Taste Profile

I send myself Instagram Reels constantly — a skincare product to look into, a pour-over recipe, an edit whose pacing I want to steal — and then never open them again. That bothered me, because sending something to yourself is a much stronger signal than liking it. There's an intention behind every one: I found this interesting enough to want to come back to it. I had a few hundred pieces of high-intent content sitting in a DM thread doing nothing. Taste Profile turns that pile into something I can actually ask questions of.

Capture: The whole thing runs off the iOS share sheet. I hit share on a reel, pick the shortcut, and it asks one question — why are you saving this? — before anything else happens. That answer is the most valuable field in the entire system: it's the only part of a saved reel that records intent rather than content. "More things I want" and "for the espresso setup" send the same video to completely different places.

I. The only question it asks

II. Four actions, no app to install

The shortcut POSTs the URL and the note to a Next.js route on Vercel. That route canonicalises the link — share sheets hand over URLs decorated with tracking parameters, and the same reel shared twice should collapse into one row, not two — and drops it onto a queue in Supabase. It replies with one plain-English sentence instead of JSON, because whatever the endpoint returns is literally what appears in the notification banner on my phone, and a wall of punctuation is a bad thing to see after saving something.

Why a queue and not just do the work: the video processing can't run on Vercel at all. There's no ffmpeg binary, the filesystem is read-only, and Hobby functions are capped at 60 seconds — while transcribing a single reel takes tens of seconds on its own. So the split is fixed: Vercel enqueues in under a second, and a separate long-running Python worker drains the queue. Every stage caches to disk and is resumable, so a failure during classification doesn't re-download the video and a failure during transcription doesn't re-fetch anything.

Three signals per reel: the worker reads each reel three ways, because any one of them is regularly empty. On-screen text comes from keyframes pushed through a vision model — I sample on scene changes rather than on a fixed interval, since a 30-second reel holding one static shot gives ten near-identical frames while a fast-cut reel misses every actual cut, and each frame costs a vision call. All frames go up in one request, which is roughly twelve times cheaper than per-frame calls and lets the model use continuity between frames to read text that's occluded in any single one. Audio goes through Whisper running locally — no key, no per-reel cost, which turned out to be the right default when the first real reel I tested came back with zero characters because it had no speech in it. Captions come along with the post. Add the note I typed on my phone, and that's what the classifier sees.

The classifier proposes, the taxonomy decides. This was the failure mode I was most worried about: left alone, a model will happily produce "coffee," "espresso," "Coffee Gear," and "brewing" as four separate clusters for the same idea, and the graph fragments into uselessness. Existing labels go into the prompt with an instruction to reuse them — but every label that comes back is then pushed through a deterministic resolver anyway, which normalises it, checks an explicit alias table, and fuzzy-matches against labels that already exist before it's allowed to mint a new one. Nothing in that resolver calls a model. Every new label that does get minted is logged to a drift file, one line each, so fragmentation is something I measure by counting events rather than by squinting at the graph.

The output is plain markdown. Each reel becomes a note with YAML frontmatter, the extracted products, the caption, the on-screen text, the keyframes copied in so the note isn't a wall of text about a video I can't see, and links to its clusters. Obsidian's backlinks do the rest — the graph below is just what falls out of 88 reels linking to 16 cluster notes. Nothing draws it; it's a side effect of the file format.

III. One cluster, and everything filed into it

What it's actually for. The queries are the whole point: what skincare products have I sent myself in the past week to try? What coffee methods should I attempt this weekend? Give me the links to the reels I saved because I wanted to make something similar. Those are questions I couldn't answer at all before — they were answerable in principle by scrolling a DM thread for twenty minutes, which means in practice they were never asked.

A note on where the data comes from. I checked the compliant routes first, and none of them carry what this needs: Instagram's data export gives you a permalink, a timestamp, and a handle but no caption; the DM webhook payload has no caption field defined at all; and the oEmbed endpoint returns the caption but prohibits persisting or deriving from it by name. So ingestion reads the post directly. Because that's the part most likely to need replacing, all of it is quarantined into two modules that nothing else imports — swapping them for a compliant source has to be sufficient — and provenance is recorded per field rather than per record, so "which of this did a scrape produce, which did a model infer, and which did I decide myself?" stays a one-line query instead of an archaeology project.

IV. Whether the worker choked on it

Built with Cursor and Claude Code, on Supabase and Vercel, with a Python worker doing everything the serverless half can't. 88 reels ingested so far across 16 clusters, and 153 tests that run with no network, no API keys, and no binaries installed.