The same question, four thousand ways

The model bill came in higher than the hosting bill. Not by a little.

The product was ordinary: a help widget in the corner of a dashboard, wired to a model with the documentation pasted into its system prompt. It worked, people used it, and every message went to the model at full price.

So I exported a week of traffic and read it. Four thousand messages. I stopped counting distinct questions at around ninety.

Three of them, from three different users, on the same afternoon:

• "how do i turn on 2fa"
• "Where is the setting for two-factor authentication?"
• "I want to add an extra login step to my account, is that possible"

One question. Three bills.

Why the obvious cache misses

A cache is a note you leave yourself so expensive work does not happen twice. Think of a receptionist with a folder of printed answers, filed by the exact question people ask. Somebody asks that question, they hand over the sheet. It costs nothing.

The filing is the whole trick, and it is where this breaks. The folder is filed by wording. Ask "where is the 2FA setting" when the sheet is filed under "how do i turn on 2fa" and the receptionist finds nothing, so you go upstairs to the expert and wait.

flowchart LR
  A["how do i turn on 2fa"] --> H1["key: 'how do i turn on 2fa'"]
  B["Where is the setting for<br/>two-factor authentication?"] --> H2["key: 'where is the setting for...'"]
  C["I want to add an extra<br/>login step to my account"] --> H3["key: 'i want to add an extra...'"]
  H1 --> S["Cache"]
  H2 --> S
  H3 --> S
  S --> R["3 keys. 3 misses on first ask.<br/>3 separate stored answers."]

Software caches have exactly this problem, and normally it does not matter, because normally the key is a URL or a user id and those repeat character for character. Sentences typed by people never do.

Attempt one: the exact string

Start with the version that is dumb and correct, because it is the floor everything else has to beat.

const key = `qa:${sha256(question)}`;
const hit = await redis.get(key);
if (hit) return hit;
 
const answer = await model.complete(question);
await redis.set(key, answer, { ex: 86400 });
return answer;

This is a real cache. It is never wrong: if the key matches, the question was byte for byte identical, so the stored answer is exactly the answer that question would get. Nothing here needs tuning and nothing here can leak a wrong answer.

I ran it against the week of real traffic. Hit rate: 2.1 percent. Almost all of that was one person hammering the send button.

Attempt two: tidy the string first

The obvious next move is to stop treating trivial differences as differences. Lowercase everything, strip punctuation, collapse whitespace, drop filler words, sort what is left alphabetically.

That took the hit rate from 2.1 percent to 3.4 percent, and it is worth understanding why so little.

String tidying measures spelling, not meaning. It fails in both directions at once:

Every string metric, including edit distance, sits on that same broken axis. Two sentences that share no words can mean one thing, and two sentences that differ by one word can mean opposite things. You cannot climb out of this by being cleverer about characters, because characters are not what you are trying to compare.

Attempt three: ask the model what it means

So ask the thing that does understand meaning. Send the question to a small, cheap model, have it return a canonical phrasing, and cache on that.

const intent = await small.complete(
  `Rewrite this support question as a short canonical intent.\n${question}`
);
// "how do i turn on 2fa"                -> "enable two factor authentication"
// "Where is the 2FA setting?"           -> "enable two factor authentication"
const hit = await redis.get(`qa:${sha256(intent)}`);

flowchart TD
  A["how do i turn on 2fa"] --> S["Small model:<br/>'write the canonical intent'"]
  B["Where is the 2FA setting?"] --> S
  S --> O1["Mon: 'enable two factor authentication'"]
  S --> O2["Tue: 'turning on 2FA'"]
  O1 --> K["sha256(intent)"]
  O2 --> K
  K --> R["Different strings.<br/>Different keys. Miss.<br/>And you paid a model call to find out."]

This works, sort of, and it is the answer most people reach for first. It has two problems, and the second one is fatal.

The first is cost and latency. A model call now happens on every request, including hits, so the response can never be faster than that call. You have made requests cheaper without making them fast.

The second: the output is a string, so you are back to exact matching, one level up. The same question yields "enable two factor authentication" on Monday and "turning on 2FA" on Tuesday, and those are different keys. Temperature zero narrows the wobble but cannot remove it, because the inputs genuinely differ.

The instinct is right and the output type is wrong. You want meaning as a position, not as a sentence. Positions can be near each other. Sentences can only be equal or not.

What a vector actually is

Picture a supermarket. Every product has an aisle number and a shelf height, so two numbers place any item in the shop. Milk and yoghurt end up close together, and neither is near the light bulbs. Nothing about their names put them there; what they are put them there.

An embedding is that idea with more room. You hand a sentence to an embedding model and it hands back a fixed-length list of numbers, typically 768 or 1536 of them rather than 2. That list is a vector, and the whole point of the model is that sentences meaning similar things come back as nearby positions.

flowchart TD
  A["how do i turn on 2fa"] --> M["Embedding model"]
  B["Where is the setting for<br/>two-factor authentication?"] --> M
  C["how do I export invoices"] --> M
  M --> V1["0.12, -0.83, 0.41, ..."]
  M --> V2["0.11, -0.80, 0.44, ..."]
  M --> V3["-0.55, 0.21, -0.09, ..."]
  V1 --> Z["A and B land beside each other.<br/>C lands somewhere else entirely."]
  V2 --> Z
  V3 --> Z

The extra dimensions are what make it work. Two numbers only give you aisle and shelf. A thousand give the model room to record politeness, tense, negation, topic and specificity all at once, each as its own direction, and that is what it learned to do from a very large amount of text. The idea goes back to word2vec; today's models come from OpenAI, Voyage, or an open one you run yourself through sentence-transformers, and you can compare them on the MTEB leaderboard.

"Nearby" needs a number, and the standard one is cosine similarity. It asks whether two vectors point the same way and answers between -1 and 1, so same direction is 1 and unrelated is around 0. The full definition is one line of trigonometry.

The cache that works

Now the pieces fit together. Embed the question, look for the nearest stored question, and if it is close enough, serve its answer.

flowchart TD
  Q["User question"] --> E["Embed<br/>~15ms, ~1/500th of a completion"]
  E --> S["Nearest-neighbour search<br/>over stored question vectors"]
  S --> T{"similarity at or above threshold?"}
  T -->|"yes"| HIT["Return the stored answer<br/>no model call"]
  T -->|"no"| MISS["Call the model"]
  MISS --> A["Answer"]
  A --> W["Store vector + question + answer"]
  A --> U["Return to user"]
  W -.->|"available to the next<br/>question that means this"| S

That is a semantic cache, and there is nothing exotic underneath it. In Postgres it is pgvector, which is one extension and one column:

create extension if not exists vector;
 
create table answer_cache (
  id          bigserial primary key,
  question    text        not null,
  answer      text        not null,
  embedding   vector(1536) not null,
  created_at  timestamptz not null default now()
);
 
create index on answer_cache
  using hnsw (embedding vector_cosine_ops);
 
-- <=> is cosine distance, so 1 - it is similarity.
select answer, 1 - (embedding <=> $1) as similarity
from answer_cache
order by embedding <=> $1
limit 1;

The index is the load-bearing line. Without it that query compares against every stored row, which is fine at ten thousand and hopeless at ten million. hnsw builds the graph from the 2016 nearest-neighbour paper and does an approximate search instead: almost always the true nearest vector, in about a millisecond. Neon ships it, and so does every dedicated store like Qdrant or Redis.

On my week of traffic: 68 percent hit rate, 40ms on a hit against 2.3 seconds on a miss. The bill stopped being the interesting line on the invoice.

The threshold is the whole design

Every number above depends on one constant I have not justified, and it is the only genuinely hard decision here.

Set the threshold high and near-identical questions miss, so you pay for answers you already have. Set it low and the cache confidently returns the wrong answer. Those failures are not symmetric, and this pair is why:

Nothing about a similarity score separates those. A missed hit costs a fraction of a cent. A false hit tells a locked-out user to check an inbox they cannot open, and that is the kind of wrong answer people screenshot.

So do not guess the number. Take a few hundred real question pairs, label them by hand, sweep the threshold, and look at what each setting actually does:

flowchart TD
  A["Label ~300 real pairs:<br/>same answer / different answer"] --> B["Sweep the threshold from 0.80 to 0.98"]
  B --> C["Plot two curves:<br/>missed hits and false hits"]
  C --> D{"Pick where the false-hit<br/>curve is still flat at zero"}
  D --> E["Ship that, minus a safety margin"]
  E --> F["Log every hit with its score.<br/>Re-sweep monthly."]

There is a second move worth knowing, because it turns the threshold from one cliff into two. Serve confidently above the high line, miss confidently below the low line, and in the uncertain band between them, ask a cheap model one yes-or-no question: would these two get the same answer? You pay the extra call only on the small fraction of traffic that is genuinely borderline.

Serve confidently above the high line, miss confidently below the low line, and escalate only what sits between them:

flowchart TD
  S["similarity score"] --> A{"score"}
  A -->|"0.95 and above"| HIT["Serve the cached answer"]
  A -->|"0.86 to 0.95"| J["Ask a cheap model:<br/>'same answer for both? yes/no'"]
  A -->|"under 0.86"| MISS["Call the full model"]
  J -->|"yes"| HIT
  J -->|"no"| MISS
  J -.-> N["~6% of traffic lands here,<br/>so the extra call is rounding error"]

The key is not just the question

Here is what I got wrong in production, and it is the failure an interviewer will push you towards.

Two identical questions do not always deserve the same answer. Change the system prompt, the model, the model version, the retrieved documents, the language, or the account asking, and the correct answer changes while the question does not move a millimetre.

flowchart LR
  subgraph K["Cache key"]
    direction TB
    NS["Namespace hash:<br/>model id + version + system prompt<br/>+ tenant + locale"]
    VEC["Question vector"]
  end
  NS --> F["Search only inside<br/>this namespace"]
  VEC --> F
  F --> H["Hit"]

Hash everything that is not the question into a namespace, and search vectors only inside it. Two consequences fall out for free. Editing the system prompt invalidates the whole affected cache automatically, because the namespace changes, which is exactly what you want and exactly what a hand-rolled purge always forgets.

And tenant separation stops being a tuning knob. A cache shared across accounts can serve one customer's answer to another, and answers on a support bot contain account details. That is not a bad hit rate, it is a data leak, so the tenant goes in the namespace and stays there.

Answers go stale

The last one is the oldest problem in the discipline. A cached answer is a snapshot, and some facts move.

Sort questions into three buckets and give each a different TTL:

Time is the blunt instrument. The sharp one is tagging: record which documents were used to build each answer, and when a document changes, purge every entry tagged with it. That way an answer lives until the thing it was derived from moves, which is the actual condition you care about.

flowchart LR
  D1["docs/security.mdx"] --> A1["cached answer: 2FA"]
  D1 --> A2["cached answer: session length"]
  D2["docs/billing.mdx"] --> A3["cached answer: invoices"]
  E["security.mdx edited"] --> P["purge where tags contains<br/>'docs/security.mdx'"]
  P -.->|"evict"| A1
  P -.->|"evict"| A2
  P -.->|"not tagged, kept"| A3

Live and personal questions need a hard exclusion, not a short TTL. Anything reading the current user's data or the current time is not cacheable across users at all, and a semantic cache that cannot tell the difference will eventually serve one person's balance to somebody else.

The short version

• Exact-string caching is correct and nearly useless on human input. Mine hit 2.1 percent. Measure yours before designing anything, because it is the floor you have to beat.
• String normalisation compares spelling. Meaning is not spelling, in both directions, so it buys almost nothing.
• Do not ask a model to write down the meaning. Its output is a sentence, and sentences can only be equal or not. Ask for a position instead.
• An embedding is that position: a fixed-length list of numbers where similar meanings land nearby. Compare two with cosine similarity, store them in something with a vector index, and the lookup costs milliseconds.
• The threshold is the design, not a detail. Label real pairs and sweep it. Escalate the uncertain band to a cheap yes-or-no check rather than picking one cliff.
• The key is the question plus everything else that changes the answer: model, version, system prompt, locale, tenant. Tenant especially, because that mistake is a leak and not a miss.
• Give every answer a TTL by how fast its facts move, tag entries with the documents they came from, and never cache anything personal or live.