Why Your Semantic Cache Is Returning Wrong Answers

A user reported that our translation plugin was returning the same cached result for every request, regardless of input. We investigated and found something worse: 95% of all semantic cache hits across our platform were false positives. 199 different translation requests, 198 unique request bodies, one cached response served to all of them.

If you care about long-lived agent state and production request handling, this post pairs well with Why Your AI Agent Keeps Losing Its Memory, the one-key chatbot guide, and the AI API rate limiting guide.

Key Takeaways

The Bug Report

The report was simple: "I disabled semantic cache, but every translation returns the same result."

Three request IDs, three different translation segments, identical cached responses. The request bodies ranged from 1,564 to 8,676 bytes. The cached response ID was the same across all of them: chatcmpl-DG6J03nhdvcF7Ek0C8rJkjh7lN9pF.

First suspicion: the user's cache settings weren't being applied. That turned out to be a separate data-source sync bug (the admin panel wrote to one table, the API gateway read from another). Fixing that only solved half the problem. Even with cache enabled and working correctly, the semantic cache was matching requests that should never match.

The Production Data

We pulled 24 hours of cache hit data from ClickHouse. The numbers were bad.

Model	Total Requests	Cache Hits	Unique Requests	Unique Responses	Hit Rate
DeepSeek V4 Flash	200	199	198	1	99.5%
glm-4.6-thinking	100	38	13	1	38%
gpt-5-nano	31	29	28	2	93.5%
gpt-oss-120b	18	17	17	1	94.4%
qwen3-vl-flash	17	16	16	1	94.1%

198 unique translation requests, all returning the same single cached response. That's not a cache. That's a broken function that returns a constant.

Every affected model shared two traits: all requests came from a single user, and all used a fixed system prompt template with varying user content. For a current list of models available on the platform, TokenLab's model directory (observed 2026-07-07) is the source of truth since lineups change often.

How to Detect This in Your Own System

You don't need our logs to find out if you have the same problem. The fastest signal is response diversity per model. If a model has a high cache hit rate but almost no unique responses, you're serving one answer to many different questions.

Here's the ClickHouse-style query we used, generalized:

SELECT
  model,
  count() AS total_hits,
  uniqExact(substring(response_body, 1, 200)) AS unique_responses,
  round(uniqExact(substring(response_body, 1, 200)) / count(), 3) AS diversity_ratio
FROM request_logs
WHERE cache_hit = true
GROUP BY model
ORDER BY total_hits DESC;

A healthy cache has a diversity_ratio near 1.0, meaning most hits return distinct responses for distinct inputs. A ratio near 0 means many requests collapse onto a handful of cached answers. Anything under roughly 0.5 on a model with real input variety is worth investigating.

If you don't log response bodies, a cheaper proxy works too: compare the count of unique request bodies against the count of unique responses served from cache. When 198 unique requests map to 1 response, the cache isn't matching meaning, it's matching boilerplate.

A second tell is user complaints that cluster on structured workloads. Translation plugins, summarizers, form-fillers, and JSON-in/JSON-out tools are the usual suspects because they wrap variable content in a fixed template.

Why Embeddings Fail on Structured Input

The translation plugin sends requests like this:

System: "Act as a translation API. Output a single raw JSON object only.
         Input: {"targetLanguage":"<lang>","title":"...","segments":[...]}"

User:   {"targetLanguage":"zh","title":"Product Page",
         "description":"Translate product descriptions",
         "tone":"formal",
         "segments":[{"text":"actual varying content here"}]}

The system prompt is identical across all requests. The user message is a JSON object where targetLanguage, title, description, and tone are fixed. Only segments[].text changes.

When our semantic cache extracts text for embedding, it concatenates the system prompt and user message. The fixed template accounts for roughly 80% of the text. The embedding model (all-mpnet-base-v2, 768 dimensions) compresses this into a vector where the template structure dominates. The actual translation content barely moves the needle.

Result: cosine similarity between "translate 'Hello world'" and "translate 'The quarterly financial report shows a 15% increase in revenue'" exceeds 0.95. Our threshold is 0.95. Every translation request matches the first cached entry.

Digging through the logs, we found three ways this breaks:

The translation plugin is the worst offender. Fixed JSON keys and values drown out the actual translation segments. DeepSeek V4 Flash and gpt-5-nano both hit this.

A context summarization assistant had a different flavor of the same problem. Its system prompt was so long that user content, ranging from 5KB to 47KB, barely registered in the embedding. That's how glm-4.6-thinking ended up returning the same summary for every conversation.

The third pattern was subtler. For gpt-oss-120b and qwen3-vl-flash, the first 500 characters of every request were byte-for-byte identical. The varying content came after, but the embedding was already dominated by the shared prefix.

What the Research Says

This isn't a novel problem. Recent papers have quantified it.

UC Berkeley's vCache project found that correct and incorrect cache hit similarity distributions overlap heavily, which means no fixed threshold cleanly separates a true match from a structurally similar false one. That finding matches exactly what we saw in production: the translation plugin's false positives clustered above 0.95, well inside the range where legitimate paraphrase matches also live.

Other recent work on semantic caching reliability reaches similar conclusions: raw embedding similarity is a necessary but not sufficient signal for cache correctness, and any production system relying on it alone should expect a meaningful false-positive rate on structured, template-heavy traffic.

The Two-Layer Fix

Layer 1 is content extraction. Before embedding, strip the fixed system prompt and template scaffolding, and embed only the variable payload: the actual segments[].text content, not the surrounding JSON keys and boilerplate. This alone dramatically increases the signal-to-noise ratio in the embedding vector.

Layer 2 is fingerprint verification. Even with better extraction, near-duplicate content can still produce high similarity scores. Before serving a cache hit, compute a fast hash (we used FNV-1a) over the extracted content of both the incoming request and the cached entry. If the hashes match exactly, serve the cache. If they don't, either fall through to a fresh completion or, for higher-value traffic, route to a cheap verification call that scores meaning, not bytes.

The mistake is skipping verification entirely and trusting raw cosine similarity. Every approach in the table beats that. Start with the cheapest one that fits your query type, and only move up the ladder when you measure real paraphrase misses.

Together, these two layers dropped our false-positive rate from roughly 95% to under 5% in the affected traffic.

When Semantic Caching Is the Wrong Tool

Caching isn't free engineering, and some workloads aren't worth caching at all.

• High-cardinality, low-repeat traffic. If almost every request is unique, one-off creative generation for example, the hit rate is too low to justify the embedding overhead. You pay to embed everything and rarely cash in.
• Outputs that must be fresh. Anything time-sensitive, live data, personalized results, anything with a "today" in it, can return stale answers from cache even when the match is technically correct. The answer was right an hour ago and wrong now.
• Strict correctness domains. For medical, legal, or financial answers, a single false positive can be worse than the cost it saved. If you cache here, the verification layer is mandatory, not optional, and an LLM-grade check may be the only acceptable one.
• Tiny prompts where the model call is already cheap. Embedding, similarity search, and verification have their own cost. If the underlying completion is a few hundred tokens on a cheap model, caching can cost more than it saves.

Caching shines on repetitive, template-heavy, expensive completions, exactly the workloads where a false positive is also easiest to introduce. That tension is why the verification layer matters. If your goal is mainly cost control, it's worth pairing caching with cheaper model routing too. The pricing comparison and the best AI models for coding guide cover where the per-token savings actually come from, and TokenLab's model directory (observed 2026-07-07) shows current options, including low-cost routing picks like DeepSeek V4 Flash and Gemini 3.5 Flash, if you're weighing which model to route cached versus uncached traffic through. Verify current pricing on the linked directory before committing to a routing plan.

Why Not Just Raise the Threshold?

Our threshold is already 0.95. Raising it doesn't help. The problem is that structurally similar inputs produce similarity scores above 0.95 no matter what the actual content says.

vCache's data backs this up: the similarity distributions of correct and incorrect hits overlap so much that no single cutoff separates them. Push the threshold to 0.99 and you'll kill legitimate cache hits for genuine paraphrases while structurally identical requests, like our translation JSON payloads, will still cluster above 0.99 regardless of content. The threshold isn't the lever. The input representation is. That's why Layer 1 (content extraction) and Layer 2 (fingerprint verification) work where a threshold bump doesn't: they change what gets compared, not how strict the comparison is.

If you're building or maintaining a semantic cache, treat the threshold as a coarse filter, not a correctness guarantee. Pair it with content extraction so the embedding actually represents the variable part of the request, then add a cheap verification step so a near-miss embedding match can never silently become a wrong answer in production.

Get Started with TokenLab's model directory to compare current pricing and benchmarks across frontier, coding, and low-cost routing models before you wire up your cache verification layer. Whichever model sits behind your completion endpoint, the extraction-plus-fingerprint approach is what actually fixes false positives.

FAQ

Does raising the similarity threshold fix semantic cache false positives? No. Research from vCache and related studies shows correct and incorrect hit distributions overlap across the threshold range, so pushing the cutoff higher blocks legitimate matches without reliably filtering out structurally similar but semantically different requests.

What's the cheapest way to verify a semantic cache hit? A fingerprint hash (FNV-1a or similar) over the extracted, meaningful content adds under a millisecond of latency and is free to compute. It won't catch paraphrases, but it eliminates exact false positives like the ones described here, which is where most of the damage in structured workloads comes from.

Does this problem depend on which model serves the completion? No, the false-positive issue lives in the embedding and matching layer, not the completion model. Any model behind a semantic cache, whether it's DeepSeek V4 Flash, glm-4.6-thinking, or something newer, will be affected the same way if the cache embeds fixed template text alongside variable content. Check TokenLab's model directory (observed 2026-07-07) for current model availability when deciding which models to route through a cached pipeline.