Prompt-cache hit rates collapsed on any request that passed through one code path, with no error and no log line. Parsing the request into a generic map, appending a message, and re-serializing produced byte-for-byte different output from byte-for-byte identical meaning — and a prefix cache only cares about bytes.
Wallets in production showed the impossible triple: deposited $0, balance $0, spent $10.53 — and kept serving. A clamp in the deduction routine absorbed every overshoot, and a write-behind cache turned the resulting lie into a renewable resource.
SageMaker’s built-in autoscaling metric counts invocations per instance, not the compute behind them. A burst of long-generation requests looked identical to a burst of trivial ones, and the endpoint scaled three minutes too late every time — for exactly the traffic that mattered most.
Users were intermittently forced to re-authenticate from scratch, for no visible reason. A background refresh and a 401-triggered refresh could race each other, and the refresh endpoint rotates the token on every use — so the loser presented a credential the winner had already spent.
Every insert failed with ‘no unique or exclusion constraint matching the ON CONFLICT specification’ — against a column that demonstrably had a unique index on it. ON CONFLICT does arbiter inference, and a partial index only qualifies if the statement’s own predicate implies the index’s.
An SSE resynthesis path emitted a whole thinking block without any of the delta events a spec-compliant client uses to build it. Clients reconstructed an empty block with a valid signature, saved it to disk, and every future replay of that conversation was rejected forever.
Folding a revoked key’s usage into an open invoice lost money under concurrency. Fixing the double-billing created silent under-billing; fixing that covered one of three race paths; and the last one was a plain lost update that a status guard cannot see.
A benchmark’s judge prompt labelled one answer as coming from the treated arm and printed the exact compression ratio applied to it. A judge primed to expect degradation will find it — and small differences between configurations turned out to be smaller than the judge’s own noise floor.
An on-premise instance whose telemetry uploads kept failing began returning 503 and never stopped. Restarting did not help; shipping a new image did not help. The health anchor was a latched flag, and the only code that cleared it could no longer run.
A published cost-and-accuracy comparison had to be retracted and restated on a smaller n. A third-party search API ran out of quota mid-sweep, and the two arms being compared died at different points in their run — so neither the sample set nor the count of dropped questions matched.
The Deployment spec showed exactly the image we intended. The pods had been serving an 18-hour-old build whose digest had already been deleted from the registry. A tag is a name; with imagePullPolicy IfNotPresent it is also a cache key.
A repository client’s internal lock worked fine in isolated tests and deadlocked or raised confusingly under a real application’s request lifecycle. It had been created once, at object construction, capturing whatever event loop happened to be running at that moment — not the one that would later serve requests.
A malformed config patch froze the entire proxy, permanently, because every request path calls Current() under the same lock. The first fix removed the self-deadlock and introduced a leaked write lock on both error paths — the common paths for a bad patch.
A hot-path call configured with a timeout could hang indefinitely. A server dribbling one byte before each read-timeout window resets the clock every time — the timeout never fires, because it was never bounding the thing that needed bounding.
A container-configuration gate failed images that passed when scanned by tag, and passed images it had never looked at. It took five attempts, each of which fixed something real and left the gate still capable of reporting on an artefact nobody asked about.
An async client call raised TypeError about unexpected arguments, only on the async path. A subclass had defined a method with the exact same name as a base-class transport method but a completely different signature, and a type: ignore comment was sitting right on top of it.
Time-to-first-token equalled full generation time for every streaming request, for months. A fast pass-through path existed and was correctly written — gated on a flag that had been set unconditionally on every request, so the branch was unreachable code nobody noticed.
Detecting a specific kind of tool call in a streaming response with a raw substring search caused duplicate upstream billing whenever the model merely mentioned the tool’s name in prose, and missed real calls whose name was split across a chunk boundary.
A config persistence routine used the standard temp-file-then-rename pattern for an atomic write. On Linux, where the system temp directory is routinely a different filesystem from the target, the rename fails outright — and the failure path was the one that caused the deadlock in an earlier story.
Usage events drained from a Redis Stream into the billing table. A helper returned an empty list on any exception, and the caller read emptiness as permission to XTRIM the stream to zero. Dashboards and invoices were quietly low, with no error anywhere.