How to build a robust AI agent

Very little of what follows is about the model. The prompt turned out to be the smallest part of the work.

Building an agent is not primarily an LLM problem. It is a systems engineering problem in which one component happens to be probabilistic. Almost all of the difficulty — and almost every bug that reached a user — lived at the boundary between that component and a system that has to be correct: the tool that writes, the cache that lies, the error frame that retries, the constraint the model trades away under pressure.

Written from nine months of building and operating an LLM agent that takes real actions on real user data. The examples throughout use an AI shopping assistant — the shape of agent most brands are now asking us about — but every lesson, bug and number came from production.

The numbers here are what worked in one system, not settings to copy. Every threshold, cadence and floor below was tuned against one product's data and one traffic pattern. Take the reasoning; re-derive the value against yours.

The order below is roughly the order you'll need to solve these problems. Parts I and II are the build. Parts III and IV are what separates a demo from something you can leave running.

Contents

Part I — Foundations

1. Picking models
2. Context engineering
3. Retrieval (RAG)
4. Tools and the action loop
5. Structured output
6. Memory and sessions

Part II — Correctness

7. Evals
8. Idempotency and write safety
9. Tool output validation
10. Learning from feedback

Part III — Safety

11. Input validation
12. PII, in and out
13. Untrusted content and prompt injection
14. Abuse and exploitation detection
15. Enforcing product constraints

Part IV — Operations

16. The reliability stack
17. Streaming
18. Proactive and scheduled work
19. Model migration
20. Cost, limits, and scale ceilings

Part I — Foundations

1. Picking models

Don't pick a model. Pick one per job.

The most common early mistake is choosing a single model for the whole product. A shopping assistant does several unrelated things — hold a conversation, extract a structured order intent, read a product photo, summarise reviews, classify a returns request — and those jobs have different requirements for accuracy, latency, cost and determinism.

Build a feature-to-model routing table as a first-class piece of config. In this project it holds twelve entries. Each declares:

Field	Why it varies
provider	Failover across vendors, and different vendors lead on different modalities
model	Capability tier — you don't need a frontier model to parse JSON
temperature	Extraction wants ~0.1. Conversation wants ~0.7. Never one global value
maxTokens	A JSON verdict needs 400. A generated gift guide needs 8,000
reasoningEffort	Only for reasoning-family models — see section 19

Concrete shape of the decisions:

• Extraction and classification → cheapest capable model, temperature ~0.1, tight token cap. "Add two of the medium in black" becomes a structured cart mutation this way, and it runs on almost every turn.
• Conversation → mid-tier model, temperature ~0.7, generous cap. This is what customers judge the product on.
• Vision → whichever vendor currently leads. Visual search — "find me something like this" from a photo — lives or dies here. Test, don't assume.
• Safety classifiers → cheapest model clearing your accuracy bar. They run on every request; cost compounds.
• Long structured generation → highest token ceiling you can afford. Truncation mid-JSON is far worse than a slightly expensive call.

Set token ceilings with headroom

Output is billed on tokens actually emitted, so a generous ceiling costs nothing on simple inputs and prevents truncation on complex ones. A truncated JSON response is a hard parse failure; headroom is free insurance.

One caveat: reasoning tokens count toward the output budget on reasoning-family models. Switching a feature to one makes its existing ceiling too low.

Build the failover chain at the same time

Every feature needs an ordered fallback chain that crosses vendors, not just models:

primary → same-vendor cheaper → different vendor → different vendor

Provider-wide outages happen. A chain that stays inside one vendor is not a chain.

Record which model actually served each request. When quality moves, the first question is always "was that the primary or a fallback," and you can't answer it retroactively.

Watch the seams between provider abstractions

A subtle failure class: a shared helper that is internally single-provider but takes a feature argument defaulting to a value that routes to a different provider's model. Every call site that omitted the argument requested a model that vendor doesn't have — a 404 on each.

Seven call sites had this bug here. It's invisible in code review because the call looks correct.

If a helper is provider-specific, make the provider explicit in its name or make the argument required. Don't let a default silently cross a vendor boundary.

2. Context engineering

This is where most of the quality lives, and it gets far less attention than prompt wording.

Before every turn, the agent is handed a structured briefing — not a blob, but a deliberately ordered, budgeted set of sections.

What goes in

Organised by how it's used, not by where it's stored:

• Identity and state — who this is, how long they've been a customer, where they are in their relationship with the brand. Account age matters: the tone that suits a fifth order is wrong on a first visit.
• Locale — timezone, local time, currency, region, shipping market. An assistant that quotes dollars to a customer in Singapore, or says "good morning" at 11pm, has broken the illusion permanently.
• Goals and current standing — what they're shopping for, their size profile, any budget they've stated, and what's in the cart right now.
• Recent activity — a window of what they've browsed, carted, bought and returned, for continuity and to avoid recommending what they already own.
• Stated preferences — extracted facts and preferences (section 6).
• Derived patterns — behavioural classifications computed offline, not inferred live: price sensitivity, brand affinity, return propensity.
• Conversation history — a bounded window of recent turns, plus summaries of older sessions.

Budget it explicitly

Context grows without bound if you let it. Set a token budget for the whole briefing — ~3,000 here — and define a truncation order in advance: which sections drop first when the budget is exceeded.

Derive that order from importance, not convenience. Conversation history, engagement stats and calibration data went first here. Identity, locale, the current cart and safety-critical constraints were never truncatable.

Without an explicit order, whatever happens to be last in your string concatenation is what gets cut. That is not a design decision.

Fetch it in parallel

The briefing assembles from many independent queries — profile, cart, order history, preferences. Collapse them into concurrent batches. Three sequential tail queries were costing ~200ms on every turn here until they were merged — pure latency, no benefit.

Structure the message array properly

Briefing in the system role. User content in the user role. Tool results in the tool role. Assistant turns in the assistant role.

This sounds obvious and was wrong here for months: the continuation call after tool execution flattened everything into a single user message. It worked — responses stayed coherent — which is exactly why nobody noticed.

Role structure is how the model distinguishes instruction from data. Collapsing it silently weakens every safety property you believe the system prompt gives you.

3. Retrieval (RAG)

Retrieve facts; don't ask the model to recall them

The largest single accuracy gain in this project came from removing a task from the model, not improving how it was asked.

A vision model was asked to look at an image and report a number about the thing in it — a fact, not a description. Error was large and roughly constant. Prompt engineering moved it a few percent, because the problem wasn't phrasing — the model was being asked to recall a number it had never reliably memorised.

Splitting the job dropped error fourfold:

• The model identifies — what is this, and what structural cues can it actually see.
• The database supplies the numbers — retrieval over an authoritative dataset, keyed off that identification.

For a shopping assistant the split is obvious once you see it. Ask a model what a photographed jacket costs and it will guess. Ask it what the jacket is — category, material, silhouette, colour — and let the catalogue supply the price, the variants and the stock. If your product depends on facts that exist in a dataset, the model's job is to find the right row, not remember its contents.

Naive top-1 retrieval is not enough

"Embed the query, take the nearest neighbour" was not accurate enough. What worked:

1. Query expansion. Several search variants per item — brand name, the colloquial name customers actually use, a component or material description, the regional term (trainers, sneakers), the abbreviated form, the generic category. Real catalogues index the same product under wildly different labels.

This costs a small model call per item and it is load-bearing. An experiment removed it to save latency and the pipeline got slower — downstream stages iterated through more candidate sets to find a match. Measure before you cut it.

2. Batch embedding with a tiered cache. In-memory map → persistent cache table → provider API. Identical input always yields an identical vector, so this is shared system state — see section 12 for the privilege bug that caused.

3. Broad retrieval, then rerank. Vector search returns a broad candidate set — ~20 here — which a cross-encoder reranks down to ~5. Bi-encoder retrieval is fast and imprecise; cross-encoder reranking is slow and precise. Use each where its cost profile fits. Running the reranker locally removes a network hop and a per-call charge from the hot path.

4. Merged inference. Rather than three calls to pick a product, pick a variant and settle a quantity, do all three in one call with the candidates in context. Fewer round trips, and the decisions come out mutually consistent rather than contradicting each other — no more "the blue one" resolving to a colour the chosen product doesn't come in.

5. Calibration and confidence. Apply known systematic corrections, then emit a numeric confidence. Below a threshold, ask a clarifying question instead of answering confidently. "Did you mean the wool coat or the wool-blend?" costs one turn. Adding the wrong one to the cart costs the sale.

Prioritise sources by context

Not all sources are equally relevant to every customer. The order that worked here, in store terms: a source matching the user's own locale (the catalogue for their market and currency) → a curated domain-specific source (the brand's own edits) → the largest general source (the full catalogue) → everything unfiltered (marketplace listings).

Falling through in a defined order beats searching everything and hoping ranking sorts it out.

Convert estimation into comparison

For the hardest sub-problem — reading a hard fact off an image — what moved accuracy was including calibrated reference images in the prompt.

The shopping version: "what shade is this" invites a hallucinated answer. "Which of these swatches is it closest to" is a comparison task, which models are markedly better at. The same holds for fit, finish and fabric weight.

Generalised: wherever a reference set exists, turn estimation into comparison or classification.

Skip the pipeline when you can

Before any of the above, check whether this work has already been done — a similarity match against the customer's own history returns instantly and free. Replenishment is the obvious case: "the same moisturiser as last time" is a lookup, not a search. In products where customers repeat themselves, this absorbs a large fraction of traffic.

Optimise the call you don't make before the model you call.

Keep safety-critical domain data out of the model

Some classifications are too important to infer per-request. Allergen matching here is a hand-built ontology — canonical buckets and a curated synonym list — not a model call. For a beauty retailer, ingredient and allergen matching is exactly this shape.

It's deliberately conservative: certain ingredients hard-exclude a whole category even where the mapping is arguable, because a false positive costs one recommendation and a false negative costs much more.

Asymmetric-cost classifications belong in a reviewed lookup table, not in a probabilistic component.

4. Tools and the action loop

Tools are how an agent stops being a chat window and starts doing things — adding to the cart, checking stock, tracking an order, starting a return. They are also your entire attack surface (section 13) and your entire data-integrity risk (section 8).

Writing a tool definition

The description is a prompt, not documentation. The model chooses tools by reading it.

• Enumerate the trigger phrasings. Not "adds an item to the cart" but "use this when the customer wants to buy something — includes any variation of 'add X', 'I'll take X', 'put X in my bag', 'get me X', 'two of those'." Models match on surface similarity more than you'd like.
• Use enums for closed sets. Sizes, colours, shipping methods. Don't accept free-text and validate after.
• Make nullability explicit — type: ["string", "null"] with a description of when null is correct.
• Say what the tool does not do. "Add to cart" and "save to wishlist" are the classic pair. Ambiguity between two similar tools is the main source of wrong selection.

Design the execution loop for the client, not just the model

The naive loop — model returns tool calls → execute all → send results back → stream reply — produces a UI that sits silent for several seconds.

Restructure so the response opens before execution:

model call (decides tools)
  └─ open stream
       for each tool:
           emit "tool_starting: <name>"   ← client shows "Checking stock…"
           execute
       stream the final reply

Emit one event per tool, immediately before it runs — not all up front. Batching makes the client flicker through labels and settle on the last, which is the wrong one.

Return structured results, not prose

Every tool returns a typed envelope: success flag, structured data, source identifier. Two consumers depend on it — the model, which summarises it, and the client, which renders it as a UI component: a product card, an order status, a returns label.

Maintain an explicit whitelist of tools whose results the client renders. Getting this wrong is invisible: nine tools here had payload parsers, card widgets and dispatch logic all shipped and working, but the server never forwarded their results, so the cards had never once rendered. A missing array entry, alive for months, erroring nowhere.

Some tools should deliberately not render a card — for a wishlist save the model's one-line confirmation is better than a widget. Decide per tool and write down why.

Ground the agent's claims in tool output

An agent with tool results in context will still occasionally state numbers that aren't in them — a price, a stock count, a delivery date.

A post-generation citation validator extracts every numeric claim, checks it against what the tools actually returned, rewrites mismatches to within tolerance, and replaces unsupported claims with a hedge. "Arrives Thursday" only survives if a tool said Thursday.

This requires tool results to carry provenance — a source_id and source_type — so the validator knows what a claim may be checked against.

Placement constraint: it runs before persistence, and on a streaming path it can only log, not rewrite. You cannot un-send tokens.

5. Structured output

Anywhere the model's output is parsed rather than displayed, treat the schema as production interface.

Schema strictness rules change under you

A production incident here: a provider tightened strict-mode function-calling validation so that every nested object now required an explicit "no additional properties" declaration. A tool parameter that had been valid for months started returning HTTP 400 on every single turn. Nothing in the codebase had changed.

Two takeaways:

• Schema validation rules are versioned by the provider, not by you. They can tighten without a migration path.
• Prefer flat schemas. The workaround was changing a nested-object parameter to a JSON string with a parse step. Less elegant, dramatically more portable across providers and versions.

Always have a parse fallback

Structured generation fails in predictable ways: truncation mid-object (raise the token ceiling — section 1), markdown fences around JSON, and prose preamble before the object. Strip fences, extract the outermost braces, then parse. When parsing still fails, that's a retryable error, not a terminal one.

Validate parsed output against a schema you own

Provider-side schema enforcement is a hint, not a guarantee. Validate the parsed result against your own schema before it reaches business logic — see section 9 for what to do when it fails.

6. Memory and sessions

Context (section 2) is what the agent knows this turn. Memory is what survives it.

Type it, source it, rank it

Every memory carries:

• Type — preference, situational context, feedback on a past recommendation, or stated fact.
• Source — explicitly stated vs. inferred by the model. Treat them differently; inferred memories should be easier to overwrite.
• Importance, 1–10. Critical constraints — sizes, an ingredient allergy, a hard budget ceiling, a brand they refuse to buy — at 9–10. One-off remarks at 1–2.

Importance drives injection: high-importance memories go into context under a heading telling the model to actively reference them; lower ones go in a general "other things I know" block; below a floor they aren't injected at all.

Extract on a cadence, not every turn

Every-message extraction is expensive and noisy. Every ~3 messages worked here, with session summaries every ~10.

Count from the persisted message count, not the array length in the current request. The in-context array is truncated to a window, so counting it fires extraction far more often than intended — a real bug here.

Deduplicate before writing

Without dedup you accumulate ten near-identical phrasings of one preference and burn context on all of them. Embed the candidate and compare by cosine similarity — ~0.88 worked here. Above the threshold, update rather than insert.

Let the agent write its own memory — with a cap

A tool letting the model explicitly record something beats relying only on background extraction, because the model knows what mattered in a way a separate pass has to guess.

Cap it — 2 writes per 30 seconds here — and keep the background extractor as a safety net.

Tier memory by how it's used

Core (stable facts — sizes, allergies), episodic (things that happened — the return last month), procedural (how this customer likes to be handled — short answers, no upsell). This makes retrieval and expiry policies tractable — episodic memories should age out, core ones shouldn't.

Sessions give you continuity across conversations

A session groups messages and carries a title and an AI-generated summary. Summaries are what let a new conversation reference an old one without replaying its full transcript.

Two things that bit here:

• Enforce one active session per user at the database level — a partial unique index on (user_id) WHERE is_active. Concurrent requests otherwise create duplicates, and every subsequent read picks an arbitrary winner. Recover from the unique-violation by re-reading the winner rather than erroring.
• Deactivate stale sessions on fresh app launch. Otherwise a client that was killed mid-conversation resumes into a session the server considers abandoned.

Generate titles from the first few messages or the first tool call, and make the write idempotent so it doesn't churn.

Part II — Correctness

7. Evals

Build this before the hardening work. Every subsequent change then has a regression signal, and you will make many.

Structure

• Cases as data, not code. Line-delimited JSON, one case per line, grouped into files by category.
• Categories mirroring your risks. Here: tool correctness, safety, groundedness, conversational continuity, injection resistance, product-constraint compliance, access gating.
• Deterministic assertions where possible. Did it call the right tool? With the right argument shape? Did output contain / not contain specific strings? Cheap, fast, reliable — prefer them.
• LLM judge only for what you can't assert — tone, helpfulness, whether a product explanation is actually correct.
• Cache judgements keyed by input, so reruns are nearly free.
• Keep a baseline file and compare against last known-good rather than an absolute score.

Full run here: ~$0.05, about two minutes. Cheap enough to run before every commit touching the agent path, which is the point.

Verify your harness can fail

An early version asserted on a shape that never matched. Roughly six assertions were no-ops — passing because they never evaluated.

Deliberately break something and confirm the relevant case goes red. An eval suite that can't fail is worse than none, because it manufactures confidence.

Calibrate the judge before trusting it

The step most teams skip. An LLM judge is a measurement instrument, and an uncalibrated instrument produces confident nonsense.

Hand-label a set of outputs, then compute against those labels:

• Cohen's κ, quadratic-weighted, for ordinal scores
• Spearman ρ
• Mean absolute error
• A confusion matrix

Set a floor and don't let the judge gate anything until it clears it. κ ≥ 0.6 is the conventional bar for substantial agreement — one of the few numbers in this guide that comes from the literature rather than from tuning.

Hand-labelling is manual and unavoidable. A few hours, once.

Non-LLM evals are underrated

Not every eval needs a model. A plain unit harness asserting the shape contract of every tool result — each tool × success × failure × malformed variant — ran 69 assertions in seconds at zero cost here, catching a class of bug an LLM judge would never look for.

Same for multimodal: a few fixture images (a clean product shot, one with adversarial text overlaid, a confusing distractor) run against your classifier asserting the verdict. Cents per run.

Keep eval traffic out of production telemetry

Eval runs make real model calls. Without a flag that short-circuits your logging, every run pollutes the tables you use to reason about real usage — and your cost dashboards.

8. Idempotency and write safety

Retries, reconnects, stream recovery and customer double-taps all replay turns. Any tool that writes needs an idempotency key. In a shopping assistant the writes are the ones that cost money: cart mutations, order placement, return initiation.

1. Client generates a request id per turn, sends it with the message.
2. Server validates the format.
3. Thread it into every write.
4. Partial unique index on (user_id, request_id) WHERE request_id IS NOT NULL — nullable rows exempt, so existing data passes unchanged.
5. On unique-violation, return the existing row instead of erroring. The caller wanted the write to have happened. It has.

Suffix the key per tool. One turn writing through two tools — add to cart, then apply a saved address — collides on a shared key. Derive {base}-{toolName}-{index}.

Don't advertise idempotency you don't have. Some operations genuinely can't be. Say so in the tool description and leave the field out of the schema — a schema accepting a request id the handler ignores is worse than none, because callers trust it.

Worth having regardless: a short time-window duplicate check (same customer, same content, within ~60s here) as a second net for clients that send no key.

9. Tool output validation

Tool results feed back into the model's context. Anything malformed becomes input to the next generation.

Validate every result against a per-tool schema and quarantine failures before they re-enter the loop.

Quarantine must be distinguishable from success

The critical detail, learned the hard way.

The first implementation returned a safe-looking {success: true, data: {confirmed: true}} for a quarantined write. Downstream logic keyed on that flag and proceeded as though the write had happened — awarding downstream side effects for an operation that never occurred. In a shopping assistant that is an order confirmation for an order that was never placed.

Safe-looking is not the same as correct. For writes, emit a failure downstream code reads unambiguously as "did not persist."

Log level should differ by risk: error for a malformed write, warning for a malformed read.

One useful property: because a quarantined result is always your own sanitised stub and never raw output, downstream consumers — including a client rendering tool results — can't leak internals from a malformed response.

10. Learning from feedback

An agent that never improves from use is a static product with a language interface. Closing the loop is what makes it an agent worth maintaining.

Capture explicit feedback with enough context to act on

A thumbs up/down needs, at minimum: the response id, what kind of response it was, an optional structured reason, a truncated prompt preview and response preview. Previews are what make the data reviewable later; truncate them (500 chars here) so the table doesn't become a PII liability.

Learn from corrections, not just ratings

Explicit ratings are sparse. Corrections are dense and higher-signal — when a customer changes the size you recommended, or swaps the colour, they've told you precisely what was wrong.

The pattern here: record each correction against a category, maintain a per-user calibration factor via exponential moving average (α ≈ 0.3), apply it automatically once there are enough samples (3+), and clamp the adjustment to a sane range (0.5×–2.0×) so one outlier can't distort everything. Fit is the natural case — a customer who sizes up in one brand's knitwear three times has told you something the size chart can't.

Read the aggregated factor first, fall back to raw per-session correction data for categories that haven't accumulated enough. Clamp on display too — showing a customer a wild calibration number is alarming and usually wrong.

Optimise behaviour with a bandit, per user

For choices with no ground truth — how to frame a complementary item, when to mention a promotion, whether to lead with reviews or with specification — a multi-armed bandit beats a hardcoded rule.

What made it work at small scale:

• Per-user posteriors, not global. Beta(α, β) per user per strategy. This needs no cross-user volume to be useful — it's learning this customer's preferences, which is the thing you actually want.
• Optimistic prior (α=2, β=1) so untried strategies get explored.
• Auto-suppression. After enough pulls at a low reward rate, suppress that strategy for a fixed window rather than continuing to sample it.
• An explainer. A "why are you seeing this" string, for UI transparency and for your own debugging.

Attribute the reward back to the decision

The part that's easy to miss: the feedback row must carry which strategy produced the response. Without that attribution the bandit has ratings it can't assign to arms, and learns nothing.

Practically: stamp the arm identifier onto the response when it's generated, and have the feedback endpoint look it up and store it alongside the rating.

Also budget for arm-name migrations. Renaming a strategy strands its history. Pass both old and new identifiers as candidates during a rollout so existing customers' learned preferences aren't reset.

Part III — Safety

11. Input validation

Before anything reaches a model, and separately from PII handling (section 12).

• Length caps on text. A hard character limit on any user string entering a prompt (~10k here). Without it, a single request can consume your whole context budget and a large part of your bill.
• Image constraints — file size, pixel dimensions, and an allowed MIME type list. All three, not just size: a small file can decode to enormous dimensions. Visual search makes this a public upload endpoint.
• Strip image metadata. EXIF carries GPS coordinates and device identifiers. If you store or forward customer photos, strip it.
• Reject rather than truncate, for structured input. Silently truncating a field that overflows produces a subtly wrong result that looks successful.
• Validate identifiers as identifiers. Anything the model supplies that will be used as a key — a request id, an order id, a SKU — gets format-validated before it touches a query.

Injection detection on input is worth having but is not a control by itself — see section 13 for why the real defence is structural.

12. PII, in and out

PII must be handled on both directions of every model call, plus everywhere either direction gets written down. A shopping assistant sees more of it than most agents — addresses, card fragments, order numbers tied to names.

Inbound — before it reaches the provider

Scrub user content before it enters the prompt: emails, phone numbers, payment card numbers, national identifiers, postal addresses.

Tune the patterns against your actual data. A real bug: the national-identifier regex was loose enough to match ordinary numeric sequences in legitimate content, silently corrupting them before the model saw them. In a store, that is order numbers and SKUs. The fix required the delimiters that make the pattern unambiguous.

Over-aggressive scrubbing is a correctness bug, not a safe default. Test the scrubber against real inputs and confirm it leaves them intact.

Outbound — before anything is persisted

The half that gets forgotten. Model output can contain PII that came from context, or that the model reconstructed.

Scrub at every persistence site. Here that was four: streaming and non-streaming paths × two providers. Missing one means a code path quietly persists unscrubbed text — and it will be the path you use least and audit least.

Enumerate your persistence sites explicitly and check them off. Four is more than you'd guess.

Logs are a persistence site

Logs leak more PII than databases, because nobody applies the same care.

• A safe logger wrapper that scrubs before writing, used everywhere instead of raw console calls.
• Mask identifiers rather than logging them whole.
• Redact by key name, not just by value pattern — a SENSITIVE_KEYS list applied when serialising objects, so a nested token or address field is removed regardless of whether its value looks sensitive.
• Truncate long strings. Full prompts and responses in logs are a liability with no operational benefit.
• Serialise errors safely. Error objects carry request bodies, headers and credentials in their properties.

Use the right privilege level for shared state

A cache keyed on a hash of input yields identical values regardless of who asked — system state, not user state.

This codebase wrote to such a cache with a user-scoped database client. Row- level security did exactly its job and silently denied every write. Reads succeeded — against a cache that was permanently empty — so every single request paid full price for work that was supposed to be cached once. Nothing surfaced for months. The client returned errors as values rather than throwing, the code never looked at the value, and the only symptom was a bill that was higher than it should have been, which nobody had a baseline to notice.

Two lessons: match privilege level to the nature of the data, and watch for clients that return errors instead of raising them — a try/catch around those is dead code.

Give users an export and a delete path

If you retain conversation data, you need both. Two details that matter:

• A transactional cascade. Deleting a customer touches dozens of tables. Do it in one transaction with per-table counts returned, not a sequence of independent statements that can half-succeed.
• Anonymise rather than delete where you have consent to retain training data — strip identifiers, keep the content. Make that consent an explicit, revocable setting.

13. Untrusted content and prompt injection

Anything a user wrote that ends up in a prompt is untrusted: names, free text, past messages, retrieved documents, text inside images. For a shopping assistant the list is longer than you'd think: product reviews are user content, supplier-written descriptions are third-party content, and a photographed label is text you didn't write.

Spotlighting — and escaping the delimiters

Wrap untrusted content in explicit delimiters and add a system-prompt section instructing the model to treat anything inside as data, never instructions:

<untrusted_content source="product_review">
...review text...
</untrusted_content>

Then escape the delimiter characters inside the wrapped content. Spotlighting is worthless if a reviewer can close the envelope themselves and write instructions after it. HTML-escaping < and > closes that hole.

This is one change, not two. The envelope without the escaping gives you the appearance of protection and none of the substance. Add an eval case that specifically attempts a closing-tag breakout.

Sanitise before storage, not just before display

Content entering long-term memory should have instruction-like patterns stripped at write time. Otherwise you've built a persistent injection vector: text written once, replayed into the system prompt on every future turn.

Canary tokens

Embed a per-instance random token in the system prompt and screen all output for it. If it appears in a response, the prompt was exfiltrated — and without this you'd never know.

Screen at every persistence site, and on streaming, screen per chunk with a hold-back buffer (section 17).

Pre-screen multimodal input

Images can carry instruction text. A cheap classifier before the expensive vision call catches the obvious cases.

It must fail open. A classifier outage must not break visual search entirely. Catch everything, log with a distinct error type so you can see the rate, and let the request through.

The asymmetry worth internalising

Safety classifiers fail open. Write validation fails closed.

A down classifier blocking every upload is worse than a missed detection. A malformed order treated as placed is worse than a rejected one. These pull in opposite directions and both are correct.

14. Abuse and exploitation detection

Separate from injection. Injection manipulates a single response; abuse is a pattern across requests — cost attacks, catalogue scraping, jailbreak probing, and fraud.

What to detect

Signal	Window	Detects
Velocity	60s	Scripted flooding, cost attacks
Repetition	1h	Same prompt hashed and repeated — scraping or automated probing
Jailbreak patterns	per-request	Known bypass phrasings, including role-play framings
Off-topic	cumulative	Using your assistant as a general-purpose model on your bill
Domain risk patterns	per-request	Content suggesting fraud or a compromised account

Hash prompts rather than storing them for the repetition check — a short hash against a timestamp array is enough, and keeps user content out of another table.

Graduated response, not binary block

The result should carry more than a boolean:

{ allowed, action: "allow" | "warn" | "delay" | "block", delayMs?, warningMessage? }

Escalate with accumulated violations rather than blocking on the first. Roughly ~5 violations to warn and ~10 to block worked here; tune both against your own false-positive rate before they gate anything. Someone who trips one heuristic once is usually not an attacker.

Include the warning message in the response. A silent throttle is indistinguishable from a broken product.

Jailbreak patterns need domain whitelists

Generic jailbreak regexes false-positive on legitimate usage. A phrase like "act as if" is a bypass attempt in the abstract and an entirely normal request in most product contexts — "act as if I'm buying for a friend who hates florals."

Maintain a domain-context whitelist that bypasses specific patterns when surrounding text is clearly on-topic. Without it, your abuse detector's main effect is blocking real customers.

Risk signals are not abuse signals

Some patterns are not rate-limiting problems: a sudden shipping-address change on a high-value order, gift-card-heavy baskets, the shape of card testing. If your assistant can place orders, detect these separately and route them differently — to fraud review, not to a throttle.

Two rules: alert on it — a log line nobody reads is not a control — and alert with masked identifiers and pattern types only, never message content. The alert must not become the leak. Dedupe per user per day so one account doesn't generate a storm.

State cannot live in process memory

The original implementation kept counters in a module-level map. On serverless that's per-instance, reset on cold start, blind across concurrent instances — unreliable exactly when an attack is in progress.

Persisting it has its own requirements:

• Decisions stay synchronous from the in-memory fast path; persistence syncs in the background. Never put a database round-trip in the critical path of every request.
• Serialise per-user syncs with a promise chain, or concurrent read-modify-write races lose counts.
• Merge with max, not sum — two instances reporting the same events must not double-count.
• Treat rows past the reset window as empty on read, removing the need for a cleanup job.
• Fail open on every database error. Abuse tracking must never block a legitimate request.

15. Enforcing product constraints

The single most important lesson in this document.

A prompt is an instruction, not a constraint

The system prompt framed the product's core behavioural constraint clearly and positively. It held for months. Then a situation arose where that constraint conflicted with another objective the agent was also pursuing — and the model traded the constraint away. Twice, in production, in front of users.

Picture it on a cruelty-free, vegan beauty brand. The prompt says: recommend only from the catalogue, never suggest anything with animal-derived ingredients. A customer presses for the most effective option for a specific concern. The model, trying to be maximally helpful, reaches past the promise.

Under competing objectives, the system prompt is exactly what gets sacrificed. It's a soft preference expressed in the same channel as everything else. If a behaviour is a brand promise, it needs enforcement outside the prompt.

Three-layer defence

Layer 1 — explicit prohibition in the prompt. State the constraint, list what's excluded, list acceptable alternatives, distinguish related-but-different situations.

Make sure every code path generating customer-facing text inherits it. Scheduled jobs, background generators and notification composers are easy to miss — and they're the ones nobody reviews.

Layer 2 — an output detector. A pure function, no I/O, scanning generated text:

• Word-boundary matching. Substring matching false-positives on compound words. This will bite you.
• A negation window. Look back a window before a match and a few characters after — ~40 characters worked here. Compliant text often mentions the excluded thing to steer away from it — "we don't carry anything with lanolin, but…" — and naive matching flags exactly the outputs behaving correctly.
• Carve-outs for legitimate compounds. Enumerate them.
• Automatic plural handling, or half your tokens silently don't match.

Layer 3 — eval cases. Encode the constraint as failing tests. Assert on specific phrasings rather than bare tokens, because bare tokens false-positive on compliant redirects — and let a rubric-based judge catch the looser cases a substring list can't.

One detector, several policies

• Background generation — regenerate once on violation, fall back to pre-written copy on the second failure. You have time.
• Streaming — log only. You cannot rewrite tokens already sent.
• Structured extraction — surface the detection as a field and let the client decide.

Detect and surface, don't refuse

When a model encounters input conflicting with your product's values, the instinct is to refuse. That's usually wrong for anything the customer is reporting rather than asking for.

The rule that emerged: never generate what you don't endorse, but always accept the customer's report of reality.

A customer telling the vegan brand's assistant "I've been using a non-vegan retinol from elsewhere and my skin reacted" is giving you their skin profile. A system that refuses to record it doesn't change what they buy. It makes your data wrong, and customers learn either to lie to it or to leave.

In practice: the extraction prompt was changed to surface a classification flag rather than refuse the input, and the stored record carries that flag. Generation stays constrained. Recording stays honest. Both enforced in code, not in wording.

Part IV — Operations

16. The reliability stack

Build this before the capabilities. Every feature added before the wrapper exists has to be retrofitted into it.

Every model call goes through the same chain:

timeout → retry (exponential backoff + jitter) → circuit breaker → fallback

• Timeouts are per-feature. A conversational turn and a long generation have completely different acceptable latencies. One global value is either too short for the slow path or useless for the fast one.
• Jitter is not optional. Without it, every client that failed during an outage retries at the same instant on recovery, and you cause the second outage.
• Circuit breaker. After N consecutive failures, stop trying for a fixed window, then allow one probe. Without it a provider outage becomes your outage plus a large bill for timed-out requests.
• Pre-written fallback copy per feature. Decide what a customer sees when the model is unavailable before you need it. "I can't check stock right now — here's the product page" is a product decision, not an error string.

Classify errors correctly or the retry layer is decorative

A real bug: the retry classifier matched error messages against the string "timeout". The error being thrown said "timed out". Timeouts were classified terminal and never retried — for months, invisibly, on the most common transient failure there is.

Test the classifier against the error objects your stack actually throws. Prefer matching on error type or name over message text.

Watch for errors returned as values

Some clients — database SDKs especially — return { error } rather than throwing. A try/catch around them is dead code and every failure vanishes silently. This pattern appeared at eight separate sites here. Grep for it.

17. Streaming

Everything above assumes a request that completes. Streaming breaks that assumption.

• Stall guards at three levels — time to first token, time between chunks, total duration.
• Buffer and split frames properly. Substring-matching the stream for a marker is not parsing. A real bug: checking whether a chunk contained a "done" marker false-triggered when tool arguments contained that string.
• Release the reader in a finally. An exception mid-stream otherwise leaks the lock.
• Wire an abort controller through. Cancelling downstream must actually stop generation upstream, or you keep paying for output nobody will see.

The error frame can be the bug

The most counterintuitive finding in this project.

Follow the sequence. A stream stalls partway through. The server does the responsible thing and sends an error frame. The client receives an error with zero tokens rendered, and does its responsible thing: it retries the whole message, non-streaming, with a fresh idempotency key — because as far as it knows, nothing happened. But the tools had already run before the stall. They run again. Users got duplicate writes — in a shopping assistant, that is the same item landing in the cart twice.

Every component behaved correctly by its own lights. The error path itself was creating the data corruption.

The fix: on stall, send a normal completion frame carrying whatever partial text was salvaged plus metadata describing side effects that already happened. The client renders it as an ordinary response. Nothing retries.

In a streaming system, "error" is a message with client-side semantics you may not control. Think about what the client does on receipt, not just what the frame means.

Screening streamed output needs a hold-back buffer

If you screen output for a marker of length N and forward tokens as they arrive, a marker split across two frames escapes detection. Hold back the last N−1 characters before forwarding; flush on completion. Handle the case where a stall interrupts mid-marker on salvaged partial text.

18. Proactive and scheduled work

An agent that only responds is a feature. One that initiates — back-in-stock alerts, replenishment reminders, the abandoned-cart nudge — is a different product, and a different set of problems, because nobody is waiting for the output and nobody consented to this particular message.

Timezone correctness is the whole game

Scheduled work runs on your infrastructure's clock and must land on the customer's. Run the job hourly and have it select customers whose local time falls in the target window, rather than scheduling one job per timezone.

Stagger jobs across the hour (:00, :05, :10…) so they don't contend for the same database and provider capacity.

Cap volume, and mean it

Hard limits per customer per day. The enforcement detail that matters: two concurrent job ticks can both read a count of 1 and both insert, producing 2× the cap. Enforce it in a single atomic database operation, not a read-then-write in application code. This was a real race here, fixed with an atomic insert function.

Throttle by engagement, not just by count

A customer ignoring your last ten messages should get fewer, not the same number. Compute an engagement signal and suppress low-priority sends below a threshold.

Make the suppression deterministic per customer per day rather than random, so behaviour is reproducible when you're debugging "why didn't this send."

Deduplicate at the data layer

Guard against sending the same category of message twice in a period — two back-in-stock alerts for the same product — and enforce it with a partial unique index, not just an application check. Application-level check-then-send has a race window, and scheduled jobs are exactly where concurrent execution happens.

Everything the prompt constrains, these paths must inherit

Background generators are the most-forgotten code path in an agent. They generate customer-facing text, often through a different helper than the main conversational path, and they're invisible in manual testing.

Every constraint from section 15, every scrub from section 12, every fallback from section 16 applies here too. When the constraint work in section 15 happened in this project, the scheduled jobs needed the same constraint added separately to their own generation helper — the main path's fix didn't cover them.

Always have a non-AI fallback

If generation fails or violates a constraint, scheduled messages should fall back to pre-written copy, not silence and not an error. Nobody is watching to retry.

19. Model migration

Benchmark on inputs that reflect reality

A deprecation forced a migration. A newer model was evaluated for the most expensive stage against the benchmark image the team had always used — and it came in ~30% faster at comparable cost. A clear win, ready to ship.

Then it was run against five inputs instead of one, and the result reversed completely: 20–37 seconds slower on three of five, confidence regressing on all five. The candidate only won on the single heaviest input — the one benchmark image.

The original benchmark image had been chosen precisely because it was a good stress test — which made it the least representative sample available. Shipping it would have penalised the common case to marginally win the edge case.

Benchmark against a set spanning your real input distribution. Include the easy cases — they're most of your traffic.

Assert the benchmark actually ran the path

Three configurations posted excellent latency. They were returning HTTP 400 and falling through to a cheap fallback that skipped the expensive stage entirely. The "fast" numbers measured the pipeline not doing the work.

Check error counts, assert the expensive stage executed, validate output shape. Never trust wall-clock alone.

Parameter vocabularies differ between families

Reasoning-family models reject parameters their predecessors accepted, and sibling families in the same generation accept different, non-overlapping value sets for the same parameter.

1. Build the parameter branching before you need it — pass optional parameters conditionally, use universally-accepted names. The next forced migration becomes a config change rather than a rewrite.
2. A rejected parameter looks like a fast response if you have a fallback. See above.

Migrations surface unrelated bugs

The shared-cache privilege bug in section 12 had existed for months. It only became visible during this migration, because pre-migration the calling code failed fast on a parameter error and never reached the cache write.

Expect this. Budget for it.

Deploy schema before code

Where a change spans both, apply the migration first. If code ships first, the failure is usually silent — a write that fails on an unknown column, swallowed by a fire-and-forget handler. Note the required order in the migration header, because whoever deploys it won't be whoever wrote it.

20. Cost, limits, and scale ceilings

Cost

• Track tokens and cost per call, attributed to feature and customer, with the model that actually served it.
• Cap the pipeline, not just the call. Multi-stage pipelines with retries and fallbacks fan out. A total-cost ceiling checked before every exit path — not just the happy one — bounds the worst case.
• Cache at the boundary. Deterministic inputs to expensive calls belong in a persistent cache.
• Attribute fan-out honestly. One user action here produced 4+ separate model-call log rows across pipeline stages. Anyone reading per-feature costs needs to know that's by design, or they'll read it as double-counting.

Rate limits and access tiers

Limits are per-feature, not global — visual search and a stock check have different unit costs and different abuse profiles.

Two implementation points:

• The rate limiter must fail closed. If the store backing it is unreachable, deny rather than allow. A limiter that fails open is not a limiter during exactly the incident you built it for.
• Return the reset time and remaining quota, not just a 429. A countdown is a usable product state; a bare error is not.

If AI features are gated by plan or account tier, gate them server-side at the route, and treat the client-side gate as presentation only.

Sample-size floors

An agent inferring patterns from behaviour needs a minimum before its inferences mean anything. Every behavioural feature here carries an explicit floor — pattern classification ~10 observations, anything rhythm-based a full week of activity, per-category calibration 3 corrections. The particular values matter far less than having one at all. Below the floor the feature returns nothing and says why.

• Counts survive small samples. "You've ordered this six times" is true at n=6.
• Distributions do not. "You usually shop on Sunday evenings" from six orders across 168 hourly buckets is noise presented as insight.

A feature reporting the modal bucket of a tiny sample isn't weak. It's confidently wrong, and customers act on it.

This was applied rigorously in one part of the codebase and not at all in another — the part written later, under more delivery pressure. Put the floor in the shared helper, not in each feature's own logic.

The tool-count ceiling

Tool count grew from 8 to 38 here. Past roughly 25 the selection reliability of this system degraded — plausible-but-wrong picks increased and multi-tool turns got less coherent. Where your own ceiling sits depends on how distinct the tool descriptions are, so watch for it rather than trusting the number.

A shopping assistant gets there fast: cart, wishlist, stock, orders, returns, addresses, payment methods, promotions, reviews, size guides.

Options, in order of preference: consolidate narrow tools into one with a mode parameter; scope the exposed subset by conversation context — no returns tools until there's an order to return; split into specialised agents with smaller surfaces.

Track it as a known risk with a number attached, rather than discovering it as unexplained quality drift.

What it adds up to

Very little of the difficulty in building a production agent is about the model. It's about the boundary between a probabilistic component and a system that has to be correct — and a system that moves money and ships goods has to be correct.

The recurring shape:

• Let the model do what it's good at — identification, comparison, classification, language — and supply the facts for everything else. Prices, stock and delivery dates come from the catalogue, never from the model.
• Anything you'd call a guarantee needs enforcement outside the prompt. A prompt is a preference.
• Every model-reachable surface is an untrusted API, and most of the vulnerabilities you'll find are ordinary ones.
• Handle PII on both directions and at every persistence site, logs included. Enumerate the sites; don't assume one function covers them.
• Close the loop. Corrections are denser signal than ratings, and a per-customer bandit works at scales where a global one wouldn't.
• Measure before you trust — the judge, the benchmark, the classifier, the retry logic, the rate limiter. All are instruments, and uncalibrated instruments produce confident nonsense.
• Failure paths deserve more design attention than the happy path, because that's where data corruption actually comes from.

The prompt is the smallest part of the work.