Infrastructure, Observability & LLMOps

The first time Cortex silently regressed, no test failed and no alarm fired. A one-word edit to a system prompt — “concise” swapped for “brief” — dropped ticket-resolution quality by nine points across a single mid-market tenant, and it took nineteen hours and an angry Slack from the customer before anyone noticed. Nothing was down. Everything was worse. That gap between “up” and “correct” is the entire discipline of this chapter, and it is why an AI startup cannot run on the DevOps playbook alone.

9.1LLMOps vs MLOps vs DevOps#

Every generation of software operations inherited the last one’s tools and then found a failure mode the old tools were blind to. DevOps made deploys fast, reversible, and observable, but it assumed the code you shipped was the code that ran: deterministic, testable with assertions, green or red. MLOps added the model as a versioned artifact — training pipelines, feature stores, drift detection — because a model’s behavior lives in weights, not source, and weights rot as the world shifts underneath them.¹ LLMOps is the third layer, and it exists because the thing you operate is neither your code nor your model: it is a prompt-plus-context-plus-frontier-API system whose most important component you did not train, cannot inspect, and do not control.²

11The uncomfortable truth of LLMOps is that your core dependency ships a new version on its own schedule, deprecates the old one, and never guarantees the numbers stay the same. You are operating someone else's model in production.

Three properties break the inherited playbook. First, non-determinism: the same input yields different outputs, so assertEqual is useless and “correctness” is a distribution, not a boolean.³ Second, cost is per-token and unbounded at request time: a single malformed prompt or a runaway agent loop can 10x a bill in an afternoon, so spend is a first-class runtime signal, not a monthly finance report.⁴ Third, the model is exogenous: providers deprecate gpt-5.2, silently re-tune a safety filter, or degrade under load, and your quality moves without a single line of your code changing. DevOps has nothing to say about any of these. LLMOps is the practice of making a non-deterministic, metered, externally-controlled system behave like production infrastructure anyway.

The founder mistake is assuming LLMOps is a tool you buy. It is a discipline spanning four surfaces — versioning, observability, the caching/routing gateway, and reliability engineering — and the rest of this chapter is those four surfaces. For the model-routing strategy (which model for which job, buy-vs-host) see Chapter 7; for the eval metrics that decide whether a change is better see Chapter 10; for unit economics see Chapter 13. This chapter owns the plumbing.

9.2Prompt and config versioning and deploys#

A prompt is code. It has inputs, control flow (few-shot examples, conditionals baked into templates), a runtime contract with the model, and it breaks silently when edited. Yet the default failure of early-stage AI teams is to treat prompts as content — a string edited in a dashboard, a Notion doc pasted into a config, a value someone changed on a Friday. Cortex’s nineteen-hour regression was exactly this: an untracked, unversioned, un-evaluated edit to production behavior. The fix is prompt CI/CD: every prompt change is versioned, evaluated against a fixed dataset, gated on quality, and rolled out progressively — the same lifecycle as a code deploy.⁵

22The versioned unit is never the prompt alone. It is the tuple: prompt text + model id (claude-opus-4-6, not claude-opus) + temperature + tool schema + max tokens. Change any element and behavior can move. Pin all of them together or you have versioned nothing.

Two architectures exist, and you will use both. Prompts-in-Git stores prompt templates as files (YAML or plain text) in your repo, syncs them to a registry, and pins versions in production for reproducible builds — LangWatch’s Prompts CLI and PromptHub’s Git-style branch/commit/merge model are built for this.⁶⁷ Prompts-in-registry stores them in a managed system (Langfuse, Braintrust) that non-engineers edit through a UI, with the runtime fetching by label (production, staging) at call time.⁸ Git gives you review, blame, and atomic rollback tied to code; the registry gives you fast iteration without a deploy and lets a support lead fix a prompt at 2 a.m. The right answer for most startups: author in Git, deploy through a registry, treating the registry as a cache of what Git already reviewed.

The non-negotiable is the quality gate. When a prompt changes, an automated eval runs against a golden dataset and reports which cases improved and which regressed; the merge blocks if quality falls below threshold.⁵ Langfuse emits a repository_dispatch event (or a signed webhook) on every prompt version change, triggering your CI to run that eval before the new version is allowed to carry the production label.⁹ This is the wall that would have stopped Cortex’s “concise → brief” edit.

33In the wildLangfuse 4 - In 2026 Langfuse shipped OpenTelemetry-native tracing and formalized a prompt CI/CD flow where a production label is applied only after a webhook-triggered eval passes, making "promote the prompt" a gated pipeline step rather than a dropdown change. See langfuse.com.

Deploy the same way you deploy code: canary a new prompt version to a small traffic slice, watch the eval and cost signals, then widen; A/B two versions live when you are choosing between them; keep the prior version one label-flip away for instant rollback.¹⁰ Cortex now ships prompts like code: each version is a Git-reviewed YAML file with a pinned model and params, promoted through staging → 5% canary → 100% only after the golden-set eval clears, and any tenant can be pinned to a known-good version if their traffic regresses. The nineteen-hour incident is now a five-minute label revert.

9.3Observability: tracing, logs, token and cost metrics#

You cannot debug what you cannot see, and an LLM request is nearly opaque by default: a string goes in, a string comes out, and everything that determined quality — which prompt version, which retrieved chunks, which tool calls, how many tokens, at what cost, in how many milliseconds — is invisible unless you instrument it. The unit of observability is the trace: a tree of spans covering the full request path, where each span carries the model, token counts, latency, and cost of one operation.¹¹ For an agent like Cortex a single ticket is one trace with a dozen child spans — retrieve knowledge-base docs, call the LLM to plan, invoke the refund tool, call the LLM again to draft the reply — and the trace is the only way to answer “why was this answer wrong and this one expensive.”

The 2026 standard that makes this portable is the OpenTelemetry GenAI semantic conventions. The GenAI SIG defined a common vocabulary of gen_ai.* attributes — model name, input/output token counts, finish reason, tool calls — so a span from a LangGraph agent looks identical to one from a raw Anthropic call. OpenTelemetry’s CNCF graduation in May 2026, plus OTLP ingestion across vendors like Datadog and Grafana, made it the default wire format rather than a niche choice.¹²¹³ Most conventions are still experimental, but they are stable enough to build on, and standardizing here means you can swap observability backends without re-instrumenting your code. Instrument with an open standard — OpenLLMetry or OpenLIT emit OTel spans into whatever backend you already run — and you are not locked to one vendor’s SDK.¹⁴

44Emit OTel GenAI spans, not vendor-proprietary events. The instrumentation is the expensive, invasive part; the backend is a commodity. Own the former, rent the latter, and keep the freedom to move.

Three signals matter above all. Traces for debugging causality. Token metrics — input, output, and cached tokens per span — because tokens are the currency of both cost and latency. And cost, computed from tokens times the per-model rate and attributed down to per-user, per-tenant, and per-feature.¹⁵ A production LLMOps dashboard shows total spend and calls, cost over time, cost by model, cost by operation, average cost per call, and cost per user — the last of which catches the one tenant whose usage pattern is quietly eating your margin.¹⁶ Cortex dashboards cost per resolved ticket per tenant and pings sales when a tenant’s number crosses its plan price; the profitability math itself is Chapter 13’s.

9.4Caching, routing and fallbacks in production#

Between your application and the frontier providers belongs exactly one component: an LLM gateway. It puts a single OpenAI-compatible endpoint in front of many providers and centralizes caching, routing, fallback, spend limits, and observability, so those concerns live in one governed place instead of being reimplemented in every service.¹⁷ Building these piecemeal across your codebase is the most common infrastructure mistake an AI startup makes; the gateway is the correction.

Caching is the highest-leverage cost lever you own, and there are two kinds. Provider prompt caching discounts the repeated prefix of a request — a large system prompt, a fixed knowledge base — and in 2026 the economics are dramatic: Anthropic charges 1.25x base input for a 5-minute cache write and just 0.10x for every cache read, a 90% discount, with OpenAI’s automatic caching matching that 90% read discount on its newest flagships.¹⁸¹⁹ For an agent like Cortex, whose every call carries the same multi-thousand-token support playbook and tenant context, structuring the prompt so stable content sits in the cacheable prefix is the difference between viable and ruinous margins. Gateway caching adds exact-match caching (identical request → stored response) and semantic caching (a fuzzy-matched, near-identical prompt served from cache) — Portkey and Cloudflare AI Gateway have the most mature implementations, while LiteLLM notably does not do semantic caching.¹⁹²⁰

Routing decides which provider serves each request — by price, latency, throughput, or capability — and fallback decides what happens when the first choice fails. The strategy is Chapter 7’s territory; the mechanism is the gateway’s, and it is what turns provider fragility into a non-event. Configure an ordered fallback chain (primary model → same model on a second provider → a different model family) with a tight health-detection window and idempotency keys to avoid double-charging on retries.²¹ The arithmetic is compelling: two providers each at 99.53% availability, combined with automatic failover, reach above 99.99% — cutting expected downtime from roughly 41 hours a year to about 12 minutes.²¹

55In the wildPortkey open-sources the gateway - In March 2026 Portkey released its full production gateway under Apache 2.0, and by May 2026 it was the most feature-complete open-source option — 1,600+ models across 250+ providers, semantic caching, guardrails (PII redaction, jailbreak and prompt-injection filters), and versioned prompt management in one self-hostable proxy. See wavect.io.

Cortex fronts everything with a self-hosted gateway: exact + semantic cache on informational queries, provider prompt caching on the fixed playbook prefix, and a fallback chain that fails over between Anthropic and a second provider on a 5–15 second health signal, with idempotency keys so a mid-flight retry never issues a refund twice.

9.5Reliability, rate limits and incident readiness#

AI is now infrastructure, and it fails like infrastructure — except the failures are stranger. 2026 made this concrete: on June 2, Anthropic’s Claude had a major global disruption with elevated error rates across Opus 4.6, the API, and Claude Code; in the nine months to July 2026 OpenAI logged roughly 166 incidents, averaging about 18 a month, with four outages inside a single four-day window in July.²²²³ If your product’s core loop runs through one provider’s API, your uptime is capped by theirs, and theirs is not 99.99%. The multi-provider failover of the previous section is not a nice-to-have; it is the only way to offer an SLA you can keep.

66The June 2026 Claude outage is the industry's "AWS us-east-1" moment for AI — the day a critical mass of companies discovered that a single provider's incident was now their incident, with no failover in place. Do not learn this lesson in production.

Rate limits are the failure you will hit first and most often. Providers meter you across usage tiers that rise with spend, and a traffic spike or a retry storm will hit 429 long before an actual outage.²⁴ Handle them at the gateway: exponential backoff with jitter, a token-bucket limiter you enforce yourself so you throttle gracefully instead of hammering a limit, request queuing for non-interactive work, and load-balancing across multiple provider accounts or regions to widen your effective ceiling. Naive retry-on-429 without backoff makes the incident worse — you DDoS your own provider relationship.

Then there is the failure mode unique to AI: the model is up but wrong. Latency crept up, a provider silently swapped a model revision, a prompt regressed, hallucination rate climbed — none of it trips a 500. This is why the three alerts every LLM team needs by tomorrow are not the classic ones: alert on cost-per-request spiking (a runaway agent loop or a prompt-injection attack), on latency-per-token degrading (provider trouble before it becomes an outage), and on quality/eval score dropping on live traffic (the silent regression).²⁵ Standard APM watches for down; LLMOps must also watch for worse.

Be sober about putting an LLM in your own on-call loop. Many 2026 reliability teams wired a model into triage and alert summarization, but research shows most frontier models degrade significantly within roughly 1,000 tokens of input — long before their advertised context limit — so a model asked to reason over a fat incident dump is wrong more often than teams assume.²⁶ Use LLMs to summarize and suggest in your incident response; keep a human on the decision.

77In the wildThe four-day July 2026 OpenAI cluster - Four significant OpenAI outages landed inside a single four-day window in July 2026, part of ~166 incidents over nine months. Teams with a configured fallback chain rode them out as elevated latency; single-provider teams took the full downtime. See thenextweb.com.

Incident readiness for an AI product means a runbook that assumes provider failure as a when, not an if: documented failover you have actually rehearsed (a fire drill that forces traffic to the secondary), a per-tenant kill switch to pin or disable a misbehaving prompt version, cached and degraded-mode responses for when everything upstream is down, and a status page that distinguishes “down” from “degraded.” Cortex’s runbook does all four; when Anthropic degraded on June 2, Cortex’s traffic shifted to its secondary provider inside the health-check window and customers saw slower replies, not silence. Full crisis management — comms, postmortems, customer trust — is Chapter 15’s subject; the engineering readiness is yours to build before the crisis, not during it.