AI Fundamentals I: Models, Modalities & Inference

You do not need to derive backpropagation to build an AI company, but you do need a mechanical model of what happens between a user’s request and a billed invoice — because that model is your product’s cost structure, its latency budget, and half its defensibility. This chapter is that physics: the smallest complete set of ideas about models, modalities, and inference that lets a non-researcher founder make correct architecture and cost decisions under pressure.

We build the picture bottom-up — tokens, transformers, modalities, the taxonomy of models you actually choose between, and the economics of running them — and we keep coming back to Cortex, our running example: an autonomous customer-support agent for mid-market SaaS companies. Cortex reads a support ticket, checks the customer’s account, and either resolves it or escalates. Every abstract idea here becomes, for Cortex, a line on a cost sheet.

2.1From tokens to transformers, for founders#

A language model does exactly one thing: given a sequence of tokens, it predicts a probability distribution over the next token. Everything else — chat, code, agents — is scaffolding around that single primitive, repeated. This loop is where your bill comes from.

A token is a chunk of text, roughly 3-4 characters or about 0.75 English words. 11A blunt but load-bearing heuristic: 1 token ≈ 4 characters ≈ 0.75 words. 1,000 English words ≈ 1,350 tokens. Code, JSON, and non-Latin scripts tokenize worse — often 1.5-3x more tokens per visible character. The model never sees letters; it sees integer token IDs produced by a tokenizer. This matters commercially for two reasons. First, you are billed per token, not per word or per request — so the shape of your data determines your margin. Cortex’s tickets are cheap prose, but the account JSON, API logs, and knowledge-base articles it stuffs into context are token-dense; a naive “just include everything” prompt can quadruple the input bill. Second, tokenization is language- and format-specific: the same Cortex flow costs materially more to serve a German or Japanese customer than an English one, purely from tokenization overhead. Bake that into pricing before a design partner in Munich becomes your worst-margin account.

The dominant architecture remains the transformer, and the one mechanism a founder must internalize is attention: to predict the next token, the model weighs its relevance against every prior token in the sequence. That “against every prior token” is the source of both the technology’s magic and its cost curve. It is why context is powerful — the model can genuinely reason over a whole ticket thread — and why long context is expensive and slow, a tension we quantify later in this chapter.

Two facts fall out of autoregression and govern your entire cost model. Input tokens are cheap; output tokens are expensive — typically 4-5x the price — because input is processed in one parallel pass while output must be generated serially. And you cannot fully predict a request’s cost in advance, because output length is emergent. A support agent that “thinks out loud” or over-explains is not just annoying; it is a variable-cost leak on every ticket.

2.2Modalities: text, vision, audio, multimodal#

A modality is a type of data the model can consume or produce: text, images, audio, video. The 2026 frontier is natively multimodal — a single model that ingests and reasons across text, images, and audio in one forward pass, rather than bolting a separate vision or speech model onto a text core. This is a genuine capability shift, and it reshapes what products are buildable.

Modalities are priced and metered differently, and conflating them is a classic founder error. Images are converted to tokens by tile — a high-resolution screenshot can cost hundreds to low-thousands of input tokens. Audio is metered per audio token or per minute of stream, at a steep premium over text. OpenAI’s GPT-Realtime-2, shipped May 2026, runs roughly $32 per million audio-input tokens and $64 per million audio-output tokens — an order of magnitude above text — with cached audio input far cheaper.¹ Google’s Gemini 3.1 Flash exposes a Live API for streaming voice and vision, with audio output around $20 per million tokens.² Voice is not “text with a microphone”; it is a different cost class.

For Cortex, the modality decision is concrete. Most tickets are text: chat transcripts, emails, form fields — the cheap, obvious backbone. But a meaningful slice include a screenshot (“here’s the error I’m seeing”), and a native-vision model that reads the screenshot directly, versus a human tagging it or a separate OCR service, is the difference between one-pass resolution and a multi-hop pipeline. The team’s decision: text is the default path; vision is invoked conditionally, only when an image is attached, so the expensive modality is billed only when it earns its keep. Voice — a customer phoning support and Cortex answering live over a real number, now trivial via the Realtime API over SIP — is deferred to a later phase, precisely because at $32/$64 per million audio tokens it would dominate the cost model before the text product is proven.

2.3Open vs closed, dense vs mixture-of-experts, reasoning models#

You are not choosing “an AI.” You are choosing a point in a three-axis taxonomy, and each axis independently moves cost, control, and capability. (Which specific provider and model to sign with is Chapter 7’s job; here we build the map so that choice is informed.)

Axis 1: open-weights vs closed / API#

Closed models (accessed only via a paid API — the frontier Claude, GPT, and Gemini families) give you top capability with zero infrastructure, but you rent them: the vendor sets the price, owns the roadmap, can deprecate a model under you, and sees your traffic. Open-weights models (weights you download and self-host — the DeepSeek, Qwen, Llama, GLM, and Kimi families) give you control, data residency, and a per-token cost that trends toward raw compute, but you own the serving problem: GPUs, autoscaling, evals (see Chapter 6), uptime. The 2026 reality is that the open-weights frontier is genuinely close on many tasks. The gap is no longer “toy vs frontier”; it is “who carries the operational burden.”

Axis 2: dense vs mixture-of-experts#

A dense model activates all its parameters for every token. A mixture-of-experts (MoE) model is partitioned into many “expert” sub-networks, and a lightweight router activates only a few per token — so a model can be huge in total knowledge yet cheap per token to run. DeepSeek’s V3-lineage models are the canonical illustration: 671B total parameters but only ~37B active per token.⁴ GLM-5.2’s 744B total run with ~40B active.³ For a founder this is not trivia: MoE is why a “very large” model can be economically servable. The total parameter count tells you roughly how much the model knows; the active count tells you roughly what it costs and how fast it runs. When you read specs, look for both.

Axis 3: reasoning vs fast#

The most important 2026 axis. A reasoning model spends extra inference compute — generating hidden or visible “thinking” tokens — before its final answer, trading latency and cost for accuracy on hard, multi-step problems. This is test-time compute: for genuinely hard tasks, accuracy scales with thinking budget, and a smaller model given room to think can beat a larger one that answers instantly.⁵ Every major lab now exposes this as a knob — Claude via a budget_tokens parameter, Gemini via a thinkingBudget — so reasoning is a dial you set per request, not a separate product.⁵ Latency has come down hard: reasoning queries that took 30-120 seconds in early 2025 now resolve in 3-15 seconds for mid-tier models.⁵ But the thinking tokens are billed, and the single biggest cost mistake of 2026 is routing every request through a reasoning model when a fast model would have nailed it.⁵

Here is where Cortex’s team makes its central technical bet. They segment tickets by difficulty and route accordingly.

The lesson generalizes: do not pick “a model” for your product; build a routing layer and pick a model per request class. Reasoning is a scalpel, not a default. Running all of Cortex’s tickets through a reasoning model would roughly 5-10x the blended cost for a single-digit accuracy gain on the easy 85% — the definition of burning money and latency.⁵

2.4Context windows, tokens and the economics of inference#

The context window is the maximum number of tokens — input plus output — the model can attend to at once. Frontier and leading open models now reach 1M tokens.³ A million-token window sounds like an invitation to stop thinking about what you include. It is a trap.

Three costs scale with how much context you actually use. Money, because you pay per input token every call. Latency, because prefill time grows with input length. And quality, because models exhibit degraded recall in the middle of very long contexts — the “lost in the middle” effect — so more context can mean worse answers, not better. A 1M-token window is a capability ceiling, not an operating point.

The economics of a single request decompose cleanly, and this is the mental model to carry:

request cost ≈ (input_tokens  × input_price)
             + (output_tokens × output_price)   # incl. reasoning tokens
             + (cached_tokens × cache_read_price)  # if caching

Two levers dominate real bills. Prompt caching: providers let you cache a stable prefix — a long system prompt, a fixed knowledge base — and pay a deep discount, typically ~90% off, on subsequent reads of those tokens. Anthropic charges roughly 1.25x the base rate to write a 5-minute cache and 0.10x to read it thereafter.⁶ Batch APIs: non-realtime work run through a batch endpoint runs ~50% off input and output at both major closed providers, and critically the discounts stack — Sonnet-tier batch-plus-cache-read has been quoted around $0.15 per million input tokens, ~95% off list.⁶

For Cortex this is decisive. Every ticket call includes a large, near-identical preamble: policies, tone guidance, tool schemas, and top knowledge-base articles — easily 8,000-15,000 stable tokens. Caching that prefix turns it from a full-price cost on every one of tens of thousands of daily tickets into a ~90%-discounted read. And the nightly job that re-embeds the knowledge base and back-tests Cortex against yesterday’s resolved tickets is pure batch work — no user is waiting — so it runs at half price. The team’s rule: realtime, uncacheable tokens are the only truly expensive tokens; engineer everything else toward cache reads and batch windows. (The full unit-economics build-out — blended cost per resolved ticket, gross margin, the effect of a bad-answer escalation — is Chapter 13.)

2.5Latency, throughput and where the compute goes#

Users feel latency; your CFO feels throughput. Both are governed by the two-phase structure of inference, and this is the section most founders skip and later regret.

An inference request runs in two phases with completely different physics (Figure 2.1). Prefill processes the entire input prompt in a single parallel pass to build the model’s internal state; it is compute-bound and its duration sets your time to first token (TTFT) — the “is it alive?” delay before anything appears.⁷ Decode then generates output tokens one at a time; each step is memory-bandwidth-bound (the whole model must be read from GPU memory per token) and its speed is measured as time per output token (TPOT) or its inverse, tokens per second.⁷

The founder consequences are direct. Long inputs hurt TTFT — Cortex stuffing a giant knowledge base into every prompt makes the agent feel sluggish before it says a word, which is exactly what caching (which lets prefill reuse cached state) and retrieval (fetch only relevant chunks — see Chapter 3) exist to fix. Long outputs hurt total latency and cost linearly, so a terse, well-instructed Cortex is both faster and cheaper than a chatty one — a rare case where the better UX is also the better margin. And reasoning models pay their tax in the decode phase: those thinking tokens are decode-phase output, which is why reasoning simultaneously raises latency and cost (Figure 2.1).

Throughput is the other side of the same coin. Providers serve many requests at once by batching them through the GPU, which is why your per-token price can be low despite frontier hardware — the vendor amortizes each GPU across many concurrent users. The 2026 landscape has specialized silicon competing hard on these numbers: Groq’s LPUs lead on consistent low-latency decode for open models, Cerebras leads raw tokens-per-second on small-to-mid models, and GPUs dominate compute-heavy prefill and high-batch decode.⁸ You will rarely touch this on a closed API — but the moment Cortex considers self-hosting an open-weights model (Chapter 7), TTFT, TPOT, and achievable batch size are the build, and the choice of accelerator is a first-order cost decision.

The synthesis for a founder: your latency and your cost are the same physics viewed from two ends. Input length drives prefill, TTFT, and input cost. Output length drives decode, total latency, and output cost. Caching attacks prefill; brevity and routing attack decode; batching attacks throughput. Cortex’s entire performance-and-margin strategy is those four moves applied to those two phases — nothing more exotic. Master this loop and you can reason about any model, any provider, and any bill you will ever see.