AI glossary

A token is the smallest chunk of text a language model reads and writes. In English it is often a word piece, not a whole word, and about three-quarters of a word on average, though the split depends on the tokenizer.

Think of tokens as Lego bricks. The model builds meaning from those bricks rather than from whole sentences at once.

Context limits, pricing, and memory use are usually counted in tokens. See also Context Window, KV Cache (Key-Value Cache), Inference.

Parameters (also called weights) are the huge set of adjustable numbers inside a trained model. After training, those numbers are what the model uses to turn input tokens into output tokens. A larger model has more of them.

You can picture them as dials on a vast mixing board. Training sets the dials. Inference uses the dial settings as they are.

File size grows with parameter count and with how many bits you store per number. Quantization stores those numbers with fewer bits so the same model takes less memory. See also Quantization, Open-Weight Model, Mixture-of-Experts (MoE).

The context window is how much text the model can see at once, measured in tokens. It covers both what you put in and what the model generates in that turn.

It is short-term working space, like the size of a whiteboard the model can look at while it answers.

Longer windows help with long documents, retrieval, and multi-step agents. They also cost more memory and usually more latency, because the key-value cache grows with length. See also Token, KV Cache (Key-Value Cache), RAG (Retrieval-Augmented Generation), Agentic AI / Agentic Systems.

The key-value cache is a memory store built during inference. As the model reads the prompt and then writes new tokens, it saves intermediate attention values so it does not recompute them from scratch for every new token.

It is like keeping a running notebook of what you already figured out, instead of re-deriving the whole page each time you write the next word.

That cache is why long context is memory-hungry, and why serving engines spend so much effort paging and compressing it. PagedAttention is one widely used approach. It pages the cache during serving so memory can be shared more efficiently across requests. It is a cache pager, not a different attention formula. See also the vLLM announcement and design notes. See also Context Window, Inference, Prefill vs Decode, Quantization, Transformer / Attention.

Inference is running a trained model to produce an output. Chatting with a model, generating code, or calling tools through a model are all inference.

Training is studying for the exam. Inference is sitting down and answering the questions.

For text, inference is usually autoregressive. The model predicts one token, appends it, and repeats. Cost and speed depend on hardware, batching, quantization, and how the key-value cache is managed. See also Token, KV Cache (Key-Value Cache), Quantization, Prefill vs Decode.

Quantization reduces how precisely each parameter is stored, for example from 16-bit floating point down to 8-bit or 4-bit integers. The model gets smaller and often cheaper to run. Quality usually drops a little, and sometimes more than a little, depending on the method and the task.

It is like compressing a high-resolution photo into a smaller file that still looks good enough for most uses.

People use quantization to fit larger models on smaller GPUs, to cut serving cost, and to run models locally. See also Parameter / Weights, Inference, Open-Weight Model.

An open-weight model is one whose trained parameters are published for download. Subject to the license, you can run it, inspect it, fine-tune it, quantize it, and serve it on your own machines.

The restaurant publishes the recipe. You can cook it yourself, change it, or serve it under your own roof. The license still sets the rules.

Open weights do not automatically mean open training data, open training code, or a permissive license. They also do not guarantee parity with whatever closed model shipped the same week. You gain control of hosting and data residency. You also take on serving, safety, and operations. See also Closed / Proprietary Model, Quantization, Parameter / Weights.

A closed or proprietary model keeps its weights private. You use it through the vendor’s API or app. You do not download the parameter files.

You eat at the restaurant. You do not get the recipe.

You usually get strong capability without owning hardware. You also accept the vendor’s pricing, rate limits, deprecations, safety filters, and retention policy. A common hedge is to keep prompts and evals portable, and to keep an open-weight fallback where that fits the job. See also Open-Weight Model, Agentic AI / Agentic Systems, Tool Calling.

A mixture-of-experts model is built from many specialist sub-networks called experts. For each token, a router activates only a small subset of those experts. Total parameter count can be very large while the compute per token stays closer to a smaller dense model.

Instead of one giant generalist brain, you have a bench of specialists and only call the ones that fit this question.

The catch is memory. Idle experts still have to sit in memory (or be sharded across machines), so serving cost tracks active compute while hardware pressure tracks total size. When you read a model card, “total parameters” and “active parameters” answer different questions. See also Parameter / Weights, Inference, Open-Weight Model.

Agentic systems put a model in a loop with tools, memory or state, and a stop condition. The system plans, calls tools, reads results, and continues until the goal is met or something stops it. That is different from a single question and a single answer.

It is the difference between a librarian who answers one question and a research assistant who goes do the work and comes back with results.

Most of the value and most of the risk live here. Failures look like compounding errors, stale context, unverified actions, and runaway spend. Tight permissions, sandboxes, verifiable checks, step and spend caps, and treating tool output as untrusted input all matter. See also Tool Calling, Computer use, Model Context Protocol (MCP), RAG (Retrieval-Augmented Generation), Prompt Injection & Guardrails.

Tool calling is the model choosing when to invoke an external tool (search, code runner, API, database, and so on), emitting a structured call with arguments, and then using the returned result in the next step.

It is the assistant who knows when to look something up or run a calculation instead of guessing.

Without tool calling, you do not get modern agents, retrieval loops, or computer-use products. Reliability depends on clear tool descriptions, validated arguments, good error messages back to the model, and treating every tool result as untrusted. See also Agentic AI / Agentic Systems, RAG (Retrieval-Augmented Generation), Model Context Protocol (MCP), Inference.

Retrieval-augmented generation looks up relevant material from an external store, puts that material into the prompt, and only then asks the model to answer. The store is often a vector database of document chunks, but the idea is broader. Retrieve first, generate second.

Instead of relying only on what the model memorized in training, you let it consult a library you trust before it answers.

RAG can reduce hallucinations on questions that have an answer in your corpus. It does not delete hallucination. Chunking, embedding quality, and how you stuff the context window all affect the result. See also Context Window, Hallucination, Tool Calling, Agentic AI / Agentic Systems.

Reasoning models are trained so that longer, checkable intermediate work pays off, and they are often allowed to spend extra compute at inference time before the final answer. Both pieces matter. The training changes what the model does. The extra inference compute is how long it is allowed to work.

A student is drilled on showing correct working, then also given extra time and scratch paper on exam day.

That usually helps on math, hard coding, and multi-step logic. It also raises latency and token cost per query, and it can make immediate streaming harder. See also Inference, Token, Eval contamination / scaffold.

Model Context Protocol is an open standard for connecting models and agents to external tools and data sources in a consistent way. Clients talk to MCP servers that expose prompts, readable resources, and callable tools. Anthropic introduced MCP as that shared protocol; the Model Context Protocol docs home and specification describe the same design.

It is meant to work like a shared plug standard. One tool server can serve many clients, instead of every app inventing its own connector.

MCP does not replace tool calling. It standardizes how tools and context are published and consumed across hosts. See also Tool Calling, Agentic AI / Agentic Systems, RAG (Retrieval-Augmented Generation).

Prompt injection is when untrusted text (a user message, an email, a web page, a retrieved document) tries to override the instructions the system meant to follow. Guardrails are the checks and limits meant to catch or contain that.

Injection is a forged note to the guard that says ignore prior orders. Guardrails are the screening that tries to spot the forgery.

Direct injection attacks the chat. Indirect injection hides instructions in content the agent retrieves or reads. The OWASP Top 10 for Large Language Model Applications is a common independent framing for these risks in LLM apps. This is a first-order risk for agents with tools. Treat tool and retrieval output as untrusted. See also Agentic AI / Agentic Systems, Tool Calling, RAG (Retrieval-Augmented Generation), Hallucination.

The transformer is the architecture almost all frontier text models use. Attention is the mechanism inside it that lets each token look at other tokens in the context and decide what matters for the next prediction. The line of work starts with Vaswani et al., Attention Is All You Need.

Imagine a meeting where everyone can glance at everyone else’s notes, with learned importance weights, instead of passing a single hat down the table.

Attention is also why the key-value cache exists and why long context is expensive. FlashAttention is an IO-aware exact-attention kernel. It is a fast way to compute the same attention, not a different architecture (implementation). Paging tricks such as [1] manage the cache during serving. They are not a different attention formula. See also KV Cache (Key-Value Cache), Token, Inference, Prefill vs Decode.

Transformer inference has two phases. Prefill reads the whole prompt and builds the key-value cache. Decode then emits new tokens one at a time, mostly limited by how fast memory can be read.

You read the whole exam paper at once, then write the answer word by word.

Time to first token is mostly prefill. Tokens per second after that are mostly decode. Long prompts hurt the first wait. Long answers hurt ongoing cost. Flat “per token” pricing can hide that split. See also Inference, KV Cache (Key-Value Cache), Token.

Hallucination is when the model produces a confident falsehood. That can be a fake citation, a wrong fact, or an invented API. Next-token prediction rewards a fluent continuation. It does not reward a well-calibrated “I do not know.”

A student would rather invent a footnote than leave the line blank.

This is not a separate module you can switch off. OpenAI’s public explainer frames the failure as partly a test-taking incentive under next-token prediction. Retrieval and tools help when the model actually uses the evidence. They do not erase the failure mode. For risk work, treat an ungrounded assertion as a control miss, not a personality quirk. See also RAG (Retrieval-Augmented Generation), Prompt Injection & Guardrails, Tool Calling.

Computer use means the model operates a real computer interface (mouse, keyboard, browser, terminal, files). It is not limited to a chat box or a single clean API call.

You hand the intern the desk, not only a comment form.

The loop is usually see the screen (or accessibility tree), choose an action, observe what changed, repeat. Variance is higher than calling a well-typed API. Wrong clicks stick. Loops happen. You want a sandbox and a clear autonomy slider. See also Agentic AI / Agentic Systems, Tool Calling, Prompt Injection & Guardrails.

Two common ways a benchmark score can mislead. Contamination means some of the test items leaked into training data. Scaffold means extra tools, prompts, retries, or voting around the model that the headline number does not explain.

Seeing last year’s exam, or taking the test with a tutor and then quoting only your name on the scoreboard.

When you read a leaderboard, ask what was public, what tools were allowed, and whether the metric is a single try or many tries. Treat vendor slides as claims until you know the setup. Independent eval homes that document methods (without any scores asserted here) include Artificial Analysis methodology, METR, OSWorld (repo), and ARC Prize. See also Reasoning Models / Test-Time Compute, Open-Weight Model, Closed / Proprietary Model.