MODULE 01 · COMPREHENSIVE KNOWLEDGE DIRECTORY
AI Concepts & Sampling Parameters Glossary
89 structured terms, sampling parameters, architectural concepts, and hardware formulas extracted directly from core research spreadsheets.
Sampling Hyperparameter Cheat Sheet
Recommended parameter configurations for common LLM deployment tasks.
Code & Mathematics
Deterministic output minimizing syntax errors and hallucinations.
Creative & Storytelling
High vocabulary diversity while filtering long-tail nonsensical tokens.
Agent Tool Calling
Strict JSON schema compliance for deterministic function arguments.
LLM VRAM Footprint & KV Cache Calculator
Estimate GPU VRAM memory requirements for inference based on model parameter count, precision, context length, and batch size.
Token
The smallest unit of text that an AI model processes. Roughly ¾ of a word on average in English. Analogy: Like individual Lego bricks that the model uses to build meaning, rather than whole sentences.
Context windows are measured in tokens. Longer context = more expensive inference and higher memory use (KV cache). Important for cost and capability planning in agents and RAG systems.
Parameter / Weights
The billions or trillions of adjustable numbers inside a model that encode its 'knowledge' after training. Analogy: Think of parameters as the dials and knobs on an enormous mixing board that the model learned to set during training.
More parameters generally = more capability (with diminishing returns). Quantization reduces parameter precision to shrink model size and speed up inference with acceptable quality loss.
Context Window
The maximum amount of text (measured in tokens) the model can 'see' at once when generating a response. Analogy: Like short-term memory or the size of a whiteboard the model can look at while thinking.
Critical for RAG, agent memory, and long-document tasks. Trade-off: longer context = higher cost and slower responses. KV Cache optimization is a major area of inference engineering.
KV Cache (Key-Value Cache)
A memory optimization during inference that stores previous computations so the model doesn't recalculate them for every new token. Analogy: Like keeping a running notebook of what you've already figured out instead of re-deriving everything from scratch each time.
Enables much faster token generation after the first token. Memory-hungry at long context lengths. Quantization and paging techniques (e.g., vLLM PagedAttention) help manage this.
Inference
The process of running a trained model to generate outputs (answers, code, images, etc.). This is what happens when you chat with an AI. Analogy: Training is like studying for an exam; inference is taking the exam and answering questions.
The dominant cost for most AI products after initial training. Quantization, distillation, and efficient serving frameworks (vLLM, TensorRT-LLM) dramatically reduce inference costs.
Training (Frontier Models)
The massive compute-intensive process of adjusting a model's parameters on enormous datasets so it learns patterns, reasoning, and capabilities. Analogy: Sending a student through years of intense, specialized education with constant testing and adjustment.
Extremely expensive (hundreds of millions to billions USD). Only a handful of organizations can train frontier models. Open-weight releases (Llama, Qwen, Gemma) let others benefit without the training cost.
Fine-tuning
Taking a pre-trained model and further training it on a smaller, specific dataset to improve performance on particular tasks or domains. Analogy: A generalist doctor completing a residency in cardiology to become a heart specialist.
The main way organizations create specialized models without training from scratch. Critical for domain adaptation, style control, and safety/alignment. LoRA makes it accessible on modest hardware.
Quantization
Reducing the numerical precision of a model's parameters (e.g., from 16-bit to 4-bit) to make the model smaller, faster, and cheaper to run with minimal quality loss. Analogy: Compressing a high-resolution photo into a smaller file size that still looks good enough for most uses.
Enables running 70B+ models on consumer GPUs or even laptops. 4-bit quantization can reduce model size by ~75% with often acceptable quality degradation. Essential for local and edge deployment.
GPU (Graphics Processing Unit)
A specialized processor originally designed for graphics, now the dominant accelerator for AI training and inference due to its massive parallelism. Analogy: Instead of one very smart person doing work sequentially, imagine thousands of moderately smart people working on different parts of the problem at the same time.
The default choice for serious AI work. CUDA ecosystem lock-in is real but powerful. Alternatives (AMD ROCm, Intel Gaudi, custom ASICs) exist but have smaller software ecosystems.
TPU (Tensor Processing Unit)
Google's custom AI accelerator chip designed specifically for neural network workloads, especially training and inference of large models. Analogy: A purpose-built race car versus a high-performance general sports car (GPU). Optimized for one job extremely well.
Google achieves excellent price/performance and efficiency for their own frontier models. Not accessible for most organizations to train on directly. Influences what capabilities Gemini excels at (long context, multimodality).
NPU / Neural Engine
Specialized hardware inside modern CPUs and SoCs (especially mobile and Apple Silicon) optimized for running AI models efficiently at low power. Analogy: A dedicated fuel-efficient engine for AI tasks versus the main powerful engine (GPU) that uses more fuel.
Enables powerful local AI on laptops, phones, and edge devices without sending data to the cloud. Apple Silicon + MLX is currently one of the best experiences for local inference on consumer hardware.
Software Stack
The complete layered collection of software that sits between raw hardware and the applications you actually use, making the hardware usable for AI work. Analogy: The foundation, wiring, plumbing, rooms, and furniture of a house — without the full stack, the powerful hardware underneath is mostly useless.
The software stack often determines what hardware you can effectively use and how easily. CUDA lock-in is strong. Moving between stacks (NVIDIA ↔ TPU ↔ Apple) has real friction. Choice of stack affects cost, performance, and developer experience.
CUDA / cuDNN
NVIDIA's proprietary software platform that lets developers program their GPUs for general-purpose computing, especially AI. cuDNN is the deep learning library on top of CUDA. Analogy: The 'language' and 'standard library' that most AI frameworks speak when they want to run fast on NVIDIA hardware.
If you want maximum performance and compatibility with the broadest set of AI tools and models, NVIDIA + CUDA is still the safest default in 2026. Many open-weight models and inference engines are optimized first for CUDA.
JAX + Pathways
Google's modern numerical computing library (JAX) and orchestration system (Pathways) used to train and serve large models efficiently across thousands of TPUs or GPUs. Analogy: A highly optimized assembly line and logistics system designed specifically for Google's custom TPU hardware.
Enables Google to achieve excellent utilization and scaling on their custom silicon. For most external users, PyTorch + CUDA or open inference engines are more practical. JAX has growing academic and research adoption.
MLX
Apple's open-source machine learning framework optimized specifically for Apple Silicon (M-series chips) and unified memory architecture. Analogy: A tailor-made suit that fits Apple hardware perfectly instead of an off-the-rack suit that works everywhere but isn't optimized.
Currently one of the best experiences for running large models locally on a laptop. Excellent performance per watt and developer ergonomics on M-series Macs. Ideal for privacy-sensitive or offline workflows.
Open-Weight Model
A model where the trained parameters (weights) are publicly released and downloadable, so anyone can run, inspect, fine-tune, or modify it (subject to license terms). Analogy: The restaurant gives you the full recipe so you can cook it yourself, tweak it, or open your own restaurant using it.
Gives sovereignty, cost control at scale, customization, and auditability. Trade-off: you take on maintenance, safety, and infrastructure responsibility. Quality has improved dramatically; many now rival closed models from 12-18 months prior.
Closed / Proprietary Model
A model where the weights and training details are kept private by the company. You can only access it through their API or chat interface. Analogy: Eating at a high-end restaurant where the chef keeps the exact recipe secret. You enjoy the meal but can't recreate or modify it yourself.
Usually leads on raw capability, especially complex reasoning and agentic workflows. Easier to use (no infrastructure to manage). Trade-off: vendor lock-in, usage-based pricing, potential sudden policy or pricing changes, less transparency.
Mixture-of-Experts (MoE)
An architecture where only a subset of the model's 'experts' (specialized sub-networks) are activated for any given input, allowing much larger total parameter counts without proportional compute cost. Analogy: Instead of one giant generalist brain, you have many specialist consultants and only call in the relevant ones for each question.
Enables higher capability at lower inference cost compared to dense models of similar active parameter count. More complex to serve efficiently. Important architectural trend in 2025-2026 frontier models.
Post-training / Alignment
The phase after pre-training where the model is further shaped to be helpful, honest, harmless, and aligned with human preferences and safety goals. Analogy: After a student learns a huge amount of raw knowledge, this is the finishing school that teaches them how to behave, communicate clearly, and avoid dangerous mistakes.
This is where much of the 'personality' and safety behavior comes from. Open models can be further aligned or 'unaligned' by users. Closed models' alignment is controlled by the provider and can change over time.
Agentic AI / Agentic Systems
AI systems that can plan, use tools, maintain memory across steps, and autonomously work toward complex, multi-step goals rather than just answering single questions. Analogy: Moving from a very knowledgeable librarian who answers questions to a proactive research assistant who can go do tasks, use tools, and come back with results.
This is where a lot of the practical value (and risk) of AI is moving. Building reliable agents is still hard — context management, error recovery, and cost control are active areas of engineering. Open-weight models + good orchestration can be very cost-effective for high-volume agent workloads.
Tool Calling
The ability of a model to decide when and how to use external tools (search, code interpreter, APIs, databases, etc.) and incorporate the results into its reasoning. Analogy: A smart assistant who knows when to look something up, run a calculation, or call a colleague instead of guessing or saying 'I don't know'.
Essential for building useful agents and reducing hallucinations on factual or computational tasks. Quality of tool calling is a major differentiator between models in 2026. Open models are closing the gap quickly.
RAG (Retrieval-Augmented Generation)
A technique where the model retrieves relevant information from an external knowledge base (usually a vector database) and includes it in the prompt before generating an answer. Analogy: Instead of relying only on what the model memorized during training, you let it look up fresh, specific information from a trusted library before answering.
One of the most practical ways to give models up-to-date or proprietary knowledge without fine-tuning. Reduces hallucinations on factual questions. Cost and latency scale with retrieval quality and context stuffing. Hybrid search + reranking often gives big quality wins.
Embeddings
Numerical vector representations of text (or images, audio, etc.) that capture semantic meaning in a way that similar items have similar vectors. Analogy: Turning words and sentences into GPS coordinates in a high-dimensional space so that 'king' and 'queen' are close to each other, and 'king - man + woman ≈ queen'.
The foundation of modern semantic search and RAG. Better embeddings = better retrieval = better RAG answers. Domain-specific fine-tuned embedders can significantly outperform general ones on specialized corpora.
TSMC (Taiwan Semiconductor Manufacturing Company)
The world's most advanced contract chip manufacturer (foundry). They actually build the physical silicon chips designed by NVIDIA, Google, Apple, AMD, and many others. Analogy: The ultra-specialized factory that builds the engines for almost every high-performance car brand, even though each brand designs its own engine.
Almost every frontier AI training run in 2026 ultimately depends on TSMC's manufacturing capacity. Supply constraints, geopolitics, and capacity allocation are real strategic factors for AI labs and cloud providers.
Colossus (xAI)
xAI's massive GPU cluster in Memphis, Tennessee, one of the largest training clusters in the world, used to train Grok models. Analogy: A purpose-built, rapidly constructed 'AI factory' using thousands of NVIDIA GPUs working in parallel.
Demonstrates that massive scale is achievable quickly with NVIDIA hardware and aggressive execution. Relevant for understanding Grok's training data freshness and iteration speed advantages.
Agentic AI
Autonomous decision-making and tool use.
Chain of Thought
Forcing step-by-step reasoning.
Few-Shot Prompting
Providing examples in the prompt to guide the model.
Groking
Sudden jump in capability after extended training.
Metaprompt
Hidden system instructions governing behavior.
Multimodal
Processing multiple data types.
ReAct
Pattern where models trace thoughts before acting.
Reasoning
Ability to logically deduce conclusions from premises.
Scaling Laws
Observation that more compute/data yields better performance.
Zero-Shot Prompting
Asking the model to do a task without examples.
Iterative Refinement
A technique where an LLM generates an output
Circle of Experts
A reasoning strategy where multiple AI agents
Attention Mechanism
Mechanism allowing model to focus on specific words in a sequence.
Chunking
Breaking large texts into smaller sections.
Context Caching
Storing processed tokens to reduce cost/latency for repeated inputs.
Diffusion Model
Generative model adding/removing noise to create data.
GAN
Two networks (generator vs discriminator) competing.
Grounding
Linking responses to verifiable sources.
Latency
Time delay between input and response.
Logits
Raw unnormalized prediction scores before probability conversion.
MoE (Mixture of Experts)
Model using sub-models (experts) for specific tasks to save compute.
Orchestration
Routing requests to correct tools.
RAG
Retrieving facts from external DB before answering.
System Prompt
Initial instructions defining the AI's role.
Temperature
Setting (0-1) for output randomness.
Tokenizer
Tool that breaks text into tokens (integers) before processing.
Top-P (Nucleus Sampling)
Sampling from the top P% cumulative probability tokens.
TPU / GPU
Specialized hardware chips required to train and run LLMs.
Transformer
Neural network architecture of LLMs.
Vector Database
Storing data as mathematical concepts.
Weights (Parameters)
The learned values inside the model that determine its output.
Alignment
Ensuring AI goals match human values.
Bias
Systematic error favoring certain groups/outcomes.
Data Poisoning
Corrupting training data to compromise the model.
Edge Cases
Unusual scenarios that break normal assumptions.
Hallucination
Factually incorrect info from probabilistic guessing.
Jailbreak
Bypassing safety filters to generate restricted content.
Model Collapse
Degradation of models trained on AI-generated data.
Prompt Injection
Attack tricking AI to ignore rules.
Red Teaming
Simulated attacks to find vulnerabilities.
Shadow AI
Unsanctioned AI tools bypassing IT.
Spectral Bias
Tendency of models to learn frequency-based shortcuts over meaning.
Zero Data Retention
Inputs deleted immediately after processing.
Persona Collapse
A breakdown in an AI system's ability to maintain a coherent identity
Backpropagation
Algorithm for adjusting weights based on error rates.
Epoch
One complete pass of the training dataset through the model.
Loss Function
Metric measuring how far the model's prediction is from actual.
MMLU
Benchmark for general intelligence (Massive Multitask Language Understanding).
Noise Spike
A sudden increase in irrelevant changes detected during file comparison.
Overfitting
Model learning noise/details too well, failing on new data.
RLHF (Reinforcement Learning from Human Feedback)
Training via human feedback ratings.
Semantic
Relating to meaning rather than surface text.
Synthetic Data
Data generated by AI to train other AI.
Underfitting
Model being too simple to capture the underlying structure.
Graceful Degradation
Partial functionality when full fails.
Multi-turn
Conversation capability maintaining context across messages.
Slop
Low-quality or generic AI-generated content.
Status Telemetry
Visible indicators of agent activity (WORKING, THINKING).
Tokenization
The process of breaking text into smaller units (tokens). A token can be a word, part of a word, or a character.
The Logits (Raw Scores)
The raw, unnormalized scores the model outputs for every possible token in its vocabulary (often 50k+ tokens).
Softmax Function
A mathematical function that converts the raw Logits into percentages (probabilities) that sum to 100%.
Decoding Strategy (Greedy)
A method where the AI always picks the highest probability token.
Decoding Strategy (Sampling)
A method where the AI rolls a die to pick the next word based on the percentages, allowing for variety.
RLHF (Reinforcement Learning)
Training that rewards the model for helpful/safe answers and punishes harmful/nonsense ones.