Inference Engineering & Optimization

Low-precision NVFP4/FP8 quantization scaling, safetensors zero-copy mmap loading, PagedAttention/RadixAttention serving schedulers, and disaggregated prefill/decode topologies.

💡 EXECUTIVE INFERENCE SUMMARY

AI serving cost is dictated by two distinct phases: Prefill (Think Time), which reads the prompt and is compute-bound, versus Decode (Typing Speed), which generates output tokens and is memory-bandwidth bound. Modern zero-copy formats (`safetensors mmap`) cut model load times from 10 minutes to 45 seconds while eliminating security risks.

🏷️ Plain-English Analogy (Prefill vs. Decode): Prefill is like a lawyer reading a 50-page contract all at once (compute heavy). Decode is the lawyer dictating their response line by line (memory fetch heavy).

PHYSICAL INFERENCE SIMULATOR

Prefill vs. Decode Latency & Throughput Calculator

Simulate compute-bound prefill latency (Time-To-First-Token) versus memory-bandwidth-bound decode throughput.

Input Prompt Context Tokens: 4,096 tokens

Generated Output Tokens: 512 tokens

Concurrent Batch Size: 16 Requests

PREFILL LATENCY (TTFT) 142 ms Compute-bound matrix multiplication ($O(N^2)$)

INTER-TOKEN LATENCY (ITL) 14.5 ms / token Memory-bandwidth-bound HBM fetch

TOTAL DECODE THROUGHPUT 1,103 tok/s Aggregated system throughput across batch

SERVING ENGINE ARCHITECTURE

Modern Serving Engine Feature Comparison Matrix

Evaluating PagedAttention virtual memory, Trie-based RadixAttention prefix caching, and Ahead-of-Time CUDA compilation.

vLLM

PagedAttention & Scale-to-Zero

Treats KV cache memory like OS virtual memory paging, reducing VRAM fragmentation to < 4%. Supports continuous batching and rapid cold starts (<90s).

Optimal For: Dynamic multi-tenant cloud APIs & scale-to-zero serverless deployments.
KV Cache Strategy: Virtual Block Page Tables allocated dynamically per request.

SGLang

RadixAttention & Agentic Reuse

Uses Trie-based RadixTree structures for automatic prompt prefix caching. Reuses KV cache pages across multi-turn branches, dropping TTFT latency by 70% – 90%.

Optimal For: Multi-agent tree search, CoT reasoning, and multi-turn chat applications.
KV Cache Strategy: Trie-based node caching with Copy-on-Write state forking.

TensorRT-LLM

AOT CUDA Graph Compilation

NVIDIA ahead-of-time (AOT) graph compiler that fuses kernels for maximum raw hardware throughput and lowest tail-latency, at the cost of a 28-minute cold-start compilation.

Optimal For: High-volume, static production workloads with strict SLA tail-latency bounds.
KV Cache Strategy: Static HBM memory allocation with AOT fused kernel graphs.

INGESTION SECURITY & BENCHMARKS

Safetensors Direct mmap vs. Legacy PyTorch (.pt)

Eliminating arbitrary code execution vulnerabilities while cutting 175B model loading times from 10 minutes to 45 seconds.

LEGACY FORMAT

PyTorch Pickle (`.pt` / `.bin`)

10 Minutes Load time for 175B model across 8 GPUs

Uses Python `pickle` module, introducing critical Remote Code Execution (RCE) vulnerabilities via `__reduce__` shell injection exploits.

ZERO-COPY SOTA

Safetensors (`mmap`)

45 Seconds 13.3x Faster zero-copy kernel loading

Direct OS kernel memory mapping (`mmap`), header restricted to 100MB, non-overlapping address validation guaranteeing zero RCE attack vectors.

QUANTIZATION BENCHMARKS

Precision Format & Throughput Scaling Matrix

Comparative benchmarks measuring VRAM footprint, prefill latency, and decode throughput across precision formats.

Format	Bit Precision	VRAM Profile	Prefill Latency	Decode Throughput	Reasoning Accuracy
FP16 / BF16	16-bit	Baseline (100%)	Baseline	Baseline (~20 tok/s)	100% SOTA
GGUF (Q6_K)	6.5-bit	Moderate (-58%)	3.73s	~45 tok/s	98.5% Retention
AWQ-Int4	4-bit	Low (-75%)	1.68s	~75 tok/s	96.0% Retention
Blackwell NVFP4	4-bit Micro-scaled	Ultra-Low (-75%)	0.45s	102,725 tok/s	95.5% Retention