Prompt Engineering & Mechanistic Interpretability

Understanding prompts as mathematical operators: QK/OV induction circuits, Function Vector (FV) heads, computational complexity classes (TC⁰ to P-Time), and continuous thought spaces.

💡 EXECUTIVE PROMPTING SUMMARY

Prompt engineering is the soft-programming of transformer attention circuits. Simple zero-shot prompts are limited to constant-depth logic ($TC^0$). Forcing the model to generate Chain-of-Thought (CoT) step-by-step reasoning expands its computational ceiling to Polynomial-Time ($P$-Time), allowing it to solve complex legal, financial, and mathematical logic.

🏷️ Plain-English Analogy (Induction Heads): Induction heads are the AI's internal copy-paste shortcut. When the AI sees a pattern earlier in your prompt, induction heads instantly complete the sequence whenever the prefix appears again.

MECHANISTIC CIRCUIT SIMULATOR

Prompting Technique & Activation Router Simulator

Select a prompting paradigm and adjust token length to simulate how attention circuits, KV cache VRAM footprint, and complexity classes dynamically morph inside the transformer decoder.

Prompting Paradigm Strategy:

Prompt & Reasoning Context Length: 4,096 Tokens

COMPUTATIONAL COMPLEXITY CLASS TC⁰ (Constant Circuit Depth) Parametric & Syntactic Heads

KV CACHE VRAM FOOTPRINT 256 MB Dynamic Key-Value State Memory

CONTEXT WINDOW UTILIZATION 3.2% of 128K Attention FLOPs: 1.07 TFLOPs

REASONING ACCURACY & ACCELERATION 62.4% Baseline Out-of-distribution reasoning accuracy

ATTENTION CIRCUIT ARCHITECTURE

The Mechanistic Engine of Transformer Prompting

How prompts dynamically rewire internal attention circuits without mutating static model weights.

STAGE 1 & 2 MECHANICS

QK & OV Induction Head Circuits

Induction heads consist of two coupled attention heads: QK (Query-Key) circuits search the context window for structural prefix patterns, while OV (Output-Value) circuits copy the subsequent token into the residual stream output projection.

QK Circuit: Locates previous occurrences of pattern [A].
OV Circuit: Copies subsequent token [B] directly to prediction.

LATENT COMPRESSION

Function Vector (FV) Heads

In few-shot prompting, Function Vector heads compress multi-example input/output pairs into compact latent vectors in key-value space. This steers model behavior toward complex tasks without explicit token copying.

Hapax Paradigm: Enables abstract in-context learning without literal token matching.
Task Steering: Dynamically shifts output distribution along task-vector axes.

RAG TRIPLE-ROUTING

RAG Multi-Circuit Attention Routing

Retrieval-Augmented Generation prompts force a tri-fold segregation of attention heads: In-Context Heads parse task rules, Parametric Heads query internal weights, and Retrieval Heads copy external document data.

In-Context Heads: Interpret instruction rules and JSON schemas.
Retrieval Heads: High-precision copying from fetched RAG documents.

COMPUTATIONAL TAXONOMY

Taxonomy of Prompting Paradigms & Complexity Classes

A rigorous classification of prompting techniques mapped to theoretical computer science complexity classes.

Prompting Paradigm	Complexity Class	Active Attention Sub-Circuits	Working Memory Mechanism	Core Computational Bottleneck
Direct Generation (Zero-Shot)	TC⁰	Parametric & Syntactic Heads	Internal Static Weights	Fixed parallel depth; cannot solve serial dependencies
Few-Shot (In-Context Learning)	L (Log-Space)	QK/OV Induction & Function Vector (FV) Heads	Latent task-vectors in KV states	Limited by context window & induction capacity
Linear Chain-of-Thought (CoT)	Context-Sensitive	Long-context concentration & Retrieval Heads	Autoregressive tokens written to context	High latency due to serial token decoding
Polynomial Chain-of-Thought (CoT)	P (Polynomial-Time)	Multi-layer routing & Semantic Induction Heads	Stepwise generated tokens in context window	High latency due to serial token decoding
Polynomial Padding Tokens	NC Parallel	Wide parallel spans over blank vector states	Hidden states of non-semantic padding tokens	Bound by parallel limits; cannot execute serial logic
Padding with Looping	Recurrent Layer	Recurrent looping attention over blank states	Hidden states of non-semantic padding tokens	Requires structural layer looping support
Continuous Stepwise (MarCos)	P (Polynomial-Time)	Continuous layer-to-layer thought mapping	Latent vector space representations	Requires non-standard continuous architectures