AI Agent & LLM Glossary of Terms
A reference guide to core terminologies in large language models, agentic workflows, and local serving infrastructures.
Model Context Protocol (MCP)
An open-source standard created by Anthropic that allows LLMs and AI agents to securely connect to external tools, databases, local file systems, and API clients through unified interfaces.
PagedAttention
A KV-cache memory management algorithm (pioneered by vLLM) inspired by virtual memory paging in operating systems. It allocates KV-cache keys and values in non-contiguous virtual memory blocks, reducing VRAM fragmentation to near 0%.
Prompt Caching
A optimization technique where the LLM provider caches system instructions or codebase contexts in model server memory. Subsequent calls containing identical context prefixes read from cache, slashing API bills by up to 90% and reducing latency.
AST (Abstract Syntax Tree) Splitting
A code chunking method for RAG systems. Instead of slicing files by fixed character counts (which breaks functional code blocks), it parses source code into an Abstract Syntax Tree to split code cleanly along class, function, or method boundaries.
Mixture-of-Experts (MoE)
A neural network architecture (used in DeepSeek-V3 and Mixtral) that replaces dense layers with multiple specialized "expert" sub-networks. A routing network directs input tokens to only a few active experts per forward pass, drastically lowering computation costs.
Cross-Encoder Re-ranking
A secondary retrieval step in RAG pipelines. While bi-encoders evaluate documents independently for fast retrieval, a Cross-Encoder scores the query and document candidates together, capturing deep semantic relationships to ensure the most relevant context is sent to the LLM.
SWE-bench Verified
A benchmark dataset consisting of 500 hand-verified real-world software engineering issues from Python open-source repositories. Used to evaluate an agent's capability to search directories, write code patches, and execute unit tests inside sandboxed environments.