LLM Architecture

What it is

LLM (Large Language Model) architecture refers to the system design required to deploy transformer-based language models at scale for production inference. While training a large language model involves massive distributed GPU clusters, LLM serving architecture focuses specifically on the infrastructure needed to efficiently process inference requests — converting user prompts into generated tokens — at throughputs and latencies acceptable for real-world applications.

The core challenge is that LLM inference is fundamentally different from traditional neural network inference. LLMs generate tokens auto-regressively: each new token requires a forward pass through the entire model, and the computation for each forward pass depends on the key-value (KV) cache accumulated from all previous tokens in the context. This means inference is memory-bandwidth bound rather than compute bound, and the cost per request scales with the sequence length. The KV cache can consume tens of gigabytes of GPU memory for long contexts, severely limiting how many concurrent requests a single GPU instance can handle.

Key architectural techniques address these constraints. Continuous batching (also called dynamic batching or iteration-level scheduling) interleaves requests at the token generation level so that finished requests immediately free their KV cache slot for new requests, dramatically improving GPU utilization. Paged attention (introduced in vLLM) manages the KV cache in fixed-size pages analogous to virtual memory, eliminating internal and external fragmentation. Quantization reduces model weights from 16-bit to 8-bit or 4-bit, halving or quartering memory requirements at the cost of some accuracy. Tensor parallelism and pipeline parallelism distribute the model across multiple GPUs to serve models that exceed a single GPU's memory.

Why it exists

Large language models present serving challenges that existing ML serving infrastructure was not designed to handle. A 70-billion parameter model requires roughly 140GB of GPU memory in FP16, which exceeds the capacity of a single GPU (even an H100 with 80GB HBM3). Even models that fit on a single GPU face the challenge that each inference request holds a growing KV cache that occupies an unpredictable and increasing share of GPU memory throughout the generation process. Naive serving approaches that allocate the maximum possible KV cache per request upfront waste enormous amounts of memory on unused reserved space.

Throughput requirements for LLM serving differ qualitatively from classification models. A classification model produces one output per input; an LLM produces hundreds of tokens sequentially, with each step dependent on the previous. This temporal dependency means traditional request-response batching — wait for N requests, process them together, return all results — is inefficient because requests have different lengths and complete at different times, causing short requests to wait for long ones. Continuous batching solves this by treating generation at the token level, not the request level.

Dedicated LLM serving infrastructure has emerged (vLLM, TensorRT-LLM, SGLang, Ollama) to address these challenges with purpose-built memory management, scheduling algorithms, and model execution backends. Without such infrastructure, organizations deploying LLMs would either severely under-utilize expensive GPU hardware or fail to meet latency SLAs under any meaningful load.

When to use

• Deploying transformer language models with more than a few billion parameters where memory management and batching significantly affect throughput and cost.
• Applications with concurrent user requests where naive request-by-request inference would leave GPU utilization below 20%.
• Long-context applications (RAG, document analysis, multi-turn chat) where KV cache management is critical to serving capacity.
• Cost-sensitive deployments where efficient GPU utilization directly impacts the cost per 1,000 tokens served.
• Organizations deploying open-weight models (Llama, Mistral, Mixtral) that need to build their own serving infrastructure rather than using a managed API.

When NOT to use

• Applications using managed LLM APIs (OpenAI, Anthropic, Google) where the serving infrastructure is entirely abstracted — focus on prompt engineering and application logic instead.
• Very low-traffic applications (<10 requests/hour) where a simple inference loop is sufficient and batching provides no benefit.
• Small models (under 1B parameters) that fit trivially in GPU memory and don't require advanced KV cache management or multi-GPU parallelism.

Typical architecture

 ┌────────────────────────────────────────────────────────────┐
 │                    CLIENT / APPLICATION                     │
 │  Chat UI · API consumer · RAG pipeline · Agent framework   │
 └───────────────────────────┬────────────────────────────────┘
                             │  HTTP / WebSocket / SSE
                             ▼
 ┌────────────────────────────────────────────────────────────┐
 │                   LLM GATEWAY / PROXY                       │
 │  Rate limiting · Auth · Prompt logging · Cost tracking      │
 │  Request routing (model selection, tenant isolation)        │
 └────────────────┬────────────────────────────────┬──────────┘
                  │ Prefill req                     │ Generation req
                  ▼                                 ▼
 ┌─────────────────────────┐     ┌──────────────────────────┐
 │   PREFILL WORKER        │     │   DECODE WORKER(S)        │
 │   (compute-intensive)   │     │   (memory-BW intensive)   │
 │   Tokenize prompt       │     │   Autoregressive decode   │
 │   Compute KV cache      │     │   Continuous batching     │
 │   [GPU: high FLOPs]     │     │   Paged KV cache          │
 └──────────┬──────────────┘     │   [GPU: high HBM BW]     │
            │  KV cache transfer │                           │
            └────────────────────┘
                  │
                  ▼
 ┌────────────────────────────────────────────────────────────┐
 │              KV CACHE MANAGER (PagedAttention)              │
 │  ┌──────┐ ┌──────┐ ┌──────┐ ┌──────┐ ┌──────┐ ┌──────┐  │
 │  │ Page │ │ Page │ │ Page │ │ Page │ │ Page │ │ Page │  │
 │  │  0   │ │  1   │ │  2   │ │  3   │ │  4   │ │  5   │  │
 │  │[req A│ │[req A│ │[req B│ │[req B│ │ FREE │ │ FREE │  │
 │  └──────┘ └──────┘ └──────┘ └──────┘ └──────┘ └──────┘  │
 └────────────────────────────────────────────────────────────┘
                  │  token stream
                  ▼
 ┌────────────────────────────────────────────────────────────┐
 │                 RESPONSE STREAMING                          │
 │  Server-Sent Events (SSE) · Streamed token-by-token         │
 └────────────────────────────────────────────────────────────┘

 TENSOR PARALLEL (multi-GPU):
 ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐
 │  GPU 0   │  │  GPU 1   │  │  GPU 2   │  │  GPU 3   │
 │ Attn H0  │  │ Attn H1  │  │ Attn H2  │  │ Attn H3  │
 │ MLP shard│  │ MLP shard│  │ MLP shard│  │ MLP shard│
 └──────────┘  └──────────┘  └──────────┘  └──────────┘
     └─────────────────────────────────────┘
              NVLink / InfiniBand AllReduce

Pros and cons

Implementation notes

Choose the right parallelism strategy for your model size and hardware topology. Tensor parallelism (splitting individual layers across GPUs) is best for reducing single-request latency on large models but requires fast inter-GPU interconnects (NVLink or InfiniBand) — it performs poorly across nodes with slow networking. Pipeline parallelism (assigning different layers to different GPUs or nodes) achieves better throughput on distributed clusters but introduces pipeline bubbles and increases latency. For models that fit on a single node with NVLink, tensor parallelism is preferred; for models requiring multiple nodes, combine both strategies.

Calibrate your quantization strategy carefully. INT8 weight quantization with activation quantization (using GPTQ, AWQ, or SmoothQuant) typically achieves 95-99% of FP16 quality at half the memory footprint. INT4 quantization pushes further but introduces more significant quality degradation, especially for reasoning tasks. Profile your specific use case: for customer service chatbots, INT4 may be entirely acceptable; for code generation or mathematical reasoning, INT8 or FP16 may be necessary. Always benchmark quantized models on your specific task distribution before deployment.

Design your context window management strategy before building the serving system. Applications with long contexts (multi-document analysis, long conversations) consume KV cache memory proportionally to context length, limiting concurrent request capacity. Techniques for managing this include prompt caching (reusing the KV cache for repeated system prompts), sliding window attention, and context compression. Define the maximum context length your application requires and size your GPU memory accordingly, accounting for the worst-case KV cache consumption multiplied by your desired concurrent request count.

Common failure modes

• KV cache exhaustion: Under high load with long-context requests, the KV cache fills completely, causing new requests to be rejected or queued indefinitely. Mitigate with queue depth limits, context length limits per request, and preemption policies that swap low-priority requests to CPU memory.
• GPU memory OOM on model loading: Attempting to load a model larger than available GPU HBM fails catastrophically. Plan GPU memory allocation accounting for model weights, KV cache working set, and PyTorch/CUDA framework overhead, which typically adds 2-3GB per GPU.
• Head-of-line blocking by long requests: A single very long generation request occupies KV cache memory and decode compute for minutes, degrading throughput for all shorter concurrent requests. Implement maximum generation length limits and priority-based scheduling.
• Tensor parallel communication bottleneck: Placing tensor-parallel shards across nodes connected by Ethernet (rather than InfiniBand or NVLink) causes AllReduce communication to dominate latency. Always use high-bandwidth interconnects for tensor-parallel deployments.

Decision checklist

• What is the model size in parameters, and how much GPU memory is required in FP16, INT8, and INT4?
• What are the latency requirements (time-to-first-token and tokens-per-second) for your application?
• What is the expected maximum context length, and how does it constrain concurrent request capacity?
• Is tensor parallelism feasible — do your GPUs have NVLink or InfiniBand interconnects for efficient AllReduce?
• Have you evaluated quantization quality on your specific task and validated that accuracy degradation is acceptable?
• Is prefill-decode disaggregation warranted given your traffic pattern (high prefill cost vs. high generation length)?

Example use cases

• Enterprise internal chatbot: A 13B parameter Llama model served on two A100 80GB GPUs with tensor parallelism; INT8 quantized to fit within memory budget; vLLM handles continuous batching for 50 concurrent employee users; system prompts are prefix-cached to reduce per-request cost.
• Code completion service: A 34B coding model deployed across four GPUs with pipeline parallelism; requests are prioritized by interactive vs. batch use; response streaming via SSE reduces perceived latency for IDE integrations.
• Document analysis API: A 70B model deployed across eight H100s for long-context (128k token) document analysis; prefill is disaggregated to a high-FLOPs node while decode workers handle token generation; sliding window KV cache manages memory for very long documents.

Related patterns

• RAG Architecture
• Model Serving
• Vector Databases
• MLOps

Further reading

• vLLM Documentation – Efficient LLM Serving
• Efficient Memory Management for Large Language Model Serving (PagedAttention) – Kwon et al. 2023
• Mastering LLM Techniques: Inference Optimization – NVIDIA Developer Blog
• TensorRT-LLM Documentation – NVIDIA