Practical AI guides
Inference & deployment
Understand serving architectures, hardware requirements, and the conditions behind performance claims.
Where to start
Read throughput and latency claims alongside model size, hardware, concurrency, and workload. An optimization that helps a busy server may introduce unnecessary complexity for a small deployment.
- Does the documented hardware match your deployment?
- Is the bottleneck compute, memory, or scheduling?
- Which operating costs sit outside the headline benchmark?
Practical guides
LLM GPU Memory Requirements: Weights, KV Cache and OOM
LLM GPU memory requirements: Budget for weights, KV cache, concurrency and runtime headroom. Choose context limits or quantization for your workload.
Explore the guideWhy AI Responses Are Slow: Trace Retrieval, Tools and Models
Why AI responses are slow: Separate first-answer delay from completion and follow the critical path. Choose a measurement before changing infrastructure.
Explore the guideExplore the reports
Semantic Router for LLM Apps: Embeddings and Intent Routing
3.9Kstars
Rapid-MLX for Coding Agents: Apple Silicon Inference
4Kstarskvcached GPU Memory Sharing: How Elastic KV Caching Works
1.5Kstars
Slotstream Explained: Running MoE Models on a Mac
412stars
GPUStack for Multi-GPU Inference: Server and Worker Roles
5.8KstarsMilvus Explained: Vector Search and Storage Separation
46.1KstarsInferenceX Explained: Comparing LLM Inference Benchmarks
1.7Kstars
KServe Explained: Choosing a Kubernetes Serving Mode
5.9Kstars
OpenVINO Deployment Overview: Conversion and Device Support
10.8KstarsBifrost AI Gateway Explained: Latency and Fallbacks
7.7Kstars
Spice Explained: Federated Queries vs Local Acceleration
3.1Kstars