概念导读
LLM inference optimization covers methods that reduce latency, memory, or cost during model serving, including speculative decoding, KV-cache optimization, routing, and compression.
- Related
LLM Inference Optimization
LLM inference optimization covers methods that reduce latency, memory, or cost during model serving, including speculative decoding, KV-cache optimization, routing, and compression.
Related
- speculative-decoding — Draft-and-verify acceleration.
- audio-kv-cache — KV-cache pressure in audio-language models.
- dflash — Diffusion-based decoding acceleration.