Cohere Integrates W4A8 Inference into vLLM for Faster Hopper Performance

CohereCohere

· Updated

Cohere released production-ready W4A8 quantization kernels for dense and Mixture of Experts models, now integrated into the vLLM inference framework. By combining 4-bit weights with 8-bit activations, the update achieves up to 58 percent faster prefill and 45 percent faster decoding on NVIDIA Hopper GPUs.

Cohere, an AI company building enterprise models for search and business applications, integrated W4A8 inference (a mixed-precision scheme using 4-bit weights and 8-bit activations) into the vLLM framework. This update targets NVIDIA Hopper architecture, optimizing for FP8 Tensor Cores to accelerate both prefill and decoding phases.
TTFT speedup
Up to 58%
TPOT speedup
45%
Weight precision
4-bit INT4
Activation precision
8-bit FP8
Calibration context
Up to 64K tokens
Target architecture
NVIDIA Hopper

Standard 4-bit quantization often relies on 16-bit activations, saving memory but missing the fastest compute engines on modern GPUs. This release mirrors the industry shift toward specialized quantization by bridging that gap. A lookup-table approach dequantizes weights without the scalar math overhead that typically bottlenecks FP8 kernels.

Deploy Command A or Mixture of Experts models with lower latency for long-context agentic workflows. This work extends recent research on MoE efficiency and includes a new token masking feature in the llm-compressor library. These kernels are available now via the official vLLM repository for production use.

View the full update on cohere.com
Cohere
Cohere
@cohere
X

Excited to share our work on production-ready W4A8 inference, now integrated in vLLM! By combining 4-bit weights (low memory) with 8-bit activations (high compute), we hit the sweet spot for both decoding and prefill — up to 58% faster TTFT and 45% faster TPOT vs W4A16 on Hopper. https://t.co/M37wT5KS8Z

35retweets240likes
View on X

Still wondering? A few quick answers below.

W4A8 is a mixed-precision quantization scheme that uses 4-bit weights to reduce memory footprint and 8-bit activations to maximize compute throughput. This approach targets the NVIDIA Hopper architecture, allowing models to run efficiently in both memory-bound decoding and compute-bound prefill regimes by utilizing high-performance FP8 Tensor Cores for matrix multiplication.

Cohere's W4A8 kernels deliver significant speed improvements on NVIDIA Hopper GPUs compared to the previous W4A16 standard. Users can expect up to a 58 percent reduction in Time to First Token and a 45 percent improvement in Time Per Output Token. These gains scale consistently across different batch sizes for both dense and Mixture of Experts models.

Token masking is a technique added to the llm-compressor library that allows developers to exclude repetitive or non-informative tokens from calibration statistics. By masking system prompts, templates, and tool descriptions, the calibration process focuses on useful optimization space. This is essential for preserving performance in long-context agentic workflows where repetitive prompts can bias quantization decisions.

Yes, Cohere has integrated its production-ready W4A8 dense and grouped GEMM kernels directly into the vLLM inference framework. This integration supports both dense models and Mixture of Experts architectures. Developers can access these performance optimizations through the official vLLM repository to deploy models like Command A with improved efficiency on compatible NVIDIA hardware.

Cohere uses two primary techniques to recover model quality: Quantization-Aware Distillation and a specialized lookup-table approach. Distillation trains a quantized student model to match a higher-precision teacher's output. Additionally, they apply per-channel scales and manual scaling to ensure weights stay within the FP8 range without clipping, restoring accuracy to within 99.5 percent of the original baseline.

Every HeadsUpAI update is written based on its original source and reviewed before it's published. Read our editorial standards →

Cohere

See all AI news & updates from Cohere →

Share this update