Cloudflare Unweight Achieves Lossless LLM Compression to Solve GPU Memory Bottlenecks

CloudflareCloudflare

· Updated

Cloudflare developed Unweight, a lossless compression system that reduces LLM footprints by up to 22% while maintaining bit-exact outputs. By decompressing weights directly in on-chip memory, the system bypasses the memory bandwidth bottleneck that typically leaves high-end GPUs idle during inference.

Cloudflare, a network and security company, introduced Unweight, a lossless compression system that shrinks LLM model weights by 15–22% without sacrificing quality. It targets the "memory wall" in inference (running a trained model to generate outputs) by using Huffman coding to compress redundant exponent bits in BF16 weights.
Compression ratio (MLP weights)
~30%
Total model footprint reduction
15-22%
VRAM savings (Llama 3.1 8B)
~3 GB
Throughput overhead
30-40%
Target hardware
NVIDIA H100 (Hopper)
Compression type
Lossless (bit-exact)

Modern GPUs like the NVIDIA H100 process data 600 times faster than memory can deliver it, creating a massive bandwidth bottleneck. Unlike standard quantization that trades accuracy for speed, Unweight preserves the original model's intelligence. This efficiency mirrors the pattern seen in specialized inference paths designed to maximize hardware throughput.

You can now access the technical paper and open-source GPU kernels to implement these reconstructive matrix multiplication techniques. While currently optimized for Llama 3.1 8B, the system follows the launch of persistent state management and extends Cloudflare's autonomous agent infrastructure to enable cheaper, faster model serving.

View the full update on blog.cloudflare.com
Cloudflare
Cloudflare
@Cloudflare
X

Running LLMs across Cloudflare’s network requires us to be smarter and more efficient about GPU memory bandwidth. That’s why we developed Unweight, a lossless inference-time compression system that achieves up to a 22% model footprint reduction. https://t.co/3wGOTTnlfQ

16retweets134likes
View on X

Still wondering? A few quick answers below.

Unweight is a lossless compression system developed by Cloudflare to reduce the memory footprint of large language models during inference. It achieves a 15% to 22% reduction in model size without any loss in accuracy. By shrinking weights, it allows more models to fit on a single GPU while improving memory bandwidth efficiency.

Unweight uses Huffman coding—a technique that assigns shorter codes to frequent values—to compress the exponent bits of model weights. Unlike lossy quantization that reduces precision, Unweight leaves the sign and mantissa bits untouched. This ensures the final output is bit-exact to the original model while requiring fewer bytes to cross the GPU memory bus.

High-end GPUs like the NVIDIA H100 often sit idle waiting for data from main memory. Unweight solves this by decompressing weights inside the GPU's fast on-chip shared memory. This reconstructive approach—where weights are rebuilt during the math operation—feeds the tensor cores directly, reducing the amount of data that must travel across the slower memory bus.

Yes, Cloudflare has open-sourced the GPU kernels for Unweight and published a technical research paper detailing the system's architecture. The project is intended to encourage innovation in GPU efficiency and compression. Developers can examine the reconstructive matrix multiplication kernels and the autotuning process used to optimize performance across different batch sizes.

While Unweight reduces memory usage, the on-chip decompression adds computational work. Current results on Llama 3.1 8B show a throughput overhead of roughly 30% to 40% depending on the batch size. Cloudflare is actively optimizing these kernels to reduce this gap, specifically by targeting the down projection weights and improving small-batch execution efficiency.

Every HeadsUpAI update is written based on its original source and reviewed before it's published. Read our editorial standards →

Cloudflare

See all AI news & updates from Cloudflare →

Share this update