Hugging Face Launches Storage Buckets for High-Throughput ML Workflows

Hugging Face

Mar 15, 2026 · Updated Apr 25, 2026

Hugging Face shipped Storage Buckets — mutable, S3-like object storage on the Hub for checkpoints, agent traces, and processed datasets. Xet chunk-based deduplication means successive checkpoints skip bytes already stored, cutting bandwidth and transfer time.

Hugging Face launched Storage Buckets, a mutable non-versioned object storage layer on the Hub for ML artifacts that change constantly — training checkpoints, optimizer states, processed data shards, and agent traces. Unlike Git-backed repos, Buckets support fast writes, overwrites, and directory sync. Addressable via hf://buckets/username/bucket-name, they're manageable via the hf CLI or huggingface_hub library (since v1.5.0).

Buckets run on Xet, Hugging Face's chunk-based storage backend — successive checkpoints with frozen model sections skip bytes already stored, cutting bandwidth and storage footprint. Enterprise billing is based on deduplicated storage, so shared chunks reduce costs. Hugging Face partners with AWS and GCP for pre-warming, bringing data close to compute before training runs.

Buckets are included in existing Hub storage plans — free accounts get storage to start; PRO and Enterprise offer higher limits. Sync a checkpoint directory with hf buckets sync or access bucket contents from any fsspec-compatible library via hf:// paths.

View the full update on huggingface.co

Hugging Face

@huggingfaceMar 10

🪣 We just shipped Storage Buckets: S3-like mutable storage, cheaper & faster Git falls short for everything on high-throughput side of AI (checkpoints, processed data, agent traces, logs etc) Buckets fixes that: fast writes, overwrites, directory sync 💨 All powered by Xet dedup so successive checkpoints skip the bytes that already exist ➡️

View on X

Every HeadsUpAI update is written based on its original source and reviewed before it's published. Read our editorial standards →

Keep reading

Hugging Face Hits One Million Open Datasets to Fuel Open Source AI

Hugging Face reached one million open datasets on its Hub. The dataset library spans multiple modalities including text, image, audio, video, geospatial, and tabular data, available in formats like Parquet, JSON, and CSV. The platform frames the milestone as community-driven, with the tagline that open models need open data.

Vercel Sandbox Adds Persistent Drives for AI Agents to Retain Memory

VercelJun 5

Vercel Sandbox Adds Persistent Drives for AI Agents to Retain Memory

Vercel Sandbox now supports persistent drives in private beta. This separates storage from compute, allowing AI agents to retain their workspaces and "memories" across disposable sandbox instances. This is crucial for building reliable, multi-step agentic systems that need to maintain state.

Anthropic enables distributed agent memory with Claude Agent SDK session adapters

ClaudeApr 24

Anthropic enables distributed agent memory with Claude Agent SDK session adapters

Anthropic launched session store adapters for the Claude Agent SDK, allowing developers to mirror agent transcripts to external databases like S3, Redis, and Postgres. This update enables persistent agent memory across distributed hosts, ensuring that sessions started on one server can be resumed on another.

Fireworks AI Uses Delta Compression to Reduce Frontier RL Training Costs

Fireworks AIMay 5

Fireworks AI Uses Delta Compression to Reduce Frontier RL Training Costs

Fireworks AI introduced a distributed reinforcement learning architecture that uses delta-compressed weight updates to sync training and inference clusters across different regions. By shipping only the 2% of weights that change between checkpoints, teams can train frontier-scale models using fragmented GPU capacity instead of expensive mega-clusters.