Google Research, in partnership with Google DeepMind, today released VaultGemma, a 1-billion-parameter open-weights language model trained from scratch with differential privacy (DP). The model, based on the Gemma 2 architecture, is claimed to be the most capable DP-trained LLM released to date. Alongside the model, Google published new scaling laws that characterize the tradeoffs between compute, privacy budget, and utility in DP training.

VaultGemma uses a sequence-level DP guarantee of (ε ≤ 2.0, δ ≤ 1.1e-10) and was trained with a modified DP-SGD algorithm that handles variable batch sizes via Poisson sampling and padding. Google reports that the model shows no detectable memorization of training data in empirical tests. On benchmarks like HellaSwag, BoolQ, and ARC, VaultGemma performs comparably to non-private models from roughly five years ago, such as GPT-2 1.5B.

The release includes weights on Hugging Face and Kaggle, along with a technical report. Google positions VaultGemma as a step toward privacy-preserving AI for regulated industries, though a utility gap remains compared to non-private models. The accompanying scaling laws suggest that optimal DP training often favors smaller models with larger batch sizes.