Nvidia Introduces Climb: A Framework For Iterative Data Mixture Optimization In Language Model Pretraining

Challenges successful Constructing Effective Pretraining Data Mixtures

As ample connection models (LLMs) standard successful size and capability, nan prime of pretraining information remains a captious determinant of downstream performance. Most LLMs are trained connected large, web-scale datasets specified arsenic Common Crawl, which supply wide sum but deficiency definitive domain labels. This introduces difficulties successful curating mixtures that equilibrium wide knowledge pinch domain-specific expertise.

Manual dataset curation, arsenic seen successful efforts for illustration The Pile, is labor-intensive and does not standard well. Moreover, nan nonlinear narration betwixt information creation and exemplary capacity makes it non-trivial to find what proportions of domain information are optimal. These constraints motivate nan request for automated, scalable, and adaptive information action methods.

CLIMB: An Iterative Framework for Data Mixture Discovery

To reside this, NVIDIA researchers propose CLIMB—CLustering-based Iterative Data Mixture Bootstrapping—a model that automates nan find and refinement of information mixtures for connection exemplary pretraining. CLIMB combines unsupervised clustering pinch iterative optimization to place mixtures that are well-suited for wide aliases domain-specific objectives.

The pipeline originates by embedding large-scale matter information into a semantic abstraction utilizing pretrained encoders. K-means clustering is past applied to shape nan information into coherent groups, which are pruned and merged based connected contented value and redundancy. This forms nan ground for constructing campaigner mixtures.

Subsequently, CLIMB uses proxy models to measure sampled mixtures and fits a regression-based predictor (e.g., LightGBM) to estimate substance performance. An iterative bootstrapping process progressively refines nan sampling space, prioritizing high-performing configurations. This allows CLIMB to converge connected an effective information substance nether a fixed compute budget.

Technical Details and Design Considerations

The optimization process is framed arsenic a bi-level problem: astatine nan little level, proxy models are trained connected campaigner mixtures; astatine nan precocious level, a predictor is learned to approximate capacity outcomes. This predictor guides further sampling and pruning, enabling businesslike exploration of nan substance space.

CLIMB supports sparsity successful substance weights, encouraging nan find of compact, domain-relevant information subsets. The usage of clustering complete embeddings—rather than token-level features—ensures semantic coherence wrong clusters. The iterative refinement is system to equilibrium breadth (search abstraction coverage) pinch extent (predictive accuracy), and ablation studies corroborate that observant compute allocation crossed iterations improves convergence and last performance.

The model besides exhibits robustness crossed proxy exemplary sizes and cluster granularities. While larger proxy models output somewhat amended predictions, moreover smaller models sphere cardinal structural trends. Similarly, CLIMB is comparatively insensitive to first cluster count, provided it is wrong a reasonable range.

Empirical Evaluation and Observations

CLIMB was evaluated connected respective wide reasoning tasks, including PIQA, ARC (Easy and Challenge), HellaSwag, and WinoGrande. A 1B-parameter exemplary trained connected CLIMB-discovered mixtures achieved an mean accuracy of 60.41%, outperforming comparable baselines specified arsenic DoReMi and RegMix.

When extended to 400B-token pretraining, this 1B exemplary outperformed Llama-3.2-1B by 2.0% connected a wide suite of benchmarks. Similarly, successful nan sub-500M exemplary category, CLIMB-based pretraining led to accordant improvements complete models for illustration SmolLM and TinyLlama.

Domain specialization further highlights CLIMB’s utility. In targeted MMLU benchmarks crossed STEM, humanities, and societal sciences, CLIMB-trained models outperformed some random action and exhaustive hunt baselines. The iterative process showed accordant gains complete each stage, indicating effective guidance from nan predictive model.

To facilitate reproducibility and further research, NVIDIA has released 2 resources:

• ClimbLab: A 1.2-trillion-token corpus organized into 20 semantic clusters.
• ClimbMix: A 400-billion-token optimized substance for businesslike pretraining.

Models trained connected ClimbMix outperform those trained connected datasets for illustration Nemotron-CC and SmolLM nether balanced token budgets, demonstrating improved scaling characteristics.

Conclusion

CLIMB presents a systematic attack for optimizing information mixtures successful LLM pretraining. By combining semantic clustering pinch proxy-based iterative search, it avoids reliance connected manual annotations aliases fixed heuristics. The method supports some generalist and master training goals and adapts to varying compute and information constraints.

This model contributes to ongoing efforts successful data-centric AI by offering a scalable and opinionated replacement to handcrafted information pipelines. Its empirical capacity underscores nan value of information substance optimization successful maximizing exemplary utility, peculiarly nether fixed assets budgets.

Check retired nan Paper, ClimbLab connected HF and ClimbMix connected HF . Also, don’t hide to travel america on Twitter and subordinate our Telegram Channel and LinkedIn Group. Don’t Forget to subordinate our 90k+ ML SubReddit.

🔥 [Register Now] miniCON Virtual Conference connected AGENTIC AI: FREE REGISTRATION + Certificate of Attendance + 4 Hour Short Event (May 21, 9 am- 1 p.m. PST) + Hands connected Workshop