The LLM360 framework requires releasing: (1) training code and recipe, (2) the full training dataset including processing/sequence order, (3) all intermediate checkpoints saved during training, (4) hyperparameters, and (5) logs, metrics and intermediate analysis results (Analysis360). This lets anyone reproduce or resume training, analyze model behavior across checkpoints, and build on intermediate model states.
Most models labeled 'open-source' release only final weights and a brief report, making it impossible to reproduce training, audit the data, or study training dynamics and emergent behavior. LLM360 addresses this lack of transparency by releasing the complete set of artifacts required for full reproduction and analysis.
The complete pre-training code together with the recipe: architecture, hyperparameters and run configuration, enabling faithful reproduction of the training run.
The complete training data together with the order/sequence in which it is fed to the model, enabling data auditing and exact reproduction of the same learning trajectory.
All intermediate weight states saved during training (not only the final checkpoint), allowing study of training dynamics, emergent phenomena, and resuming training from any point.
Recorded metrics, learning curves and logs from the training run, documenting the behavior of the model and the optimization process over time.
A companion project/repository providing analyses of intermediate checkpoints and results, supporting research into model behavior across training stages.
Official
The arXiv:2312.06550 paper defines the 'fully open' LLM standard and introduces the first two 7B models together with a complete set of training artifacts.
K2-65B (1.4T tokens, Apache 2.0) reaches performance comparable to Llama 2 70B with roughly 35% less compute; developed jointly by MBZUAI, Petuum and LLM360 with all artifacts released.
The LLM360 team continues its work as IFM (launched May 2025), moving the official Hugging Face organization to IFM and developing families such as K2 Horizon; models under Apache 2.0 and datasets under ODC-BY-style licenses.
