Llama 3.1 70B

Llama 3.1 70B is an autoregressive, multilingual large language model developed by Meta, released on 23 July 2024 as part of the Llama 3.1 family (alongside the 8B and 405B variants). The model has 70 billion parameters and uses an optimized transformer architecture with Grouped-Query Attention (GQA), RoPE positional encoding (theta 500,000), RMSNorm normalization and the SwiGLU activation.

The 70B variant comprises 80 layers, a hidden dimension of 8192, 64 attention heads and 8 key-value (KV) heads, with a vocabulary of 128,256 tokens. It supports a 128,000-token context window. It was trained on over 15 trillion tokens of publicly available data, with a knowledge cutoff of December 2023.

It officially supports eight languages: English, German, French, Italian, Portuguese, Hindi, Spanish and Thai. The pretrained (base) version serves as a foundation for fine-tuning to text-generation tasks, while the instruction-tuned variant targets conversational use and supports tool calling.

The model weights are publicly available under the Llama 3.1 Community License, permitting commercial and research use with certain restrictions (including a 700M monthly-active-users threshold requiring separate Meta approval).