Robots Atlas>ROBOTS ATLAS
Evaluation

ARC-AGI

2019ActivePublished
Key innovation
The only benchmark measuring "fluid intelligence" in AI โ€“ the ability to abstract and reason on entirely novel tasks based solely on core knowledge priors (shared by humans), without the ability to "buy" scores through massive training data.
Category
Evaluation
Abstraction level
Pattern
Operation level
Evaluation (runtime)
Use cases
measuring general AI intelligencefluid intelligence evaluationAGI researchtesting abstraction and reasoning capabilities

How it works

Each task consists of 2-5 demonstration pairs (colored pixel grids: input โ†’ output) and one or more test cases. The system must discover the rule governing the transformation and apply it. Answers are digital grids (up to 30x30 pixels, 10 colors). Scoring: binary success/fail per task; score is % of solved tasks.

Problem solved

Absence of a benchmark resistant to "buying scores" through massive training data; existing benchmarks measured stored knowledge (crystallized intelligence) instead of general reasoning ability (fluid intelligence) โ€“ preventing assessment of progress toward AGI.

Implementation

Reference implementations
ARC-AGI โ€“ official repository (GitHub)
Python
Official
Implementation pitfalls
Gap between training set and private test setHigh

Good scores on the public test set do not guarantee good performance on the private test set (ARC Prize evaluation).

Fix:Evaluate exclusively on the private set through the official ARC Prize competition.
Overfitting to known tasksCritical

Systems trained on known ARC tasks may overfit to their specific patterns without demonstrating genuine reasoning.

Fix:Use new tasks (ARC-AGI-2/3) and evaluate on the private test set.

Evolution

Original paper ยท 2019 ยท arXiv 2019 ยท Francois Chollet
On the Measure of Intelligence
Francois Chollet
2019
ARC and "On the Measure of Intelligence" paper published
Inflection point

Francois Chollet defines intelligence as skill-acquisition efficiency and introduces the ARC benchmark.

2024
ARC Prize 2024 โ€“ first systems exceed 55% on private test set
Inflection point

Public Kaggle competition with $1M prize pool attracts hundreds of teams; LLM+program synthesis hybrids exceed 55%.

2025
ARC-AGI-2 and ARC-AGI-3 โ€“ new, harder versions

ARC Prize Foundation releases new benchmark versions with harder tasks as models begin saturating ARC-AGI-1.

Hardware requirements

Primary

Pixel-grid benchmark; evaluation is hardware-agnostic although solver programs may leverage GPU.