Nuked 
Greetings, tech enthusiasts! Today, we’re diving into an exciting development in the world of artificial intelligence.
The Arc Prize Foundation, spearheaded by the influential AI researcher François Chollet, has unveiled a groundbreaking test designed to assess the general intelligence of advanced AI models. Dubbed ARC-AGI-2, this new assessment is proving to be quite the puzzle for many leading models.
Notably, reasoning-capable AI systems, like OpenAI’s o1-pro, are scoring below 2% on this challenging test. Infamous for their computational prowess, non-reasoning models such as GPT-4.5 and Claude 3.7 are also struggling similarly.
The test itself is a series of intricate problems that demand an AI’s ability to recognize visual patterns amidst colorful squares, ultimately constructing the correct answer grid. This format pushes AI systems to confront novel challenges they haven’t encountered during their training.
To set the stage, over 400 participants took the test, achieving an impressive average accuracy of 60%. An impressive feat, especially when compared to the AI models’ scores.
Chollet emphasizes the test’s value in distinguishing more genuinely capable AI from simple brute-force processors, tweaking the metrics to measure efficiency instead of merely output proportion.
The stakes are high, as the newly announced Arc Prize 2025 contest challenges developers to hit an ambitious target of 85% accuracy on the ARC-AGI-2 while keeping costs per task extremely low.
This new direction comes amidst growing calls from tech leaders for innovative metrics to effectively gauge AI accomplishments, focusing not only on performance but also efficiency in achieving those results.
