Super Mario sets the stage for revolutionary AI benchmarking

In an intriguing study, researchers from the Hao AI Lab at the University of California San Diego have introduced a new benchmark for AI by using Super Mario Bros., a classic video game that presents unique challenges for modern artificial intelligence models. Unlike previous benchmarks that utilized games like Pokémon, which many deemed manageable, the intricate mechanics of Super Mario Bros. pose significant difficulties that push AI capabilities to their limits.

The Hao AI Lab conducted live tests on various AI models while they played Super Mario Bros., discerning their ability to navigate obstacles and enemies in a real-time environment. Among the participants was Anthropic’s Claude 3.7, which emerged as the standout performer, demonstrating skill and adaptability. Following it were Claude 3.5, Google’s Gemini 1.5 Pro, and OpenAI’s GPT-4o, both of which struggled to keep up with the game’s demands.

Importantly, this test was not confined to the original 1985 version of Super Mario Bros. Instead, it was played through an emulator integrated with the innovative GamingAgent framework developed by Hao. This framework allowed the AIs to exert control over Mario after providing them with essential gaming instructions. Examples include commands to jump or dodge when an enemy or obstacle was detected, translating these maneuvers into Python code for execution in the game.

The findings from Hao AI Lab’s experiments reveal that Super Mario Bros. requires the AI models to learn complex strategies and maneuvers actively. Notably, the researchers discovered that reasoning models, like OpenAI’s o1, struggled significantly in this setting, despite typically excelling at standardized benchmarking tasks. This could be attributed to the inherent nature of reasoning models, which process decisions through systematic problem-solving approaches that often take several seconds. This delay can prove detrimental in a fast-paced game like Super Mario Bros., where split-second choices can determine success or failure.

The use of gaming as a benchmark for AI has been a long-standing practice, yet it raises questions regarding the relevance and accuracy of such assessments. Some experts have posited that the complexities of real-world applications often do not reflect the simplified and heavily structured environments typical of video games, which might provide an endless stream of training data but lack genuine challenges faced in practical scenarios.

Andrej Karpathy, a notable research scientist and founding member at OpenAI, articulated concerns over the current state of AI metrics in a recent post, questioning how to effectively evaluate the capabilities of emerging models. “I don’t really know what metrics to look at right now,” he noted, reflecting a growing sentiment of uncertainty regarding the benchmarks used to assess AI advancements.

As the AI community navigates these new challenges, the insights gained from the Super Mario Bros. benchmark may not only provide entertainment for observers but could potentially shape future approaches to AI training and evaluation. This remarkable case highlights both the prowess and limitations of current AI technology, as researchers continue to seek better methods for gauging the effectiveness and adaptability of AI in dynamically complex environments.