An Intelligence Test for Artificial Intelligence: ARCathon – Global AI Competition – Davos

The Global Swiss AI Award honours the most promising and innovative AI start-ups from across Switzerland and selects an individual who has impacted the field of AI at a global scale. The application process grounds a scouting process, tracing and following hundreds of AI companies in Switzerland and beyond. From these, several scientists, entrepreneurs, and other leaders in the field are profiled and rated along an established catalogue of criteria. An important decision criterion is the timeframe of the impact, which should encompass timeframes at maximum 3 years until a significant impact can be assessed, hence allowing young scientists and yet unknown individuals to gain visibility and become a nominee. The selection process and the jury are headed and organized by the Mindfire Foundation. In 2021, François Chollet won the last Global Swiss AI Award ‘for outstanding global impact in AI’. He is the inventor and main driver behind the Keras deep learning environment and the author of several high-impact publications.

First winners of ARCathon

Lab42 announces the names of the first winners of ARCathon, one of the most challenging and important competitions in AI. Out of 118 participating teams from 47 countries, the best solution of the first edition achieved a solution rate of 6%. At the end of the competition, ARCathon winner Michael Hodel, a master’s student in computer science at ETH Zurich, Switzerland, further optimized his and other solutions, setting a new world record of 30.5%. The previous world record was set at a Google Kaggle competition in January 2020. ARCathon will launch the next edition of ARCathon on January 31, 2023, open to all AI enthusiasts worldwide, to continue to push the boundaries of cutting-edge AI.

The ARC challenge

The ARC challenge is referred to as one of the hardest and most important competitions in AI (quote TechTalks, 2020). ARCathon is the first global AI competition of Lab42, which opened its doors on July 1st, 2022, in Davos. The committed goal of Lab42 is to develop human-level AI for humankind, addressing the global AI research community. In this spirit, 118 teams from 47 countries followed the call of Lab42 to design algorithms capable of human-level abstract thinking and reasoning. Solving the Abstraction and Reason- ing Corpus (ARC) of François Chollet, AI pioneer and Co-Host of the ARCathon, is a first step towards human thinking and reasoning capabilities in algorithms, which today’s state-of-the-art deep learning models do not possess.

An Intelligence Test for Artificial Intelligence

An example from the ARC challenge – an intelligence test for artificial intelligence. François Chollet

ARC entails over 1’000 unique and discrete puzzles (see picture 1) that require abstract problem-solving skills independent from learned expertise and knowledge about the world. Therefore, the solution of each ARC puzzle relies solely on innate abstraction and reasoning skills, such as the ability to count and detect objects and symmetries. In a study from New York University, humans solved over 80% of all ARC tasks effortlessly. Contrary, the world record for state-of-the-art models and algorithms reached a ceiling at 20%, while aggregate solutions of existing approaches did not exceed the 30% mark – until January 2023 when Michael Hodel handed in his solution to both the Kaggle and the Lab42 platform, confirming a new world record for ARC.

Current approaches to AI maneuver firmly in the direction of deep learning, a powerful tool to solve specific problems with gigantic amounts of data. Nowadays, so-called transformer neural networks can even be trained to interact with humans, like the chatbot ChatGPT (GPT = Generative Pre-trained Transformer). Alt- hough the bot outputs human-like texts, summaries, poems, and even programming code, the model cannot innovate, reason, or find new insights beyond the data it was trained with. The model calculates the probability of the next word in a sentence, which is why it sometimes produces untrue or nonsense state- ments. This bottleneck is called the Grounding Problem.

Since today’s AI approaches lack any understanding of the meaning behind the words it outputs, no new knowledge about the world can be gained autonomously. Tackling ARC requires skills such as reasoning and grounding on information to solve unfamiliar and never encountered problems. ARC is currently con- sidered the gold standard IQ test for AI, as it requires the development of entirely new algorithms to solve its 100 secret tasks. In the first edition of Lab42’s ARCathon, the most promising individuals and teams were identified globally, and the challenge has been institutionalized, paving the way for the ARCathon edition.

Picture  2

The 1st place finisher Michael Hodel (see picture 2) from Zurich, Switzerland, was also the one who pushed the existing ARC world record beyond the 30% mark after ARCathon 2022 ended. In 2nd place comes Jozef Kopanicak with his team from the Mirus Software AG based in Davos, Switzerland, and in Zilina, Slovakia. 3rd place finisher is iOS developer Simon Strandgaard from Copenhagen, Denmark. Congratulations to the podium on this outstanding achievement!

Start of the next ARCathon edition

With a strong starting point and a critical mission, Lab42 starts the next edition of ARCathon on January 31st, 2023. Lab42 is committed to harnessing this collective energy and expertise to drive progress toward human-level AI for humankind.

Registration is now open at https://lab42.global/arcathon.

Source: lab42, 31. Januar 2023

Home

 

For immediate publication: January 31, 2023 | 3/3