IQ test for artificial intelligence systems

Screen shot of a video game.
The AI systems will be evaluated based on their ability to learn and play new video games, such as Vizdoom.

Washington State University researchers are creating the first-ever “IQ test” for artificial intelligence (AI) systems that would score systems on how well they learn and adapt to new, unknown environments.

Diane Cook, Regents Professor and Huie-Rogers Chair Professor, and Larry Holder, professor in the School of Electrical Engineering and Computer Science, received a grant of just over $1 million from the Defense Advanced Research Projects Agency (DARPA) to create a framework to test the “intelligence” of AI systems.

“Previously, research on measuring intelligence in AI systems has been mostly theoretical,” Holder said. “They didn’t measure real-world performance in novel, previously unseen environments and didn’t account for the complexity of the tasks.”

Holder and Cook will design a test that will grade AI systems based on the difficulty of problems that they can solve. Creating methods to rank problems on their difficulty will be one of the major parts of the research.

Closeup of Larry Holder
Larry Holder

“The score will also take into account the systems’ accuracy, correctness, time taken and amount of data they need to perform well,” Holder said.

One of the main challenges for AI researchers is creating machines that can learn in an unfamiliar environment like a human can. Many recent advances in machine learning and AI have been in narrow, specialized tasks like detecting faces and predicting baseball match results.

Creating machines that can learn and act intelligently in novel environments is still the holy grail of AI research.

“We are focused on testing and improving systems that can be more general-purpose, like a robot assistant that can help you with many of your day-to-day tasks,” he said.

One of the behaviors Holder and Cook will be looking at is how well the machines they test can transfer their learning from one task and apply it to a new, unseen task.

“For example, you might want to learn checkers before chess because you can easily transfer your knowledge from one to the other,” Holder said.

Closeup of Diane Cook
Diane Cook

An undergraduate student who did summer research with Holder helped him come up with an evaluation environment that would be used to test AI systems on tasks like playing video games, solving SAT test problems and solving the Rubik’s cube.

Cook is also the director of the Center of Advanced Studies in Adaptive Systems (CASAS) at WSU, which focuses on creating smart homes with robot assistants that care for the elderly, among other projects. These robots can be used to monitor the safety, health, mobility and social life of elders.

Holder currently maintains a website that hosts the AIQ challenge and leaderboard. As part of the grant, Holder hopes to expand the publicly available software interface, which can be used by anyone to test their AI system while providing more data to the researchers in the process.

This grant is part of DARPA’s Science of Artificial Intelligence and Learning for Open-world Novelty (SAIL-ON).

Media contacts:

  • Diane Cook, Regents professor, School of Electrical Engineering and Computer Science, 509-335-4985,
  • Larry Holder, professor, School of Electrical Engineering and Computer Science,
  • Tina Hilding, communications director, Voiland College of Engineering and Architecture, 509-335-5095,

Next Story

Faculty, AP staff to receive 2% salary increases

Funding for a 1.6% increase was provided by the Washington Legislature, with WSU contributing additional resources to round up the pay boost for thousands of employees across the university system.

Recent News

Brad Corbin named to NCAA Division I Council

The National Collegiate Athletic Association recently appointed Corbin, deputy director of athletics, to the council for a four‑year term.

New spring wheat variety named for pioneering Black family

Bush soft white spring wheat honors settler George Bush and his family who helped indigenous populations battle disease and saved fellow settlers during the 1852 famine.

Robotic gripper for automated apple picking developed

A robotic gripper developed by WSU researchers was able to successfully grab more than 87.5% of the apples in an orchard without damaging the fruit.