IQ test for artificial intelligence systems

Screen shot of a video game.
The AI systems will be evaluated based on their ability to learn and play new video games, such as Vizdoom.

Washington State University researchers are creating the first-ever “IQ test” for artificial intelligence (AI) systems that would score systems on how well they learn and adapt to new, unknown environments.

Diane Cook, Regents Professor and Huie-Rogers Chair Professor, and Larry Holder, professor in the School of Electrical Engineering and Computer Science, received a grant of just over $1 million from the Defense Advanced Research Projects Agency (DARPA) to create a framework to test the “intelligence” of AI systems.

“Previously, research on measuring intelligence in AI systems has been mostly theoretical,” Holder said. “They didn’t measure real-world performance in novel, previously unseen environments and didn’t account for the complexity of the tasks.”

Holder and Cook will design a test that will grade AI systems based on the difficulty of problems that they can solve. Creating methods to rank problems on their difficulty will be one of the major parts of the research.

Closeup of Larry Holder
Larry Holder

“The score will also take into account the systems’ accuracy, correctness, time taken and amount of data they need to perform well,” Holder said.

One of the main challenges for AI researchers is creating machines that can learn in an unfamiliar environment like a human can. Many recent advances in machine learning and AI have been in narrow, specialized tasks like detecting faces and predicting baseball match results.

Creating machines that can learn and act intelligently in novel environments is still the holy grail of AI research.

“We are focused on testing and improving systems that can be more general-purpose, like a robot assistant that can help you with many of your day-to-day tasks,” he said.

One of the behaviors Holder and Cook will be looking at is how well the machines they test can transfer their learning from one task and apply it to a new, unseen task.

“For example, you might want to learn checkers before chess because you can easily transfer your knowledge from one to the other,” Holder said.

Closeup of Diane Cook
Diane Cook

An undergraduate student who did summer research with Holder helped him come up with an evaluation environment that would be used to test AI systems on tasks like playing video games, solving SAT test problems and solving the Rubik’s cube.

Cook is also the director of the Center of Advanced Studies in Adaptive Systems (CASAS) at WSU, which focuses on creating smart homes with robot assistants that care for the elderly, among other projects. These robots can be used to monitor the safety, health, mobility and social life of elders.

Holder currently maintains a website that hosts the AIQ challenge and leaderboard. As part of the grant, Holder hopes to expand the publicly available software interface, which can be used by anyone to test their AI system while providing more data to the researchers in the process.

This grant is part of DARPA’s Science of Artificial Intelligence and Learning for Open-world Novelty (SAIL-ON).

Media contacts:

  • Diane Cook, Regents professor, School of Electrical Engineering and Computer Science, 509-335-4985,
  • Larry Holder, professor, School of Electrical Engineering and Computer Science,
  • Tina Hilding, communications director, Voiland College of Engineering and Architecture, 509-335-5095,

Next Story

Recent News

WSU joins the College Cost Transparency Initiative

Washington State University is working alongside hundreds of institutions of higher education to make the cost of college clearer for prospective students.

Computer science project has students singing a new tune

A WSU student team developed an India raga music app for a startup company as part of a senior program that has students work on projects for real-world clients.

SURCA 2024 applications open to WSU undergraduates through Feb. 20

Students can submit abstracts to present their mentored research, scholarship, and creative activity for the Showcase for Undergraduate Research and Creative Activities this spring.

IRS special agents lead accounting exercise

Known as the IRS Citizens Academy, it is intended to showcase potential career pathways for accounting students within the agency’s enforcement arm.