New Artificial Intelligence Challenge Could Be the Next Turing Test.

A recently released biopic of Alan Turing ("The Imitation Game") tells the story of the British mathematician and cryptographer who built a machine to crack the German Enigma code during World War II. But Turing is perhaps best known for his pioneering work on artificial intelligence.

In 1950, Turing introduced a landmark test of artificial intelligence. In the so-called Turing test, a person engages in simultaneous conversations with both a human and a computer, and tries to determine which is which. If the computer can convince the person it is human, Turing would consider it artificially intelligent.

The Turing test has been a helpful gauge of progress in the field of artificial intelligence (AI), but it is more than 60 years old, and researchers are developing a successor that they say is better adapted to the field of AI today. 

The Winograd Schema Challenge consists of a set of multiple-choice questions that require common sense reasoning, which is easy for a human, but surprisingly difficult for a machine. The prize for the annual competition, sponsored by the Burlington, Massachusetts-based software company Nuance Communications, is $25,000.

"Really the only approach to measuring artificial intelligence is the idea of the Turing test," said Charlie Ortiz, senior principal manager of AI at Nuance. "But the problem is, it encourages the development of programs that can talk but don't necessarily understand."

The Turing test also encourages trickery, Ortiz told Live Science. Like politicians, instead of giving a direct answer, machines can change the subject or give a stock answer. "The Turing test is a good test for a future in politics," he said.

Earlier this year, a computer conversation program, or "chatbot," named Eugene Goostman was said to have passed the Turing test at a competition organized by the University of Reading, in England. But experts say the bot gamed the system by claiming to speak English as a second language, and by assuming the persona of a 13-year-old boy, who would dodge questions and give unpredictable answers.

In contrast to the Turing test, the Winograd Schema Challenge doesn't allow participants to change the subject or talk their way around questions — they must answer the questions asked. For example, a typical question might be, "Paul tried to call George on the phone, but he wasn't successful. Who was not successful?" The correct answer is Paul, but the response requires common sense reasoning.

"What this test tries to do is require the test taker to do some thinking to understand what's being said," Ortiz said, adding, "The winning program wouldn't be able to just guess."

Although the Winograd Schema Challenge has some advantages over the Turing test, it doesn't test every ability that a truly intelligent entity should possess. For example, Gary Marcus, a neuroscientist at New York University, has promoted the concept of a visual Turing test, in which a machine would watch videos and answer questions about them.