Intelligence Tests for Robots? Workshop on Benchmarks for Cognitive Systems

Top international researchers meet in Bielefeld to discuss new methods for benchmarking and evaluating interactive cognitive systems.

On 7–8 February, the Bielefeld Center of Excellence Cognitive Interaction Technology (CITEC) is running an international workshop on "Benchmarking and evaluating interactive cognitive systems" at the Center for Interdisciplinary Research (ZIF). This workshop makes Bielefeld University an international leader in combining the technological development of interactive cognitive systems with methods of analysis taken from psychology, linguistics, and sociology.

The goal of the workshop is to develop new methods for assessing the quality of interactive cognitive systems. Benchmarking, that is, the systematic testing of virtual agents and interactive robot systems in a clearly defined scenario, is of far-reaching significance on the path from basic research to applications in daily life. Future benchmarks in this field of research will have a lasting impact on the direction of research.

Interactive cognitive systems are technological systems (e.g. robots, virtual agents, or dialogue systems) equipped with abilities that are otherwise attributed particularly to human beings such as finding one's way in the world, grasping and manipulating objects, talking to and understanding the people around one, carrying out a conversation and passing on information, or handling social situations. In contrast to the highly precise and strongly repetitive routines so typical of computers and robots, such abilities require soft, flexible, and self-adaptive behaviour. The goal underlying the development of such technologies is for machines to adapt to human beings and to the surroundings developed for human beings and not vice versa. However, this also means that previous paradigms for measuring the performance of systems no longer hold, making quantitative progress in this field of research increasingly difficult to confirm, compare, and communicate.

In recent years, general progress in research on cognitive robotics, social agents, and human-robot interaction has led to amazing results. Nonetheless, it is still very hard to compare and evaluate the performance of different systems because of the lack of appropriate evaluation standards. The workshop will discuss not only the latest developments in the field of systems evaluation but also future demands on benchmarking.
Alongside current advances, developments, and research trends, the workshop will encourage the discussion of central questions such as:

  • How do existing robotics competitions influence the development of technological systems? And how should they be adapted or optimized to reflect the real challenges of the future?
  • What preconditions have to be met in order to compare approaches and techniques on a systems level?
  • Is it possible to measure the "cognitive abilities" of a system? If yes, then with which units of measurement? How can this be related to decisions on systems design?

To meet the strong need for such new and objective ways of making comparisons, internationally renowned researchers from Japan, the United States, Canada, Sweden, Italy, Spain, France, the Netherlands, Austria, and Germany are taking part in the workshop. Alongside top addresses for humanoid and service robots, researchers from the disciplines of linguistics, psychology, automatic language processing, machine learning, and virtual agents are also represented in this strongly interdisciplinary team. The Bielefeld Center of Excellence Cognitive Interaction Technology (CITEC) has been funded by the Excellence Initiative of the German Federal and State Governments since November 2007. By setting up and organizing this workshop, CITEC is addressing an urgent research need, thereby once gain confirming its role as an innovative international center of competence and research in the domain of "interactive intelligent systems" and strengthening international awareness of this research specialization at Bielefeld University.

To find out more in the Internet, go to