Learning How Humans Communicate Using Virtual Avatars

CITEC Social Motorics research project investigating social interaction using simulations of virtual agents

A nod of the head, an index finger touching the mouth, a wink: these are all gestures and facial expressions that we immediately understand – mostly even unconsciously. How exactly does interpersonal non-verbal communication work? To find this out, CITEC researchers Professor Dr. Stefan Kopp and Sebastian Kahl are researching virtual avatars in their project “Social Motorics.” What is entirely natural for people has to first be taught to virtual agents in small steps. By analyzing the learning process, the researchers are also obtaining new information on the human processes behind facial expressions and gestures. The Cluster of Excellence CITEC will be financing the project through December 2018.

The virtual humanoid robot Vince (left) and the artificial avatar Billie communicate with gestures. Photo: CITEC / Bielefeld University “Bill can react to speech, and understands statements like ‘aha’ and ‘m-hem,’ which is particularly important in the Kompass project. As part of the collaborate research center 673 ‘Alignment in Communication,’ we have taught Billie how to use speech and gestures in a coordinated way. Now we are using Billie in the ‘Social Motorics’ project,” says Professor Dr. Stefan Kopp, who heads the Social Motorics research project. “Billie can already move its hands, smile, and nod. In the next step, we will teach the avatars to interact with gestures and facial expressions even more naturally.” To do this, Kopp and Kahl let both avatars Vince and Billie talk to each other. The virtual humanoid robot Vince has already been in use at CITEC since 2009.

To interact with each other, Vince and Billie “meet” in a simulation. As in a game, the one avatar has to comprehend what the other one wants to convey to it non-verbally. Billie, for instance, mimes a square, and Vince has to recognize that it is a square, not a circle, and repeat the movement. The researchers make the communication game harder for Billie and Vince by interfering with the gestures: when both avatars no longer understand, they are meant to recognize the problems and solve them on their own. In doing this, the avatars act largely autonomously, confirming with a nod or by shaking their head. In order for them to really speak with each other, they have to feel that they are being addressed. For this, the researchers control the avatars’ gaze.

“At the heart of our research is the question of how humans communicate both verbally and non-verbally. Therefore, I try to recreate the cognitive communication abilities of the human brain in a computer model,” says Sebastian Kahl, who is writing his doctoral dissertation on social motorics. “Humans have two important abilities that we are investigating. On the one hand, they can intuitively comprehend and recognize via mirror neurons what their counterpart wants to say from gestures such as winking, smiling, and nodding. On the other hand, people can almost ‘read someone’s thoughts’ by imagining what their interlocutor is thinking or feeling at that moment. This ability is called ‘Theory of Mind.’” This theory is not only a scientific description, but also a mental ability that humans have. Due to the ability of Theory of Mind, humans are able to make predictions or end their counterpart’s sentences.

In order to enable Vince and Billie to learn the art of Theory of Mind, Kopp and Kahl developed their own computer model for both avatars. With this model, the researchers can see exactly how well the communication between the virtual avatars works. For their research, they draw upon their own simulation studies, which they are continuously developing further. In the future, the researchers would like to find out how the avatars are influenced when they already know their counterpart. Whether the avatars have caused a gesture, or whether it was their counterpart is something Vince and Billie still have to learn. Up to this point for the avatars, it is similar to humans who have a schizophrenic disorder, who do not always know whether they have done something, or if it was caused by someone else. Resolving this confusion for the avatars is the next challenge for Kopp and Kahl.

With their research, Stefan Kopp and Sebastian Kahl have already been able to demonstrate that Theory of Mind has a significant impact on avatars’ comprehension. The better that Billie and Vince can predict what the other wants, the easier the communication is for them. The researchers have already presented findings from their simulation studies at several conferences. Kopp and Kahl are also in close contact with other researchers at the Uniklinik Köln (University Clinic of Cologne) and the Max Planck Institute in Tübingen to bring even greater practical relevance to their work. In the future, their research may help make it easier – and more precise – for humans to communicate with virtual agents.

Social Motorics is one of five small “high-risk” research projects at CITEC. In this project line, researchers deal with particularly innovative and original topics in which the outcome is open ended. In addition to these rather small, high-risk projects, there are also four large-scale research projects and eight interdisciplinary projects. Beyond these, researchers from the Graduate School also conduct research on their own innovative individual projects.

More information is available online:

Social Motorics project description: https://cit-ec.de/en/social-motorics


Prof. Dr.-Ing. Stefan Kopp, Bielefeld University
Head of Social Cognitive Systems Group

Faculty of Technology and the Cluster of Excellence Cognitive Interaction Technology (CITEC)
Telephone: +49 521 106-12144
Email: skopp@techfak.uni-bielefeld.de