We try to capture and model these mechanisms to investigate natural and embodied social interaction with the virtual robot VINCE. Focusing on communicative hand gestures, motor cognition is defined to encompass all processes involved in the production and comprehension of one’s own and others’ gestures. We adopt the idea of shared motor structures that represent how own gestures are performed (generation) and are also exploited to make sense of the gestures other‘s are performing (perception), constituting a „perception-behavior expressway“ (Dijksterhuis & Bargh 2001). Our research questions are: (1) How can such structures be modeled and linked into gesture perception and production processes? (2) How are they acquired and learned incrementally during social interaction? (3) How can these structures root in observed or performed instances of gestural behavior (exemplar-based approach), but eventually afford the generativity of human social behavior? (4) How can we, based on this model, simulate human social capabilities such as incremental anticipation, mutual coordination and alignment in artificial agents?

Our technical testbed consists of marker-free body tracking and speech recognition technology, with which VINCE can engage with a human in verbal and nonverbal interaction. In this setup, VINCE is equipped with a computational model that puts motor knowledge in the middle between behavior generation and perception, both processes using and affecting these structures simultaneously (see Figure). In this way the agent perceives and recognizes gestures by comparing them to its own motor simulation, i.e. how it would perform that gesture. The model is graph-based and hierarchical, representing motor knowledge on three different levels of abstraction, from small movement segments (motor commands), to whole performances of a gesture (motor programs), to more general schema representations (motor schemas) leading into levels of social or communicative intention. The representation is defined as a Bayesian network. Updated probabilities of each node, resulting from Bayesian inference when predictions (forward models) are compared against actual observations, indicate neural activation of the corresponding sensorimotor element. Activations spread within each level and float in-between adjacent levels (bottom-up/top-down). Since they cease slowly, interleaved generation and perception--as in reciprocal interaction--come to affect each other.

« close