Talking Stick - sensorimotor grounding of language

How can a behaviour, which is defined by a multitude of sensorimotor processes, be linked to a condensed description like a single word? This yet unsolved problem concerns the gap between representations of sensorimotor capabilities and symbolic representations as they are used in language production and comprehension. In the Talking Stick project, we are going to extend our sensorimotor concept of cognition to language. The idea is to interconnect the following two complementing approaches:

• Our approach termed Walknet – an artificial neural network which uses biologically inspired reactive mechanisms for the control of a six-legged walker, that at the same time can be used for planning ahead. This is done by applying the same controller structures to the neurally implemented internal model thus forming a mental simulation.
• The Neural Theory of Language (NTL, ICSI Berkeley) to extend our architecture by connecting these structures to language and in this way grounding words in these structures and meaning as a whole in the internal simulation of the related behaviours.

The Neural Theory of Language project connects metaphors used in our daily language to the related motor events. This indicates a tight connection between representations and neural structures used in the execution of behaviour and those used in language. This view is supported by findings in neurobiology and is extended there to also include planning and observing of actions: It has been proposed that action understanding - in language comprehension and production, - in observing the actions of another person, - or as needed for planning ahead is based on the same neural substrate which is used for the generation of an action itself in the first place. Therefore, in the NTL project, principles of motor control are used in order to solve problems of language production and language comprehension. The NTL project links language to action or, more precisely, the meaning of language is constituted by the means of an internal simulation which can be thought of as the basis for planning ahead or even thinking itself.Connecting these two approaches is exciting because core questions of mental manipulation and of thinking in general could be addressed or even solved and could be linked to different capabilities, like language production and understanding, thinking and acting.[view:groupmembers==188]