RSS Feed

Symbiosis of Language and Action

Katharina J. Rohlfing (Supervisor), Kerstin Nachtigäller, Iris Nomikou

C, D

Abstract

The project investigates the ways in which language and action support the cognitive and linguistic development of a child, taking into account situative and intrapersonal factors. By combining observational and experimental studies of caregiver-child interactions and tutor-learner scenarios, the aim of the project is to specify parameters of caregiver or tutor behavior modification, which can be implemented in artificial systems as “top down constraints”, to allow learning, recognition and execution of human actions.

Research Questions

How does the symbiosis of language and action support the cognitive and linguistic development of a child depending on situative interpersonal factors, and
How can this symbiosis be used for technical systems (e.g. social robots), so that they can learn to recognize and carry out human actions?

The project addresses these questions by focusing on three particular aspects:

Goal-directed action as it becomes evident in the case of object manipulation. The focus lies in the variability and stability in the movement and speech of caregivers and how they convey knowledge about actions and relevant aspects of objects.
Meaning-oriented action in the form of manual gestures. Gestures support (mostly in preverbal infants) or supplement the semantic force of a verbal utterance. The research focus here is on situative and intrapersonal factors.
Conceptual development. The focus here is on the interplay between language and action as a mechanism promoting the development of meaning and the acquisition of new words. Depending on the language system, an action can be picked up conceptually and processed in memory in different ways. Everyday caregiver-infant interactions are considered by describing the ways in which a) actions make language more salient and b) language structures actions. How does the symbiosis develop over time? Another focus is on children‘s existing event knowledge and ways of associating it to their word learning processes. Can the association between children‘s event knowledge and a new word be conveyed in child-directed input and facilitate children‘s word learning?

Methods

A combination of methods is used to explore the above aspects in four different parallel projects:

Goal-directed action: In this study, computational analyses of hand trajectories are used to account for the movement adjustments that adults do in tutoring scenarios with children.
Gesture studies: In this longitudinal study, we elicit mother-child interactions in two different situations: reading a book and free play. We annotate and code gestural behavior of the mother and analyze it with respect to the specifics of the situation and the age of the child.
Language structures action: This study uses a naturalistic, longitudinal design to describe the interplay between language and action within the interactional context in which it occurs and to track how it is shaped in time by the developing infant. The subproject uses fine-grained qualitative analyses to extract behavioral features that are then used to describe the data quantitatively.
Associative event knowledge: In this project, a systematic combination of qualitative and quantitative methods is combined. In a first explorative study, the nature of the parental use of associative event knowledge of their children in child-directed input is determined. The effect of parental associative event knowledge on children‘s word learning is finally systematically investigated in a training study.

Outcomes

The symbiosis of language and action emerges as a teaching strategy as we found that –across different modalities (speech, gesture, actions and synchronized action- speech packages), behavioral modifications occur when addressing children and that robots also receive such modified input from which they can benefit. These modifications are present in the input within early interactions with infants. We found that German mothers vocalize in a tight relationship with their actions which effect is perceivable in language and tangible to the infants.

The symbiosis of action can be brought into a situation by associations. We found that embedding a new word in an action sequence (as an event unfolds) will facilitate the learning process. Our findings indicate a benefit of this associative input for children whose productive vocabulary was reported to be more advanced.

Publications

Szufnarowska, J., Nomikou, I. & Rohlfing K. J. (2012): Educating each other's attention: Mothers' and infants' eye-contact within natural interactions at 3 and 6 months of age. Poster presented at the CEU Conference on Cognitive Development, Budapest, Hungary, 12. – 14. January, 2012.[pdf]

Nomikou, I., Lohan, K. & Rohlfing, K.J. (2012): Adaptive maternal synchrony: multimodal practises are tailored to infants' attention. Poster presented at the CEU Conference on Cognitive Development, Budapest, Hungary, 12. – 14. January, 2012.[pdf]

Nachtigäller, K. & Rohlfing, K. J. (2011): A story about a word: Discoursive influences on fast mapping and retention of newly learned words. Paper presented at the XII. International Congress for Studies in Child Language, Montreal, Canada, 19. – 23. July.

Nomikou, I., Znajdek, J., & Rohlfing, K. J. (2011): Educating attention: Recruiting, maintaining and rewarding eye-contact in early natural mother-infant interactions. Paper presented at the Symposium on Asymmetric Interactions, Bielefeld, Germany, 30. – 31. May.

Nachtigäller, K., & K. Rohlfing. (2011). “Mothers' talking about early object and action concepts during picture book reading”, in: B. Kümmerling-Meibauer (Ed.), Where Literacy Begins. Children‘s Books From 0 to 3.. Amsterdam: John Benjamins.

Nachtigäller, K., & K. Rohlfing. (2011). “Einfluss von erlebten und vorgestellten Ereignissen auf die Erzählweise in kindgerichteter Sprache”. Zeitschrift für Literaturwissenschaft und Linguistik, 162, 139 - 155.

Rohlfing, K. J. & Tani, J. (2011): Grounding Language in Action. Special Issue of the IEEE Transactions on Autonomous Mental Development 3: 109–112.

Nomikou, I., & K. Rohlfing. (2011). “Language Does Something: Body Action and Language in Maternal Input to Three-Month-Olds”. IEEE Transactions on Autonomous Mental Development, 3(2), 113 - 128.

Nomikou, I. & Rohlfing, K. J. (2011): The role of intermodal synchrony in maternal input to 3-month-olds: A naturalistic study with special focus on the interplay between language and action. Poster presented at the CEU Conference on Cognitive Development, Budapest, Hungary, 14. – 16. January.[pdf]

Grimminger, A., K. Rohlfing, & P. Stenneken. (2010). “Children's lexical skills and task demands affect gestural behavior in mothers of late-talking children and children with typical language development”.Gesture, 10(2-3), 251 - 278.

Nachtigäller, K. & Rohlfing, K. J. (2010): Einfluss von erlebten und vorgestellten Ereignissen auf die Erzählweise in Kind-gerichteter Sprache. Interdisziplinärer Workshop "Spracherwerb und Kinderliteratur", Mainz, 23. – 24. September.

Nomikou, I. & Rohlfing, K. J. & Wagner, P. (2010). Exploring vocal-motor synchrony qualitatively: Maternal intermodal synchrony and rhythm in interactions with 3-4 month olds. Poster presented at the XVIIth Biennial International Conference on Infant Studies Baltimore, Maryland, 10. – 14. March. [pdf]

Nomikou, I. & Rohlfing, K. J. (2010).Exploring vocal-motor synchrony: Maternal intermodal synchrony and rhythmic patterning of interactions with 3-4 month olds. Paper presented at the Winter Symposium Meaning and Interaction. Dynamic models and social interaction, Aarhus, Denmark, 28. – 29 January.

Schillingmann, L., B. Wrede, & K. Rohlfing. (2009). “A Computational Model of Acoustic Packaging”.IEEE Transactions on Autonomous Mental Development, 1(4), 226 - 237.

Nagai, Y., & K. Rohlfing. (2009). “Computational Analysis of Motionese Toward Scaffolding Robot Action Learning”. IEEE Transactions on Autonomous Mental Development, 1, 44 - 54.