Language Comprehension in Visual Context: Children versus Adults

Research Areas: 

It has been established that adults can integrate non-linguistic information such as from a visual context rapidly and incrementally during spoken language comprehension. However, few studies have investigated children’s use of visual context for language comprehension or compared adult and child language comprehension in visual contexts. Our current research focuses on eye tracking the time course with which visual context such as depicted events affects children’s language comprehension.


Methods and Research Questions: 

How visual context affects adult language comprehension has been extensively investigated. We know that adults can use visual context rapidly and incrementally to inform language comprehension. Only few studies, in contrast, have examined how children use visual context for language comprehension, and with conflicting results.

Findings from the few studies that have examined visual context effects in children suggest that children cannot use visual context for language comprehension to the same extent as adults. Overall, the extent to which children can use visual context for real-time language comprehension is unclear. This is an important issue first because knowing more about the developmental origin of visual context effects can help us to better understand adult language comprehension. Furthermore, finding out to which extent visual context effects on comprehension are similar in children and adults can contribute to our understanding of whether children and adults rely on the same mechanisms for language comprehension. That is important for modeling human language comprehension. Do we need two fundamentally different models or can we rely on the same model but perhaps weigh the role of knowledge and working memory differently in children vs. adults. A series of eye-tracking experiments and working memory tests will be conducted to address this issue. Results from a first study are reported below.

An eye-tracking study examined whether – just like adults – 4-5-year old children can rapidly use depicted events for thematic role assignment in spoken language comprehension. For adults we know they inspect characters as they are mentioned, and when they hear a sentence fragment (e.g., “Der Bär malt…” vs. “Den Bär schubst…”) referring to an action they can anticipate the patient (e.g., the worm) or the agent (e.g., the bull-pushing) of the respective verb. A pretest ensured children understand the stimuli (Fig. 1). In the eye-tracking experiment, children saw either a picture with depicted events (Figure 1(a)) or a picture without any actions (Figure 1(b)). Concurrently they heard either a subject-initial sentence or a structurally unambiguous object-initial sentence (Fig. 1), and answered a question about either the agent or the patient of the action. Previous research revealed that 4-5 year old children had low accuracy on questions about non-canonical object-first sentences. Our design permitted us to investigate whether children benefit from the depicted scene events for understanding object-initial sentences. Furthermore, eye tracking allowed us to track the moment-by-moment processes of children’s language processing, and thus to see whether children resemble adults in their ability to rapidly use depicted events for understanding who-does-what-to-whom.


We analyzed both the accuracy with which children responded to questions about the events (e.g., “Who painted here?”) and their eye gaze to the mentioned animals (the bear, the worm, and the bull in Fig. 1). Children made more mistakes when they were answered questions about a difficult OVS sentence and the scene didn’t depict actions (Fig. 1(b)) than when the scene depicted the actions (Fig. 1(a)).

Unlike adults, they did not show clear anticipatory eye movements to the agent of a pushing event for object-initial sentences such as “Den Bär schubst…” although they looked at the agent more often shortly after the verb when the scene depicted events than when it didn’t. This finding revealed children’s incremental sensitivity to the depicted event information in visual context during sentence comprehension. It suggested that, just like adults, children are able to make rapid use of depicted events for thematic interpretation although the gaze pattern also revealed clear differences for children relative to adults.