Literacy modulates language-mediated visual attention and prediction

12. Januar 2012

In this talk I will describe my recent research on the influence of formal  literacy (i.e. the ability to read and write) on spoken language-mediated visual orienting. We used a simple look and listen task which resembles every day behavior. Main measures of interest were the timing of shifts in eye gaze and the magnitude of fixation proportions.
    In Experiment 1, 42 high (mean age 24 years; 15 years of schooling) and 32 low  (mean age 27 years, 2 years of schooling) Indian literates listened to spoken sentences containing a target word (e.g., 'magar', crocodile) while at the same time looking at a visual display of four objects (a phonological competitor of the target word, e.g., 'matar', peas; a semantic competitor, e.g., 'kachuwa', turtle, and two unrelated distractors). In Experiment 2 the semantic competitor was replaced with another unrelated distractor. Both groups of participants shifted their eye gaze to the semantic competitors (Experiment 1). In both experiments high literates shifted their eye gaze towards phonological competitors and moved their eyes away as soon as the acoustic information mismatched. Low literates in contrast only used phonological information when semantic matches between spoken word and visual referent were impossible (Experiment 2) but in contrast to high literates these phonologically-mediated shifts in eye gaze were not closely time-locked to the speech input.
    Experiment 3 investigated language-mediated anticipatory eye gaze. Low and high literates (2 and 12 years of schooling) listened to simple spoken sentences containing a target word (e.g., "door") while looking at a visual display of four objects (the door, and three distractors). The spoken sentences contained adjectives followed by the (semantically neutral) particle wala/wali and a noun (e.g., 'Abhi aap ek uncha wala darwaja dekhnge', Right now you are going to see a high door). Adjective (e.g.,  uncha/unchi, high) and  particle (wala/wali) are gender-marked in Hindi and thus participants could use syntactic information to predict the target. To maximize the likelihood to observe anticipation effects, we chose adjectives which were also semantically and associatively related to the target object. High literates started to shift their eye gaze to the target object well before target word onset. Low literates' fixations on the targets only started to differ from looks on the unrelated distractors once the spoken target word acoustically unfolded (more than a second later than the high literates). In Experiment 4, we compared anticipatory eye gaze of low literates with up to 5 years of schooling with high literates. Importantly, both groups were matched on non-verbal IQ (Raven's matrices). A word reading test established low and high levels of reading proficiency. We used the same spoken sentences and target pictures as in Experiment 3 but we replaced the three distractors so that participants could only use semantic/associative information from the adjective for prediction (all words/objects were of the same gender). As in Experiment 3, high literates (but not low literates) initiated eye movements to the target objects well before target onset.
    Overall, these findings indicate that low literates achieve word-object mapping primarily at the semantic level but appear to be unable to use this semantic information for online anticipation of up-coming words. I conclude by discussing why formal literacy and reading aptitude may influence phonological and semantic processing and language-mediated prediction, in particular the possibility that literacy may enhance individuals' abilities to generate lexical predictions, abilities that help literates to exploit contextually-relevant predictive information in other situations such as when anticipating which object an interlocutor will refer to next in one's visual environment.