BRAINSYNC

Brain oscillations and synchronization during language processing and action instruction

Supervisor(s): † Gert Rickheit
Project Member(s): Sabine Weiss

Term: 2008-03 till 2012-10
Research Area: C D

Abstract

The main goal of this project is the investigation of language processing by means of behavioral data analysis (reaction time, rating) and the recording of electrical brain activity (EEG). Behavioral and neuronal data are obtained while participants process concrete and abstract sentences dealing with motor actions and various sensory modalities. According to the embodied cognition hypotheses, the modalities that have to be integrated during sentence comprehension should influence the participants processing time, error rate, learning performance and neuronal synchronization pattern.

Research Questions and Methods

The main aim of this project is the investigation of common characteristics and differences of concrete and abstract language on the basis of behavioral experiments (reaction time, rating) and EEG recordings (ERP, EEG coherence).

Understanding object names or simple action instructions (concrete language) is often successfully implemented in technical systems, however, the parsing of complex abstract language is still a major problem. Especially problematic are cases in which the abstract meaning constitution is based on concrete lexical entities (e.g., verbs in „Sie ergreift die Gartenschere” versus "Sie ergreift die Gelegenheit”). From our point of view the combination of the „Embodied-Language-Comprehension-Theory (ELC) on the cognitive linguistic side and the Binding-by-Synchrony-Theory (BBS) on the neuronal side may allow a new view on the processing of abstract language.

According to ELC-Theory neuronal substrates underlying senso-motoric processes are not only engaged during perception and action but also during language comprehension. More specifically, it has been proposed that not only the processing of concrete but also abstract language partly base on the perception of physical entities and action but also on the mental simulation of affective states and contextual situations. As a consequence, the processing of concrete and abstract language may share common neuronal activity patterns related to perceptual and motoric processing but also different ones. An unsolved issue is the way the brain integrates the cognitive and linguistic processes ocurring more or less simultaneously during language comprehension. One hypothesis raised in the neurosciences that provides a possible solution to this “binding-problem” is the Binding-by-Synchrony-Theory (BBS).

EEG data were recorded whilst participants processed various concrete and abstract sentences and were analyzed in the light of the BBS-Theory. A central issue was the relation of linguistic processing to increased and decreased neuronal cooperation of brain oscillations in various frequency bands. Spectral analysis of the EEG signals was performed with both classical Fourier transform and more sophisticated adaptive, bivariate auto-regressive moving average models (ARMA) in order to evaluate the dynamics of functional relationships between signals associated with language-specific or language-relevant processes. In particular, the importance of EEG coherence analysis dealing with the functional temporal and frequency-specific coupling of brain activity was emphasized for the investigation of the neurobiological basis of language comprehension.

« close

Outcomes

ALT TEXT Experiments investigated the processing of concrete or abstract sentences either dealing with one or two sensory modalities (seeing, feeling, hearing, etc.). Behavioral experiments revealed a longer reaction time for visually presented sentences dealing with visual modalities during semantic judgement. Similar results were obtained for auditorily presented sentences. Furthermore, sentences dealing with only one modality were processed more quickly than sentences dealing with two modalities. These results hint at increased processing costs due to modality switching during sentence processing for both concrete and abstract language. EEG coherence data demonstrated that concrete and abstract language share neural networks in the theta- but not in the beta1- frequency band and that the degree of concreteness of a sentence correlates with the height of EEG coherence during its processing.

The present results give evidence for a model on the relation between brain oscillations, EEG frequencies, neuronal synchronization and mental simulation during language processing. Knowledge about neurocognitive processes during verbal comprehension provides a module for building theories and consequently the technical conversion in human-computer interaction.

Publications

Suggested citation:

Vasilj, M., M.A., H. Koesling, L. Sichelschmidt, & G. Rickheit. 2010. “Remember the News? What we focus on is what we forget”, in: 5th Scandinavian Workshop on Applied Eye Tracking SWAET 2010. Lund, Sweden, 33.

Suggested citation:

Rickheit, G., S. Weiss, & H.-J. Eikmeyer. 2010. Kognitive Linguistik: Theorien, Modelle, Methoden. UTB : Sprachwissenschaft, Kognitionswissenschaft ; 3408 Tübingen: Francke.

Suggested citation:

Rickheit, G. 2010. “Pragmatik und Sprachstörungen”. Sprache, Stimme, Gehör, 34, 51 - 114.

Suggested citation:

Müller, H.M., & S. Weiss. 2010. “Electrophysiological and Haemodynamic Techniques for the Investigation of Language”. Sprache-Stimme-Gehör, 34(2), 106 - 111.

Suggested citation:

Weiss, S., H.M. Müller, & G. Rickheit. 2009. “Performance of sentence processing depends on multimodal mental simulation and related neuronal coupling”, in: Proceedings 1st Neurobiology of Language Conference. Chicago Illinois, USA, 21 - 21.

Suggested citation:

Weiss, S., & H.M. Müller. 2009. “EEG frequencies as indicators of distinct cognitive processes”, in: Proceedings of the 8. International Workshop of Neural Coding. Tainan, Taiwan, 122 - 124.

Suggested citation:

Weiss, S. 2009. Gehirnoszillationen und neuronale Kommunikation während der Verarbeitung von Sprache. Universität Bielefeld: Kum. Habilitationsschrift, Fakultät für Linguistik und Literaturwissenschaften.

Suggested citation:

Swadzba, A., C. Vorwerg, S. Wachsmuth, & G. Rickheit. 2009. “A Computational Model for the Alignment of Hierarchical Scene Representations in Human-Robot Interaction”, in: International Joint Conference on Artificial Intelligence. Pasadena, CA, USA: AAAI Press, 1857 - 1863.

Suggested citation:

Müller, H.M., & S. Weiss. 2009. “Neurobiologie der Sprache: Experimentelle Neurolinguistik”, in: H.M. Müller (Ed.), Arbeitsbuch Linguistik. Paderborn: Schöningh UTB, 406 - 422.

Suggested citation:

Müller, H.M., & S. Weiss. 2009. “Meaning constitution of proper names and figurative expressions”, in: Proceedings 1st Neurobiology of Language Conference. Chicago, Illinois, USA, 32 - 33.

Suggested citation:

Vorwerg, C., H. Strohner, & G. Rickheit. 2008. “The concept of communicative competence”, in: H. Strohner, & G. Rickheit (Eds.), Communication competence. Handbook of applied linguistics, 1. New York: Mouton de Gruyter, 15 - 62.

Suggested citation:

Rickheit, G. 2008. “Inferenzen bei der Sprachverarbeitung”, in: Neue Beiträge zur Germanistik. Japanischen Gesellschaft für Germanistik, 7 - 28.

Suggested citation:

Hemmelmann, C., A. Ziegler, V. Guiard, S. Weiss, M. Walther, & R. Vollandt. 2008. “Multiple test procedures using an upper bound of the number of true hypotheses and their use for evaluating high-dimensional EEG data”. Journal of Neuroscience Methods, 170, 158 - 164.