RSS Feed
DIACOSA |
Adaptive Dialogue Coordination for Sociable Agents
- Supervisor(s): Stefan Kopp
- Project Member(s): Hendrik Buschmeier
Research Area: B C
|
|
|
Abstract
DiaCoSA explores how speakers coordinate their interaction in dialogue by providing communicative feedback and by adapting their utterances to each other's. Building on empirical analyses of human dialogues, a new type of artificial ‘attentive communicator’ is developed in the form of a virtual conversational agent that actively tries to pursue understanding by being sensitive to the user's feedback, possibly even eliciting it, and by being able to adapt its lexico-grammatical choices to the interlocutor's.
Research Questions and Methods
A major prerequisite for trouble-free and efficient interaction in dialogue is the ability of interlocutors to coordinate their communicative actions.
read more »
Outcomes
The main expected outcome of the DiaCoSA project is a new kind of artificial ‘attentive communicator’ that keeps track of and actively strives to ensure its user's understanding. This includes abilities to recognize feedback even while uttering itself, to actively elicit feedback when necessary, and to respond to it by adapting its subsequent communicative actions. As a prototype, a virtual conversational agent with these abilities in a calendar planning domain will be developed and implemented. Additional specific outcomes of the project are (1) a corpus of empirical data on multimodal feedback use in human-human dialogue, (2) manuals and coding guidelines for the annotation and analysis of feedback behavior, (3) computational techniques and models for the recognition and interpretation of vocal and nonverbal user feedback, priming-based sentence planning and incremental adaptation.
Publications
Unveiling the Information State with a Bayesian Model of the Listener
Buschmeier H, Kopp S (2011)
In: SemDial 2011: Proceedings of the 15th Workshop on the Semantics and Pragmatics of Dialogue. Los Angeles, CA: 178 - 179.
In: SemDial 2011: Proceedings of the 15th Workshop on the Semantics and Pragmatics of Dialogue. Los Angeles, CA: 178 - 179.
Download:
Conference Proceeding/Paper
|
Published
|
English
Authors:
Buschmeier, Hendrik
;
Kopp, Stefan
Department:
Center of Excellence - Cognitive Interaction Technology CITEC
Technische Fakultät
Sociable Agents
Technische Fakultät
Sociable Agents
Abstract:
Attentive speaker agents – artificial conversational agents that can attend to and adapt to listener feedback – need to attribute a mental ‘listener state’ to the user and keep track of the grounding status of their own utterances. We propose a joint model of listener state and information state, represented as a dynamic Bayesian network, that can capture the influences between dialogue context, user feedback, the mental listener state and the information state, providing an estimation of grounding.
Attentive speaker agents – artificial conversational agents that can attend to and adapt to listener feedback – need to attribute a mental ‘listener state’ to the user and keep track of the grounding status of their own utterances. We propose a joint model of listener state and information state, represented as a dynamic Bayesian network, that can capture the influences between dialogue context, user feedback, the mental listener state and the information state, providing an estimation of grounding.
Keywords:
Grounding ;
Bayesian Networks ;
Information State ;
Dialogue ;
Communicative Feedback
PUB-ID:
2325052 | Link: http://pub.uni-bielefeld.de/publication/2325052
Towards Conversational Agents That Attend to and Adapt to Communicative User Feedback
Buschmeier H, Kopp S (2011)
In: Proceedings of the 11th International Conference on Intelligent Virtual Agents. LNCS. Vilhjàlmsson H, Kopp S, Marsella S, Thórisson K (Eds.); 6895. Berlin, Heidelberg: Springer-Verlag: 169 - 182.
In: Proceedings of the 11th International Conference on Intelligent Virtual Agents. LNCS. Vilhjàlmsson H, Kopp S, Marsella S, Thórisson K (Eds.); 6895. Berlin, Heidelberg: Springer-Verlag: 169 - 182.
Download:
Conference Proceeding/Paper
|
Published
|
English
Authors:
Buschmeier, Hendrik
;
Kopp, Stefan
Editors:
Vilhjàlmsson, Hannes
;
Kopp, Stefan
;
Marsella, Stacy
;
Thórisson, Kristinn
Department:
Sociable Agents
Center of Excellence - Cognitive Interaction Technology CITEC
Center of Excellence - Cognitive Interaction Technology CITEC
Abstract:
Successful dialogue is based on collaborative efforts of the interactants to ensure mutual understanding. This paper presents work towards making conversational agents ‘attentive speakers’ that continuously attend to the communicative feedback given by their interlocutors and adapt their ongoing and subsequent communicative behaviour to their needs. A comprehensive conceptual and architectural model for this is proposed and first steps of its realisation are described. Results from a prototype implementation are presented.
Successful dialogue is based on collaborative efforts of the interactants to ensure mutual understanding. This paper presents work towards making conversational agents ‘attentive speakers’ that continuously attend to the communicative feedback given by their interlocutors and adapt their ongoing and subsequent communicative behaviour to their needs. A comprehensive conceptual and architectural model for this is proposed and first steps of its realisation are described. Results from a prototype implementation are presented.
Keywords:
communicative feedback ;
feedback interpretation ;
feedback elicitation ;
attributed listener state ;
adaptation ;
attentive speaker agents
PUB-ID:
2277951 | Link: http://pub.uni-bielefeld.de/publication/2277951
'Are you sure you're paying attention?' – 'Uh-huh'. Communicating understanding as a marker of attentiveness
Buschmeier H, Malisz Z, Wlodarczak M, Kopp S, Wagner P (2011)
In: Proceedings of INTERSPEECH 2011. Florence, Italy: International Speech Communication Association: 2057 - 2060.
In: Proceedings of INTERSPEECH 2011. Florence, Italy: International Speech Communication Association: 2057 - 2060.
Conference Proceeding/Paper
|
Published
|
English
Authors:
Buschmeier, Hendrik
;
Malisz, Zofia
;
Wlodarczak, Marcin
;
Kopp, Stefan
;
Wagner, Petra
Department:
AG Wissensbasierte Systeme
Fakultät für Linguistik und Literaturwissenschaft
Center of Excellence - Cognitive Interaction Technology CITEC
Technische Fakultät
Sociable Agents
Arbeitsgruppe Phonetik und Phonologie
Fakultät für Linguistik und Literaturwissenschaft
Center of Excellence - Cognitive Interaction Technology CITEC
Technische Fakultät
Sociable Agents
Arbeitsgruppe Phonetik und Phonologie
Abstract:
We report on the first results of an experiment designed to investigate properties of communicative feedback produced by non-attentive listeners in dialogue. Listeners were found to produce less feedback when distracted by an ancillary task. A decreased number of feedback expressions communicating understanding was a particularly reliable indicator of distractedness. We argue this finding could be used to facilitate recognition of attentional states in dialogue system users. Index Terms: communicative feedback; dialogue; distraction; engagement; attention; dual task
We report on the first results of an experiment designed to investigate properties of communicative feedback produced by non-attentive listeners in dialogue. Listeners were found to produce less feedback when distracted by an ancillary task. A decreased number of feedback expressions communicating understanding was a particularly reliable indicator of distractedness. We argue this finding could be used to facilitate recognition of attentional states in dialogue system users. Index Terms: communicative feedback; dialogue; distraction; engagement; attention; dual task
PUB-ID:
2143919 | Link: http://pub.uni-bielefeld.de/publication/2143919
Adaptive Expressiveness – Virtual Conversational Agents That Can Align to Their Interaction Partner
Buschmeier H, Bergmann K, Kopp S (2010)
In: Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems. van der Hoek W, Kaminka GA, Luck M, Sen S (Eds.); Toronto, Canada: International Foundation for Autonomous Agents and Multiagent Systems: 91 - 98.
In: Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems. van der Hoek W, Kaminka GA, Luck M, Sen S (Eds.); Toronto, Canada: International Foundation for Autonomous Agents and Multiagent Systems: 91 - 98.
Download:
Conference Proceeding/Paper
|
Published
|
English
Authors:
Buschmeier, Hendrik
;
Bergmann, Kirsten
;
Kopp, Stefan
Editors:
van der Hoek, Wiebe
;
Kaminka, Gal A.
;
Luck, Michael
;
Sen, Sandip
Department:
Sociable Agents
Center of Excellence - Cognitive Interaction Technology CITEC
B1 - Speech-gesture alignment
Center of Excellence - Cognitive Interaction Technology CITEC
B1 - Speech-gesture alignment
Abstract:
Speakers in dialogue tend to adapt to each other by starting to use similar lexical items, syntactic structures, or gestures. This behaviour, called alignment, may serve important cognitive, communicative and social functions (such as speech facilitation, grounding and rapport). Our aim is to enable and study the effects of these subtle aspects of communication in virtual conversational agents. Building upon a model for autonomous speech and gesture generation, we describe an approach to make the agent's multimodal behaviour adaptive in an interactive manner. This includes (1) an activation-based microplanner that makes linguistic choices based on lexical and syntactic priming, and (2) an empirically grounded gesture generation such that linguistic priming parallels concordant gestural adaptation. First results show that the agent aligns to its interaction partners by picking up their syntactic structures and lexical items in its subsequent utterances. These changes in the agent's verbal behaviour also have a direct influence on gestural expressions.
Speakers in dialogue tend to adapt to each other by starting to use similar lexical items, syntactic structures, or gestures. This behaviour, called alignment, may serve important cognitive, communicative and social functions (such as speech facilitation, grounding and rapport). Our aim is to enable and study the effects of these subtle aspects of communication in virtual conversational agents. Building upon a model for autonomous speech and gesture generation, we describe an approach to make the agent's multimodal behaviour adaptive in an interactive manner. This includes (1) an activation-based microplanner that makes linguistic choices based on lexical and syntactic priming, and (2) an empirically grounded gesture generation such that linguistic priming parallels concordant gestural adaptation. First results show that the agent aligns to its interaction partners by picking up their syntactic structures and lexical items in its subsequent utterances. These changes in the agent's verbal behaviour also have a direct influence on gestural expressions.
Keywords:
verbal and non-verbal expressiveness ;
modelling natural language ;
multimodal interaction ;
interactive alignment ;
user-adaptated interaction ;
Technology and Engineering ;
Languages and Literatures
PUB-ID:
1901698 | Link: http://pub.uni-bielefeld.de/publication/1901698
Modelling and Evaluation of Lexical and Syntactic Alignment with a Priming-Based Microplanner
Buschmeier H, Bergmann K, Kopp S (2010)
In: Empiricial Methods in Natural Language Generation. Krahmer E, Theune M (Eds.); Lecture Notes in Computer Science; 5790. Berlin/Heidelberg, Germany: Springer: 85 - 104.
In: Empiricial Methods in Natural Language Generation. Krahmer E, Theune M (Eds.); Lecture Notes in Computer Science; 5790. Berlin/Heidelberg, Germany: Springer: 85 - 104.
Download:
Book Chapter
|
Published
|
English
Authors:
Buschmeier, Hendrik
;
Bergmann, Kirsten
;
Kopp, Stefan
Editors:
Krahmer, Emiel
;
Theune, Mariët
Department:
Sociable Agents
Center of Excellence - Cognitive Interaction Technology CITEC
B1 - Speech-gesture alignment
Center of Excellence - Cognitive Interaction Technology CITEC
B1 - Speech-gesture alignment
Abstract:
Alignment of interlocutors is a well known psycholinguistic phenomenon of great relevance for dialogue systems in general and natural language generation in particular. In this chapter, we present the alignment-capable microplanner SPUD prime. Using a priming-based model of interactive alignment, it is flexible enough to model the alignment behaviour of human speakers to a high degree. We demonstrate that SPUD prime can account for lexical as well as syntactic alignment and present an evaluation on corpora of task-oriented dialogue that were collected in two experiments designed to investigate the alignment behaviour of humans in a controlled fashion. This will allow for further investigation of which parameters are important to model alignment and how the human?computer interaction changes when the computer aligns to its users.
Alignment of interlocutors is a well known psycholinguistic phenomenon of great relevance for dialogue systems in general and natural language generation in particular. In this chapter, we present the alignment-capable microplanner SPUD prime. Using a priming-based model of interactive alignment, it is flexible enough to model the alignment behaviour of human speakers to a high degree. We demonstrate that SPUD prime can account for lexical as well as syntactic alignment and present an evaluation on corpora of task-oriented dialogue that were collected in two experiments designed to investigate the alignment behaviour of humans in a controlled fashion. This will allow for further investigation of which parameters are important to model alignment and how the human?computer interaction changes when the computer aligns to its users.
Keywords:
natural language generation ;
microplanning ;
lexical and syntactic alignment ;
interactive alignment model ;
adaptation ;
Languages and Literatures ;
Technology and Engineering
PUB-ID:
1893922 | Link: http://pub.uni-bielefeld.de/publication/1893922
Middleware for Incremental Processing in Conversational Agents
Schlangen D, Baumann T, Buschmeier H, Kopp S, Skantze G, Yaghoubzadeh R (2010)
In: Proceedings of SIGDIAL 2010: the 11th Annual Meeting of the Special Interest Group in Discourse and Dialogue. Tokyo, Japan: Association for Computational Linguistics: 51 - 54.
In: Proceedings of SIGDIAL 2010: the 11th Annual Meeting of the Special Interest Group in Discourse and Dialogue. Tokyo, Japan: Association for Computational Linguistics: 51 - 54.
Download:
Conference Proceeding/Paper
|
Published
|
English
Authors:
Schlangen, David
;
Baumann, Timo
;
Buschmeier, Hendrik
;
Kopp, Stefan
;
Skantze, Gabriel
;
Yaghoubzadeh, Ramin
Department:
Sociable Agents
Center of Excellence - Cognitive Interaction Technology CITEC
Center of Excellence - Cognitive Interaction Technology CITEC
Abstract:
We describe work done at three sites on designing conversational agents capable of incremental processing. We focus on the ‘middleware’ layer in these systems, which takes care of passing around and maintaining incremental information between the modules of such agents. All implementations are based on the abstract model of incremental dialogue processing proposed by Schlangen and Skantze (2009), and the paper shows what different instantiations of the model can look like given specific requirements and application areas.
We describe work done at three sites on designing conversational agents capable of incremental processing. We focus on the ‘middleware’ layer in these systems, which takes care of passing around and maintaining incremental information between the modules of such agents. All implementations are based on the abstract model of incremental dialogue processing proposed by Schlangen and Skantze (2009), and the paper shows what different instantiations of the model can look like given specific requirements and application areas.
Keywords:
Technology and Engineering ;
Languages and Literatures
PUB-ID:
1903225 | Link: http://pub.uni-bielefeld.de/publication/1903225
An Alignment-capable Microplanner for Natural Language Generation
Buschmeier H, Bergmann K, Kopp S (2009)
In: Proceedings of the 12th European Workshop on Natural Language Generation. Athens, Greece: Association for Computational Linguistics: 82 - 89.
In: Proceedings of the 12th European Workshop on Natural Language Generation. Athens, Greece: Association for Computational Linguistics: 82 - 89.
Download:
Conference Proceeding/Paper
|
Published
|
English
Authors:
Buschmeier, Hendrik
;
Bergmann, Kirsten
;
Kopp, Stefan
Department:
B1 - Speech-gesture alignment
Center of Excellence - Cognitive Interaction Technology CITEC
Sociable Agents
Center of Excellence - Cognitive Interaction Technology CITEC
Sociable Agents
Abstract:
Alignment of interlocutors is a well known psycholinguistic phenomenon of great relevance for dialogue systems in general and natural language generation in particular. In this paper, we present the alignment-capable microplanner SPUD prime. Using a priming-based model of interactive alignment, it is flexible enough to model the alignment behaviour of human speakers to a high degree. This will allow for further investigation of which parameters are important to model alignment and how the human--computer interaction changes when the computer aligns to its users.
Alignment of interlocutors is a well known psycholinguistic phenomenon of great relevance for dialogue systems in general and natural language generation in particular. In this paper, we present the alignment-capable microplanner SPUD prime. Using a priming-based model of interactive alignment, it is flexible enough to model the alignment behaviour of human speakers to a high degree. This will allow for further investigation of which parameters are important to model alignment and how the human--computer interaction changes when the computer aligns to its users.
Keywords:
adaptation ;
interactive alignment model ;
natural language generation ;
microplanning ;
lexical and syntactic alignment ;
Technology and Engineering ;
Languages and Literatures
PUB-ID:
1857963 | Link: http://pub.uni-bielefeld.de/publication/1857963
Social resonance and embodied coordination in face-to-face conversation with artificial interlocutors
Kopp S (2010)
Speech Communication 52(6): 587 - 597.
Speech Communication 52(6): 587 - 597.
Download:
Journal Article
|
Published
|
English
Authors:
Kopp, Stefan
Department:
AG Wissensbasierte Systeme
Center of Excellence - Cognitive Interaction Technology CITEC
Sociable Agents
Technische Fakultät
B1 - Speech-gesture alignment
Center of Excellence - Cognitive Interaction Technology CITEC
Sociable Agents
Technische Fakultät
B1 - Speech-gesture alignment
Abstract:
Human natural face-to-face communication is characterized by inter-personal coordination. In this paper, phenomena are analyzed that yield coordination of behaviors, beliefs, and attitudes between interaction partners, which can be tied to a concept of establishing social resonance. It is discussed whether these mechanisms can and should be transferred to conversation with artificial interlocutors like ECAs or humanoid robots. It is argued that one major step in this direction is embodied coordination, mutual adaptations that are mediated by flexible modules for the top-down production and bottom-up perception of expressive conversational behavior that ground in and, crucially, coalesce in the same sensorimotor structures. Work on modeling this for ECAs with a focus on coverbal gestures is presented. (C) 2010 Elsevier B.V. All rights reserved.
Human natural face-to-face communication is characterized by inter-personal coordination. In this paper, phenomena are analyzed that yield coordination of behaviors, beliefs, and attitudes between interaction partners, which can be tied to a concept of establishing social resonance. It is discussed whether these mechanisms can and should be transferred to conversation with artificial interlocutors like ECAs or humanoid robots. It is argued that one major step in this direction is embodied coordination, mutual adaptations that are mediated by flexible modules for the top-down production and bottom-up perception of expressive conversational behavior that ground in and, crucially, coalesce in the same sensorimotor structures. Work on modeling this for ECAs with a focus on coverbal gestures is presented. (C) 2010 Elsevier B.V. All rights reserved.
Keywords:
Gesture ;
Coordination ;
Embodied conversational agents ;
Social Resonance
ISSN:
0167-6393
PUB-ID:
1795482 | Link: http://pub.uni-bielefeld.de/publication/1795482
Requirements and Building Blocks for Sociable Embodied Agents
Kopp S, Bergmann K, Buschmeier H, Sadeghipour A (2009)
In: KI 2009: Advances in Artificial Intelligence. Mertsching B, Hund M, Aziz Z (Eds.); Berlin, Heidelberg: Springer: 508 - 515.
In: KI 2009: Advances in Artificial Intelligence. Mertsching B, Hund M, Aziz Z (Eds.); Berlin, Heidelberg: Springer: 508 - 515.
Download:
Conference Proceeding/Paper
|
Published
|
English
Authors:
Kopp, Stefan
;
Bergmann, Kirsten
;
Buschmeier, Hendrik
;
Sadeghipour, Amir
Editors:
Mertsching, B.
;
Hund, M.
;
Aziz, Z.
Department:
Center of Excellence - Cognitive Interaction Technology CITEC
Sociable Agents
B1 - Speech-gesture alignment
Technische Fakultät
Sociable Agents
B1 - Speech-gesture alignment
Technische Fakultät
Abstract:
To be sociable, embodied interactive agents like virtual characters or humanoid robots need to be able to engage in mutual coordination of behaviors, beliefs, and relationships with their human interlocutors. We argue that this requires them to be capable of flexible multimodal expressiveness, incremental perception of other’s behaviors, and the integration and interaction of these models in unified sensorimotor structures. We present work on probabilistic models for these three requirements with a focus on gestural behavior.
To be sociable, embodied interactive agents like virtual characters or humanoid robots need to be able to engage in mutual coordination of behaviors, beliefs, and relationships with their human interlocutors. We argue that this requires them to be capable of flexible multimodal expressiveness, incremental perception of other’s behaviors, and the integration and interaction of these models in unified sensorimotor structures. We present work on probabilistic models for these three requirements with a focus on gestural behavior.
PUB-ID:
1903230 | Link: http://pub.uni-bielefeld.de/publication/1903230
Modeling Embodied Feedback in a Virtual Human
Kopp S, Allwood J, Ahlsen E, Grammer K, Stocksmeier T (2008)
In: Modeling Communication With Robots And Virtual Humans. LNAI 4930. Wachsmuth I, Knoblich G (Eds.); Berlin, Heidelberg: Springer: 18 - 37.
In: Modeling Communication With Robots And Virtual Humans. LNAI 4930. Wachsmuth I, Knoblich G (Eds.); Berlin, Heidelberg: Springer: 18 - 37.
Book Chapter
|
Published
|
English
Authors:
Kopp, Stefan
;
Allwood, J.
;
Ahlsen, E.
;
Grammer, K.
;
Stocksmeier, T.
Editors:
Wachsmuth, I.
;
Knoblich, G.
Department:
Research Institute for Cognition and Robotics
Technische Fakultät
Center of Excellence - Cognitive Interaction Technology CITEC
Sociable Agents
AG Wissensbasierte Systeme
Technische Fakultät
Center of Excellence - Cognitive Interaction Technology CITEC
Sociable Agents
AG Wissensbasierte Systeme
Abstract:
In natural communication, both speakers and listeners are active most of the time. While a speaker contributes new information, a listener gives feedback by producing unobtrusive (usually short) vocal or non-vocal bodily expressions to indicate whether he/she is able and willing to communicate, perceive, and understand the information, and what emotions and attitudes are triggered by this information. The simulation of feedback behavior for artificial conversational agents poses big challenges such as the concurrent and integrated perception and production of multi-modal and multi-functional expressions. We present an approach on modeling feedback for and with virtual humans, based on an approach to study ``embodied feedback'' as a special case of a more general theoretical account of embodied communication. A realization of this approach with the virtual human Max is described and results are presented.
In natural communication, both speakers and listeners are active most of the time. While a speaker contributes new information, a listener gives feedback by producing unobtrusive (usually short) vocal or non-vocal bodily expressions to indicate whether he/she is able and willing to communicate, perceive, and understand the information, and what emotions and attitudes are triggered by this information. The simulation of feedback behavior for artificial conversational agents poses big challenges such as the concurrent and integrated perception and production of multi-modal and multi-functional expressions. We present an approach on modeling feedback for and with virtual humans, based on an approach to study ``embodied feedback'' as a special case of a more general theoretical account of embodied communication. A realization of this approach with the virtual human Max is described and results are presented.
PUB-ID:
1857761 | Link: http://pub.uni-bielefeld.de/publication/1857761
New CITEC Building
Interactive Intelligent Systems
Funded in the Excellence-Initiative
