How Bartending Robots Can Respond Faster

International research team investigates pub interactions
A good bartender recognises at the first words what the customer asks for. So she can respond promptly. Robots need a little longer. Until now, they had to listen and analyse the whole utterance. Researchers from the Cluster of Excellence CITEC are working to enable robots to respond based on the beginning of the sentence – just as their human counterparts. Together with colleagues from the University of the West of England and Tufts University, USA, they are now presenting their findings in the research journal PLoS ONE.
In the study, test subjects had to operate the robot bartender, seeing only an overview of the data collected by the robot at the counter of the bar. The researchers have already dealt with the question of how to communicate at the bar. For their current study, they collaborated with scientists in England and the United States.
"When people talk to each other, they are constantly making assumptions about how their interlocutor will finish their sentences," says Sebastian Loth. He is a linguist at the research group Social Cognitive Systems (led by Stefan Kopp) that is part of the Technical Faculty and the Cluster of Excellence CITEC. For several years, Loth has investigated which parts of human communication a robot has to understand in order to serve customers. "If we do not have to wait for the end of a sentence but rather predict the user's intention quickly, we can respond faster and address the wishes of them in a socially appropriate way," says Loth.
"Service robots work with automatic speech recognition. Such software usually analyses speakers' utterances once they are complete. With incremental speech recognition, the sentences are analysed step by step as soon as the first word is spoken. However, this increases the likelihood that words are misunderstood."

The research team wants to give service robots the ability to understand customers as quickly as possible. For this, the team has asked test participants to play the role of a bartending robot. Sixteen participants had the task of controlling the robot remotely in each of 12 passes. They were in a different room than the robot and its customers. The people sat at a computer screen and got an overview of the data that the robot collected at the counter. For example, they could detect when a customer was approaching. If the customer said something, it was shown in real time as the machine indicat-ed the words.
Service robots as bartenders: researchers are investigating what a robot has to understand of human communication in order to serve customers at the bar. Based on this data, the participants selected one of about 20 possible robot actions with a mouse click. For example, they could instruct it to list the current beverage selection, ask a question, or serve a glass of water. A particular limitation was that the robot could not take back drinks once served. "We asked: When are the participants ready to tell the robot what to do," says Loth. "Do they wait for the whole sentence? Are they ready with the first words?"

The result: "The participants decide how high the error cost are. When data are uncertain, they weigh up how big the problems will be if they get lost," says Sebastian Loth.
If a customer at the bar begins her sentence with "What ...", chances are that she is about to ask what drinks are on offer. Most participants made the robot respond without knowing the rest of the sentence. "In this case, it is not a problem if the robot responds without having heard the whole sentence. If the answer was wrong, that would irritate the customer in the worst case," says Loth.
The situation is different if a specific order is placed. If sentences started like "I'd like to ..." or "I want to ..." the participants mostly waited until they had the full sentence in front of them. "In that case, a wrong response would be difficult to fix. Because serving the wrong drink could not be taken back." In cases where the data were too uncertain or muddled, the participants reassured themselves by asking questions.
CITEC researcher Dr. Sebastian Loth works to equip robot bartenders with human capabilities. "In real life, this research helps service robots or avatars on computer screens communicate more naturally by not letting their human interlocutors wait for too long and preventing misunderstandings with specific types of questions," says Loth.
The study was based on the research of the EU project "James". In this project, the Psycholinguistics research group at the University of Bielefeld lead by Prof Dr Jan de Ruiter collaborated with European partners such as the University of Edinburgh in the United Kingdom. The researchers found that barkeepers pay particular attention to body language in order to identify who wants to be served. Researchers from the Cluster of Excellence CITEC, the University of the West of England and Tufts University, USA cooperated for the current study.

Original Publication:
Sebastian Loth, Jettka K, Giuliani M, Stefan Kopp, Jan de Ruiter: Confidence in uncertainty: Error cost and commitment in early speech hypotheses. PLOS ONE., published on 1 August 2018.

More information is available online at:
“Why Bartenders Ignore Some Signals” (in German, press release from 13 November 2015):
“Conducting Research – at the Bar” (in German, press release from 9 September 2013):
James project:  

Dr. Sebastian Loth, Bielefeld University
Cluster of Excellence Cognitive Interaction Technology (CITEC) / Faculty of Technology
Telephone: +49 521 106-67505