ICSR 2011 tutorial on evaluation in human-robot interaction - integrating system and user centered perspectives

Overview

An increasing number of researchers in human-robot interaction (HRI) conduct user studies and experiments in order to evaluate their systems and to generate findings about the users and the interaction. These studies are interdisciplinary endeavors since they include both technical and social aspects of interaction. At RO-MAN 2009 we organized a first tutorial (see http://aiweb.techfak.uni-bielefeld.de/content/tutorial-roman09) on evaluation in HRI in which we showed how we cope with the plentitude of data acquired in user studies in our daily work. The tutorial at ICSR 2011 will be based on the same assumptions but is going to focus on a new aspect that evolved within the last two years: how can data from the user and the system be more strongly integrated with each other in the analysis? We will discuss how to acquire reliable data about the system itself and about the user. Connected to this issue, we will talk about the tools and approaches that we use for logging and synchronization of the data. Moreover, we will give a short introduction on qualitative and quantitative methods. Case studies will show how data were acquired and analyzed in user studies at Bielefeld University, with the aim to generate useful insights for system design and social science at the same time and in an integrated manner.

List of topics

• essentials for designing user studies and experiments
• introduction on how to acquire data in user studies
• preparation of data for analysis
• qualitative and quantitative analysis of data
• linking different types of data for analysis and visualization
• tools and devices for data acquisition and synchronization

Schedule

• 10:15-10:30 - opening talk slides (pdf)
• 10:30-11:00 - overview: from idea to study design talk slides (pdf)
• 11:00-11:30 - collecting system data for evaluation talk slides (pdf)
• 11:30-12:00 - behavior modeling: from metrics to system behavior talk slides (pdf)
• 12:00-14:00 - lunch
• 14:00-14:20 - online analysis of tutoring situations talk slides (pdf)
• 14:20-14:50 - qualitative analysis: linking system logs with video data talk slides (pdf)
• 14:50-15:30 - quantitative analysis: integrating system and subjective data talk slides (pdf)
• 15:30-15:45 - coffee break
• 16:00-16:30 - devices for acquiring additional user data talk slides (pdf)
• 16:30-17:00 - methods and tools for video synchronization talk slides (pdf)

Intended Audience

The intended audience is researchers in HRI who face the challenge of effectively analyzing the data acquired in user studies about the system AND about the user in an integrated manner. We welcome an interdisciplinary audience (computer science, robotics, social sciences, etc.) and appreciate a lot of discussion about the topic.
For further questions please contact us!

Speakers

• Sebastian Gieselmann, PhD-student in the Research Institute for Cognition and Robotics (CoR-Lab)
• Sascha S. Griffiths, research fellow at the Cluster of Excellence Cognitive Interaction Technology (CITEC) in the Applied Informatics Group, Bielefeld University.
• Katrin S. Lohan, PhD student at the Applied Informatics Group and the CoR-Lab
• Manja Lohse, postdoctoral researcher in the Applied Informatics Group and the CoR-Lab (Hybrid Society Group) at Bielefeld University
• Ingo Lütkebohle, postdoctoral researcher in the Applied Informatics Group at Bielefeld University, and affiliated with the CoR-Lab and CITEC
• Karola Pitsch, Dilthey Fellow (Volkswagenstiftung) and PostDoc researcher at Bielefeld University
• Nina Riether, doctoral researcher in the CoR-Lab at Bielefeld University and Applied Informatics Group at Bielefeld University
• Lars Schillingmann, Applied Informatics Group at Bielefeld University, associated to the CoR-Lab and currently working in the EU-Project iTalk
• Frederic Siepmann, Central Lab Facilities CITEC, founding member of team ToBI

Links

• Decision table for statistical tests with HOWTO for SPSS, Stata, SAS
• Overview table for statistical tests with further links, literature and Stata commands
• Vicon - Motion Capture Systems from Vicon
• brain products - Solutions for neurophysiological research
• BonSAI framework
• ELAN - tool for the creation of annotations on video and audio resources
• whichtest - online guide to selecting a statistical test