RSS Feed
CLAINT |
Closed-loop Auditory Interactions
- Supervisor(s): Thomas Hermann, Bettina Bläsing, Elena Carbone
- Project Member(s): Tobias Grosshauser
|
|
|
Abstract
Body movements are complex and demand multivariate coordination. CLAINT evaluates the potential of interactive motion sonification to support the learning and execution of complex coordinated motion sequences. This includes the development of sensing and sonification technology and it enables several novel applications: (a) the support of skill learning, movements in dance and musical instrument playing are focused, (b) the support of tool use, such as a drilling machine, tool-integrated interactive sonification techniques are developed, and (c) to help preventing unhealthy movement patterns, e.g. in musical instrument playing.
Research Questions and Methods
Body movements as they occur in physical activity such as musical instrument playing, dance, sports, or using tools involve the fine coordinated control of many muscle groups. Often we have to learn new patterns, such as in physiotherapy or instrument learning, or we risk our health by performing them incorrectly (e.g. lifting heavy objects in the wrong way).
read more »
Outcomes
Regarding the challenge of an unobtrusive wearable sensing and actuator system, the ARTIS system has been developed within CLAINT which allows to connect, depending on the particular demands of the application, to connect different sensors. Low-cost sensors such as goniometers, force-sensitive resistors, distance sensors, and actuators such as vibrating coils, LEDs, and mini-loudspeakers have been integrated into that platform. Bluetooth is being used as wireless interface to a PC.
Regarding the sonification, various real-time sonification approaches have been implemented under the SuperCollider language. They can be configured just-in-time by the operator or user. Tests with individual users have shown that quite simple and elementary sonifications that merely reflect the most significant variable of interest, e.g. the bending of the knee, or the contact of the feet during jumps in dance, are most accepted and effective. Over the course of a tutoring unit, settings should - and can be - adjusted to adapt to the problem that teacher and pupil work at.
Concerning applications, the CLAINT project has developed selected demonstrator systems for three areas:
- for tool-integrated auditory feedback, a drilling machine has been extended by sensors to measure the relative orientation to the wall, and actuators to display a deviation from the ideal (or nominal) angle by sound or light. A test with subjects has shown that the feedback is effective to increase drilling accuracy, that is to reduce the average deviation from a nominal drilling angle.
- for musical instrument learning, sensors and actuators have been integrated into a violin (both body and bow) to support musicians for tasks such as the linear bow stroke. A long-term study with pupils (with and without feedback) suggests that the auditory-supported practice improves movement quality and accelerates learning
- for ballet dance, particularly for jumps, sensors (goniometer, accelerometer, pressure sensor) have been integrated into the lower limbs of dance pupils, enabling to measure floor contact, knee angle and overall acceleration. This enabled the teachers, for the first time, to attend to the fine correlations between knee action and contact which are invisible (or: too fast to be analysed) from a mere observation. In a series of interactions between dancers and their teachers, where the ARTIS system augmented the information with interactive sonification, insights on how to structure and use multimodal feedback in tutoring has been gained.
Publications
New Sensors and Pattern Recognition Techniques for String Instruments
Großhauser T, Großekathöfer U, Hermann T (2010)
In: Sydney, Australia. Beilharz K , Johnston A , Ferguson S , Chen AY-C (Eds.); Proceedings of the 2010 Conference on New Interfaces for Musical Expression (NIME 2010): International Conference on New Interfaces for Musical Expression: 271 - 276.
In: Sydney, Australia. Beilharz K , Johnston A , Ferguson S , Chen AY-C (Eds.); Proceedings of the 2010 Conference on New Interfaces for Musical Expression (NIME 2010): International Conference on New Interfaces for Musical Expression: 271 - 276.
Conference Proceeding/Paper
|
Published
|
English
Authors:
Großhauser, Tobias
;
Großekathöfer, Ulf
;
Hermann, Thomas
Editors:
Beilharz, Kirsty
;
Johnston, Andrew
;
Ferguson, Sam
;
Chen , Amy Yi-Chun
Department:
Ambient Intelligence
Sociable Agents
Center of Excellence - Cognitive Interaction Technology CITEC
Sociable Agents
Center of Excellence - Cognitive Interaction Technology CITEC
Abstract:
Pressure, motion, and gesture are important parameters in musical instrument playing. Pressure sensing allows to interpret complex hidden forces, which appear during playing a musical instrument. The combination of our new sensor setup with pattern recognition techniques like the lately developed ordered means models allows fast and precise recognition of highly skilled playing techniques. This includes left and right hand analysis as well as a combination of both. In this paper we show bow position recognition for string instruments by means of support vector regression machines on the right hand finger pressure, as well as bowing recognition and inaccurate playing detection with ordered means models. We also introduce a new left hand and chin pressure sensing method for coordination and position change analysis. Our methods in combination with our audio, video, and gesture recording software can be used for teaching and exercising. Especially studies of complex movements and finger force distribution changes can benefit from such an approach. Practical applications include the recognition of inaccuracy, cramping, or malposition, and, last but not least, the development of augmented instruments and new playing techniques.
Pressure, motion, and gesture are important parameters in musical instrument playing. Pressure sensing allows to interpret complex hidden forces, which appear during playing a musical instrument. The combination of our new sensor setup with pattern recognition techniques like the lately developed ordered means models allows fast and precise recognition of highly skilled playing techniques. This includes left and right hand analysis as well as a combination of both. In this paper we show bow position recognition for string instruments by means of support vector regression machines on the right hand finger pressure, as well as bowing recognition and inaccurate playing detection with ordered means models. We also introduce a new left hand and chin pressure sensing method for coordination and position change analysis. Our methods in combination with our audio, video, and gesture recording software can be used for teaching and exercising. Especially studies of complex movements and finger force distribution changes can benefit from such an approach. Practical applications include the recognition of inaccuracy, cramping, or malposition, and, last but not least, the development of augmented instruments and new playing techniques.
Keywords:
teaching ;
piano ;
violin ;
pattern recognition ;
feedback ;
real-time ;
multimodal
ISBN:
978-0-646-53482-4
PUB-ID:
2017200 | Link: http://pub.uni-bielefeld.de/publication/2017200
Wearable Setup for Gesture and Motion based Closed Loop Audio-Haptic Interaction
Großhauser T, Hermann T (2010)
In: Proceedings of the 16th International Community for Auditory Display. Brazil E (Ed.); Washington, USA: International Community for Auditory Display: 31 - 38.
In: Proceedings of the 16th International Community for Auditory Display. Brazil E (Ed.); Washington, USA: International Community for Auditory Display: 31 - 38.
Conference Proceeding/Paper
|
Published
|
English
Authors:
Großhauser, Tobias
;
Hermann, Thomas
Editors:
Brazil, Eoin
Department:
Center of Excellence - Cognitive Interaction Technology CITEC
Ambient Intelligence
Ambient Intelligence
Abstract:
The wearable sensor and feedback system presented in this paper is a type of audio-haptic display which contains on board sensors, embedded sound synthesis, external sensors, and on the feedback side a loudspeaker and several vibrating motors. The so-called ``embedded sonification'' in this case here is an onboard IC with implemented sound synthesis. This is adjusted directly by the user and/or controlled in real-time by the sensors, which are on the board or fixed on the human body and connected to the board via cable or radio frequency transmission. Direct audio out and tactile feedback closes the loop between the wearable board and the user. In many situations, this setup can serve as a complement to visual output, e.g. exploring data in 3D-space or learning motion and gestures in dance, sports or outdoor and every-day activities. A new metaphor for interactive acoustical augmentation is introduced, the so called ``audio loupe''. In this case it means the sonification of minimal movements or state changes, which can sometimes hardly be perceived visually or corporal. These are for example small jitters or deviations of predefined ideal gestures or movements. Our system is easy to use, it even allows operation without an external computer. We demonstrate and outline the benefits of our wearable interactive setup in highly skilled motion learning scenarios in dance and sports.
The wearable sensor and feedback system presented in this paper is a type of audio-haptic display which contains on board sensors, embedded sound synthesis, external sensors, and on the feedback side a loudspeaker and several vibrating motors. The so-called ``embedded sonification'' in this case here is an onboard IC with implemented sound synthesis. This is adjusted directly by the user and/or controlled in real-time by the sensors, which are on the board or fixed on the human body and connected to the board via cable or radio frequency transmission. Direct audio out and tactile feedback closes the loop between the wearable board and the user. In many situations, this setup can serve as a complement to visual output, e.g. exploring data in 3D-space or learning motion and gestures in dance, sports or outdoor and every-day activities. A new metaphor for interactive acoustical augmentation is introduced, the so called ``audio loupe''. In this case it means the sonification of minimal movements or state changes, which can sometimes hardly be perceived visually or corporal. These are for example small jitters or deviations of predefined ideal gestures or movements. Our system is easy to use, it even allows operation without an external computer. We demonstrate and outline the benefits of our wearable interactive setup in highly skilled motion learning scenarios in dance and sports.
Keywords:
Motion ;
Gesture ;
Closed-loop ;
real-time ;
interaction ;
sonification
ISBN:
0-9670904-3-1
PUB-ID:
2017222 | Link: http://pub.uni-bielefeld.de/publication/2017222
Multimodal Closed-loop Human Machine Interaction
Großhauser T, Hermann T (2010)
In: Proceedings of the 3rd International workshop on Interactive Sonification. Bresin R (Ed.); KTH, Stockholm, Sweden: KTH School of Computer Science and Communication (CSC): 59 - 63.
In: Proceedings of the 3rd International workshop on Interactive Sonification. Bresin R (Ed.); KTH, Stockholm, Sweden: KTH School of Computer Science and Communication (CSC): 59 - 63.
Conference Proceeding/Paper
|
Published
|
English
Authors:
Großhauser, Tobias
;
Hermann, Thomas
Editors:
Bresin, Roberto
Department:
Ambient Intelligence
Center of Excellence - Cognitive Interaction Technology CITEC
Center of Excellence - Cognitive Interaction Technology CITEC
Abstract:
The paper presents a multi-modal approach to tightly close the interaction loop between a human user and any tool in operation. Every activity of a human being generates multi-modal feedback, more or less related to the eyes (visual), the skin (sensory), the nose (olfactory) and the ears (auditive). Here we show the useful augmentation or complete creation of a nonexistent or less available feedback. As an example the performance of drilling tasks, line drawing tasks, or the complex task of bowing a violin can be considered.
The paper presents a multi-modal approach to tightly close the interaction loop between a human user and any tool in operation. Every activity of a human being generates multi-modal feedback, more or less related to the eyes (visual), the skin (sensory), the nose (olfactory) and the ears (auditive). Here we show the useful augmentation or complete creation of a nonexistent or less available feedback. As an example the performance of drilling tasks, line drawing tasks, or the complex task of bowing a violin can be considered.
Keywords:
Sensor ;
Sound ;
interaction ;
Sonification ;
closed-loop ;
Wearable
PUB-ID:
2017209 | Link: http://pub.uni-bielefeld.de/publication/2017209
Wearable Multi-Modal Sensor System for Embedded Audio-Haptic Feedback
Großhauser T, Hermann T (2010)
In: Proceedings of the 3rd International workshop on Interactive Sonification. Bresin R, Hermann T, Hunt A (Eds.); KTH, Stockholm, Sweden: KTH School of Computer Science and Communication (CSC): 75 - 79.
In: Proceedings of the 3rd International workshop on Interactive Sonification. Bresin R, Hermann T, Hunt A (Eds.); KTH, Stockholm, Sweden: KTH School of Computer Science and Communication (CSC): 75 - 79.
Conference Proceeding/Paper
|
Published
|
English
Authors:
Großhauser, Tobias
;
Hermann, Thomas
Editors:
Bresin, Roberto
;
Hermann, Thomas
;
Hunt, Andy
Department:
Center of Excellence - Cognitive Interaction Technology CITEC
Ambient Intelligence
Ambient Intelligence
Abstract:
Wearable sensing technologies give the user the possibility of on-site and real-time measurements, analysis and feedback of movements and postures in everyday behaviour, learning and training situations. There are many established motion capturing technologies to support complex movements, but less mobile and wearable systems. One of the key disadvantages of the existing systems is their high complexity, for instance they demand high-speed cameras, multi-channel audio systems or the fixation to a special room or laboratory. As a low-cost and flexible solution, this paper presents an easily relocatable, flexible sensor-based system for motion capturing and multi-modal real-time feedback. Our system is easy to use, it even allows operation without an external computer. Here we introduce our wearable sensor-setup and outline its applications and benefits in typical everyday training situations in combination with multi-modal feedback and embedded systems including closed loop interactive sonification.
Wearable sensing technologies give the user the possibility of on-site and real-time measurements, analysis and feedback of movements and postures in everyday behaviour, learning and training situations. There are many established motion capturing technologies to support complex movements, but less mobile and wearable systems. One of the key disadvantages of the existing systems is their high complexity, for instance they demand high-speed cameras, multi-channel audio systems or the fixation to a special room or laboratory. As a low-cost and flexible solution, this paper presents an easily relocatable, flexible sensor-based system for motion capturing and multi-modal real-time feedback. Our system is easy to use, it even allows operation without an external computer. Here we introduce our wearable sensor-setup and outline its applications and benefits in typical everyday training situations in combination with multi-modal feedback and embedded systems including closed loop interactive sonification.
Keywords:
sonification ;
wearable ;
sensor ;
real-time
PUB-ID:
2017213 | Link: http://pub.uni-bielefeld.de/publication/2017213
Sensor Fusion and multi-modal Feedback for musical instrument learning and teaching
Großhauser T, Hermann T (2010)
In: Proceedings of the Second Vienna Talk on Music Acoustics. Goebl W (Ed.); Vienna: Institute of Musical Acoustics (Wiener Klangstil): 77 - 80.
In: Proceedings of the Second Vienna Talk on Music Acoustics. Goebl W (Ed.); Vienna: Institute of Musical Acoustics (Wiener Klangstil): 77 - 80.
Download:
Conference Proceeding/Paper
|
Published
|
English
Authors:
Großhauser, Tobias
;
Hermann, Thomas
Editors:
Goebl, Werner
Department:
Center of Excellence - Cognitive Interaction Technology CITEC
Ambient Intelligence
Ambient Intelligence
Abstract:
Pressure, motion, gesture and the coordination of all these parameters is an important task in musical instrument playing, exercising and teaching. New sensing and feedback technologies provide many possibilities to support the teacher and the student in the complex learning scenarios. In this paper we show with some practical examples the application of our current sensing technologies to teaching and everyday self exercising of violin. This starts from a left hand and chin pressure sensing method for coordination and position change analysis, a setup for force measurement and the weight allocation between the chin and left hand while playing, and a left and right hand thumb position and bending measurement method. Different individually adapted feedback methods allow assisted teaching in teacher-to-student tuition and also efficient training in single exercising situations at home without a teacher. But also in combination with audio, video, and gesture recording, our setup is useful for accurate offline music sheet alignment and analysis. Especially studies of complex movements as well as finger position and force distribution changes can benefit from this approach. We discuss the practical applications regarding the recognition of inaccuracy, cramping, and malposition.
Pressure, motion, gesture and the coordination of all these parameters is an important task in musical instrument playing, exercising and teaching. New sensing and feedback technologies provide many possibilities to support the teacher and the student in the complex learning scenarios. In this paper we show with some practical examples the application of our current sensing technologies to teaching and everyday self exercising of violin. This starts from a left hand and chin pressure sensing method for coordination and position change analysis, a setup for force measurement and the weight allocation between the chin and left hand while playing, and a left and right hand thumb position and bending measurement method. Different individually adapted feedback methods allow assisted teaching in teacher-to-student tuition and also efficient training in single exercising situations at home without a teacher. But also in combination with audio, video, and gesture recording, our setup is useful for accurate offline music sheet alignment and analysis. Especially studies of complex movements as well as finger position and force distribution changes can benefit from this approach. We discuss the practical applications regarding the recognition of inaccuracy, cramping, and malposition.
Keywords:
Network ;
Adaptive ;
Embedded ;
Teaching ;
Violin ;
Sensor
ISBN:
978-3-900914-07-3
PUB-ID:
2289017 | Link: http://pub.uni-bielefeld.de/publication/2289017
Augmented Haptics - An Interactive Feedback System for Musicians
Großhauser T, Hermann T (2009)
In: Haptic and Audio Interaction Design 4th International Conference, Proceedings. Lecture Notes in Computer Science, 5763. Altinsoy ME, Merchel S (Eds.); Berlin, Heidelberg: Springer: 100 - 108.
In: Haptic and Audio Interaction Design 4th International Conference, Proceedings. Lecture Notes in Computer Science, 5763. Altinsoy ME, Merchel S (Eds.); Berlin, Heidelberg: Springer: 100 - 108.
Conference Proceeding/Paper
|
Published
|
English
Authors:
Großhauser, Tobias
;
Hermann, Thomas
Editors:
Altinsoy, M. Ercan
;
Merchel, Sebastian
Department:
Ambient Intelligence
Center of Excellence - Cognitive Interaction Technology CITEC
Center of Excellence - Cognitive Interaction Technology CITEC
Abstract:
This paper presents integrated tactiles (or vibrotactiles), a novel interface for movement and posture tuition that provides real-time feedback in a tactile form by means of interactive haptic feedback. Tactile feedback conveys information by using non-audio and non-visual form and is a promising means for movements in 3D-space. In this paper we demonstrate haptic augmentation for applications for musicians, since it (a) doesn't affect the visual sense, occupied by reading music and communication, (b) doesn't disturb in bang sensitive situations like concerts, (c) allows to relate feedback information in the same tactile medium as the output of the musical instrument, so that an important feedback channel for musical instrument playing is extended and trained supportive. Even more, instructions from the teacher and the computer can be transmitted directly and unobtrusively in this channel. This paper presents a prototype system together with demonstrations of applications that support violinists during musical instrument learning.
This paper presents integrated tactiles (or vibrotactiles), a novel interface for movement and posture tuition that provides real-time feedback in a tactile form by means of interactive haptic feedback. Tactile feedback conveys information by using non-audio and non-visual form and is a promising means for movements in 3D-space. In this paper we demonstrate haptic augmentation for applications for musicians, since it (a) doesn't affect the visual sense, occupied by reading music and communication, (b) doesn't disturb in bang sensitive situations like concerts, (c) allows to relate feedback information in the same tactile medium as the output of the musical instrument, so that an important feedback channel for musical instrument playing is extended and trained supportive. Even more, instructions from the teacher and the computer can be transmitted directly and unobtrusively in this channel. This paper presents a prototype system together with demonstrations of applications that support violinists during musical instrument learning.
Keywords:
real-time ;
feedback ;
haptics ;
motion ;
posture ;
gesture ;
audio ;
haptics ;
feedback
ISBN:
978-3-642-04075-7
PUB-ID:
2017204 | Link: http://pub.uni-bielefeld.de/publication/2017204
tacTiles: a low-cost modular tactile sensing system for floor interactions
Anlauff J, Großhauser T, Hermann T (2010)
In: Proceedings of the 6th Nordic Conference on Human-Computer Interaction: Extending Boundaries. New York, NY, USA: ACM: 591 - 594.
In: Proceedings of the 6th Nordic Conference on Human-Computer Interaction: Extending Boundaries. New York, NY, USA: ACM: 591 - 594.
Conference Proceeding/Paper
|
Published
|
English
Authors:
Anlauff, Jan
;
Großhauser, Tobias
;
Hermann, Thomas
Department:
Ambient Intelligence
Center of Excellence - Cognitive Interaction Technology CITEC
Center of Excellence - Cognitive Interaction Technology CITEC
Abstract:
In this paper, we present a prototype of a spatially resolved force sensing floor surface. The force sensors are based on conductive paper and grouped into modules called tacTiles. Due to the cheap and widely available materials used for tacTiles, the approach is suitable as a low-cost alternative for spatially resolved tactile sensing. The necessary techniques are shared as an open source and open hardware project to provide an affordable tactile sensing for smart environments. As an interactive application of these tacTiles, we present a detection of step direction algorithm used to count steps into and out of a room.
In this paper, we present a prototype of a spatially resolved force sensing floor surface. The force sensors are based on conductive paper and grouped into modules called tacTiles. Due to the cheap and widely available materials used for tacTiles, the approach is suitable as a low-cost alternative for spatially resolved tactile sensing. The necessary techniques are shared as an open source and open hardware project to provide an affordable tactile sensing for smart environments. As an interactive application of these tacTiles, we present a detection of step direction algorithm used to count steps into and out of a room.
Keywords:
modular systems ;
force sensing ;
HCI ;
tactile floor sensing ;
tacTiles ;
paper FSR ;
open source ;
open hardware
ISBN:
978-1-60558-934-3
PUB-ID:
2277066 | Link: http://pub.uni-bielefeld.de/publication/2277066
The Sonified Music Stand - An Interactive Sonification System for Musicians
Großhauser T, Hermann T (2009)
In: SMC 2009 - Proceedings of the 6th Sound and Music Computing Conference. Gouyon F, Álvaro Barbosa XS (Eds.); 1. Porto, Portugal: Casa da Musica: 233 - 238.
In: SMC 2009 - Proceedings of the 6th Sound and Music Computing Conference. Gouyon F, Álvaro Barbosa XS (Eds.); 1. Porto, Portugal: Casa da Musica: 233 - 238.
Conference Proceeding/Paper
|
Published
|
English
Authors:
Großhauser, Tobias
;
Hermann, Thomas
Editors:
Gouyon, Fabien
;
Álvaro Barbosa, Xavier Serra
Department:
Center of Excellence - Cognitive Interaction Technology CITEC
Ambient Intelligence
Ambient Intelligence
Abstract:
This paper presents the sonified music stand, a novel interface for musicians that provides real-time feedback for professional musicians in an auditory form by means of interactive sonification. Sonifications convey information by using non-speech sound and are a promising means for musicians since they (a) leave the visual sense unoccupied, (b) address the sense of hearing which is already used and in this way further trained, (c) allow to relate feedback information in the same acoustic medium as the musical output, so that dependencies between action and reaction can be better understood. This paper presents a prototype system together with demonstrations of applications that support violinists during musical instrument learning. For that a pair of portable active loudspeaker has been designed for the music stand and a small motion sensor box has been developed to be attached to the bow, hand or hand wrist. The data are sonified in real-time according to different training objectives. We sketch several sonification ideas with sound examples and give a qualitative description of using the system.
This paper presents the sonified music stand, a novel interface for musicians that provides real-time feedback for professional musicians in an auditory form by means of interactive sonification. Sonifications convey information by using non-speech sound and are a promising means for musicians since they (a) leave the visual sense unoccupied, (b) address the sense of hearing which is already used and in this way further trained, (c) allow to relate feedback information in the same acoustic medium as the musical output, so that dependencies between action and reaction can be better understood. This paper presents a prototype system together with demonstrations of applications that support violinists during musical instrument learning. For that a pair of portable active loudspeaker has been designed for the music stand and a small motion sensor box has been developed to be attached to the bow, hand or hand wrist. The data are sonified in real-time according to different training objectives. We sketch several sonification ideas with sound examples and give a qualitative description of using the system.
Keywords:
real-time ;
sonification ;
motion ;
gesture ;
feedback ;
violin
ISBN:
ISBN: 978-989-95577-6-5
PUB-ID:
2017217 | Link: http://pub.uni-bielefeld.de/publication/2017217
New CITEC Building
Interactive Intelligent Systems
Funded in the Excellence-Initiative
