Closed-loop Auditory Interactions

Research Areas: 

Body movements are complex and demand multivariate coordination. CLAINT evaluates the potential of interactive motion sonification to support the learning and execution of complex coordinated motion sequences. This includes the development of sensing and sonification technology and it enables several novel applications: (a) the support of skill learning, movements in dance and musical instrument playing are focused, (b) the support of tool use, such as a drilling machine, tool-integrated interactive sonification techniques are developed, and (c) to help preventing unhealthy movement patterns, e.g. in musical instrument playing.

Methods and Research Questions: 

Body movements as they occur in physical activity such as musical instrument playing, dance, sports, or using tools involve the fine coordinated control of many muscle groups. Often we have to learn new patterns, such as in physiotherapy or instrument learning, or we risk our health by performing them incorrectly (e.g. lifting heavy objects in the wrong way). In many cases a trainer (teacher, physiotherapist, etc.) may want to monitor the movements to give helpful advise, but a simultaneous perception of fine-controlled movement components on different body parts is invisible, difficult or not at all perceptible. How can we better overcome the limitations of visual-only movement inspections? How can multimodal real-time display of movements provide richer information about the detailed movement execution? How can such online-feedback benefit the performers themselves to better understand, learn, conceptualize and memorize movements? CLAINT uses the methods of wearable computing, i.e., sensors and actuators that are integrated unobtrusively into the body periphery. Besides clothes, this encompasses the integration of technology into tools that the performers use for the task, such as the violin bow or a drilling machine. Engineering problems as familiar in the areas of Physical Computing to Ambient Intelligence need to be solved first. An important challenge is to develop a modular, unobtrusive, wireless, reusable, flexibly extensible sensor/actor node capable to collect, process and broadcast sensor data to the processing computers. Interactive sonification, the real-time auditory display of measured data by using non-speech sound and noise, is the main method to achieve a closed-loop interaction whereby the user is not forced to attend any visual display. This is an important prerequisite since the eyes are usually already engaged in other activities such as reading the score while making music, or the head moves like in dance so that any visual display cannot be applied. Sound furthermore offers the particular advantage that acoustic streams can simultaneously display many variables, e.g. by using pitch, level, brightness, pulse rate, spatial location of sound and more timbre parameters.


Regarding the challenge of an unobtrusive wearable sensing and actuator system, the ARTIS system has been developed within CLAINT which allows to connect, depending on the particular demands of the application, to connect different sensors. Low-cost sensors such as goniometers, force-sensitive resistors, distance sensors, and actuators such as vibrating coils, LEDs, and mini-loudspeakers have been integrated into that platform. Bluetooth is being used as wireless interface to a PC.

Regarding the sonification, various real-time sonification approaches have been implemented under the SuperCollider language. They can be configured just-in-time by the operator or user. Tests with individual users have shown that quite simple and elementary sonifications that merely reflect the most significant variable of interest, e.g. the bending of the knee, or the contact of the feet during jumps in dance, are most accepted and effective. Over the course of a tutoring unit, settings should - and can be - adjusted to adapt to the problem that teacher and pupil work at.

Concerning applications, the CLAINT project has developed selected demonstrator systems for three areas:

  • for tool-integrated auditory feedback, a drilling machine has been extended by sensors to measure the relative orientation to the wall, and actuators to display a deviation from the ideal (or nominal) angle by sound or light. A test with subjects has shown that the feedback is effective to increase drilling accuracy, that is to reduce the average deviation from a nominal drilling angle.
  • for musical instrument learning, sensors and actuators have been integrated into a violin (both body and bow) to support musicians for tasks such as the linear bow stroke. A long-term study with pupils (with and without feedback) suggests that the auditory-supported practice improves movement quality and accelerates learning
  • for ballet dance, particularly for jumps, sensors (goniometer, accelerometer, pressure sensor) have been integrated into the lower limbs of dance pupils, enabling to measure floor contact, knee angle and overall acceleration.  This enabled the teachers, for the first time, to attend to the fine correlations between knee action and contact which are invisible (or: too fast to be analysed) from a mere observation. In a series of interactions between dancers and their teachers, where the ARTIS system augmented the information with interactive sonification, insights on how to structure and use multimodal feedback in tutoring has been gained.