For digitising dance, sensor systems play an important role since they can capture and convert specific aspects of dance into quantitative data. This article gives an overview over some of the more popular sensor systems used in the field of dance. The article then goes into more detail concerning those sensor systems that have been used in the E2-Create project. Two of these sensor systems have been custom developed. For these systems, some information about the hardware and software design is provided.
On a fundamental level, a distinction can be made between sensor systems that are placed at a fixed location in space and provide an external (allocentric) point of view of a dancer’s activities and those that are placed on the dancer’s body and provide an internal (egocentric) point of view. Other distinctions can be made based on the sensing modality that a sensor system operates in (e.g. optical, auditory, magnetic, forces etc.), or the aspects of a dancer’s behaviours that are sensed (e.g. appearance, posture, kinematics etc).
Cameras
Cameras provide an allocentric point of view, operate in the optical modality, and capture the visual appearance of dancers and their surroundings. Conventional cameras record consecutive images with colour information represented as pixels. Some cameras (depth cameras) capture the distance between the camera and the recorded subject at each pixel. This distance is typically represented as intensity value. The main aspects to take into account when working with cameras for recording dance are its optics, resolution, dynamic range, and frame rate, the latter of which is particular critical for recording fast movements. In the case of depth cameras, the maximum distance that the camera can record is crucial since it likely limits the depth of the stage. Furthermore, most depth cameras are sensitive to infrared light and are therefore disturbed by infrared emitting light sources such as spotlights or the sun. Finally, all types of cameras share the same drawback: they are prone to visual occlusion effects, i.e. everything that is not in a direct line of sight of a camera is not recorded. When it comes to detecting body movements, camera-based approaches require a lot of computational power and might not provide the same level of precision as other sensors.
In the E2-Create project, two different camera models for recording colour and distance (RGBD) information have been employed: Intel Realsense D435 and Microsoft Kinect Azure.
Intel RealSense D435
The RealSense D435 camera possesses two infrared cameras, an infrared light source, and one RGB camera. The infrared light source illuminate the scene which is filmed by the two infrared cameras. Using stereovision, the images from the two infrared cameras are combined to derive depth information up to a distance of 10 meters. Both the infrared and RGB camera possess a wide angle optics. At a resolution of 1280×720 pixels, the depth image can be captured with 30 frames per second. The colour image has a resolution of 1920×1080 pixels and can also be captured at a maximum of 30 frames per second.
The experience of working with RealSense D435 camera has been very ambivalent. The main benefit of the camera over the Kinect Azure is its large recording distance. But this benefit is offset by the extremely noisy quality of the depth image.
Microsoft Kinect Azure
The Kinect Azure camera possess a depth camera, an RGB camera, an inertial measurement unit (IMU) and a seven microphone array. For depth sensing, it employs the time-of-flight principle. This principle measures the amount of time that passes between an infrared pulse being emitted and the same pulse being detected after it has been reflected from a surface back into the camera. The resolution and optical properties of the depth sensor depend on the camera settings and range from narrow to wide angle, from 320×288 to 1024×1024 pixels, and from a maximum distance of 2.21 m to 5.46 m. The RGB sensor supports a resolution of up to 2830×2160 pixels. The capture rate for depth and colour image is limited to 30 frames per second.
The Kinect Azure has been used on several occasions in the E2-Create project. In these recordings, the quality of the colour and depth image was very convincing and significantly above that of the Realsense 435. The main drawback of the Kinect with regards to our purposes is its limited depth sensing distance which usually stops at 4 meters. Another drawback concerns the logistics of buying this camera. The supply of this camera is extremely limited with long waiting times and ordering has to take place through an official institution.
During the E2-Create project, an addon entitled ofxDabVideoTracker has been developed for OpenFrameworks that supports among others the recording and playback of colour and distance videos with RealSense and Kinect Azure cameras.
Inertial Measurement Units
Inertial measurement units (IMU) are sensors that measure the acceleration of objects (such as a dancer’s limbs) to which they are attached to. IMU’s typically combine at least two sensing units for measuring two types of acceleration: An accelerometer for sensing linear acceleration, and a gyroscope for measuring rotational acceleration. Each of these accelerations are measured with respect to three orthogonal axes. IMUs provide an egocentric point of view and capture the dynamics of a dancers movements. Some IMU’s also contain a compass sensor to measure the absolute orientation of the sensor. This type of measurement provides an allocentric view. The main benefit of IMU’s is that they directly measure body movement (contrary to cameras) and can capture very nuanced aspects of body dynamics that are relevant for a dancer’s own kinaesthetic perception. Another benefit is that these sensors are usually very affordable. Furthermore, IMUs don’t suffer from occlusion effects. The main drawback of IMU’s is that they don’t provide (at least directly) positional information. Also, they can be sensitive to magnetic interference. Finally, IMU’s are usually not readily usable such as cameras but require some tinkering with microelectronics.
During the E2-Create project, simple wearable devices have been created for recording the movement of dancers. These devices integrate an IMU (Bosch BNO055) and a microcontroller with Wifi communication module (Arduino MKR1000) in a small self-contained housing that can be easily put into sleeves for wearing the sensors on the wrists or ankles. This IMU has the unique property that it calculates on board it’s absolute orientation. The Arduino has a simple firmware that gathers all sensor data and sends them as OSC messages via Wifi to a destination computer. The material list, case design, and firmware are available here.
Force Resistive Sensors
Force resistive (FSR) sensors are sensors that change their resistance when a force is exerted on them. These sensors come in different sizes and shapes and either contain a single or multiple sensing regions.
FSR sensors might not be the most obvious choice for sensing aspects of a dancer’s behaviour. One application is to use them as alternative to Electromyographic electrodes to measure muscle contraction. Another application is to integrate them into shoes to measure the forces between foot and ground. This latter application has been explored in the E2-Create project.
In the E2-Create project, a sensor shoe has been developed that employs force resistive sensors to measure the interaction between a dancer’s feet and ground. The forces generated by this interaction can reveal minute movements such as the ones representing small shifts in balance or preparatory movements that precede larger movements. The current version of a sensor shoe combines 7 FSR sensors (Flexiforce ESS301) that are taped on an inlay sole. A microcontroller, a voltage resistor array, a Wifi communication module (Arduino MKR1000), a Lithium polymer battery (Adafruit 350 mAh), and a battery charger are integrated into the sole of the shoe. The material list, case design, and firmware are available here.
Electromyographic Electrodes
Electromyographic (EMG) electrodes measure the electrical activities of muscles. These sensors are particularly interesting for recording changes in muscular tension. The most accurate sensors employ wet electrodes which are disposable and therefore costly in their usage. Dry electrodes can be used multiple times but they establish a less reliable electrical contact than wet electrodes. Obtaining a good sensor readings can be quite challenging. Often, the reading is very noisy and therefore difficult to associate with for instance a particular gesture. Commercially available EMG sensor systems such as the MyoWare Muscle Sensor from Sparkfun are geared towards a creative coding and DIY electronics community. The by now discontinued Myo armband provides are more integrated and readily useable EMG sensing device.
Physiological Sensors
Physiological sensors encompass an entire family of different sensors that measure physiological signals of the human body. Muscle tension is an example of this and has been briefly address above. Others example signals include heart rate, galvanic skin response, and skin temperature. Some of these signals can be associated with emotional states. A popular range of sensors that can easily be integrated in an electronics project is the Bitalino platform.
Motion Capture Systems
While many sensor systems are used to capture isolated aspects of body motion, it is predominantly highly integrated systems that employ multiple sensors and significant computational processing that are employed for tracking the motion of multiple body parts at the same time. On a basic level, Motion capture systems differ from each other with regards to the sensing technology. Optical motion capture employs a multi-camera setup that surrounds the performer(s). Inertial motion capture employs several IMUs that are worn on the body of the performer(s).
Marker-based Optical Motion Capture
Traditionally, camera based motion capture systems operate with infrared light and reflective markers. These markers are attached at key positions of a body suit that is worn by the performer during motion capture. The motion capture system detects the position of the markers in three dimensional space and reconstructs from them the position and orientations of the most prominent joints of a human body. For position detection, it is necessary that at least three cameras “see” a marker. This allows to triangulate the marker position and convert it from the two dimensional space of the camera image into a three dimensional space. The main benefit of marker based optical motion capture is the spatial and temporal precision with which joint positions and orientations can be inferred. Positions errors are typically in the millimetre or sub-millimetre range and temporal resolution is at 3 milliseconds or lower. This precision makes it possible to capture and reconstruct dance motions with high fidelity. The main drawback of this type of motion capture technology is its sensitivity to marker occlusion and its very high acquisition costs. The first issue has a large impact on the type of dance movements that can be captured. Movements the are on the ground, involve compact poses, or close-by performers are usually difficult to capture. The second issue limits the availability of motion capture for dance companies.
Qualisys Motion Capture System
In the context of the E2-Create Project, a marker-based motion capture system by the company Qualisys was employed. This system has been generously made available to us at no costs by the dance company Cie Gilles Jobin. This motion capture system was used for two purposes. First it served to record and analyse specific movement qualities of choreographer Muriel Romero. This analysis informed the development of a generative system in which simulated articulated entities express the same qualities. Second, it was used as part of a dance performance entitled “Embodied Machine” in which Muriel Romero controls the music and lights through her movements in real-time.
In both cases, the main joints of Muriel’s body and some joints of her hands were tracked. The remaining hand joints were then inferred from the tracked joints. Interactive control required the Qualisys Track Manager (QTM) software to communicate with a custom developed software for sound and light control. For this purpose, a small python script has been written that communicates with QTM in its native format and then translates the received data into OSC messages. This software is available here.
Markerless Optical Motion Capture System
Since recently, optical motion capture systems are available that determine joint positions and orientations without the need for performers to wear markers. This obviously provides benefits in that performers can wear regular costumes. And it also simplifies situations in which different performers have to be quickly recorded one after each other with too little time in between for changing costumes. At the moment, marker-less motion capture systems cannot compete in precision and speed with marker-based alternatives. For this reason, marker-less approaches are currently limited to recording upright body postures and simple movements such as running and walking.
The Captury Motion Capture System
In the context of the E2-Create Project, a marker-less motion capture system by the company Captury was employed. The system has been acquired by MotionBank and is permanently setup in the iLab at the Fachhochschule Mainz. Using this system, several motion capture recordings have been conducted in collaboration with professional dancers from the Staatstheater Mainz. These recordings formed part of a small study that served the purpose of comparing computational representations of dance movements with notions of movements that the dancers work with. A dedicated page describing this study is available here. In addition, the same dancers were also asked to freely improvise to music while being motion captured. The motion capture and audio recordings from these improvisations were later on used to train a machine learning model on the translation of movement into audio. More information about this machine learning model is available here.
While working for a while with this motion capture system, we realised that many of the performance tasks where too challenging for the motion capture system. As a result, many of the motion capture recordings needed a large amount of cleaning in order to usable for example as datasets for machine learning. Because of this, we conclude that the usefulness of marker-less motion capture in contemporary dance is at least for the moment very limited.
Inertial Motion Capture
Inertial motion capture systems employ IMUs instead of cameras for tracking the motion of joints of a human body. Since IMUs don’t provide any positional information, joint positions have to be derived by integrating the joints’ acceleration. This operation is fairly inaccurate and often leads to a position error that accumulates over time (drift). Nevertheless, inertial motion capture suits are interesting for dance purposes for several reasons: they are much affordable than optical motion capture systems, they don’t suffer from occlusion effects, and they can be used in situations where the light conditions would prevent an optical approach. The two most prominent suppliers of inertial motion capture systems are Rokoko and Perception Neuron.
Pose Estimation
Last but not least, a method worthwhile of mentioning is Pose Estimation. Pose Estimation is machine-learning based approach for analysing regular camera images and deriving joints positions from them. Most Pose Estimation systems can perform this task on images the represent a single camera perspective. The joint positions obtained from these images are two-dimensional. There also exist 3D Pose Estimation systems but these typically don’t operate directly on images but instead require 2D poses as input. The most recent Pose Estimation systems not only detect joint positions but also reconstruct an approximation of the tracked person(s) body volume. So far, Pose Estimation systems have mostly been trained on images showing common poses. Accordingly, many of these systems fail to correctly locate joint positions in images of dancers. Also, many Pose Estimation systems require considerable computational power and are therefor barely applicable for real-time applications. For this reason, Pose Estimation systems are still far away from competing with marker-based motion capture system. Nevertheless, Pose Estimation systems can be useful for example as interactive element in public installations in which the movement of users are neither very unusual nor do they have to be tracked precisely and quickly to achieve the desired effect.
The following is a brief list of 2D Pose Estimation systems that are relatively easy for creative coders who are not experts in machine learning to setup and run. Some of these Pose Estimation systems are included in the “Model Zoo” of GluonCV, a toolkit that makes it particularly easy to work with machine learning models for computer vision.