See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/237137338

Creating Musical Expression using Kinect

Article · January 2011

CITATIONS READS

12 1,123

3 authors, including:

Min-Joon Yoo In-Kwon Lee

Yonsei University Yonsei University
13 PUBLICATIONS   37 CITATIONS    99 PUBLICATIONS   697 CITATIONS   

SEE PROFILE SEE PROFILE

All content following this page was uploaded by In-Kwon Lee on 21 May 2014.

The user has requested enhancement of the downloaded file.

 Proceedings of the International Conference on New Interfaces for Musical Expression, 30 May - 1 June 2011, Oslo, Norway

Creating Musical Expression using Kinect

Min-Joon Yoo Jin-Wook Beak In-Kwon Lee

Yonsei University, South Korea Yonsei University, South Korea Yonsei University, South Korea
Eng Bld C533, Sinchondong Eng Bld C533, Sinchondong Eng Bld C533, Sinchondong
Seoul, South Korea Seoul, South Korea Seoul, South Korea
[email protected] [email protected] [email protected]

ABSTRACT data, which was defined previously. To translate the data, we

Recently, Microsoft introduced a game interface called Kinect implemented a Kinect-to-MIDI data translator.
for the Xbox 360 video game platform. This interface enables Since the data from the user’s body is now translated to
users to control and interact with the game console without the MIDI data, general programs responding to the MIDI
need to touch a controller. It largely increases the users’ degree commands can be used to create sound or visualization. We
of freedom to express their emotion. In this paper, we first tested Max/MSP for this purpose. We created various sounds
describe the system we developed to use this interface for with parameters converted from body motions in Max/MSP.
sound generation and controlling musical expression. The We then applied this interface to our adlib generation program.
skeleton data are extracted from users’ motions and the data are The movements of the entire body were interpreted to generate
translated to pre-defined MIDI data. We then use the MIDI data control parameters for adlib.
to control several applications. To allow the translation between
the data, we implemented a simple Kinect-to-MIDI data
convertor, which is introduced in this paper. We describe two
2. KINECT
applications to make music with Kinect: we first generate Kinect, officially launched in October 2010, is the first to
sound with Max/MSP, and then control the adlib with our own achieve the elaborate full-body control with 3D motion capture.
adlib generating system by the body movements of the users. This interface is based on technology developed by Rare and
PrimeSense for game technology and image-based 3D
reconstruction. Aided by these technologies, this interface
Keywords provides several useful functions to enable natural interaction.
Kinect, gaming interface, sound generation, adlib generation Kinect consists of three devices: the RGB camera, the depth
sensor, and the multi-array microphone (Figure 1). The camera
can output video at 30 Hz frames with 8-bit VGA resolution
1. INTRODUCTION (640 x 480 pixels). The monochrome depth sensor can then
Kinect (http://www.xbox.com/kinect) is a new game interface sense the depth information, also with VGA resolution with 11-
for Microsoft’s Xbox 360 game console. This interface enables bit depth, which provides 2,048 levels of sensitivity. The
users to control the console with their natural motion. This practical range limit of the sensor is 1.2-3.5 m (3.9-11 ft), and
freedom is achieved by analyzing the image and sound of the the sensing range is adjustable. By interpreting the data
users that the camera and microphone of the Kinect capture. obtained from the camera and depth sensor, we can realize the
Since this ‘controller-free’ interface has the ability to extend the 3D motion capture of the users. Kinect also has a microphone
degree of freedom and expressiveness of the users, many array consisting of four microphone capsules. These devices
researchers and developers have tried to apply the interface in operate with each channel processing 16-bit audio at a sampling
such a way that it not only controls the game console, but also rate of 16 kHz. You can find more detailed information about
controls their own applications. Through their efforts, open the devices on the Kinect homepage and Wikipedia
source drivers for Kinect have recently been developed and (http://en.wikipedia.org/wiki/Kinect ).
released, and various applications of Kinect have now been
presented.
In this paper, we introduce our system, designed to create
and control music according to the user’s body motion via
Kinect. Through Kinect’s ability to interpret a user’s whole
motion, users can control the sound generation system with the
rich range of movement of the body.
First, we extract skeleton data from the user’s body. We can
obtain the position and velocity of each joint of the user’s body
from the skeleton data. The joint data is then converted to MIDI

Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that
copies bear this notice and the full citation on the first page. To copy
otherwise, to republish, to post on servers or to redistribute to lists,
requires prior specific permission and/or a fee.
NIME’11, 30 May–1 June 2011, Oslo, Norway. Figure 1. Kinect consists of three components: (a) RGB
Copyright remains with the author(s). camera, (b) depth sensors, and (c) microphone array.

324
 Proceedings of the International Conference on New Interfaces for Musical Expression, 30 May - 1 June 2011, Oslo, Norway

3. SYSTEM DESCRIPTION 5. APPLICATIONS

We used modules developed in OpenNI
(http://www.openni.org/) to connect Kinect with a PC. The 5.1 With Max/MSP
PrimeSense’s NITE provided useful APIs for the manipulation With our Kinect-MIDI convertor, any music application
of naïve data. We also use FAAST controlled MIDI data can be used to create music and sound.
(http://projects.ict.usc.edu/mxr/faast/) to easily access the joint We chose the Max/MSP program to test the controllability of
data extracted from the body motion. The FAAST streams the Kinect because of its efficiency in creating and modifying
user’s skeleton data over a VRPN server sound. We implemented a Max/MSP patch, which generates
(http://www.cs.unc.edu/Research/vrpn/). sounds by several parameters.
We then implemented a Kinect-to-MIDI convertor (see The skeleton data can be roughly divided into five segments:
Figure 2). This program listens to messages from the VRPN center (0-3), left arm (4-9), right arm (10-15), left leg (16-19),
server embedded in FAAST, and responds if proper messages and right leg (20-23). Thus, we generated five sounds
are received. We predefined a mapping between the joint data corresponding to the body segments, controlled by 4-6
and the MIDI data. Thus, the input joint messages are translated parameters also corresponding to the joint data belonging to
to MIDI messages using this mapping. Finally the MIDI each segment. For example, one sound corresponding to the left
messages are sent to a MIDI-IN port. leg had four parameters, and these parameters are controlled by
Our translator is operated in a similar way to the Wii-to- the movements of the left hip (16), left knee (17), left ankle
MIDI translators such as GlovePIE or OSCulator. We were (18) and left foot (19).
inspired by these programs for creating music using game We then mapped the velocity of the joints to the parameters
controllers. of the sound. The velocity of a joint can be easily calculated by
the difference between the position of the joint in one video
frame and the position in the previous video frame. By using
velocity data, users could change the sound more intuitively.

5.2 With Our Adlib Generator

We tested our system in our own adlib software. The software
was originally implemented to generate an adlib sequence via
the user’s line drawing. The user drew lines using a mouse
interface, and the adlib was generated by data obtained using
the movement and position of the mouse. We extended the
software by changing the input from mouse to Kinect.
Figure 2. Kinect-to-MIDI convertor. The speed and pitch of the adlib was controlled by the average
velocity and position of each joint, respectively. The scale of
the adlib was then determined by the relative position of each
4. SKELETON DATA end-joint (head (0), left fingertip (9). right fingertip (15), left
FAAST streams a total of 24 skeleton joint data tracked in the foot (19), and right foot (23)). We predefined the positions of
input video. The joints are ordered corresponding to the the end-joints generating a normal pose. Then, if the end-joints
OpenNI framework (Table 1). As you can see in this table, the were closer to each other than the normal pose, (the user’s pose
skeleton data contains all the positions of the major parts of the shrunk), the adlib was created with a diminished scale. Also,
body, and we can reconstruct the movement of the body the farther the end-joints were from each other, the tenser the
approximately from the skeleton data. scales used to generate adlib. Compared with the previous
control using a mouse, this system can express more adlib
styles because of the more intuitive controller. We are now
Table 1. The skeleton joint order by OpenNI framework. conducting user surveys about the control method and resulting
Sensor Joint Sensor Joint adlibs.

0 Head 12 Right Elbow

6. CONCLUSION
1 Neck 13 Right Wrist We first reported a system to generate and control sound with
2 Torso 14 Right Hand Kinect. Several open-source drivers and modules were used to
extract the position data from the movement of a user’s body.
3 Waist 15 Right Fingertip We then converted the data into the MIDI messages by using
our Kinect-to-MIDI translator. Because the data from the body
4 Left Collar 16 Left Hip motion can be presented in MIDI messages, any music
5 Left Shoulder 17 Left Knee application responding to MIDI data can be used to create and
control music. We first tested this method with Max/MSP
6 Left Elbow 18 Left Ankle software, then with our adlib generator. In our prototype tests,
we could control the music more intuitively using body motions.
7 Left Wrist 19 Left Foot We are exploring more applications and methods for
8 Left Hand 20 Right Hip controlling music using Kinect and also doing more theoretical
research by studying literature related to analysis of
9 Left Fingertip 21 Right Knee movement. .
10 Right Collar 22 Right Ankle
11 Right Shoulder 23 Right Foot

325

View publication stats