In November 2010 Microsoft launched Kinect for the Xbox 360: a sensor bar with an infrared depth camera, a colour camera and a microphone array, sold as a way to play games without a controller. Within days, hackers had open drivers talking to it from ordinary computers. For electronic musicians, that opened something no MIDI controller had offered at a consumer price — a way to read the whole body as data, with nothing to hold.
This is the first part of Body as Interface, our series on playing music with gestures, hands, headsets and brain signals. It starts with Kinect because so many of the ideas — and mistakes — that shape today’s tools were worked out on that cheap black bar.
A controller with nothing to touch
Kinect’s trick was depth. Instead of seeing a flat picture, it measured distance, so software could separate a person from the room and estimate a skeleton of joints — head, hands, elbows, hips, knees — many times a second. Every joint became an x, y and z value. Map those to MIDI control changes or OSC messages, and a raised arm could open a filter, a step to the left could crossfade two loops.
The music community built the bridges quickly. Ryan Challinor’s Synapse sent Kinect skeleton data as OSC into Max, Max for Live and Quartz Composer. In Australia, Chris Vik developed Kinectar and posted videos of himself live-looping in Ableton Live using nothing but body movement. Forums filled with patches, and a generation of performers learned the basic grammar of mapping: scale the range, smooth the jitter, decide which moves are instrument and which are just dancing.
They also learned the limits. Skeleton tracking lagged behind fast gestures. It needed floor space and consistent light. And there was no resistance, no key or string pushing back — performers had to find the edges of a control by watching a screen or by memory.
Imogen Heap: the glove and the room
Imogen Heap came at the problem from the other end. In 2010 she began assembling a team to build musical gloves — a way, as she described it, to take the studio’s tricks out of the laptop and put them in her hands on stage. The project drew in engineers, musicians and textile designers, including Tom Mitchell at UWE Bristol, Kelly Snook, Adam Stark and designer Rachel Freire.
The gloves carried their own sensors: bend sensors along the fingers and an inertial measurement unit for orientation and motion, sending data wirelessly to a laptop. Machine learning recognised hand postures — a fist, an open palm, a point — and each posture and movement was mapped to Ableton Live: recording and looping her voice, grabbing and bending sounds, steering effects.
“The gloves are like a second skin. They are part of me. An extension of me.”
Where Kinect comes in is scale. The gloves were superb at fine detail but knew nothing about where Heap stood. For a performance of Me the Machine, the team combined glove data with position data from a Kinect, so her location on stage became part of the instrument alongside the postures of her hands. Walking toward the audience could shift the sound as surely as closing a fist.
That split is the most useful lesson in this whole story. Hands are for detail; the body is for space. Heap’s rig used each sensor for the job it was good at, instead of asking one camera to do everything.
The acts who played the room
Heap was not alone. In Auckland in 2012, The V Motion Project put a dancer in front of Kinect sensors and a giant projection, so his movements triggered and shaped the music in real time — one of the most widely shared demonstrations of the era. Melbourne’s Ethno Tekh built full-body audiovisual sets around Kinect data, and Chris Vik kept pushing Kinectar into live shows.
The gloves themselves travelled furthest. Developed into the commercial MiMU gloves with their own Glover mapping software, they reached pop stages when Ariana Grande used them on tour in 2015, and they have since been taken up by artists working with voice, synths, lights and visuals. Heap’s original aim — performing electronic music as visibly as a guitarist plays a guitar — became a product other people could buy.
When the camera went away
Kinect’s own story ended in stages. Microsoft stopped manufacturing the consumer sensor in 2017. Its developer successor, the Azure Kinect DK, was discontinued in 2023, with the technology passed to partners — Orbbec’s Femto Bolt is now the depth camera most often suggested as a replacement for existing body-tracking patches.
For musicians, the disappearance mattered less than you might think. The ideas had already moved on: into gloves and wearables, into dedicated hand trackers like Leap Motion, and into the cameras built into every modern XR headset.
What Kinect taught the next wave
Combine scales. The best rigs layer sensors: a camera or headset for body and room, a glove or hand tracker for fingers, and — increasingly — biosignals such as EEG or muscle sensing for intent. Each covers a different resolution of movement.
Mapping is the instrument. The hardware only produces numbers. What made Heap’s performances work was the mapping: a small, memorable set of gestures, each with an obvious musical result the audience could follow.
Latency and feedback decide everything. Hand tracking such as Ultraleap’s Hyperion now chases tiny finger movements, and headsets track hands at close range — but players still need to feel where a control begins and ends, whether through haptics, sound or visual cues.
It is not only the metaverse. The richest uses of body tracking have happened on real stages, in galleries and in studios. Headsets are one more environment to add to that list, not a replacement for it — which is the thread we will pick up in the rest of this series.