While rehearsing at SXSW last month, Chagall van den Berg ran into an unusual issue: her digital knee shot out in the wrong direction. “My friend laughed and said, 'Wow, that's a problem no. SXSW artist has ever had,'” van den Berg tells The Verge . gloves and a full-body suit covered in sensors, which, during this SXSW performance, not only control a projection of a digital avatar that appears behind here, but also control almost every instrument and effect in the music and her voice. If she moves across the stage, her avatar, floating in space, moves in sync. When she stretches her arms above her head, grains or audio slow to a gravel and stutter. Every hand and body movement has cause and effect, crafting a pop-infused dreamscape to watch.
van den Berg performs using motion-tracking gloves in 2017. Performance begins at 3:50
Born in Amsterdam, Holland, van den Berg was obsessed with both music and computers from an early age, but had always viewed them as separate things. She played instruments in bands, was a singer / songwriter, and traditionally worked with other producers to create beats for her tracks. Then in 2011, she entered a "real" studio for the first time. Watching producers noodle around with her songs to make remixes, it clicked: "I did this." She taught herself music production and shortly after, in 2012, released her debut EP. "I immediately noticed how much freedom and independence that gave me," van den Berg says. “I didn't need other people producing my songs anymore. My musical expression became so much better because I could just make the sounds I had in my head instead of explaining them to another human being. "" I'm kind of a guinea pig, but that's fine with me. "
van den Berg had solved one problem for herself, but releasing music created another: how would she perform the songs she'd made? The nature of most electronic tracks she had two options – behind a table with a bunch of gear and knobs and faders, or play and backing track and sing on top. Neither was acceptable here. “I had this choice,” says van den Berg. “Either it was going to be real and live, but boring to watch and distant from the audience, or to play and be able to dance around onstage. Dancing around and being one with the audience was more appealing, but the musician really didn't like the idea of singing along to a track. So I had a dilemma. ”
This dilemma van den Berg faced is a problem many DIY and electronic artists encounter – how do you incorporate movement and expressiveness when you essentially perform at a desk, using an interface the audience will probably never see? And then make it interesting? Acts like The Glitch Mobiles use electronic drum kits and hacked Microsoft Surfaces tilted towards the audience, while several startups, like Enhancia and Genki Instruments, are banking on MIDI controllers. van den Berg's initial solution was Mi.Mu, a pair of movement-tracking gloves created by musician Imogen Heap. Each glove has nine sensors and triggers that are completely customizable. Almost any movement can be assigned to any musical parameter, so you could, for example, drop your arm down to add reverb, or pinch in the air to add chorus to your voice. van den Berg was inspired by the Mi.Mu gloves, she wrote "Sappho Song" the day she received them.
That was in 2014, and it launched a quest for van den Berg to fluidly bridge the gap between her body movements onstage, and control over audio and visual effects. While the gloves allowed van den Berg to control audio and video with her hands, wanted to do more: how could she use her whole body instead of just her hands? The iteration she's now working on combines the gloves with a motion-capture suit, the child of thing generally used to record people's movements for video games or movies. For now, she has one song programmed to use all the tech she's wearing, but the goal is to eventually expand to a full, hour-long show. That's not an easy task. Along the way, she had to learn C ++, find her sensors (she owns the system, estimating the total cost at around $ 12,000), and continuously experiment with existing technologies across platforms to tie it all together with  van den Berg's setup appears sleek and minimal onstage at SXSW, but there is a complex web of hardware and software needed in order to make it run. She wears the Mi.Mu gloves on her hands, and then puts on a custom bodysuit embedded with 15 or Xsens' 3D motion-tracking sensors spanning her arms, shoulders, head, pelvis, legs, and feet. Three computers – Mac laptop running three apps for controlling audio playback and effects, on Mac mini using the "creative coding" tool OpenFrameworks to manage all the visuals controlled by the Mi.Mu gloves, and on a Windows laptop that runs the sensor's sensors and connects them in real time to the game engine Unity using the wireless full-body VR platform VRee. There is also additional hardware, like microphones and an audio interface. To make sure there aren't any hiccups, the show runs on its own Wi-Fi network, which requires venues to turn off their house Wi-Fi or move it to a different frequency to prevent interference. It's a lot. But the result – a realization of five years of work to connect body, voice, music, and video into one reactive thing – is incredibly exciting.
The whole process needs to get a lot easier if other electronic musicians are going to adopt it, van den Berg says. "[My setup] is not how we are going to change the world of electronic music performance because no one is going to spend all of this money and then be super bothered about IP addresses and networking during soundcheck," she says. “Who wants to tour with three expensive computers and has all of that up during a 20-minute changeover at a festival? That's crazy. ”
van den Berg isn't the only electronic artist thinking about the connection between her physical self and technology – musician Laura Escudé is also movement-based controllers and reactive visuals, for instance – but she's one of the most ambitious. And although every hack is self-serving that it brings here closer to the show she has her head, there ultimately a bigger goal: van den Berg hopes her experimentation will ask others to think about how their performances can be more immersive, and that she can make it easier for artists who want to do similar things.
"My dream is that I'm not the only one in the world doing stuff like this," says van den Berg. "That electronic music performance becomes more human and more expressive." That means being the one to deal with all the technical headaches, so others don't have to.
"I'm kind of a guinea pig," she laughs. . "But that's fine with me."
van den Berg performs at SXSW. Video credit: New Dutch Wave.