A DUET WITH A ROBOT: How the Japanese, Americans, and Europeans taught machines to play the theremin

The idea of “teaching” robots to play music—or simply to interact with it—goes back many decades. The concept of “mechanical” music, from music boxes to orchestrions, found a natural continuation in robotics. For years, engineers and designers have demonstrated robot drummers, robot guitarists, and robot violinists. Music—a subtle art that demands precision, advanced motor skills, and expressiveness—has always been a challenge for engineers and a clear, compelling way to show just how advanced a robotic system can be.

Over recent years, robotics has made significant strides. We do not yet live in a world where robots have replaced humans or become part of everyday life; you do not see humanoid robots walking down the street or working in local shops. But that could change over the coming decade.

The theremin is one of the most difficult musical instruments to play, requiring stillness and precision—qualities that robots naturally excel at. It has no keys or strings, which simplifies a robot’s interaction with it. As we will see from several examples, even a relatively simple device can achieve basic pitch control on a theremin. It is no surprise that this has attracted the attention of robotics companies and developers.

Recently, I experimented with a robot model created in China, and it could easily replicate movements and produce sounds on a theremin. But just the other day, we witnessed one of the most advanced humanoid robots perform a duet with a human—not merely producing sounds, but playing an actual melody.

On 29 May 2026, Carolina Eyck shared a video in which she performed a duet with a robot. Together they played the famous “Flower Duet” from Léo Delibes’ opera Lakmé on two theremins.

“Before anyone starts worrying about thereminists being replaced by robots: a humanoid robot is only as precise as its motors and motion control allow—and Ameca was still discovering just how demanding a theremin can be 🙂 The final audio recording included a little editing and some re-recorded theremin passages to make the performance as pleasant as possible to listen to,” writes Carolina.

According to her, she first became acquainted with the robot Ameca in 2023. The duet with Carolina Eyck was commissioned for a smart-city exhibition and developed in collaboration with the companies Waiys and SpiNNcloud.

“We are thrilled to present a demonstration video showcasing cutting-edge capabilities of human–robot interaction in real-time AI systems. In the video, the humanoid robot Ameca joins renowned theremin performer Carolina Eyck in a musical duet, achieving precise, expressive, and highly effective interaction thanks to our hybrid AI architecture. This architecture was developed for an innovative smart-city project and integrates neuromorphic processing (Intel Loihi 2 and SpiNNaker 2) with deep learning and symbolic AI. The impressively low latency achieved in this project demonstrates what WAIYS does best: creating synergies in hybrid architectures, whether in robotics or AI assistants,” according to Waiys, which published a demonstration video of the project earlier this year.

Carolina later shared some details about the project:

“Ameca was programmed using my system of eight finger positions, derived from my playing technique and translated into movement sequences. After calibrating the D-Lev theremin, the MIDI pitch information was mapped to the finger positions and corresponding motion sequences. Hours of training, programming and fine adjustments—and Ameca was finally ready to perform.”

Ameca was created by the engineers of Engineered Arts at the company’s headquarters in Falmouth, Cornwall, UK. Founded in 2004 by Will Jackson, Engineered Arts originally focused on building interactive installations for museums and visitor attractions.

The idea for Ameca was conceived in February 2021, and by December of the same year the robot’s first teaser video had spread across social media, attracting millions of views. Ameca made its major public debut at CES 2022 in Las Vegas. The developers deliberately gave the robot an androgynous appearance and neutral facial features, avoiding the discomfort that can sometimes accompany highly realistic humanoid designs.

Since then, three generations of the robot have been released. Ameca Gen 2.6 was introduced in 2024, while the third-generation model was first presented at the ICRA 2025 conference.

Ameca is a sophisticated technological system housed within a remarkably realistic body. The robot stands 1.87 metres tall and weighs 62 kilograms. Its “facial muscles” consist of 61 actuators, 27 of which are dedicated solely to facial expressions, allowing it to reproduce more than 50 realistic expressions. Two 8-megapixel cameras embedded in its eyes provide visual perception, while its audio system combines microphones located in the ears with a multichannel microphone in the chest.

At the core of Ameca is the proprietary operating system Tritium robotOS. The latest version, Tritium 3, supports more than 55 languages, including Russian and Arabic. The robot can also analyse facial expressions and vocal cues to identify a speaker’s emotional state.

Ameca regularly appears at major technology events around the world. It has been showcased at conferences such as MWC 2025 in Barcelona and featured in museums dedicated to future technologies. In 2022, it even delivered an alternative Christmas message on British television.

We wholeheartedly congratulate Carolina Eyck on this remarkable collaboration. The theremin remains one of the most innovative musical instruments of the twenty-first century, and it is fascinating to see it intersect with the latest developments in robotics and artificial intelligence.

We have not yet reached the point where robots are precise and intelligent enough to fully master the motor skills required for advanced theremin performance. Nevertheless, this project represents an important and intriguing attempt to push beyond the current limitations of robotic technology.

The idea that a robot might one day master the theremin—an instrument played without physical contact—is not as new as it may seem. Long before Ameca, researchers had already begun exploring this unusual and captivating possibility. The history of robotic thereminists now spans almost three decades and includes contributions from researchers in the United States, Japan, Germany, and the United Kingdom.

Let us take a look back at how engineers and scientists have gradually taught robots to perform on one of the world’s most unusual musical instruments.

🇺🇸 USA: Students, Innovation, and a Robot That Plays by Ear

Among the first to tackle this challenge were researchers at Vanderbilt University. Their solution was both elegant and ahead of its time. In 1998, student Jason Barile and his supervisor, Professor Kazuhiko Kawamura, presented the bipedal robot ISAC, equipped with pneumatic artificial muscles known as rubbertuators.

What made ISAC unique was that it did not simply memorise hand positions. Instead, it played by ear. Using specialised pitch-recognition software, the robot could listen to the note it was producing in real time, calculate the error, and immediately adjust the position of its hand. Thanks to its flexible pneumatic muscles, ISAC could even add expressive techniques such as vibrato.

This represented a genuine breakthrough: a shift from simple mechanical reproduction towards something resembling musical hearing.

🇯🇵 Japan: A Musical Ensemble with a Robotic Partner

Researchers at Kyoto University, led byb pursued an even more ambitious goal. They wanted to create not merely a robot soloist, but an ensemble musician capable of performing alongside a human partner in real time.

Their instrument of choice was the humanoid robot HRP-2. The researchers equipped it with three key capabilities. First, it learned to suppress its own sound so that it would not interfere with listening to its human partner. Second, it developed a robust system for tracking musical tempo. Third, it employed compensation-based control techniques to improve pitch accuracy.

In a 2011 video featured on Synthtopia, musician Tatsuhiko Itohara plays guitar while HRP-2 performs on a theremin. Using cameras and microphones, the robot tracks both the musician’s movements and the tempo of the performance.

As the researchers explained, “HRP-2 can synchronise with the sound of an acoustic guitar.”

This was far more than a technological curiosity. The project formed part of serious academic research and eventually led to a joint patent with Honda in 2014 for what the authors described as a “robot thereminist.”

The key innovation was that the robot did not simply move its hands along a predetermined trajectory. Instead, it relied on audio feedback: it listened to itself, calibrated before performing, and adjusted its hand position in real time.

Such adaptation is essential on the theremin, an instrument that is extremely sensitive to changes in its surroundings. Without it, the robot quickly drifts out of tune and produces unstable pitch.

🇩🇪🇬🇧 Germany and the United Kingdom: Flexibility, Expressiveness, and Artificial Intelligence

Researchers in Germany explored a different path—one that could eventually make robotic musicians far more autonomous.

At Dresden University of Technology, scientists developed a system capable of recognising musical notation and converting it directly into performance. Rather than programming every movement in advance, the robot could read a score and perform unfamiliar melodies on its own.

This marked an important step away from pre-programmed demonstrations and towards genuinely intelligent musical systems.

Another notable project emerged in London. Presented at the Kinetica Art Fair in 2010, the Robotic Theremin Ensemble brought together artist and musician Ray Lee with three robotic assistants to form a theremin quartet.

Using foot switches and MIDI-controlled robotic mechanisms, Lee created four-part harmonies that blended human performance with machine precision. His work combines analogue electronics with digital technologies, creating a unique musical and visual experience.

Lee’s insect-like mechanical constructions perform on real theremins, and his background as a musician allows him to create a compelling fusion of sound, movement, and visual art.

That same year, researchers at Imperial College London unveiled an “anthropomorphic robot thereminist.” Unlike many earlier systems, this robot used multiple degrees of freedom to imitate the movement of a human hand.

As a result, it could perform sophisticated techniques such as air fingering and, perhaps more importantly, introduce a greater degree of musical expressiveness into its performances.

🛠 DIY Movement: Robots Built from Everyday Materials

Alongside university laboratories and large research projects, a more grassroots approach to robotic music-making also emerged.

One of the most memorable examples was Lev, a robot built in 2007 by artist and enthusiast Ranjit Bhatnagar. Constructed from an old lamp, tin cans, and a collection of microprocessors, Lev demonstrated that creating a robot musician did not necessarily require a large budget. Creativity, engineering ingenuity, and a passion for music could be just as important.

More than anything, Lev showed that the idea of a robot thereminist had captured the imagination of people far beyond academic research laboratories.

In the same year, British musician and engineer Sarah Angliss introduced Clara 2.0, a robotic doll capable of listening to a human performer and adapting its own playing in response.

“Hope you enjoy watching her talents (or lack of them),” Angliss joked when presenting the project.

Unfortunately, the original video documentation has since disappeared from the internet. The robot’s name was a tribute to the legendary theremin virtuoso Clara Rockmore.

🧠 A New Stage: Machine Learning

The next major step in the evolution of robotic thereminists came when researchers began moving away from manually programming every movement of the robot.

Instead of calculating the exact hand position required for each note, scientists at the University of Hamburg explored the use of reinforcement learning. Their idea was simple but powerful: give the robot a musical objective and allow it to discover for itself how to achieve the desired pitch at the required moment.

By 2020, the team had achieved significant results.

Particularly important was the study “Reinforcement Learning with Time-dependent Goals for Robotic Musicians,” which introduced the concept of time-dependent goals. This approach enabled a robot to learn complete melodies rather than isolated notes.

Unlike traditional control systems, reinforcement learning opens the possibility of further self-improvement, adaptation to new instruments, and even limited forms of musical improvisation.

The system used two robotic manipulators positioned near the theremin’s antennas. Their movements generated the musical performance, while sophisticated signal-processing algorithms translated information from a musical score into physical actions.

The project was carried out under the supervision of Professor Christina Kelber, whose group focused on the challenging task of converting visual musical information into accurate and expressive movement.

Looking back at nearly three decades of experiments, it becomes clear that researchers around the world have repeatedly turned to the theremin as a unique testing ground for robotic intelligence and motor control.

The instrument presents an unusual challenge. Unlike a piano, violin, or guitar, it offers no physical points of reference. Every note must be located in empty space, requiring constant feedback, precision, and adaptation. In many ways, the theremin forces robots to confront some of the same challenges faced by human performers.

From university laboratories in the United States and Japan to art installations in Britain and machine-learning experiments in Germany, each project has explored a different aspect of what it means for a machine to become a musician.

Today, robots are still far from matching the sensitivity, flexibility, and artistic intuition of a skilled thereminist. Yet the progress made over the past three decades is remarkable. What once seemed like a curious experiment has gradually evolved into a serious field of research at the intersection of robotics, artificial intelligence, and music.

And who knows? Perhaps one day we will see a robot on stage creating genuine art—not merely reproducing notes, but performing with expression, imagination, and individuality.

If that day comes, the theremin may well be one of the instruments that helped make it possible.