SoundSculpt: Sculpting Music Through Motion and Immersion

Yiqi Zhao’s interactive synthesizer redefines creative expression with XR technology

Inspired by the ancient art of sculpting, SoundSculpt by Yiqi Zhao invites users to shape sound as if molding clay, bridging the gap between physical and digital creativity. This interactive music synthesizer leverages the tactile satisfaction of hands-on art, empowering individuals to become active participants in the creative process. By integrating spatial audio and visual elements, SoundSculpt blurs the lines between creator and creation, offering a unique, emotionally resonant experience.

At the heart of SoundSculpt is its innovative use of Bose AR Frames, wearable technology equipped with gyro sensors that capture six degrees of head movement. These movements are translated into real-time audio modulation through a MIDI framework, allowing users to “sculpt” sound and visuals within a virtual bubble. Developed on the Unity platform with custom C# scripts, the system employs mesh deformation and vertex manipulation, resulting in a responsive interface that reacts fluidly to user gestures.

SoundSculpt’s technical prowess lies in its seamless integration of spatial audio, motion tracking, and interactive visuals. The system maps head orientation data to sound pitch and volume, while touch inputs enable further manipulation of the virtual environment. Synesthesia principles enhance the sensory experience, creating a holistic flow that bridges physical and digital art forms. The device is lightweight, water-resistant, and offers up to 5.5 hours of battery life, making it suitable for extended creative sessions.

Research conducted at the MIT Media Lab demonstrated SoundSculpt’s potential for emotional regulation and well-being. User testing with both designers and non-designers revealed enhanced perception of spatial sound and motion, as well as improved emotional stability. These findings suggest promising applications in art therapy and psychiatry, where playful creation can support mental health and emotional balance.

Overcoming challenges in synchronizing multisensory feedback and real-time interaction, Yiqi Zhao’s design stands out for its originality and technical sophistication. The project’s iterative development process addressed limitations in sensor accuracy and data processing, ultimately delivering a fluid, captivating experience that fosters creativity and emotional connection.

SoundSculpt’s recognition with the Iron A’ Digital Art Award in 2025 underscores its impact as a practical, innovative tool that integrates industry best practices. By inviting users to sculpt sound with movement, the project not only redefines music-making but also contributes to a more expressive and emotionally fulfilling world.