Synphony

Mixed reality experience|game|app

Synphony is a mixed-reality interactive system that augments the experience of playing instruments. Inspired by the naturally-occurring phenomenon of synaesthesia, Synphony gives participants the superpower of seeing music in a new way, providing a playground for a creative exploration of harmonies using immersive visuals. The experience introduces beginners to the wonders of musical improvisation by facilitating the first steps and expands the cross-modal perception of music in advanced musicians.

Synphony

Photo by Vivek Menon

Synphony

Synphony being tested as a music education game.

The project also addresses the problem of how technology tends to alienate us from one another more and more. Synphony encourages co-creation of music with multiple people by catalysing a new musical conversation and collaboration. So far, in its early form, the project has been tested by two musicians playing simultaneously on the guitar and the piano, but the ultimate goal is to increase the number of instruments to even accommodate a symphony orchestra of up to a hundred musicians. We could imagine a concert where each musician releases a colourful visual representation of the played tone, key, timbre and velocity that subsequently merges with the rest to create a poetic but also informative visual interpretation like straight from Fantasia, visible in space to both the musicians as well as the audience.

Synphony
Synphony

In its current form, the project utilises a pair of DIY AR glasses through which one of the musicians can enjoy the spatial holograms. Harmonies, individual notes and the key are conveyed through colourful volumetric visuals through which the musicians communicate with each other as well as with the audience through an additional screen. The colours are related to one another based on the harmonies they create (the closer the colours, the more consonant the resulting sound combination). There is also an interactive key recognition script invented by Antek that picks up the tonality in which the musician is playing in real-time. The system’s awareness of the key enables interesting ways of playing with the visual representation of consonance and dissonance, as well as of assisting the musician in harmony exploration through an intuitive UI indicating creative possibilities.