SONOTROPE
Logline
The Sonotrope is a motion-to-music performance tool, a spatial music score creation and playback system for choreography driven sound-artist. It enables one to capture movement as sculpted tracks of sound, resulting in an interactive 3D music score, a spatial crystallised soundscape composition as recorded multi track sound score constellation with a synesthetic visual idiom.
The Sonotrope is a motion-to-music performance tool, a spatial music score creation and playback system for choreography driven sound-artist. It enables one to capture movement as sculpted tracks of sound, resulting in an interactive 3D music score, a spatial crystallised soundscape composition as recorded multi track sound score constellation with a synesthetic visual idiom.
Sonotrope History
The original Sonotrope started as a R&D trajectory at the lectureship of the Netherlands Film Academy (NFA) by Robin Noorda and Jilt van Moorst in 2016. Facilities and two grand were provided by the NFA. In 2017 composer Alfred Marseille joint and the Stimulerings Fonds for the Creative Industry granted funding. The Mondriaan Fonds provided two grands and the most important performance was in 2019 at Cinedans Film Festival in Amsterdam with choreographer Heidi Viertaler in collaboration with dancers Luca Cacitti and Shay Partush. After an music performance at the Overkill Festival in Enschede, the team received an invitation by the New Adventures in Sound Art (NAISA), a Canadian non-profit for experimental sound-art, to do a series of performances, make an installation and conduct workshops using the Sonotrope.
Due to Corona measures the plans, research and development of Sonotrope came to a hold. The project reached a pilot prototype version that was a clear and promising proof of concept.
Recently, after recovering from long covid, Robin Noorda picked up the pieces in order to make a new and better version of the Sonotrope.
The original Sonotrope started as a R&D trajectory at the lectureship of the Netherlands Film Academy (NFA) by Robin Noorda and Jilt van Moorst in 2016. Facilities and two grand were provided by the NFA. In 2017 composer Alfred Marseille joint and the Stimulerings Fonds for the Creative Industry granted funding. The Mondriaan Fonds provided two grands and the most important performance was in 2019 at Cinedans Film Festival in Amsterdam with choreographer Heidi Viertaler in collaboration with dancers Luca Cacitti and Shay Partush. After an music performance at the Overkill Festival in Enschede, the team received an invitation by the New Adventures in Sound Art (NAISA), a Canadian non-profit for experimental sound-art, to do a series of performances, make an installation and conduct workshops using the Sonotrope.
Due to Corona measures the plans, research and development of Sonotrope came to a hold. The project reached a pilot prototype version that was a clear and promising proof of concept.
Recently, after recovering from long covid, Robin Noorda picked up the pieces in order to make a new and better version of the Sonotrope.
The Sonotrope dance performance at Cinedans 2019
Origin and inspiration
Noorda: Sonotrope is an installation and research trajectory by artist, developer and lecturer Immersive Media of the Netherlands Film Academy (NFA), Jilt van Moorst, Sound artist Alfred Marseille and me, synaesthetic artist, installation maker and cinematographer Robin Noorda. We all have a profound fascination for the subset area of sound, image and movement and wanted to incorporate this in an immersive world.
When writing my lecture 'Sound and Image', part of a cross disciplinary workshop for art academy- and music academy students, I collected and researched many sound-scores of Minimal Music and Music Concrète composers as samples of the visually strong and intriguing idioms that emerged as visual soundscapes. Personally, I experienced a connection between the visual attractiveness of a score and the musical quality. During this research, I came across the fascinating 2D scores translated into 3D objects of soundscape Nathalie Miebach. Looking at her sculptures, one would like to be able to listen to the sounds they represent. And so, the basic idea of Sonotrope, an immersive 3D sound score notation and playback system, was born.
Below the work of Nathalie Miebach. She translates her 2D sound-scores of weather and storm based soundscapes into 3D sculptures.
Noorda: Sonotrope is an installation and research trajectory by artist, developer and lecturer Immersive Media of the Netherlands Film Academy (NFA), Jilt van Moorst, Sound artist Alfred Marseille and me, synaesthetic artist, installation maker and cinematographer Robin Noorda. We all have a profound fascination for the subset area of sound, image and movement and wanted to incorporate this in an immersive world.
When writing my lecture 'Sound and Image', part of a cross disciplinary workshop for art academy- and music academy students, I collected and researched many sound-scores of Minimal Music and Music Concrète composers as samples of the visually strong and intriguing idioms that emerged as visual soundscapes. Personally, I experienced a connection between the visual attractiveness of a score and the musical quality. During this research, I came across the fascinating 2D scores translated into 3D objects of soundscape Nathalie Miebach. Looking at her sculptures, one would like to be able to listen to the sounds they represent. And so, the basic idea of Sonotrope, an immersive 3D sound score notation and playback system, was born.
Below the work of Nathalie Miebach. She translates her 2D sound-scores of weather and storm based soundscapes into 3D sculptures.
2D and 3D soundscape scores of Nathalie Miebach, the inspiration of Sonotrope.
The new objective
Recently we prepared to restated the project. Sonotrope 2.0 is not merly the further development of the early version. It will be a revision of the concept and original goals. It means reinventing the instrument and the fundamental principles of it. It probably means rewrite the code from scratch.
Synaesthetic associative principles will be involved in the design of the audio objects. E.g. a relation of pitch and colour is an obvious association. A low sound is represented in a dark warm red and a distant high tone looks blueish.Texture, materials, dynamics and behaviour are other translation domains of sound representation.
First of all the fundamental question has to be answered if any one of the elements motion, sound, colour, image, texture, time and space, or all, will be the source of the Sonotrope manifestation. Does one experiences causality or synchronicity? Can synaesthetism be simulated for non synaesthetic people and, if not, does it trigger a feeling of synchronicity and, if so, does that delivers any experience of relevance?
Directions of manifestations
An initial promising range of applications in a spectrum of different domains has emerged that have to be explored and judged for their potential. Sonotrope is a system or instrument that can thrive in a diverse array of applications. The range of seven utilisations of the Sonotrope described below are to be explored and examen as potential directions of development. More directions to emerge are very likely. The first three applications might have priority, though future developments and influences will determine directions.
1) Primary aim is further develop the performance based Sonotrope instrument to a better and versatile performance tool and idiom. An instrument to make a synaesthetic on-stage audible drawing or sculpting of a spatial music score. It is a movement based music score sculpture including synthesiser/sampler/sequencer or multitrack-recorder that enables to real time and intuitive create, edit an replay.
2) An immersive dance film based on a cinematographic translation of a Sonotrope performance in a magic realism world of visual and tangible sound, in which the synaesthetic relation of the two contributes to a recognisable idiom.
3) Another implementation for the system might be an AR installation in the public space where visitors can play the music constellation with their mobile device and add, alter or delete parts of the never finished, dynamic and evolving music composition.
4) A potential important and interesting use for the system is in the medical field.
There are clear indications and interest for the Sonotrope system to implement as a friendly and fun therapy for autism.
5) Another route is the adaptation of the Sonotrope concept by the Virtual Reality Academy Amsterdam into their curriculum enabling students to develop concepts for the instrument or develop the concept of the open source application further.
6) An obvious option is an AR/VR online multi-user application in general release that allows users to experience the Sonotrope and create and share their creations with a Sonotrope community.
7) Finally, we aim at releasing the Sonotrope as an open source application in order to sustain an evolving lifespan of the Sonotrope.
1) Sonotrope as performance instrument
The ultimate theatre for the Sonotrope carousel would be a 180º plus projection space in the NXT Museum or ultimatly the new to be build LED dome planetarium of Artis. Implementation of the 4DSound system would be nessesary.
The Sonotrope could be seen as an instrument. A multi disciplinary instrument. So the performance concept depends on users that would be willing to apply cross discipline assets as in Synaesthecy. That might be composers with an interest in motion, colour and shapes or dancers, performers, sculptors and painters with a musical talent or ambition. The Sonotrope might also be described as an instrument that works like a synaesthecy simulator.
Creation of a score is done by selecting sound aspects by means of a controler and drawing 3D soundtracks in a 360º space. The combination of soundtracks form a slowly turning constellation with a playback meridian or a playback membrane. Each part of the floating constellation that hits the membrane, will playback the sound it represents. One can edit, add or erase parts of the constellation. Multi users can simultaneous create, alter and playback the dynamic composition. It is so to say an immersive composing orchestra that creates and plays a Synaesthetic sculpture, projected around the public.
The colourful objects of the music score constellation represent single musical notes as geometric shapes, glissando's as tubes and samples as lathed 3D waves. The colour of the notes stands for the pitch of the sound or other synaesthetic harmonic associations.
The composer can instals a kind of playback meridian or membrane by means of placing sensors. The sensors are physical on stage and the subsequent playback membrane is also represented in the projection. The growing and slowly turning music composition constellation is now being played back when the tone representing objects of it cross the playback membrane.
The composer builds a projected sound/music score sculpture that is being played back while growing.
The 2019 Cinedans prototype try-out performance was the first proof of concept with an audience. The renown choreographer Heidy Vierthaler, who worked with William Forsythe, was trusted with the choreography for Sonotrope together with the dancers Luca Cacitti and Shay Partush while Tropism member Alfred Marseille and coder and synthesiser wizard Jilt van Moorst both performed as sound artists besides me as synaesthetic visual artist.
2) Sonotrope movie
A possible movie synopsis might be a cinematographic, perhaps set in 360º, of the Sonotrope instrument. It offers an intriguing array of possibilities to emphasise the immersive visual music world of surround-sound, shape, colour, texture and movement in a magic realistic way.
It starts as a 360 registration of a Sonotrope theatre performance. Slowly the spectator gets more involved and the Sonotrope sculpture constellation expands into a real surrounding sound-land-scape world with waving hills and dancing candy like trees. Even the dancers themselves turn into furry creatures, flocking abstractions or emitting particle systems.
At the end we are back in the theatre, the dancers bow for the public but still do have the colourful and furry costumes.
Recently we prepared to restated the project. Sonotrope 2.0 is not merly the further development of the early version. It will be a revision of the concept and original goals. It means reinventing the instrument and the fundamental principles of it. It probably means rewrite the code from scratch.
Synaesthetic associative principles will be involved in the design of the audio objects. E.g. a relation of pitch and colour is an obvious association. A low sound is represented in a dark warm red and a distant high tone looks blueish.Texture, materials, dynamics and behaviour are other translation domains of sound representation.
First of all the fundamental question has to be answered if any one of the elements motion, sound, colour, image, texture, time and space, or all, will be the source of the Sonotrope manifestation. Does one experiences causality or synchronicity? Can synaesthetism be simulated for non synaesthetic people and, if not, does it trigger a feeling of synchronicity and, if so, does that delivers any experience of relevance?
Directions of manifestations
An initial promising range of applications in a spectrum of different domains has emerged that have to be explored and judged for their potential. Sonotrope is a system or instrument that can thrive in a diverse array of applications. The range of seven utilisations of the Sonotrope described below are to be explored and examen as potential directions of development. More directions to emerge are very likely. The first three applications might have priority, though future developments and influences will determine directions.
1) Primary aim is further develop the performance based Sonotrope instrument to a better and versatile performance tool and idiom. An instrument to make a synaesthetic on-stage audible drawing or sculpting of a spatial music score. It is a movement based music score sculpture including synthesiser/sampler/sequencer or multitrack-recorder that enables to real time and intuitive create, edit an replay.
2) An immersive dance film based on a cinematographic translation of a Sonotrope performance in a magic realism world of visual and tangible sound, in which the synaesthetic relation of the two contributes to a recognisable idiom.
3) Another implementation for the system might be an AR installation in the public space where visitors can play the music constellation with their mobile device and add, alter or delete parts of the never finished, dynamic and evolving music composition.
4) A potential important and interesting use for the system is in the medical field.
There are clear indications and interest for the Sonotrope system to implement as a friendly and fun therapy for autism.
5) Another route is the adaptation of the Sonotrope concept by the Virtual Reality Academy Amsterdam into their curriculum enabling students to develop concepts for the instrument or develop the concept of the open source application further.
6) An obvious option is an AR/VR online multi-user application in general release that allows users to experience the Sonotrope and create and share their creations with a Sonotrope community.
7) Finally, we aim at releasing the Sonotrope as an open source application in order to sustain an evolving lifespan of the Sonotrope.
1) Sonotrope as performance instrument
The ultimate theatre for the Sonotrope carousel would be a 180º plus projection space in the NXT Museum or ultimatly the new to be build LED dome planetarium of Artis. Implementation of the 4DSound system would be nessesary.
The Sonotrope could be seen as an instrument. A multi disciplinary instrument. So the performance concept depends on users that would be willing to apply cross discipline assets as in Synaesthecy. That might be composers with an interest in motion, colour and shapes or dancers, performers, sculptors and painters with a musical talent or ambition. The Sonotrope might also be described as an instrument that works like a synaesthecy simulator.
Creation of a score is done by selecting sound aspects by means of a controler and drawing 3D soundtracks in a 360º space. The combination of soundtracks form a slowly turning constellation with a playback meridian or a playback membrane. Each part of the floating constellation that hits the membrane, will playback the sound it represents. One can edit, add or erase parts of the constellation. Multi users can simultaneous create, alter and playback the dynamic composition. It is so to say an immersive composing orchestra that creates and plays a Synaesthetic sculpture, projected around the public.
The colourful objects of the music score constellation represent single musical notes as geometric shapes, glissando's as tubes and samples as lathed 3D waves. The colour of the notes stands for the pitch of the sound or other synaesthetic harmonic associations.
The composer can instals a kind of playback meridian or membrane by means of placing sensors. The sensors are physical on stage and the subsequent playback membrane is also represented in the projection. The growing and slowly turning music composition constellation is now being played back when the tone representing objects of it cross the playback membrane.
The composer builds a projected sound/music score sculpture that is being played back while growing.
The 2019 Cinedans prototype try-out performance was the first proof of concept with an audience. The renown choreographer Heidy Vierthaler, who worked with William Forsythe, was trusted with the choreography for Sonotrope together with the dancers Luca Cacitti and Shay Partush while Tropism member Alfred Marseille and coder and synthesiser wizard Jilt van Moorst both performed as sound artists besides me as synaesthetic visual artist.
2) Sonotrope movie
A possible movie synopsis might be a cinematographic, perhaps set in 360º, of the Sonotrope instrument. It offers an intriguing array of possibilities to emphasise the immersive visual music world of surround-sound, shape, colour, texture and movement in a magic realistic way.
It starts as a 360 registration of a Sonotrope theatre performance. Slowly the spectator gets more involved and the Sonotrope sculpture constellation expands into a real surrounding sound-land-scape world with waving hills and dancing candy like trees. Even the dancers themselves turn into furry creatures, flocking abstractions or emitting particle systems.
At the end we are back in the theatre, the dancers bow for the public but still do have the colourful and furry costumes.
Moodboard illustration of a candy like sound-land-scape for the Sonotrope movie.
Samples of motion capture imagery as illustrations for possible dance-locomotion manifestations.
3) Sonotrope as AR installation
The Sonotrope installation enables anyone with the special AR app on his/her mobile device to encounter physically in space an augmented music constellation. Searching and moving with your phone you will see the constellation and when interacting with elements these will release their sounds on collision. It is like moving a kind of meridian playback head of a recorder along the tracks.
An ideal location would e.g. be the hall of the Music Building in Amsterdam. When visitors enter the building they can see the colour-full installation floating ans slowly turning between and above the people.
In a later stage of development of the app, a possible array of features might enable the creation of new Sonotropes at specified local areas like in parks. A user could start making a Sonotrope in a variation of three modes: private, listen only or edit and interact. In the later mode the Sonotrope constellation is visible and accessible for anyone with the app in the neighbourhood. A new kind of music capturing sort of Thai Chi might emerge in parks. In the mode of edit and interact clearly, a new way of social-music encounters in will develop. A new way to socialise in the park instead of walking the dog.
4) Sonotrope as therapy for autism
We had contact with Albert Rizzo, PhD Director, Medical Virtual Reality - Institute for Creative Technologies, Research Professor - Dept. of Psychiatry and School of Gerontology, University of Southern California.
After seeing one of our very crude prototypes, we have received an invitation by him to work together with VR researchers at USC in order to use the Sonotrope for a new type of autism VR therapy.
As shown in his letter of intent, he is eager to implement the Sonotrope in therapy.
VR Music therapy has proven to have a positive effect on patients with autism. Rizzo, who is specialised in VR therapies, therefor foresees great results using the Sonotrope as therapy for autism. This is a very potential and promising trajectory which we would like to pick up.
5) Sonotrope in education like the Virtual Reality Academy
The possible adaptation of the Sonotrope immersive concept by the Virtual Reality Academy (VRA), part of the HKA, or other educational institutes into their curriculum might be a clear sign of support and recognition. The VRA can facilitate, stimulate development and provide workshops with students of practically all academies of the Amsterdam School of Arts like the Conservatorium, Academy of Dance and Theatre, Academy of Architecture and The Netherlands Film Academy. The Sonotrope system suits perfectly to be the base for cross-disciplinary workshops. As a platform it would enable a jet unknown range of promising applications made by unexpected combinations of students producing new and unseen idioms.
The workshops are in line with the principle to offer the Sonotrope as open source software to be developed further into customised applications.
6) Online multi-user app
There are many possible implementations of an online application in which the Sonotrope can manifest itself. The principal capabilities might be: playback existing Sonotrope creations, explore them, edit and create one yourself or with others. An improvisation jam with a group implies certain potential. It might evolve into a creative social music medium.
7) Open source
After the basic development and a range of performances and a successful installation, we would like to give the Sonotrope its freedom to spread out its wings and explore the immersive or virtual world out there without control in order to sustain its evolutionary lifespan.
The Sonotrope installation enables anyone with the special AR app on his/her mobile device to encounter physically in space an augmented music constellation. Searching and moving with your phone you will see the constellation and when interacting with elements these will release their sounds on collision. It is like moving a kind of meridian playback head of a recorder along the tracks.
An ideal location would e.g. be the hall of the Music Building in Amsterdam. When visitors enter the building they can see the colour-full installation floating ans slowly turning between and above the people.
In a later stage of development of the app, a possible array of features might enable the creation of new Sonotropes at specified local areas like in parks. A user could start making a Sonotrope in a variation of three modes: private, listen only or edit and interact. In the later mode the Sonotrope constellation is visible and accessible for anyone with the app in the neighbourhood. A new kind of music capturing sort of Thai Chi might emerge in parks. In the mode of edit and interact clearly, a new way of social-music encounters in will develop. A new way to socialise in the park instead of walking the dog.
4) Sonotrope as therapy for autism
We had contact with Albert Rizzo, PhD Director, Medical Virtual Reality - Institute for Creative Technologies, Research Professor - Dept. of Psychiatry and School of Gerontology, University of Southern California.
After seeing one of our very crude prototypes, we have received an invitation by him to work together with VR researchers at USC in order to use the Sonotrope for a new type of autism VR therapy.
As shown in his letter of intent, he is eager to implement the Sonotrope in therapy.
VR Music therapy has proven to have a positive effect on patients with autism. Rizzo, who is specialised in VR therapies, therefor foresees great results using the Sonotrope as therapy for autism. This is a very potential and promising trajectory which we would like to pick up.
5) Sonotrope in education like the Virtual Reality Academy
The possible adaptation of the Sonotrope immersive concept by the Virtual Reality Academy (VRA), part of the HKA, or other educational institutes into their curriculum might be a clear sign of support and recognition. The VRA can facilitate, stimulate development and provide workshops with students of practically all academies of the Amsterdam School of Arts like the Conservatorium, Academy of Dance and Theatre, Academy of Architecture and The Netherlands Film Academy. The Sonotrope system suits perfectly to be the base for cross-disciplinary workshops. As a platform it would enable a jet unknown range of promising applications made by unexpected combinations of students producing new and unseen idioms.
The workshops are in line with the principle to offer the Sonotrope as open source software to be developed further into customised applications.
6) Online multi-user app
There are many possible implementations of an online application in which the Sonotrope can manifest itself. The principal capabilities might be: playback existing Sonotrope creations, explore them, edit and create one yourself or with others. An improvisation jam with a group implies certain potential. It might evolve into a creative social music medium.
7) Open source
After the basic development and a range of performances and a successful installation, we would like to give the Sonotrope its freedom to spread out its wings and explore the immersive or virtual world out there without control in order to sustain its evolutionary lifespan.
Principle of the Sonotrope and Synaesthesia
The basic idea is to develop a system or instrument enabling a sound artist to build in an intuitive way a music annotation sculpture in AR space. It is being built by spacial soundtracks that can be played back instantly like a multitrack recorder. A fundamental aspect is to simulate the synaesthetic experience to both synaesthetic and non synaesthetic people.
Robin Noorda: "I am a synaesthete. When I was a six-year-old kid, I said to a young woman: What a beautiful light blue voice you have, my father has a dark brown one.”
It are these synaesthetic interdisciplinary associations that are still a guide in my film and installation works. I personally see a clear connection between synaesthesia and synchronicity, aleatory and synchresis. The Sonotrope system / instrument appears to be the ultimate result of this synaesthetic cross-fertilisation.
A basic feature we would like to implement in the Sonotrope is an intuitive user interface. One way to avoid a lot of menus and tools is an associative approach. We think there are many self-evident and axiomatic ways to make things aphoristic using synaesthetic associations. One such approach is to superimpose the spectrum of sound to the spectrum of light in a synaesthetic way. Both are waves, one in an array of frequencies we can hear and the other in frequencies we can see. An obvious way of linking the two is to aline the low frequencies; red light and low pitched sound all the way up to the high frequencies; blue light and high pitched sounds. The association with the colour blue in a landscape painting is distant. Blue light diffracts easier and therefore the sky is blue. The high-frequency wavelength of blue is associated with a cool temperature. A high pitched sound diffracts easier as well and is often associated with a coolish blue colour. The association with the colour red, a low frequency in the visible light spectrum, is warm and close by. A warm close sound is also low pitched. Low-frequency light and sound both have more energy to travel further as the higher frequencies.
So we present the sounds with colours accordingly to the superimposed spectrum of visible light to that of hearable sound. The result is that you can see the pitch range of sounds, represented by objects in the music constellation, by their colour.
Another way of synaesthetic association of colour and tone is a colour-code within an octave. Most synaesthetic scientists and composers used this harmonic approach and came up with quite similar colour schemes. In the Sonotrope the user is also able to address the 12 tones of the octave with his/her own colour scale associations.
The shapes of the sounds represent an associative aspect of the sounds as well. The colourful objects represent single musical notes as geometric shapes. Cubes represent block wave based sounds, spheres represent sinus based sounds and pyramids stand for sawtooth based sounds. Glissando's and long tones are represented as tubes and samples as lathed 3D waves. Textures on the objects are generated by interference, modulation and harmonics.
Materials, behaviour and looks like copper, wood, fur, neon, lightning, vibrate and moiré pattern complete the textures and character of the sound shapes.
The superimposed spectra of hearable sound and visible light.
The synaesthetic colour spectra within an octave by a range of composers, artists and scientists.
Sonotrope audio performance at the Overkill Festival
The first Sonotrope sound-art piece was performed at the Overkill Festival 2019 in Enschede by Alfred Marseille, Jilt van Moorst and Robin Noorda. The Sonotrope installation acts as 'motion to music sculpture' and controls some modular synthesizers.
Below: Sonotrope at the Overkill Festival
The internationally renowend media artist Vesna Petresin using the Proto Sonotrope in de VR studio of the Netherlands Film Academy.
Above: Jilt van Moorst demonstrating the very first rudimentary version in 2017 of the Sonotrope VR music sculpting tool. He draws a color-tone related soundtrack in the air with his right tilt brush and does a scratchy playback with his left tilt brush.
Below: sound artist Alfred Marseille and choreographer Susanne Ohmann try the first rudimentary prototype of Sonotrope in 2017. The constellation looks like floating meatballs and sounds like a pinball machine. It is a first user test proving only the principle, so please do not pay any attention to the look and sounds, as they are not designed and composed yet.
Below: sound artist Alfred Marseille and choreographer Susanne Ohmann try the first rudimentary prototype of Sonotrope in 2017. The constellation looks like floating meatballs and sounds like a pinball machine. It is a first user test proving only the principle, so please do not pay any attention to the look and sounds, as they are not designed and composed yet.
Above: Dick Rijken of the STEIM Foundation drawing the first soundtracks of Sonotrope.
Below: Very early design sketches:
Below: Very early design sketches:
