VISUALIZATION AND REVERSIBILITY OF SOUND: THE SOFTWARE SONOS EXAMPLE Jean-Baptiste Thiebaut CICM Université Paris 8 [email protected]

ABSTRACT This paper presents strategies of sound transformation according to the visualization of sound and visual interfaces. Based on previous researches [8][15][16][17] on sound visualization, I present more explicitly the importance of visualization as an intuitive tool or guideline for composers, collaborative musicians [4] and the audience. The concepts of visualizability [8], orthothécité [2] applied to music are exposed. Willing to use - as a composer – visual interfaces close to the insight, I develop the concept of reversible analysis to provide interfaces that transform sound based on the representation of its characteristics. As an example, The software Sonos [17] is described with new features of real-time frequency filtering of two sound sources in order to enhance the perception of both. The use of a video as a parameter controller is presented. Perspectives of sound transformations according to similar processes are presented. 1.

INTRODUCTION

The visualization is a prospective area of research for music and audio effects. In several disciplines such as medicine, physics and mathematics, scientists never ceased to put into images hidden parts of the body, atomic configuration or non-Euclidian geometry. It often allows validating or invalidating theory and always retrieves information on the studied subject. In Arts, visualization takes several forms like painting, photography, cinema and video. Then the visualization is a subjective vision of the artist’s perception. In science, visualization is more often a consensual process to represent theories and hypothesis. According to Arthur I. Miller, [11] “Visualization pertains to visual imagery that we abstract from phenomena that we have actually witnessed in the world of perceptions”.

In both science and art, visualization belongs to the perception. At the edge of visualization, the concept of visualizability is defined by [11]: “the visual imagery to which we are led by the mathematics of a physical theory”

Visualizability of sound can be understood as the efficiency of visualization to represent musical characteristics according to high level physical analysis. Among many works in the area of parameters extraction in the past few years, we can notice the works of [2], [3], [7] and [9]. The next step is to define a framework (or many) to map the extracted parameters to an

intuitive and formal visualization that lead to visualizability. A particular visualization that we call reversible, or orthothétique [1], would allow to transform the visualize sound according to its graphical properties which rely on sound properties. In [1], orthotheticité is defined as “ce qui pose exactement comme sens”, i.e. what project exactly the meaning. In our particular case, the meaning is obviously a musical meaning, and the projection is on a computer screen. If we can obtain an orthothétique representation of music, we can get back to the sound by the reverse process. 2.

ABOUT THE VISUALIZATION

Visualization of sound takes many forms, depending on the process, which can be artistic or scientific. In the case of artistic visualizations, one can talk about visualization of music instead of sound, as most artists aim to represent music rather than sound phenomenon. The image is then a guideline of the musical content. Considering the powerful effect of images on the mind, such representations could have an important impact on the audience. This impact raise several aesthetic and perceptive questions: Why do we want to visualize music ? What do we want to represent? Which strategy use to visualize music: to represent the perception, the composer’s mind, abstract images, non related images as for clips? In most case, due to the power of images, the visual perception takes over the auditory perception. The most important thing to be think about for a musical purpose is how the relation between images and music emphasize the perception of music. In particular it could be useful to use the parameter extraction to build a representation thanks to sound characteristics. Formal representations of perceptive phenomenon can lead to a better understanding of sound and music. Actually, the most used representation of sound is based on the time/frequency/amplitude FFT analysis. Obviously, the FFT alone can’t show anything else than time, frequency and amplitude. Thanks to several algorithms [2], [3], [7] and [9], we can extract some more parameters. The framework that can be used to elaborate new way of representing sound should take under consideration some of Pierre Schaeffer typology’s works, improved by Michel Chion and R. Murray Schaefer [14][6][13]. Similar works have been initiated by Antonio de Sousa Dias at the University Paris 8, and will be investigated in the future by the author. The second interest of visualizing sound within formal representations is to use the result of the analysis as an

interface for controlling sound characteristics through time. As soon as visualization is reversible, it is possible to use this visualization to transform the sound according to graphical actions. Visual interfaces to control sound have been explored for a while, and we can notice among them UPIC (Unité Polyagogique Informatique du Cemamu ) initiated by Iannis Xenakis and developed by engineers [10.] The software Phonogramme and Sonographe by Vincent Lesbros, Metasynth by Eric Wenger and Audiosculpt from Ircam. The concept of reversibility is the point we choose to investigate. In a previous work, the software Sonos [15][16][17], displays a STFT (Short Time Fourier Transform) and a graphical filter to transform graphically sound. As STFT represent frequency, time and intensity it is possible to act on each of these sound parameters. The goal of our research is to define strategies to use a graphical interface that can be used as a transformational tool of sound in order to produce, hopefully, music. 3.

REVERSIBLE VISUALIZATION

Visualization in sciences is a formal representation of phenomena according to scientific rules. In Arts, visualization is the artist’s personal vision of phenomenon. Some video artists use both possibilities to illustrate music. One can also distinguish visualization of sound parameters and visualization of the sound itself. The first help the musicians to produce music thanks to a computer while the second retrieve information to the viewer according to its analysis (sonogram, waveform, wavelet,…). So far, the visualization of music has taken many forms. The waveform is the first reversible representation of sound, used for visualization and manipulation as well. In most software, the user can graphically re-draw the waveform, i.e. redefine the dynamic of the sound. Some synthesis techniques are based on graphical design of waveform. Others useful visualizations are virtual sequencers that allow to sort musical data through time. The visualization can also be used as a score, or a musical guideline with software like Acousmographe that enables to display a sonogram and to annotate it with graphical shapes [8]. The representation drawn in figure 1 has been made with the Acousmographe.

Figure 1. Subjective representation of Sud by J.-C. Risset drawn by Pierre Couprie

The different representations of sound depends on different analysis such as waveform, sonagram, wavelets,... Such representations retrieve perceptive information like amplitude, timbre complexity,

frequency and duration, amongst other. The figure 1 shows a representation that retrieves musical information according to a subjective code. About the analysis of Sud Pierre Couprie said “I realized this graphical representation of Sud by focusing on the perception (what the audience perceive) and not on composition. The different figures have been chosen according to their analogy with the sounds” [5]. This parti pris is a possible strategy used for illustrate a musical content, based on perception and recognition of morphologies of sound. Using visualization to control, transform, represent or sort music has always been used since the computer appeared. Using the graphical representation of the sound (and the graphical properties of the representation) to control sound effects is the purpose of many sound editors mainly according to the waveform. As visualization of sound can be done with better algorithms and can show more information than before (rythm and transients for instance) we could also think about ways to manipulate those representations to transform sound. The first step in the software Sonos has been to modify a windowed sonagram with global transformation such as blur, zoom in and out or rotation. The main problem encountered then is the window size that applies to the effect a particular temporality. In this particular case – real-time transformation -, this problem cannot be avoided due to the fact that the sound has to be represented before being processed. A solution could be explored in a non realtime context, with a large sonogram on which one can apply locally graphical transformations. But, in my opinion, the most prospective and fruitful area of research concerns graphical transformation of sound morphologies instead of graphical transformation of the FFT. 4.

NEW FEATURES OF SONOS

4.1. Background As the music materials increased with techniques of recording and software manipulations, it would be useful to provide more simple interfaces to transform sound and produce music, for composers and for listeners too. The example of Sonos, described in [15], shows how to graphically act on frequency and amplitude but also to use image properties (in this case the R, G, B planes) to define a spectral delay and a spectral feedback. The result is a spectral reverberation that has been used by the author at the theatre Le Châtelet in Paris with the Swedish baryton David Johansson on a reading around the opera Angels in America by Peter Eötvös. Sonos has also been use for the piece ‘charley’ with the French percussionist Ravi Balle. 4.2. Graphical exclusive filtering Always confronting techniques to a practical use and aesthetical needs, I developed new features in Sonos in

order to mix two sound sources in real-time according to an exclusive graphical filter. As a composer, I often use frequency filtering between sound sources to avoid mask effect [12] and to clarify the accumulation of sounds. Based on the transfer matrix used in Sonos, the exclusive filter control a frequency filtering for two sound sources. A drawable interface of 512 points in ordinate represents the frequency and is one point large, i.e. the size of a vector of a STFT. Each point is stretched for a better usage, and can be drawn with a 8bit value between 0 and 1. This value controls the amplitude of a STFT bin of the first sound while its opposite (i.e. 1-value) controls the amplitude of a STFT bin of the second sound.

Some programs exist such as the Max/MSP program called ‘Forbidden Planet’ (given as an example with the software) on which one can also filter a single sound source according to a STFT. But the goal of Sonos is to use the colour planes RGB and all the graphical possibilities, as video, to develop new strategies of sound transformations. 4.3. Future developments There are two main subjects I want to investigate for Sonos. The first is to use the green and blue plans in the context of the exclusive filtering to enhance the possible interactions between the two sound sources. Adaptive effects [18] can be use to provide to a sound the ability to control the other, designing for each frequency a particular behaviour. Such developments will be test in the pplato project [19] with the Finnish guitarist Otso Lähdeoja and the French percussionist Ravi Balle. The second goal for Sonos is to use video strategies to control parameters through time. It is actually possible with Sonos [15] to use a video to control the filter. As the filter is an image of 1 pixel per 512 pixels, one can use a video to replace this filter. The similarity between music and video is that both evolve in time. Then, the control of sound by a video can be considered as an automation that controls a huge quantity of data. Thanks to Jitter (cycling’74), it is possible to transform video through time. An actual problem in the area of interaction between instruments and computer is that the computerist doesn’t have a score that remind him what he had to do. This is a problem for archiving musical pieces and to be able to reproduce them. In Sonos, the interaction of the computerist with the graphical interface can be recorded as a video in real time. Then the video became a very precise score of what has been done, and when it occurs in time. It opens in this context a prospective area of research. 5.

Figure 2. Graphical filtering of two sound sources i n Sonos

In figure 2, one can see that the filter applied to the sound 2 is the ‘negative’ of the filter 1. The new STFTs that will be synthesized are complementary. This way of mixing two sound sources is useful to enhance characteristics of each sound (mainly frequency). Furthermore, the mix of the two sounds will not saturate on any frequency - assuming that none of them is originally saturated -, because the resulting STFT is composed by a proportion of each and can’t exceed the maximum intensity of any of the sound. As in the previous version of Sonos, the process is real-time with a latency of one STFT frame, i.e. 11,6 ms (cf. [16]).

CONCLUSION

The concepts exposed in the first part of this paper visualizability, reversibility, orthothécité – emerged from research works at the Centre de recherches Informatique et Création Musical (CICM) in Paris with Benoît Courribet and Anne Sedes. In the context of visualization of sound and sonification of visual, it has been pointed out that these concepts which come from physics, mathematics and philosophy, should be useful to explore, as they surround our current works. The way it will transform our research and its applications is not define yet, but, for sure, we will try out many ideas derived from these concepts, such as the representation of morphologies of sound and strategies to modify the representation to transform sound according to its morphologies. The software Sonos is a work in progress and aims to provide aesthetical results. The current version is used by the author in the musical context of improvisation and composition. Future works will enhance the use of the video as a musical controller. Interfaces of control of sound in space, developed in [16], will be investigated.

Electroacoustic, electronic and contemporary music are very related to a perceptive and sensitive approach of the sound, a very deep listening of the micro characteristics (morphology) of the sound in terms of frequency, time and volume amongst many others. Visualization of them could be very useful in order to help the listener to understand what’s happening. Furthermore, sounddriven visualization may help musicians in a collaborative musical context. 6.

REFERENCES

[ 1 ] Bachimont, B. (1999). De “l'hypertexte à l'hypotexte : les parcours de la mémoire documentaire. In Technologies, Idéologies, Pratiques”.in Mémoire de la technique et techniques de la mémoire, sous la direction de Charles Lenay et Véronique Havelange), 195-225. [2] Bello, J.-P. Monti G. and Sandler M. « Techniques for automatic music transcription », i n proceedings of International Symposium o n Music Information Retrieval (ISMIR 2000) [ 3 ] Brossier, P., Bello J.-P. and Plumbley. M. D., “Real-time temporal segmentation of note objects in music signals”, in Proceedings of the International Computer Music Conference (ICMC 2004), Miami, Florida, USA, November 1-6, 2004. [4] Bryan-Kinns, N., Healey, P., Thirlwell, M., & Leach, J. (2003a). “Designing for Group Creativity”. Supplementary Proceedings of HCI International 2003. [5] Castanet, P.-A.O.Meston, J.C.Risset, D.Teruggi, P.Couprie. 2001. Portrait polychrome n° 2, JeanClaude Risset. Paris : INA- CDMC [6] Chion, M. 1995. Guide des objets sonores. Paris : Buchet Chastel. [7] Gerhard, D.; D. H. Hepting and M. McKague (2004) “Exploration of the correspondence between visual and acoustic parameter spaces”. Proceedings of the 2004 Conference on New Interfaces for Musical Expression (NIME04), Hamamatsu, Japan, pp 96-99. [ 8 ] Geslin, Y.; A. Lefevre. “Sound and musical representation : the acousmographe software”. Proceedings of the 2004 International Computer Music Conference. Miami: International Computer Music Association. [9] Jehan, T. « Perceptual segment clustering for music description and time-axis redundancy cancellation » in proceedings of International Symposium on Music Information Retrieval (ISMIR 2004) [10] Marino, G.; Serra, M-H and Raczinski, J-M., "The UPIC System: Origins and Innovations", Perspectives of New Music, vol. 31, no. 1, 1993, pp 258-269 [ 1 1 ]Miller, Arthur I., 1996. Insight of Genius. Cambridge : The MIT Press. [ 1 2 Miller, ] D. Language and communication, McGraw-Hill, NY, 1963, pp 54-55.

[13] Schaefer, R. Murray. 1977. The soundscape.Our Sonic Environment and the Tuning of the World. Vermont : Destiny Books, One Park Street, Rochester. [14] Schaeffer, P. 1966. Traité des objets musicaux. Paris : Editions du seuil. [15] Sedes, A; Courribet B. and Thiebaut JB. “From the visualization of sound to real-time sonification : different prototypes in the Max/MSP/jitter environment.” in Proceedings of the International Computer Music Conference (ICMC 2004), Miami, Florida, USA, November 1-6, 2004. [ 1 6 ]Sedes A., Courribet B. and Thiebaut J.-B.: “Egosound, an egocentric interactive and realtime approach of sound space”, in Proceedings of the 6th International Conference of Digital Audio Effects DAFX-03, London, 2003. [17] Thiebaut, J-B. 2004. Sonos. Software.

[ 1 8 ]Verfaille, V., 2003, Effets Audionumériques

Adaptatifs: Théorie, Mise en Oeuvre et Applications en Création Musicale Numérique, PhD dissertation, Université AixMarseille 88

[19] http://pplato.free.fr/