Augmented Listening

By Tue Haste Andersen - October 9, 2012

reBlogged from: design mind
 

Stop for a second and listen. Close your eyes, use your ears, and just listen.

Whether you are in a quiet office environment or out on a busy street, you'll be amazed by how many sounds there are around you. Most of us do not pay attention to the ambient sounds that surround us. Our brains filter them out and we don't listen. Yet the sounds we miss can be very enjoyable.

Designed Sounds

Today, what we hear in our daily lives is often designed sound- music and sound effects carefully crafted for games, devices, and products. For example, mission-critical products, such as heart rate monitors used during medical surgery or a plane’s flight deck controls, use distinctive alarming sounds that are designed to be easy to perceive and raise a sense of urgency or danger.

In interfaces for everyday tasks, sound is used to create engaging and beautiful experiences. Sounds can generate a special feeling or underline brand identity while simultaneously providing cues that a command has been received by the system. Most smart phones today come with subtle sounds that indicate the pressing of a touch screen’s virtual buttons. Since there is no way to feel if a virtual button has been pressed, the sounds reinforce the action for the user. Another example can be found in industrial design, where the latest electric cars are being designed with artificial motor sounds. The sounds alert pedestrians to the car as well as reinforce the sense of driving a powerful vehicle. These examples underline the overall trend of sound being used to create an aesthetic experience rather than serving as purely a functional aid to improve interaction.

Blurring the Border Between Listening and Composition

While systems and products are becoming more enjoyable and pleasant to listen to, they are usually not intentionally designed for sound interaction. The emergence of accessible music software on computers and mobile devices is changing this. These programs allow for easy modification of sound by the average user and blur the border between listening and sound creation. The small form and limited complexity of mobile interfaces has forced music software designers to reduce the complexity of their products, resulting in music software that is widely used by average mobile phone users.

Music apps are often top sellers. Popular applications allow people to become mobile DJs, to transform sounds, and to design ringtones.

I was interested in exploring the blur between sound creation and listening when my friend and colleague Matteo Penzo put me in contact with Matteo Milani from the U.S.O. Project sound art group. The ideas and compositions of the U.S.O. Project revolve around the use of noise and ambient sound as a foundation for sound installations and music composition. Together we wanted to create a mobile experience that would support active listening to the everyday sounds that surround us, making the listener a part of a personal sound installation. Instead of creating a tool for recording and transforming sound, we wanted to start from the sounds themselves. Our goal was to reinforce the sounds of the listener’s environment while blending them with more musical sounds. Together the sounds would form a unique experience that could be enjoyed by anybody that has an interest in sound and art.  

Early Experiments

We started with a small prototype app for iOS using simple sound algorithms to blend U.S.O. music with live recording from the iPhone microphone. The prototype was tested with real use cases that included listening to the app while taking a long walk as well as while sitting at the computer in the office. We added many parameters for the user to be able to tweak and play with the sound transformation.The parameters were mapped to on-screen sliders and buttons and to sensors like the accelerometer.

While doing the informal tests we found that the users were struggling to understand the relationship between the parameters and the sound output. Also, in most cases they would end up spending time experimenting with the parameters to discover how they work. The visual interface and controls were clearly distracting, taking attention away from the app’s original goal of reinforcing ambient sounds for the listener.  

Following these early experiments, we decided to take a drastically different approach. We limited the visual interface as much as possible and provided a set of sound themes in the app for the listener to select. This worked much better. All of a sudden the users would pick up the app and, once started, would tuck it away in a pocket while listening to the sounds. Each theme takes sounds from the microphone and blends them with sounds composed by U.S.O. Project. The sounds are blended using sound algorithms, unique to each theme. Each algorithm is carefully calibrated to replicate the work and skill that goes into producing a great listening experience.

Lis10er

The result is Lis10er (pronounced Listener), an augmented sound installation app. Sounds are blended from the listener’s surroundings, creating dynamic music that changes while maintaining its identity. Lis10er provides users with a creative way of listening to their environment and a unique experience with every listen. 

[www.lis10er.com]

Tue Haste Andersen is Senior Software Architect based in frog’s Milan studio. Tue is a Human Computer Interaction and Computer Music expert, with research ranging from DJ work practices to the use of sound and music in common interaction tasks. He is also the founder and original author of the popular open source DJ software, Mixxx.

Open Sound Control (OSC) for Kyma

OpenSound Control is an open, transport-independent, message-based protocol developed for communication among computers, sound synthesizers, and other multimedia devices.

OSC is often used as an alternative to the 1983 MIDI standard, where higher performance, higher resolution and a richer musical parameter space is desired. OSC messages are commonly transported across the internet and within home and studio subnets using (UDP-IP, Ethernet).

The advantages of OSC over MIDI are primarily speed and the comparative ease of specifying a symbolic path, as opposed to specifying all connections as 8-bit numbers. 

Symbolic Sound Corporation has expanded the list of devices and applications that can communicate with its Kyma sound design environment by adding support for OSC to control parameters of Kyma sound synthesis and processing algorithms. OSC communication requires the Paca or Pacarana sound engine.

Using Open Sound Control, users can interact with Kyma using a variety of real-time controllers and software that offer higher resolution and faster update rates than can be achieved using traditional, standard MIDI controllers, such as the newly announced Apple iPad (and compatible iPhone and iPod Touch), the award-winning JazzMutant Lemur multi-touch surface, among the others.

OSC-enabled Kyma X.74 is a free software update for registered Kyma X owners.

[Jazz Mutant Lemur with Kyma]

Wrapping Other Protocols Inside OSC: MIDI over OSC

Symbolic Sound's protocol for sending bi-directional MIDI streams over OSC

Users can make good use of one of the following applications for sending MIDI over OSC to Kyma (removing the need for a MIDI interface on the computer):

• Andy Capon's OSCKymaMIDI (for Max and Max for Live)
• Delora's PacaMidi
• Camille Troillard's OSCulator

[Read more: Open Sound Control (OSC) for Kyma]
[Listen U.S.O. Project exploring the SlipStick Synthesis in Kyma]
[See Kyma at these upcoming events!]

Generative Music: an interview with Peter Chilvers

by Matteo Milani, U.S.O. Project, September 2009

Generative music is a term popularized by Brian Eno to describe music that is ever-different and changing, and that is created by a system (Wikipedia). I recently had the chance to interview musician and software designer Peter Chilvers, who created the new iPhone/iPod touch application called Air (© Opal Ltd).
Based on concepts developed by Brian Eno, with whom Chilvers created Bloom, Air assembles vocal (by Sandra O'Neill) and piano samples into a beautiful, still and ever changing composition, which is always familiar, but never the same.

Air features four ‘Conduct’ modes, which let the user control the composition by tapping different areas on the display, and three ‘Listen’ modes, which provide a choice of arrangement. For those fortunate enough to have access to multiple iPhones and speakers, an option has been provided to spread the composition over several players.

"Air is like Music for Airports made endless, which is how I always wanted it to be." - Brian Eno

“About 20 years ago or more I became interested in processes that could produce music which you hadn’t specifically designed. The earliest example of that is wind chimes. If you make a set of wind chimes, you define the envelope within which the music can happen, but you don’t precisely define the way the music works out over time. It’s a way of making music that’s not completely deterministic.” - Brian Eno

[via apple.com]

Matteo Milani: Thanks for your time. Peter Chilvers as a musician first: few words about 'A Marble Calm' project.

Peter Chilvers: I happened across the phrase 'A Marble Calm' on holiday a few years ago, thought it sounded like an interesting band name, then started thinking about the type of band that might be. The more I thought about it, the more it seemed to tie up a number of ideas that were interesting to me: drifting textural ambient pieces, improvisation and song. By making it a loose collective, it's enabled me to bring in other vocalists and musicians I've enjoyed working with on other projects - vocalists Sandra O'Neill (who also worked with me on 'Air' for the iPhone) and Tim Bowness, marimba player Jon Hart and flautist Theo Travis.


MM: When did you start working with generative music?

PC: In the 90's I worked as a software developer on the 'Creatures' series of games. When we started on Creatures 2, I was given the opportunity to take over the whole soundtrack. The game wasn't remotely linear - you spent arbitrary amounts of time in different locations around an artificial world, so I wanted to create a soundtrack that acted more as a landscape. I ended developing a set of 'virtual improvisers', constantly generating an ambient soundscape in the background - it was quite involved actually, with its own simple programming language, although little of that was visible to the user.

[...] Peter chose to use his background in improvised music to create an array of "virtual musicians" that would play along to the action on screen. Each composition in Creatures contains a set of "players", each with their own set of instructions for responding to the mood of the norns on screen.

Peter was able to generate much more interesting effects using recorded instruments rather than using General MIDI sounds generated by a soundcard, which can often be quite restrictive. This meant that he could take advantage of the many different ways that a note on a "live" instrument can be played - for example, on a guitar the sound changes greatly depending on the part of the finger used to strike a string, and on a piano when one note is played, all the other strings vibrate too. Also by altering the stereo effects, he could fatten the sound at certain times.

He also made use of feedback loops within the soundtrack. Feedback loops were first experimented with in the 1970s - if any of you can remember Brian Eno, you may be interested to know he composed most of his music then using this method. The idea is that you play a track and record it into RAM (onto a tape back in the 1970s). After about a short while (around 8 seconds in Creatures 2), the loop starts and the original sounds are played back so the composer carries on creating sounds in response to what's gone before.

Behind the scenes, scripts control the music engine and set the volume, panning and interval between notes as the mood and threat changes.

[via gamewaredevelopment.co.uk]

MM: Why did you choose the Apple platform to develop the applications?

PC: I've been a huge fan of Apple products for a long time, and their timing in releasing the iPhone couldn't have been better. Bloom actually existed in some form before the iPhone SDK was announced - possibly before even the iPhone itself was announced. From the second we tried running the prototype, it was obvious that it really suited a touch screen. And Apple provided one!

The difficulty developers have faced with generative music to date has been the platform. Generative music typically requires a computer, and it's just not that enjoyable to sit at a computer and listen to music. The iPhone changed that - it was portable, powerful and designed to play music.

MM: Who designed the visualizations of Bloom? Eno himself?

PC: It was something of a two way process. I came up with the effect of circles expanding and disappearing as part of a technology experiment - Brian saw it and stopped me making it more complex! Much of the iPhone development has worked that way - one of us would suggest something and the other would filter it, and this process repeats until we end up with something neither of us imagined. Trope, our new iPhone application went through a huge number of iterations, both sonically and visually before we were happy with it.

MM: What kind of algorithms define Bloom's musical structure? Are they specifically based on Brian's requests or just an abstraction based on his previous works?

PC: Again, this is something that went back and forth between us a number of times. As you can see, anything you play is repeated back at you after a delay. But the length of that delay varies in subtle, but complex ways, and keeps the music interesting and eccentric. It's actually deliberately 'wrong' - you can't play exactly in time with something you've already played, and a few people have mistaken this for a bug. Actually, it was a bug at one point - but Brian liked the effect, and we ended up emphasising it. "Honour they error as a hidden intention" is something of a recurring theme in Brian's work.
A forthcoming update to Bloom adds two new 'operation modes', one of which was designed specifically to work with the way Brian prefers playing Bloom.

MM: Does the graphic and audio engine include audio and video standard libraries or you wrote your own classes?

PC: I've built up my own sound engine, which I'm constantly refining and use across all the applications. It went through several fairly substantial rewrites before I found something reliable and reusable.

MM: Is all the code in 'Objective C' or did you use any external application?

PC: It's all Objective-C. I hadn't used the language before, although I'd worked extensively in C++ in the past. It's an odd language to get used to, but I really like it now.


MM: Is Bloom sample based? What is music engine actually controlling (e.g. triggering, volume, panning, effects)? What about the algorithmic side of the music engine?

PC: Bloom is entirely sample based. Brian has a huge library of sounds he's created, which I was curating while we were working on the Spore soundtrack and other projects. It's funny, but the ones I picked were just the first I came across that I thought would suit Bloom. We later went through a large number of alternatives, but those remained the best choices.

The version of Bloom that's currently live uses fixed stereo samples, but an update we're releasing soon applies some panning to the sounds depending on the position of each 'bloom' on screen. It's a subtle effect, but it works rather well.

MM: Would you like to describe your actual and next projects?

PC: I've been involved in two new applications for the iPhone: Trope and Air. Both Apps were intended to be released simultaneously. Trope is my second collaboration with Brian Eno, and takes some of the ideas from Bloom in a slightly different, slightly darker direction. Instead of tapping on the screen, you trace shapes and produce constantly evolving abstract soundscapes.

Air is a collaboration with Irish vocalist Sandra O'Neill, and is quite different to Bloom. It's a generative work centred around Sandra's vocal textures and a slowly changing image. It draws heavily on techniques that Brian has evolved over his many years working on ambient music and installations, as well as a number of the generative ideas we've developed more recently.

I have just had some interesting news: Trope has been approved, it's now available in the App Store!

More information can be found at www.generativemusic.com.

"Trope is a different emotional experience - more introspective, more atmospheric. It shows that generative music, as one of the newest forms of sonema, can draw on a broad palette of moods." Brian Eno

[all iPhone and iPod touch applications by Opal Limited - App Store link]
[burningshed.com]

[Brian Eno discussing Generative Music at the Imagination Conference, 1996]

UPDATE: Trope in action!

"[...] I had realised three or four years ago that I wasn't going to be able to do generative music properly – in the sense of giving people generative music systems that they could use themselves – without involving computers. And it kind of stymied me: I hate things on computers and I hate the idea that people have to sit there with a mouse to get a piece of music to work. So then when the iPhone came out I thought: oh good, it's a computer that people carry in their pockets and use their fingers on, so suddenly that was interesting again." - Brian Eno
[via timeoutsydney.com.au]

Related Post: Deep Green: sound design for iPhone App

About 'Haptics'

What is Haptics?

"Haptics" comes from the Greek word Hapesthai, meaning "the science of touch."

The world of haptics is expansive by definition. It is the field of science and technology dedicated to tactile sensation, and it has applications for everything from handheld electronic devices to remotely operated robots. Yet outside of the research and engineering community, it is a virtually unknown concept. “People don’t even recognize the word ‘haptics’ yet,” says Ralph Hollis, director of the Microdynamic Systems Laboratory. “You have to spell it for them.”

In an age of digital devices that stimulate and amaze the eyes and ears with increasingly high fidelity, haptics has been employed mostly in relatively un­sophisticated applications—rumbling video-game controllers and buzzers that alert you to a cellphone call. But as our digital tools have become more complex and capable, our interfaces with these devices are beginning to run into the limitations of sight and sound. “It’s really only now that we’re seeing a migration from keyboards and mechanical switches to touchscreens and touch-sensitive surfaces,” said Immersion, a company that produces haptic interfaces. “We’re losing that tactile feel that we had before, and now we’re trying to bring it back.”

Touchable Technology

Moving haptics out of the lab can be challenging. Tactile feedback in consumer electronics must be both convincing to the user and appropriate for the device. Broadly speaking, touch can be divided into cutaneous sensing through the skin surface (feeling the pebbly surface of a basketball), and deeper kinesthetic sensing from muscles and tendons (experiencing the impact when hitting a ball with a bat). But much of the recent haptic development in consumer electronics has focused on fooling the fingertips into feeling onscreen buttons that aren’t physically there.

General consumers will first encounter haptics on these touchscreen gadgets and desktop controllers, but the most sophisticated touch technology outside of the lab is found in industrial, military and medical applications.

[read more - via popularmechanics.com]

Max for Live

After two years of development, Cycling '74 and Ableton have finally revealed their integration project for extending Live using Max. Don't miss David Zicarelli's perspective on the Max for Live project, and read about the new Max tools for building Live devices. Here's the press release.

[update: Create Digital Music - Make Max Patches that Integrate with Ableton]

Ableton proudly presents...

[ableton.com/extend]

And, if you'll be in Los Angeles, please take note of the following:

Touch Controlling Ableton Live Seminar presented by Lemur experts Gareth Williams (aka Raw Hedroom) and Bryant Place (aka CPU) focusing on the unique interfacing capabilities of the JazzMutant Multitouch Controller with Ableton Live.

Topics include:

• "Performing a powerful hybrid live/DJ set with the Lemur"
• "Using the Lemur with Live as a powerful sound design tool"
• "Natural and musical sequencing with Lemur and Live"

Ableton Live User Group-Los Angeles
January 22, 2009 - 8 p.m.
SAE Institute | DFC Theater | 6565 Sunset Blvd., Suite 100
Los Angeles, CA 90028

[read more - pdf]

Jazzmutant Lemur V2 on the blogosphere

The public beta of the v2 firmware update will launched by Jazzmutant at AES (October 3rd to 5th) with the full version released Q4 2008. The full update will be free.

Brand new features:

Breakpoint object: multi-segment envelope editor
Gesture object: trackpad emulation with advanced gesture recognition
Alias: memory and time saver
Tabbed Container
Mouse and keyboard control (!)
Improved and new-look Jazzeditor
New multi-line script pane

[Sonic State]
[Harmony Central]
[Mac Music]
[Gearjunkies]

TouchOSC

TouchOSC is an iPhone/iPod Touch application that lets you send and receive Open Sound Control messages over a Wi-Fi network using the UDP protocol.

Availabe on the AppStore now.

This allows to remote control and receive feedback from software and hardware that implements the OSC protocol such as Pure Data and Max/MSP.

The interface provides a number of different touch controls to send/receive messages:

• Faders
• Rotary controls
• Push buttons
• Toggle buttons
• XY pads
• Multi-faders
• Multi-toggles
• LEDs

[http://hexler.net/touchosc]

Announcement: OSCulator 2.6

My french friend Camille Troillard has released OSCulator 2.6 (Cam, thanks for thinking of me for the Italian translations, it was a pleasure to work with you). And with the help of Ramirez Mora, OSCulator is now available in Spanish (and obviously French) as well.

Here's a selection of new features:

• added the possibility to generate a single MIDI note
• preset management system (also preset changing from osc, and wiimote led follows preset #)
• OSC routing editor: forge custom messages or change incoming OSC messages with a graphical editor
• added support for two virtual HID joysticks
• input lock: avoid to make any changes to the document when the lock is on
• added preliminary support for the TUIO protocol
• wii guitar hero preliminary support
• added the ability to "split" an input in two (from 0 to 0.5 and 0.5 to 1) in order to get two events from that range from 0 to 1
• upgraded max count of connected Wiimotes to 8
• added smoothing to Raw IR
• added double and triple clicks to mouse events
• simplified keyboard combo creation (now you just have to strike the combo)
  

If you would like to know more about Camille Troillard and his vision on the future of OSCulator, please read this nice article written by Kenneth Stewart for the SEAMUS newsletter from the Conservatory of Rice University, Houston.

If you are an iPhone user, please take a look at OSCemote. This simple application, written by Josh Minor, turns your iPhone into a clever tangible user interface with sliders, buttons, accelerometer information, and a multi-touch surface.
Are three iPhones a better choice then my brand new lonely Lemur? ;-)

[Lemur, the renowned multi-touch control surface for audio and media applications, was launched in 2005 by Stantum, formerly known as JazzMutant, the brand name of the company’s music product division. In the December 2007 issue of Information Display magazine, Stantum’s CEO tells the development story of PMatrix technology: Developing the First Commercial Product that Uses Multi-Touch Technology. Everything You Always Wanted to Know About Multitouch (But Were Afraid to Ask)]

P.S. Many thanks to W. Brent Latta and Peter Kirn for the consideration on the CDM pages! Peter, you've been one of my first subscriptions in Google Reader.

Matteo Milani