Tag: synesthesia

Mary Ellen Childs: On Merging Sound and Scent

Mary Ellen Childs sitting on a bench in front of a branches.

Even though Mary Ellen Childs had a lot to say about sound when she visited with us last month, we also spent a lot of time talking about the other senses since many of the works she has created have been immersive multi-sensory experiences.

Still I was totally unprepared for Childs’s ruminations about the aesthetic possibilities of scent, despite my harboring a personal fascination with the relationship between music and perfumery ever since I attended a performance of something called a “scent opera” seven years ago. As a result of our mutual obsession, when I asked her about her attempts to incorporate olfactory perception into her own work our conversation took a significant detour. While this digression ultimately seemed a distraction from our larger musical conversation, it is yet another manifestation of Childs’s unique approach to the creative process.

A conversation at New Music USA
May 6, 2016—2:30 p.m.
Video presentations and photography by Molly Sheridan
Transcribed by Julia Lu

FJO: Literally 15 minutes before you walked in the door I just read an article about you wanting to develop a piece that involves olfactory sensation.

MEC: I’m really fascinated with this. I’ve been researching it now for years. I’ve done some kind of cursory, more experimental sort of public events, but I haven’t yet created the piece that I really want to create, which would be instrumental music written to pair with senses. Specifically I’m interested in writing the music first and having a scent designer create pieces inspired by those pieces of music. And then giving the audience a way of experiencing the two things together.

FJO: There’s an ephemerality to both sound and scent. They’re both produced by physical objects, but they are also both non-corporeal. We live in such a visually dominated world, and these are two phenomena that you can’t actually see. At least sound we can now preserve on recordings. Scent is more elusive. You can bottle a scent, but when you use up that bottle, it’s gone. The other fascinating parallel for me is that the people who make perfumes are called compositeurs. They’re composers. It’s the same word, and it’s the only other usage of this word I know.

MEC: Well, and they work with notes. I have done a ton of research on this. I’ve read a lot, I’ve smelled a lot, and I’ve talked to neuroscientists who work in this field. I’ve gone to France and I visited the Museum of Perfume in Grasse, and I’ve met with a master perfumer near Paris and talked with the director of the Institute for Art and Olfaction in Los Angeles. I’ve thought a lot about how this is going to work. I think it just takes the right moment with the right amount of money. It’s one of those pieces that has not really gone forward because it needs some financial support to make it all happen. But I’m very keen to do it.

Not too long ago, I came across reference to a study that was done showing that what we listen to affects how we smell. I was so pleased. I knew it. That was my hunch, because both those senses can make us have an emotional reaction. They might even make us experience time or space differently. Think of a smell that you want to recoil from. You physically want to contract. So you feel space differently. Sound can do the same thing. You may actually feel more spacious because of what you’re listening to. And you can slow down your perception of time through music. I think you can also do those things with scent. And so you put the two together, and the impact that they would have on you could be quite interesting.

I did an installation-style event as part of Northern Spark a little less than a year ago. Northern Spark is a festival in the Twin Cities that happens overnight, usually around the time of the solstice. It starts at sundown and it ends at sun up. I did a piece called Ear Plus Nose, because I just wanted to sort of get it out there. I just wanted to see how an audience would respond. I used recorded music, my own music. It was a room bigger than this, but not a lot bigger than this, and there was one fragrance that was diffused into the space. Then there was one long table where there were four fragrances. You could take a smell strip and dip it into the fragrance and then could combine that experience with what was diffused into the room along with what you were hearing.

I asked people to write their responses afterwards. I had two stations with two different sets of cards—two questions. The first question was basically just to write about your experience in this room. You know, what was it like? What did you respond to? And the other one was to write a scent memory. And once you’ve written that scent memory, tell me if there was any sound associated with it. I thought maybe some people will do that. But I still haven’t gotten through all the responses! I have a huge bag full. In the course of those hours, I also learned which of my pieces worked in that setting and which didn’t. I was going to try to cycle through various kind of moods and sound worlds, and keep track of when that music was playing and what responses I was getting. That became too much of a headache to keep track of, so that went out the window pretty fast. I ended up gravitating towards one piece that was pretty steady state. It created a mood, but it didn’t have a lot of ups and downs and changes, and I just let that play over and over again. And that was the thing to do in that particular setting.

FJO: Well, what’s interesting is that music has this narrative arc over time and changes. Even most very hardcore minimalist pieces are about the very subtle changes that occur in them over time, because when changes happen so slowly it makes the moment of change feel extremely significant. In a way, such pieces are similar to the way we experience a scent. It initially seems like a steady state but, due to the physical properties of chemicals, it changes over time and certain notes that were less prominent become more foregrounded. Perfume writers often compare a fragrance’s top notes, the olfactory combination you originally encounter when you smell it, with the base notes that appear after its dry down. Of course, this is a function of physical reality.

MEC: Right, so in a concert setting the way that I’m curious about exploring the use of scent is with diffusion techniques and to see if the landscape of scent will change over time. I actually wanted to have additive scents. Let’s say you start out with a citrus note, then you add something more floral, then after that something a little more savory, stronger in some way. Over time you change the scent in a way that actually works. But there are some challenges with that. It’s not a straight forward thing to do because what happens with the way our noses work is that once you smell something you tend to stop smelling it after a while. So, in order to change the scent landscape over time, I would probably have to diffuse the citrus note and then bring in more citrus plus the second scent. And then also consider that the experience of somebody who might walk in midway through would be different because they’re getting the first impact of everything.

It would take some work, and it would take some very knowledgeable people to explore this with me. I don’t know that that particular thing has been done very much, and I’m really interested in exploring it. But on a different note, the other thing that I’ve been doing just to try to start working with this material is I’ve been hosting a series of scent dinners at my home where we have a food course, then a scent course, then a food course, and a scent course. I love to have people over for dinner. I love to cook. I love to make an experience of the evening, and so adding in scent was kind of an interesting thing for me to do. And so I did the first few scent dinners without music, then I thought I should just be taking advantage of these people who are here to pair the scents with music, and so I started doing that and then asking people for their responses. And that’s been pretty interesting, too.

When I talk about working with scent and music, mostly people are very curious and very interested and they want to know more. And they say, “I want to come whenever you do this piece.” But I also get people whom you can tell they’re put off by it, or they’re a little bit worried about it, or they’re not sure they would want to come to a performance like that. And I think it’s because you can close your eyes, we can even plug our ears, but scent actually comes right into our bodies. It comes right inside your lungs. People have strong reactions to it and it is different for everybody. You have different scent preferences. And you have different things that you associate with scent. I was surprised when I heard there were some people who love the scent of skunk. I always thought that that was a miserable scent that you wouldn’t want to be around if you didn’t absolutely have to. But there are people who actually enjoy that scent. And I think that just tells us how individual people’s responses to scent can be. It has to do with your own personal history. I also think it has to do with your cultural background. So I feel like this is such a rich thing to be exploring with music, and I’m actually very eager to do more.

Vinfonies, Nessun Dorma, and Gastromorphology

A bottle of wine and a wine glass on a tale alongside an electronic keyboard, music notation paper, headphones, and a computer screen.

(Photo by Ben Houge)

At just about every technology-oriented conference I go to, I bump into some representative of the Music Technology Group of the Universitat Pompeu Fabra in Barcelona. Probably best known for developing the Reactable, the interactive music control system prominently used by Björk, the MTG is one of the most well regarded centers for music technology in Europe. I believe it was at the Game Developers Conference in San Francisco back in 2012 where I first met the MTG’s Jordi Janer, who has been conducting fascinating research into voice analysis and audio source separation, work that he has made available commercially via his side company, Voctro Labs.

Jordi has another side project as one of the founders of Vinfonies, a series of events pairing wine and music that goes back to 2009. Jordi himself has done a number of sound installations related to various aspects of viticulture, developing pieces based on resonating cava bottles, an interactive grape press, and the sonic byproducts of fermentation. With my food opera events I’ve sought to bring the sounds of the farm into an urban restaurant, but Vinfonies goes the other direction, using the festival as a way to bring wine-themed new media art to rural settings. The events happen annually at harvest time in Vilafranca del Penedès (part of the well-regarded Penedès winemaking region, one of Spain’s sixty-five or so regions that have earned the Denominación de Origen appellation) about 50km from Barcelona. The festival includes concerts, sound installations, and wine tasting sessions. Similar to the work of Spanish sound artist Francisco López, participants experience the wine and music pairings blindfolded, eliminating visual distractions to focus on taste, smell, touch, and sound.

While I was living in Spain, I had a chance to reconnect with Jordi at Music Hack Day Barcelona, sponsored by the MTG as part of the huge Sónar electronic music festival. (Check out my brainwave sonification hack from 2014.) The MTG partnered with Berklee to bring computer music pioneer John Chowning to Spain for a series of events last summer (including his first visit to Valencia since he docked there in 1953, back when he was a drummer in the U.S. Navy), and in the planning conversations, Jordi and I realized we shared an interest in music and gastronomy. He invited me to write a piece for the 2015 edition of Vinfonies, which took place last September. Over the summer, he mailed me a bottle of Azul y Garanza 2012 Garciano (the name, which might perplex some oenophiles, derives from its composition: 50% Garnacha, 50% Graciano), and I set to work composing my musical accompaniment.

BTW, a playlist of previous Vinfonies compositions (including one by Edwin Van Der Heide, whose fascinating sound installation “Spectral Diffractions” took over Mies van der Rohe’s Barcelona Pavilion as part of Sónar in 2014) is available on SoundCloud.

Composing music to accompany a taste-based experience presents some unique challenges, particularly when it comes to synchronization. One of the epiphanies that sparked my food opera project was the recognition of a meal as a time-based art form. It’s easy to recognize several time scales to a meal, from the succession of courses (even simply saving dessert for last) to the entropy that occurs as a hot dish cools or a frozen dish melts to the succession of individual bites.

Recognizing these time scales is straightforward, but synchronizing music to them is a much trickier proposition. Typically, restaurants don’t even try, and in this they resemble far too many video games that simply loop the same piece of music over and over or shuffle a playlist: the only point of coordination between the music and the restaurant is that while you’re in this space, this is the music that’s playing. (See the paper I presented at Invisible Places Sounding Cities in Portugal for a more detailed discussion.) A few restaurants that explicitly stress the multisensory angle go further: Ultraviolet or the various Kitchen Theory dinners will present a track of music synchronized with each course, but for this to work, everyone has to be eating at the same time. This approach—when the dish is served, someone presses play on the CD player—works best when there are a greater number of relatively small dishes that don’t take too long to eat.

To coordinate music with individual bites is still trickier. One clever solution is the Tasteful Turntable, devised by Lars and Nikolaj Kynde to synchronize several small bites with key moments in a soundtrack presented to four diners at a time. Another, somewhat less practical approach is Naoya Koizumi’s Chewing Jockey, which uses a photoreflector sensor and a bone conduction speaker to alter the sound of chewing. But when the dining experience is opened up to multiple diners eating asynchronously (i.e., the way the usually eat), the logistics of synchronization become much harder to address. One solution is to give diners headphones, as Heston Blumenthal does in his famous Sound of the Sea dish, or in performance artist Marina Abramović’s collaboration with chef Kevin Lasko, Volcano Flambé. Headphones works as a personal experience, potentially a profound one, but they also cut people off from the environment and those around them. The other solution is to put a speaker at each seat, broadcasting sound unfettered into the restaurant space, and then (this is the hard part) avoiding cacophony by finding a way to incorporate the bleed of music from adjacent tables into the experience, so that all the sound in the restaurant is coordinated. This is what I’ve done in my food operas, and from my observation, this is the only solution that scales to an evening-length experience.

This solution requires all of the musical compositions written for a food opera to be modular, so that they can be put together in any combination and still sound harmonious. At the events I organized with chef Jason Bond in his restaurant Bondir, there were five courses, and diners chose between two possible dishes per course, so that’s up to ten different pieces of music that could be playing at the same time, each at different points in the composition, from the twenty-six seats in the restaurant. I wrote custom software to generate a new version of each musical texture and play it on the correct speaker every time a diner was served the corresponding dish; because the music was generated on the fly, the software could also ensure that all of these individual streams of music were coordinated in harmony and rhythm. In order to make this coordination as apparent as possible, I limited myself to a diatonic scale (slowly transposing over the course of the evening, with all of the music in the restaurant programmed to conform to the current key) and a steady underlying beat referential of 188 beats per minute.

However, with this Vinfonies commission as a stand-alone piece, I had no such constraints, and so I made it an objective to make the most of my liberty. I also wanted to avoid some of the techniques I had used in the past that I decided were a bit “too easy,” so I decided to eschew steady drones, for example, and the overt diatonicism and rhythmic grid of my previous work.

When I’m developing music to accompany some gustatory stimulus, my process isn’t so different from how I’d approach a video game or a choreography: I sample the experience of the thing I’m scoring (or at least a description or concept art or whatever state of completion the thing is in), evaluate my response, and try to capture what I hear in my mind’s ear. In this, the composer has a great advantage over the researcher; whereas academic studies might try to compare the appropriateness of different types of wine to existing music (as in a recent study contrasting Debussy with Rachmaninoff), a composer can hone in on and express an ideal imagined sound without having to choose between existing examples. (This may lead to less quantifiable data, but in some ways it resembles another study, in which participants were asked to find the single note on a keyboard that best corresponded to a scent stimulus.)

An interesting point came up in Oxford last February, when I collaborated with researcher and wine expert Janice Wang in an event at Alistair Cooper’s 1855 Oxford Wine Bar. We presented three pairs of wines alongside three pairs of musical textures and asked participants to determine which musical texture best matched each wine. In the ensuing conversation, it became clear that some people chose music to evoke the wine, whereas others chose music to complement the wine. This may be frustrating to the researcher, but it’s exciting to the composer, at it shows how much room there is for creativity in devising music/food pairings. Music really can serve as another kind of seasoning, flexible in the same way that an ingredient like mint might be equally at home in a sweet or savory dish. We can recognize this phenomenon from the world of film scoring; it’s an exercise we even give to students at Berklee, to compose different soundtracks to the same film clip to give it a different emotional spin, to make it happy or sad, wistful, nostalgic, or ominous.

The Azul y Garanza 2012 Garciano provided a great opportunity to stretch out a bit. It’s a complex wine, and working at the intersection of food and music, I’m sometimes bemused by the fact that, while it’s common for people to decry complexity in music, complexity in wine is pretty universally considered a positive attribute. Jordi Janer shares an interesting observation on this point; he tells me that when pairing music and wine at the Vinfonies events, he’s found participants much more receptive to challenging, experimental soundtracks than they might ordinarily be. Unlike the whisky project I wrote about earlier, my objective here was not to translate aspects of the wine into sound so much as to create a sonic context for it, and in this case I envisioned a kind of mysterious, mystical setting.

I decided to stick mostly to an octatonic scale, allowing the music to float for long stretches without reinforcing a particular key. I sought a harmonic language that was more dissonant than what I had used in the past. The spiky, disjunct marimba patterns evoke a bit of the spicy quality of the wine. The music has no pulse, evoking a kind of timelessness. I wanted a density to the music that reflected the wine’s complexity, and the delay lines help achieve that, providing a sense of depth; I imagine myself peering into the wine as though it were a dense forest. I tend to prefer delay lines to reverberation in these kinds of pieces, and I think there’s also something liquid in the rolling echoes. The lengths of my delays are periodically changing, which results in a bit of pitch shifting during the transition, and this subtle detuning evokes a kind of tartness. (I’m not sure if this kind of detuning has been subjected to a formal study, but I wonder if there isn’t a link to the expression “a sour note.”) Periodically, as a kind of refrain, a jutting theme outlines an augmented chord, stretching out over a tenth, and alters the mode, swapping a D sharp for a D natural, serving as a kind of punctuation. At these moments, a sustained bed based on cello sounds makes a rare excursion into bass clef territory. In some of my previous work, low cello phrases have been successfully linked to tannins in wine, so I decided to include cello here for the same reason. But in general, the frequency range of the music is in a high register, which reflects the sweet and sour fruit components I notice in the wine. Musical ideas float along as objects for the listener’s consideration, not developed, but juxtaposed for the listener to parse and contemplate. Overall, the music is peaceful, emphasizing stability and resolution, reflecting this well-balanced wine.

(If this description is a bit tedious, note that one of my original reasons for pairing music with food was the recognition of how poor words are for describing music and food, suspecting that they might do a better job of describing each other.)

The instrumentation comes from manipulated recordings of myself (whistling and playing piano strings with mallets) and my Berklee Valencia colleagues Victor Mendoza (marimba master) and Sergio Martínez (percussion savant), with Berklee alum Ro Rowan’s cello recurring periodically.

In the above description, you may readily detect the influence of a composer whose music has deeply influenced my thinking about real-time musical systems and video games—Olivier Messiaen. Much of his music exists in a continuous present, which is exactly what game music must do, as it waits for the next event to signal a transition. By making this piece almost a kind of homage, I sought to link my work to an ongoing musical tradition, something I think is important when venturing into new territory, especially in light of some of the cultural crosstalk I mentioned in my previous post.

My compositional process was idiosyncratic. First I determined the scale and the types of sonorities and simultaneities I wanted to use, and I sketched these out on staff paper, along with some thematic ideas. Then I took some of the instrumental recordings I had prepared to use as source material and turned them into playable instruments in Max/MSP. I wrote a program that would take any note I played on the keyboard and build one of my previously determined chords, in any approved inversion, on top of it; in this way I generated the “mallet piano” part, allowing me to manually control the range, rhythm, and dynamics of that part, while letting the computer generate the chords. Then I wrote a program that would look at the notes of the most recent two chords played by the mallet piano and choose from among them to generate a melody to play on the marimba; whenever I pushed a button, a new melody would be generated according to parameters I specified, and a different button allowed me to repeat a phrase with subtle rhythmic variations. The whistle part, except for the recurring theme (which was played manually), was also generated by looking at the most recent mallet piano chords and choosing a note from among them, although in this case I was determining the octave transposition, rhythm, volume, and duration manually by playing the keyboard. The rattle and cello parts were performed or input manually. To sequence everything into the requested time frame, I used Ableton Live, embedding my Max patches as instruments using Max for Live.

But wait: for this Vinfonies commission, since a linear recording of about three minutes’ duration was what was requested, I could have composed and sequenced everything linearly. So why this weird process involving programmed behaviors?

It comes back to the idea of synchronization. In most cases, eating and drinking are activities that can continue for an indeterminate duration, so I’ve been very interested in composing music that can continue indefinitely. (This is the same problem we face in video games, and much of my creative effort since starting in the game industry in 1996 has been addressed to it.) It’s possible that someday I might want to use this music again and present it in a real-time, indeterminate duration form. If I do, having these algorithmic processes already in place will make it easy to adapt; by reducing the input into the system (playing single notes or pushing buttons instead of playing full chords or phrases), I make it easier to replace my manual input with some automated mechanism. I also wanted to fit this piece into my growing body of “food opera” work, using instruments that I’ve built from acoustic samples in Max as my orchestra, organized around generative or procedural processes to accommodate the constraints of modular, real-time deployment discussed earlier. And I have to admit, composing algorithmic music in this way is simply an interesting challenge. So, since much of my work hovers around this idea, it just made sense to apply the techniques on my current workbench to this new commission. But more than any of these considerations, I feel that this approach to composing—using algorithms to develop textures of indeterminate duration—brings to the fore not just what things sound like, but how they work, and I fundamentally think there’s a real link between the way this music works and the way eating or drinking works.

Wine, perhaps more than any other comestible, brings the question of time to the fore. One of the commonly discussed attributes of a wine at a wine tasting is its finish, how long it lingers on the palette after tasting, which I’ve heard some sommeliers describe in very precise measurements. But thinking about how a sensation changes in the mouth is not limited to wine. After I gave my presentation to the R&D team at Heston Blumenthal’s Fat Duck restaurant in Bray, England, last June, I sat around with some of the chefs and played some of my musical textures, while we tasted caramel and graphed flavor profile changes over time on paper.

When I was studying computer music back in grad school at the University of Washington in Seattle, I remember encountering Dennis Smalley’s concept of spectromorphology. It was fascinating to think of a method for categorizing the different ways a sound could transform over time. Electronic musicians synthesizing sounds are familiar with the concept of an envelope, a shape that describes how a parameter changes over time (the most common variant being an Attack-Decay-Sustain-Release envelope, allowing a synthesist to specify different volume levels for different points in time), and game audio designers are familiar with the concept of a real-time parameter curve (RTPC) that links these parameters to real-time input. I have often wondered about a taste-based equivalent. Is there something about our rich musical language for working with time that can apply to the rhythms of the kitchen? How precisely can we quantify and categorize the way that taste sensations change over time? Maybe we could call it “gastromorphology.”

This issue comes up at the end of one of NPR’s recent features on Charles Spence. First Spence is quoted:

[For example,] a dark chocolate or coffee-tasting dessert, then something like Pavarotti’s [performance of Puccini’s] “Nessun Dorma,” making much more low-pitched sounds, seem to be the perfect complement to help bring out those bitter tastes in the dark chocolate or the coffee.

Then the author of the article gets in a little zinger of a last word:

Of course, ‘Nessun Dorma’ gets a little more high-pitched near the end—so there are still challenges in finding the perfect sound for a constant flavor experience.

This is exactly the problem I’m working to solve. “Nessun Dorma” does change at the end, building to a glorious climax; like most music, it evolves. This is the form of the work, and determining the form is traditionally a big part of the composer’s job. When composing for dance or film or setting text, that evolution may be hitched to another structure, but the end result is a fixed trajectory over time. And this is also why it may be problematic to use a finished piece by Debussy or Rachmaninoff in evaluating taste to music correspondences. Not only are there so many parameters to track, but the rate of change of these parameters (what we might call morphology, or, at a higher level, simply musical form) is also changing at a rate that is likely very different from the rate of change of the tasting experience.

So I return to Messiaen as a useful reference point. I sometimes find myself in the semantically awkward situation of using the antonyms “static” and “dynamic” to refer to the same thing. Sound by its nature is motion, vibration; it is dynamic. But when I talk about static music, I mean music that isn’t going anywhere, music that is nonteleological, music that is not progressing towards a specific goal. I often talk to students of game music about the challenge of taking the pre-rendered dramatic trajectory out of a piece of music they’re writing, so that the game can put it back in. Messiaen exemplifies this kind of stasis.

This project draws on a lot of ideas: Messiaen’s approach to static music, video game music that responds to user input, algorithmic or generative processes, crossmodal sensation, multimedia pairings, the rhythmic profile of a meal. Taken together, they suggest a flexible, dynamic approach to composition, and the applications are not limited to wine, or to a meal. In fact, a real-time, responsive system such as I’ve described could be equally put to use to create a customized soundtrack for any of the unpredictable events of daily life, opening up a whole new arena for creative work.

Music Inspired by Visual Art

Man standing in front of a musical score which is exhibited like a painting on a wall of a gallery.

My previous column posed some broad possibilities and potentials for new music as a catalyst for learning. Now I’d like to turn my attention to one specific realm of music for learning I’ve explored in my work as a composer: music inspired by visual art.

“Inspired by” is, of course, a problematically vague descriptor. (If you think of something better, please share it with me!) My intention has been to create music that draws the listener into the world of the artwork—music that gives the listener a new lens through which to see art.

I’ve come across many pieces of music, by different composers, in which the composer has been drawn to a work of visual art as a source of inspiration—as a writing prompt, in a sense. To my ear, such works oftentimes result in music with an abstract or imperceptible connection to their source material: the visual art has had more to do with process than product. There is, of course, absolutely nothing wrong with this approach or the music that results. However, as both a listener and a composer, I feel a distinction needs to be made between this kind of work and music that has been envisioned expressly for the purpose of illuminating, commenting upon, and conversing with visual art—music where viewing the art while listening to the music is, in some sense, essential to the full realization of the composer’s vision.

Teaching Art Appreciation through Music and Online Media

My love of visual art, and my desire to increase appreciation of relatively under-recognized visual artists, has led me to incorporate art into my musical projects in many ways. These projects have ranged from a one-act monodrama depicting the artistic evolution of painter Charles E. Burchfield (The Coming of Spring, 2014), to a piece for flute and guitar following the work of Japanese-American artist Chiura Obata from Yosemite National Park to an internment camp (Dai-Shizen (Great Nature), 2014). At the heart of many of my visual art-inspired projects have been multimedia concerts, which I’ve brought to art museums as well as concert halls, in which the music is performed live with images of the artworks projected above the performers.

Using music to teach visual art appreciation has been core to my art-inspired endeavors. For one project, I put together an evening-length program of chamber music inspired by visual art and produced an in-depth complementary online “concert companion,” containing videos featuring interviews with senior museum curators about the artists and written commentary with excerpts from the music demonstrating how I translated art into musical ideas (Beyond the Notes: Music Inspired by Art, 2011). This website and related videos are now a standalone educational resource.

Smartphone showing an image of a painting with a text description below.

Illuminating the connections between music and visual art through a mobile website. Photograph by Nell Shaw Cohen.

The specific compositional techniques with which I’ve attempted to evoke visual art are beyond the scope of this article. Generally speaking, in addition to finding musical equivalents to color, texture, line, shape, etc., I’ve sought to represent and even guide the gradual wandering of one’s gaze across an image. Different sections or elements of the music are envisioned as responses to different areas within the image.

Music Helps Us to See

So, how exactly does music deepen an understanding of visual art? It has been empirically shown that pairing visual art with music can stimulate learning experiences and promote art appreciation skills. A study by Victoria Pavlou and Georgina Asthanasiou (2014) found that presenting images of an art object simultaneously with a complementary musical recording (which the researchers selected for pairing) led to significant improvements in art appreciation amongst college students who were inexperienced viewers of art. In post-experiment interviews, the participants reported that listening to the music made a substantial positive impact on their ability to derive meaning from both representational and abstract artworks and generally prompted stronger emotional responses to art.

Anecdotal evidence has confirmed this phenomenon for me. Audience members at performances of my work have commented that listening to the music while viewing video projections of the art enabled them to understand or appreciate the artworks in ways that they hadn’t without the stimuli of music. In some cases, these audience members had actually experienced a music-free viewing of the original painting in the museum’s gallery directly prior to the concert.

In addition to the aesthetic and emotional responses that music can incite, some of the positive effects of pairing music with visuals might simply stem from the support of sustained viewing. Lachapelle, et al. (2009) conducted a study in which they tested the common assumption that there is a positive correlation between duration of viewing and greater appreciation of art. This was found to be true in the majority of non-expert viewers that they studied. Music has the advantage of being a temporal art: in performances of some of my works, listeners are shown a projected still image for five or six minutes at a time during the musical performance. In our highly interconnected and hyper-stimulating age of the Internet, motivating people to simply look at an artwork for a sustained period of time is no small matter!

Of course, well-chosen music can certainly complement art viewing even if the composer hadn’t envisioned the music for that purpose. Conversely, music inspired by art should certainly have the potential to be fully engaging without being presented in tandem with the artwork (most of my art-inspired works have, at some point, been performed and heard sans video projections). I’m simply seeking to create work that is optimized for that pairing, in which the art and music are enriched and elevated for audiences through their juxtaposition.

A musical score in front of a framed artwork showing a landscape scene of a mountain surrounded by water.

Interpreting visual art through music. Artwork shown: Point Reyes from Chimney Rock by Tom Killion (2011). Photograph by Nell Shaw Cohen.

Engaging New Listeners

In addition to enhancing appreciation of visual art, my other motivation for creating music inspired by visual art is to bring experiences of new music to audiences that are interested in visual art and attending arts events in general, but would be very unlikely to listen to new music in another context. Ideally, this works on two levels: first, by attracting new audiences through strategic marketing, then supporting the listening experience through visual stimuli.

I strongly suspect that, for people unfamiliar with new music, there is a substantial boost in the immediate appeal of a new music concert when placed within a larger arts context. For example: the multimedia concerts of my chamber music inspired by paintings on display at the Peabody Essex Museum in Salem, Massachusetts, and the Parrish Art Museum in The Hamptons, New York, attracted some of the largest audiences my music has enjoyed—most of which, I perceived, were not usual new music listeners or even particularly devoted consumers of “classical” music. I can’t imagine the Peabody Essex concerts, for example, would’ve been nearly so well attended if they had been publicized as performances of “String Quartet No. 1” rather than a musical representation of paintings featured in the museum’s special exhibit.

(Disclaimer: it’s extraordinarily irritating when music is publicized as having a close relationship to visual art but doesn’t deliver on the promise! Furthermore, an irrelevant-seeming or super distracting video projection can tend to overshadow any enjoyment I might’ve derived from the music alone.)

Once the audience is present, attaching musical experiences to visual content may actually help many listeners to focus on, comprehend, and enjoy the music as they listen—especially if there is a strong, carefully considered resonance and balance between the visuals and music. This phenomenon has been demonstrated in a range of contexts, from cartoons to music videos. Many composers have long been exploring the integration of custom, dynamic visuals with musical performances. I think the potential for visual content to support listening deserves further investigation and consideration in contexts where music is being used as a learning tool for visual art appreciation.


Has music ever altered the way you perceived a work of art, whether or not the music was intended for that purpose? Are there any works of music explicitly inspired by visual art that you’ve found particularly effective or impactful? What do you think are the most important problems and possibilities of pairing these two art forms in the ways I’ve described?

Bridging Gastronomy and Art Requires Making Connections

The Basque Culinary Center's building, which is an impressive architectural structure , in front of which is an impressive piece of contemporary sculpture.

The Basque Culinary Center is as much a monument to contemporary art as it is an incubator of ideas for contemporary gastronomy.
(All photos by Ben Houge.)

Spain has been at the forefront of contemporary cuisine for many years, rising to international prominence with Ferran Adrià’s acclaimed and highly influential restaurant elBulli, which closed in 2011. Given my work combining music and food, when I was recruited to transfer to Berklee College of Music’s new campus in Valencia in 2013, this was no small consideration in deciding to accept the gig. (I was brought on to help develop curriculum for and serve as the full time faculty member in the new Music Production, Technology, and Innovation master’s program, and I also took charge of the video game component of our Scoring for Film, Television, and Video Games master’s program.)

As a result of spending two years in Spain, I’ve learned a lot about the key figures in the fascinating gastronomy scene, a recurring theme of which is the desire to reach out and engage with ideas from other artistic disciplines. In fact, merging taste with other sensory experiences was central to the topic—La Vanguardia (the Avant-Garde)—of this year’s Diálogos de Cocina conference. Now in its sixth iteration, this biennial event, founded in 2007, is a product of the Basque Culinary Center and Euro-Toques, and from the beginning the focus has been on interdisciplinary dialog. The conference took place over two days, March 9-10, 2015, at the Basque Culinary Center’s gorgeous new building on the outskirts of San Sebastián (the city with, not coincidentally, the highest number of Michelin-starred restaurants per capita in the world). This was the first conference I had attended that was devoted solely to gastronomy—I typically find myself at video game or digital art conferences—but after experiencing the amazing dishes served up at every coffee break, this is the only kind of conference I want to attend from now on.

This is what gets served at a "coffee break" during Diálogos de Cocina

This is what gets served at a “coffee break” during Diálogos de Cocina.

The entire event was focused on the future, investigating ideas from other art forms as well as innovations in technology. Leading crossmodal psychologist Charles Spence was one of the presenters (in fact, it was he who commended the conference to my attention, following my presentation to his Crossmodal Research Laboratory at Oxford University last February), and in his fascinating talk, I finally got to hear the sound of Heston Blumenthal’s influential multimedia dish The Sound of the Sea (which was heavier on the gulls than I had expected). Adrian Cheok of the Mixed Reality Lab in Singapore shared some really wacky progress on his efforts to digitize taste and smell; one lucky volunteer got to taste one of Cheok’s digitized flavors via a device he put in his mouth. Other presentations included a history of avant-garde art, an overview of recent technological trends, and a meditation on communication in the internet age, plus panel discussions on the dining experience of the future, socializing culinary innovation, and what experimental art can bring to cooking (and vice versa).

The consistent theme was how to draw on ideas from other creative practices to enhance what’s going on in the kitchen, an investigation many of the participating chefs were already pursuing in their own restaurants.

Members of the board of the Basque Culinary Center were the main hosts of the conference, and one chef whose inviting presence was most continuously felt was Andoni Luis Aduriz of Mugaritz.  His smiling demeanor extended throughout the sessions and into a group dinner at the amazing sidrería Zelaia, where he was seen cutting enormous traditional Basque steaks for conference attendees to ensure that everyone felt welcomed. Andoni has pushed Mugaritz to explore unique collaborations over the years, working with musician Felipe Ugarte on the Mugaritz BSO (Banda Sonora Original, or Original Soundtrack) in a project that involved visits to Ethiopia and Peru, developing a food ritual with the choreographers Idoia Zabaleta and Filipa Francisco, and providing the climactic meal to a production by Barcelona-based theater group La Fura dels Baus of Titus Andronicus. After I met him and told him about my food opera project, he invited me back to give a presentation to the whole R&D team at Mugaritz, who received me with warm inquisitiveness last August.

On the other side of the country, on Spain’s northeast coast, the three Roca brothers of El Celler de Can Roca have also stretched the interdisciplinary boundaries of gastronomy, most notably with an immersive multimedia “gastropera” called El Somni (The Dream). Twelve invited guests were treated to an extravagantly high tech, one-time event on May 6, 2013. The event, documented in a film by Franc Aleu (another presenter at Diálogos de Cocina), was in twelve eclectic movements, each with music by a different composer, ranging from robotic string instruments to traditional Catalan vocalizations to neo-romantic piano, while motion captured 3D graphics were projected onto the table and onto screens surrounding the diners. Sound also plays a role in one of the dishes on their regular menu: an edible reenactment of a goal scored by soccer star Lionel Messi, who plays for the Barcelona hometown team, which comes accompanied by a recorded sports announcer commentary.

Since elBulli closed in 2011, Ferran’s brother Albert Adrià has been working to keep the family business flourishing. Currently there are five Adrià-branded restaurants in Barcelona (all located within a few blocks near the Plaça d’Espanya, dubbed El Barri Adrià), including the acclaimed Tickets, with a new restaurant named Enigma to open soon. But their most recent opening occurred last summer fifty leagues south, in the Balearic island of Ibiza, an ambitious interdisciplinary collaboration with Cirque de Soleil called Heart Ibiza. Advertised as a fusion of gastronomy, music, art, and performance, this elaborate take on dinner theater proposed a tantalizing opportunity to observe how live performance might complement a meal.

But unfortunately the meal I experienced there last summer was less than the sum of its parts. Throughout the restaurant, dancers danced, actors acted, and bodies were painted, while live video feeds illuminated the walls, but servers clashed with dancers in the aisles (resulting in having a drink spilled on me, for which the waiter did not apologize), and the performances were completely out of sync with the dining experience. Halfway through a course, the lights would dim, leaving me to munch in a lurid blue glow, which, as Charles Spence will confirm, has been shown to have a deleterious effect on food. When I could see it, the food was fantastic, served in some imaginatively sculptural tableware, such as a porcelain frog you had to kiss in order to extract the first bite, but the chaotic surroundings, completely out of sync with the meal, prohibited the performance from enhancing the dining experience. Some issues may be a result of poor logistics and layout (despite reassurances to the contrary, I missed a lot of what was happening on a stage around a corner), but I suspect that the very presence of live performers in the restaurant inherently distracts from the meal on the table.

Ibiza is also home to Paco Roncero’s infamous Sublimotion. Representatives from InHedit, the Madrid-based company that provided some of Roncero’s technology, shared the stage with Adrian Cheok at Diálogos de Cocina to discuss the use of new interfaces in gastronomy. The cost of the technology is surely one reason Sublimotion proudly proclaims itself to be the most expensive restaurant in the world, although I have to say that this elitist stance is anathema to my goals as an artist. But fortunately I already had the Sublimotion experience when I visited Ultraviolet, Paul Pariet’s innovative multimedia restaurant in Shanghai that Sublimotion has been widely derided for having plagiarized.

As Ultraviolet demonstrates, Spain does not hold a monopoly on multimedia dining experiences. Ultraviolet seats ten people per night for a twenty-two course meal in a room that has been outfitted with video projectors, a sound system, and a mechanism to waft in different smells throughout the evening. Back in 2012, my experience at Ultraviolet was wonderful, including a few truly transcendent moments. The one that sticks with me the most was the most understated: a simple slice of bread in meunière sauce with a few truffle slices, experienced in a projected forest, while subtle ambient sound played in the background, a profound synergy of the senses that remains for me a benchmark of what can be achieved in this arena.

But I must point out another course that exemplifies the pitfalls in trying to bridge the worlds of gastronomy and art. Towards the end of the meal, there was a riff on traditional gazpacho (in the world of food, Spain is never far away) that involved two different elixirs separated by an edible shot glass. As the dish was served with a lighthearted “Olé” and flamenco music began to play, the video screens metamorphosed to display not an idyllic Iberian landscape or a boisterous cervecería, but Picasso’s devastating Guernica, commemorating the horrific slaughter of civilians in a Nationalist bombing raid during the Spanish Civil War. When I first saw the painting in 1998 at the Reina Sofía Museum in Madrid, I was somberly transfixed for twenty minutes. Projecting it on the walls of a restaurant as a shorthand for Spanishness was an egregious miscalculation, insensitive to the language of art, but more important, to the tragedy depicted.

I think all of these examples suggest that, as we’re moving beyond the mere novelty of pairing sound or visuals with a meal, the focus needs to shift to what is being communicated by the resulting amalgamation; it’s not enough to simply put things side by side, to project a painting or dance in the aisles next to a diner. The languages of art and food are very different, and in many cases artists and chefs seem to be talking past each other. I think the key is to look beyond the end product of a meal or a performance or a composition to examine the processes and considerations and motivations that produced it. Despite the pervasive trend for chefs to develop dishes that visually evoke famous artworks, we should be thinking not about how a plate of food looks like a painting, but about how the work of a chef is like the work of an artist. This is where fruitful interdisciplinary conversations can occur.

At Ultraviolet, as well as in El Somni, which prominently features a 3D animation of Michelangelo’s iconic David (at one point depicted in flagrante delicto, and later shown shattering to pieces), a famous artwork is reproduced to serve as a cipher for art. I understand this impetus; for much of my career in the video game industry, I’ve observed a desire for games to be taken more seriously, to aspire to the artistic credibility and cultural respectability of film, and sometimes that means invoking the conventions of cinema in cut scenes, casting, and marketing. But in fact, in gastronomy as well as in games, these ciphers are unnecessary.

I lived in Shanghai for six years, from 2004 to 2010, and during this time, Paul Pairet was my favorite chef. When I first conceived my food opera project, back in 2006, I had his innovative cooking at his first Shanghai restaurant, Jade on 36, in mind. The food was spectacular, iconoclastic, playful, but with a serious rigor, and wildly inventive: candied foie gras on a stick, ice cream disguised in a lemon rind, sardine mousse served in tin cans. It dawned on me that the experience I was enjoying was exactly the reason I went to new music concerts, to have my preconceptions shaken, to fully engage my senses to interrogate and evaluate new stimuli, not relying on conventions of naming or presentation or other culturally learned tropes. That was when I made the leap, realizing that the kind of music system I was then designing for Ubisoft (as audio director of Tom Clancy’s EndWar) could be equally applied to the unpredictable, real-time input of the dining room, highlighting and harmonizing with the music inherent in the meal, responding to its intrinsic rhythms. Ultraviolet opened much later (in May 2012, about a week after my first food opera at Harvard), but by recognizing a parallel creative process, one that, like music, unfolds over time, I already saw the tremendous potential of pairing music with food in a way that builds on the language of each.

Having completed my two-year appointment in Valencia, I returned to Berklee’s Boston campus this fall, and I now teach music programming in the Electronic Production and Design department. One of the perks of being back in the Boston area is being able to attend Harvard’s fantastic Science and Cooking Lecture Series, which has been host to a parade of luminaries from the culinary world over the past six years. A few weeks ago I got to meet up with Andoni Luis Aduriz once again, while he and one of his chefs, Ramón Perisé, were in town to present a fascinating talk on science and emotion as part of the series.

The next day, I organized a Berklee tour for the two of them, showing them some of the fun musical gadgets we have in the EPD department before visiting the new studio facilities in Berklee’s brand new building at 160 Massachusetts Avenue. As part of the tour, my boss, EPD Department Chair Michael Bierylo, demoed a Moog System 55 synthesizer. My take on the recent resurgence of interest in modular synths is that, in comparison to the vast array of sounds available as plug-ins in today’s digital audio workstations, the constraints of an analog modular rig help focus creativity. While this may have been the first time he’d seen a Moog synth, Andoni immediately recognized this concept, the notion of freedom within limitations, as being just as true of his work in the kitchen. Identifying these kinds of correspondences is what makes working at the intersection of different practices so fascinating and why I have been incorporating these ideas into my classes, to teach students about creativity and innovation by drawing parallels to other disciplines.

The last stop on our Berklee tour was the cafeteria in our new building. The previous cafeteria (a former hotel swimming pool, I’m told) had a longstanding tradition of being converted every evening into a performance venue for student ensembles, so when designing the new cafeteria, the priorities were inverted: instead of a cafeteria that also serves as concert hall, we built a concert hall that doubles as a cafeteria. Seeing Andoni and Ramón in that space, I recognized it as a perfect embodiment of the ideals of interdisciplinary collaboration. By accommodating the concerns of two different creative practices, the potential of each is expanded, and a welcoming space emerges, awaiting unforeseeable new expressions.

Andoni and Ramón at the Berklee Cafeteria

Andoni and Ramón visiting the Berklee Cafeteria.

Tasting Notes

A group of people sitting around a long table with whisky snifters in front of them in a darkly lit room.

All the folks at this table are about to taste Chivas’s Ultimate Cask Collection, but when they do so they will also be listening, on headphones, to music that was specifically created to bring out certain taste sensations in the whisky. Can what they’re listening to affect their perception of what they are tasting? Chivas thinks so. (All photos courtesy Chivas Regal.)

A few weeks ago, I wrote some music to accompany a whisky. The premiere was the unveiling of the Chivas Regal 18 Ultimate Cask Collection at an event in London, hosted by Chivas Master Blender Colin Scott and chef Jozef Youssef. The music was not there to simply provide background ambiance to a swanky event; my objective was to change the way the whisky tastes.

Jozef Youssef is the founder of Kitchen Theory, a London-based group whose experimental pop-up dinners explore the frontiers of science and gastronomy. I met him at one of their Synaesthesia dinners last April, which drew from the growing field of crossmodal psychology to examine how the senses can overlap even in those who don’t experience classic synaesthesia.

At the whisky launch event, Youssef guided participants through a series of experiments designed to provide new perspectives on the whisky. Chivas Regal 18 is a blended Scotch whisky in which each component has been aged for at least 18 years. The Ultimate Cask Collection is a reimagining of Chivas Regal 18 that has been aged in American oak, resulting in pronounced vanilla, caramel, and orange marmalade notes. Some of these experiments one might expect at a whisky tasting, observing how dilution, temperature, and aroma affect one’s impressions. Less expected might be the texture cube I first experienced at Synaesthesia, with different materials (from velvet to velcro) on each face, allowing imbibers to experience how different tactile sensations affected their experience of taste. And similarly unorthodox was the experiment that involved listening to two musical textures while sipping; one was designed to bring out the sweet notes in the whisky, while the other emphasized its bitter qualities. Composing these textures was my job.

Jozef Youssef standing and holding a carafe of whisky as participants seating at a table look on.

Jozef Youssef guides folks through a whisky tasting featuring sound.

I actually do a lot of this kind of thing. Most of my work these days involves exploring correspondences between music and food in a series of audio-gustatory events that I call “food operas.” I’ve collaborated with a couple of chefs in recent years, and I wrote a detailed article about three events I did with Boston-area chef Jason Bond that was published in NewMusicBox in 2013. Since then, I’ve also branched out into composing music for wine and, as in the aforementioned event, whisky. Over the summer I taught a class to students in Berklee College of Music’s Music Production, Technology, and Innovation master’s program that probed the real-time technology and aesthetics behind these events; students presented their final projects alongside an eight-course tasting menu at Quique Dacosta’s Michelin-starred restaurant El Poblet in Valencia, Spain. My next food opera is a collaboration with St. Paul-based new music ensemble Zeitgeist, which will premiere in the spring.

The original inspiration for my food opera project was the idea of using real-time music deployment techniques borrowed from my work designing audio for video games over the past twenty years to score the indeterminate events of the dining room. I sought to apply recent technical advancements in games as well as ideas from the discipline of sound installation (e.g., a big multichannel speaker array to deliver sound to diners) to achieve an unprecedented level of synchronization between the senses of taste and hearing. It’s been a fascinating challenge to think about what kinds of sounds go with what kinds of foods, in a way that heightens diners’ sensory awareness of the music as well as the meal.

Since I started working on the project, I’ve become aware of a growing body of work in the field of psychology that explores crossmodal links between the senses. At the forefront of this field is a psychologist named Charles Spence, who heads up the Crossmodal Research Laboratory at Oxford University. His recent book The Perfect Meal: The Multisensory Science of Food and Cooking, co-authored with Betina Piqueras-Fiszman, is a wonderful compendium of the occasionally outlandish research being undertaken in this burgeoning field. His work has been prominently featured in the press (in The New Yorker’s current “Food Issue,” for example, and several times on NPR), and he has served as consultant to celebrity chefs including Heston Blumenthal, whose Sound of the Sea dish, introduced in 2007 at his renowned three Michelin star restaurant The Fat Duck in Bray, England, is probably the most famous pairing of sound with food. This seafood dish, plated to resemble foam on a slice of sandy beach, is served with an iPod in a conch shell, its protruding headphones immersing diners in ocean sounds as they eat, a byproduct of some of the sound pairing experiments the two conducted together. Spence has also collaborated with Jozef Youssef, and his research formed the backbone of Kitchen Theory’s Synaesthesia dinners.

Spence is widely cited for asserting that, by playing different sounds, “we’re able to show that we can change the experience in [the] mouth by about 5 or 10 percent,” as he told NPR. He goes so far as to assert that sound can be employed to fight obesity, removing sugar from foods and sonically making up for the lost sweetness.

I had a chance to meet Spence when I visited Oxford last February to present my work to his Crossmodal Research Lab. My friend Janice Wang, who helped out with my Bondir food operas, recently moved from Boston to Oxford to pursue her PhD in Spence’s group. Her own research was recently featured in the Financial Times, in an article entitled, “I Use Music to Change How Food Tastes.” Janice is also president of the Oxford University Blind Wine Tasting Society, which, at the time of my last visit, had recently administered a crushing defeat to rival Cambridge University. While I was in town, Janice and I collaborated on an event at the delightfully cozy 1855 Wine Bar in Oxford, working with sommelier Alistair Cooper to pair six different wines with six different musical textures. She also accompanied me on my visit to Kitchen Theory and introduced me to Jozef Youssef in person.

Janice is part of a group who identify themselves as Crossmodalists, committed to promoting experiences that engage and reinforce the links between all the senses. Perhaps due to the influence of figures like Spence and Blumenthal, there seems to be quite a scene for crossmodal dining in the UK at the moment, and music figures prominently. I attended one of this group’s rehearsals last April, which included a parade of scents, live piano, an edible painting, and choreography that responded to flavor—a fascinating mix.

Dining is a profoundly multi-sensory experience. A lot of times we might describe a good meal strictly in terms of taste, but in fact eating involves all of the senses, and psychologists are quantifying to what extent this is true. Smell is the most obvious supporting sense, but also the feel of the silverware, the color of the plate, the food’s appearance, even the mood of the diner, all of these elements combine to affect our perception of a meal. Even after food enters the mouth, temperature and texture merge with taste in our evaluation of a dish. In Spence and Piqueras-Fiszman’s book, I learned that we actually have two different senses of smell: the orthonasal is what we typically think of as “smell,” applying to external odors in our environment, whereas the retronasal is concerned with what is already in the mouth and is thought to have evolved independently and much earlier. This is one reason (alongside aeration) that sommeliers sometimes slurp wine, to aid in retronasal evaluation.

Sound plays a huge role in eating. Studies show that loud sounds dull our sense of taste, which may explain why airplane food tastes so bland, and why people order more tomato juice on planes, as umami flavors are the most resilient to volume. The sound of mastication also has been shown to affect our perception of an item’s freshness, improving the impression of carrots and potato chips. To me, all of these observations underscore the notion that developing sound to pair with food represents an exciting new arena for aesthetic expression.

Three people at a table wearing headphones and sipping whisky

Applying the role that sound has in affecting taste perception to drinking whisky.

My mandate for the Chivas event was very specific, and so these whisky textures represent my first attempt to write music that explicitly drew on the crossmodal psychology research. As Charles Spence will tell you, high frequency sounds have been shown to make things taste sweeter, while low frequency sounds (low brass, in Spence’s experiments) make them more bitter. My textures are short, simple, and very consistent throughout their short durations, attempting to hew closely to the script, although I still wanted to provide a bit more musical interest than a static sustained tone. They are similar enough that useful comparisons can be made where they diverge.

My sweet texture is built from recordings of a flute, a clarinet, and wind chimes. The two-voice main melodic part is in short diatonic phrases, slow and legato, a type of phrasing I often use to create a sense of peaceful, suspended time. The motion is largely stepwise, the harmonies consonant, predominantly thirds and sixths. There may be a slight evocation of the type of traditional melodies one might expect to hear in the Scottish highlands, albeit in fragments. In the background is a slow moving harmonic pad to provide a bit of context, and wind chimes tinkle throughout. There’s no steady pulse. The overall impression should be of stability and resolution with mellow timbres in a floating, high register.

In my bitter texture, the overall frequency range is much lower, as the cello takes the dominant role, playing much more aggressively and roughly than anything in the sweet texture. The cello part is broken into short, intermittent phrases, similar to the lead part in the sweet texture, and similarly in two voices, as it’s all double stops, but here there is a steady pulse, and phrases tend to emphasize more dissonant intervals. Wood blocks and temple bells further emphasize the pulse, while a low drone underneath provides a harmonic reference point. Rather than emphasize brass, as in Spence’s study, I chose to use sounds relating to wood, as the bitter elements were also linked to the whisky’s having been aged in American oak casks, and I wanted to bring out that quality as well.

There’s a question that could be raised about linking the sound of wood (woody resonance of string instruments, sound of struck wood) to the taste of wood (oak). On one hand, we might say that the taste and sound are not related at the level of sensation, that one does not necessarily evoke the other. On the other, we might observe that most humans have learned to associate these things through a lifetime of interaction with wood. The question of what’s learned or culturally conditioned as opposed to what’s innate comes up all the time in designing these pairings.

As I’ve learned more about the psychological approach to the dining experience, I’ve wondered a lot about the point at which the work of the researcher ends and the work of the artist begins. Whereas in a lab experiment, one might want to isolate certain parameters of a taste experience, a lot of times as a composer, I’m trying to blend and merge and complement. Rather than dealing with pure tastes like sweet or bitter, I’m typically interested in a complete dish that has balanced flavors and textures, thinking how music can join in as another set of ingredients. It’s usually not my objective to simply replicate a dish in sound, but to complement and transform it, guided by my experience writing music to accompany dance or video games. In a way, maybe the rules of psychological association are like the rules of music theory, serving as a reference point for a composer, a framework or a palette that can then be applied, twisted, or inverted to far ranging aesthetic ends.

A group of beakers filled with various types of Chivas whisky.

Ultimately a satisfying aesthetic experience—whether eating a meal, savoring a glass of whisky, listening to a piece of music, or participating in a synaesthetic immersion that involves all of the above—must be more than an experiment.

Food Opera: Merging Taste and Sound in Real Time

[Note: Portions of this article are borrowed from my paper “Food Opera: A New Genre for Audio-Gustatory Expression,” co-authored with Jutta Friedrichs (producer of our food operas), to be presented at the Second International Workshop on Musical Metacreation, and are reproduced here with gratitude.]

Beside the White Chickens: A Summer Food Opera (excerpt) from Ben Houge on Vimeo.


Since 2012, I’ve co-produced three audio-gustatory events that I decided to call “food operas.” For each of these events, I composed a real-time generative soundtrack to accompany a multicourse meal, designed by chef Jason Bond of Bondir restaurant in Cambridge, MA. This music responds to events cued by diners, drawing on event-driven scoring techniques adapted from my work in the video game industry since 1996. Each table in the restaurant is outfitted with speakers, one for each diner, totaling thirty channels of coordinated, real-time, algorithmic, spatially deployed sound in all.

The concept is predicated on the acknowledgement of dining as a time-based art form, akin to film, dance, and music. I use the term “opera” for its multimedia associations, describing a hybrid, interdisciplinary work that engages and explores the junctures between multiple senses.

I identify the main innovations in these food operas as follows:

• Real-time scoring techniques adapted from the field of video game music create a custom soundtrack that follows unpredictable, live input from diners.
• Associating sound with taste at the level of texture (somewhere between an individual note and a complete musical phrase) allows for a heterophony in which the two senses can be closely linked while still maintaining a degree of autonomy.
• Coordinating sound across a massively multichannel speaker array allows us to preserve the interpersonal and social aspect of communal dining while also elaborating a large-scale sonic environment. (I spent some time debating whether thirty channels could be described as “massively multichannel,” but I decided in the end to keep the term, as it’s more than what can be deployed from a single computer using a single audio interface, more than you typically find in a movie theater, and far more than a typical home listening environment.)

Each of these innovations on its own may represent a small step forward, but collectively, they allow for what we (producer/artist Jutta Friedrichs, chef Jason Bond, and I) consider to be a new genre of artistic production: the food opera.

One reason we consider food opera to be a new genre is that its expressive potentialities are so vast. The potential to link sound to food, scent, and the tactile sensations of the mouth creates an entirely new field of sensory interplay, which may be harnessed to a wide range of expressive ends. Approaches include theatrical narrative structures (including non-linear or generative narratives) that might tell a story, spatial or landscape meditations that might resemble a sound installation, and ritual events such as a Passover Seder or a wedding ceremony.

As an example, in our second and third food operas, alongside purely musical elements, we incorporated field recordings and interviews from some of the local, organic farms that provide Bondir’s ingredients. Our goal was to tell the story of sustainable food practices by sonically connecting diners to the sources of their meals. We feel that this message was conveyed more deeply and memorably than it would have been using sound or taste alone.

Through the food opera project, we hope to show how the application of real-time music generation techniques, originally developed for use in the virtual worlds of video games, can increasingly be applied to score the unpredictable, real-time phenomena of our everyday physical environment, which we observe to be a rapidly growing arena of cultural production.


The chef and the composer. Photo by Melissa Rivard/Andrew Janjigian.

History and Previous Work

The impetus to associate eating with music goes back to our earliest human history; in myriad historical documents, music has been present at feasts, banquets, rituals, and celebrations. Since the development of the restaurant, performers have often been present to provide background music for diners. And following the advent of recorded sound in the twentieth century, recorded music, from jukeboxes to iPods, has increased in prominence in restaurants to the point of ubiquity. But in all of these situations, music, by the very means of its production, has been physically distanced from the table and at best only coarsely synchronized with the meal, with none of the sophistication of interplay between music and other art forms to which we are accustomed in a ballet, a film, or an opera. (For a lively and detailed overview of historical pairings of food with sound, check out Qian [Janice] Wang’s freshly minted MIT master’s thesis: Music, Mind, and Mouth: Exploring the Interaction Between Music and Flavor Perception.)

In recent years, there has been an increase in artistic attention to the sense of taste, while at the same time chefs have been making forays into the art world. Perhaps the most famous milestone in this rapprochement was Spanish chef Ferran Adrià’s “G Pavilion” at the Documenta 12 art exhibition in 2007. Other touchstones include Natalie Jeremijenko and Mihir Desai’s Cross(x)Species Adventure Club and renowned French pastry chef Pierre Hermé being invited to lecture at the Harvard Graduate School of Design.

Examples of food-based experiences that incorporate sound are more rare. The most well-known example is perhaps The Sound of the Sea, developed at Heston Blumenthal’s Fat Duck restaurant in Bray, England; this dish is served with an iPod in a conch shell allowing diners to hear a field recording of ocean surf on headphones while they dine. A similar approach is used in Volcano Flambé, developed by chef Kevin Lasko and artist Marina Abramović at Park Avenue Winter in New York, which was accompanied by a recording of the artist describing the ice cream and merengue-based dessert and thanking diners “for eating with awareness.” Paul Pairet, a French chef based in Shanghai, opened a new restaurant named Ultraviolet in the summer of 2012, which seats only ten diners per night, and each of the twenty-two courses is accompanied by video projections on the walls and a soundtrack of previously-composed music. (The Beatles’ “Ob-la-di, Ob-la-da” accompanies a riff on traditional English fish and chips, for example.)

While there has been an increase in art world interest in eating in the last century, much of it ignores the sensory nuance of taste. I dismiss as irrelevant to the current discussion art that incorporates food for purely visual or conceptual effect (e.g., Luciano Fabro’s Sisyphus, Paola Pivi’s It’s a Cocktail Party), performance work that involves eating without an exploration of the sensation of taste (Yoko Ono’s Tunafish Sandwich Piece, Alison Knowles’ Identical Lunch, Emily Katrencik’s Consuming 1.956 Inches Each Day For 41 Days), or merely visual representations of food (Claes Oldenburg’s Baked Potato, Francesc Guillamet’s photographs of dishes from El Bulli).

Small Asparagus Opera

Diners waiting in anticipation at the first food opera. Photo by Melissa Rivard/Andrew Janjigian.

My food opera work has its genesis in conversations and sketches dating back to 2006, including a workshop in Shanghai in 2010. The first public presentation of these ideas, and also my first collaboration with chef Jason Bond, was entitled “Food Opera: Four Asparagus Compositions” and took place in May 2012 at Harvard’s Graduate School of Design, as part of a food-oriented series curated by Jutta Friedrichs, Elizabeth MacWillie, and Sara Hendren, then students in the GSD’s Art, Design and the Public Domain program, headed by professors Sanford Kwinter and Krzysztof Wodiczko. The second event, entitled “Sensing Terroir: A Harvest Food Opera,” took place in November 2012 at chef Jason Bond’s Bondir restaurant in Cambridge, sponsored by Artists in Context; this event incorporated field recordings and interviews with local, organic food providers and sought to explore dining as a narrative form to tell the story of sustainable food practices. The third event, entitled “Beside the White Chickens: A Summer Food Opera,” took place in July 2013 at Bondir and highlighted chef Bond’s local, organic poultry providers, Pete and Jen’s Backyard Birds of Concord, MA, taking as its inspiration the William Carlos Williams poem, “The Red Wheelbarrow,” around which sous chef Rachel Miller organized the menu.

Disclaimer: renderings of individual textures are interspersed throughout this article. Please note, however, that at a food opera event, none of these would be heard in isolation, but they would merge in space with twenty-five other textures at various stages of deployment to create a dense, rich soundscape.

Space and Massively Multichannel Sound

Pam Joy and Margaret Experiencing Food Opera

Diners enjoying Chicken Galantine au Foin, Lions Mane Mushrooms, White Sage Peach Confit, Broccoli Leaf and Rhode Island White Flint Corn Flour Polenta, Charred Eggplant, Snow Bok Choy, and Cherry Tomatoes in Tomato Honey with corresponding music. (Note the placement of the speakers on the table.) Photo by Jutta Friedrichs.

An overriding goal throughout this project is to respect the integrity and the history of the experience of communal dining: the focus is at the table, not on a stage situated elsewhere in the dining environment. Diners are free to converse during the event. To quote Brian Eno, the music should be “as interesting as it is ignorable,” a statement that also parallels chef Jason Bond’s observation of the manner in which people enjoy his food. The sound is there to enhance and supplement the meal, not to supersede or distract from it. In our conception of food opera, the very physical presence of a live performer alters the calculus of the dining experience to an unacceptable degree; this is why we rely on recorded sound for our event. (However, because the sound is being deployed in real time, it exists, as all video game soundtracks do, somewhere between recording and performance. The combination of video game music techniques and multichannel sound diffusion are the technologies that enable food opera as we define it to exist.)

In our second and third events at Bondir restaurant, each diner has a dedicated speaker through which she or he hears the accompaniment to her or his meal. The small speaker, about 3 inches in diameter, takes the place of a traditional table centerpiece (e.g., a vase of flowers) and rests in a small speaker stand designed by Jutta Friedrichs, which allows two speakers pointed in opposite directions to be directed at two diners facing each other. Bondir seats twenty-six diners, so twenty-six channels of sound are deployed simultaneously to individual diners throughout the restaurant. Six additional speakers (four around the perimeter of the room plus two custom hemisphere speakers, designed by Stephan Moore, positioned on stands in the middle of the room) play a slowly evolving low frequency drone that provides a harmonic context for the musical textures of each diner’s discrete accompaniment.

We do not use any particularly focused or directional speaker technology, and in fact, it is not particularly desirable for our planned experience. Sounds from nearby speakers will naturally blend together, providing a rich field of music, in effect transforming the entire restaurant into a generative sound installation. At our second and third events, diners could make reservations anytime between 5pm and 9:30pm, such that, at any given time, different people in the restaurant would be at different points in their meals; this is why the harmonic and rhythmic coordination described above is particularly important. The overall form of the piece is an aggregate of every table’s individual dining trajectory, which results in an emergent foreshadowing and recapitulation as other diners in the restaurant order the same dishes at different times.

Mimesis and Abstraction

The primary sound sources in our first three events are traditional, acoustic musical instruments: flute, clarinet, bassoon, viola, cello, accordion, a tourist nyatiti from Kenya, a Chinese hulusi, and miscellaneous percussion. This may seem an unnecessary restriction, as the sound is played through speakers, allowing any electroacoustic sound to be employed. We used traditional musical instrument sounds to create an association with the alchemy of cooking: recognizable physical materials are juxtaposed, manipulated, and transformed to an aesthetic end. The sample manipulations we employ are fairly straightforward (e.g., splicing, transposing, applying amplitude envelopes), so that the acoustic source material remains clear.

There is an essential abstractness that is shared by food and music (distinct from visual art and writing, for example), which we sought to underscore. One of the original inspirations for this whole project was to explore the idea of using sound to describe taste, bypassing words entirely. It is difficult for the taste of asparagus, like the sound of a cello, to be “about” anything other than itself. As in dining, mimesis in acoustic music is by far the exception (e.g., the thunder at the end of Berlioz’s “Scène aux Champs” from Symphonie Fantastique).

In addition to these abstract sounds (abstract in the sense of an abstract photograph) intended to evoke or compliment the sensory experience of each dish, the second and third food operas also incorporated field recordings and interviews from the farms that supply Bondir’s ingredients. With the field recordings, the goal was to get people thinking about where their food comes from, using sound to connect diners at Bondir to the farms on which the ingredients for the meal were produced. And in the interviews, diners could hear directly from farmers stories about the labor that goes into producing organic produce, the challenges of sustainable farming, and the histories of obscure heirloom vegetables (such as the Waldoboro green neck turnip).
Interviews accompanied only some of the courses, and the ambient sound served primarily as buffer or interstitial behavior between courses. (After we finished our setup and sound check, but before the first guests arrived, the room was filled with 26 channels of chickens clucking, which was a pretty fantastic sound, and remarkably evocative of the experience of walking on Pete and Jen’s farm.) But since different diners are on different courses at the same time, these three modes of listening (abstract [or conventionally musical], mimetic, and linguistic) were almost always active somewhere in the space and contributed to the richness of the experience.

Video Games in the Restaurant

As video games are inherently organized around the unpredictable input of players, video game music has evolved to be uniquely sensitive to real-time input. In a video game, any event may be associated with a musical parameter, and this concept of mapping between two different types of data is at the core of video game music, and indeed, of digital art in general. These mappings have evolved to be quite sophisticated and can be thought of as the points of convergence between two autonomous but linked systems.
The challenge of composing music for real-time deployment may be summarized as follows: deciding what to do when an event is received (typically the managing of a musical event or transition), and what to do for the indeterminate amount of time between events (typically some kind of continuous or state-based musical behavior). The simplest solution, and historically the most prevalent, is to simply loop a piece of music indefinitely until an event is registered, at which point the music fades to silence over a certain amount of time, while a new piece of looping music may fade in. The disadvantages to this approach, however, are numerous, including tedious repetition and inelegant transitions.

Many more sophisticated techniques have been developed, and while it is beyond the scope of this article to present a full taxonomy, the techniques that we have incorporated into our food opera work to date include the coordination of musical events along a metric timeline, transposition of MIDI-like note data, and the algorithmic generation of musical melodies, pitch aggregates, rhythms, and phrases. In some aspects, our focus on notes and short phrases recalls the DirectMusic system released by Microsoft in 1999. It also draws from work by aleatoric composers of the mid-twentieth century, including John Cage, Earle Brown, and Christian Wolff. (For more information, see my paper Cell-Based Musical Deployment in Tom Clancy’s EndWar, presented at the First International Conference on Musical Metacreation in 2012.)

In our food operas, the sources of real time events are constrained to a fairly limited range: we observe which dishes are chosen by a diner, when each course arrives, and when it is finished. The number of courses is fixed (four for our first food opera, five for the second and third), and for most courses, the diner has a choice between two dishes.

In addition to the events indicating the beginning and ending of each course, the music responds to the elapsed time since the beginning of each course. In most cases, over the progression of each course, there is a gradual decrease in musical intensity, which may be quantified in density of musical events, volume, or number of concurrent layers. An additional behavior, limited in duration, may be associated with the beginning or ending of each course.

These parameters may seem few, but given that diners are coming and going throughout a span of roughly five hours, and that any of the twenty-six diners may be at any point in their meal at any time, the richness and complexity of the overall, emergent sonic environment is significant.

The field recordings and interviews also draw on video game techniques, as recordings of environmental ambiences and dialog are a key component of most video games. Our field recordings are chopped up and shuffled, providing infinite variation. Lines of dialog are also edited into individual files and balanced for volume before being deployed intermittently.

Texture and Musical Granularity

What we mean by musical texture is a convenient middle ground between an individual note and a fully composed musical composition. It is a musical behavior with a clear identity, characterized by instrumentation, melodic contour and duration, harmonic structure, voicing style, rhythmic density, and number of constituent layers. While textures may contain melodic elements, they are not primarily melodic in character. Our textures are designed to continue indefinitely without repetition, using algorithmic techniques to vary, shuffle, offset, juxtapose, and in some cases generate musical material.

Game music may be categorized according to the granularity of its musical components, from through-composed pieces of several minutes in duration to individual notes (or even, in considering real-time synthesis systems, subdivisions of notes). Granularity is linked to responsiveness, which is to say, the maximum amount of time that could elapse between the arrival of a game event and a musical system’s response. By orienting music around texture, we achieve a highly responsive musical fabric.

Texture in our usage is independent of pitch, so that the music may modulate to a new key area while preserving texture identity. Some textures incorporate longer phrases, which are categorized according to the harmonic areas with which they are compatible.
We emphasize texture, because it offers a useful way of defining a musical identity without large-scale temporal expectation; a key aspect of our use of texture is that it can continue indefinitely. This allows for sustained and concentrated evaluation, which in turn makes it well suited to pairing with other multimedia input, such as taste.

We find this approach preferable to playing a completed piece of music, which may contain a conflicting or distracting dramatic trajectory (manifested, for example, as the climax of a phrase or a juncture in a piece’s formal articulation), or which may end prematurely, or which may loop unvaryingly, resulting in fatigue or annoyance for listeners due to unmitigated repetition.
This approach is also preferable to simply sustaining or repeating a single tone, timbre, or sonority, since it provides rhythmic and harmonic context for individual tones and allows for the investigation of the modes of meaning with which music has long been associated, including harmony, rhythm, and melody—the syntax and grammar of music.

Coordination and Modularity

The music for the food opera is coordinated in harmony and rhythm and composed so that this coordination would be readily apparent. As most of our diners are assumed to be musical non-specialists, we decided to compose music that is diatonic and organized around a clear beat referential of 188 beats per minute.

All rhythmic activity is coordinated in reference to a common pulse that is present throughout the work. New events are, for the most part, quantized to start on a beat (although random delays of a few milliseconds may be introduced as a humanizing gesture). There is no concept of meter, only pulse, although events may be assigned to happen on multiples of beats, which creates a kind of local meter within a specific texture.

As the project has evolved over our three events so far, we determined a desire to blur the rhythm at certain points, to avoid an overly metronomic effect. For this reason we introduced the idea of rubato phrases that could be played intermittently, uncoordinated with the underlying pulse.

Over the course of the evening, the underlying harmony modulates very slowly in a random walk among five diatonic key areas excerpted from the circle of fifths: D, G, C, F, and B flat. This provides a sense of harmonic progression and combats key fatigue. A bass drone (generated in real-time from transposed recordings of a viola, a human voice, a synthesizer, and an electric generator used in a chicken slaughtering we observed) articulates this harmonic movement. The bass drone slowly (every forty to eighty seconds) chooses scale degrees from the current tonality (excluding the leading tone) according to a first order Markov chain of acceptable progressions; this allows each texture to be heard with any of six possible scale degrees in the bass, providing additional variation. Key modulations happen every three to eight bass notes (with a weighting towards the short end of that range). We don’t really use MIDI as an interface, but a lot of the system works similarly to the way MIDI does, e.g. handling notes with integer ID numbers.

Our musical textures are transposed according to one of two principles. For musical textures that are rendered using this MIDI-like system, transposition is a trivial affair, involving an integer semitone offset. For textures that involve recorded phrases, transposition is slightly more complicated: based on the current key, we choose from a pre-compiled list representing the subset of phrases compatible with the current key. In some cases, we perform real-time transposition of musical phrases. For rubato phrases that do not need to be rhythmically precise, we have found it musically acceptable to transpose as much as two semi-tones up or down to achieve compatibility with the current key. For phrases that do require rhythmic precision, we may transpose in simple multiples of the original speed (e.g., 0.5, 0.667, 1.5, 2.0) to achieve musical variety and to maximize the contexts in which a recorded phrase may be used (an important concept in game audio development).

As for the musical textures, each can be thought of as an independent algorithmic study unto itself. The only constraint is that it must in some way conform to the current key area and acknowledge the underlying pulse. Textures are composed to be somewhat sparse, with an awareness that they will be deployed alongside other textures, and are designed with registral variation in mind.
The system we used for the first three food operas was programmed in Max. The menu was preloaded into the system, and as diners placed their orders, we input the choices into a patch representing the layout of the restaurant. As dishes came and went, we advanced the musical behavior for each seat. An interstitial behavior, consisting of footsteps from one of the farms providing the ingredients for the meal, quantized to the pulse of the music, played between courses. Ambient sound from the farms played before the first course and after the last.


Soft Poached Egg (already consumed), Cucumber Foam, Lemon Balm Brioche, Prairie Fire Chili, Brown Butter Vinaigrette, and speaker in position in front of the plate. Photo by Jutta Friedrichs.

Scoring vs. Synaesthesia

A question that has come up repeatedly while working on this project is whether the goal is to attain a kind of synaesthesia. I have observed an increase in discussion around the notion of synaesthesia in recent years, and it is my guess that the reason has to do with the importance of mapping in the growing field of digital art. By now, this concept should be familiar to anyone with a passing interest in the genre; mapping, simply defined, is the binding of two sets of data. A classic video game example would be the linking of a vehicle’s velocity to its engine sound’s pitch. In an interactive installation, it might be the association of a viewer’s location in a video image with the volume of an accompanying sound.

Synaesthesia is a hard-wired mapping between the senses. I think of Messiaen hearing a certain sonority as blue-violet or mauve. There’s a notion that a sound has a certain smell, or a color has a certain sound, and it is not a choice or assignation, but rather an inalienable attribute of the thing.

What mapping has in common with synaesthesia is the notion of consistency; once established, there is a fixed method for converting one kind of information into another. The difference between mapping and synaesthesia is that, whereas synaesthesia is somehow hard-wired, a mapping is assigned (we might just as well say, “designed,” or “composed”). This creative aspect of mapping, of system design, is only slowly gaining recognition as a field of artistic activity, but it is not far removed from an aleatoric composition like Christian Wolff’s Burdocks, for example. Tatsuya Mizuguchi’s video game Rez is a famous example of an effort to suggest synaesthesia via a tightly coordinated merging of visuals, music, and (if you bought the Trance Vibrator peripheral) haptic information. On the other hand, there is a preponderance of software that makes it possible to completely ignore this creative exercise, resulting in some very blunt conversions; the famous Aphex Twin example works as a goofy in-joke, but countless others fall flat from what amounts to a lack of understanding of basic artistic materials.

However, neither of these perspectives is quite what I’m attempting to do with the food opera. Instead, I come at it from the perspective of scoring. In scoring a film or choreography or any other time-based medium, the task of the composer is not to simply convert the visual information into sound. Rather, the idea is to add an additional layer of meaning on top of the original source, to create an additional stream of information. (In fact, this is a common exercise for beginning film scoring students at Berklee College of Music, where I teach: to take a film clip without sound and use music to give it several different emotional spins, now comical, now tense, now poignant, for example.) The result is a kind of counterpoint, in which this new stream contextualizes and informs the original information. There’s a potential for ambiguity, which can provide richness and nuance to someone experiencing the work. To simply replicate it in sound would be redundant.

So it is with the food opera; I am presenting not so much the sound of Chicken Galantine au Foin with Lion’s Mane Mushrooms, White Sage Peach Confit, and Broccoli Leaf; but rather a sound of Chicken Galantine au Foin with Lion’s Mane Mushrooms, White Sage Peach Confit, and Broccoli Leaf. Of course, there is an intense effort to present a score that supports the meal in a meaningful way, but the associations are intuitive and arbitrary, and any number of alternate solutions to this compositional problem may be equally valid.

Future Work

We have a lot of ideas about how to take this idea forward; as I mentioned earlier, the expressive possibilities are vast.
On the technical side, there is enormous potential to increase the amount of input into the system generating the music; we plan to explore this potential in future food operas via a range of sensing mechanisms. Accelerometers could be attached to stemware or silverware, or cameras could be mounted above the tables, to name only the most obvious subsequent steps. In some situations, it might be desirable to have live musicians performing remotely, similar to what is currently done in some Broadway productions, incorporating real-time scoring and cueing systems.

There is also considerable potential to expand the concept in terms of visual art, incorporating projections and video displays, by coordinating table settings, clothing, and other elements of interior and set design.

Food Opera – Four Asparagus Compositions from Jutta Friedrichs on Vimeo.


Ben Houge

Ben Houge. Photo by Jutta Friedrichs

Ben Houge is an artist working at the nexus of video games, sound installation, digital art, generative video, music composition, and performance. Also an active composer of choral music, his Lao Zhang was premiered by Dale Warland at the American Composers Forum’s ChoralConnections conference in 2012, and other choral compositions have been commissioned by the Esoterics and the Minnesota Compline Choir. He holds degrees in music from St. Olaf College and the University of Washington and recently moved to Spain to help develop and teach in the new Music Technology Innovation master’s program at the Valencia campus of the Berklee College of Music.

Troy Herion: Sonic Imaging

The concluding work on coLABoratory, the American Composers Orchestra’s April 5 Zankel Hall concert, was an extremely effective symbiosis of music and film called New York: A City Symphony by Troy Herion. Throughout its roughly fifteen-minute duration, audience members occasionally gasped or laughed—not a frequent occurrence at a performance of contemporary classical music. I know I was at the edge of my seat for most of it. And at the end, the audience gave the most resounding applause that I had ever witnessed following an ACO performance. So was that reaction due to the music, or was it because they were watching a movie? Ultimately, it was a little bit of both.

Admittedly, it is not out of the ordinary for films to make us laugh or cry or to keep us completely riveted as we anticipate what will happen next. But often part of what makes the cinematic experience so effective is the musical soundtrack that accompanies the visual images we are watching on the screen. The most celebrated motion picture directors were extremely aware of this and chose the composers they worked with very carefully—think Eisenstein and Prokofiev, Alfred Hitchcock and Bernard Herrmann, or Federico Fellini and Nino Rota. In more recent times, it would be difficult to imagine Stephen Spielberg’s adventures being as quite as epic without John Williams’s exultant orchestrations or David Lynch’s narratives being nearly as creepy without Angelo Badalamenti’s deceptively serene harmonies. Peter Greenaway famously cut his films to the music that Michael Nyman wrote for them and, in that reversal of the usual process, further solidified the painterly quality of his work. Directors from silent era icon Charlie Chaplin to horror filmmaker John Carpenter occasionally created their own music for their films, further heightening how crucial the sonic element was to their particular cinematic visions, and French nouvelle vague pioneer Jean-Luc Godard’s approach to sound in his films has been so idiosyncratic that he has been frequently dubbed a composer in his own right as well.
Troy Herion, however, approaches this creative fusion from the other direction. When we visited him in his Brooklyn apartment, a pair of vintage keyboards immediately caught my attention as did piles of CDs. Other than an extremely well-crafted table, a large provocative painting on the wall, and an art object that was a cross between a camera and a can of soda (a gift from a friend), there was little evidence that this was the pad of someone who made films in addition to making music. Largely self-taught as a filmmaker but heavily trained as a composer (he has an MFA from Princeton and is currently enrolled in the Ph.D. program there), Herion’s interest in making movies grew directly out of making music. It was a way to further extend the possibilities of what music can be:

I definitely think of myself as a composer first. I’m a composer who works with sounds and images. I’m learning the techniques of a filmmaker, but I work 95% with the instincts of a composer….I start with the musical impulse and everything else comes from that, even though they end up affecting one another. If I have a musical impulse that makes me think of an image, then I capture that image and it’s different from what my imagination was. So the real image will then change the music that I originally thought of and it becomes this feedback loop….By doing it myself, things stay in this intuitive state….But I think that music can be really anything. It’s an attention to a certain type of balance, a certain type of consonance and dissonance of material….Anything can be a musical appreciation; it’s how we direct our attention.

The first large-scale manifestation of Herion’s concept of “visual music” is his Baroque Suite, in which a group of dancers filmed in a series of tableaux that evoke Baroque-style paintings is fused with similarly Baroque-inspired music, albeit scored for a band including synths, electric guitar, and drum kit. As a result, although its five movements sport such period titles as “sarabande” and “gavotte” and were derived from these centuries-old dances, the work feels very contemporary, particularly in its sonic kinship to neo-prog rock. A signature device in Herion’s musical language that comes directly from his immersion into filmmaking is to subvert expectations by playing with people’s familiarity with various musical genres.

One of the things I ask myself—I’m critiquing my work as I go forward—is, “Do I care what happens next?” Even though I don’t know what happens next yet. This is something that I think that makes syntax very important. If you’re working in a style, you have an expectation of what will happen next. But if you don’t have any syntax from a previous style that people have already become accustomed to—Baroque music or classical music or rock music or whatever it is you are using—you have to generate your own, generate some sort of momentum so people can predict what’s going to happen next and then you can divert or fulfill that….I’m influenced by cliché almost. I look for opportunities to set up a cliché on purpose. I’ll try to make something almost boring. Boring is when you know what’s going to happen next but it takes too long to get there. I try to find that point right before you tune out, but you have an extremely strong projection of what’s going to happen next. At that point I feel like I have a common experience with the audience, and that’s when I like to twist it. And I think that resembles a joke, but it’s really something that holds your attention.

Despite Herion thinking of himself as a composer who makes films, some of his recent films have featured the music of other composers. He and his girlfriend Elan Bogarin fashioned what could best be described as a music video around the pianist Michael Mizrahi’s recording of Marc Dancigers’s The Bright Motion. The Dark City, a poignant rendering of the aftermath of Hurricane Sandy (also created in collaboration with Bogarin and which was featured in The New York Times’s Metropolitan Diary), used Franz Schubert’s music.

“I’m interested in advocating for music,” Herion explains. “It was really exciting for me to take Marc Dancigers’s music and Michael Mizrahi’s playing and create a film around that that was interpreted by a musician and composer—I felt it was an analysis of the music visually. While I didn’t generate the music, I felt that I was very close to the music and I put on my composer hat [to think] about the deeper meaning of phrases. Those are the details that are often not prioritized by people who are just filmmakers and not musicians.”

New York: A City Symphony is clearly the most ambitious synthesis of his musical and cinematic ideas thus far. For him, the visual and sonic elements form a seamless whole and are really not intended to be experienced independently. Even his use of the term “symphony” is multidisciplinary. Though the term carries significant weight in music history and Herion’s symphony calls to mind such elaborate programmatic works as Hector Berlioz’s Symphonie fantastique or—closer to our own time and place—Michael Daugherty’s Metropolis Symphony, Herion is also very mindful of the tradition of “city symphonies” made by filmmakers around the world since the 1920s:

Using a term like symphony I realize is a loaded term. It means a lot of things in the musical world and I sacrificed a little bit of what it means in the music world for what it means in the cinematic world. A city symphony is a whole genre of filmmaking. Why did filmmakers in the 1920s call their films symphonies? Was it for the epic quality of their films, because they were associating that with the term symphony, or was it because they were trying to conjure a musical interpretation?

The other tradition that Herion had to make peace with was New York City itself. Originally from Philadelphia, he’s only been living here for the past three years. At first, being a relatively recent transplant made conveying New York City seem too daunting a task, but eventually his enthusiasm for his adopted home took over and it shows. New York: A City Symphony captures simultaneously the overwhelming grandeur and non-stop energy of this town in ways that only a handful of other pieces of music do—scores by Gershwin and Bernstein, perhaps, or Charles Ives’s Central Park in the Dark.

“I was nervous when I was making the piece,” Herion confesses. “I’d go back between being very confident—that’s a really great shot, I nailed it—and then another part of me would be like, ‘How dare I comment on that! I’ve only been in New York for three years; I don’t have enough New York cred to get into the dirt here.’ But there is a whole culture of New York which is immigrants. Everybody’s an immigrant to some extent. So I focused on the idea that anyone is a New Yorker as soon as they get here. Nobody cares how long you’ve been here. If you’re taking up space, you’re a New Yorker almost. And the other thing I tried to focus on was how impressive New York is when you haven’t lived here so long that it starts to melt into the common experience. It’s spectacular to your senses—the architecture, the sounds, the activity; it’s almost maddening if you pay too much attention to it, so it’s in our interests to tune it out a bit. But I tried to say, ‘Let me keep this heightened awareness for as long as possible.’”

That heightened awareness of visual images as well as sound and how these two sets of sensory information can feed off each other makes Troy Herion’s creations some of the most interesting “music” I’ve heard (and seen) in quite some time.

The Realm of the Senses

“We search for visual beauty in art and in nature, and take care to arrange our homes in a way that pleases the eye. We seek out new music and musicians to add to our CD collections; perhaps we have learned to play an instrument ourselves. We spend time and money on sampling new and exotic cuisines, even learn to cook them. We pamper our sense of touch with cashmere sweaters, silk pajamas, and crisp linen shirts—we can hardly help refining it through our constant interaction with an infinitely varied tactile world. Yet most of us take our sense of smell for granted, leaving it to its own devices in a monotonous and oversaturated olfactory environment. We never think about its cultivation or enrichment, even though some of life’s most exquisite pleasures consequently elude us.”

—Mandy Aftel, Essence and Alchemy: A Natural History of Perfume (North Point Press, 2001; re-issued in paperback by Gibbs Smith, 2004)

“Some say Fougère Royale [first issued in 1881] is to fragrance what Kandinsky’s first abstract gouache of 1910 is to painting: a turning point. Even the tongue-in-cheek name (‘royal fern’) announces that the game has changed: ferns, of course, have no smell and there is nothing royal about them. […] Fougère Royale starts the way some Bruckner symphonies do, with a muted pianissimo of strings, giving an impression of tremendous ease and quiet power. It does smell of coumarin, to be sure, but it is also fresh, clean, austere, almost bitter. […] But wait! There’s a funny thing in there, something not altogether pleasant. It’s touch of natural civet…”

—Luca Turin, The Secret of Scent (Harper, 2006)

“Dune [by Dior] is a strong contender for Bleakest Beauty in all perfumery […] dissonant but interesting […] [I]t’s as if every perfumery accord had become a Ligeti cluster chord…”

—Luca Turin in Perfumes: The Guide by Luca Turin and Tania Sanchez (Viking, 2008)

Despite all of the marvelous experiences in other cities I had over the past two months, the event that still remains most vivid to me is Green Aria—a Scent Opera, which I caught during my two day return to New York City in between flights. My encounter with that completely unique artistic creation triggered a ton of thoughts at the time which have only gotten more elaborate as time has gone by, and have been even further fueled by the writings of perfume guru Luca Turin. In one of life’s inexplicable yet fortuitous coincidences, in the guest room I stayed in several days later in Toronto there were copies of both of Turin’s books as well as Chandler Burr’s biography of the man. Sneaking fleeting glances at these books during the few hours I was actually in the room hooked me into wanting to learn about olfactory concoctions and to see if my experience of them could in any way be analogous to the way I perceive visual art, literature, film, or any of the performing arts—obviously, most of all, music.

So still immersed in Luca Turin’s The Secret of Scent (which I ordered along with several other nasally-oriented volumes upon my return) and armed with notes culled Perfumes: The Guide (which Turin co-wrote with Tania Sanchez) plus a list of perfume retailers in New York City, I embarked on a completely new (at least for me) experiential adventure—a jaunt through various perfume counters in an effort to figure out how much I could distinguish about the various offerings on hand.


A few of the many scents available for sampling at one of the many NYC branches of Sephora. But it is extremely difficult to process a large amount of olfactory information.

I started my quest at one of the many local branches of Sephora which allows visitors to sample a great many of the perfumes on sale without terribly forceful sales personnel hovering over you to guilt you into buying something. Conveniently placed smelling strips placed near the “tester” bottles allow store visitors to get a sense of what different perfumes smell like in the same way that headphone kiosks allow curious listeners to check out the latest recordings. So far so good. Perfumes in Sephora are divided by male and female fragrances—there’s definitely nothing analogous to that in a record store—and within those divisions arranged by company name. To me it seemed the equivalent of alphabetizing CDs by record label or music publisher. But since perfume composers (they actually use the same word!) are even less known to the general public than the composers that usually get talked about on these pages, it must be conceded that sorting fragrances by their creators would make little sense to potential consumers.

I wound up visiting three different branches of Sephora since the stock is not consistent from store to store, but eventually found my way to the first floor of Macy’s which I suppose is analogous to a Metropolitan Museum of Art or Lincoln Center for perfume. At Macy’s, specific areas are consigned to specific manufacturers whose representatives hawk their wares quite aggressively. I’d argue that the scene there is way more intimidating for a novice potential perfume audience member than attending an exhibition or an orchestra performance would be for a neophyte museum visitor or concertgoer. At the same time, you might be more likely to find a remarkable aroma in Macy’s than at Sephora. Of course, there are even higher end places to seek out the perfect scent such as Bergdorf-Goodman or specialty shops for various manufacturers which I suppose are analogous to blue chip art galleries or historic music shrines like Carnegie Hall, the Village Vanguard, or La Scala.

But after nearly two hours of sniffing, my nose was completely overwhelmed. Admittedly I’m a complete amateur at this sort of thing, but I would contend that is more difficult to stay focused nasally for long periods of time than it is to read for hours on end, look at floors of museum walls, or listen to music for hours uninterruptedly. And there’s a real bodily limitation in addition to the perceptual one. Despite my overhearing someone in the music business claim that being exposed to minimalist music made her physically ill, it seems physically impossible (though probably not psychologically so) to have an allergic reaction to music, whereas I imagine the same is not true for perfumes.