Tag: synaesthesia

Do You Hear What I See?

Allen Otte's set up for Begin Again (photo courtesy Allen Otte)

I spent my youth playing notes on a page. And if you’re reading this, there’s a good chance you did too. This notation, particular for what we think of as Western music, is merely one graphic, albeit specific, representation of musical sound. And some of it is quite pleasingly arranged on the page, with calligraphy and shaped staves. But connections of music to visual art are as old as music notation itself.

Chant was notated with beautiful framing on the pages. Mussorgsky’s Pictures at an Exhibition translates the paintings of Richard Hartmann just as Debussy’s La Mer is a sonic response to Katsushika Hokusai’s Great Wave Off Kanagawa. William Grant Still took as his subject works by Richmond Barthé, Sargent Johnson, and Augusta Savage in his Suite for Violin and Piano. Gian Carlo Menotti broke through his writer’s block when he visited Botticelli’s Adoration of the Magi to come up with perennial holiday favorite Amahl and the Night Visitors, and Lady Gaga was likewise inspired by the same artist’s Birth of Venus for her own “Venus.”

These visual connections give the listener a starting point for understanding, which is especially useful in the field of experimental music. What is unidentifiable sonically can trigger a memory or a feeling when it’s attached to a visual. A visual that inspires the composer or improviser is sure to also inspire audiences to a fuller and more moving experience.

A visual that inspires the composer or improviser is sure to also inspire audiences to a fuller and more moving experience.

The Kentler International Drawing Center is driving this connection home with its now-touring exhibition Music as Image and Metaphor. The Kentler Flatfiles have been accessible to Brooklyn visitors for three decades, and curators planned to bring a selection of the collection to the Bartlett’s Center in Columbus, GA this past year. This would have combined with performances by composer/pianist Michael Kowalski and percussionist/composer Allen Otte via the music department at Columbus State University.

In a dilemma familiar to many last year, by October 2020 it was decided that the plans had to change. But Kowalski and Otte did not completely abandon the concert – they instead created a lasting musical installation, able to reach far more visitors than a single performance, with an opening in January 2021. For 40 pieces from the collection, Kowalski and Otte would create individual short musical responses. 40 new pieces of music, connected to visual works, accessible in the gallery and also online. A setup that allows the visitor to absorb themselves in the aesthetic conversation, or, exist within the infinity mirror of creativity.

Both Kowalski and Otte, as well as curators David Houston and Florence Neal, were happy with the result, and now the exhibition is headed to the Ohr-O’Keefe Museum in Biloxi, MS this month.

Allen Otte is a member of the Percussive Arts Society Hall of Fame. With the Black Earth and later Percussion Group Cincinnati he has been on the cutting edge of percussion-based chamber music. (Note: the author is a former student of Allen Otte.) Michael Kowalski was a pioneer of computer-based composition, who moved from chamber music to opera when he founded The Postindustrial Players. The two overlapped as students at Oberlin, and have collaborated before. But while being quite like-minded artistically, their approaches could best be described as opposites.

Knowing the likely answer, I asked both men if it was easier to write one 20-minute piece or 20 one-minute pieces.

Otte found the episodic nature delightful. “I could boom, you know, get an idea, make a response and not be responsible for actually much more than than the idea and the response. And in a minute or 90 seconds, it’s gone.” Percussion being an area where less is more in many cases likely made this more intuitive. “If it were twenty one minutes from me, I would have been uncomfortable,” he said. But he had expected Kowalski, who lists “composer” first among his occupations, to keep the game at a high level.

Kowalski agreed that the two are of a different mind, and thinks an attentive listener could take note of different kinds of craftsmanship happening. But that’s part of the fun, “because you don’t get in one person’s groove and stay there. It takes 45 or 50 minutes to actually hear the whole thing. If you just walk through the show and spend a minute on every piece, that’s how long it would take.”

Guests can take a tour through the exhibition, listening to pieces inspired by each piece of art. There is no stated theme, and no planned progression. The locations in Columbia and Biloxi are set up differently, with the images in a different order, so if a story can be extrapolated, it will be different than any other version of the exhibition. This includes an online visit, which can of course be in any order one likes.

In the compositional process, nearly opposite approaches were both successful.

Kowalski outlined specific procedures for himself, almost like a game:

Music as image:

  • Provide a soundtrack (as if the image is a film) or
  • Use the image as a graphic score

Or music as metaphor:

  • If the artist were making music, what would this image sound like? or
  • Enter a dialogue with the visual art

Random selection of these approaches created structure – more of a puzzle to solve and less of a blank page. And he applied these four procedures with a simple shuffle of the deck – mostly sticking to whatever process came up, no matter the image.

Otte was more intuitive, keeping a chart of the images he had an immediate reaction to, and curating himself from there: asking “whether I was doing too much of one kind of thing and whether I really ought to find a way to push myself to think about a piece in a different way.”

Both Otte and Kowalski spent time studying with composer Herbert Brün, who was a pioneer of graphic notation, and who is also represented as a visual artist in the Flatfiles. In Otte’s hands, responding to Brün’s piece was unexpectedly his most difficult assignment.

Three computer generated graphics by Herbert Brün

Three computer generated graphics by Herbert Brün–Orchestra Model One (1971), Ensemble Analogue Four (1974), and Web I (1971), image courtesy Kentler International Drawing Center.

“Herbert’s piece was one of the hardest ones to do and one of the last ones that I came up with,” he said. But also pointed out that throughout the project, difficulty often yielded a better result. This is possibly because some of the pictures presented a challenge, or because the challenge demanded more time be taken, and led to more self-questioning. Of Brün’s work he noted, “Well, actually, that’s the one that’s somewhat strong, that has some substance to it.”

Hear Otte’s response to Brün’s Orchestra Model One here.

For Kowalski, who is a white man, this challenge came in the form of an image of musicians at New York’s iconic Five Spot by biracial artist Robin Holder. His randomly selected procedure was to create a soundtrack – something that could easily have come across as an appropriation.

Five Spot 2 is one of the more literal images in the entire exhibition, so there was no way to ignore what was in it.

Five Spot 2 is one of the more literal images in the entire exhibition, so there was no way to ignore what was in it. “I had to be honest and embrace that. So that was a toughie.” So in this one case, he did break his procedural “rules,” writing what he felt was a more appropriate musical response. He also recruited an ensemble. Once again, having to think a little harder being a good impulse “that just forced me to come up with something else, maybe something better.”

Robin Holder's drawing of a group of jazz musicians playing instruments and singing.

Robin Holder: Five Spot 2, stencil monotype, 22″x30″ (2005), image courtesy Kentler International Drawing Center.

Hear Kowalski’s response to Five Spot 2 here.

While the ensemble was an enjoyable addition to the project, one standout is fairly minimal, as Otte responds to art by Mary Judge using just an amplified pencil.

The museum’s notes call the music “often surprising, sometimes baffling, always illuminating.” The connection between the 40 works chosen (out of 2000 options) by David Houston and Florence Neal is up to the beholder.  The same can be said about the pieces of music.

Otte felt a connection with the works by relating to what he called the performative aspect of an artist–the idea of still engaging an audience while the visual artist’s work remains still. Whereas Kowalski found a kinship with the act of creation – making a picture being analogous to making a sound. Different results, but the mindset implies a similar procedure.

All of which are ideas that can apply to other visuals when they combine with music–especially dance, where both Otte and Kowalski have a great deal of experience.

“I can only say that I’ve been, more often than not, astounded at what dancers are hearing in music and how they experience music and it’s often fascinating,” Otte said. In his experience dancers may give apologies for not “knowing” an appropriate musical term, while their assessment of the piece is generally quite insightful.

Kowalski also noted the complexity of choreography as a visual form: existing in three dimensions and moving. “If you’re sitting beyond about row 12, you’re seeing a great deal of usually very complicated forms, tracing patterns, on a fairly large stage.”

A previous collaboration between the two featured this interaction. Kowalski wrote a piece for the Percussion Group Cincinnati called Rebus, which includes choreography with flag signals. Initially composing a storyboard, once again the visual existed before the sounds. But, that piece was quite concrete – something Kowalski has always found essential working with dancers.

“Unlike musicians, dancers don’t notate, usually they don’t go into a rehearsal with a bunch of things in their head already,” he pointed out. “They work it out. It’s a very different way of working from most musicians that I know.”

There is no one right way to do a project like this. But like any collaboration, trust must be involved. If the creators are open and welcoming to each other’s vision, then brilliant combinations are possible. If we were to call the visual and the musical participants “sides” of the equation – the sides have to balance, and be somewhat open to the other’s contributions. Kowalski describes this as a tension, much like a conversation. But to be successful, each factor, visual and musical alike, must point to the other.

There is no one right way to do a project like this. But like any collaboration, trust must be involved.

“Some people dig it more visual, and then they get into the music and the other people the other way around, and I just think that’s ideal,” he explained. “I’m very happy about that.”

Despite their different approaches, both musicians planned and charted and graphed to create each of these responses. Otte describes the planning as a math problem. “The calculations that went into that final one minute; that final 60 seconds repeated for each of us 20 times in one way or another.” But also occasionally the minute of music came quickly and easily. “The ones which just came in in some burst of fun, we stuck with a few of those.”

Otte and Kowalski will be live at the Ohr-O’Keefe Museum for a talk and performance of even three more premieres. Forms of falling dust is a work for prepared yang-qin by Rachel C. Walker, a former student of Otte. Another collaboration between Otte and Kowalski called How To Compose Yourself involves a fairly frenzied piano part with percussive commentary. And the concert includes a new iteration of Begin Again, a work by Kowalski whose material stretches from the year 1597 to 1977 and now to 2021. In Begin Again a treatise by Thomas Morley was interpreted on an IBM computer by Ed Miller. A 1977 rendition included the voice of soprano Marlene Rosen, and this version it will include today’s additions from Otte and Kowalski.

The act of drawing on decades of material is part of what makes the project feel so substantial. Music originally created on an IBM the size of a linen closet, being watched and heard on a phone that fits in my hand, still feels fresh and new in this context. And while these pieces of music once again come to life thanks to fresh realizations, they also have renewed meaning thanks to the pairing with another artist’s visual material.

Music originally created on an IBM the size of a linen closet, being watched and heard on a phone that fits in my hand, still feels fresh and new in this context.

The clichés about art and music would tell us that the two aesthetic forms are bound to go together. I leaned into one of these, by Jean-Michel Basquiat, in my conversation with Otte and Kowalski.

“Art is how we decorate space. Music is how we decorate time.”

“Decoration,” said Otte. “That’s a loaded word.” Kowalski objected as well.

But at the surface level he immediately conceded that music could be “delightful if it is in fact decorative and entertaining.” And Kowalski identified “entertaining” as a secret word.

“That’s the word that overlaps: ‘decoration,’” Kowalski said. “Decoration is congenial and attractive and so is entertainment when it’s any good, I think. And so I would use the word ‘shape’ instead of ‘decorate.’”

So Basquiat is possibly correct, depending on what the music has to say. Whether or not you can welcome the word “decorate” for a serious piece of music is up to you, just as whether or not a piece of art “shapes” your space. And the fact that we’ve returned to these kinds of philosophical artistic conversations is another sign that we’re emerging from the harshest closure in the history of music with our thoughtfulness intact.

Development: musical image / Michael Kowalski's music sketches for "Untitled" by Kazuhiro Nishijima

Development: musical image / Michael Kowalski’s music sketches for “Untitled” by Kazuhiro Nishijima, images courtesy Kentler International Drawing Center.

As a pandemic-pivot, this project was enormously successful in that some music-making happened at all. While the music world navigates a bumpy road to a new normalcy, this project is quite possibly a model. Not just of the value of interdisciplinary connections, but also one of flexibility and access.

While the music world navigates a bumpy road to a new normalcy, this project is quite possibly a model.

Music as Image and Metaphor has visual and aural elements that are complete statements on their own. It can be experienced at an individual level, at one’s own pace. And it’s available in varying degrees of in-person participation, including online. And geographically, it has been available to viewers in the southeastern USA. While the Kentler Flatfiles reside in Brooklyn, they have been available in this form to viewers in Georgia and Mississippi. Modeling and sparking conversations – musical dialogues – that allow us to grow our audience, our depth as artists, and our own creativity.

40 Flatfiles down, 1,960 to go.


This exhibition of the Kentler Flatfiles includes pieces by the following visual artists: Herbert Brün, Beth Caspar, Phillip Chen, Abby Goldstein, Takuji Hamanaka, Keiko Hara, robin holder, Richard Howe, Hannah Israel, Mary Judge, Kazuhiro Nishijima, Ralph Kiggell, Rosalinda Kolb, Jiří Kornatovský, Robert Lansden, Simon Lewandowski, Jim Napierala, Florence Neal, Margaret Neill, Morgan O’Hara, Gahae Park, Jaanika Peerna, Scott Pfaffman, Orlando Richards, Susan Schwalb, Viviane Rombaldi Seppey, Molly Snyder-Fink, and Hugh Williams.

Understanding the Polychromatic System

We seem to be at an accelerated phase in our musical evolution, where isolated methods of music practice are rapidly multiplying without a framework of integration and orientation for musicians and listeners to grasp. The polychromatic system is one framework of integration for the various scale configurations of micro-pitch music.

Isolated methods of music practice are rapidly multiplying.

The polychromatic system is oriented toward exploring the outer limits of micro-pitch awareness and its expression in music, from a perceptual rather than conceptual (i.e. mathematical, theoretical, analytical) perspective.

The polychromatic system is based on principles of associative synesthesia: learned associations and conceptual/perceptual integration of audible pitch with visual color. Recent research has shown that associative synesthesia can be developed with practice. Music is an optimal area in which to extend the possibilities and potential of synesthetic awareness, both aesthetically and scientifically.


  1. To simplify and encourage musical curiosity—exploration and discovery of infinite             possible pitch scales and their sonic combinations (intervallic or other).
  2. To open new worlds of musical expression, experience, and composition.
  3. To aid in the development of new ways of ‘hearing’ sound/music and the world.

The standard 7-white, 5-black key layout of keyboard instruments.

Imagine a chromatic keyboard where each key is split. The front half of the key plays the conventional chromatic semitone pitch while the back half of the key plays the quartertone in between each front-key pitch (24 pitches per octave). We could distinguish these pitches clearly by assigning the quarter tones at the back of the key to a pitch-color (let’s say violet).

A keyboard layout for a polychromatic system of 24 equal divisions of the octave.

Moving up to 36 edo (equal divisions of the octave) brings new complexity. Now we can imagine a pitch-color above (say, blue) and a pitch-color below (say, red) each chromatic pitch. One way to describe them would be to say that, in the key of ‘C’, C-red is flat-ish and C-blue is sharp-ish relative to the chromatic pitch. The problem here is that the terminology of flat and sharp are embedded in the chromatic language both as a pitch definition (Db, C#) and as a pitch modifier (bb, x). By applying the concept of pitch-color, we can avoid both the confusion in terms and extreme notational complexity (countless, incompatible pitch-modifier symbols) ‘bolted-on’ to chromatic black and white notation.

A keyboard layout for a polychromatic system of 36 equal divisions of the octave.

This describes how I proceeded in trying to work with very high pitch-resolutions. The polychromatic system evolved to allow the creation of a simple notation and theoretical language for writing, learning, and memorizing micro-pitch music within a pitch division scheme of 106 and 72 edo.

A keyboard layout for a polychromatic system of 72 equal divisions of the octave.

The pitch-color concept is not absolutely defined, so the values (in Hertz) of C-red are different depending on the scale division method used. In this way, the language says remains generally consistent, while being adaptable to any conceivable pitch scale. Unique pitch definitions are defined at the beginning of each composition. This enables an efficiency of learning and possibility of using of multiple pitch scales (simultaneously or via ‘modulations’), all under a unifying and intuitive pitch-color concept.

The pitch-color concept is intuitive by way of an easy association of micro-pitch to the visual color spectrum—from infrared to ultraviolet. A simple association can be made with a continuum of flatness to sharpness for each reference pitch (chromatic or other) to the visual color sequence we already know from the experience of seeing rainbows and other light diffraction phenomena.

I use equal divisions of the octave as a method of pitch division because it is a rudimentary and self-explanatory element to begin from in my early explorations of auditory limits of pitched-sound differentiation. My approach is to use the highest possible edo scales within each new keyboard design.

While there is one option for a major 3rd in a chromatic system, the polychromatic system offers several pitch-color variations of that interval.

With regard to intervals, the polychromatic system uses relatively defined pitch-color definitions and is based on an idea of intervallic relativity. So, while there is one option for a major 3rd in a chromatic system, the polychromatic system offers several pitch-color variations of that interval. This intervallic flexibility has audibly compelling implications and effects when exploring variations of each component within increasingly complex harmony.

When listeners hear microtonal music as ‘out of tune’, this impression arises from a chromatic frame of reference. A new foundational frame of reference or perspective needs to be established in order to appreciate microtonal music on its own terms and not in comparison to the macro-pitches of the chromatic system. This foundation is what I have been seeking in developing a polychromatic system. In general, microtonal music can seem extremely abstract for unprepared minds and ears, especially without a new framework of understanding (for reference and comparison).

The polychromatic system builds upon the fundamental concepts of the chromatic system (note definitions, harmonic principles, and music theory) as a common point of reference – and departure. From this common framework, increasingly refined levels of micro-pitch discrimination can be explored within a known system of musical understanding. As greater refinements of pitch and harmony recognition are developed, increased awareness can enable the recognition of further pitch-colors, as entities in themselves, without trying to force them into the conventional frame of reference: a coarse pitch-resolution, chromatic system.

In analogy, imagine the chromatic system as an old monochrome (black & white) dot matrix printer, with its chunky, quantized images. If you input a high-resolution image into that framework, it is ‘processed’ into a chunky, quantized image. The foundational framework of the coarse resolution, dot matrix system design must be addressed first.

A Low resolution dot-matrix image showing very pixellated letters as well as a very pixellated photo close-up of an eye of an animal

Paradoxically, ‘tradition’ has vital importance in the creation of new musical systems. Here, we are talking about tradition as our accumulated experience of the past, a shared frame of reference, an implicit basis and context of listening to and composing music—and not in the sense of calcified conventions of the past and present.

Paradoxically, ‘tradition’ has vital importance in the creation of new musical systems.

Polychromaticism is an approach and practice which uses chromatic music theory as an initial basis for conceptual anchors. In this way, extensions of established tonal and harmonic principles assist in the understanding of the new possibilities and implications of the polychromatic system. This creates a conceptual and perceptual bridge between our current and many future systems of music.

The use of chromatic conceptual anchors comes into play when I am trying to memorize a piece for performance. Using the conceptual anchor of a dominant 11th chord, I remember the constituent pitch-colors of the chord as minor modifiers within this larger conceptual structure, instead of having to remember each pitch-color individually.

My composing process is much more intuitive than analytical. As a result, I have neglected doing harmonic analyses, after the fact, to determine possible new theoretical models. My hope is that others with a passion for analysis will create new music theory models to aid in teaching the polychromatic system more efficiently. Philosophically, I see this as part of a larger 21st-century process which is based on collaborative innovation in music, science, technology, and the sound arts.

As a musician, it seemed impossible to gain a fundamental grasp of microtonal music because it consisted of so many discrete pitch-scale methods, all separately existing without a fundamental, underlying system.

With a polychromatic basis in notation and generalized pitch-color concept, any micro-pitch language could be quickly assimilated without having to relearn scale-specific notations. The current expansion of esoteric and fragmented micro-pitch scale methods, of reinventing-the-wheel (in notation) with each, can make wide exploration within microtonal music intimidating and difficult.

The use of non-chromatic color schemes:

A diagram of various hex-color schemes.

While the color schemes here are different, my impression is that these are examples of a modally optimized layout – in the sense that adjacent keys are 3rd, 4th, 5th, and 6th intervals rather than chromatic minor 2nd’s (or smaller) intervals. In the polychromatic system there are multiple pitch-color variations of each interval, rather than the single fixed-interval value in these layouts. Why be limited to just one major 3rd interval, when multiple intervallic pitch-color choices—singularly or simultaneously—could be available?

Ultimately, the polychromatic system exists to make the process of micro-pitch exploration and creation easier. And just as the polychromatic system has evolved from the chromatic system, it too will eventually become a (legacy) reference and conceptual point of departure for many increasingly sophisticated (non-chromatic) music systems of the future. The polychromatic system is one way of making the musical evolution toward triple-digit, pitch-scale resolutions easier to understand and create with.

As a musician and a listener, I experience music as a dynamic and evolving process, a creative interaction that we choose to engage in. Ultimately, the meaning and value of music comes from the quality and depth of this creative interaction.

Art doesn’t come to us, we must come to art.

The possibilities of growth and awareness gained through our engagement with art remind me of the idea that art doesn’t come to us, we must come to art. This idea expresses both a necessary receptivity to new perspectives as well as an active personal involvement which engages the listener’s creativity and imagination, in a similar way to that of the composer. In this process, the listener becomes a receptive-artist interacting with the compositions of the expressive-artist (composer). If we become what we practice, then exposure to, and interaction with challenging art can help us expand our integrated perpetual/conceptual awareness, and expose us to new dimensions of emotion and insight beyond the limits of spoken language.

In the next article, I will describe technical (physical technique and technology) aspects of my approach to polychromatic music, as well as some discoveries and implications that I have become aware of in my early explorations.

Mary Ellen Childs: Engaging All the Senses

Mary Ellen Childs sitting on a bench on a street filled with plants

While the music of Mary Ellen Childs has a distinctive and recognizable sound, she has long been interested in engaging the other senses as well. Perhaps this is because, when growing up, she was actively involved in dance and theater in addition to playing the flute and piano. In fact, she was choreographing musicals at her high school before she ever composed a note because writing music, at the time, felt “too mysterious and too unattainable” to her. To this day, her experience as a choreographer informs both her own music and the advice she gives when teaching composition.

“I really feel like doing choreography were my first lessons in composition,” she says. “Although the materials are different, the general concepts are the same. When I teach composition, I take that point of view, too. We’re really looking at how to create something that has interest and coherence and surprise.”

Perhaps the area where Childs’s background in dance is most pronounced is in her music for percussion ensemble. She first got interested in composing for percussion as a graduate student at the University of Illinois. “I started to notice how interesting it was to watch percussion players when they played,” she remembers. “They always moved in really interesting ways. … Using movement with percussion is the easiest of all, if you’re going to really exploit movement, because percussionists’ playing technique is already very physical.”

But what Childs ultimately wanted from percussionists went even further than what they usually do. She wanted to them to perform in immersive pieces that are as much about sight and touch as they are about sound. Most of what she wanted couldn’t be effectively communicated through music notation so it needed to be learned through an intensive rehearsal process. So she wound up forming her own ensemble, CRASH, by initially hiring one percussionist and two dancers.

“The reason I wanted dancers was because I knew I was going to have to work in rehearsal and ask the players to remember what they did from one rehearsal to the next,” she explains. “I was worried that if I brought in three musicians that they might say, ‘Where’s my notation? Where’s my score?’ It worked really well because I had one percussionist who really knew how to work with rhythm, and then I had two dancers who really knew how to work with movement and how to work from memory, so I didn’t have to convince anybody of anything.”

CRASH’s idiosyncratic repertoire includes Hands, a piece performed exclusively by the players’ hands, and Click, in which the players navigate a series of movements across three sets of claves—playing their own as well as each other’s instruments. According to Childs, “Part of the rehearsal process is figuring out who’s the active strike and who’s the passive strike—where exactly in the air is that going to be so that you don’t end up either missing, or hitting someone’s knuckle, which is very painful and has happened. You want to minimize that because it’s not pleasant.”

Though Childs acknowledges that it was not very practical, the most unusual piece that CRASH performed involved a series of three exercise bicycles that were turned into musical instruments—the pipe-cercycle (which incorporates organ pipes), the xylo-cycle (in which triggered mallets strike a series of xylophone-like bars right above the handle bars), and the string-cercycle (in which the pedaling motion triggers a variety of strings, including a ukulele and a cello). She is currently reworking another piece originally created for CRASH called Sight of Hand which involves baseball coaching signals turned into percussion. If all goes according to plan, the piece will be performed at a minor league baseball park in August in between innings at a St. Paul Saints game:

The point is having the piece and the live game be part of the same event, then taking all that material and making a piece out of it that exists on video to put my baseball percussion movement into the context of real game plays happening.

Another idea she has been wanting to flesh out for years is a way to merge a musical experience with an olfactory one. Our discussion about the relationship between scent and sound got so involved that we will publish it as an independent post later this month.

However, while engaging other senses has been key to many of Childs’s compositional ideas, listening is still primary. Even though for the premiere of Dream House—her evening-length work for the string quartet ETHEL—incorporated projections on seven different video screens scattered around the space, Childs insisted that the sonic material was foregrounded. “Visual imagery can take over so easily and make music an accompaniment,” she opined to her collaborators. “How are we really going to keep this so that, if anything, the visual material is the accompaniment to what you’re hearing?” Similarly, when composing the music for Wreck, a full-evening dance piece choreographed by Carl Flink for the Black Label Movement, she insisted on creating music that would stand on its own rather than “shrinking violet music.” One of my personal favorite compositions of hers will always be Kilter for two pianos, a work I only know through a recording. Though from its title alone it’s clear that it is inspired by motion, it is sonically ravishing.

A conversation at New Music USA
May 6, 2016—2:30 p.m.
Video presentations and photography by Molly Sheridan
Transcribed by Julia Lu

Frank J. Oteri: The conversations we have with composers and interpreters for NewMusicBox are inevitably always about sound, because that’s primarily what people think of when they think of music. That’s what’s foregrounded, and certainly that’s pretty much the only aspect of music that people experience when they listen to a recording. But I still remember the first performance I saw of your ensemble CRASH during the Meet The Composer’s THE WORKS marathon in Minneapolis in 2002 and how mesmerized I was—not just by what I was hearing, but by what I was seeing, both of which were the result of the tactile connectivity of the performers. This body of work clearly aims to be about more than just sound.

Mary Ellen Childs: I’ve certainly written a lot of pieces that are concert pieces or that now primarily exist on recordings, and those are certainly first and foremost sound and pretty much only that. But I also think I’m very attuned to all the senses and to what the total experience is for someone, especially if it’s a live performance. If you’re there with the musicians, your ears are engaged, but your eyes are also working. All the senses are working. I don’t do so much anymore, but I produced a series of concerts where I thought about the whole experience for the audience from the time they walked in the door. Really paying attention to that, you make certain choices about what the lighting is like when people walk into the theater, how the musicians enter and exit the space, or what happens between pieces—what’s the pacing of that time so that things don’t sag in between and you lose the audience’s attention?

I think that’s just part of my makeup. Maybe it has to do with the fact that I grew up not only playing the flute and piano, but also dancing and working in the theater. In fact, not a lot of people know this, but I used to work as a choreographer before I was writing music. Then when I started writing music, it felt like there was a natural pull for at least some of my music to be in collaboration with choreographers. So I was still keeping one foot in the dance world, although this time as a composer. It was really from all that experience that I started writing movement into percussion pieces. That’s kind of the short arc of how all that came about. Using movement with percussion is the easiest of all, if you’re going to really exploit movement, because percussionists’ playing technique is already very physical. They’re already ready and willing to do all different kinds of things and play all different kinds of instruments. And, for the most part, percussionists are open to a new playing technique that they might learn for a specific piece. So working with percussionists has all those things built in. To write special sticking patterns or to setup the instruments in a certain way or set them up around the space and put the players on wheels—that’s all something you can do in the percussion world.

FJO: Most people play instruments before they start composing, but I had no idea that you had been a choreographer also. What made you decide that you wanted to write music?

MEC: In my heart of hearts I was always interested and curious about it, but it felt intimidating. It felt like you either just knew how to do it or you didn’t. It was like a channel. I grew up on classical music, and so the composers who were revered were often like Mozart, writing operas when he was a teenager. So it just felt too mysterious and too unattainable. On the other hand, choreography seemed like I could figure that out. I could figure out how to move and, with the somewhat limited dance background that I had, I could make things up. Why not? It felt like something anybody could do. And so I did.

FJO: It’s so funny to hear you say that, because to me choreography is perhaps the most mysterious art of them all. Maybe that’s because I’ve made up songs since I was nine, but I never knew how to move around properly. And the folks who think music notation is complicated probably have never seen Benesh notation or Labanotation, which seem more arcane than hieroglyphics. I don’t understand that stuff at all.

MEC: I don’t either, and most people don’t. That’s certainly not a part of being a choreographer for most people. But I don’t know why it felt so attainable to me. Maybe in part because I saw my older sister do it, so I thought, “Well, I can do that, too. Why not?” I also used to do things to get inspiration. I would go to the public library, which was right down the street from where I went to high school on the shores of Lake Michigan, and I would check out these big picture books of dance. I would look at a still photo and I would imagine how they got to that moment and what happened after that moment. I would just imagine from a still shot what the movement around it might be.

FJO: If you’re a dancer, coming up with your own movements makes sense. It’s analogous to playing the piano or the guitar and then improvising and creating your own music. From there it branches out. That’s certainly what happens for a lot of composers: they’ll play an instrument and eventually write something for themselves to play, and then suddenly they write something for other people. But when you write music, it seems like that can grow into something else. You can take a solo piece and then arrange it for string quartet or even an orchestra. But how do you go from one person moving to, say, people moving? As a spectator, I’m never sure what to watch when I’m watching dancers. If I focus on one person I’m losing the larger totality, but if I’m always looking at everyone I lose the details.

I really feel like doing choreography were my first lessons in composition. And I drew upon it when I started writing music, because although the materials are different, the general concepts are the same. When I teach composition, I take that point of view, too. We’re really looking at how to create something that has interest and coherence and surprise.

MEC: But it’s not that different for music. I really feel like doing choreography were my first lessons in composition. And I drew upon it when I started writing music, because although the materials are different, the general concepts are the same. When I teach composition, I take that point of view, too. We’re really looking at how to create something that has interest and coherence and surprise. I probably should tell you what kind of choreography I was doing. I wasn’t doing modern dance masterpieces that came out of nothing. When I started choreographing, I was choreographing the high school musicals and the swing choir. There was already a framework in which what I was doing had to fit, and I had a lot of information given to me when I was creating dance steps—something had to be communicated or something had to happen, and there was already music selected that this had to go with.

FJO: So I think I understand your transition to composing music and why you continued to be involved with dance as a composer, but I wonder when you realized that music itself could incorporate movement, that writing a piece of music could be more than just the notes.

MEC: Let me think about that. When I was in graduate school at the University of Illinois, I took a class in composing for percussion that was taught by a composer and a percussion instructor, both of them together. That’s when I started writing for percussion. I think somewhere around that time, I started to notice how interesting it was to watch percussion players when they played. Whether it was the timpani player or one of the other percussionists at the back of the orchestra or somebody in a new music ensemble, they always moved in really interesting ways.
But the very first piece that I wrote that had movement written into it is a piece called Still Life, which is for three drummers playing nine drums and gongs. The drums are placed in a palindromic line up, so that the center is a small drum—I think now I use a small tom—and then on either side of that, there’s a bass drum, but they’re placed vertically on stands, absolutely vertically, which is not how you’d ever see a bass drum placed for playing. Then moving outside of that on either side, there are two more toms of varying sizes, and on the end of the setup on either end, there’s a wind gong. So the players stand. Player one is in the center and plays the inside of the two bass drums and the center drum, then the two outside players have mirror image setups where they’re each playing the outside of each bass drum, two toms, and a gong. I was able to do things with sticking patterns that would require the outside players to move in one direction or the other direction, and I asked the inside player to do sticking patterns that might be crossing arms. When I was writing it, I was really thinking about how these sticking patterns are going to force the players to move in a certain way.

FJO: Does the music sound different than it would have if they weren’t moving around that way?

MEC: No. In fact, I had written out the diagram and when I went to the first rehearsal, they had already rehearsed a bit. The players had taken the bass drums and laid them flat. They said, “Oh yeah, we put them flat because it’s easier that way.” And so I had to say, “Well, this is not just about the ease of playing, but it’s about how it looks.” We had to have a little discussion, and we turned the bass drums up and of course it’s very, very, very different.

FJO: Sure, but in terms of the question of it sounding different, they might be able to get it to sound the same way doing something else, but certainly the perception of the person watching it is going to be different, and that’s where the other senses come into play.

MEC: The way a piece sounds and how the listener will absorb it is actually different depending on what they’re seeing, even though the sound doesn’t change. If you were listening to an audio recording, that might change. But if you’re live in the performance space and watching it, you actually absorb it differently based on whether it’s under fluorescent lights or whether it’s in a little intimate pool of light that’s really a bit dim. And it’s different based on how the performer is moving. So I do think that makes a huge difference.

FJO: And you use the word palindromic, which is a word that I love and a structural concept that I frequently gravitate toward, so I’m already fascinated by this. You know, composers have created musical palindromes for many centuries, but most of the time you can’t hear that the pieces are palindromic. That famous Machaut motet, Ma Fin est Mon Commencement—My End is My Beginning—is the exact same music whether you play it forward or backwards, but if you weren’t told that and didn’t examine the score with your eyes—

MEC: —You wouldn’t hear it. Right.

FJO: And I doubt many people can hear Webern’s palindromes. But when you create a situation with these percussionists, and you have a palindromic setup and they’re doing this kind of thing, people see it. And they can infer what they’re hearing from what they’re seeing, so you’re able to create certain kinds of symmetries and formal patterns that you might not be able to do if people were left to their ears alone.

MEC: That’s right, although the piece of music itself is not a palindrome. That’s totally unrelated. But visually, there are these mirror image things. And I think maybe one example in that piece—there’s a place where there’s a two-against-three pattern between the two outside players, and they’re playing with one mallet. So one is doing two, and the other is doing it in three. And I think if you put your attention there, you can see that very clearly.

Later on, after that piece was done as a concert piece, I also created a 16-monitor video wall out of it that took the component parts of the piece and broke them up across 16 screens in a square, so you could really focus in on some of that material highlighted in another way by juxtaposing over the top another set of patterns. That’s when I could take that two-against-three pattern and put those two arms, hands, and mallets next to each other so you can see that as you’re hearing it, which makes a really big difference. It’s a very simple example of how what you’re seeing might affect what you’re hearing. I think it’s usually much more subtle and complicated, but that’s a very simple, straightforward example of how what you see changes how you hear or reinforces what you hear or draws your attention to what you hear.

FJO: There’s also a communal aspect to this that I find really interesting. I’m thinking back to the piece Click which, I think, is the first piece for CRASH.

MEC: Um, well Still Life is a piece that CRASH does and Click actually predates the group as well, but we call it CRASH’s signature piece because that is the piece that’s been done and done and done and done and done.

FJO: There’s also a piece called Crash. Is that how the group got its name?

MEC: Yes, there is a full-evening piece called Crash.

FJO: Before we get into the specifics of how the ensemble got put together, I want to talk with you about the communal performance aspects of Click. It’s interesting that the group that originally performed Still Life initially wanted to move the drums and play the piece more conventionally. That would completely miss the point of Click, which is so much about the players playing each other’s sets of claves. The whole piece seems to be about the sharing of resources.

MEC: I never thought of it as a sharing of resources exactly. I don’t know really where that idea came from, though I remember when I got it. But I also want to say, there is some of that sharing of resources—as you call it—in the marching band world where you might see a drummer reach over and play another drummer’s drum.

FJO: And certainly also in many traditional African groups, like the Chopi Timbila mallet percussion ensemble pieces, where you have many people playing on the same mallet instrument and they’re playing interlocking patterns. If one person was out of step, it would throw the whole thing off.

MEC: I was not the first person in the world to think about how visual and aural things go together. There were hundreds and hundreds of years and cultures that thought that way, and so there are lots of musical traditions, especially percussion with visual things or marrying percussive dance and rhythm.

FJO: Probably the closest thing we have in Western classical music to that is piano four-hand repertoire.

MEC: Oh, yeah. Right.

FJO: But, as far as I know, you’ve never written a four-hand piano piece.

MEC: That could be an interesting thing to exploit visually, too, with crossing hands, or intertwining, something like that. But I don’t have a four-hand piano piece, just the two-piano piece Kilter.

FJO: Which is a really great piece.

MEC: Oh, thank you.

FJO: Kilter is word we tend only to use when we talk about the negation of it, being off kilter.

MEC: Exactly.

FJO: So I imagine that there is sort of a motion and a balance idea within that piece, too.

MEC: Definitely, there’s a motion. I was really trying to get at, well, just what you said. It’s a word that’s usually only used in a phrase that means it’s negative. So I was really playing with those two things. Something and its complement. Something and its opposite. And I find that has been something that has recurred and recurred in my work. I think the interplay of seeming opposites is something that interests me a lot. And that’s where Kilter came from.

FJO: To bring it back to Click, what I find so interesting about it is how effectively it realizes a physical process. First you play your instrument, then you play someone else’s instrument. It really is about the group and coming together.

MEC: Absolutely. It’s very much about the group. That’s one of those pieces where it’s really hard for the players to rehearse on their own, although they tell me that when they absolutely know their own part they can sort of do it on their own. But it’s very much about how you interact with the others; it’s not just about putting your stick that way. It’s about putting your stick that way to hit someone else’s stick. So there’s all that sort of figuring out in rehearsals—where’s your stick going to be and when—because the other thing about a piece like Click that is not something percussionists usually deal with, is that you’re often aiming for a striking surface that isn’t there yet. The striking surface is often in motion, too, so part of the rehearsal process is figuring out who’s the active strike and who’s the passive strike—where exactly in the air is that going to be so that you don’t end up either missing, or hitting someone’s knuckle, which is very painful and has happened. You want to minimize that because it’s not pleasant.

Click is one of those pieces that has ideas that have continued to spin off into more and more possibilities. I actually have a couple of other Click-style pieces, and I put together a whole of evening where Click-like material came back as the connecting thread. But those pieces we haven’t done since. That was probably in about ’92, ‘93, ‘91, somewhere in there. It did feel like I could keep mining this material. There’s another piece I did with CRASH called Talking Stick which also uses only sticks as the instruments and, in that case, we used drum sticks. The main figure in that is this thing that I saw one of my players do in rehearsal one day during a break, and I said, “Show me what that is; there’s a piece in there.” That’s how that piece got started. One stick rests against the cheek, and this becomes the resonating cavity. And then the other stick hits the end of that stick. Then as your mouth opens and closes, you get a change in pitch. That’s why it’s called Talking Stick. That’s the recurring theme that comes back, that talking stick pattern. Then there are all kinds of other things that you do with sticks—whether it’s on the floor, or tossing and hitting them, or making sculptural patterns out of them. So that was not a Click-like piece, but it has some similarities with it.

FJO: So, in order to do pieces like that, you formed your own ensemble. I imagine this isn’t the kind of stuff you could put on a piece of staff notation and hand to some group somewhere.

MEC: No, you can’t. That’s right. I actually experimented with notation early on with the history of Click, but it was so complicated to try to get it all down on paper that it wasn’t worth it. Notation should be something that assists in the communication between composer and performer. And what I was finding with the notation I was trying to get down on paper was that it was something that would take the performers much longer to take off the page and into sound and movement when I could just say to them, “Hold your claves like this and do this.”

Notation should be something that assists in the communication between composer and performer.

And that was quick, so that really became the score, with some kind of an outline. I would name the various patterns, so that we could talk to each other without always having to refer back to the number on the outline. I could say, “Start at the triple patty cake” or “Let’s go from the everybody cross.” But this would be meaningless to anybody else except for me and my group, who have that language in common that we’ve created. Since then, after the fact, now I’ve gone back and I’ve written out some pretty detailed notes for each part. But they are sort of a combination of prose-style language and maybe some rhythms written out. They’re meant to be used in conjunction with a video of my group performing it. They’re also never to be put up on a music stand with your eyes glued to it because that’s just wrong for so many reasons. They’re just meant to be referred to. As other groups learn Click, I want them to be memorizing from day one and not be thinking, “First I’m going to read off the page, and then I’m going to take it into my memory from there.” No, this has to be right into movement and sound from the very beginning. Your eyes have to be available.

FJO: There are people who assume that if something doesn’t have a score on the page, that it’s improvised. But this isn’t.

MEC: No, not at all. I think this is one place where my dance background had a big influence, maybe without me even completely thinking about it. You don’t really write down dances for the most part. As a choreographer you may have some notes for yourself, but they exist in the dancer’s memory and maybe now on film or video so that there’s a record of it. But let me back up.

The very first version of Click was at the Yellow Springs Institute in Pennsylvania. I was there for a week with Relâche. That would have been about 1988, if I remember correctly. Because I had the ensemble for half a day, I could work on several ideas. I had them there in the room with me as I was working. So I wrote an instrumental piece for them that was completely written out and which became Parterre. Then I had this idea to do the piece with claves, and I knew I had to have at least three people to make it work. And the three people were the percussionist in the group, Flossie Lerardi if I remember her name correctly, Laurel Wykoff who was the flute player, and Guy Klucevsek who was the accordion player. So I had the three of them, and I think I created maybe just the first minute or minute and a half of the piece. It was really because I had access to the musicians while I was composing.

Then I went home and I knew I wanted to continue with this idea, so I thought I’d have to bring people in for rehearsal. This is not a piece that I could compose in my head and write down and send off to players like I might with a string quartet. So I hired one percussionist and two dancers to be in rehearsal with me to create the piece. The reason I wanted dancers was because I knew I was going to have to work in rehearsal and ask the players to remember what they did from one rehearsal to the next. And I thought, “Well, that’s how dancers work. They’ll get it immediately.” I was worried that if I brought in three musicians that they might say, “Where’s my notation? Where’s my score?” Even if it was just from this rehearsal to the next rehearsal. So that was the reason, and it worked. It worked really well because I had one percussionist who really knew how to work with rhythm, and then I had two dancers who really knew how to work with movement and how to work from memory, so I didn’t have to convince anybody of anything.

FJO: For me, one of the most fascinating of those CRASH pieces is Hands, which doesn’t use any instruments at all. It’s just the body. And because it’s based on such a primal, human activity—just using people’s hands—I imagine other people who see this being performed might think it is something we all could do, but then the minute you start paying attention to it, you realize, it’s actually pretty hard.

HANDS by Mary Ellen Childs from Mary Ellen Childs on Vimeo.

MEC: You know, that piece was a really surprising thing for me. I think it’s only like three or four minutes; it’s very short. It was part of a much longer evening that was all about percussion in various ways, and it felt to me like just a little kind of a filler thing, like something that I was going to use for that purpose and then completely forget about. But somehow that piece has taken off. There are some groups who’ve taken it into their repertoire and have just played it and played it and played it. And it gets this big response that has always surprised me.

FJO: Well, I think it’s because it’s just so basic.

MEC: It is basic.

FJO: The one thing that audiences are allowed to do during Western classical music concerts is use their hands and clap at the end. But here they are being used to make the content of the music that you’re experiencing without any other filter. Yeah, they hit chairs at some point. But mostly they’re hitting each other. And there’s something very satisfying about that.

MEC: Why does that capture people’s imagination? I don’t understand it. But I think that I understand what you’re saying. I’ve done at least one other body percussion piece, Sight of Hand, which sort of draws from clapping games and hamboning and baseball coaching signals turned into percussion, but it’s a little more complex and varied than Hands is.

FJO: I haven’t heard it or seen it, but I’ve read about it and was very curious about it. It’s a new piece, right?

MEC: Oh no, this is from 1998 or ‘99.

FJO: That’s weird, because one of the sites I went to said it was from 2016.

MEC: Oh, what that could be is now I’m going to do a version of this piece with the St. Paul Saints minor league team.

FJO: A-ha! I want to hear about that.

MEC: Yes, so if all the funding comes through—that’s a big if right now—then we’re going to do the piece at the Saints’ game in August of this year. The St. Paul Saints are a minor league team, you know, serious baseball. But there’s also a very fun atmosphere. There are always other things going on in between innings or in the stands. So I had this idea to do the piece that is influenced by baseball coaching signals there at a game. And then we’re also going to shoot video, live at the game, to capture game plays and crowd responses. We’re going to have a second day of video where we can setup some specific shots with Saints players and coaches and my players, and then create a little percussion music video out of it. So that’s the 2016 project.

FJO: There are all these connections yet disconnections between music and sports: the virtuoso, superhuman performers and the ritual of the event—whether it’s a concert or a game that an audience goes to and witnesses. Both sports and music also engender a rabid fandom. Plus there’s a ton of jargon that’s very specific to those two realms. And, as the guy who’s on the music side of this rather than the sports side, I’m always sort of envious of how sports can be as specialized and even as erudite as we are, and still millions of people care about this. So if we could reach sports fans and show them how similar our worlds are, I wonder if this piece is the kind of piece that could do that.

MEC: I don’t know. We’re just in the middle of conceiving how this is going to work in the stadium. It’s actually a pretty quiet piece; body percussion is not going to read very far, though visually maybe it will. We’re also going to work with a group of usher-tainers—they call them—there, people who are hired to be part of the entertainment, and they’re great. I’m going to have them learn just a few of the patterns out of our piece, so that they can participate. We’re going to amplify it in that way. Instead of three players, I’m going to have seven of my own people, and then we’ll probably have about 10 or 12 or so of these usher-tainers. So we could have a pretty big group doing some of these body percussion patterns.

FJO: No amplification?

MEC: I don’t know. I’ve got to think that through. Like I said, we’re still in the planning process. We might see if we can do something on their jumbotron, which would just be a little snippet—certainly not the whole piece—so that everybody can see and hear it that way. But a baseball stadium is not a concert hall, so this is not going to be the kind of performance where people are going to have their attention focused on this piece that, even at four and a half minutes, is still pretty long. We’re never going to get four and a half minutes of an audience’s attention. So we’ll have to conceive of it a little bit differently: breaking it down into component parts; doing a little bit here, going over to that part of the crowd and doing another bit; interacting with a vendor; getting it all on camera.

FJO: Why do you think that it would be so difficult to get them to focus on your piece? They’re certainly capable of being totally focused on the game.

MEC: Right, but they will never give us four and a half minutes. The time between innings—they told me—is two minutes. And they have a lot of stuff going on in that two minutes. So even if we had the whole time between two innings, we couldn’t do the piece. And we probably will only get a fraction of that out on the field with everybody’s attention.

FJO: They couldn’t work extra time into the schedule just to make it work with your piece? Why not give them a little more than two minutes between innings?

MEC: I doubt it, and that’s not really the point of this. The point is having the piece and the live game be part of the same event, then taking all that material and making a piece out of it that exists on video to put my baseball percussion movement into the context of real game plays happening. So I don’t know. We’ll see how it all comes together.

FJO: Another thing that interests me about all of these projects we’ve been talking about is that even though you work very closely with the performers, you’re not actually performing. But you are so involved in the rehearsal process. That becomes tricky if you have pieces that are being done, say, all over the world.

MEC: Well, there are now percussion groups who do my pieces and I’m not part of that rehearsal process. So that does happen now. Early on, I felt extraordinarily protective of these pieces and I did not want anyone else to do them unless I could go and coach it. Then after a while, I just didn’t feel as strongly about that. I think maybe because there were some groups who learned the pieces and put their own mark on them, because every group is going to do it a little bit differently. Even if the rhythms are exactly precise, their quality of movement is going to be a little different. One might have a more precise feel to it, another one might have a little bit more lyrical feel, just based on what the movement qualities of the performers are like. So I got more curious and interested in having performers take on these pieces. And I did find ways where I could convey the information from my written notes plus a video of my program. Now sometimes I even do Skype rehearsals where I can be at a rehearsal enough to see what’s going on and give them some coaching from my own home studio, which often works better.

FJO: But some things would still be very difficult to do without your involvement, perhaps most of all the music you wrote for these crazy bicycle instruments.

MEC: There were three of them, and they’re no longer in my studio. I donated them to the Schubert Club Musical Instrument Museum. So they’re now in the museum in St. Paul. I hear that they are wildly popular because, at the museum, they allow people to get on and ride them. They’re actually musical instruments powered by exercise bicycles that were created for my group by Norman Anderson, a visual artist in the Twin Cities. A lot of his visual art uses old musical instrument parts and a lot of it runs on small motors. These beings that he creates move and make sound. They sort of come to life and each have their own personality and idiosyncrasies. So I was really fascinated by Norman’s work and I asked him if he would create a piece that I could use with my group. The moment I asked him, I had envisioned something that moves and comes to life, but in some way also has some space in it, both literally and figuratively for my group to play along with them or to interact with this thing coming to life. So Norman said sure and started working on it. Then he called me up one day and he said, “I got an idea. I was driving through my alley and I saw someone had put out, to be taken away, an old exercise bicycle. I know I could find a couple others, no problem.” And he does—many of his materials are found items. So he said, “What if I did three pieces that each were powered by an exercise bicycle?” And so I said, “That sounds like fun. Let’s go for it.” Then we started talking back and forth about what the sounds were going to be for each and so we landed on one that was the pipe-cercycle, which uses organ pipes as its sound and physical material. Then there’s the string-cercycle, which is all strings of various kinds. A little ukulele in the front, a cello in the back, some other resonating strings that Norman created. And then the xylo-cycle, which has xylophone-like bars right above the handle bars and a kind of a music box-style cylinder with screws sticking out of it. When you pedal, it will turn and trigger the mallets over the top of the xylophone-like bars. That’s how those three pieces came about.

Tri-Cycles by Mary Ellen Childs from Mary Ellen Childs on Vimeo.

FJO: There’s actually been a significant history of writing music involving bicycles. In the early 1980s there was an audio artist based in Arizona named Richard Lerman who created an outdoor piece called Travelon Gamelon that was recorded by Folkways and which is now available from the Smithsonian. Before that, one of the earliest musical efforts by Frank Zappa that attracted some notoriety was a film score he wrote using bicycles. He even performed a selection from it on network TV in 1963. It was a few years before he became a famous rock musician; he was announced as being an avant-garde composer.

MEC: Okay, that I did not know.

FJO: It’s pretty wacky. And just last year I heard a piece by Ruby Fulton during the SONiC Festival that involved a bicycle. For some reason bicycles keep reappearing in musical contexts, but in the trajectory of your work, I think, it’s yet another example of calling attention to the physical process of making music.

MEC: The piece I just described didn’t start out being about bicycles. It was about invented musical instruments and how a human can interact with them. But it ended up being about bicycles.

FJO: Now that you’ve donated the instruments, does that mean the piece is out of CRASH’s repertoire?

MEC: We did it one more time, at the museum—must have been the fall of 2014. That was the last time it’s been done. Occasionally I would use the xylo-cycle in concert. But frankly, those instruments were sitting in my studio for a good decade unused. They were hard to get in and out of my house and were difficult to transport. You could not fly them anywhere, so they could really only be used pretty close to home. And what I realized after I got the pieces and started working with them is they’re not well suited for a concert-style performance, unless we started working with a sound engineer. If I had gone to the next stage with that piece, I would have done that. Because they don’t balance very well. Some things are too quiet to really hear, and you have to accept the fact that the sound of pedaling is going to be part of the piece because the bikes themselves are noisy. What I also found about these instruments is that they’re best suited for a more installation-style performance. You can do some very interesting things with them to create kind of an atmosphere or a mood or a sound world. But to shape a musical arc over a period of time, which is what you want to do in a concert setting, was difficult to impossible.

If I was going to continue with those instruments and those pieces, I think I probably would have added some other instruments, singers, texts, or something like that. Plus a sound engineer to finesse all those materials in a way that was going to be fully satisfying if you really have a rapt audience in front of you, rather than an installation work where people are kind of walking through. The performance that I did with my players was meant for a gallery or an installation. People do come and actually pay attention for the whole time. And the players are dressed in our usual black suits, and then they wear bowler hats. They get off the bikes and they do other things with them. I don’t know what will happen to it in the future.

FJO: Now once again, I imagine that there are some written instructions, but no score.

MEC: There are some written instructions, and that is the score. That piece actually does have a fair amount of structured improvisation in it. If I had the time or the resources I’d work that even more, but it certainly is not one of those pieces that anybody else is going to do because the instruments are so specific and so is the process of putting it together, as we discovered when recreating it ten years later. But as I said, I would sometimes take the xylo-cycle out and use it as a little solo piece in the context of a bigger CRASH concert.

FJO: Did you ever get on the bike and play it, or was it always one of the players?

MEC: It was always one of the players. There was a period of time during CRASH concerts when I would do little cameo performances. I thought of them as my Alfred Hitchcock moments—you know, you blink and you might miss me. But no, I never really performed in that group. I haven’t performed in a long time.

FJO: Thankfully, in the last decade, two really significant, large-scale compositions of yours are now available on commercial recordings. Dream House is an extremely beautiful piece that very effectively weaves the sound of a live string quartet with an electronic sound environment. But in some ways I feel I don’t completely know the piece because I only know it from the audio recording and it was created as part of a much more multisensory and omnidirectional experience involving multiple video screens. I’ve only seen a few snippets of that posted online.

DREAM HOUSE by Mary Ellen Childs from Mary Ellen Childs on Vimeo.

MEC: The original piece was conceived as a full-evening piece for live string quartet. It was written for ETHEL and some sound montage interwoven into some of the movements, and then multi-image video.

The theme of the piece was construction and destruction and how those two things are so intertwined; one just doesn’t exist without the other. It’s not that one’s good and one’s bad—although sometimes we think of them that way. It’s not that we prefer one over the other. I chose to take the roof off my house to build my studio up. That was destroying something so that I could create something else.

It was also about cycles of time, and then rhythms of work. The video sometimes filled the performance space, but sometimes was as small as a single flower in a window pane of a relatively very small image. It wasn’t meant to be a 2-D, single-screen rectangle behind the players where you see the image. I really wanted the projected imagery to be able to use the space. I think we had seven projectors used in different ways in the course of the evening. And then the lighting design was also a very integral part of the whole performance because after all, video projection is projected light. The lighting designer used and also picked up on some of these same images that were in the video and physically in the space. It was meant to be a blending of what’s projected and what’s a real image. Then we used quadrophonic sounds so that the sound could also envelop the audience. But I have to say that though that was the original performance and how the piece was conceived, I also wanted the piece to stand on its own musically. That was where it started. My videographer worked from the music. I wasn’t writing music to existing film; it was actually the other way around. And we really talked a lot with my collaborators about how to keep it so that the music is really the primary experience, because visual imagery can take over so easily and make music an accompaniment. How are we really going to keep this so that, if anything, the visual material is the accompaniment to what you’re hearing?

FJO: So it was a deliberate decision to issue Dream House on CD in order for the music to stand on its own, rather than to release it on, say, a 5.1 Surround Sound DVD?

MEC: Yeah, I’ve been thinking about this lately. I don’t know how you’d recreate the experience, except to restage the whole thing and have a live audience, just because of how the lighting design worked. If you were to take this experience of the video and flatten it down to one screen again, that goes against what the original experience was like. The only way to experience that piece again would be to restage the whole thing, which has not happened. I would be happy to have it happen, but it hasn’t happened.

FJO: Wreck also has a ton of extra-musical elements. It was created for dance and I saw just a little snippet of video that the dance company put up online, but again, I really only know the music because it was released as an audio recording.

MEC: Well, that happened because Carl Flink—a choreographer whose group is Black Label Movement—was creating a full-evening piece called Wreck. It was about an oar boat on Lake Superior that sinks and the crew knows they’re not going to survive. They end up in the last airtight compartment. It’s a rather dark piece. This is not a true story. But it’s about facing death and also about the power of that great lake.

There was another composer who was slated to do it who tragically died very unexpectedly at the very beginning stages of the work. So that’s why Carl came to me. He had already started working on the piece. My schedule was such that, by the time I was really ready to write it, much of the choreography had been created. He was still ordering pieces and making final decisions, but I came to it pretty late in the game, so I was writing to existing dance. I felt very much like I was scoring a film, and that’s really how I approached it. But I told Carl right away, “If you need to have me mirror your phrases musically and work with your phrase structure, that’s not what I’m going to do.” And he said, “No, no, no, I don’t work with music like that at all.” So I felt very freed up to create a sonic environment for the dance to exist in. But I did find ways where, through musical decisions, I could coordinate with the dancers, because it was going to be performed to live music, so the live performers could cue off the dancers.

This can’t be shrinking violet music. It can’t be wallpaper music. It can’t be background music. This had to be music that makes its own very powerful statement.

Of course, what your perception will do when you’re hearing something and seeing something is that you’ll make the connection that the two were meant to happen in time at that moment, even when they weren’t. I don’t know if that makes sense. I was kind of talking in the abstract there. But I said to Carl early on, “The movement you’ve created is so strong, I think this needs to have very powerful music that goes along with it that stands on its own, that doesn’t take second place to the dance. If that sounds appealing to you, that’s what I will give you. This can’t be shrinking violet music. It can’t be wallpaper music. It can’t be background music. This had to be music that makes its own very powerful statement and can be an equal partner with the dance.” And so again, I think that’s why it works on the recording without the dance. But the recording is not the dance piece from beginning to end. When we did the recording, there were some pieces we didn’t put on the recording. It was like releasing a film score; you order the pieces in a different way to make the recording a satisfying experience. So although it’s the same material, the dance performance and the recording are two different, both satisfying, experiences I hope.

FJO: One of the things that I find curious about both of these pieces is that even though they are large-scale and immersive, they’re both chamber music. You’ve never done a thing like this for a very large ensemble. Of course, working with an orchestra imposes a regimented and quite limited rehearsal process. I also imagine that you couldn’t tell orchestra players to move around, or to have video screens scattered around them on stage.

MEC: Well, yeah, I think that’s a part of it. The kinds of things that I do take a little more rehearsal time. But there’s always a limit to rehearsal time. When I did Click, I counted up how many rehearsals I had over the course of a year. There were like 38 or 40 rehearsals. And that was to create the piece and, of course, the players were learning it at the same time. But in our world, that’s pretty hard to come by. And I don’t think that I’ll ever do that work that way again. The kinds of things that I’m talking about take a lot of rehearsal time. That’s probably why Dream House hasn’t been done again, because it would take a lot to get that piece up on its feet again. We’d have to redo a little bit of the video. You need special equipment. You need to rebuild these set pieces. Simple, but they need to be rebuilt to fly in and out. We’d need very high powered projectors. These are also not pieces that tour very easily. They don’t get multiple performances very easily.

As for orchestras, there are just some practical considerations. It’s just going to be harder to do something like that with a professional orchestra. So if I were to do it, I’d have to think about it very differently. Could something have as much impact that wouldn’t take a week of technical rehearsals in order to pull off? And as far as the movement pieces, it’s just harder with other instruments. I’ve experimented with it a little bit. Percussion is so ripe for using movement. With other instruments, you can’t change the playing technique in the same way. It isn’t as physical to begin with, so it’s just harder to do.

I haven’t actually done new pieces for CRASH in 15 years. That’s a whole body of work that exists, and when CRASH travels it tends to be this existing work. But how I’ve incorporated other elements has kind of evolved a little bit. Eventually I starting to be interested in scent, Dream House came several years after the work that I was doing with CRASH, and that was using multi-image video and lighting design.

Last fall I did a piece at the Farnsworth House, which is a Mies van der Rohe-designed house in Plano, Illinois, about an hour and a half west of Chicago on the Fox River. And it’s this beautiful, historic, all-glass house. I was commissioned to write a piece inspired by the house and performed inside the house as part of the Chicago Architectural Biennial. So I got to spend some time in the house and absorb it. Then I wrote the piece based on what I had experienced, knowing the performance was going to take place in the house, to feel what the audience would feel to be in a space like that and see what they see because it’s glass and so the outdoors and the indoors blend so much. For that, I started feeling like the space itself was my collaborator. I’m really interested in doing more of that, writing for an already existing space. I’d like to do something with the night sky—the night sky is sort of, again, my collaborator, my existing other element. The audience’s experience would be of the music, but in this particular space, in this particular moment seeing the stars or looking out. And it’s not meant to be performances where you’re sitting in chairs, facing the musicians, and your visual focus is on the musicians. So it’s, again, using and thinking about space in a different way.

Complicity and the Chemical Senses

Blindfolded patrons sit around a table and eat various food items, smell perfume, and listen to music.

Jozef Youssef presents a multisensory meal for the launch of a new Hugo Boss fragrance. (Photo courtesy Kitchen Theory)

I spent the weekend composing two new pieces of music for another collaboration with chef Jozef Youssef of Kitchen Theory. This time around I was tasked with bringing out the sweet and sour elements of a lavender and pomegranate-based tapioca dessert, part of a multisensory meal that Youssef was organizing for the launch of a new Hugo Boss fragrance. (Previously, I composed music to bring out the sweet and bitter notes in Chivas Regal 18 Ultimate Cask Collection blended Scotch whisky for a similar event.)

My sweet texture is pretty similar to the sweet texture I wrote for the whisky pairing: consonant intervals and mellow timbres in a relatively high register, organized in short, ascending phrases, with a sprinkling of wind chimes.

The sour texture, while similarly drawing on the psychological literature of Charles Spence’s Crossmodal Research Lab at Oxford University, is a bit looser in its interpretation of the research. I used bright, buzzy timbres and descending phrases for contrast, and I picked up on an idea from the music I wrote for a 2012 Azul y Garanza Garciano to depict tartness with dissonance, not only in the musical intervals, but also in the slight detuning of my subtly shifting delay effect. To give me a wider continuum of intervals from which to choose, I composed my music in Harry Partch’s 43-note per octave scale, which allowed me to incorporate his notions of comparative consonance (the simpler the ratio, the more consonant the interval, which is the best explanation of consonance vs. dissonance I’ve encountered and serves as a finely graduated yardstick for crossmodal inquiry). I interpreted the faster tempi that have been linked to sourness in psychological studies more generally as high rhythmic density, reflected in my texture’s pulsating elements. Sourness feels higher than sweetness to me, so this piece is in a slightly higher register as well.

While I wasn’t specifically composing music to accompany a scent, scent plays a vital role in dining, and I think the challenges to accompanying scent with music are similar to taste.

Taste and smell are our chemical senses. They are what allow us to interrogate the makeup of our environment, providing information that perhaps might lead us in the direction of dinner or cause us to spit out something poisonous. The visual arts and the sonic arts arrive to us from a distance, via electromagnetic radiation or fluctuations in air pressure, but taste and smell require direct contact.

Philosophers have long debated whether the fundamentally different nature of these chemical senses precludes the elaboration of an art of ideas based on them, something that goes beyond the ancient and sophisticated traditions of perfumery or cuisine. Taste, which requires a visible delivery mechanism, perhaps has a leg up on smell, which is invisible (like music).

A few years ago, Chandler Burr lobbied New York’s Museum of Arts and Design to establish a new Department of Olfactory Art and was subsequently brought on as its head. His brief tenure culminated in the 2012 exhibition The Art of Scent 1889-2012. A man of passionate pronouncements, he comes right out and says of perfume, “It’s art.”

I’m happy to accept, along with Marcel Duchamp, that if someone proclaims his or her work as art, it’s art. However I’m not sure whether, during the long history of perfume-making, most perfumers have actually been making that claim. We’re not talking about the gradual acceptance of a new technology; unlike photography or video games (my primary arena of activity for the past twenty years, which only recently emerged from its own “is it art?” debate), perfume has been around for a very long time. It might take a little more work than the wholesale rebranding of an existing discipline to bring perfume and art into the same conversation. Perhaps we can repurpose Wittgenstein’s dictum that not every building is architecture to suggest that not every perfume is a work of olfactory art.

(Incidentally, while I’ve seen a couple of announcements in recent years about companies bringing the sense of smell into gaming, my very first game, the maybe not so great Leisure Suit Larry 7: Love for Sail!, may well be the first, way back in 1996. The game shipped with a scratch and sniff card, and at key junctures in the game, the “Cybersniff 2000” logo would flash and a sound would play (my processed voice), informing the player which of the nine squares to scratch, neatly addressing the question of synchronization. The soundtrack also featured a 12-tone faux-jazz composition, the sure mark of a young composer fresh out of music school.)

One of the considerations is practicality. In an interview with The New York Times, Holly Hotchner, the former director of the Museum of Art and Design in New York City who brought Chandler Burr on board, conceded that “perfume by its nature has to be wearable, which is not true of other art forms.” Paul A. Young, a London-based chocolatier who devised a charcoal-flavored chocolate to accompany a Francis Bacon painting at the Tate Sensorium exhibition acknowledges the assumption that food should be pleasurable when he says, “Some people will find it repulsive and not want to eat it, some will find it engaging and some people will love it.”

Music has always been considered an art form, but in the last few decades, sound art has emerged and evolved as a parallel practice distinct from music (although I think sound artists ignore the history of music at their peril). There’s no quantifiable difference between sound art and music. The materials are exactly the same; it’s a question of focus. With all of the current interest surrounding food and multisensory perception, I’ve been thinking for a while that taste and smell may be next in line to be appropriated by the art world. But this question of focus highlights the challenges that can arise when people from different creative disciplines come together (as I described in an earlier essay).

As an example, when I attended “The Architecture of Taste,” Pierre Hermé’s presentation at the Harvard Graduate School of Design a few years ago, it was fascinating to get a glimpse into a great chef’s creative process, and the pastry samples were amazing, but it was clear, especially during the Q&A afterwards, that he was thinking of his work in very different terms from the concerns of art and design grad students. Another leading pastry chef, Jordi Roca of El Celler de Can Roca, discussed his relationship to art in the film that documents the multimedia meal El Somni, devised alongside the other two Roca brothers, concluding simply, “I’m just a pastry chef.”

An assortment of desserts on a tray.

Desserts served at Pierre Hermé’s Harvard lecture included an “infringement citron” and other desserts made from unconventional ingredients such as wasabi. (Photo by Ben Houge.)

Dean Dutton provides a good overview of the issues surrounding the notion of olfactory art in his book The Art Instinct, and as I’ve been doing projects that combine food and music, it’s been useful for me to consider these arguments from a musical perspective.

One issue that he dismisses a bit too quickly, I think, is the idea of “internal relations.” He borrows this term from philosopher Monroe Beardsley, pointing out that, while we may recognize corresponding proportional relationships in the spatial layouts of a painting or the intervals of a musical chord, chemical compounds don’t lend themselves to this kind of perceptual organization that can serve as the basis for large, sophisticated structures. There’s no equivalent to an octave or a major 7 sharp 11 chord or a retrograde inversion that would allow a composer like Bach to spin out a ten-minute fugue from a subject of just a few notes. The wine aroma wheel is nowhere near as precise as a color wheel (can you rotate a raspberry note 180 degrees to obtain its opposite?) or the circle of fifths (what’s the secondary dominant of leather?).

A second obstacle is a question of the delivery mechanism, and here I’m thinking about a time-based sequencing or counterpoint of multiple scents that goes beyond the natural diffusion of a scent in the air once it is sprayed or applied. (This is more of an issue for scent than taste.) To think of scent in musical, time-based terms, it’s a challenge to consistently deliver a scent and take it away, allowing for temporal contrast. This issue thwarted some of the historical efforts to bring scent into cinema, such as Smell-O-Vision from 1959.

Many approaches to scent diffusion have been developed with varying levels of technological overhead. The Museum of Art and Design’s Art of Scent exhibition took place in a spare, white room, designed by Diller Scofidio + Renfro, laudably devoid of packaging and other marketing detritus. As visitors inclined their heads over small indentations in the wall, a puff of scent emerged from a sophisticated mechanism designed by Scent Communication in Germany. At Green Aria: A Scentopera, a collaboration between entrepreneur/writer/director Stewart Matthew, perfumer Christophe Laudamiel, and composers Nico Muhly and Valgeir Sigurdsson which premiered at the Guggenheim in New York City in 2009 (about which NewMusicBox’s Frank J. Oteri has written extensively), scent was delivered to each spectator via a “scent microphone” (functionally more like a “scent speaker”) at each seat, which could be positioned as close to the nose as desired. The inaugural installation of the recently opened Museum of Food and Drink in Brooklyn, masterminded by cocktail impresario Dave Arnold, features a “Smell Synth” that allows visitors to experiment with different odor compound combinations at the push of a button. This echoes the “Olfactiano,” a “scent piano” developed by Peter De Cupere, which he has used for time-based scent-oriented performances such as his Scentsonata for Brussels presented at the Cordoba festival in 2004. At the Crossmodalist rehearsal I attended in London last April, the technique was low-tech but effective; I was blindfolded, and different perfumes (designed by Nadjib Achaibou to evoke concepts such as “lust” and “sorrow”) were manually wafted in my direction, sometimes simultaneously, by unseen hands at different moments of Chris Lloyd’s live performance of Liszt’s piano transcription of Wagner’s “Liebestod.”

A third consideration is representation. I’ve written in the past about what I called the “abstract” quality that food and sound share. By this I meant that, while music can be linked to an external narrative via text or theatrical context, coherence is essentially the result of internal relations, to return to Monroe Beardsley’s term, i.e., a note has meaning in the context of a chord or scale or harmonic structure and how these structures evolve in time. Even without recourse to the same kinds of internal relations as music, tastes and scents are constructed and developed by combining chemical compounds. This is in contrast to the visual and literary arts like painting, sculpture, choreography, cinema, literature, poetry, and video games, in which meaning-making via external representation is the norm. I think it’s important to consider that an extracted chemical essence of an object is still a form or an attribute of the object; the scent of a rose derived from a rose cannot be said to represent a rose in the way that a painting represents a rose, because in a real sense that essence actually is the rose. Representation is the use of one thing to depict another thing; as Magritte put it, “Ceci n’est pas une pipe,” and no pipe extracts are required to produce a painting of a pipe.

My usage is different from the way the term is used, for example, in Annick Le Guérer’s article “Olfaction and Cognition: a Philosophical and Psychoanalytic View” in the weighty 2002 compendium Olfaction, Taste, and Cognition. Here, the historical assertion (shared by numerous philosophers) is that scent is not “capable of abstraction,” by which she means that a common attribute cannot be extracted from multiple scents in the way that we might refer to an abstract color. This is reflected in the lack of vocabulary for describing scents (at least in English); we generally simply describe scents according to their sources or, in the case of synthetic compounds, by analogy.

So perhaps my earlier usage of the term is nonstandard, but I think the ideas are related: just as mimetic music was by far the exception prior to the advent of recording technology (with timpani evoking thunder in Berlioz’s Symphonie Fantastique serving as an example of a rare exception), it’s difficult for a taste or a smell to be “about” anything other than itself.

I’ve heard people assert that what music represents is emotion, and I could imagine this argument being applied to smell or taste, but evoking and representing are not the same thing. As Frank J. Oteri pointed out in our recent correspondence on the subject, “Associations (e.g. major = happy, minor = sad) are the result of acculturation and not universal. The same, though, could be said for scent.” He notes the curious coincidence that, as I mentioned above, music and scent are completely invisible.

I’ve also heard some people conflate tonality with representation, and there certainly are clear parallels between figurative painting in the visual arts and tonal structures in music, but tonality strictly speaking still does not represent anything outside of music’s “internal relations,” and in fact it is these relations that give tonality meaning. Nonetheless it is certainly worth observing that abstraction in painting and tonality in music experienced parallel ruptures in the tumultuous early 20th century, and it is surely a result of the same cultural forces that Chanel No. 5 was introduced in 1921, unprecedented in its lack of reference to natural scents, “a perfume like nothing else,” in the words of Coco Chanel.

At a certain point, while pondering these questions, I realized that music has been here before. Like many undergraduate music students, I read about the absolute vs. program music debate in my music history classes, and (like a lot of students, I suspect) I was left wondering what the big deal was. On one side of this virulent debate you had Richard Wagner espousing opera as a total art form driven by narrative and deriving power from the coupling of art forms, while on the other side Eduard Hanslick argued for purely instrumental music as the art form’s pinnacle of expression. I recently picked up Mark Evan Bonds’s excellent new book on the subject and realized just how far back this question goes. He organizes his argument around the ideas of essence (what music is) and effect (what it does). This historical perspective has the potential to serve as a useful framing device for discussions of an art based on the chemical senses.

There is a category of “olfactory artists” that has emerged as a distinct practice from that of perfumers, and it seems to be expanding rapidly. Many of the artists working with scent are not working only with scent, and often they’re not working alone. Some of these artists are crafting the scents themselves, but oftentimes the chemical component is outsourced. Pamela Rosenkranz worked with perfumers Dominique Ropion and Frédéric Malle and sound designer Emar Vegt on Our Product, an installation at this year’s Venice Biennale that presented a huge pool of liquid the color of an averaged European.skin tone. Belgian artist Peter De Cupere, perhaps the best known artist in this arena, has been working with scent since the 90s, often incorporating smells into large-scale mixed media sculptures, as in The Smell of a Stranger, presented at the Havana Biennial, which engineers indigenous Cuban plants to give off scents redolent of Western culture. Brian Goeltzenleuchter has been working on a project to capture the smell of each neighborhood in Los Angeles.

Arnica Yi’s 6,070,430k of Digital Spit, presented last summer at MIT’s List Visual Arts Center, bridged the worlds of scent and taste by building an immersive installation inspired by the “Mint Pond” dish at Ferran Adrià’s elBulli restaurant. She notes, “I think the most radical artistic statements are being made in the world of cuisine. That interest translates and seeps into my approach to smell. Even though I don’t work with food, I feel the sensibilities are shared.”

Soundscape for Pamela Rosenkranz – Our Product. Swiss Pavilion at Biennale Venice 2015 from Emar Vegt on Vimeo.
The high degree of collaboration that many artists are already exploring points one way forward for an art of the chemical senses. Maybe the lack of “internal relations” and problems of diffusion preclude certain types of standalone olfactory expressions, but by linking art forms, sophisticated new kinds of experiences are possible. Let music aid with the time-based elements. It’s not unlike setting a text or scoring a film; another art form defines the structure. Ramón Perisé of Mugaritz writes that, “how cooking can participate in a multisensory spectacle is something that, in my opinion, is in the first phases of exploration.” The idea of putting different art forms together into a multimedia event is the reason I chose the term “food opera” for my project in the first place.

Listening to music is something we do together. So is dining. As I’ve been working on my food opera project, I’ve spent a lot of time thinking about the overlap between these two spaces: the restaurant and the concert hall. Food represents community, maybe literally, in the same sense of the word I used when writing about “representation” earlier. When we think of fellowship, we often speak of breaking bread together. Dining is one mode of being in the world together.

To orchestrate a communal dining experience as an artwork is the kind of tactic associated with the relational aesthetics movement of the 90’s. Rirkrit Tiravanija’s 1992 piece Untitled (Free), now in the collection of New York’s Museum of Modern Art, involved distributing free curry and rice to visitors in a gallery space, and since then, Tiravanija has presented a number of other pieces that involve people coming together to eat. Together with his longtime gallerist, Gavin Brown (a collaborator on the original exhibition of Untitled [Free]), he has been developing a commercial kitchen in Hancock, NY, called Unclebrother. In a recent interview, he highlighted some of the same community-oriented values I emphasize in my work, “It’s about eating from your surroundings.”

This is a big part of what my food opera project is about. I have observed that these events promote community in a unique way. Because each seat in a restaurant has its own speaker and functions as a source of music, diners become more aware of the people around them and in fact depend on the presence of other diners to complete their experience. The music foregrounded at each diner’s table becomes the accompaniment for the person at the adjacent table, such that everyone in the restaurant is involved, through their choice of dish and the rhythms of their meal, in facilitating the overall musical experience. There’s a lateral exchange of information that is unlike a typical concert; sound passes not just from performer to audience, but from diner to diner. This strikes me as a rich yet underexplored model for musical communication (something I’ve also explored in my piece The Tomb of the Grammarian Lysias, a setting of a Greek poem by Constantine P. Cavafy for unamplified voice and audience mobile devices).

At several events, we’ve gone so far as to incorporate field recordings from the farms that provide the ingredients and interviews with farmers into the soundscape, since Jason Bond, the chef with whom I’ve collaborated on these projects, is a passionate supporter of local, sustainable agriculture. We used sound to literally bring farm to table. My hope is that diners leave the restaurant with an increased sense of their interdependent roles in a larger ecosystem, which may encourage more responsible food consumption choices.

When I was attempting, in my poor Spanish, to describe this aspect of my work to chef Dani Lasa at Mugaritz, he understood immediately and responded with the perfect word: “complicidad.” Complicity. I asked if Mugaritz had ever done a dish involving sound, not in their various multimedia collaborations, but actually in the dining room. Dani told me about a dish called Mortar Soup with Spices, Seeds, Fish Broth, and Fresh Herbs. No matter at what stage people were in their meals, the entire restaurant was served the dish at the same time, and as each diner applied the pestle to the mortar, the whole room rang with a sound like Tibetan singing bowls. This experience was enabled by the restaurant, but enacted by each individual diner; everyone was complicit in the resulting sound.

A mortar, a pestle, and food.

Sopa de mortero con especias, semillas, caldo de pescados y hierbas frescas. (Photo courtesy Mugaritz.)

This is the goal of my food operas: to bring things together, connecting creative disciplines, connecting farms to restaurants, connecting people to their environments and to each other.

I hesitate to add this coda, but I feel this is an important point. I’ve been writing this essay in the wake of the terrorist attacks in Paris, and in Beirut before that, and of course you can keep going as far back as you like. In these factious times, more than ever we need to find ways to understand those whom we perceive as being different from ourselves as part of the same ecosystem, complicit in the well being of the world. As someone who’s lived a third of his life outside of his home country, I believe that cultivating empathy for others via direct interaction is the best hope for peace. One of the best ways to do this is to come together over a meal. Just as our chemical senses—Le Guérer uses the term “proximity senses”—require contact to perceive our surroundings, we need situations that allow us to come together and better understand our neighbors on this planet. When we break bread together, a meal can do that.

Vinfonies, Nessun Dorma, and Gastromorphology

A bottle of wine and a wine glass on a tale alongside an electronic keyboard, music notation paper, headphones, and a computer screen.

(Photo by Ben Houge)

At just about every technology-oriented conference I go to, I bump into some representative of the Music Technology Group of the Universitat Pompeu Fabra in Barcelona. Probably best known for developing the Reactable, the interactive music control system prominently used by Björk, the MTG is one of the most well regarded centers for music technology in Europe. I believe it was at the Game Developers Conference in San Francisco back in 2012 where I first met the MTG’s Jordi Janer, who has been conducting fascinating research into voice analysis and audio source separation, work that he has made available commercially via his side company, Voctro Labs.

Jordi has another side project as one of the founders of Vinfonies, a series of events pairing wine and music that goes back to 2009. Jordi himself has done a number of sound installations related to various aspects of viticulture, developing pieces based on resonating cava bottles, an interactive grape press, and the sonic byproducts of fermentation. With my food opera events I’ve sought to bring the sounds of the farm into an urban restaurant, but Vinfonies goes the other direction, using the festival as a way to bring wine-themed new media art to rural settings. The events happen annually at harvest time in Vilafranca del Penedès (part of the well-regarded Penedès winemaking region, one of Spain’s sixty-five or so regions that have earned the Denominación de Origen appellation) about 50km from Barcelona. The festival includes concerts, sound installations, and wine tasting sessions. Similar to the work of Spanish sound artist Francisco López, participants experience the wine and music pairings blindfolded, eliminating visual distractions to focus on taste, smell, touch, and sound.

While I was living in Spain, I had a chance to reconnect with Jordi at Music Hack Day Barcelona, sponsored by the MTG as part of the huge Sónar electronic music festival. (Check out my brainwave sonification hack from 2014.) The MTG partnered with Berklee to bring computer music pioneer John Chowning to Spain for a series of events last summer (including his first visit to Valencia since he docked there in 1953, back when he was a drummer in the U.S. Navy), and in the planning conversations, Jordi and I realized we shared an interest in music and gastronomy. He invited me to write a piece for the 2015 edition of Vinfonies, which took place last September. Over the summer, he mailed me a bottle of Azul y Garanza 2012 Garciano (the name, which might perplex some oenophiles, derives from its composition: 50% Garnacha, 50% Graciano), and I set to work composing my musical accompaniment.

BTW, a playlist of previous Vinfonies compositions (including one by Edwin Van Der Heide, whose fascinating sound installation “Spectral Diffractions” took over Mies van der Rohe’s Barcelona Pavilion as part of Sónar in 2014) is available on SoundCloud.

Composing music to accompany a taste-based experience presents some unique challenges, particularly when it comes to synchronization. One of the epiphanies that sparked my food opera project was the recognition of a meal as a time-based art form. It’s easy to recognize several time scales to a meal, from the succession of courses (even simply saving dessert for last) to the entropy that occurs as a hot dish cools or a frozen dish melts to the succession of individual bites.

Recognizing these time scales is straightforward, but synchronizing music to them is a much trickier proposition. Typically, restaurants don’t even try, and in this they resemble far too many video games that simply loop the same piece of music over and over or shuffle a playlist: the only point of coordination between the music and the restaurant is that while you’re in this space, this is the music that’s playing. (See the paper I presented at Invisible Places Sounding Cities in Portugal for a more detailed discussion.) A few restaurants that explicitly stress the multisensory angle go further: Ultraviolet or the various Kitchen Theory dinners will present a track of music synchronized with each course, but for this to work, everyone has to be eating at the same time. This approach—when the dish is served, someone presses play on the CD player—works best when there are a greater number of relatively small dishes that don’t take too long to eat.

To coordinate music with individual bites is still trickier. One clever solution is the Tasteful Turntable, devised by Lars and Nikolaj Kynde to synchronize several small bites with key moments in a soundtrack presented to four diners at a time. Another, somewhat less practical approach is Naoya Koizumi’s Chewing Jockey, which uses a photoreflector sensor and a bone conduction speaker to alter the sound of chewing. But when the dining experience is opened up to multiple diners eating asynchronously (i.e., the way the usually eat), the logistics of synchronization become much harder to address. One solution is to give diners headphones, as Heston Blumenthal does in his famous Sound of the Sea dish, or in performance artist Marina Abramović’s collaboration with chef Kevin Lasko, Volcano Flambé. Headphones works as a personal experience, potentially a profound one, but they also cut people off from the environment and those around them. The other solution is to put a speaker at each seat, broadcasting sound unfettered into the restaurant space, and then (this is the hard part) avoiding cacophony by finding a way to incorporate the bleed of music from adjacent tables into the experience, so that all the sound in the restaurant is coordinated. This is what I’ve done in my food operas, and from my observation, this is the only solution that scales to an evening-length experience.

This solution requires all of the musical compositions written for a food opera to be modular, so that they can be put together in any combination and still sound harmonious. At the events I organized with chef Jason Bond in his restaurant Bondir, there were five courses, and diners chose between two possible dishes per course, so that’s up to ten different pieces of music that could be playing at the same time, each at different points in the composition, from the twenty-six seats in the restaurant. I wrote custom software to generate a new version of each musical texture and play it on the correct speaker every time a diner was served the corresponding dish; because the music was generated on the fly, the software could also ensure that all of these individual streams of music were coordinated in harmony and rhythm. In order to make this coordination as apparent as possible, I limited myself to a diatonic scale (slowly transposing over the course of the evening, with all of the music in the restaurant programmed to conform to the current key) and a steady underlying beat referential of 188 beats per minute.

However, with this Vinfonies commission as a stand-alone piece, I had no such constraints, and so I made it an objective to make the most of my liberty. I also wanted to avoid some of the techniques I had used in the past that I decided were a bit “too easy,” so I decided to eschew steady drones, for example, and the overt diatonicism and rhythmic grid of my previous work.

When I’m developing music to accompany some gustatory stimulus, my process isn’t so different from how I’d approach a video game or a choreography: I sample the experience of the thing I’m scoring (or at least a description or concept art or whatever state of completion the thing is in), evaluate my response, and try to capture what I hear in my mind’s ear. In this, the composer has a great advantage over the researcher; whereas academic studies might try to compare the appropriateness of different types of wine to existing music (as in a recent study contrasting Debussy with Rachmaninoff), a composer can hone in on and express an ideal imagined sound without having to choose between existing examples. (This may lead to less quantifiable data, but in some ways it resembles another study, in which participants were asked to find the single note on a keyboard that best corresponded to a scent stimulus.)

An interesting point came up in Oxford last February, when I collaborated with researcher and wine expert Janice Wang in an event at Alistair Cooper’s 1855 Oxford Wine Bar. We presented three pairs of wines alongside three pairs of musical textures and asked participants to determine which musical texture best matched each wine. In the ensuing conversation, it became clear that some people chose music to evoke the wine, whereas others chose music to complement the wine. This may be frustrating to the researcher, but it’s exciting to the composer, at it shows how much room there is for creativity in devising music/food pairings. Music really can serve as another kind of seasoning, flexible in the same way that an ingredient like mint might be equally at home in a sweet or savory dish. We can recognize this phenomenon from the world of film scoring; it’s an exercise we even give to students at Berklee, to compose different soundtracks to the same film clip to give it a different emotional spin, to make it happy or sad, wistful, nostalgic, or ominous.

The Azul y Garanza 2012 Garciano provided a great opportunity to stretch out a bit. It’s a complex wine, and working at the intersection of food and music, I’m sometimes bemused by the fact that, while it’s common for people to decry complexity in music, complexity in wine is pretty universally considered a positive attribute. Jordi Janer shares an interesting observation on this point; he tells me that when pairing music and wine at the Vinfonies events, he’s found participants much more receptive to challenging, experimental soundtracks than they might ordinarily be. Unlike the whisky project I wrote about earlier, my objective here was not to translate aspects of the wine into sound so much as to create a sonic context for it, and in this case I envisioned a kind of mysterious, mystical setting.

I decided to stick mostly to an octatonic scale, allowing the music to float for long stretches without reinforcing a particular key. I sought a harmonic language that was more dissonant than what I had used in the past. The spiky, disjunct marimba patterns evoke a bit of the spicy quality of the wine. The music has no pulse, evoking a kind of timelessness. I wanted a density to the music that reflected the wine’s complexity, and the delay lines help achieve that, providing a sense of depth; I imagine myself peering into the wine as though it were a dense forest. I tend to prefer delay lines to reverberation in these kinds of pieces, and I think there’s also something liquid in the rolling echoes. The lengths of my delays are periodically changing, which results in a bit of pitch shifting during the transition, and this subtle detuning evokes a kind of tartness. (I’m not sure if this kind of detuning has been subjected to a formal study, but I wonder if there isn’t a link to the expression “a sour note.”) Periodically, as a kind of refrain, a jutting theme outlines an augmented chord, stretching out over a tenth, and alters the mode, swapping a D sharp for a D natural, serving as a kind of punctuation. At these moments, a sustained bed based on cello sounds makes a rare excursion into bass clef territory. In some of my previous work, low cello phrases have been successfully linked to tannins in wine, so I decided to include cello here for the same reason. But in general, the frequency range of the music is in a high register, which reflects the sweet and sour fruit components I notice in the wine. Musical ideas float along as objects for the listener’s consideration, not developed, but juxtaposed for the listener to parse and contemplate. Overall, the music is peaceful, emphasizing stability and resolution, reflecting this well-balanced wine.

(If this description is a bit tedious, note that one of my original reasons for pairing music with food was the recognition of how poor words are for describing music and food, suspecting that they might do a better job of describing each other.)

The instrumentation comes from manipulated recordings of myself (whistling and playing piano strings with mallets) and my Berklee Valencia colleagues Victor Mendoza (marimba master) and Sergio Martínez (percussion savant), with Berklee alum Ro Rowan’s cello recurring periodically.

In the above description, you may readily detect the influence of a composer whose music has deeply influenced my thinking about real-time musical systems and video games—Olivier Messiaen. Much of his music exists in a continuous present, which is exactly what game music must do, as it waits for the next event to signal a transition. By making this piece almost a kind of homage, I sought to link my work to an ongoing musical tradition, something I think is important when venturing into new territory, especially in light of some of the cultural crosstalk I mentioned in my previous post.

My compositional process was idiosyncratic. First I determined the scale and the types of sonorities and simultaneities I wanted to use, and I sketched these out on staff paper, along with some thematic ideas. Then I took some of the instrumental recordings I had prepared to use as source material and turned them into playable instruments in Max/MSP. I wrote a program that would take any note I played on the keyboard and build one of my previously determined chords, in any approved inversion, on top of it; in this way I generated the “mallet piano” part, allowing me to manually control the range, rhythm, and dynamics of that part, while letting the computer generate the chords. Then I wrote a program that would look at the notes of the most recent two chords played by the mallet piano and choose from among them to generate a melody to play on the marimba; whenever I pushed a button, a new melody would be generated according to parameters I specified, and a different button allowed me to repeat a phrase with subtle rhythmic variations. The whistle part, except for the recurring theme (which was played manually), was also generated by looking at the most recent mallet piano chords and choosing a note from among them, although in this case I was determining the octave transposition, rhythm, volume, and duration manually by playing the keyboard. The rattle and cello parts were performed or input manually. To sequence everything into the requested time frame, I used Ableton Live, embedding my Max patches as instruments using Max for Live.

But wait: for this Vinfonies commission, since a linear recording of about three minutes’ duration was what was requested, I could have composed and sequenced everything linearly. So why this weird process involving programmed behaviors?

It comes back to the idea of synchronization. In most cases, eating and drinking are activities that can continue for an indeterminate duration, so I’ve been very interested in composing music that can continue indefinitely. (This is the same problem we face in video games, and much of my creative effort since starting in the game industry in 1996 has been addressed to it.) It’s possible that someday I might want to use this music again and present it in a real-time, indeterminate duration form. If I do, having these algorithmic processes already in place will make it easy to adapt; by reducing the input into the system (playing single notes or pushing buttons instead of playing full chords or phrases), I make it easier to replace my manual input with some automated mechanism. I also wanted to fit this piece into my growing body of “food opera” work, using instruments that I’ve built from acoustic samples in Max as my orchestra, organized around generative or procedural processes to accommodate the constraints of modular, real-time deployment discussed earlier. And I have to admit, composing algorithmic music in this way is simply an interesting challenge. So, since much of my work hovers around this idea, it just made sense to apply the techniques on my current workbench to this new commission. But more than any of these considerations, I feel that this approach to composing—using algorithms to develop textures of indeterminate duration—brings to the fore not just what things sound like, but how they work, and I fundamentally think there’s a real link between the way this music works and the way eating or drinking works.

Wine, perhaps more than any other comestible, brings the question of time to the fore. One of the commonly discussed attributes of a wine at a wine tasting is its finish, how long it lingers on the palette after tasting, which I’ve heard some sommeliers describe in very precise measurements. But thinking about how a sensation changes in the mouth is not limited to wine. After I gave my presentation to the R&D team at Heston Blumenthal’s Fat Duck restaurant in Bray, England, last June, I sat around with some of the chefs and played some of my musical textures, while we tasted caramel and graphed flavor profile changes over time on paper.

When I was studying computer music back in grad school at the University of Washington in Seattle, I remember encountering Dennis Smalley’s concept of spectromorphology. It was fascinating to think of a method for categorizing the different ways a sound could transform over time. Electronic musicians synthesizing sounds are familiar with the concept of an envelope, a shape that describes how a parameter changes over time (the most common variant being an Attack-Decay-Sustain-Release envelope, allowing a synthesist to specify different volume levels for different points in time), and game audio designers are familiar with the concept of a real-time parameter curve (RTPC) that links these parameters to real-time input. I have often wondered about a taste-based equivalent. Is there something about our rich musical language for working with time that can apply to the rhythms of the kitchen? How precisely can we quantify and categorize the way that taste sensations change over time? Maybe we could call it “gastromorphology.”

This issue comes up at the end of one of NPR’s recent features on Charles Spence. First Spence is quoted:

[For example,] a dark chocolate or coffee-tasting dessert, then something like Pavarotti’s [performance of Puccini’s] “Nessun Dorma,” making much more low-pitched sounds, seem to be the perfect complement to help bring out those bitter tastes in the dark chocolate or the coffee.

Then the author of the article gets in a little zinger of a last word:

Of course, ‘Nessun Dorma’ gets a little more high-pitched near the end—so there are still challenges in finding the perfect sound for a constant flavor experience.

This is exactly the problem I’m working to solve. “Nessun Dorma” does change at the end, building to a glorious climax; like most music, it evolves. This is the form of the work, and determining the form is traditionally a big part of the composer’s job. When composing for dance or film or setting text, that evolution may be hitched to another structure, but the end result is a fixed trajectory over time. And this is also why it may be problematic to use a finished piece by Debussy or Rachmaninoff in evaluating taste to music correspondences. Not only are there so many parameters to track, but the rate of change of these parameters (what we might call morphology, or, at a higher level, simply musical form) is also changing at a rate that is likely very different from the rate of change of the tasting experience.

So I return to Messiaen as a useful reference point. I sometimes find myself in the semantically awkward situation of using the antonyms “static” and “dynamic” to refer to the same thing. Sound by its nature is motion, vibration; it is dynamic. But when I talk about static music, I mean music that isn’t going anywhere, music that is nonteleological, music that is not progressing towards a specific goal. I often talk to students of game music about the challenge of taking the pre-rendered dramatic trajectory out of a piece of music they’re writing, so that the game can put it back in. Messiaen exemplifies this kind of stasis.

This project draws on a lot of ideas: Messiaen’s approach to static music, video game music that responds to user input, algorithmic or generative processes, crossmodal sensation, multimedia pairings, the rhythmic profile of a meal. Taken together, they suggest a flexible, dynamic approach to composition, and the applications are not limited to wine, or to a meal. In fact, a real-time, responsive system such as I’ve described could be equally put to use to create a customized soundtrack for any of the unpredictable events of daily life, opening up a whole new arena for creative work.

Bridging Gastronomy and Art Requires Making Connections

The Basque Culinary Center's building, which is an impressive architectural structure , in front of which is an impressive piece of contemporary sculpture.

The Basque Culinary Center is as much a monument to contemporary art as it is an incubator of ideas for contemporary gastronomy.
(All photos by Ben Houge.)

Spain has been at the forefront of contemporary cuisine for many years, rising to international prominence with Ferran Adrià’s acclaimed and highly influential restaurant elBulli, which closed in 2011. Given my work combining music and food, when I was recruited to transfer to Berklee College of Music’s new campus in Valencia in 2013, this was no small consideration in deciding to accept the gig. (I was brought on to help develop curriculum for and serve as the full time faculty member in the new Music Production, Technology, and Innovation master’s program, and I also took charge of the video game component of our Scoring for Film, Television, and Video Games master’s program.)

As a result of spending two years in Spain, I’ve learned a lot about the key figures in the fascinating gastronomy scene, a recurring theme of which is the desire to reach out and engage with ideas from other artistic disciplines. In fact, merging taste with other sensory experiences was central to the topic—La Vanguardia (the Avant-Garde)—of this year’s Diálogos de Cocina conference. Now in its sixth iteration, this biennial event, founded in 2007, is a product of the Basque Culinary Center and Euro-Toques, and from the beginning the focus has been on interdisciplinary dialog. The conference took place over two days, March 9-10, 2015, at the Basque Culinary Center’s gorgeous new building on the outskirts of San Sebastián (the city with, not coincidentally, the highest number of Michelin-starred restaurants per capita in the world). This was the first conference I had attended that was devoted solely to gastronomy—I typically find myself at video game or digital art conferences—but after experiencing the amazing dishes served up at every coffee break, this is the only kind of conference I want to attend from now on.

This is what gets served at a "coffee break" during Diálogos de Cocina

This is what gets served at a “coffee break” during Diálogos de Cocina.

The entire event was focused on the future, investigating ideas from other art forms as well as innovations in technology. Leading crossmodal psychologist Charles Spence was one of the presenters (in fact, it was he who commended the conference to my attention, following my presentation to his Crossmodal Research Laboratory at Oxford University last February), and in his fascinating talk, I finally got to hear the sound of Heston Blumenthal’s influential multimedia dish The Sound of the Sea (which was heavier on the gulls than I had expected). Adrian Cheok of the Mixed Reality Lab in Singapore shared some really wacky progress on his efforts to digitize taste and smell; one lucky volunteer got to taste one of Cheok’s digitized flavors via a device he put in his mouth. Other presentations included a history of avant-garde art, an overview of recent technological trends, and a meditation on communication in the internet age, plus panel discussions on the dining experience of the future, socializing culinary innovation, and what experimental art can bring to cooking (and vice versa).

The consistent theme was how to draw on ideas from other creative practices to enhance what’s going on in the kitchen, an investigation many of the participating chefs were already pursuing in their own restaurants.

Members of the board of the Basque Culinary Center were the main hosts of the conference, and one chef whose inviting presence was most continuously felt was Andoni Luis Aduriz of Mugaritz.  His smiling demeanor extended throughout the sessions and into a group dinner at the amazing sidrería Zelaia, where he was seen cutting enormous traditional Basque steaks for conference attendees to ensure that everyone felt welcomed. Andoni has pushed Mugaritz to explore unique collaborations over the years, working with musician Felipe Ugarte on the Mugaritz BSO (Banda Sonora Original, or Original Soundtrack) in a project that involved visits to Ethiopia and Peru, developing a food ritual with the choreographers Idoia Zabaleta and Filipa Francisco, and providing the climactic meal to a production by Barcelona-based theater group La Fura dels Baus of Titus Andronicus. After I met him and told him about my food opera project, he invited me back to give a presentation to the whole R&D team at Mugaritz, who received me with warm inquisitiveness last August.

On the other side of the country, on Spain’s northeast coast, the three Roca brothers of El Celler de Can Roca have also stretched the interdisciplinary boundaries of gastronomy, most notably with an immersive multimedia “gastropera” called El Somni (The Dream). Twelve invited guests were treated to an extravagantly high tech, one-time event on May 6, 2013. The event, documented in a film by Franc Aleu (another presenter at Diálogos de Cocina), was in twelve eclectic movements, each with music by a different composer, ranging from robotic string instruments to traditional Catalan vocalizations to neo-romantic piano, while motion captured 3D graphics were projected onto the table and onto screens surrounding the diners. Sound also plays a role in one of the dishes on their regular menu: an edible reenactment of a goal scored by soccer star Lionel Messi, who plays for the Barcelona hometown team, which comes accompanied by a recorded sports announcer commentary.

Since elBulli closed in 2011, Ferran’s brother Albert Adrià has been working to keep the family business flourishing. Currently there are five Adrià-branded restaurants in Barcelona (all located within a few blocks near the Plaça d’Espanya, dubbed El Barri Adrià), including the acclaimed Tickets, with a new restaurant named Enigma to open soon. But their most recent opening occurred last summer fifty leagues south, in the Balearic island of Ibiza, an ambitious interdisciplinary collaboration with Cirque de Soleil called Heart Ibiza. Advertised as a fusion of gastronomy, music, art, and performance, this elaborate take on dinner theater proposed a tantalizing opportunity to observe how live performance might complement a meal.

But unfortunately the meal I experienced there last summer was less than the sum of its parts. Throughout the restaurant, dancers danced, actors acted, and bodies were painted, while live video feeds illuminated the walls, but servers clashed with dancers in the aisles (resulting in having a drink spilled on me, for which the waiter did not apologize), and the performances were completely out of sync with the dining experience. Halfway through a course, the lights would dim, leaving me to munch in a lurid blue glow, which, as Charles Spence will confirm, has been shown to have a deleterious effect on food. When I could see it, the food was fantastic, served in some imaginatively sculptural tableware, such as a porcelain frog you had to kiss in order to extract the first bite, but the chaotic surroundings, completely out of sync with the meal, prohibited the performance from enhancing the dining experience. Some issues may be a result of poor logistics and layout (despite reassurances to the contrary, I missed a lot of what was happening on a stage around a corner), but I suspect that the very presence of live performers in the restaurant inherently distracts from the meal on the table.

Ibiza is also home to Paco Roncero’s infamous Sublimotion. Representatives from InHedit, the Madrid-based company that provided some of Roncero’s technology, shared the stage with Adrian Cheok at Diálogos de Cocina to discuss the use of new interfaces in gastronomy. The cost of the technology is surely one reason Sublimotion proudly proclaims itself to be the most expensive restaurant in the world, although I have to say that this elitist stance is anathema to my goals as an artist. But fortunately I already had the Sublimotion experience when I visited Ultraviolet, Paul Pariet’s innovative multimedia restaurant in Shanghai that Sublimotion has been widely derided for having plagiarized.

As Ultraviolet demonstrates, Spain does not hold a monopoly on multimedia dining experiences. Ultraviolet seats ten people per night for a twenty-two course meal in a room that has been outfitted with video projectors, a sound system, and a mechanism to waft in different smells throughout the evening. Back in 2012, my experience at Ultraviolet was wonderful, including a few truly transcendent moments. The one that sticks with me the most was the most understated: a simple slice of bread in meunière sauce with a few truffle slices, experienced in a projected forest, while subtle ambient sound played in the background, a profound synergy of the senses that remains for me a benchmark of what can be achieved in this arena.

But I must point out another course that exemplifies the pitfalls in trying to bridge the worlds of gastronomy and art. Towards the end of the meal, there was a riff on traditional gazpacho (in the world of food, Spain is never far away) that involved two different elixirs separated by an edible shot glass. As the dish was served with a lighthearted “Olé” and flamenco music began to play, the video screens metamorphosed to display not an idyllic Iberian landscape or a boisterous cervecería, but Picasso’s devastating Guernica, commemorating the horrific slaughter of civilians in a Nationalist bombing raid during the Spanish Civil War. When I first saw the painting in 1998 at the Reina Sofía Museum in Madrid, I was somberly transfixed for twenty minutes. Projecting it on the walls of a restaurant as a shorthand for Spanishness was an egregious miscalculation, insensitive to the language of art, but more important, to the tragedy depicted.

I think all of these examples suggest that, as we’re moving beyond the mere novelty of pairing sound or visuals with a meal, the focus needs to shift to what is being communicated by the resulting amalgamation; it’s not enough to simply put things side by side, to project a painting or dance in the aisles next to a diner. The languages of art and food are very different, and in many cases artists and chefs seem to be talking past each other. I think the key is to look beyond the end product of a meal or a performance or a composition to examine the processes and considerations and motivations that produced it. Despite the pervasive trend for chefs to develop dishes that visually evoke famous artworks, we should be thinking not about how a plate of food looks like a painting, but about how the work of a chef is like the work of an artist. This is where fruitful interdisciplinary conversations can occur.

At Ultraviolet, as well as in El Somni, which prominently features a 3D animation of Michelangelo’s iconic David (at one point depicted in flagrante delicto, and later shown shattering to pieces), a famous artwork is reproduced to serve as a cipher for art. I understand this impetus; for much of my career in the video game industry, I’ve observed a desire for games to be taken more seriously, to aspire to the artistic credibility and cultural respectability of film, and sometimes that means invoking the conventions of cinema in cut scenes, casting, and marketing. But in fact, in gastronomy as well as in games, these ciphers are unnecessary.

I lived in Shanghai for six years, from 2004 to 2010, and during this time, Paul Pairet was my favorite chef. When I first conceived my food opera project, back in 2006, I had his innovative cooking at his first Shanghai restaurant, Jade on 36, in mind. The food was spectacular, iconoclastic, playful, but with a serious rigor, and wildly inventive: candied foie gras on a stick, ice cream disguised in a lemon rind, sardine mousse served in tin cans. It dawned on me that the experience I was enjoying was exactly the reason I went to new music concerts, to have my preconceptions shaken, to fully engage my senses to interrogate and evaluate new stimuli, not relying on conventions of naming or presentation or other culturally learned tropes. That was when I made the leap, realizing that the kind of music system I was then designing for Ubisoft (as audio director of Tom Clancy’s EndWar) could be equally applied to the unpredictable, real-time input of the dining room, highlighting and harmonizing with the music inherent in the meal, responding to its intrinsic rhythms. Ultraviolet opened much later (in May 2012, about a week after my first food opera at Harvard), but by recognizing a parallel creative process, one that, like music, unfolds over time, I already saw the tremendous potential of pairing music with food in a way that builds on the language of each.

Having completed my two-year appointment in Valencia, I returned to Berklee’s Boston campus this fall, and I now teach music programming in the Electronic Production and Design department. One of the perks of being back in the Boston area is being able to attend Harvard’s fantastic Science and Cooking Lecture Series, which has been host to a parade of luminaries from the culinary world over the past six years. A few weeks ago I got to meet up with Andoni Luis Aduriz once again, while he and one of his chefs, Ramón Perisé, were in town to present a fascinating talk on science and emotion as part of the series.

The next day, I organized a Berklee tour for the two of them, showing them some of the fun musical gadgets we have in the EPD department before visiting the new studio facilities in Berklee’s brand new building at 160 Massachusetts Avenue. As part of the tour, my boss, EPD Department Chair Michael Bierylo, demoed a Moog System 55 synthesizer. My take on the recent resurgence of interest in modular synths is that, in comparison to the vast array of sounds available as plug-ins in today’s digital audio workstations, the constraints of an analog modular rig help focus creativity. While this may have been the first time he’d seen a Moog synth, Andoni immediately recognized this concept, the notion of freedom within limitations, as being just as true of his work in the kitchen. Identifying these kinds of correspondences is what makes working at the intersection of different practices so fascinating and why I have been incorporating these ideas into my classes, to teach students about creativity and innovation by drawing parallels to other disciplines.

The last stop on our Berklee tour was the cafeteria in our new building. The previous cafeteria (a former hotel swimming pool, I’m told) had a longstanding tradition of being converted every evening into a performance venue for student ensembles, so when designing the new cafeteria, the priorities were inverted: instead of a cafeteria that also serves as concert hall, we built a concert hall that doubles as a cafeteria. Seeing Andoni and Ramón in that space, I recognized it as a perfect embodiment of the ideals of interdisciplinary collaboration. By accommodating the concerns of two different creative practices, the potential of each is expanded, and a welcoming space emerges, awaiting unforeseeable new expressions.

Andoni and Ramón at the Berklee Cafeteria

Andoni and Ramón visiting the Berklee Cafeteria.

Tasting Notes

A group of people sitting around a long table with whisky snifters in front of them in a darkly lit room.

All the folks at this table are about to taste Chivas’s Ultimate Cask Collection, but when they do so they will also be listening, on headphones, to music that was specifically created to bring out certain taste sensations in the whisky. Can what they’re listening to affect their perception of what they are tasting? Chivas thinks so. (All photos courtesy Chivas Regal.)

A few weeks ago, I wrote some music to accompany a whisky. The premiere was the unveiling of the Chivas Regal 18 Ultimate Cask Collection at an event in London, hosted by Chivas Master Blender Colin Scott and chef Jozef Youssef. The music was not there to simply provide background ambiance to a swanky event; my objective was to change the way the whisky tastes.

Jozef Youssef is the founder of Kitchen Theory, a London-based group whose experimental pop-up dinners explore the frontiers of science and gastronomy. I met him at one of their Synaesthesia dinners last April, which drew from the growing field of crossmodal psychology to examine how the senses can overlap even in those who don’t experience classic synaesthesia.

At the whisky launch event, Youssef guided participants through a series of experiments designed to provide new perspectives on the whisky. Chivas Regal 18 is a blended Scotch whisky in which each component has been aged for at least 18 years. The Ultimate Cask Collection is a reimagining of Chivas Regal 18 that has been aged in American oak, resulting in pronounced vanilla, caramel, and orange marmalade notes. Some of these experiments one might expect at a whisky tasting, observing how dilution, temperature, and aroma affect one’s impressions. Less expected might be the texture cube I first experienced at Synaesthesia, with different materials (from velvet to velcro) on each face, allowing imbibers to experience how different tactile sensations affected their experience of taste. And similarly unorthodox was the experiment that involved listening to two musical textures while sipping; one was designed to bring out the sweet notes in the whisky, while the other emphasized its bitter qualities. Composing these textures was my job.

Jozef Youssef standing and holding a carafe of whisky as participants seating at a table look on.

Jozef Youssef guides folks through a whisky tasting featuring sound.

I actually do a lot of this kind of thing. Most of my work these days involves exploring correspondences between music and food in a series of audio-gustatory events that I call “food operas.” I’ve collaborated with a couple of chefs in recent years, and I wrote a detailed article about three events I did with Boston-area chef Jason Bond that was published in NewMusicBox in 2013. Since then, I’ve also branched out into composing music for wine and, as in the aforementioned event, whisky. Over the summer I taught a class to students in Berklee College of Music’s Music Production, Technology, and Innovation master’s program that probed the real-time technology and aesthetics behind these events; students presented their final projects alongside an eight-course tasting menu at Quique Dacosta’s Michelin-starred restaurant El Poblet in Valencia, Spain. My next food opera is a collaboration with St. Paul-based new music ensemble Zeitgeist, which will premiere in the spring.

The original inspiration for my food opera project was the idea of using real-time music deployment techniques borrowed from my work designing audio for video games over the past twenty years to score the indeterminate events of the dining room. I sought to apply recent technical advancements in games as well as ideas from the discipline of sound installation (e.g., a big multichannel speaker array to deliver sound to diners) to achieve an unprecedented level of synchronization between the senses of taste and hearing. It’s been a fascinating challenge to think about what kinds of sounds go with what kinds of foods, in a way that heightens diners’ sensory awareness of the music as well as the meal.

Since I started working on the project, I’ve become aware of a growing body of work in the field of psychology that explores crossmodal links between the senses. At the forefront of this field is a psychologist named Charles Spence, who heads up the Crossmodal Research Laboratory at Oxford University. His recent book The Perfect Meal: The Multisensory Science of Food and Cooking, co-authored with Betina Piqueras-Fiszman, is a wonderful compendium of the occasionally outlandish research being undertaken in this burgeoning field. His work has been prominently featured in the press (in The New Yorker’s current “Food Issue,” for example, and several times on NPR), and he has served as consultant to celebrity chefs including Heston Blumenthal, whose Sound of the Sea dish, introduced in 2007 at his renowned three Michelin star restaurant The Fat Duck in Bray, England, is probably the most famous pairing of sound with food. This seafood dish, plated to resemble foam on a slice of sandy beach, is served with an iPod in a conch shell, its protruding headphones immersing diners in ocean sounds as they eat, a byproduct of some of the sound pairing experiments the two conducted together. Spence has also collaborated with Jozef Youssef, and his research formed the backbone of Kitchen Theory’s Synaesthesia dinners.

Spence is widely cited for asserting that, by playing different sounds, “we’re able to show that we can change the experience in [the] mouth by about 5 or 10 percent,” as he told NPR. He goes so far as to assert that sound can be employed to fight obesity, removing sugar from foods and sonically making up for the lost sweetness.

I had a chance to meet Spence when I visited Oxford last February to present my work to his Crossmodal Research Lab. My friend Janice Wang, who helped out with my Bondir food operas, recently moved from Boston to Oxford to pursue her PhD in Spence’s group. Her own research was recently featured in the Financial Times, in an article entitled, “I Use Music to Change How Food Tastes.” Janice is also president of the Oxford University Blind Wine Tasting Society, which, at the time of my last visit, had recently administered a crushing defeat to rival Cambridge University. While I was in town, Janice and I collaborated on an event at the delightfully cozy 1855 Wine Bar in Oxford, working with sommelier Alistair Cooper to pair six different wines with six different musical textures. She also accompanied me on my visit to Kitchen Theory and introduced me to Jozef Youssef in person.

Janice is part of a group who identify themselves as Crossmodalists, committed to promoting experiences that engage and reinforce the links between all the senses. Perhaps due to the influence of figures like Spence and Blumenthal, there seems to be quite a scene for crossmodal dining in the UK at the moment, and music figures prominently. I attended one of this group’s rehearsals last April, which included a parade of scents, live piano, an edible painting, and choreography that responded to flavor—a fascinating mix.

Dining is a profoundly multi-sensory experience. A lot of times we might describe a good meal strictly in terms of taste, but in fact eating involves all of the senses, and psychologists are quantifying to what extent this is true. Smell is the most obvious supporting sense, but also the feel of the silverware, the color of the plate, the food’s appearance, even the mood of the diner, all of these elements combine to affect our perception of a meal. Even after food enters the mouth, temperature and texture merge with taste in our evaluation of a dish. In Spence and Piqueras-Fiszman’s book, I learned that we actually have two different senses of smell: the orthonasal is what we typically think of as “smell,” applying to external odors in our environment, whereas the retronasal is concerned with what is already in the mouth and is thought to have evolved independently and much earlier. This is one reason (alongside aeration) that sommeliers sometimes slurp wine, to aid in retronasal evaluation.

Sound plays a huge role in eating. Studies show that loud sounds dull our sense of taste, which may explain why airplane food tastes so bland, and why people order more tomato juice on planes, as umami flavors are the most resilient to volume. The sound of mastication also has been shown to affect our perception of an item’s freshness, improving the impression of carrots and potato chips. To me, all of these observations underscore the notion that developing sound to pair with food represents an exciting new arena for aesthetic expression.

Three people at a table wearing headphones and sipping whisky

Applying the role that sound has in affecting taste perception to drinking whisky.

My mandate for the Chivas event was very specific, and so these whisky textures represent my first attempt to write music that explicitly drew on the crossmodal psychology research. As Charles Spence will tell you, high frequency sounds have been shown to make things taste sweeter, while low frequency sounds (low brass, in Spence’s experiments) make them more bitter. My textures are short, simple, and very consistent throughout their short durations, attempting to hew closely to the script, although I still wanted to provide a bit more musical interest than a static sustained tone. They are similar enough that useful comparisons can be made where they diverge.

My sweet texture is built from recordings of a flute, a clarinet, and wind chimes. The two-voice main melodic part is in short diatonic phrases, slow and legato, a type of phrasing I often use to create a sense of peaceful, suspended time. The motion is largely stepwise, the harmonies consonant, predominantly thirds and sixths. There may be a slight evocation of the type of traditional melodies one might expect to hear in the Scottish highlands, albeit in fragments. In the background is a slow moving harmonic pad to provide a bit of context, and wind chimes tinkle throughout. There’s no steady pulse. The overall impression should be of stability and resolution with mellow timbres in a floating, high register.

In my bitter texture, the overall frequency range is much lower, as the cello takes the dominant role, playing much more aggressively and roughly than anything in the sweet texture. The cello part is broken into short, intermittent phrases, similar to the lead part in the sweet texture, and similarly in two voices, as it’s all double stops, but here there is a steady pulse, and phrases tend to emphasize more dissonant intervals. Wood blocks and temple bells further emphasize the pulse, while a low drone underneath provides a harmonic reference point. Rather than emphasize brass, as in Spence’s study, I chose to use sounds relating to wood, as the bitter elements were also linked to the whisky’s having been aged in American oak casks, and I wanted to bring out that quality as well.

There’s a question that could be raised about linking the sound of wood (woody resonance of string instruments, sound of struck wood) to the taste of wood (oak). On one hand, we might say that the taste and sound are not related at the level of sensation, that one does not necessarily evoke the other. On the other, we might observe that most humans have learned to associate these things through a lifetime of interaction with wood. The question of what’s learned or culturally conditioned as opposed to what’s innate comes up all the time in designing these pairings.

As I’ve learned more about the psychological approach to the dining experience, I’ve wondered a lot about the point at which the work of the researcher ends and the work of the artist begins. Whereas in a lab experiment, one might want to isolate certain parameters of a taste experience, a lot of times as a composer, I’m trying to blend and merge and complement. Rather than dealing with pure tastes like sweet or bitter, I’m typically interested in a complete dish that has balanced flavors and textures, thinking how music can join in as another set of ingredients. It’s usually not my objective to simply replicate a dish in sound, but to complement and transform it, guided by my experience writing music to accompany dance or video games. In a way, maybe the rules of psychological association are like the rules of music theory, serving as a reference point for a composer, a framework or a palette that can then be applied, twisted, or inverted to far ranging aesthetic ends.

A group of beakers filled with various types of Chivas whisky.

Ultimately a satisfying aesthetic experience—whether eating a meal, savoring a glass of whisky, listening to a piece of music, or participating in a synaesthetic immersion that involves all of the above—must be more than an experiment.