Electroacoustic Music with Video: Comparison with Sound for Film

If a concert composer is using video with their music, that composer is writing a film score.

Written By

Alice Shields

Electroacoustic music with video is now frequently programmed in classical concert venues. In such a concert, a video is projected on a screen and seen.  It can present changes in tempo color, shape, distance, density, and gradual or abrupt transitions.  The electroacoustic composition played with this video is projected from loudspeakers and heard. It can also present changes in tempo, pitch, loudness, distance, duration, timbre, rhythm, and gradual or abrupt transitions.  With so many parameters available in both these sense modalities for artistic coordination and relationship with each other, there are clearly many exciting and dramatic or delicate and subtle ways in which video and sound can be organized together into an effective work of art.

But after attending a number of recent concerts in which pieces for electroacoustic music with video were performed, it struck me that in many of these pieces there is only the most minimal relationship between the video and the audio.  Whatever parameter was changing or remaining static in the video was not occurring at the same rate as changes in any parameter of the electroacoustic music. The differences in their rate of change and the type of change did not seem to be patterned or deliberate.

Spoken text has the power to bridge over and meld together music and video, whatever their relationship to each other. But most of these pieces of electroacoustic music with video had no spoken text. That lack of spoken text draws attention all the more to the lack of coherent organization between the video and the audio.

But could what I experienced in those concerts of electroacoustic music with video be a deliberate aesthetic, in which the composer and video artist are consciously avoiding any tie-in with the other sensory modality?

The Theoretical Aesthetic of Independent Music and Visuals

Could there be a deliberate aesthetic in which the composer and video artist are consciously avoiding any tie-in with the other sensory modality?

Earlier examples cited of such an aesthetic often include John Cage and Merce Cunningham’s collaborations of the 1960s. These were often presumed to be in a style in which there was a deliberate non-relation of music and dance, in which neither supposedly related to the other. This aesthetic still persists in certain collaborations between composers and choreographers.

Nonetheless, in the performances of Cage and Cunningham that I attended, the two artists managed to limit and control their music and movement quite carefully, so that the random synchrony I’ve seen in current music with video seemed rarely, if ever, to occur. What I witnessed was a palpable tension between the live dancer and the sound. One could see Cunningham’s bodily resistance to falling into moving in the pattern of the music. His dancing became counter-movements that opposed the patterns being heard in the sound. This tension was for me the aesthetic they were aiming for, and was for me the most interesting part of their performances. But as far as I could tell, that interesting tension relied primarily upon Cunningham’s highly trained professional movement skills, and secondarily upon the tendency for Cage to choose a limited number of timbral possibilities in his musical accompaniment.  The limited timbres created a bland, tension-less sound palatte that helped frame the subtle tension of the dancer’s patterned muscular resistance.

While moving, the aesthetic required Cunningham to counter what was going on in the music, to not fall into the natural tendency to move his body with the music.  By moving in patterns not present in the music, Cunningham created a distinct relationship of opposites between his movements and the music. In short, the aesthetic between Cage’s music and Cunnigham’s movement was indeed a relationship, one of opposites, of avoidance of the pattern of the other modality, not a random or chance relationship.

But the aesthetic of opposing patterns is not what I observed in these recent electroacoustic concerts with video.  Instead, there was essentially no patterned relationship at all.

Weak correlation communicates irrelevance.

What we see on the video and what we hear from the live or prerecorded music is mostly unrelated to each other: not rigorously opposed, just weakly different. They are not radically different from each other and therefore there is little tension or drama evoked.  The music and video even sometimes, for brief moments, resemble each other’s degree of change, and then change into something else.  The result is weak correlation, not opposition. Opposition communicates tension and drama. Weak correlation in anything onstage or off is likely to communicate irrelevance, non-relation.

If a concert composer is using video with their music, that composer is writing a film score.

I found myself needing a more precise way to analyze what I felt is successful and what I thought might be improved upon in these works of electroacoustic music and video. The basic issue seems to be how to clarify and intensify the relationship between sound and video image.  So I looked into some concepts from the world of film sound. After all, if a concert composer is using video with their music, that composer is writing a film score.

abstract business background with media screens and silhouettes of business people holding two giant puzzle pieces

Sound and Visual Categories in Film

The three sound components used in the film industry are: speech, music, sound effects/ambient sound, and the interactions between them.  For concert electroacoustic music, we could also include sounds constructed through software synthesis or other electronic means in either the “music” or “sound effects/ ambient sound” categories, based on the dominance of identifiable pitch or unpitched noise in each sound.

Two aspects of film that may be relevant to this discussion are mise-en-scène, which is what is visible in the frame, and the way it’s arranged and shown, through editing the transitions between different shots.

Ways of Relating Video to Electroacoustic Audio, Based on Film Sound Techniques

A. The direction and depth of the image, in relation to the direction and depth of the sound

The composer has a choice when a particular image occurs on the video. He or she can:

(1.) Match the perceived direction and depth of the visual image.  When the visual image changes its position, change the direction and depth of the sound to match.

(2.) Contradict the perceived direction and depth of the visual image, by using the opposite in the sound, e.g. if the image appears far away, then have the sound be very present, close to the audience.

(3.) Maintain a constant sound placement, regardless of the direction and depth distance of the video image.  This is the current preference of the film industry, which tries to produce a continuous sound track with approximately the same sound volume and depth of distance to the microphone for every character, regardless of where the character appears.

Any of these choices could be effective in electroacoustic music with video, if applied with enough precision in the use of reverb and timbral filtering to move the sound to its intended depth.

B. Continuity of sound, as opposed to disjunction of sound: Smooth versus Rough editing

In smooth editing the sound is continuous, a flow with no sharp breaks. All of the sound edits are gradual. In rough editing or “jump cutting” the sound begins and ends abruptly, a series of sounds with sharp breaks.  All of the sound edits are abrupt or even separated by silences.

Through smooth audio editing, continuous sounds can bind together disparate images.

Through smooth video editing, continuous visuals can bind together disparate sounds.

The composer has a choice when a smooth visual flow occurs on the video. He or she can:

(1.) Match the smoothness of visual flow with the smooth editing of the sound score, resulting in a continuous flow of sound score. Any rough edges have been smoothed over.

(2.) Contradict the smoothness of visual flow with rough jump cut editing of the sound score.

The composer has a choice when a smooth visual flow alternates with jump cuts on the video.

He or she can:

(1.) Match the video’s changes from smooth visual flow to jump cuts, resulting in a smooth flow in both video and audio, which then changes into a sharp jump cuts in both video and audio.

(2.) Contradict the video’s changes from smooth visual flow to jump cuts, resulting in smooth visual flow along with rough jump cuts in the sound, and then rough jump cuts in the video along with smooth, continuous flow of sound.

(3.) Maintain a consistent smooth or a consistent abrupt editing of sound events, regardless of whether the video is moving smoothly or in jumps.

However, when sound and visual image are synchronized, that is, moving in the same tempo in the same smooth or abrupt manner, then the video movement is perceived more clearly.

C. Some other possible relationships between a sound and a video image:

Empathetic relation: The sound expresses the same emotion as the video image.

Anempathetic relation: The sound is without emotion in relation to the image.

Ironic sound: The sound expresses the opposite emotion from the image.

Surrealistic sound: Without rational connection to the video image.

Counterpoint: Loosely connected in time, a series of musical phrases of similar content is heard while a series of images of similar content, unrelated to the sounds, is seen.

Expressive editing: A sound is exaggerated and paired with a similar visual activity in an image.

Some additional references:

Zetti, Herbert. Sight, Sound, Motion: Applied Media Aesthetics, 7th Edition. New York: Cengage Learning (The Wadsworth Series in Broadcast and Production), 2013.

Chion, Michel. Audiovision: Sound on Screen. New York: Columbia University Press, 1994.

Ribrant, Gunnar. “Style parameters in film sound.” Stockholm: Stockholms Universitet Filmvetenskapliga insitutionen, 1999. (This article is available directly online as a PDF download, or through emailing the author.)