Music and sound on (and off) screen

It seems a long time since I visited the books on screen sound, but my exciting new second supervisor Beth said I should check out one of the foundation texts. This is Michel Chion’s Audio-vision: Sound on Screen. The first thing that grabs my attention is his pointing out that while “the image” in cinema is framed by the edges of the screen. “What is specific to film is that it has just one place for images – as opposed to video installations, slide shows, and other multimedia  genres, which can have several.” (He talks about how little, since the very earliest days of cinema, directors have attempted to change the edges of that frame, and “the rare experiment” of changing the aspect ratio within a film. There’s a lot more on that “rare experiment” in this blog. The one I’ll always remember fondly is in fact the two aspect ratio changes in Galaxy Quest.)

But this is to preface a point that sound in cinema has no frame.

Luckily for the makers of films with sound in the first four decades, our human need to mentally connect sounds with what we’re looking at tricked audiences into believing that sound came from (for example) the feet of our protagonist walking across the screen, when in fact there was just one speaker behind the screen. With the arrival of Dolby, Chion says, directors and their audio designers are able to create soundspaces that exist beyond the borders of the screen. Gorbman (who by the way also translated this book) introduced me to diegetic and non-diegetic sounds – those that happen in the world of the film (dialogue, music that seems to come from a radio on screen), and those that happen outside that world for the audience’s perception only (the rousing orchestral piece as the cowboys gallop across the plain). Chion splits diegetic music into two: visualised sound that happens on screen, and off-screen or acousmatic sound, which happens offscreen but is still part of the story world.

However he admits that this split has been criticised for being oversimplistic. Where does the adult voiceover representing the internal voice of the baby in Look Who’s Talking go in this model? And is that radio on-screen? Or, given that its transmitting sound from some distant studio in the movie world does that count as offscreen? So he ends up with a model that includes: onscreen; offscreen; on-the-air; internal; ambient and nondiegetic.

I am also interested in a earlier chapter which describes three listening modes.

1. Causal (not casual) listening is where we listen to a sound in order to gather informational about its source. Chion says this is the most common form of listening (but without evidence for that claim).
2. Semantic listening is where we listen to a “code or language” to interpret a message. Chion points out that Casual and Semantic listening can be employed at the same time.
3. Reduced Listening  – This is the intriguing concept coined by Pierre Schaeffer, for the listening mode that focuses on the traits of the sound itself, independent of its cause and meaning. It seems hard to do: participants in reduced listening experiments constantly seek cause or meaning. He goes on to say that, though its hard to concentrate on reduced listening, we all do a rudimentary form of it when “we identify a pitch or tone or figure out the interval between two notes.” However, a full description of the sound can not be achieved in a single hearing. So it must be fixed, a recording, not a live performance. Intriguingly, Chion says that such sounds “thereby acquire the status of veritable objects”

Whats interesting about this last form is the relationship between reduced listening and acousmatic sound. Chion says Schaeffer “thought that the acousmatic situation could encourage reduced listening.” But he disagrees. Our first instinct will be causal, he argues, seeking the source of the sound. But a “seasoned auditor” he argues  “can exercise causal listening and reduced listening in tandem.”

I feel there is something in this. Chion isn’t writing about games, but in my reading of Red Dead Redemption especially I realised I was learning to listen to the repeating sound loops in a different way. At first I was reacting in a causal way, swirling my point of view around, seeking the source of every cue. As I grew more “seasoned” I was listening in a different way. I am not sure it it was Reduced or Semantic, but it was more than Causal. Of course, that might have something to do with the other distinction Chion makes, between Active and Passive perception.

There is something here I feel that might help in better evaluating immersive audio experiences in heritage.