Resonance: Sound, music and emotion in historic house interpretation

Just drafted an abstract for my Sound Heritage presentation:

This presentation explores what computer games can teach us about emotional engagement in cultural heritage interpretation. Beginning with a model of emotional affect drawn from the work of Panksepp and Biven (Panksepp, 2012), Lazarro (Lazarro, 2009), Sylvester (Sylvester, 2013)and Hamari et al (Hamari et al., 2014), it reveals how music especially has become a versatile emotional trigger in game design.

Drawing on the work of Cohen (Cohen, 1998)and Collins (Collins, 2008)eight functions that music has in games:

Masking – Just as music was played in the first movie theaters, partly to mask the sound of the projector, so music in new media can be used to mask the whir of the console’s or PC’s fan.

Provision of continuity – A break in the music can signal a change in the narrative, or continuous music signals the continuation of the current theme.”

Direction of attention – patterns in the music can correlate to patterns in the visuals, directing the attention of the user.

Mood induction; and,
Communication of Meaning- the nice distinction here is between music that makes the user sad, and music that tells the user “this is a sad event” without necessarily changing the user’s mood.

A cue for memory – The power of the music to invoke memories or prepare the mind for a type of cognitive activity is well recognized in advertising and sonic brands such as those created for Intel and Nokia.

Arousal and focal attention – With the user’s brain stimulated by music s/he is more able to concentrate on the diagesis of the presentation.

Aesthetics – The presentation argues that all too often music is used for aesthetic value only in museums and heritage sites, even if the pieces of music used are connected historically with the site or collection.

As an example, the presentation describes a project to improve the way music is used in the chapel at the Vyne, near Basingstoke. Currently, a portable CD player is used to fill the silence with a recording of a cathedral choir, pretty, but inappropriate for the space and for it’s story. A new recording is being made to recreate about half an hour of a pre-reformation Lady Mass, with choisters, organ and officers of the church, to be delivered via multiple speakers, which will be even more pretty but also a better tool for telling the place’s story.

With a proposed experiment at Chawton House as an example, we briefly explore narrative structure, extending the concept of story  Kernels and Satellites described by Shires and Cohan (Shires and Cohan, 1988)to imagine the cultural heritage site as a collection of narrative atoms, or Natoms (Hargood, 2012), both physical (spaces, collection) and ephemeral (text, video, music etc.). Music, the presentation concludes is often considered as a “mere” satellite, but with thought and careful design there is no reason why music can not also become the narrative kernals of interpretation.

 

COHEN, A. J. 1998. The Functions of Music in Multimedia: A Cognitive Approach. Fifth International Conference on Music Perception and Cognition. Seoul, Korea: Western Music Research Institute, Seoul National University.

COLLINS, K. 2008. An Introduction to the Participatory and Non-Linear Aspects of Video Games Audio. In: RICHARDSON, J. A. H., S. (ed.) Essays on Sound and Vision. Helsinki: Helsinki University Press.

HAMARI, J., KOIVISTO, J. & SARSA, H. Does Gamification Work? — A Literature Review of Empirical Studies on Gamification.  System Sciences (HICSS), 2014 47th Hawaii International Conference on, 6-9 Jan. 2014 2014. 3025-3034.

HARGOOD, C., JEWELL, M.O. AND MILLARD, D.E. 2012. The Narrative Braid: A Model for Tackling The Narrative Paradox in Adaptive Documentaries. NHT12@HT12. Milwaukee.

LAZARRO, N. 2009. Understand Emotions. In: BATEMAN, C. (ed.) Beyond Game Design: Nine Steps Towards Creating Better Videogames. Boston MA: Course Technology / Cangage Learning.

PANKSEPP, J. A. B., L. 2012. The Archaeology of Mind: Neuroevolutionary origins of human emotions, New York, W. W. Norton & Company.

SHIRES, L. M. & COHAN, S. 1988. Telling Stories : A Theoretical Analysis of Narrative Fiction, Florence, KY, USA, Routledge.

SYLVESTER, T. 2013. Designing Games – A Guide to Engineering Experiences, Sebastolpol, CA, O’Reilly Media.

Narratives in social science

I ought to watch out for the literature rabbit holes I can fall down. After my last foray into narratives and sociology, I got sucked into another work that was only tangentially about what I’m studying. This one though did at least have a few quotable quotes I might want to use later.

Brian Alleyne kicks off his Narrative Networks; Storied Approaches in a Digital Age by asking (on page 2) “What is narrative?”

“Narrative, in its simplest sense, consists of a series of connected events, and a particular way in which theses events are told. The first element is the story, and the second element is the narrative discourse… It follows from this that a story can be rendered through different narrative discourses”

He goes on (page 40) to reference “Jerome Bruner (1986; 1991) [who] has argued that humans make sense of the world in two fundamental ways, in two cognitive modes: paradigmatic and narrative. In the Paradigmatic mode the human mind recognises elements as belonging to two categories in a classifying operation.So in this cognitive mode, when we encounter an object, we seek to map it onto already existing classificatory schemes pf objects, trying to work out what kind of object it is. […] For Bruner this mode of cognition is most characteristic of science.
“In the the narrative mode of cognition we seek to connect people and events into a temporally coherent whole. The passage of time is important in this mode.As Paul Ricoeur (1984) argies, some sense of the passage of time is quite fundamental to how humans understand themselves and the world around them. The narrative mode of cognition is one with organises ideas and experiences into stories and is seen to contrast with the paradigmatic, scientific mode in that it operates in an emotive or emotional and expressive register as opposed to the rational register of paradigmatic cognition. These ideas are obviously abstract ideal models of cognition and cannot always be easily separated from one another in seeking to account for how people go about making sense of the the world.”

So it can the argued that a label in a museum or cultural heritage site, which categorizes an object, can’t connect the visitor emotionally to that object, unless they bring a story with them. But a purely story led interpretation of a site or collection can’t help the visitor understand it.

Later on in the book (page 92)he touches upon the Narrative Paradox “The fundamental issue here is that however defined, narrative text is characterised by a coherence that links human, and non-human agents, their actions, experiences and other happenings into a temporal chain – the following of which leads us to some kind of conclusion. The problem here for thinking about “hypertext narrative” lies in the very nature of hypermedia: unless the author of a hypertext network deliberately imposes a narrative structure on that collection of texts, the collection will have a degree of openness which militates against narrative coherence” using classic text adventure games Zork and The Hitchhikers Guide to the Galaxy as examples he concludes “Interactive fiction therefore sacrifices the open-ended possibilities of hypertext in order to maintain some degree of narrative coherence.”

He also summarises the Narratology/Ludology debate, on page 93 and later (pages 116-121. As a gamer and a narrativist, he offers a balanced view, citing many ways in which games are not narratives but also pointing out that “Narratology has been part of the videogame designer’s toolkit from the start (Crawford, 2003; Rollings & Adams, 2003; Ryan, 2001; Salen and Zimmerman, 2010). Many computer are based on the three-act narrative structure of situation, conflict and resolution, that that same structure being repeated as the player moves through the game.”

In his debate on whether games can be analyzed narratively, he looks specifically (page 119) at history themed games like my old favourite Civilization: “In order to make these games worth playing narrativity has to be be balanced with playability, which means departing from the tight emplotment of historical that is at the core of historical narrative.”

Does any of this leave me any more enlightened? Not particularly, but I enjoyed reading it.

Ludology vs Narratology Revisited

My previous post on the Ludology vs Narratology debate is one of my most visited, and I note that that the term frequently appears in searches that bring people to this site. So, in the spirit of “give the people what they want”, let me offer up this morsel.

I’ve been reading Espen Aarseth’s paper, A Narrative Theory of Games, and he both offers insight into the debate (as, it seems, a pretty early participant), and, more importantly, does a reasonable job of debunking the whole thing. Along the way, he demonstrates a masterclass in academic rhetoric, but you can’t help but feel its personal too.

In reality this is not one, but two debates conflated: one is the design-oriented discussion of the potential and failings of game-based narratives, and another is the discussion of whether games can be said to be stories.

Aarseth points the finger at Henry Jenkins’ Game Design as Narrative Architecture, for setting up the two sides of Ludologists and Narratologists. (Though in that paper, Jenkins appears to point the finger back at Aarseth for coining the word ludology in the first place.) Aarseth argues that pitting one side against the other was “unfortunate, because it obscured the fact that all the so-called “ludologists” were trained in narratology and used narratology in their studies of games.”

Aarseth argues:

The “ludologist” position was not, as has been claimed, “to see the focus shift onto the mechanics of game play” (Jenkins 2001) but to emphasize the crucial importance of combining the  mechanical and the semiotic aspects and to caution against and criticize the uncritical and unqualified application of terms such as “narrative” and “story” to games. In other words, the ludologists’ critique was a reaction to sloppy scholarship (in which key terms are not defined), one-sided focus and poor theorizing, and not a
ban against the application of narrative theory to games as such

(This next bit, I love)

That this challenge has been mistaken for a ban on the use of narrative theory in game studies is nothing less than amazing, and perhaps goes to show that humanist academics are often less astute readers, scholars and interpreters than their training gives them occasion to presume.

Oh, but what’s this?

Anyone who echoes Jenkins’ misleading nomenclature of “ludologists” vs “narratologists” simply has not read the literature itself.

That’s me well and truly told.

Kernels and Satellites

Last week I reminded myself that I hadn’t sought out Cohen and Shires’ Telling Stories: A Theoretical Analysis of Narrative Fiction, which had been referenced in an article on game narrative. It was low down my list of priorities, mainly because it was written in 1988 – which feels like ancient history in citation terms. That shows in chapter one, where defining “narrative” as recounting “a story, a series of events in a temporal sequence,” the authors explain that:

our culture depends upon numerous types of narrative: novels, short stories, films, television shows myths, anecdotes, songs, music videos, comics, paintings, advertisements, essays, biographies, and news accounts.

Games aren’t mentioned, and I guess that’s no surprise, given that in 1988 computer games were still a relatively youthful medium, and the audience for games were relatively youthful too. The investment of Hollywood amounts of money in game narratives was still a twinkle in programmers eyes. If they looked at games at all back then, the authors might well have consciously excluded them from their analysis, because, the central premise of their book is “the events making up a story are only available to us through telling”, which might (arguably) exclude the procedurally generated narratives that most games provide.

But one of their ideas does have some relevance to game narratives. The article I looked at last week made reference to this passage (page 54):

From the vantage point of a completed sequence, events function either as kernels or as satellites. Kernel events raise possibilities of succeeding or alternative events – what we can call, taking the term rather literally, “eventuality.” They initiate, increase, or conclude an uncertainty, so they advance or outline a sequence of transformations. Satellite events, on the other hand, amplify or fill in the outline of a sequence by maintaining, retarding, or prolonging the kernel events they accompany or surround.

In game narrative terms this is a neat summary of how games work as a storytelling medium. In more scripted games such as Red Dead Redemption, the sequence of Kernals is quite rigid, and the satellites are optional or even (in the case of games like Skyrim) procedurally generated. I remember nearing the end of RDR: I’d helped John Marston, the character the game had been following, to track down and (mostly) kill his old buddies from the gang he had run with, and confront his old boss, who throw himself off a mountain. Marston had been given back his farm, and wife and child, and the game challenges had become less about death and destruction, and more about production and family life – rounding up cattle and and the like. Then a blicking icon had appeared on the game map, telling me that I was ready to play the nest kernel event.

I didn’t want to, I knew the game was nearing the end, and having discovered Marston’s life story, I knew it wouldn’t end well. I wanted to prolong the rural idyll of farming, hearth and home. So I found satellite quests to prolong the current kernel. I became obsessed with beaver hunting, promising myself I wouldn’t play on to the next kernal event until I’d found the five beaverskins a crazy glider pilot Marston had met in Mexico needed for his glue. I spent days and days hunting beaver. It became a running joke with my wife.

But after shooting the first two, it seemed the beavers had gone into hiding. There was beaver drought, it seemed, by every river in the gameworld – and yes I did try every one. So with a heavy heart, I turned John Marston back towards his fate. Damn, was I emotionally engaged.

But even in purely procedural games, the idea of kernels and satellites works. As Tynan Sylvester points out, in a game like The Sims, the narrative is reliant on the interpretation of the player:

This story was co-authored between the player and the game. The game simulated some simple event (attraction between redhead and roommate), and the player ascribed meaning to it (jealousy and frustration) the same way he might have for the Michotte balls, even though that emotion was not actually in the simulation. The next part of the story was cued by him when he orchestrated the murder. The game simulated the logistics of firey deaths, but the sense of sorrow and revenge was, again, ascribed completely by the player. Most of this story is apophenia – present of the Player Model, absent from the Game Model.

While not talking about games, Cohen and Shires manage to predict how the random calculations of a procedural game can become an emotionally engaging story:

While kernels may appear to function as primary events and satellites as secondary ones, satellites are as important as kernels to a story sequence. Furthermore, an event’s status as a kernel or satellite depends entirely upon a particular sequence and not on the event itself, which does not possess the ability to advance or amplify a transformation on its own. An event acquires its kernel or satellite function for a given sequence through its placement in the sequence, because the sequence is what sets the events in relation to each other.

I like to play Civilization, which is an example of unscripted, procedural game. Some games are more satisfying than others, when the random generation of events becomes, in my mind, the thrilling story a plucky little nation that could. Sometimes, despite my best efforts to manage my nascent state “events, dear boy, events” conspire to make the game boring – but the advantage of procedural games is that if its boring, you can start again. Well designed procedural games are the ones that keep you restarting because of the all the great narratives you’ve discovered on previous plays. Ones that are consistently boring don’t get restarted, they get turned off.

The challenge for cultural heritage sites is that they can’t be restarted, so a purely procedural approach of interactive narrative would not be constructive. Some degree of scripting – the selection and ordering of narrative kernals is required.

Narrative Structure and Games – Backstory?

I’ve started writing up my literature review. And that has sent me back to the literature itself, to try and make head or tale of the cryptic comments I made to myself when I read it the fist time. Take for example Barry Ip’s two part article in Games and Culture, Narrative Structures in Computer and Video Games. Ip offers, in part one, his own pretty complete literature review of story in games. Indeed I could quote him extensively and move on, except there are some things he said that obviously prickled me. And now I’ve had to re-read him to find our why.

Overall, its the useful summary of game narrative I thought it was. It saves me having to play games for months, with a stopwatch to hand. And looking at it again, I’m reminded of a particular reference to a now out of print and distribution book I was going to check for in the library, but never did*. It needs a bit of updating, mostly by references to Tynan Sylvester’s work, and Terence Lee’s piece on emergent narrative.

But the thing that gets my goat is his use of the term “back story”. I was obviously annoyed this quote:

Backstories are usually presented just before a game begins or seen written on the back of game packaging or in its instruction manual to capture a player’s attention as well as set the scene for the entire game.

Now to my mind, what he is describing is the “blurb”, or at best a prologue that states “what has gone before” and, maybe sets the scene. Whereas I think of backstory as the background created for a fictional character, which isn’t explained at the start of the narrative (where it really becomes part of the narrative) but may be referred to as the narrative progresses. It is complete (if anywhere) only in the author’s head, but the reader (or player) can construct their own understanding of it from the clues peppered throughout the narrative. This was one aspect of Red Dead Redemption that I liked, the player’s avatar had a backstory (and not a very pleasant one) that the player could only piece together during the game. In contrast the player’s avatar in Skyrim has no-backstory – other than he is a captive at the start of the game.

So this time, rather than tap out a barely understandable note to myself, i went to the dictionary to be proved right. The online Oxford dictionary says:”A history or background created for a fictional character in a film or television programme” Aha! I was right! But then it goes on to give an example: “‘a brief prologue detailing our hero’s backstory'” Curses! That’s more like Ip’s definition … And Merriam-Webster agrees with Ip. On the other hand, Wikipedia backs up my understanding (today at least).

Oh! I don’t know, maybe I should just live with it. It seems I’ve spent more time niggling at the word than actually writing – which may of course have been the point.

 

*Kernels and Satellites from Cohen and Shire’s 1988 Telling stories: A theoretical analysis of narrative fiction

Story, Time and Place

This is the Prezi and below are my notes in preparation for a short presentation I gave to a Digital Humanities seminar group at University today. Hosted WordPress still can’t deal with embedded Prezi’s yet so click the link at the start to see the slides. And my notes below are just notes, so you’ll have to imagine me riffing off them to make an entertaining, compelling and coherent (I hope!)  presentation.

The Lindisfarne Gospels is an illuminated manuscript gospel book produced around the year 700 in a monastery off the coast of Northumberland at Lindisfarne and which is now on display in the British Library in London.

Illuminated

Very little structure to the text, no paragraphs etc

In the 10th century an Old English translation of the Gospels was made: a word-for-word gloss inserted between the lines of the Latin text by Aldred, Provost of Chester-le-Street.

This is the oldest extant translation of the Gospels into the English language, and a great example of a reader interacting with the text.

Laurence Stern created one of the first texts to be interacted with. Tristram shandy is epistolary novel, but it’s more than that, sampling other works of literature to bring new meanings.

He chose the format, paper, type and layout of the novel. It’s a book to be played with.

Last year’s Building Stories. Like Tristram Shandy, a story to be played with. Dan Clowes (author) suggest leave bits of it around your own building to chance upon.

Gorge Méliès, regarded as the first person to recognize the potential of narrative film. Goes beyond sequential time/movement and to imaginary places.

Voyage Dans La Lune , special effects, Jump cuts, locations etc started a century of narrative experimentation.

For example music

diegetic music (where musicians are playing in the story, or charcters are listening to the radio for example),
nondiegetic music (where as she says “an orchestra plays as coyboys chase indians upon the desert”) and
metadiegegtic music (where we hear a character “remember” a bit of music).
She also talk about themes, and what Wagner called “motifs or reminisence.”

But despite all this innovation, don’t you find some films “Same-y”?

Not every film has been a success of course. After some test screenings Walt Disney called in “script doctors” to fix The Lion King

Christopher Vogler – Joseph Cambell, Hero’s journey applied to Lion King, then book The Writer’s Journey.

Save the Cat! Blake Snyder’s Beat Sheet – Almost an algorithm for scripting film. 110 pages

Opening Image – page 1 A visual that represents the struggle & tone of the story. Set-up – ten pages Expand on the “before” snapshot. Present the normal world. Including: Theme Stated page 5 – say it “with great power comes great responsibility. Catalyst page 12 – the world turns upside down. Emotional shock. Debate for thirteen pages – Dare our heroes actually explore the new world? Break Into Act Two page 25– The main character makes a choice and the journey begins. B Story begins on page 30— This is when there’s a discussion about the Theme – Timon and Pumba in the Lion King. Fun and games twenty five pages— the action, the roller coaster ride the caper. Midpoint p55 — Success!’ But Bad Guys Close In for twenty pages.bAll is Lost page75 – The opposite of Success. And emotional Nadir.
Dark Night of the Soul for ten pages – woe is me. Hit rock bottom. Break Into Three (page 85) – the B story provides the solution to the A-story. Finale twentyfive pages – This time around, the main character incorporates the Theme – the nugget of truth that now makes sense to them – into their fight for the goal because they have experience from the A Story and context from the B Story. Act Three is about Synthesis! Final Image page 110 – ride into the sunset, a changed character.

Of course the audience have to see each frame of the film in the order in which it is presented. Only the director gets to play with chronology.

Games give back the power to explore the narrative

Procedural narratives versus authored narratives.

Describe RDR, starts off interactive, but delivers fewer and fewer choices towards an inevitable end. Authored, nor procedural. Are procedural stories only in need to great endings?

Is this an insight on the Narrative Paradox?

I’ve been analysing the data collected for my evaluation of Ghosts in the Garden. Yesterday I sent my preliminary observations to the guys who created it, and by the end of today I hope to have completed the first draft of my full report. If everyone approves I’ll share it all here in future.

But I did want to share, and possibly sense-check, my key bit of insight. We asked participants to rate how strongly they agreed with a number of statements about the experience, using a seven point Likert scale. So here’s a sample of the sort of response we got to a simple statement, “The Ghosts in the Garden experience added to my enjoyment of the visit today”:

A simple bar chart, showing that most visitors strongly agreed that the Ghosts in the Garden experience added to the enjoyment of their visit

Which is very nice and positive. But I’m looking for emotional engagement, and the responses to the statement “The story I heard had a real emotional impact on me” were less positive:

Most users were non-committal about emotional engagement, and some did not agree that the story had any emotional impact.

Now, to be honest I’m not sure I’m asking the right question here. I used this wording only because we ask the question in a similar way at the National Trust where I work, and this being my first bit of research I wanted something that I could easily compare these data with. (For comparison, some of the National Trust’s most emotionally engaging places get something over 20% of visitors ticking to top (number seven) box, in this sample, only about 8% did.)

Asking people to rate their emotional response is according to many, a futile task, and there are likely better ways to measure it, but allow me to indulge myself for a moment. If I can assume that the story was indeed not as emotionally engaging as it might be, I might ask myself “why not”?

Remember, Ghosts is the Garden has been described by its creators as a “choose-your-own-adventure style story.” When you pick up the “listening device” your make your first choice – balloons or fireworks – and then, at every point you are offered a choice of two locations to explore, and the narration explains that the choices you make will affect the outcome of the story. And yet when we asked users whether they agreed that the choices they made changed the story, quite a bit of skepticism was evident:

A number of people agreed that the choices they made changed the story, but more were less convinced.

So my next overriding question is, did confidence that they were changing the story affect users’ emotional engagement? I think I can do a cut of the data to find that out, but the sample size is too small to be really confident in what it might show. Given what I’ve been uncovering about the story structures of the video games I’ve played though, I beginning to wonder if there’s any value to this sort of interactivity. For me, Skyrim, with its wider story structure has been a lot less emotionally involving then either Red Dead Redemption or Dear Esther, both of which take the player towards one single, inevitable, ending. And then there’s the Narrative Paradox.

I wonder whether, rather than trying to construct a number of possible endings, Splash and Ripple (the creators of Ghosts in the Garden) might have better used their time, and the interactive nature of the device, to offer visitors a choice of points-of-view on one single story. And if they had done so, would that have made the narrative stronger, and more emotionally compelling?

Holiday Reamde

Last week, for my holiday in Cornwall, I took some “hard” reading with me, but I was determined to have some holiday reading too. Having mentioned Neal Stephenson in a previous post, I was reminded that I hadn’t ever picked up one of his more recent books, Reamde. Shopping around, it was pretty cheap on Kindle so I downloaded it, and took it with me.

I wasn’t expecting it to immerse me back in the world of games and cultural heritage, in fact, I was hoping to be taken on some flight of scientific fantasy. But as Mick Jagger once sang “you don’t always get what you want…”

IF THERE WERE going to be K’Sheteriae and Dwinn, and if Skeletor and Don Donald and their acolytes were going to clog the publishing industry’s distribution channels with works of fiction detailing their historical exploits going back thousands of years, then it was necessary for those two races to be distinct in what archaeologists would call their material culture: their clothing, architecture, decorative arts, and so on. Accordingly Corporation 9592 had hired artists and architects and musicians and costume designers to create those material cultures consistent with the “bible” of T’Rain as laid down by Skeletor and Don Donald.

 Reamde page 46

Reamde follows the adventures around one Richard Forthrast, co-founder of a company that produces a wildly successful MMORPG called T’Rain. The game is based on (and portrayed as a competitor to) World of Warcraft but the attention to detail in material culture is reminiscent of Skyrim, which has of course inspired more than one “ludic archaeologist“.

I got quite excited as the opening chapters progressed. The last Stephenson book I read, Anathem taught me a lot about mathatics and quantum theory, and I thought he might blow my mind about game design too. Sadly (though entertainingly) the novel became an extended transcontinental shootout involving the various members of  Forthrast family, a couple of chinese teenagers, a Hungarian hacker, a Russian “security consultant”, a British MI6 agent and a Welsh muslim terrorist.

The references to the game are quite fun and experimental though. They do suggest that the author is a narrativist:

Because Corporation 9592, at bottom, didn’t make anything in the way that a steel mill did. And it didn’t even really sell anything in the sense that, say, Amazon.com did. It just extracted cash flow from the players’ desire to own virtual goods that could confer status on their fictional characters as they ran around T’Rain acting out greater or lesser parts in a story. And they all suspected, though they couldn’t really prove, that a good story was as foundational to that business as, say, a blast furnace was to a steel mill.

Reamde page 209

Which is why this fictional company has a department called Narrative Dynamics. But his leading character does think ludically too: the novel recounts how they come up with the idea that the core “Medieval Armed Combat” mechanic could be used to help with monotonous real-world jobs. This is like an idea my wife had mentioned a couple of years back. The example in the book was airport security, but it made me laugh when I saw the story about Fraxinus.

The other thing that I liked about T’Rain (and something that I miss in Skyrim) was the vassal system – players were not simply lone adventurers, but could recruit (or be recruited into) a gang, warband, household or army, in something like a pyramid selling scheme, all of which feels like a more realistic medieval style world than one in which everyone is equal. The novel recounts how this eventually divides the players into two factions, not the artificial Good and Evil factions invented by the games creators, but the Forces of Brightness (Manga inspired players who dressed their characters in lurid colours) and the Earthtone Coalition (more eurocentric gamers who enjoyed more Tolkienesque fantasy). These two factions of course starting to produce material cultures that built on the created archaeology of the world, but which were something entirely new and unplanned.

A fun read, even if not quite the escape I was hoping for.

A Young Lady’s Illustrated Primer

There are a lot of things in Neal Stephenson‘s The Diamond Age which I love. If I’m honest with myself I hope to see mediatronic paper and animated digital chops, for example,  become real in my lifetime. There are other aspect of the world created in that novel, for example massive inequality in a post-scarcity society, which I hope we won’t see, but I fear we are already walking down the path towards. At the core of the book though is one idea that some of my recent reading has prompted me to think about again.

The 2009 paper, Serious Games in Cultural Heritage, by Anderson et al., is a fun read, reporting on the state of the art at the time. There are some lovely lines which I’d like to take issue with. The authors, for example, hint at an opinion that a serious game doesn’t need to be fun. To which my reply that if its not fun, then its all “serious” and not a “game,” even if it does make use of gaming technology. The authors cite two examples of virtual reconstructions of Roman life, Rome Reborn and Ancient Pompeii, which use gaming technology as a research tool: “[Rome Reborn] aims to develop a researchers’ toolkit for allowing archeologists to test past a current hypotheses surrounding architecture, crowd behavior, social interactions, topography, and urban planning and development.”  More fun comes from the Virtual Egyptian Temple, and The Ancient Olympic Games examples which have playful or ludic elements in them, even its its only piecing pots back together or successfully answering quizzes set by what the paper calls a “pedagogical agent.” (Crikey! I’m returning to the Ludology vs Narratology debate again – on the side of the Ludologists!)

The paper also discusses the pedalogical value of some commercial games, which Burton calls “documentary games.” The most recent example of this genre brought to my attention is Call of Juarez: Gunslinger (with thanks to Chad at westernreboot). Of course another feature of many modern commercial games that the paper highlights is the bundled content creation tools that allow you to create your own cultural heritage environment, and indeed the Virtual Egyptian Temple mentioned above was built with the Unreal Engine toolset.

There’s also a section on all the various “realities” that gaming technology has to offer, which I’ll return to when I finally get round to writing up Pine and Korn’s Infinite Possibilities. and a section on the various gaming technologies (rendering effects and artificial intelligence) and the like, which a cultural heritage modeler can use, which makes the paper a very good primer on the subject (and one I wish I’d found earlier).

What led me to that paper was looking deeper at one of the poster presentations I saw last week. I didn’t get a chance to talk to (I guess) Joao Neto who was deep in a conversation I didn’t want to interrupt, so I did some Googling. Part of a team working to interpret Monserrate Palace in Sintra, Portugal, Joao and Maria Neto did some of the usual stuff: creating a 3D model from architectural drawings and laser scanning to show how the palace developed over time; an interactive application called The Lords of Monserrate, exploring the lives of the different owners of the palace over the centuries; and The Restoration, which appears to be a mobile app which recognizes the distinctive plasterwork in each room and interprets the restoration process in that room. But they also experimented with what they called Embodied Conversational Agents.

These are virtual historical characters, “equipped with the complete vital informational [sic] of a heritage site.” The idea was that the virtual character would capture the visitor’s interest with a non-interactive animated opening scene, in the manner of a cut-scene on a video game, but then would open up a real time conversation that would immerse the visitor with realistic “face movements, full-body animations and complex human emotions.”  The conversation would be more sophisticated than a simple question and answer system, by being “context aware,” breaking up the knowledge base into modules, to make interactive responses more possible.

In order to achieve this ambition, we developed an Embodied Conversational Agent Framework – ECA Framework. This framework allows the creation, configuration and usage of virtual agents throughout various kinds of multimedia applications. Based on a spoken dialogue system, an Automatic Speech Recognition (ASR), Text-to-Speech (TTS) engines, a Language Interpretation, VHML Processing, Question & Answer and Behavior modules are used. These essential features have very different roles in the global virtual agent framework procedure, but they all work together to accomplish realistic facial and body animations, as well as complex behavior and disposition.

Which all sounds like an amazing feat,even if the end result is (and I’m sure it must be) a little bit clunky. I’d love to see it in action. But what does this have to do with Neal Stephenson and The Diamond Age? Well, the subtitle of that book and the McGuffin (though plot wise, it’s much more than a McGuffin)  is A Young Lady’s Illustrated Primer. In the story,  A Young Lady’s Illustrated Primer is an interactive book, a pedagogic tool commissioned by a very wealthy nobleman to ensure that his daughter’s educational development is superior to her peers. Many of the characters that the reader meets in the Primer are sophisticated virtual agents like those described by Neto and Neto. But some are voiced by a “ractor,” an interactive actor whose voice, expressions and movements are transmitted live to become the voice, expressions and movements of the character in the Primer. One of the characters in Stephenson’s novel make her living as a ractor, playing characters like Kate “in the ractive version of Taming of the Shrew (which was a a butcherous kludge, but popular with a certain sort of male user),” and to “fill in the blanks when things got slow, she also had standing bids, under another name, for easier work: mostly narration jobs, plus anything having to do with children’s media.”

I used to be a “ractor” of sorts, as a costumed interpreter in all sorts of historic sites. I’m proud that my colleagues and I became one of the most interactive and immersive of all the interpretation media available. But having professional people on site is expensive, and not all volunteers have the skills, confidence or desire to take on historical roles. So I’m wondering if another approach to Neto and Neto’s Embedded Conversational Agents is now, technically a possibility.

Could a virtual character be distantly controlled in real time by a human “ractor”? And could that ractor fill their working day becoming different characters (and even at different cultural heritage sites) as and when required? The relatively small audience for cultural heritage after all makes a live ractor experiment a more realistic possibility than it would be for a popular commercial video game.

I REALLY want to try this out. Who wants to help me?

Twitter is your friend

I note that one of the most popular searches driving traffic to this blog is “narratology vs ludology.” I must admit, I’m not entirely sure why. I’ve written only one post addressing that debate, and over all, I guess I’m taking quite a narratological point of view. This post however may begin to address the balance, as this is where I begin to get all “ludological.”

When I wrote my funding proposal, I predicted that I’d struggle to find much literature around narrative in games. I haven’t found much so far. I suppose I should not be surprised, all the people who know about games are rightly making games rather than writing about how to make games.

However, a couple of days back, just as I was packing up for the evening, and shutting down Tweetdeck,I glimpsed an interesting looking item:

Great article on how most of the of a game’s storytelling happen in our minds: http://t.co/NYxZ6nIwrs

— Thomas Grip (@ThomasGrip) June 6, 2013

I followed the link and had a quick look at the article, which was intriguing, but I had plans for the evening. So I retweeted it in lieu of making a note and shut down my computer.

When I came back to it the next day, I read the article. The author Tynan Sylvester had worked on Bioshock, which was interesting because I’d recently read a paper on the use of music in that game, and also had my niece’s boyfriend recommend it as a “must play”. The article is about simulation and emergent story, and Sylvester related how the stories in Bioshock had been intended to come out of a complex simulated ecology, however:

While BioShock retained some valuable vestiges of its simulation-heavy beginnings, the game as released was really a heavily-scripted authored story. There was no systemic ecology at all. It worked fantastically as a game – but it wasn’t a deep simulation.

Attempts to create realistic models in games are misguided, he says, because:

What we really want is not a system that is complex, but a system that is story–rich. […] Interestingly, real life and most fictional worlds are not story-rich! Most days for most people on Earth or in Middle Earth are quite mundane. It’s only very rarely that someone has to drop the Ring into Mount Doom. Follow a random hobbit in Hobbiton, and you’ll be bored soon.

He goes on to point out that whatever the model in the computer program, “The whole value of a game is in the mental model of itself it projects into the player’s mind” [his emphasis]. He calls this the Player Model Principle. He goes on to talk about apophenia, the human mind’s tendency to project human patterns and behaviors onto non-sentient objects (and in this case, computer animations). Using an example from the Sims, he shows how a story of love, jealousy and murder can be imagined out of a couple of variables in computer code interacting. He discusses how to encourage apophenia in the player, and concludes that modelling can create successful and compelling narratives as long as the designer remembers to “Choose the minimum representation that supports the kinds of stories you want to generate.” Which is to say, keep the complexity of the model as simple as you can get away with, adding complexity for the sake of realism only creates noise.

Which is all very interesting, even if its relevance to those in my field, cultural heritage interpretation, is mostly a useful reminder  not to over complicate things. Sylvester writes well, and explains complex ideas in very understandable ways. So I was particularly interested to see that he’s recently published a book called Designing Games, a Guide to Engineering Experiences. Could this be, I wondered, the elusive literature on designing narrative in games that I’d been looking for?

YES IT BLOODY CAN!

I downloaded a preview, and the first page set out Sylvester’s thesis, in the bold title of the first part (and then the first chapter) of the book “Engines of Experience.”

These are the droids you’re looking for.

I devoured that preview and wasn’t disappointed. I bought the full e-book (direct from the publishers). This is exactly the sort of book I envisaged finding when I wrote that funding proposal last year – not a guide to 3D modelling or programming games, but rather a games designer explaining (as he says) “the trade-offs in every design decision.”

And what gets me, is that I didn’t find it in a literature search, slogging away on Google, library catalogues or trawling though endnotes. It came to me on Twitter. I don’t know Thomas Grip, who posted that original tweet. I can’t even recall why I started following him. But thank you, Thomas, for posting that link.

And what if I had turned off five minutes earlier? Or ignored that tweet in my hurry to shut down? Would I have found this brilliant, helpful book at all? I hope so, but this has been a massive shortcut. I can see why my supervisors were so keen when I started my studies that I should up my Social game. Twitter is truly your friend.

But so is Google, and so for all those to find their way to this blog searching for ludology vs. narratology, let me quote Sylvester’s take on that debate.

This fiction-mechanics conflict is why some see a great debate between mechanics and fiction. The ludologists (from the Latin ludus, for “play”) argue that games draw their most important properties from mechanical systems and interactions. The narratologists argue that the mechanics are just a framework on which to hang the fictional elements players actually care about. This debate is the game designer’s nature versus nurture, our plot versus character, our individualism versus collectivism. But like all such debates, the conflict exists only on the surface. The pinnacle of game design is combining perfect mechanics and compelling fiction into one seamless system of meaning. Fiction and mechanics need not fight (though they easily can), and neither one need be given primacy (though one often is). Used together, they can enhance and extend each other in ways that each can not do alone.

Sylvester, T. Designing Games, O’Reilly Media, 2013-01-03. ePub.

I’ve got a suspicion you’ll be seeing a few more posts from me about this book.