Decoding the Digital Part 2

In my last post, I wrote  about the presentations that made me think the most, but all the speakers at Decoding the Digital were great to listen to. It was a wide ranging and eclectic mix of digital humanities.

There was good contingent from the University’s own humanities school, including Joel Burgess who doesn’t like the phrase digital humanities any more because everything in the humanities is becoming digital. By way of example, he showed us a digital analysis of TV. Where, once upon a time, people would have to sit with stopwatches  to measure, say, the length of each shot in a TV program, now the same program can be loaded into a digital editing software package, which will identify the cuts and time the whole thing pretty much automatically. This is the technique they used to see if the common perception that TV has been getting steadily faster, and audience attention spans shorter, is true. So far, on a pretty small sample size admittedly, that’s not the case.

Morris Eaves, who had a hand in editing one of my favorite books, an illuminated collection of the works of William Blake, talked about digital collaboration, with collections (such as Blake manuscripts) and expertise that are spread around the globe. Joan Saab talked about writing books on-line (in Scalar, which looks interesting) and shared some of her work an local architect Claude Brugden. She is thing about how to tell his story in a 3D model of his work. She could be one for me to follow, or collaborate with…

(A presentation from Kathleen Fitzpatrick the following day, continued the theme of collaborative writing using digital tools, where blogs become eternal drafts, and modern students don’t expect to have to review their work.)

Cary Peppermint is an environmental artist with a little bit of the situationist or psycho-geographer about him. One of his works Indeterminate Hikes, sounds a lot like an app I blogged about a while ago called Serendipitor. I caught him after his talk at a reception and asked him if he’d heard about it. He said he’d worked with Mark Shepard, the creator of that app, and looking at their two websites, you can see a lot of similarity in their interests.

The conference kicked off with a keynote from Thomas Gewecke, Chief Digital Office and Executive VP, Strategy and Business Development, Warner Bros. He talked about Embracing Disruption. An entertaining and charismatic speaker, he talked about the lesson learned from Napster and iTunes in how to deal with piracy, which was essentially, make stuff available to down-load legally as soon, and as conveniently as possible. He and Warner Bros have been instrumental in creating the Ultraviolet standard, to that end. He also talked about crowd-sourcing the funding for the Veronica Mars movie through Kickstarter. There was some controversy about this at the time. People complained that a major studio should put up their own money for such a project and leave Kickstarter to the struggling “little people.”  Personally I think if a studio offered me the chance to pre-order anther series of Firefly, I’d be there, and I doubt the Veronica Mars movie would have got past the bean counters at the studio if they hadn’t shown that a lot of fans had put so much money down. I never saw the TV series, but the extended clip-reel Gewecke showed us makes the movie look pretty good.

Todd Havens, a Rochester graduate, continued the Hollywood thread by sharing some of the secrets of the social marketing machine that now surrounds movies. He is rightlly proud this successful viral campaign for the Last Exorcism:

The Last Exorcism – Chatroulette Viral Video by dreadcentral

The last day started with an interesting presentation from Robert Markley. He told us about a digital analysis of maps of the great lakes that he’s been working on, which has suggested that there was  a massive change in water levels in the early 19th century. For about twenty years, maps show what we think of as one long island as a chain of smaller islands instead. According to Markley, people who know the area believe that the change in water level must be massive to make the shape of the island so much. All the work so far having been digital, a trip is planned to survey the island next year. I noticed he’s writing a book Kim Stanley Robinson, so we spent the following lunchtime sharing our mutual love of The Years of Rice and Salt. Later in the day Henry Kautz explained an experiment that mined twitter to map public heath in New York.

There was also a strong celebratory theme running across the conference in the choice of speakers to end each day. Graduates of the university’s Eastman School of Music, all of whom are now veterans of the music industry, joined us to share not so much academic research, but some great reminiscences. Thomas Mowrey, Bob Ludwig and Don Pulse are names you’ll find in the production credits of some of your favourite albums. I could have listened to them all night. In fact on the last day, I got to sit next to them and Todd Havens for dinner, so in a way I got my wish. There was also a great explanation of the Loudness Wars from David Temperley, and by skype, Jeff Beal,(composer one of favourite TV soundtracks: Rome) who talked about composing for Netflix’s recent version of House of Cards.

I’m sure I’ve missed one or two speakers out, but suffice to say, it was a full and very enjoyable two and a half days. I feel honoured to have spoken on the same platform.

Decoding the Digital Part 1: The Playful Aristocracy

Two weeks have passed since Decoding the Digital at Rochester University, and if I don’t write it up now I’ll have forgotten what my notes mean.

A small aside – I write terrible notes. I always have. I find that I can either listen or write notes, never both. If I try and write notes, then i realise that I’ve be concentrating so much on putting the words down and I’ve totally lost track of the lesson, lecture or presentation. That was especially true of Physics at school. I passed my a-level on just eleven pages of notes for a whole two year course. So in the end I trained myself to just listen at University, and forego notes. That worked back then when my mind was agile, but now I’m older I find I’m not quite so good at remembering stuff. So I’ve taken to writing singles words down in the hope that it will trigger the memory of something I felt was important. I fancy getting one of the new iPads, and that programme that records while you are note taking, so when you tap on a word, it plays back the recording of what was actually being said at the time.

This though is a combination of written notes (from the handy-dandy note-pad and pen that was in our welcome packs), a couple of notes typed on my phone (I’ve taken to tweeting as a form of notetaking but the University of Rochester wifi wouldn’t play with my phone for some reason), and my own imperfect memory.

The meat of the conference, for me, started with the second presentation, from Australian New Yorker, McKenzie Wark. He told us he’s been on “Facebook” since the early nineties. Of course Facebook itself  didn’t exist back then, but the early social internet instead occurred on newservers and listservers. Listserve was an email sharing protocol whereby people shared an email diatribe with a community of subscribers. I used it a lot when I was doing my MA, taking a text to the pub then ranting about it (sorry, I mean critically appraising it) when I got home in the middle of the night. Now of course I’d do the same thing on a blog, however, as a family man, I go to the pub to read texts less often, and drink less, so my posts are somewhat more sober.

Enough about me though, Wark described his time on the listserve NetTime as the Silver age of Social Media, and spoke with a great deal of nostalgia about the proto-utopian, post-perestroikan days of the early internet. His presentation felt like a piece of digital archaeology. The community of NetTime, he said, were explorers of the digital frontier, creating a new culture and new digital artifacts without the limits of real world real-politik (other than the disdain that post-communist eastern European NetTimers had for their western peers’ enthusiasm for a Marxist internet). Some of the people from the virtual commune of NetTime went on to carve careers out of the digital age, but Wark wondered where all the others went. He lamented the freedoms of at early experimental time, commenting that the internet had been enclosed by corporate interests, and the likes of Apple’s iPad. I’d counter that more people have probably jailbroken iOS devices than had even had access to the internet in the 90s, but Wark said in questions afterwards that the same spirit of invention and creativity now only exists in play, in the gamer culture of the twentyfirst century.

Which made me think that all those NetTimers of the nineties were playing, really. Playing at politics, playing around with code, and trying on different identities. And maybe those names who’d disappear from NetTime without becoming academics or movers and shakers, probably only did so, because in the real world, family and work and responsibility had curtailed their playtime.

The social media of today takes a lot less effort than back then when people had to book time on university servers, and wait for the 1200 baud dial-up modems to whistle and whir through their digital handshakes. Back then, to play on the internet took effort and time. NetTime was therefore a privilege, not a universal right. And those who managed to carry on playing, like Wark were (or are?) a playful aristocracy, not the Marxists they (at least the western ones) thought they were.

Once I had the idea of a Playful Aristocracy in my head, it  kept coming up in later presentations. Sarah Higley, Professor of English at Rochester (and creator of Star Trek: TNG’s popular re-occurring guest character Reg Barclay), co-presented, with her Second Life avatar, Hypatia Pickens, a short history of Machinima, the art of creating movies out of virtual worlds like second life and mod-able games. Second Life is another example of how people with time to play, to devote to crafting can build themselves a not just a virtual pad, but a real-world reputation. I watch my kids build intricate Minecraft homes, and wish I had the time to do the same.

(Who am I kidding – I make time to play pen and paper RPGs. I’m not entirely sure I’d choose Second Life to spend my playtime in, I had a look at it once, but decided it would take too much of my time to even look good, that I couldn’t be bothered to continue.)

Aubrey Anable returned to Second Life the following day, this time with a slightly archaeological gaze. After suddenly becoming a media darling and the subsequent massive surge in popularity, the reported population of Second Life has now shrunk to about 30,000 to 60,000 active avatars. Given that users might have more than one identity in Second Life (Higley had previously mentioned a number of avatars that she used), that suggests the number of active users is even smaller. The digital homes, landscapes and other creations that laspsed users created still remain though, empty worlds that few visit any more. The active parts of second life have also been segregated into “adult” areas where participants gather to share alternative sexualities, and the somewhat less popular “clean” areas. Anable introduced us to two artists who explore the ruins of this virtual world, creating machinima of two very different styles. Cao Fei, or China Tracy as her Second Life avatar is know, produced a haunting documentary of the wastelands left by the Second Life property boom, called i.Mirror.

Jon Rafman, on the other hand, offers tours of Secondlife with his avatar, Kool-Aid Man. The sight of this American advertising icon in the silent discarded worlds and even the busy eternal nightclubs of the adult areas is strangely compelling.

Though it feels dilapidated, Second Life has, it seems, shrugged off the the tourists and hangers on, the newbs and time wasters. and those 60,000 avatars belong to a community of dedicated players, like Cao Fei, Rafman, and Higley, another playful aristocracy, who have the space (and time) to create new virtual artforms. Is it sustainable with such a small active population? We’ll have to wait and see. Its a commercial enterprise after all, and the company behind it could turn off their servers at any time.

But who are those who won’t or can’t dedicate their time to creatively play? Lu Wei of Zhejiang University shared some research into the digital divide. Once, he said, the digital divide was easy to define. On one side, a minority had access to the the internet, and on the other, the majority didn’t. But now that so many people, even (as one slide portrayed) peasant farmers in China, have access to the internet through their phones, does the digital divide still exist? He ask the audience who used the internet for email. We all put our hands up. Who used the internet for news? Again, we all put up our hands. Then he asked how many of use played games on the internet. Only half of us put our hands up. Lu Wei had been looking at what people used the internet for. And using date from a US survey in 2008, he showed that of the 11 activities listed in the survey, only a few were used by the majority of respondents. He demonstrated that the people who used the internet for more things tended to be better educated, wealthier etc. So he suggested, the digital divide remains. And it is somewhere around four activities, if you do more, he suggests, you are likely to be empowered by the internet, for example, by actively participating in local politics. If you do four or less, you are likely to be more of a consumer. Now, this data was from 2008, and although the iPhone was released the previous year, the explosion in smartphones mobile internet hadn’t quite happened. And as one of the audience pointed out, the eleven activities in the survey didn’t include playing games, or (possibly more notably) watching porn, so the data could do with updating, but the theory is a useful one, which has some resonance.

It made me think of a counterpart to the Playful Aristocracy which I’ve taken calling the Leviathan, the unawakened digital majority who now have access to the internet, but no time to play with it. Lisa Nakamura and Sara Perry offered two insights into darker side of the Leviathan. In her presentation Blowing up the Digital Humanities, Nakamura touched on identity and race in the digital domain, and how easy it is, and how willing some are, to push people of colour into “the other” on the internet. Perry, meanwhile, explored the transition between the pseudonymous internet (in which Higley, among others could build a playful reputation with her Hypatia Pickens avatar, and post-Facebook “real name” internet, which we have to use now, to build our real-world reputations on Linked-in, etc. Its an uncomfortable period for many of us used to the pseudonymous norms of yesteryear. Back then gender played a different role and, as in Second Life, one could present on-line as a different gender if one so wished. So gender and race, it seemed through rose tinted nostalgia-goggles at least, was less of an issue than now. Now we have awful threats directed towards a woman who simply campaigned for a notable female figure on a bank note, and a respected academic castigated for her looks while appearing on television debating programme,

I don’t want to tar the whole Leviathan with the same brush, but it seems there is a contingent there who, possibly feeling dis-empowered by the those who wield an on-line reputation of any sort, seek to drive their targets off the internet, though constant anonymous abuse. It feels as though we are approaching a fork in the road. Will the Leviathan learn how to engage socially and politically for the good of society? Or will the rule of the loudest drown out the voices of reason, and the internet descend into an angry dystopia?

Check out Sara’s Gender and Digital Culture project here.

Woah! This is in danger of turning into a dissertation, and I’ve covered less than half the speakers. So I better wrap up for the day, and cover the rest in another post,

Presence

Plymouth Plantation from Boston.com (David L. Ryan/Globe Staff/file)

In my paper last week, struggling to find a way to describe the environmental “sense (or spirit) of place” that may drive emotional engagement in (games and) cultural heritage environments, I chose to use the word “Presence.”  I first came across in Pinchbeck’s writing but I was nervous about using it, until, by coincidence, Erik Champion also used the word when he commented on this post.

There are two reasons why I hadn’t used it before, and why I’m still unsure about it: Firstly, presence is a term used to talk about virtual environments (like games) but I’m looking to apply what I learn to real environments (cultural heritage). Secondly, I worry (as I mentioned in my presentation) that presence may encompass all (or most) of what I’m currently calling “emotional drivers,” rather than being one driver.

But there needed to be a word there, so i flippantly settled on presence, and promised myself I’d investigate it later,

Later starts now, and Erik handily left me some links in a comment on this post. I’ve only been able to read his own co-authored “Evaluating presence in cultural heritage projects” (2011Pujol, P.  & Champion, E.) so far the other two come from a journal Southampton doesn’t subscribe to, so I’m hoping I can fix an inter-library loan. However, this paper works quite well as a primer on what Erik calls “Cultural Presence.”

Lets kick off with how they start their paper with a little bit of  history:

Presence originates from the term ‘telepresence’, made famous by the computer scientist Marvin Minksy in a 1980 paper of the same name (Minsky 1980). From around 1991 (the date of the first issue of the MIT journal Presence: Teleoperators and Virtual Environments), presence has been typically defined as the capacity of the technology to make the user feel transported into a remote place and be able to efficiently interact with it.

Of course, to be honest, it looks like it started out as, as my science adviser used to say in Civilization, “a mainly military technology,” designed to help drone pilots in Colorado feel transported to remote mountains on the Afgan/Pakistan border and “efficiently interact” with jihadi hideouts. But it soon found civilian uses.

Archaeologists have long been using computer modelling to help visualise sites, “rebuilding” what might today be ruins, or virtually “restoring” buildings that have seen centuries of modification. The primary purpose of such models has been experimental, helping the researcher compare hypotheses. Of course once these models have been made, people want to share them, with each other and with the public. Sometime they are shared incredibly badly, even now. One of the presenters at the York Digital Heritage conference seemed surprised that a computer model that was a outcome of a research project was, when displayed in a shop window, totally ignored by the passers-by. But when its done well, its done with the intention of transporting the audience to a particular place and time, and that’s where presence comes in. As Pujol and Champion say:

Presence is typically seen in academic research as the aim of virtual reality environments. Since ‘virtual heritage’ is the name VR applications are given when used for the dissemination of cultural heritage, it logically follows that in VR applied to cultural heritage, a meaningful sense of presence is also the intended outcome.

However they follow that up with a warning that cultural heritage interpretation isn’t just about how buildings look. They argue that to thing of virtual heritage as simply the re-creation of buildings or other “tangible” artifacts in the digital domain is to ignore the importance of  human interaction, ritual, communication, symbolism and representation and all the other intangibles that are part of culture. They also quote Stone and Ojika (2000, Virtual heritage: what next?. IEEE Multimedia, 7 (2), 73–74):

[Virtual heritage is] . . . the use of computer-based interactive technologies to record, preserve, or recreate artefacts, sites and actors of historic, artistic, religious, of cultural significance and to deliver the results openly to a global audience in such a way as to provide formative educational experiences through electronic manipulations of time and space.

But here is where I begin to worry about the idea of presence. Because I’m not convinced that “the use of computer-based interactive technologies … to provide formative educational experiences through electronic manipulations of time and space” needs to be about immersive virtualisation.

You see, I very much enjoy the model of “Infinite Possibility” set out bu Pine and Korn (2011) in the book of the same name. Their claim is that Digital Technology offers so much more than Virtual (or even Augmented) Reality. We should, they argue, be thinking in terms of all the possible combinations of the variables of the Reality (Time, Space and Matter) and the equivalent variables of the digital Virtuality (for want of better words: no-time, no-space and no-matter). The diagesis of a computer game, or VR model, is a function of these three no-variables, as it is made of bits of computer code. But run that virtual world through a pair of VR goggles, or even a humble Tom-Tom navigator, and you suddenly have Augmented reality (Time and space, but no-matter). Conversely, use a wii-controller to  are intmanipluate a computer game and suddenly you find yourself in the realm of no-time and no-space, but with matter – what Pine and Korn call Augmented Virtuality.

Superimpose a different time (or no-time) on  space and matter, and you have what they call warped reality:

Such reality-based time travel happens whenever experiences simulate another time… such as Renaissance Fairs and living history museums (Plimouth Plantation, Colonial Williamsburg, and the like) or transport us … even into the future (albeit a fictional future) such as at, yes, Star Trek conventions.

In these warped realities, a cultural heritage audience is able to participate in the construction of realities that capture objects and processes of scientific, social or spiritual value [and presents them] as accurately, authentically, and engagingly as possible. Places like Plimouth Plantation share their work  in a sensitive, safe and durable manner to as wide and long-term an audience as possible, to provide an effective and inspirational learning environment that best communicates the intended pedagogical aims. Every italicised statement here actually comes from Pujol and Champion’s summary of what every good Virtual Heritage project should be. And yet Plimouth Plantation and its like a currently entirely analogue creations, and that no-one would consider applying the word “virtual” to. I think, and Pine and Korn imply, that digital technology has the potential to greatly enhance warped reality experiences, without making them virtual. I’d also argue that cultural “presence” occurs in these warped reality spaces, and yet … and yet, can it only apply to Virtual worlds?

Handily, Pujol and Champion have a crack at unpicking the definitions of presence for me. Starting with the idea that the ideal is a sense of being there, or blanking out the digital mediation of screen and controller, they touch upon immersion as a product of field of view and optical resolution. They also briefly summarize the idea that the human component of the system is likely to respond to the affordancies offered by the VR according to their interests, if the virtual component can in turn respond in a realistic way. They touch upon co-presence (sharing the VR with other users) before arguing that social presence (interacting with other users and virtual agents) is vitally important idea in the “potential [their emphasis] convergence between the presence and cultural heritage fields.”

So already we can see that the concept of presence is a very complex and even ambiguous construct: there are several definitions based on disparate theories, which focus on specific aspects or give different names to the same concept, partially overlapping or even contradicting each other. Therefore, the conventional notion of presence as the sensation of ‘being there’ is a highly simplified way of expressing an internal perception of the environment and of ourselves in relation to it. A more comprehensive explanation would be that the sense of presence results from the interaction of various factors. These factors depend both on the system (immersivity, visual accuracy, real-time physical and social interactivity, invisibility of devices, consistency of the content) and on the participant (perception, attention, empathy, engagement, meaningfulness or relevance of the content, control, suspension of disbelief).

They go on to try and define “cultural presence,” citing and earlier work by Champion (2005, Cultural presence. In: S. Dasgupta, ed. Encyclopedia of virtual communities ad technologies) to suggest that “cultural presence corresponds to the feeling that people from a specific culture occupy or have occupied a virtual environment and transformed it into a culturally meaningful place.” This is something I recognise from what the National Trust tries to do in the places it looks after. But, they say, such “environments represent a palimpsest in which past social interactions are layered and carved into the fabric of the environment. Although visitors can see ‘culture’, they cannot participate in it, either due to a lack of culturally constrained creative understanding or because the originators have long since passed away.”

So, is the previously mentioned “social presence” the key? Possibly not. Pujol and Champion briefly look at chatrooms, virtual communities and video games. The social interaction in chatrooms is fleeting and non-permanent. Virtual communities on the other hand, “do establish rules and elements of identity; nonetheless, their limited virtuality and transient ubiquity ironically prevents them from owning a sense of cultural place, where identity is expressed and recognized through dynamic processes that are materially situated.” (Hmmm Second Lifers may disagree – see my forthcoming report from Decoding the Digital). Of the three, they think, games have the most potential for social presence, because of their interactivity and exploration, virtual agents and (sometimes) co-operative play. Game mechanics though, can sometimes get in the way of cultural learning.

They summarise their discussion of cultural presence with the following:

So cultural presence in the cultural heritage field is not limited to the reconstruction of a place; ideally it would also encourage empathy, interaction and collaboration to enhance awareness and understanding of past or foreign cultures. So for cultural presence, ‘presence’ is the means and ‘culture’ is the goal. Unlike the test environments of typical presence research, virtual heritage projects should not aim at the fidelity of representation of the world in general, but towards a cultural context, containing not only objects and active agents but also the inter-relationship of their situated beliefs and values. Hence, presence becomes a ‘being – not only physically but also socially, culturally – there and then’.

Which is interesting, because in that whole paragraph virtual reality isn’t once mentioned. Is it taken as read? Or is it not required?

The Strong, National Museum of Play

Before last week’s Decoding the Digital conference, I visited the National Museum of play at the Strong. There are a number of Strong endowed institutions in Rochester, including the university Hospital, but unlike the city’s other famous sons and benefactors, George Eastman of Kodak fame, and the Xerox corporation, none of the locals seemed to know who the Strong family was or how they made their money. In contrast to the technological marvels of photography and xerography, the  Margaret Woodbury Strong  fortune, it turns out, came from the manufacture of whips! Buggy-whips to be precise. Margaret’s family wisely invested some of their whip money in Kodak too, so their fortune grew with that company. She was a keen collector of dolls, who donated her collection, and an endowment, to create the museum.

Margaret Strong’s collection and money endowed the museum

The Strong actually now contains five institutions (or as they call them, play partners): the National Museum of Play; the National Toy Hall of Fame; the International Center for the History of Electronic Games; the Brian Sutton-Smith Library and Archives of Play; and, the American Journal of Play.

Downstairs the museum mostly resembles the Discovery Centres that grew around the world after (if I recall correctly) the creation of first in Boston: all sorts of opportunities for children from birth to 12(ish) to learn through play, and some handy interpretive panels for parents and other grown-ups to explain just how brilliant learning through play is.

A panel about play

These spaces included a recreation of Sesame Street, but my favorite gallery was the one inspired by comic books, which explored the history of superhero comics, told the stories of some of the classic super-heroes, and offered play experiences based on the theme. The super hero gallery also displayed some of the museum’s collection of classic comic books.

Part of the comic heroes gallery

I actually remember buying and reading this issue of Iron Man

But most of the collection-based displays were upstairs, away from the ludo-didactic spaces on the ground floor. The Toy all of Fame  has been selecting toys annually since 1998. All the toys your expect are there: Lego, the Slinky, etc, but some of my favourite exhibits celebrate the play potential of the such brilliant inventions as “the stick” and “the card-board box.”

The Stick – in the Toy Hall of Fame (with marshmallows in a supporting role)

There’s a big gallery of games, exploring the history of boardgames of all sorts, and including table-top role-playing games like Dungeons and Dragons too. Its funny to see the D&D Basic Set which I remember saving up for, excitedly buying, then being perplexed by, behind the glass of a display cabinet.

Part of the Role-Playing Games display

But I must admit I spend most time in the video games exhibit, which was not just a collection of vitrines containing historical items from the very first computer game console, SpaceWar and the “brown box” that was the prototype for Pong all the way up to the iPhone (or rather, not an actual iPhone but a label asking “How will the iPhone change video games?” which usefully dates the completion of this gallery to late 2007, early 2008). There was also plenty of video game arcade consoles and pin-ball machine to play, and a machine that sold you five tokens for a dollar.

So I happily played the brilliant Star Wars game, which I remember only ever seeing once on holiday in the US with my parents, in which you control your vector graphics X-wing to defeat a bunch of Tie-fighters and make a run on the Death Star tench to try and hit that infamous exhaust port. Back in Nevada when i was 11, I only had one chance to play that, and lost my last life in the trench. Finally the museum allowed me to have another go and this time, a little bit more experienced with playing video games, I managed to target the exhaust port on my second run and blow up the Death Star. Huzzah! I also managed to get on the high score table on Tempest, another vector graphics game I’d enjoyed in my youth.

Tempest, that’s me at number 4! (Of course they do get reset every day)

Margaret Strong’s own collection of dolls and toys (or at least those bits of it that didn’t appear in other themed galleries) was relegated to a rather dismal gallery about as far from the entrance as it was possible to be. But there was a lovely little space nearby exploring the cold war, entitled “When Barbie Married GI Joe.”

Those were the highlights, but there’s plenty more. Well worth a visit for anybody interested in play.

Oh, one more thing. Just inside the entrance, the museum cafe, is a recreated “diner.” My pre-visit googling had revealed that a local culinary “speciality” is the Garbage Plate (or, as the outlet that invented it rigorously protects its trademark, just Plate in most eateries). The diner at the museum offers one, and it was lunchtime, so I ordered a Plate consisting of “white hots”, saute potatoes, macaroni salad, onions and hot sauce.

A “plate” aka “Garbage Plate” is a Rochester speciality

My advice?

Don’t.

Just… don’t.

My Decoding the Digital presentation

I delivered my presentation at University of Rochester’s Decoding the Digital conference today, and it seemed pretty well received. So I present the text in full below, and you can see the accompanying Prezi here. At some point I’ll see if I can do something to add a synced recording to the Prezi, but for now, I’ll make it a quiz. Can you work out which frames apply top which bits of the talk?

Regular readers will find some parts of this familiar, which shouldn’t surprise, as this presentation is a synthesis of pretty much everything I’ve been thinking about so far. There was an interesting presentation today from Kathleen Fitzpatrick, during which she said that one result of “Digital” is that we academics should share more of our drafts, so I hope this blog does what she means.

There were lots of interesting presentations over the last three days, but I’ll have to blog about them at a later date.

The Interpretation Game: Research into digital narratives and cultural heritage interpretation

Matthew Tyler-Jones
Digital Economy USRG
University of Southampton

In his book, Designing Games, Tynan Sylvester says:

“If we look around, we find interactive narrative everywhere. Museums and art galleries are interactive nonlinear narratives where visitors explore a story or an art movement in a semi directed, personal way. Ancient Ruins and urban graffiti tell stories…

“These interactive forms – museums, galleries, real spaces, and life – should be [games designers’] first touchstones as we search for narrative tools. These older forms address our most fundamental challenge: creating a story that flexes and reshapes itself around the player’s choices, and deepens the meaning of everything the player does.”

Sylvester points game designers towards the heritage industry (and elsewhere) in an attempt to dissuade them from focusing on cinema as the sole source of narrative instruction. My own studies arise from wanting to look beyond linear storyforms (text, film) which exert a strong influence on museum and heritage interpretation designers looking to engage visitors’ emotions. Games do a great job of getting players to care about watching mathematical algorithms choose what colour each pixel on a screen is.

So, thank you (on behalf of my profession) for the kudos, Mr Sylvester, but I think the learning can be two-way.

To that end I’ve been looking, these past few months, at three digital narratives (or computer games) that each take a different approach to create so-called “open worlds,” three-dimensional virtual story spaces around which the player can wander with apparent freedom. All succeed in creating emotionally engaging (story-worlds or) diageses.

The first, Dear Esther, is described by its creator as an interactive ghost story. This was designed as an academic experiment in digital narrative, and has seen two incarnations, first as a “mod” or player-created content for Half Life 2, and subsequently as a standalone game, with a number of improvements.

The next, Red Dead Redemption is at the opposite end of the commercial scale, a big budget production from international production company Rock Star. It’s a western themed adventure that received a very positive critical and popular reaction.

Finally, I’ve been looking at is The Elder Scrolls V: Skyrim. An open world adventure with a traditional European fantasy theme. This offers a more dynamic narrative that either of the other two, and enables a greater degree of agency for the player.

The designers of these three games experimented with form as they explored story telling within each game’s virtual spaces. Cultural heritage institutions, including museums, built heritage, historic and ancient sites and heritage landscapes, have long been telling stories in three dimensions. Where it’s done well, visitors to those sites, and players of the best games, can make an emotional connection with the stories that they co-author as they make choices  about what to look at first and subsequently, and how deeply they want to explore individual points of interest.

So what drives emotional engagement in digital narratives, and what can cultural heritage institutions learn from them to improve interpretation?

I’m going to take a few minutes to run through the emotional drivers I’ve identified in the games I’ve played, and with each, explore equivalents in real-world cultural heritage.

Spectacle and sensation

When I’d just started playing Red Dead Redemption for this research, I spotted this tweet. (Prezi)

Hennigan’s Stead is one of the locations in Red Dead, so I followed the trail to this blog entry (Prezi), which demonstrates the power of visual spectacle to drive emotional engagement

The visual spectacle of all three games that I’ve been looking at is frequently lauded by players and in reviews.

Emotion through spectacle/beauty is something that my own organisation recognises. The full corporate name of the National Trust is, after all, “National Trust for Places of Historic Interest or Natural Beauty.” Respondents to surveys at the Trust’s most beautiful places, like Ightham Mote, for example, do report a higher emotional impact than elsewhere.

But does the National Trust as an organisation rely too much on the beauty inherent in our places? Good museums know how to manipulate spectacle, by creating “wow” moments, often at the threshold of galleries. These can involve impressive exhibits, multimedia “shows”, interpretive “set design” or even the design of the spaces itself, such as the Great Court at the British Museum.

Presence

But the virtual worlds that these games create are not simply beautiful. They are immersive, creating environments which themselves tell stories. The way the long grass sways as your avatar walks through it, the shadow of a bird that you notice a moment before you sight the bird itself, the changing weather, all add to your immersion in the diagesis. The architectural details, textures and ephemera of your surroundings all have the power, to inform the story.

Dan Pinchbeck, the academic and creator of Dear Esther calls this “presence,” and indeed, Dear Esther was created as an experiment in the manipulation of narrative and presence. Of course one might argue that Pinchbeck’s definition of presence thus includes all the emotional drivers that I’m discussing. But I’ll think about that and save it for a later paper.

Museums and archaeological sites often build reconstructed environments to encourage visitors’ sense of presence among the exhibits. And many historic houses are presented in such a way as to suggest the visitor has walked in on a room only recently vacated by the building’s historic inhabitants. Perhaps those that manage presence most successfully are the living history museums such as Plimouth Plantation

Society

There’s still an unfair perception of computer gamers as solitary types with no friends, but of course most games, not just team sports, are ways of bringing two or more people together. Whether it’s gathering around the screen in the amusement arcade, bedroom-coding or simply comparing high scores, computer games have always been social. But of course fast, cheap data communications have resulted in all sorts of social gaming from Words with Friends to World of Warcraft. So, how do my three games do?

In fact, only Red Dead Redemption has a multi-player mode, and that only after you download an expansion. Dear Esther’s isolation is part of the appeal of that game, and Skyrim offers stilted, repetitive, scripted conversations in place of interactions with real human beings.

So this is one area where, it seems, cultural heritage is way ahead of computer games. People use museums and heritage sites as social spaces, to spend quality time with friends and family, without any intervention from the cultural intuition itself. Guides and docents have long been a part of the heritage infrastructure, and part of the enjoyment of visiting a place like Plimouth Plantation is navigating the sometimes archaic language and social mores of the costumed interpreters.

Of course, people are an expensive asset for cultural organisations, and even volunteers are not free, so there are experiments inspired by gaming technology. A team working to interpret Monserrate Palace in Sintra, Portugal, have been experimenting with what they called Embodied Conversational Agents. The idea is that the virtual character would capture the visitor’s interest with a non-interactive animated opening scene, in the manner of a cut-scene on a video game, but then would open up a real time conversation that would immerse the visitor with realistic “face movements, full-body animations and complex human emotions.”  The conversation would be more sophisticated than a simple question and answer system, by being “context aware,” breaking up the knowledge base into modules, to make interactive responses more possible.

Acquisition

Computer games often simulate the acquisition of wealth, equipment, or simply points. High totals are rewarded with new abilities, or unlocked levels and new play experiences. However the emotional value of simply beating your best score, or getting to the top of the high scores table should not be ignored.

In Skyrim, you can pick up, buy or steal almost every object you see, and a new player’s character will quickly get weighed down with useless cheap tableware and other ephemera, before making more rational choices and building a useful supply of weapons, potions, spell books and apparel. Eventually your character will be able to acquire one or more houses in which to keep the stuff you accumulate.

In Red Dead Redemption, apart from the usual money and weapons, players can devote time to completing various quests which are rewarded with new outfits for their avatar to wear.

Of the games I’ve researched, only Dear Esther eschews acquisition as an emotional driver. But of course Dear Esther is all about loss.

Cultural Heritage sites often relegate acquisition to children’s trails. Many of which are “I spy” checklists, sometimes rewarded with a sticker to wear. The National Trusts “50 things to do before you are 11 ¾” campaign encourages families to acquire 50 fifty life experiences (like making a mud-slide or flying a kite).

I was also intrigued to recently discover this question (which measures an acquisitive impulse in visitors of all ages) among those evaluating a mobile guide app.

Challenge

As we’re talking about games, the most obvious emotion driver should be the ludic one (which is to say, the one all about play). Games test the player’s dexterity, pattern learning, and puzzle solving ability before rewarding him or her, not just with a sense of accomplishment, but progression within the game. Both Red Dead Redemption and Skyrim use the traditional (to video games) challenge of having to kill people to get where you need to be. Deconstruct computer game conflict however, and what the players do is very similar to what they do playing Tetris or Candy Crush Saga – the core of the challenge is to point at as many icons as you can before the time runs out.

The National Museums of Scotland have experimented with introducing a time-based, ludic element to the cultural heritage experience. In Capture the Museum “visitors download an app to their smartphone and sign up to either the Red or Blue clan. The two sides plan their strategies then spread out across the National Museum of Scotland. A map that updates in real-time shows which clan owns which ‘territories’ – the differently themed galleries in the Museum. Players scan into territories using their phone’s camera, where they prove their understanding of the exhibits to earn the high score. After 30 fast and furious minutes the clan with the most territories is crowned the winner.”

Visitors coming to the museum’s programme of late-openings appear to enjoy the game, but time limits and territory capture may not be compatible with the every-day visitor’s wants and needs. However Sylvester argues that challenge is not as essential to games as it might appear, citing Dear Esther as a game that can “create powerful emotions without players struggling.”

Learning

All three games put learning at the core of their gameplay. And I don’t mean simply learning the patterns of the Space Invaders’ irresistible advance. Books describing the history and culture of Skyrim are liberally scattered around that game’s world. The player of Red Dead Redemption learns uncomfortable truths about their character, and the whole point of Dear Esther is to piece together seemingly random memories to reveal a tragic story. “If a lesson is obvious,” says Tynan Sylvester “there’s not much buzz in finally getting it because it was always fairly clear.” Instead, he advocates a moment of insight, where everything that has come before “clicks into place and reveals the shape of the whole.”

I’m not sure how successful cultural heritage sites are at shaping that moment of insight.

A couple of decades ago (which is in itself a scary fact) I was at a conference like this, sitting where you are now, listening to Judy Rand describe a model of interpretation which starts off with the main message, or theme, which she describes as the “single most important idea you want people to leave with.” With the theme in place, and informing all the subsequent decisions, all the other story elements or messages fall into one of three categories, primary, secondary, or tertiary, in decending order of importance.

Rand explains that this process is more than an arrangement of the relative value of the messages, it suggests a floorplan, with primary messages becoming sub-divisions of the exhibition and secondary messages indicating the contents of individual exhibits. Tertiary messages are only found by the most interested visitors.

It’s a sensible model, and one you can see in exhibition galleries right around the world, most obviously those that begin with an introductory video to put everything else in context. But it front-loads the moment of revelation, of insight.

The best attempt to subvert expectations I’ve seen recently was the Life and Death in Pompeii and Herculaneum exhibition, recently hosted by the British Museum. The introductory video was still there of course, but the final exhibit was an opportunity to remember that the reason we have discovered so much about the Roman way of life was a terrible intimate catastrophe.

Character Arc

Playing Dear Esther, you are intrigued to discover snippets of insight into a number of characters shared by a nameless narrator that may, or may not, be your avatar in the virtual world. Though there is an element of randomness in what you hear, the little world you inhabit is so tightly structured, that there is a very definite character arc.

Similarly the apparent open world of Red Dead Redemption, seems to offer a wide range of player choices, at first. But the choices are illusionary, you can’t change the story. As play progresses, you are forced towards your avatar, John Marston’s inevitable fate.

So Red Dead Redemption takes a more cinematic approach to narrative than Skyrim. In that game, you have more control over who your avatar is to begin with, and the avatar you design will affect how the game-world reacts to you to some degree. There’s far less control on where you go than in either Red Dead or Dear Esther, and a far wider choice of stories to explore.

In fact, each story turns out to be a lot more linear than Red Dead Redemption. Yes you can hop between stories, and tackle challenges in a wider variety of ways, so your avatar’s personal narrative may well be unique. But… the ending of Red Dead Redemption is powerful, and in Skyrim, I’ve saved the world twice already, and yet I’m still waiting for, longing for, the end credits.

You’ll see that, of the two, Skyrim’s narrative structure is closest to Rand’s model of interpretation. It just fizzles out when we get to bored to enjoy it any more. So my challenge to cultural heritage, is can we structure our stories to end with something closer to Red Dead Redemption and Dear Esther’s emotional punch?

The Holburne Museum in Bath recently experimented with a ludic interactive narrative form of interpretation. The project, called Ghosts in the Garden, involved giving groups a steam-punk listening device, which would allow twenty-first century visitors the opportunity to listen in on early nineteenth century conversations, taking place around the museum’s gardens, which were once a popular “pleasure grounds” attraction. At every location, users were given a choice about where to guide their nineteenth century interlocutor next, in the knowledge that they choice they made would change the outcome of the story, just like a “choose your own adventure book.”

I was interested in the emotional impact this innovative story-telling method might have, and offered to help evaluate the project with a user questionnaire. I received the data only recently, and I’ve not yet had time to analyse it properly. But one result I’ve already noticed is an apparent scepticism from users that they were actually changing the story.

So I’m just beginning to wonder if making a story interactive, is of less value than creating a strong character arc?

Music

And finally, we come to music.

Tynan Sylvester has this to say about music in games “Nobody ever gives it the credit it deserves because nobody consciously pays attention to it during play. But even though the conscious mind is oblivious, the unconscious is still processing the music into a continuous flow of feeling. You can tell because music is easily separable from the rest of the experience. Listen to a game soundtrack by itself, and you’ll feel much of what you felt during play. Play the game in silence, and you’ll be surprised at how hollow it feels.”

I know that’s true, the song that plays over the end credits of Red Dead Redemption has the power to bring a tear to my eye.

I’m going to lay my cards on the table here and say I think cultural heritage uses music incredibly badly (at least when music isn’t the subject at the centre of the heritage experience). Museums and Heritage sites too often resorting to clichéd soundtrack choices or only-just-appropriate royalty free generic music.

There’s an opportunity to use music far more intelligently in heritage spaces, Cohen (Prezi) explains how computer games use music to indicate breaks in the narrative, direct attention to particular spaces, communicate meaning, and trigger moods. The Dear Esther soundtrack is exemplary in this regard. The power of the music to invoke memories or “prepare the mind for a type of cognitive activity” is well recognized in advertising and sonic brands such as those created for Intel and Nokia. Why is the leitmotiv, such a useful tool in operatic storytelling for hundreds of years, so seldom used by museum spaces?

Conclusion – where next?

The sample size for my research around Ghosts in the Garden is too small, and of course, I’m sure I’ll discover I’ve asked the wrong questions. But I’ve got the opportunity to improve on the method with a similar project taking place at Bodiam Castle in Kent. I want to use that to test the validity of the emotional drivers I’ve defined today.

I also want to explore in more detail in the impact of, for want of a better word “designed music” on emotional engagement with cultural heritage sites. There’s something about how the games I’m examined use music (and silence) dynamically, that I’d like to find collaborators to help me test.

And I’m curious that participants of Ghosts in the Garden didn’t believe that their interaction really changed the story. It reminds me of what Aylett refers to as the Narrative Paradox  – allowing your audience to interact with, and change the story, reduces its cohesion. Colleagues at Southampton wrote a paper last year on the Narrative Braid, which attempts to tackle the narrative paradox in documentaries. I’m convinced that combining their ontologies, and concepts of narrative atoms, molecules and threads, with location and object based orientation, could make for exciting possibilities in heritage interpretation.

I wonder if interpretive narratives can become, if not interactive, then dynamic, in a similar way to computer game music, sticking to a well written engaging narrative arc, but layering different elements, tones and details into the story, depending on the agency of the visitor and where they choose to go with the heritage space.

Thank you.

Is it 1:20 am? Or is it 8:20pm?

Trying not to go to bed at 8:20, though I REALLY wanted to a couple of hours ago. I’m trying to stay awake though to better adjust to local time. So forgive this little travelog, It is, after all my very first international conference, so I’m allowed just a little excitement. My day started at 5am. UK time. My lovely wife drove me to the airport in time to get through security by 6:33. Then of course, having got through so quickly I had an hour to wait before my gate was announced. When it was it was a good long walk away, so that took about fifteen minutes, which was nice, then I sat down for a few minutes with Country Life. Seriously, when did they start giving Country Life away at airports? Anyhow, I picked it up because it had some lovely Light Sussex chickens on the cover, just like my girls. These particular chicken cover-girls were part of a feature article encouraging the landed gentry who usually read the magazine to use their local breeds.

I got a good view of the snow and mountains on Greenland as we flew over.

A couple of hours later I got the sort of views of Canadian lakes that made me want to come back as a tourist. But I only had long enough at Toronto Airport, to go through US border control, and then with three other passengers, board a little Beechcraft 1900 turboprop for the hop across Lake Ontario to Rochester.

Goodbye Canada

Hello USA

 

Evaluating emotional triggers

The organisation I work for asks a question of it’s visitors, along the lines of “how strongly do you agree or disagree with the statement ‘this place had a real emotional impact on me’?” We can see that the more people agree with that statement (even if only the minority strongly agree), the more likely people are to have a very enjoyable day, and recommend a visit to friends and relations. But we don’t really measure what drives that emotional response.

I asked a similar question when evaluating Ghosts in the Garden, and I experienced a familiar sense of frustration about how little insight it allowed me. So I plan to work out a set of questions that might be more informative about what drives an emotional connection with a place.

I’m going to start by looking in more detail at this thesis by Mohd Kamal Othman, which I first heard about at the CDH conference in July. Othman was evaluating mobile experiences, and in doing so created a Museum Experience Scale and an Church Experience Scale (one of the projects he was evaluating was a mobile guide for Churches).

I’m particularly interested in the questions the evaluation asked to measure what he termed “emotional connection”:

• The exhibition enabled me to reminisce about my past
• My sense of being in the exhibition was stronger than my sense of being in the real world
• I was overwhelmed with the aesthetic/beauty aspect of the exhibits
• I wanted to own exhibits like those that I saw in the exhibition
• I felt connected with the exhibits
• I like text-based information as supporting material at museum exhibitions
• I felt spiritually involved with the church and its features
• I felt connected with the church and its features
• I felt emotionally involved with the church and its features
• I felt moved in the church
• The church had a spiritual atmosphere
• My sense of being in the church was stronger than my sense of being in the rest of the world

Now it strikes me that the church specific questions are a little less specific than the ones created for exhibitions, but I’ve not yet read about the reasoning behind them. Some of the questions though (touching on presence, spectacle and acquisition, for example) resonate with what I’ve been discovering about emotional triggers in games. I feel there’s something here to build on.

(Now, I better get back to writing that presentation for Decoding the Digital)