One of the challenges of my viva was that I had presented cultural heritage as only a learning environment. Which was definitely not my intention. the places I have worked with especially, historic houses and palaces with gardens and parks, and countryside estate have long been recognised as places of spiritual recharging. I argued the case (obviously adequately as I passed) that I had not focussed only on learning, but perhaps some references to papers like this one will restore any intended imbalance.
“This one” is Packer, J. & Bond, N.. 2010. Museums as Restorative Environments. Curator 54: 421 – 436.and it starts of with a doozy of a quote that might be all I need: “Mental fatigue, caused by the stresses and strains of everyday life, is a common complaint in today’s society, and the need to escape from the personal and interpersonal demands of life is one of the major reasons that people have for engaging in tourism and leisure experiences”
The paper goes on to explain that al lot of the theory of restorative environments is based on the work of Kaplan and his Attention Restoration Theory. “The capacity to continually focus attention on a particular activity can be reduced or lost through mental exhaustion.” Recovery from just mental exhaustion requires that your “attention is engaged involuntarily or effortlessly,” which (I think) Kaplan calls “fascination.” Fascination allows your directed attention be rested.
Well, I think cultural heritage is fascinating, and this theory echos the theory of Flow, which I write about in my thesis, so it seems we are on to a winner with this paper. But there are three other components of restorative environments: being away (from routine); extent (the environment need to have enough content to keep you occupied a while) and compatibility (of interest – being bored is not fascinating). These sorts of things point to the infinite horizons of a walk in the countryside as being restorative (if the countryside is compatible with your interests) but analysis of responses to museums, art museums, gardens and zoos shows that the countryside is not the only place you can recharge.
So this paper looks to see if there are factors that make one museum a better restorative environment than another. To do so they use a “satisfying experiences framework” which focuses on:
“object experiences, which focus on something outside the visitor, such as seeing rare, valuable, or beautiful objects;
cognitive experiences, which focus on the interpretive or intellectual aspects of the experience, such as gaining information or understanding;
introspective experiences, which focus on private feelings and experiences, such as imagining, reflecting, reminiscing, and connecting; and
social experiences, which focus on interactions with friends, family, other visitors, or museum staff.”
They measured this and four different sites in Australia: a museum; an aquarium; a garden; and, an art gallery. Immediately they spotted that visitors to each find different experiences the most satisfying. The object experience of the fish in the aquarium was by far the most satisfying experience in any place, the cognitive experience of the museum was the most satisfying part of that visit (hmmmm). The social experience most satisfying in the gardens and, tough the art gallery was the most balanced between the four experiences, the social experience mattered to only 10.7% of visitors and the introspective experience mattered most to 29%. But it “was found that local visitors placed more importance on social and introspective experiences, and tourists placed more importance on cognitive and object experiences.” Tourists of course “are more likely to be looking for a learning and discovery experience—they want to discover new things and often try to ‘‘see as much as they can.’’ These experiences may be incompatible with a restorative experience.”
The study concludes “Not only were national parks and beaches considered more restorative than urban environments, but among the research sites, those that were focused on natural heritage (especially the botanic garden) were considered more restorative, both in attributes and benefits, than those focused on cultural heritage (the museum and art gallery).” But for frequent visitors (rather than first time visitors) museums can “offer an alternative to natural settings as a restorative experience.”
And more importantly, if “greater attention were given to visitors’ comfort, first-time and infrequent visitors, who are less familiar with the site, may be more able to experience restorative benefits as a result of their visit.” The authors also suggest that museums should “explore ways in which introspective experiences might be encouraged and supported” and this I feel supports what I concluded at Chawton – while we insist on authenticity in our storytelling, we are privileging cognitive experiences over introspective ones.
At my viva my external examiner pointed me towards this interesting paper, which she had co-authored – partly, I think, as an example of how I should restructure the discussion of my Chawton experiment in my thesis. But it contains some real gems ( like “the museums studies literature points out the restorative value of an aesthetic experience that is clear of any information acquisition or learning objective and is centred instead on the sensorial experience of being there”) that makes me regret missing it in my literature review: Marshall, M. , PETRELLI, D., DULAKE, N., NOT, E., MARCHESONI, M., TRENTI, E. & PISETTI, A.. 2015. Audio-based narratives for the trenches of World War I : intertwining stories, places and interaction for an evocative experience. International Journal of Human-Computer Studies 27-39.
It’s a case study of a prototype “visitor aware personalised multipoint auditory narrative system that automatically plays sounds and stories depending on a combination of features such as physical location, visitor proximity and visitor preferences” Voices from the Trenches for a First World War exhibition at the Museo Storico Italiano della Guerra in Italy. What particularly interest me is that its part of the Mesch project which has some other outcomes which I refer to in my thesis. The paper describes their intent to move away from what they call “the information centric approach of cultural heritage.” I am sure a number of my professional colleagues would bridle somewhat at this accusation. After all, did not Tilden tell us in the 50’s that interpretation was more than mere information? But one od the things that my Chawton experiment uncovered was that actually too much “interpretation” turns out to be mere information after all.
The authors summarise previous experiments in responsive soundscapes, such as LISTEN, which “composes a soundscape of music and/or commentaries depending on the detected visitor’s behaviour: visitors that are not close or are moving are classified as unfocussed and for them a soundscape is created, while visitors that are standing still and close to the artwork are classified as focussed and a narrative (e.g. the curator describing the artwork) is played over the headphones.” Though many soundscapes are delivered by headphone, to avoid sound pollution for other visitors, the interesting project SottoVoce is designed around eavesdropping on what other people in our party are listening to. Half the respondents (in groups of two) heard the soundscape from each others phone speakers, while the other half had headphones. “When in loudspeaker mode visitors focussed on what was displayed on the screen of the mobile device and stayed close to the sound source while partners linked via the same audio on their headphones had a more dynamic visit driven by each other’s interest in the exhibits.”
“The ability to convey and evoke emotion is a fundamental aspect of sound” they say, and explain “The affective power of voice and audio storytelling has been recognised as creating a connection to the listener and is even amplified when spoken words are not coupled with the visual capture of the storyteller, creating a sense of intimacy and affective engagement.” An they built their soundscapes using the same sort of mix of music, speech and other sounds that I used (in a limited fashion) at Chawton. Some of the primary source material was recorded to sound more like oral history, with actors reading the words “with palpable emotion” to be more affective. The responsiveness is similar to that of LISTEN, but the “staying still” metric isn’t used, instead a simpler proximity method is used. woven into that soundscape are voice recordings for attentive listening, which is selected by the visitor choosing from a selection of cards. The sound was delivered by loudspeakers but, unlike SottoVoce, not on people’s own devices, rather places around the site. This was what I did for Chawton UNtours too.
The particular challenge with this project was that it was outdoors.The difficulties of maintaining equipment, connecting power and data etc means that most sites resort to delivering via mobile device. But on the other hand: “While engagement in a museum tends to be via prolonged observation, in an outdoor setting multiple senses are stimulated: there is the physical, fullbody experience of being there, the sight and the sound of the surroundings, possibly the smell too. The multi-sensory setting places the visitor in direct connection with the heritage and enables engagement at an emotional, affective level rather than at a pure informative level.” (p6) The danger of using a mobile device to deliver interpretation is one I wrote about here, but essentially it stake them out of the where they are, it is the antithesis off presence.
With all this in mind the designers of the project set out five clear principles:
To engage at multiple levels, not just cognitive
To focus the visitors’ attention on the heritage, not the technology
To deal with group dynamics sensibly
To be provocative and surprise visitors, but design simple and straightforward interactions
To personalize content on the basis of clear conditions
The choice of sound over anything screen-based was an outcome of the second principle. Loudspeakers rather than headphones was also an attempt to focus attention on the heritage: “During a small experiment in a local outdoor heritage site, we observed that audio creates a wider attraction zone where passersby become aware of the sound source, and a closer engagement zone around the emitting point where one has to stop and listen in order to understand what the voice says.”
So they designed a soundscape that featured music nd sound to attract visitor to a location and then vice recording to hold them there. The narratives are arranged thematically, with different voices (authoritative and intimate) indicating the nature of the content. Quite how the visitor chooses is not really made clear but I expect it is by approaching the voices that most attract them.
The team trialed the idea by observing the visitors behaviour using about 23 minutes of content, but I was disappointed that they did not come up with any solutions to the problems we encounter trying to evaluate the soundscape at The Vyne. It is hard to observe and distinguish between active listening and background listening. The authors seen to assume that if the active listening content is playing, then the partiocilapants are actively listening. The only evidence they have for this is a qualitative questionnaire, which I am not convinced is an accurate measure on engagment. Yes they said they enjoyed an benefitted from the experience, but if they did not know that was what was being tested, what proportion would have even mentioned the soundscape.
Of course they identified a number of challenges, not least fine-tuning the volume to be loud enough to attract attention and yet not so loud to cause discomfort. This is especially true of the different voices, with some by necessity quieter and more intimate. Of course they also predicted issues overs scalability – similar to the ones I planned fro but wasn’t able to properly test at Chawton “how well would such a system work in a busy environment with many groups interacting.”
I’ve been head down, completing my upgrade package for weeks and so you have seen very little form me on this blog. But that package was submitted last Friday, and this week, I am at the University’s Hartley Residency, which is a refreshing opportunity to just learn and think.
Yesterday we had a seminar, and then a lecture form Steven Rings, of the University of Chicago. I always thought I was a bit a fraud in the archaeology department, but now I am in the music department (Did I tell you I have transferred to the music department? That’s another story.) am I listening to stuff I know nothing about. “But,” as they say, “I know what I like.” So thank the stars that Steve was talking about liking music, and writing academically about music that we like. He kicked of with his talk, quoting Max Weber, who, correct me if I am wrong, accused modern thought, secularism, science and academia of “disenchantment of the world.” Rings argues that there is an academic pressure to distance oneself from the music one studies, to analyse it scientifically, reducing it to numbers, or socio-politically, reducing it to a series of choices made within a dominant ideology. In a way, to destroy it – to remove from it any sense of aesthetic pleasure, or “enchantment”.
All this talk of enchantment, reminds me of a couple of papers I read months ago, but didn’t blog about. To be honest, and didn’t think either contributed much to my thesis either. But they are in my mind because I recently went back to them and added a couple of bits from as least one of them to my draft. Keirsten Latham writes about “the Numinous Museum”, and something about enchantment, and secularism made me think about that term. In the 2007 paper, The Poetry of the Museum: A Holistic Model of Numinous Museum Experiences she says “Numinous experiences (also referred to as reverential, pivotal, profound) with any museum objects/exhibits are akin to aesthetic experiences with objects of art and encounters with the beautiful.”
“Reverential, pivotal, profound…” is this the same, or similar to “enchanting”? She goes on to say that “Numinous experiences are seen as a deeply felt, connective encounter with any object not just artistic works or beautiful things and can happen anywhere and anytime, depending on the coming together of many things at one point in time.” Which is interesting because it potentially equates the mundane with the spiritual. For example, visiting Crete recently, I was was intellectually stimulated by my day-trip to Knossos, but my own numinous experience was at a less visited palace at Malia. There, it was the act of looking down at threshold stone as I stepped on it, that gave a profound, emotional feeling of stepping on the same stone as someone had thousands of years ago. I understood it a Knossos but I hadn’t felt it.
Latham explains the term she uses comes, via Catherine Cameron and John Gatewood, from Rudolf Otto who, in his book, The Idea of the Holy used the word numen to describe a religious emotion or experience that can be awakened in the presence of something holy.
Which brings us back to secularisation: are we reluctant to talk about music (or anything) we love in terms of enchantment, for fear of being seen to worship it?
“What are Emotions?” is a question asked by William Reddy (2001) in his book The Navigation of Feeling: A Framework for the History of History of Emotions. The first part of that books looks at the answers, from Cognitive Psychology in the first chapter, and Anthropology in the second. He points out early, however that:
Western specialists who study emotion cannot even agree on what the term emotion means.
He references Paul Ekman (et al)’s attempts to codify emotions by the few facial expressions that all cultures seem to attach a shared meaning to. That this is in the Cognitive Psychology chapter is interesting – I would have suggested it’s a more anthropological approach, but as Reddy points out it did encourage a lot of study looking at biological indicators of emotion, such as heart rates, hormone levels, skin conductivity etc. Reddy also points out two “nagging problems: (1) What happened to emotions when arousals subsided and the face returned to normal? (2) How were emotions such as love. shame or nostalgia to be fit into the scheme, when they had no obvious single facial expressions to go with them?” Not only that, he argues that “twenty years of work by many researchers” has shown only that “In the absence of forced choice and pretest training, agreement on other than happy faces was weak. If photographs of spontaneous facial expressions were used (i.e., naturally occurring ones, rather than the carefully posed ones of the initial tests), agreement sagged further.”
He also highlights the problematic relationship between emotion and cognition: “If
a sudden sense of fear redirects attention to a dark corner of the room, why not conclude that this sense of fear is the cognition of the potential danger of that corner? No experimental or test procedure has been offered so far that would allow one to rule out this possibility; it is resisted solely on the grounds that it counters the commonsense belief that emotion is something separate from thought, something opposed to ‘reason.'” That “reason” he argues is questioned more and more. Try as they might, philosophers and researchers are finding it harder, not easier to distinguish between “what counts as voluntary or controlled, and what counts as involuntary or automatic.” This is not helped by subjects’ mis-attribution of arousal.
Reddy also explores the difficulties measuring emotion. Most psychologists use valence, a measure of how pleasant or unpleasant an emotion is, and intensity, a scale of how difficult it is to override the emotion. Of course its not so black and white, fear is an “unpleasant” emotion, and yet horror movie audiences, theme park riders and gamers actively seek it out. Similarly Reddy argues that am emotion’s intensity may be altered by the triggering event’s relevance to a person’s goals – here he uses health anxiety as an example, “we generally pursue health for its own sake; but it is obvious that health is a means or condition for the pursuit of many, many other goals. As a result, pursuit of health as a means and pursuit of health as an end in its own right are likely to be indistinguishable (both to observers and to the person involved). Like-wise, loss of health is widely regarded with fear or anxiety. Such fear or anxiety is a “badge” of the deep goal relevance of health.”
When he goes on to explore mental control, I fear he strays more into the study of cognition that emotion, but he is striving to support an argument that “emotions can be regarded as overlearned cognitive habits.” I think Panksepp and Bevan might disagree, but I’m not seeking to argue the point – for my studies, whether emotions exist beyond cognition, or are tightly intertwined with cognitive thought is hardly relevant. When it comes to a cultural, or anthropological, approach I think my conclusions will be the same – my model already separates out social emotions from the more visceral ones identified by Panksepp and Bevan.
Reddy begins his examination of an anthropological approach by outlining the idenitity crisis of the discipline itself, especially in the area of the “production of knowledge”. He spills little ink on the psychocultural model of emotions. This approach, as with cognitive psychology, is built on the idea that there is “a broad commonality in human emotions”. He tackles the contructionist approach first, with the work of Michelle Z Rosaldo. Her study of of the Ilongot people of a mountainous region of the Philippines led her to conclude that emotion was, at least in part, a product of language. However a roughly contemporaneous study in Tahiti by Robert Levy concluded that people suffered the symptoms of grief even if they didn’t have a word for it. Its interesting to note that the two in these studies examples given are opposed on my model. Grief is one of the mammalian emotions identified by Panksepp and Bevan, while Rosaldo’s word liget, while not having a direct analogue in English seems to fit well with Fiero in my model.
However, Reddy also relates a constructionist argument that is also a more cutting critique of the psychological view of emotions, from Catherine Lutz. “In Lutz’s view, the notion that emotions are biologically based is not simply erroneous, it is part of a larger, insidious, gender-biased Western view of the self that privileges alleged male rationality over the supposedly natural emotionalism of women. Expert and lay assumptions coincided, Lutz charged, in regarding emotions as internal, involuntary, irrational, potentially dangerous or sublime, and female. Men were rational and therefore better suited to action in the public sphere. Ethnographic research showed, however, that emotions were a product of social interactions and showed, as well, that outside the West, emotions were generally not distinguished from thinking in the peculiarly sharp way Westerners distinguished them, and were generally regarded as an outcome of social interaction, rather than as rising up, ineffably, from within.” This argument requires a change in philosophy. If knowledge is constructed by culture, it hard to criticize constructed knowledge, be that headhunting in the Philippines, or Western cognitive science. So, according to Reddy, Lutz’s approach was less about culture and more about discourse as defined by Foucault. But that is an argument well beyond the scope of my work.
Reddy gives some attention to other anthropological models too, but not much. What clearly interests him about those me mentions are steers towards a concept of emotion as performance. Indeed his long conclusion to the chapter, which starts out as an attempt to reconcile cognitive psychology and anthropology, is mostly about control of emotion through performance – oversimplified as putting on an happy face, or to use the example he prefers a “‘bright face’ (mue cedang)” in Balinese.
I have spent far too long on rewriting this section of my thesis these last few days, constantly going back to the model itself and tweaking it, then readjusting it, them moving things around and finally tweaking and correcting it once more. And the texts has been through so many edits I’m not even sure it makes sense to anyone anymore. If you have the time and inclination, I’d appreciate it if you could give this the once over and tell me if its in English…
(Its about how games trigger emotions, by the way. Oh and I’m not doing links, or proper references at this stage – you’ll have to make do with my EndNote metatext.)
Let’s start with some neurobiological research. Panksepp and Biven’s{, 2012 #96} thesis is that all mammals share seven instinctive emotions, even if different species’ higher brain functions can be very different. They label the seven core emotions: SEEKING; RAGE; FEAR; LUST; CARE; PANIC/GRIEF; and, PLAY
The core mammalian emotions defined by Panksepp and Biven
Everything else, they say, are sensory affects such as hunger, or socially constructed. It’s interesting to note that play is here identified as an emotion, not just a behavior. What evolutionary benefit does a play emotion give us, and other mammals? Panksepp and Biven suggest “to learn nonsocial physical skills like hunting, foraging and so on. It is also surely important for acquiring many social capacities, especially nascent aggressive, courting, sexual and in some species competitive and perhaps even parenting skills. It may be an essential force for the construction of the many higher functions of our social brains. Playful activities may help young animals learn to identify individuals with whom they can develop cooperative relationships and know who to avoid […] In short, the brain’s PLAY networks may help stitch individuals into the stratified social fabric that will be the staging ground for their lives.”
Nicole Lazarro{, 2009 #69} draws on the work of Paul Ekman in her study of emotions experienced by gamers. She explains how his research identified six emotions which appear to have universal facial expressions and a number of other emotions which while not quite meeting his criteria for being universal, come very close: emotions like Naches (pride in another’s achievements), Fiero. These two, along with schadenfreude and embarrassment add handy descriptors to the social emotions that Panksepp and Biven lump together and ignore in their study. Similarly, disgust puts a name to one of the sensory affects. I’ve also added her “ wonder” to that sensory category. And, while descriptors for the six core emotions sit well with her research, some of the emotions she lists in her study, for example Amusement, Curiosity, and Anger/Frustration, add nuance to some of Panksepp and Biven’s descriptors, so I’ve included them in the diagram below. All these emotions, she says, can frequently be recorded and recognised when watching players of video games.
Lazarro’s observed emotions in gamers
So, now we have a model of the core emotions, as described by Pankspepp and Bevin, overlaid with emotions attributed to gamers in the work of Lazarro. To that I have below added a third layer. The are affective triggers, drawn from Lazarro and Sylvester’s recommendations for game mechanics, or affordancies, and that trigger emotion in gamers, and Hamari{i, 2014 #148} et al’s motivation mechanics from their study of gamification. Given that each is a conflation of descriptions from up to three sources, they deserve some further explanation.
The final (?) model
Environment and spectacle – While there is an enduring fashion for games that retain the “eight bit aesthetic”{Stuart, 2012 #165} the general trend of graphic development has been towards cinematic realism and “presence” {Riva, 2014 #4; Pinchbeck, 2005 #30; Pinchbeck, 2009 #33}. “Presence originates from the term ‘telepresence’, made famous by the computer scientist Marvin Minksy in a 1980 paper of the same name. From around 1991 (the date of the first issue of the MIT journal Presence: Teleoperators and Virtual Environments), presence has been typically defined as the capacity of the technology to make the user feel transported into a remote place and be able to efficiently interact with it.” {Pujol, 2012 #56}
Pujol and Champion attempt to unpick the definitions of presence. Starting with the idea that the ideal is a sense of being there, or blanking out the digital mediation of screen and controller, they touch upon immersion as a product of field of view and optical resolution. They also briefly summarize the idea that the human component of the system is likely to respond to the affordancies offered by the VR according to their interests, if the virtual component can in turn respond in a realistic way. They touch upon co-presence (sharing the VR with other users) before arguing that social presence (interacting with other users and virtual agents) is vitally important idea in the “potential [their emphasis] convergence between the presence and cultural heritage fields.”
“The conventional notion of presence as the sensation of ‘being there’ is a highly simplified way of expressing an internal perception of the environment and of ourselves in relation to it. A more comprehensive explanation would be that the sense of presence results from the interaction of various factors. These factors depend both on the system (immersivity, visual accuracy, real-time physical and social interactivity, invisibility of devices, consistency of the content) and on the participant (perception, attention, empathy, engagement, meaningfulness or relevance of the content, control, suspension of disbelief).” {Pujol, 2012 #56} In one game, set in the tunnels of a post-apocalyptic Moscow underground, Sylvester describes how play moved to the ruined, frozen, surface and the powerful effect it had on him as a player: “Though most would call Metro 2033 a shooter or RPG, I wouldn’t, because I don’t think its about shooting or roleplaying. I think its about discovering how a place like that makes you feel.”
Whether or not the gamers Lazarro studied felt they had been transported to the virtual world, she did see evidence of Wonder in their expressions, often from beautifully rendered or animated environmental effects. Sylvester separates out the beauty of “a sunset over the ocean” from the spectacle of “a slow-motion dive to dodge an incoming rocket.” He also offers a warning not to over use either.
Acquisition, points and leaderboards are reasonably self explanatory. At the crudest level, they are sharable social evidence of success, skill or commitment. Their affective and motivational power comes from sharing, but acquisition also has an impact on the individual. As Sylvester acknowledges, gambling games are all about acquisition, and computer games often simulate the acquisition of wealth (or simply points). Of course gambling works in two ways, and the bitter emotions of loss shouldn’t be disregarded.
Insight – Learning rewards and encourages curiosity and seeking. Not just any old learning though. “If a lesson is obvious,” says Sylvester “there’s not much buzz in finally getting it because it was always fairly clear.” Instead, he advocates a moment of insight, where everything that has come before “clicks into place and reveals the shape of the whole.”
Challenge is the emotional trigger that we most readily associate with video games, testing the the player’s dexterity and pattern learning before rewarding him or her not just with a sense of accomplishment, but progression within the game. Sylvester argues that it is not as essential to games as it might appear, and cites Dear Esther as a game that can “create powerful emotions without players struggling.” However, Lazarro says that the opportunity for challenge and subsequent mastery can be an important motivation for some, if not all gamers. She quotes the anonymous wife of one “hardcore” gamer “I always know how my husband feels about a game. If he screams ‘I hate it! I hate it! Ihate it!’ then I know two things. A) He’s going to finish it. B) He’s going to buy version two. If he doesn’t say these things he will put it down after a couple of hours.”
Threat – Sylvester conflates revulsion (disgust) and fear, but here I’m using Threat as an affordance of the core Fear affect. Some things (Sylvester cites spiders and snakes) scare enough people (whether instinctively or through cultural learning) that they are often used in games to provoke fear. Of course the fear response on a gamer sitting in an armchair in front of a monitor is going to be different from the fear of the same person lost in a dark forest, but some games work hard to create a close approximation. Sometimes it’s what you can’t see that scares you, and so game designers have learned to use cinematic effects such as jump-cuts and music to to increase tension and invoke a fear response.
Sex is a hard thing to get right in video games, though many have tried. “Let’s be honest, sex in games is rarely super sexy sex.” Say the video game journalists Nielsen and Grey {Nielsen, 2016 #166} “… the key to effective video game sex: good writing, good characters.”
Which brings us to the narrative arc, this is what film can do so well, engaging the audience’s empathy with one or more characters, as they face internal conflicts, grow and change. Change is a common factor for a lot of the affordances described above, and indeed, Sylvester declares that “the bedrock principle behind all emotional triggers is change”. I will address narrative in more detail in chapter 3. (XX)
Feedback, or rather the absence of it, is a trigger for panic/grief which, as Panksepp and Bevin explain has a lot to do with isolation in mammals. When our character stops responding to your controls, is when a player feels a fleeting moment of grief. Feedback is offered to gamers through what happens on screen, haptic engines in controllers, and by sounds and music.
There’s a lot to say about music, which I will address more fully in the next section. Before we move on, note how, in the model I’ve described, affordancies are placed in boxes radiating away from the core, social or sensory affects that they are most likely to trigger. In my text though I’ve already mentioned how some affordancies might contribute to the triggering of other affects: revulsion might help trigger fear for example. Sound and music can be used to trigger, or contribute to triggering, all the affects. Imagine the soundtrack to Jaws or Psycho, and contrast that to the music of Love Story for example. Music is playful, yes, so I’ve placed the label there, but its influence is more diffuse, an I’ve represented it in the diagram with a fuzzy halo around all the affects, to indicate how it can trigger our emotions without us even knowing:
“And music is wonderfully subtle – even more than most emotional triggers. Nobody ever gives it the credit it deserves because nobody consciously pays attention to it during play. But even though the conscious mind is oblivious, the unconscious is still processing the music into a continuous flow of feeling. You can tell because music is easily separable from the rest of the experience. Listen to a game soundtrack by itself, and you’ll feel much of what you felt during play. Play the game in silence, and you’ll be surprised at how hollow it feels.” {Sylvester, 2013 #89}
The Story Extension Process, from Mei Yii Lim and Ruth Aylett (2007) Narrative Construction in a Mobile Tour Guide
Another great piece from Ruth Aylett, this time from 2007. Here, she and collaborator Mei Yii Lim are getting closer to what I’m aiming for, if taking a different approach. They kick off by describing Terminal Time, a system that improvises documentaries according to the user’s ideological preference, and an intelligent guide for virtual environments which take into account the distance between locations, the already told story, and the affinity between the the story element and the guide’s profile when selecting the next story element and location combination to take users to. They note that this approach could bring mobile guides “a step nearer to the creation of an ‘intelligent guide with personality'” but that it “omits user [visitor] interests”. (I can think of many of a human tour guide that does the same). They also touch on a conversation agent that deals with the same issues they are exploring.
This being a 2007 conference paper, they are of course using a PDA as their medium. Equipped with GPS and text to speech software, a server does all the heavy lifting.
“After [an ice-breaking session where the guide extracts information about the user’s name
and interests], the guide chooses attractions that match the user’s interests, and plans the shortest possible route to the destinations. The guide navigates the user to the chosen locations via directional instructions as well as via an animated directional arrow. Upon arrival, it notifies the user and starts the storytelling process. The system links electronic data to actual physical locations so that stories are relevant to what is in sight. During the interaction, the user continuously expresses his/her interest in the guide’s stories and agreement to the guide’s argument through a rating bar on the graphical user interface. The user’s inputs affect the guide’s emotional state and determine the extensiveness of stories. The system’s outputs are in the form of speech, text and an animated talking head.”
So, in contrast to my own approach, this guide is still story lead, rather than directly user led, but it decides where to take the user based on their interests. But they are striving for an emotional connection with the visitor. So their story elements (SE) are composed of “semantic memories [-] facts, including location-related information” and “emotional memories […] generated through simulation of past experiences”. Each story element has a number of properties, sematic memories for example incude: name ( a coded identifier); type; subjects; objects; effects (this is interesting its lists the story elements that are caused by this story element, with variable weight); event; concepts (this that might need a further definition when fist mentioned); personnel (who was involved); division; attributes (relationship to interest areas in the ontology); location; and, text. Emotional story elements don’t include “effects and subjects attributes because the [emotional story element] itself is the effect of a SE and the guide itself is the subject.” These emotional memories are tagged with “arousal” and “valence” tags. The arousal tags are based on Emotional Tagging, while the valence tag “denotes how favourable or unfavourable an event was to the guide. When interacting with the user, the guide is engaged in meaningful reconstruction of its own past,” hmmmmm.
So their prototype, a guide to the Los Alamos site of the Manhatten project, the guide could be either “a scientist who is interested in topics related to Science and Politics, and a member of the military who is interested in topics related to Military and Politics. Both guides also have General knowledge about the attractions.” I’m not convinced by the artifice of layering onto the interpretation two different points of view – as both such are being authored by a team who in their creation of the two points of view will, even if striving to be objective, will make editorial decisions that reveal a third, authentic PoV.
When selecting which SE to tell next, the guide filters out the ones that are not connected to the current location. Then “three scores corresponding to: previously told stories; the guide’s interests; and the user’s interests are calculated. A SE with the highest overall
score will become the starting spot for extension.” The authors present a pleasingly simple (for a non-coder like me) algorithm for working out which SE goes next. But the semantic elements are not the only story elements that get told. The guide also measures the Emotional, Ideological story elements against the user’s initial questionnaire answers and reactions to previous story elements and decides whether or not to add the guide’s “own” ideological experience on to the interpretation, a bit like a human guide might. So you might be told:
“Estimates place the number of deaths caused by Little Boy in Hiroshima up to the end of 1945 at one hundred and forty thousands where the dying continued, five-year deaths related to the bombing reached two hundred thousands.”
Or, if the guide’s algorithms think you’ll appreciate it’s ideological perspective, you could hear:
“Estimates place the number of deaths caused by Little Boy in Hiroshima up to the end of 1945 at one hundred and forty thousands where the dying continued, five-year deaths related to the bombing reached two hundred thousands. The experience of Hiroshima and Nagasaki bombing was the opening chapter to the possible annihilation of mankind. For men to choose to kill the innocent as a means to their ends, is always murder, and murder is one of the worst of human action. In the bombing of Japanese cities it was certainly decided to kill the innocent as a means to an end.”
I guess that’s the scientist personality talking, perhaps the military personality would instead add a different ideological interpretation of the means to an end. As I mentioned before, I’m not convinced that two (or more) faux points of view are required when the whole project and every story element that the guide gets to choose from are already authored with a true point of view. But in many other aspects this paper is really useful and will get a good deal of referencing in my thesis.
Among the mince pies and over-cooked turkey over Christmas, I managed to find a little time to read an interesting paper. #Scanners: Exploring the Control of Adaptive Films using Brain-Computer Interaction shows once again, that the cool people are all at the University of Nottingham. What these particular four cool guys did was put a mini cinema in an old caravan. But this particular cinema wasn’t showing an ordinary film. Rather, the “film was created with four parallel channels of footage, where blinking and levels of attention and meditation, as recorded by a commercially available EEG device, affected which footage participants saw.”
Building on research in Brain Computer Interface (BCI) the team worked with an artist to create a filmed narrative that “ran for 16 minutes, progressing through 18 scenes. However, each scene was filmed as four distinct layers, two showing different views of the central protagonist’s external Reality and the other two showing different views of their internal dream-world.” Which layers each viewer saw was selected by the EEG device, for rather by the viewers’ blinks and states of “attention” or “meditation” as recorded by the device. The authors admit to some skepticism from the research community about the accuracy of the device in question, but that was not what as being tested here. Rather, they were interested in the viewers’ awareness of the ability to control the narrative, and their reaction to that awareness.
I was interested in the paper for two reasons. First of all, their conclusions touch upon an observation I made very early in in my own research, looking at Ghosts in the Garden, I got a small number (therefore not a very robust sample) of users of that interactive narrative to fill out a short questionnaire, and I was surprised by the number of respondents who were not aware that they could control (were controlling) the story through the choices they made. The #Scanners team noticed a similar variation in awareness, but more than that, they found that “while the BCI based adaptation made the experience more immersive for many viewers, thinking about control often brought them out of the experience.”
They conclude that “a traditional belief in HCI is that Direct Manipulation (being able to control exactly what you want to control) sits at the top of both these dimensions. We examined, however, how users deviate from line, and enjoyed the experience more by either not knowing exactly how it worked, or by giving up control and becoming re-immersed in the experience. […] these deviations from the line between knowledge and conscious control over interaction are most interesting design opportunities to explore within future BCI adaptive multimedia experiences.”
With which, I think I agree.
The other reason the paper interests me is that they described their research as “Performance-Led Research in the Wild” and pointed me towards another paper to read.
Just drafted an abstract for my Sound Heritage presentation:
This presentation explores what computer games can teach us about emotional engagement in cultural heritage interpretation. Beginning with a model of emotional affect drawn from the work of Panksepp and Biven (Panksepp, 2012), Lazarro (Lazarro, 2009), Sylvester (Sylvester, 2013)and Hamari et al (Hamari et al., 2014), it reveals how music especially has become a versatile emotional trigger in game design.
Drawing on the work of Cohen (Cohen, 1998)and Collins (Collins, 2008)eight functions that music has in games:
Masking – Just as music was played in the first movie theaters, partly to mask the sound of the projector, so music in new media can be used to mask the whir of the console’s or PC’s fan.
Provision of continuity – A break in the music can signal a change in the narrative, or continuous music signals the continuation of the current theme.”
Direction of attention – patterns in the music can correlate to patterns in the visuals, directing the attention of the user.
Mood induction; and,
Communication of Meaning- the nice distinction here is between music that makes the user sad, and music that tells the user “this is a sad event” without necessarily changing the user’s mood.
A cue for memory – The power of the music to invoke memories or prepare the mind for a type of cognitive activity is well recognized in advertising and sonic brands such as those created for Intel and Nokia.
Arousal and focal attention – With the user’s brain stimulated by music s/he is more able to concentrate on the diagesis of the presentation.
Aesthetics – The presentation argues that all too often music is used for aesthetic value only in museums and heritage sites, even if the pieces of music used are connected historically with the site or collection.
As an example, the presentation describes a project to improve the way music is used in the chapel at the Vyne, near Basingstoke. Currently, a portable CD player is used to fill the silence with a recording of a cathedral choir, pretty, but inappropriate for the space and for it’s story. A new recording is being made to recreate about half an hour of a pre-reformation Lady Mass, with choisters, organ and officers of the church, to be delivered via multiple speakers, which will be even more pretty but also a better tool for telling the place’s story.
With a proposed experiment at Chawton House as an example, we briefly explore narrative structure, extending the concept of story Kernels and Satellites described by Shires and Cohan (Shires and Cohan, 1988)to imagine the cultural heritage site as a collection of narrative atoms, or Natoms (Hargood, 2012), both physical (spaces, collection) and ephemeral (text, video, music etc.). Music, the presentation concludes is often considered as a “mere” satellite, but with thought and careful design there is no reason why music can not also become the narrative kernals of interpretation.
COHEN, A. J. 1998. The Functions of Music in Multimedia: A Cognitive Approach. Fifth International Conference on Music Perception and Cognition. Seoul, Korea: Western Music Research Institute, Seoul National University.
COLLINS, K. 2008. An Introduction to the Participatory and Non-Linear Aspects of Video Games Audio. In: RICHARDSON, J. A. H., S. (ed.) Essays on Sound and Vision. Helsinki: Helsinki University Press.
HAMARI, J., KOIVISTO, J. & SARSA, H. Does Gamification Work? — A Literature Review of Empirical Studies on Gamification. System Sciences (HICSS), 2014 47th Hawaii International Conference on, 6-9 Jan. 2014 2014. 3025-3034.
HARGOOD, C., JEWELL, M.O. AND MILLARD, D.E. 2012. The Narrative Braid: A Model for Tackling The Narrative Paradox in Adaptive Documentaries. NHT12@HT12. Milwaukee.
LAZARRO, N. 2009. Understand Emotions. In: BATEMAN, C. (ed.) Beyond Game Design: Nine Steps Towards Creating Better Videogames. Boston MA: Course Technology / Cangage Learning.
PANKSEPP, J. A. B., L. 2012. The Archaeology of Mind: Neuroevolutionary origins of human emotions, New York, W. W. Norton & Company.
SHIRES, L. M. & COHAN, S. 1988. Telling Stories : A Theoretical Analysis of Narrative Fiction, Florence, KY, USA, Routledge.
SYLVESTER, T. 2013. Designing Games – A Guide to Engineering Experiences, Sebastolpol, CA, O’Reilly Media.
In the afternoon Graham Festenstein, lighting consultant, kicked off a discussion about using lighting as a tool for interpretation. New technology, he said, especially LED, presents new opportunities, “new revolution” in lighting. It’s smaller, with better optics, and control. And also more affordable! He used cave paintings as an example. Lighting designers could take one of three approaches to lighting such a subject: they might try and recreate the historical lighting which, for a cave painting, would have been primitive indeed, a tallow bowl light, revealing small parts of the painting at time and with an orange light; it’s more likely, given the needs of the visitor, that they might go for more wide angle lighting, revealing the whole of the painting at once; or, they might light for close up inspection of the work, to show the mark making techniques. Traditionally, a lighting designer would have had to chose just one of these approaches. But with the flexibility and versatile control of modern lighting technology, we can do all three things – caveman lighting, wide angle panorama, and close up technical lighting.
Graham’s presentation was not the strongest. heexplained that he was sceptical about LED lights at first pilot as a sceptic. He recalls a visit to a pilot project at the National Portrait Gallery. His first impressions were disappointing, but then he realised that what heith missed about the tungsten lighting to the way it it the gilded frames, and the the LED lighting was better serving the pictures. Then he went on to talk about colour, how the warm lights of the Tower of London’s torture expedition undermined the theme, but the presentation overall was somewhat woolly.
Zerlina Hughes, of studio ZNA, came next, with a very visual presentation which I found myself watching rather than taking notes.It explained her “toolkit” of interpretive lighting techniques, but I didn’t manage to list all the tools. A copy of the presentation is coming my way though, so I might return with more detail on that toolkit in a later post. One of her most recent jobs looks great however, and I’m keen to go. Say You Want a Revolution, at the V&A follows on from the Bowie show a year or so ago, but with (she promises) less clunky audio technology. I want to go.
Jonathan Howard, of DHA design, explained that like Zerlina, “most of us started as Theatre designers.” I (foolishly, I think, in retrospect) passed up an invitation to do theatre design at Central St Martins, and I think I would have been fascinated by lighting design, if I had gone, so I might have ended up at the same event, if on the other side of the podium. Museums audiences today are expecting more drama in museums, having experienced theatrical presentations like Les Miserables, and theme parks etc. I was interested to learn that in theatre, cooler colours throw objects into the background, and warmer colours push them into the foreground. This is apparently because we find the blue end of the spectrum more difficult to to focus on. In a museum space, he says, you can light the walls blue so that the edges of the gallery fall away completely. But he did have a caveat about using new lighting technology. Before rushing in to replace your lighting with LEDs and and the modern bells and whistles, ask youself:
Why are we using new tech?
Who will benefit?
Who will maintain it?
Who will support it?
Kevan Shaw, offered the most interesting insight into the State of the Art. He pointed out that lighting on the ceiling has line of sight to most things, because light travels in straight lines (mostly), and we tend to point it at things. So, he said, your ligthting letwork could make a useful communications network too. He wasn’t the first presenter to include and image of a yellow centered squat cylinder in their slide deck. And they spoke as though we all knew what it was. I had to ask, after the presentation, and they explained that it was one of these. These LED modules slip into many exisiting lamps or lumieres. They are not just a light source, but also a platform for sensors and a communications device. Lighting, Kevan argues could be the beachhead of the Internet of Things in museums.
He briefly discussed two competing architectures for smart lighting, Bluetooth, which we all know, and Zigbee, which you may be aware of through the Philip’s Hue range (which I was considering for the the Chawton experiment). He also mentioned Casambi and eyenut, I’m not sure why he thinks these are not part of the two horse race. He argues that we need interoperability. So I guess he’s saying that eventually the competing systems will eventually see a business case in adopting either Bluetooth or Zigbee as an industry standard.
With our lightbulbs communicating with each other, we can get rid of some of our wires, he argues, but it needs to be robust, reliable. And the secret to reliability is a mesh networking, robust networks for local areas. Lighting is a great place for that network to be. That capability already exits in Zigbee (so I think zigbee is what I should be using for Chawton), but its coming soon in Bluetooth. And I think Kevan believes that when it does, Bluetooth will become the VHS of the lighting system wars, and relegate Zigbee to the role of Betamax.
But the really exciting thing is Visible Light Communication, by which the building can communicate with any user with a mobile device that has a front facing camera (and the relevant software installed. He showed us a short video of the technology in Carrefour (mmm the own-brand soft goat cheese is delicious).
The opportunities for museums are obvious but, he warns, to be effectively used, museums will need resource to manage and get insight from all the data these lighting units could produce in resource. Though he says optimisticly to his fellow lighting consultants, “that need could be an opportunity for us!”
Finally we heard from Pavlina Akritas, of Arup, who took the workshop in the direction of civil engineering. Using LA’s Broad Museum as an example, she explained how in this new build, Arup engineered clever (North facing) light-wells which illuminated the museum with daylight, while ensuring that no direct Los Angeles sun fell directly onto any surface within the galleries. The light-wells included blackout blinds to limit overall light hours and photocells to measure the amountof light coming in and if neccessary, automatically supplement the light with LEDs. She also talked briefly about a project to simulate skylight for the Gagosian gallery, Grosvenor Hill.
My son pointed out that one thing the game has, especially over its Niantic stablemate, Ingress, is the Pokemon brand, which two or three generations have grown up with since the mid-nineties. Speaking as someone who didn’t grow up with Pokemon however I could not believe this was the only reason for its success.
A huge difference from Ingress is ease of entry. As my survey a couple of years ago may have indicated (though I could not disprove the null hypothesis) here-to-fore locatative games have only held any interest for Hard Fun (otherwise Hardcore) gamers. Pokemon seems to be the first truly casual locatative game (though some might give that honour to Foursquare, I don’t think it was much of a game).
So lets run it though the model:
Leaderboards – Although Pokemon Go doesn’t have a leaderboard as such, it does have Gyms. Just today with my 11 year old son’s advice I managed to (very temporarily) take over a local gym from some quite high powered Pokemon of an opposing team. So a few minutes after that, I was indeed at the top of a very local leaderboard.
Badges – There are a huge variety of medals you can win for achievements like, for example, collecting ten Pokemon of a particular type.
Rewards – Pokemon Go has LOADS of rewards. For a start, visit a Pokestop, and spin the dial and you will acquire a randomly generated reward of Pokeballs, Eggs, and Potions etc, all of which will be useful in the game. Take over a Gym (as I did this afternoon) and you can claim a reward of 10 Pokecoins every day that you keep the Gym under your control. Capture a Pokemon, and not only can you add it to your collection, but you are also rewarded with Stardust and Evolution Candy. Every time you go up a level you also earn rewards such as new equipment.
Points and Levels – To level up, you need to earn experience points, which you get for pretty much everything you do, collecting Pokemon, especially new types, spinning Pokestops, hatching eggs, earning badges, evolving Pokemon, battling in gyms, etc.
Story/theme – There isn’t much of a story inherent within the Pokemon Go game, but players who have been brought up on the other computer games and TV series, will know of quite a complex backstory. However, not knowing this story does not seem to be a disadvantage to players. Story knowledge isn’t essential to play, and the lack of story within the game seems to attract (or at least not be a barrier to) players of all generations, many too old to have been captured by the original Pokemon game. My son also points out that as you play you do procedurally generate a story of your own trainer avatar, even if that is only in your head, as Sylvester describes.
Progress – As you go up in level, you do get better equipment, and are more like to catch Pokemon with higher combat power, and you are more likely to encounter rarer Pokemon.
Feedback – The game is casual enough that you don’t need to be looking at the screen all the time, but because the game does not allow you to put your device into sleep mode, you end up holding it, waiting for the tell-tale buzz of nearby Pokemon.
Spectacle and Environment – The graphics and Augmented reality are not very sophisticated, but they are fun. Two things make it so. One is that creatures that only exist in fiction now appear in our real life world. The other is that they can (with some luck and an little movement of the screen) appear in amusing places (on your knee, in your dinner or drink, on your friend’s head), and if they do, you can take and keep a photo.
Challenge – There isn’t much skill based challenge in the gameplay. Capturing rare Pokemon, is more of a feat of luck than skill. There is a real-world challenge of sorts though, and that is to walk around, which is the only way to hatch eggs. Some eggs only require two kilometer walks, but other more rewarding eggs require ten kilometres.
The game lacks (or doesn’t make the most of) a number of emotional triggers:
Music – My son likes the music, but I turned it off early in my play. The music isn’t a very sophisticated feedback generator. One track plays pretty much continuously, and the only changes are for evolution cut-scenes (my boy likes this track best) and Pokemon encounters.
Insight – There is very little learning through play. My son teaches me most of what I need to learn, and he has leaned most of that through YouTube.
There is no Threat or Sex (even when you capture Male and Female Nidoran), and no real character arc.
So, given the affordancies listed above, we can predict which emotions players will be feeling: playful Amusement (from humorously placed AR Pokemon); the social emotions Fiero and Naches (because though the gameplay isn’t inherently social there are enough players currently on the streets for conversation; advice and insight; and even a degree of cooperation; to take place); the seeking emotions, Excitement and Curiosity (especially when find new types of Pokemon); Frustration, a rage affect (when Pokemon randomly break out of your Pokemon); and some degree of Care (from nicknaming, nuturing and powering up your stable of Pokemon).
And let us not forget the Panic/grief, when nothing makes your phone buzz -you are out of mobile reception or have a weak signal, and especially when your phone battery is running low!
The Interpretation Game 