Getting EMOTIVE in Athens

At the beginning of November I attended a workshop hosted by the EMOTIVE project (https://emotiveproject.eu/), an EU-funded heritage project that aims to use emotional storytelling in digital contexts to change how people experience heritage sites. Halfway through their three-year grant, they invited a group of outside colleagues to help them discern how to best evaluate the different digital & virtual experiences and tools that they developed to date. Over the course of two action-packed days, we tried out a variety of prototypes that had been developed for heritage sites in Scotland and Turkey, and had some wide-ranging conversations on how and what to measure. It was, in many ways, a great follow up to the AAM/Knight Foundation workshop on immersion in museums that I attended in October and wrote about for AAM and highlighted many of the issues in contemporary museum experience design that I keep running into.

Before I dive in, though, I want to single out the EMOTIVE team for being open about their process and progress, and being willing to share works in progress in all their messy incompleteness. It’s always nerve wracking to invite outsiders in to poke at your work, but in my experience, the results more than make up for the momentary discomfort. It was on the whole, a sterling example of the kind of collaborative fearlessness that I would like to see much, much more of in museums. Mad respect to Maria Economou, Sara Perry, Maria Roussou, and the whole EMOTIVE team!

There was a ton of interesting conversation that I won’t try to do justice to capturing. Instead, I want to focus on four themes that arose for me while I was was there:

• the perils of storytelling,
• the power of relevant touch,
• how socialization can profoundly impact learning, and
• the ever-slippery world of immersion in museum contexts.

Storytelling, sensory disconnect and emotion

A common theme with several of the prototypes we tried revolved around constructing emotional narratives with objects in the collection, and assembling objects to illustrate longer stories. After all, an object without a story is a just a thing. As Susan Stewart writes in “On Longing”, her brilliant study of collecting, collections and souvenirs, “The experience of the object lies outside the body’s experience – it is saturated with meanings that will never be fully revealed to us.” Exploring object stories and constructing a narrative is of course a staple of modern museology, but one technique EMOTIVE was exploring was how much of that narrative could be manufactured, given the minimal amount known about most archaeological objects. We need the stories to connect to the object. Compelling stories well told are powerful tools, but they’re not enough on their own.

In three different prototypes, I had varying degrees of engagement with the content and objects presented. The most significant barrier I found had nothing to do with the narrative and whether it was fictionalized, but the extent to which I was using all my senses to focus on one thing, or being asked to look at one thing while listening to another. This disconnect had the effect of pulling me out of the narrative. When the objects and the narrative reflected each other, I had little trouble. One prototype used a series of objects from the Antonine Wall in Scotland to tell a story of a Roman soldier faced with a difficult choice. The Romans abandoned the wall and retreated south. Many of the soldiers who built the wall had been there for years. Some had started families they couldn’t take south with them. The tour asked visitors to put themselves in the place of one of those Romans and decide whether to stay with their family, or stay with their comrades. The objects chosen to represent the characters in the story helped paint a fuller picture of the time and place.

Another prototype was a walking tour of the Athenian Agora. The team had constructed a detailed story of daily life in ancient Athens, centered around an enslaved person and the family he served. There were multiple perspectives, a variety of voices, and dilemmas enough to hook listeners. But there were hardly any connections to the landscape I was walking through. I found myself standing in front of the ruins of the Tholos, listening to a story that told me nothing about the ruins I was looking at. The cognitive burden of trying to find something to look at that pertained to the story in my ears – and failing – plucked me out of the story. Why was this building circular when all the others seemed to be rectangular? What might this place have looked like in Pericles’ day? I never had those questions answered, and they stopped me from being carried away by the story.

This is not a knock against storytelling approaches. There is undeniable impact and staying power in a good narrative. The great grand-daddy of all successful audio tours, the Alcatraz tour developed by Antenna Theater back in the late 1980s (?) was mentioned a couple of times as a successful example of place-based storytelling. There’s a whole body of research to support the assertion that storytelling works to change the listener in ways that other delivery styles don’t. Narrative transportation theory asserts that people who are deeply engaged with (or “transported”) by a story, find themselves changed by the story. Their views tend to align more with those of the story than they did before. This kind of empathy generation is at the heart of experiences like Alejandro Iñárritu’s “CARNE y ARENA” VR installation that places viewers in the shoes of migrants attempting to illegally cross the US/Mexico border. This persuasive power is what EMOTIVE is trying to harness. They want visitors to heritage sites to care more about cultural heritage and feel a personal stake in its preservation.

If you change the word “transportation” to “transformation”, you’ll find a lot of resonances with the missions of many cultural organizations, not just those dealing with history. Storytelling is a vehicle for making emotional experiences that are more memorable and compelling ways to get visitors to learn about the content and themselves, as opposed to more traditional interpretive forms.

The power of touch

I’ve long advocated the use of tactile elements in interactive exhibits, even ones that could be entirely screen-based. The experience of engaging the body as well as the brain can make a big difference in visitors’ engagement with an activity. And EMOTIVE is exploring this territory in interesting ways. One ambitious activity we tested was a multi station school group activity where students worked in small groups to explore several houses at the archeological site of Çatalhöyük, Turkey. The activity seeks to get them to model some of the behaviors recorded in the archaeological record, most notably the dispersal of personal objects across the site, as opposed to being clustered in familial groups. This is one of the key pieces of evidence for archaeologists determining that Çatalhöyük’s society was an egalitarian one, so it’s kind of a big deal for students to grasp this.

Everyone in the group was given a quick Buzzfeed style personality quiz and assigned an object based on the result. These objects were small 3D prints of actual Çatalhöyük artifacts, and we were instructed to personalize “our” object with a variety of craft supplies. Then as our group progressed through the activity and learned about different aspects of Çatalhöyük society, we were instructed to leave or take objects at each station for various reasons. With multiple groups going through the activity at the same time, you could quickly see how objects travelled from place to place and group to group. And the conversations our group had about which person would leave or take an object and why was remarkable. Even with little scaffolding on how to decide whether to leave an object, and whose object would get left, our group very quickly started having very personal conversations about “our” objects and what they meant to us. We might not have learned enough about the original object the scans were created from to suit me, but knowing they were “real” objects and we were using them in a context similar to the original use context was enough for me.

Socializing with chatbots

We tried out two different chatbot activities that the EMOTIVE team were developing; a school group activity that was a follow up to the tangible activity prototype, and Facebook Messenger-based chatbot that challenges users to examine (and re-examine) their beliefs. I confess I didn’t have high expectations for either. Most of us have probably interacted online with bots, knowingly or unknowingly, by now. To date, I’d found my interactions with museum bots to be less than engaging, so I was surprised at how much I got out of these bot experiences. I think the reason for this is twofold; in the first case, it was that the bot conversation was designed as a social experience; we interacted with the bot as a small group. In the second case, they used a class of bot that does more than respond to inputs with information like the typical info bot.

The social prototype used a Snatchbot-based bot to stimulate students to state their beliefs, discuss them in a group, share that with the bot, and then in the face of confrontational information from the bot, restate their beliefs and see if they’d changed. Gutsy stuff. We split up into small (3-5 person) groups and were directed to appoint people to take on the tasks of typing our answers, and reading back the bot’s responses. These gave enough structure to our otherwise random group so that we were able to navigate the experience without further scaffolding. The act of having a typist and needing to agree on what our responses should say so the typist could enter them provided a natural vehicle for discussion and reflection. And quite a bit of laughing and learning about each other. I think everyone in the group was surprised when it turned out our response to the question did in fact change after conversation with each other and provocation from the bot.

The Facebook bot, called ChatҪat, is a kind referred to as a protest bot by Zack Whelan or bot of conviction by Mark Sample. According to Sample, “Bots of conviction share at least five characteristics: they are topical, data-based, cumulative, oppositional, and uncanny.” Read either article to get a better sense. Both are good. This bot mixed the functionalities of both an info bot with a protest bot by sequencing the interaction and conversation to include knowledge sharing (info bot) with asking questions to probe visitors’ convictions (protest bot) about whether we’d want to live in an egalitarian society like Çatalhöyük’s.

The most fraught part of the whole experience for me was the Facebook connection. The upside for the team was the prevalence of Facebook use in their target demographic, and the tools available to stand up and launch a chatbot. Strong inducements in times of austerity. Meeting people where they already are rather than making yet another boutique destination for them to (hopefully) visit is a philosophy I regularly espouse. Does the world really need another app, or will a good mobile website do the trick? But, nowadays, especially with continuing use and misuse of AI by platform providers, a question that needs to be asked is “Can we trust you not to do something bad with the data you collect?” I know that the team wrestled with those questions and decided to try out Facebook. It’s a conversation we should be having more and more often, since we rely so much on platform providers.

I am woefully unread when it comes to AI, so I was happy when someone at the event recommended a book by Sara Wachter-Boechter, on the ethics embedded in AI. I haven’t ordered it yet, but it’s on top of my list: Technically Wrong: Sexist Apps, Biased Algorithms, and Other Threats of Toxic Tech. The American Alliance of Museums also recently hosted a convening on “Museums and new intelligences” that produced a great primer on the subject of AI. Pick it up!

Immersion in VR environments

On Day Two, we went ot the University of Athens and tried out a virtual reality activity that let you explore a building in Çatalhöyük. As with most VR installations, navigating (or trying to navigate) inside the VR was neither easy nor intuitive. Using controllers in both hands and trying to click the correct button at the right time is challenging for even simple activities like moving around. I’ve done enough of them that I can do it, but I know it’s going to be a stretch to expect museum goers to pick up the interface without substantial scaffolding. That said, it is still valuable to be able to feel embodied in a space and able to move around in it. I may never get to Çatalhöyük, but I feel like I have a sense of the place I wouldn’t have just through looking at images.

As with all the EMOTIVE experiments, the developers had tried to pack a lot into every activity. The VR experience also included a human avatar dressed in period clothing in one of the rooms. I found the contrast between the photorealism of the room, and the relatively low-polygon avatar to be jarring. I had a similar response to The Enemy at MIT Museum, where the difference in fidelity between the avatars and the surroundings made them seem even less “real” than they might have seemed in isolation. In fact, low res simulations I think can be more attractive, when there is opportunity to interact. So, while I enjoyed this particular experience, in general, I’m still on the fence about most VR implementations in museums. It usually feels like a solution looking for a problem.

That is starting to change, though. I wrote previously about the Jewish Museum’s use of VR in the Pierre Chareau exhibition. The Tate’s Modigliani VR is another great example of museums getting VR right. VR experiences like The Enemy and CARNE y ARENA, mentioned above, has been drawing rave reviews for its ability to provoke empathy. And VR has also been deployed as tool to attract audiences that don’t see themselves as museum goers. I find Shelley Bernstein’s work at the Barnes on using VR to engage audiences to be deeply interesting. UPDATE: the next article on the project just dropped on Medium.

Conclusion

That gives you a flavor for what our time in Athens was like; action-packed and intellectually stimulating and exhausting. It was very interesting and encouraging to see how many different kinds of approaches the team were testing, and how solicitous they were of outside opinions to inform their evaluation. One thing that struck me after the fact was how much EMOTIVE seemed to have tapped into the zeitgeist of museum experience in 2018. Digital storytelling? Check. AI? Check! Immersive experiences? Yup! All they needed was a gamified experience and they would’ve won Museum Buzzword Bingo. Yet, having said that, I came away impressed with how many of their prototypes seemed to be onto something useful. And isn’t that what a good research project is supposed to do; show fruitful paths for further exploration? 

I look forward to seeing what the project achieves when it concludes.

Postscript: Putting all the pieces together

The EMOTIVE workshop was the latest in a series of events where terms like emotion and presence keep cropping up in conjunction with one another. One of the notes I wrote to myself on the flight home from Athens was “Gaming and immersion, where do they fit in? Is it the way games produce immersion and a sense of presence? Does that have something to do with producing a flow state?”

I also happen to be partway through Kraut and Resnick’s classic “Building Successful Online Communities: Evidence-based Social Design” which, though several years old now, is a solid piece of scholarship. In Chapter Two, they map game design principles (based on Sweetser and Wyeth’s Gameflow model) onto Csikszentmihalyi’s flow criteria in way I hadn’t seen before. And the game design principle, “Make players feel emotionally involved in the game” is linked with the flow criterion of immersion, “a deep but effortless involvement that removes awareness of the frustrations of everyday life.” Seems obvious now that I’m writing it, but seeing it in writing really closed the loop for me in how these ideas can relate to each other.

I’m going to have to write a book to sort this all out…