Research on Using AI, AR, and Narrative to Customize Visitor Experience: An interview with Maria Roussou about the CHESS Project

At April’s Museums and the Web conference in Chicago, I heard a fascinating presentation by Maria Roussou, talking about the CHESS Project, a research study exploring how museums could personalize a visitor’s experience before they arrived and throughout their visit. The strong narrative component and use of digital media reflected many of the design elements we are incorporating in our upcoming Microrangers game, so I just HAD to learn more about what she’d have to say about the power of user-centric design processes, storytelling, and creating personas, and more.

 (Check out Maria’s slides from her presentation here)

Welcome to Mooshme. Would you please introduce yourself?

I am Maria Roussou. I have a company called makebelieve. We do interactive exhibits, mostly for museums but also for informal education. I also teach at the University of Athens, Human Computer Interaction in Computer Science, and Digital Technology in Museums in the graduate program of Museum Studies. I also teach User-centered design & Web Design at the American College of Greece.

And I am an evaluator and reviewer for the European Commission on research projects. The CHESS project, which I presented here at Museums on the Web, is one of these European funded research projects.

Screen Shot 2015-05-08 at 3.19.48 PM

So please tell us about it. Let’s start by understanding all the partners involved.

It’s a project that ran from 2011 to 2014. It involved two universities – the University of Athens (their Computer Science Department) and the University of Nottingham (their Mixed Reality Laboratory, famous for their human-computer interaction work); two companies from France, DIGINEXT, who developed the authoring tool, which allows authors to put together the narratives, and Real Fusio, who worked on the media content development – 3D models, videos, games, things like that. There was a research center in Germany, the Fraunhofer, who worked on Augmented Reality; last but not least, there were two museums – the Acropolis Museum in Athens, Greece and a museum in Toulouse, France called the Cité de l’Espace (the City of Space) – it’s a science center about space and its conquest. 90% of the visitors at the Cité de l’Espace are children, so their education department was involved in this project.

Before we get into the details of what you all created, let’s delve into the main research focus of this project. What did you set out to learn?

The research focus was to enhance adaptive experiences in museums, adaptive digital experiences. In the beginning storytelling wasn’t so much what was sought – it was mostly personalization and adaptivity. So this was a computer science research project. The idea was to develop the tools, the artificial intelligence if you will, of the programs that would support such a cultural experience for visitors, one that is adaptive and personalized to their needs and wants.

So how might a visitor experience CHESS?

The visitor might begin by preparing at home. They would go online and answer a quiz, so that this system can understand more about this particular person, demographic information – age and whatever – but also things like interests and likes; whether, for instance, they like to read the political column in the newspaper or the comics. This kind of information can inform the system about the way they want information presented to them and the duration of that information, the length of it and all of this stuff.

But if they don’t prepare beforehand, they could do this quiz in the museum on a tablet that the museum can give to them or on their own phone or tablet. The system creates a profile for them which is matched underneath with one of the predefined personas that we have made; for the Acropolis Museum we had made 6 and for the Cité de l’Espace we had made 2.

So that’s how the system works at the beginning, it tries to solve what we call the cold start problem. Cold start means that the visitor comes in and we know nothing about them. How do you actually start a personalized experience if you know nothing about this visitor.

So I arrive at the museum there is now a persona associated with me?

Yes, but you don’t see this persona. This persona is a tool that the design team of the CHESS project created with the museum staff. We can’t personalize to each and every individual; everybody is different, so we had to make these personas.

Screen Shot 2015-05-08 at 3.24.53 PM

So with a persona associated with me, one that I am not aware of, it means the museum will treat me in a very different way than someone else, in a way more customized to my needs?

It means that you will begin your museum visit with the experience that the museum has designed for the persona that matches you the most. So you come in, you do this quiz, this system takes the answers that you provide and sees what’s the best match with the persona, and then takes you through the story that the museum has already prepared for that persona.

Such as?

So, for example, if you are interested in everyday life in Athens and you are at the Acropolis Museum, there is a story about a woman who talks about what she did, how she was married, how her son was killed during the Persian War, how she offered a statue to Goddess Athena (who is the goddess protecting Athens), and so on.

And this is a story that has a beginning, a middle, and an end, but it involves different exhibits on the museum floor, ending with that offering, the statue that this woman who is the narrator of this story offered to Goddess Athena. This woman actually existed, we can see her name inscribed on the little statue on the offering, but you don’t see that until the end of the story.  So if you are interested in this, if you are matched with that persona, you will get this story.

And the narrative is where? On my phone, speaking? Is it a video?

Yes, she is on your tablet or a phone. There is a sketch created of this woman, she looks really old, she was only 50 years old but at the time 50 years old was actually quite old. And you can hear a voice of an actor, an actress in this case, who is telling the story. So you see the sketch of the person, the avatar, if you will, of the narrator and you can hear her voice talking to you.

Is it just like I have a guide leading me along guiding me to interact with the exhibit?

When this person says that she was married, for example, she points you to the various pottery that depict marriage and love and things like that. If you are not interested you can skip this part and the story will resume or it will start another branch that you are maybe more interested in. You have control on the interface, so you can skip or continue or choose a branch in the story – the personalization system in the background records these choices and will automatically take you on certain paths that it assumes interest you more based on your previous interaction with the system. So the internal bits and pieces of the story change as you go along, but the general structure of the story has a beginning, a middle, and an end.

Can you say a little bit more about how this system facilitates people’s observing or interacting with these objects?

Okay, so this has been a really difficult part. When we evaluated our initial iterations, we saw that people tend to look at the screen very often, for most of their experience. And some people are bothered by this. The younger generation, teenagers and kids, are very engulfed in looking at the screen and doing things on the screen, so they don’t actually observe the physical exhibits in the space. This was a real problem, of course, for the museum. So what we did in subsequent iterations is we tried to include in the narrative ways of looking at the exhibits, and requiring observation before you can go on.

For one of the stories – about Theseus, the Hero of Athens – you have to actually look at parts of the exhibits. You have to decide whether these exhibits are worth adding to Theseus’ army so he can have enough strength to be able to fight King Minos’ army after exiting the Minotaur’s labyrinth. In order to do that you have to collect the appropriate elements from within the exhibit and add them to the armory. We tried these tricks to get people to look, to observe more carefully. I can’t say that it worked fully but it was significantly better than what we had done before. We had not thought about this problem at the beginning.

What role did augmented reality play supporting those acts of observing?

The statues in ancient Greece were all colored. This is something we don’t really see when we are in the museum. We all think that they were white marble, but that was not at all the case. They had really bright colors – blues and reds and golds. What we have done with the augmented reality application is laser scan these statutes and create 3D models. Then an artist has applied the marble material on them and then drawn on the digital marble based on what the conservators and the chief archaeologist (i.e., Professor Pandermalis, the President of the Acropolis Museum) told her. So when you raise your tablet in front of the sculpture you can see, super-imposed, the same sculpture, colorized.  So this really enhances the experience. Visitors can also tap on various parts of the model through their tablet and see or hear about details of those parts. They explain about what the artist at the time was thinking when colorizing the statues, and so on.

How did the teams find the balance between the fictional story that was being created to contextualize the experience and the reality of what was in the museum and its history that needed to be observed and respected?

The work started from the accurate text that the archeologists provided, and the archeologists don’t want too many detours from this text. But this text may not work, as is, in terms of a narrative script — you cannot make a really powerful storytelling experience out of it; visitors that don’t usually come to the museum find this rather boring. The first evaluation showed this and so we brought in a storyteller, a fiction writer, who had nothing to do with that content. He came in with a fresh eye, took this text and converted it into make stories out of them. Sometimes that took us to the other extreme.

Ooh, like how extreme?

For example, he made a story where a journalist from the future, reporting for the Journal of the Ancient Worlds, lands with the help of his time machine into the museum and wants to learn more about the past (i.e., the year 2014, which is ancient history for his time). So, okay, this was a nice wrapping, and the archeologist liked it as the context, but when it got into the details our storyteller made many mistakes, because he took the information and, in the process of making stories out of it, changed some of it. We were kind of worried about achieving a balance between the original, factual, text and fiction. And it took museum staff a while to actually understand also the reason for changing the descriptive text so much.

It sounds like there was a process of going back and forth, between academic content and creative storytelling.

It was. I just seemed like it never stopped. And we changed a lot. We evaluated again and again. There were many iterations. We kept elements that worked really well and the archeologists felt comfortable with… and one of those elements was humor. Humor worked really well with all visitors. For example, there is a part of the story that talks about the significance of horses in ancient Athens. And so the story says, “We horses were very important in ancient Athens, we were the fastest, just like a flashy convertible of today.”  So there were plenty of metaphors and humorous examples given relating to the present to illustrate some of the important facts of the past. We tried to find that balance after many iterations and had to also calm our storyteller down a bit.

Where does the project stand?

This is a research project, which means that it hasn’t actually launched as a product and it’s not used in the museum, at least not yet. Ideally it should have been but there were so many issues in the process and it’s so difficult to create these stories because of the many iterations and the work that’s involved. It needs this extra step of a company or somebody taking it, refining it, and making it more robust so it can be deployed in a more mass market.

What would you say then at this stage in the project are some of the key takeaways, given the initial research questions about how to use backend systems to prepare people before they came into the museum and then give them more customized, personalized experiences that speak to the way that they learn and engage in museums.

I think one big takeaway is the process design and the methods used to design these kinds of experiences. The participatory design methodology, which included the museum professionals and the creative industry professionals that designed these experiences, from the beginning of the process, was very informative and can inform later endeavors of this kind.

The visitor studies world has talked about different visiting styles for a long time, but actually making, putting a face on visitors, through the personas, really helped the design team to speak the same language. Usually, when museum people will talk about the visitor, they will talk about a general form of visitor but they may have somebody completely different in their mind. So when they come together to collaborate, everybody is thinking of their own user (the so-called “elastic” user) and they are not speaking the same language. Personas may look like a simple tool but, as a design method, it really helps everybody talk about the same user. So the interdisciplinary design team talked about Natalie or John or Jim or whoever and they knew exactly what Natalie or John or Jim represents, wants and likes.

It sounds like one of the main lessons that was learned in the process was the importance of a rich user-centric design process?

Yes, the user-centric design process. Exactly. And then of course that storytelling really works. I mean, at present, people like it; they come into the museum and they are immersed in an experience. They pay attention to something that they wouldn’t see before. So yes, it works especially for bringing in visitors who would not come to the museum otherwise.

Maria, thank you so much.

Screen Shot 2015-05-08 at 3.24.49 PM


Thank you!

About Barry

The Associate Director For Digital Learning, Youth Initiatives at the American Museum of Natural History.
This entry was posted in Interviews, Practice and tagged , , . Bookmark the permalink.