Our Video Bridge – using a telepresence robots to connected with Canadian First Nations – in the New York Times

Today the New York Times features an article, Let a Robot Be Your Museum Tour Guide, that explores the emerging role of telepresence robots within museum halls. Please check it out. However, if you only have a few more seconds, please check out the few lines about our work here in NYCL

“The American Museum of Natural History has also used robots powered by distant tour guides: indigenous people from Haida Gwaii, an archipelago off the north coast of British Columbia, for an exhibition involving their culture.”

Posted in From My Work | Tagged | Leave a comment

Jan Update: Gamifying Science Data

This post is part of an ongoing monthly series of posts that will focus on our current efforts in the Museum’s Science Bulletins team to create and test prototypes of Hall-based digital interactions using AR and VR using our scientists’ digital science data, and to share some of the lessons we learn along the way.

For the last few months we’ve been exploring what happens when visitors have the opportunity to interact with science data, using digital tools, in our Museum halls. In January we aimed to explore what happens when that interaction comes in the form of a game.


Does gamifying the interaction change how visitors experience or feel about the science data, and if so how? And does it change their interactions with the people around them?

To collect some data we created Escape the Planet. Inspired by the rise of both escape rooms and their educational applications, our goal was to create a 20-30 minute experience, in the Big Bang Theater (a small room off the lower level of the Hall of the Universe) focused on astro content. The narrative involved the crew of a crashed ship, on Mars, struggling to survive and contact a rescue ship. To do that, a series of puzzles needed to be resolved, which ended with the delivery of a Communications Crystal (an amethyst purchased in our Museum store) to the game facilitators.

The two puzzles each required their own Hololens to resolve. One used the VR Constellations we tested in December, which put the viewer within a real star field, with a number of constellations that can be observed from different angles. The second used a solar system simulation. The idea was that each would need to be manipulated in some way in conjunction with work being done by others in the room. In other words, neither the person in the Hololens nor those working with the physical materials would have complete knowledge – only by collaborating together could the puzzles be solved.

Below is a short video giving a sense of what it was like when we ran a play session with a local school to test the prototype.

(and LOTS of great photos can be viewed here)

Some (and far from all) of the key lessons we took away from this round of prototyping:

  • GAMIFICATION can provide motivation for visitors to engage with science data. Initial evidence suggests that interacting with the digital science data was often driven by intrinsic motivation generated through game play.
  • GAMIFICATION can increase visitor interaction. Hololens users playing Escape the Planet regularly maintained social contact with the rest of their group, as required by the design of the game, and appeared to have done so more often and with more intensity than when the same data was offered to the visitors in the Museum outside the context of the game.
  • GAMIFYING science can be an effective Hall-based pedagogy. Half of those surveyed report that Escape the Planet increased their interest in the science content, and the vast majority (15 of 16) expressed a preference for learning about astronomy in museums through games like Escape the Planet over more traditional methods (e.g. interacting with content, watching a movie, reading text or looking at an image on the wall).

(If you want to learn more, check out this earlier post on Youth as Co-developers in a Process to Gamify Science Data.)

Come back soon to see some of what we did, and learned in March.



Posted in From My Work | Tagged , , , | Leave a comment

My Weekend With Mixed Reality


My son playing with a Hololens

I’ve had my conversion moment – I now believe in the power and potential of mixed reality. In this post I want to explain how it happened and what it means for the future of museums.

I first wrote about the concept of “mixed reality” last summer, after attending the Games For Change conference, and after reading a Wired Magazine cover article. At the time I wasn’t sure if MR was just the latest marketing term or if there was something really there there. I think it’s worth quoting in whole, as you can see me trying to get my head around it:

The May cover article in Wired Magazine, “The Untold Story of Magic Leap,” by founding executive editor Kevin Kelly, is a crash course in the latest and greatest in virtual reality (and its relatives). … Kelly brings in the term “mixed reality” (or MR for short) to describe a number of devices that a user wears like a pair of super goggles and, more often than not, while tethered by hardline to a computer. The best I can figure is that MR is augmented reality through a worn device. When I saw Graeme Devine, of Magic Leap, present on his company last week at the Games For Change Festival, he, too, used the term MR (which is where Kelly might have picked it up). Devine defined it as “the mixture of the real world and virtual worlds so that one understands the other.” The power of MR, as I came to understand it, is that AR typically uses one discrete target (like a coin or sticker) to trigger an augmented experience while MR maps the shape of a room and uses that entire map as the canvass upon which to paint. In other words, with MR, the entire surrounding space is in play for layers of augmentation (an exciting prospect for museums).

That excitement I referenced at the end was more aspirational. I certainly wasn’t feeling it. Sure, I saw the value in a device understanding the space around it but, okay, then what? It sounded like a nice feature, but for a limited use case.

Then I started my new position at the Museum. I was no longer reading about Google Tango, Microsoft’s Hololens and the HTV Vive – I was now developing in them and could play with these tools whenever I wanted.

Our first prototype, as a team, brought an AR Shark into our halls using a Hololens:

It was certainly cool watching a physical shark hanging over my head turn into an interactive shark skeleton. But even then I thought of the Hololens strictly as an augmented reality device – a way to bring a single virtual object into the space around our visitors. I still wasn’t getting it.

Over the Thanksgiving weekend, I took the Hololens home. I played with it all weekend, on my own, with my kids, and with my wife. I used the apps (if that’s the right word for them) currently available for download. Some are short experiences you can play in a few minutes while others are designed to last for hours. As we transformed our home into a palette for mixed reality, I finally understood how MR differs from both AR and VR. More importantly, I fell in love with the experience.

First, a bit about Hololens. The device maps Continue reading

Posted in From My Work, Theory | Tagged | Leave a comment

A Suggested Model for Future-proofing Digital Interactives

Last fall, after my experience observing visitors interacting with biological science data through an augmented shark, it got me thinking about how we could learn about future museum visitors by develop cutting-edge prototypes for today’s visitors. Clearly, the tools are going to change, becoming more powerful and ubiquitous. And the digital literacies of our visitors will increase over time as a result. So if the tools will change along with visitor’s abilities to use them, how do we future-proof the lessons for tomorrow that we can glean today?

That lead me to consider the following. Please hang in there with me – I’m still working it through, all these months later, and sharing it with you, in part, to see how it might align with how others approach this topics. So please consider this the start of a conversation.

So while the tech will sure change, and visitors will become more capable integrating digital media into their daily lives, there is one thing that might not change: their personal threshold for how much effort they are willing to put in to learn something new (let’s call this their Learning Curve Threshold) in order to experience something new (let’s call this their Innovation Motivation). When the two are combined, we can discover a visitor’s Zone of Engagement – a window between the point after which the level of innovation is worth the effort yet before an experience requires too much learning.

Screen Shot 2017-01-11 at 4.33.18 PMLet me break this down. All new technology has Continue reading

Posted in From My Work, Theory | Leave a comment

Youth as Co-developers in a Process to Gamify Science Data

How do you support youth learners to participate as co-creators in a rapid prototyping process?


For over four years, our youth programming has invited youth to create with us – card games, mobile games, augmented activity guides and more. Since this past summer, we’ve been developing prototypes that invite visitors to interactive with digital data, seeking insight into the museum of the future. Recently, we ran a four day sprint to see what happens when we bring the two together, and called it Escape the Planet. Throughout we had a team of internal evaluators (graduate student interns from NYU and Columbia, and one high school student intern) track the process of youth claiming their roles as co-developers.



We ran the program with 19 high school students, all day, for four days, on a Tuesday through Friday. In New York City, there’s a test called the Regents, offered for a week in January. On most days most students don’t have any tests, and don’t go to school. This is a perfect opportunity for us to offer new high-risk learning experiences. Enter Escape the Planet.

Escape the Planet was inspired by the rise of escape rooms (see recent post), in which a small group are “locked” in a room and solve problems together, and the recent movement to adapt them for education. Like all of our prototypes this year, we started with the data – in this case slices of our Digital Universe database – and two driving questions: What are the opportunities for increasing the social aspect of interacting with digital data? and How might games and play affect how visitors interact with digital data?

At the start of the first day of the program, led by our staff science educator and an outside digital learning expert, the youth were told that by the end of the week they will have worked with us to create a prototype for a science-based escape room, which they would test with the public. At first blush, this is a rather abstract concept, and most had little way to understand what this could mean, but the challenge had been established: they were going to need to work together to make… something.

Our job on day 1 was to provide them with Continue reading

Posted in From My Work | Tagged , , | Leave a comment

My interview with the author of The Learning Affordances of Augmented Reality For Museum Exhibits on Human Health

Below is a re-blog of my most recent post on DMLcentral, an interview with Camillia Matuk. Read it at its original location, or check it out below.

Augmented Reality and Learning in Museums

When I read Camillia Matuk’s The Learning Affordances of Augmented Reality For Museum Exhibits on Human Health, I knew I wanted to speak with her about AR and learning. Camillia is assistant professor of educational communication and technology at New York University (with a Ph.D. in the learning sciences from Northwestern University, an MSc in biomedical communications from the University of Toronto, and a BSc in biological sciences from the University of Windsor.) She does design-based research investigations to better understand how innovative technologies and learning environments can better support teaching and learning.

Q: Camillia, thank you for joining us in discussion. What came first? An interest in learning, augmented reality or museums?

I’d say it was first my interest in museums, then in learning, and, much later, in augmented reality.

Museums were always a favorite pastime growing up, and were partially responsible for the interest I eventually developed in visualizing science. Although I eventually left the field, my four years as a medical illustrator studying at the University of Toronto and then working at INVIVO made a big impact on me. Still, it was only when I left that field and delved into research did learning come to the fore as a topic of study. As a Ph.D. student, I researched how people made sense of science from visual representations, like the cartoons of evolution that were on display at the Field Museum of Natural History. Later as a postdoc, I studied the role of technology in supporting K-12 classroom science inquiry.

Now, I’m at the Media and Games Network at NYU, and surrounded by people who are creating, studying, and playing with new technologies, including augmented reality. With my research focus on learning technologies, and my background in visualization, an interest in augmented reality was natural.

Q: How did this lead you to your recently published study, “The Learning Affordances of Augmented Reality For Museum Exhibits on Human Health?”

I had begun a project with a neuroscientist and NYU Ph.D. candidate, Oliver Vikbladh, to create an immersive experience of how the brain creates memories of space. The idea was to bring to life Nobel prize-winning research on the brain’s positioning system by allowing people to explore visualizations of place cell activation as they navigated a physical environment.

With Andre Fenton, Ken Perlin, and Jan Plass, we had won a University Research Challenge Grant from NYU to kick start the idea. With artists and former NYU graduates, Adrian Sas and Javier Molina, and with two NYU digital media students, LaJune McMillian and Mahe Dewan, we created a virtual reality experience that was exhibited at Creative Tech Week in 2016.

At the beginning of this project, we considered many different potential technologies. Although we eventually settled on virtual reality, augmented reality was high on our list.

Around this same time, Judy Diamond, a professor and curator at the University of Nebraska State Museum, was putting together a special issue of Museums & Social Issues, all about the ways that museums are taking on the challenge of educating the public about the dynamic and sometimes controversial issues surrounding human health and medicine. Her invitation to contribute to this issue sparked the idea for this article.

Q: What were some of your key findings?

Well, I found a lot of activity by various groups developing AR technologies, and examples of its applications in diverse settings, such as tourism and advertising, cultural heritage museums, K-12 science education, construction, and space navigation. Some of the empirical studies I found on AR in education reported its value in engaging and holding learners’ attention, and also in enhancing their conceptual understanding and collaboration. But these were exceptions. In all, there was limited systematic research to be found on how people learn from AR.

So, the paper argues that AR is well suited for communicating science, particularly topics in human health. It explains and illustrates its potential affordances, based on research in the learning sciences. These affordances included its ability to represent relationships in space, time, and context, which are critical for understanding, for example, anatomic structures and molecular processes; its ability to immerse learners in interactive narratives, which can help to keep the human element of health and medicine at the forefront of a learning experience; and its abilities to personalize learners’ experiences and to support them in learning from others, which are generally beneficial for learning.

Q: What do you think are some of the most important unanswered questions regarding AR and learning? Or, conversely, what do you think are some of the most promising findings that are just waiting to be fully-realized in a big way?

This is such a big question, because AR is a relatively new technology to education, and research on AR in educational settings is only just beginning. So, I want to say that all questions about AR and learning are important and unanswered! One review I came across called AR “a solution looking for a problem” (Dunleavy & Dede, 2014, p. 743), which I think is so appropriate.

We do know that people get really excited about it. Most studies find that learners report increased motivation and positive attitudes, which are important for learning. But when the novelty wears off and AR becomes as common as web-based video, it would help to have research-backed principles for using it effectively.

For example, how do we use AR as a conceptual scaffold — to give timely and targeted guidance and feedback — without cognitively overloading learners? How might we best take advantage of its location and user-awareness capabilities to personalize learning experiences across various domains and contexts, and for different learners? What are effective patterns for coordinating interactions between collaborators and the digital and physical materials that they work and play with? Building up examples and documenting lessons learned will give designers and educators ideas, resources and guidance for integrating AR into classrooms, museums and game-based learning experiences.

I’m looking forward to new technological developments that may overcome the usability issues with AR, namely, the difficulty with tracking and image processing that can cause AR information to jump around and not stay superimposed where it’s supposed to. Bacca et al. (2014) discuss this in their review as a cause of frustration for learners.

I’m also looking forward to how these developments might improve AR as an expressive tool for designers to create even richer interactive narratives, and for learners, who might learn through creating their own AR experiences.

Ultimately, I’m eager to see how research on AR in education might transform our ideas about learning in situated, authentic contexts.

Posted in Interviews | Tagged , | Leave a comment

Using VR to Frame Furniture at the Jewish Museum

UntitledCan virtual reality in a museum be used to enhance the visitor experience, to strengthen appreciation for the collections? At the same time, can VR assist families and friends to have meaningful conversations about the objects? Or will this all backfire, will VR just be the latest shiny distraction, isolating visitors from the spaces and people around them?

The jury is still out, but a strong argument is being made for the pro-VR side by a new exhibit at the Jewish Museum in New York City. Open until March 26, Pierre Chareau: Modern Architecture and Design is the first U.S. exhibition focused on this French designer and architect.

VR is certainly not the most dramatic element in the show – that distinction would go to the furniture silhouettes (below), with the real objects on one side of a sheet while the reverse features not only silhouettes of the chairs, tables and divans but of animated people interacting with them.

Photo: Will Ragozzino/SocialShutterbug.com

VR is also not the most technologically innovative – that would be what I’d describe as a building-scale CT-scan, with slices shown on a slowly moving screen above the blueprints on the floor while rooms are highlighted and featured on a side wall with video (below).

VR, nonetheless, plays a powerful role contextualizing the furniture, and is so integrated into the exhibit experience that it makes this future-focused tech feel natural (an achievement in and of itself). The VR room can be entered from four directions (see below). There are four chairs next to the center of the room, each stationed before each of the four corners. Each corner features well-lit furniture in a black void. And on each visitor chair is a tethered VR viewer, inviting visitors to pick it up, have a seat, and use it like a pair of binoculars.

Photo: Will Ragozzino/SocialShutterbug.com

Before I describe the experience of using the VR, here’s the context. The signage before entering the room explains that this segment is about interior design. We are told that Chareau was “greatly admired for the way his interiors displayed his exquisite furniture to best advantage.” He had an “almost sculptural conception of space, characterized by an imaginative interplay of space and voids.” While none of these interiors have survived, photographs exist and were used to create VR simulations of the rooms that originally housed the furniture we will see.

Sitting in the seat, facing the appropriate corner, one views in front of them some pieces of furniture. Looking through the Gear VR viewer (a static 360 degree photo) you will still see the same furniture, but now they appear (through computer generated art) in context: in a French salon, or garden, or living room, or office. Twisting around on your chair, you might look up at the light streaming down, or the carpet on the floor, at how all the pieces of the room fit together.


And when you remove the VR viewer, there’s the furniture, still in front of you as before, taking their bows as stars of the show as your mind fills in the dark void around them with memories of your recent visit. It was immersive, seamless, and compelling.

The VR did what VR can do best – take you somewhere you couldn’t otherwise go, to experience something which no longer exists (and in some cases never did), to enhance your appreciation of something real before you. And because it’s just a silent 360 photo – not a video – we (my family and I) were free to talk with each other throughout.

I was there with my family. Here are some details they and I noticed, in no particular order:


  • Other than a security guard (as there was in each of the exhibit’s rooms), there was no staff in the space. Visitors were left to manage the VR on their own, which, when it worked, worked fine.
  • Only three of the four viewers were functioning during our visit. The security guard had to repeatedly explain that to us and other visitors.
  • That also means no one cleaned the viewers between visitors (at least not while we were there). My wife, who has a super-natural sense of smell, told me she could tell.
  • My son noted that since the VR was computer generated, the textures were not 100%. “But I think I liked it like that”.
  • I had presumed the cord connecting the device to the chair was for purposes of security. The security guard explained it was the power cord.
  • Because there was no video to start or stop, no screen or gestural interface that needed to be learned, no instruction was required beyond “Please sit and view”. There was little to no learning curve challenging the visitors. It played off the natural interface of picking something up and looking through it.


Posted in Critiques | Leave a comment