My Weekend With Mixed Reality


My son playing with a Hololens

I’ve had my conversion moment – I now believe in the power and potential of mixed reality. In this post I want to explain how it happened and what it means for the future of museums.

I first wrote about the concept of “mixed reality” last summer, after attending the Games For Change conference, and after reading a Wired Magazine cover article. At the time I wasn’t sure if MR was just the latest marketing term or if there was something really there there. I think it’s worth quoting in whole, as you can see me trying to get my head around it:

The May cover article in Wired Magazine, “The Untold Story of Magic Leap,” by founding executive editor Kevin Kelly, is a crash course in the latest and greatest in virtual reality (and its relatives). … Kelly brings in the term “mixed reality” (or MR for short) to describe a number of devices that a user wears like a pair of super goggles and, more often than not, while tethered by hardline to a computer. The best I can figure is that MR is augmented reality through a worn device. When I saw Graeme Devine, of Magic Leap, present on his company last week at the Games For Change Festival, he, too, used the term MR (which is where Kelly might have picked it up). Devine defined it as “the mixture of the real world and virtual worlds so that one understands the other.” The power of MR, as I came to understand it, is that AR typically uses one discrete target (like a coin or sticker) to trigger an augmented experience while MR maps the shape of a room and uses that entire map as the canvass upon which to paint. In other words, with MR, the entire surrounding space is in play for layers of augmentation (an exciting prospect for museums).

That excitement I referenced at the end was more aspirational. I certainly wasn’t feeling it. Sure, I saw the value in a device understanding the space around it but, okay, then what? It sounded like a nice feature, but for a limited use case.

Then I started my new position at the Museum. I was no longer reading about Google Tango, Microsoft’s Hololens and the HTV Vive – I was now developing in them and could play with these tools whenever I wanted.

Our first prototype, as a team, brought an AR Shark into our halls using a Hololens:

It was certainly cool watching a physical shark hanging over my head turn into an interactive shark skeleton. But even then I thought of the Hololens strictly as an augmented reality device – a way to bring a single virtual object into the space around our visitors. I still wasn’t getting it.

Over the Thanksgiving weekend, I took the Hololens home. I played with it all weekend, on my own, with my kids, and with my wife. I used the apps (if that’s the right word for them) currently available for download. Some are short experiences you can play in a few minutes while others are designed to last for hours. As we transformed our home into a palette for mixed reality, I finally understood how MR differs from both AR and VR. More importantly, I fell in love with the experience.

First, a bit about Hololens. The device maps Continue reading

Posted in From My Work, Theory | Tagged | Leave a comment

A Suggested Model for Future-proofing Digital Interactives

Last fall, after my experience observing visitors interacting with biological science data through an augmented shark, it got me thinking about how we could learn about future museum visitors by develop cutting-edge prototypes for today’s visitors. Clearly, the tools are going to change, becoming more powerful and ubiquitous. And the digital literacies of our visitors will increase over time as a result. So if the tools will change along with visitor’s abilities to use them, how do we future-proof the lessons for tomorrow that we can glean today?

That lead me to consider the following. Please hang in there with me – I’m still working it through, all these months later, and sharing it with you, in part, to see how it might align with how others approach this topics. So please consider this the start of a conversation.

So while the tech will sure change, and visitors will become more capable integrating digital media into their daily lives, there is one thing that might not change: their personal threshold for how much effort they are willing to put in to learn something new (let’s call this their Learning Curve Threshold) in order to experience something new (let’s call this their Innovation Motivation). When the two are combined, we can discover a visitor’s Zone of Engagement – a window between the point after which the level of innovation is worth the effort yet before an experience requires too much learning.

Screen Shot 2017-01-11 at 4.33.18 PMLet me break this down. All new technology has Continue reading

Posted in From My Work, Theory | Leave a comment

Youth as Co-developers in a Process to Gamify Science Data

How do you support youth learners to participate as co-creators in a rapid prototyping process?


For over four years, our youth programming has invited youth to create with us – card games, mobile games, augmented activity guides and more. Since this past summer, we’ve been developing prototypes that invite visitors to interactive with digital data, seeking insight into the museum of the future. Recently, we ran a four day sprint to see what happens when we bring the two together, and called it Escape the Planet. Throughout we had a team of internal evaluators (graduate student interns from NYU and Columbia, and one high school student intern) track the process of youth claiming their roles as co-developers.



We ran the program with 19 high school students, all day, for four days, on a Tuesday through Friday. In New York City, there’s a test called the Regents, offered for a week in January. On most days most students don’t have any tests, and don’t go to school. This is a perfect opportunity for us to offer new high-risk learning experiences. Enter Escape the Planet.

Escape the Planet was inspired by the rise of escape rooms (see recent post), in which a small group are “locked” in a room and solve problems together, and the recent movement to adapt them for education. Like all of our prototypes this year, we started with the data – in this case slices of our Digital Universe database – and two driving questions: What are the opportunities for increasing the social aspect of interacting with digital data? and How might games and play affect how visitors interact with digital data?

At the start of the first day of the program, led by our staff science educator and an outside digital learning expert, the youth were told that by the end of the week they will have worked with us to create a prototype for a science-based escape room, which they would test with the public. At first blush, this is a rather abstract concept, and most had little way to understand what this could mean, but the challenge had been established: they were going to need to work together to make… something.

Our job on day 1 was to provide them with Continue reading

Posted in From My Work | Tagged , , | Leave a comment

My interview with the author of The Learning Affordances of Augmented Reality For Museum Exhibits on Human Health

Below is a re-blog of my most recent post on DMLcentral, an interview with Camillia Matuk. Read it at its original location, or check it out below.

Augmented Reality and Learning in Museums

When I read Camillia Matuk’s The Learning Affordances of Augmented Reality For Museum Exhibits on Human Health, I knew I wanted to speak with her about AR and learning. Camillia is assistant professor of educational communication and technology at New York University (with a Ph.D. in the learning sciences from Northwestern University, an MSc in biomedical communications from the University of Toronto, and a BSc in biological sciences from the University of Windsor.) She does design-based research investigations to better understand how innovative technologies and learning environments can better support teaching and learning.

Q: Camillia, thank you for joining us in discussion. What came first? An interest in learning, augmented reality or museums?

I’d say it was first my interest in museums, then in learning, and, much later, in augmented reality.

Museums were always a favorite pastime growing up, and were partially responsible for the interest I eventually developed in visualizing science. Although I eventually left the field, my four years as a medical illustrator studying at the University of Toronto and then working at INVIVO made a big impact on me. Still, it was only when I left that field and delved into research did learning come to the fore as a topic of study. As a Ph.D. student, I researched how people made sense of science from visual representations, like the cartoons of evolution that were on display at the Field Museum of Natural History. Later as a postdoc, I studied the role of technology in supporting K-12 classroom science inquiry.

Now, I’m at the Media and Games Network at NYU, and surrounded by people who are creating, studying, and playing with new technologies, including augmented reality. With my research focus on learning technologies, and my background in visualization, an interest in augmented reality was natural.

Q: How did this lead you to your recently published study, “The Learning Affordances of Augmented Reality For Museum Exhibits on Human Health?”

I had begun a project with a neuroscientist and NYU Ph.D. candidate, Oliver Vikbladh, to create an immersive experience of how the brain creates memories of space. The idea was to bring to life Nobel prize-winning research on the brain’s positioning system by allowing people to explore visualizations of place cell activation as they navigated a physical environment.

With Andre Fenton, Ken Perlin, and Jan Plass, we had won a University Research Challenge Grant from NYU to kick start the idea. With artists and former NYU graduates, Adrian Sas and Javier Molina, and with two NYU digital media students, LaJune McMillian and Mahe Dewan, we created a virtual reality experience that was exhibited at Creative Tech Week in 2016.

At the beginning of this project, we considered many different potential technologies. Although we eventually settled on virtual reality, augmented reality was high on our list.

Around this same time, Judy Diamond, a professor and curator at the University of Nebraska State Museum, was putting together a special issue of Museums & Social Issues, all about the ways that museums are taking on the challenge of educating the public about the dynamic and sometimes controversial issues surrounding human health and medicine. Her invitation to contribute to this issue sparked the idea for this article.

Q: What were some of your key findings?

Well, I found a lot of activity by various groups developing AR technologies, and examples of its applications in diverse settings, such as tourism and advertising, cultural heritage museums, K-12 science education, construction, and space navigation. Some of the empirical studies I found on AR in education reported its value in engaging and holding learners’ attention, and also in enhancing their conceptual understanding and collaboration. But these were exceptions. In all, there was limited systematic research to be found on how people learn from AR.

So, the paper argues that AR is well suited for communicating science, particularly topics in human health. It explains and illustrates its potential affordances, based on research in the learning sciences. These affordances included its ability to represent relationships in space, time, and context, which are critical for understanding, for example, anatomic structures and molecular processes; its ability to immerse learners in interactive narratives, which can help to keep the human element of health and medicine at the forefront of a learning experience; and its abilities to personalize learners’ experiences and to support them in learning from others, which are generally beneficial for learning.

Q: What do you think are some of the most important unanswered questions regarding AR and learning? Or, conversely, what do you think are some of the most promising findings that are just waiting to be fully-realized in a big way?

This is such a big question, because AR is a relatively new technology to education, and research on AR in educational settings is only just beginning. So, I want to say that all questions about AR and learning are important and unanswered! One review I came across called AR “a solution looking for a problem” (Dunleavy & Dede, 2014, p. 743), which I think is so appropriate.

We do know that people get really excited about it. Most studies find that learners report increased motivation and positive attitudes, which are important for learning. But when the novelty wears off and AR becomes as common as web-based video, it would help to have research-backed principles for using it effectively.

For example, how do we use AR as a conceptual scaffold — to give timely and targeted guidance and feedback — without cognitively overloading learners? How might we best take advantage of its location and user-awareness capabilities to personalize learning experiences across various domains and contexts, and for different learners? What are effective patterns for coordinating interactions between collaborators and the digital and physical materials that they work and play with? Building up examples and documenting lessons learned will give designers and educators ideas, resources and guidance for integrating AR into classrooms, museums and game-based learning experiences.

I’m looking forward to new technological developments that may overcome the usability issues with AR, namely, the difficulty with tracking and image processing that can cause AR information to jump around and not stay superimposed where it’s supposed to. Bacca et al. (2014) discuss this in their review as a cause of frustration for learners.

I’m also looking forward to how these developments might improve AR as an expressive tool for designers to create even richer interactive narratives, and for learners, who might learn through creating their own AR experiences.

Ultimately, I’m eager to see how research on AR in education might transform our ideas about learning in situated, authentic contexts.

Posted in Interviews | Tagged , | Leave a comment

Using VR to Frame Furniture at the Jewish Museum

UntitledCan virtual reality in a museum be used to enhance the visitor experience, to strengthen appreciation for the collections? At the same time, can VR assist families and friends to have meaningful conversations about the objects? Or will this all backfire, will VR just be the latest shiny distraction, isolating visitors from the spaces and people around them?

The jury is still out, but a strong argument is being made for the pro-VR side by a new exhibit at the Jewish Museum in New York City. Open until March 26, Pierre Chareau: Modern Architecture and Design is the first U.S. exhibition focused on this French designer and architect.

VR is certainly not the most dramatic element in the show – that distinction would go to the furniture silhouettes (below), with the real objects on one side of a sheet while the reverse features not only silhouettes of the chairs, tables and divans but of animated people interacting with them.

Photo: Will Ragozzino/

VR is also not the most technologically innovative – that would be what I’d describe as a building-scale CT-scan, with slices shown on a slowly moving screen above the blueprints on the floor while rooms are highlighted and featured on a side wall with video (below).

VR, nonetheless, plays a powerful role contextualizing the furniture, and is so integrated into the exhibit experience that it makes this future-focused tech feel natural (an achievement in and of itself). The VR room can be entered from four directions (see below). There are four chairs next to the center of the room, each stationed before each of the four corners. Each corner features well-lit furniture in a black void. And on each visitor chair is a tethered VR viewer, inviting visitors to pick it up, have a seat, and use it like a pair of binoculars.

Photo: Will Ragozzino/

Before I describe the experience of using the VR, here’s the context. The signage before entering the room explains that this segment is about interior design. We are told that Chareau was “greatly admired for the way his interiors displayed his exquisite furniture to best advantage.” He had an “almost sculptural conception of space, characterized by an imaginative interplay of space and voids.” While none of these interiors have survived, photographs exist and were used to create VR simulations of the rooms that originally housed the furniture we will see.

Sitting in the seat, facing the appropriate corner, one views in front of them some pieces of furniture. Looking through the Gear VR viewer (a static 360 degree photo) you will still see the same furniture, but now they appear (through computer generated art) in context: in a French salon, or garden, or living room, or office. Twisting around on your chair, you might look up at the light streaming down, or the carpet on the floor, at how all the pieces of the room fit together.


And when you remove the VR viewer, there’s the furniture, still in front of you as before, taking their bows as stars of the show as your mind fills in the dark void around them with memories of your recent visit. It was immersive, seamless, and compelling.

The VR did what VR can do best – take you somewhere you couldn’t otherwise go, to experience something which no longer exists (and in some cases never did), to enhance your appreciation of something real before you. And because it’s just a silent 360 photo – not a video – we (my family and I) were free to talk with each other throughout.

I was there with my family. Here are some details they and I noticed, in no particular order:


  • Other than a security guard (as there was in each of the exhibit’s rooms), there was no staff in the space. Visitors were left to manage the VR on their own, which, when it worked, worked fine.
  • Only three of the four viewers were functioning during our visit. The security guard had to repeatedly explain that to us and other visitors.
  • That also means no one cleaned the viewers between visitors (at least not while we were there). My wife, who has a super-natural sense of smell, told me she could tell.
  • My son noted that since the VR was computer generated, the textures were not 100%. “But I think I liked it like that”.
  • I had presumed the cord connecting the device to the chair was for purposes of security. The security guard explained it was the power cord.
  • Because there was no video to start or stop, no screen or gestural interface that needed to be learned, no instruction was required beyond “Please sit and view”. There was little to no learning curve challenging the visitors. It played off the natural interface of picking something up and looking through it.


Posted in Critiques | Leave a comment

Using Escape Rooms to Gamify Learning: an interview with Breakout EDU

Below is a re-post from my latest column on DMLcentral. Read it here or check it out below:

Using Escape Rooms to Gamify Learning


Escape Rooms first came to America in 2012-2013 from Asia and Europe, quickly spreading across the country. As defined by Professor Scott Nicholson, “escape rooms are live action team based games where players discover clues, solve puzzles, and accomplish tasks in one or more rooms in order to accomplish a specific goal (usually escaping from the room) in a limited amount of time.” It should come as little surprise, but before long, innovative educators were adapting the escape room format for a wide-range of content.

minecraft-escape-roomLast year, at Minefaire, I met Adam Bellow. Adam was honored by ISTE in 2011 as Outstanding Young Educator of the Year. He founded both eduTecher and eduClipper. At Minefaire, however, Adam was doing something new: running a Minecraft-themed escape room, promoting his growing company Breakout EDU. Supporting educators to bring escape rooms into learning settings, Breakout EDU offers kits and curriculum for an emerging community of games-based learning educators.

Before long, we developed a program at my day job (the American Museum of Natural History)  to work with youth to create a prototype for an escape room, with an astro theme, using mixed reality (a Microsoft Hololens). Between advising sessions, Adam and I sat down to talk about the history of escape rooms, education and digital learning.

Adam, for the uninitiated, just what is an escape room?

Escape rooms are physical spaces that started popping up around the world about a decade ago where small groups of people (usually 4-10) get locked into a room and are challenged to solve a series of puzzles in order to escape. While the theme of the room and the particular scenarios vary drastically, from things like “Escape the Zombie Hospital” to “Escape the Prison,” there are usually a few key similarities between all the games.

In most games, the players have a 60-minute time limit to escape. Players are encouraged to collaborate and work as a team — even if the group is made of strangers. Players are watched by a facilitator (usually on closed circuit TV systems) and can make use of a limited number of hints when needed. These rooms have become quite popular and can be found in most major cities around the world.

I went to my first escape room a year and a half ago with some friends. It was a blast but I couldn’t see how the puzzles might work in a museum context. A few months later, at the Games, Learning and Society Conference, I experienced a version of a room that had been developed by Scott Nicholson, at a historic fort in upstate New York. This time, the puzzles had historical resonance, connected to the history of the fort, and the way soldiers in the revolutionary war had used cryptography. Hazzah! Escape rooms could be used for public education. When did you first come across escape rooms, and when did you realize they had educational potential?

Scott is a great guy and has been a big fan of Breakout EDU.

His White Paper on escape rooms is invaluable and a great place for anyone to start.

He actually just won an award for his Breakout EDU game, “Ballot Box Bumble.”

I personally found out about escape rooms the week after my friend and co-founder of Breakout EDU, James Sanders, went to one. He was telling me about how he had gone to an escape room with some educators and students. He had never seen students work so hard to solve really difficult problems in the classroom and wanted to harness that excitement and level of grit to be part of the school day. That is when Breakout EDU was born.

James told me about the idea and I have to admit that I had my doubts. But, they were all removed when I had the privilege of having my first experience of an escape room as a player of the very first Breakout EDU experiment — where there was a locked bag and teachers had to solve a series of complex puzzles in order to break into the bag. Over the following few months, James and his friend Mark Hammons helped to codify the idea and the rest is history. It was clear from the first minute of the game that this was lightning in the bottle. I was immediately on fire with ideas on how this platform in the hands of educators and students would help to change education.

So what is Breakout EDU? How is it helping educators teach content?

Over the next several months, the idea of creating a Breakout EDU box was solidified, a physical kit that could be locked with numerous and different kinds of locks relating to academic puzzles. We began creating and selling these kits in addition to developing an online platform where we could host (and allow others to create and host) games using the materials that we were selling. What started with a small handful of games has grown into a wide-ranging online library of educator-created and shared games that work with the Breakout EDU kit. There are games for all ages and targeting all subject levels. If there isn’t a game that is perfect for your classroom, then the option is always there to use our templates to create one that is and we love to see those games shared back to the community of educators on the site.

Let’s explore the intersection of escape rooms and digital learning. Where do you see the two come together?

Almost all of the Breakout EDU games have both physical and digital elements. Whether it is using the black light to illuminate coordinates that can be researched on the Internet or using automated forms and digital puzzle elements that work in tandem with the real world. We recently launched a series of computer science games that focus on coding problem solving skills. The future is bright for more higher tech puzzle elements. For example, we love games like “Keep Talking and Nobody Explodes” that engage players in VR and the real world at the same time.

The idea of incorporating elements like blended reality into escape rooms and Breakout EDU is something that we are starting to see and experiment more with. Like all other game elements, when done well, it enhances the experience for the players.

Posted in Interviews | Leave a comment

A Series of Epistolary Romances: Games and Museums

Last year I received a rather unusual request: Would I have a conversation with my friend and colleague James Collins – over many months, like letters sent in days of old – and then publicly share it, inviting others to join in.

The request came from CODE│WORDS, an ongoing effort to “gather and harness the discourse occurring among the museum technology community” – which I guess describes me (as well as James, who formerly worked at the Smithsonian). This experiment is called “A Series of Epistolary Romances” designed to generate and facilitate online discussion about “topics of import to the international museum community… Twelve pairs of authors  will correspond with each other over the course of at least a month about a particular topic of interest to the community.”

Students at Museum teaching parents how to play Gutsy after visiting exhibit on microbiomes.

James was invited to be part of one of these pairs through Ed Rodley, editor of CODE│WORDS and Assoc Dir. of Integrated Media at Peabody Essex Museum, and James invited me to join him in this epistolary romances.

As professionals with long experience in game-based learning in museums, we were interested in exploring and unpacking all the complexities that hide within the seemingly straightforward idea of using games for museums-based learning. As with any intersectional issue, there turns out to be a ton of translation problems and misunderstandings among the domains of museum knowledge, game play, and game design.

Selecting Slack as our correspondence weapon of choice, we spent many months hammering the topic, and occasionally each other. Our correspondence has now come to a close, and we are ready to share it all with you.

Over the next few weeks, we’ll release it all on Medium. Here is where it all begins…

May 27, 2016. 10:28 AM joined #epistolary-correspond

 James, Just a short missive to thank you for hosting me today at your offices at the DOE. It was great to see your new offices, now that you’ve left the Smithsonian, and talk with you about Games-based learning. (Btw, I named this Slack GBLinMuseums, which, when turned lower case, as Slack is wont to, scans as “museum goblins”).

[continue the conversation here…]

Posted in Theory | Tagged | Leave a comment