Overview

Goal: create an engaging experience combining an interactive sculpture (in this case a giant ride-able robot affectionately known as “The Mech”) with Magic Leap’s mixed reality headset technology.

Challenges

Mixed Reality Meets Reality

While Magic Leap had demonstrated to us some of the possibilities of their technology through their own demo materials, our goals for The Navigator differed significantly from what Magic Leap had hitherto been developed. Most of the demos Magic Leap provided were geared towards spatial computing-assisted shopping platforms, games, 360 media viewing, and to some extent digital artwork. Rather than creating an experience where the mixed reality content was agnostic to the user’s physical surroundings, we wanted to create an experience where the mixed reality content was dependent on and complimentary to a real world interface. In other words, where the overlaid content responded to interactions between the user and the real world sculpture. An experience of this kind was unprecedented and as with most Meow Wolf assignments any issues that arose would be highly unGoogleable (Stack Overflow would be no help either).

More Possibilities, More Questions

Combining the capabilities of Magic Leap with our own custom hardware systems provided seemingly limitless possibilities for interactivity, but also introduced many questions. How do we avoid creating “interactivity soup” or overwhelming the visitor? How do we avoid “tech for tech’s sake” and instead create an experience which contributes to world building through immersive storytelling? How do we design the interactivity using both systems to create a singular, seamless visitor experience?

Cross-Team Collaboration

The design for the Mech was challenging for all teams including fabrication, technical fabrication, and our designer / developers (turns out it’s hard to build a giant robot!) It became clear early on that fabrication and technical fabrication would have to alternate active production to prevent damage to the electronics and of course the developers couldn’t be working on the Mech while either of those two teams still had work in progress. In order to complete the project, hitting deadlines for exhibitions and pop ups along the way, we’d need to have excellent cross-team communication and attention to detail so as to avoid stepping on each other’s toes.

Solutions, Process, and Approach

Keep Storytelling at the Center

We found that the answer to most of our questions regarding interactivity could be found in the decision to keep storytelling at the center of our experience because what makes an engaging, seamless experience is a good story. Instead of starting from the perspective of “how can we demonstrate this sick new technology?” We started from the position of, “what story do we want to tell with this experience?” We let narrative take the reins and from there came up with a plan for interactivity which furthered that story-line, combining the strengths of Magic Leap and our tactile interface to help the visitor uncover that story.

Putting the Visitor in the Driver’s Seat of Their Experience

To engage the visitor, we immediately position them at the start of our story with a plea for help. They are asked to help in the fight to save their race from endless winter by learning to pilot the Mech through a series of puzzles, eventually finding a place to relocate by navigating a star system. In order to do so, they must manipulate the control panel of the Mech, which in turn triggers overlays in the headset of the star system, a worm hole, and other cinematic sequences which further the story. To create the digital content, we partnered with Magnopus, a design firm in Los Angeles. We created an interactivity matrix to keep track of our inputs (things like buttons, faders, cap touch nodes, etc.) and outputs. This helped us identify the relationship between different components and later helped with the programming of our state machine.

Eemia is one of the planets visitors can explore in the Navigator.

An Unbreakable and Beautiful Control Panel

The control panel was the main focus for our team. We wanted an interface which would be in keeping with the story, something alien but at the same time intuitive enough that visitors would be enticed to explore. And like any control panel, we wanted to provide lots of opportunities for interactivity! Additionally, because the Mech would be a key narrative element permanently installed in our Denver exhibition, it would need to be built to withstand some serious abuse (we had learned many lessons from our exhibit in Santa Fe the House of Eternal Return which contains an arcade that is often in need of tech maintenance).

Towards this end, we performed ongoing testing throughout the development of the project- from doing research on the world’s beefiest buttons and joysticks to how our elements would compliment headset content. We decided to avoid dials, buttons, and switches which could be breaky and opt for contact-less hardware such as hall effect joysticks, roller balls with magnetic rotary encoders and capacitive touch nodes.

To give feedback to the visitor that they were on the right path along the experience, and add to that alien tech aesthetic, we used responsive LED lighting and embedded retina displays, illuminating certain segments and scenes related to and complimenting the Magic Leap content.

Testing, Testing, and More Testing

It was essential for us to have hands on testing of users of various ages as Meow Wolf’s demographic largely includes multi-generational families. As with most projects, we did in-house testing but also brought in consultants and specialists to give us a leg up on how our experience compared to existing XR experiences, whether or not it was exciting as well as intuitive, and overall what we could improve upon. To avoid delays or breaks in testing, we created a mini control panel for hardware / software integration testing (allowing our fabrication and technical fabrication teams to work on the physical install of the real control panel). We also created an iPad simulation of the control panel using TouchOSC for Magnopus so they could continue develop their content without needing access to the physical panel in Santa Fe. We elected to use Unity as our game engine. Because the Mech was one of the premier projects of both Meow Wolf’s Denver story world and Magic Leap, we wanted to make sure that if the opportunity arose to show it off, that it would be portable. This meant that it needed to be constructed and programmed in a way that it could be assembled and calibrated by a small team off-site. These site visits offered us a rich opportunity to hear from technology enthusiasts as well as other core demographics such as XR gamers and influencers.

Results!