HoloLens for

sequential training

and building 

muscle memory

A speculative design response providing actors with on-demand remote access to dance choreography and fight sequence training

Design Challenge

How might we use mixed reality to help filmmakers externalize creative ideas and collaborate with different stakeholders?

Project Type

Capstone Studio

(HCID 541 A: Summer 2019)

Sponsor

Microsoft HoloLens - MRW Team

 

Duration 

6 months

 

Role

UX Designer + Researcher

Skills

Research, Prototyping, Interaction Design, Visual Design, Usability Testing, Film, Video Editing

Team

Surabhi Wadhwa, Saransh Solanki, Dolcie Dass

Here's a short video I made to showcase our final concept

Background

The Future of Filmmaking

Emerging technologies have had a huge impact on the entertainment industry. Recently, filmmakers have embraced immersive technologies to assist in filmmaking. Our research uncovered the possibility of using Mixed Reality to assist various stakeholders in the filmmaking process. Due to the nature of our working relationship with the MRW team at Microsoft, we were constrained by HoloLens as an end product.

Steven Spielberg used Oculus Rift, HTC Vive, and HoloLens during the making of Ready Player One

The Problem

Using 2D Artifacts to Communicate 3D Concepts

Through secondary research, we found that filmmakers use artifacts such as storyboards, previs, and animatics to visualize and externalize their ideas, shots, and sequences. We discovered gaps in communication of these ideas largely due to the fact that 2D artifacts cannot represent 3D concepts effectively. We also found gaps in collaboration between stakeholders at different stages of film production. We decided to focus our attention on the externalization of creative ideas and investigate pain points that exist within collaboration.

DEVELOPMENT

PRE

PRODUCTION

PRODUCTION

POST

PRODUCTION

DISTRIBUTION

The five stages of filmmaking

Research Strategy

Diversification is Key

As we learned more about filmmaking, we realized that the number of stakeholders involved in the process is huge. We knew we needed diversify our research pool in order to understand the bigger picture. We looked for triangulation across different stakeholders.

2 Directors

2 MR Experts

2 Assistant Cameras

1 Film Professor

3 Storyboard /

VFX Artists

3 Film Students

Semi-Structured Interviews

Learning from Experts

With no systems in place, lack of training, and limited budgets—hobbyist and indie filmmakers would make a good candidates for usability testing, but would give us limited insights into the broader filmmaking industry.

1/4

At this point, we determined that the audience we gather insights from should be one that has worked on big-budget productions. My colleague, Dolcie, defined a structure based on which I conducted four interviews to identify pain points and uncover how Mixed Reality could be interjected into the filmmaking process.  

How does the director communicate

envisioned ideas for execution?

What are the existing gaps in collaboration between different stakeholders?

Would HoloLens be an appropriate design response given the context of use?

Insights

Making Sense of the Data

By cross-referencing the information gathered from primary and secondary research, I was able to form connections and developed key insights by synthesizing data. The best way to contextualize these insights would be to group them under three themes. 

Data synthesis

Understanding Space

The production crew relies on written documents or 2D artifacts for blocking a scene on set, which results in gaps in communication of spatial information.

Lower fidelity of computer-generated characters and objects allows for faster exploration of their position, scale, and movement.

Visualizing Virtual Objects

Actors and camera crew cannot envision computer-generated imagery while shooting on set

Directors find it difficult to iterate and communicate their vision for 3D characters and objects through existing 2D pre-visualization tools.

Viability

Big production houses are adopting and innovating new media technology. However, without a design system in place, small productions have limited access to these resources.

Research Report

Documenting our Findings

I compiled our findings and documented our process in a comprehensive report. You can learn more about our research and find additional insights here.

Ideation

90 Concepts

Based on our insights, we identified certain opportunity spaces where we felt HoloLens would make an appropriate design response. I came up with 33 concepts and was interested in taking forward ideas that had a broader scope and could be adapted to other industries. Keeping tech feasibility in mind, we narrowed down to two ideas as a team.

Down-selecting from 90 concepts

Concept I

Planning Position and Placement of Actors

From research, I learned that actors have trouble envisioning CGI while working with green screens and are dependent on verbal cues from the director. I came up with the concept of using mixed reality to help actors plan and rehearse their movement on set in a scenario where computer-generated elements are added in post-production.

Storyboarding the concept

Can HoloLens be used to understand and memorize spatial context?

Before diving into tech, I wanted to test the impact HoloLens could have on spatial memory. I designed and facilitated a non-tech experiment to compare how actors perform with verbal cues vs. virtual props and test the accuracy with which movements are recreated.

Wizard of Oz Prototyping

As a proponent of the method, I suggested validating our concept using the Wizard of Oz prototyping technique and testing it with users. I asked participants to assume the role of an actor, while me and my colleague, Saransh, alternated between the role of a director. To simulate the HoloLens experience, I recreated a rudimentary model of the set by using real-world objects to represent virtual props.

In phase I, you can see the director walking the actor through the scene. The colored markers you see have been added in post and weren’t visible to the actors at the time. Red markers are the obstacles actors must avoid. Blue are computer generated characters. Green is where the actors are supposed to stand.

We then asked the actor to recreate the action sequence with verbal guidance from the director. As you can see, the participant walks right through the virtual obstacles and misses most of his marks. He also seems to be confused and is moving slowly.

For phase II, my colleague and I recreated the virtual set to-scale using real props. Two of us acted as holograms and represented the CG characters in the scene. 

We then asked the actor to reenact the scene without the props. While the pace of the participants improved, they were still missing their marks.

Observations

We tested our prototype with 4 participants. Overall, I learned that our design intervention gave participants more confidence and improved their acting performance. While the experiment helped participants respond to CGI better, they had trouble remembering spatial context without the props.

Concept II

Choreography and Action Sequence Training

Another concept I came up with, explored the idea of using mixed reality to train actors for fight and dance sequences, especially when they don’t have on-demand access to trainers or are working with CGI.

Chris Evans rehearsing for a fight sequence in Captain America

Chris Evans rehearsing for the same sequence with extras

Can HoloLens be used for sequential training and building muscle memory?

I was curious about the impact HoloLens could have on muscle memory and designed an experiment to compare whether learning choreography from a video recording is more effective than learning from a hologram.

Bodystorming

I thought of using bodystorming to observe how people respond to holograms. We gave participants a cardboard headset to simulate HoloLens's limited field of view. My colleague, Dolcie, pretended to be the hologram while I facilitated the user study sessions with 5 participants. 

Participants asked to learn a dace choreography by watching a video. They were allowed to use all standard video controls such as pause, replay, rewind, etc. Participants took longer to understand and replicate the movement they saw on screen. As you can see, there is a lot of back and forth with the participant having to go to the laptop every time he wants to pause playback or repeat a step. 

I then gave the participant our HoloLens prototype. Participants were asked to control the hologram using voice commands or physical manipulation while I closely studied their interactions. Here, the participant has instructed the hologram to move at half speed.

Participants were quick to realize that they were mirroring the hologram instead of mimicking it. Here, the participant instructs the hologram to rotate.

Participants were more comfortable and were able to learn the dance sequence much faster.

Observations

Overall, participants found the 3D hologram easier to follow in comparison to the video. They enjoyed the handsfree experience of interacting with the hologram using voice commands.

Prototyping

Testing Functionality

The non-tech prototyping methods helped us validate our concept and gauge how people respond to mixed reality training. We decided to take the concept forward with an iterative design approach. My colleague, Saransh, built a total of three mixed reality prototypes on Unity. I tested each prototype with users and defined subsequent versions based on my learnings.

First version of our interactive prototype

This is me trying out the prototype for the first time

Design Response

Defining Interactions

After synthesizing the information I gathered from the formative usability tests, I defined the interface requirements and interaction models during whiteboarding sessions with my team. 

Optimized View

While facilitating the user tests, I learned that participants could not see the entire body of the life-sized trainer due to the limited field of view. They could only focus on either the arm movement or the leg movement at a time. To solve for this we made the virtual trainer smaller and pinned it to eye level.

Spatial Context

My assumption was that users would want to view the virtual trainer from the back, so as to not mirror the steps. However, I quickly realized that participants wanted to view the trainer from both angles—specifically participants with prior dance experience or those familiar with the studio model of learning. We added a rotate feature so users could view the trainer from different angles and map their steps more accurately.

Audio Feedback

While conducting usability tests on earlier prototypes, I noted that participants were repeating voice commands multiple times because they didn't know whether the first command had been registered by the device. I advocated for audio feedback to address the lag between voice commands and desired action.

[ This video clip requires sound ]

Sequential Training

Both our non-tech and interactive prototype tests reinforced my assumption that people learn sequences by breaking them up into smaller sections. I proposed breaking up the sequence into 8-count segments for advanced users and 4-count segments for beginners.

3D Carousel

We also explored the use of a 3D carousel for the training library and steps menu. A virtual carousel surrounds the user in their physical environment and the user's gaze triggers a preview of the sequence.

Untitled-1.gif