ezgif.com-video-to-gif (34).gif

Masterpiece Motion VR | Usability Testing

User interviews & usability testing for Masterpiece Motion, a 3D rigging tool in VR.

Masterpiece Motion VR| Usability Testing

 

About the App

Masterpiece Motion is a VR software from Masterpiece VR that allows to rig 3D models in VR.

The tool is currently in Beta and is planned to be released later on this month (May 2019).

In this project, I conducted usability testing to understand how clear the on-boarding tutorials are, and to assess if there are any flaws in usability.

ezgif.com-video-to-gif (33).gif

Usability Evaluation and testing

To run this usability test, I recruited 5 people, who have had no experience in using Masterpiece Motion, have had some experience in using VR applications, and have interest in VR creation tools and 3D modelling tools.

I first conducted usability evaluation for the tool, following the cognitive walkthrough and heuristic evaluation methodologies. Based on that exercise, I formulated the following hypotheses to be tested with participants:

  1. Participants would not be able to follow all the steps of the on-boarding tutorial correctly;

  2. Participants would be able to rig and export a character after going through the on-boarding tutorial;

  3. Participants would be able to use Masterpiece Motion in VR for over 20 minutes straight without any physical discomfort.


Methodology

After creating a script with tasks for the participants to complete, I ran moderated face-to-face usability testing sessions. I used Oculus Rift to run the experience. To record each session, I used the Windows 10 Screen Recorder and an Android phone video camera. Having had 5 users to test on, I spent 18–30 minutes for each session.

For the hygiene purposes, I used the washable eye masks for HMD. (The one-time-use masks I used when running testing for Tvori have proven to be somewhat uncomfortable to manage)

My VR usability testing setup

My VR usability testing setup

The script consisted of the warm up questions to learn about participants’ experience with VR and 3D modelling tools; series of tasks; and general feedback questions after the test.

Before and after the test I also ran the Simulator Sickness Questionnaire to measure the level of comfort in players, and whether the VR experience like Masterpiece Motion can cause any physical discomfort.


Findings

To document observations, I created a Matrix, with the list of tasks as rows and players as columns, so that the performance of each task can be easily viewed in a row, and the completion rate ranked.

Usability testing matrix

Usability testing matrix

After filling in the observations and players’ quotes into the matrix, I measured the tasks completion rate and looked for patterns in user behaviours. I then prioritised findings from major problems to minor problems, general observations, positive findings, and ideas.

The task completion rate matrix

The task completion rate matrix


Findings examples

To showcase usability findings, I created short clips and screenshots that demonstrated the observed issues. Here are some of the examples:

An example of a participant stretching their arms while trying to reach further letters on the keyboard.

An example of a participant stretching their arms while trying to reach further letters on the keyboard.

An example of when the participant didn’t see how the tutorial instructions changed from Rigging to Skinning; and so they continued with the previous step.

An example of when the participant didn’t see how the tutorial instructions changed from Rigging to Skinning; and so they continued with the previous step.

An example of a participant failing to interact with the Import button, trying to grab it.

An example of a participant failing to interact with the Import button, trying to grab it.


Results

Out of 5 participants, 3 were tested and recorded in the in-house lab, and 2 were just observed. Having had 5 participants altogether allowed to reach saturation with findings, and collect valuable observations and feedback.

The results of the Simulation Sickness Questionnaire showed that for the new user, the experience didn’t cause too much discomfort. However, some actions like Auto-Rigging and Auto-Skinning caused discomfort and dizziness. One participant asked to stop the session, and another reported discomfort every time they completed auto-rigging or auto-skinning.

The observations highlighted over 40 findings, all of which were coded and grouped under 6 focus areas, prioritised, and given UX recommendations. The focus areas include:

  1. Tutorials

  2. Panels

  3. Scene

  4. Rigging 3D models

  5. Skinning & posing 3D models

  6. Ergonomics & physical comfort

With this exercise, we were able to validate the two of the three hypotheses:

  1. Participants would not be able to follow all the steps of the on-boarding tutorial correctly;

  2. Participants would be able to rig and export a character after going through the on-boarding tutorial;

Hypothesis 3 “Players were able to use Masterpiece Motion for 20 minutes straight without any physical discomfort” was not validated, as there still were signs of physical discomfort in the participants while performing Auto-Rigging and Auto-Skinning.

After sessions, I also ran interviews and asked participants for their general feedback. Most of the participants agreed that the tool was certainly useful and fun to play with, although it wasn’t easy to comprehend and it needed some more work to be more intuitive.


Feedback

This study was evaluated to be an excellent assessment from the Masterpiece team! A number of the issues found were already worked on by the team, and fixed prior to launch. However, some areas in the report were commented to have been overlooked and the study was found helpful in that regards. The team also suggested a possibility to do more similar studies to further the assessment.

 

Client: Masterpiece VR

Time: May 2019

Role: User interviews, Usability testing

Hardware: Oculus Rift