PianoBoard | AR game

Research, UX and UI for PianoBoard, a mobile AR game that teaches one to play piano

PianoBoard | Mobile AR Game

Piano board.gif

PianoBoard is a mobile AR experience that teaches one to play piano. Using an Aryzon or a mobile VR headset with the passthrough camera view, the player would see the sequence of AR keys ‘falling’ onto the keyboard in a certain order and rhythm to let them practice piano.

In this project, I collaborated with an engineer Serge Shack, and was tasked to conduct a usability testing for the app, improve its usability and UI. The results of the testing proved the concept was of a great interest to the target audience, however, there were a number of problems highlighted in both on-boarding and in-game experiences.

The main goals of the testing were to find out users’ main pain points, as well as expectations for how to use the app and where to find the content. There were a few technical specifications that were known to be of inconvenience to the users, but would have to be left in the app – such as printing out the marker images.

On–boarding / Pre–Game Experience

Original solution:


Pain points:

  1. Unclear order of actions.

  2. Unclear which parts of Home screen UI are interactive.

  3. Poor UI organisation.

  4. While on-boarding, users can undertake two different routs, depending on the keyboard device they use - these flows are not reflected in the on-boarding UI.

  5. The flow for the returning user is not reflected in the original UI.

To solve the problem, there were suggested 3 flows: for the new user who’s playing an analogue piano, for the new user who is playing a digital piano with a MIDI output, and for the returning user. Each of the scenarios provides a clear path and gives clear instructions as for what the user is meant to do to start practicing. The interactive prototype for the on-boarding experience can be viewed here.

Screen Shot 2018-02-15 at 22.54.24.png

In–Game Experience

After on-boarding, the in-game experience offers a user to place their phone in the VR headset with the passthrough camera view. User is then asked to calibrate the AR piano and pick a melody to practice, after which the sequence of AR keys would ‘fall’ onto the keyboard in a certain order and rhythm to let them practice piano.

The original in-game UI

The original in-game UI

After play testing the experience on a number of users, there were a few common problems highlighted:

  1. Information architecture seemed rather complex to navigate.

  2. Labelling wasn’t clear enough.

  3. In some instances, labels were too small to be able to read.

  4. Users would have to move their head closer and further, up and down, to see buttons and labelling on the buttons.

  5. While playing, users would accidentally trigger buttons by gaze.

  6. Experience would offer to switch between the VR view and the 2D view, depending on the buttons clicked.

  7. The Menu icon was too small, while it was surrounded by 4 blocks that are meaningless and are not interactive.


  1. To not have Menu items triggered by gaze while playing piano.

  2. To have Menu items visible, readable, and of an accessible size, shape and tilt.

  3. Simplify interactions for controllers like adjusting speed, timeline, and volume.

Based on new Information Architecture map and challenges, there was created a 2D interactive prototype, that was then implemented by the developer into an app.

Screenshot 2019-06-10 at 22.50.13.png

Voice Activated Controls

The pre-game and in-game experiences were validated to be intuitive enough, however, gaze controls slowed down the experience significantly.

To simplify interaction within an app we suggested to implement voice activated controls, where we would use template matching recognition method – for the trigger word and commands listed in Menu.

The trigger word would simply be “PianoBoard”, while the list of commands should include:
Open Menu
Close Menu
Speed Up
Speed Down
Volume Up
Volume Down
Depth Up
Depth Down
Maximum Depth
Minimum Depth
Height Up
Height Down
Maximum Height
Minimum Height
Song Library


Team: Serge Shack (technical)

Role: User Research, UX, UI

Tools: Adobe XD