admin – Page 3 – Irish National Lottery UX

4th December 201710th December 2017

Paper Prototypes & Design Iteration

Crazy Eights

For the initial drawings it was suggested we could run a Crazy Eights Exercise[1] where each of us would select one of the task and design where we folded an A3 into quarters and used each area for one screen (8 screens in total), this has to be done in a total time of 8 minutes. After finishing this up we would vote for the favourite ideas.

The idea is to generate as many ideas as possible within a short timeframe, focusing on quantity of ideas and not quality, and then once you’ve got a bunch of divergent thinking on one topic, to begin converging on some winning ideas by voting on the favourites.

For the first Crazy Eights, we focused on the Onboarding process of the Lotto App.

And for the second Crazy Eights, we focused on the Playing Draw process of the Lotto App.

The Crazy 8s of my fellow researchers can be seen here.

Brainstorming

After discussing our favourite ideas in the Crazy Eigths we started to organize how the first prototype would look like, we sketched a few ideas in the whiteboard for 3 tasks:

Onboarding process (sign up for the Lotto app)
Playing a draw
Checking numbers
- Checking a physical ticket
- Checking ticket bought via app

We took the approach of looking into the personas, their frustrations and goals, and asked if we were solving each of the issues that came up during the user interviews. Each of us designed the screens based on the brainstorming session and designed the screens for each of the tasks. Below are my paper prototypes resulting from the brainstorm:

My fellow researcher’ screens can be found here.

First Prototype

After we put all our designs together we prepared our first prototype. The screens can be seen in Marvel. We ran a pilot, made a few adjustments on the screens and scheduled the interviews with our users.

First Iterations

For the first iteration, we decided to set our screens on Marvel and link them to create the navigation flow. For the recording, we decided to use Lookback. After a few interviews, we saw that the feedbacks from using Marvel would not be as rich in feedback as recording the user with the phone in his hands, so after a brief talk, we decided to change the recording approach for the last iteration we made. Also before each interview, users were asked to sign a consent form.

Users were being asked to perform some tasks on the app and were told that this exercise was to evaluate the design of the app and not themselves, to put them at ease while iterating with the app. Users were also asked to provide a verbal feedback as they were performing the tasks; this information can be used after to formulate recommended design changes [3] to the Lotto app.

The tasks to be performed by the users were:

Sign-up to the Lotto app.
Play and buy 3 lines of the Lotto draw.
Check the numbers of a physical ticket.
Check the numbers of a ticket bought on the app.

And below the different approach to record the user interacting with the paper prototype.

Just after each of the interviews, we applied a questionnaire, with a slightly modified version of the SUS questionnaire and its raw results will be discussed in the next section.

Results and Logs

After analyzing the videos, we generated a heat map of all the screens used during the interviews, that allowed us to visualize any actions that were not expected by us.

Sign-up process

An issue with no sign-up button on the first screen, users clicked on the burger menu.

Users were surprised when asked credit card at first part of sign up

All users chose to scan the card than trying to input the card details on the keyboard.

Most users preferred to use Touch ID as a login method than the PIN number.

Play a draw

When asked to play, one of the users clicked on the burger menu.

Users had no issues selecting numbers and adding lines to their basket on the heat-map but during the videos there was a certain hesitation about the correct procedure.

When the last screen was presented to buy the tickets, all users pressed the buy button.

When asked about being notified about the game they just played, all users clicked in the correct button.

Check Numbers

To check the numbers on a physical ticket, all users selected Scan instead of inputing the numbers manually.

When asked to check the e-tickets, they all proceeded to the Touch ID button apart from the only user that selected to add a PIN Number during the registration.

The responses of the SUS questionnaires were positive. For the Onboarding process, the users found the process easy and one user found that it was worth mentioning that the process to scan the card is a nice feature. A positive feedback also came from the Play Draw task and the Check Numbers task, “easy and fun” and “easy and straight forward” was one of the responses. The SUS score for this prototype was 86.5, well above 68, which is considered the average mark [2].

[1] Levey, Y. (2016, July 27). How to: Run a Crazy Eights exercise to generate design ideas. Retrieved from https://www.iamnotmypixels.com/how-to-use-crazy-8s-to-generate-design-ideas/
[2] Brooke, J. (n.d.). SUS – A quick and dirty usability scale. Retrieved from http://www.usabilitynet.org/trump/documents/Suschapt.doc
[3] Lewis, C. H. (1982). Using the “Thinking Aloud” Method In Cognitive Interface Design (Technical report). IBM. RC-9265.

3rd December 201710th December 2017

Second Prototype & Iterations

Preparing Second Prototype

After the analysis of the user feedback and the questionnaire, we gather the information and the pain points found during the iterations and started to brainstorm for the second prototype.

For the pain points of the first iteration, we had many clicks on the burger menu that was unnecessary and could be removed. Users also went with a debit/credit card payment instead of an alternative payment method but also placed payments with Apple Pay, so we decided to keep that.

During the onboarding process, many users were surprised that the first screen was asking for a payment method, so we switched the onboarding process to ask for Name and E-mail first, Password and then finally Card Details. The results screens were sequenced in the example videos below.

User Iterations

We ran a total of 6 tests for the second prototype. We decided to run with paper prototypes for the second iteration, placing the screens in front of the user one by one. Again, before the interviews, we asked users to sign a consent form. The videos were shot with our phones on a support and pointing to the paper prototypes in a table.

All videos can be seen here:

Analysis

After watching the videos, we created a heat map of the clicks performed during the interview for each screen, adding small notes to clicks that were performed by mistake or were designed to have another behaviour than expected by the user.

Below are the highlights and issues found during the process.

User A missed next button on entering the name and email screen.
User A incorrectly navigated to add a line of Quick Pick.
User B clicked on Touch ID icon in order to sign up.
User C says it would be important to have PIN and TouchID in order to log in as she previously burned her finger.
User C got confused when adding multiple Quick Pick lines, she would have preferred to be prompted for how many Quick Pick lines she wants to buy.
User C had to skip the adding line process as she could not figure out how to proceed.
User D clicked on the wrong button in order to Sign Up.
User D is confused by the feedback icon to indicate how many lines have been added
Used D wants some sort of feedback when checking tickets bought via the app.
User E found confusing to add Quick Pick lines.
User E did not understand the use of the pop-up screen.
User F found confusing the play draw process, did not know where to click and how to proceed.
User F hesitated over what to do on the final payment screen.
User F was not sure where to click when checking out numbers of previously bought tickets. The full data can be seen at this link.

Data is Beautiful

The graph below shows the numbers of errors found throughout the tasks being performed.

The graph below shows the SUS Score per each user.

The graph below shows the average SUS Score of the first prototype and the second prototype.

Our SUS Score decreased from the first prototype to the second prototype, we got a SUS Score of 71.5, above the average 68 but slightly. We believe there are two main reasons for the huge decrease from 86.5 to 71.5, one of them could be that the first iteration was done using Marvel, which allows us to add our paper prototypes into a mobile device. When users click on an area that is not set, Marvel highlights the possible actions in the screen, which allows the user to go ahead and complete the tasks. The other reason is one of the users that tested our prototype was confused during the Play Draw task, which will be discussed below. This particular user gave us a total SUS Score of 35, if we eliminate the lowest SUS score and the highest SUS score, our score goes to 76.7, which is still below our first mark.

During the evaluation of the Onboarding and the Check Number tasks performed by the users, the errors were mainly related to button text labels and button placement.

The Sign-Up button at the top of the screen (Fig. 1) made some users hesitate when asked to Sign-Up for the Lotto App. The Quick Scan button also confused users that bought their tickets online.
Fig. 1. Misleading labels and misplaced buttons.

The main errors occurred during the Play Draw task (Fig. 2), there were too many buttons with misleading labels that made the users confused.

Fig. 2. Too many buttons confused the users

After the test, we applied a questionnaire based on the System Usability Scale and the responses can be seen here.

For the Onboarding screens even though there were mistakes the users responded positively, none of the users gave a bad feedback when asked what they thought about the account setup. It’s interesting to note that for the Onboarding task, all users, even the ones that had issues during the Onboarding process, commented that the process was easy. The same for the checking numbers functionality, where 4 users mentioned it’s easy to understand and one user got confused with the e-ticket functionality.

When asked about the playing draw functionality, the answers were more critical: “The usability was a little complicated, I couldn’t understand which button was to Save (confirm) the row, and which to get a new row. Probably, the names can be improved.” said one of them, and the majority of the answers mentioned that there was a certain confusion as to where to click when adding lines, buying tickets, selecting the Quick Pick and navigating to the checkout screen.

2nd December 201710th December 2017

Third Prototype

After the analysis of the second prototype, the area that needed more attention was the Play Draw task. This is the main functionality of the app and it is a core functionality for the success of the Lotto app according to the goals of our Personas.

During the brainstorm session, we sketched a few ideas on how to solve the main issues that arose during the last iteration.

These are the sketches that came up with annotations as to what we will change for the 3rd Prototype based on the last iteration.

As a means to test quickly the main screen, we produced a version – Play Draw (Modified) – and marked the main actions with numbers, 1 for Add Line, 2 for Clear, 3 for Quick Pick, 4 for Checkout Screen and 5 for the Back button, and schedule a quick test via Whatsapp where we asked the tester a few questions based on the screen below:

If you have all numbers displayed at the top, where would you click to add a line?
Where would you click in order to Clear the numbers selected?
Where would you click to do a Quick Pick?
How would you go to the Checkout screen from that screen?
Where would you click to go back?

All responses were positive, the user understood and clicked on the correct buttons for all questions. We decided to go with the new design for that screen and run a new test on it.

Third Prototype

As the Check Numbers and the Onboarding tasks came back with good results from the last iteration and were validated by the users we decided that the changes on labels and button positions were enough to fix issues detected. For the Play Draw task, we redesigned the button elements, changed the label of the Quick Pick button from “QP” to “Quick Pick”. We also decided to go for a clearer design for this iteration, a step up from the drawings but still using a low fidelity prototype [1], we discussed and perceived that the sketched prototypes were more challenging for users to understand.

Onboarding

Play Draw

Check Numbers

Testing the 3rd Prototype

We tested the 3rd Prototype with three users. We wanted confirmation of some design changes we did before producing the final prototype.

User A: Onboarding | Play Draw | Check Numbers
User B: Onboarding | Play Draw | Check Numbers

During the account setup, users felt the Touch ID option was a bit confusing and during the videos, some users hesitated during the process. User C did not understand the arrows as Next or Previous buttons but that is the first user that came back with this feedback but he navigated the onboarding process without any issues after exploring the arrows. All users had no major issues during the onboard process.

When playing a draw, all users mentioned that the process was straightforward, one user commented that having the numbers being displayed when clicking “Quick Pick” is much better than the current Lotto app that only displays a series of input boxes with the letters “QP” in them. User A also got confused by the Touch ID option during payment and found that other payment options weren’t clear on the screen. All other users had no issues with the new button placements, it was clear to them the actions that each of them would perform.

For the Check Ticket process, users found the button to go to the e-tickets was not clear and User B found it by elimination. The other 2 users responded that it was an easy process even though User A had issues to find her previous tickets bought on the app and with the whole process of the tickets bought via Lotto app.

Again, we applied a SUS questionnaire and the raw results can be visualized here. The average SUS Score was 78.33, which was higher than the score of the 2nd prototype and below the score of the 1st prototype.

And below the SUS mark of the 3rd Prototype compared to the two first iterations.

And finally, we move to the final prototype on the next section, doing the tweaks we found necessary to make. The SUS mark didn’t show a massive increase but the qualitative data was promising.

[1] Pernice, K. (2016, December 18). UX Prototypes: Low Fidelity vs. High Fidelity. Retrieved from https://www.nngroup.com/articles/ux-prototype-hi-lo-fidelity/