ARithmetic - AR Assignment 5

ARithmetic is a fun and interactive way for students to learn basic mathematics with the help of augmented reality. Students arrange playing cards to try to get the correct answer to mathematical problems that appear onscreen. Whenever the cards are rearranged, the answer updates in real time, so students get immediate feedback on their answers.

Usability testing

1. Testing design and planning

The main goal of the usability testing was to assess how well the application functioned as a learning tool. The main focus was to identify how useful the application is for supporting the learning of simple mathematical concepts. With respect to the current state of the application’s development, this meant assessing how easy and engaging the card swapping interface is for users, as well as trying to judge whether it is helpful for supporting their learning. Some aspects were purposefully not included in the scope of the testing, such as the appearance of the application’s visuals, given these are in an early placeholder stage.

In order to do this, the testing process was divided into three stages.

First, participants filled out a questionnaire (Appendix 1) asking them about their level of prior experience with augmented reality technology, and their age.

Second, participants completed a hands-on test of the application. They were given three tasks to complete: solve two onscreen mathematics problem, and solve the onscreen mathematics problem with the help of a hint. The time taken to complete these tasks was recorded. Three questions were chosen in order to account for the fact that participants would likely take much longer to solve their first question as they learnt to use the application.

Third, participants were asked to complete a final questionnaire (Appendix 2) that asked them for their opinions on questions relating to how usable the application was, using a five-point Likert scale. In addition, there was also space for participants to leave feedback in an open-ended format, in case they had encountered any issues or had any suggestions.

1. Participant recruitment

Five participants were recruited for the usability testing, in line with Nielsen’s (2000) suggestion that three to five users are enough to uncover the majority of usability issues. Participants were of a wide range of ages between 18-65. Two of the participants had some level of prior familiarity with augmented reality technology, and three were not familiar at all.

One issue with the profile of the participants is that the demographics of our participant group are not representative of the target audience of the application, namely primary school-aged children. As a result, we are unable to gather data relating to performance improvements, as the adults will likely not experience a gain in mathematics ability. To an extent, this limits the ability of the testing to assess whether the application is an effective way of learning mathematics. In order to address this, participants were asked to consider their experience of using the application from the perspective of the target audience.

Another issue is that participants were drawn from friends and family. This could have resulted in testing feedback being less honest and bias being introduced into the results, which could have potentially covered up issues with the usability of the application. To combat this, participants were advised that the purpose of the usability testing is to assess the user interface, rather than their ability to solve the mathematics problems.

1. Testing protocol

The testing was carried out in person, in a relatively informal manner given that the participants were known to the testers. Before the test, participants were introduced to the purpose of the application and a brief overview of the concept (rearranging cards to get the correct answer). The purpose of the testing was also explained to participants, namely that we were trying to identify flaws in the user interface rather than assess their performance. They were also advised that given the application’s target audience are primary school-aged students, that they should try and consider the application from these students’ point of view, as we are trying to judge the application’s effectiveness as a teaching tool. There was no time limit on solving the questions.

1. Results

Quantitative

Question	Strongly disagree	Disagree	Neutral	Agree	Strongly agree
The application was intuitive to use			20%	80%
The application was confusing to use		60%	20%	20%
It was easy to solve the maths problem using the cards			20%	80%
It was easy to get a hint to solve the maths problem			40%	60%
The application was engaging				60%	40%
The application was helpful for practicing arithmetic				80%	20%
This approach to learning arithmetic would be helpful for children				80%	20%
It was clear what to do to solve the maths problem			20%	80%

Table 1. Results of testing questionnaire

This table illustrates a number of findings. On the whole, participants’ response to the application was positive. Participants agreed that the application was engaging, and a useful tool for learning arithmetic for children. Also, 2 out of 5 participants did not think the application was not confusing to use. This is likely related to a number of issues detailed further in the qualitative results.

	Participant number
Time taken (seconds)	1	2	3	4	5
First question	62	132	41	45	55
Second question	24	47	20	22	25
Third question with hint	23	43	15	26	20

Table 2. Results of timed tests

The timing tests reveal that participants improved significantly after solving the first question, as they were able to quickly pick up on how the application worked. There is no significant difference between the questions solved with and without hints, likely due to the adults’ prior knowledge of arithmetic.

Qualitative responses

Participants identified a number of recurring issues with the current state of the application. These included:

• Hard to manage getting cards on screen and rearranging them while holding a phone

• Not obvious what the operator cards are
• Lack of instructions
• Technical issues around image recognition, such as cards not being recognised properly or glitching

1. Analysis

Despite the issues around recruitment, the results of the user testing revealed some useful insights into issues with the application. Participants found some aspects of the user interface confusing. Three participants suggested that it would be helpful if there were instructions explaining how to use the application, especially for people who have not used augmented reality and are not familiar with the idea of using the phone camera to capture the images on the cards. Also, that the operator cards should be outlined beforehand, rather than the user having to trial different cards to find out what they represent. Participants took much longer to solve the first question, however took much less longer on second and third tries, suggesting that the interface mechanics were easy to pick up. However, some noted that for people unfamiliar with AR, the coordination between holding the phone and rearranging cards while figuring out how to use the application was difficult, again highlighting the need for instructions.

However, on the whole, users agreed that the application is an engaging way to learn mathematics that would be useful for children, and it was easy for them to learn how to use the software. In future, more testing with the actual target audience would help to assess whether the application has a noticeable positive impact on student learning.

Addressing user feedback

At this stage of prototyping the application, development mainly focussed on how the game interface functions, rather than supporting interfaces. However, in future versions of the application, it would be beneficial to have much more instructional material presented to the user. Given the target audience, it would be useful if these instructions were very visually oriented, so they could be understood regardless of the user’s age or literacy level.

Additionally, the image targets used with Vuforia should be swapped out with better quality images in order to ensure they perform better, in order to avoid glitches which are distracting to players and interfere with the application’s use.

Changes made

Appendix 1: Pre-test questionnaire

1. What is your age group?
2. Do you have any familiarity with augmented reality applications?

Appendix 2: Post-test questionnaire

1. The application was intuitive to use
2. The application was confusing to use
3. It was easy to solve the maths problem using the cards
4. It was easy to get a hint to solve the maths problem
5. The application was engaging
6. The application was helpful for practicing arithmetic
7. This approach to learning arithmetic would be helpful for children
8. It was clear what to do to solve the maths problem

References

Nielsen, J 2000, Why You Only Need to Test with 5 Users, Nielsen Norman Group, available at https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/.

Images for Cards

macrovector, ‘Full deck of poker playing cards’ [image], available at:
https://www.freepik.com/free-vector/full-deck-poker-playing-
cards_6086127.htm#query=playing%20card&position=4&from_view=keyword

brgfx (n.d) 'Poker Cards - Hearts' [image], freepik, available at: https://www.freepik.com/free-vector/poker-cards_1185219.htm#query=playing%20card...

Vuforia AR

Vuforia Engine Package for Unity, Vuforia Library, available at
https://library.vuforia.com/getting-started/vuforia-engine-package-unity

Images for Animals

Quanternius, 'Animated Animal Pack' [website], available at: https://poly.pizza/bundle/Animated-Animal-Pack-ILAPXeUYiS

Stack Overflow (StringToMath)

Brandon, 'Parse Math Expression [duplicate]' [website], answered by Aryabhatta, available at: https://stackoverflow.com/questions/3972854/parse-math-expression