We are Karro and Veronica, two engineers who wrote our master’s thesis about user testing of mobile games with help of session recording tools. In this blog post we are going to talk about what a session recording tool is, how to carry out user testing with these tools and what the benefits are. We are also going to explain important aspects to consider regarding session recording tools and test services.
The thesis research, which this blog post is based on, was conducted at the mobile games development company MAG Interactive at their headquarters in Stockholm and the main focus was on how to test mobile games. During our research, we evaluated a total of 10 session recording tools and test services with the goal to find the tool and service most suitable for user testing of mobile games. One of the objectives for the thesis was also to establish a workflow for how to perform user testing with session recording tools, this workflow will be presented further down.
For those who do not know what a session recording tool is, it’s a software that can be used to record the test user while they are interacting with, for example, an app. These recordings are then used to evaluate the user experience. In recent years, new tools have become available making it possible to record the user’s behaviour directly in the mobile device. These tools are integrated into the test app itself and makes the tool somewhat invisible to the user, which contributes to a more natural experience for the test users when they are testing an application.
There are also other kinds of session recording tools which makes use of separate cameras but in this blog post we will only be talking about session recording tools which are integrated in the mobile app itself and henceforth a session recording tool will be referred to as a SRT.
Some tools can record both the screen of the device, the sound input and also the face of the user by recording the input from the front camera of the device. This makes it possible to get a more accurate understanding of the user experience. After the test session has been recorded, the video is uploaded to the tool’s website where it is possible to view it. From here it is usually also possible to share the video and to make annotations at different timestamps in the recording. An annotation can for example contain a comment regarding a user’s frustration or annoyance in a specific part of the app, or feedback from the test user about what’s working and what’s not.
When conducting user testing it is not always enough to have a SRT, it can also be necessary to use additional test services. An online service for setting up the test can be necessary. The test set up service should provide the possibility to create test instructions and explanations about what the test user should do while interacting with the app. It is also important to instruct the user in how to start recording the session and to include additional questions which the user can answer before and/or after the test session. Another important online service that is required is distribution of the app. We need somewhere to upload the application (containing the installed SRT) where the test users can download it in order to take part in the test. There are also more services which can come in handy, for example test user recruitment. Some companies provides several of these services, while others offer only a session recording tool or only test user recruitment etc.
But why test mobile apps using SRTs? We have summed up the benefits and risks in the lists below.
Benefits
- The test users can perform the test on their own devices in a natural environment, and the SRT itself has little impact on the user experience
- There is no need for a testing facility and additional recording equipment
- The tests can be performed remotely. UX researchers and the test users do not have to be at the same location, which reduces travel time and costs
- It is easy and cheap to get access to a diverse group of test users
- Since the test users are using their own devices, the app can be tested on many different devices without having to buy all of them
- It allows the team to review the recorded test sessions multiple times
- The test users can perform their tests simultaneously
Risks
- l Bigger risk for technical issues and puts higher dem ands on the test user. This can however be reduced by writing detailed instructions and doing thorough pilot testing
- Since many of the SRTs are still in an early development stage, the tools and websites are constantly being updated and they are therefore not completely reliable
- Could be problematic to handle app crashes, depending on which SRT that is being used
Workflow for user testing of mobile games using SRTs
During our research, we realised that there is a need for the UX researcher to make decisions regarding several different parts of the test process. For example, not only decisions regarding test tasks and interview questions, but also choice of tool, distribution and test set up service, etc.
It soon also became clear that there was a lack of research and information regarding how to approach testing with mobile built-in SRTs, especially when it came to testing of mobile games. We encountered several problems during our testing process, but we hope that our research and workflow can help others reduce and prevent at least some of the issues. The workflow contains the most important steps when conducting user testing with SRTs. Although our research focused on mobile games, the workflow is pretty general and can hence also be applied when conducting user testing of other mobile apps and softwares. Theworkflow contains 8 important steps and is described below.
1. Test Objective
First, decide on the test objective:
– What questions are in need of being answered?
– What part of the app should be tested?
2. Test Users
Decide on whom should test the game, should it be the main target group of the game, or a group of test users that have never played the game before? Some test user demographics can be:
– Age
– Nationality
– Gender
– Casual or hardcore gamer (i.e. how often they play mobile games)
3. Session Recording Tool and Test Service
A test service can be referred to, as explained above, an online platform where a user test can be created, the installation file for the mobile application can be uploaded, test users can be recruited and both test and app can be distributed to the test users.
The decision about which test service to use depends on:
– Resources
How much time and money can be spent on the user test?
– Control
How much should the researcher be able to control and customise the test, and what specifications are needed in the test set up? Some test set up services only provide a limited number of questions, and some do not provide any post gameplay questions (questions that are answered after playing the game).
– Confidentiality
If the game or concept is not yet launched and should be kept private, it might be desirable to not use a third party company to handle the testing. Confidentiality aspects also affect if the user testing should be conducted locally or remotely, for privacy reasons it might be better to conduct the test locally at the office.
When deciding on a SRT, consider the factors:
– Development Stage
Some tools do not continue recording after an app crash. A crash can also result in the recorded session not being uploaded at all. Also other recording issues can occur. The risk for application crashes is generally higher in an early development stage. Different tools might be more suitable depending on which development stage the application is currently in.
– Platform
Make sure that the tool supports the platform of the application, for example if the app is only available on iOS.
– Facial Recordings
Not all tools provide facial recordings. If there is a need to test the user experience of the game using facial recordings, make sure that the tool provides this.
4. Time Plan
Make a schedule and set a time frame for all parts of the user testing process. This can include planning and preparing test details, integrating the SRT, distributing application and test instructions to pilot testers, correcting issues found during pilot test, distribute app to test users, analyse recordings and questionnaire and finally summarise and share the results with the team.
5. Prepare test details
Prepare the test details for the test users, such as test instructions, tasks, how long to play the game and post gameplay questionnaire.
6. Perform Test
First, a pilot test should be conducted in order to make sure everything is working properly. Make corrections (if necessary) and let the test users install the app, follow the instructions and upload their recordings.
7. Analysis
Analyse the recorded test sessions in order to understand the UX and identify issues, such as if the user is experiencing any difficulties or frustration in any part of the game. Make annotations in the tool’s dashboard where the recorded sessions are stored.
8. Summarise Results and Share with the Team
Share the analysis results with the team and try to explain why an issue occurred. Share both positive and negative feedback from the test users.
Is there an ideal tool?
Needless to say there is no ideal tool which will work in any situation, with any test object, any test objectives or work optimally on any device. However there are some things we found to be of extra importance and which facilitates user testing for both UX researchers and test users. In our research we tried several different tools and services and found that none of the investigated SRTs and test services worked perfectly. However each of the investigated tools had their advantages as well as drawbacks, and the ideal tool would be a mixture of the good parts from all of these tools.
- Our ideal SRT would provide features and properties like:
- Possibility to display tasks and questions to the test user directly in the app
- Feedback directly in the mobile phone, stating if the recording succeeded to upload or if something went wrong
- Possibility to upload the recordings again
- Possibility to pause the recording (for example if someone interrupts during the test session and this should not be part of the test)
- Possibility to change settings from the online dashboard (and not only from inside the code)
- Possibility to customise the tool, for example to turn on/off preview and pause functionality and facial recordings
- Preview possibility for the test user, in order to make sure everything was recorded correctly
- Statistics, metrics and auto generated diagrams in the dashboard
Possibility to adjust camera position (possibility for the test user to see the face for a few seconds before starting the test, in order to make sure the face is visible to the camera) - No loss of information due to application crashes
- A well functioning and easy to use annotation functionalityOur ideal tool would also be connected to a service which provides both test user recruitment, test set up and distribution of test instructions and application. Test user recruitment is generally time consuming and it would be appropriate to store everything in the same place without having to worry about for example connecting the test user’s questionnaires with the corresponding recordings. Another valuable feature would be the possibility to summarise the results and receive auto generated diagrams of statistics and quantitative data directly on the tool’s website. It would then be possible to share the results with the rest of the team instantly, without using multiple services for writing and storing documents, recordings and other data. This would both facilitate organisation of test data and save time. To be able to do all parts of the testing process using the same tool and test service would be optimal.
Based on our study, we have concluded that there is no one perfect method for conducting user tests with SRTs and there is no “one method fits all”. There are many factors to consider, for example: it is important to know what to test and whom to test it on. Based on the answer to these questions, a decision has to be made regarding which test service and SRT to use. Different tools may suit different needs. Which SRT to use also depends on how much control the UX researcher wants to have over the test and also which resources that are available.
It might seem a big leap to start using SRTs, but it’s really rewarding once you get started and we highly recommend it. It is possible to retrieve so much information regarding your user’s experiences and behaviour and you don’t have to be a UX researcher or have previous knowledge and years of experience in the area, you will still gain valuable insights. Since app development is a rapidly changing area and the competition is fierce, it is not enough to just have a working product, it also has to be user friendly and afford a better user experience than your competitors’. By doing frequent user tests early in the development process and figuring out what the users really feel about your product all through the course of development, It is possible to save both time and money. The improvements resulting from these user tests can be what separates you from your competitor.
Thank you for reading our blog post, we hope that this will help you a little bit in the testing tool djungle.
If you have any questions feel free to contact us at karjo132+srt@student.liu.se or veronica.borjesson@gmail.com.
If you are interested in reading our thesis (and a more detailed description of the workflow and evaluations of the examined tools and services) you can find it here: http://liu.diva-portal.org/smash/record.jsf?pid=diva2%3A838963&dswid=326.
Over and out!
//Karro and Veronica