Betafamily Blog

We are Karro and Veronica, two engineers who wrote our master’s thesis about user testing of mobile games with help of session recording tools. In this blog post we are going to talk about what a session recording tool is, how to carry out user testing with these tools and what the benefits are. We are also going to explain important aspects to consider regarding session recording tools and test services.

The thesis research, which this blog post is based on, was conducted at the mobile games development company MAG Interactive at their headquarters in Stockholm and the main focus was on how to test mobile games. During our research, we evaluated a total of 10 session recording tools and test services with the goal to find the tool and service most suitable for user testing of mobile games. One of the objectives for the thesis was also to establish a workflow for how to perform user testing with session recording tools, this workflow will be presented further down.

For those who do not know what a session recording tool is, it’s a software that can be used to record the test user while they are interacting with, for example, an app. These recordings are then used to evaluate the user experience. In recent years, new tools have become available making it possible to record the user’s behaviour directly in the mobile device. These tools are integrated into the test app itself and makes the tool somewhat invisible to the user, which contributes to a more natural experience for the test users when they are testing an application.

There are also other kinds of session recording tools which makes use of separate cameras but in this blog post we will only be talking about session recording tools which are integrated in the mobile app itself and henceforth a session recording tool will be referred to as a SRT.

Some tools can record both the screen of the device, the sound input and also the face of the user by recording the input from the front camera of the device. This makes it possible to get a more accurate understanding of the user experience. After the test session has been recorded, the video is uploaded to the tool’s website where it is possible to view it. From here it is usually also possible to share the video and to make annotations at different timestamps in the recording. An annotation can for example contain a comment regarding a user’s frustration or annoyance in a specific part of the app, or feedback from the test user about what’s working and what’s not.

When conducting user testing it is not always enough to have a SRT, it can also be necessary to use additional test services. An online service for setting up the test can be necessary. The test set up service should provide the possibility to create test instructions and explanations about what the test user should do while interacting with the app. It is also important to instruct the user in how to start recording the session and to include additional questions which the user can answer before and/or after the test session. Another important online service that is required is distribution of the app. We need somewhere to upload the application (containing the installed SRT) where the test users can download it in order to take part in the test. There are also more services which can come in handy, for example test user recruitment. Some companies provides several of these services, while others offer only a session recording tool or only test user recruitment etc.

But why test mobile apps using SRTs? We have summed up the benefits and risks in the lists below.

Benefits

• The test users can perform the test on their own devices in a natural environment, and the SRT itself has little impact on the user experience
• There is no need for a testing facility and additional recording equipment
• The tests can be performed remotely. UX researchers and the test users do not have to be at the same location, which reduces travel time and costs
• It is easy and cheap to get access to a diverse group of test users
• Since the test users are using their own devices, the app can be tested on many different devices without having to buy all of them
• It allows the team to review the recorded test sessions multiple times
• The test users can perform their tests simultaneously

Risks

• l Bigger risk for technical issues and puts higher dem ands on the test user. This can however be reduced by writing detailed instructions and doing thorough pilot testing
• Since many of the SRTs are still in an early development stage, the tools and websites are constantly being updated and they are therefore not completely reliable
• Could be problematic to handle app crashes, depending on which SRT that is being used

Workflow for user testing of mobile games using SRTs

During our research, we realised that there is a need for the UX researcher to make decisions regarding several different parts of the test process. For example, not only decisions regarding test tasks and interview questions, but also choice of tool, distribution and test set up service, etc.
It soon also became clear that there was a lack of research and information regarding how to approach testing with mobile built-in SRTs, especially when it came to testing of mobile games. We encountered several problems during our testing process, but we hope that our research and workflow can help others reduce and prevent at least some of the issues. The workflow contains the most important steps when conducting user testing with SRTs. Although our research focused on mobile games, the workflow is pretty general and can hence also be applied when conducting user testing of other mobile apps and softwares. Theworkflow contains 8 important steps and is described below.

1. Test Objective
First, decide on the test objective:
– What questions are in need of being answered?
– What part of the app should be tested?

2. Test Users
Decide on whom should test the game, should it be the main target group of the game, or a group of test users that have never played the game before? Some test user demographics can be:
– Age
– Nationality
– Gender
– Casual or hardcore gamer (i.e. how often they play mobile games)

3. Session Recording Tool and Test Service
A test service can be referred to, as explained above, an online platform where a user test can be created, the installation file for the mobile application can be uploaded, test users can be recruited and both test and app can be distributed to the test users.

The decision about which test service to use depends on:

– Resources
How much time and money can be spent on the user test?

– Control
How much should the researcher be able to control and customise the test, and what specifications are needed in the test set up? Some test set up services only provide a limited number of questions, and some do not provide any post gameplay questions (questions that are answered after playing the game).

– Confidentiality
If the game or concept is not yet launched and should be kept private, it might be desirable to not use a third party company to handle the testing. Confidentiality aspects also affect if the user testing should be conducted locally or remotely, for privacy reasons it might be better to conduct the test locally at the office.

When deciding on a SRT, consider the factors:

– Development Stage
Some tools do not continue recording after an app crash. A crash can also result in the recorded session not being uploaded at all. Also other recording issues can occur. The risk for application crashes is generally higher in an early development stage. Different tools might be more suitable depending on which development stage the application is currently in.

– Platform
Make sure that the tool supports the platform of the application, for example if the app is only available on iOS.

– Facial Recordings
Not all tools provide facial recordings. If there is a need to test the user experience of the game using facial recordings, make sure that the tool provides this.

4. Time Plan
Make a schedule and set a time frame for all parts of the user testing process. This can include planning and preparing test details, integrating the SRT, distributing application and test instructions to pilot testers, correcting issues found during pilot test, distribute app to test users, analyse recordings and questionnaire and finally summarise and share the results with the team.

5. Prepare test details
Prepare the test details for the test users, such as test instructions, tasks, how long to play the game and post gameplay questionnaire.

6. Perform Test
First, a pilot test should be conducted in order to make sure everything is working properly. Make corrections (if necessary) and let the test users install the app, follow the instructions and upload their recordings.

7. Analysis
Analyse the recorded test sessions in order to understand the UX and identify issues, such as if the user is experiencing any difficulties or frustration in any part of the game. Make annotations in the tool’s dashboard where the recorded sessions are stored.

8. Summarise Results and Share with the Team
Share the analysis results with the team and try to explain why an issue occurred. Share both positive and negative feedback from the test users.

Is there an ideal tool?
Needless to say there is no ideal tool which will work in any situation, with any test object, any test objectives or work optimally on any device. However there are some things we found to be of extra importance and which facilitates user testing for both UX researchers and test users. In our research we tried several different tools and services and found that none of the investigated SRTs and test services worked perfectly. However each of the investigated tools had their advantages as well as drawbacks, and the ideal tool would be a mixture of the good parts from all of these tools.

• Our ideal SRT would provide features and properties like:
• Possibility to display tasks and questions to the test user directly in the app
• Feedback directly in the mobile phone, stating if the recording succeeded to upload or if something went wrong
• Possibility to upload the recordings again
• Possibility to pause the recording (for example if someone interrupts during the test session and this should not be part of the test)
• Possibility to change settings from the online dashboard (and not only from inside the code)
• Possibility to customise the tool, for example to turn on/off preview and pause functionality and facial recordings
• Preview possibility for the test user, in order to make sure everything was recorded correctly
• Statistics, metrics and auto generated diagrams in the dashboard
  Possibility to adjust camera position (possibility for the test user to see the face for a few seconds before starting the test, in order to make sure the face is visible to the camera)
• No loss of information due to application crashes
• A well functioning and easy to use annotation functionalityOur ideal tool would also be connected to a service which provides both test user recruitment, test set up and distribution of test instructions and application. Test user recruitment is generally time consuming and it would be appropriate to store everything in the same place without having to worry about for example connecting the test user’s questionnaires with the corresponding recordings. Another valuable feature would be the possibility to summarise the results and receive auto generated diagrams of statistics and quantitative data directly on the tool’s website. It would then be possible to share the results with the rest of the team instantly, without using multiple services for writing and storing documents, recordings and other data. This would both facilitate organisation of test data and save time. To be able to do all parts of the testing process using the same tool and test service would be optimal.

Based on our study, we have concluded that there is no one perfect method for conducting user tests with SRTs and there is no “one method fits all”. There are many factors to consider, for example: it is important to know what to test and whom to test it on. Based on the answer to these questions, a decision has to be made regarding which test service and SRT to use. Different tools may suit different needs. Which SRT to use also depends on how much control the UX researcher wants to have over the test and also which resources that are available.

It might seem a big leap to start using SRTs, but it’s really rewarding once you get started and we highly recommend it. It is possible to retrieve so much information regarding your user’s experiences and behaviour and you don’t have to be a UX researcher or have previous knowledge and years of experience in the area, you will still gain valuable insights. Since app development is a rapidly changing area and the competition is fierce, it is not enough to just have a working product, it also has to be user friendly and afford a better user experience than your competitors’. By doing frequent user tests early in the development process and figuring out what the users really feel about your product all through the course of development, It is possible to save both time and money. The improvements resulting from these user tests can be what separates you from your competitor.

Thank you for reading our blog post, we hope that this will help you a little bit in the testing tool djungle.

If you have any questions feel free to contact us at karjo132+srt@student.liu.se or veronica.borjesson@gmail.com.

If you are interested in reading our thesis (and a more detailed description of the workflow and evaluations of the examined tools and services) you can find it here: http://liu.diva-portal.org/smash/record.jsf?pid=diva2%3A838963&dswid=326.

Over and out!

//Karro and Veronica

Having worked a lot with ­­offline-capable apps lately, I started to see similar trends everywhere. Of course, that is partly because of frequency illusion, also known as Baader-Meinhof phenomenon – “When a thing you just found out about suddenly seems to crop up everywhere.”

Nonetheless, offline is really trending. As broadb and connections and 4G cell phones become commonplace, and everyone and their mom talk about IoT, streaming and The Cloud™, it might seem strange to focus on the eventuality that you would temporarily lose your precious connection. But I find it perfectly reasonable. The more we get used to online services, the more we depend on them, and the worse the damage done when they unexpectedly stop working. Considering and handling offline scenarios is proactive damage control.

YouTube supporting offline

During a recent trip to Mumbai, India, I noticed while browsing YouTube on my Android phone that a new button had appeared in the app. I was able to download videos for offline viewing! Eureka! There were some restrictions; only videos uploaded by non-commercial accounts seemed to work, and only if the clips were ad-free. But still, a great feature. You could even choose the video quality, balancing a trade-off between resolution and disk space.

Alas, when returning home to Sweden, my new favorite button was gone. India only feature, it seems. Apparently, in most markets, this ‘YouTube offline’ feature will instead be marketed and sold as YouTube Red, a premium subscription option that initially launched in the end of October in US, at $9.99 per month. Oh well.

Viaplay promoting the offline use case

Another offline-capable app catching my eye lately is that of Viaplay, probably the biggest Swedish movie/TV streaming service, originating from a major TV network. Even though the stability and speed of internet connections in Sweden is very good, Viaplay is currently running ads in the Stockholm subway network, inspiring customers to view movies that were pre-downloaded to their devices. How come? In a time when Netflix shouts from the rooftops that they will never, ever, ever support offline viewing, Viaplay is simply filling a gap. Offline is now one of their USP’s. And of course, it helps their case that mobile data is getting increasingly expensive.

There is no such thing as an unlimited data plan

Because it is no secret that mobile carriers are charging more for data compared to a few years ago. We make fewer voice calls, and EU legislations have enforced lowered roaming costs, so the carriers have to make their money somewhere else. If you are one of the lucky ones to still be on an old mobile plan with unlimited data, treat your SIM card like a precious valuable. And if you do have such a contract, chances are that your carrier has already tried to sweet talk you with a free iPhone or something similar, just in exchange for “renewing your subscription”.

How to develop apps for offline…

If you as a developer are including offline or caching features in your app, of course you should test out the functionality in realistic conditions. Set the device to flight mode and give it a go, right? No, a bit more complex. There are design choices involved with supporting sudden network loss. A good list of considerations from Infoq:

• Displaying comprehensive error messages when network calls fail.
• Allowing the use of the application in “guest mode”, where certain features can be delayed until the user actually signs in.
• Visually displaying the absence of network connectivity on the UI (connected mode/offline mode).
• Disabling controls in the absence of network connectivity.
• Allowing the user to query and act on data while no network connection (offline data access).
• Testing the application under different network conditions!

… and how to test offline

Testers, on the other hand, will do well to try out a few different scenarios of connection loss, reconnections, app booting, and how to handle data access when your device is totally isolated from communication. Google has some offline testing guidelines which are primarily intended for web apps, but which go just as well for native mobile apps for Android or iOS.

Make sure your app works well under the following circumstances:

• The app is installed,and then immediately goes offline. In other words, the first use of the app is offline.
• The app is installed on one computer and then synced to another.
• The app is uninstalled and then immediately installed again.
• The app is running on two computers at the same time, with the same profile.
• The app must behave reasonably when one computer goes offline, the user does a bunch of stuff on that computer, and then the computer comes online again.
• The app has intermittent connectivity, switching often between online and offline.

PUBLISHED BY

Peter Skogsberg
Peter Skogsberg is an ISTQB certified software tester and holds a Master’s degree in Information Technology. He has been on both the development and the testing sides of several mobile and web applications.

It’s a new month and time for another awesome beta tester to take the limelight here at the Beta Family Blog. This month we’re featuring Amit Kulkarni who’s been a part of our community for several years. He has some interesting tips on how to become a successful beta tester so keep reading and have your notepad ready!

Name: Amit KulkarniAmit

Age: I am still young in testing ????

Nationality: Indian

Interests: Cycling, Music, Writing, and of course testing variety of apps

Devices you own:

iOS: iPhone 6, iPhone 5, iPhone 4, iPhone 3GS, iPad mini, iPad 3

Android: Galaxy Note II, Galaxy S 2, Galaxy Ace, Galaxy Tab 2(7”), Galaxy Tab 10.1, HTC Desire C, LG Optimus Vu

Windows: Lumia 520

BlackBerry: BB8520

How many apps have you beta tested?

At the Beta Family community I have tested 46 applications. It may sound less when compared with other testers around the world and part of this community, but there are other factors that affect the actual numbers.

What’s your best quality as a tester?

Perseverance, creative thinking, and of course patience.

What’s your favorite app?

Well, this is kind of a question where you not necessarily have a specific answer. I always keep trying new applications as and when they are available in their respective stores. But since you need specifics – I’d say that Strava (Fitness ), Peak (Education), Microsoft Outlook (Productivity), Swipes (Productivity), Wally (Finance), hike (Social networking) , 8 Ball Pool (Games), and any casual games like Candy crush, Daddy was a thiefand many more.

There are others as well but that list will be quite long so I tried to list one from each category that I use the most. To sum up; any application with interesting concept, excellent UI and feature set would be my favorite app. It does not need to be from a big company and can be from a startup or sole developer.

Do you have a favorite app developer or company?

I truly believe that all developers try their best and it would be harsh to pick one or for that matter the company. Just to give the background I like the developers/companies which actually listen to their user base. This is very important today since we are becoming more impatient and with growing large number of options, we are always on a lookout for a switch if needed. Users are ready to pay more for a good deal so I feel that all the companies/developers try to match up to the expectation of their user base.

Over here at BetaFamily I like to work with some developers like Dave V and Tamas Laszlo. Having said that, it doesn’t mean that I do not like others but I just wanted to be specific. (Not diplomatic :D)

How come you started with beta testing?

I really don’t know. But, I guess I got to know about BetaFamily while going through some article. They have come a long way and there is quite a long path to cover, but I believe I was the early adapter to the BetaFamily community.

What kind of apps do you most like to test?

I like to test any kind of application where there is a new concept behind its implementation. By that I mean, not necessarily that it’s a totally new idea but the implementation is such that it makes the app different from the rest.

Over here at BetaFamily I feel that developers take a chance to actually see if there is any interest for the idea they have in mind before going to the larger audience. I like to play or test any games or an ideas that is something new and I haven’t had a chance to use any app like that before.

As you’ve tested lots of apps you must have encountered many bugs and faults. Is there one bug or fault you see often that you feel developers should know about and fix before starting the beta testing?

More often I find that there are quite a few applications which have not been tested when offline (no connectivity) and that results in a crash.

Why are you a part of Beta Family’s community?

I am part of the community because I get a chance to play with the apps before other people. I get to interact with developers and underst and their point of view. There have been instances when the developers have helped me with something I thought could be different. Also, the community has quite a good testers and that’s helps.

The support team here is quite good as they take time to respond to each and every small question, and fix an issue with the platform at the earliest. The good part is they don’t mind you contacting them – it may take some time but there is always a reply.

How long have you been a member at Beta Family?

It has been almost 2 years. I signed up with Beta Family in the month of December, 2012 so I am counting from 2013.

Do you have any tips for newly started beta testers?

• Be practical while applying to the tests (make sure that you can dedicate them the time it requires.)
• Interact with developers when unsure.
• And most important focus on quality.
• Bonus tip (for reading thus far): Imagine developers as a friend of yours who need constructive criticism from you even if it hurts.
• Try to win the confidence of the developers with your testing as that could go a long way.

What’s your philosophy towards beta testing?

I feel that beta testing in general is quite good as that gives developers a platform where they can see if the idea that they have in mind could attract audience. It has two fold advantages as it gives developers a platform and new testers to get better at testing. However, new testers can also spend time on open source application under some mentor to learn more about testing and get better at it.

I feel beta testing is also useful for the companies who feel they need testing but can not afford too much of money pouring into it. Such companies can take advantage of the platforms like BetaFamily.

What do you need to be successful in beta testing?

• Most importantly you need to be patience, as it is really something that is going to help you in long run.
• Need to be honest with your criticism while submitting the feedback. A developer does not mind if you tell them truth of what you actually feel about the application, rather they appreciate your honesty and not just good words for the app.
• Try to be as helpful as possible and when needed – do not hesitate to interact with the developers to put forth your point if you missed out something in your report
• Refrain from using words that are too harsh in your feedback even if it is constructive criticism.
• Help BetaFamily if you encounter any issues with their site as that way they would be in better position to fix something that needs immediate attention
• What are the best and worst things about beta testing?

Best:

• It is your choice what to and what not test.
• Chance to work on the applications not yet gone live.
• A platform with quite useful features that helps you with your feedback.
• Helpful team of developers and support team.
• And of course a chance to earn money while you test the apps.Bad:
• Less communication with stakeholders.
• At times the information related to the app is not adequate or sufficient.Would you rather test apps for bigger companies or new start ups?

That is good question. What matters to me the most is communication. I love to work with the companies which are ready to communicate, helps you understand and answers any of your question unless they are not too obvious. There I feel start ups are more open and they tend to be more co-operative with you. Other advantage of working with startups is that they are ready to listen and willing to change something if that makes the app better.

I believe that both startups and sole developers who are working on these apps and needs some constructive opinion are the most often willing to communicate if you have any question. There is no way for us (testers) to actually come to know whether it is a big company or startups at Beta Family so this I am saying from my personal experience. It has been pleasure to be a part of this community.

What makes an app exciting to test?

Creativity.

How many apps are on your phone right now?

As you can see I have a lot of devices and it would take some time to count them all but just to go by the new device that I own i.e. iPhone 6 – I have 100+ (107 to be very specifics and growing).

Are you a gamer? If so, what kind of games do you like?

Hmmm, yeah! But I would point out that I do have control where I do not keep on playing the games for hours. Mostly I enjoy any new game with something new in it. I do not have any special choice or specific category as long as the game keeps me interested. At times I had to clear some space and it is hard choice which one to remove. ????  I kind of feel I should just keep on adding any new game that serves the purpose of having a good/fun time.

Thanks to Amit for taking time to answer my questions about beta testing. It’s been a pleasure having Amit in our community here at Beta Family and we hope he’ll stay with us for many more years. Check out some of our previous posts to learn from other Beta Tester of The Month winners. Stay tuned by subscribing to our blog.

PUBLISHED BY

Malin Klockare Gullesjö
Malin Klockare Gullesjö is working with Beta Family’s online profile. She has previously worked as a community manager on social media in the tourism industry.

For your products to become successful you need to know your target market and audience. If you don’t know these you’ll waste time and money on irrelevant people. But how do you know your true target audience? The people you think are your target market might actually be the complete opposite to those who end up buying your products. 

One way to find your target is to run focus groups. In focus groups you’ll get an idea of who likes your product, however there are only a few people in focus groups and you probably need more data to find your market. This can lead to some quantitative research. However, if you’re already doing quantitative testing of your app I believe this can actually help you find your target market.

It’s a real advantage to have testers involved with and interested in your app. They give a lot of important information, like finding bugs and problems with usability, but why not get these interested testers more involved? They can do more than finding bugs, in fact, they can help you nail down your target audience. Start a dialogue and ask them for their feedback over a longer period of time. Ultimately, the testers are part of your target so invite them to help you understand them better and learn what kind of people they are. This can end up saving you a lot of money.

When using a beta testing platform such as Beta Family you get testers from all over the world that have a passion for apps and you can use this as a base to help find your target market and audience. Something that might help you with this is our feature called Teams. It lets you create different teams depending on your goals. You can add our top testers to a team and have good clear communication with them. Since this feature gives you better contact with your testers you can have an honest dialogue and learn more about them. You can also use it to create teams of people that you believe are in your target audience and it will help you underst and if they would indeed buy and/or use your app.

A quick word of caution for when using the Team function to test your app. We suggest you use both chosen and random testers to maintain credibility. If you only use chosen testers, and pay them too, any reviews of the app could lack credibility and make potential customers skeptical about the results and how good the app is.

It’s October and we’ve got a new beta tester show you. Ken Wong Zhi Bin is our Beta Tester of The Month. He’s 34 years old and there were two things that made him start with beta testing. Read on to find out what they are.

Name: Ken Wong Zhi Bin

Age: 34

Nationality: Singaporean

Interests: Technology

Devices you own: PC, laptop, android phone, Ipad, smart TV, etc

How many apps have you beta tested?

More than 40

What’s your best quality as a tester?

My patience and capability to look into details

What’s your favorite app?

I have a few. Whatsup and Google Drive is two of my favorites.

Do you have a favorite app developer or company?

I admire of all those one-man developer/company worldwide, kudos to you all!

How come you started with beta testing?

To make internet a better place and to earn some $

What kind of apps do you most like to test?

Game apps or social apps

As you’ve tested lots of apps you must have encountered many bugs and faults. Is there one bug or fault you see often that you feel developers should know about and fix before starting the beta testing?

The connection to server. It must be ready before beta testing.

Why are you part of Beta Family’s community?

It came out as top from the search result of search engine.

How long have you been a member at Beta Family?

More than 3 months

Do you have any tips for newly started beta testers?

Actively participate all the available testing.

What’s your philosophy towards beta testing?

It’s a matter of manner towards everything, it actually reflect your mentality towards your life too.

What three things do you need to be successful in beta testing?

The three things I think you need to be in order to be successful at beta testing are patience, honesty and speed.

What are the best and worst things about beta testing?

The worst is to forget where you have stopped at.

The best thing about beta testing is when you encounter a bug-free app.

Would you rather test apps for bigger companies or new start ups?

I think I rather beta test for new start up because I believe that they have better ideas.

What makes an app exciting to test?

Innovative idea

How many apps are on your phone right now?

I have lost count but at least over a 100.

Are you a gamer? If so, what kind of games do you like?

Casual gamer, strategy game is my fav.

PUBLISHED BY

Malin Klockare Gullesjö
Malin Klockare Gullesjö is working with Beta Family’s online profile. She has previously worked as a community manager on social media in the tourism industry.

Since I joined Beta Family I’ve discovered a tool called Mention, which lets users monitor certain phrases that are important to them. For example, I’m always on the lookout for app development companies that need beta testers and with Mention I get an overview of all social media channels and forums where companies are searching for them. It’s great!

Using Mention I noticed it’s common for fans of games and other applications to contact companies directly and offer to be beta testers. This is quite interesting and I reckon worth chatting about, because there are some aspects to consider when using fans as testers.

In a previous blog post I discussed the idea of using your testers as marketeers. However, when talking about fans and their ability to test new apps we’ve reached a dilemma, because I believe they’re not the best beta testers.

Why you might ask? Well, I’ll tell you.

When you’re dealing with fans you have to realize they have high expectations and a preset idea, sometimes unrealistic, of how the app should be. If the app doesn’t deliver on those expectations the feedback might be overly harsh. It can also lead to the beta testing fans bad mouthing the app on social media and forums where other fans are. If fans are used in the testing it’s important you clearly communicated your vision and what you want the app to achieve to avoid this situation. At the same time, this sometimes harsh feedback can be very useful and help you make improvements in upcoming updates.

Testers that don’t know about you don’t have this problem. They’re beginners with completely fresh eyes and will give you unbiased constructive feedback. Often the goal with an app is to get as many people as possible to use it, right? Well, since your fans are already familiar with your products, things that are easy to them might be complicated to beginners. Therefore, using feedback from fans you might accidentally make the app too difficult and block off potential new users. And let’s be honest, we don’t wanna miss out on new customers.

But aren’t there positive aspects as well? Of course there are!

There are a lot of great reasons to let fans do the testing. They’re devoted to your products and are often happy to do the testing for free. They truly want you to make better apps and succeed. You don’t have to spend time and money finding beta testers as they’re right there and willing to work, often contacting you asking to be testers. And you can be sure your fans will do their best to help you!

We talked earlier about the pitfall of not meeting high expectations. Obviously the opposite can apply. If you smash it out of the park and get fans excited they will tell their friends and other people about their experience, creating word of mouth and free advertising.

Also, you can learn a lot about beta testing fans if you ask the right questions. When you know a lot about your users you can find their “twins” on social media sites like Facebook or LinkedIn, because those sites know so much about their users. I don’t mean actual twins of course, but rather people with the same interests etc. In other words, potential new customers.

So who is the optimal tester then?

There is no universal answer to this question as there are so many different reasons for testing products and services, however there are some characteristics to keep an eye out for.

Since the tests can be quite difficult it’s always helpful if the testers have a technical background, or at least an interest in learning the technical side of things. It’s also good if they have an interest in testing the app’s limits to find errors and don’t just do it to be among first trying it. They should have good communication skills too, both verbal and written, as they’ll have to explain their findings.

Great testers practice to improve these skills so you should look for testers with a lot of experience. Unfortunately it can sometimes be hard to find all these characteristics in a tester and perhaps even more so in a fan without testing experience.

So to conclude, I believe your products and services will be better off if you use experienced unbiased beta testers and clearly communicate your expectations to them. Having said that, it’s also valuable to invite dedicated fans to join the testing once in a while, just make sure you don’t solely rely on them. Also, a really cool thing about involving fans is that they tend to get real happy when they’re involved!

Wow, time flies! It’s already September and time for another Beta Tester of The Month. This month we’re happy to feature Jerrod Kerwin-Sweeney! He’s got an eye for user interface design and enjoys innovative and creative solutions. Keep reading to find out his tips for beginner beta testers.

Name: Jerrod Kerwin-Sweeneyjerrod

Age: 20

Nationality: American

Interests: Game Development and Design

Devices you own: Samsung S3, iPhone 4

How many apps have you beta tested?

I have tested over 60 applications, including the apps I tested on the Beta Family site.

What’s your best quality as a tester?

I personally think my best quality as a tester is my eye for UI design. I’ve worked with UI and UX designers for a while know and I like to share what I’ve seen in my own work.

What’s your favorite app?

My favorite app at the moment is Reddit is Fun. I use the app daily and I think the mobile platform is very convenient. My other apps are mainly for work, media, and scheduling.

Do you have a favorite app developer or company?

I don’t play mobile games as frequently as I used to. It would be unfair for me to list a specific developer as my favorite without playing the most recent releases, and my devices are somewhat dated so it’s difficult for me to judge the quality of a game on what could now be considered a low-end device.

How come you started with beta testing?

I started with QA and beta testing because it complemented my chosen field.

What kind of apps do you most like to test?

I enjoy testing a variety of apps. I enjoy seeing the innovation and creative UI solutions that smaller devs can present. I’d say I enjoy testing apps that aren’t music or messaging related the most.

As you’ve tested lots of apps you must have encountered many bugs and faults. Is there one bug or fault you see often that you feel developers should know about and fix before starting the beta testing?

The one issue I notice the most is bad text formatting. It doesn’t matter how robust or attractive your UI is. Simple formatting issues like messy text wrapping or alignment will throw off the appearance and polish of your entire application. It’s an easy fix but it is too commonly seen.

Why are you part of Beta Family’s community?

I am a part of the Beta Family community simply because of the convenience on the testers’ side of things. I like being able to help smaller teams that are trying to get new apps off of the ground. It’s been interesting to see how the service has developed over time, and my overall experience has been very positive over the years.

How long have you been a member at Beta Family?

I’ve been a member at Beta Family for 3 years.

Do you have any tips for newly started beta testers?

Be critical, test beyond what the test report is asking you to test, and don’t be afraid to be honest.

What’s your philosophy towards beta testing?

I think beta testing is useful for smaller companies, but I also think it is something that should be geared towards testing usability rather than bug tracking. I don’t mind doing bug reports, but a straight forward test report allows for more feedback about the application itself to make its way back to the developer.

What three things do you need to be successful in beta testing?

To be successful in beta testing you need excellent written communication, an honest opinion, and an eye for UI design.

What are the best and worst things about beta testing?

The worst thing about beta testing is the test reports. Sometimes the language barrier or poorly structured test reports make giving feedback very difficult. I’d say the best thing about beta testing is the feedback you receive from the developers and test owners for delivering a high quality test report.

Would you rather test apps for bigger companies or new start ups?

I’d rather test apps for bigger companies. I have no problem testing apps for smaller startups but it’s interesting to see both sides of the market, and in the past I’ve done more testing for smaller companies.

What makes an app exciting to test?

Seeing creative solutions to common problems is what makes an app exciting to test. I enjoying seeing unique designs and concepts in their earliest stages, and it’s even more rewarding seeing those apps make it onto Google Play nd the App Store.

How many apps are on your phone right now?

I have almost 40 apps on my personal phone, not counting stock apps. Most of them are for daily use and not game related apps.

Are you a gamer? If so, what kind of games do you like?

I’m an avid game and I enjoy FPS games for the most part. I tend to play more PC games in my own time.

Thanks to Jerrod for being our September Beta Tester of The Month! We love having so many fantastic testers here at Beta Family and hope you enjoy reading about some of them each month. You can check out other featured beta testers in our older blog posts.

PUBLISHED BY

Malin Klockare Gullesjö
Malin Klockare Gullesjö is working with Beta Family’s online profile. She has previously worked as a community manager on social media in the tourism industry.

It’s important to treat your customers and beta testers right. In this post I explore how it can lead to your testers becoming happy customers too.

When doing research for a previous blog post about getting your testers involved to help find your target audience I realized that customers often feel like beta testers when they’re actually paying customers. In other words, the app they’ve bought has some bugs or limited usability and they reckon it should’ve been tested more before hitting the paying market. Unfortunately many organizations lack the time and money to do extensive quality controls of their apps and therefore release their products when it still have some kinks to work out. It might still be at an acceptable level, but it can be enough to annoy the consumer.

Here in the app development community we benefit from having beta testing as a huge part of our development process. When you clearly label your product as a beta version then your potential customers know that if they want to test it, they can to expect it to be flawed. It is a big no no in my book to treat your paying customers as unpaid beta testers, because you only get one chance to impress.

So instead of making the early adopter customers your beta testers, make the beta testers your customers. There are a few things you can do to increase the chance of this transformation from tester to customer. To start with obviously treat the testers well and have a good dialogue with them. When the beta testing is done you can offer them a special deal. Perhaps give them the production version of the product, or a reduced price or some other incentive that will turn them into customers. Just make sure to not offer this incentive until after the testing is done to avoid getting extra positive feedback in “exchange” for this offer. If you have a good app that they enjoyed testing it should be easy turning them into customers, as long as you genuinely appreciate them and show it.

So keep in mind, everyone is a potential customer and make sure to treat them that way, including your testers!

It’s a new month and time to showcase another awesome beta tester here at Beta Family. Taras Honcharenko has only been a part of our family for a few months, but has already tested lots of apps! He’s got a very interesting philosophy. Read on to check it out.

Name: Taras Honcharenko

Age: 32

Nationality: Ukrainian

Devices you own: Sony Xperia z3compact, HTC ONE, HTC Sensation XL with Beats Audio X315e And iPhone 6 (is temporary unavailable).

How many apps have you beta tested?

At this moment I have tested 50 apps on Beta Family.

What’s your best quality as a tester?

I am target-oriented, perseverant and ingenious.

What’s your favorite app?

I can’t indicate any specific app because there are a lot.

Do you have a favorite app developer or company?

Cloudburst games LLC and Fábio Guimarães Cunha are definitely my favorite developers regarding the quality of apps and feedback.

How come you started with beta testing?

I’m trying to improve my testing skills thus I’m permanently searching for opportunities in different areas. Therefore when my brother mentioned Beta Family as an excellent testing field I was glad to join the family.

What kind of apps do you most like to test?

I’m a huge fan of logical games.

As you’ve tested lots of apps you must have encountered many bugs and faults. Is there one bug or fault you see often that you feel developers should know about and fix before starting the beta testing?

The most crucial constituent of any app is its main functionality. In search for various features some developers can easily neglect the importance of proper and stable performance of primary function.

Why are you part of Beta Family’s community?

I consider the Beta Family by far the greatest place to improve testing skills for a novice tester. It’s extremely convenient and user friendly.

How long have you been a member at Beta Family?

I joined the community on 14 December 2014.

Do you have any tips for newly started beta testers?

Be patient, perseverant, and passionate about your job.

What’s your philosophy towards beta testing?

“Be water, my friend”. (Quote from martial art legend Bruce Lee).

To get the meaning of the quote, click here.

What three things do you need to be successful in beta testing?

• Efficiency, detail-orientation and good communicative skills.
• Would you rather test apps for bigger companies or new start ups?
• I’m currently working on the position of QA Engineer in pretty big company dealing with different e-commerce projects but i’m still highly interested in any high-quality project despite the company format.

What makes an app exciting to test?

The basic aspect of any application is IDEA. The more interesting is the IDEA, the more exciting is testing

How many apps are on your phone right now?

It’s about 20.

Are you a gamer? If so, what kind of games do you like?

I’m hardly a gamer nowadays, but i still get excited while playing old console games.

A big thank you to Taras. We are happy to have him and his beta testing skills in our community. Next month we will feature another successful beta tester so make sure to subscribe!

PUBLISHED BY

Malin Klockare Gullesjö
Malin Klockare Gullesjö is working with Beta Family’s online profile. She has previously worked as a community manager on social media in the tourism industry.

This week I’ve interviewed the app development company DM Digital who are releasing the app game Hoops Rivals on iTunes and Google Play. We talked about beta testing and how hoops-rivals-logothey used this method to create a basketball app with crisp graphics and stimulating user interface.

Hi guys! Tell us about your company and what you’re doing.

DM Digital, our company, is set up as two interwoven teams: one provides mobile and web development to local companies and startups (to bring home the bread), the other focuses on our own startup products. During these years we managed to grow slowly but steadily, creating a young, motivated and passionate 10-person team. We are in constant contact with a local university (me and our CEO Daniele Grassi wrote an academic paper last year) and we are very active in the community.We are not very consultancy-like people: we feel that Italian companies strongly undervalue well-crafted software and the “price-is-the-only-thing-that-matters” mentality we perceive around us constantly frustrates us. That’s why our dreams, hopes and longest hours are invested on our own startup ideas. How come you started with app development?</strong>We started building mobile apps in 2011 because we strongly felt that the market had a clear growth potential, and we were extremely engaged by the endless interaction possibilities that mobile apps provided to users.

DM Digital, our company, is set up as two interwoven teams: one provides mobile and web development to local companies and startups (to bring home the bread), the other focuses on our own startup products. During these years we managed to grow slowly but steadily, creating a young, motivated and passionate 10-person team. We are in constant contact with a local university (me and our CEO Daniele Grassi wrote an academic paper last year) and we are very active in the community.We are not very consultancy-like people: we feel that Italian companies strongly undervalue well-crafted software and the “price-is-the-only-thing-that-matters” mentality we perceive around us constantly frustrates us. That’s why our dreams, hopes and longest hours are invested on our own startup ideas.How come you started with app development?We started building mobile apps in 2011 because we strongly felt that the market had a clear growth potential, and we were extremely engaged by the endless interaction possibilities that mobile apps provided to users.

How long do you stick with an idea for an app before giving up on it? Do you prototype, do you test your ideas on people etc?

We think that early prototyping is the key to test ideas right away without risking to commit too much time to a losing one. We try to build functional prototypes using at most 20 % of the time-budget we allocate to a project, then test the ideas on friends, family and involving people who can be considered domain-experts in the context of the idea.

What motivates you and what are your goals with the company?

It would be too easy to say that we want to revolutionize the world with our ideas. The truth is that we love creating new things, and we get a great satisfaction in the process. We love nice and smooth user experience and solutions that make you say “wow!”, and keep you engaged with them. That’s our goal, providing great experiences to our users and keeping our creativity flowing.

What would you say are the top three skills needed to be a successful app developer?

Being a successful app developer means having the right mix of different skills: Great sense of design, great sense of technology and a great sense of the market.

What have been some of your failures and what have you learned from them?

In 2012 we launched a startup called inbooki that allowed people to write and read immersive and interactive e-books. “inbooks” could be written to change their content and their story depending on the reader’s choices, their profile and environment. We won a couple of awards for this app, but we made the mistake of focusing on the Italian market (too small) and of listening too much to so-called “business advisors”, instead of concentrating our efforts in providing the best possible experience to our user base.

What advice would you give to someone who wants to be a successful app developer?

Listen to your gut, test your ideas as soon as possible and iterate rapidly.

Can you tell us a bit about your new app and how you got the idea for it?

Our developers and CEO have a great passion for basketball but sometimes is not just possible to go out an play against your friends for different reasons: Bad weather conditions, lack of free time, etc. We believed that developing a basketball management game could have helped us to play basketball everyday.

As there were no realistic basketball management game on the market we believed that that could have been a great chance to build the first basketball management game for mobile devices. We wanted to make a realistic game in order to help young basketball players to understand tactics and stats in basketball and help them spread this sport worldwide.

Like all sports games, the quest for the number one spot sits at the center of Hoops Rivals goal. Once players download the app, they will love its crisp graphics and stimulating user interface.
Assembling a team is simple, making that team a wolf pack of winners is the challenge. True fans of the game will welcome that challenge, as they develop rivalries across the leagues and watch their amateur players grow into statistical giants. Users can create and manage their basketball team, purchase players from the global market, and train them according to the manager’s preferences for play style, strategy, and tactics.

Side activities include upgrading your stadium and facilities to accommodate your growing fan base, earning tickets and cash from sponsorships, prizes, player sales, and game victories. Managers can determine the aggressiveness and rotation of their players, with complete access to all player stats at every instant.

Daily player training encourages team growth and performance and enhances overall team standings. Pre season drafts each month ensure each manager 2 new players with fresh stats to grow from. Once assigned a team, managers will all have their unique set of players to begin with. What happens from there is the essence of the excitement that this radical new game promises.

Regular season team match ups can be complemented with friendlies coordinated with other managers. Stats include rebounding, steals, 2 pt shooting, 3 pt shooting, free throws, fatigue, form, and much, much more. Managers and their teams will “level up” after each season, giving them a draft advantage based on league averages.

What problems are you facing right now?

We have made a soft launch of the game and we are already getting an unexpected number of daily downloads. We are now trying to work on ASO in order to rank higher in both iTunes Store and Google Play as we believe that many people could appreciate our game.

What do you feel are the positive aspects of using a crowdtesting platform for your beta testing?

We believe that using a crowdtesting platform is essential to test an app under different realistic platforms which makes the test more reliable, cost-effective, fast, and it’s also easier to identify bugs. In addition to that, crowdsource testing is great to target a specific group of tester thanks to the different filters that can be applied when looking for testers.

What challenges are you facing in the beta testing process? Can Beta Family can help with them?

It’s not easy to find testers that are interested in giving an honest feedback. Our main goal was to find out what needed to be edited, removed or added in our app. We weren’t just looking for people playing the beta version of our game before releasing it; we wanted feedbacks outside our office.

Beta Family allows us to find many testers for both iOS and Android devices and thanks to the easy reporting system, we received plenty of honest and inspiring feedbacks.

Can you tell us more about your testing plans?

Our official testing plans are ended as our game is now available for iOS and Android devices. However, as we are the first users of Hoops Rivals, we are running performances and usability test every day.

Why did you choose Beta Family over other similar services?

We chose Beta Family as subscribing and starting our first tests was straight simple. Many other services require too many information and too much time to run a test. We also chose Beta Family as exporting and analyzing users reports was very intuitive and fast.

I hope you enjoyed this interview on beta testing with DM Digital. I like how they learned to concentrate their efforts on providing the best possible experience to their user base instead of listening to others about how to run their business.

PUBLISHED BY

Malin Klockare Gullesjö
Malin Klockare Gullesjö is working with Beta Family’s online profile. She has previously worked as a community manager on social media in the tourism industry.