In this article we emphasize the importance of asking open questions, encouraging your app testers to think freely and add their own input. We also discuss a few techniques for testers to narrow down root causes to potential issues.
App owners: Avoid Yes or No questions
Let’s face it – we’re all lazy. We’re designed by evolution to preserve our energy, so when offered an easy way out, we tend to take it. App testing is no exception, of course. If you as an app developer ask questions like “Do you think my app is user friendly”, the tester will conclude that a simple yes or no response will do. While the percentage of users agreeing might interest you, you’d be even better off knowing which complaints the unhappy bunch might have.
Naturally, it would be better to ask an open ended question that invites the tester to elaborate and provide more useful feedback. Ask Why, How and What (arguably the most relevant of the “Five W’s” of questioning. Feel free to ask the tester to compare your app with your competitors and cherry pick feature requests from them. You should strive to formulate your test instructions so that each test report will yield qualitative data, rather than quantitative.
Testers: Describe the root cause
The car manufacturer Toyota is widely regarded as an innovative company when it comes to refining work processes and methodologies. One of the techniques attributed to them is “Five Whys”. It’s an iterative process, repeating the question Why five times, each time trying to dig a little bit deeper until the root cause is found.
Wikipedia provides an example based on, of course, a car failing to start. The investigator begins by asking Why, and responds that the battery is dead. Then follows increasingly narrow reasons until the root cause is found: failure to follow the maintenance service schedule. Each Why questions the previous answer, funneling down until the bottom.
A common criticism against the Five Whys technique is that the end result is not reproducible – that is to say, two different people applying the method on the same problem are likely to end up with different root causes. This might be due to different a priori knowledge, experiences or opinions. While in some cases this could be problematic, it actually works to our advantage in the case of app testing, as all feedback should be welcomed.
Four Whys might be plenty
Let’s try out the Five Whys technique described above by applying it to a possible problem relevant to our app testing scenario. The following might be a little technical, and keep in mind that the technique works just as well on even smaller issues than this one.
Problem: The mobile app I’m testing drains the battery very quickly.
Why? (1)
It activates the Wi-Fi or 3G connection/antenna very frequently.
(This could be seen on the top bar Wi-Fi / data traffic icon)
Why? (2)
The app downloads a lot of data.
(This could be measured by the smart phone OS, usually in a Settings menu)
Why? (3)
It does’nt seem to cache data, so it re-downloads every time I return to the view.
(Again, the top bar icon)
Why? (4)
For the app’s purposes it needs freshly updated, current data.
(Let’s assume that the app is in this category)
Let’s stop right there, even though we haven’t reached the fifth level yet. At this point, we as testers could terminate the process as we can already use our knowledge about similar apps and their technology choices to provide a creative and concrete suggested solution to the problem. In fact, any of the following features would be helpful:
- Switch from periodic data polling to push notifications
- Offer an app user setting of the refresh interval
- Setting whether to only refresh often while on Wi-Fi connection and/or charging battery
- Et cetera…So now we have found the root cause to the battery drainage, a poor design choice, which assumes unlimited data quota and battery power. Typically such choices come from the developer not testing the app enough in the wild. The developer might think that everything works just fine, while sitting at his computer, charging the device and within Wi-Fi range. The Beta Family is obviously a helpful tool for getting more perspectives in these cases.Good examples, comin’ upOver the next few months, we will set out to interview a selection of our top-rated testers to get their input on how to deliver a great test report. This will give you as a developer the chance to hear from the other side of the fence, while testers can improve their methodology by learning from the best of breed. Watch this space!
PUBLISHED BY
Peter Skogsberg
Peter Skogsberg is an ISTQB certified software tester and holds a Master’s degree in Information Technology. He has been on both the development and the testing sides of several mobile and web applications. View all posts by Peter Skogsberg