Why you need strategy for mobile apps testing?

Authors: Tomasz Soroka

16.07.2015

Introduction


At Leaware we already developed more than 70 mobile apps using Xamarin technology. Most of them are multiplatform, means developed in most cases for IOS, Android and Windows platform.
In recent years we spent a lot of time against finding best way to test our applications in different stages of their life cycle.

Most typical stages during app life cycle is like on following picture.

Application development


In this stage we work a lot on developing app from scratch. Typically we work in scrum approach, so this means that we work in sprint cycles, where at end of every cycle we deliver to customer a shippable product. 

But we know that later or faster we will have to perform a transition of our development effort into something what we can name a product version 1.0. During this stage we transit development into alpha version, then into beta, then into release candidate and finally into first production release. 
Is means that from beta version we are more focused on polishing our product, we are not so focused on adding new functions, but more on fixing these which we have already,  doing some change requests … still using scrum approach.

When we are in release candidate stage, then we in most cases don’t add anything new, event don’t implement change requests. We are focused on polishing, fixing and improving stability, scalability, user interface and so on.

Here is how it looks:


How about testing?


Testing in Leaware is a strategy. This is really important for us. It doesn’t matter that we did 1000 functionalities well, when 1001 doesn’t work. People typically see this was is wrong, event if’s just one thing on thousands of others :) . It’s a human nature. And this is a challenge for software development teams.

Especially in mobile…
Some facts:

We have on market many different devices with different screen sizes




So many different screens, resolutions, proportions….

Devices are very fragmented


•    Android – more than 19000 unique devices on the market!
•    IOS – more than 20 different combinations of different configurations on the market

In US You have to test your app on more than 134 different devices to cover 80% of devices market


Who has so many different devices, different configurations? Who is testing apps that way? 


How we do it at Leaware ?


As you remember we internally divide development into two stages:

We also divided testing into two stages.

First one is manual testing. We perform manual testing in early development stage. It means that developers and testers test app manually on every commit, and of every sprint. We use typical test case scenarios testing as well as clicking app also by others, where for example we give app to chosen users and lets them use the app as it is.
Such testing is helpful for finding bugs on early stages which could be harmful for example during sprint presentation to Customer. But it’s not a Leaware quality :) as well as strategy.

This is why we use automated tests on real devices.


What’s a difference between manual testing and automated testing? It’s something totally different… 
Some examples:
When we use automated testing we can test app for example on 300 DIFFERENT devices in 20 minutes. In manual… eghh 300 DIFFERENT devices? Impossible, no one has so many.

When we use automated testing we can choose 300 DIFFERENT devices from more than 1000 DIFFERENT devices which are in the cloud – in manual? Impossible!

We can perform a test scenario on for example 100, 200, 300, 400 different devices in parallel – such test is performed in average 10-20 minutes (with reports, information about bugs and so on).
Let’s calculate performing such test manually. Let’s imagine
We have 50 different configurations in company (we don’t even think about 100 or more –it’s almost impossible in typical software house)
We have one person who is doing the test.

So let’s assume:
Taking device – 1 minute
Connecting device to computer – 1 minute
Loading a build of app to device – 5 minutes
Opening test scenario excel file – 2 minutes
Performing a test – 10 minutes 
Filling bug reports document – 10 minutes (bugs description, print screens)
If we have more bugs – another 5 minutes for filling 
STOP.
Assume = 1+1+5+2+10+10+5=34 minutes.
Its optimistic scenario and it doesn’t cover work of developer after getting a report regarding checking if bugs are repeatable or not, on which exactly devices bugs were found and so on.

But okay. Let’s say that such test takes 30 minutes so if we have to perform a test for 50 devices, our person will need 30*50=1500 minutes = 25 hours, so seems like 4 days of testing.

SO THIS IS WHY MANUAL TESTING IN LEAWARE IS NOT A SOLUTION. It’s only helpful in early stages of development where we are focused only on one, maybe two devices which are used by us for showing progress to customer.

When it’s going to RC stage or PRODUCTION we need a way of proving that application will work properly on thousands of different configurations. This is why we test our apps on cloud of real devices.

If You are interested in testing your apps on many different real devices, contact with us. We will show You how much more differences.

For example Do You imagine to get screenshots of every screen of app during testing? To find bugs like squished images, keyboard overlays and so on? In manual testing its almost impossible, in our approach – it works. See example:




There is much more differences, contact with us to get detailed information.

Application is already released

Great day for every team of developers… as well as for hole company  and of course for our customers as well. 
Let’s say that we released an app a month ago. No bugs, everything was working well. No ANY CHANGE IN CODE in last 30 days.

But what happens when Apple release new version of IOS or Google new version of Android? Are you prepared? 
What happens when any producer will release new version of smartphone or tablet?

Are you prepared for that?
Example:
MobileTAS application. Released several months ago for some enterprise customers and then Samsung Galaxy 6 Edge appears on market.

Normally we wouldn’t do anything, but our mobile testing strategy push us to test every device on new smartphones when only they appears on market.
So we did a test:



And what happened… Application didn’t even start on Galaxy S6 Edge.

So thanks to our strategy we found problem with application before MobileTAS users.  Without strategy for testing – it wouldn’t be possible.

Conclusion? We test our mobile apps all the time, not only during development, not only when we release something new… It’s not enough in current sophisticated world of mobile.

If you wish to get more information – how we can help with testing your apps, please contact us.