Mobile Testing Strategy for Enterprise apps

Smartphone and tablet adoption has pushed the growth of mobile application in last couple years. People are spending more and more time on these devices and it has slowly replaced any other device as primary means of communication. Compared to web, mobile space is very challenging when comes to testing. Gone are the days when you had enough time on hand to develop a feature and push to production after thorough testing.

Even though most of the mobile app has only couple of screens (may be 5 or 6 max if it’s transactional in nature) and very limited input mechanism (mostly dominated by keyboard input or selecting by scroll) then what makes it complex?

Most of the mobile app (especially enterprise mobile app) has the need of releasing faster and releasing frequently. This is primarily due to new feature being pushed or user feedback incorporation. It could also be due to UX improvement or changes. So when we are talking about mobile app testing we need to have a strategy which supports:

• rapid feature push
• continuous testing
• device coverage
• team collaboration

While creating we should be leveraging our learning from web automation and pick all the concepts which are proven and working. For a test strategy to support rapid feature push and continuous testing, we need to have smoke test and functional test in place. While smoke test will make sure build has not broken any critical scenario, functional test would make sure that all major flows and scenarios are working via thorough testing.

Every tool has their own strength and limitation. While appium let’s you write test which are device agnostic, it’s limited in testing only with in app. For example appium (till version 1.5.0) doesn’t support testing toast messages on android. However, if you have to test toast messages, Espresso would be the right choice of tool and hence for building comprehensive unit test. So smoke test should be running on a build which has passed unit test, thereby subjecting the app to device level test (comprising of business critical flows ) and service level test (business critical end points). This will make sure that your app’s core functionality is working and would build confidence.

Functional test at the same time should be very comprehensive and test all the flows of the application. Functional test should also be testing app behaviour by simulating network condition (turning data off, GPS off) etc. Majority of the mobile app needs data connection and t’s important to capture how they handle data/app flows when network is off or limiting. 

Also when it comes to mobile there are tools like appium, calabash which let folks explore functional test automation but at the same time it’s important to promote team collaboration via the solution. Especially for mobile app I would strongly recommend users to try cucumber style test so that right from dev to product manager everyone should be able to read and comprehend the test report. Below is a snippet from Cucumber website on how the automated script would look like. 

The idea is down the line product manager should be comfortable in writing the user scenarios which should serve as an input for quality team. Also the tools are evolving in a way to promote team collaboration for example there is a jenkins plug-in which post the build results on to slack, which is really cool. I am sure they will improve it to share test results as well.

While we have talked about importance of Smoke Test and how we could have a functional test to promote team collaboration, it’s time to also touch upon the very critical aspect of releasing app on devices, which is device coverage. Android throws in a very interesting problem to it’s developer group which is device fragmentation. Let me quote some pictures than text to give you a rough idea of what problem I am talking about. Below is the API distribution posted by google recently.

If you see the above break up of OS version, you would notice that to cover 80% of user base you need to have your test running on API level 16 to 23, this itself is a big problem. Also you don’t know which device family are being used by this fragment of user base. There are lot of popular ones like Samsung, Sony, Xiaomi and there are many which are unknown. If we see the brand fragmentation it will further worsen our problem.

So how do we deal with this?  There are device cloud offered by Google, Amazon and SauceLabs but even then which devices to run test on and which all features? One pays for every minute of device and hence the need is not only of automated test but “performant” test which do not have any place for false positive and false negative.

Your test strategy has to answer all this questions satisfactorily. So basically it boils down to two things:

• how much of testing to be performed ?
• on what devices this testing has to be performed?

Now certainly manual testing is not going to help you out seeing the OS fragmentation and device size fragmentation. If you have been relying on manual testing you were just lucky as you were siting on a time bomb which could explode any time.

You need to have a well thought of strategy to handle how much to test and what to test on. And majority of it goes down to how mature the test framework is to handle above mentioned things? Some of the asks from test framework would be:

• able to run code on device or emulator
• able to trigger test on locally connected device
• support parallel test between devices
• support easy integration with device providers in cloud

I have summarised a probable approach for app automation (in the pic below), which is just one piece of the mobile testing strategy. One certainly needs to look into their organisation need and demand to put in other components in the strategy which helps them release faster and release reliable product.

Please feel to reach out to me on nishant@testvagrant.com