Brief Intro: There was a time when everything was manual, then came automation and now the device farms. I am talking in the context of Mobile app testing. When you change focus from Web to Mobile, one of the thing which ideally should get detailed attention from you is “device coverage”.
Why device coverage matters: Let me quote some numbers for Android OS first to give you a rough idea. Below is a snapshot from OpenSignal. According to the report, android device fragmentation has increased 28% over last year.
When it comes to iOS the scene is relatively better. Below is the screenshot of iOS to Android comparison in terms of fragmentation.
Probable solutions to device Coverage: The good news is there are lot of vendors offering device farm and to name the prominent ones:
Xamarin Test Cloud Service
Cloud Test Lab by Google
AWS Device Farm
But the not so good news is most of these are paid service and the charges for engaging a device could go up to $240 per device per month (on AWS device farm for example). Till device farm became easily accessible and popular, folks were majorly testing on emulator and some handful of devices. Beta testing phase used to be a little longer and most of the bugs were found during that and field testing of app.
Now that device farms brings you plethora of devices but the concern is every minute of a device in a farm costs you. What should be your strategy for mobile testing and automation given the limited testing time you have and the money to spend on?
Contextual information about the app: A lot in your strategy would be shaped by what kind of app it is ? Is it a B2B or B2C ? Is it a hybrid app or native app? Do you know your targeted customer segment ? Do you know the age segment of those customers ? Possible usage patterns of your app?
Once you have these info, you could potentially decide what are the typical devices which might get used. There are other set of questions which can definitely lead you to some pointers to test strategy for mobile apps.
Should you invest in devices locally ?
What should be your emulator configuration to run test?
Can your functional test run in parallel ?
Which section of functional test you should run for device coverage?
What devices you should opt when using device farm ?
When should you run your tests on Device Farm ?
Probable approach: We have tried to diffuse how you can help shape your mobile testing strategy in the below picture. It’s more to prepare you and open you up for discussion which should help you arrive at your device testing strategy.
Apart from the test strategy you also need to have a highly optimised code where you are handling “wait” smartly in your test code. Remember that with each minute of your test execution on actual device, there is a cost associated.
Want us to help you with your mobile testing strategy and automation, reach out to us.