According to analyst firm App Annie, the app economy will be worth $6.3 trillion by 2021 — that's a 380% jump from the $1.3 trillion app market of 2016. In order to account for this tremendous growth Quality Assurance (QA) departments must quickly scale experienced resources to meet demand.
In this fast-paced software development environment many unique QA challenges exist. Important factors beyond hardware and software to take into consideration for Mobile App testing include:
Device variation is the #1 mobile app testing challenge due primarily to compatibility issues, as a mobile app can be deployed across different operating systems (iOS, Android etc.), manufacturers (Apple, Samsung, Google, LG, HTC etc.), keypad types (virtual keypad, hard keypad) or the Display/Screen resolutions. Furthermore, QA cannot be 100% sure if a tested app working well with a given device will run smoothly on other devices.
Experienced test engineers know it is impossible to cover every possible angle in Mobile App testing. It is not practical to expect to cover a universal combination of devices and OS versions, so it is best to target the most commonly used devices within the targeted demographic in order to derive the greatest value from your testing.
Using the USA demographic as a sample case, we have identified various categories:
The table below captures the market share in US for the iOS and Android Smartphones based on the OS Versions in use, Display Size/Screen Resolution and different Vendors:
Based on the above statistics, our test strategy is to create a matrix for the iOS and Android platforms with different OS Versions, Vendors and models.
In the case of iOS, the complexity is lower as compared to the Android. The table below shows the iOS devices and versions targeted for achieving sufficient test coverage for an iOS App to be released in USA demographic.
Note: The # of Test iterations is based on the type of target devices on which the App needs to be tested.
The below table shows the Android devices and versions targeted for achieving sufficient test coverage for an Android App to be released in USA demographic.
Note: The # of Test iterations is based on the type of target devices on which the App needs to be tested.
The above strategy enables a test team to provide predictable test coverage for the USA demographic. To increase test coverage you’d increase the number of Mobile Test Devices (from different vendors with different OS and OS versions), which proportionality increases the number of testing cycles. If testing is done manually the schedule and cost will be a major challenge. In such a scenario, Test Automation becomes a necessity with maximum benefit derived from its inclusion as an integral aspect of the design process.
