Mobile application testing is a process by which application software developed for handheld mobile devices is tested for its functionality, usability and consistency. Mobile application testing can be an automated or manual type of testing.Mobile applications either come pre-installed or can be installed from mobile software distribution platforms. Mobile devices have witnessed a phenomenal growth in the past few years. A study conducted by the Yankee Group.Predicts the generation of $4.2 billion in revenue by 2013 through 7 billion U.S. smartphone app downloads.
- Functional Testing
- Laboratory Testing
- Performance Testing
- Memory Leakage Testing
- Interrupt Testing
- Usability testing
- Installation testing
- Certification Testing
- Security Testing
- Location Testing
- Outdated Software Testing
- Load Testing
- Black box Testing
Description Of Application Testing
- Functional Testing: Functional testing ensures that the application is working as per the requirements. Most of the test conducted for this is driven by the user interface and call flow
- Laboratory Testing: Laboratory testing, usually carried out by network carriers, is done by simulating the complete wireless network. This test is performed to find out any glitches when a mobile application uses voice and/or data connection to perform some functions.
- Performance Testing: This testing process is undertaken to check the performance and behavior of the application under certain conditions such as low battery, bad network coverage, low available memory, simultaneous access to application’s server by several users and other conditions. Performance of an application can be affected from two sides: the application’s server side and the client’s side. Performance testing is carried out to check both.
- Memory Leakage Testing: Memory leakage happens when a computer program or application is unable to manage the memory it is allocated resulting in poor performance of the application and the overall slowdown of the system. As mobile devices have significant constraints of available memory, memory leakage testing is crucial for the proper functioning of an application
- Interrupt Testing: An application while functioning may face several interruptions like incoming calls or network coverage outage and recovery. The different types of interruptions are:
- Incoming and Outgoing SMS and MMS
- Incoming and Outgoing calls
- Incoming Notifications
- Battery Removal
- Cable Insertion and Removal for data transfer
- Network outage and recovery
- Media Player on/off
- Device Power cycle
An application should be able to handle these interruptions by going into a suspended state and resuming afterward.
- Usability testing: Usability testing is carried out to verify if the application is achieving its goals and getting a favorable response from users. This is important as the usability of an application is its key to commercial success (it is nothing but user-friendliness).Another important part of usability testing is to make sure that the user experience is uniform across all devices. This section of testing hopes to address the key challenges of the variety of mobile devices and the diversity in mobile platforms/OS, which is also called device fragmentation. One key portion of this type of usability testing is to be sure that there are no major errors in the functionality, placement, or sizing of the user interface on different devices.,
- Installation testing: Certain mobile applications come pre-installed on the device whereas others have to be installed from the store. Installation testing verifies that the installation process goes smoothly without the user having to face any difficulty. This testing process covers installation, updating and uninstalling of an application
- Certification Testing: To get a certificate of compliance, each mobile device needs to be tested against the guidelines set by different mobile platforms.
- Security Testing: To check for vulnerabilities to hacking, authentication and authorization policies, data security, session management and other security standards.
- Location Testing: Connectivity changes with network and location, but you can’t mimic those fluctuating conditions in a lab. Only in Country non-automated testers can perform comprehensive usability and functionality testing.
- Outdated Software Testing:ost: Not everyone regularly updates their operating system. Some Android users might not even have access to the newest version. Professional Testers can test outdated software.
- Load Testing: When many users all attempt to download, load, and use your app or game simultaneously, slow load times or crashes can occur causing many customers to abandon your app, game, or website. In-country human testing done manually is the most effective way to test load.
- Black box Testing: This type of testing doesn’t include the internally coding logic of the application. Tester tests the application with functionality without peering with an internal structure of the application. This method of test can be applied virtually to every level of software testing: unit, integration, system, and acceptance.
Difference between Tablets, E-book Readers, and Smartphones
Mobile devices are typically handheld computers. They have many variants based on their characteristics such as physical dimension, hardware, and software capability, what are they meant for, etc.
Take a look at the following table. It differentiates tablets, e-book readers, and smartphones based on their characteristics.
|What it is||Tablets are portable computer devices. Unlike traditional computers, they don’t have keyboards or mouse, however the entire screen is touch sensitive.||E-book readers—also called e-readers—are similar to tablet computers, except they are mainly designed for reading e-books (digital, downloadable books).||A smartphone is a powerful mobile phone that is designed to run a variety of applications in addition to providing phone service.|
|Used for||Almost all the jobs which we can do with traditional computers or desktops.||Reading e-books||Web browsing, watching videos, reading e-books, and playing games|
|Example||Samsung Tablets||Amazon Kindle, Barnes & Noble Nook.||Sony smartphones, Samsung smartphones, Apple iPhone.|
One thing is self-explanatory in the case of mobile testing. To perform mobile testing, you need a mobile device. This is to access that how our product will work and look like on a given mobile set.
Suppose we are developing an application for flight ticket booking system. Once the product is entirely developed, as a part of mobile testing, we need to check if the application is working as expected with all the majorly used devices like Android phones, iOS, Blackberry phones, and other different types of tablets and iPads.
of tablets and iPads.
To do this kind of check, we need to acquire each such device and then we can check if the application behaves as per expectation. Yes you thought right, as a product owner one will defiantly find this very expensive to procure such a large number of mobile devices and carry out testing. So is there any smart alternate available?
The solution to this problem is to use Mobile Simulators and Mobile Emulators. These are primarily software programs designed to provide simulation for important features of a smartphone. They are very similar in nature, so sometimes, they are used interchangeably.
How testing on an Emulator/Simulator is different from testing
|Real Device||Emulator / Simulator|
|Price||Getting real devices will cost you a lot.||It is almost free, we just need to download and install them|
|Processing Speed||It has faster processing; however network latency may be normal.||It is slower as compared to actual devices. It has observed less latency than real devices connected to the local network or in the cloud.|
|Debugging||Debugging is not that easy.||It provides step-by-step debugging of an application. Also, it provides an efficient way for capturing screenshots.|
|Web-app Testing||Web applications can be tested in a normal way.||Testing a web application is much easier.|
|Reliability||Testing on a real device has a major advantage that it always gives accurate results.||It cannot simulate all types of user interactions; hence it may lead to false results sometimes. So it scores low when it comes to reliability.|
A simulator/emulator cannot mimic the following features −
- Mobile device battery
- Mobile device’s camera
- Difficult to mimic interruptions like incoming calls and SMS.
- Not so much realistic simulation for mobile device memory usage.
Let us now understand more about mobile simulators and mobile emulators. There are specific differences between the two. The following table lists the major difference between a simulator and an emulator.
|What it mimics||Mobile device softwareMobile device hardwareMobile operating system||Internal behavior of the device.It does not mimic hardware.|
|How to get it||It is generally provided by the device manufacturer.||It is generally provided by the device manufacturer or some other company.|
|Internal structure||It is written in machine-level assembly language.||It is written in high-level language.|
|Debugging||It is more suitable for debugging.||It is not suitable for debugging purpose.|
|Performance||Emulators are really slow. Emulating the actual hardware usually makes the software run slower than it would natively.||Faster than emulators.|
|Example||Google’s Android SDK||Apple’s iOS Simulator|
The best choice for mobile testing
There are quite a few tools available in the market to make mobile UI testing smoother and simpler. For example −
- Google chrome extension
- Browser Stack
Let’s understand a little more about these tools and their usefulness.
Google chrome extension
It’s a free feature available with Google chrome web browser. We have given here a step-by-step explanation of how to test mobile web with Google Chrome Extension −
Step 1 − Open the website under testing in “Google Chrome Web browser”.
Step 2 − Press F12. It will open the Developer tool window, as shown in the following screenshot.
Step 3 − Click the mobile device like icon. Refer the following screenshot.
Step 4 − Select the mobile device with which you want to test the website. You can choose the different available devices in order to do the UI verification.
FRAMEWORK FOR TESTING
Robotium is an open-source test framework for writing automatic gray box testing cases for Android applications.With the support of Robotium, test case developers can write function, system and acceptance test scenarios, spanning multiple Android activities. Robotium can be used both for testing applications where the source code is available and applications where only the APK file is available and the implementation details are not known.
Benefits of Robotium
- Automatic timing and delays.
- Automatically follows current Activity.
- Automatically finds Views.
- Automatically makes its own decisions (ex: When to scroll, etc.)
- Easy to write, shorter code. Minimal time needed to write solid test cases.
- You can develop powerful test cases, with minimal knowledge of the application under test.
Drawbacks of Robotium
- Robotium cannot handle Flash or Web components.
- It handles only one application at a time.
- It cannot simulate clicking on soft keyboard using Robotium (need to use ‘enterText()’ to enter text into an EditText field)
- Robotium cannot interact with Status Bar Notifications − that is, pull down the Notification area and click on a specified Notification.
- Can be a bit slow, especially running on older devices.
Getting started with Selendroid
Selendroid is based on the Android instrumentation framework. Currently only the testing of one specific app is supported. Selendroid requires the following permission for your app:
- Selendroid can be used on Mac, Linux and Windows.
- Java SDK (minimum 1.6) must be installed and JAVA_HOME configured. IMPORTANT: If JAVA_HOME is pointing to a Java runtime environment, selendroid will produce errors because tools like the jarsigner are not available!
- Latest Android-Sdk must be installed and ANDROID_HOME set. If detailed instructions are needed, have a look at this guide.
- If you run selendroid on a 64bit Linux machine, please install:
- sudo dpkg –add-architecture i386
- sudo apt-get update
- sudo apt-get install libc6:i386 libncurses5:i386 libstdc++6:i386
- At least one Android virtual device must exist or an Android hardware device must be plugged in to the computer. For best practices about Android devices please read this section.
Getting an application under test
Selendroid can be used to test already built apps. Those Android apps (apk file) must exist on the machine, where the selendroid-standalone server will be started. The reason for this is that a customized selendroid-server for the app under test (AUT) will be created. Both apps (selendroid-server and AUT) must be signed with the same certificate in order to install the apks on the device.
If you want to play around with selendroid, you can use our selendroid-test-app which we use to verify that actually selendroid works as expected. You can download this app here.
Download selendroid-standalone.jar and cd to the folder where you downloaded the jar, and run:
java -jar selendroid-standalone-0.17.0-with-dependencies.jar -app selendroid-test-app-0.17.0.apk
Selendroid-standalone will start a http server on port 4444 and will scan all Android virtual devices (avd) that the user has created (~/.android/avd/). The Android target version and the screen size will be identified. If an emulator is running, it can be used since version 0.11.0. Even an emulator that has been started manually after the selendroid-standalone got started can be used. If there are Android hardware devices plugged in, they will also be added to the device store.
You can check that the application(s) and the devices are recognized by opening a browser and navigating to: http://localhost:4444/wd/hub/status.
You should see a result similar to this:
Tests are written using the Selenium 2 client API. For Java we provide selendroid-client libraries that are used in the following example:
SelendroidCapabilities capa = new SelendroidCapabilities(“io.selendroid.testapp:0.17.0”);
WebDriver driver = new SelendroidDriver(capa);
WebElement inputField = driver.findElement(By.id(“my_text_field”));
If, for example, you prefer to write your tests in Python, the same test can be found here, and here in Ruby.
In order to create a new test session in the desired capabilities, the id of the app under test must be provided in the format: io.selendroid.testapp:0.17.0. Please note, if the version is not specified in the id of the app under test then the latest version of the app will be automatically selected. Based on that information a matching Android device will be identified, otherwise the test session will throw an error and not start. Important: before filing a bug: please make sure the desired capabilities in your test match the supported apps and devices available.
After the found device has been initialized, a customized selendroid-server will be created and automatically installed on the device. The app under test will also be installed and the selendroid-server on the device will then be automatically started.
After the test session has been successfully initialized, the test commands such as ‘find an element’ and ‘element interactions’ are routed to this device. If the test session has ended, the emulator will stop automatically.
Desired Capabilities (SelendroidCapabilities)
We recommend the use of SelendroidCapabilities. There you find convenient methods like:
- new SelendroidCapabilities( … ): This requests an app for the test session without any preference about emulator or hardware device.
- SelendroidCapabilities.emulator( … ): This requests an emulator and will fail the test session if no emulator is available.
- SelendroidCapabilities.device( … ) This requests a hardware device and will fail the test session if no device is plugged in.
- SelendroidCapabilities.android( … ) This starts a mobile web testing session.
The SelendroidCapabilities are only available for Java projects. For other programming languages use the following properties in the DesiredCapabilities of the Selenium client apis:
For testing native or hybrid apps:
- Specify the app id of the app under test (aut). E.g. io.selendroid.testapp:0.17.0. If the version of the aut is not specified, then the latest version is assumed.
For testing Mobile Web:
- Value must be android to start the ‘Android driver webview app’.
- Please note that either aut or browserName can be used for a test session.
Optional properties in the desired capabilities
- If True, an emulator will be requested. If False, a hardware device will be requested.
- Specify the Android target API version of the device. E.g. for KitKat it is: 19
- Specify the locale of an emulator to be used in the test session. During the start the locale will be automatically configured. E.g. de_CH
- Specify the screen size of the device: e.g. 720×1280
- Specify the display number that is used to start the emulator on. Supported only in Linux platforms. E.g. 1
- Specify a list of adb commands that will be executed on the target device before selendroid-server will be started. E.g. shell setprop name selendroid, please note that the adb command itself and the serial will be added by selendroid automatically.
- Specify the model (e.g. Nexus One, Nexus 7, Galaxy S3) of an emulator when selecting a device.
- Specify the api library you need on your device. (e.g. google) if you need a device that has Google APIs.
Selendroid supports testing on hardware devices as well as using Android emulators. In the capabilities the properties are used to find the device for the test execution. The supported properties in the capabilities are listed here.
Example for selecting a device
For running a test on the Android target platform 18 and on an emulator which is simulating a Nexus 7, the capabilities for the selendroid-test-app are:
SelendroidCapabilities cap = new SelendroidCapabilities(“io.selendroid.testapp:0.8.0”);
Selendroid can start and stop Android Virtual Devices (Avd). It cannot create new emulators, they must be created manually by the tester. After an emulator has been created, we recommend the first start be done manually in order to be sure everything works as expected.
When creating ads, please read the following configuration recommendations:
- Whenever possible, use the Intel x86 ABI
- Install Intel x86 Emulator Accelerator on Mac and Windows and use KVM on Linux to enable hardware acceleration to massively speed up the Emulator
- Use at least 1024MB of RAM per emulator
- Use at least 32MB of VM Heap per emulator
- Configure the hardware keyboard to be used: hw.keyboard=yes
- Use the Host GPU. If only a black screen is displayed, please deactivate this option.
- The number of emulators you can run in parallel per machine depends heavily on the hardware of the machine you use.
Using Hardware Devices
- Please make sure the device has no screen lock configured.
- Devices must be plugged in via USB to the computer that the selendroid-standalone component is running on