24 August, 2016
Let’s get physical….physical
You may have been tempted by virtual solutions to your problem, and you may have valid reasons for using them, but we here believe that there is nothing better than the real thing.
5 key reasons for using physical devices for mobile/compatibility testing
There are certainly valid reasons to consider using emulators as a means of testing. There are many free emulators on the market that offer various screen resolutions or device mimicking that allow you to view what your product would look like on a specified device. Another reason emulators are commonly used is that typical user interface issues found on real devices, such as screen sizes, have been alleviated. They further allow for real time data extraction which provides developers with the necessary information required to debug any issues. Emulators clearly provide benefits to the testing process or they simply wouldn’t be used. However, we believe that they are not sufficient enough for the final testing of a product purely because they are not able to cover all of the functionality that needs to be tested to ensure that the product is fully compatible. From our own experience, testing on ‘physical’ devices provides the clearest, most accurate, results. In doing so you avoid a common pitfall of using an emulator which is finding a false positive or negative result. Although it is completely impossible to cover the full range of devices available, here are 5 key reasons why we feel that you should still use real devices.
1. Look and Feel
Emulators cannot sufficiently guarantee the look and feel of an application. You will also not be able to adequately judge the applications responsiveness on a particular device via emulation. Other look and feel emulator drawbacks are that you will be unable to test the brightness of the display, the pixel density, colour scheme under varying lighting conditions or while moving around. Device orientation is an another key reason to use physical devices. Emulators are simply not reliable enough to confirm how your application responds when the device is being switched from portrait to landscape or to replicate the real world user experience.
An emulator will not be able to reproduce the the true user experience of a touch screen device. Traditional computers utilize the power and precision of mouse control, something which is not used in every day life by those using their phones/tablets to access applications. With the continuing increase in handheld devices being used over traditional PCs and laptops to access information, execution is now more commonly performed by hand. This is both slower and less accurate, so it is important that the UI reflects this form of operation. For example, with an emulator you are unable to replicate the pinch/zoom functionality that is widely used.
It’s called a “mobile” device for a reason. Testing using emulators does not provide an accurate account of the scenarios/conditions the application will face when being accessed by a user who is moving around. They are unable to account for the real world effects such as location, carrier signal strength or simply the difference between indoor and outdoor conditions. An emulator is unable to show you the impact of low signal strength, 3G or 4G networks, different Wi-Fi speeds, switching from network to WI-FI or offline use.
Most emulators will use a vanilla version of the OS that you would do your compatibility/mobile testing on, and as manufacturers and network providers will add their own middleware to the stock OS user journey steps can be missed. The mobile sector is much more fragmented despite being newer, the key providers are Android, iOS, Symbian and Windows. Developers have to work in multiple development environments, this complicates the development and testing processes. Newer OS will have the capability to support newer functionality, actions and rendering, something that older OS versions will not. Emulators are unable to provide an account of this difference.
5. The Hardware
Memory related issues are commonly found in compatibility/application testing. Device fragmentation has increased significantly over the past year with over 4000 more device configurations found this year compared to last. Emulators have difficulty replicating the user experience between high and low end devices. This is because the processor of the PC is ten times more powerful than that of a given android device. While emulators are able to test some senor functionality they are highly limited and requires a physical device to ensure the application is fully tested. Finally, there is no other way other than using the physical device itself to test the power consumption and efficiency of your application. If your application causes severe strain on a user’s device they are highly unlikely to continue using it.
One of the key reasons many organisations choose to use emulators over physical devices is the cost of procuring enough devices. However, any real ‘physical’ device testing is preferable to none. Emulators are a useful (and cheap) tool for testing devices, and certainly have their place, but to provide an accurate true to life evaluation of user experience and an in-depth analysis of functionality, there is no substitution for a real device.
A potential solution to this would be to hire an external test lab such as the one provided by Edge Testing Solutions.
By Brian Gilmour, Test Analyst & William McGavin, Digital Test Hub Manager, Edge Testing
Back to Blog