Written by on 8 September 2015

How we test our mobile website

In the last three years, we’ve seen the number of mobile- and tablet users grow substantially. From 19% in 2013 to 50% of our current total visitors are on mobile devices. When we began making our mobile website (not responsive). We started off with one break point consisting of 320px minimal and 668px maximum width. Before we had our device lab, the way to test our mobile site was on a few physical devices like an iPhone 4 and a Nexus, emulators like BrowserStack, and just by scaling down the browser’s size.

Scaling the browser

Scaling the browser down is fast and easy as it’s part of our development flow. But the representation on a desktop screen is especially higher than a device’s screen itself. Compared to your fingers, the mouse or trackpad is much more accurate as well. So when we actually did test on a device, it seemed that the interface wasn’t built as good as we  thought. Especially users with a bigger fingers. Sometimes wrong buttons were pressed or just didn’t work at all because of the overlapping hidden elements blocking the interaction. Basic bugs in the difference in how the viewport reacts to JavaScript calculations were one of those things that you cannot test well on a desktop. The speed difference is huge compared to (older) mobile devices. However, the good part is that you can use development tools for profiling on desktop will help to remove bugs and performance on mobile devices.

  • Pros:
    • – Fast development
    • – Good code inspection tools
    • – Easier to test screen variations in height and width
    • – Cheap
  • Cons:
    • – Screen is bigger than physical device
    • – Mouse interaction does not represent touch screen interaction, even not with a bigger cursor
    • – Not ideal for testing touch events
    • – Fast CPU/GPU does not represent mobile device
    • – Browser/OS quirks are not visible

Testing with emulators

Actual device screen versus emulated device screenWhen you want a better representation
of a device, testing on an emulator can be quite useful. The quirks of a browser
and operating system of a device will be faster noticed. You can test touch events and use most sensors unlike the desktop browser. The added benefit is to be able switch must faster between devices when using an emulator hub like BrowserStack. The downside is still that emulated device speeds are not the same. The issue screen and pointer device for desktop counts with the emulators as well. The emulated screen is bigger than the actual device and the touch emulator does not represent a finger that well. But even more important: not all devices, operating systems and browsers are available. An example why this can be an issue is the following:

The main call-to-action button in the checkout process was falling off the screen. This caused a substantial conversion drop on tablets. We actually found that out during a business Intelligence presentation. The simple cause was we just did not test thoroughly enough and not had the right diversity of devices available.

Another example is Chrome for Android and the native Android browser have a different implementation for scaling to fit the screen. Switching browsers was not productive when working just in BrowserStack. If you can’t do a side-by-side comparison,
some quirks will just not be visible enough.

  • Pros:
    • – Better device representation than desktops
    • – Sensors accessible
    • – Easy to switch browser & operating system
  • Cons:
    • – No side-by-side comparison
    • – Device speed not representable
    • – Hardware of devices is emulated, and they are not identical
    • – Harder to set-up if you have an internal development environment
    • – There are some costs involved
    • – Your specific need for certain device/os are maybe not available

Testing on physical devices

With physical devices, you have the best real-world examples. We can test the actual screen sizes and how using different finger surfaces work. We can test performance and spot quirks in different browsers, operating system specific quirks easy. But there are a lot of downsides. Filling in the URLs on the devices by hand or by QR code is time consuming. When you don’t have the devices organized, they can end up without power, missing cables, unwanted upgrades and malware. There is no efficient way to do remote code inspect all devices on the fly. You need to hook up the device to your machine, perhaps install drivers, in order to inspect code. Side-by-side comparison is great,
but hard when timing page load.

  • Pros:
    • – The real deal: testing real-world software and testing real-world interaction.
    • – Side-by-side testing
    • – Able to program for sensors
  • Cons:
    • – Expensive
    • – Device individual entering the URL (by hand or QR code)
    • – Harder to use development tools
    • – Maintenance (cables, power, network, availability)
    • – Time consuming

Final thoughts

Our biggest lesson learned is desktop browser scaling and emulators are good for initial development. They however, do not sufficiently represent real-world devices both in technical aspect as well as from a user’s experience point of view. Developing a responsive “mobile first” website is a great method to ensure that the most important content will be on-screen. Physical devices are irreplaceable when designing and developing your when your substantial user base have mobile devices, and are the only real representation of your website. Unfortunately, physical testing can be an expensive, time consuming and tedious task.

COMMENTGive your two cents.