It’s time to write about the research I’ve been doing to find a solution to the testing limitations of mobile testing on a desktop computer. I know that all of the issues could not be solved, but I wanted to see what we could accomplish with online software.
I think it’s important to first define the difference between Simulators and Emulators. While these two are nearly synonymous, there are slight differences that could assist in your determination to look into one product or another. Simulators imitate the appearance, design, or the basic features of a device, whereas an emulator will reproduce the features and actions of the device.
Lastly, my research was to find something that could resolve URLs locally, because we want to test our websites in lower testing environments before we release to production, so it’s important to be able to test our sites this way. That being said, let’s get started.
I’ve got to start with my favorite product that I tested with, and say the most about it. BrowserStack has it together. They offer a laundry list of devices to choose from, with different browsers to test on with each device. They have both emulators and simulators, so you can test on actual real devices, and also by simulation if the device is a bit older. You are also able to resolve URLs locally so we were able to access our sites in house. It allows automation testing using Selenium and Appium, using actual mobile devices. I didn’t look into this feature too much, I focused more on functional testing for my efforts. Lastly, it has a built in code debugger, and the ability to inspect and read the console, which is really nice for testing iOS.
A few notable items is that the trial is 30 minutes of functional testing, but this can be overcome by signing up with different email addresses once that expires. The trial only allows you to tap into a couple iOS devices, and a couple older Android devices. The trial is a little laggy and doesn’t always display the design without minor glitches, but it worked pretty well for my needs. On the plus side, they show the entire phone so it really feels like you’re testing on a device, and you get the full experience.
The cost of the full version can add up quickly, but they do have a variety of offerings to allow you to customize to your needs. The other downside of this product (and most the others) was that the OS versions are not the most up to date, but rather uses the OS version that was released on the phone you are testing (for example, iPhone 5 uses iOS 6.0). The last downside is their screenshot tool cannot resolve URLs behind a login screen, but that’s only an issue for our type of testing.
Overall BrowserStack gets my vote.
A product of SmartBear, CrossBrowserTesting has a lot of really great features to test with. Their trial is 100 minutes of manual testing, which allows you to get a lot done to see if you want to pursue the product further, however it only allows 5 minutes at a time, so you have to be fast. Like others, their trial only allows a few Apple and Android devices, but upon payment, unlocks a ton more options. They also have a lot of devices to choose from, and many of their devices are available for testing both as real emulators and simulators. They even offer real devices as far back as iPhone 3GS and Galaxy S3, if that is a need for you.
Their trial is extremely laggy, and pretty slow to load up (which cuts into your 5 minutes), but unsure if that is just the trial or if that would be the paid version as well. Another minor downside is that their viewable area is just the screen, not the full device, but that part didn’t bother me too much, it just doesn’t give you the entire feel of using the mobile device.
They offer a great screenshot tool that allows you to quickly compare up to 25 different browsers, and they’ll even tell you design differences that they find. This tool even allows you to go behind a log in screen by passing in a username and password (though I wasn’t able to get this to work on their mobile devices, only the desktop versions).
Overall this product was great, other than it being very slow on connection, and a 5 minute trial window, this obviously has a lot of great features.
GenyMotion is a pretty cool product, and is locally installed on your computer, which in theory provides some speed increases and doesn’t fully rely on a solid web connection. This product also allows you to test things like interruptions, battery usage, network connectivity changes, and more. It actually seems to do a lot of really cool things that were not available in the above products.
However, a few things to note is that this is only for testing Android devices, there is no iOS or other available devices. Also, because it’s locally installed, you are required to have the space available and the memory requirements. It also requires some additional software, and to run against local URLs, it requires a bit of server configurations.
This was a fun product to try, because SauceLabs offers every device and every OS version to all those devices. You simply select a device, and then an available OS and get started. However, it takes a very long time to spin up the testing simulator. It appears that you can test local sites, but it doesn’t come without a lot of server/proxy work, of which I was not going to test in my short research period.
The coolest thing about this was all the available devices and OS versions, allowing you to test different scenarios and configurations.
There was another handful of products that I tried that didn’t give me what I needed or were old and outdated. These products may help in some areas, but for testing locally, or needing to test on the latest device, they didn’t step up to the plate. Or, some required money up front or further work to even try it out, of which I also wasn’t interested in my research period.
- Requires $20 to download.
- Air iPhone
- Old and outdated. Not even worth the time.
- Xamarin Testflight
- Requires in depth knowledge of Visual Studio and runs locally.
- Old and outdated, does not work like I needed it to work.
- Safari browser
- Doesn’t work great on Windows and doesn’t do what I needed.
- More of an App builder and tester, not for testing our sites.
- I just could not get it to install, and then was contacted by their sales team a bunch.
- Sigos AppExperience
- Free trial requires meeting with a sales rep. I didn’t do this, but the app looks cool.
If you find yourself needing this type of testing software, I recommend BrowserStack. It costs some money, but will give you a lot of really great features. I was able to find a handful of bugs during my trial period as well, which was extra fun to realize the value before looking further into this product.