Menu
Menu
The diversity in parameters of available mobile devices requires software developers to optimize the view of applications for various screen sizes and resolutions. However, this process requires testing the application on different devices, which in turn brings the need of purchasing many devices, as well as separate installation of new versions of the application on different devices or emulators. Such a traditional method of testing mobile applications leads to numerous problems. In particular, the time needed for conducting tests increases linearly depending on the number of devices and along with the development of new versions of applications. To solve this problem, Xamarin provides suitable tools, among others, the Test Cloud service, which supports testing mobile applications using a large number of physical devices. The objective of this article is to present selected functions of the Xamarin tools for conducting automatic tests of mobile applications.
It is best to start the description of the genesis of this article by asking the fundamental question. Namely, why is testing mobile applications important? So, according to research conducted in the USA, it was estimated that for an application to cover 80% of the mobile market, it should be tested on a minimum of 134 different devices. This research includes the two most popular platforms – iOS and Android.
The chart of application share on the market depends on the number of tested devices. As you can see, the function shows exponential growth. This means that the percentage coverage of the market increases quickly with the number of devices used in the tests.
According to another research, if an application does not load in three seconds from the moment of running it, 60% of users will close it, and what is even worse, 43% of users declare that they will never go back to such an application. On the other hand, a correctly operating application will be recommended to others by more than 45% of users.
Mobile applications are easily available, a few clicks are enough to download and run them from a proper shop. The selection of applications is huge and quality is the keyword in the process of creating them, as it is.
The quality that is decisive in the success of a given application. One of the elements which guarantee high quality is proper testing. In the case of mobile applications, however, it is not an easy job. One of the fundamental problems is the diversity of screens of mobile devices and a very large fragmentation of the market, which includes:
different versions of operating systems (iOS, Android, Windows Phone, WinRT, Windows 10), different versions of devices (smartphones, tablets), different types of smartphones (different resolutions, screen sizes, types of processors, amount of memory), different types of tablets (different resolutions, screen sizes, types of processors, amount of memory), different versions of operating systems of the same producer, e.g. Android from version 2.3 to version 6. X, iOS from version 7 to version 9. X, Windows Phone 8.0 – 10.
Considering the number of possible combinations of problems that may arise during the development of the application, it turns out that the implementation of a properly working application is quite a challenge. For this reason, testing the application becomes a very big challenge for the developer.
There are a lot of ways to test desktop and web applications, starting with manual tests – popular clicking, through crowd testing, and finishing with automatic tests (unit testing). The scope of areas for testing in mobile applications is similar, starting with individual tests, through integrating tests, system tests, and finishing with automatic UI testing, in which software simulates typical actions performed by users using the user interface, e.g. clicking (touching) buttons, selecting elements of drop-down lists, scrolling views, etc.
Manual tests are the most realistic, however, they require a lot of time for the realization of individual iterations. Traditional individual tests make it possible to reduce the time of a single cycle at the expense of reduced realism of the test. Automatic UI tests guarantee high realism, and at the same time, they reduce the time needed for carrying out a given test. Based on this comparison we can see that automatic UI tests seem to be a good solution during the process of application development. For this reason, we are going to focus in this article on automatic UI tests. In particular, using Xamarin tools will be discussed. These tools are not limited exclusively to mobile applications created based on Xamarin technologies.
Xamarin provides two basic tools for conducting automatic tests. The first of them is the framework for writing tests (Xamarin.UITest). It allows to write code that emulates the user’s behavior and it allows to conduct a test on an emulator or a device that is connected to the computer. The second tool is Xamarin Test Cloud (XTC). This tool allows conducting a previously written test in a cloud on over 2000 real devices. In other words, XTC provides an interface that automatically installs and then runs a given application on all of the selected devices. As the final result, XTC presents a report from conducted tests.
The main objective of the Xamarin tools is the automation of the testing process and performing tests on a large number of different devices. In a basic approach, tests are designed, run, and adjusted locally, in particular on a physical device connected to a computer or an emulator, after that the same test can be published and executed in the XTC cloud on hundreds of selected devices.
Writing a test starts with creating scripts in C# or Ruby. In this article, we are going to present an example test designed in C#. So, we create a test, whose purpose will be to check if a given button fits (is visible) on the screen of a mobile device. For a specific example, we created a relatively simple application designed in the Xamarin Forms technology. The application consists of two screens. On the first screen of the application, there is a button labeled Start app, whose only purpose is to switch the user to the second screen (navigation to the other view), which displays the text reading Hello!
Let’s create a test whose purpose will be running an application, turning the screen to the horizontal position, and pressing the button labeled Start app. This will result in displaying the second screen of the application reading Hello! The last element of the test will be returning to the welcome screen of the application.
The source code of the test is presented in Listing 1.
Listing 1. Source code of a UI test
[Test]
public void AppScreensPortraitAndLandscape ()
{
app.SetOrientationPortrait ();
System.Threading.Thread.Sleep (3000);
app.Screenshot ("Start page in portrait mode");
app.WaitForElement (c => c.Button("Start app"));
app.SetOrientationLandscape ();
System.Threading.Thread.Sleep (3000);
app.Screenshot ("Start page in landscape mode");
System.Threading.Thread.Sleep (3000);
app.WaitForElement (c => c.Button("Start app"));
app.Tap (p => p.Button ("Start app"));
app.WaitForElement (c => c.Text("Hello!"));
app.Back ();
System.Threading.Thread.Sleep (3000);
app.Screenshot ("Start page back");
}
As you can see, the above text is composed of very simple commands, which simulate actions performed by the user. The command app.SetOrientationPortrait() sets the mobile device screen in a horizontal position. The next step is the command System.Threading.Thread.Sleep(3000), triggered from an auxiliary function Suspend, suspends conducting the test for 3 seconds. This kind of delay is useful in a situation, where we test the performance of the application to verify whether a specific operation will be performed in a specified time. In this case, within 3 seconds. Next, use the method app.SetOrientationLandscape() we change the device orientation and after that the function app.Screenshot(“Start page in portrait mode”) takes a screenshot and saves it to the hard drive. With this function, we can save a visual status of the application to check what a given view looks like on a particular device from the Xamarin cloud. The command app.WaitForElement (c => c.Button(“Start app”)) checks whether the button labeled Start app is visible on the device screen. Next, we simulate clicking (touching) this button using the command app. Tap (p => p.Button (“Start app”)). After that the application navigates to the welcome screen (app.Back method) and again, after waiting 3 seconds, we save the screenshot to the hard drive.
In the main section of the figure, we can see the source code of the created test. On the left side, we can see the structure of the project, and on the right, we can see the window which allows us to run a test. After clicking the button Run All the test will be executed. Example results received from the emulator Galaxy Nexus 4.3 (window Unit Tests in Figure 7) show that the test was performed without any problems. However, a question remains: what about other devices available on the market?
To answer the question asked in the previous subchapter we need to run a test on a larger number of devices. The test from Listing 1 is perfect for this purpose.
Xamarin Test Cloud is a cloud of over two thousand mobile devices (with Android and iOS systems), on which the developer can run individual UI tests (Figure 8). Directly before every test run, we can select target devices. Next, a compiled application as well as a test are sent to the Xamarin cloud, and after the test is performed its results are presented in a form of a very attractive report. Furthermore, we can visually inspect particular application views saved using the method app. Screenshot
The reason for that is that on devices of a smaller size/screen density, in the horizontal view, the button labeled Start app did not fit on the screen. Thus the command: app.WaitForElement(c => c.Button(“Start app”)), was not executed correctly (Figure 12 and Figure 13). It is an evident developer’s mistake – the application should be designed so that the button labeled Start app is always visible. Although this error was generated for this article, in our everyday work we have encountered applications many times, which contained errors of this type. Usually, the scenario was that the developer created an application that was tested manually on devices/emulators, which the developer had access to – usually 2-3 devices. Next, the application was published and it turned out that on a large number of devices, it did not work correctly. The developer did not have an idea why it is the case, as he could not test it on a larger number of devices.
There are a lot of testing strategies and selecting an appropriate one depends on numerous factors. The process of testing mobile applications includes the following stages: first, the scenario of the test is drawn up. During this stage, the specification of the test is prepared, which includes a list of example uses of individual elements of a given view and the expected outcomes of these actions.
When the scenario is ready, we can go on to programming the test. The level of test complexity corresponds to the scenario. For this reason, the final source code can consist of a few commands or even many classes.
Next, the prepared test is run on a physical device to test its correct operation. After that, possible bugs in the source code of the test are corrected, which is followed by a publication and execution in the XTC cloud on many devices. Because access to the XTC cloud is paid, an adaptation of the test on a physical device is essential for optimizing the use of the XTC cloud and for minimizing costs.
After conducting the test, its results are presented in the form of reports from XTC (see Figure 10 and Figure 11).
Running tests on local devices can be automated using a server of continuous integration (CI). Tests are published within the repository of the application source code and are run on local devices available in the local network. The CI server, in specified time intervals, checks if programming works were conducted. If yes, a test compilation is created, which is then automatically tested based on the prepared test scripts. If any bugs are found, appropriate information is sent to the programmers using specified information channels: communicators, electronic mail, and notifications. As a result, a very quick reaction is possible and the source code can be fixed. Additionally, each programmer can run all or selected tests (ad-hoc) and see the results locally.
The testing process for mobile applications presented above illustrates a general way of conduct. Specific automatic tests can be directed at the functional analysis of the application and finding regression bugs, testing application views (visual tests), differential (analyzing differences between devices), and performance.
Functional tests mean testing the functionality of the created solution. They concentrate mainly on the part related to executing programmed functionalities of applications that react to actions performed by the user. Such tests simulate various actions of the application user and its purpose is to verify the correctness of application operation described in functional requirements.
Functional tests cover the most important scenarios of the application use (so-called “user story”). Furthermore, functional tests are also used to analyze regression bugs. They verify whether implemented corrections and changes in the source code led to bringing back earlier bugs.
The purpose of these tests is to check whether screens of the created application are displayed correctly – whether there aren’t any serious problems with the view on various devices. They are specifically prepared tests, which focus on going through all the application screens.
These tests are run in the XTC cloud before each sprint is finished. We select a few devices, which cover the most popular resolutions and densities of devices. Next, test results are verified by the Quality Assurance department (QA).
The scenario in which we run selected tests on a large number of devices in XTC. The purpose is to run tests on e.g. the fastest and the slowest devices, on devices with high and low resolutions, and with large and small amounts of operating memory. As a result, we can detect potential problems related to hardware.
Performance tests are similar to differential tests in the context of hardware. Their purpose is to check the performance of mobile applications and network services connected to them. For this purpose, the source code of the test is directed at verification of the performance of the application, and then it is run on a large number of devices. Thanks to this we can check, how the application is behaving, if it runs on 200 devices at the same time, and whether it leads to incorrect operation of the service, delays, etc.
The fundamental limitation of Xamarin.UITest is the incapability to perform certain operations automatically. For example, we cannot reply to certain system events (e.g. acceptance of application permissions for executing PUSH notifications) or simulate a 3G network. There are several ways to bypass those limitations. We can for example simulate taking a picture by providing the application with a previously taken picture from the device memory (Listing 2). In the application testing process, interactive tests are also very useful, where during the test a tester’s action is needed. Although we cannot achieve full automation in such a case, its unquestioned advantage is the fact that the test, even with the tester’s interference, is performed in the same way every time – it is repeatable, and a large part of the test is performed automatically.
[Test]
public void FoldersInOfflineMode()
{
string randomfolderName = "aat_"+DateTime.Now.ToString("yyyyMMddHHmmssfff");
app.Screenshot("Open Application");
app.WaitForElement(p => p.Id("Content"), "Login Page doesn't load", new TimeSpan(0,0,15));
app.Screenshot("Login Page");
app.Tap(e => e.Class("EntryEditText").Index(1));
app.EnterText(userLogin);
app.Tap(e => e.Class("EntryEditText").Index(2));
app.EnterText(userPassword);
app.Screenshot("Enter Email and Password");
app.Tap(p => p.Text("Login"));
app.Screenshot("After Click On Login Button");
app.WaitForElement(p => p.Id("action_bar_title"), "Home Page doesn't load", new TimeSpan(0,0,35));
app.Screenshot("Home Page");
app.Tap(x => x.Text(mainFolder2));
app.Screenshot("Enter into folder");
app.ScrollDownTo(mainFolder2TestImg1,null,0,0.67,500,true,new TimeSpan(0,0,100));
app.Tap(mainFolder2TestImg1);
app.Repl();
//CHECK: wait as long as the image will load
//CHECK: turn on flight mode
//CHECK: type quit in Repl
app.Back();
app.Back();
app.Tap(x => x.Text(mainFolder2));
app.ScrollDownTo(mainFolder2TestImg1,null,0,0.67,500,true,new TimeSpan(0,0,100));
app.Tap(mainFolder2TestImg1);
app.Repl();
//CHECK: wait as long as the image will load
//CHECK: check if the image was loaded
//CHECK: turn off airplane mode
//CHECK: type quit in Repl
} Listing 2. An example of a semi-automatic test – the test is stopped at the command app.Reply() and awaits interaction from the tester
There are a lot more ways of using the tools described above than those that were presented in this article. If the readers are interested, we will gladly present “examples taken from real-life experience”, where using Xamarin. UI. Test and Xamarin Test Cloud saved more than one project, which was taken over by another foreign software company, and it initially had very serious problems related to the quality of the code and a large number of regression bugs. Furthermore, integration of the Xamarin tools with the process of continuous integration is a very interesting issue. It is not an easy job, but it is very beneficial, saves time, and improves the quality of solutions created in the company
Do you want to know how to
estimate the budget needed
for your project?
