TESTING AND QUALITY MANAGEMENT. Automatic testing of mobile applications using Xamarin tools

Authors: Tomasz Soroka

02.05.2017

INTRODUCTION

It is best to start the description of the genesis of this article with asking the fundamental question. Namely, why is testing mobile applications important? So, according to research conducted in the USA, it was estimated that in order for an application to cover 80% of the mobile market, it should be tested on the minimum of 134 different devices. This research includes two most popular platforms – iOS and Android (Figure 1).


Figure 1. The chart of application share on the market depending on the number of tested devices. As you can see, the function shows exponential growth. This means that percentage coverage of the market increases quickly with the number of devices used in the tests.

According to another research, if an application does not load in three seconds from the moment of running it, 60% of users will close it, and what is even worse, 43% of users declares that they will never go back to such an application. On the other hand, a correctly operating application will be recommended to others by more than 45% users.

Mobile applications are easily available, a few clicks is enough to download and run them from a proper shop. The selection of applications is huge and quality is the key word in the process of creating them, as it is.

The quality that is decisive in the success of a given application. One of the elements which guarantees high quality is proper testing. In the case of mobile applications however, it is not an easy job. One of the fundamental problems is the diversity of screens of mobile devices (Figure 2) and a very large fragmentation of the market, which includes:

  • different versions of operating systems (iOS, Android, Windows Phone, WinRT, Windows 10),
  • different versions of devices (smartphones, tablets), 
  • different types of smartphones (different resolutions, screen sizes, types of processors, amount of memory),
  • different types of tablets (different resolutions, screen sizes, types of processors, amount of memory),
  • different versions of operating systems of the same producer, e.g. Android from version 2.3 to version 6.X, iOS from version 7 to version 9.X, Windows Phone 8.0 – 10.


Figure 2. The diversity of screen sizes of mobile devices


Biorąc pod uwagę ilość możliwych kombinacji problemów, które mogą się pojawić w trakcie tworzenia aplikacji, okazuje się iż wykonanie prawidłowo działającej aplikacji jest sporym wyzwaniem. Z tego powodu przetestowanie aplikacji staje się bardzo dużym wyzwaniem dla developera.


METHODS FOR TESTING MOBILE APPLICATIONS


There are a lot of ways to test desktop and web applications, starting with manual tests – popular clicking, through crowd testing, and finishing with automatic tests (unit testing). The scope of areas for testing in mobile applications is similar, starting with individual tests, through integrating tests, system tests, and finishing with automatic UI testing, in which software simulates typical actions performed by users using the user interface, e.g. clicking (touching) buttons, selecting elements of drop down lists, scrolling views etc.

Figure 3 illustrates a comparison of various testing scenarios. Manual tests are the most realistic, however, they require a lot of time for realisation of individual iterations. Traditional individual tests make it possible to reduce the time of a single cycle at the expense of reduced realism of the test. Automatic UI tests guarantee high realism, and at the same time they reduce the time needed for carrying out a given test. On the basis of this comparison we can see that automatic UI tests seem to be a good solution during the process of application development. For this reason we are going to focus in this article on automatic UI tests. In particular, using Xamarin tools will be discussed. These tools are not limited exclusively to mobile applications created on the basis of Xamarin technologies.

Figure 3. Characteristics of application testing


TOOLS PROVIDED BY THE XAMARIN COMPANY

Xamarin provides two basic tools for conducting automatic tests. The first of them is the framework for writing tests (Xamarin.UITest). It allows to write code which emulates the user’s behaviour and it allows to conduct a test on an emulator or a device which is connected to the computer. The second tool is Xamarin Test Cloud (XTC). This tool allows to conduct a previously written test in a cloud on over 2000 real devices. In other words, XTC provides an interface which automatically installs and then runs a given application on all of the selected devices. As the final result, XTC presents a report from conducted tests.

 

TYPES OF TESTS WHICH CAN BE CONDUCTED USING XAMARIN TOOLS

The main objective of the Xamarin tools is automation of the testing process and performing tests on a large number of different devices. In a basic approach tests are designed, run and adjusted locally, in particular on a physical device connected to a computer or an emulator, after that the same test can be published and executed in the XTC cloud on hundreds selected devices.

 

HOW DO YOU DESIGN TESTS USING XAMARIN TOOLS?

Writing a test starts with creating scripts in C# or Ruby. In this article we are going to present an example test designed in C#. So, we create a test, whose purpose will be to check if a given button fits (is visible) on the screen of a mobile device. For the purpose of a specific example we created a relatively simple application designed in the Xamarin Forms technology. The application consists of two screens. On the first screen of the application there is a button labelled Start app, whose only purpose is to switch the user to the second screen (navigation to the other view), which displays the text reading Hello!. Both views of the application are illustrated in screenshots presented in Figures 4-6.

Figure 4. Welcome screen of an application run in the GenyMotion emulator


Figure 5. Second view of the application


Figure 6. Welcome screen in the horizontal view


Let’s create a test whose purpose will be running an application, turning the screen to the horizontal position and pressing the button labelled Start app. This will result in displaying the second screen of the application reading Hello!. The last element of the test will be returning to the welcome screen of the application.

The source code of the test is presented in Listing 1.


Listing 1. Source code of a UI test

[Test]
        public void AppScreensPortraitAndLandscape ()
        {
            app.SetOrientationPortrait ();
            System.Threading.Thread.Sleep (3000);
            app.Screenshot ("Start page in portrait modee");
            app.WaitForElement (c => c.Button("Start app"));
            app.SetOrientationLandscape ();
            System.Threading.Thread.Sleep (3000);
            app.Screenshot ("Start page in landscape mode");
            System.Threading.Thread.Sleep (3000);
            app.WaitForElement (c => c.Button("Start app"));
            app.Tap (p => p.Button ("Start app"));
            app.WaitForElement (c => c.Text("Hello!"));
            app.Back ();
            System.Threading.Thread.Sleep (3000);
            app.Screenshot ("Start page back");
        } 


As you can see, the above text is composed of very simple commands, which simulate actions performed by the user. The command app.SetOrientationPortrait() sets the mobile device screen in a horizontal position. In the next step, the command System.Threading.Thread.Sleep(3000), triggered from an auxiliary function Suspend, suspends conducting the test for 3 seconds. This kind of delay is useful in a situation, where we test the performance of the application to verify whether a specific operation will be performed in a specified time. In this case, within 3 seconds. Next, using the method app.SetOrientationLandscape() we change the device orientation, after that the function app.Screenshot("Start page in portrait mode") takes a screenshot and saves it to the hard drive. With this function we can save a visual status of the application in order to check what a given view looks like on a particular device from the Xamarin cloud. The command app.WaitForElement (c => c.Button("Start app")) checks whether the button labelled Start app is visible on the device screen. Next, we simulate clicking (touching) this button using the command app.Tap (p => p.Button ("Start app")). After that the application navigates to the welcome screen (app.Back method) and again, after waiting 3 seconds, we save the screenshot to the hard drive.

 

Figure 7 illustrates the view of the Xamarin Studio environment. In the main section of the figure we can see the source code of the created test. On the left side we can see the structure of the project, and on the right we can see the window which allows to run a test. After clicking the button Run All the test will be executed. Example results received from the emulator Galaxy Nexus 4.3 (window Unit Tests in Figure 7) show that the test was performed without any problems. However, a question remains: what about other devices available on the market?

Figure 7. The view of the Xamarin Studio environment


Running a test in the Xamarin Test Cloud

 In order to answer the question asked in the previous subchapter we need to run a test on a larger number of devices. The test from Listing 1 is perfect for this purpose. 

Xamarin Test Cloud is a cloud of over two thousand mobile devices (with Android and iOS systems), on which the developer can run individual UI tests (Figure 8). Directly before every test run we can select target devices. Next, a compiled application as well as a test are sent to the Xamarin cloud, and after the test is performed its results are presented in a form of a very attractive report (Figure 9). Furthermore, we can visually inspect particular application views saved using the method app.Screenshot (Figure 10 and Figure 11).



Figure 8. Selection of devices in Xamarin Test Cloud, which will be included in the test


For the purpose of the described example, the test from Listing 1 was run on 213 different devices with the operating system Android (full report is available at: https://testcloud.xamarin.com/test/xtcexampleapp_c4b44135-9331-496c-a7db-bd9f8355e783 ). The results are as they are because on a large number of devices the test did not run successfully (Figure 9). 


The reason for that is that on devices of a smaller size/screen density, in the horizontal view, the button labelled Start app did not fit on the screen. Thus the command: app.WaitForElement(c => c.Button("Start app")), was not executed correctly (Figure 12 and Figure 13). Obviously, it is an evident developer’s mistake – the application should be designed so that the button labelled Start app is always visible. Although this error was generated for the purpose of this article, in our everyday work we have encountered applications many times, which contained errors of this type. Usually the scenario was that the developer created an application which was tested manually on devices/emulators, which the developer had access to – usually 2-3 devices. Next, the application was published and it turned out that on a large number of devices it did not work correctly. The developer did not have an idea why it is the case, as he could not test it on a larger number of devices.



Figure 9. Test result. Test was not successful on 20 devices.



Figure 10. The view of the first step of the test – was performed correctly on all 213 devices.


Figure 11. The third step of the test. The test was not successful on 20 devices, as the button labelled Start is not visible on them.


Figure 12. Details of a bug which makes it impossible to finish the test correctly



Figure 13. Screenshot from a real device – we can see a partial button labelled Start app



HOW TO USE THE XAMARIN TOOLS WHICH SUPPORT AUTOMATIC TESTING?

There are a lot of testing strategies and selecting an appropriate one depends on numerous factors. The process of testing mobile applications includes the following stages: first, the scenario of the test is drawn up. During this stage the specification of the test is prepared, which includes a list of example uses of individual elements of a given view and expected outcomes of these actions.

When the scenario is ready, we can go on to programming the test. The level of test complexity corresponds to the scenario. For this reason the final source code can consist of a few commands or even many classes.

Next, the prepared test is run on a physical device in order to test its correct operation. After that, possible bugs in the source code of the test are corrected, which is followed by a publication and execution in the XTC cloud on many devices. Due to the fact that the access to the XTC cloud is paid, adaptation of the test on a physical device is essential for optimising the use of the XTC cloud and for minimising costs.

After conducting the test, its results are presented in the form of reports from XTC (see Figure 10 and Figure 11).

Running tests on local devices can be automated using a server of continuous integration (CI). Tests are published within the repository of the application source code and are run on local devices available in the local network. The CI server, in specified time intervals, checks if programming works were conducted. If yes, a test compilation is created, which is then automatically tested on the basis of the prepared test scripts. If any bugs are found, appropriate information is sent to the programmers using specified information channels: communicators, electronic mail, notifications. As a result, a very quick reaction is possible and the source code can be fixed. Additionally, each programmer can run all or selected tests (ad-hoc) and see the results locally.

The testing process for mobile applications presented above illustrates a general way of conduct. Specific automatic tests can be directed at: functional analysis of the application and finding regression bugs, testing application views (visual tests), differential (analysing differences between devices) and performance.

Functional tests, finding regression bugs

Functional tests mean testing the functionality of the created solution. They concentrate mainly on the part related to executing programmed functionalities of applications which react to actions performed by the user. Such tests simulate various actions of the application user and its purpose is to verify the correctness of application operation described in functional requirements.

Functional tests cover the most important scenarios of the application use (so called “user story”). Furthermore, functional tests are also used to analyse regression bugs. They verify whether implemented corrections and changes in the source code led to bringing back earlier bugs.

Performing visual tests

The purpose of these tests is to check whether screens of the created application are displayed correctly – whether there aren’t any serious problems with the view on various devices. They are specifically prepared tests, which focus on going through all the application screens.

These tests are run in the XTC cloud before each sprint is finished. We select a few devices, which cover the most popular resolutions and densities of devices. Next, test results are verified by the Quality Assurance department (QA).

 

Performing tests which test the differences between devices

The scenario in which we run selected tests on a large amount of devices in XTC. The purpose is to run tests on e.g. the fastest and the slowest devices, on devices with high and low resolutions, with large and small amount of operating memory. As a result we can detect potential problems related to hardware.

 

Testing the performance of a mobile application

Performance tests are similar to differential tests in the context of hardware. Their purpose is to check the performance of a mobile application and network services connected to them. For this purpose the source code of the test is directed at verification of the performance of the application, and then it is run on a large number of devices. Thanks to this we can check, how the application is behaving, if it runs on 200 devices at the same time, and whether it leads to incorrect operation of the service, delays etc.

 

LIMITATIONS OF THE TESTING TOOLS

The fundamental limitation of Xamarin.UITest is the incapability to perform certain operations automatically. For example, we cannot reply to certain system events (e.g. acceptance of application permissions for executing PUSH notifications) or simulate a 3G network. There are several ways to bypass those limitations. We can for example simulate taking a picture by providing the application with a previously taken picture from the device memory (Listing 2). In the application testing process interactive tests are also very useful, where during the test a tester’s action is needed. Although, we cannot achieve full automation in such case, its unquestioned advantage is the fact that the test, even with the tester’s interference, is performed in the same way every time – it is repeatable and a large part of the test is performed automatically.


[Test]
        public void FoldersInOfflineMode()
        {
            string randomfolderName = "aat_"+DateTime.Now.ToString("yyyyMMddHHmmssfff");

            app.Screenshot("Open Application");
            app.WaitForElement(p => p.Id("content"), "Login Page doesn't load", new TimeSpan(0,0,15));
            app.Screenshot("Login Page");
            app.Tap(e => e.Class("EntryEditText").Index(1));
            app.EnterText(userLogin);
            app.Tap(e => e.Class("EntryEditText").Index(2));
            app.EnterText(userPassword);
            
            app.Screenshot("Enter Email and Password");
            app.Tap(p => p.Text("Login"));
            app.Screenshot("After Click On Login Button");
            app.WaitForElement(p => p.Id("action_bar_title"), "Home Page doesn't load", new TimeSpan(0,0,35));
            app.Screenshot("Home Page");
            app.Tap(x => x.Text(mainFolder2));
            app.Screenshot("Enter into folder");
            app.ScrollDownTo(mainFolder2TestImg1,null,0,0.67,500,true,new TimeSpan(0,0,100));   
            app.Tap(mainFolder2TestImg1);


            app.Repl();
            //CHECK: wait as long as image will load
            //CHECK: turn on flight mode
            //CHECK: type quit in Repl

            app.Back();
            app.Back();
            app.Tap(x => x.Text(mainFolder2));
            app.ScrollDownTo(mainFolder2TestImg1,null,0,0.67,500,true,new TimeSpan(0,0,100));   
            app.Tap(mainFolder2TestImg1);
            app.Repl();
            //CHECK: wait as long as image will load
            //CHECK: check if image was loaded
            //CHECK: turn off airplane mode
            //CHECK: type quit in Repl

        } 

Listing 2. An example of a semi-automatic test – the test is stopped at the command app.Repl() and awaits interaction from the tester


SUMMARY

There are a lot more ways of using the tools described above than those that were presented in this article. If the readers are interested, we will gladly present “examples taken from real life experience”, where using Xamarin.UI.Test and Xamarin Test Cloud saved more than one project, which was taken over from another foreign software company, and it initially had very serious problems related to the quality of the code and a large number of regression bugs. Furthermore, integration of the Xamarin tools with the process of continuous integration is a very interesting issue. It is not an easy job, but it is very beneficial, saves time and improves the quality of solutions created in the company