Contents:

1. Introduction
2. Black Box GUI Automation Overview
3. Object Oriented Testing
4. Image Based Testing
5. Implementation Factors
6. Summary

1. Introduction

This document compares object oriented and image based approaches of black box GUI automated software testing. It is intended to respond frequently asked questions about these two technologies and to provide basic information for those who are looking for automated testing solutions.

2. Black Box GUI Automation Overview

Hardware devices operated by humans nowadays have a lot in common, such as:

• Display device, such as computer or TV screen, mobile phone display,
• Keyboard device, such as a computer or mobile phone keyboard,
• Pointer device, such as a computer mouse, touch pad or stylus.

Figure 1-1: GUI enabled hardware devices

Most modern GUI applications for these devices are implemented on top of a programming platform, such as Java or .NET. Platforms allow developers to build complex GUIs from the platform's library of graphical primitives such as buttons, drop downs, panels and windows. When the application is being run, the platform (or code injected to the application by the platform) paints the application GUI image onto the display device, usually through the screen image buffer operated by the OS. HTML documents generated by web applications can be also considered GUIs save that their image gets painted by a browser rather than the underlying platform.

Software applications typically do not approach the keyboard and pointer devices directly. They rather act as consumers of keyboard and pointer events delivered by the underlying OS.

Black box GUI automated testing tools are software applications which behave in a slightly different way. They generate keyboard and pointer events or invoke GUI component actions in order to navigate through GUI of the tested application. They also allow to analyze the GUI to verify whether the application under test (AUT) behaves as expected.

Automated testing tools can be roughly classified by the way they approach the GUI to:

• Object Oriented Tools
  
• Image Based Tools

3. Object Oriented Testing

Object oriented testing benefits from tight integration with the programming platform. Tools based on this principle understand the platform's technology and they are able to identify individual GUI objects (components), read their properties and interact with them. They can handle test steps such as "verify that there are two buttons labeled 'OK' and 'Cancel'" or "find out whether the 'OK' button is enabled or disabled".

Object oriented testing tools typically invoke actions on the GUI components through the platform APIs. For example, to simulate a mouse click on a Java component JButton one may call the button's doClick() method. Some tools in this category are also able to generate the keyboard and pointer events on the OS level through posting to the system event queues. Verification of test results is usually done through checking of existence or state of a particular GUI component, such as "Check that there's a text field labeled 'Result' and contains the text of '1'".

Object oriented testing approach prevails among automated testing tools today thanks to:

• Vertical capability focus allowing to exploit specific features of the programming platform,
• Robustness of test scripts thanks to ability to identify properties and invoke actions of individual GUI components,
• Good test result verification means,
  
• Easy to understand for engineers familiar with the particular programming platform.

There are however also many disadvantages, such as

• Ability to test just applications of the particular programming platform. A typical test scenario often contains steps which can not be covered by such a tool, for example starting of the application from CLI or verification of outputs in an editor or a third party application. To achieve full test automation additional technologies and/or tools have to be used which increases complexity of the test suite and automation costs.
• Proprietary platform test tool lock-in. When your application gets enhanced with a component implemented using another technology, eventual automation may require to bring in another tool. This inevitably increases automation costs as well as test suite complexity.
  
• Frequent inability to automate all features supported by the underlying platform. For example, many web application test tools can't handle pages with embedded Flash content or a Java applet.
• Inability to reveal GUI layout errors such as component overlaps. The tool can usually confirm that the component is there and has certain properties but it has no means to verify whether it is correctly displayed.
  
• Upgrade risks. Platform upgrades usually require to upgrade the testing tool as well. This runs compatibility risks and increases costs. Some producers even introduce support of a higher platform version with a significant delay from the platform release which poses a schedule risk as well.
  
• Higher qualification requirements on test engineers. Both testing and programming experience is usually necessary. This may or may not increase automation costs.

4. Image Based Testing

Image based testing tools typically automate on the level of operating system, eventually on top of another layer allowing to access all the necessary devices (such as virtualization or remote desktop software). Automation is driven through keyboard and pointer events injected into the system event queues. GUI of the tested application is accessed through the system display buffer on the pixel level. Verification is usually peformed through image comparison methods, such as image search or object recognition.

Advantages of image based testing are quite obvious:

• True "end user approach". The tool navigates through the GUI the same way a user does and verifies the test results visually seeing the same image output as the user.
  
• Complete application technology independence. As automation inputs and outputs are produced and consumed on the OS level, the application technology (programming platform) becomes irrelevant and a single testing tool can automate any GUI application or a set of application based on various technologies. The tool can be often used to automate testing of GUI of the OS itself.
  
• Non-invasive testing. As image based testing tools usually do not need any other other software besides the OS, they can be applied in cases where the test specifications require to keep the system under test free of any third party software.
• Easy to learn; short learning curve even for engineers with no programming experience and/or with no knowledge of the underlying application platform.
  

This testing concept also suffers from numerous disadvantages:

• Lack of access to the application GUI components. An often used workaround is to save images of the GUI components during the test design phase and then use them together with image search to navigate and verify the result during test suite execution.
• Limited verification means. Image comparison doesn't always suffice to verify whether your application works well or not.
• Vulnerability to test environment changes. As just a mere change of desktop background color may cause image search failure, image testing requires a very stable test environment.
  
• Migration risks resulting from vulnerability to test environment changes. Migration of a test suite to another test environment usually requires to recreate images of GUI components used for verification.

5. Implementation Factors

Choosing the right automated testing tool for your project is crucial. Before you make a decision, figure out what your test environment is like and what your expectations, resources and future development plans are. When deciding between the object oriented and image based technologies, consider the following factors:

• Required test depth. Do you need access to the GUI objects or are does a simpler image approach work for you?
• Technology scope. Does the application you are going to automate consist of one technology or rather a number of different technologies? Is it possible that the project will expand with new technologies in the future?
• Migration and upgrade plans. Is your test environment going to be stable, for example on a dedicated test server? Will there be a need to migrate the test suites to another environment and/or platform version?
  
• QA resources at hand. Do your test engineers have a programming experience with the underlying platform? Do you have enough time to get your resources trained to master the platform specific features?

6. Summary

From the capability point of view, object oriented testing focuses verticaly on a particular programming technology and it performs well in single technology environments. Some test steps out of the particular technology however can't be usually automated with the same tool. Automation development costs are usually higher while maintenance costs are lower thanks to good robustness.

Image based testing is more generic and universal. It performs well in environments with mixed technologies and in certain cases where object oriented tools fail or lack support of a certain feature or technology. Automation development costs are typically lower because of simplicity but long term maintenance costs may be higher due to environment vulnerabilities.

- Robert Pes, T-Plan Ltd, August 2009