2. Black Box GUI Automation Overview
3. Object Oriented Testing
4. Image Based Testing
5. Implementation Factors
This document compares object oriented and image
based approaches of black box GUI automated software testing. It is
intended to respond frequently asked questions about these two
technologies and to provide basic information for those who are looking
for automated testing solutions.
Black Box GUI Automation Overview
Hardware devices operated by humans nowadays have a lot in common, such as:
- Display device, such as computer or TV screen, mobile phone display,
device, such as a computer or mobile phone keyboard,
device, such as a computer mouse, touch pad or stylus.
|Figure 1-1: GUI enabled hardware
Most modern GUI applications for these devices are implemented on top
of a programming platform,
such as Java
or .NET. Platforms allow developers to build complex GUIs from the
platform's library of
graphical primitives such as buttons, drop downs, panels and windows.
When the application is being run, the platform (or code injected to
the application by the platform) paints the
application GUI image onto the display device, usually through the
screen image buffer operated by the OS. HTML documents generated by web
applications can be also considered GUIs save that
their image gets painted by a browser rather than the
Software applications typically do not approach the keyboard and
pointer devices directly. They rather act as consumers of keyboard and
pointer events delivered by the underlying OS.
1-2: Application architecture
Black box GUI automated testing tools are software applications
behave in a slightly different way. They generate keyboard and
pointer events or invoke GUI component actions in
order to navigate through GUI of the tested application. They also
allow to analyze the GUI to verify whether the application under test
(AUT) behaves as expected.
Automated testing tools can be roughly classified by the way they
approach the GUI to:
- Image Based Tools
3. Object Oriented Testing
Object oriented testing benefits from tight integration with the
programming platform. Tools based on this principle understand the
platform's technology and they are able to identify individual GUI
objects (components), read their properties and interact with them.
They can handle test steps such as "verify
that there are two buttons labeled 'OK' and 'Cancel'" or "find out whether the 'OK' button is
enabled or disabled".
Object oriented testing tools typically invoke actions on the GUI
components through the platform APIs. For example, to simulate a mouse
click on a Java component JButton one may call the button's doClick() method. Some tools in this category are also able to generate the
keyboard and pointer events on the OS level through posting to the
system event queues. Verification of test results is usually done
through checking of existence or state of a particular GUI component,
such as "Check that there's a text
field labeled 'Result' and contains the text of '1'".
2-1: Object oriented testing approach
Object oriented testing approach prevails
among automated testing tools today thanks to:
capability focus allowing to exploit specific features of the
- Robustness of test scripts thanks to ability to identify properties and invoke
actions of individual GUI components,
- Good test
result verification means,
- Easy to
understand for engineers familiar with the particular
There are however also many disadvantages, such as
- Ability to
test just applications of the particular programming platform. A
typical test scenario often contains steps which can not be covered by
such a tool, for example starting of the application from CLI or
verification of outputs in an editor or a third party application. To
achieve full test automation additional technologies
and/or tools have to be used which increases complexity of the test
suite and automation costs.
platform test tool lock-in. When your application gets
enhanced with a component implemented using another technology,
eventual automation may require to bring in another tool. This
inevitably increases automation costs as well as test
- Frequent inability
features supported by the underlying platform.
For example, many web application test tools can't handle
pages with embedded Flash content or a Java applet.
- Inability to
reveal GUI layout errors such as component
overlaps. The tool can usually confirm that the component is there and
has certain properties but it has no means to verify whether it is
- Upgrade risks.
require to upgrade the testing tool as well.
This runs compatibility risks and increases costs. Some producers even
of a higher platform version with a significant delay from the platform
release which poses a schedule risk as well.
qualification requirements on test engineers. Both
testing and programming experience is usually necessary. This may or
may not increase automation costs.
4. Image Based Testing
Image based testing tools typically automate on the level of operating
system, eventually on top of another layer allowing to access
all the necessary devices (such as virtualization or remote desktop
software). Automation is driven through keyboard and pointer events
injected into the system event queues. GUI of the tested application is
accessed through the system display buffer on the pixel level.
Verification is usually peformed through image comparison methods, such
as image search or object recognition.
3-1: Image based testing approach
Advantages of image based testing are quite obvious:
- True "end
user approach". The tool navigates through the GUI the same way
a user does and verifies the test results visually seeing the same
image output as the user.
- Complete application
independence. As automation inputs and outputs are
produced and consumed on the OS level, the application technology
(programming platform) becomes irrelevant and a single testing tool can
automate any GUI application or a set of application based on various
technologies. The tool can be often used to automate testing of GUI
of the OS itself.
testing. As image based testing tools usually do not need any
other other software besides the OS, they can be applied in cases where
the test specifications require to keep the system under test free of
any third party software.
- Easy to learn;
even for engineers with no programming experience
and/or with no knowledge of the underlying application platform.
This testing concept also suffers from numerous disadvantages:
- Lack of access
to the application GUI components. An often used workaround is
to save images of the GUI components during the test design phase and
then use them together with image search to navigate and verify the
result during test suite execution.
verification means. Image comparison doesn't always suffice to
verify whether your application works well or not.
to test environment changes. As just a mere change of desktop
background color may cause image search failure, image testing requires
a very stable test environment.
- Migration risks resulting from vulnerability to test environment changes. Migration of
a test suite to another test environment usually requires to recreate
images of GUI components used for verification.
Choosing the right automated testing tool for your project is crucial.
Before you make a decision, figure out what your test environment is
like and what your expectations, resources and future development plans
are. When deciding between the object oriented and image based
consider the following factors:
- Required test
depth. Do you need access to the GUI objects or are does a
simpler image approach work for you?
scope. Does the application you are going to automate consist of
one technology or rather a number of different technologies? Is it
possible that the project will expand with new technologies in the
- Migration and
upgrade plans. Is your test environment going to be stable, for
example on a dedicated test server? Will there be a need to migrate the
test suites to another environment and/or platform version?
- QA resources
at hand. Do your test engineers have a programming experience
with the underlying platform? Do you have enough time to get your
resources trained to master the platform specific features?
From the capability point of view, object
testing focuses verticaly on a particular programming
technology and it performs well in single technology environments. Some
test steps out of the particular technology however can't be usually
automated with the same tool. Automation development costs are usually
higher while maintenance costs are lower thanks to good robustness.
Image based testing is more generic and universal. It performs well in environments with
mixed technologies and in certain cases where object oriented tools
fail or lack support of a certain feature or technology. Automation
development costs are typically lower because of simplicity but long
term maintenance costs may be higher due to environment vulnerabilities.
- Robert Pes,
T-Plan Ltd, August 2009