The year 2021 marks a 100 years since the Czech playwrite Karel Čapek introduced the world to the word “Robot”, when his play ‘Rossum’s Universal Robots (RUR)‘, premiered in Prague in January 1921.
Initially the writers brother gave him the idea to call these creatures in his play ‘roboti’ after the Czech word for serf or forced labourer, but Karel settled on ‘robot’.
At the time the play was a great success and fathered debates of whether technology might free and liberate workers from their manual labour. The play ‘RUR’ even played to audiences abroad and in 1927 was the first full-length play to be aired by the BBC in full length. Additionally in 1938 Rossum’s Universal Robots (RUR) became possibly the worlds first televised science fiction.
We knew nothing of the naming significance at the time, but it is certainly strange to think that in 2005 our very own Robert Pes, T-Plan CTO, created ‘VNCRobot’ in the Czech Republic, and when T-Plan purchased the tool in 2009 we kept the Czech Republic history, naming our award winning Test Automation product ‘T-Plan Robot’.
Our new office is situated in the Lady Bee Enterprise Centre, which is a located just outside Brighton in Southwick, East Sussex.
With direct train line access to London and the West we have great lines of transportation to our customers.
Directions to T-Plan Unit 8
Unit 8, Lady Bee Enterprise Centre,
Tired of not having drag and drop capabilities when performing your Test Management activities on a Mac? Frustrated at working in a browser using a web app built for a windows environment with half of the normal user functionality that you would expect?
If so… why not try our Mac supported Test Management Tool with all of the rich functionality that you would expect of a desktop application.
Don’t believe me… ask us for a demonstration. FREE to TRY… before you buy!
Did you know that T-Pan Robot can be used to automate the testing of CAD software applications? Well you do now!
Test automation of this type of CAD application showcases our products uniqe ability to automate an application or any environment. In the example video Robot is automating on a Linux Ubuntu environment, but the same script would also work on a MacOSx or Windows platform.
Please see the video below of Robot automating Paraview. ‘Paraview is an open-source, multi-platform data analysis and visualisation application.’
This document compares object oriented and image based approaches of black box GUI automated software testing. It is intended to respond frequently asked questions about these two technologies and to provide necessary information for those who are looking for automated testing solutions.
Hardware devices operated by humans nowadays have a lot in common, such as:
- Display device, such as computer or TV screen, mobile phone display,
- Keyboard device, such as a computer or mobile phone keyboard,
- Pointer device, such as a computer mouse, touch pad or stylus.
Figure 1-1: GUI enabled hardware devices
Most modern GUI applications for these devices are implemented on top of a programming platform, such as Java or .NET. Platforms allow developers to build complex GUIs from the platform’s library of graphical primitives such as buttons, dropdowns, panels and windows. When the application is run, the platform (or code injected to the application by the platform) paints the application GUI image onto the display device, usually through the screen image buffer operated by the OS. HTML documents generated by web applications can also be considered GUIs save that their image gets painted by a browser rather than the underlying platform.
Software applications typically do not approach the keyboard and pointer devices directly. They rather act as consumers of keyboard and pointer events delivered by the underlying OS.
Figure 1-2: Application architecture
Black box GUI automated testing tools are software applications which behave in a slightly different way. They generate keyboard and pointer events or invoke GUI component actions to navigate through GUI of the tested application. They also allow to analyze the GUI to verify whether the application under test (AUT) behaves as expected.
Automated testing tools can be classified by the way they approach the GUI automation:
- Object Oriented Tools
- Image Based Tools
Object oriented testing benefits from tight integration with the programming platform. Tools based on this principle understand the platform’s technology, and are able to identify individual GUI objects (components), read their properties and interact with them. They can handle test steps such as “verify that there are two buttons labelled ‘OK’ and ‘Cancel'” or “find out whether the ‘OK’ button is enabled or disabled”.
Object oriented testing tools typically invoke actions on the GUI components through the platform APIs. For example, to simulate a mouse click on a Java component JButton, one may call the button’s doClick() method. Some tools in this category are also able to generate the keyboard and pointer events on the OS level through posting to the system event queues. Verification of test results is commonly done through checking of existence or state of a particular GUI component, such as “Check that there’s a text field labelled ‘Result’ and contains the text of ‘1’”.
Figure 2-1: Object oriented testing approach
Object oriented testing approach prevails among automated testing tools today thanks to:
- Vertical capability focus allowing to exploit specific features of the programming platform,
- Robustness of test scripts thanks to ability to identify properties and invoke actions of individual GUI components,
- Good test result verification means,
- Easy to understand for engineers familiar with the particular programming platform.
There are however also many disadvantages, such as
- Ability to test just applications of the particular programming platform. A typical test scenario often contains steps which can not be covered by such a tool, for example, starting of the application from CLI or verification of outputs in an editor or a third party application. To achieve full test automation, additional technologies and/or tools have to be used, which increases the complexity of the test suite and automation costs.
- Proprietary platform test tool lock-in. When your application gets enhanced with a component implemented using another technology, eventual automation may require to bring in another tool. This inevitably increases automation costs as well as test suite complexity.
- Frequent inability to automate all features supported by the underlying platform. For example, many web application test tools can’t handle pages with embedded Flash content or a Java applet.
- Inability to reveal GUI layout errors such as component overlaps. The tool can usually confirm that the component is there and has certain properties, but it has no means to verify whether it is correctly displayed.
- Upgrade risks. Platform upgrades usually require an upgrade of the testing tool as well, which in turn causes compatibility risks, and increases costs. Some producers even introduce support of a higher platform version with a significant delay from the platform release, which poses a schedule risk as well.
- Higher qualification requirements on test engineers. Both testing and programming experience is usually necessary, and this commonly increases automation costs.
Image based testing tools typically automate at the level of an operating system, eventually on top of another layer allowing to access all the necessary devices (such as virtualization or remote desktop software). Automation is driven through keyboard and pointer events injected into the system event queues. GUI of the tested application is accessed through the system display buffer on the pixel level.
Verification is usually performed through image comparison methods, such as image search or object recognition.
Figure 3-1: Image based testing approach
Advantages of image based testing:
- True “end-user approach”. The tool navigates through the GUI the same way a user does and verifies the test results visually seeing the same image output as the user.
- Complete application technology independence. As automation inputs and outputs are produced and consumed on the OS level, the application technology (programming platform) becomes irrelevant, and a single testing tool can automate any GUI application or a set of application based on various technologies. The tool can be often used to automate testing of GUI of the OS itself.
- Non-invasive testing. As image based testing tools usually do not need any other other software besides the OS, they can be applied in cases where the test specifications require to keep the system under test free of any third party software.
- Easy to learn. A Short learning curve, even for engineers with no programming experience and/or with no knowledge of the underlying application platform.
This testing concept also suffers from some minor disadvantages:
- Lack of access to the application GUI components. An often used workaround is to save images of the GUI components during the test design phase and then use them together with image search to navigate and verify the result during test suite execution.
- Vulnerability to test environment changes. Image testing often requires a stable test environment, however this is often mitigated with clever image doctor techniques etc.
- Migration risks resulting from vulnerability to test environment changes. Migration of a test suite to another test environment usually requires to recreate images of GUI components used for verification. A solution to this vulnerability is to make sure you build up throughout the SDLC, the image collections of your system under test, under the different environments to which it is to be deployed. In this way you will be able to truly test cross-platform and cross-device.
Choosing the right automated testing tool for your project is crucial. Before you make a decision, figure out what your test environment is like and what your expectations, resources and future development plans are. When deciding between the object oriented and image based technologies, consider the following factors:
- Required test depth. Do you need access to the GUI objects or are does a more straightforward image approach work for you?
- Technology scope. Does the application you are going to automate consist of one technology or rather a number of different technologies? Is it possible that the project will expand with new technologies in the future?
- Migration and upgrade plans. Is your test environment going to be stable, for example on a dedicated test server? Will there be a need to migrate the test suites to another environment and/or platform version?
- QA resources at hand. Do your test engineers have programming experience with the underlying platform? Do you have enough time to get your resources trained to master the platform-specific features?
From the capability point of view, object oriented testing focuses vertically on a particular programming technology and it performs well in single technology environments. Some test steps out of the particular technology however can’t be usually automated with the same tool. Automation development costs are usually higher while maintenance costs are lower thanks to good robustness.
Image based testing is more generic and universal. It performs well in environments with mixed technologies and in certain cases where object oriented tools fail or lack support of a certain feature or technology. Automation development costs are typically lower because of simplicity, but long term maintenance costs may be higher due to environment vulnerabilities.
– Robert Pes, T-Plan Ltd
In checking back through some presentations we have given over the years… I found a slide pack given by our CEO at the Sydney Australia Testing Tools Fair many moons ago.
I have copied the headline items below as I still believe they stand the test of time, even to this day!
1) Not agreeing the Testing Stages
Need a “Strategy”
2) Not identifying resource requirements
Need a “Plan”
3) Not clearly defining the scope
Need “Exit Criteria”
4) Not establishing the purpose of each script
Need “Minimum Effort”
5) Not establishing suitable automation
Need “Maximum Efficiency”
6) Not knowing where you are
7) Not identifying problem areas
Need “Incident Management”
Implementation of successful testing processes must include:
- A strategy and detailed Plan
- Establishing ‘What’ and ‘How’ to achieve the testing
- Optimisation using appropriate testing tools
- Management and Control of the Testing Process
- Management and control of the Incidents
The latest version of T-Plan Robot Enterprise is the easiest to use UI test automation tool on the market… well that is our view anyhow!
Why you ask? Well… we have developed a new screen recorder smart technology, which interacts with your use of ANY system to correctly and accurately perform the image and code creation, as you create the automated test.
This saves you time and money! However the other main benefit is that any user of any capability can use our test automation tool. Simply click, add and move to the next, to create you test automation scripts. Being UI driven with images, you can also quickly understand where you are in your test creation, with no code knowledge required.
Don’t believe me… well take it for a spin, or ask us for a demonstration. FREE to TRY… before you buy!
Test Automation on iOS 8 is now possible with T-Plan Robot Enterprise. Driving a NON-Jailbroken (not rooted) REAL device, or simulator, has never been easier!
Utilising T-Plan Robot’s amazingly new simple image capture & playback technology, we can now shout that we have focused mobile test automation, back into the testers domain.
With alternative solutions being very technical and prone to many weeks of development by developers, T-Plan’s solution is not only easy to use, but more importantly easy to learn and setup.
This means that return on investment can be realised almost immediately, and managers returning upstairs with a smile on their face!
Open Source tools have been around for years, and it would seem that the downturn in the economy over the past few years, has pushed the use of these tools into the forefront of management consideration, like we have never seen before.
There is often something quite intangible, and some would say magical about using a free solution, and one that has particular steer in the psyche when looking at software tools. However… as you will see from the previous sentence… it is all too easy to mix the words “free” and open source” together as meaning the same thing.
What people forget when caught up in the religious zeal of open source is that there is a real tangible cost of implementing this type of solution, which often far outstrips the cost of a paid commercial product. Indeed… if people were just to slow down and pick the correct tool for the job, many would see that the commercial world of software has also changed exponentially over the same short period, and I certainly would argue now offers a more attractive overall packaged solution than any you would expect to find in the “free” world!
For example, in situations where a small test team has to test multiple software applications, with limited overall technical skill, and a wide range of supported environments (pc, mobile etc.); that a commercial solution should be your one and only focus…
Many teams these days have limited technical experience with automation. Yes you may have automation experts in the team etc. but scratch the surface and you will find limited experience past HP QTP, or anything outside a total all consuming indoctrination of HP UFT.
Even worse is the team with an automation guru, where his or her technical skill set is so one shop and biased towards a particular form of vegetable (cucumber, gerkin). In this instance the team will struggle to any other type of testing, and be left high and dry and open to ransom demands if he or she chooses to leave etc.
1. Free / open source tools are very technical to setup and run.
You will have to hire in, or technically train resources to implement and maintain your testing solution going forwards.
This is very costly and restricts you to a particular technology and process moving forwards.
You will find that your resources become developers, and it breeds a mentality of tester “fencing”, as I call it, where employees erect “Chinese walls” around what they develop, so that they become indispensable and therefore impossible to replace.
2. Free / open source tools require technical development.
Rarely, rarely, rarely are these tools “out of the box solutions”. All extra work, extensions, addins, bug fixes, issues etc. is work that you will have to pay for and develop yourselves. Can you afford this cost?
3. Free / open source tools require internal support.
Like the previous points, you will have to hire in, or technically train a leader to implement and maintain your testing solution going forwards. This presents a problem when sickness or holidays or etc. come into play. Resource unavailability could seriously mean your inability to test for the amount of time your resource is unavailable. Can you really afford this situation?
4. Free / open source tools can disappear at any moment.
There are numerous examples on the web, where companies have implemented a testing tool and embedded it into their testing framework, only to find that the project dies, or is taken over by a commercial vendor, which then charges for you to continue using new releases moving forwards.
Free to start, could mean an escalation of unknown huge costs in the future.
5. Free / open source tools are NOT supported! [without a fee!]
Many companies charge a support fee. In most cases it would have been cheaper to buy a commercial solution!
6. Free / open source tools are bound by different licensing.
You need to be very careful what you do with open source and free tools. Free does NOT mean that you are free to do ANYHING you want with the software in many cases.
For example… Most open source or free tools are covered under GPL. GPL licensing requires you to distribute any derivative work as open source under GPL. This is known as “copyleft”. The practical impact is that if you write a any new feature, plugin or enhancement, it is in terms of GPL considered to be a derivative work of the tool, and therefore covered by GPL.
I.e. If you decide to distribute your work to a 3rd party, you have to do so under GPL, and with the full source code. If you do not… you can be prosecuted.
7. Free / open source tools rarely integrate in with other tools out of the box, without development of integration features and plugins etc.
Again… work that you will have to do yourselves.
Can you afford all the above 7 deadly sins… I wonder… is the open source world looking less free, and commercial solutions more attractive… I hope so… but then I would, wouldn’t I!