Image Comparison With VNCRobot 1.3


Contents
1. Introduction
2. Image Comparison Method 'default'
    2.1 Algorithm Description
    2.2 Usage
3. Image Comparison Method 'search'
    3.1 Algorithm Description
    3.2 Usage
4. VNCRobot Scripting Language and Image Comparison
5. Troubleshooting Image Comparisons
    5.1 Image Comparison Factors
    5.2 Real Life Example
    5.3 Debugging Of Failed Image Searches

1. Introduction

Unlike other automated testing software, tools working over the RFB (VNC) protocol like VNCRobot work just with the image of the remote desktop. That's why image comparison is one of the few means to verify the contents of the server desktop and behavior of the tested object (software, application, OS, ...).

Imager comparison algorithm (method) is a module which accepts two images on input and produces a number (usually percentage) indicating how much the images match. VNCRobot v1.3 provides two basic image comparison methods:
Users wishing to use custom image comparison algorithms may take advantage of the VNCRobot 1.3 Open API to write their own Java code and plug it into VNCRobot.


2. Image Comparison Method 'default'

2.1 Algorithm Description

The default method is based on histogram comparison. An image histogram is in simple words a list of colors located in the image together with number of pixels of each color. The method first calculates histograms both of the remote desktop image and the template image and compares them. Result of the image comparison is then calculated as a number of matching pixels divided by the total number of image pixels.

The following example illustrates how the method works. Let us have two simple images, image 1 and image 2:

Default method example images

Each image has 120x100=12000 pixels. The following table shows the histograms and number of matching pixels:

Color
Image 1
Image 2
Matching pixels
White 9000 pixels
6000 pixels
6000
Red
3000 pixels
6000 pixels
3000
Sum of matching pixels:
9000

The comparison result is in this case 9000px/12000px, which is 0.75, or better 75%.

2.2 Usage

Though this image comparison method is very simple and has many limitations, it is quite reliable and has a few advantages. It is for example robust against changes in location of application windows on the remote desktop. The following example displays three different images which will always produce 100% match if compared:

Examples of matching images

In order to get reliable results, consider the following hints:
Note that the default method is designed to compare two images of the same size. If your template image has different size than the remote desktop, both images are cropped to the size of their intersection befor comparison. The following picture illustrates such a situation:

Cropping of comparison images

When a custom comparison area is specified via the cmparea parameter, the remote desktop image is cropped from the area location, i.e. from its [x,y] coordinates. The values of width and height of the area are ignored in this case a are rather derived from the intersection size.

It is not recommended to use the default method together with a custom comparison area because it is easy to make a mistake and compare rectangles which do not correspond. If you need to check a certain area of your remote desktop image whether it contains a certain image, use the 'search' image comparison method instead.


3. Image Comparison Method 'search'

3.1 Algorithm Description

The search method performs a simple pixel by pixel search of a template image in the specified remote desktop image area. Though the algorithm was slightly modified for better performance, it basically runs through all pixels of the remote desktop image (or its part) and tries to find match(es) of the template image. The template image must be smaller than the remote desktop image.

Version 1.3.10 introduces limited support of images with transparent pixels. When a search image comparison is used together with a transparent template image, all transparent or semi-transparent pixels (i.e. pixels with Alpha < 0xFF) are skipped. It means that if you have a template with 100 pixels where 90 pixels are transparent, only the remaining 10 non-transparent pixels will be compared against the desktop image. Transparent pixels are always counted as passed. If e.g. one pixel out of those 10 non-transparent ones in our example will be different, result of the comparison will be 99% (90 'correct' transparent pixels plus 9 equal ones).

Transparency can be used to ensure that image comparison is robust against background or component color changes. When you e.g. search an icon rendered on your remote desktop, marking the background color pixels transparent will make sure that the image comparison passes regardless of the remote desktop color changes. Note that VNCRobot 1.3.x doesn't have any tools to add transparency to template images and such features are planned for VNCRobot 2.0. Linux/Unix users may take advantage of Gimp graphic editor to set up transparent pixels as follows:

1. Open the template image with Gimp
2. Select Layer->Transparency->Add Alpha Channel
3. Select Select->By Color and click on the color you want to make transparent in the image. Note that functionality of By Color is affected by the threshold parameter which specifies similarity of colors to be selected.
4. Select Edit->Cut to remove the selected pixels (and make them transparent).
5. Save the template image and retest with VNCRobot.

Older implementations of the search algorithm were not able to handle the passrate option. It was always looking for an exact pixel-by-pixel match regardless of the desired pass rate. This behavior was changed in VNCRobot 1.3.6 and the new implementation is able to search even for image occurencies which have a certain amount of pixels different from the template image. The pass rate is newly converted to a number of pixels (template_pixel_count * (1 - passrate)) and the remote desktop image is searched for all instances having a number of different pixels lower than or equal to this number. If your template image size is e.g. 100 pixels and you specify 99% pass rate, VNCRobot will find all areas of remote server desktop having up to 1 different pixel. Note that the lower pass rate you specify, the lower the performance will be and the longer the search will take.

The method returns 0% match if the template is not found or 100% if at least one match is located. Coordinates of the found matches are then provided in form of variables which can be accessed during script execution and further processed. See the Compareto command specification for more info on the variable format. Also note that the number of occurencies is limited by a parameter in the Compareto command preferences which can be customized in the VNCRobot Preferences window.

It is recommended to save the template images in PNG format through the Compareto, Screenshot of Waitfor match windows of VNCRobot. There are tools allowing you to crop the desired area of the remote desktop image and save it as a template. If you use images saved by other applications, it may happen that VNCRobot will not be able to match the image due to an incompatible pixel format.

3.2 Usage

The method can be widely used to verify that the remote desktop image contains a desired component or window. It is also possible to search for a component or an icon and then click onto its coordinates.

It is recommended to use the default method together with a custom comparison area (parameter cmparea of the scripting language commands) to narrow the search area. Though you may search the whole remote desktop, the algorithm is quite inefficient and it may take up to seconds to finish. The performance can be also severely affected if there are too many matches. You can improve the performance by proper selection of the template image. The following example suggests how to create templates for an efficient search.

Example remote desktop

Example of a remote desktop

Template Image
Comment
Template #1
Incorrect. There will be thousands of matches within the remote desktop's red rectangle.
Template #2
Better. Though there will be just one match, the large white stripe in the upper part will cause delays in searching because the desktop background is white and the method processes pixels by rows from the top left corner.
Template #3
Best. The first red pixel is located close to the top left corner which will make the search efficient.


4. VNCRobot Scripting Language and Image Comparison

Image comparison modules are closely integrated with 3 commands of the VNCRobot 1.3 scripting language:
All three commands support almost the same image comparison parameters. There are Compareto, Screenshot of Waitfor match windows allowing to construct and maintain the commands easily. See the following picture showing a fragment of code with these commands together with the corresponding image comparison GUI.

Comparison parameters


It is usually necessary to branch the script behavior based on the image comparison result. There are two ways to ensure it:
The following two examples are equivalent and terminate the script execution with an exit code 1 when the image comparison fails:

# Example #1 - 'onfail' parameter
Compareto test.png method=default onfail="exit 1"

# Example #2 - 'if' statement
if ({_EXIT_CODE} > 0) {
  exit 1
}

Another example demonstrates how to search for a button template image and then click it:

Compareto button.png method="search"

# If the button is found, the exit code will be 0
if ({_EXIT_CODE} == 0) {
  Mouse click to=x:{_SEARCH_X}+5,y:{_SEARCH_Y}+5
}


See the VNCRobot 1.3 Language Reference document for additional examples and more information on the scripting language.

5. Troubleshooting Image Comparisons

5.1 Image Comparison Factors

A number of users reported failing image comparisons with VNCRobot, especially searches. Vast majority of these issues are not fault of VNCRobot and they happen due to a number of factors. Consider the these tips first:
The following two chapters are intended to give you an overview of how to reveal causes of failing image comparisons. Be sure to give it a try before you proceed to contacting us.

5.2 Real Life Example

A real life example of a failing image comparison follows. Let's suppose you are testing a web page displayed in Firefox 2.0.0.7 on Ubuntu Linux 6.0.6. Your remote desktop looks like the following one:

Ubuntu 6.0.4 desktop with Firefox 2.0

To verify whether Firefox is correctly displayed you decide to cut a piece of the image with one of the browser buttons and use it as a template. This can be easily done through the Compareto window. Let's suppose that the template will contain the Refresh button from the following area:

Cutting out the Refresh Firefox 2.0 button

If you set 100% pass rate together with the 'search' image comparison method and hit the 'Compare...' button, you get a nice 100% match:

First match

So far so good. Leave the Compareto Command window with OK. It should create a Compareto command in your active script editor:

Compareto command

Then go to the VNC viewer and move your mouse over the Refresh button. Note that Firefox will highlight it:

Highlighted Refresh button

Now move your mouse pointer away from the button so that its original state is restored. Right click the Compareto command in the editor and select 'Properties' in the pop up menu to reopen the 'Compareto Command' window. If you retry the image comparison, VNCRobot fails to find the button for no obvious reason. When you cut out the same image again and compare it with the old one, they visually appear to be the same:

Old Image
Old template
New Image
New Image

Do you think that there's no difference? Yes, there unfortunately is, and VNCRobot provides powerful tools to debug it.

5.2 Debugging Of Failed Image Searches

VNCRobot v1.3.8 introduced two important debugging features. The 'search' image comparison module was enhanced to handle non-exact searches (i.e. searches with pass rate lower than 100%) and tracking of differencies between two images was implemented. This new functionality allowed the 'Compareto Command', 'Screenshot Command' and 'Waitfor Command' windows to debug different pixels of a template image and a certain match area of the remote desktop image (or any other image loaded through the 'Load RD Image' button).

To access this functionality you have to achieve a search match first. If you are doing image comparison through the 'default' method, switch to the 'search' one temporarily. There are two techniques to achieve a match in failing cases:
  1. Decrease the pass rate gradually and verify existence of a match through the 'Compare...' button. This approach helps in most of the cases.
  2. Enter the comparison area equal to (in case you know it exactly) or a bit bigger than the original cut rectangle. This will allow you to decrease the pass rate to values close to 0% without having thousands of matches in the whole picture. It also allows the image comparison to run much faster. In our example the area equals to x:94,y:81,w:30,h:30.
To achieve a match in our Firefox example above one has to decrease the pass rate to approx. 68%. Review the modified parameters of pass rate and comparison area highlighted in the following picture:

Getting a search match

Once you have a match in the expected area, select the 'Show Matches' button to display more details. Part of the window is shown on the following picture:

Debug details

Our template image has 900 pixels (30x30). The window reveals that there are 281 different pixels which means that more than 31% of the template image pixels don't match with the specified area of the remote desktop image. These pixels are displayed in green. When you place your mouse pointer above one of them, the label below the drop down in the top left corner displays the pixel coordinates both in the remote dektop and template image coordinate systems together with the corresponding pixel colors. Though the mouse pointer is not captured in the picture above, you can clearly see that pixel at [100,93] of the remote desktop corresponds to template pixel [6,12] but they have different colors.

Note that this is an example of non-standard behavior of an application (Firefox) rather than VNCRobot failure. Firefox applies an image filter to the button icon in different modes and leaves residual changes in the image displayed. If you detect such a problem, you should stop using image comparison for such areas. Though decreasing of the pass rate to 68% temporarily resolves the failure, you can't be really sure whether this behavior will be stable. An appropriate solution in case of Firefox would be to check existence of an image in a loaded document or to cut a piece of text from the browser's menu instead.

Should you have problems with image comparison which can't be debugged the way we described above, contact us through the Contacts page. To allow us to debug the comparison process send us the VNCRobot version string together with the images which you compare, i.e. your template image and a screenshot of your remote desktop image or at least a subimage of the area you are searching.