Image Comparison With VNCRobot 1.3
Contents
1. Introduction
2. Image Comparison Method 'default'
2.1 Algorithm Description
2.2 Usage
3. Image Comparison Method 'search'
3.1 Algorithm Description
3.2 Usage
4. VNCRobot Scripting Language and Image Comparison
5. Troubleshooting Image
Comparisons
5.1 Image Comparison Factors
5.2 Real Life Example
5.3 Debugging Of Failed Image Searches
1. Introduction
Unlike other automated testing software, tools working over the RFB
(VNC) protocol like VNCRobot work just with the image of the remote
desktop. That's why image comparison is one of the few means to verify
the contents of the server desktop and behavior of the tested object
(software, application, OS, ...).
Imager comparison algorithm (method)
is a module which accepts two
images on input and produces a number (usually percentage) indicating
how much the images match. VNCRobot v1.3 provides two basic image
comparison methods:
- 'default' - Histogram based image
comparison method.
- 'search' - Method allowing to search
certain remote desktop image area whether it contains another image.
Users wishing to use custom image comparison algorithms may take
advantage of the VNCRobot 1.3 Open API
to write their own Java code and plug it into VNCRobot.
2. Image Comparison Method 'default'
2.1 Algorithm Description
The default method is based on histogram comparison. An image histogram
is in simple words a list of colors located in the image together with
number of pixels of each color. The method first calculates histograms
both of the remote desktop image and the template image and compares
them. Result of the image comparison is then calculated as a number of
matching pixels divided by the total number of image pixels.
The following example illustrates how the method works. Let us have two
simple images, image 1 and image 2:
Each image has 120x100=12000 pixels. The following table shows the
histograms and number of matching pixels:
Color
|
Image 1
|
Image 2
|
Matching pixels |
| White |
9000 pixels
|
6000 pixels
|
6000 |
Red
|
3000 pixels
|
6000 pixels
|
3000 |
Sum of matching pixels:
|
9000 |
The comparison result is in this case 9000px/12000px, which is 0.75, or
better 75%.
2.2 Usage
Though this image comparison method is very simple and has many
limitations, it is quite reliable and has a few advantages. It is for
example robust against changes in location of application windows
on
the remote desktop. The following example displays three
different images which will always produce 100% match if compared:
In order to get reliable results, consider the following hints:
- Change the remote desktop
background to a solid color to minimize
the number of mismatching pixels. Choose such a color which is not
contained in the application window which you are going to test.
- Remove all icons and items from
the remote server desktop and
make it as simple as possible. Note that most desktops also display a
clock/date which introduce mismatching image areas.
- It is a good practise to use the 'Mouse move' command to move
your mouse pointer to a constant position before comparison.
Mouse
pointer forms part of the remote desktop image and it is a source of
mismatching pixels.
- Do not perform image comparison against images with lossy
compression like e.g. JPEG. PNG, BMP and GIF formats are
preferred because they
preserve
100% of the image information and guarantees reliable and stable
comparison results.
- Check if the application window(s) you are testing have stable
sizes. Variable window size significantly affects the histogram
based
comparison results.
Note that the default method is designed to compare two images of
the same size. If your template image has different size than the
remote desktop, both images are cropped to the size of their
intersection befor comparison. The following picture illustrates such a
situation:
When a custom comparison area is specified via the cmparea
parameter, the remote desktop image is cropped from the area location,
i.e. from its [x,y] coordinates. The values of width and height of the
area are ignored in this case a are rather derived from the
intersection
size.
It is not recommended to use the default method together with a
custom comparison area because it is easy to make a mistake and compare
rectangles which do not correspond. If you need to check a certain area
of your remote desktop image whether it contains a certain image, use
the 'search' image comparison
method instead.
3. Image Comparison Method 'search'
3.1 Algorithm Description
The search method performs a simple pixel by pixel search of a template
image in the specified remote desktop image area. Though the algorithm
was slightly modified for better performance, it basically runs through
all pixels of the remote desktop image (or its part) and tries to find
match(es) of the template image. The template image must be
smaller than the remote desktop image.
Version 1.3.10 introduces limited support
of images with transparent pixels. When a search image
comparison is used together with a transparent template image, all
transparent or semi-transparent pixels (i.e. pixels with Alpha <
0xFF) are skipped. It means that if you have a template with 100 pixels
where 90 pixels are transparent, only the remaining 10 non-transparent
pixels will be compared against the desktop image. Transparent pixels
are always counted as passed. If e.g. one pixel out of those 10
non-transparent ones in our example will be different, result of the
comparison will be 99% (90 'correct' transparent pixels plus 9 equal
ones).
Transparency can be used to ensure that image comparison is robust
against background or component color changes. When you e.g. search an
icon rendered on your remote desktop, marking the background color
pixels transparent will make sure that the image comparison passes
regardless of the remote desktop color changes. Note that VNCRobot
1.3.x doesn't have any tools to add transparency to template images and
such features are planned for VNCRobot 2.0. Linux/Unix users may take
advantage of Gimp graphic editor to set up transparent pixels as
follows:
1. Open the template image with Gimp
2. Select Layer->Transparency->Add
Alpha Channel
3. Select Select->By Color
and click on the color you want to make
transparent in the image. Note that functionality of By Color is affected by
the threshold parameter which specifies similarity of colors to be
selected.
4. Select Edit->Cut to
remove the selected pixels (and make them
transparent).
5. Save the template image and retest with VNCRobot.
Older implementations of the search algorithm were not
able to handle the passrate
option. It was always looking for an exact pixel-by-pixel match
regardless of the desired pass rate. This behavior was changed in
VNCRobot 1.3.6 and the new implementation is able to search even for
image occurencies which have a certain amount of pixels different from
the template image. The pass rate is newly converted to a number of
pixels (template_pixel_count * (1 - passrate))
and the
remote desktop image is searched for all instances having a number of
different pixels lower than or equal to this number. If your template
image size is e.g. 100 pixels and you specify 99% pass rate,
VNCRobot will find all areas of remote server desktop having up to 1
different pixel. Note that the lower pass rate you specify, the lower
the performance will be and the longer the search will take.
The method returns 0% match if the template is not found or 100% if at
least one match is located. Coordinates of the found matches are then
provided in form of variables which can be accessed during script
execution and further processed. See the Compareto command
specification for more info on the variable format. Also note that
the number of occurencies is limited by a parameter in the
Compareto command preferences which can be customized in the VNCRobot
Preferences window.
It is recommended to save the template images in PNG format through the
Compareto, Screenshot
of Waitfor match windows of VNCRobot. There
are tools allowing you to crop the desired area of the remote desktop
image and save it as a template. If you use images saved by other
applications, it may happen that VNCRobot will not be able to match the
image due to an incompatible pixel format.
3.2 Usage
The method can be widely used to verify that the remote desktop image
contains a desired component or window. It is also possible to search
for a component or an icon and then click onto its coordinates.
It is recommended to use the default method together with a custom
comparison area (parameter cmparea of the scripting
language commands) to narrow the search area. Though you may search the
whole remote desktop, the algorithm is quite inefficient and it may
take
up to seconds to finish. The performance can be also severely affected
if there are too many matches. You can improve the performance by
proper
selection of the template image. The following example suggests how to
create templates for an efficient search.
Example of a remote desktop
Template Image
|
Comment
|
|
Incorrect. There will be
thousands of matches within the remote desktop's red rectangle.
|
|
Better. Though there will
be just one match, the large white stripe in the upper part will cause
delays in searching because the desktop background is white and the
method processes pixels by rows from the top left corner. |
|
Best. The first red pixel
is located close to the top left corner which will make the search
efficient. |
4. VNCRobot Scripting Language and Image Comparison
Image comparison modules are closely integrated with 3 commands of the
VNCRobot 1.3 scripting language:
- Compareto -
Compare a template image to the remote desktop.
- Screenshot -
Save a remote desktop screenshot to a file and eventually compare it to
a template image.
- Waitfor match/mismatch
- Pause the execution until ('match')
or while ('mismatch') the
remote
desktop matches to a template image.
All three commands support almost the same image comparison parameters.
There are Compareto, Screenshot of Waitfor
match windows allowing to construct and maintain the commands
easily. See the following picture showing a fragment of code with these
commands together with the corresponding image comparison GUI.
It is usually necessary to branch the script behavior based on the
image comparison result. There are two ways to ensure it:
- Use the
onpass and onfail command
parameters ontimeout in case of the Waitfor
command) to specify what should be done when image comparison succeeds
or fails. A disadvantage is that it is allowed to specify just one
command or procedure name.
- Use the
if statement together with the command exit
code.
The following two examples are equivalent and terminate the script
execution with an exit code 1 when the image comparison fails:
# Example #1 - 'onfail' parameter
Compareto test.png method=default onfail="exit 1"
# Example #2 - 'if' statement
if ({_EXIT_CODE} > 0) {
exit 1
}
Another example demonstrates how to search for a button template image
and then click it:
Compareto button.png
method="search"
# If the button is found, the exit code will be 0
if ({_EXIT_CODE} == 0) {
Mouse click to=x:{_SEARCH_X}+5,y:{_SEARCH_Y}+5
}
See the VNCRobot 1.3 Language
Reference document for additional examples and more information on
the scripting language.
5. Troubleshooting Image Comparisons
5.1 Image Comparison Factors
A number of users reported failing image comparisons with VNCRobot,
especially searches. Vast majority of these issues are not fault of
VNCRobot and they happen due to a number of factors. Consider the these
tips first:
- Conflicting mouse pointer. Yes, mouse pointer forms part of the
image, so move it out of the area of interest when you're going to
create template images and before you execute image comparison!
- Watch out for sources of regular changes of the remote desktop,
e.g. clock and date on your system task bar.
- Some systems and applications tend to highlight buttons and menu
items when mouse pointer is above them.
- Change in the environment or display settings may also affect
functionality of image comparison. If you create your templates on a 24
bit color remote desktop and later on restart the VNC server in 8 bit
color depth, it won't work.
- A few cases of unstable behavior of VNC servers were also
reported. Note that VNCRobot is primarily tested with TightVNC on
Ubuntu and we don't have resources to cover all VNC clones and
operating systems. If your server implementation deviates from expected
behavior, we can't do much about it.
The following two chapters are intended to give you an overview of how
to reveal causes of failing image comparisons. Be sure to give it a try
before you proceed to contacting us.
5.2 Real Life Example
A real life example of a failing image comparison follows. Let's
suppose you are testing a web page displayed in Firefox 2.0.0.7 on
Ubuntu
Linux 6.0.6. Your remote desktop looks like the following one:
To verify whether Firefox is correctly displayed you decide to cut
a piece of the image with one of the browser buttons and use it as a
template.
This can be easily done through the Compareto window. Let's suppose
that the template will contain the Refresh button
from the following
area:
If you set 100% pass rate together with the 'search' image comparison method
and hit the 'Compare...'
button, you get a
nice 100% match:
So far so good. Leave the Compareto Command window with OK. It should
create a Compareto command in your active script editor:
Then go to the
VNC viewer and move your mouse over the Refresh button. Note that
Firefox will highlight it:
Now move your mouse pointer away from the button so that its original
state is restored. Right click the Compareto
command in the editor and select 'Properties'
in the pop up menu to reopen the 'Compareto
Command' window. If you retry the
image comparison, VNCRobot fails to find the button for no
obvious reason. When you cut out the same image again and compare it
with the old one, they visually appear to be the same:
Old Image
|
|
New Image
|
|
Do you think that there's no difference? Yes, there unfortunately is,
and VNCRobot
provides powerful tools to debug it.
5.2 Debugging Of Failed Image Searches
VNCRobot v1.3.8 introduced two important debugging features. The 'search' image comparison module
was enhanced to handle non-exact searches (i.e. searches with pass rate
lower than 100%) and tracking of differencies between two images was
implemented. This new functionality allowed the 'Compareto Command', 'Screenshot Command' and 'Waitfor Command' windows to debug
different pixels of a template image and a certain match area of the
remote desktop image (or any other image loaded
through the 'Load RD Image'
button).
To access this functionality you have to achieve a search match first.
If you are doing image comparison through the 'default' method, switch to the 'search' one temporarily. There are
two techniques to achieve a match in failing cases:
- Decrease the pass rate gradually and verify existence of a match
through the 'Compare...'
button. This approach helps in most of the cases.
- Enter the comparison area equal to (in case you know it exactly)
or
a bit bigger than the original cut rectangle. This will allow you to
decrease the pass rate to values close to 0% without having thousands
of matches in the whole picture. It also allows the image comparison to
run much faster. In our example the area equals to x:94,y:81,w:30,h:30.
To achieve a match in our Firefox example above one has to decrease the
pass rate to approx. 68%. Review the modified parameters of pass rate
and comparison area highlighted in the following picture:
Once you have a match in the expected area, select the 'Show Matches' button to display
more details. Part of the window is shown on the following picture:
Our template image has 900 pixels (30x30). The window reveals that
there are 281 different pixels which means that more than 31% of the
template image pixels don't match with the specified area of the remote
desktop image. These pixels are displayed in green. When you place your
mouse pointer above one of them, the label below the drop down in the
top left corner displays the pixel coordinates both in the remote
dektop and template image coordinate systems together with the
corresponding pixel colors. Though the mouse pointer is not captured in
the picture above, you can clearly see that pixel at [100,93] of the
remote desktop corresponds to template pixel [6,12] but they have
different colors.
Note that this is an example of non-standard behavior of an application
(Firefox) rather than VNCRobot failure. Firefox applies an image filter
to the button icon in different modes and leaves residual changes in
the image displayed. If you detect such a problem,
you should stop using image comparison for such areas. Though
decreasing of the pass rate to 68% temporarily resolves the failure,
you can't be really sure whether this behavior will be stable. An
appropriate solution in case of Firefox would be to check existence of
an image in a loaded document or to cut a piece of text from the
browser's menu instead.
Should you have problems with image comparison which can't be debugged
the way we described above, contact us through the Contacts page. To
allow us to debug the comparison process send us the VNCRobot version
string together with the images which you compare, i.e. your template
image and a screenshot of your remote desktop image or at least a
subimage of the area you are searching.