Open Source tools have been around for years, and it would seem that the downturn in the economy over the past few years, has pushed the use of these tools into the forefront of management consideration, like we have never seen before.

There is often something quite intangible, and some would say magical about using a free solution, and one that has particular steer in the psyche when looking at software tools. However… as you will see from the previous sentence… it is all too easy to mix the words “free” and open source” together as meaning the same thing.

What people forget when caught up in the religious zeal of open source is that there is a real tangible cost of implementing this type of solution, which often far outstrips the cost of a paid commercial product. Indeed… if people were just to slow down and pick the correct tool for the job, many would see that the commercial world of software has also changed exponentially over the same short period, and I certainly would argue now offers a more attractive overall packaged solution than any you would expect to find in the “free” world!

For example, in situations where a small test team has to test multiple software applications, with limited overall technical skill, and a wide range of supported environments (pc, mobile etc.); that a commercial solution should be your one and only focus…

Many teams these days have limited technical experience with automation. Yes you may have automation experts in the team etc. but scratch the surface and you will find limited experience past HP QTP, or anything outside a total all consuming indoctrination of HP UFT.
Even worse is the team with an automation guru, where his or her technical skill set is so one shop and biased towards a particular form of vegetable (cucumber, gerkin). In this instance the team will struggle to any other type of testing, and be left high and dry and open to ransom demands if he or she chooses to leave etc.

1. Free / open source tools are very technical to setup and run.

You will have to hire in, or technically train resources to implement and maintain your testing solution going forwards.
This is very costly and restricts you to a particular technology and process moving forwards.
You will find that your resources become developers, and it breeds a mentality of tester “fencing”, as I call it, where employees erect “Chinese walls” around what they develop, so that they become indispensable and therefore impossible to replace.

2. Free / open source tools require technical development.

Rarely, rarely, rarely are these tools “out of the box solutions”. All extra work, extensions, addins, bug fixes, issues etc. is work that you will have to pay for and develop yourselves. Can you afford this cost?

3. Free / open source tools require internal support.

Like the previous points, you will have to hire in, or technically train a leader to implement and maintain your testing solution going forwards. This presents a problem when sickness or holidays or etc. come into play. Resource unavailability could seriously mean your inability to test for the amount of time your resource is unavailable. Can you really afford this situation?

4. Free / open source tools can disappear at any moment.

There are numerous examples on the web, where companies have implemented a testing tool and embedded it into their testing framework, only to find that the project dies, or is taken over by a commercial vendor, which then charges for you to continue using new releases moving forwards.
Free to start, could mean an escalation of unknown huge costs in the future.

5. Free / open source tools are NOT supported! [without a fee!]

Many companies charge a support fee. In most cases it would have been cheaper to buy a commercial solution!

6. Free / open source tools are bound by different licensing.

You need to be very careful what you do with open source and free tools. Free does NOT mean that you are free to do ANYHING you want with the software in many cases.
For example… Most open source or free tools are covered under GPL. GPL licensing requires you to distribute any derivative work as open source under GPL. This is known as “copyleft”. The practical impact is that if you write a any new feature, plugin or enhancement, it is in terms of GPL considered to be a derivative work of the tool, and therefore covered by GPL.
I.e. If you decide to distribute your work to a 3rd party, you have to do so under GPL, and with the full source code. If you do not… you can be prosecuted.

7. Free / open source tools rarely integrate in with other tools out of the box, without development of integration features and plugins etc.

Again… work that you will have to do yourselves.

Can you afford all the above 7 deadly sins… I wonder… is the open source world looking less free, and commercial solutions more attractive… I hope so… but then I would, wouldn’t I!

T-Plan supports non-jailbroken device testing as well as rooted… Update for iOS 7.x…

If you want to use Veency to remote drive the device under test. I.e. You want to automate the whole device, and all the applications on the device as part of your testing, you will need to install a VNC Server – aka Veency – on your device. Once done you then connect to the device using T-Plan Robot’s in built VNC viewer.

This method is incredibly easy and allows the tester full UI testing of the device to mimic all actions as a user would. It also avoids the need for any source instrumentation code hooks etc. to be placed into the application at development time, allowing therefore testing of any application from the appstore, or pre-live from development etc.

The only proviso for Veency is that your device needs to be rooted, commonly called jailbroken.

For the testing of iOS 7.0.x devices, details on jailbreaking your device can be found on the Evasion website.

For the testing of iOS 7.1.1 devices, there is a new rooting option called Pangu.

Great news also is that Veency, by Jay Freeman (saurik) is being upgraded to the latest iOS versions.

1) Background…

Since it was founded in 1989, T-Plan has continued to develop software solutions for the Quality Assurance (QA) industry. Almost every organisation dealing with software, will have a testing department, or a team or group of persons responsible for the testing or verification of the software being implemented, or produced for market.

The challenges presented to testing the software, before it goes into production, have lead companies like us to develop solutions that aid the testing process, in industries such as mobile applications, defence, banking and gaming etc.

The T-Plan Product Suite allows you to manage every aspect of the Testing Process. The T-Plan Professional product, SaaS Cloud or On-Premise, is modular in design, clearly differentiating between the Analysis, Design, Management and Monitoring of the Test Assets. Test Automation is also integrated into the test suite package using T-Plan Robot, therefore creating a full testing solution.

For the purposes of this blog I want to now concentrate on our Test Automation solution…

2) Where do we sit?

Our product “T-Plan Robot Enterprise” is a Test Automation software tool.

In an industry such as gaming, speed of delivery is of the utmost importance. One of the biggest challenges to any auditable verification process is time. When squeezed for time due to delivery timescales, the testing or the software is normally the first to suffer.

Test Automation in this instance can really help in reducing the business risks of releasing a product, that has not had time to go through the proper battery of tests.

How… by getting computers to do the job of ‘regression testing’ whilst the workers are asleep, or by freeing up your workers so that they can concentrate on critical areas of the system that have just been produced.

The justifications for automation are well documented on the web… so in the interests of keeping this short, I will not reiterate these further here.

3) How does our tool work?

By creating a series of scripts or actions, T-Plan Robot can exercise or drive applications in the same way as would an end user.

In essence, imagine yourself sitting at your PC and running through a list of ‘pre-flight’ checks on the software, and recording these. Now imagine the PC, using our Robot software, playing this list back to you over and over again, perhaps under different circumstances etc. and then telling you the results when it was finished.

For example… you want to conduct the same list of checks simultaneously on a Windows PC / Windows Tablet / Mac / iPad / Linux PC (Ubuntu etc.) etc. to check that the software works in all of these environments. Welcome to the world of test automation…

4) How are we different?

The key to answering this is in the first sentence of point number 3) : “The same way as would an end user”.

Robot is an “image based” test automation tool and approaches validation from the end-user perspective, providing critical validation that what you want the customer to see, they will actually get. We do this by capturing images and checking these images have remain unchanged during the test execution etc. I.e. just like a human would do by comparing 2 photos… or 2 pictures from memory… but doing it programmatically, and therefore eliminating human error and saving time and therefore money!

The fact is that if you are not using Robot you may be testing your code and qualifying that it does not have any bugs that a computer can detect, but a computer isn’t a person. A person doesn’t read code when they are using an application they work with, or a game. Therefore, without testing your software’s interface (GUI) by “seeing it” or “reading it”, can you really confidently say that you know your application will deliver the desired experience to the end user.

Robot is also platform independent (Java) [environment agnostic] and runs on and automates all major systems such as Windows, Linux and Unix plus certain mobile platforms.  The product has the same functionality, look and feel regardless of Operating System (OS). There is no need to relearn the product on various platforms, and your test scripts are completely transferable between platforms. This is a huge point for the gaming industry as many games now have to sit with no alteration of functionality in many different environments, mobile included. Our solution is a one size fits all approach!

5) Time for a change!

Yes IBM / HP etc. provide testing tools… but their approach is very developer centric. The ‘big boys’ all approach test automation with very complicated “object oriented” test tools. These tools require a development knowledge of the system under test (SUT) and a technical checking of the code behind the displayed result.

In gaming the visual result is all important, and this in many circumstances cannot be verified at the component / object / code level. For example the work we have done with Zynga and Disney are classic examples of this where flash / flex / Silverlight etc. just cannot be easily tested using the ‘old’ tools provided by IBM and HP etc. Please view our YouTube Channel to see some examples of what and how we do it.

6) Resourcing and skill transfer.

As Robot fully supports Java as a programming language, I guarantee that you will find more people skilled in Java, than any proprietary scripting language supported by other vendors.

This makes your resourcing easier and more expandable across multiple locations, and off shoring of resource also should this be a consideration.

7) We are…

A UK based company with excellent support and customer service. We have an extensive global partner network, and pride ourselves in being competitive, flexible and addicted to customer retention!

We are totally customer focused, with a simple development focus and ethos to provide customers with functionality they desire.

Whilst I was watching back some GoPro video of me kitesurfing the other day… it struck me that a birds eye view of the situation often reveals some startling results. Sometimes it is good to take stock… look around and re-confirm that the world is round. However… in the often complex world of mobile device testing, this seldom happens. It should do… and needs to!

Between you, me and the gate post… there are some high level ‘gotchas’ that only surface when you have spent quite a lot of time researching and looking at solutions.

For what it’s worth (people will say that this is a sales pitch… but really this is me trying to be as honest as possible), my check list is as follows:

Device or not to Device. That is the Question…

1. Need to target specific OS’s as they are all different.
   1. I.e. IOS, Android etc. Unless you have a lot of resource you will just not manage to test everything… for example blackberry and windows etc. etc.
2. Need to decide whether you are doing simulators/emulators or real devices.
   1. Or a mixture of both.
   2. A lot of people I have spoken to, forget or ignore the power of simulators/emulators for testing.Whilst Simulators/emulators are used in development, their usage is not always carried forward into the testing environment. No real reason for this I think, other than a lack of educational swap-over, from development to test… Simulators/emulators can appear to be complicated and technical to setup (hardware and software), but can add real benefit to your testing solution. Do not ignore.
3. If (you should be!) testing real devices… You need to target specific devices as again the differences are complex.
   1. E.g. Differences between Android devices will be present… Differences between Samsung Galaxy S2 and HTC equivalent, can be keyboard, navigation, screen resolution etc.
   2. Also remember that IOS is again different from everything… has no Adobe Flash Players etc.

Simulation versus Emulation… Not Assimilation!

1. Need to properly understand the differences between simulators and emulators.
   1. iPhone Simulator in Apples’s Xcode only mimics the software environment. NOT the hardware resources such as memory, disk space, processor speed etc.
   2. Android Emulator mimics the software and hardware environments found on actual devices. The drawback is that they can be more complex to setup as you have to target real device configurations etc. BUT they are more true life obviously…
      1. Note: Though android emulator emulates the ARM processors and certain hardware, it still doesn’t do a good job of matching the CPU performance.
2. Simulators/emulators require e.g SDK installation according to manufacturer OS you are testing.
   1. Android needs Android SDK with then optional device environments for Samsung / HTC etc.
   2. IOS simulator testing only works on Mac with Apple SDK (Free download) installed. Need to be on recent Apple OS to support latest Apple SDK and therefore get latest device simulators for IOS 5 / iPad 3 and iPhone 5 etc.
3. Be wary of free Javascript simulators, or web plugins etc. These are most likely not going to give you a true environment and are most likely to give false positives etc. when you are testing. Go the extra mile and get setup properly. There are very few shortcuts in this game…
4. Can be very difficult to load the application under test into the simulator or emulator correctly. Development help and planning often required. E.g. An apple developer license and Xcode environment is often required… or a windows development license and environment setup etc.

Conclusion… so far… Device testing is very important…

Hardware… go Hard or go Home!

1. Most device testing requires a USB connection… so you need to think about desktop PC’s with the ability to have drivers etc. installed on them.
   1. E.g. Advanced connection to Android phones require you to have a) Android SDK installed for ADB connection and b) Model drivers etc. E.g. Samsung Kies, or HTC Sync for USB device connection etc.
   2. E.g. For IOS, Apple requires iTunes.
2. Device testing will largely be played out on the PC desktop… so Citrix and Terminal services etc. will make life exceptionally difficult I would imagine.
3. Device testing is usually singular in operation. I.e. normally you will have a single virtual machine and then host, or single machine setup at any one time, for each device model you are testing. For Testing this means multiple licenses will need to be purchased to carry out multiple device testing.

Applications… Angry Birds or Angry Nerds!

1. Getting the application onto the device is problematic, whether you are dealing with real device testing or simulation/emulation.
2. Application load in development environment:
   1. Requires a close synergy with your development team to load application from development and test in a proper production like environment.
      1. All too easy to take short cuts on setting up a proper environment to make life easy for the development team. However mistakes here will give false positives and questionable test results.
   2. More suited to simulation/emulation than real device testing due to publishing lag on markets etc.
3. Application load from Market:
   1. Testing can invariably start quite late in the cycle as publishing to Google Play (Android Market) or iTunes can take a lot of time and patience.
   2. If you are testing production ready applications, and depending on the number of devices you are testing; be prepared for multiple sign ups to Google and iTunes etc.

Testing Tools… or Testing Fools… Do not always trust what it says on the tin!

1. IF you are still reading this… and can remember the silver bullet era of testing by HP… then read on Macduff! Because as Nasa would say… we have re-entry!
2. There are a lot of testing companies that are advertising mobile testing tools, but only a few companies that are actually building them.
   1. A few of the big boys e.g. HP… Borland… IBM are OEM white labelling other products. E.g. Borland’s Silk Mobile is actually Experitest SeeTest.
3. Cloud or no Cloud.
   1. Cloud – Positives are that you do not have to own the devices… Negatives are that you do not own the devices…
   2. I know that this is a flippant remark… but the feedback to me from customers who have gone down the cloud route is that it is far more costly in the long run, and prone to availability downtime and configuration nightmares for clean environment testing etc.
4. There are in my opinion 2 main divisions of testing tools…
   1. Object based application automation – Application under Test – This is where all the big boys are playing. With slick marketing and lots of images, this is advertised as a ‘quick’ (anything but, I would say!) method of getting up and running with your mobile test automation.
      1. Invasive Source Instrumentation (compile-time) code required. – In nearly all cases… you will need access to developers and development environments… This is so that you can get the specific test tool hooks and object recognition code embedded and compiled into your application under test… so that you can drive, test and receive results. This is an application under test scenario… not a system under test! As most of these tools require development of the testing into the application being tested, you need to really look at the tools you are evaluating and ascertain the level of technical implementation impact your testing will have on the application development.
      2. With so much embedded code compiled into your application to allow the automation to function… how on earth do you call this a clean testing environment?
      3. How do you test the production release? Do you leave the embedded testing code in place? Or do you strip your testing code out and then hope for the best when you load it into the market for release?
      4. Very difficult in most circumstances to test outside the application sandbox. I.e. Application interaction with other applications like email / SMS etc. is hard to achieve.
      5. Very difficult to test “off the market” applications developed by third party suppliers or OEM partners etc.
      6. Having the testing code in the application can allow however for greater ease when testing multiple devices using the same or similar test scripts etc.
      7. Often does not require a jail broken or rooted device.
      8. Object orientated approach is often very technical. Needs careful resourcing of your test team etc.
   2. Image based application automation – System under Test [T-Plan Robot Enterprise] – Tests the mobile device (System under Test) by connecting via WiFi or USB, to a VNC server, or other server, running on the device.
      1. Due to the presence of a VNC server, or other server to command the device (keyboard / mouse etc.), there are 2 main options:
         1. NON-Jailbroken:
            • For iOS you need an Apple Development License (very inexpensive & easy to enrol) to install (using Xcode & iTunes) the server components etc. (Not available in the App Store).
            • For Android you can choose between VNC Server (VMLite), or use the  ADB (Android Debug Bridge with Android SDK, over USB or WiFi).
            • For other mobile environments such as Windows Mobile, Windows CE, Blackberry there are also VNC Server solutions available.
              • For Windows Phone life is tricky at the moment with a distinct lack of Microsoft & developer interest to provide tools for controlling applications, or the device, from a PC etc.
         2. Jailbroken (Root):
            • Rooting the device is a consideration that has to be planned for and decided upon. E.g. For iOS, whilst rooting and installing Veency on the device is difficult, the plethora of options this then gives in terms of fully controlling the device etc. is enormous. However rooting can invalidate your warranty so it would need to be sanctioned etc.
              For Android, as usual, the process is easy and the rewards are the same.
      2. Non-invasive – No application authoring to include test tool hooks etc. I.e. NO Source Instrumentation (compile-time) code required. Allows your ‘vanilla’ application to be received (from the developers or third party) / downloaded (from the market or test portal etc.) / installed (like it would be in a production environment) and tested. I.e. True production environment readiness can be achieved, as no foreign code has been included into the application for testing.
      3. Tests the application interaction with other application on the mobile device. As you are testing by connecting to the device… these tools truly test the system under test, and the application under test connectivity, to other applications on the device etc.
      4. Very easy to test outside the application sandbox. I.e. Application interaction with other applications like email / SMS etc. is hard to achieve.
      5. Very easy to test “off the market” applications developed by third party suppliers or OEM partners etc. Test per application, or test a suite of applications interaction synergy on the device.
      6. Multiple devices using the same script is supported with careful planning of the test image repositories etc.
      7. This GUI lead testing, tends to be much less technical than the development orientated approach of object recognition. Resourcing can be easier, as often particular skill sets are not required.

Futher Reading: Unbiased white paper on object versus image

The Opportunities for the Future of Test Automation

The opportunities in this space right now are tremendous and with the plethora of tools and services being provided at the moment, it is an exciting industry to be part of.
With this sentiment in mind it is my belief that we will continue to see an increased collaboration and communication between different companies, and therefore technologies, when testing projects are being carried out for customers. We have heard for some time now about grey box testing (essentially a combination of black box and white box testing), and it is my belief that we shall see an explosion of these sort of testing practices, being more and more deployed in test environments. For some this will be a use of a combination of different toolsets, whereas for others we will see developers and testers alike working to get a coherent test structure, utilising a combination of different techniques, data sets and of course tools.
For example: take the case of a company that is having to produce a product that now has to function in not just a windows environment, but in a mobile and Linux one also. Users expect a unique but consistent experience across the different technologies, but above all the present day consumer expects quality and security. We have become failure intolerant when using technologies that are no longer emerging, and have become intertwined with everyday life.
What this means for the software testing industry is a procurement of a selection of tools to test systems across a wide range of different environments. Historically we have been able to control the testing into fairly neat packages, as the tests conducted on the different environments, all used similar hardware and networks etc. E.g. tests carried out against different operating systems of Windows XP, Vista, and 7 etc. However in the mobile and tablet world, not only does the hardware, and networks need to change, but also the nature of the tests being deployed, as you are now in a world of widely different operating systems etc. Last but not least, you add Linux, Android and Chrome into the mix alongside the traditional desktop testing, and you have a whole range of tools that have to act in harmony.

Automating the Future

Coming back to the grey box testing ethos it means that in the future we can expect testing automation tools to have a firm footfall into the development of the systems under test, and the data deployed end to end within them. Test automation tools in this rapidly changing landscape need to be positioned at the black box or user level, as after all we need to make sure that the end user is getting the same unique and consistent experience, whatever the system or device they are running. What I mean by unique is that an Apple user still expects a Mac look and feel to their application, as does a Windows user also expects a windows look and feel to their application. I.e. a unique experience, but consistent functionality across the different platforms.
The positioning of these tools at the black box level is also important as the traditional object oriented automation tools, offered by the large providers, cannot keep pace with the wide range technologies. The rapid development interaction required to make sure that objects are embedded at the correct code level, to ensure workable automating solutions, is no longer the intelligent solution in a world now of many different devices and platforms.
Of course, in these situations, environment agnostic tools built on platforms like Java, which therefore function across multiple different systems, are ideally placed to drive the test automation. Indeed our test automation tool T-Plan Robot with its open architecture, can readily achieve this level of synergy by controlling multiple toolsets and interacting with data feeds. Indeed one can truly say that image based testing tools as a whole, have become more and more integrated into the full test process and the system development life cycle.
In order to satisfy the number and diversity of systems under test, the traditional demarcation lines between toolsets, have become blurred. It is my belief that a company can no longer consider just one suite of tools from a traditional vendor, but more a variety of tools from different vendors to achieve blanket coverage for their testing activities.

Read the full article in Test Magazine….