Testing Tools

We discussed how to plan quality activities in my last article in this Quality Management series, including how to plan test tool acquisitions. This article will expand on the theme of automated test tools, their impact on the project, and how to choose a set that is suitable to your project. This is not a consumer report on all the various tools available; once you have decided on the type of tools (if any) that suit your project, you should use your existing vendor selection process to choose the right vendor.

I mentioned in my first article that the type of software being developed, and the development platform used, constrains the types of tools available to you. Other influences on your selection will be the budget available to purchase these tools, the degree of training necessary, and the applicability of the tools on future projects. You will need to consider all these factors before making your final decision on the tool set. The type of tools that are suitable to testing requirements will change as the project phase changes, tools appropriate for development use will not be suitable for QA use.

Testing has 2 purposes: to ensure that the software product we deliver meets all the quality standards set for it by the project and that the quality is delivered as cost effectively as possible. Both these objectives are achievable without automated testing tools, but the tools make testing quicker and less time consuming. There is a maxim in the world of software testing that says bugs caught early in the development cycle are the cheapest to fix. Bugs caught in production are the most expensive. Your testing, and the tools to be used should be planned accordingly.

Development Test Tools

Development test tools are designed to be used in the development environment, by software developers to perform automated unit, function, thread, and system testing. I can best explain the difference between these types of tests by referring to the size of the chunk of code being tested. A unit test tests the smallest chunk of code that is capable of standing alone. Unit tests are usually performed on “modules”, although any chunk of code that can be compiled and driven with inputs and outputs can be unit tested. Function or thread testing tests the next largest chunk of code. This chunk can be made up of several units or modules and comprise a whole function. The log on function in a software system is a typical function and this function may be made up of modules to display the log on screen and capture the username and id, a module to check the password against the database, and a module that assigns appropriate privileges to the userid.

These tools are very much platform dependent as they must be integrated into the development environment. Once the choice of programming language and platform has been made, you can choose a developer test tool that is compatible with your choice. Different tools also offer different feature sets that their vendors feel are most compatible with the code under test.

Features of the various test tools can be divided into 2 groups or categories: static tests, and dynamic tests. Static testing examines the code at rest for things such as correct variable initialization, boundary conditions, memory leeks, etc. Dynamic tests exercise the code using stubs and drivers, including wrapper code which will automatically initialize global variables. Other features will measure code compliance with coding standards (the standard will vary with the language). These tools will also allow you to collect metrics on code coverage, and error detection. The feature set your project will require will depend on the size and complexity of the code, and the size and complexity of your budget. Don’t forget that a learning curve always accompanies the acquisition of new tools and the more feature rich the tool you choose the more essential training will be.

Integration Testing

Integration testing can only be done once unit and function testing is completed but should be done before Quality Assurance testing begins. These tests exercise the system as a whole and while it may be appropriate to repeat some or all of the tests performed during development, tests should also include those that exercise the system as a whole. For example, rather than simply test the log on function, an integration test would verify that the correct privileges are assigned to the logged in user so that they can perform another function, such as order placement.

Integration testing can be done using most Continuous Integration tools such as Hudson or Confluence. The value of these tools is not only their ability to automate integration testing, but the automation of the build process. Compiles can be scheduled periodically or triggered by events such as a developer checking their code into the source library. The advantage these frequent builds gives the project is the avoidance of resolving all the integration bugs at the last minute when the system is to be passed along to the Quality Assurance group for testing.

Quality Assurance Test Tools

QA requires a different tool set because of the different nature of the testing. QA testing should take a “black box” approach, testing the system against the requirements and specifications defined for the system. There are 2 categories of tool that facilitate this testing: automated test tools and bug reporting and tracking tools. Regression testing is important to the QA test phase to avoid the risk of a bug fix in one area of the system causing another area to fail. Regression testing detects this by exercising the entire system with the complete suite of tests. Performing these tests manually takes a great deal of time and effort, the more often they are run, the more effort required because they will take just as long the 10^th time they are run as the 1^st. Automated test tools will relieve the strain of multiple cycles of regression testing.

Automating your regression testing will require different tools depending on the programming language the application is written in. Any application that is based on a GUI interface will require a tool capable of key stroke capture and replay. Applications that don’t can use tools that use test harnesses, scripts, and data files to perform testing. Regression testing tools designed for non-GUI applications will differ depending on the platforms the application is designed to run on. Applications that run in a client-server environment will require a different tool than those that don’t. Tools may also differ depending on the protocol used – HTTP, SMTP, Telnet, DNS, etc.

Key stroke capture and playback tools rely on the simple mechanism of storing a sequence of keystrokes, including pauses, and then playing them back to simulate a user exercising the system. One advantage of these tools is they are independent of the architecture they are running on because they simulate an actual user rather than exercise the system artificially. Another advantage of these tools is their ability to record other facets of system behavior such as how many simultaneous users can log on, how quickly the system performs under load, etc. These capabilities make these tools very useful for performance, load, and stress testing. Their disadvantage is the time and effort it takes to create the regression test suite to begin with and to maintain it after creation.

Mercury Interactive was a leader in the field of automated GUI, performance, and load testing with tools such as Winrunner and Loadrunner. Mercury was acquired by HP recently and the tools that used to be sold under the Mercury Interactive brand now appear under the HP brand, but are still the same tools. These tools include performance measuring tools in addition to Winrunner and Loadrunner. I’m not here to promote HP products, but would advise project managers to review the range of QA test tools offered on HP’s web site to get an understanding of the breadth and depth of tools available.

There are a wide variety of bug reporting tools available, with equally diverse feature sets. When you research these tools, you’ll find that they are sometimes referred to as “issue tracking” and sometimes as “bug reporting”. I’m not certain what the difference is, if any, but would recommend that you evaluate the tools in terms of their feature sets regardless of what it is called.

It is unlikely that you will be undertaking a software development project in an environment without a bug reporting tool, unless this is your organizations first software development project. The goal here is to assess your project’s needs and determine if the existing tool provides all the functionality necessary to meet them. One approach to take to bug reporting is to integrate that activity with the other project communications activities using a wiki. Confluence is one example of a wiki and it uses a bug reporting tool called JIRA which is integrated with the rest of the Confluence features. Other bug reporting tools are integrated with software management systems, such as ClearQuest which is integrated with IBM’s Rational Rose system. Prices for these tools will also vary according to their feature sets, from freeware such as Bugzilla, to expensive such as Rational. There are two things to keep in mind when choosing your tools: 1) Do I really need all the features in the tool, and 2) How much will training cost. The more feature rich and complex the tool the greater the need for training.

A good reporting tool is an important component of your project management tool set. The reporting function may be supported by a feature in the bug reporting tool, or may be purchased separately. Look for a reporting tool that will allow you to isolate bugs by cause, by developer group or individual developer, and by severity. You also need to know how many were opened during a reporting period, how many were closed, and how many were re-opened. These reports should support showing your stakeholders how the project is progressing towards meeting its quality goals and objectives so should support the creation of charts and graphs, or at least exporting data to Excel. You also need to identify problem areas so preventive or corrective action can be taken to address the problem. Queue size is a key indicator of quality issues so you need to know if the number of bugs being opened growing faster than the number being closed.

Selling the Benefits

Software testing tools are organizational assets, not project assets. These tools, and the training in their use, should serve future projects and operational activities so it is unfair to base any business case for their purchase solely on the benefits to be derived by the project. You may want to make a case for the purchase and implementation of these tools to be performed outside the project, in which case you must ensure that your project’s needs are part of the selection criteria.

Should you be required to state the business case for the purchase of testing tools as part of your project, do include the benefits to be derived from operations and future projects 2, 3, or 5 years down the road. Gathering this information may seem beyond your responsibilities as the project manager, but will make or break the business case for buying these tools.

Conclusions

Analyze the needs of your project, using the goals, objectives, and standards specified in your Project Charter, Statement of Work, or Scope Statement to determine your need for automated test tools. You will need to make a business case for the acquisition of any tools not already owned by your organization. Future benefits to the organization from the use of the tools should be included in the business case, in addition to the benefits to be derived from your project.

Training costs are part of the acquisition costs for a lot of these tools, don’t neglect to identify those costs in your business case and to include the training in your product/vendor evaluation. Allow some budget room for training of any new users who join the project team after the initial round of training.

Identify the additional hardware and software that the testing tools will require. Hardware will include PCs, servers, disc space, internet connections, etc. and software may include such things as extra developer licenses for testing platforms. Also include any database instances that must be acquired to support the testing which will also include servers and licenses.

Finally, make sure you consider your own needs when selecting testing tools. You have a need for tools to facilitate the creation of QA reports to report quality progress to your stakeholders and to monitor that progress to determine when to take preventive or corrective actions.