We discussed how to plan quality activities in my last
article in this Quality Management series, including how to plan test tool
acquisitions. This article will expand on the theme of automated test tools,
their impact on the project, and how to choose a set that is suitable to your
project. This is not a consumer report on all the various tools available; once
you have decided on the type of tools (if any) that suit your project, you
should use your existing vendor selection process to choose the right vendor.
I mentioned in my first article that the type of software
being developed, and the development platform used, constrains the types of
tools available to you. Other influences on your selection will be the budget
available to purchase these tools, the degree of training necessary, and the
applicability of the tools on future projects. You will need to consider all
these factors before making your final decision on the tool set. The type of
tools that are suitable to testing requirements will change as the project
phase changes, tools appropriate for development use will not be suitable for
Testing has 2 purposes: to ensure that the software product
we deliver meets all the quality standards set for it by the project and that
the quality is delivered as cost effectively as possible. Both these objectives
are achievable without automated testing tools, but the tools make testing
quicker and less time consuming. There is a maxim in the world of software
testing that says bugs caught early in the development cycle are the cheapest
to fix. Bugs caught in production are the most expensive. Your testing, and the
tools to be used should be planned accordingly.
Development Test Tools
Development test tools are designed to be used in the development
environment, by software developers to perform automated unit, function,
thread, and system testing. I can best explain the difference between these
types of tests by referring to the size of the chunk of code being tested. A
unit test tests the smallest chunk of code that is capable of standing alone.
Unit tests are usually performed on “modules”, although any chunk of code that
can be compiled and driven with inputs and outputs can be unit tested. Function
or thread testing tests the next largest chunk of code. This chunk can be made
up of several units or modules and comprise a whole function. The log on
function in a software system is a typical function and this function may be
made up of modules to display the log on screen and capture the username and
id, a module to check the password against the database, and a module that
assigns appropriate privileges to the userid.
These tools are very much platform dependent as they must be
integrated into the development environment. Once the choice of programming
language and platform has been made, you can choose a developer test tool that
is compatible with your choice. Different tools also offer different feature
sets that their vendors feel are most compatible with the code under test.
Features of the various test tools can be divided into 2
groups or categories: static tests, and dynamic tests. Static testing examines
the code at rest for things such as correct variable initialization, boundary
conditions, memory leeks, etc. Dynamic tests exercise the code using stubs and
drivers, including wrapper code which will automatically initialize global
variables. Other features will measure code compliance with coding standards
(the standard will vary with the language). These tools will also allow you to
collect metrics on code coverage, and error detection. The feature set your
project will require will depend on the size and complexity of the code, and
the size and complexity of your budget. Don’t forget that a learning curve
always accompanies the acquisition of new tools and the more feature rich the
tool you choose the more essential training will be.
Integration testing can only be done once unit and function
testing is completed but should be done before Quality Assurance testing
begins. These tests exercise the system as a whole and while it may be
appropriate to repeat some or all of the tests performed during development,
tests should also include those that exercise the system as a whole. For
example, rather than simply test the log on function, an integration test would
verify that the correct privileges are assigned to the logged in user so that
they can perform another function, such as order placement.
Integration testing can be done using most Continuous
Integration tools such as Hudson
or Confluence. The value of these tools is not only their ability to automate
integration testing, but the automation of the build process. Compiles can be
scheduled periodically or triggered by events such as a developer checking
their code into the source library. The advantage these frequent builds gives
the project is the avoidance of resolving all the integration bugs at the last
minute when the system is to be passed along to the Quality Assurance group for
Quality Assurance Test Tools
QA requires a different tool set because of the different
nature of the testing. QA testing should take a “black box” approach, testing
the system against the requirements and specifications defined for the system.
There are 2 categories of tool that facilitate this testing: automated test
tools and bug reporting and tracking tools. Regression testing is important to
the QA test phase to avoid the risk of a bug fix in one area of the system
causing another area to fail. Regression testing detects this by exercising the
entire system with the complete suite of tests. Performing these tests manually
takes a great deal of time and effort, the more often they are run, the more
effort required because they will take just as long the 10th time
they are run as the 1st. Automated test tools will relieve the
strain of multiple cycles of regression testing.
Automating your regression testing will require different
tools depending on the programming language the application is written in. Any
application that is based on a GUI interface will require a tool capable of key
stroke capture and replay. Applications that don’t can use tools that use test
harnesses, scripts, and data files to perform testing. Regression testing tools
designed for non-GUI applications will differ depending on the platforms the
application is designed to run on. Applications that run in a client-server
environment will require a different tool than those that don’t. Tools may also
differ depending on the protocol used – HTTP, SMTP, Telnet, DNS, etc.
Key stroke capture and playback tools rely on the simple
mechanism of storing a sequence of keystrokes, including pauses, and then
playing them back to simulate a user exercising the system. One advantage of
these tools is they are independent of the architecture they are running on
because they simulate an actual user rather than exercise the system
artificially. Another advantage of these tools is their ability to record other
facets of system behavior such as how many simultaneous users can log on, how
quickly the system performs under load, etc. These capabilities make these
tools very useful for performance, load, and stress testing. Their disadvantage
is the time and effort it takes to create the regression test suite to begin
with and to maintain it after creation.
Mercury Interactive was a leader in the field of automated
GUI, performance, and load testing with tools such as Winrunner and Loadrunner.
Mercury was acquired by HP recently and the tools that used to be sold under
the Mercury Interactive brand now appear under the HP brand, but are still the
same tools. These tools include performance measuring tools in addition to
Winrunner and Loadrunner. I’m not here to promote HP products, but would advise
project managers to review the range of QA test tools offered on HP’s web site
to get an understanding of the breadth and depth of tools available.
There are a wide variety of bug reporting tools available,
with equally diverse feature sets. When you research these tools, you’ll find
that they are sometimes referred to as “issue tracking” and sometimes as “bug
reporting”. I’m not certain what the difference is, if any, but would recommend
that you evaluate the tools in terms of their feature sets regardless of what
it is called.
It is unlikely that you will be undertaking a software
development project in an environment without a bug reporting tool, unless this
is your organizations first software development project. The goal here is to
assess your project’s needs and determine if the existing tool provides all the
functionality necessary to meet them. One approach to take to bug reporting is
to integrate that activity with the other project communications activities
using a wiki. Confluence is one example of a wiki and it uses a bug reporting
tool called JIRA which is integrated with the rest of the Confluence features.
Other bug reporting tools are integrated with software management systems, such
as ClearQuest which is integrated with IBM’s Rational Rose system. Prices for
these tools will also vary according to their feature sets, from freeware such
as Bugzilla, to expensive such as Rational. There are two things to keep in
mind when choosing your tools: 1) Do I really need all the features in the
tool, and 2) How much will training cost. The more feature rich and complex the
tool the greater the need for training.
A good reporting tool is an important component of your
project management tool set. The reporting function may be supported by a
feature in the bug reporting tool, or may be purchased separately. Look for a
reporting tool that will allow you to isolate bugs by cause, by developer group
or individual developer, and by severity. You also need to know how many were
opened during a reporting period, how many were closed, and how many were
re-opened. These reports should support showing your stakeholders how the
project is progressing towards meeting its quality goals and objectives so
should support the creation of charts and graphs, or at least exporting data to
Excel. You also need to identify problem areas so preventive or corrective
action can be taken to address the problem. Queue size is a key indicator of
quality issues so you need to know if the number of bugs being opened growing
faster than the number being closed.
Selling the Benefits
Software testing tools are organizational assets, not
project assets. These tools, and the training in their use, should serve future
projects and operational activities so it is unfair to base any business case
for their purchase solely on the benefits to be derived by the project. You may
want to make a case for the purchase and implementation of these tools to be
performed outside the project, in which case you must ensure that your
project’s needs are part of the selection criteria.
Should you be required to state the business case for the
purchase of testing tools as part of your project, do include the benefits to
be derived from operations and future projects 2, 3, or 5 years down the road. Gathering
this information may seem beyond your responsibilities as the project manager,
but will make or break the business case for buying these tools.
Analyze the needs of your project, using the goals,
objectives, and standards specified in your Project Charter, Statement of Work,
or Scope Statement to determine your need for automated test tools. You will
need to make a business case for the acquisition of any tools not already owned
by your organization. Future benefits to the organization from the use of the
tools should be included in the business case, in addition to the benefits to
be derived from your project.
Training costs are part of the acquisition costs for a lot
of these tools, don’t neglect to identify those costs in your business case and
to include the training in your product/vendor evaluation. Allow some budget
room for training of any new users who join the project team after the initial
round of training.
Identify the additional hardware and software that the
testing tools will require. Hardware will include PCs, servers, disc space,
internet connections, etc. and software may include such things as extra
developer licenses for testing platforms. Also include any database instances
that must be acquired to support the testing which will also include servers and
Finally, make sure you consider your own needs when
selecting testing tools. You have a need for tools to facilitate the creation
of QA reports to report quality progress to your stakeholders and to monitor
that progress to determine when to take preventive or corrective actions.