Development testing covers unit, function/thread,
integration, and system testing. The purpose of this testing is to ensure that
the software performs all the functions specified in the design document before
being passed to the Quality Assurance group to be tested against business
requirements. This testing may be done with or without automated testing tools.
Automated test tools are designed to reduce the manual effort required to
perform the tests and provide metrics for test results, test coverage, and in
some cases to enforce coding standards. Automated test tools and their use are
described in a separate article; this article will describe the functions
required for development testing and leave the decision to use automated test tools,
and the choice of those tools, to the reader.
The project manager should ensure that there a standards set
for the application or system that is passed on to the Quality Assurance group
for their testing. These standards will seldom be specified in the SOW, Scope
Statement, or Project Charter so must be derived from the overall goals set for
the final product. Standards for development testing should include standards
for code coverage and number of unresolved bugs. Quality standards for the
software application, or system, under test must be enforced during integration
or system testing but developers should be held accountable for the quality of
their individual code. Accountability can be enforced by reporting test results
for unit, function/thread testing, or by analyzing test results during
integration/system testing to determine if any of the errors should have been
caught during unit testing.
The project manager should take advantage of any reporting
capabilities of the automated tools employed. They should rely on a manual
reporting approach in the absence of automated reports to ensure that all
planned testing activities are being carried out and that all quality goals are
Unit testing exercises the smallest chunks of code to ensure
they work properly and satisfy the requirements described in the design
document. Units will usually be functions, procedures, web pages, or any other
piece of code that performs a particular task. To perform unit testing the
developer will need to create stubs and drivers to replicate the system or
application the unit will function in. The stubs replicate functions that the
unit outputs information to and the drivers replicate functions that input data
to the unit. Many software languages employ the concept of global variables, or
variables that are available to the entire application rather than to a single
function. These variables must be replicated by some form of wrapper. With
stubs, drivers, and wrapper in place the developer can now compile the unit and
test with test data. Unit tests are sometimes referred to as "white box” tests,
meaning that the tester has visibility to the code under test. That visibility
enables the tester to determine the degree of code coverage, among other things.
White box testing differs from black box testing in that, in black box testing, the tester has no
visibility to the source code under test, and doesn’t need it. They are simply
testing a sub-program against a set of requirements to ensure the code
satisfies the requirements.
Unit tests should include such things as boundary tests and
exception tests in addition to verifying the unit meets the requirements of the
design. Boundary tests verify that the unit handles all inputs
between the specified range (e.g. if the input is a number between 1 and 10 the
unit should handle 1 and 10 as well as the numbers in between). Exception tests
verify that the unit handles a number outside the expected range, usually
through an error message. In the example just quoted, the unit should handle
each number between 1 and 10 and display, or "throw”, an error if presented
with an input of 0.
Ensure that the development team knows what unit test
activities are expected of them and of the quality standards their code must
meet. Standards will include the number of permissible bugs and the percentage
of code coverage their testing must provide. Ensure that the team is trained on
any automated test tools provided for the project. Many automated test tools
come with tool training provided by the vendor; you may want to consider
combining project training with the vendor’s tool training to ensure tools are
being used correctly.
Unit testing will add time to code writing, after all it is
far quicker to write a few hundred lines of code and chuck that into the source
library and let someone else worry about any bugs in the code, than it is to
perform thorough testing. Ensure that you allow sufficient time in your effort
and duration estimates to allow for proper unit testing. The subject matter
expert responsible for estimating effort and duration should be aware of the
project plans for testing and factor those into their estimates. Now that the
team has been given the tools and time they need to perform the planned unit
testing it's up to you to monitor results to ensure the plan is being followed.
You may also wish to capture metrics on number of tests, number of bugs found
and fixed, code coverage, and any other areas that you measure.
Function or thread testing differs from unit testing in that
function testing will combine units previously tested into a chunk of code that
performs one or more complete functions. Developers can perform function tests
by combining the previously tested units which perform the function under test.
Frequently unit tests and function tests are combined by choosing a chunk of
code that encompasses the entire function for the unit test. There is nothing
wrong with this approach except where this combination makes testing too
complex. Both white box and black box testing must occur where unit and
function testing are combined.
Function tests call for test cases to be written by the
developer, or tester, which are based on the requirements captured in the
business requirements document. Here’s a tip that could save a lot of
development and QA time: make the test cases written by the QA group available
to the developers. This will require the QA group to begin writing their test
cases as soon as the business requirements documentation is completed but can
avoid developers wasting time coding a non-existent requirement, or misreading
a requirement. The test cases written by the QA group can also be used by the
development team either intact or with slight modifications. I’m not saying
that QA testers are always right when it comes to interpreting business
requirements, but they should be your arbiter until such time as the
application is tested by the users.
Many software systems employ a database as a data
repository. Software development projects that deliver a database as part of
the system will require an instance of that database for the development phase.
Each developer will also need their own instance of the database for their
testing. Organizing the database licenses in this fashion, as opposed to a
single shared database, will avoid much wasted effort and confusion around
sharing test data. Each developer should be responsible for the creation and
management of the test data unique to their area of development. Data that is
common across all developers should be provided in the shared database.
Examples of common data would be customer ids, addresses, telephone numbers, or
product ids. Although the developer must know the attributes of every data
element handled by their code, making this data available "off the shelf” will
make the developers’ lives much easier and can be provided with very little
effort. Ultimately the data required to complete functional testing must be the
Functional testing should also have goals and objectives.
The goals and objectives you select for your project will be unique to the
project, but here are a few which are universal: 100% of written test cases executed,
no severity one bugs open, no more than __ severity two bugs open, etc. Now
that functional testing is complete the code can be checked into the source
library ready for integration, or system, testing. You may want to track some
metrics from this activity as well, such as number of test cases, number of
test cases passed, number failed, etc.
The principal difference between integration testing and
system testing is that integration testing determines how well the various
software pieces perform together when they are integrated (or compiled),
whereas system testing tests the software application, hardware platform, and
database together to verify they perform properly. System testing requires that
the same type of hardware components are installed in the hardware platform as
will be installed in the production environment. This doesn’t necessarily mean
that the system test environment is an exact duplicate of the production
environment, but the classes and types of servers and database should be
duplicated. The configuration of the production environment will be unique to
the system being developed so the system test environment required will also be
unique. The environments, hardware, and software licenses necessary for the
various test environments should be identified in advance and their implementation
should be a part of the quality management plan. For our purposes, it is
sufficient to note that a test environment is required to perform this testing
and that the testing is of the "black box” variety.
Integration or system testing must be performed before the
system is turned over to the QA group for testing and that testing may be done
manually, or it may be automated as part of a Continuous Integration tool.
These tools were described in a previous article on automated test tools so we
won’t go into too much detail here, other than to mention that some of these
tools will store tests and execute them as part of the build process. Having a
suite of function tests put together by the development team is a good start to
the integration/system test library. The aim of this testing is to ensure the
pieces work together as a whole, so executing every test case for every piece
is not necessary.
Test cases must be written for functionality that spans the
entire system. For example, a software system consisting of order capture,
order processing, and order delivery sub-systems will have each of those
functionalities tested individually but integration/system testing should track
one order through the system from capture to delivery to ensure that the data each
successor sub-system expects from its predecessor is what it gets and that what
gets delivered by the system is what was ordered. Members of the team will be
responsible for writing these test cases and team members must also be
responsible for assembling the data necessary to execute these tests. The QA
team can be of help here; making their test cases available to the development
team, or system test team, will ensure that the system they test will be as
clean as possible and test data will be available to the QA team from the
development or system test environment.
A database administrator, or someone who can fill that role,
should be responsible for creation of the system/integration test environment.
This person may also be responsible for creating the individual scripts that
create the database, or each developer may be responsible for creating their
own portion of the database. Creation of the database will not be part of any
continuous build process so a database administrator should own the integration/system
test database and be responsible for its creation and refreshment with test
data for each new round of integration/system testing.
Test goals and objectives should be set for the
integration/system test phase of the project that supports the overall quality
objectives for the project. The goal of unit, function/thread, and
integration/system testing should be to eliminate all development type bugs
from the system before hand-off to the QA group. Development type bugs are
those that arise from a failure of the system to perform the way it was
designed to. Access to QA test cases can also eliminate bugs due to the system
not satisfying the business requirements (as interpreted by the QA group), and
this is a desirable thing. Remember that any bugs that the QA group finds in
the system must be reported using the bug reporting/issue tracking tool
implemented for the project. The QA tester must spend time reporting the bug,
the developer must spend time on re-work and report the solving of the problem,
and finally the QA tester must re-test the system and close the bug. Finding
these bugs before the QA test phase eliminates all the administrative work with
the bug reporting tool.
Your system should be ready for QA testing now. You should
keep track of progress to the integration/system testing goals and you may also
want to report system testing metrics such as number of system test cases,
number passed, etc. You should also analyze test results to determine if any of
the bugs discovered indicate insufficient function or unit testing. Reports of
quality metrics at this stage of development should serve to assure the project
stakeholders that the quality activities planned for the development phase have
been executed and the results support the quality goals and objectives set for
Monitoring and controlling activities necessary to meet
quality goals and objectives are described for you in the PMBOK Fourth Edition.
Getting your Project Management Professional (PMP) certification will not only
provide you with the overview necessary to use the techniques in this article,
it will demonstrate to your project stakeholders that you are competent in the
area of quality management. Look for a good PMP
Course, or other PMP Exam Preparation training to get you ready to pass the PMP