Quality Assurance (QA) testing should be done for every
software development project regardless of whether the performing organization
has a Quality Assurance department or not. Organizations that create QA
departments do so because of recognition that the skills required to perform
thorough QA testing are unique and different from those required to create the
software, not because the organization suddenly recognizes the need for QA
testing. QA testing should deliver a product which meets all the stated
requirements and all quality goals and objectives.
Your organization may view quality and QA testing as the
domain of the QA department but you must remember that you have overall
responsibility for quality as project manager. You must ensure that the QA
resources devoted to testing your project are sufficient to deliver quality
goals and objectives. The QA department may take responsibility for planning
their activities but you must review the plan for its ability to deliver on quality
goals. Project managers who don’t have a QA department must acquire testing
resources and plan the testing activities themselves.
The Test Plan
QA activities, including the acquisition of resources should
be set out in a Quality Management Plan. This can be something as simple as an
e-mail, or it can be a full blown project document. The Quality Management plan
should capture the project’s quality goals and objectives and describe the
approach the project will take to meeting quality goals and objectives. It may
also include details such as roles and responsibilities, reports to be produced
etc. The activities called for by the plan should be captured in the WBS and
scheduled. The first activity to be performed will be the creation of test
Test cases should be created from the Business Requirements
Document, Commercial Specification, or other document used to capture business
requirements for the software system. It is important for the testers to work
from a business document to create their test cases, particularly if the
testers are not part of a QA department. The purpose of QA testing is to test
the system against business requirements, not the design. Creating test cases
based on system design could cause design errors to be missed. For example, the
system design may be delivered by the code but the design may not satisfy the
business requirements. Test cases may be grouped into test suites by
sub-system, functional area, tester, or some other means. Test cases which have
been grouped together should be treated as a single entity.
Test cases should be reviewed by the Business Analyst to
ensure the test cases exercise the system the way the business intended. Test
cases should be completed before the coding phase begins and developers should
have access to the test cases so there is less risk of the system design or
code failing to deliver the business requirements. Taking this approach will
require testers to be on board during the planning phase of the project, but
the return on this investment is the elimination of bugs caused by requirements
miss-interpretation. The test cases become components of the software system
and should be controlled by the organization’s configuration management system.
If your project doesn’t have a QA department to write test
cases, try to acquire team members who have experience with QA testing (not
just development testing). Making these people responsible for writing test
cases is the closest you can come to having a QA department. Try to keep test case
writing and QA testing activities separate from development activities if you
don’t have a QA department and if you have to mix testing and development
activities try to at least have a developer other than the one responsible for
the code write the test cases. Having a developer who is responsible for the
design and code write the test cases risks propagation of errors caused by miss-interpretation
of the requirements.
QA testing should start with a fresh environment with all
software applications and necessary software licenses. A database instance
should also be installed and populated with test data. Data can be sourced from
several areas. The common or global data used in the development environment
can be ported to the QA environment, data created by developers to perform
tests similar to the QA test cases can be ported, and testers can supply data.
QA testing should try to make as efficient use of data as possible; an order
created by one test case can be the order that is processed by the next test
Monitor the bug reports in the queue used to report QA
testing results. Rapid growth of the queue will indicate a deluge of re-work.
This may have a negative impact on the development phase of an iterative
project, or overload the remaining developers on a waterfall project. Bug
reports should also be monitored for rapid closure of severity 1 bugs. I
usually reserve the severity 1 designation to bugs that prevent further testing
of at least 1 major function. These bugs have the potential of delaying the test
schedule and therefore final delivery of the project. You should also monitor
bug reports for bugs that should have been detected in development testing.
Examples of this type of bug are code that crashes instead of displaying an error
message, or code that doesn’t handle inputs outside the acceptable range
cleanly. A high ratio of development related bugs may indicate inadequate
development testing. Be especially suspicious if these bugs are related to one
You should also monitor reports for bugs that are caused by
a difference between tester and developer expectations of how the system should
behave. An example of this type of bug is a system which displays an error
message complaining of an invalid character in a log in userid and then exits
when the tester expects the system to collect the password before it displays
the error message. These bugs should be rare when the developer has access to
the test case before coding begins but will occur when they don’t.
Report on Quality Assurance results to the project
stakeholders. Your reports should communicate information required by those
stakeholders, but you can help set those expectations so that you inform them
of the number of executed test cases, test cases to be executed, the number of
bugs by severity, the number of bugs opened this period and the number closed
this period. Don’t try to hide bad news in the form of excessive bugs to
report, but don’t be alarmist either. There may be legitimate reasons for a
high volume of bug reports in a given period: a larger volume of code being
tested, or a particularly complex piece of code, for example. The execution of a higher
volume of test cases may also be an explanation. If it is, put the metric in
context by reporting on the increase in test cases executed, as well as the increase in bug reports.
Every time code is touched there is a risk of breaking it.
This includes when the code is updated during re-work to fix a bug. Regression
testing is the only way to mitigate this risk, not just the re-execution of the
test that failed, but the re-execution of all the tests. Regression testing may
be costly, particularly if the system has a GUI interface. The most effective
way of reducing this cost is to automate regression testing. This will
significantly reduce the cost of regression testing, although maintaining the
test suite during the project will have a cost. The cost of the tool will
include the cost of creating the tests initially and this can be costly.
Failing the acquisition of automated regression testing tools, you can try to
eliminate duplication by regression testing by sub-system for example.
Excessive failures during regression testing should be
analyzed for possible root causes. A change in one piece of code that causes an
unexpected failure in another area not visibly related is cause for concern.
One cause for this phenomenon is the improper handling of global variables.
Check the entire source library for more instances of the error when you find
this type of problem.
Performance, Load, and Stress Testing
These tests should be performed against benchmarks
established for the system in the Project Charter or SOW. Performance testing
should measure key system functions against benchmarks. Examples of these
benchmarks would be a maximum of 1 second for every page on the site to load, a
maximum of 2 seconds to log in after the userid and password are supplied, or a
maximum of 5 seconds to check an order with a maximum of 25 items. Performance
testing can be accomplished manually or by an automated test tool such as
Xrunner or Loadrunner.
Load testing is almost impossible to perform without an
automated tool to replicate the load. Load Testing tests system capability to
handle certain conditions resulting from user demand. The system may have a
requirement to support a peak logged in user threshold of 1,000 users for
instance. No-one should attempt to replicate this condition manually; if
capacity handling is critical buy an automated test tool. Load testing and performance
testing often go hand in hand. For example you may be required to deliver a
system which can handle 1,000 logged in users and log the last user in under 2
seconds, or process 50 orders of 25 items simultaneously and have each process
in a maximum of 5 seconds.
Stress testing is simply the systems response to a demand
outside of the limits set for it. In the previous examples your stress test
would be to log 1,000 users into the system and then observe system behavior
when the 1,001stuser attempts to log in. Normally the system should
fail "gracefully”, that is an error message notifying the 1,001stuser that the maximum number of users has been exceeded rather than have the
Performance, load, and system testing require a platform which
duplicates the production platform in terms of hardware capacity, processing
capability, network composition, etc. Performing this type of testing on a
platform that has less capacity, or does not have the constraints of the
production platform will render test results unusable. Regression, performance,
load, and stress testing should all form part of the quality reports you
communicate to the project’s stakeholders.
The tips and tricks described in this article implement some
of the best practices promoted by the PMI (Project Management Institute). These
are taught in most PMP®courses
and other PMP® exam preparation training products. If you haven't
been certified as a PMP® (Project Management Professional) by the
PMI and would like to learn more about certification, visit the three O Project
Solutions website at: http://threeo.ca/pmpcertifications29.php.
three O Project Solutions also offers a downloadable software based training
tool that has prepared project managers around the world to pass their
certification exams. For more information about this product, AceIt, visit the
three O website at: http://threeo.ca/aceit-features-c1288.php.