User Acceptance Testing

User Acceptance Testing is approached differently depending on whether the users of the system are internal to the organization, an external organization who have paid for the project, or individual external customers who buy a copy of the system or a license to use it. User Acceptance Testing is usually used to verify the system with internal users and external customers who sponsor the project. Testing by individual customers is most often referred to as Beta testing and requires a different approach when engaging the testers.

There are 2 goals this testing should accomplish: to demonstrate the system’s quality to the user community and to improve the product so that it is more acceptable to users. User Acceptance Testing should never be used to detect programming bugs, integration bugs, or design flaws. These should be detected by the previous testing phases: development testing and Quality Assurance testing. To achieve these goals UAT must be conducted in a production-like environment and the system exercised by at least one representative of every functional group in the user community.

Enlisting Beta Testers

Beta testing of a new product should be part of the marketing campaign for that product. The testers are enticed to take part in the testing program by some form of inducement. A standard part of the inducement should always be that the beta version is replaced by the production version at the end of beta testing, unless the customer chooses to keep their beta version. Other inducements might be a lower price, the opportunity to have input into product design, or a competitive advantage from having access to the beta version. Beyond targeting inducements at the various market segments, the organization developing the system does not have the ability to force customers into becoming beta testers.

Enlisting User Acceptance Testers

The users responsible for conducting User Acceptance Testing will be the responsibility of the customer organization in the case where you’ve built a system for a single external customer. It is in their best interests to ensure that coverage is complete and they are confident in the system before signing off on the contract. Your responsibility is to ensure that the users identified to you by the customer are provided with all the tools they need to conduct their tests.

You do have more influence over the user community in your own organization than users of external organizations, but will probably need some enticement to enlist users onto the test team. You should discuss your enlistment approach with your project’s business sponsor; this is likely to be a person who has authority over the user community and can help you with your enlistment campaign.

From your perspective the inducements are obvious: the system you are producing will replace an antiquated manual system, or offers great new features that the old software system doesn’t offer, which will make the users work much easier. UAT will eliminate any problems not detected by development testing or QA testing, rendering the perfect system to production. These advantages are not all that clear to the user community. The users have yet to see proof that your system will deliver the promised improvements and besides, they have been getting along nicely with the existing system. They may also believe that your job is to deliver the perfect system to them and you shouldn’t need their help to do so. After all, they don’t ask you for help to do their jobs.

The principal objection most users will have to participating in your testing is that they must accomplish the work assigned to them by their bosses plus do your UA testing. The best way of offering an effective inducement to overcome their objection is to change this dynamic so that the tester is not made to suffer as a result of participating in testing. The simplest way of accomplishing this is to take the testers out of their functional organizations and have other resources cover for them. You will definitely need the support of your business sponsor to accomplish this and should make plans for the transfer of the users to the project as part of your Quality Management Plan. This will add cost to the project but will ensure that you have a committed group of user/testers.

Committed full-time user/testers may be a luxury your project can’t afford, or the perceived benefits don’t outweigh the costs. You’ll need to offer the user/testers an inducement to make the commitment of their time a little more palatable. Offering the users the ability to port the results of their testing to the production environment is one possible inducement. Let me illustrate this approach with an example. Let’s say your users configure software orders. They use the system to choose the software features for their customers and check the orders against software and marketing rules. The result is a software package that can then be shipped to the customer. Porting the order they have configured to the production environment at the time of the cut over will reduce the amount of time the tester will "waste” on testing. This approach requires you to selectively merge data from the UAT environment with production data at the time of cut over. This is just one example of making the users life easier. Look for other means of accomplishing the same thing. This may require some innovation on your part but the payback in terms of user sign-up to UAT, will be worth the effort.

Training is always a desirable feature of any new system. Training should be a part of the work of the project; without adequate training the new system will be a hindrance rather than a help. To induce the user community to participate in testing make the training a part of the User Acceptance Testing program. The 2 activities are not all that different. The users should receive training before using the new system and testing can be made part of "hands on” training. Combining these two inducements can make participation in UAT especially attractive to users.

Test Environment

The test environment must be as similar to the production environment as possible. This is simple to accomplish when the application or system is installed on a user’s computer but a little more difficult in a client server or hosted environment. The environment should have the same software, hardware, and network configuration as the production environment. The environment should have all the common or shared data from the production environment. It may be possible to simply port this data from QA to UAT. The UAT environment should also have sufficient privately owned data to enable testing. This may be done by selecting one customer to test on, or one product, and to port this data from the production environment where the new system replaces an existing one, or to create the data manually where a manual system is being replaced. Another approach is to simply port all the data in the production software system into the UAT environment. This will require someone to groom the data so that it is compatible with the new system’s data dictionary (assuming data is handled by a database).

The hardware installed in the UAT environment should be as close to the production environment as possible to provide users with the same capacity and performance they will experience in production. Saving money here will guarantee hard discussions about system performance during UAT. You can attempt to save money here by having one environment for both QA and UAT which is a duplicate of the production environment in which case all QA testing must be finished before UAT can start.

Testers must be set up with user accounts which reflect their accounts in the production system, including their passwords and privileges. The testers will also need access to a bug reporting system. This won’t be a problem when the testers belong to your organization; it may require an administrator to add a few privileges to the testers account, or to create a new account for them. Accessing testers from an external organization may be more difficult, for starters they will almost certainly have to penetrate your organizations fire wall. The simplest way to solve this problem is to use a wiki to do bug reporting. This approach will allow anyone with access to the internet to be added to the project’s tester community.

Beta testing will require you to set up a bug reporting system that allows the user to report bugs from their own environments. This can be done by providing access to your bug reporting system via a web portal (providing you host a web site), or to use a publicly accessible bug reporting system such as BUGtrack. This approach can also work with an external customer’s organization.

Test Results

Your development testing and QA testing should have eliminated software bugs from the system by this time. The bugs that will most likely be reported during UAT can be grouped into 3 categories:

1. What I like to refer to as requirements interpretation issues. These bugs result from the developer/QA tester having one view of the function that meets a requirement and the UA tester having another.
2. "Cosmetic” bugs – the user dislikes the look and feel of a screen, or dislikes the layout of the screen.
3. Minor bugs such as the text in an error message, e.g. the system is developed in the USA and tested in England and the error message displays "behavior” rather than "behaviour”.