Testing Best Practice

Overview
This white paper looks at defining some guidelines for best practice when planning for the testing effort for any development project or when recruiting new testers. Guidelines that are covered here are:

1.1 The role of testing in an organisation or project
1.2 Planning the complete testing effort
1.3 Choosing the right people and resources for testing
1.4 Planning the test activities



1.1 The role of testing in an organization or project
1.1.1 The Test Team is not responsible for all Quality Assurance
Quality should be a project or organisational activity and driven by management not just the Test Team. Everyone taking part in the development process is responsible for the quality of their work. Systems that are developed from quality requirements, with quality development standards means that Testing is there to validate this quality rather than ensure the quality.

1.1.2 The purpose of testing is to find software bugs
While this statement is correct to an extent however the purpose of testing is not just to find bugs bet to ensure that the important bugs are found. It is often too easy to spend time testing less important functionality and log superficial bugs while critical bugs are being missed because they are not being tested.

1.1.3 Usability issues are important and should be raised
Testers are often the only people in an organization who will use the system as heavily as an expert. Formal usability testing should be recognised as part of the testing effort and if valid be addressed by the development team. Choosing to ignore usability issues may affect client satisfaction and lead to lost revenue.

1.1.4 Ensure that all bug reporting and metrics is put into context for management
When presenting bug reports all limitations to the data should be explained to eliminate the risk of management taking a “too optimistic” view towards the results.

1.1.5 Don’t leave testing to the end
An effective testing process involves extensive planning for the testing effort. Including testing earlier on in the development life cycle will promote a preventative approach to development with the test team providing a supporting role in the early stages with their input increasing as the project progresses. The test team will become familiar with the product and any issues identified can be resolved earlier rather than later.

Tests designed before the development team start coding can help to improve quality. The test team can inform the developer of the kinds of tests that they will run which in turn may help the developer during the design phases and in unit testing.


1.2 Planning the complete testing effort
1.2.1 Do not bias your testing effort to functional testing
This could be a danger as functional testing usually tests features in isolation. While this will ensure that the requirements for the project are met it means that critical bugs may be missed. Testers who are planning the testing effort need to be a little more insightful to identify test cases and scenarios that cover off critical paths that while they test a certain feature also ensures that dependant or affected functionality is also tested.

1.2.2 Do not underestimate the importance of configuration and integration testing
Often this type of testing is either forgotten or given less emphasis in the development process. These tests ensure that the developed product works on different hardware configurations and with different third party software.

While these tests are often forgotten because it is expensive to maintain test environments with the necessary hardware and software the use of virtual machines etc. will mean that most companies can cover a baseline for this type of testing.

1.2.3 Remember to factor in time for performance and stress testing
Often this type of testing is left to the end because it means that major development has stopped, most of the testing is complete and the system is the closest to it has ever been to being a potential release candidate. While this is the best time to start intensive performance or stress testing remember to factor in enough time for refactoring, optimisation and re-testing of the code in the instance that your product will just not scale to the level required.

Developers and testers should keep in mind any performance requirements while they are designing their code, planning their tests and performing unit testing or functional testing. Obvious performance issues are quite easy to detect in the early stages of development and testing.

1.2.4 Remember to test the installation procedures
To avoid embarrassing mistakes where the product won’t install on first go make sure that each release provides detailed Release Notes and Installation Procedures that have been verified by both the development and test teams.

1.2.5 Remember to test the supporting functionality such as online help or documentation
Testing the documentation means checking that all the procedures and examples in the documentation work. Remember to factor in enough time so that any modifications can be made. As often is the case the product may have changes from the original specifications so procedures and screenshots will need to be updated.

1.2.6 Ensure you have sufficient testing coverage
Plan to get coverage for all critical functionality across all areas of the system early in the piece. This may mean that you do not get to test all functionality on first pass but you should cover all the critical tests.

It is better to know the status of the system across all areas rather then falling into the scenario where you don’t start testing a new area until you finished the last. You will get a general feel for the stability of the system rather that having some areas that are tested to a high level while others are tested less thoroughly if shortage of time becomes a factor. You can also identify areas where there are high numbers of defects and work with the developers to mitigate this risk.

1.2.7 Correctly identify high risk areas
As part of the test planning all key stakeholders should work with the development and test teams to identify high risk areas in the product or system. These areas are typically high risk because they high visibility to system users or if failure could lead to either loss of property or in some cases even life. Some businesses have extremely complex business rules that need detailed tests.

Another good way to identify high risk areas is to review historical system data. Old bug reports and metrics may give you insight into the areas that require thorough testing.

1.2.8 Realise that testing requirements will change and be prepared to accommodate for this
It is inherent with development projects that requirements and hence testing requirements will change. Therefore no matter how well you plan your testing efforts make sure that you keep up to date with development changes so that you can make changes to your test plans and cases as necessary.


1.3 Choosing the right people for testing
1.3.1 Don’t use testing as a transitional job for new programmers
If you have hired someone who wants to be a developer don’t put them into a testing role. While some may argue that spending a couple of months as a tester will help the new person to understand the system and allow you to analyse how they grasp concepts before starting on the code this is not conducive to good quality. If a person has been hired as a programmer then that should be there primary focus.

1.3.2 Be careful of recruiting failed programmers as testers
While there are many good testers who are not good programmers and there could be equally as many good programmers who are good testers. Some programmers genuinely need a change programming and make a real difference to the testing effort.

However there are situations where the reasons that made a person a bad programmer will also make them a bad tester. For example, a programmer who had poor quality code with a high bug count because they had low attention to detail will more than likely be a bad tester as they will miss bugs for the same reason.

1.3.3 Look for the following qualities when recruiting your testers
When interviewing concentrate on the candidates intelligence and thought process. A good tester should have the following qualities:

> Methodical and systematic
> Tactful and diplomatic but firm when necessary
> Good troubleshooting skills. Able to notice and pursue oddities in the system.
> Skeptical about any assumptions and be able to validate whether they are true or not
> Good written and verbal skills in order to explain bugs clearly and concisely
> A knack for anticipating what others are likely to misunderstand this is useful for UAT and usability testing. If a tester has problems understanding then how will the client feel?
> An ability to think outside of the box to experiment and consider all possible scenarios.

1.3.4 Try and recruit testers who are domain experts in the area you need
While this is hard, especially in a consultancy firm where the domain expertise is varied try and identify the areas where Change is most likely to focus its work effort. Testers who do not have domain expertise find it hard to distinguish between important and irrelevant or superficial bugs.


1.4 Planning the testing activities
1.4.1 Plan tests before executing them
Historically poor test design like poor code design leads to poor quality. Planning testing activities allows the tester to analyse the scenarios and any special cases in advance. While jumping straight into testing activities may mean that these special scenarios are never identified and hence never tested.

1.4.2 Encourage peer review of test documentation
It is good practice to encourage reviews where a quick check of the testing approach and the defined test cases to ensure that sufficient coverage has been met and no critical tests have been missed.

1.4.3 Set guidelines and implement tools for good issue tracking and issue reporting
An integral part of testing is how to report and track the issues that are a result of the testing process. There are a number of free test bug tracking tools that can help you to manage this activity. Similarly there are many off the shelf products that offer the same functionality and then some.

What needs to be kept in mind is that while tools can help they need to be implemented and used properly. Guidelines need to be set so that testers and anyone reporting issues provide specific data that will help developers to investigate and if necessary fix the issue. Workflow should also be set so that issues don’t get lost in the system and are routed to the correct individuals as needed. An administrator may need to be set to oversee the whole issue tracking process.