When creating test scripts via “record and playback”, however, the tool will usually record page delays and play them back during the script run – and so the problem is solved. Isn’t it?

There are two issues that must be considered here. The first follows on naturally from the comment above: If the recorded wait time is not realistic then the resultant loading will not be realistic, and so the test will be better but still not correct. This could be fixed by simply using a user who is likely to have realistic waits – assuming you can identify such a user and make them available for the testing. I can be surprising just how long real users do spend on think time.

The next issue is more important, and harder to understand. For illustration let us assume we have 10 users with a 5s think time. Let us further assume that requests usually take 0.1s to process, and that simulated users are started 0.5s apart. In normal operation these requests will never coincide. This could be an issue in itself, but this isn’t the worst of it.

Let us now presume there is a problem – a temporary server issue would be sufficient – that causes all of the responses to arrive back at the client at the same time. It does happen in load testing for various reasons. Now all of the scripts will wait 5s and send the next requests at exactly the same time. The message loading on the server suddenly goes up dramatically, and is likely to continue in this pattern for some time. The result is that the system doesn’t seem to be able to cope with even 10 users in realistic conditions.

The solution is to introduce a level of randomisation in think time delays. It is then possible to introduce an appropriate statistical think time profile – and to break up bursts. The cost is extra test script development time, but the benefit is much more realistic testing.

If you would like advice, or to discuss how to decide on the most appropriate way to set up load testing in your circumstances then please feel free to contact me at dh@sarquol.com, or call on +44 7887 536083.

Where there are no manufacturer guidelines, or as the next stage, you would usually to produce your own early tests. The same issues apply though, unless you are able to work with production-like data on a production-like platform. I am not saying not to use this approach – in fact I strongly recommend doing so – only that care needs to be taken in understanding where the test is realistic. As with other forms of planning, consider the risk carefully and plan in a level of contingency that is appropriate for capacity plans.

If you need help in interpreting manufacturers figures, or in setting up a benchmarking exercise then feel free to contact me at dh@sarquol.com, or call on +44 7887 536083.

• A very low load increasing the load will actually improve the performance of the system slightly. This is because most systems use various caching mechanisms and if there is too little load none of the caches will have data in.
• As the load increases a near flat region will be experienced, where all requests act as if they have near dedicated resources available to them.
• As the load increases further requests will tend to be contending for resources on an increasingly frequent basis. Thus, the resultant response time will show a near-linear increase with increasing load.
• If further load is added the system will start to demonstrate queuing behaviour, with most requests having to queue for each resource they require. The system response will now degrade far more rapidly with increasing load, and eventually the system will fail.

This behaviour is well known among people used to stress testing, but is less well known outside that circle. The important point to note is that if a system does not show this behaviour under load it is worth investigating why not. The chances are that there is either a system problem or a problem with the test process being used. If you have a system that does not show this profile and need some support in the analysis then please feel free to contact me at dh@sarquol.com, or call on +44 7887 536083.

The key to deciding is to consider the level of risk that performance problems will pose to the business. There seem to be relatively few organisations that use a blend of testing and future projection based on a risk-driven approach. It is this approach that is going to produce the most cost effective projects, since the degree of testing and proof provided will reflect the business risk that is inherent in the system. Spend sufficient to offset the cost of risk that a system carries if the performance is insufficient in the long term. Remember that it is likely to be much more expensive to solve a performance problem a number of years after initial delivery than when the project is in full flow. However important the initial delivery may be, a project is not complete until it has been proven to be capable of being operational for the long term.

Suggestion:

Use a simple set of standard criteria for a project representing “average” performance risk for the business:

1. A soak test to prove that the system will run for an extended period, say 12 hours, at average loading without performance degradation.
2. A load test to demonstrate that the system can process peak demand volumes for a predefined period, such as an hour, without degrading.
3. A volume test, to prove that the system can operate with production volumes at a defined point in the future – say 2 years hence.
4. Within any performance tests use a small set of key business functions to test that the system performs adequately.

If the risk related to poor system performance, or failure, is especially high then these performance criteria would be made more stringent. The degree of performance proof required, however, should be kept proportional to the level of risk in the system failing to perform well. If the system was business critical, for example, it may be considered appropriate to require that the system be run using a much wider set of business functions, for 72 hours continuously, and at 5 times the expected peak volumes for the system for the next 20 years. Unsurprisingly, as the proof requirements become more stringent the cost and time needed to prove them will increase significantly.