When creating test scripts via “record and playback”, however, the tool will usually record page delays and play them back during the script run – and so the problem is solved. Isn’t it?

There are two issues that must be considered here. The first follows on naturally from the comment above: If the recorded wait time is not realistic then the resultant loading will not be realistic, and so the test will be better but still not correct. This could be fixed by simply using a user who is likely to have realistic waits – assuming you can identify such a user and make them available for the testing. I can be surprising just how long real users do spend on think time.

The next issue is more important, and harder to understand. For illustration let us assume we have 10 users with a 5s think time. Let us further assume that requests usually take 0.1s to process, and that simulated users are started 0.5s apart. In normal operation these requests will never coincide. This could be an issue in itself, but this isn’t the worst of it.

Let us now presume there is a problem – a temporary server issue would be sufficient – that causes all of the responses to arrive back at the client at the same time. It does happen in load testing for various reasons. Now all of the scripts will wait 5s and send the next requests at exactly the same time. The message loading on the server suddenly goes up dramatically, and is likely to continue in this pattern for some time. The result is that the system doesn’t seem to be able to cope with even 10 users in realistic conditions.

The solution is to introduce a level of randomisation in think time delays. It is then possible to introduce an appropriate statistical think time profile – and to break up bursts. The cost is extra test script development time, but the benefit is much more realistic testing.

If you would like advice, or to discuss how to decide on the most appropriate way to set up load testing in your circumstances then please feel free to contact me at dh@sarquol.com, or call on +44 7887 536083.

It provides a record-playback interface, and the generated scripts can then be customised using a simple basic-like language. It can also gather performance monitoring data from the load testing environment. For Windows machines the capture data can come directly from any of the performance monitor counters available on the system. For other operating systems, can be monitored using SNMP-based counters. This is slightly more complicated, but is feasible without too much difficulty.

It can also be used to simulate high loads, with a network of load generation machines being able to be controlled from a single master machine. Depending on the specification of the systems and the script involved each load machine can generate over 1,000 simulated users. Thus even very high user loading levels can be simulated using the tool without incurring excessive costs. To put this in context, the cost of using commercial software to simulate load usually increases with the number of connections being simulated.

Lastly on the feature set, the tool has a set of relatively simple analysis reports that are built into it. These allow some of the most common performance analysis questions to be answered almost immediately. If these built in reports aren’t sufficient, however, the raw data can be exported to Excel and analysed there. This allows analysis such as a percentile-based examination of response times.

Like any software tool it does have its difficulties and limitations. One of these is that it is a Windows tool, and requires Excel to be installed to get the best out of it. The user interface and scripting language also have limitations and frustrations. Probably the most important limitation is that it doesn’t support the range of protocols that the more expensive tools do. Thus, if you need support for testing a range of protocols you will need to look elsewhere. If what you need is a web testing tool without excessive costs, however, I can recommend starting here and choosing a full commercial tool only if you find you really need it.

If you need help in choosing or using suitable tools for performance management work, whether open source or commercial tools, then e-mail me. The choice of tool, and the way it is used, will make a big difference to the results and time required for a performance management initiative.