It seems that each new version of a product tends to be claimed by its manufacturer to perform better than the last, and be more scaleable and so forth. It is also true, however, that the system products around at the moment could not have run on the systems available even a few years ago.

The cycle of upgrades definitely serves manufacturers well, as the need to stay supported fuels software purchase, which fuels hardware purchase and so forth. On the back of this manufacturers also add new software features that increase the power and flexibility of the tool set. It is common, however, for upgrades to be performed purely to stay on a supported release – indicating that the customers often upgrade because they have to rather than because they see benefit in doing so.

Do we really need all this to do the job – or are we just buying in to the products the manufacturers want to sell us?

An interesting example of technical innovation has come to my attention recently. Data federation was recommended as a way forwards for an information management system. This is an innovative way to build a data warehouse without copying and moving data into a central store. It comes from the Service Oriented architecture stable and has been “main stream” for a while. It should have significant benefits if you believe it really works. One of these benefits should be that good performance is achieved by having a caching and management layer above the production databases. The fundamental question was whether it could perform well enough in the environment we needed it for. Was acceptable performance achievable using this innovative technology?

This is where technologies such as Netezza and its competition approach the problem a different angle. Their solution is to break a data warehouse into pieces spread across large numbers of processing elements. The individual elements then have processing capacity and storage space. They provide a fragment of the answer for each query, and pass their part of the answer back to a process to perform a merge. The fragments aren’t necessarily indexed because the data fragment it runs on can be small enough that it doesn’t provide a performance improvement. The time to process the query improves linearly with the number of processing elements – so pick the number of elements to achieve the required response time. There are many systems that would find this sort of solution very useful.

There are potential pitfalls here too though, especially where there are large numbers of parallel queries occurring, or queries are relatively complex. As usual there is “no silver bullet” and the solution needs to fit the needs of the problem. If you practical experience of these technologies then please share your experience of the approach with me at dh@sarquol.com, or call on +44 7887 536083.

Where there are no manufacturer guidelines, or as the next stage, you would usually to produce your own early tests. The same issues apply though, unless you are able to work with production-like data on a production-like platform. I am not saying not to use this approach – in fact I strongly recommend doing so – only that care needs to be taken in understanding where the test is realistic. As with other forms of planning, consider the risk carefully and plan in a level of contingency that is appropriate for capacity plans.

If you need help in interpreting manufacturers figures, or in setting up a benchmarking exercise then feel free to contact me at dh@sarquol.com, or call on +44 7887 536083.

• A very low load increasing the load will actually improve the performance of the system slightly. This is because most systems use various caching mechanisms and if there is too little load none of the caches will have data in.
• As the load increases a near flat region will be experienced, where all requests act as if they have near dedicated resources available to them.
• As the load increases further requests will tend to be contending for resources on an increasingly frequent basis. Thus, the resultant response time will show a near-linear increase with increasing load.
• If further load is added the system will start to demonstrate queuing behaviour, with most requests having to queue for each resource they require. The system response will now degrade far more rapidly with increasing load, and eventually the system will fail.

This behaviour is well known among people used to stress testing, but is less well known outside that circle. The important point to note is that if a system does not show this behaviour under load it is worth investigating why not. The chances are that there is either a system problem or a problem with the test process being used. If you have a system that does not show this profile and need some support in the analysis then please feel free to contact me at dh@sarquol.com, or call on +44 7887 536083.

There have been tools available to inject traffic into a system and monitor that data for a while. There have also been tools available to allow such monitoring on a one-off basis when a problem is known or suspected. This doesn’t, however, really go far enough in overall system monitoring since both approaches would usually be used retrospectively. It would seem, however, that the tool manufacturers have started to fill this gap in the performance monitoring market.

The Compuware “Vantage” toolset is a good example of this class of tool. This suite has included a strong range of monitoring, investigation and analysis tools so some time. They have now added a “passive monitoring” facility that starts to meet the requirement to be able to monitor the performance of all client transactions. If you would like to know more about this particular toolset then Compuware have published a fact sheet about it here.

The key to deciding is to consider the level of risk that performance problems will pose to the business. There seem to be relatively few organisations that use a blend of testing and future projection based on a risk-driven approach. It is this approach that is going to produce the most cost effective projects, since the degree of testing and proof provided will reflect the business risk that is inherent in the system. Spend sufficient to offset the cost of risk that a system carries if the performance is insufficient in the long term. Remember that it is likely to be much more expensive to solve a performance problem a number of years after initial delivery than when the project is in full flow. However important the initial delivery may be, a project is not complete until it has been proven to be capable of being operational for the long term.

Suggestion:

Use a simple set of standard criteria for a project representing “average” performance risk for the business:

1. A soak test to prove that the system will run for an extended period, say 12 hours, at average loading without performance degradation.
2. A load test to demonstrate that the system can process peak demand volumes for a predefined period, such as an hour, without degrading.
3. A volume test, to prove that the system can operate with production volumes at a defined point in the future – say 2 years hence.
4. Within any performance tests use a small set of key business functions to test that the system performs adequately.

If the risk related to poor system performance, or failure, is especially high then these performance criteria would be made more stringent. The degree of performance proof required, however, should be kept proportional to the level of risk in the system failing to perform well. If the system was business critical, for example, it may be considered appropriate to require that the system be run using a much wider set of business functions, for 72 hours continuously, and at 5 times the expected peak volumes for the system for the next 20 years. Unsurprisingly, as the proof requirements become more stringent the cost and time needed to prove them will increase significantly.