There have been tools available to inject traffic into a system and monitor that data for a while. There have also been tools available to allow such monitoring on a one-off basis when a problem is known or suspected. This doesn’t, however, really go far enough in overall system monitoring since both approaches would usually be used retrospectively. It would seem, however, that the tool manufacturers have started to fill this gap in the performance monitoring market.

The Compuware “Vantage” toolset is a good example of this class of tool. This suite has included a strong range of monitoring, investigation and analysis tools so some time. They have now added a “passive monitoring” facility that starts to meet the requirement to be able to monitor the performance of all client transactions. If you would like to know more about this particular toolset then Compuware have published a fact sheet about it here.

The most valuable way to validate a performance model is to monitor a production environment and then to check that all of the predictions of the model are reflected by the production system within a reasonable tolerance. This allows all of the assumptions and mathematics to be checked in a single stage. The practice should be a matter of routine after the implementation of systems, and acts as an important operational performance assurance activity.

In the time before the initial implementation, however, it is only practical to check the technical aspects of the model using a set of performance tests. Thus, it is possible to test using a defined level of user load, and to make sure that the predictions of the model match the performance profile experienced to within an acceptable tolerance. If not, the reason for the differential needs to be investigated and resolved. It will usually be necessary to run a series of tests until the model’s accuracy is sufficient for the model’s use.

It is important to stress, however, that this form of validation still leaves a significant level of uncertainty about the level of demand to be placed on the system. It is worthwhile, therefore, using the model to predict the likely performance under a range of demand-profile assumptions. At least the impact of the level of unknown for the demand can then be investigated.

When a system has been implemented it is likely to either require upgrade or to experience performance problems at some point, and then the basis for modelling can be much stronger. There is a system in place with real users performing real transactions, and so detailed information should be available. This is especially important when investigating performance problems. This is because how often certain operations occur, and what else was happening on the system at the same time, can be critical in understanding the likely root cause.

It is possibly surprising how rare it is to be able to gather meaningful data from a system on how it is being used, and the experience that users are having from the system. Without this being considered early in a system’s design, or in the evaluation of a product, it is often the case that information on the system’s usage in production is either completely unavailable or only available indirectly using analysis techniques that are difficult to produce. There are a significant number of ways to make this information available, but it has to be implemented and tested in the original system design. This data gathering is very difficult to retro-fit to a system, but generally easy to add during development.

So what should the requirements for a system be?

The following is a starting list:

1. Information should be recorded about all operations that users perform, and the experience they obtain from those operations. This should not only be limited to where an error is experienced.
2. The system should have standard reports from this data that provides key metrics on a regular basis. These metrics should be directly used to monitor the service level, and used to detect if there are problems in the system as early as possible.
3. Such a data gathering system can generate large volumes of data, and so a data archiving and management mechanism must be in place. Without this it is likely that the growth of monitoring data will itself cause performance problems in the future.
4. The solution for monitoring and reporting on user operations should not negatively impact the user experience.
5. It should be possible to recover retrospectively what was happening at a certain point in time on the system, down to user IDs and arguments used in operations.

It is worth noting that having this sort of information will be valuable throughout the delivery of a system anyway, in particular during load testing and performance model calibration.

The next question is how does such information get collected? There are many different way, and the most effective way varies with the technology being used. Where possible the information should be gathered as a side effect of something else. In web systems for example it is sometimes possible to gather this sort of data from server log files. In systems where the code is generated it may be possible to add data gathering into the templates. If all else fails then place timers and system logging throughout the code and make sure you have the necessary log processing tools in available.

If you need help in working out an effective approach to collecting data, or in framing the non-functional requirements necessary to manage the performance of a system then e-mail me. It can be surprising how often having access to this information can help to resolve issues – and not just those in the scope of system performance problems.