Project Assurance is, as the name suggests, assurance of the way that the project is being run. Relevant experience is around project management rather than the implementation solution space. The concept od Solution Assurance is more related to confirming that solution being delivered will be fit for purpose once in place. In PRINCE2 terms the project assurance role will be most closely aligned to the project executive, whereas the the Solution Assurance will be aligned with the senior suplier. (Arguably the senior user could also be looking to take on solution assurance, but in PRINCE terms it would be the senior supplier.)

The idea of a turnaround specialist would be slightly different again. To require “turnaround” there must be an identified issue to address. The assurance role is required whether or not there is a problem, and its presence would hopefully avoid problems. The turnaround specialists task is more one of a troubleshooter – identifying the symptoms, analysing the cause and leading the solution. This might be in Project or Solution terms.

Let me provide some examples:

• In my last role I was employed as a Solutions Architect, and my primary focus was on assuring that the solution to be delivered met its functional and non-functional requirements. The skills I employed for this were primarily technical, and example questions were around whether the particular solution would fully deliver the functional and non-functional requirements.
• At the same time I also performed a level of project assurance for the programme manager for projects I was less directly involved in. In this case the experience I was drawing on was one of a project manager with years of delivering solutions under my belt. The examples of questions I was asking here were around the potential issues to do with processes being used for risk and issue management and whether they were appropriate.
• In a turnaround role I would employ analysis based on *both* of the other assurance roles, but would go further to address an identified issue. I would use my own creative skills to provide potential approaches, and would use an engagement approach that would get buy-in and participation from the delivery team. I would then propose an exception plan, in PRINCE terms, in conjunction with the project or programme manager. The skills being used are more active and leadership based.

In the description above I have assumed a project/programme context for turenaround. The approach, however, is as likely to be applied in an operational setting, and in this case the set of techniques will differ somewhat. The result of business level turnaround application is likely to be one or more change programmes.

Table of contents for Performance Model Example

1. Performance Model Example
2. Example: Function definition
3. Example: Defining users and locations
4. Example: Usage Definition
5. Example: User demand estimation

Location definition

Specification of the locations within the model

Behaviour type definition

The next set of information to define is the behaviour types to be used in the model. In this case the main behaviour types being modelled are: Retail Banking, Corporate Users and Corporate Managers. To this are added behaviour types for the email processor and standing order processor, which will be used to model the batch processing within the system.

Specification of the behaviour type within the spreadsheet

 User Section Definition

Now the sets of users themselves can be defined to link locations and behaviour types. Thus, there would be corporate banking users and managers in the UK, New York and Europe. For this example, Retail baking users are only to be found in the UK, whereas the batch processors are located in the primary data centre. This is defined within the “User Section” sheet, which literally defines the different sections of users by their behaviour type and location giving each a representative name.

Specification of the user structure within the model

When creating test scripts via “record and playback”, however, the tool will usually record page delays and play them back during the script run – and so the problem is solved. Isn’t it?

There are two issues that must be considered here. The first follows on naturally from the comment above: If the recorded wait time is not realistic then the resultant loading will not be realistic, and so the test will be better but still not correct. This could be fixed by simply using a user who is likely to have realistic waits – assuming you can identify such a user and make them available for the testing. I can be surprising just how long real users do spend on think time.

The next issue is more important, and harder to understand. For illustration let us assume we have 10 users with a 5s think time. Let us further assume that requests usually take 0.1s to process, and that simulated users are started 0.5s apart. In normal operation these requests will never coincide. This could be an issue in itself, but this isn’t the worst of it.

Let us now presume there is a problem – a temporary server issue would be sufficient – that causes all of the responses to arrive back at the client at the same time. It does happen in load testing for various reasons. Now all of the scripts will wait 5s and send the next requests at exactly the same time. The message loading on the server suddenly goes up dramatically, and is likely to continue in this pattern for some time. The result is that the system doesn’t seem to be able to cope with even 10 users in realistic conditions.

The solution is to introduce a level of randomisation in think time delays. It is then possible to introduce an appropriate statistical think time profile – and to break up bursts. The cost is extra test script development time, but the benefit is much more realistic testing.

If you would like advice, or to discuss how to decide on the most appropriate way to set up load testing in your circumstances then please feel free to contact me at dh@sarquol.com, or call on +44 7887 536083.

There have been tools available to inject traffic into a system and monitor that data for a while. There have also been tools available to allow such monitoring on a one-off basis when a problem is known or suspected. This doesn’t, however, really go far enough in overall system monitoring since both approaches would usually be used retrospectively. It would seem, however, that the tool manufacturers have started to fill this gap in the performance monitoring market.

The Compuware “Vantage” toolset is a good example of this class of tool. This suite has included a strong range of monitoring, investigation and analysis tools so some time. They have now added a “passive monitoring” facility that starts to meet the requirement to be able to monitor the performance of all client transactions. If you would like to know more about this particular toolset then Compuware have published a fact sheet about it here.

When a system has been implemented it is likely to either require upgrade or to experience performance problems at some point, and then the basis for modelling can be much stronger. There is a system in place with real users performing real transactions, and so detailed information should be available. This is especially important when investigating performance problems. This is because how often certain operations occur, and what else was happening on the system at the same time, can be critical in understanding the likely root cause.

It is possibly surprising how rare it is to be able to gather meaningful data from a system on how it is being used, and the experience that users are having from the system. Without this being considered early in a system’s design, or in the evaluation of a product, it is often the case that information on the system’s usage in production is either completely unavailable or only available indirectly using analysis techniques that are difficult to produce. There are a significant number of ways to make this information available, but it has to be implemented and tested in the original system design. This data gathering is very difficult to retro-fit to a system, but generally easy to add during development.

So what should the requirements for a system be?

The following is a starting list:

1. Information should be recorded about all operations that users perform, and the experience they obtain from those operations. This should not only be limited to where an error is experienced.
2. The system should have standard reports from this data that provides key metrics on a regular basis. These metrics should be directly used to monitor the service level, and used to detect if there are problems in the system as early as possible.
3. Such a data gathering system can generate large volumes of data, and so a data archiving and management mechanism must be in place. Without this it is likely that the growth of monitoring data will itself cause performance problems in the future.
4. The solution for monitoring and reporting on user operations should not negatively impact the user experience.
5. It should be possible to recover retrospectively what was happening at a certain point in time on the system, down to user IDs and arguments used in operations.

It is worth noting that having this sort of information will be valuable throughout the delivery of a system anyway, in particular during load testing and performance model calibration.

The next question is how does such information get collected? There are many different way, and the most effective way varies with the technology being used. Where possible the information should be gathered as a side effect of something else. In web systems for example it is sometimes possible to gather this sort of data from server log files. In systems where the code is generated it may be possible to add data gathering into the templates. If all else fails then place timers and system logging throughout the code and make sure you have the necessary log processing tools in available.

If you need help in working out an effective approach to collecting data, or in framing the non-functional requirements necessary to manage the performance of a system then e-mail me. It can be surprising how often having access to this information can help to resolve issues – and not just those in the scope of system performance problems.