Table of contents for Project Estimation Model

1. Project Estimation Model
2. Project Estimation Model Example
3. Project Estimation using Scenarios

Defining estimation scenarios

The first stage in the steps to estimating a project was defined to be the definition of the scenarios to be supported in the solution.  The scenarios should be defined at as low a level of detail as possible, with the following diagram showing the example scenario definition.

Scenario definiton

Definition of the scenarios sizing

In the spreadsheet the light blue fields are those that you would normally fill in. The light purple fields are intended to fill in automatically based on the calibration parameters. If you prefer, however, it is suggested that you replace the calculated value with your own value where a better value is available. In this case I would suggest changing the colour of the cell so that you know this has been done later. I use bright highlighter yellow for this.The purple values are filled in from the “Calibration” sheet based on the complexity selected in the drop-down within those cells. The values represent typical operation counts for scenarios of the defined complexity.Scenario sizing calibrationThe scenario sizing information is drawn from the following part of the calibration sheet:Scenario calibration

As for all the calibration, the provided values are likely to be useful but the optimal values will depend on the business context of the estimator. It is suggested that the calibration should be adjusted from project to project until the most appropriate values are found. For a more rigorous approach a function-point counting style could be adopted on a specification of some suitable scenarios. The idea is to count the number of entities (c.f. database tables) that need to be managed by each scenario.

In the scenario definition sheet these filled in values are then summed at the bottom of the page, and each operation type is multiplied by a weighting factor to produce a total size. This can be seen at the bottom of the scenario definition sheet in the example:

Scenario sizing

The scenario sizing calculation

As can be seen, a standard “Operation Weight” is applied to each of the operation types within the columns of the estimation spreadsheet.  These are drawn from this section within the calibration sheet:

Scenario operation weight

Operation weighting calibration

This process is similar in principle to function point analysis, but the constants have not been adjusted to provide a function point count. It would be possible to complete this and hence to provide an estimate very close to the true function point count of the solution. This, however, is left as an exercise for the reader since I don’t have access to a library of function point analysis data which would make it worthwhile.

Table of contents for Project Estimation Model

1. Project Estimation Model
2. Project Estimation Model Example
3. Project Estimation using Scenarios

As a means for discussing the model I will follow through an example of the model being used. To do this the needs to be an example usage scenario. In this case I have chosen to use the following:

“A banking web site has retail and corporate clients, and well as a set of automated processes that must be completed overnight in the bank’s overnight batch window. The site has a standard Java web architecture, with the batch processes being initiated using a batch process at the application server. The purpose behind the model is to examine the capacity required in the major system components, and to make ensure the ongoing capability of the host systems for the site.”

If you have followed through the performance model series (here), you will note that this is exactly the same scenario. I have chosen this in order to be able to compare and contrast the application of the performance and project estimation models. There are clear differences in approach that need to be considered. For those that like to skip to the end I have provided a copy of the final estimation example filled out here:

Example project estimate

Table of contents for Performance Model Example

1. Performance Model Example
2. Example: Function definition
3. Example: Defining users and locations
4. Example: Usage Definition
5. Example: User demand estimation

Having defined the key system functions and the user population it is now necessary to define how the users place a functional demand on the system. (The online version of this article has more detail again.) The first part of this definition is to lay out how the usage of the system varies over time. Thus the relative usage of the system needs to be defined. The initial definitions are on an intra-hour, hourly, daily, weekly and monthly basis. These figures effectively have no units and simply provide a relative level of usage. Thus any metrics that are available to allow calibration of this data can be used. As an example, estimated percentage utilisation of the system for the time period could be provided. Alternatively, there may be historical data available from a production system.

In the majority of the usage profile sheets the data entry allows a default profile to be entered at the top of the sheet. This can then be varied for individual user sections where they vary significantly. Thus, for example, most of the user sections have similar behaviour this would be used for the default profile. The significant variations from this behaviour would then be entered individually for the separate user section rows that this was required for.

There is a further possibility of varying the use of different functions within a time period, in our example there is likely to be a bias in the number of payments created and approved around the cut off times for making those payments.

The facility extends to a monthly and weekly specification:

The “Intrahour” entry is different. In this case a simple ratio is provided between the peak activity within an hour and the average activity for the hour. This value represents the relative activity instability of the system under consideration, with a higher value representing a system with very unstable usage within any given hour.

The “Annual” data entry is different again. In this case a reference year for the modelling is defined, and the modelling extends for ten years including this starting year. The number of working days per year is also entered in this sheet, and then the size of the user population for each user segment in each year. Thus, the expected population growth for each of the user segments may be defined over a ten year period.

Note:In this exmple it is worth noting that the “User population” count for the email and standing order systems has been set to the expected number of items per day. This is a useful modelling mechanism in the case of batch oriented processes.

Lastly in the definition of the system demand by the system’s behaviour types (“Behaviour type function usage”). In this case the average use of each function by each user per day broken down by behaviour type. Using all of this information it is possible to calculate a detailed profile of the expected system demand. This information and its usage will be examined next.

Table of contents for Performance Model Example

1. Performance Model Example
2. Example: Function definition
3. Example: Defining users and locations
4. Example: Usage Definition
5. Example: User demand estimation

Location definition

Specification of the locations within the model

Behaviour type definition

The next set of information to define is the behaviour types to be used in the model. In this case the main behaviour types being modelled are: Retail Banking, Corporate Users and Corporate Managers. To this are added behaviour types for the email processor and standing order processor, which will be used to model the batch processing within the system.

Specification of the behaviour type within the spreadsheet

 User Section Definition

Now the sets of users themselves can be defined to link locations and behaviour types. Thus, there would be corporate banking users and managers in the UK, New York and Europe. For this example, Retail baking users are only to be found in the UK, whereas the batch processors are located in the primary data centre. This is defined within the “User Section” sheet, which literally defines the different sections of users by their behaviour type and location giving each a representative name.

Specification of the user structure within the model

Where there are no manufacturer guidelines, or as the next stage, you would usually to produce your own early tests. The same issues apply though, unless you are able to work with production-like data on a production-like platform. I am not saying not to use this approach – in fact I strongly recommend doing so – only that care needs to be taken in understanding where the test is realistic. As with other forms of planning, consider the risk carefully and plan in a level of contingency that is appropriate for capacity plans.

If you need help in interpreting manufacturers figures, or in setting up a benchmarking exercise then feel free to contact me at dh@sarquol.com, or call on +44 7887 536083.

Table of contents for Performance Model Example

1. Performance Model Example
2. Example: Function definition
3. Example: Defining users and locations
4. Example: Usage Definition
5. Example: User demand estimation

Once the function, behaviour type and usage pattern data have been provided the model starts to produce useful information based on its estimations. The most important of this is the daily function usage estimation. This information is presented on the “Estimated Function Demand” sheet within the model, and provides as estimate, by year, of the average daily usage for each function. This is then used to provide an estimate of the average and peak operations per second for the system.

Calculated daily function use

Calculated function usage per second

The first use of this information is often to check the model against other data that has been provided. In this example the bank is likely have provided an idea of their expectation of the daily levels of system usage for key functions. It is unlikely that their values will exactly match those estimated by the model. A serious difference should be considered carefully, however, as it may represent a serious issue in underlying assumptions about system usage.

The next set of usage data that may be of value is on the “Annual Function Demand” sheet. This provides the average daily usage data multiplied up to provide total annual usage, and cumulative annual usage of the different functions. Again, this information is often provided for comparison purposes. If the information hasn’t been previously provided the overall function usage estimates can provide a strong indication of the overall scale of the system being analysed.

This information can be used to support an understanding of the likely levels of technical and business risk in the system before the detailed architecture has been elaborated. It may also be used in the evaluation of the technology needed to produce the system under consideration. The next stage of the modelling, however, will start to provide the basis for architectural part of the estimation process.

Table of contents for Performance Model Example

1. Performance Model Example
2. Example: Function definition
3. Example: Defining users and locations
4. Example: Usage Definition
5. Example: User demand estimation

I will skip the title sheet, which is merely descriptive, and start to work with the function definitions. In this case I highly recommend starting with a few key functions that are likely to provide the majority of the key performance behaviour. If this model turns out to be insufficient at a later stage then add more information as the model develops. This means that the level of detail being worked on remains manageable.

In this case I am going to use the following key functions to define the system:

1. Login
2. Logout
3. Balance Enquiry
4. Create Payment
5. Create Standing Order (SO)
6. Approve payment
7. Approve SO
8. Cancel SO
9. Send SO
10. Send status e-mail

This set has been chosen as much to illustrate the modelling process as for their importance in a real system. There are two other major sets of information that need to be entered into the page.

The first is a “Key Function”, which is used where a particular function needs to be reported on, rather than all functions. This is used in places to simplify analysis and reporting within the model. In this case the “Create Payment” function is used.

The other information needed is the “daily usage window” for each function. This information is used during the capacity analysis, and for most functions in an internet banking system is likely to be 24 hours. For the example, however, I have limited the sending of standing orders by the system to a 3 hour batch window, with the sending of status emails to a larger 6 hour window.

Later in the modelling more detailed information will be provided.

The images shows this information entered into the spreadsheet.

Table of contents for Performance Model Example

1. Performance Model Example
2. Example: Function definition
3. Example: Defining users and locations
4. Example: Usage Definition
5. Example: User demand estimation

I have received feedback that the performance model would be more useful if it was better documented. The is a tour of the model available (here), but that doesn’t make it clear how to apply the model ina typical usage scenario. I have, therefore, decided to illustrate the use of the model using a fictional example.

Modelling Scenario

First, I’ll outline the scenario for the modelling:

“A banking web site has retail and corporate clients, and well as a set of automated processes that must be completed overnight in the bank’s overnight batch window. The site has a standard Java web architecture, with the batch processes being initiated using a batch process at the application server. The purpose behind the model is to examine the capacity required in the major system components, and to make ensure the ongoing capability of the host systems for the site.”

 Note: Before starting to work with the spreadsheet turn off automatic recalculation of the cell values. The calculations in the model take a while to run, and so it is better to be in control of them.

http://jmt.sourceforge.net/

These tools provide a strong set of analytical and simulation capabilities that are designed for computer system modelling. They provide an integrated set of tools with a graphical user interface, meaning that the learning curve necessary to use the tools and understand the models will not be too steep. If you need to apply queuing theory models to a piece of performance analysis these tools are well worth looking at.