This is where technologies such as Netezza and its competition approach the problem a different angle. Their solution is to break a data warehouse into pieces spread across large numbers of processing elements. The individual elements then have processing capacity and storage space. They provide a fragment of the answer for each query, and pass their part of the answer back to a process to perform a merge. The fragments aren’t necessarily indexed because the data fragment it runs on can be small enough that it doesn’t provide a performance improvement. The time to process the query improves linearly with the number of processing elements – so pick the number of elements to achieve the required response time. There are many systems that would find this sort of solution very useful.

There are potential pitfalls here too though, especially where there are large numbers of parallel queries occurring, or queries are relatively complex. As usual there is “no silver bullet” and the solution needs to fit the needs of the problem. If you practical experience of these technologies then please share your experience of the approach with me at dh@sarquol.com, or call on +44 7887 536083.

The quick answer is that “yes it can be true”. A little more analysis would be needed to be absolutely sure it is true for a particular implementation. This can be validated by considering what happens for a disk read: The OS will request a data block from the disk. The block may be cached, but if it isn’t then a disk read is necessary. It is not uncommon for such a read to take of the order of 10ms to complete, and the request is likely to be queued behind other requests. This is then returned to the processor.

This process is basically the same whether the disk is a storage device or a local disk. For a network device, however, there is a cache at the client and another at the server and network communication between them. The storage on a network device, however, is also likely to be spread over more disks and so the number of requests being queued to each disk will be lower. The data may also be stored as copies in multiple places, which again allows optimisation of requests and reduced queuing. The server side cache is also likely to be much larger than the cache on a standard production server. The network communication overheads are therefore traded for improved disk storage mechanisms in a dedicated device. To look at a particular instance, however, would need consideration of:

a)     Network interface efficiency;

b)    Network loading and communication efficiency;

c)     Relative disk performance;

d)    Disk duplication mechanisms, allocation strategy and efficiency;

In most cases a high quality storage device with a well designed network link will outperform a standard set of local server disks.

Firms waste billions on network over-design

Who needs desktop Gigabit Ethernet anyway?

http://www.theregister.co.uk/2006/05/22/network_overdesign_gartner/

It is true that simply upgrading hardware until the system performs well can look like an effective solution in many cases. It can avoid often complicated analysis and testing work, and in many cases will be effective in the short term. Simply continuing to throw hardware at problems as a strategy, however, doesn’t really work. The issue is that solving a solution in this manner will generally be a short term fix. If the solution continues to be to keep adding more resources then eventually the costs stack up to be unacceptable. In many cases there will also be a point of diminishing returns where even the best hardware on the market will fail to keep up. A more informed decision can be made by characterising a system and estimating the capacity that may be required to meet future requirements.