Meeting the requirements

by Nick de Voil 4. February 2009 15:12

My house is about 160 years old. It has a certain Victorian charm, but a couple of years ago the roof started leaking. Actually, there had been small leaks for years previously, which we had fixed, but it was only then that the small leak turned into a big one in bad weather. I feared that the time might have come to bite the bullet and get the main roof replaced. The smaller, higher roof at the front of the house had been replaced perhaps 25 years ago, before we bought the house.

We called a reputable local company, and a man came to look at the leak. Naturally, I didn't tell him that we needed a new roof - instead, I showed him the leak and asked what to do about it. Surprise! He told me that we needed a new roof. I placed the order and we made a date for the work to be done.

A few weeks later, after we had cleared everything out of the loft space, the team arrived to do the job. They worked fast and efficiently, and by the end of the day we had a handsome new roof.

Then it rained.

Guess what? The leak was still there. We had spent thousands of pounds on - as we thought - solving the problem once and for all, but to no avail. The leak was still right there.

I won't go into all the painful details of the phone calls, letters etc. that followed. Suffice it to say that eventually they fixed it. Apparently the root cause of the leak was in the 25-year-old roof, not the 160-year-old one. The water was running down inside an inaccessible part of the roof space from the higher roof to the lower one, giving a misleading impression of the leak's origin.

Afterwards, I started thinking about this as a project. It has a lot in common with systems projects I've seen.

  • The customer (me) had a problem - the roof was leaking. I wanted the problem fixed - I wanted to have a situation where water was not coming into my house.
  • I turned to a supplier with expertise in this problem domain.
  • I told them the problem.
  • They proposed a solution to the problem - supply a product, i.e. build a new roof.
  • I accepted their proposal.
  • They carried out the work. Note that, as far as creating the deliverable was concerned, they did a fine job, apparently using trained staff, good materials and best practice techniques.
  • We tested the product by waiting for it to rain (and hoping that this would happen before payment was due).
  • The product comprehensively failed the test.

What conclusions can we draw?

The first thing that occurred to me was that I had failed to communicate the requirements correctly. It wasn't quite as simple as that - rather, I had failed to ensure that the agreement between customer and supplier was specified in terms of the requirement, rather than the product. Should I have accepted the proposal with a letter stating that I considered the absence of incoming water to be the key performance indicator, or critical parameter, for the project? Perhaps. Do systems project contracts get drawn up this way? Very seldom, in my experience.

What guarantee did I have that the product was the correct solution? Was this a classic case of the customer being sold an expensive product that they didn't need and didn't meet the requirements? Was the (prima facie) high intrinsic quality of the deliverable irrelevant? Well, actually I wanted a new roof because I knew that the old one had reached the end of its life and would have become a source of increasing frequent problems. So no, it wasn't irrelevant. But it is true that I got something I hadn't asked for to begin with, even though I wanted it.

Did we have a good user acceptance testing strategy? Well, it was risky, because it might not have rained before we were compelled to pay for the work. Note also that we needed to wait for rain a second time in order to carry out a second, successful test. But it would have been impracticable to simulate the rain before putting the new "system" into production.

How did we manage to reach a satisfactory conclusion? The honest answer is that I'm not sure. Was it because of the leverage we had obtained by not paying too soon? Was it because the supplier felt an obligation to satisfy the customer? Maybe a bit of both. But in the software world, it's quite normal for suppliers to charge extra for remedying problems which aren't within the contracted product scope.

And finally, how typical that the test failure was caused by a problem at the interface between two components!

My advice? Don't buy a Victorian house.