Monday, December 6, 2010

Brakes and Defect Prediction...

Brake prediction and defect prediction are related. Defect Prediction is defined by me as the ability to predict the number of defects that would occur in the next development cycle. How is Defect prediction possible? Read on for my thoughts.

I did the following exercise. My workplace is an hour’s drive from my home. I challenged myself to predict the number of times I’d brake on a day, while driving from home to office.

What I did was as follows:-

1) From Day 1 - Day 5, I calculated the number of brakes on each day to create a data repository.

2) From Day 6 - Day 15 onwards, I used the information gathered from the previous days to try and predict the number of times I braked every day. I also updated the data repository at the end of every day.

Top 7 of my observations in italized below with their corrosponding learning in Software testing in BOLD inline…

1) Every day, I applied the brake at least once on my way to work.

All of us can predict at least 1 defect in the system. (I guess that's the closest we can get to with predicting defects).

2) Though the traffic conditions and road crowd was same on all 15 days, my predictions were wrong. They had a variance of + and – 50%.

Number of lines of code cannot be used as a factor for predicting defects

3) Regardless of how many times I braked, I always reached office in  60-70 mins. Average time to reach destination was not hampered by the number of brakes.

Your project schedule will not be changed by your number of defects. Rather they are determined by your ability to detect and fix them in quick time.

4) Sometimes, when I braked, I had to remain braked for long periods of time, due to different reasons. Traffic, crowd, signals, etc. etc. etc.

Severity of defects can never be predicted at all

5) Mostly, I thought I’d hit the brakes at the same spots on the days.Wrong. More than 50% of the time, I hit the brakes at different locations.

Defect Predictions becomes a highly mis-guiding factor. It also gives you the false notion that the total number of defects will not exceed a given number. It may be proved false after the application is launched.

6) On 2 days a week at least, I had to take a different path to work. My algorithm to predict defects went haywire on those days.

During the path, teams need to display a lot of agility. Many a time, you have to take a different direction to reach your goal. Your defect prevention algorithm, may not factor in for these change of direction

7) And False Alarms!!! Sometimes, I expected an obstruction and hit the brake, but it turned out to be a false alarm.

You cannot predict false alarms. Sometimes, you might spend a few hours on a trivial or "Difficult To reproduce" defect.

Well, I certainly was not accurate on predicting defects or my brakes, but at least learnt enough information I thought is worth a blog post :)!!!

And last, the fake tester's gyan

1) Dont bother about the future. Worry about the present.

2) Ask yourself the question --- If you had 5 mins at your disposal, would you try to test the product, or use the time to predict the total no. of defects the next development cycle would throw up?

3) And lastly, when we are saying that it's not possible to fully test a product, how can you predict the number of defects in the future?


If the number of defects varies by 50%, then how do you estimate work for dev teams during testing? Well, more on that another day!!!