All the Fake Numbers

Someone once said "Keep It Simple, Stupid"!!!. I'd say, "Keep it Simple". If not, it means "You're Stupid!!!".

Let's try to see a scene that we’d all encounter during the end of the test cycle.

The Cast
Test Representative - Test Lead, Test Manager, Senior Tester, whoever. I call the person as a test representative since that's what he/she is
Boss - Master of the Daylight hours of the test representative
Big Boss - Master of the Daylight hours of the boss of the boss of the test representative,
Bigger Boss - Senior Management , Project Sponsors, etc. etc. etc...
Peer - Close friend/Best Buddy/Pal of the test representative, who is stuck in another project or wants to spend time browsing, or a workaholic, 1 who stays late @ office...

Project Stage ---> 2 Weeks before shipping

Time - 7 PM - Day 1

Scene - The test representative has been asked to send a daily status report on testing status by his boss.

Mental State of Test Representative ---> Panic Stricken!!! He has not sent such a report earlier in his life!!! His boss has re-directed him to the archives of the company to find a similar report. His peers have responded to his "call of panic" by flooding his mail boxes with format of a 10,000 status reports, some from his email archives and some from his google search.

What does the Test Representative do? - With a little time, he looks at whatever metrics is available with him, whatever he knows, and plugs it into a test status report and sends it to the whole world.

Fast Forward to next day.

Time - Day 2, 10 AM

Venue - Big boss's email

Activity - Big boss has received an email in the choicest language from his bigger boss's. They indicate a worthless status report.

Time - Day 2, 10:14 AM

Venue - Office of Big Boss

Scene - Big boss faulting the boss for a hopeless report, says the magic word “Appraisal”!!!

Language – The choicest words :) !!!!!

Time – 11:04 AM

Venue - Boss's office

Scene - Boss shouting at the Test representative for a very bad status report. Boss also mentions the word "Appraisal"!!!

Language – The very same choicest words :)!!!

What now? – Today, the boss wants to send the Test Report himself.

Time – 2:08 PM

Venue - Browser of the Test Representative

Activity - Test Representative having opened 7 Windows in IE and doing a search on test status reports and Metrics management. Finds a lot of data that he incorporates into the current day's test report.

Time frame– From 6:11 PM – 9:56 PM

Activity – 582 e-mails between the boss and the test representative on improving the report quality

Time - 10:02 PM in the evening

Venue - Boss's mail-box

Activity - Having reviewed the test report multiple times, and after so many updates between him and the test representative, the boss opens the current day's test status report and hits the "Send" button!!!

Boss’s Mental State - Relieved.

Time - 10:03 PM in the evening

Venue – Bigger boss’s mail-box

Activity - Opening the test status report sent by the boss and reads it.

Mental State – More Confusion!!! Is this what I said? Yes, he is confused. He is seeing so many details in the report, and he is confused on what to make of it.

Time - 10:46 PM

Venue - Hospital

Hospital? Yes, the bigger boss has been admitted for head injury. Witness seem to indicate that he has banged his head against a wall. His laptop has the latest status report open!!!

Time – Next day morning…

Venue – Bigger boss’s office

People - A bandaged bigger boss, the big boss and the test representative.

Activity – Well, I really don’t want to go into that… This blog is about fake software testing and I think we need to come back to our subject of software testing!!!

And the truth is--- Among all of the above people, there is definitely at least "1 fake software tester"!!!

What none of the above cast realized ---> The answer to Why, Why and Why. What questions does the status report, sent on an every day basis towards the end of the release cycle, try to indicate?

What these people should realize ---> The only 2 questions that people want to be answered from the test status report are ...--- How soon can we release? What is the health of the current build?

That's all, I believe, should the test status report try to indicate. It's very important for the test team to project these numbers so that all other stakeholders understand project health and ability to meet release dates.

Take a look at some of the metrics on test status reports that you can find defect reports - Time taken to test, no. of defects in all builds in the last year, Review count of test cases reworked on (from the previous generation), review time for test cases, Various delays that happenned in the project, Root Cause Analysis, Defect injected rate, Life-history of the entire project, Defect Removal Rate, Remaining defects rate, etc. etc. etc...

As in the story above, the boss and the bigger boss try to project a lot of information in these reports – The poor test team representative spends hours and hours of time in trying to get the above metrics into the report. Though each data point is valid at some stage, their usefulness is defined by the way it is useful to the project in the current stage. With very little time to ship the application, these numbers become useless.

Neither do these indicate the current health, nor do they indicate how soon can you ship the product.

Having said that, what do you think I say should be in the report?

1) Number of Open S1-P1/S2-P2 defects & time taken to resolve/re-test them - You will need to work very closely with the development team to predict these numbers.

2) No. of blocked test cases and the business priority of these blocked test cases - You will need to work very closely with Business/Dev team to say these numbers.

3) Ratio of no. of passed test cases to no. of total test cases - This is something that you should be able to say. Also indicate business priority of the passed test cases.

4) Sometimes people want to see a list of issues. Please feel free to include a list of open defects, but what will add value is an estimate on the time to fix/re-test these defects as well.

5) List of existing issues/blockers – Add to this list, details of who owns it and the ETA.

These not only tell you a status of the current health of the build, but also give you a view on how soon you can ship. That's all you need!!!

Of course, it would be very easy for all of you to tell me trillions of other metrics that you need...but then, if you subscribe to the thought of a trillion metrics in your daily test status reports, then you'd fall into the category of the fake software testers!!!

Happy Reporting!!!