Non-Reproducible defects, sometimes, are even more lethal than existing defects. If ignored, you can never predict when the non-reproducible defect would rear it's ugly face and kill the product. Trying to present a personal perspective of "top 5 myths" around non-repro defects.
Myth 1: That non-reproducible defects are harmless
Not really. When it's happened once, it's more likely that it can happen again. Non-occurance in a protected environment does not mean permanent non-occurance in production. Secondly, can the defect can be ignored since it cannot be re-produced? Definitely not. Test around areas of the defect to explore it further. You never know what you can unearth unless you dig further!!!
Myth 2: That the development team have fixed the non-reproducible defect
A personal favorite. I have heard many stories of dev fixing non-repro defects. How do you verify a bug-fix that's not reproducible in the first place? I don’t know. I have never figured it out. Maybe they know something that you don't. If you have good interpersonal relationships with the developers, you should be able to talk to them to understand the root cause to try and re-create the error conditions to reproduce the defect and confirm if it’s fixed.
Myth 3: Large number of non-reproducible defects? No, it doesn’t’ mean much
Maybe a large number of non-reproducible defects means the following
a) Test Conditions are varying due to test environment instability
b) Testers have not understood product design
c) Dev and testers don’t talk to each other
Whatever it is, it means chaos!!! And that's a ill-omen and bad-health for the product
Myth 4: That the non-reproducible defect is a scenario that's never happenned before
If it’s an application in production, (a maintenance application), maybe it’s happened before. Look around.
a) Talk to the support teams to check if it's happenned in the past.
b) Talk to the veteran project members to see if similar defects have been unearthed in the pasts.
c) Look into the bug database.
d) Check the support log files (if you have access to support calls).
e) Network with older project testers who you are in touch with (even though they have moved to other projects or companies).
f) Maybe it's happenned earlier in life and you can get some info from earlier people. Also talk to people who have tested the application earlier to see if it's happenned in the past.
Remember, History is a great teacher!!!
Myth 5: That the sole owner of a non-reproducible defect is the tester
Non-Repro defects are mentinoed as errors of the tester. I Disagree. It's more like a case of a person being punished for trying to reach objectives beyond boundaries. When a tester tries to invest time in trying to showcase a rarely occuring problem, we are actually punishing him by thrusting ownership on him. Metrics crazy companies have assigned metrics for each developer and tester, and I am pretty sure that this results in the tester being blamed if a defect’s non-reproducible.
My take --> I feel that this defect should be owned by the entire project team. No single person ownership for non-reproducible defects.
Fake Tester's Gyan
1) Dont' ignore them
Most of us tend to ignore the defects which cannot be reproduced again. It's akin to commiting suicide. Please ensure that you address all non-repro defects before going live. Ensure that the right people get to view them and discuss them accordingly, before deciding on next steps of these defects.
2) A non-repro defect session
Why is it non-reproducible? Have a session with the entire project team to brainstorm regularly on all non-reproducible defects in the project. Maybe we would understand patterns on why a defect is non-reproducible!!!.
3) Business Severity
And look at business severity, as always. If it's high priority, then please spend some more time looking into areas. Try defining test conditions. Look for cases outside the requirements. Maybe it’s the slow system, slow connection speed, memory leak, etc. that’s resulting in this behavior. Classify high priority non-reproducible defects into a bucket and spend 1 hr a week analyzing them. It's better spending time on bug research now instead of doing it after the system goes live.
That's all!!!
As always, happy testing!!!
Your several posts including this one i liked very much but one thing is not clear to me, what is the reason of your anonymity. Your writing is good and interesting but you lost your credibility as being anonymous.
ReplyDeleteHope we will be able to know more about you very soon :)
- Selim
Selim, Thank you for the kind note. I have emailed you a note. I am not anonymous, all you have to do is a series of tests to find out the name.
ReplyDeleteReally good article. i wish even developers and other people in the business can understand the importance of non-reproducible defect rather than ignoring it.
ReplyDeleteAll About automation
Sacred drops
I work in a product based company & really use to face this problem very often(Non-Reproducible defect)Sometimes crashes occurs but does not reproduced,it creates mess when such types of defects comes at client end..& dev generally disagree with it...
ReplyDeleteReally Nice article !!