5 Steps to Bug Regression:
Day #3

You have the build and understand the fix. Now all you have to do is see if the bug still occurs. Easy! Test the bug one single time and if you don’t see it, all is well, right? Uh, no…

Step #3:
Validate Appropriately

In order to validate appropriately, you must do the right things so that you can report the right things. What does this mean? What are the right things to do? I thought you’d never ask…

When regressing a bug, you follow the exact Setup and Steps to Reproduce to see if the bug occurs. That part is simple and straightforward. Make sure you follow the exact same Setup and Steps that are reported in the bug.

If you see the bug occur – even once, then the test for this fix fails and you can now move on to Step #4: Comment Thoroughly. But, if the bug does not occur, what should you do?

If the bug you are testing has a Reproducibility of Always, then you should test it at least 3-5 times and maybe even as many as 10 times. If you do not see the bug occur, you should feel confident that the fix was successful. Since this bug was occurring 100% of the time, you can feel certain after 10 or so attempts that this bug has indeed been fixed and you can move on.

The key to this Step is in testing until you are comfortable stating with confidence that the bug no longer occurs. That level of confidence takes more than a single try to attain.

Test until you feel confident that the bug has been successfully fixed. And, since you are documenting everything, make sure you note how many attempts you made to reproduce the bug without being successful.

If the bug you are testing has a Reproducibility of Intermittent, the number of attempts you will need to make to feel comfortable probably just went up. How many tries would you need in order to feel comfortable that an Intermittent bug was no longer occurring? 5? 10? 20? 100?

It is really up to you – and the Intermittency. Of course, the Reproducibility was reported in a format that was impactful, right? The Reproducibility reads: “Intermittent: Occurs 6 out of 10 times.” Right?

If that is the case, then you can probably test 10 times. If the bug doesn’t occur at all but you don’t yet feel confident, give it another 10 tries. If you don’t see that bug in those 20 attempts, you can report that the bug no longer occurs with a high degree of certainty, right? Great!

But what if the Reproducibility as reported reads: “Sometimes. It happens about 10% of the time.” Then you need to find the person that wrote the bug and…no, no, never mind…let’s not do any of that. Being a true Quality Assurance Professional, you seize the challenge and do your best.

You attempt to reproduce the bug 20 times. If you see the bug at all, fine. The fix fails (when you comment the bug, please update its Reproducibility with information that is actually useful).

But what if you attempt to reproduce the bug 30 times or 50 times and don’t see it? What if you don’t yet feel comfortable stating with certainty that the bug is fixed?

What if the Reproducibility, as reported, leaves you without adequate confidence to close the bug? In this case, you should talk to your lead and explain the situation to them.

It would be a mistake to close a bug that you were not certain was fixed. There is a risk in this scenario that you may not be sure the bug is fixed, so what should you do? I’ll tell you.

Talk with your QA Lead or Manager and explain the situation to them: Due to the reported Reproducibility, you are not comfortable closing the bug yet. You have made 30 attempts to reproduce the bug and have not yet seen it, but tell them that it is unclear how often the bug occurred in the first place. Suggest that you comment the bug with your most recent data and leave the bug open and assigned to you for continued testing in the next build.

If you will be getting another build, then there is less harm in testing the bug again than there is in closing it when it might still occur. So, what you should do is to properly comment the bug with the latest information (we will cover that tomorrow in Step #4: Comment Thoroughly) and leave it open to be tested when the next build arrives.

Then, of course, you must be sure to test it in the next build and add further data to it then. If after another 30 attempts, the bug has not occurred, you may feel comfortable closing it.

One place that I worked, we had a rule of “3 consecutive builds”. That is: if a bug were very intermittent or otherwise difficult to reproduce, once a fix was in place we would thoroughly test the bug in 3 consecutive builds.

If we didn’t see the bug in 3 consecutive builds then, and only then, we would close it. It would also be added to our list of “most wanted” that we kept at our workstations to keep an eye out for.

The real key to regressing a bug in a way that Validates Appropriately, is to test until you can state with a high degree of confidence that the bug no longer occurs. Or, alternatively, test until you see the bug occur and can state with confidence that it still needs to be fixed.

Either way, test until you are sure. Test until you feel confident. Don’t trust anyone else to tell you whether or not everything is good and happy. You are a Quality Assurance Professional. You carry the standard and set the bar for excellence. You are the consumer’s advocate!

When you say that a bug no longer occurs, others will believe you. They will believe you because you are QA. They will believe you because they want the bug to be fixed. They will believe you because you have built a relationship of trust with them. So, be right and be accurate. When you claim a bug no longer occurs and can be closed – have confidence!