5 Keys to a Bulletproof Bug:
Day #2

After you have written your Expanded Description (see Day #1), you will want to define your bug’s Reproducibility. You should write this directly under the Expanded Description.

This way the reader will immediately have an understanding of how often the bug you have just described occurs. This is the order that a reader can most easily understand the Severity of your bug and the Priority that should be assigned to it.

Key #2: Reproducibility

• Always
• Intermittent
• Seen Once – OR – One Time

Let’s look at when you would use each selection.

Always
The bug you are writing will happen 100% of the time, without fail, when the Steps that you write are followed. In this instance “Always” really means always.

When you write up a bug and state that it has a Reproducibility of Always, be certain that this is true. Do enough testing to determine that the bug will happen every time. Not most of the time, ALL of the time when the reader follows the Setup and Steps to Reproduce that you detail.

It is possible to have a bug that will only occur under specific circumstances or only when certain conditions are met. If you include these details in your Setup and Steps, then you still will have an Always bug.

Conditions that may affect, create, or contribute to a situation-specific bug include (but are not limited to):

• Operating System
• Browser Version
• Media Player Version
• Internet Connectivity Type
• CPU Speed
• Available RAM
• DirectX Version
• Any combination of the above

When this is the case, you must include these situation-specific factors in your bug. If you detail these factors correctly, your Reproducibility should be solid. It is ok to write that a bug Always occurs when using Windows 98SE and then following the Steps as written.

Intermittent
In your journeys you will encounter bugs that occur sometimes, and only sometimes. This can be frustrating, but it does happen and you will have to accept it as a reality of testing.

However, make sure in your testing that you don’t fall into the trap of accepting a bug as Intermittent too quickly. Many novice testers too easily accept that the bug they have just encountered is Intermittent. Often they are wrong. They just haven’t spent enough time and effort to determine the Reproducibility with surety.

When it comes to light, as it always does, that the bug is not Intermittent but occurs more frequently (see: Always) and the tester simply didn’t complete the work to do their job then the perception of QA suffers. The team loses trust in the tester’s skill and attentiveness.

As I stress throughout this and every other course, this is the opposite outcome you are working to achieve. Remember that in all you do, you are working to build trust. So, be thorough. If a bug is truly Intermittent, so be it…just make sure that you are certain it is.

To reiterate, when you encounter a bug that appears to be Intermittent, you must remember two things:

First: Test and re-test until you are certain that you are unable to find the exact circumstances and Steps that will give you a Reproducibility of Always. Once you have done that and are sure your bug is truly Intermittent, then move on to the second thing.

Second: Whenever you write up a bug with a Reproducibility of Intermittent, you must detail the Intermittency. What I mean by this is that you must state that the bug occurs “X-out-of-Y” times.

You must clearly state the bug occurred "how many times" out of "how many attempts". This way everyone can get an informed understanding of the true impact of the bug you have written.

If requested or if you like, you can state this Intermittency as a percentage also, but you must state “X out of Y”. This clearly quantifies the current impact of the bug.

Only stating that the bug occurs 20% of the time is not an impactful statement. This could mean:

• 1 out of 5
• 2 out of 10
• 20 out of 100

A sample size of 100 is a far more accurate count of the bug’s impact than a sample size of 5, therefore “20 out of 100” is a more impactful statement than “1 out of 5” – so be sure to always include “X out of Y” ANY time you enter an Intermittent bug.

Seen Once – OR – One Time
It will happen, but I hope for you it will happen only rarely. You will encounter bugs that, despite your best efforts, you will have to enter as Seen Once. What this means is that you were able to produce a bug once, but only once. You have tried and tried, but are unable to establish a circumstance and Steps that will entice the bug to occur a second time.

Although sad, this happens to everyone. Everyone that stays in testing long enough, that is. When this happens to you, be a professional and do the right thing

“Pretend it never happened?!?”

As nice as that would be, no…don’t do that. You can do the next best thing.

As a Quality Assurance Professional, you must perform due diligence and ensure that you are unable to find a way to reproduce the bug. If you can do this, then you can move on and enter the bug.

When entering a bug with a Reproducibility of Seen Once, be sure to state how many attempts you have made to reproduce the bug. This will help others understand whether or not attempts have yet been made to reproduce the bug – or if you had to enter it because it was closing time and you had to get the bug into the bugbase.

Write it up as you would any other bug, but make sure that you include all the information you have about what the circumstances were when the bug occurred. Include the details of:

• What actions led up to the bug occurring?
• What environment were you working in when the bug occurred?
• What testing did you perform when attempting to reproduce the bug?

That last piece of information is key: What have you tried that did NOT result in the bug occurring again? With this information, it will be easier to narrow down the variables going forward if the bug is ever seen again. If nothing else, it will give the only other data you have about the issue – this can be key if anyone is to have any chance at tracking down the issue.

Whenever possible, this Reproducibility should be upgraded to a more precise entry as soon as possible. However, there are times when you simply will not be able to figure out what particular circumstances caused an issue to occur – these should be few and far between, but it does happen. So after giving it your best effort, update the bug with as much information as possible and move on.