Day #3: Define and Quantify – Separate Yourself!

Define and Quantify the bugs you find. Everyone using software has seen bugs. What makes you a QA professional is your ability to explain what happened and the overall effect. Separate yourself from your coworkers by reporting your issues with more accuracy and weight.

In Day #1, we looked at the flow of the testing process and your role in it.
In Day #2, you learned how to execute tests and unearth bugs.

Today, in Day #3, we will turn that work of executing test cases into tangible defect reports. Today you will learn to take the issue you’ve encountered and report with accuracy and impact.

What To Do When You Find A Bug

Anyone who has ever used a piece of software has encountered a bug (whether they know it or not). What you do once you find a bug is what separates you from the uninformed layperson that thinks anyone can excel as a tester.

Now you add the value that only a true Quality Assurance Professional can. Now you define the bug.

Once you have found a bug you must determine:
• What happened
• Why it happened
• How often it will happen
• Under what circumstances it will always happen
• If there are circumstances in which it will never happen
• What is the true impact of the bug occurring

You can establish all of these points by following just three steps. In just three simple steps you can gather all of the information you will need to write great bugs!

How to Define and Quantify a Bug in 3 Simple Steps:
1. Eliminate the Variables
2. Establish the Reproducibility
3. Define the Impact

Having found a bug, you must now Define and Quantify it. This means you need to be able to eliminate the variables and boil the bug down into its most fundamental parts. You must determine what the exact steps are to make the bug happen again and eliminate all else. This way the bug that you write will have value.

If you don’t do this, then you have alerted people that something is wrong, but otherwise have only created more work for someone else, either your team or others. Obviously, that is never a good thing.

Step 1: Eliminate the Variables:
Let’s look at exactly how to Define and Quantify your bugs. First, back to the bug that you found: What did you do to make it happen?

Since you have been documenting all of the tests you have been performing, you should have a clear idea of how to reproduce the bug. You should have steps you can follow to recreate the bug. Do that now. Run those same steps again. Did the bug happen?

This is the opening salvo you fire in your work to Define and Quantify. Eliminate the variables and determine how to reproduce your bug.

If the bug did happen, review your steps to see if there are any that can be omitted and still get the bug to occur. Selectively leave out steps that likely don’t affect the outcome and attempt to reproduce it again.

As you do this, remember that your goal is to Define and Quantify each bug one at a time. Make this your mantra “Define and Quantify”. Take the necessary steps every time: Define and Quantify. Do this and you can’t go wrong, trust me. Always remember: Define and Quantify!

Were you able to skip any steps and still reproduce the bug? If so, repeat this process of removing steps until you have broken down the sequence needed to reproduce the bug into the fewest steps possible and make sure you document them for use when entering the bug.

If the bug did not occur again when executing all steps, you will have to delve deeper into what you did to make the bug happen in the first place. Try using the exact same steps and path that got the bug to occur originally.

Did it happen yet? It should have. If it didn’t, chances are you are forgetting something because although it is true that “sometimes computers just do that”, there is usually a reproducible cause.

As you work to Define and Quantify, keep in mind that your goal is accuracy of information. Speed is never the determining factor in this process. It is better to be slower and right than fast and wrong.

Try it from the top. Be curious; are there other things that you did that may have caused the bug? Are there other steps you could add that would make the bug occur?

This is where the great tester is separated from the average tester. A QA Professional that understands their craft will take the time to Define and Quantify. Test and experiment until you have firmly established the precise actions and sequence with which you can reproduce the bug.

Step 2: Establish the Reproducibility:
Once you are able to reproduce the bug, you need to discover how often the bug occurs. You must be confident that you have determined without question how often the bug will occur when you follow the steps you have outlined.

The way that you do this is to attempt to reproduce the bug several times. Follow the steps that you know will make the bug occur and execute them multiple times. Do this until you are certain you clearly understand how often the bug will occur.

If the bug occurs every time that you execute the steps, great! This is the simplest and most straightforward scenario. This is a bug that will always occur.

But what do you do if the bug doesn’t occur every time you execute the steps? What you must do is repeat the steps over and over until you can establish with certainty how often the bug does occur. Remember: Define and Quantify…

Execute the steps 20 times and make a note or a check mark each time the bug occurs. After you have attempted 20 reproductions, look at how many check marks you have made. Is there one check mark? Are there 2? Are there 10?

You should now have the information to state with conviction how often the bug occurs. Your testing shows that the bug occurs:
• 1 out of 20 times
• 2 out of 20 times
• 10 out of 20 times
• 18 out of 20 times

You can now report with accuracy what the reproducibility of the bug is. You can state how often the bug occurs when the steps are followed. Being able to report with this level of accuracy will help the bug get prioritized properly and will show that you bring integrity to your work.

If the bug has such a low reproduction rate, like 1 out of 20, that you don’t feel confident reporting its reproducibility, what should you do? Test more!

Attempt to reproduce the bug 20 more times. Then you will have the results of 40 of the exact same test steps. Test until you feel confident enough to report your bug and stand by its accuracy.

It is very important in this phase of your work to test until you are confident. You are the one that unearthed the issue and are the one that is reporting it. You must know that what you are reporting is accurate and must feel confident enough to defend it.

This is key in your quest to Define and Quantify. Gather the correct information and be sure that you are confident in the data you will report.

What do you do if you can’t get the bug to occur a second time? What do you do if a bug has occurred once, but you are unable to discover how to make it happen again? Don’t fret. We cover this scenario (the “Seen Once” bug) in the course: The 5 Keys to a Bulletproof Bug.

Step 3: Define the Impact:
Once you have Eliminated the Variables so that you know exactly what steps to follow to reproduce the bug, and after you have clearly Established the Reproducibility by executing multiple replications, you will need to ascertain what the ultimate effect of the bug is. Your goal is to be able to understand what the true outcome is when the bug occurs.

This goes beyond bug itself. The bug is the result that was incorrect. The bug is the reason you are logging the issue. The bug is the element that will need to be fixed.

For all of these reasons, the bug receives the focus – and rightfully so. But in order to excel as a tester; you must learn to Define the Impact of the bug. Always remember: Define and Quantify!

In Defining the Impact, you will communicate what is affected due to the occurrence of the bug. What I mean by this is that you must do more than state the first thing that happened (the Result of the bug). Once you state what happened (the Result), you will explain what has been adversely affected by that result.

It is not enough to report that the program crashed. Once you state that the program has crashed, you must state what that means. There are many outcomes that can occur.

Once you tell the reader that the program has crashed, you then quantify that crash for them. Your explanation could be either of the following examples:
• That the user just needs to restart the program and then can continue to use the program
– OR –
• That the computer became unresponsive and the user has to reboot the computer before they can continue

Can you see the difference in those two scenarios? In both cases, the program crashed. However, in the first case the computer was unaffected. In the second scenario, the computer had to be rebooted before anyone could do anything with it. These are very different impacts to an end user.

Is it clearer to you now how being able to Define the Impact and clearly communicate it to the project team would affect the priority they place on your bug? Can you see how this gives the bug deeper meaning?

A crash that affects the entire computer is far more severe to the end-user than a crash that only affects a single program. This is the type of differentiation you need to be able to provide for others.

Once you know how to make the bug occur and know how often it will occur, you will determine what the effect of the bug is to the end-user. When you can communicate this impact clearly, you will add a great deal of informed weight to your bugs.

Being able to describe the bug is very important (Define). To take your bugs to the next level, explain what the real-world impact will be to the consumer (Quantify). This will show that you understand the true nature of the bugs you are reporting.

Now that you know how to Define and Quantify, you have all the information you need to begin writing your bug, I will show you how to do just that. Tomorrow in Day #4 you will get all you need to enter your first bug with confidence.