Are Quality Assurance Metrics Important?

Quality Assurance Metrics are the tool to use to create improvement in your testing, your team, and your company. Establishing target Quality Assurance standards that you can create benchmarking tests to monitor, allows a QA team to develop software testing metrics that gather the necessary information for improvement.

quality assurance metrics,benchmarking,software testing metrics,quality assurance standards

Is your QA team improving? Have you reached the highest state of efficacy? Is your testing as thorough as it can be? Are you able to do everything you want within budget?

If the answer to any one of those questions is “no”, then you should take the implementation of effective software testing metrics seriously. Fortunately, Quality Assurance offers a unique opportunity for creating these improvements.

Where Do You Begin?

To create improvement in any endeavor, you must first know what is happening. How are you doing? What sort of progress do you make?

You must create an objective picture of what is happening before you can create an effective map for improvement. The software quality assurance field offers a unique opportunity to gather this information.

Because testing and measurement are an integral part of QA, capturing the results of any activity is ready-made. You can gather the results of any test, report, meeting, etc. that you want at will.

So your first step is to decide what to capture. Attempting to measure everything at first is too daunting a task, so begin with something small. Once you begin gathering results, you will have data to measure.

Once you have data to measure you can create a clear picture of what is happening. When you know what is happening, you can choose what you want to change. You will be armed with the necessary information to create higher QA standards.

Begin With the End In Mind

Where should you start measuring? Anywhere that you want to apply improvement, that’s where!

It is generally best to start small. Identify an area where you want to determine how efficient and effective you and you team are. Then begin capturing metrics. This is the first step.

There are many software testing metrics that are used to measure productivity, efficacy, and efficiency. Which ones are right for you is a determination only you and your team can make. It really depends on where you want to create improvement.

I recommend beginning small. Don’t try to tackle your entire development process yet. Choose an area that provides feedback quickly and clearly. That way you can begin measuring (and practicing) sooner rather than later. Select an area where you are confident of the data that will be produced – and produces data that you understand.

If you start where much reliable, easily understandable data is created quickly, you will be able to start without delay. If you don’t understand the produced data, you won’t know what to do with it or how to measure it. If the data you are capturing yields only a small amount or yields it slowly, it will take much longer for you to gather enough data to be able to take any action.

What You Measure Matters

There are many standard benchmarks that are used when creating quality assurance metrics. As stated, what you choose to measure is up to you. Just keep in mind that you can only create measurable improvement in areas that you have quantified.

Which quality assurance metrics should you implement? Here are some standards:

  • Number of bugs found per build
  • Number of bugs found per module
  • Time between bugs found
  • Qualified Priority of bugs found
  • Mean time between P1 bugs
  • Time to perform tests
  • Bugs found per test

These are only a start, but let’s look at the possible use of each.

Number of bugs found per build
Common Usage: Progressive Build Improvement
One way to measure the stability of builds over time – and in comparison to one another – is the number of bugs that are found in each build. The number of bugs found should decrease from one build to the next over the course of the project.

There may be a data point spike in this metric when builds that contain new features are delivered. This should be expected. New features will result in higher bug counts, but over the course of the project, from build to build, the number of bugs found in each build should go down.

This, of course, assumes that your test coverage, methods, and accuracy do not fluctuate. In order to create good quality assurance metrics, there must be a baseline somewhere to begin with. In this case, the baseline is assumed to be standard tests (same number, type, even the same tests performed) from one build to the next.

Number of bugs found per module
Common Usage: Determination of Bug Density
Again, assuming your testing as a baseline standard, this quality assurance metric is about discovering which of your test modules (or functionality modules) yields the most bugs. This helps show bug density based on how you have your tests set up.

This measurement is often used as a benchmarking tool to determine test efficacy and productivity. Using this quality assurance metric in conjunction with a measurement of how long each test (or test module) takes will quickly show you a cost involved in your testing.

You should be able to see how long it takes to find a bug (average and total) in each tested area. You will be able to see of you are spending many hours testing an area that never yields bugs. With this software testing metric in-hand, you should be able to make more informed decisions about where to expend your test effort.

Time between bugs found
Common Usage: Build Improvement
The amount of time that passes between bugs being found is a quality assurance metric that can is often used to determine whether a project is progressing as it should. As a project gets closer to completion, the amount of time that passes between bugs found should become greater.

If there is no change, or if suddenly the time between bugs found gets shorter as a project nears its shipping date, this should be a cause for concern. If the time between bugs found does not increase, then the project is not advancing quickly enough.

This creates the risk of shipping the product with a high number of known bugs and/or with many bugs that are not found before it goes live. The other option is to miss the ship date in order to make sure the bugs are fixed – initially costly but often creating a better long-term updside.

Qualified Priority of bugs found
Common Usage: Testing Efficacy in Specified Timeframe
This Quality Assurance standard is used to ascertain if the test team is finding the high Severity and high Priority bugs. If the majority of bugs found are nit-picky low Priority issues, then either the release is low-risk, or QA is missing critical issues that will arise later.

When measuring this information it is important to know which tests have been executed. Are the tests that are being performed finding the issues that are important to the company? Is the test team using its time and resources as effectively as possible?

Not every bug found will be of the highest priority. There should, however, be enough synchronicity between the development team and the project stakeholders that the tests run find issues that are deemed to be important to fix. Otherwise effort is being wasted.

Mean time between P1 bugs
Common Usage: Build Improvement / Shipping Readiness
As the project progresses and nears shipping readiness, the average time between P1 (Priority = 1) bugs should become greater. Any P1 found will be a showstopper, so it becomes important that the product progress well enough for P1s to stop appearing.

When the mean time between P1s found stagnates or lessens, this should be a warning sign that your project is regressing. This is one of the clearest quality assurance metrics that you can track. It can carry a great deal of weight and impact. Use it wisely.

Time to perform tests
Common Usage: Testing Efficiency
Internal team quality assurance metrics can be used to create efficiencies throughout and after a project. One simple measurement of your team’s testing efficiency is how long it takes to perform selected tests. The first time a test (or set of tests) is executed, this number should be higher than during subsequent executions.

As your team becomes familiar with each test and learns the nuances to make them run smoother, you should notice that your test time falls. Specific tests (automated or activity dependent) may not vary, but the setup and subsequent result capture should take less time.

With a good team, you will also find that they will identify which tests can be run concurrently or in parallel to gain time efficiency. This is an important quality assurance metric to have available when estimating future projects and when used in conjunction with bug density quality assurance metrics, will show you where you can best spend your time testing.

Bugs found per test
Common Usage: Test Efficacy / Bug Density
This is one of the quality assurance metrics that you should track so that you can begin to evaluate whether your tests are targeted and thorough or not. Assuming your testing does not fluctuate, this metric will show you if you are testing areas that are yielding bugs.

Knowing which tests yield how many bugs – both as a total number and as a percentage – allows you to evaluate your testing. When measured in conjunction with a “time to perform tests” metric, you can quickly see if you are wasting your testing effort or leveraging your team effectively.

Absolutes or Trends?

Many QA Engineers like to use formulas to gather the results from their quality assurance metrics. This is a very common practice and can work very well if you use a formula that gets you the right information.

I have found that there is no special quality assurance metrics formula that gathers the data or makes it easier for me to understand, but many like them. If you are going to use a formula to gather your quality assurance metrics results, make sure it gathers the correct data. Also be careful to ensure that it is producing accurate numbers.

For projects that have achieved an advanced stage after many releases, some teams use absolute numbers to determine if a build is ready to ship. If they have released a product for several years in a row and know that their quality assurance metrics final numbers (bug count, time between bugs found, etc.) fall within a small window, then when those numbers appear, they determine the product is ready for release.

**Side Note:
I don’t personally like or use this method as it leaves to many other variables unaccounted for in my experience. It also precludes the QA Professional’s expertise in being an advocate for the consumer by understanding the details of the product as well as validating the holistic end-to-end experience.

There are, however, many QA Engineers and companies that find this practice acceptable. Even though I am not a proponent of it, I would be remiss to not inform you of its use. End of rant…

Once you have your quality assurance metrics results, then you can look to see what needs improvement. The key to this is being able to identify trends. This is, to me, a critical skill whether you use absolute numbers or not – you must be able to identify the trends that you see.

Finding a trend is basically just pattern recognition. The data will show that more is happening, less is happening, or your numbers are stable. Knowing which it is will allow you to take the next step and make informed decisions.

Being able to make decisions based on accurate, insightful information is the way to begin creating improvements. Identify what is happening, determine what you want to happen, and make the adjustments.

Then continue to gather your quality assurance metrics to measure your progress. With a little practice, you will be able to create whatever improvement you desire. You will be armed with the data you need to show others what is happening and why. This will make any recommendation you offer much weightier and impactful.

So, what is your opinion now? Do you think quality assurance metrics are important? Do you see a way that they can serve you? If you had the right data, could you make improvements in your own work? Could you make improvements in your team’s approach and methodology?

Leave Are Quality Assurance Metrics Important? and go to Successful Quality Assurance Home

E-mail Address
First Name (optional)

Don't worry — your e-mail address is totally secure.
I promise to use it only to send you SQA the Right Way!.