This month we look at the difference between properly trained, elite QA Testers and the more expensive alternative. The story below illustrates how a real QA Professional, grounded in the fundamentals, can make a world of difference to a project.

The elite tester adds value to their team, their projects, and their company. The elite tester builds upon each project and advances their career by becoming more and more in demand.

The elite tester can be employed like a scalpel – the run-of-the-mill tester can only be employed with many others in what is known as a “shotgun technique”. The choice is yours.

(Story excerpted from the book Software Tester: Elite by Phillip Bailey of the Successful Quality Assurance Team)

Shotgun vs. Scalpel – OR – How Two Outperformed Twelve

This is often referred to as Shotgun Testing. Fire as many people as you can at the product, no matter the skill level, and hope that they find “enough” bugs. “Enough” seems to be defined by:
• Achieving target sales numbers
• Unfixed or unfound bugs being tolerated by the consumer
• Keeping the consumer confident enough in your product that they will purchase the next product

Unfortunately, Shotgun Testing often has several detrimental results, whether or not the company’s target sales are achieved. All of these results cost a company money, sooner or later:
• Delays in getting the product to market
• Shipping a product that still has WAY too many bugs for easy consumer use
• Having to manage a constant turnover in the staff that has been “trained” to test for the company
• Consumers that tolerate a far lower quality of software than is possible

There are, of course, other ways to do things. Successful Quality Assurance is only possible when a project is staffed by real QA Professionals. These QA Testers, Leads, Analysts, and Managers understand how to achieve the most thorough testing with the least amount of time and effort (and therefore, cost).

Even novice QA Testers can bring integrity and value to any project if they are trained in the fundamentals of Quality Assurance. Understanding how to succeed when testing software is the key.

Let me relate a brief anecdote to illustrate this point…

We had one client in particular for whom we had been testing for a few years. As they migrated their products from paper-based games to CD versions, we delivered great results for them time and again. As that company grew, they were acquired by a larger game company. This larger company had originally built its success on board games and had recently been trying to become competitive in the software game industry.

When our long-time client was acquired by the gaming giant, our client was told that all QA would be performed “in-house” by the larger company’s “interactive” software testing department. Would you be surprised to learn that this larger testing department applied the Shotgun Testing method? Well, if so, you shouldn’t be.

As a new product came up for testing, the producer for the product (whom we had worked with quite successfully before) called us to say that she was sorry but she was not going to be able to give us the business. She was, however, working on a proposal to give us a chance to show the cost savings the larger company could realize if they contracted us for testing in the future as opposed to using their own “in-house” team.

A couple of weeks later, that same producer called again to tell us she had been given the go-ahead. She had been given a budget for 2 of our testers for 2 weeks. She proposed that we test the same builds at the same time that the other team was testing and she would show the difference in the results of testing and in the cost (since we were not to be given access to the results of the “in-house” testing team).

We jumped at the chance! Not only did we want the very lucrative contract back in our house, this was a chance to show off our superior testing skills (at least, that was our opinion).

Being a project that had not been taken into account on our schedule, we had to decide who we would assign to the testing. We had one tester that was finishing up a project. For the sake of this story we will call him Jimmy. Jimmy was one, but we needed one more. Since I was spending the bulk of my time managing the QA Leads and the interaction with their clients, it was decided that I would be the other tester. Jimmy and me against a team of, what we discovered later, was twelve testers that the gaming giant thought were some of their best.

I was thrilled to have the chance to test with Jimmy. He was, and is to this day, one of the finest testers I have ever seen. He is not the fastest tester I have worked with, but he never missed a bug. Nothing got by him. He was the most meticulous, detailed tester I have ever known. It was rare that Jimmy ever made any kind of mistake and if he did, he never made it again. Testing with Jimmy raised the level of everyone who tested with him!

We received our first build late Monday afternoon. Just enough time to install it and read through the release notes. Our testing began in earnest on Tuesday morning. Jimmy and I looked at the program for the first time and met to decide on a plan of attack. We decided to go to our strengths. I would hammer all of the functionality and Jimmy would dissect the datasets. And off we went!

By mid-afternoon Tuesday we had logged 40 bugs, most of which we identified as bugs that were keeping us from being able to test other parts of the program. Every bug we logged was preventing us from accessing further functionality. At that point we had done all we could and we were stuck.

We emailed the producer that we had done what testing could be done and would need another build before we could continue into any more testing. She responded immediately that she had seen our bugs, understood that we were unable to continue, and was wondering what the other team was doing.

As it turned out, the “group-of-twelve” was still testing away. They were, apparently, unaware of all of the issues in the build that prevented further testing (chalk up “1” for the home team!). The producer was not happy at this most fundamental shortcoming, but said she would be sure they got a new build to us ASAP.

True to her word, the developers got some fixes in and overnighted a build to us that we received on Wednesday morning. Our assault continued!

Jimmy and I spent the rest of the week executing our respective tests. I logged tens of issues relating to blocked functionality, incorrect functionality, repeated patterns that precluded proper gameplay, etc. and Jimmy tore apart the datasets!

Jimmy logged every typo, incorrect copyright, incomplete sentences, inaccuracy, and every other issue that affected the game’s datasets. One bug in particular stands out to me to this day.

Jimmy logged a misspelling of one of the game character’s names. The bug was assigned back to him later that day with a note that read (I’m paraphrasing here): “The name is spelled correctly. We have checked the documentation and with the lead programmer. This is not a misspelling.”

But Jimmy knew better. This character had a Japanese name and Jimmy could see that it was not possible to spell a name that way in Japanese, so he persisted.

He assigned the bug back to the developer with a very diplomatic note explaining that it was not possible to spell a Japanese name in this way. He also included, very diplomatically, three different links to official game web pages where the character’s name was spelled correctly. He pointed out that these were official game sites (that the company we were testing for owned) and detailed where on the page you would find an accurate spelling.

The bug was again assigned back to Jimmy with a note: ”The name is spelled as it is supposed to be spelled. Please don’t tell us how to spell the names of our characters. This is not a bug.”

Knowing that this error would be major to a consumer who was an avid player of the game, Jimmy persevered. He again very diplomatically explained why it was not possible to spell a Japanese name is this way (this had to do with the name being spelled using letters/sounds that exist in English, but not in Japanese – and he was, of course, right). Jimmy then included 3 links to websites that showed how to read Hiragana and Katakana (“simplified” Japanese alphabets).

The next time we saw the bug it had only one additional comment: ”Fixed.” And, sure enough, it was!

The next Monday, a new build arrived. Jimmy and I tested all of the fixes and completed our testing sweep, hoping all the time that our work would show that although they were spending more per tester with us, if the gaming giant contracted with us, they would not need to spend as much overall as we could do more with fewer, higher quality testers. We finished testing and waited to hear back from the producer about how the “experiment” went.

As you might have guessed, the results showed that we provided far superior testing with far fewer testers. Although we did not get to test every game that the company subsequently developed, they definitely kept us busy going forward.

There was, however, almost a major glitch after all of that work we did.

We had two upcoming releases planned for testing with that company when, one day, my boss came flying out of his office and landed on my desk. He had just gotten off the phone with the producer in our story above and he was frantic.

“They shipped 800,000 units internationally and they have to recall them! How did we miss a bug that will make them recall 800,000 units?” He asked me. I felt a gurgle in the pit of my stomach. Not sure if I just needed to go get lunch or look for a new job, I ventured on.

“What bug? What recall?” I queried.

“The project you and Jimmy worked on! We missed a bug that is making them recall 800,000 units worldwide! How did you guys miss something like that? We just got the contract back and now they are threatening to pull their projects that are scheduled with us! How did you guys miss that?” He calmly explained.

I tried again, “What was the bug?” I thought about it a little more and realized that this didn’t make sense. I think rather highly of my own testing skills and I knew Jimmy wouldn’t have missed something like that. What could this be? “What bug is making them recall 800,000 units?” I asked.

Not to be placated by my most calming demeanor, by boss blurted, “Foreign characters! The game doesn’t recognize foreign characters. You can type them in, but it won’t remember the username. English is fine, but the game can’t deal with any non-English character!”

“Wha…?” It seemed to me that, indeed, that could be an issue since the game had been shipped to several countries worldwide. “We couldn’t have missed that!” I hoped. “Give me 30 minutes to look at our documentation. I’ll let you know in 30 minutes what happened.” I offered.

“Hurry!” and he was back in his office.

So, Jimmy and I compared notes. We looked through the documentation that had accompanied the builds. We loaded up the latest version that we tested. And then both exhaled a deep sigh of relief…

Fifteen minutes after the unrest had begun; we were ready to put it to rest. We summoned my boss to explain and, if necessary, show him what we found. While we were testing, the part of the program in question had not been enabled. In short, we never got a chance to test it because in the two weeks we tested, that functionality was never implemented. We didn’t miss it! It didn’t exist in the builds we had.

My boss, feeling much better now, called the producer back to explain the situation. As it turned out, her final report about the project and a “team-of- 2” vs. a “group-of-12” had one additional item to be added. Not only did we outperform them, they also missed a bug that we never had the chance to test. A bug that resulted in the gaming giant recalling 800,000 units.

Take two things away from this story:
1. A scalpel is a fine tool when applied correctly
2. Document Everything!! (you may have encountered this advice somewhere before)