Wednesday, January 16, 2013

How to measure and analyze the testing efficiency?

Measurements or Metrics or Stats are the common terms you would hear in every management meeting. What exactly does that mean? A bunch of numbers? Hyped up colorful graphs? Why should I bother about those? Well. Right from pre-KG education to K12 to college, we are so used to something called "marks". These are a set of numbers that reflect the academic capabilities of students (note - marks are not the real indicators of IQ levels). These marks in turn help academic institutions to select candidates, along with other parameters. In the same way, if one wants to measure the extent of testing, what are the numbers one must look at?

Fundamental rule. Do not complicate the numbers. The more you complicate, the more will be the confusion. We need to start with some basic numbers that reflect speed of testing, coverage of testing, efficiency of testing. If all these indicators move up, we can definitely be confident that the testing efficiency is getting better and better.

Every number collected from projects, help to fine tune the processes in that project and across the company itself. 

Test planning rate (TPR). TPR = Total number of test cases planned / total person-hours spent on planning. This number indicates how fast the testing team thinks, articulates the tests and documents the tests.

Test execution rate (TER). TER = Total number of test cases executed / total person-hours spent on execution. This indicates the speed of testers in executing the same. 

Requirements coverage (RC). Ideal goal is 100% coverage. But it is very tough to say how many test cases will cover 100% of requirements. But there is a simple range you mus assume. If we test each requirement in just 2 different ways - 1 positive and 1 negative, we need 2N number of test cases, where N is the number of distinct requirements. On an average, most of the commercial app requirements can be done with 8N test cases. So, the chances of achieving 100% coverage is high if you try to test every requirement in 8 different ways. Not all requirements may need an eight-way approach.

Planning Miss (PM).  PM = Number of adhoc test cases that are framed at the time of execution / Number of test cases planned before execution. This indicates, whether the testers are able to plan the tests based on the documentation and understanding levels. This number must be as less as possible, but it is very difficult to achieve zero level in this.

Bug Dispute Rate (BDR). BDR = Number of bugs rejected by development team / Number of total bugs posted by testing team. A high number here leads to unwanted arguments between the two teams.

There is a set of metrics that reflect the efficiency of the development team, based on the bugs found by the testing team. Those metrics do not really reflect the efficiency of the testing team; but without testing team, those metrics cannot be calculated. Here are a few of those.

Bug Fix Rate (BFR). BFR = Total number of hours spent on fixing bugs / total number of bugs fixed by dev team. This indicates the speed of developers  in fixing the bugs.

Number of re-opened bugs. This absolute number is an indicator of how many potential bad-fixes or regression effects are injected into the application, by the development team. Ideal goal is zero for this.

Bug Bounce Chart (BBC). BBC is not just a number, but a line chart. On the X axis, we need to plot the build numbers in sequence. Y axis contains how many New+ReOpen bugs are found in each build. Ideally this graph must keep dropping towards zero, as quickly as possible. But if we see a swinging pattern, like sinusoidal wave, it indicates, new bugs are getting injected build over build, due to regression effects. After code-freeze, product companies must keep a keen watch on this chart.

For a free test management tool, visit

For free recorded training, visit Also available on

Sunday, January 6, 2013

Regression Testing

Can someone tell whether the last bug is removed from the product? Can someone give 100.00% guarantee that the testing is fully done? Yes and No. Given the time, it can be done; but then what is the timeframe one needs to get to this level? Karate Schools believe in repeating certain exercises to attain perfection. Practice leads to perfection. Practice is nothing but doing it again and again, and ever improving. This holds good for software testing also.

When we walk thru the regular SDLC, the system testing happens once the product is given to the QA team. A set of bugs are published and those bugs get fixed by the development team. What next? Again testing  happens. The question is , how much testing must happen again? Well, the ideal goal is to do all tests on the product once again. This will ensure that a fix in one place does not break something else. But, do we have people, time and money to do all tests again? The following principles are used in most of the organization when it comes to repeating tests. 

Whatever tests we did in build 1, that have passed in build 1, must not fail in build 2. This is fundamental. So ideally we must execute all test cases in build 1 and 2. Assume that there are 500 tests and we executed all of them in build 1 and around 60 test cases failed. When build 2 comes, the first priority goes to checking those failed test cases for the correctness of the fix. This is called re-testing. That means, a test case failed and it was fixed and we execute that test case again. Forgetting all other tests, we check this test case alone; this is re-testing. To find out whether there are any side effects due to the fix, we must do regression tests. Regression testing means, to re-test the test cases that have passed in previous build also. Every time you go to the doctor, the doctor checks the pulse, temperature etc. That part is regression. Same way we must repeat passed tests also, to ensure functional stability.

But how much regression is good? A few simple principles you may try. These are some generic thumb-rules used by most of the companies.
  • If a test case fails for a field, repeat all test cases for that field.
  • If a test case fails for a screen, repeat all test cases for that screen.
  • If a test case fails for a data entity (e.g. purchase order, Invoice etc.), repeat all test cases for that data entity wherever that is used.
  • If a test case fails for a functional module, repeat all test cases for that module.
You are the ultimate authority to decide which suits better for your product within the timeframe you have got. If you are able to cover up a larger area of the application, around the failed test cases, your chances of blocking all loopholes is high.

When it comes to large products such as ERP, when the product reaches the system testing phase, the work of developer may be just 10 minutes to fix a bug; but the testing will go on for days, due to regression. Hence, testing takes a huge effort build after build. You can ask me whether companies are doing proper regression testing. The answer I can give is that those companies that do proper regression have stable products  and larger customer base; this is due to the quality of the product being tested again and again.

For a free test management tool, visit

For free recorded training, visit