How to measure and analyze the testing efficiency?

Measurements or Metrics or Stats are the common terms you would hear in every management meeting. What exactly does that mean? A bunch of numbers? Hyped up colorful graphs? Why should I bother about those? Well. Right from pre-KG education to K12 to college, we are so used to something called "marks". These are a set of numbers that reflect the academic capabilities of students (note - marks are not the real indicators of IQ levels). These marks in turn help academic institutions to select candidates, along with other parameters. In the same way, if one wants to measure the extent of testing, what are the numbers one must look at?

Fundamental rule. Do not complicate the numbers. The more you complicate, the more will be the confusion. We need to start with some basic numbers that reflect speed of testing, coverage of testing, efficiency of testing. If all these indicators move up, we can definitely be confident that the testing efficiency is getting better and better.

Every number collected from projects, help to fine tune the processes in that project and across the company itself. 

Test planning rate (TPR). TPR = Total number of test cases planned / total person-hours spent on planning. This number indicates how fast the testing team thinks, articulates the tests and documents the tests.

Test execution rate (TER). TER = Total number of test cases executed / total person-hours spent on execution. This indicates the speed of testers in executing the same. 

Requirements coverage (RC). Ideal goal is 100% coverage. But it is very tough to say how many test cases will cover 100% of requirements. But there is a simple range you mus assume. If we test each requirement in just 2 different ways - 1 positive and 1 negative, we need 2N number of test cases, where N is the number of distinct requirements. On an average, most of the commercial app requirements can be done with 8N test cases. So, the chances of achieving 100% coverage is high if you try to test every requirement in 8 different ways. Not all requirements may need an eight-way approach.

Planning Miss (PM).  PM = Number of adhoc test cases that are framed at the time of execution / Number of test cases planned before execution. This indicates, whether the testers are able to plan the tests based on the documentation and understanding levels. This number must be as less as possible, but it is very difficult to achieve zero level in this.

Bug Dispute Rate (BDR). BDR = Number of bugs rejected by development team / Number of total bugs posted by testing team. A high number here leads to unwanted arguments between the two teams.

There is a set of metrics that reflect the efficiency of the development team, based on the bugs found by the testing team. Those metrics do not really reflect the efficiency of the testing team; but without testing team, those metrics cannot be calculated. Here are a few of those.

Bug Fix Rate (BFR). BFR = Total number of hours spent on fixing bugs / total number of bugs fixed by dev team. This indicates the speed of developers  in fixing the bugs.

Number of re-opened bugs. This absolute number is an indicator of how many potential bad-fixes or regression effects are injected into the application, by the development team. Ideal goal is zero for this.

Bug Bounce Chart (BBC). BBC is not just a number, but a line chart. On the X axis, we need to plot the build numbers in sequence. Y axis contains how many New+ReOpen bugs are found in each build. Ideally this graph must keep dropping towards zero, as quickly as possible. But if we see a swinging pattern, like sinusoidal wave, it indicates, new bugs are getting injected build over build, due to regression effects. After code-freeze, product companies must keep a keen watch on this chart.

For a free test management tool, visit http://www.qamonitor.net

For free recorded training, visit http://www.openmentor.net. Also available on http://www.youtube.com/freeopenmentor

Regression Testing

Can someone tell whether the last bug is removed from the product? Can someone give 100.00% guarantee that the testing is fully done? Yes and No. Given the time, it can be done; but then what is the timeframe one needs to get to this level? Karate Schools believe in repeating certain exercises to attain perfection. Practice leads to perfection. Practice is nothing but doing it again and again, and ever improving. This holds good for software testing also.

When we walk thru the regular SDLC, the system testing happens once the product is given to the QA team. A set of bugs are published and those bugs get fixed by the development team. What next? Again testing  happens. The question is , how much testing must happen again? Well, the ideal goal is to do all tests on the product once again. This will ensure that a fix in one place does not break something else. But, do we have people, time and money to do all tests again? The following principles are used in most of the organization when it comes to repeating tests. 

Whatever tests we did in build 1, that have passed in build 1, must not fail in build 2. This is fundamental. So ideally we must execute all test cases in build 1 and 2. Assume that there are 500 tests and we executed all of them in build 1 and around 60 test cases failed. When build 2 comes, the first priority goes to checking those failed test cases for the correctness of the fix. This is called re-testing. That means, a test case failed and it was fixed and we execute that test case again. Forgetting all other tests, we check this test case alone; this is re-testing. To find out whether there are any side effects due to the fix, we must do regression tests. Regression testing means, to re-test the test cases that have passed in previous build also. Every time you go to the doctor, the doctor checks the pulse, temperature etc. That part is regression. Same way we must repeat passed tests also, to ensure functional stability.

But how much regression is good? A few simple principles you may try. These are some generic thumb-rules used by most of the companies.

• If a test case fails for a field, repeat all test cases for that field.
• If a test case fails for a screen, repeat all test cases for that screen.
• If a test case fails for a data entity (e.g. purchase order, Invoice etc.), repeat all test cases for that data entity wherever that is used.
• If a test case fails for a functional module, repeat all test cases for that module.

You are the ultimate authority to decide which suits better for your product within the timeframe you have got. If you are able to cover up a larger area of the application, around the failed test cases, your chances of blocking all loopholes is high.

When it comes to large products such as ERP, when the product reaches the system testing phase, the work of developer may be just 10 minutes to fix a bug; but the testing will go on for days, due to regression. Hence, testing takes a huge effort build after build. You can ask me whether companies are doing proper regression testing. The answer I can give is that those companies that do proper regression have stable products  and larger customer base; this is due to the quality of the product being tested again and again.

For a free test management tool, visit http://www.qamonitor.net

For free recorded training, visit http://www.openmentor.net

 

Bug Life Cycle

Like a butterfly going from egg to larva to pupa to a fully-grown butterfly,  the bug also has a life cycle. Each bug should go through the life cycle to reach a logical end. A specific life cycle ensures that the process is standardized. A life cycle is nothing but taking one thru a set of stages, as time passes by. If you take a civil lawsuit, the case goes thru a set of stages such as case filed, case hearing done, sessions court verdict given, applied to high court etc. Treat a bug as a case filed by the testing team and the project manager needs to see all cases are closed before release.

Typical stage or status codes are New, Open, Fixed (or Resolved), Retested, Closed. The codes may slightly vary from company to company, but these are the status codes that are universally accepted.

New: When the bug is posted for the first time, its state will be “NEW”. This means that the bug is not yet approved.

Open: After a tester has posted a bug, the lead approves that the bug is genuine and he/she changes the state as “OPEN”. This also means that the development team has acknowledged the bug. 

Fixed (Resolved): Once the developer fixes the bug, he has to change the status as Fixed and assign the bug to the testing team for retesting. Developer must also provide what kind of resolution he has done to fix the bug. 

Retested: Once the bug is fixed, the tester retests the bug in the subsequent build or in a hot-fix. If the bug is not present in the software, he sets the status to “Retested”.  

Closed: When the bug is successfully retested and the bug is not found any more, usually the test lead or manager closes the bug. This state means that the bug is fixed, retested and approve and no more exists.

Apart from the above regular stage/status codes, there are additional codes used to exactly position the bug's status. 

Reopened: If the bug still exists during retesting even after the bug is fixed by the developer, the tester changes the status to “Reopened”. The bug traverses the life cycle once again.Reopen is on the same line with Open.

 Deferred: This means the bug is expected to be fixed in next releases and not immediately. The reasons for changing the bug to this state have many factors. Some of them are priority of the bug may be low, lack of time for the release or the bug may not have major effect on the software. 
 

Rejected: If the developer feels that the bug is not genuine, he rejects the bug. 

Closed-Duplicate:If the bug is already logged by same tester or another tester in a different context, this can happen. so every tester must do a simple search in the bug tracking system whether a similar bug exists or not.

Closed-Irreproducible: Sometimes, the bug may happen due to a strange environment or data problem. The development team may not be able to exactly reproduce the bug and hence they cannot fix the same. In such a case, the bug may land up in this stage.

Closed-By-Design: When test cases were not exactly matching the specs or design, the understanding of the tester may be wrong. In such case, the bug will be closed in this manner.

Typical disputes on bugs

When someone says that you are wrong will you accept on the first shot or will you try to defend? This is typically the human behavior. More than that, if I come and tell you that your son or daughter damaged my car glass, will you defend or not?  More than defending self, one will defend more when it comes to one’s son or daughter. Same way, when a tester finds an issue on a program, many programmers will defend their program, as if it were their kids !  Quite natural. But in a professional environment this has to be handled carefully. This is usually handled in triage meetings. This section discusses the soft issues that occur due to these disputes.

This is not a bug:

 

The first resistance will be to say that it is not a bug. When this comes, we need to remove our emotions out and look at the requirements and design. As long as the product does not work as per requirements and design, it is a bug. But in some cases, the documents may be wrong or old; in that case, assign the bug to the business analyst or systems analyst to modify the requirements and/or design documents.

Over-testing of one module:

If a tester posts a bug on a module and that is overruled in triage meeting, the tester will defend the bug found. When his/her kid is ignored, when the next build comes for testing, the tester will focus more on that module irrespective of any project priorities and bring out more issues. If they are genuine, it is good. But most of the times, the issues are brought out for the sake of it with a human dislike.

If this is sensed by the Test Lead in advance, he can roll the tester to another module till time heals the attitude of the tester.

The developer pinch:

When the developer is flooded with bugs, he/she may try to talk to the tester and will show the technical superiority. It may go to the extent that the developer telling the tester “what do you know in java ? we create and you do only the testing….”. But, remember, the tester knows what customer wants. So there may be a rift between the developer and tester. If this is identified, we must ensure that the team members do not interact directly except triage meetings. Many companies put these two teams in two different buildings !

Blame it on requirements:

When more bugs are found, and development lead and test lead not getting well, both will start blaming the ambiguity and gaps in requirements. This affects project progress. The position of the business analyst will be very difficult. In this situation, the manager must stress to stick on to what is stated in specifications – nothing more, nothing less. First do, what is stated and then resolve gaps.

Push to Deliver:

When project manager overrules the results published by testing team and approves delivery to customer, the testing team will feel hurt. When the customer sends the product back pointing out the issues, again the same blame will come on the testing team only. To resolve this, we must publish delivery with known issues list to customer. This will give a better feel to testing team.

For further details, please visit http://www.freeqamonitor.com.

Tester's friend - The BUG

Every normal human being gets pleasure while finding mistakes in others, sad, but true. Tester is also a human being and when he/she finds a bug, it further motivates him/her to find more. A tester aims to fail the software in any way possible, and tester succeeds when bugs do appear. I do not worry about the theoretical definition of bug; to me, anything that the software does in a manner that is not intended as per specs or by implied needs, is a bug. Hence bug, error, problem, issue, failure etc. all come under the same category, though you can find zillion definitions for these words if you google.

When a bug appears while you test, do the test again. Ensure it appears again. This will prove that, you as a tester, had not done any mistake in executing the test. Soon after confirming its consistent nature, do not call the developer. Developer has a tendency to reject or dispute. Do not send a mail either. You need to ask a few questions to yourselves before you formally state that it is a bug. 

Try to describe the bug in 1 line. This is like writing the email subject. "User does not get confirmation email after registration", "Application accepts invalid SSN in Add Customer Page". Try to convey the exact problem with minimal words.

Mention the screen name where the bug appears. This will help the developer directly to go to that place, without wasting time. If possible, take a screenshot where the bug appears. A picture means more than 1000 words.

Explain the bug in detail, by telling the steps to reproduce. You get bugs when you execute a test case. Thus, the bug must be appearing against one of the test steps. Copy paste the test case itself and mention the exact problem against the test step. If the developer does the same thing, he/she should get the same bug.

How does this bug affect the product? The answer to this question will help you to set the Severity of the bug. Are you able to proceed further to do other test cases? If not, this bug is termed as a Show Stopper or Blocker. This means, you have no way to test anymore unless development team fixes the same. 

Is there a crash or data loss or a feature is missing in the product? This falls under High or Major category, provided you are able to do other tests. This means, that the product has a serious problem. For example, when we page customer details, Save operation itself does now work and hence the data entered is lost. Technically a developer may say that he can fix this in 2 minutes, but always look from customer's angle. For the customer, data is more important than the product itself. Any loss of data is a major bug. The severity code Major is also termed as High in many companies.


Bug does not stop me from testing, there is no data loss or crash, but the functionality is wrongly implemented. The app saves data, but some part of the results are not accurate. In this case, companies usually treat this as Minor or Medium severity.

The app does the job, but there are spelling mistakes or improper messages to users or imporper field alignments in the UI. Usually companies treat this as Cosmetic or Low severity.

To sum up, if we can write the following, then we are good in reporting bugs to dev team.

• One line description of bug
• Details to reproduce the bug
• Screen name where the bug appears
• Screenshot of the bug
• Severity of the bug

By principle, every problem must be reported and tracked. We will see tracking mechanism in the next post. For more details on bugs, please see this bug tracking demo.

Test Execution - How can we do faster?

Development deadlines always slip, but release dates do not change; caught between these 2, the testing time is always compromised. Hence the testing teams need to work faster. If the application has withstood the first one hour of critical tests, it indicates the basic stability of the app. But every test is important from functional point of view. Here are some simple techniques to make it faster.

Testing teams may have a mix of people with different experience levels. It is very uncommon to see a testing team with uniform experience level. The first thing we need to do is to assign navigation and field level validation tests to the junior-most members of the team. The number of tests will be more, but they are relatively simpler to execute. For every screen, let them do the same, and in one or  two rounds they will get familiarized with the app.

Assign the form level tests to mid-level experienced testers. This will involve submitting a variety of data thru forms and trying add, modify, delete, view test cases. 

Assign business flow tests to seniors and leads. They need to wear the cap of the end users and move data across screens to achieve the business flow. These people are the best bet to test the reports. The senior people must start feeling the semantics behind the data that flow across the application and understand the business value of the flows. If this is not done, end user testing will always find finer problems and that may lead to escalation.

Always people feel happy to execute new tests (progression tests, for new features added or for change requests). But by principle, a fix or new feature introduction may create issues at some other place. So we try to cover those areas by doing regression tests. The first set of regression tests must be performed for the form in which  the new features/changes are introduced. The next set of regression tests must be done at the module to which the form belongs. The next set of regression tests must address the path in which the feature or change is introduced. Once these are covered, then only do other regression tests. This will ensure the the impact of the changes are minimal.

For every round of testing, reallocate people. Doing the same test again and again will make them slow. You cannot reallocate a person to a totally unrelated module. Every application has a flow of information from one module to another; e.g. request spare item in spares module, cater the service in service module, raise invoice in accounts module. If tester A has done testing of round 1 for spares module, for round 2, assign tester A to service module and in the 3rd round assign to accounts module. This way, the tester also logically moves to the next step rather than landing on a dark area.

Last but not the least. Ask every tester to stand up and do a simple neck and shoulder exercise for 20-30 seconds every hour. Remember, you need good blood flow towards head. When you are too attentive for hours, doing testing, with eyes locked to the monitor, you will develop neck and shoulder pains. You must be physically good to perform better. So take care of your body well.

For free learning courses, please visit http://www.openmentor.net

The first hour of testing

When a person suffers from heart attack or stroke, the first one hour is crucial. If we can treat the person in that first hour, there is about 90% chance that the heart or brain reacts positively to the medicines. This is true in testing also. The first one hour of testing can bring out a great insight about the health of the application being tested. To achieve best results, we have to execute the tests in a structured manner. Here is how.

At this point we assume that the test bed is ready and the application is already installed and ready to be tested. We may have 100s or even 1000s of tests documented and reviewed. Before test execution, ask this question on every test case - "If this test fails, is there any point to continue testing?". Mark all those test cases for which you get an answer "No". That means, those are the vital test cases that must necessarily work or pass. Let us take an example of testing a car. To test a car, we may have to do 1000s of tests; but if ignition does not work, if door does not open, if clutch does not work, if brakes do not work, if accelerator or gas pedal does not work, if steering does not work, is it worth saying that as a car? In your software testing, identify those test cases before test execution.

In the first 5 minutes, navigate to all screens in the application. Simply check the loading of forms and the look and feel. End users do care about screens more and more nowadays.

After this, in every form, if there is a provision to add or create new records, perform those. If no data is added, no data can be viewed; hence do add data. Add with valid data. negative data conditions can be done a little later. Ensure all these tests work well. In every screen try to view all existing data. If you see missing data on screen, we have a problem.

Then, execute critical workflows with valid data. One of the flows in a retail banking scenario may be, create an account, perform a few deposit transactions, perform a few withdrawal transactions, view account balance and transfer some amount from checking to savings account. The flow touches a set of screens with connected data. Usually every application will have top-10 workflows or top-20 workflows. Perform those meticulously. 

Generate a set of key reports and charts that the management people would view on the application on a daily basis. These are the heartbeats of the application to the customer's management; one wrong data in that, the management will lose the confidence in the product.

If we can pack these in the first 1 hour of testing, we can definitely see whether the application is ready to do what it is supposed to do, i.e. it works for positive data. All other invalid data validations, invalid flow tests. non-core features can be tested after the first hour. We call this as BVT (Build Verification Tests). No failure can happen in BVT. Only when all these tests pass, we ask our testing teams to proceed further. You may still ask a question, brakes in car not working does not mean that the horn may not work. I agree. But, if a critical part or path is not working, there is no point in testing less critical part. So let us not waste time and this will ensure that the build or dev teams to ensure they pay proper attention to the paths of the application according to their importance. 

Usually 10-15% of the total tests will fall under BVT category in every application. Attack those first.

For free learning content, visit us at http://www.openmentor.net

Test environment (test bed) setup - some important points

The environment on which we carry out testing, is the very first challenge for every testing team. My mentor always says, if our hands are not clean before taking food, we are prone to illness. The environment (also known as Test Bed) plays a vital part during test execution stage. Due to delivery deadlines, delay in hardware readiness, etc., people hurry up in setting up test bed and at a later point, it becomes the finger-pointing item in any bug triage meetings. Many to most of the times, the application may display a misleading message and people will start troubleshooting in one direction; but the actual problem will be at the test bed level. 

To preempt such issues, certain rules we must follow,, every time we start testing.

Rule 1. Test bed must not have development studio or IDE or other tools, except the product binaries and configuration files. If we have Eclipse or NetBeans in a java product test bed, that itself will resolve many conflicts and the product may tend to work fine. But in reality, in production environment, no dev tools will be present. Hence we must ensure that test bed resembles the production software environment to the extent possible.

Rule 2. Check the versions of OS along with all service packs, patches and hot-fixes.  Many times, people just check the OS version and start executing tests. The service packs and hot fixes do have a lot of compatibility and security related dependencies.

Rule 3. Check the webserver properties. Usually the configuration xml files will have memory settings, connection settings, database connectivity settings etc. Unless they are correctly configured, those may start showing up at different paths of the application.

Rule 4. Check database properties.  Things such as case sensitivity, connections, connection pooling settings, unicode settings etc will have a definite impact on the application behavior.

Rule 5. Check the free disk space, memory space and log file space. These items may not immediately cause issues in functional testing, but do have an effect on performance of the system. 

Rule 6. Check the system users who will run the application. This is mostly for rich-client applications. But if the application login is obtained from the LDAP settings, for all applications, the security settings of the logged-in user is key for carrying out certain activities at application and system levels.


Rule 7. Check the background processes/services. These services must be brought up, before others access the application. See if these are up and running without any exceptions.  Adequate configurations must be done to restart these if these fail to run during test execution.

Rule 8. Check the connectivity of the system with server and other machines. Try to ping or connect to the machines that need connectivity. Check the proxies and firewalls as well.

Rule 9. Check your antivirus settings. Bad settings may prevent blocking of requests or unwanted quarantine effects on your application data. This may cause connection or even timeout issues.

Rule 10. Check your browser version  and settings. Especially when you have auto upgrade option ON in your browser, you may not be able to track what happened to your browser since the last access, and silently an upgrade might have happened. That latest version may not be in scope of testing at all.

For  free learning content, visit us at http://www.openmentor.net