Test Automation - Checkpoints

A test must end with a pass or fail status. Humans execute test steps, see the actual results in front of their eyes, compare that with expected results and declare the pass/fail status. Using record/replay/data, test automation tool executes the test steps. But the tool does not know expected results. We need to tell the tool about what is expected and the tool can then declare the results. This is implemented using checkpoint or verification point feature, in every tool. 

Tester feeds data on UI objects such as text box, combo box, radio button etc. When the form is submitted, usually thru a button click to save or update or delete, the application does the operation and posts a message back on the screen. The result is usually a text message such as "Your ticket is successfully booked. Thanks for choosing us.". The tester will then state that the booking test case has passed. If this message does not come, it is treated as failed. Usually the messages will be displayed in specific places of the screen or status bar or the screen may refresh with the data that is entered with an auto-generated id etc. 

As part of automating test cases, every test case will have steps and will have checkpoints. In the checkpoint, we need to feed what is the object to be looked for results, which property of the object reflects the result and what is the expected result. e.g. look in the status bar, look at the text, the text must be "The transaction is completed successfully.". In some cases, it may not be text, instead we will be looking for the price to be updated in the price text box. When you go to a travel portal, usually the number of passengers will be set to 1 and the price will be for 1 person, say $110. When you change the number of passengers to 2, the price must change to $220. So, based on one event, the expected outcome changes in another field. In this case, we must place a checkpoint on the price text box, for its text property with an expected value $220. 

In some cases, only when I check the "I accept the terms" check button, the continue button will be enabled. Till then it will be disabled. In this, we need to place a check point for the continue button, for the enabled property with the expected outcome as true/on. In an email inbox, when I delete a mail, the number of mails shown at the top will be reduced by one. Like this, the screen will show the actual results in specific places. We must find those places and put appropriate checkpoints. Some organizations will use standardization of their UI elements, such as the width and height of all push buttons must be 90 and 30 pixels respectively; in such case we must place checkpoints for the width and height properties of the buttons.

Every test case will have the standard sequence as follows.

• Test steps to establish pre-requisite state of the application
• Test steps for the test case
• Checkpoint-1
• More test steps for the test case
• Checkpoint-2
• Some more test steps for the test case
• Checkpoint-3
• Test steps to clean-up

When the checkpoint passes, the tool will declare the results in green color and when it fails, the result will be painted in red color. Ideally for every expected result in my test case, I must have a checkpoint command in the test script.

For free lessons on automation tools, visit us at http://www.openmentor.net.

Test Automation - Record and Replay

"I need a tester who tests my app very fast!", yells the test manager. This is the same tone we hear from every other test manager. When there are tight deadlines, 100s of customers using the product, one small change needs 3000 test cases to be executed. High time you start automating your test cases. Automate or lose your people and customers - it is your call now.

Take 1 minute to understand how automation works. Take electric fan. Before fan was designed, we used our hands and a fan made of palm leaves; our hand will swing that fan and you can feel the air breeze on your face. To feel that breeze, there is a physical movement. The electric fan automates that. It has the long wings, acting as the palm leaf fan; it rotates and that physical motion pushes air towards you. Well, how can we achieve that in software testing? Instead of me, a tool doing my physical actions such as click, type, drag and drop etc.? 

All tools use this record and replay feature. A microphone records sound waves and speaker reproduces the same; camera records light waves and screen reproduces the same. Same way, inside the test automation tool, there is a recording mechanism. This recording mechanism uses win32/64 API events. When you click on a button on some screen, a win api is generated and is broadcast across the system. That api will tell a single click, left button on mouse has happened on screen X on button with label as "Submit". Our room is full of radio waves, TV channel waves, mobile phone waves; yet we cannot feel or see or hear those? Why, because our eyes and ears are not tuned to detect those. The moment we have a radio and tune the frequency, we hear. The radio has a mechanism to detect and convert those waves to a humanly audible form. Same way, the testing tool has a mechanism to "listen" to those apis and tools produce script. 

The script lines will have the details such as Window name, screen object name, event, event details. e.g.

Notepad - myfile.txt, Edit-Find Menu, Click, left button/single
Find, Find text:, Type, Hello

The script is readable and understandable. When this script is replayed, the tool produces instructions to the operating system, via the reverse win apis, to perform the same action. The application will receive a click event on the object and it will respond to the same. If these apis are not exposed or suppressed, none of the tools will work.

Record and replay are the powerful features of every tool. Do not underestimate the same. The level 1 automation can be quickly achieved by recording every test case and replaying them one after the other. Record once and you can reply any number of times, without spending your time and energy. If you have 100 test cases, record all of those and replay those. While the tool is replaying, manually oversee the screens for any issues. This itself will save at least 30-40% of your testing time.

To see the record and replay (also known as capture-playback), click here to see the demo video.

Test Automation Basics

One of the standard issues every test manager faces, is to have a proof of what is tested and what is not. If I have 300- 400 test cases, I am in full control. But if I have 5000 odd test cases how do I know that the 3617^th test case is executed or not ? I trust my testers. Imagine I need to test 200 test cases in 3 different browsers, it multiplies my effort. I can not have 3 testers to do it in three different browsers, nevertheless it must be done. When I need some testing to be done on some critical builds, I do not know, on that day my tester gets illness. He does not turn up to work! My client is waiting to get the status. Oh, what a mess!

4 to 6 weeks from the day of first test execution cycle, my testers get bored of the test cases. Their eyes are not as sharp as before. They feel tired. But they want salary revision! One of my testers claim that he has done 80 test cases since morning I am more than 100% sure that he could not have done that much. How can I be assured of some one did it or did not do it?

The one best answer is automation. Instead of manually executing the test cases, do the testing using a tool. This can solve all the above mentioned problems. Tool never gets tired, tools never get bored, they do not ask salary revisions, they are fast, they do not apply leave and they are consistent!

Before doing any automation test, we must carry out a small proof of concept (POC) or a feasibility study of automation tool on our application. This may take from 4 hours to 8 hours. But this can solve a lot of issues that will be faced  by the team at a later time.

When it comes to test automation, a tester becomes a developer of automated test scripts. This means, the tester generates code using the tool to test the application. There are a variety of tools available in the market. QTP by HP, SilkTest by Borland, Rational Functional Tester by IBM, TestComplete by SmartBear, Selenium, Ranorex etc. are a few to name. Some tools work only on browser based web apps, some work on only rich/thick clients app and some work on both. But all these tools use the UI of the application to run tests. The human tester uses the UI to carry out functional tests and these tools do the same. Instead of human doing a click, the tool clicks on a button, instead of human typing, the tool mimics the key strokes.  

The following are the most common features that almost all tools share.

1. Recording (Capture test steps)
2. Replaying  (Playback test steps)
3. Object Identification (knowing the forms and fields attributes on screen)
4. Data Driven Test (use same steps with different data sets)
5. Check points (Verification points, compare the actual results to expected results)
6. Scripting (use a programming language to add intelligence to test scripts)
7. File and Database handling (if results are stored on disk)
8. Exception handling (recovery path when test script itself fails)

You need to think of test automation if you say Yes to one or more of the following points.

1. The number of test cases for my product is large and I have many regression rounds
2. My application is a product and not just a 4 months project
3. My product needs to be tested on multiple environments for compatibility
4. My product is being used by 1000s of customers and we cannot have a single regression issue
5. I test my product very frequently, almost everyday
6. My team costs me more and more and project bleeds for profitability

In the coming sections, we will discuss more about each of the automation features, in detail.

For free automation courses, visit http://www.openmentor.net.
 

How to measure and analyze the testing efficiency?

Measurements or Metrics or Stats are the common terms you would hear in every management meeting. What exactly does that mean? A bunch of numbers? Hyped up colorful graphs? Why should I bother about those? Well. Right from pre-KG education to K12 to college, we are so used to something called "marks". These are a set of numbers that reflect the academic capabilities of students (note - marks are not the real indicators of IQ levels). These marks in turn help academic institutions to select candidates, along with other parameters. In the same way, if one wants to measure the extent of testing, what are the numbers one must look at?

Fundamental rule. Do not complicate the numbers. The more you complicate, the more will be the confusion. We need to start with some basic numbers that reflect speed of testing, coverage of testing, efficiency of testing. If all these indicators move up, we can definitely be confident that the testing efficiency is getting better and better.

Every number collected from projects, help to fine tune the processes in that project and across the company itself. 

Test planning rate (TPR). TPR = Total number of test cases planned / total person-hours spent on planning. This number indicates how fast the testing team thinks, articulates the tests and documents the tests.

Test execution rate (TER). TER = Total number of test cases executed / total person-hours spent on execution. This indicates the speed of testers in executing the same. 

Requirements coverage (RC). Ideal goal is 100% coverage. But it is very tough to say how many test cases will cover 100% of requirements. But there is a simple range you mus assume. If we test each requirement in just 2 different ways - 1 positive and 1 negative, we need 2N number of test cases, where N is the number of distinct requirements. On an average, most of the commercial app requirements can be done with 8N test cases. So, the chances of achieving 100% coverage is high if you try to test every requirement in 8 different ways. Not all requirements may need an eight-way approach.

Planning Miss (PM).  PM = Number of adhoc test cases that are framed at the time of execution / Number of test cases planned before execution. This indicates, whether the testers are able to plan the tests based on the documentation and understanding levels. This number must be as less as possible, but it is very difficult to achieve zero level in this.

Bug Dispute Rate (BDR). BDR = Number of bugs rejected by development team / Number of total bugs posted by testing team. A high number here leads to unwanted arguments between the two teams.

There is a set of metrics that reflect the efficiency of the development team, based on the bugs found by the testing team. Those metrics do not really reflect the efficiency of the testing team; but without testing team, those metrics cannot be calculated. Here are a few of those.

Bug Fix Rate (BFR). BFR = Total number of hours spent on fixing bugs / total number of bugs fixed by dev team. This indicates the speed of developers  in fixing the bugs.

Number of re-opened bugs. This absolute number is an indicator of how many potential bad-fixes or regression effects are injected into the application, by the development team. Ideal goal is zero for this.

Bug Bounce Chart (BBC). BBC is not just a number, but a line chart. On the X axis, we need to plot the build numbers in sequence. Y axis contains how many New+ReOpen bugs are found in each build. Ideally this graph must keep dropping towards zero, as quickly as possible. But if we see a swinging pattern, like sinusoidal wave, it indicates, new bugs are getting injected build over build, due to regression effects. After code-freeze, product companies must keep a keen watch on this chart.

For a free test management tool, visit http://www.qamonitor.net

For free recorded training, visit http://www.openmentor.net. Also available on http://www.youtube.com/freeopenmentor

Regression Testing

Can someone tell whether the last bug is removed from the product? Can someone give 100.00% guarantee that the testing is fully done? Yes and No. Given the time, it can be done; but then what is the timeframe one needs to get to this level? Karate Schools believe in repeating certain exercises to attain perfection. Practice leads to perfection. Practice is nothing but doing it again and again, and ever improving. This holds good for software testing also.

When we walk thru the regular SDLC, the system testing happens once the product is given to the QA team. A set of bugs are published and those bugs get fixed by the development team. What next? Again testing  happens. The question is , how much testing must happen again? Well, the ideal goal is to do all tests on the product once again. This will ensure that a fix in one place does not break something else. But, do we have people, time and money to do all tests again? The following principles are used in most of the organization when it comes to repeating tests. 

Whatever tests we did in build 1, that have passed in build 1, must not fail in build 2. This is fundamental. So ideally we must execute all test cases in build 1 and 2. Assume that there are 500 tests and we executed all of them in build 1 and around 60 test cases failed. When build 2 comes, the first priority goes to checking those failed test cases for the correctness of the fix. This is called re-testing. That means, a test case failed and it was fixed and we execute that test case again. Forgetting all other tests, we check this test case alone; this is re-testing. To find out whether there are any side effects due to the fix, we must do regression tests. Regression testing means, to re-test the test cases that have passed in previous build also. Every time you go to the doctor, the doctor checks the pulse, temperature etc. That part is regression. Same way we must repeat passed tests also, to ensure functional stability.

But how much regression is good? A few simple principles you may try. These are some generic thumb-rules used by most of the companies.

• If a test case fails for a field, repeat all test cases for that field.
• If a test case fails for a screen, repeat all test cases for that screen.
• If a test case fails for a data entity (e.g. purchase order, Invoice etc.), repeat all test cases for that data entity wherever that is used.
• If a test case fails for a functional module, repeat all test cases for that module.

You are the ultimate authority to decide which suits better for your product within the timeframe you have got. If you are able to cover up a larger area of the application, around the failed test cases, your chances of blocking all loopholes is high.

When it comes to large products such as ERP, when the product reaches the system testing phase, the work of developer may be just 10 minutes to fix a bug; but the testing will go on for days, due to regression. Hence, testing takes a huge effort build after build. You can ask me whether companies are doing proper regression testing. The answer I can give is that those companies that do proper regression have stable products  and larger customer base; this is due to the quality of the product being tested again and again.

For a free test management tool, visit http://www.qamonitor.net

For free recorded training, visit http://www.openmentor.net

 

Tester's friend - The BUG

Every normal human being gets pleasure while finding mistakes in others, sad, but true. Tester is also a human being and when he/she finds a bug, it further motivates him/her to find more. A tester aims to fail the software in any way possible, and tester succeeds when bugs do appear. I do not worry about the theoretical definition of bug; to me, anything that the software does in a manner that is not intended as per specs or by implied needs, is a bug. Hence bug, error, problem, issue, failure etc. all come under the same category, though you can find zillion definitions for these words if you google.

When a bug appears while you test, do the test again. Ensure it appears again. This will prove that, you as a tester, had not done any mistake in executing the test. Soon after confirming its consistent nature, do not call the developer. Developer has a tendency to reject or dispute. Do not send a mail either. You need to ask a few questions to yourselves before you formally state that it is a bug. 

Try to describe the bug in 1 line. This is like writing the email subject. "User does not get confirmation email after registration", "Application accepts invalid SSN in Add Customer Page". Try to convey the exact problem with minimal words.

Mention the screen name where the bug appears. This will help the developer directly to go to that place, without wasting time. If possible, take a screenshot where the bug appears. A picture means more than 1000 words.

Explain the bug in detail, by telling the steps to reproduce. You get bugs when you execute a test case. Thus, the bug must be appearing against one of the test steps. Copy paste the test case itself and mention the exact problem against the test step. If the developer does the same thing, he/she should get the same bug.

How does this bug affect the product? The answer to this question will help you to set the Severity of the bug. Are you able to proceed further to do other test cases? If not, this bug is termed as a Show Stopper or Blocker. This means, you have no way to test anymore unless development team fixes the same. 

Is there a crash or data loss or a feature is missing in the product? This falls under High or Major category, provided you are able to do other tests. This means, that the product has a serious problem. For example, when we page customer details, Save operation itself does now work and hence the data entered is lost. Technically a developer may say that he can fix this in 2 minutes, but always look from customer's angle. For the customer, data is more important than the product itself. Any loss of data is a major bug. The severity code Major is also termed as High in many companies.


Bug does not stop me from testing, there is no data loss or crash, but the functionality is wrongly implemented. The app saves data, but some part of the results are not accurate. In this case, companies usually treat this as Minor or Medium severity.

The app does the job, but there are spelling mistakes or improper messages to users or imporper field alignments in the UI. Usually companies treat this as Cosmetic or Low severity.

To sum up, if we can write the following, then we are good in reporting bugs to dev team.

• One line description of bug
• Details to reproduce the bug
• Screen name where the bug appears
• Screenshot of the bug
• Severity of the bug

By principle, every problem must be reported and tracked. We will see tracking mechanism in the next post. For more details on bugs, please see this bug tracking demo.

SDLC - Coding, Testing, Implementation, Maintenance Phases

Coding Phase

The LLD document contains all the details for a programmer to convert it into a working program. Designed in the right way, the job of the developer is to convert the pseudo code into a programming language in the specified platform. The developer will be given specific guidelines that tell how to code. This guideline includes the naming convention of procedures, variables, commenting methods etc. As per the guidelines, the developer codes and at the end of the coding, he/she compiles the program unit.

By compiling and correcting the errors, all syntax error and removed. Syntax errors are the easy ones to rectify. The logical errors in the programs are the difficult ones to identify and then rectify. By logical error, we mean, the program behaves in a way, that it is not expected to behave. The role of testers is to identify the logical errors at various stages.

Testing

*This section provides only a shorter version of the testing techniques.

Testing is the process of evaluating a system or application, to check whether the application meets all requirements of the client and to detect the errors.

Generally testing can be classified into static testing and dynamic testing.

Again Dynamic Testing is classified into two types: Structural Testing (or) white box , Functional Testing (or) Black Box testing.

Static Testing

Verification activities fall into the category of Static Testing. Static testing refers to testing something that’s not running. It is examining and reviewing it. i.e., to check whether the work done meets the standards of the organization. Reviews, Inspections and Walk-throughs are static testing methodologies.

Example: The specification is a document and not an executing program. When we read it to find out the issues, it is considered as static testing.

Dynamic Testing

Dynamic Testing involves working with the software, giving input values and checking if the output is as expected. These are the Validation activities. Unit Tests, Integration Tests, System Tests and Acceptance Tests are few of the Dynamic Testing methodologies.

Techniques used are determined by type of testing that must be conducted.

• Functional ("black box") testing.

• Structural (usually called "white box") testing.

Black box testing involves looking at the specifications and does not require examining the code of a program. Tests that examine the observable behavior of software as evidenced by its outputs without referencing to internal functions is black box testing. It is not based on any knowledge of internal design or code and tests are based on requirements and functionality. Nowadays there are automatic code generation tools and code re-use becomes more prevalent, analysis of source code itself becomes less important and functional tests become more important.

Black box testing is easy as it is based on system’s response for a given user’s input. If we know the business functionality of the product, we can do black box testing. On the other hand, white box testing requires programming knowledge to know the internals of the code. Also it is time consuming. So only a developer can become a white box tester.

Unit Testing

Soon after the program is corrected for syntax errors, the program has to be checked for logical errors, at the unit level. Programs may have simple user inputs or outputs thru screens or reports. The inputs must be validated for their format, data type, boundary conditions etc. Also, the elementary functionality of the program must be verified. In most of the cases, the programmers themselves will do this unit testing. Many inputs will be verified in this way and the program unit will be checked for the basic functionality in this phase.

If there are any problems while doing this testing, the programmers will debug the program and the bugs will be fixed and will be tested again for the bug fix. While doing the unit testing, that particular unit need not wait for another unit to be completed. It can be tested independently. Unit testing is also termed as component testing. For example, if we need to test a java class and method, even without any screen, the programmer can test the same. Thus any piece of code, that can be independently executed, can be independently tested as well. This is termed as unit testing.

Integration Testing

When all the individual program units are tested in the unit testing phase and all units are clear of any known bugs, the interfaces between those modules will be tested, to establish that they communicate to each other property via the specified APIs and thus they can be integrated into an application. The integration test may be performed by the independent testers or by the development team members.

System Testing

After all the interfaces are tested between multiple modules, the whole set of software is tested to establish that all modules work together correctly as an application or system or package. This is again performed by independent testes. The testers will have to do the system testing as though they are the end users of the application. Systems testing includes special testing methods like performance testing, interoperability testing, stability testing etc.

Acceptance Testing

After the software product is tested by the SDU in 3 different stages (unit, integration and system), the client will test it, in their place, in a near-real-time or simulated environment. This establishes that the software meets the requirements of the end user. This testing is carried out as though they are using the system in their real office, to mimic the real time scenarios.

Implementation Phase/Support

This Phase will provide users with the documentation and training required to use the system effectively. Data Conversion will only occur once, but user documentation will be required. Deployment of the product will be carried out, on the hardware that is going to be used in production (on live systems). Deployment itself requires careful planning. Once the product is deployed, initial data will be populated, user training will happen.

Release to Production and warranty Period

When the clients to the acceptance testing and finds no problems, then they will accept the software and now they have to start using the software in their real office. It may be a real bank for a banking application. During the acceptance testing also, there may be some bug fixed done to the software. Before the software is put into production (real time environment), the SDU must release the latest versions of the software, which has no known bugs to the client. This may be in a CD or tape or an Internet download. The clients will then install the latest software in their production system and will start using. This is called go-live process.

In the same way every product has a guarantee period software also has a warranty period. Normally it will be a 60-day or a 90-day warranty period for most kinds of software. Depending upon the complexity and size of the software the warranty period may be extended or shortened. During this warranty period, in case of any problems, the SDU has the responsibility to rectify the problem, at no charges. Again, it depends upon the contract, whether to charge for the fix or not and may vary from industry to industry.

Maintenance Phase

After the warranty period, the software enters the maintenance phase. During the maintenance phase, 3 things happen to the software. They are :

• Bug fixing

• Upgrade

• Enhancement

During the maintenance phase, because of some untested scenarios, the software may give errors or logical problems. This has to be fixed. This is same as any other bug fixing. This is similar to any repair works done to a vehicle, by its owner.

In the software field, the software versions are constantly changing and the applications may have to keep themselves upgraded to the newer versions of the software. For example, the anti-virus software must be constantly upgraded to address new viruses, which come very day. If the operating system version charges, it may necessitate to introduce some changes in the programs. So, this kind of upgrade works to the existing software applications, because of operating system or environmental changes or hardware changes may have to be incorporated.

Any software is not all complete and there are enough rooms to add new features to an existing software. For example, today the railways reservation system may not be in the internet; but tomorrow there may be a need to go in the internet way. So, new modules are to be added to the existing system to address this. This is enhancement.

After some time, the software may become obsolete and will reach a point that it cannot be used. At that time, it will be replaced by another software which is superior to that. This is the end of the software. The software will reach its end, when the maintenance cost of the software becomes more than the new software available. Software can become obsolete when the company does not upgrade it to work on new environments such as databases or operating systems. When the software becomes incompatible with the environment, users will move on to another software, dumping the existing one.

For recorded training content, please visit http://www.openmentor.net.