Friday, April 19, 2013

Test Automation - Object Identification

There are about 30 students in a class; you need to find out whether the person you want to talk to, is there or not; assume you have not seen him. You first go and ask "Is John here? I have a parcel to deliver". So, you use the first name to identify a person. If there are 2 Johns in the same class, then you will ask "John Bosco". So you use last name also as additional qualifier to uniquely identify that person, among 30 other students. As a rare case, if both are John Bosco, what will you do? You may say "John Bosco, son of  David P Bosco". The goal is to deliver the parcel to the right person. Testing tools also work on very similar way to carry out click or type actions, on the exact objects on the application UI.

If one has a very good grasp on object identification concepts, 80% of the automation problems are already solved. An object means an item on the user interface, on which you either click or type or you read it and infer some information; e.g. first name text box, state combo box, save button, etc. Recording feature in every tool, internally does this object identification. Most of the tools use a way of storing the details of objects in a separate file. In HP QTP it is called Object Repository, in IBM Rational Functional Tester it is called Object Map, in Borland SilkTest it is called frames.inc, and so on. 

Every object has a type, aka class. The most common types are text box, check button, radio button, combo box, push button, menu bar, tool bar, label, link, image, grid (rows, columns, cell), scroll bar, status bar, progress bar. These are the typical items we see in every application. But the exact way, every technology calls these classes will be different; for example, native Windows C++ will say Push Button, Java will say JButton, Turbo C++ may say TButton, for the same push button you see on screen. This specific text is called nativeclass.

They also have a name associated with them. This is not mandatory, but preferable. The objects may display a name on screen for the user to read, but it may internally use another name. This is like the nick name we use for people. Tools understand the internal name of the objects, rather than the display name of the objects. Ideally developers of the screens must use same, meaningful names for display as well as internal purposes. 

The best way to identify an object is to use the type/class of the object and its internal name. e.g. button/OK, combobox/States, textbox/ZIP code etc. Sometimes, the same display name may appear 2 times on the UI. Consider a screen that takes your personal and office addresses; both will have street/city/state text boxes. So the display name is same; the humans understand this well; but for the tools to understand two objects with same display name, it is better to use the internal name given by developers. 

When we need additional details to uniquely identify an object, there are other properties such as location (x, y coordinates), index of the object (nth object from top of the screen) are usually used. But position based identification is a risky option; this may fail when screen is updated with new objects or a few objects are removed, that may affect the relative position of this field on the screen. 

To understand the objects and their properties, tools provide object spy or object inspector utilities. Before you start automating, spend a few hours to see the class, name and other properties of every object on every single screen. If you see objects with confusing properties or non-unique properties, negotiate with development team to have consistent naming of objects.

Remember: Proper naming of objects reduces the time to automate significantly. This simple process discipline is required by the dev team to make test automation is easy.

For a live demo of this, please click on this url.

Friday, April 5, 2013

Test Automation - Record and Replay

"I need a tester who tests my app very fast!", yells the test manager. This is the same tone we hear from every other test manager. When there are tight deadlines, 100s of customers using the product, one small change needs 3000 test cases to be executed. High time you start automating your test cases. Automate or lose your people and customers - it is your call now.

Take 1 minute to understand how automation works. Take electric fan. Before fan was designed, we used our hands and a fan made of palm leaves; our hand will swing that fan and you can feel the air breeze on your face. To feel that breeze, there is a physical movement. The electric fan automates that. It has the long wings, acting as the palm leaf fan; it rotates and that physical motion pushes air towards you. Well, how can we achieve that in software testing? Instead of me, a tool doing my physical actions such as click, type, drag and drop etc.? 

All tools use this record and replay feature. A microphone records sound waves and speaker reproduces the same; camera records light waves and screen reproduces the same. Same way, inside the test automation tool, there is a recording mechanism. This recording mechanism uses win32/64 API events. When you click on a button on some screen, a win api is generated and is broadcast across the system. That api will tell a single click, left button on mouse has happened on screen X on button with label as "Submit". Our room is full of radio waves, TV channel waves, mobile phone waves; yet we cannot feel or see or hear those? Why, because our eyes and ears are not tuned to detect those. The moment we have a radio and tune the frequency, we hear. The radio has a mechanism to detect and convert those waves to a humanly audible form. Same way, the testing tool has a mechanism to "listen" to those apis and tools produce script. 

The script lines will have the details such as Window name, screen object name, event, event details. e.g.

Notepad - myfile.txt, Edit-Find Menu, Click, left button/single
Find, Find text:, Type, Hello

The script is readable and understandable. When this script is replayed, the tool produces instructions to the operating system, via the reverse win apis, to perform the same action. The application will receive a click event on the object and it will respond to the same. If these apis are not exposed or suppressed, none of the tools will work.

Record and replay are the powerful features of every tool. Do not underestimate the same. The level 1 automation can be quickly achieved by recording every test case and replaying them one after the other. Record once and you can reply any number of times, without spending your time and energy. If you have 100 test cases, record all of those and replay those. While the tool is replaying, manually oversee the screens for any issues. This itself will save at least 30-40% of your testing time.

To see the record and replay (also known as capture-playback), click here to see the demo video.






Monday, February 25, 2013

Test Automation Basics



One of the standard issues every test manager faces, is to have a proof of what is tested and what is not. If I have 300- 400 test cases, I am in full control. But if I have 5000 odd test cases how do I know that the 3617th test case is executed or not ? I trust my testers. Imagine I need to test 200 test cases in 3 different browsers, it multiplies my effort. I can not have 3 testers to do it in three different browsers, nevertheless it must be done. When I need some testing to be done on some critical builds, I do not know, on that day my tester gets illness. He does not turn up to work! My client is waiting to get the status. Oh, what a mess!

4 to 6 weeks from the day of first test execution cycle, my testers get bored of the test cases. Their eyes are not as sharp as before. They feel tired. But they want salary revision! One of my testers claim that he has done 80 test cases since morning I am more than 100% sure that he could not have done that much. How can I be assured of some one did it or did not do it?

The one best answer is automation. Instead of manually executing the test cases, do the testing using a tool. This can solve all the above mentioned problems. Tool never gets tired, tools never get bored, they do not ask salary revisions, they are fast, they do not apply leave and they are consistent!

Before doing any automation test, we must carry out a small proof of concept (POC) or a feasibility study of automation tool on our application. This may take from 4 hours to 8 hours. But this can solve a lot of issues that will be faced  by the team at a later time.

When it comes to test automation, a tester becomes a developer of automated test scripts. This means, the tester generates code using the tool to test the application. There are a variety of tools available in the market. QTP by HP, SilkTest by Borland, Rational Functional Tester by IBM, TestComplete by SmartBear, Selenium, Ranorex etc. are a few to name. Some tools work only on browser based web apps, some work on only rich/thick clients app and some work on both. But all these tools use the UI of the application to run tests. The human tester uses the UI to carry out functional tests and these tools do the same. Instead of human doing a click, the tool clicks on a button, instead of human typing, the tool mimics the key strokes.  

The following are the most common features that almost all tools share.
  1. Recording (Capture test steps)
  2. Replaying  (Playback test steps)
  3. Object Identification (knowing the forms and fields attributes on screen)
  4. Data Driven Test (use same steps with different data sets)
  5. Check points (Verification points, compare the actual results to expected results)
  6. Scripting (use a programming language to add intelligence to test scripts)
  7. File and Database handling (if results are stored on disk)
  8. Exception handling (recovery path when test script itself fails)
You need to think of test automation if you say Yes to one or more of the following points.
  1. The number of test cases for my product is large and I have many regression rounds
  2. My application is a product and not just a 4 months project
  3. My product needs to be tested on multiple environments for compatibility
  4. My product is being used by 1000s of customers and we cannot have a single regression issue
  5. I test my product very frequently, almost everyday
  6. My team costs me more and more and project bleeds for profitability
In the coming sections, we will discuss more about each of the automation features, in detail.

For free automation courses, visit http://www.openmentor.net.
 

Tuesday, February 12, 2013

Compatibility Testing

What shapes a person - nature or nurture? All said and done, the environment in which a person is nurtured, has a tremendous impact on the person's nature, IQ and all other aspects. Software is no exception to this. The environment in which it runs, determines the behavior of the software. Let us see the list of items that affect a software from the environment angle. We can broadly divide this into client side compatibility and server side compatibility.

Let us first take the client side compatibility. This is where the end customer sees your product. The Operating System. This is the ultimate controller of a physical computer. Assume that we develop a product in Windows XP, compile it in Windows XP and test it in Windows XP. There is a very high probability that it will work fine. When we install the same product in Windows Vista or Windows 7 or Windows 8, what is the guarantee it will work the same way it did in Win XP? Absolutely no guarantee. But, we cannot predict the customer's operating environment. So, it is our responsibility to test the product in different versions of the same OS family. Usually, we need to test the product in the latest, latest - 1, latest - 2 versions of the OS.

If you carefully look at the operating system releases, you will notice service packs (SP), and HotFixes (HF). These are patches applied to the OS itself. If I had tested the product in OS + SP1, and if there is a new SP2 release, in order to ensure the quality of the product in that environment, we need to test our product again in OS + SP2 combination. 

With internet penetration to every nook and corner, browser war is always on. The top companies compete to gain browser share. Hence browser is the primary interface to end users. If a product is tested in Internet Explorer (IE), if customers prefer Firefox (FF) or Chrome, then it is mandatory to test the product across different browsers. The way each browser handles html/xml/json etc., the rendering may differ slightly. But when the alignment or rendering gets affected, the end user experience differs. 

Now, to put a bigger bomb, what about testing the product in WinXP and IE 8, Win7 and IE 9, Win8 and Firefox 12 combination? The OS-Browser pair along with OS-Browser versions, the validation matrix will be huge. Do we need to worry about this large matrix? If your end customer uses one such combination, and if that person spends 1000s of dollars on your eCom portal, will you not take effort to make it work? The more combinations you test, the more market share you can gain, as it covers a large customer base.

With lot of different PCs, laptops, models coming up with different screen size and resolutions, consumer facing application such as  online shopping portal etc., we need to test with different screen resolutions. A simple vertical or horizontal scroll bar may irritate the end user. That one small dislike may make the user leave your product.

The compatibility tests are large. You execute the same functional test cases, but on different environments. This takes more time and needs more people. But now, we have a lot of tools on cloud to automatically setup such combinations of environments. This will help you to test faster. You can see browserstack.com as an example. There are many providers like this, it is up to you to make the choice, that suits you better.

For video lessons, please visit www.openmentor.net.


 

License To Crack - A blog-book on Software Testing

Hello,

Welcome to the world of software testing. This page contains a series of links that will help you understand software testing, step by step. Along with this text content, you can use www.openmentor.net videos to learn software testing. 

Enjoy learning.

Functional Testing

  1. License To Crack - Introduction to Testing
  2. SDLC - First 3 Critical Phases 
  3. SDLC - Coding, Testing, Implementation, Maintenance
  4. Five angles to look at everything 
  5. Top 100 Test Scenarios for Online Shopping Apps 
  6. Writing Test Cases 
  7. Top 100 Test Scenarios – Inventory Management
  8. Boundary Value Analysis (BVA) 
  9. Equivalence Partitions 
  10. The CRUD Approach 
  11. Review before you execute tests 
  12. Test environment (test bed) setup
  13. The first hour of testing 
  14. Test Execution - How can we do faster? 
  15. Tester's friend - The BUG 
  16. Bug Life Cycle 
  17. Regression Testing 
  18. How to measure and analyze the testing efficiency?  
  19. What does one need to get job? 
  20. Compatibility Testing 
Test Automation

  1. Test Automation Basics 
  2. Record and Replay 
  3. Object Identification 
  4. Data Driven Test 
  5. Checkpoints 
  6. File/DB Checkpoints 
  7. Scripting Essentials 
  8. Exception Handling 
 
 


Wednesday, January 16, 2013

How to measure and analyze the testing efficiency?

Measurements or Metrics or Stats are the common terms you would hear in every management meeting. What exactly does that mean? A bunch of numbers? Hyped up colorful graphs? Why should I bother about those? Well. Right from pre-KG education to K12 to college, we are so used to something called "marks". These are a set of numbers that reflect the academic capabilities of students (note - marks are not the real indicators of IQ levels). These marks in turn help academic institutions to select candidates, along with other parameters. In the same way, if one wants to measure the extent of testing, what are the numbers one must look at?

Fundamental rule. Do not complicate the numbers. The more you complicate, the more will be the confusion. We need to start with some basic numbers that reflect speed of testing, coverage of testing, efficiency of testing. If all these indicators move up, we can definitely be confident that the testing efficiency is getting better and better.

Every number collected from projects, help to fine tune the processes in that project and across the company itself. 

Test planning rate (TPR). TPR = Total number of test cases planned / total person-hours spent on planning. This number indicates how fast the testing team thinks, articulates the tests and documents the tests.

Test execution rate (TER). TER = Total number of test cases executed / total person-hours spent on execution. This indicates the speed of testers in executing the same. 

Requirements coverage (RC). Ideal goal is 100% coverage. But it is very tough to say how many test cases will cover 100% of requirements. But there is a simple range you mus assume. If we test each requirement in just 2 different ways - 1 positive and 1 negative, we need 2N number of test cases, where N is the number of distinct requirements. On an average, most of the commercial app requirements can be done with 8N test cases. So, the chances of achieving 100% coverage is high if you try to test every requirement in 8 different ways. Not all requirements may need an eight-way approach.

Planning Miss (PM).  PM = Number of adhoc test cases that are framed at the time of execution / Number of test cases planned before execution. This indicates, whether the testers are able to plan the tests based on the documentation and understanding levels. This number must be as less as possible, but it is very difficult to achieve zero level in this.

Bug Dispute Rate (BDR). BDR = Number of bugs rejected by development team / Number of total bugs posted by testing team. A high number here leads to unwanted arguments between the two teams.

There is a set of metrics that reflect the efficiency of the development team, based on the bugs found by the testing team. Those metrics do not really reflect the efficiency of the testing team; but without testing team, those metrics cannot be calculated. Here are a few of those.

Bug Fix Rate (BFR). BFR = Total number of hours spent on fixing bugs / total number of bugs fixed by dev team. This indicates the speed of developers  in fixing the bugs.

Number of re-opened bugs. This absolute number is an indicator of how many potential bad-fixes or regression effects are injected into the application, by the development team. Ideal goal is zero for this.

Bug Bounce Chart (BBC). BBC is not just a number, but a line chart. On the X axis, we need to plot the build numbers in sequence. Y axis contains how many New+ReOpen bugs are found in each build. Ideally this graph must keep dropping towards zero, as quickly as possible. But if we see a swinging pattern, like sinusoidal wave, it indicates, new bugs are getting injected build over build, due to regression effects. After code-freeze, product companies must keep a keen watch on this chart.

For a free test management tool, visit http://www.qamonitor.net

For free recorded training, visit http://www.openmentor.net. Also available on http://www.youtube.com/freeopenmentor








Sunday, January 6, 2013

Regression Testing

Can someone tell whether the last bug is removed from the product? Can someone give 100.00% guarantee that the testing is fully done? Yes and No. Given the time, it can be done; but then what is the timeframe one needs to get to this level? Karate Schools believe in repeating certain exercises to attain perfection. Practice leads to perfection. Practice is nothing but doing it again and again, and ever improving. This holds good for software testing also.

When we walk thru the regular SDLC, the system testing happens once the product is given to the QA team. A set of bugs are published and those bugs get fixed by the development team. What next? Again testing  happens. The question is , how much testing must happen again? Well, the ideal goal is to do all tests on the product once again. This will ensure that a fix in one place does not break something else. But, do we have people, time and money to do all tests again? The following principles are used in most of the organization when it comes to repeating tests. 

Whatever tests we did in build 1, that have passed in build 1, must not fail in build 2. This is fundamental. So ideally we must execute all test cases in build 1 and 2. Assume that there are 500 tests and we executed all of them in build 1 and around 60 test cases failed. When build 2 comes, the first priority goes to checking those failed test cases for the correctness of the fix. This is called re-testing. That means, a test case failed and it was fixed and we execute that test case again. Forgetting all other tests, we check this test case alone; this is re-testing. To find out whether there are any side effects due to the fix, we must do regression tests. Regression testing means, to re-test the test cases that have passed in previous build also. Every time you go to the doctor, the doctor checks the pulse, temperature etc. That part is regression. Same way we must repeat passed tests also, to ensure functional stability.

But how much regression is good? A few simple principles you may try. These are some generic thumb-rules used by most of the companies.
  • If a test case fails for a field, repeat all test cases for that field.
  • If a test case fails for a screen, repeat all test cases for that screen.
  • If a test case fails for a data entity (e.g. purchase order, Invoice etc.), repeat all test cases for that data entity wherever that is used.
  • If a test case fails for a functional module, repeat all test cases for that module.
You are the ultimate authority to decide which suits better for your product within the timeframe you have got. If you are able to cover up a larger area of the application, around the failed test cases, your chances of blocking all loopholes is high.

When it comes to large products such as ERP, when the product reaches the system testing phase, the work of developer may be just 10 minutes to fix a bug; but the testing will go on for days, due to regression. Hence, testing takes a huge effort build after build. You can ask me whether companies are doing proper regression testing. The answer I can give is that those companies that do proper regression have stable products  and larger customer base; this is due to the quality of the product being tested again and again.

For a free test management tool, visit http://www.qamonitor.net

For free recorded training, visit http://www.openmentor.net