Test Automation - File/DB Checkpoints

Most of the apps take the input from the user thru UI, do some logic, store the data in a file or database, finally show a success or failure message on the UI. Checking the message alone is not good enough; what if the data was truncated or wrongly formatted or wrongly stored or lost? This is why you need to check the data in files and database. Testers can see the results of a test case on the UI or in a file or in database tables. Checking the text on UI is very simple as one can immediately see it in front of the eyes. But Checking the results in a file takes some more time and a few extra steps.

Let us take the file storage first. First of all, the tester must know the folder and filename in which the app stores the results. The file may be a flat file or a csv or a tsv or an XML file. Hence the tester must also know the format of the file. First, the tester must manually enter data thru UI and save the record and see how the file stores the data. Imagine the app stores the employee record like this (one record per one line): emp_code,emp_name,emp_dob,designation,salary,branch,emp_doj. This is a csv format. Here, we do not know the length of the line; it can vary based on the length of the name or other fields. But all I know is that if I give some data in the UI, the same must appear in the file, in the given order. Hence, to check whether the data is present in the file or not, we must know how to open a file, read the records from file, compare the data with expected results and then close the file. All tools provide a fiel interface thru file system object (VBScript) or file input stream (Java). Using these commands, one must read record after record, compare with expected results and close the file.

If the app stores data in XML file, it is a little bit easier. Because, every scripting language provides simple commands to check whether a node (<tag> </tag>) is present, how many child nodes are present and the data embedded within a specific tag. These commands use XML object to store the file data. Tools like HP QTP provide a very good UI also to view the whole XML structure like a tree and we need not even write code to check data in an XML file.

When it comes to checking records in a database, you need to know the tables and their structure. Soon after every add, edit or delete transaction thru the app UI, we must do a select query from the respective table, extract the data and compare whether the same is correct or not. The important point in writing the query is to narrow down the data that is very very specific to the test. Do not grab a lot of records that are not required for the test; for example, if I cancel a ticket with reservation number XB45890, I need to just select the record for that reservation number only; so that I get only the record I want to compare. Try to select records based on some unique key values, rather than a generic where clause.

When you use file or database checkpoints, make sure you close the file or database connection, soon after the checkpoint.

For free lessons on automation tools, visit us at http://www.openmentor.net.

Test Automation - Checkpoints

A test must end with a pass or fail status. Humans execute test steps, see the actual results in front of their eyes, compare that with expected results and declare the pass/fail status. Using record/replay/data, test automation tool executes the test steps. But the tool does not know expected results. We need to tell the tool about what is expected and the tool can then declare the results. This is implemented using checkpoint or verification point feature, in every tool. 

Tester feeds data on UI objects such as text box, combo box, radio button etc. When the form is submitted, usually thru a button click to save or update or delete, the application does the operation and posts a message back on the screen. The result is usually a text message such as "Your ticket is successfully booked. Thanks for choosing us.". The tester will then state that the booking test case has passed. If this message does not come, it is treated as failed. Usually the messages will be displayed in specific places of the screen or status bar or the screen may refresh with the data that is entered with an auto-generated id etc. 

As part of automating test cases, every test case will have steps and will have checkpoints. In the checkpoint, we need to feed what is the object to be looked for results, which property of the object reflects the result and what is the expected result. e.g. look in the status bar, look at the text, the text must be "The transaction is completed successfully.". In some cases, it may not be text, instead we will be looking for the price to be updated in the price text box. When you go to a travel portal, usually the number of passengers will be set to 1 and the price will be for 1 person, say $110. When you change the number of passengers to 2, the price must change to $220. So, based on one event, the expected outcome changes in another field. In this case, we must place a checkpoint on the price text box, for its text property with an expected value $220. 

In some cases, only when I check the "I accept the terms" check button, the continue button will be enabled. Till then it will be disabled. In this, we need to place a check point for the continue button, for the enabled property with the expected outcome as true/on. In an email inbox, when I delete a mail, the number of mails shown at the top will be reduced by one. Like this, the screen will show the actual results in specific places. We must find those places and put appropriate checkpoints. Some organizations will use standardization of their UI elements, such as the width and height of all push buttons must be 90 and 30 pixels respectively; in such case we must place checkpoints for the width and height properties of the buttons.

Every test case will have the standard sequence as follows.

• Test steps to establish pre-requisite state of the application
• Test steps for the test case
• Checkpoint-1
• More test steps for the test case
• Checkpoint-2
• Some more test steps for the test case
• Checkpoint-3
• Test steps to clean-up

When the checkpoint passes, the tool will declare the results in green color and when it fails, the result will be painted in red color. Ideally for every expected result in my test case, I must have a checkpoint command in the test script.

For free lessons on automation tools, visit us at http://www.openmentor.net.

Test Automation - Data Driven Test

You need more data for testing. I would rather re-phrase this to state you need more varieties of data rather than more data itself. Each data set must help the tester to drive the application thru a different logic or path, so that we can do better testing. Take the example of booking airline ticket. It is just 1 screen that takes data and users feed these fields to get their ticket booked. Each ticket may be a different combination. A human tester gets bored when repeatedly seeing and operating on the same screen. But this has to be done any way. 

Here comes the data driven test (DDT) feature of the test automation tools. Test steps are same, but data is different, for each transaction I must use a different data set - If this is your case, use data driven test. Every testing tool comes up with a data sheet or data pool. Here you need to provide data in a spreadsheet like file. For booking a ticket you need from place, to place, one way or round trip, date of journey, return date, number of persons etc. Though there are lot more details for ticket booking, let us limit to these data at this point. 

The first thing for DDT is to create a file that has these data. Usually you can use a csv file or xls file. The first line is usually the title for the data. See the example below.


FROM-PLACE,TO-PLACE,TRIP-TYPE,JOURNEYDT,RETURNDT,NUMPERSONS
Los Angeles,New York,TWOWAY,10-JUN-2013,14-JUN-2013,2

Los Angeles,Denver,ONEWAY,10-JUN-2013,10-JUN-2013,1

Los Angeles,London,TWOWAY,10-JUN-2013,18-AUG-2013,5
...

We need to prepare each line of data that represent some equivalent partition set to test our application. 

Once data is created, we need to record a script using the tool, that does one ticket booking, by entering the data in the above fields. While recording we will give some data and that will reflect in the script. e.g.

BookingScreen.Clear
BookingScreen.FromAirPort.Set "Los Angeles"
BookingScreen.ToAirPort.Set "Denver"
BookingScreen.TripType.Select "TwoWay"
BookingScreen.DateOfJourney.Set "10-May-2013"
BookingScreen.DateOfReturn.Set "12-May-2013"
BookingScreen.NumberOfPersons.Set 1
BookingScreen.Submit

The syntax given above is generic and not specific to any tool. To ensure that the script uses the data from our file, we may have to modify the script manually or use the data driver wizard of the tool. We need to set the value for each UI field data entry, to point to a column in the data sheet. Also, we must loop thru all rows in the data sheet, so that the script executes for all records and books multiple tickets. It may  look like the following.

DataSheet = "C:\\mydata.csv"

For currRow = 1 to DataSheet.RowCount

   DataSheet.CurrentRow = currRow
   BookingScreen.Clear
   BookingScreen.FromAirPort.Set DataSheet.getColumnvalue("FROM-PLACE")
   BookingScreen.ToAirPort.Set DataSheet.getColumnvalue("TO-PLACE")
   BookingScreen.TripType.Select DataSheet.getColumnvalue("TRIP-TYPE")
   BookingScreen.DateOfJourney.Set DataSheet.getColumnvalue("JOURNEYDT")
   BookingScreen.DateOfReturn.Set DataSheet.getColumnvalue("RETURNDT")
   BookingScreen.NumberOfPersons.SetDataSheet.getColumnvalue("NUMPERSONS")
   BookingScreen.Submit
Next

The simple code, will get all records from the csv file and repeat feeding the screen with different data sets from the csv file. This way, a 10 line code can enter 10000 records with little human interference.

For a demo of data driven test, see this video. 

Test Automation - Object Identification

There are about 30 students in a class; you need to find out whether the person you want to talk to, is there or not; assume you have not seen him. You first go and ask "Is John here? I have a parcel to deliver". So, you use the first name to identify a person. If there are 2 Johns in the same class, then you will ask "John Bosco". So you use last name also as additional qualifier to uniquely identify that person, among 30 other students. As a rare case, if both are John Bosco, what will you do? You may say "John Bosco, son of  David P Bosco". The goal is to deliver the parcel to the right person. Testing tools also work on very similar way to carry out click or type actions, on the exact objects on the application UI.

If one has a very good grasp on object identification concepts, 80% of the automation problems are already solved. An object means an item on the user interface, on which you either click or type or you read it and infer some information; e.g. first name text box, state combo box, save button, etc. Recording feature in every tool, internally does this object identification. Most of the tools use a way of storing the details of objects in a separate file. In HP QTP it is called Object Repository, in IBM Rational Functional Tester it is called Object Map, in Borland SilkTest it is called frames.inc, and so on. 

Every object has a type, aka class. The most common types are text box, check button, radio button, combo box, push button, menu bar, tool bar, label, link, image, grid (rows, columns, cell), scroll bar, status bar, progress bar. These are the typical items we see in every application. But the exact way, every technology calls these classes will be different; for example, native Windows C++ will say Push Button, Java will say JButton, Turbo C++ may say TButton, for the same push button you see on screen. This specific text is called nativeclass.

They also have a name associated with them. This is not mandatory, but preferable. The objects may display a name on screen for the user to read, but it may internally use another name. This is like the nick name we use for people. Tools understand the internal name of the objects, rather than the display name of the objects. Ideally developers of the screens must use same, meaningful names for display as well as internal purposes. 

The best way to identify an object is to use the type/class of the object and its internal name. e.g. button/OK, combobox/States, textbox/ZIP code etc. Sometimes, the same display name may appear 2 times on the UI. Consider a screen that takes your personal and office addresses; both will have street/city/state text boxes. So the display name is same; the humans understand this well; but for the tools to understand two objects with same display name, it is better to use the internal name given by developers. 

When we need additional details to uniquely identify an object, there are other properties such as location (x, y coordinates), index of the object (nth object from top of the screen) are usually used. But position based identification is a risky option; this may fail when screen is updated with new objects or a few objects are removed, that may affect the relative position of this field on the screen. 

To understand the objects and their properties, tools provide object spy or object inspector utilities. Before you start automating, spend a few hours to see the class, name and other properties of every object on every single screen. If you see objects with confusing properties or non-unique properties, negotiate with development team to have consistent naming of objects.

Remember: Proper naming of objects reduces the time to automate significantly. This simple process discipline is required by the dev team to make test automation is easy.

For a live demo of this, please click on this url.

Test Automation - Record and Replay

"I need a tester who tests my app very fast!", yells the test manager. This is the same tone we hear from every other test manager. When there are tight deadlines, 100s of customers using the product, one small change needs 3000 test cases to be executed. High time you start automating your test cases. Automate or lose your people and customers - it is your call now.

Take 1 minute to understand how automation works. Take electric fan. Before fan was designed, we used our hands and a fan made of palm leaves; our hand will swing that fan and you can feel the air breeze on your face. To feel that breeze, there is a physical movement. The electric fan automates that. It has the long wings, acting as the palm leaf fan; it rotates and that physical motion pushes air towards you. Well, how can we achieve that in software testing? Instead of me, a tool doing my physical actions such as click, type, drag and drop etc.? 

All tools use this record and replay feature. A microphone records sound waves and speaker reproduces the same; camera records light waves and screen reproduces the same. Same way, inside the test automation tool, there is a recording mechanism. This recording mechanism uses win32/64 API events. When you click on a button on some screen, a win api is generated and is broadcast across the system. That api will tell a single click, left button on mouse has happened on screen X on button with label as "Submit". Our room is full of radio waves, TV channel waves, mobile phone waves; yet we cannot feel or see or hear those? Why, because our eyes and ears are not tuned to detect those. The moment we have a radio and tune the frequency, we hear. The radio has a mechanism to detect and convert those waves to a humanly audible form. Same way, the testing tool has a mechanism to "listen" to those apis and tools produce script. 

The script lines will have the details such as Window name, screen object name, event, event details. e.g.

Notepad - myfile.txt, Edit-Find Menu, Click, left button/single
Find, Find text:, Type, Hello

The script is readable and understandable. When this script is replayed, the tool produces instructions to the operating system, via the reverse win apis, to perform the same action. The application will receive a click event on the object and it will respond to the same. If these apis are not exposed or suppressed, none of the tools will work.

Record and replay are the powerful features of every tool. Do not underestimate the same. The level 1 automation can be quickly achieved by recording every test case and replaying them one after the other. Record once and you can reply any number of times, without spending your time and energy. If you have 100 test cases, record all of those and replay those. While the tool is replaying, manually oversee the screens for any issues. This itself will save at least 30-40% of your testing time.

To see the record and replay (also known as capture-playback), click here to see the demo video.

Test Automation Basics

One of the standard issues every test manager faces, is to have a proof of what is tested and what is not. If I have 300- 400 test cases, I am in full control. But if I have 5000 odd test cases how do I know that the 3617^th test case is executed or not ? I trust my testers. Imagine I need to test 200 test cases in 3 different browsers, it multiplies my effort. I can not have 3 testers to do it in three different browsers, nevertheless it must be done. When I need some testing to be done on some critical builds, I do not know, on that day my tester gets illness. He does not turn up to work! My client is waiting to get the status. Oh, what a mess!

4 to 6 weeks from the day of first test execution cycle, my testers get bored of the test cases. Their eyes are not as sharp as before. They feel tired. But they want salary revision! One of my testers claim that he has done 80 test cases since morning I am more than 100% sure that he could not have done that much. How can I be assured of some one did it or did not do it?

The one best answer is automation. Instead of manually executing the test cases, do the testing using a tool. This can solve all the above mentioned problems. Tool never gets tired, tools never get bored, they do not ask salary revisions, they are fast, they do not apply leave and they are consistent!

Before doing any automation test, we must carry out a small proof of concept (POC) or a feasibility study of automation tool on our application. This may take from 4 hours to 8 hours. But this can solve a lot of issues that will be faced  by the team at a later time.

When it comes to test automation, a tester becomes a developer of automated test scripts. This means, the tester generates code using the tool to test the application. There are a variety of tools available in the market. QTP by HP, SilkTest by Borland, Rational Functional Tester by IBM, TestComplete by SmartBear, Selenium, Ranorex etc. are a few to name. Some tools work only on browser based web apps, some work on only rich/thick clients app and some work on both. But all these tools use the UI of the application to run tests. The human tester uses the UI to carry out functional tests and these tools do the same. Instead of human doing a click, the tool clicks on a button, instead of human typing, the tool mimics the key strokes.  

The following are the most common features that almost all tools share.

1. Recording (Capture test steps)
2. Replaying  (Playback test steps)
3. Object Identification (knowing the forms and fields attributes on screen)
4. Data Driven Test (use same steps with different data sets)
5. Check points (Verification points, compare the actual results to expected results)
6. Scripting (use a programming language to add intelligence to test scripts)
7. File and Database handling (if results are stored on disk)
8. Exception handling (recovery path when test script itself fails)

You need to think of test automation if you say Yes to one or more of the following points.

1. The number of test cases for my product is large and I have many regression rounds
2. My application is a product and not just a 4 months project
3. My product needs to be tested on multiple environments for compatibility
4. My product is being used by 1000s of customers and we cannot have a single regression issue
5. I test my product very frequently, almost everyday
6. My team costs me more and more and project bleeds for profitability

In the coming sections, we will discuss more about each of the automation features, in detail.

For free automation courses, visit http://www.openmentor.net.
 

Compatibility Testing

What shapes a person - nature or nurture? All said and done, the environment in which a person is nurtured, has a tremendous impact on the person's nature, IQ and all other aspects. Software is no exception to this. The environment in which it runs, determines the behavior of the software. Let us see the list of items that affect a software from the environment angle. We can broadly divide this into client side compatibility and server side compatibility.

Let us first take the client side compatibility. This is where the end customer sees your product. The Operating System. This is the ultimate controller of a physical computer. Assume that we develop a product in Windows XP, compile it in Windows XP and test it in Windows XP. There is a very high probability that it will work fine. When we install the same product in Windows Vista or Windows 7 or Windows 8, what is the guarantee it will work the same way it did in Win XP? Absolutely no guarantee. But, we cannot predict the customer's operating environment. So, it is our responsibility to test the product in different versions of the same OS family. Usually, we need to test the product in the latest, latest - 1, latest - 2 versions of the OS.

If you carefully look at the operating system releases, you will notice service packs (SP), and HotFixes (HF). These are patches applied to the OS itself. If I had tested the product in OS + SP1, and if there is a new SP2 release, in order to ensure the quality of the product in that environment, we need to test our product again in OS + SP2 combination. 

With internet penetration to every nook and corner, browser war is always on. The top companies compete to gain browser share. Hence browser is the primary interface to end users. If a product is tested in Internet Explorer (IE), if customers prefer Firefox (FF) or Chrome, then it is mandatory to test the product across different browsers. The way each browser handles html/xml/json etc., the rendering may differ slightly. But when the alignment or rendering gets affected, the end user experience differs. 

Now, to put a bigger bomb, what about testing the product in WinXP and IE 8, Win7 and IE 9, Win8 and Firefox 12 combination? The OS-Browser pair along with OS-Browser versions, the validation matrix will be huge. Do we need to worry about this large matrix? If your end customer uses one such combination, and if that person spends 1000s of dollars on your eCom portal, will you not take effort to make it work? The more combinations you test, the more market share you can gain, as it covers a large customer base.

With lot of different PCs, laptops, models coming up with different screen size and resolutions, consumer facing application such as  online shopping portal etc., we need to test with different screen resolutions. A simple vertical or horizontal scroll bar may irritate the end user. That one small dislike may make the user leave your product.

The compatibility tests are large. You execute the same functional test cases, but on different environments. This takes more time and needs more people. But now, we have a lot of tools on cloud to automatically setup such combinations of environments. This will help you to test faster. You can see browserstack.com as an example. There are many providers like this, it is up to you to make the choice, that suits you better.

For video lessons, please visit www.openmentor.net.