Non-functional Testing

You are never alone. The environment around you changes every second. Your behavior in a changing environment - is it consistent or is it unpredictable? This is true for software applications also. Testing the behavior for a given input and expecting a definitive output is termed as functionality testing. But the same input, same product, but a different environment or an external factor, need not give a consistent output. Testing that is a non-functional testing. 

There are a variety of non-functional testing topics, that we are going to discuss in detail. The key areas that need to be addressed are given below.

1. Performance Testing
2. Compatibility Testing
3. Interoperability Testing
4. Security Testing
5. Recovery Testing
6. Usability Testing
7. Localization Testing
8. Globalization Testing
9. Adhoc Testing

Let us first take performance testing. In today's world, internet is everything and it is everywhere. That connects PCs, servers, mobiles and people. Hence it is inseparable from our life. Internet is as important as electricity today. This means, more people use the application. Let us take google, facebook, amazon, youtube, msdn etc. All these sites/portals are used by millions of people. When more users use the system, company gets more visibility and hence more money. But the crowd comes with problem also.

When more people use, the system slows down or crashes. How many people are ready to tolerate the home page loading beyond 5-7 seconds. If your product does not load or start quickly, there are enough competitor products that users can try out. Hence the speed is the single factor that wins the hearts of users, right at the first shot. 

Remember the trinity - Users, Data and Time. If any one of these factors is increased, it uses more system resources such as cpu, memory, disk and network; and that causes slowness. But how will I test my app with 1000s of users, hitting the app at the same time? Can we assemble the real users in beach and give them laptops or tablets and coordinate to test the app? No way. Hence, instead of relying on real users, we go for virtual users. Performance testing is now the key factor for an app to be released. 

Performance testing has different sub-types.

1. Load testing
2. Stress testing
3. Scalability or Capacity testing
4. Volume Testing
5. Endurance or Availability testing

To carry out these testing, we need to have proper tools. There are priced tools such as HP Load Runner, IBM Rational Performance Tester, Borland SilkPerformer and there are free open source tools such as Apache JMeter etc. In the coming sections, we will see the concepts of load testing and how to use these tools etc.


For free lessons on automation tools, visit us at http://www.openmentor.net.

 

Test Automation - Exception Handling

Test automation is all about a tester generating test scripts, i.e. code. So, all problems of what a developer would face, a tester would also face. Hence test script must also be fully tested by the tester. But, there are 4 major factors that can affect the test script and those 4 are unpredictable. Hence, the test script must gracefully handle those areas. This is called exception handling or known as recovery scenarios.

Hurdle 1 - Unknown pop-ups. When the script executes, step by step, application reacts to those steps, and there is a specific expectation out of each step. For example, if a user enters a valid account number in account number field and press tab, the account name must be auto populated in name field; when script executes, due to a wrong data fed into account number field, if the application shows a pop-up stating "Hey, the account number does not exist; please check the data", that pop-up is a blocker. Without closing that, nothing can be done and script cannot proceed. This is just one example; similarly in many occasions, due to data issues or application functional issues, OS related issues, unwanted popups will crop up. The tester must ensure all such pop-ups are addressed when script runs. Ideal solution is close the pop-up, and continue or go to next test case.

Hurdle 2 - Objects or pages not found or disabled for input. This happens due to a functional bug in the application. Take an example. When user goes to account balance screen, the account number must be enabled for data entry. Occasionally if that field does not appear or disabled, the script will go ahead try to enter the account number and the type event will fail. It is not possible to check the enabled or displayed property for every field to be ON, before every step. A field not found error may happen, if the display of the field, goes outside the display resolution of the monitor. If this kind of errors happen, it is better to relogin to the app and move to the next test case.

Hurdle 3 - Application crashes. This is usually due to some critical bug in the application. The test script will not find the application itself to execute the next step. In this case, it is ideal to restart the app and start the next test case. If we try to start the same test case, it may crash again and it may go in a loop.

Hurdle 4 - Script error. This is due to a wrong logic by the tester in the test script. This may be due to accessing wrong array locations, divide by zero, trying to open a non-existing file etc. If this kind of errors happen, it is better to relogin to the app and move to the next test case.

Tools provide a variety of mechanisms to handle the exceptions, either thru coding or by configurations or by both. Every exception is a learning. It is very difficult to identify all exceptions and handle those before script runs. It  is an evolving process. So, as and when new unforeseen exceptions do happen, add them to the exception handler library.


For free lessons on automation tools, visit us at http://www.openmentor.net.

Test Automation - Scripting Essentials

Record and replay alone cannot solve test automation problems; also record/replay may not be always possible. The test script may need some kind of decision making at run time; script may need some intelligence in handling situations; the test may have to alter its path based on some values coming on screen at run time. The simple solution for all kinds of such issues is - Scripting. Every tool provides some scripting language such as VBScript, Java, JavaScript, C#, Ruby etc. So, if we mix record/replay and scripting, we increase the power of the tool by thousand times.

Rule 1: Never build application logic in the scripting language. Example. You recorded how to book one-way ticket for 1 person; for 2 people, if app needs the price to be multiplied by 2, do not build that multiplication logic in your script. Because, whenever the app logic changes, script needs to be modified and retested. You may make some bugs in coding in the scripting language, while modifying that logic. Always, see what is the input, manually determine what must be the expected output, feed that output as checkpoint values, go on. Your brain is the best tool, better than any automation tool.

Rule 2: Comment your script well. Maintenance of script is very important. Hence make sure another tester can easily understand your script. 

Rule 3: Unit test your test script. Remember, developers make mistakes in their code. When you program, you will also do mistakes in your code. Being a tester, nothing guarantees that your test scripts will work without testing.

Rule 4: Ensure your test functions, work for a variety of parameters. A single function may feed 1000s of data thru data driven test. Hence test your script with multiple data.

Rule 5: Avoid nested if conditions. At the most you can do 2 levels. This itself will consume more time for you to unit test your test scripts.

Rule 6: Avoid nested loops. Nested loops may not be actually required in 95% of the cases. So be judicial when using nested loops.

Rule 7: Have a traceability matrix for your test scripts. You need to have a spreadsheet that documents the input params, output params, file details, function details, caller details for every test script/function. Else, when it grows to 1000 test scripts, changing one script may affect another, if you are not aware of the dependencies.

Rule 8: Always put all test scripts in a batch and run. This will eliminate the base state problems.
 
Rule 9: Always make another tester (other than the author of the test script) to run the test batch. This will eliminate human related issues and documentation issues.


Rule 10: Always run the test batch from another machine (bot used for building the test scripts). This will eliminate system related issues, hardcoding of drive/folder names etc. and documentation issues.


For free lessons on automation tools, visit us at http://www.openmentor.net.







 

Test Automation - File/DB Checkpoints

Most of the apps take the input from the user thru UI, do some logic, store the data in a file or database, finally show a success or failure message on the UI. Checking the message alone is not good enough; what if the data was truncated or wrongly formatted or wrongly stored or lost? This is why you need to check the data in files and database. Testers can see the results of a test case on the UI or in a file or in database tables. Checking the text on UI is very simple as one can immediately see it in front of the eyes. But Checking the results in a file takes some more time and a few extra steps.

Let us take the file storage first. First of all, the tester must know the folder and filename in which the app stores the results. The file may be a flat file or a csv or a tsv or an XML file. Hence the tester must also know the format of the file. First, the tester must manually enter data thru UI and save the record and see how the file stores the data. Imagine the app stores the employee record like this (one record per one line): emp_code,emp_name,emp_dob,designation,salary,branch,emp_doj. This is a csv format. Here, we do not know the length of the line; it can vary based on the length of the name or other fields. But all I know is that if I give some data in the UI, the same must appear in the file, in the given order. Hence, to check whether the data is present in the file or not, we must know how to open a file, read the records from file, compare the data with expected results and then close the file. All tools provide a fiel interface thru file system object (VBScript) or file input stream (Java). Using these commands, one must read record after record, compare with expected results and close the file.

If the app stores data in XML file, it is a little bit easier. Because, every scripting language provides simple commands to check whether a node (<tag> </tag>) is present, how many child nodes are present and the data embedded within a specific tag. These commands use XML object to store the file data. Tools like HP QTP provide a very good UI also to view the whole XML structure like a tree and we need not even write code to check data in an XML file.

When it comes to checking records in a database, you need to know the tables and their structure. Soon after every add, edit or delete transaction thru the app UI, we must do a select query from the respective table, extract the data and compare whether the same is correct or not. The important point in writing the query is to narrow down the data that is very very specific to the test. Do not grab a lot of records that are not required for the test; for example, if I cancel a ticket with reservation number XB45890, I need to just select the record for that reservation number only; so that I get only the record I want to compare. Try to select records based on some unique key values, rather than a generic where clause.

When you use file or database checkpoints, make sure you close the file or database connection, soon after the checkpoint.

For free lessons on automation tools, visit us at http://www.openmentor.net.

Test Automation - Checkpoints

A test must end with a pass or fail status. Humans execute test steps, see the actual results in front of their eyes, compare that with expected results and declare the pass/fail status. Using record/replay/data, test automation tool executes the test steps. But the tool does not know expected results. We need to tell the tool about what is expected and the tool can then declare the results. This is implemented using checkpoint or verification point feature, in every tool. 

Tester feeds data on UI objects such as text box, combo box, radio button etc. When the form is submitted, usually thru a button click to save or update or delete, the application does the operation and posts a message back on the screen. The result is usually a text message such as "Your ticket is successfully booked. Thanks for choosing us.". The tester will then state that the booking test case has passed. If this message does not come, it is treated as failed. Usually the messages will be displayed in specific places of the screen or status bar or the screen may refresh with the data that is entered with an auto-generated id etc. 

As part of automating test cases, every test case will have steps and will have checkpoints. In the checkpoint, we need to feed what is the object to be looked for results, which property of the object reflects the result and what is the expected result. e.g. look in the status bar, look at the text, the text must be "The transaction is completed successfully.". In some cases, it may not be text, instead we will be looking for the price to be updated in the price text box. When you go to a travel portal, usually the number of passengers will be set to 1 and the price will be for 1 person, say $110. When you change the number of passengers to 2, the price must change to $220. So, based on one event, the expected outcome changes in another field. In this case, we must place a checkpoint on the price text box, for its text property with an expected value $220. 

In some cases, only when I check the "I accept the terms" check button, the continue button will be enabled. Till then it will be disabled. In this, we need to place a check point for the continue button, for the enabled property with the expected outcome as true/on. In an email inbox, when I delete a mail, the number of mails shown at the top will be reduced by one. Like this, the screen will show the actual results in specific places. We must find those places and put appropriate checkpoints. Some organizations will use standardization of their UI elements, such as the width and height of all push buttons must be 90 and 30 pixels respectively; in such case we must place checkpoints for the width and height properties of the buttons.

Every test case will have the standard sequence as follows.

• Test steps to establish pre-requisite state of the application
• Test steps for the test case
• Checkpoint-1
• More test steps for the test case
• Checkpoint-2
• Some more test steps for the test case
• Checkpoint-3
• Test steps to clean-up

When the checkpoint passes, the tool will declare the results in green color and when it fails, the result will be painted in red color. Ideally for every expected result in my test case, I must have a checkpoint command in the test script.

For free lessons on automation tools, visit us at http://www.openmentor.net.

Test Automation - Data Driven Test

You need more data for testing. I would rather re-phrase this to state you need more varieties of data rather than more data itself. Each data set must help the tester to drive the application thru a different logic or path, so that we can do better testing. Take the example of booking airline ticket. It is just 1 screen that takes data and users feed these fields to get their ticket booked. Each ticket may be a different combination. A human tester gets bored when repeatedly seeing and operating on the same screen. But this has to be done any way. 

Here comes the data driven test (DDT) feature of the test automation tools. Test steps are same, but data is different, for each transaction I must use a different data set - If this is your case, use data driven test. Every testing tool comes up with a data sheet or data pool. Here you need to provide data in a spreadsheet like file. For booking a ticket you need from place, to place, one way or round trip, date of journey, return date, number of persons etc. Though there are lot more details for ticket booking, let us limit to these data at this point. 

The first thing for DDT is to create a file that has these data. Usually you can use a csv file or xls file. The first line is usually the title for the data. See the example below.


FROM-PLACE,TO-PLACE,TRIP-TYPE,JOURNEYDT,RETURNDT,NUMPERSONS
Los Angeles,New York,TWOWAY,10-JUN-2013,14-JUN-2013,2

Los Angeles,Denver,ONEWAY,10-JUN-2013,10-JUN-2013,1

Los Angeles,London,TWOWAY,10-JUN-2013,18-AUG-2013,5
...

We need to prepare each line of data that represent some equivalent partition set to test our application. 

Once data is created, we need to record a script using the tool, that does one ticket booking, by entering the data in the above fields. While recording we will give some data and that will reflect in the script. e.g.

BookingScreen.Clear
BookingScreen.FromAirPort.Set "Los Angeles"
BookingScreen.ToAirPort.Set "Denver"
BookingScreen.TripType.Select "TwoWay"
BookingScreen.DateOfJourney.Set "10-May-2013"
BookingScreen.DateOfReturn.Set "12-May-2013"
BookingScreen.NumberOfPersons.Set 1
BookingScreen.Submit

The syntax given above is generic and not specific to any tool. To ensure that the script uses the data from our file, we may have to modify the script manually or use the data driver wizard of the tool. We need to set the value for each UI field data entry, to point to a column in the data sheet. Also, we must loop thru all rows in the data sheet, so that the script executes for all records and books multiple tickets. It may  look like the following.

DataSheet = "C:\\mydata.csv"

For currRow = 1 to DataSheet.RowCount

   DataSheet.CurrentRow = currRow
   BookingScreen.Clear
   BookingScreen.FromAirPort.Set DataSheet.getColumnvalue("FROM-PLACE")
   BookingScreen.ToAirPort.Set DataSheet.getColumnvalue("TO-PLACE")
   BookingScreen.TripType.Select DataSheet.getColumnvalue("TRIP-TYPE")
   BookingScreen.DateOfJourney.Set DataSheet.getColumnvalue("JOURNEYDT")
   BookingScreen.DateOfReturn.Set DataSheet.getColumnvalue("RETURNDT")
   BookingScreen.NumberOfPersons.SetDataSheet.getColumnvalue("NUMPERSONS")
   BookingScreen.Submit
Next

The simple code, will get all records from the csv file and repeat feeding the screen with different data sets from the csv file. This way, a 10 line code can enter 10000 records with little human interference.

For a demo of data driven test, see this video. 

Test Automation - Object Identification

There are about 30 students in a class; you need to find out whether the person you want to talk to, is there or not; assume you have not seen him. You first go and ask "Is John here? I have a parcel to deliver". So, you use the first name to identify a person. If there are 2 Johns in the same class, then you will ask "John Bosco". So you use last name also as additional qualifier to uniquely identify that person, among 30 other students. As a rare case, if both are John Bosco, what will you do? You may say "John Bosco, son of  David P Bosco". The goal is to deliver the parcel to the right person. Testing tools also work on very similar way to carry out click or type actions, on the exact objects on the application UI.

If one has a very good grasp on object identification concepts, 80% of the automation problems are already solved. An object means an item on the user interface, on which you either click or type or you read it and infer some information; e.g. first name text box, state combo box, save button, etc. Recording feature in every tool, internally does this object identification. Most of the tools use a way of storing the details of objects in a separate file. In HP QTP it is called Object Repository, in IBM Rational Functional Tester it is called Object Map, in Borland SilkTest it is called frames.inc, and so on. 

Every object has a type, aka class. The most common types are text box, check button, radio button, combo box, push button, menu bar, tool bar, label, link, image, grid (rows, columns, cell), scroll bar, status bar, progress bar. These are the typical items we see in every application. But the exact way, every technology calls these classes will be different; for example, native Windows C++ will say Push Button, Java will say JButton, Turbo C++ may say TButton, for the same push button you see on screen. This specific text is called nativeclass.

They also have a name associated with them. This is not mandatory, but preferable. The objects may display a name on screen for the user to read, but it may internally use another name. This is like the nick name we use for people. Tools understand the internal name of the objects, rather than the display name of the objects. Ideally developers of the screens must use same, meaningful names for display as well as internal purposes. 

The best way to identify an object is to use the type/class of the object and its internal name. e.g. button/OK, combobox/States, textbox/ZIP code etc. Sometimes, the same display name may appear 2 times on the UI. Consider a screen that takes your personal and office addresses; both will have street/city/state text boxes. So the display name is same; the humans understand this well; but for the tools to understand two objects with same display name, it is better to use the internal name given by developers. 

When we need additional details to uniquely identify an object, there are other properties such as location (x, y coordinates), index of the object (nth object from top of the screen) are usually used. But position based identification is a risky option; this may fail when screen is updated with new objects or a few objects are removed, that may affect the relative position of this field on the screen. 

To understand the objects and their properties, tools provide object spy or object inspector utilities. Before you start automating, spend a few hours to see the class, name and other properties of every object on every single screen. If you see objects with confusing properties or non-unique properties, negotiate with development team to have consistent naming of objects.

Remember: Proper naming of objects reduces the time to automate significantly. This simple process discipline is required by the dev team to make test automation is easy.

For a live demo of this, please click on this url.