Performance Testing - Performance Counters

When the servers are loaded with 100s of virtual user hits, the application response will slow down. This slowness is caused by a variety of factors. If we need to improve the speed of the app, we need to know exactly which causes the slowness. We must understand that the internet app will have web servers, app servers, database servers, load balancers, proxies, firewalls, network etc. along the chain. A single bad component can pull down the response of the whole app. To identify the exact place of slowness, we must rely on performance counters.

Imagine a total health check. Our whole body will undergo so many tests. Weight, heart beat rate, blood pressure, cholesterol levels, sugar levels, RBC count, WBC count, treadmill test results etc. will all be taken and analyzed by the doctor. each part of our body is an object and each object has many measurements based on tests, and these are translated to numbers. When all these details are translated to numbers, it is easy to compare and isolate problem areas. Now, treat each object in the entire web application chain as health components and measure it. That means, we must monitor server health!

Each machine or device will have cpu, memory, disk and network connection to transfer data in and out. If we measure these objects in every machine, we will be in a better position to analyze. Every load testing tool also provides performance counter collection component along with the tool. There are other independent tools as well, available in the market. Usually we start collecting these performance object counters from 10 minutes before the test run, collect data when test is in progress, keep collecting the data until 10 minutes after the test is complete. The data will be usually collected every 5 or 10 seconds.

What to measure in CPU? There are 100s of items to be monitored; this list has only the vital counters. % cpu usage, % cpu used by system, % cpu used by user application, number of processes waiting in queue to grab cpu.

What to measure in memory? % memory in use, page faults, swap ratio, cache hits.

What to measure in disk? Number of disk read/sec, number of disk writes/sec, read/write errors, disk queue length.

What to measure in network? Number of packets sent, number of packets received, packet errors, available bandwidth, tcp retransmissions, network queue length.

The above counters must be collected from all servers that are part of the application environment. Over and above these, a lot of specific counters are available and they must also be collected, after consulting with respective system/server admins.

Apart from hardware related counters, we must also collected software related counters. For example, if you use Apache Tomcat, we must collected a few counters such as number of active sessions, number of active connections, cache hit ratio, memory used by webserver, pages cached etc. When you use RDBMS, we must collect counters such as number of active connections to db, index usage percentage, number of waited locks, number of nowait locks, number of open tables, reads per second, writes per second, number of open cursors etc.

This means, we need to collect 100s of such measurements for every run, and analyze after the run. If these are not collected, we cannot isolate where the problem exists. How to identify exact bottleneck - we will see in the next post.

For high end load testing tool, visit http://www.floodgates.co.in. 

For free video lessons on load testing, visit  http://www.openmentor.net.

Performance Testing - Load Generators

In today's context, 100 vusers is the minimum load expectation for any web application. If you want to test for just 100 or 200 users, you just need one machine to generate the load. Each vuser will in turn be a thread or process, running in background, in the same machine where your load testing tool is installed. This machine is usually called controller. Each thread/process will occupy 1MB to 20MB memory space, based on script size and data size and each will consume some amount of cpu and disk. When I need to run 2000 vusers, one single machine is not enough to generate load. Here we need to distribute the load generation process itself.

Imagine each vuser consuming 5MB memory space; if we run 1000 vusers, we will require 5GB just for the vusers alone; over and above OS and other software will consume memory. If we have a machine with 4GB memory, we cannot run 1000 vu from that machine; because, that load generating machine itself will crash. Also, when responses for all 1000 vusers are sent to the same machine, that network port of that machine will choke. Every tool has a facility to generate load from different machines. These are called load generators or load agents. From the controller machine, these load generator machines must be accessible via LAN. A small program needs to be installed on all these machines, called remote agent process.

From controller, we must specify how many users are to be executed from each of the load generator machines. If our total user count is 1000, we can specify 300 from load-gen-1, 400 from load-gen-2 and 300 from load-gen-3. The target server for which we do the load test, must be accessible from all load gens; else scripts will fail. Once the load is distributed to all load gens, when the run starts, the tool will send the scripts to the load generators and instruct those to start the vuser thread/processes in those load gens. Hence, the memory and cpu of the controller will not be consumed. Every 5 or 10 seconds, the status will be sent back to the controller from all load gens.

This helps us in 2 ways. First, it helps us to run large number of vusers using multiple regular desktops/laptops, without a need for high end machines, for the sake of generating load. Second, we can run tests on target server, from remote machine other than where the tool license is installed. You may be in New York, USA, target server may be in Ireland and tool load generator may be in Los Angeles, USA. So, this helps us to do a load test with load being generated at a different geography, not from the same place where the server is installed.

For high end load testing tool, visit http://www.floodgates.co.in. 

For free video lessons on load testing, visit  http://www.openmentor.net.

Performance Testing - Configure vuser count, duration

Executing performance tests is a relatively easier task than doing scripting. Because, the tool is going to do more work and the tester needs to do just a set of configurations. The 2 key configurations are user count and duration. A performance test will not usually have just 1 script running; rather a set of scripts will be executed in parallel as a combined scenario. This is to reflect different sets of users doing different operations on the same server. So it is very important for us to do proper configuration before hitting the start button.

Recollect our first few lessons in load test planning. We identify a set of most frequently used scenarios and identify their priorities. We may want to run 1000 virtual users, but how to distribute 1000 virtual users across different scripts? It is better to get stats from both business team and the webserver admin team. They can tell the historic usage of the transactions. In a banking scenario, we may see x% of users doing balance inquiry, y% doing deposits, z% doing withdrawals, etc. Usually the business team can provide how many deposits happened in last quarter/month in terms of number of transactions, number of withdrawals, number of balance inquiry, number of utility bill payments etc. From that number, we can arrive at the % of transactions for that activity. If total transactions are 100,000 and deposits are 12500, we can say deposit transaction has 12.5% consumption of total transactions with the server and so on.

We now have to fix the total duration of the run. We usually try to run scenarios at least for 1 hour with all users in peak load. It is better to run for a longer duration to get better statistics. It also ensures the reliability and consistency of the servers and apps. If a branch of a bank works from 9 to 3, better we run our tests for 3 hours (half of it). Again this is our way of planning; different consultants suggest anywhere between 25% to 75% of the total duration of the office hours.

But there is one important aspect on releasing virtual users to hit the server. If I need to run 1000 vu, all 1000 vu will not start at the same time and hit the server. In real life, crowd slowly builds up - both on roads as well as on web. Hence we need to slowly ramp up the user count, rather than doing a big bang. If I need to run 1000 users for 3 hours (180 minutes) at peak load, what is the time I must keep in mind for user ramp up? We usually suggest 80:20 principle. Take 20% of the total peak load duration, and allocate that for ramp-up. Thus to run a scenario for 180 minutes, I may allow 30-35 minutes for users to ramp up and then run 180 minutes at peak. This means, the test will run for 35+180 minutes. Some companies try to include the ramp-up time within total duration and some do not. It does not really affect in a big way.

If 1000 users need to ramp-up in 35 minutes, how to release new users to the load pool? You can either evenly distribute or release in batches. If I need to evenly distribute, I can release 1 user every 2 seconds, and that will give 1000 users at the end of 2000th second. This means, first 1 user will start, after 2 seconds one more user will get added, after another 2 seconds another user will get added and so on. The other way is releasing in batches. Release 30 users every minute. This is purely a subjective decision and it will vary from project to project. In an online examination scenario, all users will ramp-up within 5 minutes, even though the exam duration is 2 or 3 hours.

For free lessons on automation tools, visit us at http://www.openmentor.net.

Performance Testing - Scripting Part 3

I always hear this from people "scripting is the best part in tools and I want to master scripting". Most people misunderstand scripting vs hand-coding. Hand-coding is remembering commands/syntax and typing the same whereas scripting includes both hand-typing as well as generating code thru wizards. Tools provide a lot of ways to generate code. A machine generated code is more trust-worthy than a hand-typed code and it consumes less time.  But people somehow feel good when they hand-type - known as "my code" syndrome. Let us see when and where we must use scripting efficiently.

Mere record and replay cannot solve 100% of our problems. We need to use scripting as well. Typically it will range from 5-15% overall in any load testing project.

1. Manual correlation. When we need to manually correlate some dynamic data, we must use correlation commands. Nowadays many tools provide a facility to locate a text and add command automatically. If that does not work well, you can do the same manually.

2. Taking a decision. Imagine a scenario in a ticket booking app. A page is displayed with flights data and we need to choose a flight between 8am and 9am. If the screen does not have filters for such time selection, the only way we can achieve is by scripting. You can get the whole response text in a variable thru correlation and then use string manipulation commands to select a flight with given criteria. Here we treat the html response as pure text data and locate the data we need.

3. Skipping actions/iterations due to data issues. If we need to skip subsequent steps, when we do not get proper data from parameter csv files or null data inputs or no data from server response, we can use this technique. Before any form is populated with data, check the validity of data using if conditions and then send the request.

4. Repeat a portion of a script. Many tools provide iterations for folders/containers/actions/blocks of code. If you want to repeat an entire block, you can use that built-in feature. If you need a set of lines within a block to repeat, you can write your own loops.

5. Custom logging. If you need to log some text or data in the way you need, you can use file handling open-write-close methods to achieve the same.

For free lessons on automation tools, visit us at http://www.openmentor.net.

Performance Testing - Scripting Part 2

When people move from functional test automation to performance test automation, there is always a question on how to handle dynamic data. All functional automation tools provide data driven test, where you feed data in a csv file or spreadsheet. But how to get or extract data that is sent by server? Most of the functional tools provide GetRunTimeValues from UI. This will dump the UI data in an array, from that we can extract the data we want. But in load test, when 1000s of vusers are running, we do not have UI at all. How do we handle dynamic data?

Look at this sequence of events. 

Client sends request REQ1. This is to get the list of purchase orders (PO).

Server sends response RES1. This has a lot of purchase order IDs.

Client must choose the first purchase order from response RES1 and pass that in request REQ2 with a few modified values. 

You do not know what PO ID will come from server. That is purely dynamic data. Unless you send one of those IDs coming in RES1, the REQ2 will fail. All these are happening in background without you seeing UI. To achieve this, you need to do correlation.

Correlation means - extract some details from a response, and pass that as part of subsequent requests. 

All load testing tools use the same principle to handle correlation. The response is a pure html text. See this example. List of POs are coming like this.

<table>

<tr class="po"><td>POID</td><td>CUSTOMERID</td><td>PODATE</td></tr>

<tr class="po"><td>9235</td><td>Navy Corp</td><td>01-10-2013</td></tr>

<tr class="po"><td>9845</td><td>Blue Minds</td><td>10-05-2013</td></tr>

<tr class="po"><td>9876</td><td>Blue Fields</td><td>06-03-2013</td></tr>

<tr class="po"><td>9989</td><td>Red Grove</td><td>07-04-2013</td></tr>

</table>

When I send REQ1, after sometime, the table data may be different in response. Hence every time, it is necessary that we get the first row from the response. How do we extract the row 1? There are 5 rows in the table and the first row is the header. 

Here is the simple trick. "Chase the data". I want to pick up 9235. Locate what appears to its left and what appears to its right. The text <tr class="po"><td> appears to its left and </td> appears to its right. But there are 5 such rows, where I can see the same left and right text. But this 9235 text appears in the 2nd position or ordinal. So if we tell the load test script, to locate the text with <tr class="po"><td> as left boundary and </td> as right boundary, it will give me an array [POID, 9235, 9845, 9876, 9989]. In this array, my text appears in ordinal 2. Look at this command.

locate_dynamic_data(dynavar1, LB=<tr class="po"><td>, RB=</td>, Ordinal=2);

where, dynavar1 is the variable to which it will load the value, LB is left boundary and RB is right boundary. When load test tool sees this instruction/command, it will scan the response, locate the text as per your LB, RB and Ordinal, and will give the extracted data in variable dynavar1. Once it is in variable, you can pass that to subsequent requests.

Everywhere, you see dynamic data, you need to correlate. You need to ensure that the correlation command gets proper data from the response. If response itself is not received or the text you look for is not present, subsequent requests may fail. You need to handle that part in scripting.

In java framework, in web apps, there is usually a dynamic text jessionid; same way, in asp.net apps, you will see a dynamic text __VIEWSTATE__, __VIEWINFO__. Unless you handle these framework related dynamic text, load test script will not work. But the load test tools are intelligent. Whenever the text during recording and text during replay are changing, they give a warning to the tester, that these changing areas are potential correlation areas. Hence pay attention when tools warn you on dynamic data.

For free lessons on automation tools, visit us at http://www.openmentor.net.

Performance Testing - Scripting - Part 1

Web applications will typically follow n-tier architecture, i.e. presentation layer, business logic layer, data access layer, data layer, external interfaces etc. But a website will mostly static pages, may be with javscript. The entry and exit to websites is unpredictable, as every page is a potential entry and/or exit, they are all independent. I a web application (webapp), the sequence of tasks one does is very important, to complete a business transaction. Hence there is a difference between load testing a website and load testing a webapp. As we saw in the previous post, the very first step is to identify the most frequently used scenarios and the steps for those scenarios. From that point, we need to focus on scripts.

Every load test tool provides recording feature. This is the most economical, easy and powerful way of scripting. First the tester must record the sequence of operations, as one user does. This means, start recording, start doing the business transaction steps on the application. The tool will record all the requests and responses, going to server and coming back from the server, using an internal proxy. Ideally every load test tool sniffs the request/response between the browser and the server. The request may be a get or post or ajax request. The tool will identify the url, the query string parameters for the request and will identify the server response data and redirection pages. Soon after recording, if we replay the script, the tool must be in a position to send the request to the server.

Mere recording and replaying will not solve many business rules. Some apps will require unique data, some need random data, some need advanced correlation and that is what we must do as the next step with the recorded script. Also, it is better we organize the requests under folders/containers. This will help towards easy maintenance of the scripts. For example, if our sequence of operations look like the following,

• Go to home page
• Fill userid and password, login
• Navigate to items page
• Load items list grid
• Select an item and edit
• Enter new details and save
• Refresh items grid
• Logout

It is better to organize the same like this.

• Initialize
• Initialization
• Go to home page
• Fill userid and password, login
• Items Grid Load
• Navigate to items page
• Load items list grid
• Modify Item
• Select an item and edit
• Enter new details and save
• Items Grid Refresh
• Refresh items grid
• Finalize
• Logout

Once the requests/steps are organized, the next step is to provide proper data to the script. We cannot use the same data we gave during recording. There are 2 parts to the data  - static data, dynamic data. Static data is different data supplied/typed by the user on the screen, dynamic data is the data that server sends back to the screen. Providing static data is called parameterization and handling dynamic data is called correlation. 

Static data can be provided thru variables. Tools provide variable manager module. We can create a variable and load the variable with different values, at run time. Some data that we may often use, can be obtained from the system itself, such as current date and time, user name, machine name, random number, random text, etc. 

Some data will be application specific. This data will have dependencies on other application data as well. For this, we usually create a file to have such data and modify the hard coded data values in the script, to use the data variables. The variables will be mapped to the file and specific column in that file. This is very similar to data driven tests in functional test automation.

For example if item creation page requires item code, item name, UOM, price as user supplied data, create a csv file like this.

ITEMCODE,ITEMNAME,UOM,PRICE

1001,Maxx Soap,NOS,18.90

1002,Vixor Biscuits,PCK,12.60

..

1099,Brainee Rice,KGS,14.50

Create a variable in the tool (say myItemData) and map that to this file.

In the script, replace the hard coded values with
myItemData.ITEMCODE, myItemData.ITEMNAME, myItemData.UOM,

myItemData.PRICE. During run time, the tool will read the data from this file and supply the values from the respective columns to the right variables. Usually, the tools will read lines sequentially and send that to the script. This can be changed as well. We will see these in the next post. Stay tuned.

For free lessons on automation tools, visit us at http://www.openmentor.net.



 

Performance Testing - Protocol Selection, Script Recording

If you know one load testing tool, it is easy to learn another. You need to be clear in the load testing fundamentals, in order to learn a new tool and to master it. We will help you step by step in achieving that mastery over load testing tools. Let us take the first part of the tools, protocol selection.

Protocol is nothing but the format in which the client and server communicate with each other. Eg., http, https, ftp, smtp, wap, tcpip, rdp, etc. Every application developer must first freeze the protocol and architecture, as changing these at a later stage will mess up a whole lot of things. But for a load testing person, all it needs is to understand what goes as request and what comes as response. Ultimately every thing will go as a stream of bytes; but to operate on the requests for parameterization etc., the load tester must know the parts of the requests and parts of the response. 

It is difficult to manage too many protocols and learning them at the bits and bytes level. Instead, if the tool can parse the request and response, and display the same in a clear user interface, most of the problems for the load tester are solved. The tester must refer to the design documents as well as consult with the development team to identify and choose the right protocols. A few applications may use multiple protocols to carry out a specific transaction; in that case, the tester must select all those protocols before recording the script.

The catch here is, that the load testing tools will charge you based on the protocol modules you want to buy! There will be a base license cost and there will be an add-on cost for every protocol module. You may be thinking that it is a simple web application, but that app may use Google Web Toolkit (GWT) or Flex related formats; without having those protocols as part of the tool, you cannot get a clean script. Hence one needs to be careful why purchasing the tool and add-on licenses.

Once you freeze the protocol, the you need to focus on scripts. We must create load testing script by recording a typical business scenario, as though one user is doing it on the application. For example, a user logs into an HR application, submits a travel request and logs out. The actual load test will send 100s of such requests (simulating 100s of users). The key point here is what scenarios we must record as part of scripting?

Identify the most frequently used user scenarios. All said and done, you and I go to google and do search 80% of the times.  Some other person may go to google stocks page to get stock quotes. So, the user priorities vary. Though google has 1000s of pages, only a small set of pages are more frequently used, by  most of the users. In the same manner, in your application, identify which are the most frequent ones and tabulate the same. How many users will execute those scripts, how long the users will run etc., we will deal with those in subsequent sections.

If you take any ecom site, the most frequently used scenarios are:

1. Go to home page, type a keyword, do a search, load search results, view an item from the results
2. Go to home page, type a keyword, do a search, load search results, view an item from the results, add to cart
3. Go to home page, type a keyword, do a search, load search results, view an item from the results, add to cart, provide payment details, buy

In the above 3 scenarios, many activities are common, but for every one customer actually buying, 100s of other customers, just "surf" and "window-shop" without adding to cart. Though they do not contribute to revenue, they occupy your system and network space. After attracting a customer to the site, thru marketing, it is very hard to see the customer abandoning the shopping cart without a buy! Usually these things happen due to slow response. So, buckle up, and make it faster!

For free lessons on automation tools, visit us at http://www.openmentor.net.