Performance Testing - Configure vuser count, duration

Executing performance tests is a relatively easier task than doing scripting. Because, the tool is going to do more work and the tester needs to do just a set of configurations. The 2 key configurations are user count and duration. A performance test will not usually have just 1 script running; rather a set of scripts will be executed in parallel as a combined scenario. This is to reflect different sets of users doing different operations on the same server. So it is very important for us to do proper configuration before hitting the start button.

Recollect our first few lessons in load test planning. We identify a set of most frequently used scenarios and identify their priorities. We may want to run 1000 virtual users, but how to distribute 1000 virtual users across different scripts? It is better to get stats from both business team and the webserver admin team. They can tell the historic usage of the transactions. In a banking scenario, we may see x% of users doing balance inquiry, y% doing deposits, z% doing withdrawals, etc. Usually the business team can provide how many deposits happened in last quarter/month in terms of number of transactions, number of withdrawals, number of balance inquiry, number of utility bill payments etc. From that number, we can arrive at the % of transactions for that activity. If total transactions are 100,000 and deposits are 12500, we can say deposit transaction has 12.5% consumption of total transactions with the server and so on.

We now have to fix the total duration of the run. We usually try to run scenarios at least for 1 hour with all users in peak load. It is better to run for a longer duration to get better statistics. It also ensures the reliability and consistency of the servers and apps. If a branch of a bank works from 9 to 3, better we run our tests for 3 hours (half of it). Again this is our way of planning; different consultants suggest anywhere between 25% to 75% of the total duration of the office hours.

But there is one important aspect on releasing virtual users to hit the server. If I need to run 1000 vu, all 1000 vu will not start at the same time and hit the server. In real life, crowd slowly builds up - both on roads as well as on web. Hence we need to slowly ramp up the user count, rather than doing a big bang. If I need to run 1000 users for 3 hours (180 minutes) at peak load, what is the time I must keep in mind for user ramp up? We usually suggest 80:20 principle. Take 20% of the total peak load duration, and allocate that for ramp-up. Thus to run a scenario for 180 minutes, I may allow 30-35 minutes for users to ramp up and then run 180 minutes at peak. This means, the test will run for 35+180 minutes. Some companies try to include the ramp-up time within total duration and some do not. It does not really affect in a big way.

If 1000 users need to ramp-up in 35 minutes, how to release new users to the load pool? You can either evenly distribute or release in batches. If I need to evenly distribute, I can release 1 user every 2 seconds, and that will give 1000 users at the end of 2000th second. This means, first 1 user will start, after 2 seconds one more user will get added, after another 2 seconds another user will get added and so on. The other way is releasing in batches. Release 30 users every minute. This is purely a subjective decision and it will vary from project to project. In an online examination scenario, all users will ramp-up within 5 minutes, even though the exam duration is 2 or 3 hours.

For free lessons on automation tools, visit us at http://www.openmentor.net.

Performance Testing - Scripting Part 3

I always hear this from people "scripting is the best part in tools and I want to master scripting". Most people misunderstand scripting vs hand-coding. Hand-coding is remembering commands/syntax and typing the same whereas scripting includes both hand-typing as well as generating code thru wizards. Tools provide a lot of ways to generate code. A machine generated code is more trust-worthy than a hand-typed code and it consumes less time.  But people somehow feel good when they hand-type - known as "my code" syndrome. Let us see when and where we must use scripting efficiently.

Mere record and replay cannot solve 100% of our problems. We need to use scripting as well. Typically it will range from 5-15% overall in any load testing project.

1. Manual correlation. When we need to manually correlate some dynamic data, we must use correlation commands. Nowadays many tools provide a facility to locate a text and add command automatically. If that does not work well, you can do the same manually.

2. Taking a decision. Imagine a scenario in a ticket booking app. A page is displayed with flights data and we need to choose a flight between 8am and 9am. If the screen does not have filters for such time selection, the only way we can achieve is by scripting. You can get the whole response text in a variable thru correlation and then use string manipulation commands to select a flight with given criteria. Here we treat the html response as pure text data and locate the data we need.

3. Skipping actions/iterations due to data issues. If we need to skip subsequent steps, when we do not get proper data from parameter csv files or null data inputs or no data from server response, we can use this technique. Before any form is populated with data, check the validity of data using if conditions and then send the request.

4. Repeat a portion of a script. Many tools provide iterations for folders/containers/actions/blocks of code. If you want to repeat an entire block, you can use that built-in feature. If you need a set of lines within a block to repeat, you can write your own loops.

5. Custom logging. If you need to log some text or data in the way you need, you can use file handling open-write-close methods to achieve the same.

For free lessons on automation tools, visit us at http://www.openmentor.net.