Monday, August 26, 2013

Performance Testing - Basics

Squeeze the app before release. If the app withstands that, it is fit for release. But how to squeeze? How will we determine the number of users, data volume etc.? Let us take this step by step and learn. Performance tests are usually postponed until customers feel the pinch. The primary reason is the cost of the tools and the capability to use the tools. If one wants to earn millions of dollars thru a hosted app, a good, proven and simple way is to increase users and reduce price. If one does this, the business volume will grow - but it brings the performance issues as well along with that.

Most of the tools use the same concept of emulating the requests from the client side of the application. This has to be done programmatically. When one is able to generate requests, processing response is a relatively easier task. When you choose the tools, it is better you look for the must-be-in features and then for nice-if features. 

The must-be-in features are listed below.

  1. Select protocols (HTTP, FTP, SMTP etc.)
  2. Record the user sequence and generate script
  3. Parameterize the script to supply a variety of data
  4. Process dynamic data sent by server side (correlation)
  5. Configure user count, iterations and pacing between iterations
  6. Configure user ramp-up
  7. Process secondary requests
  8. Configure network speed, browser types
  9. Check for specific patterns in the response
  10. Execute multiple scripts in parallel
  11. Measure hits, throughput, response time for every page 
  12. Log important details and server response data for troubleshooting
  13. Provide custom coding facility to add additional logic
The nice-if features are listed below.
  1. Configure performance counters for OS, webserver, app server, database server. This way, you can get all results under one single tool
  2. Automatically correlate standard dynamic texts based on java or .net framework. This will reduce scripting time
  3. Provide a visual UI to script and build logic
  4. Generate data as needed - sequential, random and unique data
  5. Provide a flexible licensing model - permanent as well as pay-per-use will be great
  6. Integrate withe profiling tools to pinpoint issues at code level
When one evaluates a performance testing tool, one must do a simple proof of concept on the above features, to see how effectively the tool handles these features. No need to say, that the tool must be simple to use.

Here are a few simple terms you need to be clear - at least academically. There are so many different definitions for the phrases given below, but we try to take the mostly accepted definitions from various project groups.

Load Testing - Test the app for an expected number of users. Usually customers know their current user base (example - total number of account holders in a bank). The number of online users will be usually between 5% and 10% of customer base. But an online user may be just a logged in user, doing no transaction with server. Our interest is always on the concurrent users. Concurrent users are usually between 5% and 10% of online users. So if 100,000 is the total customer base, then 10% of it, 10,000 will be online users and 10% of that, 1000 will be concurrent users.

Stress Testing - Overload the system by x%. That x% may be 10% more than normal load or even 300% more than the normal load. But usually load tests happen for a longer duration and stress tests happen for a shorter duration as spikes, with abnormally more users. Stress is like a flash flood. 

Scalability/Capacity Testing -  See the level at which the system crashes. Keep increasing users and you will see a lot of failures and eventually crash. Some companies use the term stress testing itself to include the capacity testing as well.

Volume Testing - keep increasing the data size for requests as well as process requests when the application database has 100s of millions of records. This usually checks the robustness and speed of the data retrieval and processing.

Endurance/Availability Tests - test the system for a very long period of time. Let the users keep sending requests 24 by 7 may be even for a week or month. See if system consistently behaves over a period of time.

For free lessons on automation tools, visit us at

Monday, August 12, 2013

Non-functional Testing

You are never alone. The environment around you changes every second. Your behavior in a changing environment - is it consistent or is it unpredictable? This is true for software applications also. Testing the behavior for a given input and expecting a definitive output is termed as functionality testing. But the same input, same product, but a different environment or an external factor, need not give a consistent output. Testing that is a non-functional testing. 

There are a variety of non-functional testing topics, that we are going to discuss in detail. The key areas that need to be addressed are given below.
  1. Performance Testing
  2. Compatibility Testing
  3. Interoperability Testing
  4. Security Testing
  5. Recovery Testing
  6. Usability Testing
  7. Localization Testing
  8. Globalization Testing
  9. Adhoc Testing
Let us first take performance testing. In today's world, internet is everything and it is everywhere. That connects PCs, servers, mobiles and people. Hence it is inseparable from our life. Internet is as important as electricity today. This means, more people use the application. Let us take google, facebook, amazon, youtube, msdn etc. All these sites/portals are used by millions of people. When more users use the system, company gets more visibility and hence more money. But the crowd comes with problem also.

When more people use, the system slows down or crashes. How many people are ready to tolerate the home page loading beyond 5-7 seconds. If your product does not load or start quickly, there are enough competitor products that users can try out. Hence the speed is the single factor that wins the hearts of users, right at the first shot. 

Remember the trinity - Users, Data and Time. If any one of these factors is increased, it uses more system resources such as cpu, memory, disk and network; and that causes slowness. But how will I test my app with 1000s of users, hitting the app at the same time? Can we assemble the real users in beach and give them laptops or tablets and coordinate to test the app? No way. Hence, instead of relying on real users, we go for virtual users. Performance testing is now the key factor for an app to be released. 

Performance testing has different sub-types.
  1. Load testing
  2. Stress testing
  3. Scalability or Capacity testing
  4. Volume Testing
  5. Endurance or Availability testing
To carry out these testing, we need to have proper tools. There are priced tools such as HP Load Runner, IBM Rational Performance Tester, Borland SilkPerformer and there are free open source tools such as Apache JMeter etc. In the coming sections, we will see the concepts of load testing and how to use these tools etc.

For free lessons on automation tools, visit us at