Tester's friend - The BUG

Every normal human being gets pleasure while finding mistakes in others, sad, but true. Tester is also a human being and when he/she finds a bug, it further motivates him/her to find more. A tester aims to fail the software in any way possible, and tester succeeds when bugs do appear. I do not worry about the theoretical definition of bug; to me, anything that the software does in a manner that is not intended as per specs or by implied needs, is a bug. Hence bug, error, problem, issue, failure etc. all come under the same category, though you can find zillion definitions for these words if you google.

When a bug appears while you test, do the test again. Ensure it appears again. This will prove that, you as a tester, had not done any mistake in executing the test. Soon after confirming its consistent nature, do not call the developer. Developer has a tendency to reject or dispute. Do not send a mail either. You need to ask a few questions to yourselves before you formally state that it is a bug. 

Try to describe the bug in 1 line. This is like writing the email subject. "User does not get confirmation email after registration", "Application accepts invalid SSN in Add Customer Page". Try to convey the exact problem with minimal words.

Mention the screen name where the bug appears. This will help the developer directly to go to that place, without wasting time. If possible, take a screenshot where the bug appears. A picture means more than 1000 words.

Explain the bug in detail, by telling the steps to reproduce. You get bugs when you execute a test case. Thus, the bug must be appearing against one of the test steps. Copy paste the test case itself and mention the exact problem against the test step. If the developer does the same thing, he/she should get the same bug.

How does this bug affect the product? The answer to this question will help you to set the Severity of the bug. Are you able to proceed further to do other test cases? If not, this bug is termed as a Show Stopper or Blocker. This means, you have no way to test anymore unless development team fixes the same. 

Is there a crash or data loss or a feature is missing in the product? This falls under High or Major category, provided you are able to do other tests. This means, that the product has a serious problem. For example, when we page customer details, Save operation itself does now work and hence the data entered is lost. Technically a developer may say that he can fix this in 2 minutes, but always look from customer's angle. For the customer, data is more important than the product itself. Any loss of data is a major bug. The severity code Major is also termed as High in many companies.


Bug does not stop me from testing, there is no data loss or crash, but the functionality is wrongly implemented. The app saves data, but some part of the results are not accurate. In this case, companies usually treat this as Minor or Medium severity.

The app does the job, but there are spelling mistakes or improper messages to users or imporper field alignments in the UI. Usually companies treat this as Cosmetic or Low severity.

To sum up, if we can write the following, then we are good in reporting bugs to dev team.

• One line description of bug
• Details to reproduce the bug
• Screen name where the bug appears
• Screenshot of the bug
• Severity of the bug

By principle, every problem must be reported and tracked. We will see tracking mechanism in the next post. For more details on bugs, please see this bug tracking demo.

Test Execution - How can we do faster?

Development deadlines always slip, but release dates do not change; caught between these 2, the testing time is always compromised. Hence the testing teams need to work faster. If the application has withstood the first one hour of critical tests, it indicates the basic stability of the app. But every test is important from functional point of view. Here are some simple techniques to make it faster.

Testing teams may have a mix of people with different experience levels. It is very uncommon to see a testing team with uniform experience level. The first thing we need to do is to assign navigation and field level validation tests to the junior-most members of the team. The number of tests will be more, but they are relatively simpler to execute. For every screen, let them do the same, and in one or  two rounds they will get familiarized with the app.

Assign the form level tests to mid-level experienced testers. This will involve submitting a variety of data thru forms and trying add, modify, delete, view test cases. 

Assign business flow tests to seniors and leads. They need to wear the cap of the end users and move data across screens to achieve the business flow. These people are the best bet to test the reports. The senior people must start feeling the semantics behind the data that flow across the application and understand the business value of the flows. If this is not done, end user testing will always find finer problems and that may lead to escalation.

Always people feel happy to execute new tests (progression tests, for new features added or for change requests). But by principle, a fix or new feature introduction may create issues at some other place. So we try to cover those areas by doing regression tests. The first set of regression tests must be performed for the form in which  the new features/changes are introduced. The next set of regression tests must be done at the module to which the form belongs. The next set of regression tests must address the path in which the feature or change is introduced. Once these are covered, then only do other regression tests. This will ensure the the impact of the changes are minimal.

For every round of testing, reallocate people. Doing the same test again and again will make them slow. You cannot reallocate a person to a totally unrelated module. Every application has a flow of information from one module to another; e.g. request spare item in spares module, cater the service in service module, raise invoice in accounts module. If tester A has done testing of round 1 for spares module, for round 2, assign tester A to service module and in the 3rd round assign to accounts module. This way, the tester also logically moves to the next step rather than landing on a dark area.

Last but not the least. Ask every tester to stand up and do a simple neck and shoulder exercise for 20-30 seconds every hour. Remember, you need good blood flow towards head. When you are too attentive for hours, doing testing, with eyes locked to the monitor, you will develop neck and shoulder pains. You must be physically good to perform better. So take care of your body well.

For free learning courses, please visit http://www.openmentor.net

The first hour of testing

When a person suffers from heart attack or stroke, the first one hour is crucial. If we can treat the person in that first hour, there is about 90% chance that the heart or brain reacts positively to the medicines. This is true in testing also. The first one hour of testing can bring out a great insight about the health of the application being tested. To achieve best results, we have to execute the tests in a structured manner. Here is how.

At this point we assume that the test bed is ready and the application is already installed and ready to be tested. We may have 100s or even 1000s of tests documented and reviewed. Before test execution, ask this question on every test case - "If this test fails, is there any point to continue testing?". Mark all those test cases for which you get an answer "No". That means, those are the vital test cases that must necessarily work or pass. Let us take an example of testing a car. To test a car, we may have to do 1000s of tests; but if ignition does not work, if door does not open, if clutch does not work, if brakes do not work, if accelerator or gas pedal does not work, if steering does not work, is it worth saying that as a car? In your software testing, identify those test cases before test execution.

In the first 5 minutes, navigate to all screens in the application. Simply check the loading of forms and the look and feel. End users do care about screens more and more nowadays.

After this, in every form, if there is a provision to add or create new records, perform those. If no data is added, no data can be viewed; hence do add data. Add with valid data. negative data conditions can be done a little later. Ensure all these tests work well. In every screen try to view all existing data. If you see missing data on screen, we have a problem.

Then, execute critical workflows with valid data. One of the flows in a retail banking scenario may be, create an account, perform a few deposit transactions, perform a few withdrawal transactions, view account balance and transfer some amount from checking to savings account. The flow touches a set of screens with connected data. Usually every application will have top-10 workflows or top-20 workflows. Perform those meticulously. 

Generate a set of key reports and charts that the management people would view on the application on a daily basis. These are the heartbeats of the application to the customer's management; one wrong data in that, the management will lose the confidence in the product.

If we can pack these in the first 1 hour of testing, we can definitely see whether the application is ready to do what it is supposed to do, i.e. it works for positive data. All other invalid data validations, invalid flow tests. non-core features can be tested after the first hour. We call this as BVT (Build Verification Tests). No failure can happen in BVT. Only when all these tests pass, we ask our testing teams to proceed further. You may still ask a question, brakes in car not working does not mean that the horn may not work. I agree. But, if a critical part or path is not working, there is no point in testing less critical part. So let us not waste time and this will ensure that the build or dev teams to ensure they pay proper attention to the paths of the application according to their importance. 

Usually 10-15% of the total tests will fall under BVT category in every application. Attack those first.

For free learning content, visit us at http://www.openmentor.net

Test environment (test bed) setup - some important points

The environment on which we carry out testing, is the very first challenge for every testing team. My mentor always says, if our hands are not clean before taking food, we are prone to illness. The environment (also known as Test Bed) plays a vital part during test execution stage. Due to delivery deadlines, delay in hardware readiness, etc., people hurry up in setting up test bed and at a later point, it becomes the finger-pointing item in any bug triage meetings. Many to most of the times, the application may display a misleading message and people will start troubleshooting in one direction; but the actual problem will be at the test bed level. 

To preempt such issues, certain rules we must follow,, every time we start testing.

Rule 1. Test bed must not have development studio or IDE or other tools, except the product binaries and configuration files. If we have Eclipse or NetBeans in a java product test bed, that itself will resolve many conflicts and the product may tend to work fine. But in reality, in production environment, no dev tools will be present. Hence we must ensure that test bed resembles the production software environment to the extent possible.

Rule 2. Check the versions of OS along with all service packs, patches and hot-fixes.  Many times, people just check the OS version and start executing tests. The service packs and hot fixes do have a lot of compatibility and security related dependencies.

Rule 3. Check the webserver properties. Usually the configuration xml files will have memory settings, connection settings, database connectivity settings etc. Unless they are correctly configured, those may start showing up at different paths of the application.

Rule 4. Check database properties.  Things such as case sensitivity, connections, connection pooling settings, unicode settings etc will have a definite impact on the application behavior.

Rule 5. Check the free disk space, memory space and log file space. These items may not immediately cause issues in functional testing, but do have an effect on performance of the system. 

Rule 6. Check the system users who will run the application. This is mostly for rich-client applications. But if the application login is obtained from the LDAP settings, for all applications, the security settings of the logged-in user is key for carrying out certain activities at application and system levels.


Rule 7. Check the background processes/services. These services must be brought up, before others access the application. See if these are up and running without any exceptions.  Adequate configurations must be done to restart these if these fail to run during test execution.

Rule 8. Check the connectivity of the system with server and other machines. Try to ping or connect to the machines that need connectivity. Check the proxies and firewalls as well.

Rule 9. Check your antivirus settings. Bad settings may prevent blocking of requests or unwanted quarantine effects on your application data. This may cause connection or even timeout issues.

Rule 10. Check your browser version  and settings. Especially when you have auto upgrade option ON in your browser, you may not be able to track what happened to your browser since the last access, and silently an upgrade might have happened. That latest version may not be in scope of testing at all.

For  free learning content, visit us at http://www.openmentor.net

Review before you execute tests

I have been using English as the official language to communicate for the last 20 years. When I write a paragraph, and if I read the same once again, I see some   issues and I correct those. This happens every time. In simple terms, I can find out at least 2-3 issues in a single page that I write. When I see the text written by others, I do see issues that need correction. This is true whether I write my personal diary or letter or short story or test cases. Every document needs proof reading by the author and by at least one other person. 

When we write test cases, we think and start writing; the speed of thinking varies from speed of typing. The ability to choose the right word at the right place need not be consistent always. When we change one particular sentence    in a paragraph, that tone of that may be slightly different from that of others. Hence our human limitations of expressing what we think, in terms of an understandable text by other people, lead to issues in documents. Hence we must have a review done by other people.

Any document in any phase of SDLC, must go thru review. This is also termed as peer review, as your peers will do the review of the document. Reviews can be formal or informal, but must be effective. Usually the author of the document must coordinate to arrange a review. If you write test cases, check with a few other members in the project, fix up a time (30-60 minutes), fix a meeting room for review. Send the document that needs to be reviewed, to the invitees at least 12 hours before. They get time to glance thru it before they come for review.

Usually a formal review will have one senior person as moderator; usually, the project manager or business analyst or test manager will play that role. The author will give a brief introduction on the document to be reviewed. Let us assume we all review a test case document. Then everyone must start reading thru test cases one by one; take one, read it thoroughly, raise any concerns if it is not clear to you. Many people do not talk much during review even if they see issues in the test cases. This is not right and we are not doing any value add if we do not raise concerns. 

Keep these in mind while reviewing.

1. Is the test case clear? Is it simple to understand?
2. Are there any grammar or spelling issues?
3. Is this test case relevant to a requirement? 
4. Is this test case similar to another test case or is it a duplicate of another?
5. Are the sentences written in proper format / tense? (e.g. test steps cannot be in past tense)

If any one raises any concerns, author must note down the test case id, the issue raised, the person who raised the same. If it is a valid concern, then the author has the responsibility to correct the same. Correction may happen at a later time. When one test case is cleared, let all reviewers move to next one and review that, raise any concerns. This process repeats for all test cases. At the end, the author will collate all the review findings/comments into a single sheet and circulate to all. This must be used as a reference for correction.

Sometimes, the concerns raised by a reviewer may end up in a heated argument between the author and the reviewer. At that time, the moderator needs to resolve it with logical explanation by author. If this is not resolved, it may trigger ego clashes among people.

After the review, the author must make all those corrections and send the document to all or selected invitees, to ensure that the corrections are made without any further issues. If the number of issues raised during review go beyond certain limits, the moderator may call for another review session as well. Once the corrections are made, the document will go for further approval by project manager or test manager.

There are different review methods adopted by companies. Some of the common approaches are given below.

1. Walkthru - this is the formal approach we explained above. This is very effective, needs coordination within project
2. Inspection - this is usually a review conducted by people from other projects.  This is very effective, needs coordination across projects
3. Desk checking - this is quick review, by just one person, directly coming to your desk and reviewing the document. This is less effective, but needs less coordination

If an organization has a strong reviewing team, there is no doubt that the organization will be strong in delivering products as well. 

For free online training programs, visit us at http://www.openmentor.net

The CRUD Approach

With the knowledge of "what" and "how" to test, let us go one more step forward in testing. Majority of the tests fall under the category of functionality. Functionality is nothing but the way the software responds to its inputs based on the programmed logic. The functionality itself can be broken to smaller pieces to make it simple. Here is a CRUDe way to do it.


Let us take a simple social networking application. You can create a new account, edit your profile, add a friend, delete a friend, add comments, delete comments, reload your photograph, view the comments by others etc. Take an instant messenger like Skype. You can add a contact, rename the contact, edit your text, delete a contact etc. Look at a documenting tool like MS Word. You can create a document, modify its contents, rename it, delete some portions of it etc.


So, what do you see as a common set of activities in all the above? Add something, edit or modify it, read or view it, delete it. In other words, Create Read Update Delete and that is CRUD. There is no exception to this in software. Every software will have to necessarily support these features; so if we chase these, we can test a software in a structural manner. Once you read the specifications of the software, see whether that requirement falls under create or read or update or delete. Once you identify that, you can blindly apply the following test conditions to that particular requirement.


If the requirement falls under Create (also termed as New, Add, Insert) category, you need to define test conditions for:

1. Creating a brand new item one at a time
2. Creating a duplicate item
3. Creating more than one items in a single shot (if applicable)

If the requirement falls under Read (also termed as View, Select, Open) category, you need to define test conditions for:

1. Reading an existing item one at a time
2. Reading more than one existing items at a time
3. Reading a non-existing item
4. Reading in a sorted order (such as sort a grid in ascending or descending order)
5. Read an item based on wild cards (such as search or filter items)

If the requirement falls under Update (also termed as Edit, Modify, Change) category, you need to define test conditions for:

1. Update an existing item one at a time
2. Update more than one existing items 
3. Update a non-existing item 

If the requirement falls under Delete (also termed as Remove, Trash, Erase) category, you need to define test conditions for:

1. Delete an existing item one at a time
2. Delete more than one existing items 
3. Delete a non-existing item 

So, if we apply this CRUD method to every single requirement, we start getting more and more branches for every such operation. 

What is the guarantee that most of the software operations are restricted to only this CRUD concept? If you look at any database, you can see just four important operations - SELECT, INSERT, UPDATE and DELETE. Whether it is Oracle or MySQL or MS SQL or DB2, all use these 4 operations day in and day out. And ultimately every software must deal with a database  to finally store the data. Hope this is quite a convincing argument.

For more in testing, please visit http://www.openmentor.net.

Equivalence Partitions

When we deal with a lot of data, it seems to be ever increasing to test and exhaust. Let us take a simple courier management application. Typical inputs are - from address, to address, weight of the packet to be couriered, any insurance to be used for the parcel. Based on these, the price is calculated by the system. Imagine you are the tester for this application, what kind of test conditions you will write?

Let us divide the problem into different segments. See the conditions below.

1. Sending a packet from your home to another house in the same street
2. Sending a packet from your home to another house in same zip code or area code
3. Sending a packet from your home to another house in another city, but within the same state
4. Sending a packet from your home to another house in another city, but outside the state
5. Sending a packet from your home to another house in another city, but outside the country

Do you see a pattern? We try to get a sample of each business scenario that can happen in real time. One sample is a representative of a lot of similar data. The trick is to identify as many number of good samples, as possible. Another set of samples would be based on the weight of the packet. Usually the charges would be x1 for 10-100 grams,  x2 for 101-250 grams, x3 for 251-500 grams, x4 for 501-1000 grams and so on. If we need to get samples for these different slabs of the packets, what values should I choose? Should I take 4 packets with weights 10, 101, 251, 501 grams each or 100, 250, 500, 1000 grams each? There is no hard an fast rule here - but the boundary values will always be taken care in your BVA test conditions. When it comes to Equivalence Partition, it is advised to take the middle values when it comes to numbers. So if I need to give 4 samples based on weight of the packet, I would choose 50, 175, 375, 750 grams each. These 4 samples clearly represent the 4 slabs of the packet weights.

Let us take a bank that takes deposits. There are many parameters that determine the interests on deposits - but let me take 2 key parameters - Interest Type - Compound or Simple,  and the other one is Deposit Period Tag - Daily, Monthly, Yearly. If we alter these 2 parameters, the interest varies. Look at the following options:

1. Simple interest, applied on Daily basis
2. Simple interest, applied on Monthly basis
3. Simple interest, applied on Yearly basis
4. Compound interest, applied on Daily basis
5. Compound interest, applied on Monthly basis
6. Compound interest, applied on Yearly basis
   

If we carefully look at the above samples, 2 parameters with 2 and 3 different values, resulted in 6 combination samples. Thus, when the number of parameters and values of the parameters keep increasing, the number of samples you have to choose will also keep increasing.


Exercise: Take a travel booking site. From city, to city, round trip/one way, adults/children count, accommodation required etc. will appear on the screen as input parameters. Arrive at the combinations of these as test samples based on equivalence partition technique.


For recorded sessions on software testing, visit us at www.openmentor.net.