Christian Shadis

Self-Directed Blog Post #3

Two common testing techniques in Black-Box Software Testing are Boundary Value Testing (or Boundary Value Analysis) or Equivalence Partition Testing (or Equivalence Class Testing). Both techniques examine the output of a program given its inputs.

Consider a function eligibleForScholarship which accepts two inputs, SAT score (for simplicity, only considering 1600 scale) and GPA (on a 4.0 scale) and returns in dollars the total reward a student is eligible to receive. At CompUniversity, a student will receive a scholarship if their SAT score is at least 1200 and their GPA is at least 3.5. The amount of that scholarship is determined by the criteria in the table below.

In addition to the SAT reward, additional money is given based on GPA.

For example, a student with an SAT score of 1450 and GPA of 3.8 will receive $2500 plus an additional 20% for a total of $3000. A student with an SAT score of 1600 and a GPA of 3.49 will receive $0.

If software tester wanted to write test bases based on Robust Boundary Value Analysis, they would arrive at the following set of test cases:

Boundaries:
SAT Score upper bound: 1600 (physical limit)
SAT Score lower bound: 0 (physical limit)
GPA upper bound: 4.00 (physical limit)
GPA lower bound: 0.00 (physical limit)

Disadvantages of Boundary Value Testing:
There are possible outputs that this set of test cases never checks (e.g. $3000, $3750). Additionally, this set of test cases fails to test the behavior inside the program, instead focusing on outermost limits.

If the software tester instead wanted to write test cases based off of Strong Normal Equivalence Partitioning, they would arrive at the following set of test cases:

Disadvantages of Equivalence Partitioning:
This set of test cases does not successfully account for behavior at the boundaries of valid inputs, nor does it account for the behavior of the program with every possible combination of inputs.

The disadvantages of boundary value testing and equivalence partitioning can be addressed by instead implementing edge testing.

Edge Testing

In Edge testing, we want to examine any point at which behavior changes, and create test cases for all of those ‘boundaries’, along with values close to the boundaries on either side.

The SAT edge cases can be listed as {-1, 0, 1, 1199, 1200, 1201, 1399, 1400, 1401, 1599, 1600, 1601}.
The GPA edge cases can be listed as {-0.01, 0.00, 0.01, 3.49, 3.50, 3.51, 3.74, 3.75, 3.76, 3.89, 3.90, 3.91, 3.99, 4.00, 4.01}.

Test cases can now be created by taking all pairwise combinations of edge cases (Worst-case). There are too many test cases to list here without taking up unnecessary space, but now our tests account for each and every possible combination of inputs.

Edge testing combines the best attributes from Boundary Value testing and Equivalence Partitioning into one comprehensive technique. Edge testing successfully tests the upper and lower bounds for each input, and it successfully examining inner behavior at each partition.

Edge testing has a major drawback: the number of test cases needed. In this example, there are 12 * 15 = 180 test cases, but it has unrivaled behavior coverage. The decision of which testing technique to use depends largely on the program for which the tests are being developed, but for a smaller number of partitions in the inputs, edge testing is a relatively simple but comprehensive technique for testing all possible behavior.

Articles used for my research:
https://professionalqa.com/equivalence-class-testing
https://artoftesting.com/boundary-value-analysis
https://www.mindfulqa.com/edge-cases/
Class materials from CS-443 at Worcester State University

From the blog CS@Worcester – Christian Shadis' Blog by ctshadis and used with permission of the author. All other rights reserved by the author.

By Christian Shadis

CS-443 Self-Directed Blog Post #2

In this post, I will compare the implementations of Robust Boundary Value Testing and Equivalence Partitioning using simple examples and actual JUnit code.

For the purposes of this post, you have been elected ChairBee of HoneyComb Council in your neighborhood’s local beehive. Your main job is to determine whether a worker bee has produced enough honey to retire. There are two criteria by which you base your decision, age and number of flowers pollinated. As a bee, your brain was incapable of making this complex of a decision every time a bee requests retirement, so you decided to write a Java program to do the heavy lifting for you.

Your function needs two inputs: the number of flowers pollinated, and the age of the bee in days. Your decision is based on these conditions: A bee is eligible to retire at age 18, but only if they have pollinated at least 20 flowers.

boolean readyToRetire(int age, int flowersPollinated) throws IllegalArgumentException{

   if(age < 0 || age > 50)
      throw new IllegalArgumentException();
   if(flowersPollinated < 0 || flowersPollinated > 200)
      throw new IllegalArgumentException();
   if(age >= 18 && age <= 50) {
      if(flowersPollinated < 20) 
         return false;
      else
         return true;
   }

   else
      return false;
}

Being the good little former working bee you are, you decide that no program is worth its weight in beetles (joke of an insect, really) if it doesn’t have a JUnit Test Suite to go along with it. You decide that you want to develop test cases with one of two methods: Boundary Value Testing or Equivalence Partitioning.

Boundary Value Testing

There are physical and arbitrary limits implemented on each input of the function. The age of a bee, obviously, cannot be a negative number, and for a creature with a life expectancy of 28 days, you cannot expect it to live past 50 days. Our acceptable inputs for age are 0 <= age <= 50. Likewise, a negative number of flowers being pollinated is impossible, and a bee in our hive is extremely unlikely to pollinate close to 150 flowers, let alone 200. Our acceptable inputs for flowers pollinated are 0 <= flowers <= 200. Thus, we implement a physical limit of 0 to both age and flowers pollinated, and arbitrary upper limits of 50 and 200, respectively.

In Robust Boundary Value testing, we take each boundary of an input, and one step in the positive and negative directions, and test each against the nominal (or ideal) value of the other input. We arrive at the following test cases:

Now we write JUnit 5 Test cases for each row in the table above. We will assume the readyToRetire function is inside a Bee class.

@Test
void test1(){
   Bee testBee = new Bee();
   assertThrows(IllegalArgumentException.class, 
               () -> { testBee.readyToRetire(-1, 30); } );
}

@Test
void test2(){
   Bee testBee = new Bee();
   boolean result = testBee.readyToRetire(0, 30);
   assertFalse(result);
}

@Test
void test3(){
   Bee testBee = new Bee();
   boolean result = testBee.readyToRetire(1, 30);
   assertFalse(result);
}

@Test
void test4(){
   Bee testBee = new Bee();
   boolean result = testBee.readyToRetire(49, 30);
   assertTrue(result);
}

@Test
void test5(){
   Bee testBee = new Bee();
   boolean result = testBee.readyToRetire(50, 30);
   assertTrue(result);
}

@Test
void test6(){
   Bee testBee = new Bee();
   assertThrows(IllegalArgumentException.class, 
               () -> { testBee.readyToRetire(51, 30); } );
}

@Test
void test7(){
   Bee testBee = new Bee();
   boolean result = testBee.readyToRetire(20, 30);
   assertTrue(result);
}

@Test
void test8(){
   Bee testBee = new Bee();
   assertThrows(IllegalArgumentException.class, 
               () -> { testBee.readyToRetire(20, -1); } );
}

@Test
void test9(){
   Bee testBee = new Bee();
   boolean result = testBee.readyToRetire(20, 0);
   assertFalse(result);
}

@Test
void test10(){
   Bee testBee = new Bee();
   boolean result = testBee.readyToRetire(20, 1);
   assertFalse(result);
}

@Test
void test11(){
   Bee testBee = new Bee();
   boolean result = testBee.readyToRetire(20, 199);
   assertTrue(result);
}

@Test
void test12(){
   Bee testBee = new Bee();
   boolean result = testBee.readyToRetire(20, 200);
   assertTrue(result);
}

@Test
void test13(){
   Bee testBee = new Bee();
   assertThrows(IllegalArgumentException.class, 
               () -> { testBee.readyToRetire(20, 201); } );
}

Equivalence Class Partitioning

Equivalence class partitioning arises from the foundational concept that with this function, there will be one of three results. An Exception will be thrown, true will be returned, or false will be returned. Based on our information about the criteria for deciding if a bee can retire, the behavior of the function can be modeled with a table.

Each cell in this table is an equivalence class and can represent a test case in the JUnit Test Suite. With your bee work ethic, you decide to implement Strong Robust Equivalence Testing, which is the most thorough method, and uses every cell in the table above as a test case.

@Test
void test1(){
   Bee testBee = new Bee();
   assertThrows(IllegalArgumentException.class, 
               () -> { testBee.readyToRetire(51, -1); } );
}

@Test
void test2(){
   Bee testBee = new Bee();
   assertThrows(IllegalArgumentException.class, 
               () -> { testBee.readyToRetire(20, -1); } );
}

@Test
void test3(){
   Bee testBee = new Bee();
   assertThrows(IllegalArgumentException.class, 
               () -> { testBee.readyToRetire(15, -1); } );
}

@Test
void test4(){
   Bee testBee = new Bee();
   assertThrows(IllegalArgumentException.class, 
               () -> { testBee.readyToRetire(-1, -1); } );
}

@Test
void test5(){
   Bee testBee = new Bee();
   assertThrows(IllegalArgumentException.class, 
               () -> { testBee.readyToRetire(51, 10); } );
}

@Test
void test6(){
   Bee testBee = new Bee();
   boolean result = testBee.readyToRetire(20, 10);
   assertFalse(result);
}

@Test
void test7(){
   Bee testBee = new Bee();
   boolean result = testBee.readyToRetire(15, 10);
   assertFalse(result);
}

@Test
void test8(){
   Bee testBee = new Bee();
   assertThrows(IllegalArgumentException.class, 
               () -> { testBee.readyToRetire(-1, 10); } );
}

@Test
void test9(){
   Bee testBee = new Bee();
   assertThrows(IllegalArgumentException.class, 
               () -> { testBee.readyToRetire(51, 30); } );
}


@Test
void test10(){
   Bee testBee = new Bee();
   boolean result = testBee.readyToRetire(20, 30);
   assertTrue(result);
}

@Test
void test11(){
   Bee testBee = new Bee();
   boolean result = testBee.readyToRetire(15, 30);
   assertFalse(result);
}

@Test
void test12(){
   Bee testBee = new Bee();
   assertThrows(IllegalArgumentException.class, 
               () -> { testBee.readyToRetire(-1, 30); } );
}

@Test
void test13(){
   Bee testBee = new Bee();
   assertThrows(IllegalArgumentException.class, 
               () -> { testBee.readyToRetire(51, 201); } );
}

@Test
void test14(){
   Bee testBee = new Bee();
   assertThrows(IllegalArgumentException.class, 
               () -> { testBee.readyToRetire(20, 201); } );
}

@Test
void test15(){
   Bee testBee = new Bee();
   assertThrows(IllegalArgumentException.class, 
               () -> { testBee.readyToRetire(15, 201); } );
}

@Test
void test16(){
   Bee testBee = new Bee();
   assertThrows(IllegalArgumentException.class, 
               () -> { testBee.readyToRetire(-1, 201); } );
}

Conclusion

We now have JUnit 5 Test Suites for both testing methods (boundary value testing and equivalence class testing). You are now prepared to serve your term as ChairBee knowing every decision will be made correctly, and having peace of mind that your test cases verified all desired behavior.

Equivalence class testing, at least with examples like this one, is a more thorough and robust technique that tests the programs behaviors as well as its input bounds, whereas boundary value testing focuses only on the extremes for each input. Each technique has its pros and cons, but I hope this simple example shed some light on implementing the theory behind boundary-value testing and equivalence class testing in JUnit 5.

From the blog CS@Worcester – Christian Shadis&#039; Blog by ctshadis and used with permission of the author. All other rights reserved by the author.

Boundary value testing is often seen as an ineffective testing methodology in comparison to other techniques. And while other techniques such as equivalence case testing might be better at testing in some situations opposed to boundary value testing, I believe there are some situations where boundary value testing is a better solution.

For instance, in a situation where you want to test that one particular variable (ie: price of a product in a product catalog) which should exist within a certain definable range (suppose price should stay between 49.99 and 28.99 depending on sales or coupons) then you have both a minimum and a maximum value already specified, and a nominal value can be as simple as the value halfway between the minimum and maximum values. In situations similar to this, boundary value testing has the potential to be much easier than some other testing methods, such as creating partitions and going through equivalence case testing.

On the other hand, in a situation where you want to evaluate a boolean value, boundary value testing makes no sense whatsoever. Since the output will on ever be true or false, there is no obvious specified minimum or maximum. And without a minimum or maximum value it would be difficult to find a nominal value which represents typical output for the variable/function being tested.

Additionally, when working with variables, sometimes the output may have an obvious boundary in one direction (ie: maximum age of a bond is 10 years) but no specified boundary in the other direction (no specified minimum age of a bond). In these cases, a limit will often have to be chosen as a minimum or maximum based off of the functionality of the software or what it’s purpose is (ie: bond cannot be less than 0 years old, can say minimum is 0).

This can be confusing at times when the upper limit is not bounded, forcing you to choose an arbitrary upper limit (ie: price is the amount someone pays for a product, specified must be greater than 0, but no specified maximum price) such as 10,000, 50, etc. This can be somewhat subjective and might vary significantly depending on the context of the program being tested, and can often introduce complications.

In terms of concrete advantages and disadvantages, this article provides a good summary of some of the major upsides and downsides (https://qtp.blogspot.com/2009/07/boundary-value-analysis-bva-problems.html).

Regarding disadvantages, boundary value testing is a poor choice for boolean values as previously discussed, and is also ineffective when working with dependent variables (if you try to use BVT with a dependent variable, you would likely also have to test for the independent variable associated which could be better accomplished through other testing procedures).

Conversely, the advantages of boundary value testing largely involve how simple it is to implement and execute, relative ease of finding lower and upper boundaries (often specified within the program being evaluated), and ability to use more or less test cases as needed (robust boundary value testing could provide a more comprehensive analysis than normal, and worst-case robust BVT could provide even more detailed information).

Because it is often dependent on the inclusion of obvious upper and lower limits, and can’t be used effectively with boolean values, the applications of BVT are not universal, but when it can be applied effectively, boundary value testing is easy to use and can provide representation of the majority of expected inputs.

Article referenced: https://qtp.blogspot.com/2009/07/boundary-value-analysis-bva-problems.html

From the blog CS@Worcester – CodeRoad by toomeymatt1515 and used with permission of the author. All other rights reserved by the author.

Software testing often involves the process of writing tests in order to ascertain the proper functioning and capabilities of software, within expected operating conditions. When coming up with test cases for a calculator program for instance, you might find it important to test that the addition of two numbers results in a correct sum following the operation.

Testing against the expected value of two arbitrarily chosen numbers (ie: 5 + 4 should equal 9) would be able to provide insight into whether the calculator is functioning as intended, but you could also choose to test for addition of negative numbers, negative and positive numbers, addition resulting in 0 and so on. Because there could be a large number of potential cases to check for, it might seem unclear what exactly would make for an optimal or objectively “good” test in this situation.

When looking into this concept, I found an article produced by Ministry of Testing (https://www.ministryoftesting.com/) a community focused around software testing and related subjects. The article (https://www.ministryoftesting.com/dojo/lessons/designing-tests-what-s-the-difference-between-a-good-test-and-a-bad-test) discusses some attributes to look out for which could be indicative of good or bad tests during the process of creating tests for a piece of software.

The process described in this article follows a series of steps, first identifying and considering risks involved with the program, things which could go wrong and which could affect the program as well as people or entities dependent on the program (users, other programs which access the one being tested).

Following the identification of risks, forming test ideas is described in terms of asking questions. So good tests would need to answer questions about the software and it’s functionality, maybe entering in a negative number, or something which is not a number in terms of the calculator example. This allows for unexpected behaviors to be considered even when they might not seem readily apparent.

Next, the test should be conducted or executed, and depending on whether or not the test reveals any relevant information (do you learn anything from it?) the information gained can be applied to improve the program, or further testing may be required if the test did not reveal anything.

I would say that this process makes sense as a basic framework to follow when writing test cases. In some cases, such as with very simple applications or non-commercial development, the risk can be practically nonexistent; the risk aspect may not always apply to every situation. But in general, this article brings up some good points and provides a helpful sequence of processes which can be followed and repeated as needed.

Article referenced:

https://www.ministryoftesting.com/dojo/lessons/designing-tests-what-s-the-difference-between-a-good-test-and-a-bad-test

From the blog CS@Worcester – CodeRoad by toomeymatt1515 and used with permission of the author. All other rights reserved by the author.

Decision Tables have been one of the tricker concepts for me to understand and work with, and was one of the tougher sections on the recent exam for me. I think part of it is me mixing some of the terms up, such as at the conditions and rules, which leads to weird looking tables or I would just get stumped. I wanted to read more articles about decision tables to better understand what I else I might be missing and how to improve.

I understand the main concept of using the tables as another way to show different combinations of values for the variables, with the conditions acting similar to the previous boundary and equivalence classes that we have been working with, and actions simply being the output. So I think it’s the idea of the rules and what goes in those columns that is what I am having trouble with.

At first I though that rules were the classes and was confused on what the conditions were, but after understanding the difference between them, I had a better grasp on making the tables. I was able to set up the conditions and actions after this, and now I wanted to focus on understand rules and what is put in the columns.

The rules themselves are similar to case numbers from boundary and class testing, where they identify each combination of input and output values. Because of these, we are usually able to group rules together that have several input values matching a condition and have a shared output , such as multiple values that fall in the range between 0 and 10 that match an action.

I have a better understanding of decision table testing now after doing more reading on the subject from other articles and activity 8 and seeing my mistakes with my understanding of rules, possible input and output values, and the conditions and actions.

https://www.guru99.com/decision-table-testing.html

Decision Table Testing

From the blog Jeffery Neal&#039;s Blog by jneal44 and used with permission of the author. All other rights reserved by the author.

During my introductory programming classes, I tended to think that simply running my code and using random values as input to see if any errors popped up was more than enough to ensure that my code was (at least at the time) sound. Perhaps, in ideal circumstances this could be an ideal testing method. After all, if I had an inkling of an idea of what is considered a valid or invalid input, I could simply use some somewhat random input values, see what happens, and call it a day. It works, right?

As tempting as this may be, I am still working on understanding how functional software testing works, what techniques exist and can be used, and how to implement the tools that are created specifically for functional testing. In fact, I did talk about JUnit testing in a previous post, which can automate the process of testing for valid and invalid inputs that we provide to the software. If we were to just use random values without having a specific pattern in mind while testing software, what conclusions we would end up with by the end of testing would not mean much in the long run.

One efficient technique of testing for valid and invalid inputs is Boundary Value Testing or Boundary Value Analysis (two terms that I may be using interchangeably), a software testing technique for which test cases are being written based on what the programs define an acceptable range of inputs to be. To be more specific, when using boundary value testing, a tester may use the values at the boundaries of a range as inputs. In essence, we use the range or set of acceptable values for any variables in the code as the test cases, rather than simply coming up with random valid or invalid values at our own discretion. Moreover, to conduct boundary value testing we take values near and at the boundaries of a defined range [min, max], with some such values being:
1) max and min: the values at the extremes of a range
2) max-: the value just below the upper limit of a range
3) min+: the value just above the lower limit of a range
Boundary value analysis can include more values near the extremes of a range based on the needs of specific test cases, thus increasing the testing complexity, though the above mentioned are actually very common among all variations of boundary value testing, which may even include values that fall outside of the valid range. The above values fall within the acceptable (or valid) range of values, thus if they are provided as inputs during testing, we can expect to get reasonable outputs and behaviors from the software in return.

Here is an example of boundary value analysis using mathematical sets:
Suppose we have the variable x of type int that has acceptable values within the range [a,b]. This essentially means that we can define the valid range of x as {x: x ≥ a and x ≤ b}, meaning that any value within and including a and b will result in the program behaving in a reasonable manner. Any value that fits the invalid range of {x: x < a and x > b}, on the other hand, may result in the program throwing exceptions.

A simple example of boundary value testing being used in JAVA is the following:

Though boundary value analysis may not absolutely prevent errors and bugs from occurring while testing software, it is an extremely good point of reference that a tester can use when they are testing code. If one expects the software to behave in certain ways depending on specific value intervals, it can be fairly easy for the tester to adjust their approach around such specific intervals and have a better understanding of the software they are testing and how it should work.

Articles/Websites I read that discuss the topic of this post in greater detail:
1) https://celestialsys.com/blog/software-testing-boundary-value-analysis-equivalence-partitioning/
2) https://www.eviltester.com/2019/01/what-is-boundary-value-analysis.html
3) https://qualitance.com/blog/boundary-value-analysis-and-equivalence-partitioning/
4) https://blog.qatestlab.com/2016/02/19/boundary-value-testing/

From the blog CS@Worcester – CompSci Log by sohoda and used with permission of the author. All other rights reserved by the author.