One of the last types of testing techniques that we learned about this semester is known is Combinatorial Testing. Combinatorial testing is a testing technique that is used for software applications with a lot of different input possibilities and a high complexity. Even if you create a large number of different test cases, you will most likely still miss a test scenario. I’m not the best at explaining what something is, so I did some research in order to find an article that helps describe the aspects of combinatorial testing as well as how to use it and what its benefits are. This website is called testsigma.com

I think that this website does a great job at explaining what combinatorial testing is as well as all of its different benefits. In the article by Shanika Wickramasinghe, it states “Combinatorial testing is a testing method that uses multiple combinations of input parameters to perform testing for a software application. The main goal of combinatorial testing is to make sure that the software product can handle different combinations of test data as input parameters and configuration options.” This means that combinatorial testing takes a bunch of different input parameters, similar to pairwise testing, and uses it to test a bunch of different cases for the program. This can be extremely useful because some of the errors with a program can only be found with specific inputs. I’ve actually had this happen to me before in one of my classes. I wrote a program and testing a bunch of different inputs and they all worked, but when my teacher tried an input I never used, it failed. I think combinational testing is going to be extremely useful for me in the future. I know it seems like combinatorial testing can only be used in certain scenarios so it might be better to not learn it, but it actually has a lot of benefits according to Wickramasinghe:

• Covers a broad range of input combinations using a minimum number of test cases.
• Increases test coverage compared to normal component testing since it always considers multiple input combinations.
• Helps to detect bugs, defects, vulnerabilities, and unexpected outputs that might not be detected during the usual component and regression testing phases.
• Reduces testing effort, cost, and time. (Since combinatorial tests use fewer test cases to cover a wide scope of testing.)
• Identifies issues at the earliest while allowing the team to address and fix those earlier in the software development life cycle.
• Optimizes the testing process by removing unwanted test cases while ensuring that the cost and effort are not wasted on repeating the same test scenarios again and again.
• Helps to test complex software applications with a large number of parameters, settings, and options.
• Reduces the risk of critical defects going unnoticed, which can occur only when handling specific input combinations.

I recommend that everyone tries to use combinatorial testing at least once so they know how it works in case they ever need to use it again in the future to make sure all of their different input possibilities work.

Link: https://testsigma.com/blog/combinatorial-testing/

From the blog CS@Worcester – One pixel at a time by gizmo10203 and used with permission of the author. All other rights reserved by the author.

During this semester we have learned about many different testing techniques, some being a lot easier than others. One of the testing techniques that we have learned is called Stochastic Testing. Stochastic testing is a type of black box testing method in which random tests are conducted over time. These tests are performed by automated testing tools in order to see if the software can pass a large number of individual tests.

Another type of testing technique that we have talked about is Property-based Testing. Property-based Testing is very similar to Stochastic testing. In this testing technique, the testing is also automated. However, it isn’t just executing the tests that is automated, it is also generating the tests. While researching the topic, I came across an article called “What is Property-based Testing?” by Alex Robert. In this article, Robert states “Property-based testing automates that work for us. Test automation allows us to generate better tests with less work and less code, so that we can focus our effort on the less mechanical work developers excel at. Instead of writing tests with manually created examples, a property-based test defines the types of inputs it needs. In the example above with CommonPrefix, the input would be two strings. The property-based test framework will generate hundreds if not thousands of examples and feed them to your test function.” He talks about how property-based testing can create a ton of different examples for your test function in order to help you test your program.

In this article, Robert also talks about input generators and what they do. A generator is a function that returns an instance of a given type from a source of randomness. These generators are great for randomly creating new inputs for the test function. However, they are only efficient if the generators have a good framework in order to help uncover more issues with the code. Robert gives a list of qualities that he thinks would make a good generator. His qualities include: a generator should be fast, a generator should be deterministic, a generator should not waste randomness, and a generator should cover the code under test. Most of these property-based testing frameworks come with a built-in generator which can be extremely helpful for testers, depending on the type input parameter. I really enjoyed reading this article as looking at all of the examples given by Robert helped my understanding of the topic. I know that if I ever need to use this type of testing technique, I will definitely use this article as a reference.

Link: https://www.mayhem.security/blog/what-is-property-based-testing

From the blog CS@Worcester – One pixel at a time by gizmo10203 and used with permission of the author. All other rights reserved by the author.

Another important topic that we have discussed in CS-443, or Software Quality Assurance and Testing, is known as Pairwise Testing. Pairwise Testing is yet another form of testing, but this type is a little bit different than the rest. Pairwise Testing, sometimes known as all-pairs testing, tests each pair of input parameters in order to make sure that the functions in the system run correctly no matter what the input is, guaranteeing that it will run for every combination. Pairwise Testing is known as a Permutations and Combinations (P&C) based software testing technique. A blog that I found to be really helpful in explaining Pairwise Testing is known as Pairwise Testing | What It Is, When & How to Perform by Kiruthika D. In the blog, she gives an example that helped me understand more. She states “Let’s say you have an application that allows users to enter two numbers, and the application will output the sum of the two numbers. You can use pairwise testing to test all possible combinations of two numbers, such as (1, 2), (2, 3), (3, 4), (4, 5), etc. By testing all the combinations of two numbers, you can be sure that the application is working correctly and will not fail when given different numbers.” This shows that you don’t actually test every single combination, but you test every single input with another input. This way, it makes sure that all of the inputs work instead of testing a potentially infinite amount of combinations.

The actual purpose/use of Pairwise Testing is exactly what I previously stated. It is used to make sure that all combinations of inputs are possible, but you don’t need to test every single combination. It can be extremely helpful as it reduces the amount of time it takes to test the program as well as the amount of effort. While Pairwise testing is a great testing technique, you obviously can’t use it all the time as it involves pairs. As for when to use it, Kiruthika states “Pairwise testing is helpful when testing complex systems that have multiple input parameters and multiple possible values for each parameter. It can significantly reduce the number of test cases that need to be created while ensuring that all possible discrete combinations of parameters are tested. This can help reduce test case creation time and cost and improve the software’s overall quality. Pairwise testing is not appropriate for all types of software testing. As we discussed, it is most effective for systems with multiple parameters and multiple possible values for each parameter. If a system has only a few parameters and a small number of possible values for each parameter, pairwise testing may be unnecessary. Pairwise testing, also, will not be useful if the values of inputs are inappropriate.” Essentially she is saying that Pairwise testing is used for functions that have multiple parameters with multiple values, and the order of parameters doesn’t matter. On top of that, depending on the type of parameter, the technique might not work either. While I personally don’t see myself using this technique in the future, I think that it has the opportunity to be very useful in certain situations, so I’m glad that I was able to understand it more in case I ever need to use it.

Link: https://testsigma.com/blog/pairwise-testing

From the blog CS@Worcester – One pixel at a time by gizmo10203 and used with permission of the author. All other rights reserved by the author.

The last few weeks of class have all been about Test Driven Development. Test Driven Development is when you use tests in order to guide you into developing software. According to a website called martinfowler, there are three main steps that we follow repeatedly. Those steps are:

1. “Write a test for the next bit of functionality you want to add.
2. Write the functional code until the test passes.
3. Refactor both new and old code to make it well structured.”

Essentially what this means is you create a list of things that you want included in your test, then you create a test for just one of those items. Once you create the test, you implement the code in order to get the test to pass. Once it passes, then you move on to creating the next test and getting that test to pass. If it causes the previous test you fail, you refractor until all of the current tests pass, then you repeat until all of the items you need are done. I personally find writing the test first to be a lot easier than writing the code first. That way, while you’re creating the code, you have an example to look back at so you know exactly what you want it to do. On top of that, focusing on one part of the code at a time makes it a little easier to develop the code without making mistakes, as you aren’t focusing on the entire program at once, just one small part at a time. According to the article, this form of development has two main benefits. It says “Most obviously it’s a way to get SelfTestingCode, since we can only write some functional code in response to making a test pass. The second benefit is that thinking about the test first forces us to think about the interface to the code first. This focus on interface and how you use a class helps us separate interface from implementation, a key element of good design that many programmers struggle with.” I chose this article as my source because I thought it did it a really good job explaining what to do step by step so you don’t get confused. On top of that, it explains the benefits and consequences of this method and how to avoid/achieve them. This article helped me further understand Test Driven Development and why it is useful.

Source: https://martinfowler.com/bliki/TestDrivenDevelopment.html

From the blog CS@Worcester – One pixel at a time by gizmo10203 and used with permission of the author. All other rights reserved by the author.

A little while ago we learned how Define-Use Testing works as well as how to use it ourselves. However at the time, it still confused me a bit and I wasn’t really sure what it was used for. After reviewing the previous work we’ve done and coming back across this topic, I realized that I still didn’t fully understand what was going on. I decided to look further into it and to do some research in order to understand it more. As a result, I found a website called testsigma which I believe does a really good job at explaining what Define-Use case testing is, its advantages and its disadvantages, and when to use it.

With the website’s help, I’m going to explain everything that you need to know about Define-Use case testing in order to understand it and to know when to use it. As for what a use-case itself is, the website states “The concept of a Use Case can be likened to a real-life scenario in which users and their objectives are present. A Use Case outlines the events unfolding when these users interact with the system to achieve their objectives. The Use Case allows for a thorough understanding of the overall behavior and functionality of the system from the user’s point of view.” Essentially this means that a Use Case shows all of the decisions that were made while the user was trying to achieve their outcome, allowing you to understand all of the behavior and functionality from the user’s perspective. Use Case Testing was designed to make sure that the system that is currently being tested meets all of the expectations. It essentially simulates different real-life scenarios in order to make sure the software does everything that it is supposed to. While the website gives a lot of different advantages and disadvantages of Use Case testing, I’m just going to list a few. Some advantages include helping to identify and validate the functional requirements of a system or software, explaining how the software will be used in real-world scenarios, and helping uncover potential defects. Some disadvantages include it could be time-consuming and require a lot of effort, it may not cover all scenarios which could leave to gaps in test coverage, and it relies on the accuracy of the use case documentation. As for when to use Use Case testing, the website gives a checklist which I found to be extremely useful in my understanding of when to use it. I’m really glad I found this website as it further helped my understanding of the topic, and it has helped me prepare for my final exam. The checklist states:

“1. Review the software requirements, specifications, design documentation, and use case scenarios. 

2. Identify the possible scenarios related to the use case. 

3. Determine the functional requirements of the use case. 

4. Create test cases for each possible use case scenario. 

5. Define the expected results of the use case scenarios. 

6. Execute the test cases and compare them with the expected results. 

7. Retest the fixes after identifying failed test cases. 

8. Check for exceptions and errors for each use case scenario. 

9. Verify compliance and security requirements for each use case scenario. 

10. Monitor the system’s overall functionality across use scenarios. 

11. Validate the overall system performance with user expectations”

From the blog CS@Worcester – One pixel at a time by gizmo10203 and used with permission of the author. All other rights reserved by the author.

Recently in my Cs-443 class, Software Quality Assurance and Testing, we have been talking about unit testing using stubs. Honestly, I didn’t understand what stubs were during the classes themselves, but a blog called Why Stubs in Unit Testing Improve Integration Testing helped me understand what a stub is, as well as where and when it is used. In this blog written by Miroslaw Zielinski, he states ” Unit testing is more about isolating the function, method, or procedure, otherwise referred to as a unit. This isolation is done by stubbing out dependencies and forcing specific paths of execution. Stubs take the place of the code in the unit that is dependent on code outside the unit.” Stubs allow the person testing the code to test specific paths of execution within the code, allowing them to view different things such as if different parts of the code work and are reliable. I think that Zielinski gives a great example of why these stubs are used in his article. He states, “Once the stub is added, it will be consistently applied to the tested code. A user working on the “allocation failure” test case will have an easy way to install a special callback function into the stub, which will simulate the desired effect: allocation failure or do nothing since by default the stub returns a null pointer, which is expected for the test case.” This helped me understand why stubs are used, and the rest of the article helped me understand when they are used.

After reading more of the article, Zielinski began to explain why stubs are used. Although his explanation confused me a bit, I believe I got the jist of what he was trying to say. I believe that they are being used when you are trying to test lower-level modules while the upper-level modules have not been developed yet. Therefore you can use a stub, which acts like an upper-level module, in order to see how it will be treated. Zielinski also gives examples of when stubs aren’t good to use and what the limitations are “In instances where there’s not an original definition available for a stubbed function, what happens? How does the stub behave without a callback that defines the alternative behavior? The beauty of C/C++test is that it automatically detects this kind of situation. The stub will reconfigure itself during the test harness build time. It won’t call the original definition when no callback is installed and will return a safe default value.” I’m not really sure what he means by this, but I hope to learn more in the future.

Link: https://www.parasoft.com/blog/using-stubs-in-integration-level-testing/#:~:text=Unit%20testing%20is%20more%20about,on%20code%20outside%20the%20unit.

From the blog CS@Worcester – One pixel at a time by gizmo10203 and used with permission of the author. All other rights reserved by the author.

During these first few weeks of class, we have been talking about JUnit and how it has features that allow us to test our code. We mainly talked about annotating methods to use as test cases as well as different assertions from Java and third-party libraries in order to help determine if a test was successful. While we kept using it throughout the week in order to help us test our code, I wasn’t exactly sure what JUnit actually was. Luckily when I was doing some research on it, I came across a blog on Parasoft by Nathan Jakubiak which gave a lot of useful information on what unit testing is, what JUnit is, as well as how to set up JUnit testing and to write different tests. I think that Jakubiak gave great descriptions as to what each of them are and what they are used for. Firstly, he stated “Unit testing is a form of white box testing in which test cases are based on internal structure. The tester chooses inputs to exercise particular paths through the code and configures assertions that validate the output. The purpose of unit testing is to examine the individual components or pieces of methods/classes to verify functionality, ensuring the behavior is as expected.” This means that unit testing is a type of testing where test cases are created based on certain parts of the program that you want to test, rather than testing the entire code in one test. For JUnit testing, Jakubiak stated “JUnit is the most popular Java unit testing framework. An open-source framework, it’s used to write and run repeatable automated tests.” Essentially he is just letting us know that it is an open-sourced framework that allows us to run different tests for our code. In his blog he also talks about how to set up JUnit testing which I personally don’t understand, probably because I just started working with it. However later in his blog he talks about all of the different parts of a JUnit test and how to write them. He also gave examples for some of the different parts, such as annotations, which I found to be really helpful in my understanding of the subject. I chose this Jakubiak’s blog because in all of the research I did on the topics of JUnit and Unit Testing, his blog did the best job of explaining them and overall gave me a much better understanding of the topics.

Link: https://www.parasoft.com/blog/junit-tutorial-setting-up-writing-and-running-java-unit-tests/

From the blog CS@Worcester – One pixel at a time by gizmo10203 and used with permission of the author. All other rights reserved by the author.