For my final blog post for my Software Testing course, I wanted to go over something that also ties with other projects I’ve been working on during this semester in some way. In our Softawre Development Capstone team, we’ve had a couple of issues where we’ve had to touch upon the Thea’s Pantry continuous integration system through GitLab to have linter tests run automatically when pushing commits, so that commits that don’t match up to specification, the pipeline fails and the person who pushed those changes gets notified to fix any issues that arise. In this post, I want to go over a bit more detail about what continuous integration is and how it’s useful for developers.

According to Stephen Roddewig’s blog post on HubSpot, contiguous integration is an approach to development where code changes are regularly merged into a shared repository, built into a test application automatically, and the results of running this test application go back to the developer if any bugs or defects are found.

In practice, a continuous integration tool typically is an automated system where the source code is compiled with tests being automatically run on every individual push from a developer. This means that the developer’s contribution to the project is automatically tested on push, then either pushed forward to the maintainers or pushed back as a result of a pipeline failure.

The benefit of this is clear: bugs are caught quicker, the source code is updated on a regular basis with greater confidence because of the automated tests, and the pipeline provides an explicit and clear vision of what the specifications are for the project.

While working with the Thea’s Pantry system for my capstone, I could see these benefits in action. If a developer forgets to run tests locally (in the case of the Thea’s Pantry system, all tests are run in a script, and linters are run in their own script as well), the pipeline will catch any problems seamlessly, and the developer can easily see the pipeline failure, look at the output on GitLab, and determine what they need to fix in their branch.

In addition, it clarifies the specifications of what commits should look like, and what code should look like as well, on the basis that you can even add linters to the pipeline. It’s very useful in case someone forgets that we use conventional commits to have our changes be more clear with what they do, as the pipeline will detect that and function as a reminder of what things should look like.

All around, continuous integration is always a benefit for everyone involved in the software development process, creating a smooth system for testing that functions automatically rather than requiring developers to remember to run tests so they don’t mess up their branch.

From the blog CS@Worcester – V's CompSCi Blog by V and used with permission of the author. All other rights reserved by the author.

The blog post titled “Pairwise Testing: A Complete Guide” by Rajkumar (https://www.softwaretestingmaterial.com/pairwise-testing/) delves into the concept of pairwise testing, also known as all-pairs testing. I chose this blog because of its in-depth examples of  how to implement pairwise testing. These examples really made it easier to understand how pairwise testing can be used to reduce the amount of test cases while still maintaining high test coverage. The post also outlines the importance of pairwise testing and its advantages. Additionally, it compares pairwise testing with other testing techniques, highlights tools available for automating pairwise testing, and discusses its application in various scenarios.

What is Pairwise Testing?

Pairwise testing is a black-box testing technique used to reduce the number of test cases while still maintaining high test coverage. By focusing on combinations of input pairs, pairwise testing ensures that all possible pairs of input parameters are tested at least once. For example, if you had a program that took three inputs, X, Y, and Z, pairwise testing would create test cases for each combination of input pairs: (X, Y), (X,Z), and (Y,Z). At first glance this may seem like you have to create a ridiculous amount of test cases, but by following the steps for pairwise testing you can see how quickly the number of test cases is reduced.

Applying Pairwise Testing

The first step of pairwise testing is to identify all of your input variables. Next list all possible values for each variable. Variables that have numeric values may be reduced to valid and invalid to begin reducing the number of test cases needed. Then, by creating a table content columns of each variable and its possible outputs, you can get a list of each test case. 

This blog’s examples use tables to illustrate how each test case is created by filling in each column one by one.

The example from the blog uses a bookstore and creates a table of every variable. In this example you can see how each pair of variables covers all unique combinations except the book category and enquiry. These variables only test fiction with valid and non fiction with invalid. The other columns fixed this by simply reversing the order of one of the test pairs, but doing this would create the same issue for another combination of variables. In this instance two more test cases are added to the list to ensure the unique combination of book category and enquiry.

From the blog CS@Worcester – CS Learning by kbourassa18 and used with permission of the author. All other rights reserved by the author.

Like my previous blogs about performance and AI tools, I wanted to discuss another area of software testing that covers a specific development aspect. I chose security testing because while performance and functionality are essential, we must ensure our applications are safe for their users. This blog post, “Security Testing: Types, Tools, and Best Practices,” by Oliver Moradov, will outline various aspects of security testing, its importance, and its methodologies.

Oliver’s blog starts by defining security testing, listing its main goals and principles, and providing an index for many other aspects of security testing. It stresses that security testing is a non-functional testing process that determines the resiliency of software from cyber-attacks and data breaches. Non-functional testing focuses on how the system works rather than if it works. The primary goals are identifying assets, threats, vulnerabilities, and risks and performing remediation. 

The blog then discusses the types of security testing, including penetration testing, application security testing, API security testing, and vulnerability management. It also covers security test cases, and security testing approaches such as black box, white box, and grey box testing. The blog finishes by discussing best practices in security testing, such as shifting security testing left, testing internal interfaces, and automating tests.  

 I felt the distinction between functional and non-functional testing was important to discuss. Knowing how our data or the company’s data is handled is much more important for security than making sure the software functions correctly. It could function perfectly within the development team’s parameters but still fail to protect the company’s and users assets. 

I found it interesting that black box, white box, and grey box testing are applied differently than how they’re typically used in unit testing. The fundamental uses are the same, limiting and granting information, but there is much more to be gained when using this method in security testing. Testing what a malicious hacker could do with limited or abundant details on the software would be critical to protecting our assets. It also highlights the difference between non-functional and functional because developers use box testing to determine how to make errors occur within the system. In contrast, security tests would use the system functionality itself to create a breach in security. 

The sections detailing what security vulnerabilities they look for were enlightening. The authentication, input validation, and application logic sections highlighted what scenarios the security tests may encounter to validate system safety. I’m glad the blog discussed tools for developers to ensure their code can’t be exploited. Overall, this blog gave plenty of rudimentary information on security testing, and I urge those interested to check it out. In the future, I plan on using the tools and testing techniques to ensure that my code is safe for its users and owners. 

Blog: https://brightsec.com/blog/security-testing/

From the blog CS@Worcester – KindlCoding by jkindl and used with permission of the author. All other rights reserved by the author.

During the time I’ve spent in my software development concentration in my computer science studies (and even in general), we’ve mostly been concerned with the software development life cycle, where we focus mostly on getting a finished product as efficiently as possible that matches specifications, and build it up better and better over time. There is an interesting counterpart to this in the software testing life cycle, which is technically a part of the software development life cycle, but has it’s own specific steps.

In this post, I will be referencing this blog post from Testim on the STLC.

The point of STLC is similar to the point of SDLC at its core, getting a functional testing suitebased on specifications. The end goal has to do with finding problems and reporting them, however, rather than having a functional piece of software, which makes sense considering that testing is a step toward that piece of software.

The software testing life cycle is split up into 6 phases:

1. Requirement Analysis: Understand what the product should do, prioritize issues and brainstorm potential solutions (and whether they can be automated) with the team.
2. Test Planning: This is where the scope, tools and objectives are set for the following phases. It’s similar to a sprint planning meeting where tasks are assigned, time is estimated and issues are weighted.
3. Test Case Designing and Development: This is where the tests are, well, designed and created based on the specifications and priorities set up from previous phases.
4. Test Environment Setup: Software is ran on different configurations and setups to determine levels of performance and minimum requirements. We want to make sure our software works well on all possible configurations where it would be used, making a smoother experience for the end-user.
5. Test Execution: The tests are actually run all together, and the results are logged with details, and rerun with any changes to the main project as needed. Automated testing tools are preferred, as it makes this process significantly more refined.
6. Test Closure: Evaluate the testing result, taking into account things like test coverage, quality, and review the testing process. This is analogous to a sprint review, where the team comes together to review the results.

In an agile environment, these phases should all be covered in every sprint. All things considered, this is a necessary step in having working, quality software, as without a good testing environment your software could behave unexpectedly, and bugs will be more obfuscated.

From the blog CS@Worcester – V's CompSCi Blog by V and used with permission of the author. All other rights reserved by the author.

This post I will cover integration testing and why we use it today. Integration testing is a critical phase in the software development lifecycle, focusing on the integration of individual components into a cohesive system. It ensures that various modules or subsystems work together as intended. One of the primary challenges in integration testing is ensuring comprehensive coverage of interactions between different components. Identifying the right integration points and scenarios to test can be complex, especially in large-scale projects with numerous dependencies.

Selecting appropriate test cases to validate integration points is crucial. It requires understanding how components interact and designing tests to simulate these interactions effectively. Failure to cover all integration scenarios may lead to undetected defects, impacting the reliability and functionality of the software.

Moreover, integration testing often involves testing across different environments and platforms, adding to the complexity. Ensuring compatibility and consistency across various configurations is essential for delivering a robust product. One of the primary hurdles is achieving comprehensive test coverage across all integration points. Prioritizing critical integration points and designing effective test scenarios are essential to address this challenge.

Another challenge is managing the dependencies and external services during integration testing. Mocking or simulating external dependencies may be necessary to isolate various parts for testing, but it can also introduce its own set of challenges, such as maintaining realistic testing environments.

Furthermore, integration testing requires coordination among development teams working on different modules or services. Synchronizing changes and ensuring compatibility between components can be challenging, particularly in agile or distributed development environments.

Frameworks like Selenium are helpful for automating web browser interactions to test integrations between web components. For broader integration testing needs, companies might choose tools like Katalon Studio, which offers a comprehensive suite for web, mobile, desktop, and API testing. Additionally, some companies leverage enterprise-grade solutions like IBM Rational Integration Tester that provide robust features for complex integrations and compliance requirements. Ultimately, the choice of tool depends on the specific needs of the project and the company’s development environment.

Integration testing verifies the interactions between software modules, ensuring they function seamlessly as a unified system. Unlike unit testing, which examines individual components in isolation, integration testing focuses on how these components integrate and communicate with each other. It plays a crucial role in detecting issues arising from the integration of diverse elements, such as incompatible interfaces or conflicting behaviors. By identifying and addressing these issues early in the development process, integration testing helps prevent costly errors from surfacing in production. It’s step towards delivering reliable, high-quality software that meets user expectations and business requirements.

Blog Post: https://www.opkey.com/blog/integration-testing-a-comprehensive-guide-with-best-practices and https://www.testlearning.net/en/posts/integration-testing

From the blog CS@Worcester – Computer Science Through a Junior by Winston Luu and used with permission of the author. All other rights reserved by the author.

When I was in the process of learning how to build a PC as well as planning out what parts to choose, I had to weigh my options on the scales of price and performance. Some time after building it and using it almost daily, I was confident that my shiny new PC could handle nearly everything that I was using it for. I have yet to run into any major performance issues despite how my last sentence sounded but the fear of that still lingers in my mind when I try or want to try pushing my machine a bit. To prevent these worries, performance testing is done on software applications. 

Performance testing “is a type of software testing that ensures software applications perform properly under their expected workload.” It is meant to test and measure a system’s performance “in terms of sensitivity, reactivity, and stability,” using metrics such as response time, scalability, and resource usage (GeeksforGeeks). Doing so ensures that a system can handle the expected workload efficiently and effectively. 

Performance testing can be done on multiple levels such as on the application itself, the system running the application, and other levels that go a bit out of my knowledge.

There are many types of performance testing but the three that seemed most common and simple were load testing, stress testing, and endurance testing. Load testing involves testing the product’s performance under expected loads. This could include simulating user experiences through testing with different tiers of hardware, using the application for a certain amount of time, and doing things that a user may do. This ensures that the application is accessible to as many people as possible as hardware limits and mild activity will not do anything to the application nor the system. Stress testing involves putting the application under extreme loads, pushing to and even beyond its limits. This helps to identify the point in which the product breaks or affects the system, and gives an idea of what to optimize or fix in order to avoid breaking as much as possible. Endurance testing involves putting the application under stress for extended periods of time. Simply running the application over the course of a few hours, days, and possible weeks helps to identify memory leaks and how it handles a different type of load. 

Performance testing is really interesting to me as, on my PC building journey, I was very much interested in the performance of computers and how some applications require better and faster components to enjoy it at its true maximum. I, though not on the level of actual performance testing, have tested my own machine to see its limits, how it handles the maximum of certain applications, and how well it handles my use. Luckily, I’ve had very few crashing of applications, freezing, or other issues. 

https://www.geeksforgeeks.org/performance-testing-software-testing

From the blog CS@Worcester – Kyler's Blog by kylerlai and used with permission of the author. All other rights reserved by the author.

Security testing is a form of testing which is rapidly growing with consumers’ needs for security on the internet. As the article states, cybersecurity is becoming very important when you are online not only for personal reasons but also for business purposes. Many companies now use multiple online resources just to run their business day to day which may include payment systems, payroll, database management or any other range of services which are offered to companies through different platforms in order to help business owners run their business as efficiently as possible. These platforms have to be trusted so that consumers continue to use them and recommend them to friends, family, etc. Some examples provided of security issues within an  application are a student management system being insecure if the admission branch of the system can edit the exam branch, online shopping malls or any online storefront is not secure if the details from users credit cards or other payment methods is not encrypted as this opens up the users to credit card fraud, and even custom software can have security issues which is highlighted by the articles example of an SQL query retrieving the actual passwords associated with users accounts. Two tools recommended in this article for security testing are Invicti which is a web application that is used to scan both legacy and modern applications and Indusface which includes scanners for web, mobile and API applications. There are also different techniques involved with security testing some of these being access to application which involves going through the different roles in a system one by one ensuring they all only have access to what they should, data protection which similarly to access to application is meant to ensure that a specific user/role cannot see an aspect or menu of the application they are not supposed to and error handling which ensures detailed error messages cannot be used to aid in hacking.

Security testing while not being something we directly worked on during the semester it would have been interesting to work with as the many different types of security testing and the many different risks associated with application security. This type of testing seems to be much more manual in many aspects versus directly writing test cases as we did in most cases but being able to test for access or test the permissions of a role in a system would be very interesting to work with.

From the blog CS@Worcester – Dylan Brown Computer Science by dylanbrowncs and used with permission of the author. All other rights reserved by the author.

https://www.opentext.com/what-is/performance-testing#:~:text=Performance%20testing%20is%20a%20non,up%20under%20a%20given%20workload.

Performance testing is a form of testing in software development which is called non functional testing. Non functional testing means that the software’s function is not directly tested during this type of testing and in turn this makes a large majority of people view performance testing to be an afterthought or unimportant compared to other types of testing. Performance testing is specifically responsible for testing how a system or software may perform under a heavy amount of traffic or stress provided by many requests or users concurrently using the system. Some of the reasons that an organization may choose to undertake performance testing are to ensure a specific amount of users can be handles (for example 1000 concurrent users), to locate bottlenecks within an application which hinder performance or result in errors, to ensure that software is performing up to the standard that was given by the softwares vendor and to measure general stability as traffic goes up and down. Performance testing can require many different steps depending on the type of application being tested, the tester must make sure to look at the environment they wish to use for testing as well reading documentation on the environment or systems hardware in order to ensure the proper environment is used as performance testing may or may not involve testing within the production environment. The tester must also decide what is deemed acceptable performance wise which may involve meetings with the product owner or production team in order to set the correct standards for your tests and then you must also plan and design your original tests for performance. These tests may be all that you need, but if you require further testing due to necessary changes in a system if the system is originally unable to meet the requirements you will have to redesign and re-run your tests.

I feel as though performance testing is a lot more important than some people may think, especially to very large companies or social media platforms as they need to be able to accommodate a certain number of users not only everyday but also a certain number of users concurrently. Companies like amazon, microsoft, facebook, etc have to deal with thousands of customers or users at once and they can even reach millions of concurrent users which means they must do thorough performance testing in order to maintain the stability of their platforms and keep consumers happy.

From the blog CS@Worcester – Dylan Brown Computer Science by dylanbrowncs and used with permission of the author. All other rights reserved by the author.