When testing a software product the testers need access to some amount of information. The can be given access to just the finished product, just the code, or both. In each case the type of testing can vary.

Black-box testing refers to testing software where you have not been given the source code. In most cases the tested would be given a set of specifications. The tested the preform test on the  product and ensure it complies with the specifications. The main way in which black box testing works is by providing input and verifying the output is what is expected.

With white-box testing the testers have access to the source code. This means the they can examine how the code works. This can prove to be more complex but allows for much more thorough testing. This is where unit testing can really work and test cases can be written to test the code. Metrics can also be generated to show how much of the code was actually covered by test.

The third options involves a little of both categories, this is known as gray-box testing. In most cases involving gray-box testing the tested don’t have access to the source code but do have some knowledge of the inner workings of the program. One example of this would be a team that understands the formulas and methods used by software. This allows them to create more specific test than black-box testing that can target areas that are more likely to have issues. This is a good place for boundary value testing.

From the blog CS@WSU – :(){ :|: & };: by rmurphy12blog and used with permission of the author. All other rights reserved by the author.

One way of ensuring good quality is by creating a test suit that actively test the code to fine flaws. Another way is to comb through the code and verify there are no errors in the logic. This process in known as a software technical review.

A software technical review consists of 4 parties, the producer, the recorder, the review leader, and the reviewers.

The producer is the entity that develops the code this could be a single developer and development team or an outside company. They are responsible for providing a frozen copy of the code and a set of requirements and expectations. It is important that the code does not change during the technical review process.

The recorder is responsible for taking notes and documenting what takes place during the review.

The review leader is responsible for scheduling the review, suppling what is necessary for the review, as well as producing the final report and sharing the findings.

The code reviewers as well as the leader and the reported are responsible for looking through the code before the meeting and documenting all errors of other issues found in the code.

During the review each member shares what they found and compares it to the others. As a group the determine if what is found is actually an issue that needs to be fixed. The team does not make suggestions to how the issues can be resolved.

The review leader then takes the information from the review session and generates a final report to be shared with the producer.

From the blog CS@WSU – :(){ :|: & };: by rmurphy12blog and used with permission of the author. All other rights reserved by the author.

Most software development companies don’t start with a fresh idea of there own and see it through to the end. In many cases a client will contact a development software company with and idea for a project the would like built to fill a certain need of theirs. In this type of situation there must be communications between the client and the development team. Most of the time the developers themselves wont have specific knowledge in the area of the new software.

This process starts with getting the client to form a set of requirement and expectations. This process can be aided by the developer by asking questions to clarify what the client is looking for. Even with a great plan and list of what the client is looking for it the final program doesn’t always do what is really wanted.

I have seen this first hand working a a small Internet company. There has been several times the owner of the company has gone to the development team with and idea. As a team they listen to what the owner is looking for and repeat their interpretation of it. They continue this back and forth until both side are satisfied. The development team then works to produce the discussed program. After the program is complete the final product is presented to the owner. Usually everything is all set but in some cases the response is “this doesn’t do what I want it to”.

There are several reasons that can lead to situations like this. First is an unreal expectation. Some projects are to complex or just not feasible. Some times it is because the client didn’t, themselves, really know what they wanted. Other times it is because the developer didn’t understand the requirements completely.

In any case, the situation could benefit for acceptance testing along the way. As the development team starts to develop starts to produce a product they can go back to the client and show them the current status to ensure it is what they are looking for. On method that would help is the use of Wireframes. I have written about this is a previous article.  https://rmurphy12blog.wordpress.com/2016/10/29/wireframe-testing/

From the blog CS@WSU – :(){ :|: & };: by rmurphy12blog and used with permission of the author. All other rights reserved by the author.

Integration testing is one of the three major categories of testing. It falls between unit testing and functional testing. Integration testing is the step when you begin to test the individual parts of a program with each other. This can be a tricky task as there are often times when not all the required parts are developed.

There are three ways of preforming integration testing:

~ Top Down

~ Bottom Up

~ Big Bang

In top down integration testing you start with the top level and work your way to the bottom. As you start with the top many lower levels have not been coded yet. For these you write stubs that hold the palace and provide canned answers. These stubs do just enough so that you can ensure you upper levels work. A disadvantage of this method is the amount of throw away code required for the stubs.

The next approach is bottom up. This is, as it sounds, the opposite of top down. In this method you start by testing the lowest level parts and work your way up. Like in the top down approach there is a certain amount of throw away code required. The difference is instead of stubs you must write drivers. These drivers are used to call the necessary actions to ensure the low levels are working. This method leaves on main issue, design flaws in the project are usually not discovered until a lot of effort has been put in.

The Final method of integration testing known as the big bang approach. This involves waiting for most of the code to be developed and integrating all at once. This is good because there is no throw away code but it becomes very hard to identify issues.

From the blog CS@WSU – :(){ :|: & };: by rmurphy12blog and used with permission of the author. All other rights reserved by the author.

When talking about software testing there are different levels to consider. These different stages of testing go along with the different stages of development. It is obvious you can’t test the final program while only several small parts have been coded. And it would not make sense to wait until you have a completed project to start testing. The two processes need to work together. These different levels of testing can be summarized by three categories; Unit Testing, Integrations Testing, and Functional Testing.
I like the cell phone example Sushma S. uses in her blog.

~ Unit testing: the battery is checked for its life, capacity and other parameters. Sim card is checked for its activation.

~ Integration Testing: battery and sim card are integrated i.e. assembled in order to start the mobile phone.

~ Functional Testing: the functionality of the mobile phone is checked in terms of its features and also battery usage as well as sim card facilities.

Unit testing is the lowest level of testing. As a developer begins to code the parts of a system it is important to test that the it preform as expected. These individual parts are tested in units. Once there are several units developed one must ensure they work together. This could include having access to the necessary information as well as making sure the output of one part lines up with the input of the next. The final level of testing is functional testing. This is where the overall product can be tested to see if it operates in the way it was intended and to see that if satisfies all the needs of the end user.

http://www.softwaretestinghelp.com/the-difference-between-unit-integration-and-functional-testing/

From the blog CS@WSU – :(){ :|: & };: by rmurphy12blog and used with permission of the author. All other rights reserved by the author.

Many programs are interactive and require input from the user to function. In a perfect world, the input provided by the user would be exactly what is required, this is obviously isn’t a safe assumption. There are some many ways input can cause issues to the program. Many times invalid input will cause a program to return an incorrect answer or behave in an unexpected way. Other time it will cause and exceptions that causes the program to crash. In more serious cases, like buffer overflow type attacks, these issues with input can cause security risks. Because of this programs need to be able to validate and check input values. Most programmers are aware of what may be entered and program defensively as to catch the invalid input and prompt the use to make a correction.

This being said it is hard to think of all the input could be entered incorrectly. To check to see how your program handles the different types of input it is useful to try entering input and seeing what happens. This is very time consuming and suffers from the same flaw, its hard to come up with invalid input to test. This is where fuzz testing comes in. Fuzz testing, commonly referred to as fuzzing aims to do this automatically. The fuzzing software will attempt to generate input of all sorts to see how the program responds. This input can include items like numbers random letters and symbols, it also tries very large input. With the informations provided by the testing a programmer can make changes to the code to better catch the issues before it is an actual issue.

From the blog CS@WSU – :(){ :|: & };: by rmurphy12blog and used with permission of the author. All other rights reserved by the author.

Mutation Testing

It is one thing to use the many techniques to come up with a set of test. There is software that will go through you code and test cases and provides a measure which show how much of your code is actually covered by the tests. This is a great start but how do you know if your tests are really helpful. That is to say are do the test do a good job at testing the code and would they pick up on errors. To really judge how strong your test suite is you need to be able to test your test. One way of doing this is by purposely making changes to your code and seeing if your tests catch it. This is called mutation testing.

Manually going through your code would be very time consuming and tedious. The benefits would not be worth the effort. Luckily there are programs that automate this process. I personally looked at Pit Mutation Testing. It can be found at pitest.org.  In the case of Pitest it works with Java and the software directly modifies the byte code which makes the process much faster as you don’t need to edit the source then recompile.

The code systematically works through the code and makes changes, these changes are referred to as mutants. It then runs the tests to see if these mutants survive or are killed. It then generates a report that shows what lines are covered by the test as well as what mutations it generated. For each mutation it will show if it survived or was killed. In the case of a surviving mutation it will indicate whether it was tested and survived or if it was simply never tested.

Some of the mutation it tries include changing logical operators, changing the the operations in equations and returning null.

these changes include changes logical statements.

From the blog CS@WSU – :(){ :|: & };: by rmurphy12blog and used with permission of the author. All other rights reserved by the author.

A lot of software testing focuses on testing the code, which is very important, but there is another area of testing to consider, the user experience. When it comes to designing a product it is important to keep in mind who will be using it and what they are expecting. This area of testing ranges depending on the nature of the software. If, for example, you are developing software that is to be server based and will be run from command line it will require less time than a program that is to be used by the average employee.

It is a good idea to start this testing before you have invested too much time in developing the final user interface. A quote by Frank Lloyd Wright that summarizes this is “You can use an eraser on the drafting table or a sledge hammer on the job site.”

One way to accomplish this is to make use of wireframe testing. Wireframe testing is a method of testing that involves generating a mock up so that the end user can get a feel for what the final program will look like and how it will behave. There is also a variation where an expert analyzes the software instead of the end user.

Wireframes can vary in details. The simplest form may be a simple sketch the allows the user to see how items are laid out. More detailed wireframes fall into to categories high fidelity and low fidelity. Low fidelity wireframes are mock ups that provide a fairly similar look to the final product and some functionality where as high fidelity wireframes are very similar to the final product in look and feel.

Original Articles:
http://curioustester.blogspot.com/2015/08/wireframes-testing-part-i.html
http://curioustester.blogspot.com/2015/09/wireframes-testing-part-ii-how-did-i-do.html

From the blog CS@WSU – :(){ :|: & };: by rmurphy12blog and used with permission of the author. All other rights reserved by the author.

When it comes to estimating the amount of time a project will take one must consider the subproblem, how much time will it take to properly test the code. This turns out to be a very difficult task. There are so many variables that influence testing time.

One of the large factors in determining how long it will take to test a particular project is the team. How large is the testing team? How often does the team work together? Do members come and go? What is the skill level of each of the members? Is this the teams only task? All of these questions affect how efficiently a testing team can work together which in turn can help determines how long testing will take.

Another large factor in figuring out testing time is the complexity of the code. Obviously the more complex the code the more complex the testing process. Some programs just need to be tested to show the can take in a set of data and properly process it. Other programs have many possible paths though the code and each one must be tested.

The quality of the code that needs testing plays a role in the testing process as well. If the development team produces high quality code that is not laden with bugs the test process can move faster, but if there are many errors in the code the test team may be held up as the developers work out the bugs or will need to take the time to correct the errors them selves.

You must also keep in mind the importance of an error free program and the consequences of errors. Based on the answers to these questions you can determine what level of coverage is required. If the code is for a piece for medical equipment or part of a flight control system the consequences can be very severe. Cases like these obviously demand much more attention than the code for, say, a video game.

To actually start putting a number on the time for testing, it is a good idea to start logging data for each project. This data can then be looked at to help predict future projects. It can also help to show areas where the process can be bettered which will lead to a more efficient process and shorted estimates.

Original article: https://www.stickyminds.com/article/why-estimating-software-testing-time-so-difficult

From the blog CS@WSU – :(){ :|: & };: by rmurphy12blog and used with permission of the author. All other rights reserved by the author.