Category Archives: Software Testing

Mutation Testing

Mutation Testing

It is one thing to use the many techniques to come up with a set of test. There is software that will go through you code and test cases and provides a measure which show how much of your code is actually covered by the tests. This is a great start but how do you know if your tests are really helpful. That is to say are do the test do a good job at testing the code and would they pick up on errors. To really judge how strong your test suite is you need to be able to test your test. One way of doing this is by purposely making changes to your code and seeing if your tests catch it. This is called mutation testing.

Manually going through your code would be very time consuming and tedious. The benefits would not be worth the effort. Luckily there are programs that automate this process. I personally looked at Pit Mutation Testing. It can be found at pitest.org.  In the case of Pitest it works with Java and the software directly modifies the byte code which makes the process much faster as you don’t need to edit the source then recompile.

The code systematically works through the code and makes changes, these changes are referred to as mutants. It then runs the tests to see if these mutants survive or are killed. It then generates a report that shows what lines are covered by the test as well as what mutations it generated. For each mutation it will show if it survived or was killed. In the case of a surviving mutation it will indicate whether it was tested and survived or if it was simply never tested.

Some of the mutation it tries include changing logical operators, changing the the operations in equations and returning null.

these changes include changes logical statements.

From the blog CS@WSU – :(){ :|: & };: by rmurphy12blog and used with permission of the author. All other rights reserved by the author.

7 Best Practices for Agile Test Driven Development

Original Article

This article goes over seven different practices that will help with Test Driven Development when using the Agile approach.

The first thing that should be done is make sure you “avoid functional complexity.” The point of this type of approach is to keep it simple. The article suggests going over with your team to make sure that the test covers all the functionality that is needed.

The second thing is to make sure you have a clear understanding on what it is you are trying to achieve. It’s advised that you follow standard documentation and naming for your tests. This is so that developers are able to come back to it and understand what was the intended purpose for the set of tests.

The third item is that code is not overly complected. You want to make sure that your test is kept simple so that the code is easily readable and has room for any necessary improvements.

Fourth on the list is to make sure you are testing everything repeatedly. Always run tests before and after coding, and after refactoring since you never know when adding something new might break your tests. Also remember that when you refactor, to make sure that the node is still “maintainable and adheres to standards.”

Fifth point that is made is to “maintain code sanctity.” This means that you should be using version control tools to keep your code in check; which is extremely import if you have multiple developers working on the same code.

The sixth thing is making sure that your team is aware and understands how the application works. To do this make sure that the system documentation is clear and that all team members understand how the system and tests work. This will not only make sure that when new code is added, the overall program will not break, but also that new tests will work in the first run.

The last item that the article points out is to “know when to use TDD.” Test Driven Development is not suppose to be used for any testing that will take a long time. Always remember TDD primary function is to be used for projects that can be tested quickly.

From the blog CS WSU – Techni-Cat by clamberthutchinson and used with permission of the author. All other rights reserved by the author.

4 Warning Signs That Your Beta Testing Process Is in Danger

Original Article

In this article, Ran Rachlin goes over the four warning signs that may signal that your current beta testing process might be in trouble.

The first point that he talks about is if your beta testing is having huge delays. The main way Rachlin says to over come this obstacle is to set strict deadlines for the team (this includes testers). He also points out to make sure that the objectives for the deadline are clear and reasonable; this is to make sure that everyone is on the same page and no one gets worried about what needs to be done. The other thing that this step includes is to make sure that constant contact is made with the testers so that communication is not lost and you receive better feedback.

The second item Rachlin touches upon is making sure that you don’t underestimate how much time it will take to go through testing. If you are on a tight deadline to get your product out, he recommends a few things that you should do. One is to make sure you have experienced tests for your product. Two is to stay in contact with the testers (as mentioned above), and inform them of the “time crunch” they are dealing with. These steps will help make sure that the testers know what they are doing before hand and are aware of the challenges they need to overcome.

The third thing is to be aware of the well being of the testers you’ve hired. If you’re testers become frustrated with the product they are working on, this can hinder the speed of testing the application. Again it helps to keep contact with the testers to make sure they are on track with the work, but you also want to make sure that they feel appreciated for the work they are doing; especially if they are in a time crunch. It may also help to give incentives near the end of a deadline to encourage testers to finish early or for a job well done. Although you should always have backup plan just in case things become too much for the testers that you have.

The last point that Rachlin makes is to ask yourself “Are we testing the target market and devices?” He says that you must make sure of two things; that you make sure the testers you have are from the target market and that you are using the “most popular devices and carriers in this target market” for your tests. If all of these issues are addressed, you should have minimal issues with they beta testing process for your app.

From the blog CS WSU – Techni-Cat by clamberthutchinson and used with permission of the author. All other rights reserved by the author.

Coverage Testing

One of the trickiest things about software testing is knowing when we have tested enough. We could spend hours testing the software and believe that we have done a good job and later find have a bug show up that was triggered by an input that we did not think to test.

One way that we could quantify the amount of testing done is to use a coverage metric. The coverage metric is a score that based on the number of functions tested in the source code.

I used the EclEmma code coverage plugin for Eclipse and I used the code from my previous blog. The following was the output of the code coverage.

 

capture

One thing that I noticed was that the coverage metric only tests to see if a particular method was traversed by a test case and not whether the test case passes. One of the test cases could have failed and the test coverage metric would stay the same. Obviously!

From the blog Software Testing – The blog about software by Sudarshan and used with permission of the author. All other rights reserved by the author.

Mutation Testing

This week I decided to play around with mutation testing and the pitest system for mutation testing.

So what is mutation testing? The idea is to mutate your code and check to see if your tests still pass. If your tests fail that means that your tests were able to detect the mutant. In mutation testing terms this means that the mutant was killed. The quality of your tests can be gauged from the percentage of mutations killed.

If the test cases pass after the mutation, that may indicate an issue with the test suite. The whole idea of mutation testing is to test the effectiveness of the tests.

I decided to run the pitest tool on the tictactoe example I found on the Test-Driven java Development book by Alex Garcia.

I have attached a link to the code and the corresponding pitest results (https://drive.google.com/drive/folders/0B586m1QhLSUXQzBlWHVZNkxnMmc?usp=sharing). The mutation coverage was 95% and all the mutations were killed. I wonder why all of the mutations were killed?

From the blog Software Testing – The blog about software by Sudarshan and used with permission of the author. All other rights reserved by the author.

Exploring Software

Imagine you are giving a large software system to test, how do you test the system? Where do you start? Chapter 4 of the exploratory software testing book (by James Whittaker) answers this question.

The author uses the metaphor of the tourist. Suppose you visit a city like London. How do you go about touring the city? Do you walk around? Use the bus? Use a guide? There are many ways to travel a city and so does exploring a large software system. There has to be some sort of structure to help us achieve our goals.

The following are the goals of exploratory testing:
1) Gain an understanding of how an application works, what its interface looks like and what functionality it implements.
2) To force the software to exhibit its capabilities.
3) To find bugs.

The author develops a tourist guidebook for software to help us explore software. The author separates software functionality into overlapping districts as follows:
1. business district (The main part or “where stuff gets done” in the software systems)
2. Historical district (Legacy code)
3. Tourist district (Many cities have districts where only tourists go. Software is similar in                                       that novice users will use certain features that experienced users would                                        not.)

4.  Hotel district (Every tourist needs a place to rest. Software is actually quite busy when it                                     is “at rest”, and testing tours need to test those features)
5. Seedy district (Places where illegal and bad stuff happen. Seedy tours find places of                                              vulnerability in the software)

The author goes on to explain how to test each of these “districts” and what kind of bugs they might release.

Tours is mechanism to organize a tester’s thinking about how to approach exploring an application and in organizing actual testing. The tours help testers make the myriad decisions about which path to choose, inputs to apply, or parameters to select.

From the blog Software Testing – The blog about software by Sudarshan and used with permission of the author. All other rights reserved by the author.

Four main functional tests to ensure software quality (WEEK 11)

These post is about a collection of four most recommend tests that a developer must have knowledge. According to info-Tech report there are four functional tests that can ensure software quality. The six functional tests are unit test, regression test, system integration, tests  and acceptance test. 


To ensure every line of code executes correctly is recommender to use Unit Testing, Unit Testing is the process of testing each unit of code  in a single component. One unit is the smallest part in a software, one unit can be a single program or function.  


To ensure every function produces its expected outcome developers have to use Functional Testing. Functional testing indicates concerns about the correct decision of functional requirements. Functional testing is like a black box testing which don’t require any knowledge of the underlying implementation.


To ensure that all of the functions combine to deliver the desired business results developers have to use System testing. This type of testing executes end to end functional tests that can cross software units, in order to help become fully aware of the goals of assure that components deliver the desired business results. 


To ensure new changes did not adversely affect other parts of the system developers should use Regression testing. Regression  testing is used to ensure code modifications that have not accidentally introduced bugs into the system. When using Regression  testing is recommended to include plans from the original unit and system tests phases which can show that existing functionality behaves as planed. 


I wrote in my previous blogs posts about these four of these functional tests i really recommend you to read my previous posts because there you can find dip details about unit testing and etc.




Citation


http://searchsoftwarequality.techtarget.com/report/Six-functional-tests-to-ensure-software-quality






From the blog Table of Code by Andon S and used with permission of the author. All other rights reserved by the author.

Four main functional tests to ensure software quality (WEEK 11)

These post is about a collection of four most recommend tests that a developer must have knowledge. According to info-Tech report there are four functional tests that can ensure software quality. The six functional tests are unit test, regression test, system integration, tests  and acceptance test. 


To ensure every line of code executes correctly is recommender to use Unit Testing, Unit Testing is the process of testing each unit of code  in a single component. One unit is the smallest part in a software, one unit can be a single program or function.  


To ensure every function produces its expected outcome developers have to use Functional Testing. Functional testing indicates concerns about the correct decision of functional requirements. Functional testing is like a black box testing which don’t require any knowledge of the underlying implementation.


To ensure that all of the functions combine to deliver the desired business results developers have to use System testing. This type of testing executes end to end functional tests that can cross software units, in order to help become fully aware of the goals of assure that components deliver the desired business results. 


To ensure new changes did not adversely affect other parts of the system developers should use Regression testing. Regression  testing is used to ensure code modifications that have not accidentally introduced bugs into the system. When using Regression  testing is recommended to include plans from the original unit and system tests phases which can show that existing functionality behaves as planed. 


I wrote in my previous blogs posts about these four of these functional tests i really recommend you to read my previous posts because there you can find dip details about unit testing and etc.




Citation


http://searchsoftwarequality.techtarget.com/report/Six-functional-tests-to-ensure-software-quality






From the blog Table of Code by Andon S and used with permission of the author. All other rights reserved by the author.

Four main functional tests to ensure software quality (WEEK 11)

These post is about a collection of four most recommend tests that a developer must have knowledge. According to info-Tech report there are four functional tests that can ensure software quality. The six functional tests are unit test, regression test, system integration, tests  and acceptance test. 


To ensure every line of code executes correctly is recommender to use Unit Testing, Unit Testing is the process of testing each unit of code  in a single component. One unit is the smallest part in a software, one unit can be a single program or function.  


To ensure every function produces its expected outcome developers have to use Functional Testing. Functional testing indicates concerns about the correct decision of functional requirements. Functional testing is like a black box testing which don’t require any knowledge of the underlying implementation.


To ensure that all of the functions combine to deliver the desired business results developers have to use System testing. This type of testing executes end to end functional tests that can cross software units, in order to help become fully aware of the goals of assure that components deliver the desired business results. 


To ensure new changes did not adversely affect other parts of the system developers should use Regression testing. Regression  testing is used to ensure code modifications that have not accidentally introduced bugs into the system. When using Regression  testing is recommended to include plans from the original unit and system tests phases which can show that existing functionality behaves as planed. 


I wrote in my previous blogs posts about these four of these functional tests i really recommend you to read my previous posts because there you can find dip details about unit testing and etc.




Citation


http://searchsoftwarequality.techtarget.com/report/Six-functional-tests-to-ensure-software-quality






From the blog Table of Code by Andon S and used with permission of the author. All other rights reserved by the author.

Four main functional tests to ensure software quality (WEEK 11)

These post is about a collection of four most recommend tests that a developer must have knowledge. According to info-Tech report there are four functional tests that can ensure software quality. The six functional tests are unit test, regression test, system integration, tests  and acceptance test. 


To ensure every line of code executes correctly is recommender to use Unit Testing, Unit Testing is the process of testing each unit of code  in a single component. One unit is the smallest part in a software, one unit can be a single program or function.  


To ensure every function produces its expected outcome developers have to use Functional Testing. Functional testing indicates concerns about the correct decision of functional requirements. Functional testing is like a black box testing which don’t require any knowledge of the underlying implementation.


To ensure that all of the functions combine to deliver the desired business results developers have to use System testing. This type of testing executes end to end functional tests that can cross software units, in order to help become fully aware of the goals of assure that components deliver the desired business results. 


To ensure new changes did not adversely affect other parts of the system developers should use Regression testing. Regression  testing is used to ensure code modifications that have not accidentally introduced bugs into the system. When using Regression  testing is recommended to include plans from the original unit and system tests phases which can show that existing functionality behaves as planed. 


I wrote in my previous blogs posts about these four of these functional tests i really recommend you to read my previous posts because there you can find dip details about unit testing and etc.




Citation


http://searchsoftwarequality.techtarget.com/report/Six-functional-tests-to-ensure-software-quality






From the blog Table of Code by Andon S and used with permission of the author. All other rights reserved by the author.