Category Archives: Software Testing

If It’s Broke Don’t Fix it: Treating Bugs Like Buddha

For my last blog for this course instead of taking one
particular course subject, summarizing it, and theorizing about what actual
implementations may look like – I’d like to look at the whole course and do the
same. More specifically, I would like to cover how bugs or defects are actually
addressed, or not, in Software Development and Testing. To do this I have found
some interesting blog posts which argue that 100% of bugs may be able to be
fixed, but shouldn’t be. Instead, one should focus on serving the vast majority
of users under expected circumstances.

           Both blogs
focus on how impractical, and expensive, maintaining 100% stability or up-time
is. In the first focusing on bugs specifically they give many reasons why this
goal is unadvisable. The first is the prevalence of Agile Development,
dominating nearly the entire software development landscape. As such, the constraints
of this fast-paced development style limit the ability to do traditional QA testing;
if a program can have several revisions in a week, maybe even a day, then how
could a team reasonable test all these iterations. Instead, the author suggest
a stability monitoring tool to automatically test each revision.

           In
addition, they suggest that eliminating 100% of bugs would eliminate many which
users would never see, so why waste resources addressing them? Even if you
could fix everything how could you possibly know? This focus is reinforced when
considering the truly incredible breadth of devices that one may have consider,
and specifically in mobile development: where they cite the over 24,000
different android devices on the market. One must focus on the average expected
user experience and not waste time fussing with the outliers until, presumably,
a bug report is filed.

The second blog discusses
defects in systems in a similar way, covering more or less the same points, although
mentioning a rather obscure possibility: being legally challenged for claiming
that your product has no defects. Instead, what I believe they are trying to emphasize
is that products are always fallible, and the amount of resources required to
get them even close is impossible or impractical. As a result, as we move
forward in this class and towards graduation I think we should resist the impulse
to try for complete perfection and instead focus on what is achievable and
provides the best experience for the majority of users.

Sources

Not all bugs are worth fixing and that’s okay
The Zero Defect Fallacy

From the blog CS@Worcester – Press Here for Worms by wurmpress and used with permission of the author. All other rights reserved by the author.

Clarifying Edge Testing


I happened upon a very
interesting blog recently, searching for more examples of edge case testing,
wherein the author provides an interesting set of common edge cases. While they
use the same names or similar ones as those we’ve seen lately in class some
have new meaning here. In addition, they may have an actual value tied to the
name – such as zero, one, and two – but their values are defined otherwise.

 In order they
are as such:

  • Zero “represents
    any form of null input”, including the actual value zero but also null, an empty
    List or Array, et cetera. This of course is to test the capability of a program
    to deal with input that isn’t properly usable by it.
  • One represents what
    we have referred to as nominal up to this point, meaning a valid normal input
    which should test proper functionality of a program under ideal conditions.
  • Two does not
    correspond to the value two, but rather refers to testing the same code twice,
    usually in sequence, to see how repeated executions affect a system.
  • Two to Max-1 is
    most like one, in that it represents a nominal value as well, but in opposition
    this value should not be the absolute simplest needed to function but an average
    use case; meaning possibly complicated.
  • Max is
    fairly self-explanatory, used to test the upper limit accepted by a program, and
    can sometimes be an extreme value. As such, it can test the limits of the
    program under incredible load.
  • Max + 1 is
    used to ensure that limits placed on an application are working and that anything
    that does not correspond to a valid range is rejected in a reasonable manner.

These would most closely correspond to Normal Boundary Value Testing as we have covered in class but they are each less abstract than those counterparts. They provide an insight on what these values look like in actual QA testing, as well as expectations upon being used. Two for example does not have an actual value associated with it, but rather refers to a testing orthodoxy outlined above. One example of a test in this vein is found in another blog, in which the author proposes the edge case of the same user trying to log in from two different computers. I believe between the two of these blogs a clearer picture of the concepts we have covered can be found through these more concrete examples of the testing procedure.

Sources

A Beginner’s Guide to Testing: Error Handling Edge Cases
Build Strong Edge Test Cases

From the blog CS@Worcester – Press Here for Worms by wurmpress and used with permission of the author. All other rights reserved by the author.

Test Driven Development: Formal Trial-and-Error


Test Driven Development (TDD), like many concepts in
Computer Science, is very familiar to even newer programming students but they
lack the vocabulary to formally describe it. However, in this instance they
could probably informally name it: trail-and-error. Yes, very much like the
social sciences, computer science academics love giving existing concepts fancy
names. If we were to humor them, they would describe it in five-ish steps:

  1. Add
    test
  2. Run
    tests, check for failures
  3. Change
    code to address failures/Add another test
  4. Run
    tests again, refactor code
  5. Repeat

The TDD process comes
with some assumptions as well, one being that you are not building the system
to test while writing tests, these tests are for functionally complete
projects. As well, this technique is used to verify that code achieves some
valid outcome outlined for it, with a successful test being one that fails,
rather than “successful” tests that reveal an error as in traditional testing.
Related as well to our most recent classwork, TDD should achieve complete
coverage by testing every single line of code – which in the parlance of said
classwork would be complete node and edge coverage.

Additionally, TDD has
different levels, two to be more precise: Acceptance TDD and Developer TDD. The
first, ATDD, involves creating a test to fulfill the specifications of the
program and correcting the program as necessary to allow it to pass this test.
This testing is also known as Behavioral Driven Development. The latter, DTDD, is
usually referred to as just TDD and involves writing tests and then code to
pass them to, as mentioned before, to test functionality of all aspects of a
program.

As it relates to our coursework, the second assignment involved writing tests to test functionality based on the project specifications. While we did not modify the given program code, at least very little, we used the iterative process of writing and re-writing tests in order to verify the correct functioning of whatever method or feature we were hoping to test. In this way, the concept is very simple, though it remains to be seen if it stays that way given different code to test.

Sources:

Guru99 – Test-Driven Development

From the blog CS@Worcester – Press Here for Worms by wurmpress and used with permission of the author. All other rights reserved by the author.

Path of Most Resistance


In my last blog, I sought
to cover Integration Testing and in doing so we covered the two distinct types
outlined by Mr. Fowler. Of these, Broad Integration Testing (BIT to save time)
is most relevant to the next subject I wish to cover: Path Testing. BIT covers
the interactions between all ‘services’ within a program – meaning a program’s
completed modules are tested to ensure that their interactions match
expectations and do not fail some tests created for them. In this way, Path
Testing is very similar but with a focus on how paths through various aspects/modules
of a program, hopefully, work or do not.

           As
opposed to BIT, Path Testing (PT) seeks to identify not just the interactions
between modules, but instead any and all possible paths through an application
– and discover those parts of the application that have no path. The ultimate
goal is to find and test all “linearly independent paths”, which is defined as
a path that covers a partition that has yet to be covered. PT is made up of,
and can integrate, other testing techniques as well, including what we’ve
covered most recently: equivalence testing. Using this technique, paths can be
grouped by their shared functionality into classes, in order to eliminate
repetition in testing.

           When
determining which paths to take, one could be mistaken for wanting to avoid the
same module more than once; as stated previously, we are seeking paths we have
yet to take. However, it is very often that the same path must be taken, at
least initially, which leads to several modules. In fact, a path might be near
or actually identical to one that has come before it, but if it is required
that several values be tested along this path then it as well is considered
distinct as well. An excellent example of this made by the article I chose
states that loops or recursive calls are very often dictated by data, and
necessarily will require multiple test values.

           However, after this point the author begins to move away from the purely conceptual to actual graphs representing these paths, specifically directed graphs. While it was painful to see these again after thinking I had long escaped discrete math, they provide a perfect illustration for the individual modules you expect a path to trace through, as well as possible breaking points. Directed graphs represent tightly coupled conditions, and in this way they express how a program’s run in order and the cause and effect of certain commands upon execution. In this way, it offers a much more concise visual presentation of the testing process as opposed to something like equivalence testing. As well, these graphs are quite self-explanatory but I look forward to applying these concepts in class to actual code.

Sources

Path Testing: The Theory

From the blog CS@Worcester – Press Here for Worms by wurmpress and used with permission of the author. All other rights reserved by the author.

Static VS Dynamic Testing


In Software Testing, the two
most popular or important methods for testing are static and dynamic. While both
are obviously named, its good to go over the distinctions, and more importantly,
the actual implementations of both.

           Static
testing involves review of project documents, including source code, to weed
out any actual or potential security flaws and general mistakes. This takes the
form of Informal and Technical Reviews, Inspections, Walkthroughs, and more. This
process can involve an inspection tool or be performed manually, searching
for potential run-time problems, but not actually running the code. Dynamic testing
conversely involves actually executing code looking for functionality,
resource usage, and performance. We have been using dynamic testing, specifically
unit testing, in our class so far. Dynamic testing is used mainly to verify the
program runs in accordance with the specifications outlined for it, specifically
called System Testing.

           Of
course, both have their own advantages and disadvantages. In static testing you
have the potential to find bugs that may have made later development more
troublesome early in the process, which can save lots of time and frustration in
the future. However, there may exist some flaw that only a running program can
reveal, but it only in the code that is executed. It would seem to me, that
these should not be exclusive and in fact, they have a logical order.

           You
would begin with static testing, like proofreading a paper, searching for misspelled
words (poor variable/method names), run on sentences (needless complexity or
repetition), and reorganizing to best get your point across (logical order of
declarations/methods, documentation). Like writing a paper, it is best to do
this ahead of run-time (reading through), that way you don’t have to constantly
stop to bandage small errors. In the same way it seems essential to ‘proof-read’
your code before execution to make sure it is free from identifiable errors
before you move on to seeking out run-time errors. You wouldn’t want to have to
fix both at the same time like with proof-reading.

           In sum,
these two could be grouped as pro-active or reactive, static and dynamic respectively,
and best explain their use. As mentioned, these also make sense to utilize in a
specific order, ensuring code is as correct as possible upon inspection, then
running it to see where flaws undetectable in a static environment have arisen.
These together ensure quality software that is optimized and secure.

Static Testing vs Dynamic Testing: What’s the Difference?
Static Testing vs. Dynamic Testing

From the blog CS@Worcester – Press Here for Worms by wurmpress and used with permission of the author. All other rights reserved by the author.

Evaluating Software Testing Strategies

https://www.mitre.org/publications/systems-engineering-guide/se-lifecycle-building-blocks/test-and-evaluation

From the blog CS@Worcester – Caleb's Computer Science Blog by calebscomputerscienceblog and used with permission of the author. All other rights reserved by the author.

Software Testing With Security in Mind

For this weeks blog post I wanted to take a look at the security aspect of software testing.  I feel that we have discussed many aspects of software testing in our CS 443 class but one of the aspects we haven’t really gone over is how to determine whether code we write is secure.  Many of us, at some point in our carriers, will probably write software that will be used in a web/ desktop environment. By testing such code for vulnerabilities before it is released we can save ourselves and the companies we work for from falling victim to data breaches and stolen information. I found this article titled, How to Test Application Security – Web and Desktop Application Security Testing Techniques, and it discusses the issues I have just introduced.

The author of the article defines security as meaning “that authorized access is granted to protected data and unauthorized access is restricted.”  They then go on to distinguish between desktop and web-based software and the different security needs for both. Essentially, they suggest that both types of software require similar security measures to protect sensitive data, however, most web based software will require a little extra security measures since this type of software is accessible to anyone on the internet.

In the Article the author brings up a number of interesting points regarding testing how secure a piece of software is but I would like to focus on three of their main points as I feel they are really important. The three points I’d like to focus on are data protection, brut-force attacks, and SQL injections/ XSS.  To test for data protection in your software, the author suggests, you should ensure all passwords in your DB are being encrypted when they are transmitted. Also, if your software is web based, you should be using the HTTPS protocol rather than HTTP and you should test certificate validity on the server side. When it comes to testing whether your software is vulnerable to brut force attacks, the author says you should have some kind include “some mechanism of account suspension” into your software.  Finally, in order to test for SQL injections and XSS attacks we must treat any part of the code that accepts user input as a vulnerability.  The author advises that make sure there is a maximum length of characters for valid input as well as a checking mechanism for basic SQL injection techniques.

From the blog CS@Worcester – Caleb's Computer Science Blog by calebscomputerscienceblog and used with permission of the author. All other rights reserved by the author.

What to Consider When Doing a Technical Review on Object Oriented Code

For this week’s blog post I decided to write about important things to consider when doing a technical review on object-oriented code.  I chose this topic as I figured it would serve as a good refresher on key object-oriented concepts that we should keep in mind going forward with the next software testing project.  I thought that reminiscing on good practices of object oriented design would enable us to be more aware of what to look for while writing out software technical review on the Sir-Tommy Solitaire program.

I found a short book entitled Object Oriented Programing that was written by a software professional named Carl Erickson.  Erickson is also the CEO and founder of a software company called Atomic Object.  Although the book was written in 2009, I still think it is relative in the world of object-oriented design today.  There are many sections to Erickson’s book but I plan on looking at just a few sections that I thought were most relevant to writing our software technical review.  The sections I intend to focus on in this blog post are the following: sections 4 – OO Naming Conventions, 8 – Encapsulation & Modularity, 9 – Object-Oriented Hierarchy.

In section 4, Naming Conventions, Erickson briefly touches upon good practice for naming classes, methods, and variables.  He explains that class names should have the first letter of the first word capitalized along with the first letter of every word after that whereas, object variable and method names should have the first word begin with a lowercase letter and all following words begining with capitalized letters.  This probably isn’t news to any of use in CS-443 but it’s definitely something to consider when writing our software technical reviews.

Next, in sections 8 and 9, Erickson explores the ideas of class encapsulation and hierarchy.  Encapsulation is an important idea to consider when writing software technical reviews because improper use of encapsulation such as instantiating a variable as public when it should be private can lead to serious headache down the road.  For example, if something in the code were to go wrong with regards to a public variable that you instantiated, it may be very difficult to track down the culprit of the bug since the variable in question is not private. Also, Erickson brings up the notion of the “is a” relationship between classes, sub-classes, and interfaces which will be significant in determining the flexibility of the code in question.

It’s also worth reading over the rest of the sections in Erickson’s book. In particular sections 10 -14 as they go over essential concepts regarding to object-oriented design.  Section 13 does a good job of explaining how to evaluate whether or not a piece of object-oriented code is designed and implemented efficiently by focusing whether the code adheres to user specifications and object-oriented design techniques. Keeping in mind the foundational concepts of object-oriented design may allow us to make more insightful suggestions and write better technical reviews.

 

 

 

 

 

 

From the blog CS@Worcester – Caleb's Computer Science Blog by calebscomputerscienceblog and used with permission of the author. All other rights reserved by the author.

Code Coverage Alone Probably Won’t Ensure Your Code is Fully Tested.

For this week’s CS-443 self-directed professional development blog entry I read a blog post written by Mark Seemann, a professional programmer/ software architect from Copenhagen Denmark. The blog post I read is entitled “Code coverage is a useless target measure,” and I found it to be quite relevant to the material we’ve been discussing in class the past couple of weeks, especially regarding path testing and data-flow testing. In this blog post, Seemann urges project managers and test developers not to just set a “code coverage goal” as a means for measuring whether their code is completely tested or not. Seemann explains that he finds this to be a sort of “perverse incentive” as it could encourage developers to write bad unit tests for the simple purpose of just covering the code as the project’s deadline approaches and the pressure on them increases. He provides some examples of what this scenario might look like using the C# programming language.

In his examples, Seemann shows that it is pretty easy to achieve 100% code coverage for a specific class, however, that doesn’t mean the code in that class is sufficiently tested for correct functionality. In his first example test, Seemann shows that it is possible to write an essentially useless test by using a try/catch block and no assertions existing solely for the purpose of covering code. Next, he gives an example of a test with an assertion that might seem like a legitimate test but Seemann shows that “[the test] doesn’t prevent regressions, or [prove] that the System Under Test works as intended.” Finally, Seemann gives a test in which he uses multiple boundary values and explains that even though it is a much better test, it hasn’t increased code coverage over the previous two tests. Hence, Seemann concludes that in order to show that software is working as it is supposed to, you need to do more than just make sure all the code is covered by unit tests, you need to write multiple tests for certain portions of code and ensure correct outputs are generated when given boundary-value inputs as well.

The reason I chose Mark’s blog post for my entry this week was because I thought it related to the material we’ve been discussing recently in class, especially data-flow testing. I think it is important for us to remember, when we are using code based testing techniques, that writing unit tests to simply cover the code are not sufficient to ensuring software is completely functional. Therefore, it’s probably a good idea to use a combination of code and specification based techniques when writing unit tests.

From the blog CS@Worcester – Caleb's Computer Science Blog by calebscomputerscienceblog and used with permission of the author. All other rights reserved by the author.

Finding & Testing Independent Paths

Since we have been going over path testing in class this past week I decided to find a blog post relating to that material. The post I found titled, “Path Testing: Independent Paths,” is a continuation of a couple previous posts, Path Testing: The Theory & Path Testing: The Coverage, written by the same author, Jeff Nyman. In this blog post, Nyman offers an explanation into what basis path testing is as well as how to determine the number of linearly independent paths in a chunk of code.

Nyman essentially describes a linearly independent path as any unique path in  the graph that does not contain the same combinations of nodes as any other linearly independent path. He also brings up the point that even though path testing is mainly a code-based approach to testing, by assessing what the inputs and outputs should be of a certain piece of code it is still possible “to figure out and model paths.” He gives the specific example of a function that takes in arbitrary values and determines their Greatest Common Denominator. Nyman uses the following diagram to show how he is able to determine each linearly independent path:

I really liked how he was able to break down the logic in the form of processes, edges and decisions without looking at the code. I feel like sometimes when we are building our graphs strictly based on code it’s easy to get confused and forget about the underlying logic that will determine the amount of tests that are necessary to ensure our code is completely tested. It also helped me understand how basis path testing should work and how it should be implemented.

Nyman goes on by showing that he is able to calculate the number of independent paths using the above graph and the formula for cyclomatic complexity. First he points out that number of nodes is equal to the sum of the number of decisions and the number of processes, which in this case is 6. Then, by plugging numbers into cyclomatic complexity formula (V(G) = e – n +2p), Nyman was able to obtain the following results:

screenshot

Finally, Nyman ends the post by showing that the same results are obtained when going over the actual code for the Greatest Common Denominator function. He also shows that this same graph could be applicable to something like an amazon shopping cart/wishlist program. I think the biggest take-away from this post was that there is a strong relationship between cyclomatic complexity and testing which can prevent bugs through determining each linearly independent path and making sure they are producing the desired functionality.

October 1, 2017

-Caleb Pruitt

 

From the blog CS@Worcester – Caleb's Computer Science Blog by calebscomputerscienceblog and used with permission of the author. All other rights reserved by the author.