Goals
- to write specific repeatable black-box tests for a program
- to develop white-box tests that cover the code's branches
- to show that you can think carefully about a program and prove
that it matches its specification
Parts of the Assignment
40% - creating a black box test set for checkout program
30% - building test sets that cover the conditions
of the checkout program
30% - writing an axiomatic proof of the z = x + y program
Overview
We've discussed white-box and black-box testing.
Although we could use JUnit within Eclipse, some of you have not
set up Eclipse in your own accounts. Instead, we'll take a different
approach and do the testing
exploration on cs in C++.
You'll use the checkout problem for the two testing parts
and an entirely different program for the proof.
The Checkout Problem for Testing
Consider the checkout program described by Dr. Hutchens in the attached
handout and here.
I adjusted this from his CS 161 assignment.
You do not need to write the program. I did that.
You will find my answer in
checkout.cpp.
Like a good student, I executed the program with the provided data.
It prints the sample output.
Black Box Testing
Read through the specifications for the program. Develop several good tests.
You should have at least ten tests but may have more.
Remember that a good test has a high likelihood of revealing a previously unknown bug.
Make your tests diverse.
Realize that you need both item files and order data as input. You may use an item
file in multiple tests. For each test, describe the input, the expected output,
and what bug you are trying to reveal.
If it seems appropriate, you may describe some item files and order data instead
of listing the actual contents, but for most tests, giving the actual contents
is essential.
You need them to be in files to run them.
You are not proving that the program is correct or even showing it is correct.
You are trying to uncover bugs.
Run the program with your tests.
For example, if you put the sample input file into itemfile and the
order information into a test1 and the executable program is in a.out,
run the program by typing
./a.out < test1
Note whether the actual output matched your expected output.
I will grade these on the suitability of your tests as well as your description of them. Try to break my program. But don't look at it. This is supposed to be black box testing where you don't look inside.
White Box Branch Coverage Testing
You have a set of tests you developed above an stored them in files
so that you could keep track of them.
Now, instead of working only from the program specification to create
your tests, look at the program, and see if you can execute every branch
of the program.
It's possible that the tests you have from above are enough.
Refer to the handout about coverage and especially that you compile the program
with
-fprofile-arcs -ftest-coverage options.
Get a summary report using gcov -b sourceCodeName.
Look at the report in sourceCodeName.gcov.
To start over, remove *.gcov and *.gcda files.
In doing this part of the assignment, look at your tests and choose the ones
you think will cover the most branches. Describe why you think those
will cover many of the branches. Then run them. How did they do?
Note that I care only about the information about the program I wrote.
You'll get information about many of the STL files.
The summary may state that 100% of the branches are executed without both
directions taken on the branch. You have to look at the checkout.cpp.gcov file
to see which branches were taken.
I think you can get 100% branch coverage with only a few tests.
When you get 100% or as close as you believe possible, print the checkout.cpp.gcov file, and describe which tests you used.
I will grade these on your description of which tests you chose and why, how close you got to 100% of branch coverage, and having relatively few tests to get there.
Axiomatic Proof
Use an axiomatic proof to show that the following program matches its
specification in that when its precondition P is true, after it executes,
its postcondition Q is true.
z := y;
i := 0;
while (i < x) {
z := z + 1;
i := i + 1;
}
P = (x > 0, y > 0)
Q = (z = x + y)
Loop invariant I: z = i + y and i <= x
Show that the loop terminates.
Show that the invariant and the loop condition applied to the loop statements give the invariant.
Show that the precondition P for the program is strong enough, when applied to the initialization statements, to establish the precondition for the invariant.
Show that the loop invariant and the negation of the loop condition imply the postcondition Q for the program.
Collaboration
This assignment is to be completed on your own.
It is NOT a group assignment.
Check it out early so that if you get stuck, you can ask me questions.