Testing Part I - the Problem Description
CS 330 - Spring 2008 - Dr. Liffick

Goals of Overall Assignment
- to practice using procedural dataflow design techniques
- to consider software development as a process beyond implementation
- to think about the Testing problem before you develop it
- to practice developing software using good development techniques
- to explore resources and find answers to implementation details on your own

Overview
The project is not difficult. Don't make it harder than it is. It doesn't require complex data structures; use arrays and strings as appropriate. It does use command line arguments, and your program must ensure that the correct number of arguments are available before attempting anything else. It also manipulates files and creates commands that are executed within your program.

You will need to find answers to some implementation details in part II of the assignment. You should be able to look for and find answers to many of your questions.

The testing program is a simplified "oracle" for determining whether an executable program gives the expected answers for provided test cases. Command line inputs to the program are the file name of an executable program, the name of a directory containing input data files (one file per test case), and the name of a directory containing expected output data files (one file per test case with file names matching the corresponding input data file). Output is a report on any time the actual output from the provided executable program does not match the expected output for a test case. There should be a report for each failed test case. Matching ignores white space. I'm expecting you to read input from the actual output and expected output as strings and note when the strings to do not match. A value of 3.5 will not match a value of 3.50. The wording of prompts and labels will have to match exactly.

We might enhance this program later in the term. Therefore, remember to separate concerns and localize details of representation. All aspects of the problem will be fair game for modification. Plan accordingly. Use plenty of functions/methods and pass information through parameters.

Example
For example, suppose I have a program to compute the square of its input. All of the items in this paragraph exist outside the testing system you are developing. I compile my program and store its executable version in a file named square. I think about test cases that include both input and output. I have three test cases named four, negThree, and two to remind me of their contents. I create a directory named sqInput with three files (named as above) with each one containing the input part of the test case. I create a directory named sqOutput with three files (named as above) with each one containing the expected output part of the test case.

To use your testing system (named testing), I might use this command: 

   testing  square  sqInput  sqOutput
Suppose my program has a bug that causes the program to give the wrong answer when the input is negative. So when your system checks my program's output on the test cases, the actual output for the negThree input will not match the expected output. Your program will report that it doesn't match. If you wish, it may report that the outputs do match on the other tests, but that is your choice.

Note that we won't be running your testing programs through themselves because I am not being picky about the specific output. Also, the programs that can be tested with this system perform their input and output only through standard input and output, and the testing system uses command line arguments.