Software Testing, Verification and Validation Karl J. Lieberherr COM 3220 Spring 1999 =========================================================================== Due date: April 22, 1999 This assignment is stored in file $TVV/hw/3 in file assign.txt On CCS Northeastern machines, TVV=/home/lieber/.www/com3220/ On the WWW, TVV = http://www.ccs.neu.edu/home/lieber/com3220/sp99 ========= The DJ home page is: http://www.ccs.neu.edu/research/demeter/DJ/ The AP Library home page is: http://www.ccs.neu.edu/research/demeter/AP-Library/ Connection between DJ and AP Library: DJ uses the AP Library. When you test DJ you test the code in the AP Library. A good question: is testing DJ testing all of the AP Library? The AP Library Specification home page is: http://www.ccs.neu.edu/research/demeter/biblio/strategies.html ASSIGNMENT 3 ============================================================= TESTING of a Java Productivity tool:DJ Testing along three loosely coupled dimensions. Building simple testing scripts for regression testing ============================================================== Use the following header for your electronic homework submissions and put it at the top of the message: Course number: COM 3220 Name: Account name: Assignment number: Date: Whenever you have questions about course material, please send e-mail to: mail lieberherr@ccs.neu.edu com3220-grader@ccs.neu.edu ccs.courses.com3220@ccs.neu.edu All messages which I send to the class are archived in /proj/demsys/logs-lieber/com3220 ========= READING: Read the web pages mentioned at the beginning of the hw. Find suitable chapters in the textbook that help you with the testing task. Which ones were most helpful? =================================================== The purpose of this homework is to have a concrete, interesting example where you can apply the testing process described in the text book. Use a black and white box testing approach (called greybox testing). Study the code for the AP library reachable from the AP Library home page to find better test cases. Your goal is to find enough test cases so that you achieve the following coverage: 100% statement coverage and 100% conditional coverage (each conditional is executed at least once for the two (or more) branches). We will measure test coverage for the AP library in a later homework. Here you only try to achieve good coverage through good test requirements and corresponding tests. The good test coverage should come for free. Write test requirements In this homework you have now access to a detailed specification for an important part of DJ: the specification that says how a strategy and a class graph defines traversals of objects. The AP Library Specification home page gives you the specification of how the traversals should be done. The specification of the rest of DJ should be in the DJ home page. If it is not there, list it as a bug you found that needs to be fixed. tests (run them) This time you automate running of the tests. bug reports scripts See page 11 of the text book for an overview. (no parts 1 and 2) PART 3: ======= This part is modified from the previous homework where you kept the strategy graph constant ("A->B"). In this homework you test for variability not only in the class graph and visitors but also in the strategies. Reuse your test cases from the previous hw as much as possible. Follow the instructions in the DJ homepage to learn about strategies. Here are some examples that you should test as a minimum: "from A via B to C" "from A bypassing B to C" "from A bypassing X via B bypassing Y to C" "from A bypassing {X,X1} via {B,B1} bypassing {Y,Y1} to C" "from A bypassing {Q, ->X,y,Y, ->Y,x1,X1} via B bypassing {Y,Y1} to C" "from A to {B,C} What happens if there is a syntax error in the strategy, e.g., "A-->B" What happens if there is no path? Test the fetch method with different strategies. What happens if you call fetch with a strategy that yields multiple paths? There should be an error message. Test the gather method with different strategies. Test now all features of visitors described in the DJ home page. This includes code on edges not just vertices of the class graph. In this homework you test along the following only loosely coupled dimensions: class graph, strategy graph, visitor. This leads to quick growth of the number of test cases. If you have n class graphs and n strategy graphs and n visitors, you have already n^3 different test cases. For n=10, you get 1000 test cases. Can you test them all? How do you choose a subset of the 1000 cases systematically? As one of your many test cases, solve the following problem: Write a method int count(String s) that is defined for class A and that counts the number of B-objects reachable from the A-object guided by strategy s. Test your method with different strategies and class graphs. What to turn in: test requirements tests indicate how your tests cover your test requirements. (a regression test suite; run it and show the output) bug reports Two test scripts and their outputs Two test scripts for running regression tests: one (test-script-good) for the test cases where you expect a correct answer and one (test-script-bad) where you expect error messages. Use your favorite scripting language to write the scripts, e.g. a UNIX shell script or Java. Directory names for the test cases. Put each test case into a separate directory. Your script test-script-good should go to each directory, run the Java compiler and execute the program and print SUCCESS if the test is successful and print FAILURE otherwise. What is your strategy for deciding whether you have success or failure? Write an evaluation script that reports whether a regression test ran without problems (e.g. no failures). These test scripts will be useful to run after DJ has been maintained. OLD PART 3 from hw 2: ===================== This part is about testing a Java programming tool that facilitates navigation through object structures and programming with visitors. The tool is called DJ http://www.ccs.neu.edu/research/demeter/DJ/ I used DJ on a simple example as described in: /course/com3362/sp99/DJ Test the following features of DJ: Test the traverse method using a statement of the form: cg.traverse(a, new StrategyGraph("A->B"), new MyVisitor()); Use only strategies of the form "A->B". (Find all B-objects contained in a A-object.) but use different visitors and different objects to be traversed. The class graph cg is constructed as follows: ClassGraph cg = new ClassGraph(); // constructed from *.java // will analyze your Java program Test the following features of visitors: before(A host) {}; // code executed before entering A-objects after(A host) {}; // code executed after leaving A-objects void start() {}; // executed before the traversal void finish() {}; // executed after the traversal Have before/after methods for both abstract and final classes. When a class has multiple super classes with before/after methods, in which order are the methods called? In your Java classes use the following features and make sure that the traversal is done right in all cases: abstract classes final classes Can you traverse through a Java array object? Can you traverse through a Java java.util.Vector object? In your Java objects use the following features and make sure that the traversal is done right in all cases: tree objects only (the Java object graph is a tree) cyclic objects (the Java object graph contains cycles; visitor has to stop infinite loop) use instances of classes that have subclasses. ===================== What to turn in: test requirements test specifications indicate how your tests cover your test requirements. (a regression test suite; run it and show the output) bug reports PART 4: Test the class graph constructors of DJ. Is the class graph constructed correctly in all cases?