Software Testing, Verification and Validation Karl J. Lieberherr COM 3220 Spring 1999 =========================================================================== Due date: April 15, 1999 This assignment is stored in file $TVV/hw/2 in file assign.txt ========= ASSIGNMENT 2 TESTING OF A PARSER GENERATOR: The Java Compiler Compiler from SUNSOFT http://www.suntest.com/JavaCC/ TESTING of a Java Productivity tool:DJ ============================================================== On CCS Northeastern machines, TVV=/home/lieber/.www/com3220/ On the WWW, TVV = http://www.ccs.neu.edu/home/lieber/com3220/sp99 Use the following header for your electronic homework submissions and put it at the top of the message: Course number: COM 3220 Name: Account name: Assignment number: Date: Whenever you have questions about course material, please send e-mail to: mail lieberherr@ccs.neu.edu com3220-grader@ccs.neu.edu ccs.courses.com3220@ccs.n eu.edu All messages which I send to the class are archived in /proj/demsys/logs-lieber/com3220 ========= READING: Read chapters 5 and 6 of your text book. Read chapters 16 (Syntax Testing) and 17 (Example about Syntax Testing) of the text book. Learn EBNF style grammar notation: http://www.ccs.neu.edu/research/demeter/DemeterJava/UserManual/ Known problems: http://www.ccs.neu.edu/research/demeter/DemeterJava/use/latest-demjava/ BUGS CHANGES Those files tell you about problems which are already known and about recent changes which are not yet in the user manual. =================================================== The purpose of this homework is to have a concrete, interesting example where you can apply the testing process described in the text book. Use a black box testing approach (you don't have to look at the source code). Write test requirements test specifications tests (run them) bug reports See page 11 of the text book for an overview. Parsing is a very common task and parsers need to be tested too. We test JavaCC by translating from an EBNF style notation to the JavaCC input notation. Consider directory: $TVV/hw/2/busroute In program.cd you have a grammar for describing BusRoute-objects. In gen/grammar.jj you have the JavaCC input for the grammar. In program.input is the input sentence which the parser will parse. In order to run the parser, you copy the directory and call: demjava test PART 1: ======= IMPORTANT: DO THIS PART ONLY IF YOU HAD COM 3351 (Principles of Programming Languages) or equivalent. Otherwise skip this part and test Part 3 more thoroughly. Test the parser in $TVV/hw/2/busroute. To feed it different inputs, modify demeter-input and call demjava test PART 2: ======= IMPORTANT: DO THIS PART ONLY IF YOU HAD COM 3351 (Principles of Programming Languages) or equivalent. Otherwise skip this part and test Part 3 more thoroughly. Test The Java Compiler Compiler along with Demeter. To create a test case, perform the following steps: Create a new directory Create a grammar in file program.cd Make sure that Main = . is at the end of the file. Create an input sentence in file program.input Copy file $TVV/hw/2/busroute/program.beh and replace BusRoute in three places by the start class of your grammar. Create a new project file using demjava new Call Demeter/Java and the Java Compiler Compiler and the Java compiler and parse the input using demjava test In part 2 you should also test the predefined classes: Line, Word http://www.ccs.neu.edu/research/demeter/DemeterJava/use/latest-demjava/CHANGES and also Ident and Text. Testing a parser generator is a big task. You should only spend so much time on the testing as you find it useful for your learning process. Many test cases for Demeter/Java have already been written: See: http://www.ccs.neu.edu/research/demeter/DemeterJava/testing/ When you report bugs please provide a directory in the regression suite format for addition to the suite. ========================= What to turn in: test requirements test specifications indicate how your tests cover your test requirements. (a regression test suite; run it and show the output) bug reports A description on how to improve and automate the testing. Describe a test driver and a suite driver for your regression test suite. Make a critique of the regression test technique described in: http://www.ccs.neu.edu/research/demeter/DemeterJava/testing/ In your bug reports, identify whether you think it is a bug in the Java Compiler Compiler or in the preprocessing done by Demeter/Java or the Java Just-In-Time (JIT) compiler. There are actually many things which could go wrong in Demeter/Java such as: class generation, visitor generation, traversal generation, besides grammar generation. This is an example of a situation where you are not given a complete specification. We have not said what should happen if the input contains some syntactic fault. It is your task to check whether the program handles those situations reasonably well. If you are eager to do some white box testing, the source code for Demeter/Java is in: http://www.ccs.neu.edu/research/demeter/DemeterJava/use/latest-demjava/ PART 3: ======= NOTE: This part is different: it does not use Demeter/Java. This part is about testing a Java programming tool that facilitates navigation through object structures and programming with visitors. The tool is called DJ http://www.ccs.neu.edu/research/demeter/DJ/ I used DJ on a simple example as described in: /course/com3362/sp99/DJ Test the following features of DJ: Test the traverse method using a statement of the form: cg.traverse(a, new StrategyGraph("A->B"), new MyVisitor()); Use only strategies of the form "A->B". (Find all B-objects contained in a A-object.) but use different visitors and different objects to be traversed. The class graph cg is constructed as follows: ClassGraph cg = new ClassGraph(); // constructed from *.java // will analyze your Java program Test the following features of visitors: before(A host) {}; // code executed before entering A-objects after(A host) {}; // code executed after leaving A-objects void start() {}; // executed before the traversal void finish() {}; // executed after the traversal Have before/after methods for both abstract and final classes. When a class has multiple super classes with before/after methods, in which order are the methods called? In your Java classes use the following features and make sure that the traversal is done right in all cases: abstract classes final classes Can you traverse through a Java array object? Can you traverse through a Java java.util.Vector object? In your Java objects use the following features and make sure that the traversal is done right in all cases: tree objects only (the Java object graph is a tree) cyclic objects (the Java object graph contains cycles; visitor has to stop infinite loop) use instances of classes that have subclasses. ===================== What to turn in: test requirements test specifications indicate how your tests cover your test requirements. (a regression test suite; run it and show the output) bug reports =============== Info about JavaCC: I have installed JavaCC 1.0 and Metamata Parse 1.06 in /proj/demsys/demjava/packages. Since JavaCC 1.0 claims to be fully backward-compatible, and all of the Demeter/Java regression tests passed with it, I've put a link to it in /proj/demsys/demjava/bin, so it will be the default if that's near the front of your path. Let me know if you run into any problems and I'll rename it so it's not the default; otherwise I'll probably get systems to install it as the default for everyone after a few weeks. Metamata Parse is a new parser generator from the folks who wrote JavaCC, and it has a different (though similar) grammar language. Demeter/Java doesn't support it yet but the next release should. You can use "mparse" to run it, or "metamata" to run the GUI (which also lets you run the other parts of the Metamata Suite, such as Debug, but our evaluation license runs out soon-- Parse, however, is not licensed and will always be free). It translates its input to the JavaCC syntax and then runs JavaCC, so you'll still need that in your path; future versions of Metamata Parse will include the JavaCC functionality directly, but I guess they're still wrangling with legal issues with Sun. See http://www.metamata.com for more info. --Doug From dougo@ccs.neu.edu Wed Apr 7 17:08:28 1999