Software Testing, Verification and Validation Karl J. Lieberherr COM 3220 Spring 1998 =========================================================================== Due date: April 14, 1998 This assignment is stored in file $TVV/hw/2 in file assign.txt ========= SET-UP INFO: You may put setenv CLASSPATH into your initialization file since both demjava and apstudio set the CLASSPATH correctly. Alternatively, you can take out CLASSPATH from your initialization file. ========== ASSIGNMENT 2 TESTING OF A PARSER GENERATOR: The Java Compiler Compiler from SUNSOFT http://www.suntest.com/JavaCC/ ============================================================== On CCS Northeastern machines, TVV=/home/lieber/.www/com3220/ On the WWW, TVV = http://www.ccs.neu.edu/home/lieber/com3220 Use the following header for your electronic homework submissions and put it at the top of the message: Course number: COM 3220 Name: Account name: Assignment number: Date: Whenever you have questions about course material, please send e-mail to: mail lieberherr@ccs.neu.edu com3220-grader@ccs.neu.edu ccs.courses.com3220@ccs.n eu.edu All messages which I send to the class are archived in /proj/demsys/logs-lieber/com3220 ========= READING: Read chapters 5 and 6 of your text book. Read chapters 16 (Syntax Testing) and 17 (Example about Syntax Testing) of the text book. Learn EBNF style grammar notation: http://www.ccs.neu.edu/research/demeter/DemeterJava/UserManual/node27.html Known problems: http://www.ccs.neu.edu/research/demeter/DemeterJava/use/latest-demjava/ BUGS CHANGES Those files tell you about problems which are already known and about recent changes which are not yet in the user manual. =================================================== The purpose of this homework is to have a concrete, interesting example where you can apply the testing process described in the text book. Use a black box testing approach (you don't have to look at the source code). Write test requirements test specifications tests (run them) bug reports See page 11 of the text book for an overview. Parsing is a very common task and parsers need to be tested too. We test JavaCC by translating from an EBNF style notation to the JavaCC input notation. Consider directory: $TVV/hw/2/busroute In program.cd you have a grammar for describing BusRoute-objects. In gen/grammar.jj you have the JavaCC input for the grammar. In program.input is the input sentence which the parser will parse. In order to run the parser, you copy the directory and call: demjava test PART 1: ======= Test the parser in $TVV/hw/2/busroute. To feed it different inputs, modify demeter-input and call demjava test PART 2: ======= Test The Java Compiler Compiler along with Demeter. To create a test case, perform the following steps: Create a new directory Create a grammar in file program.cd Make sure that Main = . is at the end of the file. Create an input sentence in file program.input Copy file $TVV/hw/2/busroute/program.beh and replace BusRoute in three places by the start class of your grammar. Create a new project file using demjava new Call Demeter/Java and the Java Compiler Compiler and the Java compiler and parse the input: demjava test In part 2 you should also test the predefined classes: Line, Word http://www.ccs.neu.edu/research/demeter/DemeterJava/use/latest-demjava/CHANGES and also Ident and Text. Testing a parser generator is a big task. You should only spend so much time on the testing as you find it useful for your learning process. Many test cases for Demeter/Java have already been written: See: http://www.ccs.neu.edu/research/demeter/DemeterJava/testing/ When you report bugs please provide a directory in the regression suite format for addition to the suite. ========================= What to turn in: test requirements test specifications indicate how your tests cover your test requirements. (a regression test suite; run it and show the output) bug reports (see earlier email message) A description on how to improve and automate the testing. Describe a test driver and a suite driver for your regression test suite. Make a critique of the regression test technique described in: http://www.ccs.neu.edu/research/demeter/DemeterJava/testing/ In your bug reports, identify whether you think it is a bug in the Java Compiler Compiler or in the preprocessing done by Demeter/Java or the Java Just-In-Time (JIT) compiler. There are actually many things which could go wrong in Demeter/Java such as: class generation, visitor generation, traversal generation, besides grammar generation. This is an example of a situation where you are not given a complete specification. We have not said what should happen if the input contains some syntactic fault. It is your task to check whether the program handles those situations reasonably well. If you are eager to do some white box testing, the source code for Demeter/Java is in: http://www.ccs.neu.edu/research/demeter/DemeterJava/use/latest-demjava/ PART 3: ======= This part revisits assignment 1 but now in the context of a graphical user interface. While in assignment one you used a textual representation of the graphs, you use now a graphical representation. The graphical user interface for drawing graphs is called apstudio and resides at /proj/demsys/apstudio/bin/apstudio You must run this program on Solaris or you can download it into your computer (including source code). Instructions for downloading at: http://www.ccs.neu.edu/research/demeter/DemeterJava/ Your task in this part of the homework is to test AP Studio for the purpose of drawing graphs where the nodes are squares and the edges are labeled. AP Studio has many other capabailities to draw UML class diagrams (www.rational.com/uml/) but here we just focus on simple graphs. The documentation for AP Studio is at: http://www.ccs.neu.edu/research/demeter/APStudio/UserManual/ For Postscript, see: http://www.ccs.neu.edu/research/demeter/DemeterJava/ If you are curious, the source code for AP Studio is in: http://www.ccs.neu.edu/research/demeter/DemeterJava/use/latest-apstudio/ What to turn in: Draw three different graphs which you used in hw 1 and then save them. This will produce files with extensions *.cd.apcd which contain the textual representations of your graphs. For Part 3 turn in the three graphs in their textual and graphical form (use the Print command in the File menu) and a discussion of how you could develop a regression test suite for AP Studio, and Java GUIs in general. See http://www.sun.com/suntest/ for some ideas. If you encounter problems using AP Studio, please turn in a list of bugs or things which should be improved.