Main Projects:

We will also set up a course Wiki, where you will describe the status of your projects. The Wiki will also have a space for general issues/comments in supporting Roomy and DMTCP.

DMTCP (list of projects still being revised)

1. Using suspend-to-disk mode to enable checkpointing of graphics programs (including OpenGL (3D graphics))
2. MTT (MPI Testing Tool) for automated testing for Open MPI and DMTCP
3. Checkpointing Qemu, first step towards checkpointing malware, and then running the malware reversibly
4. Another option that may or may not work is the Linux lguest simple hypervisor
5. Fast process migration (e.g. for servers):
   Step 1: Add an MTCP module capability
   Step 2: Write an MTCP checkpoint module that checkpoints to remote RAM
   Step 3: Restart on the remote machine
   Step 4: Tune it for speed
   
6. DMTCP attach using new ptrace capability.
7. Checkpoint the job control/suspend feature of your favorite shell (^Z)
8. Have MTCP use a standard ELF linker script.
9. Hijack/Attach to already running process and checkpoint (There is a question here about how to follow socket connections, if that process is already talking to other processes.)
10. Thread race condition detector: A traditional race condition eventually causes a crash. But since it's a race condition, it doesn't always crash at that location. Experiment with different checkpoints, until you find a checkpoint location for which the process always crashes upon restart. Then modify MTCP to only allow a subset of the threads to resume, and keep the other threads suspended. By trial and error, discover which two threads have a race condition.
11. Portable Linux Apps: DMTCP checkpoint images include any libraries that have been loaded. If the environment variable LD_BIND_NOW is set (set to anything), then the loader will preload every library that it will need. This should enable one to copy a checkpoint image from Debian Linux to OpenSuse Linux to RedHat Linux to Ubuntu Linux to (etc.). Does this work? If not, what's needed to make it work?
12. Incremental Checkpoint: DMTCP may want to keep multiple checkpoint images, so that it can return to any of several execution points in the past. This would normally require a lot of disk space. How does one efficiently store a diff between checkpoints. (This may be a somewhat easier project, for those who are looking for that. With other projects, one will often find that at the end of the semester, one has to report that some parts are still not working, and why. This project offers the opportunity of finishing most of the project, if there are no surprises.)
13. Checkpoint valgrind or other binary rewriters. Examples of programs using dynamic binary translation include: Valgrind, Pin, and Paradyn/Dyninst. Currently, it appears that DMTCP cannot checkpoint these packages. Why? Can it be fixed?

FReD (list of projects still being revised)

1. Reversibly debugging Cilk programs
2. Reversibly debugging OpenMP programs
3. Extend the FReD reversible debugger to Java programs via 'jdb'
4. Haskell and reversible debugging (Due to the functional lazy design of the language, Haskell has to worry about side effects in debugging)
5. Combine FReD with DThreads (highly speculative, but very high impact)
6. Memory leak detector: If a memory leak occurs later in the program, valgrind runs too slowly to easily find it. So, use a malloc debugger or your own memory/free interceptor. A good one is the DUMA library (libduma) (Detect Unintended Memory Access), which is a newer replacement for the classic Electric Fence (libefence). This defines regions created through malloc. Late in the program, it will be easy to find a region of memory that is a memory leak (that no one ever touches again).
   
   Alternatively, using checkpoint/restart tricks, find the last time that anyone touched that memory segment. Report that line of code using a standard tool to convert between a line of assembly language and the source code line. To guarantee that no one ever uses that memory again, remove read-write protection from that region of memory and add a segfault handler to trap any accesses. Then automate the many checkpoint-restart to automatically find where the memory segment was last touched.

Project Software:

DMTCP

DMTCP is available through the sourceforge web page. The easiest way to start is (in Linux) to type:

  svn co https://dmtcp.svn.sourceforge.net/svnroot/dmtcp/trunk dmtcp
  cd dmtcp
  ./configure
  [ OR:   ./configure --enable-debug ]
  make
  make check  [OPTIONAL]

FReD

Debugging and Other Systems Tricks:

1. pgrep -n a.out
pkill -9 a.out, where you replace a.out by the name of your binary. Also, consider pgrep with -o, and with no flags.
2. GDB (beyond the basics): The use of gdb is essential. Note the introduction to UNIX tools.
   1. I invoke gdb --args <COMMAND STRING>
      This form allows you to include the command and its arguments.
   2. Within gdb, try this ASCII graphics mode: ^Xa (control-X a)
   3. Include useful utility functions in your code. You can then do things like (gdb) print my-utility-len-linked-list(mylist) This also works on system calls: (gdb) print lseek(4, 0, 0) This finds the file offset for file descriptor number 4, where the final 0 corresponds to SEEK_SET, whose value is found from grep -R SEEK_SET /usr/include . Similarly, you can find out what fd 4 corresponds to: (gdb) set $pid = getpid()
(gdb) shell ls -l /proc/$pid/fd NOTE: getpid() can be called only after the gdb run (after the target is running).
   4. set follow-fork-mode child will follow the child process on fork. parent will have it follow the parent process. break fork will have GDB stop before executing the fork system call.
   5. If your program under GDB exits too soon, try (gdb) break _exit . (You can slso break on other system calls in libc.)
   6. gdb a.out PID OR
      gdb a.out
(gdb) attach PID
(where the attach command is given within gdb); a convenient single command that finds the PID is: gdb a.out `pgrep a.out | tail -1`
   7. Consider delay loops in combination with "gdb attach". These are better than calls to 'sleep' since they do not access external libraries. For example, insert into your source code: {int x = 1; while(x);} Then do "gdb attach" and (gdb) print x=0 An alternate form to stop at the 5th occurrence is: {static int x = 1; if (x++ > 5) while(x);}
   8. To use gdb with assembly, consider x/10i $pc, and x/10i $pc-12, etc. A convenient version is: (gdb) display/5i $pc followed by stepi (si) or nexti (ni). To set a breakpoint in assembly, try: (gdb) break *0xbfdea000 for a breakpoint at the given address.
   9. To use gdb on multi-threaded programs, try:
      (gdb) info threads
(gdb) thread <NUM>
(gdb) thread apply all where
(gdb) thread set scheduler-locking on NOTE: the last command (scheduler-locking) has the possibility of creating deadlock -- for example, if one thread is holding a low-level libc or C++ lock, and you try to advance a second thread whose execution requires the lock.
3. strace -o myoutput a.out (trace system calls based on kernel API: /usr/include/asm/unistd*.h ; decide in advance if it should trace all child processes or not; the flags -f and -ff exist for tracing parent and all children)
4. ltrace -o myoutput a.out (not as useful as strace, but sometimes interesting: trace library calls instead of system calls).
5. ps auxw | grep a.out
6. pstree -pu $USER or pstree -lu $USER (tree of processes and child process; names in curly braces are additional threads); Note idioms like: pstree -p | grep -C2 a.out
7. top
8. When your program runs too slowly, it might not be CPU-bound. Check man iostat
man vmstat for disk/file I/O (Blk_read/s / Blk_wrtn/s), and paging to disk (bi/bo/id), respectively. A local disk (not on the network; SANs are different) can sequentially read or write (not both at once) roughly at a rate from 50 MB/s to 100 MB/s. If you are accessing files mostly and you don't see that bandwidth, then your program is not efficient. If you are paging to disk and you do see a bandwidth anywhere near that bandwidth, then you are using too much RAM.
9. watch -d ls -l /tmp/myfile.txt
watch -d "pstree -l | grep -A1 `basename $SHELL`" (repeatedly execute COMMAND for watch -d COMMAND)
10. Search for SUBSTRING in all dmtcp/src/* files: find dmtcp| xargs grep SUBSTRING
    or alternatively: grep -r SUBSTRING
    Don't forget: grep -C3 ...; grep -A5 ...; (and so on.)
11. grep and google are your friends when searching for information. Besides "grep'ing" through source code, here is a grep trick you may not have seen:
    find /usr/share/man/man3 | xargs zgrep MYSTRING
find /usr/share/man/man3 | xargs gzip -dc | grep -C3 MYSTRING
12. less /proc/PID/maps<
13. ls -l /proc/PID/fd
14. List open file descriptors: lsof | grep a.out If you discover an interesting socket with SOCKET_ID through 'lsof' or 'ls -l /proc/PID/fd', then find the other end of the socket: lsof | grep SOCKET_ID
15. List environment variables of a process: cat /proc/PID/environ | tr '\000' '\n'
16. nm a.out (or nm library.so) Note the form nm -o for printing out filenames. This can be useful with brute force strategies:
    nm -o /usr/lib/lib* | grep MY_SYMBOL
    (Also see discussion of readelf and objdump below.)
17. strings -a a.out (for some binary, a.out)
18. Sometimes, a macro was expanded, and it's hard to track down what happened. Try: cpp -dM -I. -Iother/path/to/include/files myfile.c
19. Sometimes, it's not clear what includes paths to use. In the above example, try: rm myfile.o; make myfile.o and copy the command line used by 'make' to build myfile.o. If 'make' uses libtools, you may also have to remove hidden directories with names like .libs .
20. Get to know your loader. It executes before your executable file: man ld.so
21. env LD_DEBUG=help a.out
env LD_DEBUG=files a.out (and try other options to LD_DEBUG)
22. ldd a.out (for some binary, a.out)
23. Replace PID in following:
    pushd /proc/PID;
ls -l exe;
echo -n "cmdline: "; cat -v cmdline;
echo ""; cat -v environ; echo "";
popd
24. DMTCP: ./configure --enable-debug; make clean; make and then run and look at /tmp/dmtcp-USER@HOST/jassert* files for your value of USER and HOST. Before your next test, rm -rf /tmp/dmtcp-USER@HOST .
25. MTCP: Look at mtcp/Makefile and uncomment the line taht adds to CFLAGS the flag: -DDEBUG
26. More on gdb: Using gdb with C++ : For a C++ function with namespace, class, and signature (e.g.: dmtcp::myClass::foo(int, bool) ), try listing it first:
    list 'dmtcp::myC<TAB>
    It will autocomplete. Extend it, and type the final quote mark ('). Once you're sure you can list it, you can do things like set a breakpoint:
    b 'dmtcp::myC<TAB> (and complete it with quote mark as before).
27. Using gdb with errno: In glibc, the global variable errno (see man errno) is a macro that is redefined to:
    *(int *)__errno_location()
    If you want to p errno within gdb, you will have to modify this into p *(int *)__errno_location() On 64-bit Linux, glibc seems to do something even more complicated, requiring a more complicated solution.
28. If you look at gdb and some call frames on the stack have no information (only a hex address and "?"), then find out where the call frames come from. Look at the hexadecimal address. Then do:
    (gdb) shell cat /proc/PID/maps (where the PID of the current process is given by (gdb) info proc ). Alternatively:
    (gdb) info proc mappings
    Find which library or other memory segment the unknown hexadecimal address came from. Knowing which library was called is useful, but you may be able to find out more. If it comes from libc.so (or some other well-known library), then see the next two tips for how to get the library to show you its internal debugging information.
29. (Continued) If you need a libc.so (or other well-known library) with debugging symbols, then:
    1. Install the package libc6-dbg. (The package name might differ for you. Also, this assumes you have root privilege on your Linux.) This will install a special libc.so in the directory /usr/lib/debug . Please note that the CCIS Ubuntu Linux machines already have a debugging version of libc installed, currently as /usr/lib/debug/libc-2.7.so .
    2. Next, do:
       env LD_LIBRARY_PATH=/usr/lib/debug dmtcp_checkpoint a.out (Presumably, after you checkpoint, the restarted a.out process will be using the pre-checkpoint libraries and hence the debugging versions. So, probably you don't need to use env LD_LIBRARY_PATH=/usr/lib/debug for the restart command. But if you're unsure, it doesn't hurt.)
    3. The a.out process above should now be using a debugging version of libc.so and perhaps other libraries. You can verify this by looking at /proc/PID/maps for your process. Now, in gdb, you will see the symbol information in the call frame and a source code file and line number.
    4. To read the corresponding source code, you can either download it from the main source code location: http://www.gnu.org/software/libc/libc.html#Availability (try to choose the same libc version, and note that the line numbers may be different in your Linux distro), or download the source package for your particular Linux distro.
30. (Continued) [You can find a more conceptual version of this discussion here.] If gdb still shows some call frames with "?", and you have the full pathname of the library on disk, then you can often fix it as follows. (Once you understand this procedure, you may want to try the bin/gdb-add-symbol-file shell script found in DMTCP.)
    1. In /proc/PID/maps look up the full pathname of the library you need to load. The address of the call frame with missing information should be in the address range of that library.
    2. In gdb, read help add-symbol-file
    3. In gdb, type add-symbol-file FILE ADDR where FILE is the full pathname you identified in the /proc/PID/maps file. The ADDR will be the hexadecimal sum of:
       1. beginning of text segment address (text segment normally has r-x permission) in /proc/PID/maps; and
       2. hexadecimal address for Addr heading corresponding to .text when you look it up under Headers: with either of the following command: readelf -S FILE
objdump -h FILE
    4. In the last step, the maps file provided the beginning address of the whole segment, but the binary library on disk contains many sections for a segment, and the .text section need not be the first section in the file. So, we must add the offset of the .text section, found by analyzing the binary library on disk.
    5. In gdb, a convenient way to add hexadecimal numbers is:
       p/x addr1 + addr2 where addr1 and addr2 are the two addresses we discussed. If those addresses are in hexadecimal, make sure to include 0x at the beginning of each hexadecimal number.
    6. Now do 'where' in gdb, and you should see full call frame information.
31. If gdb is inconvenient, you can set a breakpoint directly in your program using a technique of Nikolay Igotti. In short, one defines a handler for SIGTRAP, forks a child, and uses ptrace on the child to set some of the child's x86 hardware debug registers using the POKEUSER option of ptrace and include/sys/user.h.
32. The two commands readelf and objdump are useful for inspecting the contents of binary files. These are related to the other commands, nm and strings, but these commands have many more options, including the ability to disassemble into assembly code, the ability to display section headers, etc. Scan the man pages quickly to see if something might be useful for you.
33. For an assembly level listing as you do stepi in gdb, try objdump -S a.out > a.out.listing where a.out should be replaced by your binary. For a more verbose form, try one of: gcc -c -g -Wa,-alh,-L file.c > file.s
gcc -c -g -Wa,-ahls=file.s file.c Variations of this can also produce assembly code that can be directly assembled by gcc or by as. For example, if you want to modify and re-compile the source code for libc.so, this is normally quite painful. A nice trick is to disassemble libc.so into assembly, and then cut or copy out the particular assembly routines that you want to assemble into a modified library.
34. UNIX system calls, by Open Group; (enter system call in search box); This is the clearest, most precise man pages for system calls you will ever find.
35. Valgrind (Memory and leak detection utility); This is easy-to-use and surprisingly powerful.
36. If you want to see the stack just before a segfault, a quick idea that may help is: catchsegv COMMAND_LINE
37. Another method for catching segfaults that may give you more control is to try the glibc call backtrace: man backtrace It mangles any C++ names, but they are mostly readable (and utilities exist for demangling the names). Read the notes of man backtrace (for example, compile with gcc -rdynamic to get symbol names. Also, note man addr2line.
    Look at the example file, backtrace.c for this course.
    Also, for any call frames with no symbol name, look up the hex address in /proc/<PID>/maps. Use addr2line to translate hex addresses into line numbers in source code. (If it's a .so dynamic library, give it the offset, the hex address minus the beginning library address as shown by /proc/<PID>/maps.
38. Understand addresses of symbols:
    
    1. Your executable will be loaded into RAM at an unknown base address. But once it is loaded, you can find the base address it was loaded to: less /proc/PID/maps
    2. Your executable file provides the offset from the base address at which you will find the beginning of: 'text', 'data' (and possibly 'bss' segment). Commands like 'objdump' and 'readelf' will show you this offset in the file. readelf -S a.out | grep '\.text '
    3. You may want to know the address of a particular symbol within the 'text', 'data', or 'bss' segment. Use 'nm', 'objdump', or 'readelf'. SHELL% nm a.out | grep main
080483e4 T main (As described in the 'man' pages, 't', 'T', 'd', 'D', 'b', 'B', 'U' tell you if the symbol is in text, data, bss, or undefined (presumably defined in a different library). Lower case means file-private, and upper case means a globally visible symbol. Look up __attribute__ ((visibility ("hidden"))) for declaring a symbol library-private: globally visible within a .o file, but file-private within the .so (library) file.)
    4. In gdb, if it fails to correctly show you the stack, maybe some memory was mmapped in (e.g. by DMTCP) that confused it. The command (gdb) help add-symbol-file along with the above information will allow you to tell gdb at what address in RAM the executable or library file on disk was loaded. The file on disk contains the symbol information.
    5. Some of these calculations can be automated for you by a DMTCP utility: (gdb) shell utils/gdb-add-symbol-file
39. An interesting Linux command: addr2line
    ('main' is on line 2 of tmp.c in the example below.) SHELL% nm a.out
...
080483e4 T main
SHELL% addr2line -C -e /tmp/a.out 080483e4
/tmp/tmp.c:2
40. In comparing two versions of a file, consider programs such as: kompare, kdiff3, meld, gvimdiff (or text-based vimdiff).
41. To examine the Linux source code, try "google LXR" (Linux Cross Reference), which should lead you to lxr.linux.no . LXR is free software for hyper-linking large code bases. Another popular choice is Doxygen (available as a package in many Linux distros).
42. To see which virtual memory pages are currently mapped to physical memory, see /proc/PID/pagemap .
43. To find out information known to BIOS: sudo dmiprobe -t help
44. When using google to get technical information, stackoverflow.com tends to have high quality answers. Try those hits first.