Adaptive Programming of Large Objects

Phase 1
 DemJava abandoned in favor of plain java adaptive programming.  Plain Jane version with 4 different visitors.  Objects created and destroyed in traversal.  (David Wagstaff)

Phase 2
 Objects created and destroyed in visitor.  (David Wagstaff)

Phase 3
Visitor clones itself inside the traversal.  The clone  traverses a subdirectory (making even more clones for its subdirectories.) (David Wagstaff)

Phase 4
Tried some reflection method for funky cloning.  Interesting, but confusing. (David Wagstaff)

Phase 5
 More funky cloning. (David Wagstaff)

Phase 6
 Visitor takes care of ALL cloning, not the traversal. (David Wagstaff)

Phase 7
Put all the action back in traversal.  Definitely back to better design. (David Wagstaff)

Phase 8
 Added threading.  Actually thread waits until it's child is dead, so the behavior is single threaded. (David Wagstaff)

Phase 9
 Added ThreadPool because large directory trees can't have a thread for every directory.  Composite visitor made of simple visitors.  Used an instance variable as a way to pass info to run() method. threading and cloning controlled by visitor.  Just extend with the right supervisitor and you have threading and cloning.   If there are no more threads (ThreadPool empty), then threads have to traverse into subdirectories (my colleague called it "Boss with a Deadline Algorithm:  if you can't get an employee to do it, do it yourself.") (David Wagstaff)

Phase 10
 Tried to separate ThreadPool from the object.  Abandoned this approach.(David Wagstaff)

Phase 11
Put cloning in run method. (David Wagstaff)

Phase 12
 Separate ThreadPool object. (David Wagstaff)

• DemeterJava assumes that when you extend a class that the class super constructor doesn't take any arguments.  File demands it.  Documented in Phase 0.
• ThreadPool is a misnomer.  It's presently only a ThreadCounter.  I'll keep developing it.
• Big bug with objects and threading.  The bug never appears if you use all cloning or no clones.  Only when some directories are threaded/cloned and others are not.  I had visitors report back to their parents after finishing a directory.  If no clones, then no parents to report back to; only the one visitor collects all info.  If all clones, then each correctly reports back on one directory.  If some clones, some not, then a clone that is forced to traverse into n directories reports back to its parent after EACH directory.  I only wanted it to report ONCE.  Therefore I was getting n + n-1 + n-2 + ... + 1 instead of n.  Fixed in phase 11.
• The design revolved around the power struggle between traversal and visitor.  Either one could be in control.  I vacillated one way, then the other.  The final design feels right.
• The power struggle changed when cloning and threading entered the fray.
• Speed tests in Windows 95 JVM.  NoClone/NoThread is the fastest.  Cloned/Threaded is faster than NoClone/Threaded.  Why would the overhead of creating and destroying clones make it faster?!??  Because only ONE thread at a time can access the countingVisitors sum.  Without cloning then EVERY thread in the whole tree is waiting on one method to increment sum.  With cloning, then every child is waiting on it's parent's method to increment the parent's sum.  Translation:  although counterintuitive at first, cloned threads have less bottlenecks for summing visitors than plain threads.
• The development process was fun.  I didn't know it where it would end.  New problems/opportunities arose with each phase.  It just didn't seem done until phase 12.  I'll call this "serendipity programming."
• An improvement to Threads would fix the "idle parents" problem.  Parent threads "join()" (wait until children die).  The parent thread is idle.  It could be doing something.  I suggest the parent change the parent of it's children to the grandparent and then destroy itself, freeing up a valuable thread.