Larry brought the article below to my attention. I think that with Demeter/C++ we followed a cathedral style. With Demeter/Java we follow more a bazaar style. Have you seen the message of yesterday where a developer noticed an incompatibility with the latest version of the Java Compiler Compiler and he decided to fix Demeter/Java himself and he told us how he did it. I think that events like this are frequent in the bazaar approach. Should we move completely towards a bazaar approach? -- Karl From laf@ccs.neu.edu Mon Jan 5 16:07:53 1998 X-Sender: laf@ccs.neu.edu To: lieber@ccs.neu.edu From: Larry Finkelstein Subject: Cathedral vs. Bazaar Karl, Here's a pointer to the article together with a review of it due to Ken Laws published in the Computists Communique. --Larry 4> Software development: I recently ran across Eric S. Raymond's "The Cathedral and the Bazaar." The latest version is at , and is also available in German (in an earlier version) and as an audio presentation. If you like it, you'll probably also like Raymond's "How to Become a Hacker" FAQ, , and perhaps the jargon file he maintains at . [Loren S Osborn , comp.emulators.ms-windows.wine, 08Nov97.] Raymond is commenting on the Linux software development model, with corroboration from his own team development of fetchmail. The idea is that even large, complex operating systems can be developed through the collective efforts of thousands of programmers on the Internet. This differs from the "cathedral" approach of the Free Software Foundation, in which specific individuals contribute the design and implementation of specific modules and functions. FSF is used as a foil because it's GNU unix is similar to Linux, and because both efforts use open source code and (mostly) volunteer effort. Both follow the Unix gospel of small tools, rapid prototyping, and evolutionary programming. However, FSF encourages deep, careful work "by individual wizards or small bands of mages working in splendid isolation, with no beta to be released before its time," and copyright protection for any contribution of twenty lines or more. Large corporate software development is usually even more "cathedral" in style, and dependent on the talents of just a few programmers. This relates to Fred Brooks' observation that programming output goes up linearly with the number of programmers, but communication and coordination complexity goes up as the square. If Brooks' model held for bazaar-style development, Linux should fail miserably. Instead, it is an astonishing success. Raymond suggests that one key is to release software often -- even daily. Then no one's ego gets tied up with any one release. (Stable versions can be maintained, for those who don't want to be on the bleeding edge.) You also need a large enough development community to have several interested people scanning each new section of code. Gerald Weinberg advocated this "egoless programming" in his classic "The Psychology of Computer Programming." ("One has to smile at the thought of Internet hackers as egoless.") Weinberg noted that software improvement is dramatically faster where developers encourage other people to look for bugs and improvements. Prior to the Internet, only places like Bell Labs, the MIT AI Lab, and UC Berkeley had enough skilled developers to make this feasible. Linux was the first software project (other than the Internet itself) to enlist a global talent pool. Raymond also sees a tie to the Dephi effect: that the averaged opinion of a communicating group is more reliable than the views of any one member (assuming equal expertise for all). Many heads are better than one. Linus Torvalds' style of development -- which sparked and shaped the Linux project -- involves delegation of all of the work to anyone who is interested. All contributions are taken back in and made available to others. Developers can then comment on any new work, and code improvements as they see them. "Plan to throw one away; you will, anyhow." [Fred Brooks, "The Mythical Man-Month."] Don't hesitate to throw away obsolete features. Aircraft designer (and children's author) Antoine de Saint-Exupery noted: "Perfection (in design) is achieved not when there is nothing more to add, but rather when there is nothing more to take away." Aside from some initial structuring and motivation, the project leader's job is to recognize and promote good work once it appears. "The next best thing to having good ideas is recognizing good ideas from your users. Sometimes the latter is better." A bazaar project coordinator or leader must also have good people and communications skills. You need to attract people, interest them in what you're doing, and keep them happy about the amount of work they're doing. Incidentally, people will treat you as no less a genius if you modestly give credit to the people who really did the work. "Every good work of software starts by scratching a developer's personal itch." (Necessity is the mother of invention.) "To solve an interesting problem, start by finding a problem that is interesting to you." "If you have the right attitude, interesting problems will find you." Too often software developers spend their days grinding away for pay at programs they neither need nor love. Love may explain the high average quality of Linux software. "Good programmers know what to write. Great ones know what to rewrite (and reuse)." Great programmers are constructively lazy. "They know that you get an A not for effort but for results, and that it's almost always easier to start from a good partial solution than from nothing at all." Also, if you join someone else's project you inherit their development/user community. Then, "when you lose interest in a program, your last duty to it is to hand it off to a competent successor." Keep your hacker/users constantly stimulated and rewarded, to maximize the number of person-hours thrown at debugging and development. "Given a large enough beta-tester and co-developer base, almost every problem will be characterized quickly and the fix obvious to someone." (Raymond poses this as Linus' Law: "Given enough eyeballs, all bugs are shallow." Jeff Dutky phrased it as "Debugging is parallelizable," as it doesn't require communication among the debuggers except via a central authority -- thus getting around Brooks' law.) On occasion someone will even suggest an entirely new way to think about the problem. The result is rapid simplification and improvement of the code (which may help uncover bugs elsewhere). "Often, the most striking and innovative solutions come from realizing that your concept of the problem was wrong." Here, I think, is the core difference underlying the cathedral-builder and bazaar styles. In the cathedral-builder view of programming, bugs and development problems are tricky, insidious, deep phenomena. It takes months of scrutiny by a dedicated few to develop confidence that you've winkled them all out. Thus the long release intervals, and the inevitable disappointment when long-awaited releases are not perfect. In the bazaar view, on the other hand, you assume that bugs are generally shallow phenomena -- or, at least, that they turn shallow pretty quick when exposed to a thousand eager co-developers pounding on every single new release. Accordingly you release often in order to get more corrections, and as a beneficial side effect you have less to lose if an occasional botch gets out the door. Torvalds says that finding problems is a bigger challenge than fixing them. As Brooks noted, "The total cost of maintaining a widely used program is typically 40 percent or more of the cost of developing it. Surprisingly this cost is strongly affected by the number of users. More users find more bugs." Under the bazaar model, having more beta-testers increases the probability that someone's view or toolkit will make the bugs shallow. The utility function to be maximized is the hackers' own ego satisfaction and reputation among other hackers. Selfishness is thus harnessed to cooperation. You might expect a culture of self-directed egotists to be "fragmented, territorial, wasteful, secretive, and hostile," but in fact these hackers produce better code and documentation than do paid programmers and tech writers. "If you treat your beta-testers as if they're your most valuable resource, they will respond by becoming your most valuable resource." In his fetchmail project, Raymond discovered that by cultivating and chatting with his user/developer community, "I got bug reports of a quality most developers would kill for, often with good fixes attached. I got thoughtful criticism, I got fan mail, I got intelligent feature suggestions." Very few companies could afford the number of talented developers who have contributed to Linux or even fetchmail. "The cutting edge of free software -- or all software? -- will belong to people who start from individual vision and brilliance, then amplify it through the effective construction of voluntary communities of interest." Incidentally, Raymond recommends the 25th Anniversary addition of Frederick P. Brooks' "The Mythical Man-Month" (Addison-Wesley, ISBN 0-201-83595-9), which adds his 1986 "No Silver Bullet" paper and a retrospective essay. Another good "bazaar" essay is Richard P. Gabriel's 1989 "Lisp: Good News, Bad News, and How to Win Big." Of particular interest is the section on "Worse is Better." (As I recall, it's about how cheap, easy-to-implement, nearly-always-OK solutions spread like viruses, whereas complex, well-thought-out, bulletproof solutions will attract few developers. Hence Unix and C gain market share over Lisp.) -- Ken