A Generation Lost in The Bazaar - ACM Queue

RELATED CONTENT
BARBARIANS AT THE GATEWAYS

High-frequency Trading and Exchange
Technology
Jacob Loveless
ONLINE ALGORITHMS IN HIGH-FREQUENCY
TRADING
The challenges faced by competing HFT
algorithms
Jacob Loveless, Sasha Stoikov, Rolf
Waeber
THE ESSENCE OF SOFTWARE ENGINEERING:
THE SEMAT KERNEL
A thinking framework in the form of an
actionable kernel
Ivar Jacobson, Pan-Wei Ng, Paul McMahon,
Ian Spence, Svante Lidman
MANAGING TECHNICAL DEBT
Shortcuts that save money and time
today can cost you down the road.
Eric Allman
BROWSE THIS TOPIC:
DEVELOPMENT
QUEUE ON SLASHDOT
- More Encryption Is Not the Solution
- The Antifragile Organization
- Realtime GPU Audio
QUEUE ON REDDIT
- Nonblocking Algorithms and Scalable
Multicore Programming
- FPGA Programming for the Masses
- There's just no getting around it: You
are Building a Distributed System
A Generation Lost in the Bazaar
Quality happens only when someone is responsible for it.
POUL-HENNING KAMP
Thirteen years ago, Eric Raymond's book The Cathedral and the Bazaar (O'Reilly Media, 2001) redefined our
vocabulary and all but promised an end to the waterfall model and big software companies, thanks to the
new grass-roots open source software development movement. I found the book thought provoking, but
it did not convince me. On the other hand, being deeply involved in open source, I couldn't help but think
that it would be nice if he was right.
The book I brought to the beach house this summer is also thought provoking, much more so than
Raymond's (which it even mentions rather positively): Frederick P. Brooks's The Design of Design (Addison-
Wesley Professional, 2010). As much as I find myself nodding in agreement and as much as I enjoy
Brooks's command of language and subject matter, the book also makes me sad and disappointed.
Thirteen years ago also marks the apogee of the dot-com euphoria, where every teenager was a Web
programmer and every college dropout had a Web startup. I had genuine fun trying to teach some of
those greenhorns about the good old-fashioned tricks of the tradetest-restoring backups, scripting
operating-system installs, version control, etc. Hindsight, of course, is 20/20 (i.e., events may have been
less fun than you remember), and there is no escaping that the entire dot-com era was a disaster for
IT/CS in general and for software quality and Unix in particular.
I have not seen any competent analysis of how much bigger the IT industry became during the dot-com
years. My own estimate is thatcounted in the kinds of jobs that would until then have been behind the
locked steel doors of the IT departmentour trade grew by two orders of magnitude, or if you prefer, by
more than 10,000 percent.
Getting hooked on computers is easyalmost anybody can make a program work, just as almost anybody
can nail two pieces of wood together in a few tries. The trouble is that the market for two pieces of wood
nailed togetherinexpertlyis fairly small outside of the "proud grandfather" segment, and getting from
there to a decent set of chairs or fitted cupboards takes talent, practice, and education. The extra 9,900
percent had neither practice nor education when they arrived in our trade, and before they ever had the
chance to acquire it, the party was over and most of them were out of a job. I will charitably assume that
those who managed to hang on were the most talented and most skilled, but even then there is no
escaping that as IT professionals they mostly sucked because of their lack of ballast.
The bazaar meme advocated by Raymond, "Just hack it," as opposed to the carefully designed cathedrals
of the pre-dot-com years, unfortunately did, not die with the dot-com madness, and today Unix is rapidly
sinking under its weight.
I updated my laptop. I have been running the development version of FreeBSD for 18 years straight now,
and compiling even my Spartan work environment from source code takes a full day, because it involves
trying to make sense and architecture out of Raymond's anarchistic software bazaar.
At the top level, the FreeBSD ports collection is an attempt to create a map of the bazaar that makes it
easy for FreeBSD users to find what they need. In practice this map consists, right now, of 22,198 files
that give a summary description of each stall in the bazaara couple of lines telling you roughly what that
stall offers and where you can read more about it. Also included are 23,214 Makefiles that tell you what to
do with the software you find in each stall. These Makefiles also try to inform you of the choices you should
consider, which options to choose, and what would be sensible defaults for them. The map also
conveniently comes with 24,400 patch files to smooth over the lack of craftsmanship of many of the wares
offered, but, generally, it is lack of portability that creates a need for these patch files.
Finally, the map helpfully tells you that if you want to have www/firefox, you will first need to get devel/nspr, security/nss, databases/sqlite3, and
so on. Once you look up those in the map and find their dependencies, and recursively look up their dependencies, you will have a shopping list
of the 122 packages you will need before you can get to www/firefox.
Modularity and code reuse are, of course, A Good Thing. Even in the most trivially simple case, however, the CS/IT dogma of code reuse is totally
foreign in the bazaar: the software in the FreeBSD ports collection contains at least 1,342 copied and pasted cryptographic algorithms.
If that resistance/ignorance of code reuse had resulted in self-contained and independent packages of software, the price of the code duplication
might actually have been a good tradeoff for ease of package management. But that was not the case: the packages form a tangled web of
haphazard dependencies that results in much code duplication and waste.
Here is one example of an ironic piece of waste: Sam Leffler's graphics/libtiff is one of the 122 packages on the road to www/firefox, yet the
resulting Firefox browser does not render TIFF images. For reasons I have not tried to uncover, 10 of the 122 packages need Perl and seven
need Python; one of them, devel/glib20, needs both languages for reasons I cannot even imagine.
Further down the shopping list are repeated applications of the Peter Principle, the idea that in an organization where promotion is based on
achievement, success, and merit, that organization's members will eventually be promoted beyond their level of ability. The principle is commonly
phrased, "Employees tend to rise to their level of incompetence." Applying the principle to software, you will find that you need three different
versions of the make program, a macroprocessor, an assembler, and many other interesting packages. At the bottom of the food chain, so to
speak, is libtool, which tries to hide the fact that there is no standardized way to build a shared library in Unix. Instead of standardizing how to
do that across all Unixensomething that would take just a single flag to the ld(1) commandthe Peter Principle was applied and made it
libtool's job instead. The Peter Principle is indeed strong in this casethe source code for devel/libtool weighs in at 414,740 lines. Half that line
count is test cases, which in principle is commendable, but in practice it is just the Peter Principle at work: the tests elaborately explore the
functionality of the complex solution for a problem that should not exist in the first place. Even more maddening is that 31,085 of those lines are
in a single unreadably ugly shell script called configure. The idea is that the configure script performs approximately 200 automated tests, so that
the user is not burdened with configuring libtool manually. This is a horribly bad idea, already much criticized back in the 1980s when it appeared,
as it allows source code to pretend to be portable behind the veneer of the configure script, rather than actually having the quality of portability
ACM QueueArchitecting Tomorrow's Computing
A Generation Lost in the Bazaar
Tweet 431
COLUMNS > THE BIKESHED
view issue
by Poul-Henning Kamp | August 15, 2012
Topic: Development
Like 113 people like this. Sign Up to see what your f riends like.
A Special Offer to Join ACM Why Join ACM?
to begin with. It is a travesty that the configure idea survived.
The 1980s saw very different Unix implementations: Cray-1s with their 24-bit pointers, Amdahl UTS mainframe Unix, a multitude of more or less
competently executed SysV+BSD mashups from the minicomputer makers, the almostbut not quiteUnix shims from vendors such as Data
General, and even the genuine Unix clone Coherent from the paint company Mark Williams.
The configure scripts back then were written by hand and did things like figure out if this was most like a BSD- or a SysV-style Unix, and then
copied one or the other Makefile and maybe also a .h file into place. Later the configure scripts became more ambitious, and as an almost
predictable application of the Peter Principle, rather than standardize Unix to eliminate the need for them, somebody wrote a program, autoconf,
to write the configure scripts.
Today's Unix/Posix-like operating systems, even including IBM's z/OS mainframe version, as seen with 1980 eyes are identical; yet the 31,085
lines of configure for libtool still check if <sys/stat.h> and <stdlib.h> exist, even though the Unixen, which lacked them, had neither sufficient
memory to execute libtool nor disks big enough for its 16-MB source code.
How did that happen?
Well, autoconf, for reasons that have never made sense, was written in the obscure M4 macro language, which means that the actual tests look
like this:
## Whether `make' supports order-only prerequisites.
AC_CACHE_CHECK([whether ${MAKE-make} supports order-only prerequisites],
[lt_cv_make_order_only],
[mkdir conftest.dir
cd conftest.dir
touch b
touch a
cat >confmk << 'END'
a: b | c
a b c:
touch $[]@
END
touch c
if ${MAKE-make} -s -q -f confmk >/dev/null 2>&1; then
lt_cv_make_order_only=yes
else
lt_cv_make_order_only=no
fi
cd ..
rm -rf conftest.dir
])
if test $lt_cv_make_order_only = yes; then
ORDER='|'
else
ORDER=''
fi
AC_SUBST([ORDER])
Needless to say, this is more than most programmers would ever want to put up with, even if they had the skill, so the input files for autoconf
happen by copy and paste, often hiding behind increasingly bloated standard macros covering "standard tests" such as those mentioned earlier,
which look for compatibility problems not seen in the past 20 years.
This is probably also why libtool's configure probes no fewer than 26 different names for the Fortran compiler my system does not have, and then
spends another 26 tests to find out if each of these nonexistent Fortran compilers supports the -g option.
That is the sorry reality of the bazaar Raymond praised in his book: a pile of old festering hacks, endlessly copied and pasted by a clueless
generation of IT "professionals" who wouldn't recognize sound IT architecture if you hit them over the head with it. It is hard to believe today,
but under this embarrassing mess lies the ruins of the beautiful cathedral of Unix, deservedly famous for its simplicity of design, its economy of
features, and its elegance of execution. (Sic transit gloria mundi, etc.)
One of Brooks's many excellent points is that quality happens only if somebody has the responsibility for it, and that "somebody" can be no more
than one single personwith an exception for a dynamic duo. I am surprised that Brooks does not cite Unix as an example of this claim, since we
can pinpoint with almost surgical precision the moment that Unix started to fragment: in the early 1990s when AT&T spun off Unix to
commercialize it, thereby robbing it of its architects.
More than once in recent years, others have reached the same conclusion as Brooks. Some have tried to impose a kind of sanity, or even to lay
down the law formally in the form of technical standards, hoping to bring order and structure to the bazaar. So far they have all failed
spectacularly, because the generation of lost dot-com wunderkinder in the bazaar has never seen a cathedral and therefore cannot even imagine
why you would want one in the first place, much less what it should look like. It is a sad irony, indeed, that those who most need to read it may
find The Design of Design entirely incomprehensible. But to anyone who has ever wondered whether using m4 macros to configure autoconf to
write a shell script to look for 26 Fortran compilers in order to build a Web browser was a bit of a detour, Brooks offers well-reasoned hope that
there can be a better way.
LOVE IT, HATE IT? LET US KNOW
feedback@queue.acm.org
Poul-Henning Kamp (phk@FreeBSD.org) has programmed computers for 26 years and is the inspiration behind bikeshed.org. His software has
been widely adopted as under-the-hood building blocks in both open source and commercial products. His most recent project is the Varnish HTTP
accelerator, which is used to speed up large Web sites such as Facebook.
2012 ACM 1542-7730/12/0800 $10.00
Originally published in Queue vol. 10, no. 8
see this item in the ACM Digital Library
POUL-HENNING KAMP (phk@FreeBSD.org) is one of the primary developers of the FreeBSD operating system, which he has worked on from the
very beginning. He is widely unknown for his MD5-based password scrambler, which protects the passwords on Cisco routers, Juniper routers,
and Linux and BSD systems. Some people have noticed that he wrote a memory allocator, a device file system, and a disk encryption method that
is actually usable. Kamp lives in Denmark with his wife, his son, his daughter, about a dozen FreeBSD computers, and one of the world's most
precise NTP (Network Time Protocol) clocks. He makes a living as an independent contractor doing all sorts of stuff with computers and networks.
For additional information see the ACM Digital Library Author Page for: Poul-Henning Kamp

COMMENTS
William Payne | Mon, 20 Aug 2012 14:31:12 UTC
So should we then flee to the relative oasis of sanity that is Plan 9?
phd_student_doom | Mon, 20 Aug 2012 15:10:25 UTC
Yawn, I can't wait till these type of people retire. I've only been alive 2/3 of the time he's been programming but the 'chaos'
is what makes programming fun.
Matt Welsh | Mon, 20 Aug 2012 15:13:22 UTC
Nice article. I think we all predicted this would happen when the open source "movement" started to gain momentum a decade ago.
To be fair, a lot of good engineering goes into open source projects, although it's often hard to see because of the smoke
screen of amateur contributions and longwinded threads on the email lists. It's not all hackers working in their spare time:
Google, for example, contributes a lot of code back to open source projects (e.g., Guido van Rossum works at Google). There
also seems to be a great deal of entropy at work: Witness all of the failed efforts to "clean up" BSD and Linux that never went
anywhere. Chaos is king.
Poul-Henning Kamp | Mon, 20 Aug 2012 15:25:27 UTC
Matt, My point is not that there are no cathedrals today, but that people don't recognize them as such or see any value in
them.
Ken Stox | Mon, 20 Aug 2012 15:48:13 UTC
Like many other "good" ideas, the Bazaar has been over applied, and used in situations where it is a bad solution. Give a man a
hammer... That being said, there are situations where the Bazaar is a great path to take.
I would argue that UNIX began to fragment in the late 1980's when AT&T's licensing terms became more onerous than the customers
were willing to accept, leading to the creation of OSF. AIX and HP/UX forked off the System V.2 codebase. The only reason SUN
veered from BSD to System V.4 was because AT&T owned a substantial portion of them at the time.
I suspect that we will oscillate around the strange attractor that is the Cathedral and the other that is the Bazaar. After
all, isn't that what Chaos is all about?
iain | Mon, 20 Aug 2012 15:48:55 UTC
Contained within this rant are some good points:
Code reuse is good
Removing unneeded dependancies is good
Fixing up libtool to not check for fortran would be brilliant
However, surrounding these good points are a lot of non-sequiturs and nostalgic romanticised thinkings harking back to the good
ol' days when all you needed was a terminal and a |. I'm sure it was lovely back then, but these days some of us like being
able to plug in our USB devices and have them automatically detected, being able to just click on the wifi network we want to
connect to.
So libtiff is an implicit dependency for Firefox even though Firefox cannot display tiff images, and this is an example of how
badly designed the whole system is. Is it not more likely that someone somewhere (possibly the author, in his haste to show us
how hardcore he was because he compiles everything from source) had forgotten to pass a --disable-tiff to one of the other 121
packages?
Surely even the complaint that Firefox uses 122 other packages shows that code reuse, rather than being "totally foreign in the
bazaar" is actually quite a common occurance?
The bazaar (I'm assuming this is to mean the free software world) is anarchistic? From all the software projects I've worked on
over the past 15 years in Free Software, the one thing they have in common is a very strong sense of hierarchy so that clearly
can't be an anarchy, can they? Large distributions have smart people who are thinking about the best way to put them together,
people can't just fling their code into them, so these aren't anarchies either.
There is also a fair greater sense of reverence to Eric Raymond and his ramblings (the man has written articles about how he is
the embodiment of the god Pan, for goodness sake) in this article than I have seen in the Free Software world. The man is a
hack, anyone who has ever had to look at his code would be able to tell you he isn't very good at it. He stopped being relevant
to the Free Software community circa 1999.
This rant seems to be the ranting of an out of touch old hacker, who's a bit annoyed technology has got too complicated for him
to understand, but rather than seeing that as the inevitable progress of technology (a modern car is considerably more complex
than Henry Ford's, the modern aeroplane is more complicated than the one the Wright Bros used, and the modern computer OS is
more complex than the operating systems of the early 80s), he concludes that it was better in the old days, and everyone today
is a moron.
Kevin Vilbig | Mon, 20 Aug 2012 15:54:38 UTC
What are the cathedrals?
philip andrew | Mon, 20 Aug 2012 15:54:51 UTC
There are problems when building a Cathedral, a small problem in one part of the code has a tendency to be difficult to
discover and fix when the system gets very large. Well, from my experience on a large C++ project, 400,000 lines of code, we
found a few problems.
One, its very hard to debug some problems even with the core dump from the customer with 100 threads dumped, looking through
the threads call stacks.
Two, its very hard to understand the whole thing.
Three, the dependancies all had to be compatible with each other.
Four, when something broke, the whole system crashed as it was one single process.
We ended up breaking the large program up into many small programs which communicated via Corba, you could insert JNI here or
any other communications method. The small programs were then understandable, maintainable, testable. What they depended upon
was relative small, one program did Authentication, one did Authorization, one did Document Storage.
Basically the architecture changed from a single large C++ program to a broken up system.
A Cathedral could work, communism could work if everyone followed the rules, if everything worked well and worked according to
good rules. However, in our real world, nothing works the way we expected. Software has bugs, bugs you have never even through
of. Things fail. People fail.
At least in the Bazzar model I can understand the many small programs. In the Cathedral model, I cannot, at least with C++, it
reaches a complexity limit for the human brain, thats also why we don't program in data-flow. With some other language and
tools and rules perhaps it can be done. Perhaps ADA, that was the idea. (Well I prefer Haskell for the perfect world).
Mark dugger | Mon, 20 Aug 2012 15:57:16 UTC
I respectfully disagree. The Cathedral model only works if one is capable of defining the scope of the problem completely from
the outset of the program. In systems programming, hardware doesn't change nearly as much as an organic customer. Therefore
development processes need to change given what the code is interfacing.
Poul is an excellent developer and I've enjoyed his contributions to FreeBSD and varnish, but his criticisms of the bazaar
model do not hold water. Poul argues that code quality can only be assured if quality can be accounted for easily, I would
argue that code quality can only be assured if the users of the code had leverage. In free software, the currency that the
users yield is popularity and reputation, which developers can flip into fame and respect. Code quality is hidden from the
user most of the time, but the incentive that the programmer can provide the user is useful features that are released often,
which may not heed the best architectural design. In this case, the incentive for developers to write code first fulfilled the
needs of the user first before engineering concerned were addressed.
What he says is true when he described the computing life as simpler once upon a time. People once used computers for
computers to do work for the most part, and now modern open source unix has to be built in a manner where computers are used
both for computers and people to do work directly. While code was simpler, many users of UNIX found the platform(s) wanting.
By the early 90s commercial UNIX was certainly written in a Cathedral model. All of them, most certainly by virtue of a market
that lacked choice given that hardware and OS were tethered, were found wanting.
His thesis of the Cathedral model being superior holds true if and only if the Cathedral in question reacts quickly and
effectively to the demands of the user. Unfortunately for the Unix world, before the rise of the bazaar model, said Cathedrals
in the Unix world have only cooperated enough to turn into an oligopoly. Before the rise of Linux and *BSD, the world had to
suffer through Unix as its detractors described in the Unix Haters Handbook. To wit, the question shouldn't be whether or not
which development process gives one better code, but which one ships better products. The Bazaar model is the development
process gives one access to better user feedback as it welcomes development contributions directly, and henceforth ships
successful products more often.
Phil Regnauld | Mon, 20 Aug 2012 16:04:28 UTC
Poul-Henning, spot on. But it's an onset of different factors, not just the "open source" movement. This perfect storm: a lack
of vision, the internet, combined with a penchant for instant gratification ("TL;DR" mindset) has brought this about. You only
have to read the second comment to convince yourself ("the 'chaos' is what makes programming fun"). No chance I'll be flying a
plane where someone like this has contributed code :) But I think you may have another article to write on quality and
responsibility. Cf. Dan's excellent talk on software testing (Flying Linux - www.usenix.org/event/lisa04/tech/talks/klein.pdf)
Jeremy Huffman | Mon, 20 Aug 2012 16:04:49 UTC
Maybe instead of writing a rant, you could start building the replacment for the autotools - encompassing a more declaritive
shared library management system? If it worked, if there was a clear migration from the current tools and if it presented clear
advantages while being simpler to use then it would eventually be adopted. Its a massive challenge; anyone who has tried to
bite off a small piece of it has just made the problem worse so far, but that doesn't mean it cannot be done.
KP Freds | Mon, 20 Aug 2012 16:10:45 UTC
phd_student_doom, I honestly can't tell if your post is satire or not.
Assuming it's not, go hang up your keyboard and stick to the amateur leagues. In the same way that an amateur woodworker should
not be building houses, an amateur programmer should not be writing software.
Poul Henning-Kamp doesn't address this, but a side-effect of the IT boom -- including the current VC bubble -- was that anyone
that could remotely do the job was hired. This lead to inexperienced engineers having a voice and role grossly disproportionate
to their abilities, and ultimately, lead to 15+ years of wheel re-invention and history repetition amongst junior engineers who
lacked any adult supervision.
In fact, companies such as ycombinator are fueled by and intentionally encourage youthful naivety, as they require a steady
stream of heady grist for the VC mill. All it takes for them to turn tremendous profits is one successful exit out of a hundred
failures.
Your distasteful lack of respect for expertise and experience is the result of this lack of adult supervision.
metageek | Mon, 20 Aug 2012 16:11:54 UTC
This is a typical engineering point of view. Engineers like to think that the world could be designed, however Nature shows the
opposite. Any sufficiently complex system cannot be built out of design, it has to be *evolved*. Evolution is messy
(anarchic?): pieces that work get reused, pieces that do not are lost. There must be something good about autoconf that has
enabled it to survive and eliminate those awful "designed" makefiles that needed manual tweeking even though they did not check
for 15 different Fortran compilers (the status quo of the 1980s). Does it matter that autoconf looks for those now extinct
compilers? No, but it gives it robustness which is why it has survived. Someday a new, less bloated, alternative to autoconf
will appear and it will slowly overtake autoconf. In the meantime we are doing very well with it, thank you very much.
Software is the most complex construction that mankind has so far created. It is so complex that it can only be evolved much
like Nature's products are. Engineers can continue living in their nice little artificial world of linear systems and planned
devices. Those that venture in the real-world of complexity and are successful will take up Nature's ideas.
Goodbye Unix of the 80s. Linux is here to stay
rdm | Mon, 20 Aug 2012 16:15:22 UTC
1. I agree that Eric Raymond's characterization of good software development processes leaves something to be desired.
2. But I also agree with the author, here, that the "destruction of the cathedral" predates Eric Raymond's prose
3. I agree that modern systems have many inefficiencies and inadequacies, and that if someone were responsible for fixing them
that they would get fixed.
I think, though, that if we want to understand why these inefficiencies last as long as they do, we need to look at their
costs: the incremental costs are trivial, for the people currently responsible for propagating them.
(The systemic risks are much higher -- these kinds of inefficiencies are what would be called a "business opportunity" in some
circles. But there are also other "business issues" lurking in the wings. For example: the incremental value of a lot of
software is approximately the same as a [perhaps badly printed] marketing flyer, and the incremental production costs are much
lower.)
drsoong | Mon, 20 Aug 2012 16:32:25 UTC
> Yawn, I can't wait till these type of people retire. I've only been alive 2/3 of the time he's been programming but the
'chaos' is what makes programming fun. -- phd_student_doom
From this attitude, I can see that you're never going to be a good programmer. Other people are going to end up having to fix
your crap. Typical attitude for "rockstar" programmers who lack the skill, intelligence, or foundation to be decent software
engineers. You'd better find a different line of work.
Fazal Majid | Mon, 20 Aug 2012 16:38:59 UTC
I have my own set of makefiles and patches, grown organically over the last 12 years to do something similar to FreeBSD's
ports, except on Solaris and OS X. One of the reasons why it takes a full day for PHK to rebuild is the shocking lack of
parallelism. Only 215 out of 578 packages build properly when GNU make's -j flag for parallel compiles is used, and even then,
more often than not, more than half the total build time is spent in the non-parallelizable "configure" step, limiting your
opportunities to parallelize by Amdahl's law.
The open source ecosystem cannot be a soviet-style planned economy, some duplication of features by competing projects is
inevitable, and probably even desirable as competition fosters innovation. The question is when too much is too much, as with
the proliferation of crypto libraries or crackpot build automation tools.
KP Freds | Mon, 20 Aug 2012 16:47:20 UTC
metageek, sir, that is a load of bollocks: a simple-minded excuse for compounding ill-considered decisions.
You don't have to look far for better examples of engineering decision making.
As a side note, this may be the first time I've seen someone defending autoconf in the past 15 years.
@Iain: "The bazaar (I'm assuming this is to mean the free software world) [...]" This clearly proves that you are one of the
lost generation: You have no idea what a cathedral is in the first place.
@philip andrew, @Mark and others: I'm not arguing that cathedrals is a better solution, they are certainly not without their
own kind of problems. What I'm pointing out that is people like iain don't even know what they are in the first place.
@Jeremy Huffman: I have only a small calendar and must live with it.
@metageek: You're _totally_ missing the point.
Jacob Halln | Mon, 20 Aug 2012 16:54:10 UTC
The *BSD distributions are legacy code. To maintain an operating system distribution you need a massive effort. While nobody
does it perfectly, there are some organizations in the Linux sphere that do quite a good job. Debian/Ubuntu for instance.
The fact that your Firefox build-depends on 122 different packages is not strange. To build some graphics libraries that are
prerequisites for building Firefox, you will need TIFF support, even though Firefox doesn't support TIFF.
The way around this problem for a user is to not build the entire software suite, but instead use binary packages from a
trusted source.
To think that the problems of distribution management at this scale can be solved by up-front design is very nave.
David Kastrup | Mon, 20 Aug 2012 16:56:12 UTC
At Mark Dugger: "I respectfully disagree. The Cathedral model only works if one is capable of defining the scope of the
problem completely from the outset of the program. In systems programming, hardware doesn't change nearly as much as an
organic customer. Therefore development processes need to change given what the code is interfacing."
You confuse the Cathedral/Bazaar dichotomy, which is about continuity of the executive, with working to plan, which is about
continuity of targets.
In fact, combining changing targets with changing responsible programmers is a recipe for cruft: they don't have the knowledge
which code pieces have been tailored to different targets and should be replaced, rewritten, or removed. So some idioms may
stay around when they are no longer a good match to reality.
For example, take a look at the original fork system call: why was this a sensible design choice? Because it had a very
elegant implementation sitting well with the memory subsystem: fork just swapped out the current process but proceeded without
marking the in-memory copy invalid. In the time of demand-paging, this idiom makes rather little sense as a primitive.
Tom Alexander | Mon, 20 Aug 2012 17:13:21 UTC
I'd also recommend reading this much shorter rant from him about Autotools. https://www.varnish-
cache.org/docs/2.1/phk/autocrap.html
metageek | Mon, 20 Aug 2012 17:14:44 UTC
@Poul-Henning Kamp: I thought the point was whether you can plan the entirety of the construct versus building it in an
evolutionary way. My argument is:
1) it is impossible to plan the entirety of a complex system at the outset,
2) most software is indeed sufficiently complex to fit point 1
3) evolutionary construction is the solution: smaller parts of the system are assembled through objective criteria
4) code bloat that is not harmful will last for a long time
if someone cared to remove the fortran checks, autoconf would not work much more efficiently than it does now, so leaving it in
is fitness neutral (which is why it is still there, since it costs energy to remove it)
Apologies if I am still missing the point ;)
@KP Freds: I am not defending autoconf, just explaining why it has so far dominated. I am not making excuses for not believing
that classical engineering approaches do not scale up to large and nonlinear constructions. Nature keeps showing how it is
done, and while it is messy and chaotic, it is incredibly efficient and robust.
I have cursed against autoconf many times too, but the energy to go around its problems is orders of magnitude less than to
make an "efficient" alternative.
maht | Mon, 20 Aug 2012 17:26:33 UTC
Talking of plan9, we have a saying, coined by Boyd Roberts, "this is not Linux, there are rules".
Dave Presotto also graced us with "Linux, by amateurs, for amateurs".
Which, I think, summarises the spirit of your post.
Plan9, where even #ifdef is eschewed.
metageek: cathedrals can evolve, many of them did, some of them have changed major style three times or more. Some of them
evolve to this day. And those are built out of stone :-)
My point really is that Brooks has written a wonderful book, but a lot of people are simply not equipped with the necessary
mental models and background knowledge to understand what The Old Man is talking about.
If plan9 had been available in anywhere near runnable form in 1992, I wouldn't have spent two decades on FreeBSD. I actually
asked and couldn't get a license.
Steve | Mon, 20 Aug 2012 17:36:12 UTC
@Matt Welsh: Funny you should mention Google. They are far from innocent, having built their own Makefile generator known as
"gyp", which have in turn their own portability problems, and often bundling dependencies along with their source, not updating
software to work with newer versions of those dependencies. These things happen out of necessity though. By definition, "hacks"
are things that accomplish the short term goal in the short term. PHK rightly laments the long term cost. Whether you are
talking about software or business, the short term goal is usually the winner. Twas ever thus...
Shawn Garbett | Mon, 20 Aug 2012 17:48:31 UTC
I view the problem is that the chance of project failure increases with scope. This causes most "cathedral" designs to fail.
Address this key issue and the bazaar will move into the background. It'll be like the discovery of steel impacting on the
architecture of buildings if you succeed. Till then the market consists mostly of yurts and tents.
This said, I wouldn't run national security, a mars probe or space shuttle on bazaar code. I.e., how important is the project?
Important enough there will be an investment of time/money to solve the design issues?
Eric S. Raymond | Mon, 20 Aug 2012 17:52:05 UTC
There are a number of ways in which this rant is defective. One of them is mistaking a local problem for a global one; yes,
autoconf is a pile of festering crocks, but the step where this demonstrates that the bazaar model is a failure is just a
handwave and an assertion.
But the largest failure is in the unstated assumption that there is an alternative - that at the scale of Linux or *BSD you can
have anything *but* what Kamp dismisses as "chaos". It is worth remembering that these artifacts are orders of magnitudes
larger in LOC and other measures of complexity than the successful cathedrals of programming legend. They are not just
operating systems in the traditional sense but application suites of unprecedented size.
Nothing this vast can ever have a single person responsible for overall quality, because that task is beyond the cognitive
capability of any individual human being. The implicit alternative Kamp is proposing is impossible - it doesn't scale.

iain | Mon, 20 Aug 2012 17:53:09 UTC
@Poul-Henning Kamp: Great come back. I was going to leave it, but no, I won't. I know perfectly well what the cathedral is,
however none of your complaints stem from any difference between the two models, and rather stem from your belief that all
systems should be simple enough for you to understand and build in under a day. For example, I'm pretty certain your libtiff
complaint was your own fault and if you had passed the correct options to the configure file you would have not needed to build
it.
None of the good points in your rant are anything new, people have been asking why libtool needs to check for fortran for
years, people have been reusing code and trying to reduce dependencies for years.
@ESR: Your reading of my piece is highly defective :-)
I'm not saying the bazaar method is defective, I'm saying it is so "successful" that most people don't even know what a
cathedral is anymore.
Try checking the comments in hackernews or reddit, people are forcefully making that point for me :-)
@iain: I am pretty sure my piece includes this text: "This is a horribly bad idea, already much criticized back in the 1980s
when it appeared," :-)
RORVI | Mon, 20 Aug 2012 18:07:41 UTC
Let me ask you something ....Why do you think these "cathedral breaking programmers" appeared ? Is it because people like you
oftenly refuse to answer a "dumb question" ? Or is it because books like "Teach yourself C++ in 24 hours" appeared ? When was
the last time you answered a question of one of these "dumb programmers" ? The lack of consistency in coding nowadays can also
be the result of a very fast growing industry,that needs more,better code that is written very fast.Also,nobody mentors
nowadays.I have yet to find somebody to spend some of their time to answer a question or two and give some advice.(i myself am
18,and i started programming 2 years ago !).So one of the most painful questions is : where are the mentors,the experienced
programmers and ,for example,a website where they can select beginners in order to mentor them(this should be done for free,or
for symbolic prices,not for 200$ a course,since in some countries the average salary is low-in my country is 300 $).So,in the
conclusion,i would like to point out the fact that you ask for better,but you don't do anything to help shape the experts that
will create great software.
Andre | Mon, 20 Aug 2012 18:11:00 UTC
"Patches welcome."
The sound of crickets that come after those words is the problem. Not the Baazar, Cathedral, Developer, Engineer or Hack.
David F. Skoll | Mon, 20 Aug 2012 18:14:35 UTC
Your article was interesting and thought provoking. You are right: Open-source systems like Linux are a mess for developers.
They are insanely inconsistent.
But like democracy, the system is the worst system except for all the others. Proprietary systems like Microsoft Windows have
plenty of API inconsistencies, weird API warts, magical and mysterious registry settings, etc.
I think these system simply reflect human nature, which is to be wildly creative but (alas) inconsistent, lazy and messy.
@RORVI:
I think you pinpoint a lot of the auxillary causes correctly.
Please notice that I don't blame all the dot-com wunderkinds for grabbing opportunity, I merely note that it was a catastrophy
for CS/IT as a discipline that they got the opportunity in the first place.
With respect to mentoring, I have mentored a fair number of people myself, most of them FreeBSD committers, and I have always
done it for free.
But I have to make a living, so I can't mentor everybody who wants me to.
I also have to be pretty ruthless to my mailbox, and I hate that, but I cannot spend 2-3 hours a day on email that doesn't put
bread on my table. You'd be surprised at the number of emails I get from students whenever a professor gives homework related
to code I have written...
So all in all, I don't feel that your claim that "I don't do anything" is justified.
But I can certainly understand your frustration: I started out in a back-water country, with no net-access, and had to teach
myself everything from books I had to order home from abroad, at great expense. Mentors were nowhere to be had.
rayohauno | Mon, 20 Aug 2012 18:28:56 UTC
@metageek beautiful words !!! I fully agree.
anon | Mon, 20 Aug 2012 18:42:23 UTC
"as opposed to the carefully designed cathedrals of the pre-dot-com years"
Right there in paragraph 6 is the flaw in your thesis.
Neither Cathedral nor Bazaar entails Quality.
CuriousWayne | Mon, 20 Aug 2012 18:44:50 UTC
It's been awhile, but some cobwebs shook loose while reading this post. The Cathedral and the Bazaar was presented as a
metaphor for centralized control of code in the Data Center (AKA the priesthood of Data Processing, hence, a Cathedral) versus
distributed programming (AKA Linux).
From the first reading, I sensed a weakness in the metaphor; while the Data Center as Cathedral works OK, Linux was not a
Bazaar. Linus Torvalds was King of Linux and dictated what got in and what did not for the longest time. Linus was a single
programmer scratching his own itch for a usable, affordable OS (the most effecient and effective programming team possible).
Furthermore, the DotCom bubble was not a Bazaar. Bazaars are competitive but predictable; it is highly unlikely for someone
with too much money to come into a Bazaar and buy out all of the vendors' stuff.
Yet that was exactly how a Silicon Valley VC, after the bubble burst, described to me why the bubble happened in the first
place; too much money was chasing too few opportunities, and the VC community lost their quality controls on their investment
process.
I have respect for Mr. Kamp and his perspective on software engineering. I have respect for the modern paradigm of Web
programming that breaks a simple idea like a website into a billion moving, dependent parts with specialists running all over
the place. The two perspectives are somewhat mutually exclusive (or similar) in that they are solving different technical
problems using coding techniques.
Having worked in other engineering disciplines, none have the challenges or the leverage that software has. Software has no
universal and unambiguous design language, nor is it likely to have one anytime soon (stuff changes too fast; the design
language is more dependent on the makeup of the team than the problem).
By contrast, any junior engineer in any other discipline (anyone know of an exception to this?) can spec a design and get it
made on the other side of the planet by people that share nothin else than a universal design language.
Here's what I think is a more apt metaphor for today vs legacy than the cathedral and the bazaar: if you had a performance
problem in legacy days, you might get fired if you 'threw hardware at it'. If you have a performance problem today, you might
get fired if you don't, because hardware is now often cheaper than paying to tune code.
@anon:
Where do you see "quality" in the text you quote ?
Just because something is carefully designed doesn't mean that it is good.
See "galloping gertie" for just one of many examples of the opposite.
Damn, I hate it when I remember an even better example seconds later:
@anon: Best example: The Hubble telescopes main mirror: The mirror that was most precisely ground wrong, ever.
Tommy McGuire | Mon, 20 Aug 2012 18:53:45 UTC
PHK is said to have been programming computers for 26 years. I would have thought he would have remembered the '80's and '90's
at least as well as I do, but I could be wrong. I also thought that the whole point of the BSD effort was to build software
that anyone could do anything they wanted do with; certainly IBM did that, when they created an ld(1) that required a good deal
more than one flag to build a shared library, among other things.
Should standards have done something about that? Sure, but standards are not handed down by the Almighty, or even by Fred
Brooks. They're built by people with many agendas, including ensuring that the standard does not inconvenience their market
share. (This might make PHK smile; it does for me: the memory of the environment variable POSIX_ME_HARDER.)
Now, I suppose you could blame the hideous environment that resulted in the terrible solution of the autotools on the bazaar,
in that AT&T did commercially license Unix in the early '90's and thereby allow more people to soil it's pristine beauty. But
to say that was "robbing it of its architects" seems to me to be a bit too much. I knew some of the people working on AIX at
IBM at that time, and they were all very good, and all as dedicated to the beauty of the cathedral as anyone you could put
inside any fortified wall you want to build. But, heck, if I really wanted to pinpoint the moment Unix began to fragment, I
might have to go back to the seventh edition, which did not have the odd proliferation of system calls that includes socket(2).
So, if any wunderkinder really deserve blame, they are probably not kinder any more. And since the bazaar is so bad, remember
that there are cathedrals out there for you to worship at; they must be better. Right?
Paul W. Homer | Mon, 20 Aug 2012 18:57:00 UTC
You are right. The stuff we have has been degenerating for years, and each new wave of kids blows us off because they lack the
knowledge to really understand the technologies that they are wielding or creating. I also think that it's gotten significantly
worse in the last decade, that we've right in the middle of a 'dark ages' that is dragging us backwards.
At first I just ranted but nobody wants to listen, they're too busy re-writing the same wheels over and over again. These days
when I blog I try to be more positive, but not always:
http://theprogrammersparadox.blogspot.ca/2011/05/software-virgin.html
It's all very disappointing, and the trend has been for it to get worse. I think it's going to be a long time (generations)
before the winds change, people tend to wear blinders when exposed to too much complexity, which forms a vicious cycle.
Paul.
PS. I miss Unix too, my favorite version was a pure, clean, Sys V from AT&T in the early nineties.
@Tommy McGuire:
I have it from first hand, that people at Bell Labs found the socket API hideous, and the SVID IPC and STREAMS was even less to
their liking.
With respect to standards, yes: The UNIX-vendors shot themselves (and us) in the foot bigtime, by photocopying the SVR manuals
into "the UNIX standard" (how ever many there were of those), rather than look at what the future would require of them.
That's why we still call N system calls just to open a TCP socket connection to somewhere, rather than a convenient single
function in a well designed API.
But the lost dot-com generation is an entirely different issue: They have essentially wiped out all CS history before 1990, by
never bothering to read any of it, and that means we have to slowly relearn what we once knew about the craft of IT.
And the particular fact that the do not even recognize the architected cathedral, is a particular expensive bit of amnesia.
Justin Alan Ryan | Mon, 20 Aug 2012 19:10:58 UTC
Cathedrals are built by some of the greediest, yet least financially stable companies in the world.
I also find it really interesting that the author insists on using FreeBSD on his laptop, but clearly knows less about free
software and dependencies than I know about my Ubuntu hosts.
There are some sticky points when it comes to dependencies. A cathedral would provide us with less choices. Why does glib
depend on both Perl and Python? Because glib is a library that many people use from Perl and Python, and it's set to build
those bindings.
Buy a fucking mac, install XCode, and tell me about the beauty of the cathedral.
Dear Justin,
Thanks for illustrating my point about the ignorance of the lost generation so forcefully.
You clearly have no idea about what a cathedral is in this context, you clearly did not bother to find out and you spew random
insults at the author, without even bothering to check if you were about to make a total ass of youself thereby.
The reason I run FreeBSD is that I wrote a very significant part of it and therefore I probably know more about dependencies
than you ever will. Building the perl/python bindings as part of glibc is ass backwards in my book. As for your "fucking
mac", last I checked, my name was 35 times in the kernel of the open source version ("Darwin") of the OSX kernel.
Peter Lindhard | Mon, 20 Aug 2012 19:43:38 UTC
Perhaps it could be a general point in OSS projects, that contributors are reminded/persuaded to reread their parts at a later
date. I honestly found it disturbing, to read what seems like disappointment that newcomers don't/can't do 20 years of code-
clean up.
Author seems blind to new challenges/interests, that new developers face/embrace. To be very honest antisocial ranting apparent
in several publications by this author, makes for yet another terrible read.
mcur | Mon, 20 Aug 2012 19:50:47 UTC
Mark Williams wasn't a paint company.
http://en.wikipedia.org/wiki/Mark_Williams_Company
Sven Trpe | Mon, 20 Aug 2012 19:54:50 UTC
You are free to go to a restaurant rather than to the bazaar. You seem, however, to insist on preparing your meal yourself.
Yes, this involves a variety of imperfect tools, and it makes a mess that you'll have to clean up befor you're done. But such
was your choice, you wanted to prepare a meal and not go to a restaurant where the chef would have done it for you. In the
restaurant, you install Firefox by dragging an orange icon over to the Applications folder, that's it. Somebody else takes care
of the 122 ingredients for you.
Rick Russell | Mon, 20 Aug 2012 20:26:14 UTC
Mr. Kamp, your credentials are without fault, and I certainly am not qualified to question any of the specifics you discuss.
But despite all these perfectly legitimate observations, a good Linux distribution like Ubuntu is not only easy to install, but
it is easy to maintain. The issues of package dependency, whatever they may be, are effectively hidden behind a rock-solid
package management system that seems to have very few failure modes, at least to this 80s UNIX user and occasional programmer.
I remember installing software on AIX, SunOS, Solaris and IRIX systems back in the late 80s & early 90s, before the Linux
revolution and I would say that, subjectively, it was much more difficult to get working software back then. I don't just mean
open source, although open source was part of it. Perhaps it was because the dependency mapping was not as complete -- you'd
install a piece of commercial software on your Solaris environment and find out that one part of the software required a
version of Motif you didn't have, and then you find out that version of Motif isn't compatible with the X terminals your labs
use, and then... ?
There may be errors in the dependency tree for some applications in FreeBSD, but at least the dependency tree *exists* in some
form. If individual packages are claiming unnecessary dependencies, then maybe the people who write that software need to clean
up their act. In the bazaar, you have to tolerate such sloppiness. In the cathedral, that software might not be available at
all because it never sees the light of day. Neither solution is optimal, but the bazaar seems preferable, to me, since at least
I have a shot at getting it to work.
Elizabeth Marston | Mon, 20 Aug 2012 20:30:09 UTC
Great article. It's a shame how much ignorant ageism is in the comments --- winning an argument on reason rather than emotion
is, ironically, an application of rationality at which engineers are often poor.
What cathedrals have going for them is that they are beautiful. What bazaars have going for them is life.
FOSS didn't win the mindshare wars through propaganda and hazy hippie ideas about sharing; it won by being better; by building
more slowly; by being inclusive. And inclusivity always means mess, because not everyone has the same intentions or tastes.
And how is UNIX dying? Please support this claim. From where I sit, it looks like it's winning by default. Code I write for
Linux tends to resist bitrot. Unlike, say, the code I've written for Windows-land, where MS effectively says "and now for
something completely different" every four years (R.I.P., C#) and thereby generates enormous waste.
It's that careful, incremental, inclusivity-loving bazaar mentality that allows us to bring the full force of our UNIX and C
heritage to bear on any problem, and is the essence of code reuse.
The results speak for themselves. My apartment has a single, old, moldering Windows box and a slew of iOS, OSX, Android and
Linux devices. That UNIX:Windows ratio has been constant for about ten years. What's changed is that my retired parents' place
is now approaching that ratio as well.
Put more clearly: software that has no unnecessary parts is software that can never be put to any purpose except those foreseen
(and permitted) by the designer. It is as fragile and as beautiful as a snowflake, and tends to last about as long before
someone has a better, entirely incompatible idea. It's what killed Esperanto, and it's what will kill Windows (and many other
commercial projects.)
Top-down design works great for as long as everyone shares (or at least respects) the intentions of the designer. But that
necessarily means quashing other voices. Perhaps some of those voices aren't as clear, or those speaking have impoverished
mental models of how the OS functions; but others will simply be different opinions about an underdetermined solution space.
(Quick clarification on my last post: -- the phrase 'what will kill Windows' isn't clear in context, as Windows has many
unnecessary parts :3 What will kill Windows is the _anxiety_ that its developers have about the unnecessary parts, and their
repeated attempts to deprecate whatever isn't cool this week.)
Hector Correa | Mon, 20 Aug 2012 20:53:47 UTC
I find it interesting that even systems owned by a single company (like MS Windows) are also messy.
Microsoft has done a great job of simplifying the installation of Windows (particularly compared to what you describe for your
system!) but as David F. Skoll mentioned in a previous comment the API is equally messy if you look behind the scenes. Heck,
there were even some well publicized stories about some of the hacks buried in the Windows source code to account for wacky
scenarios (e.g. http://www.kuro5hin.org/story/2004/2/15/71552/7795)
Even more interesting, is that none of these systems (Windows or Unix) are typically maintained by "casual software developers"
that just took on development overnight during the dot-com era.
I think the mess inside these systems is just the nature of large systems, maintained by a large number of people over a long
time period.
Mike Smith | Mon, 20 Aug 2012 21:16:28 UTC
Poul,
Thanks for putting this together so cogently. Your rants are always thought-provoking and a good read.
Your point about design had me wondering; why is "good design" currently trendy when it comes to UX, but uncool when it comes
to software (and many other engineering disciplines). Is it really just a socialisation problem?
= Mike
Isn't that simply because "civilians" see UX, but not source code, and there are more people with a sense of style, sthetics
and architecture amongst the "civilians" ?
Poul: With a background that straddles both UX and low-level dev, I've been wondering this myself. At first I thought it was an
aesthetic thing. But now I think that it's just straight-up conflict of interest, like we're used to seeing elsewhere in the
public sphere. Developers can't advocate for users because they're _developers_ -- their best interests are in long-term
architectural stability and elegance, not in ease-of-use or rapid production. Developers are intrinsically lazy, and do not
wish to make work for themselves; and in every other context, this is a good thing. Do not expect a developer to agitate
passionately against the best interests of developers, as a UX specialist must often do.
Jonathan Abbey | Mon, 20 Aug 2012 21:45:25 UTC
I hope you will follow this column with one that more fully illustrates the nature of these cathedrals, ideally with examples
for us to enjoy.
David F. Skoll | Mon, 20 Aug 2012 22:01:29 UTC
Oh, some more random thoughts... I've been programming professionally for about 18 years and have participated in my share of
both open-source and closed-source projects.
In my experience, no matter how careful you are and how good your intentions are, almost all large software projects tend to
have a few ugly dark nuggets at their core. Perhaps it was done that way to work around limitations of the development
environment, or perhaps the original design was found lacking and it was easier to patch things up than do a major redesign.
But we're only human: We can't predict everything and sometimes a workaround (aka a hack) is more efficient and more productive
than a redesign. Sometimes the perfect really is the enemy of the good.
Again writing from personal experience, I'm much more likely to try to exorcise the ugly dark nuggets from an open-source
project than from a closed-source one. In the open-source project, my peers can read my code and I might be subject to their
derision. In a closed-source project, everything can be swept nicely under the rug.
(I realize that "open source" does not mean "bazaar", but "closed source" usually does mean "cathedral.")
Kartik Agaram | Mon, 20 Aug 2012 22:09:57 UTC
@Mike Smith, @phk: Reading your conversation it occurred to me that for all the benefits of commandlines that we geeks like to
rant about, a GUI has one key benefit. Anybody who looks at it can instantly spot ugliness and cruft to some extent. By
contrast, a commandline-oriented stack is a lot less discoverable, and encourages the formation of various 'commons' like
dotfiles, extra commandline flags, open file handles. Since nobody is monitoring these areas for cruft, we developers dump
indiscriminately on them.
This point is independent of how savvy the reader is, I think. My visual cortex instantly shows me what's crufty about a UI.
Dotfiles on the other hand I have to deliberately go dig around in to criticize.
Anton Striklov | Mon, 20 Aug 2012 22:11:16 UTC
As much as I like FreeBSD one cannot equate the success / failure of the whole bazaar ecosystem to it. FreeBSD is awesome in
some aspects and terrible at others, and while many projects lag behind it, plenty far surpass it. It's package management
system for example if far inferior to the one present in Gentoo. But why stick with Unix OS's for analysis ? The sheer volume
of non-OS related successful projects developed under the bazaar model is staggering !
ford p | Mon, 20 Aug 2012 23:54:18 UTC
This article is horseshit. Total flamebait and founded on the obviously false assumption that you ever need to compile a whole
distro from source. Was this article written in the 1980s and you just got around to UUCPing it off from paper tape?
Bernd Jendrissek | Tue, 21 Aug 2012 02:54:11 UTC
Much of your rant seems predicated on the badness of the autotools. While some of your criticisms are certainly valid (even
though I might not necessarily agree what the response should be), you seem to be burning the sources of inelegant software in
effigy, by attacking the autotools. The creators of the autotools are not the ones responsible for the problems which the
autotools were designed to address!
Secondly, some of your criticisms (here and in your autocrap rant) against the autotools are again misplaced: you seem to want
to hold the autotools' creators responsible for the misuses to which others put them, sometimes due to innocent ignorance,
other times due to an arrogant sense of superiority (poor translation: I want to convey the german word "besserwisserisch").
As others have pointed out, I'm not surprised either that there are features in a large dependency graph that go into a black
hole, like the TIFF support you mention. My curiosity is somewhat piqued to know how this irony resolves. And specifically, is
this a run-time dependency or a build-time dependency?
But here's another irony: you don't recognize the deliberate architecture to which the autotools were built. I wasn't around at
the time to know if they were originally built cathedral-style (their development now seems very bazaar-like), and you might
not appreciate the architectural style used, but boy, you'd better believe it, much of your interaction with them was
*designed*. In my experience most of the "friction" in interacting with the autotools is due to the very sort of meta-ignorance
you write of: not even knowing what the tools' raison d'tre are, or "disagreeing" with them (as if that is possible). I like
to say: Those who do not understand the autotools are doomed to reinvent them, poorly. Please understand that I mean a
philosophical understanding; I'm sure you know very well what AC_CHECK_LIBS does, for example. And yes, I think I can agree
with your criticism of keeping around 20-years-superseded feature checks.
P.S. I know I'm conflating cathedral-as-designed-artifact with cathedral-as-home-of-a-priesthood.
mike mccafferty | Tue, 21 Aug 2012 03:19:19 UTC
The dilution of a profession's core skills due to population explosion strikes me as an interesting twist on Naomi Klein's
"Shock Doctrine." Dotcom-era opportunities create a virtual Gold Rush, people with wildly varying degrees of of technical
skill rush into a space, and the IT population growth rate is such that experienced hands simply cannot transmit best practices
fast enough to keep pace with the population increase. Soon the notion of "state of the art" has changed, because a large
percentage of the population doesn't know where to look, and doesn't know who to ask for pointers. It's the "telephone" game,
played out on the clock and at a velocity that prevents error correction.
Thanks for recommending out the book. I know that I will benefit from the lessons contained within.
Terry Lambert | Tue, 21 Aug 2012 03:51:25 UTC
When Sun switched over to SVR4, it became impossible to to get BattleMap to run on any decent hardware. Now it's sold by
McCabe software as McCabe IQ. It's down from $50,000 a seat to $45,000 for three floating licenses.
I think one of the issues with the Bazaar model is that younger programmers, particularly the ones Poul-Henning identified as
coming out of the .bomb era as being hired out of their second year in CS coursework to be cubicle warmers at startups which
went nowhere, wouldn't recognize a decent CASE tool if it bit them on the ass.
Complexity is manageable through abstraction, and through good tools, and anyone who doesn't understand that is doomed to
recreate their own version of the autotools mess in their own projects. Eventually I will probably run into them in McDonalds
while they are asking me "Do you want fries with that".
Also, I take umbrage at Phillip Andrew's "break everything into tiny programs" model of development; an inability to manage
complexity is no reason to drink the Daniel J. Bernstein Kool-Aid that has djbdns _still_ unable to do zone transfers, and
synchronizing primary and secondary DNS servers "is an exercise left for the student". The attack surface on 50 small programs
is way larger than the attack surface on one large program. It's why we still build monolithic kernels.
Poul-Henning and I haven't always seen eye to eye on everything, but his post here is spot on. Good on you, PHK.
Max Kanat-Alexander | Tue, 21 Aug 2012 05:33:19 UTC
First, let me say that this is an excellent article. For many years, I was the Chief Architect of the Bugzilla Project and
heavily involved in the open source community as a whole, and I can assure all readers here that there will never be a model of
software development in which developer responsibility can be neglected. In every model that will ever exist, developer
understanding and responsibility will be the key component to quality. There are methodologies that make quality hard and
methodologies that make quality easy, but there are no models that create quality code without guidance from some unified
viewpoint and individual responsibility for software design down to the lowest levels of development.
The bazaar has all the problems that anarchy has--if a few good people step up to accomplish and lead things, then they go
well. If nobody steps up to be responsible for quality, then even though things get built, they can't be nice things.
The cathedral has all the problems of a dictatorship. If the dictator is wise and benevolent, then the project prospers and is
beautiful. If the dictator is ignorant or possessed of motives beyond the good of the project, then the project can fail and be
disastrous.
Our modern economy throws in one additional crowbar to the cathedral as well--that most people making centrally-controlled
projects are creating them under the banner of organizations that do not want external contributions and which do not which to
collaborate with others. The cathedral has been creating just as much duplication of effort in the past many decades as has the
bazaar.
And finally, an even more salient point to touch on is that perhaps "the bazaar" is itself a myth. The best open-source
projects are, in reality, cathedrals in which anybody is allowed to build pieces, but only after they have passed the approval
of the central control. We could certainly image a "bazaar" to exist--some set of projects that have somebody birthed
themselves out of anarchy. But there's no situation in which that's true. Software is conceived, created, and guided by people,
and when those people have great understanding and all take strong responsibility, excellent work is done. When they neglect
their code or assume that "somebody else will take care of it," then ugly disorder ensues.
-Max
Justen | Tue, 21 Aug 2012 05:45:25 UTC
Sounds like your problem is a crap package management system more than anything. I can't believe you have to actually manually
research and add dependencies. What decade are you living in?
I'd also like to note that development methodology has little do to with the problem of quality. The internet is swimming with
horrific windows shareware that is not only *not* package managed, and full of bugs and compatibility issues even though they
depend on an essentially monolithic platform and set of libraries, but still inexplicably expects us to pay for "full"
features. One of the greatest joys I discovered in moving to Linux was all the free, relatively bug-free, fully functional
basic utility software of the kind that every computer needs to be useful - all in a central packaging system that not only
installs software in one click (along with all its dependencies) but keeps track of when it needs to be updated too! And don't
even get me started on Mac, a platform so terrified of free (as in beer or as in freedom) and open source that I was just asked
to pay $8 for a script that finds and consolidates duplicate music files in iTunes. Are you *serious*? How is that not a built-
in feature of every media library manager in existence, let alone one of the keystones of the Apple experience.
Nah man, I think you're looking at this through too narrow a lens. And maybe you need to get off FreeBSD, the red headed
stepchild of of ghetto *nixes.
Justen | Tue, 21 Aug 2012 06:43:22 UTC
After reading the further discussion in the comments, I have a greater understanding of what you're actually trying to complain
about here. But you've turned what ought to be a simple lamentation about cruft into a jermiad. There are reasons why this
cruft sticks around: because somewhere out there, someone wants to install a bit of *nix software on a truly ancient machine
where it will actually matter; because a few thousand clocks is infinitesimal in cost to any given user compared to the cost to
a developer to carefully excise it; because the vast majority of users don't know it's there and wouldn't even know why it's
bad if they did; because most of us use binary package systems, only supplementing with compiled-from-source binaries when they
are too obscure to merit inclusion in the repositories.
The *nix ecosystem is incredibly massive. If it's true that you can only get a cathedral from continuity of executive, and that
a single virtuoso, or rarely a dynamic duo, must manage an entire project with an iron fist in order to achieve cathedraldom;
then where is this godlike ego you propose to manage all of *nix? Or shall we commit the even bigger sin of tossing it all out
and starting over so that, this time, we can do it "right"? Even when standards are proposed, or when they emerge from common
practices, there will always be the outliers (*coughbsdcough*) who insist on doing it differently and they'll still want to run
common *nix software. Inevitably some monster like autoconf and friends will arrive on the scene to facilitate that.
Now as to M4, I have no apologies for that. Ugh. I thought Perl was opaque until the first time I tried to configure sendmail.
Pete | Tue, 21 Aug 2012 07:10:25 UTC
There are some good points here, but there's a couple glaring problems.
The first half of the article laments the mass of "college dropouts" who entered the technology field around the turn of the
millennium. The second half of the article laments the mess of legacy technical decisions that we are stuck with. But for the
most part, the former did not cause the latter!
For example, you spend a while talking about the disaster that is autoconf. (I think this is one of those tar-babies that most
people don't bother talking about any more, because everybody knows how bad it is.) But it was about the furthest thing from a
dot-com-era college project: it predates that party by about a decade. It was created to solve a real problem (hand-editing
configure scripts), back in the day.
This leads straight into the second problem. It's perhaps not fair to complain about autoconf itself, since it solved a real
problem back in the early 1990's, but it is fair to complain about modern projects like Firefox continuing to adopt it, rather
than being more portable from the start. But Firefox, for the most part, *works*. It exists. It delivers what people want.
Imagine a web browser built with more solid software engineering principles. Does it give users anything they don't have
today? (Firefox is pretty stable, and it does everything I think I need.) Does it make life better for its developers?
(Apparently there are developers willing to work on Firefox as it is today for free, but I don't see anyone doing the same for
a serious alternative.) What tangible benefit would it actually have, except to make some of us feel better about it? We have
working, free, open-source web browsers, which are faster and more capable than ever. I wish they had a better architecture,
but it's hard to complain too much when things are so much better than 10 years ago.
Finally, you also dish out against Unix and autoconf quite a bit, but I'm not sure if this is because other operating systems
are any better, or because it's simply the most public one, so we see all the problems.
holger | Tue, 21 Aug 2012 10:22:05 UTC
The true cause indeed seems to be the explosion of IT "professionals" we saw in the years leading up to and during the dot com
boom. Things just had to get done (and quickly) because otherwise someone else could have been first to market. Creating an
incredible demand for IT "somethings" and attracting lots of people with little or no formal education in the field. But the
problem is not only lack of education: today the core problem applies to cathedral and bazaar software in exactly the same way.
Because what coincided with the dot com boom (of not being the cause of it) was that hardware advanced faster than ever before
in those years. Meaning you could pretty much get away with less then optimal code (and coding practices) most of the time.
Almost nobody cares about efficient design (much less code efficiency) anymore, it's all about quick design and development
efficiency. Once something (mostly) works, it's good enough. Needless to say, it usually isn't,
but that isn't easily noticable on the developers machine. But deploy that code to a hundred thousand servers or a couple
million users, and the overall balance shifts - inefficient code translates to wasted computing resources and wasted energy on
a very large scale. And to make matters worse, object oriented languages, while being in invaluable design tool to the
programmer who is experienced with them, make it unbelievably easy to write horribly inefficient and bloated code without even
noticing if you lack said experience. (Not to mention to include more dependencies than you can count if code reuse means
"someone else probably already wrote something similar, so i'll just google" to you).
Someone actually suggested using the libgmp bindings for bit manipulations in some language that apparently didn't readily
offer them in a comment on thedailywtf.com because he googled and found out that it had something similar (for bignums if I
remember correctly, but hey, you can always convert and back, right?). That was one of my largest wtfs i ever saw there.
And even the better programmers out there often end up adding layers over layers of abstraction to make an overly complex
problem somewhat tractable without ever questioning the underlying design once there is one that seems to work "good enough".
In a bazaar style project, there's at least the off chance someone is going to revisit things. In a cathedral style project,
design decisions tend to be set in stone once the first release is out "that's just the way things are done around here".
Alkonaut | Tue, 21 Aug 2012 10:41:13 UTC
Don't confuse the "bazaar" with Linux. Linux is a perfect example of a cathedral. Complete with its high priest. The linux
*distros* (e.g. GNU/Linux) shows the real "chaos" of the bazaar.
foljs | Tue, 21 Aug 2012 10:42:18 UTC
@phd_student_doom
"""Yawn, I can't wait till these type of people retire. I've only been alive 2/3 of the time he's been programming but the
'chaos' is what makes programming fun."""
Well, speaking as someone of your age, sure, it will be a very "fun" IT landscape indeed when ignorants like you take over
(even more so considering this BS non-response was written by a "phd student").
This "chaos" is what makes programming fun?
Spoken like someone who doesn't understand accidental and essential complexity at all, and cherishes trivial bullshit tinkering
over actually making something, much less making something beautiful.
Glenn French II | Tue, 21 Aug 2012 12:32:15 UTC
I figured that I would chime in once I read all of the other unprepared and ill-advised comments.
On another note, thanks for all of the contributions.
David MacKenzie | Tue, 21 Aug 2012 13:34:45 UTC
As the creator of Autoconf, I'd like to apologize to everyone for using m4. I chose it to minimize dependencies, as it was on
every Unix-like system already. Many of us were on 28kbps dialup and people would complain about the time needed to download a
big package like Perl. Then I hit so many Unix m4 limitations and bugs that I ended up requiring the GNU version anyway. I
probably should have started over at that point with another approach. It's true that I did most of that work before I'd earned
my CS degree, but no one more experienced deigned to tackle the practical problems I needed to solve, so I had to. (Larry Wall
had half-tackled them.)
When I did the initial design of Automake a few years later, Perl had become widespread enough that I could get away with
specifying it as the implementation language. (I wasn't involved with libtool.)
The core GNU project (not to be confused with the ports or other packages like X11) does have a quality assurance dictator of
sorts in Richard Stallman and the GNU Coding Standards. A lot of the work I and other maintainers did on GNU was bringing other
peoples' code up to those quality standards, so that our versions of POSIX commands and libraries had fewer arbitrary
limitations and buggy corner cases than the Unix versions that originated in Bell Labs. Our source code was also supposed to be
readable and well documented. We had the advantage of another decade or two in processing power and RAM at our disposal
compared to the original Unix developers, who by then were working on Plan 9 which was elegant but never really became
available to use until it was itself a dead project.
When I started Autoconf, there were no full Unix-like operating systems with all the source code available to everyone, so we
had to bootstrap ours on the variety of existing systems and accomodate their variations.
It doesn't solve any practical problems to say that applications like Firefox should be written in a more portable way so there
isn't a need for something like Autoconf. Unless you want to reduce the whole global software industry to a single company or
government regulation, communist-style, you can't control what other people are going to do. And so you need to work around the
diversity.
The dependencies-of-dependencies tree can indeed get large when you're leveraging general-purpose tools. And keep in mind that
although you need to build some seemingly unrelated things in order to end up with Firefox, those things might end up getting
used for something else you're also building. Most people just download binary distributions, anyway.
What I haven't seen so far in this discussion is a workable real-world proposal for what to do differently other than a few
incidental details about particular software. The complaint seems to be about the big-picture approach, but I suspect Eric
Raymond is right that no cathedral can be large enough to encompass a whole industry. And that's effectively what the FreeBSD
ports system contains.
Frank Greco | Tue, 21 Aug 2012 14:48:55 UTC
Brilliant article. I've been thinking of the same stuff since the Cathedral book. It seems in other endevours there exists
quick spike, multi-creator mediocrities and high-priest design masterpieces. From building architecture to music to paintings
to TV shows to finance, et al. Apparently software is no different.
Poul-Henning Kamp | Tue, 21 Aug 2012 17:58:42 UTC
@David MacKenzie:
I don't dispute that autoconf was a pragmatic solution to the madness of UNIX vendors back in the 1980'ies and 1990'ies. It
certainly saved time for me back then.
But today it no longer serves a useful purpose, other than add incomprehensible onamentation and woodoo incantations to open
source software, for no credible gain in overall quality or portability.
I am certainly not looking for "The One Unix To Rule Them All", but I would really love to see some intelligent standardization
fill the worst pot-holes, so we don't have to waste so much time while autoconf checks for and libtool looks for 26 fortran
compilers, twice.
The one thing cathedrals is better at than bazaars, is throwing junk overboard, that's probably the main reason why I wish we
had more cathedral and less bazaar.
As for M4: Apology accepted, but can we please loose it now ?
Marty Fouts | Tue, 21 Aug 2012 19:09:26 UTC
Mr Kamp is right, but it is clear from the responses that he is not well understood.
I have programmed professionally since 1972. At that time the computer industry was already twenty years old, but those of us
who entered it studied its history and learned from it. Sometime between 1980 and 1990 that willingness to learn lessons from
past projects disappeared, while a sense of hubris entered the industry, epitomized by the conversion of our job titles from
"programmer" to "engineer" that took root in the 1980s.
Now, the computer industry is always twenty years old, and those who come to the profession no longer learn the lessons of the
past. Recently, an algorithms "expert" I interviewed, admitted to having no idea who Don Knuth was, or why they should be
aware of his work. Such interviews are becoming more, not less, common. I can point at hundreds of such examples of dumbing-
down in the industry: OS kernel developers who can not describe the difference between a write-through and a write-back cache;
people claiming to be C++ experts who can not enumerate the member function in the class definition 'class f {};' C#
programmers who can associate the concepts involved in boxing with the underlying mechanisms of stacks and heaps, nor describe
the machine level implementation of either. The list is endless.
Nor is it limited to the people who entered the industry in the early 2000s and remain. It is a trend that continues to this
day. University computer science programs are laughable, and the produce both the above, and at the other extreme, the sort of
highly intelligent but extremely inexperienced people that companies like Google hire and that go on to produce abominations
like the build system for ChromiumOS. Or the people at Microsoft, who because of the nature of its career path are always
moving to new projects for which they have no experience. Microsoft's software looks like it was written by people with three
years of experience and no mentoring because it was. Even programmers who have been there for two decades have rarely worked
in one area for more than a couple of years before moving on to something else.
We are sinking in a morass of mediocrity. In a world where every programmer, regardless their background, experience, or
training, is a 'senior software engineer', we should expect no less.
Roboknight | Tue, 21 Aug 2012 20:02:19 UTC
I think most of the commenters here pretty much missed the point.
1) Yes, this is a rant.
2) The point of the article isn't the fact that it requires 122 packages to build Firefox, or even the additional bloat that
goes with it. Poul is mainly concerned with how we got here. It does wander around the mark a bit, but generally, I'd have to
agree with the argument.
I myself have built MANY projects and typically don't lament the fact that it takes 122 packages to build something (although I
have been there too), but you can see the inexperience in many of the projects, especially the ones that have a claim along the
lines: "This project was great, but was poorly designed, so we are going to rewrite it." This doesn't just lead to something
requiring 122 packages to build, but to having things like Gnome and KDE installed on the same system because some package you
wanted to use wasn't built with the libraries you chose to install on your system. This especially occurs when someone has
written a nice tool that you would like to use, but uses a different set of libraries than you do. Compviz, while a really
cool effort, suffered from this as well as other issues.
I also agree with Poul that while I'm excited that open software has taken off, I'm saddened to see that the Cathedral model is
more difficult to find and has pretty much been overrun by the Bazaar model. I had hoped that the two models of software
development would feed off of one another and the deficiencies in both models would be overcome and platforms like Linux/BSD
would have been able to avoid "development bloat" and been a little more cross-compatible than they are. Maybe avoiding
clashes like Gnome v KDE or having several thousand copies of encryption code.
Finally, to those that completely disagree with Poul, look at the opensource landscape. Take a look at how many repeat
projects, dead ends, and restarts there are out there. In this kind of environment, I'd agree that what is out there is
certainly an "evolution", but even evolution has a plan that does typically strip away inefficiency. I can't say that truly
occurs with opensource. The projects that seem to succeed beyond others are the ones that have a single driving force moving
them toward a common end goal. Apple has picked up clang, for example, and that seems to keep moving forward. Google has
picked up a few projects. The ones that have not been "picked up" and are successful, still have dedicated developers working
everyday to make them better. I'd liken those developers less to a cathedral and more to a glass tower, where everyone waits
and watches for the next development. Every once in a while, something comes in and out of the tower (i.e. outside patches and
so forth), but in general, things are added and integrated on the schedule laid out by those that are dedicated. Typically,
those dedicated individuals have put in the time and effort it takes to become the "good carpenter" that Poul talked about.
But there are many more "wood pushers", to use an old Chessmaster term, out there. And their influence really turns the Bazaar
into a difficult place to develop, or even build. Yes, you can be successful, but it seems that it was once easier to do so.
Alexander Vlasov | Tue, 21 Aug 2012 22:30:41 UTC
I, for one, can't believe that a person with PHK's skills and experience really doesn't know why compiling glib needs both perl
& python or why libtiff may be required to build firefox & its dependencies from scratch.
So this is merely a provocation. Nice try.
sl | Wed, 22 Aug 2012 01:06:33 UTC
@Poul-Henning:
Come to Plan 9. We don't have autoconf at all.
Ivan | Wed, 22 Aug 2012 02:34:30 UTC
Seems silly to rant about code reuse but then suggest rewriting autotools when sane build systems like cmake exist.
Klapauzius | Wed, 22 Aug 2012 09:47:13 UTC
I think the poster bases his argument - or rant, if you will - on an intellectually questionable trick: Yes, there are the
dotcom-JavaScript-PhP-Wunderkinds (what Jamie zawinsky describes as "Attention Deficit Teenagers"), but what they do is build
silly, bloated Webpages. They don't really program on an application or even system level because they can't. If they try, they
fail spectactularly, case in point being the disaster that is GNOME 3 (which in the end shows that the OSS model works very
well in axing failed projects, much better than in the corporate world). But the author makes the leap from these Web-spamming
"New media" whizkids to system programming which is a totally different animal.
I also believe that the author is somewhat behind the curve as far as the consolidation efforts in the UNIX world are
concerned. Whereas there used to be dozens of more or less incompatible UNIX'en out there (Tru64 anyone ?), today there are
basically Linux and BSD. And yes, there are the remnants of having to support all these platforms (most of which have joined
the choir invisible), but so what. If there's a line of shell code in configure that never gets executed any more, i can
imagine worse problems. And i *LOVE* the fact that i'm currently running more or less identical Linux installs more or less
hassle-free on Hardware as different as a PowerPC box, an x86 PC and an ARM-powered Netbook. I could surely run it just as
hassle-free on an IBM mainframe, but i haven't stacked one in my cellar just yet :-)
If that isn't progress, i don't know what is.
Poul-Henning Kamp | Wed, 22 Aug 2012 11:05:51 UTC
@sl: If I had been able to get hold of p9 when I asked for it back around 1990, I would not have spent two decades on FreeBSD.
@Klapauzius: I think there is plenty of time before progress goes too far :-)
Christophe de Dinechin | Wed, 22 Aug 2012 14:31:41 UTC
> One of Brooks's many excellent points is that quality happens only
> if somebody has the responsibility for it, and that "somebody" can
> be no more than one single personwith an exception for a dynamic duo.
I think that this argument is bogus. Quality is defined by usage, not by manufacturing. Today's cars don't rust anymore, and
it's not because a single person took charge, it's on the contrary because solid processes were put into place. The same holds
true for programming.
In the late 1980s, I wrote a 3D game that entered the Guinness Book of Records. Back then, me, myself and I, as a single
person, could deliver a world-class video game on my spare time as a student. It was considered "high quality" back then. And
it would seem to support the point that a single person can deliver quality.
Yet that game wouldn't pass any standard today. It was all of 17000 lines of code (puny by today's standard), and all in
assembly, very ugly by today's standards. It didn't use any well-defined 3D API, since there weren't any. Instead, it blasted
pixels on a fixed screen geometry using self-modifying code. Nobody would do that today. It computed approximations of 3D
transforms that used mostly additions because multiplications were expensive. Who would avoid multiplications today when doing
3D transforms? The name of "quality" has changed.
There's no way one of my kids, who are brighter than I am, can deliver a world class game all by themselves today. A single
texture in a modern video game contains more data than my entire game. More importantly, that humble texture refers to more
accumulated knowledge about color spaces, compression, lighting and dozens of other complex mathematical fields than was
necessary back in 1990. Who is responsible for the quality of the texture? The single guy who, all by himself, invented the
JPEG, TIFF and PNG standards? Or the dedicated artist who tries to implement in his texture the vision shared with an entire
team about the game?
We always need to anthropomorphize, to put a single face behind every great achievement. For many people, Linux is the
brainchild of Linus Torvalds, the iPhone is Steve Job's vision. That is just not true. Cathedrals were never built by a single
person, never dreamt of by a single person.
I'd actually go further. Quality needs at least two people: someone to build, and someone to use. And quality emerges when the
user is happy with what was built. Currently, more and more, cathedrals emerge only from the bazaar. The crafting process is
messy, but who cares. The result is solid, beautiful, interesting, usable, whatever.
Dan Cross | Wed, 22 Aug 2012 14:38:18 UTC
I think this article is decent, but the thesis is insufficiently stated and supported, and many readers are missing the point.
The point, as I understand it, is that programmers work in environments that lack good example s of design. This is secondary
to the development model used (beautiful things evolve in bazaars just as they do in cathedrals, despite Brooks's apparent
suggestion that an individual who mandates quality is essential at some level), and more indicative of a lack of relevant
experience, aesthetics, and a rejection of the responsibility to learn these things because it is not "the way it's always been
done." Except what we do now is not the way it's always been done, and too many people don't know enough about the history of
the profession to realize that.
The autotools vignette is an aside, but is drawing most of the attention in the response. Perhaps Kamp's point is that
autotools solved the wrong problem; treating a symptom instead of the underlying issue, like a bandaid that became a permanent
fixture. How could it have been done better? The answer to that specific question doesn't matter.
The real problem is that no one now is asking the question, "how can we do better?" (present tense as opposed to past). And no
one is asking that because they don't even realize that they should. The intellectual curiosity to ask isn't there. That's
the real problem, not whether libtool searches for FORTRAN compilers from systems that no one has used in 15 years, or whether
some command has options that haven't mattered in 20. If someone bothered to ask why and decided to clean it up, it would get
fixed. E.g., if someone decided to ask why invocations of 'ld' are so different across systems and get them cleaned up, it
might happen (though that's a much larger and harder problem).
This is the inevitable result when people stop questioning the basics and start taking them for granted. Other disciplines
stress proficiency in the fundamentals: professional basketball players practice basic shooting and dribbling. The computer
industry does not, in favor of 'graduating' to flashy whizz-bang fluff. The result is that people do not stop to say, "hey,
maybe there's a better way to do this?" and cruft accumulates for decades: why did it take nearly 25 years after the Morris
Internet worm to remove gets() from the C standard library?
On the other hand, there are many who have (and would continue) to argue that Unix is not the model of good design and elegance
that Kamp holds to be. For example, why does the command line interface on my Macintosh laptop (which has a high-resolution
hardware accelerated bitmapped display, multicore processor with a >2GHz clock, 4 gigabytes of RAM and integrated keyboard and
mouse) emulate a TTY communicating at 9600 BAUD? Why the applications continue to use a cursor addressing command set from the
1970s? Plan 9 did better, and even though it's been generally available as an example for most of the past decade, the
Unix/Linux folks don't want to evolve. Perhaps Kamp couldn't get a license in 1990; that doesn't excuse him for not learning
and applying its lessons and from 2003 on.
So that's the real problem: people are unwilling to question the most basic premises on which they build systems. It has
little to do with cathedrals or bazaars or myriad PHP-jockeys who can't tell the difference between a regular expression and an
arithmetic expression, it's about the unwillingness to question entrenched assumptions, and cut through the complexity to get
to the underlying simple, elegant solution at the most basic levels. Or maybe people do understand these things and don't care
because it's not worth the effort along dimensions they care about (but Kamp does not).
This article was a rant against autotools so much as a plea to start questioning whether things are as good as they can be.
Unfortunately, the short answer to that is almost always going to be, "No."
Irene Knapp | Wed, 22 Aug 2012 15:07:29 UTC
I agree with everything you say, and based on the number of negative comments I thought I should add my vote for cathedrals.
But there's no point in saying it; the bazaar-dwellers are too convinced they're right.
denisroyle | Wed, 22 Aug 2012 16:55:01 UTC
The computer paradigm affords the absolute precedence of the opcode over the operand. This leads to the unquestioned precedence
of source over data of delivery mechanism over content of industry over user. The binary abstraction at its core is a
profoundly irrelevant and divisive concept capable only of encouraging naval gazing and generating counter-productive noise.
Stop IT.
Different enough?
@Dan Cross:
Nice rant :-)
I disagree with you about the lack of curiosity: There is plenty of curiosity, which is why the lost generation throws
themselves at any fad that comes around, no matter how hare-brained it might be.
Many of these fads are reinventions of stuff that was invented long ago, and explored and reported on, but nobody in the lost
generation would ever dream about reading CS and IT research from before year 2000: They cannot even imagine what they might
learn from it.
When I recommended my readers read The Mythical Man-Month, I subsequently received several emails from people saying "You said
this was a good book, it's about *mainframes* for goodness sake, why should I read about mainframes ??!"
That attitude, is also represented in may comments to the current column, and to me indicates that a large number of young-ish
practitioners simply cannot fathom that there might be a higher level of abstraction at all.
In other words: They wouldn't recognize a cathedral if they saw one.
Poul-Henning
PS: With respect to your terminal window, that's an interesting case, and it's a very good question. My guess is that command
line interfaces simply have qualities, for instance economy of bandwidth, that allow their wrapping in antique simulations of
printing terminals to survive. I have yet to see a credible substitute or improvement, but I have seen several graphically
designed programs getting much improved with the addition of a command line, even if just in the form of ALT-F, ALT-S menu
shortcuts.
PPS: As for what I have and have not done inspired by p9, that's a long story, told in details in FreeBSD's version control
system.
Dan Cross | Wed, 22 Aug 2012 20:25:35 UTC
Poul-Henning:
The intent with my intellectual curiosity remark had to do with questioning basic assumptions (e.g., "we use autoconf because
we've always used autoconf"). This should not be taken to imply that people aren't curious about other things; my concern is
that the curiosity about the basics is no longer there ("do we need to use autoconf?").
Had I written your polemic, I would have avoided the cathedral metaphor: it's too overloaded, and I feel that your point has
become obscured in the ensuing rhetoric. If I understand you correctly, you mention cathedrals because, by and large, they are
beautiful. You are then lamenting the seeming inability of the 'lost generation' to recognize beauty in design and
implementation of software, as evidenced by the bloated layers upon layers of cruft that have become the norm (autoconf is just
a specific example). I would have used another metaphor to more clearly highlight the argument; say, contrasting between the
works of a dutch master or a child's finger painting. Or maybe between a Jackson Pollock and a child's finger painting. There
are superficial similarities, but one has sublime aspects of beauty that the other does not, and these take a certain about of
refinement to appreciate. It's that refinement that's missing in too many cases. Many would argue that similar refinement has
been missing from the Unix community almost since its inception: are they wrong? To some extent these things ARE subjective,
and calling one group a 'lost generation' without acknowledging that one may be lost oneself is a tad bit dishonest. Like the
Sphere from "flatland" that cannot fathom a 4th or higher dimension, some of the older generation struggle similarly; I know I
sure do.
Sure, things are reinvented and the reinventers aren't even aware of it because they haven't studied the literature well enough
to become acquainted with the full body of work in the relevant field, but the same is true of experienced practitioners as
well: e.g., your rediscovery of d-heaps. The seemingly recent rediscovery of DSLs by the design-patterns set. Lisp being
rediscovered again and again and again. Etc.
PS: I have nothing against command line interfaces. In fact, by and large I prefer them. What I object to is the Unix-centric
notion that command line interfaces must be tied to the implementation details of 1970s era hardware devices that have become
mostly irrelevant (I realize people still occasionally use serial terminals, but as a primary environment that has become as
rare as caring about the name of the FORTRAN-77 compiler on some obscure Unix variant). As far as credible substitutes for the
Unix pseudo-tty model: Plan 9 has already been mentioned and the command windows of rio and acme are perfectly usable (many
would argue superior) and don't emulate VT100s or have any notion of a BAUD rate.
Jussi Pakkanen | Wed, 22 Aug 2012 21:11:52 UTC
Autotools truly is terrible. For those who want more gory details, here's a blog post I wrote some time ago on the subject:
http://voices.canonical.com/jussi.pakkanen/2011/09/13/autotools/
@Dan Cross:
The main reason I chose the Cathedral metaphor, as bad and overloaded as it is, was because I think ESR's ideological manifesto
is a large part of the problem. "Bazaar won, f**k cathedrals" as one commenter summed up his view to me.
And I think "lost generation" is a both fair and apt description, because most of that generation have grown used to things
"being shit all the way down" as my friend Arthur is fond of saying, and cannot even imagine that it can be any better.
We are talking about people here who expect printer drivers to crash their kernel, who troubleshoots by pointing and clicking
on any icon which seems remotely related and who thinks that virus scanners and firewalls are where security comes from.
And yes, it's the IT-people I'm talking about here, not the average user/person, but the people who are employed in IT jobs.
A lot of us tend to forget this part of the IT world, because we have long since made choices in our lives that puts as much
distances between us and that IT world as possible.
"I don't do windows" as the T-shirt says.
But that world is still there, and the lost generation are still there, and they are in charge of IT at your dentist, you
plumber, your town-hall and your sewage treatment plant.
sl | Thu, 23 Aug 2012 00:59:00 UTC
@Poul-Henning:
It's never too late: http://code.google.com/p/plan9front/
denisroyle@gmail.com | Thu, 23 Aug 2012 02:38:32 UTC
Well there's a few things that we can say about the kids. They didn't create the environment in which they can tinker. They
didn't set the criteria for entry. They didn't set up the education system that is meant to train them. & They didn't allow the
standard to fragment.
The fact that a carbuncle can even be imagined let alone implemented demonstrates that the computer software engineer is no
engineer at all. If you viewed the heat emissions off every process rather than how terse and **obtuse you can get your C code
then you might have gathered the empirical data you require to protect your standard.
Even the French knew that long long ago.
Dan Cross | Thu, 23 Aug 2012 12:40:02 UTC
Poul-Henning:
I understand what you are getting at, and appreciate your point. But by presenting your argument in the language of the
Cathedral and the Bazaar, you create an implicit false dichotomy: that if cathedrals are beautiful, then bazaars must be ugly.
While that may often hold true (e.g. autoconf), it does not follow that it is always true. And I'm afraid you have constructed
a situation where the majority of discussion now focuses on this false dichotomy rather than the real issue: finding and
communicating examples of beauty in software, no matter what their source, in order to impart a sense of both quality and
aesthetics in newer programmers.
Did Eric Raymond's essay create a 'lost' generation of "IT professionals" who came into the industry to meet a huge demand for
bodies during the dot-com boom, and then never left? That's debatable, and I think it's more likely that Raymond gave many
people the sense that they had the right to contribute, even if they didn't have the skill, knowledge or sophistication to do
so. And I'm positive it motivated many of those to contribute, even if they'd been better off learning a bit more beforehand.
But I also remember the dot-com era pretty well, and my sense was that most of the 'lost' types you are referring to had never
heard of Eric S. Raymond, let alone the Cathedral and the Bazaar, that most wanted to work with resume-padding commercial
systems, not "open source" and that most were just there for the money. The IT types you refer to are much more closely
aligned with the Windows administrators who are struggling to retain their positions as Microsoft and organizations like it
(and the products they produce) slide further and further from relevance.
Regardless, if we assume for a moment that Raymond's missive was one of the catalysts in creating this lost generation, is
directly attacking his metaphor the most effective way to repair that damage? Certainly not, as evidenced by many of the
purely reactionary responses you see here. An axiom of military training is that frontal assaults rarely work; enfilade fire
through the flank is far more effective. Attacking the Cathedral and the Bazaar is a frontal assault.
But all of this is an aside. It has little to do with what is beautiful and how to educate the next generation about that.
My further point in my responses is that beauty is subjective. I don't mean to retreat into the intellectual dishonesty of
relativism (where one might start to question the difference between Jackson Pollock and a child finger-painting. That's
absurd. Unless the child is some kind of virtuoso, of course Pollock is the better artist), but holding up Unix as a
'cathedral' is, at best, subjective. Rob Pike, one of the guys who literally wrote the book on Unix, is often quoted as
saying, "Not only is Unix dead, it's starting to smell really bad." And that was in the early 1990s, well before both the dot-
com boom and the Cathedral and the Bazaar.
Yet there are still people telling us we should use pseudo-ttys and applications that use character addressing. Are these the
cathedrals we should aspire to? I rather hope not. In the early 1990s, there were still people pointing at Unix and Linux and
deriding them as toys compared to systems like VMS and mainframes. Much of the criticism was valid, even if those systems have
largely disappeared (note that that has less to do with their quality and more to do with tying themselves to vendors and
hardware platforms that have become increasingly irrelevant). I remember VMS systems programers complaining about the "lost
generation" of people trained on Unix, who didn't understand efficiency or the value of doing things like checking array
boundaries! Clearly, both are better than Windows, but despite that, there's never been any sort of golden age.
Poul-Henning Kamp | Thu, 23 Aug 2012 19:39:19 UTC
@Dan:
I think we are in violent agreement here, I certainly agree that there's never been a golden age (-- yet!), and that no silver
bullet is going to get us there.
And that is the crux of the matter: This column was inspired by the fact that the best book about the hard problem of software
design I have read in a long time, is totally out of reach for the majority of people who most need to understand it.
Vadim Goncharov | Thu, 23 Aug 2012 22:26:22 UTC
Good article, and enough is said in discussion (especially on curiosity about the basics). I agree, however, that it is not
directly Cathedral vs Bazaar (it's more effective not to attack this directly but better through flanks, though). It's rather
social. Now IT is just a model of all other socium. First decades were just as in scientific community, then it became just
like all other mankind, just a Bell curve. It is questionable, though, that IT could be better just like any other engineering
dicippline - then we should like to the point about hubris and "Patches are welcome"-response.
About autocrap: there is project for autoconf/automake replacement, http://sourceforge.net/projects/mk-configure/ - based on a
different idea than autotools, rather than test about every other feature (like fortran compilers), just let the platform (and
it's maintainers, you always have maintainers in distros these days) know what it supports. This leads to very compact and
clean code, written in BSD make. Unfortunatelly it's NetBSD's bmake, which prevent it's spreading to at least FreeBSD's ports
and then to being more popular. That's, BTW just another side of rant's main question and "Patches welcome"-response - author
is just not interested in having it's child being more popular. It's not enough to me curious how to do better, it's also (and
may be much more!) important to *push* better solutions to outside world. Or else mediocre solutions will dominate, just as in
living nature's struggle for existence.
P.S. If '...that quality happens only if somebody has the responsibility for it, and that "somebody" can be no more than one
single person-with an exception for a dynamic duo.' - then... what about FreeBSD core@ vs single Linus? ;)
Joseph | Thu, 23 Aug 2012 23:52:40 UTC
I think a lot of the blame for the sorry state of things falls on our universities. All the CS classes I took in college in the
early 00's either made us prove theorems or were auto-graded. This tends to foster a "if it works, it works" attitude.
Sometimes the instructor claimed a part of the grade depended on coding "style", but this tended to be trivial things like
having comments and descriptive variable names. The only software engineering course offered at my school taught OO with Java,
which I think was simultaneously too general (it doesn't teach you how to solve real problems) and too specific (only one
programming paradigm). So I feel like I never developed a systematic way to think about software design. I should mention that
this school is considered a very prestigious technical university.
Recently I've started to wonder why there aren't CS courses built around "reading the classics" e.g. the SysV source code. The
only exception I know of is John Lion's course back in the day for which the Lion book was written. Without a good foundation
of case studies to draw upon, is it any surprise that the prevalent problem-solving methodology of young programmers is to bang
their heads against the wall until something gives?
Ganesan Rajagopal | Fri, 24 Aug 2012 12:31:50 UTC
Excellent rant :-). The comments about autoconf/libtool mess are spot on. Every other week I have the urge to sit down and
rewrite libtool but don't have the energy to get into the cess pool. However, I think that the cathedral/bazaar metaphor is
misdirected - especially looking at the number of comments which equate bazaar to the open source/free software movement.
FreeBSD is open source but is typically thought of as the Cathedral model because historically very few people with commit
access.
I am sure many people would disagree but I consider Linux kernel also to be a cathedral in the sense that a single person or a
small group steering the overall design/architecture with rigorous code review and emphasis on quality. What defines a "bazaar"
is the lower barrier of entry with no private cliques.
Neither cathedral nor bazaar ensure quality. I find the rant a bit off track from that perspective. The festering mess that is
autoconf/libtool is not a problem of the bazaar model. libtool made sense when HPUX/AIX/Tru64/Solaris all build shared
libraries differently. It should just be thrown away today. But if you look at autoconf/libtool contributors over the years,
you'll find one or two active contributors - certainly not an example for the bazaar model of development. autoconf/libtool are
bad because they're outdated and need a replacement, period. It has absolutely nothing to do with Cathedral vs Bazaar.
Dan Cross | Fri, 24 Aug 2012 19:18:48 UTC
Poul-Henning:
Yes, violent agreement in good ol' USENET fashion indeed.
I agree with the crux of your argument, I just think it could be framed differently in a way that would make it more effective.
I hope you found my comments constructive.
Zygo | Sat, 25 Aug 2012 07:09:04 UTC
I think it is worth pointing out here that Firefox does, in fact, display TIFF images.
Dan Haffey | Sat, 25 Aug 2012 23:39:06 UTC
"and as an almost predictable application of the Peter Principle, rather than standardize Unix to eliminate the need for them,
somebody wrote a program, autoconf, to write the configure scripts"
Good rant, but a questionable invocation of the Peter Principle. It looks to me like regular old local reactions to unsolved
global problems. I'm as repulsed by autoconf as the next guy, but I wouldn't accuse its creators of incompetence merely because
they needed a solution and couldn't solve a big collective action problem like Unix standardization.
Soren | Mon, 27 Aug 2012 09:02:10 UTC
What's even more sad about the whole configure nonsense is that the amount of applications (or libraries or whatever is being
configured) that actually use the information provided in config.h is miniscule. So not only is configure checking for a bunch
of things that make no sense in modern day, but the output is almost invariably discarded.
Most applications simply use configure to check if a specific set of libraries are installed and to allow users to
enable/disable certain features. I'm fairly sure this could be accomplished with a fraction of the code and complexity of
autoconf.
Peter | Tue, 28 Aug 2012 02:49:26 UTC
In my opinion, the problem is not if a piece of software stems from a cathedral or a bazaar.
It is also not about if it is configured with autotools (some claim "autohell" ;) ) or manually.
Configuring only happens only once, running some seldom used tests won't kill anybody, if the results are correct. (But it is
indeed evil that no programmer can grasp autotools *easy* - tools should be as easy to grasp like a hammer or a drilling
machine )
It is more about applications becoming overfeatured complexity hell, complexity which is not needed in 99.x% of all cases -
like layers of wrapper libraries, using a complete library only for 1 or 2 functions, etc.
It is also about things like "company standard" and "strategic alliance" where you are unfortunately forced to use that huge
commercial database with the red trademark where a spreadsheet did work before (dept. leaders are ranked by burnt budget, you
know).
And it is about that cathedral project leaders as well as bazaar hackers are rewarded in money and fame by feature count and
LOC and not by carefully asking for the real needs and leaving out unnecessary stuff.
(sorry, if s/o calls the grammar police, I'm no native english speaker and it is late over here ;) )
Martin | Tue, 28 Aug 2012 22:32:10 UTC
Read the column and found myself nodding in agreement with some of the points that were being made. Also read some of the
comments. Then went on with my business but apparently my subconscious went on thinking.
Having considered the case, I guess my main issue with the rant is that there is an implicit appraisal; that the divine design
of the cathedral is superior to the wasteful redundancy and haggling of the bazaar.
But I think that is missing the point of the bazaar. It was never meant to be a cathedral. It serves different and far more
mundane purposes. Blaming the bazaar for not being a cathedral is akin to blaming a workhorse for not being a Lipizzaner horse.
Also, the wastefulness in the design of the bazaar is mirrored by the wastefulness of the cathedral. Figuratively speaking; the
thick walls, the oversize windows, the architectural obsession with symmetries, the amount of energy it takes to heat up the
place, the impracticality of the whole thing.
In the real world, there are few bazaars and there are even fewer cathedrals. Most buildings fall somewhere in the middle
between divinity and chaos. There are a lot of tradeoffs between usability, cost to manufacture, importance of maintenance,
etc. I believe the same to be true for software. As developers, we don't want to fall into the pitfall of creating
unmaintainable, fragile software, so we embrace modularity, test harnesses, separation of concerns, dependency injection, etc.
On the other hand there is always a tradeoff between effort and reward, and although we may be professionally inclined to build
divine masterpieces, we'll usually stop somewhere short of that. At a state where we can fix the worst inconsistencies and live
with the remaining ones.
Gregor | Fri, 31 Aug 2012 23:30:34 UTC
For modern day attempt to recreate a modern computing environment as a 20,000loc cathedral see
http://www.vpri.org/html/work/ifnct.htm
Worth checking out.
Charles | Wed, 05 Sep 2012 08:37:15 UTC
Thanks for the thought-provoking article. I find it regrettable that folks turn things into a battle of ideologies and OS's. I
don't think that was your point at all. But I can see why it can go that way. Still, the issues are more general than the
article's examples.
For example, serious contributors to TeX and friends tend to ask those in the know for the benefit and knowledge of their
experience. Love it or hate it, one tends to get both good mentoring and swift bugfixes. There's a certain community
cohesiveness and dedication. The output of the typesetter depends by nature on the elegance of design. Do people still see such
necessity with programming, or has economics clouded that?
By comparison, in spite of using paragraph/character styles, you can still get formatting anomalies in LibreOffice. One needs a
plugin (!) to convert local formatting to named styles. It seems that emulating Office has resulted in inheriting problematic
approaches from the design goals of Office. This is not the LO developers' fault; they inherited it. Can they hope to fix it?
A more fundamental and general question involves whether economics is selecting for the brutish and ugly over the elegant and
intelligent.
I have to ask myself why I experience more problems with the latest Ubuntu LTS release than the previous one. Wasn't Linux
supposed to free users from the frustrations of Windoze regression hell? So to get the broadest functionality, one tolerates
frustration in Linux and begins to question the general development of software, open or closed. And for those who say to dump
an OS if one doesn't like it, please. You are not going to dump your person-years of investment. Don't ask that of others.
Maybe the barbarians are the cost accountants and executives who grab managers by the short hairs when enough financial
gravitas is in play. I have seen the "enemy of done" argument used to justify questionable expectations for production
deadlines and quality decisions. Maybe the suits trashed the cathedrals because today's computers hide the necessary complexity
behind them. "When they're all supers, no one is super." Programmers and their craft have lost value as craftspeople and have
become mere laborers. Open or closed, financial pressure and "instant experts" work against good design, fixing inadequacies,
and correctness.
Erik Nordstroem | Fri, 07 Sep 2012 01:36:25 UTC
Having just finished reading the post and the comments (yes, all of them), I am left with a few questions.
Like RORVI, I have a hard time finding good, reliable information and answers.
I have read some books (I especially enjoyed many books published by O'Reilly), I am a bachelor student, and I have a part time
job where I have responsibility over a few servers running Solaris 10. On my spare time, I use Ubuntu (at the student group of
which I am a member), FreeBSD and Mac OS X.
For many of the day-to-day things I do, the information I need is available in the relevant manual pages and user guides, and
the assignments we get at the university are usually divided up in manageable "bite size" chunks such that they do not provide
significant trouble (as long as i start in time; I've been known to postpone assignments untill the very end and not having
enough time left to deliver when it is due).
When it comes to larger projects, however, I have no real experience. I have tried to start a few projects on my own, but have
rarely been able to get far with them (one exception was when an older and more experienced member of the student group I am in
gave me some mentoring).
One problem is that I don't know how much I should know.
The reason that I write this comment, is because I was wondering if you could tell me what material I should read so that I get
the foundation that I need to read and understand this book (The Design of Design).
Tom Barrett | Mon, 10 Sep 2012 17:55:45 UTC
Architecture is not an accident of nature, and neither is natural law. Disdain for one will bring about the downfall of the
other.
My own experience with open source has gone from, "This is cool!" (mid-90s) to, "This is...getting difficult to navigate,"
(early 2000s) to today's, "How the hell am I supposed to use this???"
clockskew | Fri, 14 Sep 2012 11:42:06 UTC
"Only open your mouth if it improves on silence."
Ralph Dratman | Fri, 14 Sep 2012 17:49:23 UTC
My belief is that we don't yet have computers powerful enough to put software development onto a rational footing. More
precisely, even if we do already have computers marginally powerful enough, we are not yet routinely configuring them with the
vast RAM spaces necessary to turn the existing base of open source code into a usable construction and optimization kit. The
newly-architected computers I am discussing will cost significantly more per seat than this year's standard common-denominator
system. Most hackers won't be able to afford them.
All this is necessary because the problems being discussed here are so large and intertwined that, as a practical matter, they
will only ever be fixed by a very smart computer program running on a very powerful computer. Here are some of the attributes
this (currently imaginary) program will have:
First, the reconstruction program will be able to extract many algorithms and translate them to run in a new environment. It
will be able to do this extraction from a combination of source code and fully linked running code.
Second, the program will be able to construct appropriately scaled emulation frameworks for many or most of the old
environments, in order to deal with situations where the first-mentioned method fails.
Third, and perhaps most importantly, this advanced program-manipulation program must be able to study the ensemble of existing
code statistically, as a series of random constructs, in order to follow the best-working existing implementations in the
course of rebuilding things.
The reconstruction program I describe is simultaneously simple in concept and stunningly complex in implementation detail. That
is why the program will have to incrementally write itself in an evolutionary fashion, and why it will use techniques from the
new bioinformatics world to manipulate all these program fragments created by all of us biological beings.
For that to work, I don't need to tell anyone reading this that we will need -- at the very least -- significantly more
powerful computing architectures and a hell of a lot of RAM space.
Rod Grimes | Mon, 24 Sep 2012 10:19:25 UTC
Your always full of such wonderfully flavored writing!!!
IGnatius T Foobar | Fri, 05 Oct 2012 12:28:54 UTC
I call bullshit. You're using FreeBSD as an example of excessive complexity and then declaring that all of Unix has this
problem.
I've got news for ya sunshine, there's this thing called Linux where it's easy. Applications install with one command and all
of the dependencies come on board automatically. It's just as easy as Windows and Mac.
Howard B. Golden | Sat, 03 Nov 2012 21:17:43 UTC
Mr. Kamp, what you wrote needed to be said. Thank you for saying it.
Along this line, I have been thinking about a "quality improvement institute, factory and service" that might help. Open source
is well established now in industry, and the problems are apparent to all. However, the benefits are seen as even greater. This
creates an opportunity to improve existing open source software and future development and maintenance processes. (Not
surprisingly, this same opportunity exists for closed source software.) Therefore, there should be a willingness on the part of
government and industry to fund quality improvement. (I don't attempt to specify whether such organization(s) are for-profit or
not-for-profit, but their products would need to be open for use without compensation for them to benefit open source
developers.)
There are several approaches that should be pursued: (1) Improved training and education for developers; (2) Identification and
promulgation of better or best practices (in choices of languages, tools and development processes); (3) Identification of
existing software most in need of remediation or replacement; (4) Funding remediation/replacement of the most important needs;
(5) Outreach to existing projects with offers to help adopt better practices; (6) Outreach to academia to encourage training in
better practices; (7) Advice for starting new projects following best practices appropriate for the project size; (8) Funding
development of tools to support better or best practices. (9) Review of existing tools and offering constructive suggestions
for improvement, perhaps with funding.
I would be happy to elaborate on these ideas with anyone interested.
Daniel Dumitriu | Tue, 06 Nov 2012 06:07:47 UTC
Yessss, for the "violent agreement" !
It's just horribly disheartening how many of the "contributors" to the comments didn't get it. Most rushed to "contribute"
without checking, first, who Don Knuth or Peter Naur, Per Brinch Hansen, Dijkstra, Fred Brooks and, even, Alan Turing,
are/were.
Nor did they figure out why they should care...
k | Wed, 07 Nov 2012 03:28:32 UTC
fortunately for you people, configure has been replaced: http://aiju.de/b/configure
Cellar | Fri, 09 Nov 2012 11:56:05 UTC
Apropos requiring perl AND python, last time I checked that was an artifact of the ports collection where an unexplained
digotomy exists between running sanity tests and building the damn thing, since if you'd care to look up the original project
page it only lists things like libiconv, and neither python nor perl. And lo and behold, it compiles fine if you kick both of
them out, and add the documented requirements.
Now, I'm not a fan in the least of the concept of glib or the people that wrote it, and there are many projects under the same
banner where such requirements specifications are only inferrable through repeated runs of configure and figuring out what the
crashes and error messages mean, but in this particular case it appears some of that unprofessionalism has seeped into the
ports collection's fabric itself. Which is... not good, for that particular package is pretty widely needed.
HELLYO protocol | Fri, 09 Nov 2012 13:29:20 UTC
@Erik Nordstroem
> When it comes to larger projects, however, I have no real experience.
Do you mean "larger projects that still can be handled by one person"? I would say the most important point is, when starting
out, make abstraction of coding. Think about what problem you want to solve, what elements and objects should be "there", how
they interact, what the architecture is, what the design is (go to the library and borrow Thimbleby's "Press On - Principles of
Interaction Programming"). Read up on the principles behind the infrastructure you will be building on (OS, VMs, database, web
interface). Study existing architecture and its philosophy (go to the library and borrow Eric Raymond's "Art of Unix
Programming") but not religiously, search for exiting solutions to connect to, study the APIs you intend to use. Select the
tools you will need, if necessary script your own. Learn good coding style (go to the library and borrow Steve McConnell's
"Code Complete"). Take a look at other programming languages (like Erlang), read a book on "Programming Patterns" and a book on
Object-Oriented Design, too (avoid UML if you can). Once your project connects many people, start reading about and applying an
Agile Method like SCRUM, that will keep things under control and in movement. Fumble onwards from there.
> One problem is that I don't know how much I should know. The reason that I write this comment, is because I was wondering if
you could tell me what material I should read so that I get the foundation that I need to read and understand this book (The
Design of Design).
Don't get hang up on that. Understanding comes with time and experience, and failure, not with reading. But start reading
anyway.
I am currently at the point that I know that I will never know what I should know. Once you start getting Excel sheets of
labeled and checkboxed requirements appearing on your desk while folders of commitee-designed standards block access to the
fire exit, it's time to put your affairs in order.
Justin Wiley | Fri, 09 Nov 2012 15:51:51 UTC
This is an old, pointless argument, and the fact the ACM has chosen to highlight it is another sign of their increasing
irrelevance. What Poul is really saying is: "remember the good old days? Weren't they better than now? Aren't we smarter
than our children?"
No, they were not, and no you are not. I remember cathedral systems. They used proprietary protocols. They were based on
proprietary hardware that couldn't evolve and couldn't be upgraded. And only the chosen few knew how they worked. ATT Unix
had proprietary manuals for gods sake...you really want to go back to that?
Progress in the 21rst century will not come from a giant ivory tower with the anointed few slowly building masterpieces.
There's too much demand, and too much complexity to tackle. A decent solution and a working system today will always beat a
perfect system 15 years from now. I think 21rst century developers and architects understand and embrace that reality.
If you can't Poul, please step aside...or by all means keep posting articles for the folks at the ACM retirement home to read.
sed gawk | Fri, 09 Nov 2012 21:28:50 UTC
Speaking as a younger *NIX coder,GNU/Linux (Suse) was my introduction to *NIX, I'd not seen BSD until a couple of years later,
so in my mind the coreutils was the GNU core utils.

This I think has something to do with why I use Linux over BSD as my workflow is available across all *NIX (gcc/vi/coreutils) -
perhaps the same for other people.
The autotools are a bit difficult to learn but I think they are worth the effort, M4 is a bit fugly but
with a bit of practise and ruthless factoring into small macros its not that bad, (Top tip add banner lines to your macros so
you can spot the output in the generated configure script)
With the autotools, I get cross-compilation/packaging/ cross-compiled unit-tests execute in a cross environment/transparent
replacement of missing functions at link time/standardised argument handling which generates help messages/binutil access and
ability to mung various assets (images/sql etc) in to my code with very little effort.
Mostly I copy my build-aux and m4 directories into a new project and write a simple configure. My heart sinks when I have to
work on project that doesn't use autotools.
So I think the autotools survived because when you take into account everything it provides, it's streets ahead of everything
else, so @David MacKenzie - I do not accept your apology and I thank you for making the choices you did.
Daniel Martinez | Sat, 10 Nov 2012 16:53:16 UTC
Even if the article made some good points it should also admit that the lost generation is a result of a something better. Just
to be practical let's atalk about possible solutions:
How can you stop poor designs in the bazaar model? Education? (i.e. universal education) it would be great but, how? On the
other hand, how can you prevent the uneducated and unexperienced programmers from contributing/reusing code in your project?
Copyright??? (please no). Better designs? If I understood "The Cathedral and the Bazaar" well, the bazaar model did not
override Brooks' work on design, it just enriched and complemented it. I think it is a bit unfair to blame the bazaar for
everything.
fishburger | Mon, 12 Nov 2012 11:06:21 UTC
You do realize that autoconf was written before the "lost generation" was out of their nappies? Autoconf also fails as example
of the lack of responsibility, we even had the person responsible showing up here to comment. As for an elegant solution to
that particular problem - how could anyone have been responsible for standardizing the plethora of Unixen when their very
existence spun on the premise of them being non-standard? And the Unix cataclysm itself: again brought to us by you the pre-
lost-generation dinosaurs.
Nobody has the incentive to optimize their software compilation times just so Mr. Unix Geek can build an entire software
universe in less than a day on his laptop. I think it's a freaking miracle how little repetition you found, when you consider
what's involved: a decentralized global software factory.
Short of it, you can only have responsibility if you also have control. You don't control the people and projects making the
software that you cram into your ports collection, so stop moaning.
warren | Tue, 13 Nov 2012 23:50:37 UTC
I learned from this article that CRUFT IS BAD. But nobody takes time out to remove CRUFT until it breaks.
Open source needs a Department of Cruft Removal. Because Raymond was right but so is Poul. Design is overrated (Sorry Poul)
but Architecture is 90% about Cruft Control (Point for Poul).
W
Farid Hajji | Sun, 02 Dec 2012 21:53:24 UTC
As a long time user of all kinds of Unices in the Eighties, and FreeBSD aficionado since FreeBSD 1.x up until now, I totally
agree with phk@. This said, there's a troubling point that didn't get enough attention yet, IMHO.
The FreeBSD ports collection, just like full Linux-like (F)OS(S) distributions, are actually big meta-programs full of internal
inter-dependencies and hidden couplings. That's the nature of the beast.
While we have figured out how to refactor individual programs under one's control, even big ones, we have yet to figure out how
to refactor such huge meta-programs like our FOSS ecosystem in order to clean them of their increasing number of meta-warts and
meta-bitrot.
Maybe, this is not a mere engineering task, but something bigger in need of fundamental methodologies. Methodologies we don't
know about yet because they were never needed before. I guess this is an active area of research, or it ought to be.
Ron Minnich | Wed, 05 Dec 2012 18:12:30 UTC
A good example is worth much discussion. I'm working to start a web site where we link to good code and good articles about
code. A link to this article would be the headline. Examples of high quality, portable code, written with good practices, not
afflicted with the current set of mistakes we see (autotools, libtool, and so on) would include:
- Go
- Plan 9 ports
- Inferno
to start. These systems all build on many OSes and there is not one single bit of configure scripts in the group. All have many
contributors and benefit from the bazaar, but are hardly chaotic. I'm sure we could find more. I've just gotten burned again
(by autoconf, of course) and I feel we really need to start educating people. The current state of the art in most open source
software is not very good.
Some of you know me; if interested, let me know, I could use the help. I'm not that good at setting up web pages :-)
Ranjit Singh | Fri, 21 Dec 2012 13:46:15 UTC
For people asking "where do we start?" I'd highly recommend: "The Unix Programming Environment" (Kernighan & Pike, 1984) --
even today it is highly relevant and will open your eyes
to the Unix philosophy simply by doing. Additionally you get to see what has been around since the
beginning, which is useful for code portability.
It also has the best yacc tutorial I've seen anywhere: I defy anyone to work through chapter 8 and not feel enlightened.
(Making it work with ANSI C is a useful exercise, and not at all difficult.)
The only caveat I'd mention is that he uses $* for shell parameters, where he should use "$@"; he mentions the latter after a
while, but doesn't use it later on where it would be better.
I've been coding for decades, and I wish I'd read it when it first came out, as it would have changed my life even more back
then. It's available quite cheaply second-hand. You'll need a basic understanding of C, and should really have done the
exercises in "The C Programming Language" (Kernighan & Ritchie 2nd ed) first. (Even if you haven't, buy both:)
The best modern reference is: "Advanced Programming in the Unix Environment" [2nd ed.] (Stevens & Rago, 2005) www.apuebook.com
but it's a much heavier read. If you trace Stevens' work back, the original "Unix Network Programming" (one volume, from 1990)
is in the same series as K&P(1984) and "The Design of the Unix Operating System" Bach (1986) both of which it refers to quite a
bit. The latter is an excellent companion piece to "Lions' Commentary on UNIX 6th edition with Source Code" John Lions (Peer-
to-Peer) which was mentioned above. Again, highly recommended, as are UNP volume 1: Sockets (3rd ed) and vol 2: IPC (2nd ed
afaik.) www.unpbook.com
Also, consider #friendly-coders on IRC chat.freenode.net if you want some helpful advice. They'll likely redirect you to
language-specific channels, but they're there if you need an explanation of what they've said, advice on approaches or
algorithms, or you just need some reassurance (especially given the prevalence of idiocy-elitism on IRC.) (Disclaimer: I work
with one of the founders.)
M. Simon | Wed, 26 Dec 2012 06:08:40 UTC
The problem is "C". But people are so used to it that they don't even see it.
E. Sarmas | Wed, 26 Dec 2012 12:52:51 UTC
I totally agree with above comment. The problem is in the use of "C".
We need at last a new start (maybe Java or a cleaner C with Java influences but certainly not C++).
M. Simon | Wed, 26 Dec 2012 22:11:51 UTC
Let me add that I'm currently working with a guy who managed 5,000 programmers at AT&T and who is a crack programmer himself
and he agrees with me. If all you have is a 75 tool pocket knife and what you really need is a hammer....
David Ormand | Wed, 02 Jan 2013 19:03:16 UTC
Good article, and controversial. The takeaway I'm getting is that the root problem is the lack of a "proper" education, even
from "accredited" public universities and tech schools, for CS/IT people. Well, I'm not CS/IT, I'm just a relic who graduated
from the TI-99/4A to doing embedded code (assembly, C, C++, Ada) in defense products (which for the most part do NOT have any
formal software quality requirements!), and I'm very aware of my lack of training. Given the shortcomings of regular school-
based education (especially for those with full-time employment) and the difficulty of getting mentoring from old-timers, any
suggestions for self-teaching? Like a preferred reading list; I see suggestions for McConnell's "Code Complete", and I've
heard of the "Mythical Man-Month" (and see its regular application - in the wrong way - in my company), and I've got the K&R
books and the Humphrey's "Personal Software Process" book (my company is a CMM level 5 outfit, so I grok process) and some
textbook for "Design Patterns" that is heavy on UML. What I think would be sweet is a website with recommended reading lists
for various software-related disciplines, such as system programming, network, crypto, embedded, etc.
Dan Cross | Thu, 14 Feb 2013 21:22:03 UTC
My list is here: http://pub.gajendra.net/2012/09/stone_age
Dave Wyland | Mon, 01 Apr 2013 18:55:21 UTC
Great discussion.
One responder proposed an interesting idea, that Moore's law and advancing computer hardware performance indirectly allowed
sloppy coding and software anarchy. An interesting idea. Moore's Law tended to improve performance by ~30%/year (CPU clock
rate), year-after-year, inexorably. Hard to adjust to on a continuing basis. At some point, a new programmer can just assume
that hardware performance is infinite, and that we should just go play in the new, digital ocean.
But that has quietly changed. Moore's Law went into the ICU in 2004. I had a 3.5 GHz Pentium 4 in 2004. There has never been a
4 GHz Pentium, to this day. Moore's law hit the scaling wall. Before 2004, each shrink meant cheaper silicon, lower power and
higher speed. After 2004 (~90 nm), each shrink meant cheaper silicon, *higher* power and *lower* speed. Intel and others
struggle with magic chemicals to make 2.5 GHz processors today. And no proven, near term solution to this problem has emerged.
I think this means we are in the clean up phase, where we get better results by cleaning up, reducing and streamlining code
rather than adding code. This will be interesting to watch, whether you subscribe to the cathedral or bazaar model.
ZenBowman | Wed, 08 May 2013 01:31:53 UTC
Great discussion, and it reminds me of an Alan Kay interview:
"""
Binstock: You once referred to computing as pop culture.
Kay: It is. Complete pop culture. I'm not against pop culture.... I consider jazz to be a developed part of high culture.
Anything that's been worked on and developed and you [can] go to the next couple levels.
Binstock: One thing about jazz aficionados is that they take deep pleasure in knowing the history of jazz.
Kay: Yes! Classical music is like that, too. But pop culture holds a disdain for history. Pop culture is all about identity and
feeling like you're participating. It has nothing to do with cooperation, the past or the future it's living in the present. I
think the same is true of most people who write code for money. They have no idea where [their culture came from] and the
Internet was done so well that most people think of it as a natural resource like the Pacific Ocean, rather than something that
was man-made. When was the last time a technology with a scale like that was so error-free? The Web, in comparison, is a joke.
The Web was done by amateurs.
"""
I'm a relative newbie, at the age of 30, but I found the vast majority of my CS curriculum (apart from a few courses on
compilers, operating systems and AI) very superficial. This past year I studied SICP and received more illumination from those
video lectures than I ever did in any coursework done during my PhD. I was lucky enough to work at a really good lab (USC/ISI)
during my PhD, and the one thing I'm very thankful for is that I met some genuine programming wizards, who imbibed in me an
appreciation for the history of our field.
Without an appreciation for our history, we will just continue to loop in cycles. The author is spot-on.
ksym | Thu, 26 Sep 2013 21:25:48 UTC
I am a FreeBSD user and I fully agree that FreeBSD ports is a chaotic mesh full of traps ready to blow up the sysadmins' faces.
Wanna upgrade your software today? Oh, please do read the UPDATING document very carefully to avoid failing builds AND to make
sure the ports do not screw up your system (like updating perl often does).
ksym | Thu, 26 Sep 2013 21:28:31 UTC
I could argue that the only free Unix-like system that has no problems with software maintenance is Debian with the Stable -
repository. But then again, Debian is the only Linux I have ever used since it is relatively simplistic at it's core, yet has a
working package manager.
2013 ACM, Inc. All Rights Reserved.

A Generation Lost in The Bazaar - ACM Queue

Uploaded by

Copyright:

Available Formats

A Generation Lost in The Bazaar - ACM Queue

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

A Generation Lost in The Bazaar - ACM Queue

Uploaded by

Copyright:

Available Formats

RELATED CONTENT

BARBARIANS AT THE GATEWAYS

You might also like

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.