A Conversation with

Bruce Lindsay

interview
Designing for
FAILURE MAY BE THE

I
f you were looking for an expert in designing database KEY TO SUCCESS. and Linux. Lindsay devel-
management systems, you couldn’t find many more oped the concept of data-
qualified than IBM Fellow Bruce Lindsay. He has been base extenders, which treat
involved in the architecture of RDBMS (relational data- multimedia data—images,
base management systems) practically since before there voice, and audio—as
were such systems. In 1978, fresh out of graduate school objects that are extensions of standard relational database
at the University of California at Berkeley with a Ph.D. and can be queried using standard SQL (Structured Query
in computer science, he joined IBM’s San Jose Research Language). Today he is still at work deep in the data man-
Laboratory, where researchers were then working on what agement lab at IBM’s Almaden Research Center, helping
would become the foundation for IBM’s SQL and DB2 to create the next generation in database management
database products. Lindsay has had a guiding hand in the products.
evolution of RDBMS ever since. Our interviewer this month is Steve Bourne, of Unix
In the late 1980s he helped define the DRDA (Distrib- “Bourne Shell” fame. He has spent 20 years in senior
uted Relational Database Architecture) protocol and later engineering management positions at Cisco Systems, Sun
was the principal architect of Starburst, an extensible Microsystems, Digital Equipment, and Silicon Graph-
database system that eventually became the query opti- ics, and is now chief technology officer at the venture
mizer and interpreter for IBM’s DB2 on Unix, Windows, capital partnership El Dorado Ventures in Menlo Park,
PHOTOGRAPHY BY TOM UPTON

22 November 2004 QUEUE rants: feedback@acmqueue.com

 interview

California. Earlier in his career he spent nine years at Bell SB What are your thoughts on what’s an error and what
Laboratories as a member of the Seventh Edition Unix isn’t?
team. While there, he designed the Unix Command BL An error is whenever any component being used does
Language (“Bourne Shell”), which is used for scripting not deliver the result it is expected to deliver, whether it
in the Unix programming environment, and he wrote be a memory reference that comes up with a parity prob-
the ADB debugger tool. Bourne graduated with a degree lem in the machine, a disk I/O in which no result comes
in mathematics from King’s College, London, and has a back, or a subroutine call that throws an exception. “You
Ph.D. in mathematics from Trinity College in Cambridge, asked me to do X, I didn’t do it.”
England. A good design principle is: either do what you’re told
to do or tell us you didn’t do it and why, but don’t do
STEVE BOURNE Why don’t we start off with the thought something completely different.
that you can’t recover from an error until you’ve detected SB I guess the other choice is, “I’m not going to do that,
the error. so find out if there’s some other route to get the answer to
BRUCE LINDSAY Let’s think a little bit about how errors the question you were looking for or to get the function
happen—and they happen at all the different levels of the complete in the way that you were asked to do it.” This is
system, from an alpha particle discharging a capacitor in the multiple-case situation.
your memory to a fire, flood, or insurrection wiping out BL Well, certainly that comes up all the time. The func-
the entire site. From program logic blunders to the disk tion is enroll a UserID and we find out when we try to
coming back with data from the wrong sector, things go enroll it that it’s a duplicate. The next level up, above
wrong. You have to engineer for failure at all the different the level of enroll this UserID interface, is going to have
levels of the system, from the circuit level on up through to think of what to do. It can say, “Sorry, we allow only
subsystems like the database or the file system and on one Bruce in our system and you can’t be in our system.
into the application programs themselves. Please go away forever,” or “Would you like to change
“Engineering for failure” sounds like a bad phrase, your name to Steve?”
but that’s really what’s required to deliver reliable and
dependable information processing.
SB It’s certainly true that one of the mind-sets you have SB When you’re designing a system—as opposed to a sim-
to have when you’re writing code and designing systems ple application—where the system is quite complicated,
is: What’s going to break? There’s a broad range of pos- may be quite large, and involves some issues of scale, how
sibilities for approaching this, depending on the type do you design for failure?
of application or software. If you’re writing a Microsoft BL Certainly, you take the medical profession’s position
Word–type program, the way you approach this might be of do-no-harm in case of failure. That is, don’t blunder
different from if you’re designing a heart monitor. forward if you’re not sure what you’re doing. Another
BL In the heart monitor case, you better keep the heart way of saying this is fail fast, and I’m speaking here of
going, whereas in the Microsoft Word case, you can just systems that maintain persistent state for the most part.
give them a blue screen and everybody is used to that. It can be very dangerous to try to go forward if you don’t
SB But also in the heart monitor case, it’s hard to ask really understand what has gone wrong. So the idea of fail
users if they want to keep the heart going because the fast and pick up the pieces from the redundancy that you
answer is pretty obvious, whereas in the Word case, you have carefully maintained has been pretty successful.
can ask the user in some cases what to do about it. From my own area of database systems, you can talk a
BL You can sometimes ask the user, although it is better lot about the beauty of the data models and their power
to ask the subsystem what it is going to do to get itself as and abilities to get the work done and to make it easy for
healthy as it can, or abandon as the case may be, depend- you to write your applications and store vast amounts of
ing on its analysis of the situation. data and all that; but bottom line, the success of these
SB Are you really thinking of system failures as opposed systems is due to their reliability. Until the reliability of
to user errors? the system to maintain the data, not lose it, was good
BL I don’t think of user errors, such as improper input enough to gain the confidence of users, nobody was
kinds of things, as “failures.” Those are normal occur- going to put critical data into these systems.
rences. I don’t think of a compiler saying you misspelled The approach used in these systems is basically to
goto as really being an error. That’s expected. maintain two copies of the books. One we call the

24 November 2004 QUEUE rants: feedback@acmqueue.com

recovery log, which is the true copy of the books. That’s checking the consistency of our state to see if we should
everything that happened in the order it happened. Then go into error handling. Timeout is not exactly a redun-
there’s a representation of the database optimized for dancy but that’s another way of detecting errors.
access and update of the “current.” The key point here is that if you go to sea with only
The ability of database systems to preserve the latest one clock, you can’t tell whether it’s telling you the right
committed state of the data despite the failure of the time. You need to have some way to check. For example,
processor, the storage, and the communications, coupled if you read a message from a network, you might want to
with the important transaction, atomicity, and durabil- check the header to see if it is really a message that you
ity principles, made these systems reliable enough that were expecting—that is, look for some bits and some posi-
people trusted them to keep critical information, without tion that says, “Aha, this seems like I should go further.”
which they would be pretty much lost and confused. Or if you read from the disk, you might want to check
a label on the disk block to see if it was the block you
thought you were asking for. It’s always some kind of
SB One thing we could explore here is what techniques redundancy in the state that allows you to detect the
we have in our toolkit for error detection. occurrence of an error. If you hadn’t thought about fail-
BL Fault handling always begins with the detection of ures, why would you put the address of a disk block into
the fault—most often by use of some kind of redundancy the disk block?
in the system, whether it be parity, sanity checks in the SB So, really what you’re trying to do is establish confi-
code where we may spend 10 percent of the lines of code dence in your belief about what’s going on in the system?

more queue: www.acmqueue.com QUEUE November 2004 25

 interview

BL In a large sense, that’s right. And to validate, as you exception in the language is something the logic of the
go, that the data or state upon which you’re going to program does.
operate is self-consistent. Most of the scripting languages, for example, have
very little support at the language and semantic level for
dealing with exceptions. And at the end of the day, most
SB What language support do we have for error detec- of what’s in the languages is stuff that you could have
tion? coded yourself.
BL In fact, there is zero language support for detection. There are some fairly dangerous features in lan-
What we see in the languages are facilities for dealing guages—in particular, the raise error or throw exception
with the error once it has been discovered. Throwing an and the handlers. How does that relate to the stack of

26 November 2004 QUEUE rants: feedback@acmqueue.com

 interview

procedure calls? What we see in some early approaches case, I’m faced with a detected error. The first thing to do
to language-supported error handling is that the stack is is fold your tent—that is, put the state back so that the
peeled back without doing anything until you find some state that you manage is coherent. Then you report to the
level in the stack that has declared that it’s interested in guy who called you, possibly making some dumps along
handling the particular exception that’s in the error at the the way, or you can attempt alternate logic to circumvent
moment. the exception.
In general, folding a procedure or subroutine acti- In our database projects, what typically happens is it
vation—method activation—without cleaning up the gets reported up, up, up the chain until you get to some
mess that may have already been made, the partially very high level that then says, “Oh, I see this as one of
completed state transformation of that function, is very those really bad ones. I’m going to initiate the massive
dangerous. If there have been memory allocations, for dumping now.” When you report an error, you should
example, and you just peel back the stack entry, those classify it. You should give it a name. If you’re a compo-
memory allocations are likely not to be undone. nent that reports errors, there should be an exhaustive list
So it’s very important that at every level of the proce- of the errors that you would report.
dure activation, those procedures be given a chance to That’s one of the real problems in today’s program-
fold up their tent neatly, even if they can’t deal with the ming language architecture for exception handling. Each
exception. component should list the exceptions that were raised:
SB It certainly looks like memory allocation is an area typically if I call you and you say that you can raise A, B,
that people need to think about. I don’t know if there are and C, but you can call Joe who can raise D, E, and F, and
any good rules or not. Have you got any other thoughts you ignore D, E, and F, then I’m suddenly faced with D,
on that? E, and F at my level and there’s nothing in your interface
BL Well, garbage collection, of course, covers a multitude that said D, E, and F errors were things you caused. That
of sins in this area. The automatic invocation of destruc- seems to be ubiquitous in the programming and the lan-
tors for objects in the stack frame of C++ is also helpful. guage facilities. You are never required to say these are all
While garbage collection creates some serious perfor- the errors that might escape from a call to me. And that’s
mance problems, especially in multithreaded applica- because you’re allowed to ignore errors. I’ve sometimes
tions, it does tend to eliminate storage leaks, which are advocated that, no, you’re not allowed to ignore any
among the hardest of bugs to isolate. error. You can reclassify an error and report it back up, but
SB It seems that some of the more modern languages you’ve got to get it in the loop.
at least address the issue of error recovery—Python, for SB The system doesn’t know exactly the right informa-
example, has some facilities in it. It seems to me the good tion to report, but you don’t necessarily want to be in a
news is that people are starting to think about it and put- situation where you have to wade through 20 megabytes
ting these things in languages. I don’t know if they are of stuff to find something.
getting it right or not. BL In the case of database systems, we seem to be moving
BL It’s good that people are starting to think about it, if more and more toward conservative dumping—that is,
only for the reason that there’s this new language facility dump anything that might be of interest. The reason is
that makes them think about error recovery. There’s noth- that when you have an application in the field, and when
ing magic in the language that’s going to save you here. you’re dealing with things like Heisenbugs that don’t
You still have to think about the reasons that the things reproduce easily, or concurrency bugs, it is quite often dif-
you call might fail to deliver and what you are going to ficult to get the user to make a special run when you say,
do about it if they don’t deliver. “Hey, I want you to crash your system again while I have
my special monitors on to see what happens.”
They will usually say, “Huh? It’s running now. It’s been
SB Once you’ve detected the error, now what? You can running for six hours. Yes, this problem happens once
report it, but the question is who do you report it back to a week, and we’re quite mad that it’s happening. Fix it,
and what do you report back? damn it. But we’re not going to go crash the system just
BL There are two classes of detection. One is that I looked for you.”
at my own guts and they didn’t look right, and so I say So, we’re dumping more and more information. Now,
this is an error situation. The other is I called some other we don’t dump the database itself, but if we find a page
component that failed to perform as requested. In either that is suspicious, we’ll certainly dump all the contents

28 November 2004 QUEUE rants: feedback@acmqueue.com

 interview

of that page—almost bitwise—for possible analysis. cate reservations, because it turns out it doesn’t matter if
Dumping too much is probably not a bad thing in this somebody has two reservations in the system. Eventually,
area if you have to service code in the field and you’re somebody will figure it out and do something about it.
not allowed to say, “Hey, please crash the system for me That’s not a black-and-white system. It’s one that has a
again.” fuzzy edge to it and is willing to tolerate some inconsis-
SB When things do go wrong and you decide to dump tencies because they have other ways of recovering from
out everything you can think of, the question is, “Gee, the inconsistency.
wouldn’t it be nice to know what was happening in this BL At all levels of the system, we are finding a flirtation of
place at the time?” as opposed to just getting a stack degraded operation, from the memory systems in which
dump and trying to figure it all out from those photo- a bad memory chip will be simply removed from the
graphs. It’s almost like you need the movie rather than memory and no longer used—hopefully after recovering
the photograph. its state, using ECC redundancy—on up to whole system-
BL Well, the movie of course comes when you turn on level things where in case of the failure of the system, you
tracing, but then again, that requires special ability and may restart it on alternative hardware that may already be
usually is done back in the development shop. Often, doing part of the load. So the degradation is in perfor-
that’s the first thing that the developers are going to try mance at this point.
if it’s a reproducible error and they can look at all the You overload one system because another has failed
dumps. They’re going to reproduce it with the micro- and you are willing to tolerate the reduced performance
scope turned on the system and look at the tracing until the first system can be repaired.
dumps—which does give you the movie. Similarly, in redundant array storage systems, during
SB So the real question is, when you’re designing the reconstruction of a failed disk, we find reduced perfor-
code for failure, do you think of things that you should mance for accesses to the data that was once stored on
be recording that would help you in these situations? that disk because the value has to be reconstructed from
BL It’s not unusual that when the root cause is dis- the value of the alternative array blocks.
covered, the dump routines will be modified to dump So graceful degradation is showing up at all levels of
additional information that would have been useful to the system.
enhance the discovery of the problem. One has to be very careful about degradations in
which the result returned by the system may be compro-
mised in some way. Consider the airline control applica-
SB It would be interesting to talk about file systems and tion in which if you lose a reservation, the guy will tell
registries, and just get your views on how far we’ve come you about it and you’ll figure it out later. That requires a
from the old days when we had fsck (file system check) to real gut-check at the system design level to say, “Is this
fix up the file system when it fell over. acceptable for the application, for the deployment of this
BL I think it has been quite a revolution in file systems system?” Certainly we’re seeing this in the Web environ-
that they have moved away from scanning or fsck to ment. If you completely blow off one in 100 requests for
recover from emergency restarts toward a database- your Web, you’re fine. The guy is just going to refresh and
motivated/inspired logging system that logs the meta- it will work the next time. If you blow off 100 percent of
data changes and then lazily in the background writes the the requests for even a few minutes, however, it’s in the
changes to the disk. New York Times.
This has two beneficial effects. One is typically that the So we see that the decision on how to respond to
log can be written to a high-performance device, whereas errors—whether to give no answer or even a partial
the meta-data changes represented by log changes can be answer—really depends on what the answer is going to be
lazily written to their home locations on disk at a later used for and what the reaction of your client or customer
time and not necessarily inline with the user request. It is going to be to a wrong answer or lack of answer.
has also made emergency restart of the file systems much
faster because only the most recent changes need to be
redone or double-checked. SB What about the relationship between system debug-
SB Let’s consider an example of system design for graceful ging and detection and recovery?
degradation. In the Sabre system for airline reservations, BL There are two broad classes of error messages: one
they’re willing to have something like one in 1,000 dupli- is that the system had anticipated this problem, like a

30 November 2004 QUEUE rants: feedback@acmqueue.com

duplicate username or parameter-out-of-range, and is SB When you produce an error message, you have to
now trying to tell you what problem it observed. Most know who it’s for and what they know about what’s
systems are particularly bad at this. The error feedback going on. To what extent do people normally test the
you get is cryptic at best and deceptive at worst. This error messages with the user community? It seems to me
stems from a number of problems. One is that in many that feedback would be quite constructive.
software development environments, there’s a whole BL Oftentimes the text of the error message is phrased
little subsystem to deal with error messages. To create a in terms of the developer who’s down in the guts of the
new error message, you’ve got a half-hour of typing to fill system and doesn’t clearly relate to what the end user was
in the error message, and you have to make sure it gets to doing at that time, or like I say, it’s not specific enough.
National Language Translation and give it an error code Now that’s the first class of error messages—that is,
that’s not already allocated to some other error message. there is some fairly clear relationship between the user’s
To avoid this, sometimes people reuse old error messages, input and the error. The other class of error messages is
saying, “This is sort of like that other error. It’s already in when the system realizes it screwed up. In this case, the
the system. I can just use this.” error messages are really not intended for the end user at
Laziness in programming leads to lack of specificity in all and often don’t tell the end user anything. They say,
error messages. You might say, “You know, I have a choice “We couldn’t do the statement. Sorry, go away. Try a dif-
between an unparameterized message that sort of says ferent statement.”
what’s happening, and a much more complicated param- Basically, it’s an admission by the system that there’s
eterized message for which I’d have to write 15 lines of a bug, and at this point, of course, to service that fault—
code.” Guess what I’m going to choose? whether it be a software or a hardware fault—a lot more
I think that, in general, software developers—and information is needed. Dumping all that on the poor end
hardware people aren’t a heckuvalot better—are not user is not helpful.
investing enough money in correctly stating the errors So most major systems—hardware and software—
and being more precise in describing the error. maintain error-logging data sets, and when these kinds of

more queue: www.acmqueue.com QUEUE November 2004 31

 interview

errors happen, the system ends up dumping quite a lot of under normal messaging circumstances—or in extreme
information into the error-log data sets, saying, “Here’s messaging circumstances—to tell whether another system
the situation. I found this was bad and here’s all the stuff has failed or is just slow or you have failed. That is, your
surrounding this stuff that might have caused it to be bad communication outlink has failed.
and here’s what I was doing when it went bad. Hope you In practical terms, it is possible to write and imple-
can figure out what happened.” ment these algorithms. There are good ones around. They
are a bit expensive to run, but they are used to maintain
the list of who’s alive and keep everybody who is alive
SB What are Heisenbugs? aware of who else is alive, because in general these sys-
BL Heisenbug as originally defined—and I was there tems have to collaborate among all the alive members in
when it happened—are bugs in which clearly the behav- maintaining the distributed state.
ior of the system is incorrect, and when you try to look to SB One of the interesting aspects of this is the trade-off
see why it’s incorrect, the problem goes away. Typically, between how long it takes to detect something and how
when you are trying to see what’s incorrect, you turn on much time you really have to recover in the system.
tracing or you add some more parameters or you change BL And what it means to remove the failed component,
something. And the change causes the problem to go because there is a split brain problem that I think you’re
away. Quite often these problems occur because of con- out and you think I’m out. Who’s in charge?
current executions. Or they occur because of the way the SB Right, and while they’re arguing about it, nothing is
memory happened to get laid out in a particular case. happening.
So the real definition of a Heisenbug is when you look, BL That’s also possible, although some of these systems
it goes away—in deference to Dr. [Werner] Heisenberg, can continue service. It’s rare that a distributed system
who said, “The more closely you look at one thing, the needs the participation of all active members to perform a
less closely can you see something else.” single action.
SB I guess the good news when that happens is that you There is also the issue of dealing out the failed mem-
know you’re in real trouble. Because it usually means bers. If there are five of us and four of us think that you’re
there’s a lower layer of the system—such as memory allo- dead, the next thing to do is make sure you’re dead by
cation or something like that—that’s screwed up? putting three more bullets in you.
BL They’re very hard to find, but, of course, the hard ones We see that particularly in the area of shared man-
are the fun ones. agement of disks, where you have two processors, two
systems, connected to the same set of storage. The prob-
lem is that if one system is going to take over the storage
SB How do we deal with error detection and recovery for the other system, then the first system better not be
in distributed systems, which by their nature tend to be using the storage anymore. We actually find architectural
highly parallel, with a bunch of loosely coupled parts? facilities in the storage subsystems for freezing out partici-
BL We can define distributed systems into two broad cat- pants—so-called fencing facilities.
egories. One category is where one system uses another So if I think you’re dead and I want to take over your
system for some function. That is a dependent function. use and responsibility for the storage, the first thing I
Web services are an example of that kind of dependent want to do is tell the storage, “Pay no attention to him
use. The other category is where distributed machines anymore. I’m in charge now.” If you were to continue to
are cooperating on some single service. Distributed file use the storage while I blithely go forward and think I’m
systems are an example of that. the one who’s in charge of it, terrible things can happen.
The second one is actually where people put most of There are two aspects of collaboration in distributed
their thought into error recovery and reliability, because systems. One is figure out who’s playing, and the second
the idea in such distributed systems is quite often that one is, if someone now is considered not playing, make
any server can do it—you just have to find one that’s up damn sure somehow that they’re not playing.
and running and is part of the game. But because the
distributed state is then maintained by these servers,
there is an enormous problem of figuring out who’s in the SB One thing that makes error detection and recovery
game. These are the group consensus algorithms. There’s hard is multithreaded applications. Should we just not
an important theorem that says that it is impossible write multithreaded systems?

32 November 2004 QUEUE rants: feedback@acmqueue.com

 BL I think we need to write multithreaded systems, but it SB How is that represented in the code?
is difficult. I used to say to people, “If you haven’t written BL Usually by mechanisms in the thread management
multiprocessing code for five years, you’re not qualified that you end up having to write for any multithreaded
to write multiprocessing code.” And then I realized that application. A thread can die, and then the thread man-
nobody is going to replace me if I keep up this attitude. ager will say, “Oh, that was one of the kinds of threads
So I have been working to help people understand how that wants to restart if it dies.”
to write multiprocessing code. There’s a whole raft of For example, in a database system, you may get a
techniques, but basically synchronization of critical sec- statement and the system may freak out and say, “Oh, I
tions and lots of checking are necessary. The advantages can’t figure out the syntax here.” That’s a user error, and
of multiprocessing code and using shared memory data we would be happy to tell the user, but the thread that
structures are so compelling that we have to do this. did that is happy to accept another statement or even the
SB I guess the state of the art in detection and recovery same statement again and try to process it.
hasn’t moved a lot in the last 20 years. At the other end of the spectrum, the log disk has
BL One of the things we’re getting a little bit better at is failed and we can’t write log records. You better stop the
what we call the scope of the error. Did it mess up just whole show at that point. You have to be able to give up
this one function? You called this function, it didn’t do and restart and provide the service on alternative hard-
it. It says, “I fail, return.” At that point the scope is that ware. We have to deal with site disasters, right? That’s
function. Of course, if you return, if you throw an excep- real. You have to think that through from the beginning
tion to the code, now the scope has gotten bigger. You in your application and subsystem design. Q
have to decide if the scope of this error is just this thread.
If the error occurred outside of any shared data manipula- LOVE IT, HATE IT? LET US KNOW
tions, you can say, “Maybe if we just killed this thread, feedback@acmqueue.com or www.acmqueue.com/forums
the other threads could keep going, and maybe we can
even restart this thread and it will be happier next time.” © 2004 ACM 1542-7730/04/1100 $5.00
���������������������� ����� ��������������������� �����������������������������������

��������������������������������������������������������������������
�������� �������������������������������������������������������

��������
�
������������������������������������������
�����������������������������������������
�����������������������������������
�����������������������
� ���������������������������������������������������������������������
� �������������������������������������������������������������
���� ��������������������������
� ������������������������������������������������������������������
���������������������������������������������������������������������

�������������������
� ������������������������������������������������������
����������������
� �����������������������������������������������������������
� ����������������� ����������������������������������������
����������
� ������������������������������������ ��������������������
� ��������������������������������������������������������
� �����
� ���������������������������� �������������������������
�����������

��������������������������
���������������� � ��������������������������������������������
���������������������
������������������ � ���������������������������� ��������������� ����������

more queue: www.acmqueue.com QUEUE November 2004 33