19. Beyond Exception Handling: Conditions and Restarts

One of Lisp's great features is its condition system. It serves a similar purpose to the exception handling systems in Java, Python, and C++ but is more flexible. In fact, its flexibility extends beyond error handling--conditions are more general than exceptions in that a condition can represent any occurrence during a program's execution that may be of interest to code at different levels on the call stack. For example, in the section "Other Uses for Conditions," you'll see that conditions can be used to emit warnings without disrupting execution of the code that emits the warning while allowing code higher on the call stack to control whether the warning message is printed. For the time being, however, I'll focus on error handling.

The condition system is more flexible than exception systems because instead of providing a two-part division between the code that signals an error1 and the code that handles it,2 the condition system splits the responsibilities into three parts--signaling a condition, handling it, and restarting. In this chapter, I'll describe how you could use conditions in part of a hypothetical application for analyzing log files. You'll see how you could use the condition system to allow a low-level function to detect a problem while parsing a log file and signal an error, to allow mid-level code to provide several possible ways of recovering from such an error, and to allow code at the highest level of the application to define a policy for choosing which recovery strategy to use.

To start, I'll introduce some terminology: errors, as I'll use the term, are the consequences of Murphy's law. If something can go wrong, it will: a file that your program needs to read will be missing, a disk that you need to write to will be full, the server you're talking to will crash, or the network will go down. If any of these things happen, it may stop a piece of code from doing what you want. But there's no bug; there's no place in the code that you can fix to make the nonexistent file exist or the disk not be full. However, if the rest of the program is depending on the actions that were going to be taken, then you'd better deal with the error somehow or you will have introduced a bug. So, errors aren't caused by bugs, but neglecting to handle an error is almost certainly a bug.

So, what does it mean to handle an error? In a well-written program, each function is a black box hiding its inner workings. Programs are then built out of layers of functions: high-level functions are built on top of the lower-level functions, and so on. This hierarchy of functionality manifests itself at runtime in the form of the call stack: if high calls medium, which calls low, when the flow of control is in low, it's also still in medium and high, that is, they're still on the call stack.

Because each function is a black box, function boundaries are an excellent place to deal with errors. Each function--low, for example--has a job to do. Its direct caller--medium in this case--is counting on it to do its job. However, an error that prevents it from doing its job puts all its callers at risk: medium called low because it needs the work done that low does; if that work doesn't get done, medium is in trouble. But this means that medium's caller, high, is also in trouble--and so on up the call stack to the very top of the program. On the other hand, because each function is a black box, if any of the functions in the call stack can somehow do their job despite underlying errors, then none of the functions above it needs to know there was a problem--all those functions care about is that the function they called somehow did the work expected of it.

In most languages, errors are handled by returning from a failing function and giving the caller the choice of either recovering or failing itself. Some languages use the normal function return mechanism, while languages with exceptions return control by throwing or raising an exception. Exceptions are a vast improvement over using normal function returns, but both schemes suffer from a common flaw: while searching for a function that can recover, the stack unwinds, which means code that might recover has to do so without the context of what the lower-level code was trying to do when the error actually occurred.

Consider the hypothetical call chain of high, medium, low. If low fails and medium can't recover, the ball is in high's court. For high to handle the error, it must either do its job without any help from medium or somehow change things so calling medium will work and call it again. The first option is theoretically clean but implies a lot of extra code--a whole extra implementation of whatever it was medium was supposed to do. And the further the stack unwinds, the more work that needs to be redone. The second option--patching things up and retrying--is tricky; for high to be able to change the state of the world so a second call into medium won't end up causing an error in low, it'd need an unseemly knowledge of the inner workings of both medium and low, contrary to the notion that each function is a black box.

The Lisp Way

Common Lisp's error handling system gives you a way out of this conundrum by letting you separate the code that actually recovers from an error from the code that decides how to recover. Thus, you can put recovery code in low-level functions without committing to actually using any particular recovery strategy, leaving that decision to code in high-level functions.

To get a sense of how this works, let's suppose you're writing an application that reads some sort of textual log file, such as a Web server's log. Somewhere in your application you'll have a function to parse the individual log entries. Let's assume you'll write a function, parse-log-entry, that will be passed a string containing the text of a single log entry and that is supposed to return a log-entry object representing the entry. This function will be called from a function, parse-log-file, that reads a complete log file and returns a list of objects representing all the entries in the file.

To keep things simple, the parse-log-entry function will not be required to parse incorrectly formatted entries. It will, however, be able to detect when its input is malformed. But what should it do when it detects bad input? In C you'd return a special value to indicate there was a problem. In Java or Python you'd throw or raise an exception. In Common Lisp, you signal a condition.

Conditions

A condition is an object whose class indicates the general nature of the condition and whose instance data carries information about the details of the particular circumstances that lead to the condition being signaled.3 In this hypothetical log analysis program, you might define a condition class, malformed-log-entry-error, that parse-log-entry will signal if it's given data it can't parse.

Condition classes are defined with the DEFINE-CONDITION macro, which works essentially the same as DEFCLASS except that the default superclass of classes defined with DEFINE-CONDITION is CONDITION rather than STANDARD-OBJECT. Slots are specified in the same way, and condition classes can singly and multiply inherit from other classes that descend from CONDITION. But for historical reasons, condition classes aren't required to be instances of STANDARD-OBJECT, so some of the functions you use with DEFCLASSed classes aren't required to work with conditions. In particular, a condition's slots can't be accessed using SLOT-VALUE; you must specify either a :reader option or an :accessor option for any slot whose value you intend to use. Likewise, new condition objects are created with MAKE-CONDITION rather than MAKE-INSTANCE. MAKE-CONDITION initializes the slots of the new condition based on the :initargs it's passed, but there's no way to further customize a condition's initialization, equivalent to INITIALIZE-INSTANCE.4

When using the condition system for error handling, you should define your conditions as subclasses of ERROR, a subclass of CONDITION. Thus, you might define malformed-log-entry-error, with a slot to hold the argument that was passed to parse-log-entry, like this:

(define-condition malformed-log-entry-error (error)

((text :initarg :text :reader text)))

Condition Handlers

In parse-log-entry you'll signal a malformed-log-entry-error if you can't parse the log entry. You signal errors with the function ERROR, which calls the lower-level function SIGNAL and drops into the debugger if the condition isn't handled. You can call ERROR two ways: you can pass it an already instantiated condition object, or you can pass it the name of the condition class and any initargs needed to construct a new condition, and it will instantiate the condition for you. The former is occasionally useful for resignaling an existing condition object, but the latter is more concise. Thus, you could write parse-log-entry like this, eliding the details of actually parsing a log entry:

(defun parse-log-entry (text)

(if (well-formed-log-entry-p text)

(make-instance 'log-entry ...)

(error 'malformed-log-entry-error :text text)))

What happens when the error is signaled depends on the code above parse-log-entry on the call stack. To avoid landing in the debugger, you must establish a condition handler in one of the functions leading to the call to parse-log-entry. When a condition is signaled, the signaling machinery looks through a list of active condition handlers, looking for a handler that can handle the condition being signaled based on the condition's class. Each condition handler consists of a type specifier indicating what types of conditions it can handle and a function that takes a single argument, the condition. At any given moment there can be many active condition handlers established at various levels of the call stack. When a condition is signaled, the signaling machinery finds the most recently established handler whose type specifier is compatible with the condition being signaled and calls its function, passing it the condition object.

The handler function can then choose whether to handle the condition. The function can decline to handle the condition by simply returning normally, in which case control returns to the SIGNAL function, which will search for the next most recently established handler with a compatible type specifier. To handle the condition, the function must transfer control out of SIGNAL via a nonlocal exit. In the next section, you'll see how a handler can choose where to transfer control. However, many condition handlers simply want to unwind the stack to the place where they were established and then run some code. The macro HANDLER-CASE establishes this kind of condition handler. The basic form of a HANDLER-CASE is as follows:

(handler-case expression

error-clause*)

where each error-clause is of the following form:

(condition-type ([var]) code)

If the expression returns normally, then its value is returned by the HANDLER-CASE. The body of a HANDLER-CASE must be a single expression; you can use PROGN to combine several expressions into a single form. If, however, the expression signals a condition that's an instance of any of the condition-types specified in any error-clause, then the code in the appropriate error clause is executed and its value returned by the HANDLER-CASE. The var, if included, is the name of the variable that will hold the condition object when the handler code is executed. If the code doesn't need to access the condition object, you can omit the variable name.

For instance, one way to handle the malformed-log-entry-error signaled by parse-log-entry in its caller, parse-log-file, would be to skip the malformed entry. In the following function, the HANDLER-CASE expression will either return the value returned by parse-log-entry or return NIL if a malformed-log-entry-error is signaled. (The it in the LOOP clause collect it is another LOOP keyword, which refers to the value of the most recently evaluated conditional test, in this case the value of entry.)

(defun parse-log-file (file)

(with-open-file (in file :direction :input)

(loop for text = (read-line in nil nil) while text

for entry = (handler-case (parse-log-entry text)

(malformed-log-entry-error () nil))

when entry collect it)))

When parse-log-entry returns normally, its value will be assigned to entry and collected by the LOOP. But if parse-log-entry signals a malformed-log-entry-error, then the error clause will return NIL, which won't be collected.

JAVA-STYLE EXCEPTON HANDLING

HANDLER-CASE is the nearest analog in Common Lisp to Java- or Python-style exception handling. Where you might write this in Java:

try {

doStuff();

doMoreStuff();

} catch (SomeException se) {

recover(se);

}

or this in Python:

try:

doStuff()

doMoreStuff()

except SomeException, se:

recover(se)

in Common Lisp you'd write this:

(handler-case

(progn

(do-stuff)

(do-more-stuff))

(some-exception (se) (recover se)))

This version of parse-log-file has one serious deficiency: it's doing too much. As its name suggests, the job of parse-log-file is to parse the file and produce a list of log-entry objects; if it can't, it's not its place to decide what to do instead. What if you want to use parse-log-file in an application that wants to tell the user that the log file is corrupted or one that wants to recover from malformed entries by fixing them up and re-parsing them? Or maybe an application is fine with skipping them but only until a certain number of corrupted entries have been seen.

You could try to fix this problem by moving the HANDLER-CASE to a higher-level function. However, then you'd have no way to implement the current policy of skipping individual entries--when the error was signaled, the stack would be unwound all the way to the higher-level function, abandoning the parsing of the log file altogether. What you want is a way to provide the current recovery strategy without requiring that it always be used.

Restarts

The condition system lets you do this by splitting the error handling code into two parts. You place code that actually recovers from errors into restarts, and condition handlers can then handle a condition by invoking an appropriate restart. You can place restart code in mid- or low-level functions, such as parse-log-file or parse-log-entry, while moving the condition handlers into the upper levels of the application.

To change parse-log-file so it establishes a restart instead of a condition handler, you can change the HANDLER-CASE to a RESTART-CASE. The form of RESTART-CASE is quite similar to a HANDLER-CASE except the names of restarts are just names, not necessarily the names of condition types. In general, a restart name should describe the action the restart takes. In parse-log-file, you can call the restart skip-log-entry since that's what it does. The new version will look like this:

(defun parse-log-file (file)

(with-open-file (in file :direction :input)

(loop for text = (read-line in nil nil) while text

for entry = (restart-case (parse-log-entry text)

(skip-log-entry () nil))

when entry collect it)))

If you invoke this version of parse-log-file on a log file containing corrupted entries, it won't handle the error directly; you'll end up in the debugger. However, there among the various restarts presented by the debugger will be one called skip-log-entry, which, if you choose it, will cause parse-log-file to continue on its way as before. To avoid ending up in the debugger, you can establish a condition handler that invokes the skip-log-entry restart automatically.

The advantage of establishing a restart rather than having parse-log-file handle the error directly is it makes parse-log-file usable in more situations. The higher-level code that invokes parse-log-file doesn't have to invoke the skip-log-entry restart. It can choose to handle the error at a higher level. Or, as I'll show in the next section, you can add restarts to parse-log-entry to provide other recovery strategies, and then condition handlers can choose which strategy they want to use.

But before I can talk about that, you need to see how to set up a condition handler that will invoke the skip-log-entry restart. You can set up the handler anywhere in the chain of calls leading to parse-log-file. This may be quite high up in your application, not necessarily in parse-log-file's direct caller. For instance, suppose the main entry point to your application is a function, log-analyzer, that finds a bunch of logs and analyzes them with the function analyze-log, which eventually leads to a call to parse-log-file. Without any error handling, it might look like this:

(defun log-analyzer ()

(dolist (log (find-all-logs))

(analyze-log log)))

The job of analyze-log is to call, directly or indirectly, parse-log-file and then do something with the list of log entries returned. An extremely simple version might look like this:

(defun analyze-log (log)

(dolist (entry (parse-log-file log))

(analyze-entry entry)))

where the function analyze-entry is presumably responsible for extracting whatever information you care about from each log entry and stashing it away somewhere.

Thus, the path from the top-level function, log-analyzer, to parse-log-entry, which actually signals an error, is as follows:

Assuming you always want to skip malformed log entries, you could change this function to establish a condition handler that invokes the skip-log-entry restart for you. However, you can't use HANDLER-CASE to establish the condition handler because then the stack would be unwound to the function where the HANDLER-CASE appears. Instead, you need to use the lower-level macro HANDLER-BIND. The basic form of HANDLER-BIND is as follows:

(handler-bind (binding*) form*)

where each binding is a list of a condition type and a handler function of one argument. After the handler bindings, the body of the HANDLER-BIND can contain any number of forms. Unlike the handler code in HANDLER-CASE, the handler code must be a function object, and it must accept a single argument. A more important difference between HANDLER-BIND and HANDLER-CASE is that the handler function bound by HANDLER-BIND will be run without unwinding the stack--the flow of control will still be in the call to parse-log-entry when this function is called. The call to INVOKE-RESTART will find and invoke the most recently bound restart with the given name. So you can add a handler to log-analyzer that will invoke the skip-log-entry restart established in parse-log-file like this:5

(defun log-analyzer ()

(handler-bind ((malformed-log-entry-error

#'(lambda (c)

(invoke-restart 'skip-log-entry))))

(dolist (log (find-all-logs))

(analyze-log log))))

In this HANDLER-BIND, the handler function is an anonymous function that invokes the restart skip-log-entry. You could also define a named function that does the same thing and bind it instead. In fact, a common practice when defining a restart is to define a function, with the same name and taking a single argument, the condition, that invokes the eponymous restart. Such functions are called restart functions. You could define a restart function for skip-log-entry like this:

(defun skip-log-entry (c)

(invoke-restart 'skip-log-entry))

Then you could change the definition of log-analyzer to this:

(defun log-analyzer ()

(handler-bind ((malformed-log-entry-error #'skip-log-entry))

(dolist (log (find-all-logs))

(analyze-log log))))

As written, the skip-log-entry restart function assumes that a skip-log-entry restart has been established. If a malformed-log-entry-error is ever signaled by code called from log-analyzer without a skip-log-entry having been established, the call to INVOKE-RESTART will signal a CONTROL-ERROR when it fails to find the skip-log-entry restart. If you want to allow for the possibility that a malformed-log-entry-error might be signaled from code that doesn't have a skip-log-entry restart established, you could change the skip-log-entry function to this:

(defun skip-log-entry (c)

(let ((restart (find-restart 'skip-log-entry)))

(when restart (invoke-restart restart))))

FIND-RESTART looks for a restart with a given name and returns an object representing the restart if the restart is found and NIL if not. You can invoke the restart by passing the restart object to INVOKE-RESTART. Thus, when skip-log-entry is bound with HANDLER-BIND, it will handle the condition by invoking the skip-log-entry restart if one is available and otherwise will return normally, giving other condition handlers, bound higher on the stack, a chance to handle the condition.

Providing Multiple Restarts

Since restarts must be explicitly invoked to have any effect, you can define multiple restarts, each providing a different recovery strategy. As I mentioned earlier, not all log-parsing applications will necessarily want to skip malformed entries. Some applications might want parse-log-file to include a special kind of object representing malformed entries in the list of log-entry objects; other applications may have some way to repair a malformed entry and may want a way to pass the fixed entry back to parse-log-entry.

To allow more complex recovery protocols, restarts can take arbitrary arguments, which are passed in the call to INVOKE-RESTART. You can provide support for both the recovery strategies I just mentioned by adding two restarts to parse-log-entry, each of which takes a single argument. One simply returns the value it's passed as the return value of parse-log-entry, while the other tries to parse its argument in the place of the original log entry.

(defun parse-log-entry (text)

(if (well-formed-log-entry-p text)

(make-instance 'log-entry ...)

(restart-case (error 'malformed-log-entry-error :text text)

(use-value (value) value)

(reparse-entry (fixed-text) (parse-log-entry fixed-text)))))

The name USE-VALUE is a standard name for this kind of restart. Common Lisp defines a restart function for USE-VALUE similar to the skip-log-entry function you just defined. So, if you wanted to change the policy on malformed entries to one that created an instance of malformed-log-entry, you could change log-analyzer to this (assuming the existence of a malformed-log-entry class with a :text initarg):

(defun log-analyzer ()

(handler-bind ((malformed-log-entry-error

#'(lambda (c)

(use-value

(make-instance 'malformed-log-entry :text (text c))))))

(dolist (log (find-all-logs))

(analyze-log log))))

You could also have put these new restarts into parse-log-file instead of parse-log-entry. However, you generally want to put restarts in the lowest-level code possible. It wouldn't, though, be appropriate to move the skip-log-entry restart into parse-log-entry since that would cause parse-log-entry to sometimes return normally with NIL, the very thing you started out trying to avoid. And it'd be an equally bad idea to remove the skip-log-entry restart on the theory that the condition handler could get the same effect by invoking the use-value restart with NIL as the argument; that would require the condition handler to have intimate knowledge of how the parse-log-file works. As it stands, the skip-log-entry is a properly abstracted part of the log-parsing API.

Other Uses for Conditions

While conditions are mainly used for error handling, they can be used for other purposes--you can use conditions, condition handlers, and restarts to build a variety of protocols between low- and high-level code. The key to understanding the potential of conditions is to understand that merely signaling a condition has no effect on the flow of control.

The primitive signaling function SIGNAL implements the mechanism of searching for an applicable condition handler and invoking its handler function. The reason a handler can decline to handle a condition by returning normally is because the call to the handler function is just a regular function call--when the handler returns, control passes back to SIGNAL, which then looks for another, less recently bound handler that can handle the condition. If SIGNAL runs out of condition handlers before the condition is handled, it also returns normally.

The ERROR function you've been using calls SIGNAL. If the error is handled by a condition handler that transfers control via HANDLER-CASE or by invoking a restart, then the call to SIGNAL never returns. But if SIGNAL returns, ERROR invokes the debugger by calling the function stored in *DEBUGGER-HOOK*. Thus, a call to ERROR can never return normally; the condition must be handled either by a condition handler or in the debugger.

Another condition signaling function, WARN, provides an example of a different kind of protocol built on the condition system. Like ERROR, WARN calls SIGNAL to signal a condition. But if SIGNAL returns, WARN doesn't invoke the debugger--it prints the condition to *ERROR-OUTPUT* and returns NIL, allowing its caller to proceed. WARN also establishes a restart, MUFFLE-WARNING, around the call to SIGNAL that can be used by a condition handler to make WARN return without printing anything. The restart function MUFFLE-WARNING finds and invokes its eponymous restart, signaling a CONTROL-ERROR if no such restart is available. Of course, a condition signaled with WARN could also be handled in some other way--a condition handler could "promote" a warning to an error by handling it as if it were an error.

For instance, in the log-parsing application, if there were ways a log entry could be slightly malformed but still parsable, you could write parse-log-entry to go ahead and parse the slightly defective entries but to signal a condition with WARN when it did. Then the larger application could choose to let the warning print, to muffle the warning, or to treat the warning like an error, recovering the same way it would from a malformed-log-entry-error.

A third error-signaling function, CERROR, provides yet another protocol. Like ERROR, CERROR will drop you into the debugger if the condition it signals isn't handled. But like WARN, it establishes a restart before it signals the condition. The restart, CONTINUE, causes CERROR to return normally--if the restart is invoked by a condition handler, it will keep you out of the debugger altogether. Otherwise, you can use the restart once you're in the debugger to resume the computation immediately after the call to CERROR. The function CONTINUE finds and invokes the CONTINUE restart if it's available and returns NIL otherwise.

You can also build your own protocols on SIGNAL--whenever low-level code needs to communicate information back up the call stack to higher-level code, the condition mechanism is a reasonable mechanism to use. But for most purposes, one of the standard error or warning protocols should suffice.

You'll use the condition system in future practical chapters, both for regular error handling and, in Chapter 25, to help in handling a tricky corner case of parsing ID3 files. Unfortunately, it's the fate of error handling to always get short shrift in programming texts--proper error handling, or lack thereof, is often the biggest difference between illustrative code and hardened, production-quality code. The trick to writing the latter has more to do with adopting a particularly rigorous way of thinking about software than with the details of any particular programming language constructs. That said, if your goal is to write that kind of software, you'll find the Common Lisp condition system is an excellent tool for writing robust code and one that fits quite nicely into Common Lisp's incremental development style.

Writing Robust Software

For information on writing robust software, you could do worse than to start by finding a copy of Software Reliability (John Wiley & Sons, 1976) by Glenford J. Meyers. Bertrand Meyer's writings on Design By Contract also provide a useful way of thinking about software correctness. For instance, see Chapters 11 and 12 of his Object-Oriented Software Construction (Prentice Hall, 1997). Keep in mind, however, that Bertrand Meyer is the inventor of Eiffel, a statically typed bondage and discipline language in the Algol/Ada school. While he has a lot of smart things to say about object orientation and software reliability, there's a fairly wide gap between his view of programming and The Lisp Way. Finally, for an excellent overview of the larger issues surrounding building fault-tolerant systems, see Chapter 3 of the classic Transaction Processing: Concepts and Techniques (Morgan Kaufmann, 1993) by Jim Gray and Andreas Reuter.

In the next chapter I'll give a quick overview of some of the 25 special operators you haven't had a chance to use yet, at least not directly.

1Throws or raises an exception in Java/Python terms

2Catches the exception in Java/Python terms

3In this respect, a condition is a lot like an exception in Java or Python except not all conditions represent an error or exceptional situation.

4In some Common Lisp implementations, conditions are defined as subclasses of STANDARD-OBJECT, in which case SLOT-VALUE, MAKE-INSTANCE, and INITIALIZE-INSTANCE will work, but it's not portable to rely on it.

5The compiler may complain if the parameter is never used. You can silence that warning by adding a declaration (declare (ignore c)) as the first expression in the LAMBDA body.

Condition Handling in the Lisp Language Family

This paper appears in

Advances in Exception Handling Techniques,

edited by A. Romanovsky, C. Dony, J.L. Knudsen, and A. Tripathi.

This book, published in 2001, is part of Lecture Notes in Computer Science, Volume 2022,

published by Springer.

This paper was originally written as an HTML document, exactly as shown below. Any final reformatting that was done for hardcopy publication in LaTeX may have been lost in this version. Any new text that has been added for this annotated version appears bracketed and in color green; such text is intended to help clarify the historical context as time passes.

--Kent Pitman, 28-Feb-2002.

Annotated original document follows.

Click here for an index of other titles by Kent Pitman.

Condition Handling in the Lisp Language Family

by Kent M. Pitman

Copyright © 2001, Kent M. Pitman. All Rights Reserved.

[ Web version Copyright © 2002, Kent M. Pitman. All Rights Reserved. ]

Introduction

The Lisp family of languages has long been a rich source of ideas and inspiration in the area of error handling. Here we will survey some of the abstract concepts and terminology, as well as some specific language constructs that Lisp has contributed.

Although there are numerous dialects of Lisp, several of which offer the modern concepts and capabilities described herein, we will focus specifically on Common Lisp as described in the ANSI standard, X3.226-1994 [X3J13 1994].

Condition Systems vs Error Systems

The Common Lisp community typically prefers to speak about its condition system rather than its error system to emphasize that there are not just fatal but also non-fatal situations in which the capabilities provided by this system are useful.

Not all exceptional situations are represented, or sometimes even detected. A situation that  represented within the language is referred to in Common Lisp as a condition; an object of class CONDITION is used to represent such a situation.

A condition is said to be the generalization of an error. Correspondingly, within the language the class CONDITION is a superclass of another class ERROR, which represents situations that would be fatal if not appropriately managed.

So the set of all situations involving conditions includes not only descriptions of outright erroneous situations but also descriptions of situations that are merely unusual or questionable. Even in the case of non-error conditions, the programmer may, as a matter of expressive freedom, choose to use the same capabilities and protocols as would be used for "real" error handling.

Condition Handling Is Primarily a Protocol Matter

To properly understand condition handling, it is critical to understand that it is primarily about protocol, rather than mere computational ability. The establishment of protocols is a sort of before-the-fact hedge against the "prisoner's dilemma"; that is, it creates an obvious way for two people who are not directly communicating to structure independently developed code so that it works in a manner that remains coherent when such code is later combined.

For example, if we want to write a program that searches a list for an object, returning true if the object is present and false otherwise, we could write the following, but would ordinarily not:

(defun search-list (goal-item list-to-search)

(handler-case

;; Main body

(progn (dolist (item list-to-search)

(when (eq item goal-item)

(return-from search-list t)))

;; Search has failed, signal an error.

(error 'search-failure

:item goal-item))

;; Upon error, just return false.

(error () nil)))

The reason not to write this is not that it will not work, but that it involves undue amounts of unneeded mechanism. The language already contains simpler ways of expressing transfer of control from point A to point B in a program where the same programmer, acting in the same role, controls the code at both points. For example, the following would suffice:

(defun search-list (goal-item list-to-search)

(dolist (item list-to-search)

(when (eq item goal-item)

(return-from search-list t)))

nil)

(defun search-list (goal-item list-to-search)

(dolist (item list-to-search)

(when (eq item goal-item)

(return-from search-list t)))

(error 'search-failure :item goal-item))

Protocol is simply not needed when communicating lexically among parts of a program that were written as a unit and that are not called by other programs that are either logically separated or, at minimum, logically separable. The distinction is subjective, but it is nevertheless important.

Historical Influences

Before beginning to look in detail at the features of Common Lisp's condition system, it's useful to observe that computer languages, like human languages, evolve over time both to accommodate current needs and to repair problems observed in past experience. The interesting features of the Common Lisp condition system were not suddenly designed one day as a spontaneous creative act, but rather grew from many years of prior experience in other languages and systems.

Influence of Multics PL/I on Symbolics Zetalisp

The PL/I language, designed at IBM in the early 1960's, included an extensive condition mechanism which had an extensible set of named signals and dynamic handlers running in the dynamic context of the signal. Since PL/I included "downward" lexical closures these handlers had access to the erring environment, and sometimes to the details of the error.

Multics, whose official language was PL/I, adapted and extended this, including the addition of any_other, cleanup, and unclaimed_signal, the passing of machine conditions and other arbitrary data, cross-protection-domain signals, and the use of this mechanism to manage multiple suspended environments on one stack.

For security reasons, Multics had an elaborate and rigidly enforced separation of various kinds of code into protected "rings". The Multics operating system relied on their expanded signaling system for several critical system functions, mostly in the user ring, although cross-ring signals were possible in certain cases involving paging or memory errors.

Historically, Multics was an "early" environment, but it was not, by any analysis, a toy. Its condition system was carefully designed, heavily tested, and had many important characteristics that influenced the later design of Lisp:

- It separated the notion of condition signaling from condition handling.

- It offered the possibility, through program_interrupt conditions, of resuming an erring computation, presumably after correcting the offending situation.

- It began to deal with the mediation of signaling and handling through not only a condition type but also a set of associated data appropriate to that type: By using a system-defined operator called "signal_" instead of the normal PL/I "signal", a data block could be associated with the condition being signaled, a crude precursor to the idea of object-oriented condition descriptions that followed later in Lisp.

- It began to deal with the notion of default handling, through the use of the unclaimed_signal pseudocondition.

In the early 1980's, some former users of the Multics system, including Daniel L. Weinreb, Bernard Greenberg and David Moon, harvested the good ideas of the Multics PL/I condition system and recast them into Zetalisp, a dialect running on the Symbolics Lisp Machines. This redesign was called simply the "New Error System" (or sometimes just "NES").

Influence of Symbolics Zetalisp on Common Lisp

Key elements of the New Error System (NES) in Symbolics Zetalisp were:

- NES had an object-oriented nature.

- NES clearly separated the treatment of exceptional situations into three logically distinct programming activities:

- Establishing handlers.

- Managing "proceed types" (what Common Lisp later called "restarts").

- Detecting exceptional situations and signaling appropriate conditions.

- NES provided for erring programs to be resumed either interactively or non-interactively, separating information about prompting for replacement data from the conduits that would carry such data so that programs wishing to do mechanical recovery could bypass the prompting but use the rest of the recovery pipeline.

NES directly and strongly influenced the design of the Common Lisp condition system. In fact, one initial concern voiced by a number of vendors was that they were fearful that somehow the ideas of the condition system, being taken from the Lisp Machine environment, would not perform well on standard hardware. It took several months of discussion, and the availability of a free public implementation of the ideas, before these fears were calmed and the Common Lisp community was able to adopt them. Even so, numerous small changes and a few major changes were made in the process.

In both Zetalisp and Common Lisp, handlers are functions that are called in the dynamic context of the signaling operation. No stack unwinding has yet occurred when the handlers are called. Potential handlers are tried in order until one decides to handle the condition. Probably the most conspicuous change between NES and the Common Lisp Condition System was the choice of how a handler function communicated its decision to elect a specific mode of recovery for the condition being signaled.

NES used a passive recovery mechanism. That is, in all cases, the handler would return one or more values. The nature of the return values would determine which recovery mode (called a "proceed type" in Zetalisp) was to be used. If NIL was returned, the handler had elected no proceed type, and the next handler was tried. Otherwise, the handler must return at least one value, a keyword designating the proceed type, and, optionally, additional values which were data appropriate to that manner of proceeding.

Common Lisp uses an active recovery mechanism. That is, any handler that wishes to designate a recovery mechanism (called a "restart" in Common Lisp) must imperatively transfer control to that restart. If the handler does not transfer control, that is, if the handler returns normally, any returned values are ignored and the handler is said to have "declined" (i.e., elected no restart), and the next handler is tried.

The Maclisp Experience

I am commonly credited with the "creation" of the Common Lisp Condition System, although I hope to show through this paper that my role in the design was largely to take the ideas of others and carefully transplant them to Common Lisp. In doing this, I relied on my personal experiences to guide me, and many of my formative experiences came from my work with Maclisp [Pitman 1983], which originated at MIT's Project MAC (later renamed to be the Laboratory for Computer Science), and which ran on the Digital Equipment Corporation (DEC) PDP10, DEC TOPS20 and Honeywell Multics systems.

Maclisp, had a relatively primitive error system, which I had used extensively. At the time I came to the Lisp Machine's NES, I did not know what I was looking for in an error system, but I knew, based on my experience with Maclisp, what I was not looking for. So what impressed me initially about NES was that it had fixed many of the design misfeatures that I had seen in Maclisp.

One important bit of background on Maclisp, at least on the PDP10 implementation that I used, was that it had no STRING datatype. In almost all cases where one might expect strings to be used, interned symbols were used instead. Symbols containing characters that might otherwise confuse the tokenizer were bounded on either end by a vertical bar (|). Also, since symbols would normally name variables, they generally had to be quoted with a leading single quote (') to protect them from the Lisp evaluation mechanism and allow them to be used as pseudostrings.

'|This is a quoted Maclisp symbol.|

Poor Separation of Signaling and Handling in Maclisp

Maclisp had two forms of the function ERROR. In the simple and most widely used form, one merely called ERROR with one argument, a description of the error. Such errors would stop program execution with no chance of recovery other than to transfer to the innermost ERRSET, the approximate Maclisp equivalent of Common Lisp's IGNORE-ERRORS.

(error '|YOU LOSE|)

It was possible, however, in a limited way, to specify the particular kind of error. There were about a dozen predefined kinds of errors that one could identify that did allow recovery. For example,

(error '|VARIABLE HAS NO VALUE| 'A 'UNBND-VRBL)

The "keyword" UNBND-VRBL was a system-defined name that indicated to the system that this was an error of kind "unbound variable". A specific recovery strategy was permitted in this case. One could, either interactively in a breakpoint or through the dynamic establishment of a handler for such errors, provide a value for the variable. If that happened, the call to ERROR would then return a list of that value and the caller of ERROR was expected to pick up that value and use it.

This worked fine for the case where the programmer knew the kind of error and was prepared to recover from it. But a strange situation occurred when one knew the kind of error but was not prepared to recover. Sometimes one knew one had an unbound variable, and wanted to call ERROR, but was not prepared to recover. In this case, the programmer was forced to lie and to say that it was an error of arbitrary type, using just the short form, to avoid the misperception on the part of potential handlers that returning a recovery value would be useful.

(error '|VARIABLE HAS NO VALUE| 'A)

One feature of the NES, which I personally found very attractive, was the notion that I could freely specify the class of error without regard to whether I was prepared to handle it in some particular way. The issue of how to handle the error was specified orthogonally.

Error Message Identity In Maclisp

In PDP10 Maclisp, error messages were historically all uppercase, since the system's primitive error messages were that way and many users found it aesthetically unpleasant to have some messages in mixed case while others were entirely uppercase. At some point, however, there was pressure to provide mixed case error messages. The decision made by the Maclisp maintainers of the time was not to yield to such pressure.

The problem was that many programs faced with an error message were testing it for object identity. For example:

(eq msg '|UNBOUND VARIABLE|)

Had we changed the case of all of the error messages in the Maclisp system to any other case, lower or mixed, these tests would have immediately begun to fail, breaking a lot of installed code and costing a lot of money to fix. The change would have been seen to be gratuitous.

The lesson from this for all of us in the Maclisp community, which became magnified later when we confronted the broader community of international users, was that the identity of an error should not be its name. That is, had we to do it over again, we would not have used |unbound variable| nor |Unbound Variable| as the identity of the error, but rather would have created objects whose slots or methods were responsible for yielding the presented string, but whose identity and nature was controlled orthogonally. This was another thing that NES offered that drew me immediately to it.

Terminological Influences

At the time of the Common Lisp design, Scheme did not have an error system, and so its contribution to the dialog on condition systems was not that of contributing an operator or behavior. However, it still did have something to contribute: the useful term continuation. For our purposes here, it is sufficient to see a continuation as an actual or conceptual function that represents, in essence, one of possibly several "future worlds", any of which can be entered by electing to call its associated continuation.

This metaphor was of tremendous value to me socially in my efforts to gain acceptance of the condition system, because it allowed a convenient, terse explanation of what "restarts" were about in Common Lisp. Although Scheme continuations are, by tradition, typically passed by explicit data flow, this is not a requirement. And so I have often found myself thankful for the availability of a concept so that I could talk about the establishment of named restart points as "taking a continuation, labeling it with a tag, and storing it away on a shelf somewhere for possible later use."

Likewise, I found it useful in some circles to refer to some of the concepts of reflective Lisps, such as Brian Smith's "3Lisp" [Smith 1982], and later work inspired by it. I feel that the condition system's ability to introspect (through operators such as FIND-RESTART) about what possible actions are pending, without actually invoking those pending actions, is an important reflective capability. Even though Common Lisp does not offer general-purpose reflection, the ability to use this metaphor for speaking about those aspects of the language that are usefully described by it simplifies conversations.

Abstract Concepts

Having now hopefully firmly established that the formative ideas in the Common Lisp Condition System did not all spring into existence with the language itself, and are really the legacy of the community using the continuum of languages of which Common Lisp is a part, we can now turn our attention to a survey of some of the important features that Common Lisp provides.

Separating Signaling and Handling

Traditionally, "error handling" has been largely a process of programs stopping and the only real question has been "how much of the program stops?" or "how far out do I throw?" It is against that backdrop that modern condition handling can be best understood.

The proper way to think about condition handling is this:

The process of programming is about saying what to do in every circumstance. In that regard, a computer has been sometimes characterized as a "relentless judge of incompleteness". When a program reaches a place where there are several possible next steps and the program is unwilling or incapable of choosing among them, the program has detected an exceptional situation.

The possible next steps are called restarts. Restarts are, effectively, named continuations.

The process of asking for help in resolving the problem of selecting among the possible next steps is called signaling.

The independently contributed pieces of code which are consulted during the signaling process are called handlers. In Common Lisp, these are functions contributed by the dynamic call chain that are tried in order from innermost (i.e., most specific) to outermost (i.e., most general). Each handler is called with an argument that is a description of the problem situation. The handler will transfer control (by GO, RETURN or THROW) if it chooses to handle the problem described by its argument.

In describing condition handling, I tell the following story to help people visualize the need for its various parts:

Think of the process of signaling and handling as analogous to finding a fork in a road that you do not commonly travel. You don't know which way to go, so you make known your dilemma, that is, you signal a condition. Various sources of wisdom (handlers) present themselves, and you consult each, placing your trust in them because you have no special knowledge yourself of what to do. Not all sources of wisdom are experts on every topics, so some may decline to help before you find one that is confident of its advice. When an appropriately confident source of wisdom is found, it will act on your behalf. The situation has been handled.

In the case that the situation is not handled, the next action depends on which operator was used to signal. The function signal will just return normally when a condition goes unhandled. The function error is like signal, but rather than return, it enters the debugger. The Common Lisp debugger might allow access to low-level debugging features such as examination of individual storage locations, but it is not required to. Its  role is to be an interactive handler; that is, to present the human user interactively with various options about how computation might be resumed. Conceptually, this is the same as if it were acting as the human user's proxy in being the element on the list of handlers, so that the human user is the source of wisdom whose choice will determine how to proceed. Other capabilities that the debugger might offer in support of that human's decision are probably very important in practice, but are conceptually uninteresting to this understanding of the debugger's role in signaling and handling.

Note, too, that in some possible future world, knowledge representation may have advanced enough that handlers could, rather than act unconditionally on behalf of the signaler, merely return a representation of a set of potential actions accompanied by descriptive information respresenting motivations, consequences, and even qualitative representations of the goodness of each. Such information might be combined with, compared to, or confirmed by recommendations from other sources of wisdom in order to produce a better result. This is how consultation of sources of wisdom would probably work in the real world. Consider that even a doctor who is sure of what a patient needs will ask the patient's permission before acting. However, this last step of confirmation, which would allow more flexibility in the reasoning process, is not manifest in Common Lisp as of the time of writing this paper. It is an open area for future research.

Some of these issues are discussed in much greater detail in my 1990 conference paper [Pitman 1990].

Generalized Conditions

It was mentioned earlier that the space of conditions that can be used in the Common Lisp Condition System is more general than the space of mere errors. Here are some examples.

Serious, non-error Conditions

The superclass of error is serious-condition. This kind of condition is a subclass of condition but is serious enough that conditions of this kind should generally enter the debugger if unhandled. Serious conditions, which the Zetalisp NES called "debugger conditions", exist as a separately named concept from "error conditions" to accommodate things that are not semantic errors in a program, but are instead resource limitations and other incidental accomodations to pragmatics.

Suppose one writes the following:

(ignore-errors (open "some.file"))

This will trap errors during the file open. However, what if a stack overflow occurs, not for reasons of infinite recursion, but merely because the call is nested very deeply in other code? The answer is that a stack overflow is considered a serious condition, but not an error. The above code is equivalent to:

(handler-case (open "some.file")

(error (c)

(values nil c)))

And since any condition representing a stack overflow is going to be a kind of SERIOUS-CONDITION, but not a kind of ERROR, the use of IGNORE-ERRORS will succeed in trapping a file error but not a stack overflow. If one wanted to catch serious conditions as well, one would write instead:

(handler-case (open "some.file")

(serious-condition (c)

(values nil c)))

Non-serious conditions

Some conditions are not at all serious. Such conditions might be handled, but there is an obvious default action in the case of their going unhandled.

Consider a program doing line-at-a-time output to a console. One might assume the screen to have infinite height, and the output might look like:

(defvar *line-number* 0)

(defun show-lines (lines)

(dolist (line lines)

(show-line line)))

However, it might be useful to specify screen line height, and to have the console pause every so many lines for a human reader to confirm that it's ok to proceed. There are, of course, a number of ways such a facility could be programmed, but one possible such way is to use the condition system. For example,

(defvar *line-number* 0)

(defvar *page-height* nil)

(define-condition end-of-page (condition) ())

(defun show-lines (lines)

(dolist (line lines)

(show-line line)  ; Oops. Omitted this call from original paper!

(incf *line-number*)

(when (and *page-height*

(zerop (mod *line-number* *page-height*)))

(restart-case (signal 'end-of-page)

(continue ())))))

In the above, there is only one way to proceed. A restart named CONTINUE is offered as a way of imperatively selecting this option (imperatively bypassing any other pending handlers), but if the handler declines and the condition goes unhandled, the same result will be achieved.

A similar kind of facility could be used to manage end of line handling. There, it's common to allow various modes, and so a corresponding set of restarts has to be established, which handlers would choose among. If no handler elected to handle the condition, however, no great harm would come. Here's an example of how that might look:

(defvar *line-length* nil)

(define-condition end-of-line (condition) ())

(defun show-line (line)

(let ((eol (or *line-length* -1)) (hpos 0))

(loop for pos below (length line)

for ch = (char line pos)

do (write-char ch)

when (= hpos eol)

do (restart-case (signal 'end-of-line)

(wrap ()

(write-char #\Newline)

(setq hpos 0))

(truncate ()

(return))

(continue ()

;; just allow to continue

))

else do (incf hpos))))

Independent, Reflective Specification of Restarts

It has long been the case that Lisp offered the ability to dynamically make the decision to transfer to a computed return point using the special operator THROW. However, without reflective capabilities, there has not been the ability for a programmer to determine if there was a pending CATCH to which control could be thrown other than relatively clumsy idioms such as the following:

(ignore-errors (throw 'some-tag 'some-value))

The problem with the above idiom is that while it "detects" the presence or absence of a pending tag, it only retains local control and the ability to reason about this knowledge in the case of the tag's non-existence. The price of detecting the tag's existence is transfer to that tag.

The Common Lisp Condition System adds a limited kind of reflective capability in the form of a new kind of catch point, called a restart, the presence or absence of which can be reasoned about without any attempt to actually perform a transfer. A restart can also have associated with it a descriptive textual string that a human user can be shown by the debugger to describe the potential action offered by the restart.

Restart points that require transfer of control but no data can be established straightforwardly with WITH-SIMPLE-RESTART. For example:

(defun lisp-top-level-loop ()

(with-simple-restart (exit "Exit from Lisp.")

(loop

(with-simple-restart (continue "Return to Lisp toplevel.")

(print (eval (read)))))))

Restarts that require data can also be established using a slightly more elaborate syntax. This syntax not only accommodates the programmatic data flow to the restart, but also enough information for the Common Lisp function INVOKE-RESTART-INTERACTIVELY to properly prompt for any appropriate values to be supplied to that restart. For example:

(defun my-symbol-value (name)

(if (boundp name)

(symbol-value name)

(restart-case (error 'unbound-variable :name name)

(use-value (value)

:report "Specify a value to use."

:interactive (lambda ()

(format t "~&Value to use: ")

(list (eval (read))))

value)

(store-value (value)

:report "Specify a value to use and store."

:interactive (lambda ()

(format t "~&Value to use and store: ")

(list (eval (read))))

(setf (symbol-value name) value)

value))))

Code that inquires about such restarts typically makes use of FIND-RESTART to test for the availability of a restart, and then INVOKE-RESTART to invoke a restart. For example:

(handler-bind ((unbound-variable

#'(lambda (c) ;argument is condition description

;; Try to make unbound variables get value 17

(dolist (tag '(store-value use-value))

(let ((restart (find-restart tag c)))

(when restart

(invoke-restart restart 17)))))))

(+ (my-symbol-value 'this-symbol-has-no-value)

(my-symbol-value 'pi))) ;pi DOES have a value!

=> 20.141592653589793

Absent such a handler, the restart would be offered interactively by the debugger, as in:

(+ (my-symbol-value 'this-symbol-has-no-value)

(my-symbol-value 'pi))

Error: The variable THIS-SYMBOL-HAS-NO-VALUE is unbound.

Please select a restart option:

1 - Specify a value to use.

2 - Specify a value to use and store.

3 - Return to Lisp toplevel.

4 - Exit from Lisp.

Option: 1

Value to use: 19

=> 22.141592653589793

Handling in the Context of the Signaler

A key capability provided by Common Lisp is the fact that, at the most primitive level, handling can be done in the dynamic context of the signaler, while certain very critical dynamic state information is still available that would be lost if a stack unwind happened before running the handler.

This capability is reflected in the ability of the operator handler-bind to take control of a computation  any transfer of control occurs. Note that the Common Lisp operator handler-case, which is more analogous to facilities offered in other languages, does not allow programmer-supplied code to run until after the transfer of control; this is useful for some simple situations, but is less powerful.

Consider a code fragment such as:

(handler-case (main-action)

(error (c) (other-action)))

In this example, the expression (other-action) will run  unwinding from wherever in (main-action) signaled the error, regardless of how deep into main-action that signaling occured.

By contrast, handler-bind takes control  the dynamic context of the call to SIGNAL, and so is capable of accessing restarts that are dynamically between the call to SIGNAL and the use of HANDLER-BIND. Consider this example:

(with-simple-restart (foo "Outer foo.")

(handler-case (with-simple-restart (foo "Inner foo.")

(error "Lossage."))

(error (c) (invoke-restart 'foo))))

In the above, the outer FOO restart will be selected, as contrasted with the following, where the inner FOO restart will be selected:

(with-simple-restart (foo "Outer foo.")

(handler-bind ((error #'(lambda (c) (invoke-restart 'foo))))

(with-simple-restart (foo "Inner foo.")

(error "Lossage."))))

This is important because error handling tends to want to make use of all available restarts, but  those that are in that code region that HANDLER-BIND can see but HANDLER-CASE cannot. Consider another example:

(handler-case (foo)

(unbound-variable (c)

(let ((r (find-restart 'use-value c)))

(if r (invoke-restart r nil)))))

The above example will not achieve its presumed intent, which is to supply NIL as the default value for any unbound variable encountered during the call to FOO. The problem is that any USE-VALUE restart that is likely to be found will also be within the call to FOO, and will no longer be active by the time the ERROR clause of the HANDLER-CASE expression is executed.

Use of HANDLER-BIND allows this example to work in a way that is not possible with HANDLER-CASE and its analogs in other programming languages. Consider:

(handler-bind (error

#'(lambda (c)

(let ((r (find-restart 'use-value c)))

(if r (invoke-restart r nil)))))

(foo))

Default Handling

Zetalisp contained a facility not only for asserting handlers to be used for conditions, but also an additional facility for asserting handlers that should be provisionally used only if no normal handlers were found. In effect, this meant there were two search lists, a handlers list and a default handlers list, which were searched in order.

In my use of Zetalisp's NES, I became convinced that it was conceptually incorrect to search the default handlers list in order; I felt it should be searched in reverse order. I had reported this as a bug, but it was never fixed. In all honesty, I'm not sure there was enough data then or perhaps even now to say whether I was right, although I continue to believe that default handling is something that should proceed from the outside in, not the inside out. Nevertheless, whether I was right or not is not so much relevant in this context as is the fact that it was a point of controversy that ended up influencing the design of Common Lisp's condition system. I was distrustful of the operator condition-bind-default that was offered by NES, and so I omitted it from the set of offerings that I transplanted to Common Lisp.

The Common Lisp Condition System does provide a way to implement the concept of a default handler, but it is idiomatic. And, perhaps not entirely coincidentally, it has the net effect of seeking default handlers from the outside in rather than the inside out, as I had always felt was right.

The Zetalisp mechanism looked like this:

(condition-bind-default ((error

#'(lambda (c)

...default handling...)))

...body in which handler is in effect...)

The corresponding Common Lisp idiom looks like this:

(handler-bind ((error

#'(lambda (c)

(signal c) ;resignal

...default handling...)))

...body in which handler is in effect...)

In effect, the Common Lisp idiom continues the signaling process but without explicitly relinquishing control. If the resignaled condition is unhandled, control will return to this handler and the default handling will be done. If, on the other hand, some outer handler does handle the condition, the default handling code will never be reached and so will not be run.

Unifying "Signals" and "Exceptions"

In some systems, such as Unix, "signals" are an asynchronous mechanism primarily used for implementing event-driven programming interfaces, but are not generally used within ordinary, synchronous programming.

While it is beyond the scope of the ANSI Common Lisp standard to address the issue of either interrupts or multitasking, most Common Lisp implementations have a convergent manner of coping with these issues that is sufficiently stable as to be worth some mention. The approach has been to separate the notion of "interrupting" from the notion of "signaling".

That is, in Common Lisp, all signaling is synchronous. However, such synchronous behavior can be usefully coupled with a process interruption to produce interesting effects.

In this separation, process interruptions without signaling might be done for any reason that involved the need to read or modify dynamic state of another process. Here is an example that merely reads the dynamic state of another process:

(defvar *my-dynamic-variable* 1)

(let ((temporary-process

(mp:process-run-function "temp" '()

;; Launch a temporary process that

;;  merely dynamically binds a

;;  certain variable and then

;;  sleeps for a minute.

#'(lambda ()

(let ((*my-dynamic-variable* 2))

(sleep 60)))))

(result-value nil))

;; Now interrupt our temporary process

;; to see the value of the variable

(mp:process-interrupt temporary-process

#'(lambda ()

(setq result-value *my-dynamic-variable*)))

;; Now wait for the interrupt to occur

(mp:process-wait "not yet assigned"

#'(lambda () result-value)) ;tests for a non-null value

;; If we get this far, the result-value has been assigned

;; and can be returned.

result-value)

=> 2

Note that this merely examines the dynamic state of our temporary process, but does not invoke any signaling mechanism at all. And while the process of interruption is inherently asynchronous, the actions to be done in the interrupted process are synchronous.

If we instead intertwine the notion of process interruption with signaling, we get what some systems might call "asynchronous signaling", but which Common Lisp views as just the composition of two orthogonal facilities. So, for example, a keyboard interrupt to a process might be accomplished by:

(define-condition keyboard-interrupt ()

((character :initarg :character :reader kbd-char))

(:report (lambda (condition stream)

(format t "The character ~@:C was typed."

(kbd-char condition)))))

(defun keyboard-interrupt (process character)

(mp:process-interrupt process

#'(lambda ()

;; Offer the process a chance to handle the condition.

;; If the condition is not handled, the call to SIGNAL returns

;; and the interrupt is completed.  Normal process execution

;; then continues.

(signal 'keyboard-interrupt :character character))))

Using such a facility, a keyboard process (itself a synchronous activity) can asynchronously interrupt another process (presumably, the window selected at the point an interrupt character is seen).

(defvar *selected-window* nil)

(defun keyboard-process (raw-keyboard-stream)

(loop (let ((char (read-char raw-keyboard-stream)))

(when *selected-window*

(if (is-interrupt-character? char)

(keyboard-interrupt (window-process *selected-window*)

char)

;; otherwise...

(add-to-input-buffer *selected-window* char))))))

Although the KEYBOARD-PROCESS shown here will interrupt the window process, an understanding of what happens at that point does not require any special knowledge of asynchrony. It merely requires observing that at the time of interruption, the other process was about to execute some expression (exp) and will now execute instead

(progn (funcall interrupt-function) (exp))

This kind of structured approach removes much of the mystery and unpredictability normally associated with asynchronous interrupts in other systems, where the description of the effect is often not linguistic at all but deals in overly concrete terms of bits and registers in a way that only career experts can hope to navigate. The sense in the Common Lisp community is that a correct conceptual treatment of these issues makes these sorts of capabilities something that "mere mortals" can safely and conveniently employ in their programming.

Open Issues

The Dylan language patterned its efforts after the Common Lisp Condition System, but made some interesting changes. I probably lack the appropriate experience and surely the appropriate objectivity to conclude whether their changes are clear improvements over the Common Lisp approaches. But it's plain that by making divergent decisions in some places, the Dylan community has identified certain areas of the Common Lisp design as "controversial".

Restarts vs Handlers

Common Lisp provides parallel but unrelated operators such as HANDLER-BIND and HANDLER-CASE for dealing with handlers, and RESTART-BIND and RESTART-CASE for dealing with restarts. It was thought that these were orthogonal operations, requiring unrelated dataflow, that really didn't belong intermingled. The Dylan community has sought to coalesce these by making restarts into a kind of condition, and eliminating special binding forms for them.

The "Condition Firewall"

Probably the most controversial semantic component of the Common Lisp condition system is what has come to be called the "condition firewall". The idea behind the condition firewall is that a given handler should be executed in an environment that does not "see" intervening handlers that might have been established since its establishment.

So, for example, consider this code:

(handler-case

(handler-bind ((error

#'(lambda (condition)

(declare (ignore condition))

(error 'unbound-variable :name 'fred))))

(handler-case ;; Signal an arbitrary error:

(error "Not an UNBOUND-VARIABLE error.")

(unbound-variable (c) (list 'inner c))))

(unbound-variable (c) (list 'outer c)))

This sets up two handlers for conditions of class UNBOUND-VARIABLE, one outside of the scope of the general-purpose handler for conditions of class ERROR and one inside of its scope. At the time the "arbitrary" error signaled, both handlers are in effect. This means that if the error being signaled  been of class UNBOUND-VARIABLE, it would have been caught by the inner HANDLER-CASE for UNBOUND-VARIABLE. However, as the search for a handler proceeds outward, the handlers that are tried are executed in a context where the inner handlers are no longer visible. As such, the above example yields

(OUTER #<ERROR UNBOUND-VARIABLE 12A39B87>)

By contrast, the following code:

(handler-case

(handler-bind ((error #'(lambda (condition)

(declare (ignore condition))

(error 'unbound-variable :name 'fred))))

(handler-case (error 'unbound-variable :name 'marvin)

(unbound-variable (c) (list 'inner c))))

(unbound-variable (c) (list 'outer c)))

yields

(INNER #<ERROR UNBOUND-VARIABLE 12A39B87>)

It is interesting to note as an aside that the "resignaling trick" used earlier in the discussion of default handling relies implicitly on the condition firewall in order to avoid infinite recursion. Without the condition firewall, a different mechanism for implementing default handling is needed.

The designers of the Dylan language chose to eliminate the condition firewall, perhaps out of necessity since the most useful restarts almost always occur in the dynamic space near the point of signal, and the handlers usually occur farther out. If handlers could only see the restarts farther out than where they were established, they would not see the most useful restarts. (I am personally doubtful of this argument, and am more inclined to believe that this is why restarts should not have been turned into a kind of condition in Dylan, but I could be wrong and time will tell.)

The Dylan notation is different in many ways from Common Lisp, but the approximately equivalent code to the above two examples would , I believe, return

(INNER #<ERROR UNBOUND-VARIABLE 12A39B87>)

Summary

Language features don't originate spontaneously out of nowhere. We have surveyed some of the origins of the Common Lisp Condition System in an effort to demonstrate how prior experiences, both good and bad, have influenced the present design. Nor is this the end of the story. The ideas in Common Lisp have had some influence on other languages and will, I hope, continue to do so, since there are a number of things the Common Lisp makes easy through its condition system that other languages do not.

We have also seen that good terminology is important both to the specification of a programming language and to its community acceptance. Programming is not only a technical endeavor, but a social one. So much of so many lives is spent doing programming, that it is critical that we have good terminology, beyond the terms of the language itself, for talking among each other about what we are doing within the language.

And we have surveyed some of the key features that distinguish Common Lisp's condition system from those offered by other languages, and highlighted some open issues, where Common Lisp's answers to certain problems have already met with challenges.

A Personal Footnote

During the design of Common Lisp, I headed the committee that produced the design of the condition system. At that time, there were many questions and doubts about the design: Were the decisions sound? Were all of the alternatives explored, or were there better ways we might later wish we'd tried? Were there problems lurking under the surface, waiting to bite someone when used under heavier stress?

It wasn't that people doubted our committee's competence, but rather many would-be reviewers lacked the relevant experience to critically analyze our proposals. Yet the design seemed mostly right to me, and my larger concern was that if we didn't at some point release it to a community of users to try, we'd be back at the same design table a few years later with the same questions and the same lack of community experience to answer them. A leap of faith seemed to be required to move ahead. So I and my committee nodded our collective heads and said we stood by the design. Personally, I had some doubts about some details, but I found it counterproductive to raise them because I believed the risk of not trying these things out was higher than the risk of trying them.

In my experience, much of language design is like this. We think we know how it will all come out, but we don't always. Usage patterns are often surprising, as one learns if one is around long enough to design a language or two and then watch how expectations play out in reality over a course of years. So it's a gamble. But the only way not to gamble is not to move ahead.

I once saw an interview on television with a font designer from Bitstream Inc. about how he conceptualized the process of font design. It is not about designing the shape of the letters, he explained, much to my initial surprise. Then he went on to explain that it was really about the shape of words. The font shapes play into that, but they are not, in themselves, the end goal. Programming language design is like that, too. It's not about the semantics of individual operators, but about how those operators fit together to form sentences in programs.

Unlike the situation with fonts, where whole books can be viewed instantly in a new font to see how the design works, we don't know in advance what sentences will be made in a programming language. We have to wait and see what people choose to write. Common Lisp took a step forward, and while we can quibble endlessly over whether any given design decision was right, the one design decision I'm most certain was right was to offer the community a rich set of capabilities that would empower them not only to write programs, but also to have a stake in future designs. Never again will I fear sending out e-mail to a design group asking for advice about what the semantics of HANDLER-BIND should be and finding that no one has an opinion! To me, that kind of progress, the evolution of a whole community's understanding, is the best kind of progress of all.

Acknowledgements

I would like to thank Keith Corbett, Christophe Dony, Bernard Greenberg, and Erik Naggum for reviewing drafts of this text. Any lingering errors after they got done looking at it are still my responsibility, but I'm quite sure the editorial, technical, and historical quality of this text was improved measurably through their helpful scrutiny.

Bibliography

[Multicians 2000]

Historical references to Multics can probably best be obtained from

http://www.multicians.org/

[Pitman 1983]

Kent M. Pitman, The Revised Maclisp Manual, Technical Report 295, MIT Laboratory for Computer Science, Cambridge, MA, May 1983.

[Pitman 1990]

Kent M. Pitman, "Exceptional Situations in Lisp", proceedings for the First European Conference on the Practical Application of Lisp (EUROPAL'90), Churchill College, Cambridge, UK, March 27-29, 1990.

http://www.nhplace.com/kent/Papers/Exceptional-Situations-1990.html

[Smith 1982]

B.C. Smith, Reflection and Semantics in a Procedural Language, Technical Report 272, MIT Laboratory for Computer Science, Cambridge, MA, January 1982.

[X3J13 1994]

American National Standard for Information Systems--Programming Language--Common Lisp (X3.226-1994)

http://www.lispworks.com/documentation/HyperSpec/Front/

[Weinreb 1983]

D.L. Weinreb and D.A. Moon, Lisp Machine Manual, MIT Artificial Intelligence Laboratory, Cambridge, MA, July 1981.

Original printed text document

Copyright 2001 by Kent M. Pitman. All Rights Reserved.

HTML hypertext version of document

Copyright 2002, Kent M. Pitman. All rights reserved.

The following limited, non-exclusive, revokable licenses are granted:

Browsing of this document (that is, transmission and display of a temporary copy of this document for the ordinary purpose of direct viewing by a human being in the usual manner that hypertext browsers permit such viewing) is expressly permitted, provided that no recopying, redistribution, redisplay, or retransmission is made of any such copy.

Bookmarking of this document (that is, recording only the document's title and Uniform Resource Locator, or URL, but not its content, for the purpose of remembering an association between the document's title and the URL, and/or for the purpose of making a subsequent request for a fresh copy of the content named by that URL) is also expressly permitted.

All other uses require negotiated permission.

A Tutorial on Conditions and Restarts in Clojure

This tutorial is adapted from Common Lisp: A Tutorial on Conditions and Restarts to use Clojure with the Pure Conditioning library.

A good introduction to the concepts we'll use here is the chapter on conditions and restarts in Peter Seibel’s excellent book, Practical Common Lisp. To get a more complete grasp of the condition system, so you might want to read that chapter. Another good read is Kent Pitman’s paper, Condition Handling in the Lisp Language Family. You should be able to follow this tutorial without reading that first, though.

I’ll attempt to show how effective the pure-conditioning system can be by building a validator for CSV (comma-separated values) files. The validator will check that all the fields in each row of the file are valid (according to some defined criteria).

Getting started

Our deps.edn file can be very simple for this example. We only need the pure-conditioning library (it's at the bottom of the page).

We'll start by pulling in pure-conditioning plus some dependencies we'll use to parse CSV and do some testing.

(ns tutorial

(:require [conditions :refer :all]

[clojure.java.io :refer [reader]]

[clojure.string :as str]

[clojure.test :refer [deftest is]]))

The CSV

The first row of the CSV will be a comma-separated list of headers, followed by rows with each column corresponding to the headers in the first row. A sample file looks like this:

rating,url,visitors,date

4,http://chaitanyagupta.com/home,1233445,2000-01-01

5,http://chaitanyagupta.com/blog,33333,2006-02-02

5,http://chaitanyagupta.com/code,2121212,2007-03-03

First we write functions to validate fields for the four headers we used above: rating, url, visitors, and date.

(defn validate-url

"The URL of the page; should start with http:// or https://."

[string]

(when-not (re-matches #"^https?://.*" string)

(condition :url-invalid (restarts {:url string})

(error "URL invalid"))))

(defn validate-rating

"String should contain an integer between 1 and 5, inclusive."

[string]

(let [rating (read-string string)]

(when-not (and (int? rating) (<= 1 rating 5))

(condition :invalid-rating (restarts {:rating rating})

(error "Rating is not an integer in range")))))

(defn validate-visitors

"The number of visitors to the page; string should contain an

integer more than or equal to zero."

[string]

(let [visitors (read-string string)]

(when-not (nat-int? visitors)

(condition :invalid-visitors (restarts {:visitors visitors})

(error "Number of visitors invalid")))))

(defn validate-date

"The published date of the URL. Should be in yyyy-mm-dd format."

[string]

(when-not (re-matches #"^\d{4}-\d{2}-\d{2}$" string)

(condition :invalid-date (restarts {:date string})

(error "Published date not in valid format"))))

All of these functions check the validation criteria and if there is an error, they signal a restartable condition which will raise an error with the given message if not handled. They all attach their relevant data to the condition as well.

To associate these functions to the fields they validate, we will register them in a simple dictionary:

(def validators {"url" validate-url

"rating"   validate-rating

"visitors" validate-visitors

"date"     validate-date})

Signalling validation errors

The Common Lisp tutorial has a section on defining an error type, but this system works differently and does not need condition types to be predefined.

Parsing the CSV

The parser converts raw CSV text into a list of lists – each item in these lists corresponds to a field in the CSV.

(Now is a good time to mention that this CSV parser is for illustration purposes only, and you should use a real CSV library for parsing.)

(defn parse-csv-file [file]

(with-open [f (reader file)]

(mapv #(str/split % #",") (line-seq f))))

The Validator (sans the restarts)

Finally, we get down to writing the validator, validate-csv. If the validation is successful (i.e. all the fields in the CSV are valid), the function returns normally. If any invalid field is present, an error will be signalled (using the validator functions defined above).

This version of the validator doesn’t contain any restarts though.

(defn validate-field

"Takes a header name and a string value as arguments; checks the

validity of the value by calling the appropriate validator function."

[header value]

(if-let [f (validators header)]

(f value)

(condition :invalid-header {:header header})))

(defn validate-csv [file]

(let [[headers & rows :as all-rows] (parse-csv-file file)]

(map (fn [line-number row]

(if (not= (count row) (count headers))

(condition :wrong-field-count {:line-number line-number}

(error "Number of fields doesn't equal number of headers."))

(manage [any? (fall-through #(assoc % :line-number line-number))]

(mapv validate-field headers row))))

(range 2 (count all-rows))

rows)))

Unhandled conditions

If the condition is not handled, the default action will be called. For instance this code, will by default raise a normal exception much like if the following code were there in its place instead , meaning that this system is fully compatible with the standard Java exceptions used by Clojure.

(try

(condition :the-condition 'the-data (error "The condition happened"))

;; if not handled, equivalent to:

(throw (ex-info "The condition happened"

{:condition :the-condition :value 'the-data}))

(catch Exception e e))

Putting restarts in place

There are a few actions we can take once an “invalid” field has been detected (i.e. a condition is signalled), e.g. we can abort the validation, we can continue validation on the next row, or we continue validation with the remaining fields in the same row (to name just a few).

To enable restarts, we just wrap the value passed to the condition in (restarts ...), which attaches the necessary information needed to perform the restart to the condition.

(defn validate-csv [file]

(let [[headers & rows :as all-rows] (parse-csv-file file)]

(map (fn [line-number row]

(manage [:continue-next-row (result! nil)]

(if (not= (count row) (count headers))

(condition :wrong-field-count

(restarts {:line-number line-number})

(error "Number of fields doesn't equal number of headers."))

(manage [any? (fall-through #(assoc % :line-number line-number))]

(manage [:continue-next-field (result! nil)]

(mapv validate-field headers row))))))

(range 2 (count all-rows))

rows)))

We'll also make the :invalid-header condition restartable.

(defn validate-field [header value]

(if-let [f (validators header)]

(f value)

(condition :invalid-header (restarts {:header header}))))

Time for some fun now. Pass an invalid file to the validator, and what do we see? Our two restart handlers are visible in the exception: :continue-next-field, and :continue-next-row.

(comment

(validate-csv "tutorial.csv"))

;; ExceptionInfo:

URL invalid

{:condition :url-invalid,

:value

{:data {:url "gopher://untether.ai", :line-number 3},

:handlers

[{}

{:continue-next-row #function[clojure.lang.AFunction/1]}

{#function[clojure.core/any?] #function[conditions.handlers/fall-through/fn--15590]}

{:continue-next-field #function[clojure.lang.AFunction/1]}],

:condition :url-invalid,

:message "URL invalid"}}

We'll see how to use the restarts in the next section.

Retrying the whole block

We’ll add one more restart now: this will allow us to revalidate the whole file if an error is signalled. retry! is a special handler since in an immutable language you usually need to be able to provide some update to the data in order to effectively retry. Here we use retryable and add the extra argument [file] which tells us that when we call (retry! file) the file argument of the retryable body should be set to the new value provided. In this case, however, we are relying on the file itself being changed before the upstream handler retries, so the retry is performed without modification to the arguments.

(defn validate-csv [file]

(retryable [file] [:retry-file (retry! file)]

(let [[headers & rows :as all-rows] (parse-csv-file file)]

(doall

(map (fn [line-number row]

(manage [:continue-next-row (result! nil)]

(if (not= (count row) (count headers))

(condition :wrong-field-count

(restarts {:line-number line-number})

(error "Number of fields doesn't equal number of headers."))

(manage [any? (fall-through #(assoc %

:line-number line-number))]

(manage [:continue-next-field (result! nil)]

(mapv validate-field headers row))))))

(range 2 (count all-rows))

rows)))))

Now what happens if we pass an invalid file to validate-csv? We get the :retry-file handler in the exception. This means that we can fix the problematic field, save the file, and start the validation all over again, without having restarted the overarching process, even if the handler is far up the call stack.

Managing restarts

To activate a restart, we can use restart or restart-any, the latter allowing an ordered list of restarts, where it will use the first one present.

For example, the following function will continue validating the file as long as conditions that it can handle are signalled and one of :continue-next-field or :continue-next-row restarts are available. It collects those errors in a list and returns it.

(defn list-csv-errors [file]

(let [result (atom [])]

(manage [any? (restart-any :continue-next-field :continue-next-row)]

(manage [any? (fall-through

^:restart

(fn [error]

(swap! result conj (assoc (:data error)

:condition (:condition error)

:message (:message error)))

error))]

(validate-csv file)))

@result))

(list-csv-errors "tutorial.csv")

If we want a non-programmer to use the validator, we can provide a way to upload the CSV file and give a nicely formatted output of list-csv-errors in the browser.

Conclusion

If we wanted list-csv-errors to list only one error per each row, that change would have been trivial, thanks to the restarts we have provided. This separation of logic, IMHO, makes it a very elegant tool in dealing with problems like these.

What I really like about the condition system is how it allows one to defer decisions to higher-level functions. The low-level functions provide different ways to move forward in case of exceptions (this is what validate-csv does), while the higher-level functions actually get to decide what path to take (like list-csv-errors).

Testing this code

The test data and test definition are below.

(deftest correct-error-list

(is (= [{:url "gopher://untether.ai",:line-number 3

:condition :url-invalid

:message "URL invalid"}

{:rating 'five,:line-number 4

:condition :invalid-rating

:message "Rating is not an integer in range"}

{:line-number 5

:condition :wrong-field-count

:message "Number of fields doesn't equal number of headers."}]

(list-csv-errors "tutorial.csv"))))

(correct-error-list)

"If we got here, success, the test passed!"

{:deps {org.clojure/clojure {:mvn/version "1.10.1"}

;; complient is used for autocompletion

;; add your libs here

com.xn--lgc/pure-conditioning {:mvn/version "0.1.1"}

compliment/compliment {:mvn/version "0.3.9"}}}