With Common Lisp Contexts

Common Lisp programmers may write many with-something overt their careers; the language specification itself is ripe with such constructs: witness with-open-file. Many other libraries also introduce a slew of with- macros dealing with this or that case.

So, if this is the case, what prevents Common Lisp programmers from coming up with a generalized with macro?

It appears that the question has been answered, rather satisfactorily, in Python and Julia (at least). Python offers the with statement, alongside a library of "contexts" (Python introduced the with statement in 2005 with PEP 343) and Julia offers its do blocks.

The present library, WITH-CONTEXTS, is a Common Lisp answer to the question. The library is patterned after the ideas embodied in the Python solution, but with several (common) "lispy" twists.

Here is the standard - underwhelming - example:

      (with f = (open "foo.bar") do
         (do-something-with f))

That's it as far as syntax is concerned (the 'var =' being optional, obviously not in this example; the syntax was chosen to be loop-like, instead of using Python's as keyword). Things become more interesting when you look under the hood.

Traditional Common Lisp with- macros expand in variations of unwind-protect or handle-case (and friends). The example above, if written with with-open-file would probably expand into something like the following:

      (let ((f nil))
         (unwind-protect
              (progn
                 (setq f (open "foo.bar"))
                 (do-something-with f))
            (when f (close f))))

Python generalizes this scheme by introducing a enter/exit protocol that is invoked by the with statement. Please refer to the Python documentation on contexts and their __enter__ and __exit__ methods.

The "WITH" Macro in Common Lisp: Contexts and Protocol

In order to introduce a with macro in Common Lisp that mimicked what Python programmers expect and what Common Lisp programmers are used to some twists are necessary. To achieve this goal, a protocol of three generic functions is provided alongside a library of contexts.

The ENTER/HANDLE/EXIT Context Protocol

The WITH-CONTEXTS library provides three generic functions that are called at different times within the code resulting from the expansion of the onvocation of the with macro.

enter: this generic function is invoked when the with macro "enters" the context; its main argument is the result of the expression that is the argument of the with macro.
handle: this generic function is called to take care of exceptional situations that may arise during the call to enter or during the execution of the body of the with macro.
exit: this generic function is called to "clean up" before exiting the context entered by means of the with macro.

Given the protocol (from now on referred to as the "EHE-C protocol"), the (undewhelming) "open file" example expands in the following:

      (let ((f nil))
	(unwind-protect
	    (progn
	      (setq f (enter (open "contexts.lisp")))
	      (handler-case (open-stream-p f)
		(error (#:ctcx-err-e-41883)
		  (handle f #:ctcx-err-e-41883))))
	  (exit f)))

Apart from the gensymmed variable the expansion is pretty straightforward. The function enter is called on the newly opened stream (and is essentially an identity function) and sets the variable. If some error happens while the body of the macro is executing then control is passed to the handle function (which, in its most basic form just re-signals the condition). Finally, the unwind-protect has a chance to clean up by calling exit (which, when passed an open stream, just closes it).

One unexpected behavior for Common Lisp programmers is that the variable (f in the case above) escapes the with constructs. This is in line with what Python does, and it may have its uses. The file opening example thus has the following behavior:

    CL-prompt > (with f = (open "contexts.lisp") do
                    (open-stream-p f))
    T

    CL-prompt > (open-stream-p f)
    NIL

To ensure that this behavior is reflected in the implementation, the actual macroexpansion of the with call becomes the following.

      (let ((#:ctxt-esc-val-41882 nil))
        (multiple-value-prog1
	    (let ((f nil))
	      (unwind-protect
		  (progn
		    (setq f (enter (open "contexts.lisp")))
		    (handler-case
			(open-stream-p f)
		      (error (#:ctcx-err-e-41883)
			(handle f #:ctcx-err-e-41883))))
		(multiple-value-prog1
		    (exit f)
		  (setf #:ctxt-esc-val-41882 f))))
	  (setf f #:ctxt-esc-val-41882)))

This "feature" will help in - possibly - porting some Python code to Common Lisp.

"Contexts"

Python attaches to the with statement the notion of contexts. In Common Lisp, far as the with macro is concerned, anything that is passed as the expression to it, must respect the enter/handle/exit. protocol. The three generic functions enter, handle, exit, have simple defaults that essentially let everything "pass through", but specialized context classes have been defined that parallel the Python context library classes.

First of all, the current library defines the EHE-C protocol for streams. This is the strightforward way to obtain the desired behavior for opening and closing files as with with-open-file.

Next, the library defines the following "contexts" (as Python does).

null-context:
this is a full "pass through" context, just encapsulating the expression passed to it.
managed-resource-context:
this is a first cut implementation of a "managed resource", which implements also an acquire/release protocol; of course, this would become more useful in presence of mutltiprocessing (see notes in Limitations).
redirect-context:
this is a context that redirects output to a different stream.
suppress-context:
this is a context that selectively handles some conditions, while ignoring other ones.
exit-stack-context:
this is a context that essentially allows a programmer to manipulate the "state of the computation" within it body and combine other "contexts"; to achieve its design goal, it leverages the functions of a protocol comprising the enter-context, push-context, callback, pop-all and unwind (this is equivalent to the Python close() context method).

This should be a good enough base to start working with contexts in Common Lisp. It is unclear whether the Python decorator interface would provide some extra functionality in this Common Lisp implementation of contexts and the with macro.

Limitations

The current implementation has a semantics that is obviously not the same as the corresponding Python one, but it is hoped that it still provided useful functionality. There are some obvious limitations that should be taken into account.

The current implementation of the library does not take into consideration threading issues. It could, by providing a locking-context based on a portable multiprocessing API (e.g., bordeaux-threads).

The Python implementation of contexts relies heavily on the yield statement. Again, the current implementation does not provide similar functionality, although it could possibly be implemented using a delimited continuation library (e.g., cl-cont).

Disclaimer

The code associated to these documents is not completely tested and it is bound to contain errors and omissions. This documentation may contain errors and omissions as well. Moreover, some design choices are recognized as sub-optimal and may change in the future.

License

The file COPYING that accompanies the library contains a Berkeley-style license. You are advised to use the code at your own risk. No warranty whatsoever is provided, the author will not be held responsible for any effect generated by your use of the library, and you can put here the scariest extra disclaimer you can think of.

Repository and Downloads

The with-contexts library is available on Quicklisp (not yet).

The with-contexts library. is hosted at common-lisp.net.

The git repository can be gotten from the common-lisp.net Gitlab instance in the with-macro project page.