With Common Lisp Contexts

Common Lisp programmers may write many with-something overt their careers; the language specification itself is ripe with such constructs: witness with-open-file. Many other libraries also introduce a slew of with- macros dealing with this or that case.

So, if this is the case, what prevents Common Lisp programmers from coming up with a generalized with macro?

It appears that the question has been answered, rather satisfactorily, in Python and Julia (at least). Python offers the with statement, alongside a library of "contexts" (Python introduced the with statement in 2005 with PEP 343) and Julia offers its do blocks.

The present library, WITH-CONTEXTS, is a Common Lisp answer to the question. The library is patterned after the ideas embodied in the Python solution, but with several (common) "lispy" twists.

Here is the standard - underwhelming - example:

      (with f = (open "foo.bar") do
         (do-something-with f))
    
That's it as far as syntax is concerned (the 'var =' being optional, obviously not in this example; the syntax was chosen to be loop-like, instead of using Python's as keyword). Things become more interesting when you look under the hood.

Traditional Common Lisp with- macros expand in variations of unwind-protect or handle-case (and friends). The example above, if written with with-open-file would probably expand into something like the following:

      (let ((f nil))
         (unwind-protect
              (progn
                 (setq f (open "foo.bar"))
                 (do-something-with f))
            (when f (close f))))
    

Python generalizes this scheme by introducing a enter/exit protocol that is invoked by the with statement. Please refer to the Python documentation on contexts and their __enter__ and __exit__ methods.

The "WITH" Macro in Common Lisp: Contexts and Protocol

In order to introduce a with macro in Common Lisp that mimicked what Python programmers expect and what Common Lisp programmers are used to some twists are necessary. To achieve this goal, a protocol of three generic functions is provided alongside a library of contexts.

The ENTER/HANDLE/EXIT Context Protocol

The WITH-CONTEXTS library provides three generic functions that are called at different times within the code resulting from the expansion of the onvocation of the with macro.

Given the protocol (from now on referred to as the "EHE-C protocol"), the (undewhelming) "open file" example expands in the following:

      (let ((f nil))
	(unwind-protect
	    (progn
	      (setq f (enter (open "contexts.lisp")))
	      (handler-case (open-stream-p f)
		(error (#:ctcx-err-e-41883)
		  (handle f #:ctcx-err-e-41883))))
	  (exit f)))
    

Apart from the gensymmed variable the expansion is pretty straightforward. The function enter is called on the newly opened stream (and is essentially an identity function) and sets the variable. If some error happens while the body of the macro is executing then control is passed to the handle function (which, in its most basic form just re-signals the condition). Finally, the unwind-protect has a chance to clean up by calling exit (which, when passed an open stream, just closes it).

One unexpected behavior for Common Lisp programmers is that the variable (f in the case above) escapes the with constructs. This is in line with what Python does, and it may have its uses. The file opening example thus has the following behavior:

    CL-prompt > (with f = (open "contexts.lisp") do
                    (open-stream-p f))
    T

    CL-prompt > (open-stream-p f)
    NIL
  

To ensure that this behavior is reflected in the implementation, the actual macroexpansion of the with call becomes the following.

      (let ((#:ctxt-esc-val-41882 nil))
        (multiple-value-prog1
	    (let ((f nil))
	      (unwind-protect
		  (progn
		    (setq f (enter (open "contexts.lisp")))
		    (handler-case
			(open-stream-p f)
		      (error (#:ctcx-err-e-41883)
			(handle f #:ctcx-err-e-41883))))
		(multiple-value-prog1
		    (exit f)
		  (setf #:ctxt-esc-val-41882 f))))
	  (setf f #:ctxt-esc-val-41882)))
    

This "feature" will help in - possibly - porting some Python code to Common Lisp.

"Contexts"

Python attaches to the with statement the notion of contexts. In Common Lisp, far as the with macro is concerned, anything that is passed as the expression to it, must respect the enter/handle/exit. protocol. The three generic functions enter, handle, exit, have simple defaults that essentially let everything "pass through", but specialized context classes have been defined that parallel the Python context library classes.

First of all, the current library defines the EHE-C protocol for streams. This is the strightforward way to obtain the desired behavior for opening and closing files as with with-open-file.

Next, the library defines the following "contexts" (as Python does).

This should be a good enough base to start working with contexts in Common Lisp. It is unclear whether the Python decorator interface would provide some extra functionality in this Common Lisp implementation of contexts and the with macro.

Limitations

The current implementation has a semantics that is obviously not the same as the corresponding Python one, but it is hoped that it still provided useful functionality. There are some obvious limitations that should be taken into account.

The current implementation of the library does not take into consideration threading issues. It could, by providing a locking-context based on a portable multiprocessing API (e.g., bordeaux-threads).

The Python implementation of contexts relies heavily on the yield statement. Again, the current implementation does not provide similar functionality, although it could possibly be implemented using a delimited continuation library (e.g., cl-cont).

Disclaimer

The code associated to these documents is not completely tested and it is bound to contain errors and omissions. This documentation may contain errors and omissions as well. Moreover, some design choices are recognized as sub-optimal and may change in the future.

License

The file COPYING that accompanies the library contains a Berkeley-style license. You are advised to use the code at your own risk. No warranty whatsoever is provided, the author will not be held responsible for any effect generated by your use of the library, and you can put here the scariest extra disclaimer you can think of.

Repository and Downloads

The with-contexts library is available on Quicklisp (not yet).

The with-contexts library. is hosted at common-lisp.net.

The git repository can be gotten from the common-lisp.net Gitlab instance in the with-macro project page.