Cluck and Crow
Cluck implements a generalised web gateway interface that enables a Common Lisp CGI script to run as a persistent process whilst communicating transparently with a web server using the SCGI protocol, in a similar way to Perl's SCGI module. It does this by redirecting standard input and output through a socket stream. It also provides an interface in the style of WSGI, Rack, PSGI/Plack and Clack but without sacrificing the underlying compatibility with CGI.
Crow is a micro web-framework which, together with supporting packages, provides a suite of macros which generate HTML 5 elements, access form parameters and implement some Common Lisp language and database utilities.
The project consists of four package files:
Here is a simple example which gives a flavour of how it can be used:
(let ((n 0)) (with-gi "scgi" 8080 (setf n (+ n 1)) (with-env-params (format t (strcat ;; Print HTTP headers... "Status: 200 OK~%" "Content-Type: text/html; charset=ISO-8859-1~%~%" ;; Print number of times called... (<P> () (mkstr "Called: " n " times")) (<HR>) ;; Print env vars... (<P> () "Env: " (princ-to-string env )) (<HR>) ;; Print parameters... (<P> () "Params: " (princ-to-string params )) (<HR>))))))
This project has several objectives, the main ones being:
- A Common Lisp implementation of the SCGI protocol which enables a CGI script to run as a persistent process in a similar way to Perl's SCGI module.
- A Common Lisp interface-agnostic middleware component serving both CGI and SCGI requests. This is influenced by, but does not mimic, WSGI, Rack, PSGI/Plack and Clack.
- A Common Lisp micro web-framework, compatible with a standard Apache HTTP web server.
I have been guided by the following ideals, although do not claim to have necessarily fulfilled them:
- Minimalism - include only essential functionality;
- Simplicity - only as complex as required by the minimalist requirements;
- Elegance - code should be well structured, e.g. loosely coupled and well factored.
On a personal note, this project has been a journey into Common Lisp as well as an opportunity to give something back to the open source community.
An important consideration, which has determined a number of technical decisions, has been my desire to maintain compatibility with the original CGI specification and it's implementation in commonly used web servers and operating systems. Unlike the requirements in the WSGI and PSGI specifications, a Cluck application, like a CGI script, can communicate with the web server through meta variables implemented by operating system environment variables and,if present, a request message-body accessible through the standard input file handle. I say can because it is equally possible to build a Plack/Clack-like layer that allows the web application to interface with the web server through the parameter list of an application function as demonstrated below.
A note on the names Cluck and Crow: I have always found naming things one of the more challenging aspects of programming and this project has been no exception. The name Cluck is onomatopoeicly derived from the word Clack, the quieter noise reflecting its more lowly ambitions. Crow builds on Cluck, the louder noise reflecting its higher position up the application stack.
The following lines should be uncommented in your Apache httpd.conf file:
# Uncomment these lines... LoadModule proxy_module modules/mod_proxy.so LoadModule proxy_scgi_module modules/mod_proxy_scgi.so
In addition, add the following lines, or similar depending on your requirements:
ProxyPass /scgi/demo scgi://localhost:8080/ <Proxy *> Require ip 127.0.0.1 </Proxy>
The simplest way of running the examples below is to place the module package and example code into the same folder or directory. For the CGI examples, the files must be placed in the cgi-bin folder. For the SCGI examples, the files can be placed in any folder. Note that mod_proxy_scgi does not start the SCGI application automatically - it must be started independently before any requests are sent to the web server. The easiest way to do this is to load the file containing the application or example into the Lisp REPL which will cause it to connect to a port and start waiting on requests.
Packages, Modules and Forms
Each file implements a single package (name space) as well as a single module. The following diagram shows the dependencies between the packages:
Implements the SCGI protocol as well as the generalised gateway interface.
The principal forms are:
with-scgi - SCGI handler with-gi - generalised gateway interface cluck - inspired by WSGI, Rack, PSGI/Plack and Clack.
A micro web-framework which provides a set of macro forms for generating HTML 5 elements, as well as higher level utility forms. The elements are generated in a generic manner using macros which define macros.
Implements low-level CGI functionality, including the retrieval of GET and POST parameters via application/x-www-form-urlencoded and multipart/form-data content types. The principal form exported by utiles-cgi.lisp is:
Basic language-level utility forms for handling strings, lists, arrays, io, regular expressions etc.
Forms for handling database connectivity based on plain-odbc. This is a very simple interface, the sole purpose of which is to execute stored procedures in a database-agnostic fashion. The motivation for focusing on stored procedures is a) a desire to avoid the direct use of SQL in applications and b) a desire to avoid the added complexity of database middleware.
Currently, only Postgres plpgsql and SQL Server T-SQL are supported. These forms are still very much experimental,
with-env-params is an anaphoric macro which introduces the following variables:
- env: contains an association list of the CGI meta variables;
- params: contains an association list of the parameters and values from the query string or the message request body.
Note that each value in params is itself a list of one or two values:
- one value for a normal parameter
- two values for a file parameter:
- the first value is the name of the file;
- the second is the file contents.
An example of such a parameter list might be:
(("page" "select_file") ("file_1" "file_name_" "file content..."))
Note that files can contain binary or character data.
The following example demonstrates the use of with-env-params with with plain CGI:
#! C:\Program Files\clisp-2.49\clisp.exe -K full --quiet ;; CGI script demonstrating use of utils-cgi:with-env-param (require :utils-cgi) (import '( utils-cgi:with-env-params utils-cgi:env utils-cgi:params)) (with-env-params ;; Print HTTP headers... (format t "Status: 200 OK~%") (format t "Content-Type: text/html~%~%") ;; Print env vars... (format t "<P> Env: ~A </P>~%" (princ-to-string env)) ;; Print parameters... (format t "<P> Params: ~A </P>~%" (princ-to-string params)))
HTML Element Generation
The full list of HTML 5 elements is supported. This is achieved through two types of macro, both of which generate HTML elements as strings. Void elements are generated by a macro form the name of which is the element tag name enclosed in angle brackets with no spaces, followed by pairs of attribute names and values as follows:
( <tag> "attribute 1" "value 1" "attribute 2" "value 2" ...)
( <input> "type" "submit" "name" "submit_button" "value" "submit file" )
Non-void elements are generated by a macro form, the naming of which folows that for void elements, followed by a list of attribute/value pairs, followed by the element contents, as follows:
( <tag> ("attribute 1" "value 1" "attribute 2" "value 2") "some content" "more content" ... )
( <P> ("class" "some_class") "This is" " a paragraph" )
The html macros are named with angle brackets surrounding the tag name, and not with the bare tag name, because of the conflict between a small number of tag names and reserved words in Common Lisp , specifically map and time.
The previous example of with-env-params can be rewritten as follows:
#! C:\Program Files\clisp-2.49\clisp.exe -K full --quiet ;; CGI script demonstrating; ;; - use of with-env-params ;; - HTML-generating macros <P>, <HR> ;; - CGI utility macros: send-params, http-status and http-conent-type (require :utils-lang) (require :utils-cgi) (require :crow) (import '( utils-lang:nl utils-cgi:with-env-params utils-cgi:env utils-cgi:params utils-cgi:http-status utils-cgi:http-content-type utils-cgi:send-response crow:<p> crow:<hr> )) (with-env-params (send-response ;; Print HTTP headers... (http-status 200) (nl) (http-content-type 'html) (nl) (nl) ;; Print env vars... (<P> () "Env: " (princ-to-string env )) (<HR>) ;; Print parameters... (<P> () "Params: " (princ-to-string params )) (<HR>)))
Note some additional macro forms have been introduced above:
send-responseconcatenates its arguments and writes them to standard output;
http-statustales a integer denoting an HTTP response status and returns an HTTP status header string;
http-content-typetakes a symbol, either 'text or 'html, and generates the appropriate HTTP header.
A fundamental feature of Cluck is that the SCGI layer, implemented by the form with-scgi, can be used as a wrapper around existing CGI code without need for modification in any way. Assuming the web server has been correctly configured for SCGI, it is used as follows:
;; SCGI script demonstrating use of with-scgi (require :utils-lang) (require :utils-cgi) (require :crow) (require :cluck) (import '( utils-lang:nl utils-lang:mkstr utils-cgi:with-env-params utils-cgi:env utils-cgi:params utils-cgi:http-status utils-cgi:http-content-type utils-cgi:send-response crow:<p> crow:<hr> cluck:with-scgi )) (let ((n 0)) (with-scgi 8080 (setf n (+ n 1)) (with-env-params (send-response ;; Print HTTP headers... (http-status 200) (nl) (http-content-type 'html) (nl) (nl) ;; Print number of times called... ( <P> () (mkstr "Called: " n " times")) (<HR>) ;; Print env vars... (<P> () "Env: " (princ-to-string env )) (<HR>) ;; Print parameters... (<P> () "Params: " (princ-to-string params )) (<HR>)))))
Note that, in the above example, the macro form with-scgi binds the special variables *standard-input* and *standard-output* to port 8080. Therefore, within the form's scope, these file handles can be used to receive data from, and send data to, the web server in the manner of a traditional CGI script.
The form with-gi abstracts away from the specific interface and allows for handlers to be implemented by defining a form prefixed with the name with-. The form with-scgi is such a handler which is invoked from the form with-gi as follows:
;; SCGI script demonstrating use of with-gi (require :utils-lang) (require :utils-cgi) (require :crow) (require :cluck) (import '( utils-lang:nl utils-lang:mkstr utils-cgi:with-env-params utils-cgi:http-status utils-cgi:http-content-type utils-cgi:send-response crow:<p> cluck:with-gi cluck:with-scgi )) (let ((n 0)) (with-gi "scgi" 8080 (setf n (+ n 1)) (with-env-params (send-response ;; Print HTTP headers... (http-status 200) (nl) (http-content-type 'html) (nl) (nl) ;; Print number of times called... ( <P> () (mkstr "Called: " n " times"))))))
Note the following:
- The symbol cluck:with-scgi has to be imported into the package where with-gi is used;
- A CGI handler form, with-cgi, is also provided. It effectively does nothing more than allow the same programme structure to be used for both CGI and SCGI applications, meaning that the programme is interface-agnostic;
- Additional handlers could be written, e.g. for FastCGI.
The form cluck implements a layer on top of the forms with-gi and with-env-params and permits a web application to be structured in a similar way to a Clack application. Here is a simple example:
;; CGI script demonstrating use of cluck (require :utils-lang) (require :utils-cgi) (require :crow) (require :cluck) (import '( utils-lang:mkstr utils-cgi:http-status utils-cgi:http-content-type crow:<p> crow:<hr> cluck:with-scgi cluck:cluck )) (defun demo-app ( state env params ) ;; Increment the global counter... (if (null (gethash 'count state)) (setf (gethash 'count state) 0) (setf (gethash 'count state) (+ 1 (gethash'count state)))) ;; Return a list whose first two members are http status and content type. ;; The remaining members comprise the response body. (list (http-status 200) (http-content-type 'html) ;; Print number of times called... ( <P> () (mkstr "Called: " (gethash 'count state) " times")) (<HR>) ;; Print env vars... (<P> () "Env: " (princ-to-string env )) (<HR>) ;; Print parameters... (<P> () "Params: " (princ-to-string params )) (<HR>))) (cluck #'demo-app "scgi" 8080)
Note the following:
- The call to cluck results in the function demo-app being called with the arguments:
- state: a hash that can be used to maintain state between requests - in the above example it is used to hold a counter of the number of times the application has been called;
- env: an association list of CGI meta-variables;
- params: an association list of parameters parsed from the query string and/or the request message-body;
- The contents of the hash variable state are changed in the calling environment as a consequence of Common Lisp's mutable data types and the fact that, although Common Lisp function arguments are passed by value, for compound data types such as lists and hashes the places that they contain are effectively passed by reference;
- It is necessary for with-scgi to be imported as the call to cluck gets replaced with this.
The principal forms are:
(db-connect "dsn" "server" "port" "database" "username" "password")
db-connect connects to a database using an ODBC dsn. The username and password parameters are optional for SQL Server drivers. If not supplied then a trusted connection is made.
(db-proc-to-list dbcon "stored_proc_name")
dbcon is the connection object returned by db-connect. It assumes the stored procedure does not write to the database and therefore no commit is done. It returns a list of rows each of which is a list of column values.
(db-proc dbcon "stored_proc_name")
Assumes that the the stored procedure does write to the database and therefore does a commit after executing the procedure. It returns the first value returned by the procedure. This could be a rowcount for example, or -1 in case of an exception, error, rollback etc.
The project has been implemented using the following systems:
There are several areas in the code that are likely to give rise to portability issues.
- Use of CLISP's ext packge:
The setf-able property of function stream-element-type is exploited to convert standard input to UNSIGNED-BYTE 8:
(setf (stream-element-type *standard-input*) '(UNSIGNED-BYTE 8) )
This is a useful feature of CLISP (see Implementation Notes for GNU CLISP on stream-element-type and socket streams) which has been used to enable the reading of binary files in the message request body.
HTML documents are currently encoded with charset iso-8859-1. I need to find a portable way of converting a binary array into a string using the charset specified in Content-Type http header. The code currently uses code-char which uses the 'default' charset but can't it's not clear to me how CLISP chooses a default charset. I suspect babel might be a portable way to do this.
The following are some of the areas I would like to develop or add support for in future:
- Support for utf-8
- Additional error handling and increased robustness;
- Increased portability across other Common Lisp platforms;
- Better support for database access, including support for other databases such as MySQL and Oracle;
- Support for FastCGI;
- Port to Arc and/or Clojure;
- Addition of asynchronous handling of requests, e.g. using cl-async;
- Session handling
- LDAP Authentication
I would like to thank all those who took the time and effort to respond to my questions on the CLISP mailing list, including Sam Steingold and Michael Kappert who provided invaluable help and advice with REGEXP and PCRE issues (in early versions - I now use CL-PPCRE), as well as Raymond Toy and Pascal J. Bourguignon for their suggestions regarding reading of character and binary data from socket streams. Thanks to Stephen Shorrock for trying out the examples above and for his comments and feedback.
I have endeavoured to reference sources of information, ideas and code within the code itself. However, I would like to mention the following two sources which I found particularly useful: