I had originally included a section on macros in my introductory Enlive tutorial but in the end thought better of it. My Enlive tutorial turned out to be far more popular than I had imagined it would be. I think the notion of keeping your HTML and your code completely separate struck a nerve with web developers. Now that the dust has settled I realize there is no better introduction to Lisp macros then the topic of HTML templating and I’m writing this more advanced tutorial to explain Lisp macros to the uninitiated as well illustrate how you can use the features of Enlive to build higher level HTML templating abstractions.
I strongly recommend that you go through the first six tutorials if you aren’t already familiar with Clojure. If you are familiar with Clojure you can skim this one as well.
Coding web pages is largely about taking one textual representation and transforming it into another based on some data you are given. Every web developer knows this. Well, it turns out that many Lisps, including Clojure, allow you to do the very same thing with Lisp code itself.
So while Enlive gives you the full power of Clojure to generate your HTML, Clojure gives you the full power of Clojure to generate your code! It’s one of those features of Lisp that is often talked about but rarely well described. This why I think HTML templating is a great introduction to Lisp macros, if you’re a web developer, HTML templating is an activity where the concept of transforming structures is already internalized. If you understand HTML templating you can understand Lisp macros.
Imagine that you’re working with Enlive and you find yourself writing the following snippet:
(defsnippet book-widget “resources/widgets.html” [:.book]
[{:keys [title author publisher date edition ISBN] :as ctxt}]
[:.title] (content title)
[:.author] (content author)
[:.publisher] (content publisher)
[:.date] (content date)
[:.edition] (content edition)
[:.ISBN] (content ISBN))
In my experience most programmers consider this standard fare- an unavoidable bit of tediousness. However a Lisp programmer looks at this and thinks “I know I’m going to be writing stuff like this all the time, I want this to be generated automatically for me.”
What is the pattern here that can be abstracted away? We have a HTML file of widgets handed to us by the designer. The first CSS class name of all the elements in that file can be used for templating in values. If there’s a lot of widgets in that file you’re going to have a lot of widgets in your code that look exactly the same as the above. There’s no reason to type this over and over again. We have a real pattern that can be abstracted away. I am going to show you how to do this.
By the end of this tutorial you will be able to write the following:
(quick-template book-widget "resources/widgets.html" [:.book])
And it will automatically generate the verbose code above.
Now if your favorite programming language is not a competent Lisp there’s not much you can do except approach the templating library authors and ask them to make an enhancement. What would the requested enhancement be?
“Could you pretty please make it so that I can quickly generate a template from the CSS classes that are already in markup?”
Imagine taking this kind of request to the Django templating library maintainers, they probably wouldn’t take you too seriously and honestly, rightly so. In languages which don’t support macros often the approach is to design a Domain Specific Langauge to tackle the problem at hand. The problem with DSLs is that they are generally are even more restrictive they the language they were written in. In Lisp writing a DSL is just writing some macros. The beauty of macros is that you can write macros that use those macros. At no point is there is a decrease in the amount of expressive power even as your syntax becomes simpler and more domain specific.
The beauty of Lisp is that when we are confronted with the wrong level of expression we can just write a macro and we don’t need to wait around for the library maintainers. And the best part is that you don’t need to understand their implementation to write a macro. You can just program to the agreed upon interface.
This is powerful stuff.
Before we get into macros let’s run the following code. Start up a REPL with “lein repl” in the repository:
user=> (load "tutorial/books") nil user=> (in-ns 'tutorial.books) nil tutorial.books=> (start-app) ... output ...
Point your favorite browser at http://localhost:8080/. You should see a list books. If you remember the previous tutorial the following should shock you. This is how much templating code we have:
(m/quick-snippet book "resources/widgets.html" [:.book]) (html/deftemplate index "resources/index.html" [ctxt] [:.books] (html/content (map book (:books ctxt))))
Intrigued yet? :)
Let’s start up the Clojure REPL in the usual way:
user=> (load “tutorial/syntax”) nil user=> (in-ns ’tutorial.syntax) nil
The first we need to understand is that when we template code, we are working with lists.
tutorial/syntax=> () ()
This is a list. But it’s also useful to think of it as being something very similar to, say, a DOM node ;)
tutorial/syntax=> (a b) () (a b) java.lang.Exception: Unable to resolve symbol: a in this context ...
We’ve tried to evaluate a list which includes two symbols which have never been defined. Thus the error.
tutorial/syntax=> ‘(a b) '(a b) (a b)
Interesting eh? No error. This is a quoted list. A quoted listed will not attempt to evaluate it’s contents. The following looks remarkably similar.
tutorial/syntax=> `(a b) ‘(a b) (tutorial.syntax/a tutorial.syntax/b)
Interesting. ` is the syntax quote. But what is this good for?
tutorial/syntax=> (let [a 1 b 2] `(~a ~b)) (let [a 1 b 2] `(~a ~b)) (1 2)
Say hello to your first Lisp template. Syntax quote allows us to splice in values with ~. Again if you’ve done some HTML templating this should not be so alien to you.
But this is not a macro. We’re just creating lists here. What we want is a way to transform a piece of code and replace it with the transformation. This is the whole point of defmacro. We’ll get to this in a moment. Let’s show another useful feature before we move on.
tutorial.syntax=> (let [f + v [1 2 3]] `(~f ~@v)) (let [f + v [1 2 3]] `(~f ~@v)) (#<core$_PLUS ...> 1 2 3)
Wow. ~@ lets us splice in the values of vector! When transform code this is a particularly useful operation (keep thinking about how this relates to HTML templating).
Okay it’s time to talk about defmacro. Be warned the following is completely contrived, it’s only for the purposes of demonstration. We’ll get to the real world macro soon. Before we look at the code let’s run the following:
tutorial.syntax=> (slow-add 1 2) (slow-add 1 2) "Elapsed time: 93.175 msecs" "Elapsed time: 87.948 msecs" "Elapsed time: 82.617 msecs" "Elapsed time: 95.717 msecs" "Elapsed time: 81.954 msecs" nil
Then run the following:
tutorial.syntax=> (fast-add 1 2) (fast-add 1 2) "Elapsed time: 51.211 msecs" "Elapsed time: 35.186 msecs" "Elapsed time: 32.346 msecs" "Elapsed time: 29.503 msecs" "Elapsed time: 38.929 msecs" nil
What’s the difference here? If we look at the code we see the following:
(defmacro int-add [a b]
`(+ (int ~a) (int ~b)))
(defn slow-add [a b]
(dotimes [_ 5]
(time
(dotimes [_ 5000000]
(+ a b)))))
(defn fast-add [a b]
(dotimes [_ 5]
(time
(dotimes [_ 5000000]
(int-add a b)))))
In order to get the best arithmetic performance out of the JVM, you can provide type-hints to the compiler:
(+ (int 4) (int 5))
With this extra bit of information the JVM can call the correct implementation of + and avoid losing time in reflection. Again this is not something you should concern yourself with, and cleraly the timing numbers above show that operations on boxed numbers is passableif you’re not doing serious number crunching.
Type the following into the REPL:
tutorial.syntax=> (macroexpand '(int-add 1 2)) (macroexpand-1 '(int-add 1 2)) (clojure.core/+ (clojure.core/int 1) (clojure.core/int 2))
macroexpand is a convenience that outpus what the code would expand to. Note that here we quote the code so that it doesn’t actually get compiled. We’ve converted code into dat. Just as an HTML template give us transformed markup, macroexpand applies the transformations and hands us back the transformed code.
If this data appeared as real code in our program the compiler would replace every occurence of int-add with it’s macroexpansion. If you’re familiar with C/C++ preprocessing you should be used to this idea. But if you conclude from this that Lisp macros really don’t add much beyond that you would be very wrong. Macro’s have full access the programming language. In the following section we will load the markup, analyze it with Enlive and use that information to generate an Enlive template.
Again this int-add example is a trivial, unrealistic, and fairly pointless example only for the purposes of illustration. Now let’s solve some real world problems.
TODO: defmacro a function to show issues around symbols
TODO: show gensym
Now I won’t lie, as simple as the previous parts were the following is going to be pretty rough going but I hope you stick with it. There is not much code involved in this transformation, but what’s there packs a punch.
First let’s see how this actually works. We’ve created a macro called quick-snippet which loads a analyzes a widget so it can produce the right code. With quick-snippet instead of writing out the whole template you can write the following:
(m/quick-snippet book "resources/widgets.html" [:.book]))
It’s important to be able to expand macros in order to debug them. Try the following at your REPL:
tutorial.books=> (pprint (macroexpand '(m/quick-snippet book "resources/widgets.html" [:.book]))) (def book (net.cgrand.enlive-html/snippet "resources/widgets.html" [:.book] [ctxt] [:span.title] (net.cgrand.enlive-html/content (:title ctxt)) [:span.author] (net.cgrand.enlive-html/content (:author ctxt)) [:span.publisher] (net.cgrand.enlive-html/content (:publisher ctxt)) [:span.date] (net.cgrand.enlive-html/content (:date ctxt)) [:span.edition] (net.cgrand.enlive-html/content (:edition ctxt)) [:span.ISBN] (net.cgrand.enlive-html/content (:ISBN ctxt)))) nil
Wow. Without changing a line of Enlive, here’s a whole new level of functionality on top of Enlive.
Lets dive into the code. Open the file macroology.clj. It’s a tiny file. This is all the code that it takes to augment Enlive.
(ns tutorial.macroology
(:use [tutorial.utils :only [render]])
(:require [net.cgrand.enlive-html :as html]
[clojure.contrib.str-utils2 :as s]))
(def *child-nodes-with-class* (html/selector [html/root [:* (html/attr? :class)]]))
(defn- selector-for-node [{tag :tag, attrs :attrs}]
(let [css-class (s/trim (first (s/split (:class attrs) #" ")))]
`([~(keyword (str (name tag) "." css-class))] (html/content (~(keyword css-class) ~'ctxt)))))
(defmacro quick-snippet [name rsrc selector]
(let [nodes (-> (html/html-resource (eval rsrc))
(html/select (eval `(html/selector ~selector)))
(html/select *child-nodes-with-class*))]
`(html/defsnippet ~name ~rsrc ~selector
[~'ctxt]
~@(reduce concat (map selector-for-node nodes)))))
Let’s break it down. First:
(defn- selector-for-node [{tag :tag, attrs :attrs}]
(let [css-class (s/trim (first (s/split (:class attrs) #" ")))]
`([~(keyword (str (name tag) "." css-class))] (html/content (~(keyword css-class) ~'ctxt)))))
This function takes a node and automatically generates the selector/function pair that Enlive templates and snippets use to transform HTML. The first big simply extracts the first CSS class. So this hints there will be a one to one correspondence between the keys in the map passed into the final snippet and first CSS class name of a node.
You should be familiar with syntax quote now. Note that we can run regular code before splice in the final value.
To get a better sense of what’s going on lets give this function content at the REPL:
tutorial.books=> (m/selector-for-node {:tag :div :attrs {:class "foo bar"}})
([:div.foo] (net.cgrand.enlive-html/content (:foo ctxt)))
It should be quite clear what this function does now. Given an HTML node it creates the matching selector/function pair based on the CSS class.
The next function is not so simple:
(defmacro quick-snippet [name rsrc selector]
(let [nodes (-> (html/html-resource (eval rsrc))
(html/select (eval `(html/selector ~selector)))
(html/select *child-nodes-with-class*))]
`(html/defsnippet ~name ~rsrc ~selector
[~'ctxt]
~@(reduce concat (map selector-for-node nodes)))))
The first thing we do is use eval. This is generally not necessary for macros and in fact is discouraged. However for what we mean to accomplish there is no other way that doesn’t generate more code than we’d like. We need information about the HTML file.
When a macro is executed you are not handed “values” you’re simply handle whatever symbols or expressions are present in the code. This is a common error to think you are being handed values. You are always transforming code not the values present in your program. But there are ways to get at those values if you really need them, eval is one way to get at them.
So we evaluate rsrc so that we can load an HTML resource. We then also need to evaluate the selector so that we can match only a specific part of the document. Finally we select only the child nodes of that part of the document that actually have CSS classes.
We then emit the defsnippet splicing in the name, rsrc, and selector unchanged. ~'ctxt is there so that the macro doesn’t tryto resolve this symbol, it’s jut a local var. Finally we take all the matching nodes, map over them with selector-for-node and splice in the selector/fn pairs.
This is definitely a doozy. And programming macros is not a simple task by any means.
But clearly the dividends are huge in this case. We now have a general way to automatically generate templates for many, many kinds of widgets with one line of code.