Exploring Data with Clojure and Clerk

by Kira McLean

This is the first of two tutorials which focus on downloading a dataset and using tools from Clojure’s data science ecosystem to visualise it.

Clojure Tutorial Image 1

At TPXimpact we help government agencies across the UK publish their data as 5 star open data. That means it’s published under an open licence in standardised, machine readable formats, and conforms to open standards as closely as possible in an attempt to join it to the broader web of linked data.

For these projects, the platform we use is mostly built with Clojure, which has gained popularity over the past decade as a general purpose programming language. It’s not as widely used yet in the data science community, but over the past couple of years several tools and libraries have been developed that make Clojure and its ecosystem an increasingly attractive choice for data scientists.

In this series of tutorials, we’ll learn how to use the PublishMyData platform to download a dataset and then use tools from Clojure’s emerging data science ecosystem to visualise it. In this first post of the series, we’ll download a dataset from PublishMyData and learn how to set up a Clojure programming environment and a new Clerk notebook. Clerk is a computational notebook for Clojure and will serve as the “container” for our data visualisation project. We’ll load our dataset to begin working with it, and then in the next and final post we’ll learn how to visualise the data.

Downloading a dataset from PublishMyData

All instances of PublishMyData host, at a minimum, a catalogue of datasets and a way to query them. For this project we’ll start with a dataset on the Integrated Data Service website (which is in beta at time of writing): “Annual mean temp with trends actual”. You can narrow down the list of datasets in the dataset catalogue by publisher or theme to find it (“Met Office” and “Climate Change”, respectively, in this case), or search for it by name.

Our ultimate goal is to make a simple plot showing how the average temperature in the UK has changed over time. From the dataset explorer, we have the option of downloading the entire dataset as tidy data (one observation per row), or we could apply filters and download the resulting subset of data. We are only interested in the data for all of the UK, so we’ll toggle “Filter data” on, then edit the “Geography” filter and select just “UK”, clicking “Apply” to filter the dataset. The filter bank above the data table should look like this:

Clojure Filter Bank Image 2

Now we've filtered, we only see rows which are for all of the UK. We can now click the “Download” button, leaving the toggle set to “Filtered data only”, and then click “Download zip” to retrieve the data. If this particular subset of data hasn’t been downloaded in a while, the button might say “Prepare download” instead. If that’s the case just click on it and the download will be prepared after a few seconds.

Clojure Tutorial Image 3

There should be two files in the downloaded zip: a data.csv file and a README.txt file. We’ll use the data.csv file for our project.

Setting up a Clojure Environment

Running a Clerk notebook locally requires a working Clojure environment. This requires installing both the Clojure programming language itself and an editor that is suited for Clojure development.

Clojure Programming Language

There are instructions on the Clojure website for installing Clojure itself on different operating systems. Clojure requires that Java also be installed. All Java LTS releases (currently Java 8, 11, and 17) are officially supported. Some installation methods will include installation of this dependency as well, others will require that Java be installed separately. See the instructions for your operating system to get Clojure installed on your machine.

Clojure IDE

Clojure developers typically program at a REPL. There are several editors well suited to Clojure development with learning curves of varying steepness. A relatively simple and free option that includes good REPL support is Visual Studio Code with the Calva plugin. If you already use a different editor of course feel free to stick with it and see what kind of Clojure support is available. Most major editors have good support for a Clojure REPL via plugins or packages.

Make a New Notebook

A Clerk notebook is just a Clojure namespace. Clerk interprets the code and comments in the namespace to render them as a web page. To make a new Clojure project with a blank Clerk notebook, we just need to create a new directory for our project and add two files — one to specify our project’s dependencies and another for the notebook itself.

Create a new directory for your project, then add a file called deps.edn and a directory called notebooks containing a single file for our visualisation project. I called mine annual_mean_temp_uk.clj. The name of this file is arbitrary but needs to end with the .clj (for “Clojure”) extension and use underscores to separate words (as opposed to hyphens, spaces, or any other delimiter). I’ve also added the dataset that we downloaded above inside a separate data directory (renamed to annual-mean-temp-uk.csv for clarity). The project structure should look like this:

visualisation-project

├── data

│   └── annual-mean-temp-uk.csv

├── deps.edn

└── notebooks

└── annual_mean_temp_uk.clj

The notebook file (annual_mean_temp_uk.clj) can remain blank for now. The deps.edn file is where we’ll specify Clerk as a dependency of our project. After updating this file to include Clerk as a dependency, it should look like this:

In deps.edn:

{:deps {io.github.nextjournal/clerk {:mvn/version "0.8.445"}}}

Note version 0.8.445 just happens to be the latest Clerk version at the time of this writing. To be sure the rest of the steps work as described use the same version of Clerk.

Start the Clojure REPL

Open this project directory in your editor of choice, and start a REPL. How you do that will depend on which Clojure IDE you’ve chosen to work with. If you’re using Visual Studio Code as described above, you’ll start a REPL by selecting the command palette option “Calva: Start a Project REPL and Connect (aka Jack-In)”. You can bring up the command palette in Visual Studio Code with ctrl+shift+P (or cmd+shift+P on a mac), then search for this command. If you’re asked to select a project type, choose deps.edn.

This should download the Clerk dependency we specified in our deps.edn file and start a REPL.

You might see several new hidden files appear in your project directory. These are called “dotfiles” (because they start with a dot), and are used by the editor and Clerk to track project state. They can be ignored for development purposes.

Set up a Clerk notebook

So far we have a deps.edn file with a single dependency (Clerk) specified in it, an empty visualisation namespace, and a REPL running and connected to this project. Next we’ll learn how to actually run a Clerk notebook.

As mentioned earlier, a Clerk notebook is just a Clojure namespace. To see what this means, try adding a basic namespace declaration and comment to the annual_mean_temp_uk.clj file:

In notebooks/annual_mean_temp_uk.clj:

(ns notebooks.annual-mean-temp-uk)

;; This is a Clerk notebook

Note the conventions for Clojure namespaces — the namespace name matches the name of the file, with the extension removed; underscores are converted to hyphens, and directory separators are converted to periods. These details are important so that the namespaces can be loaded properly. It’s also good practice for namespaces to include at least 2 segments, to avoid name collisions and potential problems with Java interop.

Now try starting the Clerk notebook. First enter this into the running REPL to require Clerk:

Evaluate in the running REPL:

(require '[nextjournal.clerk :as clerk])

Then start a local Clerk server and tell it to show your new notebook:

Evaluate in the running REPL:

(clerk/serve! {:watch-paths ["."] :browse? true})

(clerk/show! "notebooks/annual_mean_temp_uk.clj")

We’ve given two arguments to Clerk’s built-in server. The first is to instruct it to watch the root of our project for changes so that the page automatically reloads whenever changes are saved (watch-paths). The second is to tell clerk to open a new browser window for our notebook (browse).

You should see something that looks like this in your browser, assuming you copied the same placeholder text from above:

Clerk Notebook

Loading data into the Notebook

Clerk comes with several built-in viewers for data. In the next post in this series we’ll learn how to use the vega viewer to build visualisations. For now we can quickly inspect our data using Clerk’s built-in table viewer, which is where we’ll leave this tutorial for now.

To read our CSV file, we can use Clojure’s standard CSV library. We need to include it as a dependency, so our deps.edn file will look like this now:

In deps.edn:

{:deps {io.github.nextjournal/clerk {:mvn/version "0.8.445"}

org.clojure/data.csv {:mvn/version "1.0.1"}}}

To download the newly added library, restart the REPL.

We’ll require this CSV library in our namespace and use it to parse the contents of our file into a Clojure data structure. Then we can use Clerk’s built-in table viewer to render our data as a simple table. All that should result in an annual-mean-temp-uk namespace that looks like this:

(ns notebooks.annual-mean-temp-uk

(:require [clojure.data.csv :as csv]

[nextjournal.clerk :as clerk]))

(-> "data/annual-mean-temp-uk.csv" ;; name of the file

slurp              ;; read the contents of the file

csv/read-csv       ;; parse the result as csv data

clerk/use-headers  ;; tell Clerk to use the first row as headers

clerk/table)       ;; render the data as a table

The notebook should refresh automatically when the file is saved, showing the table.

Conclusion

We have a working Clojure environment running locally and know how to set up and display a new Clerk notebook, using some of its most basic functionality to begin to explore our data. We’ve only covered the bare minimum about Clojure required to make this notebook load, so if it’s new to you and of further interest there are dozens of high quality, free resources available for you to dig deeper. The source code for this demo is also available on github (the code up to this point is on the main branch) for comparison.

One of the best parts of working with data in Clojure is the welcoming and helpful community, so if there are any issues you can usually find help on the Clojurian’s slack or Clojureverse forum, or of course feel free to ping me in any of the many places I lurk around the internet!

In the next and final post of this series we’ll learn how to build data visualisations with Clerk’s vega viewer and Hanami; a library for writing succinct vega and vega-lite specifications in Clojure.

Visualising data in Clojure with Hanami

by Kira McLean

This is the second post in a tutorial series, which builds on exploring data with Clojure and Clerk. In this post we look at visualising the data.

(ns notebooks.annual-mean-temp-uk

(:require [clojure.data.csv :as csv]

[nextjournal.clerk :as clerk]))

(-> "data/annual-mean-temp-uk.csv" ;; name of the file

slurp              ;; read the contents of the file

csv/read-csv       ;; parse the result as csv data

clerk/use-headers  ;; tell Clerk to use the first row as headers

clerk/table)       ;; render the data as a table

At TPXimpact we publish a lot of data, but what can you do with a good dataset once it exists? In the previous post in this series, we learned how to download a dataset from the PMD open data platform; set up a Clojure programming environment, and create a new Clerk notebook. We left off with a working Clerk notebook and a preview of our data being rendered in a simple table (as shown above). In this post, we’ll learn how to use some tools from Clojure’s emerging data science ecosystem to visualise the data.

We’ll be taking advantage of Clerk’s built-in Vega viewer to render a Vega-lite specification to see our visualisation. We’ll also be using the Clojure library Hanami to help simplify writing the specification.

The code for the project as we left it is available on github. We’ll be building on top of where we left off in the first post.

Brief background on grammar for graphics

Before diving into the code again, it‘s helpful to have a basic understanding of how Vega approaches visualising data. Vega is a “grammar for graphics”, so it’s like a programming language but, rather than describing arbitrary instructions for a computer to execute, it describes what a data visualisation should look like in JSON. This is why it’s also commonly described as a “declarative” way to describe how to visually encode data and interactions into a format that can be rendered in a browser. We tell Vega what our visualisation should look like, and all of the details about how it gets rendered are left to the library. Like a language, we can think of Vega as being made up of words and rules for combining those words.

Vega-lite is built on top of Vega. It’s effectively a more concise version of the same library that abstracts away some of the details common to most visualisations. We’ll be working with Vega-lite for the rest of this tutorial.

The main components of the Vega-lite “language” are described below:

- data: input for the visualisation

- mark: shape/type of graphics to use

- encoding: mapping between data and marks

- transform: e.g. filter, aggregate, bin, etc.

- scale: meta-info about how the data should fit into the visualisation

- guides: legends, labels

These are some of the available JSON keys in a Vega-lite spec. The corresponding values are the “rules” for combining these “words”. Some examples are concat, layer, repeat, facet, resolve. There is more to Vega-lite, but this is enough to get started and to build the visualisation for our project. For many more examples of how this grammar is used to piece together “sentences” that Vega-lite can interpret and render as a graphic, see the Vega-lite website.

Rendering a Vega spec in Clerk

Now we can start exploring a bit in our notebook. As mentioned above, a Vega-lite spec is just some JSON data that uses specific, supported keys and values. Clojure can work with JSON, but conventionally uses EDN as a data transfer format. The “E” stands for “extensible”, so it supports more complex data structures than JSON does. This means that all JSON can be directly converted to EDN, but not the other way around. This is relevant because our Clerk notebook will expect Vega-lite specifications to be written in EDN, not JSON.

In order to get started, and see what Clerk does with a Vega spec, we can copy a simple example directly from the Vega-lite website. As an example, copy-pasting this Vega-lite specification for a visualisation of carbon dioxide in the atmosphere over time, into a JSON to EDN converter online gives us an EDN version of the specification.

For now, paste the EDN result into the notebook that we started last time. For a reminder about how to render the Clerk notebook in your browser, see the previous post.

In annual_meal_temp_uk.clj:

{:$schema "https://vega.github.io/schema/vega-lite/v5.json"

:data {:url "data/co2-concentration.csv"

:format {:parse {:Date "utc:'%Y-%m-%d'"}}}

:width 800

:height 500

:transform [{:calculate "year(datum.Date)" :as "year"}

{:calculate "floor(datum.year / 10)" :as "decade"}

{:calculate "(datum.year % 10) + (month(datum.Date)/12)"

:as "scaled_date"}

{:calculate "datum.first_date === datum.scaled_date ? 'first' : datum.last_date === datum.scaled_date ? 'last' : null"

:as "end"}]

:encoding {:x {:type "quantitative" :title "Year into Decade" :axis {:tickCount 11}}

:y {:title "CO2 concentration in ppm" :type "quantitative" :scale {:zero false}}

:color {:field "decade" :type "ordinal" :scale {:scheme "magma"} :legend nil}}

:layer [{:mark "line" :encoding {:x {:field "scaled_date"} :y {:field "CO2"}}}

{:mark {:type "text" :baseline "top" :aria false}

:encoding {:x {:aggregate "min" :field "scaled_date"}

:y {:aggregate {:argmin "scaled_date"} :field "CO2"}

:text {:aggregate {:argmin "scaled_date"} :field "year"}}}

{:mark {:type "text" :aria false}

:encoding {:x {:aggregate "max" :field "scaled_date"}

:y {:aggregate {:argmax "scaled_date"} :field "CO2"}

:text {:aggregate {:argmax "scaled_date"} :field "year"}}}]

:config {:text {:align "left" :dx 3 :dy 1}}}

Your copy-pasted EDN from the JSON conversion might appear more verbose, depending on how much whitespace is added for readability. Note that commas are optional in Clojure and they are interpreted as whitespace by the reader - exactly the same as spaces, tabs, and newlines.

To get Clerk to render this Vega-lite spec, we just need to pass it as the only argument to Clerk’s Vega viewer, clerk/vl, as in:

(clerk/vl {... that whole big chunk of EDN from above ...})

This won’t quite work yet, since the data source is specified as a relative path. To make it work we just need to update the value of :data > :url to a full URL: “https://vega.github.io/vega-lite/data/co2-concentration.csv"

This should render the visualisation in the notebook:

Visualising Data Hanami

This spec is quite elaborate, but a useful demonstration of how all the components of Vega-lite come together to describe a data visualisation.

Hanami Setup

The next (and final) topic we’ll cover is Hanami. As you can see, Vega-lite specs can quickly become very verbose, so Hanami is a Clojure library that was developed to simplify writing these Vega specs. Using Hanami we can write compact, declarative, composable, and recursively parameterised visualisation templates.

The first step here is to add Hanami to our project. First we have to add it as a dependency and restart the REPL. Our deps.edn should look like this now:

In deps.edn:

{:deps {io.github.nextjournal/clerk {:mvn/version "0.8.445"}

org.clojure/data.csv {:mvn/version "1.0.1"}

aerial.hanami/aerial.hanami {:mvn/version "0.17.0"}}}

Next, we require the relevant parts of Hanami in our notebook namespace declaration. We’ll be using Hanami’s commonand templates namespaces, so our namespace declaration should look like this:

In annual_meal_temp_uk.clj:

(ns notebooks.annual-mean-temp-uk

(:require [clojure.data.csv :as csv]

[nextjournal.clerk :as clerk]

[aerial.hanami.common :as hc]

[aerial.hanami.templates :as ht]))

For the rest of this tutorial, it will be useful to have loaded and evaluated the notebook namespace in the REPL. In VSCode with Calva this can be done by opening the command palette and selecting “Calva: Switch Namespace in Output/REPL Window to Current Namespace” and then “Calva: Load/Evaluate Current File and its Requires/Dependencies”. This allows us access to all of these required namespaces directly from the REPL.

Hanami background

In Hanami, a “template” is just a Clojure map of substitution keys. The library defines many default substitution keys and a set of base templates to use as starting points, which you can see by entering @hc/_defaults in a REPL. This will print a big map with all of the default values that Hanami uses in the templates. By convention, Hanami substitution keys are all-caps symbols. To get the default value for a specific substitution key, Hanami includes a helper function hc/get-default. For example, to find the default value Hanami uses for all visualisation backgrounds, we can execute (hc/get-default :BACKGROUND) in the REPL, which returns"floralwhite".

The main function in Hanami is hc/xform. This is the transformation function that takes a template and optionally a series of extra transformation key/value pairs, inserting the values wherever the keys are specified in the template. If no key/value pairs are provided, Hanami uses the built-in defaults. To see what all this means, we can use Hanami to fill in a very basic template (remember, a template is just a Clojure map where the values are Hanami substitution keys. These are all-caps symbols, some of which have default values):

In the REPL (input> and output> are just prompts and will most likely be different in your REPL):

input> (hc/xform {:my-key :BACKGROUND})

output> {:my-key "floralwhite"}

If we were to supply a custom value for :BACKGROUND, Hanami would use that instead, for example:

input> (hc/xform {:my-key :BACKGROUND} :BACKGROUND "orange")

output> {:my-key "orange"}

Transformations are recursive, i.e. the definitions of substitution keys can themselves contain substitution keys. For example, inspect the default value for :TOOLTIP:

input> (hc/get-default :TOOLTIP)

output> [{:field :X, :type :XTYPE, :title :XTTITLE, :format :XTFMT}

{:field :Y, :type :YTYPE, :title :YTTITLE, :format :YTFMT}]

The result includes several values which are themselves substitution keys. When used in a transformation and given no extra parameters, all of these values are replaced with the Hanami defaults:

input> (hc/xform {:my-key :TOOLTIP})

output> {:my-key [{:field "x", :type "quantitative"}

{:field "y", :type "quantitative"}]}

If we supply any custom values to override these defaults, they will be used in the transformation:

input> (hc/xform {:my-key :TOOLTIP} :X "my custom x value")

output> {:my-key [{:field "my custom x value", :type "quantitative"}

{:field "y", :type "quantitative"}]}

This is a good time to note Hanami’s special “nothing” value, RMV. Notice in the above transformed :TOOLTIPexample that several of the keys that were specified in its definition have been removed. If you execute hc/RMV in the REPL, you'll see that it's just an alias for the special "nothing" value of the underlying library which Hanami uses for data manipulation (which is called Specter). If we check, for example, (hc/get-default :XTTITLE), we can see it's the same thing. This is a useful shortcut in Hanami to specify a default of "nothing" but to still include a given key in a template, which can be helpful to demonstrate to users which keys are available for writing generic, composable templates. Vega-lite does not gracefully ignore null values in specs, so we have to remove them before trying to render a visualisation.

The last relevant piece of background to understand in order to get started with Hanami is its built-in templates. The aerial.hanami.templates namespace defines several common Vega-lite chart templates and other useful composable chart components. See, for example, ht/bar-chart or ht/point-chart, by executing those statements in the REPL.

Putting it all together

We finally have all the necessary pieces to visualise our dataset from last time. To get started we can use Hanami’s built-in line-chart template. To render the skeleton of the chart, we can use Hanami to transform a bare-bones line chart template into a Vega-lite spec and tell Clerk to render it:

In annual_meal_temp_uk.clj:

(-> (hc/xform ht/line-chart)

clerk/vl)

Next, we need to supply our data to the graph. Vega-lite expects data to be in a tabular format, like a spreadsheet or database table. In JSON or EDN format this means the data looks like a collection of maps. Hanami will automatically convert data in CSV format to a vector of maps for us.

Hanami supports many different ways of specifying the data source. We can point Hanami to the data that’s already in a file in our project using the :FDATA substitution key (F for file):

(-> (hc/xform ht/line-chart

:FDATA "data/annual-mean-temp-uk.csv")

clerk/vl)

We also need to tell Vega-lite which columns to use for the x and y axes:

(-> (hc/xform ht/line-chart

:FDATA "data/annual-mean-temp-uk.csv"

:X "Year"

:Y "Annual Mean Temperature")

clerk/vl)

This should be enough to get our data showing in our notebook!

Hanami Post Graph 2

From here we can make some minor improvements. The effect of each extra value is explained in the comment beside it:

(-> (hc/xform ht/line-chart

:FDATA "data/annual-mean-temp-uk.csv"

:X "Year"

:Y "Annual Mean Temperature"

:XTITLE "Year" ;; label for the x axis

:YTITLE "Average temperature (ºC)" ;; label for the y axis

:XTYPE "temporal" ;; display x axis labels as years, not numbers

:YSCALE {:zero false} ;; don't force the baseline to be zero

:WIDTH 700 ;; make the graph wider

)

clerk/vl)

This tidies up our graph to something we can more easily interpret:

Hanami Post Graph 3

Vega-lite has many built-in transformations, including locally-estimated scatterplot smoothing, which can be used to produce a trend line. If we wanted to render a trend line, rather than plot each point individually, we just have to tell Vega-lite which axes to use in the transformation:

(-> (hc/xform ht/line-chart

:FDATA "data/annual-mean-temp-uk.csv"

:X "Year"

:Y "Annual Mean Temperature"

:XTITLE "Year"

:YTITLE "Average temperature (ºC)"

:XTYPE "temporal"

:YSCALE {:zero false}

:WIDTH 700

:TRANSFORM [{:loess :Y :on :X}]) ;; <<<- this is new

clerk/vl)

This will give us a smooth trendline, rather than a line connecting each point:

Hanami Post Graph 4

Vega-lite also supports layering two charts on the same set of axes, so we can layer these two graphs on top of each other to generate our final visualisation. Hanami includes a template to simplify this, ht/layer-chart. I’ve also changed the colour for the mark on the second graph by specifying a different value for :MCOLOR:

(-> (hc/xform ht/layer-chart

:FDATA "data/annual-mean-temp-uk.csv"

:LAYER [(hc/xform ht/line-chart

:X "Year"

:Y "Annual Mean Temperature"

:XTITLE "Year"

:YTITLE "Average temperature (ºC)"

:XTYPE "temporal"

:YSCALE {:zero false}

:WIDTH 700)

(hc/xform ht/line-chart

:X "Year"

:Y "Annual Mean Temperature"

:XTITLE "Year"

:YTITLE "Average temperature (ºC)"

:XTYPE "temporal"

:YSCALE {:zero false}

:WIDTH 700

:TRANSFORM [{:loess :Y :on :X}]

:MCOLOR "orange")])

clerk/vl)

Hanami Post Overlay Graph

Conclusion

We have our Vega-lite spec being generated by Hanami and rendered with Clerk. Almost any visualisation you can imagine can be written as a Vega or Vega-lite specification, and Hanami also supports much more than we covered here. If you’re interested in learning more, both libraries have excellent documentation and lots of examples. The source code for this demo is also available on GitHub (the code up to this point is on the completed-visualisation branch). And, as always, feel free to reach out to me or the Clojure community in general wherever you find us on the internet.