March 14, 2022

On the not-so-easy transition from lein-figwheel to figwheel-main

Table of Contents

- Setting the stage

- Not so fast.

- Updating dependencies

- The Figwheel problem

- Back to the basics

- figwheel-main-template

- Fixing problems - step by step

- lein aliases

- Wrong output file name

- JS imports

- reagent.core vs reagent.dom

- Optimized build (for production)

- The "figwheel build name" problem

- Injecting the build name

- Making server port configuration flexible

- Figwheel/ClojureScript REPL vs. Emacs/Cider

- Google Closure compiler and classpath problems

- Chasing the clojurescript artifacts

- Meet enrich.classpath

- Resources

Recently, I went trough the process of upgrading a rather old project using ClojureScript with lein-figwheel to figwheel-main.

It was a painful experience and I struggled a lot. Hereby, I describe various problems I came across.

SETTING THE STAGE

It all started because I couldn’t fire up figwheel build anymore:

1. Unhandled java.util.MissingResourceException

Can't find resource for bundle java.util.PropertyResourceBundle, key jsdoc.primitives

ResourceBundle.java:  564  java.util.ResourceBundle/getObject

ResourceBundle.java:  521  java.util.ResourceBundle/getString

ParserRunner.java:  116  com.google.javascript.jscomp.parsing.ParserRunner/initResourceConfig

ParserRunner.java:   78  com.google.javascript.jscomp.parsing.ParserRunner/createConfig

Compiler.java: 2686  com.google.javascript.jscomp.Compiler/createConfig

Compiler.java: 2667  com.google.javascript.jscomp.Compiler/getParserConfig

JsAst.java:  155  com.google.javascript.jscomp.JsAst/parse

JsAst.java:   55  com.google.javascript.jscomp.JsAst/getAstRoot

externs.clj:  169  cljs.externs/parse-externs

externs.clj:  156  cljs.externs/parse-externs

externs.clj:  204  cljs.externs/externs-map*/fn

...

env.cljc:   51  cljs.env$default_compiler_env_STAR_/invokeStatic

env.cljc:   46  cljs.env$default_compiler_env_STAR_/invoke

env.cljc:   62  cljs.env$default_compiler_env/invokeStatic

env.cljc:   59  cljs.env$default_compiler_env/invoke

utils.clj:  108  figwheel-sidecar.utils/compiler-env

utils.clj:  105  figwheel-sidecar.utils/compiler-env

build_utils.clj:    8  figwheel-sidecar.build-utils/add-compiler-env

After upgrading to latest clojurescript it still failed but in a different way:

Caused by: java.lang.NoSuchMethodError: 'java.util.stream.Collector com.google.common.collect.ImmutableSortedSet.toImmutableSortedSet(java.util.Comparator)'

at com.google.javascript.jscomp.deps.ModuleLoader.createRootPaths(ModuleLoader.java:257)

at com.google.javascript.jscomp.deps.ModuleLoader.<init>(ModuleLoader.java:147)

at com.google.javascript.jscomp.deps.ModuleLoader.<init>(ModuleLoader.java:48)

at com.google.javascript.jscomp.deps.ModuleLoader$Builder.build(ModuleLoader.java:139)

at com.google.javascript.jscomp.deps.ModuleLoader.<clinit>(ModuleLoader.java:408)

at com.google.javascript.jscomp.DiagnosticGroups.<clinit>(DiagnosticGroups.java:182)

at cljs.closure__init.load(Unknown Source)

at cljs.closure__init.<clinit>(Unknown Source)

...

at cljs.repl$loading__5569__auto____4083.invoke(repl.cljc:9)

at cljs.repl__init.load(Unknown Source)

at cljs.repl__init.<clinit>(Unknown Source)

...

at figwheel_sidecar.repl$eval52111$loading__6737__auto____52112.invoke(repl.clj:1)

I tried a few times, but couldn’t solve the problem. I looked at the dependencies - they were all terribly outdated. Moreover, lein-figwheel had been replaced a long time ago by figwheel-main. I decided to upgrade the whole project…

NOT SO FAST.

Updating dependencies

Updating the depedencies was straightforward. I just went through everything in the project.clj file, looked up the latest versions on clojars.org and used those.

To my surprise, there were no major problems caused by the upgrade itself; except the problem with upgrading Figwheel.

The Figwheel problem

figwheel-main is quite different from lein-figwheel.

I checked out https://github.com/bhauman/figwheel-main and https://figwheel.org/ and started sketching out the new configuration for my project.

I struggled a lot and couldn’t make a proper config. For instance, I was really confused by the distinction between dev.cljs.edn and figwheel-main.edn configuration files.

On top of that, I hadn’t worked with this project and ClojureScript tooling for a few years.

BACK TO THE BASICS

Enough! It was time to learn the basics. Taking a more structured approach and reading Figwheel documentation step by step helped me a lot. After reading the first half, the things got much clearer.

I realized that the typical Figwheel configuration might actually be split into 2+ files:

- This can also contain Figwheel specific configuration attached as metadata.

- figwheel-main.edn figwheel specific configuration shared by all the builds

- <build-name>.cljs.edn for ClojureScript build specific configuration

- Typically, you have at least dev.cljs.edn.

figwheel-main-template

What helped me a lot was to try figwheel-main-template:

lein new figwheel-main hello-world.core -- --reagent

Then I examined project.clj, dev.cljs.edn and figwheel-main.edn to get better understanding how things are wired in a real project.

FIXING PROBLEMS - STEP BY STEP

lein aliases

First, I didn’t get a good idea of the aliases I’m supposed to use. After checking out the sample hello-world.core project I figured out that I not only need need the "fig" alias, but, crucially, also fig:build:

:aliases {"fig" ["trampoline" "run" "-m" "figwheel.main"]

"fig:build" ["trampoline" "run" "-m" "figwheel.main" "--build" "dev" "--repl"]

;; a separate figwheel build is used for optimized build

"fig:min"   ["run" "-m" "figwheel.main" "-O" "advanced" "--build-once" "min"]

"fig:test"  ["run" "-m" "figwheel.main" "-co" "test.cljs.edn" "-m" "hello-world.test-runner"]}

Wrong output file name

Figwheel compiler configuration docs incorrectly state that the default :output-to (compiled javascript) file name is

target/public/cljs-out/[build-name]-main.js

When I tried that, I got 404 Not Found in the browser:

figwheel js build file - Not Found

The proper build file name is actually [build-name]/main.js not [build-name]-main.js:

target/public/cljs-out/[build-name]/main.js

JS imports

After fixing the build file name, the next error I got (in the browser) was:

Uncaught SyntaxError: Cannot use import statement outside a module   main.js:1

# main.js:1

import {npmDeps} from "./npm_deps.js";

I was quite confused for a while and searched around but couldn’t find much.

Originally, the project used Selmer’s script tag to import JavaScript files. After looking at the hello-world.core sample project I simply adopted the <script> tag style used there:

<script type="text/javascript" src="cljs-out/dev/main_bundle.js"></script>

I’m not sure why Selmer’s {% script %} wasn’t working. There may well have been another problem but I didn’t find what that was.

reagent.core vs reagent.dom

The fun was not over. Trying to build it again, I got another error, this time related to the Reagent’s render function:

reagent.core/render error

As I said, the project was using really old versions of dependencies and one of those was reagent. In the new version, they simply moved the render function from reagent.core to reagent.dom. The fix was simple: update my main cljs namespace to use reagent.dom instead.

I also had to hard-refresh the webpage to get rid of the error.

And the dev build was finally working. Yes!

OPTIMIZED BUILD (FOR PRODUCTION)

While the build was working in the dev environment I had to also produce a minimized build for production. That process brought more surprises.

I read the Advance Compile docs and found a related github issue In Leiningen, setting :resource-paths to include "target" is bad for uberjars #134. There they say:

With Leinigen I suggest making :target-dir resources/public.

— Bruce Hauman (Figwheel author)

I rushed to adop that advice - except that the :target-dir should really be just resources, not resources/public (you would end up with the build directory resources/public/public/cljs-out/…).

I again checked the hello-world.core sample project and indeed there was this configuration:

:target-dir "resources"

The "figwheel build name" problem

:target-dir configuration was just the beginning.

A bigger problem was that the JS artifact name depends on the build name.

So using the lein aliases we defined:

"fig:build" ["trampoline" "run" "-m" "figwheel.main" "--build" "dev" "--repl"]

;; a separate figwheel build is used for optimized build

"fig:min"   ["run" "-m" "figwheel.main" "-O" "advanced" "--build-once" "min"]

- If we run lein fig:build we get cljs-out/dev/main_bundle.js.

- If we run lein fig:min we get cljs-out/min/main_bundle.js.

Having two (or more) different artifact names isn’t great, because we need to include it in the main HTML file - remember, we had this:

<script type="text/javascript" src="cljs-out/dev/main_bundle.js"></script>

Advance Compile docs shows an example how to produce an optimized artifact using the dev build/config

$ clj -m figwheel.main -O advanced -bo dev

If I used this approach, I would have the same artifact name for both unoptimized and optimized builds. However, I don’t like it because you include the dev configuration in the build of the artifact intended for the production. You also cannot use both "min" and "dev" build at the same time.

Injecting the build name

After thinking about it for a while, I decided to make the backend responsible for generating proper artifact name. After all, it was the server side logic who rendered the main HTML template (with Selmer).

I updated my main HTML file to use a variable:

<script type="text/javascript" src="cljs-out/{{figwheel/build-name}}/main_bundle.js"></script>

This is set in the backend clojure code responsible for rendering. The app gets this name simply as a configuration setting in profiles.clj. It’s a dev-only setting with a fallback to "min":

(defn home-page []

;; dynamic injection of figwheel build is needed to insert proper minified js into uberjar

(let [figwheel-build (or (:figwheel-build-name env) "min")]

(layout/render "home.html" {:figwheel/build-name figwheel-build})))

Making server port configuration flexible

To glue everything together, I needed one more piece: the dynamic configuration of the URL serving the HTML file. Since my app has both backend and frontend, early on I added custom :open-url to the figwheel config:

:open-url "http://localhost:5001/app"

The problem was that a developer can choose another port. In fact, the default port is 5000, it’s only me who’s using 5001 to avoid conflicts with other applications running on my machine.

Figwheel makes this port configuration flexible to some extent - you can use [[server-port]] placeholder and it’s expanded to the actual port used by Figwheel. But this is not the same thing as my backend server port! It’s the port of the internal ring server that Figwheel uses. As such, it doesn’t work in our setup. I had to find a different solution.

I simply renamed figwheel-main.edn to figwheel-main.edn.template and added a script (dev-configure Make target) to parse the port from profiles.clj and replace the placeholder in figwheel-main.edn.template with the correct port. This output is then saved as the final figwheel-main.edn file.

This is an extra step that the developer has to do, but it’s typically a one-time thing - you run it once and that’s it. Of course, if you change your port configuration, you have to run it again.

The code

- profiles.clj

{:profiles/dev  {:env {:options {:port 5001}

;; use the "dev" figwheel build instead of optimized "min" build

;; - see ui routes serving home.html and also project.clj

:figwheel-build-name "dev"

- figwheel-main.edn.template <1>

;;; This is a template file to generate proper `figwheel-main.edn` file.

;;; The [[server-port]] variable here will be replaced with the actual value parsed from profiles.clj

;;; This is necessary because we need to use the backend server port (by default 5000),

;;; not the figwheel's ring server port (by default 9500).

;;; We could just hardcode 5000 here but then developers wouldn't be able to change the port easily.

;;;

;;; Resources:

;;; - https://figwheel.org/docs/create_a_build.html#configuring-a-build

;;; - https://figwheel.org/config-options.html

;; overriding default index.html page - we need to serve home.html from the server

{:open-url "http://localhost:[[server-port]]/app" ; https://figwheel.org/docs/your_own_page.html#providing-your-own-page ;; set target-dir different than "target" - see https://github.com/bhauman/figwheel-main/pull/138

:target-dir "resources"

:css-dirs ["resources/public/css"] ; https://figwheel.org/docs/live_css.html

:watch-dirs ["src/cljc" "src/cljs" "env/dev/cljs"] ; https://figwheel.org/docs/hot_reloading.html

;; automatically bundle JS dependencies like react: https://figwheel.org/docs/npm.html

:auto-bundle :webpack

}

- Makefile

# dev-configure is for dev / local installation only

# it parses the server port from profiles.clj and saves it into figwheel-main.edn

dev-configure: profiles.clj

server_port := $(shell cat profiles.clj | clj -e '(get-in (clojure.edn/read *in*) [:profiles/dev :env :options :port])')

sed "s/\[\[server-port\]\]/$(server_port)/" figwheel-main.edn.template > figwheel-main.edn

You can see that I renamed figwheel-main.edn to figwheel-main.edn.template and the final config file is now generated dynamically.

That’s it - the transition to figwheel-main is now complete and I can run both Clojure and ClojureScript REPLs again!

Or?!

FIGWHEEL/CLOJURESCRIPT REPL VS. EMACS/CIDER

The transition was almost complete - there was one missing piece: editor integration.

I use Emacs with Cider and being able to interact with the application from within the editor is an essential part of my development workflow. I couldn’t develop the backend code without it and I wanted to have a similar experience on frontend.

Fortunately, Cider has a first-class support for figwheel-main: https://figwheel.org/docs/emacs.html Unfortunately, it didn’t quite work.

Google Closure compiler and classpath problems

After launching the REPL within Emacs via cider-jack-in-clj&cljs), I got a surprisping error:

1. Unhandled java.lang.NullPointerException

Null closurePrimitiveNames

AutoValue_Config.java:  196  com.google.javascript.jscomp.parsing.AutoValue_Config$Builder/setClosurePrimitiveNames

ParserRunner.java:   91  com.google.javascript.jscomp.parsing.ParserRunner/createConfig

Compiler.java: 2686  com.google.javascript.jscomp.Compiler/createConfig

Compiler.java: 2667  com.google.javascript.jscomp.Compiler/getParserConfig

JsAst.java:  155  com.google.javascript.jscomp.JsAst/parse

JsAst.java:   55  com.google.javascript.jscomp.JsAst/getAstRoot

externs.clj:  169  cljs.externs/parse-externs

After trying to find the cause of this error I was becoming desparate. Until I clicked, by coincidence, on the stacktrace in the Emacs cider-error buffer. It opened the corresponding Compiler.java. That wasn’t surprising (well, I was a bit surprised that Cider was able to locate the file seamlessly) but the weird piece was the file path:

/Users/jumar/.m2/repository/com/google/javascript/closure-compiler/v20130603/closure-compiler-v20130603-sources.jar:com/google/javascript/jscomp/Compiler.java

You’ll notice that there was a really old version of closure-compiler in place, that is v20130603. But leiningen didn’t report the same version - instead it reported much newer v20200315:

lein deps :tree >& deps.out && less deps.out

...

[org.clojure/clojurescript "1.10.773"]

[com.google.javascript/closure-compiler-unshaded "v20200315"]

Chasing the clojurescript artifacts

I was really puzzled how the Cider REPL can use a different version when leiningen doesn’t report anything like that.

I ended up going through the clojurescript repo commits. After a while, I found the commit that contained version v20130603. A simple dependency upgrade made in Dec 2013!

So something was bringing in a really old clojurescript version - but why?!

Searching through more commits, I found CLJS-1640: Use the unshaded version of the closure compiler It was only then that I really noticed the difference: the old clojurescript version used the com.google.javascript/closure-compiler artifact but the newer versions have been using com.google.javascript/closure-compiler-unshaded artifact.

That introduces a tricky problem: if you transitively depend on multiple different versions of clojurescript, you can end up with both closure-compiler and closure-compiler-unshaded on the classpath!

But still, this was a long time ago and it was only a problem when starting the REPL with Cider. There was probably something else in play…

Meet enrich.classpath

Indeed!

I examined the exact command that Cider is using to start the REPL:

/usr/local/bin/lein update-in :dependencies conj \[nrepl/nrepl\ \"0.9.0\"\] -- \

update-in :dependencies conj \[refactor-nrepl/refactor-nrepl\ \"3.3.2\"\] -- \

update-in :dependencies conj \[cider/piggieback\ \"0.5.2\"\] -- \

update-in :plugins conj \[refactor-nrepl/refactor-nrepl\ \"3.3.2\"\] -- \

update-in :plugins conj \[cider/cider-nrepl\ \"0.28.1\"\] -- \

update-in :plugins conj \[mx.cider/enrich-classpath\ \"1.9.0\"\] -- \

update-in :middleware conj cider.enrich-classpath/middleware -- \

repl :headless :host localhost

Then I got an idea: how about running the deps :tree and checking the dependencies?

lein update-in :dependencies conj \[nrepl/nrepl\ \"0.9.0\"\] -- update-in :dependencies conj \[refactor-nrepl/refactor-nrepl\ \"3.3.2\"\] -- update-in :dependencies conj \[cider/piggieback\ \"0.5.2\"\] -- update-in :plugins conj \[refactor-nrepl/refactor-nrepl\ \"3.3.2\"\] -- update-in :plugins conj \[cider/cider-nrepl\ \"0.28.1\"\] -- update-in :plugins conj \[mx.cider/enrich-classpath\ \"1.9.0\"\] -- update-in :middleware conj cider.enrich-classpath/middleware -- \

deps :tree >& deps-cider.out

I opened deps-cider.out and searched for "v20130603":

[com.google.javascript/closure-compiler-unshaded "v20200315" :classifier "javadoc" :exclusions [[*]]]

[com.google.javascript/closure-compiler-unshaded "v20200315" :classifier "sources" :exclusions [[*]]]

[com.google.javascript/closure-compiler "v20130603" :classifier "javadoc" :exclusions [[*]]]

[com.google.javascript/closure-compiler "v20130603" :classifier "sources" :exclusions [[*]]]

Here we go! It’s right there!

Except that it’s only "sources" and "javadocs", not a JAR file. I checked the Cider repl startup command again and enrich.classpath caught my attention. I had known this plugin from before - it’s a new way how to automatically add sources and javadocs for all the project’s dependencies.

This looked liked a problem with this plugin so I removed it from the Cider startup command and voila! It all worked.

I created a minimal repro based on the real project and submitted a bug report: enrich.classpath doesn’t work with CLJS repl (fighweel-main): Null closurePrimitiveNames error after starting the REPL #20

Until this is fixed, I keep enrich.classpath disabled for my project.

RESOURCES

- figwheel-main

- Figwheel documentation

- figwheel-main-template

- Fighweel docs: Host Page - Providing your own page

- In Leiningen, setting :resource-paths to include "target" is bad for uberjars #134 Figwheel - editor integration

- Leiningen profiles merging

- enrich.classpath doesn’t work with CLJS repl (fighweel-main): Null closurePrimitiveNames error after starting the REPL #20

Tags: clojure frontend leiningen clojurescript

« WEEKLY BITS 06/2022 - CODE RED, DEBUGGING ON PRODUCTION, 'DIRTY PIPE' VULNERABILITY, AND MACOS LOCAL SNAPSHOTS WEEKLY BITS 05/2022 - UKRAINE, CLOUDFRONT & ORIGINCOMMERROR, DOUBLE-SUBMIT COOKIES, FRONTEND MONITORING, EMACS' QUERY-REPLACE-REGEXP »

COPYRIGHT © 2022 JURAJ MARTINKA

POWERED BY CRYOGEN | FREE WEBSITE TEMPLATE BY DOWNLOAD WEBSITE TEMPLATES

CURIOUS (CLOJURE) PROGRAMMER

Simplicity matters

- HOME

ARCHIVES

TAGS

ABOUT

MY TALKS

WEEKLY BITS & PIECES

CLOJURE TIP OF THE DAY SCREENCAST

RSS