Motivation

We are building dynamic multi-tenant systems where data models should be first-class: they should drive the runtime value transformations, forms and processes. We should be able to edit the models at runtime, persist them and load them back from a database and over the wire, for both Clojure and ClojureScript. Think of JSON Schema, but for Clojure/Script.

Hasn't the problem been solved (many times) already?

There is Schema, which is an awesome, proven and collaborative open-source project, and we absolutely love it. We still use it in many of our projects. The sad part: serializing & de-serializing schemas is non-trivial and there is no proper support on branching.

Spec is the de facto data specification library for Clojure. It has many great ideas, but it is opinionated with macros, global registry, and it doesn't have any support for runtime transformations. Spec-tools was created to "fix" some of the things, but after five years of developing it, it's still a kind of hack and not fun to maintain.

So, we decided to spin out our own library, which would do all the things we feel is important for dynamic system development. It's based on the best parts of the existing libraries and several project-specific tools we have done over the years.

If you have expectations (of others) that aren't being met, those expectations are your own responsibility. You are responsible for your own needs. If you want things, make them.

- Rich Hickey, Open Source is Not About You

The library

Clojars Project

Malli requires Clojure 1.10+ and is tested against 1.10 and 1.11.

Syntax

Malli supports Vector, Map and Lite syntaxes.

Vector syntax

The default syntax uses vectors, inspired by hiccup:

type

[type & children]

[type properties & children]

Examples:

;; just a type (String)

:string

;; type with properties

[:string {:min 1, :max 10}]

;; type with properties and children

[:tuple {:title "location"} :double :double]

;; a function schema of :int -> :int

[:=> [:cat :int] :int]

Usage:

(require '[malli.core :as m])

(def non-empty-string

(m/schema [:string {:min 1}]))

(m/schema? non-empty-string)

; => true

(m/validate non-empty-string "")

; => false

(m/validate non-empty-string "kikka")

; => true

(m/form non-empty-string)

; => [:string {:min 1}]

Map syntax

Alternative map-syntax, similar to cljfx:

NOTE: For now, Map syntax in considered as internal, so don't use it as a database persistency model.

;; just a type (String)

{:type :string}

;; type with properties

{:type :string

:properties {:min 1, :max 10}

;; type with properties and children

{:type :tuple

:properties {:title "location"}

:children [{:type :double}

{:type :double}]}

;; a function schema of :int -> :int

{:type :=>

:input {:type :cat, :children [{:type :int}]}

:output :int}

Usage:

(def non-empty-string

(m/from-ast {:type :string

:properties {:min 1}}))

(m/schema? non-empty-string)

; => true

(m/validate non-empty-string "")

; => false

(m/validate non-empty-string "kikka")

; => true

(m/ast non-empty-string)

; => {:type :string,

;     :properties {:min 1}}

Map-syntax is also called the Schema AST.

Why multiple syntaxes?

Malli started with just the Vector syntax. It's really powerful and relatively easy to read, but not optimal for all use cases.

We introduced Map Syntax as we found out that the overhead of parsing large amount of vector-syntaxes can be a deal-breaker when running on slow single-threaded environments like Javascript on mobile phones. Map-syntax allows lazy and parseless Schema Creation.

We added Lite Syntax for simplified schema creation for special cases, like to be used with reitit coercion and for easy migration from data-specs.

Example Address schema

Following example schema is assumed in many of the following examples.

(def Address

[:map

[:id string?]

[:tags [:set keyword?]]

[:address

[:map

[:street string?]

[:city string?]

[:zip int?]

[:lonlat [:tuple double? double?]]]]])

Validation

Validating values against a schema:

;; with schema instances

(m/validate (m/schema :int) 1)

; => true

;; with vector syntax

(m/validate :int 1)

; => true

(m/validate :int "1")

; => false

(m/validate [:and :int [:> 6]] 7)

; => true

(m/validate [:qualified-keyword {:namespace :aaa}] :aaa/bbb)

; => true

;; optimized (pure) validation function for best performance

(def valid?

(m/validator

[:map

[:x :boolean]

[:y {:optional true} :int]

[:z :string]]))

(valid? {:x true, :z "kikka"})

; => true

Schemas can have properties:

(def Age

[:and

{:title "Age"

:description "It's an age"

:json-schema/example 20}

:int [:> 18]])

(m/properties Age)

; => {:title "Age"

;     :description "It's an age"

;     :json-schema/example 20}

Maps are open by default:

(m/validate

[:map [:x :int]]

{:x 1, :extra "key"})

; => true

Maps can be closed with :closed property:

(m/validate

[:map {:closed true} [:x :int]]

{:x 1, :extra "key"})

; => false

Maps keys are not limited to keywords:

(m/validate

[:map

["status" [:enum "ok"]]

[1 :any]

[nil :any]

[::a :string]]

{"status" "ok"

1 'number

nil :yay

::a "properly awesome"})

; => true

Most core-predicates are mapped to Schemas:

(m/validate string? "kikka")

; => true

See the full list of default schemas.

Qualified keys in a map

You can also use decomplected maps keys and values using registry references. References must be either qualified keywords or strings.

(m/validate

[:map {:registry {::id int?

::country string?}}

::id

[:name string?]

[::country {:optional true}]]

{::id 1

:name "kikka"})

; => true

Homogeneous maps

Other times, we use a map as a homogeneous index. In this case, all our key-value pairs have the same type. For this use case, we can use the :map-of schema.

(m/validate

[:map-of :string [:map [:lat number?] [:long number?]]]

{"oslo" {:lat 60 :long 11}

"helsinki" {:lat 60 :long 24}})

;; => true

Sequence schemas

You can use :sequential for any homogeneous Clojure sequence, :vector for vectors and :set for sets.

(m/validate [:sequential any?] (list "this" 'is :number 42))

;; => true

(m/validate [:vector int?] [1 2 3])

;; => true

(m/validate [:vector int?] (list 1 2 3))

;; => false

A :tuple describes a fixed length Clojure vector of heterogeneous elements:

(m/validate [:tuple keyword? string? number?] [:bing "bang" 42])

;; => true

Malli also supports sequence regexes like Seqexp and Spec. The supported operators are :cat & :catn for concatenation / sequencing

(m/validate [:cat string? int?] ["foo" 0]) ; => true

(m/validate [:catn [:s string?] [:n int?]] ["foo" 0]) ; => true

:alt & :altn for alternatives

(m/validate [:alt keyword? string?] ["foo"]) ; => true

(m/validate [:altn [:kw keyword?] [:s string?]] ["foo"]) ; => true

and :?, :*, :+ & :repeat for repetition:

(m/validate [:? int?] []) ; => true

(m/validate [:? int?] [1]) ; => true

(m/validate [:? int?] [1 2]) ; => false

(m/validate [:* int?] []) ; => true

(m/validate [:* int?] [1 2 3]) ; => true

(m/validate [:+ int?] []) ; => false

(m/validate [:+ int?] [1]) ; => true

(m/validate [:+ int?] [1 2 3]) ; => true

(m/validate [:repeat {:min 2, :max 4} int?] [1]) ; => false

(m/validate [:repeat {:min 2, :max 4} int?] [1 2]) ; => true

(m/validate [:repeat {:min 2, :max 4} int?] [1 2 3 4]) ; => true (:max is inclusive, as elsewhere in Malli)

(m/validate [:repeat {:min 2, :max 4} int?] [1 2 3 4 5]) ; => false

:catn and :altn allow naming the subsequences / alternatives

(m/explain

[:* [:catn [:prop string?] [:val [:altn [:s string?] [:b boolean?]]]]]

["-server" "foo" "-verbose" 11 "-user" "joe"])

;; => {:schema [:* [:map [:prop string?] [:val [:map [:s string?] [:b boolean?]]]]],

;;     :value ["-server" "foo" "-verbose" 11 "-user" "joe"],

;;     :errors ({:path [0 :val :s], :in [3], :schema string?, :value 11}

;;              {:path [0 :val :b], :in [3], :schema boolean?, :value 11})}

while :cat and :alt just use numeric indices for paths:

(m/explain

[:* [:cat string? [:alt string? boolean?]]]

["-server" "foo" "-verbose" 11 "-user" "joe"])

;; => {:schema [:* [:cat string? [:alt string? boolean?]]],

;;     :value ["-server" "foo" "-verbose" 11 "-user" "joe"],

;;     :errors ({:path [0 1 0], :in [3], :schema string?, :value 11}

;;              {:path [0 1 1], :in [3], :schema boolean?, :value 11})}

As all these examples show, the "seqex" operators take any non-seqex child schema to mean a sequence of one element that matches that schema. To force that behaviour for a seqex child :schema can be used:

(m/validate

[:cat [:= :names] [:schema [:* string?]] [:= :nums] [:schema [:* number?]]]

[:names ["a" "b"] :nums [1 2 3]])

; => true

;; whereas

(m/validate

[:cat [:= :names] [:* string?] [:= :nums] [:* number?]]

[:names "a" "b" :nums 1 2 3])

; => true

Although a lot of effort has gone into making the seqex implementation fast

(require '[clojure.spec.alpha :as s])

(require '[criterium.core :as cc])

(let [valid? (partial s/valid? (s/* int?))]

(cc/quick-bench (valid? (range 10)))) ; Execution time mean : 27µs

(let [valid? (m/validator [:* int?])]

(cc/quick-bench (valid? (range 10)))) ; Execution time mean : 2.7µs

it is always better to use less general tools whenever possible:

(let [valid? (partial s/valid? (s/coll-of int?))]

(cc/quick-bench (valid? (range 10)))) ; Execution time mean : 1.8µs

(let [valid? (m/validator [:sequential int?])]

(cc/quick-bench (valid? (range 10)))) ; Execution time mean : 0.12µs

String schemas

Using a predicate:

(m/validate string? "kikka")

Using :string Schema:

(m/validate :string "kikka")

;; => true

(m/validate [:string {:min 1, :max 4}] "")

;; => false

Using regular expressions:

(m/validate #"a+b+c+" "abbccc")

;; => true

;; :re with string

(m/validate [:re ".{3,5}"] "abc")

;; => true

;; :re with regex

(m/validate [:re #".{3,5}"] "abc")

;; => true

;; NB: re-find semantics

(m/validate [:re #"\d{4}"] "1234567")

;; => true

;; anchor with ^...$ if you want to strictly match the whole string

(m/validate [:re #"^\d{4}$"] "1234567")

;; => false

Maybe schemas

Use :maybe to express that an element should match some schema OR be nil:

(m/validate [:maybe string?] "bingo")

;; => true

(m/validate [:maybe string?] nil)

;; => true

(m/validate [:maybe string?] :bingo)

;; => false

Fn schemas

:fn allows any predicate function to be used:

(def my-schema

[:and

[:map

[:x int?]

[:y int?]]

[:fn (fn [{:keys [x y]}] (> x y))]])

(m/validate my-schema {:x 1, :y 0})

; => true

(m/validate my-schema {:x 1, :y 2})

; => false

Error messages

Detailed errors with m/explain:

(m/explain

Address

{:id "Lillan"

:tags #{:artesan :coffee :hotel}

:address {:street "Ahlmanintie 29"

:city "Tampere"

:zip 33100

:lonlat [61.4858322, 23.7854658]}})

; => nil

(m/explain

Address

{:id "Lillan"

:tags #{:artesan "coffee" :garden}

:address {:street "Ahlmanintie 29"

:zip 33100

:lonlat [61.4858322, nil]}})

;{:schema [:map

;          [:id string?]

;          [:tags [:set keyword?]]

;          [:address [:map

;                     [:street string?]

;                     [:city string?]

;                     [:zip int?]

;                     [:lonlat [:tuple double? double?]]]]],

; :value {:id "Lillan",

;         :tags #{:artesan :garden "coffee"},

;         :address {:street "Ahlmanintie 29"

;                   :zip 33100

;                   :lonlat [61.4858322 nil]}},

; :errors ({:path [:tags 0]

;           :in [:tags 0]

;           :schema keyword?

;           :value "coffee"}

;          {:path [:address :city],

;           :in [:address :city],

;           :schema [:map

;                    [:street string?]

;                    [:city string?]

;                    [:zip int?]

;                    [:lonlat [:tuple double? double?]]],

;           :type :malli.core/missing-key}

;          {:path [:address :lonlat 1]

;           :in [:address :lonlat 1]

;           :schema double?

;           :value nil})}

Note! If you need error messages that serialize neatly to EDN/JSON, use malli.util/explain-data instead.

Humanized error messages

Explain results can be humanized with malli.error/humanize:

(require '[malli.error :as me])

(-> Address

(m/explain

{:id "Lillan"

:tags #{:artesan "coffee" :garden}

:address {:street "Ahlmanintie 29"

:zip 33100

:lonlat [61.4858322, nil]}})

(me/humanize))

;{:tags #{["should be a keyword"]}

; :address {:city ["missing required key"]

;           :lonlat [nil ["should be a double"]]}}

Custom error messages

Error messages can be customized with :error/message and :error/fn properties:

(-> [:map

[:id int?]

[:size [:enum {:error/message "should be: S|M|L"}

"S" "M" "L"]]

[:age [:fn {:error/fn (fn [{:keys [value]} _] (str value ", should be > 18"))}

(fn [x] (and (int? x) (> x 18)))]]]

(m/explain {:size "XL", :age 10})

(me/humanize

{:errors (-> me/default-errors

(assoc ::m/missing-key {:error/fn (fn [{:keys [in]} _] (str "missing key " (last in)))}))}))

;{:id ["missing key :id"]

; :size ["should be: S|M|L"]

; :age ["10, should be > 18"]}

Messages can be localized:

(-> [:map

[:id int?]

[:size [:enum {:error/message {:en "should be: S|M|L"

:fi "pitäisi olla: S|M|L"}}

"S" "M" "L"]]

[:age [:fn {:error/fn {:en (fn [{:keys [value]} _] (str value ", should be > 18"))

:fi (fn [{:keys [value]} _] (str value ", pitäisi olla > 18"))}}

(fn [x] (and (int? x) (> x 18)))]]]

(m/explain {:size "XL", :age 10})

(me/humanize

{:locale :fi

:errors (-> me/default-errors

(assoc-in ['int? :error-message :fi] "pitäisi olla numero")

(assoc ::m/missing-key {:error/fn {:en (fn [{:keys [in]} _] (str "missing key " (last in)))

:fi (fn [{:keys [in]} _] (str "puuttuu avain " (last in)))}}))}))

;{:id ["puuttuu avain :id"]

; :size ["pitäisi olla: S|M|L"]

; :age ["10, pitäisi olla > 18"]}

Top-level humanized map-errors are under :malli/error:

(-> [:and [:map

[:password string?]

[:password2 string?]]

[:fn {:error/message "passwords don't match"}

(fn [{:keys [password password2]}]

(= password password2))]]

(m/explain {:password "secret"

:password2 "faarao"})

(me/humanize))

; {:malli/error ["passwords don't match"]}

Errors can be targeted using :error/path property:

(-> [:and [:map

[:password string?]

[:password2 string?]]

[:fn {:error/message "passwords don't match"

:error/path [:password2]}

(fn [{:keys [password password2]}]

(= password password2))]]

(m/explain {:password "secret"

:password2 "faarao"})

(me/humanize))

; {:password2 ["passwords don't match"]}

By default, only direct erroneous schema properties are used:

(-> [:map

[:foo {:error/message "entry-failure"} :int]] ;; here, :int fails, no error props

(m/explain {:foo "1"})

(me/humanize))

; => {:foo ["should be an integer"]}

Looking up humanized errors from parent schemas with custom :resolve (BETA, subject to change):

(-> [:map

[:foo {:error/message "entry-failure"} :int]]

(m/explain {:foo "1"})

(me/humanize {:resolve me/-resolve-root-error}))

; => {:foo ["entry-failure"]}

Spell checking

For closed schemas, key spelling can be checked with:

(-> [:map [:address [:map [:street string?]]]]

(mu/closed-schema)

(m/explain

{:name "Lie-mi"

:address {:streetz "Hämeenkatu 14"}})

(me/with-spell-checking)

(me/humanize))

;{:address {:street ["missing required key"]

;           :streetz ["should be spelled :street"]}

; :name ["disallowed key"]}

Values in error

Just to get parts of the value that are in error:

(-> Address

(m/explain

{:id "Lillan"

:tags #{:artesan "coffee" :garden "ground"}

:address {:street "Ahlmanintie 29"

:zip 33100

:lonlat [61.4858322, "23.7832851,17"]}})

(me/error-value))

;{:tags #{"coffee" "ground"}

; :address {:lonlat [nil "23.7832851,17"]}}

Masking irrelevant parts:

(-> Address

(m/explain

{:id "Lillan"

:tags #{:artesan "coffee" :garden "ground"}

:address {:street "Ahlmanintie 29"

:zip 33100

:lonlat [61.4858322, "23.7832851,17"]}})

(me/error-value {::me/mask-valid-values '...}))

;{:id ...

; :tags #{"coffee" "ground" ...}

; :address {:street ...

;           :zip ...

;           :lonlat [... "23.7832851,17"]}}

Pretty errors

For pretty development-time error printing, try malli.dev.pretty/explain

Value transformation

(require '[malli.transform :as mt])

Two-way schema-driven value transformations with m/decode and m/encode using a Transformer instance.

Default Transformers include:

name

description

mt/string-transformer

transform between strings and EDN

mt/json-transformer

transform between JSON and EDN

mt/strip-extra-keys-transformer

drop extra keys from maps

mt/default-value-transformer

applies default values from schema properties

mt/key-transformer

transforms map keys

mt/collection-transformer

conversion between collections (e.g. set -> vector)

NOTE: the included transformers are best-effort, i.e. they won't throw on bad input, they will just pass the input value through unchanged. You should make sure your schema validation catches these non-transformed values. Custom transformers should follow the same idiom.

Simple usage:

(m/decode int? "42" mt/string-transformer)

; 42

(m/encode int? 42 mt/string-transformer)

; "42"

For performance, precompute the transformations with m/decoder and m/encoder:

(def decode (m/decoder int? mt/string-transformer))

(decode "42")

; 42

(def encode (m/encoder int? mt/string-transformer))

(encode 42)

; "42"

Coercion

For both decoding + validating the results (throwing exception on error), there is m/coercer and m/coerce:

(m/coerce :int "42" mt/string-transformer)

; 42

(m/coerce :int "invalid" mt/string-transformer)

; =throws=> :malli.core/invalid-input {:value "invalid", :schema :int, :explain {:schema :int, :value "invalid", :errors ({:path [], :in [], :schema :int, :value "invalid"})}}

Coercion can be applied without transformer, doing just validation:

(m/coerce :int 42)

; 42

(m/coerce :int "42")

; =throws=> :malli.core/invalid-input {:value "42", :schema :int, :explain {:schema :int, :value "42", :errors ({:path [], :in [], :schema :int, :value "42"})}}

Advanced Transformations

Transformations are recursive:

(m/decode

Address

{:id "Lillan",

:tags ["coffee" "artesan" "garden"],

:address {:street "Ahlmanintie 29"

:city "Tampere"

:zip 33100

:lonlat [61.4858322 23.7854658]}}

mt/json-transformer)

;{:id "Lillan",

; :tags #{:coffee :artesan :garden},

; :address {:street "Ahlmanintie 29"

;           :city "Tampere"

;           :zip 33100

;           :lonlat [61.4858322 23.7854658]}}

Transform map keys:

(m/encode

Address

{:id "Lillan",

:tags ["coffee" "artesan" "garden"],

:address {:street "Ahlmanintie 29"

:city "Tampere"

:zip 33100

:lonlat [61.4858322 23.7854658]}}

(mt/key-transformer {:encode name}))

;{"id" "Lillan",

; "tags" ["coffee" "artesan" "garden"],

; "address" {"street" "Ahlmanintie 29"

;            "city" "Tampere"

;            "zip" 33100

;            "lonlat" [61.4858322 23.7854658]}}

Transforming homogenous :enum or :=s (supports automatic type detection of :keyword, :symbol, :int and :double):

(m/decode [:enum :kikka :kukka] "kukka" mt/string-transformer)

; => :kukka

Transformers can be composed with mt/transformer:

(def strict-json-transformer

(mt/transformer

mt/strip-extra-keys-transformer

mt/json-transformer))

(m/decode

Address

{:id "Lillan",

:EVIL "LYN"

:tags ["coffee" "artesan" "garden"],

:address {:street "Ahlmanintie 29"

:DARK "ORKO"

:city "Tampere"

:zip 33100

:lonlat [61.4858322 23.7854658]}}

strict-json-transformer)

;{:id "Lillan",

; :tags #{:coffee :artesan :garden},

; :address {:street "Ahlmanintie 29"

;           :city "Tampere"

;           :zip 33100

;           :lonlat [61.4858322 23.7854658]}}

Schema properties can be used to override default transformations:

(m/decode

[string? {:decode/string clojure.string/upper-case}]

"kerran" mt/string-transformer)

; => "KERRAN"

This works too:

(m/decode

[string? {:decode {:string clojure.string/upper-case}}]

"kerran" mt/string-transformer)

; => "KERRAN"

Decoders and encoders as interceptors (with :enter and :leave stages):

(m/decode

[string? {:decode/string {:enter clojure.string/upper-case}}]

"kerran" mt/string-transformer)

; => "KERRAN"

(m/decode

[string? {:decode/string {:enter #(str "olipa_" %)

:leave #(str % "_avaruus")}}]

"kerran" mt/string-transformer)

; => "olipa_kerran_avaruus"

To access Schema (and options) use :compile:

(m/decode

[int? {:math/multiplier 10

:decode/math {:compile (fn [schema _]

(let [multiplier (:math/multiplier (m/properties schema))]

(fn [x] (* x multiplier))))}}]

12

(mt/transformer {:name :math}))

; => 120

Going crazy:

(m/decode

[:map

{:decode/math {:enter #(update % :x inc)

:leave #(update % :x (partial * 2))}}

[:x [int? {:decode/math {:enter (partial + 2)

:leave (partial * 3)}}]]]

{:x 1}

(mt/transformer {:name :math}))

; => {:x 24}

To and from JSON

The m/encode and m/decode functions work on clojure data. To go from clojure data to JSON, you need a JSON library like jsonista. Additionally, since m/decode doesn't check the schema, you need to run m/validate (or m/explain) if you want to make sure your data conforms to your schema.

To JSON:

(def Tags

(m/schema [:map

{:closed true}

[:tags [:set :keyword]]]))

(jsonista.core/write-value-as-string

(m/encode Tags

{:tags #{:bar :quux}}

mt/json-transformer))

; => "{\"tags\":[\"bar\",\"quux\"]}"

From JSON without validation:

(m/decode Tags

(jsonista.core/read-value "{\"tags\":[\"bar\",[\"quux\"]]}"

jsonista.core/keyword-keys-object-mapper)

mt/json-transformer)

; => {:tags #{:bar ["quux"]}}

From JSON with validation:

(m/explain Tags

(m/decode Tags

(jsonista.core/read-value "{\"tags\":[\"bar\",[\"quux\"]]}"

jsonista.core/keyword-keys-object-mapper)

mt/json-transformer))

; => {:schema [:map {:closed true} [:tags [:set :keyword]]],

;     :value {:tags #{:bar ["quux"]}},

;     :errors ({:path [:tags 0], :in [:tags ["quux"]], :schema :keyword, :value ["quux"]})}

(m/validate Tags

(m/decode Tags

(jsonista.core/read-value "{\"tags\":[\"bar\",\"quux\"]}" ; <- note! no error

jsonista.core/keyword-keys-object-mapper)

mt/json-transformer))

; => true

For performance, it's best to prebuild the validator, decoder and explainer:

(def validate-Tags (m/validator Tags))

(def decode-Tags (m/decoder Tags mt/json-transformer))

(-> (jsonista.core/read-value "{\"tags\":[\"bar\",\"quux\"]}"

jsonista.core/keyword-keys-object-mapper)

decode-Tags

validate-Tags)

; => true

Default values

Applying default values:

(m/decode [:and {:default 42} int?] nil mt/default-value-transformer)

; => 42

With custom key and type defaults:

(m/decode

[:map

[:user [:map

[:name :string]

[:description {:ui/default "-"} :string]]]]

nil

(mt/default-value-transformer

{:key :ui/default

:defaults {:map (constantly {})

:string (constantly "")}}))

; => {:user {:name "", :description "-"}}

With custom function:

(m/decode

[:map

[:os [:string {:property "os.name"}]]

[:timezone [:string {:property "user.timezone"}]]]

{}

(mt/default-value-transformer

{:key :property

:default-fn (fn [_ x] (System/getProperty x))}))

; => {:os "Mac OS X", :timezone "Europe/Helsinki"}

Optional Keys are not added by default:

(m/decode

[:map

[:name [:string {:default "kikka"}]]

[:description {:optional true} [:string {:default "kikka"}]]]

{}

(mt/default-value-transformer))

; => {:name "kikka"}

Adding optional keys too via ::mt/add-optional-keys option:

(m/decode

[:map

[:name [:string {:default "kikka"}]]

[:description {:optional true} [:string {:default "kikka"}]]]

{}

(mt/default-value-transformer {::mt/add-optional-keys true}))

; => {:name "kikka", :description "kikka"}

Single sweep of defaults & string encoding:

(m/encode

[:map {:default {}}

[:a [int? {:default 1}]]

[:b [:vector {:default [1 2 3]} int?]]

[:c [:map {:default {}}

[:x [int? {:default 42}]]

[:y int?]]]

[:d [:map

[:x [int? {:default 42}]]

[:y int?]]]

[:e int?]]

nil

(mt/transformer

mt/default-value-transformer

mt/string-transformer))

;{:a "1"

; :b ["1" "2" "3"]

; :c {:x "42"}}

Programming with schemas

(require '[malli.util :as mu])

Updating Schema properties:

(mu/update-properties [:vector int?] assoc :min 1)

; => [:vector {:min 1} int?]

Lifted clojure.core function to work with schemas: select-keys, dissoc, get, assoc, update, get-in, assoc-in, update-in

(mu/get-in Address [:address :lonlat])

; => [:tuple double? double?]

(mu/update-in Address [:address] mu/assoc :country [:enum "fi" "po"])

;[:map

; [:id string?]

; [:tags [:set keyword?]]

; [:address

;  [:map [:street string?]

;   [:city string?]

;   [:zip int?]

;   [:lonlat [:tuple double? double?]]

;   [:country [:enum "fi" "po"]]]]]

(-> Address

(mu/dissoc :address)

(mu/update-properties assoc :title "Address"))

;[:map {:title "Address"}

; [:id string?]

; [:tags [:set keyword?]]]

Making keys optional or required:

(mu/optional-keys [:map [:x int?] [:y int?]])

;[:map

; [:x {:optional true} int?]

; [:y {:optional true} int?]]

(mu/required-keys [:map [:x {:optional true} int?] [:y int?]])

;[:map

; [:x int?]

; [:y int?]]

Closing and opening all :map schemas recursively:

(def abcd

[:map {:title "abcd"}

[:a int?]

[:b {:optional true} int?]

[:c [:map

[:d int?]]]])

(mu/closed-schema abcd)

;[:map {:title "abcd", :closed true}

; [:a int?]

; [:b {:optional true} int?]

; [:c [:map {:closed true}

;      [:d int?]]]]

(-> abcd

mu/closed-schema

mu/open-schema)

;[:map {:title "abcd"}

; [:a int?]

; [:b {:optional true} int?]

; [:c [:map

;      [:d int?]]]]

Merging Schemas (last value wins):

(mu/merge

[:map

[:name string?]

[:description string?]

[:address

[:map

[:street string?]

[:country [:enum "finland" "poland"]]]]]

[:map

[:description {:optional true} string?]

[:address

[:map

[:country string?]]]])

;[:map

; [:name string?]

; [:description {:optional true} string?]

; [:address [:map

;            [:street string?]

;            [:country string?]]]]

With :and, first child is used in merge:

(mu/merge

[:and {:type "entity"}

[:map {:title "user"}

[:name :string]]

map?]

[:map {:description "aged"} [:age :int]])

;[:and {:type "entity"}

; [:map {:title "user", :description "aged"}

;  [:name :string]

;  [:age :int]]

; map?]

Schema unions (merged values of both schemas are valid for union schema):

(mu/union

[:map

[:name string?]

[:description string?]

[:address

[:map

[:street string?]

[:country [:enum "finland" "poland"]]]]]

[:map

[:description {:optional true} string?]

[:address

[:map

[:country string?]]]])

;[:map

; [:name string?]

; [:description {:optional true} string?]

; [:address [:map

;            [:street string?]

;            [:country [:or [:enum "finland" "poland"] string?]]]]]

Adding generated example values to Schemas:

(m/walk

[:map

[:name string?]

[:description string?]

[:address

[:map

[:street string?]

[:country [:enum "finland" "poland"]]]]]

(m/schema-walker

(fn [schema]

(mu/update-properties schema assoc :examples (mg/sample schema {:size 2, :seed 20})))))

;[:map

; {:examples ({:name "", :description "", :address {:street "", :country "poland"}}

;             {:name "W", :description "x", :address {:street "8", :country "finland"}})}

; [:name [string? {:examples ("" "")}]]

; [:description [string? {:examples ("" "")}]]

; [:address

;  [:map

;   {:examples ({:street "", :country "finland"} {:street "W", :country "poland"})}

;   [:street [string? {:examples ("" "")}]]

;   [:country [:enum {:examples ("finland" "poland")} "finland" "poland"]]]]]

Finding first value (prewalk):

(mu/find-first

[:map

[:x int?]

[:y [:vector [:tuple

[:or [:and {:salaisuus "turvassa"} boolean?] int?]

[:schema {:salaisuus "vaarassa"} false?]]]]

[:z [:string {:salaisuus "piilossa"}]]]

(fn [schema _ _]

(-> schema m/properties :salaisuus)))

; => "turvassa"

Finding all subschemas with paths, retaining order:

(def Schema

(m/schema

[:maybe

[:map

[:id string?]

[:tags [:set keyword?]]

[:address

[:and

[:map

[:street {:optional true} string?]

[:lonlat {:optional true} [:tuple double? double?]]]

[:fn (fn [{:keys [street lonlat]}] (or street lonlat))]]]]]))

(mu/subschemas Schema)

;[{:path [], :in [], :schema [:maybe

;                             [:map

;                              [:id string?]

;                              [:tags [:set keyword?]]

;                              [:address

;                               [:and

;                                [:map

;                                 [:street {:optional true} string?]

;                                 [:lonlat {:optional true} [:tuple double? double?]]]

;                                [:fn (fn [{:keys [street lonlat]}] (or street lonlat))]]]]]}

; {:path [0], :in [], :schema [:map

;                              [:id string?]

;                              [:tags [:set keyword?]]

;                              [:address

;                               [:and

;                                [:map

;                                 [:street {:optional true} string?]

;                                 [:lonlat {:optional true} [:tuple double? double?]]]

;                                [:fn (fn [{:keys [street lonlat]}] (or street lonlat))]]]]}

; {:path [0 :id], :in [:id], :schema string?}

; {:path [0 :tags], :in [:tags], :schema [:set keyword?]}

; {:path [0 :tags :malli.core/in], :in [:tags :malli.core/in], :schema keyword?}

; {:path [0 :address], :in [:address], :schema [:and

;                                               [:map

;                                                [:street {:optional true} string?]

;                                                [:lonlat {:optional true} [:tuple double? double?]]]

;                                               [:fn (fn [{:keys [street lonlat]}] (or street lonlat))]]}

; {:path [0 :address 0], :in [:address], :schema [:map

;                                                 [:street {:optional true} string?]

;                                                 [:lonlat {:optional true} [:tuple double? double?]]]}

; {:path [0 :address 0 :street], :in [:address :street], :schema string?}

; {:path [0 :address 0 :lonlat], :in [:address :lonlat], :schema [:tuple double? double?]}

; {:path [0 :address 0 :lonlat 0], :in [:address :lonlat 0], :schema double?}

; {:path [0 :address 0 :lonlat 1], :in [:address :lonlat 1], :schema double?}

; {:path [0 :address 1], :in [:address], :schema [:fn (fn [{:keys [street lonlat]}] (or street lonlat))]}]

Collecting unique value paths and their schema paths:

(->> Schema

(mu/subschemas)

(mu/distinct-by :id)

(mapv (juxt :in :path)))

;[[[] []]

; [[] [0]]

; [[:id] [0 :id]]

; [[:tags] [0 :tags]]

; [[:tags :malli.core/in] [0 :tags :malli.core/in]]

; [[:address] [0 :address]]

; [[:address] [0 :address 0]]

; [[:address :street] [0 :address 0 :street]]

; [[:address :lonlat] [0 :address 0 :lonlat]]

; [[:address :lonlat 0] [0 :address 0 :lonlat 0]]

; [[:address :lonlat 1] [0 :address 0 :lonlat 1]]

; [[:address] [0 :address 1]]]

Schema paths can be converted into value paths:

(mu/get-in Schema [0 :address 0 :lonlat])

; => [:tuple double? double?]

(mu/path->in Schema [0 :address 0 :lonlat])

; => [:address :lonlat]

and back, returning all paths:

(mu/in->paths Schema [:address :lonlat])

; => [[0 :address 0 :lonlat]]

Declarative schema transformation

There are also declarative versions of schema transforming utilities in malli.util/schemas. These include :merge, :union and :select-keys:

(def registry (merge (m/default-schemas) (mu/schemas)))

(def Merged

(m/schema

[:merge

[:map [:x :string]]

[:map [:y :int]]]

{:registry registry}))

Merged

;[:merge

; [:map [:x :string]]

; [:map [:y :int]]]

(m/deref Merged)

;[:map

; [:x :string]

; [:y :int]]

(m/validate Merged {:x "kikka", :y 6})

; => true

Persisting schemas

Writing and Reading schemas as EDN, no eval needed.

Following example requires sci as external dependency because it includes a function definition. See Serializable functions.

(require '[malli.edn :as edn])

(-> [:and

[:map

[:x int?]

[:y int?]]

[:fn '(fn [{:keys [x y]}] (> x y))]]

(edn/write-string)

(doto prn) ; => "[:and [:map [:x int?] [:y int?]] [:fn (fn [{:keys [x y]}] (> x y))]]"

(edn/read-string)

(doto (-> (m/validate {:x 0, :y 1}) prn)) ; => false

(doto (-> (m/validate {:x 2, :y 1}) prn))) ; => true

;[:and

; [:map

;  [:x int?]

;  [:y int?]]

; [:fn (fn [{:keys [x y]}] (> x y))]]

Multi schemas

Closed dispatch with :multi schema and :dispatch property:

(m/validate

[:multi {:dispatch :type}

[:sized [:map [:type keyword?] [:size int?]]]

[:human [:map [:type keyword?] [:name string?] [:address [:map [:country keyword?]]]]]]

{:type :sized, :size 10})

; true

Default branch with ::m/default:

(def valid?

(m/validator

[:multi {:dispatch :type}

["object" [:map-of :keyword :string]]

[::m/default :string]]))

(valid? {:type "object", :key "1", :value "100"})

; => true

(valid? "SUCCESS!")

; => true

(valid? :failure)

; => false

Any function can be used for :dispatch:

(m/validate

[:multi {:dispatch first}

[:sized [:tuple keyword? [:map [:size int?]]]]

[:human [:tuple keyword? [:map [:name string?] [:address [:map [:country keyword?]]]]]]]

[:human {:name "seppo", :address {:country :sweden}}])

; true

:dispatch values should be decoded before actual values:

(m/decode

[:multi {:dispatch :type

:decode/string #(update % :type keyword)}

[:sized [:map [:type [:= :sized]] [:size int?]]]

[:human [:map [:type [:= :human]] [:name string?] [:address [:map [:country keyword?]]]]]]

{:type "human"

:name "Tiina"

:age "98"

:address {:country "finland"

:street "this is an extra key"}}

(mt/transformer mt/strip-extra-keys-transformer mt/string-transformer))

;{:type :human

; :name "Tiina"

; :address {:country :finland}}

Recursive schemas

To create a recursive schema, introduce a local registry and wrap all recursive positions in the registry with :ref. Now you may reference the recursive schemas in the body of the schema.

For example, here is a recursive schema using :schema for singly-linked lists of positive integers:

(m/validate

[:schema {:registry {::cons [:maybe [:tuple pos-int? [:ref ::cons]]]}}

[:ref ::cons]]

[16 [64 [26 [1 [13 nil]]]]])

; => true

Without the :ref keyword, malli eagerly expands the schema until a stack overflow error is thrown:

(m/validate

[:schema {:registry {::cons [:maybe [:tuple pos-int? ::cons]]}}

::cons]

[16 [64 [26 [1 [13 nil]]]]])

; StackOverflowError

Technically, you only need the :ref in recursive positions. However, it is best practice to :ref all references to recursive variables for better-behaving generators:

;; Note:

[:schema {:registry {::cons [:maybe [:tuple pos-int? [:ref ::cons]]]}}

::cons]

;; produces the same generator as the "unfolded"

[:maybe [:tuple pos-int? [:schema {:registry {::cons [:maybe [:tuple pos-int? [:ref ::cons]]]}} ::cons]]]

;; while

[:schema {:registry {::cons [:maybe [:tuple pos-int? [:ref ::cons]]]}}

[:ref ::cons]]

;; has a direct correspondance to the following generator:

(gen/recursive-gen

(fn [rec] (gen/one-of [(gen/return nil) (gen/tuple rec)]))

(gen/return nil))

Mutual recursion works too. Thanks to the :schema construct, many schemas could be defined in the local registry, the top-level one being promoted by the :schema second parameter:

(m/validate

[:schema {:registry {::ping [:maybe [:tuple [:= "ping"] [:ref ::pong]]]

::pong [:maybe [:tuple [:= "pong"] [:ref ::ping]]]}}

::ping]

["ping" ["pong" ["ping" ["pong" ["ping" nil]]]]])

; => true

Nested registries, the last definition wins:

(m/validate

[:schema {:registry {::ping [:maybe [:tuple [:= "ping"] [:ref ::pong]]]

::pong any?}} ;; effectively unreachable

[:schema {:registry {::pong [:maybe [:tuple [:= "pong"] [:ref ::ping]]]}}

::ping]]

["ping" ["pong" ["ping" ["pong" ["ping" nil]]]]])

; => true

Value generation

Schemas can be used to generate values:

(require '[malli.generator :as mg])

;; random

(mg/generate keyword?)

; => :?

;; using seed

(mg/generate [:enum "a" "b" "c"] {:seed 42})

;; => "a"

;; using seed and size

(mg/generate pos-int? {:seed 10, :size 100})

;; => 55740

;; regexs work too (only clj and if [com.gfredericks/test.chuck "0.2.10"+] available)

(mg/generate

[:re #"^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,63}$"]

{:seed 42, :size 10})

; => "CaR@MavCk70OHiX.yZ"

;; :gen/elements (note, are not validated)

(mg/generate

[:and {:gen/elements ["kikka" "kukka" "kakka"]} string?]

{:seed 10})

; => "kikka"

;; :gen/fmap

(mg/generate

[:and {:gen/fmap (partial str "kikka_")} string?]

{:seed 10, :size 10})

;; => "kikka_WT3K0yax2"

;; portable :gen/fmap (requires `org.babashka/sci` dependency to work)

(mg/generate

[:and {:gen/fmap '(partial str "kikka_")} string?]

{:seed 10, :size 10})

;; => "kikka_nWT3K0ya7"

;; :gen/schema

(mg/generate

[:any {:gen/schema [:int {:min 10, :max 20}]}]

{:seed 10})

; => 19

;; :gen/min & :gen/max for numbers and collections

(mg/generate

[:vector {:gen/min 4, :gen/max 4} :int]

{:seed 1})

; => [-8522515 -1433 -1 1]

;; :gen/infinite? & :gen/NaN? for :double

(mg/generate

[:double {:gen/infinite? true, :gen/NaN? true}]

{:seed 1})

; => ##Inf

(require '[clojure.test.check.generators :as gen])

;; gen/gen (note, not serializable)

(mg/generate

[:sequential {:gen/gen (gen/list gen/neg-int)} int?]

{:size 42, :seed 42})

; => (-37 -13 -13 -24 -20 -11 -34 -40 -22 0 -10)

Generated values are valid:

(mg/generate Address {:seed 123, :size 4})

;{:id "H7",

; :tags #{:v?.w.t6!.QJYk-/-?s*4

;         :_7U

;         :QdG/Xi8J

;         :*Q-.p*8*/n-J9u}

; :address {:street "V9s"

;           :city ""

;           :zip 3

;           :lonlat [-2.75 -0.625]}}

(m/validate Address (mg/generate Address))

; => true

Sampling values:

;; sampling

(mg/sample [:and int? [:> 10] [:< 100]] {:seed 123})

; => (25 39 51 13 53 43 57 15 26 27)

Integration with test.check:

(require '[clojure.test.check.generators :as gen])

(gen/sample (mg/generator pos-int?))

; => (2 1 2 2 2 2 8 1 55 83)

Inferring schemas

Inspired by F# Type providers:

(require '[malli.provider :as mp])

(def samples

[{:id "Lillan"

:tags #{:artesan :coffee :hotel}

:address {:street "Ahlmanintie 29"

:city "Tampere"

:zip 33100

:lonlat [61.4858322, 23.7854658]}}

{:id "Huber",

:description "Beefy place"

:tags #{:beef :wine :beer}

:address {:street "Aleksis Kiven katu 13"

:city "Tampere"

:zip 33200

:lonlat [61.4963599 23.7604916]}}])

(mp/provide samples)

;[:map

; [:id :string]

; [:tags [:set :keyword]]

; [:address

;  [:map

;   [:street :string]

;   [:city :string]

;   [:zip :int]

;   [:lonlat [:vector :double]]]]

; [:description {:optional true} :string]]

All samples are valid against the inferred schema:

(every? (partial m/validate (mp/provide samples)) samples)

; => true

For better performance, use mp/provider:

(require '[criterium.core :as p])

;; 5ms

(p/bench (mp/provide samples))

;; 500µs (10x)

(let [provider (mp/provider)]

(p/bench (provider samples)))

:map-of inferring

Inferring :map-of requires 8 identical key & value schemas:

(mp/provide

[{:a [1]

:b [1 2]

:c [1 2 3]

:d [1 2]

:e [1 2 3]

:f [1]

:g [1]

:h [1 2 3 4]}])

; [:map-of :keyword [:vector :int]]

This can be configured via ::mp/map-of-threshold options:

(mp/provide

[{:a [1]

:b [1 2]}]

{::mp/map-of-threshold 2})

; [:map-of :keyword [:vector :int]]

Sample-data can be type-hinted with ::mp/hint:

(mp/provide

[^{::mp/hint :map-of}

{:a {:b 1, :c 2}

:b {:b 2, :c 1}

:c {:b 3}

:d nil}])

;[:map-of

; :keyword

; [:maybe [:map

;          [:b :int]

;          [:c {:optional true} :int]]]]

:tuple inferring

By default, tuples are not inferred:

(mp/provide

[[1 "kikka" true]

[2 "kukka" true]

[3 "kakka" true]])

; [:vector :some]

There is ::mp/tuple-threshold option:

(mp/provide

[[1 "kikka" true]

[2 "kukka" true]

[3 "kakka" false]]

{::mp/tuple-threshold 3})

; [:tuple :int :string :boolean]

Sample-data can be type-hinted with ::mp/hint:

(mp/provide

[^{::mp/hint :tuple}

[1 "kikka" true]

["2" "kukka" true]])

; [:tuple :some string? boolean?]

value decoding in inferring

By default, no decoding is applied for (leaf) values:

(mp/provide

[{:id "caa71a26-5fe1-11ec-bf63-0242ac130002"}

{:id "8aadbf5e-5fe3-11ec-bf63-0242ac130002"}])

; => [:map [:id string?]]

Adding custom decoding via ::mp/value-decoders option:

(mp/provide

[{:id "caa71a26-5fe1-11ec-bf63-0242ac130002"

:time "2021-01-01T00:00:00Z"}

{:id "8aadbf5e-5fe3-11ec-bf63-0242ac130002"

:time "2022-01-01T00:00:00Z"}]

{::mp/value-decoders {'string? {:uuid mt/-string->uuid

'inst? mt/-string->date}}})

; => [:map [:id :uuid] [:time inst?]

Destructuring

Schemas can also be inferred from Clojure Destructuring Syntax.

(require '[malli.destructure :as md])

(def infer (comp :schema md/parse))

(infer '[a b & cs])

; => [:cat :any :any [:* :any]]

Malli also supports adding type hints as an extension to the normal Clojure syntax (enabled by default), inspired by Plumatic Schema.

(infer '[a :- :int, b :- :string & cs :- [:* :boolean]])

; => [:cat :int :string [:* :boolean]]

Pulling out function argument schemas from Vars:

(defn kikka

([a] [a])

([a b & cs] [a b cs]))

(md/infer #'kikka)

;[:function

; [:=> [:cat :any] :any]

; [:=> [:cat :any :any [:* :any]] :any]]

md/parse uses the following options:

key

description

::md/inline-schemas

support plumatic-style inline schemas (true)

::md/sequential-maps

support sequential maps in non-rest position (true)

::md/references

qualified schema references used (true)

::md/required-keys

destructured keys are required (false)

::md/closed-maps

destructured maps are closed (false)

A more complete example:

(infer '[a [b c & rest :as bc]

& {:keys [d e]

:demo/keys [f]

g :demo/g

[h] :h

:or {d 0}

:as opts}])

;[:cat

; :any

; [:maybe [:cat

;          [:? :any]

;          [:? :any]

;          [:* :any]]]

; [:altn

;  [:map

;   [:map

;    [:d {:optional true} :any]

;    [:e {:optional true} :any]

;    [:demo/f {:optional true}]

;    [:demo/g {:optional true}]

;    [:h {:optional true} [:maybe [:cat

;                                  [:? :any]

;                                  [:* :any]]]]]]

;  [:args

;   [:*

;    [:alt

;     [:cat [:= :d] :any]

;     [:cat [:= :e] :any]

;     [:cat [:= :demo/f] :demo/f]

;     [:cat [:= :demo/g] :demo/g]

;     [:cat [:= :h] [:maybe [:cat

;                            [:? :any]

;                            [:* :any]]]]

;     [:cat :any :any]]]]]]

Parsing values

Schemas can be used to parse values using m/parse and m/parser:

m/parse for one-time things:

(m/parse

[:* [:catn

[:prop string?]

[:val [:altn

[:s string?]

[:b boolean?]]]]]

["-server" "foo" "-verbose" true "-user" "joe"])

;[{:prop "-server", :val [:s "foo"]}

; {:prop "-verbose", :val [:b true]}

; {:prop "-user", :val [:s "joe"]}]

m/parser to create an optimized parser:

(def Hiccup

[:schema {:registry {"hiccup" [:orn

[:node [:catn

[:name keyword?]

[:props [:? [:map-of keyword? any?]]]

[:children [:* [:schema [:ref "hiccup"]]]]]]

[:primitive [:orn

[:nil nil?]

[:boolean boolean?]

[:number number?]

[:text string?]]]]}}

"hiccup"])

(def parse-hiccup (m/parser Hiccup))

(parse-hiccup

[:div {:class [:foo :bar]}

[:p "Hello, world of data"]])

;[:node

; {:name :div

;  :props {:class [:foo :bar]}

;  :children [[:node

;              {:name :p

;               :props nil

;               :children [[:primitive [:text "Hello, world of data"]]]}]]}]

Parsing returns tagged values for :orn, :catn, :altn and :multi.

(def Multi

[:multi {:dispatch :type}

[:user [:map [:size :int]]]

[::m/default :any]])

(m/parse Multi {:type :user, :size 1})

; => [:user {:type :user, :size 1}]

(m/parse Multi {:type "sized", :size 1})

; => [:malli.core/default {:type "sized", :size 1}]

Unparsing values

The inverse of parsing, using m/unparse and m/unparser:

(->> [:div {:class [:foo :bar]}

[:p "Hello, world of data"]]

(m/parse Hiccup)

(m/unparse Hiccup))

;[:div {:class [:foo :bar]}

; [:p "Hello, world of data"]]

Serializable functions

Enabling serializable function schemas requires sci as external dependency. If it is not present, the malli function evaluator throws :sci-not-available exception.

For ClojureScript, you also need to require sci.core manually, either directly or via :preloads.

For GraalVM, you need to require sci.core manually, before requiring any malli namespaces.

(def my-schema

[:and

[:map

[:x int?]

[:y int?]]

[:fn '(fn [{:keys [x y]}] (> x y))]])

(m/validate my-schema {:x 1, :y 0})

; => true

(m/validate my-schema {:x 1, :y 2})

; => false

NOTE: sci is not termination safe so be wary of sci functions from untrusted sources. You can explicitly disable sci with option ::m/disable-sci and set the default options with ::m/sci-options.

(m/validate [:fn 'int?] 1 {::m/disable-sci true})

; Execution error

; :malli.core/sci-not-available {:code int?}

Schema AST

Implemented with protocol malli.core/AST. Allows lossless round-robin with faster schema creation.

NOTE: For now, the AST syntax in considered as internal, e.g. don't use it as a database persistency model.

(def ?schema

[:map

[:x boolean?]

[:y {:optional true} int?]

[:z [:map

[:x boolean?]

[:y {:optional true} int?]]]])

(m/form ?schema)

;[:map

; [:x boolean?]

; [:y {:optional true} int?]

; [:z [:map

;      [:x boolean?]

;      [:y {:optional true} int?]]]]

(m/ast ?schema)

;{:type :map,

; :keys {:x {:order 0

;            :value {:type boolean?}},

;        :y {:order 1, :value {:type int?}

;            :properties {:optional true}},

;        :z {:order 2,

;            :value {:type :map,

;                    :keys {:x {:order 0

;                               :value {:type boolean?}},

;                           :y {:order 1

;                               :value {:type int?}

;                               :properties {:optional true}}}}}}}

(-> ?schema

(m/schema) ;; 3.4µs

(m/ast)

(m/from-ast) ;; 180ns (18x, lazy)

(m/form)

(= (m/form ?schema)))

; => true

Schema transformation

Schemas can be transformed using post-walking, e.g. the Visitor Pattern.

The identity walker:

(m/walk

Address

(m/schema-walker identity))

;[:map

; [:id string?]

; [:tags [:set keyword?]]

; [:address

;  [:map

;   [:street string?]

;   [:city string?]

;   [:zip int?]

;   [:lonlat [:tuple double? double?]]]]]

Adding :title property to schemas:

(m/walk

Address

(m/schema-walker #(mu/update-properties % assoc :title (name (m/type %)))))

;[:map {:title "map"}

; [:id [string? {:title "string?"}]]

; [:tags [:set {:title "set"} [keyword? {:title "keyword?"}]]]

; [:address

;  [:map {:title "map"}

;   [:street [string? {:title "string?"}]]

;   [:city [string? {:title "string?"}]]

;   [:zip [int? {:title "int?"}]]

;   [:lonlat [:tuple {:title "tuple"} [double? {:title "double?"}] [double? {:title "double?"}]]]]]]

Transforming schemas into maps:

(m/walk

Address

(fn [schema _ children _]

(-> (m/properties schema)

(assoc :malli/type (m/type schema))

(cond-> (seq children) (assoc :malli/children children)))))

;{:malli/type :map,

; :malli/children [[:id nil {:malli/type string?}]

;                  [:tags nil {:malli/type :set

;                              :malli/children [{:malli/type keyword?}]}]

;                  [:address nil {:malli/type :map,

;                                 :malli/children [[:street nil {:malli/type string?}]

;                                                  [:city nil {:malli/type string?}]

;                                                  [:zip nil {:malli/type int?}]

;                                                  [:lonlat nil {:malli/type :tuple

;                                                                :malli/children [{:malli/type double?}

;                                                                                 {:malli/type double?}]}]]}]]}

JSON Schema

Transforming Schemas into JSON Schema:

(require '[malli.json-schema :as json-schema])

(json-schema/transform Address)

;{:type "object",

; :properties {:id {:type "string"},

;              :tags {:type "array"

;                     :items {:type "string"}

;                     :uniqueItems true},

;              :address {:type "object",

;                        :properties {:street {:type "string"},

;                                     :city {:type "string"},

;                                     :zip {:type "integer", :format "int64"},

;                                     :lonlat {:type "array",

;                                              :items [{:type "number"} {:type "number"}],

;                                              :additionalItems false}},

;                        :required [:street :city :zip :lonlat]}},

; :required [:id :tags :address]}

Custom transformation via :json-schema namespaced properties:

(json-schema/transform

[:enum

{:title "Fish"

:description "It's a fish"

:json-schema/type "string"

:json-schema/default "perch"}

"perch" "pike"])

;{:title "Fish"

; :description "It's a fish"

; :type "string"

; :default "perch"

; :enum ["perch" "pike"]}

Full override with :json-schema property:

(json-schema/transform

[:map {:json-schema {:type "file"}}

[:file any?]])

; {:type "file"}

Swagger2

Transforming Schemas into Swagger2 Schema:

(require '[malli.swagger :as swagger])

(swagger/transform Address)

;{:type "object",

; :properties {:id {:type "string"},

;              :tags {:type "array"

;                     :items {:type "string"}

;                     :uniqueItems true},

;              :address {:type "object",

;                        :properties {:street {:type "string"},

;                                     :city {:type "string"},

;                                     :zip {:type "integer", :format "int64"},

;                                     :lonlat {:type "array",

;                                              :items {},

;                                              :x-items [{:type "number", :format "double"}

;                                                        {:type "number", :format "double"}]}},

;                        :required [:street :city :zip :lonlat]}},

; :required [:id :tags :address]}

Custom transformation via :swagger and :json-schema namespaced properties:

(swagger/transform

[:enum

{:title "Fish"

:description "It's a fish"

:swagger/type "string"

:json-schema/default "perch"}

"perch" "pike"])

;{:title "Fish"

; :description "It's a fish"

; :type "string"

; :default "perch"

; :enum ["perch" "pike"]}

Full override with :swagger property:

(swagger/transform

[:map {:swagger {:type "file"}}

[:file any?]])

; {:type "file"}

Custom schema types

Schema Types are described using m/IntoSchema protocol, which has a factory method (-into-schema [this properties children options]) to create the actual Schema instances. See malli.core for example implementations.

Simple schema

For simple cases, there is m/-simple-schema:

(require '[clojure.test.check.generators :as gen])

(def Over6

(m/-simple-schema

{:type :user/over6

:pred #(and (int? %) (> % 6))

:type-properties {:error/message "should be over 6"

:decode/string mt/-string->long

:json-schema/type "integer"

:json-schema/format "int64"

:json-schema/minimum 6

:gen/gen (gen/large-integer* {:min 7})}}))

(m/into-schema? Over6)

; => true

m/IntoSchema can be both used as Schema (creating a Schema instance with nil properties and children) and as Schema type to create new Schema instances without needing to register the types:

(m/schema? (m/schema Over6))

; => true

(m/schema? (m/schema [Over6 {:title "over 6"}]))

; => true

:pred is used for validation:

(m/validate Over6 2)

; => false

(m/validate Over6 7)

; => true

:type-properties are shared for all schema instances and are used just like Schema (instance) properties by many Schema applications, including error messages, value generation and json-schema transformations.

(json-schema/transform Over6)

; => {:type "integer", :format "int64", :minimum 6}

(json-schema/transform [Over6 {:json-schema/example 42}])

; => {:type "integer", :format "int64", :minimum 6, :example 42}

Content dependent simple schema

You can also build content-dependent schemas by using a callback function of properties children -> opts instead of static opts:

(def Over

(m/-simple-schema

(fn [{:keys [value]} _]

(assert (int? value))

{:type :user/over

:pred #(and (int? %) (> % value))

:type-properties {:error/fn (fn [error _] (str "should be over " value ", was " (:value error)))

:decode/string mt/-string->long

:json-schema/type "integer"

:json-schema/format "int64"

:json-schema/minimum value

:gen/gen (gen/large-integer* {:min (inc value)})}})))

(-> [Over {:value 12}]

(m/explain 10)

(me/humanize))

; => ["should be over 12, was 10"]

Schema registry

Schemas are looked up using a malli.registry/Registry protocol, which is effectively a map from schema type to a schema recipe (Schema, IntoSchema or vector-syntax schema). Maps can also be used as a registry.

Custom Registry can be passed into all/most malli public APIs via the optional options map using :registry key. If omitted, malli.core/default-registry is used.

;; the default registry

(m/validate [:maybe string?] "kikka")

; => true

;; registry as explicit options

(m/validate [:maybe string?] "kikka" {:registry m/default-registry})

; => true

The default immutable registry is merged from multiple parts, enabling easy re-composition of custom schema sets. See built-in schemas for list of all Schemas.

Custom registry

Here's an example to create a custom registry without the default core predicates and with :neg-int and :pos-int Schemas:

(def registry

(merge

(m/class-schemas)

(m/comparator-schemas)

(m/base-schemas)

{:neg-int (m/-simple-schema {:type :neg-int, :pred neg-int?})

:pos-int (m/-simple-schema {:type :pos-int, :pred pos-int?})}))

(m/validate [:or :pos-int :neg-int] 'kikka {:registry registry})

; => false

(m/validate [:or :pos-int :neg-int] 123 {:registry registry})

; => true

We did not register normal predicate schemas:

(m/validate pos-int? 123 {:registry registry})

; Syntax error (ExceptionInfo) compiling

; :malli.core/invalid-schema {:schema pos-int?}

Local registry

Any schema can define a local registry using :registry schema property:

(def Adult

[:map {:registry {::age [:and int? [:> 18]]}}

[:age ::age]])

(mg/generate Adult {:size 10, :seed 1})

; => {:age 92}

Local registries can be persisted:

(-> Adult

(malli.edn/write-string)

(malli.edn/read-string)

(m/validate {:age 46}))

; => true

See also Recursive Schemas.

Changing the default registry

Passing in custom options to all public methods is a lot of boilerplate. For the lazy, there is an easier way - we can swap the (global) default registry:

(require '[malli.registry :as mr])

;; the default registry

(-> m/default-registry (mr/schemas) (count))

;=> 140

;; global side-effects! free since 0.7.0!

(mr/set-default-registry!

{:string (m/-string-schema)

:maybe (m/-maybe-schema)

:map (m/-map-schema)})

(-> m/default-registry (mr/schemas) (count))

; => 3

(m/validate

[:map [:maybe [:maybe :string]]]

{:maybe "sheep"})

; => true

(m/validate :int 42)

; =throws=> :malli.core/invalid-schema {:schema :int}

NOTE: mr/set-default-registry! is an imperative api with global side-effects. Easy, but not simple. If you want to disable the api, you can define the following compiler/jvm bootstrap:

- cljs: :closure-defines {malli.registry/mode "strict"}

- clj: :jvm-opts ["-Dmalli.registry/mode=strict"]

DCE and schemas

The default schema registry is defined as a Var, so all Schema implementation (100+) are dragged in. For ClojureScript, this means the schemas implementations are not removed via Dead Code Elimination (DCE), resulting a large (37KB, zipped) js-bundle.

Malli allows the default registry to initialized with empty schemas, using the following compiler/jvm bootstrap:

- cljs: :closure-defines {malli.registry/type "custom"}

- clj: :jvm-opts ["-Dmalli.registry/type=custom"]

;; with the flag set on

(-> m/default-registry (mr/schemas) (count))

; => 0

With this, you can register just what you need and rest are DCE'd. The previous example results in just a 3KB gzip bundle.

Registry implementations

Malli supports multiple type of registries.

Immutable registry

Just a Map.

(require '[malli.registry :as mr])

(mr/set-default-registry!

{:string (m/-string-schema)

:maybe (m/-maybe-schema)

:map (m/-map-schema)})

(m/validate

[:map [:maybe [:maybe :string]]]

{:maybe "sheep"})

; => true

Mutable registry

clojure.spec introduces a mutable global registry for specs. The mutable registry in malli forced you to bring in your own state atom and functions how to work with it:

Using a custom registry atom:

(def registry*

(atom {:string (m/-string-schema)

:maybe (m/-maybe-schema)

:map (m/-map-schema)}))

(defn register! [type ?schema]

(swap! registry* assoc type ?schema))

(mr/set-default-registry!

(mr/mutable-registry registry*))

(register! :non-empty-string [:string {:min 1}])

(m/validate :non-empty-string "malli")

; => true

The mutable registry can also be passed in as an explicit option:

(def registry (mr/mutable-registry registry*))

(m/validate :non-empty-string "malli" {:registry registry})

; => true

Dynamic registry

If you know what you are doing, you can also use dynamic scope to pass in default schema registry:

(mr/set-default-registry!

(mr/dynamic-registry))

(binding [mr/*registry* {:string (m/-string-schema)

:maybe (m/-maybe-schema)

:map (m/-map-schema)

:non-empty-string [:string {:min 1}]}]

(m/validate :non-empty-string "malli"))

; => true

Lazy registries

You can provide schemas at runtime using mr/lazy-registry - it takes a local registry and a provider function of type registry -> schema as arguments:

(def registry

(mr/lazy-registry

(m/default-schemas)

(fn [type registry]

;; simulates pulling CloudFormation Schemas when needed

(let [lookup {"AWS::ApiGateway::UsagePlan" [:map {:closed true}

[:Type [:= "AWS::ApiGateway::UsagePlan"]]

[:Description {:optional true} string?]

[:UsagePlanName {:optional true} string?]]

"AWS::AppSync::ApiKey" [:map {:closed true}

[:Type [:= "AWS::AppSync::ApiKey"]]

[:ApiId string?]

[:Description {:optional true} string?]]}]

(println "... loaded" type)

(some-> type lookup (m/schema {:registry registry}))))))

;; lazy multi, doesn't realize the schemas

(def CloudFormation

(m/schema

[:multi {:dispatch :Type, :lazy-refs true}

"AWS::ApiGateway::UsagePlan"

"AWS::AppSync::ApiKey"]

{:registry registry}))

(m/validate

CloudFormation

{:Type "AWS::ApiGateway::UsagePlan"

:Description "laiskanlinna"})

; ... loaded AWS::ApiGateway::UsagePlan

; => true

(m/validate

CloudFormation

{:Type "AWS::ApiGateway::UsagePlan"

:Description "laiskanlinna"})

; => true

Composite registry

Registries can be composed, a full example:

(require '[malli.core :as m])

(require '[malli.registry :as mr])

(def registry (atom {}))

(defn register! [type schema]

(swap! registry assoc type schema))

(mr/set-default-registry!

;; linear search

(mr/composite-registry

;; immutable registry

{:map (m/-map-schema)}

;; mutable (spec-like) registry

(mr/mutable-registry registry)

;; on the perils of dynamic scope

(mr/dynamic-registry)))

;; mutate like a boss

(register! :maybe (m/-maybe-schema))

;; ☆.。.:*・°☆.。.:*・°☆.。.:*・°☆.。.:*・°☆

(binding [mr/*registry* {:string (m/-string-schema)}]

(m/validate

[:map [:maybe [:maybe :string]]]

{:maybe "sheep"}))

; => true

Function schemas

See Working with functions.

Instrumentation

See Instrumentation.

Clj-kondo

Clj-kondo is a linter for Clojure code that sparks joy.

Given functions and function Schemas:

(defn square [x] (* x x))

(m/=> square [:=> [:cat int?] nat-int?])

(defn plus

([x] x)

([x y] (+ x y)))

(m/=> plus [:function

[:=> [:cat int?] int?]

[:=> [:cat int? int?] int?]])

Generating clj-kondo configuration from current namespace:

(require '[malli.clj-kondo :as mc])

(-> (mc/collect *ns*) (mc/linter-config))

;{:lint-as #:malli.schema{defn schema.core/defn},

; :linters

; {:type-mismatch

;  {:namespaces

;   {user {square {:arities {1 {:args [:int]

;                               :ret :pos-int}}}

;          plus {:arities {1 {:args [:int]

;                             :ret :int},

;                          2 {:args [:int :int]

;                             :ret :int}}}}}}}}

Emitting confing into ./.clj-kondo/configs/malli/config.edn:

(mc/emit!)

In action:

malli

Static type checking via Typed Clojure

Typed Clojure is an optional type system for Clojure.

typed.malli can consume a subset of malli schema syntax to statically type check and infer Clojure code.

See this in action in the malli-type-providers example project.

(ns typed-example.malli-type-providers

(:require [typed.clojure :as t]

[malli.core :as m]))

;; just use malli instrumentation normally

(m/=> foo [:=> [:cat :int] :int])

;; Typed Clojure will statically check `foo` against its schema (after converting it to a type)

(defn foo [t] (inc t))

;; Typed Clojure will automatically infer `foo`s type from its schema

(foo 1)

(comment (t/check-ns-clj)) ;; check this ns

Visualizing schemas

DOT

Transforming Schemas into DOT Language:

(require '[malli.dot :as md])

(def Address

[:schema

{:registry {"Country" [:map

[:name [:enum :FI :PO]]

[:neighbors [:vector [:ref "Country"]]]]

"Burger" [:map

[:name string?]

[:description {:optional true} string?]

[:origin [:maybe "Country"]]

[:price pos-int?]]

"OrderLine" [:map

[:burger "Burger"]

[:amount int?]]

"Order" [:map

[:lines [:vector "OrderLine"]]

[:delivery [:map

[:delivered boolean?]

[:address [:map

[:street string?]

[:zip int?]

[:country "Country"]]]]]]}}

"Order"])

(md/transform Address)

; "digraph { ... }"

Visualized with Graphviz:

PlantUML

Transforming Schemas into PlantUML:

(require '[malli.plantuml :as plantuml])

(plantuml/transform Address)

; "@startuml ... @enduml"

Visualized with PlantText:

Lite

Simple syntax sugar, like data-specs, but for malli.

As the namespace suggests, it's experimental, built for reitit.

(require '[malli.experimental.lite :as l])

(l/schema

{:map1 {:x int?

:y [:maybe string?]

:z (l/maybe keyword?)}

:map2 {:min-max [:int {:min 0 :max 10}]

:tuples (l/vector (l/tuple int? string?))

:optional (l/optional (l/maybe :boolean))

:set-of-maps (l/set {:e int?

:f string?})

:map-of-int (l/map-of int? {:s string?})}})

;[:map

; [:map1

;  [:map

;   [:x int?]

;   [:y [:maybe string?]]

;   [:z [:maybe keyword?]]]]

; [:map2

;  [:map

;   [:min-max [:int {:min 0, :max 10}]]

;   [:tuples [:vector [:tuple int? string?]]]

;   [:optional {:optional true} [:maybe :boolean]]

Options can be used by binding a dynamic l/*options* Var:

(binding [l/*options* {:registry (merge

(m/default-schemas)

{:user/id :int})}]

(l/schema {:id (l/maybe :user/id)

:child {:id :user/id}}))

;[:map

; [:id [:maybe :user/id]]

; [:child [:map [:id :user/id]]]]

Performance

Malli tries to be really, really fast.

Validation performance

Usually as fast (or faster) as idiomatic Clojure.

(require '[criterium.core :as cc])

(def valid {:x true, :y 1, :z "zorro"})

;; idomatic clojure (54ns)

(let [valid? (fn [{:keys [x y z]}]

(and (boolean? x)

(if y (int? y) true)

(string? z)))]

(assert (valid? valid))

(cc/quick-bench (valid? valid)))

(require '[malli.core :as m])

;; malli (39ns)

(let [valid? (m/validator

[:map

[:x :boolean]

[:y {:optional true} :int]

[:z :string]])]

(assert (valid? valid))

(cc/quick-bench (valid? valid)))

Same with Clojure Spec and Plumatic Schema:

(require '[clojure.spec.alpha :as spec])

(require '[schema.core :as schema])

(spec/def ::x boolean?)

(spec/def ::y int?)

(spec/def ::z string?)

;; clojure.spec (450ns)

(let [spec (spec/keys :req-un [::x ::z] :opt-un [::y])]

(assert (spec/valid? spec valid))

(cc/quick-bench (spec/valid? spec valid)))

;; plumatic schema (660ns)

(let [valid? (schema/checker

{:x schema/Bool

(schema/optional-key :y) schema/Int

:z schema/Str})]

(assert (not (valid? valid)))

(cc/quick-bench (valid? valid)))

Transformation performance

Usually faster than idiomatic Clojure.

(def data {:x "true", :y "1", :z "kikka"})

(def expected {:x true, :y 1, :z "kikka"})

;; idiomatic clojure (290ns)

(let [transform (fn [{:keys [x y] :as m}]

(cond-> m

(string? x) (update :x #(Boolean/parseBoolean %))

(string? y) (update :y #(Long/parseLong %))))]

(assert (= expected (transform data)))

(cc/quick-bench (transform data)))

;; malli (72ns)

(let [schema [:map

[:x :boolean]

[:y {:optional true} int?]

[:z string?]]

transform (m/decoder schema (mt/string-transformer))]

(assert (= expected (transform data)))

(cc/quick-bench (transform data)))

Same with Clojure Spec and Plumatic Schema:

(require '[spec-tools.core :as st])

(require '[schema.coerce :as sc])

(spec/def ::x boolean?)

(spec/def ::y int?)

(spec/def ::z string?)

;; clojure.spec (19000ns)

(let [spec (spec/keys :req-un [::x ::z] :opt-un [::y])

transform #(st/coerce spec % st/string-transformer)]

(assert (= expected (transform data)))

(cc/quick-bench (transform data)))

;; plumatic schema (2200ns)

(let [schema {:x schema/Bool

(schema/optional-key :y) schema/Int

:z schema/Str}

transform (sc/coercer schema sc/string-coercion-matcher)]

(assert (= expected (transform data)))

(cc/quick-bench (transform data)))

The transformation engine is smart enough to just transform parts of the schema that need to be transformed. If there is nothing to transform, identity function is returned.

(def json->user

(m/decoder

[:map

[:id :int]

[:name :string]

[:address [:map

[:street :string]

[:rural :boolean]

[:country [:enum "finland" "poland"]]]]]

(mt/json-transformer)))

(= identity json->user)

; => true

;; 5ns

(cc/quick-bench

(json->user

{:id 1

:name "tiina"

:address {:street "kotikatu"

:rural true

:country "poland"}}))

Parsing performance

;; 37µs

(let [spec (s/* (s/cat :prop string?,

:val (s/alt :s string?

:b boolean?)))

parse (partial s/conform spec)]

(cc/quick-bench

(parse ["-server" "foo" "-verbose" "-verbose" "-user" "joe"])))

;; 2.4µs

(let [schema [:* [:catn

[:prop string?]

[:val [:altn

[:s string?]

[:b boolean?]]]]]

parse (m/parser schema)]

(cc/quick-bench

(parse ["-server" "foo" "-verbose" "-verbose" "-user" "joe"])))

Built-in schemas

malli.core/predicate-schemas

Contains both function values and unqualified symbol representations for all relevant core predicates. Having both representations enables reading forms from both code (function values) and EDN-files (symbols): any?, some?, number?, integer?, int?, pos-int?, neg-int?, nat-int?, pos?, neg?, float?, double?, boolean?, string?, ident?, simple-ident?, qualified-ident?, keyword?, simple-keyword?, qualified-keyword?, symbol?, simple-symbol?, qualified-symbol?, uuid?, uri?, decimal?, inst?, seqable?, indexed?, map?, vector?, list?, seq?, char?, set?, nil?, false?, true?, zero?, rational?, coll?, empty?, associative?, sequential?, ratio?, bytes?, ifn? and fn?.

malli.core/class-schemas

Class-based schemas, contains java.util.regex.Pattern & js/RegExp.

malli.core/comparator-schemas

Comparator functions as keywords: :>, :>=, :<, :<=, := and :not=.

malli.core/type-schemas

Type-like schemas: :any, :some, :nil, :string, :int, :double, :boolean, :keyword, :qualified-keyword, :symbol, :qualified-symbol, and :uuid.

malli.core/sequence-schemas

Sequence/regex-schemas: :+, :*, :?, :repeat, :cat, :alt, :catn, :altn.

malli.core/base-schemas

Contains :and, :or, :orn, :not, :map, :map-of, :vector, :sequential, :set, :enum, :maybe, :tuple, :multi, :re, :fn, :ref, :=>, :function and :schema.

malli.util/schemas

:merge, :union and :select-keys.

Links (and thanks)

- Schema https://github.com/plumatic/schema

- Clojure.spec https://clojure.org/guides/spec

- Spell-spec https://github.com/bhauman/spell-spec

- JSON Schema https://json-schema.org/understanding-json-schema/index.html

- Spec-provider: https://github.com/stathissideris/spec-provider

- F# Type Providers: https://docs.microsoft.com/en-us/dotnet/fsharp/tutorials/type-providers/

- Minimallist https://github.com/green-coder/minimallist

- malli-instrument https://github.com/setzer22/malli-instrument

- Core.typed https://github.com/clojure/core.typed

- TypeScript https://www.typescriptlang.org/

- Struct https://funcool.github.io/struct/latest/

- Seqexp https://github.com/cgrand/seqexp

- yup https://github.com/jquense/yup

- JOI https://github.com/hapijs/joi

Alpha

The public API of Malli has been quite stable already in pre-alpha and in alpha, we try not to break things. Still, the library is evolving and things like value destructuring could affect public APIs and most likely affect the library extenders, e.g. need to implement a new protocol method for custom schemas.

All changes (breaking or not) will be documented in the CHANGELOG and there will be a migration guide and path if needed.

The API layers and stability:

- public API: public vars, name doesn't start with -, e.g. malli.core/validate. The most stable part of the library, should not change (much) in alpha

- extender API: public vars, name starts with -, e.g. malli.core/-collection-schema. Not needed with basic use cases, might evolve during the alpha, follow CHANGELOG for details

- experimental: stuff in malli.experimental ns, code might change be moved under a separate support library, but you can always copy the old implementation to your project, so ok to use.

- private API: private vars and malli.impl namespaces, all bets are off.

Supported Java versions

Malli aims to support the LTS releases Java 8 and 11 and the latest Java release, Java 15.

Development

Malli is open for contributions. Before contributing with a PR, please open an issue for it.

Adding new schema types

To add a new schema type, e.g. :float, you should adding the following:

- schema definition to malli.core + tests

- default encoder/decoder mappings into malli.transform + tests

- JSON Schema mappings into malli.json-schema + tests

- Generators into malli.generator + tests

- OPTIONALLY adding inferrers into malli.provider + tests

- update README.md

Running tests

We use Kaocha and cljs-test-runner as a test runners. Before running the tests, you need to install NPM dependencies.

npm install

./bin/kaocha

./bin/node

Installing locally

clj -Mjar

clj -Minstall

Bundle size for cljs

With default registry (37KB+ Gzipped)

# no sci

npx shadow-cljs run shadow.cljs.build-report app /tmp/report.html

# with sci

npx shadow-cljs run shadow.cljs.build-report app-sci /tmp/report.html

With minimal registry (2.4KB+ Gzipped)

# no sci

npx shadow-cljs run shadow.cljs.build-report app2 /tmp/report.html

# with sci

npx shadow-cljs run shadow.cljs.build-report app2-sci /tmp/report.html

Formatting the code

clojure-lsp format

clojure-lsp clean-ns

Checking the generated code

npx shadow-cljs release app --pseudo-names

Testing on GraalVM

Without sci (11Mb)

./bin/native-image demo

./demo '[:set :keyword]' '["kikka" "kukka"]'

With sci (18Mb):

./bin/native-image demosci

./demosci '[:fn (fn [x] (and (int? x) (> x 10)))]]' '12'

Babashka

Since version 0.8.9 malli is compatible with babashka, a native, fast starting Clojure interpreter for scripting.

You can add malli to bb.edn:

{:deps {metosin/malli {:mvn/version "0.9.0"}}}

or directly in a babashka script:

(ns bb-malli

(:require [babashka.deps :as deps]))

(deps/add-deps '{:deps {metosin/malli {:mvn/version "0.9.0"}}})

(require '[malli.core :as malli])

(prn (malli/validate [:map [:a [:int]]] {:a 1}))

(prn (malli/explain [:map [:a [:int]]] {:a "foo"}))

3rd party libraries

- Aave, a code checking tool for Clojure.

- Gungnir, a high level, data driven database library for Clojure data mapping.

- Regal, Royally reified regular expressions

- Reitit, a fast data-driven router for Clojure/Script.

- wasm.cljc - Spec compliant WebAssembly compiler and decompiler

- malli-instrument - Instrumentation for malli mimicking the clojure.spec.alpha API

- Snoop - Function instrumentation using Malli schemas.

- malli-key-relations - Relational schemas about map keys for malli

- malli-cli - Command-line processing

- malapropism - malli-backed configuration library

License

Copyright © 2019-2022 Metosin Oy and contributors.

Available under the terms of the Eclipse Public License 2.0, see LICENSE.

Malli, Data-Driven Schemas for Clojure/Script

Thursday, 08 October, 2020 by Tommi Reiman

Introduction

After over a year in incubation, we are proud to announce the first alpha-version of Malli, a new data validation and specification library for Clojure/Script. It provides unified tools for building data-driven schema systems covering schema definition, validation, (human) errors, value and schema transformation, value and schema generation, registries and much more.

It's an open system, aiming to be both a fun to use and easy to extend. It has first class apis for both basic users and advanced/library developers. Using the lessons learned on making reitit, Malli is written from the group up to be fast.

There is a long list of prior art, but special thanks to Plumatic Schema and clojure.spec for showing the way - and to Clojurists Together for funding to ship the lib out.

The simplest thing that works:

(require '[malli.core :as m])

(m/validate int? 1)

; => true

Specification Language

One of the key lessons learned from building successful multi-tenant, data-oriented and data-driven systems with Clojure is that we should aim to define our data models as literal/serializable data we can easily program with. This gives us big leverage as we can roundtrip our schemas into database and into the web clients for local validation or editing. The specification language should be expressive and easy to understand, also by business and domain people.

There is a great post by Valentin Waeselynck about the benefits of having a common definition language and deriving more or less everything from it.

Example of a recursive Malli Schema, describing an Order:

(def Order

[:schema

{:registry {"Country" [:map

[:name [:enum :FI :PO]]

[:neighbors [:vector [:ref "Country"]]]]

"Burger" [:map

[:name string?]

[:description {:optional true} string?]

[:origin [:maybe "Country"]]

[:price pos-int?]]

"OrderLine" [:map

[:burger "Burger"]

[:amount int?]]

"Order" [:map

[:lines [:vector "OrderLine"]]

[:delivery [:map

[:delivered boolean?]

[:address [:map

[:street string?]

[:zip int?]

[:country "Country"]]]]]]}}

"Order"])

You don't have to know much about Malli's internals to be able to read that. And, we didn't need any dependencies, it's just data and functions. You can play with the above schema in malli.io (the schema is serialized as a query parameter).

Example of transforming Order into DOT, the Graph Description Language:

(require '[malli.dot :as md])

(md/transform Order)

; => "digraph {...}"

Visualized using graphviz:

Defining Schemas

Malli defaults to a Reagent-style hiccup-syntax for defining schemas:

boolean?                               ;; type

[:tuple double? double?]               ;; [type & children]

[:enum {:title "Color"} "red" "black"] ;; [type properties & children]

[MyType {:my "props"} :my :children]   ;; reagent-style types

Internally, Schemas are represented as protocol instances. Schema syntax can be compiled into Schema protocol instances using m/schema function, backed by a scoped Registry.

Using the default registry:

(m/schema [:or int? string?])

; => [:or int? string?]

Using an explicit registry:

(m/schema [:or int? string?] {:registry m/default-registry})

; => [:or int? string?]

Malli supports several types of registries: immutable, mutable, dynamic, lazy, local and composite.

Core protocols

For internal elegance, Malli is built using protocols, but unless you are creating library extensions, you don't have to know about those. For the curious, here are some of the most important ones:

(defprotocol IntoSchema

(-into-schema [this properties children options] "creates a new schema instance"))

(defprotocol Schema

(-type [this] "returns type of the schema")

(-type-properties [this] "returns schema type properties")

(-validator [this] "returns a predicate function that checks if the schema is valid")

(-explainer [this path] "returns a function of `x in acc -> maybe errors` to explain the errors for invalid values")

(-transformer [this transformer method options] "returns an interceptor map with :enter and :leave functions to transform the value for the given schema and method")

(-walk [this walker path options] "walks the schema and it's children")

(-properties [this] "returns original schema properties")

(-options [this] "returns original options")

(-children [this] "returns schema children")

(-form [this] "returns original form of the schema"))

(defprotocol MapSchema

(-entries [this] "returns sequence of `key -val-schema` MapEntries"))

(defprotocol LensSchema

(-keep [this] "returns truthy if schema contributes to value path")

(-get [this key default] "returns schema at key")

(-set [this key value] "returns a copy with key having new value"))

(defprotocol RefSchema

(-ref [this] "returns the reference name")

(-deref [this] "returns the referenced schema"))

Validation and Errors

The most common Schema application is runtime validation and for that, there is validate. Like most Malli core functions, it supports both Schema instances and raw Schema syntax - calling schema behind the scenes:

(m/validate (m/schema [:maybe string?]) "sheep")

; => true

(m/validate [:maybe string?] "sheep")

; => true

Below is an example of validating a closed map (maps are open by default) with the constraint that the two password fields must match. Malli integrates with sci to add support for serializable anonymous function schemas without needing macros or eval.

(def UserForm

[:and

[:map {:closed true}

[:name string?]

[:age pos-int?]

[:password string?]

[:password2 string?]]

[:fn {:error/message "passwords must match"

:error/path [:password2]}

'(fn [{:keys [password password2]}]

(= password password2))]])

(m/validate

UserForm

{:name "Liisa"

:age 64

:password "Liisa4"

:password2 "Liisa444"})

; => false

To get more programmatic details about errors, there is m/explain, inspired by clojure.spec:

(-> UserForm

(m/explain

{:name "Liisa"

:age "64"

:password "Liisa4"

:password2 "Liisa444"}))

;{:schema [:and

;          [:map {:closed true}

;           [:name string?]

;           [:age pos-int?]

;           [:password string?]

;           [:password2 string?]]

;          [:fn {:error/path [:password2]

;                :error/message "passwords must match"}

;           (fn [{:keys [password password2]}]

;             (= password password2))]]

; :value {:name "Liisa"

;         :age "64"

;         :password "Liisa4"

;         :password2 "Liisa444"},

; :errors ({:path [0 :age]

;           :in [:age]

;           :schema pos-int?

;           :value "64"}

;          {:path [1],

;           :in [],

;           :schema [:fn {:error/path [:password2]

;                         :message "passwords must match"}

;                    (fn [{:keys [password password2]}]

;                      (= password password2))]

;           :value {:name "Liisa"

;                   :age "64"

;                   :password "Liisa4"

;                   :password2 "Liisa444"}})}

Malli also provides humanized errors, supporting localization:

(require '[malli.error :as me])

(-> UserForm

(m/explain

{:name "Liisa"

:age "64"

:password "Liisa4"

:password2 "Liisa444"})

(me/humanize))

;{:age ["should be a positive int"]

; :password2 ["passwords must match"]}

... and ships with a spell checker for :map and :multi keys, thanks to Bruce Hauman's code originally found in spell-spec:

(-> UserForm

(m/explain

{:name "Liisa"

:age "64"

:passwordz "Liisa4"})

(me/with-spell-checking)

(me/humanize))

;{:age ["should be a positive int"]

; :passwordz ["should be spelled :password2 or :password"]}

Value Transformation

Runtime value transformation, aka coercion is also important. Schemas should be used to derive the required transformations between different formats like JSON and EDN. Malli has a performant built-in two-way transformation engine, inspired by Plumatic Schema and the lessons learned from developing spec-tools.

(require '[malli.transform :as mt])

(m/decode [:set keyword?] ["kikka" "kukka"] (mt/string-transformer))

; => #{:kukka :kikka}

By default, the decoding and encoding logic is derived from schema types, but schema properties can be used too. Example of a bidirectional transfromation:

(def LegacyString

[:string {:decode/string '(fn [x] (str/lower-case (subs x 4)))

:encode/string '(fn [x] (str/upper-case (str "HAL_" x)))}])

(m/decode LegacyString "HAL_KIKKA" (mt/string-transformer))

; => "kikka"

(m/encode LegacyString "kikka" (mt/string-transformer))

; => "HAL_KIKKA"

Transformation chains are composable. Below is an example where we decode data from JSON to EDN, stripping away all extra keys and applying default values:

(def Address

[:map

[:id string?]

[:tags [:set keyword?]]

[:address

[:map

[:street string?]

[:city string?]

[:zip {:default 33100} int?]

[:lonlat [:tuple double? double?]]]]])

(m/decode

Address

{:id "Lillan",

:EVIL "LYN"

:tags ["coffee" "artesan" "garden"],

:address {:street "Ahlmanintie 29"

:DARK "ORKO"

:city "Tampere"

:lonlat [61.4858322 23.7854658]}}

(mt/transformer

(mt/strip-extra-keys-transformer)

(mt/default-value-transformer)

(mt/json-transformer)))

;{:id "Lillan",

; :tags #{:coffee :artesan :garden},

; :address {:street "Ahlmanintie 29"

;           :city "Tampere"

;           :zip 33100

;           :lonlat [61.4858322 23.7854658]}}

Thanks to the internal ahead-of-time resolver, the value transformation are quite fast. It's currently 1-3 orders of magnitude faster than with clojure.spec + spec-tools and we aim to make it even faster.

Schema Transformation

One of the key aspect of Clojure is data-orientation. We build our values using immutable maps, sets, lists and vectors, without needing any predefined classes or schemas:

(def user

{:name "tommi"

:age 45

:address {:street "Hämeenkatu"

:country "Finland"}})

The Joy of Clojure:

(-> user

(select-keys [:name :address])

(assoc-in [:address :zip] 33100))

;{:name "tommi"

; :address {:street "Hämeenkatu"

;           :country "Finland"

;           :zip 33100}}

Malli aims to bring the same power to working with Schemas. Schema instances are immutable values and thanks to generic -walk, -get and -set protocol methods and helpers in malli.util, we can transform Schemas just like normal data.

(def User

[:map

[:name string?]

[:age int?]

[:address [:map

[:street string?]

[:country string?]]]])

(m/validate User user)

; => true

Transforming it like in the example above:

(require '[malli.util :as mu])

(-> User

(mu/select-keys [:name :address])

(mu/assoc-in [:address :zip] int?))

;[:map

; [:name string?]

; [:address [:map

;            [:street string?]

;            [:country string?]

;            [:zip int?]]]]

This still holds:

(m/validate

(-> User

(mu/select-keys [:name :address])

(mu/assoc-in [:address :zip] int?))

(-> user

(select-keys [:name :address])

(assoc-in [:address :zip] 33100)))

; => true

There are also utilities for schema merge, union, find-first, pre- and postwalk. Below is an example how to transform a Schema from hiccup-syntax to cljfx-style map-syntax:

(m/walk

User

(fn [schema _ children _]

(let [properties (m/properties schema)]

(cond-> {:type (m/type schema)}

(seq properties) (assoc :properties properties)

(seq children) (assoc :children children)))))

;{:type :map,

; :children [[:name nil {:type string?}]

;            [:age nil {:type int?}]

;            [:address nil {:type :map

;                           :children [[:street nil {:type string?}]

;                                      [:country nil {:type string?}]]}]]}

Malli to DOT and Malli to JSON Schema transformers are also available. To highlight power of the transforming utilities, the DOT-transformer is just 69 lines of code.

Value Generation

Value generation is a process of creating valid example values out of a Schema. Both Plumatic and Spec have had this for years and so does Malli. Thanks to test.check for all the heavy lifting:

(require '[malli.generator :as mg])

(mg/sample string? {:size 5, :seed 42})

; ("" "o" "1" "p" "gQl")

Schema properties can be used to control the value generation. These properties include::gen/fmap, :gen/elements, :gen/min, :gen/max and :gen/gen:

(mg/generate [:string {:gen/fmap '(partial str "kikka_")}])

; "kikka_WT3K0yax2"

Random Orders coming your way:

(mg/generate Order {:size 5, :seed 42})

;{:lines [{:burger {:name "5n156", :origin nil, :price 1, :description ""}, :amount -3}

;         {:burger {:name "rxH0", :origin {:name :FI, :neighbors []}, :price 7}, :amount 0}

;         {:burger {:name "JU", :origin nil, :price 12, :description "Wa3"}, :amount -4}],

; :delivery {:delivered true, :address {:street "5", :zip -1, :country {:name :PO, :neighbors []}}}}

Schema Generation

Inspired by F# Type providers, there are two built-in ways to generate Schemas from external data: inferring schemas and lazy/provider registries.

Inferring Schemas

Inferring schemas is the inverse of value generation. Given a set of values, a minimal schema that all values are valid against is returned. Inspired by spec-provider for Clojure Spec.

(require '[malli.provider :as mp])

(mp/provide [user])

;[:map

; [:name string?]

; [:age int?]

; [:address [:map

;            [:street string?]

;            [:country string?]]]]

The current implementation is naive (69 lines of code) but already quite useful.

Lazy/Provider Registries

Another way to generate schemas is to use a special lazy-registry that allows a callback-function to be used to return a schema given a registry key. There is a nice example that pulls out AWS CloudFormation Schemas on-demand and converts them into Malli Schemas.

Beyond Runtime Validation

The primary goal of Malli has been to create a tool for communicating the structure of data, to be enforced at runtime. In JavaScript, it would be more like yup than TypeScript. But we don't have to stop there: Malli will have function schemas, it should provide pretty errors for developers, it most likely will integrate with Clojure Spec, definitely with clj-kondo, maybe even with Typed Clojure.

We could build providers for GraphQL, PostgreSQL and TypeScript to simplify interop in the real world projects we build. Maybe infer schemas with HugSQL the in spirit of pgtyped?

A demo of function schemas with clj-kondo from my talk at ClojureD:

Malli vs Clojure.Spec

But wait, there is already clojure.spec, which is the supposed future of Clojure, why create a library doing about the same? Short answer: the libraries have different goals: Spec is not going to be a runtime transformation engine, period.

Spec builds around a global registry and strong ideologies around it, Malli builds on schemas and registries as values and aims to be pragmatic. The world is not perfect, it's riddled with nulls and optional keys and we should embrace them.

You could use a single global registry in Malli, but you don't have to. You can build a strict Spec2 Schema/Select-style system with decomplected maps keys and values with Malli, but you don't have to. Malli is just a library, putting the user in control of things.

Spec2 example schema with Malli:

(def registry

{::street string?

::city string?

::state string?

::zip int?

::addr [:map ::street ::city ::state ::zip]

::id int?

::first string?

::last string?

::user [:map ::id ::first ::last ::addr]})

(def User

[:schema {:registry registry}

::user])

(mg/generate User {:size 21, :seed 21})

;{:user/id -1669,

; :user/first "5qU1TP3",

; :user/last "mbQN36L",

; :user/addr {:user/street "W901"

;             :user/city "cA7AZkvLu0Dl6lXg"

;             :user/state ""

;             :user/zip 61730}}

I believe Spec and Malli can co-exist and can be even friends together. Spec is an awesome tool for describing the core language and library apis, and Malli the world we are living in.

It's Alpha-time!

Since today, the following artifact is found in Clojars:

;; Leiningen/Boot

[metosin/malli "0.1.0"]

;; Clojure CLI/deps.edn

metosin/malli {:mvn/version "0.1.0"}

Big thanks to all contributors, everyone at Metosin and to Clojurists Together for the final nudge to get the stable version out. We are all excited and it has been real joy to see the current design emerged from original drafts.

Despite having been in pre-alpha, Malli has over 96k downloads, 459 stars on github and many people are already using it in production. All design decisions are openly discussed both in #malli Slack an in Github Issues. After this release, all changes will be properly tracked in the CHANGELOG.

What's Next?

There are still many weeks of the Clojurist Together funding, which enables me to focus on Malli. The roadmap is mostly laid out as Github Issues, but here is some ongoing work:

- Better support for dates - needed in day-to-day projects

- Declarative schema manipulation - just because it's possible

- Sequence schemas - enabling schema-based destructuring

- Schema defn syntax - for domain code, enables linting

- Full clj-kondo integration - a linter that sparks joy

- Validating schema properties and children using Malli

- Two-way bindings for JSON Schema

We are using Malli actively in our new projects, and will continue to explore things we can do with it. Busy times, so help is welcome.

Getting Started

Simplest way to get started with Malli is to join #malli, read the README, try things in malli.io or use it via an external library, including reitit or gungnir. Google's version of how to pronounce Malli.

The coordinates:

- https://github.com/metosin/malli

- https://clojars.org/metosin/malli

You can discuss this post in Clojureverse.

Cheers,

Tommi

#clojure

Structure and Interpretation of Malli Regex Schemas

Wednesday, 31 March, 2021 by Pauli Jaakkola

Seqex schemas shipped with Malli 0.3.0. This is their story. A story with many twists and turns and technical details Malli users don't need to know.

Introduction

In December 2020 Tommi called on me to implement "regex schemas" for Malli. Now what do we even mean by "regex schemas"? Hasn't Malli had support for

(m/validate #"foo|bar" "foo") ;=> true

for quite a while now?

What I am talking about here are regular expressions of sequences and not just of strings. In an attempt to reduce confusion I will call them "seqexes" from now on, Seqexp having pioneered the concept before Clojure Spec was even a thing.

Most developers are probably familiar with regular expressions of strings and their basic forms. Here they are in (JVM) Clojure and Malli forms:

- The empty string #"" or sequence [:cat]

- Literal character #"a" or element schema :int

- Concatenation #"ab" or [:cat :string :int]

- Alternation #"a|b" or [:alt :string :int]

- Kleene Star #"a*" or [:* :int]

The formalism also contains the empty set but regex and seqex engines do not because a regex/seqex/schema that never matches anything is not very useful in practice. Because Malli uses an S-expression or Hiccup-like syntax the empty sequence is just the concatenation of zero subexpressions [:cat].

I am quite the theoretical computer science person, which is why I was tasked with this feature. More pragmatic programmers might regard these additional operators as equally "basic":

- Optional #"a?" or [:? :int]

- Kleene Plus #"a+" or [:+ :int]

- Repeat #"a{1, 5}" or [:repeat {:min 1, :max 5} :int]

even though they can often be implemented as syntactic sugar for the five fundamental constructs. There are many more regex operators in the wild but like Seqexp and Spec, Malli has a more minimal set of seqex operators. Many regex operators are text-specific while others are considered harmful. Seqexp does support lookaheads and non-greedy quantifiers, but even lookaheads may have been a mistake.

Expressivity

Seqexes are useful for function schemas and they can handle quite complicated DSL:s too:

(m/parse

[:schema {:registry {"hiccup" [:or

[:catn

[:name keyword?]

[:props [:? [:map-of keyword? any?]]]

[:children [:* [:schema [:ref "hiccup"]]]]]

[:or nil? boolean? number? string?]]}}

"hiccup"]

[:div {:class [:foo :bar]}

"Hello, world of data"])

;=> {:name :div, :props {:class [:foo :bar]}, :children ["Hello, world of data"]}

Macro arguments can of course be targeted as well as data DSL:s. Go for it!

Although Malli seqex schemas are actually limited to regular languages by design, so you can't do this:

(require '[clojure.spec.alpha :as s])

(s/def ::ws (s/* #(Character/isWhitespace %)))

(s/def ::sexpr

(s/cat

:lws ::ws

:sexpr (s/alt

:list (s/cat :lparen #(= \( %) :sexprs (s/* ::sexpr) :rparen #(= \) %))

:symbol (s/cat :c #(Character/isJavaIdentifierStart %)

:cs (s/* #(Character/isJavaIdentifierPart %)))

:int (s/+ #(Character/isDigit %)))

:rws ::ws))

(s/valid? ::sexpr (seq "(foo (bar 1 2))"))

; => true

(s/valid? ::sexpr (seq "(foo (bar 1 2)"))

; => false

(Regular expressions are insufficient to handle recursively nested constructs like our lovely Lispy pairs of parentheses or HTML. That is also why we have Instaparse. And please don't do it with Spec either even if it happens to work as relying on emergent behaviour can be risky!)

Requirements

We have to extend the Malli schema functionalities to seqexes, which means:

- Validation with validate; this is just language recognition. Language recognition is rarely sufficient but here we have it.

- Getting validation errors with explain; similar to validate, but have to collect all errors from invalid seq elements as well as a seq that is too short or long.

- Transformers (encode and decode); these are essentially parsing where the result just happens to be a seq of the same length with transformed elements (higher level encoder or decoder functions could of course slice and dice that seq later).

- Destructuring and constructing sequences with parse and unparse; parse is well, parsing with seqexes. Full parsing with regular expressions is also a bit unusual; usually we just check whether the input matches and possibly extract some substrings of it. unparse is a kind of pretty printing but producing sequences instead of just strings and quite simple compared to the other functionalities. parse and unparse were actually added after the seqex schemas and are no-ops unless you use seqexes or :orn (which was only added with parse) or :multi (starting with Malli 0.4.0).

Possible Implementation Strategies

There are three well-known approaches to implement regular expression matching:

- Convert the regular expression to a deterministic finite automaton (DFA) (usually via a nondeterministic finite automaton (NFA) and the powerset construction) and run the DFA on the input. This is what foundational tools like grep and lex do.

- Convert the regular expression to a NFA and run the NFA on the input. The Seqexp library uses the "PikeVM" variant of this, where the NFA is stored as a bytecode program for a highly multithreaded virtual machine (the threads are green and even more lightweight than general purpose green threads like running go blocks).

- One can also add an incrementally constructed DFA as a caching layer, sort of like a "DFA JIT". ANTLR 4's ALL(*) lookahead also works like this, except it has to simulate the augmented transition network (ATN) for a context-free or even context-sensitive grammar instead of just a NFA for a regular expression.

- Use a backtracking parser similar to the typical implementation of parser combinators. This is how Perl, PCRE, Java etc. regular expressions work and must work because they have gone beyond regular languages with crazy stuff like backreferences. Caching can be added similarly to the memoizing Packrat parsing algorithms.

And then there is the forgotten jewel of Brzozowski's regular expression derivatives:

- Derivatives can be used instead of NFA-conversion and the powerset construction to turn the regular expression into a DFA. This is used in the SML ml-ulex and Racket parser-tools/lex lexer generators.

- It is also possible to do parsing by computing regex derivatives on the fly, which is how Clojure Spec regex operators are implemented.

Now suppose the length of the input is n and the length of the regex is r:

- DFA matching takes  time and  space (just the index of the current state). However constructing the DFA takes  time and the size of the DFA is also  (although the usual case is much more reasonable of course).

- NFA matching takes  time and  space (e.g. the PikeVM needs  threads at most). Constructing the NFA only takes  time and so does storing it. The DFA cache gives speedup especially on longer inputs but does not change the bounds.

- Backtracking takes  time and  space. Memoization can reduce the time to  while only increasing the space usage to .

- On the fly derivative matching takes  time and  space. The optimizations from parsing with derivatives are surely applicable and might be able to reduce the space requirement to  but AFAIK Spec does not try to do that.

Obviously the backtracking strategy is a depth-first search. The other techniques are essentially breadth-first search. Typically depth-first search uses exponential time but linear space while breadth-first search uses linear time but exponential space. However because a string or even a general seq is quite a simple input there are tricks to make breadth-first search take very little space (converting to a DFA, deduplicating threads in the PikeVM) and the usual depth-first perf lifeline of memoization only requires linear space.

Overall the NFA approach seems like the winner; nice linear bounds without even the ahead of time DFA construction exponentiality issues. Actually since we are matching general seqs instead of strings the DFA approach isn't really applicable as it requires simulating running the NFA on every possible input string in advance. That is just fine on ASCII, tricky on UTF (but there are solutions) and a non-starter on every possible Clojure datum host object.

PikeVM Thread Deduplication Did not Last

So I started implementing the regex operators based on the Seqexp code. We noticed that Seqexp is actually quite slow because its PikeVM engine is implemented in idiomatic Clojure with lots of intermediate sequences when ideally it would allocate just the thread table and no intermediate sequences at all. By writing Java in Clojure (or is it Rust in Clojure because I haven't written much Java at all...) I was able to make it 20x faster.

But then I realized that in Malli we need more than regular language recognition and subsequence extraction; at the time we already had transformers (encode/decode) and recently we added parse and unparse as well. Those produce sequences where potentially every element has been transformed somehow. The prefix of that sequence (or something from which it can be reconstructed) needs to be a part of the thread state, which makes the thread deduplication ineffective and we are back to  space usage. I saw no way around that, so had to pivot!

My perf Seqexp branch is still out there if anyone is interested.

Implementing Seqex Parsing

So I went for (memoized) backtracking instead. Since the implementation is quite interesting both as a final achievement and as a journey I thought I'd explain it here. (Also, I do worry about the bus factor with this!)

Warning: Here Be Code (but, no Dragon Books!)

Introductions to parsing like to emphasize that the parser input is an abstract sequence of any "symbols of an alphabet" while in practise it is almost always a string or stream of characters or tokens. But our input really is any seq(uence) of any values, and it really does not affect this implementation much (but, cannot use DFA).

As usual with parsing combinators, our (sub)parsers are functions that take the input and return a tuple of the parsed prefix and the remaining input. Clojure vectors make bulky tuples (thus the attempts at clj-tuple and cambrian-collections) but we will later get rid of them rather incidentally. If the parser fails we just return :malli.core/invalid; explain has a separate implementation.

Parsing a single item consists of just using parse-item on the first element of the seq. parse-item should be a function that transforms the item or returns :malli.core/invalid:

(defn item-parser [parse-item]

(fn [coll]

(if (seq coll)

(let [res (parse-item (first coll))]

(if-not (miu/-invalid? res)

[res (rest coll)]

res))

:malli.core/invalid)))

In practice the parse-item functions come from malli.core/-parser.

end-parser checks that the collection is empty. The old-school way to do this was to check for an 'EOF' sentinel or null byte, but with Clojure sequences we have empty?:

(defn end-parser [coll]

(if (empty? coll)

[nil coll]

:malli.core/invalid))

As with many parsing libraries and parser generators, end-parser is needed to ensure that the entire input is consumed by the overall parser. Parsing just a prefix of the input makes the parsing functions composable, but the top level parser must not be satisfied with just a correct prefix -- that would lead to silent failures, the worst kind of bugs. Obviously all of the input must be validated when we have web-facing API:s; offering s/every in addition to s/coll-of is just another sign that Spec is not about web API:s.

The parser for :cat may look formidable, but really it is mostly just boring plumbing. The parsers are just run in sequence, the results collected into a vector with the remaining input threade through and any error returned immediately, also canceling the remaining parsers:

(defn cat-parser [& parsers]

(reduce (fn [parser parser*]

(fn [coll]

(let [res (parser coll)]

(if-not (miu/-invalid? res)

(let [[vs coll] res]

(let [res* (parser* coll)]

(if-not (miu/-invalid? res*)

(let [[v coll] res*]

[(conj vs v) coll])

res*)))

res))))

(pure-parser []) parsers))

(And part of the bulk in cat-parser is just due to it being variadic (because we want :cat to be). In Haskell >>= and <*> are binary infix operators and the reduce (or, uh, foldr? foldl'?) part would not appear.)

cat-parser also uses a little utility pure-parser which just returns the fixed v and leaves the input alone:

(defn pure-parser [v] (fn [coll] [v coll]))

(In fancy terms, pure-parser 'embeds a pure value into the domain of parsing functions'; it is the 'monadic unit operation'.)

Complementarily, the parser for :alt tries the parsers for the alternatives in sequence and stops when it finds one that succeeds:

(defn alt-parser [& parsers]

(reduce (fn [parser parser*]

(fn [coll]

(let [res (parser coll)]

(if-not (miu/-invalid? res)

res

(parser* coll)))))

parsers))

Note that the reduce in alt-parser and cat-parser is done at parser generation time. Combined with the JIT inlining the effect is similar to loop unrolling. This "compilation to closures" approach was pervasive in Malli already before the seqex schemas and is the reason to prefer e.g. m/validator over m/validate. The approach is familiar from Plumatic Schema, but is also described in SICP (section 4.1.7: Separating Syntactic Analysis from Execution), which mentions Jonathan Rees already using it in 1982! To get similarly specialized machine code for the PikeVM approach we would need an unlikely meta-tracing JIT.

For contrast, the parser for :* must loop at parsing time, because we do not know how many times its child schema will match:

(defn *-parser [parser]

(letfn [(parser* [vs coll]

(let [res (parser coll)]

(if-not (miu/-invalid? res)

(let [[v coll] res]

(recur (conj vs v) coll))

[vs coll])))]

(fn [coll] (parser* [] coll))))

Aside from the looping, it is similar to cat-parser in collecting the results to a vector and threading the remaining input through. But note how when the child parser fails, *-parser succeeds with the input and the previously collected vs; :* can just stop matching at any time or match nothing at all, so it never fails. (The overall parser can still fail thanks to the use of end-parser: consider e.g. (= false (m/validate [:* int?] ["foo"])).)

Since we have parser combinators, the parsers for :? and :+ can be derived in practice like in theory, by composing the combinators (Always Be Composing!):

(defn ?-parser [parser] (alt-parser parser (pure-parser nil)))

(defn +-parser [parser]

(map-parser (fn [[v vs]] (cons v vs))

(cat-parser parser (*-parser parser))))

+-parser also needs map-parser, which just parses with parser and, if successful, applies f to the parse result:

(defn map-parser [f parser]

(fn [coll]

(let [res (parser coll)]

(if-not (miu/-invalid? res)

(let [[v coll] res]

[(f v) coll])

res))))

Having map-parser technically makes our collection of parsing functions a functor (even if we are not extending any fancy Functor typeclass protocol). But don't worry, we won't make it a monad; that is only required for context-dependent parsing and we only want to handle regular grammars.

(Not that making it a monad would be any scarier, just change [] to ():

(defn flat-map-parser [f parser]

(fn [coll]

(let [res (parser coll)]

(if-not (miu/-invalid? res)

(let [[v coll] res]

((f v) coll))

res))))

And a monad would also require pure-parser, but we had that already.)

In theory the :repeat parser could also be implemented with composition, but that would require duplicating the child parser max times and building some other closures besides. We prefer to implement the :repeat parser directly:

(defn repeat-parser [min max parser]

(letfn [(compulsories [n vs coll]

(if (< n min)

(let [res (parser coll)]

(if-not (miu/-invalid? res)

(let [[v coll] res]

(recur (inc n) (conj vs v) coll))

res))

(optionals n vs coll)))

(optionals [n vs coll]

(if (< n max)

(let [res (parser coll)]

(if-not (miu/-invalid? res)

(let [[v coll] res]

(recur (inc n) (conj vs v) coll))

[vs coll]))

[vs coll])]

(fn [coll] (compulsories 0 [] coll))))

Quite similar to *-parser, but we also have a counter to deal with. Brings me back to Programming 101... after switching to FP I rarely get such juicy chances for off-by-one errors.

An NFA implementation like Seqexp has not choice but to add at least max nodes to the NFA. Well, the PikeVM used by Seqexp could be extended with a stack of registers for loops, but that would marr its elegance...

A Gotcha

Consider

(m/parse [:cat [:* pos?] [:= 4]] [2 4])

which boils down to

(let [parse (cat-parser (*-parser (item-parser (m/parser pos?)))

(item-parser (m/parser [:= 4])))]

(parse [2 4]))

According to the semantics we have for seqex parsing, it should return [[2] 4]. But the result is :malli.core/invalid.

The problem is that not only is :* greedy (as it should be); it also commits to every consumption of a sequence element. Since (pos? 4) is true, [:* pos?] parses out [2 4], leaving [:= 4] with an empty seq when it wants [4]. For the overall parse to succeed, it needs to backtrack to when [:* pos?] only had [2] and exit the loop at that point, leaving [:= 4] with the input it needs. I think I have run into this already on my previous parser combinator adventures (of which I have had more than I care to admit).

CPS-Conversion and Trampolining

But how can the backtracking into :* be implemented? The (implicit) loop in *-parser cannot actually be restarted after it has been exited. We could give up on the 'compilation to closures' approach and make seqex parsing 'interpret' the schema tree instead. Then backtracking could be handled by resetting the current position in the schema, similar to Rob Pike's elegant C regex demo implementation which just rewinds a char *. But that would lose all the performance gained by precomputation when building the closures.

The solution is simple but potentially mindblowing: capture the continuation inside the loop. Since Clojure does not have any sort of call-with-current-continuation we have to convert the parsers to continuation passing style (CPS) ourselves to get the continuation. And because Clojure does not do general tail-call optimization either we also have to implement that by trampolining the CPS-converted code to replace the recur in *-parser. Buckle up, it is going to be a wild ride!

As in standard CPS, the converted parsers take a continuation parameter k that they can call to return or pass to other functions in tail calls. Usually continuations would have just one parameter, the return value. But we pass the parsed value and the remainder of the input as separate arguments, conveniently avoiding the vector allocations as promised. This threading through of the input coll resembles the handling of the state value in State monads, if you are familiar with those (and their implementation).

More unusually even for CPS the converted parsers also get a stack of fallback continuations. Calls to parsers may be pushed to that stack by 'parking' them. This is similar to how Prolog implementations mark backtracking points on their stacks. The Prolog connection is no accident; parsing can be conceptualized as a search on the grammar that is also guided by the input string and Prolog also uses a depth-first search.

*-parser parks epsilon on each iteration to make those available as restart locations (the line marked "remember fallback"):

(defn *-parser [parser]

(letfn [(epsilon (fn [_ vs coll k] (k vs coll)))

(parser* [stack vs coll k]

(park*! stack epsilon vs coll k) ; remember fallback

(parser stack coll

(fn [v coll] (park*! stack parser* (conj vs v) coll k))))] ; TCO

(fn [stack coll k] (parser* stack [] coll k))))

This arrangement requires an external control loop to restart the call on top of the stack when parsing fails. We can also use that loop as a trampoline for TCO, as happens on the line marked "TCO".

The converted repeat does the same tricks, it just has more bookkeeping (as before):

(defn repeat-parser [min max parser]

(letfn [(epsilon (fn [_ vs coll k] (k vs coll)))

(compulsories [stack n vs coll k]

(if (< n min)

(parser stack coll

(fn [v coll] (park**! stack compulsories (inc n) (conj vs v) coll k))) ; TCO

(optionals stack n vs coll k)))

(optionals [stack n vs coll k]

(if (< n max)

(do

(park*! stack epsilon vs coll k) ; remember fallback

(parser stack coll

(fn [v coll] (park**! stack optionals (inc n) (conj vs v) coll k)))) ; TCO

(k vs coll)))]

(fn [stack coll k] (compulsories stack 0 [] coll k))))

We use parking for all backtracking, including alt-parser:

(defn alt-parser [& parsers]

(reduce (fn [parser parser*]

(fn [stack coll k]

(park! stack parser* coll k) ; remember fallback

(parser stack coll k)))

parsers))

That is not strictly necessary at this point, but will be useful when we add caching later on.

The other basic parsers are just converted to our flavor of CPS, taking k and stack and pushing coll around:

(defn item-parser [parse-item]

(fn [_ coll k]

(when (seq coll)

(let [res (parse-item (first coll))]

(when-not (miu/-invalid? res)

(k res (rest coll)))))))

(defn end-parser [_ coll k]

(when (empty? coll)

(k nil coll)))

(defn cat-parser [& parsers]

(let [p (reduce (fn [rest-parser parser]

(fn [stack vs coll k]

(parser stack coll

(fn [v coll] (rest-parser stack (conj vs v) coll k)))))

(reverse parsers))]

(fn [stack coll k] (p stack [] coll k))))

Since ?-parser and +-parser are just defined in terms of the other combinators, they don't need to change. But the pure-parser and map-parser that they use do need to be CPS-converted:

(defn pure-parser [v] (fn [_ coll k] (k v coll)))

(defn map-parser [f parser]

(fn [stack coll k]

(parser stack coll

(fn [v coll] (k (f v) coll)))))

parser makes sure that the entire input is consumed by appending end-parser. It also contains the backtracking and trampolining loop, which just performs calls from the stack until one of them finishes successfully by calling the final success continuation which sets success and result or until the backtracking stack is empty:

(defn parser [p]

(let [p (cat-parser p (end-parser))]

(fn [coll]

(if (sequential? coll)

(let [stack (make-stack)

success (volatile! false)

result (volatile! nil)]

(p stack coll (fn [v _ _] (vreset! success true) (vreset! result v)))

(if @success

(first @result)

(loop []

(if-some [thunk (pop-thunk! stack)]

(do

(thunk)

(if @success

(first @result)

(recur)))

:malli.core/invalid))))

:malli.core/invalid))))

Before adding caching, the stack is just a java.util.ArrayDeque or a JS array:

(defn make-stack [] #?(:clj (ArrayDeque.), :cljs #js []))

(defn- empty-stack? [^ArrayDeque stack]

#?(:clj (.isEmpty stack), :cljs (zero? (alength stack))))

(defn park! [^ArrayDeque stack parser coll k] (.push stack #(parser stack coll k)))

(defn pop-thunk! [^ArrayDeque stack] (when-not (empty-stack? stack) (.pop stack)))

We can use volatile! and mutable stacks freely because the parsers we create are single threaded. parse-item can (at least theoretically) be multi-threaded, but that cannot affect the parsing thread since we do not pass continuations into parse-item or anything like that. Besides, the parsers returned by parser are externally pure, just like into uses transients internally whenever possible.

Memoization

Opposite to the NFA, backtracking is very space-efficient but has exponential worst-case time behaviour. As is often the case, we can use caching to gain linear time behaviour at the expense of more space usage. Happily, our cache will only use linear space also, as previously promised.

We can avoid visiting backtracking points more than once by simply discarding the parsing function and its arguments instead of pushing them to the stack when we have already seen the particular combination of input and seqex positions. The position in the input is already captured by the remainder of the input coll but I added the input index pos everywhere because integers make much better hash map keys than arbitrary sequences. The position in the seqex could be implemented as a reference to the schema subtree but instead I found it convenient to just guarantee that every schema subtree gets its own parsing function which can be used as the key (by pointer equality and identity hash).

Perhaps surprisingly the counter in :repeat adds quite a bit of complication to this memoization. Separate iterations of :repeat with a different counter should be recognized as separate positions in the seqex, so the counter needs to be added to the hash key along with the seq index and the parsing function. Furthermore there can be nested :repeats so a stack of all in-progress :repeat counters is actually required. Oh well, at least the counters themselves are just integers.

Having dealt with all these algorithmic complications I was faced with a more industrial one; I wanted a mutable hash set for the memoization cache (the parsers and arguments going to the separate stack), but Javascript only got one in ES6. I did try to use transient persistent sets in volatile!s for #?(:cljs ... but the performance was completely unacceptable. Google Closure can polyfill ES6 sets with the right options, but those options can only be set in the project that actually does the compilation to JS, e.g. your frontend app. And Malli is so general (and divisive) that it is likely to end up in some applications as a transitive dependency only.

So I made my own mutable hash set in some 40 lines of Clojure. It uses quadratic probing with power of two sizes and triangular numbers, another neat trick that could fill another blog post. Amusingly it was actually slightly faster than java.util.HashSet on Java 8 and roughly equivalent with it on newer JDK:s so I switched to it on JVM as well. But more importantly Malli seqexes now had good performance on the ClojureScript side too.

You can see the actual Malli code for the final algorithm with all the complications from memoization and associated issues tackled and also all the variations required to implement validator, explainer, decoder and encoder in addition to parse. The unparse implementation is as straightforward for seqexes as for everything else and has nothing to do with all these parsing techniques.

Performance

As the README says and as you might have gathered by now:

Although a lot of effort has gone into making the seqex implementation fast

(require '[clojure.spec.alpha :as s])

(require '[criterium.core :as cc])

(let [valid? (partial s/valid? (s/* int?))]

(cc/quick-bench (valid? (range 10)))) ; Execution time mean : 27µs

(let [valid? (m/validator [:* int?])]

(cc/quick-bench (valid? (range 10)))) ; Execution time mean : 2.7µs

it is always better to use less general tools whenever possible:

(let [valid? (partial s/valid? (s/coll-of int?))]

(cc/quick-bench (valid? (range 10)))) ; Execution time mean : 1.8µs

(let [valid? (m/validator [:sequential int?])]

(cc/quick-bench (valid? (range 10)))) ; Execution time mean : 0.12µs

Going Beyond

- The implementation here is quite similar to GLL parsing except we don't have to care about continuations in the caching since regular languages don't include (nontail) recursion. https://epsil.github.io/gll/ is a nice explanation of GLL parser combinators and has links to papers etc. It also inspired Instaparse, which Engelberg had a presentation about at Clojure/West 2014.

- If you caught the parsing bug Parsing Techniques is the hefty tome to scratch that itch / scare you away.

Assessment

The Malli seqex implementation was an exceptional project. It is not often that parsing theory is so directly applicable to a practical problem and there is no pre-existing parsing tool that can just be used (although Seqexp did come close!). In retrospect all the issues from pivot-level to merely annoying also made the experience greater; it is exhilarating when you are able to pull aces like CPS and custom hash tables (with bit-twiddling and all) from your sleeve instead of having to admit defeat.

I feel that the most masterful code comes from "hunting high and low", reaching for mindbending theory as well as the bits and JITs at hand. Of course that is also the recipe to find amazing ways to shoot yourself in the foot with C++. As with many other things, the hard-earned skill is to know the difference (and no, it is not just C++!).

#clojure#clojurescript#malli

High-Performance Schemas in Clojure/Script with Malli 1/2

Tuesday, 07 December, 2021 by Tommi Reiman

Introduction

It's over a year since the initial launch of Malli, the data-driven data specification library for Clojure/Script. Since then, it has become a de-facto tool in our projects and we are happy to see it getting adopted in the Clojure community. While writing this, Malli has 110 forks, over 60 contributors, several high-quality community-built add-ons, and over 750k downloads on Clojars.

This post walks through the performance engineering done in the last few revisions.

Performance

Inspired by the performance of Reitit, we wanted to make Malli fast too. We have benchmarked Malli against both idiomatic hand-written Clojure code and some well-known libraries like clojure.spec and Plumatic Schema.

Initially, only validation and value transformation were optimized for performance, but over time, more performance targets have been added, currently including the following:

- Validation

- Explaining

- Value Transformation

- Parsing and Unparsing

- Inferring Schemas

- Schema Creation

- Schema Worker Creation

- Transforming Schemas

Besides just counting the CPU cycles, the following are also important:

- Library load time (on CLJ)

- Bundle-size (on CLJS)

- Memory Usage

Let's start our journey with a sample domain model.

The Mushroom Forest

This was a great year to find delicious mushrooms in the (Finnish) forests. Here's some:

(def valid1

{:name "Kanttarelli"

:eatability 3

:data {:taste [:peppery :fruity :apricot]

:latin "Cantharellus cibarius"}})

(def valid2

{:name "Suppilovahvero"

:eatability 3

:data {:taste [:sweet :sour]}})

(def valid3

{:name "Punakärpässieni"

:eatability -2

:data nil})

Punakärpässieni is actually poisonous, so don't eat it.

Besides sample data, we need a domain model. We can use the Schema Inferring to pull the schema out of the samples:

(require '[malli.provider :as mp])

(def schema (mp/provide [valid1 valid2 valid3]))

schema

;[:map

; [:name string?]

; [:eatability int?]

; [:data [:maybe [:map

;                 [:taste [:vector keyword?]]

;                 [:latin {:optional true} string?]]]]]

The returned schema looks legit, :data can be nil and :latin key is optional. Let's also create an invalid mushroom to ensure our test domain works correctly:

(def invalid1

{:name "Känsätuhkelo"

:eatability 1

:data {:taste ["mild"]}})

Checking the values:

(require '[malli.core :as m])

(m/validate schema valid1) ; => true

(m/validate schema valid2) ; => true

(m/validate schema valid3) ; => true

(m/validate schema invalid) ; => false

All set, let's dive into the performance.

Validation

We are using Criterium to run the tests. There is also jmh-clj, but Criterium is much easier to use and gives good-enough results. The test harness:

(require '[criterium.core :as cc])

(defn bench-validate! [valid?]

(assert (every? valid? [valid1 valid2 valid3]))

(assert (not (valid? invalid)))

(cc/quick-bench (valid? valid1)))

Starting with idiomatic Clojure:

;; 440ns

(bench-validate!

(fn [{:keys [name eatability data] :as mushroom}]

(and (map? mushroom)

(int? eatability)

(string? name)

(or (nil? data)

(let [{:keys [taste latin]} data]

(and (map? data)

(vector? taste)

(every? keyword? taste)

(if latin (string? latin) true)))))))

That's pretty fast and compact. Same with Malli:

;; 160ns

(bench-validate! (m/validator schema))

Thanks to the optimized validation engine of Malli, it's over 2x faster than the idiomatic hand-written Clojure. Also, as the validation is derived from the schema, it's always correct and up-to-date.

Validation using clojure.spec:

(require '[clojure.spec.alpha :as s])

(s/def ::name string?)

(s/def ::eatability int?)

(s/def ::taste (s/coll-of keyword? :kind vector?))

(s/def ::latin string?)

(s/def ::data (s/nilable (s/keys :req-un [::taste], :opt-un [::lating])))

(s/def ::mushroom (s/keys :req-un [::name ::eatability ::data]))

;; 950ns

(bench-validate!

(fn [x]

(let [res (s/conform ::mushroom x)]

(not (s/invalid? res)))))

And with Plumatic Schema:

(require '[schema.core :as schema])

(def Mushroom

{:name schema/Str

:eatability schema/Int

:data (schema/maybe {:taste [schema/Keyword]

(schema/optional-key :latin) schema/Str})})

;; 1500ns

(bench-validate!

(let [check (schema/checker Mushroom)]

(fn [x] (nil? (check x)))))

Malli is the fastest here. You should not take the micro-benchmarks too seriously and always conduct your tests with your data models to see the differences. If the performance differences are in order(s) of magnitude, it starts to matter.

Transformation

Mushrooms are entering our system over the wire as JSON. To decode the values from JSON to Clojure, we have to apply a set of transformations. Here's an example of a JSON kanttarelli, parsed from string using jsonista:

(require '[jsonista.core :as j])

(def json

(-> valid1

(j/write-value-as-string)

(j/read-value j/keyword-keys-object-mapper)))

json

;{:name "Kanttarelli"

; :eatability 3

; :data {:taste ["peppery" "fruity" "apricot"]

;        :latin "Cantharellus cibarius"}}

JSON can't handle the keywords, otherwise, it looks correct. A new benchmark suite:

(defn bench-transform! [decode]

(assert (= valid1 (decode json)))

(cc/quick-bench (decode json)))

With idiomatic Clojure:

;; 520ns

(bench-transform!

(fn [{:keys [data] :as mushroom}]

(cond-> mushroom

data (update :data (fn [{:keys [taste] :as data}]

(cond-> data

taste (update :taste (partial mapv keyword))))))))

Despite we only need to transform one value, the transforming function is far from being simple as it has to handle the potential missing values too. We could have used Specter or Meander to simplify the code, but neither can be considered as idiomatic Clojure.

FlameGraph looks calm:

Same transformation with Malli:

(require '[malli.transform :as mt])

;; 250ns

(bench-transform! (m/decoder schema (mt/json-transformer)))

Again, 2x faster. Why would you want to build and maintain the transformation functions manually if you can derive them from a schema and get better performance?

Clojure.spec doesn't support runtime transformations, but we can use spec-tools for the job:

(require '[spec-tools.core :as st])

;; 55000ns

(bench-transform! #(st/coerce ::mushroom % st/json-transformer))

That's two orders of magnitude slower than hand-written Clojure. As spec is built around a global registry, spec-tools walks the spec at runtime to ensure that correct registry references are used. FlameGraph looks like the Himalayas.

Plumatic Schema can cache the transformation functions is much faster than spec-tools:

(require '[schema.coerce :as sc])

;; 3200ns

(bench-transform! (sc/coercer Mushroom sc/json-coercion-matcher))

But wait, Plumatic defines coercion as a process of both transforming and validating in a single sweep. We should test that with Malli too to get a fair comparison.

(defn coercer [schema transformer]

(let [valid? (m/validator schema)

decode (m/decoder schema transformer)

explain (m/explainer schema)]

(fn [x]

(let [res (decode x)]

(cond-> res (not (valid? res)) (explain))))))

;; 470ns

(bench-transform! (coercer schema (mt/json-transformer)))

Malli needs to do two sweeps over the data to get the same functionality, but having two fast sweeps is still an order of magnitude faster than the one sweep of Plumatic in this test.

Inferring Schemas

The original Schema Inferrer in Malli was really slow. Since 0.7.0, it is orders of magnitude faster, thanks to a clear separation of ahead-of-time and actual runtime computations. And it's still just 71 lines of code.

(defn bench-provide! [provide]

(let [samples [valid1 valid2 valid3]]

(cc/quick-bench (provide samples))))

;; 79000µs (0.5.1)

(bench-provide! mp/provide)

;; 280µs (0.7.0) -> 280x

(bench-provide! (mp/provider))

The performance is now in the same ballpark as with spec-provider:

(require '[spec-provider.provider :as sp])

;; 440µs

(bench-provide! #(sp/infer-specs % ::mushroom))

There is still much more we can do to make the Malli providers faster (and better).

Creating Schemas and Workers

One of the main reasons Malli is so fast is that we have pushed computations from runtime (e.g., validation) into the schema and worker creation phase. For the JVM, it's ok to pay a one-time cost to enable a fast runtime. Still, for single-threaded environments like JavaScript, the initial cost of building everything does matter.

In 0.7.0, we measured and optimized the creation phase making most of the operations 1-2 orders of magnitude faster. Here are the relevant parts of that work:

1. new EntryParser protocol for parsing with primitive arrays and supporting laziness

2. patching EntrySchema values without re-parsing

3. new SchemaAST protocol for parse-free schema creation

4. new Cached protocol for memoizing validators, parsers, explainers, and generators

5. removed all calls to satisfy?, go vote up CLJ-1814

6. support for pass-through walking of schemas

Schema creation:

;; 57µs -> 5µs (11x)

(m/schema schema)

;[:map

; [:name string?]

; [:eatability int?]

; [:data [:maybe [:map

;                 [:taste [:vector keyword?]]

;                 [:latin {:optional true} string?]]]]]

Schema creation, using the new Schema AST:

(def ast (m/ast schema))

ast

;{:type :map,

; :keys {:name {:order 0

;               :value {:type string?}},

;        :eatability {:order 1

;                     :value {:type int?}},

;        :data {:order 2,

;               :value {:type :maybe,

;                       :child {:type :map,

;                               :keys {:taste {:order 0

;                                              :value {:type :vector

;                                                      :child {:type keyword?}}},

;                                      :latin {:order 1

;                                              :value {:type string?}

;                                              :properties {:optional true}}}}}}}}

;; 160ns (300x, lazy)

(m/from-ast ast)

Schema Walking, using the identity walker (no-op) as an example:

(def Mushroom (m/schema schema))

;; 28µs -> 2µs (14x)

(m/walk Mushroom (m/schema-walker identity))

;[:map

; [:name string?]

; [:eatability int?]

; [:data [:maybe [:map

;                 [:taste [:vector keyword?]]

;                 [:latin {:optional true} string?]]]]]

Closing a nested schema:

(require '[malli.util :as mu])

;; 55µs -> 5.8µs (9x)

(mu/closed-schema Mushroom)

;[:map {:closed true}

; [:name string?]

; [:eatability int?]

; [:data [:maybe [:map {:closed true}

;                 [:taste [:vector keyword?]]

;                 [:latin {:optional true} string?]]]]]

Removing key from a map-schema:

;; 22µs -> 1.2µs (18x)

(mu/dissoc Mushroom :data)

;[:map

; [:name string?]

; [:eatability int?]]

Transparent caching of schema workers:

;; 60µs -> 190ns (320x)

(m/validate Mushroom valid1)

; => true

Schema creation performance starts to be ok, but there's still room for improvement. We could push all the schema transformation work from initialization time into development time and "freeze" the final schemas as Schema AST, which is the fastest way to load the schemas. This could also reduce the js-bundle size as the transformation utilities could be DCE'd from production builds.

When Performance Matters

Performance is usually not the first concern when selecting a data specification library for Clojure, but when it matters, Malli is there for you. Malli is currently as fast or faster than hand-written idiomatic Clojure on validation and value transformation while still being fully declarative. The new 0.7.0 version is a significant internal rewrite for making schema creation, inferring, and schema transformation order(s) of magnitude faster.

Big thanks to Clojurists Together for funding the Malli development. Also, special thanks to all contributors, especially to Ben Sless for some crazy performance improvements.

For discussion and getting help, there is #malli in Clojurians Slack.

Transforming Data With Malli and Meander

Tuesday, 15 November, 2022 by Tommi Reiman

Prologue

Malli is a high-performance, data-driven data specification library for Clojure. It supports both runtime data validation and static type linting via tools like clj-kondo and core.typed. This is the fourth post on Malli, focusing on data-driven value transformations with the help of Meander. Older posts include:

- Malli, Data-Driven Schemas for Clojure/Script

- Structure and Interpretation of Malli Regex Schemas

- High-Performance Schemas in Clojure/Script with Malli 1/2

Data Transformation

When working with real-life interconnected systems, we are transforming data between different Sources and Targets. You could be reading 3rd party JSON or GraphQL APIs, Legacy XML endpoints, databases and messages from queues in various formats into your target domain models.

In the simplest form, data is transformed via Type Coercion. Malli has a fast and extendable engine for both decoding and encoding values based on schema type and properties. It's covered in the earlier posts, so not going to go into details now.

The second form is transforming different shapes of data, for example reading a flat row from a CSV-file and converting that into domain data with deeply nested values. In the early days of my career, XSLT was the king and there were a lot of commercial tools for doing graphical mapping of the data forms using it.

Screenshot of XSLT-mapper from Oracle SOA Suite 11g Copyright © 2005, 2009, Oracle and/or its affiliates.

Nowadays, most cloud providers offer their own serverless dataflow glue factories and there are also good open source solutions available.

But wouldn't it be nice if we could just use our favourite programming languages for the job?

Clojure

Clojure is our go-to tool for modern cloud development, and it's a great language for working with data. The core library has a lot of tools for transforming maps, sequences and values, while retaining the immutable data semantics.

(-> {:id "1"}

(update :id parse-long)

(assoc :name "Elegia")

(update :tags (fnil conj #{}) "poem"))

; => {:id 1, :name "Elegia", :tags #{"poem"}}

For advanced data processing, you can use libraries like tech.ml.dataset and scicloj.ml.

(require '[tech.v3.dataset :as ds])

(def orders (ds/->dataset "orders.csv" {:key-fn keyword, :parser-fn :string}))

orders

;| :id | :firstName | :lastName |         :street | :item1 | :item2 |  :zip |

;|----:|------------|-----------|-----------------|--------|--------|-------|

;|   1 |      Sauli |  Niinistö |    Mariankatu 2 | coffee |   buns | 00170 |

;|   2 |      Sanna |     Marin | Kesärannantie 1 |  juice |  pasta | 00250 |

(dissoc orders :zip :street)

;| :id | :firstName | :lastName | :item1 | :item2 |

;|----:|------------|-----------|--------|--------|

;|   1 |      Sauli |  Niinistö | coffee |   buns |

;|   2 |      Sanna |     Marin |  juice |  pasta |

Transforming data programmatically is great, but we don't have to stop there. We can describe also the data models and data transformations as data and write an interpreter or compiler for it. Being able to modify the models and processes like the data itself enables us to build truly dynamic systems.

Data, you say? Let's add Malli and Meander to the mix.

The Pipeline

We are building an imaginary service to reduce food waste by consolidating food orders from various systems into a single food delivery system. One of the sources is a CSV-file.

Our target Malli schema looks like this:

(def Order

[:map {:db/table "Orders"}

[:id :uuid]

[:source [:enum "csv" "online"]]

[:source-id :string]

[:name {:optional true} :string]

[:items [:vector :keyword]]

[:delivered {:default false} :boolean]

[:address [:map

[:street :string]

[:zip :string]]]])

Example order:

(require '[malli.generator :as mg])

(mg/generate Order {:seed 3})

;{:id #uuid"b36c2541-2db8-4d75-b87d-3413bdacdb7d",

; :source "online",

; :source-id "",

; :items [:y!Aw11EA :PUPjb-_T :DPXc!g:e],

; :delivered true,

; :address {:street "MG7rxPm6jywJSPqEs"

;           :zip "116iS2c74JGKv90oAhJP7aq7iL8iyk"}}

Extract

We can start by inferring the source schema from samples:

(require '[malli.provider :as mp])

(def CSVOrder (mp/provide (ds/rows orders)))

CSVOrder

;[:map

; [:id :string]

; [:firstName :string]

; [:lastName :string]

; [:street :string]

; [:item1 :string]

; [:item2 :string]

; [:zip :string]]

Validating the result:

(require '[malli.core :as m])

(->> (ds/rows orders)

(map (m/validator CSVOrder))

(every? true?))

; => true

A helper function to coerce the values, throwing an exception on error:

(defn coercer [schema transformer]

(let [valid? (m/validator schema)

decode (m/decoder schema transformer)

explain (m/explainer schema)]

(fn [x]

(let [value (decode x)]

(when-not (valid? value)

(m/-fail! ::invalid-input {:value value

:schema schema

:explain (explain value)}))

value))))

Loading and validating the data, no coercion needed:

(defn load-csv [file]

(ds/rows (ds/->dataset file {:key-fn keyword, :parser-fn :string})))

(require '[malli.transform :as mt])

(def validate-input (coercer CSVOrder (mt/no-op-transformer)))

Testing the data pipeline so far:

(->> (load-csv "orders.csv")

(map validate-input))

;({:id "1",

;  :firstName "Sauli",

;  :lastName "Niinistö",

;  :street "Mariankatu 2",

;  :item1 "coffee",

;  :item2 "buns",

;  :zip "00170"}

; {:id "2",

;  :firstName "Sanna",

;  :lastName "Marin",

;  :street "Kesärannantie 1",

;  :item1 "juice",

;  :item2 "pasta",

;  :zip "00250"})

Transform

We are using Meander to build the transformations. Meander is a great library for creating transparent data transformations in Clojure.

First we build a little helper that allows us to define the Meander patterns and expressions as data, to be compiled later using eval. I'll later describe why this is important.

(require '[meander.match.epsilon :as mme])

(defn matcher [{:keys [pattern expression]}]

(eval `(fn [data#]

(let [~'data data#]

~(mme/compile-match-args

(list 'data pattern expression)

nil)))))

We define a pattern with logic variables (?id, !items etc.) to match the source data and expression to create the target data from the matched variables.

(def transform

(matcher

{:pattern '{:id ?id

:firstName ?firstName

:lastName ?lastName

:street ?street

:item1 !item

:item2 !item

:zip ?zip}

:expression '{:id (random-uuid)

:source "csv"

:source-id ?id

:name (str ?firstName " " ?lastName)

:items !item

:address {:street ?street

:zip ?zip}}}))

We are changing the shape of the data:

- adding the :id and :source fields

- mapping :id to :source-id

- concatenating :name

- collecting the :items vector

- creating a submap :address with :street and :zip

Adding the transformation into our pipeline:

(->> (load-csv "orders.csv")

(map validate-input)

(map transform))

;({:id #uuid"7f765cf9-24a2-4bd8-950f-35f6c8724c65",

;  :source "csv",

;  :source-id "1",

;  :name "Sauli Niinistö",

;  :items #{"buns" "coffee"},

;  :address {:street "Mariankatu 2", :zip "00170"}}

; {:id #uuid"3fc14e81-170f-4234-983a-9d9cc2a47ed5",

;  :source "csv",

;  :source-id "2",

;  :name "Sanna Marin",

;  :items #{"pasta" "juice"},

;  :address {:street "Kesärannantie 1", :zip "00250"}})

We are almost there.

Conform

Last step is to coerce and validate the data against our target schema. For this, we compose a Malli transformer:

(def validate-output

(coercer

Order

(mt/transformer

(mt/string-transformer)

(mt/default-value-transformer))))

Our final transformation pipeline:

(->> (load-csv "orders.csv")

(map validate-input)

(map transform)

(map validate-output))

;({:id #uuid"a3fae918-9b5a-4f54-9d32-4c22b74e8922",

;  :source "csv",

;  :source-id "1",

;  :name "Sauli Niinistö",

;  :items #{:buns :coffee},

;  :address {:street "Mariankatu 2", :zip "00170"},

;  :delivered false}

; {:id #uuid"5f4cb486-6990-457b-8fca-bebc25b23277",

;  :source "csv",

;  :source-id "2",

;  :name "Sanna Marin",

;  :items #{:pasta :juice},

;  :address {:street "Kesärannantie 1", :zip "00250"},

;  :delivered false})

- :items are now coerced into keywords

- :delivered key was set to the default value

The pipeline didn't throw an exception, so the data is valid. We are done.

It's all data

We have defined both the data models (Malli Schemas) and the transformations (Meander Patterns and Expressions) as data. Let's now extract these into a separate transformation map:

(def transformation

{:registry {:csv/order [:map

[:id :string]

[:firstName :string]

[:lastName :string]

[:street :string]

[:item1 :string]

[:item2 :string]

[:zip :string]]

:domain/order [:map {:db/table "Orders"}

[:id :uuid]

[:source [:enum "csv" "online"]]

[:source-id :string]

[:name {:optional true} :string]

[:items [:vector :keyword]]

[:delivered {:default false} :boolean]

[:address [:map

[:street :string]

[:zip :string]]]]}

:mappings {:source :csv/order

:target :domain/order

:pattern '{:id ?id

:firstName ?firstName

:lastName ?lastName

:street ?street

:item1 !item

:item2 !item

:zip ?zip}

:expression '{:id (random-uuid)

:source "csv"

:source-id ?id

:name (str ?firstName " " ?lastName)

:items !item

:address {:street ?street

:zip ?zip}}}})

As it's just data, we can serialize it as EDN or Transit over the wire, inspect and edit it on the browser, send it back and store it in the database.

(require '[clojure.edn :as edn])

(-> transformation

(pr-str)

(edn/read-string)

(= transformation))

; => true

Note: As we use eval to compile the Meander definitions, we should sandbox the execution or use tools like SCI if we are reading the definitions from untrusted sources.

Function to compile the transformation data structure into a Clojure function:

(defn transformer [{:keys [registry mappings]} source-transformer target-transformer]

(let [{:keys [source target]} mappings

xf (comp (map (coercer (get registry source) source-transformer))

(map (matcher mappings))

(map (coercer (get registry target) target-transformer)))]

(fn [data] (into [] xf data))))

Transformation pipeline:

(def pipeline

(transformer

transformation

(mt/no-op-transformer)

(mt/transformer

(mt/string-transformer)

(mt/default-value-transformer))))

Note: It's not yet fully declarative as we are passing source & target transformers into the transformer function, but that would be easy to fix.

Trying it out:

(pipeline (load-csv "orders.csv"))

;({:id #uuid"e809652f-8399-467e-9bab-a0e2bba1548e",

;  :source "csv",

;  :source-id "1",

;  :name "Sauli Niinistö",

;  :items #{:buns :coffee},

;  :address {:street "Mariankatu 2", :zip "00170"},

;  :delivered false}

; {:id #uuid"fc255810-7235-43ba-93bb-e5abc7c78168",

;  :source "csv",

;  :source-id "2",

;  :name "Sanna Marin",

;  :items #{:pasta :juice},

;  :address {:street "Kesärannantie 1", :zip "00250"},

;  :delivered false})

It works. We only needed coercer, matcher, transform and pipeline functions to turn our data into a transformation pipeline. That's around 30 lines of code which could be added to the libraries as the functions are generic.

Performance

Let's compare the performance of our data-driven approach to just writing a simple transformation function in Clojure (without any validation).

(defn clojure-pipeline [x]

(mapv (fn [{:keys [id firstName lastName street item1 item2 zip]}]

{:id (random-uuid)

:source "csv"

:source-id id

:name (str firstName " " lastName)

:items [(keyword item1) (keyword item2)]

:address {:street street

:zip zip}

:delivered false}) x))

Loading 1000 random users for the test:

(require '[hato.client :as hc])

(def data

(let [!id (atom 0)

rand-item #(rand-nth ["bun" "coffee" "pasta" "juice"])]

(->> (hc/get "https://randomuser.me/api/"

{:query-params {:results 1000, :seed 123}

:as :json})

:body

:results

(map (fn [user]

{:id (str (swap! !id inc))

:firstName (-> user :name :first)

:lastName (-> user :name :last)

:street (str (-> user :location :street :name) " "

(-> user :location :street :number))

:item1 (rand-item)

:item2 (rand-item)

:zip (-> user :location :postcode str)})))))

(take 2 data)

;({:id "1"

;  :firstName "Heldo"

;  :lastName "Campos"

;  :street "Rua Três 9120"

;  :item1 "bun"

;  :item2 "juice"

;  :zip "73921"}

; {:id "2",

;  :firstName "Candice",

;  :lastName "Long",

;  :street "Northaven Rd 4744",

;  :item1 "pasta",

;  :item2 "coffee",

;  :zip "25478"})

Benchmarking with Criterium:

(require '[criterium.core :as cc])

;; 480µs

(cc/quick-bench (clojure-pipeline data))

;; 840µs

(cc/quick-bench (pipeline data))

Our data-driven pipeline is about 2x slower in this test, but it validates both the input & output schemas, while the pure Clojure implementation is doing no validation. Flamegraphs below:

idiomatic clojure (no validation)  data-driven transformation (double validation)

Visualization

Another benefit of using a data-oriented approach is that we can visualize or even create visual editors for it. Our transformation visualized:

Epilogue

Clojure is a great language for writing data transformations and for simple cases, that's all we need. But to ensure data correctness, we should also validate data at the borders. Malli is a tool for defining the data schemas and validating & coercing the data. Meander is a data-driven transformation tool that complements Malli. Together they enable fully data-driven transformation pipelines.

Being able to describe systems as data allows us to blur the boundaries between development time and runtime. We can inspect, visualize and edit the system at runtime on the browser, send it over the wire and store it in a database. With the right tools, we can compile applications from the system without sacrificing any runtime performance. With Malli, we can also generate sample data from schemas and use generative testing to verify the transformations logic. We can also pull development-time schemas from runtime data, enabling static type-linting on dynamic schemas. I'll write another post about it.

If Clojure, Malli or building dynamic data-driven systems got you curious, drop me an email and lets talk. Or join Clojurians Slack and find me (@ikitommi) there.

And yes, it's all Open Source.

Table of Contents

Prologue

Epilogue