MXNet made simple: Image Manipulation with OpenCV and MXNet

- 25 MINS

CLOJURE MXNET DEEPLEARNING NDARRAY IMAGE OPENCV TUTORIAL

In this post, we will learn how to perform some common image manipulation operations that are used for preprocessing or postprocessing image datasets. OpenCV and MXNet can manipulate images and NDArrays directly.

Nothing specific to Deep Learning will be covered in this post although this is probably something you need to learn before being able to build your own Computer Vision models.

Before we begin…

We will need to import certain packages:

(require '[clojure.java.io :as io])

;; MXNet namespaces

(require '[org.apache.clojure-mxnet.image :as mx-img])

(require '[org.apache.clojure-mxnet.ndarray :as ndarray])

(require '[org.apache.clojure-mxnet.shape :as mx-shape])

;; OpenCV namespaces

(require '[opencv4.colors.rgb :as rgb])

(require '[opencv4.mxnet :as mx-cv])

(require '[opencv4.core :as cv])

(require '[opencv4.utils :as cvu])

(import org.opencv.core.Mat)

(import java.awt.image.DataBufferByte)

Downloading images from URLs

We can write a small util function that lets us download an image from a URL and save it locally - similar to wget.

(defn download!

"Download `uri` and store it in `filename` on disk"

[uri filename]

(with-open [in (io/input-stream uri)

out (io/output-stream filename)]

(io/copy in out)))

Now, one can download images from URLs

;; Download a cat image from a `uri` and save it into `images/cat.jpg`

(download! "https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/doc/tutorials/python/predict_image/cat.jpg" "images/cat.jpg")

;; Download a dog image from a `uri` and save it into `images/dog.jpg`

(download! "https://github.com/dmlc/web-data/blob/master/mxnet/doc/tutorials/python/predict_image/dog.jpg?raw=true" "images/dog.jpg")

Cat

Dog

Plotting an image from disk

It is often really useful to plot an image that is loaded from a filename on disk

(defn preview!

"Preview image from `filename` and display it on the screen in a new window

Ex:

(preview! \"images/cat.jpg\")

(preview! \"images/cat.jpg\" :h 300 :w 200)"

([filename]

(preview! filename {:h 400 :w 400}))

([filename {:keys [h w]}]

(-> filename

(cv/imread)

(cv/resize! (cv/new-size h w))

(cvu/imshow))))

The function preview! lets you load images in memory and display them

;; Preview an image from disk

(preview! "images/cat.jpg")

;; Preview with different size

(preview! "images/cat.jpg" {:h 300 :w 200})

Image preprocessing

Resizing, Scaling, Normalizing

When working with image datasets, we always have to preprocess them. Below is a list of preprocessing steps that are commonly done:

- Subtract mean of a pixel value for Red, Green and Blue channels

- Resize an image to fit the expected dimensions of a model

- Rescale pixel values linearly from a range to another - Eg. (-128, 127) -> (0, 127)

- Normalize pixel values into the (0.0, 1.0) range

(defn preprocess-mat

"Preprocessing steps on a `mat` from OpenCV.

Example of commons preprocessing tasks"

[mat]

(-> mat

;; Subtract mean

(cv/add! (cv/new-scalar 103.939 116.779 123.68))

;; Resize

(cv/resize! (cv/new-size 400 400))

;; Maps pixel values from [-128, 127] to [0, 127]

(cv/convert-to! cv/CV_8SC3 0.5)

;; Normalize pixel values into (0.0, 1.0)

; (cv/normalize! 0.0 1.0 cv/NORM_MINMAX cv/CV_32FC1)

))

One can now visualize what the processing steps do to an image

(-> "images/cat.jpg"

(cv/imread)

(preprocess-mat)

(cvu/imshow))

Preprocessed Cat

Raw Byte Values

Origami, the Clojure OpenCV wrapper, provides serveral utility functions to make it simple to look at raw byte values. One would need to do this when performing preprocessing steps in order to check that is was done properly at the pixel value level.

;; Looking at the raw bytes

(-> "images/dog.jpg"

;; Read image from disk

(mx-img/read-image {:to-rgb false})

;; Resizing image

(mx-img/resize-image 200 200)

;; Convert NDArray to Mat

(mx-cv/ndarray-to-mat)

;; Mat to bytes

(cv/<<)

(cv/->bytes)

(vec))

- cv/<<: converts the mat to bytes

- cv/->bytes returns the byte values from the mat as a byte-array

Color channels

A color image is represented as a matrix of pixel values for each red, green and blue channels. It is generally useful to look at a specific color channel of an image.

(defn mat->color-mat

"Returns Mat of the selected channel.

Assumes the Mat is in RGB format.

`mat`: Mat object from OpenCV

`color`: value in #{:red :green :blue}

Ex

(mat->color-mat m :green)

(mat->color-mat m :blue)"

[mat color]

(let [color->selector #(get {:red first :green second :blue last} % first)]

((color->selector color) (cv/split! mat))))

We can now use the mat->color-mat function to display the color channel of the cat image

;; Display all channels (blue, green and red) of a picture

(-> "images/cat.jpg"

;; Read image from disk

(mx-img/read-image {:to-rgb true})

;; Resize

(mx-img/resize-image 200 200)

;; Convert NDArray to Mat

(mx-cv/ndarray-to-mat)

;; Extract the different color channels

((juxt #(mat->color-mat % :blue)

#(mat->color-mat % :green)

#(mat->color-mat % :red)))

;; Display the images

(#(map cvu/imshow %)))

Red Channel

Blue Channel

Green Channel

Cat Red Channel

Cat Blue Channel

Cat Green Channel

Type conversions

MXNet Image API

As we mentioned in a previous blog post, MXNet models can only consume NDArrays. Therefore, one needs to turn raw images into NDArrays.

Unsurprisingly, MXNet provides an Image API to load image files from disk and convert them directly to NDArrays. Under the hood, MXNet uses OpenCV to perform the conversion.

The code below demonstrates how to:

1. Load an image from disk using MXNet Image API

2. Resize an image using MXNet Image API

3. Convert it to a Java BufferedImage

4. Save it back on disk

(-> "images/dog.jpg"

;; Convert filename to NDArray

(mx-img/read-image {:to-rgb true}) ;; option :to-rgb, true by default

;; Resizing image to height = 400, width = 400

(mx-img/resize-image 400 400)

;; Convert to BufferedImage

(mx-img/to-image)

;; Saving BufferedImage to disk

(javax.imageio.ImageIO/write "jpg" (java.io.File. "test2.jpg")))

The code below demonstrates how to convert a BufferedImage into a Mat

;; Showing an image using `buffered-image-to-mat`

(-> "images/dog.jpg"

;; Read image from disk

(mx-img/read-image {:to-rgb true})

;; Convert to BufferedImage - Can be very slow...

(mx-img/to-image)

;; Convert to Mat

(cvu/buffered-image-to-mat)

;; Show Mat

(cvu/imshow))

OpenCV API

One can also work directly with OpenCV to achieve the same results. OpenCV uses a Java Mat Object internally. We can write some util functions to convert from mat to ndarray and the other way around too.

(defn mat->ndarray

"Convert a `mat` from OpenCV to an MXNet `ndarray`"

[mat]

(let [h (.height mat)

w (.width mat)

c (.channels mat)]

(-> mat

cvu/mat->flat-rgb-array

(ndarray/array [c h w]))))

(defn ndarray->mat

"Convert a `ndarray` to an OpenCV `mat`"

[ndarray]

(let [shape (mx-shape/->vec ndarray)

[h w _ _] (mx-shape/->vec (ndarray/shape ndarray))

bytes (byte-array shape)

mat (cv/new-mat h w cv/CV_8UC3)]

(.put mat 0 0 bytes)

mat))

Finally, we can write a function that loads an image from disk and returns an NDArray

(defn filename->ndarray!

"Convert an image stored on disk `filename` into an `ndarray`

`filename`: string representing the image on disk

`shape-vec`: is the actual shape of the returned `ndarray`

return: ndarray"

[filename shape-vec]

(-> filename

(cv/imread)

(mat->ndarray)))

Image Postprocessing

Bounding boxes are returned by models doing image detection tasks. We can easily plot bounding boxes on an image using OpenCV

(defn draw-bounding-box!

"Draw bounding box on `img` given the `top-left` and `bottom-right` coordonates.

Add `label` when provided.

returns: nil"

[img {:keys [label top-left bottom-right]}]

(let [[x0 y0] top-left

[x1 y1] bottom-right

top-left-point (cv/new-point x0 y0)

bottom-right-point (cv/new-point x1 y1)]

(cv/rectangle img top-left-point bottom-right-point rgb/white 1)

(when label

(cv/put-text! img label top-left-point cv/FONT_HERSHEY_DUPLEX 1.0 rgb/white 1))))

We can now draw the bounding box of the cookie

;; Drawing one bounding box on an image of dog

(let [img (cv/imread "images/dog.jpg")]

(draw-bounding-box! img {:top-left [200 440]

:bottom-right [350 525]

:label "cookie"})

(cvu/imshow img))

Preprocessed Cat

Models that perform object detection tasks return a collection of bounding boxes. We iterate over the predictions and draw the bounding boxes with their associated labels

(defn draw-predictions!

"Draw all predictions on an `img` passing `results` which is a collection

of bounding boxes data.

returns: nil"

[img results]

(doseq [{:keys [label top-left bottom-right] :as result} results]

(draw-bounding-box! img result)))

And below is an example

;; Drawing multiple bounding boxes on an image of dog

(let [img (cv/imread "images/dog.jpg")

results [{:top-left [200 70] :bottom-right [830 430] :label "dog"}

{:top-left [200 440] :bottom-right [350 525] :label "cookie"}]]

(draw-predictions! img results)

(cvu/imshow img))

Preprocessed Cat

Conclusion

Image manipulation is simple with MXNet and OpenCV. Image data will need to be preprocessed to be consumed by MXNet models: raw bytes to NDArrays. We have covered different APIs that are useful for performing such operations: the MXNet Image API and the OpenCV API.

References and Resources

- MXNet NDArray API Reference

- MXNet Image API Reference

- OpenCV website

- Clojure OpenCV wrapper - origami

- MXNet Made Simple: Clojure NDArray API

Here is also the code used in this post - also available in this repository

(ns mxnet-clj-tutorials.image-manipulation

"Image manipulation tutorial."

(:require

[clojure.java.io :as io]

[org.apache.clojure-mxnet.image :as mx-img]

[org.apache.clojure-mxnet.ndarray :as ndarray]

[org.apache.clojure-mxnet.shape :as mx-shape]

[opencv4.colors.rgb :as rgb]

[opencv4.mxnet :as mx-cv]

[opencv4.core :as cv]

[opencv4.utils :as cvu])

(:import org.opencv.core.Mat java.awt.image.DataBufferByte))

(defn download!

"Download `uri` and store it in `filename` on disk"

[uri filename]

(with-open [in (io/input-stream uri)

out (io/output-stream filename)]

(io/copy in out)))

(defn preview!

"Preview image from `filename` and display it on the screen in a new window

Ex:

(preview! \"images/cat.jpg\")

(preview! \"images/cat.jpg\" :h 300 :w 200)"

([filename]

(preview! filename {:h 400 :w 400}))

([filename {:keys [h w]}]

(-> filename

cv/imread

(cv/resize! (cv/new-size h w))

cvu/imshow)))

(defn preprocess-mat

"Preprocessing steps on a `mat` from OpenCV.

Example of commons preprocessing tasks"

[mat]

(-> mat

;; Subtract mean

(cv/add! (cv/new-scalar 103.939 116.779 123.68))

;; Resize

(cv/resize! (cv/new-size 400 400))

;; Maps pixel values from [-128, 128] to [0, 127]

(cv/convert-to! cv/CV_8SC3 0.5)

;; Normalize pixel values into (0.0, 1.0)

; (cv/normalize! 0.0 1.0 cv/NORM_MINMAX cv/CV_32FC1)

))

(defn mat->ndarray

"Convert a `mat` from OpenCV to an MXNet `ndarray`"

[mat]

(let [h (.height mat)

w (.width mat)

c (.channels mat)]

(-> mat

cvu/mat->flat-rgb-array

(ndarray/array [c h w]))))

(defn ndarray->mat

"Convert a `ndarray` to an OpenCV `mat`"

[ndarray]

(let [shape (mx-shape/->vec ndarray)

[h w _ _] (mx-shape/->vec (ndarray/shape ndarray))

bytes (byte-array shape)

mat (cv/new-mat h w cv/CV_8UC3)]

(.put mat 0 0 bytes)

mat))

(defn filename->ndarray!

"Convert an image stored on disk `filename` into an `ndarray`

`filename`: string representing the image on disk

`shape-vec`: is the actual shape of the returned `ndarray`

return: ndarray"

[filename shape-vec]

(-> filename

cv/imread

mat->ndarray))

(defn draw-bounding-box!

"Draw bounding box on `img` given the `top-left` and `bottom-right` coordonates.

Add `label` when provided.

returns: nil"

[img {:keys [label top-left bottom-right]}]

(let [[x0 y0] top-left

[x1 y1] bottom-right

top-left-point (cv/new-point x0 y0)

bottom-right-point (cv/new-point x1 y1)]

(cv/rectangle img top-left-point bottom-right-point rgb/white 1)

(when label

(cv/put-text! img label top-left-point cv/FONT_HERSHEY_DUPLEX 1.0 rgb/white 1))))

(defn draw-predictions!

"Draw all predictions on an `img` passing `results` which is a collection

of bounding boxes data.

returns: nil"

[img results]

(doseq [{:keys [label top-left bottom-right] :as result} results]

(draw-bounding-box! img result)))

(defn mat->color-mat

"Returns Mat of the selected channel.

Assumes the Mat is in RGB format.

`mat`: Mat object from OpenCV

`color`: value in #{:red :green :blue}

Ex

(mat->color-mat m :green)

(mat->color-mat m :blue)"

[mat color]

(let [color->selector #(get {:red first :green second :blue last} % first)]

((color->selector color) (cv/split! mat))))

(comment

;; Download a cat image from a `uri` and save it into `images/cat.jpg`

(download! "https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/doc/tutorials/python/predict_image/cat.jpg" "images/cat.jpg")

;; Download a dog image from a `uri` and save it into `images/dog.jpg`

(download! "https://github.com/dmlc/web-data/blob/master/mxnet/doc/tutorials/python/predict_image/dog.jpg?raw=true" "images/dog.jpg")

;; Preview an image from disk

(preview! "images/cat.jpg")

;; Preview with different size

(preview! "images/cat.jpg" {:h 300 :w 200})

;; Visualize preprocessing steps

(-> "images/cat.jpg"

(cv/imread)

(preprocess-mat)

(cvu/imshow))

;; Writing image to disk

(-> "images/dog.jpg"

;; Convert filename to NDArray

(mx-img/read-image {:to-rgb true}) ;; option :to-rgb, true by default

;; Resizing image to height = 400, width = 400

(mx-img/resize-image 400 400)

;; Convert to BufferedImage

(mx-img/to-image)

;; Saving BufferedImage to disk

(javax.imageio.ImageIO/write "jpg" (java.io.File. "test2.jpg")))

;; Writing image to disk

(-> "images/mnist_digit_8.jpg"

;; Load image from disk

(mx-img/read-image {:to-rgb true})

;; Convert NDArray to Mat

(ndarray->mat)

;; Save Image to disk

(cv/imwrite "test-digit.jpg"))

; cvu/imshow

;; Showing an image using `buffered-image-to-mat`

(-> "images/dog.jpg"

;; Read image from disk

(mx-img/read-image {:to-rgb true})

;; Convert to BufferedImage - Can be very slow...

(mx-img/to-image)

;; Convert to Mat

(cvu/buffered-image-to-mat)

;; Show Mat

(cvu/imshow))

;; Showing an image using `ndarray->mat`

(-> "images/dog.jpg"

;; Read image from disk

(mx-img/read-image {:to-rgb false})

;; Convert NDArray to Mat

(ndarray->mat)

;; Show Mat

(cvu/imshow))

;; Showing an image using `mx-cv/ndarray-to-mat` from `origami`

;; MXNet default channel ordering is: RGB

;; OpenCV default channel ordering is: BGR

(-> "images/dog.jpg"

;; Read image from disk

(mx-img/read-image {:to-rgb false})

;; Convert NDArray to Mat

(mx-cv/ndarray-to-mat)

;; Show Mat

(cvu/imshow))

;; Looking at the raw bytes: -128 to 127

(-> "images/dog.jpg"

;; Read image from disk

(mx-img/read-image {:to-rgb false})

;; Resizing image

(mx-img/resize-image 200 200)

;; Convert NDArray to Mat

(mx-cv/ndarray-to-mat)

;; Mat to bytes

(cv/<<)

(cv/->bytes)

(vec))

;; Display green channel of a picture

(-> "images/cat.jpg"

;; Read image from disk

(mx-img/read-image {:to-rgb true})

;; Convert NDArray to Mat

(mx-cv/ndarray-to-mat)

;; Extract the red color channel

(mat->color-mat :green)

;; Display the image

(cvu/imshow))

;; Display all channels (blue, green and red) of a picture

(-> "images/cat.jpg"

;; Read image from disk

(mx-img/read-image {:to-rgb true})

;; Resize

(mx-img/resize-image 200 200)

;; Convert NDArray to Mat

(mx-cv/ndarray-to-mat)

;; Extract the different color channels

((juxt #(mat->color-mat % :blue)

#(mat->color-mat % :green)

#(mat->color-mat % :red)))

;; Display the images

(#(map cvu/imshow %)))

;; Drawing one bounding box on an image of dog

(let [img (cv/imread "images/dog.jpg")]

(draw-bounding-box! img {:top-left [200 440]

:bottom-right [350 525]

:label "cookie"})

(cvu/imshow img))

;; Drawing multiple bounding boxes on an image of dog

(let [img (cv/imread "images/dog.jpg")

results [{:top-left [200 70] :bottom-right [830 430] :label "dog"}

{:top-left [200 440] :bottom-right [350 525] :label "cookie"}]]

(draw-predictions! img results)

(cvu/imshow img))

MXNet made simple: Pretrained Models for image classification - Inception and VGG

- 22 MINS

CLOJURE MXNET DEEPLEARNING INCEPTION VGG IMAGE CLASSIFICATION MODEL ZOO TUTORIAL

In this post, we will learn how to leverage pretrained models to perform image classification. The Computer Vision task is to associate a label with an unseen image. State of the art Machine Learning models have hundred of layers and require days and sometimes even weeks to train on GPUs. MXNet enables us to leverage such models very easily. We will see how to use three common Deep Learning models: Inception and VGG.

Before we begin…

We will need to import certain packages:

(require '[clojure.string :as string])

;; MXNet namespaces

(require '[org.apache.clojure-mxnet.module :as m])

(require '[org.apache.clojure-mxnet.ndarray :as ndarray])

;; OpenCV namespaces

(require '[opencv4.core :as cv])

(require '[opencv4.utils :as cvu])

The MXNet Model Zoo

The MXNet Model Zoo is a set of pretrained models including the computation graphs and their trained parameters. We can download the models from the Model Zoo. First, let’s download VGG16 by running the following bash script

#!/bin/bash

set -evx

mkdir -p model

cd model

wget http://data.mxnet.io/models/imagenet/vgg/vgg16-symbol.json

wget http://data.mxnet.io/models/imagenet/vgg/vgg16-0000.params

cd ..

- vgg16-symbol.json: serialized computation graph that can be loaded by MXNet

- vgg16-0000.params: binary file containing the NDArrays of the trained parameters

Let’s also download the Inception pretrained models

#!/bin/bash

set -evx

mkdir -p model

cd model

wget http://data.mxnet.io/models/imagenet/inception-bn/Inception-BN-symbol.json

wget http://data.mxnet.io/models/imagenet/inception-bn/Inception-BN-0126.params

mv Inception-BN-0126.params Inception-BN-0000.params

cd ...

We will also need to download the ImageNet categories which are essentially a mapping from integer classes to human readable strings.

$ wget http://data.mxnet.io/models/imagenet/synset.txt

$ head -10 synset.txt

n01440764 tench, Tinca tinca

n01443537 goldfish, Carassius auratus

n01484850 great white shark, white shark, man-eater, man-eating shark, Carcharodon carcharias

n01491361 tiger shark, Galeocerdo cuvieri

n01494475 hammerhead, hammerhead shark

n01496331 electric ray, crampfish, numbfish, torpedo

n01498041 stingray

n01514668 cock

n01514859 hen

n01518878 ostrich, Struthio camelus

Pretrained Models

Before we dive into the code, we can compare the different pretrained models that we have downloaded. Choosing the right model for your application comes down to understanding the tradeoffs and the choices you have.

- What is your RAM constraint for running the model?

- How fast does the model run on CPU, GPU?

- What is the error rate you can deal with?

Below is a table that summarizes some key metrics for the three different models we will be using:

VGG16

Inception v3

RAM required

528 MB

43 MB

Number of Parameters

140 million

25 million

Error Rate (ImageNet Challenge)

7.4%

5.6%

Topology

VGG16 Topology

Inception Topology

If you want to learn more about the MXNet vizualization API, it is covered in a previous post here.

VGG16

Winner of the 2014 ImageNet challenge by achieving a 7.4% error rate on image classification. It is a model built from 16 layers - Research Paper

Inception v3

Published in December 2015. It achieved 15-25% more accurate predictions than the best models at the time while being six times cheaper computationally and using five time less parameters - Research Paper

Performing image Classification with pretrained Models

Loading the models

Now that the models are on your disk, MXNet can load their computation graphs and the pretrained parameters (weights and biases). First, we define where the models are located and the size of the images that we will feed into the Models.

(def model-dir "model/")

(def h 224) ;; Image height

(def w 224) ;; Image width

(def c 3)   ;; Number of channels: Red, Green, Blue

Time has come to load the models with the MXNet Module API.

;; Loading VGG16

(defonce vgg-16-mod

(-> {:prefix (str model-dir "vgg16") :epoch 0}

(m/load-checkpoint)

;; Define the shape of input data and bind the name of the input layer

;; to "data"

(m/bind {:for-training false

:data-shapes [{:name "data" :shape [1 c h w]}]})))

;; Loading Inception v3

(defonce inception-mod

(-> {:prefix (str model-dir "Inception-BN") :epoch 0}

(m/load-checkpoint)

;; Define the shape of input data and bind the name of the input layer

;; to "data"

(m/bind {:for-training false

:data-shapes [{:name "data" :shape [1 c h w]}]})))

The last step is to load the ImageNet labels that we downloaded before.

(defonce image-net-labels

(-> (str model-dir "/synset.txt")

(slurp)

(string/split #"\n")))

;; ImageNet 1000 Labels check

(assert (= 1000 (count image-net-labels)))

Preparing the Data

Our models expect the data to have the following shape [batch-size c h w] where

- batch-size: number of datapoints to feed - we will set batch-size to 1 here since we want to predict one image at a time and not batch them

- c: number of color channels - Red, Green, Blue - 3

- h: height of the image - 244

- w: width of the image - 244

We also need to normalize the image pixel values according to the way the different models were trained. In our case, we only need to subtract the mean (from the ImageNet dataset) color channels which are well known values.

(defn preprocess-img-mat

"Preprocessing steps on an `img-mat` from OpenCV to feed into the Model"

[img-mat]

(-> img-mat

;; Resize image to (w, h)

(cv/resize! (cv/new-size w h))

;; Maps pixel values from [-128, 128] to [0, 127]

(cv/convert-to! cv/CV_8SC3 0.5)

;; Substract mean pixel values from ImageNet dataset

(cv/add! (cv/new-scalar -103.939 -116.779 -123.68))

;; Flatten matrix

(cvu/mat->flat-rgb-array)

;; Reshape to (1, c, h, w)

(ndarray/array [1 c h w])))

If you need a refresher on image manipulation and processing, it is covered in a previous post here.

Predicting

We would like to make new predictions on images that we will feed to the models. The three models return an NDArray of probabilities:

- 1000 values between 0.0 and 1.0

- The sum of these 1000 values is equal to 1.0

- Each index in the NDArray corresponds to a label (Eg. Dog, Cat, Guitar, …)

We will look at the top k labels that the model returns for a given image.

(defn- top-k

"Return top `k` from prob-maps with :prob key"

[k prob-maps]

(->> prob-maps

(sort-by :prob)

(reverse)

(take k)))

(defn predict

"Predict with `model` the top `k` labels from `labels` of the ndarray `x`"

([model labels x]

(predict model labels x 5))

([model labels x k]

(let [probs (-> model

(m/forward {:data [x]})

(m/outputs)

(ffirst)

(ndarray/->vec))

prob-maps (mapv (fn [p l] {:prob p :label l}) probs labels)]

(top-k k prob-maps))))

To run the model, one only needs to use the m/forward function to perform a forward pass.

Prediction Comparisons

We can now run the models on different images and look at the top 5 predictions to get a feel for how they perform.

Cat

(->> "images/cat2.jpg"

(cv/imread)

(preprocess-img-mat)

(predict inception-mod image-net-labels))

;({:prob 0.9669817, :label "n02124075 Egyptian cat"}

;{:prob 0.020066999, :label "n02123045 tabby, tabby cat"}

;{:prob 0.0071042357, :label "n02123159 tiger cat"}

;{:prob 0.005353994, :label "n02127052 lynx, catamount"}

;{:prob 4.658187E-5, :label "n02123597 Siamese cat, Siamese"})

(->> "images/cat2.jpg"

(cv/imread)

(preprocess-img-mat)

(predict vgg-16-mod image-net-labels))

;({:prob 0.9030159, :label "n02124075 Egyptian cat"}

;{:prob 0.05147686, :label "n02123045 tabby, tabby cat"}

;{:prob 0.024212556, :label "n02123159 tiger cat"}

;{:prob 0.0099070445, :label "n02127052 lynx, catamount"}

;{:prob 3.7205187E-4, :label "n04040759 radiator"})

Dog

(->> "images/dog2.jpg"

(cv/imread)

(preprocess-img-mat)

(predict inception-mod image-net-labels))

;({:prob 0.7363852, :label "n02110958 pug, pug-dog"}

;{:prob 0.23988461, :label "n02108422 bull mastiff"}

;{:prob 0.013495497, :label "n02108915 French bulldog"}

;{:prob 0.0019004685, :label "n02093428 American Staffordshire terrier, Staffordshire terrier, American pit bull terrier, pit bull terrier"}

;{:prob 0.0013417465, :label "n04409515 tennis ball"})

(->> "images/dog2.jpg"

(cv/imread)

(preprocess-img-mat)

(predict vgg-16-mod image-net-labels))

;({:prob 0.95628285, :label "n02110958 pug, pug-dog"}

;{:prob 0.02271582, :label "n02108422 bull mastiff"}

;{:prob 0.0075261267, :label "n02108915 French bulldog"}

;{:prob 0.0014686864, :label "n02086079 Pekinese, Pekingese, Peke"}

;{:prob 0.0012910544, :label "n02108089 boxer"})

Guitar Player

(->> "images/guitarplayer2.jpg"

(cv/imread)

(preprocess-img-mat)

(predict inception-mod image-net-labels))

;({:prob 0.647201, :label "n03272010 electric guitar"}

;{:prob 0.3371953, :label "n04296562 stage"}

;{:prob 0.008809802, :label "n02676566 acoustic guitar"}

;{:prob 0.0024602208, :label "n02787622 banjo"}

;{:prob 0.0018765739, :label "n03759954 microphone, mike"})

(->> "images/guitarplayer2.jpg"

(cv/imread)

(preprocess-img-mat)

(predict vgg-16-mod image-net-labels))

;({:prob 0.73966444, :label "n03272010 electric guitar"}

;{:prob 0.105860166, :label "n04296562 stage"}

;{:prob 0.059584185, :label "n04141076 sax, saxophone"}

;{:prob 0.029627431, :label "n02787622 banjo"}

;{:prob 0.016049441, :label "n02676566 acoustic guitar"})

Overall the predictions look really good! Don’t you think?

Below is a summary of the predictions to compare how the different models perform on those images.

VGG16

Inception v3

DogDog

1. pug, pug-dog

1. pug, pug-dog

2. bull mastiff

2. bull mastiff

3. French bulldog

3. French bulldog

4. Pekinese, Pekingese, Peke

4. American Staffordshire terrier, Staffordshire terrier, American pit bull terrier, pit bull terrier

5. boxer

5. tennis ball

Cat

1. Egyptian cat

1. Egyptian cat

2. tabby, tabby cat

2. tabby, tabby cat

3. tiger cat

3. tiger cat

4. lynx, catamount

4. lynx, catamount

5. radiator

5. Siamese cat, Siamese

Guitar PlayerGuitar Player

1. electric guitar

1. electric guitar

2. stage

2. stage

3. sax, saxophone

3. acoustic guitar

4. banjo

4. banjo

5. acoustic guitar

5. microphone, mike

Conclusion

Now you can use pretrained, state of the art Deep Learning Models for image classification. Most of the code is about data preparation. The actual prediction code is only 4 lines of code.

References and Resources

- ImageNet Challenge

- VGG16 Research Paper

- Inception v3 Research Paper

- MXNet Model Zoo

- An Introduction to the MXNet API - Part 4

- MXNet made simple: Image Manipulation with OpenCV and MXNet

- MXNet made simple: Clojure Symbol Visualization API

Here is also the code used in this post - also available in this repository

(ns mxnet-clj-tutorials.pretrained

"Tutorial on pretrained models with MXNet: Inception and VGG."

(:require

[clojure.string :as string]

[opencv4.core :as cv]

[opencv4.utils :as cvu]

[org.apache.clojure-mxnet.module :as m]

[org.apache.clojure-mxnet.ndarray :as ndarray]))

;;; Loading the Models

(def model-dir "model/")

(def h 224) ;; Image height

(def w 224) ;; Image width

(def c 3)   ;; Number of channels: Red, Green, Blue

;; Pretrained Inception BN model loaded from disk

(defonce inception-mod

(-> {:prefix (str model-dir "Inception-BN") :epoch 0}

(m/load-checkpoint)

;; Define the shape of input data and bind the name of the input layer

;; to "data"

(m/bind {:for-training false

:data-shapes [{:name "data" :shape [1 c h w]}]})))

(defonce vgg-16-mod

(-> {:prefix (str model-dir "vgg16") :epoch 0}

(m/load-checkpoint)

;; Define the shape of input data and bind the name of the input layer

;; to "data"

(m/bind {:for-training false

:data-shapes [{:name "data" :shape [1 c h w]}]})))

;; ImageNet 1000 Labels

(defonce image-net-labels

(-> (str model-dir "/synset.txt")

(slurp)

(string/split #"\n")))

(assert (= 1000 (count image-net-labels)))

;;; Preparing the Data

(defn preprocess-img-mat

"Preprocessing steps on an `img-mat` from OpenCV to feed into the Model"

[img-mat]

(-> img-mat

;; Resize image to (w, h)

(cv/resize! (cv/new-size w h))

;; Maps pixel values from [-128, 128] to [0, 127]

(cv/convert-to! cv/CV_8SC3 0.5)

;; Substract mean pixel values from ImageNet dataset

(cv/add! (cv/new-scalar -103.939 -116.779 -123.68))

;; Flatten matrix

(cvu/mat->flat-rgb-array)

;; Reshape to (1, c, h, w)

(ndarray/array [1 c h w])))

;;; Predicting

(defn- top-k

"Return top `k` from prob-maps with :prob key"

[k prob-maps]

(->> prob-maps

(sort-by :prob)

(reverse)

(take k)))

(defn predict

"Predict with `model` the top `k` labels from `labels` of the ndarray `x`"

([model labels x]

(predict model labels x 5))

([model labels x k]

(let [probs (-> model

(m/forward {:data [x]})

(m/outputs)

(ffirst)

(ndarray/->vec))

prob-maps (mapv (fn [p l] {:prob p :label l}) probs labels)]

(top-k k prob-maps))))

(comment

(->> "images/guitarplayer.jpg"

(cv/imread)

(preprocess-img-mat)

(predict inception-mod image-net-labels))

;({:prob 0.68194896, :label "n04296562 stage"}

;{:prob 0.06861413, :label "n03272010 electric guitar"}

;{:prob 0.04886661, :label "n10565667 scuba diver"}

;{:prob 0.044686787, :label "n03250847 drumstick"}

;{:prob 0.029348794, :label "n02676566 acoustic guitar"})

(->> "images/guitarplayer.jpg"

(cv/imread)

(preprocess-img-mat)

(predict vgg-16-mod image-net-labels))

;({:prob 0.31067622, :label "n03272010 electric guitar"}

;{:prob 0.14873363, :label "n04296562 stage"}

;{:prob 0.04211086, :label "n04141076 sax, saxophone"}

;{:prob 0.032480247, :label "n04536866 violin, fiddle"}

;{:prob 0.022555437, :label "n03110669 cornet, horn, trumpet, trump"})

(->> "images/guitarplayer2.jpg"

(cv/imread)

(preprocess-img-mat)

(predict inception-mod image-net-labels))

;({:prob 0.647201, :label "n03272010 electric guitar"}

;{:prob 0.3371953, :label "n04296562 stage"}

;{:prob 0.008809802, :label "n02676566 acoustic guitar"}

;{:prob 0.0024602208, :label "n02787622 banjo"}

;{:prob 0.0018765739, :label "n03759954 microphone, mike"})

(->> "images/guitarplayer2.jpg"

(cv/imread)

(preprocess-img-mat)

(predict vgg-16-mod image-net-labels))

;({:prob 0.73966444, :label "n03272010 electric guitar"}

;{:prob 0.105860166, :label "n04296562 stage"}

;{:prob 0.059584185, :label "n04141076 sax, saxophone"}

;{:prob 0.029627431, :label "n02787622 banjo"}

;{:prob 0.016049441, :label "n02676566 acoustic guitar"})

(->> "images/cat.jpg"

(cv/imread)

(preprocess-img-mat)

(predict inception-mod image-net-labels))

;({:prob 0.5226559, :label "n02119789 kit fox, Vulpes macrotis"}

;{:prob 0.14540964, :label "n02112018 Pomeranian"}

;{:prob 0.13845555, :label "n02119022 red fox, Vulpes vulpes"}

;{:prob 0.06784552, :label "n02120505 grey fox, gray fox, Urocyon cinereoargenteus"}

;{:prob 0.024868377, :label "n02441942 weasel"})

(->> "images/cat.jpg"

(cv/imread)

(preprocess-img-mat)

(predict vgg-16-mod image-net-labels))

;({:prob 0.41937035, :label "n02119789 kit fox, Vulpes macrotis"}

;{:prob 0.26819462, :label "n02119022 red fox, Vulpes vulpes"}

;{:prob 0.07655225, :label "n02124075 Egyptian cat"}

;{:prob 0.049807232, :label "n02123159 tiger cat"}

;{:prob 0.034435965, :label "n02123045 tabby, tabby cat"})

(->> "images/cat2.jpg"

(cv/imread)

(preprocess-img-mat)

(predict inception-mod image-net-labels))

;({:prob 0.9669817, :label "n02124075 Egyptian cat"}

;{:prob 0.020066999, :label "n02123045 tabby, tabby cat"}

;{:prob 0.0071042357, :label "n02123159 tiger cat"}

;{:prob 0.005353994, :label "n02127052 lynx, catamount"}

;{:prob 4.658187E-5, :label "n02123597 Siamese cat, Siamese"})

(->> "images/cat2.jpg"

(cv/imread)

(preprocess-img-mat)

(predict vgg-16-mod image-net-labels))

;({:prob 0.9030159, :label "n02124075 Egyptian cat"}

;{:prob 0.05147686, :label "n02123045 tabby, tabby cat"}

;{:prob 0.024212556, :label "n02123159 tiger cat"}

;{:prob 0.0099070445, :label "n02127052 lynx, catamount"}

;{:prob 3.7205187E-4, :label "n04040759 radiator"})

(->> "images/dog.jpg"

(cv/imread)

(preprocess-img-mat)

(predict inception-mod image-net-labels))

;({:prob 0.89285797, :label "n02110958 pug, pug-dog"}

;{:prob 0.06376573, :label "n04409515 tennis ball"}

;{:prob 0.01919549, :label "n03942813 ping-pong ball"}

;{:prob 0.014978847, :label "n02108422 bull mastiff"}

;{:prob 0.0012790044, :label "n02808304 bath towel"})

(->> "images/dog.jpg"

(cv/imread)

(preprocess-img-mat)

(predict vgg-16-mod image-net-labels))

;({:prob 0.96750915, :label "n02110958 pug, pug-dog"}

;{:prob 0.01833086, :label "n02108422 bull mastiff"}

;{:prob 0.005593519, :label "n04409515 tennis ball"}

;{:prob 0.0017559915, :label "n02108089 boxer"}

;{:prob 8.5579534E-4, :label "n02096585 Boston bull, Boston terrier"})

(->> "images/dog2.jpg"

(cv/imread)

(preprocess-img-mat)

(predict inception-mod image-net-labels))

;({:prob 0.7363852, :label "n02110958 pug, pug-dog"}

;{:prob 0.23988461, :label "n02108422 bull mastiff"}

;{:prob 0.013495497, :label "n02108915 French bulldog"}

;{:prob 0.0019004685, :label "n02093428 American Staffordshire terrier, Staffordshire terrier, American pit bull terrier, pit bull terrier"}

;{:prob 0.0013417465, :label "n04409515 tennis ball"})

(->> "images/dog2.jpg"

(cv/imread)

(preprocess-img-mat)

(predict vgg-16-mod image-net-labels))

;({:prob 0.95628285, :label "n02110958 pug, pug-dog"}

;{:prob 0.02271582, :label "n02108422 bull mastiff"}

;{:prob 0.0075261267, :label "n02108915 French bulldog"}

;{:prob 0.0014686864, :label "n02086079 Pekinese, Pekingese, Peke"}

;{:prob 0.0012910544, :label "n02108089 boxer"})

)

« MXNet made simple: Image Manipulation with OpenCV and MXNet

MXNet made simple: Image RecordIO with im2rec and Data Loading »