Build Your Own Static Vector Tile Pipeline

19 minute read

r

Following on from James Milner’s introduction to Tiler, this post will drill into a little bit more detail about vector tiles, and the low-level commands tools like Tiler use behind the scenes. By understanding the commands, you’ll be able to build your own static vector tile pipeline.

NOTE: If you want to generate vector tiles that frequently need updating, you would be better off importing your data into PostGIS and then using a tool such as Tegola to generate them dynamically from the database as they are requested. The approach described in this post is better for generating a set of vector tiles all in one go.

This post describes the exact commands you’ll need to run assuming you are on the latest macOS (the most common set up amongst Engineers at the Hub at the moment). The same pipeline will work on Linux as well as previous versions of MacOS if you install the core tools we use yourself.

We’ll also assume you have a basic geospatial knowledge. If you haven’t, it is worth knowing that shapefiles, GeoJSON and TopoJSON are just ways of storing points, lines and areas. Within each file, each point, line or area is called a feature. Web maps generally load the data they need to display in tiles, small squares of data that each represent one small part of the map you are looking at.

Read this GIS File Formats page first for an overview of different approaches and their pros and cons.

It is also worth knowing that, slightly confusingly, a single shapefile is actually distributed as four separate files, and that you need all four to be able to work with the shapefile, not just the .shp file itself.

Throughout this tutorial we’ll be using compressed vector tiles that are smaller to store and quicker to transfer than their uncompressed counterparts.

Also, if you stop the tutorial and pick it up again, you’ll need to re-run all the export commands to set up all your paths correctly again.

Mapbox Vector Tiles

Mapbox Vector Tiles are a modern way of storing and transmitting the same sort of feature data you might ordinarily find in a shapefile, GeoJSON or TopoJSON file.

There are two features of vector tiles that make them particularly interesting as a source format for displaying data on maps:

- In preparing the tiles, the data for the features the tileset represents is chopped up into individual tiles. This means the data for a large feature like a coastline will no longer exist as a single complex shape, but instead each tile that needs to display a portion of it will contain just the information needed to render the part of the coastline it is responsible for.

- Mapbox Vector Tiles are a compact binary encoding of the vector data called Google Protocol Buffers that is smaller than corresponding JSON, and usually smaller than a corresponding traditional raster tile too.

Because of these innovations, vector tiles have a major advantage over non-tiled formats like GeoJSON and TopoJSON:

- Map clients only need to download the data for the tiles they need to display, not all the data for all features that have some part on the map (as they would with GeoJSON or TopoJSON)

- Even if source GeoJSON or TopoJSON was prepared as separate tiles, Vector tiles can be obtained by browsers more quickly because they are a more compact binary format so file sizes are lower.

Being a vector format (containing portions of the actual source data), vector tiles have advantages over traditional raster tiles (just pictures of the pre-coloured data) too:

- Map clients can access the vector data directly so have the ability to re-colour or style the data dynamically themselves without needing to load more data from the server (something that can’t be done with raster tiles, which are just images of the data)

- Map clients can zoom in and out smoothly between the different zoom levels because the data can be easily scaled while new data is loaded. This is known as over-zooming or under-zooming.

- Vector tiles often have smaller file sizes than raster tiles. For example imagine you want to draw a line. In an image, you’d need to colour in each pixel between the start and end point (perhaps 20 numbers), whereas as a vector, you just specify numbers for the start and end point.

These advantages of vector tiles all add up to:

- A faster and smoother experience for the end user

- Lower bandwidth costs for the hosting

- The potential for building more advanced map applications that make use of the data transmitted

Hosting

Vector tiles are also fairly easy to host cheaply since they can be represented as a simple directory structure of files.

Google Cloud Storage is one place to host the tiles so that they can be retrieved by a map client. The advantage over running your own server is that Google will look after keeping the files accessible so that you don’t need hosting expertise or 24 hour monitoring and response to keep your tiles available. It is also a cheaper option than hosting directly with Mapbox.

Modern browsers are able to read compressed files directly if they are compressed in gzip format and served with the following header:

Content-Encoding: gzip

In this post we’ll assume you have a host that is capable of adding this header to HTTP responses so that we can store and transfer compressed tiles, rather than uncompressed ones which take up more space and take longer to transfer.

Getting Started

To get started, why not download some shapefiles for London to experiment with from the Ordnance Survey Open Local download page.

You’ll need to tick the box to the right of the National Grid Reference squares label, in the Download column, and then scroll down to select TQ from the list next to the grid squares diagram. Scroll to the very bottom of the page, click Continue and follow the instructions.

The page mentions that it can take up to 2 hours to fulfil the request. In practice it is nearly instant so don’t be put off.

Repeat the process to download the TR grid square data too. (You can also request them both at the same time by selecting them both to start with).

Note: Each download has a data directory containing different shapefiles representing the different types of data available. For example, the shapefile for the roads in the TQ square actually comes as TQ_Road.shp, TQ_Road.shp, TQ_Road.shp and TQ_Road.shp. Tiler will automatically extract the data from the archives you’ve downloaded as long as you use the correct name, in this case Road.

Next you must download the ostn02-ntv2-data.zip file from OS here. It contains the OSTN02_NTv2.gsb file you need in order to get the high resolution transforms from British National Grid into WGS84 that we need.

At this point make a directory named src. Extract the ostn02-ntv2-data.zip archive and copy the OSTN02_NTv2.gsb file into the src directory.

Finally put the two zip files you’ve downloaded for TQ and TR in the src directory too, next to OSTN02_NTv2.gsb.

You should now have an src directory containing 3 files and nothing else:

- OSTN02_NTv2.gsb

- opmplc_essh_tq.zip

- opmplc_essh_tr.zip

We’ll now write some re-usable commands to process this data, without any manual steps. This will then scale to more layers later.

Install the Tools

If you have Homebrew installed you can use the brew command to install GDAL:

brew install gdal tippecanoe

If you already have them installed, upgrade to the latest versions:

brew update

brew upgrade gdal tippecanoe

Otherwise, follow the advanced instructions below.

Advanced Install

NOTE: You can skip this section if you used the brew option above.

Download and install GDAL by following the download instructions.

Next install the very latest version tippecanoe. To do this you’ll need the Xcode command line tools. They come automatically if you install Xcode, or if you have tried to use command line tools in the past. If you don’t already have Xcode command line tools installed, you can install them from a terminal.

To get a terminal, click the spotlight search icon in the bar at the top right of your screen (it looks like a magnifying glass) and type “Terminal”. Click the terminal icon.

At the terminal type cc and press enter and you’ll be prompted to install the Xcode command line tools. Follow the instructions. If instead you see a clang error, that means you already have the tools.

Next download the latest tippecanoe from here:

- https://github.com/mapbox/tippecanoe/archive/master.zip

Double click it to unzip it to a tippecanoe-master folder

You can now install it by typing these commands at a terminal:

cd ~/Downloads/tippecanoe-master

make

export PATH=~/Downloads/tippecanoe-master:$PATH

You should now be able to run the following two commands:

ogr2ogr --version

tippecanoe -v

On my computer I get:

GDAL 2.1.2, released 2016/10/24

tippecanoe v1.18.1

You’ll need at least tippecanoe 1.18.1.

Extract the files from the downloads in src

First create a build directory for your work, next to the src directory (not inside it).

Now run this command in the directory that contains your src and build directories. It will extract the OS data into build/shapefiles:

find "src" | grep ".zip" | xargs -n1 unzip -d "build/shapefiles"

If you run it from the wrong place, it won’t be able to find the src directory.

Find the Road shapefiles from each grid square

The OS road layer is named Road so we need to find all the road shapefiles that we just extracted into /build/shapefiles.

We can find them like this:

export LAYER=Road

find "build/shapefiles" | grep "_${LAYER}.shp" > "build/${LAYER}_shapefiles.txt"

You can see the shapefiles we found with:

cat "build/${LAYER}_shapefiles.txt"

I get:

build/shapefiles/OS OpenMap Local (ESRI Shape File) TQ/data/TQ_Road.shp

build/shapefiles/OS OpenMap Local (ESRI Shape File) TR/data/TR_Road.shp

Merge the Road shapefiles togehter

Now we need to put all the shapefiles together. Slightly confusingly the ogr2ogr tool from GDAL requires us to do this in two commands (it is just the way it works).

The first command will take the first shapefile and makes a copy of it. The second command is used for appending the other shapefiles to the first.

Let’s set the layer we want again:

export LAYER=Road

Now here’s the first command for copying the first shapefile:

ogr2ogr -f 'ESRI Shapefile' "build/$LAYER.shp" -nln "${LAYER}" "`cat "build/${LAYER}_shapefiles.txt" | head -n 1`"

You can now see the four files that make up the Road shapefile in build.

Now let’s append all the other shapefiles (in this case there is only one, TQ, but there might be more later:

cat "build/${LAYER}_shapefiles.txt" | tail -n +2 | while read line; do

ogr2ogr -f 'ESRI Shapefile' -update -append "build/$LAYER.shp" "$line" -nln "${LAYER}"

done

Generate a GeoJSON file

tippecanoe which will eventually generate out vector tiles requires a GeoJSON file as input, so we need to convert the Road shapefile to GeoJSON.

One complication is that our source data is in a British National Grid projection but we need it in WGS84 (a.k.a EPSG:4326) for GeoJSON, so as well as converting formats, we’ll re-project the data too.

We can do this all in one step like this, using the OSTN02_NTv2.gsb file you put in the src directory earlier:

mkdir build/geojson

export LAYER=Road

ogr2ogr -dim 2 -f 'GeoJSON' -s_srs "+proj=tmerc +lat_0=49 +lon_0=-2 +k=0.999601 +x_0=400000 +y_0=-100000 +ellps=airy +units=m +no_defs +nadgrids=./src/OSTN02_NTv2.gsb" -t_srs 'EPSG:4326' "build/geojson/$LAYER.json" "build/$LAYER.shp"

This is a little complex, because it is reprojecting data as well as changing its format, but you can look at these links to understand it:

- Why transforming data in the UK is so important for GIS

- OSTN02 – NTv2 format

- Transformations to OSGB36 and WGS84 in GDAL/OGR

- GDAL info about a number of SRS sources

- A guide to coordinate systems in Great Britain PDF

Now we have the Road.json file in build/geojson.

Run tippecanoe to generate a directory of Mapbox Vector Tiles

We can finally generate compressed vector tiles with this command for hosting:

tippecanoe --no-feature-limit --no-tile-size-limit --exclude-all --minimum-zoom=5 --maximum-zoom=g --output-to-directory "build/www/tiles" `find ./build/geojson -type f | grep .json`

By default, tippecanoe will create one Vector Tile layer per .json file, and name it according to the JSON filename. In this case with just one .json file, it will create a set of tiles with just one layer named Road. Rename the file before running the command if you would like it named differently.

There are lots of options you can use, but our experience has been that the automatic simplification options such as --drop-fraction-as-needed don’t produce good results. Simplifying isn’t easy, and it becomes partly an issue of cartography.

Just to describe two of the options chosen in particular:

- --minimum-zoom=5 - Don’t generate tiles for layers below 5, there will be too much data for the clients to be able to display them smoothly anyway.

- --maximum-zoom=g - Let tippecanoe guess the maximum zoom level based on how close together the data is (makes sure we don’t generate more tiles than we need to)

- --exclude-all - Don’t include attributes about the features in the tiles, only include the vector data itself (keeps the tiles small)

It is likely that in future, you’ll want to use --include instead of --exclude-all to keep specific attributes attached to the vector data (such as road names for example) but for now we’ll strip all the extra data out to keep the tiles small.

Have a look at the tippecanoe docs and feel free to experiment with different options.

ADVANCED TIP: Through a lot of trial and error I’ve found that if your tiles end up too big to display well in a particular client, it is much better to filter the data in the input files at the start of the process for different zoom levels, than to try to automatically simplify later.

Pipeline

Putting everything together, we now have this set of commands to perform all the steps run so far to generate a new build directory when run in a folder with the src directory:

export LAYER=Road

mkdir -p build/www/tiles/

find "src" | grep ".zip" | xargs -n1 unzip -d "build/shapefiles"

find "build/shapefiles" | grep "_${LAYER}.shp" > "build/${LAYER}_shapefiles.txt"

ogr2ogr -f 'ESRI Shapefile' "build/$LAYER.shp" -nln "${LAYER}" "`cat "build/${LAYER}_shapefiles.txt" | head -n 1`"

cat "build/${LAYER}_shapefiles.txt" | tail -n +2 | while read line; do

ogr2ogr -f 'ESRI Shapefile' -update -append "build/$LAYER.shp" "$line" -nln "${LAYER}"

done

mkdir build/geojson

ogr2ogr -dim 2 -f 'GeoJSON' -s_srs "+proj=tmerc +lat_0=49 +lon_0=-2 +k=0.999601 +x_0=400000 +y_0=-100000 +ellps=airy +units=m +no_defs +nadgrids=./src/OSTN02_NTv2.gsb" -t_srs 'EPSG:4326' "build/geojson/$LAYER.json" "build/$LAYER.shp"

tippecanoe --no-feature-limit --no-tile-size-limit --exclude-all --minimum-zoom=5 --maximum-zoom=g --output-to-directory "build/www/tiles" `find ./build/geojson -type f | grep .json`

Generate a sample Mapbox GL JS web app

Now you’ve generated the tiles, let’s create a web map to have a look at them.

Add some files to the /build/www directory created by tippecanoe:

cat << EOF > "build/www/index.html"

<!DOCTYPE html>

<html>

<head>

<meta charset="utf-8" />

<title>Map</title>

<meta name="viewport" content="initial-scale=1,maximum-scale=1,user-scalable=no" />

<script src="https://api.mapbox.com/mapbox-gl-js/v0.35.1/mapbox-gl.js"></script>

<link href="https://api.mapbox.com/mapbox-gl-js/v0.35.1/mapbox-gl.css" rel="stylesheet" />

<style>

body { margin:0; padding:0; }

#map { position:absolute; top:0; bottom:0; width:100%; }

</style>

</head>

<body>

<div id="map"></div>

<script>

mapboxgl.accessToken = "not-needed-unless-using-mapbox-styles";

var map = new mapboxgl.Map({

container: "map",

style: "http://localhost:8000/style.json",

center: [-0.1047, 51.5236],

// These affect the hard limits of the zoom controls

minZoom: 5,

zoom: 11,

maxZoom: 22

});

map.addControl(new mapboxgl.NavigationControl());

</script>

</body>

</html>

EOF

cat << EOF > "build/www/style.json"

{

"version": 8,

"name": "Custom",

"metadata": {

"mapbox:autocomposite": true

},

"glyphs": "mapbox://fonts/mapbox/{fontstack}/{range}.pbf",

"sources": {

"composite": {

"type": "vector",

"tiles": ["http://localhost:8000/tiles/{z}/{x}/{y}.pbf"],

"minzoom": 0,

"maxzoom": 15

}

},

"layers": [

{

"id": "background",

"type": "background",

"paint": {

"background-color": "#e3decb"

}

},

{

"id": "Road",

"type": "line",

"source": "composite",

"source-layer": "Road",

"minzoom": 0,

"maxzoom": 22,

"paint": {

"line-color": "#FFFFFF",

"line-width": 0.5

}

}

]

}

EOF

Keeping the styles all in an external style.json is a bit different from the examples you’ll see online for Mapbox GL JS, but the advantage is that the same style formats will work with the other clients such as Mapbox GL iOS and Mapbox GL Android.

Under and Overzooming

One of the nice features of vector tiles is that you don’t necessarily need to generate tiles for every zoom layer the browser supports. As long as you tell Mapbox which tiles exist in each source, it can use the vector data to dynamically resize the information from the closest tiles that are available. This is known as under- or over-zooming.

The minzoom and maxzoom settings in the composite source specify which zoom levels the source physically contains files for (which zoom levels you generated with tippecanoe).

The minzoom and maxzoom in each style layer can be different from the values specified in the source (as long as they are within the bounds specified by maxZoom and minZoom (note the capital Z) in index.html. If they extend beyond the values you have tiles for, the Mapbox tile client will over-zoom or under-zoom based on the nearest available tile.

In this case, it means roads will still be visible in the map at zoom level 22, even though we only have tiles for level 15.

Since we’re only using up to zoom level 5, we didn’t technically need to generate tiles 0-5.

Local Server

Now you can run a server capable of correctly serving gzipped vector tiles by adding a Content-Encoding: gzip header to each response. (The added header means the browser will unzip the tiles before passing them to the Mapbox client, so that Mapbox still gets the unzipped tiles, but you only need to transmit and store compressed ones).

One suitable server is live-server running with a middleware plugin, but it is a bit fiddly to set up.

To use it you’ll need to install Node Current, for example with this command:

brew install node@7

Then install live-server like this:

npm install live-server

This will create a node_modules/.bin directory containing live-server. Add this to your path:

export PATH=$PWD/node_modules/.bin:$PATH

You can create support for gzipped tiles by creating this file:

cat << EOF > "build/gzip.js"

module.exports = function(req, res, next) {

if (req.url.endsWith('.pbf')) {

next();

res.setHeader('Content-Encoding', 'gzip');

} else {

next();

}

}

EOF

Then run the server like this:

live-server --port=8000 --middleware="${PWD}/build/gzip.js" --host=localhost build/www

There you go, your very own Vector Tiles!

Styling for Different Zoom Levels

If you play with the example as it stands, you’ll notice that things get pretty crowded as you zoom out. You might want to change the road thickness to become thinner as you zoom out.

Try replacing the single road style layer you have so far in build/www/style.json with these three style layers, which change the width depending on the zoom range:

{

"id": "Road",

"type": "line",

"source": "composite",

"source-layer": "Road",

"minzoom": 13,

"maxzoom": 15,

"paint": {

"line-color": "#FFFFFF",

"line-width": 3.5

}

},

{

"id": "Road2",

"type": "line",

"source": "composite",

"source-layer": "Road",

"minzoom": 10,

"maxzoom": 13,

"paint": {

"line-color": "#ffffff",

"line-width": 1.5

}

},

{

"id": "Road3",

"type": "line",

"source": "composite",

"source-layer": "Road",

"minzoom": 0,

"maxzoom": 10,

"paint": {

"line-color": "#ffffff",

"line-width": 0.5

}

}

CAUTION:

- When you are changing the styling, make sure you put the background layer first, otherwise you’ll paint a background over the top of everything and see nothing!

- Make sure you give each layer a different id, even if it is actually using the same source and layer in the underlying vector tiles otherwise Mapbox won’t be able to render it correctly.

When you’ve finished styling the map, live-server should automatically reload and you’ll get his map:

Vector Tiles Map

If you zoom out a bit you’ll see the extent of the data you’ve generated tiles for. Notice that all the roads are shown, but the individual road widths is thinner because of the styles you’ve set for each zoom range.

Vector Tiles Map

These screenshots contain OS Open Local data © Crown copyright and database right 2017

Advanced Styling Based on Attributes

Whilst changing styling based on zoom level is handy, it is also useful to be able to style the one vector tile layer in different ways according to the data it contains. In the case of roads, you might want main roads like A roads and motorways to be styled differently from B roads. You might want minor roads to only be visible when you are quite zoomed in.

You can achieve this with attributes and layer style filters.

When you ran tippecanoe you used the --exclude-all option to exclude all the attribute data from the original shapefiles. This time, let’s use --include to include just one attribute of the Road shapefile called CLASSIFICA:

mv build/www/tiles build/www/tiles.old

time tippecanoe --no-feature-limit --no-tile-size-limit --include="CLASSIFICA" --maximum-zoom=16 --output-to-directory "build/www/tiles" `find ./build/geojson -type f | grep .json`

The tiles generated this time will include this extra data.

There is plenty of documntation on the OS Open Local data and in particular, the Product Guide pag 14 describes the Road attribute classifications:

- A Road

- B Road

- Local Road

- Local Access Road

- Restricted Local Access Road

- Minor Road

- Primary Road

- Motorway

- A Road, Collapsed Dual Carriageway

- B Road, Collapsed Dual Carriageway

- Minor, Collapsed Dual Carriageway

- Primary, Collapsed Dual Carriageway

- Motorway, Collapsed Dual Carriageway

- Shared use Carriageway

- Guided Busway Carriageway

ADVANCED TIP: There is a program called QGIS that you can install that helps you work with shapefiles. If you drag the TQ_Roads.shp file into QGIS it will draw all the roads for you. If you click the Open Attribute Table button you’ll see a table of all the attribute data available, and the names of the columns.

This means that now in the style layers, you can choose which features get styled by filtering only the features that have the attributes you want to style.

Here’s an example where we create three road style layers, each with a different filter that uses the CLASSIFICA attribute. Note that some of the style laters only show at certain zoom levels.

{

"id": "Road",

"type": "line",

"source": "composite",

"source-layer": "Road",

"minzoom": 12,

"maxzoom": 15,

"paint": {

"line-color": "#FFFFFF",

"line-width": 1.5

},

"filter": ["!in", "CLASSIFICA", "A Road", "A Road, Collapsed Dual Carriageway", "Motorway", "Primary Road", "Primary Road, Collapsed Dual Carriageway", "B Road"]

},

{

"id": "MajorRoad",

"type": "line",

"source": "composite",

"source-layer": "Road",

"minzoom": 0,

"maxzoom": 15,

"paint": {

"line-color": "#000000",

"line-width": 3.5

},

"filter": ["in", "CLASSIFICA", "A Road", "A Road, Collapsed Dual Carriageway", "Motorway", "Primary Road", "Primary Road, Collapsed Dual Carriageway"]

},

{

"id": "BRoad",

"type": "line",

"source": "composite",

"source-layer": "Road",

"minzoom": 0,

"maxzoom": 15,

"paint": {

"line-color": "#888",

"line-width": 1.8

},

"filter": ["in", "CLASSIFICA", "B Road", "B Road, Collapsed Dual Carriageway"]

}

Summary

At this point you’ve successfully build a map using Vector Tiles that has different layers (roads and tidal water), and has different style layers for the road layer.

This is a great achievement! There’s still a lot more to learn about vector tiles including sprites, labels and simplification, but hopefully getting this far will have given you a great starting point for your explorations.

Feel free to leave questions or comments below and I’ll try my best to answer them.

Creative Commons Share-a-like James Gardner © Geovation

Tags: notes

Categories: tech

Updated:

Vector tiles standards

On this page

- Format

- Encoding geometry

- Encoding attributes

- Winding order

- Implementations

- What the spec doesn't cover

- How to use vector tiles as a dataset

- How to encode attributes that aren't strings or numbers

- Clipping

- Simplification

Ready to get started?

Create a free account to start building with Mapbox.

Sign Up

Join the Mapbox Developers Discord Community

arrow-right

This guide on vector tile standards explains how data is stored and encoded in vector tiles.

Just as applications make assumptions about what information exists in a database, so do Mapbox tools make assumptions about how vector tiles store geographic information. Understanding these standards can help you understand how to work with Mapbox vector tiles.

This guide covers file format, examples of geometry encoding and attribute encoding, an explanation of the importance of winding order, and a list of concepts not included in the Mapbox vector tile specification.

For information about file formats and extensions, projections and bounds, and the internal structure of vector tiles, you can read the Mapbox Vector Tile Specification on GitHub.

Format

Vector tiles are encoded as Google Protobufs (PBF), which allow for serializing structured data. For clarity, Mapbox Vector Tiles use the .mvt file suffix. The specification details are largely structured around the rules implemented in the base .proto file.

Mapbox Vector Tiles are not related at all to OpenStreetMap PBF files. PBFs are a format, much like XML and can take many forms. Mapbox Vector Tiles and OpenStreetMap PBFs are protobuf files, but conform to completely different specifications and are used in different ways.

Encoding geometry

To encode geographic information into a vector tile a tool must convert geographic coordinates, such as latitude and longitude, into vector tile grid coordinates. Vector tiles hold no concept of geographic information. They encode points, lines, and polygons as x/y pairs relative to the top left of the grid in a right-down manner.

This is a step-by-step example showing how a single vector tile encodes geometry in the grid. It follows the commands of the "pen" to encode two rings.

Step 0

An empty vector tile

The vector tile to the right is a 10x10 grid with 2 cell buffer. Let's encode some geometry to the grid starting with a blue polygon. The following commands will be relative to the pen (black dot).

Encoding attributes

Attributes are encoded in a series of tags that exist within a feature in the vector that have integer values that reference keys and values designating the original key:value pairs from the geometry. For large geometry, this removes redundancy for attributes that have the same keys and similar values.

Take a look at the original GeoJSON FeatureCollection on the left and see how its individual parts are encoded into the proper tags of the vector tile protobuf.

Hover over the features and the properties of the GeoJSON.

Original geojson

{

"type": "FeatureCollection",

"features": [

{

"geometry": { ... },

"type": "Feature",

"properties": {

"hello": "world",

"h": "world",

"count": 1.23

}

},

{

"geometry": { ... },

"type": "Feature",

"properties": {

"hello": "again",

"count": 2

}

}

]}

Final vector tile

layers {

version: 2 name: "points"

features: {

id: 1

tags: 0

tags: 0

tags: 1

tags: 0

tags: 2

tags: 1

type: Point geometry: ...

}

features {

id: 2

tags: 0

tags: 2

tags: 2

tags: 3

type: Point geometry: ...

}

keys: "hello"

keys: "h"

keys: "count"

values: { string_value: "world" }

values: { double_value: 1.23 }

values: { string_value: "again" }

values: { int_value: 2 }

extent: 4096

}

Winding order

Winding order refers to the direction the vector tile draws a ring, either clockwise or counter-clockwise. Many geometries are multipolygons with "holes," which are also represented as polygon rings. It is important to be able to infer winding order to extract source data from a vector tile and understand if the geometry is part of a multipolygon or a unique polygon.

Extracting the original data from images has been difficult on maps in the past, because of the loss of underlying metadata from the geometry that might have been used to create the image. Yet, with the introduction of client side rendering of vector tiles via GL technologies, the raw geometry data has become useful for a source of information outside of rendering.

For renderers to appropriately distinguish which polygons are holes and which are unique geometries, the specification requires all polygons to be valid (OGC validity). Any polygon interior ring must be oriented with the winding order opposite that of their parent exterior ring and all interior rings must directly follow the exterior ring to which they belong. Exterior rings must be oriented clockwise and interior rings must be oriented counter-clockwise (when viewed in screen coordinates).

The following example geometries show how encoding a ring's winding order can affect the rendered result. Each example assumes all rings are part of the same multipolygon.

Note: the Y axis is positive-downward in vector tile coordinates!

Description

Winding order

Rendered

A single ring, in clockwise order is rendered as a single, solid polygon.

Ring 1: Clockwise

Two rings with the same winding order will render as two unique polygons overlapping.

Ring 1: Clockwise Ring 2: Clockwise

Two rings, the first (exterior) ring is in clockwise order, while the second is counter-clockwise. This results in a "hole" in the final render.

Ring 1: Clockwise Ring 2: Counter-Clockwise

Three rings in a multipolygon that alternate winding order.

Ring 1: Clockwise Ring 2: Counter-Clockwise Ring 3: Clockwise

Implementations

Implementations of the Mapbox Vector Tile Specification are far and wide. Many of them fall into one of the following categories:

- Parsers & generators: libraries that read and/or encode vector tiles, some also have command line utilities

- Clients: web-based tools that render vector tiles that conform to the specification

- Applications: browser-based tools for creating and visualizing vector tiles

- Servers: support rendering and serving up vector tiles (note: the specification doesn't go into how to do this explicitly)

Visit Mapbox Vector Tiles for a list of tools implementing vector tiles.

What the spec doesn't cover

This specification is explicit in the way a vector tile should pack data. But, there are some related concepts that this specification does not cover.

How to use vector tiles as a dataset

This specification IS NOT intended to explain how to use vector tiles as a dataset. This is something that has been considered for the future, but it will likely be a separate specification. This specification does not cover how to store, request, or share vector tiles. Consider this specification like how the PNG spec explains how to pack data.

How to encode attributes that aren't strings or numbers

Attributes in geometric data, such as the properties object in GeoJSON can include more than strings and numbers. They can be arrays or objects. The specification doesn't cover how to encode these values and is up to the encoder to decide. Tools like Mapnik will turn arrays and objects into strings, which requires the decoder to parse them. For example, if you had a GeoJSON property called categories and it was an array:

"categories": ["one", "two", "three"]

It would be converted into a string and stored in the protobuf like this:

keys: "categories"

values: {

string_value: "[\"one\",\"two\",\"three\"]"

}

Clipping

The specification does not explain how geographic data should be clipped between vector tiles since clipping, like simplification, can be executed in many ways. Mapbox specifically clips features at a buffer around the tile (see the encoding example above). Any geometry within this buffer is assumed to carry over to another tile. This is up for consideration for a future release.

Note: encoded geometry in vector tiles can extend beyond the bound of the tile. This means features are not required to be clipped.

A common question, when it comes to clipping is "how do renderers know which lines to connect for clipped geometry?". This is the reason Mapbox adds a buffer to vector tiles and clipped geometry. When it is time to render, Mapbox sets the canvas to the exact tile size, which sets the edges outside of the visual frame, thus the tiles all line up. So, there is no need to know which nodes are part of others for rendering purposes. Although, you could use the id field in the protobuf to store information necessary for reconstructing polygons.

Simplification

The conversion from geographic coordinates (latitude and longitude) to vector tile coordinates (x, y) is an important step, but can be implemented in many different ways before vector tile encoding. It is not included in this specification, but there are some important GOTCHAs worth noting.

Simplification can cause invalid polygons according to the OGC standards by oversimplifying polygon rings to the point where their edges overlap. See below how simplifying one line changes the rendering of a polygon by pushing the interior ring outside of the exterior ring.

When spatial coordinates are converted to tile coordinates, they are rounded to integers. Simplifying (rounding) the coordinates can reverse the winding order. Consider a triangle polygon that is simplified to the vector tile grid. The rounded point can cross over the polygon and "flip" it, rendering its winding order reversed.

Vector Tile Specification

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.

1. Purpose

This document specifies a space-efficient encoding format for tiled geographic vector data. It is designed to be used in browsers or server-side applications for fast rendering or lookups of feature data.

2. File Format

The Vector Tile format uses Google Protocol Buffers as a encoding format. Protocol Buffers are a language-neutral, platform-neutral extensible mechanism for serializing structured data.

2.1. File Extension

The filename extension for Vector Tile files SHOULD be mvt. For example, a file might be named vector.mvt.

2.2. Multipurpose Internet Mail Extensions (MIME)

When serving Vector Tiles the MIME type SHOULD be application/vnd.mapbox-vector-tile.

3. Projection and Bounds

A Vector Tile represents data based on a square extent within a projection. A Vector Tile SHOULD NOT contain information about its bounds and projection. The file format assumes that the decoder knows the bounds and projection of a Vector Tile before decoding it.

Web Mercator is the projection of reference, and the Google tile scheme is the tile extent convention of reference. Together, they provide a 1-to-1 relationship between a specific geographical area, at a specific level of detail, and a path such as https://example.com/17/65535/43602.mvt.

Vector Tiles MAY be used to represent data with any projection and tile extent scheme.

4. Internal Structure

This specification describes the structure of data within a Vector Tile. The reader should have an understanding of the Vector Tile protobuf schema document and the structures it defines.

4.1. Layers

A Vector Tile consists of a set of named layers. A layer contains geometric features and their metadata. The layer format is designed so that the data required for a layer is contiguous in memory, and so that layers can be appended to a Vector Tile without modifying existing data.

A Vector Tile SHOULD contain at least one layer. A layer SHOULD contain at least one feature.

A layer MUST contain a version field with the major version number of the Vector Tile specification to which the layer adheres. For example, a layer adhering to version 2.1 of the specification contains a version field with the integer value 2. The version field SHOULD be the first field within the layer. Decoders SHOULD parse the version first to ensure that they are capable of decoding each layer. When a Vector Tile consumer encounters a Vector Tile layer with an unknown version, it MAY make a best-effort attempt to interpret the layer, or it MAY skip the layer. In either case it SHOULD continue to process subsequent layers in the Vector Tile.

A layer MUST contain a name field. A Vector Tile MUST NOT contain two or more layers whose name values are byte-for-byte identical. Prior to appending a layer to an existing Vector Tile, an encoder MUST check the existing name fields in order to prevent duplication.

Each feature in a layer (see below) may have one or more key-value pairs as its metadata. The keys and values are indices into two lists, keys and values, that are shared across the layer's features.

Each element in the keys field of the layer is a string. The keys include all the keys of features used in the layer, and each key may be referenced by its positional index in this set of keys, with the first key having an index of 0. The set of keys SHOULD NOT contain two or more values which are byte-for-byte identical.

Each element in the values field of the layer encodes a value of any of several types (see below). The values represent all the values of features used in the layer, and each value may be referenced by its positional index in this set of values, with the first value having an index of 0. The set of values SHOULD NOT contain two or more values of the same type which are byte-for-byte identical.

In order to support values of varying string, boolean, integer, and floating point types, the protobuf encoding of the value field consists of a set of optional fields. A value MUST contain exactly one of these optional fields.

A layer MUST contain an extent that describes the width and height of the tile in integer coordinates. The geometries within the Vector Tile MAY extend past the bounds of the tile's area as defined by the extent. Geometries that extend past the tile's area as defined by extent are often used as a buffer for rendering features that overlap multiple adjacent tiles.

For example, if a tile has an extent of 4096, coordinate units within the tile refer to 1/4096th of its square dimensions. A coordinate of 0 is on the top or left edge of the tile, and a coordinate of 4096 is on the bottom or right edge. Coordinates from 1 through 4095 inclusive are fully within the extent of the tile, and coordinates less than 0 or greater than 4096 are fully outside the extent of the tile. A point at (1,10) or (4095,10) is within the extent of the tile. A point at (0,10) or (4096,10) is on the edge of the extent. A point at (-1,10) or (4097,10) is outside the extent of the tile.

4.2. Features

A feature MUST contain a geometry field.

A feature MUST contain a type field as described in the Geometry Types section.

A feature MAY contain a tags field. Feature-level metadata, if any, SHOULD be stored in the tags field.

A feature MAY contain an id field. If a feature has an id field, the value of the id SHOULD be unique among the features of the parent layer.

4.3. Geometry Encoding

Geometry data in a Vector Tile is defined in a screen coordinate system. The upper left corner of the tile (as displayed by default) is the origin of the coordinate system. The X axis is positive to the right, and the Y axis is positive downward. Coordinates within a geometry MUST be integers.

A geometry is encoded as a sequence of 32 bit unsigned integers in the geometry field of a feature. Each integer is either a CommandInteger or a ParameterInteger. A decoder interprets these as an ordered series of operations to generate the geometry.

Commands refer to positions relative to a "cursor", which is a redefinable point. For the first command in a feature, the cursor is at (0,0) in the coordinate system. Some commands move the cursor, affecting subsequent commands.

4.3.1. Command Integers

A CommandInteger indicates a command to be executed, as a command ID, and the number of times that the command will be executed, as a command count.

A command ID is encoded as an unsigned integer in the least significant 3 bits of the CommandInteger, and is in the range 0 through 7, inclusive. A command count is encoded as an unsigned integer in the remaining 29 bits of a CommandInteger, and is in the range 0 through pow(2, 29) - 1, inclusive.

A command ID, a command count, and a CommandInteger are related by these bitwise operations:

CommandInteger = (id & 0x7) | (count << 3)

id = CommandInteger & 0x7

count = CommandInteger >> 3

A command ID specifies one of the following commands:

Command

Id

Parameters

Parameter Count

MoveTo

1

dX, dY

2

LineTo

2

dX, dY

2

ClosePath

7

No parameters

0

Example Command Integers

Command

ID

Count

CommandInteger

Binary Representation   [Count][Id]

MoveTo

1

1

9

[00000000 00000000 00000000 00001][001]

MoveTo

1

120

961

[00000000 00000000 00000011 11000][001]

LineTo

2

1

10

[00000000 00000000 00000000 00001][010]

LineTo

2

3

26

[00000000 00000000 00000000 00011][010]

ClosePath

7

1

15

[00000000 00000000 00000000 00001][111]

4.3.2. Parameter Integers

Commands requiring parameters are followed by a ParameterInteger for each parameter required by that command. The number of ParameterIntegers that follow a CommandInteger is equal to the parameter count of a command multiplied by the command count of the CommandInteger. For example, a CommandInteger with a MoveTo command with a command count of 3 will be followed by 6 ParameterIntegers.

A ParameterInteger is zigzag encoded so that small negative and positive values are both encoded as small integers. To encode a parameter value to a ParameterInteger the following formula is used:

ParameterInteger = (value << 1) ^ (value >> 31)

Parameter values greater than pow(2,31) - 1 or less than -1 * (pow(2,31) - 1) are not supported.

The following formula is used to decode a ParameterInteger to a value:

value = ((ParameterInteger >> 1) ^ (-(ParameterInteger & 1)))

4.3.3. Command Types

For all descriptions of commands the initial position of the cursor shall be described to be at the coordinates (cX, cY) where cX is the position of the cursor on the X axis and cY is the position of the cursor on the Y axis.

4.3.3.1. MoveTo Command

A MoveTo command with a command count of n MUST be immediately followed by n pairs of ParameterIntegers. Each pair (dX, dY):

1. Defines the coordinate (pX, pY), where pX = cX + dX and pY = cY + dY.

- Within POINT geometries, this coordinate defines a new point.

- Within LINESTRING geometries, this coordinate defines the starting vertex of a new line.

- Within POLYGON geometries, this coordinate defines the starting vertex of a new linear ring.

2. Moves the cursor to (pX, pY).

4.3.3.2. LineTo Command

A LineTo command with a command count of n MUST be immediately followed by n pairs of ParameterIntegers. Each pair (dX, dY):

1. Defines a segment beginning at the cursor (cX, cY) and ending at the coordinate (pX, pY), where pX = cX + dX and pY = cY + dY.

- Within LINESTRING geometries, this segment extends the current line.

- Within POLYGON geometries, this segment extends the current linear ring.

2. Moves the cursor to (pX, pY).

For any pair of (dX, dY) the dX and dY MUST NOT both be 0.

4.3.3.3. ClosePath Command

A ClosePath command MUST have a command count of 1 and no parameters. The command closes the current linear ring of a POLYGON geometry via a line segment beginning at the cursor (cX, cY) and ending at the starting vertex of the current linear ring.

This command does not change the cursor position.

4.3.4. Geometry Types

The geometry field is described in each feature by the type field which must be a value in the enum GeomType. The following geometry types are supported:

- UNKNOWN

- POINT

- LINESTRING

- POLYGON

Geometry collections are not supported.

4.3.4.1. Unknown Geometry Type

The specification purposefully leaves an unknown geometry type as an option. This geometry type encodes experimental geometry types that an encoder MAY choose to implement. Decoders MAY ignore any features of this geometry type.

4.3.4.2. Point Geometry Type

The POINT geometry type encodes a point or multipoint geometry. The geometry command sequence for a point geometry MUST consist of a single MoveTo command with a command count greater than 0.

If the MoveTo command for a POINT geometry has a command count of 1, then the geometry MUST be interpreted as a single point; otherwise the geometry MUST be interpreted as a multipoint geometry, wherein each pair of ParameterIntegers encodes a single point.

4.3.4.3. Linestring Geometry Type

The LINESTRING geometry type encodes a linestring or multilinestring geometry. The geometry command sequence for a linestring geometry MUST consist of one or more repetitions of the following sequence:

1. A MoveTo command with a command count of 1

2. A LineTo command with a command count greater than 0

If the command sequence for a LINESTRING geometry type includes only a single MoveTo command then the geometry MUST be interpreted as a single linestring; otherwise the geometry MUST be interpreted as a multilinestring geometry, wherein each MoveTo signals the beginning of a new linestring.

4.3.4.4. Polygon Geometry Type

The POLYGON geometry type encodes a polygon or multipolygon geometry, each polygon consisting of exactly one exterior ring that contains zero or more interior rings. The geometry command sequence for a polygon consists of one or more repetitions of the following sequence:

1. An ExteriorRing

2. Zero or more InteriorRings

Each ExteriorRing and InteriorRing MUST consist of the following sequence:

1. A MoveTo command with a command count of 1

2. A LineTo command with a command count greater than 1

3. A ClosePath command

An exterior ring is DEFINED as a linear ring having a positive area as calculated by applying the surveyor's formula to the vertices of the polygon in tile coordinates. In the tile coordinate system (with the Y axis positive down and X axis positive to the right) this makes the exterior ring's winding order appear clockwise.

An interior ring is DEFINED as a linear ring having a negative area as calculated by applying the surveyor's formula to the vertices of the polygon in tile coordinates. In the tile coordinate system (with the Y axis positive down and X axis positive to the right) this makes the interior ring's winding order appear counterclockwise.

If the command sequence for a POLYGON geometry type includes only a single exterior ring then the geometry MUST be interpreted as a single polygon; otherwise the geometry MUST be interpreted as a multipolygon geometry, wherein each exterior ring signals the beginning of a new polygon. If a polygon has interior rings they MUST be encoded directly after the exterior ring of the polygon to which they belong.

Linear rings MUST be geometric objects that have no anomalous geometric points, such as self-intersection or self-tangency. The position of the cursor before calling the ClosePath command of a linear ring SHALL NOT repeat the same position as the first point in the linear ring as this would create a zero-length line segment. A linear ring SHOULD NOT have an area calculated by the surveyor's formula equal to zero, as this would signify a ring with anomalous geometric points.

Polygon geometries MUST NOT have any interior rings that intersect and interior rings MUST be enclosed by the exterior ring.

4.3.5. Example Geometry Encodings

4.3.5.1. Example Point

An example encoding of a point located at:

- (25,17)

This would require a single command:

- MoveTo(+25, +17)

Encoded as: [ 9 50 34 ]

| |  `> Decoded: ((34 >> 1) ^ (-(34 & 1))) = +17

| `> Decoded: ((50 >> 1) ^ (-(50 & 1))) = +25

| ===== relative MoveTo(+25, +17) == create point (25,17)

`> [00001 001] = command id 1 (MoveTo), command count 1

4.3.5.2. Example Multi Point

An example encoding of two points located at:

- (5,7)

- (3,2)

This would require two commands:

- MoveTo(+5,+7)

- MoveTo(-2,-5)

Encoded as: [ 17 10 14 3 9 ]

|  |  | | `> Decoded: ((9 >> 1) ^ (-(9 & 1))) = -5

|  |  | `> Decoded: ((3 >> 1) ^ (-(3 & 1))) = -2

|  |  | === relative MoveTo(-2, -5) == create point (3,2)

|  |  `> Decoded: ((34 >> 1) ^ (-(34 & 1))) = +7

|  `> Decoded: ((50 >> 1) ^ (-(50 & 1))) = +5

| ===== relative MoveTo(+5, +7) == create point (5,7)

`> [00010 001] = command id 1 (MoveTo), command count 2

4.3.5.3. Example Linestring

An example encoding of a line with the points:

- (2,2)

- (2,10)

- (10,10)

This would require three commands:

- MoveTo(+2,+2)

- LineTo(+0,+8)

- LineTo(+8,+0)

Encoded as: [ 9 4 4 18 0 16 16 0 ]

|      |      ==== relative LineTo(+8, +0) == Line to Point (10, 10)

|      | ==== relative LineTo(+0, +8) == Line to Point (2, 10)

|      `> [00010 010] = command id 2 (LineTo), command count 2

| === relative MoveTo(+2, +2)

`> [00001 001] = command id 1 (MoveTo), command count 1

4.3.5.4. Example Multi Linestring

An example encoding of two lines with the points:

- Line 1:

- (2,2)

- (2,10)

- (10,10)

- Line 2:

- (1,1)

- (3,5)

This would require the following commands:

- MoveTo(+2,+2)

- LineTo(+0,+8)

- LineTo(+8,+0)

- MoveTo(-9,-9)

- LineTo(+2,+4)

Encoded as: [ 9 4 4 18 0 16 16 0 9 17 17 10 4 8 ]

|      |           |        | === relative LineTo(+2, +4) == Line to Point (3,5)

|      |           |        `> [00001 010] = command id 2 (LineTo), command count 1

|      |           | ===== relative MoveTo(-9, -9) == Start new line at (1,1)

|      |           `> [00001 001] = command id 1 (MoveTo), command count 1

|      |      ==== relative LineTo(+8, +0) == Line to Point (10, 10)

|      | ==== relative LineTo(+0, +8) == Line to Point (2, 10)

|      `> [00010 010] = command id 2 (LineTo), command count 2

| === relative MoveTo(+2, +2)

`> [00001 001] = command id 1 (MoveTo), command count 1

4.3.5.5. Example Polygon

An example encoding of a polygon feature that has the points:

- (3,6)

- (8,12)

- (20,34)

- (3,6) Path Closing as Last Point

Would encoded by using the following commands:

- MoveTo(3, 6)

- LineTo(5, 6)

- LineTo(12, 22)

- ClosePath

Encoded as: [ 9 6 12 18 10 12 24 44 15 ]

|       |              `> [00001 111] command id 7 (ClosePath), command count 1

|       |       ===== relative LineTo(+12, +22) == Line to Point (20, 34)

|       | ===== relative LineTo(+5, +6) == Line to Point (8, 12)

|       `> [00010 010] = command id 2 (LineTo), command count 2

| ==== relative MoveTo(+3, +6)

`> [00001 001] = command id 1 (MoveTo), command count 1

4.3.5.6. Example Multi Polygon

An example of a more complex encoding of two polygons, one with a hole. The position of the points for the polygons are shown below. The winding order of the polygons is VERY important in this example as it signifies the difference between interior rings and a new polygon.

- Polygon 1:

- Exterior Ring:

- (0,0)

- (10,0)

- (10,10)

- (0,10)

- (0,0) Path Closing as Last Point

- Polygon 2:

- Exterior Ring:

- (11,11)

- (20,11)

- (20,20)

- (11,20)

- (11,11) Path Closing as Last Point

- Interior Ring:

- (13,13)

- (13,17)

- (17,17)

- (17,13)

- (13,13) Path Closing as Last Point

This polygon would be encoded with the following set of commands:

- MoveTo(+0,+0)

- LineTo(+10,+0)

- LineTo(+0,+10)

- LineTo(-10,+0) // Cursor at 0,10 after this command

- ClosePath // End of Polygon 1

- MoveTo(+11,+1) // NOTE THAT THIS IS RELATIVE TO LAST LINETO!

- LineTo(+9,+0)

- LineTo(+0,+9)

- LineTo(-9,+0) // Cursor at 11,20 after this command

- ClosePath // This is a new polygon because area is positive!

- MoveTo(+2,-7) // NOTE THAT THIS IS RELATIVE TO LAST LINETO!

- LineTo(+0,+4)

- LineTo(+4,+0)

- LineTo(+0,-4) // Cursor at 17,13

- ClosePath // This is an interior ring because area is negative!

Encoded as: [ 9 0 0 26 20 0 0 20 19 0 15 9 22 2 26 18 0 0 18 17 0 15 9 4 13 26 0 8 8 0 0 7 15 ]

|      |                |  |       |                |  |       |              `> [00001 111] (ClosePath)

|      |                |  |       |                |  |       `> [00011 010] = (LineTo), command count 3

|      |                |  |       |                |  `> [00001 001] = command id 1 (MoveTo), command count 1

|      |                |  |       |                `> [00001 111] (ClosePath)

|      |                |  |       `> [00011 010] = (LineTo), command count 3

|      |                | `> [00001 001] = command id 1 (MoveTo), command count 1

|      |                `> [00001 111] (ClosePath)

|      `> [00011 010] = (LineTo), command count 3

`> [00001 001] = command id 1 (MoveTo), command count 1

4.4. Feature Attributes

Feature attributes are encoded as pairs of integers in the tag field of a feature. The first integer in each pair represents the zero-based index of the key in the keys set of the layer to which the feature belongs. The second integer in each pair represents the zero-based index of the value in the values set of the layer to which the feature belongs. Every key index MUST be unique within that feature such that no other attribute pair within that feature has the same key index. A feature MUST have an even number of tag fields. A feature tag field MUST NOT contain a key index or value index greater than or equal to the number of elements in the layer's keys or values set, respectively.

4.5. Example

For example, a GeoJSON feature like:

{

"type": "FeatureCollection",

"features": [

{

"geometry": {

"type": "Point",

"coordinates": [

-8247861.1000836585,

4970241.327215323

]

},

"type": "Feature",

"properties": {

"hello": "world",

"h": "world",

"count": 1.23

}

},

{

"geometry": {

"type": "Point",

"coordinates": [

-8247861.1000836585,

4970241.327215323

]

},

"type": "Feature",

"properties": {

"hello": "again",

"count": 2

}

}

]

}

Could be structured like:

layers {

version: 2

name: "points"

features: {

id: 1

tags: 0

tags: 0

tags: 1

tags: 0

tags: 2

tags: 1

type: Point

geometry: 9

geometry: 2410

geometry: 3080

}

features {

id: 2

tags: 0

tags: 2

tags: 2

tags: 3

type: Point

geometry: 9

geometry: 2410

geometry: 3080

}

keys: "hello"

keys: "h"

keys: "count"

values: {

string_value: "world"

}

values: {

double_value: 1.23

}

values: {

string_value: "again"

}

values: {

int_value: 2

}

extent: 4096

}

Keep in mind the exact values for the geometry would differ based on the projection and extent of the tile.