osrm-backend/third_party/vtzero/doc/writing.md

Writing vector tiles

Writing vector tiles start with creating a tile_builder. This builder will then be used to add layers and features in those layers. Once all this is done, you call serialize() to actually build the vector tile from the data you provided to the builders:

#include <vtzero/builder.hpp> // always needed when writing vector tiles

vtzero::tile_builder tbuilder;
// add lots of data to builder...
std::string buffer = tbuilder.serialize();

You can also serialize the data into an existing buffer instead:

std::string buffer; // got buffer from somewhere
tbuilder.serialize(buffer);

Adding layers to tiles

Once you have a tile builder, you'll first need some layers:

vtzero::tile_builder tbuilder;
vtzero::layer_builder layer_pois{tbuilder, "pois", 2, 4096};
vtzero::layer_builder layer_roads{tbuilder, "roads"};
vtzero::layer_builder layer_forests{tbuilder, "forests"};

Here three layers called "pois", "roads", and "forests" are added. The first one explicitly specifies the vector tile version used and the extent. The values specified here are the default, so all layers in this example will have a version of 2 and an extent of 4096.

If you have read a layer from an existing vector tile and want to copy over some of the data, you can use this layer to initialize the new layer in the new vector tile with the name, version and extent from the existing layer like this:

vtzero::layer some_layer = ...;
vtzero::layer_builder layer_pois{tbuilder, some_layer};
// same as...
vtzero::layer_builder layer_pois{tbuilder, some_layer.name(),
                                           some_layer.version(),
                                           some_layer.extent()};

If you want to copy over an existing layer completely, you can use the add_existing_layer() function instead:

vtzero::layer some_layer = ...;
vtzero::tile_builder tbuilder;
tbuilder.add_existing_layer(some_layer);

Or, if you have the encoded layer data available in a data_view this also works:

vtzero::data_view layer_data = ...;
vtzero::tile_builder tbuilder;
tbuilder.add_existing_layer(layer_data);

Note that this call will only store a reference to the data to be added in the tile builder. The data will only be copied when the final serialize() is called, so the input data must still be available then!

You can mix any of the ways of adding a layer to the tile mentioned above. The layers will be added to the tile in the order you add them to the tile_builder.

The tile builder is smart enough to not add empty layers, so you can start out with all the layers you might need and if some of them stay empty, they will not be added to the tile when serialize() is called.

Adding features to layers

Once we have one or more layer_builders instantiated, we can add features to them. This is done through the following feature builder classes:

• point_feature_builder to add a feature with a (multi)point geometry,
• linestring_feature_builder to add a feature with a (multi)linestring geometry,
• polygon_feature_builder to add a feature with a (multi)polygon geometry, or
• geometry_feature_builder to add a feature with an existing geometry you got from reading a vector tile.

In all cases you need to instantiate the feature builder class, optionally add the feature ID using the set_id() method, add the geometry and then add all the properties of this feature. You have to keep to this order!

...
vtzero::layer_builder lbuilder{...};
{
    vtzero::point_feature_builder fbuilder{lbuilder};
    // optionally set the ID
    fbuilder.set_id(23);
    // add the geometry (exact calls are different for different feature builders)
    fbuilder.add_point(99, 33);
    // add the properties
    fbuilder.add_property("amenity", "restaurant");
    // call commit() when you are done
    fbuilder.commit()
}

You have to call commit() on the feature builder object after you set all the data to actually add it to the layer. If you don't do this, the feature will not be added to the layer! This can be useful, for instance, if you detect that you have an invalid geometry while you are adding the geometry to the feature builder. In that case you can call rollback() explicitly or just let the feature builder go out of scope and it will do the rollback automatically.

Only the first call to commit() or rollback() will take effect, any further calls to these functions on the same feature builder object are ignored.

Adding a geometry to the feature

There are different ways of adding the geometry to the feature, depending on the geometry type.

Adding a point geometry

Simply call add_point() to set the point geometry. There are three different overloads for this function. One takes a vtzero::point, one takes two uint32_ts with the x and y coordinates and one takes any type T that can be converted to a vtzero::point using the create_vtzero_point function. This templated function works on any type that has x and y members and you can create your own overload of this function. See the [advanced.md](advanced topics documentation).

Adding a multipoint geometry

Call add_points() with the number of points in the geometry as the only argument. After that call set_point() for each of those points. set_point() has multiple overloads just like the add_point() method described above.

There is also the add_points_from_container() function which copies the point from any container type supporting the size() function and which iterator yields a vtzero::point or something convertible to it.

Adding a linestring geometry

Call add_linestring() with the number of points in the linestring as only argument. After that call set_point() for each of those points. set_point() has multiple overloads just like the add_point() method described above.

...
vtzero::layer_builder lbuilder{...};
try {
    vtzero::linestring_feature_builder fbuilder{lbuilder};
    // optionally set the ID
    fbuilder.set_id(23);
    // add the geometry
    fbuilder.add_linestring(2);
    fbuilder.set_point(1, 2);
    fbuilder.set_point(3, 4);
    // add the properties
    fbuilder.add_property("highway", "primary");
    fbuilder.add_property("maxspeed", 80);
    // call commit() when you are done
    fbuilder.commit()
} catch (const vtzero::geometry_exception& e) {
    // if we are here, something was wrong with the geometry.
}

Note that we have wrapped the feature builder in a try-catch-block here. This will ignore all geometry errors (which can happen if two consective points are the same creating a zero-length segment).

There are two other versions of the add_linestring() function. They take two iterators defining a range to get the points from. Dereferencing those iterators must yield a vtzero::point or something convertible to it. One of these functions takes a third argument, the number of points the iterator will yield. If this is not available std::distance(begin, end) is called which internally by the add_linestring() function which might be slow depending on your iterator type.

Adding a multilinestring geometry

Adding a multilinestring works just like adding a linestring, just do the calls to add_linestring() etc. repeatedly for each of the linestrings.

Adding a polygon geometry

A polygon consists of one outer ring and zero or more inner rings. You have to first add the outer ring and then the inner rings, if any.

Call add_ring() with the number of points in the ring as only argument. After that call set_point() for each of those points. set_point() has multiple overloads just like the add_point() method described above. The minimum number of points is 4 and the last point must be the same as the first point (or call close_ring() instead of the last set_point()).

...
vtzero::layer_builder lbuilder{...};
try {
    vtzero::polygon_feature_builder fbuilder{lbuilder};
    // optionally set the ID
    fbuilder.set_id(23);
    // add the geometry
    fbuilder.add_ring(5);
    fbuilder.set_point(1, 1);
    fbuilder.set_point(1, 2);
    fbuilder.set_point(2, 2);
    fbuilder.set_point(2, 1);
    fbuilder.set_point(1, 1); // or call fbuilder.close_ring() instead
    // add the properties
    fbuilder.add_property("landuse", "forest");
    // call commit() when you are done
    fbuilder.commit()
} catch (const vtzero::geometry_exception& e) {
    // if we are here, something was wrong with the geometry.
}

Note that we have wrapped the feature builder in a try-catch-block here. This will ignore all geometry errors (which can happen if two consective points are the same creating a zero-length segment or if the last point is not the same as the first point).

There are two other versions of the add_ring() function. They take two iterators defining a range to get the points from. Dereferencing those iterators must yield a vtzero::point or something convertible to it. One of these functions takes a third argument, the number of points the iterator will yield. If this is not available std::distance(begin, end) is called which internally by the add_ring() function which might be slow depending on your iterator type.

Adding a multipolygon geometry

Adding a multipolygon works just like adding a polygon, just do the calls to add_ring() etc. repeatedly for each of the rings. Make sure to always first add an outer ring, then the inner rings in this outer ring, then the next outer ring and so on.

Adding an existing geometry

The geometry_feature_builder class is used to add geometries you got from reading a vector tile. This is useful when you want to copy over a geometry from a feature without decoding it.

auto geom = ... // get geometry from a feature you are reading
...
vtzero::tile_builder tb;
vtzero::layer_builder lb{tb};
vtzero::geometry_feature_builder fb{lb};
fb.set_id(123); // optionally set ID
fb.add_geometry(geom) // add geometry
fb.add_property("foo", "bar"); // add properties
fb.commit();
...

Adding properties to the feature

A feature can have any number of properties. They are added with the add_property() method called on the feature builder. There are two different ways of doing this. The simple approach which does all the work for you and the advanced approach which can be more efficient, but you have to to some more work. It is recommended that you start out with the simple approach and only switch to the advanced approach once you have a working program and want to get the last bit of performance out of it.

The difference stems from the way properties are encoded in vector tiles. While properties "belong" to features, they are really stored in two tables (for the keys and values) in the layer. The individual feature only references the entries in those tables by index. This make the encoded tile smaller, but it means somebody has to manage those tables. In the simple approach this is done behind the scenes by the layer_builder object, in the advanced approach you handle that yourself.

Do not mix the simple and the advanced approach unless you know what you are doing.

The simple approach to adding properties

For the simple approach call add_property() with two arguments. The first is the key, it must have some kind of string type (std::string, const char*, vtzero::data_view, anything really that converts to a data_view). The second argument is the value, for which most basic C++ types are allowed (string types, integer types, double, ...). See the API documentation for the constructors of the encoded_property_value class for a list.

vtzero::layer_builder lb{...};
vtzero::linestring_feature_builder fb{lb};
...
fb.add_property("waterway", "stream"); // string value
fb.add_property("name", "Little Creek");
fb.add_property("width", 1.5); // double value
...

Sometimes you need to specify exactly which type should be used in the encoding. The encoded_property_value constructor can take special types for that like in the following example, where you force the sint encoding:

fb.add_property("layer", vtzero::sint_value_type(2));

You can also call add_property() with a single vtzero::property argument (which is handy if you are copying this property over from a tile you are reading):

while (auto property = feature.next_property()) {
    if (property.key() == "name") {
        feature_builder.add_property(property);
    }
}

The advanced approach to adding properties

In the advanced approach you have to do the indexing yourself. Here is a very basic example:

vtzero::tile_builder tbuilder;
vtzero::layer_builder lbuilder{tbuilder, "test"};
const vtzero::index_value highway = lbuilder.add_key("highway");
const vtzero::index_value primary = lbuilder.add_value("primary");
...
vtzero::point_feature_builder fbuilder{lbuilder};
...
fbuilder.add_property(highway, primary);
...

The methods add_key() and add_value() on the layer builder are used to add keys and values to the tables in the layer. They both return the index (of type vtzero::index_value) of those keys or values in the tables. You store those index values somewhere (in this case in the highway and primary variables) and use them when calling add_property() on the feature builder.

In some cases you only have a few property keys and know them beforehand, then storing the key indexes in individual variables might work. But for values this usually doesn't make much sense, and if all your keys and values are only known at runtime, it doesn't work either. For this you need some kind of index data structure mapping from keys/values to index values. You can implement this yourself, but it is easier to use some classes provided by vtzero. Then the code looks like this:

#include <vtzero/index.hpp> // use this include to get the index classes
...
vtzero::layer_builder lb{...};
vtzero::key_index<std::map> key_index{lb};
vtzero::value_index_internal<std::unordered_map> value_index{lb};
...
vtzero::point_feature_builder fb{lb};
...
fb.add_property(key_index("highway"), value_index("primary"));
...

In this example the key_index template class is used for keys, it uses std::map internally as can be seen by its template argument. The value_index_internal template class is used for values, it uses std::unordered_map internally in this example. Whether you specify std::map or std::unordered_map or something else (that needs to be compatible to those classes) is up to you. Benchmark your use case and decide then.

Keys are always strings, so they are easy to handle. For keys there is only the single key_index in vtzero.

For values this is more difficult. Basically there are two choices:

1. Encode the value according to the vector tile encoding rules which results in a string and store this in the index. This is what the value_index_internal class does.
2. Store the un-encoded value in the index. The index lookup will be faster, but you need a different index type for each value type. This is what the value_index classes do.

The value_index template classes need three template arguments: The type used internally to encode the value, the type used externally, and the map type.

In this example the user program has the values as int, the index will store them in a std::map<int>. The integer value is then encoded in an sint int the vector tile:

vtzero::value_index<vtzero::sint_value:type, int, std::map> index;

Sometimes these generic indexes based on std::map or std::unordered_map are inefficient, that's why there are specialized indexes for special cases:

• The value_index_bool class can only index boolean values.
• The value_index_small_uint class can only index small unsigned integer values (up to uint16_t). It uses a vector internally, so if all your numbers are small and densely packed, this is very efficient. This is especially useful for enum types.

The add_property() function.

The last chapters already talked about the add_property() function of the feature_builder class. But because it is a bit difficult to see all the different ways add_property() can be called, here is some more information.

The add_property() function is called with either two parameters for the key and value or with one parameter that combines the key and value.

If it is called with an index_value for the key or value, that index value is stored directly into the feature. If it is called with an index_value_pair, the index values in the index_value_pair are stored directly in the feature.

If it is called with something that is not an index_value or index_value_pair, the function will interpret the data as keys or values. It will add those keys and values to the layer (if they are not already there), find the corresponding index values and store them in the feature.

You can mix index-use with non-index use. For instance

index_value key_maxspeed = lbuilder.add_key("maxspeed");
...
fbuilder.add_property(key_maxspeed, 30);

In this case the key ("maxspeed") was added to the layer once and its index value (key_maxspeed) can later be reused. The value (30), on the other hand, is only added to the layer in the add_property() call.

So for keys, you can have as argument:

• An index_value.
• A data_view or something that converts to it like a const char* or std::string.

For values, you can have as argument:

• An index_value.
• A property_value.
• An encoded_property_value or anything that converts to it.

For combined keys and values, you can have as argument:

• An index_value_pair.
• A property.

Deriving from layer_builder and feature_builder

The vtzero::layer_builder and vtzero::feature_builder classes have been designed in a way that they can be derived from easily. This allows you to encapsulate part of your vector tile writing code if some aspects of your layers/features are always the same, such as the layer name and the names and types of properties.

Say you want to write a layer named "restaurants" with point geometries. Each feature should have a name and a 5-star-rating. First you create a class derived from the layer_builder with all the indexes you want to use. For the keys you don't need indexes in this case, because there are only two keys for which we can easily store the index values in the layer.

class restaurant_layer_builder : public vtzero::layer_builder {

public:

    // The index we'll use for the "name" property values
    vtzero::value_index<vtzero::string_value_type, std::string, std::unordered_map> string_index;

    // The index we'll use for the "stars" property values
    vtzero::value_index_small_uint stars_index;

    // The index value of the "name" key
    vtzero::index_value key_name;

    // The index value of the "stars" key
    vtzero::index_value key_stars;

    restaurant_layer_builder(vtzero::tile_builder& tile) :
        layer_builder(tile, "restaurants"), // the name of the layer
        string_index(*this),
        stars_index(*this),
        key_name(add_key_without_dup_check("name")),
        key_stars(add_key_without_dup_check("stars")) {
    }

};

The we'll add a class derived from feature_builder to help with adding features:

class restaurant_feature_builder : public vtzero::feature_builder {

    restaurant_layer_builder& m_layer;

public:

    restaurant_feature_builder(restaurant_layer_builder& layer, uint64_t id) :
        vtzero::point_feature_builder(layer), // always a point geometry
        m_layer(layer) {
        set_id(id); // we always have an ID in this case
    }

    void add_location(mylocation& loc) { // restaurant location is stored in your own type
        add_point(loc.lon(), loc.lat());
    }

    void set_name(const std::string& name) {
        add_property(m_layer.key_name,
                     m_layer.string_index(vtzero::encoded_property_value{name}));
    }

    void set_stars(stars s) { // your own "stars" type
        vtzero::encoded_property_value svalue{ s.num_stars() }; // convert stars type to small integer

        add_property(m_layer.key_stars,
                     m_layer.stars_index(svalue));
    }

};

This example only shows a general pattern you can follow to derive from the layer_builder and feature_builder classes. In some cases this makes more sense then in others. The derived classes make it easy for you to mix your own functions (for instance when you need to convert from your own types to vtzero types like with the mylocation and stars types above) or just use the functions in the base classes.

Using a custom buffer type

Usually std::string is used as a buffer type:

std::string buffer;
tbuilder.serialize(buffer);

But you can also use other buffer types supported by Protozero, like std::vector<char>:

#include <protozero/buffer_vector.hpp>

std::vector<char> buffer;
tbuilder.serialize(buffer);

Or you can use your own buffer types and write special adaptors for it. See the Protozero documentation for details.

Note that while in theory this allows you to also use fixed-sized buffers through the protozero::fixed_sized_buffer_adaptor class, vtzero will still use std::string for additional buffers internally.