Open-Harbor/osrm-backend
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 3 Wiki Activity

Add support for disabling feature datasets (#6666)

Browse Source 
This change adds support for disabling datasets, such that specific
files are not loaded into memory when running OSRM. This enables users
to not pay the memory cost for features they do not intend to use.

Initially, there are two options:
- ROUTE_GEOMETRY, for disabling overview, steps, annotations and waypoints.
- ROUTE_STEPS, for disabling steps only.

Attempts to query features for which the datasets are disabled will
lead to a DisabledDatasetException being returned.
This commit is contained in:
Michael Bell 2023-08-04 18:43:37 +01:00 committed by GitHub
parent 522d0f066e
commit db7946d762
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
34 changed files with 1005 additions and 200 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
CHANGELOG.md
View File

@ -2,6 +2,7 @@
- Changes from 5.27.1
- Features
- ADDED: Add support for a default_radius flag. [#6575](https://github.com/Project-OSRM/osrm-backend/pull/6575)
- ADDED: Add support for disabling feature datasets. [#6666](https://github.com/Project-OSRM/osrm-backend/pull/6666)
- Build:
- ADDED: Add CI job which builds OSRM with gcc 12. [#6455](https://github.com/Project-OSRM/osrm-backend/pull/6455)
- CHANGED: Upgrade to clang-tidy 15. [#6439](https://github.com/Project-OSRM/osrm-backend/pull/6439)

31
docs/http.md
View File

@ -88,10 +88,11 @@ Every response object has a `code` property containing one of the strings below
| `InvalidService` | Service name is invalid. |
| `InvalidVersion` | Version is not found. |
| `InvalidOptions` | Options are invalid. |
| `InvalidQuery` | The query string is syntactically malformed. |
| `InvalidQuery` | The query string is syntactically malformed. |
| `InvalidValue` | The successfully parsed query parameters are invalid. |
| `NoSegment` | One of the supplied input coordinates could not snap to the street segment. |
| `NoSegment` | One of the supplied input coordinates could not snap to the street segment. |
| `TooBig` | The request size violates one of the service-specific request size restrictions. |
| `DisabledDataset` | The request tried to access a disabled dataset. |
- `message` is a **optional** human-readable error message. All other status types are service-dependent.
- In case of an error the HTTP status code will be `400`. Otherwise, the HTTP status code will be `200` and `code` will be `Ok`.
@ -130,7 +131,7 @@ In addition to the [general options](#general-options) the following options are
|number |`integer >= 1` (default `1`) |Number of nearest segments that should be returned. |
As `waypoints` is a single thing, returned by that service, using it with the option `skip_waypoints` set to `true` is quite useless, but still
possible. In that case, only the `code` field will be returned.
possible. In that case, only the `code` field will be returned.
**Response**
@ -953,11 +954,11 @@ The object is used to describe the waypoint on a route.
## Flatbuffers format
The default response format is `json`, but OSRM supports binary [`flatbuffers`](https://google.github.io/flatbuffers/) format, which
The default response format is `json`, but OSRM supports binary [`flatbuffers`](https://google.github.io/flatbuffers/) format, which
is much faster in serialization/deserialization, comparing to `json`.
The format itself is described in message descriptors, located at `include/engine/api/flatbuffers` directory. Those descriptors could
be compiled to provide protocol parsers in Go/Javascript/Typescript/Java/Dart/C#/Python/Lobster/Lua/Rust/PHP/Kotlin. Precompiled
be compiled to provide protocol parsers in Go/Javascript/Typescript/Java/Dart/C#/Python/Lobster/Lua/Rust/PHP/Kotlin. Precompiled
protocol parser for C++ is supplied with OSRM.
`Flatbuffers` format provides exactly the same data, as `json` format with a slightly different layout, which was optimized to minimize
@ -971,7 +972,7 @@ Root object is the only object, available from a 'raw' `flatbuffers` buffer. It
**Properties**
- `error`: `bool` Marks response as erroneous. An erroneous response should include the `code` fieldset, all the other fields may not be present.
- `error`: `bool` Marks response as erroneous. An erroneous response should include the `code` fieldset, all the other fields may not be present.
- `code`: `Error` Error description object, only present, when `error` is `true`
- `waypoints`: `[Waypoint]` Array of `Waypoint` objects. Should present for every service call, unless `skip_waypoints` is set to `true`. Table service will put `sources` array here.
- `routes`: `[RouteObject]` Array of `RouteObject` objects. May be empty or absent. Should present for Route/Trip/Match services call.
@ -983,21 +984,21 @@ Contains error information.
**Properties**
- `code`: `string` Error code
- `code`: `string` Error code
- `message`: `string` Detailed error message
### Waypoint object
Almost the same as `json` Waypoint object. The following properties differ:
- `location`: `Position` Same as `json` location field, but different format.
- `location`: `Position` Same as `json` location field, but different format.
- `nodes`: `Uint64Pair` Same as `json` nodes field, but different format.
### RouteObject object
Almost the same as `json` Route object. The following properties differ:
- `polyline`: `string` Same as `json` geometry.polyline or geometry.polyline6 fields. One field for both formats.
- `polyline`: `string` Same as `json` geometry.polyline or geometry.polyline6 fields. One field for both formats.
- `coordinates`: `[Position]` Same as `json` geometry.coordinates field, but different format.
- `legs`: `[Leg]` Array of `Leg` objects.
@ -1012,7 +1013,7 @@ Almost the same as `json` Leg object. The following properties differ:
Almost the same as `json` Step object. The following properties differ:
- `polyline`: `string` Same as `json` geometry.polyline or geometry.polyline6 fields. One field for both formats.
- `polyline`: `string` Same as `json` geometry.polyline or geometry.polyline6 fields. One field for both formats.
- `coordinates`: `[Position]` Same as `json` geometry.coordinates field, but different format.
- `maneuver`: `StepManeuver` Same as `json` maneuver field, but different format.
@ -1035,7 +1036,7 @@ Almost the same as `json` Step object. The following properties differ:
| `ExitRoundabout` | Describes a maneuver exiting a roundabout (usually preceded by a `roundabout` instruction) |
| `ExitRotary` | Describes the maneuver exiting a rotary (large named roundabout) |
- `driving_side`: `bool` Ttrue stands for the left side driving.
- `driving_side`: `bool` Ttrue stands for the left side driving.
- `intersections`: `[Intersection]` Same as `json` intersections field, but different format.
### Intersection object
@ -1049,7 +1050,7 @@ Almost the same as `json` Intersection object. The following properties differ:
Almost the same as `json` Lane object. The following properties differ:
- `indications`: `Turn` Array of `Turn` enum values.
- `indications`: `Turn` Array of `Turn` enum values.
| `value` | Description |
|------------------------|---------------------------------------------------------------------------------------------------------------------------|
@ -1089,16 +1090,16 @@ Almost the same as `json` StepManeuver object. The following properties differ:
| `ExitRoundabout` | Describes a maneuver exiting a roundabout (usually preceded by a `roundabout` instruction) |
| `ExitRotary` | Describes the maneuver exiting a rotary (large named roundabout) |
- `modifier`: `Turn` Maneuver turn (enum)
- `modifier`: `Turn` Maneuver turn (enum)
### Annotation object
Exactly the same as `json` annotation object.
Exactly the same as `json` annotation object.
### Position object
A point on Earth.
A point on Earth.
***Properties***
- `longitute`: `float` Point's longitude

1
docs/nodejs/api.md
View File

@ -31,6 +31,7 @@ var osrm = new OSRM('network.osrm');
Old behaviour: Path to a file on disk to store the memory using mmap. Current behaviour: setting this value is the same as setting `mmap_memory: true`.
- `options.mmap_memory` **[Boolean](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Boolean)?** Map on-disk files to virtual memory addresses (mmap), rather than loading into RAM.
- `options.path` **[String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)?** The path to the `.osrm` files. This is mutually exclusive with setting {options.shared_memory} to true.
- `options.disable_feature_dataset` **[Array](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Array)?** Disables a feature dataset from being loaded into memory if not needed. Options: `ROUTE_STEPS`, `ROUTE_GEOMETRY`.
- `options.max_locations_trip` **[Number](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Number)?** Max. locations supported in trip query (default: unlimited).
- `options.max_locations_viaroute` **[Number](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Number)?** Max. locations supported in viaroute query (default: unlimited).
- `options.max_locations_distance_table` **[Number](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Number)?** Max. locations supported in distance table query (default: unlimited).

21
features/lib/osrm_loader.js
View File

@ -68,8 +68,9 @@ class OSRMDirectLoader extends OSRMBaseLoader {
super(scope);
}
load (inputFile, callback) {
this.inputFile = inputFile;
load (ctx, callback) {
this.inputFile = ctx.inputFile;
this.loaderArgs = ctx.loaderArgs;
this.shutdown(() => {
this.launch(callback);
});
@ -78,7 +79,7 @@ class OSRMDirectLoader extends OSRMBaseLoader {
osrmUp (callback) {
if (this.osrmIsRunning()) return callback(new Error("osrm-routed already running!"));
const command_arguments = util.format('%s -p %d -i %s -a %s', this.inputFile, this.scope.OSRM_PORT, this.scope.OSRM_IP, this.scope.ROUTING_ALGORITHM);
const command_arguments = util.format('%s -p %d -i %s -a %s %s', this.inputFile, this.scope.OSRM_PORT, this.scope.OSRM_IP, this.scope.ROUTING_ALGORITHM, this.loaderArgs);
this.child = this.scope.runBin('osrm-routed', command_arguments, this.scope.environment, (err) => {
if (err && err.signal !== 'SIGINT') {
this.child = null;
@ -101,8 +102,9 @@ class OSRMmmapLoader extends OSRMBaseLoader {
super(scope);
}
load (inputFile, callback) {
this.inputFile = inputFile;
load (ctx, callback) {
this.inputFile = ctx.inputFile;
this.loaderArgs = ctx.loaderArgs;
this.shutdown(() => {
this.launch(callback);
});
@ -111,7 +113,7 @@ class OSRMmmapLoader extends OSRMBaseLoader {
osrmUp (callback) {
if (this.osrmIsRunning()) return callback(new Error("osrm-routed already running!"));
const command_arguments = util.format('%s -p %d -i %s -a %s --mmap', this.inputFile, this.scope.OSRM_PORT, this.scope.OSRM_IP, this.scope.ROUTING_ALGORITHM);
const command_arguments = util.format('%s -p %d -i %s -a %s --mmap %s', this.inputFile, this.scope.OSRM_PORT, this.scope.OSRM_IP, this.scope.ROUTING_ALGORITHM, this.loaderArgs);
this.child = this.scope.runBin('osrm-routed', command_arguments, this.scope.environment, (err) => {
if (err && err.signal !== 'SIGINT') {
this.child = null;
@ -134,8 +136,9 @@ class OSRMDatastoreLoader extends OSRMBaseLoader {
super(scope);
}
load (inputFile, callback) {
this.inputFile = inputFile;
load (ctx, callback) {
this.inputFile = ctx.inputFile;
this.loaderArgs = ctx.loaderArgs;
this.loadData((err) => {
if (err) return callback(err);
@ -148,7 +151,7 @@ class OSRMDatastoreLoader extends OSRMBaseLoader {
}
loadData (callback) {
const command_arguments = util.format('--dataset-name=%s %s', this.scope.DATASET_NAME, this.inputFile);
const command_arguments = util.format('--dataset-name=%s %s %s', this.scope.DATASET_NAME, this.inputFile, this.loaderArgs);
this.scope.runBin('osrm-datastore', command_arguments, this.scope.environment, (err) => {
if (err) return callback(new Error('*** osrm-datastore exited with ' + err.code + ': ' + err));
callback();

141
features/options/data/disabled_dataset.feature Normal file
View File

@ -0,0 +1,141 @@
@routing @disable-feature-dataset
Feature: disable-feature-dataset command line options
Background:
Given the profile "testbot"
And the node map
"""
0
a b c
"""
And the ways
| nodes |
| ab |
| bc |
Scenario: disable-feature-dataset - geometry disabled error
Given the data load extra arguments "--disable-feature-dataset ROUTE_GEOMETRY"
# The default values
And the query options
| overview | simplified |
| annotations | false |
| steps | false |
| skip_waypoints | false |
When I route I should get
| from | to | code |
| a | c | DisabledDataset |
When I plan a trip I should get
| waypoints | code |
| a,b,c | DisabledDataset |
When I match I should get
| trace | code |
| abc | DisabledDataset |
Scenario: disable-feature-dataset - geometry disabled error table
Given the data load extra arguments "--disable-feature-dataset ROUTE_GEOMETRY"
When I request nearest I should get
| in | code |
| 0 | DisabledDataset |
When I request a travel time matrix with these waypoints I should get the response code
| waypoints | code |
| a,b,c | DisabledDataset |
Scenario: disable-feature-dataset - geometry disabled success
Given the data load extra arguments "--disable-feature-dataset ROUTE_GEOMETRY"
# No geometry values returned
And the query options
| overview | false |
| annotations | false |
| steps | false |
| skip_waypoints | true |
When I route I should get
| from | to | code |
| a | c | Ok |
When I plan a trip I should get
| waypoints | code |
| a,b,c | Ok |
When I match I should get
| trace | code |
| abc | Ok |
Scenario: disable-feature-dataset - geometry disabled error table
Given the data load extra arguments "--disable-feature-dataset ROUTE_GEOMETRY"
And the query options
| skip_waypoints | true |
# You would never do this, but just to prove the point.
When I request nearest I should get
| in | code |
| 0 | Ok |
When I request a travel time matrix with these waypoints I should get the response code
| waypoints | code |
| a,b,c | Ok |
Scenario: disable-feature-dataset - steps disabled error
Given the data load extra arguments "--disable-feature-dataset ROUTE_STEPS"
# Default + annotations, steps
And the query options
| overview | simplified |
| annotations | true |
| steps | true |
When I route I should get
| from | to | code |
| a | c | DisabledDataset |
When I plan a trip I should get
| waypoints | code |
| a,b,c | DisabledDataset |
When I match I should get
| trace | code |
| abc | DisabledDataset |
Scenario: disable-feature-dataset - geometry disabled error table
Given the data load extra arguments "--disable-feature-dataset ROUTE_STEPS"
When I request nearest I should get
| in | code |
| 0 | Ok |
When I request a travel time matrix with these waypoints I should get the response code
| waypoints | code |
| a,b,c | Ok |
Scenario: disable-feature-dataset - steps disabled success
Given the data load extra arguments "--disable-feature-dataset ROUTE_STEPS"
# Default + steps
And the query options
| overview | simplified |
| annotations | true |
| steps | false |
When I route I should get
| from | to | code |
| a | c | Ok |
When I plan a trip I should get
| waypoints | code |
| a,b,c | Ok |
When I match I should get
| trace | code |
| abc | Ok |

5
features/step_definitions/data.js
View File

@ -33,6 +33,11 @@ module.exports = function () {
callback();
});
this.Given(/^the data load extra arguments "(.*?)"$/, (args, callback) => {
this.loaderArgs = this.expandOptions(args);
callback();
});
this.Given(/^a grid size of ([0-9.]+) meters$/, (meters, callback) => {
this.setGridSize(meters);
callback();

63
features/step_definitions/distance_matrix.js
View File

@ -5,6 +5,7 @@ var FBResult = require('../support/fbresult_generated').osrm.engine.api.fbresult
module.exports = function () {
const durationsRegex = new RegExp(/^I request a travel time matrix I should get$/);
const durationsCodeOnlyRegex = new RegExp(/^I request a travel time matrix with these waypoints I should get the response code$/);
const distancesRegex = new RegExp(/^I request a travel distance matrix I should get$/);
const estimatesRegex = new RegExp(/^I request a travel time matrix I should get estimates for$/);
const durationsRegexFb = new RegExp(/^I request a travel time matrix with flatbuffers I should get$/);
@ -17,6 +18,7 @@ module.exports = function () {
const FORMAT_FB = 'flatbuffers';
this.When(durationsRegex, function(table, callback) {tableParse.call(this, table, DURATIONS_NO_ROUTE, 'durations', FORMAT_JSON, callback);}.bind(this));
this.When(durationsCodeOnlyRegex, function(table, callback) {tableCodeOnlyParse.call(this, table, 'durations', FORMAT_JSON, callback);}.bind(this));
this.When(distancesRegex, function(table, callback) {tableParse.call(this, table, DISTANCES_NO_ROUTE, 'distances', FORMAT_JSON, callback);}.bind(this));
this.When(estimatesRegex, function(table, callback) {tableParse.call(this, table, DISTANCES_NO_ROUTE, 'fallback_speed_cells', FORMAT_JSON, callback);}.bind(this));
this.When(durationsRegexFb, function(table, callback) {tableParse.call(this, table, DURATIONS_NO_ROUTE, 'durations', FORMAT_FB, callback);}.bind(this));
@ -27,6 +29,64 @@ const durationsParse = function(v) { return isNaN(parseInt(v)); };
const distancesParse = function(v) { return isNaN(parseFloat(v)); };
const estimatesParse = function(v) { return isNaN(parseFloat(v)); };
function tableCodeOnlyParse(table, annotation, format, callback) {
const params = this.queryParams;
params.annotations = ['durations','fallback_speed_cells'].indexOf(annotation) !== -1 ? 'duration' : 'distance';
params.output = format;
var got;
this.reprocessAndLoadData((e) => {
if (e) return callback(e);
var testRow = (row, ri, cb) => {
var afterRequest = (err, res) => {
if (err) return cb(err);
for (var k in row) {
var match = k.match(/param:(.*)/);
if (match) {
if (row[k] === '(nil)') {
params[match[1]] = null;
} else if (row[k]) {
params[match[1]] = [row[k]];
}
got[k] = row[k];
}
}
var json;
got.code = 'unknown';
if (res.body.length) {
json = JSON.parse(res.body);
got.code = json.code;
}
cb(null, got);
};
var params = this.queryParams,
waypoints = [];
if (row.waypoints) {
row.waypoints.split(',').forEach((n) => {
var node = this.findNodeByName(n);
if (!node) throw new Error(util.format('*** unknown waypoint node "%s"', n.trim()));
waypoints.push({ coord: node, type: 'loc' });
});
got = { waypoints: row.waypoints };
this.requestTable(waypoints, params, afterRequest);
} else {
throw new Error('*** no waypoints');
}
};
this.processRowsAndDiff(table, testRow, callback);
});
}
function tableParse(table, noRoute, annotation, format, callback) {
const parse = annotation == 'distances' ? distancesParse : (annotation == 'durations' ? durationsParse : estimatesParse);
@ -62,9 +122,6 @@ function tableParse(table, noRoute, annotation, format, callback) {
});
}
var actual = [];
actual.push(table.headers);
this.reprocessAndLoadData((e) => {
if (e) return callback(e);
// compute matrix

46
features/step_definitions/nearest.js
View File

@ -12,35 +12,43 @@ module.exports = function () {
var inNode = this.findNodeByName(row.in);
if (!inNode) throw new Error(util.format('*** unknown in-node "%s"', row.in));
var outNode = this.findNodeByName(row.out);
if (!outNode) throw new Error(util.format('*** unknown out-node "%s"', row.out));
this.requestNearest(inNode, this.queryParams, (err, response) => {
if (err) return cb(err);
var coord;
var headers = new Set(table.raw()[0]);
if (response.statusCode === 200 && response.body.length) {
var got = { in: row.in};
if (response.body.length) {
var json = JSON.parse(response.body);
got.code = json.code;
coord = json.waypoints[0].location;
if (response.statusCode === 200) {
var got = { in: row.in, out: row.out };
if (headers.has('data_version')) {
got.data_version = json.data_version || '';
}
Object.keys(row).forEach((key) => {
if (key === 'out') {
if (this.FuzzyMatch.matchLocation(coord, outNode)) {
got[key] = row[key];
} else {
row[key] = util.format('%s [%d,%d]', row[key], outNode.lat, outNode.lon);
}
if (headers.has('data_version')) {
got.data_version = json.data_version || '';
}
});
if (json.waypoints && json.waypoints.length && row.out) {
coord = json.waypoints[0].location;
got.out = row.out;
var outNode = this.findNodeByName(row.out);
if (!outNode) throw new Error(util.format('*** unknown out-node "%s"', row.out));
Object.keys(row).forEach((key) => {
if (key === 'out') {
if (this.FuzzyMatch.matchLocation(coord, outNode)) {
got[key] = row[key];
} else {
row[key] = util.format('%s [%d,%d]', row[key], outNode.lat, outNode.lon);
}
}
});
}
}
cb(null, got);
}
else {

3
features/step_definitions/trip.js
View File

@ -91,7 +91,7 @@ module.exports = function () {
var encodedResult = '';
if (json.trips) row.trips.split(',').forEach((sub, si) => {
if (json.trips && row.trips) row.trips.split(',').forEach((sub, si) => {
if (si >= subTrips.length) {
ok = false;
} else {
@ -134,7 +134,6 @@ module.exports = function () {
} else {
var params = this.queryParams,
waypoints = [];
params['steps'] = 'true';
if (row.from && row.to) {
var fromNode = this.findNodeByName(row.from);
if (!fromNode) throw new Error(util.format('*** unknown from-node "%s"', row.from));

3
features/support/data.js
View File

@ -280,10 +280,11 @@ module.exports = function () {
};
this.reprocessAndLoadData = (callback) => {
let p = {loaderArgs: this.loaderArgs, inputFile: this.processedCacheFile};
let queue = d3.queue(1);
queue.defer(this.writeAndLinkOSM.bind(this));
queue.defer(this.extractContractPartitionAndCustomize.bind(this));
queue.defer(this.osrmLoader.load.bind(this.osrmLoader), this.processedCacheFile);
queue.defer(this.osrmLoader.load.bind(this.osrmLoader), p);
queue.awaitAll(callback);
};

1
features/support/hooks.js
View File

@ -37,6 +37,7 @@ module.exports = function () {
this.contractArgs = '';
this.partitionArgs = '';
this.customizeArgs = '';
this.loaderArgs = '';
this.environment = Object.assign(this.DEFAULT_ENVIRONMENT);
this.resetOSM();

3
features/support/route.js
View File

@ -101,7 +101,8 @@ module.exports = function () {
this.requestTrip = (waypoints, userParams, callback) => {
var defaults = {
output: 'json'
output: 'json',
steps: 'true'
},
params = this.overwriteParams(defaults, userParams);

139
include/engine/datafacade/contiguous_internalmem_datafacade.hpp
View File

@ -31,6 +31,26 @@
namespace osrm::engine::datafacade
{
static const std::string DATASET_TURN_DATA = "TurnData";
static const std::string DATASET_TURN_LANE_DATA = "NameLaneData";
static const std::string DATASET_NAME_DATA = "NameData";
static const std::string DATASET_INTERSECTION_BEARINGS = "IntersectionBearings";
static const std::string DATASET_ENTRY_CLASS = "EntryClass";
/**
* Macro is not ideal. But without it we either have to:
* a) Write this boiler-plate for every usage of an optional dataset.
* b) Convert to a function and add lots of polluting NOLINT(bugprone-unchecked-optional-access)
* comments. This macro keeps the API code readable.
*/
#define CHECK_DATASET_DISABLED(val, dataset) \
{ \
if (!(val)) \
{ \
throw osrm::util::DisabledDatasetException((dataset)); \
} \
}
template <typename AlgorithmT> class ContiguousInternalMemoryAlgorithmDataFacade;
template <>
@ -141,18 +161,15 @@ class ContiguousInternalMemoryDataFacadeBase : public BaseDataFacade
std::string_view m_data_timestamp;
util::vector_view<util::Coordinate> m_coordinate_list;
extractor::PackedOSMIDsView m_osmnodeid_list;
util::vector_view<std::uint32_t> m_lane_description_offsets;
util::vector_view<extractor::TurnLaneType::Mask> m_lane_description_masks;
std::optional<util::vector_view<std::uint32_t>> m_lane_description_offsets;
std::optional<util::vector_view<extractor::TurnLaneType::Mask>> m_lane_description_masks;
util::vector_view<TurnPenalty> m_turn_weight_penalties;
util::vector_view<TurnPenalty> m_turn_duration_penalties;
extractor::SegmentDataView segment_data;
extractor::EdgeBasedNodeDataView edge_based_node_data;
guidance::TurnDataView turn_data;
std::optional<guidance::TurnDataView> turn_data;
util::vector_view<char> m_datasource_name_data;
util::vector_view<std::size_t> m_datasource_name_offsets;
util::vector_view<std::size_t> m_datasource_name_lengths;
util::vector_view<util::guidance::LaneTupleIdPair> m_lane_tupel_id_pairs;
std::optional<util::vector_view<util::guidance::LaneTupleIdPair>> m_lane_tuple_id_pairs;
util::vector_view<extractor::StorageManeuverOverride> m_maneuver_overrides;
util::vector_view<NodeID> m_maneuver_override_node_sequences;
@ -161,16 +178,24 @@ class ContiguousInternalMemoryDataFacadeBase : public BaseDataFacade
std::unique_ptr<SharedGeospatialQuery> m_geospatial_query;
boost::filesystem::path file_index_path;
extractor::IntersectionBearingsView intersection_bearings_view;
std::optional<extractor::IntersectionBearingsView> intersection_bearings_view;
extractor::NameTableView m_name_table;
std::optional<extractor::NameTableView> m_name_table;
// the look-up table for entry classes. An entry class lists the possibility of entry for all
// available turns. Such a class id is stored with every edge.
util::vector_view<util::guidance::EntryClass> m_entry_class_table;
std::optional<util::vector_view<util::guidance::EntryClass>> m_entry_class_table;
// allocator that keeps the allocation data
std::shared_ptr<ContiguousBlockAllocator> allocator;
bool isIndexed(const storage::SharedDataIndex &index, const std::string &name)
{
bool result = false;
index.List(name,
boost::make_function_output_iterator([&](const auto &) { result = true; }));
return result;
}
void InitializeInternalPointers(const storage::SharedDataIndex &index,
const std::string &metric_name,
const std::size_t exclude_index)
@ -183,7 +208,17 @@ class ContiguousInternalMemoryDataFacadeBase : public BaseDataFacade
exclude_mask = m_profile_properties->excludable_classes[exclude_index];
m_check_sum = *index.GetBlockPtr<std::uint32_t>("/common/connectivity_checksum");
// We no longer use "/common/connectivity_checksum", as osrm.edges is an optional dataset.
// Instead, we load the value from the MLD or CH graph, whichever is loaded.
if (isIndexed(index, "/mld/connectivity_checksum"))
{
m_check_sum = *index.GetBlockPtr<std::uint32_t>("/mld/connectivity_checksum");
}
else
{
BOOST_ASSERT(isIndexed(index, "/ch/connectivity_checksum"));
m_check_sum = *index.GetBlockPtr<std::uint32_t>("/ch/connectivity_checksum");
}
m_data_timestamp = make_timestamp_view(index, "/common/timestamp");
@ -196,13 +231,23 @@ class ContiguousInternalMemoryDataFacadeBase : public BaseDataFacade
edge_based_node_data = make_ebn_data_view(index, "/common/ebg_node_data");
turn_data = make_turn_data_view(index, "/common/turn_data");
if (isIndexed(index, "/common/turn_data"))
{
turn_data = make_turn_data_view(index, "/common/turn_data");
}
m_name_table = make_name_table_view(index, "/common/names");
if (isIndexed(index, "/common/names"))
{
m_name_table = make_name_table_view(index, "/common/names");
}
std::tie(m_lane_description_offsets, m_lane_description_masks) =
make_turn_lane_description_views(index, "/common/turn_lanes");
m_lane_tupel_id_pairs = make_lane_data_view(index, "/common/turn_lanes");
if (isIndexed(index, "/common/turn_lanes"))
{
std::tie(m_lane_description_offsets, m_lane_description_masks) =
make_turn_lane_description_views(index, "/common/turn_lanes");
m_lane_tuple_id_pairs = make_lane_data_view(index, "/common/turn_lanes");
}
m_turn_weight_penalties = make_turn_weight_view(index, "/common/turn_penalty");
m_turn_duration_penalties = make_turn_duration_view(index, "/common/turn_penalty");
@ -211,10 +256,12 @@ class ContiguousInternalMemoryDataFacadeBase : public BaseDataFacade
m_datasources = index.GetBlockPtr<extractor::Datasources>("/common/data_sources_names");
intersection_bearings_view =
make_intersection_bearings_view(index, "/common/intersection_bearings");
m_entry_class_table = make_entry_classes_view(index, "/common/entry_classes");
if (isIndexed(index, "/common/intersection_bearings"))
{
intersection_bearings_view =
make_intersection_bearings_view(index, "/common/intersection_bearings");
m_entry_class_table = make_entry_classes_view(index, "/common/entry_classes");
}
std::tie(m_maneuver_overrides, m_maneuver_override_node_sequences) =
make_maneuver_overrides_views(index, "/common/maneuver_overrides");
@ -305,7 +352,8 @@ class ContiguousInternalMemoryDataFacadeBase : public BaseDataFacade
osrm::guidance::TurnInstruction
GetTurnInstructionForEdgeID(const EdgeID edge_based_edge_id) const override final
{
return turn_data.GetTurnInstruction(edge_based_edge_id);
CHECK_DATASET_DISABLED(turn_data, DATASET_TURN_DATA);
return turn_data->GetTurnInstruction(edge_based_edge_id);
}
std::vector<RTreeLeaf> GetEdgesInBox(const util::Coordinate south_west,
@ -406,27 +454,32 @@ class ContiguousInternalMemoryDataFacadeBase : public BaseDataFacade
std::string_view GetNameForID(const NameID id) const override final
{
return m_name_table.GetNameForID(id);
CHECK_DATASET_DISABLED(m_name_table, DATASET_NAME_DATA);
return m_name_table->GetNameForID(id);
}
std::string_view GetRefForID(const NameID id) const override final
{
return m_name_table.GetRefForID(id);
CHECK_DATASET_DISABLED(m_name_table, DATASET_NAME_DATA);
return m_name_table->GetRefForID(id);
}
std::string_view GetPronunciationForID(const NameID id) const override final
{
return m_name_table.GetPronunciationForID(id);
CHECK_DATASET_DISABLED(m_name_table, DATASET_NAME_DATA);
return m_name_table->GetPronunciationForID(id);
}
std::string_view GetDestinationsForID(const NameID id) const override final
{
return m_name_table.GetDestinationsForID(id);
CHECK_DATASET_DISABLED(m_name_table, DATASET_NAME_DATA);
return m_name_table->GetDestinationsForID(id);
}
std::string_view GetExitsForID(const NameID id) const override final
{
return m_name_table.GetExitsForID(id);
CHECK_DATASET_DISABLED(m_name_table, DATASET_NAME_DATA);
return m_name_table->GetExitsForID(id);
}
std::string_view GetDatasourceName(const DatasourceID id) const override final
@ -459,46 +512,60 @@ class ContiguousInternalMemoryDataFacadeBase : public BaseDataFacade
util::guidance::BearingClass
GetBearingClass(const NodeID node_based_node_id) const override final
{
return intersection_bearings_view.GetBearingClass(node_based_node_id);
CHECK_DATASET_DISABLED(intersection_bearings_view, DATASET_INTERSECTION_BEARINGS);
return intersection_bearings_view->GetBearingClass(node_based_node_id);
}
guidance::TurnBearing PreTurnBearing(const EdgeID edge_based_edge_id) const override final
{
return turn_data.GetPreTurnBearing(edge_based_edge_id);
CHECK_DATASET_DISABLED(turn_data, DATASET_TURN_DATA);
return turn_data->GetPreTurnBearing(edge_based_edge_id);
}
guidance::TurnBearing PostTurnBearing(const EdgeID edge_based_edge_id) const override final
{
return turn_data.GetPostTurnBearing(edge_based_edge_id);
CHECK_DATASET_DISABLED(turn_data, DATASET_TURN_DATA);
return turn_data->GetPostTurnBearing(edge_based_edge_id);
}
util::guidance::EntryClass GetEntryClass(const EdgeID edge_based_edge_id) const override final
{
auto entry_class_id = turn_data.GetEntryClassID(edge_based_edge_id);
return m_entry_class_table.at(entry_class_id);
CHECK_DATASET_DISABLED(m_entry_class_table, DATASET_ENTRY_CLASS);
CHECK_DATASET_DISABLED(turn_data, DATASET_TURN_DATA);
auto entry_class_id = turn_data->GetEntryClassID(edge_based_edge_id);
return m_entry_class_table->at(entry_class_id);
}
bool HasLaneData(const EdgeID edge_based_edge_id) const override final
{
return turn_data.HasLaneData(edge_based_edge_id);
CHECK_DATASET_DISABLED(turn_data, DATASET_TURN_DATA);
return turn_data->HasLaneData(edge_based_edge_id);
}
util::guidance::LaneTupleIdPair
GetLaneData(const EdgeID edge_based_edge_id) const override final
{
CHECK_DATASET_DISABLED(turn_data, DATASET_TURN_DATA);
CHECK_DATASET_DISABLED(m_lane_tuple_id_pairs, DATASET_TURN_LANE_DATA);
BOOST_ASSERT(HasLaneData(edge_based_edge_id));
return m_lane_tupel_id_pairs.at(turn_data.GetLaneDataID(edge_based_edge_id));
return m_lane_tuple_id_pairs->at(turn_data->GetLaneDataID(edge_based_edge_id));
}
extractor::TurnLaneDescription
GetTurnDescription(const LaneDescriptionID lane_description_id) const override final
{
CHECK_DATASET_DISABLED(m_lane_description_offsets, DATASET_TURN_LANE_DATA);
CHECK_DATASET_DISABLED(m_lane_description_masks, DATASET_TURN_LANE_DATA);
if (lane_description_id == INVALID_LANE_DESCRIPTIONID)
return {};
else
return extractor::TurnLaneDescription(
m_lane_description_masks.begin() + m_lane_description_offsets[lane_description_id],
m_lane_description_masks.begin() +
m_lane_description_offsets[lane_description_id + 1]);
m_lane_description_masks->begin() +
m_lane_description_offsets->at(lane_description_id),
m_lane_description_masks->begin() +
m_lane_description_offsets->at(lane_description_id + 1));
}
bool IsLeftHandDriving(const NodeID edge_based_node_id) const override final

3
include/engine/engine_config.hpp
View File

@ -29,9 +29,11 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
#define ENGINE_CONFIG_HPP
#include "storage/storage_config.hpp"
#include "osrm/datasets.hpp"
#include <boost/filesystem/path.hpp>
#include <set>
#include <string>
namespace osrm::engine
@ -89,6 +91,7 @@ struct EngineConfig final
boost::filesystem::path memory_file;
bool use_mmap = true;
Algorithm algorithm = Algorithm::CH;
std::vector<storage::FeatureDataset> disable_feature_dataset;
std::string verbosity;
std::string dataset_name;
};

43
include/nodejs/node_osrm_support.hpp
View File

@ -205,9 +205,50 @@ inline engine_config_ptr argumentsToEngineConfig(const Napi::CallbackInfo &args)
}
}
auto disable_feature_dataset = params.Get("disable_feature_dataset");
if (disable_feature_dataset.IsArray())
{
Napi::Array datasets = disable_feature_dataset.As<Napi::Array>();
for (uint32_t i = 0; i < datasets.Length(); ++i)
{
Napi::Value dataset = datasets.Get(i);
if (!dataset.IsString())
{
ThrowError(args.Env(), "disable_feature_dataset list option must be a string");
return engine_config_ptr();
}
auto dataset_str = dataset.ToString().Utf8Value();
if (dataset_str == "ROUTE_GEOMETRY")
{
engine_config->disable_feature_dataset.push_back(
osrm::storage::FeatureDataset::ROUTE_GEOMETRY);
}
else if (dataset_str == "ROUTE_STEPS")
{
engine_config->disable_feature_dataset.push_back(
osrm::storage::FeatureDataset::ROUTE_STEPS);
}
else
{
ThrowError(
args.Env(),
"disable_feature_dataset array can include 'ROUTE_GEOMETRY', 'ROUTE_STEPS'.");
return engine_config_ptr();
}
}
}
else if (!disable_feature_dataset.IsUndefined())
{
ThrowError(args.Env(),
"disable_feature_dataset option must be an array and can include the string "
"values 'ROUTE_GEOMETRY', 'ROUTE_STEPS'.");
return engine_config_ptr();
}
if (!path.IsUndefined())
{
engine_config->storage_config = osrm::StorageConfig(path.ToString().Utf8Value());
engine_config->storage_config = osrm::StorageConfig(path.ToString().Utf8Value(),
engine_config->disable_feature_dataset);
engine_config->use_shared_memory = false;
}

15
include/osrm/datasets.hpp Normal file
View File

@ -0,0 +1,15 @@
#ifndef DATASETS_HPP
#define DATASETS_HPP
namespace osrm::storage
{
enum class FeatureDataset
{
ROUTE_STEPS,
ROUTE_GEOMETRY,
};
} // namespace osrm::storage
#endif

3
include/osrm/error_codes.hpp
View File

@ -23,7 +23,8 @@ enum ErrorCode
FileIOError,
UnexpectedEndOfFile,
IncompatibleDataset,
UnknownAlgorithm
UnknownAlgorithm,
UnknownFeatureDataset
#ifndef NDEBUG
// Leave this at the end. In debug mode, we assert that the size of
// this enum matches the number of messages we have documented, and __ENDMARKER__

5
include/storage/io_config.hpp
View File

@ -35,6 +35,11 @@ struct IOConfig
return {base_path.string() + fileName};
}
bool IsRequiredConfiguredInput(const std::string &fileName) const
{
return IsConfigured(fileName, required_input_files);
}
boost::filesystem::path base_path;
protected:

75
include/storage/storage_config.hpp
View File

@ -31,10 +31,61 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
#include <boost/filesystem/path.hpp>
#include "storage/io_config.hpp"
#include "osrm/datasets.hpp"
#include <set>
namespace osrm::storage
{
std::istream &operator>>(std::istream &in, FeatureDataset &datasets);
static std::vector<boost::filesystem::path>
GetRequiredFiles(const std::vector<storage::FeatureDataset> &disabled_feature_dataset)
{
std::set<boost::filesystem::path> required{
".osrm.datasource_names",
".osrm.ebg_nodes",
".osrm.edges",
".osrm.fileIndex",
".osrm.geometry",
".osrm.icd",
".osrm.maneuver_overrides",
".osrm.names",
".osrm.nbg_nodes",
".osrm.properties",
".osrm.ramIndex",
".osrm.timestamp",
".osrm.tld",
".osrm.tls",
".osrm.turn_duration_penalties",
".osrm.turn_weight_penalties",
};
for (const auto &to_disable : disabled_feature_dataset)
{
switch (to_disable)
{
case FeatureDataset::ROUTE_STEPS:
for (const auto &dataset : {".osrm.icd", ".osrm.tld", ".osrm.tls"})
{
required.erase(dataset);
}
break;
case FeatureDataset::ROUTE_GEOMETRY:
for (const auto &dataset :
{".osrm.edges", ".osrm.icd", ".osrm.names", ".osrm.tld", ".osrm.tls"})
{
required.erase(dataset);
}
break;
}
}
return std::vector<boost::filesystem::path>(required.begin(), required.end());
;
}
/**
* Configures OSRM's file storage paths.
*
@ -42,29 +93,17 @@ namespace osrm::storage
*/
struct StorageConfig final : IOConfig
{
StorageConfig(const boost::filesystem::path &base) : StorageConfig()
StorageConfig(const boost::filesystem::path &base,
const std::vector<storage::FeatureDataset> &disabled_feature_datasets_ = {})
: StorageConfig(disabled_feature_datasets_)
{
IOConfig::UseDefaultOutputNames(base);
}
StorageConfig()
StorageConfig(const std::vector<storage::FeatureDataset> &disabled_feature_datasets_ = {})
: IOConfig(
{".osrm.ramIndex",
".osrm.fileIndex",
".osrm.edges",
".osrm.geometry",
".osrm.turn_weight_penalties",
".osrm.turn_duration_penalties",
".osrm.datasource_names",
".osrm.ebg_nodes",
".osrm.names",
".osrm.nbg_nodes",
".osrm.timestamp",
".osrm.tls",
".osrm.tld",
".osrm.properties",
".osrm.icd",
".osrm.maneuver_overrides"},
GetRequiredFiles(disabled_feature_datasets_),
{".osrm.hsgr", ".osrm.cells", ".osrm.cell_metrics", ".osrm.mldgr", ".osrm.partition"},
{})
{

31
include/util/exception.hpp
View File

@ -62,7 +62,7 @@ class exception : public std::exception
* user supplied bad data, etc).
*/
constexpr const std::array<const char *, 11> ErrorDescriptions = {{
constexpr const std::array<const char *, 12> ErrorDescriptions = {{
"", // Dummy - ErrorCode values start at 2
"", // Dummy - ErrorCode values start at 2
"Fingerprint did not match the expected value", // InvalidFingerprint
@ -75,7 +75,8 @@ constexpr const std::array<const char *, 11> ErrorDescriptions = {{
// NOLINTNEXTLINE(bugprone-suspicious-missing-comma)
"The dataset you are trying to load is not " // IncompatibleDataset
"compatible with the routing algorithm you want to use.", // ...continued...
"Incompatible algorithm" // IncompatibleAlgorithm
"Incompatible algorithm", // IncompatibleAlgorithm
"Unknown feature dataset" // UnknownFeatureDataset
}};
#ifndef NDEBUG
@ -84,6 +85,32 @@ static_assert(ErrorDescriptions.size() == ErrorCode::__ENDMARKER__,
"ErrorCode list and ErrorDescription lists are different sizes");
#endif
class DisabledDatasetException : public exception
{
public:
explicit DisabledDatasetException(const std::string &dataset_)
: exception(BuildMessage(dataset_)), dataset(dataset_)
{
}
const std::string &Dataset() const { return dataset; }
private:
// This function exists to 'anchor' the class, and stop the compiler from
// copying vtable and RTTI info into every object file that includes
// this header. (Caught by -Wweak-vtables under Clang.)
virtual void anchor() const override;
const std::string dataset;
static std::string BuildMessage(const std::string &dataset)
{
return "DisabledDatasetException: Your query tried to access the disabled dataset " +
dataset +
". Please check your configuration: "
"https://github.com/Project-OSRM/osrm-backend/wiki/Disabled-Datasets";
}
};
class RuntimeError : public exception
{
using Base = exception;

1
src/nodejs/node_osrm.cpp
View File

@ -74,6 +74,7 @@ Napi::Object Engine::Init(Napi::Env env, Napi::Object exports)
* Old behaviour: Path to a file on disk to store the memory using mmap. Current behaviour: setting this value is the same as setting `mmap_memory: true`.
* @param {Boolean} [options.mmap_memory] Map on-disk files to virtual memory addresses (mmap), rather than loading into RAM.
* @param {String} [options.path] The path to the `.osrm` files. This is mutually exclusive with setting {options.shared_memory} to true.
* @param {Array} [options.disable_feature_dataset] Disables a feature dataset from being loaded into memory if not needed. Options: `ROUTE_STEPS`, `ROUTE_GEOMETRY`.
* @param {Number} [options.max_locations_trip] Max. locations supported in trip query (default: unlimited).
* @param {Number} [options.max_locations_viaroute] Max. locations supported in viaroute query (default: unlimited).
* @param {Number} [options.max_locations_distance_table] Max. locations supported in distance table query (default: unlimited).

98
src/server/request_handler.cpp
View File

@ -9,21 +9,16 @@
#include "util/log.hpp"
#include "util/string_util.hpp"
#include "util/timing_util.hpp"
#include "util/typedefs.hpp"
#include "engine/status.hpp"
#include "osrm/osrm.hpp"
#include "util/json_container.hpp"
#include <boost/iostreams/copy.hpp>
#include <boost/iostreams/filter/gzip.hpp>
#include <boost/iostreams/filtering_streambuf.hpp>
#include <ctime>
#include <algorithm>
#include <iostream>
#include <iterator>
#include <string>
#include <thread>
@ -36,6 +31,48 @@ void RequestHandler::RegisterServiceHandler(
service_handler = std::move(service_handler_);
}
void SendResponse(ServiceHandler::ResultT &result, http::reply &current_reply)
{
current_reply.headers.emplace_back("Access-Control-Allow-Origin", "*");
current_reply.headers.emplace_back("Access-Control-Allow-Methods", "GET");
current_reply.headers.emplace_back("Access-Control-Allow-Headers",
"X-Requested-With, Content-Type");
if (result.is<util::json::Object>())
{
current_reply.headers.emplace_back("Content-Type", "application/json; charset=UTF-8");
current_reply.headers.emplace_back("Content-Disposition",
"inline; filename=\"response.json\"");
util::json::render(current_reply.content, result.get<util::json::Object>());
}
else if (result.is<flatbuffers::FlatBufferBuilder>())
{
auto &buffer = result.get<flatbuffers::FlatBufferBuilder>();
current_reply.content.resize(buffer.GetSize());
std::copy(buffer.GetBufferPointer(),
buffer.GetBufferPointer() + buffer.GetSize(),
current_reply.content.begin());
current_reply.headers.emplace_back(
"Content-Type", "application/x-flatbuffers;schema=osrm.engine.api.fbresult");
}
else
{
BOOST_ASSERT(result.is<std::string>());
current_reply.content.resize(result.get<std::string>().size());
std::copy(result.get<std::string>().cbegin(),
result.get<std::string>().cend(),
current_reply.content.begin());
current_reply.headers.emplace_back("Content-Type", "application/x-protobuf");
}
// set headers
current_reply.headers.emplace_back("Content-Length",
std::to_string(current_reply.content.size()));
}
void RequestHandler::HandleRequest(const http::request &current_request, http::reply &current_reply)
{
if (!service_handler)
@ -96,43 +133,7 @@ void RequestHandler::HandleRequest(const http::request &current_request, http::r
std::to_string(position) + ": \"" + context + "\"";
}
current_reply.headers.emplace_back("Access-Control-Allow-Origin", "*");
current_reply.headers.emplace_back("Access-Control-Allow-Methods", "GET");
current_reply.headers.emplace_back("Access-Control-Allow-Headers",
"X-Requested-With, Content-Type");
if (result.is<util::json::Object>())
{
current_reply.headers.emplace_back("Content-Type", "application/json; charset=UTF-8");
current_reply.headers.emplace_back("Content-Disposition",
"inline; filename=\"response.json\"");
util::json::render(current_reply.content, result.get<util::json::Object>());
}
else if (result.is<flatbuffers::FlatBufferBuilder>())
{
auto &buffer = result.get<flatbuffers::FlatBufferBuilder>();
current_reply.content.resize(buffer.GetSize());
std::copy(buffer.GetBufferPointer(),
buffer.GetBufferPointer() + buffer.GetSize(),
current_reply.content.begin());
current_reply.headers.emplace_back(
"Content-Type", "application/x-flatbuffers;schema=osrm.engine.api.fbresult");
}
else
{
BOOST_ASSERT(result.is<std::string>());
current_reply.content.resize(result.get<std::string>().size());
std::copy(result.get<std::string>().cbegin(),
result.get<std::string>().cend(),
current_reply.content.begin());
current_reply.headers.emplace_back("Content-Type", "application/x-protobuf");
}
// set headers
current_reply.headers.emplace_back("Content-Length",
std::to_string(current_reply.content.size()));
SendResponse(result, current_reply);
if (!std::getenv("DISABLE_ACCESS_LOGGING"))
{
@ -168,6 +169,19 @@ void RequestHandler::HandleRequest(const http::request &current_request, http::r
<< request_string;
}
}
catch (const util::DisabledDatasetException &e)
{
current_reply.status = http::reply::bad_request;
ServiceHandler::ResultT result = util::json::Object();
auto &json_result = result.get<util::json::Object>();
json_result.values["code"] = "DisabledDataset";
json_result.values["message"] = e.what();
SendResponse(result, current_reply);
util::Log(logWARNING) << "[disabled dataset error][" << tid << "] code: DisabledDataset_"
<< e.Dataset() << ", uri: " << current_request.uri;
}
catch (const std::exception &e)
{
current_reply = http::reply::stock_reply(http::reply::internal_server_error);

100
src/storage/storage.cpp
View File

@ -294,17 +294,20 @@ std::vector<std::pair<bool, boost::filesystem::path>> Storage::GetStaticFiles()
std::vector<std::pair<bool, boost::filesystem::path>> files = {
{IS_OPTIONAL, config.GetPath(".osrm.cells")},
{IS_OPTIONAL, config.GetPath(".osrm.partition")},
{IS_REQUIRED, config.GetPath(".osrm.icd")},
{IS_REQUIRED, config.GetPath(".osrm.properties")},
{IS_REQUIRED, config.GetPath(".osrm.nbg_nodes")},
{IS_REQUIRED, config.GetPath(".osrm.ebg_nodes")},
{IS_REQUIRED, config.GetPath(".osrm.tls")},
{IS_REQUIRED, config.GetPath(".osrm.tld")},
{IS_REQUIRED, config.GetPath(".osrm.timestamp")},
{IS_REQUIRED, config.GetPath(".osrm.maneuver_overrides")},
{IS_REQUIRED, config.GetPath(".osrm.edges")},
{IS_REQUIRED, config.GetPath(".osrm.names")},
{IS_REQUIRED, config.GetPath(".osrm.ramIndex")}};
{IS_REQUIRED, config.GetPath(".osrm.nbg_nodes")},
{IS_REQUIRED, config.GetPath(".osrm.ramIndex")},
{IS_REQUIRED, config.GetPath(".osrm.properties")},
{IS_REQUIRED, config.GetPath(".osrm.timestamp")}};
for (const auto &file : {".osrm.edges", ".osrm.names", ".osrm.icd", ".osrm.tls", ".osrm.tld"})
{
if (config.IsRequiredConfiguredInput(file))
{
files.push_back({IS_REQUIRED, config.GetPath(file)});
}
}
for (const auto &file : files)
{
@ -394,12 +397,6 @@ void Storage::PopulateStaticData(const SharedDataIndex &index)
absolute_file_index_path.begin(), absolute_file_index_path.end(), file_index_path_ptr);
}
// Name data
{
auto name_table = make_name_table_view(index, "/common/names");
extractor::files::readNames(config.GetPath(".osrm.names"), name_table);
}
// Timestamp mark
{
auto timestamp_ref = make_timestamp_view(index, "/common/timestamp");
@ -412,25 +409,39 @@ void Storage::PopulateStaticData(const SharedDataIndex &index)
}
// Turn lane data
if (config.IsRequiredConfiguredInput(".osrm.tld"))
{
auto turn_lane_data = make_lane_data_view(index, "/common/turn_lanes");
extractor::files::readTurnLaneData(config.GetPath(".osrm.tld"), turn_lane_data);
}
// Turn lane descriptions
if (config.IsRequiredConfiguredInput(".osrm.tls"))
{
auto views = make_turn_lane_description_views(index, "/common/turn_lanes");
extractor::files::readTurnLaneDescriptions(
config.GetPath(".osrm.tls"), std::get<0>(views), std::get<1>(views));
}
// Load edge-based nodes data
// Load intersection data
if (config.IsRequiredConfiguredInput(".osrm.icd"))
{
auto node_data = make_ebn_data_view(index, "/common/ebg_node_data");
extractor::files::readNodeData(config.GetPath(".osrm.ebg_nodes"), node_data);
auto intersection_bearings_view =
make_intersection_bearings_view(index, "/common/intersection_bearings");
auto entry_classes = make_entry_classes_view(index, "/common/entry_classes");
extractor::files::readIntersections(
config.GetPath(".osrm.icd"), intersection_bearings_view, entry_classes);
}
// Name data
if (config.IsRequiredConfiguredInput(".osrm.names"))
{
auto name_table = make_name_table_view(index, "/common/names");
extractor::files::readNames(config.GetPath(".osrm.names"), name_table);
}
// Load original edge data
if (config.IsRequiredConfiguredInput(".osrm.edges"))
{
auto turn_data = make_turn_data_view(index, "/common/turn_data");
@ -441,6 +452,12 @@ void Storage::PopulateStaticData(const SharedDataIndex &index)
config.GetPath(".osrm.edges"), turn_data, *connectivity_checksum_ptr);
}
// Load edge-based nodes data
{
auto node_data = make_ebn_data_view(index, "/common/ebg_node_data");
extractor::files::readNodeData(config.GetPath(".osrm.ebg_nodes"), node_data);
}
// Loading list of coordinates
{
auto views = make_nbn_data_view(index, "/common/nbn_data");
@ -466,15 +483,6 @@ void Storage::PopulateStaticData(const SharedDataIndex &index)
metric_name = profile_properties_ptr->GetWeightName();
}
// Load intersection data
{
auto intersection_bearings_view =
make_intersection_bearings_view(index, "/common/intersection_bearings");
auto entry_classes = make_entry_classes_view(index, "/common/entry_classes");
extractor::files::readIntersections(
config.GetPath(".osrm.icd"), intersection_bearings_view, entry_classes);
}
if (boost::filesystem::exists(config.GetPath(".osrm.partition")))
{
auto mlp = make_partition_view(index, "/mld/multilevelpartition");
@ -545,15 +553,18 @@ void Storage::PopulateUpdatableData(const SharedDataIndex &index)
contractor::files::readGraph(
config.GetPath(".osrm.hsgr"), metrics, graph_connectivity_checksum);
auto turns_connectivity_checksum =
*index.GetBlockPtr<std::uint32_t>("/common/connectivity_checksum");
if (turns_connectivity_checksum != graph_connectivity_checksum)
if (config.IsRequiredConfiguredInput("osrm.edges"))
{
throw util::exception(
"Connectivity checksum " + std::to_string(graph_connectivity_checksum) + " in " +
config.GetPath(".osrm.hsgr").string() + " does not equal to checksum " +
std::to_string(turns_connectivity_checksum) + " in " +
config.GetPath(".osrm.edges").string());
auto turns_connectivity_checksum =
*index.GetBlockPtr<std::uint32_t>("/common/connectivity_checksum");
if (turns_connectivity_checksum != graph_connectivity_checksum)
{
throw util::exception(
"Connectivity checksum " + std::to_string(graph_connectivity_checksum) +
" in " + config.GetPath(".osrm.hsgr").string() +
" does not equal to checksum " + std::to_string(turns_connectivity_checksum) +
" in " + config.GetPath(".osrm.edges").string());
}
}
}
@ -573,15 +584,18 @@ void Storage::PopulateUpdatableData(const SharedDataIndex &index)
customizer::files::readGraph(
config.GetPath(".osrm.mldgr"), graph_view, graph_connectivity_checksum);
auto turns_connectivity_checksum =
*index.GetBlockPtr<std::uint32_t>("/common/connectivity_checksum");
if (turns_connectivity_checksum != graph_connectivity_checksum)
if (config.IsRequiredConfiguredInput("osrm.edges"))
{
throw util::exception(
"Connectivity checksum " + std::to_string(graph_connectivity_checksum) + " in " +
config.GetPath(".osrm.hsgr").string() + " does not equal to checksum " +
std::to_string(turns_connectivity_checksum) + " in " +
config.GetPath(".osrm.edges").string());
auto turns_connectivity_checksum =
*index.GetBlockPtr<std::uint32_t>("/common/connectivity_checksum");
if (turns_connectivity_checksum != graph_connectivity_checksum)
{
throw util::exception(
"Connectivity checksum " + std::to_string(graph_connectivity_checksum) +
" in " + config.GetPath(".osrm.mldgr").string() +
" does not equal to checksum " + std::to_string(turns_connectivity_checksum) +
" in " + config.GetPath(".osrm.edges").string());
}
}
}
}

23
src/storage/storage_config.cpp Normal file
View File

@ -0,0 +1,23 @@
#include "osrm/datasets.hpp"
#include "osrm/exception.hpp"
#include "util/exception_utils.hpp"
#include <boost/algorithm/string/case_conv.hpp>
namespace osrm::storage
{
std::istream &operator>>(std::istream &in, FeatureDataset &datasets)
{
std::string token;
in >> token;
boost::to_lower(token);
if (token == "route_steps")
datasets = FeatureDataset::ROUTE_STEPS;
else if (token == "route_geometry")
datasets = FeatureDataset::ROUTE_GEOMETRY;
else
throw util::RuntimeError(token, ErrorCode::UnknownFeatureDataset, SOURCE_REF);
return in;
}
} // namespace osrm::storage

9
src/tools/routed.cpp
View File

@ -4,6 +4,7 @@
#include "util/meminfo.hpp"
#include "util/version.hpp"
#include "osrm/datasets.hpp"
#include "osrm/engine_config.hpp"
#include "osrm/exception.hpp"
#include "osrm/osrm.hpp"
@ -23,7 +24,6 @@
#include <exception>
#include <future>
#include <iostream>
#include <memory>
#include <new>
#include <string>
#include <thread>
@ -69,6 +69,7 @@ std::istream &operator>>(std::istream &in, EngineConfig::Algorithm &algorithm)
throw util::RuntimeError(token, ErrorCode::UnknownAlgorithm, SOURCE_REF);
return in;
}
} // namespace osrm::engine
// overload validate for the double type to allow "unlimited" as an input
@ -155,6 +156,10 @@ inline unsigned generateServerProgramOptions(const int argc,
value<EngineConfig::Algorithm>(&config.algorithm)
->default_value(EngineConfig::Algorithm::CH, "CH"),
"Algorithm to use for the data. Can be CH, CoreCH, MLD.") //
("disable-feature-dataset",
value<std::vector<storage::FeatureDataset>>(&config.disable_feature_dataset)->multitoken(),
"Disables a feature dataset from being loaded into memory if not needed. Options: "
"ROUTE_STEPS, ROUTE_GEOMETRY") //
("max-viaroute-size",
value<int>(&config.max_locations_viaroute)->default_value(500),
"Max. locations supported in viaroute query") //
@ -276,7 +281,7 @@ try
if (!base_path.empty())
{
config.storage_config = storage::StorageConfig(base_path);
config.storage_config = storage::StorageConfig(base_path, config.disable_feature_dataset);
}
if (!config.use_shared_memory && !config.storage_config.IsValid())
{

17
src/tools/store.cpp
View File

@ -2,6 +2,7 @@
#include "storage/shared_memory.hpp"
#include "storage/shared_monitor.hpp"
#include "storage/storage.hpp"
#include "osrm/storage_config.hpp"
#include "osrm/exception.hpp"
#include "util/log.hpp"
@ -9,6 +10,7 @@
#include "util/typedefs.hpp"
#include "util/version.hpp"
#include <boost/algorithm/string/case_conv.hpp>
#include <boost/filesystem.hpp>
#include <boost/program_options.hpp>
@ -100,7 +102,8 @@ bool generateDataStoreOptions(const int argc,
std::string &dataset_name,
bool &list_datasets,
bool &list_blocks,
bool &only_metric)
bool &only_metric,
std::vector<storage::FeatureDataset> &disable_feature_dataset)
{
// declare a group of options that will be allowed only on command line
boost::program_options::options_description generic_options("Options");
@ -125,6 +128,12 @@ bool generateDataStoreOptions(const int argc,
boost::program_options::value<std::string>(&dataset_name)->default_value(""),
"Name of the dataset to load into memory. This allows having multiple datasets in memory "
"at the same time.") //
("disable-feature-dataset",
boost::program_options::value<std::vector<storage::FeatureDataset>>(
&disable_feature_dataset)
->multitoken(),
"Disables a feature dataset from being loaded into memory if not needed. Options: "
"ROUTE_STEPS, ROUTE_GEOMETRY") //
("list",
boost::program_options::value<bool>(&list_datasets)
->default_value(false)
@ -239,6 +248,7 @@ try
bool list_datasets = false;
bool list_blocks = false;
bool only_metric = false;
std::vector<storage::FeatureDataset> disable_feature_dataset;
if (!generateDataStoreOptions(argc,
argv,
verbosity,
@ -247,7 +257,8 @@ try
dataset_name,
list_datasets,
list_blocks,
only_metric))
only_metric,
disable_feature_dataset))
{
return EXIT_SUCCESS;
}
@ -260,7 +271,7 @@ try
return EXIT_SUCCESS;
}
storage::StorageConfig config(base_path);
storage::StorageConfig config(base_path, disable_feature_dataset);
if (!config.IsValid())
{
util::Log(logERROR) << "Config contains invalid file paths. Exiting!";

3
src/util/exception.cpp
View File

@ -1,5 +1,7 @@
#include "util/exception.hpp"
#include <utility>
// This function exists to 'anchor' the class, and stop the compiler from
// copying vtable and RTTI info into every object file that includes
// this header. (Caught by -Wweak-vtables under Clang.)
@ -16,4 +18,5 @@ namespace osrm::util
void exception::anchor() const {}
void RuntimeError::anchor() const {}
void DisabledDatasetException::anchor() const {}
} // namespace osrm::util

37
test/nodejs/index.js
View File

@ -163,6 +163,43 @@ test('constructor: throws on invalid custom limits', function(assert) {
})
});
});
test('constructor: throws on invalid disable_feature_dataset option', function(assert) {
assert.plan(1);
assert.throws(function() {
var osrm = new OSRM({
path: monaco_path,
disable_feature_dataset: ['NOT_EXIST'],
})
});
});
test('constructor: throws on non-array disable_feature_dataset', function(assert) {
assert.plan(1);
assert.throws(function() {
var osrm = new OSRM({
path: monaco_path,
disable_feature_dataset: 'ROUTE_GEOMETRY',
})
});
});
test('constructor: ok on valid disable_feature_dataset option', function(assert) {
assert.plan(1);
var osrm = new OSRM({
path: monaco_path,
disable_feature_dataset: ['ROUTE_GEOMETRY'],
});
assert.ok(osrm);
});
test('constructor: ok on multiple overlapping disable_feature_dataset options', function(assert) {
assert.plan(1);
var osrm = new OSRM({
path: monaco_path,
disable_feature_dataset: ['ROUTE_GEOMETRY', 'ROUTE_STEPS'],
});
assert.ok(osrm);
});
require('./route.js');
require('./trip.js');

62
test/nodejs/match.js
View File

@ -398,3 +398,65 @@ test('match: match in Monaco with waypoints', function(assert) {
}));
});
});
test('match: throws on disabled geometry', function (assert) {
assert.plan(1);
var osrm = new OSRM({'path': data_path, 'disable_feature_dataset': ['ROUTE_GEOMETRY']});
var options = {
coordinates: three_test_coordinates.concat(three_test_coordinates),
};
osrm.match(options, function(err, route) {
console.log(err)
assert.match(err.message, /DisabledDatasetException/);
});
});
test('match: ok on disabled geometry', function (assert) {
assert.plan(2);
var osrm = new OSRM({'path': data_path, 'disable_feature_dataset': ['ROUTE_GEOMETRY']});
var options = {
steps: false,
overview: 'false',
annotations: false,
skip_waypoints: true,
coordinates: three_test_coordinates.concat(three_test_coordinates),
};
osrm.match(options, function(err, response) {
assert.ifError(err);
assert.equal(response.matchings.length, 1);
});
});
test('match: throws on disabled steps', function (assert) {
assert.plan(1);
var osrm = new OSRM({'path': data_path, 'disable_feature_dataset': ['ROUTE_STEPS']});
var options = {
steps: true,
coordinates: three_test_coordinates.concat(three_test_coordinates),
};
osrm.match(options, function(err, route) {
console.log(err)
assert.match(err.message, /DisabledDatasetException/);
});
});
test('match: ok on disabled steps', function (assert) {
assert.plan(8);
var osrm = new OSRM({'path': data_path, 'disable_feature_dataset': ['ROUTE_STEPS']});
var options = {
steps: false,
overview: 'simplified',
annotations: true,
coordinates: three_test_coordinates.concat(three_test_coordinates),
};
osrm.match(options, function(err, response) {
assert.ifError(err);
assert.ok(response.tracepoints);
assert.ok(response.matchings);
assert.equal(response.matchings.length, 1);
assert.ok(response.matchings[0].geometry, "the match has geometry");
assert.ok(response.matchings[0].legs, "the match has legs");
assert.notok(response.matchings[0].legs.every(l => { return l.steps.length > 0; }), 'every leg has steps');
assert.ok(response.matchings[0].legs.every(l => { return l.annotation;}), 'every leg has annotations');
});
});

37
test/nodejs/nearest.js
View File

@ -103,3 +103,40 @@ test('nearest: nearest in Monaco without motorways', function(assert) {
assert.equal(response.waypoints.length, 1);
});
});
test('nearest: throws on disabled geometry', function(assert) {
assert.plan(1);
var osrm = new OSRM({path: data_path, 'disable_feature_dataset': ['ROUTE_GEOMETRY']});
var options = {
coordinates: [two_test_coordinates[0]],
};
osrm.nearest(options, function(err, response) {
console.log(err)
assert.match(err.message, /DisabledDatasetException/);
});
});
test('nearest: ok on disabled geometry', function(assert) {
assert.plan(2);
var osrm = new OSRM({path: data_path, 'disable_feature_dataset': ['ROUTE_GEOMETRY']});
var options = {
coordinates: [two_test_coordinates[0]],
skip_waypoints: true,
};
osrm.nearest(options, function(err, response) {
assert.ifError(err);
assert.notok(response.waypoints);
});
});
test('nearest: ok on disabled steps', function(assert) {
assert.plan(2);
var osrm = new OSRM({path: data_path, 'disable_feature_dataset': ['ROUTE_STEPS']});
var options = {
coordinates: [two_test_coordinates[0]],
};
osrm.nearest(options, function(err, response) {
assert.ifError(err);
assert.equal(response.waypoints.length, 1);
});
});

64
test/nodejs/route.js
View File

@ -761,4 +761,66 @@ test('route: snapping parameter passed through OK', function(assert) {
assert.ifError(err);
assert.equal(Math.round(route.routes[0].distance * 10), 1315); // Round it to nearest 0.1m to eliminate floating point comparison error
});
});
});
test('route: throws on disabled geometry', function (assert) {
assert.plan(1);
var osrm = new OSRM({'path': monaco_path, 'disable_feature_dataset': ['ROUTE_GEOMETRY']});
var options = {
coordinates: three_test_coordinates,
};
osrm.route(options, function(err, route) {
console.log(err)
assert.match(err.message, /DisabledDatasetException/);
});
});
test('route: ok on disabled geometry', function (assert) {
assert.plan(2);
var osrm = new OSRM({'path': monaco_path, 'disable_feature_dataset': ['ROUTE_GEOMETRY']});
var options = {
steps: false,
overview: 'false',
annotations: false,
skip_waypoints: true,
coordinates: three_test_coordinates,
};
osrm.route(options, function(err, response) {
assert.ifError(err);
assert.equal(response.routes.length, 1);
});
});
test('route: throws on disabled steps', function (assert) {
assert.plan(1);
var osrm = new OSRM({'path': monaco_path, 'disable_feature_dataset': ['ROUTE_STEPS']});
var options = {
steps: true,
coordinates: three_test_coordinates,
};
osrm.route(options, function(err, route) {
console.log(err)
assert.match(err.message, /DisabledDatasetException/);
});
});
test('route: ok on disabled steps', function (assert) {
assert.plan(8);
var osrm = new OSRM({'path': monaco_path, 'disable_feature_dataset': ['ROUTE_STEPS']});
var options = {
steps: false,
overview: 'simplified',
annotations: true,
coordinates: three_test_coordinates,
};
osrm.route(options, function(err, response) {
assert.ifError(err);
assert.ok(response.waypoints);
assert.ok(response.routes);
assert.equal(response.routes.length, 1);
assert.ok(response.routes[0].geometry, "the route has geometry");
assert.ok(response.routes[0].legs, "the route has legs");
assert.notok(response.routes[0].legs.every(l => { return l.steps.length > 0; }), 'every leg has steps');
assert.ok(response.routes[0].legs.every(l => { return l.annotation;}), 'every leg has annotations');
});
});

42
test/nodejs/table.js
View File

@ -19,7 +19,7 @@ test('table: flatbuffer format', function(assert) {
assert.ok(table instanceof Buffer);
const fb = FBResult.getRootAsFBResult(new flatbuffers.ByteBuffer(table));
assert.ok(fb.table());
});
});
@ -369,3 +369,43 @@ tables.forEach(function(annotation) {
});
});
test('table: throws on disabled geometry', function (assert) {
assert.plan(1);
var osrm = new OSRM({'path': data_path, 'disable_feature_dataset': ['ROUTE_GEOMETRY']});
var options = {
coordinates: [three_test_coordinates[0], three_test_coordinates[1]],
};
osrm.table(options, function(err, table) {
console.log(err)
assert.match(err.message, /DisabledDatasetException/);
});
});
test('table: ok on disabled geometry', function (assert) {
assert.plan(4);
var osrm = new OSRM({'path': data_path, 'disable_feature_dataset': ['ROUTE_GEOMETRY']});
var options = {
coordinates: [three_test_coordinates[0], three_test_coordinates[1]],
skip_waypoints: true
};
osrm.table(options, function(err, table) {
assert.ifError(err);
assert.ok(table.durations, 'distances table result should exist');
assert.notok(table.sources)
assert.notok(table.destinations)
});
});
test('table: ok on disabled steps', function (assert) {
assert.plan(4);
var osrm = new OSRM({'path': data_path, 'disable_feature_dataset': ['ROUTE_STEPS']});
var options = {
coordinates: [three_test_coordinates[0], three_test_coordinates[1]],
};
osrm.table(options, function(err, table) {
assert.ifError(err);
assert.ok(table.durations, 'distances table result should exist');
assert.ok(table.sources.length, 2)
assert.ok(table.destinations.length, 2)
});
});

17
test/nodejs/tile.js
View File

@ -23,3 +23,20 @@ test.test('tile interface pre-conditions', function(assert) {
assert.throws(function() { osrm.tile(17059, 11948, 15, function(err, result) {}) }, /must be an array \[x, y, z\]/);
assert.throws(function() { osrm.tile([17059, 11948, -15], function(err, result) {}) }, /must be unsigned/);
});
test.test('tile fails to load with geometry disabled', function(assert) {
assert.plan(1);
var osrm = new OSRM({'path': data_path, 'disable_feature_dataset': ['ROUTE_GEOMETRY']});
osrm.tile(tile.at, function(err, result) {
console.log(err)
assert.match(err.message, /DisabledDatasetException/);
});
});
test.test('tile ok with steps disabled', function(assert) {
assert.plan(2);
var osrm = new OSRM({'path': data_path, 'disable_feature_dataset': ['ROUTE_STEPS']});
osrm.tile(tile.at, function(err, result) {
assert.ifError(err);
assert.equal(result.length, tile.size);
});
});

62
test/nodejs/trip.js
View File

@ -368,3 +368,65 @@ test('trip: trip in Monaco without motorways', function(assert) {
});
});
test('trip: throws on disabled geometry', function (assert) {
assert.plan(1);
var osrm = new OSRM({'path': data_path, 'disable_feature_dataset': ['ROUTE_GEOMETRY']});
var options = {
coordinates: three_test_coordinates.concat(three_test_coordinates),
};
osrm.trip(options, function(err, route) {
console.log(err)
assert.match(err.message, /DisabledDatasetException/);
});
});
test('trip: ok on disabled geometry', function (assert) {
assert.plan(2);
var osrm = new OSRM({'path': data_path, 'disable_feature_dataset': ['ROUTE_GEOMETRY']});
var options = {
steps: false,
overview: 'false',
annotations: false,
skip_waypoints: true,
coordinates: three_test_coordinates.concat(three_test_coordinates),
};
osrm.trip(options, function(err, response) {
assert.ifError(err);
assert.equal(response.trips.length, 1);
});
});
test('trip: throws on disabled steps', function (assert) {
assert.plan(1);
var osrm = new OSRM({'path': data_path, 'disable_feature_dataset': ['ROUTE_STEPS']});
var options = {
steps: true,
coordinates: three_test_coordinates.concat(three_test_coordinates),
};
osrm.trip(options, function(err, route) {
console.log(err)
assert.match(err.message, /DisabledDatasetException/);
});
});
test('trip: ok on disabled steps', function (assert) {
assert.plan(8);
var osrm = new OSRM({'path': data_path, 'disable_feature_dataset': ['ROUTE_STEPS']});
var options = {
steps: false,
overview: 'simplified',
annotations: true,
coordinates: three_test_coordinates.concat(three_test_coordinates),
};
osrm.trip(options, function(err, response) {
assert.ifError(err);
assert.ok(response.waypoints);
assert.ok(response.trips);
assert.equal(response.trips.length, 1);
assert.ok(response.trips[0].geometry, "trip has geometry");
assert.ok(response.trips[0].legs, "trip has legs");
assert.notok(response.trips[0].legs.every(l => { return l.steps.length > 0; }), 'every leg has steps');
assert.ok(response.trips[0].legs.every(l => { return l.annotation;}), 'every leg has annotations');
});
});