Open-Harbor/osrm-backend
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 3 Wiki Activity

Add support for disabling feature datasets (#6666)

Browse Source 
This change adds support for disabling datasets, such that specific
files are not loaded into memory when running OSRM. This enables users
to not pay the memory cost for features they do not intend to use.

Initially, there are two options:
- ROUTE_GEOMETRY, for disabling overview, steps, annotations and waypoints.
- ROUTE_STEPS, for disabling steps only.

Attempts to query features for which the datasets are disabled will
lead to a DisabledDatasetException being returned.
This commit is contained in:
Michael Bell 2023-08-04 18:43:37 +01:00 committed by GitHub
parent 522d0f066e
commit db7946d762
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
34 changed files with 1005 additions and 200 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
CHANGELOG.md
View File

@ -2,6 +2,7 @@
- Changes from 5.27.1 - Changes from 5.27.1
- Features - Features
- ADDED: Add support for a default_radius flag. [#6575](https://github.com/Project-OSRM/osrm-backend/pull/6575) - ADDED: Add support for a default_radius flag. [#6575](https://github.com/Project-OSRM/osrm-backend/pull/6575)
- ADDED: Add support for disabling feature datasets. [#6666](https://github.com/Project-OSRM/osrm-backend/pull/6666)
- Build: - Build:
- ADDED: Add CI job which builds OSRM with gcc 12. [#6455](https://github.com/Project-OSRM/osrm-backend/pull/6455) - ADDED: Add CI job which builds OSRM with gcc 12. [#6455](https://github.com/Project-OSRM/osrm-backend/pull/6455)
- CHANGED: Upgrade to clang-tidy 15. [#6439](https://github.com/Project-OSRM/osrm-backend/pull/6439) - CHANGED: Upgrade to clang-tidy 15. [#6439](https://github.com/Project-OSRM/osrm-backend/pull/6439)

31
docs/http.md
View File

@ -88,10 +88,11 @@ Every response object has a `code` property containing one of the strings below
| `InvalidService` | Service name is invalid. | | `InvalidService` | Service name is invalid. |
| `InvalidVersion` | Version is not found. | | `InvalidVersion` | Version is not found. |
| `InvalidOptions` | Options are invalid. | | `InvalidOptions` | Options are invalid. |
| `InvalidQuery` | The query string is syntactically malformed. | | `InvalidQuery` | The query string is syntactically malformed. |
| `InvalidValue` | The successfully parsed query parameters are invalid. | | `InvalidValue` | The successfully parsed query parameters are invalid. |
| `NoSegment` | One of the supplied input coordinates could not snap to the street segment. | | `NoSegment` | One of the supplied input coordinates could not snap to the street segment. |
| `TooBig` | The request size violates one of the service-specific request size restrictions. | | `TooBig` | The request size violates one of the service-specific request size restrictions. |
| `DisabledDataset` | The request tried to access a disabled dataset. |
- `message` is a **optional** human-readable error message. All other status types are service-dependent. - `message` is a **optional** human-readable error message. All other status types are service-dependent.
- In case of an error the HTTP status code will be `400`. Otherwise, the HTTP status code will be `200` and `code` will be `Ok`. - In case of an error the HTTP status code will be `400`. Otherwise, the HTTP status code will be `200` and `code` will be `Ok`.
@ -130,7 +131,7 @@ In addition to the [general options](#general-options) the following options are
|number |`integer >= 1` (default `1`) |Number of nearest segments that should be returned. | |number |`integer >= 1` (default `1`) |Number of nearest segments that should be returned. |
As `waypoints` is a single thing, returned by that service, using it with the option `skip_waypoints` set to `true` is quite useless, but still As `waypoints` is a single thing, returned by that service, using it with the option `skip_waypoints` set to `true` is quite useless, but still
possible. In that case, only the `code` field will be returned. possible. In that case, only the `code` field will be returned.
**Response** **Response**
@ -953,11 +954,11 @@ The object is used to describe the waypoint on a route.
## Flatbuffers format ## Flatbuffers format
The default response format is `json`, but OSRM supports binary [`flatbuffers`](https://google.github.io/flatbuffers/) format, which The default response format is `json`, but OSRM supports binary [`flatbuffers`](https://google.github.io/flatbuffers/) format, which
is much faster in serialization/deserialization, comparing to `json`. is much faster in serialization/deserialization, comparing to `json`.
The format itself is described in message descriptors, located at `include/engine/api/flatbuffers` directory. Those descriptors could The format itself is described in message descriptors, located at `include/engine/api/flatbuffers` directory. Those descriptors could
be compiled to provide protocol parsers in Go/Javascript/Typescript/Java/Dart/C#/Python/Lobster/Lua/Rust/PHP/Kotlin. Precompiled be compiled to provide protocol parsers in Go/Javascript/Typescript/Java/Dart/C#/Python/Lobster/Lua/Rust/PHP/Kotlin. Precompiled
protocol parser for C++ is supplied with OSRM. protocol parser for C++ is supplied with OSRM.
`Flatbuffers` format provides exactly the same data, as `json` format with a slightly different layout, which was optimized to minimize `Flatbuffers` format provides exactly the same data, as `json` format with a slightly different layout, which was optimized to minimize
@ -971,7 +972,7 @@ Root object is the only object, available from a 'raw' `flatbuffers` buffer. It
**Properties** **Properties**
- `error`: `bool` Marks response as erroneous. An erroneous response should include the `code` fieldset, all the other fields may not be present. - `error`: `bool` Marks response as erroneous. An erroneous response should include the `code` fieldset, all the other fields may not be present.
- `code`: `Error` Error description object, only present, when `error` is `true` - `code`: `Error` Error description object, only present, when `error` is `true`
- `waypoints`: `[Waypoint]` Array of `Waypoint` objects. Should present for every service call, unless `skip_waypoints` is set to `true`. Table service will put `sources` array here. - `waypoints`: `[Waypoint]` Array of `Waypoint` objects. Should present for every service call, unless `skip_waypoints` is set to `true`. Table service will put `sources` array here.
- `routes`: `[RouteObject]` Array of `RouteObject` objects. May be empty or absent. Should present for Route/Trip/Match services call. - `routes`: `[RouteObject]` Array of `RouteObject` objects. May be empty or absent. Should present for Route/Trip/Match services call.
@ -983,21 +984,21 @@ Contains error information.
**Properties** **Properties**
- `code`: `string` Error code - `code`: `string` Error code
- `message`: `string` Detailed error message - `message`: `string` Detailed error message
### Waypoint object ### Waypoint object
Almost the same as `json` Waypoint object. The following properties differ: Almost the same as `json` Waypoint object. The following properties differ:
- `location`: `Position` Same as `json` location field, but different format. - `location`: `Position` Same as `json` location field, but different format.
- `nodes`: `Uint64Pair` Same as `json` nodes field, but different format. - `nodes`: `Uint64Pair` Same as `json` nodes field, but different format.
### RouteObject object ### RouteObject object
Almost the same as `json` Route object. The following properties differ: Almost the same as `json` Route object. The following properties differ:
- `polyline`: `string` Same as `json` geometry.polyline or geometry.polyline6 fields. One field for both formats. - `polyline`: `string` Same as `json` geometry.polyline or geometry.polyline6 fields. One field for both formats.
- `coordinates`: `[Position]` Same as `json` geometry.coordinates field, but different format. - `coordinates`: `[Position]` Same as `json` geometry.coordinates field, but different format.
- `legs`: `[Leg]` Array of `Leg` objects. - `legs`: `[Leg]` Array of `Leg` objects.
@ -1012,7 +1013,7 @@ Almost the same as `json` Leg object. The following properties differ:
Almost the same as `json` Step object. The following properties differ: Almost the same as `json` Step object. The following properties differ:
- `polyline`: `string` Same as `json` geometry.polyline or geometry.polyline6 fields. One field for both formats. - `polyline`: `string` Same as `json` geometry.polyline or geometry.polyline6 fields. One field for both formats.
- `coordinates`: `[Position]` Same as `json` geometry.coordinates field, but different format. - `coordinates`: `[Position]` Same as `json` geometry.coordinates field, but different format.
- `maneuver`: `StepManeuver` Same as `json` maneuver field, but different format. - `maneuver`: `StepManeuver` Same as `json` maneuver field, but different format.
@ -1035,7 +1036,7 @@ Almost the same as `json` Step object. The following properties differ:
| `ExitRoundabout` | Describes a maneuver exiting a roundabout (usually preceded by a `roundabout` instruction) | | `ExitRoundabout` | Describes a maneuver exiting a roundabout (usually preceded by a `roundabout` instruction) |
| `ExitRotary` | Describes the maneuver exiting a rotary (large named roundabout) | | `ExitRotary` | Describes the maneuver exiting a rotary (large named roundabout) |
- `driving_side`: `bool` Ttrue stands for the left side driving. - `driving_side`: `bool` Ttrue stands for the left side driving.
- `intersections`: `[Intersection]` Same as `json` intersections field, but different format. - `intersections`: `[Intersection]` Same as `json` intersections field, but different format.
### Intersection object ### Intersection object
@ -1049,7 +1050,7 @@ Almost the same as `json` Intersection object. The following properties differ:
Almost the same as `json` Lane object. The following properties differ: Almost the same as `json` Lane object. The following properties differ:
- `indications`: `Turn` Array of `Turn` enum values. - `indications`: `Turn` Array of `Turn` enum values.
| `value` | Description | | `value` | Description |
|------------------------|---------------------------------------------------------------------------------------------------------------------------| |------------------------|---------------------------------------------------------------------------------------------------------------------------|
@ -1089,16 +1090,16 @@ Almost the same as `json` StepManeuver object. The following properties differ:
| `ExitRoundabout` | Describes a maneuver exiting a roundabout (usually preceded by a `roundabout` instruction) | | `ExitRoundabout` | Describes a maneuver exiting a roundabout (usually preceded by a `roundabout` instruction) |
| `ExitRotary` | Describes the maneuver exiting a rotary (large named roundabout) | | `ExitRotary` | Describes the maneuver exiting a rotary (large named roundabout) |
- `modifier`: `Turn` Maneuver turn (enum) - `modifier`: `Turn` Maneuver turn (enum)
### Annotation object ### Annotation object
Exactly the same as `json` annotation object. Exactly the same as `json` annotation object.
### Position object ### Position object
A point on Earth. A point on Earth.
***Properties*** ***Properties***
- `longitute`: `float` Point's longitude - `longitute`: `float` Point's longitude

1
docs/nodejs/api.md
View File

@ -31,6 +31,7 @@ var osrm = new OSRM('network.osrm');
Old behaviour: Path to a file on disk to store the memory using mmap. Current behaviour: setting this value is the same as setting `mmap_memory: true`. Old behaviour: Path to a file on disk to store the memory using mmap. Current behaviour: setting this value is the same as setting `mmap_memory: true`.
- `options.mmap_memory` **[Boolean](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Boolean)?** Map on-disk files to virtual memory addresses (mmap), rather than loading into RAM. - `options.mmap_memory` **[Boolean](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Boolean)?** Map on-disk files to virtual memory addresses (mmap), rather than loading into RAM.
- `options.path` **[String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)?** The path to the `.osrm` files. This is mutually exclusive with setting {options.shared_memory} to true. - `options.path` **[String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)?** The path to the `.osrm` files. This is mutually exclusive with setting {options.shared_memory} to true.
- `options.disable_feature_dataset` **[Array](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Array)?** Disables a feature dataset from being loaded into memory if not needed. Options: `ROUTE_STEPS`, `ROUTE_GEOMETRY`.
- `options.max_locations_trip` **[Number](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Number)?** Max. locations supported in trip query (default: unlimited). - `options.max_locations_trip` **[Number](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Number)?** Max. locations supported in trip query (default: unlimited).
- `options.max_locations_viaroute` **[Number](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Number)?** Max. locations supported in viaroute query (default: unlimited). - `options.max_locations_viaroute` **[Number](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Number)?** Max. locations supported in viaroute query (default: unlimited).
- `options.max_locations_distance_table` **[Number](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Number)?** Max. locations supported in distance table query (default: unlimited). - `options.max_locations_distance_table` **[Number](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Number)?** Max. locations supported in distance table query (default: unlimited).

21
features/lib/osrm_loader.js
View File

@ -68,8 +68,9 @@ class OSRMDirectLoader extends OSRMBaseLoader {
super(scope); super(scope);
} }
load (inputFile, callback) { load (ctx, callback) {
this.inputFile = inputFile; this.inputFile = ctx.inputFile;
this.loaderArgs = ctx.loaderArgs;
this.shutdown(() => { this.shutdown(() => {
this.launch(callback); this.launch(callback);
}); });
@ -78,7 +79,7 @@ class OSRMDirectLoader extends OSRMBaseLoader {
osrmUp (callback) { osrmUp (callback) {
if (this.osrmIsRunning()) return callback(new Error("osrm-routed already running!")); if (this.osrmIsRunning()) return callback(new Error("osrm-routed already running!"));
const command_arguments = util.format('%s -p %d -i %s -a %s', this.inputFile, this.scope.OSRM_PORT, this.scope.OSRM_IP, this.scope.ROUTING_ALGORITHM); const command_arguments = util.format('%s -p %d -i %s -a %s %s', this.inputFile, this.scope.OSRM_PORT, this.scope.OSRM_IP, this.scope.ROUTING_ALGORITHM, this.loaderArgs);
this.child = this.scope.runBin('osrm-routed', command_arguments, this.scope.environment, (err) => { this.child = this.scope.runBin('osrm-routed', command_arguments, this.scope.environment, (err) => {
if (err && err.signal !== 'SIGINT') { if (err && err.signal !== 'SIGINT') {
this.child = null; this.child = null;
@ -101,8 +102,9 @@ class OSRMmmapLoader extends OSRMBaseLoader {
super(scope); super(scope);
} }
load (inputFile, callback) { load (ctx, callback) {
this.inputFile = inputFile; this.inputFile = ctx.inputFile;
this.loaderArgs = ctx.loaderArgs;
this.shutdown(() => { this.shutdown(() => {
this.launch(callback); this.launch(callback);
}); });
@ -111,7 +113,7 @@ class OSRMmmapLoader extends OSRMBaseLoader {
osrmUp (callback) { osrmUp (callback) {
if (this.osrmIsRunning()) return callback(new Error("osrm-routed already running!")); if (this.osrmIsRunning()) return callback(new Error("osrm-routed already running!"));
const command_arguments = util.format('%s -p %d -i %s -a %s --mmap', this.inputFile, this.scope.OSRM_PORT, this.scope.OSRM_IP, this.scope.ROUTING_ALGORITHM); const command_arguments = util.format('%s -p %d -i %s -a %s --mmap %s', this.inputFile, this.scope.OSRM_PORT, this.scope.OSRM_IP, this.scope.ROUTING_ALGORITHM, this.loaderArgs);
this.child = this.scope.runBin('osrm-routed', command_arguments, this.scope.environment, (err) => { this.child = this.scope.runBin('osrm-routed', command_arguments, this.scope.environment, (err) => {
if (err && err.signal !== 'SIGINT') { if (err && err.signal !== 'SIGINT') {
this.child = null; this.child = null;
@ -134,8 +136,9 @@ class OSRMDatastoreLoader extends OSRMBaseLoader {
super(scope); super(scope);
} }
load (inputFile, callback) { load (ctx, callback) {
this.inputFile = inputFile; this.inputFile = ctx.inputFile;
this.loaderArgs = ctx.loaderArgs;
this.loadData((err) => { this.loadData((err) => {
if (err) return callback(err); if (err) return callback(err);
@ -148,7 +151,7 @@ class OSRMDatastoreLoader extends OSRMBaseLoader {
} }
loadData (callback) { loadData (callback) {
const command_arguments = util.format('--dataset-name=%s %s', this.scope.DATASET_NAME, this.inputFile); const command_arguments = util.format('--dataset-name=%s %s %s', this.scope.DATASET_NAME, this.inputFile, this.loaderArgs);
this.scope.runBin('osrm-datastore', command_arguments, this.scope.environment, (err) => { this.scope.runBin('osrm-datastore', command_arguments, this.scope.environment, (err) => {
if (err) return callback(new Error('*** osrm-datastore exited with ' + err.code + ': ' + err)); if (err) return callback(new Error('*** osrm-datastore exited with ' + err.code + ': ' + err));
callback(); callback();

141
features/options/data/disabled_dataset.feature Normal file
View File

@ -0,0 +1,141 @@
@routing @disable-feature-dataset
Feature: disable-feature-dataset command line options
Background:
Given the profile "testbot"
And the node map
"""
0
a b c
"""
And the ways
| nodes |
| ab |
| bc |
Scenario: disable-feature-dataset - geometry disabled error
Given the data load extra arguments "--disable-feature-dataset ROUTE_GEOMETRY"
# The default values
And the query options
| overview | simplified |
| annotations | false |
| steps | false |
| skip_waypoints | false |
When I route I should get
| from | to | code |
| a | c | DisabledDataset |
When I plan a trip I should get
| waypoints | code |
| a,b,c | DisabledDataset |
When I match I should get
| trace | code |
| abc | DisabledDataset |
Scenario: disable-feature-dataset - geometry disabled error table
Given the data load extra arguments "--disable-feature-dataset ROUTE_GEOMETRY"
When I request nearest I should get
| in | code |
| 0 | DisabledDataset |
When I request a travel time matrix with these waypoints I should get the response code
| waypoints | code |
| a,b,c | DisabledDataset |
Scenario: disable-feature-dataset - geometry disabled success
Given the data load extra arguments "--disable-feature-dataset ROUTE_GEOMETRY"
# No geometry values returned
And the query options
| overview | false |
| annotations | false |
| steps | false |
| skip_waypoints | true |
When I route I should get
| from | to | code |
| a | c | Ok |
When I plan a trip I should get
| waypoints | code |
| a,b,c | Ok |
When I match I should get
| trace | code |
| abc | Ok |
Scenario: disable-feature-dataset - geometry disabled error table
Given the data load extra arguments "--disable-feature-dataset ROUTE_GEOMETRY"
And the query options
| skip_waypoints | true |
# You would never do this, but just to prove the point.
When I request nearest I should get
| in | code |
| 0 | Ok |
When I request a travel time matrix with these waypoints I should get the response code
| waypoints | code |
| a,b,c | Ok |
Scenario: disable-feature-dataset - steps disabled error
Given the data load extra arguments "--disable-feature-dataset ROUTE_STEPS"
# Default + annotations, steps
And the query options
| overview | simplified |
| annotations | true |
| steps | true |
When I route I should get
| from | to | code |
| a | c | DisabledDataset |
When I plan a trip I should get
| waypoints | code |
| a,b,c | DisabledDataset |
When I match I should get
| trace | code |
| abc | DisabledDataset |
Scenario: disable-feature-dataset - geometry disabled error table
Given the data load extra arguments "--disable-feature-dataset ROUTE_STEPS"
When I request nearest I should get
| in | code |
| 0 | Ok |
When I request a travel time matrix with these waypoints I should get the response code
| waypoints | code |
| a,b,c | Ok |
Scenario: disable-feature-dataset - steps disabled success
Given the data load extra arguments "--disable-feature-dataset ROUTE_STEPS"
# Default + steps
And the query options
| overview | simplified |
| annotations | true |
| steps | false |
When I route I should get
| from | to | code |
| a | c | Ok |
When I plan a trip I should get
| waypoints | code |
| a,b,c | Ok |
When I match I should get
| trace | code |
| abc | Ok |

5
features/step_definitions/data.js
View File

@ -33,6 +33,11 @@ module.exports = function () {
callback(); callback();
}); });
this.Given(/^the data load extra arguments "(.*?)"$/, (args, callback) => {
this.loaderArgs = this.expandOptions(args);
callback();
});
this.Given(/^a grid size of ([0-9.]+) meters$/, (meters, callback) => { this.Given(/^a grid size of ([0-9.]+) meters$/, (meters, callback) => {
this.setGridSize(meters); this.setGridSize(meters);
callback(); callback();

63
features/step_definitions/distance_matrix.js
View File

@ -5,6 +5,7 @@ var FBResult = require('../support/fbresult_generated').osrm.engine.api.fbresult
module.exports = function () { module.exports = function () {
const durationsRegex = new RegExp(/^I request a travel time matrix I should get$/); const durationsRegex = new RegExp(/^I request a travel time matrix I should get$/);
const durationsCodeOnlyRegex = new RegExp(/^I request a travel time matrix with these waypoints I should get the response code$/);
const distancesRegex = new RegExp(/^I request a travel distance matrix I should get$/); const distancesRegex = new RegExp(/^I request a travel distance matrix I should get$/);
const estimatesRegex = new RegExp(/^I request a travel time matrix I should get estimates for$/); const estimatesRegex = new RegExp(/^I request a travel time matrix I should get estimates for$/);
const durationsRegexFb = new RegExp(/^I request a travel time matrix with flatbuffers I should get$/); const durationsRegexFb = new RegExp(/^I request a travel time matrix with flatbuffers I should get$/);
@ -17,6 +18,7 @@ module.exports = function () {
const FORMAT_FB = 'flatbuffers'; const FORMAT_FB = 'flatbuffers';
this.When(durationsRegex, function(table, callback) {tableParse.call(this, table, DURATIONS_NO_ROUTE, 'durations', FORMAT_JSON, callback);}.bind(this)); this.When(durationsRegex, function(table, callback) {tableParse.call(this, table, DURATIONS_NO_ROUTE, 'durations', FORMAT_JSON, callback);}.bind(this));
this.When(durationsCodeOnlyRegex, function(table, callback) {tableCodeOnlyParse.call(this, table, 'durations', FORMAT_JSON, callback);}.bind(this));
this.When(distancesRegex, function(table, callback) {tableParse.call(this, table, DISTANCES_NO_ROUTE, 'distances', FORMAT_JSON, callback);}.bind(this)); this.When(distancesRegex, function(table, callback) {tableParse.call(this, table, DISTANCES_NO_ROUTE, 'distances', FORMAT_JSON, callback);}.bind(this));
this.When(estimatesRegex, function(table, callback) {tableParse.call(this, table, DISTANCES_NO_ROUTE, 'fallback_speed_cells', FORMAT_JSON, callback);}.bind(this)); this.When(estimatesRegex, function(table, callback) {tableParse.call(this, table, DISTANCES_NO_ROUTE, 'fallback_speed_cells', FORMAT_JSON, callback);}.bind(this));
this.When(durationsRegexFb, function(table, callback) {tableParse.call(this, table, DURATIONS_NO_ROUTE, 'durations', FORMAT_FB, callback);}.bind(this)); this.When(durationsRegexFb, function(table, callback) {tableParse.call(this, table, DURATIONS_NO_ROUTE, 'durations', FORMAT_FB, callback);}.bind(this));
@ -27,6 +29,64 @@ const durationsParse = function(v) { return isNaN(parseInt(v)); };
const distancesParse = function(v) { return isNaN(parseFloat(v)); }; const distancesParse = function(v) { return isNaN(parseFloat(v)); };
const estimatesParse = function(v) { return isNaN(parseFloat(v)); }; const estimatesParse = function(v) { return isNaN(parseFloat(v)); };
function tableCodeOnlyParse(table, annotation, format, callback) {
const params = this.queryParams;
params.annotations = ['durations','fallback_speed_cells'].indexOf(annotation) !== -1 ? 'duration' : 'distance';
params.output = format;
var got;
this.reprocessAndLoadData((e) => {
if (e) return callback(e);
var testRow = (row, ri, cb) => {
var afterRequest = (err, res) => {
if (err) return cb(err);
for (var k in row) {
var match = k.match(/param:(.*)/);
if (match) {
if (row[k] === '(nil)') {
params[match[1]] = null;
} else if (row[k]) {
params[match[1]] = [row[k]];
}
got[k] = row[k];
}
}
var json;
got.code = 'unknown';
if (res.body.length) {
json = JSON.parse(res.body);
got.code = json.code;
}
cb(null, got);
};
var params = this.queryParams,
waypoints = [];
if (row.waypoints) {
row.waypoints.split(',').forEach((n) => {
var node = this.findNodeByName(n);
if (!node) throw new Error(util.format('*** unknown waypoint node "%s"', n.trim()));
waypoints.push({ coord: node, type: 'loc' });
});
got = { waypoints: row.waypoints };
this.requestTable(waypoints, params, afterRequest);
} else {
throw new Error('*** no waypoints');
}
};
this.processRowsAndDiff(table, testRow, callback);
});
}
function tableParse(table, noRoute, annotation, format, callback) { function tableParse(table, noRoute, annotation, format, callback) {
const parse = annotation == 'distances' ? distancesParse : (annotation == 'durations' ? durationsParse : estimatesParse); const parse = annotation == 'distances' ? distancesParse : (annotation == 'durations' ? durationsParse : estimatesParse);
@ -62,9 +122,6 @@ function tableParse(table, noRoute, annotation, format, callback) {
}); });
} }
var actual = [];
actual.push(table.headers);
this.reprocessAndLoadData((e) => { this.reprocessAndLoadData((e) => {
if (e) return callback(e); if (e) return callback(e);
// compute matrix // compute matrix

46
features/step_definitions/nearest.js
View File

@ -12,35 +12,43 @@ module.exports = function () {
var inNode = this.findNodeByName(row.in); var inNode = this.findNodeByName(row.in);
if (!inNode) throw new Error(util.format('*** unknown in-node "%s"', row.in)); if (!inNode) throw new Error(util.format('*** unknown in-node "%s"', row.in));
var outNode = this.findNodeByName(row.out);
if (!outNode) throw new Error(util.format('*** unknown out-node "%s"', row.out));
this.requestNearest(inNode, this.queryParams, (err, response) => { this.requestNearest(inNode, this.queryParams, (err, response) => {
if (err) return cb(err); if (err) return cb(err);
var coord; var coord;
var headers = new Set(table.raw()[0]); var headers = new Set(table.raw()[0]);
if (response.statusCode === 200 && response.body.length) { var got = { in: row.in};
if (response.body.length) {
var json = JSON.parse(response.body); var json = JSON.parse(response.body);
got.code = json.code;
coord = json.waypoints[0].location; if (response.statusCode === 200) {
var got = { in: row.in, out: row.out }; if (headers.has('data_version')) {
got.data_version = json.data_version || '';
if (headers.has('data_version')) {
got.data_version = json.data_version || '';
}
Object.keys(row).forEach((key) => {
if (key === 'out') {
if (this.FuzzyMatch.matchLocation(coord, outNode)) {
got[key] = row[key];
} else {
row[key] = util.format('%s [%d,%d]', row[key], outNode.lat, outNode.lon);
}
} }
});
if (json.waypoints && json.waypoints.length && row.out) {
coord = json.waypoints[0].location;
got.out = row.out;
var outNode = this.findNodeByName(row.out);
if (!outNode) throw new Error(util.format('*** unknown out-node "%s"', row.out));
Object.keys(row).forEach((key) => {
if (key === 'out') {
if (this.FuzzyMatch.matchLocation(coord, outNode)) {
got[key] = row[key];
} else {
row[key] = util.format('%s [%d,%d]', row[key], outNode.lat, outNode.lon);
}
}
});
}
}
cb(null, got); cb(null, got);
} }
else { else {

3
features/step_definitions/trip.js
View File

@ -91,7 +91,7 @@ module.exports = function () {
var encodedResult = ''; var encodedResult = '';
if (json.trips) row.trips.split(',').forEach((sub, si) => { if (json.trips && row.trips) row.trips.split(',').forEach((sub, si) => {
if (si >= subTrips.length) { if (si >= subTrips.length) {
ok = false; ok = false;
} else { } else {
@ -134,7 +134,6 @@ module.exports = function () {
} else { } else {
var params = this.queryParams, var params = this.queryParams,
waypoints = []; waypoints = [];
params['steps'] = 'true';
if (row.from && row.to) { if (row.from && row.to) {
var fromNode = this.findNodeByName(row.from); var fromNode = this.findNodeByName(row.from);
if (!fromNode) throw new Error(util.format('*** unknown from-node "%s"', row.from)); if (!fromNode) throw new Error(util.format('*** unknown from-node "%s"', row.from));

3
features/support/data.js
View File

@ -280,10 +280,11 @@ module.exports = function () {
}; };
this.reprocessAndLoadData = (callback) => { this.reprocessAndLoadData = (callback) => {
let p = {loaderArgs: this.loaderArgs, inputFile: this.processedCacheFile};
let queue = d3.queue(1); let queue = d3.queue(1);
queue.defer(this.writeAndLinkOSM.bind(this)); queue.defer(this.writeAndLinkOSM.bind(this));
queue.defer(this.extractContractPartitionAndCustomize.bind(this)); queue.defer(this.extractContractPartitionAndCustomize.bind(this));
queue.defer(this.osrmLoader.load.bind(this.osrmLoader), this.processedCacheFile); queue.defer(this.osrmLoader.load.bind(this.osrmLoader), p);
queue.awaitAll(callback); queue.awaitAll(callback);
}; };

1
features/support/hooks.js
View File

@ -37,6 +37,7 @@ module.exports = function () {
this.contractArgs = ''; this.contractArgs = '';
this.partitionArgs = ''; this.partitionArgs = '';
this.customizeArgs = ''; this.customizeArgs = '';
this.loaderArgs = '';
this.environment = Object.assign(this.DEFAULT_ENVIRONMENT); this.environment = Object.assign(this.DEFAULT_ENVIRONMENT);
this.resetOSM(); this.resetOSM();

3
features/support/route.js
View File

@ -101,7 +101,8 @@ module.exports = function () {
this.requestTrip = (waypoints, userParams, callback) => { this.requestTrip = (waypoints, userParams, callback) => {
var defaults = { var defaults = {
output: 'json' output: 'json',
steps: 'true'
}, },
params = this.overwriteParams(defaults, userParams); params = this.overwriteParams(defaults, userParams);

139
include/engine/datafacade/contiguous_internalmem_datafacade.hpp
View File

@ -31,6 +31,26 @@
namespace osrm::engine::datafacade namespace osrm::engine::datafacade
{ {
static const std::string DATASET_TURN_DATA = "TurnData";
static const std::string DATASET_TURN_LANE_DATA = "NameLaneData";
static const std::string DATASET_NAME_DATA = "NameData";
static const std::string DATASET_INTERSECTION_BEARINGS = "IntersectionBearings";
static const std::string DATASET_ENTRY_CLASS = "EntryClass";
/**
* Macro is not ideal. But without it we either have to:
* a) Write this boiler-plate for every usage of an optional dataset.
* b) Convert to a function and add lots of polluting NOLINT(bugprone-unchecked-optional-access)
* comments. This macro keeps the API code readable.
*/
#define CHECK_DATASET_DISABLED(val, dataset) \
{ \
if (!(val)) \
{ \
throw osrm::util::DisabledDatasetException((dataset)); \
} \
}
template <typename AlgorithmT> class ContiguousInternalMemoryAlgorithmDataFacade; template <typename AlgorithmT> class ContiguousInternalMemoryAlgorithmDataFacade;
template <> template <>
@ -141,18 +161,15 @@ class ContiguousInternalMemoryDataFacadeBase : public BaseDataFacade
std::string_view m_data_timestamp; std::string_view m_data_timestamp;
util::vector_view<util::Coordinate> m_coordinate_list; util::vector_view<util::Coordinate> m_coordinate_list;
extractor::PackedOSMIDsView m_osmnodeid_list; extractor::PackedOSMIDsView m_osmnodeid_list;
util::vector_view<std::uint32_t> m_lane_description_offsets; std::optional<util::vector_view<std::uint32_t>> m_lane_description_offsets;
util::vector_view<extractor::TurnLaneType::Mask> m_lane_description_masks; std::optional<util::vector_view<extractor::TurnLaneType::Mask>> m_lane_description_masks;
util::vector_view<TurnPenalty> m_turn_weight_penalties; util::vector_view<TurnPenalty> m_turn_weight_penalties;
util::vector_view<TurnPenalty> m_turn_duration_penalties; util::vector_view<TurnPenalty> m_turn_duration_penalties;
extractor::SegmentDataView segment_data; extractor::SegmentDataView segment_data;
extractor::EdgeBasedNodeDataView edge_based_node_data; extractor::EdgeBasedNodeDataView edge_based_node_data;
guidance::TurnDataView turn_data; std::optional<guidance::TurnDataView> turn_data;
util::vector_view<char> m_datasource_name_data; std::optional<util::vector_view<util::guidance::LaneTupleIdPair>> m_lane_tuple_id_pairs;
util::vector_view<std::size_t> m_datasource_name_offsets;
util::vector_view<std::size_t> m_datasource_name_lengths;
util::vector_view<util::guidance::LaneTupleIdPair> m_lane_tupel_id_pairs;
util::vector_view<extractor::StorageManeuverOverride> m_maneuver_overrides; util::vector_view<extractor::StorageManeuverOverride> m_maneuver_overrides;
util::vector_view<NodeID> m_maneuver_override_node_sequences; util::vector_view<NodeID> m_maneuver_override_node_sequences;
@ -161,16 +178,24 @@ class ContiguousInternalMemoryDataFacadeBase : public BaseDataFacade
std::unique_ptr<SharedGeospatialQuery> m_geospatial_query; std::unique_ptr<SharedGeospatialQuery> m_geospatial_query;
boost::filesystem::path file_index_path; boost::filesystem::path file_index_path;
extractor::IntersectionBearingsView intersection_bearings_view; std::optional<extractor::IntersectionBearingsView> intersection_bearings_view;
extractor::NameTableView m_name_table; std::optional<extractor::NameTableView> m_name_table;
// the look-up table for entry classes. An entry class lists the possibility of entry for all // the look-up table for entry classes. An entry class lists the possibility of entry for all
// available turns. Such a class id is stored with every edge. // available turns. Such a class id is stored with every edge.
util::vector_view<util::guidance::EntryClass> m_entry_class_table; std::optional<util::vector_view<util::guidance::EntryClass>> m_entry_class_table;
// allocator that keeps the allocation data // allocator that keeps the allocation data
std::shared_ptr<ContiguousBlockAllocator> allocator; std::shared_ptr<ContiguousBlockAllocator> allocator;
bool isIndexed(const storage::SharedDataIndex &index, const std::string &name)
{
bool result = false;
index.List(name,
boost::make_function_output_iterator([&](const auto &) { result = true; }));
return result;
}
void InitializeInternalPointers(const storage::SharedDataIndex &index, void InitializeInternalPointers(const storage::SharedDataIndex &index,
const std::string &metric_name, const std::string &metric_name,
const std::size_t exclude_index) const std::size_t exclude_index)
@ -183,7 +208,17 @@ class ContiguousInternalMemoryDataFacadeBase : public BaseDataFacade
exclude_mask = m_profile_properties->excludable_classes[exclude_index]; exclude_mask = m_profile_properties->excludable_classes[exclude_index];
m_check_sum = *index.GetBlockPtr<std::uint32_t>("/common/connectivity_checksum"); // We no longer use "/common/connectivity_checksum", as osrm.edges is an optional dataset.
// Instead, we load the value from the MLD or CH graph, whichever is loaded.
if (isIndexed(index, "/mld/connectivity_checksum"))
{
m_check_sum = *index.GetBlockPtr<std::uint32_t>("/mld/connectivity_checksum");
}
else
{
BOOST_ASSERT(isIndexed(index, "/ch/connectivity_checksum"));
m_check_sum = *index.GetBlockPtr<std::uint32_t>("/ch/connectivity_checksum");
}
m_data_timestamp = make_timestamp_view(index, "/common/timestamp"); m_data_timestamp = make_timestamp_view(index, "/common/timestamp");
@ -196,13 +231,23 @@ class ContiguousInternalMemoryDataFacadeBase : public BaseDataFacade
edge_based_node_data = make_ebn_data_view(index, "/common/ebg_node_data"); edge_based_node_data = make_ebn_data_view(index, "/common/ebg_node_data");
turn_data = make_turn_data_view(index, "/common/turn_data"); if (isIndexed(index, "/common/turn_data"))
{
turn_data = make_turn_data_view(index, "/common/turn_data");
}
m_name_table = make_name_table_view(index, "/common/names"); if (isIndexed(index, "/common/names"))
{
m_name_table = make_name_table_view(index, "/common/names");
}
std::tie(m_lane_description_offsets, m_lane_description_masks) = if (isIndexed(index, "/common/turn_lanes"))
make_turn_lane_description_views(index, "/common/turn_lanes"); {
m_lane_tupel_id_pairs = make_lane_data_view(index, "/common/turn_lanes"); std::tie(m_lane_description_offsets, m_lane_description_masks) =
make_turn_lane_description_views(index, "/common/turn_lanes");
m_lane_tuple_id_pairs = make_lane_data_view(index, "/common/turn_lanes");
}
m_turn_weight_penalties = make_turn_weight_view(index, "/common/turn_penalty"); m_turn_weight_penalties = make_turn_weight_view(index, "/common/turn_penalty");
m_turn_duration_penalties = make_turn_duration_view(index, "/common/turn_penalty"); m_turn_duration_penalties = make_turn_duration_view(index, "/common/turn_penalty");
@ -211,10 +256,12 @@ class ContiguousInternalMemoryDataFacadeBase : public BaseDataFacade
m_datasources = index.GetBlockPtr<extractor::Datasources>("/common/data_sources_names"); m_datasources = index.GetBlockPtr<extractor::Datasources>("/common/data_sources_names");
intersection_bearings_view = if (isIndexed(index, "/common/intersection_bearings"))
make_intersection_bearings_view(index, "/common/intersection_bearings"); {
intersection_bearings_view =
m_entry_class_table = make_entry_classes_view(index, "/common/entry_classes"); make_intersection_bearings_view(index, "/common/intersection_bearings");
m_entry_class_table = make_entry_classes_view(index, "/common/entry_classes");
}
std::tie(m_maneuver_overrides, m_maneuver_override_node_sequences) = std::tie(m_maneuver_overrides, m_maneuver_override_node_sequences) =
make_maneuver_overrides_views(index, "/common/maneuver_overrides"); make_maneuver_overrides_views(index, "/common/maneuver_overrides");
@ -305,7 +352,8 @@ class ContiguousInternalMemoryDataFacadeBase : public BaseDataFacade
osrm::guidance::TurnInstruction osrm::guidance::TurnInstruction
GetTurnInstructionForEdgeID(const EdgeID edge_based_edge_id) const override final GetTurnInstructionForEdgeID(const EdgeID edge_based_edge_id) const override final
{ {
return turn_data.GetTurnInstruction(edge_based_edge_id); CHECK_DATASET_DISABLED(turn_data, DATASET_TURN_DATA);
return turn_data->GetTurnInstruction(edge_based_edge_id);
} }
std::vector<RTreeLeaf> GetEdgesInBox(const util::Coordinate south_west, std::vector<RTreeLeaf> GetEdgesInBox(const util::Coordinate south_west,
@ -406,27 +454,32 @@ class ContiguousInternalMemoryDataFacadeBase : public BaseDataFacade
std::string_view GetNameForID(const NameID id) const override final std::string_view GetNameForID(const NameID id) const override final
{ {
return m_name_table.GetNameForID(id); CHECK_DATASET_DISABLED(m_name_table, DATASET_NAME_DATA);
return m_name_table->GetNameForID(id);
} }
std::string_view GetRefForID(const NameID id) const override final std::string_view GetRefForID(const NameID id) const override final
{ {
return m_name_table.GetRefForID(id); CHECK_DATASET_DISABLED(m_name_table, DATASET_NAME_DATA);
return m_name_table->GetRefForID(id);
} }
std::string_view GetPronunciationForID(const NameID id) const override final std::string_view GetPronunciationForID(const NameID id) const override final
{ {
return m_name_table.GetPronunciationForID(id); CHECK_DATASET_DISABLED(m_name_table, DATASET_NAME_DATA);
return m_name_table->GetPronunciationForID(id);
} }
std::string_view GetDestinationsForID(const NameID id) const override final std::string_view GetDestinationsForID(const NameID id) const override final
{ {
return m_name_table.GetDestinationsForID(id); CHECK_DATASET_DISABLED(m_name_table, DATASET_NAME_DATA);
return m_name_table->GetDestinationsForID(id);
} }
std::string_view GetExitsForID(const NameID id) const override final std::string_view GetExitsForID(const NameID id) const override final
{ {
return m_name_table.GetExitsForID(id); CHECK_DATASET_DISABLED(m_name_table, DATASET_NAME_DATA);
return m_name_table->GetExitsForID(id);
} }
std::string_view GetDatasourceName(const DatasourceID id) const override final std::string_view GetDatasourceName(const DatasourceID id) const override final
@ -459,46 +512,60 @@ class ContiguousInternalMemoryDataFacadeBase : public BaseDataFacade
util::guidance::BearingClass util::guidance::BearingClass
GetBearingClass(const NodeID node_based_node_id) const override final GetBearingClass(const NodeID node_based_node_id) const override final
{ {
return intersection_bearings_view.GetBearingClass(node_based_node_id); CHECK_DATASET_DISABLED(intersection_bearings_view, DATASET_INTERSECTION_BEARINGS);
return intersection_bearings_view->GetBearingClass(node_based_node_id);
} }
guidance::TurnBearing PreTurnBearing(const EdgeID edge_based_edge_id) const override final guidance::TurnBearing PreTurnBearing(const EdgeID edge_based_edge_id) const override final
{ {
return turn_data.GetPreTurnBearing(edge_based_edge_id); CHECK_DATASET_DISABLED(turn_data, DATASET_TURN_DATA);
return turn_data->GetPreTurnBearing(edge_based_edge_id);
} }
guidance::TurnBearing PostTurnBearing(const EdgeID edge_based_edge_id) const override final guidance::TurnBearing PostTurnBearing(const EdgeID edge_based_edge_id) const override final
{ {
return turn_data.GetPostTurnBearing(edge_based_edge_id); CHECK_DATASET_DISABLED(turn_data, DATASET_TURN_DATA);
return turn_data->GetPostTurnBearing(edge_based_edge_id);
} }
util::guidance::EntryClass GetEntryClass(const EdgeID edge_based_edge_id) const override final util::guidance::EntryClass GetEntryClass(const EdgeID edge_based_edge_id) const override final
{ {
auto entry_class_id = turn_data.GetEntryClassID(edge_based_edge_id); CHECK_DATASET_DISABLED(m_entry_class_table, DATASET_ENTRY_CLASS);
return m_entry_class_table.at(entry_class_id); CHECK_DATASET_DISABLED(turn_data, DATASET_TURN_DATA);
auto entry_class_id = turn_data->GetEntryClassID(edge_based_edge_id);
return m_entry_class_table->at(entry_class_id);
} }
bool HasLaneData(const EdgeID edge_based_edge_id) const override final bool HasLaneData(const EdgeID edge_based_edge_id) const override final
{ {
return turn_data.HasLaneData(edge_based_edge_id); CHECK_DATASET_DISABLED(turn_data, DATASET_TURN_DATA);
return turn_data->HasLaneData(edge_based_edge_id);
} }
util::guidance::LaneTupleIdPair util::guidance::LaneTupleIdPair
GetLaneData(const EdgeID edge_based_edge_id) const override final GetLaneData(const EdgeID edge_based_edge_id) const override final
{ {
CHECK_DATASET_DISABLED(turn_data, DATASET_TURN_DATA);
CHECK_DATASET_DISABLED(m_lane_tuple_id_pairs, DATASET_TURN_LANE_DATA);
BOOST_ASSERT(HasLaneData(edge_based_edge_id)); BOOST_ASSERT(HasLaneData(edge_based_edge_id));
return m_lane_tupel_id_pairs.at(turn_data.GetLaneDataID(edge_based_edge_id)); return m_lane_tuple_id_pairs->at(turn_data->GetLaneDataID(edge_based_edge_id));
} }
extractor::TurnLaneDescription extractor::TurnLaneDescription
GetTurnDescription(const LaneDescriptionID lane_description_id) const override final GetTurnDescription(const LaneDescriptionID lane_description_id) const override final
{ {
CHECK_DATASET_DISABLED(m_lane_description_offsets, DATASET_TURN_LANE_DATA);
CHECK_DATASET_DISABLED(m_lane_description_masks, DATASET_TURN_LANE_DATA);
if (lane_description_id == INVALID_LANE_DESCRIPTIONID) if (lane_description_id == INVALID_LANE_DESCRIPTIONID)
return {}; return {};
else else
return extractor::TurnLaneDescription( return extractor::TurnLaneDescription(
m_lane_description_masks.begin() + m_lane_description_offsets[lane_description_id], m_lane_description_masks->begin() +
m_lane_description_masks.begin() + m_lane_description_offsets->at(lane_description_id),
m_lane_description_offsets[lane_description_id + 1]); m_lane_description_masks->begin() +
m_lane_description_offsets->at(lane_description_id + 1));
} }
bool IsLeftHandDriving(const NodeID edge_based_node_id) const override final bool IsLeftHandDriving(const NodeID edge_based_node_id) const override final

3
include/engine/engine_config.hpp
View File

@ -29,9 +29,11 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
#define ENGINE_CONFIG_HPP #define ENGINE_CONFIG_HPP
#include "storage/storage_config.hpp" #include "storage/storage_config.hpp"
#include "osrm/datasets.hpp"
#include <boost/filesystem/path.hpp> #include <boost/filesystem/path.hpp>
#include <set>
#include <string> #include <string>
namespace osrm::engine namespace osrm::engine
@ -89,6 +91,7 @@ struct EngineConfig final
boost::filesystem::path memory_file; boost::filesystem::path memory_file;
bool use_mmap = true; bool use_mmap = true;
Algorithm algorithm = Algorithm::CH; Algorithm algorithm = Algorithm::CH;
std::vector<storage::FeatureDataset> disable_feature_dataset;
std::string verbosity; std::string verbosity;
std::string dataset_name; std::string dataset_name;
}; };

43
include/nodejs/node_osrm_support.hpp
View File

@ -205,9 +205,50 @@ inline engine_config_ptr argumentsToEngineConfig(const Napi::CallbackInfo &args)
} }
} }
auto disable_feature_dataset = params.Get("disable_feature_dataset");
if (disable_feature_dataset.IsArray())
{
Napi::Array datasets = disable_feature_dataset.As<Napi::Array>();
for (uint32_t i = 0; i < datasets.Length(); ++i)
{
Napi::Value dataset = datasets.Get(i);
if (!dataset.IsString())
{
ThrowError(args.Env(), "disable_feature_dataset list option must be a string");
return engine_config_ptr();
}
auto dataset_str = dataset.ToString().Utf8Value();
if (dataset_str == "ROUTE_GEOMETRY")
{
engine_config->disable_feature_dataset.push_back(
osrm::storage::FeatureDataset::ROUTE_GEOMETRY);
}
else if (dataset_str == "ROUTE_STEPS")
{
engine_config->disable_feature_dataset.push_back(
osrm::storage::FeatureDataset::ROUTE_STEPS);
}
else
{
ThrowError(
args.Env(),
"disable_feature_dataset array can include 'ROUTE_GEOMETRY', 'ROUTE_STEPS'.");
return engine_config_ptr();
}
}
}
else if (!disable_feature_dataset.IsUndefined())
{
ThrowError(args.Env(),
"disable_feature_dataset option must be an array and can include the string "
"values 'ROUTE_GEOMETRY', 'ROUTE_STEPS'.");
return engine_config_ptr();
}
if (!path.IsUndefined()) if (!path.IsUndefined())
{ {
engine_config->storage_config = osrm::StorageConfig(path.ToString().Utf8Value()); engine_config->storage_config = osrm::StorageConfig(path.ToString().Utf8Value(),
engine_config->disable_feature_dataset);
engine_config->use_shared_memory = false; engine_config->use_shared_memory = false;
} }

15
include/osrm/datasets.hpp Normal file
View File

@ -0,0 +1,15 @@
#ifndef DATASETS_HPP
#define DATASETS_HPP
namespace osrm::storage
{
enum class FeatureDataset
{
ROUTE_STEPS,
ROUTE_GEOMETRY,
};
} // namespace osrm::storage
#endif

3
include/osrm/error_codes.hpp
View File

@ -23,7 +23,8 @@ enum ErrorCode
FileIOError, FileIOError,
UnexpectedEndOfFile, UnexpectedEndOfFile,
IncompatibleDataset, IncompatibleDataset,
UnknownAlgorithm UnknownAlgorithm,
UnknownFeatureDataset
#ifndef NDEBUG #ifndef NDEBUG
// Leave this at the end. In debug mode, we assert that the size of // Leave this at the end. In debug mode, we assert that the size of
// this enum matches the number of messages we have documented, and __ENDMARKER__ // this enum matches the number of messages we have documented, and __ENDMARKER__

5
include/storage/io_config.hpp
View File

@ -35,6 +35,11 @@ struct IOConfig
return {base_path.string() + fileName}; return {base_path.string() + fileName};
} }
bool IsRequiredConfiguredInput(const std::string &fileName) const
{
return IsConfigured(fileName, required_input_files);
}
boost::filesystem::path base_path; boost::filesystem::path base_path;
protected: protected:

75
include/storage/storage_config.hpp
View File

@ -31,10 +31,61 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
#include <boost/filesystem/path.hpp> #include <boost/filesystem/path.hpp>
#include "storage/io_config.hpp" #include "storage/io_config.hpp"
#include "osrm/datasets.hpp"
#include <set>
namespace osrm::storage namespace osrm::storage
{ {
std::istream &operator>>(std::istream &in, FeatureDataset &datasets);
static std::vector<boost::filesystem::path>
GetRequiredFiles(const std::vector<storage::FeatureDataset> &disabled_feature_dataset)
{
std::set<boost::filesystem::path> required{
".osrm.datasource_names",
".osrm.ebg_nodes",
".osrm.edges",
".osrm.fileIndex",
".osrm.geometry",
".osrm.icd",
".osrm.maneuver_overrides",
".osrm.names",
".osrm.nbg_nodes",
".osrm.properties",
".osrm.ramIndex",
".osrm.timestamp",
".osrm.tld",
".osrm.tls",
".osrm.turn_duration_penalties",
".osrm.turn_weight_penalties",
};
for (const auto &to_disable : disabled_feature_dataset)
{
switch (to_disable)
{
case FeatureDataset::ROUTE_STEPS:
for (const auto &dataset : {".osrm.icd", ".osrm.tld", ".osrm.tls"})
{
required.erase(dataset);
}
break;
case FeatureDataset::ROUTE_GEOMETRY:
for (const auto &dataset :
{".osrm.edges", ".osrm.icd", ".osrm.names", ".osrm.tld", ".osrm.tls"})
{
required.erase(dataset);
}
break;
}
}
return std::vector<boost::filesystem::path>(required.begin(), required.end());
;
}
/** /**
* Configures OSRM's file storage paths. * Configures OSRM's file storage paths.
* *
@ -42,29 +93,17 @@ namespace osrm::storage
*/ */
struct StorageConfig final : IOConfig struct StorageConfig final : IOConfig
{ {
StorageConfig(const boost::filesystem::path &base) : StorageConfig()
StorageConfig(const boost::filesystem::path &base,
const std::vector<storage::FeatureDataset> &disabled_feature_datasets_ = {})
: StorageConfig(disabled_feature_datasets_)
{ {
IOConfig::UseDefaultOutputNames(base); IOConfig::UseDefaultOutputNames(base);
} }
StorageConfig() StorageConfig(const std::vector<storage::FeatureDataset> &disabled_feature_datasets_ = {})
: IOConfig( : IOConfig(
{".osrm.ramIndex", GetRequiredFiles(disabled_feature_datasets_),
".osrm.fileIndex",
".osrm.edges",
".osrm.geometry",
".osrm.turn_weight_penalties",
".osrm.turn_duration_penalties",
".osrm.datasource_names",
".osrm.ebg_nodes",
".osrm.names",
".osrm.nbg_nodes",
".osrm.timestamp",
".osrm.tls",
".osrm.tld",
".osrm.properties",
".osrm.icd",
".osrm.maneuver_overrides"},
{".osrm.hsgr", ".osrm.cells", ".osrm.cell_metrics", ".osrm.mldgr", ".osrm.partition"}, {".osrm.hsgr", ".osrm.cells", ".osrm.cell_metrics", ".osrm.mldgr", ".osrm.partition"},
{}) {})
{ {

31
include/util/exception.hpp
View File

@ -62,7 +62,7 @@ class exception : public std::exception
* user supplied bad data, etc). * user supplied bad data, etc).
*/ */
constexpr const std::array<const char *, 11> ErrorDescriptions = {{ constexpr const std::array<const char *, 12> ErrorDescriptions = {{
"", // Dummy - ErrorCode values start at 2 "", // Dummy - ErrorCode values start at 2
"", // Dummy - ErrorCode values start at 2 "", // Dummy - ErrorCode values start at 2
"Fingerprint did not match the expected value", // InvalidFingerprint "Fingerprint did not match the expected value", // InvalidFingerprint
@ -75,7 +75,8 @@ constexpr const std::array<const char *, 11> ErrorDescriptions = {{
// NOLINTNEXTLINE(bugprone-suspicious-missing-comma) // NOLINTNEXTLINE(bugprone-suspicious-missing-comma)
"The dataset you are trying to load is not " // IncompatibleDataset "The dataset you are trying to load is not " // IncompatibleDataset
"compatible with the routing algorithm you want to use.", // ...continued... "compatible with the routing algorithm you want to use.", // ...continued...
"Incompatible algorithm" // IncompatibleAlgorithm "Incompatible algorithm", // IncompatibleAlgorithm
"Unknown feature dataset" // UnknownFeatureDataset
}}; }};
#ifndef NDEBUG #ifndef NDEBUG
@ -84,6 +85,32 @@ static_assert(ErrorDescriptions.size() == ErrorCode::__ENDMARKER__,
"ErrorCode list and ErrorDescription lists are different sizes"); "ErrorCode list and ErrorDescription lists are different sizes");
#endif #endif
class DisabledDatasetException : public exception
{
public:
explicit DisabledDatasetException(const std::string &dataset_)
: exception(BuildMessage(dataset_)), dataset(dataset_)
{
}
const std::string &Dataset() const { return dataset; }
private:
// This function exists to 'anchor' the class, and stop the compiler from
// copying vtable and RTTI info into every object file that includes
// this header. (Caught by -Wweak-vtables under Clang.)
virtual void anchor() const override;
const std::string dataset;
static std::string BuildMessage(const std::string &dataset)
{
return "DisabledDatasetException: Your query tried to access the disabled dataset " +
dataset +
". Please check your configuration: "
"https://github.com/Project-OSRM/osrm-backend/wiki/Disabled-Datasets";
}
};
class RuntimeError : public exception class RuntimeError : public exception
{ {
using Base = exception; using Base = exception;

1
src/nodejs/node_osrm.cpp
View File

@ -74,6 +74,7 @@ Napi::Object Engine::Init(Napi::Env env, Napi::Object exports)
* Old behaviour: Path to a file on disk to store the memory using mmap. Current behaviour: setting this value is the same as setting `mmap_memory: true`. * Old behaviour: Path to a file on disk to store the memory using mmap. Current behaviour: setting this value is the same as setting `mmap_memory: true`.
* @param {Boolean} [options.mmap_memory] Map on-disk files to virtual memory addresses (mmap), rather than loading into RAM. * @param {Boolean} [options.mmap_memory] Map on-disk files to virtual memory addresses (mmap), rather than loading into RAM.
* @param {String} [options.path] The path to the `.osrm` files. This is mutually exclusive with setting {options.shared_memory} to true. * @param {String} [options.path] The path to the `.osrm` files. This is mutually exclusive with setting {options.shared_memory} to true.
* @param {Array} [options.disable_feature_dataset] Disables a feature dataset from being loaded into memory if not needed. Options: `ROUTE_STEPS`, `ROUTE_GEOMETRY`.
* @param {Number} [options.max_locations_trip] Max. locations supported in trip query (default: unlimited). * @param {Number} [options.max_locations_trip] Max. locations supported in trip query (default: unlimited).
* @param {Number} [options.max_locations_viaroute] Max. locations supported in viaroute query (default: unlimited). * @param {Number} [options.max_locations_viaroute] Max. locations supported in viaroute query (default: unlimited).
* @param {Number} [options.max_locations_distance_table] Max. locations supported in distance table query (default: unlimited). * @param {Number} [options.max_locations_distance_table] Max. locations supported in distance table query (default: unlimited).

98
src/server/request_handler.cpp
View File

@ -9,21 +9,16 @@
#include "util/log.hpp" #include "util/log.hpp"
#include "util/string_util.hpp" #include "util/string_util.hpp"
#include "util/timing_util.hpp" #include "util/timing_util.hpp"
#include "util/typedefs.hpp"
#include "engine/status.hpp" #include "engine/status.hpp"
#include "osrm/osrm.hpp" #include "osrm/osrm.hpp"
#include "util/json_container.hpp" #include "util/json_container.hpp"
#include <boost/iostreams/copy.hpp> #include <boost/iostreams/copy.hpp>
#include <boost/iostreams/filter/gzip.hpp>
#include <boost/iostreams/filtering_streambuf.hpp>
#include <ctime> #include <ctime>
#include <algorithm> #include <algorithm>
#include <iostream>
#include <iterator>
#include <string> #include <string>
#include <thread> #include <thread>
@ -36,6 +31,48 @@ void RequestHandler::RegisterServiceHandler(
service_handler = std::move(service_handler_); service_handler = std::move(service_handler_);
} }
void SendResponse(ServiceHandler::ResultT &result, http::reply &current_reply)
{
current_reply.headers.emplace_back("Access-Control-Allow-Origin", "*");
current_reply.headers.emplace_back("Access-Control-Allow-Methods", "GET");
current_reply.headers.emplace_back("Access-Control-Allow-Headers",
"X-Requested-With, Content-Type");
if (result.is<util::json::Object>())
{
current_reply.headers.emplace_back("Content-Type", "application/json; charset=UTF-8");
current_reply.headers.emplace_back("Content-Disposition",
"inline; filename=\"response.json\"");
util::json::render(current_reply.content, result.get<util::json::Object>());
}
else if (result.is<flatbuffers::FlatBufferBuilder>())
{
auto &buffer = result.get<flatbuffers::FlatBufferBuilder>();
current_reply.content.resize(buffer.GetSize());
std::copy(buffer.GetBufferPointer(),
buffer.GetBufferPointer() + buffer.GetSize(),
current_reply.content.begin());
current_reply.headers.emplace_back(
"Content-Type", "application/x-flatbuffers;schema=osrm.engine.api.fbresult");
}
else
{
BOOST_ASSERT(result.is<std::string>());
current_reply.content.resize(result.get<std::string>().size());
std::copy(result.get<std::string>().cbegin(),
result.get<std::string>().cend(),
current_reply.content.begin());
current_reply.headers.emplace_back("Content-Type", "application/x-protobuf");
}
// set headers
current_reply.headers.emplace_back("Content-Length",
std::to_string(current_reply.content.size()));
}
void RequestHandler::HandleRequest(const http::request &current_request, http::reply &current_reply) void RequestHandler::HandleRequest(const http::request &current_request, http::reply &current_reply)
{ {
if (!service_handler) if (!service_handler)
@ -96,43 +133,7 @@ void RequestHandler::HandleRequest(const http::request &current_request, http::r
std::to_string(position) + ": \"" + context + "\""; std::to_string(position) + ": \"" + context + "\"";
} }
current_reply.headers.emplace_back("Access-Control-Allow-Origin", "*"); SendResponse(result, current_reply);
current_reply.headers.emplace_back("Access-Control-Allow-Methods", "GET");
current_reply.headers.emplace_back("Access-Control-Allow-Headers",
"X-Requested-With, Content-Type");
if (result.is<util::json::Object>())
{
current_reply.headers.emplace_back("Content-Type", "application/json; charset=UTF-8");
current_reply.headers.emplace_back("Content-Disposition",
"inline; filename=\"response.json\"");
util::json::render(current_reply.content, result.get<util::json::Object>());
}
else if (result.is<flatbuffers::FlatBufferBuilder>())
{
auto &buffer = result.get<flatbuffers::FlatBufferBuilder>();
current_reply.content.resize(buffer.GetSize());
std::copy(buffer.GetBufferPointer(),
buffer.GetBufferPointer() + buffer.GetSize(),
current_reply.content.begin());
current_reply.headers.emplace_back(
"Content-Type", "application/x-flatbuffers;schema=osrm.engine.api.fbresult");
}
else
{
BOOST_ASSERT(result.is<std::string>());
current_reply.content.resize(result.get<std::string>().size());
std::copy(result.get<std::string>().cbegin(),
result.get<std::string>().cend(),
current_reply.content.begin());
current_reply.headers.emplace_back("Content-Type", "application/x-protobuf");
}
// set headers
current_reply.headers.emplace_back("Content-Length",
std::to_string(current_reply.content.size()));
if (!std::getenv("DISABLE_ACCESS_LOGGING")) if (!std::getenv("DISABLE_ACCESS_LOGGING"))
{ {
@ -168,6 +169,19 @@ void RequestHandler::HandleRequest(const http::request &current_request, http::r
<< request_string; << request_string;
} }
} }
catch (const util::DisabledDatasetException &e)
{
current_reply.status = http::reply::bad_request;
ServiceHandler::ResultT result = util::json::Object();
auto &json_result = result.get<util::json::Object>();
json_result.values["code"] = "DisabledDataset";
json_result.values["message"] = e.what();
SendResponse(result, current_reply);
util::Log(logWARNING) << "[disabled dataset error][" << tid << "] code: DisabledDataset_"
<< e.Dataset() << ", uri: " << current_request.uri;
}
catch (const std::exception &e) catch (const std::exception &e)
{ {
current_reply = http::reply::stock_reply(http::reply::internal_server_error); current_reply = http::reply::stock_reply(http::reply::internal_server_error);

100
src/storage/storage.cpp
View File

@ -294,17 +294,20 @@ std::vector<std::pair<bool, boost::filesystem::path>> Storage::GetStaticFiles()
std::vector<std::pair<bool, boost::filesystem::path>> files = { std::vector<std::pair<bool, boost::filesystem::path>> files = {
{IS_OPTIONAL, config.GetPath(".osrm.cells")}, {IS_OPTIONAL, config.GetPath(".osrm.cells")},
{IS_OPTIONAL, config.GetPath(".osrm.partition")}, {IS_OPTIONAL, config.GetPath(".osrm.partition")},
{IS_REQUIRED, config.GetPath(".osrm.icd")},
{IS_REQUIRED, config.GetPath(".osrm.properties")},
{IS_REQUIRED, config.GetPath(".osrm.nbg_nodes")},
{IS_REQUIRED, config.GetPath(".osrm.ebg_nodes")}, {IS_REQUIRED, config.GetPath(".osrm.ebg_nodes")},
{IS_REQUIRED, config.GetPath(".osrm.tls")},
{IS_REQUIRED, config.GetPath(".osrm.tld")},
{IS_REQUIRED, config.GetPath(".osrm.timestamp")},
{IS_REQUIRED, config.GetPath(".osrm.maneuver_overrides")}, {IS_REQUIRED, config.GetPath(".osrm.maneuver_overrides")},
{IS_REQUIRED, config.GetPath(".osrm.edges")}, {IS_REQUIRED, config.GetPath(".osrm.nbg_nodes")},
{IS_REQUIRED, config.GetPath(".osrm.names")}, {IS_REQUIRED, config.GetPath(".osrm.ramIndex")},
{IS_REQUIRED, config.GetPath(".osrm.ramIndex")}}; {IS_REQUIRED, config.GetPath(".osrm.properties")},
{IS_REQUIRED, config.GetPath(".osrm.timestamp")}};
for (const auto &file : {".osrm.edges", ".osrm.names", ".osrm.icd", ".osrm.tls", ".osrm.tld"})
{
if (config.IsRequiredConfiguredInput(file))
{
files.push_back({IS_REQUIRED, config.GetPath(file)});
}
}
for (const auto &file : files) for (const auto &file : files)
{ {
@ -394,12 +397,6 @@ void Storage::PopulateStaticData(const SharedDataIndex &index)
absolute_file_index_path.begin(), absolute_file_index_path.end(), file_index_path_ptr); absolute_file_index_path.begin(), absolute_file_index_path.end(), file_index_path_ptr);
} }
// Name data
{
auto name_table = make_name_table_view(index, "/common/names");
extractor::files::readNames(config.GetPath(".osrm.names"), name_table);
}
// Timestamp mark // Timestamp mark
{ {
auto timestamp_ref = make_timestamp_view(index, "/common/timestamp"); auto timestamp_ref = make_timestamp_view(index, "/common/timestamp");
@ -412,25 +409,39 @@ void Storage::PopulateStaticData(const SharedDataIndex &index)
} }
// Turn lane data // Turn lane data
if (config.IsRequiredConfiguredInput(".osrm.tld"))
{ {
auto turn_lane_data = make_lane_data_view(index, "/common/turn_lanes"); auto turn_lane_data = make_lane_data_view(index, "/common/turn_lanes");
extractor::files::readTurnLaneData(config.GetPath(".osrm.tld"), turn_lane_data); extractor::files::readTurnLaneData(config.GetPath(".osrm.tld"), turn_lane_data);
} }
// Turn lane descriptions // Turn lane descriptions
if (config.IsRequiredConfiguredInput(".osrm.tls"))
{ {
auto views = make_turn_lane_description_views(index, "/common/turn_lanes"); auto views = make_turn_lane_description_views(index, "/common/turn_lanes");
extractor::files::readTurnLaneDescriptions( extractor::files::readTurnLaneDescriptions(
config.GetPath(".osrm.tls"), std::get<0>(views), std::get<1>(views)); config.GetPath(".osrm.tls"), std::get<0>(views), std::get<1>(views));
} }
// Load edge-based nodes data // Load intersection data
if (config.IsRequiredConfiguredInput(".osrm.icd"))
{ {
auto node_data = make_ebn_data_view(index, "/common/ebg_node_data"); auto intersection_bearings_view =
extractor::files::readNodeData(config.GetPath(".osrm.ebg_nodes"), node_data); make_intersection_bearings_view(index, "/common/intersection_bearings");
auto entry_classes = make_entry_classes_view(index, "/common/entry_classes");
extractor::files::readIntersections(
config.GetPath(".osrm.icd"), intersection_bearings_view, entry_classes);
}
// Name data
if (config.IsRequiredConfiguredInput(".osrm.names"))
{
auto name_table = make_name_table_view(index, "/common/names");
extractor::files::readNames(config.GetPath(".osrm.names"), name_table);
} }
// Load original edge data // Load original edge data
if (config.IsRequiredConfiguredInput(".osrm.edges"))
{ {
auto turn_data = make_turn_data_view(index, "/common/turn_data"); auto turn_data = make_turn_data_view(index, "/common/turn_data");
@ -441,6 +452,12 @@ void Storage::PopulateStaticData(const SharedDataIndex &index)
config.GetPath(".osrm.edges"), turn_data, *connectivity_checksum_ptr); config.GetPath(".osrm.edges"), turn_data, *connectivity_checksum_ptr);
} }
// Load edge-based nodes data
{
auto node_data = make_ebn_data_view(index, "/common/ebg_node_data");
extractor::files::readNodeData(config.GetPath(".osrm.ebg_nodes"), node_data);
}
// Loading list of coordinates // Loading list of coordinates
{ {
auto views = make_nbn_data_view(index, "/common/nbn_data"); auto views = make_nbn_data_view(index, "/common/nbn_data");
@ -466,15 +483,6 @@ void Storage::PopulateStaticData(const SharedDataIndex &index)
metric_name = profile_properties_ptr->GetWeightName(); metric_name = profile_properties_ptr->GetWeightName();
} }
// Load intersection data
{
auto intersection_bearings_view =
make_intersection_bearings_view(index, "/common/intersection_bearings");
auto entry_classes = make_entry_classes_view(index, "/common/entry_classes");
extractor::files::readIntersections(
config.GetPath(".osrm.icd"), intersection_bearings_view, entry_classes);
}
if (boost::filesystem::exists(config.GetPath(".osrm.partition"))) if (boost::filesystem::exists(config.GetPath(".osrm.partition")))
{ {
auto mlp = make_partition_view(index, "/mld/multilevelpartition"); auto mlp = make_partition_view(index, "/mld/multilevelpartition");
@ -545,15 +553,18 @@ void Storage::PopulateUpdatableData(const SharedDataIndex &index)
contractor::files::readGraph( contractor::files::readGraph(
config.GetPath(".osrm.hsgr"), metrics, graph_connectivity_checksum); config.GetPath(".osrm.hsgr"), metrics, graph_connectivity_checksum);
auto turns_connectivity_checksum = if (config.IsRequiredConfiguredInput("osrm.edges"))
*index.GetBlockPtr<std::uint32_t>("/common/connectivity_checksum");
if (turns_connectivity_checksum != graph_connectivity_checksum)
{ {
throw util::exception( auto turns_connectivity_checksum =
"Connectivity checksum " + std::to_string(graph_connectivity_checksum) + " in " + *index.GetBlockPtr<std::uint32_t>("/common/connectivity_checksum");
config.GetPath(".osrm.hsgr").string() + " does not equal to checksum " + if (turns_connectivity_checksum != graph_connectivity_checksum)
std::to_string(turns_connectivity_checksum) + " in " + {
config.GetPath(".osrm.edges").string()); throw util::exception(
"Connectivity checksum " + std::to_string(graph_connectivity_checksum) +
" in " + config.GetPath(".osrm.hsgr").string() +
" does not equal to checksum " + std::to_string(turns_connectivity_checksum) +
" in " + config.GetPath(".osrm.edges").string());
}
} }
} }
@ -573,15 +584,18 @@ void Storage::PopulateUpdatableData(const SharedDataIndex &index)
customizer::files::readGraph( customizer::files::readGraph(
config.GetPath(".osrm.mldgr"), graph_view, graph_connectivity_checksum); config.GetPath(".osrm.mldgr"), graph_view, graph_connectivity_checksum);
auto turns_connectivity_checksum = if (config.IsRequiredConfiguredInput("osrm.edges"))
*index.GetBlockPtr<std::uint32_t>("/common/connectivity_checksum");
if (turns_connectivity_checksum != graph_connectivity_checksum)
{ {
throw util::exception( auto turns_connectivity_checksum =
"Connectivity checksum " + std::to_string(graph_connectivity_checksum) + " in " + *index.GetBlockPtr<std::uint32_t>("/common/connectivity_checksum");
config.GetPath(".osrm.hsgr").string() + " does not equal to checksum " + if (turns_connectivity_checksum != graph_connectivity_checksum)
std::to_string(turns_connectivity_checksum) + " in " + {
config.GetPath(".osrm.edges").string()); throw util::exception(
"Connectivity checksum " + std::to_string(graph_connectivity_checksum) +
" in " + config.GetPath(".osrm.mldgr").string() +
" does not equal to checksum " + std::to_string(turns_connectivity_checksum) +
" in " + config.GetPath(".osrm.edges").string());
}
} }
} }
} }

23
src/storage/storage_config.cpp Normal file
View File

@ -0,0 +1,23 @@
#include "osrm/datasets.hpp"
#include "osrm/exception.hpp"
#include "util/exception_utils.hpp"
#include <boost/algorithm/string/case_conv.hpp>
namespace osrm::storage
{
std::istream &operator>>(std::istream &in, FeatureDataset &datasets)
{
std::string token;
in >> token;
boost::to_lower(token);
if (token == "route_steps")
datasets = FeatureDataset::ROUTE_STEPS;
else if (token == "route_geometry")
datasets = FeatureDataset::ROUTE_GEOMETRY;
else
throw util::RuntimeError(token, ErrorCode::UnknownFeatureDataset, SOURCE_REF);
return in;
}
} // namespace osrm::storage

9
src/tools/routed.cpp
View File

@ -4,6 +4,7 @@
#include "util/meminfo.hpp" #include "util/meminfo.hpp"
#include "util/version.hpp" #include "util/version.hpp"
#include "osrm/datasets.hpp"
#include "osrm/engine_config.hpp" #include "osrm/engine_config.hpp"
#include "osrm/exception.hpp" #include "osrm/exception.hpp"
#include "osrm/osrm.hpp" #include "osrm/osrm.hpp"
@ -23,7 +24,6 @@
#include <exception> #include <exception>
#include <future> #include <future>
#include <iostream> #include <iostream>
#include <memory>
#include <new> #include <new>
#include <string> #include <string>
#include <thread> #include <thread>
@ -69,6 +69,7 @@ std::istream &operator>>(std::istream &in, EngineConfig::Algorithm &algorithm)
throw util::RuntimeError(token, ErrorCode::UnknownAlgorithm, SOURCE_REF); throw util::RuntimeError(token, ErrorCode::UnknownAlgorithm, SOURCE_REF);
return in; return in;
} }
} // namespace osrm::engine } // namespace osrm::engine
// overload validate for the double type to allow "unlimited" as an input // overload validate for the double type to allow "unlimited" as an input
@ -155,6 +156,10 @@ inline unsigned generateServerProgramOptions(const int argc,
value<EngineConfig::Algorithm>(&config.algorithm) value<EngineConfig::Algorithm>(&config.algorithm)
->default_value(EngineConfig::Algorithm::CH, "CH"), ->default_value(EngineConfig::Algorithm::CH, "CH"),
"Algorithm to use for the data. Can be CH, CoreCH, MLD.") // "Algorithm to use for the data. Can be CH, CoreCH, MLD.") //
("disable-feature-dataset",
value<std::vector<storage::FeatureDataset>>(&config.disable_feature_dataset)->multitoken(),
"Disables a feature dataset from being loaded into memory if not needed. Options: "
"ROUTE_STEPS, ROUTE_GEOMETRY") //
("max-viaroute-size", ("max-viaroute-size",
value<int>(&config.max_locations_viaroute)->default_value(500), value<int>(&config.max_locations_viaroute)->default_value(500),
"Max. locations supported in viaroute query") // "Max. locations supported in viaroute query") //
@ -276,7 +281,7 @@ try
if (!base_path.empty()) if (!base_path.empty())
{ {
config.storage_config = storage::StorageConfig(base_path); config.storage_config = storage::StorageConfig(base_path, config.disable_feature_dataset);
} }
if (!config.use_shared_memory && !config.storage_config.IsValid()) if (!config.use_shared_memory && !config.storage_config.IsValid())
{ {

17
src/tools/store.cpp
View File

@ -2,6 +2,7 @@
#include "storage/shared_memory.hpp" #include "storage/shared_memory.hpp"
#include "storage/shared_monitor.hpp" #include "storage/shared_monitor.hpp"
#include "storage/storage.hpp" #include "storage/storage.hpp"
#include "osrm/storage_config.hpp"
#include "osrm/exception.hpp" #include "osrm/exception.hpp"
#include "util/log.hpp" #include "util/log.hpp"
@ -9,6 +10,7 @@
#include "util/typedefs.hpp" #include "util/typedefs.hpp"
#include "util/version.hpp" #include "util/version.hpp"
#include <boost/algorithm/string/case_conv.hpp>
#include <boost/filesystem.hpp> #include <boost/filesystem.hpp>
#include <boost/program_options.hpp> #include <boost/program_options.hpp>
@ -100,7 +102,8 @@ bool generateDataStoreOptions(const int argc,
std::string &dataset_name, std::string &dataset_name,
bool &list_datasets, bool &list_datasets,
bool &list_blocks, bool &list_blocks,
bool &only_metric) bool &only_metric,
std::vector<storage::FeatureDataset> &disable_feature_dataset)
{ {
// declare a group of options that will be allowed only on command line // declare a group of options that will be allowed only on command line
boost::program_options::options_description generic_options("Options"); boost::program_options::options_description generic_options("Options");
@ -125,6 +128,12 @@ bool generateDataStoreOptions(const int argc,
boost::program_options::value<std::string>(&dataset_name)->default_value(""), boost::program_options::value<std::string>(&dataset_name)->default_value(""),
"Name of the dataset to load into memory. This allows having multiple datasets in memory " "Name of the dataset to load into memory. This allows having multiple datasets in memory "
"at the same time.") // "at the same time.") //
("disable-feature-dataset",
boost::program_options::value<std::vector<storage::FeatureDataset>>(
&disable_feature_dataset)
->multitoken(),
"Disables a feature dataset from being loaded into memory if not needed. Options: "
"ROUTE_STEPS, ROUTE_GEOMETRY") //
("list", ("list",
boost::program_options::value<bool>(&list_datasets) boost::program_options::value<bool>(&list_datasets)
->default_value(false) ->default_value(false)
@ -239,6 +248,7 @@ try
bool list_datasets = false; bool list_datasets = false;
bool list_blocks = false; bool list_blocks = false;
bool only_metric = false; bool only_metric = false;
std::vector<storage::FeatureDataset> disable_feature_dataset;
if (!generateDataStoreOptions(argc, if (!generateDataStoreOptions(argc,
argv, argv,
verbosity, verbosity,
@ -247,7 +257,8 @@ try
dataset_name, dataset_name,
list_datasets, list_datasets,
list_blocks, list_blocks,
only_metric)) only_metric,
disable_feature_dataset))
{ {
return EXIT_SUCCESS; return EXIT_SUCCESS;
} }
@ -260,7 +271,7 @@ try
return EXIT_SUCCESS; return EXIT_SUCCESS;
} }
storage::StorageConfig config(base_path); storage::StorageConfig config(base_path, disable_feature_dataset);
if (!config.IsValid()) if (!config.IsValid())
{ {
util::Log(logERROR) << "Config contains invalid file paths. Exiting!"; util::Log(logERROR) << "Config contains invalid file paths. Exiting!";

3
src/util/exception.cpp
View File

@ -1,5 +1,7 @@
#include "util/exception.hpp" #include "util/exception.hpp"
#include <utility>
// This function exists to 'anchor' the class, and stop the compiler from // This function exists to 'anchor' the class, and stop the compiler from
// copying vtable and RTTI info into every object file that includes // copying vtable and RTTI info into every object file that includes
// this header. (Caught by -Wweak-vtables under Clang.) // this header. (Caught by -Wweak-vtables under Clang.)
@ -16,4 +18,5 @@ namespace osrm::util
void exception::anchor() const {} void exception::anchor() const {}
void RuntimeError::anchor() const {} void RuntimeError::anchor() const {}
void DisabledDatasetException::anchor() const {}
} // namespace osrm::util } // namespace osrm::util

37
test/nodejs/index.js
View File

@ -163,6 +163,43 @@ test('constructor: throws on invalid custom limits', function(assert) {
}) })
}); });
}); });
test('constructor: throws on invalid disable_feature_dataset option', function(assert) {
assert.plan(1);
assert.throws(function() {
var osrm = new OSRM({
path: monaco_path,
disable_feature_dataset: ['NOT_EXIST'],
})
});
});
test('constructor: throws on non-array disable_feature_dataset', function(assert) {
assert.plan(1);
assert.throws(function() {
var osrm = new OSRM({
path: monaco_path,
disable_feature_dataset: 'ROUTE_GEOMETRY',
})
});
});
test('constructor: ok on valid disable_feature_dataset option', function(assert) {
assert.plan(1);
var osrm = new OSRM({
path: monaco_path,
disable_feature_dataset: ['ROUTE_GEOMETRY'],
});
assert.ok(osrm);
});
test('constructor: ok on multiple overlapping disable_feature_dataset options', function(assert) {
assert.plan(1);
var osrm = new OSRM({
path: monaco_path,
disable_feature_dataset: ['ROUTE_GEOMETRY', 'ROUTE_STEPS'],
});
assert.ok(osrm);
});
require('./route.js'); require('./route.js');
require('./trip.js'); require('./trip.js');

62
test/nodejs/match.js
View File

@ -398,3 +398,65 @@ test('match: match in Monaco with waypoints', function(assert) {
})); }));
}); });
}); });
test('match: throws on disabled geometry', function (assert) {
assert.plan(1);
var osrm = new OSRM({'path': data_path, 'disable_feature_dataset': ['ROUTE_GEOMETRY']});
var options = {
coordinates: three_test_coordinates.concat(three_test_coordinates),
};
osrm.match(options, function(err, route) {
console.log(err)
assert.match(err.message, /DisabledDatasetException/);
});
});
test('match: ok on disabled geometry', function (assert) {
assert.plan(2);
var osrm = new OSRM({'path': data_path, 'disable_feature_dataset': ['ROUTE_GEOMETRY']});
var options = {
steps: false,
overview: 'false',
annotations: false,
skip_waypoints: true,
coordinates: three_test_coordinates.concat(three_test_coordinates),
};
osrm.match(options, function(err, response) {
assert.ifError(err);
assert.equal(response.matchings.length, 1);
});
});
test('match: throws on disabled steps', function (assert) {
assert.plan(1);
var osrm = new OSRM({'path': data_path, 'disable_feature_dataset': ['ROUTE_STEPS']});
var options = {
steps: true,
coordinates: three_test_coordinates.concat(three_test_coordinates),
};
osrm.match(options, function(err, route) {
console.log(err)
assert.match(err.message, /DisabledDatasetException/);
});
});
test('match: ok on disabled steps', function (assert) {
assert.plan(8);
var osrm = new OSRM({'path': data_path, 'disable_feature_dataset': ['ROUTE_STEPS']});
var options = {
steps: false,
overview: 'simplified',
annotations: true,
coordinates: three_test_coordinates.concat(three_test_coordinates),
};
osrm.match(options, function(err, response) {
assert.ifError(err);
assert.ok(response.tracepoints);
assert.ok(response.matchings);
assert.equal(response.matchings.length, 1);
assert.ok(response.matchings[0].geometry, "the match has geometry");
assert.ok(response.matchings[0].legs, "the match has legs");
assert.notok(response.matchings[0].legs.every(l => { return l.steps.length > 0; }), 'every leg has steps');
assert.ok(response.matchings[0].legs.every(l => { return l.annotation;}), 'every leg has annotations');
});
});

37
test/nodejs/nearest.js
View File

@ -103,3 +103,40 @@ test('nearest: nearest in Monaco without motorways', function(assert) {
assert.equal(response.waypoints.length, 1); assert.equal(response.waypoints.length, 1);
}); });
}); });
test('nearest: throws on disabled geometry', function(assert) {
assert.plan(1);
var osrm = new OSRM({path: data_path, 'disable_feature_dataset': ['ROUTE_GEOMETRY']});
var options = {
coordinates: [two_test_coordinates[0]],
};
osrm.nearest(options, function(err, response) {
console.log(err)
assert.match(err.message, /DisabledDatasetException/);
});
});
test('nearest: ok on disabled geometry', function(assert) {
assert.plan(2);
var osrm = new OSRM({path: data_path, 'disable_feature_dataset': ['ROUTE_GEOMETRY']});
var options = {
coordinates: [two_test_coordinates[0]],
skip_waypoints: true,
};
osrm.nearest(options, function(err, response) {
assert.ifError(err);
assert.notok(response.waypoints);
});
});
test('nearest: ok on disabled steps', function(assert) {
assert.plan(2);
var osrm = new OSRM({path: data_path, 'disable_feature_dataset': ['ROUTE_STEPS']});
var options = {
coordinates: [two_test_coordinates[0]],
};
osrm.nearest(options, function(err, response) {
assert.ifError(err);
assert.equal(response.waypoints.length, 1);
});
});

64
test/nodejs/route.js
View File

@ -761,4 +761,66 @@ test('route: snapping parameter passed through OK', function(assert) {
assert.ifError(err); assert.ifError(err);
assert.equal(Math.round(route.routes[0].distance * 10), 1315); // Round it to nearest 0.1m to eliminate floating point comparison error assert.equal(Math.round(route.routes[0].distance * 10), 1315); // Round it to nearest 0.1m to eliminate floating point comparison error
}); });
}); });
test('route: throws on disabled geometry', function (assert) {
assert.plan(1);
var osrm = new OSRM({'path': monaco_path, 'disable_feature_dataset': ['ROUTE_GEOMETRY']});
var options = {
coordinates: three_test_coordinates,
};
osrm.route(options, function(err, route) {
console.log(err)
assert.match(err.message, /DisabledDatasetException/);
});
});
test('route: ok on disabled geometry', function (assert) {
assert.plan(2);
var osrm = new OSRM({'path': monaco_path, 'disable_feature_dataset': ['ROUTE_GEOMETRY']});
var options = {
steps: false,
overview: 'false',
annotations: false,
skip_waypoints: true,
coordinates: three_test_coordinates,
};
osrm.route(options, function(err, response) {
assert.ifError(err);
assert.equal(response.routes.length, 1);
});
});
test('route: throws on disabled steps', function (assert) {
assert.plan(1);
var osrm = new OSRM({'path': monaco_path, 'disable_feature_dataset': ['ROUTE_STEPS']});
var options = {
steps: true,
coordinates: three_test_coordinates,
};
osrm.route(options, function(err, route) {
console.log(err)
assert.match(err.message, /DisabledDatasetException/);
});
});
test('route: ok on disabled steps', function (assert) {
assert.plan(8);
var osrm = new OSRM({'path': monaco_path, 'disable_feature_dataset': ['ROUTE_STEPS']});
var options = {
steps: false,
overview: 'simplified',
annotations: true,
coordinates: three_test_coordinates,
};
osrm.route(options, function(err, response) {
assert.ifError(err);
assert.ok(response.waypoints);
assert.ok(response.routes);
assert.equal(response.routes.length, 1);
assert.ok(response.routes[0].geometry, "the route has geometry");
assert.ok(response.routes[0].legs, "the route has legs");
assert.notok(response.routes[0].legs.every(l => { return l.steps.length > 0; }), 'every leg has steps');
assert.ok(response.routes[0].legs.every(l => { return l.annotation;}), 'every leg has annotations');
});
});

42
test/nodejs/table.js
View File

@ -19,7 +19,7 @@ test('table: flatbuffer format', function(assert) {
assert.ok(table instanceof Buffer); assert.ok(table instanceof Buffer);
const fb = FBResult.getRootAsFBResult(new flatbuffers.ByteBuffer(table)); const fb = FBResult.getRootAsFBResult(new flatbuffers.ByteBuffer(table));
assert.ok(fb.table()); assert.ok(fb.table());
}); });
}); });
@ -369,3 +369,43 @@ tables.forEach(function(annotation) {
}); });
}); });
test('table: throws on disabled geometry', function (assert) {
assert.plan(1);
var osrm = new OSRM({'path': data_path, 'disable_feature_dataset': ['ROUTE_GEOMETRY']});
var options = {
coordinates: [three_test_coordinates[0], three_test_coordinates[1]],
};
osrm.table(options, function(err, table) {
console.log(err)
assert.match(err.message, /DisabledDatasetException/);
});
});
test('table: ok on disabled geometry', function (assert) {
assert.plan(4);
var osrm = new OSRM({'path': data_path, 'disable_feature_dataset': ['ROUTE_GEOMETRY']});
var options = {
coordinates: [three_test_coordinates[0], three_test_coordinates[1]],
skip_waypoints: true
};
osrm.table(options, function(err, table) {
assert.ifError(err);
assert.ok(table.durations, 'distances table result should exist');
assert.notok(table.sources)
assert.notok(table.destinations)
});
});
test('table: ok on disabled steps', function (assert) {
assert.plan(4);
var osrm = new OSRM({'path': data_path, 'disable_feature_dataset': ['ROUTE_STEPS']});
var options = {
coordinates: [three_test_coordinates[0], three_test_coordinates[1]],
};
osrm.table(options, function(err, table) {
assert.ifError(err);
assert.ok(table.durations, 'distances table result should exist');
assert.ok(table.sources.length, 2)
assert.ok(table.destinations.length, 2)
});
});

17
test/nodejs/tile.js
View File

@ -23,3 +23,20 @@ test.test('tile interface pre-conditions', function(assert) {
assert.throws(function() { osrm.tile(17059, 11948, 15, function(err, result) {}) }, /must be an array \[x, y, z\]/); assert.throws(function() { osrm.tile(17059, 11948, 15, function(err, result) {}) }, /must be an array \[x, y, z\]/);
assert.throws(function() { osrm.tile([17059, 11948, -15], function(err, result) {}) }, /must be unsigned/); assert.throws(function() { osrm.tile([17059, 11948, -15], function(err, result) {}) }, /must be unsigned/);
}); });
test.test('tile fails to load with geometry disabled', function(assert) {
assert.plan(1);
var osrm = new OSRM({'path': data_path, 'disable_feature_dataset': ['ROUTE_GEOMETRY']});
osrm.tile(tile.at, function(err, result) {
console.log(err)
assert.match(err.message, /DisabledDatasetException/);
});
});
test.test('tile ok with steps disabled', function(assert) {
assert.plan(2);
var osrm = new OSRM({'path': data_path, 'disable_feature_dataset': ['ROUTE_STEPS']});
osrm.tile(tile.at, function(err, result) {
assert.ifError(err);
assert.equal(result.length, tile.size);
});
});

62
test/nodejs/trip.js
View File

@ -368,3 +368,65 @@ test('trip: trip in Monaco without motorways', function(assert) {
}); });
}); });
test('trip: throws on disabled geometry', function (assert) {
assert.plan(1);
var osrm = new OSRM({'path': data_path, 'disable_feature_dataset': ['ROUTE_GEOMETRY']});
var options = {
coordinates: three_test_coordinates.concat(three_test_coordinates),
};
osrm.trip(options, function(err, route) {
console.log(err)
assert.match(err.message, /DisabledDatasetException/);
});
});
test('trip: ok on disabled geometry', function (assert) {
assert.plan(2);
var osrm = new OSRM({'path': data_path, 'disable_feature_dataset': ['ROUTE_GEOMETRY']});
var options = {
steps: false,
overview: 'false',
annotations: false,
skip_waypoints: true,
coordinates: three_test_coordinates.concat(three_test_coordinates),
};
osrm.trip(options, function(err, response) {
assert.ifError(err);
assert.equal(response.trips.length, 1);
});
});
test('trip: throws on disabled steps', function (assert) {
assert.plan(1);
var osrm = new OSRM({'path': data_path, 'disable_feature_dataset': ['ROUTE_STEPS']});
var options = {
steps: true,
coordinates: three_test_coordinates.concat(three_test_coordinates),
};
osrm.trip(options, function(err, route) {
console.log(err)
assert.match(err.message, /DisabledDatasetException/);
});
});
test('trip: ok on disabled steps', function (assert) {
assert.plan(8);
var osrm = new OSRM({'path': data_path, 'disable_feature_dataset': ['ROUTE_STEPS']});
var options = {
steps: false,
overview: 'simplified',
annotations: true,
coordinates: three_test_coordinates.concat(three_test_coordinates),
};
osrm.trip(options, function(err, response) {
assert.ifError(err);
assert.ok(response.waypoints);
assert.ok(response.trips);
assert.equal(response.trips.length, 1);
assert.ok(response.trips[0].geometry, "trip has geometry");
assert.ok(response.trips[0].legs, "trip has legs");
assert.notok(response.trips[0].legs.every(l => { return l.steps.length > 0; }), 'every leg has steps');
assert.ok(response.trips[0].legs.every(l => { return l.annotation;}), 'every leg has annotations');
});
});