add changelog

With @karenzshea's name / ref split (ref. #2857) in master we want to
make use of it and reduce `NewName` instructions when ever possible.
This is a first step towards #2744 by using the already existing name
change heuristic from the extractor now in post-processing as well.

Limitations: at the moment we don't have the `SuffixTable` in
post-processing; this would require us serializing and subsequently
deserializing the table, passing it through from the profiles to the
API.

.travis.yml

@@ -13,6 +13,7 @@ notifications:
 branches:
   only:
     - master
+    - 5.4
 
 cache:
   ccache: true

CHANGELOG.md

@@ -1,5 +1,25 @@
+# 5.4.0-rc.5
+  - Changes from 5.4.0-rc.4
+    - Guidance
+      - Improved detection of obvious name changes
+    - Profiles
+      - The defautl profile for car now excludes HOV only routes in navigation by default
+    - Bugfixes
+      - Fixed a bug that could result in endless loops in combination with sliproads
+
+# 5.4.0-rc.4
+  - Changes from 5.4.0-rc.3
+    - Bugfixes
+      - Fixed a bug where roundabout intersections could result in breaking assertions when immediately exited
+
+# 5.4.0-rc.3
+  - Changes from 5.4.0-rc.2
+    - Bugfixes
+      - BREAKING: Fixed a bug where some roads could be falsly identified as sliproadsi This change requires reprocessing datasets with osrm-extract and osrm-contract
+      - BREAKING: Fixed a bug that resulted in false names/ref/destination/pronunciation This change requires reprocessing datasets with osrm-extract and osrm-contract
+
 # 5.4.0
-  Changes from 5.3.0
+  - Changes from 5.3.0
     - Profiles
       - includes library guidance.lua that offers preliminary configuration on guidance.
       - added left_hand_driving flag in global profile properties
@@ -20,9 +40,14 @@
       - Fixed an issue that could emit `invalid` as instruction when ending on a sliproad after a traffic-light
       - Fixed an issue that would detect turning circles as sliproads
       - Fixed a bug where post-processing instructions (e.g. left + left -> uturn) could result in false pronunciations
+      - Fixes a bug where a bearing range of zero would cause exhaustive graph traversals
+      - Fixes a bug where certain looped geometries could cause an infinite loop during extraction
+
+    - Infrastructure:
+      - Adds a feature to limit results in nearest service with a default of 100 in `osrm-routed`
 
 # 5.3.0
-  Changes from 5.3.0-rc.3
+  - Changes from 5.3.0-rc.3
     - Guidance
       - Only announce `use lane` on required turns (not using all lanes to go straight)
       - Moved `lanes` to the intersection objects. This is BREAKING in relation to other Release Candidates but not with respect to other releases.
@@ -30,7 +55,7 @@
       - Fix BREAKING: bug that could result in failure to load 'osrm.icd' files. This breaks the dataformat
       - Fix: bug that results in segfaults when `use lane` instructions are suppressed
 
-  Changes form 5.2.7
+  - Changes form 5.2.7
     - API
       - Introduces new `TurnType` in the form of `use lane`. The type indicates that you have to stick to a lane without turning
       - Introduces `lanes` to the `Intersection` object. The lane data contains both the markings at the intersection and a flag indicating if they can be chosen for the next turn
@@ -54,7 +79,7 @@
       - Fix devide by zero on updating speed data using osrm-contract
 
 # 5.3.0 RC3
-  Changes from 5.3.0-rc.2
+  - Changes from 5.3.0-rc.2
     - Guidance
       - Improved detection of obvious turns
       - Improved turn lane detection

CMakeLists.txt

@@ -9,7 +9,7 @@ endif()
 
 project(OSRM C CXX)
 set(OSRM_VERSION_MAJOR 5)
-set(OSRM_VERSION_MINOR 3)
+set(OSRM_VERSION_MINOR 4)
 set(OSRM_VERSION_PATCH 0)
 
 # these two functions build up custom variables:

cucumber.js

@@ -1,10 +1,8 @@
 module.exports = {
-    default: '--require features --tags ~@stress --tags ~@todo',
-    verify: '--require features --tags ~@todo --tags ~@bug --tags ~@stress -f progress',
-    jenkins: '--require features --tags ~@todo --tags ~@bug --tags ~@stress --tags ~@options -f progress',
-    bugs: '--require features --tags @bug',
-    todo: '--require features --tags @todo',
-    all: '--require features'
+    default: '--strict --tags ~@stress --tags ~@todo --require features/support --require features/step_definitions',
+    verify: '--strict --tags ~@stress --tags ~@todo -f progress --require features/support --require features/step_definitions',
+    todo: '--strict --tags @todo --require features/support --require features/step_definitions',
+    all: '--strict --require features/support --require features/step_definitions'
 }
 
 

docs/http.md

@@ -30,11 +30,11 @@ http://{server}/{service}/{version}/{profile}/{coordinates}[.{format}]?option=va
   
     | Service     |           Description                                     |
     |-------------|-----------------------------------------------------------|
-    | [`route`](#service-route)     | shortest path between given coordinates                   |
+    | [`route`](#service-route)     | fastest path between given coordinates                   |
     | [`nearest`](#service-nearest)   | returns the nearest street segment for a given coordinate |
     | [`table`](#service-table)     | computes distance tables for given coordinates            |
     | [`match`](#service-match)     | matches given coordinates to the road network             |
-    | [`trip`](#service-trip)      | Compute the shortest round trip between given coordinates |
+    | [`trip`](#service-trip)      | Compute the fastest round trip between given coordinates |
     | [`tile`](#service-tile)      | Return vector tiles containing debugging info             |
   
 - `version`: Version of the protocol implemented by the service.
@@ -302,7 +302,7 @@ All other fields might be undefined.
 ## Service `trip`
 
 The trip plugin solves the Traveling Salesman Problem using a greedy heuristic (farthest-insertion algorithm).
-The returned path does not have to be the shortest path, as TSP is NP-hard it is only an approximation.
+The returned path does not have to be the fastest path, as TSP is NP-hard it is only an approximation.
 Note that if the input coordinates can not be joined by a single trip (e.g. the coordinates are on several disconnected islands)
 multiple trips for each connected component are returned.
 

features/car/access.feature

@@ -156,6 +156,25 @@ Feature: Car - Restricted access
             | primary | yes        | x     |
             | primary | no         | x     |
 
+    Scenario: Car - a way with all lanes HOV-designated is inaccessible by default (similar to hov=designated)
+        Then routability should be
+            | highway | hov:lanes:forward      | hov:lanes:backward     | hov:lanes              | oneway | forw | backw |
+            | primary | designated             | designated             |                        |        |      |       |
+            | primary |                        | designated             |                        |        | x    |       |
+            | primary | designated             |                        |                        |        |      | x     |
+            | primary | designated\|designated | designated\|designated |                        |        |      |       |
+            | primary | designated\|no         | designated\|no         |                        |        | x    | x     |
+            | primary | yes\|no                | yes\|no                |                        |        | x    | x     |
+            | primary |                        |                        |                        |        | x    | x     |
+            | primary | designated             |                        |                        | -1     |      |       |
+            | primary |                        | designated             |                        | -1     |      | x     |
+            | primary |                        |                        | designated             | yes    |      |       |
+            | primary |                        |                        | designated             | -1     |      |       |
+            | primary |                        |                        | designated\|designated | yes    |      |       |
+            | primary |                        |                        | designated\|designated | -1     |      |       |
+            | primary |                        |                        | designated\|yes        | yes    | x    |       |
+            | primary |                        |                        | designated\|no         | -1     |      | x     |
+
      Scenario: Car - these toll roads always work
         Then routability should be
             | highway | toll        | bothw |

features/car/names.feature

@@ -35,6 +35,23 @@ Feature: Car - Street names in instructions
             | a    | d  | My Way,My Way      | ,meyeway             | ,A1   |
             | 1    | c  | Your Way,Your Way  | yourewaye,yourewaye  | ,     |
 
+    # See #2860
+    Scenario: Car - same street name but different pronunciation
+        Given the node map
+            | a | b | c |
+            |   | d |   |
+            |   | e |   |
+
+        And the ways
+            | nodes | name       | name:pronunciation |
+            | abc   | Houston St | hew-stun           |
+            | bde   | Houston St | how-stun           |
+
+        When I route I should get
+            | from | to | route                            | pronunciations             |
+            | a    | c  | Houston St,Houston St            | hew-stun,hew-stun          |
+            | a    | e  | Houston St,Houston St,Houston St | hew-stun,how-stun,how-stun |
+
     @todo
     Scenario: Car - Use way type to describe unnamed ways
         Given the node map

features/car/traffic_speeds.feature

@@ -25,7 +25,7 @@ Feature: Traffic - speeds
             | fb    | primary |
         Given the profile "testbot"
         Given the extract extra arguments "--generate-edge-lookup"
-        Given the contract extra arguments "--segment-speed-file speeds.csv"
+        Given the contract extra arguments "--segment-speed-file {speeds_file}"
         Given the speed file
         """
         1,2,0
@@ -69,7 +69,7 @@ Feature: Traffic - speeds
             | fb    | primary |
         Given the profile "testbot"
         Given the extract extra arguments "--generate-edge-lookup"
-        Given the contract extra arguments "--segment-speed-file speeds.csv"
+        Given the contract extra arguments "--segment-speed-file {speeds_file}"
         Given the speed file
         """
         1,2,0
@@ -112,7 +112,6 @@ Feature: Traffic - speeds
             | fb    | primary |
         Given the profile "testbot"
         Given the extract extra arguments "--generate-edge-lookup"
-        Given the contract extra arguments "--segment-speed-file speeds.csv"
         Given the speed file
         """
         1,2,-10
@@ -123,6 +122,6 @@ Feature: Traffic - speeds
         4,1,-5
         """
         And the data has been extracted
-        When I run "osrm-contract --segment-speed-file speeds.csv {extracted_base}.osrm"
+        When I try to run "osrm-contract --segment-speed-file {speeds_file} {processed_file}"
         And stderr should contain "malformed"
-        And it should exit with code not 0
+        And it should exit with an error

features/car/traffic_turn_penalties.feature

@@ -58,7 +58,7 @@ Feature: Traffic - turn penalties
             8,11,12,23
             1,4,5,-0.2
             """
-        And the contract extra arguments "--turn-penalty-file penalties.csv"
+        And the contract extra arguments "--turn-penalty-file {penalties_file}"
         When I route I should get
             | from | to | route                 | speed   | time      |
             | a    | h  | ad,dhk,dhk            | 63 km/h | 11.5s +-1 |
@@ -81,7 +81,7 @@ Feature: Traffic - turn penalties
                                                                               # double left - hdc penalty ever so slightly higher than imn; forces all the way around
 
     Scenario: Too-negative penalty clamps, but does not fail
-        Given the contract extra arguments "--turn-penalty-file penalties.csv"
+        Given the contract extra arguments "--turn-penalty-file {penalties_file}"
         And the profile "testbot"
         And the turn penalty file
             """

features/guidance/advanced-lanes.feature

@@ -103,7 +103,7 @@ Feature: Turn Lane Guidance
             | a,j       | in,turn,other,other  | depart,turn right,turn left,arrive  | ,,left:true right:false,               |
             | a,i       | in,turn,other,other  | depart,turn right,turn right,arrive | ,,left:false right:true,               |
 
-    @todo @bug @2654 @none
+    @todo @2654 @none
     #https://github.com/Project-OSRM/osrm-backend/issues/2645
     #http://www.openstreetmap.org/export#map=19/52.56054/13.32152
     Scenario: Kurt-Schuhmacher-Damm
@@ -131,7 +131,7 @@ Feature: Turn Lane Guidance
             | a,f       | ,ksd,ksd | depart,turn left,arrive  | ,left:true none:true right:false, |
             | a,i       | ,ksd,ksd | depart,turn right,arrive | ,left:false none:true right:true, |
 
-    @todo @bug @2650 @sliproads
+    @todo @2650 @sliproads
     #market and haight in SF, restricted turn
     #http://www.openstreetmap.org/#map=19/37.77308/-122.42238
     Scenario: Market/Haight without Through Street
@@ -177,7 +177,7 @@ Feature: Turn Lane Guidance
             | a,j       | ghough,market,market | depart,turn left,arrive            | ,none:true straight:false straight:false straight:false, |
             | a,f       | ghough,ghough,ghough | depart,continue slight left,arrive | ,none:true straight:true straight:false straight:false,  |
 
-    @todo @2650 @bug @sliproads
+    @todo @2650 @sliproads
     #market and haight in SF, unrestricted
     #http://www.openstreetmap.org/#map=19/37.77308/-122.42238
     Scenario: Market/Haight without Through Street
@@ -218,3 +218,19 @@ Feature: Turn Lane Guidance
             | a,h       | ghough,market,market | depart,turn slight right,arrive    | ,none:false straight:false straight:true straight:true,  |
             | a,j       | ghough,market,market | depart,turn left,arrive            | ,none:true straight:false straight:false straight:false, |
             | a,f       | ghough,ghough,ghough | depart,continue slight left,arrive | ,none:true straight:true straight:false straight:false,  |
+
+    Scenario: Check sliproad handler loop's exit condition, Issue #2896
+      # http://www.openstreetmap.org/way/198481519
+        Given the node locations
+            | node | lat        | lon         |
+            | a    | 7.6125350  | 126.5708309 |
+            | b    | 7.6125156  | 126.5707219 |
+            | c    | 7.6125363  | 126.5708337 |
+
+        And the ways
+            | nodes | name |
+            | cbac  |      |
+
+        When I route I should get
+            | from | to | route | turns         |
+            | a    | c  | ,     | depart,arrive |

features/guidance/anticipate-lanes.feature

@@ -422,7 +422,7 @@ Feature: Turn Lane Guidance
                | waypoints | route               | turns                                          | lanes                                                                                                                                                             |
                | a,e       | main,main,main,main | depart,use lane straight,continue right,arrive | ,left:false straight:false straight:false straight:false straight:true straight:true right:false,straight:false straight:false right:false right:true right:true, |
 
-    @anticipate @todo @bug @2661
+    @anticipate @todo @2661
     Scenario: Anticipate with lanes in roundabout: roundabouts as the unit of anticipation
         Given the node map
             |   |   | e |   |   |
@@ -667,7 +667,7 @@ Feature: Turn Lane Guidance
             | a,f       | abc,bdeh,feg,feg | depart,turn right,turn right,arrive | ,none:false none:false right:false right:true,left:false none:false none:false right:true, |
 
     @anticipate
-    Scenario: Tripple Right keeping Left
+    Scenario: Triple Right keeping Left
         Given the node map
             | a |   |   |   | b |   | i |
             |   |   |   |   |   |   |   |

features/guidance/dedicated-turn-roads.feature

@@ -264,3 +264,88 @@ Feature: Slipways and Dedicated Turn Lanes
      When I route I should get
             | waypoints | bearings     | route           | turns         |
             | b,a       | 90,10 270,10 | circled,circled | depart,arrive |
+
+    #http://www.openstreetmap.org/#map=19/38.90597/-77.01276
+    Scenario: Don't falsly classify as sliproads
+        Given the node map
+            # 0   1   2   3   4   5   6   7   8   9  10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27  28  29  30
+            |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   | j |   |   |   |   |   |   |   |
+            | a | b |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   | c |   |   |   |   |   |   | d |
+            |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |
+            |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |
+            |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |
+            |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |
+            |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |
+            |   |   |   |   | e |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |
+            |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |
+            |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |
+            |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |
+            |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |
+            |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |
+            |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |
+            |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |
+            |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |
+            |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |
+            |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |
+            |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |
+            |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |
+            |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |
+            |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |
+            |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   | 1 |   |   |   |   |   |   |   |
+            |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |
+            |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   | f |   |   |   |   | g |   |   |   |   |   |   |   |
+            |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |
+            |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   | i |   |   |   |   |   |   | h |
+
+        And the ways
+            | nodes | name       | highway   | oneway | maxspeed |
+            | abcd  | new york   | primary   | yes    | 35       |
+            | befgh | m street   | secondary | yes    | 35       |
+            | igcj  | 1st street | tertiary  | no     | 20       |
+
+        And the nodes
+            | node | highway         |
+            | c    | traffic_signals |
+            | g    | traffic_signals |
+
+        When I route I should get
+            | waypoints | route                                   | turns                              | #                                    |
+            | a,d       | new york,new york                       | depart,arrive                      | this is the sinatra route            |
+            | a,j       | new york,1st street,1st street          | depart,turn left,arrive            |                                      |
+            | a,1       | new york,m street,1st street,1st street | depart,turn right,turn left,arrive | this can false be seen as a sliproad |
+
+    # Merging into degree two loop on dedicated turn detection / 2927
+    Scenario: Turn Instead of Ramp
+        Given the node map
+            |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   | f |
+            |   |   |   |   | g |   |   |   |   |   | h |   |   |   |   |   |
+            |   |   |   |   |   |   |   |   |   |   |   |   | d |   |   | e |
+            | i |   |   |   | c |   |   |   |   |   | j |   |   |   |   |   |
+            |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |
+            |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |
+            |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |
+            |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |
+            |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |
+            |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |
+            |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |
+            |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |
+            |   |   | b |   |   |   |   |   |   |   |   |   |   |   |   |   |
+            |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |
+            |   |   | a |   |   |   |   |   |   |   |   |   |   |   |   |   |
+
+        And the ways
+            | nodes | highway | name | oneway |
+            | abi   | primary | road | yes    |
+            | bcjd  | primary | loop | yes    |
+            | dhgf  | primary | loop | yes    |
+            | fed   | primary | loop | yes    |
+
+        And the nodes
+            | node | highway         |
+            | g    | traffic_signals |
+            | c    | traffic_signals |
+
+        # We don't actually care about routes here, this is all about endless loops in turn discovery
+        When I route I should get
+            | waypoints | route          | turns                          |
+            | a,i       | road,road,road | depart,fork slight left,arrive |

features/guidance/new-name.feature

@@ -164,3 +164,172 @@ Feature: New-Name Instructions
             | waypoints | route                    | turns                           |
             | a,e       | name,with-name,with-name | depart,new name straight,arrive |
             | b,e       | with-name,with-name      | depart,arrive                   |
+
+    Scenario: Both Name and Ref Empty
+        Given the node map
+            | a |  | b |  | c |
+
+        And the ways
+            | nodes | name | ref |
+            | ab    |      |     |
+            | bc    |      |     |
+
+        When I route I should get
+            | waypoints | route | turns         |
+            | a,c       | ,     | depart,arrive |
+
+    Scenario: Same Name, Ref Extended
+        Given the node map
+            | a |  | b |  | c |
+
+        And the ways
+            | nodes | name | ref   |
+            | ab    | A    | B1    |
+            | bc    | C    | B1;B2 |
+
+        When I route I should get
+            | waypoints | route | turns                           |
+            | a,c       | A,C,C | depart,new name straight,arrive |
+
+    Scenario: Same Name, Ref Removed
+        Given the node map
+            | a |  | b |  | c |
+
+        And the ways
+            | nodes | name | ref   |
+            | ab    | A    | B1;B2 |
+            | bc    | C    | B1    |
+
+        When I route I should get
+            | waypoints | route | turns                           |
+            | a,c       | A,C,C | depart,new name straight,arrive |
+
+    Scenario: Name Removed, Ref Extended
+        Given the node map
+            | a |  | b |  | c |
+
+        And the ways
+            | nodes | name | ref   |
+            | ab    | A    | B1    |
+            | bc    |      | B1;B2 |
+
+        When I route I should get
+            | waypoints | route | turns         |
+            | a,c       | A,    | depart,arrive |
+
+    Scenario: Name Added, Ref Removed
+        Given the node map
+            | a |  | b |  | c |
+
+        And the ways
+            | nodes | name | ref   |
+            | ab    |      | B1    |
+            | bc    | A    |       |
+
+        When I route I should get
+            | waypoints | route | turns                           |
+            | a,c       | ,A,A  | depart,new name straight,arrive |
+
+    Scenario: Prefix Change
+        Given the node map
+            | a |   |   |   | b |   |   |   | c |
+
+        And the ways
+            | nodes | name                     | ref   | highway  |
+            | ab    | North Central Expressway | US 75 | motorway |
+            | bc    | Central Expressway       | US 75 | motorway |
+
+        When I route I should get
+            | waypoints | route                                                          | turns                           |
+            | a,c       | North Central Expressway,Central Expressway,Central Expressway | depart,new name straight,arrive |
+
+    Scenario: Prefix Change
+        Given the node map
+            | a |   |   |   | b |   |   |   | c |
+
+        And the ways
+            | nodes | name                     | ref   | highway  |
+            | ba    | North Central Expressway | US 75 | motorway |
+            | cb    | Central Expressway       | US 75 | motorway |
+
+        When I route I should get
+            | waypoints | route                                                                | turns                           |
+            | c,a       | Central Expressway,North Central Expressway,North Central Expressway | depart,new name straight,arrive |
+
+    Scenario: No Name, Same Reference
+        Given the node map
+            | a |   |   |   | b |   |   |   | c |
+
+        And the ways
+            | nodes | name               | ref   | highway  |
+            | ab    | Central Expressway | US 75 | motorway |
+            | bc    |                    | US 75 | motorway |
+
+        When I route I should get
+            | waypoints | route               | turns         |
+            | a,c       | Central Expressway, | depart,arrive |
+
+    Scenario: No Name, Same Reference
+        Given the node map
+            | a |   |   |   | b |   |   |   | c |
+
+        And the ways
+            | nodes | name               | ref   | highway  |
+            | ab    |                    | US 75 | motorway |
+            | bc    | Central Expressway | US 75 | motorway |
+
+        When I route I should get
+            | waypoints | route               | turns         |
+            | a,c       | ,Central Expressway | depart,arrive |
+
+    Scenario: No Name, Same Reference
+        Given the node map
+            | a |   |   |   | b |   |   |   | c |
+
+        And the ways
+            | nodes | name | ref         | highway  |
+            | ab    |      | US 75;US 69 | motorway |
+            | bc    |      | US 75       | motorway |
+
+        When I route I should get
+            | waypoints | route | turns         |
+            | a,c       | ,     | depart,arrive |
+
+    Scenario: No Name, Same Reference
+        Given the node map
+            | a |   |   |   | b |   |   |   | c |
+
+        And the ways
+            | nodes | name | ref         | highway  |
+            | ab    |      | US 69;US 75 | motorway |
+            | bc    |      | US 75       | motorway |
+
+        When I route I should get
+            | waypoints | route | turns         |
+            | a,c       | ,     | depart,arrive |
+
+    Scenario: No Name, Same Reference
+        Given the node map
+            | a |   |   |   | b |   |   |   | c |
+
+        And the ways
+            | nodes | name | ref         | highway  |
+            | ab    |      | US 75       | motorway |
+            | bc    |      | US 75;US 69 | motorway |
+
+        When I route I should get
+            | waypoints | route | turns         |
+            | a,c       | ,     | depart,arrive |
+
+    Scenario: No Name, Same Reference
+        Given the node map
+            | a |   |   |   | b |   |   |   | c |
+
+        And the ways
+            | nodes | name | ref         | highway  |
+            | ab    |      | US 75       | motorway |
+            | bc    |      | US 69;US 75 | motorway |
+
+        When I route I should get
+            | waypoints | route | turns         |
+            | a,c       | ,     | depart,arrive |

features/guidance/roundabout-turn.feature

@@ -491,3 +491,23 @@ Feature: Basic Roundabout
            | h,a       | gh,ab,ab | depart,roundabout turn left exit-3,arrive     |
            | h,d       | gh,cd,cd | depart,roundabout turn straight exit-2,arrive |
            | h,f       | gh,ef,ef | depart,roundabout turn right exit-1,arrive    |
+
+    #http://www.openstreetmap.org/#map=19/41.03275/-2.18990
+    #at some point we probably want to recognise these situations and don't mention the roundabout at all here
+    Scenario: Enter And Exit Throughabout
+        Given the node map
+            |   |   |   |   |   | h |   |   |   |   |
+            |   |   |   |   |   |   |   |   |   |   |
+            | c | b |   | d |   |   |   | e |   | f |
+            |   |   |   |   |   |   |   |   |   |   |
+            |   | a |   |   |   | g |   |   |   |   |
+
+        And the ways
+            | nodes | highway       | name    | junction   | oneway |
+            | dghd  | tertiary_link |         | roundabout |        |
+            | cbdef | trunk         | through |            | no     |
+            | ab    | residential   | in      |            |        |
+
+        When I route I should get
+            | waypoints | turns                                                    | route                      |
+            | a,f       | depart,turn right,roundabout turn straight exit-1,arrive | in,through,through,through |

features/guidance/roundabout.feature

@@ -36,6 +36,25 @@ Feature: Basic Roundabout
            | h,c       | gh,bcegb,bcegb | depart,roundabout-exit-undefined,arrive |
            | h,e       | gh,bcegb,bcegb | depart,roundabout-exit-undefined,arrive |
 
+    #2927
+    Scenario: Only Roundabout
+        Given the node map
+            |   |   | a |   |   |
+            |   |   |   |   |   |
+            |   |   |   |   |   |
+            | b |   |   |   | d |
+            |   |   |   |   |   |
+            |   |   |   |   |   |
+            |   |   | c |   |   |
+
+       And the ways
+            | nodes  | junction   |
+            | abcda  | roundabout |
+
+       When I route I should get
+           | waypoints | route       | turns         |
+           | a,c       | abcda,abcda | depart,arrive |
+
     Scenario: Only Exit
         Given the node map
             |   |   | a |   |   |

features/guidance/turn-lanes.feature

@@ -145,7 +145,6 @@ Feature: Turn Lane Guidance
             | a,c       | road,road      | depart,arrive           | ,                           |
 
     # This tests whether empty/invalid PSV tags cause osrm-extract to crash
-    @bug
     Scenario: Turn with Bus-Lane
         Given the node map
             | a |   | b |   | c |
@@ -181,7 +180,7 @@ Feature: Turn Lane Guidance
             | a,c       | road,road      | depart,arrive            |
 
     #turn lanes are often drawn at the incoming road, even though the actual turn requires crossing the intersection first
-    @todo @bug @collapse @partition-lanes
+    @todo @collapse @partition-lanes
     Scenario: Turn Lanes at Segregated Road
         Given the node map
             |   |   | i | l |   |   |
@@ -383,7 +382,7 @@ Feature: Turn Lane Guidance
             | a,f       | road,turn,turn   | depart,turn left,arrive  | ,left:true right:false, |
             | a,g       | road,right,right | depart,turn right,arrive | ,left:false right:true, |
 
-    @todo @bug @partition-lanes @previous-lanes
+    @todo @partition-lanes @previous-lanes
     Scenario: Narrowing Turn Lanes
         Given the node map
             |   |   |   |   | g |   |
@@ -428,7 +427,7 @@ Feature: Turn Lane Guidance
             | a,d       | road,road      | depart,arrive            | ,                           |
             | a,e       | road,turn,turn | depart,turn right,arrive | ,straight:false right:true, |
 
-    @bug @todo @roundabout
+    @todo @roundabout
     Scenario: Theodor Heuss Platz
         Given the node map
             |   |   |   | i | o |   |   | l |   |
@@ -555,7 +554,7 @@ Feature: Turn Lane Guidance
             | a,e       | road,road,road         | depart,turn uturn,arrive        | ,left:true left:false left:false straight:false straight:false, |
             | a,g       | road,straight,straight | depart,new name straight,arrive | ,left:false left:false left:false straight:true straight:true,  |
 
-    @bug @todo @roundabout
+    @todo @roundabout
     Scenario: Passing Through a Roundabout
         Given the node map
             |   |   | h |   | g |   |   |
@@ -620,7 +619,7 @@ Feature: Turn Lane Guidance
             | a,d       | hwy,hwy       | depart,arrive                       | ,                                                                            |
             | a,e       | hwy,ramp,ramp | depart,off ramp slight right,arrive | ,straight:false straight:false straight;slight right:true slight right:true, |
 
-    @bug @todo
+    @todo
     Scenario: Turning Off Ramp
         Given the node map
             |   | a |   |
@@ -864,7 +863,7 @@ Feature: Turn Lane Guidance
             | a,c       | ab,bc,bc  | depart,turn left,arrive | ,left:true right:false, |
 
     # http://www.openstreetmap.org/#map=19/47.97685/7.82933&layers=D
-    @bug @todo
+    @todo
     Scenario: Lane Parsing Issue #2706: None Assignments I
         Given the node map
             |   | f |   |   | j  |   |
@@ -904,7 +903,7 @@ Feature: Turn Lane Guidance
             # Note: at the moment we don't care about routes, we care about the extract process triggering assertions
 
     # https://www.openstreetmap.org/#map=19/47.99257/7.83276&layers=D
-    @bug @todo
+    @todo
     Scenario: Lane Parsing Issue #2706: None Assignments II
         Given the node map
             |   | k | l |   |
@@ -939,7 +938,7 @@ Feature: Turn Lane Guidance
             | i,e ||||
             # Note: at the moment we don't care about routes, we care about the extract process triggering assertions
 
-    @bug @todo
+    @todo
     Scenario: Lane Parsing Issue #2706: None Assignments III - Minimal reproduction recipe
         Given the node map
             |   |   | l |   |

features/lib/hash.js

@@ -0,0 +1,31 @@
+'use strict';
+
+const fs = require('fs');
+const crypto = require('crypto');
+const d3 = require('d3-queue');
+
+module.exports =  {
+    hashOfFiles: (paths, cb) => {
+        let queue = d3.queue();
+        for (let i = 0; i < paths.length; ++i) {
+            queue.defer(fs.readFile, paths[i]);
+        }
+        queue.awaitAll((err, results) => {
+            if (err) return cb(err);
+            let checksum = crypto.createHash('md5');
+            for (let i = 0; i < results.length; ++i) {
+                checksum.update(results[i]);
+            }
+            cb(null, checksum.digest('hex'));
+        });
+    },
+
+     hashOfFile: (path, cb) => {
+        fs.readFile(path, (err, result) => {
+            if (err) return cb(err);
+            let checksum = crypto.createHash('md5');
+            checksum.update(result);
+            cb(null, checksum.digest('hex'));
+        });
+    }
+};

features/lib/osm.js

@@ -1,11 +1,7 @@
 'use strict';
 
-var builder = require('xmlbuilder');
-
-var ensureDecimal = (i) => {
-    if (parseInt(i) === i) return i.toFixed(1);
-    else return i;
-};
+const builder = require('xmlbuilder');
+const ensureDecimal = require('./utils').ensureDecimal;
 
 class DB {
     constructor () {

features/lib/osrm_loader.js

@@ -0,0 +1,169 @@
+'use strict';
+
+const fs = require('fs');
+const util = require('util');
+const Timeout = require('node-timeout');
+const tryConnect = require('../lib/try_connect');
+const errorReason = require('./utils').errorReason;
+
+class OSRMBaseLoader{
+    constructor (scope) {
+        this.scope = scope;
+        this.child = null;
+    }
+
+    launch (callback) {
+        var limit = Timeout(this.scope.TIMEOUT, { err: new Error('*** Launching osrm-routed timed out.') });
+
+        var runLaunch = (cb) => {
+            this.osrmUp(() => { this.waitForConnection(cb); });
+        };
+
+        runLaunch(limit((e) => { if (e) callback(e); else callback(); }));
+    }
+
+    shutdown (callback) {
+        if (!this.osrmIsRunning()) return callback();
+
+        var limit = Timeout(this.scope.TIMEOUT, { err: new Error('*** Shutting down osrm-routed timed out.')});
+
+        this.osrmDown(limit(callback));
+    }
+
+    osrmIsRunning () {
+        return this.child && !this.child.killed;
+    }
+
+    osrmDown (callback) {
+        if (this.osrmIsRunning()) {
+            this.child.on('exit', (code, signal) => {callback();});
+            this.child.kill();
+        } else callback();
+    }
+
+    waitForConnection (callback) {
+        var retryCount = 0;
+        let retry = (err) => {
+          if (err) {
+            if (retryCount < 10) {
+              retryCount++;
+              setTimeout(() => { tryConnect(this.scope.OSRM_PORT, retry); }, 10);
+            } else {
+              callback(new Error("Could not connect to osrm-routed after ten retries."));
+            }
+          }
+          else
+          {
+            callback();
+          }
+        };
+
+        tryConnect(this.scope.OSRM_PORT, retry);
+    }
+};
+
+class OSRMDirectLoader extends OSRMBaseLoader {
+    constructor (scope) {
+        super(scope);
+    }
+
+    load (inputFile, callback) {
+        this.inputFile = inputFile;
+        this.shutdown(() => {
+            this.launch(callback);
+        });
+    }
+
+    osrmUp (callback) {
+        if (this.osrmIsRunning()) return callback(new Error("osrm-routed already running!"));
+
+        this.child = this.scope.runBin('osrm-routed', util.format("%s -p %d", this.inputFile, this.scope.OSRM_PORT), this.scope.environment, (err) => {
+          if (err) {
+              throw new Error(util.format('osrm-routed %s: %s', errorReason(err), err.cmd));
+          }
+        });
+        callback();
+    }
+};
+
+class OSRMDatastoreLoader extends OSRMBaseLoader {
+    constructor (scope) {
+        super(scope);
+    }
+
+    load (inputFile, callback) {
+        this.inputFile = inputFile;
+
+        this.loadData((err) => {
+            if (err) return callback(err);
+            if (!this.osrmIsRunning()) this.launch(callback);
+            else {
+                this.scope.setupOutputLog(this.child, fs.createWriteStream(this.scope.scenarioLogFile, {'flags': 'a'}));
+                callback();
+            }
+        });
+    }
+
+    loadData (callback) {
+        this.scope.runBin('osrm-datastore', this.inputFile, this.scope.environment, (err) => {
+            if (err) return callback(new Error('*** osrm-datastore exited with ' + err.code + ': ' + err));
+            callback();
+        });
+    }
+
+    osrmUp (callback) {
+        if (this.osrmIsRunning()) return callback();
+
+        this.child = this.scope.runBin('osrm-routed', util.format('--shared-memory=1 -p %d', this.scope.OSRM_PORT), this.scope.environment, (err) => {
+            if (err) {
+                throw new Error(util.format('osrm-routed %s: %s', errorReason(err), err.cmd));
+            }
+        });
+
+        // we call the callback here, becuase we don't want to wait for the child process to finish
+        callback();
+    }
+};
+
+class OSRMLoader {
+    constructor (scope) {
+        this.scope = scope;
+        this.sharedLoader = new OSRMDatastoreLoader(this.scope);
+        this.directLoader = new OSRMDirectLoader(this.scope);
+        this.method = scope.DEFAULT_LOAD_METHOD;
+    }
+
+    load (inputFile, callback) {
+        if (this.method === 'datastore') {
+            this.directLoader.shutdown((err) => {
+              if (err) return callback(err);
+              this.loader = this.sharedLoader;
+              this.sharedLoader.load(inputFile, callback);
+            });
+        } else if (this.method === 'directly') {
+            this.sharedLoader.shutdown((err) => {
+              if (err) return callback(err);
+              this.loader = this.directLoader;
+              this.directLoader.load(inputFile, callback);
+            });
+        } else {
+            callback(new Error('*** Unknown load method ' + method));
+        }
+    }
+
+    setLoadMethod (method) {
+        this.method = method;
+    }
+
+    shutdown (callback) {
+        if (!this.loader) return callback();
+
+        this.loader.shutdown(callback);
+    }
+
+    up () {
+        return this.loader ? this.loader.osrmIsRunning() : false;
+    }
+};
+
+module.exports = OSRMLoader;

features/lib/table_diff.js

@@ -0,0 +1,54 @@
+'use strict';
+
+var util = require('util');
+var path = require('path');
+var fs = require('fs');
+var chalk = require('chalk');
+
+var unescapeStr = (str) => str.replace(/\\\|/g, '\|').replace(/\\\\/g, '\\');
+
+module.exports = function (expected, actual) {
+    let headers = expected.raw()[0];
+    let expected_keys = expected.hashes();
+    let diff = [];
+    let hasErrors = false;
+
+    var good = 0, bad = 0;
+
+    expected_keys.forEach((row, i) => {
+        var rowError = false;
+
+        for (var j in row) {
+            if (unescapeStr(row[j]) != actual[i][j]) {
+                rowError = true;
+                hasErrors = true;
+                break;
+            }
+        }
+
+        if (rowError) {
+            bad++;
+            diff.push(Object.assign({}, row, {c_status: 'undefined'}));
+            diff.push(Object.assign({}, actual[i], {c_status: 'comment'}));
+        } else {
+            good++;
+            diff.push(row);
+        }
+    });
+
+    if (!hasErrors) return null;
+
+    var s = ['Tables were not identical:'];
+    s.push(headers.map(key => '    ' + key).join(' | '));
+    diff.forEach((row) => {
+        var rowString = '| ';
+        headers.forEach((header) => {
+            if (!row.c_status) rowString += chalk.green('    ' + row[header] + ' | ');
+            else if (row.c_status === 'undefined') rowString += chalk.yellow('(-) ' + row[header] + ' | ');
+            else rowString += chalk.red('(+) ' + row[header] + ' | ');
+        });
+        s.push(rowString);
+    });
+
+    return s.join('\n') + '\nTODO this is a temp workaround waiting for https://github.com/cucumber/cucumber-js/issues/534';
+};

features/lib/try_connect.js

@@ -0,0 +1,13 @@
+'use strict';
+
+const net = require('net');
+const Timeout = require('node-timeout');
+
+module.exports = function tryConnect(port, callback) {
+  net.connect({ port: port, host: '127.0.0.1' })
+    .on('connect', () => { callback(); })
+    .on('error', () => {
+        callback(new Error('Could not connect.'));
+    });
+}
+

features/lib/utils.js

@@ -0,0 +1,17 @@
+'use strict';
+
+const util = require('util');
+
+module.exports = {
+
+    ensureDecimal: (i) => {
+        if (parseInt(i) === i) return i.toFixed(1);
+        else return i;
+    },
+
+    errorReason: (err) => {
+        return err.signal ?
+            util.format('killed by signal %s', err.signal) :
+            util.format('exited with code %d', err.code);
+    }
+};

features/options/contract/datasources.feature

@@ -1,8 +1,7 @@
 @prepare @options @files
 Feature: osrm-contract command line options: datasources
 # expansions:
-# {extracted_base} => path to current extracted input file
-# {profile} => path to current profile script
+# {processed_file} => path to .osrm file
 
     Background:
         Given the profile "testbot"
@@ -24,7 +23,6 @@ Feature: osrm-contract command line options: datasources
         And the data has been extracted
 
     Scenario: osrm-contract - Passing base file
-        When I run "osrm-contract --segment-speed-file speeds.csv {extracted_base}.osrm"
-        Then stderr should be empty
-        And datasource names should contain "lua profile,speeds"
-        And it should exit with code 0
+        When I run "osrm-contract --segment-speed-file {speeds_file} {processed_file}"
+        Then datasource names should contain "lua profile,25_osrmcontract_passing_base_file_speeds"
+        And it should exit successfully

features/options/contract/files.feature

@@ -1,9 +1,5 @@
 @prepare @options @files
 Feature: osrm-contract command line options: files
-# expansions:
-# {extracted_base} => path to current extracted input file
-# {profile} => path to current profile script
-
     Background:
         Given the profile "testbot"
         And the node map
@@ -14,12 +10,11 @@ Feature: osrm-contract command line options: files
         And the data has been extracted
 
     Scenario: osrm-contract - Passing base file
-        When I run "osrm-contract {extracted_base}.osrm"
-        Then stderr should be empty
-        And it should exit with code 0
+        When I run "osrm-contract {processed_file}"
+        Then it should exit successfully
 
     Scenario: osrm-contract - Missing input file
-        When I run "osrm-contract over-the-rainbow.osrm"
+        When I try to run "osrm-contract over-the-rainbow.osrm"
         And stderr should contain "over-the-rainbow.osrm"
         And stderr should contain "not found"
-        And it should exit with code 1
+        And it should exit with an error

features/options/contract/help.feature

@@ -2,7 +2,7 @@
 Feature: osrm-contract command line options: help
 
     Scenario: osrm-contract - Help should be shown when no options are passed
-        When I run "osrm-contract"
+        When I try to run "osrm-contract"
         Then stderr should be empty
         And stdout should contain "osrm-contract <input.osrm> [options]:"
         And stdout should contain "Options:"
@@ -13,7 +13,7 @@ Feature: osrm-contract command line options: help
         And stdout should contain "--core"
         And stdout should contain "--level-cache"
         And stdout should contain "--segment-speed-file"
-        And it should exit with code 1
+        And it should exit with an error
 
     Scenario: osrm-contract - Help, short
         When I run "osrm-contract -h"
@@ -27,7 +27,7 @@ Feature: osrm-contract command line options: help
         And stdout should contain "--core"
         And stdout should contain "--level-cache"
         And stdout should contain "--segment-speed-file"
-        And it should exit with code 0
+        And it should exit successfully
 
     Scenario: osrm-contract - Help, long
         When I run "osrm-contract --help"
@@ -41,4 +41,4 @@ Feature: osrm-contract command line options: help
         And stdout should contain "--core"
         And stdout should contain "--level-cache"
         And stdout should contain "--segment-speed-file"
-        And it should exit with code 0
+        And it should exit successfully

features/options/contract/invalid.feature

@@ -5,8 +5,8 @@ Feature: osrm-contract command line options: invalid options
         Given the profile "testbot"
 
     Scenario: osrm-contract - Non-existing option
-        When I run "osrm-contract --fly-me-to-the-moon"
+        When I try to run "osrm-contract --fly-me-to-the-moon"
         Then stdout should be empty
         And stderr should contain "option"
         And stderr should contain "fly-me-to-the-moon"
-        And it should exit with code 1
+        And it should exit with an error

features/options/contract/version.feature

@@ -12,11 +12,11 @@ Feature: osrm-contract command line options: version
         Then stderr should be empty
         And stdout should contain 1 line
         And stdout should contain /(v\d{1,2}\.\d{1,2}\.\d{1,2}|\w*-\d+-\w+)/
-        And it should exit with code 0
+        And it should exit successfully
 
     Scenario: osrm-contract - Version, long
         When I run "osrm-contract --version"
         Then stderr should be empty
         And stdout should contain 1 line
         And stdout should contain /(v\d{1,2}\.\d{1,2}\.\d{1,2}|\w*-\d+-\w+)/
-        And it should exit with code 0
+        And it should exit successfully

features/options/extract/files.feature

@@ -14,17 +14,15 @@ Feature: osrm-extract command line options: files
         And the data has been saved to disk
 
     Scenario: osrm-extract - Passing base file
-        When I run "osrm-extract {osm_base}.osm --profile {profile}"
-        Then stderr should be empty
-        And it should exit with code 0
+        When I run "osrm-extract {osm_file} --profile {profile_file}"
+        Then it should exit successfully
 
     Scenario: osrm-extract - Order of options should not matter
-        When I run "osrm-extract --profile {profile} {osm_base}.osm"
-        Then stderr should be empty
-        And it should exit with code 0
+        When I run "osrm-extract --profile {profile_file} {osm_file}"
+        Then it should exit successfully
 
     Scenario: osrm-extract - Missing input file
-        When I run "osrm-extract over-the-rainbow.osrm --profile {profile}"
+        When I try to run "osrm-extract over-the-rainbow.osrm --profile {profile_file}"
         And stderr should contain "over-the-rainbow.osrm"
         And stderr should contain "not found"
-        And it should exit with code 1
+        And it should exit with an error

features/options/extract/help.feature

@@ -16,7 +16,7 @@ Feature: osrm-extract command line options: help
         And stdout should contain "--threads"
         And stdout should contain "--generate-edge-lookup"
         And stdout should contain "--small-component-size"
-        And it should exit with code 0
+        And it should exit successfully
 
     Scenario: osrm-extract - Help, short
         When I run "osrm-extract -h"
@@ -30,7 +30,7 @@ Feature: osrm-extract command line options: help
         And stdout should contain "--threads"
         And stdout should contain "--generate-edge-lookup"
         And stdout should contain "--small-component-size"
-        And it should exit with code 0
+        And it should exit successfully
 
     Scenario: osrm-extract - Help, long
         When I run "osrm-extract --help"
@@ -44,4 +44,4 @@ Feature: osrm-extract command line options: help
         And stdout should contain "--threads"
         And stdout should contain "--generate-edge-lookup"
         And stdout should contain "--small-component-size"
-        And it should exit with code 0
+        And it should exit successfully

features/options/extract/invalid.feature

@@ -5,8 +5,8 @@ Feature: osrm-extract command line options: invalid options
         Given the profile "testbot"
 
     Scenario: osrm-extract - Non-existing option
-        When I run "osrm-extract --fly-me-to-the-moon"
+        When I try to run "osrm-extract --fly-me-to-the-moon"
         Then stdout should be empty
         And stderr should contain "option"
         And stderr should contain "fly-me-to-the-moon"
-        And it should exit with code 1
+        And it should exit with an error

features/options/extract/version.feature

@@ -12,11 +12,11 @@ Feature: osrm-extract command line options: version
         Then stderr should be empty
         And stdout should contain 1 line
         And stdout should contain /(v\d{1,2}\.\d{1,2}\.\d{1,2}|\w*-\d+-\w+)/
-        And it should exit with code 0
+        And it should exit successfully
 
     Scenario: osrm-extract - Version, long
         When I run "osrm-extract --version"
         Then stderr should be empty
         And stdout should contain 1 line
         And stdout should contain /(v\d{1,2}\.\d{1,2}\.\d{1,2}|\w*-\d+-\w+)/
-        And it should exit with code 0
+        And it should exit successfully

features/options/routed/files.feature

@@ -29,4 +29,4 @@ Feature: osrm-routed command line options: files
         And stdout should contain /^\[info\] loaded plugin: viaroute/
         And stdout should contain /^\[info\] trial run/
         And stdout should contain /^\[info\] shutdown completed/
-        And it should exit with code 0
+        And it should exit successfully

features/options/routed/help.feature

@@ -21,7 +21,7 @@ Feature: osrm-routed command line options: help
         And stdout should contain "--max-trip-size"
         And stdout should contain "--max-table-size"
         And stdout should contain "--max-matching-size"
-        And it should exit with code 0
+        And it should exit successfully
 
     Scenario: osrm-routed - Help, short
         When I run "osrm-routed -h"
@@ -40,7 +40,7 @@ Feature: osrm-routed command line options: help
         And stdout should contain "--max-trip-size"
         And stdout should contain "--max-table-size"
         And stdout should contain "--max-matching-size"
-        And it should exit with code 0
+        And it should exit successfully
 
     Scenario: osrm-routed - Help, long
         When I run "osrm-routed --help"
@@ -59,4 +59,4 @@ Feature: osrm-routed command line options: help
         And stdout should contain "--max-table-size"
         And stdout should contain "--max-table-size"
         And stdout should contain "--max-matching-size"
-        And it should exit with code 0
+        And it should exit successfully

features/options/routed/invalid.feature

@@ -5,14 +5,14 @@ Feature: osrm-routed command line options: invalid options
         Given the profile "testbot"
 
     Scenario: osrm-routed - Non-existing option
-        When I run "osrm-routed --fly-me-to-the-moon"
+        When I try to run "osrm-routed --fly-me-to-the-moon"
         Then stdout should be empty
         And stderr should contain "unrecognised"
         And stderr should contain "fly-me-to-the-moon"
-        And it should exit with code 1
+        And it should exit with an error
 
     Scenario: osrm-routed - Missing file
-        When I run "osrm-routed over-the-rainbow.osrm"
+        When I try to run "osrm-routed over-the-rainbow.osrm"
         Then stderr should contain "over-the-rainbow.osrm"
         And stderr should contain "not found"
-        And it should exit with code 1
+        And it should exit with an error

features/options/routed/version.feature

@@ -12,11 +12,11 @@ Feature: osrm-routed command line options: version
         Then stderr should be empty
         And stdout should contain 1 line
         And stdout should contain /(v\d{1,2}\.\d{1,2}\.\d{1,2}|\w*-\d+-\w+)/
-        And it should exit with code 0
+        And it should exit successfully
 
     Scenario: osrm-routed - Version, long
         When I run "osrm-routed --version"
         Then stderr should be empty
         And stdout should contain 1 line
         And stdout should contain /(v\d{1,2}\.\d{1,2}\.\d{1,2}|\w*-\d+-\w+)/
-        And it should exit with code 0
+        And it should exit successfully

features/raster/extract.feature

@@ -1,9 +1,5 @@
 @raster @extract
 Feature: osrm-extract with a profile containing raster source
-# expansions:
-# {osm_base} => path to current input file
-# {profile} => path to current profile script
-
     Scenario: osrm-extract on a valid profile
         Given the profile "rasterbot"
         And the node map
@@ -11,8 +7,15 @@ Feature: osrm-extract with a profile containing raster source
         And the ways
             | nodes |
             | ab    |
+        And the raster source
+            """
+            0  0  0   0
+            0  0  0   250
+            0  0  250 500
+            0  0  0   250
+            0  0  0   0
+            """
         And the data has been saved to disk
-        When I run "osrm-extract {osm_base}.osm -p {profile}"
-        Then stderr should be empty
-        And stdout should contain "source loader"
-        And it should exit with code 0
+        When I run "osrm-extract {osm_file} -p {profile_file}"
+        Then stdout should contain "source loader"
+        And it should exit successfully

features/raster/weights.feature

@@ -32,8 +32,8 @@ Feature: Raster - weights
 
     Scenario: Weighting not based on raster sources
         Given the profile "testbot"
-        When I run "osrm-extract {osm_base}.osm -p {profile}"
-        And I run "osrm-contract {osm_base}.osm"
+        When I run "osrm-extract {osm_file} -p {profile_file}"
+        And I run "osrm-contract {processed_file}"
         And I route I should get
             | from | to | route    | speed   |
             | a    | b  | ab,ab    | 36 km/h |
@@ -44,9 +44,9 @@ Feature: Raster - weights
 
     Scenario: Weighting based on raster sources
         Given the profile "rasterbot"
-        When I run "osrm-extract {osm_base}.osm -p {profile}"
+        When I run "osrm-extract {osm_file} -p {profile_file}"
         Then stdout should contain "evaluating segment"
-        And I run "osrm-contract {osm_base}.osm"
+        And I run "osrm-contract {processed_file}"
         And I route I should get
             | from | to | route    | speed   |
             | a    | b  | ab,ab    | 8 km/h  |
@@ -62,9 +62,9 @@ Feature: Raster - weights
 
     Scenario: Weighting based on raster sources
         Given the profile "rasterbotinterp"
-        When I run "osrm-extract {osm_base}.osm -p {profile}"
+        When I run "osrm-extract {osm_file} -p {profile_file}"
         Then stdout should contain "evaluating segment"
-        And I run "osrm-contract {osm_base}.osm"
+        And I run "osrm-contract {processed_file}"
         And I route I should get
             | from | to | route    | speed   |
             | a    | b  | ab,ab    | 8 km/h  |

features/step_definitions/data.js

@@ -2,19 +2,23 @@ var util = require('util');
 var path = require('path');
 var fs = require('fs');
 var d3 = require('d3-queue');
-var OSM = require('../support/build_osm');
+var OSM = require('../lib/osm');
 
 module.exports = function () {
     this.Given(/^the profile "([^"]*)"$/, (profile, callback) => {
-        this.setProfile(profile, callback);
+        this.profile = profile;
+        this.profileFile = path.join(this.PROFILES_PATH, this.profile + '.lua');
+        callback();
     });
 
     this.Given(/^the extract extra arguments "(.*?)"$/, (args, callback) => {
-        this.setExtractArgs(args, callback);
+        this.extractArgs = this.expandOptions(args);
+        callback();
     });
 
     this.Given(/^the contract extra arguments "(.*?)"$/, (args, callback) => {
-        this.setContractArgs(args, callback);
+        this.contractArgs = this.expandOptions(args);
+        callback();
     });
 
     this.Given(/^a grid size of ([0-9.]+) meters$/, (meters, callback) => {
@@ -228,58 +232,46 @@ module.exports = function () {
     });
 
     this.Given(/^the raster source$/, (data, callback) => {
-        this.updateFingerprintExtract(data);
-        fs.writeFile(path.resolve(this.TEST_FOLDER, 'rastersource.asc'), data, callback);
+        // TODO: Don't overwrite if it exists
+        fs.writeFile(this.rasterCacheFile, data, callback);
+        // we need this to pass it to the profiles
+        this.environment = Object.assign({OSRM_RASTER_SOURCE: this.rasterCacheFile}, this.environment);
     });
 
     this.Given(/^the speed file$/, (data, callback) => {
-        this.updateFingerprintContract(data);
-        fs.writeFile(path.resolve(this.TEST_FOLDER, 'speeds.csv'), data, callback);
+        // TODO: Don't overwrite if it exists
+        fs.writeFile(this.speedsCacheFile, data, callback);
     });
 
     this.Given(/^the turn penalty file$/, (data, callback) => {
-        this.updateFingerprintContract(data);
-        fs.writeFile(path.resolve(this.TEST_FOLDER, 'penalties.csv'), data, callback);
+        // TODO: Don't overwrite if it exists
+        fs.writeFile(this.penaltiesCacheFile, data, callback);
     });
 
     this.Given(/^the data has been saved to disk$/, (callback) => {
-        try {
-            this.reprocess(callback);
-        } catch(e) {
-            this.processError = e;
-            callback(e);
-        }
+        this.reprocess(callback);
     });
 
     this.Given(/^the data has been extracted$/, (callback) => {
-        this.osmData.populate(() => {
-            this.writeAndExtract((err) => {
-                if (err) this.processError = err;
-                callback();
-            });
-        });
+        this.reprocess(callback);
     });
 
     this.Given(/^the data has been contracted$/, (callback) => {
-        this.reprocess((err) => {
-            if (err) this.processError = err;
-            callback();
-        });
+        this.reprocess(callback);
     });
 
     this.Given(/^osrm\-routed is stopped$/, (callback) => {
-        this.OSRMLoader.shutdown((err) => {
-            if (err) this.processError = err;
-            callback();
-        });
+        this.OSRMLoader.shutdown(callback);
     });
 
-    this.Given(/^data is loaded directly/, () => {
-        this.loadMethod = 'directly';
+    this.Given(/^data is loaded directly/, (callback) => {
+        this.osrmLoader.setLoadMethod('directly');
+        callback();
     });
 
-    this.Given(/^data is loaded with datastore$/, () => {
-        this.loadMethod = 'datastore';
+    this.Given(/^data is loaded with datastore$/, (callback) => {
+        this.osrmLoader.setLoadMethod('datastore');
+        callback();
     });
 
     this.Given(/^the HTTP method "([^"]*)"$/, (method, callback) => {

features/step_definitions/distance_matrix.js

@@ -53,8 +53,6 @@ module.exports = function () {
                 });
 
                 var testRow = (row, ri, cb) => {
-                    var ok = true;
-
                     for (var k in result[ri]) {
                         if (this.FuzzyMatch.match(result[ri][k], row[k])) {
                             result[ri][k] = row[k];
@@ -62,15 +60,9 @@ module.exports = function () {
                             result[ri][k] = '';
                         } else {
                             result[ri][k] = result[ri][k].toString();
-                            ok = false;
                         }
                     }
 
-                    if (!ok) {
-                        var failed = { attempt: 'distance_matrix', query: this.query, response: response };
-                        this.logFail(row, result[ri], [failed]);
-                    }
-
                     result[ri][''] = row[''];
                     cb(null, result[ri]);
                 };

features/step_definitions/hooks.js

@@ -1,18 +0,0 @@
-var util = require('util');
-
-module.exports = function () {
-    this.Before((scenario, callback) => {
-        this.scenarioTitle = scenario.getName();
-
-        this.loadMethod = this.DEFAULT_LOAD_METHOD;
-        this.queryParams = {};
-        var d = new Date();
-        this.scenarioTime = util.format('%d-%d-%dT%s:%s:%sZ', d.getFullYear(), d.getMonth()+1, d.getDate(), d.getHours(), d.getMinutes(), d.getSeconds());
-        this.resetData();
-        this.hasLoggedPreprocessInfo = false;
-        this.hasLoggedScenarioInfo = false;
-        this.setGridSize(this.DEFAULT_GRID_SIZE);
-        this.setOrigin(this.DEFAULT_ORIGIN);
-        callback();
-    });
-};

features/step_definitions/matching.js

@@ -157,7 +157,6 @@ module.exports = function () {
                         } else {
                             got.matchings = encodedResult;
                             row.matchings = extendedTarget;
-                            this.logFail(row, got, { matching: { query: this.query, response: res } });
                         }
 
                         cb(null, got);

features/step_definitions/nearest.js

@@ -22,24 +22,16 @@ module.exports = function () {
 
                         var got = { in: row.in, out: row.out };
 
-                        var ok = true;
-
                         Object.keys(row).forEach((key) => {
                             if (key === 'out') {
                                 if (this.FuzzyMatch.matchLocation(coord, outNode)) {
                                     got[key] = row[key];
                                 } else {
                                     row[key] = util.format('%s [%d,%d]', row[key], outNode.lat, outNode.lon);
-                                    ok = false;
                                 }
                             }
                         });
 
-                        if (!ok) {
-                            var failed = { attempt: 'nearest', query: this.query, response: response };
-                            this.logFail(row, got, [failed]);
-                        }
-
                         cb(null, got);
                     }
                     else {

features/step_definitions/options.js

@@ -2,36 +2,58 @@ var assert = require('assert');
 var fs = require('fs');
 
 module.exports = function () {
-    this.When(/^I run "osrm\-routed\s?(.*?)"$/, { timeout: this.TIMEOUT }, (options, callback) => {
-        this.runBin('osrm-routed', options, () => {
-            callback();
+    this.resetOptionsOutput = () => {
+        this.stdout = null;
+        this.stderr = null;
+        this.exitCode = null;
+        this.termSignal = null;
+    };
+
+    this.runAndSafeOutput = (binary, options, callback) => {
+        this.runBin(binary, this.expandOptions(options), this.environment, (err, stdout, stderr) => {
+            this.stdout = stdout;
+            this.stderr = stderr;
+            this.exitCode = err && err.code || 0;
+            this.termSignal = err && err.signal || '';
+            callback(err);
         });
+    };
+
+    this.When(/^I run "osrm\-routed\s?(.*?)"$/, { timeout: this.TIMEOUT }, (options, callback) => {
+        this.runAndSafeOutput('osrm-routed', options, callback);
     });
 
     this.When(/^I run "osrm\-extract\s?(.*?)"$/, (options, callback) => {
-        this.runBin('osrm-extract', options, () => {
-            callback();
-        });
+        this.runAndSafeOutput('osrm-extract', options, callback);
     });
 
     this.When(/^I run "osrm\-contract\s?(.*?)"$/, (options, callback) => {
-        this.runBin('osrm-contract', options, () => {
-            callback();
-        });
+        this.runAndSafeOutput('osrm-contract', options, callback);
+    });
+
+    this.When(/^I try to run "osrm\-routed\s?(.*?)"$/, (options, callback) => {
+        this.runAndSafeOutput('osrm-routed', options, () => { callback(); });
+    });
+
+    this.When(/^I try to run "osrm\-extract\s?(.*?)"$/, (options, callback) => {
+        this.runAndSafeOutput('osrm-extract', options, () => { callback(); });
+    });
+
+    this.When(/^I try to run "osrm\-contract\s?(.*?)"$/, (options, callback) => {
+        this.runAndSafeOutput('osrm-contract', options, () => { callback(); });
     });
 
     this.When(/^I run "osrm\-datastore\s?(.*?)"$/, (options, callback) => {
-        this.runBin('osrm-datastore', options, () => {
-            callback();
-        });
+        this.runAndSafeOutput('osrm-datastore', options, callback);
     });
 
-    this.Then(/^it should exit with code (\d+)$/, (code) => {
-        assert.equal(this.exitCode, parseInt(code));
+    this.Then(/^it should exit successfully$/, () => {
+        assert.equal(this.exitCode, 0);
+        assert.equal(this.termSignal, '');
     });
 
-    this.Then(/^it should exit with code not (\d+)$/, (code) => {
-        assert.notEqual(this.exitCode, parseInt(code));
+    this.Then(/^it should exit with an error$/, () => {
+        assert.ok(this.exitCode !== 0 || this.termSignal);
     });
 
     this.Then(/^stdout should contain "(.*?)"$/, (str) => {
@@ -65,7 +87,7 @@ module.exports = function () {
     });
 
     this.Then(/^datasource names should contain "(.+)"$/, (expectedData) => {
-        var actualData = fs.readFileSync(this.osmData.extractedFile + '.osrm.datasource_names', {encoding:'UTF-8'}).trim().split('\n').join(',');
+        var actualData = fs.readFileSync(this.processedCacheFile + '.datasource_names', {encoding:'UTF-8'}).trim().split('\n').join(',');
         assert.equal(actualData, expectedData);
     });
 

features/step_definitions/requests.js

@@ -51,7 +51,7 @@ module.exports = function () {
     });
 
     this.Then(/^"([^"]*)" should return code (\d+)$/, (binary, code) => {
-        assert.ok(this.processError instanceof this.OSRMError);
+        assert.ok(this.processError instanceof Error);
         assert.equal(this.processError.process, binary);
         assert.equal(parseInt(this.processError.code), parseInt(code));
     });

features/step_definitions/routability.js

@@ -13,7 +13,7 @@ module.exports = function () {
             }
 
             this.reprocessAndLoadData((e) => {
-                if (e) callback(e);
+                if (e) return callback(e);
                 var testRow = (row, i, cb) => {
                     var outputRow = row;
 
@@ -41,10 +41,6 @@ module.exports = function () {
                             }
                         });
 
-                        if (outputRow != row) {
-                            this.logFail(row, outputRow, result);
-                        }
-
                         cb(null, outputRow);
                     });
                 };
@@ -116,7 +112,7 @@ module.exports = function () {
                     sq.defer(parseRes, key);
                 });
 
-                sq.awaitAll(() => { cb(null, result); });
+                sq.awaitAll((err) => { cb(err, result); });
             });
     };
 };

features/step_definitions/trip.js

@@ -85,23 +85,14 @@ module.exports = function () {
                     } else {
                         got.trips = encodedResult;
                         got.trips = extendedTarget;
-                        this.logFail(row, got, { trip: { query: this.query, response: res }});
                     }
 
-                    ok = true;
-
                     for (var key in row) {
                         if (this.FuzzyMatch.match(got[key], row[key])) {
                             got[key] = row[key];
-                        } else {
-                            ok = false;
                         }
                     }
 
-                    if (!ok) {
-                        this.logFail(row, got, { trip: { query: this.query, response: res }});
-                    }
-
                     cb(null, got);
                 };
 

features/support/cache.js

@@ -0,0 +1,184 @@
+'use strict';
+
+const d3 = require('d3-queue');
+const fs = require('fs');
+const util = require('util');
+const path = require('path');
+const mkdirp = require('mkdirp');
+const hash = require('../lib/hash');
+const rimraf = require('rimraf');
+
+module.exports = function() {
+    this.initializeCache = (callback) => {
+        this.getOSRMHash((err, osrmHash) => {
+            if (err) return callback(err);
+            this.osrmHash = osrmHash;
+            callback();
+        });
+    };
+
+    // computes all paths for every feature
+    this.setupFeatures = (features, callback) => {
+        this.featureIDs = {};
+        this.featureCacheDirectories = {};
+        this.featureProcessedCacheDirectories = {};
+        let queue = d3.queue();
+
+        function initializeFeature(feature, callback) {
+            let uri = feature.getUri();
+
+            // setup cache for feature data
+            hash.hashOfFile(uri, (err, hash) => {
+                if (err) return callback(err);
+
+                // shorten uri to be realtive to 'features/'
+                let featurePath = path.relative(path.resolve('./features'), uri);
+                // bicycle/bollards/{HASH}/
+                let featureID = path.join(featurePath, hash);
+                let featureCacheDirectory = this.getFeatureCacheDirectory(featureID);
+                let featureProcessedCacheDirectory = this.getFeatureProcessedCacheDirectory(featureCacheDirectory, this.osrmHash);
+                this.featureIDs[uri] = featureID;
+                this.featureCacheDirectories[uri] = featureCacheDirectory;
+                this.featureProcessedCacheDirectories[uri] = featureProcessedCacheDirectory;
+
+                d3.queue(1)
+                  .defer(mkdirp, featureProcessedCacheDirectory)
+                  .defer(this.cleanupFeatureCache.bind(this), featureCacheDirectory, hash)
+                  .defer(this.cleanupProcessedFeatureCache.bind(this), featureProcessedCacheDirectory, this.osrmHash)
+                  .awaitAll(callback);
+            });
+        }
+
+        for (let i = 0; i < features.length; ++i) {
+            queue.defer(initializeFeature.bind(this), features[i]);
+        }
+        queue.awaitAll(callback);
+    };
+
+    this.cleanupProcessedFeatureCache = (directory, osrmHash, callback) => {
+        let parentPath = path.resolve(path.join(directory, '..'));
+        fs.readdir(parentPath, (err, files) => {
+            let q = d3.queue();
+            function runStats(path, callback) {
+                fs.stat(path, (err, stat) => {
+                    if (err) return callback(err);
+                    callback(null, {file: path, stat: stat});
+                });
+            }
+            files.map(f => { q.defer(runStats, path.join(parentPath, f)); });
+            q.awaitAll((err, results) => {
+                if (err) return callback(err);
+                let q = d3.queue();
+                results.forEach(r => {
+                    if (r.stat.isDirectory() && r.file.search(osrmHash) < 0) {
+                        q.defer(rimraf, r.file);
+                    }
+                });
+                q.awaitAll(callback);
+            });
+        });
+    };
+
+    this.cleanupFeatureCache = (directory, featureHash, callback) => {
+        let parentPath = path.resolve(path.join(directory, '..'));
+        fs.readdir(parentPath, (err, files) => {
+            let q = d3.queue();
+            files.filter(name => { return name !== featureHash;})
+                 .map((f) => { q.defer(rimraf, path.join(parentPath, f)); });
+            q.awaitAll(callback);
+        });
+    };
+
+    this.setupFeatureCache = (feature) => {
+        let uri = feature.getUri();
+        this.featureID = this.featureIDs[uri];
+        this.featureCacheDirectory = this.featureCacheDirectories[uri];
+        this.featureProcessedCacheDirectory = this.featureProcessedCacheDirectories[uri];
+    };
+
+    this.setupScenarioCache = (scenarioID) => {
+        this.scenarioCacheFile = this.getScenarioCacheFile(this.featureCacheDirectory, scenarioID);
+        this.processedCacheFile = this.getProcessedCacheFile(this.featureProcessedCacheDirectory, scenarioID);
+        this.inputCacheFile = this.getInputCacheFile(this.featureProcessedCacheDirectory, scenarioID);
+        this.rasterCacheFile = this.getRasterCacheFile(this.featureProcessedCacheDirectory, scenarioID);
+        this.speedsCacheFile = this.getSpeedsCacheFile(this.featureProcessedCacheDirectory, scenarioID);
+        this.penaltiesCacheFile = this.getPenaltiesCacheFile(this.featureProcessedCacheDirectory, scenarioID);
+    };
+
+    // returns a hash of all OSRM code side dependencies
+    this.getOSRMHash = (callback) => {
+        let dependencies = [
+            this.OSRM_EXTRACT_PATH,
+            this.OSRM_CONTRACT_PATH,
+            this.LIB_OSRM_EXTRACT_PATH,
+            this.LIB_OSRM_CONTRACT_PATH
+        ];
+
+        var addLuaFiles = (directory, callback) => {
+            fs.readdir(path.normalize(directory), (err, files) => {
+                if (err) return callback(err);
+
+                var luaFiles = files.filter(f => !!f.match(/\.lua$/)).map(f => path.normalize(directory + '/' + f));
+                Array.prototype.push.apply(dependencies, luaFiles);
+
+                callback();
+            });
+        };
+
+        // Note: we need a serialized queue here to ensure that the order of the files
+        // passed is stable. Otherwise the hash will not be stable
+        d3.queue(1)
+            .defer(addLuaFiles, this.PROFILES_PATH)
+            .defer(addLuaFiles, this.PROFILES_PATH + '/lib')
+            .awaitAll(hash.hashOfFiles.bind(hash, dependencies, callback));
+    };
+
+    // test/cache/bicycle/bollards/{HASH}/
+    this.getFeatureCacheDirectory = (featureID) => {
+        return path.join(this.CACHE_PATH, featureID);
+    };
+
+    // converts the scenario titles in file prefixes
+    this.getScenarioID = (scenario) => {
+        let name = scenario.getName().toLowerCase().replace(/[\/\-'=,\(\)]/g, '').replace(/\s/g, '_').replace(/__/g, '_').replace(/\.\./g, '.');
+        return util.format('%d_%s', scenario.getLine(), name);
+    };
+
+    // test/cache/{feature_path}/{feature_hash}/{scenario}_raster.asc
+    this.getRasterCacheFile = (featureCacheDirectory, scenarioID) => {
+        return path.join(featureCacheDirectory, scenarioID) + '_raster.asc';
+    };
+
+    // test/cache/{feature_path}/{feature_hash}/{scenario}_speeds.csv
+    this.getSpeedsCacheFile = (featureCacheDirectory, scenarioID) => {
+        return path.join(featureCacheDirectory, scenarioID) + '_speeds.csv';
+    };
+
+    // test/cache/{feature_path}/{feature_hash}/{scenario}_penalties.csv
+    this.getPenaltiesCacheFile = (featureCacheDirectory, scenarioID) => {
+        return path.join(featureCacheDirectory, scenarioID) + '_penalties.csv';
+    };
+
+    // test/cache/{feature_path}/{feature_hash}/{scenario}.osm
+    this.getScenarioCacheFile = (featureCacheDirectory, scenarioID) => {
+        return path.join(featureCacheDirectory, scenarioID) + '.osm';
+    };
+
+    // test/cache/{feature_path}/{feature_hash}/{osrm_hash}/
+    this.getFeatureProcessedCacheDirectory = (featureCacheDirectory, osrmHash) => {
+        return path.join(featureCacheDirectory, osrmHash);
+    };
+
+    // test/cache/{feature_path}/{feature_hash}/{osrm_hash}/{scenario}.osrm
+    this.getProcessedCacheFile = (featureProcessedCacheDirectory, scenarioID) => {
+        return path.join(featureProcessedCacheDirectory, scenarioID) + '.osrm';
+    };
+
+    // test/cache/{feature_path}/{feature_hash}/{osrm_hash}/{scenario}.osm
+    this.getInputCacheFile = (featureProcessedCacheDirectory, scenarioID) => {
+        return path.join(featureProcessedCacheDirectory, scenarioID) + '.osm';
+    };
+
+
+    return this;
+};

features/support/config.js

@@ -1,127 +0,0 @@
-var fs = require('fs');
-var path = require('path');
-var util = require('util');
-var d3 = require('d3-queue');
-var OSM = require('./build_osm');
-var classes = require('./data_classes');
-
-module.exports = function () {
-    this.initializeOptions = (callback) => {
-        this.profile = this.profile || this.DEFAULT_SPEEDPROFILE;
-
-        this.OSMDB = this.OSMDB || new OSM.DB();
-
-        this.nameNodeHash = this.nameNodeHash || {};
-
-        this.locationHash = this.locationHash || {};
-
-        this.nameWayHash = this.nameWayHash || {};
-
-        this.osmData = new classes.osmData(this);
-
-        this.OSRMLoader = this._OSRMLoader();
-
-        this.PREPROCESS_LOG_FILE = path.resolve(this.TEST_FOLDER, 'preprocessing.log');
-
-        this.LOG_FILE = path.resolve(this.TEST_FOLDER, 'fail.log');
-
-        this.HOST = 'http://127.0.0.1:' + this.OSRM_PORT;
-
-        this.DESTINATION_REACHED = 15;  // OSRM instruction code
-
-        this.shortcutsHash = this.shortcutsHash || {};
-
-        var hashLuaLib = (cb) => {
-            fs.readdir(path.normalize(this.PROFILES_PATH + '/lib/'), (err, files) => {
-                if (err) cb(err);
-                var luaFiles = files.filter(f => !!f.match(/\.lua$/)).map(f => path.normalize(this.PROFILES_PATH + '/lib/' + f));
-                this.hashOfFiles(luaFiles, hash => {
-                    this.luaLibHash = hash;
-                    cb();
-                });
-            });
-        };
-
-        var hashProfile = (cb) => {
-            this.hashProfile((hash) => {
-                this.profileHash = hash;
-                cb();
-            });
-        };
-
-        var hashExtract = (cb) => {
-            var files = [ util.format('%s/osrm-extract%s', this.BIN_PATH, this.EXE),
-                          util.format('%s/libosrm_extract%s', this.BIN_PATH, this.LIB) ];
-            this.hashOfFiles(files, (hash) => {
-                this.binExtractHash = hash;
-                cb();
-            });
-        };
-
-        var hashContract = (cb) => {
-            var files = [ util.format('%s/osrm-contract%s', this.BIN_PATH, this.EXE),
-                          util.format('%s/libosrm_contract%s', this.BIN_PATH, this.LIB) ];
-            this.hashOfFiles(files, (hash) => {
-                this.binContractHash = hash;
-                cb();
-            });
-        };
-
-        var hashRouted = (cb) => {
-            var files = [ util.format('%s/osrm-routed%s', this.BIN_PATH, this.EXE),
-                          util.format('%s/libosrm%s', this.BIN_PATH, this.LIB) ];
-            this.hashOfFiles(files, (hash) => {
-                this.binRoutedHash = hash;
-                this.fingerprintRoute = this.hashString(this.binRoutedHash);
-                cb();
-            });
-        };
-
-        d3.queue()
-            .defer(hashLuaLib)
-            .defer(hashProfile)
-            .defer(hashExtract)
-            .defer(hashContract)
-            .defer(hashRouted)
-            .awaitAll(() => {
-                this.AfterConfiguration(() => {
-                    callback();
-                });
-            });
-    };
-
-    this.updateFingerprintExtract = (str) => {
-        this.fingerprintExtract = this.hashString([this.fingerprintExtract, str].join('-'));
-    };
-
-    this.updateFingerprintContract = (str) => {
-        this.fingerprintContract = this.hashString([this.fingerprintContract, str].join('-'));
-    };
-
-    this.setProfile = (profile, cb) => {
-        var lastProfile = this.profile;
-        if (profile !== lastProfile) {
-            this.profile = profile;
-            this.hashProfile((hash) => {
-                this.profileHash = hash;
-                this.updateFingerprintExtract(this.profileHash);
-                cb();
-            });
-        } else {
-            this.updateFingerprintExtract(this.profileHash);
-            cb();
-        }
-    };
-
-    this.setExtractArgs = (args, callback) => {
-        this.extractArgs = args;
-        this.updateFingerprintExtract(args);
-        callback();
-    };
-
-    this.setContractArgs = (args, callback) => {
-        this.contractArgs = args;
-        this.updateFingerprintContract(args);
-        callback();
-    };
-};

features/support/data.js

@@ -1,11 +1,14 @@
-var fs = require('fs');
-var path = require('path');
-var util = require('util');
-var exec = require('child_process').exec;
-var d3 = require('d3-queue');
+'use strict';
 
-var OSM = require('./build_osm');
-var classes = require('./data_classes');
+const fs = require('fs');
+const util = require('util');
+const d3 = require('d3-queue');
+
+const OSM = require('../lib/osm');
+const classes = require('./data_classes');
+const tableDiff = require('../lib/table_diff');
+const ensureDecimal = require('../lib/utils').ensureDecimal;
+const errorReason = require('../lib/utils').errorReason;
 
 module.exports = function () {
     this.setGridSize = (meters) => {
@@ -94,13 +97,8 @@ module.exports = function () {
         q.awaitAll(callback);
     };
 
-    this.ensureDecimal = (i) => {
-        if (parseInt(i) === i) return i.toFixed(1);
-        else return i;
-    };
-
     this.tableCoordToLonLat = (ci, ri) => {
-        return [this.origin[0] + ci * this.zoom, this.origin[1] - ri * this.zoom].map(this.ensureDecimal);
+        return [this.origin[0] + ci * this.zoom, this.origin[1] - ri * this.zoom].map(ensureDecimal);
     };
 
     this.addOSMNode = (name, lon, lat, id) => {
@@ -132,10 +130,6 @@ module.exports = function () {
         return this.nameWayHash[s.toString()] || this.nameWayHash[s.toString().split('').reverse().join('')];
     };
 
-    this.resetData = () => {
-        this.resetOSM();
-    };
-
     this.makeOSMId = () => {
         this.osmID = this.osmID + 1;
         return this.osmID;
@@ -143,206 +137,88 @@ module.exports = function () {
 
     this.resetOSM = () => {
         this.OSMDB.clear();
-        this.osmData.reset();
         this.nameNodeHash = {};
         this.locationHash = {};
+        this.shortcutsHash = {};
         this.nameWayHash = {};
         this.osmID = 0;
     };
 
     this.writeOSM = (callback) => {
-        fs.exists(this.DATA_FOLDER, (exists) => {
-            var mkDirFn = exists ? (cb) => { cb(); } : fs.mkdir.bind(fs.mkdir, this.DATA_FOLDER);
-            mkDirFn((err) => {
-                if (err) return callback(err);
-                var osmPath = path.resolve(this.DATA_FOLDER, util.format('%s.osm', this.osmData.osmFile));
-                fs.exists(osmPath, (exists) => {
-                    if (!exists) fs.writeFile(osmPath, this.osmData.str, callback);
-                    else callback();
+        fs.exists(this.scenarioCacheFile, (exists) => {
+            if (exists) callback();
+            else {
+                this.OSMDB.toXML((xml) => {
+                    fs.writeFile(this.scenarioCacheFile, xml, callback);
                 });
-            });
-        });
-    };
-
-    this.isExtracted = (callback) => {
-        fs.exists(util.format('%s.osrm', this.osmData.extractedFile), (core) => {
-            if (!core) return callback(false);
-            fs.exists(util.format('%s.osrm.names', this.osmData.extractedFile), (names) => {
-                if (!names) return callback(false);
-                fs.exists(util.format('%s.osrm.restrictions', this.osmData.extractedFile), (restrictions) => {
-                    return callback(restrictions);
-                });
-            });
-        });
-    };
-
-    this.isContracted = (callback) => {
-        fs.exists(util.format('%s.osrm.hsgr', this.osmData.contractedFile), callback);
-    };
-
-    this.writeTimestamp = (callback) => {
-        fs.writeFile(util.format('%s.osrm.timestamp', this.osmData.contractedFile), this.OSM_TIMESTAMP, callback);
-    };
-
-    this.writeInputData = (callback) => {
-        this.writeOSM((err) => {
-            if (err) return callback(err);
-            this.writeTimestamp(callback);
-        });
-    };
-
-    this.extractData = (callback) => {
-        this.logPreprocessInfo();
-        this.log(util.format('== Extracting %s.osm...', this.osmData.osmFile), 'preprocess');
-        var cmd = util.format('%s/osrm-extract %s.osm %s --profile %s/%s.lua >>%s 2>&1',
-            this.BIN_PATH, this.osmData.osmFile, this.extractArgs || '', this.PROFILES_PATH, this.profile, this.PREPROCESS_LOG_FILE);
-        this.log(cmd);
-        process.chdir(this.TEST_FOLDER);
-        exec(cmd, (err) => {
-            if (err) {
-                this.log(util.format('*** Exited with code %d', err.code), 'preprocess');
-                process.chdir('../');
-                return callback(this.ExtractError(err.code, util.format('osrm-extract exited with code %d', err.code)));
             }
-
-            var q = d3.queue();
-
-            var rename = (file, cb) => {
-                this.log(util.format('Renaming %s.%s to %s.%s', this.osmData.osmFile, file, this.osmData.extractedFile, file), 'preprocess');
-                fs.rename([this.osmData.osmFile, file].join('.'), [this.osmData.extractedFile, file].join('.'), (err) => {
-                    if (err) return cb(this.FileError(null, 'failed to rename data file after extracting'));
-                    cb();
-                });
-            };
-
-            var renameIfExists = (file, cb) => {
-                fs.stat([this.osmData.osmFile, file].join('.'), (doesNotExistErr, exists) => {
-                    if (exists) rename(file, cb);
-                    else cb();
-                });
-            };
-
-            ['osrm', 'osrm.ebg', 'osrm.edges', 'osrm.enw', 'osrm.fileIndex', 'osrm.geometry', 'osrm.icd',
-             'osrm.names', 'osrm.nodes', 'osrm.properties', 'osrm.ramIndex', 'osrm.restrictions', 'osrm.tld', 'osrm.tls'].forEach(file => {
-                 q.defer(rename, file);
-             });
-
-            ['osrm.edge_penalties', 'osrm.edge_segment_lookup'].forEach(file => {
-                q.defer(renameIfExists, file);
-            });
-
-            q.awaitAll((err) => {
-                this.log('Finished extracting ' + this.osmData.extractedFile, 'preprocess');
-                process.chdir('../');
-                callback(err);
-            });
         });
     };
 
-    this.contractData = (callback) => {
-        this.logPreprocessInfo();
-        this.log(util.format('== Contracting %s.osm...', this.osmData.extractedFile), 'preprocess');
-        var cmd = util.format('%s/osrm-contract %s %s.osrm >>%s 2>&1',
-            this.BIN_PATH, this.contractArgs || '', this.osmData.extractedFile, this.PREPROCESS_LOG_FILE);
-        this.log(cmd);
-        process.chdir(this.TEST_FOLDER);
-        exec(cmd, (err) => {
-            if (err) {
-                this.log(util.format('*** Exited with code %d', err.code), 'preprocess');
-                process.chdir('../');
-                return callback(this.ContractError(err.code, util.format('osrm-contract exited with code %d', err.code)));
+    this.linkOSM = (callback) => {
+        fs.exists(this.inputCacheFile, (exists) => {
+            if (exists) callback();
+            else {
+                fs.link(this.scenarioCacheFile, this.inputCacheFile, callback);
             }
+        });
+    };
 
-            var rename = (file, cb) => {
-                this.log(util.format('Renaming %s.%s to %s.%s', this.osmData.extractedFile, file, this.osmData.contractedFile, file), 'preprocess');
-                fs.rename([this.osmData.extractedFile, file].join('.'), [this.osmData.contractedFile, file].join('.'), (err) => {
-                    if (err) return cb(this.FileError(null, 'failed to rename data file after contracting.'));
-                    cb();
-                });
-            };
+    this.extractData = (p, callback) => {
+        let stamp = p.processedCacheFile + '.extract';
+        fs.exists(stamp, (exists) => {
+            if (exists) return callback();
 
-            var renameIfExists = (file, cb) => {
-                fs.stat([this.osmData.extractedFile, file].join('.'), (doesNotExistErr, exists) => {
-                    if (exists) rename(file, cb);
-                    else cb();
-                });
-            };
-
-            var copy = (file, cb) => {
-                this.log(util.format('Copying %s.%s to %s.%s', this.osmData.extractedFile, file, this.osmData.contractedFile, file), 'preprocess');
-                fs.createReadStream([this.osmData.extractedFile, file].join('.'))
-                    .pipe(fs.createWriteStream([this.osmData.contractedFile, file].join('.'))
-                            .on('finish', cb)
-                        )
-                    .on('error', () => {
-                        return cb(this.FileError(null, 'failed to copy data after contracting.'));
-                    });
-            };
-
-            var q = d3.queue();
-
-            ['osrm', 'osrm.core', 'osrm.datasource_indexes', 'osrm.datasource_names', 'osrm.ebg','osrm.edges',
-             'osrm.enw', 'osrm.fileIndex', 'osrm.geometry', 'osrm.hsgr', 'osrm.icd','osrm.level', 'osrm.names',
-             'osrm.nodes', 'osrm.properties', 'osrm.ramIndex', 'osrm.restrictions', 'osrm.tld', 'osrm.tls'].forEach((file) => {
-                 q.defer(rename, file);
-             });
-
-            ['osrm.edge_penalties', 'osrm.edge_segment_lookup'].forEach(file => {
-                q.defer(renameIfExists, file);
-            });
-
-            [].forEach((file) => {
-                q.defer(copy, file);
-            });
-
-            q.awaitAll((err) => {
-                this.log('Finished contracting ' + this.osmData.contractedFile, 'preprocess');
-                process.chdir('../');
-                callback(err);
+            this.runBin('osrm-extract', util.format('%s --profile %s %s', p.extractArgs, p.profileFile, p.inputCacheFile), p.environment, (err) => {
+                if (err) {
+                    return callback(new Error(util.format('osrm-extract %s: %s', errorReason(err), err.cmd)));
+                }
+                fs.writeFile(stamp, 'ok', callback);
             });
         });
     };
 
-    var noop = (cb) => cb();
+    this.contractData = (p, callback) => {
+        let stamp = p.processedCacheFile + '.contract';
+        fs.exists(stamp, (exists) => {
+            if (exists) return callback();
+
+            this.runBin('osrm-contract', util.format('%s %s', p.contractArgs, p.processedCacheFile), p.environment, (err) => {
+                if (err) {
+                    return callback(new Error(util.format('osrm-contract %s: %s', errorReason(err), err)));
+                }
+                fs.writeFile(stamp, 'ok', callback);
+            });
+        });
+    };
+
+    this.extractAndContract = (callback) => {
+        // a shallow copy of scenario parameters to avoid data inconsistency
+        // if a cucumber timeout occurs during deferred jobs
+        let p = {extractArgs: this.extractArgs, contractArgs: this.contractArgs,
+                 profileFile: this.profileFile, inputCacheFile: this.inputCacheFile,
+                 processedCacheFile: this.processedCacheFile, environment: this.environment};
+        let queue = d3.queue(1);
+        queue.defer(this.extractData.bind(this), p);
+        queue.defer(this.contractData.bind(this), p);
+        queue.awaitAll(callback);
+    };
 
     this.reprocess = (callback) => {
-        this.osmData.populate(() => {
-            this.isContracted((isContracted) => {
-                if (!isContracted) {
-                    this.writeAndExtract((e) => {
-                        if (e) return callback(e);
-                        this.contractData((e) => {
-                            if (e) return callback(e);
-                            this.logPreprocessDone();
-                            callback();
-                        });
-                    });
-                } else {
-                    this.log('Already contracted ' + this.osmData.contractedFile, 'preprocess');
-                    callback();
-                }
-            });
-        });
-    };
-
-    this.writeAndExtract = (callback) => {
-        this.writeInputData((e) => {
-            if (e) return callback(e);
-            this.isExtracted((isExtracted) => {
-                var extractFn = isExtracted ? noop : this.extractData;
-                if (isExtracted) this.log('Already extracted ' + this.osmData.extractedFile, 'preprocess');
-                extractFn((e) => {
-                    callback(e);
-                });
-            });
-        });
+        let queue = d3.queue(1);
+        queue.defer(this.writeOSM.bind(this));
+        queue.defer(this.linkOSM.bind(this));
+        queue.defer(this.extractAndContract.bind(this));
+        queue.awaitAll(callback);
     };
 
     this.reprocessAndLoadData = (callback) => {
-        this.reprocess((e) => {
-            if (e) return callback(e);
-            this.OSRMLoader.load(util.format('%s.osrm', this.osmData.contractedFile), callback);
-        });
+        let queue = d3.queue(1);
+        queue.defer(this.writeOSM.bind(this));
+        queue.defer(this.linkOSM.bind(this));
+        queue.defer(this.extractAndContract.bind(this));
+        queue.defer(this.osrmLoader.load.bind(this.osrmLoader), this.processedCacheFile);
+        queue.awaitAll(callback);
     };
 
     this.processRowsAndDiff = (table, fn, callback) => {
@@ -352,7 +228,9 @@ module.exports = function () {
 
         q.awaitAll((err, actual) => {
             if (err) return callback(err);
-            this.diffTables(table, actual, {}, callback);
+            let diff = tableDiff(table, actual);
+            if (diff) callback(new Error(diff));
+            else callback();
         });
     };
 };

features/support/data_classes.js

@@ -1,7 +1,6 @@
 'use strict';
 
-var util = require('util');
-var path = require('path');
+const util = require('util');
 
 module.exports = {
     Location: class {
@@ -11,43 +10,6 @@ module.exports = {
         }
     },
 
-    osmData: class {
-        constructor (scope) {
-            this.scope          = scope;
-            this.str            = null;
-            this.hash           = null;
-            this.fingerprintOSM = null;
-            this.osmFile        = null;
-            this.extractedFile  = null;
-            this.contractedFile   = null;
-        }
-
-        populate (callback) {
-            this.scope.OSMDB.toXML((str) => {
-                this.str = str;
-
-                this.hash = this.scope.hashString(str);
-                this.fingerprintOSM = this.scope.hashString(this.hash);
-
-                this.osmFile        = path.resolve(this.scope.DATA_FOLDER, this.fingerprintOSM);
-
-                this.extractedFile  = path.resolve([this.osmFile, this.scope.fingerprintExtract].join('_'));
-                this.contractedFile   = path.resolve([this.osmFile, this.scope.fingerprintExtract, this.scope.fingerprintContract].join('_'));
-
-                callback();
-            });
-        }
-
-        reset () {
-            this.str            = null;
-            this.hash           = null;
-            this.fingerprintOSM = null;
-            this.osmFile        = null;
-            this.extractedFile  = null;
-            this.contractedFile   = null;
-        }
-    },
-
     FuzzyMatch: class {
         match (got, want) {
             var matchPercent = want.match(/(.*)\s+~(.+)%$/),

features/support/env.js

@@ -1,32 +1,46 @@
-var path = require('path');
-var util = require('util');
-var fs = require('fs');
-var exec = require('child_process').exec;
-var d3 = require('d3-queue');
+'use strict';
 
+const path = require('path');
+const util = require('util');
+const fs = require('fs');
+const d3 = require('d3-queue');
+const child_process = require('child_process');
+const tryConnect = require('../lib/try_connect');
+
+// Sets up all constants that are valid for all features
 module.exports = function () {
     this.initializeEnv = (callback) => {
-        this.OSRM_PORT = process.env.OSRM_PORT && parseInt(process.env.OSRM_PORT) || 5000;
         this.TIMEOUT = process.env.CUCUMBER_TIMEOUT && parseInt(process.env.CUCUMBER_TIMEOUT) || 5000;
+        // set cucumber default timeout
         this.setDefaultTimeout(this.TIMEOUT);
-        this.ROOT_FOLDER = process.cwd();
+        this.ROOT_PATH = process.cwd();
+
+        this.TEST_PATH = path.resolve(this.ROOT_PATH, 'test');
+        this.CACHE_PATH = path.resolve(this.TEST_PATH, 'cache');
+        this.LOGS_PATH = path.resolve(this.TEST_PATH, 'logs');
+
+        this.PROFILES_PATH = path.resolve(this.ROOT_PATH, 'profiles');
+        this.FIXTURES_PATH = path.resolve(this.ROOT_PATH, 'unit_tests/fixtures');
+        this.BIN_PATH = process.env.OSRM_BUILD_DIR && process.env.OSRM_BUILD_DIR || path.resolve(this.ROOT_PATH, 'build');
+        var stxxl_config = path.resolve(this.ROOT_PATH, 'test/.stxxl');
+        if (!fs.existsSync(stxxl_config)) {
+            return callback(new Error('*** '+stxxl_config+ 'does not exist'));
+        }
+
+        this.DEFAULT_ENVIRONMENT = Object.assign({STXXLCFG: stxxl_config}, process.env);
+        this.DEFAULT_PROFILE = 'bicycle';
+        this.DEFAULT_INPUT_FORMAT = 'osm';
+        this.DEFAULT_LOAD_METHOD = 'datastore';
+        this.DEFAULT_ORIGIN = [1,1];
         this.OSM_USER = 'osrm';
         this.OSM_GENERATOR = 'osrm-test';
         this.OSM_UID = 1;
-        this.TEST_FOLDER = path.resolve(this.ROOT_FOLDER, 'test');
-        this.DATA_FOLDER = path.resolve(this.TEST_FOLDER, 'cache');
         this.OSM_TIMESTAMP = '2000-01-01T00:00:00Z';
-        this.DEFAULT_SPEEDPROFILE = 'bicycle';
         this.WAY_SPACING = 100;
-        this.DEFAULT_GRID_SIZE = 100;    // meters
-        this.PROFILES_PATH = path.resolve(this.ROOT_FOLDER, 'profiles');
-        this.FIXTURES_PATH = path.resolve(this.ROOT_FOLDER, 'unit_tests/fixtures');
-        this.BIN_PATH = process.env.OSRM_BUILD_DIR && process.env.OSRM_BUILD_DIR || path.resolve(this.ROOT_FOLDER, 'build');
-        this.DEFAULT_INPUT_FORMAT = 'osm';
-        this.DEFAULT_ORIGIN = [1,1];
-        this.DEFAULT_LOAD_METHOD = 'datastore';
-        this.OSRM_ROUTED_LOG_FILE = path.resolve(this.TEST_FOLDER, 'osrm-routed.log');
-        this.ERROR_LOG_FILE = path.resolve(this.TEST_FOLDER, 'error.log');
+        this.DEFAULT_GRID_SIZE = 100; // meters
+
+        this.OSRM_PORT = process.env.OSRM_PORT && parseInt(process.env.OSRM_PORT) || 5000;
+        this.HOST = 'http://127.0.0.1:' + this.OSRM_PORT;
 
         // TODO make sure this works on win
         if (process.platform.match(/indows.*/)) {
@@ -37,36 +51,49 @@ module.exports = function () {
         } else {
             this.TERMSIGNAL = 'SIGTERM';
             this.EXE = '';
-            this.LIB = '.so';
+            // TODO autodetect if this was build with shared or static libraries
+            this.LIB = process.env.BUILD_SHARED_LIBS && '.so' || '.a';
             this.QQ = '';
         }
 
+        this.OSRM_EXTRACT_PATH = path.resolve(util.format('%s/%s%s', this.BIN_PATH, 'osrm-extract', this.EXE));
+        this.OSRM_CONTRACT_PATH = path.resolve(util.format('%s/%s%s', this.BIN_PATH, 'osrm-contract', this.EXE));
+        this.OSRM_ROUTED_PATH = path.resolve(util.format('%s/%s%s', this.BIN_PATH, 'osrm-routed', this.EXE));
+        this.LIB_OSRM_EXTRACT_PATH = util.format('%s/libosrm_extract%s', this.BIN_PATH, this.LIB),
+        this.LIB_OSRM_CONTRACT_PATH = util.format('%s/libosrm_contract%s', this.BIN_PATH, this.LIB),
+        this.LIB_OSRM_PATH = util.format('%s/libosrm%s', this.BIN_PATH, this.LIB);
+
         // eslint-disable-next-line no-console
         console.info(util.format('Node Version', process.version));
         if (parseInt(process.version.match(/v(\d)/)[1]) < 4) throw new Error('*** PLease upgrade to Node 4.+ to run OSRM cucumber tests');
 
-        fs.exists(this.TEST_FOLDER, (exists) => {
-            if (!exists) throw new Error(util.format('*** Test folder %s doesn\'t exist.', this.TEST_FOLDER));
-            callback();
+        fs.exists(this.TEST_PATH, (exists) => {
+            if (exists)
+                return callback();
+            else
+                return callback(new Error('*** Test folder doesn\'t exist.'));
         });
     };
 
-    this.verifyOSRMIsNotRunning = () => {
-        if (this.OSRMLoader.up()) {
-            throw new Error('*** osrm-routed is already running.');
-        }
+    this.getProfilePath = (profile) => {
+        return path.resolve(this.PROFILES_PATH, profile + '.lua');
+    };
+
+    this.verifyOSRMIsNotRunning = (callback) => {
+        tryConnect(this.OSRM_PORT, (err) => {
+            if (!err) return callback(new Error('*** osrm-routed is already running.'));
+            else callback();
+        });
     };
 
     this.verifyExistenceOfBinaries = (callback) => {
-        var verify = (bin, cb) => {
-            var binPath = path.resolve(util.format('%s/%s%s', this.BIN_PATH, bin, this.EXE));
+        var verify = (binPath, cb) => {
             fs.exists(binPath, (exists) => {
-                if (!exists) throw new Error(util.format('%s is missing. Build failed?', binPath));
+                if (!exists) return cb(new Error(util.format('%s is missing. Build failed?', binPath)));
                 var helpPath = util.format('%s --help > /dev/null 2>&1', binPath);
-                exec(helpPath, (err) => {
+                child_process.exec(helpPath, (err) => {
                     if (err) {
-                        this.log(util.format('*** Exited with code %d', err.code), 'preprocess');
-                        throw new Error(util.format('*** %s exited with code %d', helpPath, err.code));
+                        return cb(new Error(util.format('*** %s exited with code %d', helpPath, err.code)));
                     }
                     cb();
                 });
@@ -74,23 +101,12 @@ module.exports = function () {
         };
 
         var q = d3.queue();
-        ['osrm-extract', 'osrm-contract', 'osrm-routed'].forEach(bin => { q.defer(verify, bin); });
-        q.awaitAll(() => {
-            callback();
-        });
-    };
-
-    this.AfterConfiguration = (callback) => {
-        this.clearLogFiles(() => {
-            this.verifyOSRMIsNotRunning();
-            this.verifyExistenceOfBinaries(() => {
-                callback();
-            });
-        });
+        [this.OSRM_EXTRACT_PATH, this.OSRM_CONTRACT_PATH, this.OSRM_ROUTED_PATH].forEach(bin => { q.defer(verify, bin); });
+        q.awaitAll(callback);
     };
 
     process.on('exit', () => {
-        if (this.OSRMLoader.loader) this.OSRMLoader.shutdown(() => {});
+        this.osrmLoader.shutdown(() => {});
     });
 
     process.on('SIGINT', () => {

features/support/exception_classes.js

@@ -1,132 +0,0 @@
-'use strict';
-
-var util = require('util');
-var path = require('path');
-var fs = require('fs');
-var chalk = require('chalk');
-
-var OSRMError = class extends Error {
-    constructor (process, code, msg, log, lines) {
-        super(msg);
-        this.process = process;
-        this.code = code;
-        this.msg = msg;
-        this.lines = lines;
-        this.log = log;
-    }
-
-    extract (callback) {
-        this.logTail(this.log, this.lines, callback);
-    }
-
-    // toString (callback) {
-    //     this.extract((tail) => {
-    //         callback(util.format('*** %s\nLast %s from %s:\n%s\n', this.msg, this.lines, this.log, tail));
-    //     });
-    // }
-
-    logTail (logPath, n, callback) {
-        var expanded = path.resolve(this.TEST_FOLDER, logPath);
-        fs.exists(expanded, (exists) => {
-            if (exists) {
-                fs.readFile(expanded, (err, data) => {
-                    var lines = data.toString().trim().split('\n');
-                    callback(lines
-                        .slice(lines.length - n)
-                        .map(line => util.format('    %s', line))
-                        .join('\n'));
-                });
-            } else {
-                callback(util.format('File %s does not exist!', expanded));
-            }
-        });
-    }
-};
-
-var unescapeStr = (str) => str.replace(/\\\|/g, '\|').replace(/\\\\/g, '\\');
-
-module.exports = {
-    OSRMError: OSRMError,
-
-    FileError: class extends OSRMError {
-        constructor (logFile, code, msg) {
-            super ('fileutil', code, msg, logFile, 5);
-        }
-    },
-
-    LaunchError: class extends OSRMError {
-        constructor (logFile, launchProcess, code, msg) {
-            super (launchProcess, code, msg, logFile, 5);
-        }
-    },
-
-    ExtractError: class extends OSRMError {
-        constructor (logFile, code, msg) {
-            super('osrm-extract', code, msg, logFile, 3);
-        }
-    },
-
-    ContractError:  class extends OSRMError {
-        constructor (logFile, code, msg) {
-            super('osrm-contract', code, msg, logFile, 3);
-        }
-    },
-
-    RoutedError: class extends OSRMError {
-        constructor (logFile, msg) {
-            super('osrm-routed', null, msg, logFile, 3);
-        }
-    },
-
-    TableDiffError: class extends Error {
-        constructor (expected, actual) {
-            super();
-            this.headers = expected.raw()[0];
-            this.expected = expected.hashes();
-            this.actual = actual;
-            this.diff = [];
-            this.hasErrors = false;
-
-            var good = 0, bad = 0;
-
-            this.expected.forEach((row, i) => {
-                var rowError = false;
-
-                for (var j in row) {
-                    if (unescapeStr(row[j]) != actual[i][j]) {
-                        rowError = true;
-                        this.hasErrors = true;
-                        break;
-                    }
-                }
-
-                if (rowError) {
-                    bad++;
-                    this.diff.push(Object.assign({}, row, {c_status: 'undefined'}));
-                    this.diff.push(Object.assign({}, actual[i], {c_status: 'comment'}));
-                } else {
-                    good++;
-                    this.diff.push(row);
-                }
-            });
-        }
-
-        get string () {
-            if (!this.hasErrors) return null;
-
-            var s = ['Tables were not identical:'];
-            s.push(this.headers.map(key => '    ' + key).join(' | '));
-            this.diff.forEach((row) => {
-                var rowString = '| ';
-                this.headers.forEach((header) => {
-                    if (!row.c_status) rowString += chalk.green('    ' + row[header] + ' | ');
-                    else if (row.c_status === 'undefined') rowString += chalk.yellow('(-) ' + row[header] + ' | ');
-                    else rowString += chalk.red('(+) ' + row[header] + ' | ');
-                });
-                s.push(rowString);
-            });
-
-            return s.join('\n') + '\nTODO this is a temp workaround waiting for https://github.com/cucumber/cucumber-js/issues/534';
-        }
-    }
-};

features/support/exceptions.js

@@ -1,15 +0,0 @@
-var exceptions = require('./exception_classes');
-
-module.exports = function () {
-    this.OSRMError = exceptions.OSRMError,
-
-    this.FileError = (code, msg) => new (exceptions.FileError.bind(exceptions.FileError, this.PREPROCESS_LOG_FILE))(code, msg);
-
-    this.LaunchError = (code, launchProcess, msg) => new (exceptions.LaunchError.bind(exceptions.LaunchError, this.ERROR_LOG_FILE))(code, launchProcess, msg);
-
-    this.ExtractError = (code, msg) => new (exceptions.ExtractError.bind(exceptions.ExtractError, this.PREPROCESS_LOG_FILE))(code, msg);
-
-    this.ContractError = (code, msg) => new (exceptions.ContractError.bind(exceptions.ContractError, this.PREPROCESS_LOG_FILE))(code, msg);
-
-    this.RoutedError = (msg) => new (exceptions.RoutedError.bind(exceptions.RoutedError, this.OSRM_ROUTED_LOG_FILE))(msg);
-};

features/support/hash.js

@@ -1,43 +0,0 @@
-var fs = require('fs');
-var path = require('path');
-var crypto = require('crypto');
-var d3 = require('d3-queue');
-
-module.exports = function () {
-    this.hashOfFiles = (paths, cb) => {
-        paths = Array.isArray(paths) ? paths : [paths];
-        var shasum = crypto.createHash('sha1'), hashedFiles = false;
-
-        var q = d3.queue(1);
-
-        var addFile = (path, cb) => {
-            fs.readFile(path, (err, data) => {
-                if (err && err.code === 'ENOENT') cb(); // ignore non-existing files
-                else if (err) cb(err);
-                else {
-                    shasum.update(data);
-                    hashedFiles = true;
-                    cb();
-                }
-            });
-        };
-
-        paths.forEach(path => { q.defer(addFile, path); });
-
-        q.awaitAll(err => {
-            if (err) throw new Error('*** Error reading files:', err);
-            if (!hashedFiles) throw new Error('*** No files found: [' + paths.join(', ') + ']');
-            cb(shasum.digest('hex'));
-        });
-    };
-
-    this.hashProfile = (cb) => {
-        this.hashOfFiles(path.resolve(this.PROFILES_PATH, this.profile + '.lua'), cb);
-    };
-
-    this.hashString = (str) => {
-        return crypto.createHash('sha1').update(str).digest('hex');
-    };
-
-    return this;
-};

features/support/hooks.js

@@ -1,36 +1,61 @@
-var util = require('util');
+'use strict';
+
+var d3 = require('d3-queue');
+var path = require('path');
+var mkdirp = require('mkdirp');
+var rimraf = require('rimraf');
+var OSM = require('../lib/osm');
+var OSRMLoader = require('../lib/osrm_loader');
 
 module.exports = function () {
-    this.BeforeFeatures((features, callback) => {
-        this.pid = null;
-        this.initializeEnv(() => {
-            this.initializeOptions(callback);
-        });
+    this.registerHandler('BeforeFeatures', {timeout: 30000},  (features, callback) => {
+        this.osrmLoader = new OSRMLoader(this);
+        this.OSMDB = new OSM.DB();
+
+        let queue = d3.queue(1);
+        queue.defer(this.initializeEnv.bind(this));
+        queue.defer(this.verifyOSRMIsNotRunning.bind(this));
+        queue.defer(this.verifyExistenceOfBinaries.bind(this));
+        queue.defer(this.initializeCache.bind(this));
+        queue.defer(this.setupFeatures.bind(this, features));
+        queue.awaitAll(callback);
     });
 
-    this.Before((scenario, callback) => {
-        this.scenarioTitle = scenario.getName();
-
-        this.loadMethod = this.DEFAULT_LOAD_METHOD;
-        this.queryParams = {};
-        var d = new Date();
-        this.scenarioTime = util.format('%d-%d-%dT%s:%s:%sZ', d.getFullYear(), d.getMonth()+1, d.getDate(), d.getHours(), d.getMinutes(), d.getSeconds());
-        this.resetData();
-        this.hasLoggedPreprocessInfo = false;
-        this.hasLoggedScenarioInfo = false;
-        this.setGridSize(this.DEFAULT_GRID_SIZE);
-        this.setOrigin(this.DEFAULT_ORIGIN);
-        this.fingerprintExtract = this.hashString([this.luaLibHash, this.binExtractHash].join('-'));
-        this.fingerprintContract = this.hashString(this.binContractHash);
+    this.BeforeFeature((feature, callback) => {
+        this.profile = this.DEFAULT_PROFILE;
+        this.profileFile = path.join(this.PROFILES_PATH, this.profile + '.lua');
+        this.setupFeatureCache(feature);
         callback();
     });
 
+    this.Before((scenario, callback) => {
+        this.osrmLoader.setLoadMethod(this.DEFAULT_LOAD_METHOD);
+        this.setGridSize(this.DEFAULT_GRID_SIZE);
+        this.setOrigin(this.DEFAULT_ORIGIN);
+        this.queryParams = {};
+        this.extractArgs = '';
+        this.contractArgs = '';
+        this.environment = Object.assign(this.DEFAULT_ENVIRONMENT);
+        this.resetOSM();
+
+        this.scenarioID = this.getScenarioID(scenario);
+        this.setupScenarioCache(this.scenarioID);
+
+        // setup output logging
+        let logDir = path.join(this.LOGS_PATH, this.featureID);
+        this.scenarioLogFile = path.join(logDir, this.scenarioID) + '.log';
+        d3.queue(1)
+            .defer(mkdirp, logDir)
+            .defer(rimraf, this.scenarioLogFile)
+            .awaitAll(callback);
+    });
+
     this.After((scenario, callback) => {
-        this.setExtractArgs('', () => {
-            this.setContractArgs('', () => {
-                if (this.loadMethod === 'directly' && !!this.OSRMLoader.loader) this.OSRMLoader.shutdown(callback);
-                else callback();
-            });
-        });
+        this.resetOptionsOutput();
+        callback();
+    });
+
+    this.AfterFeatures((features, callback) => {
+        callback();
     });
 };

features/support/http.js

@@ -19,6 +19,9 @@ module.exports = function () {
         return paramString;
     };
 
+    // FIXME this needs to be simplified!
+    // - remove usage of node-timeout
+    // - replace with node's native timout mechanism
     this.sendRequest = (baseUri, parameters, callback) => {
         var limit = Timeout(this.TIMEOUT, { err: { statusCode: 408 } });
 
@@ -28,9 +31,9 @@ module.exports = function () {
 
             request(this.query, (err, res, body) => {
                 if (err && err.code === 'ECONNREFUSED') {
-                    throw new Error('*** osrm-routed is not running.');
+                    return cb(new Error('*** osrm-routed is not running.'));
                 } else if (err && err.statusCode === 408) {
-                    throw new Error();
+                    return cb(new Error());
                 }
 
                 return cb(err, res, body);
@@ -40,11 +43,10 @@ module.exports = function () {
         runRequest(limit((err, res, body) => {
             if (err) {
                 if (err.statusCode === 408)
-                    return callback(this.RoutedError('*** osrm-routed did not respond'));
+                    return callback(new Error('*** osrm-routed did not respond'));
                 else if (err.code === 'ECONNREFUSED')
-                    return callback(this.RoutedError('*** osrm-routed is not running'));
+                    return callback(new Error('*** osrm-routed is not running'));
             }
-            //console.log(body+"\n");
             return callback(err, res, body);
         }));
     };

features/support/launch.js

@@ -1,5 +0,0 @@
-var launchClasses = require('./launch_classes');
-
-module.exports = function () {
-    this._OSRMLoader = () => new (launchClasses._OSRMLoader.bind(launchClasses._OSRMLoader, this))();
-};

features/support/launch_classes.js

@@ -1,164 +0,0 @@
-'use strict';
-
-var fs = require('fs');
-var spawn = require('child_process').spawn;
-var util = require('util');
-var net = require('net');
-var Timeout = require('node-timeout');
-
-var OSRMBaseLoader = class {
-    constructor (scope) {
-        this.scope = scope;
-    }
-
-    launch (callback) {
-        var limit = Timeout(this.scope.TIMEOUT, { err: this.scope.RoutedError('Launching osrm-routed timed out.') });
-
-        var runLaunch = (cb) => {
-            this.osrmUp(() => { this.waitForConnection(cb); });
-        };
-
-        runLaunch(limit((e) => { if (e) callback(e); else callback(); }));
-    }
-
-    shutdown (callback) {
-        var limit = Timeout(this.scope.TIMEOUT, { err: this.scope.RoutedError('Shutting down osrm-routed timed out.')});
-
-        var runShutdown = (cb) => {
-            this.osrmDown(cb);
-        };
-
-        runShutdown(limit((e) => { if (e) callback(e); else callback(); }));
-    }
-
-    osrmIsRunning () {
-        return !!this.scope.pid && this.child && !this.child.killed;
-    }
-
-    osrmDown (callback) {
-        if (this.scope.pid) {
-            process.kill(this.scope.pid, this.scope.TERMSIGNAL);
-            this.waitForShutdown(callback);
-            this.scope.pid = null;
-        } else callback(true);
-    }
-
-    waitForConnection (callback) {
-        var retryCount = 0;
-        var connectWithRetry = () => {
-            net.connect({ port: this.scope.OSRM_PORT, host: '127.0.0.1' })
-               .on('connect', () => { callback(); })
-               .on('error', () => {
-                   if (retryCount < 2) {
-                       retryCount++;
-                       setTimeout(connectWithRetry, 100);
-                   } else {
-                       callback(new Error('Could not connect to osrm-routed after three retires'));
-                   }
-               });
-        };
-
-        connectWithRetry();
-    }
-
-    waitForShutdown (callback) {
-        var check = () => {
-            if (!this.osrmIsRunning()) return callback();
-        };
-        setTimeout(check, 100);
-    }
-};
-
-var OSRMDirectLoader = class extends OSRMBaseLoader {
-    constructor (scope) {
-        super(scope);
-    }
-
-    load (inputFile, callback) {
-        this.inputFile = inputFile;
-        this.shutdown(() => {
-            this.launch(callback);
-        });
-    }
-
-    osrmUp (callback) {
-        if (this.scope.pid) return callback();
-        var writeToLog = (data) => {
-            fs.appendFile(this.scope.OSRM_ROUTED_LOG_FILE, data, (err) => { if (err) throw err; });
-        };
-
-        var child = spawn(util.format('%s/osrm-routed', this.scope.BIN_PATH), [this.inputFile, util.format('-p%d', this.scope.OSRM_PORT)]);
-        this.scope.pid = child.pid;
-        child.stdout.on('data', writeToLog);
-        child.stderr.on('data', writeToLog);
-
-        callback();
-    }
-};
-
-var OSRMDatastoreLoader = class extends OSRMBaseLoader {
-    constructor (scope) {
-        super(scope);
-    }
-
-    load (inputFile, callback) {
-        this.inputFile = inputFile;
-        this.loadData((err) => {
-            if (err) return callback(err);
-            if (!this.scope.pid) return this.launch(callback);
-            else callback();
-        });
-    }
-
-    loadData (callback) {
-        this.scope.runBin('osrm-datastore', this.inputFile, (err) => {
-            if (err) return callback(this.scope.LaunchError(this.exitCode, 'datastore', err));
-            callback();
-        });
-    }
-
-    osrmUp (callback) {
-        if (this.scope.pid) return callback();
-        var writeToLog = (data) => {
-            fs.appendFile(this.scope.OSRM_ROUTED_LOG_FILE, data, (err) => { if (err) throw err; });
-        };
-
-        var child = spawn(util.format('%s/osrm-routed', this.scope.BIN_PATH), ['--shared-memory=1', util.format('-p%d', this.scope.OSRM_PORT)]);
-        this.child = child;
-        this.scope.pid = child.pid;
-        child.stdout.on('data', writeToLog);
-        child.stderr.on('data', writeToLog);
-
-        callback();
-    }
-};
-
-module.exports = {
-    _OSRMLoader: class {
-        constructor (scope) {
-            this.scope = scope;
-            this.loader = null;
-        }
-
-        load (inputFile, callback) {
-            var method = this.scope.loadMethod;
-            if (method === 'datastore') {
-                this.loader = new OSRMDatastoreLoader(this.scope);
-                this.loader.load(inputFile, callback);
-            } else if (method === 'directly') {
-                this.loader = new OSRMDirectLoader(this.scope);
-                this.loader.load(inputFile, callback);
-            } else {
-                callback(new Error('*** Unknown load method ' + method));
-            }
-        }
-
-        shutdown (callback) {
-            this.loader.shutdown(callback);
-        }
-
-        up () {
-            return this.loader ? this.loader.osrmIsRunning() : false;
-        }
-    }
-};

features/support/log.js

@@ -1,90 +0,0 @@
-var fs = require('fs');
-
-module.exports = function () {
-    this.clearLogFiles = (callback) => {
-        // emptying existing files, rather than deleting and writing new ones makes it
-        // easier to use tail -f from the command line
-        fs.writeFile(this.OSRM_ROUTED_LOG_FILE, '', err => {
-            if (err) throw err;
-            fs.writeFile(this.PREPROCESS_LOG_FILE, '', err => {
-                if (err) throw err;
-                fs.writeFile(this.LOG_FILE, '', err => {
-                    if (err) throw err;
-                    callback();
-                });
-            });
-        });
-    };
-
-    var log = this.log = (s, type) => {
-        s = s || '';
-        type = type || null;
-        var file = type === 'preprocess' ? this.PREPROCESS_LOG_FILE : this.LOG_FILE;
-        fs.appendFile(file, s + '\n', err => {
-            if (err) throw err;
-        });
-    };
-
-    this.logScenarioFailInfo = () => {
-        if (this.hasLoggedScenarioInfo) return;
-
-        log('=========================================');
-        log('Failed scenario: ' + this.scenarioTitle);
-        log('Time: ' + this.scenarioTime);
-        log('Fingerprint osm stage: ' + this.osmData.fingerprintOSM);
-        log('Fingerprint extract stage: ' + this.fingerprintExtract);
-        log('Fingerprint contract stage: ' + this.fingerprintContract);
-        log('Fingerprint route stage: ' + this.fingerprintRoute);
-        log('Profile: ' + this.profile);
-        log();
-        log('```xml');               // so output can be posted directly to github comment fields
-        log(this.osmData.str.trim());
-        log('```');
-        log();
-        log();
-
-        this.hasLoggedScenarioInfo = true;
-    };
-
-    this.logFail = (expected, got, attempts) => {
-        this.logScenarioFailInfo();
-        log('== ');
-        log('Expected: ' + JSON.stringify(expected));
-        log('Got:      ' + JSON.stringify(got));
-        log();
-        ['route','forw','backw'].forEach((direction) => {
-            if (attempts[direction]) {
-                log('Direction: ' + direction);
-                log('Query: ' + attempts[direction].query);
-                log('Response: ' + attempts[direction].response.body);
-                log();
-            }
-        });
-    };
-
-    this.logPreprocessInfo = () => {
-        if (this.hasLoggedPreprocessInfo) return;
-        log('=========================================', 'preprocess');
-        log('Preprocessing data for scenario: ' + this.scenarioTitle, 'preprocess');
-        log('Time: ' + this.scenarioTime, 'preprocess');
-        log('', 'preprocess');
-        log('== OSM data:', 'preprocess');
-        log('```xml', 'preprocess');            // so output can be posted directly to github comment fields
-        log(this.osmData.str, 'preprocess');
-        log('```', 'preprocess');
-        log('', 'preprocess');
-        log('== Profile:', 'preprocess');
-        log(this.profile, 'preprocess');
-        log('', 'preprocess');
-        this.hasLoggedPreprocessInfo = true;
-    };
-
-    this.logPreprocess = (str) => {
-        this.logPreprocessInfo();
-        log(str, 'preprocess');
-    };
-
-    this.logPreprocessDone = () => {
-        log('Done with preprocessing at ' + new Date(), 'preprocess');
-    };
-};

features/support/route.js

@@ -1,7 +1,8 @@
 'use strict';
 
-var Timeout = require('node-timeout');
-var request = require('request');
+const Timeout = require('node-timeout');
+const request = require('request');
+const ensureDecimal = require('../lib/utils').ensureDecimal;
 
 module.exports = function () {
     this.requestPath = (service, params, callback) => {
@@ -42,7 +43,7 @@ module.exports = function () {
     };
 
     var encodeWaypoints = (waypoints) => {
-        return waypoints.map(w => [w.lon, w.lat].map(this.ensureDecimal).join(','));
+        return waypoints.map(w => [w.lon, w.lat].map(ensureDecimal).join(','));
     };
 
     this.requestRoute = (waypoints, bearings, userParams, callback) => {

features/support/run.js

@@ -1,40 +1,52 @@
-var fs = require('fs');
-var util = require('util');
-var exec = require('child_process').exec;
+'use strict';
+
+const fs = require('fs');
+const util = require('util');
+const child_process = require('child_process');
 
 module.exports = function () {
-    this.runBin = (bin, options, callback) => {
-        var opts = options.slice();
+    // replaces placeholders for in user supplied commands
+    this.expandOptions = (options) => {
+        let opts = options.slice();
+        let table = {
+            '{osm_file}': this.inputCacheFile,
+            '{processed_file}': this.processedCacheFile,
+            '{profile_file}': this.profileFile,
+            '{rastersource_file}': this.rasterCacheFile,
+            '{speeds_file}': this.speedsCacheFile,
+            '{penalties_file}': this.penaltiesCacheFile
+        };
 
-        if (opts.match('{osm_base}')) {
-            if (!this.osmData.osmFile) throw new Error('*** {osm_base} is missing');
-            opts = opts.replace('{osm_base}', this.osmData.osmFile);
+        for (let k in table) {
+            opts = opts.replace(k, table[k]);
         }
 
-        if (opts.match('{extracted_base}')) {
-            if (!this.osmData.extractedFile) throw new Error('*** {extracted_base} is missing');
-            opts = opts.replace('{extracted_base}', this.osmData.extractedFile);
+        return opts;
+    };
+
+    this.setupOutputLog = (process, log) => {
+        if (process.logFunc) {
+            process.stdout.removeListener('data', process.logFunc);
+            process.stderr.removeListener('data', process.logFunc);
         }
 
-        if (opts.match('{contracted_base}')) {
-            if (!this.osmData.contractedFile) throw new Error('*** {contracted_base} is missing');
-            opts = opts.replace('{contracted_base}', this.osmData.contractedFile);
-        }
+        process.logFunc = (message) => { log.write(message); };
+        process.stdout.on('data', process.logFunc);
+        process.stderr.on('data', process.logFunc);
+    };
 
-        if (opts.match('{profile}')) {
-            opts = opts.replace('{profile}', [this.PROFILES_PATH, this.profile + '.lua'].join('/'));
-        }
-
-        var cmd = util.format('%s%s/%s%s%s %s 2>%s', this.QQ, this.BIN_PATH, bin, this.EXE, this.QQ, opts, this.ERROR_LOG_FILE);
-        process.chdir(this.TEST_FOLDER);
-        exec(cmd, (err, stdout, stderr) => {
-            this.stdout = stdout.toString();
-            fs.readFile(this.ERROR_LOG_FILE, (e, data) => {
-                this.stderr = data ? data.toString() : '';
-                this.exitCode = err && err.code || 0;
-                process.chdir('../');
-                callback(err, stdout, stderr);
-            });
-        });
+    this.runBin = (bin, options, env, callback) => {
+        let cmd = util.format('%s%s/%s%s%s', this.QQ, this.BIN_PATH, bin, this.EXE, this.QQ);
+        let opts = options.split(' ').filter((x) => { return x && x.length > 0; });
+        let log = fs.createWriteStream(this.scenarioLogFile, {'flags': 'a'});
+        log.write(util.format('*** running %s %s\n', cmd, options));
+        // we need to set a large maxbuffer here because we have long running processes like osrm-routed
+        // with lots of log output
+        let child = child_process.execFile(cmd, opts, {maxBuffer: 1024 * 1024 * 1000, env: env}, callback);
+        child.on('exit', function(code) {
+            log.write(util.format('*** %s exited with code %d\n', bin, code));
+        }.bind(this));
+        this.setupOutputLog(child, log);
+        return child;
     };
 };

features/support/shared_steps.js

@@ -93,7 +93,7 @@ module.exports = function () {
                             if (headers.has('distance')) {
                                 if (row.distance.length) {
                                     if (!row.distance.match(/\d+m/))
-                                        throw new Error('*** Distance must be specified in meters. (ex: 250m)');
+                                        return cb(new Error('*** Distance must be specified in meters. (ex: 250m)'));
                                     got.distance = instructions ? util.format('%dm', distance) : '';
                                 } else {
                                     got.distance = '';
@@ -102,7 +102,7 @@ module.exports = function () {
 
                             if (headers.has('time')) {
                                 if (!row.time.match(/\d+s/))
-                                    throw new Error('*** Time must be specied in seconds. (ex: 60s)');
+                                    return cb(new Error('*** Time must be specied in seconds. (ex: 60s)'));
                                 got.time = instructions ? util.format('%ds', time) : '';
                             }
 
@@ -113,7 +113,7 @@ module.exports = function () {
                             if (headers.has('speed')) {
                                 if (row.speed !== '' && instructions) {
                                     if (!row.speed.match(/\d+ km\/h/))
-                                        throw new Error('*** Speed must be specied in km/h. (ex: 50 km/h)');
+                                        cb(new Error('*** Speed must be specied in km/h. (ex: 50 km/h)'));
                                     var speed = time > 0 ? Math.round(3.6*distance/time) : null;
                                     got.speed = util.format('%d km/h', speed);
                                 } else {
@@ -139,20 +139,12 @@ module.exports = function () {
                             putValue('destinations', destinations);
                         }
 
-                        var ok = true;
-
                         for (var key in row) {
                             if (this.FuzzyMatch.match(got[key], row[key])) {
                                 got[key] = row[key];
-                            } else {
-                                ok = false;
                             }
                         }
 
-                        if (!ok) {
-                            this.logFail(row, got, { route: { query: this.query, response: res }});
-                        }
-
                         cb(null, got);
                     } else {
                         cb(new Error('request failed to return valid body'));
@@ -189,11 +181,11 @@ module.exports = function () {
 
                     if (row.from && row.to) {
                         var fromNode = this.findNodeByName(row.from);
-                        if (!fromNode) throw new Error(util.format('*** unknown from-node "%s"'), row.from);
+                        if (!fromNode) return cb(new Error(util.format('*** unknown from-node "%s"'), row.from));
                         waypoints.push(fromNode);
 
                         var toNode = this.findNodeByName(row.to);
-                        if (!toNode) throw new Error(util.format('*** unknown to-node "%s"'), row.to);
+                        if (!toNode) return cb(new Error(util.format('*** unknown to-node "%s"'), row.to));
                         waypoints.push(toNode);
 
                         got.from = row.from;
@@ -202,13 +194,13 @@ module.exports = function () {
                     } else if (row.waypoints) {
                         row.waypoints.split(',').forEach((n) => {
                             var node = this.findNodeByName(n.trim());
-                            if (!node) throw new Error('*** unknown waypoint node "%s"', n.trim());
+                            if (!node) return cb(new Error('*** unknown waypoint node "%s"', n.trim()));
                             waypoints.push(node);
                         });
                         got.waypoints = row.waypoints;
                         this.requestRoute(waypoints, bearings, params, afterRequest);
                     } else {
-                        throw new Error('*** no waypoints');
+                        return cb(new Error('*** no waypoints'));
                     }
                 }
             };

features/support/table_patch.js

@@ -1,11 +0,0 @@
-var DifferentError = require('./exception_classes').TableDiffError;
-
-module.exports = function () {
-    this.diffTables = (expected, actual, options, callback) => {
-        // this is a temp workaround while waiting for https://github.com/cucumber/cucumber-js/issues/534
-
-        var error = new DifferentError(expected, actual);
-
-        return callback(error.string);
-    };
-};

features/testbot/bad.feature

@@ -11,8 +11,8 @@ Feature: Handle bad data in a graceful manner
         Given the ways
             | nodes |
 
-        When the data has been contracted
-        Then "osrm-extract" should return code 1
+        When I try to run "osrm-extract {osm_file} --profile {profile_file}"
+        Then it should exit with an error
 
     Scenario: Only dead-end oneways
         Given the node map

features/testbot/bugs.feature

@@ -1,5 +0,0 @@
-@routing @testbot @bug
-Feature: Known bugs
-
-    Background:
-        Given the profile "testbot"

features/testbot/matching.feature

@@ -124,7 +124,7 @@ Feature: Basic Map Matching
         1,2,36
         """
 
-        And the contract extra arguments "--segment-speed-file speeds.csv"
+        And the contract extra arguments "--segment-speed-file {speeds_file}"
 
         When I match I should get
             | trace | matchings | annotation                                                                                     |

features/testbot/traffic_turn_penalties.feature

@@ -26,7 +26,7 @@ Feature: Traffic - turn penalties applied to turn onto which a phantom node snap
             1,2,5,0,comment
             3,4,7,-20
             """
-        And the contract extra arguments "--turn-penalty-file penalties.csv"
+        And the contract extra arguments "--turn-penalty-file {penalties_file}"
         When I route I should get
             | from | to | route    | speed   | time    |
             | a    | e  | ab,be,be | 36 km/h | 40s +-1 |

features/testbot/via.feature

@@ -173,7 +173,7 @@ Feature: Via points
             | c,d,a     | abc,bd,bd,bd,abc,abc |
 
     # See issue #2349
-    @bug @todo
+    @todo
     Scenario: Via point at a dead end with oneway
         Given the node map
             | a | b | c |

include/engine/engine_config.hpp

@@ -51,6 +51,7 @@ namespace engine
  *  - Route
  *  - Table
  *  - Match
+ *  - Nearest
  *
  * In addition, shared memory can be used for datasets loaded with osrm-datastore.
  *
@@ -65,6 +66,7 @@ struct EngineConfig final
     int max_locations_viaroute = -1;
     int max_locations_distance_table = -1;
     int max_locations_map_matching = -1;
+    int max_results_nearest = -1;
     bool use_shared_memory = true;
 };
 }

include/engine/guidance/post_processing.hpp

@@ -1,7 +1,6 @@
 #ifndef ENGINE_GUIDANCE_POST_PROCESSING_HPP
 #define ENGINE_GUIDANCE_POST_PROCESSING_HPP
 
-#include "engine/datafacade/datafacade_base.hpp"
 #include "engine/guidance/leg_geometry.hpp"
 #include "engine/guidance/route_step.hpp"
 #include "engine/phantom_node.hpp"

include/engine/plugins/nearest.hpp

@@ -15,9 +15,12 @@ namespace plugins
 class NearestPlugin final : public BasePlugin
 {
   public:
-    explicit NearestPlugin(datafacade::BaseDataFacade &facade);
+    explicit NearestPlugin(datafacade::BaseDataFacade &facade, const int max_results);
 
     Status HandleRequest(const api::NearestParameters &params, util::json::Object &result);
+
+  private:
+    const int max_results;
 };
 }
 }

include/extractor/extraction_containers.hpp

@@ -39,10 +39,6 @@ class ExtractionContainers
     void WriteNodes(std::ofstream &file_out_stream) const;
     void WriteRestrictions(const std::string &restrictions_file_name) const;
     void WriteEdges(std::ofstream &file_out_stream) const;
-    void
-    WriteTurnLaneMasks(const std::string &file_name,
-                       const stxxl::vector<std::uint32_t> &turn_lane_offsets,
-                       const stxxl::vector<guidance::TurnLaneType::Mask> &turn_lane_masks) const;
     void WriteCharData(const std::string &file_name);
 
   public:

include/extractor/extractor_callbacks.hpp

@@ -18,14 +18,17 @@ class Way;
 
 namespace std
 {
-template <> struct hash<std::tuple<std::string, std::string, std::string>>
+template <> struct hash<std::tuple<std::string, std::string, std::string, std::string>>
 {
-    std::size_t operator()(const std::tuple<std::string, std::string, std::string> &mk) const noexcept
+    std::size_t
+    operator()(const std::tuple<std::string, std::string, std::string, std::string> &mk) const
+        noexcept
     {
         std::size_t seed = 0;
         boost::hash_combine(seed, std::get<0>(mk));
         boost::hash_combine(seed, std::get<1>(mk));
         boost::hash_combine(seed, std::get<2>(mk));
+        boost::hash_combine(seed, std::get<3>(mk));
         return seed;
     }
 };
@@ -51,8 +54,9 @@ struct ExtractionWay;
 class ExtractorCallbacks
 {
   private:
-    // used to deduplicate street names and street destinations: actually maps to name ids
-    using MapKey = std::tuple<std::string, std::string, std::string>;
+    // used to deduplicate street names, refs, destinations, pronunciation: actually maps to name
+    // ids
+    using MapKey = std::tuple<std::string, std::string, std::string, std::string>;
     using MapVal = unsigned;
     std::unordered_map<MapKey, MapVal> string_map;
     guidance::LaneDescriptionMap lane_description_map;

include/extractor/guidance/toolkit.hpp

@@ -11,7 +11,6 @@
 
 #include "extractor/compressed_edge_container.hpp"
 #include "extractor/query_node.hpp"
-#include "extractor/suffix_table.hpp"
 
 #include "extractor/guidance/discrete_angle.hpp"
 #include "extractor/guidance/intersection.hpp"
@@ -26,7 +25,6 @@
 #include <utility>
 
 #include <boost/algorithm/string.hpp>
-#include <boost/algorithm/string/predicate.hpp>
 #include <boost/functional/hash.hpp>
 #include <boost/tokenizer.hpp>
 
@@ -151,122 +149,6 @@ getRepresentativeCoordinate(const NodeID from_node,
     }
 }
 
-inline std::pair<std::string, std::string> getPrefixAndSuffix(const std::string &data)
-{
-    const auto suffix_pos = data.find_last_of(' ');
-    if (suffix_pos == std::string::npos)
-        return {};
-
-    const auto prefix_pos = data.find_first_of(' ');
-    auto result = std::make_pair(data.substr(0, prefix_pos), data.substr(suffix_pos + 1));
-    boost::to_lower(result.first);
-    boost::to_lower(result.second);
-    return result;
-}
-
-inline bool requiresNameAnnounced(const std::string &from,
-                                  const std::string &to,
-                                  const SuffixTable &suffix_table)
-{
-    // first is empty and the second is not
-    if (from.empty() && !to.empty())
-        return true;
-
-    // FIXME, handle in profile to begin with?
-    // this uses the encoding of references in the profile, which is very BAD
-    // Input for this function should be a struct separating streetname, suffix (e.g. road,
-    // boulevard, North, West ...), and a list of references
-    std::string from_name;
-    std::string from_ref;
-    std::string to_name;
-    std::string to_ref;
-
-    // Split from the format "{name} ({ref})" -> name, ref
-    auto split = [](const std::string &name, std::string &out_name, std::string &out_ref) {
-        const auto ref_begin = name.find_first_of('(');
-        if (ref_begin != std::string::npos)
-        {
-            if (ref_begin != 0)
-                out_name = name.substr(0, ref_begin - 1);
-            const auto ref_end = name.find_first_of(')');
-            out_ref = name.substr(ref_begin + 1, ref_end - ref_begin - 1);
-        }
-        else
-        {
-            out_name = name;
-        }
-    };
-
-    split(from, from_name, from_ref);
-    split(to, to_name, to_ref);
-
-    // check similarity of names
-    const auto names_are_empty = from_name.empty() && to_name.empty();
-    const auto name_is_contained =
-        boost::starts_with(from_name, to_name) || boost::starts_with(to_name, from_name);
-
-    const auto checkForPrefixOrSuffixChange = [](
-        const std::string &first, const std::string &second, const SuffixTable &suffix_table) {
-
-        const auto first_prefix_and_suffixes = getPrefixAndSuffix(first);
-        const auto second_prefix_and_suffixes = getPrefixAndSuffix(second);
-        // reverse strings, get suffices and reverse them to get prefixes
-        const auto checkTable = [&](const std::string &str) {
-            return str.empty() || suffix_table.isSuffix(str);
-        };
-
-        const auto getOffset = [](const std::string &str) -> std::size_t {
-            if (str.empty())
-                return 0;
-            else
-                return str.length() + 1;
-        };
-
-        const bool is_prefix_change = [&]() -> bool {
-            if (!checkTable(first_prefix_and_suffixes.first))
-                return false;
-            if (!checkTable(second_prefix_and_suffixes.first))
-                return false;
-            return !first.compare(getOffset(first_prefix_and_suffixes.first),
-                                  std::string::npos,
-                                  second,
-                                  getOffset(second_prefix_and_suffixes.first),
-                                  std::string::npos);
-        }();
-
-        const bool is_suffix_change = [&]() -> bool {
-            if (!checkTable(first_prefix_and_suffixes.second))
-                return false;
-            if (!checkTable(second_prefix_and_suffixes.second))
-                return false;
-            return !first.compare(0,
-                                  first.length() - getOffset(first_prefix_and_suffixes.second),
-                                  second,
-                                  0,
-                                  second.length() - getOffset(second_prefix_and_suffixes.second));
-        }();
-
-        return is_prefix_change || is_suffix_change;
-    };
-
-    const auto is_suffix_change = checkForPrefixOrSuffixChange(from_name, to_name, suffix_table);
-    const auto names_are_equal = from_name == to_name || name_is_contained || is_suffix_change;
-    const auto name_is_removed = !from_name.empty() && to_name.empty();
-    // references are contained in one another
-    const auto refs_are_empty = from_ref.empty() && to_ref.empty();
-    const auto ref_is_contained =
-        from_ref.empty() || to_ref.empty() ||
-        (from_ref.find(to_ref) != std::string::npos || to_ref.find(from_ref) != std::string::npos);
-    const auto ref_is_removed = !from_ref.empty() && to_ref.empty();
-
-    const auto obvious_change =
-        (names_are_empty && refs_are_empty) || (names_are_equal && ref_is_contained) ||
-        (names_are_equal && refs_are_empty) || (ref_is_contained && name_is_removed) ||
-        (names_are_equal && ref_is_removed) || is_suffix_change;
-
-    return !obvious_change;
-}
-
 // To simplify handling of Left/Right hand turns, we can mirror turns and write an intersection
 // handler only for one side. The mirror function turns a left-hand turn in a equivalent right-hand
 // turn and vice versa.

include/extractor/suffix_table.hpp

@@ -25,6 +25,7 @@ class SuffixTable final
   private:
     std::unordered_set<std::string> suffix_set;
 };
+
 } /* namespace extractor */
 } /* namespace osrm */
 

include/util/bearing.hpp

@@ -67,7 +67,7 @@ inline bool CheckInBounds(const int A, const int B, const int range)
 
     if (range >= 180)
         return true;
-    if (range <= 0)
+    if (range < 0)
         return false;
 
     // Map both bearings into positive modulo 360 space

include/util/guidance/toolkit.hpp

@@ -12,8 +12,13 @@
 #include "util/simple_logger.hpp"
 
 #include <algorithm>
+#include <string>
 #include <vector>
 
+#include <boost/algorithm/string.hpp>
+#include <boost/algorithm/string/predicate.hpp>
+#include <boost/functional/hash.hpp>
+
 namespace osrm
 {
 namespace util
@@ -113,7 +118,7 @@ inline bool entersRoundabout(const extractor::guidance::TurnInstruction instruct
             instruction.type == extractor::guidance::TurnType::EnterRoundaboutIntersectionAtExit ||
             instruction.type == extractor::guidance::TurnType::EnterAndExitRoundabout ||
             instruction.type == extractor::guidance::TurnType::EnterAndExitRotary ||
-            instruction.type == extractor::guidance::TurnType::EnterAndExitRotary);
+            instruction.type == extractor::guidance::TurnType::EnterAndExitRoundaboutIntersection);
 }
 
 inline bool leavesRoundabout(const extractor::guidance::TurnInstruction instruction)
@@ -131,6 +136,128 @@ inline bool staysOnRoundabout(const extractor::guidance::TurnInstruction instruc
     return instruction.type == extractor::guidance::TurnType::StayOnRoundabout;
 }
 
+// Name Change Logic
+// Used both during Extraction as well as during Post-Processing
+
+inline std::pair<std::string, std::string> getPrefixAndSuffix(const std::string &data)
+{
+    const auto suffix_pos = data.find_last_of(' ');
+    if (suffix_pos == std::string::npos)
+        return {};
+
+    const auto prefix_pos = data.find_first_of(' ');
+    auto result = std::make_pair(data.substr(0, prefix_pos), data.substr(suffix_pos + 1));
+    boost::to_lower(result.first);
+    boost::to_lower(result.second);
+    return result;
+}
+
+// Note: there is an overload without suffix checking below.
+// (that's the reason we template the suffix table here)
+template <typename SuffixTable>
+inline bool requiresNameAnnounced(const std::string &from_name,
+                                  const std::string &from_ref,
+                                  const std::string &to_name,
+                                  const std::string &to_ref,
+                                  const SuffixTable &suffix_table)
+{
+    // first is empty and the second is not
+    if ((from_name.empty() && from_ref.empty()) && !(to_name.empty() && to_ref.empty()))
+        return true;
+
+    // FIXME, handle in profile to begin with?
+    // Input for this function should be a struct separating streetname, suffix (e.g. road,
+    // boulevard, North, West ...), and a list of references
+
+    // check similarity of names
+    const auto names_are_empty = from_name.empty() && to_name.empty();
+    const auto name_is_contained =
+        boost::starts_with(from_name, to_name) || boost::starts_with(to_name, from_name);
+
+    const auto checkForPrefixOrSuffixChange = [](
+        const std::string &first, const std::string &second, const SuffixTable &suffix_table) {
+
+        const auto first_prefix_and_suffixes = getPrefixAndSuffix(first);
+        const auto second_prefix_and_suffixes = getPrefixAndSuffix(second);
+
+        // reverse strings, get suffices and reverse them to get prefixes
+        const auto checkTable = [&](const std::string &str) {
+            return str.empty() || suffix_table.isSuffix(str);
+        };
+
+        const auto getOffset = [](const std::string &str) -> std::size_t {
+            if (str.empty())
+                return 0;
+            else
+                return str.length() + 1;
+        };
+
+        const bool is_prefix_change = [&]() -> bool {
+            if (!checkTable(first_prefix_and_suffixes.first))
+                return false;
+            if (!checkTable(second_prefix_and_suffixes.first))
+                return false;
+            return !first.compare(getOffset(first_prefix_and_suffixes.first),
+                                  std::string::npos,
+                                  second,
+                                  getOffset(second_prefix_and_suffixes.first),
+                                  std::string::npos);
+        }();
+
+        const bool is_suffix_change = [&]() -> bool {
+            if (!checkTable(first_prefix_and_suffixes.second))
+                return false;
+            if (!checkTable(second_prefix_and_suffixes.second))
+                return false;
+            return !first.compare(0,
+                                  first.length() - getOffset(first_prefix_and_suffixes.second),
+                                  second,
+                                  0,
+                                  second.length() - getOffset(second_prefix_and_suffixes.second));
+        }();
+
+        return is_prefix_change || is_suffix_change;
+    };
+
+    const auto is_suffix_change = checkForPrefixOrSuffixChange(from_name, to_name, suffix_table);
+    const auto names_are_equal = from_name == to_name || name_is_contained || is_suffix_change;
+    const auto name_is_removed = !from_name.empty() && to_name.empty();
+    // references are contained in one another
+    const auto refs_are_empty = from_ref.empty() && to_ref.empty();
+    const auto ref_is_contained =
+        from_ref.empty() || to_ref.empty() ||
+        (from_ref.find(to_ref) != std::string::npos || to_ref.find(from_ref) != std::string::npos);
+    const auto ref_is_removed = !from_ref.empty() && to_ref.empty();
+
+    const auto obvious_change =
+        (names_are_empty && refs_are_empty) || (names_are_equal && ref_is_contained) ||
+        (names_are_equal && refs_are_empty) || (ref_is_contained && name_is_removed) ||
+        (names_are_equal && ref_is_removed) || is_suffix_change;
+
+    const auto needs_announce =
+        // " (Ref)" -> "Name "
+        (from_name.empty() && !from_ref.empty() && !to_name.empty() && to_ref.empty());
+
+    return !obvious_change || needs_announce;
+}
+
+// Overload without suffix checking
+inline bool requiresNameAnnounced(const std::string &from_name,
+                                  const std::string &from_ref,
+                                  const std::string &to_name,
+                                  const std::string &to_ref)
+{
+    // Dummy since we need to provide a SuffixTable but do not have the data for it.
+    // (Guidance Post-Processing does not keep the suffix table around at the moment)
+    struct NopSuffixTable final
+    {
+        NopSuffixTable(){}
+        bool isSuffix(const std::string &) const { return false; }
+    } static const table;
+
+    return requiresNameAnnounced(from_name, from_ref, to_name, to_ref, table);
+}
+
 } // namespace guidance
 } // namespace util
 } // namespace osrm

include/util/name_table.hpp

@@ -23,7 +23,13 @@ class NameTable
 
   public:
     NameTable(const std::string &filename);
+
+    // This class provides a limited view over all the string data we serialize out.
+    // The following functions are a subset of what is available.
+    // See the data facades for they provide full access to this serialized string data.
+    // (at time of writing this: get{Name,Ref,Pronunciation,Destinations}ForID(name_id);)
     std::string GetNameForID(const unsigned name_id) const;
+    std::string GetRefForID(const unsigned name_id) const;
 };
 } // namespace util
 } // namespace osrm

include/util/static_rtree.hpp

@@ -17,6 +17,7 @@
 #include <boost/assert.hpp>
 #include <boost/filesystem.hpp>
 #include <boost/filesystem/fstream.hpp>
+#include <boost/format.hpp>
 #include <boost/iostreams/device/mapped_file.hpp>
 
 #include <tbb/parallel_for.h>

package.json

@@ -7,9 +7,11 @@
     "chalk": "^1.1.3",
     "cucumber": "^1.2.1",
     "d3-queue": "^2.0.3",
+    "mkdirp": "^0.5.1",
     "node-timeout": "0.0.4",
     "polyline": "^0.2.0",
     "request": "^2.69.0",
+    "rimraf": "^2.5.4",
     "xmlbuilder": "^4.2.1"
   },
   "bin": {

profiles/car.lua

@@ -238,6 +238,10 @@ function way_function (way, result)
     return
   end
 
+  -- default to driving mode, may get overwritten below
+  result.forward_mode = mode.driving
+  result.backward_mode = mode.driving
+
   -- we dont route over areas
   local area = way:get_value_by_key("area")
   if ignore_areas and area and "yes" == area then
@@ -245,10 +249,58 @@ function way_function (way, result)
   end
 
   -- respect user-preference for HOV-only ways
-  local hov = way:get_value_by_key("hov")
-  if ignore_hov_ways and hov and "designated" == hov then
-    return
-  end
+  if ignore_hov_ways then
+    local hov = way:get_value_by_key("hov")
+    if hov and "designated" == hov then
+      return
+    end
+
+    -- also respect user-preference for HOV-only ways when all lanes are HOV-designated
+    local function has_all_designated_hov_lanes(lanes)
+      local all = true
+      for lane in lanes:gmatch("(%w+)") do
+        if lane and lane ~= "designated" then
+          all = false
+          break
+        end
+      end
+      return all
+    end
+
+    local hov_lanes = way:get_value_by_key("hov:lanes")
+    local hov_lanes_forward = way:get_value_by_key("hov:lanes:forward")
+    local hov_lanes_backward = way:get_value_by_key("hov:lanes:backward")
+
+    local hov_all_designated = hov_lanes and hov_lanes ~= ""
+                               and has_all_designated_hov_lanes(hov_lanes)
+
+    local hov_all_designated_forward = hov_lanes_forward and hov_lanes_forward ~= ""
+                                       and has_all_designated_hov_lanes(hov_lanes_forward)
+
+    local hov_all_designated_backward = hov_lanes_backward and hov_lanes_backward ~= ""
+                                        and has_all_designated_hov_lanes(hov_lanes_backward)
+
+    -- forward/backward lane depend on a way's direction
+    local oneway = way:get_value_by_key("oneway")
+    local reverse = oneway and oneway == "-1"
+
+    if hov_all_designated or hov_all_designated_forward then
+      if reverse then
+        result.backward_mode = mode.inaccessible
+      else
+        result.forward_mode = mode.inaccessible
+      end
+    end
+
+    if hov_all_designated_backward then
+      if reverse then
+        result.forward_mode = mode.inaccessible
+      else
+        result.backward_mode = mode.inaccessible
+      end
+    end
+
+  end -- hov handling
 
   -- respect user-preference for toll=yes ways
   local toll = way:get_value_by_key("toll")
@@ -278,9 +330,6 @@ function way_function (way, result)
     return
   end
 
-  result.forward_mode = mode.driving
-  result.backward_mode = mode.driving
-
   -- handling ferries and piers
   local route_speed = speed_profile[route]
   if (route_speed and route_speed > 0) then

profiles/rasterbot.lua

@@ -21,8 +21,12 @@ function way_function (way, result)
 end
 
 function source_function ()
+  local path = os.getenv('OSRM_RASTER_SOURCE')
+  if not path then
+    path = "rastersource.asc"
+  end
   raster_source = sources:load(
-    "../test/rastersource.asc",
+    path,
     0,    -- lon_min
     0.1,  -- lon_max
     0,    -- lat_min

profiles/rasterbotinterp.lua

@@ -21,8 +21,12 @@ function way_function (way, result)
 end
 
 function source_function ()
+  local path = os.getenv('OSRM_RASTER_SOURCE')
+  if not path then
+    path = "rastersource.asc"
+  end
   raster_source = sources:load(
-    "../test/rastersource.asc",
+    path,
     0,    -- lon_min
     0.1,  -- lon_max
     0,    -- lat_min

src/engine/api/json_factory.cpp

@@ -235,7 +235,8 @@ util::json::Object makeRouteStep(guidance::RouteStep step, util::json::Value geo
     route_step.values["distance"] = std::round(step.distance * 10) / 10.;
     route_step.values["duration"] = std::round(step.duration * 10) / 10.;
     route_step.values["name"] = std::move(step.name);
-    if (!step.ref.empty()) route_step.values["ref"] = std::move(step.ref);
+    if (!step.ref.empty())
+        route_step.values["ref"] = std::move(step.ref);
     if (!step.pronunciation.empty())
         route_step.values["pronunciation"] = std::move(step.pronunciation);
     if (!step.destinations.empty())

src/engine/engine.cpp

@@ -146,7 +146,7 @@ Engine::Engine(EngineConfig &config)
 
     route_plugin = create<ViaRoutePlugin>(*query_data_facade, config.max_locations_viaroute);
     table_plugin = create<TablePlugin>(*query_data_facade, config.max_locations_distance_table);
-    nearest_plugin = create<NearestPlugin>(*query_data_facade);
+    nearest_plugin = create<NearestPlugin>(*query_data_facade, config.max_results_nearest);
     trip_plugin = create<TripPlugin>(*query_data_facade, config.max_locations_trip);
     match_plugin = create<MatchPlugin>(*query_data_facade, config.max_locations_map_matching);
     tile_plugin = create<TilePlugin>(*query_data_facade);

src/engine/engine_config.cpp

@@ -15,11 +15,15 @@ bool EngineConfig::IsValid() const
         storage_config.datasource_names_path.empty() &&
         storage_config.datasource_indexes_path.empty() && storage_config.names_data_path.empty();
 
-    const bool limits_valid =
-        (max_locations_distance_table == -1 || max_locations_distance_table > 2) &&
-        (max_locations_map_matching == -1 || max_locations_map_matching > 2) &&
-        (max_locations_trip == -1 || max_locations_trip > 2) &&
-        (max_locations_viaroute == -1 || max_locations_viaroute > 2);
+    const auto unlimited_or_more_than = [](const int v, const int limit) {
+        return v == -1 || v > limit;
+    };
+
+    const bool limits_valid = unlimited_or_more_than(max_locations_distance_table, 2) &&
+                              unlimited_or_more_than(max_locations_map_matching, 2) &&
+                              unlimited_or_more_than(max_locations_trip, 2) &&
+                              unlimited_or_more_than(max_locations_viaroute, 2) &&
+                              unlimited_or_more_than(max_results_nearest, 0);
 
     return ((use_shared_memory && all_path_are_empty) || storage_config.IsValid()) && limits_valid;
 }

src/engine/guidance/post_processing.cpp

@@ -83,18 +83,6 @@ bool isCollapsableInstruction(const TurnInstruction instruction)
 
 bool compatible(const RouteStep &lhs, const RouteStep &rhs) { return lhs.mode == rhs.mode; }
 
-double nameSegmentLength(std::size_t at, const std::vector<RouteStep> &steps)
-{
-    double result = steps[at].distance;
-    while (at + 1 < steps.size() && steps[at + 1].name_id == steps[at].name_id)
-    {
-        ++at;
-        result += steps[at].distance;
-    }
-
-    return result;
-}
-
 // invalidate a step and set its content to nothing
 void invalidateStep(RouteStep &step) { step = getInvalidRouteStep(); }
 
@@ -129,6 +117,28 @@ double turn_angle(const double entry_bearing, const double exit_bearing)
     return angle > 360 ? angle - 360 : angle;
 }
 
+// Checks if name change happens the user wants to know about.
+// Treats e.g. "Name (Ref)" -> "Name" changes still as same name.
+bool isNoticeableNameChange(const RouteStep &lhs, const RouteStep &rhs)
+{
+    // TODO: at some point we might want to think about pronunciation here.
+    // Also rotary_name is not handled at the moment.
+    return util::guidance::requiresNameAnnounced(lhs.name, lhs.ref, rhs.name, rhs.ref);
+}
+
+double nameSegmentLength(std::size_t at, const std::vector<RouteStep> &steps)
+{
+    BOOST_ASSERT(at < steps.size());
+
+    double result = steps[at].distance;
+    while (at + 1 < steps.size() && !isNoticeableNameChange(steps[at], steps[at + 1]))
+    {
+        at += 1;
+        result += steps[at].distance;
+    }
+    return result;
+}
+
 OSRM_ATTR_WARN_UNUSED
 RouteStep forwardInto(RouteStep destination, const RouteStep &source)
 {
@@ -203,6 +213,8 @@ bool setUpRoundabout(RouteStep &step)
         instruction.type == TurnType::EnterRoundaboutAtExit ||
         instruction.type == TurnType::EnterRoundaboutIntersectionAtExit)
     {
+        // Here we consider an actual entry, not an exit. We simply have to count the additional
+        // exit
         step.maneuver.exit = 1;
         // prevent futher special case handling of these two.
         if (instruction.type == TurnType::EnterRotaryAtExit)
@@ -215,6 +227,7 @@ bool setUpRoundabout(RouteStep &step)
 
     if (leavesRoundabout(instruction))
     {
+        // This set-up, even though it looks the same, is actually looking at entering AND exiting
         step.maneuver.exit = 1; // count the otherwise missing exit
 
         // prevent futher special case handling of these two.
@@ -406,12 +419,16 @@ void collapseTurnAt(std::vector<RouteStep> &steps,
         if (compatible(one_back_step, current_step))
         {
             steps[one_back_index] = elongate(std::move(steps[one_back_index]), steps[step_index]);
+
             if ((TurnType::Continue == one_back_step.maneuver.instruction.type ||
                  TurnType::Suppressed == one_back_step.maneuver.instruction.type) &&
-                current_step.name_id != steps[two_back_index].name_id)
+                isNoticeableNameChange(steps[two_back_index], current_step))
+            {
+
                 steps[one_back_index].maneuver.instruction.type = TurnType::Turn;
+            }
             else if (TurnType::Turn == one_back_step.maneuver.instruction.type &&
-                     current_step.name_id == steps[two_back_index].name_id)
+                     !isNoticeableNameChange(steps[two_back_index], current_step))
             {
                 steps[one_back_index].maneuver.instruction.type = TurnType::Continue;
 
@@ -457,7 +474,7 @@ void collapseTurnAt(std::vector<RouteStep> &steps,
     {
         BOOST_ASSERT(two_back_index < steps.size());
         // the simple case is a u-turn that changes directly into the in-name again
-        const bool direct_u_turn = steps[two_back_index].name_id == current_step.name_id;
+        const bool direct_u_turn = !isNoticeableNameChange(steps[two_back_index], current_step);
 
         // however, we might also deal with a dual-collapse scenario in which we have to
         // additionall collapse a name-change as welll
@@ -468,7 +485,7 @@ void collapseTurnAt(std::vector<RouteStep> &steps,
              isCollapsableInstruction(steps[next_step_index].maneuver.instruction));
         const bool u_turn_with_name_change =
             continues_with_name_change &&
-            steps[next_step_index].name_id == steps[two_back_index].name_id;
+            !isNoticeableNameChange(steps[two_back_index], steps[next_step_index]);
 
         if (direct_u_turn || u_turn_with_name_change)
         {
@@ -792,9 +809,8 @@ std::vector<RouteStep> collapseTurns(std::vector<RouteStep> steps)
                     // Turn Types in the response depend on whether we find the same road name
                     // (sliproad indcating a u-turn) or if we are turning onto a different road, in
                     // which case we use a turn.
-                    if (steps[getPreviousIndex(one_back_index)].name_id ==
-                            steps[step_index].name_id &&
-                        steps[step_index].name_id != EMPTY_NAMEID)
+                    if (!isNoticeableNameChange(steps[getPreviousIndex(one_back_index)],
+                                                steps[step_index]))
                         steps[one_back_index].maneuver.instruction.type = TurnType::Continue;
                     else
                         steps[one_back_index].maneuver.instruction.type = TurnType::Turn;
@@ -828,7 +844,7 @@ std::vector<RouteStep> collapseTurns(std::vector<RouteStep> steps)
         // These have to be handled in post-processing
         else if (isCollapsableInstruction(current_step.maneuver.instruction) &&
                  current_step.maneuver.instruction.type != TurnType::Suppressed &&
-                 steps[getPreviousNameIndex(step_index)].name_id == current_step.name_id &&
+                 !isNoticeableNameChange(steps[getPreviousNameIndex(step_index)], current_step) &&
                  // canCollapseAll is also checking for compatible(step,step+1) for all indices
                  canCollapseAll(getPreviousNameIndex(step_index) + 1, next_step_index))
         {
@@ -853,8 +869,7 @@ std::vector<RouteStep> collapseTurns(std::vector<RouteStep> steps)
             const auto two_back_index = getPreviousIndex(one_back_index);
             BOOST_ASSERT(two_back_index < steps.size());
             // valid, since one_back is collapsable or a turn and therefore not depart:
-            const auto &coming_from_name_id = steps[two_back_index].name_id;
-            if (current_step.name_id == coming_from_name_id)
+            if (!isNoticeableNameChange(steps[two_back_index], current_step))
             {
                 if (compatible(one_back_step, steps[two_back_index]))
                 {
@@ -886,7 +901,7 @@ std::vector<RouteStep> collapseTurns(std::vector<RouteStep> steps)
             else if (step_index + 2 < steps.size() &&
                      current_step.maneuver.instruction.type == TurnType::NewName &&
                      steps[next_step_index].maneuver.instruction.type == TurnType::NewName &&
-                     one_back_step.name_id == steps[next_step_index].name_id)
+                     !isNoticeableNameChange(one_back_step, steps[next_step_index]))
             {
                 if (compatible(steps[step_index], steps[next_step_index]))
                 {

src/engine/plugins/nearest.cpp

@@ -8,6 +8,7 @@
 #include <string>
 
 #include <boost/assert.hpp>
+#include <boost/numeric/conversion/cast.hpp>
 
 namespace osrm
 {
@@ -16,13 +17,25 @@ namespace engine
 namespace plugins
 {
 
-NearestPlugin::NearestPlugin(datafacade::BaseDataFacade &facade) : BasePlugin{facade} {}
+NearestPlugin::NearestPlugin(datafacade::BaseDataFacade &facade, const int max_results_)
+    : BasePlugin{facade}, max_results{max_results_}
+{
+}
 
 Status NearestPlugin::HandleRequest(const api::NearestParameters &params,
                                     util::json::Object &json_result)
 {
     BOOST_ASSERT(params.IsValid());
 
+    if (max_results > 0 &&
+        (boost::numeric_cast<std::int64_t>(params.number_of_results) > max_results))
+    {
+        return Error("TooBig",
+                     "Number of results " + std::to_string(params.number_of_results) +
+                         " is higher than current maximum (" + std::to_string(max_results) + ")",
+                     json_result);
+    }
+
     if (!CheckAllCoordinates(params.coordinates))
         return Error("InvalidOptions", "Coordinates are invalid", json_result);
 

src/extractor/extraction_containers.cpp

@@ -110,7 +110,8 @@ ExtractionContainers::ExtractionContainers()
     // Check if stxxl can be instantiated
     stxxl::vector<unsigned> dummy_vector;
 
-    // Insert three empty strings offsets for name, destination and pronunciation
+    // Insert three empty strings offsets for name, ref, destination and pronunciation
+    name_offsets.push_back(0);
     name_offsets.push_back(0);
     name_offsets.push_back(0);
     name_offsets.push_back(0);

src/extractor/extractor_callbacks.cpp

@@ -34,8 +34,8 @@ namespace TurnLaneType = guidance::TurnLaneType;
 ExtractorCallbacks::ExtractorCallbacks(ExtractionContainers &extraction_containers)
     : external_memory(extraction_containers)
 {
-    // we reserved 0, 1, 2 for the empty case
-    string_map[MapKey("", "", "")] = 0;
+    // we reserved 0, 1, 2, 3 for the empty case
+    string_map[MapKey("", "", "", "")] = 0;
     lane_description_map[TurnLaneDescription()] = 0;
 }
 
@@ -142,7 +142,7 @@ void ExtractorCallbacks::ProcessWay(const osmium::Way &input_way, const Extracti
 
     // FIXME this need to be moved into the profiles
     const guidance::RoadClassification road_classification = parsed_way.road_classification;
-    const auto laneStringToDescription = [](std::string lane_string) -> TurnLaneDescription {
+    const auto laneStringToDescription = [](const std::string &lane_string) -> TurnLaneDescription {
         if (lane_string.empty())
             return {};
 
@@ -214,7 +214,7 @@ void ExtractorCallbacks::ProcessWay(const osmium::Way &input_way, const Extracti
 
     // convert the lane description into an ID and, if necessary, remembr the description in the
     // description_map
-    const auto requestId = [&](std::string lane_string) {
+    const auto requestId = [&](const std::string &lane_string) {
         if (lane_string.empty())
             return INVALID_LANE_DESCRIPTIONID;
         TurnLaneDescription lane_description = laneStringToDescription(std::move(lane_string));
@@ -233,15 +233,16 @@ void ExtractorCallbacks::ProcessWay(const osmium::Way &input_way, const Extracti
         }
     };
 
-    // Deduplicates street names, destination names and refs based on the string_map map.
-    // In case we do not already store the name, inserts (name, id) tuple and return id.
+    // Deduplicates street names, refs, destinations, pronunciation based on the string_map.
+    // In case we do not already store the key, inserts (key, id) tuple and return id.
     // Otherwise fetches the id based on the name and returns it without insertion.
     const auto turn_lane_id_forward = requestId(parsed_way.turn_lanes_forward);
     const auto turn_lane_id_backward = requestId(parsed_way.turn_lanes_backward);
 
     const constexpr auto MAX_STRING_LENGTH = 255u;
     // Get the unique identifier for the street name, destination, and ref
-    const auto name_iterator = string_map.find(MapKey(parsed_way.name, parsed_way.destinations, parsed_way.ref));
+    const auto name_iterator = string_map.find(
+        MapKey(parsed_way.name, parsed_way.destinations, parsed_way.ref, parsed_way.pronunciation));
     unsigned name_id = EMPTY_NAMEID;
     if (string_map.end() == name_iterator)
     {
@@ -256,7 +257,8 @@ void ExtractorCallbacks::ProcessWay(const osmium::Way &input_way, const Extracti
         name_id = external_memory.name_offsets.size() - 1;
 
         external_memory.name_char_data.reserve(external_memory.name_char_data.size() + name_length +
-                                               destinations_length + pronunciation_length);
+                                               destinations_length + pronunciation_length +
+                                               ref_length);
 
         std::copy(parsed_way.name.c_str(),
                   parsed_way.name.c_str() + name_length,
@@ -278,7 +280,8 @@ void ExtractorCallbacks::ProcessWay(const osmium::Way &input_way, const Extracti
                   std::back_inserter(external_memory.name_char_data));
         external_memory.name_offsets.push_back(external_memory.name_char_data.size());
 
-        auto k = MapKey{parsed_way.name, parsed_way.destinations, parsed_way.ref};
+        auto k = MapKey{
+            parsed_way.name, parsed_way.destinations, parsed_way.ref, parsed_way.pronunciation};
         auto v = MapVal{name_id};
         string_map.emplace(std::move(k), std::move(v));
     }

src/extractor/guidance/intersection_generator.cpp

@@ -1,11 +1,12 @@
-#include "extractor/guidance/intersection_generator.hpp"
 #include "extractor/guidance/constants.hpp"
+#include "extractor/guidance/intersection_generator.hpp"
 #include "extractor/guidance/toolkit.hpp"
 
 #include <algorithm>
 #include <iomanip>
 #include <iterator>
 #include <limits>
+#include <unordered_set>
 #include <utility>
 
 #include <boost/range/algorithm/count_if.hpp>
@@ -268,8 +269,11 @@ bool IntersectionGenerator::CanMerge(const NodeID node_at_intersection,
         return becomes_narrower;
     };
 
+    const bool is_y_arm_first = isValidYArm(first_index, second_index);
+    const bool is_y_arm_second = isValidYArm(second_index, first_index);
+
     // Only merge valid y-arms
-    if (!isValidYArm(first_index, second_index) || !isValidYArm(second_index, first_index))
+    if (!is_y_arm_first || !is_y_arm_second)
         return false;
 
     if (angle_between < 60)
@@ -577,10 +581,14 @@ IntersectionGenerator::GetActualNextIntersection(const NodeID starting_node,
     // to prevent endless loops
     const auto termination_node = node_based_graph.GetTarget(via_edge);
 
-    while (result.size() == 2 &&
-           node_based_graph.GetEdgeData(via_edge).IsCompatibleTo(
-               node_based_graph.GetEdgeData(result[1].turn.eid)))
+    // using a maximum lookahead, we make sure not to end up in some form of loop
+    std::unordered_set<NodeID> visited_nodes;
+    while (visited_nodes.count(node_at_intersection) == 0 &&
+           (result.size() == 2 &&
+            node_based_graph.GetEdgeData(via_edge).IsCompatibleTo(
+                node_based_graph.GetEdgeData(result[1].turn.eid))))
     {
+        visited_nodes.insert(node_at_intersection);
         node_at_intersection = node_based_graph.GetTarget(incoming_edge);
         incoming_edge = result[1].turn.eid;
         result = GetConnectedRoads(node_at_intersection, incoming_edge);

src/extractor/guidance/intersection_handler.cpp

@@ -4,6 +4,7 @@
 
 #include "util/coordinate_calculation.hpp"
 #include "util/guidance/toolkit.hpp"
+#include "util/guidance/toolkit.hpp"
 #include "util/simple_logger.hpp"
 
 #include <algorithm>
@@ -90,9 +91,11 @@ TurnInstruction IntersectionHandler::getInstructionForObvious(const std::size_t
         const auto &in_data = node_based_graph.GetEdgeData(via_edge);
         const auto &out_data = node_based_graph.GetEdgeData(road.turn.eid);
         if (in_data.name_id != out_data.name_id &&
-            requiresNameAnnounced(name_table.GetNameForID(in_data.name_id),
-                                  name_table.GetNameForID(out_data.name_id),
-                                  street_name_suffix_table))
+            util::guidance::requiresNameAnnounced(name_table.GetNameForID(in_data.name_id),
+                                                  name_table.GetRefForID(in_data.name_id),
+                                                  name_table.GetNameForID(out_data.name_id),
+                                                  name_table.GetRefForID(out_data.name_id),
+                                                  street_name_suffix_table))
         {
             // obvious turn onto a through street is a merge
             if (through_street)
@@ -597,17 +600,18 @@ std::size_t IntersectionHandler::findObviousTurn(const EdgeID via_edge,
             if (deviation_ratio < DISTINCTION_RATIO / 1.5)
                 return 0;
 
-            // in comparison to another continuing road, we need a better distinction. This prevents
-            // situations where the turn is probably less obvious. An example are places that have a
-            // road with the same name entering/exiting:
-            //
-            //         d
-            //        /
-            //       /
-            // a -- b
-            //       \
-            //        \
-            //         c
+            /* in comparison to another continuing road, we need a better distinction. This prevents
+               situations where the turn is probably less obvious. An example are places that have a
+               road with the same name entering/exiting:
+
+                       d
+                      /
+                     /
+               a -- b
+                     \
+                      \
+                       c
+            */
 
             if (turn_data.name_id == continue_data.name_id &&
                 deviation_ratio < 1.5 * DISTINCTION_RATIO)

src/extractor/guidance/roundabout_handler.cpp

@@ -222,48 +222,46 @@ RoundaboutType RoundaboutHandler::getRoundaboutType(const NodeID nid) const
     std::unordered_set<unsigned> roundabout_name_ids;
     std::unordered_set<unsigned> connected_names;
 
-    const auto getNextOnRoundabout =
-        [this, &roundabout_name_ids, &connected_names](const NodeID node) {
-            EdgeID continue_edge = SPECIAL_EDGEID;
-            for (const auto edge : node_based_graph.GetAdjacentEdgeRange(node))
+    const auto getNextOnRoundabout = [this, &roundabout_name_ids, &connected_names](
+        const NodeID node) {
+        EdgeID continue_edge = SPECIAL_EDGEID;
+        for (const auto edge : node_based_graph.GetAdjacentEdgeRange(node))
+        {
+            const auto &edge_data = node_based_graph.GetEdgeData(edge);
+            if (!edge_data.reversed && edge_data.roundabout)
             {
-                const auto &edge_data = node_based_graph.GetEdgeData(edge);
-                if (!edge_data.reversed && edge_data.roundabout)
+                if (SPECIAL_EDGEID != continue_edge)
                 {
-                    if (SPECIAL_EDGEID != continue_edge)
-                    {
-                        // fork in roundabout
-                        return SPECIAL_EDGEID;
-                    }
-
-                    if (EMPTY_NAMEID != edge_data.name_id)
-                    {
-                        bool add = true;
-                        for (auto name_id : roundabout_name_ids)
-                        {
-
-                            if (!requiresNameAnnounced(name_table.GetNameForID(name_id),
-                                                       name_table.GetNameForID(edge_data.name_id),
-                                                       street_name_suffix_table))
-                            {
-                                add = false;
-                                break;
-                            }
-                        }
-                        if (add)
-                            roundabout_name_ids.insert(edge_data.name_id);
-                    }
-
-                    continue_edge = edge;
+                    // fork in roundabout
+                    return SPECIAL_EDGEID;
                 }
-                else if (!edge_data.roundabout)
+
+                if (EMPTY_NAMEID != edge_data.name_id)
                 {
-                    // remember all connected road names
-                    connected_names.insert(edge_data.name_id);
+
+                    const auto announce = [&](unsigned id) {
+                        return util::guidance::requiresNameAnnounced(
+                            name_table.GetNameForID(id),
+                            name_table.GetRefForID(id),
+                            name_table.GetNameForID(edge_data.name_id),
+                            name_table.GetRefForID(edge_data.name_id),
+                            street_name_suffix_table);
+                    };
+
+                    if (std::all_of(begin(roundabout_name_ids), end(roundabout_name_ids), announce))
+                        roundabout_name_ids.insert(edge_data.name_id);
                 }
+
+                continue_edge = edge;
             }
-            return continue_edge;
-        };
+            else if (!edge_data.roundabout)
+            {
+                // remember all connected road names
+                connected_names.insert(edge_data.name_id);
+            }
+        }
+        return continue_edge;
+    };
     // the roundabout radius has to be the same for all locations we look at it from
     // to guarantee this, we search the full roundabout for its vertices
     // and select the three smallest ids

src/extractor/guidance/sliproad_handler.cpp

@@ -1,6 +1,6 @@
-#include "extractor/guidance/sliproad_handler.hpp"
 #include "extractor/guidance/constants.hpp"
 #include "extractor/guidance/intersection_scenario_three_way.hpp"
+#include "extractor/guidance/sliproad_handler.hpp"
 #include "extractor/guidance/toolkit.hpp"
 
 #include "util/guidance/toolkit.hpp"
@@ -37,9 +37,9 @@ SliproadHandler::SliproadHandler(const IntersectionGenerator &intersection_gener
 // included for interface reasons only
 bool SliproadHandler::canProcess(const NodeID /*nid*/,
                                  const EdgeID /*via_eid*/,
-                                 const Intersection & /*intersection*/) const
+                                 const Intersection &intersection) const
 {
-    return true;
+    return intersection.size() > 2;
 }
 
 Intersection SliproadHandler::
@@ -56,13 +56,30 @@ operator()(const NodeID, const EdgeID source_edge_id, Intersection intersection)
             auto intersection = intersection_generator(at_node, road.turn.eid);
             auto in_edge = road.turn.eid;
             // skip over traffic lights
-            while (intersection.size() == 2)
+            // to prevent ending up in an endless loop, we remember all visited nodes. This is
+            // necessary, since merging of roads can actually create enterable loops of degree two
+            std::unordered_set<NodeID> visited_nodes;
+            auto node = at_node;
+            while (intersection.size() == 2 && visited_nodes.count(node) == 0)
             {
-                const auto node = node_based_graph.GetTarget(in_edge);
+                visited_nodes.insert(node);
+                node = node_based_graph.GetTarget(in_edge);
+                if (node == at_node)
+                {
+                    // we ended up in a loop without exit
+                    output_node = SPECIAL_NODEID;
+                    intersection.clear();
+                    return intersection;
+                }
                 in_edge = intersection[1].turn.eid;
                 output_node = node_based_graph.GetTarget(in_edge);
                 intersection = intersection_generator(node, in_edge);
             }
+            if (intersection.size() <= 2)
+            {
+                output_node = SPECIAL_NODEID;
+                intersection.clear();
+            }
             return intersection;
         };
 
@@ -77,11 +94,15 @@ operator()(const NodeID, const EdgeID source_edge_id, Intersection intersection)
             // a one-sided sliproad, however, the non-sliproad can be considered `obvious`. Here we
             // assume that this could be the case and check for a potential sliproad/non-sliproad
             // situation.
-            NodeID intersection_node_one, intersection_node_two;
+            NodeID intersection_node_one = SPECIAL_NODEID, intersection_node_two = SPECIAL_NODEID;
             const auto intersection_following_index_one = findNextIntersectionForRoad(
                 intersection_node_id, intersection[1], intersection_node_one);
             const auto intersection_following_index_two = findNextIntersectionForRoad(
                 intersection_node_id, intersection[2], intersection_node_two);
+            // in case of broken roads, we return
+            if (intersection_following_index_one.empty() ||
+                intersection_following_index_two.empty())
+                return 0;
 
             // In case of loops at the end of the road, we will arrive back at the intersection
             // itself. If that is the case, the road is obviously not a sliproad.
@@ -128,7 +149,6 @@ operator()(const NodeID, const EdgeID source_edge_id, Intersection intersection)
         return intersection;
 
     const auto source_edge_data = node_based_graph.GetEdgeData(source_edge_id);
-
     // check whether the continue road is valid
     const auto check_valid = [this, source_edge_data](const ConnectedRoad &road) {
         const auto road_edge_data = node_based_graph.GetEdgeData(road.turn.eid);
@@ -152,6 +172,9 @@ operator()(const NodeID, const EdgeID source_edge_id, Intersection intersection)
     const auto next_road_next_intersection =
         findNextIntersectionForRoad(intersection_node_id, next_road, next_intersection_node);
 
+    if (next_road_next_intersection.empty())
+        return intersection;
+
     // If we are at a traffic loop at the end of a road, don't consider it a sliproad
     if (intersection_node_id == next_intersection_node)
         return intersection;
@@ -181,6 +204,22 @@ operator()(const NodeID, const EdgeID source_edge_id, Intersection intersection)
                 return intersection;
             }(intersection_node_id);
 
+            const auto link_data = node_based_graph.GetEdgeData(road.turn.eid);
+            // Check if the road continues here
+            const bool is_through_street =
+                !target_intersection.empty() &&
+                target_intersection.end() !=
+                    std::find_if(target_intersection.begin() + 1,
+                                 target_intersection.end(),
+                                 [this, &link_data](const ConnectedRoad &road) {
+                                     return node_based_graph.GetEdgeData(road.turn.eid).name_id ==
+                                            link_data.name_id;
+                                 });
+
+            // if the sliproad candidate is a through street, we cannot handle it as a sliproad
+            if (is_through_street)
+                continue;
+
             for (const auto &candidate_road : target_intersection)
             {
                 const auto &candidate_data = node_based_graph.GetEdgeData(candidate_road.turn.eid);

src/extractor/guidance/turn_analysis.cpp

@@ -94,7 +94,8 @@ Intersection TurnAnalysis::assignTurnTypes(const NodeID from_nid,
         }
     }
     // Handle sliproads
-    intersection = sliproad_handler(from_nid, via_eid, std::move(intersection));
+    if (sliproad_handler.canProcess(from_nid, via_eid, intersection))
+        intersection = sliproad_handler(from_nid, via_eid, std::move(intersection));
 
     // Turn On Ramps Into Off Ramps, if we come from a motorway-like road
     if (node_based_graph.GetEdgeData(via_eid).road_classification.IsMotorwayClass())

src/extractor/guidance/turn_lane_augmentation.cpp

@@ -1,5 +1,3 @@
-#include "util/debug.hpp"
-
 #include "extractor/guidance/turn_lane_augmentation.hpp"
 #include "extractor/guidance/turn_lane_types.hpp"
 #include "util/simple_logger.hpp"

src/tools/routed.cpp

@@ -63,7 +63,8 @@ inline unsigned generateServerProgramOptions(const int argc,
                                              int &max_locations_trip,
                                              int &max_locations_viaroute,
                                              int &max_locations_distance_table,
-                                             int &max_locations_map_matching)
+                                             int &max_locations_map_matching,
+                                             int &max_results_nearest)
 {
     using boost::program_options::value;
     using boost::filesystem::path;
@@ -100,7 +101,10 @@ inline unsigned generateServerProgramOptions(const int argc,
          "Max. locations supported in distance table query") //
         ("max-matching-size",
          value<int>(&max_locations_map_matching)->default_value(100),
-         "Max. locations supported in map matching query");
+         "Max. locations supported in map matching query") //
+        ("max-nearest-size",
+         value<int>(&max_results_nearest)->default_value(100),
+         "Max. results supported in nearest query");
 
     // hidden options, will be allowed on command line, but will not be shown to the user
     boost::program_options::options_description hidden_options("Hidden options");
@@ -189,7 +193,8 @@ int main(int argc, const char *argv[]) try
                                                               config.max_locations_trip,
                                                               config.max_locations_viaroute,
                                                               config.max_locations_distance_table,
-                                                              config.max_locations_map_matching);
+                                                              config.max_locations_map_matching,
+                                                              config.max_results_nearest);
     if (init_result == INIT_OK_DO_NOT_START_ENGINE)
     {
         return EXIT_SUCCESS;

src/util/name_table.cpp

@@ -22,6 +22,9 @@ NameTable::NameTable(const std::string &filename)
 
     name_stream >> m_name_table;
 
+    if (!name_stream)
+        throw exception("Unable to deserialize RangeTable for NameTable");
+
     unsigned number_of_chars = 0;
     name_stream.read(reinterpret_cast<char *>(&number_of_chars), sizeof(number_of_chars));
     if (!name_stream)
@@ -64,5 +67,23 @@ std::string NameTable::GetNameForID(const unsigned name_id) const
     }
     return result;
 }
+
+std::string NameTable::GetRefForID(const unsigned name_id) const
+{
+    // Way string data is stored in blocks based on `name_id` as follows:
+    //
+    // | name | destination | pronunciation | ref |
+    //                                      ^     ^
+    //                                      [range)
+    //                                       ^ name_id + 3
+    //
+    // `name_id + offset` gives us the range of chars.
+    //
+    // Offset 0 is name, 1 is destination, 2 is pronunciation, 3 is ref.
+    // See datafacades and extractor callbacks for details.
+    const constexpr auto OFFSET_REF = 3u;
+    return GetNameForID(name_id + OFFSET_REF);
+}
+
 } // namespace util
 } // namespace osrm

taginfo.json

@@ -62,6 +62,24 @@
             "object_types": [ "way" ],
             "description": "Roads with hov=designated are ignored by default."
         },
+        {
+            "key": "hov:lanes",
+            "value": "designated",
+            "object_types": [ "way" ],
+            "description": "Roads with hov:lanes all-designated are ignored by default."
+        },
+        {
+            "key": "hov:lanes:forward",
+            "value": "designated",
+            "object_types": [ "way" ],
+            "description": "Roads with hov:lanes:forward all-designated are ignored by default."
+        },
+        {
+            "key": "hov:lanes:backward",
+            "value": "designated",
+            "object_types": [ "way" ],
+            "description": "Roads with hov:lanes:backward all-designated are ignored by default."
+        },
         {
             "key": "toll",
             "value": "yes",

unit_tests/library/limits.cpp

@@ -4,6 +4,7 @@
 #include "args.hpp"
 
 #include "osrm/match_parameters.hpp"
+#include "osrm/nearest_parameters.hpp"
 #include "osrm/route_parameters.hpp"
 #include "osrm/table_parameters.hpp"
 #include "osrm/trip_parameters.hpp"
@@ -136,4 +137,33 @@ BOOST_AUTO_TEST_CASE(test_match_limits)
     BOOST_CHECK(code == "TooBig"); // per the New-Server API spec
 }
 
+BOOST_AUTO_TEST_CASE(test_nearest_limits)
+{
+    const auto args = get_args();
+    BOOST_REQUIRE_EQUAL(args.size(), 1);
+
+    using namespace osrm;
+
+    EngineConfig config;
+    config.storage_config = {args[0]};
+    config.use_shared_memory = false;
+    config.max_results_nearest = 2;
+
+    OSRM osrm{config};
+
+    NearestParameters params;
+    params.coordinates.emplace_back(util::FloatLongitude{}, util::FloatLatitude{});
+    params.number_of_results = 10000;
+
+    json::Object result;
+
+    const auto rc = osrm.Nearest(params, result);
+
+    BOOST_CHECK(rc == Status::Error);
+
+    // Make sure we're not accidentally hitting a guard code path before
+    const auto code = result.values["code"].get<json::String>().value;
+    BOOST_CHECK(code == "TooBig"); // per the New-Server API spec
+}
+
 BOOST_AUTO_TEST_SUITE_END()