Commit Graph

1594 Commits

Author SHA1 Message Date
Michael Bell
04c2167c2d Fix for_each_indexed util function
- Fix typo in util function name for_each_indexed.
- Use the overloaded functions for_each_indexed and for_each_pair
  with a container argument where possible to improve readability.
2021-12-22 23:40:06 +00:00
Daniel Patterson
d5cd702242
Enable all unit tests (#5248)
* Add missing profile name to library extract test.

* Support both tzid and TZID properties on timezone geometry.  Improve validation of timezone polygons.

* Missing tzid property wasn't a geojson validation issue, shouldn't have been tested there.

* Use filesystem glob to loop over all test executables so we don't miss any in the future.

Co-authored-by: Michael Bell <michael@mjjbell.com>
2021-10-22 22:10:25 +01:00
Michael Bell
56a427f839 Upgrade Boost to 1.70, fix inefficient connection handling
A request to osrm-routed can be assigned to a thread which
is currently busy processing another request, even when there
are other threads/cores available. This unnecessarily delays
the response, and can make requests appear to hang when
awaiting CPU intensive requests to finish.

The issue looks like a bug in Boost.Asio multithreaded
networking stack.

osrm-routed server implementation is
heavily influenced by the HTTP server 3 example in the
Boost.Asio docs. By upgrading to Boost 1.70 and updating the
server connections to match the example provided in the 1.70
release, the problem is resolved.

The diff of the changes to the Boost.Asio stack are
vast, so it's difficult to identify the exact cause. However
the implementation change is to push the strand of execution
into the socket (and timer) objects, which suggests it could
fix the type of threading issue we are observing.
2021-09-30 22:51:12 +01:00
Michael Bell
f6349a7fbe
Fix metric offset overflow for large MLD partitions (#6124)
Each MLD cell has source and destination nodes.
MLD is keeping a |source| x |destination| sized table
for various metrics (distances, durations, etc) from each
source to all destinations in a cell.

It stores all of the values for a metric in one large array, with
an offset for each cell to find its values. The offset is currently
limited to 32 bit values, which overflows on very large graphs
(e.g. Planet OSM).

We fix this by changing the offsets to be uint64_t types.
2021-09-21 19:25:30 +01:00
Michael Bell
f1f96166c5
Fix MLD level mask generation to support 64-bit masks. (#6123)
The generation of level masks for compactly storing partition cells
supports sizes that can be stored in 64 bits.

The current implementation fails if the total bit sum is 64 bits
exactly. A bit shift mechanism is used that is undefined when the
shift size is equal to the bit size of the underlying type. This
generates an incorrect mask value.

We fix this by adding a special case for a 64 bit offset. Given this
code is called at most |level| times, there will be no effect on
performance. We also update the assertions to reflect 64 bit masks
are now supported.
2021-09-21 18:09:38 +01:00
Michael Bell
f1a6056953 Immediately close bad connections to prevent file exhaustion
osrm-routed does not immediately clean up a keep-alive connection
when the client closes it. Instead it waits for five seconds
of inactivity before removing.

Given a setup with low file limits and clients opening and
closing a lot of keep-alive connections, it's possible for
osrm-routed to run out of file descriptors whilst it waits for
the clean-up to trigger.

Furthermore, this causes the connection acceptor loop to exit.
Even after the old connections are cleaned up, new ones
will not be created. Any new requests will block until the
server is restarted.

This commit improves the situation by:

- Immediately closing connections on error. This includes EOF errors
indicating that the client has closed the connection. This releases
resources early (including the open file) and doesn't wait for the
timer.

- Log when the acceptor loop exits. Whilst this means the behaviour
can still occur for reasons other than too many open files,
we will at least have visibility of the cause and can investigate further.
2021-09-04 01:55:36 +01:00
Michael Bell
50f5a753ea Increase PackedOSMIDs size to 34 bits
OSM node 2^33 was created in early April 2021. This and all
subsequently created IDs will be overflowing OSRM node storage
which only support 33 bit IDs.
Bump the number of bits to 34 to double node ID capacity. This
is a breaking change to the data format as it alters the layout
of .osrm.nbg_nodes.
2021-04-24 23:41:02 +01:00
Karen Shea
a613375460
Validate source/destination indices correctly in nodejs support (#5595)
* validate source/destination indices correctly

Co-authored-by: Denis Chapligin <denis.chaplygin@wolt.com>
Co-authored-by: Daniel Patterson <danpat@danpat.net>
2021-01-28 06:02:01 -08:00
Richard Fairhurst
960269f95a
Merge pull request #5131 from themylogin/master
Fix turn.roads_on_the_left and turn.roads_on_the right for two-way roads
2021-01-28 14:00:17 +00:00
Michael Bell
bd3eb6591e
Undo libosrm API break by adding old interface as method overload (#5861)
Removes the breaking libosrm API change by adding the old interface to
the new. This does not introduce any new breaks.

The downside of this is that it allows for multiple ways to
return JSON responses.
2021-01-27 09:14:44 -08:00
themylogin
b0b8069ab0 clang-format 2021-01-27 13:12:20 +01:00
themylogin
d9d873903f Account merged edges when processing turn legs 2021-01-27 10:53:48 +01:00
Michael Bell
e554624438 Reduce copying in API parameter constructors
When using non-default constructors for the API parameter classes,
vector arguments like coordinates and hints are copied at least once
(twice when passed as lvalue arguments).

Enable perfect forwarding of BaseParameter arguments and
pass-by-value in the constructor that uses the argument. This
ensures we copy at most once (zero for rvalue arguments).
2021-01-06 23:15:35 +00:00
Michael Bell
5266ac1635
Add support for multiple via-way restrictions (#5907)
Currently OSRM only supports turn restrictions with a single via-node or one
via-way. OSM allows for multiple via-ways to represent longer and more
complex restrictions.

This PR extends the use of duplicate nodes for representng via-way turn
restrictions to also support multi via-way restrictions. Effectively, this
increases the edge-based graph size by the number of edges in multi via-way
restrictions. However, given the low number of these restrictions it
has little effect on total graph size.

In addition, we add a new step in the extraction phase that constructs
a restriction graph to support more complex relationships between restrictions,
such as nested restrictions and overlapping restrictions.
2020-12-20 13:59:57 -08:00
Denis Chapligin
eb1d399f3b
Merge pull request #5894 from xlaussel/avoid_samelookup_in_heap_map
Avoid samelookup in heap map
2020-12-20 22:17:49 +02:00
Alin Mindroc
33b2a193f9
Fix node14 compilation issues (#5918)
* Use Nan:: getters and setters for node 12/14 compatibility
* Drop support for publishing node 8 bindings, add publishing support for node 12, 14

Co-authored-by: Daniel Patterson <danpat@danpat.net>
2020-12-19 15:37:01 -08:00
Desone Burns II
0e10b94d38 Merge branch 'master' of github.com:Project-OSRM/osrm-backend into HEAD 2020-12-01 19:24:11 -07:00
Denis Chapligin
18a0f6c1f8
Merge pull request #5891 from mloskot/ml/check-indexed-data-for-empty-values
Avoid dereferencing iterators over empty indexed data values
2020-12-01 15:26:58 +02:00
xlaussel
687a3cdfbe Merge branch 'master' into avoid_samelookup_in_heap_map 2020-11-26 21:29:27 +01:00
Daniel Patterson
50d9632ed7
Upgrade formatting to clang-format 10 (#5895)
* Update formatting tools to clang-format-10

* Reformat using clang-format-10.0.09
2020-11-26 07:21:39 -08:00
xlaussel
fcb7dd2a21 format code 2020-11-25 12:42:01 +01:00
xlaussel
aa06029801 References removed for extracted heapNode: could lead to bugs because the same was sometimes modified after when relaxing outgoing edges 2020-11-25 11:22:30 +01:00
Desone Burns II
cf13a9714d clang-format-3.8 changed files 2020-11-24 08:06:26 -07:00
xlaussel
8697a6b14a Changes and corrections before change request
Cucumber successfull
2020-11-24 12:43:08 +01:00
xlaussel
697c2ff88d other improvements.
Variable renaming with better names
2020-11-23 23:11:41 +01:00
xlaussel
e181cb325c other improvements.
Variable renaming with better names
2020-11-23 23:10:22 +01:00
xlaussel
41af9615cd Improvements 2020-11-23 23:00:27 +01:00
xlaussel
13067844ee Changes made 2020-11-23 22:33:08 +01:00
Mateusz Łoskot
58fbda0483
Avoid dereferencing iterators over empty indexed data values
It seems the values can be empty even if size of blocks
is non-zero (e.g. blocks.size()==1)
2020-11-23 21:21:35 +01:00
Desone Burns II
e693b8963a Remove unsigned 0 comparison 2020-11-19 17:15:17 -07:00
Desone Burns II
43b920bfb5 Merge branch 'master' of https://github.com/Project-OSRM/osrm-backend into upgrade_libsol2_v2.20.6 2020-11-18 17:11:02 -07:00
Desone Burns II
deacc8fb5c Update to sol2 v2.20.6 2020-11-18 17:01:43 -07:00
Michael Bell
96acdaf0d5 Fix osrm-contract, tests, on Windows
As part of graph contraction, node renumbering leads to
in-place permuting of graph state, including boolean vector elements.

std::vector<bool> returns proxy objects when referencing individual
bits. To correctly swap bool elements using MSVC, we need to explicitly
apply std::vector<bool>::swap.

Making this change fixes osrm-contract on Windows.

We also correct failing tests and other undefined behaviours
(mainly iterator access outside boundaries) highlighted by MSVC.
2020-11-15 14:22:26 +00:00
Denis Chapligin
98fd17589d
Merge pull request #5868 from wangyoucao577/feature/unify-turn-index-dump
Unify .osrm.turn_penalties_index dump same with turn_weight_penalties/turn_duration_penalties
2020-11-10 11:20:55 +02:00
Michael Bell
b840c0be95 Fix bit-shift overflow in MLD partition step for Windows builds
For very large graphs, generation of MLD level masks fail on Windows
due to bit shift overflow of unsigned long values.

Correct by using unsigned long long literals, which are 64 bit on
all major systems.
2020-11-08 03:51:11 +00:00
Jay Zhang
17eb39d0a9 fix: clangformat 2020-10-26 01:19:17 +00:00
Jay Zhang
7d28b7aa3a feat: unify turn_penalties_index dump same with turn_weight_penalties and turn_duration_penalties 2020-10-23 09:38:56 +00:00
Daniel Patterson
3451d1ec82
Lock access to facade_factory in data_watchdog to avoid accessing destructed object (#5844)
* Wrap access to facade_factory in a shared lock so it doesn't get changed partway through access which leads to a crash.
2020-09-30 18:45:44 -07:00
Michael Bell
4799b46eeb
Incorrect error message when unable to snap all input coordinates (#5846)
In cases where we are unable to find a phantom node for an input
coordinate, we return an error indicating which coordinate failed.

This would always refer to the coordinate with index equal to the
number of valid phantom nodes found.

We fix this by instead returning the first index for which a
phantom node could not be found.
2020-09-30 18:44:22 -07:00
Vyacheslav Napadovsky
f545f2293e Make MMapMemoryAllocator map files in read-only (O_RDONLY) mode. Mapping with O_RDWR will cause copy_up across Docker layers. 2020-09-09 00:51:42 +03:00
Denis Chaplygin
933e36190f
Removed catching of obsolete tbb:captured_exception, as TBB now supports C++ native std::exception_ptr inter-thread exception passing mechanism. 2020-09-04 09:33:43 +03:00
Denis Chaplygin
2d0b2ca987
Removed STXXL support. 2020-08-20 11:54:02 +03:00
Lev Dragunov
699ca2bbd0
Merge pull request #5758 from woltapp/gcc10
Fixes signed/unsigned comparision spotted by gcc10.
2020-08-20 10:59:23 +03:00
Łukasz Jagielski
9861faa456 nodejs: user worker thread friendly initialization 2020-07-07 09:01:52 +02:00
Denis Chaplygin
919fe74c40
Fixes signed/unsigned comparision spotted by gcc10.
Fixed by moving assertion to the actual point where signed values are used.
2020-06-17 13:18:21 +03:00
Łukasz Jagielski
cbec111eb6 include/nodejs/node_osrm.hpp: clang-format 2020-05-12 15:16:38 +02:00
Łukasz Jagielski
7d369b9ab9 Try to fix node 12 compilation issues 2020-05-05 23:34:26 +02:00
marius
6d8cf826d7 modified FSM so that the whitespace between header-key and value is optional 2019-11-19 18:30:21 +01:00
Lev Dragunov
0b139ff05d
Merge pull request #5572 from Tomonobu3110/tomo_less_memory
Reduce memory usage for raster source handling.
2019-11-14 12:58:59 +03:00
Tomonobu Saito
9da6cf8764 add: comment for RasterCache class 2019-11-13 11:01:07 +09:00
Tomonobu Saito
ee177efe41 fix: remove unused include / add const for necessity portion 2019-11-08 16:15:38 +09:00
Tomonobu Saito
fd0f1b60bb fix by revier comments 2019-11-06 11:06:29 +09:00
Tomonobu Saito
17f32f4ca1 fix coding format 2019-10-09 13:35:19 +09:00
Tomonobu Saito
542c3ba872 Add singletone class RasterCache to handle global cache data. 2019-10-09 13:02:59 +09:00
Tomonobu Saito
f36707d1fb revert file mode to 664 2019-10-04 17:39:05 +09:00
Tomonobu Saito
a587b14006 destruct static vector and unorderd_map when last RasterContainer is destructed. 2019-10-04 17:30:35 +09:00
Tomonobu Saito
f9ee74d78e Add x range check to avoid data corruption. 2019-10-04 16:47:31 +09:00
Tomonobu Saito
eef072234e update delim chars 2019-10-04 15:14:20 +09:00
Tomonobu Saito
432d49e23d bugfix: support multiple delimitor. 2019-10-04 13:50:13 +09:00
Tomonobu Saito
e4aaf07879 remove unused lines 2019-10-03 17:24:42 +09:00
Tomonobu Saito
d316ff9d41 Improvement of raster source loading to reduce consumed memory size. 2019-10-03 16:44:27 +09:00
Tomonobu Saito
62c8b70f78 use boost::filesystem::file_size() to get the file size (instead of seeking the file). 2019-10-02 19:04:01 +09:00
Tomonobu.Saito (Desktop PC)
d8d9ac8686 Add assert to check memory allocation 2019-10-02 15:47:16 +09:00
Denis Chaplygin
016c77a4de Flatbuffers is not a boost::optional friendly 2019-10-01 17:26:10 +03:00
Tom Peoples
28895373fb Fixed flatbufferbuiler copy issues.
Compiling under gcc9.1 we get copy issues.
It appears we shouldn't pass builder classes by value, only ref.
2019-10-01 21:54:00 +10:00
Tom Peoples
018a9bc804 Removed un-needed calls to std::move
These calls were throwing a pessimistic move error and stopping compilation.
2019-09-26 17:55:19 +10:00
Denis Chaplygin
e97ec13064 Formatted the sources 2019-09-17 17:42:52 +03:00
Denis Chaplygin
660c0cc602 Implemented 'skip_waypoints' for the 'Table' service. 2019-09-17 17:42:52 +03:00
Denis Chaplygin
600f48e15a Implemented 'skip_waypoints' support in 'Route', 'Trip', 'Match' services. 2019-09-17 17:42:52 +03:00
Denis Chaplygin
1b47242a58 Implemented 'skip_waypoints' for the 'Nearest' service.
It actually makes no sence, as the only output on the 'Nearest' service
is a list of wypoints. On the other hand it can be used now as a simple
health check.
2019-09-17 17:42:52 +03:00
Denis Chaplygin
10c1b38139 Added new base parameter 'skip_waypoints' 2019-09-17 17:42:52 +03:00
Denis Chaplygin
0be87c73cf Fixed typo in flatbuffers: longitute -> longitude. 2019-08-26 14:31:51 +03:00
Denis Chaplygin
b8182dd7ff Replaced 'in' and 'out' at the flatbuffers 'instersection' object
with 'in_bearing' and 'out_bearing' to get flatbuffers schema javascript friendly.

Added a cucumber test for flatbuffers support.
2019-08-26 14:03:29 +03:00
Denis Chaplygin
80e1abba0c Added setting 'data_version' to all flatbuffers response functions. 2019-08-26 11:09:38 +03:00
Denis Chaplygin
97270ae473 Fixed nested offsets inside of the flatbuffers structure 2019-08-26 11:09:38 +03:00
Denis Chaplygin
a9c187c99b Updated changelog entry 2019-08-26 11:09:38 +03:00
Denis Chaplygin
2a1593fe05 Flatbuffers now uses floats instead of double. We do not use that precision anyway and it make output much smaller. 2019-08-26 11:08:56 +03:00
Denis Chaplygin
73f544d4da Slightly changed the serialization order. 2019-08-26 11:08:56 +03:00
Denis Chaplygin
889c82516f Set flatbuffers root type and removed unneded array for rows/cols 2019-08-26 11:08:56 +03:00
Denis Chaplygin
306c047c14 Generalized flatbuffers response generation in 'Route','Trip','Match' services. 2019-08-26 11:08:56 +03:00
Denis Chaplygin
533f1ffc6b Inlined routes and waypoints, as all services are using at least one of those object, so there is no need for another one level of indirection. 2019-08-26 11:08:56 +03:00
Denis Chaplygin
90191c9f34 Added flatbuffers output support to the 'Match' service 2019-08-26 11:08:56 +03:00
Denis Chaplygin
b7e9e5f2c0 Added flatbuffers output support to the 'Trip' service. 2019-08-26 11:08:56 +03:00
Denis Chaplygin
13287d9970 Optimizing FB schema:
* Replaced Response union with nullable fields
* "Ok" code replaced with a boolean and a separate structure for a error code and message.
* Inlined geometry, thus removing one layer of indirection.
2019-08-26 11:08:56 +03:00
Denis Chaplygin
affa8a4653 Added flatbuffers output support to the 'Route' service. 2019-08-26 11:08:56 +03:00
Denis Chaplygin
9d7a74445d Added flatbuffers output support to the 'Nearest' service. 2019-08-26 11:08:56 +03:00
Denis Chaplygin
acd3e81ab4 Removed ineffectivness in the serialization schema. Positions are
structs now and distance/duration matrices are just arrays.
2019-08-26 11:08:56 +03:00
Denis Chaplygin
ea111129dd Added full flatbuffers support to the Table service 2019-08-26 11:08:56 +03:00
Denis Chaplygin
7ddda105a3 Added flatbuffers support to the PluginBase::Error
This changes BREAKS osrm.
2019-08-26 11:08:56 +03:00
Denis Chaplygin
75aadb0f3f Passed ResultT down to engine plugins, so now they can form replies in different formats. 2019-08-26 11:08:56 +03:00
Denis Chaplygin
f6f86b2a52 Added flatbuffers schema. 2019-08-26 11:08:56 +03:00
Denis Chaplygin
59a83bd537 Added output format parser to the other services.
Same as for Route service, you can now specify either json or flatbuffers
as output format for the other services (except Tile service, where it
makes no sense).
2019-08-26 11:08:56 +03:00
Denis Chaplygin
59ca7840f4 Added output format parser to the Route service.
It is possible now to choose between json and flatbuffers (not yet implemented) output format.
Only route service is supported atm.
2019-08-26 11:08:56 +03:00
Denis Chaplygin
9efcab2108 Receive timeout should only be active for a second and
following requests on a keep-alive connections.
2019-08-23 12:21:54 +03:00
Denis Chaplygin
a7b7d77e1f Formatted the code. 2019-08-23 12:21:54 +03:00
Denis Chaplygin
22550d078f Added timeout handling for keep-alive operations. 2019-08-23 12:21:54 +03:00
Denis Chaplygin
a0582a3e68 Added keep-alive support to the http server. 2019-08-23 12:21:54 +03:00
Denis Chaplygin
92c7b6fbd1 Added 'Connection' header extraction. 2019-08-23 12:21:54 +03:00
Přemysl Šťastný
b4f849adaa Solved this warning:
In file included from /home/travis/build/Project-OSRM/osrm-backend/include/storage/shared_data_index.hpp:4,
                 from /home/travis/build/Project-OSRM/osrm-backend/include/engine/datafacade/contiguous_block_allocator.hpp:4,
                 from /home/travis/build/Project-OSRM/osrm-backend/include/engine/datafacade/contiguous_internalmem_datafacade.hpp:5,
                 from /home/travis/build/Project-OSRM/osrm-backend/include/engine/plugins/nearest.hpp:5,
                 from /home/travis/build/Project-OSRM/osrm-backend/src/engine/plugins/nearest.cpp:1:
/home/travis/build/Project-OSRM/osrm-backend/include/storage/shared_datatype.hpp:102:26: warning: inline function ‘virtual void* osrm::storage::BaseDataLayout::GetBlockPtr(void*, const string&) const’ used but never defined
     virtual inline void *GetBlockPtr(void *base_ptr, const std::string &name) const = 0;
2019-05-28 12:45:47 +02:00
Daniel Patterson
2c7c18fd24 Fix bug in snapping=any when bearings or radiuses are supplied. 2019-02-15 20:42:30 -08:00