Open-Harbor/osrm-backend
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 3 Wiki Activity

Improve compile data in docker and publish data (#21)

Browse Source 
* chore: misc improve for telenav/osrm-backend docker

* chore: improve entrypoint for easy integrated by CI

* chore: fix profiles not found issue

* chore: also support package and publish compiled data by .tar.gz

* docs: update example

* docs: update for compile data automation

* docs: update item names

* docs: add docker run

* docs: update design doc
This commit is contained in:
Jay 2019-06-19 12:18:20 +08:00 committed by Xun(Perry) Liu
parent 79d565b2ec
commit b1446e4add
6 changed files with 125 additions and 34 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
docker-orchestration/README.md
View File

@ -6,7 +6,7 @@ Base image for telenav osrm-backend development, include all building and runnin
See details in [osrm-backend-dev docker](./osrm-backend-dev/). See details in [osrm-backend-dev docker](./osrm-backend-dev/).
### osrm-backend ### osrm-backend
Image within built osrm binaries(`osrm-extract/osrm-partition/osrm-customize/...`) and running dependencies. Image within built osrm binaries(`osrm-extract/osrm-partition/osrm-customize/...`) and running dependencies. It can be used to **compile data** or **startup routed**.
See details in [osrm-backend docker](./osrm-backend/) See details in [osrm-backend docker](./osrm-backend/)
### osrm-frontend ### osrm-frontend

10
docker-orchestration/osrm-backend/Dockerfile
View File

@ -39,15 +39,13 @@ RUN DEBIAN_FRONTEND=noninteractive apt-get update && \
libboost-iostreams1.62.0 libboost-thread1.62.0 expat liblua5.2-0 libtbb2 curl ca-certificates && \ libboost-iostreams1.62.0 libboost-thread1.62.0 expat liblua5.2-0 libtbb2 curl ca-certificates && \
rm -rf /var/lib/apt/lists/* rm -rf /var/lib/apt/lists/*
RUN mkdir /osrm-build RUN mkdir /osrm-build && mkdir /osrm-data
COPY --from=builder /osrm-build /osrm-build/ COPY --from=builder /osrm-build /osrm-build/
COPY --from=builder /osrm-data /osrm-data/
COPY --from=gobuilder /workspace/go/bin /osrm-build/ COPY --from=gobuilder /workspace/go/bin /osrm-build/
WORKDIR /osrm-build
# Publish
COPY ./docker-entrypoint.sh / COPY ./docker-entrypoint.sh /
RUN chmod +x /docker-entrypoint.sh RUN chmod +x /docker-entrypoint.sh
EXPOSE 5000
ENTRYPOINT ["/docker-entrypoint.sh"] ENTRYPOINT ["/docker-entrypoint.sh"]

72
docker-orchestration/osrm-backend/README.md
View File

@ -1,18 +1,74 @@
# telenav osrm-backend docker # telenav osrm-backend docker
Image within built osrm binaries(`osrm-extract/osrm-partition/osrm-customize/...`) and running dependencies. It can be used to **compile data** or **startup routed**.
## Build Image ## Build Image
Please use below jenkins job to build docker image within osrm binaries from source code.
- [(Telenav Internal) Jenkins Job - Build_Telenav_OSRM_Backend_Docker](https://shd-routingfp-01.telenav.cn:8443/view/OSRM/job/Build_Telenav_OSRM_Backend_Docker/)
## Compile Data
Please use below jenkins job to compile mapdata from PBF to osrm data.
There're two options for publish the compiled osrm data:
- generate new docker image from current one within compile data
- export as `map.tar.gz`
See below job for details:
- [(Telenav Internal) Jenkins Job - Compile_Mapdata_In_Telenav_OSRM_Backend_Docker](https://shd-routingfp-01.telenav.cn:8443/view/OSRM/job/Compile_Mapdata_In_Telenav_OSRM_Backend_Docker/)
## Run
### Run with osrm-data inside
```bash ```bash
$ cd docker-orchestration/osrm-backend $ docker pull artifactory.telenav.com/telenav-docker-preprod/osrm-backend:8__9__master-telenav__20190618T135121CST__20190514T140904CST-california-latest
$
# build source from default branch master-telenav # run with default trafficproxy
$ DOCKER_BUILDKIT=1 docker build -t telenav/osrm-backend . $ docker run -d -p 5000:5000 artifactory.telenav.com/telenav-docker-preprod/osrm-backend:8__9__master-telenav__20190618T135121CST__20190514T140904CST-california-latest routed_startup
# build source from specified branch, e.g. feature/telenav-import-internal-pbf
$ DOCKER_BUILDKIT=1 docker build -t telenav/osrm-backend:telenav-import-internal-pbf --build-arg BRANCH_NAME=feature/telenav-import-internal-pbf .
# run with another trafficproxy(e.g. 10.189.102.81)
$ docker run -d -p 5000:5000 artifactory.telenav.com/telenav-docker-preprod/osrm-backend:8__9__master-telenav__20190618T135121CST__20190514T140904CST-california-latest routed_startup 10.189.102.81
``` ```
## Example ### Run with osrm-data outside
```bash
# prepare compiled data first (sample data: california)
$ cd osrm-data
$ ll -lh
total 243M
-rw-r--r-- 1 mapuser appuser 21M Jun 17 23:37 map.osrm.cell_metrics
-rw-r--r-- 1 mapuser appuser 193K Jun 17 23:34 map.osrm.cells
-rw-r--r-- 1 mapuser appuser 1.9M Jun 17 23:34 map.osrm.cnbg
-rw-r--r-- 1 mapuser appuser 1.9M Jun 17 23:34 map.osrm.cnbg_to_ebg
-rw-r--r-- 1 mapuser appuser 68K Jun 17 23:37 map.osrm.datasource_names
-rw-r--r-- 1 mapuser appuser 9.8M Jun 17 23:34 map.osrm.ebg
-rw-r--r-- 1 mapuser appuser 2.8M Jun 17 23:34 map.osrm.ebg_nodes
-rw-r--r-- 1 mapuser appuser 2.9M Jun 17 23:34 map.osrm.edges
-rw-r--r-- 1 mapuser appuser 2.7M Jun 17 23:34 map.osrm.enw
-rwx------ 1 mapuser appuser 5.6M Jun 17 23:34 map.osrm.fileIndex
-rw-r--r-- 1 mapuser appuser 7.3M Jun 17 23:37 map.osrm.geometry
-rw-r--r-- 1 mapuser appuser 1.1M Jun 17 23:34 map.osrm.icd
-rw-r--r-- 1 mapuser appuser 5.0K Jun 17 23:34 map.osrm.maneuver_overrides
-rw-r--r-- 1 mapuser appuser 11M Jun 17 23:37 map.osrm.mldgr
-rw-r--r-- 1 mapuser appuser 218K Jun 17 23:34 map.osrm.names
-rw-r--r-- 1 mapuser appuser 4.0M Jun 17 23:34 map.osrm.nbg_nodes
-rw-r--r-- 1 mapuser appuser 1.8M Jun 17 23:34 map.osrm.partition
-rw-r--r-- 1 mapuser appuser 6.0K Jun 17 23:34 map.osrm.properties
-rw-r--r-- 1 mapuser appuser 29K Jun 17 23:34 map.osrm.ramIndex
-rw-r--r-- 1 mapuser appuser 4.0K Jun 17 23:34 map.osrm.restrictions
-rw-r--r-- 1 mapuser appuser 3.5K Jun 17 23:34 map.osrm.timestamp
-rw-r--r-- 1 mapuser appuser 5.5K Jun 17 23:34 map.osrm.tld
-rw-r--r-- 1 mapuser appuser 8.0K Jun 17 23:34 map.osrm.tls
-rw-r--r-- 1 mapuser appuser 836K Jun 17 23:34 map.osrm.turn_duration_penalties
-rw-r--r-- 1 mapuser appuser 4.9M Jun 17 23:34 map.osrm.turn_penalties_index
-rw-r--r-- 1 mapuser appuser 836K Jun 17 23:34 map.osrm.turn_weight_penalties
$ cd ..
# pull & run
$ docker pull wangyoucao577/osrm-backend:9__master-telenav__20190618T135121CST
$ docker run -d -p 5000:5000 "src=$(pwd)/osrm-data,dst=/osrm-data,type=bind" wangyoucao577/osrm-backend:9__master-telenav__20190618T135121CST routed_startup
```
## Example By Manual
- [Build Berlin Server with OSM data](./example-berlin-osm.md) - [Build Berlin Server with OSM data](./example-berlin-osm.md)

53
docker-orchestration/osrm-backend/docker-entrypoint.sh
View File

@ -1,5 +1,8 @@
#!/bin/bash #!/bin/bash -x
BUILD_PATH=${BUILD_PATH:="/osrm-build"}
DATA_PATH=${DATA_PATH:="/osrm-data"} DATA_PATH=${DATA_PATH:="/osrm-data"}
OSRM_EXTRA_COMMAND="-l --verbosity"
MAPDATA_NAME_WITH_SUFFIX=map
_sig() { _sig() {
kill -TERM $child 2>/dev/null kill -TERM $child 2>/dev/null
@ -7,19 +10,49 @@ _sig() {
if [ "$1" = 'routed_startup' ]; then if [ "$1" = 'routed_startup' ]; then
trap _sig SIGKILL SIGTERM SIGHUP SIGINT EXIT trap _sig SIGKILL SIGTERM SIGHUP SIGINT EXIT
./osrm-routed $DATA_PATH/$2.osrm -a MLD --max-table-size 8000 &
TRAFFIC_FILE=traffic.csv
TRAFFIC_PROXY_IP=${2:-"10.189.102.81"}
cd ${DATA_PATH}
${BUILD_PATH}/osrm_traffic_updater -c ${TRAFFIC_PROXY_IP} -d=false -f ${TRAFFIC_FILE}
ls -lh
${BUILD_PATH}/osrm-customize ${MAPDATA_NAME_WITH_SUFFIX}.osrm --segment-speed-file ${TRAFFIC_FILE} ${OSRM_EXTRA_COMMAND}
${BUILD_PATH}/osrm-routed ${MAPDATA_NAME_WITH_SUFFIX}.osrm -a MLD --max-table-size 8000 &
child=$! child=$!
wait "$child" wait "$child"
elif [ "$1" = 'compile_mapdata' ]; then elif [ "$1" = 'compile_mapdata' ]; then
trap _sig SIGKILL SIGTERM SIGHUP SIGINT EXIT trap _sig SIGKILL SIGTERM SIGHUP SIGINT EXIT
if [ ! -f $DATA_PATH/$2.osrm ]; then
if [ ! -f $DATA_PATH/$2.osm.pbf ]; then PBF_FILE_URL=${2}
curl $3 > $DATA_PATH/$2.osm.pbf KEEP_COMPILED_DATA=${3:-"false"}
fi GENERATE_DATA_PACKAGE=${4:-"false"}
./osrm-extract $DATA_PATH/$2.osm.pbf -p profiles/car.lua
./osrm-partition $DATA_PATH/$2.osrm curl ${PBF_FILE_URL} > $DATA_PATH/${MAPDATA_NAME_WITH_SUFFIX}.osm.pbf
./osrm-customize $DATA_PATH/$2.osrm ${BUILD_PATH}/osrm-extract $DATA_PATH/${MAPDATA_NAME_WITH_SUFFIX}.osm.pbf -p ${BUILD_PATH}/profiles/car.lua ${OSRM_EXTRA_COMMAND}
fi ${BUILD_PATH}/osrm-partition $DATA_PATH/${MAPDATA_NAME_WITH_SUFFIX}.osrm ${OSRM_EXTRA_COMMAND}
${BUILD_PATH}/osrm-customize $DATA_PATH/${MAPDATA_NAME_WITH_SUFFIX}.osrm ${OSRM_EXTRA_COMMAND}
# clean source pbf and temp .osrm
rm -f $DATA_PATH/${MAPDATA_NAME_WITH_SUFFIX}.osm.pbf
rm -f $DATA_PATH/${MAPDATA_NAME_WITH_SUFFIX}.osrm
# package and publish compiled mapdata
if [ ${GENERATE_DATA_PACKAGE} == "true" ]; then
cd ${DATA_PATH}
tar -zcf ${MAPDATA_NAME_WITH_SUFFIX}.tar.gz *
SAVE_DATA_PACKAGE_PATH=/save-data
mkdir -p ${SAVE_DATA_PACKAGE_PATH}
mv ${DATA_PATH}/${MAPDATA_NAME_WITH_SUFFIX}.tar.gz ${SAVE_DATA_PACKAGE_PATH}/
fi
# rm compiled data if not needed
if [ ${KEEP_COMPILED_DATA} != "true" ]; then
rm -f $DATA_PATH/*
fi
else else
exec "$@" exec "$@"
fi fi

15
docker-orchestration/osrm-backend/example-berlin-osm.md
View File

@ -2,21 +2,28 @@
- Generate docker image - Generate docker image
```bash ```bash
DOCKER_BUILDKIT=1 docker build --no-cache -t telenav/osrm-backend:docker-orchestration-perry --build-arg BRANCH_NAME=feature/docker-orchestration-perry . $ DOCKER_BUILDKIT=1 docker build --no-cache -t telenav/osrm-backend:compile-data-inside-docker --build-arg BRANCH_NAME=feature/compile-data-inside-docker .
``` ```
- Generate OSRM data - Generate OSRM data
```bash ```bash
docker run -d -v /Users/ngxuser/osrm-data/berlin_osm:/osrm-data --name osrm-data telenav/osrm-backend:docker-orchestration-perry compile_mapdata berlin "https://download.geofabrik.de/europe/germany/berlin-latest.osm.pbf" $ mkdir -p compiled-data
$ docker run -it --mount "src=$(pwd)/compiled-data,dst=/save-data,type=bind" telenav/osrm-backend:compile-data-inside-docker compile_mapdata "https://download.geofabrik.de/europe/germany/berlin-latest.osm.pbf" false true
$ ll compiled-data/
total 34968
-rw-r--r-- 1 root root 35805710 Jun 17 23:34 map.tar.gz
``` ```
- Start OSRM server - Start OSRM server
```bash ```bash
docker run -d -p 5000:5000 -v /Users/ngxuser/osrm-data/berlin_osm:/osrm-data --name osrm-api telenav/osrm-backend:docker-orchestration-perry routed_startup berlin $ cd compiled-data
$ tar -zxf map.tar.gz
$ docker run -d -p 5000:5000 --mount "src=$(pwd),dst=/osrm-data,type=bind" --name osrm-api telenav/osrm-backend:compile-data-inside-docker routed_startup
05227a108a66e7c59f7515f8f174a65f9932f36fe3807f83991806c0194a7e50
``` ```
You should see such logs in docker You should see such logs in docker
``` ```
# docker logs -f container_id $ docker logs 05227a108a66e7c59f7515f8f174a65f9932f36fe3807f83991806c0194a7e50
[info] starting up engines, v5.22.0 [info] starting up engines, v5.22.0
[info] Threads: 10 [info] Threads: 10
[info] IP address: 0.0.0.0 [info] IP address: 0.0.0.0

7
docs/design/osrm-with-telenav-traffic.md
View File

@ -3,18 +3,15 @@
## Architecture ## Architecture
![osrm-with-telenav-traffic-architecture](./graph/osrm-with-telenav-traffic-architecture.mmd.png) ![osrm-with-telenav-traffic-architecture](./graph/osrm-with-telenav-traffic-architecture.mmd.png)
### OSRMTrafficUpdater(need to implement) ### OSRMTrafficUpdater
- as client - as client
- connect `TrafficProxy` by `RPC` - connect `TrafficProxy` by `RPC`
- convert contents from `RPC` protocol to `OSRM` required `csv` format, then write to file - convert contents from `RPC` protocol to `OSRM` required `csv` format, then write to file
### TrafficProxy(need to implement) ### TrafficProxy
- as server - as server
- provide traffic contents by region - provide traffic contents by region
- contents include at least `from node, to node, speed`(both `from node` and `to node` are come from original mapdata) - contents include at least `from node, to node, speed`(both `from node` and `to node` are come from original mapdata)
- known issues/questions:
- can not compile `OSM` mapdata to traffic graph?
## OSRM with Traffic Startup Flow ## OSRM with Traffic Startup Flow
![osrm-with-traffic-startup-flow-chart](./graph/osrm-with-traffic-startup-flow-chart.mmd.png) ![osrm-with-traffic-startup-flow-chart](./graph/osrm-with-traffic-startup-flow-chart.mmd.png)