From b1446e4add0cd3e7d1f0a5d45cd18bcc1c51e65d Mon Sep 17 00:00:00 2001 From: Jay Date: Wed, 19 Jun 2019 12:18:20 +0800 Subject: [PATCH] Improve compile data in docker and publish data (#21) * chore: misc improve for telenav/osrm-backend docker * chore: improve entrypoint for easy integrated by CI * chore: fix profiles not found issue * chore: also support package and publish compiled data by .tar.gz * docs: update example * docs: update for compile data automation * docs: update item names * docs: add docker run * docs: update design doc --- docker-orchestration/README.md | 2 +- docker-orchestration/osrm-backend/Dockerfile | 10 ++- docker-orchestration/osrm-backend/README.md | 72 ++++++++++++++++--- .../osrm-backend/docker-entrypoint.sh | 53 +++++++++++--- .../osrm-backend/example-berlin-osm.md | 15 ++-- docs/design/osrm-with-telenav-traffic.md | 7 +- 6 files changed, 125 insertions(+), 34 deletions(-) diff --git a/docker-orchestration/README.md b/docker-orchestration/README.md index a573dad5d..6003bedeb 100644 --- a/docker-orchestration/README.md +++ b/docker-orchestration/README.md @@ -6,7 +6,7 @@ Base image for telenav osrm-backend development, include all building and runnin See details in [osrm-backend-dev docker](./osrm-backend-dev/). ### osrm-backend -Image within built osrm binaries(`osrm-extract/osrm-partition/osrm-customize/...`) and running dependencies. +Image within built osrm binaries(`osrm-extract/osrm-partition/osrm-customize/...`) and running dependencies. It can be used to **compile data** or **startup routed**. See details in [osrm-backend docker](./osrm-backend/) ### osrm-frontend diff --git a/docker-orchestration/osrm-backend/Dockerfile b/docker-orchestration/osrm-backend/Dockerfile index 1235903bd..eb7519254 100644 --- a/docker-orchestration/osrm-backend/Dockerfile +++ b/docker-orchestration/osrm-backend/Dockerfile @@ -39,15 +39,13 @@ RUN DEBIAN_FRONTEND=noninteractive apt-get update && \ libboost-iostreams1.62.0 libboost-thread1.62.0 expat liblua5.2-0 libtbb2 curl ca-certificates && \ rm -rf /var/lib/apt/lists/* -RUN mkdir /osrm-build +RUN mkdir /osrm-build && mkdir /osrm-data COPY --from=builder /osrm-build /osrm-build/ -COPY --from=builder /osrm-data /osrm-data/ COPY --from=gobuilder /workspace/go/bin /osrm-build/ - -WORKDIR /osrm-build - -# Publish COPY ./docker-entrypoint.sh / RUN chmod +x /docker-entrypoint.sh + +EXPOSE 5000 ENTRYPOINT ["/docker-entrypoint.sh"] + diff --git a/docker-orchestration/osrm-backend/README.md b/docker-orchestration/osrm-backend/README.md index 4e2d21036..c262bfafa 100644 --- a/docker-orchestration/osrm-backend/README.md +++ b/docker-orchestration/osrm-backend/README.md @@ -1,18 +1,74 @@ # telenav osrm-backend docker +Image within built osrm binaries(`osrm-extract/osrm-partition/osrm-customize/...`) and running dependencies. It can be used to **compile data** or **startup routed**. ## Build Image +Please use below jenkins job to build docker image within osrm binaries from source code. + +- [(Telenav Internal) Jenkins Job - Build_Telenav_OSRM_Backend_Docker](https://shd-routingfp-01.telenav.cn:8443/view/OSRM/job/Build_Telenav_OSRM_Backend_Docker/) + +## Compile Data +Please use below jenkins job to compile mapdata from PBF to osrm data. +There're two options for publish the compiled osrm data: +- generate new docker image from current one within compile data +- export as `map.tar.gz` + +See below job for details: +- [(Telenav Internal) Jenkins Job - Compile_Mapdata_In_Telenav_OSRM_Backend_Docker](https://shd-routingfp-01.telenav.cn:8443/view/OSRM/job/Compile_Mapdata_In_Telenav_OSRM_Backend_Docker/) + + +## Run +### Run with osrm-data inside ```bash -$ cd docker-orchestration/osrm-backend - -# build source from default branch master-telenav -$ DOCKER_BUILDKIT=1 docker build -t telenav/osrm-backend . - -# build source from specified branch, e.g. feature/telenav-import-internal-pbf -$ DOCKER_BUILDKIT=1 docker build -t telenav/osrm-backend:telenav-import-internal-pbf --build-arg BRANCH_NAME=feature/telenav-import-internal-pbf . +$ docker pull artifactory.telenav.com/telenav-docker-preprod/osrm-backend:8__9__master-telenav__20190618T135121CST__20190514T140904CST-california-latest +$ +# run with default trafficproxy +$ docker run -d -p 5000:5000 artifactory.telenav.com/telenav-docker-preprod/osrm-backend:8__9__master-telenav__20190618T135121CST__20190514T140904CST-california-latest routed_startup +# run with another trafficproxy(e.g. 10.189.102.81) +$ docker run -d -p 5000:5000 artifactory.telenav.com/telenav-docker-preprod/osrm-backend:8__9__master-telenav__20190618T135121CST__20190514T140904CST-california-latest routed_startup 10.189.102.81 ``` -## Example +### Run with osrm-data outside + +```bash +# prepare compiled data first (sample data: california) +$ cd osrm-data +$ ll -lh +total 243M +-rw-r--r-- 1 mapuser appuser 21M Jun 17 23:37 map.osrm.cell_metrics +-rw-r--r-- 1 mapuser appuser 193K Jun 17 23:34 map.osrm.cells +-rw-r--r-- 1 mapuser appuser 1.9M Jun 17 23:34 map.osrm.cnbg +-rw-r--r-- 1 mapuser appuser 1.9M Jun 17 23:34 map.osrm.cnbg_to_ebg +-rw-r--r-- 1 mapuser appuser 68K Jun 17 23:37 map.osrm.datasource_names +-rw-r--r-- 1 mapuser appuser 9.8M Jun 17 23:34 map.osrm.ebg +-rw-r--r-- 1 mapuser appuser 2.8M Jun 17 23:34 map.osrm.ebg_nodes +-rw-r--r-- 1 mapuser appuser 2.9M Jun 17 23:34 map.osrm.edges +-rw-r--r-- 1 mapuser appuser 2.7M Jun 17 23:34 map.osrm.enw +-rwx------ 1 mapuser appuser 5.6M Jun 17 23:34 map.osrm.fileIndex +-rw-r--r-- 1 mapuser appuser 7.3M Jun 17 23:37 map.osrm.geometry +-rw-r--r-- 1 mapuser appuser 1.1M Jun 17 23:34 map.osrm.icd +-rw-r--r-- 1 mapuser appuser 5.0K Jun 17 23:34 map.osrm.maneuver_overrides +-rw-r--r-- 1 mapuser appuser 11M Jun 17 23:37 map.osrm.mldgr +-rw-r--r-- 1 mapuser appuser 218K Jun 17 23:34 map.osrm.names +-rw-r--r-- 1 mapuser appuser 4.0M Jun 17 23:34 map.osrm.nbg_nodes +-rw-r--r-- 1 mapuser appuser 1.8M Jun 17 23:34 map.osrm.partition +-rw-r--r-- 1 mapuser appuser 6.0K Jun 17 23:34 map.osrm.properties +-rw-r--r-- 1 mapuser appuser 29K Jun 17 23:34 map.osrm.ramIndex +-rw-r--r-- 1 mapuser appuser 4.0K Jun 17 23:34 map.osrm.restrictions +-rw-r--r-- 1 mapuser appuser 3.5K Jun 17 23:34 map.osrm.timestamp +-rw-r--r-- 1 mapuser appuser 5.5K Jun 17 23:34 map.osrm.tld +-rw-r--r-- 1 mapuser appuser 8.0K Jun 17 23:34 map.osrm.tls +-rw-r--r-- 1 mapuser appuser 836K Jun 17 23:34 map.osrm.turn_duration_penalties +-rw-r--r-- 1 mapuser appuser 4.9M Jun 17 23:34 map.osrm.turn_penalties_index +-rw-r--r-- 1 mapuser appuser 836K Jun 17 23:34 map.osrm.turn_weight_penalties +$ cd .. + +# pull & run +$ docker pull wangyoucao577/osrm-backend:9__master-telenav__20190618T135121CST +$ docker run -d -p 5000:5000 "src=$(pwd)/osrm-data,dst=/osrm-data,type=bind" wangyoucao577/osrm-backend:9__master-telenav__20190618T135121CST routed_startup +``` + +## Example By Manual - [Build Berlin Server with OSM data](./example-berlin-osm.md) diff --git a/docker-orchestration/osrm-backend/docker-entrypoint.sh b/docker-orchestration/osrm-backend/docker-entrypoint.sh index f4c137f96..48747b029 100755 --- a/docker-orchestration/osrm-backend/docker-entrypoint.sh +++ b/docker-orchestration/osrm-backend/docker-entrypoint.sh @@ -1,5 +1,8 @@ -#!/bin/bash +#!/bin/bash -x +BUILD_PATH=${BUILD_PATH:="/osrm-build"} DATA_PATH=${DATA_PATH:="/osrm-data"} +OSRM_EXTRA_COMMAND="-l --verbosity" +MAPDATA_NAME_WITH_SUFFIX=map _sig() { kill -TERM $child 2>/dev/null @@ -7,19 +10,49 @@ _sig() { if [ "$1" = 'routed_startup' ]; then trap _sig SIGKILL SIGTERM SIGHUP SIGINT EXIT - ./osrm-routed $DATA_PATH/$2.osrm -a MLD --max-table-size 8000 & + + TRAFFIC_FILE=traffic.csv + TRAFFIC_PROXY_IP=${2:-"10.189.102.81"} + + cd ${DATA_PATH} + ${BUILD_PATH}/osrm_traffic_updater -c ${TRAFFIC_PROXY_IP} -d=false -f ${TRAFFIC_FILE} + ls -lh + ${BUILD_PATH}/osrm-customize ${MAPDATA_NAME_WITH_SUFFIX}.osrm --segment-speed-file ${TRAFFIC_FILE} ${OSRM_EXTRA_COMMAND} + ${BUILD_PATH}/osrm-routed ${MAPDATA_NAME_WITH_SUFFIX}.osrm -a MLD --max-table-size 8000 & child=$! wait "$child" + elif [ "$1" = 'compile_mapdata' ]; then trap _sig SIGKILL SIGTERM SIGHUP SIGINT EXIT - if [ ! -f $DATA_PATH/$2.osrm ]; then - if [ ! -f $DATA_PATH/$2.osm.pbf ]; then - curl $3 > $DATA_PATH/$2.osm.pbf - fi - ./osrm-extract $DATA_PATH/$2.osm.pbf -p profiles/car.lua - ./osrm-partition $DATA_PATH/$2.osrm - ./osrm-customize $DATA_PATH/$2.osrm - fi + + PBF_FILE_URL=${2} + KEEP_COMPILED_DATA=${3:-"false"} + GENERATE_DATA_PACKAGE=${4:-"false"} + + curl ${PBF_FILE_URL} > $DATA_PATH/${MAPDATA_NAME_WITH_SUFFIX}.osm.pbf + ${BUILD_PATH}/osrm-extract $DATA_PATH/${MAPDATA_NAME_WITH_SUFFIX}.osm.pbf -p ${BUILD_PATH}/profiles/car.lua ${OSRM_EXTRA_COMMAND} + ${BUILD_PATH}/osrm-partition $DATA_PATH/${MAPDATA_NAME_WITH_SUFFIX}.osrm ${OSRM_EXTRA_COMMAND} + ${BUILD_PATH}/osrm-customize $DATA_PATH/${MAPDATA_NAME_WITH_SUFFIX}.osrm ${OSRM_EXTRA_COMMAND} + + # clean source pbf and temp .osrm + rm -f $DATA_PATH/${MAPDATA_NAME_WITH_SUFFIX}.osm.pbf + rm -f $DATA_PATH/${MAPDATA_NAME_WITH_SUFFIX}.osrm + + # package and publish compiled mapdata + if [ ${GENERATE_DATA_PACKAGE} == "true" ]; then + cd ${DATA_PATH} + tar -zcf ${MAPDATA_NAME_WITH_SUFFIX}.tar.gz * + + SAVE_DATA_PACKAGE_PATH=/save-data + mkdir -p ${SAVE_DATA_PACKAGE_PATH} + mv ${DATA_PATH}/${MAPDATA_NAME_WITH_SUFFIX}.tar.gz ${SAVE_DATA_PACKAGE_PATH}/ + fi + + # rm compiled data if not needed + if [ ${KEEP_COMPILED_DATA} != "true" ]; then + rm -f $DATA_PATH/* + fi + else exec "$@" fi diff --git a/docker-orchestration/osrm-backend/example-berlin-osm.md b/docker-orchestration/osrm-backend/example-berlin-osm.md index d1d402572..49c8bca52 100644 --- a/docker-orchestration/osrm-backend/example-berlin-osm.md +++ b/docker-orchestration/osrm-backend/example-berlin-osm.md @@ -2,21 +2,28 @@ - Generate docker image ```bash -DOCKER_BUILDKIT=1 docker build --no-cache -t telenav/osrm-backend:docker-orchestration-perry --build-arg BRANCH_NAME=feature/docker-orchestration-perry . +$ DOCKER_BUILDKIT=1 docker build --no-cache -t telenav/osrm-backend:compile-data-inside-docker --build-arg BRANCH_NAME=feature/compile-data-inside-docker . ``` - Generate OSRM data ```bash - docker run -d -v /Users/ngxuser/osrm-data/berlin_osm:/osrm-data --name osrm-data telenav/osrm-backend:docker-orchestration-perry compile_mapdata berlin "https://download.geofabrik.de/europe/germany/berlin-latest.osm.pbf" +$ mkdir -p compiled-data +$ docker run -it --mount "src=$(pwd)/compiled-data,dst=/save-data,type=bind" telenav/osrm-backend:compile-data-inside-docker compile_mapdata "https://download.geofabrik.de/europe/germany/berlin-latest.osm.pbf" false true +$ ll compiled-data/ +total 34968 +-rw-r--r-- 1 root root 35805710 Jun 17 23:34 map.tar.gz ``` - Start OSRM server ```bash -docker run -d -p 5000:5000 -v /Users/ngxuser/osrm-data/berlin_osm:/osrm-data --name osrm-api telenav/osrm-backend:docker-orchestration-perry routed_startup berlin +$ cd compiled-data +$ tar -zxf map.tar.gz +$ docker run -d -p 5000:5000 --mount "src=$(pwd),dst=/osrm-data,type=bind" --name osrm-api telenav/osrm-backend:compile-data-inside-docker routed_startup +05227a108a66e7c59f7515f8f174a65f9932f36fe3807f83991806c0194a7e50 ``` You should see such logs in docker ``` -# docker logs -f container_id +$ docker logs 05227a108a66e7c59f7515f8f174a65f9932f36fe3807f83991806c0194a7e50 [info] starting up engines, v5.22.0 [info] Threads: 10 [info] IP address: 0.0.0.0 diff --git a/docs/design/osrm-with-telenav-traffic.md b/docs/design/osrm-with-telenav-traffic.md index fb0f6a8bc..ec7b9a1e3 100644 --- a/docs/design/osrm-with-telenav-traffic.md +++ b/docs/design/osrm-with-telenav-traffic.md @@ -3,18 +3,15 @@ ## Architecture ![osrm-with-telenav-traffic-architecture](./graph/osrm-with-telenav-traffic-architecture.mmd.png) -### OSRMTrafficUpdater(need to implement) +### OSRMTrafficUpdater - as client - connect `TrafficProxy` by `RPC` - convert contents from `RPC` protocol to `OSRM` required `csv` format, then write to file -### TrafficProxy(need to implement) +### TrafficProxy - as server - provide traffic contents by region - contents include at least `from node, to node, speed`(both `from node` and `to node` are come from original mapdata) -- known issues/questions: - - can not compile `OSM` mapdata to traffic graph? - ## OSRM with Traffic Startup Flow ![osrm-with-traffic-startup-flow-chart](./graph/osrm-with-traffic-startup-flow-chart.mmd.png)