Improve compile data in docker and publish data (#21)
* chore: misc improve for telenav/osrm-backend docker * chore: improve entrypoint for easy integrated by CI * chore: fix profiles not found issue * chore: also support package and publish compiled data by .tar.gz * docs: update example * docs: update for compile data automation * docs: update item names * docs: add docker run * docs: update design doc
This commit is contained in:
Jay
Xun(Perry) Liu
parent
79d565b2ec
commit
b1446e4add
@ -6,7 +6,7 @@ Base image for telenav osrm-backend development, include all building and runnin
|
||||
See details in [osrm-backend-dev docker](./osrm-backend-dev/).
|
||||
|
||||
### osrm-backend
|
||||
Image within built osrm binaries(`osrm-extract/osrm-partition/osrm-customize/...`) and running dependencies.
|
||||
Image within built osrm binaries(`osrm-extract/osrm-partition/osrm-customize/...`) and running dependencies. It can be used to **compile data** or **startup routed**.
|
||||
See details in [osrm-backend docker](./osrm-backend/)
|
||||
|
||||
### osrm-frontend
|
||||
|
||||
@ -39,15 +39,13 @@ RUN DEBIAN_FRONTEND=noninteractive apt-get update && \
|
||||
libboost-iostreams1.62.0 libboost-thread1.62.0 expat liblua5.2-0 libtbb2 curl ca-certificates && \
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
|
||||
RUN mkdir /osrm-build
|
||||
RUN mkdir /osrm-build && mkdir /osrm-data
|
||||
|
||||
COPY --from=builder /osrm-build /osrm-build/
|
||||
COPY --from=builder /osrm-data /osrm-data/
|
||||
COPY --from=gobuilder /workspace/go/bin /osrm-build/
|
||||
|
||||
WORKDIR /osrm-build
|
||||
|
||||
# Publish
|
||||
COPY ./docker-entrypoint.sh /
|
||||
RUN chmod +x /docker-entrypoint.sh
|
||||
|
||||
EXPOSE 5000
|
||||
ENTRYPOINT ["/docker-entrypoint.sh"]
|
||||
|
||||
|
||||
@ -1,18 +1,74 @@
|
||||
# telenav osrm-backend docker
|
||||
Image within built osrm binaries(`osrm-extract/osrm-partition/osrm-customize/...`) and running dependencies. It can be used to **compile data** or **startup routed**.
|
||||
|
||||
## Build Image
|
||||
Please use below jenkins job to build docker image within osrm binaries from source code.
|
||||
|
||||
- [(Telenav Internal) Jenkins Job - Build_Telenav_OSRM_Backend_Docker](https://shd-routingfp-01.telenav.cn:8443/view/OSRM/job/Build_Telenav_OSRM_Backend_Docker/)
|
||||
|
||||
## Compile Data
|
||||
Please use below jenkins job to compile mapdata from PBF to osrm data.
|
||||
There're two options for publish the compiled osrm data:
|
||||
- generate new docker image from current one within compile data
|
||||
- export as `map.tar.gz`
|
||||
|
||||
See below job for details:
|
||||
- [(Telenav Internal) Jenkins Job - Compile_Mapdata_In_Telenav_OSRM_Backend_Docker](https://shd-routingfp-01.telenav.cn:8443/view/OSRM/job/Compile_Mapdata_In_Telenav_OSRM_Backend_Docker/)
|
||||
|
||||
|
||||
## Run
|
||||
### Run with osrm-data inside
|
||||
|
||||
```bash
|
||||
$ cd docker-orchestration/osrm-backend
|
||||
|
||||
# build source from default branch master-telenav
|
||||
$ DOCKER_BUILDKIT=1 docker build -t telenav/osrm-backend .
|
||||
|
||||
# build source from specified branch, e.g. feature/telenav-import-internal-pbf
|
||||
$ DOCKER_BUILDKIT=1 docker build -t telenav/osrm-backend:telenav-import-internal-pbf --build-arg BRANCH_NAME=feature/telenav-import-internal-pbf .
|
||||
$ docker pull artifactory.telenav.com/telenav-docker-preprod/osrm-backend:8__9__master-telenav__20190618T135121CST__20190514T140904CST-california-latest
|
||||
$
|
||||
# run with default trafficproxy
|
||||
$ docker run -d -p 5000:5000 artifactory.telenav.com/telenav-docker-preprod/osrm-backend:8__9__master-telenav__20190618T135121CST__20190514T140904CST-california-latest routed_startup
|
||||
|
||||
# run with another trafficproxy(e.g. 10.189.102.81)
|
||||
$ docker run -d -p 5000:5000 artifactory.telenav.com/telenav-docker-preprod/osrm-backend:8__9__master-telenav__20190618T135121CST__20190514T140904CST-california-latest routed_startup 10.189.102.81
|
||||
```
|
||||
|
||||
## Example
|
||||
### Run with osrm-data outside
|
||||
|
||||
```bash
|
||||
# prepare compiled data first (sample data: california)
|
||||
$ cd osrm-data
|
||||
$ ll -lh
|
||||
total 243M
|
||||
-rw-r--r-- 1 mapuser appuser 21M Jun 17 23:37 map.osrm.cell_metrics
|
||||
-rw-r--r-- 1 mapuser appuser 193K Jun 17 23:34 map.osrm.cells
|
||||
-rw-r--r-- 1 mapuser appuser 1.9M Jun 17 23:34 map.osrm.cnbg
|
||||
-rw-r--r-- 1 mapuser appuser 1.9M Jun 17 23:34 map.osrm.cnbg_to_ebg
|
||||
-rw-r--r-- 1 mapuser appuser 68K Jun 17 23:37 map.osrm.datasource_names
|
||||
-rw-r--r-- 1 mapuser appuser 9.8M Jun 17 23:34 map.osrm.ebg
|
||||
-rw-r--r-- 1 mapuser appuser 2.8M Jun 17 23:34 map.osrm.ebg_nodes
|
||||
-rw-r--r-- 1 mapuser appuser 2.9M Jun 17 23:34 map.osrm.edges
|
||||
-rw-r--r-- 1 mapuser appuser 2.7M Jun 17 23:34 map.osrm.enw
|
||||
-rwx------ 1 mapuser appuser 5.6M Jun 17 23:34 map.osrm.fileIndex
|
||||
-rw-r--r-- 1 mapuser appuser 7.3M Jun 17 23:37 map.osrm.geometry
|
||||
-rw-r--r-- 1 mapuser appuser 1.1M Jun 17 23:34 map.osrm.icd
|
||||
-rw-r--r-- 1 mapuser appuser 5.0K Jun 17 23:34 map.osrm.maneuver_overrides
|
||||
-rw-r--r-- 1 mapuser appuser 11M Jun 17 23:37 map.osrm.mldgr
|
||||
-rw-r--r-- 1 mapuser appuser 218K Jun 17 23:34 map.osrm.names
|
||||
-rw-r--r-- 1 mapuser appuser 4.0M Jun 17 23:34 map.osrm.nbg_nodes
|
||||
-rw-r--r-- 1 mapuser appuser 1.8M Jun 17 23:34 map.osrm.partition
|
||||
-rw-r--r-- 1 mapuser appuser 6.0K Jun 17 23:34 map.osrm.properties
|
||||
-rw-r--r-- 1 mapuser appuser 29K Jun 17 23:34 map.osrm.ramIndex
|
||||
-rw-r--r-- 1 mapuser appuser 4.0K Jun 17 23:34 map.osrm.restrictions
|
||||
-rw-r--r-- 1 mapuser appuser 3.5K Jun 17 23:34 map.osrm.timestamp
|
||||
-rw-r--r-- 1 mapuser appuser 5.5K Jun 17 23:34 map.osrm.tld
|
||||
-rw-r--r-- 1 mapuser appuser 8.0K Jun 17 23:34 map.osrm.tls
|
||||
-rw-r--r-- 1 mapuser appuser 836K Jun 17 23:34 map.osrm.turn_duration_penalties
|
||||
-rw-r--r-- 1 mapuser appuser 4.9M Jun 17 23:34 map.osrm.turn_penalties_index
|
||||
-rw-r--r-- 1 mapuser appuser 836K Jun 17 23:34 map.osrm.turn_weight_penalties
|
||||
$ cd ..
|
||||
|
||||
# pull & run
|
||||
$ docker pull wangyoucao577/osrm-backend:9__master-telenav__20190618T135121CST
|
||||
$ docker run -d -p 5000:5000 "src=$(pwd)/osrm-data,dst=/osrm-data,type=bind" wangyoucao577/osrm-backend:9__master-telenav__20190618T135121CST routed_startup
|
||||
```
|
||||
|
||||
## Example By Manual
|
||||
- [Build Berlin Server with OSM data](./example-berlin-osm.md)
|
||||
|
||||
|
||||
@ -1,5 +1,8 @@
|
||||
#!/bin/bash
|
||||
#!/bin/bash -x
|
||||
BUILD_PATH=${BUILD_PATH:="/osrm-build"}
|
||||
DATA_PATH=${DATA_PATH:="/osrm-data"}
|
||||
OSRM_EXTRA_COMMAND="-l --verbosity"
|
||||
MAPDATA_NAME_WITH_SUFFIX=map
|
||||
|
||||
_sig() {
|
||||
kill -TERM $child 2>/dev/null
|
||||
@ -7,19 +10,49 @@ _sig() {
|
||||
|
||||
if [ "$1" = 'routed_startup' ]; then
|
||||
trap _sig SIGKILL SIGTERM SIGHUP SIGINT EXIT
|
||||
./osrm-routed $DATA_PATH/$2.osrm -a MLD --max-table-size 8000 &
|
||||
|
||||
TRAFFIC_FILE=traffic.csv
|
||||
TRAFFIC_PROXY_IP=${2:-"10.189.102.81"}
|
||||
|
||||
cd ${DATA_PATH}
|
||||
${BUILD_PATH}/osrm_traffic_updater -c ${TRAFFIC_PROXY_IP} -d=false -f ${TRAFFIC_FILE}
|
||||
ls -lh
|
||||
${BUILD_PATH}/osrm-customize ${MAPDATA_NAME_WITH_SUFFIX}.osrm --segment-speed-file ${TRAFFIC_FILE} ${OSRM_EXTRA_COMMAND}
|
||||
${BUILD_PATH}/osrm-routed ${MAPDATA_NAME_WITH_SUFFIX}.osrm -a MLD --max-table-size 8000 &
|
||||
child=$!
|
||||
wait "$child"
|
||||
|
||||
elif [ "$1" = 'compile_mapdata' ]; then
|
||||
trap _sig SIGKILL SIGTERM SIGHUP SIGINT EXIT
|
||||
if [ ! -f $DATA_PATH/$2.osrm ]; then
|
||||
if [ ! -f $DATA_PATH/$2.osm.pbf ]; then
|
||||
curl $3 > $DATA_PATH/$2.osm.pbf
|
||||
|
||||
PBF_FILE_URL=${2}
|
||||
KEEP_COMPILED_DATA=${3:-"false"}
|
||||
GENERATE_DATA_PACKAGE=${4:-"false"}
|
||||
|
||||
curl ${PBF_FILE_URL} > $DATA_PATH/${MAPDATA_NAME_WITH_SUFFIX}.osm.pbf
|
||||
${BUILD_PATH}/osrm-extract $DATA_PATH/${MAPDATA_NAME_WITH_SUFFIX}.osm.pbf -p ${BUILD_PATH}/profiles/car.lua ${OSRM_EXTRA_COMMAND}
|
||||
${BUILD_PATH}/osrm-partition $DATA_PATH/${MAPDATA_NAME_WITH_SUFFIX}.osrm ${OSRM_EXTRA_COMMAND}
|
||||
${BUILD_PATH}/osrm-customize $DATA_PATH/${MAPDATA_NAME_WITH_SUFFIX}.osrm ${OSRM_EXTRA_COMMAND}
|
||||
|
||||
# clean source pbf and temp .osrm
|
||||
rm -f $DATA_PATH/${MAPDATA_NAME_WITH_SUFFIX}.osm.pbf
|
||||
rm -f $DATA_PATH/${MAPDATA_NAME_WITH_SUFFIX}.osrm
|
||||
|
||||
# package and publish compiled mapdata
|
||||
if [ ${GENERATE_DATA_PACKAGE} == "true" ]; then
|
||||
cd ${DATA_PATH}
|
||||
tar -zcf ${MAPDATA_NAME_WITH_SUFFIX}.tar.gz *
|
||||
|
||||
SAVE_DATA_PACKAGE_PATH=/save-data
|
||||
mkdir -p ${SAVE_DATA_PACKAGE_PATH}
|
||||
mv ${DATA_PATH}/${MAPDATA_NAME_WITH_SUFFIX}.tar.gz ${SAVE_DATA_PACKAGE_PATH}/
|
||||
fi
|
||||
./osrm-extract $DATA_PATH/$2.osm.pbf -p profiles/car.lua
|
||||
./osrm-partition $DATA_PATH/$2.osrm
|
||||
./osrm-customize $DATA_PATH/$2.osrm
|
||||
|
||||
# rm compiled data if not needed
|
||||
if [ ${KEEP_COMPILED_DATA} != "true" ]; then
|
||||
rm -f $DATA_PATH/*
|
||||
fi
|
||||
|
||||
else
|
||||
exec "$@"
|
||||
fi
|
||||
|
||||
@ -2,21 +2,28 @@
|
||||
|
||||
- Generate docker image
|
||||
```bash
|
||||
DOCKER_BUILDKIT=1 docker build --no-cache -t telenav/osrm-backend:docker-orchestration-perry --build-arg BRANCH_NAME=feature/docker-orchestration-perry .
|
||||
$ DOCKER_BUILDKIT=1 docker build --no-cache -t telenav/osrm-backend:compile-data-inside-docker --build-arg BRANCH_NAME=feature/compile-data-inside-docker .
|
||||
```
|
||||
|
||||
- Generate OSRM data
|
||||
```bash
|
||||
docker run -d -v /Users/ngxuser/osrm-data/berlin_osm:/osrm-data --name osrm-data telenav/osrm-backend:docker-orchestration-perry compile_mapdata berlin "https://download.geofabrik.de/europe/germany/berlin-latest.osm.pbf"
|
||||
$ mkdir -p compiled-data
|
||||
$ docker run -it --mount "src=$(pwd)/compiled-data,dst=/save-data,type=bind" telenav/osrm-backend:compile-data-inside-docker compile_mapdata "https://download.geofabrik.de/europe/germany/berlin-latest.osm.pbf" false true
|
||||
$ ll compiled-data/
|
||||
total 34968
|
||||
-rw-r--r-- 1 root root 35805710 Jun 17 23:34 map.tar.gz
|
||||
```
|
||||
|
||||
- Start OSRM server
|
||||
```bash
|
||||
docker run -d -p 5000:5000 -v /Users/ngxuser/osrm-data/berlin_osm:/osrm-data --name osrm-api telenav/osrm-backend:docker-orchestration-perry routed_startup berlin
|
||||
$ cd compiled-data
|
||||
$ tar -zxf map.tar.gz
|
||||
$ docker run -d -p 5000:5000 --mount "src=$(pwd),dst=/osrm-data,type=bind" --name osrm-api telenav/osrm-backend:compile-data-inside-docker routed_startup
|
||||
05227a108a66e7c59f7515f8f174a65f9932f36fe3807f83991806c0194a7e50
|
||||
```
|
||||
You should see such logs in docker
|
||||
```
|
||||
# docker logs -f container_id
|
||||
$ docker logs 05227a108a66e7c59f7515f8f174a65f9932f36fe3807f83991806c0194a7e50
|
||||
[info] starting up engines, v5.22.0
|
||||
[info] Threads: 10
|
||||
[info] IP address: 0.0.0.0
|
||||
|
||||
@ -3,18 +3,15 @@
|
||||
## Architecture
|
||||
|
||||
|
||||
### OSRMTrafficUpdater(need to implement)
|
||||
### OSRMTrafficUpdater
|
||||
- as client
|
||||
- connect `TrafficProxy` by `RPC`
|
||||
- convert contents from `RPC` protocol to `OSRM` required `csv` format, then write to file
|
||||
|
||||
### TrafficProxy(need to implement)
|
||||
### TrafficProxy
|
||||
- as server
|
||||
- provide traffic contents by region
|
||||
- contents include at least `from node, to node, speed`(both `from node` and `to node` are come from original mapdata)
|
||||
- known issues/questions:
|
||||
- can not compile `OSM` mapdata to traffic graph?
|
||||
|
||||
|
||||
## OSRM with Traffic Startup Flow
|
||||
|
||||
|
||||
Loading…
Reference in New Issue
Block a user