Open-Harbor/osrm-backend
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 3 Wiki Activity

Squashed 'third_party/libosmium/' changes from ce865381f..d2c1d872b

Browse Source 
d2c1d872b Release v2.14.0
9c9aedd50 Update embedded catch.hpp to version 1.12.1.
f94e70d32 Change comment on namespace closing brace to make clang-tidy happy.
ad9f03afc Prefer prefix increment operator.
a141b9c0c Update change log.
363fc7a15 Add more XML tests.
303a3090f Refactor XML tests.
a1ee5dd56 Make util namespace inline.
6f1709a83 Cleanup progress bar code.
1b02b65e0 Use Location setters taking a string in osmium_tiles example.
7ffdc7618 Disable clang-tidy warning.
882631085 Better implementation of str_to_int.
4cbf53725 Fix another bitwise op on signed integer.
903bfac9d Disable clang-tidy warning.
01de3f518 Don't change location if set_lon() or set_lat() functions throw.
9a27e899d Fix bitwise operation on signed integer.
61650224f Disable some clang-tidy warnings.
9dfa8437e Provide a way to better mock the getenv function in tests.
ae2c172a2 Explicitly set integer type for integers in test code.
878850068 Simpler implementation. We don't need to look at errno at all.
8f9b886f5 Formatting
cf4760cb0 Use bit operations on unsigned where possible.
c7cb4cde8 No conversion needed later if we declare it this way.
01357a540 Add missing inline for free function.
3feef81ee Disable clang-tidy warning in place we can't do anything about.
bf083b8ee Avoid binary bitwise operations on signed integers.
f70b1c803 Move content of osmium::util::detail namespace into osmium::detail.
caf2f0033 Remove use of atoi and atol from code.
f564cef07 Formatting.
f0885fc73 Disable travis build with sanitizers.
858c58444 Use explicit asserts and then casts, not static_cast_with_assert.
a613b1b0a Disable clang-tidy warning.
50e327677 Always build and run data tests on travis.
20d7a09b0 Update submodule to add more XML tests.
6c8f652bf Merge pull request #250 from Nakaner/object-comparison-without-timestamp
38ea437c0 Make XML parser more strict.
7788f5108 Remove object_order_type_id_reverse_version_without_timestamp
fe2223d25 Ignore timestamps if only one object has a valid timestamp
a4515161a Code cleanup.
eb5ea1a0e Use explicit conversions.
105905c45 Only disable warning on non-MSVC compilers.
80fc1b35d Change the way IDs are parsed from strings.
78aedf91e More tests for StringMatcher and disable a clang-tidy warning.
644a00752 Move catch.hpp into its own directory.
19842756c Update to newest osm-testdata submodule.
0402f64d7 Fix the problem with "git submodule" in appveyor.
ec4095d32 Cleanup metadata_options code.
d7390741e Add static_cast to avoid conversion warning.
dd16be1c3 Remove else after throw.
ef4d618de Remove unused variable.
0fdb58d13 Check out submodule in appveyor builds.
6406282d3 Merge pull request #248 from Nakaner/metadata-options-more
eb138a902 make clang < 3.9 work
e703b0d1d make sections their own independent test cases
f68888006 detect_available_metadata must be inline
b53224588 move metadata_options into osmium namespace
edd363cfa make const what could be const
302a485e5 remove unnecessary include, add missing include
93a76e07e give osmium::io::metadata_from_object a better name
91dfbc427 update documentation
453cd6395 replace REQUIRE(!foo) by REQUIRE_FALSE(foo)
247c3b021 replace friend function by setter methods
0a62e589c performance improvement: check first character instead using std::strlen()
73568bbff move ctor of metadata_options which taks an OSMObject out of class
1879c9e47 restore metadata_options::to_string
71a303608 add more unit tests for osmium::io::metadata_options
843cdc89c replace osmium::io::metdata_options::to_string fully by operator<<
6402a8793 Fix bug in code which determines if an OSM object has a valid user.
a6d8143ae Move metadata_options out of "detail" subnamespace
61e32df10 add metadata_options::operator|=
d5f7cc0ab Use newest osm-testdata submodule.
31e34f10c add operator&=, operator<< and a to_string method to metadata_options
77d507172 Formatting fixes.
1c38b2a61 Update some URLs. Use https where possible.
10066cb3e Merge pull request #247 from Nakaner/issue-245
3c938ce82 Allow version=-1 and changeset=-1 for PBF input.
9a542c133 Add ReaderWithProgressBar class.
49839a1ce Replace strftime by our own implementation.
b32b90942 Update osm-testdata version used.
f88b6e548 Add more tests.
cb7e778f6 Handle NULL geometries in test cases.
1961a9c95 Clean up some test cases.
d7f45e071 Update osm-testdata submodule.
73a2ef18f Remove check for lost ways in multipolygon assembler.
1244e498a Update submodule also in msys2 build.
229fc2453 Revert "Make util namespace inline."
9f942d2e9 Revert "Add inline declaration to one util namespace we forgot."
40d4835ea Really fix appveyor build (I hope).
5ae071135 Add inline declaration to one util namespace we forgot.
e0916257c Clean up debug message.
2ddc474cc Make util namespace inline.
4b4e25f87 Travis config: Do not install clang-tidy, we are not using it.
5a608fba1 Fix travis build.
a075cc323 Fix appveyor build.
41ee9920b Add submodule with osm-testdata repository.
4d42245c0 Revert "Add clang-tidy build to travis."
9df355b64 Fix issue on some compilers.
c8a3c00d6 Add clang-tidy build to travis.
2fc80ad17 Fixed clang-tidy config file order.
2adf05267 Fix various clang-tidy warnings
1dca92459 Use the right syntax to override clang-tidy settings in the code.
25baecb46 Fix some issues found by clang-tidy.
54a8de6c0 Disable more clang-tidy tests.
de46d562e Also look for clang-tidy-6.0 in CMake config.
2172553d8 Add build with sanitizers on travis.
7e4e45562 Add object comparators ignoring the timestamp.
92fb231a1 Fix test code.
e541df0fc Cleanup test code.
5640269da Add comment in appveyor script about unusual parameter.
acb872376 Workaround for failing appveyor MSYS2 build.
0634bcf70 Set default value for m_file_size member.
3a2fe5781 Set default values on output options.
43d289fda Make metadata_options constructor explicit.
6e54992b4 Update change log.
e54b955cf Update included `catch.hpp` to version 1.12.0.
e50b40d53 Update copyright date.
0d8b401b6 Merge branch 'metadata'
c7da3a3b6 Remove superfluous has_visibles boolean.
7f74d33d3 Some cleanups of metadata code and fixes test.
5b3c34990 Output XML attributes uid and user independently of each other.
384167334 Merge pull request #243 from Nakaner/metadata_other
733f1916c Change path of nuget installs in appveyor config.
e48672999 Debug output from FindOsmium.cmake script is better formatted now.
79663be5d Change appveyor scripts to work without deprecated prebuilt libs.
443f7a9e4 Remove dependency on (win)getopt completely.
c133b8b14 Fix osmium_convert: Long options with = syntax.
b55daf00a Remove dependency on (win)getopt from area_test and convert examples.
92f7275a9 output visibility directly after the object ID in debug format
a771279f0 deprecate format option pbf_add_metadata
18e3f6005 The value "no" should be treated as "none" by metadata_options
c8b0b2e28 re-assign metadata_options instead of calling a public method of it
15bc4a200 Add more flexible metadata handling to PBF output
b085632b7 add more flexible metadata handling to OPL and debug output
a12c65974 Remove now wrong check.
767ad883a Allow PBF DenseNodes with only some attributes to be read.
fd5a7eb6c More flexible metadata encoding in OSM files.
d16c18581 Add instructions for bug reporting.
db52591ab Use __COUNTER__ instead of name for internal dummy name.
89e634dfc Cleanup.
931ec30ca Marks some false positives for clang-tidy google-runtime-int warning.
b59ef02a2 More rule of zero/rule of five fixes.
1b72b3386 Pass by value and use std::move.
b7d7017cc Handle some clang-tidy google-runtime-int warnings.
c043ca016 Fix special functions.
41d9f0795 Use fixed sized ints where possible.
0899a66a7 Revert some changes older compilers don't like.
41e05989a Fix several rule of zero/rule of five issues.
4e78a828f Various fixes based on clang-tidy reports.
636bda372 Default initialize arrays.
466e5aa6a Handle various clang-tidy warnings.
9f47aa07d Disable clang-tidy misc-forwarding-reference-overload warning.
5e97bf29f Mark Reader and Writer non-movable.
3a2221b27 Add minimal tests for thread-safe queue class.
affdeed6f Add some tests.
5ed055d50 Handle some warnings regarding "explicit".
b9d4c9bf5 Disable clang-tidy cert-err58-cpp warning.
254c25349 Make ADL work for begin()/end() of InputIterator<Reader>.
b64cb32ae Disable clang-tidy android-cloexec-* warnings.
6d6934ffc Fix various warnings reported by clang-tidy.
30dcabaed Another travis build fix.
88c59e669 Fix travis build.
2facb461f Fix travis and appveyor builds.
bba631a51 Remove protozero from repository.
aebe4a914 Use pass by value and std::move in some constructors.
c8c24c8fd Name unused parameters in comments.
9c2e321f9 Make deleted special functions public.
265c16284 Fix some problems with older compilers.
174e95211 Modernize class initialization code for some classes.
4e3e584f8 Disable more clang-tidy warnings.
6b53981bb Fix some class initializations.
fe1405ef8 Add/remove special functions for some classes.
c6dd930d9 Disable cppcoreguidelines-pro-type-static-cast-downcast warning.
0d09210ec Disable misc-unused-parameters warning in clang-tidy.
1dbc43112 Refactor to avoid misc-suspicious-string-compare clang-tidy warning.
91262cc86 Merge pull request #232 from worace/wkt-geojson-polygons
438076b09 Cleanup: Avoid else after return/break.
e0368e92c Disable modernize-raw-string-literal and hicpp-no-assembler check.
c822888fc Mark C style cast in macro from a library as NOLINT.
662d4a641 Moving a TypedMemoryMapping is noexcept.
aec1c4526 Use auto (clang-tidy check modernize-use-auto).
64857589d Consistent include order.
d1ce8de00 Disable hicpp-invalid-access-moved clang-tidy check.
0d3902420 Refactor test without SECTION.
32e9e9e1e Rewrite test avoiding nested sections.
eeb532902 Reorder includes according to best practice.
249e7b553 Revert "Add build with sanitizers to travis."
362cb419a Set CC env var in travis config.
9f520eca3 Add build with sanitizers to travis.
cfc18fef0 Travis config cleanup.
212c6cd42 Remove tests that triggers undefined behaviour in the std lib.
f30985884 Bugfix: Avoid undefined behaviour.
90191619a Make test work even if unused memory isn't returned to OS.
876867e68 Make older compilers happy.
925d1a797 Add names for unused parameters.
014b3a056 Avoid repeating the return type from the declaration
c4e705f92 Disable hicpp-invalid-access-moved clang-tidy warnings.
8c03b3827 Disable some warnings.
c49962ffe Refactor test to avoid spurious use-after-move warning.
04a0afbba Sort includes.
2aaab9a89 Use auto where possible.
e2d84109b More changes for old compilers.
5364ca2a8 Changes for old compilers.
098fc1fa2 Move initializations of class members to the declaration.
eea463026 Fix buffer overflow in o5m parser.
5dc74f2d5 Remove check from static cast (and document why).
7a2dfdcdc Add NOLINT for using namespace osmium::builder::attr.
01fa05f1e Clang-tidy config cleanup.
5e8f8df3f Order includes.
6b78d9b24 Various small fixes for problems detected by clang-tidy.
b815f71b7 Fix gdal check in CMake config.
d47487093 Fix integer type error.
dec9de9ec Bugfix: Throw if elemens in XML file are nested too deep.
85e37723b Various small cleanups.
5f40d726a Additional checks in o5m parser.
b59b4b6a6 Check before static cast and throw.
e57b15700 Disable compiling some header files if gdal lib is not available.
1d3056646 Better checking that PBF data is in range.
c586521d3 Add some consts to the code.
df17e8189 Trying to get the types right...
8e9b759e0 Check read and write system call for EINTR.
28c676054 Rename license file so that Github finds it.
be29e9a52 Remove duplicate space character.
36c14136f Update protozero to newest version (1.6.0).
a19181998 Make libosmium work with different protozero versions.
5c5d9fb36 Use tag and type from protozero to make PBF parser more robust.
03f894400 Add repology.org badge to README.md.
1840739dd Merge pull request #234 from AMDmi3/patch-1
d8e737196 Add support for DragonFly BSD
8de181c8d Revert "Return const object from postfix ++ operator."
0dfb1d649 Add polygon implementation for WKT and GeoJSON Geometry Factories
150e05aea Fix indentation.
6fdee974a Merge pull request #229 from alex85k/no-winsock
f302f08ce Merge pull request #228 from alex85k/fix-test-win
bfb60c888 remove ws2_32 from needed libraries
ef187851c no winsock2 on windows: replace htonl with protozero::detail::byteswap_inplace
9343f8453 Fix testdata-multipolygon test on Windows (executable path differs)
381cf9272 Do not return const from postfix increment operator.
46ec985e9 Add const version of OutputIterator::operator*.
c67de374f Disable clang-tidy test for const params.
f5eda165f Make functions noexcept and return const value from post increment op.
1e4e3160f Use {} when calling constructors. Use .empty() to check empty strings.
8893d16be Mark some tests using size() with NOLINT.
f8a74a8aa Make functions noexcept.
45c1e072c Use REQUIRE_FALSE(...) instead of REQUIRE(!...) for readability.
fc173dae7 Merge pull request #227 from alex85k/msys2-appveor
0a281c064 Add MSYS2 building to Appveyor
c93f73120 fix FindGem for Windows Ruby distributions
8bf6702b3 Merge pull request #226 from alex85k/master
e8dad731d gmtime_s should be used on both MSVC and GCC from MinGW64
d0d98e9c1 remove "osmium::util::MemoryMapping::" from class body
6830ee9bd ftruncate is not available only on MSVC (MinGW has it)
c0597584f Disable some clang-tidy checks.
578db8e4a Use empty() instead of size() or == "" to check for empty collections.
53c8f3a21 Return const object from postfix ++ operator.
bd5e4d109 Use emplace_back() instead of push_back().
0a0a4f24c Use correct C++ header.
35226ea86 Remove unused move.
df843a9cf Fix header guard.
759e76131 Avoid default argument on virtual function.
db4f1bdb9 Update included catch.hpp to version 1.10.0.
59e70e5c6 Pass constructor arguments by value and std::move() into members.
ee9b598ed Add tests for thread_handler.
ac661209a CMake config: Update list of clang-tidy versions we look for.
d7281215c Add comment to unusual code.
cbb4c09fb Split up some overlong test functions.
9870a6b90 Fix noexcept specification.
5e27c37d0 Use array instead of map to store input/output format creators.
987d12922 Mark some constructors noexcept.
17f4d8b7e Avoid static variable only needed as constant value.
fe705cef0 Avoid else after return in several cases.
d4a85878e Beautifying travis config.
f07d1e375 Make some special functions noexcept.
4bb0f7f22 Do not declare special functions on some classes.
bc6c5943d Disable some clang-tidy checks that produce false positives.
6b6598a5b Remove superfluous std::move() of POD type.
71c3f5e16 Make some special functions noexcept.
1cb32778c Remove explicit true/false in tests.
929ba89cf Always catch by const ref.
a9a255820 Code readability: No else clause after return.
f14364bd3 Code formatting fixes.
f41feb6a5 Simplify code.
727f6eff4 Mark dummy includes for IWYU.
72d20f835 Switch travis build to 'trusty'.
c480e6808 Do not run gcov from codecov script.
d16c6e90a Clean up lambda captures in xml_input_format.hpp.

git-subtree-dir: third_party/libosmium
git-subtree-split: d2c1d872bc915d9579350007383ce1ebfea3665f
This commit is contained in:
Michael Krasnyk
2018-04-19 22:03:25 +03:00
parent 6eb4f090f9
commit a3a7a822e1
353 changed files with 6183 additions and 7253 deletions
Download Patch File Download Diff File Expand all files Collapse all files
include/osmium/io/detail/debug_output_format.hpp
+45 -45
View File
@@ -5,7 +5,7 @@
This file is part of Osmium (http://osmcode.org/libosmium).
Copyright 2013-2017 Jochen Topf <jochen@topf.org> and others (see README).
Copyright 2013-2018 Jochen Topf <jochen@topf.org> and others (see README).
Boost Software License - Version 1.0 - August 17th, 2003
@@ -33,16 +33,6 @@ DEALINGS IN THE SOFTWARE.
*/
#include <cinttypes>
#include <cmath>
#include <cstring>
#include <iterator>
#include <memory>
#include <string>
#include <utility>
#include <boost/crc.hpp>
#include <osmium/io/detail/output_format.hpp>
#include <osmium/io/detail/queue_util.hpp>
#include <osmium/io/detail/string_util.hpp>
@@ -56,6 +46,7 @@ DEALINGS IN THE SOFTWARE.
#include <osmium/osm/crc.hpp>
#include <osmium/osm/item_type.hpp>
#include <osmium/osm/location.hpp>
#include <osmium/osm/metadata_options.hpp>
#include <osmium/osm/node.hpp>
#include <osmium/osm/node_ref.hpp>
#include <osmium/osm/object.hpp>
@@ -68,6 +59,16 @@ DEALINGS IN THE SOFTWARE.
#include <osmium/util/minmax.hpp>
#include <osmium/visitor.hpp>
#include <boost/crc.hpp>
#include <cinttypes>
#include <cmath>
#include <cstring>
#include <iterator>
#include <memory>
#include <string>
#include <utility>
namespace osmium {
namespace io {
@@ -92,18 +93,19 @@ namespace osmium {
struct debug_output_options {
/// Should metadata of objects be added?
bool add_metadata;
/// Which metadata of objects should be added?
osmium::metadata_options add_metadata;
/// Output with ANSI colors?
bool use_color;
bool use_color = false;
/// Add CRC32 checksum to each object?
bool add_crc32;
bool add_crc32 = false;
/// Write in form of a diff file?
bool format_as_diff;
};
bool format_as_diff = false;
}; // struct debug_output_options
/**
* Writes out one buffer with OSM data in Debug format.
@@ -144,7 +146,8 @@ namespace osmium {
*m_out += '-';
*m_out += color_reset;
return;
} else if (m_diff_char == '+') {
}
if (m_diff_char == '+') {
*m_out += color_backg_green;
*m_out += color_white;
*m_out += color_bold;
@@ -218,26 +221,36 @@ namespace osmium {
void write_meta(const osmium::OSMObject& object) {
output_int(object.id());
*m_out += '\n';
if (m_options.add_metadata) {
if (object.visible()) {
*m_out += " visible\n";
} else {
write_error(" deleted\n");
}
if (m_options.add_metadata.version()) {
write_fieldname("version");
*m_out += " ";
output_int(object.version());
if (object.visible()) {
*m_out += " visible\n";
} else {
write_error(" deleted\n");
}
*m_out += '\n';
}
if (m_options.add_metadata.changeset()) {
write_fieldname("changeset");
output_int(object.changeset());
*m_out += '\n';
}
if (m_options.add_metadata.timestamp()) {
write_fieldname("timestamp");
write_timestamp(object.timestamp());
}
if (m_options.add_metadata.user() || m_options.add_metadata.uid()) {
write_fieldname("user");
*m_out += " ";
output_int(object.uid());
*m_out += ' ';
write_string(object.user());
if (m_options.add_metadata.uid()) {
output_int(object.uid());
*m_out += ' ';
}
if (m_options.add_metadata.user()) {
write_string(object.user());
}
*m_out += '\n';
}
}
@@ -261,8 +274,9 @@ namespace osmium {
*m_out += " ";
write_string(tag.key());
auto spacing = max() - std::strlen(tag.key());
while (spacing--) {
while (spacing > 0) {
*m_out += " ";
--spacing;
}
*m_out += " = ";
write_string(tag.value());
@@ -322,14 +336,6 @@ namespace osmium {
m_utf8_suffix(options.use_color ? color_blue : "") {
}
DebugOutputBlock(const DebugOutputBlock&) = default;
DebugOutputBlock& operator=(const DebugOutputBlock&) = default;
DebugOutputBlock(DebugOutputBlock&&) = default;
DebugOutputBlock& operator=(DebugOutputBlock&&) = default;
~DebugOutputBlock() noexcept = default;
std::string operator()() {
osmium::apply(m_input_buffer->cbegin(), m_input_buffer->cend(), *this);
@@ -522,19 +528,13 @@ namespace osmium {
public:
DebugOutputFormat(osmium::thread::Pool& pool, const osmium::io::File& file, future_string_queue_type& output_queue) :
OutputFormat(pool, output_queue),
m_options() {
m_options.add_metadata = file.is_not_false("add_metadata");
OutputFormat(pool, output_queue) {
m_options.add_metadata = osmium::metadata_options{file.get("add_metadata")};
m_options.use_color = file.is_true("color");
m_options.add_crc32 = file.is_true("add_crc32");
m_options.format_as_diff = file.is_true("diff");
}
DebugOutputFormat(const DebugOutputFormat&) = delete;
DebugOutputFormat& operator=(const DebugOutputFormat&) = delete;
~DebugOutputFormat() noexcept final = default;
void write_header(const osmium::io::Header& header) final {
if (m_options.format_as_diff) {
return;
include/osmium/io/detail/input_format.hpp
+30 -26
View File
@@ -5,7 +5,7 @@
This file is part of Osmium (http://osmcode.org/libosmium).
Copyright 2013-2017 Jochen Topf <jochen@topf.org> and others (see README).
Copyright 2013-2018 Jochen Topf <jochen@topf.org> and others (see README).
Boost Software License - Version 1.0 - August 17th, 2003
@@ -33,14 +33,6 @@ DEALINGS IN THE SOFTWARE.
*/
#include <exception>
#include <functional>
#include <future>
#include <map>
#include <memory>
#include <string>
#include <utility>
#include <osmium/io/detail/queue_util.hpp>
#include <osmium/io/error.hpp>
#include <osmium/io/file.hpp>
@@ -50,6 +42,14 @@ DEALINGS IN THE SOFTWARE.
#include <osmium/osm/entity_bits.hpp>
#include <osmium/thread/pool.hpp>
#include <array>
#include <exception>
#include <functional>
#include <future>
#include <memory>
#include <string>
#include <utility>
namespace osmium {
namespace io {
@@ -177,37 +177,41 @@ namespace osmium {
private:
using map_type = std::map<osmium::io::file_format, create_parser_type>;
std::array<create_parser_type, static_cast<std::size_t>(file_format::last) + 1> m_callbacks;
map_type m_callbacks;
ParserFactory() noexcept = default;
ParserFactory() :
m_callbacks() {
create_parser_type& callbacks(const osmium::io::file_format format) noexcept {
return m_callbacks[static_cast<std::size_t>(format)];
}
const create_parser_type& callbacks(const osmium::io::file_format format) const noexcept {
return m_callbacks[static_cast<std::size_t>(format)];
}
public:
static ParserFactory& instance() {
static ParserFactory& instance() noexcept {
static ParserFactory factory;
return factory;
}
bool register_parser(osmium::io::file_format format, create_parser_type&& create_function) {
const auto result = m_callbacks.emplace(format, std::forward<create_parser_type>(create_function));
return result.second;
bool register_parser(const osmium::io::file_format format, create_parser_type&& create_function) {
callbacks(format) = std::forward<create_parser_type>(create_function);
return true;
}
create_parser_type get_creator_function(const osmium::io::File& file) const {
const auto it = m_callbacks.find(file.format());
if (it == m_callbacks.end()) {
throw unsupported_file_format_error{
std::string{"Can not open file '"} +
file.filename() +
"' with type '" +
as_string(file.format()) +
"'. No support for reading this format in this program."};
const auto func = callbacks(file.format());
if (func) {
return func;
}
return it->second;
throw unsupported_file_format_error{
std::string{"Can not open file '"} +
file.filename() +
"' with type '" +
as_string(file.format()) +
"'. No support for reading this format in this program."};
}
}; // class ParserFactory
include/osmium/io/detail/o5m_input_format.hpp
+69 -53
View File
@@ -5,7 +5,7 @@
This file is part of Osmium (http://osmcode.org/libosmium).
Copyright 2013-2017 Jochen Topf <jochen@topf.org> and others (see README).
Copyright 2013-2018 Jochen Topf <jochen@topf.org> and others (see README).
Boost Software License - Version 1.0 - August 17th, 2003
@@ -33,18 +33,6 @@ DEALINGS IN THE SOFTWARE.
*/
#include <algorithm>
#include <cstddef>
#include <cstdint>
#include <cstring>
#include <future>
#include <memory>
#include <string>
#include <utility>
#include <protozero/exception.hpp>
#include <protozero/varint.hpp>
#include <osmium/builder/osm_object_builder.hpp>
#include <osmium/io/detail/input_format.hpp>
#include <osmium/io/detail/queue_util.hpp>
@@ -63,9 +51,21 @@ DEALINGS IN THE SOFTWARE.
#include <osmium/osm/types.hpp>
#include <osmium/osm/way.hpp>
#include <osmium/thread/util.hpp>
#include <osmium/util/cast.hpp>
#include <osmium/util/delta.hpp>
#include <protozero/exception.hpp>
#include <protozero/varint.hpp>
#include <algorithm>
#include <cstddef>
#include <cstdint>
#include <cstring>
#include <future>
#include <limits>
#include <memory>
#include <string>
#include <utility>
namespace osmium {
namespace builder {
@@ -90,7 +90,7 @@ namespace osmium {
namespace detail {
// Implementation of the o5m/o5c file formats according to the
// description at http://wiki.openstreetmap.org/wiki/O5m .
// description at https://wiki.openstreetmap.org/wiki/O5m .
class ReferenceTable {
@@ -137,7 +137,7 @@ namespace osmium {
if (m_table.empty() || index == 0 || index > number_of_entries) {
throw o5m_error{"reference to non-existing string in table"};
}
auto entry = (current_entry + number_of_entries - index) % number_of_entries;
const auto entry = (current_entry + number_of_entries - index) % number_of_entries;
return &m_table[entry * entry_size];
}
@@ -145,13 +145,13 @@ namespace osmium {
class O5mParser : public Parser {
static constexpr int buffer_size = 2 * 1000 * 1000;
static constexpr std::size_t buffer_size = 2 * 1000 * 1000;
osmium::io::Header m_header;
osmium::io::Header m_header{};
osmium::memory::Buffer m_buffer;
std::string m_input;
std::string m_input{};
const char* m_data;
const char* m_end;
@@ -163,7 +163,7 @@ namespace osmium {
}
bool ensure_bytes_available(std::size_t need_bytes) {
if ((m_end - m_data) >= long(need_bytes)) {
if ((m_end - m_data) >= static_cast<int64_t>(need_bytes)) {
return true;
}
@@ -190,7 +190,7 @@ namespace osmium {
void check_header_magic() {
static const unsigned char header_magic[] = { 0xff, 0xe0, 0x04, 'o', '5' };
if (std::strncmp(reinterpret_cast<const char*>(header_magic), m_data, sizeof(header_magic))) {
if (std::strncmp(reinterpret_cast<const char*>(header_magic), m_data, sizeof(header_magic)) != 0) {
throw o5m_error{"wrong header magic"};
}
@@ -231,15 +231,15 @@ namespace osmium {
set_header_value(m_header);
}
osmium::util::DeltaDecode<osmium::object_id_type> m_delta_id;
osmium::DeltaDecode<osmium::object_id_type> m_delta_id;
osmium::util::DeltaDecode<int64_t> m_delta_timestamp;
osmium::util::DeltaDecode<osmium::changeset_id_type> m_delta_changeset;
osmium::util::DeltaDecode<int64_t> m_delta_lon;
osmium::util::DeltaDecode<int64_t> m_delta_lat;
osmium::DeltaDecode<int64_t> m_delta_timestamp;
osmium::DeltaDecode<osmium::changeset_id_type> m_delta_changeset;
osmium::DeltaDecode<int64_t> m_delta_lon;
osmium::DeltaDecode<int64_t> m_delta_lat;
osmium::util::DeltaDecode<osmium::object_id_type> m_delta_way_node_id;
osmium::util::DeltaDecode<osmium::object_id_type> m_delta_member_ids[3];
osmium::DeltaDecode<osmium::object_id_type> m_delta_way_node_id;
osmium::DeltaDecode<osmium::object_id_type> m_delta_member_ids[3];
void reset() {
m_reference_table.clear();
@@ -263,18 +263,21 @@ namespace osmium {
throw o5m_error{"string format error"};
}
return *dataptr;
} else { // get from reference table
auto index = protozero::decode_varint(dataptr, end);
return m_reference_table.get(index);
}
// get from reference table
const auto index = protozero::decode_varint(dataptr, end);
return m_reference_table.get(index);
}
std::pair<osmium::user_id_type, const char*> decode_user(const char** dataptr, const char* const end) {
bool update_pointer = (**dataptr == 0x00);
const bool update_pointer = (**dataptr == 0x00);
const char* data = decode_string(dataptr, end);
const char* start = data;
auto uid = protozero::decode_varint(&data, end);
const auto uid = protozero::decode_varint(&data, end);
if (uid > std::numeric_limits<user_id_type>::max()) {
throw o5m_error{"uid out of range"};
}
if (data == end) {
throw o5m_error{"missing user name"};
@@ -285,7 +288,7 @@ namespace osmium {
if (uid == 0 && update_pointer) {
m_reference_table.add("\0\0", 2);
*dataptr = data;
return std::make_pair(0, "");
return {0, ""};
}
while (*data++) {
@@ -299,14 +302,14 @@ namespace osmium {
*dataptr = data;
}
return std::make_pair(static_cast_with_assert<osmium::user_id_type>(uid), user);
return {static_cast<osmium::user_id_type>(uid), user};
}
void decode_tags(osmium::builder::Builder& parent, const char** dataptr, const char* const end) {
osmium::builder::TagListBuilder builder{parent};
while (*dataptr != end) {
bool update_pointer = (**dataptr == 0x00);
const bool update_pointer = (**dataptr == 0x00);
const char* data = decode_string(dataptr, end);
const char* start = data;
@@ -316,6 +319,10 @@ namespace osmium {
}
}
if (data == end) {
throw o5m_error{"no null byte in tag value"};
}
const char* value = data;
while (*data++) {
if (data == end) {
@@ -338,8 +345,13 @@ namespace osmium {
if (**dataptr == 0x00) { // no info section
++*dataptr;
} else { // has info section
object.set_version(static_cast_with_assert<object_version_type>(protozero::decode_varint(dataptr, end)));
auto timestamp = m_delta_timestamp.update(zvarint(dataptr, end));
const auto version = protozero::decode_varint(dataptr, end);
if (version > std::numeric_limits<object_version_type>::max()) {
throw o5m_error{"object version too large"};
}
object.set_version(static_cast<object_version_type>(version));
const auto timestamp = m_delta_timestamp.update(zvarint(dataptr, end));
if (timestamp != 0) { // has timestamp
object.set_timestamp(timestamp);
object.set_changeset(m_delta_changeset.update(zvarint(dataptr, end)));
@@ -368,8 +380,8 @@ namespace osmium {
builder.set_visible(false);
builder.set_location(osmium::Location{});
} else {
auto lon = m_delta_lon.update(zvarint(&data, end));
auto lat = m_delta_lat.update(zvarint(&data, end));
const auto lon = m_delta_lon.update(zvarint(&data, end));
const auto lat = m_delta_lat.update(zvarint(&data, end));
builder.set_location(osmium::Location{lon, lat});
if (data != end) {
@@ -389,7 +401,7 @@ namespace osmium {
// no reference section, object is deleted
builder.set_visible(false);
} else {
auto reference_section_length = protozero::decode_varint(&data, end);
const auto reference_section_length = protozero::decode_varint(&data, end);
if (reference_section_length > 0) {
const char* const end_refs = data + reference_section_length;
if (end_refs > end) {
@@ -417,7 +429,7 @@ namespace osmium {
}
std::pair<osmium::item_type, const char*> decode_role(const char** dataptr, const char* const end) {
bool update_pointer = (**dataptr == 0x00);
const bool update_pointer = (**dataptr == 0x00);
const char* data = decode_string(dataptr, end);
const char* start = data;
@@ -438,7 +450,7 @@ namespace osmium {
*dataptr = data;
}
return std::make_pair(member_type, role);
return {member_type, role};
}
void decode_relation(const char* data, const char* const end) {
@@ -452,7 +464,7 @@ namespace osmium {
// no reference section, object is deleted
builder.set_visible(false);
} else {
auto reference_section_length = protozero::decode_varint(&data, end);
const auto reference_section_length = protozero::decode_varint(&data, end);
if (reference_section_length > 0) {
const char* const end_refs = data + reference_section_length;
if (end_refs > end) {
@@ -480,23 +492,23 @@ namespace osmium {
}
void decode_bbox(const char* data, const char* const end) {
auto sw_lon = zvarint(&data, end);
auto sw_lat = zvarint(&data, end);
auto ne_lon = zvarint(&data, end);
auto ne_lat = zvarint(&data, end);
const auto sw_lon = zvarint(&data, end);
const auto sw_lat = zvarint(&data, end);
const auto ne_lon = zvarint(&data, end);
const auto ne_lat = zvarint(&data, end);
m_header.add_box(osmium::Box{osmium::Location{sw_lon, sw_lat},
osmium::Location{ne_lon, ne_lat}});
}
void decode_timestamp(const char* data, const char* const end) {
auto timestamp = osmium::Timestamp(zvarint(&data, end)).to_iso();
const auto timestamp = osmium::Timestamp(zvarint(&data, end)).to_iso();
m_header.set("o5m_timestamp", timestamp);
m_header.set("timestamp", timestamp);
}
void flush() {
osmium::memory::Buffer buffer(buffer_size);
osmium::memory::Buffer buffer{buffer_size};
using std::swap;
swap(m_buffer, buffer);
send_to_output_queue(std::move(buffer));
@@ -516,7 +528,7 @@ namespace osmium {
void decode_data() {
while (ensure_bytes_available(1)) {
dataset_type ds_type = dataset_type(*m_data++);
const auto ds_type = static_cast<dataset_type>(*m_data++);
if (ds_type > dataset_type::jump) {
if (ds_type == dataset_type::reset) {
reset();
@@ -591,13 +603,17 @@ namespace osmium {
explicit O5mParser(parser_arguments& args) :
Parser(args),
m_header(),
m_buffer(buffer_size),
m_input(),
m_data(m_input.data()),
m_end(m_data) {
}
O5mParser(const O5mParser&) = delete;
O5mParser& operator=(const O5mParser&) = delete;
O5mParser(O5mParser&&) = delete;
O5mParser& operator=(O5mParser&&) = delete;
~O5mParser() noexcept final = default;
void run() final {
include/osmium/io/detail/opl_input_format.hpp
+12 -6
View File
@@ -5,7 +5,7 @@
This file is part of Osmium (http://osmcode.org/libosmium).
Copyright 2013-2017 Jochen Topf <jochen@topf.org> and others (see README).
Copyright 2013-2018 Jochen Topf <jochen@topf.org> and others (see README).
Boost Software License - Version 1.0 - August 17th, 2003
@@ -33,11 +33,6 @@ DEALINGS IN THE SOFTWARE.
*/
#include <cstdint>
#include <memory>
#include <string>
#include <utility>
#include <osmium/io/detail/input_format.hpp>
#include <osmium/io/detail/opl_parser_functions.hpp>
#include <osmium/io/file_format.hpp>
@@ -45,6 +40,11 @@ DEALINGS IN THE SOFTWARE.
#include <osmium/memory/buffer.hpp>
#include <osmium/thread/util.hpp>
#include <cstdint>
#include <memory>
#include <string>
#include <utility>
namespace osmium {
namespace io {
@@ -120,6 +120,12 @@ namespace osmium {
set_header_value(osmium::io::Header{});
}
OPLParser(const OPLParser&) = delete;
OPLParser& operator=(const OPLParser&) = delete;
OPLParser(OPLParser&&) = delete;
OPLParser& operator=(OPLParser&&) = delete;
~OPLParser() noexcept final = default;
void parse_line(const char* data) {
include/osmium/io/detail/opl_output_format.hpp
+36 -39
View File
@@ -5,7 +5,7 @@
This file is part of Osmium (http://osmcode.org/libosmium).
Copyright 2013-2017 Jochen Topf <jochen@topf.org> and others (see README).
Copyright 2013-2018 Jochen Topf <jochen@topf.org> and others (see README).
Boost Software License - Version 1.0 - August 17th, 2003
@@ -33,12 +33,6 @@ DEALINGS IN THE SOFTWARE.
*/
#include <cstdint>
#include <iterator>
#include <memory>
#include <string>
#include <utility>
#include <osmium/io/detail/output_format.hpp>
#include <osmium/io/detail/queue_util.hpp>
#include <osmium/io/detail/string_util.hpp>
@@ -51,6 +45,7 @@ DEALINGS IN THE SOFTWARE.
#include <osmium/osm/changeset.hpp>
#include <osmium/osm/item_type.hpp>
#include <osmium/osm/location.hpp>
#include <osmium/osm/metadata_options.hpp>
#include <osmium/osm/node.hpp>
#include <osmium/osm/node_ref.hpp>
#include <osmium/osm/object.hpp>
@@ -61,6 +56,12 @@ DEALINGS IN THE SOFTWARE.
#include <osmium/thread/pool.hpp>
#include <osmium/visitor.hpp>
#include <cstdint>
#include <iterator>
#include <memory>
#include <string>
#include <utility>
namespace osmium {
namespace io {
@@ -69,16 +70,16 @@ namespace osmium {
struct opl_output_options {
/// Should metadata of objects be added?
bool add_metadata;
/// Which metadata of objects should be added?
osmium::metadata_options add_metadata;
/// Should node locations be added to ways?
bool locations_on_ways;
bool locations_on_ways = false;
/// Write in form of a diff file?
bool format_as_diff;
bool format_as_diff = false;
};
}; // struct opl_output_options
/**
* Writes out one buffer with OSM data in OPL format.
@@ -123,19 +124,29 @@ namespace osmium {
void write_meta(const osmium::OSMObject& object) {
output_int(object.id());
if (m_options.add_metadata) {
*m_out += ' ';
write_field_int('v', object.version());
if (m_options.add_metadata.any()) {
if (m_options.add_metadata.version()) {
*m_out += ' ';
write_field_int('v', object.version());
}
*m_out += " d";
*m_out += (object.visible() ? 'V' : 'D');
*m_out += ' ';
write_field_int('c', object.changeset());
*m_out += ' ';
write_field_timestamp('t', object.timestamp());
*m_out += ' ';
write_field_int('i', object.uid());
*m_out += " u";
append_encoded_string(object.user());
if (m_options.add_metadata.changeset()) {
*m_out += ' ';
write_field_int('c', object.changeset());
}
if (m_options.add_metadata.timestamp()) {
*m_out += ' ';
write_field_timestamp('t', object.timestamp());
}
if (m_options.add_metadata.uid()) {
*m_out += ' ';
write_field_int('i', object.uid());
}
if (m_options.add_metadata.user()) {
*m_out += " u";
append_encoded_string(object.user());
}
}
write_tags(object.tags());
}
@@ -167,14 +178,6 @@ namespace osmium {
m_options(options) {
}
OPLOutputBlock(const OPLOutputBlock&) = default;
OPLOutputBlock& operator=(const OPLOutputBlock&) = default;
OPLOutputBlock(OPLOutputBlock&&) = default;
OPLOutputBlock& operator=(OPLOutputBlock&&) = default;
~OPLOutputBlock() noexcept = default;
std::string operator()() {
osmium::apply(m_input_buffer->cbegin(), m_input_buffer->cend(), *this);
@@ -285,18 +288,12 @@ namespace osmium {
public:
OPLOutputFormat(osmium::thread::Pool& pool, const osmium::io::File& file, future_string_queue_type& output_queue) :
OutputFormat(pool, output_queue),
m_options() {
m_options.add_metadata = file.is_not_false("add_metadata");
OutputFormat(pool, output_queue) {
m_options.add_metadata = osmium::metadata_options{file.get("add_metadata")};
m_options.locations_on_ways = file.is_true("locations_on_ways");
m_options.format_as_diff = file.is_true("diff");
}
OPLOutputFormat(const OPLOutputFormat&) = delete;
OPLOutputFormat& operator=(const OPLOutputFormat&) = delete;
~OPLOutputFormat() noexcept final = default;
void write_buffer(osmium::memory::Buffer&& buffer) final {
m_output_queue.push(m_pool.submit(OPLOutputBlock{std::move(buffer), m_options}));
}
include/osmium/io/detail/opl_parser_functions.hpp
+12 -11
View File
@@ -5,7 +5,7 @@
This file is part of Osmium (http://osmcode.org/libosmium).
Copyright 2013-2017 Jochen Topf <jochen@topf.org> and others (see README).
Copyright 2013-2018 Jochen Topf <jochen@topf.org> and others (see README).
Boost Software License - Version 1.0 - August 17th, 2003
@@ -33,15 +33,6 @@ DEALINGS IN THE SOFTWARE.
*/
#include <cstdint>
#include <cstdlib>
#include <iterator>
#include <limits>
#include <stdexcept>
#include <string>
#include <utf8.h>
#include <osmium/builder/osm_object_builder.hpp>
#include <osmium/io/error.hpp>
#include <osmium/memory/buffer.hpp>
@@ -56,6 +47,15 @@ DEALINGS IN THE SOFTWARE.
#include <osmium/osm/types.hpp>
#include <osmium/osm/way.hpp>
#include <utf8.h>
#include <cstdint>
#include <cstdlib>
#include <iterator>
#include <limits>
#include <stdexcept>
#include <string>
namespace osmium {
namespace builder {
@@ -187,7 +187,8 @@ namespace osmium {
while (true) {
if (*s == '\0' || *s == ' ' || *s == '\t' || *s == ',' || *s == '=') {
break;
} else if (*s == '%') {
}
if (*s == '%') {
++s;
opl_parse_escaped(&s, result);
} else {
include/osmium/io/detail/output_format.hpp
+31 -26
View File
@@ -5,7 +5,7 @@
This file is part of Osmium (http://osmcode.org/libosmium).
Copyright 2013-2017 Jochen Topf <jochen@topf.org> and others (see README).
Copyright 2013-2018 Jochen Topf <jochen@topf.org> and others (see README).
Boost Software License - Version 1.0 - August 17th, 2003
@@ -33,13 +33,6 @@ DEALINGS IN THE SOFTWARE.
*/
#include <cstdint>
#include <functional>
#include <map>
#include <memory>
#include <string>
#include <utility>
#include <osmium/handler.hpp>
#include <osmium/io/detail/queue_util.hpp>
#include <osmium/io/error.hpp>
@@ -48,6 +41,13 @@ DEALINGS IN THE SOFTWARE.
#include <osmium/memory/buffer.hpp>
#include <osmium/thread/pool.hpp>
#include <array>
#include <cstdint>
#include <functional>
#include <memory>
#include <string>
#include <utility>
namespace osmium {
namespace io {
@@ -121,23 +121,23 @@ namespace osmium {
public:
OutputFormat(osmium::thread::Pool& pool, future_string_queue_type& output_queue) :
OutputFormat(osmium::thread::Pool& pool, future_string_queue_type& output_queue) noexcept :
m_pool(pool),
m_output_queue(output_queue) {
}
OutputFormat(const OutputFormat&) = delete;
OutputFormat(OutputFormat&&) = delete;
OutputFormat& operator=(const OutputFormat&) = delete;
OutputFormat(OutputFormat&&) = delete;
OutputFormat& operator=(OutputFormat&&) = delete;
virtual ~OutputFormat() noexcept = default;
virtual void write_header(const osmium::io::Header&) {
virtual void write_header(const osmium::io::Header& /*header*/) {
}
virtual void write_buffer(osmium::memory::Buffer&&) = 0;
virtual void write_buffer(osmium::memory::Buffer&& /*buffer*/) = 0;
virtual void write_end() {
}
@@ -159,32 +159,34 @@ namespace osmium {
private:
using map_type = std::map<osmium::io::file_format, create_output_type>;
std::array<create_output_type, static_cast<std::size_t>(file_format::last) + 1> m_callbacks;
map_type m_callbacks;
OutputFormatFactory() noexcept = default;
OutputFormatFactory() :
m_callbacks() {
create_output_type& callbacks(const osmium::io::file_format format) noexcept {
return m_callbacks[static_cast<std::size_t>(format)];
}
const create_output_type& callbacks(const osmium::io::file_format format) const noexcept {
return m_callbacks[static_cast<std::size_t>(format)];
}
public:
static OutputFormatFactory& instance() {
static OutputFormatFactory& instance() noexcept {
static OutputFormatFactory factory;
return factory;
}
bool register_output_format(osmium::io::file_format format, create_output_type create_function) {
if (! m_callbacks.insert(map_type::value_type(format, create_function)).second) {
return false;
}
bool register_output_format(const osmium::io::file_format format, create_output_type&& create_function) {
callbacks(format) = std::forward<create_output_type>(create_function);
return true;
}
std::unique_ptr<osmium::io::detail::OutputFormat> create_output(osmium::thread::Pool& pool, const osmium::io::File& file, future_string_queue_type& output_queue) {
const auto it = m_callbacks.find(file.format());
if (it != m_callbacks.end()) {
return std::unique_ptr<osmium::io::detail::OutputFormat>((it->second)(pool, file, output_queue));
std::unique_ptr<osmium::io::detail::OutputFormat> create_output(osmium::thread::Pool& pool, const osmium::io::File& file, future_string_queue_type& output_queue) const {
const auto func = callbacks(file.format());
if (func) {
return std::unique_ptr<osmium::io::detail::OutputFormat>((func)(pool, file, output_queue));
}
throw unsupported_file_format_error{
@@ -208,6 +210,9 @@ namespace osmium {
BlackholeOutputFormat(const BlackholeOutputFormat&) = delete;
BlackholeOutputFormat& operator=(const BlackholeOutputFormat&) = delete;
BlackholeOutputFormat(BlackholeOutputFormat&&) = delete;
BlackholeOutputFormat& operator=(BlackholeOutputFormat&&) = delete;
~BlackholeOutputFormat() noexcept final = default;
void write_buffer(osmium::memory::Buffer&& /*buffer*/) final {
include/osmium/io/detail/pbf.hpp
+7 -7
View File
@@ -5,7 +5,7 @@
This file is part of Osmium (http://osmcode.org/libosmium).
Copyright 2013-2017 Jochen Topf <jochen@topf.org> and others (see README).
Copyright 2013-2018 Jochen Topf <jochen@topf.org> and others (see README).
Boost Software License - Version 1.0 - August 17th, 2003
@@ -33,18 +33,18 @@ DEALINGS IN THE SOFTWARE.
*/
#include <cstdint>
#include <string>
#include <osmium/io/error.hpp>
#include <osmium/osm/location.hpp>
// needed for htonl and ntohl
// needed for htonl and ntohl or their equivalent in protozero
#ifndef _WIN32
# include <netinet/in.h>
#else
# include <winsock2.h>
# include <protozero/byteswap.hpp>
#endif
#include <osmium/io/error.hpp>
#include <osmium/osm/location.hpp>
#include <cstdint>
#include <string>
namespace osmium {
include/osmium/io/detail/pbf_decoder.hpp
+170 -163
View File
@@ -5,7 +5,7 @@
This file is part of Osmium (http://osmcode.org/libosmium).
Copyright 2013-2017 Jochen Topf <jochen@topf.org> and others (see README).
Copyright 2013-2018 Jochen Topf <jochen@topf.org> and others (see README).
Boost Software License - Version 1.0 - August 17th, 2003
@@ -33,19 +33,6 @@ DEALINGS IN THE SOFTWARE.
*/
#include <cstdint>
#include <cstring>
#include <limits>
#include <memory>
#include <stdexcept>
#include <string>
#include <utility>
#include <vector>
#include <protozero/iterators.hpp>
#include <protozero/pbf_message.hpp>
#include <protozero/types.hpp>
#include <osmium/builder/osm_object_builder.hpp>
#include <osmium/io/detail/pbf.hpp> // IWYU pragma: export
#include <osmium/io/detail/protobuf_tags.hpp>
@@ -63,9 +50,21 @@ DEALINGS IN THE SOFTWARE.
#include <osmium/osm/timestamp.hpp>
#include <osmium/osm/types.hpp>
#include <osmium/osm/way.hpp>
#include <osmium/util/cast.hpp>
#include <osmium/util/delta.hpp>
#include <protozero/iterators.hpp>
#include <protozero/pbf_message.hpp>
#include <protozero/types.hpp>
#include <cstdint>
#include <cstring>
#include <limits>
#include <memory>
#include <stdexcept>
#include <string>
#include <utility>
#include <vector>
namespace osmium {
namespace builder {
@@ -103,7 +102,7 @@ namespace osmium {
}
protozero::pbf_message<OSMFormat::StringTable> pbf_string_table{data};
while (pbf_string_table.next(OSMFormat::StringTable::repeated_bytes_s)) {
while (pbf_string_table.next(OSMFormat::StringTable::repeated_bytes_s, protozero::pbf_wire_type::length_delimited)) {
const auto str_view = pbf_string_table.get_view();
if (str_view.size() > osmium::max_osm_string_length) {
throw osmium::pbf_error{"overlong string in string table"};
@@ -115,20 +114,20 @@ namespace osmium {
void decode_primitive_block_metadata() {
protozero::pbf_message<OSMFormat::PrimitiveBlock> pbf_primitive_block{m_data};
while (pbf_primitive_block.next()) {
switch (pbf_primitive_block.tag()) {
case OSMFormat::PrimitiveBlock::required_StringTable_stringtable:
switch (pbf_primitive_block.tag_and_type()) {
case protozero::tag_and_type(OSMFormat::PrimitiveBlock::required_StringTable_stringtable, protozero::pbf_wire_type::length_delimited):
decode_stringtable(pbf_primitive_block.get_view());
break;
case OSMFormat::PrimitiveBlock::optional_int32_granularity:
case protozero::tag_and_type(OSMFormat::PrimitiveBlock::optional_int32_granularity, protozero::pbf_wire_type::varint):
m_granularity = pbf_primitive_block.get_int32();
break;
case OSMFormat::PrimitiveBlock::optional_int32_date_granularity:
case protozero::tag_and_type(OSMFormat::PrimitiveBlock::optional_int32_date_granularity, protozero::pbf_wire_type::varint):
m_date_factor = pbf_primitive_block.get_int32();
break;
case OSMFormat::PrimitiveBlock::optional_int64_lat_offset:
case protozero::tag_and_type(OSMFormat::PrimitiveBlock::optional_int64_lat_offset, protozero::pbf_wire_type::varint):
m_lat_offset = pbf_primitive_block.get_int64();
break;
case OSMFormat::PrimitiveBlock::optional_int64_lon_offset:
case protozero::tag_and_type(OSMFormat::PrimitiveBlock::optional_int64_lon_offset, protozero::pbf_wire_type::varint):
m_lon_offset = pbf_primitive_block.get_int64();
break;
default:
@@ -139,11 +138,11 @@ namespace osmium {
void decode_primitive_block_data() {
protozero::pbf_message<OSMFormat::PrimitiveBlock> pbf_primitive_block{m_data};
while (pbf_primitive_block.next(OSMFormat::PrimitiveBlock::repeated_PrimitiveGroup_primitivegroup)) {
while (pbf_primitive_block.next(OSMFormat::PrimitiveBlock::repeated_PrimitiveGroup_primitivegroup, protozero::pbf_wire_type::length_delimited)) {
protozero::pbf_message<OSMFormat::PrimitiveGroup> pbf_primitive_group = pbf_primitive_block.get_message();
while (pbf_primitive_group.next()) {
switch (pbf_primitive_group.tag()) {
case OSMFormat::PrimitiveGroup::repeated_Node_nodes:
switch (pbf_primitive_group.tag_and_type()) {
case protozero::tag_and_type(OSMFormat::PrimitiveGroup::repeated_Node_nodes, protozero::pbf_wire_type::length_delimited):
if (m_read_types & osmium::osm_entity_bits::node) {
decode_node(pbf_primitive_group.get_view());
m_buffer.commit();
@@ -151,7 +150,7 @@ namespace osmium {
pbf_primitive_group.skip();
}
break;
case OSMFormat::PrimitiveGroup::optional_DenseNodes_dense:
case protozero::tag_and_type(OSMFormat::PrimitiveGroup::optional_DenseNodes_dense, protozero::pbf_wire_type::length_delimited):
if (m_read_types & osmium::osm_entity_bits::node) {
if (m_read_metadata == osmium::io::read_meta::yes) {
decode_dense_nodes(pbf_primitive_group.get_view());
@@ -163,7 +162,7 @@ namespace osmium {
pbf_primitive_group.skip();
}
break;
case OSMFormat::PrimitiveGroup::repeated_Way_ways:
case protozero::tag_and_type(OSMFormat::PrimitiveGroup::repeated_Way_ways, protozero::pbf_wire_type::length_delimited):
if (m_read_types & osmium::osm_entity_bits::way) {
decode_way(pbf_primitive_group.get_view());
m_buffer.commit();
@@ -171,7 +170,7 @@ namespace osmium {
pbf_primitive_group.skip();
}
break;
case OSMFormat::PrimitiveGroup::repeated_Relation_relations:
case protozero::tag_and_type(OSMFormat::PrimitiveGroup::repeated_Relation_relations, protozero::pbf_wire_type::length_delimited):
if (m_read_types & osmium::osm_entity_bits::relation) {
decode_relation(pbf_primitive_group.get_view());
m_buffer.commit();
@@ -191,35 +190,45 @@ namespace osmium {
protozero::pbf_message<OSMFormat::Info> pbf_info{data};
while (pbf_info.next()) {
switch (pbf_info.tag()) {
case OSMFormat::Info::optional_int32_version:
switch (pbf_info.tag_and_type()) {
case protozero::tag_and_type(OSMFormat::Info::optional_int32_version, protozero::pbf_wire_type::varint):
{
const auto version = pbf_info.get_int32();
if (version < 0) {
if (version < -1) {
throw osmium::pbf_error{"object version must not be negative"};
}
object.set_version(static_cast_with_assert<object_version_type>(version));
if (version == -1) {
object.set_version(0U);
} else {
object.set_version(static_cast<object_version_type>(version));
}
}
break;
case OSMFormat::Info::optional_int64_timestamp:
case protozero::tag_and_type(OSMFormat::Info::optional_int64_timestamp, protozero::pbf_wire_type::varint):
object.set_timestamp(pbf_info.get_int64() * m_date_factor / 1000);
break;
case OSMFormat::Info::optional_int64_changeset:
case protozero::tag_and_type(OSMFormat::Info::optional_int64_changeset, protozero::pbf_wire_type::varint):
{
const auto changeset_id = pbf_info.get_int64();
if (changeset_id < 0) {
throw osmium::pbf_error{"object changeset_id must not be negative"};
if (changeset_id < -1 || changeset_id >= std::numeric_limits<changeset_id_type>::max()) {
throw osmium::pbf_error{"object changeset_id must be between 0 and 2^32-1"};
}
if (changeset_id == -1) {
object.set_changeset(0U);
} else {
object.set_changeset(static_cast<changeset_id_type>(changeset_id));
}
object.set_changeset(static_cast_with_assert<changeset_id_type>(changeset_id));
}
break;
case OSMFormat::Info::optional_int32_uid:
case protozero::tag_and_type(OSMFormat::Info::optional_int32_uid, protozero::pbf_wire_type::varint):
object.set_uid_from_signed(pbf_info.get_int32());
break;
case OSMFormat::Info::optional_uint32_user_sid:
case protozero::tag_and_type(OSMFormat::Info::optional_uint32_user_sid, protozero::pbf_wire_type::varint):
user = m_stringtable.at(pbf_info.get_uint32());
break;
case OSMFormat::Info::optional_bool_visible:
case protozero::tag_and_type(OSMFormat::Info::optional_bool_visible, protozero::pbf_wire_type::varint):
object.set_visible(pbf_info.get_bool());
break;
default:
@@ -249,7 +258,7 @@ namespace osmium {
}
}
int32_t convert_pbf_coordinate(int64_t c) const {
int32_t convert_pbf_coordinate(int64_t c) const noexcept {
return int32_t((c * m_granularity + m_lon_offset) / resolution_convert);
}
@@ -266,27 +275,27 @@ namespace osmium {
protozero::pbf_message<OSMFormat::Node> pbf_node{data};
while (pbf_node.next()) {
switch (pbf_node.tag()) {
case OSMFormat::Node::required_sint64_id:
switch (pbf_node.tag_and_type()) {
case protozero::tag_and_type(OSMFormat::Node::required_sint64_id, protozero::pbf_wire_type::varint):
node.set_id(pbf_node.get_sint64());
break;
case OSMFormat::Node::packed_uint32_keys:
case protozero::tag_and_type(OSMFormat::Node::packed_uint32_keys, protozero::pbf_wire_type::length_delimited):
keys = pbf_node.get_packed_uint32();
break;
case OSMFormat::Node::packed_uint32_vals:
case protozero::tag_and_type(OSMFormat::Node::packed_uint32_vals, protozero::pbf_wire_type::length_delimited):
vals = pbf_node.get_packed_uint32();
break;
case OSMFormat::Node::optional_Info_info:
case protozero::tag_and_type(OSMFormat::Node::optional_Info_info, protozero::pbf_wire_type::length_delimited):
if (m_read_metadata == osmium::io::read_meta::yes) {
user = decode_info(pbf_node.get_view(), builder.object());
} else {
pbf_node.skip();
}
break;
case OSMFormat::Node::required_sint64_lat:
case protozero::tag_and_type(OSMFormat::Node::required_sint64_lat, protozero::pbf_wire_type::varint):
lat = pbf_node.get_sint64();
break;
case OSMFormat::Node::required_sint64_lon:
case protozero::tag_and_type(OSMFormat::Node::required_sint64_lon, protozero::pbf_wire_type::varint):
lon = pbf_node.get_sint64();
break;
default:
@@ -323,30 +332,30 @@ namespace osmium {
protozero::pbf_message<OSMFormat::Way> pbf_way{data};
while (pbf_way.next()) {
switch (pbf_way.tag()) {
case OSMFormat::Way::required_int64_id:
switch (pbf_way.tag_and_type()) {
case protozero::tag_and_type(OSMFormat::Way::required_int64_id, protozero::pbf_wire_type::varint):
builder.object().set_id(pbf_way.get_int64());
break;
case OSMFormat::Way::packed_uint32_keys:
case protozero::tag_and_type(OSMFormat::Way::packed_uint32_keys, protozero::pbf_wire_type::length_delimited):
keys = pbf_way.get_packed_uint32();
break;
case OSMFormat::Way::packed_uint32_vals:
case protozero::tag_and_type(OSMFormat::Way::packed_uint32_vals, protozero::pbf_wire_type::length_delimited):
vals = pbf_way.get_packed_uint32();
break;
case OSMFormat::Way::optional_Info_info:
case protozero::tag_and_type(OSMFormat::Way::optional_Info_info, protozero::pbf_wire_type::length_delimited):
if (m_read_metadata == osmium::io::read_meta::yes) {
user = decode_info(pbf_way.get_view(), builder.object());
} else {
pbf_way.skip();
}
break;
case OSMFormat::Way::packed_sint64_refs:
case protozero::tag_and_type(OSMFormat::Way::packed_sint64_refs, protozero::pbf_wire_type::length_delimited):
refs = pbf_way.get_packed_sint64();
break;
case OSMFormat::Way::packed_sint64_lat:
case protozero::tag_and_type(OSMFormat::Way::packed_sint64_lat, protozero::pbf_wire_type::length_delimited):
lats = pbf_way.get_packed_sint64();
break;
case OSMFormat::Way::packed_sint64_lon:
case protozero::tag_and_type(OSMFormat::Way::packed_sint64_lon, protozero::pbf_wire_type::length_delimited):
lons = pbf_way.get_packed_sint64();
break;
default:
@@ -358,14 +367,14 @@ namespace osmium {
if (!refs.empty()) {
osmium::builder::WayNodeListBuilder wnl_builder{builder};
osmium::util::DeltaDecode<int64_t> ref;
osmium::DeltaDecode<int64_t> ref;
if (lats.empty()) {
for (const auto& ref_value : refs) {
wnl_builder.add_node_ref(ref.update(ref_value));
}
} else {
osmium::util::DeltaDecode<int64_t> lon;
osmium::util::DeltaDecode<int64_t> lat;
osmium::DeltaDecode<int64_t> lon;
osmium::DeltaDecode<int64_t> lat;
while (!refs.empty() && !lons.empty() && !lats.empty()) {
wnl_builder.add_node_ref(
ref.update(refs.front()),
@@ -395,30 +404,30 @@ namespace osmium {
protozero::pbf_message<OSMFormat::Relation> pbf_relation{data};
while (pbf_relation.next()) {
switch (pbf_relation.tag()) {
case OSMFormat::Relation::required_int64_id:
switch (pbf_relation.tag_and_type()) {
case protozero::tag_and_type(OSMFormat::Relation::required_int64_id, protozero::pbf_wire_type::varint):
builder.object().set_id(pbf_relation.get_int64());
break;
case OSMFormat::Relation::packed_uint32_keys:
case protozero::tag_and_type(OSMFormat::Relation::packed_uint32_keys, protozero::pbf_wire_type::length_delimited):
keys = pbf_relation.get_packed_uint32();
break;
case OSMFormat::Relation::packed_uint32_vals:
case protozero::tag_and_type(OSMFormat::Relation::packed_uint32_vals, protozero::pbf_wire_type::length_delimited):
vals = pbf_relation.get_packed_uint32();
break;
case OSMFormat::Relation::optional_Info_info:
case protozero::tag_and_type(OSMFormat::Relation::optional_Info_info, protozero::pbf_wire_type::length_delimited):
if (m_read_metadata == osmium::io::read_meta::yes) {
user = decode_info(pbf_relation.get_view(), builder.object());
} else {
pbf_relation.skip();
}
break;
case OSMFormat::Relation::packed_int32_roles_sid:
case protozero::tag_and_type(OSMFormat::Relation::packed_int32_roles_sid, protozero::pbf_wire_type::length_delimited):
roles = pbf_relation.get_packed_int32();
break;
case OSMFormat::Relation::packed_sint64_memids:
case protozero::tag_and_type(OSMFormat::Relation::packed_sint64_memids, protozero::pbf_wire_type::length_delimited):
refs = pbf_relation.get_packed_sint64();
break;
case OSMFormat::Relation::packed_MemberType_types:
case protozero::tag_and_type(OSMFormat::Relation::packed_MemberType_types, protozero::pbf_wire_type::length_delimited):
types = pbf_relation.get_packed_enum();
break;
default:
@@ -430,7 +439,7 @@ namespace osmium {
if (!refs.empty()) {
osmium::builder::RelationMemberListBuilder rml_builder{builder};
osmium::util::DeltaDecode<int64_t> ref;
osmium::DeltaDecode<int64_t> ref;
while (!roles.empty() && !refs.empty() && !types.empty()) {
const auto& r = m_stringtable.at(roles.front());
const int type = types.front();
@@ -477,17 +486,17 @@ namespace osmium {
protozero::pbf_message<OSMFormat::DenseNodes> pbf_dense_nodes{data};
while (pbf_dense_nodes.next()) {
switch (pbf_dense_nodes.tag()) {
case OSMFormat::DenseNodes::packed_sint64_id:
switch (pbf_dense_nodes.tag_and_type()) {
case protozero::tag_and_type(OSMFormat::DenseNodes::packed_sint64_id, protozero::pbf_wire_type::length_delimited):
ids = pbf_dense_nodes.get_packed_sint64();
break;
case OSMFormat::DenseNodes::packed_sint64_lat:
case protozero::tag_and_type(OSMFormat::DenseNodes::packed_sint64_lat, protozero::pbf_wire_type::length_delimited):
lats = pbf_dense_nodes.get_packed_sint64();
break;
case OSMFormat::DenseNodes::packed_sint64_lon:
case protozero::tag_and_type(OSMFormat::DenseNodes::packed_sint64_lon, protozero::pbf_wire_type::length_delimited):
lons = pbf_dense_nodes.get_packed_sint64();
break;
case OSMFormat::DenseNodes::packed_int32_keys_vals:
case protozero::tag_and_type(OSMFormat::DenseNodes::packed_int32_keys_vals, protozero::pbf_wire_type::length_delimited):
tags = pbf_dense_nodes.get_packed_int32();
break;
default:
@@ -495,9 +504,9 @@ namespace osmium {
}
}
osmium::util::DeltaDecode<int64_t> dense_id;
osmium::util::DeltaDecode<int64_t> dense_latitude;
osmium::util::DeltaDecode<int64_t> dense_longitude;
osmium::DeltaDecode<int64_t> dense_id;
osmium::DeltaDecode<int64_t> dense_latitude;
osmium::DeltaDecode<int64_t> dense_longitude;
auto tag_it = tags.begin();
@@ -531,8 +540,7 @@ namespace osmium {
}
void decode_dense_nodes(const data_view& data) {
bool has_info = false;
bool has_visibles = false;
bool has_info = false;
protozero::iterator_range<protozero::pbf_reader::const_sint64_iterator> ids;
protozero::iterator_range<protozero::pbf_reader::const_sint64_iterator> lats;
@@ -549,33 +557,32 @@ namespace osmium {
protozero::pbf_message<OSMFormat::DenseNodes> pbf_dense_nodes{data};
while (pbf_dense_nodes.next()) {
switch (pbf_dense_nodes.tag()) {
case OSMFormat::DenseNodes::packed_sint64_id:
switch (pbf_dense_nodes.tag_and_type()) {
case protozero::tag_and_type(OSMFormat::DenseNodes::packed_sint64_id, protozero::pbf_wire_type::length_delimited):
ids = pbf_dense_nodes.get_packed_sint64();
break;
case OSMFormat::DenseNodes::optional_DenseInfo_denseinfo:
case protozero::tag_and_type(OSMFormat::DenseNodes::optional_DenseInfo_denseinfo, protozero::pbf_wire_type::length_delimited):
{
has_info = true;
protozero::pbf_message<OSMFormat::DenseInfo> pbf_dense_info{pbf_dense_nodes.get_message()};
while (pbf_dense_info.next()) {
switch (pbf_dense_info.tag()) {
case OSMFormat::DenseInfo::packed_int32_version:
switch (pbf_dense_info.tag_and_type()) {
case protozero::tag_and_type(OSMFormat::DenseInfo::packed_int32_version, protozero::pbf_wire_type::length_delimited):
versions = pbf_dense_info.get_packed_int32();
break;
case OSMFormat::DenseInfo::packed_sint64_timestamp:
case protozero::tag_and_type(OSMFormat::DenseInfo::packed_sint64_timestamp, protozero::pbf_wire_type::length_delimited):
timestamps = pbf_dense_info.get_packed_sint64();
break;
case OSMFormat::DenseInfo::packed_sint64_changeset:
case protozero::tag_and_type(OSMFormat::DenseInfo::packed_sint64_changeset, protozero::pbf_wire_type::length_delimited):
changesets = pbf_dense_info.get_packed_sint64();
break;
case OSMFormat::DenseInfo::packed_sint32_uid:
case protozero::tag_and_type(OSMFormat::DenseInfo::packed_sint32_uid, protozero::pbf_wire_type::length_delimited):
uids = pbf_dense_info.get_packed_sint32();
break;
case OSMFormat::DenseInfo::packed_sint32_user_sid:
case protozero::tag_and_type(OSMFormat::DenseInfo::packed_sint32_user_sid, protozero::pbf_wire_type::length_delimited):
user_sids = pbf_dense_info.get_packed_sint32();
break;
case OSMFormat::DenseInfo::packed_bool_visible:
has_visibles = true;
case protozero::tag_and_type(OSMFormat::DenseInfo::packed_bool_visible, protozero::pbf_wire_type::length_delimited):
visibles = pbf_dense_info.get_packed_bool();
break;
default:
@@ -584,13 +591,13 @@ namespace osmium {
}
}
break;
case OSMFormat::DenseNodes::packed_sint64_lat:
case protozero::tag_and_type(OSMFormat::DenseNodes::packed_sint64_lat, protozero::pbf_wire_type::length_delimited):
lats = pbf_dense_nodes.get_packed_sint64();
break;
case OSMFormat::DenseNodes::packed_sint64_lon:
case protozero::tag_and_type(OSMFormat::DenseNodes::packed_sint64_lon, protozero::pbf_wire_type::length_delimited):
lons = pbf_dense_nodes.get_packed_sint64();
break;
case OSMFormat::DenseNodes::packed_int32_keys_vals:
case protozero::tag_and_type(OSMFormat::DenseNodes::packed_int32_keys_vals, protozero::pbf_wire_type::length_delimited):
tags = pbf_dense_nodes.get_packed_int32();
break;
default:
@@ -598,13 +605,13 @@ namespace osmium {
}
}
osmium::util::DeltaDecode<int64_t> dense_id;
osmium::util::DeltaDecode<int64_t> dense_latitude;
osmium::util::DeltaDecode<int64_t> dense_longitude;
osmium::util::DeltaDecode<int64_t> dense_uid;
osmium::util::DeltaDecode<int64_t> dense_user_sid;
osmium::util::DeltaDecode<int64_t> dense_changeset;
osmium::util::DeltaDecode<int64_t> dense_timestamp;
osmium::DeltaDecode<int64_t> dense_id;
osmium::DeltaDecode<int64_t> dense_latitude;
osmium::DeltaDecode<int64_t> dense_longitude;
osmium::DeltaDecode<int64_t> dense_uid;
osmium::DeltaDecode<int64_t> dense_user_sid;
osmium::DeltaDecode<int64_t> dense_changeset;
osmium::DeltaDecode<int64_t> dense_timestamp;
auto tag_it = tags.begin();
@@ -624,47 +631,55 @@ namespace osmium {
ids.drop_front();
if (has_info) {
if (versions.empty() ||
changesets.empty() ||
timestamps.empty() ||
uids.empty() ||
user_sids.empty()) {
// this is against the spec, must have same number of elements
throw osmium::pbf_error{"PBF format error"};
}
const auto version = versions.front();
versions.drop_front();
if (version < 0) {
throw osmium::pbf_error{"object version must not be negative"};
}
node.set_version(static_cast<osmium::object_version_type>(version));
const auto changeset_id = dense_changeset.update(changesets.front());
changesets.drop_front();
if (changeset_id < 0) {
throw osmium::pbf_error{"object changeset_id must not be negative"};
}
node.set_changeset(static_cast<osmium::changeset_id_type>(changeset_id));
node.set_timestamp(dense_timestamp.update(timestamps.front()) * m_date_factor / 1000);
timestamps.drop_front();
node.set_uid_from_signed(static_cast<osmium::signed_user_id_type>(dense_uid.update(uids.front())));
uids.drop_front();
if (has_visibles) {
if (visibles.empty()) {
// this is against the spec, must have same number of elements
throw osmium::pbf_error{"PBF format error"};
if (!versions.empty()) {
const auto version = versions.front();
versions.drop_front();
if (version < -1) {
throw osmium::pbf_error{"object version must not be negative"};
}
if (version == -1) {
node.set_version(0U);
} else {
node.set_version(static_cast<osmium::object_version_type>(version));
}
}
if (!changesets.empty()) {
const auto changeset_id = dense_changeset.update(changesets.front());
changesets.drop_front();
if (changeset_id < -1 || changeset_id >= std::numeric_limits<changeset_id_type>::max()) {
throw osmium::pbf_error{"object changeset_id must be between 0 and 2^32-1"};
}
if (changeset_id == -1) {
node.set_changeset(0U);
} else {
node.set_changeset(static_cast<osmium::changeset_id_type>(changeset_id));
}
}
if (!timestamps.empty()) {
node.set_timestamp(dense_timestamp.update(timestamps.front()) * m_date_factor / 1000);
timestamps.drop_front();
}
if (!uids.empty()) {
node.set_uid_from_signed(static_cast<osmium::signed_user_id_type>(dense_uid.update(uids.front())));
uids.drop_front();
}
if (!visibles.empty()) {
visible = (visibles.front() != 0);
visibles.drop_front();
}
node.set_visible(visible);
const auto& u = m_stringtable.at(dense_user_sid.update(user_sids.front()));
user_sids.drop_front();
builder.set_user(u.first, u.second);
if (!user_sids.empty()) {
const auto& u = m_stringtable.at(dense_user_sid.update(user_sids.front()));
user_sids.drop_front();
builder.set_user(u.first, u.second);
}
}
// even if the node isn't visible, there's still a record
@@ -722,8 +737,8 @@ namespace osmium {
protozero::pbf_message<FileFormat::Blob> pbf_blob{blob_data};
while (pbf_blob.next()) {
switch (pbf_blob.tag()) {
case FileFormat::Blob::optional_bytes_raw:
switch (pbf_blob.tag_and_type()) {
case protozero::tag_and_type(FileFormat::Blob::optional_bytes_raw, protozero::pbf_wire_type::length_delimited):
{
const auto data_len = pbf_blob.get_view();
if (data_len.size() > max_uncompressed_blob_size) {
@@ -731,27 +746,27 @@ namespace osmium {
}
return data_len;
}
case FileFormat::Blob::optional_int32_raw_size:
case protozero::tag_and_type(FileFormat::Blob::optional_int32_raw_size, protozero::pbf_wire_type::varint):
raw_size = pbf_blob.get_int32();
if (raw_size <= 0 || uint32_t(raw_size) > max_uncompressed_blob_size) {
throw osmium::pbf_error{"illegal blob size"};
}
break;
case FileFormat::Blob::optional_bytes_zlib_data:
case protozero::tag_and_type(FileFormat::Blob::optional_bytes_zlib_data, protozero::pbf_wire_type::length_delimited):
zlib_data = pbf_blob.get_view();
break;
case FileFormat::Blob::optional_bytes_lzma_data:
case protozero::tag_and_type(FileFormat::Blob::optional_bytes_lzma_data, protozero::pbf_wire_type::length_delimited):
throw osmium::pbf_error{"lzma blobs not implemented"};
default:
throw osmium::pbf_error{"unknown compression"};
}
}
if (zlib_data.size() != 0 && raw_size != 0) {
if (!zlib_data.empty() && raw_size != 0) {
return osmium::io::detail::zlib_uncompress_string(
zlib_data.data(),
static_cast<unsigned long>(zlib_data.size()),
static_cast<unsigned long>(raw_size),
static_cast<unsigned long>(zlib_data.size()), // NOLINT(google-runtime-int)
static_cast<unsigned long>(raw_size), // NOLINT(google-runtime-int)
output
);
}
@@ -767,17 +782,17 @@ namespace osmium {
protozero::pbf_message<OSMFormat::HeaderBBox> pbf_header_bbox{data};
while (pbf_header_bbox.next()) {
switch (pbf_header_bbox.tag()) {
case OSMFormat::HeaderBBox::required_sint64_left:
switch (pbf_header_bbox.tag_and_type()) {
case protozero::tag_and_type(OSMFormat::HeaderBBox::required_sint64_left, protozero::pbf_wire_type::varint):
left = pbf_header_bbox.get_sint64();
break;
case OSMFormat::HeaderBBox::required_sint64_right:
case protozero::tag_and_type(OSMFormat::HeaderBBox::required_sint64_right, protozero::pbf_wire_type::varint):
right = pbf_header_bbox.get_sint64();
break;
case OSMFormat::HeaderBBox::required_sint64_top:
case protozero::tag_and_type(OSMFormat::HeaderBBox::required_sint64_top, protozero::pbf_wire_type::varint):
top = pbf_header_bbox.get_sint64();
break;
case OSMFormat::HeaderBBox::required_sint64_bottom:
case protozero::tag_and_type(OSMFormat::HeaderBBox::required_sint64_bottom, protozero::pbf_wire_type::varint):
bottom = pbf_header_bbox.get_sint64();
break;
default:
@@ -805,11 +820,11 @@ namespace osmium {
protozero::pbf_message<OSMFormat::HeaderBlock> pbf_header_block{data};
while (pbf_header_block.next()) {
switch (pbf_header_block.tag()) {
case OSMFormat::HeaderBlock::optional_HeaderBBox_bbox:
switch (pbf_header_block.tag_and_type()) {
case protozero::tag_and_type(OSMFormat::HeaderBlock::optional_HeaderBBox_bbox, protozero::pbf_wire_type::length_delimited):
header.add_box(decode_header_bbox(pbf_header_block.get_view()));
break;
case OSMFormat::HeaderBlock::repeated_string_required_features:
case protozero::tag_and_type(OSMFormat::HeaderBlock::repeated_string_required_features, protozero::pbf_wire_type::length_delimited):
{
auto feature = pbf_header_block.get_view();
if (!std::strncmp("OsmSchema-V0.6", feature.data(), feature.size())) {
@@ -825,23 +840,23 @@ namespace osmium {
}
}
break;
case OSMFormat::HeaderBlock::repeated_string_optional_features:
case protozero::tag_and_type(OSMFormat::HeaderBlock::repeated_string_optional_features, protozero::pbf_wire_type::length_delimited):
header.set("pbf_optional_feature_" + std::to_string(i++), pbf_header_block.get_string());
break;
case OSMFormat::HeaderBlock::optional_string_writingprogram:
case protozero::tag_and_type(OSMFormat::HeaderBlock::optional_string_writingprogram, protozero::pbf_wire_type::length_delimited):
header.set("generator", pbf_header_block.get_string());
break;
case OSMFormat::HeaderBlock::optional_int64_osmosis_replication_timestamp:
case protozero::tag_and_type(OSMFormat::HeaderBlock::optional_int64_osmosis_replication_timestamp, protozero::pbf_wire_type::varint):
{
const auto timestamp = osmium::Timestamp{pbf_header_block.get_int64()}.to_iso();
header.set("osmosis_replication_timestamp", timestamp);
header.set("timestamp", timestamp);
}
break;
case OSMFormat::HeaderBlock::optional_int64_osmosis_replication_sequence_number:
case protozero::tag_and_type(OSMFormat::HeaderBlock::optional_int64_osmosis_replication_sequence_number, protozero::pbf_wire_type::varint):
header.set("osmosis_replication_sequence_number", std::to_string(pbf_header_block.get_int64()));
break;
case OSMFormat::HeaderBlock::optional_string_osmosis_replication_base_url:
case protozero::tag_and_type(OSMFormat::HeaderBlock::optional_string_osmosis_replication_base_url, protozero::pbf_wire_type::length_delimited):
header.set("osmosis_replication_base_url", pbf_header_block.get_string());
break;
default:
@@ -879,14 +894,6 @@ namespace osmium {
m_read_metadata(read_metadata) {
}
PBFDataBlobDecoder(const PBFDataBlobDecoder&) = default;
PBFDataBlobDecoder& operator=(const PBFDataBlobDecoder&) = default;
PBFDataBlobDecoder(PBFDataBlobDecoder&&) = default;
PBFDataBlobDecoder& operator=(PBFDataBlobDecoder&&) = default;
~PBFDataBlobDecoder() noexcept = default;
osmium::memory::Buffer operator()() {
std::string output;
PBFPrimitiveBlockDecoder decoder{decode_blob(*m_input_buffer, output), m_read_types, m_read_metadata};
include/osmium/io/detail/pbf_input_format.hpp
+31 -20
View File
@@ -5,7 +5,7 @@
This file is part of Osmium (http://osmcode.org/libosmium).
Copyright 2013-2017 Jochen Topf <jochen@topf.org> and others (see README).
Copyright 2013-2018 Jochen Topf <jochen@topf.org> and others (see README).
Boost Software License - Version 1.0 - August 17th, 2003
@@ -33,18 +33,6 @@ DEALINGS IN THE SOFTWARE.
*/
#include <cassert>
#include <cstddef>
#include <cstdint>
#include <cstring>
#include <memory>
#include <string>
#include <type_traits>
#include <utility>
#include <protozero/pbf_message.hpp>
#include <protozero/types.hpp>
#include <osmium/io/detail/input_format.hpp>
#include <osmium/io/detail/pbf.hpp> // IWYU pragma: export
#include <osmium/io/detail/pbf_decoder.hpp>
@@ -56,6 +44,18 @@ DEALINGS IN THE SOFTWARE.
#include <osmium/thread/util.hpp>
#include <osmium/util/config.hpp>
#include <protozero/pbf_message.hpp>
#include <protozero/types.hpp>
#include <cassert>
#include <cstddef>
#include <cstdint>
#include <cstring>
#include <memory>
#include <string>
#include <type_traits>
#include <utility>
namespace osmium {
namespace io {
@@ -64,7 +64,7 @@ namespace osmium {
class PBFParser : public Parser {
std::string m_input_buffer;
std::string m_input_buffer{};
/**
* Read the given number of bytes from the input queue.
@@ -105,7 +105,13 @@ namespace osmium {
return 0; // EOF
}
#ifndef _WIN32
const uint32_t size = ntohl(size_in_network_byte_order);
#else
uint32_t size = size_in_network_byte_order;
protozero::detail::byteswap_inplace(&size);
#endif
if (size > static_cast<uint32_t>(max_blob_header_size)) {
throw osmium::pbf_error{"invalid BlobHeader size (> max_blob_header_size)"};
}
@@ -122,11 +128,11 @@ namespace osmium {
size_t blob_header_datasize = 0;
while (pbf_blob_header.next()) {
switch (pbf_blob_header.tag()) {
case FileFormat::BlobHeader::required_string_type:
switch (pbf_blob_header.tag_and_type()) {
case protozero::tag_and_type(FileFormat::BlobHeader::required_string_type, protozero::pbf_wire_type::length_delimited):
blob_header_type = pbf_blob_header.get_view();
break;
case FileFormat::BlobHeader::required_int32_datasize:
case protozero::tag_and_type(FileFormat::BlobHeader::required_int32_datasize, protozero::pbf_wire_type::varint):
blob_header_datasize = pbf_blob_header.get_int32();
break;
default:
@@ -138,7 +144,7 @@ namespace osmium {
throw osmium::pbf_error{"PBF format error: BlobHeader.datasize missing or zero."};
}
if (std::strncmp(expected_type, blob_header_type.data(), blob_header_type.size())) {
if (std::strncmp(expected_type, blob_header_type.data(), blob_header_type.size()) != 0) {
throw osmium::pbf_error{"blob does not have expected type (OSMHeader in first blob, OSMData in following blobs)"};
}
@@ -190,10 +196,15 @@ namespace osmium {
public:
explicit PBFParser(parser_arguments& args) :
Parser(args),
m_input_buffer() {
Parser(args) {
}
PBFParser(const PBFParser&) = delete;
PBFParser& operator=(const PBFParser&) = delete;
PBFParser(PBFParser&&) = delete;
PBFParser& operator=(PBFParser&&) = delete;
~PBFParser() noexcept final = default;
void run() final {
include/osmium/io/detail/pbf_output_format.hpp
+101 -72
View File
@@ -5,7 +5,7 @@
This file is part of Osmium (http://osmcode.org/libosmium).
Copyright 2013-2017 Jochen Topf <jochen@topf.org> and others (see README).
Copyright 2013-2018 Jochen Topf <jochen@topf.org> and others (see README).
Boost Software License - Version 1.0 - August 17th, 2003
@@ -33,19 +33,6 @@ DEALINGS IN THE SOFTWARE.
*/
#include <cassert>
#include <cmath>
#include <cstdint>
#include <cstdlib>
#include <memory>
#include <string>
#include <utility>
#include <vector>
#include <protozero/pbf_builder.hpp>
#include <protozero/pbf_writer.hpp>
#include <protozero/types.hpp>
#include <osmium/handler.hpp>
#include <osmium/io/detail/output_format.hpp>
#include <osmium/io/detail/pbf.hpp> // IWYU pragma: export
@@ -61,6 +48,7 @@ DEALINGS IN THE SOFTWARE.
#include <osmium/osm/box.hpp>
#include <osmium/osm/item_type.hpp>
#include <osmium/osm/location.hpp>
#include <osmium/osm/metadata_options.hpp>
#include <osmium/osm/node.hpp>
#include <osmium/osm/node_ref.hpp>
#include <osmium/osm/object.hpp>
@@ -72,8 +60,22 @@ DEALINGS IN THE SOFTWARE.
#include <osmium/thread/pool.hpp>
#include <osmium/util/cast.hpp>
#include <osmium/util/delta.hpp>
#include <osmium/util/misc.hpp>
#include <osmium/visitor.hpp>
#include <protozero/pbf_builder.hpp>
#include <protozero/pbf_writer.hpp>
#include <protozero/types.hpp>
#include <cassert>
#include <cmath>
#include <cstdint>
#include <cstdlib>
#include <memory>
#include <string>
#include <utility>
#include <vector>
namespace osmium {
namespace io {
@@ -82,8 +84,11 @@ namespace osmium {
struct pbf_output_options {
/// Which metadata of objects should be added?
osmium::metadata_options add_metadata;
/// Should nodes be encoded in DenseNodes?
bool use_dense_nodes;
bool use_dense_nodes = true;
/**
* Should the PBF blobs contain zlib compressed data?
@@ -93,21 +98,18 @@ namespace osmium {
* the writing speed a little but the output will be 2x to 3x
* bigger.
*/
bool use_compression;
/// Should metadata of objects be written?
bool add_metadata;
bool use_compression = true;
/// Add the "HistoricalInformation" header flag.
bool add_historical_information_flag;
bool add_historical_information_flag = false;
/// Should the visible flag be added to all OSM objects?
bool add_visible_flag;
bool add_visible_flag = false;
/// Should node locations be added to ways?
bool locations_on_ways;
bool locations_on_ways = false;
};
}; // struct pbf_output_options
/**
* Maximum number of items in a primitive block.
@@ -185,7 +187,12 @@ namespace osmium {
pbf_blob_header.add_string(FileFormat::BlobHeader::required_string_type, m_blob_type == pbf_blob_type::data ? "OSMData" : "OSMHeader");
pbf_blob_header.add_int32(FileFormat::BlobHeader::required_int32_datasize, static_cast_with_assert<int32_t>(blob_data.size()));
#ifndef _WIN32
const uint32_t sz = htonl(static_cast_with_assert<uint32_t>(blob_header_data.size()));
#else
uint32_t sz = static_cast_with_assert<uint32_t>(blob_header_data.size());
protozero::detail::byteswap_inplace(&sz);
#endif
// write to output: the 4-byte BlobHeader-Size followed by the BlobHeader followed by the Blob
std::string output;
@@ -224,15 +231,15 @@ namespace osmium {
std::vector<int64_t> m_lons;
std::vector<int32_t> m_tags;
osmium::util::DeltaEncode<object_id_type, int64_t> m_delta_id;
osmium::DeltaEncode<object_id_type, int64_t> m_delta_id;
osmium::util::DeltaEncode<uint32_t, int64_t> m_delta_timestamp;
osmium::util::DeltaEncode<changeset_id_type, int64_t> m_delta_changeset;
osmium::util::DeltaEncode<user_id_type, int32_t> m_delta_uid;
osmium::util::DeltaEncode<uint32_t, int32_t> m_delta_user_sid;
osmium::DeltaEncode<uint32_t, int64_t> m_delta_timestamp;
osmium::DeltaEncode<changeset_id_type, int64_t> m_delta_changeset;
osmium::DeltaEncode<user_id_type, int32_t> m_delta_uid;
osmium::DeltaEncode<uint32_t, int32_t> m_delta_user_sid;
osmium::util::DeltaEncode<int64_t, int64_t> m_delta_lat;
osmium::util::DeltaEncode<int64_t, int64_t> m_delta_lon;
osmium::DeltaEncode<int64_t, int64_t> m_delta_lat;
osmium::DeltaEncode<int64_t, int64_t> m_delta_lon;
const pbf_output_options& m_options;
@@ -269,22 +276,30 @@ namespace osmium {
m_delta_lon.clear();
}
std::size_t size() const {
std::size_t size() const noexcept {
return m_ids.size() * 3 * sizeof(int64_t);
}
void add_node(const osmium::Node& node) {
m_ids.push_back(m_delta_id.update(node.id()));
if (m_options.add_metadata) {
if (m_options.add_metadata.version()) {
m_versions.push_back(static_cast_with_assert<int32_t>(node.version()));
}
if (m_options.add_metadata.timestamp()) {
m_timestamps.push_back(m_delta_timestamp.update(uint32_t(node.timestamp())));
}
if (m_options.add_metadata.changeset()) {
m_changesets.push_back(m_delta_changeset.update(node.changeset()));
}
if (m_options.add_metadata.uid()) {
m_uids.push_back(m_delta_uid.update(node.uid()));
}
if (m_options.add_metadata.user()) {
m_user_sids.push_back(m_delta_user_sid.update(m_stringtable.add(node.user())));
if (m_options.add_visible_flag) {
m_visibles.push_back(node.visible());
}
}
if (m_options.add_visible_flag) {
m_visibles.push_back(node.visible());
}
m_lats.push_back(m_delta_lat.update(lonlat2int(node.location().lat_without_check())));
@@ -303,14 +318,23 @@ namespace osmium {
pbf_dense_nodes.add_packed_sint64(OSMFormat::DenseNodes::packed_sint64_id, m_ids.cbegin(), m_ids.cend());
if (m_options.add_metadata) {
if (m_options.add_metadata.any() || m_options.add_visible_flag) {
protozero::pbf_builder<OSMFormat::DenseInfo> pbf_dense_info{pbf_dense_nodes, OSMFormat::DenseNodes::optional_DenseInfo_denseinfo};
pbf_dense_info.add_packed_int32(OSMFormat::DenseInfo::packed_int32_version, m_versions.cbegin(), m_versions.cend());
pbf_dense_info.add_packed_sint64(OSMFormat::DenseInfo::packed_sint64_timestamp, m_timestamps.cbegin(), m_timestamps.cend());
pbf_dense_info.add_packed_sint64(OSMFormat::DenseInfo::packed_sint64_changeset, m_changesets.cbegin(), m_changesets.cend());
pbf_dense_info.add_packed_sint32(OSMFormat::DenseInfo::packed_sint32_uid, m_uids.cbegin(), m_uids.cend());
pbf_dense_info.add_packed_sint32(OSMFormat::DenseInfo::packed_sint32_user_sid, m_user_sids.cbegin(), m_user_sids.cend());
if (m_options.add_metadata.version()) {
pbf_dense_info.add_packed_int32(OSMFormat::DenseInfo::packed_int32_version, m_versions.cbegin(), m_versions.cend());
}
if (m_options.add_metadata.timestamp()) {
pbf_dense_info.add_packed_sint64(OSMFormat::DenseInfo::packed_sint64_timestamp, m_timestamps.cbegin(), m_timestamps.cend());
}
if (m_options.add_metadata.changeset()) {
pbf_dense_info.add_packed_sint64(OSMFormat::DenseInfo::packed_sint64_changeset, m_changesets.cbegin(), m_changesets.cend());
}
if (m_options.add_metadata.uid()) {
pbf_dense_info.add_packed_sint32(OSMFormat::DenseInfo::packed_sint32_uid, m_uids.cbegin(), m_uids.cend());
}
if (m_options.add_metadata.user()) {
pbf_dense_info.add_packed_sint32(OSMFormat::DenseInfo::packed_sint32_user_sid, m_user_sids.cbegin(), m_user_sids.cend());
}
if (m_options.add_visible_flag) {
pbf_dense_info.add_packed_bool(OSMFormat::DenseInfo::packed_bool_visible, m_visibles.cbegin(), m_visibles.cend());
}
@@ -332,18 +356,14 @@ namespace osmium {
protozero::pbf_builder<OSMFormat::PrimitiveGroup> m_pbf_primitive_group;
StringTable m_stringtable;
DenseNodes m_dense_nodes;
OSMFormat::PrimitiveGroup m_type;
int m_count;
OSMFormat::PrimitiveGroup m_type = OSMFormat::PrimitiveGroup::unknown;
int m_count = 0;
public:
explicit PrimitiveBlock(const pbf_output_options& options) :
m_pbf_primitive_group_data(),
m_pbf_primitive_group(m_pbf_primitive_group_data),
m_stringtable(),
m_dense_nodes(m_stringtable, options),
m_type(OSMFormat::PrimitiveGroup::unknown),
m_count(0) {
m_dense_nodes(m_stringtable, options) {
}
const std::string& group_data() {
@@ -367,7 +387,7 @@ namespace osmium {
}
}
protozero::pbf_builder<OSMFormat::PrimitiveGroup>& group() {
protozero::pbf_builder<OSMFormat::PrimitiveGroup>& group() noexcept {
++m_count;
return m_pbf_primitive_group;
}
@@ -381,15 +401,15 @@ namespace osmium {
return m_stringtable.add(s);
}
int count() const {
int count() const noexcept {
return m_count;
}
OSMFormat::PrimitiveGroup type() const {
OSMFormat::PrimitiveGroup type() const noexcept {
return m_type;
}
std::size_t size() const {
std::size_t size() const noexcept {
return m_pbf_primitive_group_data.size() + m_stringtable.size() + m_dense_nodes.size();
}
@@ -401,7 +421,7 @@ namespace osmium {
*/
constexpr static std::size_t max_used_blob_size = max_uncompressed_blob_size * 95 / 100;
bool can_add(OSMFormat::PrimitiveGroup type) const {
bool can_add(OSMFormat::PrimitiveGroup type) const noexcept {
if (type != m_type) {
return false;
}
@@ -457,14 +477,24 @@ namespace osmium {
}
}
if (m_options.add_metadata) {
if (m_options.add_metadata.any() || m_options.add_visible_flag) {
protozero::pbf_builder<OSMFormat::Info> pbf_info{pbf_object, T::enum_type::optional_Info_info};
pbf_info.add_int32(OSMFormat::Info::optional_int32_version, static_cast_with_assert<int32_t>(object.version()));
pbf_info.add_int64(OSMFormat::Info::optional_int64_timestamp, uint32_t(object.timestamp()));
pbf_info.add_int64(OSMFormat::Info::optional_int64_changeset, object.changeset());
pbf_info.add_int32(OSMFormat::Info::optional_int32_uid, static_cast_with_assert<int32_t>(object.uid()));
pbf_info.add_uint32(OSMFormat::Info::optional_uint32_user_sid, m_primitive_block.store_in_stringtable(object.user()));
if (m_options.add_metadata.version()) {
pbf_info.add_int32(OSMFormat::Info::optional_int32_version, static_cast_with_assert<int32_t>(object.version()));
}
if (m_options.add_metadata.timestamp()) {
pbf_info.add_int64(OSMFormat::Info::optional_int64_timestamp, uint32_t(object.timestamp()));
}
if (m_options.add_metadata.changeset()) {
pbf_info.add_int64(OSMFormat::Info::optional_int64_changeset, object.changeset());
}
if (m_options.add_metadata.uid()) {
pbf_info.add_int32(OSMFormat::Info::optional_int32_uid, static_cast_with_assert<int32_t>(object.uid()));
}
if (m_options.add_metadata.user()) {
pbf_info.add_uint32(OSMFormat::Info::optional_uint32_user_sid, m_primitive_block.store_in_stringtable(object.user()));
}
if (m_options.add_visible_flag) {
pbf_info.add_bool(OSMFormat::Info::optional_bool_visible, object.visible());
}
@@ -482,21 +512,20 @@ namespace osmium {
PBFOutputFormat(osmium::thread::Pool& pool, const osmium::io::File& file, future_string_queue_type& output_queue) :
OutputFormat(pool, output_queue),
m_options(),
m_primitive_block(m_options) {
if (!file.get("pbf_add_metadata").empty()) {
throw std::invalid_argument{"The 'pbf_add_metadata' option is deprecated. Please use 'add_metadata' instead."};
}
m_options.use_dense_nodes = file.is_not_false("pbf_dense_nodes");
m_options.use_compression = file.get("pbf_compression") != "none" && file.is_not_false("pbf_compression");
m_options.add_metadata = file.is_not_false("pbf_add_metadata") && file.is_not_false("add_metadata");
m_options.add_metadata = osmium::metadata_options{file.get("add_metadata")};
m_options.add_historical_information_flag = file.has_multiple_object_versions();
m_options.add_visible_flag = file.has_multiple_object_versions();
m_options.locations_on_ways = file.is_true("locations_on_ways");
}
PBFOutputFormat(const PBFOutputFormat&) = delete;
PBFOutputFormat& operator=(const PBFOutputFormat&) = delete;
~PBFOutputFormat() noexcept final = default;
void write_header(const osmium::io::Header& header) final {
std::string data;
protozero::pbf_builder<OSMFormat::HeaderBlock> pbf_header_block{data};
@@ -535,7 +564,7 @@ namespace osmium {
const std::string osmosis_replication_sequence_number{header.get("osmosis_replication_sequence_number")};
if (!osmosis_replication_sequence_number.empty()) {
pbf_header_block.add_int64(OSMFormat::HeaderBlock::optional_int64_osmosis_replication_sequence_number, std::atoll(osmosis_replication_sequence_number.c_str()));
pbf_header_block.add_int64(OSMFormat::HeaderBlock::optional_int64_osmosis_replication_sequence_number, osmium::detail::str_to_int<int64_t>(osmosis_replication_sequence_number.c_str()));
}
const std::string osmosis_replication_base_url{header.get("osmosis_replication_base_url")};
@@ -583,7 +612,7 @@ namespace osmium {
add_meta(way, pbf_way);
{
osmium::util::DeltaEncode<object_id_type, int64_t> delta_id;
osmium::DeltaEncode<object_id_type, int64_t> delta_id;
protozero::packed_field_sint64 field{pbf_way, protozero::pbf_tag_type(OSMFormat::Way::packed_sint64_refs)};
for (const auto& node_ref : way.nodes()) {
field.add_element(delta_id.update(node_ref.ref()));
@@ -592,14 +621,14 @@ namespace osmium {
if (m_options.locations_on_ways) {
{
osmium::util::DeltaEncode<int64_t, int64_t> delta_id;
osmium::DeltaEncode<int64_t, int64_t> delta_id;
protozero::packed_field_sint64 field{pbf_way, protozero::pbf_tag_type(OSMFormat::Way::packed_sint64_lon)};
for (const auto& node_ref : way.nodes()) {
field.add_element(delta_id.update(lonlat2int(node_ref.location().lon_without_check())));
}
}
{
osmium::util::DeltaEncode<int64_t, int64_t> delta_id;
osmium::DeltaEncode<int64_t, int64_t> delta_id;
protozero::packed_field_sint64 field{pbf_way, protozero::pbf_tag_type(OSMFormat::Way::packed_sint64_lat)};
for (const auto& node_ref : way.nodes()) {
field.add_element(delta_id.update(lonlat2int(node_ref.location().lat_without_check())));
@@ -623,7 +652,7 @@ namespace osmium {
}
{
osmium::util::DeltaEncode<object_id_type, int64_t> delta_id;
osmium::DeltaEncode<object_id_type, int64_t> delta_id;
protozero::packed_field_sint64 field{pbf_relation, protozero::pbf_tag_type(OSMFormat::Relation::packed_sint64_memids)};
for (const auto& member : relation.members()) {
field.add_element(delta_id.update(member.ref()));
include/osmium/io/detail/protobuf_tags.hpp
+1 -1
View File
@@ -5,7 +5,7 @@
This file is part of Osmium (http://osmcode.org/libosmium).
Copyright 2013-2017 Jochen Topf <jochen@topf.org> and others (see README).
Copyright 2013-2018 Jochen Topf <jochen@topf.org> and others (see README).
Boost Software License - Version 1.0 - August 17th, 2003
include/osmium/io/detail/queue_util.hpp
+10 -4
View File
@@ -5,7 +5,7 @@
This file is part of Osmium (http://osmcode.org/libosmium).
Copyright 2013-2017 Jochen Topf <jochen@topf.org> and others (see README).
Copyright 2013-2018 Jochen Topf <jochen@topf.org> and others (see README).
Boost Software License - Version 1.0 - August 17th, 2003
@@ -33,15 +33,15 @@ DEALINGS IN THE SOFTWARE.
*/
#include <osmium/memory/buffer.hpp>
#include <osmium/thread/queue.hpp>
#include <cassert>
#include <exception>
#include <future>
#include <string>
#include <utility>
#include <osmium/memory/buffer.hpp>
#include <osmium/thread/queue.hpp>
namespace osmium {
namespace io {
@@ -110,6 +110,12 @@ namespace osmium {
m_has_reached_end_of_data(false) {
}
queue_wrapper(const queue_wrapper&) = delete;
queue_wrapper& operator=(const queue_wrapper&) = delete;
queue_wrapper(queue_wrapper&&) = delete;
queue_wrapper& operator=(queue_wrapper&&) = delete;
~queue_wrapper() noexcept {
drain();
}
include/osmium/io/detail/read_thread.hpp
+5 -5
View File
@@ -5,7 +5,7 @@
This file is part of Osmium (http://osmcode.org/libosmium).
Copyright 2013-2017 Jochen Topf <jochen@topf.org> and others (see README).
Copyright 2013-2018 Jochen Topf <jochen@topf.org> and others (see README).
Boost Software License - Version 1.0 - August 17th, 2003
@@ -33,16 +33,16 @@ DEALINGS IN THE SOFTWARE.
*/
#include <osmium/io/compression.hpp>
#include <osmium/io/detail/queue_util.hpp>
#include <osmium/thread/util.hpp>
#include <atomic>
#include <exception>
#include <string>
#include <thread>
#include <utility>
#include <osmium/io/compression.hpp>
#include <osmium/io/detail/queue_util.hpp>
#include <osmium/thread/util.hpp>
namespace osmium {
namespace io {
include/osmium/io/detail/read_write.hpp
+38 -10
View File
@@ -5,7 +5,7 @@
This file is part of Osmium (http://osmcode.org/libosmium).
Copyright 2013-2017 Jochen Topf <jochen@topf.org> and others (see README).
Copyright 2013-2018 Jochen Topf <jochen@topf.org> and others (see README).
Boost Software License - Version 1.0 - August 17th, 2003
@@ -33,6 +33,8 @@ DEALINGS IN THE SOFTWARE.
*/
#include <osmium/io/writer_options.hpp>
#include <cerrno>
#include <cstddef>
#include <fcntl.h>
@@ -45,8 +47,6 @@ DEALINGS IN THE SOFTWARE.
# include <io.h>
#endif
#include <osmium/io/writer_options.hpp>
namespace osmium {
namespace io {
@@ -67,14 +67,14 @@ namespace osmium {
* @throws system_error if the file can't be opened.
*/
inline int open_for_writing(const std::string& filename, osmium::io::overwrite allow_overwrite = osmium::io::overwrite::no) {
if (filename == "" || filename == "-") {
if (filename.empty() || filename == "-") {
#ifdef _WIN32
_setmode(1, _O_BINARY);
#endif
return 1; // stdout
}
int flags = O_WRONLY | O_CREAT;
int flags = O_WRONLY | O_CREAT; // NOLINT(hicpp-signed-bitwise)
if (allow_overwrite == osmium::io::overwrite::allow) {
flags |= O_TRUNC;
} else {
@@ -99,7 +99,7 @@ namespace osmium {
* @throws system_error if the file can't be opened.
*/
inline int open_for_reading(const std::string& filename) {
if (filename == "" || filename == "-") {
if (filename.empty() || filename == "-") {
return 0; // stdin
}
@@ -132,10 +132,14 @@ namespace osmium {
if (write_count > max_write) {
write_count = max_write;
}
const auto length = ::write(fd, output_buffer + offset, static_cast<unsigned int>(write_count));
if (length < 0) {
throw std::system_error{errno, std::system_category(), "Write failed"};
}
int64_t length = 0;
do {
length = ::write(fd, output_buffer + offset, static_cast<unsigned int>(write_count));
if (length < 0 && errno != EINTR) {
throw std::system_error{errno, std::system_category(), "Write failed"};
}
} while (length < 0);
offset += static_cast<size_t>(length);
} while (offset < size);
}
@@ -154,6 +158,30 @@ namespace osmium {
reliable_write(fd, reinterpret_cast<const unsigned char*>(output_buffer), size);
}
/**
* Reads a maximum of size bytes from the file descriptor into the
* input_buffer. This is just a wrapper around read(2) catching
* errors.
*
* @param fd File descriptor.
* @param input_buffer Buffer for data to be read. Must be at least size bytes long.
* @param size Maximum number of bytes to read.
* @returns the number of bytes read
* @throws std::system_error On error.
*/
inline int64_t reliable_read(const int fd, char* input_buffer, const unsigned int size) {
int64_t nread = 0;
do {
nread = ::read(fd, input_buffer, size);
if (nread < 0 && errno != EINTR) {
throw std::system_error{errno, std::system_category(), "Read failed"};
}
} while (nread < 0);
return nread;
}
inline void reliable_fsync(const int fd) {
#ifdef _WIN32
if (_commit(fd) != 0) {
include/osmium/io/detail/string_table.hpp
+9 -12
View File
@@ -5,7 +5,7 @@
This file is part of Osmium (http://osmcode.org/libosmium).
Copyright 2013-2017 Jochen Topf <jochen@topf.org> and others (see README).
Copyright 2013-2018 Jochen Topf <jochen@topf.org> and others (see README).
Boost Software License - Version 1.0 - August 17th, 2003
@@ -33,6 +33,8 @@ DEALINGS IN THE SOFTWARE.
*/
#include <osmium/io/detail/pbf.hpp>
#include <cassert>
#include <cstddef>
#include <cstdint>
@@ -43,8 +45,6 @@ DEALINGS IN THE SOFTWARE.
#include <unordered_map>
#include <utility>
#include <osmium/io/detail/pbf.hpp>
namespace osmium {
namespace io {
@@ -77,8 +77,7 @@ namespace osmium {
public:
explicit StringStore(size_t chunk_size) :
m_chunk_size(chunk_size),
m_chunks() {
m_chunk_size(chunk_size) {
add_chunk();
}
@@ -212,12 +211,12 @@ namespace osmium {
struct djb2_hash {
size_t operator()(const char* str) const noexcept {
size_t hash = 5381;
std::size_t operator()(const char* str) const noexcept {
std::size_t hash = 5381;
int c;
while ((c = *str++)) {
hash = ((hash << 5) + hash) + c; /* hash * 33 + c */
hash = ((hash << 5u) + hash) + c; /* hash * 33 + c */
}
return hash;
@@ -245,14 +244,12 @@ namespace osmium {
StringStore m_strings;
std::unordered_map<const char*, size_t, djb2_hash, str_equal> m_index;
uint32_t m_size;
uint32_t m_size = 0;
public:
explicit StringTable(size_t size = default_stringtable_chunk_size) :
m_strings(size),
m_index(),
m_size(0) {
m_strings(size) {
m_strings.add("");
}
include/osmium/io/detail/string_util.hpp
+13 -13
View File
@@ -5,7 +5,7 @@
This file is part of Osmium (http://osmcode.org/libosmium).
Copyright 2013-2017 Jochen Topf <jochen@topf.org> and others (see README).
Copyright 2013-2018 Jochen Topf <jochen@topf.org> and others (see README).
Boost Software License - Version 1.0 - August 17th, 2003
@@ -33,6 +33,8 @@ DEALINGS IN THE SOFTWARE.
*/
#include <utf8.h>
#include <cassert>
#include <cstdint>
#include <cstdio>
@@ -40,8 +42,6 @@ DEALINGS IN THE SOFTWARE.
#include <string>
#include <utility>
#include <utf8.h>
namespace osmium {
namespace io {
@@ -127,22 +127,22 @@ namespace osmium {
// Write out the value with exactly two hex digits.
inline void append_2_hex_digits(std::string& out, uint32_t value, const char* const hex_digits) {
out += hex_digits[(value >> 4) & 0xf];
out += hex_digits[ value & 0xf];
out += hex_digits[(value >> 4u) & 0xfu];
out += hex_digits[ value & 0xfu];
}
// Write out the value with four or more hex digits.
inline void append_min_4_hex_digits(std::string& out, uint32_t value, const char* const hex_digits) {
auto
v = value & 0xf0000000; if (v) { out += hex_digits[v >> 28]; }
v = value & 0x0f000000; if (v) { out += hex_digits[v >> 24]; }
v = value & 0x00f00000; if (v) { out += hex_digits[v >> 20]; }
v = value & 0x000f0000; if (v) { out += hex_digits[v >> 16]; }
v = value & 0xf0000000u; if (v) { out += hex_digits[v >> 28u]; }
v = value & 0x0f000000u; if (v) { out += hex_digits[v >> 24u]; }
v = value & 0x00f00000u; if (v) { out += hex_digits[v >> 20u]; }
v = value & 0x000f0000u; if (v) { out += hex_digits[v >> 16u]; }
out += hex_digits[(value >> 12) & 0xf];
out += hex_digits[(value >> 8) & 0xf];
out += hex_digits[(value >> 4) & 0xf];
out += hex_digits[ value & 0xf];
out += hex_digits[(value >> 12u) & 0xfu];
out += hex_digits[(value >> 8u) & 0xfu];
out += hex_digits[(value >> 4u) & 0xfu];
out += hex_digits[ value & 0xfu];
}
inline void append_utf8_encoded_string(std::string& out, const char* data) {
include/osmium/io/detail/write_thread.hpp
+5 -5
View File
@@ -5,7 +5,7 @@
This file is part of Osmium (http://osmcode.org/libosmium).
Copyright 2013-2017 Jochen Topf <jochen@topf.org> and others (see README).
Copyright 2013-2018 Jochen Topf <jochen@topf.org> and others (see README).
Boost Software License - Version 1.0 - August 17th, 2003
@@ -33,16 +33,16 @@ DEALINGS IN THE SOFTWARE.
*/
#include <osmium/io/compression.hpp>
#include <osmium/io/detail/queue_util.hpp>
#include <osmium/thread/util.hpp>
#include <exception>
#include <future>
#include <memory>
#include <string>
#include <utility>
#include <osmium/io/compression.hpp>
#include <osmium/io/detail/queue_util.hpp>
#include <osmium/thread/util.hpp>
namespace osmium {
namespace io {
include/osmium/io/detail/xml_input_format.hpp
+375 -302
View File
@@ -5,7 +5,7 @@
This file is part of Osmium (http://osmcode.org/libosmium).
Copyright 2013-2017 Jochen Topf <jochen@topf.org> and others (see README).
Copyright 2013-2018 Jochen Topf <jochen@topf.org> and others (see README).
Boost Software License - Version 1.0 - August 17th, 2003
@@ -33,15 +33,6 @@ DEALINGS IN THE SOFTWARE.
*/
#include <cassert>
#include <cstring>
#include <future>
#include <memory>
#include <string>
#include <utility>
#include <expat.h>
#include <osmium/builder/builder.hpp>
#include <osmium/builder/osm_object_builder.hpp>
#include <osmium/io/detail/input_format.hpp>
@@ -64,7 +55,16 @@ DEALINGS IN THE SOFTWARE.
#include <osmium/osm/types_from_string.hpp>
#include <osmium/osm/way.hpp>
#include <osmium/thread/util.hpp>
#include <osmium/util/cast.hpp>
#include <expat.h>
#include <cassert>
#include <cstring>
#include <future>
#include <limits>
#include <memory>
#include <string>
#include <utility>
namespace osmium {
@@ -75,8 +75,8 @@ namespace osmium {
*/
struct xml_error : public io_error {
unsigned long line;
unsigned long column;
uint64_t line = 0;
uint64_t column = 0;
XML_Error error_code;
std::string error_string;
@@ -95,8 +95,6 @@ namespace osmium {
explicit xml_error(const std::string& message) :
io_error(message),
line(0),
column(0),
error_code(),
error_string(message) {
}
@@ -112,8 +110,7 @@ namespace osmium {
std::string version;
explicit format_version_error() :
io_error("Can not read file without version (missing version attribute on osm element)."),
version() {
io_error("Can not read file without version (missing version attribute on osm element).") {
}
explicit format_version_error(const char* v) :
@@ -129,54 +126,50 @@ namespace osmium {
class XMLParser : public Parser {
static constexpr int buffer_size = 2 * 1000 * 1000;
static constexpr std::size_t buffer_size = 2 * 1000 * 1000;
enum class context {
root,
top,
osm,
osmChange,
bounds,
create_section,
modify_section,
delete_section,
node,
way,
relation,
tag,
nd,
member,
changeset,
discussion,
comment,
comment_text,
ignored_node,
ignored_way,
ignored_relation,
ignored_changeset,
in_object
text,
other
}; // enum class context
context m_context;
context m_last_context;
std::vector<context> m_context_stack;
/**
* This is used only for change files which contain create, modify,
* and delete sections.
*/
bool m_in_delete_section;
osmium::io::Header m_header;
osmium::io::Header m_header{};
osmium::memory::Buffer m_buffer;
std::unique_ptr<osmium::builder::NodeBuilder> m_node_builder;
std::unique_ptr<osmium::builder::WayBuilder> m_way_builder;
std::unique_ptr<osmium::builder::RelationBuilder> m_relation_builder;
std::unique_ptr<osmium::builder::ChangesetBuilder> m_changeset_builder;
std::unique_ptr<osmium::builder::ChangesetDiscussionBuilder> m_changeset_discussion_builder;
std::unique_ptr<osmium::builder::NodeBuilder> m_node_builder{};
std::unique_ptr<osmium::builder::WayBuilder> m_way_builder{};
std::unique_ptr<osmium::builder::RelationBuilder> m_relation_builder{};
std::unique_ptr<osmium::builder::ChangesetBuilder> m_changeset_builder{};
std::unique_ptr<osmium::builder::ChangesetDiscussionBuilder> m_changeset_discussion_builder{};
std::unique_ptr<osmium::builder::TagListBuilder> m_tl_builder;
std::unique_ptr<osmium::builder::WayNodeListBuilder> m_wnl_builder;
std::unique_ptr<osmium::builder::RelationMemberListBuilder> m_rml_builder;
std::unique_ptr<osmium::builder::TagListBuilder> m_tl_builder{};
std::unique_ptr<osmium::builder::WayNodeListBuilder> m_wnl_builder{};
std::unique_ptr<osmium::builder::RelationMemberListBuilder> m_rml_builder{};
std::string m_comment_text;
/**
* A C++ wrapper for the Expat parser that makes sure no memory is leaked.
* A C++ wrapper for the Expat parser that makes sure no memory
* is leaked.
*/
template <typename T>
class ExpatXMLParser {
XML_Parser m_parser;
@@ -198,15 +191,21 @@ namespace osmium {
// but they can be misused. See
// https://en.wikipedia.org/wiki/Billion_laughs
// The handler will just throw an error.
static void entity_declaration_handler(void*,
const XML_Char*, int, const XML_Char*, int, const XML_Char*,
const XML_Char*, const XML_Char*, const XML_Char*) {
static void entity_declaration_handler(void* /*userData*/,
const XML_Char* /*entityName*/,
int /*is_parameter_entity*/,
const XML_Char* /*value*/,
int /*value_length*/,
const XML_Char* /*base*/,
const XML_Char* /*systemId*/,
const XML_Char* /*publicId*/,
const XML_Char* /*notationName*/) {
throw osmium::xml_error{"XML entities are not supported"};
}
public:
explicit ExpatXMLParser(T* callback_object) :
explicit ExpatXMLParser(void* callback_object) :
m_parser(XML_ParserCreate(nullptr)) {
if (!m_parser) {
throw osmium::io_error{"Internal error: Can not create parser"};
@@ -218,9 +217,9 @@ namespace osmium {
}
ExpatXMLParser(const ExpatXMLParser&) = delete;
ExpatXMLParser(ExpatXMLParser&&) = delete;
ExpatXMLParser& operator=(const ExpatXMLParser&) = delete;
ExpatXMLParser(ExpatXMLParser&&) = delete;
ExpatXMLParser& operator=(ExpatXMLParser&&) = delete;
~ExpatXMLParser() noexcept {
@@ -228,7 +227,8 @@ namespace osmium {
}
void operator()(const std::string& data, bool last) {
if (XML_Parse(m_parser, data.data(), static_cast_with_assert<int>(data.size()), last) == XML_STATUS_ERROR) {
assert(data.size() < std::numeric_limits<int>::max());
if (XML_Parse(m_parser, data.data(), static_cast<int>(data.size()), last) == XML_STATUS_ERROR) {
throw osmium::xml_error{m_parser};
}
}
@@ -236,21 +236,21 @@ namespace osmium {
}; // class ExpatXMLParser
template <typename T>
static void check_attributes(const XML_Char** attrs, T check) {
static void check_attributes(const XML_Char** attrs, T&& check) {
while (*attrs) {
check(attrs[0], attrs[1]);
std::forward<T>(check)(attrs[0], attrs[1]);
attrs += 2;
}
}
const char* init_object(osmium::OSMObject& object, const XML_Char** attrs) {
const char* user = "";
if (m_in_delete_section) {
assert(m_context_stack.size() > 1);
if (m_context_stack[m_context_stack.size() - 2] == context::delete_section) {
object.set_visible(false);
}
osmium::Location location;
const char* user = "";
check_attributes(attrs, [&location, &user, &object](const XML_Char* name, const XML_Char* value) {
if (!std::strcmp(name, "lon")) {
@@ -298,9 +298,9 @@ namespace osmium {
const char* v = "";
check_attributes(attrs, [&k, &v](const XML_Char* name, const XML_Char* value) {
if (name[0] == 'k' && name[1] == 0) {
if (name[0] == 'k' && name[1] == '\0') {
k = value;
} else if (name[0] == 'v' && name[1] == 0) {
} else if (name[0] == 'v' && name[1] == '\0') {
v = value;
}
});
@@ -315,311 +315,390 @@ namespace osmium {
set_header_value(m_header);
}
void top_level_element(const XML_Char* element, const XML_Char** attrs) {
if (!std::strcmp(element, "osm")) {
m_context_stack.push_back(context::osm);
} else if (!std::strcmp(element, "osmChange")){
m_context_stack.push_back(context::osmChange);
m_header.set_has_multiple_object_versions(true);
} else {
throw osmium::xml_error{std::string{"Unknown top-level element: "} + element};
}
check_attributes(attrs, [this](const XML_Char* name, const XML_Char* value) {
if (!std::strcmp(name, "version")) {
m_header.set("version", value);
if (std::strcmp(value, "0.6") != 0) {
throw osmium::format_version_error{value};
}
} else if (!std::strcmp(name, "generator")) {
m_header.set("generator", value);
}
// ignore other attributes
});
if (m_header.get("version").empty()) {
throw osmium::format_version_error{};
}
}
void data_level_element(const XML_Char* element, const XML_Char** attrs, bool in_change_section) {
assert(!m_node_builder);
assert(!m_way_builder);
assert(!m_relation_builder);
assert(!m_changeset_builder);
assert(!m_changeset_discussion_builder);
assert(!m_tl_builder);
assert(!m_wnl_builder);
assert(!m_rml_builder);
if (!std::strcmp(element, "node")) {
m_context_stack.push_back(context::node);
mark_header_as_done();
if (read_types() & osmium::osm_entity_bits::node) {
m_node_builder.reset(new osmium::builder::NodeBuilder{m_buffer});
m_node_builder->set_user(init_object(m_node_builder->object(), attrs));
}
return;
}
if (!std::strcmp(element, "way")) {
m_context_stack.push_back(context::way);
mark_header_as_done();
if (read_types() & osmium::osm_entity_bits::way) {
m_way_builder.reset(new osmium::builder::WayBuilder{m_buffer});
m_way_builder->set_user(init_object(m_way_builder->object(), attrs));
}
return;
}
if (!std::strcmp(element, "relation")) {
m_context_stack.push_back(context::relation);
mark_header_as_done();
if (read_types() & osmium::osm_entity_bits::relation) {
m_relation_builder.reset(new osmium::builder::RelationBuilder{m_buffer});
m_relation_builder->set_user(init_object(m_relation_builder->object(), attrs));
}
return;
}
if (in_change_section) {
throw xml_error{"create/modify/delete sections can only contain nodes, ways, and relations"};
}
if (!std::strcmp(element, "changeset")) {
m_context_stack.push_back(context::changeset);
mark_header_as_done();
if (read_types() & osmium::osm_entity_bits::changeset) {
m_changeset_builder.reset(new osmium::builder::ChangesetBuilder{m_buffer});
init_changeset(*m_changeset_builder, attrs);
}
} else if (!std::strcmp(element, "create")) {
if (m_context_stack.back() != context::osmChange) {
throw xml_error{"<create> only allowed in OSM change files"};
}
m_context_stack.push_back(context::create_section);
mark_header_as_done();
} else if (!std::strcmp(element, "modify")) {
if (m_context_stack.back() != context::osmChange) {
throw xml_error{"<modify> only allowed in OSM change files"};
}
m_context_stack.push_back(context::modify_section);
mark_header_as_done();
} else if (!std::strcmp(element, "delete")) {
if (m_context_stack.back() != context::osmChange) {
throw xml_error{"<delete> only allowed in OSM change files"};
}
m_context_stack.push_back(context::delete_section);
mark_header_as_done();
} else if (!std::strcmp(element, "bounds")) {
m_context_stack.push_back(context::bounds);
osmium::Location min;
osmium::Location max;
check_attributes(attrs, [&min, &max](const XML_Char* name, const XML_Char* value) {
if (!std::strcmp(name, "minlon")) {
min.set_lon(value);
} else if (!std::strcmp(name, "minlat")) {
min.set_lat(value);
} else if (!std::strcmp(name, "maxlon")) {
max.set_lon(value);
} else if (!std::strcmp(name, "maxlat")) {
max.set_lat(value);
}
});
osmium::Box box;
box.extend(min).extend(max);
m_header.add_box(box);
} else {
m_context_stack.push_back(context::other);
}
}
void start_element(const XML_Char* element, const XML_Char** attrs) {
switch (m_context) {
case context::root:
if (!std::strcmp(element, "osm") || !std::strcmp(element, "osmChange")) {
if (!std::strcmp(element, "osmChange")) {
m_header.set_has_multiple_object_versions(true);
}
check_attributes(attrs, [this](const XML_Char* name, const XML_Char* value) {
if (!std::strcmp(name, "version")) {
m_header.set("version", value);
if (std::strcmp(value, "0.6")) {
throw osmium::format_version_error{value};
}
} else if (!std::strcmp(name, "generator")) {
m_header.set("generator", value);
}
});
if (m_header.get("version") == "") {
throw osmium::format_version_error{};
}
} else {
throw osmium::xml_error{std::string{"Unknown top-level element: "} + element};
}
m_context = context::top;
if (m_context_stack.empty()) {
top_level_element(element, attrs);
return;
}
switch (m_context_stack.back()) {
case context::osm:
// fallthrough
case context::osmChange:
data_level_element(element, attrs, false);
break;
case context::top:
assert(!m_tl_builder);
if (!std::strcmp(element, "node")) {
mark_header_as_done();
if (read_types() & osmium::osm_entity_bits::node) {
m_node_builder.reset(new osmium::builder::NodeBuilder{m_buffer});
m_node_builder->set_user(init_object(m_node_builder->object(), attrs));
m_context = context::node;
} else {
m_context = context::ignored_node;
}
} else if (!std::strcmp(element, "way")) {
mark_header_as_done();
if (read_types() & osmium::osm_entity_bits::way) {
m_way_builder.reset(new osmium::builder::WayBuilder{m_buffer});
m_way_builder->set_user(init_object(m_way_builder->object(), attrs));
m_context = context::way;
} else {
m_context = context::ignored_way;
}
} else if (!std::strcmp(element, "relation")) {
mark_header_as_done();
if (read_types() & osmium::osm_entity_bits::relation) {
m_relation_builder.reset(new osmium::builder::RelationBuilder{m_buffer});
m_relation_builder->set_user(init_object(m_relation_builder->object(), attrs));
m_context = context::relation;
} else {
m_context = context::ignored_relation;
}
} else if (!std::strcmp(element, "changeset")) {
mark_header_as_done();
if (read_types() & osmium::osm_entity_bits::changeset) {
m_changeset_builder.reset(new osmium::builder::ChangesetBuilder{m_buffer});
init_changeset(*m_changeset_builder, attrs);
m_context = context::changeset;
} else {
m_context = context::ignored_changeset;
}
} else if (!std::strcmp(element, "bounds")) {
osmium::Location min;
osmium::Location max;
check_attributes(attrs, [&min, &max](const XML_Char* name, const XML_Char* value) {
if (!std::strcmp(name, "minlon")) {
min.set_lon(value);
} else if (!std::strcmp(name, "minlat")) {
min.set_lat(value);
} else if (!std::strcmp(name, "maxlon")) {
max.set_lon(value);
} else if (!std::strcmp(name, "maxlat")) {
max.set_lat(value);
}
});
osmium::Box box;
box.extend(min).extend(max);
m_header.add_box(box);
} else if (!std::strcmp(element, "delete")) {
m_in_delete_section = true;
}
case context::create_section:
// fallthrough
case context::modify_section:
// fallthrough
case context::delete_section:
data_level_element(element, attrs, true);
break;
case context::node:
m_last_context = context::node;
m_context = context::in_object;
if (!std::strcmp(element, "tag")) {
get_tag(*m_node_builder, attrs);
m_context_stack.push_back(context::tag);
if (read_types() & osmium::osm_entity_bits::node) {
get_tag(*m_node_builder, attrs);
}
} else {
throw xml_error{std::string{"Unknown element in <node>: "} + element};
}
break;
case context::way:
m_last_context = context::way;
m_context = context::in_object;
if (!std::strcmp(element, "nd")) {
m_tl_builder.reset();
m_context_stack.push_back(context::nd);
if (read_types() & osmium::osm_entity_bits::way) {
m_tl_builder.reset();
if (!m_wnl_builder) {
m_wnl_builder.reset(new osmium::builder::WayNodeListBuilder{*m_way_builder});
}
NodeRef nr;
check_attributes(attrs, [this, &nr](const XML_Char* name, const XML_Char* value) {
if (!std::strcmp(name, "ref")) {
nr.set_ref(osmium::string_to_object_id(value));
} else if (!std::strcmp(name, "lon")) {
nr.location().set_lon(value);
} else if (!std::strcmp(name, "lat")) {
nr.location().set_lat(value);
if (!m_wnl_builder) {
m_wnl_builder.reset(new osmium::builder::WayNodeListBuilder{*m_way_builder});
}
});
m_wnl_builder->add_node_ref(nr);
NodeRef nr;
check_attributes(attrs, [&nr](const XML_Char* name, const XML_Char* value) {
if (!std::strcmp(name, "ref")) {
nr.set_ref(osmium::string_to_object_id(value));
} else if (!std::strcmp(name, "lon")) {
nr.location().set_lon(value);
} else if (!std::strcmp(name, "lat")) {
nr.location().set_lat(value);
}
});
m_wnl_builder->add_node_ref(nr);
}
} else if (!std::strcmp(element, "tag")) {
m_wnl_builder.reset();
get_tag(*m_way_builder, attrs);
m_context_stack.push_back(context::tag);
if (read_types() & osmium::osm_entity_bits::way) {
m_wnl_builder.reset();
get_tag(*m_way_builder, attrs);
}
} else {
throw xml_error{std::string{"Unknown element in <way>: "} + element};
}
break;
case context::relation:
m_last_context = context::relation;
m_context = context::in_object;
if (!std::strcmp(element, "member")) {
m_tl_builder.reset();
m_context_stack.push_back(context::member);
if (read_types() & osmium::osm_entity_bits::relation) {
m_tl_builder.reset();
if (!m_rml_builder) {
m_rml_builder.reset(new osmium::builder::RelationMemberListBuilder{*m_relation_builder});
}
item_type type = item_type::undefined;
object_id_type ref = 0;
bool ref_is_set = false;
const char* role = "";
check_attributes(attrs, [&](const XML_Char* name, const XML_Char* value) {
if (!std::strcmp(name, "type")) {
type = char_to_item_type(value[0]);
} else if (!std::strcmp(name, "ref")) {
ref = osmium::string_to_object_id(value);
ref_is_set = true;
} else if (!std::strcmp(name, "role")) {
role = static_cast<const char*>(value);
if (!m_rml_builder) {
m_rml_builder.reset(new osmium::builder::RelationMemberListBuilder{*m_relation_builder});
}
});
if (type != item_type::node && type != item_type::way && type != item_type::relation) {
throw osmium::xml_error{"Unknown type on relation member"};
item_type type = item_type::undefined;
object_id_type ref = 0;
bool ref_is_set = false;
const char* role = "";
check_attributes(attrs, [&type, &ref, &ref_is_set, &role](const XML_Char* name, const XML_Char* value) {
if (!std::strcmp(name, "type")) {
type = char_to_item_type(value[0]);
} else if (!std::strcmp(name, "ref")) {
ref = osmium::string_to_object_id(value);
ref_is_set = true;
} else if (!std::strcmp(name, "role")) {
role = static_cast<const char*>(value);
}
});
if (type != item_type::node && type != item_type::way && type != item_type::relation) {
throw osmium::xml_error{"Unknown type on relation <member>"};
}
if (!ref_is_set) {
throw osmium::xml_error{"Missing ref on relation <member>"};
}
m_rml_builder->add_member(type, ref, role);
}
if (!ref_is_set) {
throw osmium::xml_error{"Missing ref on relation member"};
}
m_rml_builder->add_member(type, ref, role);
} else if (!std::strcmp(element, "tag")) {
m_rml_builder.reset();
get_tag(*m_relation_builder, attrs);
m_context_stack.push_back(context::tag);
if (read_types() & osmium::osm_entity_bits::relation) {
m_rml_builder.reset();
get_tag(*m_relation_builder, attrs);
}
} else {
throw xml_error{std::string{"Unknown element in <relation>: "} + element};
}
break;
case context::tag:
throw xml_error{"No element inside <tag> allowed"};
case context::nd:
throw xml_error{"No element inside <nd> allowed"};
case context::member:
throw xml_error{"No element inside <member> allowed"};
case context::changeset:
m_last_context = context::changeset;
if (!std::strcmp(element, "discussion")) {
m_context = context::discussion;
m_tl_builder.reset();
if (!m_changeset_discussion_builder) {
m_changeset_discussion_builder.reset(new osmium::builder::ChangesetDiscussionBuilder{*m_changeset_builder});
m_context_stack.push_back(context::discussion);
if (read_types() & osmium::osm_entity_bits::changeset) {
m_tl_builder.reset();
if (!m_changeset_discussion_builder) {
m_changeset_discussion_builder.reset(new osmium::builder::ChangesetDiscussionBuilder{*m_changeset_builder});
}
}
} else if (!std::strcmp(element, "tag")) {
m_context = context::in_object;
m_changeset_discussion_builder.reset();
get_tag(*m_changeset_builder, attrs);
m_context_stack.push_back(context::tag);
if (read_types() & osmium::osm_entity_bits::changeset) {
m_changeset_discussion_builder.reset();
get_tag(*m_changeset_builder, attrs);
}
} else {
throw xml_error{std::string{"Unknown element in <changeset>: "} + element};
}
break;
case context::discussion:
if (!std::strcmp(element, "comment")) {
m_context = context::comment;
osmium::Timestamp date;
osmium::user_id_type uid = 0;
const char* user = "";
check_attributes(attrs, [&date, &uid, &user](const XML_Char* name, const XML_Char* value) {
if (!std::strcmp(name, "date")) {
date = osmium::Timestamp(value);
} else if (!std::strcmp(name, "uid")) {
uid = osmium::string_to_user_id(value);
} else if (!std::strcmp(name, "user")) {
user = static_cast<const char*>(value);
}
});
m_changeset_discussion_builder->add_comment(date, uid, user);
m_context_stack.push_back(context::comment);
if (read_types() & osmium::osm_entity_bits::changeset) {
osmium::Timestamp date;
osmium::user_id_type uid = 0;
const char* user = "";
check_attributes(attrs, [&date, &uid, &user](const XML_Char* name, const XML_Char* value) {
if (!std::strcmp(name, "date")) {
date = osmium::Timestamp{value};
} else if (!std::strcmp(name, "uid")) {
uid = osmium::string_to_uid(value);
} else if (!std::strcmp(name, "user")) {
user = static_cast<const char*>(value);
}
});
m_changeset_discussion_builder->add_comment(date, uid, user);
}
} else {
throw xml_error{std::string{"Unknown element in <discussion>: "} + element};
}
break;
case context::comment:
if (!std::strcmp(element, "text")) {
m_context = context::comment_text;
m_context_stack.push_back(context::text);
} else {
throw xml_error{std::string{"Unknown element in <comment>: "} + element};
}
break;
case context::comment_text:
break;
case context::ignored_node:
break;
case context::ignored_way:
break;
case context::ignored_relation:
break;
case context::ignored_changeset:
break;
case context::in_object:
assert(false); // should never be here
break;
case context::text:
throw osmium::xml_error{"No element in <text> allowed"};
case context::bounds:
throw osmium::xml_error{"No element in <bounds> allowed"};
case context::other:
throw xml_error{"xml file nested too deep"};
}
}
void end_element(const XML_Char* element) {
switch (m_context) {
case context::root:
assert(false); // should never be here
assert(!m_context_stack.empty());
switch (m_context_stack.back()) {
case context::osm:
assert(!std::strcmp(element, "osm"));
mark_header_as_done();
break;
case context::top:
if (!std::strcmp(element, "osm") || !std::strcmp(element, "osmChange")) {
mark_header_as_done();
m_context = context::root;
} else if (!std::strcmp(element, "delete")) {
m_in_delete_section = false;
}
case context::osmChange:
assert(!std::strcmp(element, "osmChange"));
mark_header_as_done();
break;
case context::create_section:
assert(!std::strcmp(element, "create"));
break;
case context::modify_section:
assert(!std::strcmp(element, "modify"));
break;
case context::delete_section:
assert(!std::strcmp(element, "delete"));
break;
case context::node:
assert(!std::strcmp(element, "node"));
m_tl_builder.reset();
m_node_builder.reset();
m_buffer.commit();
m_context = context::top;
flush_buffer();
if (read_types() & osmium::osm_entity_bits::node) {
m_tl_builder.reset();
m_node_builder.reset();
m_buffer.commit();
flush_buffer();
}
break;
case context::way:
assert(!std::strcmp(element, "way"));
m_tl_builder.reset();
m_wnl_builder.reset();
m_way_builder.reset();
m_buffer.commit();
m_context = context::top;
flush_buffer();
if (read_types() & osmium::osm_entity_bits::way) {
m_tl_builder.reset();
m_wnl_builder.reset();
m_way_builder.reset();
m_buffer.commit();
flush_buffer();
}
break;
case context::relation:
assert(!std::strcmp(element, "relation"));
m_tl_builder.reset();
m_rml_builder.reset();
m_relation_builder.reset();
m_buffer.commit();
m_context = context::top;
flush_buffer();
if (read_types() & osmium::osm_entity_bits::relation) {
m_tl_builder.reset();
m_rml_builder.reset();
m_relation_builder.reset();
m_buffer.commit();
flush_buffer();
}
break;
case context::tag:
break;
case context::nd:
break;
case context::member:
break;
case context::changeset:
assert(!std::strcmp(element, "changeset"));
m_tl_builder.reset();
m_changeset_discussion_builder.reset();
m_changeset_builder.reset();
m_buffer.commit();
m_context = context::top;
flush_buffer();
if (read_types() & osmium::osm_entity_bits::changeset) {
m_tl_builder.reset();
m_changeset_discussion_builder.reset();
m_changeset_builder.reset();
m_buffer.commit();
flush_buffer();
}
break;
case context::discussion:
assert(!std::strcmp(element, "discussion"));
m_context = context::changeset;
break;
case context::comment:
assert(!std::strcmp(element, "comment"));
m_context = context::discussion;
break;
case context::comment_text:
case context::text:
assert(!std::strcmp(element, "text"));
m_context = context::comment;
m_changeset_discussion_builder->add_comment_text(m_comment_text);
break;
case context::in_object:
m_context = m_last_context;
break;
case context::ignored_node:
if (!std::strcmp(element, "node")) {
m_context = context::top;
if (read_types() & osmium::osm_entity_bits::changeset) {
m_changeset_discussion_builder->add_comment_text(m_comment_text);
m_comment_text.clear();
}
break;
case context::ignored_way:
if (!std::strcmp(element, "way")) {
m_context = context::top;
}
case context::bounds:
assert(!std::strcmp(element, "bounds"));
break;
case context::ignored_relation:
if (!std::strcmp(element, "relation")) {
m_context = context::top;
}
break;
case context::ignored_changeset:
if (!std::strcmp(element, "changeset")) {
m_context = context::top;
}
case context::other:
break;
}
m_context_stack.pop_back();
}
void characters(const XML_Char* text, int len) {
if (m_context == context::comment_text) {
if ((read_types() & osmium::osm_entity_bits::changeset) &&
!m_context_stack.empty() &&
m_context_stack.back() == context::text) {
m_comment_text.append(text, len);
} else {
m_comment_text.resize(0);
}
}
void flush_buffer() {
if (m_buffer.committed() > buffer_size / 10 * 9) {
send_to_output_queue(std::move(m_buffer));
osmium::memory::Buffer buffer(buffer_size);
osmium::memory::Buffer buffer{buffer_size};
using std::swap;
swap(m_buffer, buffer);
}
@@ -629,27 +708,21 @@ namespace osmium {
explicit XMLParser(parser_arguments& args) :
Parser(args),
m_context(context::root),
m_last_context(context::root),
m_in_delete_section(false),
m_header(),
m_buffer(buffer_size),
m_node_builder(),
m_way_builder(),
m_relation_builder(),
m_changeset_builder(),
m_changeset_discussion_builder(),
m_tl_builder(),
m_wnl_builder(),
m_rml_builder() {
m_buffer(buffer_size) {
}
XMLParser(const XMLParser&) = delete;
XMLParser& operator=(const XMLParser&) = delete;
XMLParser(XMLParser&&) = delete;
XMLParser& operator=(XMLParser&&) = delete;
~XMLParser() noexcept final = default;
void run() final {
osmium::thread::set_thread_name("_osmium_xml_in");
ExpatXMLParser<XMLParser> parser(this);
ExpatXMLParser parser{this};
while (!input_done()) {
const std::string data{get_input()};
include/osmium/io/detail/xml_output_format.hpp
+41 -52
View File
@@ -5,7 +5,7 @@
This file is part of Osmium (http://osmcode.org/libosmium).
Copyright 2013-2017 Jochen Topf <jochen@topf.org> and others (see README).
Copyright 2013-2018 Jochen Topf <jochen@topf.org> and others (see README).
Boost Software License - Version 1.0 - August 17th, 2003
@@ -33,11 +33,6 @@ DEALINGS IN THE SOFTWARE.
*/
#include <iterator>
#include <memory>
#include <string>
#include <utility>
#include <osmium/handler.hpp>
#include <osmium/io/detail/output_format.hpp>
#include <osmium/io/detail/queue_util.hpp>
@@ -51,6 +46,7 @@ DEALINGS IN THE SOFTWARE.
#include <osmium/osm/changeset.hpp>
#include <osmium/osm/item_type.hpp>
#include <osmium/osm/location.hpp>
#include <osmium/osm/metadata_options.hpp>
#include <osmium/osm/node.hpp>
#include <osmium/osm/node_ref.hpp>
#include <osmium/osm/object.hpp>
@@ -62,6 +58,11 @@ DEALINGS IN THE SOFTWARE.
#include <osmium/thread/pool.hpp>
#include <osmium/visitor.hpp>
#include <iterator>
#include <memory>
#include <string>
#include <utility>
namespace osmium {
namespace io {
@@ -72,21 +73,22 @@ namespace osmium {
struct xml_output_options {
/// Should metadata of objects be added?
bool add_metadata;
/// Which metadata of objects should be added?
osmium::metadata_options add_metadata;
/// Should the visible flag be added to all OSM objects?
bool add_visible_flag;
bool add_visible_flag = false;
/**
* Should <create>, <modify>, <delete> "operations" be added?
* (This is used for .osc files.)
*/
bool use_change_ops;
bool use_change_ops = false;
/// Should node locations be added to ways?
bool locations_on_ways;
};
bool locations_on_ways = false;
}; // struct xml_output_options
namespace detail {
@@ -144,34 +146,35 @@ namespace osmium {
void write_meta(const osmium::OSMObject& object) {
write_attribute("id", object.id());
if (m_options.add_metadata) {
if (object.version()) {
write_attribute("version", object.version());
}
if (m_options.add_metadata.version() && object.version()) {
write_attribute("version", object.version());
}
if (object.timestamp()) {
*m_out += " timestamp=\"";
*m_out += object.timestamp().to_iso();
*m_out += "\"";
}
if (m_options.add_metadata.timestamp() && object.timestamp()) {
*m_out += " timestamp=\"";
*m_out += object.timestamp().to_iso();
*m_out += "\"";
}
if (!object.user_is_anonymous()) {
write_attribute("uid", object.uid());
*m_out += " user=\"";
append_xml_encoded_string(*m_out, object.user());
*m_out += "\"";
}
if (m_options.add_metadata.uid() && object.uid()) {
write_attribute("uid", object.uid());
}
if (object.changeset()) {
write_attribute("changeset", object.changeset());
}
if (m_options.add_metadata.user() && object.user()[0] != '\0') {
*m_out += " user=\"";
append_xml_encoded_string(*m_out, object.user());
*m_out += "\"";
}
if (m_options.add_visible_flag) {
if (object.visible()) {
*m_out += " visible=\"true\"";
} else {
*m_out += " visible=\"false\"";
}
if (m_options.add_metadata.changeset() && object.changeset()) {
write_attribute("changeset", object.changeset());
}
if (m_options.add_visible_flag) {
if (object.visible()) {
*m_out += " visible=\"true\"";
} else {
*m_out += " visible=\"false\"";
}
}
}
@@ -247,14 +250,6 @@ namespace osmium {
m_options(options) {
}
XMLOutputBlock(const XMLOutputBlock&) = default;
XMLOutputBlock& operator=(const XMLOutputBlock&) = default;
XMLOutputBlock(XMLOutputBlock&&) = default;
XMLOutputBlock& operator=(XMLOutputBlock&&) = default;
~XMLOutputBlock() noexcept = default;
std::string operator()() {
osmium::apply(m_input_buffer->cbegin(), m_input_buffer->cend(), *this);
@@ -432,19 +427,13 @@ namespace osmium {
public:
XMLOutputFormat(osmium::thread::Pool& pool, const osmium::io::File& file, future_string_queue_type& output_queue) :
OutputFormat(pool, output_queue),
m_options() {
m_options.add_metadata = file.is_not_false("add_metadata");
OutputFormat(pool, output_queue) {
m_options.add_metadata = osmium::metadata_options{file.get("add_metadata")};
m_options.use_change_ops = file.is_true("xml_change_format");
m_options.add_visible_flag = (file.has_multiple_object_versions() || file.is_true("force_visible_flag")) && !m_options.use_change_ops;
m_options.locations_on_ways = file.is_true("locations_on_ways");
}
XMLOutputFormat(const XMLOutputFormat&) = delete;
XMLOutputFormat& operator=(const XMLOutputFormat&) = delete;
~XMLOutputFormat() noexcept final = default;
void write_header(const osmium::io::Header& header) final {
std::string out{"<?xml version='1.0' encoding='UTF-8'?>\n"};
include/osmium/io/detail/zlib.hpp
+17 -9
View File
@@ -5,7 +5,7 @@
This file is part of Osmium (http://osmcode.org/libosmium).
Copyright 2013-2017 Jochen Topf <jochen@topf.org> and others (see README).
Copyright 2013-2018 Jochen Topf <jochen@topf.org> and others (see README).
Boost Software License - Version 1.0 - August 17th, 2003
@@ -33,14 +33,21 @@ DEALINGS IN THE SOFTWARE.
*/
#include <string>
#include <osmium/io/error.hpp>
#include <protozero/version.hpp>
#if PROTOZERO_VERSION_CODE >= 10600
# include <protozero/data_view.hpp>
#else
# include <protozero/types.hpp>
#endif
#include <zlib.h>
#include <protozero/types.hpp>
#include <osmium/io/error.hpp>
#include <osmium/util/cast.hpp>
#include <cassert>
#include <limits>
#include <string>
namespace osmium {
@@ -58,7 +65,8 @@ namespace osmium {
* @returns Compressed data.
*/
inline std::string zlib_compress(const std::string& input) {
unsigned long output_size = ::compressBound(osmium::static_cast_with_assert<unsigned long>(input.size()));
assert(input.size() < std::numeric_limits<unsigned long>::max());
unsigned long output_size = ::compressBound(static_cast<unsigned long>(input.size())); // NOLINT(google-runtime-int)
std::string output(output_size, '\0');
@@ -66,7 +74,7 @@ namespace osmium {
reinterpret_cast<unsigned char*>(const_cast<char *>(output.data())),
&output_size,
reinterpret_cast<const unsigned char*>(input.data()),
osmium::static_cast_with_assert<unsigned long>(input.size())
static_cast<unsigned long>(input.size()) // NOLINT(google-runtime-int)
);
if (result != Z_OK) {
@@ -89,7 +97,7 @@ namespace osmium {
* @param output Uncompressed result data.
* @returns Pointer and size to incompressed data.
*/
inline protozero::data_view zlib_uncompress_string(const char* input, unsigned long input_size, unsigned long raw_size, std::string& output) {
inline protozero::data_view zlib_uncompress_string(const char* input, unsigned long input_size, unsigned long raw_size, std::string& output) { // NOLINT(google-runtime-int)
output.resize(raw_size);
const auto result = ::uncompress(