Squashed 'third_party/libosmium/' changes from 8bcd4ea..c43f8db

c43f8db Release v2.3.0
44c135f Update README to show dependencies used internally.
ece54cd Add external licenses.
908cd5f Updated change log.
96dbf0e Change %-escape in OPL format.
98f6e27 Change write benchmark to interleave reading and writing.
39620ce Make writing of metadata configurable for XML and OPL output.
e5a4e5e Add debug output format.
597390f Remove superfluous include and pragmas.
ecc57b0 Update pbf reader/writer to use new protozero functions.
5d1e8d2 Update protozero from upstream.
ef8746b Fix build on Windows.
ddba46f Remove superfluous include.
098c57f Add some paranoia checks to pbf reader.
0f804c2 Try building with newer boost library on travis.
6f79d63 Use explicit return types on lambdas.
355f3b1 New PBF reader and writer based on protozero.
71d719b Add pbf writing benchmark.
f014b4c Fix iwyu.sh script: Works now if build directory doesn't exist.
a0ace49 Use utf8cpp header-only lib instead of boost for utf8 decoding.
796f18e Bugfix: Reading large XML files could block.
5a2bcbe Replace strcmp by std::string comparison in test.
bc49e2c Bugfix: XML writer was not writing whitespace correctly.
61222f8 Fix 64bit byte swap.
e56f090 Fix new CRC code on OSX and Windows.
70229aa Add low-level building blocks that allow calculating CRC of OSM data.
0968a66 Remove assert checking for unset version.
62e0261 Refactor test case.
4bfc7fc Allow instantiating osmium::geom::GEOSFactory with existing GEOS factory.
e70af0c Remove calls to protobuf cleanup function im benchmarks and examples.
718518d Bugfix in OPL output. Relation member roles were not encoded.
759d5cb Rename parameter that had the same name as a type.
7054cab Provide (Typed)MemoryMapping constructors for backwards compatibility.
d09f5d1 Fix typo.
b4e578f Make memory mapping utility class more flexible.
633fa8e Travis build without sudo.
7ff23f1 Improved code setting file format from suffix/format argument.
90ef3b9 Remove some tests that didn't test much and failed on FreeBSD.
af86273 Add some pragmas to disable warnings for GCC.
efac7fd Fix some include problems found by IWYU.
79d2f4c Changed add_user() and add_role() in builders. Add add_member().
9375d00 Add function to set tags from ptr + length. Improve TagBuilder tests.
bafca20 Test helper: Use more const and have sub-builders in their own scope.
f73c993 Simplify code.
fee1710 Disable warning only when compiling with GCC.
74402f3 Merge pull request #98 from dforsi/master
2c4b449 Update to new upstream catch.hpp version.
1318732 Release v2.2.0
1873998 Add missing test.
2e5ea1d Do not add timestamp to html doc pages.
1b2ea89 Remove debug output.
0be9599 Improved parsing of ids, versions, uids, etc. from strings.
4308d80 Add second version of split_string utility function.
f18c9e5 Move part of pbf.hpp into new pbf_type_conv.hpp.
d201152 Use new DeltaEncode class in pbf writer.
e205610 Add DeltaEncode/DeltaDecode utility classes.
32905d6 Bugfix: Actually throw the exception we are creating...
d3e86d8 Add functions to convert item_type to zero-based index.
daddf07 Bugfix: Programs writing OSM files can stall up to a second after writing.
00b0247 Add function to set the id of a relation member.
f85316a Fix error message.
19bc6cc Fix name of travis install script.
719cd33 spatialite-bin package now available on travis
cb03821 Shorten long test string (MSVC doesn't like it).
c3440a6 Add BoolVector index class.
da08073 Add min_op/max_op utility functions.
411d112 AppVeyor.yml: new links for binary deps
7d9095f add test for badly formatted timestamps
a073f73 Add helper methods to DiffObject.
3b9819a Add GeoJSON factory using the RapidJSON library.
107bca5 Use a reference instead of a copy.
a6943a4 Mark a few variables that are not changing as const.
51b7e53 Improved error message for geometry exceptions.
5c37a13 Some minor spelling fixes
8ae5723 Bugfix: Dense location store was written out only partially.
5994322 Add support for tiles.
2168bac Add has_map_type() method to map factory.
a9634bd Add more tests for mercator projection.
3c13e4d Add functionality to create simple polygons from ways in geom factories.
e8c5bb1 Use uint64_t as counter, so there can be no overflows.
07fc9b9 libsparsehash-dev now in travis package whitelist
820e112 Add coverage support to CMake config.
5e9f943 Bugfix: Use the right include to really allow any input file type.
d4b48eb CMake: Make version string a cached variable.
e6baccb Add (c)begin/end functions to TypedMemoryMapping. Removed get_addr().
3e32710 Use size() from MemoryMapping in TypedMemoryMapping.
96390db Improve MemoryMapping class documentation.
60a6217 Do not round memory mapped files to page size boundaries.
4907cbe Bugfix: function name.
cac01d8 Use _filelengthi64 on Windows instead of fstat(2).
6a25bdf Windows: Put invalid parameter handler into wrapper class. Re-enable test.
110df9b Add invalid parameter handler on Windows to test.
549ed5f Disable some tests (to find which one fails on appveyor).
a5b8873 Use resize_file() in memory mapping test instead of ftruncate directly.
40e41d3 Use _chsize_s() instead of _chsize() on Windows.
048397e Refactoring: Use low-level util functions in DataFile.
6a033f9 Remove now unused Windows implementation of mmap.
3eccdbb Move dword_hi/lo functions into osmium::util namespace.
be7351b Remove unused code.
b859b18 Make dword_hi/lo functions inline.
2e3bc37 Simplify mmap_vector_base/anon/file.
f819cf3 Always map full pages. Make sure files behind mapping are large enough.
d0c84b6 Add some low-level helper functions for file system access.
62e8d91 Make DataFile constructor explicit.
fba684c Fix memory mapping test for windows.
78a7fd5 Add constructor to DataFile to create tmp file with given size.
f911893 Bugfix: typo.
1cf2739 Add AnonymousMemoryMapping class.
56eac30 Implement MemoryMapping::resize() function.
1a73262 Bugfix: Counter variables were too small.
1ade32c Fix include position.
b03aec3 Fixed some bugs in new DataFile class/tests.
f109534 Add DataFile utility class.
9ed3c43 Fix/cleanup some code.
4f326c9 Fix bug: Copy-and-paste error.
78a5b2f Use reinterpret_cast instead of static_cast to get HANDLE on Windows.
7baa318 Fix typo.
e669069 Make huge value even huger to see if code reliable fails then.
66137ad Improved documentation of MemoryMapping and TypedMemoryMapping classes.
3121393 Add TypedMemoryMapping class.
f45335e Default for get_addr() template type.
685bbaf Remove unused code from tests.
ce65bd4 Fix some issue with new MemoryMapping class.
e7b8e15 Added MemoryMapping wrapper class for mmap() and Windows equivalent.
6b1effe typo fixed
33d479d Refactored travis build.
4348522 Fix xml data test.
769b1e8 Bugfix: Better check for invalid locations.
bba7e68 Appveyor: Disable test failing because of missing dependency.
3d40dc7 Link with /debug on MSVC, add note about LNK4099 warnings.
5ef051f Appveyor: Disable header builds, add benchmarks.
ce7485e Reformat Appveyor config.
c60e505 use shallow clones for faster git fetch
3b18bca Travis cleanups.
b8dfac0 Cleanup travis build.
5f19838 Trying to fix travis gcc build...
d4255a4 Remove -Wno-return-type from recommended options.
5f1a41b Add dump_as_array() function to maps.
ff22f76 Add constructors and begin()/end() functions to VectorBasedSparseMultimap.
c7e05dd Bugfix: Make REGISTER_MAP() macro work when called several time with same name parameter.
abdc317 Bugfix: Mark cbegin() and cend() of mmap_vector_base as const functions.
d81d439 Add close() function to mmap_vector_base class.
d74cff2 Add function on Buffer to get iterator to specific offset.

git-subtree-dir: third_party/libosmium
git-subtree-split: c43f8db50d93912a8bec5cd9fea733f7fec05549
This commit is contained in:
Patrick Niklaus
2015-08-28 12:42:03 +02:00
parent c0402345f7
commit 788bc67faa
150 changed files with 12325 additions and 4321 deletions
@@ -39,8 +39,6 @@ DEALINGS IN THE SOFTWARE.
#include <fcntl.h>
#include <stdexcept>
#include <string>
#include <sys/stat.h>
#include <sys/types.h>
#include <vector>
namespace osmium {
@@ -35,9 +35,6 @@ DEALINGS IN THE SOFTWARE.
#ifdef __linux__
#include <cstddef>
#include <osmium/index/detail/typed_mmap.hpp>
#include <osmium/index/detail/mmap_vector_base.hpp>
namespace osmium {
@@ -45,26 +42,16 @@ namespace osmium {
namespace detail {
/**
* This class looks and behaves like STL vector, but uses mmap internally.
* This class looks and behaves like STL vector, but uses mmap
* internally.
*/
template <typename T>
class mmap_vector_anon : public mmap_vector_base<T, mmap_vector_anon> {
class mmap_vector_anon : public mmap_vector_base<T> {
public:
mmap_vector_anon() :
mmap_vector_base<T, osmium::detail::mmap_vector_anon>(
-1,
osmium::detail::mmap_vector_size_increment,
0,
osmium::detail::typed_mmap<T>::map(osmium::detail::mmap_vector_size_increment)) {
}
void reserve(size_t new_capacity) {
if (new_capacity > this->capacity()) {
this->data(osmium::detail::typed_mmap<T>::remap(this->data(), this->capacity(), new_capacity));
this->m_capacity = new_capacity;
}
mmap_vector_base<T>() {
}
}; // class mmap_vector_anon
@@ -34,11 +34,10 @@ DEALINGS IN THE SOFTWARE.
*/
#include <cstddef>
#include <new>
#include <new> // IWYU pragma: keep
#include <stdexcept>
#include <osmium/index/detail/typed_mmap.hpp>
#include <osmium/util/compatibility.hpp>
#include <osmium/util/memory_mapping.hpp>
namespace osmium {
@@ -48,40 +47,29 @@ namespace osmium {
/**
* This is a base class for implementing classes that look like
* STL vector but use mmap internally. This class can not be used
* on it's own. Use the derived classes mmap_vector_anon or
* mmap_vector_file.
* STL vector but use mmap internally. Do not use this class itself,
* use the derived classes mmap_vector_anon or mmap_vector_file.
*/
template <typename T, template <typename> class TDerived>
template <typename T>
class mmap_vector_base {
protected:
int m_fd;
size_t m_capacity;
size_t m_size;
T* m_data;
explicit mmap_vector_base(int fd, size_t capacity, size_t size, T* data) noexcept :
m_fd(fd),
m_capacity(capacity),
m_size(size),
m_data(data) {
}
explicit mmap_vector_base(int fd, size_t capacity, size_t size) :
m_fd(fd),
m_capacity(capacity),
m_size(size),
m_data(osmium::detail::typed_mmap<T>::grow_and_map(capacity, m_fd)) {
}
void data(T* data) {
m_data = data;
}
osmium::util::TypedMemoryMapping<T> m_mapping;
public:
explicit mmap_vector_base(int fd, size_t capacity, size_t size = 0) :
m_size(size),
m_mapping(capacity, osmium::util::MemoryMapping::mapping_mode::write_shared, fd) {
}
explicit mmap_vector_base(size_t capacity = mmap_vector_size_increment) :
m_size(0),
m_mapping(capacity) {
}
typedef T value_type;
typedef T& reference;
typedef const T& const_reference;
@@ -90,12 +78,14 @@ namespace osmium {
typedef T* iterator;
typedef const T* const_iterator;
~mmap_vector_base() {
osmium::detail::typed_mmap<T>::unmap(m_data, m_capacity);
~mmap_vector_base() = default;
void close() {
m_mapping.unmap();
}
size_t capacity() const noexcept {
return m_capacity;
return m_mapping.size();
}
size_t size() const noexcept {
@@ -106,23 +96,23 @@ namespace osmium {
return m_size == 0;
}
const T* data() const noexcept {
return m_data;
const T* data() const {
return m_mapping.begin();
}
T* data() noexcept {
return m_data;
T* data() {
return m_mapping.begin();
}
T& operator[](size_t n) {
return m_data[n];
return data()[n];
}
T at(size_t n) const {
if (n >= m_size) {
throw std::out_of_range("out of range");
}
return m_data[n];
return data()[n];
}
void clear() noexcept {
@@ -134,16 +124,22 @@ namespace osmium {
}
void push_back(const T& value) {
if (m_size >= m_capacity) {
if (m_size >= capacity()) {
resize(m_size+1);
}
m_data[m_size] = value;
data()[m_size] = value;
++m_size;
}
void reserve(size_t new_capacity) {
if (new_capacity > capacity()) {
m_mapping.resize(new_capacity);
}
}
void resize(size_t new_size) {
if (new_size > capacity()) {
static_cast<TDerived<T>*>(this)->reserve(new_size + osmium::detail::mmap_vector_size_increment);
reserve(new_size + osmium::detail::mmap_vector_size_increment);
}
if (new_size > size()) {
new (data() + size()) T[new_size - size()];
@@ -152,27 +148,27 @@ namespace osmium {
}
iterator begin() noexcept {
return m_data;
return data();
}
iterator end() noexcept {
return m_data + m_size;
return data() + m_size;
}
const_iterator begin() const noexcept {
return m_data;
return data();
}
const_iterator end() const noexcept {
return m_data + m_size;
return data() + m_size;
}
const_iterator cbegin() noexcept {
return m_data;
const_iterator cbegin() const noexcept {
return data();
}
const_iterator cend() noexcept {
return m_data + m_size;
const_iterator cend() const noexcept {
return data() + m_size;
}
}; // class mmap_vector_base
@@ -33,11 +33,9 @@ DEALINGS IN THE SOFTWARE.
*/
#include <cstddef>
#include <osmium/index/detail/typed_mmap.hpp>
#include <osmium/index/detail/mmap_vector_base.hpp>
#include <osmium/index/detail/tmpfile.hpp>
#include <osmium/util/file.hpp>
namespace osmium {
@@ -48,32 +46,19 @@ namespace osmium {
* internally.
*/
template <typename T>
class mmap_vector_file : public mmap_vector_base<T, mmap_vector_file> {
class mmap_vector_file : public mmap_vector_base<T> {
public:
explicit mmap_vector_file() :
mmap_vector_base<T, osmium::detail::mmap_vector_file>(
explicit mmap_vector_file() : mmap_vector_base<T>(
osmium::detail::create_tmp_file(),
osmium::detail::mmap_vector_size_increment,
0) {
osmium::detail::mmap_vector_size_increment) {
}
explicit mmap_vector_file(int fd) :
mmap_vector_base<T, osmium::detail::mmap_vector_file>(
explicit mmap_vector_file(int fd) : mmap_vector_base<T>(
fd,
osmium::detail::typed_mmap<T>::file_size(fd) == 0 ?
osmium::detail::mmap_vector_size_increment :
osmium::detail::typed_mmap<T>::file_size(fd),
osmium::detail::typed_mmap<T>::file_size(fd)) {
}
void reserve(size_t new_capacity) {
if (new_capacity > this->capacity()) {
typed_mmap<T>::unmap(this->data(), this->capacity());
this->data(typed_mmap<T>::grow_and_map(new_capacity, this->m_fd));
this->m_capacity = new_capacity;
}
osmium::util::file_size(fd) / sizeof(T),
osmium::util::file_size(fd) / sizeof(T)) {
}
}; // class mmap_vector_file
-229
View File
@@ -1,229 +0,0 @@
#ifndef OSMIUM_INDEX_DETAIL_TYPED_MMAP_HPP
#define OSMIUM_INDEX_DETAIL_TYPED_MMAP_HPP
/*
This file is part of Osmium (http://osmcode.org/libosmium).
Copyright 2013-2015 Jochen Topf <jochen@topf.org> and others (see README).
Boost Software License - Version 1.0 - August 17th, 2003
Permission is hereby granted, free of charge, to any person or organization
obtaining a copy of the software and accompanying documentation covered by
this license (the "Software") to use, reproduce, display, distribute,
execute, and transmit the Software, and to prepare derivative works of the
Software, and to permit third-parties to whom the Software is furnished to
do so, all subject to the following:
The copyright notices in the Software and this entire statement, including
the above license grant, this restriction and the following disclaimer,
must be included in all copies of the Software, in whole or in part, and
all derivative works of the Software, unless such copies or derivative
works are solely in the form of machine-executable object code generated by
a source language processor.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE, TITLE AND NON-INFRINGEMENT. IN NO EVENT
SHALL THE COPYRIGHT HOLDERS OR ANYONE DISTRIBUTING THE SOFTWARE BE LIABLE
FOR ANY DAMAGES OR OTHER LIABILITY, WHETHER IN CONTRACT, TORT OR OTHERWISE,
ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
DEALINGS IN THE SOFTWARE.
*/
#include <cerrno>
#include <cstddef>
#include <stdexcept>
#include <system_error>
#include <sys/stat.h>
#ifndef _WIN32
# include <sys/mman.h>
#else
# include <mmap_for_windows.hpp>
#endif
#ifndef _MSC_VER
# include <unistd.h>
#else
# define ftruncate _chsize
#endif
// for bsd systems
#ifndef MAP_ANONYMOUS
# define MAP_ANONYMOUS MAP_ANON
#endif
#include <osmium/util/cast.hpp>
namespace osmium {
/**
* @brief Namespace for Osmium internal use
*/
namespace detail {
/**
* This is a helper class for working with memory mapped files and
* anonymous shared memory. It wraps the necessary system calls
* adding:
* - error checking: all functions throw exceptions where needed
* - internal casts and size calculations allow use with user defined
* type T instead of void*
*
* This class only contains static functions. It should never be
* instantiated.
*
* @tparam T Type of objects we want to store.
*/
template <typename T>
class typed_mmap {
public:
/**
* Create anonymous private memory mapping with enough space for size
* objects of type T.
*
* Note that no constructor is called for any of the objects in this memory!
*
* @param size Number of objects of type T that should fit into this memory
* @returns Pointer to mapped memory
* @throws std::system_error If mmap(2) failed
*/
static T* map(size_t size) {
void* addr = ::mmap(nullptr, sizeof(T) * size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
#pragma GCC diagnostic push
#pragma GCC diagnostic ignored "-Wold-style-cast"
if (addr == MAP_FAILED) {
throw std::system_error(errno, std::system_category(), "mmap failed");
}
#pragma GCC diagnostic pop
return reinterpret_cast<T*>(addr);
}
/**
* Create shared memory mapping of a file with enough space for size
* objects of type T. The file must already have at least the
* required size.
*
* Note that no constructor is called for any of the objects in this memory!
*
* @param size Number of objects of type T that should fit into this memory
* @param fd File descriptor
* @param write True if data should be writable
* @returns Pointer to mapped memory
* @throws std::system_error If mmap(2) failed
*/
static T* map(size_t size, int fd, bool write = false) {
int prot = PROT_READ;
if (write) {
prot |= PROT_WRITE;
}
void* addr = ::mmap(nullptr, sizeof(T) * size, prot, MAP_SHARED, fd, 0);
#pragma GCC diagnostic push
#pragma GCC diagnostic ignored "-Wold-style-cast"
if (addr == MAP_FAILED) {
throw std::system_error(errno, std::system_category(), "mmap failed");
}
#pragma GCC diagnostic pop
return reinterpret_cast<T*>(addr);
}
// mremap(2) is only available on linux systems
#ifdef __linux__
/**
* Grow memory mapping created with map().
*
* Note that no constructor is called for any of the objects in this memory!
*
* @param data Pointer to current mapping (as returned by typed_mmap())
* @param old_size Number of objects currently stored in this memory
* @param new_size Number of objects we want to have space for
* @throws std::system_error If mremap(2) call failed
*/
static T* remap(T* data, size_t old_size, size_t new_size) {
void* addr = ::mremap(reinterpret_cast<void*>(data), sizeof(T) * old_size, sizeof(T) * new_size, MREMAP_MAYMOVE);
#pragma GCC diagnostic push
#pragma GCC diagnostic ignored "-Wold-style-cast"
if (addr == MAP_FAILED) {
throw std::system_error(errno, std::system_category(), "mremap failed");
}
#pragma GCC diagnostic pop
return reinterpret_cast<T*>(addr);
}
#endif
/**
* Release memory from map() call.
*
* Note that no destructor is called for the objects in this memory!
*
* @param data Pointer to the data
* @param size Number of objects of type T stored
* @throws std::system_error If munmap(2) call failed
*/
static void unmap(T* data, size_t size) {
if (::munmap(reinterpret_cast<void*>(data), sizeof(T) * size) != 0) {
throw std::system_error(errno, std::system_category(), "munmap failed");
}
}
/**
* Get number of objects of type T that would fit into a file.
*
* @param fd File descriptor
* @returns Number of objects of type T in this file
* @throws std::system_error If fstat(2) call failed
* @throws std::length_error If size of the file isn't a multiple of sizeof(T)
*/
static size_t file_size(int fd) {
struct stat s;
if (fstat(fd, &s) < 0) {
throw std::system_error(errno, std::system_category(), "fstat failed");
}
if (static_cast<size_t>(s.st_size) % sizeof(T) != 0) {
throw std::length_error("file size has to be multiple of object size");
}
return static_cast<size_t>(s.st_size) / sizeof(T);
}
/**
* Grow file so there is enough space for at least new_size objects
* of type T. If the file is large enough already, nothing is done.
* The file is never shrunk.
*
* @param new_size Number of objects of type T that should fit into this file
* @param fd File descriptor
* @throws std::system_error If ftruncate(2) call failed
*/
static void grow_file(size_t new_size, int fd) {
if (file_size(fd) < new_size) {
if (::ftruncate(fd, static_cast_with_assert<off_t>(sizeof(T) * new_size)) < 0) {
throw std::system_error(errno, std::system_category(), "ftruncate failed");
}
}
}
/**
* Grow file to given size (if it is smaller) and mmap it.
*
* @param size Number of objects of type T that should fit into this file
* @param fd File descriptor
* @throws Errors thrown by grow_file() or map()
*/
static T* grow_and_map(size_t size, int fd) {
grow_file(size, fd);
return map(size, fd, true);
}
}; // class typed_mmap
} // namespace detail
} // namespace osmium
#endif // OSMIUM_INDEX_DETAIL_TYPED_MMAP_HPP
+9 -1
View File
@@ -68,7 +68,7 @@ namespace osmium {
m_vector(fd) {
}
~VectorBasedDenseMap() {}
~VectorBasedDenseMap() = default;
void reserve(const size_t size) override final {
m_vector.reserve(size);
@@ -97,6 +97,10 @@ namespace osmium {
return m_vector.size();
}
size_t byte_size() const {
return m_vector.size() * sizeof(element_type);
}
size_t used_memory() const override final {
return sizeof(TValue) * size();
}
@@ -106,6 +110,10 @@ namespace osmium {
m_vector.shrink_to_fit();
}
void dump_as_array(const int fd) override final {
osmium::io::detail::reliable_write(fd, reinterpret_cast<const char*>(m_vector.data()), byte_size());
}
iterator begin() {
return m_vector.begin();
}
@@ -67,6 +67,16 @@ namespace osmium {
public:
VectorBasedSparseMultimap() :
m_vector() {
}
explicit VectorBasedSparseMultimap(int fd) :
m_vector(fd) {
}
~VectorBasedSparseMultimap() = default;
void set(const TId id, const TValue value) override final {
m_vector.push_back(element_type(id, value));
}
@@ -141,6 +151,30 @@ namespace osmium {
osmium::io::detail::reliable_write(fd, reinterpret_cast<const char*>(m_vector.data()), byte_size());
}
iterator begin() {
return m_vector.begin();
}
iterator end() {
return m_vector.end();
}
const_iterator cbegin() const {
return m_vector.cbegin();
}
const_iterator cend() const {
return m_vector.cend();
}
const_iterator begin() const {
return m_vector.cbegin();
}
const_iterator end() const {
return m_vector.cend();
}
}; // class VectorBasedSparseMultimap
} // namespace multimap