Viewing 15 posts - 1 through 15 (of 16 total)
  • Author
    Posts
  • #44739
    ErwinB
    Participant

    Hi there,

    Is there any information somewhere on the site on how the OpenAndroMaps are created?
    I mean hardware, software, configurations used?

    Why? Do I want to make my own version of OAM. Absolutely not.
    I am trying to create maps for the Wahoo line of bike computers. They come with their own maps but these are very rarely updated so there is “a demand” for updated maps.
    Now I have already, with the help of others, created a python program that creates the maps needed. Basically map files of x/y tiles at zoom level 8. But I am searching for information how and what other people are doing/using to make it faster/better etc.
    Right now (btw I am on windows) the program downloads the countries needed from Geofabrik (osm.pbf files), converts them to o5m using osmconvert, strips unneeded information using osmfilter and then converts them back to osm.pbf format. Then it creates land and sea for the wanted tiles and splits the tiles from the filtered countries’ osm.pbf files. Finally the land, sea and map tiles are merged using Osmosis and Osmosis with the mapwriter plugin is used to create the final tiles.
    Again this works but is ehh not fast. But what I don’t understand for example is that if I look at the mapsforge creator project they through whole counties at the mapwriter plugin. If I do this with a single tile, without filtering it first, it already crashes (no, no out of memory). But somehow you manage to make whole country maps too.
    I also have a postgresql db with the planet file (created with osmosis) in it but when I try to extract a singe tile of map data it takes about 2 hours! Totally unusable. But other seem to be able to use these db’s just fine.

    Sorry for making this a long post but in short, any background information would be welcome!

    #44742
    mbe57
    Moderator

    Hi Erwin, first of all let’s understand what makes those Wahoo bike maps special (and different from OAM with Locus e.g.):
    – appearance / information
    — coloring
    — selection of information
    — user choices
    – technical specialities
    — your comments include tiles (pointing to ratser maps with PNG/JPG tiles) but also mapwriter, which typically produces .map vector maps. Or did I miss something here?
    — any specialities beyond?
    Cheers
    Michael

    #44744
    ErwinB
    Participant

    Hi Michael,
    Thanks for responding. Maybe I was not clear enough, my question is not about appearance, coloring etc. That has been taken care of and working at this point. An other guy in our little group made a tool to “extract” the poi’s, ways and zoom-appear levels used by Wahoo. I adapted the tag-mapping file used by mapwriter to match and think improve it.
    Wahoo puts a lot of poi’s in the map files for example but does(/can’t) use them on the actual devices and is well… doing some weird stuff there. The simplest example being they include a lot of poi’s for shops but leave out bicycle shops. Bit strange for a bicycle computer. To keep the files relatively small they then use a high simplification factor in the mapwriter. I leave out anything their rendering theme does not use and without simplification get a better looking smaller map!
    But again, not the question.
    I refer to the files as map “titles” because the x/y coordinate system is to my knowledge referred to as slippy tiles.
    The Wahoo devices are running android and have a folder structure consisting of zoom level/X/Y.map. The zoom level is fixed at 8 so that the whole globe is 256 by 256 tiles. So each x/y coordinate has its own map file/tile.

    My questions are:
    – How do the people here, and elsewhere, manage to create map files for whole countries when I have to struggle to create map files for small parts? On the one hand just being curious and more importantly wondering if maybe there are better alternatives to the software I am using. The development of Osmosis and mapwriter seems to have pretty much stopped years ago. I can’t use Osmium because its Linux only (again, as far as I know).
    – What is the data source? Are these also the Geofabrik files? (or a similar site) Or is a postgresql db being used and if so, how is data imported/extracted and how long does this take. As I said, 2 hours for a single tiles map data is not usable.

    #44748
    mbe57
    Moderator

    Hi Erwin,
    sounds like there is a hybrid of x/y breakdown that you normally see for raster type files (PNG or JPG) and mapsforge .map vector maps. Interesting indeed. Never saw such before.

    Your questions from the latter paragraph Christian can answer in more details, but the fundamental answers I may give you as well.
    He uses osmosis mapwriter and uses those pbf files as a base. Of course filtering, tag mapping etc. etc.
    So your toolset is pretty much the same. The reason why your setup breaks I cannot see, though, sorry.
    Christian, any typical traps you know?

    Cheers
    Michael

    #44761
    ErwinB
    Participant

    Thanks Michael,
    I also know the 256×256 (pixels) tile system from raster files. I am sure it was the inspiration for this solution. On these little bike computers it is a way to keep the memory usage down I guess.

    I had hoped to get clues to the existence of other faster tools. Splitting/cutting these tiles from large osm.pbf files like Russia, Canada etc is very time consuming. I heard Osmium is a lot faster there but I seem to be stuck with java/Osmosis.

    I expect the big difference to be that you guys are on Linux, mapping is a real Linux thingy it seems, and are able to through a lot of memory at it 🙂

    But if Christiaan or somebody else knows something that makes the windows mapping live easier…

    #44908
    Avatar photoChristianK
    Keymaster

    Well, creating Mapsforge Maps needs som patience.

    What you need ist RAM,RAM,RAM (min 16Gb) and a really fast SSD

    One of the key params is JavaHeap.
    So add at top of osmosis.bat:

    set osmosis_temp=c:/oam_temp (point to SSD, best is M2)
    set JAVACMD_OPTIONS=-Xms3G -Xmx15G -Djava.io.tmpdir=%osmosis_temp%

    ..where Xmx15G means 15Gb of RAM is assigned to Java Heap.
    If you want to run the MapWriter in RAM you cancalculate the JavaHeap= pbf-file /15 = amount of RAM needed AND assigned via “Xmx” to Java. But this is only possible for small maps.
    If you dont do this Java starts to swap and/or crashes.

    Normaly Mapwriter is started with “type=hd” and indexfiles are written to a fast SSD rather tha using ram.

    Most Important: Identify stupid, large Realtions and put those on a blacklist (osmfilter = first step).
    Where I need labels from such Relations I convert these Relations to a single node label and delete the relation afterwards – before mapwriting.

    Austria map need, with all these optimizations, about 12 hours for the last step = the mapwriter.
    So 24h for a map like Austria is normal.

    #44913
    ErwinB
    Participant

    Hi Christian,

    Thanks for taking the time to answer this. Unfortunately for me I already use(d) most of these options 🙁 I currently have -Xmx set to 25. Not that it is available on windows with 32GB installed and doing other stuff…
    My main ssd is a m2 model already, not a pci e x4 model unfortunately but upgrading from my current threadripper 12 core has an insane price tag at the moment, just the new processor I would want is in the 1000-5000 euro range depending on the number of cores which I would not blink about a couple of months ago but right now is a little much…
    But working with the pre-filtered maps in tiles I get away with ~ 4 parallel cmd windows. Thanks to the tiled (and filtered) setup I think Austria is done within the hour. It unfortunately is a bit hard to test this just for fun at the moment.
    I would be interested in how to blacklist “stupidly large relations” 🙂 if it’s not a secret of the trade. To be honest I don’t need the info but I am more than a little intrigued…
    My only(?) remaining problem at the moment is that osmconvert bums out with an error 28 (for example osmconvert Error: could not jump in file: c:\Windows-Wahoo-Map-Creator-Osmosis\Output\outFileFiltered-asia.o5m, osmconvert Exit: 28) on really large files. According to this can be prevented by either changing the osmconvert source and recompiling or by using a 64bit version of zlib. I have just started looking into this but to be honest am more then a little out of my depth here.
    Basically I can generate (tiled) maps of every country now except some continents.

    Not that it matters for this discussion but I am doing a osm2pgsql import of the planet file now to see if it performs better then the unusable osmosis postgresql import (painstakingly slow queries) And I an trying to get Valhalla working using a docker container to generate routing tiles with elevation info.
    Doing nothing and having fun you know… 😉

    #44952
    Avatar photoChristianK
    Keymaster

    Wel, your PC is OK, much better than my current production system.

    osmconvert is faulty on windows, esp. using zlib – no way to resolve this.
    An OSM User compiled a special version for me w/o zlib that can handle –complete-relations.

    osmconvert is the main reason why I switched to Win10 – her i run osmconvert on WSL/Debian (this really works)
    osmfilter on windows behaves strange in special cases when comparing numeric values -> WSL resolves this.

    PSql: No experience with this, just running some scripts for coastlines w/o really knowing whats happening inside.

    #44956
    ErwinB
    Participant

    Thanks for the info Christiaan.
    If you know/remember… I have been searching all day but I can’t even find a trace of a binary without zlib being available. You would not still be having it by any change?

    I don’t think I am doing any numeric comparisons with osmconvert. I am just doing:
    osmconvert -v –hash-memory=2500 –complete-ways –complete-multipolygons –complete-boundaries –drop-author –drop-version map_file -o=outFileo5m
    and
    osmconvert -v –hash-memory=2500 -b=tile[“left”], tile[“bottom”], tile[“right”],tile[“top”] –complete-ways –complete-multipolygons –complete-boundaries filtered_map_file -o=outFile

    I tried the binary which is supposed to have large file support from here. and a binary I found of the 0.8.10 osmconvert source but that also does not work. Switching to wsl would mean abandoning all the work I have done so far to create a “windows” platform.
    I am using pyosmium-up-to-date in wsl ubuntu now to keep my local maps updated though 🙂

    #44958
    mbe57
    Moderator

    The nice thing about WSL is its integration with W10. You can easily “outsource” only some jobs to WSL and then take the results back from there into your W10 workflow. Same file system, different views. Easy going.

    #44960
    Avatar photoChristianK
    Keymaster

    You would not still be having it by any change?

    Please drop a note when you downloaded this file.

    OK, downloaded by user and removed.

    #44963
    ErwinB
    Participant

    Thanks a million Christian, downloaded and testing right now!
    It has just processed the first tiles without error messages, waiting for it to finish… Splitting Florida from the US is slow… Certainly when Valhalla is running in parallel using 100% cpu and 80% memory in a Docker container… had a bit of a scare because the results where blank maps but figured out these where still the old tiles 😉 so re-running… and… YES, the US north-south and east-west layout of roads is way weird to see but it is WORKING!!! Thank you, thank you, thank you!!!


    @mbe57
    : undoubtedly but I/we are trying to build something every mor…. ehh windows user can use. Involving wsl in my current python program is also above my knowledge level which probably makes me a… windows user 😉 but that’s the way it is…

    #44966
    Avatar photoChristianK
    Keymaster

    Do you have managed to install GDAL-bindings for Python on Windows?
    The only way I know is to install whole FWTOOLS with Python2/Gdal inside

    #44970
    ErwinB
    Participant

    I believe I do and I believe I am not using it 😉
    At the time I certainly did not know what I was doing but I installed it to get my hands on ogr2ogr.
    Fortunately I kind of documented what I did:
    ——————–
    To be able to run the program you need to have Python > 3.5 installed.
    Furthermore you need to have ogr2ogr installed.
    I downloaded gdal (which contains ogr2ogr) from:
    http://www.gisinternals.com/release.php
    Following the guide here:
    https://sandbox.idre.ucla.edu/sandbox/tutorials/installing-gdal-for-windows
    This guide is good but misses a needed environment variable:
    PROJ_LIB “C:\Program Files\GDAL\projlib”
    See:
    https://stackoverflow.com/questions/56764046/gdal-ogr2ogr-cannot-find-proj-db-error

    I myself have Python 3.7.4 installed and used gdal-302-1928-core.msi
    and GDAL-3.2.1.win32-py3.7.msi
    The later looks like it wants to install Python but it is actually asking
    you for the location where Python is installed.
    ———————–
    Now I have a newer 64 bit version installed and Python 3.9

    1 user thanked author for this post.
    #45997
    bm.ffb
    Participant

    I am also struggling to get ogr2osm running on a Win10 PC. I have installed:
    Python 3.9
    gdal-304-1928-x64-core.msi
    GDAL-3.4.0.win-amd64-py3.9.msi
    and used py -m pip install ogr2osm-1.1.1-py3-none-any.whl
    A check shows:

    D:\>py -m pip list
    Package    Version
    ---------- -------
    GDAL       3.4.0
    lxml       4.6.4
    ogr2osm    1.1.1
    pip        21.2.4
    setuptools 58.1.0

    So, it looks ok. But in using this command I get an error:

    D:\>py -m ogr2osm D:\cres_land.shp -o test.osm
    Traceback (most recent call last):
      File "C:\Python39\lib\runpy.py", line 188, in _run_module_as_main
        mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
      File "C:\Python39\lib\runpy.py", line 147, in _get_module_details
        return _get_module_details(pkg_main_name, error)
      File "C:\Python39\lib\runpy.py", line 111, in _get_module_details
        __import__(pkg_name)
      File "C:\Python39\lib\site-packages\ogr2osm\__init__.py", line 16, in <module>
        from .ogr_datasource import OgrDatasource
      File "C:\Python39\lib\site-packages\ogr2osm\ogr_datasource.py", line 15, in <module>
        from osgeo import ogr
      File "C:\Python39\lib\site-packages\osgeo\ogr.py", line 245, in <module>
        import osr
    ModuleNotFoundError: No module named 'osr'

    Does anyone has experience on running 2 versions of Python on the same machine? Because I use Python 2.7 for phyghtmap (running fine) and now Python 3.9 for ogr2osm?
    Any help would be appreciated.

Viewing 15 posts - 1 through 15 (of 16 total)
  • You must be logged in to reply to this topic.