September 2011 - mkgmap-dev - The mkgmap lists

Coastline issues - analysis and possible solution
by Bartosz Fabianowski 15 Oct '11

15 Oct '11

Hi list I spent the last 12 hours debugging sea generation problems in Europe. After digging through a lot of mkgmap and splitter code, I believe I understand the source of the issues now. Since data is processed in tiles, the sea generator will often encounter coastlines clipped at the tile boundaries. There are heuristics in the code that should produce valid sea multipolygons for tiles with incomplete coastlines as well. However, the heuristics fail for the Geofabrik extracts of several European countries. Here is what I believe the problem to be: The sea generator correctly identifies incomplete coastlines. It then attempts to determine for each such coastline whether it was clipped by a tile boundary. It is this test that fails for some countries, leading to missing or inverted sea. The source of this issue can be traced back to the splitter. Here, tile boundaries are expanded to coarse multiples of Garmin map units. Tiles therefore become larger than originally requested. If map data for the entire expanded area is available, this is OK. But when dealing with country extracts, there may be no data available for some of the expanded regions. Here is an ASCII art example, a rectangular extract with a coastline inside, shown as a double line: +---------+ | | | | |=========| | | +---------+ If this data is fed to mkgmap directly, the sea generator will correctly determine that the coastline was clipped by the left and right tile boundaries. If the data is passed through the splitter first, even if no splitting into multiple tiles should be needed, the tile boundaries are rounded to the multiples of Garmin map units mentioned above. The tile may thus grow: +-----------+ | | | | | | | ========= | | | | | +-----------+ The problem is that now, the coastline no longer touches the tile boundaries. The mkgmap sea generator is confused by this and produces invalid or no sea at all. An obvious but incorrect solution is for the sea generator to check whether the coastline reaches the *original* tile boundary, not the rounded one. This would require the splitter to pass the original boundary to mkgmap along with each tile. The real issue with this solution would be that even if mkgmap correctly detects a clipped coastline, there is no valid data between the original and rounded boundaries. To construct a proper sea multipolygon, mkgmap would have to synthesize a coastline in that space, likely introducing ugly artifacts. A correct solution, already implemented in mkgmap today, is to read coastlines from a separate file. The file should contain coastlines that extend at least as far as the rounded tile boundaries. From the mail archives, it appears that WanMil has identified the same issue before and proposed this very solution [1]. Coastlines for a larger area can be extracted using: osmosis \ --rb larger_area.osm.pbf \ --tf accept-ways natural=coastline \ --tf reject-relations \ --used-node \ --wb coastlines.osm.pbf omitmetadata=true The coaslines.osm.pbf file can then be fed to mkgmap via the --coastlinefile option. While WanMil described this solution in December 2010 already, it seems to have been buried in the mailing list without becoming common knowledge. Another mailing list post [2] shows that the --coastlinefile had been broken for a while without anyone noticing, confirming that it is not in common use. After spending 12 hours on this today, I hope that others will benefit from this write-up instead of having to repeat my odyssey. Anyone wanting to use the --coastlinefile option will require coastlines for a larger region than the extract they are processing. While the option does allow several files to be specified whose contents are then concatenated, it is easiest to load a single file with coastlines for a larger area. I have prepared two such files, one from today's Geofabrik europe.osm.pbf extract, the other from the most recent planet-latest.osm.pbf. Both files can be found at [3]. I am providing these files so that anyone working with extracts can download compact coastlines only instead of having to extract them from the huge europe.osm.pbf or planet-latest.osm.pbf files. Right now, the files were generated once-off. If they are found to be useful by others, I am happy to automate the process and make updated coastline files available on a regular basis. Finally, a word of warning: When using coastlines_planet.osm.pbf, mkgmap needs a *lot* of memory. I found that even for small tiles, 4.5GB of RAM are consumed by the Java process. With coastlines_europe.osm.pbf, the memory consumption is about 2.5GB. - Bartosz [1] http://www.mkgmap.org.uk/pipermail/mkgmap-dev/2010q4/009636.html [2] http://www.mkgmap.org.uk/pipermail/mkgmap-dev/2011q1/010138.html [3] http://www.fabianowski.eu/osm/coastlines/

7 27

Memory Consumption on Index Creation
by Felix Hartmann 15 Oct '11

15 Oct '11

1. I noticed that (at least for my style-file) 7500MB of RAM available does not suffice to compile a map of Europe including index! However if I recreate the index in another go, then it is no problem. How comes? The index creation seems to be completly independent of the maptiles creation. Is mkgmap leaving stuff inside the memory, that is in reality not needed anymore? 2. Also as running out of memory primarily happens on index creation, it would be great if mkgmap writes out the index under a temporary name, and once it has passed correctly, renames the file. Else it is not possible to easily assess in a bash/batch script whether mkgmap completed the map correctly or not (and hence one uploads broken maps if everything automated). Or at least I don't see any way to find out whether it passed correctly or not.

6 26

Routing not working
by Carlos Dávila 15 Oct '11

15 Oct '11

I have compiled a map of India from Geofabrik extract, using the commands below, similar to the ones I use for other countries. The map compiles without errors but routing doesn't work on MapSource, only a straight line is drawn from point A to B. I have tried with different days extracts with the same result. I also tried reducing the parameters in the mkgmap call to the minimum, but still the same result. Any idea why routing is not working? java -Xmx1500M -jar splitter.jar --max-nodes=1400000 --mapid=55180001 india.osm.pbf java -Xmx1200m -enableassertions -Dlog.config=logging.properties -jar mkgmap.jar --max-jobs --generate-sea=polygons,extend-sea-sectors --route --tdbfile --latin1 --code-page=1252 --gmapsupp --country-name=India --country-abbr=IND --area-name=India --family-name="OSM India" --family-id=18 --product-id=1 --series-name="OSM-India" --overview-mapname=55180000 --index --ignore-maxspeeds --remove-short-arcs --merge-lines --add-pois-to-areas --adjust-turn-headings --report-similar-arcs --link-pois-to-ways --location-autofill=bounds,is_in,nearest --drive-on-left --check-roundabouts --check-roundabout-flares --style=mio -c india.args

3 12

OSM maps on the new eTrex 30
by Frank Fesevur 06 Oct '11

06 Oct '11

Hi, Does anybody already has any experience with a mkgmap generated map on the brand new Garmin eTrex 20 or 30? I know these models are available for just a couple of days now, but I'm seriously considering buying a eTrex 30 any time soon. But I only buy it when the OSM based maps work on them. Maybe I can take a gmapsupp.img with me to the shop and try myself before I buy it, but if anyone already has any experience I would really like to know. Regards, Frank

6 17

Commit: r2042: Correct the sorting of streets in mdr20 when a city
by svn commit 02 Oct '11

02 Oct '11

Version 2042 was commited by steve on 2011-09-30 14:22:50 +0100 (Fri, 30 Sep 2011) Correct the sorting of streets in mdr20 when a city is split over more than one tile. The valid sorting takes a lot more memory. If this is a problem the please use the index-memory branch, until that is merged.

2 1

Too many pois on multipolygon areas
by Minko 28 Sep '11

28 Sep '11

I have enabled the add-pois-to-areas option, because this way I can search for a certain tag that is placed on an area like landuse=*, tourism=* or building=yes. But sometimes those pois are cluttering the map, especially if this area is a type=multipoygon relation. Take for example this multipolygon relation: http://www.openstreetmap.org/browse/relation/949481 tourism=zoo is tagged only once on the outer way: http://www.openstreetmap.org/browse/way/61046378 However, the multipolygon function of mkgmap splits this area up in several parts (to render the inner islands, which is fine) but it also creates too many pois by the add-pois-to-areas for this one tourism=zoo area: http://img546.imageshack.us/img546/5763/zoow.jpg I'd like to see just one elephant icon instead of several. Is it possible to execute the add-pois-to-areas action before the multipolygon action takes place? Or is it possible to set some rules in the style file to make some exceptions for areas with type=multipolygon?

5 7

Re: [mkgmap-dev] Splitter doesn't process pbf data created with osmosis 0.39
by Michael Prinzing 25 Sep '11

25 Sep '11

On Sun, 25 Sep 2011 12:31:07 +0100, Steve Ratcliffe wrote: >> Could please someone look into the splitter code and check if the >> corrections that were done to osmosis have to by applied to the >> splitter, too? > >OK the reported problem should be fixed now. Thank you, Steve. It's working fine now. Michael

1 0

Commit: r2038: Allow decimal numbers in style file rules and in tag values.
by svn commit 25 Sep '11

25 Sep '11

Version 2038 was commited by steve on 2011-09-25 21:24:57 +0100 (Sun, 25 Sep 2011) Allow decimal numbers in style file rules and in tag values.

1 0

Commit: r2037: net read bug fix mask out top bits before attempting
by svn commit 25 Sep '11

25 Sep '11

Version 2037 was commited by steve on 2011-09-25 20:54:31 +0100 (Sun, 25 Sep 2011) net read bug fix mask out top bits before attempting to use it as a pointer. The bug happens all the time, but usually has no harmful effect. Can cause a crash however.

1 0

Re: [mkgmap-dev] Splitter doesn't process pbf data created with osmosis 0.39
by Michael Prinzing 25 Sep '11

25 Sep '11

On Tue, 20 Sep 2011 22:14:36 +0200, Michael Prinzing wrote: >On Mon, 19 Sep 2011 17:53:03 +0200, Michael Prinzing wrote: > >>there seems to be a problem with the latest splitter and data in pbf >>format if it was created with osmosis 0.39. > >After some further testing, it now looks like a problem with osmosis >only. If I am converting the pbf data created with osmosis 0.39 back to >xml then there also is an empty bounding box that did not exist in the >original data and that does not arise if the pbf file was built with >osmosis 0.38. > >I will report this on the mailing list for osmosis. Well, there really was a problem with osmosis and it is fixed in the latest osmosis-snapshot. This version now can read pbf data created with osmosis 0.39 or newer without seeing an empty bounding box where no one is present. However, the bug was in the pbf reader, not with the creation of pbf files. This means the created pbf data did not change, and therefore the problem with the splitter still persists. Igor Podolskiy who fixed the problem in osmosis also looked into the splitter code. He assumes there could be something similar wrong as it was in osmosis. Please see the thread beginning with http://lists.openstreetmap.org/pipermail/osmosis-dev/2011-September/001125.… for details. Here again how to reproduce the problem with the splitter: I am saving the following xml data (not containing a bounding box!) to data.osm: -------------------------------------------------------------------- <?xml version='1.0' encoding='UTF-8'?> <osm version="0.6" generator="Osmosis 0.35"> <node id="0000004406" version="1" timestamp="2011-09-11T03:07:14Z" lat="51.6325833" lon="6.2575833"/> <node id="0000004407" version="1" timestamp="2011-09-11T03:07:14Z" lat="51.6334167" lon="6.2584167"/> <node id="0000004408" version="1" timestamp="2011-09-11T03:07:14Z" lat="51.63425" lon="6.25925"/> <node id="0000004409" version="1" timestamp="2011-09-11T03:07:14Z" lat="51.6334167" lon="6.2600833"/> <node id="0000004410" version="1" timestamp="2011-09-11T03:07:14Z" lat="51.6325833" lon="6.2600833"/> <node id="0000004411" version="1" timestamp="2011-09-11T03:07:14Z" lat="51.63175" lon="6.25925"/> <node id="0000004412" version="1" timestamp="2011-09-11T03:07:14Z" lat="51.63175" lon="6.2584167"/> <way id="1" version="1" timestamp="2011-09-11T03:07:14Z"> <nd ref="0000004406"/> <nd ref="0000004407"/> <nd ref="0000004408"/> <nd ref="0000004409"/> <nd ref="0000004410"/> <nd ref="0000004411"/> <nd ref="0000004412"/> <nd ref="0000004406"/> <tag k="ele" v="-220"/> <tag k="contour" v="elevation"/> <tag k="contour_ext" v="elevation_medium"/> </way> </osm> -------------------------------------------------------------------- Then, I am converting this xml data to pbf with: osmosis --read-xml file="data.osm" --write-pbf omitmetadata=true file="data_nnn.osm.pbf" Depending on the osmosis version I am using for this, I am getting the following: With osmosis 0.39: -------------------------------------------------------------------- 00000000: 0000 000d 0a09 4f53 4d48 6561 6465 7218 ......OSMHeader. 00000010: 2f10 231a 2b78 9c53 e2f3 2fce 0d4e ce48 /.#.+x.S../..N.H 00000020: cd4d d40d 33d0 3353 e272 49cd 2b4e f5cb .M..3.3S.rI.+N.. 00000030: 4f49 2d6e 6264 31d0 33b6 0400 bcf0 0a2d OI-nbd1.3......- 00000040: 0000 000c 0a07 4f53 4d44 6174 6118 8c01 ......OSMData... 00000050: 109f 011a 8601 789c e372 e062 e012 48cd ......x..r.b..H. 00000060: 492d 4b2c c9cc cf8b cf4d 4dc9 2ccd e562 I-K,.....MM.,..b 00000070: 068a 70b1 27e7 e795 e497 1671 7143 19f1 ..p.'......qqC.. 00000080: a915 255c 2cba 4646 065c 9c70 2d42 9642 ..%\,.FF.\.p-B.B 00000090: e65c 1c6f 5c98 c0c0 4974 52fb 9637 cc73 .\.o\...ItR..7.s 000000a0: 9a18 6735 31ce 6c62 9c0d 2619 bc44 365d ..g51.lb..&..D6] 000000b0: be66 0d11 0522 8699 6061 2149 2971 0e46 .f..."..`a!I)q.F 000000c0: 2166 2666 1629 6656 3646 274e 9831 dc1d !f&f.)fV6F'N.1.. 000000d0: 8c29 1318 5fb0 0300 fd3e 2ca3 .).._....>,. -------------------------------------------------------------------- With osmosis-snapshot: -------------------------------------------------------------------- 00000000: 0000 000d 0a09 4f53 4d48 6561 6465 7218 ......OSMHeader. 00000010: 3b10 2f1a 3778 9c53 e2f3 2fce 0d4e ce48 ;./.7x.S../..N.H 00000020: cd4d d40d 33d0 3353 e272 49cd 2b4e f5cb .M..3.3S.rI.+N.. 00000030: 4f49 2d6e 6214 08f6 730c 08f6 f00f d12d OI-nb...s......- 00000040: cdcb cecb 2fcf 0300 5893 0f1c 0000 000c ..../...X....... 00000050: 0a07 4f53 4d44 6174 6118 8c01 109f 011a ..OSMData....... 00000060: 8601 789c e372 e062 e012 48cd 492d 4b2c ..x..r.b..H.I-K, 00000070: c9cc cf8b cf4d 4dc9 2ccd e562 068a 70b1 .....MM.,..b..p. 00000080: 27e7 e795 e497 1671 7143 19f1 a915 255c '......qqC....%\ 00000090: 2cba 4646 065c 9c70 2d42 9642 e65c 1c6f ,.FF.\.p-B.B.\.o 000000a0: 5c98 c0c0 4974 52fb 9637 cc73 9a18 6735 \...ItR..7.s..g5 000000b0: 31ce 6c62 9c0d 2619 bc44 365d be66 0d11 1.lb..&..D6].f.. 000000c0: 0522 8699 6061 2149 2971 0e46 2166 2666 ."..`a!I)q.F!f&f 000000d0: 1629 6656 3646 274e 9831 dc1d 8c29 1318 .)fV6F'N.1...).. 000000e0: 5fb0 0300 fd3e 2ca3 _....>,. -------------------------------------------------------------------- Both files can be converted back to xml only with the latest osmosis from snapshot. osmosis 0.39 or older are reading an empty bounding box from the pbf data. This was the bug that was fixed in osmosis now. When I am trying to split one of these files, the splitter sees an empty bounding box and of course does not write any data to the output: -------------------------------------------------------------------- cache= description= geonames-file= legacy-mode=false mapid=63240001 max-areas=255 max-nodes=1600000 max-threads=1 (auto) mixed=false no-trim=false output=unset output-dir= overlap=2000 resolution=13 split-file= status-freq=120 write-kml= Elapsed time: 0s Memory: Current 15MB (1MB used, 14MB free) Max 247MB Time started: Fri Sep 23 23:27:55 CEST 2011 Map is being split for resolution 13: - area boundaries are aligned to 0x800 map units - areas are multiples of 0x1000 map units wide and high Processing data_039.osm.pbf Bounding box 0.0 0.0 0.0 0.0 in 1 file Time: Fri Sep 23 23:27:56 CEST 2011 Exact map coverage is (0.0,0.0) to (2.1457672119140625E-5,2.1457672119140625E-5) Trimmed and rounded map coverage is (0.0,0.0) to (2.1457672119140625E-5,2.1457672119140625E-5) Splitting nodes into areas containing a maximum of 1.600.000 nodes each... 0 areas: Writing out split osm files Fri Sep 23 23:27:56 CEST 2011 Processing 0 areas in a single pass Time finished: Fri Sep 23 23:27:56 CEST 2011 Total time taken: 0s -------------------------------------------------------------------- If however I am creating the pbf file with osmosis 0.38, then the pbf data looks like the following: -------------------------------------------------------------------- 00000000: 0000 000c 0a07 4f53 4d44 6174 6118 8c01 ......OSMData... 00000010: 109f 011a 8601 789c e372 e062 e012 48cd ......x..r.b..H. 00000020: 492d 4b2c c9cc cf8b cf4d 4dc9 2ccd e562 I-K,.....MM.,..b 00000030: 068a 70b1 27e7 e795 e497 1671 7143 19f1 ..p.'......qqC.. 00000040: a915 255c 2cba 4646 065c 9c70 2d42 9642 ..%\,.FF.\.p-B.B 00000050: e65c 1c6f 5c98 c0c0 4974 52fb 9637 cc73 .\.o\...ItR..7.s 00000060: 9a18 6735 31ce 6c62 9c0d 2619 bc44 365d ..g51.lb..&..D6] 00000070: be66 0d11 0522 8699 6061 2149 2971 0e46 .f..."..`a!I)q.F 00000080: 2166 2666 1629 6656 3646 274e 9831 dc1d !f&f.)fV6F'N.1.. 00000090: 8c29 1318 5fb0 0300 fd3e 2ca3 .).._....>,. -------------------------------------------------------------------- The obvious difference is that there is no "OSMHeader" included. This file can be processed by the splitter without an empty bounding box showing up. Because of this the problem with the splitter does not occur with pbf files created with osmosis 0.38 or older. Could please someone look into the splitter code and check if the corrections that were done to osmosis have to by applied to the splitter, too? Thank you, Michael

2 2