August 2009 - mkgmap-dev - The mkgmap lists

New splitter version available
by Chris Miller 08 Aug '09

08 Aug '09

I've checked in a new version of the tile splitter. It's not yet available for download on the splitter homepage, but until then you can either check it out of Subversion at http://svn.mkgmap.org.uk/splitter/trunk or download a prebuilt version from http://redyeti.net/splitter.jar Here is what has changed: - Reduced memory requirements, especially during the generation of areas.list - About 10% faster overall - Uses a slightly faster and leaner XML parser (XPP instead of SAX) - Removed the 4-area limit for relations - Improved the log messages to give more information about what is happening during the split - Can handle an arbitrary number of areas. Note however that each multiple of 255 areas requires a separate pass over the XML. This means a large number of areas will slow down the splitting due to the additional passes. This is still a lot better than the previous behaviour which was for the splitter to silently put elements into the wrong areas. It should be possible to split the whole planet file with far less memory now. The amount of memory required is roughly 2.5GB(?) for the initial areas.list generation, and then roughly proportional to (--max-nodes * 255). The smaller you make --max-nodes, the less memory the split will require but also the longer it will take (since a full parsing run over the planet file is required for each multiple of 255 areas). I've just kicked off the following before I go to bed, will be interesting to see if it manages to complete by the morning without running out of memory: java -Xmx4000m -jar splitter.jar --mapid=1 --max-nodes=1200000 planet-090715.osm Enjoy! Chris

2 3

Commit: r1124: Make it possible to append to tags during iteration.
by svn commit 08 Aug '09

08 Aug '09

Version 1124 was commited by steve on 2009-08-08 12:10:14 +0100 (Sat, 08 Aug 2009) Make it possible to append to tags during iteration.

1 0

New splitter version available
by Chris Miller 08 Aug '09

08 Aug '09

I've checked in some more updates to the splitter. You can either check it out of Subversion and build it yourself, or download a copy from http://redyeti.net/splitter.jar. Here's what's new/changed: - Removed the 4 areas per way restriction. Now all ways will be split correctly regardless of how many areas they are in. - Added a --max-areas parameter. This allows you to trade off speed against memory usage. - Improved the logging to make it clearer what is happening, and to provide feedback of progress during the split. - The default ant target is now 'dist'. - A couple of minor performance improvements. Some further explanation of --max-areas is probably required. By default this is set to 255, meaning that up to 255 areas will be processed on a single pass over the XML when splitting. However if your map is going to split into say 387 areas, rather than doing one pass of 255 areas and a second of 132 the splitter will balance the passes at 184 and 183 areas each. This is because the amount of memory used by a given pass is proportional to the number of areas being processed multiplied by --max-nodes, so the lower the number of areas for a given pass the better. There is no performance cost associated with this balancing. As for the 4 areas per way restriction... I managed to come up with a way to handle the few ways that exceed this limit with some special-case code. There is a slight performance and memory cost associated with this, but with the current data it is negligible. If in the futher there end up being a huge number of ways covering more than 4 areas we might have to rethink how this works, but for the medium term at least I think we'll be fine. As always, please give this new version a try and let me know what you think or have any problems/questions. Chris

2 2

Merging roundabouts that are split because of bus route relations
by Marko Mäkelä 07 Aug '09

07 Aug '09

In the Finnish forum <http://forum.openstreetmap.org/viewtopic.php?id=3763>, user Juhe noticed that the Nüvi 205 gave directions as if a roundabout were a series of junctions (as far as I understood). But his fix was wrong: it is not OK to merge the roundabout in the OSM source, because only part of the roundabout belongs to a bus route relation. Here is the changeset, which has already been reverted according to user Daeron: http://www.openstreetmap.org/browse/changeset/2026393 Juhe writes that he got his map from <http://www.raumbezug.eu/ag/internet/osmGarmin.htm>, but he has since then compiled a map with mkgmap based on my scripts <http://www.polkupyoraily.net/osm/>. Would it be possible to merge such roundabouts in mkgmap or in a preprocessing script? I guess that this would require a style rule plugin that would merge ways based on a condition. Marko

1 0

Styling for the power user: filtering contour lines
by David 07 Aug '09

07 Aug '09

To Christian Gawron Thank you for your tip. I will try to use it for the next release of my map. David > Hi Daniel, > > you can use regular expressions to filter contour lines by elevation > - here is what I do in my style file: > > # Contours take their name from the elevation setting. > # multiples of 200m > contour=elevation & ele ~ '\d*[02468]00' > { name '${ele|conv:m=>ft}'; } > [0x22 resolution 18] > # multiples of 50m > contour=elevation & ele ~ '\d+[05]0' > { name '${ele|conv:m=>ft}'; } > [0x21 resolution 21] > contour=elevation > { name '${ele|conv:m=>ft}'; } > [0x20 resolution 23] > > You could use '\d+[27]5' for your purpose. > > Best wishes > Christian

1 0

Memory limits for mkgmap and splitter
by Paul Ortyl 07 Aug '09

07 Aug '09

Hi, I used to process the europe.osm (from geofabrik.de) file using splitter. It does not work (in finited time) any more because of the 4GB limit of my hardware RAM. I have a suggestion, which comes from my experience years ago, where I had to crunch gigabytes of textual data using PERL. In the old days I had written some scripts, which used associative arrays for data storage. I had to have access to the whole data "at once" for report generation. The trick was to test the processing on small amount of data and for the big run "tie" the "associative array" functionality to BerkeleyDB. The processing was slow, but possible without spending any more time on searching for other solution. Would it be possible to allow an option, where data storage instead of internal structures would be stored partly on disk and partly in database engine cache (see BerkeleyDB implementation). I am aware, that the processing will be 10 or 100 times slower, but will not take my workstation down grabbing all RAM and swap space and could do that in background leaving me substantial amount of RAM and one CPU core free for other tasks. In PERL it is builtin "tie" functionality, I have, however, no idea what is the used data structure in mkgmap and splitter and how to translate the trick into Java. If you think that the change is trivial and point me to the critical section I might see if I get it implemented. Any ideas? Paul -- Don't take life too seriously; you will never get out of it alive. -- Elbert Hubbard

10 34

Commit: r1123: Add a test for style rules.
by svn commit 07 Aug '09

07 Aug '09

Version 1123 was commited by steve on 2009-08-07 15:26:00 +0100 (Fri, 07 Aug 2009) Add a test for style rules.

1 0

Styling for the power user
by David 07 Aug '09

07 Aug '09

To Mark Burton, The new features you added for styling is a great step for my map. The last step is how to filter each contour line which height finishes by 25 or 75 (I don't know how to do it with style rules). These contours are only used for low level of details. At higher LOD, contours with a step of 10m are displayed (to build such a layer I need cgpsmapper). Thank you for your work, David

3 2

test map and RoadTrip?
by Greg Troxel 06 Aug '09

06 Aug '09

I am trying to make a testmap and look at it in RoadTrip. The creation process goes smoothly, and the gmapibuilder output looks reasonable. I can use MapManager to add the map, but then RoadTrip crashes when I try to select it. A similar process to build real maps (from cloudmade extracts, with the splitter) with the same mkgmap.jar produces maps that work in RoadTrip. I have not yet tried this map on a gps receiver. Is there anything obvious I am doing wrong? echo "test-map:all-elements" export BASE_LAT=42.3 export BASE_LONG=-71.1 time \ java \ -enableassertions \ -Xmx2048m \ -jar mkgmap.jar \ --overview-mapname=63500000 \ --mapname=63500001 \ --description="OSM_test" \ --country-abbr="US" \ --country-name="United States" \ --region-abbr="MA" \ --region-name="Massachusetts" \ --series-name="OSM_test_series" \ --family-id=402 \ --family-name="OSM_test_family" \ --product-id=14 \ --series-name="OSM_test_series" \ --area-name="OSM_test_area" \ --tdbfile \ test-map:all-elements > OUT.04.mkgmap 2>&1 echo "options not used:" \ --remove-short-arcs \ --add-pois-to-areas \ --route \ mkdir -p GMAPI-TEST echo "GMAPI-TEST:" time \ /Users/gdt/SOFTWARE/GMAPIBUILDER/gmapi-builder/gmapi-builder.py \ -o GMAPI-TEST \ -t 63500000.tdb \ -b 63500000.img \ -v 6350*.img \ > OUT.05.gmapi-test 2>&1

1 0

compiled mkgmap binary with contour option
by maning sambale 06 Aug '09

06 Aug '09

Any download link of mkgmap binary with the contour generation options? I can't find r1079 in the mkgmap download site -- cheers, maning ------------------------------------------------------ "Freedom is still the most radical idea of all" -N.Branden wiki: http://esambale.wikispaces.com/ blog: http://epsg4253.wordpress.com/ ------------------------------------------------------

4 9