November 2012 - mkgmap-dev - The mkgmap lists

splitter r236
by GerdP 17 Nov '12

17 Nov '12

Hi all, r236 is now available. Corrections regarding --keep-complete: - do not add members of "parent-only" relations. A p"arent-only" relation is one that has a member which is a relation and which is in the problem-list, but has no problem-ways. - some ways were detected as problem ways but still not written. This sometimes happened when one or more points of the way is normally not written to any output file. - enclosing multipolygons were not always added Improvements - reduced the needed for sorting the temp files, this also reduces heap requirement - recursive relation checks are faster Known problem: When splitting planet, a lot of boundary data is written to tiles that don't need it, e.g. the boundary of USA may be written to tiles in europe. Ciao, Gerd -- View this message in context: http://gis.19327.n5.nabble.com/splitter-r236-tp5736104.html Sent from the Mkgmap Development mailing list archive at Nabble.com.

4 11

Altitude Units for Contours
by Roger Calvert 16 Nov '12

16 Nov '12

Please can someone advise: I have an OSM file which contains contours with elevations in metres. Mapsource seems to interpret these as feet, and then, because I have its altitude setting in metres, converts them. So they all come out low by a factor of 3.3. Is there a way of adjusting for this in the style file? Thanks in advance, Roger -- ------------------------------------------------------------------------ Roger Calvert ------------------------------------------------------------------------

3 5

Handling of site-relations in style
by Henning Scholland 16 Nov '12

16 Nov '12

Hi, site-relation would become more and more importantbecause of detailed mapping of objects in OSM. I think therefore it's necessary to copy taggs from relation to a member with a special role. Is this already possible in mkgmap-style? Otherwise it would be great if something like this could be integrated in syntax of style-file. Eg.: type=site & site=parking { apply for role="perimeter,label" { set amenity=parking } }

2 1

splitter r232
by GerdP 16 Nov '12

16 Nov '12

Hi all, if you are testing the --keep-complete or the --problem-list option, please update to r232. The older versions in the branch are known to have stupid bugs causing extreme memory needs (and possibly long run times). Example: the DACH extract is now handled on a 32bit machine with -Xmx1600m without problems. Ciao, Gerd -- View this message in context: http://gis.19327.n5.nabble.com/splitter-r232-tp5735799.html Sent from the Mkgmap Development mailing list archive at Nabble.com.

8 19

Splitter Error
by Roger Calvert 14 Nov '12

14 Nov '12

In running a macro I have used many times before with a new download from geofabrik, splitter gave me the following error. (I realise that there is no need to split the file in this particular case, but it has never caused problems before.) I am not sure of the version of splitter I am using, but the files are dated 11/11/2011. Any advice would be appreciated. Roger ------------------------------------------------------------------------------- cache=DataTest description=Test geonames-file=..\Resources\cities5000.zip legacy-mode=false mapid=60555001 max-areas=255 max-nodes=2500000 max-threads=4 (auto) mixed=false no-trim=false output=pbf output-dir= overlap=2000 resolution=13 split-file= status-freq=120 write-kml= Elapsed time: 0s Memory: Current 59MB (2MB used, 57MB free) Max 2666MB Time started: Wed Nov 14 10:48:30 GMT 2012 Map is being split for resolution 13: - area boundaries are aligned to 0x800 map units - areas are multiples of 0x1000 map units wide and high Processing DataTest\cumbria.osm.pbf Bounding box -3.907471 53.90063000000001 -2.1595880000000003 55.18863 in 1 file Time: Wed Nov 14 10:48:31 GMT 2012 Exact map coverage is (53.90062093734741,-3.907463550567627) to (55.188617706298 83,-2.159585952758789) Trimmed and rounded map coverage is (53.876953125,-3.9111328125) to (55.1953125, -2.1533203125) Splitting nodes into areas containing a maximum of 2,500,000 nodes each... Area (53.876953125,-3.9111328125) to (55.1953125,-2.1533203125) contains 699,912 nodes. DONE! 1 areas: Area 60555001 covers (0x265000,0xfffd3800) to (0x274000,0xfffe7800) GB-Carlisle Writing out split osm files Wed Nov 14 10:48:32 GMT 2012 Processing 1 areas in a single pass (53.876953125,-3.9111328125) to (55.1953125,-2.1533203125) Starting pass 1 of 1, processing 1 areas (60555001 to 60555001) Making SparseMultiMap Making SparseMultiMap Processing DataTest\cumbria.osm.pbf Bounding box -3.907471 53.90063000000001 -2.1595880000000003 55.18863 Exception in thread "main" java.lang.IndexOutOfBoundsException: Index (31257321) is greater than or equal to list size (31250001) at it.unimi.dsi.fastutil.objects.ObjectArrayList.get(ObjectArrayList.jav a:258) at uk.me.parabola.splitter.SparseInt2ShortMapInline.put(SparseInt2ShortM apInline.java:128) at uk.me.parabola.splitter.SparseInt2ShortMultiMap$Inner.put(SparseInt2S hortMultiMap.java:81) at uk.me.parabola.splitter.SparseInt2ShortMultiMap.put(SparseInt2ShortMu ltiMap.java:31) at uk.me.parabola.splitter.SplitProcessor.writeNode(SplitProcessor.java: 209) at uk.me.parabola.splitter.SplitProcessor.processNode(SplitProcessor.jav a:118) at uk.me.parabola.splitter.BinaryMapParser.parseDense(BinaryMapParser.ja va:49) at crosby.binary.BinaryParser.parse(BinaryParser.java:124) at crosby.binary.BinaryParser.handleBlock(BinaryParser.java:68) at crosby.binary.file.FileBlock.process(FileBlock.java:135) at crosby.binary.file.BlockInputStream.process(BlockInputStream.java:34) at uk.me.parabola.splitter.Main.processMap(Main.java:403) at uk.me.parabola.splitter.Main.writeAreas(Main.java:368) at uk.me.parabola.splitter.Main.split(Main.java:190) at uk.me.parabola.splitter.Main.start(Main.java:118) at uk.me.parabola.splitter.Main.main(Main.java:107) -- ------------------------------------------------------------------------ Roger Calvert ------------------------------------------------------------------------

4 4

splitter improvements
by UliBaer 14 Nov '12

14 Nov '12

> Hi Uli, > > I fixed the error with r232 which I've just committed. > Thanks for reporting ! > > Gerd Hi Gerd, you're welcome! :) Only minutes ago splitter-r232 just flew thru the germany+ dataset on a 32-bit-environment using -Xmx1376m without a single problem! The problem-candidates-list sizes now to a reasonable 2698KB. Great work, thanks a lot! Kind regards, Uli

1 0

splitter improvements
by UliBaer 14 Nov '12

14 Nov '12

Hi Gerd, i just tried it once again using splitter-r231. This is the error, that occurs: 23.000.000 ways processed... id=175802045 24.000.000 ways processed... id=184296766 Stats for MultiTileProcessor pass 2 endMap Pass2 end SparseBitSet problemRels 173576 (917000 bytes) SparseBitSet neededWays 11057443 (42951840 bytes) SparseBitSet neededNodes 73769812 (212038560 bytes) Number of stored relations: 173576 starting to collect coordinates for 73.769.812 special nodes Exception in thread "main" java.lang.OutOfMemoryError: Java heap space at it.unimi.dsi.fastutil.longs.LongArrayList.<init>(LongArrayList.java:104) at uk.me.parabola.splitter.MultiTileProcessor.endMap(MultiTileProcessor.java:194) at uk.me.parabola.splitter.Main.processMap(Main.java:603) at uk.me.parabola.splitter.Main.writeAreas(Main.java:499) at uk.me.parabola.splitter.Main.split(Main.java:218) at uk.me.parabola.splitter.Main.start(Main.java:144) at uk.me.parabola.splitter.Main.main(Main.java:133) ------------------------------------------------------------------------------------------------------------- Additionally i compared the two problem-candidates-lists: Using my own germany+iceland-dataset the list has a size of 2663KB. In case of the germany+-dataset the list size is 179797KB (!!) :O I think, this explains why there is not enough memory available... ;) Kind regards, Uli

2 1

splitter improvements
by UliBaer 14 Nov '12

14 Nov '12

Hi Gerd, this is the way i create the germany + 50km + iceland .osm.pbf-file: First step is download from geofabrik (only about once a month): ..\bin\wget --no-cache http://download.geofabrik.de/openstreetmap/europe.osm.pbf Then i cut the interesting area(s), i'll attach the corresponding .poly-file: ..\osmconvert europe.osm.pbf -B=..\germany+iceland.poly -o=germany+iceland.osm.pbf I do this process only once a month. Normally i update the germany+iceland.osm.pbf on daily basis with the following command: osmupdate --daily --planet-url=http://planet.openstreetmap.org/replication/ OSM-Data\germany+iceland.osm.pbf OSM-Data\germany+iceland-new.osm.pbf -B=germany+iceland.poly -v This file was fed to splitter on a 32(!)Bit-machine with Xmx1376m without problems. Regards, Uli

1 0

splitter improvements
by UliBaer 14 Nov '12

14 Nov '12

I think the memory requirements are more dependent on the data than on the splitter version: I have a self created data set consisting of germany + ~50 kilometers plus the data from iceland cut with osmconvert.exe from the europe.osm.pbf from geofabrik, which is processed without errors with versions 225, 229, 230 and 231 using 32-Bit-jvm with Xmx1376m. When i use the about the same sized 1.9 GB "germany+"-dataset from here: ( http://140.78.94.22/osm/ ) i can't get thru the data without java heap overflow, even with 64-Bit-jvm given Xmx3000m. I think there is some kind of endless loop or the like, which causes the algorithm to overflow the memory in some circumstances. Anyway, good work until now! I'm eager to test further splitter versions don't requiring the giant overlap-values for routing.

4 6

splitter improvements
by UliBaer 14 Nov '12

14 Nov '12

> Hi Uli, > > can please also tell me the splitter parms that you use? > The speed and memory needs depends directly on the number and the position > of the tiles. > > Gerd Hi Gerd, of course here are the parameters for the splitter call: start /low /b /wait java -enableassertions -Xmx1376m -jar ..\bin\splitter.jar --max-threads=2 --keep-complete --overlap=0 --output=pbf --max-areas=256 --mapid=66%FID%001 --max-nodes=1200000 ..\OSM-Data\%osmfile% I normally use this for 32-Bit-Java on WindowsXP 32-Bit and it works ok with my dataset. If i feed this call with the above mentioned germany+-dataset i get an out of memory error, even if i change the heap to -Xmx3000m on a 64-Bit-Jvm. My own dataset is 1.85GB in size and the germany+ set is 1.88GB in size. So i think they are comparable. The call i use to converrt the germany+.osm.xz to .osm.pbf is as follows: ..\xz -d -c -k -v -v germany+.osm.xz | ..\osmconvert - -o=germany+.osm.pbf --verbose I can upload the file somewhere, but i'm at work and can only access limited pages on the internet and all file sharing sites are blocked. Regards from Wolfsburg - ;) - Uli

2 1