Weird behavior and problem with splitter r161
Hello Splitter doesn't work work anymore.... I am downloading the geofabrik european map to build my own benelux map. for this I cut the european PBF with Osmosis to generate a Benelux PBF. I then "split it" into tiles with the special r161 splitter version that supports PBF as input before running the rest of the processing. This worked and I already generated a usefull map. But since my latest tentative (European data of wednesday and thursday) the splitter stops processing (without any error message) and the generated maps are missing most of their features (ex: roads). The weird behavior I see is that when the splitter starts to process the "way occupancy" it displays "MAP occupancy: 0" and exists immediatly. This should explain why I don't see any roads on my generated maps. This is the output of my splitter: C:\Users\papa\Documents\GPS\OpenStreetMap\be_tiles>java -Xmx2000M -jar ../splitt er-r161/splitter.jar --mapid=52000001 --cache=../be_tiles_cache/ --geonames-file =../cities15000.zip --max-nodes=1000000 ../myBenelux.osm.pbf cache=../be_tiles_cache/ description= geonames-file=../cities15000.zip legacy-mode=false mapid=52000001 max-areas=255 max-nodes=1000000 max-threads=2 (auto) mixed=false no-trim=false overlap=2000 resolution=13 split-file= status-freq=120 write-kml= Elapsed time: 0s Memory: Current 61MB (1MB used, 60MB free) Max 1777MB Time started: Sat Feb 19 08:10:04 CET 2011 Map is being split for resolution 13: - area boundaries are aligned to 0x800 map units - areas are multiples of 0x1000 map units wide and high Processing ../myBenelux.osm.pbf Bounding box 1.801757999 48.908059 7.800292999000001 52.214339 Time: Sat Feb 19 08:10:07 CET 2011 Exact map coverage is (48.908042907714844,1.8017578125) to (52.21432685852051,7. 80029296875) Trimmed and rounded map coverage is (48.9111328125,1.8017578125) to (52.25097656 25,7.7783203125) Splitting nodes into areas containing a maximum of 1.000.000 nodes each... Area (49.482421875,1.8896484375) to (51.15234375,4.1748046875) contains 405.624 nodes. DONE! Area (49.04296875,4.1748046875) to (51.15234375,5.4931640625) contains 733.032 n odes. DONE! Area (51.15234375,1.8017578125) to (52.20703125,4.5263671875) contains 620.613 n odes. DONE! Area (51.15234375,4.5263671875) to (52.20703125,5.4931640625) contains 704.554 n odes. DONE! Area (48.9111328125,5.4931640625) to (49.5263671875,7.7783203125) contains 529.6 28 nodes. DONE! Area (49.5263671875,5.4931640625) to (50.4931640625,7.4267578125) contains 658.5 04 nodes. DONE! Area (50.4931640625,5.4931640625) to (51.1083984375,6.2841796875) contains 477.8 42 nodes. DONE! Area (50.4931640625,6.2841796875) to (51.1083984375,7.1630859375) contains 524.1 02 nodes. DONE! Area (51.1083984375,5.4931640625) to (51.3720703125,7.0751953125) contains 491.5 19 nodes. DONE! Area (51.3720703125,5.4931640625) to (52.0751953125,6.9873046875) contains 619.8 61 nodes. DONE! 10 areas: Area 52000001 covers (0x233800,0x3e800) to (0x23e800,0x54800) DE-Trier Area 52000002 covers (0x245800,0x3e800) to (0x248800,0x50800) DE-Dusseldorf Area 52000003 covers (0x22c800,0x3e800) to (0x233800,0x58800) DE-Saarbrucken Area 52000004 covers (0x22e000,0x2f800) to (0x246000,0x3e800) BE-Brussels Area 52000005 covers (0x23e800,0x3e800) to (0x245800,0x47800) DE-Aachen Area 52000006 covers (0x246000,0x33800) to (0x252000,0x3e800) NL-Utrecht Area 52000007 covers (0x248800,0x3e800) to (0x250800,0x4f800) DE-Duisburg Area 52000008 covers (0x233000,0x15800) to (0x246000,0x2f800) BE-Gent Area 52000009 covers (0x23e800,0x47800) to (0x245800,0x51800) DE-Koeln Area 52000010 covers (0x246000,0x14800) to (0x252000,0x33800) NL-Rotterdam Writing out split osm files Sat Feb 19 08:10:08 CET 2011 Processing 10 areas in a single pass Starting pass 1 of 1, processing 10 areas (52000001 to 52000010) Making SparseMultiMap Making SparseMultiMap Processing ../myBenelux.osm.pbf Bounding box 1.801757999 48.908059 7.800292999000001 52.214339 Making SparseMultiMap Making SparseMultiMap MAP occupancy: 861501 MAP occupancy: 130753 MAP occupancy: 7746 Making SparseMultiMap MAP occupancy: 1744524 MAP occupancy: 242713 MAP occupancy: 12515 MAP occupancy: 248 MAP occupancy: 2603236 MAP occupancy: 372778 MAP occupancy: 23341 MAP occupancy: 645 MAP occupancy: 3439177 MAP occupancy: 523803 MAP occupancy: 36362 MAP occupancy: 658 MAP occupancy: 4301030 MAP occupancy: 658871 MAP occupancy: 39441 MAP occupancy: 658 MAP occupancy: 5178067 MAP occupancy: 776748 MAP occupancy: 44358 MAP occupancy: 827 coords occupancy MAP occupancy: 5765385 MAP occupancy: 839551 MAP occupancy: 46448 MAP occupancy: 1236 ways occupancy MAP occupancy: 0 Time finished: Sat Feb 19 08:10:47 CET 2011 Total time taken: 43s C:\Users\papa\Documents\GPS\OpenStreetMap\be_tiles> - Am I using the latest version of the PBF splitter branch (I also tried the compiled r161-3) ? - Is this a known problem ? - Is something wrong with my parameters ? I was using a windows 7 PC for this. Thanks in advance to any hint or advise you could give me. Jmm Jean-Marc Meessen
Jean-Marc Meessen schrieb am 20.02.2011 14:24:
* Am I using the latest version of the PBF splitter branch (I also tried the compiled r161-3) ? * Is this a known problem ? * Is something wrong with my parameters ?
I was using a windows 7 PC for this.
Thanks in advance to any hint or advise you could give me.
This sound like a known (and already solved) problem, where splitter had problems with files >4GB on windof PCs. In the lates version of r-161 this problem was solved, previous versions of r-161 had this problem. Are you sure, you are using the latest versin of r-161? (My splitter.jar has the date 27.01.2011.) Are smaller maps working ok? Gruss Torsten
The file I am trying to split is fairly small, Torsten. It is about 30kb large (while the european data is 5gb). The splitter version is from end january. Isn't the extract too small and thus the Osmosis extract went wrong ? I am extracting 50km around the borders of Belgium. Jmm 2011/2/20 Torsten Leistikow <de_muur@gmx.de>
Jean-Marc Meessen schrieb am 20.02.2011 14:24:
* Am I using the latest version of the PBF splitter branch (I also tried the compiled r161-3) ? * Is this a known problem ? * Is something wrong with my parameters ?
I was using a windows 7 PC for this.
Thanks in advance to any hint or advise you could give me.
This sound like a known (and already solved) problem, where splitter had problems with files >4GB on windof PCs.
In the lates version of r-161 this problem was solved, previous versions of r-161 had this problem.
Are you sure, you are using the latest versin of r-161? (My splitter.jar has the date 27.01.2011.)
Are smaller maps working ok?
Gruss Torsten _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Jean-Marc Meessen schrieb am 20.02.2011 14:53:
The file I am trying to split is fairly small, Torsten. It is about 30kb large (while the european data is 5gb). The splitter version is from end january.
Isn't the extract too small and thus the Osmosis extract went wrong ? I am extracting 50km around the borders of Belgium.
I haven't read your first mail correctly: I think the Osmosis cutting is your problem. Splitter and Osmosis use the same librarys for the pbf handling, so they both had the same 4GB bug. Is your Osmosis actual? It should also have a date from end of january. Gruss Torsten
You have to use an osmosis-build, builded after 28.1.2011. Also it is faster to use splitter with an existing areas.list-file and europe.osm.pbf-file. This will be as fast as splitting your area with osmosis. So you save the time for splitting your myBenelux.osm.pbf Henning
Hi Jean-Marc, Im using an areas.list-file for the Benelux, you can use this to split the europe.osm.pbf See areas_bnl.kml and areas_bnl.list on http://mijndev.openstreetmap.nl/~ligfietser/openfietsmap/Scripts/ I've also made a bigger areas.list that divides the bnl in 3 sections: north, middle and south. First step is to split europe.osm.pbf in 3 big osm.gz files with the areas list below. (or maybe merge middle and south to one big osm.gz). Second step is to split the 3 files into smaller tiles with max-nodes=1500000 or so. I've made this workaround because I couldnt make a working extract with osmosis either (same problems). # List of areas # BNL-north 31100000: 2410496,110592 to 2516992,344064 # : 51.723633,2.373047 to 54.008789,7.382813 # BNL-middle 31200000: 2361344,110592 to 2410496,307200 # : 50.668945,2.373047 to 51.723633,6.547852 # BNL-south 31300000: 2299904,147456 to 2361344,307200 # : 49.350586,3.164063 to 50.668945,6.547852
Hello Minko, This is indeed an interesting option. I need to check if the large squares you propose fit my needs (as I'd like to have a good chunk of Germany and France. I am also concerned to use previously defined tiles as the tiles are growing very fast since the Bing pictures are available.... Jmm 2011/2/20 Minko <ligfietser@online.nl>
Hi Jean-Marc, Im using an areas.list-file for the Benelux, you can use this to split the europe.osm.pbf See areas_bnl.kml and areas_bnl.list on http://mijndev.openstreetmap.nl/~ligfietser/openfietsmap/Scripts/
I've also made a bigger areas.list that divides the bnl in 3 sections: north, middle and south. First step is to split europe.osm.pbf in 3 big osm.gz files with the areas list below. (or maybe merge middle and south to one big osm.gz). Second step is to split the 3 files into smaller tiles with max-nodes=1500000 or so.
I've made this workaround because I couldnt make a working extract with osmosis either (same problems).
# List of areas
# BNL-north 31100000: 2410496,110592 to 2516992,344064 # : 51.723633,2.373047 to 54.008789,7.382813
# BNL-middle 31200000: 2361344,110592 to 2410496,307200 # : 50.668945,2.373047 to 51.723633,6.547852
# BNL-south 31300000: 2299904,147456 to 2361344,307200 # : 49.350586,3.164063 to 50.668945,6.547852
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hello Henning, Thank you for the tip. Just came back and am trying your suggestion. I think that indeed the problem is coming from Osmosis. I used the latest stable build (previous compilations were made running on Linux but a little low on memory for the rest of the process). I am now running the latest dev build of Osmosis. The produced file is larger then the previous run (10 times larger, what is looking much better). I'll run splitter on that file now. I am optimistic. Jmm 2011/2/20 Henning Scholland <osm@aighes.de>
You have to use an osmosis-build, builded after 28.1.2011.
Also it is faster to use splitter with an existing areas.list-file and europe.osm.pbf-file. This will be as fast as splitting your area with osmosis. So you save the time for splitting your myBenelux.osm.pbf
Henning
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Youpie ! I am back on the road: it works. I was able to walk through the complete process up to my Oregon GPS. Thanks for the tips and the advice. Jmm 2011/2/20 Jean-Marc Meessen <osm@meessen-web.org>
Hello Henning,
Thank you for the tip. Just came back and am trying your suggestion. I think that indeed the problem is coming from Osmosis. I used the latest stable build (previous compilations were made running on Linux but a little low on memory for the rest of the process). I am now running the latest dev build of Osmosis. The produced file is larger then the previous run (10 times larger, what is looking much better).
I'll run splitter on that file now. I am optimistic.
Jmm
2011/2/20 Henning Scholland <osm@aighes.de>
You have to use an osmosis-build, builded after 28.1.2011.
Also it is faster to use splitter with an existing areas.list-file and europe.osm.pbf-file. This will be as fast as splitting your area with osmosis. So you save the time for splitting your myBenelux.osm.pbf
Henning
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
participants (4)
-
Henning Scholland -
Jean-Marc Meessen -
Minko -
Torsten Leistikow