After a *lot* of messages like these : Node 986712679 in too many areas. Already in areas 0xd0a0706, trying to add area 0xe Node 986712679 in too many areas. Already in areas 0xd0a0706, trying to add area 0x11 Node 986712679 in too many areas. Already in areas 0xd0a0706, trying to add area 0x12 the command java -jar splitter-r123/splitter.jar --mapid=400 --max-nodes=1000 --resolution=15 iso-nl-259.osm did produce 30 output files. This may be too much, so I tried again with max-nodes=10000 (and resolution=15). Similar result : 30 output files. This time I logged the output via script : > 3 million lines of messages. max-nodes=10000 , resolution=14 --> 6 files, "only" 120000 lines of output in typescript. BTW I also tried decreasing max-nodes without changing the resolution, this is what Felix wrote. That didn't work : the file never got split further. So how do I describe what to do, in documentation for RoadMap users ? Danny On Sun, 2010-11-14 at 16:12 -0600, Scott Crosby wrote:

Try increasing the resolution. The resolution controls the minimum tile size. Note that increasing the resolution by 1 quadruples the memory use of one of the internal arrays used to determine split points. I suggest using 13 or 14. Alternatively, you can split the geographic extent of the too-large tile manually in areas.list.

Scott

On Sun, Nov 14, 2010 at 8:16 AM, Danny Backx <danny.backx@scarlet.be> wrote: Hi,

I've been using splitter to cut European country files into pieces that RoadMap will cope with. The country files are from GeoFabrik (http://download.geofabrik.de/osm/europe), RoadMap is at http://sourceforge.net/projects/roadmap .

This works very well except for a few cases in which some of the pieces don't appear to get any smaller when I decrease max-nodes.

An example http://danny.backx.info/download/roadmap/iso-nl-259.osm.gz has been created by java -jar splitter-r123/splitter.jar --mapid=001 --max-nodes=10000 netherlands-2010.11.13.osm .

Inspection of the files created (only some shown) reveals variation in their size : -rw-rw-r-- 1 danny danny 3157377 2010-11-14 11:11 iso-nl-001.osm.gz -rw-rw-r-- 1 danny danny 1618173 2010-11-14 11:11 iso-nl-002.osm.gz -rw-rw-r-- 1 danny danny 926847 2010-11-14 11:11 iso-nl-003.osm.gz -rw-rw-r-- 1 danny danny 3367783 2010-11-14 11:11 iso-nl-004.osm.gz -rw-rw-r-- 1 danny danny 2808994 2010-11-14 11:11 iso-nl-005.osm.gz -rw-rw-r-- 1 danny danny 2907610 2010-11-14 11:11 iso-nl-006.osm.gz -rw-rw-r-- 1 danny danny 6003918 2010-11-14 11:11 iso-nl-007.osm.gz -rw-rw-r-- 1 danny danny 2919463 2010-11-14 11:11 iso-nl-008.osm.gz -rw-rw-r-- 1 danny danny 3121867 2010-11-14 11:11 iso-nl-009.osm.gz -rw-rw-r-- 1 danny danny 2277160 2010-11-14 11:11 iso-nl-010.osm.gz -rw-rw-r-- 1 danny danny 391572 2010-11-14 11:11 iso-nl-011.osm.gz -rw-rw-r-- 1 danny danny 2073021 2010-11-14 11:11 iso-nl-012.osm.gz -rw-rw-r-- 1 danny danny 14619554 2010-11-14 11:18 iso-nl-259.osm.gz

Is there anything I can do to deal with this ?

Danny -- Danny Backx ; danny.backx - at - scarlet.be ; http://danny.backx.info

_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

-- Danny Backx ; danny.backx - at - scarlet.be ; http://danny.backx.info

On Mon, 2010-11-15 at 14:28 -0600, Scott Crosby wrote:

Node 986712679 in too many areas. Already in areas 0xd0a0706, trying to add area 0x12

The tracking tables used by the old version of the splitter can only track nodes that occur in at most 4 output files. (To handler borders correctly, nodes near a region are included in it.) The warnings indicate that nodes weren't written to a file that they should have been. That's bad.

I suggest using a copy of the splitter generated from the crosby_integration branch. The code in that branch supports pbf input, runs faster, and doesn't have this limitation. http://www.mail-archive.com/mkgmap-dev@lists.mkgmap.org.uk/msg07266.html

This appears to be better indeed.

It sounds like the limit is the minimum size of a region. Note that this is a limitation of the part of the splitter that automatically finds splits. If areas.list is manually edited, smaller regions can be generated.

You're hitting that minimum-area-size limit at anything under 10,000 nodes. The only fixes are either increase the resolution, or, for regions containing too many nodes, edit areas.list file and subdivide the regions that are too big.

max-nodes=10000 , resolution=14 --> 6 files, "only" 120000 lines of output in typescript.

BTW I also tried decreasing max-nodes without changing the resolution, this is what Felix wrote. That didn't work : the file never got split further.

So how do I describe what to do, in documentation for RoadMap users ?

First question is how many nodes can RoadMap handle?

There are several limits in RoadMap. The one I'm hitting right now (with areas not sufficiently split) is the number of "polygons" which is limited to 65535. Here's a sample output of buildmap on a file that doesn't hit the limit, to give you an idea : pavilion: {1062} /home/danny/src/roadmap/roadmap.cvs.sf.net/roadmap/build/linux/buildmap_osm -c ../default/All -m maps.gb -i iso-gb-881.osm -o iso-gb-881.rdm -- Pass 1 : 262313 lines read (0 seconds) -- Pass 2 : 262313 lines read (3 seconds) -- Pass 3 : 262313 lines read (0 seconds) -- loading shape info (from 47261 ways) ... -- generating squares... -- sorting squares... -- sorting points... -- counting crossings... -- sorting lines... -- Pass 4 : 262313 lines read (0 seconds) -- Splits 39193, ways split 6103, not split 2595 -- sorting streets... -- retrieving lines and squares... -- sorting polygons' squares... -- sorting polygons... -- sorting polygon lines... -- writing results to iso-gb-881.rdm -- saving dictionary... -- saving 5 attributes... -- saving 89604 points... -- saving 57180 lines... -- Line By Point : 89602 points, 114359 lines -- saving 2406 streets... -- building the street search accelerator... -- saving 0 ranges... -- saving 9919 polygons... -- saving 36 squares... pavilion: {1063} ls -l iso-gb-881.osm maps.gb/iso-gb-881.rdm -rw-rw-r-- 2 danny danny 8988883 2010-11-13 16:18 iso-gb-881.osm -rw-rw-r-- 1 danny danny 2328604 2010-11-16 19:06 maps.gb/iso-gb-881.rdm I just ran it on the OSM file for the Netherlands, with command java -jar $SPLITTER --mapid=001 --max-nodes=40000 --resolution=14 $OSMFILE and got this error : Exception in thread "main" java.lang.OutOfMemoryError: Java heap space at it.unimi.dsi.fastutil.longs.LongArrays.ensureCapacity(LongArrays.java:107) at it.unimi.dsi.fastutil.longs.LongArrayList.ensureCapacity(LongArrayList.java:202) at it.unimi.dsi.fastutil.longs.LongArrayList.size(LongArrayList.java:271) at uk.me.parabola.splitter.SparseInt2ShortMapInline.resizeTo(SparseInt2ShortMapInline.java:95) at uk.me.parabola.splitter.SparseInt2ShortMapInline.put(SparseInt2ShortMapInline.java:123) at uk.me.parabola.splitter.SparseInt2ShortMultiMap $Inner.put(SparseInt2ShortMultiMap.java:81) at uk.me.parabola.splitter.SparseInt2ShortMultiMap $Inner.put(SparseInt2ShortMultiMap.java:79) at uk.me.parabola.splitter.SparseInt2ShortMultiMap.put(SparseInt2ShortMultiMap.java:31) at uk.me.parabola.splitter.SplitProcessor.writeNode(SplitProcessor.java:208) at uk.me.parabola.splitter.SplitProcessor.processNode(SplitProcessor.java:118) at uk.me.parabola.splitter.OSMParser.endElement(OSMParser.java:243) at uk.me.parabola.splitter.AbstractXppParser.parse(AbstractXppParser.java:57) at uk.me.parabola.splitter.Main.processMap(Main.java:336) at uk.me.parabola.splitter.Main.writeAreas(Main.java:307) at uk.me.parabola.splitter.Main.split(Main.java:152) at uk.me.parabola.splitter.Main.start(Main.java:108) at uk.me.parabola.splitter.Main.main(Main.java:97) Earlier runs with max-nodes=10000 and 20000 were going to generate too many files (> 1000) so I ended up interrupting them after pass 1 but they didn't crash before that point. Still looking for good advice on how to document this ;-) Danny -- Danny Backx ; danny.backx - at - scarlet.be ; http://danny.backx.info

I've taken some time and played with the suggestions given. I'm including two scripts. One (doit.nl) is a sample script to split the OSM file for the Netherlands into manageable pieces, and then to convert the OSM files into RoadMap format. This mostly works : - I get 339 map files - 9 maps turn out to be too big still for RoadMap So I created another script (deeper.nl, used in a subdirectory, you'll notice that in the script) that'll take the files I put in the subdirectory, and divide them in four smaller chunks. I used deeper.nl and still had problems with 3 of the smaller chunks so I reran deeper.nl one more time, producing maps that were (again) four times smaller than the offending OSM files. This worked. So the 8GB netherlands.osm file got converted into some 380 RoadMap map files, totalling about 1.6GB. I realize that I'm working around something splitter is leaving unsolved, and I'm partially automating this, my question is whether all this is a good idea. Danny On Tue, 2010-11-16 at 10:37 -0800, aighes wrote:

You should tell java, how much ram splitter can use.

example: java -Xmx1024M -jar splitter-r123/splitter.jar --mapid=400 --max-nodes=1000 --resolution=15 iso-nl-259.osm

max-nodes higher than 100000 shouldn't be a problem. I'm using 1350000.

aighes

-- Danny Backx ; danny.backx - at - scarlet.be ; http://danny.backx.info

I think I've been successfully working around this 'problem' without the need to hack Splitter, but Chris Miller did changed some things based on off-list communication (Details elude me at this moment though) to make this possible. My process in splitting the planet is essentially a 2-step method: 1. Initial split to generate large tiles, some of which might be processed successfully. The ones that don't pass go into the second step. 2. A recursive process in which: 2a. Split the tile with a diminishing --max-nodes setting until Splitter returns at least two subtiles. 2b. The resulting subtiles can be processed by Mkgmap, or RoadMap in the Danny's case. If a subtile can't be processed then goto 2. This way I can create tiles as large as possible but with varying amounts of nodes in an automated way, without the need for manually defined tile definitions. The only prerequisite for this process is the ability to 'know' when the processing of a splitted tile is successful. It is essentially the same process as the 'doit' and 'deeper' scripts but in an recursive way so that the depth of the process is variable depending on the actual needs instead of fixed 2 levels. If there is interest in my script (PHP) then I can clean it up a bit and post it here. On 2010-11-22 18:02, Scott Crosby wrote:

On Sun, Nov 21, 2010 at 1:04 PM, Danny Backx <danny.backx@scarlet.be <mailto:danny.backx@scarlet.be>> wrote:

I've taken some time and played with the suggestions given.

I'm including two scripts. One (doit.nl <http://doit.nl>) is a sample script to split the OSM file for the Netherlands into manageable pieces, and then to convert the OSM files into RoadMap format. This mostly works : - I get 339 map files - 9 maps turn out to be too big still for RoadMap

So I created another script (deeper.nl <http://deeper.nl>, used in a subdirectory, you'll notice that in the script) that'll take the files I put in the subdirectory, and divide them in four smaller chunks.

I used deeper.nl <http://deeper.nl> and still had problems with 3 of the smaller chunks so I reran deeper.nl <http://deeper.nl> one more time, producing maps that were (again) four times smaller than the offending OSM files.

I realize that I'm working around something splitter is leaving unsolved, and I'm partially automating this, my question is whether all this is a good idea.

If you're willing to do some work on the splitter to improve it, I think that yes, you could improve it in a few days of work to the point where this becomes a non-issue for you, and the splitter generates excellent and well-balanced tiles.

The core problem is that the splitter needs to track the point density around the world very densely. It, divide the world into, say 8192 tiles horizontally and 8192 vertically. It goes through the planet it counts how many nodes there are for each tile, and then uses that summary 'density map' table to derive the areas for splitting, by, say. Your problem is that you want regions smaller than 1*1, because even a 1*1 has too much data for your program. Increasing the density that stuff is tracked, to 32k*32ktiles across the world increases memory by 16x and might not solve your problem because the the minimum sized tile of 1*1 may still have too many nodes, even if it is smaller in geographic size.

The fix for you is that dividing Europe into, say 16k*16k tiles generates tiles with a smaller geographic extent than dividing the world into tiles of 32k*32k. You should be able to feed the geographic bounds data from the <bound> tag to configure the density map to only cover part of the world. There are a few catches in that you'll have to obey the alignment constraints that are required by mkgmap. (Which, alas, I do not know.). But that would solve your problem and might make the splitter more useful to others..

There are also a few smaller-scale things that may help. For instance, the minimum are size in the splitter is 2*2 tiles, not 1*1 tiles.

Scott

_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

On 2010-11-25 19:45, Danny Backx wrote:

On Thu, 2010-11-25 at 14:17 +0100, Lambertus wrote:

...

If there is interest in my script (PHP) then I can clean it up a bit and post it here.

Yes please.

Danny

The script is attached. I've added some inline comments and hope all is clear. Don't hesitate to ask if you have any questions.

On Mon, 2010-11-22 at 11:02 -0600, Scott Crosby wrote:

On Sun, Nov 21, 2010 at 1:04 PM, Danny Backx <danny.backx@scarlet.be> wrote: I've taken some time and played with the suggestions given.

I'm including two scripts. [..] I realize that I'm working around something splitter is leaving unsolved, and I'm partially automating this, my question is whether all this is a good idea.

If you're willing to do some work on the splitter to improve it, I think that yes, you could improve it in a few days of work to the point where this becomes a non-issue for you, and the splitter generates excellent and well-balanced tiles. [..] There are a few catches in that you'll have to obey the alignment constraints that are required by mkgmap. (Which, alas, I do not know.). But that would solve your problem and might make the splitter more useful to others..

I'm afraid I don't qualify. Too much work already, and I don't have a clue about those constraints (not a mkgmap user). Danny -- Danny Backx ; danny.backx - at - scarlet.be ; http://danny.backx.info

Hi

...

I suggest using a copy of the splitter generated from the crosby_integration branch. The code in that branch supports pbf input, runs faster, and doesn't have this limitation. http://www.mail-archive.com/mkgmap-dev@lists.mkgmap.org.uk/msg07266.html

This build is no longer there. Any location for a build with pbf support? Or a docu how to build splitter from this branch?

I've re-enabled that same download again. There is no new version since then. ..Steve