Hi, today I've merged the problem-list branch into trunk and committed r279. Major changes to r270: - keep-complete=true is now the default - new --precomp-sea parameter: estimate nodes that are added in mkgmap due to the equally named option, the parameter specifies the directory with the precompiled sea data - fixed potential problem with negative ids in XML writer - corrected mixed line endings for Windows systems: + all lines in areas.list, areas.poly, and template.args are written with \r\n + all lines in the kml file are written with \n only - improved throughput (a few percent) by avoiding auto-boxing - reduced memory needed in the "handle-problem-list" phase I'll update the wiki tomorrow. Ciao, Gerd -- View this message in context: http://gis.19327.n5.nabble.com/splitter-r279-tp5744035.html Sent from the Mkgmap Development mailing list archive at Nabble.com.
Hi,
today I've merged the problem-list branch into trunk and committed r279.
Major changes to r270: - keep-complete=true is now the default - new --precomp-sea parameter: estimate nodes that are added in mkgmap due to the equally named option, the parameter specifies the directory with the precompiled sea data - fixed potential problem with negative ids in XML writer - corrected mixed line endings for Windows systems: + all lines in areas.list, areas.poly, and template.args are written with \r\n + all lines in the kml file are written with \n only - improved throughput (a few percent) by avoiding auto-boxing - reduced memory needed in the "handle-problem-list" phase
I'll update the wiki tomorrow.
Ciao, Gerd
Hi Gerd, great improvements! I have just tested r277 (r279 is not ready for download yet) and observed that the output parameter is not working. No matter which value is given for output, the output is pbf always. Do you know if that's fixed in the trunk r279? WanMil
WanMil wrote
I have just tested r277 (r279 is not ready for download yet) and observed that the output parameter is not working. No matter which value is given for output, the output is pbf always.
Do you know if that's fixed in the trunk r279?
reg. output: Please check if you used the parameter two times. Only the last value given is used. If that doesn't help, please send me the full command that you use. Besides that, r279 should be exactly like r277. ciao, Gerd -- View this message in context: http://gis.19327.n5.nabble.com/splitter-r279-tp5744035p5744074.html Sent from the Mkgmap Development mailing list archive at Nabble.com.
WanMil wrote
I have just tested r277 (r279 is not ready for download yet) and observed that the output parameter is not working. No matter which value is given for output, the output is pbf always.
Do you know if that's fixed in the trunk r279?
reg. output: Please check if you used the parameter two times. Only the last value given is used. If that doesn't help, please send me the full command that you use.
Besides that, r279 should be exactly like r277.
ciao, Gerd
Arghh.... I used the output option twice. Bad luck.... Thanks! WanMil
I have tried to split a large part of Western Europe (6 Gb) on a modest laptop (Win32, java -Xmx1400m) and after several hours the splitting process is stopped: Exception in thread "main" java.lang.OutOfMemoryError: Java heap space at uk.me.parabola.splitter.MultiTileProcessor.endMap(MultiTileProcessor.java:247) at uk.me.parabola.splitter.Main.processMap(Main.java:817) at uk.me.parabola.splitter.Main.writeAreas(Main.java:705) at uk.me.parabola.splitter.Main.split(Main.java:250) at uk.me.parabola.splitter.Main.start(Main.java:156) at uk.me.parabola.splitter.Main.main(Main.java:145) I couldnt see when it happened, I guess it was after the splitting process has done several sections (8/8). There wasn't any osm.pbf file produced, only densities-out.txt, areas.list and areas.poly. java -Xmx1400m -jar %SPLITTER% --output-dir=splitter --polygon-file=w-eu2.poly --keep-complete --overlap=0 --max-areas=100 --mapid=97810001 --max-nodes=1400000 --write-kml=%areas%.kml --output=pbf --geonames-file=cities5000.txt --description=OFM_EU w-eu.osm.pbf 2>log.txt The w-eu.osm.pbf extract is a combined file with osm data (Benelux, Germany, Switzerland, France, Northern Italy and Northern Spain) and contour lines. I had no problems with a Germany extract or another Europe extract (the input files were .05m format though, will this consume less memory?). I could try to split the extract in 05m instead of osm.pbf. What other options can I try, higher than 1400m is no option, java wouldnt run. Or a lower value of max-areas?
Hi Minko, 1400m heap is probably not enough for the keep-complete algorithm. The OOME happened just before the program reached the most critical point, so maybe only 50kb are missing. You may try o5m input, it will probably consume less memory and is much faster anyway. If that doesn't help you can use --keep-complete=false or a smaller input file. The max-areas parameter does not have an influence on this part of the program, and I see no easy way to change that. If that doesn't help: Please add -XX:+HeapDumpOnOutOfMemoryError -XX:+PrintGCTimeStamps -XX:+PrintGCDetails to your JVM options to produce a log of the GC. Add also
splitter.log so that you can look at the sysout of splitter. Send me the logs and the *.
Gerd
Date: Sun, 13 Jan 2013 10:27:55 +0100 From: ligfietser@online.nl To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] splitter r279
I have tried to split a large part of Western Europe (6 Gb) on a modest laptop (Win32, java -Xmx1400m) and after several hours the splitting process is stopped:
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space at uk.me.parabola.splitter.MultiTileProcessor.endMap(MultiTileProcessor.java:247) at uk.me.parabola.splitter.Main.processMap(Main.java:817) at uk.me.parabola.splitter.Main.writeAreas(Main.java:705) at uk.me.parabola.splitter.Main.split(Main.java:250) at uk.me.parabola.splitter.Main.start(Main.java:156) at uk.me.parabola.splitter.Main.main(Main.java:145)
I couldnt see when it happened, I guess it was after the splitting process has done several sections (8/8). There wasn't any osm.pbf file produced, only densities-out.txt, areas.list and areas.poly.
java -Xmx1400m -jar %SPLITTER% --output-dir=splitter --polygon-file=w-eu2.poly --keep-complete --overlap=0 --max-areas=100 --mapid=97810001 --max-nodes=1400000 --write-kml=%areas%.kml --output=pbf --geonames-file=cities5000.txt --description=OFM_EU w-eu.osm.pbf 2>log.txt
The w-eu.osm.pbf extract is a combined file with osm data (Benelux, Germany, Switzerland, France, Northern Italy and Northern Spain) and contour lines.
I had no problems with a Germany extract or another Europe extract (the input files were .05m format though, will this consume less memory?). I could try to split the extract in 05m instead of osm.pbf. What other options can I try, higher than 1400m is no option, java wouldnt run. Or a lower value of max-areas? _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Thanks Gerd, I'll try the 05m first and see if that will work. Two (or or more) smaller areas are also an option, I can combine the tiles afterwards with mkgmap.
Minko-2 wrote
Thanks Gerd, I'll try the 05m first and see if that will work. Two (or or more) smaller areas are also an option, I can combine the tiles afterwards with mkgmap. _______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
One more option if you want to use keep-complete: try to increase max-nodes. The higher this value the fewer areas are created. Fewer areas also means fewer problems ways+relations and their nodes. Gerd -- View this message in context: http://gis.19327.n5.nabble.com/splitter-r279-tp5744035p5744256.html Sent from the Mkgmap Development mailing list archive at Nabble.com.
Gerd wrote:
One more option if you want to use keep-complete: try to increase max-nodes. The higher this value the fewer areas are created. Fewer areas also means fewer problems ways+relations and their nodes.
Ok, I'll try that. Splitting the o5m just failed too (but was indeed much, much faster). It stopped after Multitileprocessor pass 2, see attachment But then it is very likely mkgmap skips producing some tiles because they contain too many nodes?
Minko-2 wrote
Gerd wrote:
One more option if you want to use keep-complete: try to increase max-nodes. The higher this value the fewer areas are created. Fewer areas also means fewer problems ways+relations and their nodes.
Ok, I'll try that. Splitting the o5m just failed too (but was indeed much, much faster). It stopped after Multitileprocessor pass 2, see attachment But then it is very likely mkgmap skips producing some tiles because they contain too many nodes?
1) Use the latest version of mkgmap because that will less likely produce this error message 2) If you use --precomp-sea with mkgmap and you use --no-trim with splitter, try to add --precomp-sea to splitter as well. Although this will produce a few more tiles, it should not add many more problem ways. The positive effect is that you may set max-nodes to a much higher value. 3) If your PC has 2 GB memory, but Windows allows only -Xmx1400m for the JVM, it is probably because the graphic card shares memory. I assume that you'll be able to free memory by changing to fewer colours or smaller resoltion. Ciao, Gerd -- View this message in context: http://gis.19327.n5.nabble.com/splitter-r279-tp5744035p5744285.html Sent from the Mkgmap Development mailing list archive at Nabble.com.
1) Use the latest version of mkgmap because that will less likely produce this error message 2) If you use --precomp-sea with mkgmap and you use --no-trim with splitter, try to add --precomp-sea to splitter as well. Although this will produce a few more tiles, it should not add many more problem ways. The positive effect is that you may set max-nodes to a much higher value.
What is a mch higher value? I've tried it now max-nodes=2500000 but it still stopped after this message JVM Memory Info: Current 1197MB (333MB used, 864MB free) Max 1353MB Calculating tiles for problem relations... Heap def new generation total 380480K, used 79242K [0x042b0000, 0x1df80000, 0x21550000) eden space 338240K, 22% used [0x042b0000, 0x08e0b3b0, 0x18d00000) from space 42240K, 4% used [0x18d00000, 0x18f07750, 0x1b640000) to space 42240K, 0% used [0x1b640000, 0x1b640000, 0x1df80000) tenured generation total 845424K, used 275431K [0x21550000, 0x54eec000, 0x5bab0000) the space 845424K, 32% used [0x21550000, 0x32249f88, 0x3224a000, 0x54eec000) compacting perm gen total 12288K, used 5289K [0x5bab0000, 0x5c6b0000, 0x5fab0000) the space 12288K, 43% used [0x5bab0000, 0x5bfda6c0, 0x5bfda800, 0x5c6b0000) No shared spaces configured.
3) If your PC has 2 GB memory, but Windows allows only -Xmx1400m for the JVM, it is probably because the graphic card shares memory. I assume that you'll be able to free memory by changing to fewer colours or smaller resoltion.
My PC has 4Gb (but since it's 32 bits it cant use all 4gb). Changing the graphics didnt help to make more memory free. I guess I should make the area even smaller and run the splitter 4x instead of 2x. Northern France incl Paris+Benelux+Germany is probably too big for this computer.
Minko-2 wrote
What is a mch higher value? I've tried it now max-nodes=2500000 but it still stopped after this message JVM Memory Info: Current 1197MB (333MB used, 864MB free) Max 1353MB Calculating tiles for problem relations... Heap def new generation total 380480K, used 79242K [0x042b0000, 0x1df80000, 0x21550000) eden space 338240K, 22% used [0x042b0000, 0x08e0b3b0, 0x18d00000) from space 42240K, 4% used [0x18d00000, 0x18f07750, 0x1b640000) to space 42240K, 0% used [0x1b640000, 0x1b640000, 0x1df80000) tenured generation total 845424K, used 275431K [0x21550000, 0x54eec000, 0x5bab0000) the space 845424K, 32% used [0x21550000, 0x32249f88, 0x3224a000, 0x54eec000) compacting perm gen total 12288K, used 5289K [0x5bab0000, 0x5c6b0000, 0x5fab0000) the space 12288K, 43% used [0x5bab0000, 0x5bfda6c0, 0x5bfda800, 0x5c6b0000) No shared spaces configured.
Hmm, what does it mean that it stopped? Was it again an OutofMemoryError? At least you passed the point which I consider to be the most critical. Do you have a dump for me? Minko-2 wrote
3) If your PC has 2 GB memory, but Windows allows only -Xmx1400m for the JVM, it is probably because the graphic card shares memory. I assume that you'll be able to free memory by changing to fewer colours or smaller resoltion.
My PC has 4Gb (but since it's 32 bits it cant use all 4gb). Changing the graphics didnt help to make more memory free. I guess I should make the area even smaller and run the splitter 4x instead of 2x. Northern France incl Paris+Benelux+Germany is probably too big for this computer.
I used to use a WinXP machine with 4GB and java allowed to use -Xmx1600m, anyway, you are right, the file is very big and even if you get it running today the next extract might again be too big. Ciao, Gerd -- View this message in context: http://gis.19327.n5.nabble.com/splitter-r279-tp5744035p5744332.html Sent from the Mkgmap Development mailing list archive at Nabble.com.
Hi Gerd,
Hmm, what does it mean that it stopped? Was it again an OutofMemoryError? At least you passed the point which I consider to be the most critical. Do you have a dump for me?
Yes, again out of memory. See the logfile.
I used to use a WinXP machine with 4GB and java allowed to use -Xmx1600m, anyway, you are right, the file is very big and even if you get it running today the next extract might again be too big.
Ciao, Gerd
Ok, I'll try it with 1/4 of the initial intended extract. But maybe I'm doing it the wrong way? I still use the same w-eu.o5m as input which is 12 Gb big. I only make the poly file smaller and smaller. Is this the correct way to do it? BTW I can split germany.osm.pbf with contours without running out of memory.
Hi Minko, Minko-2 wrote
Hmm, what does it mean that it stopped? Was it again an OutofMemoryError? At least you passed the point which I consider to be the most critical. Do you have a dump for me?
Yes, again out of memory. See the logfile.
I don't see that in the log and I have no idea why it should have stopped at this stage. Minko-2 wrote
Ok, I'll try it with 1/4 of the initial intended extract. But maybe I'm doing it the wrong way? I still use the same w-eu.o5m as input which is 12 Gb big. I only make the poly file smaller and smaller. Is this the correct way to do it? BTW I can split germany.osm.pbf with contours without running out of memory.
Yes, I think this is the right way to do it. Please send your polygon file, maybe I can reproduce the problem. Besides that: With no-trim=false you should not use the precomp-sea parm. Gerd -- View this message in context: http://gis.19327.n5.nabble.com/splitter-r279-tp5744035p5744346.html Sent from the Mkgmap Development mailing list archive at Nabble.com.
participants (4)
-
Gerd Petermann -
GerdP
-
Minko -
WanMil