Endless loop with splitter r221
Hi, looks like I run into an endless loop with splitter r221 when trying to split my own created DACH extract. Anything I could provide to help here? Thorsten -- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg)
Thorsten, with parameter --keep-complete splitter may fail to caculate the so called pseudo areas. I did not find a robust algorithm yet, so maybe I will just change the program so that it stops when this happens. Does that fit ? ciao, Gerd
Date: Tue, 6 Nov 2012 21:10:45 +0100 From: kukuk@suse.de To: mkgmap-dev@lists.mkgmap.org.uk Subject: [mkgmap-dev] Endless loop with splitter r221
Hi,
looks like I run into an endless loop with splitter r221 when trying to split my own created DACH extract.
Anything I could provide to help here?
Thorsten -- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
On Tue, Nov 06, Gerd Petermann wrote:
Thorsten,
with parameter --keep-complete splitter may fail to caculate the so called pseudo areas. I did not find a robust algorithm yet, so maybe I will just change the program so that it stops when this happens.
Does that fit ?
Could be very well, for all other countrys I tested now it works fine. Thorsten
Date: Tue, 6 Nov 2012 21:10:45 +0100 From: kukuk@suse.de To: mkgmap-dev@lists.mkgmap.org.uk Subject: [mkgmap-dev] Endless loop with splitter r221
Hi,
looks like I run into an endless loop with splitter r221 when trying to split my own created DACH extract.
Anything I could provide to help here?
Thorsten -- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg)
Hi all, r224 fixes this endless loop and other reported problems: - correct creation of pseudo areas - correct handling of nodes in relations - handle overlapping tiles - performance improvements: + reduce number of expensive intersects() for ways + much faster pseudo area creation (still room for optimization) - improved reporting of incomplete relations or ways Please try it with --keep-complete and --overlap=0 . Ciao, Gerd -- View this message in context: http://gis.19327.n5.nabble.com/Endless-loop-with-splitter-r221-tp5734527p573... Sent from the Mkgmap Development mailing list archive at Nabble.com.
Thanks Gerd, If I understand it correctly, the option --problem-file=problem_polygons.txt is still needed? How can I save the report of the incomplete relations?
Hi all,
r224 fixes this endless loop and other reported problems: - correct creation of pseudo areas - correct handling of nodes in relations - handle overlapping tiles - performance improvements: + reduce number of expensive intersects() for ways + much faster pseudo area creation (still room for optimization) - improved reporting of incomplete relations or ways
Please try it with --keep-complete and --overlap=0 .
Ciao, Gerd
Am 09.11.2012 16:10, schrieb Minko:
Thanks Gerd,
If I understand it correctly, the option --problem-file=problem_polygons.txt is still needed? How can I save the report of the incomplete relations? No, it isn't. --keep-complete calculates problematic objects automatically. So it isn't necessary to set them manually. But --keep-complete needs more RAM and more time.
Henning
Well, I tried to split the geofabrik extract from Sweden first without --problem-file and found a broken polygon of the Mälaren (http://www.openstreetmap.org/browse/relation/1433877). After I splitted it again with --problem-file (and this relation in it), it was rendered ok. My splitter options: java -Xmx1200m -jar splitter-problem-list-r224.jar --keep-complete --problem-file=problem_polygons.txt --write-kml=areas.kml --output=pbf --no-trim --max-nodes=1000000 --overlap=0
If I understand it correctly, the option --problem-file=problem_polygons.txt is still needed? How can I save the report of the incomplete relations? No, it isn't. --keep-complete calculates problematic objects automatically. So it isn't necessary to set them manually. But --keep-complete needs more RAM and more time.
Henning
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Am 09.11.2012 16:27, schrieb Minko:
Well, I tried to split the geofabrik extract from Sweden first without --problem-file and found a broken polygon of the Mälaren (http://www.openstreetmap.org/browse/relation/1433877). After I splitted it again with --problem-file (and this relation in it), it was rendered ok. --keep-complete means splitter generates a list automatic --problem-file means you give splitter a list
I don't know what splitter does if you set both. I would assume that splitter tell the user, that something is wrong. Henning
Date: Fri, 9 Nov 2012 16:32:53 +0100 From: osm@aighes.de To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] Endless loop with splitter r221
Am 09.11.2012 16:27, schrieb Minko:
Well, I tried to split the geofabrik extract from Sweden first without --problem-file and found a broken polygon of the Mälaren (http://www.openstreetmap.org/browse/relation/1433877). After I splitted it again with --problem-file (and this relation in it), it was rendered ok. --keep-complete means splitter generates a list automatic --problem-file means you give splitter a list
I don't know what splitter does if you set both. I would assume that splitter tell the user, that something is wrong.
The current impelemtation merges the generated list with the user specified ones. I like to keep that until we know for sure the generated list is complete. Gerd
The current impelemtation merges the generated list with the user specified ones. I like to keep that until we know for sure the generated list is complete.
Gerd
Ok, So in my case this relation 1433877 wasn't detected by r224 with --keep-complete and only with the help of the problem list it was rendered ok?
Minko-2 wrote
The current impelemtation merges the generated list with the user specified ones. I like to keep that until we know for sure the generated list is complete.
Ok, So in my case this relation 1433877 wasn't detected by r224 with --keep-complete and only with the help of the problem list it was rendered ok?
Yes, I think this looks like an error. I will try to repdroduce that. Thanks for reporting. Gerd -- View this message in context: http://gis.19327.n5.nabble.com/Endless-loop-with-splitter-r221-tp5734527p573... Sent from the Mkgmap Development mailing list archive at Nabble.com.
Hi Minko, Minko-2 wrote
So in my case this relation 1433877 wasn't detected by r224 with --keep-complete and only with the help of the problem list it was rendered ok?
please review. I can't repdroduce the problem with r224. When debugging I see that 1433877 is detected as a problem case and is handled like that without specifying a problem-file containing one line with rel:1433877. The output of splitter is exactly the same regarding the pbf files. Ciao, Gerd -- View this message in context: http://gis.19327.n5.nabble.com/Endless-loop-with-splitter-r221-tp5734527p573... Sent from the Mkgmap Development mailing list archive at Nabble.com.
Hi Gerd, Probably I didn't cleared my Mapsource cache.... I tried to run the splitter again, now without the problem list option. Now the mp turns out fine :) Sorry for the noise ;-)
Minko-2 wrote
So in my case this relation 1433877 wasn't detected by r224 with --keep-complete and only with the help of the problem list it was rendered ok?
please review. I can't repdroduce the problem with r224. When debugging I see that 1433877 is detected as a problem case and is handled like that without specifying a problem-file containing one line with rel:1433877. The output of splitter is exactly the same regarding the pbf files.
Ciao, Gerd
Hi Minko, Minko-2 wrote
Probably I didn't cleared my Mapsource cache.... I tried to run the splitter again, now without the problem list option. Now the mp turns out fine :) Sorry for the noise ;-)
fine, that's good to hear :-) Gerd -- View this message in context: http://gis.19327.n5.nabble.com/Endless-loop-with-splitter-r221-tp5734527p573... Sent from the Mkgmap Development mailing list archive at Nabble.com.
Hi Gerd, I have another problem: I'm trying to process my Benelux map now but the splitter got stuck and stopped. The last lines were several "Sorry, way # is missing x nodes" First I cut an area out of europe.osm.pbf with osmconvert (maybe it gets wrong here?) Then I process the benelux.osm.pbf with splitter-problem-list-r224.jar: %DIR%\Tools\osmconvert %DIR%\Splitter\europe.osm.pbf -b=2.35,49.35,7.5,54 -o=%DIR%\Splitter\benelux.osm.pbf java -Xmx1200m -jar %SPLITTER% --output-dir=splitter --keep-complete --overlap=0 --write-kml=%areas%.kml --output=pbf --split-file=%areas% --geonames-file=%DIR%\resources\cities5000.txt --no-trim --description=OFM_BNL %DIR%\Splitter\benelux.osm.pbf
PS, Forgot to mention my areas.list file, it is here: http://mijndev.openstreetmap.nl/~ligfietser/openfietsmap/Scripts/areas_bnl.l...
First I cut an area out of europe.osm.pbf with osmconvert (maybe it gets wrong here?) Then I process the benelux.osm.pbf with splitter-problem-list-r224.jar:
%DIR%\Tools\osmconvert %DIR%\Splitter\europe.osm.pbf -b=2.35,49.35,7.5,54 -o=%DIR%\Splitter\benelux.osm.pbf java -Xmx1200m -jar %SPLITTER% --output-dir=splitter --keep-complete --overlap=0 --write-kml=%areas%.kml --output=pbf --split-file=%areas% --geonames-file=%DIR%\resources\cities5000.txt --no-trim --description=OFM_BNL %DIR%\Splitter\benelux.osm.pbf
Hi Minko, what do you mean with "got stuck" ? Is it getting slower and slower or did it crash? maybe it is a simple "too small heap" problem?. Try e.g. -Xms1600m -Xmx1600m If that doesn't help, please provide a log file created with these additional JVM parms: -XX:+PrintGCTimeStamps -XX:+PrintGCDetails I found a few errors in the handling of overlapping areas in r224 (Henning uses this). I hope I will commit the correction today. Gerd Minko-2 wrote
PS,
Forgot to mention my areas.list file, it is here: http://mijndev.openstreetmap.nl/~ligfietser/openfietsmap/Scripts/areas_bnl.l...
First I cut an area out of europe.osm.pbf with osmconvert (maybe it gets wrong here?) Then I process the benelux.osm.pbf with splitter-problem-list-r224.jar:
%DIR%\Tools\osmconvert %DIR%\Splitter\europe.osm.pbf -b=2.35,49.35,7.5,54 -o=%DIR%\Splitter\benelux.osm.pbf java -Xmx1200m -jar %SPLITTER% --output-dir=splitter --keep-complete --overlap=0 --write-kml=%areas%.kml --output=pbf --split-file=%areas% --geonames-file=%DIR%\resources\cities5000.txt --no-trim --description=OFM_BNL %DIR%\Splitter\benelux.osm.pbf
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
-- View this message in context: http://gis.19327.n5.nabble.com/Endless-loop-with-splitter-r221-tp5734527p573... Sent from the Mkgmap Development mailing list archive at Nabble.com.
No, it just stopped after those messages. No warnings, I'll try the logfile. -Xms1600m doesnt work on my computer (dont have much memory)
Ah wait, I found out that it did printed an error in a log file: Could not write KML file splitter\resources\areas_bnl.list.kml Exception in thread "main" java.lang.NullPointerException at java.awt.Rectangle.<init>(Unknown Source) at uk.me.parabola.splitter.MultiTileProcessor.addWritersOfCrossedTiles(MultiTileProcessor.java:585) at uk.me.parabola.splitter.MultiTileProcessor.addWriters(MultiTileProcessor.java:638) at uk.me.parabola.splitter.MultiTileProcessor.processWay(MultiTileProcessor.java:139) at uk.me.parabola.splitter.BinaryMapParser.parseWays(BinaryMapParser.java:170) at crosby.binary.BinaryParser.parse(BinaryParser.java:121) at crosby.binary.BinaryParser.handleBlock(BinaryParser.java:68) at crosby.binary.file.FileBlock.process(FileBlock.java:135) at crosby.binary.file.BlockInputStream.process(BlockInputStream.java:34) at uk.me.parabola.splitter.Main.processMap(Main.java:556) at uk.me.parabola.splitter.Main.writeAreas(Main.java:482) at uk.me.parabola.splitter.Main.split(Main.java:218) at uk.me.parabola.splitter.Main.start(Main.java:144) at uk.me.parabola.splitter.Main.main(Main.java:133) So it seems it didnt find the resources subdirectory?
PS I set in my batch file set areas=resources\areas_bnl.list And in my splitter command I have --split-file=%areas% --write-kml=%areas%.kml --output-dir=splitter so there is the bug, it was ignored in the splitter-r200 java -Xmx1200m -jar %SPLITTER% --output-dir=splitter --keep-complete --write-kml=%areas%.kml --output=pbf --split-file=%areas% --geonames-file=%DIR%\resources\cities5000.txt --no-trim --description=OFM_BNL --overlap=0 %DIR%\splitter\benelux.osm.pbf
Could not write KML file splitter\resources\areas_bnl.list.kml
Hi Minko, thanks, I found this problem as well. Fixed with r225 Gerd
Date: Sat, 10 Nov 2012 11:54:19 +0100 From: ligfietser@online.nl To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] Endless loop with splitter r221
Ah wait, I found out that it did printed an error in a log file:
Could not write KML file splitter\resources\areas_bnl.list.kml Exception in thread "main" java.lang.NullPointerException at java.awt.Rectangle.<init>(Unknown Source) at uk.me.parabola.splitter.MultiTileProcessor.addWritersOfCrossedTiles(MultiTileProcessor.java:585) at uk.me.parabola.splitter.MultiTileProcessor.addWriters(MultiTileProcessor.java:638) at uk.me.parabola.splitter.MultiTileProcessor.processWay(MultiTileProcessor.java:139) at uk.me.parabola.splitter.BinaryMapParser.parseWays(BinaryMapParser.java:170) at crosby.binary.BinaryParser.parse(BinaryParser.java:121) at crosby.binary.BinaryParser.handleBlock(BinaryParser.java:68) at crosby.binary.file.FileBlock.process(FileBlock.java:135) at crosby.binary.file.BlockInputStream.process(BlockInputStream.java:34) at uk.me.parabola.splitter.Main.processMap(Main.java:556) at uk.me.parabola.splitter.Main.writeAreas(Main.java:482) at uk.me.parabola.splitter.Main.split(Main.java:218) at uk.me.parabola.splitter.Main.start(Main.java:144) at uk.me.parabola.splitter.Main.main(Main.java:133)
So it seems it didnt find the resources subdirectory?
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Minko,
Ah wait, I found out that it did printed an error in a log file:
Could not write KML file splitter\resources\areas_bnl.list.kml
This is just informational. The NPE below is not related to it. Gerd
Exception in thread "main" java.lang.NullPointerException at java.awt.Rectangle.<init>(Unknown Source) at uk.me.parabola.splitter.MultiTileProcessor.addWritersOfCrossedTiles(MultiTileProcessor.java:585) at uk.me.parabola.splitter.MultiTileProcessor.addWriters(MultiTileProcessor.java:638) at uk.me.parabola.splitter.MultiTileProcessor.processWay(MultiTileProcessor.java:139) at uk.me.parabola.splitter.BinaryMapParser.parseWays(BinaryMapParser.java:170) at crosby.binary.BinaryParser.parse(BinaryParser.java:121) at crosby.binary.BinaryParser.handleBlock(BinaryParser.java:68) at crosby.binary.file.FileBlock.process(FileBlock.java:135) at crosby.binary.file.BlockInputStream.process(BlockInputStream.java:34) at uk.me.parabola.splitter.Main.processMap(Main.java:556) at uk.me.parabola.splitter.Main.writeAreas(Main.java:482) at uk.me.parabola.splitter.Main.split(Main.java:218) at uk.me.parabola.splitter.Main.start(Main.java:144) at uk.me.parabola.splitter.Main.main(Main.java:133)
So it seems it didnt find the resources subdirectory?
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Am 09.11.2012 16:10, schrieb Minko:
Thanks Gerd,
If I understand it correctly, the option --problem-file=problem_polygons.txt is still needed? How can I save the report of the incomplete relations?
I don't know for sure if that makes sense. The generated list is very specific because it contains everything that is crossing a tile border. As described before, you have thousands of these cases. On the other hand, the problem-list-generator runs quite long, so you may want to reuse the list next time or filter it somehow to reduce processing time in the following passes. At the moment I try to optimize the calculations and reduce memory needs. If that doesn't help, I think it would be good to write the list of all closed ways and mp that have a bbox size larger x. Gerd
Am 08.11.2012 01:42, schrieb GerdP:
Hi all,
r224 fixes this endless loop and other reported problems: - correct creation of pseudo areas - correct handling of nodes in relations - handle overlapping tiles - performance improvements: + reduce number of expensive intersects() for ways + much faster pseudo area creation (still room for optimization) - improved reporting of incomplete relations or ways
Please try it with --keep-complete and --overlap=0 .
Hi Gerd, I tried it with planet and all my maps. It seems to work, but there wasn't enough RAM in my PC, so it tooks to long. I stop splitter after 9 hours. Log: http://www.aighes.de/data/splitter.log If it helps you, I could also generate a detailed log. Henning
Hello Henning, yes, it got close. The long runtime in the 3rd pass of the MultiTileProcessor is caused by some lazy coded tests. I am optimistic that this can be reduced drastically. I will commit a new version soon, maybe that will already work for you. I found an error in the pseudo-area creation that causes too many problem candidates. With mixed=false, it is possible to replace a large hashmap by two arrays, I am testing this now. Gerd
Date: Fri, 9 Nov 2012 16:26:23 +0100 From: osm@aighes.de To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] Endless loop with splitter r221
Am 08.11.2012 01:42, schrieb GerdP:
Hi all,
r224 fixes this endless loop and other reported problems: - correct creation of pseudo areas - correct handling of nodes in relations - handle overlapping tiles - performance improvements: + reduce number of expensive intersects() for ways + much faster pseudo area creation (still room for optimization) - improved reporting of incomplete relations or ways
Please try it with --keep-complete and --overlap=0 .
Hi Gerd, I tried it with planet and all my maps. It seems to work, but there wasn't enough RAM in my PC, so it tooks to long. I stop splitter after 9 hours.
Log: http://www.aighes.de/data/splitter.log
If it helps you, I could also generate a detailed log.
Henning
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
participants (5)
-
Gerd Petermann -
GerdP
-
Henning Scholland -
Minko -
Thorsten Kukuk