Hi all, r236 is now available. Corrections regarding --keep-complete: - do not add members of "parent-only" relations. A p"arent-only" relation is one that has a member which is a relation and which is in the problem-list, but has no problem-ways. - some ways were detected as problem ways but still not written. This sometimes happened when one or more points of the way is normally not written to any output file. - enclosing multipolygons were not always added Improvements - reduced the needed for sorting the temp files, this also reduces heap requirement - recursive relation checks are faster Known problem: When splitting planet, a lot of boundary data is written to tiles that don't need it, e.g. the boundary of USA may be written to tiles in europe. Ciao, Gerd -- View this message in context: http://gis.19327.n5.nabble.com/splitter-r236-tp5736104.html Sent from the Mkgmap Development mailing list archive at Nabble.com.
Hi, On Fri, Nov 16, GerdP wrote:
r236 is now available.
sorry, hadn't the time to run splitter on europe before. So I did it now with r236 but run into this exception: Starting distribution pass 1 of 5, processing 241 areas (71100001 to 71100241) Processing data/osm/europe.osm.pbf Bounding box -32.2426199 27.36279 44.83212 74.95058 10.000.000 nodes processed... id=47423283 20.000.000 nodes processed... id=250816845 30.000.000 nodes processed... id=273285850 40.000.000 nodes processed... id=292255081 50.000.000 nodes processed... id=313611609 Elapsed time: 1h 10m 7s Memory: Current 5269MB (3239MB used, 2030MB free) Max 5333MB 60.000.000 nodes processed... id=337411222 Exception in thread "main" java.lang.IndexOutOfBoundsException: Index: 9580535, Size: 9580535 at java.util.ArrayList.rangeCheck(Unknown Source) at java.util.ArrayList.get(Unknown Source) at java.util.Collections$UnmodifiableList.get(Unknown Source) at crosby.binary.Osmformat$DenseNodes.getKeysVals(Unknown Source) at uk.me.parabola.splitter.BinaryMapParser.parseDense(BinaryMapParser.java:113) at crosby.binary.BinaryParser.parse(Unknown Source) at crosby.binary.BinaryParser.handleBlock(Unknown Source) at crosby.binary.file.FileBlock.process(Unknown Source) at crosby.binary.file.BlockInputStream.process(Unknown Source) at uk.me.parabola.splitter.Main.processMap(Main.java:576) at uk.me.parabola.splitter.Main.writeAreas(Main.java:528) at uk.me.parabola.splitter.Main.split(Main.java:218) at uk.me.parabola.splitter.Main.start(Main.java:144) at uk.me.parabola.splitter.Main.main(Main.java:133) -- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg)
Hi Thorsten, it seems to be a problem in the pbf read routines. Please try to process the file with r202 or another program, I expect you get similar problems. Maybe your file is corrupt, else this is a special case. Gerd Thorsten Kukuk wrote
Hi,
On Fri, Nov 16, GerdP wrote:
r236 is now available.
sorry, hadn't the time to run splitter on europe before. So I did it now with r236 but run into this exception:
Starting distribution pass 1 of 5, processing 241 areas (71100001 to 71100241) Processing data/osm/europe.osm.pbf Bounding box -32.2426199 27.36279 44.83212 74.95058 10.000.000 nodes processed... id=47423283 20.000.000 nodes processed... id=250816845 30.000.000 nodes processed... id=273285850 40.000.000 nodes processed... id=292255081 50.000.000 nodes processed... id=313611609 Elapsed time: 1h 10m 7s Memory: Current 5269MB (3239MB used, 2030MB free) Max 5333MB 60.000.000 nodes processed... id=337411222 Exception in thread "main" java.lang.IndexOutOfBoundsException: Index: 9580535, Size: 9580535 at java.util.ArrayList.rangeCheck(Unknown Source) at java.util.ArrayList.get(Unknown Source) at java.util.Collections$UnmodifiableList.get(Unknown Source) at crosby.binary.Osmformat$DenseNodes.getKeysVals(Unknown Source) at uk.me.parabola.splitter.BinaryMapParser.parseDense(BinaryMapParser.java:113) at crosby.binary.BinaryParser.parse(Unknown Source) at crosby.binary.BinaryParser.handleBlock(Unknown Source) at crosby.binary.file.FileBlock.process(Unknown Source) at crosby.binary.file.BlockInputStream.process(Unknown Source) at uk.me.parabola.splitter.Main.processMap(Main.java:576) at uk.me.parabola.splitter.Main.writeAreas(Main.java:528) at uk.me.parabola.splitter.Main.split(Main.java:218) at uk.me.parabola.splitter.Main.start(Main.java:144) at uk.me.parabola.splitter.Main.main(Main.java:133)
-- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) _______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
-- View this message in context: http://gis.19327.n5.nabble.com/splitter-r236-tp5736104p5736165.html Sent from the Mkgmap Development mailing list archive at Nabble.com.
On Fri, Nov 16, GerdP wrote:
Hi Thorsten,
it seems to be a problem in the pbf read routines. Please try to process the file with r202 or another program, I expect you get similar problems. Maybe your file is corrupt, else this is a special case.
r202 works fine. I'm trying it currently again with r236, maybe there was somewhere else a hickup, even if I had never problems with that hardware. Thorsten
Thorsten Kukuk wrote
Hi,
On Fri, Nov 16, GerdP wrote:
r236 is now available.
sorry, hadn't the time to run splitter on europe before. So I did it now with r236 but run into this exception:
Starting distribution pass 1 of 5, processing 241 areas (71100001 to 71100241) Processing data/osm/europe.osm.pbf Bounding box -32.2426199 27.36279 44.83212 74.95058 10.000.000 nodes processed... id=47423283 20.000.000 nodes processed... id=250816845 30.000.000 nodes processed... id=273285850 40.000.000 nodes processed... id=292255081 50.000.000 nodes processed... id=313611609 Elapsed time: 1h 10m 7s Memory: Current 5269MB (3239MB used, 2030MB free) Max 5333MB 60.000.000 nodes processed... id=337411222 Exception in thread "main" java.lang.IndexOutOfBoundsException: Index: 9580535, Size: 9580535 at java.util.ArrayList.rangeCheck(Unknown Source) at java.util.ArrayList.get(Unknown Source) at java.util.Collections$UnmodifiableList.get(Unknown Source) at crosby.binary.Osmformat$DenseNodes.getKeysVals(Unknown Source) at uk.me.parabola.splitter.BinaryMapParser.parseDense(BinaryMapParser.java:113) at crosby.binary.BinaryParser.parse(Unknown Source) at crosby.binary.BinaryParser.handleBlock(Unknown Source) at crosby.binary.file.FileBlock.process(Unknown Source) at crosby.binary.file.BlockInputStream.process(Unknown Source) at uk.me.parabola.splitter.Main.processMap(Main.java:576) at uk.me.parabola.splitter.Main.writeAreas(Main.java:528) at uk.me.parabola.splitter.Main.split(Main.java:218) at uk.me.parabola.splitter.Main.start(Main.java:144) at uk.me.parabola.splitter.Main.main(Main.java:133)
-- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) _______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
-- View this message in context: http://gis.19327.n5.nabble.com/splitter-r236-tp5736104p5736165.html Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev -- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg)
Hello Thorsten, thanks, please use r236 also with --keep-complete=false and no problem-list, this should bring you directly to the problem case. The other passes do not call the function that failed. If r236 fails again, please tell me where / how I can reproduce the error (input file and parms) Ciao, Gerd -- View this message in context: http://gis.19327.n5.nabble.com/splitter-r236-tp5736104p5736175.html Sent from the Mkgmap Development mailing list archive at Nabble.com.
On Sat, Nov 17, GerdP wrote:
Hello Thorsten,
thanks, please use r236 also with --keep-complete=false and no problem-list, this should bring you directly to the problem case. The other passes do not call the function that failed.
If r236 fails again, please tell me where / how I can reproduce the error (input file and parms)
This time it always passes. Don't know what happend the first time, I assume some hickups somewhere else. Sorry for the noise. But the good news is, that splitter r236 runs currently again on all of my setups with the same memory limit (which doesn't say anything about if it needs more or less memory, didn't tracked that, but at least it does not need more memory anymore than the machines have). Thorsten -- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg)
Hello Thorsten, great! Is it possible that you update the input files while splitter processes them? I don't know if that is possible, but maybe it could explain strange errors: When splitter r236 reads a pbf file for the first time, it creates an index (see blockTypeMap in main) of the blocks which tells splitter whether that block contains nodes,ways, or relations. All later read passes use this list to jump over blocks that are not needed. For each pass, the file is closed and reopend. Maybe another process is able to update the file in the time when it is closed. I'll try to find a test for this situation. Gerd Thorsten Kukuk wrote
On Sat, Nov 17, GerdP wrote:
Hello Thorsten,
thanks, please use r236 also with --keep-complete=false and no problem-list, this should bring you directly to the problem case. The other passes do not call the function that failed.
If r236 fails again, please tell me where / how I can reproduce the error (input file and parms)
This time it always passes. Don't know what happend the first time, I assume some hickups somewhere else.
Sorry for the noise.
But the good news is, that splitter r236 runs currently again on all of my setups with the same memory limit (which doesn't say anything about if it needs more or less memory, didn't tracked that, but at least it does not need more memory anymore than the machines have).
Thorsten
-- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) _______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
-- View this message in context: http://gis.19327.n5.nabble.com/splitter-r236-tp5736104p5736187.html Sent from the Mkgmap Development mailing list archive at Nabble.com.
Hi Gerd, On Sat, Nov 17, GerdP wrote:
Hello Thorsten,
great! Is it possible that you update the input files while splitter processes them?
No, that cannot happen. And in this case I even run the splitter command manually in a test directory where the normal scripts never access something. I assume a hickup of the hardware. I have seen already more than enough bit flips on different hardware in my life, but not on this one until now. Thorsten -- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg)
Hi Gerd, r236 works fine for planet. Log: http://www.aighes.de/data/mkgmap.zip Henning Another thought: What about only analysing the ways, if they cross/leave a tile and keep them in mind. Afterwards read all relations and check, if they contains a way, which is kept in mind. So relations must not be builded to, but only the needed relations would be in the tile. Henning
Henning Scholland wrote
Hi Gerd, r236 works fine for planet. Log: http://www.aighes.de/data/mkgmap.zip
Yes, that really looks much better than the previous logs. The changes in the run times for the diverse passes meet my assumptions. I guess you have also verified the missing fairy lines do appear now in 11000004 ? Henning Scholland wrote
Another thought: What about only analysing the ways, if they cross/leave a tile and keep them in mind. Afterwards read all relations and check, if they contains a way, which is kept in mind. So relations must not be builded to, but only the needed relations would be in the tile.
I think this would not detect MP that enclose a tile (without having any way or point in it or crossing it). Gerd P.S. I just found out that my tools (osmconvert, osmfilter) are sometimes producing corrupted data (longitude values like -378). I found newer versions of both, I hope this fixes the problem. I'll add a few assertions in the OSM reader routines to detect this. -- View this message in context: http://gis.19327.n5.nabble.com/splitter-r236-tp5736104p5736183.html Sent from the Mkgmap Development mailing list archive at Nabble.com.
Am 17.11.2012 11:37, schrieb GerdP:
Henning Scholland wrote
Hi Gerd, r236 works fine for planet. Log: http://www.aighes.de/data/mkgmap.zip Yes, that really looks much better than the previous logs. The changes in the run times for the diverse passes meet my assumptions. I guess you have also verified the missing fairy lines do appear now in 11000004 ? Yes, they are fine now.
Henning Scholland wrote
Another thought: What about only analysing the ways, if they cross/leave a tile and keep them in mind. Afterwards read all relations and check, if they contains a way, which is kept in mind. So relations must not be builded to, but only the needed relations would be in the tile. I think this would not detect MP that enclose a tile (without having any way or point in it or crossing it). Hmm...yes, you're right. It's very tricky: rels with type=boundary should only be in the tiles, there they have a way in and type=multipolygon should be in each tile, the area is touching.
Henning
Henning Scholland wrote
Hi Gerd, r236 works fine for planet. Log: http://www.aighes.de/data/mkgmap.zip Yes, that really looks much better than the previous logs. The changes in the run times for the diverse passes meet my assumptions. I guess you have also verified the missing fairy lines do appear now in 11000004 ? Yes, they are fine now.
Good to hear :-)
Henning Scholland wrote
Another thought: What about only analysing the ways, if they cross/leave a tile and keep them in mind. Afterwards read all relations and check, if they contains a way, which is kept in mind. So relations must not be builded to, but only the needed relations would be in the tile. I think this would not detect MP that enclose a tile (without having any way or point in it or crossing it). Hmm...yes, you're right. It's very tricky: rels with type=boundary should only be in the tiles, there they have a way in and type=multipolygon should be in each tile, the area is touching.
Hmm, so we would write complete relation if they have type=multipolygon, but allow incomplete relations if type=boundary ? For the latter I would just make sure that the ways that are crossing the tile are complete. Sounds reasonable to me. Any objections? Ciao, Gerd
participants (4)
-
Gerd Petermann -
GerdP
-
Henning Scholland -
Thorsten Kukuk