Splitter: Assertion on very large node id
Using an o5m file with a huge node-id seems to break splitter-r427: [..] 3,200,000,000 nodes processed... id=3982721872 3,210,000,000 nodes processed... id=200000782524 Exception in thread "main" java.lang.AssertionError at uk.me.parabola.splitter.O5mMapParser.readNode(O5mMapParser.java:260) at uk.me.parabola.splitter.O5mMapParser.readFile(O5mMapParser.java:187) at uk.me.parabola.splitter.O5mMapParser.parse(O5mMapParser.java:133) at uk.me.parabola.splitter.Main.processOSMFiles(Main.java:1396) at uk.me.parabola.splitter.Main.processMap(Main.java:908) at uk.me.parabola.splitter.Main.calculateAreas(Main.java:599) at uk.me.parabola.splitter.Main.split(Main.java:256) at uk.me.parabola.splitter.Main.start(Main.java:185) at uk.me.parabola.splitter.Main.main(Main.java:155) The o5m file is a combination of a recent planet dump and an address file for France, from here (banco-france-o5m.zip): https://github.com/ligfietser/mkgmap-style-sheets/tree/master/resources It looks like a signed 32 bit integer problem? Splitter is running on Linux x86_64, Java OpenJDK 1.7.0_95 64-bit (mixed mode).
Hi Lambertus, the code in line 260: assert flon >= -180.0 && flon <= 180.0; So it is not related to ids. Gerd ________________________________________ Von: mkgmap-dev-bounces@lists.mkgmap.org.uk <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Lambertus <osm@na1400.info> Gesendet: Dienstag, 1. März 2016 14:15 An: Development list for mkgmap Betreff: [mkgmap-dev] Splitter: Assertion on very large node id Using an o5m file with a huge node-id seems to break splitter-r427: [..] 3,200,000,000 nodes processed... id=3982721872 3,210,000,000 nodes processed... id=200000782524 Exception in thread "main" java.lang.AssertionError at uk.me.parabola.splitter.O5mMapParser.readNode(O5mMapParser.java:260) at uk.me.parabola.splitter.O5mMapParser.readFile(O5mMapParser.java:187) at uk.me.parabola.splitter.O5mMapParser.parse(O5mMapParser.java:133) at uk.me.parabola.splitter.Main.processOSMFiles(Main.java:1396) at uk.me.parabola.splitter.Main.processMap(Main.java:908) at uk.me.parabola.splitter.Main.calculateAreas(Main.java:599) at uk.me.parabola.splitter.Main.split(Main.java:256) at uk.me.parabola.splitter.Main.start(Main.java:185) at uk.me.parabola.splitter.Main.main(Main.java:155) The o5m file is a combination of a recent planet dump and an address file for France, from here (banco-france-o5m.zip): https://github.com/ligfietser/mkgmap-style-sheets/tree/master/resources It looks like a signed 32 bit integer problem? Splitter is running on Linux x86_64, Java OpenJDK 1.7.0_95 64-bit (mixed mode). _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Lambertus, maybe the problem is related to the file size of planet file (~32GB) I've added those assertions to detect possible IO errors. If you like, I can try to add code to report the last node that was processed before. Gerd ________________________________________ Von: mkgmap-dev-bounces@lists.mkgmap.org.uk <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Lambertus <osm@na1400.info> Gesendet: Dienstag, 1. März 2016 14:15 An: Development list for mkgmap Betreff: [mkgmap-dev] Splitter: Assertion on very large node id Using an o5m file with a huge node-id seems to break splitter-r427: [..] 3,200,000,000 nodes processed... id=3982721872 3,210,000,000 nodes processed... id=200000782524 Exception in thread "main" java.lang.AssertionError at uk.me.parabola.splitter.O5mMapParser.readNode(O5mMapParser.java:260) at uk.me.parabola.splitter.O5mMapParser.readFile(O5mMapParser.java:187) at uk.me.parabola.splitter.O5mMapParser.parse(O5mMapParser.java:133) at uk.me.parabola.splitter.Main.processOSMFiles(Main.java:1396) at uk.me.parabola.splitter.Main.processMap(Main.java:908) at uk.me.parabola.splitter.Main.calculateAreas(Main.java:599) at uk.me.parabola.splitter.Main.split(Main.java:256) at uk.me.parabola.splitter.Main.start(Main.java:185) at uk.me.parabola.splitter.Main.main(Main.java:155) The o5m file is a combination of a recent planet dump and an address file for France, from here (banco-france-o5m.zip): https://github.com/ligfietser/mkgmap-style-sheets/tree/master/resources It looks like a signed 32 bit integer problem? Splitter is running on Linux x86_64, Java OpenJDK 1.7.0_95 64-bit (mixed mode). _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Thanks, Gerd, for looking into this. The problem is now nailed down to one of the other address files that are merged with the planet file. So there is currently no need to change Splitter's code. On 02/03/2016 07:09, Gerd Petermann wrote:
Hi Lambertus,
maybe the problem is related to the file size of planet file (~32GB) I've added those assertions to detect possible IO errors. If you like, I can try to add code to report the last node that was processed before.
Gerd
________________________________________ Von: mkgmap-dev-bounces@lists.mkgmap.org.uk <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Lambertus <osm@na1400.info> Gesendet: Dienstag, 1. März 2016 14:15 An: Development list for mkgmap Betreff: [mkgmap-dev] Splitter: Assertion on very large node id
Using an o5m file with a huge node-id seems to break splitter-r427:
[..] 3,200,000,000 nodes processed... id=3982721872 3,210,000,000 nodes processed... id=200000782524 Exception in thread "main" java.lang.AssertionError at uk.me.parabola.splitter.O5mMapParser.readNode(O5mMapParser.java:260) at uk.me.parabola.splitter.O5mMapParser.readFile(O5mMapParser.java:187) at uk.me.parabola.splitter.O5mMapParser.parse(O5mMapParser.java:133) at uk.me.parabola.splitter.Main.processOSMFiles(Main.java:1396) at uk.me.parabola.splitter.Main.processMap(Main.java:908) at uk.me.parabola.splitter.Main.calculateAreas(Main.java:599) at uk.me.parabola.splitter.Main.split(Main.java:256) at uk.me.parabola.splitter.Main.start(Main.java:185) at uk.me.parabola.splitter.Main.main(Main.java:155)
The o5m file is a combination of a recent planet dump and an address file for France, from here (banco-france-o5m.zip): https://github.com/ligfietser/mkgmap-style-sheets/tree/master/resources
It looks like a signed 32 bit integer problem? Splitter is running on Linux x86_64, Java OpenJDK 1.7.0_95 64-bit (mixed mode). _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Lambertus, good to know. I still think that I should change the code to improve the handling of wrong data. As you pointed out, the status messages are misseading, on expects that the last message before a stack trace is somehow related to the problem, but it wasn't here. Gerd ________________________________________ Von: mkgmap-dev-bounces@lists.mkgmap.org.uk <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Lambertus <osm@na1400.info> Gesendet: Donnerstag, 3. März 2016 10:31 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] Splitter: Assertion on very large node id Thanks, Gerd, for looking into this. The problem is now nailed down to one of the other address files that are merged with the planet file. So there is currently no need to change Splitter's code. On 02/03/2016 07:09, Gerd Petermann wrote:
Hi Lambertus,
maybe the problem is related to the file size of planet file (~32GB) I've added those assertions to detect possible IO errors. If you like, I can try to add code to report the last node that was processed before.
Gerd
________________________________________ Von: mkgmap-dev-bounces@lists.mkgmap.org.uk <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Lambertus <osm@na1400.info> Gesendet: Dienstag, 1. März 2016 14:15 An: Development list for mkgmap Betreff: [mkgmap-dev] Splitter: Assertion on very large node id
Using an o5m file with a huge node-id seems to break splitter-r427:
[..] 3,200,000,000 nodes processed... id=3982721872 3,210,000,000 nodes processed... id=200000782524 Exception in thread "main" java.lang.AssertionError at uk.me.parabola.splitter.O5mMapParser.readNode(O5mMapParser.java:260) at uk.me.parabola.splitter.O5mMapParser.readFile(O5mMapParser.java:187) at uk.me.parabola.splitter.O5mMapParser.parse(O5mMapParser.java:133) at uk.me.parabola.splitter.Main.processOSMFiles(Main.java:1396) at uk.me.parabola.splitter.Main.processMap(Main.java:908) at uk.me.parabola.splitter.Main.calculateAreas(Main.java:599) at uk.me.parabola.splitter.Main.split(Main.java:256) at uk.me.parabola.splitter.Main.start(Main.java:185) at uk.me.parabola.splitter.Main.main(Main.java:155)
The o5m file is a combination of a recent planet dump and an address file for France, from here (banco-france-o5m.zip): https://github.com/ligfietser/mkgmap-style-sheets/tree/master/resources
It looks like a signed 32 bit integer problem? Splitter is running on Linux x86_64, Java OpenJDK 1.7.0_95 64-bit (mixed mode). _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Problem is corrupt data in the Vlaanderen set: lon min: 2.3066895 lon max: 200.0000000 lat min: -120.4967296 lat max: 103.0000000 I have fixed it with osmconvert (bbox around Belgium) https://github.com/ligfietser/mkgmap-style-sheets/commit/bd945183cdfc704d0ff...
The previous file version contained a lot of such nonsense data. Hope the rest is okay... Gerd ________________________________________ Von: mkgmap-dev-bounces@lists.mkgmap.org.uk <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Minko <ligfietser@online.nl> Gesendet: Donnerstag, 3. März 2016 12:05 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] Splitter: Assertion on very large node id Problem is corrupt data in the Vlaanderen set: lon min: 2.3066895 lon max: 200.0000000 lat min: -120.4967296 lat max: 103.0000000 I have fixed it with osmconvert (bbox around Belgium) https://github.com/ligfietser/mkgmap-style-sheets/commit/bd945183cdfc704d0ff... _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
participants (3)
-
Gerd Petermann -
Lambertus -
Minko