[Patch] Overflow in splitter node counters
Hi Thorsten, attached is a patch for the problem reported here: http://gis.19327.n5.nabble.com/Bug-in-splitter-tp5610856p5610856.html I don't have the data nor the machine to test it, so I tried to find all places were nodes are counted and changed the variables to long. I did not find a good explanation for the small difference in the numbers: osmconvert reported 2518069911 nodes, which is ~ 2.518.069.911 -(2^31 - 1) = 370.586.264 - 2^31 + 370.586.264 =* -1.776.897.384* but splitter r200 reported * -1.776.905.107* nodes The formatting routine is not the reason. Any ideas? Gerd http://gis.19327.n5.nabble.com/file/n5611808/nodes_int_overflow_v1.patch nodes_int_overflow_v1.patch -- View this message in context: http://gis.19327.n5.nabble.com/Patch-Overflow-in-splitter-node-counters-tp56... Sent from the Mkgmap Development mailing list archive at Nabble.com.
Hi Gerd, On Mon, Apr 02, GerdP wrote:
Hi Thorsten,
attached is a patch for the problem reported here:
http://gis.19327.n5.nabble.com/Bug-in-splitter-tp5610856p5610856.html
I don't have the data nor the machine to test it, so I tried to find all places were nodes are counted and changed the variables to long.
Thanks, at least for me it's working now.
I did not find a good explanation for the small difference in the numbers: osmconvert reported 2518069911 nodes, which is ~ 2.518.069.911 -(2^31 - 1) = 370.586.264 - 2^31 + 370.586.264 =* -1.776.897.384* but splitter r200 reported * -1.776.905.107* nodes The formatting routine is not the reason. Any ideas?
The SRTM data has some duplicate nodes (duplicate as of same lat/lon, but different ID). Does one of the tools ignore them? At least mkgmap warns about them. Else no, I have no idea. Thorsten -- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg)
Hi Thorsten,
Date: Mon, 2 Apr 2012 16:13:52 +0200 From: kukuk@suse.de To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] [Patch] Overflow in splitter node counters
Hi Gerd,
On Mon, Apr 02, GerdP wrote:
Hi Thorsten,
attached is a patch for the problem reported here:
http://gis.19327.n5.nabble.com/Bug-in-splitter-tp5610856p5610856.html
I don't have the data nor the machine to test it, so I tried to find all places were nodes are counted and changed the variables to long.
Thanks, at least for me it's working now.
Fine. Good to hear.
I did not find a good explanation for the small difference in the numbers: osmconvert reported 2518069911 nodes, which is ~ 2.518.069.911 -(2^31 - 1) = 370.586.264 - 2^31 + 370.586.264 =* -1.776.897.384* but splitter r200 reported * -1.776.905.107* nodes The formatting routine is not the reason. Any ideas?
The SRTM data has some duplicate nodes (duplicate as of same lat/lon, but different ID). Does one of the tools ignore them? At least mkgmap warns about them. Else no, I have no idea.
What is the value that the patched splitter reports for the coords occupancy? Here is what I see for a small file: ... *********************************************************** Final statistics *********************************************************** Needed dictionary entries: 25 of 65535 coords occupancy MAP occupancy: 7.544.535 ... Ciao, Gerd
Thorsten
-- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Gerd, On Mon, Apr 02, Gerd Petermann wrote:
I did not find a good explanation for the small difference in the numbers: osmconvert reported 2518069911 nodes, which is ~ 2.518.069.911 -(2^31 - 1) = 370.586.264 - 2^31 + 370.586.264 =* -1.776.897.384* but splitter r200 reported * -1.776.905.107* nodes The formatting routine is not the reason. Any ideas?
The SRTM data has some duplicate nodes (duplicate as of same lat/lon, but different ID). Does one of the tools ignore them? At least mkgmap warns about them. Else no, I have no idea.
What is the value that the patched splitter reports for the coords occupancy? Here is what I see for a small file: ... *********************************************************** Final statistics *********************************************************** Needed dictionary entries: 25 of 65535 coords occupancy MAP occupancy: 7.544.535 ...
With the patched version: Needed dictionary entries: 414 of 65535 coords occupancy MAP occupancy: 1.014.076.753 Thorsten -- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg)
Hi Thorsten,
Date: Tue, 3 Apr 2012 12:58:49 +0200 From: kukuk@suse.de To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] [Patch] Overflow in splitter node counters
Hi Gerd,
On Mon, Apr 02, Gerd Petermann wrote:
The SRTM data has some duplicate nodes (duplicate as of same lat/lon, but different ID). Does one of the tools ignore them? At least mkgmap warns about them. Else no, I have no idea.
What is the value that the patched splitter reports for the coords occupancy? Here is what I see for a small file: ... *********************************************************** Final statistics *********************************************************** Needed dictionary entries: 25 of 65535 coords occupancy MAP occupancy: 7.544.535 ...
With the patched version:
Needed dictionary entries: 414 of 65535 coords occupancy MAP occupancy: 1.014.076.753
Hmm, this is confusing. The map should count each different node-id as one entry, so maybe you have duplicates in your data? If yes, make sure that equal node-ids have equal lat/lon values, else you'll get wrong results. Gerd
Thorsten
-- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
It's splitting the big data now, but for me all output is broken (even for files were the old splitter worked fine without probs): Error at line 1, col 1 Bad file format: 75400000.osm.pbf Error parsing file ........ Error at line 1, col 1 Bad file format: 75400698.osm.pbf Error parsing file Exception in thread "main" java.lang.NullPointerException at uk.me.parabola.mkgmap.combiners.FileInfo.getFileInfo(FileInfo.java:139) at uk.me.parabola.mkgmap.main.Main.endOptions(Main.java:406) at uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.java:126) at uk.me.parabola.mkgmap.main.Main.main(Main.java:112) openSUSE-121-64-minimal:/home/contourlines # This also happens to srtm files. Have not tried standard osm.pbf extracts... On 02.04.2012 10:35, GerdP wrote:
Hi Thorsten,
attached is a patch for the problem reported here:
http://gis.19327.n5.nabble.com/Bug-in-splitter-tp5610856p5610856.html
I don't have the data nor the machine to test it, so I tried to find all places were nodes are counted and changed the variables to long.
I did not find a good explanation for the small difference in the numbers: osmconvert reported 2518069911 nodes, which is ~ 2.518.069.911 -(2^31 - 1) = 370.586.264 - 2^31 + 370.586.264 =* -1.776.897.384* but splitter r200 reported * -1.776.905.107* nodes The formatting routine is not the reason. Any ideas?
Gerd
http://gis.19327.n5.nabble.com/file/n5611808/nodes_int_overflow_v1.patch nodes_int_overflow_v1.patch
-- View this message in context:http://gis.19327.n5.nabble.com/Patch-Overflow-in-splitter-node-counters-tp56... Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Felix, I see those error messages when osmpbf.jar is missing. Do you try this in a different directory? Gerd Felix Hartmann-2 wrote
It's splitting the big data now, but for me all output is broken (even for files were the old splitter worked fine without probs):
Error at line 1, col 1 Bad file format: 75400000.osm.pbf Error parsing file ........ Error at line 1, col 1 Bad file format: 75400698.osm.pbf Error parsing file Exception in thread "main" java.lang.NullPointerException at uk.me.parabola.mkgmap.combiners.FileInfo.getFileInfo(FileInfo.java:139) at uk.me.parabola.mkgmap.main.Main.endOptions(Main.java:406) at uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.java:126) at uk.me.parabola.mkgmap.main.Main.main(Main.java:112) openSUSE-121-64-minimal:/home/contourlines #
This also happens to srtm files. Have not tried standard osm.pbf extracts...
On 02.04.2012 10:35, GerdP wrote:
Hi Thorsten,
attached is a patch for the problem reported here:
http://gis.19327.n5.nabble.com/Bug-in-splitter-tp5610856p5610856.html
I don't have the data nor the machine to test it, so I tried to find all places were nodes are counted and changed the variables to long.
I did not find a good explanation for the small difference in the numbers: osmconvert reported 2518069911 nodes, which is ~ 2.518.069.911 -(2^31 - 1) = 370.586.264 - 2^31 + 370.586.264 =* -1.776.897.384* but splitter r200 reported * -1.776.905.107* nodes The formatting routine is not the reason. Any ideas?
Gerd
http://gis.19327.n5.nabble.com/file/n5611808/nodes_int_overflow_v1.patch nodes_int_overflow_v1.patch
-- View this message in context:http://gis.19327.n5.nabble.com/Patch-Overflow-in-splitter-node-counters-tp56... Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@.org http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
mkgmap-dev mailing list mkgmap-dev@.org http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- View this message in context: http://gis.19327.n5.nabble.com/Patch-Overflow-in-splitter-node-counters-tp56... Sent from the Mkgmap Development mailing list archive at Nabble.com.
ups, well I had checked for that. But somehow I had a tree conflict regarding external.properties file (which should exist anymore) and therefore wrong locations where it looked for the osmpbf.jar which didn't exist anymore. I'll write again once I can confirm whether the outcome is correctly working - but that will need some time as the data amount is huge.... On 04.04.2012 16:48, GerdP wrote:
Hi Felix,
I see those error messages when osmpbf.jar is missing. Do you try this in a different directory?
Gerd
Felix Hartmann-2 wrote
It's splitting the big data now, but for me all output is broken (even for files were the old splitter worked fine without probs):
Error at line 1, col 1 Bad file format: 75400000.osm.pbf Error parsing file ........ Error at line 1, col 1 Bad file format: 75400698.osm.pbf Error parsing file Exception in thread "main" java.lang.NullPointerException at uk.me.parabola.mkgmap.combiners.FileInfo.getFileInfo(FileInfo.java:139) at uk.me.parabola.mkgmap.main.Main.endOptions(Main.java:406) at uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.java:126) at uk.me.parabola.mkgmap.main.Main.main(Main.java:112) openSUSE-121-64-minimal:/home/contourlines #
This also happens to srtm files. Have not tried standard osm.pbf extracts...
On 02.04.2012 10:35, GerdP wrote:
Hi Thorsten,
attached is a patch for the problem reported here:
http://gis.19327.n5.nabble.com/Bug-in-splitter-tp5610856p5610856.html
I don't have the data nor the machine to test it, so I tried to find all places were nodes are counted and changed the variables to long.
I did not find a good explanation for the small difference in the numbers: osmconvert reported 2518069911 nodes, which is ~ 2.518.069.911 -(2^31 - 1) = 370.586.264 - 2^31 + 370.586.264 =* -1.776.897.384* but splitter r200 reported * -1.776.905.107* nodes The formatting routine is not the reason. Any ideas?
Gerd
http://gis.19327.n5.nabble.com/file/n5611808/nodes_int_overflow_v1.patch nodes_int_overflow_v1.patch
-- View this message in context:http://gis.19327.n5.nabble.com/Patch-Overflow-in-splitter-node-counters-tp56... Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@.org http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
mkgmap-dev mailing list mkgmap-dev@.org http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- View this message in context: http://gis.19327.n5.nabble.com/Patch-Overflow-in-splitter-node-counters-tp56... Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Felix, no problem. It is easy to verify the result of the new splitter, you can simply compare the output files, for "normal" data you should see no difference to r200. Gerd Felix Hartmann-2 wrote
ups, well I had checked for that. But somehow I had a tree conflict regarding external.properties file (which should exist anymore) and therefore wrong locations where it looked for the osmpbf.jar which didn't exist anymore. I'll write again once I can confirm whether the outcome is correctly working - but that will need some time as the data amount is huge....
On 04.04.2012 16:48, GerdP wrote:
Hi Felix,
I see those error messages when osmpbf.jar is missing. Do you try this in a different directory?
Gerd
Felix Hartmann-2 wrote
It's splitting the big data now, but for me all output is broken (even for files were the old splitter worked fine without probs):
Error at line 1, col 1 Bad file format: 75400000.osm.pbf Error parsing file ........ Error at line 1, col 1 Bad file format: 75400698.osm.pbf Error parsing file Exception in thread "main" java.lang.NullPointerException at uk.me.parabola.mkgmap.combiners.FileInfo.getFileInfo(FileInfo.java:139) at uk.me.parabola.mkgmap.main.Main.endOptions(Main.java:406) at uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.java:126) at uk.me.parabola.mkgmap.main.Main.main(Main.java:112) openSUSE-121-64-minimal:/home/contourlines #
This also happens to srtm files. Have not tried standard osm.pbf extracts...
On 02.04.2012 10:35, GerdP wrote:
Hi Thorsten,
attached is a patch for the problem reported here:
http://gis.19327.n5.nabble.com/Bug-in-splitter-tp5610856p5610856.html
I don't have the data nor the machine to test it, so I tried to find all places were nodes are counted and changed the variables to long.
I did not find a good explanation for the small difference in the numbers: osmconvert reported 2518069911 nodes, which is ~ 2.518.069.911 -(2^31 - 1) = 370.586.264 - 2^31 + 370.586.264 =* -1.776.897.384* but splitter r200 reported * -1.776.905.107* nodes The formatting routine is not the reason. Any ideas?
Gerd
http://gis.19327.n5.nabble.com/file/n5611808/nodes_int_overflow_v1.patch nodes_int_overflow_v1.patch
-- View this message in context:http://gis.19327.n5.nabble.com/Patch-Overflow-in-splitter-node-counters-tp56... Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@.org http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
mkgmap-dev mailing list mkgmap-dev@.org http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- View this message in context: http://gis.19327.n5.nabble.com/Patch-Overflow-in-splitter-node-counters-tp56... Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@.org http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
mkgmap-dev mailing list mkgmap-dev@.org http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- View this message in context: http://gis.19327.n5.nabble.com/Patch-Overflow-in-splitter-node-counters-tp56... Sent from the Mkgmap Development mailing list archive at Nabble.com.
Hi WanMil, would you mind to commit the patch nodes_int_overflow_v1.patch ? Felix didn't confirm that the patched version works, but I think he confirmed that he did something wrong while testing. I like to continue working on improvements in tile splitter and it's a bit confusing to have many lose ends.. Gerd GerdP wrote
Hi Felix,
no problem. It is easy to verify the result of the new splitter, you can simply compare the output files, for "normal" data you should see no difference to r200.
Gerd
Felix Hartmann-2 wrote
ups, well I had checked for that. But somehow I had a tree conflict regarding external.properties file (which should exist anymore) and therefore wrong locations where it looked for the osmpbf.jar which didn't exist anymore. I'll write again once I can confirm whether the outcome is correctly working - but that will need some time as the data amount is huge....
On 04.04.2012 16:48, GerdP wrote:
Hi Felix,
I see those error messages when osmpbf.jar is missing. Do you try this in a different directory?
Gerd
Felix Hartmann-2 wrote
It's splitting the big data now, but for me all output is broken (even for files were the old splitter worked fine without probs):
Error at line 1, col 1 Bad file format: 75400000.osm.pbf Error parsing file ........ Error at line 1, col 1 Bad file format: 75400698.osm.pbf Error parsing file Exception in thread "main" java.lang.NullPointerException at uk.me.parabola.mkgmap.combiners.FileInfo.getFileInfo(FileInfo.java:139) at uk.me.parabola.mkgmap.main.Main.endOptions(Main.java:406) at uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.java:126) at uk.me.parabola.mkgmap.main.Main.main(Main.java:112) openSUSE-121-64-minimal:/home/contourlines #
This also happens to srtm files. Have not tried standard osm.pbf extracts...
On 02.04.2012 10:35, GerdP wrote:
Hi Thorsten,
attached is a patch for the problem reported here:
http://gis.19327.n5.nabble.com/Bug-in-splitter-tp5610856p5610856.html
I don't have the data nor the machine to test it, so I tried to find all places were nodes are counted and changed the variables to long.
I did not find a good explanation for the small difference in the numbers: osmconvert reported 2518069911 nodes, which is ~ 2.518.069.911 -(2^31 - 1) = 370.586.264 - 2^31 + 370.586.264 =* -1.776.897.384* but splitter r200 reported * -1.776.905.107* nodes The formatting routine is not the reason. Any ideas?
Gerd
http://gis.19327.n5.nabble.com/file/n5611808/nodes_int_overflow_v1.patch nodes_int_overflow_v1.patch
-- View this message in context:http://gis.19327.n5.nabble.com/Patch-Overflow-in-splitter-node-counters-tp56... Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@.org http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
mkgmap-dev mailing list mkgmap-dev@.org http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- View this message in context: http://gis.19327.n5.nabble.com/Patch-Overflow-in-splitter-node-counters-tp56... Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@.org http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
mkgmap-dev mailing list mkgmap-dev@.org http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- View this message in context: http://gis.19327.n5.nabble.com/Patch-Overflow-in-splitter-node-counters-tp56... Sent from the Mkgmap Development mailing list archive at Nabble.com.
participants (4)
-
Felix Hartmann -
Gerd Petermann -
GerdP
-
Thorsten Kukuk