I'm getting the following error when trying to split some large .osm files containing contour data. After generating the contour files I preprocess them and then split them so it's most likely an error in my preprocesing but I can't track it down. Additionally this only happens if the input file needs splitting i.e if I tweak the contour file size and the --max-nodes option to the point that the input file doesn't actually need splitting then splitter continues through the file and all is well but this approach isn't feasible for what I need to achieve. Command line is:- java -Xmx1024M -jar splitter.jar --mapid=12345001 --max-nodes=200000 --output=xml input.osm Any pointers as to where to look would be appreciated. Thanks Paul Exception in thread "main" java.lang.IndexOutOfBoundsException: Index: 2147483647, Size: 6 at java.util.ArrayList.rangeCheck(ArrayList.java:604) at java.util.ArrayList.get(ArrayList.java:382) at uk.me.parabola.splitter.WriterDictionaryInt.getBitSet(WriterDictionaryInt.java:85) at uk.me.parabola.splitter.SplitProcessor.writeNode(SplitProcessor.java:297) at uk.me.parabola.splitter.SplitProcessor.processNode(SplitProcessor.java:102) at uk.me.parabola.splitter.OSMParser.endElement(OSMParser.java:219) at uk.me.parabola.splitter.AbstractXppParser.parse(AbstractXppParser.java:61) at uk.me.parabola.splitter.Main.processMap(Main.java:831) at uk.me.parabola.splitter.Main.writeAreas(Main.java:769) at uk.me.parabola.splitter.Main.split(Main.java:277) at uk.me.parabola.splitter.Main.start(Main.java:162) at uk.me.parabola.splitter.Main.main(Main.java:146)
For additional info if I use --keep-complete=false then the input files are processed without error so although I have a solution it would be better if splitter died gracefully. Thanks Paul On 16/07/13 22:51, News wrote:
I'm getting the following error when trying to split some large .osm files containing contour data. After generating the contour files I preprocess them and then split them so it's most likely an error in my preprocesing but I can't track it down. Additionally this only happens if the input file needs splitting i.e if I tweak the contour file size and the --max-nodes option to the point that the input file doesn't actually need splitting then splitter continues through the file and all is well but this approach isn't feasible for what I need to achieve.
Command line is:-
java -Xmx1024M -jar splitter.jar --mapid=12345001 --max-nodes=200000 --output=xml input.osm
Any pointers as to where to look would be appreciated.
Thanks
Paul
Exception in thread "main" java.lang.IndexOutOfBoundsException: Index: 2147483647, Size: 6 at java.util.ArrayList.rangeCheck(ArrayList.java:604) at java.util.ArrayList.get(ArrayList.java:382) at uk.me.parabola.splitter.WriterDictionaryInt.getBitSet(WriterDictionaryInt.java:85) at uk.me.parabola.splitter.SplitProcessor.writeNode(SplitProcessor.java:297) at uk.me.parabola.splitter.SplitProcessor.processNode(SplitProcessor.java:102) at uk.me.parabola.splitter.OSMParser.endElement(OSMParser.java:219) at uk.me.parabola.splitter.AbstractXppParser.parse(AbstractXppParser.java:61) at uk.me.parabola.splitter.Main.processMap(Main.java:831) at uk.me.parabola.splitter.Main.writeAreas(Main.java:769) at uk.me.parabola.splitter.Main.split(Main.java:277) at uk.me.parabola.splitter.Main.start(Main.java:162) at uk.me.parabola.splitter.Main.main(Main.java:146) _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Paul, I have no clue yet. Are you able to reproduce the error when you convert the input file to o5m or osm.pbf first? If yes, please upload the file and send a link. Gerd
Date: Tue, 16 Jul 2013 22:51:44 +0100 From: news@pointdee.co.uk To: mkgmap-dev@lists.mkgmap.org.uk Subject: [mkgmap-dev] Splitter Crash
I'm getting the following error when trying to split some large .osm files containing contour data. After generating the contour files I preprocess them and then split them so it's most likely an error in my preprocesing but I can't track it down. Additionally this only happens if the input file needs splitting i.e if I tweak the contour file size and the --max-nodes option to the point that the input file doesn't actually need splitting then splitter continues through the file and all is well but this approach isn't feasible for what I need to achieve.
Command line is:-
java -Xmx1024M -jar splitter.jar --mapid=12345001 --max-nodes=200000 --output=xml input.osm
Any pointers as to where to look would be appreciated.
Thanks
Paul
Exception in thread "main" java.lang.IndexOutOfBoundsException: Index: 2147483647, Size: 6 at java.util.ArrayList.rangeCheck(ArrayList.java:604) at java.util.ArrayList.get(ArrayList.java:382) at uk.me.parabola.splitter.WriterDictionaryInt.getBitSet(WriterDictionaryInt.java:85) at uk.me.parabola.splitter.SplitProcessor.writeNode(SplitProcessor.java:297) at uk.me.parabola.splitter.SplitProcessor.processNode(SplitProcessor.java:102) at uk.me.parabola.splitter.OSMParser.endElement(OSMParser.java:219) at uk.me.parabola.splitter.AbstractXppParser.parse(AbstractXppParser.java:61) at uk.me.parabola.splitter.Main.processMap(Main.java:831) at uk.me.parabola.splitter.Main.writeAreas(Main.java:769) at uk.me.parabola.splitter.Main.split(Main.java:277) at uk.me.parabola.splitter.Main.start(Main.java:162) at uk.me.parabola.splitter.Main.main(Main.java:146) _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
On 24/07/13 08:43, Gerd Petermann wrote:
Hi Paul,
I have no clue yet. Are you able to reproduce the error when you convert the input file to o5m or osm.pbf first? If yes, please upload the file and send a link.
Gerd
Gerd, I've had some spare time today to do some experimentation and it seems that with --keep-complete=true (the default) then node and way id's in the order of 1,000,000,000,000,000,000 are what is causing the error. With --keep-complete=false then such node and way id's are acceptable so for now I'm re-writing the id's as part of the pre-processing but this would seem to be inconsistent behaviour in splitter Hope that helps Paul
Hi Paul, please try to describe what you mean with "node and way id's in the order of 1,000,000,000,000,000,000 are what is causing the error" Is it a special id that triggers the error? Gerd News-3 wrote
On 24/07/13 08:43, Gerd Petermann wrote:
Hi Paul,
I have no clue yet. Are you able to reproduce the error when you convert the input file to o5m or osm.pbf first? If yes, please upload the file and send a link.
Gerd
Gerd,
I've had some spare time today to do some experimentation and it seems that with --keep-complete=true (the default) then node and way id's in the order of 1,000,000,000,000,000,000 are what is causing the error. With --keep-complete=false then such node and way id's are acceptable so for now I'm re-writing the id's as part of the pre-processing but this would seem to be inconsistent behaviour in splitter
Hope that helps
Paul _______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
-- View this message in context: http://gis.19327.n5.nabble.com/Splitter-Crash-tp5770025p5771352.html Sent from the Mkgmap Development mailing list archive at Nabble.com.
On 25/07/13 14:09, GerdP wrote:
Hi Paul,
please try to describe what you mean with "node and way id's in the order of 1,000,000,000,000,000,000 are what is causing the error"
Is it a special id that triggers the error?
Gerd
So........the following simplified input would cause a crash <?xml version="1.0" encoding="utf-8"?> <osm version="0.6" generator="news"> <node id="1000000000000000000" lat="46.100416666666668" lon="3.39375" /> <node id="1000000000000000001" lat="46.100288461538462" lon="3.3929166666666668" /> <node id="1000000000000000002" lat="46.1003125" lon="3.3920833333333333" /> <node id="1000000000000000003" lat="46.10008333333333" lon="3.39125" /> <node id="1000000000000000004" lat="46.099895833333335" lon="3.3904166666666669" /> <way id="1000000000000000000"> <nd ref="1000000000000000000" /> <nd ref="1000000000000000001" /> <nd ref="1000000000000000002" /> <nd ref="1000000000000000003" /> <nd ref="1000000000000000004" /> <nd ref="1000000000000000000" /> <tag k="ele" v="300" /> <tag k="contour" v="elevation" /> </way> </osm> Whereas the following would be ok <?xml version="1.0" encoding="utf-8"?> <osm version="0.6" generator="news"> <node id="1000000000" lat="46.100416666666668" lon="3.39375" /> <node id="1000000001" lat="46.100288461538462" lon="3.3929166666666668" /> <node id="1000000002" lat="46.1003125" lon="3.3920833333333333" /> <node id="1000000003" lat="46.10008333333333" lon="3.39125" /> <node id="1000000004" lat="46.099895833333335" lon="3.3904166666666669" /> <way id="1000000000"> <nd ref="1000000000" /> <nd ref="1000000001" /> <nd ref="1000000002" /> <nd ref="1000000003" /> <nd ref="1000000004" /> <nd ref="1000000000" /> <tag k="ele" v="300" /> <tag k="contour" v="elevation" /> </way> </osm> Thanks Paul
Hi Paul, sorry, I am not able to reproduce the problem with splitter r306 on a windows 64bit machine. Attached is the log, please compare with your one. Maybe this will help to find what's going wrong. Gerd splitter.log <http://gis.19327.n5.nabble.com/file/n5771380/splitter.log> News-3 wrote
On 25/07/13 14:09, GerdP wrote:
Hi Paul,
please try to describe what you mean with "node and way id's in the order of 1,000,000,000,000,000,000 are what is causing the error"
Is it a special id that triggers the error?
Gerd
So........the following simplified input would cause a crash
<?xml version="1.0" encoding="utf-8"?> <osm version="0.6" generator="news">
<node id="1000000000000000000" lat="46.100416666666668" lon="3.39375" />
<node id="1000000000000000001" lat="46.100288461538462" lon="3.3929166666666668" />
<node id="1000000000000000002" lat="46.1003125" lon="3.3920833333333333" />
<node id="1000000000000000003" lat="46.10008333333333" lon="3.39125" />
<node id="1000000000000000004" lat="46.099895833333335" lon="3.3904166666666669" />
<way id="1000000000000000000">
<nd ref="1000000000000000000" />
<nd ref="1000000000000000001" />
<nd ref="1000000000000000002" />
<nd ref="1000000000000000003" />
<nd ref="1000000000000000004" />
<nd ref="1000000000000000000" />
<tag k="ele" v="300" />
<tag k="contour" v="elevation" />
</way> </osm> Whereas the following would be ok
<?xml version="1.0" encoding="utf-8"?> <osm version="0.6" generator="news">
<node id="1000000000" lat="46.100416666666668" lon="3.39375" />
<node id="1000000001" lat="46.100288461538462" lon="3.3929166666666668" />
<node id="1000000002" lat="46.1003125" lon="3.3920833333333333" />
<node id="1000000003" lat="46.10008333333333" lon="3.39125" />
<node id="1000000004" lat="46.099895833333335" lon="3.3904166666666669" />
<way id="1000000000">
<nd ref="1000000000" />
<nd ref="1000000001" />
<nd ref="1000000002" />
<nd ref="1000000003" />
<nd ref="1000000004" />
<nd ref="1000000000" />
<tag k="ele" v="300" />
<tag k="contour" v="elevation" />
</way> </osm> Thanks
Paul _______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
-- View this message in context: http://gis.19327.n5.nabble.com/Splitter-Crash-tp5770025p5771380.html Sent from the Mkgmap Development mailing list archive at Nabble.com.
On 25/07/13 16:14, GerdP wrote:
Hi Paul,
sorry, I am not able to reproduce the problem with splitter r306 on a windows 64bit machine. Attached is the log, please compare with your one. Maybe this will help to find what's going wrong.
Gerd
Gerd If you use the data at http://pointdee.co.uk/files/crash.osm and command line java -Xmx1024M -jar splitter.jar --mapid=12345001 --max-nodes=2500 --output=xml crash.osm then you should be able to reproduce it. I've had this crash on 64 bit Linux and 32 bit Windows so I guess it's not OS related Thanks Paul
Hi Paul, your data contains the id 9223372036854775807 which is equal to Long.MAX_VALUE (2**63-1) This value has a special meaning in the code. I'll add a check to print a proper message, but you have to use smaller ids. Gerd News-3 wrote
On 25/07/13 16:14, GerdP wrote:
Hi Paul,
sorry, I am not able to reproduce the problem with splitter r306 on a windows 64bit machine. Attached is the log, please compare with your one. Maybe this will help to find what's going wrong.
Gerd
Gerd
If you use the data at http://pointdee.co.uk/files/crash.osm and command line
java -Xmx1024M -jar splitter.jar --mapid=12345001 --max-nodes=2500 --output=xml crash.osm
then you should be able to reproduce it. I've had this crash on 64 bit Linux and 32 bit Windows so I guess it's not OS related
Thanks
Paul _______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
-- View this message in context: http://gis.19327.n5.nabble.com/Splitter-Crash-tp5770025p5771387.html Sent from the Mkgmap Development mailing list archive at Nabble.com.
On 25/07/13 17:30, GerdP wrote:
Hi Paul,
your data contains the id 9223372036854775807 which is equal to Long.MAX_VALUE (2**63-1) This value has a special meaning in the code. I'll add a check to print a proper message, but you have to use smaller ids.
Gerd
I'd already realised that smaller id's were the way to go but at least now you've found the cause Thanks Paul
On 25/07/13 14:09, GerdP wrote:
Hi Paul,
please try to describe what you mean with "node and way id's in the order of 1,000,000,000,000,000,000 are what is causing the error"
Gerd Sorry about that. In english the phrase "numbers iin the order of 1000000000000000000" means "numbers around the same size as or similar to 1000000000000000000". I'm guessing you're a not a native english speaker so you probably haven't come across that before Apologies Paul
No problem, I am always happy to learn something new :-) Gerd News-3 wrote
On 25/07/13 14:09, GerdP wrote:
Hi Paul,
please try to describe what you mean with "node and way id's in the order of 1,000,000,000,000,000,000 are what is causing the error"
Gerd
Sorry about that. In english the phrase "numbers iin the order of 1000000000000000000" means "numbers around the same size as or similar to 1000000000000000000". I'm guessing you're a not a native english speaker so you probably haven't come across that before
Apologies
Paul
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
-- View this message in context: http://gis.19327.n5.nabble.com/Splitter-Crash-tp5770025p5771381.html Sent from the Mkgmap Development mailing list archive at Nabble.com.
participants (4)
-
Gerd Petermann -
GerdP
-
News -
Paul