error handling in splitter
Hi all, I noticed that splitter is quite relaxed regarding error handling ;-) For example, all kinds of I/O errors are catched and may produce stack traces, but processing continues with more or less useless results. Example stdout for a corrupted o5m input : Processing test.o5m java.io.IOException: wrong header byte 13 at uk.me.parabola.splitter.O5mMapParser.parse(O5mMapParser.java:124) at uk.me.parabola.splitter.Main.processOSMFiles(Main.java:1379) at uk.me.parabola.splitter.Main.processMap(Main.java:891) at uk.me.parabola.splitter.Main.calculateAreas(Main.java:579) at uk.me.parabola.splitter.Main.split(Main.java:259) at uk.me.parabola.splitter.Main.start(Main.java:180) at uk.me.parabola.splitter.Main.main(Main.java:152) in 1 file Time: Wed May 14 08:24:16 CEST 2014 Failed to calculate areas. See log for details. Failed to calculate areas. Sorry. Cannot split the file without creating huge, almost empty, tiles. Please specify a bounding polygon with the --polygon-file parameter. Time finished: Wed May 14 08:24:16 CEST 2014 Total time taken: 0s I think splitter should be changed so that it stops immediately with rc 1 if - an input file is missing or can't be read (e.g. invalid numbers, corrupted binary files, etc.) - an output file can't be written (no access rights, disk full, etc) and a useful error message in stdout and stderr. I try to code it like this in the new refactoring branch. I also try to code a new log handling, so that debug info is written to its own file. Gerd
Hi Gerd, this sounds very reasonable! How about implementing this like the way mkgmap handles it? If given the option --keep-going it will continue as splitter does now and the new default would be to stop processing? BTW: Last time i suspected mkgmap to produce bad maps, it was because of a corrupted *.o5m, which splitter happily converted to useless data. So again, yes, i would love this new behavior! Cheers, Uli -- View this message in context: http://gis.19327.n5.nabble.com/error-handling-in-splitter-tp5806188p5806198.... Sent from the Mkgmap Development mailing list archive at Nabble.com.
Hi Uli,
this sounds very reasonable! How about implementing this like the way mkgmap handles it? If given the option --keep-going it will continue as splitter does now and the new default would be to stop processing?
I thought about this as well, but in what case would it be useful to continue? I see these potential cases to print warning and continue instead of stopping: - if output of densities_out.txt fails - if output of a *.poly or *.kml file fails - if input of --geonames file fails If any other file causes trouble I see no reason to continue.
BTW: Last time i suspected mkgmap to produce bad maps, it was because of a corrupted *.o5m, which splitter happily converted to useless data. So again, yes, i would love this new behavior!
yes, I kept that in mind. Gerd
Hi Gerd, i just thought of some kind of backward compatibility. In my opinion i see no reason for splitter to continue processing corrupted data. BTW: Could be there are some files missing for checkin? Somehow, the new splitter builds do not appear for download. Uli -- View this message in context: http://gis.19327.n5.nabble.com/error-handling-in-splitter-tp5806188p5806207.... Sent from the Mkgmap Development mailing list archive at Nabble.com.
Hi Uli,
BTW: Could be there are some files missing for checkin? Somehow, the new splitter builds do not appear for download.
you have to scroll down if you look at http://www.mkgmap.org.uk/download/splitter.html Gerd
Hi Gerd, you are right, i missed, it's a branch! *handtoforehead* ;) Uli -- View this message in context: http://gis.19327.n5.nabble.com/error-handling-in-splitter-tp5806188p5806213.... Sent from the Mkgmap Development mailing list archive at Nabble.com.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi Gerd, I would agree, that if input-data cannot be read or tiles cannot be written, splitter should stop and write an error-message. A special log shouldn't be required if this log will only contain stdout and/or stderr. This can be written to a file by cmd. Other things could be done: As you mainly had written the overlapping-code for me, I think it can be removed. The new method is so much better, that I would highly recommend not to use the older one. Splitter and mkgmap are using different libs of fastutil, osmpbf and protobuf. Maybe it would be a good thing to let them use the same libs. Henning -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.14 (GNU/Linux) iQEcBAEBAgAGBQJTc5NeAAoJEKXggIeC16WP/lMH/iMIHOpOoiCKZLqxf0sdwATy afRs/I1V1bg+lht8oQrgCHju1jLOQIiBAxHyATlsNUvJ2rS8jhcy+wmwF+AvxKxw x+BaD+KNbJDzq3Jj6Ll0O3VO556UVYWUv1jNr7YBU7omv0774H4vgejVsnykt0/x g8bnqU/jGpOJwcRuf+xt4HSrhF287+Dsm/dgCaGIOLeUGiz1KDVA1PwkpqyRJ467 cdJEwK2y43BYCzTIspyr3fCaAcc4EHLm1n9NL0EfUV5S3W3solIxrFj0H+RnxKlU hlYTMIi0suxWqL2xTOTGlmzAf7emFHMk0ofLv/fi8/vmAlI2ajzZpRACr0l6wbI= =2EMX -----END PGP SIGNATURE-----
Hi Henning,
I would agree, that if input-data cannot be read or tiles cannot be written, splitter should stop and write an error-message.
A special log shouldn't be required if this log will only contain stdout and/or stderr. This can be written to a file by cmd.
I think about a special log file containing more details than the current stdout and a condensed stdout.
Other things could be done:
As you mainly had written the overlapping-code for me, I think it can be removed. The new method is so much better, that I would highly recommend not to use the older one.
OK. I'll try to find out if that could help to simplify or improve processing for all "normal" users.
Splitter and mkgmap are using different libs of fastutil, osmpbf and protobuf. Maybe it would be a good thing to let them use the same libs.
I was not aware of that. The fastutil lib is quite out-aged now. The protobuf lib is newer in mkgmap, osmpbf libs have different names but equal content. @Steve: I don't know the reason for that. Is it intended? I'll check if the newer fastutil version improves anything for mkgmap or splitter. Gerd
On 14/05/14 17:43, Gerd Petermann wrote:
Splitter and mkgmap are using different libs of fastutil, osmpbf and protobuf. Maybe it would be a good thing to let them use the same libs. I was not aware of that. The fastutil lib is quite out-aged now. The protobuf lib is newer in mkgmap, osmpbf libs have different names but equal content. @Steve: I don't know the reason for that. Is it intended?
There is no particular reason. At one time we updated the mkgmap fastutil.jar so that the same one would work on both mkgmap and splitter. The names are different mostly because splitter hasn't been converted to use ivy. ..Steve
Hi Steve,
Splitter and mkgmap are using different libs of fastutil, osmpbf and protobuf. Maybe it would be a good thing to let them use the same libs. I was not aware of that. The fastutil lib is quite out-aged now. The protobuf lib is newer in mkgmap, osmpbf libs have different names but equal content. @Steve: I don't know the reason for that. Is it intended?
There is no particular reason. At one time we updated the mkgmap fastutil.jar so that the same one would work on both mkgmap and splitter.
The names are different mostly because splitter hasn't been converted to use ivy.
I think we should update fastutil to 6.5.15 and use the latest pbf libs for both programs. Reg. fastutil: I'd like to have a few more classes in it, so I've attached a small program that uses all classes which seem to be useful in splitter or mkgmap. I don't know much about ivy. Is it a lot of work to change splitter to use it? Gerd
Hi Gerd
I think we should update fastutil to 6.5.15 and use the latest pbf libs for both programs.
OK, and I would like to upgrade junit to 4.11.
Reg. fastutil: I'd like to have a few more classes in it, so I've attached a small program that uses all classes which seem to be useful in splitter or mkgmap.
OK
I don't know much about ivy. Is it a lot of work to change splitter to use it?
It wouldn't take long since it will be mostly the same as for mkgmap and display. ..Steve
Hi Steve, thanks, build works fine. I've committed it in the branch. I did not yet test if the pbf updates have an effect, will do that tomorrow. Gerd Steve Ratcliffe wrote
Hi Gerd
I don't know much about ivy. Is it a lot of work to change splitter to use it?
Here is a patch for ivy.
Everything upgraded to the latest except for fastutils.
..Steve
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
ivy.patch (11K) <http://gis.19327.n5.nabble.com/attachment/5806499/0/ivy.patch>;
-- View this message in context: http://gis.19327.n5.nabble.com/error-handling-in-splitter-tp5806188p5806500.... Sent from the Mkgmap Development mailing list archive at Nabble.com.
Hi Steve, the new pbf libs are much better. I've just tried to split an older Germany.osm.pbf on my netbook and I see an improvement 681s -> 476s So, o5m is no longer that much better. Gerd GerdP wrote
Hi Steve,
thanks, build works fine. I've committed it in the branch. I did not yet test if the pbf updates have an effect, will do that tomorrow.
Gerd Steve Ratcliffe wrote
Hi Gerd
I don't know much about ivy. Is it a lot of work to change splitter to use it?
Here is a patch for ivy.
Everything upgraded to the latest except for fastutils.
..Steve
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
ivy.patch (11K) <http://gis.19327.n5.nabble.com/attachment/5806499/0/ivy.patch>;
-- View this message in context: http://gis.19327.n5.nabble.com/error-handling-in-splitter-tp5806188p5806506.... Sent from the Mkgmap Development mailing list archive at Nabble.com.
On 16/05/14 07:00, Gerd Petermann wrote:
I'd like to have a few more classes in it, so I've attached a small program that uses all classes which seem to be useful in splitter or mkgmap.
I had to alter the program so that it actually used the classes (attached). I've now created minimal fastutil jars for mkgmap+splitter and for UseFastutil They can downloaded by changing the version to 6.5.15-mkg.1 and 6.5.15-mkg.1b respectively. mkgmap+splitter: <dependency org="it.unimi.dsi" name="fastutil" rev="6.5.15-mkg.1" conf="compile->default(*)"/> UseFastutil: <dependency org="it.unimi.dsi" name="fastutil" rev="6.5.15-mkg.1b" conf="compile->default(*)"/> ..Steve
Hi Steve, thanks again. I've changed the branch to use fastutil-6.5.15-mkg.1b.jar now. After the merge to trunk I'll update ivy.xml mkgmap as well, so that both use the same libs. BTW: On my PC I don't see the big improvements in the pbf libs, so maybe I've hit an edge case on my netbook. Gerd On 16/05/14 07:00, Gerd Petermann wrote:
I'd like to have a few more classes in it, so I've attached a small program that uses all classes which seem to be useful in splitter or mkgmap.
I had to alter the program so that it actually used the classes (attached). I've now created minimal fastutil jars for mkgmap+splitter and for UseFastutil They can downloaded by changing the version to 6.5.15-mkg.1 and 6.5.15-mkg.1b respectively. mkgmap+splitter: <dependency org="it.unimi.dsi" name="fastutil" rev="6.5.15-mkg.1" conf="compile->default(*)"/> UseFastutil: <dependency org="it.unimi.dsi" name="fastutil" rev="6.5.15-mkg.1b" conf="compile->default(*)"/> ..Steve _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
participants (5)
-
Gerd Petermann -
GerdP
-
Henning Scholland -
Steve Ratcliffe -
UliBaer