Re: [mkgmap-dev] [PATCH] Fix crash when parsing invalid UTF-8 files (WanMil)
Thanks for the patch. Can you post the exception you want to catch in your patch? The patch covers a problem which I think should be fixed at its origin. So a stacktrace would be very helpful.
Thanks! WanMil
The origin is either JVM6 or the benelux dataset, depending on how you look at it :) The complete trace and original mail message I send some time ago: mkgmap r2023 (and many older version) crash when running with --road-name-pois on file like in http://planet.openstreetmap.nl/benelux ++ /usr/lib/jvm/java-6-sun/bin/java -ea -Xmx4000m -jar ../mkgmap-r2023/mkgmap.jar --style-file=../aiostyles/basemap_style --tdbfile --gmapsupp --max-jobs --adjust-turn-headings --reduce-point-density=5 --merge-lines --route --remove-short-arcs=1 --link-pois-to-ways --family-id=4 --product-id=45 --family-name=OpenStreetmap --draw-priority=29 --add-pois-to-areas --road-name-pois 63240001.osm.pbf [...snip...] 63240075.osm.pbf ../aiostyles/basemap.TYP java.lang.IllegalStateException: Current state = FLUSHED, new state = CODING_END at java.nio.charset.CharsetEncoder.throwIllegalStateException(CharsetEncoder.java:951) at java.nio.charset.CharsetEncoder.encode(CharsetEncoder.java:537) at java.nio.charset.CharsetEncoder.encode(CharsetEncoder.java:766) at uk.me.parabola.imgfmt.app.srt.Sort.createSortKey(Sort.java:96) at uk.me.parabola.imgfmt.app.srt.Sort.createSortKey(Sort.java:150) at uk.me.parabola.mkgmap.main.MapMaker.makeRoadNamePOIS(MapMaker.java:266) at uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:64) at uk.me.parabola.mkgmap.main.Main$1.call(Main.java:222) at uk.me.parabola.mkgmap.main.Main$1.call(Main.java:219) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Regards, Rutger.
Thanks for the patch. Can you post the exception you want to catch in your patch? The patch covers a problem which I think should be fixed at its origin. So a stacktrace would be very helpful.
Thanks! WanMil
The origin is either JVM6 or the benelux dataset, depending on how you look at it :)
The complete trace and original mail message I send some time ago:
mkgmap r2023 (and many older version) crash when running with --road-name-pois on file like in http://planet.openstreetmap.nl/benelux
++ /usr/lib/jvm/java-6-sun/bin/java -ea -Xmx4000m -jar ../mkgmap-r2023/mkgmap.jar --style-file=../aiostyles/basemap_style --tdbfile --gmapsupp --max-jobs --adjust-turn-headings --reduce-point-density=5 --merge-lines --route --remove-short-arcs=1 --link-pois-to-ways --family-id=4 --product-id=45 --family-name=OpenStreetmap --draw-priority=29 --add-pois-to-areas --road-name-pois 63240001.osm.pbf [...snip...] 63240075.osm.pbf ../aiostyles/basemap.TYP java.lang.IllegalStateException: Current state = FLUSHED, new state = CODING_END at java.nio.charset.CharsetEncoder.throwIllegalStateException(CharsetEncoder.java:951) at java.nio.charset.CharsetEncoder.encode(CharsetEncoder.java:537) at java.nio.charset.CharsetEncoder.encode(CharsetEncoder.java:766) at uk.me.parabola.imgfmt.app.srt.Sort.createSortKey(Sort.java:96) at uk.me.parabola.imgfmt.app.srt.Sort.createSortKey(Sort.java:150) at uk.me.parabola.mkgmap.main.MapMaker.makeRoadNamePOIS(MapMaker.java:266) at uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:64) at uk.me.parabola.mkgmap.main.Main$1.call(Main.java:222) at uk.me.parabola.mkgmap.main.Main$1.call(Main.java:219) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662)
Regards, Rutger.
Thanks Rutger, I think the problem is that the mkgmap Sort object is unsynchronized but used by several threads. If they use the internal encoder at the same time the given exception occurs. I fear that some other problems could also be caused by this (e.g. sorting problems with the index?!?) You can use --max-jobs=1 as workaround (although map creation will take much longer :-(.) The problem should be fixed by using a separate Sort object for each thread. One solution would be if CommandArgs.getSort() returns a copy. I am not sure how to create a valid copy of a Sort object. Steve, can you help? WanMil
Hi
I think the problem is that the mkgmap Sort object is unsynchronized but used by several threads. If they use the internal encoder at the same
Ahh, well spotted. However...
The problem should be fixed by using a separate Sort object for each thread. One solution would be if CommandArgs.getSort() returns a copy. I am not sure how to create a valid copy of a Sort object.
it does return a completely new Sort object for each file... So I'm not sure what is going on, investigating...
Hi
The problem should be fixed by using a separate Sort object for each thread. One solution would be if CommandArgs.getSort() returns a copy. I am not sure how to create a valid copy of a Sort object. Steve, can you help?
The problem is that there is only one MapMaker object. Attached patch fixes it. ..Steve
Using the patch I cannot reproduce the problem any more :-)) +1 to commit. Thanks! WanMil
Hi
The problem should be fixed by using a separate Sort object for each thread. One solution would be if CommandArgs.getSort() returns a copy. I am not sure how to create a valid copy of a Sort object. Steve, can you help?
The problem is that there is only one MapMaker object.
Attached patch fixes it.
..Steve
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
participants (3)
-
Rutger Nijlunsing -
Steve Ratcliffe -
WanMil