Hi Ticker, hi Valentin my understanding is that the error message from MapSplitter was a false positive because mkgmap reports 0 MapFailedExceptions. The crash with IndexOutOfBoundsException occurs in the final combiner phase for the gmapsupp. It might be caused by an I/O error or a corrupted *.img file. @Valentin: If you didn't already try that I suggest to clear the directory and restart the compilation. If that produces the same IndexOutOfBoundsException please try again without the file 63240201.osm.pbf. Only if that works fine we can be sure that 63240201.osm.pbf is causing the crash. ciao, Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Freitag, 31. Juli 2020 17:01 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] Error in Africa? Hi Valentin & Gerd @Gerd, have you started looking at this? The initial error is related to the splitting of the tile into subdivisions and is VERY dependent on the size and layout of all map elements in the tile, which will depend on the style and many options as well as the actual OSM input. If not using the default style, can you zip up and post what you are using, also mkgmap command line and options. The exception happens later but is probably a consequence of the split failure. Ticker On Thu, 2020-07-30 at 23:33 +0300, valentin3151@gmail.com wrote:

Hi all, My daily compilation of Africa was stopped because error is appear: SEVERE (MapSplitter): 63240201.osm.pbf: Single item predicted to exceed subdivision http://www.openstreetmap.org/?mlat=10.412207&mlon=-13.010709&zoom=17 Number of MapFailedExceptions: 0 Exception in thread "main" java.lang.IndexOutOfBoundsException: Index: 113, Size: 3 at java.util.ArrayList.rangeCheck(Unknown Source) at java.util.ArrayList.get(Unknown Source) at uk.me.parabola.imgfmt.app.net.NETFileReader.getRoads(NETFileReader.ja va:116) at uk.me.parabola.imgfmt.app.map.MapReader.getRoads(MapReader.java:205) at uk.me.parabola.mkgmap.combiners.MdrBuilder.addStreets(MdrBuilder.java :298) at uk.me.parabola.mkgmap.combiners.MdrBuilder.onMapEnd(MdrBuilder.java:1 71) at uk.me.parabola.mkgmap.combiners.GmapsuppBuilder.onMapEnd(GmapsuppBuil der.java:164) at uk.me.parabola.mkgmap.main.Main.endOptions(Main.java:665) at uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.ja va:125) at uk.me.parabola.mkgmap.main.Main.mainStart(Main.java:144) at uk.me.parabola.mkgmap.main.Main.main(Main.java:115)

Link to my tile where the error is: https://maptourist.org/tmp/63240201.osm.pbf

I tried to find the cause of this error in OSM but could not. The link from above error log points to the location with completely no data somewhere in the middle of Guinea. As an experiment, I loaded the problem tile into JOSM and fixed all automatically found errors. Then I tried to compile the corrected file. No positive effect. The error is still there. I use mkgmap-r4564, but I found there is no dependency with other versions. How do I find what exactly is causing this error in the OSM data? Previously, this map was compiled many times over a long time without such problem.

_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Hello, all! Does this tile happen to contain the Port Said area near Cairo, Egypt? If so, you may be facing the same problem that I encountered in splitter. Splitter only limits the number of nodes in a tile. It ought to limit the number of ways as well. The Port Said area has an immense number of single city block length streets not joined into longer streets. I have built maps for the entire planet. That is the only place that I found that had that problem. The solution was to build Africa using a max nodes value of about 600 thousand nodes as you did, which brought the number of ways down sufficiently to avoid the problem in mkgmap. If splitter had an option to limit the number of nodes, then splitter would have divided the Port Said tile sufficiently that just those tiles would have had to be made small. Randolph J. Herber On 7/31/2020 3:09 PM, valentin3151@gmail.com wrote:

Re: Error in Africa? Hi Ticker, hi Gerd, thanks for your attention.

I continue to do my experiments. I tried to compile the map from this one 63240201.osm.pbf with and without the --gmapsupp option (of course to an empty output folder). I get the same result on my two FreeBSD servers and my Windows home computer. I am getting this error on all my systems.

Command line for testing: java -jar ^        D:\OSM\mkgmap\mkgmap-r4564\mkgmap4564.jar ^        --output-dir=D:\Garmin\test\ ^        --family-name="OSM test %date%" ^        --series-name="OSM test %date%" ^        --overview-mapname="OSM_test" ^        --area-name="OSM Test %date%" ^        --family-id=480 ^        --read-config=./config2018/optionsfile.args ^        --mapname=63080000 ^        --style-file=Config2018 ^        --tdbfile ^          --bounds=D:\OSM\mkgmap\boundary\world ^        --road-name-config=./config2018/roadNameConfig.txt ^        --name-tag-list=name:en,int_name,name ^        --code-page:65001 --style-option=code-page=1250 ^        ./config2018/OSM-2018.txt --description="OSM test" ^        --gmapsupp ^        --add-boundary-nodes-at-admin-boundaries=3 ^        D:\OSM\test\63240201.osm.pbf

Link to my Style files: https://maptourist.org/osm-garmin/CurrentConfigs/

Earlier I found a way to fix this error when I use the --max-nodes=600000 value. But this will produce an extremely large number of tiles! I really don't like this result. The same number of tiles is produced by the splitter when I making a map of the whole of Europe without the --max-nodes option. The compilation of the map of Europe is stable and looks very good without errors. But the original europe-latest.o5m is four times larger than africa-latest.o5m! I see this is an old and complex spiltter problem. I'm just trying to find out how OSM data is causing this error. It is still unpredictable for me.

Hi Ticker, hi Valentin

my understanding is that the error message from MapSplitter was a false positive because mkgmap reports 0 MapFailedExceptions.

The crash with IndexOutOfBoundsException occurs in the final combiner phase for the gmapsupp. It might be caused by an I/O error or a corrupted *.img file.

@Valentin: If you didn't already try that I suggest to clear the directory and restart the compilation. If that produces the same IndexOutOfBoundsException  please try again without the file 63240201.osm.pbf. Only if that works fine we can be sure that 63240201.osm.pbf is causing the crash.

ciao, Gerd

________________________________________ Von: mkgmap-dev <[hidden email] </user/SendEmail.jtp?type=node&node=5971915&i=0>> im Auftrag von Ticker Berkin <[hidden email] </user/SendEmail.jtp?type=node&node=5971915&i=1>> Gesendet: Freitag, 31. Juli 2020 17:01 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] Error in Africa?

Hi Valentin & Gerd

@Gerd, have you started looking at this?

The initial error is related to the splitting of the tile into subdivisions and is VERY dependent on the size and layout of all map elements in the tile, which will depend on the style and many options as well as the actual OSM input.

If not using the default style, can you zip up and post what you are using, also mkgmap command line and options.

The exception happens later but is probably a consequence of the split failure.

Ticker

On Thu, 2020-07-30 at 23:33 +0300, [hidden email] </user/SendEmail.jtp?type=node&node=5971915&i=2> wrote:

...

Hi all, My daily compilation of Africa was stopped because error is appear: SEVERE (MapSplitter): 63240201.osm.pbf: Single item predicted to exceed subdivision http://www.openstreetmap.org/?mlat=10.412207&mlon=-13.010709&zoom=17         Number  of  MapFailedExceptions:  0 Exception in thread "main"         java.lang.IndexOutOfBoundsException:  Index:  113,  Size: 3         at java.util.ArrayList.rangeCheck(Unknown Source)         at java.util.ArrayList.get(Unknown Source)         at uk.me.parabola.imgfmt.app.net.NETFileReader.getRoads(NETFileReader.ja va:116)         at uk.me.parabola.imgfmt.app.map.MapReader.getRoads(MapReader.java:205)         at uk.me.parabola.mkgmap.combiners.MdrBuilder.addStreets(MdrBuilder.java :298)         at uk.me.parabola.mkgmap.combiners.MdrBuilder.onMapEnd(MdrBuilder.java:1 71)         at uk.me.parabola.mkgmap.combiners.GmapsuppBuilder.onMapEnd(GmapsuppBuil der.java:164)         at uk.me.parabola.mkgmap.main.Main.endOptions(Main.java:665)         at uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.ja va:125)         at uk.me.parabola.mkgmap.main.Main.mainStart(Main.java:144)         at uk.me.parabola.mkgmap.main.Main.main(Main.java:115)

Link to my tile where the error is: https://maptourist.org/tmp/63240201.osm.pbf

I tried to find the cause of this error in OSM but could not. The link from  above  error  log  points  to  the  location  with completely no data somewhere in the middle of Guinea. As  an  experiment, I loaded the problem  tile  into  JOSM  and  fixed all  automatically found errors. Then I tried to compile the corrected file. No positive effect. The error is still there. I use mkgmap-r4564, but I found there is no dependency with other versions. How  do  I  find what exactly is causing this error in the OSM data? Previously, this map was compiled many times over a long time without such problem.

_______________________________________________ mkgmap-dev mailing list [hidden email] </user/SendEmail.jtp?type=node&node=5971915&i=3> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list [hidden email] </user/SendEmail.jtp?type=node&node=5971915&i=4> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

------------------------------------------------------------------------ *If you reply to this email, your message will be added to the discussion below: *http://gis.19327.n8.nabble.com/Error-in-Africa-tp5971902p5971915.html To unsubscribe from Mkgmap Development, click here <http://gis.19327.n8.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=5324443&code=dmFsZW50aW4zMTUxQGdtYWlsLmNvbXw1MzI0NDQzfC0xMDQyNTE0Njg2>. NAML <http://gis.19327.n8.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>

/-- С уважением,  Valentin3151 /mailto:valentin3151@gmail.com

_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Hi I can reproduce this problem, the first message is about subdivision splitting (in Guinea, not Egypt), and is the same as you had. SEVE: uk.me.parabola.mkgmap.build.MapSplitter Config2018/63240201.osm.pbf: Single item predicted to exceed subdivision http://www.openstreetmap.org/?mlat=10.412207&mlon=-13.010709&zoom=17 The second/follow-on exception is slightly different: java.lang.AssertionError: too many polylines at level 0 at uk.me.parabola.imgfmt.app.net.RoadDef.writeLevelCount(RoadDef.java:338) at uk.me.parabola.imgfmt.app.net.RoadDef.writeNet1(RoadDef.java:231) etc but I don't think the difference is significant. The subdivision starts off with 239 roads (OK, but the area doesn't seem to have many roads) needing an estimated size of 239710 (far to much for a subdivision), but then the logic which should split the subdivision doesn't get activated. I think the problem is that there are no polygons or points in this area and this has confused the 'splittable' test. I'll investigate more tomorrow. Ticker

Hi Forget splitting in OSM, mkgmap joins them back together. Ticker On Mon, 2020-08-03 at 11:07 +0100, Ticker Berkin wrote:

Hi

It appears that the problem caused by:

https://www.openstreetmap.org/way/690517138 and/or https://www.openstreetmap.org/way/690517135

[name="Reservoire Souapiti", waterway=drain]

which is converted into a routable road by the rule

(waterway=canal

...

waterway=river waterway=rapid | waterway=rapids | whitewater=rapid | whitewater=rapids waterway=waterfall waterway=drain waterway=yes waterway=stream) & tunnel!=* & is_closed()=false {add access=no; add taxi=yes} [0x0f road_class=0 road_speed=0 resolution 24 continue]

and also:

waterway=drain [0x18 resolution 19 continue]

but it is the road that causes the problem.

This waterway feature is already split into many separate ways, and maybe the offending segments (at the moment each containing about 1775 nodes) could be split again in OSM.

but I'll investigate long road splitting in mkgmap.

Ticker

_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Hi Gerd I was thinking the problem was in, or at least could be solved reasonably easily in MapSplitter/MapArea. However this is too late because all the logic relating to splitting/merging roads and keeping track of highway ids/counts/restrictions/etc happens in StyledConverter. All MapSplitter does with roads with lots of points is to put them in a subdivision on their own. The simplest way of fixing this is to stop roads being merged if the results probably won't fit into a subdivision. It isn't an exact calculation because the line smoothing filters haven't been run yet. I've attached a patch to do this. Ticker

Hi Gerd In theory yes, but messages in various forums seem to suggest that creating ways with more than 2000 points causes problems, and the feature in question has been chopped into multiple ways of ~1700 points. I estimated the maximum number of points before limits might be exceeded is 16200. This case is a bit pathological. I doubt if there are real roads this long without change of name/speed/class Ticker On Mon, 2020-08-03 at 19:52 +0000, Gerd Petermann wrote:

Hi Ticker, whatever Roadmerger does, the same data may be in OSM. There is code in Styledconverter to split roads and also later down the chain.

Problem should be fixed there. Cannot help more until I am back. Cioa Gerd

---- Ticker Berkin schrieb ----

Hi Gerd

I was thinking the problem was in, or at least could be solved reasonably easily in MapSplitter/MapArea. However this is too late because all the logic relating to splitting/merging roads and keeping track of highway ids/counts/restrictions/etc happens in StyledConverter. All MapSplitter does with roads with lots of points is to put them in a subdivision on their own.

The simplest way of fixing this is to stop roads being merged if the results probably won't fit into a subdivision. It isn't an exact calculation because the line smoothing filters haven't been run yet.

I've attached a patch to do this.

Ticker

_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Hi Ticker, I've now found the time to look at the details. The problem was not the size of the sub division but the number of Polyline instances created for the long road. This has to be < 0x80. The corresponding check was only implemented with an assert statement and Valentins call doesn't enable assertions, so invalid data was written. I've attached a patch which implements the splitting in StyledConverter and which throws a MapFailedException instead of using an assertion. Open problem: MapSplitter still SCHW: uk.me.parabola.mkgmap.build.MapSplitter f:\dwnload\temp\63240201.osm.pbf: Single item predicted to exceed subdivision http://www.openstreetmap.org/?mlat=10.489948&mlon=-13.065190&zoom=17 SCHW: uk.me.parabola.mkgmap.build.MapSplitter f:\dwnload\temp\63240201.osm.pbf: Single item predicted to exceed subdivision http://www.openstreetmap.org/?mlat=10.368004&mlon=-12.969704&zoom=17 while the map seems to be OK. So I think the estimation also needs a fix? Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von valentin3151@gmail.com <valentin3151@gmail.com> Gesendet: Dienstag, 4. August 2020 12:28 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] Error in Africa? Hi, Ticker. Thanks for your investigation. I am very impressed with how some people are using the wrong tags. With --x-no-mergeroads option, my problem was fixed. Great! Best regadrs, Valrntin _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Hi Ticker, mergeRoads() is called before any addRoadsWithoutLoops(). I hope this clarifies all? Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Montag, 14. September 2020 15:17 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] Error in Africa? Hi Gerd I don't understand how splitting the road in addRoadsWithoutLoops() changes anything - won't the bits (and the other OSM ways with same typ/names/class/speed) just be joined up again during StyledConverted end() processing in the call mergeRoads(). For a road with many points, it is going to be LineSplitterFilter that creates too many road segments. Probably the easiest way to prevent this many-pointed road causing this problem is to split it before invoking the filter but after mergeRoads(). However I maintain that a single OSM way won't cause the problem and so stopping RoadMerger from creating the problem is a safe solution. RoadMerger.MAX_MERGED_POINTS should be re-expressed as per StyledConverter.MAX_ROAD_POINTS. I understand the change to remove the warning in MapSplitter about a single item being too big. Ticker On Sun, 2020-09-06 at 08:56 +0000, Gerd Petermann wrote:

Hi Ticker,

I've now found the time to look at the details. The problem was not the size of the sub division but the number of Polyline instances created for the long road. This has to be < 0x80. The corresponding check was only implemented with an assert statement and Valentins call doesn't enable assertions, so invalid data was written. I've attached a patch which implements the splitting in StyledConverter and which throws a MapFailedException instead of using an assertion. Open problem: MapSplitter still SCHW: uk.me.parabola.mkgmap.build.MapSplitter f:\dwnload\temp\63240201.osm.pbf: Single item predicted to exceed subdivision http://www.openstreetmap.org/?mlat=10.489948&mlon=-13.065190&zoom=17 SCHW: uk.me.parabola.mkgmap.build.MapSplitter f:\dwnload\temp\63240201.osm.pbf: Single item predicted to exceed subdivision http://www.openstreetmap.org/?mlat=10.368004&mlon=-12.969704&zoom=17 while the map seems to be OK. So I think the estimation also needs a fix?

Gerd

________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von valentin3151@gmail.com <valentin3151@gmail.com> Gesendet: Dienstag, 4. August 2020 12:28 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] Error in Africa?

Hi, Ticker.

Thanks for your investigation. I am very impressed with how some people are using the wrong tags. With --x-no-mergeroads option, my problem was fixed. Great!

Best regadrs, Valrntin

_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev