Hi Ticker, sorry for my reluctance. I simply have no test case that shows an error (search for xyz not working). If you have one please share it so that I can understand the importance of the patch. I would also be happy if you could create a new branch to test further changes reg. --lower-case. According to MdrCheck we also produce wrong data for mdr 27 (cities are not de-duped). Found this also with Arndts *.img data but did not yet try to find out if MdrCheck is right. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Freitag, 5. November 2021 11:34 An: mkgmap-dev@lists.mkgmap.org.uk; mkgmap-svn@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4811: fix java.lang.AssertionError while building index from unicode tiles Hi Gerd I really don't like the idea of the consistent dedup part of this patch not being applied (Mdr5 & Mdr25). The Mdr7 changes are a slight refactor and a useful comment, but has no logical effect. The Mdr29 changes are an assert to detect inconsistency in Mdr5/25 index pointer lengths before these could cause a crash + .equal()/collator.compare() change that could be removed Ticker On Fri, 2021-11-05 at 09:02 +0000, svn commit wrote:

Version mkgmap-r4811 was committed by gerd on Fri, 05 Nov 2021

fix java.lang.AssertionError while building index from unicode tiles Changes extracted from mdrUnicode_v8.patch by Ticker Berkin

http://www.mkgmap.org.uk/websvn/revision.php?repname=mkgmap&rev=4811 _______________________________________________ mkgmap-svn mailing list To unsubscribe send an mail to mkgmap-svn-leave@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-svn

_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Hi Ticker, ok, so let's wait for that crash to get an example. I really have no idea how to force it :( According to MdrCheck the index is full of errors for the two tiles in China, but maybe it's MdrCheck which is wrong. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Samstag, 6. November 2021 12:27 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4811: fix java.lang.AssertionError while building index from unicode tiles Hi Gerd The crash in Carlos's original problem is due to the inconsistency in the dedups between Mdr5/Mdr25. This could be triggered with any --code-page where city names contain characters that exist in this character-set but are not given sort positions. My mistake with mdrUnicode_v1/2.patch was trying to tackle the case problem at the same time. This is going to be much more difficult. Ticker On Fri, 2021-11-05 at 11:00 +0000, Gerd Petermann wrote:

Hi Ticker,

sorry for my reluctance. I simply have no test case that shows an error (search for xyz not working). If you have one please share it so that I can understand the importance of the patch.

I would also be happy if you could create a new branch to test further changes reg. --lower-case. According to MdrCheck we also produce wrong data for mdr 27 (cities are not de-duped). Found this also with Arndts *.img data but did not yet try to find out if MdrCheck is right.

Gerd

________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Freitag, 5. November 2021 11:34 An: mkgmap-dev@lists.mkgmap.org.uk; mkgmap-svn@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4811: fix java.lang.AssertionError while building index from unicode tiles

Hi Gerd

I really don't like the idea of the consistent dedup part of this patch not being applied (Mdr5 & Mdr25). The Mdr7 changes are a slight refactor and a useful comment, but has no logical effect. The Mdr29 changes are an assert to detect inconsistency in Mdr5/25 index pointer lengths before these could cause a crash + .equal()/collator.compare() change that could be removed

Ticker

On Fri, 2021-11-05 at 09:02 +0000, svn commit wrote:

...

Version mkgmap-r4811 was committed by gerd on Fri, 05 Nov 2021

fix java.lang.AssertionError while building index from unicode tiles Changes extracted from mdrUnicode_v8.patch by Ticker Berkin

http://www.mkgmap.org.uk/websvn/revision.php?repname=mkgmap&rev=4811 _______________________________________________ mkgmap-svn mailing list To unsubscribe send an mail to mkgmap-svn-leave@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-svn

_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

H Ticker, but how do you know that the code in Mdr5 is right (with or without the patch)? Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Samstag, 6. November 2021 15:29 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4811: fix java.lang.AssertionError while building index from unicode tiles Hi Gerd I just hate the idea it is obviously wrong and trivially to make correct. It is unlikely it will ever fail. While it is in this inconsistent state it inhibits any addressing of the more complicated issue about how to tackle the case-differences in city names and what MdrCheck is reporting. Ticker On Sat, 2021-11-06 at 14:10 +0000, Gerd Petermann wrote:

Hi Ticker,

ok, so let's wait for that crash to get an example. I really have no idea how to force it :( According to MdrCheck the index is full of errors for the two tiles in China, but maybe it's MdrCheck which is wrong.

Gerd

________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Samstag, 6. November 2021 12:27 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4811: fix java.lang.AssertionError while building index from unicode tiles

Hi Gerd

The crash in Carlos's original problem is due to the inconsistency in the dedups between Mdr5/Mdr25.

This could be triggered with any --code-page where city names contain characters that exist in this character-set but are not given sort positions.

My mistake with mdrUnicode_v1/2.patch was trying to tackle the case problem at the same time. This is going to be much more difficult.

Ticker

On Fri, 2021-11-05 at 11:00 +0000, Gerd Petermann wrote:

...

Hi Ticker,

sorry for my reluctance. I simply have no test case that shows an error (search for xyz not working). If you have one please share it so that I can understand the importance of the patch.

I would also be happy if you could create a new branch to test further changes reg. --lower-case. According to MdrCheck we also produce wrong data for mdr 27 (cities are not de-duped). Found this also with Arndts *.img data but did not yet try to find out if MdrCheck is right.

Gerd

________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Freitag, 5. November 2021 11:34 An: mkgmap-dev@lists.mkgmap.org.uk; mkgmap-svn@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4811: fix java.lang.AssertionError while building index from unicode tiles

Hi Gerd

I really don't like the idea of the consistent dedup part of this patch not being applied (Mdr5 & Mdr25). The Mdr7 changes are a slight refactor and a useful comment, but has no logical effect. The Mdr29 changes are an assert to detect inconsistency in Mdr5/25 index pointer lengths before these could cause a crash + .equal()/collator.compare() change that could be removed

Ticker

On Fri, 2021-11-05 at 09:02 +0000, svn commit wrote:

...

Version mkgmap-r4811 was committed by gerd on Fri, 05 Nov 2021

fix java.lang.AssertionError while building index from unicode tiles Changes extracted from mdrUnicode_v8.patch by Ticker Berkin

http://www.mkgmap.org.uk/websvn/revision.php?repname=mkgmap&rev=4811 _______________________________________________ mkgmap-svn mailing list To unsubscribe send an mail to mkgmap-svn-leave@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-svn

_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Hi Ticker,

there is no logical change in behaviour to Mdr5 with this patch. yes, I already wrote it's only refactoring. I still try to find some "real OSM" input where the changes to the other files show any difference in the mdr file.

I found this very old svn log entry by Steve: https://www.mkgmap.org.uk/websvn/revision.php?repname=mkgmap&rev=3093 which says that "All cases where sorted names are compared with equals() need to be changed to use Collator.compare() with the SrtCollator from Sort." So, I really wonder why he didn't change all sources accordingly. Maybe it was only about cities, maybe not. I see only one way to find out: Have an input that has shows differences in the index and find out which version gives better results in the search in MapSource or - if testing is possible - on a device. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Samstag, 6. November 2021 18:46 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4811: fix java.lang.AssertionError while building index from unicode tiles Hi Gerd I don't know if Mdr5 was correct before, but, for European names at least, I presume it has been used quite a lot and no complaints. If the default strength for a Collator is TERTIARY, there is no logical change in behaviour to Mdr5 with this patch. Ticker On Sat, 2021-11-06 at 14:33 +0000, Gerd Petermann wrote:

H Ticker,

but how do you know that the code in Mdr5 is right (with or without the patch)?

Gerd

________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Samstag, 6. November 2021 15:29 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4811: fix java.lang.AssertionError while building index from unicode tiles

Hi Gerd

I just hate the idea it is obviously wrong and trivially to make correct. It is unlikely it will ever fail. While it is in this inconsistent state it inhibits any addressing of the more complicated issue about how to tackle the case-differences in city names and what MdrCheck is reporting.

Ticker

On Sat, 2021-11-06 at 14:10 +0000, Gerd Petermann wrote:

...

Hi Ticker,

ok, so let's wait for that crash to get an example. I really have no idea how to force it :( According to MdrCheck the index is full of errors for the two tiles in China, but maybe it's MdrCheck which is wrong.

Gerd

________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Samstag, 6. November 2021 12:27 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4811: fix java.lang.AssertionError while building index from unicode tiles

Hi Gerd

The crash in Carlos's original problem is due to the inconsistency in the dedups between Mdr5/Mdr25.

This could be triggered with any --code-page where city names contain characters that exist in this character-set but are not given sort positions.

My mistake with mdrUnicode_v1/2.patch was trying to tackle the case problem at the same time. This is going to be much more difficult.

Ticker

On Fri, 2021-11-05 at 11:00 +0000, Gerd Petermann wrote:

...

Hi Ticker,

sorry for my reluctance. I simply have no test case that shows an error (search for xyz not working). If you have one please share it so that I can understand the importance of the patch.

I would also be happy if you could create a new branch to test further changes reg. --lower-case. According to MdrCheck we also produce wrong data for mdr 27 (cities are not de-duped). Found this also with Arndts *.img data but did not yet try to find out if MdrCheck is right.

Gerd

________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Freitag, 5. November 2021 11:34 An: mkgmap-dev@lists.mkgmap.org.uk; mkgmap-svn@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4811: fix java.lang.AssertionError while building index from unicode tiles

Hi Gerd

I really don't like the idea of the consistent dedup part of this patch not being applied (Mdr5 & Mdr25). The Mdr7 changes are a slight refactor and a useful comment, but has no logical effect. The Mdr29 changes are an assert to detect inconsistency in Mdr5/25 index pointer lengths before these could cause a crash + .equal()/collator.compare() change that could be removed

Ticker

On Fri, 2021-11-05 at 09:02 +0000, svn commit wrote:

...

Version mkgmap-r4811 was committed by gerd on Fri, 05 Nov 2021

fix java.lang.AssertionError while building index from unicode tiles Changes extracted from mdrUnicode_v8.patch by Ticker Berkin

http://www.mkgmap.org.uk/websvn/revision.php?repname=mkgmap&rev=4811 _______________________________________________ mkgmap-svn mailing list To unsubscribe send an mail to mkgmap-svn-leave@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-svn

_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Hi Gerd While looking at the mdr output from Mapinstall and the differences in the street repeat flags, mainly where there are shield/road-ref entries, I realised my change to Sort.java for undefined sortOrder went to far and it disrupts this ordering for Unicode. Missing or 0 sortOrder (anything before the first "<" in sort/cp*.txt) means the character is ignored by the key sort and collator.compare(), so TERTIARY and EQUAL can give different answers. Attached is mdrUnicode_v9a.patch that generates a fake sortOrder only for rows where no sortOrders are defined, instead of for anything in Unicode with no/0 sortOrder. With this patch it becomes possible for Unicode to trigger the original problem again. Attached is mdrUnicode_v9b.patch that has the fix for this from the previous patch. Ticker On Tue, 2021-11-09 at 18:13 +0000, Ticker Berkin wrote:

Hi Gerd

Sorry about the delay, I'm away at the moment without access to the sources and other tools, but:

The same pointer size (1-3 bytes) is used by Mdr5 and Mdr25. I can't remember if this is determined by the actual number of Mdr5 entries or bigger than Mdr5, then this setting needs to be investigated.

To see what happens when Mdr25 uses exact name dedupe and Mdr5 uses TERTIARY collator, the best way I can think of is to remove some characters from the sort/code-page such that Mdr5 will consider some cities to be identical in Mdr5, then see what effect this has on Find>Address.

With Mdr, I can well understand unwillingness to change anything when it seems to be working. My attempts to switch to SECONDARY collator, which I think is what has to happen eventually, had confusing results

Ticker

On 07/11/2021 09:16, Gerd Petermann wrote:

...

Hi Ticker,

...

  there is no logical change in behaviour to Mdr5 with this patch. yes, I already wrote it's only refactoring. I still try to find some "real OSM" input where the changes to the other files show any difference in the mdr file.

I found this very old svn log entry by Steve: https://www.mkgmap.org.uk/websvn/revision.php?repname=mkgmap&rev=3093 which says that "All cases where sorted names are compared with equals() need to be changed to use Collator.compare() with the SrtCollator from Sort."

So, I really wonder why he didn't change all  sources accordingly. Maybe it was only about cities, maybe not. I see only one way to find out: Have an input that has shows differences in the index and find out which version gives better results in the search in MapSource or - if testing is possible - on a device.

Gerd

________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Samstag, 6. November 2021 18:46 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4811: fix java.lang.AssertionError while building index from unicode tiles

Hi Gerd

I don't know if Mdr5 was correct before, but, for European names at least, I presume it has been used quite a lot and no complaints.

If the default strength for a Collator is TERTIARY, there is no logical change in behaviour to Mdr5 with this patch.

Ticker

On Sat, 2021-11-06 at 14:33 +0000, Gerd Petermann wrote:

...

H Ticker,

but how do you know that the code in Mdr5 is right (with or without the patch)?

Gerd

________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Samstag, 6. November 2021 15:29 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4811: fix java.lang.AssertionError while building index from unicode tiles

Hi Gerd

I just hate the idea it is obviously wrong and trivially to make correct. It is unlikely it will ever fail. While it is in this inconsistent state it inhibits any addressing of the more complicated issue about how to tackle the case-differences in city names and what MdrCheck is reporting.

Ticker

On Sat, 2021-11-06 at 14:10 +0000, Gerd Petermann wrote:

...

Hi Ticker,

ok, so let's wait for that crash to get an example. I really have no idea how to force it :( According to MdrCheck the index is full of errors for the two tiles in China, but maybe it's MdrCheck which is wrong.

Gerd

________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Samstag, 6. November 2021 12:27 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4811: fix java.lang.AssertionError while building index from unicode tiles

Hi Gerd

The crash in Carlos's original problem is due to the inconsistency in the dedups between Mdr5/Mdr25.

This could be triggered with any --code-page where city names contain characters that exist in this character-set but are not given sort positions.

My mistake with mdrUnicode_v1/2.patch was trying to tackle the case problem at the same time. This is going to be much more difficult.

Ticker

On Fri, 2021-11-05 at 11:00 +0000, Gerd Petermann wrote:

...

Hi Ticker,

sorry for my reluctance. I simply have no test case that shows an error (search for xyz not working). If you have one please share it so that I can understand the importance of the patch.

I would also be happy if you could create a new branch to test further changes reg. --lower-case. According to MdrCheck we also produce wrong data for mdr 27 (cities are not de-duped). Found this also with Arndts *.img data but did not yet try to find out if MdrCheck is right.

Gerd

________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Freitag, 5. November 2021 11:34 An: mkgmap-dev@lists.mkgmap.org.uk; mkgmap-svn@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4811: fix java.lang.AssertionError while building index from unicode tiles

Hi Gerd

I really don't like the idea of the consistent dedup part of this patch not being applied (Mdr5 & Mdr25). The Mdr7 changes are a slight refactor and a useful comment, but has no logical effect. The Mdr29 changes are an assert to detect inconsistency in Mdr5/25 index pointer lengths before these could cause a crash + .equal()/collator.compare() change that could be removed

Ticker

On Fri, 2021-11-05 at 09:02 +0000, svn commit wrote:

...

Version mkgmap-r4811 was committed by gerd on Fri, 05 Nov 2021

fix java.lang.AssertionError while building index from unicode tiles Changes extracted from mdrUnicode_v8.patch by Ticker Berkin

http://www.mkgmap.org.uk/websvn/revision.php?repname=mkgmap&rev=4811 _______________________________________________ mkgmap-svn mailing list To unsubscribe send an mail to mkgmap-svn-leave@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-svn

_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Hi Gerd Any map where there is a highway with a ref=* and it exists as more than one of {motorway/trunk}, {primary}, {secondary/tertiary}. The default style will apply different highway-symbols. Try with --code-page=1252 and code-page=65001 and see the differences in MdrCheck sections: mdr7 check: (r) vs (Rr) mdr20 check: ditto mdr11 check: 0x6616/Land-Summit points, when unnamed but with a height, change position in the list. mdr19 check: differences due to above. With mdrUnicode_v9a.patch, the Unicode behaviour for typical European maps should be the same as before r4809. Ticker On Thu, 2021-11-18 at 09:21 +0000, Gerd Petermann wrote:

Hi Ticker,

please share your testdata so that I'm able to reproduce the differences.

Gerd

________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Donnerstag, 18. November 2021 10:06 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4811: fix java.lang.AssertionError while building index from unicode tiles

Hi Gerd

While looking at the mdr output from Mapinstall and the differences in the street repeat flags, mainly where there are shield/road-ref entries, I realised my change to Sort.java for undefined sortOrder went to far and it disrupts this ordering for Unicode.

Missing or 0 sortOrder (anything before the first "<" in sort/cp*.txt) means the character is ignored by the key sort and collator.compare(), so TERTIARY and EQUAL can give different answers.

Attached is mdrUnicode_v9a.patch that generates a fake sortOrder only for rows where no sortOrders are defined, instead of for anything in Unicode with no/0 sortOrder.

With this patch it becomes possible for Unicode to trigger the original problem again. Attached is mdrUnicode_v9b.patch that has the fix for this from the previous patch.

Ticker

Hi Ticker, If I got that right the result of the MdrCheck depends on the mkgmap.jar. This is a bit tricky when comparing the results. I'll use the patched binary for now. Gerd ________________________________________ Von: Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Donnerstag, 18. November 2021 14:22 An: Development list for mkgmap Betreff: AW: [mkgmap-dev] [mkgmap-svn] Commit r4811: fix java.lang.AssertionError while building index from unicode tiles Hi Ticker, mdrUnicode_v9a.patch also changes the order in NET, esp. the "Roads as sorted in NET 3 ". The order in unicode with the patch is different to that with 1252 because highway symbols are now ignored. Is that intended? I don't remember where this order is important. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Donnerstag, 18. November 2021 13:16 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4811: fix java.lang.AssertionError while building index from unicode tiles Hi Gerd Any map where there is a highway with a ref=* and it exists as more than one of {motorway/trunk}, {primary}, {secondary/tertiary}. The default style will apply different highway-symbols. Try with --code-page=1252 and code-page=65001 and see the differences in MdrCheck sections: mdr7 check: (r) vs (Rr) mdr20 check: ditto mdr11 check: 0x6616/Land-Summit points, when unnamed but with a height, change position in the list. mdr19 check: differences due to above. With mdrUnicode_v9a.patch, the Unicode behaviour for typical European maps should be the same as before r4809. Ticker On Thu, 2021-11-18 at 09:21 +0000, Gerd Petermann wrote:

Hi Ticker,

please share your testdata so that I'm able to reproduce the differences.

Gerd

________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Donnerstag, 18. November 2021 10:06 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4811: fix java.lang.AssertionError while building index from unicode tiles

Hi Gerd

While looking at the mdr output from Mapinstall and the differences in the street repeat flags, mainly where there are shield/road-ref entries, I realised my change to Sort.java for undefined sortOrder went to far and it disrupts this ordering for Unicode.

Missing or 0 sortOrder (anything before the first "<" in sort/cp*.txt) means the character is ignored by the key sort and collator.compare(), so TERTIARY and EQUAL can give different answers.

Attached is mdrUnicode_v9a.patch that generates a fake sortOrder only for rows where no sortOrders are defined, instead of for anything in Unicode with no/0 sortOrder.

With this patch it becomes possible for Unicode to trigger the original problem again. Attached is mdrUnicode_v9b.patch that has the fix for this from the previous patch.

Ticker

_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Hi Ticker, anyhow, mdrUnicode_v9a.patch looks good to me so far. mdrUnicode_v9b.patch produced no differences in output in my tests, so I guess I have to produce an osm file with some special names to get close to the 256 names boundary and force the crash. I hoped you did that already. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Donnerstag, 18. November 2021 15:20 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4811: fix java.lang.AssertionError while building index from unicode tiles Hi Gerd I hadn't thought of that! Ticker On Thu, 2021-11-18 at 13:42 +0000, Gerd Petermann wrote:

Hi Ticker,

If I got that right the result of the MdrCheck depends on the mkgmap.jar. This is a bit tricky when comparing the results. I'll use the patched binary for now.

Gerd

_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Hi Ticker, please help me: I don't want to change the source. What unicode code character is ignored? How do I find one? My understanding is that I need an input file with 255 cities and then one additional city which has the same name as one in the map and an appended ignored character. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Donnerstag, 18. November 2021 15:37 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4811: fix java.lang.AssertionError while building index from unicode tiles Hi Gerd Sort and Collator with lots of ignored characters did it for me. The new assert in Mdr29 is there to detect problems before the getting to the stage where Mdr25 ptr needs more bytes than Mdr5 ptr. Ticker On Thu, 2021-11-18 at 14:28 +0000, Gerd Petermann wrote:

Hi Ticker,

anyhow, mdrUnicode_v9a.patch looks good to me so far. mdrUnicode_v9b.patch produced no differences in output in my tests, so I guess I have to produce an osm file with some special names to get close to the 256 names boundary and force the crash. I hoped you did that already.

Gerd

_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Hi Ticker, sorry, I must have done something wrong. NET3 order seems correct with the patch and wrong without. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Donnerstag, 18. November 2021 14:22 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4811: fix java.lang.AssertionError while building index from unicode tiles Hi Ticker, mdrUnicode_v9a.patch also changes the order in NET, esp. the "Roads as sorted in NET 3 ". The order in unicode with the patch is different to that with 1252 because highway symbols are now ignored. Is that intended? I don't remember where this order is important. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Donnerstag, 18. November 2021 13:16 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4811: fix java.lang.AssertionError while building index from unicode tiles Hi Gerd Any map where there is a highway with a ref=* and it exists as more than one of {motorway/trunk}, {primary}, {secondary/tertiary}. The default style will apply different highway-symbols. Try with --code-page=1252 and code-page=65001 and see the differences in MdrCheck sections: mdr7 check: (r) vs (Rr) mdr20 check: ditto mdr11 check: 0x6616/Land-Summit points, when unnamed but with a height, change position in the list. mdr19 check: differences due to above. With mdrUnicode_v9a.patch, the Unicode behaviour for typical European maps should be the same as before r4809. Ticker On Thu, 2021-11-18 at 09:21 +0000, Gerd Petermann wrote:

Hi Ticker,

please share your testdata so that I'm able to reproduce the differences.

Gerd

________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Donnerstag, 18. November 2021 10:06 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4811: fix java.lang.AssertionError while building index from unicode tiles

Hi Gerd

While looking at the mdr output from Mapinstall and the differences in the street repeat flags, mainly where there are shield/road-ref entries, I realised my change to Sort.java for undefined sortOrder went to far and it disrupts this ordering for Unicode.

Missing or 0 sortOrder (anything before the first "<" in sort/cp*.txt) means the character is ignored by the key sort and collator.compare(), so TERTIARY and EQUAL can give different answers.

Attached is mdrUnicode_v9a.patch that generates a fake sortOrder only for rows where no sortOrders are defined, instead of for anything in Unicode with no/0 sortOrder.

With this patch it becomes possible for Unicode to trigger the original problem again. Attached is mdrUnicode_v9b.patch that has the fix for this from the previous patch.

Ticker

_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev