Commit r4810: revert changes from r4809 for now, they caused more trouble
Version mkgmap-r4810 was committed by gerd on Mon, 25 Oct 2021 revert changes from r4809 for now, they caused more trouble http://www.mkgmap.org.uk/websvn/revision.php?repname=mkgmap&rev=4810
Hi Gerd Attached is version 3 of the patch. The significant problem was the logic (in Mdr5) where change in sortKey only was used to make unique lists, implying TERTIARY differences. The related structures had been processed with a Collator with SECONDARY strength. I've added an option to set the strength in the keys. The other changes are in Mdr25.sortCities: If the same city name with the same region name was in 2 countries, it didn't spot the new country - I realise this is very unlikely. Be consistent with collator strength when spotting changes in country. Again, unlikely that the same country occurs with different letter- case. Ticker On Mon, 2021-10-25 at 08:27 +0100, svn commit wrote:
Version mkgmap-r4810 was committed by gerd on Mon, 25 Oct 2021
revert changes from r4809 for now, they caused more trouble
http://www.mkgmap.org.uk/websvn/revision.php?repname=mkgmap&rev=4810 _______________________________________________ mkgmap-svn mailing list To unsubscribe send an mail to mkgmap-svn-leave@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-svn
Hi Ticker, if I got that right the method Sort.setKeyStrength() changes the Sort instance that is also used in other classes? This looks confusing if not dangerous. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Mittwoch, 27. Oktober 2021 13:41 An: mkgmap development Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4810: revert changes from r4809 for now, they caused more trouble Hi Gerd Attached is version 3 of the patch. The significant problem was the logic (in Mdr5) where change in sortKey only was used to make unique lists, implying TERTIARY differences. The related structures had been processed with a Collator with SECONDARY strength. I've added an option to set the strength in the keys. The other changes are in Mdr25.sortCities: If the same city name with the same region name was in 2 countries, it didn't spot the new country - I realise this is very unlikely. Be consistent with collator strength when spotting changes in country. Again, unlikely that the same country occurs with different letter- case. Ticker On Mon, 2021-10-25 at 08:27 +0100, svn commit wrote:
Version mkgmap-r4810 was committed by gerd on Mon, 25 Oct 2021
revert changes from r4809 for now, they caused more trouble
http://www.mkgmap.org.uk/websvn/revision.php?repname=mkgmap&rev=4810 _______________________________________________ mkgmap-svn mailing list To unsubscribe send an mail to mkgmap-svn-leave@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-svn
Hi Gerd Yes, you're right. Two solutions: Because there is no intermingling in the use of the Sort object in the steps of getting it, generating the keys and doing the sort, each use that is going to use createSortKey must set the keyStrength beforehand. There are, I think, 6 other places where this should be done - and they probably will all use SECONDARY. Remove this usage and change Mdr5 to be more convention and use lastCity/collator.compare(city.xxx(), lastCity.xxx()). I had originally changed it to be like this, but it adds extra cost, whereas changing the key reduces cost. Ticker On Wed, 2021-10-27 at 11:57 +0000, Gerd Petermann wrote:
Hi Ticker,
if I got that right the method Sort.setKeyStrength() changes the Sort instance that is also used in other classes? This looks confusing if not dangerous.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Mittwoch, 27. Oktober 2021 13:41 An: mkgmap development Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4810: revert changes from r4809 for now, they caused more trouble
Hi Gerd
Attached is version 3 of the patch.
The significant problem was the logic (in Mdr5) where change in sortKey only was used to make unique lists, implying TERTIARY differences. The related structures had been processed with a Collator with SECONDARY strength. I've added an option to set the strength in the keys.
The other changes are in Mdr25.sortCities:
If the same city name with the same region name was in 2 countries, it didn't spot the new country - I realise this is very unlikely.
Be consistent with collator strength when spotting changes in country. Again, unlikely that the same country occurs with different letter- case.
Ticker
On Mon, 2021-10-25 at 08:27 +0100, svn commit wrote:
Version mkgmap-r4810 was committed by gerd on Mon, 25 Oct 2021
revert changes from r4809 for now, they caused more trouble
http://www.mkgmap.org.uk/websvn/revision.php?repname=mkgmap&rev=4810 _______________________________________________ mkgmap-svn mailing list To unsubscribe send an mail to mkgmap-svn-leave@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-svn
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Ticker, in what situation do we still need the comparison with TERTIARY strength? Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Mittwoch, 27. Oktober 2021 14:35 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4810: revert changes from r4809 for now, they caused more trouble Hi Gerd Yes, you're right. Two solutions: Because there is no intermingling in the use of the Sort object in the steps of getting it, generating the keys and doing the sort, each use that is going to use createSortKey must set the keyStrength beforehand. There are, I think, 6 other places where this should be done - and they probably will all use SECONDARY. Remove this usage and change Mdr5 to be more convention and use lastCity/collator.compare(city.xxx(), lastCity.xxx()). I had originally changed it to be like this, but it adds extra cost, whereas changing the key reduces cost. Ticker On Wed, 2021-10-27 at 11:57 +0000, Gerd Petermann wrote:
Hi Ticker,
if I got that right the method Sort.setKeyStrength() changes the Sort instance that is also used in other classes? This looks confusing if not dangerous.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Mittwoch, 27. Oktober 2021 13:41 An: mkgmap development Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4810: revert changes from r4809 for now, they caused more trouble
Hi Gerd
Attached is version 3 of the patch.
The significant problem was the logic (in Mdr5) where change in sortKey only was used to make unique lists, implying TERTIARY differences. The related structures had been processed with a Collator with SECONDARY strength. I've added an option to set the strength in the keys.
The other changes are in Mdr25.sortCities:
If the same city name with the same region name was in 2 countries, it didn't spot the new country - I realise this is very unlikely.
Be consistent with collator strength when spotting changes in country. Again, unlikely that the same country occurs with different letter- case.
Ticker
On Mon, 2021-10-25 at 08:27 +0100, svn commit wrote:
Version mkgmap-r4810 was committed by gerd on Mon, 25 Oct 2021
revert changes from r4809 for now, they caused more trouble
http://www.mkgmap.org.uk/websvn/revision.php?repname=mkgmap&rev=4810 _______________________________________________ mkgmap-svn mailing list To unsubscribe send an mail to mkgmap-svn-leave@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-svn
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Gerd I suspect none, but I'll have a search and see if there are any places that don't set the collator to SECONDARY and see if there is a reason. Ticker On Wed, 2021-10-27 at 12:40 +0000, Gerd Petermann wrote:
Hi Ticker,
in what situation do we still need the comparison with TERTIARY strength?
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Mittwoch, 27. Oktober 2021 14:35 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4810: revert changes from r4809 for now, they caused more trouble
Hi Gerd
Yes, you're right.
Two solutions:
Because there is no intermingling in the use of the Sort object in the steps of getting it, generating the keys and doing the sort, each use that is going to use createSortKey must set the keyStrength beforehand. There are, I think, 6 other places where this should be done - and they probably will all use SECONDARY.
Remove this usage and change Mdr5 to be more convention and use lastCity/collator.compare(city.xxx(), lastCity.xxx()). I had originally changed it to be like this, but it adds extra cost, whereas changing the key reduces cost.
Ticker
On Wed, 2021-10-27 at 11:57 +0000, Gerd Petermann wrote:
Hi Ticker,
if I got that right the method Sort.setKeyStrength() changes the Sort instance that is also used in other classes? This looks confusing if not dangerous.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Mittwoch, 27. Oktober 2021 13:41 An: mkgmap development Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4810: revert changes from r4809 for now, they caused more trouble
Hi Gerd
Attached is version 3 of the patch.
The significant problem was the logic (in Mdr5) where change in sortKey only was used to make unique lists, implying TERTIARY differences. The related structures had been processed with a Collator with SECONDARY strength. I've added an option to set the strength in the keys.
The other changes are in Mdr25.sortCities:
If the same city name with the same region name was in 2 countries, it didn't spot the new country - I realise this is very unlikely.
Be consistent with collator strength when spotting changes in country. Again, unlikely that the same country occurs with different letter- case.
Ticker
On Mon, 2021-10-25 at 08:27 +0100, svn commit wrote:
Version mkgmap-r4810 was committed by gerd on Mon, 25 Oct 2021
revert changes from r4809 for now, they caused more trouble
http://www.mkgmap.org.uk/websvn/revision.php?repname=mkgmap&rev=4810 _______________________________________________ mkgmap-svn mailing list To unsubscribe send an mail to mkgmap-svn-leave@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-svn
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Gerd Where a sorted list is required rather than de-duplication then the strength should be left as TERTIARY, eg imgfmt/app/lbl/PlacesFile.java I've also spotted a few more Mdr sections that refer to regions and countries in a case sensitive way and these should be changed to use SECONDARY matching now that the inconsistencies have been removed from elsewhere. Ticker On Wed, 2021-10-27 at 14:18 +0100, Ticker Berkin wrote:
Hi Gerd
I suspect none, but I'll have a search and see if there are any places that don't set the collator to SECONDARY and see if there is a reason.
Ticker
On Wed, 2021-10-27 at 12:40 +0000, Gerd Petermann wrote:
Hi Ticker,
in what situation do we still need the comparison with TERTIARY strength?
Gerd
Hi Gerd After considering all places where the createSortKey strength must be set I've decided it isn't a good approach: 1/ It needs changes in NET and LBL as well as MDR. 2/ Because case-differences are not sorted, an apparently sorted and unique list of city/region/country could be out of order if there are case differences in the initial letters of duplicates. This also makes mdrCheck comparisons difficult This would have been mdrUnicode_v4.patch Attached patch doesn't change the Key/Sort behaviour but follows the sort with collator.compare where necessary instead of key.compareTo() or .equals() Ticker
Hi Ticker, OK, patch looks more consistent then v2 where I wondered about the mix of String.equals() and collator.compare(). I have to run a few tests to understand the meaning of the change. Any chance to create unit tests for this? I got the impression that the indexes work quite well in the past, at least for latin1. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Donnerstag, 28. Oktober 2021 13:20 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4810: revert changes from r4809 for now, they caused more trouble Hi Gerd After considering all places where the createSortKey strength must be set I've decided it isn't a good approach: 1/ It needs changes in NET and LBL as well as MDR. 2/ Because case-differences are not sorted, an apparently sorted and unique list of city/region/country could be out of order if there are case differences in the initial letters of duplicates. This also makes mdrCheck comparisons difficult This would have been mdrUnicode_v4.patch Attached patch doesn't change the Key/Sort behaviour but follows the sort with collator.compare where necessary instead of key.compareTo() or .equals() Ticker
Hi Ticker, patch v5 corrupts address search :( I tried the following command with the data from Arndt: (Arndt_NRWplus_mkgmapstyle.zip) java -Xmx5G -jar dist\mkgmap.jar --output-dir=e:/ld --max-jobs --latin1 --gmapi --index 60050169.img 60050170.img 60050188.img The three tiles contain data for cities "de Wijk" or "De Wijk" (two different cities and different spelling of the same city) I test with Basecamp address search. With the unpatched version I can find the street Cockserve in city De Wijk (OK), but not with the patched version. Basecamp shows the city name "De Wijk" when I seach for the road only, but obviously the index doesn't work. Same when I search for street Cockserve in city de Wijk In MapSource the results are similar, but it doesn't suggest "De Wijk" as city name, only "de Wijk". Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Freitag, 29. Oktober 2021 09:13 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4810: revert changes from r4809 for now, they caused more trouble Hi Ticker, OK, patch looks more consistent then v2 where I wondered about the mix of String.equals() and collator.compare(). I have to run a few tests to understand the meaning of the change. Any chance to create unit tests for this? I got the impression that the indexes work quite well in the past, at least for latin1. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Donnerstag, 28. Oktober 2021 13:20 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4810: revert changes from r4809 for now, they caused more trouble Hi Gerd After considering all places where the createSortKey strength must be set I've decided it isn't a good approach: 1/ It needs changes in NET and LBL as well as MDR. 2/ Because case-differences are not sorted, an apparently sorted and unique list of city/region/country could be out of order if there are case differences in the initial letters of duplicates. This also makes mdrCheck comparisons difficult This would have been mdrUnicode_v4.patch Attached patch doesn't change the Key/Sort behaviour but follows the sort with collator.compare where necessary instead of key.compareTo() or .equals() Ticker _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Gerd The behaviour for identically named but different cities shouldn't have changed - the main mdr structures deduped them based on name, region & country. I have changed the behaviour around what 'identically named' means - country and region logic had a mix of case-sensitivity and I've made them consistently insensitive. Cities were generally case-sensitive (I'd have to check carefully if this was totally consistent) and are now consistently insensitive. In your test you don't specify bounds, so region may not set. It is possible that, where there were distinct identical cities, streets in all but one were not findable and no one noticed. I need to study more of the mdr logic around how streets are tied to cities to understand this. Any solution is complicated because the same city could be split across tiles so the cityIndex is not adequate. I could just restore case-sensitivity to cities to give the old behaviour, but this may be avoiding the real issue. It seems wrong that device/basecamp/mapsource should present multiple "identical" cities and the correct one needs to be selected to find its streets. Ticker On Fri, 2021-10-29 at 08:35 +0000, Gerd Petermann wrote:
Hi Ticker,
patch v5 corrupts address search :( I tried the following command with the data from Arndt: (Arndt_NRWplus_mkgmapstyle.zip) java -Xmx5G -jar dist\mkgmap.jar --output-dir=e:/ld --max-jobs -- latin1 --gmapi --index 60050169.img 60050170.img 60050188.img
The three tiles contain data for cities "de Wijk" or "De Wijk" (two different cities and different spelling of the same city)
I test with Basecamp address search.
With the unpatched version I can find the street Cockserve in city De Wijk (OK), but not with the patched version. Basecamp shows the city name "De Wijk" when I seach for the road only, but obviously the index doesn't work. Same when I search for street Cockserve in city de Wijk
In MapSource the results are similar, but it doesn't suggest "De Wijk" as city name, only "de Wijk".
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Freitag, 29. Oktober 2021 09:13 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4810: revert changes from r4809 for now, they caused more trouble
Hi Ticker,
OK, patch looks more consistent then v2 where I wondered about the mix of String.equals() and collator.compare(). I have to run a few tests to understand the meaning of the change.
Any chance to create unit tests for this? I got the impression that the indexes work quite well in the past, at least for latin1.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Donnerstag, 28. Oktober 2021 13:20 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4810: revert changes from r4809 for now, they caused more trouble
Hi Gerd
After considering all places where the createSortKey strength must be set I've decided it isn't a good approach:
1/ It needs changes in NET and LBL as well as MDR. 2/ Because case-differences are not sorted, an apparently sorted and unique list of city/region/country could be out of order if there are case differences in the initial letters of duplicates. This also makes mdrCheck comparisons difficult
This would have been mdrUnicode_v4.patch
Attached patch doesn't change the Key/Sort behaviour but follows the sort with collator.compare where necessary instead of key.compareTo() or .equals()
Ticker
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Ticker, the bounds are not needed when existing *.img input is processed. I didn't try to compile the tiles from *.osm, my understanding of the patches was that they have no influence on the map tiles, only on the mdr file (as long as --latin1 is used). I can upload the three img files if you don't have them. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Freitag, 29. Oktober 2021 13:30 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4810: revert changes from r4809 for now, they caused more trouble Hi Gerd The behaviour for identically named but different cities shouldn't have changed - the main mdr structures deduped them based on name, region & country. I have changed the behaviour around what 'identically named' means - country and region logic had a mix of case-sensitivity and I've made them consistently insensitive. Cities were generally case-sensitive (I'd have to check carefully if this was totally consistent) and are now consistently insensitive. In your test you don't specify bounds, so region may not set. It is possible that, where there were distinct identical cities, streets in all but one were not findable and no one noticed. I need to study more of the mdr logic around how streets are tied to cities to understand this. Any solution is complicated because the same city could be split across tiles so the cityIndex is not adequate. I could just restore case-sensitivity to cities to give the old behaviour, but this may be avoiding the real issue. It seems wrong that device/basecamp/mapsource should present multiple "identical" cities and the correct one needs to be selected to find its streets. Ticker On Fri, 2021-10-29 at 08:35 +0000, Gerd Petermann wrote:
Hi Ticker,
patch v5 corrupts address search :( I tried the following command with the data from Arndt: (Arndt_NRWplus_mkgmapstyle.zip) java -Xmx5G -jar dist\mkgmap.jar --output-dir=e:/ld --max-jobs -- latin1 --gmapi --index 60050169.img 60050170.img 60050188.img
The three tiles contain data for cities "de Wijk" or "De Wijk" (two different cities and different spelling of the same city)
I test with Basecamp address search.
With the unpatched version I can find the street Cockserve in city De Wijk (OK), but not with the patched version. Basecamp shows the city name "De Wijk" when I seach for the road only, but obviously the index doesn't work. Same when I search for street Cockserve in city de Wijk
In MapSource the results are similar, but it doesn't suggest "De Wijk" as city name, only "de Wijk".
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Freitag, 29. Oktober 2021 09:13 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4810: revert changes from r4809 for now, they caused more trouble
Hi Ticker,
OK, patch looks more consistent then v2 where I wondered about the mix of String.equals() and collator.compare(). I have to run a few tests to understand the meaning of the change.
Any chance to create unit tests for this? I got the impression that the indexes work quite well in the past, at least for latin1.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Donnerstag, 28. Oktober 2021 13:20 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4810: revert changes from r4809 for now, they caused more trouble
Hi Gerd
After considering all places where the createSortKey strength must be set I've decided it isn't a good approach:
1/ It needs changes in NET and LBL as well as MDR. 2/ Because case-differences are not sorted, an apparently sorted and unique list of city/region/country could be out of order if there are case differences in the initial letters of duplicates. This also makes mdrCheck comparisons difficult
This would have been mdrUnicode_v4.patch
Attached patch doesn't change the Key/Sort behaviour but follows the sort with collator.compare where necessary instead of key.compareTo() or .equals()
Ticker
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Ticker, attached is a patch that simplifies calcMdr20SortPos(), no change in output intended. Does it help? Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Freitag, 29. Oktober 2021 14:10 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4810: revert changes from r4809 for now, they caused more trouble Hi Ticker, the bounds are not needed when existing *.img input is processed. I didn't try to compile the tiles from *.osm, my understanding of the patches was that they have no influence on the map tiles, only on the mdr file (as long as --latin1 is used). I can upload the three img files if you don't have them. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Freitag, 29. Oktober 2021 13:30 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4810: revert changes from r4809 for now, they caused more trouble Hi Gerd The behaviour for identically named but different cities shouldn't have changed - the main mdr structures deduped them based on name, region & country. I have changed the behaviour around what 'identically named' means - country and region logic had a mix of case-sensitivity and I've made them consistently insensitive. Cities were generally case-sensitive (I'd have to check carefully if this was totally consistent) and are now consistently insensitive. In your test you don't specify bounds, so region may not set. It is possible that, where there were distinct identical cities, streets in all but one were not findable and no one noticed. I need to study more of the mdr logic around how streets are tied to cities to understand this. Any solution is complicated because the same city could be split across tiles so the cityIndex is not adequate. I could just restore case-sensitivity to cities to give the old behaviour, but this may be avoiding the real issue. It seems wrong that device/basecamp/mapsource should present multiple "identical" cities and the correct one needs to be selected to find its streets. Ticker On Fri, 2021-10-29 at 08:35 +0000, Gerd Petermann wrote:
Hi Ticker,
patch v5 corrupts address search :( I tried the following command with the data from Arndt: (Arndt_NRWplus_mkgmapstyle.zip) java -Xmx5G -jar dist\mkgmap.jar --output-dir=e:/ld --max-jobs -- latin1 --gmapi --index 60050169.img 60050170.img 60050188.img
The three tiles contain data for cities "de Wijk" or "De Wijk" (two different cities and different spelling of the same city)
I test with Basecamp address search.
With the unpatched version I can find the street Cockserve in city De Wijk (OK), but not with the patched version. Basecamp shows the city name "De Wijk" when I seach for the road only, but obviously the index doesn't work. Same when I search for street Cockserve in city de Wijk
In MapSource the results are similar, but it doesn't suggest "De Wijk" as city name, only "de Wijk".
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Freitag, 29. Oktober 2021 09:13 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4810: revert changes from r4809 for now, they caused more trouble
Hi Ticker,
OK, patch looks more consistent then v2 where I wondered about the mix of String.equals() and collator.compare(). I have to run a few tests to understand the meaning of the change.
Any chance to create unit tests for this? I got the impression that the indexes work quite well in the past, at least for latin1.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Donnerstag, 28. Oktober 2021 13:20 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4810: revert changes from r4809 for now, they caused more trouble
Hi Gerd
After considering all places where the createSortKey strength must be set I've decided it isn't a good approach:
1/ It needs changes in NET and LBL as well as MDR. 2/ Because case-differences are not sorted, an apparently sorted and unique list of city/region/country could be out of order if there are case differences in the initial letters of duplicates. This also makes mdrCheck comparisons difficult
This would have been mdrUnicode_v4.patch
Attached patch doesn't change the Key/Sort behaviour but follows the sort with collator.compare where necessary instead of key.compareTo() or .equals()
Ticker
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Gerd I haven't worked out logic of your patch yet. Concerning your previous post on bounds - Yes. Sorry, I didn't notice that you were using existing tile .img files. Going back to the "find" behaviour for, eg, "de Wijk" vs. "De Wikj", I think it is important that, in the address search, the user doesn't need to know the case of the Country, Region and City to get the correct set of streets. I was hoping my changes of consistent use of collator strength SECONDARY would have this effect. I'll spend some time trying to work out why it doesn't, looking more carefully at what MDR25 is doing. If I can't get this to work then then I'll change it to be consistent TERTIARY in all the relevant places. Ticker On Sat, 2021-10-30 at 09:16 +0000, Gerd Petermann wrote:
Hi Ticker,
attached is a patch that simplifies calcMdr20SortPos(), no change in output intended. Does it help?
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Freitag, 29. Oktober 2021 14:10 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4810: revert changes from r4809 for now, they caused more trouble
Hi Ticker,
the bounds are not needed when existing *.img input is processed. I didn't try to compile the tiles from *.osm, my understanding of the patches was that they have no influence on the map tiles, only on the mdr file (as long as --latin1 is used).
I can upload the three img files if you don't have them.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Freitag, 29. Oktober 2021 13:30 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4810: revert changes from r4809 for now, they caused more trouble
Hi Gerd
The behaviour for identically named but different cities shouldn't have changed - the main mdr structures deduped them based on name, region & country.
I have changed the behaviour around what 'identically named' means - country and region logic had a mix of case-sensitivity and I've made them consistently insensitive. Cities were generally case-sensitive (I'd have to check carefully if this was totally consistent) and are now consistently insensitive.
In your test you don't specify bounds, so region may not set.
It is possible that, where there were distinct identical cities, streets in all but one were not findable and no one noticed. I need to study more of the mdr logic around how streets are tied to cities to understand this. Any solution is complicated because the same city could be split across tiles so the cityIndex is not adequate.
I could just restore case-sensitivity to cities to give the old behaviour, but this may be avoiding the real issue. It seems wrong that device/basecamp/mapsource should present multiple "identical" cities and the correct one needs to be selected to find its streets.
Ticker
On Fri, 2021-10-29 at 08:35 +0000, Gerd Petermann wrote:
Hi Ticker,
patch v5 corrupts address search :( I tried the following command with the data from Arndt: (Arndt_NRWplus_mkgmapstyle.zip) java -Xmx5G -jar dist\mkgmap.jar --output-dir=e:/ld --max-jobs -- latin1 --gmapi --index 60050169.img 60050170.img 60050188.img
The three tiles contain data for cities "de Wijk" or "De Wijk" (two different cities and different spelling of the same city)
I test with Basecamp address search.
With the unpatched version I can find the street Cockserve in city De Wijk (OK), but not with the patched version. Basecamp shows the city name "De Wijk" when I seach for the road only, but obviously the index doesn't work. Same when I search for street Cockserve in city de Wijk
In MapSource the results are similar, but it doesn't suggest "De Wijk" as city name, only "de Wijk".
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Freitag, 29. Oktober 2021 09:13 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4810: revert changes from r4809 for now, they caused more trouble
Hi Ticker,
OK, patch looks more consistent then v2 where I wondered about the mix of String.equals() and collator.compare(). I have to run a few tests to understand the meaning of the change.
Any chance to create unit tests for this? I got the impression that the indexes work quite well in the past, at least for latin1.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Donnerstag, 28. Oktober 2021 13:20 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4810: revert changes from r4809 for now, they caused more trouble
Hi Gerd
After considering all places where the createSortKey strength must be set I've decided it isn't a good approach:
1/ It needs changes in NET and LBL as well as MDR. 2/ Because case-differences are not sorted, an apparently sorted and unique list of city/region/country could be out of order if there are case differences in the initial letters of duplicates. This also makes mdrCheck comparisons difficult
This would have been mdrUnicode_v4.patch
Attached patch doesn't change the Key/Sort behaviour but follows the sort with collator.compare where necessary instead of key.compareTo() or .equals()
Ticker
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Gerd Looking at what Mdr5::calcMdr20SortPos() is doing, can't it be simplified away to just an extra call in genCitiesAndMdr20s(): ... around line 106 if (!c.isSameByName(collator, lastCity)) mdr20count++; c.setMdr20Index(mdr20count); add this: c.setMdr20SortPos(mdr20count); Ticker On Sat, 2021-10-30 at 09:16 +0000, Gerd Petermann wrote:
Hi Ticker,
attached is a patch that simplifies calcMdr20SortPos(), no change in output intended. Does it help?
Gerd
Hi Ticker, I think I tried that first and got an assertion or a different result. Maybe I didn't do it right? Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Montag, 1. November 2021 11:29 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4810: revert changes from r4809 for now, they caused more trouble Hi Gerd Looking at what Mdr5::calcMdr20SortPos() is doing, can't it be simplified away to just an extra call in genCitiesAndMdr20s(): ... around line 106 if (!c.isSameByName(collator, lastCity)) mdr20count++; c.setMdr20Index(mdr20count); add this: c.setMdr20SortPos(mdr20count); Ticker On Sat, 2021-10-30 at 09:16 +0000, Gerd Petermann wrote:
Hi Ticker,
attached is a patch that simplifies calcMdr20SortPos(), no change in output intended. Does it help?
Gerd
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi I've just tried this and leaving in the original calcMdr20SortPosn and checking its value and also doing your version and checking and they all come up with the same answer. On the full NRW map from Arndt (253 tiles) the extra sort adds 21 seconds to 4 mins 21 secs do generate gmapsupp from the tile .imgs Just tried without the original calcMdr20SortPosn and back to the original time - the 2 version with the sort take the same time Ticker On Mon, 2021-11-01 at 10:32 +0000, Gerd Petermann wrote:
Hi Ticker,
I think I tried that first and got an assertion or a different result. Maybe I didn't do it right?
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Montag, 1. November 2021 11:29 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4810: revert changes from r4809 for now, they caused more trouble
Hi Gerd
Looking at what Mdr5::calcMdr20SortPos() is doing, can't it be simplified away to just an extra call in genCitiesAndMdr20s():
... around line 106 if (!c.isSameByName(collator, lastCity)) mdr20count++; c.setMdr20Index(mdr20count); add this: c.setMdr20SortPos(mdr20count);
Ticker
On Sat, 2021-10-30 at 09:16 +0000, Gerd Petermann wrote:
Hi Ticker,
attached is a patch that simplifies calcMdr20SortPos(), no change in output intended. Does it help?
Gerd
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Ticker, here's the new version. No idea what I tested before. I don't see any significant performance improvement (calcMdr20SortPos() only takes a fraction of a second with nrw++ data) but I think the code is much clearer. Maybe you already hit memory limits on your 32-bit JRE? Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Montag, 1. November 2021 11:29 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4810: revert changes from r4809 for now, they caused more trouble Hi Gerd Looking at what Mdr5::calcMdr20SortPos() is doing, can't it be simplified away to just an extra call in genCitiesAndMdr20s(): ... around line 106 if (!c.isSameByName(collator, lastCity)) mdr20count++; c.setMdr20Index(mdr20count); add this: c.setMdr20SortPos(mdr20count); Ticker On Sat, 2021-10-30 at 09:16 +0000, Gerd Petermann wrote:
Hi Ticker,
attached is a patch that simplifies calcMdr20SortPos(), no change in output intended. Does it help?
Gerd
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Gerd I was thinking that that either mdr20SortPos or mdr20Index could go, probably also int[] mdr20Index/setMdr20set() - I'd have kept mdr20SortPos I've updated the same machine (4 core but only have 3G memory) to a 64- bit OS and Java 11, but generally the performance is worse. The 21 seconds for either version of calcMdrSortPos() is doing all 232 tiles of NRW+. Ticker On Tue, 2021-11-02 at 08:39 +0000, Gerd Petermann wrote:
Hi Ticker,
here's the new version. No idea what I tested before. I don't see any significant performance improvement (calcMdr20SortPos() only takes a fraction of a second with nrw++ data) but I think the code is much clearer. Maybe you already hit memory limits on your 32-bit JRE?
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Montag, 1. November 2021 11:29 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4810: revert changes from r4809 for now, they caused more trouble
Hi Gerd
Looking at what Mdr5::calcMdr20SortPos() is doing, can't it be simplified away to just an extra call in genCitiesAndMdr20s():
... around line 106 if (!c.isSameByName(collator, lastCity)) mdr20count++; c.setMdr20Index(mdr20count); add this: c.setMdr20SortPos(mdr20count);
Ticker
On Sat, 2021-10-30 at 09:16 +0000, Gerd Petermann wrote:
Hi Ticker,
attached is a patch that simplifies calcMdr20SortPos(), no change in output intended. Does it help?
Gerd
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Ticker, OK, let's concentrate on the unicode bug for now. I hope you still have an idea how to fix that? Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Dienstag, 2. November 2021 09:59 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4810: revert changes from r4809 for now, they caused more trouble Hi Gerd I was thinking that that either mdr20SortPos or mdr20Index could go, probably also int[] mdr20Index/setMdr20set() - I'd have kept mdr20SortPos I've updated the same machine (4 core but only have 3G memory) to a 64- bit OS and Java 11, but generally the performance is worse. The 21 seconds for either version of calcMdrSortPos() is doing all 232 tiles of NRW+. Ticker On Tue, 2021-11-02 at 08:39 +0000, Gerd Petermann wrote:
Hi Ticker,
here's the new version. No idea what I tested before. I don't see any significant performance improvement (calcMdr20SortPos() only takes a fraction of a second with nrw++ data) but I think the code is much clearer. Maybe you already hit memory limits on your 32-bit JRE?
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Montag, 1. November 2021 11:29 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4810: revert changes from r4809 for now, they caused more trouble
Hi Gerd
Looking at what Mdr5::calcMdr20SortPos() is doing, can't it be simplified away to just an extra call in genCitiesAndMdr20s():
... around line 106 if (!c.isSameByName(collator, lastCity)) mdr20count++; c.setMdr20Index(mdr20count); add this: c.setMdr20SortPos(mdr20count);
Ticker
On Sat, 2021-10-30 at 09:16 +0000, Gerd Petermann wrote:
Hi Ticker,
attached is a patch that simplifies calcMdr20SortPos(), no change in output intended. Does it help?
Gerd
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Gerd I agree, sort of. Use of some unicode chars exposed a problem of inconsistent sizes of mdr5/25 due to collator vs compare. Making these consistent exposed another problem with case-differences. My attempt at grouping case-variants of the same name and, if in same region/country, considering them the same and aggregating all the streets is not working consistently across different Garmin software. This was basically mdrUnicode_v5.patch. The MDR structural changes seemed justified but it makes MdrCheck report a lot of errors (it does exact name matching), works well on eTrex HCx, but behaves differently on and between BaseCamp and MapSource which appear to filter the index results with exact name matching via the LBL names (or maybe they don't use this part of the index.) So I'm removing all excess changes and just fix the unicode sort for unknown and make the name testing consistent but case sensitive where relevant. Ticker On Tue, 2021-11-02 at 09:48 +0000, Gerd Petermann wrote:
Hi Ticker,
OK, let's concentrate on the unicode bug for now. I hope you still have an idea how to fix that?
Gerd
Hi Gerd Here is a minimal patch to stop the 2 assertion crashes: 1/ unspecified sort for some blocks of Unicode chararacters. 2/ inconsistent MDR20 city positions that could result from city names with different case or use of non-unicode multi-byte codepage For 2/ with option --lower-case, the behaviour of how and which cities with capitalisation differences are shown in the find list, and which streets are attached to them is: 1/ device/BaseCamp/MapSource dependant. 2/ depends on how the streets are spread over tiles where there is possible name conflict. 3/ Affected by the presence of city POI on a tile. Ticker
Hi Ticker, thanks, will have a closer later. Just to make sure: My understanding is that the assertion with Arndts data was caused by your patch (r4809) and everything worked fine with r4808 / r4810. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Donnerstag, 4. November 2021 11:55 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4810: revert changes from r4809 for now, they caused more trouble Hi Gerd Here is a minimal patch to stop the 2 assertion crashes: 1/ unspecified sort for some blocks of Unicode chararacters. 2/ inconsistent MDR20 city positions that could result from city names with different case or use of non-unicode multi-byte codepage For 2/ with option --lower-case, the behaviour of how and which cities with capitalisation differences are shown in the find list, and which streets are attached to them is: 1/ device/BaseCamp/MapSource dependant. 2/ depends on how the streets are spread over tiles where there is possible name conflict. 3/ Affected by the presence of city POI on a tile. Ticker
Hi Gerd Yes, patch r4809 caused the crash with Arndt's data (--lower-case and differently cased city names). Ticker On Thu, 2021-11-04 at 11:11 +0000, Gerd Petermann wrote:
Hi Ticker,
thanks, will have a closer later. Just to make sure: My understanding is that the assertion with Arndts data was caused by your patch (r4809) and everything worked fine with r4808 / r4810.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Donnerstag, 4. November 2021 11:55 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4810: revert changes from r4809 for now, they caused more trouble
Hi Gerd
Here is a minimal patch to stop the 2 assertion crashes: 1/ unspecified sort for some blocks of Unicode chararacters. 2/ inconsistent MDR20 city positions that could result from city names with different case or use of non-unicode multi-byte codepage
For 2/ with option --lower-case, the behaviour of how and which cities with capitalisation differences are shown in the find list, and which streets are attached to them is: 1/ device/BaseCamp/MapSource dependant. 2/ depends on how the streets are spread over tiles where there is possible name conflict. 3/ Affected by the presence of city POI on a tile.
Ticker
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Ticker, why collator.setStrength(Collator.SECONDARY); in Mdr29? Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Donnerstag, 4. November 2021 12:40 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4810: revert changes from r4809 for now, they caused more trouble Hi Gerd Yes, patch r4809 caused the crash with Arndt's data (--lower-case and differently cased city names). Ticker On Thu, 2021-11-04 at 11:11 +0000, Gerd Petermann wrote:
Hi Ticker,
thanks, will have a closer later. Just to make sure: My understanding is that the assertion with Arndts data was caused by your patch (r4809) and everything worked fine with r4808 / r4810.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Donnerstag, 4. November 2021 11:55 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4810: revert changes from r4809 for now, they caused more trouble
Hi Gerd
Here is a minimal patch to stop the 2 assertion crashes: 1/ unspecified sort for some blocks of Unicode chararacters. 2/ inconsistent MDR20 city positions that could result from city names with different case or use of non-unicode multi-byte codepage
For 2/ with option --lower-case, the behaviour of how and which cities with capitalisation differences are shown in the find list, and which streets are attached to them is: 1/ device/BaseCamp/MapSource dependant. 2/ depends on how the streets are spread over tiles where there is possible name conflict. 3/ Affected by the presence of city POI on a tile.
Ticker
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Gerd This was a bit arbitrary and maybe I should have reverted it to .equals(). Generally Regions and Countries don't cause a problem because they (almost always) originate from --bounds and go through PlacesFile.createRegion() / createCountry() which stops any case difference within a tile. Ticker On Thu, 2021-11-04 at 13:15 +0000, Gerd Petermann wrote:
Hi Ticker,
why collator.setStrength(Collator.SECONDARY); in Mdr29?
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Donnerstag, 4. November 2021 12:40 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4810: revert changes from r4809 for now, they caused more trouble
Hi Gerd
Yes, patch r4809 caused the crash with Arndt's data (--lower-case and differently cased city names).
Ticker
On Thu, 2021-11-04 at 11:11 +0000, Gerd Petermann wrote:
Hi Ticker,
thanks, will have a closer later. Just to make sure: My understanding is that the assertion with Arndts data was caused by your patch (r4809) and everything worked fine with r4808 / r4810.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Donnerstag, 4. November 2021 11:55 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4810: revert changes from r4809 for now, they caused more trouble
Hi Gerd
Here is a minimal patch to stop the 2 assertion crashes: 1/ unspecified sort for some blocks of Unicode chararacters. 2/ inconsistent MDR20 city positions that could result from city names with different case or use of non-unicode multi-byte codepage
For 2/ with option --lower-case, the behaviour of how and which cities with capitalisation differences are shown in the find list, and which streets are attached to them is: 1/ device/BaseCamp/MapSource dependant. 2/ depends on how the streets are spread over tiles where there is possible name conflict. 3/ Affected by the presence of city POI on a tile.
Ticker
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Ticker, My understanding of mdrUnicode_v8.patch is again that only the changes in class Sort are relevant for the unicode error reported by Carlos. Why should we do more right now? In a second step we can try to simplify the code or fix other issues that came up last weeks like weird code in Mdr25 or the obsolete sort for mdr20. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Donnerstag, 4. November 2021 14:34 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4810: revert changes from r4809 for now, they caused more trouble Hi Gerd This was a bit arbitrary and maybe I should have reverted it to .equals(). Generally Regions and Countries don't cause a problem because they (almost always) originate from --bounds and go through PlacesFile.createRegion() / createCountry() which stops any case difference within a tile. Ticker On Thu, 2021-11-04 at 13:15 +0000, Gerd Petermann wrote:
Hi Ticker,
why collator.setStrength(Collator.SECONDARY); in Mdr29?
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Donnerstag, 4. November 2021 12:40 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4810: revert changes from r4809 for now, they caused more trouble
Hi Gerd
Yes, patch r4809 caused the crash with Arndt's data (--lower-case and differently cased city names).
Ticker
On Thu, 2021-11-04 at 11:11 +0000, Gerd Petermann wrote:
Hi Ticker,
thanks, will have a closer later. Just to make sure: My understanding is that the assertion with Arndts data was caused by your patch (r4809) and everything worked fine with r4808 / r4810.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Donnerstag, 4. November 2021 11:55 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4810: revert changes from r4809 for now, they caused more trouble
Hi Gerd
Here is a minimal patch to stop the 2 assertion crashes: 1/ unspecified sort for some blocks of Unicode chararacters. 2/ inconsistent MDR20 city positions that could result from city names with different case or use of non-unicode multi-byte codepage
For 2/ with option --lower-case, the behaviour of how and which cities with capitalisation differences are shown in the find list, and which streets are attached to them is: 1/ device/BaseCamp/MapSource dependant. 2/ depends on how the streets are spread over tiles where there is possible name conflict. 3/ Affected by the presence of city POI on a tile.
Ticker
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Gerd Although the Sort change fixes the Unicode problem reported by Carlos it demonstrated that the different methods of dedupe between Mdr5 and Mdr25 can lead to the same error with certain patterns of city names and so I really think this should be fixed at the same time. If not, it is a problem waiting to happen and the use of, say, code- page=936 as in Carlos's second problem, could trigger it. Mdr25 could be reverted to have the weird logic, but I think it was a bug that would never happen and the code would always confuse. The obsolete sort/Mdr20Posn/Index can be fixed as a trivial change soon. What to do with city case-variants and their street attachments is much more complicated and needs addressing in the tile-generation phase more than the MDR generation. This should definitely wait. Ticker On Thu, 2021-11-04 at 14:10 +0000, Gerd Petermann wrote:
Hi Ticker,
My understanding of mdrUnicode_v8.patch is again that only the changes in class Sort are relevant for the unicode error reported by Carlos. Why should we do more right now?
In a second step we can try to simplify the code or fix other issues that came up last weeks like weird code in Mdr25 or the obsolete sort for mdr20.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Donnerstag, 4. November 2021 14:34 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4810: revert changes from r4809 for now, they caused more trouble
Hi Gerd
This was a bit arbitrary and maybe I should have reverted it to .equals().
Generally Regions and Countries don't cause a problem because they (almost always) originate from --bounds and go through PlacesFile.createRegion() / createCountry() which stops any case difference within a tile.
Ticker
On Thu, 2021-11-04 at 13:15 +0000, Gerd Petermann wrote:
Hi Ticker,
why collator.setStrength(Collator.SECONDARY); in Mdr29?
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Donnerstag, 4. November 2021 12:40 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4810: revert changes from r4809 for now, they caused more trouble
Hi Gerd
Yes, patch r4809 caused the crash with Arndt's data (--lower-case and differently cased city names).
Ticker
On Thu, 2021-11-04 at 11:11 +0000, Gerd Petermann wrote:
Hi Ticker,
thanks, will have a closer later. Just to make sure: My understanding is that the assertion with Arndts data was caused by your patch (r4809) and everything worked fine with r4808 / r4810.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Donnerstag, 4. November 2021 11:55 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4810: revert changes from r4809 for now, they caused more trouble
Hi Gerd
Here is a minimal patch to stop the 2 assertion crashes: 1/ unspecified sort for some blocks of Unicode chararacters. 2/ inconsistent MDR20 city positions that could result from city names with different case or use of non-unicode multi-byte codepage
For 2/ with option --lower-case, the behaviour of how and which cities with capitalisation differences are shown in the find list, and which streets are attached to them is: 1/ device/BaseCamp/MapSource dependant. 2/ depends on how the streets are spread over tiles where there is possible name conflict. 3/ Affected by the presence of city POI on a tile.
Ticker
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Ticker, OK to fix de-dup where it's wrong. So far I found no problem with the different spelling of "De Wijk". I don't fully understand the discrepancies in the existing code, but the patched code also looks wrong to me with the mixed usage of strengths and String.equals() and collator.compare() or key.compareTo() If I got that right the changes in mdr5 and mdr7 are just refactoring and OK. Mdr25 could better use Mdr5Record.isSameByName() (although the method name is a bit confusing) Mdr29 should probably use strength TERTIARY for the collator instead of SECONDARY, or there should be a comment why a different strength is needed. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Donnerstag, 4. November 2021 17:17 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4810: revert changes from r4809 for now, they caused more trouble Hi Gerd Although the Sort change fixes the Unicode problem reported by Carlos it demonstrated that the different methods of dedupe between Mdr5 and Mdr25 can lead to the same error with certain patterns of city names and so I really think this should be fixed at the same time. If not, it is a problem waiting to happen and the use of, say, code- page=936 as in Carlos's second problem, could trigger it. Mdr25 could be reverted to have the weird logic, but I think it was a bug that would never happen and the code would always confuse. The obsolete sort/Mdr20Posn/Index can be fixed as a trivial change soon. What to do with city case-variants and their street attachments is much more complicated and needs addressing in the tile-generation phase more than the MDR generation. This should definitely wait. Ticker On Thu, 2021-11-04 at 14:10 +0000, Gerd Petermann wrote:
Hi Ticker,
My understanding of mdrUnicode_v8.patch is again that only the changes in class Sort are relevant for the unicode error reported by Carlos. Why should we do more right now?
In a second step we can try to simplify the code or fix other issues that came up last weeks like weird code in Mdr25 or the obsolete sort for mdr20.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Donnerstag, 4. November 2021 14:34 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4810: revert changes from r4809 for now, they caused more trouble
Hi Gerd
This was a bit arbitrary and maybe I should have reverted it to .equals().
Generally Regions and Countries don't cause a problem because they (almost always) originate from --bounds and go through PlacesFile.createRegion() / createCountry() which stops any case difference within a tile.
Ticker
On Thu, 2021-11-04 at 13:15 +0000, Gerd Petermann wrote:
Hi Ticker,
why collator.setStrength(Collator.SECONDARY); in Mdr29?
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Donnerstag, 4. November 2021 12:40 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4810: revert changes from r4809 for now, they caused more trouble
Hi Gerd
Yes, patch r4809 caused the crash with Arndt's data (--lower-case and differently cased city names).
Ticker
On Thu, 2021-11-04 at 11:11 +0000, Gerd Petermann wrote:
Hi Ticker,
thanks, will have a closer later. Just to make sure: My understanding is that the assertion with Arndts data was caused by your patch (r4809) and everything worked fine with r4808 / r4810.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Donnerstag, 4. November 2021 11:55 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4810: revert changes from r4809 for now, they caused more trouble
Hi Gerd
Here is a minimal patch to stop the 2 assertion crashes: 1/ unspecified sort for some blocks of Unicode chararacters. 2/ inconsistent MDR20 city positions that could result from city names with different case or use of non-unicode multi-byte codepage
For 2/ with option --lower-case, the behaviour of how and which cities with capitalisation differences are shown in the find list, and which streets are attached to them is: 1/ device/BaseCamp/MapSource dependant. 2/ depends on how the streets are spread over tiles where there is possible name conflict. 3/ Affected by the presence of city POI on a tile.
Ticker
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Gerd This was intended to be a minimal patch. Where there didn't seem to be a need to replace .equals() with collator.compare() I didn't. Some of these usages are setting a flag where a name repeats and MdrCheck assumes exact equals and device behaviour is more difficult to determine. Others are tests for Region or Country which aren't a problem. Ticker On Thu, 2021-11-04 at 16:57 +0000, Gerd Petermann wrote:
Hi Ticker,
OK to fix de-dup where it's wrong. So far I found no problem with the different spelling of "De Wijk". I don't fully understand the discrepancies in the existing code, but the patched code also looks wrong to me with the mixed usage of strengths and String.equals() and collator.compare() or key.compareTo()
If I got that right the changes in mdr5 and mdr7 are just refactoring and OK. Mdr25 could better use Mdr5Record.isSameByName() (although the method name is a bit confusing) Mdr29 should probably use strength TERTIARY for the collator instead of SECONDARY, or there should be a comment why a different strength is needed.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Donnerstag, 4. November 2021 17:17 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4810: revert changes from r4809 for now, they caused more trouble
Hi Gerd
Although the Sort change fixes the Unicode problem reported by Carlos it demonstrated that the different methods of dedupe between Mdr5 and Mdr25 can lead to the same error with certain patterns of city names and so I really think this should be fixed at the same time.
If not, it is a problem waiting to happen and the use of, say, code- page=936 as in Carlos's second problem, could trigger it.
Mdr25 could be reverted to have the weird logic, but I think it was a bug that would never happen and the code would always confuse.
The obsolete sort/Mdr20Posn/Index can be fixed as a trivial change soon.
What to do with city case-variants and their street attachments is much more complicated and needs addressing in the tile-generation phase more than the MDR generation. This should definitely wait.
Ticker
On Thu, 2021-11-04 at 14:10 +0000, Gerd Petermann wrote:
Hi Ticker,
My understanding of mdrUnicode_v8.patch is again that only the changes in class Sort are relevant for the unicode error reported by Carlos. Why should we do more right now?
In a second step we can try to simplify the code or fix other issues that came up last weeks like weird code in Mdr25 or the obsolete sort for mdr20.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Donnerstag, 4. November 2021 14:34 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4810: revert changes from r4809 for now, they caused more trouble
Hi Gerd
This was a bit arbitrary and maybe I should have reverted it to .equals().
Generally Regions and Countries don't cause a problem because they (almost always) originate from --bounds and go through PlacesFile.createRegion() / createCountry() which stops any case difference within a tile.
Ticker
On Thu, 2021-11-04 at 13:15 +0000, Gerd Petermann wrote:
Hi Ticker,
why collator.setStrength(Collator.SECONDARY); in Mdr29?
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Donnerstag, 4. November 2021 12:40 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4810: revert changes from r4809 for now, they caused more trouble
Hi Gerd
Yes, patch r4809 caused the crash with Arndt's data (--lower-case and differently cased city names).
Ticker
On Thu, 2021-11-04 at 11:11 +0000, Gerd Petermann wrote:
Hi Ticker,
thanks, will have a closer later. Just to make sure: My understanding is that the assertion with Arndts data was caused by your patch (r4809) and everything worked fine with r4808 / r4810.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Donnerstag, 4. November 2021 11:55 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4810: revert changes from r4809 for now, they caused more trouble
Hi Gerd
Here is a minimal patch to stop the 2 assertion crashes: 1/ unspecified sort for some blocks of Unicode chararacters. 2/ inconsistent MDR20 city positions that could result from city names with different case or use of non-unicode multi-byte codepage
For 2/ with option --lower-case, the behaviour of how and which cities with capitalisation differences are shown in the find list, and which streets are attached to them is: 1/ device/BaseCamp/MapSource dependant. 2/ depends on how the streets are spread over tiles where there is possible name conflict. 3/ Affected by the presence of city POI on a tile.
Ticker
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Gerd I realise now why I couldn't reproduce this behaviour with the case distinction - I haven't been using --lower-case. My device shows everything neatly in mixed case, even though all the data in .img is upper. This also affects the how mdr builds the indexes - all case variants look the same and are combined. Ticker On Fri, 2021-10-29 at 08:35 +0000, Gerd Petermann wrote:
Hi Ticker,
patch v5 corrupts address search :( I tried the following command with the data from Arndt: (Arndt_NRWplus_mkgmapstyle.zip) java -Xmx5G -jar dist\mkgmap.jar --output-dir=e:/ld --max-jobs -- latin1 --gmapi --index 60050169.img 60050170.img 60050188.img
The three tiles contain data for cities "de Wijk" or "De Wijk" (two different cities and different spelling of the same city)
I test with Basecamp address search.
With the unpatched version I can find the street Cockserve in city De Wijk (OK), but not with the patched version. Basecamp shows the city name "De Wijk" when I seach for the road only, but obviously the index doesn't work. Same when I search for street Cockserve in city de Wijk
In MapSource the results are similar, but it doesn't suggest "De Wijk" as city name, only "de Wijk".
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Freitag, 29. Oktober 2021 09:13 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4810: revert changes from r4809 for now, they caused more trouble
Hi Ticker,
OK, patch looks more consistent then v2 where I wondered about the mix of String.equals() and collator.compare(). I have to run a few tests to understand the meaning of the change.
Any chance to create unit tests for this? I got the impression that the indexes work quite well in the past, at least for latin1.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Donnerstag, 28. Oktober 2021 13:20 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4810: revert changes from r4809 for now, they caused more trouble
Hi Gerd
After considering all places where the createSortKey strength must be set I've decided it isn't a good approach:
1/ It needs changes in NET and LBL as well as MDR. 2/ Because case-differences are not sorted, an apparently sorted and unique list of city/region/country could be out of order if there are case differences in the initial letters of duplicates. This also makes mdrCheck comparisons difficult
This would have been mdrUnicode_v4.patch
Attached patch doesn't change the Key/Sort behaviour but follows the sort with collator.compare where necessary instead of key.compareTo() or .equals()
Ticker
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Ticker, ah, I already wondered why we have so different ideas ;) I also don't use --lower-case for my own maps, but it was my first idea when Arndt reported the problem with --latin1. Not sure but I think one reason for using --lower-case can be that two letter words are shown better, e.g. road name "Im Hagen" is displayed "IM Hagen" on my Oregon. Anyway, I think when the index is build we must be able to handle img files that contain mixed case characters. Don't be tempted to check if option --lower-case is used in that phase. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Montag, 1. November 2021 13:31 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4810: revert changes from r4809 for now, they caused more trouble Hi Gerd I realise now why I couldn't reproduce this behaviour with the case distinction - I haven't been using --lower-case. My device shows everything neatly in mixed case, even though all the data in .img is upper. This also affects the how mdr builds the indexes - all case variants look the same and are combined. Ticker On Fri, 2021-10-29 at 08:35 +0000, Gerd Petermann wrote:
Hi Ticker,
patch v5 corrupts address search :( I tried the following command with the data from Arndt: (Arndt_NRWplus_mkgmapstyle.zip) java -Xmx5G -jar dist\mkgmap.jar --output-dir=e:/ld --max-jobs -- latin1 --gmapi --index 60050169.img 60050170.img 60050188.img
The three tiles contain data for cities "de Wijk" or "De Wijk" (two different cities and different spelling of the same city)
I test with Basecamp address search.
With the unpatched version I can find the street Cockserve in city De Wijk (OK), but not with the patched version. Basecamp shows the city name "De Wijk" when I seach for the road only, but obviously the index doesn't work. Same when I search for street Cockserve in city de Wijk
In MapSource the results are similar, but it doesn't suggest "De Wijk" as city name, only "de Wijk".
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Freitag, 29. Oktober 2021 09:13 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4810: revert changes from r4809 for now, they caused more trouble
Hi Ticker,
OK, patch looks more consistent then v2 where I wondered about the mix of String.equals() and collator.compare(). I have to run a few tests to understand the meaning of the change.
Any chance to create unit tests for this? I got the impression that the indexes work quite well in the past, at least for latin1.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Donnerstag, 28. Oktober 2021 13:20 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4810: revert changes from r4809 for now, they caused more trouble
Hi Gerd
After considering all places where the createSortKey strength must be set I've decided it isn't a good approach:
1/ It needs changes in NET and LBL as well as MDR. 2/ Because case-differences are not sorted, an apparently sorted and unique list of city/region/country could be out of order if there are case differences in the initial letters of duplicates. This also makes mdrCheck comparisons difficult
This would have been mdrUnicode_v4.patch
Attached patch doesn't change the Key/Sort behaviour but follows the sort with collator.compare where necessary instead of key.compareTo() or .equals()
Ticker
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
participants (3)
-
Gerd Petermann -
svn commit -
Ticker Berkin