Small problem with global index
Hi Ticker, while experimenting with the index and unicode I found a rather simple case that doesn't work as expected. I've attached a small example file and a patch for the default style. A map produced with these options and the patched style --style-file=d:\mkgmap\resources\styles\default --lower-case --preserve-element-order --code-page=1252 --gmapi --index --housenumbers --bounds=bounds.zip bad-search.osm produces a map that shows an error in the POI search. I search for name=abc123x in region Bayern and get the correct result list of three different restaurants named abc123x. I search for name=abc123x in region reg1 and get the correct result list of three different restaurants named abc123x. Now I also fill the search field city with Burgpreppach. with name=abc123x in city Burgpreppach and region Bayern the result is still correct. with name=abc123x in city Burgpreppach and region reg1 the result is not correct, it also lists the objects in region Bayern So far I found no correction for this. It also doesn't seem to depend on the lower-case option. Any idea? Gerd
Hi Devs, did not find a solution for this yet but I noticed that MapSource creates a different mdr for this small map. If I got that right the order in Mdr11 also depends on the city/region and in Mdr19 the "repeated name" flag also considers this. I'll try to modify MdrCheck to find out if I am right here. My current thinking is that we have to identify (in this order) 1. unique countries -> each gets a number 2. unique regions -> name and number of country, each gets a number 3. unique cities -> name and number of region (we probably need an empy region for each country), each gets a number 4. POI -> name and city number No idea yet where Garmin considers upper/lower case differences or special "characters" like the highway shield codes. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <GPetermann_muenchen@hotmail.com> Gesendet: Samstag, 20. November 2021 14:59 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: [mkgmap-dev] Small problem with global index Hi Ticker, while experimenting with the index and unicode I found a rather simple case that doesn't work as expected. I've attached a small example file and a patch for the default style. A map produced with these options and the patched style --style-file=d:\mkgmap\resources\styles\default --lower-case --preserve-element-order --code-page=1252 --gmapi --index --housenumbers --bounds=bounds.zip bad-search.osm produces a map that shows an error in the POI search. I search for name=abc123x in region Bayern and get the correct result list of three different restaurants named abc123x. I search for name=abc123x in region reg1 and get the correct result list of three different restaurants named abc123x. Now I also fill the search field city with Burgpreppach. with name=abc123x in city Burgpreppach and region Bayern the result is still correct. with name=abc123x in city Burgpreppach and region reg1 the result is not correct, it also lists the objects in region Bayern So far I found no correction for this. It also doesn't seem to depend on the lower-case option. Any idea? Gerd
Hi Gerd I can't look at this in detail at the moment but: Lower case and indexing behaviour is affected by the tile processing stages deduping using upper case on country/region/city when loading MDR data (mainly from mkgmap:country/region/city). I seem to remember that explicit city POI behave differently and might be combined into the country/region/city/data. Some of these get an index which confuses matters more. Also isIn processing can introduce names forced to upper- case. I came to the conclusion that indexing would have to be done ignoring case, but my simple attempts to do this failed. Before trying again, we need to work on EQUAL/collator.compare in various places without the complications of --lower-case and --unicode Then make sure --unicode indexing works. Some Mdr sections are suppressed when Unicode. Only then tackle --lower-case and strange find behaviour. Ticker On Sat, 2021-11-20 at 16:49 +0000, Gerd Petermann wrote:
Hi Devs,
did not find a solution for this yet but I noticed that MapSource creates a different mdr for this small map. If I got that right the order in Mdr11 also depends on the city/region and in Mdr19 the "repeated name" flag also considers this.
I'll try to modify MdrCheck to find out if I am right here.
My current thinking is that we have to identify (in this order) 1. unique countries -> each gets a number 2. unique regions -> name and number of country, each gets a number 3. unique cities -> name and number of region (we probably need an empy region for each country), each gets a number 4. POI -> name and city number
No idea yet where Garmin considers upper/lower case differences or special "characters" like the highway shield codes.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <GPetermann_muenchen@hotmail.com> Gesendet: Samstag, 20. November 2021 14:59 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: [mkgmap-dev] Small problem with global index
Hi Ticker,
while experimenting with the index and unicode I found a rather simple case that doesn't work as expected. I've attached a small example file and a patch for the default style. A map produced with these options and the patched style --style-file=d:\mkgmap\resources\styles\default --lower-case -- preserve-element-order --code-page=1252 --gmapi --index -- housenumbers --bounds=bounds.zip bad-search.osm produces a map that shows an error in the POI search. I search for name=abc123x in region Bayern and get the correct result list of three different restaurants named abc123x. I search for name=abc123x in region reg1 and get the correct result list of three different restaurants named abc123x. Now I also fill the search field city with Burgpreppach. with name=abc123x in city Burgpreppach and region Bayern the result is still correct. with name=abc123x in city Burgpreppach and region reg1 the result is not correct, it also lists the objects in region Bayern
So far I found no correction for this. It also doesn't seem to depend on the lower-case option. Any idea?
Gerd _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Gerd Although non-sorting (eg shields) characters are a handy way of examining behaviour, I don't think they should influence any changes in the handling of Country/Region/City. We shouldn't attempt to make your example with the 3 variants of DREG region work. So, the problem becomes what to do with differently cased Country, Region & City when --lower-case. I found that trying to respect the case had problems both in implementation and use. I think we have to build a case-insensitive index if possible. Country, Region & City come into MDRFile with map relative index. For Country & Region, I think this is unique regardless of case. This isn't always true of City, so this should be tackled. POIs then need to be fixed to ref the chosen City. Then need to experiment with changing .equal to collator.compare and changing strengths from TERTIARY to SECONDARY. It is possible that MapInstall, having access to all the case variants via the LBL section, might rebuild a case-sensitive index. Was hoping to get mdrUnicode_v9p.patch out of the way before this. Ticker On Sat, 2021-11-20 at 17:27 +0000, Ticker Berkin wrote:
Hi Gerd
I can't look at this in detail at the moment but:
Lower case and indexing behaviour is affected by the tile processing stages deduping using upper case on country/region/city when loading MDR data (mainly from mkgmap:country/region/city). I seem to remember that explicit city POI behave differently and might be combined into the country/region/city/data. Some of these get an index which confuses matters more. Also isIn processing can introduce names forced to upper- case.
I came to the conclusion that indexing would have to be done ignoring case, but my simple attempts to do this failed. Before trying again, we need to work on EQUAL/collator.compare in various places without the complications of --lower-case and --unicode
Then make sure --unicode indexing works. Some Mdr sections are suppressed when Unicode.
Only then tackle --lower-case and strange find behaviour.
Ticker On Sat, 2021-11-20 at 16:49 +0000, Gerd Petermann wrote:
Hi Devs,
did not find a solution for this yet but I noticed that MapSource creates a different mdr for this small map. If I got that right the order in Mdr11 also depends on the city/region and in Mdr19 the "repeated name" flag also considers this.
I'll try to modify MdrCheck to find out if I am right here.
My current thinking is that we have to identify (in this order) 1. unique countries -> each gets a number 2. unique regions -> name and number of country, each gets a number 3. unique cities -> name and number of region (we probably need an empy region for each country), each gets a number 4. POI -> name and city number
No idea yet where Garmin considers upper/lower case differences or special "characters" like the highway shield codes.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <GPetermann_muenchen@hotmail.com> Gesendet: Samstag, 20. November 2021 14:59 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: [mkgmap-dev] Small problem with global index
Hi Ticker,
while experimenting with the index and unicode I found a rather simple case that doesn't work as expected. I've attached a small example file and a patch for the default style. A map produced with these options and the patched style --style-file=d:\mkgmap\resources\styles\default --lower-case -- preserve-element-order --code-page=1252 --gmapi --index -- housenumbers --bounds=bounds.zip bad-search.osm produces a map that shows an error in the POI search. I search for name=abc123x in region Bayern and get the correct result list of three different restaurants named abc123x. I search for name=abc123x in region reg1 and get the correct result list of three different restaurants named abc123x. Now I also fill the search field city with Burgpreppach. with name=abc123x in city Burgpreppach and region Bayern the result is still correct. with name=abc123x in city Burgpreppach and region reg1 the result is not correct, it also lists the objects in region Bayern
So far I found no correction for this. It also doesn't seem to depend on the lower-case option. Any idea?
Gerd _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Ticker, sorry, seems I attached the wrong patch for the default style. The problem is neither about upper/lower case nor about special characters like highway shield codes. It might be a problem in Garmin software, but it might as well show a problem with wrongly calculated repeat flags or maybe other flags. My thinking is that we should try to learn from the gmapsupp that is produced by MapSource, so my current workflow is this: - Compile the tile(s) and the gmapi and the gmapsupp. - Start MapSource, try if search works (it still doesn't for my example). - Generate a gmapsupp with Mapsource. - Extract the tiles from the gmapsupp (or copy those produced by mkgmap) - execute MdrDisplay and MdrCheck on the two different gmapsupp.img - compare all significant difference (different flags, existence of different sections etc) My latests results are that both MdrCheck and mkgmap should use TERTIARY strength were SECONDARY is used now. Unpatched MdrCheck reports several errors reg. repeat flags on Garmin maps which disapear when I use TERTIARY (the default). There also seems to be a problem with the positions of some bit flags. It seems that some flags should be shifted depending on the size of sections, but I don't know yet the details. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Sonntag, 21. November 2021 19:46 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] Small problem with global index Hi Gerd Although non-sorting (eg shields) characters are a handy way of examining behaviour, I don't think they should influence any changes in the handling of Country/Region/City. We shouldn't attempt to make your example with the 3 variants of DREG region work. So, the problem becomes what to do with differently cased Country, Region & City when --lower-case. I found that trying to respect the case had problems both in implementation and use. I think we have to build a case-insensitive index if possible. Country, Region & City come into MDRFile with map relative index. For Country & Region, I think this is unique regardless of case. This isn't always true of City, so this should be tackled. POIs then need to be fixed to ref the chosen City. Then need to experiment with changing .equal to collator.compare and changing strengths from TERTIARY to SECONDARY. It is possible that MapInstall, having access to all the case variants via the LBL section, might rebuild a case-sensitive index. Was hoping to get mdrUnicode_v9p.patch out of the way before this. Ticker On Sat, 2021-11-20 at 17:27 +0000, Ticker Berkin wrote:
Hi Gerd
I can't look at this in detail at the moment but:
Lower case and indexing behaviour is affected by the tile processing stages deduping using upper case on country/region/city when loading MDR data (mainly from mkgmap:country/region/city). I seem to remember that explicit city POI behave differently and might be combined into the country/region/city/data. Some of these get an index which confuses matters more. Also isIn processing can introduce names forced to upper- case.
I came to the conclusion that indexing would have to be done ignoring case, but my simple attempts to do this failed. Before trying again, we need to work on EQUAL/collator.compare in various places without the complications of --lower-case and --unicode
Then make sure --unicode indexing works. Some Mdr sections are suppressed when Unicode.
Only then tackle --lower-case and strange find behaviour.
Ticker
On Sat, 2021-11-20 at 16:49 +0000, Gerd Petermann wrote:
Hi Devs,
did not find a solution for this yet but I noticed that MapSource creates a different mdr for this small map. If I got that right the order in Mdr11 also depends on the city/region and in Mdr19 the "repeated name" flag also considers this.
I'll try to modify MdrCheck to find out if I am right here.
My current thinking is that we have to identify (in this order) 1. unique countries -> each gets a number 2. unique regions -> name and number of country, each gets a number 3. unique cities -> name and number of region (we probably need an empy region for each country), each gets a number 4. POI -> name and city number
No idea yet where Garmin considers upper/lower case differences or special "characters" like the highway shield codes.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <GPetermann_muenchen@hotmail.com> Gesendet: Samstag, 20. November 2021 14:59 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: [mkgmap-dev] Small problem with global index
Hi Ticker,
while experimenting with the index and unicode I found a rather simple case that doesn't work as expected. I've attached a small example file and a patch for the default style. A map produced with these options and the patched style --style-file=d:\mkgmap\resources\styles\default --lower-case -- preserve-element-order --code-page=1252 --gmapi --index -- housenumbers --bounds=bounds.zip bad-search.osm produces a map that shows an error in the POI search. I search for name=abc123x in region Bayern and get the correct result list of three different restaurants named abc123x. I search for name=abc123x in region reg1 and get the correct result list of three different restaurants named abc123x. Now I also fill the search field city with Burgpreppach. with name=abc123x in city Burgpreppach and region Bayern the result is still correct. with name=abc123x in city Burgpreppach and region reg1 the result is not correct, it also lists the objects in region Bayern
So far I found no correction for this. It also doesn't seem to depend on the lower-case option. Any idea?
Gerd _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Gerd The first stages of investigation should avoid other complexities that might be introduced by --lower-case (and --unicode). In this environment there should be no difference between SECONDARY and TERTIARY so the question is reduced to .equals() vs .compare() and other flag settings / Mdr sections. Your tests seem be examining the behaviour of "Find" > "POI" limited by region. My devices don't seem to have the ability do this and I don't see it in BaseCamp / MapSource either. "Find" > "Address" / "Intersection" are the only places where I can limit by Country and City. Your workflow makes sense when all upper case, but might not lead us to the best solution for mixed case. Ticker On Mon, 2021-11-22 at 05:24 +0000, Gerd Petermann wrote:
Hi Ticker,
sorry, seems I attached the wrong patch for the default style. The problem is neither about upper/lower case nor about special characters like highway shield codes. It might be a problem in Garmin software, but it might as well show a problem with wrongly calculated repeat flags or maybe other flags. My thinking is that we should try to learn from the gmapsupp that is produced by MapSource, so my current workflow is this: - Compile the tile(s) and the gmapi and the gmapsupp. - Start MapSource, try if search works (it still doesn't for my example). - Generate a gmapsupp with Mapsource. - Extract the tiles from the gmapsupp (or copy those produced by mkgmap) - execute MdrDisplay and MdrCheck on the two different gmapsupp.img - compare all significant difference (different flags, existence of different sections etc)
My latests results are that both MdrCheck and mkgmap should use TERTIARY strength were SECONDARY is used now. Unpatched MdrCheck reports several errors reg. repeat flags on Garmin maps which disapear when I use TERTIARY (the default). There also seems to be a problem with the positions of some bit flags. It seems that some flags should be shifted depending on the size of sections, but I don't know yet the details.
Gerd
Hi Ticker, I test with MapSource and the POI search also offers fields for region and country. On devices the search for region was probably dropped. Basecamp search is not very useful when you want to verify the index. The result is often a mixture of index results and nearby search.
Your workflow makes sense when all upper case, but might not lead us to the best solution for mixed case. Why not? I see also reasonable results for such a gmapsupp (but I didn't try it on the device so far). BTW: A good reason to use --lower-case in Germany are road names like Hauptstraße which is displayed as Hauptstrasse without that option.
Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Montag, 22. November 2021 11:43 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] Small problem with global index Hi Gerd The first stages of investigation should avoid other complexities that might be introduced by --lower-case (and --unicode). In this environment there should be no difference between SECONDARY and TERTIARY so the question is reduced to .equals() vs .compare() and other flag settings / Mdr sections. Your tests seem be examining the behaviour of "Find" > "POI" limited by region. My devices don't seem to have the ability do this and I don't see it in BaseCamp / MapSource either. "Find" > "Address" / "Intersection" are the only places where I can limit by Country and City. Your workflow makes sense when all upper case, but might not lead us to the best solution for mixed case. Ticker On Mon, 2021-11-22 at 05:24 +0000, Gerd Petermann wrote:
Hi Ticker,
sorry, seems I attached the wrong patch for the default style. The problem is neither about upper/lower case nor about special characters like highway shield codes. It might be a problem in Garmin software, but it might as well show a problem with wrongly calculated repeat flags or maybe other flags. My thinking is that we should try to learn from the gmapsupp that is produced by MapSource, so my current workflow is this: - Compile the tile(s) and the gmapi and the gmapsupp. - Start MapSource, try if search works (it still doesn't for my example). - Generate a gmapsupp with Mapsource. - Extract the tiles from the gmapsupp (or copy those produced by mkgmap) - execute MdrDisplay and MdrCheck on the two different gmapsupp.img - compare all significant difference (different flags, existence of different sections etc)
My latests results are that both MdrCheck and mkgmap should use TERTIARY strength were SECONDARY is used now. Unpatched MdrCheck reports several errors reg. repeat flags on Garmin maps which disapear when I use TERTIARY (the default). There also seems to be a problem with the positions of some bit flags. It seems that some flags should be shifted depending on the size of sections, but I don't know yet the details.
Gerd
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Gerd I'd forgotten about MapSource's better POI searching. If a mixed case map has differently cased variants of the same name, it shouldn't be up to the user to know which variant of Country, Region, City or Street to choose to find something, hence the interface should be case-insensitive. These case variants come from inconsistent OSM data and, at the moment, mkgmap logic. It might be possible for mkgmap to make a case-insensitive index by changing all TERTIARY to SECONDARY etc. However, once this is loaded into BaseCamp and MapInstall regenerates the indexes for a gmapsupp, it sees all the name variants and might include them in the new index structures. It lacks information to do this properly. The resultant index might have the problems described earlier about the user having to guess which Country, Region ... works. Ticker On Mon, 2021-11-22 at 10:54 +0000, Gerd Petermann wrote:
Hi Ticker,
I test with MapSource and the POI search also offers fields for region and country. On devices the search for region was probably dropped. Basecamp search is not very useful when you want to verify the index. The result is often a mixture of index results and nearby search.
Your workflow makes sense when all upper case, but might not lead us to the best solution for mixed case. Why not? I see also reasonable results for such a gmapsupp (but I didn't try it on the device so far). BTW: A good reason to use --lower-case in Germany are road names like Hauptstraße which is displayed as Hauptstrasse without that option.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Montag, 22. November 2021 11:43 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] Small problem with global index
Hi Gerd
The first stages of investigation should avoid other complexities that might be introduced by --lower-case (and --unicode). In this environment there should be no difference between SECONDARY and TERTIARY so the question is reduced to .equals() vs .compare() and other flag settings / Mdr sections.
Your tests seem be examining the behaviour of "Find" > "POI" limited by region. My devices don't seem to have the ability do this and I don't see it in BaseCamp / MapSource either. "Find" > "Address" / "Intersection" are the only places where I can limit by Country and City.
Your workflow makes sense when all upper case, but might not lead us to the best solution for mixed case.
Ticker
On Mon, 2021-11-22 at 05:24 +0000, Gerd Petermann wrote:
Hi Ticker,
sorry, seems I attached the wrong patch for the default style. The problem is neither about upper/lower case nor about special characters like highway shield codes. It might be a problem in Garmin software, but it might as well show a problem with wrongly calculated repeat flags or maybe other flags. My thinking is that we should try to learn from the gmapsupp that is produced by MapSource, so my current workflow is this: - Compile the tile(s) and the gmapi and the gmapsupp. - Start MapSource, try if search works (it still doesn't for my example). - Generate a gmapsupp with Mapsource. - Extract the tiles from the gmapsupp (or copy those produced by mkgmap) - execute MdrDisplay and MdrCheck on the two different gmapsupp.img - compare all significant difference (different flags, existence of different sections etc)
My latests results are that both MdrCheck and mkgmap should use TERTIARY strength were SECONDARY is used now. Unpatched MdrCheck reports several errors reg. repeat flags on Garmin maps which disapear when I use TERTIARY (the default). There also seems to be a problem with the positions of some bit flags. It seems that some flags should be shifted depending on the size of sections, but I don't know yet the details.
Gerd
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Ticker,
it shouldn't be up to the user to know which variant of Country, Region,City or Street to choose to find something, Yes, I agree. I never noticed such a problem in MapSource. Where do you see this problem?
Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Montag, 22. November 2021 12:36 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] Small problem with global index Hi Gerd I'd forgotten about MapSource's better POI searching. If a mixed case map has differently cased variants of the same name, it shouldn't be up to the user to know which variant of Country, Region, City or Street to choose to find something, hence the interface should be case-insensitive. These case variants come from inconsistent OSM data and, at the moment, mkgmap logic. It might be possible for mkgmap to make a case-insensitive index by changing all TERTIARY to SECONDARY etc. However, once this is loaded into BaseCamp and MapInstall regenerates the indexes for a gmapsupp, it sees all the name variants and might include them in the new index structures. It lacks information to do this properly. The resultant index might have the problems described earlier about the user having to guess which Country, Region ... works. Ticker On Mon, 2021-11-22 at 10:54 +0000, Gerd Petermann wrote:
Hi Ticker,
I test with MapSource and the POI search also offers fields for region and country. On devices the search for region was probably dropped. Basecamp search is not very useful when you want to verify the index. The result is often a mixture of index results and nearby search.
Your workflow makes sense when all upper case, but might not lead us to the best solution for mixed case. Why not? I see also reasonable results for such a gmapsupp (but I didn't try it on the device so far). BTW: A good reason to use --lower-case in Germany are road names like Hauptstraße which is displayed as Hauptstrasse without that option.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Montag, 22. November 2021 11:43 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] Small problem with global index
Hi Gerd
The first stages of investigation should avoid other complexities that might be introduced by --lower-case (and --unicode). In this environment there should be no difference between SECONDARY and TERTIARY so the question is reduced to .equals() vs .compare() and other flag settings / Mdr sections.
Your tests seem be examining the behaviour of "Find" > "POI" limited by region. My devices don't seem to have the ability do this and I don't see it in BaseCamp / MapSource either. "Find" > "Address" / "Intersection" are the only places where I can limit by Country and City.
Your workflow makes sense when all upper case, but might not lead us to the best solution for mixed case.
Ticker
On Mon, 2021-11-22 at 05:24 +0000, Gerd Petermann wrote:
Hi Ticker,
sorry, seems I attached the wrong patch for the default style. The problem is neither about upper/lower case nor about special characters like highway shield codes. It might be a problem in Garmin software, but it might as well show a problem with wrongly calculated repeat flags or maybe other flags. My thinking is that we should try to learn from the gmapsupp that is produced by MapSource, so my current workflow is this: - Compile the tile(s) and the gmapi and the gmapsupp. - Start MapSource, try if search works (it still doesn't for my example). - Generate a gmapsupp with Mapsource. - Extract the tiles from the gmapsupp (or copy those produced by mkgmap) - execute MdrDisplay and MdrCheck on the two different gmapsupp.img - compare all significant difference (different flags, existence of different sections etc)
My latests results are that both MdrCheck and mkgmap should use TERTIARY strength were SECONDARY is used now. Unpatched MdrCheck reports several errors reg. repeat flags on Garmin maps which disapear when I use TERTIARY (the default). There also seems to be a problem with the positions of some bit flags. It seems that some flags should be shifted depending on the size of sections, but I don't know yet the details.
Gerd
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Gerd I had renamed a city (Upton>wherWell) to duplicate another (Wherwell) in the same region, but with different capitalisation. The streets for "wherWell" spanned tiles, one tile contained the the real "Wherwell", the other tile contained the city POI (renamed in the same way). I can't remember which, but one or both of MapSource/BaseCamp showed both entries for the city during Find>Address, but one as "wherwell" instead of "wherWell" and the other as "Wherwell". They had different sets of streets, and these didn't even corresponding to the correct city. Ticker On Mon, 2021-11-22 at 12:56 +0000, Gerd Petermann wrote:
Hi Ticker,
it shouldn't be up to the user to know which variant of Country, Region,City or Street to choose to find something, Yes, I agree. I never noticed such a problem in MapSource. Where do you see this problem?
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Montag, 22. November 2021 12:36 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] Small problem with global index
Hi Gerd
I'd forgotten about MapSource's better POI searching.
If a mixed case map has differently cased variants of the same name, it shouldn't be up to the user to know which variant of Country, Region, City or Street to choose to find something, hence the interface should be case-insensitive. These case variants come from inconsistent OSM data and, at the moment, mkgmap logic.
It might be possible for mkgmap to make a case-insensitive index by changing all TERTIARY to SECONDARY etc.
However, once this is loaded into BaseCamp and MapInstall regenerates the indexes for a gmapsupp, it sees all the name variants and might include them in the new index structures. It lacks information to do this properly. The resultant index might have the problems described earlier about the user having to guess which Country, Region ... works.
Ticker
On Mon, 2021-11-22 at 10:54 +0000, Gerd Petermann wrote:
Hi Ticker,
I test with MapSource and the POI search also offers fields for region and country. On devices the search for region was probably dropped. Basecamp search is not very useful when you want to verify the index. The result is often a mixture of index results and nearby search.
Your workflow makes sense when all upper case, but might not lead us to the best solution for mixed case. Why not? I see also reasonable results for such a gmapsupp (but I didn't try it on the device so far). BTW: A good reason to use --lower-case in Germany are road names like Hauptstraße which is displayed as Hauptstrasse without that option.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Montag, 22. November 2021 11:43 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] Small problem with global index
Hi Gerd
The first stages of investigation should avoid other complexities that might be introduced by --lower-case (and --unicode). In this environment there should be no difference between SECONDARY and TERTIARY so the question is reduced to .equals() vs .compare() and other flag settings / Mdr sections.
Your tests seem be examining the behaviour of "Find" > "POI" limited by region. My devices don't seem to have the ability do this and I don't see it in BaseCamp / MapSource either. "Find" > "Address" / "Intersection" are the only places where I can limit by Country and City.
Your workflow makes sense when all upper case, but might not lead us to the best solution for mixed case.
Ticker
On Mon, 2021-11-22 at 05:24 +0000, Gerd Petermann wrote:
Hi Ticker,
sorry, seems I attached the wrong patch for the default style. The problem is neither about upper/lower case nor about special characters like highway shield codes. It might be a problem in Garmin software, but it might as well show a problem with wrongly calculated repeat flags or maybe other flags. My thinking is that we should try to learn from the gmapsupp that is produced by MapSource, so my current workflow is this: - Compile the tile(s) and the gmapi and the gmapsupp. - Start MapSource, try if search works (it still doesn't for my example). - Generate a gmapsupp with Mapsource. - Extract the tiles from the gmapsupp (or copy those produced by mkgmap) - execute MdrDisplay and MdrCheck on the two different gmapsupp.img - compare all significant difference (different flags, existence of different sections etc)
My latests results are that both MdrCheck and mkgmap should use TERTIARY strength were SECONDARY is used now. Unpatched MdrCheck reports several errors reg. repeat flags on Garmin maps which disapear when I use TERTIARY (the default). There also seems to be a problem with the positions of some bit flags. It seems that some flags should be shifted depending on the size of sections, but I don't know yet the details.
Gerd
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Ticker, the results probably depend on the way how you "renamed a city". You may only rename a single POI or also the data in the bounds.zip or maybe also addr:city=Upton etc. That's why I keep asking for demo data to reproduce your results. Testing this stuff is quite complex and error prone and thus we should try to have common input files to be able to verify our results. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Montag, 22. November 2021 16:00 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] Small problem with global index Hi Gerd I had renamed a city (Upton>wherWell) to duplicate another (Wherwell) in the same region, but with different capitalisation. The streets for "wherWell" spanned tiles, one tile contained the the real "Wherwell", the other tile contained the city POI (renamed in the same way). I can't remember which, but one or both of MapSource/BaseCamp showed both entries for the city during Find>Address, but one as "wherwell" instead of "wherWell" and the other as "Wherwell". They had different sets of streets, and these didn't even corresponding to the correct city. Ticker On Mon, 2021-11-22 at 12:56 +0000, Gerd Petermann wrote:
Hi Ticker,
it shouldn't be up to the user to know which variant of Country, Region,City or Street to choose to find something, Yes, I agree. I never noticed such a problem in MapSource. Where do you see this problem?
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Montag, 22. November 2021 12:36 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] Small problem with global index
Hi Gerd
I'd forgotten about MapSource's better POI searching.
If a mixed case map has differently cased variants of the same name, it shouldn't be up to the user to know which variant of Country, Region, City or Street to choose to find something, hence the interface should be case-insensitive. These case variants come from inconsistent OSM data and, at the moment, mkgmap logic.
It might be possible for mkgmap to make a case-insensitive index by changing all TERTIARY to SECONDARY etc.
However, once this is loaded into BaseCamp and MapInstall regenerates the indexes for a gmapsupp, it sees all the name variants and might include them in the new index structures. It lacks information to do this properly. The resultant index might have the problems described earlier about the user having to guess which Country, Region ... works.
Ticker
On Mon, 2021-11-22 at 10:54 +0000, Gerd Petermann wrote:
Hi Ticker,
I test with MapSource and the POI search also offers fields for region and country. On devices the search for region was probably dropped. Basecamp search is not very useful when you want to verify the index. The result is often a mixture of index results and nearby search.
Your workflow makes sense when all upper case, but might not lead us to the best solution for mixed case. Why not? I see also reasonable results for such a gmapsupp (but I didn't try it on the device so far). BTW: A good reason to use --lower-case in Germany are road names like Hauptstraße which is displayed as Hauptstrasse without that option.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Montag, 22. November 2021 11:43 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] Small problem with global index
Hi Gerd
The first stages of investigation should avoid other complexities that might be introduced by --lower-case (and --unicode). In this environment there should be no difference between SECONDARY and TERTIARY so the question is reduced to .equals() vs .compare() and other flag settings / Mdr sections.
Your tests seem be examining the behaviour of "Find" > "POI" limited by region. My devices don't seem to have the ability do this and I don't see it in BaseCamp / MapSource either. "Find" > "Address" / "Intersection" are the only places where I can limit by Country and City.
Your workflow makes sense when all upper case, but might not lead us to the best solution for mixed case.
Ticker
On Mon, 2021-11-22 at 05:24 +0000, Gerd Petermann wrote:
Hi Ticker,
sorry, seems I attached the wrong patch for the default style. The problem is neither about upper/lower case nor about special characters like highway shield codes. It might be a problem in Garmin software, but it might as well show a problem with wrongly calculated repeat flags or maybe other flags. My thinking is that we should try to learn from the gmapsupp that is produced by MapSource, so my current workflow is this: - Compile the tile(s) and the gmapi and the gmapsupp. - Start MapSource, try if search works (it still doesn't for my example). - Generate a gmapsupp with Mapsource. - Extract the tiles from the gmapsupp (or copy those produced by mkgmap) - execute MdrDisplay and MdrCheck on the two different gmapsupp.img - compare all significant difference (different flags, existence of different sections etc)
My latests results are that both MdrCheck and mkgmap should use TERTIARY strength were SECONDARY is used now. Unpatched MdrCheck reports several errors reg. repeat flags on Garmin maps which disapear when I use TERTIARY (the default). There also seems to be a problem with the positions of some bit flags. It seems that some flags should be shifted depending on the size of sections, but I don't know yet the details.
Gerd
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Gerd I'm not ignoring this, but trying to put the essence of my renames into the default style and testing with a r4810 and --lower-case I find MapSource isn't giving me a Find > Address option and BaseCamp doesn't find any streets and once said "Doesn't support Address Search" So I'll try and pin down what makes it fail totally. Ticker On Mon, 2021-11-22 at 15:27 +0000, Gerd Petermann wrote:
Hi Ticker,
the results probably depend on the way how you "renamed a city". You may only rename a single POI or also the data in the bounds.zip or maybe also addr:city=Upton etc. That's why I keep asking for demo data to reproduce your results. Testing this stuff is quite complex and error prone and thus we should try to have common input files to be able to verify our results.
Gerd
Hi Gerd Here are the bits I used to investigate case strangeness The 2 tiles can be extracted from Hampshire, England, Great-Britain etc with the attached split-list. I can upload them if you want The style is "default" with additions at end of points & lines to rename "Upton" to "wherWell" and a slight mod to inc/address - diff attached. Command is: $ java -ea -jar ../mkgmap-r4810/mkgmap.jar --add-pois-to-areas \ --bounds=../bounds.zip --code-page=1252 --gmapi --gmapsupp --index \ --location-autofill=is_in,nearest --lower-case \ --preserve-element-order --route --style-file=tstStyle -c template.args --route was needed for Find > Address to show in MapSource and to work in BaseCamp. Resultant behaviour from Find > Address: BaseCamp 4.7.4. City field allows "Wherwell" or "wherwell" (NB not "wherWell"), Streets in either city can be found regardless of which version is entered, but if the street is in the top tile the city is shown as "Wherwell" (even if in "wherwell"=="Upton"). City for Streets in the lower tile show as "wherwell". MapSource 6.16.3 The Street must be chosen first. Only "wherwell" is accepted in the City field. The results show "Wherwell"/"wherwell" according to the tile, as above. eTrex HCx The City field can be set to "Wherwell" or "wherWell". If "Wherwell" then can find streets in the top tile, if "wherWell" then the bottom tile. Sample Streets: Name In HCx find>city necessary Alma Lane Upton wherWell Baybridge Lane Upton Wherwell Beech Grove Wherwell Wherwell Belmore Upton Wherwell Chant Close Wherwell Wherwell Bigpath Lane Upton Either as crosses tile Church Street Both Either, finds expected one Ticker On Tue, 2021-11-23 at 11:03 +0000, Ticker Berkin wrote:
Hi Gerd
I'm not ignoring this, but trying to put the essence of my renames into the default style and testing with a r4810 and --lower-case I find MapSource isn't giving me a Find > Address option and BaseCamp doesn't find any streets and once said "Doesn't support Address Search"
So I'll try and pin down what makes it fail totally.
Ticker
On Mon, 2021-11-22 at 15:27 +0000, Gerd Petermann wrote:
Hi Ticker,
the results probably depend on the way how you "renamed a city". You may only rename a single POI or also the data in the bounds.zip or maybe also addr:city=Upton etc. That's why I keep asking for demo data to reproduce your results. Testing this stuff is quite complex and error prone and thus we should try to have common input files to be able to verify our results.
Gerd
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Ticker, I (my tools) don't know how to use default.diff. Please can you either create a normal svn diff or zip the complete style? Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Dienstag, 23. November 2021 14:37 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] Small problem with global index Hi Gerd Here are the bits I used to investigate case strangeness The 2 tiles can be extracted from Hampshire, England, Great-Britain etc with the attached split-list. I can upload them if you want The style is "default" with additions at end of points & lines to rename "Upton" to "wherWell" and a slight mod to inc/address - diff attached. Command is: $ java -ea -jar ../mkgmap-r4810/mkgmap.jar --add-pois-to-areas \ --bounds=../bounds.zip --code-page=1252 --gmapi --gmapsupp --index \ --location-autofill=is_in,nearest --lower-case \ --preserve-element-order --route --style-file=tstStyle -c template.args --route was needed for Find > Address to show in MapSource and to work in BaseCamp. Resultant behaviour from Find > Address: BaseCamp 4.7.4. City field allows "Wherwell" or "wherwell" (NB not "wherWell"), Streets in either city can be found regardless of which version is entered, but if the street is in the top tile the city is shown as "Wherwell" (even if in "wherwell"=="Upton"). City for Streets in the lower tile show as "wherwell". MapSource 6.16.3 The Street must be chosen first. Only "wherwell" is accepted in the City field. The results show "Wherwell"/"wherwell" according to the tile, as above. eTrex HCx The City field can be set to "Wherwell" or "wherWell". If "Wherwell" then can find streets in the top tile, if "wherWell" then the bottom tile. Sample Streets: Name In HCx find>city necessary Alma Lane Upton wherWell Baybridge Lane Upton Wherwell Beech Grove Wherwell Wherwell Belmore Upton Wherwell Chant Close Wherwell Wherwell Bigpath Lane Upton Either as crosses tile Church Street Both Either, finds expected one Ticker On Tue, 2021-11-23 at 11:03 +0000, Ticker Berkin wrote:
Hi Gerd
I'm not ignoring this, but trying to put the essence of my renames into the default style and testing with a r4810 and --lower-case I find MapSource isn't giving me a Find > Address option and BaseCamp doesn't find any streets and once said "Doesn't support Address Search"
So I'll try and pin down what makes it fail totally.
Ticker
On Mon, 2021-11-22 at 15:27 +0000, Gerd Petermann wrote:
Hi Ticker,
the results probably depend on the way how you "renamed a city". You may only rename a single POI or also the data in the bounds.zip or maybe also addr:city=Upton etc. That's why I keep asking for demo data to reproduce your results. Testing this stuff is quite complex and error prone and thus we should try to have common input files to be able to verify our results.
Gerd
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Ticker, in the meanwhile I tried with the default style. I don't see any road with city=Upton nor with city=Wherwell. Road "Fair Piece" is found in Test Valley. Maybe it's because I used keep-complete=false while splitting (I used europe as input). So, better upload your two tiles and I'll update my bounds.zip now... Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Mittwoch, 24. November 2021 16:32 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] Small problem with global index Hi Gerd Here it is Ticker On Wed, 2021-11-24 at 14:57 +0000, Gerd Petermann wrote:
Hi Ticker,
I (my tools) don't know how to use default.diff. Please can you either create a normal svn diff or zip the complete style?
Gerd
Hi Gerd The default inc/address rules are not very good for UK towns and smaller - need to start from mkgmap:admin_level10. Appropriate change is in tstStyle.zip Nothing significant to this has changed in bounds.zip - I'm using a version from April 2021 Ticker On Wed, 2021-11-24 at 15:40 +0000, Gerd Petermann wrote:
Hi Ticker,
in the meanwhile I tried with the default style. I don't see any road with city=Upton nor with city=Wherwell. Road "Fair Piece" is found in Test Valley. Maybe it's because I used keep-complete=false while splitting (I used europe as input).
So, better upload your two tiles and I'll update my bounds.zip now...
Gerd
Hi Ticker, OK, with the zipped style I could reproduce (with r4808) some results. I think MapSource works well, but with my Oregon I also have the problem that I have to decide whether I want to search in wherWell or in Wherwell, and the device will only find the roads in the corresponding city. That's not nice but at least it looks correct, means, it doesn't find Alma Lane in "Wherwell", only in "wherWell", and e.g. Bigpath Lane is found in both cities, but different and correct roads for each. When I change your style to replace Upton by Wherwell the device shows only one Wherwell as city after typing wh and three occurrences for Bigpath Lane. That's good and I guess you would want the same for the special case that cities are written with only TERTIARY differences? No idea if this is a good approach. Maybe it's better to let the user stumble over this so that someone fixes the wrong OSM data. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Mittwoch, 24. November 2021 16:57 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] Small problem with global index Hi Gerd The default inc/address rules are not very good for UK towns and smaller - need to start from mkgmap:admin_level10. Appropriate change is in tstStyle.zip Nothing significant to this has changed in bounds.zip - I'm using a version from April 2021 Ticker On Wed, 2021-11-24 at 15:40 +0000, Gerd Petermann wrote:
Hi Ticker,
in the meanwhile I tried with the default style. I don't see any road with city=Upton nor with city=Wherwell. Road "Fair Piece" is found in Test Valley. Maybe it's because I used keep-complete=false while splitting (I used europe as input).
So, better upload your two tiles and I'll update my bounds.zip now...
Gerd
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Gerd In this example, choosing the correct version of the spelling doesn't necessarily work! To find "Alma Lane", which is in Upton=wherWell, "wherWell" must be selected as the city. To find "Baybridge Lane", also in Upton=wherWell, "Wherwell" must be given. Getting capitalisation consistent in OSM would be a good thing. Generally, for cities, it is pretty good; in the original problem report (Arndt / NRW + surrounding) I found only 3 examples. However, for streets, I find a lots. This is why I'm keen on getting to a nicely working case-insensitive search and against moving existing logic from SECONDARY to TERTIARY/EQUAL is-in logic unconditionally upper-casing everything isn't ideal - I don't think it adds to the problem but its effect might be visible. The strange behaviour in this example is because, although combiners/MDRBuilder attempts to do case-insensitive dedup on COUNTRY, REGION and CITY, a CITY POI (only existing in 1 tile) evades this and so, in this tile, there might be 2 entries with different city.index. This needs to be fixed before trying to change from TERTIARY/EQUAL to SECONDARY. Concerning the case where there really are 2 Cities with the same name in the same region: There isn't any way of distinguishing which streets belong to which (in my example the correct city is in another tile), so how many streets with the same name to show? With the current mkgmap implementation of not joining streets with different attributes, there might be many in the same city. mkgmap appears to dedup them (give or take the r/rr flags which I haven't understood yet), which is reasonable if all in 1 city but not if >1. Ticker On Wed, 2021-11-24 at 16:39 +0000, Gerd Petermann wrote:
Hi Ticker,
OK, with the zipped style I could reproduce (with r4808) some results. I think MapSource works well, but with my Oregon I also have the problem that I have to decide whether I want to search in wherWell or in Wherwell, and the device will only find the roads in the corresponding city. That's not nice but at least it looks correct, means, it doesn't find Alma Lane in "Wherwell", only in "wherWell", and e.g. Bigpath Lane is found in both cities, but different and correct roads for each.
When I change your style to replace Upton by Wherwell the device shows only one Wherwell as city after typing wh and three occurrences for Bigpath Lane. That's good and I guess you would want the same for the special case that cities are written with only TERTIARY differences?
No idea if this is a good approach. Maybe it's better to let the user stumble over this so that someone fixes the wrong OSM data.
Gerd
Hi Ticker, OK, sorry, did not understand the table. I can confirm that the search for "Baybridge Lane" doesn't work as expected. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Mittwoch, 24. November 2021 18:58 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] Small problem with global index Hi Gerd In this example, choosing the correct version of the spelling doesn't necessarily work! To find "Alma Lane", which is in Upton=wherWell, "wherWell" must be selected as the city. To find "Baybridge Lane", also in Upton=wherWell, "Wherwell" must be given. Getting capitalisation consistent in OSM would be a good thing. Generally, for cities, it is pretty good; in the original problem report (Arndt / NRW + surrounding) I found only 3 examples. However, for streets, I find a lots. This is why I'm keen on getting to a nicely working case-insensitive search and against moving existing logic from SECONDARY to TERTIARY/EQUAL is-in logic unconditionally upper-casing everything isn't ideal - I don't think it adds to the problem but its effect might be visible. The strange behaviour in this example is because, although combiners/MDRBuilder attempts to do case-insensitive dedup on COUNTRY, REGION and CITY, a CITY POI (only existing in 1 tile) evades this and so, in this tile, there might be 2 entries with different city.index. This needs to be fixed before trying to change from TERTIARY/EQUAL to SECONDARY. Concerning the case where there really are 2 Cities with the same name in the same region: There isn't any way of distinguishing which streets belong to which (in my example the correct city is in another tile), so how many streets with the same name to show? With the current mkgmap implementation of not joining streets with different attributes, there might be many in the same city. mkgmap appears to dedup them (give or take the r/rr flags which I haven't understood yet), which is reasonable if all in 1 city but not if >1. Ticker On Wed, 2021-11-24 at 16:39 +0000, Gerd Petermann wrote:
Hi Ticker,
OK, with the zipped style I could reproduce (with r4808) some results. I think MapSource works well, but with my Oregon I also have the problem that I have to decide whether I want to search in wherWell or in Wherwell, and the device will only find the roads in the corresponding city. That's not nice but at least it looks correct, means, it doesn't find Alma Lane in "Wherwell", only in "wherWell", and e.g. Bigpath Lane is found in both cities, but different and correct roads for each.
When I change your style to replace Upton by Wherwell the device shows only one Wherwell as city after typing wh and three occurrences for Bigpath Lane. That's good and I guess you would want the same for the special case that cities are written with only TERTIARY differences?
No idea if this is a good approach. Maybe it's better to let the user stumble over this so that someone fixes the wrong OSM data.
Gerd
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Ticker, it's all quite confusing for me because your table mentiones Upton but the rules say Upham ;) Anyway, when I remove the rules which replace Upham by wherWell the road "Baybridge Lane" is still found in the upper tile, so it is not a problem with the global index. Looking at the current data I see that the road crosses the boundary between Owslebury and Upham near https://www.openstreetmap.org/way/368825450, so it's not clear to which area it belongs. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Donnerstag, 25. November 2021 06:37 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] Small problem with global index Hi Ticker, OK, sorry, did not understand the table. I can confirm that the search for "Baybridge Lane" doesn't work as expected. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Mittwoch, 24. November 2021 18:58 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] Small problem with global index Hi Gerd In this example, choosing the correct version of the spelling doesn't necessarily work! To find "Alma Lane", which is in Upton=wherWell, "wherWell" must be selected as the city. To find "Baybridge Lane", also in Upton=wherWell, "Wherwell" must be given. Getting capitalisation consistent in OSM would be a good thing. Generally, for cities, it is pretty good; in the original problem report (Arndt / NRW + surrounding) I found only 3 examples. However, for streets, I find a lots. This is why I'm keen on getting to a nicely working case-insensitive search and against moving existing logic from SECONDARY to TERTIARY/EQUAL is-in logic unconditionally upper-casing everything isn't ideal - I don't think it adds to the problem but its effect might be visible. The strange behaviour in this example is because, although combiners/MDRBuilder attempts to do case-insensitive dedup on COUNTRY, REGION and CITY, a CITY POI (only existing in 1 tile) evades this and so, in this tile, there might be 2 entries with different city.index. This needs to be fixed before trying to change from TERTIARY/EQUAL to SECONDARY. Concerning the case where there really are 2 Cities with the same name in the same region: There isn't any way of distinguishing which streets belong to which (in my example the correct city is in another tile), so how many streets with the same name to show? With the current mkgmap implementation of not joining streets with different attributes, there might be many in the same city. mkgmap appears to dedup them (give or take the r/rr flags which I haven't understood yet), which is reasonable if all in 1 city but not if >1. Ticker On Wed, 2021-11-24 at 16:39 +0000, Gerd Petermann wrote:
Hi Ticker,
OK, with the zipped style I could reproduce (with r4808) some results. I think MapSource works well, but with my Oregon I also have the problem that I have to decide whether I want to search in wherWell or in Wherwell, and the device will only find the roads in the corresponding city. That's not nice but at least it looks correct, means, it doesn't find Alma Lane in "Wherwell", only in "wherWell", and e.g. Bigpath Lane is found in both cities, but different and correct roads for each.
When I change your style to replace Upton by Wherwell the device shows only one Wherwell as city after typing wh and three occurrences for Bigpath Lane. That's good and I guess you would want the same for the special case that cities are written with only TERTIARY differences?
No idea if this is a good approach. Maybe it's better to let the user stumble over this so that someone fixes the wrong OSM data.
Gerd
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Gerd Sorry about the confusion with Upton/Upham. Baybridge Lane has a section in Upham and there is nothing of this name in Wherwell so it is clear it should be found as a street in Upham (wherWell with the rename). I'm not saying there is a problem with the global index, except that when using --lower-case and there are case-variations in city names, it might be necessary for the user to guess which variation to use to find a street and, worse, some streets in the same city might require the choice of the other variant. Ticker On Thu, 2021-11-25 at 06:15 +0000, Gerd Petermann wrote:
Hi Ticker,
it's all quite confusing for me because your table mentiones Upton but the rules say Upham ;) Anyway, when I remove the rules which replace Upham by wherWell the road "Baybridge Lane" is still found in the upper tile, so it is not a problem with the global index. Looking at the current data I see that the road crosses the boundary between Owslebury and Upham near https://www.openstreetmap.org/way/368825450, so it's not clear to which area it belongs.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Donnerstag, 25. November 2021 06:37 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] Small problem with global index
Hi Ticker,
OK, sorry, did not understand the table. I can confirm that the search for "Baybridge Lane" doesn't work as expected.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Mittwoch, 24. November 2021 18:58 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] Small problem with global index
Hi Gerd
In this example, choosing the correct version of the spelling doesn't necessarily work!
To find "Alma Lane", which is in Upton=wherWell, "wherWell" must be selected as the city.
To find "Baybridge Lane", also in Upton=wherWell, "Wherwell" must be given.
Getting capitalisation consistent in OSM would be a good thing. Generally, for cities, it is pretty good; in the original problem report (Arndt / NRW + surrounding) I found only 3 examples. However, for streets, I find a lots. This is why I'm keen on getting to a nicely working case-insensitive search and against moving existing logic from SECONDARY to TERTIARY/EQUAL
is-in logic unconditionally upper-casing everything isn't ideal - I don't think it adds to the problem but its effect might be visible.
The strange behaviour in this example is because, although combiners/MDRBuilder attempts to do case-insensitive dedup on COUNTRY, REGION and CITY, a CITY POI (only existing in 1 tile) evades this and so, in this tile, there might be 2 entries with different city.index. This needs to be fixed before trying to change from TERTIARY/EQUAL to SECONDARY.
Concerning the case where there really are 2 Cities with the same name in the same region: There isn't any way of distinguishing which streets belong to which (in my example the correct city is in another tile), so how many streets with the same name to show? With the current mkgmap implementation of not joining streets with different attributes, there might be many in the same city. mkgmap appears to dedup them (give or take the r/rr flags which I haven't understood yet), which is reasonable if all in 1 city but not if >1.
Ticker
On Wed, 2021-11-24 at 16:39 +0000, Gerd Petermann wrote:
Hi Ticker,
OK, with the zipped style I could reproduce (with r4808) some results. I think MapSource works well, but with my Oregon I also have the problem that I have to decide whether I want to search in wherWell or in Wherwell, and the device will only find the roads in the corresponding city. That's not nice but at least it looks correct, means, it doesn't find Alma Lane in "Wherwell", only in "wherWell", and e.g. Bigpath Lane is found in both cities, but different and correct roads for each.
When I change your style to replace Upton by Wherwell the device shows only one Wherwell as city after typing wh and three occurrences for Bigpath Lane. That's good and I guess you would want the same for the special case that cities are written with only TERTIARY differences?
No idea if this is a good approach. Maybe it's better to let the user stumble over this so that someone fixes the wrong OSM data.
Gerd
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Ticker, yes, sorry, I probably wasn't fully awake this morning and forgot what the problem is :( Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Donnerstag, 25. November 2021 10:41 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] Small problem with global index Hi Gerd Sorry about the confusion with Upton/Upham. Baybridge Lane has a section in Upham and there is nothing of this name in Wherwell so it is clear it should be found as a street in Upham (wherWell with the rename). I'm not saying there is a problem with the global index, except that when using --lower-case and there are case-variations in city names, it might be necessary for the user to guess which variation to use to find a street and, worse, some streets in the same city might require the choice of the other variant. Ticker On Thu, 2021-11-25 at 06:15 +0000, Gerd Petermann wrote:
Hi Ticker,
it's all quite confusing for me because your table mentiones Upton but the rules say Upham ;) Anyway, when I remove the rules which replace Upham by wherWell the road "Baybridge Lane" is still found in the upper tile, so it is not a problem with the global index. Looking at the current data I see that the road crosses the boundary between Owslebury and Upham near https://www.openstreetmap.org/way/368825450, so it's not clear to which area it belongs.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Donnerstag, 25. November 2021 06:37 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] Small problem with global index
Hi Ticker,
OK, sorry, did not understand the table. I can confirm that the search for "Baybridge Lane" doesn't work as expected.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Mittwoch, 24. November 2021 18:58 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] Small problem with global index
Hi Gerd
In this example, choosing the correct version of the spelling doesn't necessarily work!
To find "Alma Lane", which is in Upton=wherWell, "wherWell" must be selected as the city.
To find "Baybridge Lane", also in Upton=wherWell, "Wherwell" must be given.
Getting capitalisation consistent in OSM would be a good thing. Generally, for cities, it is pretty good; in the original problem report (Arndt / NRW + surrounding) I found only 3 examples. However, for streets, I find a lots. This is why I'm keen on getting to a nicely working case-insensitive search and against moving existing logic from SECONDARY to TERTIARY/EQUAL
is-in logic unconditionally upper-casing everything isn't ideal - I don't think it adds to the problem but its effect might be visible.
The strange behaviour in this example is because, although combiners/MDRBuilder attempts to do case-insensitive dedup on COUNTRY, REGION and CITY, a CITY POI (only existing in 1 tile) evades this and so, in this tile, there might be 2 entries with different city.index. This needs to be fixed before trying to change from TERTIARY/EQUAL to SECONDARY.
Concerning the case where there really are 2 Cities with the same name in the same region: There isn't any way of distinguishing which streets belong to which (in my example the correct city is in another tile), so how many streets with the same name to show? With the current mkgmap implementation of not joining streets with different attributes, there might be many in the same city. mkgmap appears to dedup them (give or take the r/rr flags which I haven't understood yet), which is reasonable if all in 1 city but not if >1.
Ticker
On Wed, 2021-11-24 at 16:39 +0000, Gerd Petermann wrote:
Hi Ticker,
OK, with the zipped style I could reproduce (with r4808) some results. I think MapSource works well, but with my Oregon I also have the problem that I have to decide whether I want to search in wherWell or in Wherwell, and the device will only find the roads in the corresponding city. That's not nice but at least it looks correct, means, it doesn't find Alma Lane in "Wherwell", only in "wherWell", and e.g. Bigpath Lane is found in both cities, but different and correct roads for each.
When I change your style to replace Upton by Wherwell the device shows only one Wherwell as city after typing wh and three occurrences for Bigpath Lane. That's good and I guess you would want the same for the special case that cities are written with only TERTIARY differences?
No idea if this is a good approach. Maybe it's better to let the user stumble over this so that someone fixes the wrong OSM data.
Gerd
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Ticker, reg. --lower-case and city/region/country names with different capitalization: I think it would be good to keep the different capitalization within a single tile, so yes, the .toUpperCase() in PlacesFile is probably not a good idea. Results seem better when this is not done. When the global index is created we can log warnings for those cases, but I don't see yet how we can create a valid index which doesn't require the user to decide whether wherWell or Wherwell should be searched. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Donnerstag, 25. November 2021 10:43 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] Small problem with global index Hi Ticker, yes, sorry, I probably wasn't fully awake this morning and forgot what the problem is :( Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Donnerstag, 25. November 2021 10:41 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] Small problem with global index Hi Gerd Sorry about the confusion with Upton/Upham. Baybridge Lane has a section in Upham and there is nothing of this name in Wherwell so it is clear it should be found as a street in Upham (wherWell with the rename). I'm not saying there is a problem with the global index, except that when using --lower-case and there are case-variations in city names, it might be necessary for the user to guess which variation to use to find a street and, worse, some streets in the same city might require the choice of the other variant. Ticker On Thu, 2021-11-25 at 06:15 +0000, Gerd Petermann wrote:
Hi Ticker,
it's all quite confusing for me because your table mentiones Upton but the rules say Upham ;) Anyway, when I remove the rules which replace Upham by wherWell the road "Baybridge Lane" is still found in the upper tile, so it is not a problem with the global index. Looking at the current data I see that the road crosses the boundary between Owslebury and Upham near https://www.openstreetmap.org/way/368825450, so it's not clear to which area it belongs.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Donnerstag, 25. November 2021 06:37 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] Small problem with global index
Hi Ticker,
OK, sorry, did not understand the table. I can confirm that the search for "Baybridge Lane" doesn't work as expected.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Mittwoch, 24. November 2021 18:58 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] Small problem with global index
Hi Gerd
In this example, choosing the correct version of the spelling doesn't necessarily work!
To find "Alma Lane", which is in Upton=wherWell, "wherWell" must be selected as the city.
To find "Baybridge Lane", also in Upton=wherWell, "Wherwell" must be given.
Getting capitalisation consistent in OSM would be a good thing. Generally, for cities, it is pretty good; in the original problem report (Arndt / NRW + surrounding) I found only 3 examples. However, for streets, I find a lots. This is why I'm keen on getting to a nicely working case-insensitive search and against moving existing logic from SECONDARY to TERTIARY/EQUAL
is-in logic unconditionally upper-casing everything isn't ideal - I don't think it adds to the problem but its effect might be visible.
The strange behaviour in this example is because, although combiners/MDRBuilder attempts to do case-insensitive dedup on COUNTRY, REGION and CITY, a CITY POI (only existing in 1 tile) evades this and so, in this tile, there might be 2 entries with different city.index. This needs to be fixed before trying to change from TERTIARY/EQUAL to SECONDARY.
Concerning the case where there really are 2 Cities with the same name in the same region: There isn't any way of distinguishing which streets belong to which (in my example the correct city is in another tile), so how many streets with the same name to show? With the current mkgmap implementation of not joining streets with different attributes, there might be many in the same city. mkgmap appears to dedup them (give or take the r/rr flags which I haven't understood yet), which is reasonable if all in 1 city but not if >1.
Ticker
On Wed, 2021-11-24 at 16:39 +0000, Gerd Petermann wrote:
Hi Ticker,
OK, with the zipped style I could reproduce (with r4808) some results. I think MapSource works well, but with my Oregon I also have the problem that I have to decide whether I want to search in wherWell or in Wherwell, and the device will only find the roads in the corresponding city. That's not nice but at least it looks correct, means, it doesn't find Alma Lane in "Wherwell", only in "wherWell", and e.g. Bigpath Lane is found in both cities, but different and correct roads for each.
When I change your style to replace Upton by Wherwell the device shows only one Wherwell as city after typing wh and three occurrences for Bigpath Lane. That's good and I guess you would want the same for the special case that cities are written with only TERTIARY differences?
No idea if this is a good approach. Maybe it's better to let the user stumble over this so that someone fixes the wrong OSM data.
Gerd
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Gerd I was sort of thinking the opposite. PlaceFile using some method (eg what it does) to dedupe city/region/country and this is extended to the POI/MDRBuilder logic, such that combinations of these 3 have unique sets of index values regardless of case. Then the relevant MdrX sections should be able to do a SECONDARY dedup on these, to cope with case-variants coming from different tiles. Then checking that this does actually work with Garmin software (ie hope nothing cares that the index entries might not match the LBL data in some of the tiles) If this works, there should only be one city presented in the find options - eg, from the original problem data, it might be "De Wijk" or "de Wijk" Then making MdrCheck tolerant of this as well. An alternative is just to ignore the whole issue - no one else has ever noticed and complained. I was hoping to get mdrUnicode_v9b.patch accepted before tackling this. Its purpose is to fix the crash when pathological city / region / country names or incomplete sortorder codepage data causes enough difference between TERTIARY & EQUAL to make Mdr25 index size too big. Ticker On Fri, 2021-11-26 at 10:56 +0000, Gerd Petermann wrote:
Hi Ticker,
reg. --lower-case and city/region/country names with different capitalization: I think it would be good to keep the different capitalization within a single tile, so yes, the .toUpperCase() in PlacesFile is probably not a good idea. Results seem better when this is not done. When the global index is created we can log warnings for those cases, but I don't see yet how we can create a valid index which doesn't require the user to decide whether wherWell or Wherwell should be searched.
Gerd
Hi Ticker, I tried this: use your command to build a gmapi and gmapsupp, but replace r4810 by a binary compiled from mkgmap r4717 + mdrUnicode_v9b.patch (I still see no difference in the output compared to unpatched r4717) I then use MapSource to create another gmapsupp. I run MdrDisplay and MdrCheck on both gmapsupp.img and see different repeat flags. MdrCheck + my patch display-no-secondary.patch complains a lot about the gmapsupp with your patch but reports only 1 problem about a city without name. When I try this with the unpatched display tool it complains a lot about the gmapsupp from MapSource but not about the one from mkgmap. I think that shows that unpatched MdrCheck is wrong. I tried this also with a binary from r4817 with attached mkgmap-no-secondary-v2.patch with your command. The two outputs from MdrCheck are identical, and I think the outputs for MdrDisplay differ only in offsets. I consider this very good. In MapSource the search for "Baybride Lane" and "Alma Lane" both return wherWell, so that's also good. I prefer a patch that changes mkgmap to produce the same index as MapSource. I hope I've done nothing wrong during testing. Do you get other results? Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Freitag, 26. November 2021 16:27 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] Small problem with global index Hi Gerd I was sort of thinking the opposite. PlaceFile using some method (eg what it does) to dedupe city/region/country and this is extended to the POI/MDRBuilder logic, such that combinations of these 3 have unique sets of index values regardless of case. Then the relevant MdrX sections should be able to do a SECONDARY dedup on these, to cope with case-variants coming from different tiles. Then checking that this does actually work with Garmin software (ie hope nothing cares that the index entries might not match the LBL data in some of the tiles) If this works, there should only be one city presented in the find options - eg, from the original problem data, it might be "De Wijk" or "de Wijk" Then making MdrCheck tolerant of this as well. An alternative is just to ignore the whole issue - no one else has ever noticed and complained. I was hoping to get mdrUnicode_v9b.patch accepted before tackling this. Its purpose is to fix the crash when pathological city / region / country names or incomplete sortorder codepage data causes enough difference between TERTIARY & EQUAL to make Mdr25 index size too big. Ticker On Fri, 2021-11-26 at 10:56 +0000, Gerd Petermann wrote:
Hi Ticker,
reg. --lower-case and city/region/country names with different capitalization: I think it would be good to keep the different capitalization within a single tile, so yes, the .toUpperCase() in PlacesFile is probably not a good idea. Results seem better when this is not done. When the global index is created we can log warnings for those cases, but I don't see yet how we can create a valid index which doesn't require the user to decide whether wherWell or Wherwell should be searched.
Gerd
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Ticker, sorry, meant r4718 instead of 4717 before. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Freitag, 26. November 2021 18:06 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] Small problem with global index Hi Ticker, I tried this: use your command to build a gmapi and gmapsupp, but replace r4810 by a binary compiled from mkgmap r4717 + mdrUnicode_v9b.patch (I still see no difference in the output compared to unpatched r4717) I then use MapSource to create another gmapsupp. I run MdrDisplay and MdrCheck on both gmapsupp.img and see different repeat flags. MdrCheck + my patch display-no-secondary.patch complains a lot about the gmapsupp with your patch but reports only 1 problem about a city without name. When I try this with the unpatched display tool it complains a lot about the gmapsupp from MapSource but not about the one from mkgmap. I think that shows that unpatched MdrCheck is wrong. I tried this also with a binary from r4817 with attached mkgmap-no-secondary-v2.patch with your command. The two outputs from MdrCheck are identical, and I think the outputs for MdrDisplay differ only in offsets. I consider this very good. In MapSource the search for "Baybride Lane" and "Alma Lane" both return wherWell, so that's also good. I prefer a patch that changes mkgmap to produce the same index as MapSource. I hope I've done nothing wrong during testing. Do you get other results? Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Freitag, 26. November 2021 16:27 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] Small problem with global index Hi Gerd I was sort of thinking the opposite. PlaceFile using some method (eg what it does) to dedupe city/region/country and this is extended to the POI/MDRBuilder logic, such that combinations of these 3 have unique sets of index values regardless of case. Then the relevant MdrX sections should be able to do a SECONDARY dedup on these, to cope with case-variants coming from different tiles. Then checking that this does actually work with Garmin software (ie hope nothing cares that the index entries might not match the LBL data in some of the tiles) If this works, there should only be one city presented in the find options - eg, from the original problem data, it might be "De Wijk" or "de Wijk" Then making MdrCheck tolerant of this as well. An alternative is just to ignore the whole issue - no one else has ever noticed and complained. I was hoping to get mdrUnicode_v9b.patch accepted before tackling this. Its purpose is to fix the crash when pathological city / region / country names or incomplete sortorder codepage data causes enough difference between TERTIARY & EQUAL to make Mdr25 index size too big. Ticker On Fri, 2021-11-26 at 10:56 +0000, Gerd Petermann wrote:
Hi Ticker,
reg. --lower-case and city/region/country names with different capitalization: I think it would be good to keep the different capitalization within a single tile, so yes, the .toUpperCase() in PlacesFile is probably not a good idea. Results seem better when this is not done. When the global index is created we can log warnings for those cases, but I don't see yet how we can create a valid index which doesn't require the user to decide whether wherWell or Wherwell should be searched.
Gerd
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Gerd mdrUnicode_v9b.patch isn't related to the issue of case-variants; it is about keeping consistency between Mdr5 and Mdr25 indexes. This will go wrong when there is a difference between TERTIARY and EQUAL in Country, Region and City names. It may be that this doesn't matter to Garmin software, or, more likely, will introduce slight errors in what is findable. If you don't want to accept this patch, I think changes would be needed to Mdr5 to replace TERTIARY collator use with .equals(). Ticker On Fri, 2021-11-26 at 18:04 +0000, Gerd Petermann wrote:
Hi Ticker,
sorry, meant r4718 instead of 4717 before.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Freitag, 26. November 2021 18:06 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] Small problem with global index
Hi Ticker,
I tried this: use your command to build a gmapi and gmapsupp, but replace r4810 by a binary compiled from mkgmap r4717 + mdrUnicode_v9b.patch (I still see no difference in the output compared to unpatched r4717)
I then use MapSource to create another gmapsupp. I run MdrDisplay and MdrCheck on both gmapsupp.img and see different repeat flags. MdrCheck + my patch display-no-secondary.patch complains a lot about the gmapsupp with your patch but reports only 1 problem about a city without name. When I try this with the unpatched display tool it complains a lot about the gmapsupp from MapSource but not about the one from mkgmap. I think that shows that unpatched MdrCheck is wrong.
I tried this also with a binary from r4817 with attached mkgmap-no- secondary-v2.patch with your command. The two outputs from MdrCheck are identical, and I think the outputs for MdrDisplay differ only in offsets. I consider this very good. In MapSource the search for "Baybride Lane" and "Alma Lane" both return wherWell, so that's also good.
I prefer a patch that changes mkgmap to produce the same index as MapSource.
I hope I've done nothing wrong during testing. Do you get other results?
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Freitag, 26. November 2021 16:27 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] Small problem with global index
Hi Gerd
I was sort of thinking the opposite. PlaceFile using some method (eg what it does) to dedupe city/region/country and this is extended to the POI/MDRBuilder logic, such that combinations of these 3 have unique sets of index values regardless of case.
Then the relevant MdrX sections should be able to do a SECONDARY dedup on these, to cope with case-variants coming from different tiles.
Then checking that this does actually work with Garmin software (ie hope nothing cares that the index entries might not match the LBL data in some of the tiles)
If this works, there should only be one city presented in the find options - eg, from the original problem data, it might be "De Wijk" or "de Wijk"
Then making MdrCheck tolerant of this as well.
An alternative is just to ignore the whole issue - no one else has ever noticed and complained.
I was hoping to get mdrUnicode_v9b.patch accepted before tackling this. Its purpose is to fix the crash when pathological city / region / country names or incomplete sortorder codepage data causes enough difference between TERTIARY & EQUAL to make Mdr25 index size too big.
Ticker
On Fri, 2021-11-26 at 10:56 +0000, Gerd Petermann wrote:
Hi Ticker,
reg. --lower-case and city/region/country names with different capitalization: I think it would be good to keep the different capitalization within a single tile, so yes, the .toUpperCase() in PlacesFile is probably not a good idea. Results seem better when this is not done. When the global index is created we can log warnings for those cases, but I don't see yet how we can create a valid index which doesn't require the user to decide whether wherWell or Wherwell should be searched.
Gerd
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Ticker, running in circles, aren't we? I ask for sample data to show that mdrUnicode_v9b.patch makes a difference in some special case. I totally agree that either Mdr5 or Mdr25 should be changed, and probably other places, too. What I really like to have is a (small) example that shows a difference between TERTIARY and EQUAL so that I am able to compare mkgmap results with what Garmin does. The highway shield codes may not be a good idea in case Garmin treats them special, but I also would like to understand that special handling if it exists. I think all I need is a way to find a String that gives TERTIARY == 0 and String.equals() returns false for a given codepage. Maybe this is totally clear to you but it is not for me. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Samstag, 27. November 2021 11:23 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] Small problem with global index Hi Gerd mdrUnicode_v9b.patch isn't related to the issue of case-variants; it is about keeping consistency between Mdr5 and Mdr25 indexes. This will go wrong when there is a difference between TERTIARY and EQUAL in Country, Region and City names. It may be that this doesn't matter to Garmin software, or, more likely, will introduce slight errors in what is findable. If you don't want to accept this patch, I think changes would be needed to Mdr5 to replace TERTIARY collator use with .equals(). Ticker On Fri, 2021-11-26 at 18:04 +0000, Gerd Petermann wrote:
Hi Ticker,
sorry, meant r4718 instead of 4717 before.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Freitag, 26. November 2021 18:06 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] Small problem with global index
Hi Ticker,
I tried this: use your command to build a gmapi and gmapsupp, but replace r4810 by a binary compiled from mkgmap r4717 + mdrUnicode_v9b.patch (I still see no difference in the output compared to unpatched r4717)
I then use MapSource to create another gmapsupp. I run MdrDisplay and MdrCheck on both gmapsupp.img and see different repeat flags. MdrCheck + my patch display-no-secondary.patch complains a lot about the gmapsupp with your patch but reports only 1 problem about a city without name. When I try this with the unpatched display tool it complains a lot about the gmapsupp from MapSource but not about the one from mkgmap. I think that shows that unpatched MdrCheck is wrong.
I tried this also with a binary from r4817 with attached mkgmap-no- secondary-v2.patch with your command. The two outputs from MdrCheck are identical, and I think the outputs for MdrDisplay differ only in offsets. I consider this very good. In MapSource the search for "Baybride Lane" and "Alma Lane" both return wherWell, so that's also good.
I prefer a patch that changes mkgmap to produce the same index as MapSource.
I hope I've done nothing wrong during testing. Do you get other results?
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Freitag, 26. November 2021 16:27 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] Small problem with global index
Hi Gerd
I was sort of thinking the opposite. PlaceFile using some method (eg what it does) to dedupe city/region/country and this is extended to the POI/MDRBuilder logic, such that combinations of these 3 have unique sets of index values regardless of case.
Then the relevant MdrX sections should be able to do a SECONDARY dedup on these, to cope with case-variants coming from different tiles.
Then checking that this does actually work with Garmin software (ie hope nothing cares that the index entries might not match the LBL data in some of the tiles)
If this works, there should only be one city presented in the find options - eg, from the original problem data, it might be "De Wijk" or "de Wijk"
Then making MdrCheck tolerant of this as well.
An alternative is just to ignore the whole issue - no one else has ever noticed and complained.
I was hoping to get mdrUnicode_v9b.patch accepted before tackling this. Its purpose is to fix the crash when pathological city / region / country names or incomplete sortorder codepage data causes enough difference between TERTIARY & EQUAL to make Mdr25 index size too big.
Ticker
On Fri, 2021-11-26 at 10:56 +0000, Gerd Petermann wrote:
Hi Ticker,
reg. --lower-case and city/region/country names with different capitalization: I think it would be good to keep the different capitalization within a single tile, so yes, the .toUpperCase() in PlacesFile is probably not a good idea. Results seem better when this is not done. When the global index is created we can log warnings for those cases, but I don't see yet how we can create a valid index which doesn't require the user to decide whether wherWell or Wherwell should be searched.
Gerd
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Gerd The drastic case which mdrUnicode_v9b fixes is the index byte size crash, and this is most easily demonstrating by having enough ignorable characters, eg shields or Chinese & Unicode without the original Sort fix. Until this crash point is reached, I've no idea if there is a problem that can be demonstrated. However, the data structure whereby Mdr25 shares the same byte-size pointer to Mdr5 strongly indicates that these should be kept in step and anything that allows Mdr25 to be bigger than Mdr5 muse be wrong. The new version of Sort for Unicode assumes that if ordering has been defined for any characters in a [256] page, then any characters in this page but not defined will get a zero/ignore sortOrder. If nothing has been defined for the page the code will invent an sortOrder. So some diag code in Sort should be able to list some other characters that will get a zero sortOrder, hopefully there might be some nice name-like chars amongst them. Apart from the ignored sortOrder chars making a difference between TERTIARY and .equal(): A significant consideration is that Sort.java doesn't sort higher than the TERTIARY level, so it is possible to end up with a section of TERTIARY same records, but adjacent records in this set might be .equal() or there might be a non equal record between equals. Again, no idea if this will matter in areas like the repeat flag setting, but it indicates strongly that should use collator.compare rather than .equal() for dedup. Ticker On Sat, 2021-11-27 at 10:54 +0000, Gerd Petermann wrote:
Hi Ticker,
running in circles, aren't we? I ask for sample data to show that mdrUnicode_v9b.patch makes a difference in some special case. I totally agree that either Mdr5 or Mdr25 should be changed, and probably other places, too.
What I really like to have is a (small) example that shows a difference between TERTIARY and EQUAL so that I am able to compare mkgmap results with what Garmin does. The highway shield codes may not be a good idea in case Garmin treats them special, but I also would like to understand that special handling if it exists.
I think all I need is a way to find a String that gives TERTIARY == 0 and String.equals() returns false for a given codepage. Maybe this is totally clear to you but it is not for me.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Samstag, 27. November 2021 11:23 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] Small problem with global index
Hi Gerd
mdrUnicode_v9b.patch isn't related to the issue of case-variants; it is about keeping consistency between Mdr5 and Mdr25 indexes. This will go wrong when there is a difference between TERTIARY and EQUAL in Country, Region and City names. It may be that this doesn't matter to Garmin software, or, more likely, will introduce slight errors in what is findable.
If you don't want to accept this patch, I think changes would be needed to Mdr5 to replace TERTIARY collator use with .equals().
Ticker
On Fri, 2021-11-26 at 18:04 +0000, Gerd Petermann wrote:
Hi Ticker,
sorry, meant r4718 instead of 4717 before.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Freitag, 26. November 2021 18:06 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] Small problem with global index
Hi Ticker,
I tried this: use your command to build a gmapi and gmapsupp, but replace r4810 by a binary compiled from mkgmap r4717 + mdrUnicode_v9b.patch (I still see no difference in the output compared to unpatched r4717)
I then use MapSource to create another gmapsupp. I run MdrDisplay and MdrCheck on both gmapsupp.img and see different repeat flags. MdrCheck + my patch display-no-secondary.patch complains a lot about the gmapsupp with your patch but reports only 1 problem about a city without name. When I try this with the unpatched display tool it complains a lot about the gmapsupp from MapSource but not about the one from mkgmap. I think that shows that unpatched MdrCheck is wrong.
I tried this also with a binary from r4817 with attached mkgmap-no- secondary-v2.patch with your command. The two outputs from MdrCheck are identical, and I think the outputs for MdrDisplay differ only in offsets. I consider this very good. In MapSource the search for "Baybride Lane" and "Alma Lane" both return wherWell, so that's also good.
I prefer a patch that changes mkgmap to produce the same index as MapSource.
I hope I've done nothing wrong during testing. Do you get other results?
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Freitag, 26. November 2021 16:27 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] Small problem with global index
Hi Gerd
I was sort of thinking the opposite. PlaceFile using some method (eg what it does) to dedupe city/region/country and this is extended to the POI/MDRBuilder logic, such that combinations of these 3 have unique sets of index values regardless of case.
Then the relevant MdrX sections should be able to do a SECONDARY dedup on these, to cope with case-variants coming from different tiles.
Then checking that this does actually work with Garmin software (ie hope nothing cares that the index entries might not match the LBL data in some of the tiles)
If this works, there should only be one city presented in the find options - eg, from the original problem data, it might be "De Wijk" or "de Wijk"
Then making MdrCheck tolerant of this as well.
An alternative is just to ignore the whole issue - no one else has ever noticed and complained.
I was hoping to get mdrUnicode_v9b.patch accepted before tackling this. Its purpose is to fix the crash when pathological city / region / country names or incomplete sortorder codepage data causes enough difference between TERTIARY & EQUAL to make Mdr25 index size too big.
Ticker
On Fri, 2021-11-26 at 10:56 +0000, Gerd Petermann wrote:
Hi Ticker,
reg. --lower-case and city/region/country names with different capitalization: I think it would be good to keep the different capitalization within a single tile, so yes, the .toUpperCase() in PlacesFile is probably not a good idea. Results seem better when this is not done. When the global index is created we can log warnings for those cases, but I don't see yet how we can create a valid index which doesn't require the user to decide whether wherWell or Wherwell should be searched.
Gerd
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Ticker, I think I found an example in a map for China. A road name contains U+200E LEFT-TO-RIGHT MARK https://www.fontspace.com/unicode/analyzer#e=55Sw6LSd4oCO5LiJ6Lev Another road name doesn't contain this. Both strings are TERTIARY equal but not identical. It should be possible to create some test maps to find out how Garmin treats this :) Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Samstag, 27. November 2021 13:05 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] Small problem with global index Hi Gerd The drastic case which mdrUnicode_v9b fixes is the index byte size crash, and this is most easily demonstrating by having enough ignorable characters, eg shields or Chinese & Unicode without the original Sort fix. Until this crash point is reached, I've no idea if there is a problem that can be demonstrated. However, the data structure whereby Mdr25 shares the same byte-size pointer to Mdr5 strongly indicates that these should be kept in step and anything that allows Mdr25 to be bigger than Mdr5 muse be wrong. The new version of Sort for Unicode assumes that if ordering has been defined for any characters in a [256] page, then any characters in this page but not defined will get a zero/ignore sortOrder. If nothing has been defined for the page the code will invent an sortOrder. So some diag code in Sort should be able to list some other characters that will get a zero sortOrder, hopefully there might be some nice name-like chars amongst them. Apart from the ignored sortOrder chars making a difference between TERTIARY and .equal(): A significant consideration is that Sort.java doesn't sort higher than the TERTIARY level, so it is possible to end up with a section of TERTIARY same records, but adjacent records in this set might be .equal() or there might be a non equal record between equals. Again, no idea if this will matter in areas like the repeat flag setting, but it indicates strongly that should use collator.compare rather than .equal() for dedup. Ticker On Sat, 2021-11-27 at 10:54 +0000, Gerd Petermann wrote:
Hi Ticker,
running in circles, aren't we? I ask for sample data to show that mdrUnicode_v9b.patch makes a difference in some special case. I totally agree that either Mdr5 or Mdr25 should be changed, and probably other places, too.
What I really like to have is a (small) example that shows a difference between TERTIARY and EQUAL so that I am able to compare mkgmap results with what Garmin does. The highway shield codes may not be a good idea in case Garmin treats them special, but I also would like to understand that special handling if it exists.
I think all I need is a way to find a String that gives TERTIARY == 0 and String.equals() returns false for a given codepage. Maybe this is totally clear to you but it is not for me.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Samstag, 27. November 2021 11:23 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] Small problem with global index
Hi Gerd
mdrUnicode_v9b.patch isn't related to the issue of case-variants; it is about keeping consistency between Mdr5 and Mdr25 indexes. This will go wrong when there is a difference between TERTIARY and EQUAL in Country, Region and City names. It may be that this doesn't matter to Garmin software, or, more likely, will introduce slight errors in what is findable.
If you don't want to accept this patch, I think changes would be needed to Mdr5 to replace TERTIARY collator use with .equals().
Ticker
On Fri, 2021-11-26 at 18:04 +0000, Gerd Petermann wrote:
Hi Ticker,
sorry, meant r4718 instead of 4717 before.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Freitag, 26. November 2021 18:06 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] Small problem with global index
Hi Ticker,
I tried this: use your command to build a gmapi and gmapsupp, but replace r4810 by a binary compiled from mkgmap r4717 + mdrUnicode_v9b.patch (I still see no difference in the output compared to unpatched r4717)
I then use MapSource to create another gmapsupp. I run MdrDisplay and MdrCheck on both gmapsupp.img and see different repeat flags. MdrCheck + my patch display-no-secondary.patch complains a lot about the gmapsupp with your patch but reports only 1 problem about a city without name. When I try this with the unpatched display tool it complains a lot about the gmapsupp from MapSource but not about the one from mkgmap. I think that shows that unpatched MdrCheck is wrong.
I tried this also with a binary from r4817 with attached mkgmap-no- secondary-v2.patch with your command. The two outputs from MdrCheck are identical, and I think the outputs for MdrDisplay differ only in offsets. I consider this very good. In MapSource the search for "Baybride Lane" and "Alma Lane" both return wherWell, so that's also good.
I prefer a patch that changes mkgmap to produce the same index as MapSource.
I hope I've done nothing wrong during testing. Do you get other results?
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Freitag, 26. November 2021 16:27 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] Small problem with global index
Hi Gerd
I was sort of thinking the opposite. PlaceFile using some method (eg what it does) to dedupe city/region/country and this is extended to the POI/MDRBuilder logic, such that combinations of these 3 have unique sets of index values regardless of case.
Then the relevant MdrX sections should be able to do a SECONDARY dedup on these, to cope with case-variants coming from different tiles.
Then checking that this does actually work with Garmin software (ie hope nothing cares that the index entries might not match the LBL data in some of the tiles)
If this works, there should only be one city presented in the find options - eg, from the original problem data, it might be "De Wijk" or "de Wijk"
Then making MdrCheck tolerant of this as well.
An alternative is just to ignore the whole issue - no one else has ever noticed and complained.
I was hoping to get mdrUnicode_v9b.patch accepted before tackling this. Its purpose is to fix the crash when pathological city / region / country names or incomplete sortorder codepage data causes enough difference between TERTIARY & EQUAL to make Mdr25 index size too big.
Ticker
On Fri, 2021-11-26 at 10:56 +0000, Gerd Petermann wrote:
Hi Ticker,
reg. --lower-case and city/region/country names with different capitalization: I think it would be good to keep the different capitalization within a single tile, so yes, the .toUpperCase() in PlacesFile is probably not a good idea. Results seem better when this is not done. When the global index is created we can log warnings for those cases, but I don't see yet how we can create a valid index which doesn't require the user to decide whether wherWell or Wherwell should be searched.
Gerd
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
participants (3)
-
Gerd Petermann -
Gerd Petermann
-
Ticker Berkin