Index with Arabic names
Hi... I've used mkgmap-r1955 to convert the map of Damascus, Syria from OpenStreetMap to IMG. I'm interested in using find places function in MapSource, and this is the command that I've used to generate the map: java -jar mkgmap.jar --code-page=1256 --route --index --tdbfile --nsis --country-name=SYRIA --country-abbr=SYR map.osm I noticed that most of Arabic labeled places can't be found in MapSource. So I opened the file osmmap_mdr.img using text editor to inspect it, and I found that almost none of the Arabic names is written correctly. For example, the Arabic name of Damascus International Airport is: مطار دمشق الدولي but it is written in the file osmmap_mdr.img as: أطار دأشق اءدئءح Now using the incorrect name above, I managed to found the airport on MapSource. I hope you will fix this problem in the next releases.. Regards, Hosam
On Sun, Jun 05, 2011 at 11:56:40PM +0300, Hosam Arnous wrote:
For example, the Arabic name of Damascus International Airport is: مطار دمشق الدولي but it is written in the file osmmap_mdr.img as: أطار دأشق اءدئءح
Do you have the OpenStreetMap node or way ID of the airport? Or the map coordinates (lat/lon)? Is the name written correctly in the map data? Best regards, Marko
The airport POI is http://www.openstreetmap.org/browse/node/26608948 and the name is given as مطار دمشق الدولي Which is how it should be written. Marko Mäkelä (marko.makela@iki.fi) wrote:
On Sun, Jun 05, 2011 at 11:56:40PM +0300, Hosam Arnous wrote:
For example, the Arabic name of Damascus International Airport is: مطار دمشق الدولي but it is written in the file osmmap_mdr.img as: أطار دأشق اءدئءح
Do you have the OpenStreetMap node or way ID of the airport? Or the map coordinates (lat/lon)? Is the name written correctly in the map data?
Best regards,
Marko _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- Charlie
Marko Mäkelä (marko.makela@iki.fi) wrote:
On Sun, Jun 05, 2011 at 11:56:40PM +0300, Hosam Arnous wrote:
For example, the Arabic name of Damascus International Airport is: مطار دمشق الدولي but it is written in the file osmmap_mdr.img as: أطار دأشق اءدئءح
Do you have the OpenStreetMap node or way ID of the airport? Or the map coordinates (lat/lon)? Is the name written correctly in the map data?
All the Arabic names are written correctly on the map. I only gave you an example for the corruption that happens to the Arabic names. I believe there is a bug with the index created by mkgmap. Regards, Hosam -- View this message in context: http://gis.638310.n2.nabble.com/Index-with-Arabic-names-tp6443180p6444909.ht... Sent from the Mkgmap Development mailing list archive at Nabble.com.
Hi
For example, the Arabic name of Damascus International Airport is: مطار دمشق الدولي but it is written in the file osmmap_mdr.img as: أطار دأشق اءدئءح
Yes, you are right, thanks for reporting it. When names are read out of the .img file, the code page is ignored. I'm in the middle of making changes to the character set code so I've attached a new version of the patch which includes a fix. If you just want to try the fix separately The relevant part of the patch is: @@ -89,6 +100,7 @@ funcs.setEncoder(new Simple8Encoder()); } else { funcs.setEncodingType(ENCODING_FORMAT9); + funcs.setDecoder(new AnyCharsetDecoder(charset)); funcs.setEncoder(new AnyCharsetEncoder(charset)); guessCodepage(funcs, charset); } The patch, is the second version of the previous translit_first patch (which was for problems first noted in Greek names). There is still the problem that the names will not be sorted quite correctly, although they should still be findable, as they will be sorted consistently. ..Steve
Hi...
When names are read out of the .img file, the code page is ignored. I'm in the middle of making changes to the character set code so I've attached a new version of the patch which includes a fix.
If you just want to try the fix separately The relevant part of the patch is:
@@ -89,6 +100,7 @@ funcs.setEncoder(new Simple8Encoder()); } else { funcs.setEncodingType(ENCODING_FORMAT9); + funcs.setDecoder(new AnyCharsetDecoder(charset)); funcs.setEncoder(new AnyCharsetEncoder(charset)); guessCodepage(funcs, charset); }
The patch, is the second version of the previous translit_first patch (which was for problems first noted in Greek names).
Thanks for your quick answer... I'm sorry to tell you that the patch didn't resolve the issue. I've tried the patch on both the version r1955 and the svn trunk. Regards, Hosam
Hi
Thanks for your quick answer...
I'm sorry to tell you that the patch didn't resolve the issue. I've tried the patch on both the version r1955 and the svn trunk.
You mean it was unchanged or wrong in a different way? Certainly when I tried on a small example I saw the same incorrect string before the patch and the correct string as given in your email afterwards. I've uploaded my built jar in case I missed something in the patch, could you try it please? It is at URL: http://files.mkgmap.org.uk/download/27/mkgmap.jar Thanks ..Steve
Hi..
You mean it was unchanged or wrong in a different way?
Yes I mean the problem was unchanged.
Certainly when I tried on a small example I saw the same incorrect string before the patch and the correct string as given in your email afterwards.
Using the patch translit_first2.patch, I still can't find Arabic names using the correct letters.
I've uploaded my built jar in case I missed something in the patch, could you try it please?
The jar throw an exception whenever I pass the option --code-page=1256 Regards, Hosam
Hi
Yes I mean the problem was unchanged.
Well I think that you cannot be running the same patch as me then as there is certainly a difference in what goes into the osmmap_mdr.img file. I may still not work in mapsource, but I have no way of knowing that without some help.
The jar throw an exception whenever I pass the option --code-page=1256
OK I've re-uploaded it, please try it now. Thanks ..Steve
Hi
OK I've re-uploaded it, please try it now.
The old link for the jar is dead now.
Sorry, the new link is: http://files.mkgmap.org.uk/download/29/mkgmap.jar ..Steve
Hi...
Sorry, the new link is: http://files.mkgmap.org.uk/download/29/mkgmap.jar
I've tried the new jar. The Arabic names are still unfindable on MapSource using the correct letters. However, I believe you're making a progress to resolve the issue. Previously using only the patch translit_first2.patch, the Arabic names were still corrupted in the file osmmap_mdr.img. Now, the Arabic names are correct in the file osmmap_mdr.img. Regards, Hosam
Hi
names were still corrupted in the file osmmap_mdr.img. Now, the Arabic names are correct in the file osmmap_mdr.img.
OK that is good. In your first post you said that you could find the airport using the incorrect name as found in the mdr file. Now that the correct name is in the file it is strange that you cannot find it by that name. Can you find any feature at all? Do the features show up in the drop down box, when you type a single letter? If so are the names sorted correctly? ..Steve
Hi.. Sorry for the late answer. The regime here is shutting down the Internet from many cities and towns including my city.
In your first post you said that you could find the airport using the incorrect name as found in the mdr file. Now that the correct name is in the file it is strange that you cannot find it by that name.
That's true.
Can you find any feature at all? Do the features show up in the drop down box, when you type a single letter? If so are the names sorted correctly?
Non of the features can be found regardless of its category type. Only some features show up when I type a single letter, even they don't include that letter!! And when I hit "Find" button, it gives me "No items found". Moreover, I notice that the Arabic names are still corrupted in the file osmmap_mdr.img with the version r1962. Wasn't the patch applied in this version? Regards, Hosam
Hi Hosam I have now made a lot of progress and have something that works as far as I can tell. I can now hit space, select one of the displayed names and it is found. There are two patches attached. The first is the main problem I think. Previously there were two equal and opposite errors when reading the labels out of the .img files and then writing to the index. This was deliberate and works as far as getting the correct characters. However it meant that the labels were completly mis-sorted and sorting is essential for searching. The second patch includes a sort table for cp1256 and implements the character expansion feature in the SRT compiler, this is where a character like æ can be made to sort as if it were the two separate characters "ae". Previously the feature was hardwired in as I didn't know how it worked, and it may therefore have been causing problems. ..Steve
Hi Steve,
I have now made a lot of progress and have something that works as far as I can tell.
I can now hit space, select one of the displayed names and it is found.
I've tried the patches against r1969 and compiled the jar by myself. I confirm that the Arabic names are now findable in the "Find Places" window, but they are still unfindable using the correct letters in the "Find Nearest Places" window.
Also I forgot to add to the previous email that I have a ready compiled jar at http://files.mkgmap.org.uk/download/30/mkgmap.jar
Using this jar doesn't produce the same result. When I hit space in the "Find Places" window, Arabic names appear as questions marks. Regards, Hosam
Hosam Arnous <hariov@gmail.com> wrote:
Hi Steve,
I have now made a lot of progress and have something that works as far as I can tell.
I can now hit space, select one of the displayed names and it is found.
I've tried the patches against r1969 and compiled the jar by myself. I confirm that the Arabic names are now findable in the "Find Places" window, but they are still unfindable using the correct letters in the "Find Nearest Places" window.
Also I forgot to add to the previous email that I have a ready compiled jar at http://files.mkgmap.org.uk/download/30/mkgmap.jar
Using this jar doesn't produce the same result. When I hit space in the "Find Places" window, Arabic names appear as questions marks.
Regards, Hosam _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
That's great. No idea why find nearest does not work. It does as far as I can tell. Does typing in a single letter do anything useful? Exactly how does it fail? I'll remove the jar if it doesn't work. ..Steve
I'll remove the jar if it doesn't work.
A working jar now uploaded at: http://files.mkgmap.org.uk/download/31/mkgmap.jar ..Steve
Hi Steve, I'm currently using an old version of MapSource which is v6.15.11. I hope that doesn't affect on my findings.
That's great. No idea why find nearest does not work. It does as far as I can tell. Does typing in a single letter do anything useful? Exactly how does it fail?
Searching at "Find Nearest Places" fails exactly as my first post. For example, if I want to search for any place contains the word مطار I should type أطار I think "Find Nearest Places" has noting to do with MDR file because when I remove MDR key from the registry, MapSource doesn't disable "Find Nearest Places". Is that right? Regards, Hosam
Hi Hosam
Searching at "Find Nearest Places" fails exactly as my first post. For example, if I want to search for any place contains the word مطار I should type أطار
I think "Find Nearest Places" has noting to do with MDR file because when I remove MDR key from the registry, MapSource doesn't disable "Find Nearest Places". Is that right?
That is true, the index is not needed for Find Nearest. Which means that the problem must be in the individual tiles. If I create a single tile with only one POI with the name "مطار" then as far as I can tell the same string is placed into the .img file, I am not seeing the obvious corruption of letters that I could see in the other case. (I'm using the options --code-page=1256 --route --index --tdbfile) Unfortunately I don't know enough to go further with this, without more help about what might be wrong. ..Steve
Hi Steve,
That is true, the index is not needed for Find Nearest.
Which means that the problem must be in the individual tiles. If I create a single tile with only one POI with the name "مطار" then as far as I can tell the same string is placed into the .img file, I am not seeing the obvious corruption of letters that I could see in the other case.
(I'm using the options --code-page=1256 --route --index --tdbfile)
That's true, all the Arabic names in the .img file are correct. Is it possible that a similar bug exists in the code that generates .img? Because before the last two patches, the index was corrupted even that the names were correct in the mdr file.
Unfortunately I don't know enough to go further with this, without more help about what might be wrong.
Can I help you with anything? Regards, Hosam
Steve,
Yes I mean the problem was unchanged. Well I think that you cannot be running the same patch as me then as there is certainly a difference in what goes into the osmmap_mdr.img
It may still not work in mapsource, but I have no way of knowing that without some help.
I also still have the Mapsource problem that e.g. a subset of 4 specific tiles around the black sea, touching contries like Ukraine, Romania, Moldavia, Turkey and Greece, do still break with a Mapsource Error MDR_TRIM_SC.CXX-808-6.16.3.0. The tiles are not all bordering. The crazy thing is, that when I generate the index files _only_ with those 4 tiles, they work OK and I can upload a map and index to the GPS. Mapsource breaks only, when I select the same 4 tiles within the context of the whole Europe map index files generated for all 787 tiles. I'm using the Geofabrik Europe extract ... If I select only the western Europe countries I can upload tiles up to 4 GB without a problem. This means to me, that there still is a problem with cyrillic and maybe also arabic encodings -- see Hosam's emails, and perhaps other encodings, too. Even with the translit_first2 patch! Any idea what testcase I could try or how I could nail down the problem? Use smaller tiles? Peter
Peter
This means to me, that there still is a problem with cyrillic and maybe also arabic encodings -- see Hosam's emails, and perhaps other encodings, too. Even with the translit_first2 patch!
Any idea what testcase I could try or how I could nail down the problem? Use smaller tiles?
Its worth trying out the patches that I just posted for arabic. I don't expect them to make much difference when all compiled with --latin1 but you never know... Also I forgot to add to the previous email that I have a ready compiled jar at http://files.mkgmap.org.uk/download/30/mkgmap.jar ..Steve
participants (5)
-
charlie＠cferrero.net -
Hosam Arnous -
Marko Mäkelä -
Peter Lerner -
Steve Ratcliffe