mkgmap: problems with encoding file used with option --license-file
Hello together, I'm using makemap (r3696) on Windows (non UTF-8 locale) and on Linux (UTF-8 locale) and got very strange and platform dependant behaviour. So I started to use the option -Dfile.encoding=utf-8 in the java call of mkgmap to have at least the same, still strange and different behaviour on both platforms. Here the facts: * I have special characters (Latin1) in the license file: äöüéè ... * If I build a unicode map and the license file is properly encoded in Unicode, Basecamp shows at the end scrambled characters * I'm able to get a unicode map with correct license info (no scrambled chars) only on Windows (non-utf8 environment) without calling mkgmap with -Dfile.encoding=utf-8 and some additional cheats * If I use mkgmap to create a gmapsupp.img it again looks different, bit better but not good. I know these facts are a bit unspecific, but I've really tested out quite a lot and no combination / tries did help to solve the problems. And before investing even more time and get completely lost I'll ask for help. Therefore: did I miss something essential or are the parameters --license-file and --copyright-file (I use the same file for both parameters) handled specially/different than other source files for mkgmap (no/different encoding support ?). We never had any problems with utf8 encoded style files or typ source files, all went always very smooth as long as everything was utf-8...but with the license files I got stuck now. Thanks in advance for hints / solutions / confirmations. Cheers Patrik
Hi
Therefore: did I miss something essential or are the parameters --license-file and --copyright-file (I use the same file for both parameters) handled specially/different than other source files for mkgmap (no/different encoding support ?).
Yes, you are correct those two files appear to be opened using the system default character encoding, and not utf-8 as is the case for most other files. I'll investigate further tomorrow - there may be more involved to write it to the img file correctly. Best wishes ..Steve
Thanks Steve, Good to have that different behaviour explained/confirmed. If there's anything to be tested, just let me know.... I would just need patched/updated mkgmap as I'm not the Java Crack in terms of compilations... Best wishes too. Patrik On 25.11.2016 23:36, Steve Ratcliffe wrote:
Hi
Therefore: did I miss something essential or are the parameters --license-file and --copyright-file (I use the same file for both parameters) handled specially/different than other source files for mkgmap (no/different encoding support ?).
Yes, you are correct those two files appear to be opened using the system default character encoding, and not utf-8 as is the case for most other files.
I'll investigate further tomorrow - there may be more involved to write it to the img file correctly.
Best wishes ..Steve _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Patrik
If there's anything to be tested, just let me know.... I would just need patched/updated mkgmap as I'm not the Java Crack in terms of compilations...
This simply opens the file for read using utf-8. Patch attached and pre-compiled jar file is here: http://files.mkgmap.org.uk/download/315/mkgmap.jar Let me know what happens. ..Steve
Steve, Thanks for the updated version. It looks different and better, but not yet completely good yet: * it's consistent now (tested on Windows only for the moment), independant if I start with or without -Dfile.encoding=utf * If I create a Latin1 map (codepage 1252) the licens looks good in BaseCamp and when checking the img files withGMapTool * If I create a unicode map (codepage 65001) it looks good when checking the image files with GMapTool * Checking the same unicode map (installed img files) in BaseCamp, the äöü stuff is scrambled again * Converting the same unicode map into BaseCamp format (via jmc_cli) and checking in BaseCamp, äöü stuff is also scrambled Something's still sort of strange, not working as expected.... Another important thing to note about the release you created: Suddenly the cfg file has to be utf-8 also. I have some special characters in some comments line there and mkgmap complained, bailed out. Not sure if this is unintended or intended.... but it might be 'dangerous' for other users, I personally do not care, I can create the cfg file without any problems in UTF-8 on the fly as it's created automatically... ;-) With the old version I could have my cfg file in cp1252 on both linux (UTF-8 Environment) and Windows (non-utf-8). Cheers Patrik On 26.11.2016 20:44, Steve Ratcliffe wrote:
Hi Patrik
If there's anything to be tested, just let me know.... I would just need patched/updated mkgmap as I'm not the Java Crack in terms of compilations...
This simply opens the file for read using utf-8.
Patch attached and pre-compiled jar file is here:
http://files.mkgmap.org.uk/download/315/mkgmap.jar
Let me know what happens.
..Steve
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
On 26/11/16 22:37, Patrik Brunner wrote:
Another important thing to note about the release you created: Suddenly the cfg file has to be utf-8 also. I have some special
What do you mean by the cfg file? A -c or --read-config option? I didn't change anything there! That is however another place where it uses the system encoding and not utf-8. ..Steve
Steve, mkgmap ist started with (more or less): java -Xmx <heapSize> -jar mkgmap.jar -c <configfile> --check-styles I was talking about the config file handed over to mkmap.jar. Probably this behaviour was caused also when starting mkgmap (more precisely java) with the option -Dfile.encoding=UTF-8 Cheers Patrik On 27.11.2016 21:59, Steve Ratcliffe wrote:
On 26/11/16 22:37, Patrik Brunner wrote:
Another important thing to note about the release you created: Suddenly the cfg file has to be utf-8 also. I have some special
What do you mean by the cfg file? A -c or --read-config option?
I didn't change anything there!
That is however another place where it uses the system encoding and not utf-8.
..Steve _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Patrik
It looks different and better, but not yet completely good yet:
* it's consistent now (tested on Windows only for the moment), independant if I start with or without -Dfile.encoding=utf * If I create a Latin1 map (codepage 1252) the licens looks good in BaseCamp and when checking the img files withGMapTool * If I create a unicode map (codepage 65001) it looks good when checking the image files with GMapTool * Checking the same unicode map (installed img files) in BaseCamp, the äöü stuff is scrambled again * Converting the same unicode map into BaseCamp format (via jmc_cli) and checking in BaseCamp, äöü stuff is also scrambled
Something's still sort of strange, not working as expected....
That could be a problem - if GMapTool displays it correctly, but basecamp doesn't, it could be that there is something that we don't know. Perhaps the copyright and licence needs to be in cp 1252 regardless of map code page. I'll investigate some more. Cheers ..Steve
Steve, where you able to investigate some more ? Is there another build I can/have to try it with ? Cheers Patrik On 27.11.2016 22:05, Steve Ratcliffe wrote:
Hi Patrik
It looks different and better, but not yet completely good yet:
* it's consistent now (tested on Windows only for the moment), independant if I start with or without -Dfile.encoding=utf * If I create a Latin1 map (codepage 1252) the licens looks good in BaseCamp and when checking the img files withGMapTool * If I create a unicode map (codepage 65001) it looks good when checking the image files with GMapTool * Checking the same unicode map (installed img files) in BaseCamp, the äöü stuff is scrambled again * Converting the same unicode map into BaseCamp format (via jmc_cli) and checking in BaseCamp, äöü stuff is also scrambled
Something's still sort of strange, not working as expected....
That could be a problem - if GMapTool displays it correctly, but basecamp doesn't, it could be that there is something that we don't know.
Perhaps the copyright and licence needs to be in cp 1252 regardless of map code page. I'll investigate some more.
Cheers ..Steve _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Patrik
where you able to investigate some more ? Is there another build I can/have to try it with ?
As it happens I just discovered something today. There is no encoding done for the copyright info before the TRE section data. Patch attached to fix this and pre-made jar file at: http://files.mkgmap.org.uk/download/317/mkgmap.jar Cheers, ..Steve
Steve, great news. Just a very quick feedback without having done much testing... I've just exchanged mkgmap.jar and built a unicode and a cp1252 map. If I didn't mess anything up on my end I would guess: * it looks like you found the correct hook * when building unicode map either copyright-file or license-file result is now correct (which one will show further testing tomorrow, assuming license-file), the other one ist still 'screwed up' (allthough I'm not sure if I really have to/want to set both) * non-unicode map (cp1252) still seems to build fine (license/copyright wise) * One thing to mention which caught my attention: it looks like I now have only capital letters when checking with BaseCamp... before I had also lower case letters (at least in one of the two options copyright-file and license-file).... needs more investigations from my end also..... I'll do proper testing (hopefully) tomorrow and will let you know the outcome. I've to check the result also on the Garmin Devices directly (gmapsupp.im)... hopefully the result is still the same as before... Thanks again. Cheers Patrik On 03.12.2016 21:07, Steve Ratcliffe wrote:
Hi Patrik
where you able to investigate some more ? Is there another build I can/have to try it with ?
As it happens I just discovered something today. There is no encoding done for the copyright info before the TRE section data.
Patch attached to fix this and pre-made jar file at: http://files.mkgmap.org.uk/download/317/mkgmap.jar
Cheers, ..Steve
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Patrik
when building unicode map either copyright-file or license-file result is now correct (which one will show further testing tomorrow, assuming license-file), the other one ist still 'screwed up'
Yes, the fix applied to licence file. It does not affect copyright file which appeared to be OK, unless there is something unknown about it.
One thing to mention which caught my attention: it looks like I now have only capital letters when checking with BaseCamp... before I had also lower case letters (at least in one of the two options copyright-file and license-file).... needs more investigations from my end also.....
Ahh yes that will be true, I'll keep that in mind. ..Steve
Steve, as promised here the outcome from the (hopefully) proper testing... Tests performed with maps created on Windows: * licence-file option: Check in Basecamp (Windows): now ok if I build cp1252 or unicode map, thumbs up ! * lower-case option: if this option is set then I get proper capital and lower-case letters in shown license, if not set the it shows all capital letters... I haven't found any other differences in the map, Basecamp shows proper names on map. * copyright-file option: Check in BaseCamp (Windows): still shows scrambled characters in unicode map but also still ok in cp1252 maps Additional tests performed with map created on Linux and final gmap checked on BaseCamp (Windows): * created unicode map and cp1252 map * on cp1252 map: both copyright-file and license-file output looks good (as expected) * on unicode map: consistent behaviour as on windows: license-file output correct, copyright-file output scrambled That leaves up two things/questions: * I think copyright-file needs to be fixed also as this is the only one being displayed on the GPS Device, correct ? * question: the description of the option --lower-case says that lower case characters aren't displayed properly on most devices. On mine (etrex30, oregon 450t, monterra) I couldn't find any difference in the map, this option seems to affect only license and copyright files ? Do I miss something ? Is it dangerous to set this option for having a nice license as I might loose support for some devices ? Cheers Patrik On 03.12.2016 22:51, Patrik Brunner wrote:
Steve, great news.
Just a very quick feedback without having done much testing... I've just exchanged mkgmap.jar and built a unicode and a cp1252 map. If I didn't mess anything up on my end I would guess:
* it looks like you found the correct hook * when building unicode map either copyright-file or license-file result is now correct (which one will show further testing tomorrow, assuming license-file), the other one ist still 'screwed up' (allthough I'm not sure if I really have to/want to set both) * non-unicode map (cp1252) still seems to build fine (license/copyright wise) * One thing to mention which caught my attention: it looks like I now have only capital letters when checking with BaseCamp... before I had also lower case letters (at least in one of the two options copyright-file and license-file).... needs more investigations from my end also.....
I'll do proper testing (hopefully) tomorrow and will let you know the outcome. I've to check the result also on the Garmin Devices directly (gmapsupp.im)... hopefully the result is still the same as before...
Thanks again.
Cheers Patrik
On 03.12.2016 21:07, Steve Ratcliffe wrote:
Hi Patrik
where you able to investigate some more ? Is there another build I can/have to try it with ?
As it happens I just discovered something today. There is no encoding done for the copyright info before the TRE section data.
Patch attached to fix this and pre-made jar file at: http://files.mkgmap.org.uk/download/317/mkgmap.jar
Cheers, ..Steve
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Patrik
as promised here the outcome from the (hopefully) proper testing...
Great, so we only seem to have a remaining problem with unicode and the copyright file.
That leaves up two things/questions:
* I think copyright-file needs to be fixed also as this is the only one being displayed on the GPS Device, correct ?
OK this is strange as the copyright file is just converted into normal labels just like everything else on the map like street names etc. It was the licence file that was different. I will take another look as I may have missed something there. Can you tell me what characters you do see instead?
* question: the description of the option --lower-case says that lower case characters aren't displayed properly on most devices. On mine (etrex30, oregon 450t, monterra) I couldn't find any difference in the map, this option seems to affect only license and copyright files ? Do I miss something ? Is it dangerous to set this option for having a nice license as I might loose support for some devices ?
Yes, the --lower-case should affect the map; it leaves names with the same case as they appear in OSM input file in the map. That is as long as you are using one code-page option, without that you get an ascii uppercase only format. That comment in the help for --lower-case is out of date now. When it was written, most or all hand held devices then available could not display rotated lowercase text. Horizontal text was OK. Maybe even earlier devices couldn't display lower case text at all, I'm not sure. (eg. See the image in this post http://www.mkgmap.org.uk/pipermail/mkgmap-dev/2010q1/007667.html) I will try to preserve the case of the text, after all if you want or need the licence text to be in upper case you can just write it that way in the file. ..Steve
Steve, here some answers for below questions: -> Can you tell me what characters you do see instead? For unicode map option copyright-file I see characters that look like black diamonds with a question mark in it (picture attached, hope it comes through). -> lower-case option I've double checked with older maps I was definitely building without the --lower-case option and the map always showed proper case (capital and lower case) names, preserving OSM style on my Garmin devices (at least for the last few months... didn't/couldn't check back more). Looking in BaseCamp at the license/copyright information I can say that we had part in capitals and part in mixed cases, assuming that copyright-message (I was using the message, not the copyright-file option earlier) was in capital letters only. The license-file part was before always in mixed cases as it should be. And to make the statment complete: the codepage option was always used and set. Yes, preserving case for license-file and copyright-file (as well as copyright-message) would be great.... I would prefer not having to change the option lower-case while building the map, just to make sure we don't screw up stuff for old devices out there... ;-) Looks like we're getting nearer to the real problem and to the solution.... great work, many thanks. Cheers Patrik On 04.12.2016 14:55, Steve Ratcliffe wrote:
Hi Patrik
as promised here the outcome from the (hopefully) proper testing...
Great, so we only seem to have a remaining problem with unicode and the copyright file.
That leaves up two things/questions:
* I think copyright-file needs to be fixed also as this is the only one being displayed on the GPS Device, correct ?
OK this is strange as the copyright file is just converted into normal labels just like everything else on the map like street names etc. It was the licence file that was different.
I will take another look as I may have missed something there.
Can you tell me what characters you do see instead?
* question: the description of the option --lower-case says that lower case characters aren't displayed properly on most devices. On mine (etrex30, oregon 450t, monterra) I couldn't find any difference in the map, this option seems to affect only license and copyright files ? Do I miss something ? Is it dangerous to set this option for having a nice license as I might loose support for some devices ?
Yes, the --lower-case should affect the map; it leaves names with the same case as they appear in OSM input file in the map. That is as long as you are using one code-page option, without that you get an ascii uppercase only format.
That comment in the help for --lower-case is out of date now. When it was written, most or all hand held devices then available could not display rotated lowercase text. Horizontal text was OK. Maybe even earlier devices couldn't display lower case text at all, I'm not sure.
(eg. See the image in this post http://www.mkgmap.org.uk/pipermail/mkgmap-dev/2010q1/007667.html)
I will try to preserve the case of the text, after all if you want or need the licence text to be in upper case you can just write it that way in the file.
..Steve _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Patrik
-> Can you tell me what characters you do see instead? For unicode map option copyright-file I see characters that look like black diamonds with a question mark in it (picture attached, hope it comes through).
OK thanks that helps I will post something to try later.
-> lower-case option I've double checked with older maps I was definitely building without the --lower-case option and the map always showed proper case (capital and lower case) names, preserving OSM style on my Garmin devices (at least for the last few months... didn't/couldn't check back more). Looking in BaseCamp at the license/copyright information I can say that we had part in capitals and part in mixed cases, assuming that copyright-message (I was using the message, not the copyright-file option earlier) was in capital letters only. The license-file part was before always in mixed cases as it should be. And to make the statment complete: the codepage option was always used and set.
In MapSource etc it will display in upper and lower case even though the label in the map is all in uppercase. It just capitalises the first letter of each work and lower cases the rest. So "OSM Road" would be stored as "OSM ROAD" and displayed as "Osm Road". All I can do is control what is actually put into the map, how any particular device displays it is up to the device. ..Steve
Steve, that sounds like a plan... ;-) Regarding lower-case option again: I found old screenprints from one of my devices from early last year, showing all capital letters for streetnames and similar... sometime last year it seems to have changed. Newer screenprints from the device always show proper case.... I can't check how it was in earlier days regarding license and copyright information.... no real old files around anymore.... constantly running out of space here... ;-) I guess it's fine to have it in 'proper' case as written in the files inside the license and copyright information, looks nicer. And the actual devices (what I could test) do definitely support it. And I definitely prefer to unset the 'lower-case' option again if all runs fine without it. Thanks Patrik On 04.12.2016 17:31, Steve Ratcliffe wrote:
Patrik
-> Can you tell me what characters you do see instead? For unicode map option copyright-file I see characters that look like black diamonds with a question mark in it (picture attached, hope it comes through).
OK thanks that helps I will post something to try later.
-> lower-case option I've double checked with older maps I was definitely building without the --lower-case option and the map always showed proper case (capital and lower case) names, preserving OSM style on my Garmin devices (at least for the last few months... didn't/couldn't check back more). Looking in BaseCamp at the license/copyright information I can say that we had part in capitals and part in mixed cases, assuming that copyright-message (I was using the message, not the copyright-file option earlier) was in capital letters only. The license-file part was before always in mixed cases as it should be. And to make the statment complete: the codepage option was always used and set.
In MapSource etc it will display in upper and lower case even though the label in the map is all in uppercase. It just capitalises the first letter of each work and lower cases the rest. So "OSM Road" would be stored as "OSM ROAD" and displayed as "Osm Road".
All I can do is control what is actually put into the map, how any particular device displays it is up to the device.
..Steve _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Patrik OK, well try this out: http://files.mkgmap.org.uk/download/318/mkgmap.jar This writes the copyright file text as labels encoded as cp1252 regardless of what the actual code-page in use is. That is just wierd but may work. Also possible that only ascii characters are allowed, but see how that goes. ..Steve
no luck.... think it got worse as there is now suddenly text in (I assume) the copyright section which belongs to the map: 'Centre de Conf....). Here just copy paste of the text inside the Copyright Dialog Box of Basecamp: (c) Contour data: U.S. Geological Survey or J. de Ferranti (free for research and private use) (c) Map data: OSM contributors ������ (ODbl) (c) Map: FZK project äöüéèê (free for research and private use) (c) Map data: OSM contributors äöüéèê (ODbl) (c) Contour data: U.S. Geological Survey or J. de Ferranti (free for research and private use) (c) Map: FZK project ������ (free for research and private use) Centre de Conf�rence et de Congr�s International (Salle municipale) centrale de Vianden (Station �lectrique) The last two lines are what does not belong to it.... obviously. Cheers Patrik On 04.12.2016 18:07, Steve Ratcliffe wrote:
Hi Patrik
OK, well try this out: http://files.mkgmap.org.uk/download/318/mkgmap.jar
This writes the copyright file text as labels encoded as cp1252 regardless of what the actual code-page in use is. That is just wierd but may work. Also possible that only ascii characters are allowed, but see how that goes.
..Steve
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
On 04/12/16 17:16, Patrik Brunner wrote:
no luck.... think it got worse as there is now suddenly text in (I assume) the copyright section which belongs to the map: 'Centre de Conf....). Here just copy paste of the text inside the Copyright Dialog Box of Basecamp:
Hmm, what does it look like on the device? Cheers, ..Steve
When checked via BaseCamp while the eTrex or the oregon 450t is hooked up to my machine: * Scrambled Characters for the special characters.... shown only once (as the license-file thingy is missing on the device anyway) © (c) Contour data: U.S. Geological Survey or J. de Ferranti (free for research and private use) © (c) Map data: Osm contributors ������ (Odbl) © (c) Map: Fzk project ������ (free for research and private use) Directly on the device itself (tested both eTrex and oregon): * No additional stuff shown as in BaseCamp directly (the stuff that belongs to the map) * No special characters shown, looks like it cuts off everything on the line before the first special character. It looks more or less (put together manually): © (c) Map: Fzk project © (c) Map data: Osm contributors © (c) Contour data: U.S. Geological Survey or J. de Ferranti (free for research and private use) And just to clarify again: the source file for the the option license-file and for the option copyright-file is the same file, it's only one file. And this is, according to my editors, properly utf8.... :-( Strange stuff... ;-) Cheers Patrik On 04.12.2016 18:30, Steve Ratcliffe wrote:
On 04/12/16 17:16, Patrik Brunner wrote:
no luck.... think it got worse as there is now suddenly text in (I assume) the copyright section which belongs to the map: 'Centre de Conf....). Here just copy paste of the text inside the Copyright Dialog Box of Basecamp:
Hmm, what does it look like on the device?
Cheers, ..Steve _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
OK I've realised that I am looking in the wrong place. MapSource is looking at the TDB file. I've no idea where the device gets the info from. ..Steve
Additional Info regarding the 'leaked' data from the map, just noticed: Building a cp1252 map now shows also additional stuff in the BaseCamp Copyright Dialog Box: Centre de Conférence et de Congrès International (Salle municipale) And I think I have the reason for 'leaking' map data into copyright dialog box: As written in the mkgmap description the first line of the file handed over to copyright-file option is not shown on the device. Clever as I am I've just put an empty line in front of the three needed lines.... looks like this was not that clever, as it screws up the process additionally. But that, I would guess, a different construction area again, here the facts so far: * If I have either only copyright-file or only license-file activated with the empty line, nothing bad happens. * This leaking happens only if have have the file starting with an empty line handed over to both options. * A single space instead of a complete empty line does also not help. * Rolling back one release to the one you've sent me yesterday helps, the leaked information is gone. * Rolling back to the version of the 26th Nov (first one with build nr 3704) also does not create such data leak. ... just tried it out now with building cp1252 map as it happened here also. Cheers Patrik On 04.12.2016 18:47, Patrik Brunner wrote:
When checked via BaseCamp while the eTrex or the oregon 450t is hooked up to my machine:
* Scrambled Characters for the special characters.... shown only once (as the license-file thingy is missing on the device anyway)
© (c) Contour data: U.S. Geological Survey or J. de Ferranti (free for research and private use) © (c) Map data: Osm contributors ������ (Odbl) © (c) Map: Fzk project ������ (free for research and private use)
Directly on the device itself (tested both eTrex and oregon):
* No additional stuff shown as in BaseCamp directly (the stuff that belongs to the map) * No special characters shown, looks like it cuts off everything on the line before the first special character.
It looks more or less (put together manually):
© (c) Map: Fzk project © (c) Map data: Osm contributors © (c) Contour data: U.S. Geological Survey or J. de Ferranti (free for research and private use)
And just to clarify again: the source file for the the option license-file and for the option copyright-file is the same file, it's only one file. And this is, according to my editors, properly utf8.... :-( Strange stuff... ;-)
Cheers Patrik
On 04.12.2016 18:30, Steve Ratcliffe wrote:
On 04/12/16 17:16, Patrik Brunner wrote:
no luck.... think it got worse as there is now suddenly text in (I assume) the copyright section which belongs to the map: 'Centre de Conf....). Here just copy paste of the text inside the Copyright Dialog Box of Basecamp:
Hmm, what does it look like on the device?
Cheers, ..Steve _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Not sure about the leaking, but please just ignore the last jar file I sent out. Sorry about that. Steve On 4 December 2016 18:23:03 GMT+00:00, Patrik Brunner <patrik.brunner@gmx.net> wrote:
Additional Info regarding the 'leaked' data from the map, just noticed:
Building a cp1252 map now shows also additional stuff in the BaseCamp Copyright Dialog Box: Centre de Conférence et de Congrès International (Salle municipale)
And I think I have the reason for 'leaking' map data into copyright dialog box: As written in the mkgmap description the first line of the file handed over to copyright-file option is not shown on the device. Clever as I am I've just put an empty line in front of the three needed
lines.... looks like this was not that clever, as it screws up the process additionally.
But that, I would guess, a different construction area again, here the facts so far:
* If I have either only copyright-file or only license-file activated with the empty line, nothing bad happens. * This leaking happens only if have have the file starting with an empty line handed over to both options. * A single space instead of a complete empty line does also not help. * Rolling back one release to the one you've sent me yesterday helps, the leaked information is gone. * Rolling back to the version of the 26th Nov (first one with build nr 3704) also does not create such data leak.
... just tried it out now with building cp1252 map as it happened here also.
Cheers Patrik
On 04.12.2016 18:47, Patrik Brunner wrote:
When checked via BaseCamp while the eTrex or the oregon 450t is hooked up to my machine:
* Scrambled Characters for the special characters.... shown only once (as the license-file thingy is missing on the device anyway)
© (c) Contour data: U.S. Geological Survey or J. de Ferranti (free for research and private use) © (c) Map data: Osm contributors ������ (Odbl) © (c) Map: Fzk project ������ (free for research and private use)
Directly on the device itself (tested both eTrex and oregon):
* No additional stuff shown as in BaseCamp directly (the stuff that belongs to the map) * No special characters shown, looks like it cuts off everything on the line before the first special character.
It looks more or less (put together manually):
© (c) Map: Fzk project © (c) Map data: Osm contributors © (c) Contour data: U.S. Geological Survey or J. de Ferranti (free for research and private use)
And just to clarify again: the source file for the the option license-file and for the option copyright-file is the same file, it's
only one file. And this is, according to my editors, properly utf8.... :-( Strange stuff... ;-)
Cheers Patrik
On 04.12.2016 18:30, Steve Ratcliffe wrote:
On 04/12/16 17:16, Patrik Brunner wrote:
no luck.... think it got worse as there is now suddenly text in (I assume) the copyright section which belongs to the map: 'Centre de Conf....). Here just copy paste of the text inside the Copyright Dialog Box of Basecamp:
Hmm, what does it look like on the device?
Cheers, ..Steve _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
------------------------------------------------------------------------
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Patrik Please try this one: http://files.mkgmap.org.uk/download/319/mkgmap.jar The important file for this change is the tdb file. Previously the copyright information was always encoded with cp1252 whatever the declared code page was. So this makes a lot more sense, sorry about the last one as I wasn't thinking straight! This change does not affect anything on the device, if you used a gmapsupp created by mkgmap. ..Steve
Steve, I've just replaced the mkgmap.jar and started the process again.... unfortunately it's not running through and throws an exception when trying to compile the TYP files (independant if I try to build unicode things or cp1252 things): java -Xmx1536M -jar D:/fzk/develop/fzk-mde-garmin/Freizeitkarte-Entwicklung/tools/mkgmap/mkgmap.jar --max-jobs=2 --product-id=1 --family-id=6442 contrast.txt Time started: Mon Dec 05 20:01:08 CET 2016 Number of MapFailedExceptions: 0 Exception in thread "main" java.nio.charset.UnsupportedCharsetException: cp0 at java.nio.charset.Charset.forName(Unknown Source) at uk.me.parabola.imgfmt.app.labelenc.AnyCharsetDecoder.<init>(AnyCharsetDecoder.java:27) at uk.me.parabola.imgfmt.app.labelenc.CodeFunctions.createEncoderForLBL(CodeFunctions.java:104) at uk.me.parabola.imgfmt.app.labelenc.CodeFunctions.createEncoderForLBL(CodeFunctions.java:127) at uk.me.parabola.tdbfmt.Block.<init>(Block.java:56) at uk.me.parabola.tdbfmt.TdbFile.write(TdbFile.java:155) at uk.me.parabola.mkgmap.combiners.TdbBuilder.writeTdbFile(TdbBuilder.java:190) at uk.me.parabola.mkgmap.combiners.TdbBuilder.onFinish(TdbBuilder.java:182) at uk.me.parabola.mkgmap.main.Main.endOptions(Main.java:610) at uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.java:128) at uk.me.parabola.mkgmap.main.Main.mainStart(Main.java:135) at uk.me.parabola.mkgmap.main.Main.main(Main.java:106) Using the new mkgmap.jar only for the real build process runs at least through, but the result ist just different but still scrambled: For unicode map: still scrambled, just differently (c) Contour data: U.S. Geological Survey or J. de Ferranti (free for research and private use) (c) Map data: OSM contributors äöüéèê (ODbl) (c) Map: FZK project äöüéèê (free for research and private use) (c) Map: FZK project ￃﾤￃﾶￃﾼￃﾩￃﾨￃﾪ (free for research and private use) (c) Map data: OSM contributors ￃﾤￃﾶￃﾼￃﾩￃﾨￃﾪ (ODbl) (c) Contour data: U.S. Geological Survey or J. de Ferranti (free for research and private use) New Problem: cp1252 map scrambled now too: (c) Contour data: U.S. Geological Survey or J. de Ferranti (free for research and private use) (c) Map data: OSM contributors äöüéèê (ODbl) (c) Map: FZK project ¦??-|| (free for research and private use) (c) Map data: OSM contributors ¦??-|| (ODbl) (c) Contour data: U.S. Geological Survey or J. de Ferranti (free for research and private use) (c) Map: FZK project äöüéèê (free for research and private use) Cheers Patrik On 04.12.2016 22:09, Steve Ratcliffe wrote:
Hi Patrik
Please try this one: http://files.mkgmap.org.uk/download/319/mkgmap.jar
The important file for this change is the tdb file. Previously the copyright information was always encoded with cp1252 whatever the declared code page was.
So this makes a lot more sense, sorry about the last one as I wasn't thinking straight!
This change does not affect anything on the device, if you used a gmapsupp created by mkgmap.
..Steve _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Patrik
I've just replaced the mkgmap.jar and started the process again.... unfortunately it's not running through and throws an exception when trying to compile the TYP files (independant if I try to build unicode things or cp1252 things): java -Xmx1536M -jar D:/fzk/develop/fzk-mde-garmin/Freizeitkarte-Entwicklung/tools/mkgmap/mkgmap.jar --max-jobs=2 --product-id=1 --family-id=6442 contrast.txt Time started: Mon Dec 05 20:01:08 CET 2016 Number of MapFailedExceptions: 0 Exception in thread "main" java.nio.charset.UnsupportedCharsetException: cp0 at java.nio.charset.Charset.forName(Unknown Source)
OK, so that is a bug, ... but you do need one of the code-page options the same one that is used to create the img files. ..Steve
confirmed... adding the proper --code-page=something works also for compiling TYP files. And I've just noticed that creating the gmapsupp file fails with the latest patch for unicode licenses with the the same exception if I do not add the codepage.... Adding --code-page=anything there does not help, unfortunately for the gmapsupp creation. Regards Patrik On 05.12.2016 22:36, Steve Ratcliffe wrote:
Hi Patrik
I've just replaced the mkgmap.jar and started the process again.... unfortunately it's not running through and throws an exception when trying to compile the TYP files (independant if I try to build unicode things or cp1252 things): java -Xmx1536M -jar D:/fzk/develop/fzk-mde-garmin/Freizeitkarte-Entwicklung/tools/mkgmap/mkgmap.jar
--max-jobs=2 --product-id=1 --family-id=6442 contrast.txt Time started: Mon Dec 05 20:01:08 CET 2016 Number of MapFailedExceptions: 0 Exception in thread "main" java.nio.charset.UnsupportedCharsetException: cp0 at java.nio.charset.Charset.forName(Unknown Source)
OK, so that is a bug, ... but you do need one of the code-page options the same one that is used to create the img files.
..Steve _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
I've started a branch for this as the problems I'm discovering are increasing rapidly! So in addition to the copyright and licence file issues that started the thread, we also do not correctly deal with unicode in the TDB file and the MPS file (inside a gmapsupp). Some visible problems include: * --series-name with unicode characters not displayed correctly for a unicode map. * --area-name and unicode characters. * --description with non-ascii characters can overflow the header and completely corrupt the map. Eg try --description=Mẏ Ðésçriptioñ Branch is called 'file-encoding' and you can download the latest build from the branch build section of the mkgmap download page. ..Steve
sorry for having opened Pandora's Box... ;-) Consider me as a good and willing tester.... I'll just fetch the new branch to my local repo. Cheers Patrik On 11.12.2016 14:39, Steve Ratcliffe wrote:
I've started a branch for this as the problems I'm discovering are increasing rapidly!
So in addition to the copyright and licence file issues that started the thread, we also do not correctly deal with unicode in the TDB file and the MPS file (inside a gmapsupp).
Some visible problems include:
* --series-name with unicode characters not displayed correctly for a unicode map.
* --area-name and unicode characters.
* --description with non-ascii characters can overflow the header and completely corrupt the map. Eg try --description=Mẏ Ðésçriptioñ
Branch is called 'file-encoding' and you can download the latest build from the branch build section of the mkgmap download page.
..Steve _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Patrik
Consider me as a good and willing tester.... I'll just fetch the new branch to my local repo.
Thanks, but note that currently it is much less advanced than the previous patch. Just about to make a commit for the licence file soon. ..Steve
Hi The file-encoding branch now appears to work with both copyright and licence files and --unicode. I've checked with BaseCamp. Both are added in mixed case, regardless of if --lower-case is given. Use upper case letters in the licence file if that is what you actually want. Not tested with a gmapsupp, and there is probably still work to be done there. ..Steve
If I didn't do any local mistake then I still have a different behaviour than you. Tested with r3725 from file-encoding branch: Independant of building map with unicode (cp65001) or cp1252 I have following result: * utf8 src file is shown properly via option --copyright-file in BaseCamp Windows * same utf8 file is shown scrambled (at least the special characters) via option --license-file in BaseCamp Windows Here you see the output I get when having both --copyright-file and --license-file activated and building a unicode map: (c) Contour data: U.S. Geological Survey or J. de Ferranti (free for research and private use) (c) Map: FZK project äöüéèê (free for research and private use) (c) Map: FZK project ￃﾤￃﾶￃﾼￃﾩￃﾨￃﾪ (free for research and private use) (c) Map data: OSM contributors äöüéèê (ODbl) (c) Map data: OSM contributors ￃﾤￃﾶￃﾼￃﾩￃﾨￃﾪ (ODbl) (c) Contour data: U.S. Geological Survey or J. de Ferranti (free for research and private use) (c) Contour data: U.S. Geological Survey or J. de Ferranti (free for research and private use) I'll try to test more tomorrow and also confirm that I didn't do anything wrong on my end.... If I have to perform further tests, just let me know. Cheers Patrik On 11.12.2016 23:48, Patrik Brunner wrote:
I will try to test it tomorrow evening... Thanks
Patrik
Am 11.12.2016 23:29 schrieb Steve Ratcliffe <steve@parabola.me.uk>:
Hi
The file-encoding branch now appears to work with both copyright and licence files and --unicode. I've checked with BaseCamp.
Both are added in mixed case, regardless of if --lower-case is given. Use upper case letters in the licence file if that is what you actually want.
Not tested with a gmapsupp, and there is probably still work to be done there.
..Steve _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi
(c) Map: FZK project ￃﾤￃﾶￃﾼￃﾩￃﾨￃﾪ (free for research and private use)
That looks like the result from an earlier version which kind of double encoded the licence file. It didn't actually look like that in BaseCamp to me however. Just to check I am looking under Maps->Manage map products->copyright there may be something else I've missed. Your original problem implied that your system encoding is not utf-8 so maybe that is the problem. I don't know how to change that so I can't try it. I am installing the map via the registry, not loading a gmapsupp. ..Steve
Steve, I guess I was definitely using the wrong version.... I was, instead of pointing to mkgmap.jar, using hardcoded patched version with different name in the same directory... sorry for that, was a bit too quick with the tests.... with the good intention of providing quick feedback... Latest Results (with the correct version now): I was building unicode and cp1252 map and they look both allright (license and copyright wise) when checking the gmap (created via jmc_cli, not mkgmap) in Basecamp and gmapsupp (created with mkgmap) on the device itself (checked Oregon 450t only, right now). The gmapsupp version looks also good when connected to the PC and checked via BaseCamp. If I install the images directly via Registry it also looks perfect in BaseCamp. In short: All looks fine right now, I could not find/create/produce any strange behaviours... I have to clean up the map build environment a bit, as few changes where implemented to prevent problems... As soon as I have everything in shape again I'll do final tests on Windows and Linux (just to make sure all is fine there also, allthough I do not expect any new problems due to calling it always with UTF-8 env). As it looks right now: I would guess, that you solved the issues.... great. Many thanks... I'll let you know latest test results in a few days. Thanks and regards Patrik On 13.12.2016 09:03, Patrik Brunner wrote:
Steve, Thanks. I'm checking at the same location in BaseCamp... Following things can/have to be mentioned:
* I'm calling java with the option file-encoding=utf8 (or similar, can't remember by heart), this means, mkgmap is started also in utf8 environment, independant if it's run on windows or linux (tests are done on windows actually)... and the source file for the license and copyright options are properly utf8 encoded.... all will be confirmed again * I quite sure I've used the correct version, freshly compiled and moved to the correct directory.... but for sure I'll double check again.... should reflect the correct release number already ? Same as in the svn commit ? How can I else check ? * I did NOT install the img files via registry, I've used a converted GMAP Archive..... converted by jmc_cli (JaVaWa), not yet converted by mkgmap. But I will definitely go down this route also and do the Map installation via registry, to prevent any additional potential conversion issues
I will do the additional testing/confirmations latest this evening. Cheers Patrik *Gesendet:* Montag, 12. Dezember 2016 um 23:43 Uhr *Von:* "Steve Ratcliffe" <steve@parabola.me.uk> *An:* "Development list for mkgmap" <mkgmap-dev@lists.mkgmap.org.uk> *Betreff:* Re: [mkgmap-dev] mkgmap: problems with encoding file used with option --license-file
Hi
(c) Map: FZK project ￃﾤￃﾶￃﾼￃﾩￃﾨￃﾪ (free for research and private use)
That looks like the result from an earlier version which kind of double encoded the licence file. It didn't actually look like that in BaseCamp to me however.
Just to check I am looking under Maps->Manage map products->copyright there may be something else I've missed.
Your original problem implied that your system encoding is not utf-8 so maybe that is the problem. I don't know how to change that so I can't try it.
I am installing the map via the registry, not loading a gmapsupp.
..Steve _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Patrik
In short: All looks fine right now, I could not find/create/produce any strange behaviours...
That is great to hear. I shall wait for you final tests before merging. I also encourage others to test out as well as there are a fair few changes. ..Steve
Steve, In very short: all looks fine. Bit longer: I've cleaned up the building environment and created a small map (LUX) a few times: * on linux, unicode * on linux, cp1252 * on windows, unicode * on windows, cp1252 I've checked these results the following ways: * gmap archive (jmc_cli) on BaseCamp * gmapsupp.img directly on the GPS (Oregon 450t only) * gmapsupp.img in BaseCamp while having GPS connected to PC And as said at the very beginning: all looks fine. As you know: only some Latin1 characters where tested, but I do not think we'll have new conceptual problems with other characters. I would consider the issue as solved, I'm a very happy user and from my side (pure user perspective) it's ready to be merged. Many thanks for your support on this, really appreciated. Patrik On 14.12.2016 17:51, Steve Ratcliffe wrote:
Patrik
In short: All looks fine right now, I could not find/create/produce any strange behaviours...
That is great to hear. I shall wait for you final tests before merging.
I also encourage others to test out as well as there are a fair few changes.
..Steve _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Steve, did some final test with r3730 after merging the branch back to trunk (just Windows build, but as I start mkgmap on linux and windows in unicode environment I do not expect any problems): still all ok... Thanks again. Patrik On 14.12.2016 21:01, Patrik Brunner wrote:
Steve, In very short: all looks fine.
Bit longer: I've cleaned up the building environment and created a small map (LUX) a few times:
* on linux, unicode * on linux, cp1252 * on windows, unicode * on windows, cp1252
I've checked these results the following ways:
* gmap archive (jmc_cli) on BaseCamp * gmapsupp.img directly on the GPS (Oregon 450t only) * gmapsupp.img in BaseCamp while having GPS connected to PC
And as said at the very beginning: all looks fine.
As you know: only some Latin1 characters where tested, but I do not think we'll have new conceptual problems with other characters.
I would consider the issue as solved, I'm a very happy user and from my side (pure user perspective) it's ready to be merged.
Many thanks for your support on this, really appreciated. Patrik
On 14.12.2016 17:51, Steve Ratcliffe wrote:
Patrik
In short: All looks fine right now, I could not find/create/produce any strange behaviours...
That is great to hear. I shall wait for you final tests before merging.
I also encourage others to test out as well as there are a fair few changes.
..Steve _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
participants (2)
-
Patrik Brunner -
Steve Ratcliffe