Problem with compiling TYP containing Cyrillic characters (mkgmap r2673)
Gents, I have a properly translated source Text file (UTF-8 with BOM) for Russian containing, obviously, cyrillic characters. It contains definitions like (for example): [_line] Type=0x10103 UseOrientation=Y LineWidth=2 Xpm="0 0 1 0" "1 c #99CCCC" String1=0x19,?????/??? ExtendedLabels=Y FontStyle=NoLabel (invisible) CustomColor=No [end] Complete file can be downloaded at https://dl.dropboxusercontent.com/u/40846569/freizeit.zip Unfortunately mkgmap doesn't like it and fails to compile such a text file into a binary TYP file: java -Xmx1536M -jar P:/Freizeitkarte/develop/fzk-mde-garmin.git/Freizeitkarte-Entwicklung/tools/mkgmap/mkgmap.jar --max-jobs=2 --code-page=1251 --product-id=1 --family-id=6442 freizeit.txt Time started: Tue Oct 08 11:30:55 CEST 2013 java.lang.NullPointerException: charsetName at java.io.InputStreamReader.<init>(Unknown Source) at uk.me.parabola.mkgmap.main.TypCompiler.compile(TypCompiler.java:123) at uk.me.parabola.mkgmap.main.TypCompiler.makeMap(TypCompiler.java:70) at uk.me.parabola.mkgmap.main.Main$1.call(Main.java:243) at uk.me.parabola.mkgmap.main.Main$1.call(Main.java:239) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Exiting - if you want to carry on regardless, use the --keep-going option Time finished: Tue Oct 08 11:30:55 CEST 2013 Total time taken: 176ms Exact the same happens if I try to compile it with the option --code-page=65001 (for unicode). Any clue ? is this an enduser problem or a mkgmap problem ? Thanks for the help. Patrik
On 08/10/13 10:55, keenonkites wrote:
I have a properly translated source Text file (UTF-8 with BOM) for Russian containing, obviously, cyrillic characters. It contains definitions like (for example):
[_line] Type=0x10103 UseOrientation=Y LineWidth=2 Xpm="0 0 1 0" "1 c #99CCCC" String1=0x19,Канал/ров ExtendedLabels=Y FontStyle=NoLabel (invisible) CustomColor=No [end]
Hi, I will have to look into this in more detail, but the first thing to do is to remove the CodePage=1252 as mkgmap will try to read the file as though it is in written in that code page. When you do that there is an error. I think that is probably caused by the BOM. If so that should be easy to fix. ..Steve
Steve, thanks for the quick feedback.... think you're on the correct road.... taking awaz the CodePage=1251 from the source TXT file gives the following error: java -Xmx1536M -jar P:/Freizeitkarte/develop/fzk-mde-garmin.git/Freizeitkarte-Entwicklung/tools/mkgmap/mkgmap.jar --max-jobs=2 --code-page=1251 --product-id=1 --family-id=6442 freizeit.txt Time started: Tue Oct 08 14:28:04 CEST 2013 Compiling TYP txt file: Error: (freizeit.txt:1): Expecting '=' or ':' instead of ╗ Time finished: Tue Oct 08 14:28:04 CEST 2013 Total time taken: 166ms Converting the file to UTF-8 without BOM runs through properly. The resulting TYP is usable but doesn't show correct strings... looks like it interpretes the encoding incorrect, though..... you need a screenprint ? Thanks and regards Patrik On 08.10.2013 12:32, Steve Ratcliffe wrote:
On 08/10/13 10:55, keenonkites wrote:
I have a properly translated source Text file (UTF-8 with BOM) for Russian containing, obviously, cyrillic characters. It contains definitions like (for example):
[_line] Type=0x10103 UseOrientation=Y LineWidth=2 Xpm="0 0 1 0" "1 c #99CCCC" String1=0x19,Канал/ров ExtendedLabels=Y FontStyle=NoLabel (invisible) CustomColor=No [end] Hi,
I will have to look into this in more detail, but the first thing to do is to remove the CodePage=1252 as mkgmap will try to read the file as though it is in written in that code page.
When you do that there is an error. I think that is probably caused by the BOM. If so that should be easy to fix.
..Steve _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
On 08/10/13 13:42, keenonkites wrote:
Converting the file to UTF-8 without BOM runs through properly. The resulting TYP is usable but doesn't show correct strings... looks like it interpretes the encoding incorrect, though..... you need a screenprint ?
Don't need a screenshot at the moment thanks. It ought to work if the file itself is in utf-8 and there is no CodePage command in the .txt file itself. But I'll double check this tonight when I have a bit more time. Cheers ..Steve
On 08/10/13 13:42, keenonkites wrote:
Converting the file to UTF-8 without BOM runs through properly. The resulting TYP is usable but doesn't show correct strings... looks like it interpretes the encoding incorrect, though..... you need a screenprint ?
I have a patch for detecting files that begin with a BOM and forcing it to be read as utf-8 The second issue is that we allow the typ txt file to be written in the same character set as its declared CodePage as well as utf-8. As it is not possible to tell what encoding/character set a file uses this leads to errors if the file does not follow those rules. The patch now makes utf-8 a much stronger default. So if trying to read with CodePage fails it will revert to utf-8. The third issue is what should happen if the --code-page is different to the CodePage in the file. It would be consistent with the behaviour in other parts of the program for any file CodePage to be ignored if a command line --code-page is given. This is still needs doing. ..Steve
Steve, Issue 1+2: think that makes sense... should solve my actual problem. Issue 3: I did adapt the 'build' process so that the CodePage option is commented out during creation... but nevertheless I think it makes sense to be able to 'override' the things set in the file by the cli option. Could you give accesss to a 'built' version ? I'm not that familiar with the build process. Means it would be easier and for sure much faster and safer if someone else could be the complete package for me to test. Many thanks for the quick reaction anyway. Patrik On 09.10.2013 00:10, Steve Ratcliffe wrote:
On 08/10/13 13:42, keenonkites wrote:
Converting the file to UTF-8 without BOM runs through properly. The resulting TYP is usable but doesn't show correct strings... looks like it interpretes the encoding incorrect, though..... you need a screenprint ?
I have a patch for detecting files that begin with a BOM and forcing it to be read as utf-8
The second issue is that we allow the typ txt file to be written in the same character set as its declared CodePage as well as utf-8. As it is not possible to tell what encoding/character set a file uses this leads to errors if the file does not follow those rules. The patch now makes utf-8 a much stronger default. So if trying to read with CodePage fails it will revert to utf-8.
The third issue is what should happen if the --code-page is different to the CodePage in the file. It would be consistent with the behaviour in other parts of the program for any file CodePage to be ignored if a command line --code-page is given. This is still needs doing.
..Steve
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi
Could you give accesss to a 'built' version ? I'm not that familiar with the build process. Means it would be easier and for sure much faster and safer if someone else could be the complete package for me to test.
Here is a pre-built one http://files.mkgmap.org.uk/detail/154 It contains a bit more than the previous patch; the command line code-page is used when there is no CodePage present in the file. Still more to do on that. ..Steve
I did invest some time this morning into the compilation bits.... think I got it running. Therefore here the result of two tests: 1) run with the patched version (not containing everything): compilation still fails java -Xmx1536M -jar P:/Freizeitkarte/develop/fzk-mde-garmin.git/Freizeitkarte-Entwicklung/tools/mkgmap/mkgmap.jar --max-jobs=2 --code-page=1251 --product-id=1 --family-id=6442 freizeit.txt Time started: Wed Oct 09 13:56:45 CEST 2013 uk.me.parabola.imgfmt.app.typ.TypLabelException at uk.me.parabola.imgfmt.app.typ.TypElement.makeLabelBlock(TypElement.java:135) at uk.me.parabola.imgfmt.app.typ.TypElement.writeLabelBlock(TypElement.java:148) at uk.me.parabola.imgfmt.app.typ.TypPolygon.write(TypPolygon.java:49) at uk.me.parabola.imgfmt.app.typ.TYPFile.writeSection(TYPFile.java:168) at uk.me.parabola.imgfmt.app.typ.TYPFile.write(TYPFile.java:68) at uk.me.parabola.mkgmap.main.TypCompiler.writeTyp(TypCompiler.java:145) at uk.me.parabola.mkgmap.main.TypCompiler.makeMap(TypCompiler.java:102) at uk.me.parabola.mkgmap.main.Main$1.call(Main.java:243) at uk.me.parabola.mkgmap.main.Main$1.call(Main.java:239) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Exiting - if you want to carry on regardless, use the --keep-going option Time finished: Wed Oct 09 13:56:45 CEST 2013 Total time taken: 260ms 2) Test with your precompiled version (containing more fixes than the patch): Runs properly and TYP file is usable in both Qlandkarte and Basecamp java -Xmx1536M -jar P:/Freizeitkarte/develop/fzk-mde-garmin.git/Freizeitkarte-Entwicklung/tools/mkgmap/mkgmap.jar --max-jobs=2 --code-page=1251 --product-id=1 --family-id=6442 freizeit.txt Time started: Wed Oct 09 14:55:56 CEST 2013 cmd line sort sort cp=1251 order=00010008 Time finished: Wed Oct 09 14:55:56 CEST 2013 Total time taken: 410ms BTW: I get a lot of 'getting null label' messages during the build process... As mentioned in the previous post: codepage line is commented out in the source TXT file (TXT version of the TYP file). Thanks, Patrik On 09.10.2013 12:49, Steve Ratcliffe wrote:
Hi
Could you give accesss to a 'built' version ? I'm not that familiar with the build process. Means it would be easier and for sure much faster and safer if someone else could be the complete package for me to test. Here is a pre-built one http://files.mkgmap.org.uk/detail/154
It contains a bit more than the previous patch; the command line code-page is used when there is no CodePage present in the file. Still more to do on that.
..Steve _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Patrik Here is the latest version of the patch. It should now work like this: 1. If command line --code-page not given, then one in the file is used. 2. If command line --code-page is given then it is used, ignoring any the file. 3. If there is a mis-match between command line and CodePage in file, a warning is printed. 4. If the output file cannot be written in the selected code page an error message is printed instead of a stack trace. Pre-built at: http://files.mkgmap.org.uk/detail/155 I am going to add some documentation giving advice on the preferred format for a typ txt file. ..Steve
Thanks, Steve. Not sure if I'm able to test tomorrrow.... I'm out of town for a few days starting on Friday and quite busy already tomorrow with proper preparations... I'll give a proper feedback as soon as possible. Many thanks again. Patrik On 09.10.2013 22:15, Steve Ratcliffe wrote:
Hi Patrik
Here is the latest version of the patch.
It should now work like this:
1. If command line --code-page not given, then one in the file is used. 2. If command line --code-page is given then it is used, ignoring any the file. 3. If there is a mis-match between command line and CodePage in file, a warning is printed. 4. If the output file cannot be written in the selected code page an error message is printed instead of a stack trace.
Pre-built at: http://files.mkgmap.org.uk/detail/155
I am going to add some documentation giving advice on the preferred format for a typ txt file.
..Steve
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Steve, Was able to steal some time and do some tests.... but bad luck, not sure what I'm doing wrong. I've taken r2744 and replaced mkgmap.jar with the downloaded version from the link below..... but it doesn't work. I've also taken r2744 from svn, applied the patch and recompiled mkgmap.jar. Both do not compile, getting the following Exception: java -Xmx3000M -jar P:/Freizeitkarte/develop/fzk-mde-garmin.git/Freizeitkarte-Entwicklung/tools/mkgmap/mkgmap.jar --max-jobs=2 --code-page=1251 --product-id=1 --family-id=9010 freizeit.txt Time started: Thu Oct 10 14:44:31 CEST 2013 cmd line sort sort cp=1251 order=00010008 java.lang.NullPointerException at uk.me.parabola.imgfmt.app.typ.TypData.setSort(TypData.java:49) at uk.me.parabola.mkgmap.main.TypCompiler.compile(TypCompiler.java:132) at uk.me.parabola.mkgmap.main.TypCompiler.makeMap(TypCompiler.java:71) at uk.me.parabola.mkgmap.main.Main$1.call(Main.java:243) at uk.me.parabola.mkgmap.main.Main$1.call(Main.java:239) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Exiting - if you want to carry on regardless, use the --keep-going option Time finished: Thu Oct 10 14:44:31 CEST 2013 Total time taken: 160ms Am I doing something wrong ? Here's a link to 'freizeit.txt' failing above: https://dl.dropboxusercontent.com/u/40846569/freizeit-20131010.zip .... do you need some additional files ? just let me know. Patrik On 09.10.2013 22:15, Steve Ratcliffe wrote:
Hi Patrik
Here is the latest version of the patch.
It should now work like this:
1. If command line --code-page not given, then one in the file is used. 2. If command line --code-page is given then it is used, ignoring any the file. 3. If there is a mis-match between command line and CodePage in file, a warning is printed. 4. If the output file cannot be written in the selected code page an error message is printed instead of a stack trace.
Pre-built at: http://files.mkgmap.org.uk/detail/155
I am going to add some documentation giving advice on the preferred format for a typ txt file.
..Steve
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
On 10/10/13 13:50, keenonkites wrote:
java.lang.NullPointerException at uk.me.parabola.imgfmt.app.typ.TypData.setSort(TypData.java:49) at uk.me.parabola.mkgmap.main.TypCompiler.compile(TypCompiler.java:132) at uk.me.parabola.mkgmap.main.TypCompiler.makeMap(TypCompiler.java:71)
Sorry that was a last minute change that broke it again. New patch attached. Pre-built at: http://files.mkgmap.org.uk/detail/156 Cheers, ..Steve
Looks much better now: java -Xmx3000M -jar P:/Freizeitkarte/develop/fzk-mde-garmin.git/Freizeitkarte-Entwicklung/tools/mkgmap/mkgmap.jar --max-jobs=2 --code-page=1251 --product-id=1 --family-id=9010 freizeit.txt Time started: Thu Oct 10 16:06:01 CEST 2013 cmd line sort sort cp=1251 order=00010008 WARNING: SortCode in TYP txt file different from command line setting Time finished: Thu Oct 10 16:06:01 CEST 2013 Total time taken: 380ms I just have a lot of output like 'getting null label' while actually building the map (not compiling the TYP file)... assumeing that's some sort of debug output. I've just replaced mkgmap.jar again with the downloaded version (didn't test to compile it, running out of time...). You don't have to say sorry, I have to say thank you. Cheers Patrik On 10.10.2013 14:58, Steve Ratcliffe wrote:
On 10/10/13 13:50, keenonkites wrote:
java.lang.NullPointerException at uk.me.parabola.imgfmt.app.typ.TypData.setSort(TypData.java:49) at uk.me.parabola.mkgmap.main.TypCompiler.compile(TypCompiler.java:132) at uk.me.parabola.mkgmap.main.TypCompiler.makeMap(TypCompiler.java:71)
Sorry that was a last minute change that broke it again.
New patch attached.
Pre-built at: http://files.mkgmap.org.uk/detail/156
Cheers, ..Steve
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
That's what I thought.... this means: I'm a happy user... ;-) I'm trying to perform more tests over the next few days, but I'm on the road with very restricted spare time.... but I'll be on the road now with a proper cyrillic map and TYP file... Thanks Patrik On 10.10.2013 18:18, Steve Ratcliffe wrote:
On 10/10/13 15:18, keenonkites wrote:
I just have a lot of output like 'getting null label' while actually building the map Yea, just ignore those, its some debugging in my local code. Its not an error of any kind.
..Steve _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
participants (2)
-
keenonkites -
Steve Ratcliffe