New assertion, now with code-page=632 and Japan tile
The command below produces an assertion while compiling this tile <https://files.mkgmap.org.uk/download/526/31191025.o5m> from Japan. Process continues with remaining tiles and finishes without "Number of MapFailedExceptions: 1" as expected. This is with r4813, but I also tried with an old version of mkgmap with the same result. java -Xmx27G -ea -jar mkgmap.jar--code-page=632 31191025.o5m Mkgmap version 4813 Time started: Tue Nov 09 20:18:16 CET 2021 WARNING (global): Setting max-jobs to 8 Exception in thread "main" java.lang.AssertionError: found trailing 0 in chars at uk.me.parabola.imgfmt.app.labelenc.EncodedText.<init>(EncodedText.java:39) at uk.me.parabola.imgfmt.app.labelenc.AnyCharsetEncoder.encodeText(AnyCharsetEncoder.java:112) at uk.me.parabola.imgfmt.app.lbl.LBLFile.newLabel(LBLFile.java:132) at uk.me.parabola.imgfmt.app.lbl.PlacesFile.createPOI(PlacesFile.java:253) at uk.me.parabola.imgfmt.app.lbl.LBLFile.createPOI(LBLFile.java:172) at uk.me.parabola.mkgmap.build.MapBuilder.processPOIs(MapBuilder.java:670) at uk.me.parabola.mkgmap.build.MapBuilder.makeMap(MapBuilder.java:325) at uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:114) at uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:62) at uk.me.parabola.mkgmap.main.Main.lambda$processFilename$1(Main.java:291) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829)
code-page=932, sorry for the typo. El 9/11/21 a las 20:36, Carlos Dávila escribió:
The command below produces an assertion while compiling this tile <https://files.mkgmap.org.uk/download/526/31191025.o5m> from Japan. Process continues with remaining tiles and finishes without "Number of MapFailedExceptions: 1" as expected. This is with r4813, but I also tried with an old version of mkgmap with the same result.
java -Xmx27G -ea -jar mkgmap.jar--code-page=632 31191025.o5m Mkgmap version 4813 Time started: Tue Nov 09 20:18:16 CET 2021 WARNING (global): Setting max-jobs to 8 Exception in thread "main" java.lang.AssertionError: found trailing 0 in chars at uk.me.parabola.imgfmt.app.labelenc.EncodedText.<init>(EncodedText.java:39) at uk.me.parabola.imgfmt.app.labelenc.AnyCharsetEncoder.encodeText(AnyCharsetEncoder.java:112) at uk.me.parabola.imgfmt.app.lbl.LBLFile.newLabel(LBLFile.java:132) at uk.me.parabola.imgfmt.app.lbl.PlacesFile.createPOI(PlacesFile.java:253) at uk.me.parabola.imgfmt.app.lbl.LBLFile.createPOI(LBLFile.java:172) at uk.me.parabola.mkgmap.build.MapBuilder.processPOIs(MapBuilder.java:670) at uk.me.parabola.mkgmap.build.MapBuilder.makeMap(MapBuilder.java:325) at uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:114) at uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:62) at uk.me.parabola.mkgmap.main.Main.lambda$processFilename$1(Main.java:291) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829)
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi I think this assertion could be removed from the code. Looking through the definition of Shift-JIS, I read it as saying the second byte shouldn't be zero, so I don't know why this happens. As with the Chinese code-pages, mkgmap has places where multi-byte encodings are not handled correctly in the --index generation and unknown meanings of flags to the Garmin software. Ticker On 09/11/2021 19:43, Carlos Dávila wrote:
code-page=932, sorry for the typo.
El 9/11/21 a las 20:36, Carlos Dávila escribió:
The command below produces an assertion while compiling this tile <https://files.mkgmap.org.uk/download/526/31191025.o5m> from Japan. Process continues with remaining tiles and finishes without "Number of MapFailedExceptions: 1" as expected. This is with r4813, but I also tried with an old version of mkgmap with the same result.
java -Xmx27G -ea -jar mkgmap.jar--code-page=632 31191025.o5m Mkgmap version 4813 Time started: Tue Nov 09 20:18:16 CET 2021 WARNING (global): Setting max-jobs to 8 Exception in thread "main" java.lang.AssertionError: found trailing 0 in chars at uk.me.parabola.imgfmt.app.labelenc.EncodedText.<init>(EncodedText.java:39)
at uk.me.parabola.imgfmt.app.labelenc.AnyCharsetEncoder.encodeText(AnyCharsetEncoder.java:112)
at uk.me.parabola.imgfmt.app.lbl.LBLFile.newLabel(LBLFile.java:132) at uk.me.parabola.imgfmt.app.lbl.PlacesFile.createPOI(PlacesFile.java:253) at uk.me.parabola.imgfmt.app.lbl.LBLFile.createPOI(LBLFile.java:172) at uk.me.parabola.mkgmap.build.MapBuilder.processPOIs(MapBuilder.java:670) at uk.me.parabola.mkgmap.build.MapBuilder.makeMap(MapBuilder.java:325) at uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:114) at uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:62) at uk.me.parabola.mkgmap.main.Main.lambda$processFilename$1(Main.java:291) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Ticker Not sure if relevant, but note in this case assertion occurs while compiling the tile, not the index. In fact, --index is not included in the command. El 9/11/21 a las 21:55, Ticker Berkin escribió:
Hi
I think this assertion could be removed from the code.
Looking through the definition of Shift-JIS, I read it as saying the second byte shouldn't be zero, so I don't know why this happens.
As with the Chinese code-pages, mkgmap has places where multi-byte encodings are not handled correctly in the --index generation and unknown meanings of flags to the Garmin software.
Ticker
On 09/11/2021 19:43, Carlos Dávila wrote:
code-page=932, sorry for the typo.
El 9/11/21 a las 20:36, Carlos Dávila escribió:
The command below produces an assertion while compiling this tile <https://files.mkgmap.org.uk/download/526/31191025.o5m> from Japan. Process continues with remaining tiles and finishes without "Number of MapFailedExceptions: 1" as expected. This is with r4813, but I also tried with an old version of mkgmap with the same result.
java -Xmx27G -ea -jar mkgmap.jar--code-page=632 31191025.o5m Mkgmap version 4813 Time started: Tue Nov 09 20:18:16 CET 2021 WARNING (global): Setting max-jobs to 8 Exception in thread "main" java.lang.AssertionError: found trailing 0 in chars at uk.me.parabola.imgfmt.app.labelenc.EncodedText.<init>(EncodedText.java:39)
at uk.me.parabola.imgfmt.app.labelenc.AnyCharsetEncoder.encodeText(AnyCharsetEncoder.java:112)
at uk.me.parabola.imgfmt.app.lbl.LBLFile.newLabel(LBLFile.java:132) at uk.me.parabola.imgfmt.app.lbl.PlacesFile.createPOI(PlacesFile.java:253) at uk.me.parabola.imgfmt.app.lbl.LBLFile.createPOI(LBLFile.java:172) at uk.me.parabola.mkgmap.build.MapBuilder.processPOIs(MapBuilder.java:670) at uk.me.parabola.mkgmap.build.MapBuilder.makeMap(MapBuilder.java:325) at uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:114) at uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:62) at uk.me.parabola.mkgmap.main.Main.lambda$processFilename$1(Main.java:291) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Carlos, I'll try to debug this. BTW: I see you use *.o5m for the tiles (output from splitter). I think this is no longer a good choice, pbf is a lot smaller and almost as fast. Esp. when it comes to the goal of reducing disk I/O (as with --gmapi-minimal) Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Carlos Dávila <carlos@alternativaslibres.org> Gesendet: Dienstag, 9. November 2021 22:54 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] New assertion, now with code-page=632 and Japan tile Hi Ticker Not sure if relevant, but note in this case assertion occurs while compiling the tile, not the index. In fact, --index is not included in the command. El 9/11/21 a las 21:55, Ticker Berkin escribió:
Hi
I think this assertion could be removed from the code.
Looking through the definition of Shift-JIS, I read it as saying the second byte shouldn't be zero, so I don't know why this happens.
As with the Chinese code-pages, mkgmap has places where multi-byte encodings are not handled correctly in the --index generation and unknown meanings of flags to the Garmin software.
Ticker
On 09/11/2021 19:43, Carlos Dávila wrote:
code-page=932, sorry for the typo.
El 9/11/21 a las 20:36, Carlos Dávila escribió:
The command below produces an assertion while compiling this tile <https://files.mkgmap.org.uk/download/526/31191025.o5m> from Japan. Process continues with remaining tiles and finishes without "Number of MapFailedExceptions: 1" as expected. This is with r4813, but I also tried with an old version of mkgmap with the same result.
java -Xmx27G -ea -jar mkgmap.jar--code-page=632 31191025.o5m Mkgmap version 4813 Time started: Tue Nov 09 20:18:16 CET 2021 WARNING (global): Setting max-jobs to 8 Exception in thread "main" java.lang.AssertionError: found trailing 0 in chars at uk.me.parabola.imgfmt.app.labelenc.EncodedText.<init>(EncodedText.java:39)
at uk.me.parabola.imgfmt.app.labelenc.AnyCharsetEncoder.encodeText(AnyCharsetEncoder.java:112)
at uk.me.parabola.imgfmt.app.lbl.LBLFile.newLabel(LBLFile.java:132) at uk.me.parabola.imgfmt.app.lbl.PlacesFile.createPOI(PlacesFile.java:253) at uk.me.parabola.imgfmt.app.lbl.LBLFile.createPOI(LBLFile.java:172) at uk.me.parabola.mkgmap.build.MapBuilder.processPOIs(MapBuilder.java:670) at uk.me.parabola.mkgmap.build.MapBuilder.makeMap(MapBuilder.java:325) at uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:114) at uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:62) at uk.me.parabola.mkgmap.main.Main.lambda$processFilename$1(Main.java:291) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi devs, the problem occurs with node https://www.osm.org/node/5692472121 name=키타가키 고로케 Google translate says the name is Korean. The (utf8) name cannot be translated into code-page 932 (japanese) and thus mkgmap converts the internal utf16 representation of the name to bytes. This happens in method AnyCharsetEncoder.encodeText(String text) in this loop: for (int i = 0; i < s.length(); i++) outBuf.put((byte) s.charAt(i)); The name 키타가키 고로케 ends with 케 and the char value is \ucf00, so it is converted to \0x00. Maybe we should simply stop transliteration when this happens and return an empty string for the label? If mkgmap is executed without the -ea run time option the map shows name ､ﾀ for the restaurant which is just wrong. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Mittwoch, 10. November 2021 09:43 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] New assertion, now with code-page=632 and Japan tile Hi Carlos, I'll try to debug this. BTW: I see you use *.o5m for the tiles (output from splitter). I think this is no longer a good choice, pbf is a lot smaller and almost as fast. Esp. when it comes to the goal of reducing disk I/O (as with --gmapi-minimal) Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Carlos Dávila <carlos@alternativaslibres.org> Gesendet: Dienstag, 9. November 2021 22:54 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] New assertion, now with code-page=632 and Japan tile Hi Ticker Not sure if relevant, but note in this case assertion occurs while compiling the tile, not the index. In fact, --index is not included in the command. El 9/11/21 a las 21:55, Ticker Berkin escribió:
Hi
I think this assertion could be removed from the code.
Looking through the definition of Shift-JIS, I read it as saying the second byte shouldn't be zero, so I don't know why this happens.
As with the Chinese code-pages, mkgmap has places where multi-byte encodings are not handled correctly in the --index generation and unknown meanings of flags to the Garmin software.
Ticker
On 09/11/2021 19:43, Carlos Dávila wrote:
code-page=932, sorry for the typo.
El 9/11/21 a las 20:36, Carlos Dávila escribió:
The command below produces an assertion while compiling this tile <https://files.mkgmap.org.uk/download/526/31191025.o5m> from Japan. Process continues with remaining tiles and finishes without "Number of MapFailedExceptions: 1" as expected. This is with r4813, but I also tried with an old version of mkgmap with the same result.
java -Xmx27G -ea -jar mkgmap.jar--code-page=632 31191025.o5m Mkgmap version 4813 Time started: Tue Nov 09 20:18:16 CET 2021 WARNING (global): Setting max-jobs to 8 Exception in thread "main" java.lang.AssertionError: found trailing 0 in chars at uk.me.parabola.imgfmt.app.labelenc.EncodedText.<init>(EncodedText.java:39)
at uk.me.parabola.imgfmt.app.labelenc.AnyCharsetEncoder.encodeText(AnyCharsetEncoder.java:112)
at uk.me.parabola.imgfmt.app.lbl.LBLFile.newLabel(LBLFile.java:132) at uk.me.parabola.imgfmt.app.lbl.PlacesFile.createPOI(PlacesFile.java:253) at uk.me.parabola.imgfmt.app.lbl.LBLFile.createPOI(LBLFile.java:172) at uk.me.parabola.mkgmap.build.MapBuilder.processPOIs(MapBuilder.java:670) at uk.me.parabola.mkgmap.build.MapBuilder.makeMap(MapBuilder.java:325) at uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:114) at uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:62) at uk.me.parabola.mkgmap.main.Main.lambda$processFilename$1(Main.java:291) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi all,
Maybe we should simply stop transliteration when this happens and return an empty string for the label?
any thoughts on this? Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Mittwoch, 10. November 2021 11:17 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] New assertion, now with code-page=632 and Japan tile Hi devs, the problem occurs with node https://www.osm.org/node/5692472121 name=키타가키 고로케 Google translate says the name is Korean. The (utf8) name cannot be translated into code-page 932 (japanese) and thus mkgmap converts the internal utf16 representation of the name to bytes. This happens in method AnyCharsetEncoder.encodeText(String text) in this loop: for (int i = 0; i < s.length(); i++) outBuf.put((byte) s.charAt(i)); The name 키타가키 고로케 ends with 케 and the char value is \ucf00, so it is converted to \0x00. Maybe we should simply stop transliteration when this happens and return an empty string for the label? If mkgmap is executed without the -ea run time option the map shows name ､ﾀ for the restaurant which is just wrong. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Mittwoch, 10. November 2021 09:43 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] New assertion, now with code-page=632 and Japan tile Hi Carlos, I'll try to debug this. BTW: I see you use *.o5m for the tiles (output from splitter). I think this is no longer a good choice, pbf is a lot smaller and almost as fast. Esp. when it comes to the goal of reducing disk I/O (as with --gmapi-minimal) Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Carlos Dávila <carlos@alternativaslibres.org> Gesendet: Dienstag, 9. November 2021 22:54 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] New assertion, now with code-page=632 and Japan tile Hi Ticker Not sure if relevant, but note in this case assertion occurs while compiling the tile, not the index. In fact, --index is not included in the command. El 9/11/21 a las 21:55, Ticker Berkin escribió:
Hi
I think this assertion could be removed from the code.
Looking through the definition of Shift-JIS, I read it as saying the second byte shouldn't be zero, so I don't know why this happens.
As with the Chinese code-pages, mkgmap has places where multi-byte encodings are not handled correctly in the --index generation and unknown meanings of flags to the Garmin software.
Ticker
On 09/11/2021 19:43, Carlos Dávila wrote:
code-page=932, sorry for the typo.
El 9/11/21 a las 20:36, Carlos Dávila escribió:
The command below produces an assertion while compiling this tile <https://files.mkgmap.org.uk/download/526/31191025.o5m> from Japan. Process continues with remaining tiles and finishes without "Number of MapFailedExceptions: 1" as expected. This is with r4813, but I also tried with an old version of mkgmap with the same result.
java -Xmx27G -ea -jar mkgmap.jar--code-page=632 31191025.o5m Mkgmap version 4813 Time started: Tue Nov 09 20:18:16 CET 2021 WARNING (global): Setting max-jobs to 8 Exception in thread "main" java.lang.AssertionError: found trailing 0 in chars at uk.me.parabola.imgfmt.app.labelenc.EncodedText.<init>(EncodedText.java:39)
at uk.me.parabola.imgfmt.app.labelenc.AnyCharsetEncoder.encodeText(AnyCharsetEncoder.java:112)
at uk.me.parabola.imgfmt.app.lbl.LBLFile.newLabel(LBLFile.java:132) at uk.me.parabola.imgfmt.app.lbl.PlacesFile.createPOI(PlacesFile.java:253) at uk.me.parabola.imgfmt.app.lbl.LBLFile.createPOI(LBLFile.java:172) at uk.me.parabola.mkgmap.build.MapBuilder.processPOIs(MapBuilder.java:670) at uk.me.parabola.mkgmap.build.MapBuilder.makeMap(MapBuilder.java:325) at uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:114) at uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:62) at uk.me.parabola.mkgmap.main.Main.lambda$processFilename$1(Main.java:291) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi How about something like: If the full string fails to encode in the target charset, process char at a time. If a char can't be represented, try transliteration on it and, if none defined, use "?", then go through the resultant string char at a time, and if this can't be represented, drop it. Maybe a final warning at end if no transliteration for a char or transliteration couldn't be represented. Ticker On Mon, 2021-11-15 at 13:04 +0000, Gerd Petermann wrote:
Hi all,
Maybe we should simply stop transliteration when this happens and return an empty string for the label?
any thoughts on this?
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Mittwoch, 10. November 2021 11:17 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] New assertion, now with code- page=632 and Japan tile
Hi devs,
the problem occurs with node https://www.osm.org/node/5692472121 name=키타가키 고로케 Google translate says the name is Korean. The (utf8) name cannot be translated into code-page 932 (japanese) and thus mkgmap converts the internal utf16 representation of the name to bytes. This happens in method AnyCharsetEncoder.encodeText(String text) in this loop: for (int i = 0; i < s.length(); i++) outBuf.put((byte) s.charAt(i)); The name 키타가키 고로케 ends with 케 and the char value is \ucf00, so it is converted to \0x00. Maybe we should simply stop transliteration when this happens and return an empty string for the label?
If mkgmap is executed without the -ea run time option the map shows name ､ﾀ for the restaurant which is just wrong. Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Mittwoch, 10. November 2021 09:43 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] New assertion, now with code- page=632 and Japan tile
Hi Carlos,
I'll try to debug this.
BTW: I see you use *.o5m for the tiles (output from splitter). I think this is no longer a good choice, pbf is a lot smaller and almost as fast. Esp. when it comes to the goal of reducing disk I/O (as with --gmapi-minimal)
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Carlos Dávila <carlos@alternativaslibres.org> Gesendet: Dienstag, 9. November 2021 22:54 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] New assertion, now with code- page=632 and Japan tile
Hi Ticker
Not sure if relevant, but note in this case assertion occurs while compiling the tile, not the index. In fact, --index is not included in the command.
El 9/11/21 a las 21:55, Ticker Berkin escribió:
Hi
I think this assertion could be removed from the code.
Looking through the definition of Shift-JIS, I read it as saying the second byte shouldn't be zero, so I don't know why this happens.
As with the Chinese code-pages, mkgmap has places where multi-byte encodings are not handled correctly in the --index generation and unknown meanings of flags to the Garmin software.
Ticker
On 09/11/2021 19:43, Carlos Dávila wrote:
code-page=932, sorry for the typo.
El 9/11/21 a las 20:36, Carlos Dávila escribió:
The command below produces an assertion while compiling this tile <https://files.mkgmap.org.uk/download/526/31191025.o5m> from Japan. Process continues with remaining tiles and finishes without "Number of MapFailedExceptions: 1" as expected. This is with r4813, but I also tried with an old version of mkgmap with the same result.
java -Xmx27G -ea -jar mkgmap.jar--code-page=632 31191025.o5m Mkgmap version 4813 Time started: Tue Nov 09 20:18:16 CET 2021 WARNING (global): Setting max-jobs to 8 Exception in thread "main" java.lang.AssertionError: found trailing 0 in chars at uk.me.parabola.imgfmt.app.labelenc.EncodedText.<init>(EncodedTe xt.java:39)
at uk.me.parabola.imgfmt.app.labelenc.AnyCharsetEncoder.encodeText (AnyCharsetEncoder.java:112)
at uk.me.parabola.imgfmt.app.lbl.LBLFile.newLabel(LBLFile.java:132 ) at uk.me.parabola.imgfmt.app.lbl.PlacesFile.createPOI(PlacesFile.j ava:253) at uk.me.parabola.imgfmt.app.lbl.LBLFile.createPOI(LBLFile.java:17 2) at uk.me.parabola.mkgmap.build.MapBuilder.processPOIs(MapBuilder.j ava:670) at uk.me.parabola.mkgmap.build.MapBuilder.makeMap(MapBuilder.java: 325) at uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:114) at uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:62) at uk.me.parabola.mkgmap.main.Main.lambda$processFilename$1(Main.j ava:291) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:2 64) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Thr eadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Th readPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi all, this small patch would be my approach. It replaces those characters which don't fit into a byte by '?' This fixes the problems with japanese codepage 932. Gerd BTW: SparseTransliterator is very sparse. We could add a few more character mappings, for example there is a housenumber that contains "１２３７−１" instead of "1237-1". https://www.fontspace.com/unicode/analyzer#e=77yR77yS77yT77yX4oiS77yR ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Montag, 15. November 2021 15:59 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] New assertion, now with code-page=632 and Japan tile Hi How about something like: If the full string fails to encode in the target charset, process char at a time. If a char can't be represented, try transliteration on it and, if none defined, use "?", then go through the resultant string char at a time, and if this can't be represented, drop it. Maybe a final warning at end if no transliteration for a char or transliteration couldn't be represented. Ticker On Mon, 2021-11-15 at 13:04 +0000, Gerd Petermann wrote:
Hi all,
Maybe we should simply stop transliteration when this happens and return an empty string for the label?
any thoughts on this?
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Mittwoch, 10. November 2021 11:17 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] New assertion, now with code- page=632 and Japan tile
Hi devs,
the problem occurs with node https://www.osm.org/node/5692472121 name=키타가키 고로케 Google translate says the name is Korean. The (utf8) name cannot be translated into code-page 932 (japanese) and thus mkgmap converts the internal utf16 representation of the name to bytes. This happens in method AnyCharsetEncoder.encodeText(String text) in this loop: for (int i = 0; i < s.length(); i++) outBuf.put((byte) s.charAt(i)); The name 키타가키 고로케 ends with 케 and the char value is \ucf00, so it is converted to \0x00. Maybe we should simply stop transliteration when this happens and return an empty string for the label?
If mkgmap is executed without the -ea run time option the map shows name ､ﾀ for the restaurant which is just wrong. Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Mittwoch, 10. November 2021 09:43 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] New assertion, now with code- page=632 and Japan tile
Hi Carlos,
I'll try to debug this.
BTW: I see you use *.o5m for the tiles (output from splitter). I think this is no longer a good choice, pbf is a lot smaller and almost as fast. Esp. when it comes to the goal of reducing disk I/O (as with --gmapi-minimal)
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Carlos Dávila <carlos@alternativaslibres.org> Gesendet: Dienstag, 9. November 2021 22:54 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] New assertion, now with code- page=632 and Japan tile
Hi Ticker
Not sure if relevant, but note in this case assertion occurs while compiling the tile, not the index. In fact, --index is not included in the command.
El 9/11/21 a las 21:55, Ticker Berkin escribió:
Hi
I think this assertion could be removed from the code.
Looking through the definition of Shift-JIS, I read it as saying the second byte shouldn't be zero, so I don't know why this happens.
As with the Chinese code-pages, mkgmap has places where multi-byte encodings are not handled correctly in the --index generation and unknown meanings of flags to the Garmin software.
Ticker
On 09/11/2021 19:43, Carlos Dávila wrote:
code-page=932, sorry for the typo.
El 9/11/21 a las 20:36, Carlos Dávila escribió:
The command below produces an assertion while compiling this tile <https://files.mkgmap.org.uk/download/526/31191025.o5m> from Japan. Process continues with remaining tiles and finishes without "Number of MapFailedExceptions: 1" as expected. This is with r4813, but I also tried with an old version of mkgmap with the same result.
java -Xmx27G -ea -jar mkgmap.jar--code-page=632 31191025.o5m Mkgmap version 4813 Time started: Tue Nov 09 20:18:16 CET 2021 WARNING (global): Setting max-jobs to 8 Exception in thread "main" java.lang.AssertionError: found trailing 0 in chars at uk.me.parabola.imgfmt.app.labelenc.EncodedText.<init>(EncodedTe xt.java:39)
at uk.me.parabola.imgfmt.app.labelenc.AnyCharsetEncoder.encodeText (AnyCharsetEncoder.java:112)
at uk.me.parabola.imgfmt.app.lbl.LBLFile.newLabel(LBLFile.java:132 ) at uk.me.parabola.imgfmt.app.lbl.PlacesFile.createPOI(PlacesFile.j ava:253) at uk.me.parabola.imgfmt.app.lbl.LBLFile.createPOI(LBLFile.java:17 2) at uk.me.parabola.mkgmap.build.MapBuilder.processPOIs(MapBuilder.j ava:670) at uk.me.parabola.mkgmap.build.MapBuilder.makeMap(MapBuilder.java: 325) at uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:114) at uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:62) at uk.me.parabola.mkgmap.main.Main.lambda$processFilename$1(Main.j ava:291) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:2 64) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Thr eadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Th readPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi wouldn't: if ((c & 0xff) == 0) c = "?"; be safer I don't understand the point of sparseTranslitorator and why it is only used for cp932 (japanese), unless this charset includes quite a few european accented character. If this is the case then wouldn't it be much better to do as I described, the essence of which is not to transliterate the complete string into the small ascii/latin1 set just because some chars can't be mapped. The TableTranslitorator (ascii & latin1) map these FULLWIDTH digits (and letters). MATHS MINUS isn't defined but easy to add. Handling char at a time might allow removal of the 'ascii' table - if transliteration changes char to [string of] another, for each of these, if can't be represented, transliterate them. Ticker On Tue, 2021-11-16 at 15:48 +0000, Gerd Petermann wrote:
Hi all,
this small patch would be my approach. It replaces those characters which don't fit into a byte by '?' This fixes the problems with japanese codepage 932.
Gerd BTW: SparseTransliterator is very sparse. We could add a few more character mappings, for example there is a housenumber that contains "１２３７−１" instead of "1237-1". https://www.fontspace.com/unicode/analyzer#e=77yR77yS77yT77yX4oiS77yR
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Montag, 15. November 2021 15:59 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] New assertion, now with code-page=632 and Japan tile
Hi
How about something like:
If the full string fails to encode in the target charset, process char at a time.
If a char can't be represented, try transliteration on it and, if none defined, use "?", then go through the resultant string char at a time, and if this can't be represented, drop it.
Maybe a final warning at end if no transliteration for a char or transliteration couldn't be represented.
Ticker
On Mon, 2021-11-15 at 13:04 +0000, Gerd Petermann wrote:
Hi all,
Maybe we should simply stop transliteration when this happens and return an empty string for the label?
any thoughts on this?
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Mittwoch, 10. November 2021 11:17 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] New assertion, now with code- page=632 and Japan tile
Hi devs,
the problem occurs with node https://www.osm.org/node/5692472121 name=키타가키 고로케 Google translate says the name is Korean. The (utf8) name cannot be translated into code-page 932 (japanese) and thus mkgmap converts the internal utf16 representation of the name to bytes. This happens in method AnyCharsetEncoder.encodeText(String text) in this loop: for (int i = 0; i < s.length(); i++) outBuf.put((byte) s.charAt(i)); The name 키타가키 고로케 ends with 케 and the char value is \ucf00, so it is converted to \0x00. Maybe we should simply stop transliteration when this happens and return an empty string for the label?
If mkgmap is executed without the -ea run time option the map shows name ､ﾀ for the restaurant which is just wrong. Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Mittwoch, 10. November 2021 09:43 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] New assertion, now with code- page=632 and Japan tile
Hi Carlos,
I'll try to debug this.
BTW: I see you use *.o5m for the tiles (output from splitter). I think this is no longer a good choice, pbf is a lot smaller and almost as fast. Esp. when it comes to the goal of reducing disk I/O (as with --gmapi-minimal)
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Carlos Dávila <carlos@alternativaslibres.org> Gesendet: Dienstag, 9. November 2021 22:54 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] New assertion, now with code- page=632 and Japan tile
Hi Ticker
Not sure if relevant, but note in this case assertion occurs while compiling the tile, not the index. In fact, --index is not included in the command.
El 9/11/21 a las 21:55, Ticker Berkin escribió:
Hi
I think this assertion could be removed from the code.
Looking through the definition of Shift-JIS, I read it as saying the second byte shouldn't be zero, so I don't know why this happens.
As with the Chinese code-pages, mkgmap has places where multi- byte encodings are not handled correctly in the --index generation and unknown meanings of flags to the Garmin software.
Ticker
On 09/11/2021 19:43, Carlos Dávila wrote:
code-page=932, sorry for the typo.
El 9/11/21 a las 20:36, Carlos Dávila escribió:
The command below produces an assertion while compiling this tile <https://files.mkgmap.org.uk/download/526/31191025.o5m> from Japan. Process continues with remaining tiles and finishes without "Number of MapFailedExceptions: 1" as expected. This is with r4813, but I also tried with an old version of mkgmap with the same result.
java -Xmx27G -ea -jar mkgmap.jar--code-page=632 31191025.o5m Mkgmap version 4813 Time started: Tue Nov 09 20:18:16 CET 2021 WARNING (global): Setting max-jobs to 8 Exception in thread "main" java.lang.AssertionError: found trailing 0 in chars at uk.me.parabola.imgfmt.app.labelenc.EncodedText.<init>(Encoded Te xt.java:39)
at uk.me.parabola.imgfmt.app.labelenc.AnyCharsetEncoder.encodeTe xt (AnyCharsetEncoder.java:112)
at uk.me.parabola.imgfmt.app.lbl.LBLFile.newLabel(LBLFile.java:1 32 ) at uk.me.parabola.imgfmt.app.lbl.PlacesFile.createPOI(PlacesFile .j ava:253) at uk.me.parabola.imgfmt.app.lbl.LBLFile.createPOI(LBLFile.java: 17 2) at uk.me.parabola.mkgmap.build.MapBuilder.processPOIs(MapBuilder .j ava:670) at uk.me.parabola.mkgmap.build.MapBuilder.makeMap(MapBuilder.jav a: 325) at uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:114 ) at uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:62) at uk.me.parabola.mkgmap.main.Main.lambda$processFilename$1(Main .j ava:291) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java :2 64) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(T hr eadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run( Th readPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Ticker, remember that cs932 is a double-byte character set. With your code only a few unmappable utf-16 characters are replaced, for the rest one of cs932 is used, but without any good reason. The result is typically garbage. I've modified the patch to replace any unmappable character that was not transliterated by '?' . I've also attached a debug version that shows what goes on. A possible change in SparseTransliterator would be to add a mapping for the MATH MINUS, the other FULLWIDTH digits are supported in cs932. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Dienstag, 16. November 2021 17:33 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] New assertion, now with code-page=632 and Japan tile Hi wouldn't: if ((c & 0xff) == 0) c = "?"; be safer I don't understand the point of sparseTranslitorator and why it is only used for cp932 (japanese), unless this charset includes quite a few european accented character. If this is the case then wouldn't it be much better to do as I described, the essence of which is not to transliterate the complete string into the small ascii/latin1 set just because some chars can't be mapped. The TableTranslitorator (ascii & latin1) map these FULLWIDTH digits (and letters). MATHS MINUS isn't defined but easy to add. Handling char at a time might allow removal of the 'ascii' table - if transliteration changes char to [string of] another, for each of these, if can't be represented, transliterate them. Ticker On Tue, 2021-11-16 at 15:48 +0000, Gerd Petermann wrote:
Hi all,
this small patch would be my approach. It replaces those characters which don't fit into a byte by '?' This fixes the problems with japanese codepage 932.
Gerd BTW: SparseTransliterator is very sparse. We could add a few more character mappings, for example there is a housenumber that contains "１２３７−１" instead of "1237-1". https://www.fontspace.com/unicode/analyzer#e=77yR77yS77yT77yX4oiS77yR
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Montag, 15. November 2021 15:59 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] New assertion, now with code-page=632 and Japan tile
Hi
How about something like:
If the full string fails to encode in the target charset, process char at a time.
If a char can't be represented, try transliteration on it and, if none defined, use "?", then go through the resultant string char at a time, and if this can't be represented, drop it.
Maybe a final warning at end if no transliteration for a char or transliteration couldn't be represented.
Ticker
On Mon, 2021-11-15 at 13:04 +0000, Gerd Petermann wrote:
Hi all,
Maybe we should simply stop transliteration when this happens and return an empty string for the label?
any thoughts on this?
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Mittwoch, 10. November 2021 11:17 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] New assertion, now with code- page=632 and Japan tile
Hi devs,
the problem occurs with node https://www.osm.org/node/5692472121 name=키타가키 고로케 Google translate says the name is Korean. The (utf8) name cannot be translated into code-page 932 (japanese) and thus mkgmap converts the internal utf16 representation of the name to bytes. This happens in method AnyCharsetEncoder.encodeText(String text) in this loop: for (int i = 0; i < s.length(); i++) outBuf.put((byte) s.charAt(i)); The name 키타가키 고로케 ends with 케 and the char value is \ucf00, so it is converted to \0x00. Maybe we should simply stop transliteration when this happens and return an empty string for the label?
If mkgmap is executed without the -ea run time option the map shows name ､ﾀ for the restaurant which is just wrong. Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Mittwoch, 10. November 2021 09:43 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] New assertion, now with code- page=632 and Japan tile
Hi Carlos,
I'll try to debug this.
BTW: I see you use *.o5m for the tiles (output from splitter). I think this is no longer a good choice, pbf is a lot smaller and almost as fast. Esp. when it comes to the goal of reducing disk I/O (as with --gmapi-minimal)
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Carlos Dávila <carlos@alternativaslibres.org> Gesendet: Dienstag, 9. November 2021 22:54 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] New assertion, now with code- page=632 and Japan tile
Hi Ticker
Not sure if relevant, but note in this case assertion occurs while compiling the tile, not the index. In fact, --index is not included in the command.
El 9/11/21 a las 21:55, Ticker Berkin escribió:
Hi
I think this assertion could be removed from the code.
Looking through the definition of Shift-JIS, I read it as saying the second byte shouldn't be zero, so I don't know why this happens.
As with the Chinese code-pages, mkgmap has places where multi- byte encodings are not handled correctly in the --index generation and unknown meanings of flags to the Garmin software.
Ticker
On 09/11/2021 19:43, Carlos Dávila wrote:
code-page=932, sorry for the typo.
El 9/11/21 a las 20:36, Carlos Dávila escribió:
The command below produces an assertion while compiling this tile <https://files.mkgmap.org.uk/download/526/31191025.o5m> from Japan. Process continues with remaining tiles and finishes without "Number of MapFailedExceptions: 1" as expected. This is with r4813, but I also tried with an old version of mkgmap with the same result.
java -Xmx27G -ea -jar mkgmap.jar--code-page=632 31191025.o5m Mkgmap version 4813 Time started: Tue Nov 09 20:18:16 CET 2021 WARNING (global): Setting max-jobs to 8 Exception in thread "main" java.lang.AssertionError: found trailing 0 in chars at uk.me.parabola.imgfmt.app.labelenc.EncodedText.<init>(Encoded Te xt.java:39)
at uk.me.parabola.imgfmt.app.labelenc.AnyCharsetEncoder.encodeTe xt (AnyCharsetEncoder.java:112)
at uk.me.parabola.imgfmt.app.lbl.LBLFile.newLabel(LBLFile.java:1 32 ) at uk.me.parabola.imgfmt.app.lbl.PlacesFile.createPOI(PlacesFile .j ava:253) at uk.me.parabola.imgfmt.app.lbl.LBLFile.createPOI(LBLFile.java: 17 2) at uk.me.parabola.mkgmap.build.MapBuilder.processPOIs(MapBuilder .j ava:670) at uk.me.parabola.mkgmap.build.MapBuilder.makeMap(MapBuilder.jav a: 325) at uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:114 ) at uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:62) at uk.me.parabola.mkgmap.main.Main.lambda$processFilename$1(Main .j ava:291) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java :2 64) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(T hr eadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run( Th readPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Gerd My description didn't quite mean what I hoped it did - sorry. I was thinking that there would be a single attempt at encoding the whole string, and if that fails, start again but char-by-char. But, assuming result.length() works and charBuffer.get() and outBuff.put() maintain positions used by main encoder, within the loop the failed component needs to be processed input char-by-char, transliterated (if no change replaced by "?") and encoded with another encoder. Any variable length nature of the output charset shouldn't be a problem. The variable length input UTF-16 will need care. Ticker On Wed, 2021-11-17 at 11:16 +0000, Gerd Petermann wrote:
Hi Ticker,
remember that cs932 is a double-byte character set. With your code only a few unmappable utf-16 characters are replaced, for the rest one of cs932 is used, but without any good reason. The result is typically garbage.
I've modified the patch to replace any unmappable character that was not transliterated by '?' . I've also attached a debug version that shows what goes on. A possible change in SparseTransliterator would be to add a mapping for the MATH MINUS, the other FULLWIDTH digits are supported in cs932.
Gerd
Hi Ticker, result.length() works and most times returns 1, sometimes higher values for unicode characters which cannot be represented by a single char. OK to commit v2? Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Mittwoch, 17. November 2021 15:37 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] New assertion, now with code-page=632 and Japan tile Hi Gerd My description didn't quite mean what I hoped it did - sorry. I was thinking that there would be a single attempt at encoding the whole string, and if that fails, start again but char-by-char. But, assuming result.length() works and charBuffer.get() and outBuff.put() maintain positions used by main encoder, within the loop the failed component needs to be processed input char-by-char, transliterated (if no change replaced by "?") and encoded with another encoder. Any variable length nature of the output charset shouldn't be a problem. The variable length input UTF-16 will need care. Ticker On Wed, 2021-11-17 at 11:16 +0000, Gerd Petermann wrote:
Hi Ticker,
remember that cs932 is a double-byte character set. With your code only a few unmappable utf-16 characters are replaced, for the rest one of cs932 is used, but without any good reason. The result is typically garbage.
I've modified the patch to replace any unmappable character that was not transliterated by '?' . I've also attached a debug version that shows what goes on. A possible change in SparseTransliterator would be to add a mapping for the MATH MINUS, the other FULLWIDTH digits are supported in cs932.
Gerd
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Gerd Not quite - The transliteration / "?" doesn't get encoded into the target charset. In this case with cp932, there seems to be an assumption that SparseTranslitorator will convert all unicode chars that are not in CP932. There must be lots of these. Ticker On Wed, 2021-11-17 at 16:00 +0000, Gerd Petermann wrote:
Hi Ticker,
result.length() works and most times returns 1, sometimes higher values for unicode characters which cannot be represented by a single char.
OK to commit v2? Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Mittwoch, 17. November 2021 15:37 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] New assertion, now with code-page=632 and Japan tile
Hi Gerd
My description didn't quite mean what I hoped it did - sorry. I was thinking that there would be a single attempt at encoding the whole string, and if that fails, start again but char-by-char.
But, assuming result.length() works and charBuffer.get() and outBuff.put() maintain positions used by main encoder, within the loop the failed component needs to be processed input char-by-char, transliterated (if no change replaced by "?") and encoded with another encoder.
Any variable length nature of the output charset shouldn't be a problem. The variable length input UTF-16 will need care.
Ticker
On Wed, 2021-11-17 at 11:16 +0000, Gerd Petermann wrote:
Hi Ticker,
remember that cs932 is a double-byte character set. With your code only a few unmappable utf-16 characters are replaced, for the rest one of cs932 is used, but without any good reason. The result is typically garbage.
I've modified the patch to replace any unmappable character that was not transliterated by '?' . I've also attached a debug version that shows what goes on. A possible change in SparseTransliterator would be to add a mapping for the MATH MINUS, the other FULLWIDTH digits are supported in cs932.
Gerd
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Ticker, yes, sure, a lot of unicode characters cannot be represented in cs932. SparseTransliterator only handles 5 of them. All others are now translated to ? instead of a more or less unpredictable character. My patch doesn't try to implement a good transliteration, just a better handling of unmapped chars. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Mittwoch, 17. November 2021 17:35 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] New assertion, now with code-page=632 and Japan tile Hi Gerd Not quite - The transliteration / "?" doesn't get encoded into the target charset. In this case with cp932, there seems to be an assumption that SparseTranslitorator will convert all unicode chars that are not in CP932. There must be lots of these. Ticker On Wed, 2021-11-17 at 16:00 +0000, Gerd Petermann wrote:
Hi Ticker,
result.length() works and most times returns 1, sometimes higher values for unicode characters which cannot be represented by a single char.
OK to commit v2? Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Mittwoch, 17. November 2021 15:37 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] New assertion, now with code-page=632 and Japan tile
Hi Gerd
My description didn't quite mean what I hoped it did - sorry. I was thinking that there would be a single attempt at encoding the whole string, and if that fails, start again but char-by-char.
But, assuming result.length() works and charBuffer.get() and outBuff.put() maintain positions used by main encoder, within the loop the failed component needs to be processed input char-by-char, transliterated (if no change replaced by "?") and encoded with another encoder.
Any variable length nature of the output charset shouldn't be a problem. The variable length input UTF-16 will need care.
Ticker
On Wed, 2021-11-17 at 11:16 +0000, Gerd Petermann wrote:
Hi Ticker,
remember that cs932 is a double-byte character set. With your code only a few unmappable utf-16 characters are replaced, for the rest one of cs932 is used, but without any good reason. The result is typically garbage.
I've modified the patch to replace any unmappable character that was not transliterated by '?' . I've also attached a debug version that shows what goes on. A possible change in SparseTransliterator would be to add a mapping for the MATH MINUS, the other FULLWIDTH digits are supported in cs932.
Gerd
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Gerd It makes a big assumption that transliterated chars can have their low byte written as a sequence that will be taken as some other charset. For cp932, because so few chars are transliterated, and, by chance, these have the same representation, it won't crash or give an invalid sequence, but will be strange that only macron chars, with the macron removed, will show amongst the "?"s. For some other character sets the result could be invalid or garbage. Ticker On Wed, 2021-11-17 at 16:40 +0000, Gerd Petermann wrote:
Hi Ticker,
yes, sure, a lot of unicode characters cannot be represented in cs932. SparseTransliterator only handles 5 of them. All others are now translated to ? instead of a more or less unpredictable character. My patch doesn't try to implement a good transliteration, just a better handling of unmapped chars.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Mittwoch, 17. November 2021 17:35 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] New assertion, now with code-page=632 and Japan tile
Hi Gerd
Not quite - The transliteration / "?" doesn't get encoded into the target charset.
In this case with cp932, there seems to be an assumption that SparseTranslitorator will convert all unicode chars that are not in CP932. There must be lots of these.
Ticker
On Wed, 2021-11-17 at 16:00 +0000, Gerd Petermann wrote:
Hi Ticker,
result.length() works and most times returns 1, sometimes higher values for unicode characters which cannot be represented by a single char.
OK to commit v2? Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Mittwoch, 17. November 2021 15:37 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] New assertion, now with code-page=632 and Japan tile
Hi Gerd
My description didn't quite mean what I hoped it did - sorry. I was thinking that there would be a single attempt at encoding the whole string, and if that fails, start again but char-by-char.
But, assuming result.length() works and charBuffer.get() and outBuff.put() maintain positions used by main encoder, within the loop the failed component needs to be processed input char-by-char, transliterated (if no change replaced by "?") and encoded with another encoder.
Any variable length nature of the output charset shouldn't be a problem. The variable length input UTF-16 will need care.
Ticker
On Wed, 2021-11-17 at 11:16 +0000, Gerd Petermann wrote:
Hi Ticker,
remember that cs932 is a double-byte character set. With your code only a few unmappable utf-16 characters are replaced, for the rest one of cs932 is used, but without any good reason. The result is typically garbage.
I've modified the patch to replace any unmappable character that was not transliterated by '?' . I've also attached a debug version that shows what goes on. A possible change in SparseTransliterator would be to add a mapping for the MATH MINUS, the other FULLWIDTH digits are supported in cs932.
Gerd
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Ticker,
For some other character sets the result could be invalid or garbage. OK, I assumed that '?' is always at the same position, might be wrong with that. SparseTransliterator is only used for cs932.
Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Mittwoch, 17. November 2021 18:23 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] New assertion, now with code-page=632 and Japan tile Hi Gerd It makes a big assumption that transliterated chars can have their low byte written as a sequence that will be taken as some other charset. For cp932, because so few chars are transliterated, and, by chance, these have the same representation, it won't crash or give an invalid sequence, but will be strange that only macron chars, with the macron removed, will show amongst the "?"s. For some other character sets the result could be invalid or garbage. Ticker On Wed, 2021-11-17 at 16:40 +0000, Gerd Petermann wrote:
Hi Ticker,
yes, sure, a lot of unicode characters cannot be represented in cs932. SparseTransliterator only handles 5 of them. All others are now translated to ? instead of a more or less unpredictable character. My patch doesn't try to implement a good transliteration, just a better handling of unmapped chars.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Mittwoch, 17. November 2021 17:35 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] New assertion, now with code-page=632 and Japan tile
Hi Gerd
Not quite - The transliteration / "?" doesn't get encoded into the target charset.
In this case with cp932, there seems to be an assumption that SparseTranslitorator will convert all unicode chars that are not in CP932. There must be lots of these.
Ticker
On Wed, 2021-11-17 at 16:00 +0000, Gerd Petermann wrote:
Hi Ticker,
result.length() works and most times returns 1, sometimes higher values for unicode characters which cannot be represented by a single char.
OK to commit v2? Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Mittwoch, 17. November 2021 15:37 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] New assertion, now with code-page=632 and Japan tile
Hi Gerd
My description didn't quite mean what I hoped it did - sorry. I was thinking that there would be a single attempt at encoding the whole string, and if that fails, start again but char-by-char.
But, assuming result.length() works and charBuffer.get() and outBuff.put() maintain positions used by main encoder, within the loop the failed component needs to be processed input char-by-char, transliterated (if no change replaced by "?") and encoded with another encoder.
Any variable length nature of the output charset shouldn't be a problem. The variable length input UTF-16 will need care.
Ticker
On Wed, 2021-11-17 at 11:16 +0000, Gerd Petermann wrote:
Hi Ticker,
remember that cs932 is a double-byte character set. With your code only a few unmappable utf-16 characters are replaced, for the rest one of cs932 is used, but without any good reason. The result is typically garbage.
I've modified the patch to replace any unmappable character that was not transliterated by '?' . I've also attached a debug version that shows what goes on. A possible change in SparseTransliterator would be to add a mapping for the MATH MINUS, the other FULLWIDTH digits are supported in cs932.
Gerd
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Gerd For any code-page except Japanese/cp932, AnyCharSetEncoder takes anything that can't be represented, tries to find a reasonable ascii representation or "?", then writes this to the output. This is a big assumption for far-eastern charsets, most likely generating garbage with possible invalid shift-in/out requests... SparseTranslitorator is a very strange special case, without any explanation. Doing a bit of searching, it was submitted as a change because user had map that needed to be in Japanese/cp932 and it also contained latin characters. The characters with macrons couldn't be encoded. Many others could. The rest of Unicode that can't be encoded resulted in garbage. Your patch fixes the "rest of Unicode" problem for cp932. It misses any ability of the 'latin1' transliterator to provide reasonable replacement chars that can be encoded. It doesn't deal with possible problems for other (non-european) charsets. I've attached cs932-V3.patch that addresses both of these issues. SparseTranslitorator.java can the be removed. Ticker On Wed, 2021-11-17 at 18:00 +0000, Gerd Petermann wrote:
Hi Ticker,
For some other character sets the result could be invalid or garbage. OK, I assumed that '?' is always at the same position, might be wrong with that. SparseTransliterator is only used for cs932.
Gerd
Hi Gerd Please ignore the CodeFunctions.java change in this patch. A better / hybrid TableTranslitorator is needed before SparseTranslitorator can go. Ticker On Thu, 2021-11-18 at 17:47 +0000, Ticker Berkin wrote:
Hi Gerd For any code-page except Japanese/cp932, AnyCharSetEncoder takes anything that can't be represented, tries to find a reasonable ascii representation or "?", then writes this to the output. This is a big assumption for far-eastern charsets, most likely generating garbage with possible invalid shift-in/out requests...
SparseTranslitorator is a very strange special case, without any explanation. Doing a bit of searching, it was submitted as a change because user had map that needed to be in Japanese/cp932 and it also contained latin characters. The characters with macrons couldn't be encoded. Many others could. The rest of Unicode that can't be encoded resulted in garbage.
Your patch fixes the "rest of Unicode" problem for cp932. It misses any ability of the 'latin1' transliterator to provide reasonable replacement chars that can be encoded. It doesn't deal with possible problems for other (non-european) charsets.
I've attached cs932-V3.patch that addresses both of these issues.
SparseTranslitorator.java can the be removed.
Ticker
On Wed, 2021-11-17 at 18:00 +0000, Gerd Petermann wrote:
Hi Ticker,
For some other character sets the result could be invalid or garbage. OK, I assumed that '?' is always at the same position, might be wrong with that. SparseTransliterator is only used for cs932.
Gerd
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Ticker, no problem, still trying to find a case where mdrUnicode_v9b.patch changes something... Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Freitag, 19. November 2021 08:46 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] New assertion, now with code-page=632 and Japan tile Hi Gerd Please ignore the CodeFunctions.java change in this patch. A better / hybrid TableTranslitorator is needed before SparseTranslitorator can go. Ticker On Thu, 2021-11-18 at 17:47 +0000, Ticker Berkin wrote:
Hi Gerd
For any code-page except Japanese/cp932, AnyCharSetEncoder takes anything that can't be represented, tries to find a reasonable ascii representation or "?", then writes this to the output. This is a big assumption for far-eastern charsets, most likely generating garbage with possible invalid shift-in/out requests...
SparseTranslitorator is a very strange special case, without any explanation. Doing a bit of searching, it was submitted as a change because user had map that needed to be in Japanese/cp932 and it also contained latin characters. The characters with macrons couldn't be encoded. Many others could. The rest of Unicode that can't be encoded resulted in garbage.
Your patch fixes the "rest of Unicode" problem for cp932. It misses any ability of the 'latin1' transliterator to provide reasonable replacement chars that can be encoded. It doesn't deal with possible problems for other (non-european) charsets.
I've attached cs932-V3.patch that addresses both of these issues.
SparseTranslitorator.java can the be removed.
Ticker
On Wed, 2021-11-17 at 18:00 +0000, Gerd Petermann wrote:
Hi Ticker,
For some other character sets the result could be invalid or garbage. OK, I assumed that '?' is always at the same position, might be wrong with that. SparseTransliterator is only used for cs932.
Gerd
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
I'm aware of the supposed advantages of pbf, but I did some tests in the past and I found o5m to be quite faster than pbf, but I'll test again. El 10/11/21 a las 9:43, Gerd Petermann escribió:
Hi Carlos,
I'll try to debug this.
BTW: I see you use *.o5m for the tiles (output from splitter). I think this is no longer a good choice, pbf is a lot smaller and almost as fast. Esp. when it comes to the goal of reducing disk I/O (as with --gmapi-minimal)
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Carlos Dávila <carlos@alternativaslibres.org> Gesendet: Dienstag, 9. November 2021 22:54 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] New assertion, now with code-page=632 and Japan tile
Hi Ticker
Not sure if relevant, but note in this case assertion occurs while compiling the tile, not the index. In fact, --index is not included in the command.
El 9/11/21 a las 21:55, Ticker Berkin escribió:
Hi
I think this assertion could be removed from the code.
Looking through the definition of Shift-JIS, I read it as saying the second byte shouldn't be zero, so I don't know why this happens.
As with the Chinese code-pages, mkgmap has places where multi-byte encodings are not handled correctly in the --index generation and unknown meanings of flags to the Garmin software.
Ticker
On 09/11/2021 19:43, Carlos Dávila wrote:
code-page=932, sorry for the typo.
El 9/11/21 a las 20:36, Carlos Dávila escribió:
The command below produces an assertion while compiling this tile <https://files.mkgmap.org.uk/download/526/31191025.o5m> from Japan. Process continues with remaining tiles and finishes without "Number of MapFailedExceptions: 1" as expected. This is with r4813, but I also tried with an old version of mkgmap with the same result.
java -Xmx27G -ea -jar mkgmap.jar--code-page=632 31191025.o5m Mkgmap version 4813 Time started: Tue Nov 09 20:18:16 CET 2021 WARNING (global): Setting max-jobs to 8 Exception in thread "main" java.lang.AssertionError: found trailing 0 in chars at uk.me.parabola.imgfmt.app.labelenc.EncodedText.<init>(EncodedText.java:39)
at uk.me.parabola.imgfmt.app.labelenc.AnyCharsetEncoder.encodeText(AnyCharsetEncoder.java:112)
at uk.me.parabola.imgfmt.app.lbl.LBLFile.newLabel(LBLFile.java:132) at uk.me.parabola.imgfmt.app.lbl.PlacesFile.createPOI(PlacesFile.java:253) at uk.me.parabola.imgfmt.app.lbl.LBLFile.createPOI(LBLFile.java:172) at uk.me.parabola.mkgmap.build.MapBuilder.processPOIs(MapBuilder.java:670) at uk.me.parabola.mkgmap.build.MapBuilder.makeMap(MapBuilder.java:325) at uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:114) at uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:62) at uk.me.parabola.mkgmap.main.Main.lambda$processFilename$1(Main.java:291) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Only as input for splitter it changes a lot. As input for mkgmap it's very little difference. On Wed, 10 Nov 2021, 17:38 Carlos Dávila <carlos@alternativaslibres.org> wrote:
I'm aware of the supposed advantages of pbf, but I did some tests in the past and I found o5m to be quite faster than pbf, but I'll test again.
El 10/11/21 a las 9:43, Gerd Petermann escribió:
Hi Carlos,
I'll try to debug this.
BTW: I see you use *.o5m for the tiles (output from splitter). I think this is no longer a good choice, pbf is a lot smaller and almost as fast. Esp. when it comes to the goal of reducing disk I/O (as with --gmapi-minimal)
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Carlos Dávila <carlos@alternativaslibres.org> Gesendet: Dienstag, 9. November 2021 22:54 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] New assertion, now with code-page=632 and Japan tile
Hi Ticker
Not sure if relevant, but note in this case assertion occurs while compiling the tile, not the index. In fact, --index is not included in the command.
El 9/11/21 a las 21:55, Ticker Berkin escribió:
Hi
I think this assertion could be removed from the code.
Looking through the definition of Shift-JIS, I read it as saying the second byte shouldn't be zero, so I don't know why this happens.
As with the Chinese code-pages, mkgmap has places where multi-byte encodings are not handled correctly in the --index generation and unknown meanings of flags to the Garmin software.
Ticker
On 09/11/2021 19:43, Carlos Dávila wrote:
code-page=932, sorry for the typo.
El 9/11/21 a las 20:36, Carlos Dávila escribió:
The command below produces an assertion while compiling this tile <https://files.mkgmap.org.uk/download/526/31191025.o5m> from Japan. Process continues with remaining tiles and finishes without "Number of MapFailedExceptions: 1" as expected. This is with r4813, but I also tried with an old version of mkgmap with the same result.
java -Xmx27G -ea -jar mkgmap.jar--code-page=632 31191025.o5m Mkgmap version 4813 Time started: Tue Nov 09 20:18:16 CET 2021 WARNING (global): Setting max-jobs to 8 Exception in thread "main" java.lang.AssertionError: found trailing 0 in chars at
uk.me.parabola.imgfmt.app.labelenc.EncodedText.<init>(EncodedText.java:39)
at
uk.me.parabola.imgfmt.app.labelenc.AnyCharsetEncoder.encodeText(AnyCharsetEncoder.java:112)
at uk.me.parabola.imgfmt.app.lbl.LBLFile.newLabel(LBLFile.java:132) at
uk.me.parabola.imgfmt.app.lbl.PlacesFile.createPOI(PlacesFile.java:253)
at uk.me.parabola.imgfmt.app.lbl.LBLFile.createPOI(LBLFile.java:172) at
uk.me.parabola.mkgmap.build.MapBuilder.processPOIs(MapBuilder.java:670)
at uk.me.parabola.mkgmap.build.MapBuilder.makeMap(MapBuilder.java:325) at uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:114) at uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:62) at
uk.me.parabola.mkgmap.main.Main.lambda$processFilename$1(Main.java:291)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
O5m is always faster, but needs more disk space. Converting geofabrik osm.pbf to o5m makes sense however (especially if you can write to Ramdisk or slow Harddisk) On Wed, 10 Nov 2021, 17:38 Carlos Dávila <carlos@alternativaslibres.org> wrote:
I'm aware of the supposed advantages of pbf, but I did some tests in the past and I found o5m to be quite faster than pbf, but I'll test again.
El 10/11/21 a las 9:43, Gerd Petermann escribió:
Hi Carlos,
I'll try to debug this.
BTW: I see you use *.o5m for the tiles (output from splitter). I think this is no longer a good choice, pbf is a lot smaller and almost as fast. Esp. when it comes to the goal of reducing disk I/O (as with --gmapi-minimal)
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Carlos Dávila <carlos@alternativaslibres.org> Gesendet: Dienstag, 9. November 2021 22:54 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] New assertion, now with code-page=632 and Japan tile
Hi Ticker
Not sure if relevant, but note in this case assertion occurs while compiling the tile, not the index. In fact, --index is not included in the command.
El 9/11/21 a las 21:55, Ticker Berkin escribió:
Hi
I think this assertion could be removed from the code.
Looking through the definition of Shift-JIS, I read it as saying the second byte shouldn't be zero, so I don't know why this happens.
As with the Chinese code-pages, mkgmap has places where multi-byte encodings are not handled correctly in the --index generation and unknown meanings of flags to the Garmin software.
Ticker
On 09/11/2021 19:43, Carlos Dávila wrote:
code-page=932, sorry for the typo.
El 9/11/21 a las 20:36, Carlos Dávila escribió:
The command below produces an assertion while compiling this tile <https://files.mkgmap.org.uk/download/526/31191025.o5m> from Japan. Process continues with remaining tiles and finishes without "Number of MapFailedExceptions: 1" as expected. This is with r4813, but I also tried with an old version of mkgmap with the same result.
java -Xmx27G -ea -jar mkgmap.jar--code-page=632 31191025.o5m Mkgmap version 4813 Time started: Tue Nov 09 20:18:16 CET 2021 WARNING (global): Setting max-jobs to 8 Exception in thread "main" java.lang.AssertionError: found trailing 0 in chars at
uk.me.parabola.imgfmt.app.labelenc.EncodedText.<init>(EncodedText.java:39)
at
uk.me.parabola.imgfmt.app.labelenc.AnyCharsetEncoder.encodeText(AnyCharsetEncoder.java:112)
at uk.me.parabola.imgfmt.app.lbl.LBLFile.newLabel(LBLFile.java:132) at
uk.me.parabola.imgfmt.app.lbl.PlacesFile.createPOI(PlacesFile.java:253)
at uk.me.parabola.imgfmt.app.lbl.LBLFile.createPOI(LBLFile.java:172) at
uk.me.parabola.mkgmap.build.MapBuilder.processPOIs(MapBuilder.java:670)
at uk.me.parabola.mkgmap.build.MapBuilder.makeMap(MapBuilder.java:325) at uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:114) at uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:62) at
uk.me.parabola.mkgmap.main.Main.lambda$processFilename$1(Main.java:291)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
I don't use Geofabrik extracts but my own ones. I keep country.o5m files that are updated with osmupdate as a first step for map updates. El 10/11/21 a las 22:59, Felix Hartmann escribió:
O5m is always faster, but needs more disk space. Converting geofabrik osm.pbf to o5m makes sense however (especially if you can write to Ramdisk or slow Harddisk)
On Wed, 10 Nov 2021, 17:38 Carlos Dávila <carlos@alternativaslibres.org <mailto:carlos@alternativaslibres.org>> wrote:
I'm aware of the supposed advantages of pbf, but I did some tests in the past and I found o5m to be quite faster than pbf, but I'll test again.
El 10/11/21 a las 9:43, Gerd Petermann escribió: > Hi Carlos, > > I'll try to debug this. > > BTW: I see you use *.o5m for the tiles (output from splitter). I think this is no longer a good choice, pbf is a lot smaller and almost as fast. Esp. when it comes to the goal of reducing disk I/O (as with --gmapi-minimal) > > Gerd > > ________________________________________ > Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk <mailto:mkgmap-dev-bounces@lists.mkgmap.org.uk>> im Auftrag von Carlos Dávila <carlos@alternativaslibres.org <mailto:carlos@alternativaslibres.org>> > Gesendet: Dienstag, 9. November 2021 22:54 > An: mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> > Betreff: Re: [mkgmap-dev] New assertion, now with code-page=632 and Japan tile > > Hi Ticker > > Not sure if relevant, but note in this case assertion occurs while > compiling the tile, not the index. In fact, --index is not included in > the command. > > El 9/11/21 a las 21:55, Ticker Berkin escribió: >> Hi >> >> I think this assertion could be removed from the code. >> >> Looking through the definition of Shift-JIS, I read it as saying the >> second byte shouldn't be zero, so I don't know why this happens. >> >> As with the Chinese code-pages, mkgmap has places where multi-byte >> encodings are not handled correctly in the --index generation and >> unknown meanings of flags to the Garmin software. >> >> Ticker >> >> >> >> On 09/11/2021 19:43, Carlos Dávila wrote: >>> code-page=932, sorry for the typo. >>> >>> El 9/11/21 a las 20:36, Carlos Dávila escribió: >>>> The command below produces an assertion while compiling this tile >>>> <https://files.mkgmap.org.uk/download/526/31191025.o5m <https://files.mkgmap.org.uk/download/526/31191025.o5m>> from Japan. >>>> Process continues with remaining tiles and finishes without "Number >>>> of MapFailedExceptions: 1" as expected. This is with r4813, but I >>>> also tried with an old version of mkgmap with the same result. >>>> >>>> java -Xmx27G -ea -jar mkgmap.jar--code-page=632 31191025.o5m >>>> Mkgmap version 4813 >>>> Time started: Tue Nov 09 20:18:16 CET 2021 >>>> WARNING (global): Setting max-jobs to 8 >>>> Exception in thread "main" java.lang.AssertionError: found trailing >>>> 0 in chars >>>> at >>>> uk.me.parabola.imgfmt.app.labelenc.EncodedText.<init>(EncodedText.java:39) >>>> >>>> at >>>> uk.me.parabola.imgfmt.app.labelenc.AnyCharsetEncoder.encodeText(AnyCharsetEncoder.java:112) >>>> >>>> at >>>> uk.me.parabola.imgfmt.app.lbl.LBLFile.newLabel(LBLFile.java:132) >>>> at >>>> uk.me.parabola.imgfmt.app.lbl.PlacesFile.createPOI(PlacesFile.java:253) >>>> at >>>> uk.me.parabola.imgfmt.app.lbl.LBLFile.createPOI(LBLFile.java:172) >>>> at >>>> uk.me.parabola.mkgmap.build.MapBuilder.processPOIs(MapBuilder.java:670) >>>> at >>>> uk.me.parabola.mkgmap.build.MapBuilder.makeMap(MapBuilder.java:325) >>>> at >>>> uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:114) >>>> at >>>> uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:62) >>>> at >>>> uk.me.parabola.mkgmap.main.Main.lambda$processFilename$1(Main.java:291) >>>> at >>>> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) >>>> at >>>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) >>>> >>>> at >>>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) >>>> >>>> at java.base/java.lang.Thread.run(Thread.java:829) >>>> >>>> >>>> _______________________________________________ >>>> mkgmap-dev mailing list >>>> mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> >>>> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev <https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev> >>> _______________________________________________ >>> mkgmap-dev mailing list >>> mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> >>> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev <https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev> >> _______________________________________________ >> mkgmap-dev mailing list >> mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> >> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev <https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev> > _______________________________________________ > mkgmap-dev mailing list > mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> > https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev <https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev> > _______________________________________________ > mkgmap-dev mailing list > mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> > https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev <https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev>
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev <https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev>
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Following are the results processing Portugal: Size osmupdate split into o5m split into pbf mkgmap portugal.o5m 713 MB 0:00:36 0:00:38 0:00:46 o5m: 0:01:25 portugal.pbf 410 MB 0:01:02 0:02:09 0:02:20 pbf: 0:01:27 With these figures the only reasonable option for me would be to output pbf from splitter to reduce disk I/O. Time penalty for the other steps is too high for me. Currently my server is working some 12 hours/day updating maps. El 10/11/21 a las 22:59, Felix Hartmann escribió:
O5m is always faster, but needs more disk space. Converting geofabrik osm.pbf to o5m makes sense however (especially if you can write to Ramdisk or slow Harddisk)
On Wed, 10 Nov 2021, 17:38 Carlos Dávila <carlos@alternativaslibres.org <mailto:carlos@alternativaslibres.org>> wrote:
I'm aware of the supposed advantages of pbf, but I did some tests in the past and I found o5m to be quite faster than pbf, but I'll test again.
El 10/11/21 a las 9:43, Gerd Petermann escribió: > Hi Carlos, > > I'll try to debug this. > > BTW: I see you use *.o5m for the tiles (output from splitter). I think this is no longer a good choice, pbf is a lot smaller and almost as fast. Esp. when it comes to the goal of reducing disk I/O (as with --gmapi-minimal) > > Gerd > > ________________________________________ > Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk <mailto:mkgmap-dev-bounces@lists.mkgmap.org.uk>> im Auftrag von Carlos Dávila <carlos@alternativaslibres.org <mailto:carlos@alternativaslibres.org>> > Gesendet: Dienstag, 9. November 2021 22:54 > An: mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> > Betreff: Re: [mkgmap-dev] New assertion, now with code-page=632 and Japan tile > > Hi Ticker > > Not sure if relevant, but note in this case assertion occurs while > compiling the tile, not the index. In fact, --index is not included in > the command. > > El 9/11/21 a las 21:55, Ticker Berkin escribió: >> Hi >> >> I think this assertion could be removed from the code. >> >> Looking through the definition of Shift-JIS, I read it as saying the >> second byte shouldn't be zero, so I don't know why this happens. >> >> As with the Chinese code-pages, mkgmap has places where multi-byte >> encodings are not handled correctly in the --index generation and >> unknown meanings of flags to the Garmin software. >> >> Ticker >> >> >> >> On 09/11/2021 19:43, Carlos Dávila wrote: >>> code-page=932, sorry for the typo. >>> >>> El 9/11/21 a las 20:36, Carlos Dávila escribió: >>>> The command below produces an assertion while compiling this tile >>>> <https://files.mkgmap.org.uk/download/526/31191025.o5m <https://files.mkgmap.org.uk/download/526/31191025.o5m>> from Japan. >>>> Process continues with remaining tiles and finishes without "Number >>>> of MapFailedExceptions: 1" as expected. This is with r4813, but I >>>> also tried with an old version of mkgmap with the same result. >>>> >>>> java -Xmx27G -ea -jar mkgmap.jar--code-page=632 31191025.o5m >>>> Mkgmap version 4813 >>>> Time started: Tue Nov 09 20:18:16 CET 2021 >>>> WARNING (global): Setting max-jobs to 8 >>>> Exception in thread "main" java.lang.AssertionError: found trailing >>>> 0 in chars >>>> at >>>> uk.me.parabola.imgfmt.app.labelenc.EncodedText.<init>(EncodedText.java:39) >>>> >>>> at >>>> uk.me.parabola.imgfmt.app.labelenc.AnyCharsetEncoder.encodeText(AnyCharsetEncoder.java:112) >>>> >>>> at >>>> uk.me.parabola.imgfmt.app.lbl.LBLFile.newLabel(LBLFile.java:132) >>>> at >>>> uk.me.parabola.imgfmt.app.lbl.PlacesFile.createPOI(PlacesFile.java:253) >>>> at >>>> uk.me.parabola.imgfmt.app.lbl.LBLFile.createPOI(LBLFile.java:172) >>>> at >>>> uk.me.parabola.mkgmap.build.MapBuilder.processPOIs(MapBuilder.java:670) >>>> at >>>> uk.me.parabola.mkgmap.build.MapBuilder.makeMap(MapBuilder.java:325) >>>> at >>>> uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:114) >>>> at >>>> uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:62) >>>> at >>>> uk.me.parabola.mkgmap.main.Main.lambda$processFilename$1(Main.java:291) >>>> at >>>> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) >>>> at >>>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) >>>> >>>> at >>>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) >>>> >>>> at java.base/java.lang.Thread.run(Thread.java:829) >>>> >>>> >>>> _______________________________________________ >>>> mkgmap-dev mailing list >>>> mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> >>>> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev <https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev> >>> _______________________________________________ >>> mkgmap-dev mailing list >>> mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> >>> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev <https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev> >> _______________________________________________ >> mkgmap-dev mailing list >> mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> >> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev <https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev> > _______________________________________________ > mkgmap-dev mailing list > mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> > https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev <https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev> > _______________________________________________ > mkgmap-dev mailing list > mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> > https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev <https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev>
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev <https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev>
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Carlos, yes, I only suggested to use pbf for the output of splitter. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Carlos Dávila <carlos@alternativaslibres.org> Gesendet: Donnerstag, 11. November 2021 19:22 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] New assertion, now with code-page=632 and Japan tile Following are the results processing Portugal: Size osmupdate split into o5m split into pbf mkgmap portugal.o5m 713 MB 0:00:36 0:00:38 0:00:46 o5m: 0:01:25 portugal.pbf 410 MB 0:01:02 0:02:09 0:02:20 pbf: 0:01:27 With these figures the only reasonable option for me would be to output pbf from splitter to reduce disk I/O. Time penalty for the other steps is too high for me. Currently my server is working some 12 hours/day updating maps. El 10/11/21 a las 22:59, Felix Hartmann escribió:
O5m is always faster, but needs more disk space. Converting geofabrik osm.pbf to o5m makes sense however (especially if you can write to Ramdisk or slow Harddisk)
On Wed, 10 Nov 2021, 17:38 Carlos Dávila <carlos@alternativaslibres.org <mailto:carlos@alternativaslibres.org>> wrote:
I'm aware of the supposed advantages of pbf, but I did some tests in the past and I found o5m to be quite faster than pbf, but I'll test again.
El 10/11/21 a las 9:43, Gerd Petermann escribió: > Hi Carlos, > > I'll try to debug this. > > BTW: I see you use *.o5m for the tiles (output from splitter). I think this is no longer a good choice, pbf is a lot smaller and almost as fast. Esp. when it comes to the goal of reducing disk I/O (as with --gmapi-minimal) > > Gerd > > ________________________________________ > Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk <mailto:mkgmap-dev-bounces@lists.mkgmap.org.uk>> im Auftrag von Carlos Dávila <carlos@alternativaslibres.org <mailto:carlos@alternativaslibres.org>> > Gesendet: Dienstag, 9. November 2021 22:54 > An: mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> > Betreff: Re: [mkgmap-dev] New assertion, now with code-page=632 and Japan tile > > Hi Ticker > > Not sure if relevant, but note in this case assertion occurs while > compiling the tile, not the index. In fact, --index is not included in > the command. > > El 9/11/21 a las 21:55, Ticker Berkin escribió: >> Hi >> >> I think this assertion could be removed from the code. >> >> Looking through the definition of Shift-JIS, I read it as saying the >> second byte shouldn't be zero, so I don't know why this happens. >> >> As with the Chinese code-pages, mkgmap has places where multi-byte >> encodings are not handled correctly in the --index generation and >> unknown meanings of flags to the Garmin software. >> >> Ticker >> >> >> >> On 09/11/2021 19:43, Carlos Dávila wrote: >>> code-page=932, sorry for the typo. >>> >>> El 9/11/21 a las 20:36, Carlos Dávila escribió: >>>> The command below produces an assertion while compiling this tile >>>> <https://files.mkgmap.org.uk/download/526/31191025.o5m <https://files.mkgmap.org.uk/download/526/31191025.o5m>> from Japan. >>>> Process continues with remaining tiles and finishes without "Number >>>> of MapFailedExceptions: 1" as expected. This is with r4813, but I >>>> also tried with an old version of mkgmap with the same result. >>>> >>>> java -Xmx27G -ea -jar mkgmap.jar--code-page=632 31191025.o5m >>>> Mkgmap version 4813 >>>> Time started: Tue Nov 09 20:18:16 CET 2021 >>>> WARNING (global): Setting max-jobs to 8 >>>> Exception in thread "main" java.lang.AssertionError: found trailing >>>> 0 in chars >>>> at >>>> uk.me.parabola.imgfmt.app.labelenc.EncodedText.<init>(EncodedText.java:39) >>>> >>>> at >>>> uk.me.parabola.imgfmt.app.labelenc.AnyCharsetEncoder.encodeText(AnyCharsetEncoder.java:112) >>>> >>>> at >>>> uk.me.parabola.imgfmt.app.lbl.LBLFile.newLabel(LBLFile.java:132) >>>> at >>>> uk.me.parabola.imgfmt.app.lbl.PlacesFile.createPOI(PlacesFile.java:253) >>>> at >>>> uk.me.parabola.imgfmt.app.lbl.LBLFile.createPOI(LBLFile.java:172) >>>> at >>>> uk.me.parabola.mkgmap.build.MapBuilder.processPOIs(MapBuilder.java:670) >>>> at >>>> uk.me.parabola.mkgmap.build.MapBuilder.makeMap(MapBuilder.java:325) >>>> at >>>> uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:114) >>>> at >>>> uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:62) >>>> at >>>> uk.me.parabola.mkgmap.main.Main.lambda$processFilename$1(Main.java:291) >>>> at >>>> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) >>>> at >>>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) >>>> >>>> at >>>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) >>>> >>>> at java.base/java.lang.Thread.run(Thread.java:829) >>>> >>>> >>>> _______________________________________________ >>>> mkgmap-dev mailing list >>>> mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> >>>> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev <https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev> >>> _______________________________________________ >>> mkgmap-dev mailing list >>> mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> >>> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev <https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev> >> _______________________________________________ >> mkgmap-dev mailing list >> mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> >> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev <https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev> > _______________________________________________ > mkgmap-dev mailing list > mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> > https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev <https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev> > _______________________________________________ > mkgmap-dev mailing list > mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> > https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev <https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev>
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev <https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev>
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
participants (5)
-
Carlos Dávila -
Felix Hartmann -
Gerd Petermann -
Ticker Berkin -
Ticker Berkin