Hi Ticker, no problem, still trying to find a case where mdrUnicode_v9b.patch changes something... Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Freitag, 19. November 2021 08:46 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] New assertion, now with code-page=632 and Japan tile Hi Gerd Please ignore the CodeFunctions.java change in this patch. A better / hybrid TableTranslitorator is needed before SparseTranslitorator can go. Ticker On Thu, 2021-11-18 at 17:47 +0000, Ticker Berkin wrote:
Hi Gerd
For any code-page except Japanese/cp932, AnyCharSetEncoder takes anything that can't be represented, tries to find a reasonable ascii representation or "?", then writes this to the output. This is a big assumption for far-eastern charsets, most likely generating garbage with possible invalid shift-in/out requests...
SparseTranslitorator is a very strange special case, without any explanation. Doing a bit of searching, it was submitted as a change because user had map that needed to be in Japanese/cp932 and it also contained latin characters. The characters with macrons couldn't be encoded. Many others could. The rest of Unicode that can't be encoded resulted in garbage.
Your patch fixes the "rest of Unicode" problem for cp932. It misses any ability of the 'latin1' transliterator to provide reasonable replacement chars that can be encoded. It doesn't deal with possible problems for other (non-european) charsets.
I've attached cs932-V3.patch that addresses both of these issues.
SparseTranslitorator.java can the be removed.
Ticker
On Wed, 2021-11-17 at 18:00 +0000, Gerd Petermann wrote:
Hi Ticker,
For some other character sets the result could be invalid or garbage. OK, I assumed that '?' is always at the same position, might be wrong with that. SparseTransliterator is only used for cs932.
Gerd
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev