Hi, maybe a regression from r4167? http://www.mkgmap.org.uk/websvn/revision.php?repname=mkgmap&rev=4167 Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von osm@pinns <osm@pinns.co.uk> Gesendet: Samstag, 6. April 2019 10:01 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] Building map with Hebrew characters Have done some checking using older versions It works on 4143 but not on 4179 I'm afraid I don't have anything in between r Nick On 05/04/2019 21:25, Carlos Dávila wrote:

I'm trying to build a map of Israel with Hebrew characters, but using --code-page=1255 a get the following assertion:

at uk.me.parabola.imgfmt.app.srt.SRTFile.writeWeights(SRTFile.java:117) at uk.me.parabola.imgfmt.app.srt.SRTFile.writeSrt4Chars(SRTFile.java:99) at uk.me.parabola.imgfmt.app.srt.SRTFile.write(SRTFile.java:63) at uk.me.parabola.mkgmap.combiners.MdrBuilder.init(MdrBuilder.java:117) at uk.me.parabola.mkgmap.main.Main.endOptions(Main.java:605) at uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.java:128) at uk.me.parabola.mkgmap.main.Main.mainStart(Main.java:143) at uk.me.parabola.mkgmap.main.Main.main(Main.java:114)

With --unicode I also get an assertion: Exception in thread "main" java.lang.AssertionError: -125 at uk.me.parabola.imgfmt.app.BufferedImgFileWriter.put1u(BufferedImgFileWriter.java:154) at uk.me.parabola.imgfmt.app.SectionWriter.put1u(SectionWriter.java:78) at uk.me.parabola.imgfmt.app.srt.SRTFile.writeWeights(SRTFile.java:113) at uk.me.parabola.imgfmt.app.srt.SRTFile.writeSrt8(SRTFile.java:166) at uk.me.parabola.imgfmt.app.srt.SRTFile.write(SRTFile.java:68) at uk.me.parabola.mkgmap.combiners.MdrBuilder.init(MdrBuilder.java:117) at uk.me.parabola.mkgmap.main.Main.endOptions(Main.java:605) at uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.java:128) at uk.me.parabola.mkgmap.main.Main.mainStart(Main.java:143) at uk.me.parabola.mkgmap.main.Main.main(Main.java:114)

Probably I'm missing something obvious, but I always used --code-page=1251 and have no idea what it may be

_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Hi Gerd and Nick @nick, when you say it works on 4143 but not on 4179, did you test code -page=1255 on 4143, or just unicode? It looks like there are 2 problems. The --unicode assertion error is a 'byte' to 'int' sign-extension problem, introduced in r4167 or r4168 and I've attached a patch that fixes this and also removes some other confusing/obscuring type representation changes in this area of the code. The patch also has 4 commented out (because not appropriate for - -unicode) asserts that seem to point to the Hebrew/1255 problem being in imgfmt/app/srt/Sort.java:finish(): $ java -ea -jar /mkgmap/mkgmap.jar --code-page=1255 --gmapsupp test -map:all-elements Time started: Mon Apr 08 09:30:06 BST 2019 Exception in thread "main" java.lang.AssertionError: 16 at uk.me.parabola.imgfmt.app.srt.Sort$Page.setSecondary(Sort.java:642) at uk.me.parabola.imgfmt.app.srt.Sort.finish(Sort.java:161) at uk.me.parabola.mkgmap.srt.SrtTextReader.read(SrtTextReader.java:182) at uk.me.parabola.mkgmap.srt.SrtTextReader.<init>(SrtTextReader.java:117) at uk.me.parabola.mkgmap.srt.SrtTextReader.<init>(SrtTextReader.java:106) at uk.me.parabola.mkgmap.srt.SrtTextReader.sortForCodepage(SrtTextReader.j ava:141) at uk.me.parabola.mkgmap.main.Main.getSort(Main.java:764) at uk.me.parabola.mkgmap.main.Main.processFilename(Main.java:283) at uk.me.parabola.mkgmap.CommandArgsReader$Filename.processArg(CommandArgs Reader.java:272) at uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.java :125) at uk.me.parabola.mkgmap.main.Main.mainStart(Main.java:143) at uk.me.parabola.mkgmap.main.Main.main(Main.java:114) this is corresponds to: } else { p.setSecondary(i, p.getSecondary(i) + maxSecondary); } where this increase in the secondary value is governed by some bits in the pages.flags and the flags testing was changed in r3968, along with the introduction of the original assert that was failing (SRTFile.java:117) and quite a few other changes in srt. @gerd, I'm afraid I don't understand the srt logic so can you have a look at it Regards Ticker On Sat, 2019-04-06 at 08:16 +0000, Gerd Petermann wrote:

Hi, maybe a regression from r4167? http://www.mkgmap.org.uk/websvn/revision.php?repname=mkgmap&rev=4167

Gerd

________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von osm@pinns <osm@pinns.co.uk> Gesendet: Samstag, 6. April 2019 10:01 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] Building map with Hebrew characters

Have done some checking using older versions

It works on 4143 but not on 4179

I'm afraid I don't have anything in between

r

Nick

On 05/04/2019 21:25, Carlos Dávila wrote:

...

I'm trying to build a map of Israel with Hebrew characters, but using --code-page=1255 a get the following assertion:

at uk.me.parabola.imgfmt.app.srt.SRTFile.writeWeights(SRTFile.java:117 ) at uk.me.parabola.imgfmt.app.srt.SRTFile.writeSrt4Chars(SRTFile.java:9 9) at uk.me.parabola.imgfmt.app.srt.SRTFile.write(SRTFile.java:63) at uk.me.parabola.mkgmap.combiners.MdrBuilder.init(MdrBuilder.java:117 ) at uk.me.parabola.mkgmap.main.Main.endOptions(Main.java:605) at uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader. java:128) at uk.me.parabola.mkgmap.main.Main.mainStart(Main.java:143) at uk.me.parabola.mkgmap.main.Main.main(Main.java:114)

With --unicode I also get an assertion: Exception in thread "main" java.lang.AssertionError: -125 at uk.me.parabola.imgfmt.app.BufferedImgFileWriter.put1u(BufferedImgFi leWriter.java:154) at uk.me.parabola.imgfmt.app.SectionWriter.put1u(SectionWriter.java:78 ) at uk.me.parabola.imgfmt.app.srt.SRTFile.writeWeights(SRTFile.java:113 ) at uk.me.parabola.imgfmt.app.srt.SRTFile.writeSrt8(SRTFile.java:166) at uk.me.parabola.imgfmt.app.srt.SRTFile.write(SRTFile.java:68) at uk.me.parabola.mkgmap.combiners.MdrBuilder.init(MdrBuilder.java:117 ) at uk.me.parabola.mkgmap.main.Main.endOptions(Main.java:605) at uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader. java:128) at uk.me.parabola.mkgmap.main.Main.mainStart(Main.java:143) at uk.me.parabola.mkgmap.main.Main.main(Main.java:114)

Probably I'm missing something obvious, but I always used --code-page=1251 and have no idea what it may be

_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

---

mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Hi I'll be able to take a look later this evening. I think it is just all sign extension problems. Steve On 8 April 2019 11:50:58 BST, Gerd Petermann <gpetermann_muenchen@hotmail.com> wrote:

Hi Ticker,

sorry, can't help with this part of the source.

@Steve: Please help

Gerd

________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Montag, 8. April 2019 12:38 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] Building map with Hebrew characters

Hi Gerd and Nick

@nick, when you say it works on 4143 but not on 4179, did you test code -page=1255 on 4143, or just unicode?

It looks like there are 2 problems.

The --unicode assertion error is a 'byte' to 'int' sign-extension problem, introduced in r4167 or r4168 and I've attached a patch that fixes this and also removes some other confusing/obscuring type representation changes in this area of the code.

The patch also has 4 commented out (because not appropriate for - -unicode) asserts that seem to point to the Hebrew/1255 problem being in imgfmt/app/srt/Sort.java:finish():

$ java -ea -jar /mkgmap/mkgmap.jar --code-page=1255 --gmapsupp test -map:all-elements

Time started: Mon Apr 08 09:30:06 BST 2019 Exception in thread "main" java.lang.AssertionError: 16 at uk.me.parabola.imgfmt.app.srt.Sort$Page.setSecondary(Sort.java:642) at uk.me.parabola.imgfmt.app.srt.Sort.finish(Sort.java:161) at uk.me.parabola.mkgmap.srt.SrtTextReader.read(SrtTextReader.java:182) at uk.me.parabola.mkgmap.srt.SrtTextReader.<init>(SrtTextReader.java:117) at uk.me.parabola.mkgmap.srt.SrtTextReader.<init>(SrtTextReader.java:106) at uk.me.parabola.mkgmap.srt.SrtTextReader.sortForCodepage(SrtTextReader.j ava:141) at uk.me.parabola.mkgmap.main.Main.getSort(Main.java:764) at uk.me.parabola.mkgmap.main.Main.processFilename(Main.java:283) at uk.me.parabola.mkgmap.CommandArgsReader$Filename.processArg(CommandArgs Reader.java:272) at uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.java :125) at uk.me.parabola.mkgmap.main.Main.mainStart(Main.java:143) at uk.me.parabola.mkgmap.main.Main.main(Main.java:114)

this is corresponds to: } else {

p.setSecondary(i, p.getSecondary(i) + maxSecondary);

}

where this increase in the secondary value is governed by some bits in the pages.flags and the flags testing was changed in r3968, along with the introduction of the original assert that was failing (SRTFile.java:117) and quite a few other changes in srt.

@gerd, I'm afraid I don't understand the srt logic so can you have a look at it

Regards Ticker

On Sat, 2019-04-06 at 08:16 +0000, Gerd Petermann wrote:

...

Hi, maybe a regression from r4167? http://www.mkgmap.org.uk/websvn/revision.php?repname=mkgmap&rev=4167

Gerd

________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von osm@pinns <osm@pinns.co.uk> Gesendet: Samstag, 6. April 2019 10:01 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] Building map with Hebrew characters

Have done some checking using older versions

It works on 4143 but not on 4179

I'm afraid I don't have anything in between

r

Nick

On 05/04/2019 21:25, Carlos Dávila wrote:

...

I'm trying to build a map of Israel with Hebrew characters, but using --code-page=1255 a get the following assertion:

at uk.me.parabola.imgfmt.app.srt.SRTFile.writeWeights(SRTFile.java:117 ) at uk.me.parabola.imgfmt.app.srt.SRTFile.writeSrt4Chars(SRTFile.java:9 9) at uk.me.parabola.imgfmt.app.srt.SRTFile.write(SRTFile.java:63) at uk.me.parabola.mkgmap.combiners.MdrBuilder.init(MdrBuilder.java:117 ) at uk.me.parabola.mkgmap.main.Main.endOptions(Main.java:605) at uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader. java:128) at uk.me.parabola.mkgmap.main.Main.mainStart(Main.java:143) at uk.me.parabola.mkgmap.main.Main.main(Main.java:114)

With --unicode I also get an assertion: Exception in thread "main" java.lang.AssertionError: -125 at uk.me.parabola.imgfmt.app.BufferedImgFileWriter.put1u(BufferedImgFi leWriter.java:154) at uk.me.parabola.imgfmt.app.SectionWriter.put1u(SectionWriter.java:78 ) at uk.me.parabola.imgfmt.app.srt.SRTFile.writeWeights(SRTFile.java:113 ) at uk.me.parabola.imgfmt.app.srt.SRTFile.writeSrt8(SRTFile.java:166) at uk.me.parabola.imgfmt.app.srt.SRTFile.write(SRTFile.java:68) at uk.me.parabola.mkgmap.combiners.MdrBuilder.init(MdrBuilder.java:117 ) at uk.me.parabola.mkgmap.main.Main.endOptions(Main.java:605) at uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader. java:128) at uk.me.parabola.mkgmap.main.Main.mainStart(Main.java:143) at uk.me.parabola.mkgmap.main.Main.main(Main.java:114)

Probably I'm missing something obvious, but I always used --code-page=1251 and have no idea what it may be

_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

---

mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

---

mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Just tested 4136 with --code-page=1255 and it fails  :( Again, fine with unicode :) On 08/04/2019 12:03, osm@pinns wrote:

Hi Ticker

I just used unicode

r

Nick

On 08/04/2019 11:38, Ticker Berkin wrote:

...

Hi Gerd and Nick

@nick, when you say it works on 4143 but not on 4179, did you test code -page=1255 on 4143, or just unicode?

It looks like there are 2 problems.

The --unicode assertion error is a 'byte' to 'int' sign-extension problem, introduced in r4167 or r4168 and I've attached a patch that fixes this and also removes some other confusing/obscuring type representation changes in this area of the code.

The patch also has 4 commented out (because not appropriate for - -unicode) asserts that seem to point to the Hebrew/1255 problem being in imgfmt/app/srt/Sort.java:finish():

$ java -ea -jar /mkgmap/mkgmap.jar --code-page=1255 --gmapsupp test -map:all-elements

Time started: Mon Apr 08 09:30:06 BST 2019 Exception in thread "main" java.lang.AssertionError: 16 at uk.me.parabola.imgfmt.app.srt.Sort$Page.setSecondary(Sort.java:642) at uk.me.parabola.imgfmt.app.srt.Sort.finish(Sort.java:161) at uk.me.parabola.mkgmap.srt.SrtTextReader.read(SrtTextReader.java:182) at uk.me.parabola.mkgmap.srt.SrtTextReader.<init>(SrtTextReader.java:117) at uk.me.parabola.mkgmap.srt.SrtTextReader.<init>(SrtTextReader.java:106) at uk.me.parabola.mkgmap.srt.SrtTextReader.sortForCodepage(SrtTextReader.j ava:141) at uk.me.parabola.mkgmap.main.Main.getSort(Main.java:764) at uk.me.parabola.mkgmap.main.Main.processFilename(Main.java:283) at uk.me.parabola.mkgmap.CommandArgsReader$Filename.processArg(CommandArgs Reader.java:272) at uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.java :125) at uk.me.parabola.mkgmap.main.Main.mainStart(Main.java:143) at uk.me.parabola.mkgmap.main.Main.main(Main.java:114)

this is corresponds to: } else {

p.setSecondary(i, p.getSecondary(i) + maxSecondary);

}

where this increase in the secondary value is governed by some bits in the pages.flags and the flags testing was changed in r3968, along with the introduction of the original assert that was failing (SRTFile.java:117) and quite a few other changes in srt.

@gerd, I'm afraid I don't understand the srt logic so can you have a look at it

Regards Ticker

On Sat, 2019-04-06 at 08:16 +0000, Gerd Petermann wrote:

...

Hi, maybe a regression from r4167? http://www.mkgmap.org.uk/websvn/revision.php?repname=mkgmap&rev=4167

Gerd

________________________________________ Von: mkgmap-dev<mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von osm@pinns<osm@pinns.co.uk> Gesendet: Samstag, 6. April 2019 10:01 An:mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] Building map with Hebrew characters

Have done some checking using older versions

It works on 4143 but not on 4179

I'm afraid I don't have anything in between

r

Nick

On 05/04/2019 21:25, Carlos Dávila wrote:

...

I'm trying to build a map of Israel with Hebrew characters, but using --code-page=1255 a get the following assertion:

at uk.me.parabola.imgfmt.app.srt.SRTFile.writeWeights(SRTFile.java:117 ) at uk.me.parabola.imgfmt.app.srt.SRTFile.writeSrt4Chars(SRTFile.java:9 9) at uk.me.parabola.imgfmt.app.srt.SRTFile.write(SRTFile.java:63) at uk.me.parabola.mkgmap.combiners.MdrBuilder.init(MdrBuilder.java:117 ) at uk.me.parabola.mkgmap.main.Main.endOptions(Main.java:605) at uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader. java:128) at uk.me.parabola.mkgmap.main.Main.mainStart(Main.java:143) at uk.me.parabola.mkgmap.main.Main.main(Main.java:114)

With --unicode I also get an assertion: Exception in thread "main" java.lang.AssertionError: -125 at uk.me.parabola.imgfmt.app.BufferedImgFileWriter.put1u(BufferedImgFi leWriter.java:154) at uk.me.parabola.imgfmt.app.SectionWriter.put1u(SectionWriter.java:78 ) at uk.me.parabola.imgfmt.app.srt.SRTFile.writeWeights(SRTFile.java:113 ) at uk.me.parabola.imgfmt.app.srt.SRTFile.writeSrt8(SRTFile.java:166) at uk.me.parabola.imgfmt.app.srt.SRTFile.write(SRTFile.java:68) at uk.me.parabola.mkgmap.combiners.MdrBuilder.init(MdrBuilder.java:117 ) at uk.me.parabola.mkgmap.main.Main.endOptions(Main.java:605) at uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader. java:128) at uk.me.parabola.mkgmap.main.Main.mainStart(Main.java:143) at uk.me.parabola.mkgmap.main.Main.main(Main.java:114)

Probably I'm missing something obvious, but I always used --code-page=1251 and have no idea what it may be

_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

---

mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Hi Gerd Attached is a patch that takes Steve's changes for Hebrew and a merges his other changes with my patch from 8th Apr that removes a lot of pointless swapping between byte/char and int and also fixes the sign -extension problem. Regards Ticker On Mon, 2019-04-08 at 21:22 +0100, Steve Ratcliffe wrote:

Hi

There is a problem that is specific to cp1255 in that there is a primary character with more than 15 secondary variations.

I've split the first line up in cp1255.txt for lack of any better idea of what to do.

The attached patch fixes both 1255 and unicode.

Steve _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Hi Gerd This patch never got applied - I've attached the patch based on current trunk. Ticker On Tue, 2019-04-16 at 06:01 +0000, Gerd Petermann wrote:

Hi,

@Steve: I see nothing obvious wrong in this patch but I have no idea how to test it. Please commit if you like the changes.

Gerd

________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Montag, 15. April 2019 17:49 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] Building map with Hebrew characters

Hi Gerd

Attached is a patch that takes Steve's changes for Hebrew and a merges his other changes with my patch from 8th Apr that removes a lot of pointless swapping between byte/char and int and also fixes the sign -extension problem.

Regards Ticker

On Mon, 2019-04-08 at 21:22 +0100, Steve Ratcliffe wrote:

...

Hi

There is a problem that is specific to cp1255 in that there is a primary character with more than 15 secondary variations.

I've split the first line up in cp1255.txt for lack of any better idea of what to do.

The attached patch fixes both 1255 and unicode.

Steve _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

---

mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev