Re: [mkgmap-dev] [mkgmap-svn] Commit r4854: - fix possible error when negativ index was written to lookup table
Hi Gerd Just starting to read the new code more carefully and a few comments: In Mdr17 the following line should be deleted: len = (len << 1) + 1; Mdr15 is not written forDevice so can skip all the compression stuff. Have you seen an example with unicode. There seem to be too many assumption about fixed/8-bit charsets to attempt compression. In Mdr15.createString, for compressed can just use ++nextOffset, the string length is meaningless Does sizes.getStrOffSize() need fixing after the final forms of the strings have been written? Will look at the nitty-gritty detail next. Ticker On Sun, 2022-01-09 at 11:03 +0000, svn commit wrote:
Version mkgmap-r4854 was committed by gerd on Sun, 09 Jan 2022 BRANCH: mdr2 - fix possible error when negativ index was written to lookup table - rename initBits to lookupBits - use common code for variable length integers in MDR16 and Mdr17 - simplify code, add comments to be continued...
http://www.mkgmap.org.uk/websvn/revision.php?repname=mkgmap&rev=4854 _______________________________________________ mkgmap-svn mailing list To unsubscribe send an mail to mkgmap-svn-leave@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-svn
Hi Ticker, thanks for reviewing, fixed the bug in MDR17. I've not even started to optimize anything reg. performance. I'm still trying to understand the idea of the shifted minCode values and I think I understand it now after looking at the code in GPXSee (huffmantable.cpp method HuffmanTable::symbol) I have a few sample MDR16 with unicode maps (from private mails). They all use 6 bits for the lookup table. "Does sizes.getStrOffSize() need fixing after the final forms of the strings have been written?" Maybe yes. I didn't recognize that nextOffset is used to calculate that. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Montag, 10. Januar 2022 09:22 An: mkgmap-dev@lists.mkgmap.org.uk; mkgmap-svn@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4854: - fix possible error when negativ index was written to lookup table Hi Gerd Just starting to read the new code more carefully and a few comments: In Mdr17 the following line should be deleted: len = (len << 1) + 1; Mdr15 is not written forDevice so can skip all the compression stuff. Have you seen an example with unicode. There seem to be too many assumption about fixed/8-bit charsets to attempt compression. In Mdr15.createString, for compressed can just use ++nextOffset, the string length is meaningless Does sizes.getStrOffSize() need fixing after the final forms of the strings have been written? Will look at the nitty-gritty detail next. Ticker On Sun, 2022-01-09 at 11:03 +0000, svn commit wrote:
Version mkgmap-r4854 was committed by gerd on Sun, 09 Jan 2022 BRANCH: mdr2 - fix possible error when negativ index was written to lookup table - rename initBits to lookupBits - use common code for variable length integers in MDR16 and Mdr17 - simplify code, add comments to be continued...
http://www.mkgmap.org.uk/websvn/revision.php?repname=mkgmap&rev=4854 _______________________________________________ mkgmap-svn mailing list To unsubscribe send an mail to mkgmap-svn-leave@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-svn
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Gerd More minor stuff - Still haven't understood the mincode parts mdr/Mdr16.java: loopupBits/lookupBits is very confusing. Can you have lookupBits for table2 size and something else for table1 size If maxDepth < initial size (5 or 6), can't you just set table1 size to maxDepth and set table2 size to 0. Or did you find this combination didn't work. Should it need: while (pos >= 0) { tab2[pos * 2] = 0; tab2[pos * 2 + 1] = (byte) lastIndex; pos--; } Won't the data have been exhausted, with v0 being the first index of table1 (0) and v2 correct MdrDisplay: would be clearer to spot/remember the bottom bit flag in table2, and
1 the data in both cases, then present the char data as numBits/char and the non-char data as minLev/maxLev
Ticker
Hi Ticker, table1 size depends on the number of levels with values and the existence of the lookup table, empty levels are not stored. When there is a lookup table (table2) we must not store the corresponding top 5 /6 levels in table1. I don't understand the part after "Should it need ..." . It worked in my tests but of course I might have missed a special case. It's all work in progress. I plan to add unit tests, something like "given these frequencies and codepage xyz the result should look like ...". I plan to recode MdrDisplay now that I undestand almost all data. I also want to implement this in MdrCheck during the next days. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Montag, 10. Januar 2022 17:49 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4854: - fix possible error when negativ index was written to lookup table Hi Gerd More minor stuff - Still haven't understood the mincode parts mdr/Mdr16.java: loopupBits/lookupBits is very confusing. Can you have lookupBits for table2 size and something else for table1 size If maxDepth < initial size (5 or 6), can't you just set table1 size to maxDepth and set table2 size to 0. Or did you find this combination didn't work. Should it need: while (pos >= 0) { tab2[pos * 2] = 0; tab2[pos * 2 + 1] = (byte) lastIndex; pos--; } Won't the data have been exhausted, with v0 being the first index of table1 (0) and v2 correct MdrDisplay: would be clearer to spot/remember the bottom bit flag in table2, and
1 the data in both cases, then present the char data as numBits/char and the non-char data as minLev/maxLev
Ticker _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Gerd I think I got table1 / table2 names the wrong way round. The bit about "should it need" is probably necessary when all the levels fit into the prefix table but it isn't full. I won't look at it again until you want me to. Ticker On Mon, 2022-01-10 at 19:16 +0000, Gerd Petermann wrote:
Hi Ticker,
table1 size depends on the number of levels with values and the existence of the lookup table, empty levels are not stored. When there is a lookup table (table2) we must not store the corresponding top 5 /6 levels in table1.
I don't understand the part after "Should it need ..." . It worked in my tests but of course I might have missed a special case. It's all work in progress. I plan to add unit tests, something like "given these frequencies and codepage xyz the result should look like ...".
I plan to recode MdrDisplay now that I undestand almost all data. I also want to implement this in MdrCheck during the next days.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Montag, 10. Januar 2022 17:49 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r4854: - fix possible error when negativ index was written to lookup table
Hi Gerd
More minor stuff - Still haven't understood the mincode parts
mdr/Mdr16.java:
loopupBits/lookupBits is very confusing. Can you have lookupBits for table2 size and something else for table1 size
If maxDepth < initial size (5 or 6), can't you just set table1 size to maxDepth and set table2 size to 0. Or did you find this combination didn't work.
Should it need: while (pos >= 0) { tab2[pos * 2] = 0; tab2[pos * 2 + 1] = (byte) lastIndex; pos--; } Won't the data have been exhausted, with v0 being the first index of table1 (0) and v2 correct
MdrDisplay: would be clearer to spot/remember the bottom bit flag in table2, and
1 the data in both cases, then present the char data as numBits/char and the non-char data as minLev/maxLev
Ticker
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
participants (2)
-
Gerd Petermann -
Ticker Berkin