January 2022 - mkgmap-dev - The mkgmap lists

Re: [mkgmap-dev] mkgmap-dev Digest, Vol 162, Issue 25
by John Thorn 17 Jan '22

17 Jan '22

Hi Gerd et al, Downloaded 4865 and ran it to make a .GMAP with no changes to data nor options Compared with running v4286 (which works for me so I still use) : MDR is now 43092K was 47347K Runtime (reading .MP files) seemed slower, maybe 30% slower Some .RGN files are bigger : now 5855K was 5823K, for example. Searching in Basecamp runs with no problems. Keep up the good work! John On Sun, 16 Jan 2022 at 12:00, <mkgmap-dev-request(a)lists.mkgmap.org.uk> wrote: > Send mkgmap-dev mailing list submissions to > mkgmap-dev(a)lists.mkgmap.org.uk > > To subscribe or unsubscribe via the World Wide Web, visit > https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > or, via email, send a message with subject or body 'help' to > mkgmap-dev-request(a)lists.mkgmap.org.uk > > You can reach the person managing the list at > mkgmap-dev-owner(a)lists.mkgmap.org.uk > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of mkgmap-dev digest..." > Today's Topics: > > 1. Please try and report results of Huffman compression > (Gerd Petermann) > 2. Re: Please try and report results of Huffman compression (7770) > 3. Re: Please try and report results of Huffman compression > (Gerd Petermann) > > > > ---------- Forwarded message ---------- > From: Gerd Petermann <GPetermann_muenchen(a)hotmail.com> > To: "mkgmap-dev(a)lists.mkgmap.org.uk" <mkgmap-dev(a)lists.mkgmap.org.uk> > Cc: > Bcc: > Date: Sat, 15 Jan 2022 15:59:24 +0000 > Subject: [mkgmap-dev] Please try and report results of Huffman compression > Hi all, > > during the last weeks Ticker and I decoded Garmin's String compression of > the global index for the PC (the *_mdr.img) . > In the mdr2 branch I've coded support for that in mkgmap and I think its > ready for a first beta test. > > It works with codepages 1250 ..1258 or 65001. > I've no idea how the 2-byte pages work, but Garmin seems to support that, > too. Maybe I'll work on this later. > > The branch version produces around 200 lines of diagnostic output to show > what happens when compression is enabled. > This will be changed to debug level later. > > The encoding is enabled by default, special option > --x-no-mdr15-compression can be used to turn it off. > > Note that the size of the *_mdr.img file will be smaller (which is esp. > intersting for those who get close to the 2G limit) > Note also that a zipped *_mdr.img can and will be larger since zip cannot > remove as much redundancy. > Note also that this branch includes the changes from the faster-mp branch > and introduces a few changes > which are relavant for --lower-case maps. > > How to test? > Install the map in MapSource and check if you see any unexpected > characters in the result lists or maybe even crashes while searching. > If so, please try the same input files with the --x-no-mdr15-compression > to verify if this is caused by compression or maybe other changes. > > Please don't yet publish the maps, but let me know if you see anything > unexpected. Some changes reg. lower-case in this branch are still > experimental. > > The binary is here: > https://www.mkgmap.org.uk/download/mkgmap-mdr2-r4865.zip > > If you see errors which are caused by compression, please attach the > diagnostic output. > > @Ticker: Please review the source. > > Gerd > > > > ---------- Forwarded message ---------- > From: 7770 <7770(a)foskan.eu> > To: "mkgmap-dev(a)lists.mkgmap.org.uk" <mkgmap-dev(a)lists.mkgmap.org.uk> > Cc: > Bcc: > Date: Sat, 15 Jan 2022 18:53:37 +0100 > Subject: Re: [mkgmap-dev] Please try and report results of Huffman > compression > Hi. > https://files.mkgmap.org.uk/detail/544 > > ALLVARLIG (MultiPolygonRelation): folder/77700046.osm.pbf: Internal error: > Failed to render MultiPolygonRelation 2970267 > that was with --latin1 > > mkgmap version 4839 works well. > > > Regards > Karl > > On lördag 15 januari 2022 16:59:24 CET Gerd Petermann wrote: > > Hi all, > > > > during the last weeks Ticker and I decoded Garmin's String compression of > > the global index for the PC (the *_mdr.img) . In the mdr2 branch I've > coded > > support for that in mkgmap and I think its ready for a first beta test. > > > > It works with codepages 1250 ..1258 or 65001. > > I've no idea how the 2-byte pages work, but Garmin seems to support that, > > too. Maybe I'll work on this later. > > > > The branch version produces around 200 lines of diagnostic output to show > > what happens when compression is enabled. This will be changed to debug > > level later. > > > > The encoding is enabled by default, special option > --x-no-mdr15-compression > > can be used to turn it off. > > > > Note that the size of the *_mdr.img file will be smaller (which is esp. > > intersting for those who get close to the 2G limit) Note also that a > zipped > > *_mdr.img can and will be larger since zip cannot remove as much > > redundancy. Note also that this branch includes the changes from the > > faster-mp branch and introduces a few changes which are relavant for > > --lower-case maps. > > > > How to test? > > Install the map in MapSource and check if you see any unexpected > characters > > in the result lists or maybe even crashes while searching. If so, please > > try the same input files with the --x-no-mdr15-compression to verify if > > this is caused by compression or maybe other changes. > > > > Please don't yet publish the maps, but let me know if you see anything > > unexpected. Some changes reg. lower-case in this branch are still > > experimental. > > > > The binary is here: > > https://www.mkgmap.org.uk/download/mkgmap-mdr2-r4865.zip > > > > If you see errors which are caused by compression, please attach the > > diagnostic output. > > > > @Ticker: Please review the source. > > > > Gerd > > _______________________________________________ > > mkgmap-dev mailing list > > mkgmap-dev(a)lists.mkgmap.org.uk > > https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > > > > > > > ---------- Forwarded message ---------- > From: Gerd Petermann <gpetermann_muenchen(a)hotmail.com> > To: "mkgmap-dev(a)lists.mkgmap.org.uk" <mkgmap-dev(a)lists.mkgmap.org.uk> > Cc: > Bcc: > Date: Sun, 16 Jan 2022 09:38:14 +0000 > Subject: Re: [mkgmap-dev] Please try and report results of Huffman > compression > Hi all, > > see https://www.mkgmap.org.uk/websvn/revision.php?repname=mkgmap&rev=4866 > and https://www.mkgmap.org.uk/websvn/revision.php?repname=mkgmap&rev=4867 > > Please use https://www.mkgmap.org.uk/download/mkgmap-mdr2-r4867.zip for > further tests. > > Gerd > > ________________________________________ > Von: 7770 <7770(a)foskan.eu> > Gesendet: Samstag, 15. Januar 2022 18:53 > An: mkgmap-dev(a)lists.mkgmap.org.uk > Cc: Gerd Petermann > Betreff: Re: [mkgmap-dev] Please try and report results of Huffman > compression > > Hi. > https://files.mkgmap.org.uk/detail/544 > > ALLVARLIG (MultiPolygonRelation): folder/77700046.osm.pbf: Internal error: > Failed to render MultiPolygonRelation 2970267 > that was with --latin1 > > mkgmap version 4839 works well. > > > Regards > Karl > > On lördag 15 januari 2022 16:59:24 CET Gerd Petermann wrote: > > Hi all, > > > > during the last weeks Ticker and I decoded Garmin's String compression of > > the global index for the PC (the *_mdr.img) . In the mdr2 branch I've > coded > > support for that in mkgmap and I think its ready for a first beta test. > > > > It works with codepages 1250 ..1258 or 65001. > > I've no idea how the 2-byte pages work, but Garmin seems to support that, > > too. Maybe I'll work on this later. > > > > The branch version produces around 200 lines of diagnostic output to show > > what happens when compression is enabled. This will be changed to debug > > level later. > > > > The encoding is enabled by default, special option > --x-no-mdr15-compression > > can be used to turn it off. > > > > Note that the size of the *_mdr.img file will be smaller (which is esp. > > intersting for those who get close to the 2G limit) Note also that a > zipped > > *_mdr.img can and will be larger since zip cannot remove as much > > redundancy. Note also that this branch includes the changes from the > > faster-mp branch and introduces a few changes which are relavant for > > --lower-case maps. > > > > How to test? > > Install the map in MapSource and check if you see any unexpected > characters > > in the result lists or maybe even crashes while searching. If so, please > > try the same input files with the --x-no-mdr15-compression to verify if > > this is caused by compression or maybe other changes. > > > > Please don't yet publish the maps, but let me know if you see anything > > unexpected. Some changes reg. lower-case in this branch are still > > experimental. > > > > The binary is here: > > https://www.mkgmap.org.uk/download/mkgmap-mdr2-r4865.zip > > > > If you see errors which are caused by compression, please attach the > > diagnostic output. > > > > @Ticker: Please review the source. > > > > Gerd > > _______________________________________________ > > mkgmap-dev mailing list > > mkgmap-dev(a)lists.mkgmap.org.uk > > https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > > > > _______________________________________________ > mkgmap-dev mailing list > mkgmap-dev(a)lists.mkgmap.org.uk > https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

2 1

Re: [mkgmap-dev] [mkgmap-svn] Commit r4854: - fix possible error when negativ index was written to lookup table
by Ticker Berkin 10 Jan '22

10 Jan '22

Hi Gerd Just starting to read the new code more carefully and a few comments: In Mdr17 the following line should be deleted: len = (len << 1) + 1; Mdr15 is not written forDevice so can skip all the compression stuff. Have you seen an example with unicode. There seem to be too many assumption about fixed/8-bit charsets to attempt compression. In Mdr15.createString, for compressed can just use ++nextOffset, the string length is meaningless Does sizes.getStrOffSize() need fixing after the final forms of the strings have been written? Will look at the nitty-gritty detail next. Ticker On Sun, 2022-01-09 at 11:03 +0000, svn commit wrote: > Version mkgmap-r4854 was committed by gerd on Sun, 09 Jan 2022 > BRANCH: mdr2 > - fix possible error when negativ index was written to lookup table > - rename initBits to lookupBits > - use common code for variable length integers in MDR16 and Mdr17 > - simplify code, add comments > to be continued... > > > http://www.mkgmap.org.uk/websvn/revision.php?repname=mkgmap&rev=4854 > _______________________________________________ > mkgmap-svn mailing list > To unsubscribe send an mail to mkgmap-svn-leave(a)lists.mkgmap.org.uk > https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-svn

2 4

LBL file encoding - Huffman
by Ticker Berkin 10 Jan '22

10 Jan '22

It looks like the LBL file could be Huffman compressed with an ENCODING value of 11. Is there scope for having the new Huffman logic with app/labelenc/CharacterEncoder /CharacterDecoder interface? Ticker

2 1

Commit r4855: improve documentation, esp. about --mdr7-del (options2.patch by Mike Baggaley)
by svn commit 10 Jan '22

10 Jan '22

Version mkgmap-r4855 was committed by gerd on Mon, 10 Jan 2022 improve documentation, esp. about --mdr7-del (options2.patch by Mike Baggaley) http://www.mkgmap.org.uk/websvn/revision.php?repname=mkgmap&rev=4855

1 0

Minor documentation patch
by Mike Baggaley 09 Jan '22

09 Jan '22

Hi Gerd, Please find attached a minor documentation patch. Cheers, Mike

2 2

Commit r4845: merge from faster-mp branch
by svn commit 07 Jan '22

07 Jan '22

Version mkgmap-r4845 was committed by gerd on Wed, 05 Jan 2022 merge from faster-mp branch - don't render natural=coastline in lines - implement faster (but possibly less reliable) algorithm to render complex multipolygons - don't render MP without tags (old code did that when all tags were removed because they did not appear in style rules, see field tagsIncomplete) http://www.mkgmap.org.uk/websvn/revision.php?repname=mkgmap&rev=4845

3 16

Commit r4850: revert merge from faster-mp branch, causes crashes in some maps
by svn commit 06 Jan '22

06 Jan '22

Version mkgmap-r4850 was committed by gerd on Thu, 06 Jan 2022 revert merge from faster-mp branch, causes crashes in some maps http://www.mkgmap.org.uk/websvn/revision.php?repname=mkgmap&rev=4850

1 0

Re: [mkgmap-dev] [mkgmap-svn] Commit r4848: catch some special cases which caused infinite loop in mkgmap or wrong data in MDR16 if there is not much data
by Ticker Berkin 06 Jan '22

06 Jan '22

Hi Gerd Did the '5' nibble in byte 2 indicate the size of lookup table? if so, could reduce the table size so it overflows. I assume that setting byte 3 to <= table size didn't work Ticker On Thu, 2022-01-06 at 09:25 +0000, svn commit wrote: > Version mkgmap-r4848 was committed by gerd on Thu, 06 Jan 2022 > BRANCH: mdr2 > catch some special cases which caused infinite loop in mkgmap or > wrong data in MDR16 if there is not much data > in the index > - I don't know yet what Garmin expects when there are very few > characters in the Huffman tree so that all fit into the lookup table. > Maybe this is what the unknown flag in the initial bits field is > about > - don't try to encode with invalid data in mdr16 > > http://www.mkgmap.org.uk/websvn/revision.php?repname=mkgmap&rev=4848 > _______________________________________________ > mkgmap-svn mailing list > To unsubscribe send an mail to mkgmap-svn-leave(a)lists.mkgmap.org.uk > https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-svn

2 1

do people think mkgmap tests pass?
by Greg Troxel 05 Jan '22

05 Jan '22

(After doing other things for a long time, I'm updating spliter, mkgmap, and data prepartory to building new img files.) I ran the mkgmap tests, and got some failures, 7 in args and 2 others. I am using NetBSD, but that shouldn't matter. Do people think "ant test" gets a clean pass with the current svn? If so I can send info about the failing tests. Thanks, Greg

2 3

Error in MdrCheck?
by Gerd Petermann 04 Jan '22

04 Jan '22

Hi devs, I think there is a bug in MdrCheck, probably also in other Check programs. The program doesn't read the SRT file content from the map, instead it uses the corresponding data from mkgmap. If the builtin sort order in mkgmap doesn't match the SRT file content the program will report errors about wrong order or missing repeat flags etc. I guess this explains why MdrCheck complains about the Garmin demo map for Turkey? I once started to implement a SrtFileReader but I don't know if that can be used instead. Gerd

3 10