April 2021 - mkgmap-dev - The mkgmap lists

Re: [mkgmap-dev] Embedding raster maps
by Oliver Eichler 25 Jan '22

25 Jan '22

Hi Steve, thanks for the file. On my device nothing is visible at all. But that's kind of an information, too. There is no way to circle down bugs by simply skipping the extended part. Probably it is still some missing encoding. In the Isle of Man map (and others) there is much more information in the RGN header. Note the values in the "filling bytes". And there seems to be an additional section after the extended point2 section. I called it offset1/length2. --- RGN header --- size 007D type 47 41 52 4D 49 4E 20 52 47 4E | GARMIN RGN byte0x0000000C 01 flag 00 year 2010 month 7 day 21 hour 9 minute 53 second 6 offset1 0000D54E length1 00000000 offset_polyg2 0000D54E length_polyg2 00001A1F byte0x00000025_0x00000038 02 00 00 00 00 00 00 00 FF 00 00 20 FD FC 03 00 00 00 00 00 offset_polyl2 0000EF6D length_polyl2 00000000 byte0x00000041_0x00000054 00 00 00 00 00 00 00 00 3F 00 00 20 FD 0F 00 00 00 00 00 00 offset_point2 0000EF6D length_point2 00000000 byte0x0000005D_0x00000070 00 00 00 00 00 00 00 00 FF 07 00 20 3F F7 3F 00 00 00 00 00 offset2 0000EF6D length2 0000020B unknown 000000E3 This section has a variable length in different sub-files. The unknown value E3 can be E5, too. It doesn't seem to be a record size. I had a look at several sections. They are not completely random, thus there seem to be no "compression". However the patterns do not ring a bell for me. I copied a couple of them into a file and loaded it into my cloud share: https://www.magentacloud.de/share/s429h1z-r6 Maybe one of you has an idea what it's about. I also uploaded my current version of the test map with a single raster map tile. It does not show. Top left coordinate is N46° 57.768 E011° 47.484. Sadly I am stuck here and I am running out of quality time to concentrate on the topic. I also have to update QMapShack's decoding. While reading the mkgmap source code I noticed a lot of new stuff that is not covered by the decoder. Cheers Oliver

4 6

Empty rectangles in map
by Gerd Petermann 10 May '21

10 May '21

Hi Experts, while working on the faster-mp branch I experimented with the monster relation r9488835. I added tag landuse=forest to it so that it is actually rendered and found huge (wrong) empty rectangles in the forest. The original file is very complex but the same problem also occurs with a simplified file, see http://files.mkgmap.org.uk/download/505/bad-map-split.osm.pbf Just use e.g. java -jar mkgmap.jar --gmapi bad-map-split.osm.pbf Problem: Basecamp and Mapsource don't show this map, GpsMapEdit shows a large white rectangle near 54.97743 -58.30292 I assume that we hit some unknown limit , possibly in LinePreparer or MapSplitter. I've also tried some older versions, so far they all show the problem. Any idea where to look? RgnDisplay shows this data for the problemat area: --------- Level 0, Subdiv 279 -------------------------------------------------- | | | final at 265f, (end 25e2) | | | polygon start 2634 to 265f 00002634 | 000000 | 4b | Polygon 1 | | | Type 0x4b 00002635 | 000001 | 00 00 00 | Label offset: 0x000000 () 00002638 | 000004 | d0 3b | Long delta 15312 0000263a | 000006 | 2a 3a | Lat delta 14890 | | | Start point 55.24913,-61.91139 2574796/-2885280 0000263c | 000008 | 0c | Bitstream length 12 0000263d | 000009 | bb | Base bb 0000263e | 00000a | 06 43 04 00 00 00 4c d1 | Bitstream (len: 12, 0xc) 00002646 | 000012 | 41 ef 00 00 | | | | xsame false, xneg false | | | ysame true, yneg true | | | xbase 16, ybase 15 | | | Coord 55.24913,-62.56851 2574796/-2915904 [-30624,+0] | | | Coord 54.61014,-62.56851 2545017/-2915904 [+0,-29779] | | | Coord 54.61014,-61.91139 2545017/-2885280 [+30624,+0] | | | 0 bits left, 0 | | | 0000264a | 000016 | 50 | Polygon 2 | | | Type 0x50 0000264b | 000017 | 00 00 00 | Label offset: 0x000000 () 0000264e | 00001a | d0 3b | Long delta 15312 00002650 | 00001c | b4 08 | Lat delta 2228 | | | Start point 54.97743,-61.91139 2562134/-2885280 00002652 | 00001e | 0b | Bitstream length 11 00002653 | 00001f | 9b | Base 9b 00002654 | 000020 | f2 8d 04 40 a7 fe d0 4b | Bitstream (len: 11, 0xb) 0000265c | 000028 | e7 00 00 | | | | xsame false, xneg false | | | ysame true, yneg false | | | xbase 16, ybase 11 | | | Coord 54.97743,-62.51706 2562134/-2913506 [-28226,+0] | | | Coord 55.01732,-62.54665 2563993/-2914885 [-1379,+1859] | | | Coord 55.01732,-61.91139 2563993/-2885280 [+29605,+0] | | | 4 bits left, 0 | | | Gerd

3 14

Please try branch check-precomp-sea
by Gerd Petermann 04 May '21

04 May '21

Hi all, I've implemented some basic checks to detect large flooded areas in the precomp-sea data in this new branch. A link to the latest binary (r4579 as of now) is a the bottom of the normal download page : http://www.mkgmap.org.uk/download/mkgmap.html - If PrecompSeaGenerator is used to calculate the sea data and the index check fails a warning is printed and the program ends with rc -1. - If mkgmap is used with a "dubious" sea.zip one or more messages about the problematic areas are written and the program stops with an ExitException unless option --x-check-precomp-sea=0 is given before the first input file. The index test is normally performed once for the whole planet, so it doesn't care about the actual tile boundaries. If e.g. only a part of Australia is flooded and you create a map for the UK you'll still see the errors. BTW: The current code (also in trunk version) doesn't support to use --precomp-sea with different values. Results are rather unpredictable. Gerd

4 4

Re: [mkgmap-dev] mkgmap-dev Digest, Vol 153, Issue 40 Resolution 23 raster problems
by Felix Hartmann 04 May '21

04 May '21

Well yes in that case they must. However for overhanging rocks the hgt files are incompatible. All sources we use for phyghtmap or srt2osm or other tools rely on 2D models to show the altitude. So for what we have right now - contourlines shall never cross. On Mon, 26 Apr 2021 at 18:22, Thomas Morgenstern <webmaster(a)img2ms.de> wrote: > I think , your wish, never cross contourlines, shows not the reality. In > fact there are some rocks, cliffs with 90 ° declination. For exampel: the > cliffs near Los Gigantes (tenerife) and the Baranco del Inferno > (tenerife) . In such situations, the contourlines must touch each other. > And how to procede, if the rocks hang over ? Contourlines must cross !? > thomas > > Am 26.04.2021 um 11:02 schrieb Felix Hartmann: > > I'm not fully sure of the exact location anymore - not too far away > from N47° 31.285' E12° 55.889' however. And using Basecamp map details at > lowest (this will allow you to see the problems much easier). > > On Mon, 26 Apr 2021 at 16:40, Thomas Morgenstern <webmaster(a)img2ms.de> > wrote: > >> @ Felix : please, can you post coordinates of your screenshots ?. i will >> look at my own contourlines at this position. >> thomas >> >> Am 26.04.2021 um 10:15 schrieb Felix Hartmann: >> >> yes - for contourlines it would be great if the algo could check that >> lines never cross - and if moving one direction vs the other means not >> touching vs touching - then move the other direction. So it would need to >> loop at for each line, at the next 2 lines. The big advantage would be for >> contourlines only maps (which makes most sense IMHO, as you don't need to >> update contourlines except if there is newer better data which is rare) >> that you can create them with resolution 23 as highest resolution >> instead of 24. Saving about 50% of the space. Old Garmin devices scroll >> notably slower with resolution 24 contourlines vs 23, even more of course >> if 10m equidistance instead of 20m equidistance is used. >> >> As 0x20, 0x21 and 0x22 are always contourlines - those lines could easily >> be distinguished (there is one second set of contour line types in newer >> garmin maps, I don't know if that matters, and cant remember the types >> right now). If this treatment is a lot slower - then it should have a >> command option, so whoever integrates the contourlines has the choice of >> chosing it. Because for separate contourlines even doubling or >> tripling compilation times do not matter much. >> >> On Mon, 26 Apr 2021 at 15:45, Gerd Petermann < >> gpetermann_muenchen(a)hotmail.com> wrote: >> >>> Hi all, >>> >>> the major problem with resolution 23 and lower is the rounding to garmin >>> units without looking at the distortions (WrongAngleFixer does that for res >>> 24) >>> >>> At res 23 we have to draw the lines using a "bed of nails" where the >>> nails have ~5m distance to each other. If the original line "meanders" >>> horizontally close to the nails there is no big problem, nodes are probably >>> rounded to a straight line. If the same line is moved by ~2.5 m up or down >>> we often hit the worst case scenario where one node is rounded to the north >>> and the next is rounded to the south. This produces visible zig-zagging. >>> If two or more almost parallel lines meat this worst case you see the >>> obvious errros where contour lines overlap. >>> >>> The logic in WrongAngleFixer tries to detect this and sometimes decides >>> to round to a different place to reduce zig-zagging. The problem is that >>> this has to be done looking at all lines, not just one, at least with >>> normal OSM data where we have intersections. >>> >>> For contourlines it would be best if the generator would only calculate >>> nodes which are exactly at the positions of the Garmin nails, but that is >>> probaby not easy. >>> The special case here is that it doesn't really matter where the nodes >>> are, we just need "good looking" lines and that each node is only used by >>> one contour way. >>> >>> Maybe it is possible to do this with a separate program or a special >>> method in mkgmap which just treats contour lines >>> - take generated contour data as input >>> - rerender each way with resolution 23 nodes so that the line looks >>> almost like before. This means it may add nodes somewhere and remove others. >>> - maybe re-iterate if the rerendering caused overlapping contour lines >>> where original lines where not overlapping >>> >>> Gerd >>> >>> >>> >>> ________________________________________ >>> Von: mkgmap-dev <mkgmap-dev-bounces(a)lists.mkgmap.org.uk> im Auftrag von >>> Felix Hartmann <extremecarver(a)gmail.com> >>> Gesendet: Montag, 26. April 2021 08:55 >>> An: Development list for mkgmap >>> Betreff: Re: [mkgmap-dev] mkgmap-dev Digest, Vol 153, Issue 40 >>> Resolution 23 raster problems >>> >>> Thanks Andrzej for the patch. >>> It produces better results in about 2/3 of cases, and worse in about 1/3 >>> of cases for resolution 23. So overall it's an improvement, and at least >>> for compiling contourlines I didn't notice a significant difference in >>> compile time. I could test this more in detail also against normal map data >>> if needed... >>> Using --simplifyContoursEpsilon= in phyghtmap helps a bit too, >>> especially for resolution 24. However it runs super slow. It's like 1-30 >>> times as much time, if not more (except for a value of 0.0 which is not >>> changing the maps at all, and just making the .pbf file 10% smaller). >>> >>> This doesn't solve phyghtmap not using b-spline interpolation. That >>> would improve even more (at least up to resolution 23, for resolution 22 >>> maybe this patch matters more). In general if using 20m equidistance I feel >>> in most cases it's good enough to compile contourlines only starting from >>> resolution 23, much faster, less data just of course in very steep areas >>> problematic. For 10m equidistance however it kinda means also going for >>> resolution 24, else the improvement from 20m to 10m is more of a nuisance. >>> >>> On Mon, 26 Apr 2021 at 03:41, Andrzej Popowski <popej(a)poczta.onet.pl >>> <mailto:popej@poczta.onet.pl>> wrote: >>> Hi Gerd, >>> >>> I don't observe any significant differences in compilation. But to make >>> it more optimized, we can put SizeFilter before DouglasPeuckerFilter. I >>> have attached a second patch here. >>> >>> There is a difference in results. See pictures with DouglasPeuckerFilter >>> after RoundCoordsFilter and new version with DouglasPeuckerFilter >>> before. This is for 22-bit resolution, so effects are probably more >>> clear. >>> >>> -- >>> Best regards, >>> Andrzej >>> >>> _______________________________________________ >>> mkgmap-dev mailing list >>> mkgmap-dev(a)lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk> >>> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev >>> >>> >>> -- >>> Felix Hartman - Openmtbmap.org & VeloMap.org >>> >>> _______________________________________________ >>> mkgmap-dev mailing list >>> mkgmap-dev(a)lists.mkgmap.org.uk >>> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev >>> >> >> >> -- >> Felix Hartman - Openmtbmap.org & VeloMap.org >> >> >> >> _______________________________________________ >> mkgmap-dev mailing listmkgmap-dev@lists.mkgmap.org.ukhttps://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev >> >> >> > > -- > Felix Hartman - Openmtbmap.org & VeloMap.org > > > -- Felix Hartman - Openmtbmap.org & VeloMap.org

3 33

Request - set --reduce-point-density= for each resolution
by Felix Hartmann 01 May '21

01 May '21

As I'm not sure everyone is following the other topic, I would like to ask to introduce the possibility to set different values for the DP filter based on resolution. Why is this important? a) maps which are produced with resolution 23 as highest resolution need to have a much lower level than subsequent resolutions. b) For resolution 23 and 22 a lower level makes sense - it is not needed as much here to speed up the gps device map drawing speed, while for lower resolutions like 18-21 a high level is really needed else map panning becomes painfully slow (usually basemap kicks in at 17). If I'm correct right now 2.6 at resolution 23 would be 5.4 at resolution 22, 10.8 at resolution 21 and so on. It's okay to keep that behaviour - but the value doubled just needs to be possible to be set for each resolution. It doesn't matter to me if we keep it like that or if we change the numbers than ourselves by resolution. Important thing is that it's possible to be set for each resolution c) maybe also introduce the possibility to set DP for resolution 24. I think a value of 0.3-0.6 could be worth it. I'm not sure if there were some problems preventing DP to be used at 24. However as by now I think mkgmap is able to produce routable maps starting at 23 instead of 24, those problems should be fixed... In general I think a map made for automotive use (car, motorcycle) would be fine using resolution 23 as a start instead of 24. Like Garmin City Navigator maps do. But then yes it's really important to not have a high DP value on resolution 23. Same as contourlines only maps which make sense at resolution 23 instead of 24. -- Felix Hartman - Openmtbmap.org & VeloMap.org

2 8

tile takes very long time to generate
by 7770 29 Apr '21

29 Apr '21

Hi. I observe a situation that one of the very first tiles (often the first or the third) that mkgmap generates takes 20 - 30 minutes to generate, wheres the others take about 30 seconds each. The map data produced by splitter is a total of around 700 files with splitter option --max-nodes=1275000. sea and bouds are used for mkgmap. At first i thought i am running low on memory but changing to max-jobs=1 (instead of the possible max of 2) did not make any change. Is mkgmap doing something specific in the beginning that can explain this long time generating one of the very first tiles? I can provide more details, let me know how i can best collect those details in case i need to set some parameters to have mkgmap to say more about what it is doing. Regards Karl

6 40

Commit r4680: Use https for all ivy downloads
by svn commit 26 Apr '21

26 Apr '21

Version mkgmap-r4680 was committed by steve on Mon, 26 Apr 2021 Use https for all ivy downloads http://www.mkgmap.org.uk/websvn/revision.php?repname=mkgmap&rev=4680

1 0

Check Roundabouts
by Mike Baggaley 26 Apr '21

26 Apr '21

Hi Gerd, I haven't posted it yet. I'm away on holiday at the moment, hence the lack of input from me as it is difficult to reply to the digest from a phone. I will post the patch when I return home. Cheers, Mike

2 1

check roundabouts
by Mike Baggaley 26 Apr '21

26 Apr '21

Hi Gerd, I have a largely completed change that extends the --check-roundabouts option to become --check-roundabouts=[all|loop|direction[:fix]|junctions|flares] and had made fixing reversed roundabouts an optional part of the direction option loop - check that roundabouts are formed from a single loop with no forks or gaps direction[:fix] - check and optionally fix the direction of travel around the roundabout junctions - check that no more than one connecting highway joins at each node flares - check that roundabout flare roads are one-way, are in the right direction, and don't extend beyond the flare However, separating the options completely allows for correction without displaying any message if desired, so I will look at that and I will change the reporting option name to --report-roundabout-issues as suggested, which brings it into line with the other diagnostic options. Also included in the change is code to more accurately try to identify roundabout flares, as the existing code produces more false results than wanted ones. It replaces --max-flare-length-ratio=NUM with --roundabout-flare-rules-config=filename where the config file can contain a set of rules for identifying flares. Cheers, Mike -----Original Message----- From: Gerd Petermann [mailto:gpetermann_muenchen@hotmail.com] Sent: 12 April 2021 16:10 To: Development list for mkgmap <mkgmap-dev(a)lists.mkgmap.org.uk> Subject: Re: [mkgmap-dev] logging Hi Mike, please have a look at the check-roundabouts and --check-roundabout-flares code. I think we missed the fact that --check-roundabouts may change the direction of roundabouts (in StyledConverter). So, I think we need a new option like report-roundabout-issues=[NUM] and another one for the reversing? Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces(a)lists.mkgmap.org.uk> im Auftrag von Mike Baggaley <mike(a)tvage.co.uk> Gesendet: Montag, 12. April 2021 17:02 An: 'Development list for mkgmap' Betreff: [mkgmap-dev] logging Hi Gerd, In the attached patch there are a few improvements to messages as well as some more use of logging: In ImgFS, the message from the FileNotWritableException was displaying the filename twice, so I have fixed that. In TdbBuilder, I have removed an unnecessary log.error In StyledConverter, I have added the synchronisation code for the removal of duplicate messages In O5mBinHandler, I have changed a thrown exception from IOException to FormatException so that it gets handled by its callers Cheers, Mike

2 1

Re: [mkgmap-dev] mkgmap-dev Digest, Vol 153, Issue 40 Resolution 23 raster problems
by John Thorn 26 Apr '21

26 Apr '21

I don't think this can be improved. As I understand it, internally Garmin devices hold coordinates in the number of bits specified in the level. So 24 bits has a resolution of 2.4 metres, 23 bits has a resolution of 4.8 meters and so on. If you look carefully, all the vertices of the lines lie on a square grid. Having said that, that must be a very steep slope - unless something else is creating the granularity. On Fri, 23 Apr 2021 at 12:06, <mkgmap-dev-request(a)lists.mkgmap.org.uk> wrote: > Send mkgmap-dev mailing list submissions to > mkgmap-dev(a)lists.mkgmap.org.uk > > To subscribe or unsubscribe via the World Wide Web, visit > https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > or, via email, send a message with subject or body 'help' to > mkgmap-dev-request(a)lists.mkgmap.org.uk > > You can reach the person managing the list at > mkgmap-dev-owner(a)lists.mkgmap.org.uk > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of mkgmap-dev digest..." > Today's Topics: > > 1. Resolution 23 raster problems (Felix Hartmann) > > > > ---------- Forwarded message ---------- > From: Felix Hartmann <extremecarver(a)gmail.com> > To: Development list for mkgmap <mkgmap-dev(a)lists.mkgmap.org.uk> > Cc: > Bcc: > Date: Fri, 23 Apr 2021 18:01:15 +0800 > Subject: [mkgmap-dev] Resolution 23 raster problems > Any ideas why this is happening? > Some of the contourlines intersect each other at resolution 23 - I > especially used reduce-point-density=1.0 to try to stop it. > > java -jar -Xmx47000M /home/contourlines/mkgmap.jar --keep-going > --dem-poly=/home/contourlines/bounds/bayern.poly > --series-name=openmtbmap-cntrs-DBY-23-Apr-2021 > --dem=/home/contourlines/hgt/VIEW1/,/home/contourlines/hgt/SRTM1v3.0,/home/contourlines/hgt/VIEW3/,/home/contourlines/hgt/SRTM3v3.0/ > --dem-dists=5520 --max-jobs=7 --reduce-point-density=1 --transparent > --style-file=srtm24 --latin1 95260000.osm.pbf > > Is there any way to improve this? > > Look at the screenshots - the small one is resolution 23, the big one 24. > If using 10m interval contourlines this is pretty obviously not ideal. > Resolution 24 could be nicer - but I guess this is more a problem of > phyghtmap (while groundtruth can produce higher quality), but resolution 23 > really becomes confusing vs 24...: > [image: Simplification_23.PNG] > [image: Oiginal_24.PNG] > > If pictures don't show > https://openmtbmap.org/23.PNG > https://openmtbmap.org/24.PNG > -- > Felix Hartman - Openmtbmap.org & VeloMap.org > > _______________________________________________ > mkgmap-dev mailing list > mkgmap-dev(a)lists.mkgmap.org.uk > https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

4 8