January 2018 - mkgmap-dev - The mkgmap lists

Is dem-tdb branch ready for trunk ?
by Gerd Petermann 29 Jan '18

29 Jan '18

Hi all, I am not aware of any erros in r4091, so I think it is time to merge it into trunk. I see only one problem: HGT data changes rarely, so I'd prefer to do the costly DEM calculations only once and be able to store the results. I've already described how to do this here [1] but I'd prefer to have a container format that allows mkgmap to extract the Garmin DEM bitstream data for a given lat/lon pair from a file. Such a container could contain the DEM data for one or more dem-dist values. It could be empty first and grow each time you calculate DEM for a new area. In subsequent executions of mkgmap it would check if the container already contains the data for the wanted area. Advantage would be a faster tile compilation and less power consumption, disadvantage would be the additional disk space and higher complexity. [1] http://gis.19327.n8.nabble.com/Performance-with-zipped-hgt-files-tp5909756p… Gerd

7 14

gmapi reader?
by Gerd Petermann 29 Jan '18

29 Jan '18

Hi Steve, I hope you can help me out here. I've implemented a new option --x-dem-reuse which allows to copy the DEM file from an older map. My idea is that the option specifies the path to either 1) a gmapsupp or 2) a gmap folder or 3) a directory containing *.img files from a previous run with similar dem-dist option and tile boundaries. If given, mkgmap tries to find the matching *.DEM in any of these formats, checks if the content matches the wanted boundary and dem-dist option and - if ok - uses the existing file instead of doing the DEM calculations again. At the moment I have some working quick+dirty code (see attached patch). I assume the new code in MapBuilder should go somewhere in package uk.me.parabola.imgfmt.app, maybe a new class ImgFinder or so? Or do we already have such code in mkgmap? What I also don't like is that it copies the data from the existing file into memory. Is there already an easy way to avoid that? Gerd

3 7

splitter r590: (undocumented) option align-for-dem was removed
by Gerd Petermann 27 Jan '18

27 Jan '18

Hi all, see http://www.mkgmap.org.uk/websvn/revision.php?repname=splitter&rev=590 Please make sure to remove the option from your scripts. Gerd

1 0

Splitter doc patch
by Carlos Dávila 26 Jan '18

26 Jan '18

Hi Gerd Attached is a patch that tries to improve splitter documentation and fix some typos. Please commit it if you find it correct.

2 1

WG: How to handle missing hgt files or files with voids?
by Gerd Petermann 25 Jan '18

25 Jan '18

Hi all, replied to David instead of the list... See below Gerd ________________________________________ Von: Gerd Petermann <gpetermann_muenchen(a)hotmail.com> Gesendet: Donnerstag, 25. Januar 2018 18:08 An: Pazapas Betreff: AW: [mkgmap-dev] How to handle missing hgt files or files with voids? Hi all, see http://www.mkgmap.org.uk/websvn/revision.php?repname=mkgmap&rev=4089 and http://www.mkgmap.org.uk/websvn/revision.php?repname=mkgmap&rev=4090 @David: You are free to use any source, the program will always try to read hgt data. This is only about the message that is produced when a hgt file is not found. With 4090 you will see a list of files that were not found although we know that hgt info exists. Hope that makes it clearer now? Gerd ________________________________________ Von: Pazapas <pazapas(a)orange.fr> Gesendet: Donnerstag, 25. Januar 2018 17:38 An: Gerd Petermann Betreff: Re: [mkgmap-dev] How to handle missing hgt files or files with voids? Hello, Why a list of missing hgt is needed ? I would prefer that mkgmap adds itself avirtual void tile and writes a warning in console or output a report of the compilation in a text file. We then are free to use any sourc of custom hgt files. Again, chapeau bas for all your work, David Pazapas > Le 25 janv. 2018 à 13:11, Gerd Petermann <gpetermann_muenchen(a)hotmail.com> a écrit : > > Hi Nick, > > I think it is not a matter of speed, the file is only read once. > I think I'll keep the human readable format, the size of mkgmap.jar probably doesn't > matter much. > > Reg. --dem-interpolation see > http://www.mkgmap.org.uk/websvn/revision.php?repname=mkgmap&rev=4087 > > Gerd > > ________________________________________ > Von: mkgmap-dev <mkgmap-dev-bounces(a)lists.mkgmap.org.uk> im Auftrag von osm@pinns <osm(a)pinns.co.uk> > Gesendet: Donnerstag, 25. Januar 2018 13:03 > An: mkgmap-dev(a)lists.mkgmap.org.uk > Betreff: Re: [mkgmap-dev] How to handle missing hgt files or files with voids? > > Hi Gerd > > > A binary file sounds very 'Garminesque' and if its a matter of speed then I would prefer such a format - > > btw has the --dem-interpolation option been implemented yet ? > > Nick > > On 25/01/2018 11:59, thesurveyor(a)wolke7.net<mailto:thesurveyor@wolke7.net> wrote: > Hi Gerd, > > first many thanks for your work on mkgmap. > > for the list of available hgt files: I prefer a human readable format > > Regards, > Gert > > > > Gesendet: Donnerstag, 25. Januar 2018 um 12:40 Uhr > Von: "Gerd Petermann" <gpetermann_muenchen(a)hotmail.com><mailto:gpetermann_muenchen@hotmail.com> > An: "Development list for mkgmap" <mkgmap-dev(a)lists.mkgmap.org.uk><mailto:mkgmap-dev@lists.mkgmap.org.uk> > Betreff: Re: [mkgmap-dev] How to handle missing hgt files or files with voids? > Hi all, > > I've played with the list of available hgt files created by Henning. > I've coded a reader for the list so that mkgmap can > check if a missing hgt file should exist or if it covers ocean. > The idea is to print a warning if a file was not found when it should be. > The check is fast, but the list is long, so I wonder > if I should store it "as is" in the mkgmap binary like we do with LocatorConfig.xml > or if I should use another format. > The most obvious for me would be a binary file containing 180*360 bits, a 1 bit for every > file that should exist. > > What do you think? > > Gerd > > > ________________________________________ > Von: mkgmap-dev <mkgmap-dev-bounces(a)lists.mkgmap.org.uk><mailto:mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <gpetermann_muenchen(a)hotmail.com><mailto:gpetermann_muenchen@hotmail.com> > Gesendet: Montag, 15. Januar 2018 14:25 > An: Development list for mkgmap > Betreff: Re: [mkgmap-dev] How to handle missing hgt files or files with voids? > > Hi Henning, > > I'll see if I can use srtm.zip. > I think voids are interesting. When I started coding DEM support I searched my PC for *.hgt files and found them in several places. > Later I found out that some of those files have voids and that better data exists (assuming that files without voids have fewer errors) > So, I think it would be good to have a list of those files that have voids. > > I'll see what is needed for this. > > Gerd > > ________________________________________ > Von: mkgmap-dev <mkgmap-dev-bounces(a)lists.mkgmap.org.uk><mailto:mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Henning Scholland <osm(a)hscholland.de><mailto:osm@hscholland.de> > Gesendet: Montag, 15. Januar 2018 12:21:09 > An: mkgmap-dev(a)lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk> > Betreff: Re: [mkgmap-dev] How to handle missing hgt files or files with voids? > > This would be the list of all hgt-files available on viewfinder. > > In each zip-file of viewfinder there are 6° in E-W direction and 3° in > N-S. The scheme of naming is also simple. That list I also attached. > > In general I think it's better to just output a list, which > hgt-files/viefinder-tiles are missing than writing a downloader. I mean > in total it's 13GB of data. If you need to download it manually, you are > more aware of what you are doing and not just download them all. > > Regarding voids: I think if there is a possibility the user can fix the > hgt file, A list can be helpful. Otherwise I don't know if a list would > be helpful. Only use case I see is to ask the user to use better data. > But then we need a list, which source is best in which area. And finally > all the different sources need to match each other. Otherwise it will > lead to a hard border between 2 sources. I think this will be much to > complicated... > > Maybe for advanced users there can be a Warning message, hgt-file > N00E100.hgt contains 3 voids. So user can compare different sources > manually. > > Henning > >> On 15.01.2018 16:55, Gerd Petermann wrote: >> Hi Henning, >> >> checking data is not that easy, but I'll think about that option. >> >> I'd like to add a short howto that describes ways to get good hgt data, but I don't know any method for Windows users. >> phyghtmap is good, but installation on Win is complex and doc for that looks out-aged. >> Srtm2Osm seems out-dated, as well as Groundtruth. >> >> So, up to now I try to guess what areas I have to download from viewfinder. >> Any hints ? >> >> Gerd >> >> ________________________________________ >> Von: mkgmap-dev <mkgmap-dev-bounces(a)lists.mkgmap.org.uk><mailto:mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Henning Scholland <osm(a)hscholland.de><mailto:osm@hscholland.de> >> Gesendet: Montag, 15. Januar 2018 09:05:24 >> An: Development list for mkgmap >> Betreff: Re: [mkgmap-dev] How to handle missing hgt files or files with voids? >> >> Hi Gerd, >> >> regarding 1) I can create a list of existing tiles of Viewfinder (which covers all land area so far I see). But it will be a long list. I remember it's about 25000 files in total. So maybe it's easier to create a polygon or at least take the list and let software create the polygon. An easier way can be to check if data is available. No data might be sea. Of course it's not 100% sure, as there are deserts and ferries. >> >> Another idea I had during the weekend was to output a osm-file with missing areas. So then user can check. But I came to conclusion, that it's easily visible in the map later. So I'm not sure it's useful at all. >> >> Henning >> >> On 2018-01-15 15:45, Gerd Petermann wrote: >> >> Hi all, >> >> 1) The current code (only) writes a log message with severity WARNING for a missing hgt file: >> "file xxx.hgt not found. Is expected to cover sea." >> I wonder if we can create a list of files that can exist or maybe a polygon that can be tested so that >> tiles which are knwon to cover land are reported as errors. >> When you look at http://viewfinderpanoramas.org/Coverage%20map%20viewfinderpanoramas_org3.htm >> you can see that such a polygon would be quite complex. >> Did anybody already try to define such a polygon or list of files that would cover only ocean? >> >> 2) Should mkgmap print a warning when voids in the hgt file(s) have an influence on the DEM data? >> I think yes, but it is a bit more complex to measure the real effect. >> Reason: We read the 4 hgt values next to the wanted DEM point and use interpolation to calculate the height >> that is stored in the DEM file. >> If one or two of the 4 points are voids they may still have no influence if the wanted point is very close to one that >> is not a void. Another problem is that a single void might be used several times. >> So, another option would be to add an option like --dem-check-hgt which could read all existing hgt files and >> report the number of voids for each. I'd prefer this because it would have no impact on performance for normal >> processing. Or maybe there is already a tool for this that works on all platforms (win,linux,mac) ? >> >> Comments? >> >> Gerd >> >> >> >> >> >> _______________________________________________ >> mkgmap-dev mailing list >> mkgmap-dev(a)lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk><mailto:mkgmap-dev@lists.mkgmap.org.uk><mailto:mkgmap-dev@lists.mkgmap.org.uk> >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev >> >> >> >> >> _______________________________________________ >> mkgmap-dev mailing list >> mkgmap-dev(a)lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk> >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev >> > > _______________________________________________ > mkgmap-dev mailing list > mkgmap-dev(a)lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > _______________________________________________ > mkgmap-dev mailing list > mkgmap-dev(a)lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > > > > _______________________________________________ > mkgmap-dev mailing list > mkgmap-dev(a)lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >

1 0

How to handle missing hgt files or files with voids?
by Gerd Petermann 25 Jan '18

25 Jan '18

Hi all, 1) The current code (only) writes a log message with severity WARNING for a missing hgt file: "file xxx.hgt not found. Is expected to cover sea." I wonder if we can create a list of files that can exist or maybe a polygon that can be tested so that tiles which are knwon to cover land are reported as errors. When you look at http://viewfinderpanoramas.org/Coverage%20map%20viewfinderpanoramas_org3.htm you can see that such a polygon would be quite complex. Did anybody already try to define such a polygon or list of files that would cover only ocean? 2) Should mkgmap print a warning when voids in the hgt file(s) have an influence on the DEM data? I think yes, but it is a bit more complex to measure the real effect. Reason: We read the 4 hgt values next to the wanted DEM point and use interpolation to calculate the height that is stored in the DEM file. If one or two of the 4 points are voids they may still have no influence if the wanted point is very close to one that is not a void. Another problem is that a single void might be used several times. So, another option would be to add an option like --dem-check-hgt which could read all existing hgt files and report the number of voids for each. I'd prefer this because it would have no impact on performance for normal processing. Or maybe there is already a tool for this that works on all platforms (win,linux,mac) ? Comments? Gerd

7 12

Commit r4088: Fix problems in LocatorConfig.xml
by svn commit 25 Jan '18

25 Jan '18

Version mkgmap-r4088 was committed by gerd on Thu, 25 Jan 2018 Fix problems in LocatorConfig.xml - Tonga is a drive-on-left country - Pitcairn is one island, the country is called "Pitcairn Islands" LocatorConfig.patch by Carlos Davida http://www.mkgmap.org.uk/websvn/revision.php?repname=mkgmap&rev=4088

1 0

missing link
by nwillink 24 Jan '18

24 Jan '18

@Steve Hi Steve Just to say that clicking on 'mkgmap' gives me a 404 : http://files.mkgmap.org.uk/ r Nick -- Sent from: http://gis.19327.n8.nabble.com/Mkgmap-Development-f5324443.html

3 4

Minor corrections in LocatorConfig
by Carlos Dávila 24 Jan '18

24 Jan '18

Attached is a patch that fixes a couple of issues regarding "both drive-on-left and drive-on-right" wrong warnings.

1 0

DEM: performance with bicubic interpolation
by Gerd Petermann 24 Jan '18

24 Jan '18

Hi all, the bicubic interpolation that was introduced with r4072 is quite slow compared to bilinear interpolation. I have already changed the code to use a faster method in r4076 but you should still expect that run time for DEM calculation increases by up to ~100% compared to code r4071, actual values depend on hgt resolution and your dem-dists values. So, I wonder if I should add an option that allows to switch this - probably better - interpolation off or maybe add an option to switch it on so that faster interpolation is used by default. The differences in the map are rather small and may be smaller than the errros that come with the input hgt data. Gerd

5 21