- mkgmap-dev - The mkgmap lists

mkgmap r4042 improves interpolation and thus reduces DEM sizes
by Gerd Petermann 10 Jan '18

10 Jan '18

Hi all, see http://www.mkgmap.org.uk/websvn/revision.php?repname=mkgmap&rev=4042 Details are discussed here: http://www.mkgmap.org.uk/pipermail/mkgmap-dev/2018q1/027628.html Thanks to Andrzej for this patch, I think it is another good improvement. A nice side effect of the better interpolation is that DEM sizes are a bit smaller. So, maybe also try higher dem-dist values now. Gerd

1 0

one mistake less in DEM
by Frank Stinner 09 Jan '18

09 Jan '18

Hi, there is a new BuildDEMFile-version with a rectified encoder. This should help on maps with strong changes in height. Example: East 79,335° / North 42,554°, PointDistance 55000 -> 0,004610° Hi Gerd, there seems to be a limitation for my variable "sumH" for the calculation of hunit. Perhaps garmin use only a 2-byte-var. I have included in CodingTypeStd.addValue() after "SumH += dh;": if (SumH + unitdelta + 1 >= 0xFFFF) SumH -= 0x10000; I have no idea, why not "if (SumH + unitdelta > 0xFFFF)" or so. I have done the same with CodingTypePlateauFollowerNotZero.AddValue() and CodingTypePlateauFollowerZero.AddValue(). Frank

2 1

Re: [mkgmap-dev] Performance with zipped hgt files
by Frank Stinner 09 Jan '18

09 Jan '18

Hi Gerd, at last you need the data uncompressed in a buffer, right? That's why i see no difference between using compressed or uncompressed hgt files. Compressed files need a little bit more time for uncompressing, but after that the data need the same size in memory. That's why i see no vantage for mkgmap if the data are stored in an other form. It can only save disc space, no heap space. What speaks against a limitation for the extent of a maptile? This limited the count of hgt's for one maptile too. You have the option --max-areas for splitter. We need only an additional option like --maxextent or whatever. Perhaps we have then maptiles without points, lines or areas (on oceans, deserts or so), but why not? Frank

3 7

DEM Noddy Guide
by Mike Baggaley 09 Jan '18

09 Jan '18

Hi all, I've been following the discussions of DEM with interest, but have so far not attempted to create a map including this. Is there a guide to the steps involved in producing the DEM data? I have contour data in OSM format - is there a simple way to get this into a suitable input format (or is there a specification of the input format available, so I can knock up something to convert it)? I'm assuming that the height data contains the same information as contour data but in a different format. Thanks, Mike

4 16

Performance with zipped hgt files
by Gerd Petermann 08 Jan '18

08 Jan '18

Hi all, I see no way to improve the handling of zipped files much more. We have two conflicting optimization goals here: 1) reduce heap memory by storing less data in buffers (risking additional rather slow unzip actions) 2) reduce run time be storing more data in buffers (risking OutOfMemoryException) The problem is that the DEM algo needs more or less random access to the data of one or more *.hgt.zip files, and I see no simple way to change that now. I did a few tests now with r4036 and a map for scandinavia and hgt files in 1'' resolution. The 380 unzipped hgt files require ~ 9.4 GB. When stored in a NTFS compressed folder the required disk space is ~ 37%, when stored in single zipped files (*.hgt.zip) the size is ~1.3 GB, means ~14%. I've used a simple poly file to cut scandinavia out of an older europe.o5m with osmconvert and used splitter with max-nodes=1000000. This gave me 174 tiles. I used the following command to create a map: java -Xmx6800m -jar d:\mkgmap\dist\mkgmap.jar -c d:\dbg\dev-addr.opt --output-dir=.\map --block-size=2048 --max-jobs=4 --overview-dem-dist=55000 --dem-poly=f:\osm\scandinavia.poly --dem=f:\srtm3_1_zip --dem-dists=3312,5846,8948,12646,20282 --gmapi -c e:\scand\tiles\template.args I repeated this command two times, once with the uncompressed files, once with the NTFS compressed files. All worked fine, java heap memory was no problem, but run time was different: 953 secs for the uncompressed files 1084 secs for the NTFS compressed files 1538 secs for the zipped files I think NTFS compressed folder is a good compromise if disk space matters. I hope the numbers are similar for compressing Linux file systems. I plan to implement a new dem-temp=[dir] option. If given, mkgmap could create temp files for the unzipped hgt data. Gerd

4 8

Error when not using dem-poly
by nwillink 08 Jan '18

08 Jan '18

Hi Gerd I'm using latest r4036 I'm applying what you suggested to Minko to avoid contours been given a dem. In my dem.args I have dem=e:\dem\europe\ overview-dem-dist=888888 dem-dists=3312,13248,26512,53024 and get the following error: Number of MapFailedExceptions: 0 Exception in thread "main" java.lang.NullPointerException at uk.me.parabola.mkgmap.reader.hgt.HGTConverter.getElevation(HGTConvert er.java:89) at uk.me.parabola.mkgmap.reader.hgt.HGTConverter.getHeights(HGTConverter .java:287) at uk.me.parabola.imgfmt.app.dem.DEMSection.calcTiles(DEMSection.java:10 5) at uk.me.parabola.imgfmt.app.dem.DEMSection.<init>(DEMSection.java:78) at uk.me.parabola.imgfmt.app.dem.DEMFile.calc(DEMFile.java:60) at uk.me.parabola.mkgmap.build.MapBuilder.makeMap(MapBuilder.java:340) at uk.me.parabola.mkgmap.combiners.OverviewBuilder.writeOverviewMap(Over viewBuilder.java:191) at uk.me.parabola.mkgmap.combiners.OverviewBuilder.onFinish(OverviewBuil der.java:103) at uk.me.parabola.mkgmap.main.Main.endOptions(Main.java:617) at uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.ja va:128) at uk.me.parabola.mkgmap.main.Main.mainStart(Main.java:136) at uk.me.parabola.mkgmap.main.Main.main(Main.java:107) 1) When I add dem-poly=areas.poly the error disappears. 2) I also don't get the error if I replace -c dem-args with --dem=e:\dem\europe\ --overview-dem-dist=888888 --dem-dists=3312,13248,26512,53024 3) This only seems to affect certain pbfs. file: http://files.mkgmap.org.uk/download/385/12345678.osm.pbf r Nick -- Sent from: http://gis.19327.n8.nabble.com/Mkgmap-Development-f5324443.html

4 9

Re: [mkgmap-dev] r4033: new dem options are now documented
by EthnicFood IsGreat 06 Jan '18

06 Jan '18

> Date: Sat, 6 Jan 2018 16:51:43 +0000 > From: Gerd Petermann <gpetermann_muenchen(a)hotmail.com> > To: Development list for mkgmap <mkgmap-dev(a)lists.mkgmap.org.uk> > Subject: Re: [mkgmap-dev] r4033: new dem options are now documented > Message-ID: > <AM2PR08MB029297B845D858CD564D57BD9E1D0(a)AM2PR08MB0292.eurprd08.prod.outlook.com> > > Content-Type: text/plain; charset="us-ascii" > > Hi Mark, > > my understanding is that files with version 2.1 have many voids. You find them here: > https://dds.cr.usgs.gov/srtm/version2_1/SRTM3/ > The files from here seem to have few voids : http://viewfinderpanoramas.org/dem3.html > > Gerd > > ________________________________________ > Von: mkgmap-dev <mkgmap-dev-bounces(a)lists.mkgmap.org.uk> im Auftrag von EthnicFood IsGreat <ethnicfoodisgreat(a)gmail.com> > Gesendet: Samstag, 6. Januar 2018 17:42:41 > An: mkgmap-dev(a)lists.mkgmap.org.uk > Betreff: Re: [mkgmap-dev] r4033: new dem options are now documented > > >> Date: Sat, 6 Jan 2018 11:09:39 +0000 >> From: Gerd Petermann <GPetermann_muenchen(a)hotmail.com> >> To: "mkgmap-dev(a)lists.mkgmap.org.uk" <mkgmap-dev(a)lists.mkgmap.org.uk> >> Subject: [mkgmap-dev] r4033: new dem options are now documented >> >> Hi all, >> >> see http://www.mkgmap.org.uk/doc/options >> I've added them in a chapter headed "Hill Shading (DEM) options" >> This also means that you no longer need the --x- prefix when using r4033 or newer. >> >> Please suggest improvements, esp. if you have found rules to determine "good" dem-dists values. >> If possible I'd like to do the calculations in mkgmap instead of asking each user to find out what is good. >> Maybe option --dem should be changed to --dem-source before merging to trunk? >> >> I think these are the other open problems now (no particular order): >> -- determine optimal block-size to avoid MapFailedExeption with large DEM files, if that is too complex we'll change the default from 512 to 2048. >> -- find possible encoding error for low resolutions, esp. in overview maps. Frank Stinner tries to find a solution for this :-) >> -- Problems with java heap when creating large or highly detailed maps: >> Zipped files require more heap memory because the current implementation has to keep a copy of the unzipped content while unzipped files >> are kept in MappedByteBuffers which don't require java heap. On the other hand they maybe cause other problems as those reported by Carlos: >> http://gis.19327.n8.nabble.com/mkgmap-r4025-implements-x-dem-poly-option-tp… >> One work around would be to keep the unzipped data in temporary files (more I/O), another solution might be to further change the algorithm >> so that it doesn't require full random access to all(!) hgt files "touched" by the overview map. I've already tried to change that with r4027 >> so that only some rows are required, but that code is not very elegant. I hope I find a better solution for this. >> @Frank: I think BuildDEMFile also has this problem, right? >> -- The code needs further clean up , javadoc and unit tests. >> >> Thanks for the good feedback so far, I think this will be another big improvement if merged into trunk. >> >> Gerd >> >> ------------------------------ > Thanks for the documentation Gerd. I look forward to creating my first > map with the DEM included. You mention that some sources of .hgt files > contain voids. Can you specify which ones you're aware of that should > be avoided? > > Mark > Thanks Gerd!

1 0

r4033: new dem options are now documented
by Gerd Petermann 06 Jan '18

06 Jan '18

Hi all, see http://www.mkgmap.org.uk/doc/options I've added them in a chapter headed "Hill Shading (DEM) options" This also means that you no longer need the --x- prefix when using r4033 or newer. Please suggest improvements, esp. if you have found rules to determine "good" dem-dists values. If possible I'd like to do the calculations in mkgmap instead of asking each user to find out what is good. Maybe option --dem should be changed to --dem-source before merging to trunk? I think these are the other open problems now (no particular order): -- determine optimal block-size to avoid MapFailedExeption with large DEM files, if that is too complex we'll change the default from 512 to 2048. -- find possible encoding error for low resolutions, esp. in overview maps. Frank Stinner tries to find a solution for this :-) -- Problems with java heap when creating large or highly detailed maps: Zipped files require more heap memory because the current implementation has to keep a copy of the unzipped content while unzipped files are kept in MappedByteBuffers which don't require java heap. On the other hand they maybe cause other problems as those reported by Carlos: http://gis.19327.n8.nabble.com/mkgmap-r4025-implements-x-dem-poly-option-tp… One work around would be to keep the unzipped data in temporary files (more I/O), another solution might be to further change the algorithm so that it doesn't require full random access to all(!) hgt files "touched" by the overview map. I've already tried to change that with r4027 so that only some rows are required, but that code is not very elegant. I hope I find a better solution for this. @Frank: I think BuildDEMFile also has this problem, right? -- The code needs further clean up , javadoc and unit tests. Thanks for the good feedback so far, I think this will be another big improvement if merged into trunk. Gerd

5 14

r4035 fixes problem with *.hgt.zip files
by Gerd Petermann 06 Jan '18

06 Jan '18

Hi Minko, Since r4027 single zipped hgt files (*.hgt.zip) caused problems, mkgmap remembered the wrong name and tried to find the file again and again (silentiy and without success) for each height value needed for the DEM calculation. Gerd

2 1

Re: [mkgmap-dev] r4033: new dem options are now documented
by EthnicFood IsGreat 06 Jan '18

06 Jan '18

> Date: Sat, 6 Jan 2018 11:09:39 +0000 > From: Gerd Petermann <GPetermann_muenchen(a)hotmail.com> > To: "mkgmap-dev(a)lists.mkgmap.org.uk" <mkgmap-dev(a)lists.mkgmap.org.uk> > Subject: [mkgmap-dev] r4033: new dem options are now documented > > Hi all, > > see http://www.mkgmap.org.uk/doc/options > I've added them in a chapter headed "Hill Shading (DEM) options" > This also means that you no longer need the --x- prefix when using r4033 or newer. > > Please suggest improvements, esp. if you have found rules to determine "good" dem-dists values. > If possible I'd like to do the calculations in mkgmap instead of asking each user to find out what is good. > Maybe option --dem should be changed to --dem-source before merging to trunk? > > I think these are the other open problems now (no particular order): > -- determine optimal block-size to avoid MapFailedExeption with large DEM files, if that is too complex we'll change the default from 512 to 2048. > -- find possible encoding error for low resolutions, esp. in overview maps. Frank Stinner tries to find a solution for this :-) > -- Problems with java heap when creating large or highly detailed maps: > Zipped files require more heap memory because the current implementation has to keep a copy of the unzipped content while unzipped files > are kept in MappedByteBuffers which don't require java heap. On the other hand they maybe cause other problems as those reported by Carlos: > http://gis.19327.n8.nabble.com/mkgmap-r4025-implements-x-dem-poly-option-tp… > One work around would be to keep the unzipped data in temporary files (more I/O), another solution might be to further change the algorithm > so that it doesn't require full random access to all(!) hgt files "touched" by the overview map. I've already tried to change that with r4027 > so that only some rows are required, but that code is not very elegant. I hope I find a better solution for this. > @Frank: I think BuildDEMFile also has this problem, right? > -- The code needs further clean up , javadoc and unit tests. > > Thanks for the good feedback so far, I think this will be another big improvement if merged into trunk. > > Gerd > > ------------------------------ Thanks for the documentation Gerd. I look forward to creating my first map with the DEM included. You mention that some sources of .hgt files contain voids. Can you specify which ones you're aware of that should be avoided? Mark

2 1