January 2018 - mkgmap-dev - The mkgmap lists

DEM Noddy Guide
by Mike Baggaley 09 Jan '18

09 Jan '18

Hi all, I've been following the discussions of DEM with interest, but have so far not attempted to create a map including this. Is there a guide to the steps involved in producing the DEM data? I have contour data in OSM format - is there a simple way to get this into a suitable input format (or is there a specification of the input format available, so I can knock up something to convert it)? I'm assuming that the height data contains the same information as contour data but in a different format. Thanks, Mike

4 16

Performance with zipped hgt files
by Gerd Petermann 08 Jan '18

08 Jan '18

Hi all, I see no way to improve the handling of zipped files much more. We have two conflicting optimization goals here: 1) reduce heap memory by storing less data in buffers (risking additional rather slow unzip actions) 2) reduce run time be storing more data in buffers (risking OutOfMemoryException) The problem is that the DEM algo needs more or less random access to the data of one or more *.hgt.zip files, and I see no simple way to change that now. I did a few tests now with r4036 and a map for scandinavia and hgt files in 1'' resolution. The 380 unzipped hgt files require ~ 9.4 GB. When stored in a NTFS compressed folder the required disk space is ~ 37%, when stored in single zipped files (*.hgt.zip) the size is ~1.3 GB, means ~14%. I've used a simple poly file to cut scandinavia out of an older europe.o5m with osmconvert and used splitter with max-nodes=1000000. This gave me 174 tiles. I used the following command to create a map: java -Xmx6800m -jar d:\mkgmap\dist\mkgmap.jar -c d:\dbg\dev-addr.opt --output-dir=.\map --block-size=2048 --max-jobs=4 --overview-dem-dist=55000 --dem-poly=f:\osm\scandinavia.poly --dem=f:\srtm3_1_zip --dem-dists=3312,5846,8948,12646,20282 --gmapi -c e:\scand\tiles\template.args I repeated this command two times, once with the uncompressed files, once with the NTFS compressed files. All worked fine, java heap memory was no problem, but run time was different: 953 secs for the uncompressed files 1084 secs for the NTFS compressed files 1538 secs for the zipped files I think NTFS compressed folder is a good compromise if disk space matters. I hope the numbers are similar for compressing Linux file systems. I plan to implement a new dem-temp=[dir] option. If given, mkgmap could create temp files for the unzipped hgt data. Gerd

4 8

Error when not using dem-poly
by nwillink 08 Jan '18

08 Jan '18

Hi Gerd I'm using latest r4036 I'm applying what you suggested to Minko to avoid contours been given a dem. In my dem.args I have dem=e:\dem\europe\ overview-dem-dist=888888 dem-dists=3312,13248,26512,53024 and get the following error: Number of MapFailedExceptions: 0 Exception in thread "main" java.lang.NullPointerException at uk.me.parabola.mkgmap.reader.hgt.HGTConverter.getElevation(HGTConvert er.java:89) at uk.me.parabola.mkgmap.reader.hgt.HGTConverter.getHeights(HGTConverter .java:287) at uk.me.parabola.imgfmt.app.dem.DEMSection.calcTiles(DEMSection.java:10 5) at uk.me.parabola.imgfmt.app.dem.DEMSection.<init>(DEMSection.java:78) at uk.me.parabola.imgfmt.app.dem.DEMFile.calc(DEMFile.java:60) at uk.me.parabola.mkgmap.build.MapBuilder.makeMap(MapBuilder.java:340) at uk.me.parabola.mkgmap.combiners.OverviewBuilder.writeOverviewMap(Over viewBuilder.java:191) at uk.me.parabola.mkgmap.combiners.OverviewBuilder.onFinish(OverviewBuil der.java:103) at uk.me.parabola.mkgmap.main.Main.endOptions(Main.java:617) at uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.ja va:128) at uk.me.parabola.mkgmap.main.Main.mainStart(Main.java:136) at uk.me.parabola.mkgmap.main.Main.main(Main.java:107) 1) When I add dem-poly=areas.poly the error disappears. 2) I also don't get the error if I replace -c dem-args with --dem=e:\dem\europe\ --overview-dem-dist=888888 --dem-dists=3312,13248,26512,53024 3) This only seems to affect certain pbfs. file: http://files.mkgmap.org.uk/download/385/12345678.osm.pbf r Nick -- Sent from: http://gis.19327.n8.nabble.com/Mkgmap-Development-f5324443.html

4 9

Re: [mkgmap-dev] r4033: new dem options are now documented
by EthnicFood IsGreat 06 Jan '18

06 Jan '18

> Date: Sat, 6 Jan 2018 16:51:43 +0000 > From: Gerd Petermann <gpetermann_muenchen(a)hotmail.com> > To: Development list for mkgmap <mkgmap-dev(a)lists.mkgmap.org.uk> > Subject: Re: [mkgmap-dev] r4033: new dem options are now documented > Message-ID: > <AM2PR08MB029297B845D858CD564D57BD9E1D0(a)AM2PR08MB0292.eurprd08.prod.outlook.com> > > Content-Type: text/plain; charset="us-ascii" > > Hi Mark, > > my understanding is that files with version 2.1 have many voids. You find them here: > https://dds.cr.usgs.gov/srtm/version2_1/SRTM3/ > The files from here seem to have few voids : http://viewfinderpanoramas.org/dem3.html > > Gerd > > ________________________________________ > Von: mkgmap-dev <mkgmap-dev-bounces(a)lists.mkgmap.org.uk> im Auftrag von EthnicFood IsGreat <ethnicfoodisgreat(a)gmail.com> > Gesendet: Samstag, 6. Januar 2018 17:42:41 > An: mkgmap-dev(a)lists.mkgmap.org.uk > Betreff: Re: [mkgmap-dev] r4033: new dem options are now documented > > >> Date: Sat, 6 Jan 2018 11:09:39 +0000 >> From: Gerd Petermann <GPetermann_muenchen(a)hotmail.com> >> To: "mkgmap-dev(a)lists.mkgmap.org.uk" <mkgmap-dev(a)lists.mkgmap.org.uk> >> Subject: [mkgmap-dev] r4033: new dem options are now documented >> >> Hi all, >> >> see http://www.mkgmap.org.uk/doc/options >> I've added them in a chapter headed "Hill Shading (DEM) options" >> This also means that you no longer need the --x- prefix when using r4033 or newer. >> >> Please suggest improvements, esp. if you have found rules to determine "good" dem-dists values. >> If possible I'd like to do the calculations in mkgmap instead of asking each user to find out what is good. >> Maybe option --dem should be changed to --dem-source before merging to trunk? >> >> I think these are the other open problems now (no particular order): >> -- determine optimal block-size to avoid MapFailedExeption with large DEM files, if that is too complex we'll change the default from 512 to 2048. >> -- find possible encoding error for low resolutions, esp. in overview maps. Frank Stinner tries to find a solution for this :-) >> -- Problems with java heap when creating large or highly detailed maps: >> Zipped files require more heap memory because the current implementation has to keep a copy of the unzipped content while unzipped files >> are kept in MappedByteBuffers which don't require java heap. On the other hand they maybe cause other problems as those reported by Carlos: >> http://gis.19327.n8.nabble.com/mkgmap-r4025-implements-x-dem-poly-option-tp… >> One work around would be to keep the unzipped data in temporary files (more I/O), another solution might be to further change the algorithm >> so that it doesn't require full random access to all(!) hgt files "touched" by the overview map. I've already tried to change that with r4027 >> so that only some rows are required, but that code is not very elegant. I hope I find a better solution for this. >> @Frank: I think BuildDEMFile also has this problem, right? >> -- The code needs further clean up , javadoc and unit tests. >> >> Thanks for the good feedback so far, I think this will be another big improvement if merged into trunk. >> >> Gerd >> >> ------------------------------ > Thanks for the documentation Gerd. I look forward to creating my first > map with the DEM included. You mention that some sources of .hgt files > contain voids. Can you specify which ones you're aware of that should > be avoided? > > Mark > Thanks Gerd!

1 0

r4033: new dem options are now documented
by Gerd Petermann 06 Jan '18

06 Jan '18

Hi all, see http://www.mkgmap.org.uk/doc/options I've added them in a chapter headed "Hill Shading (DEM) options" This also means that you no longer need the --x- prefix when using r4033 or newer. Please suggest improvements, esp. if you have found rules to determine "good" dem-dists values. If possible I'd like to do the calculations in mkgmap instead of asking each user to find out what is good. Maybe option --dem should be changed to --dem-source before merging to trunk? I think these are the other open problems now (no particular order): -- determine optimal block-size to avoid MapFailedExeption with large DEM files, if that is too complex we'll change the default from 512 to 2048. -- find possible encoding error for low resolutions, esp. in overview maps. Frank Stinner tries to find a solution for this :-) -- Problems with java heap when creating large or highly detailed maps: Zipped files require more heap memory because the current implementation has to keep a copy of the unzipped content while unzipped files are kept in MappedByteBuffers which don't require java heap. On the other hand they maybe cause other problems as those reported by Carlos: http://gis.19327.n8.nabble.com/mkgmap-r4025-implements-x-dem-poly-option-tp… One work around would be to keep the unzipped data in temporary files (more I/O), another solution might be to further change the algorithm so that it doesn't require full random access to all(!) hgt files "touched" by the overview map. I've already tried to change that with r4027 so that only some rows are required, but that code is not very elegant. I hope I find a better solution for this. @Frank: I think BuildDEMFile also has this problem, right? -- The code needs further clean up , javadoc and unit tests. Thanks for the good feedback so far, I think this will be another big improvement if merged into trunk. Gerd

5 14

r4035 fixes problem with *.hgt.zip files
by Gerd Petermann 06 Jan '18

06 Jan '18

Hi Minko, Since r4027 single zipped hgt files (*.hgt.zip) caused problems, mkgmap remembered the wrong name and tried to find the file again and again (silentiy and without success) for each height value needed for the DEM calculation. Gerd

2 1

Re: [mkgmap-dev] r4033: new dem options are now documented
by EthnicFood IsGreat 06 Jan '18

06 Jan '18

> Date: Sat, 6 Jan 2018 11:09:39 +0000 > From: Gerd Petermann <GPetermann_muenchen(a)hotmail.com> > To: "mkgmap-dev(a)lists.mkgmap.org.uk" <mkgmap-dev(a)lists.mkgmap.org.uk> > Subject: [mkgmap-dev] r4033: new dem options are now documented > > Hi all, > > see http://www.mkgmap.org.uk/doc/options > I've added them in a chapter headed "Hill Shading (DEM) options" > This also means that you no longer need the --x- prefix when using r4033 or newer. > > Please suggest improvements, esp. if you have found rules to determine "good" dem-dists values. > If possible I'd like to do the calculations in mkgmap instead of asking each user to find out what is good. > Maybe option --dem should be changed to --dem-source before merging to trunk? > > I think these are the other open problems now (no particular order): > -- determine optimal block-size to avoid MapFailedExeption with large DEM files, if that is too complex we'll change the default from 512 to 2048. > -- find possible encoding error for low resolutions, esp. in overview maps. Frank Stinner tries to find a solution for this :-) > -- Problems with java heap when creating large or highly detailed maps: > Zipped files require more heap memory because the current implementation has to keep a copy of the unzipped content while unzipped files > are kept in MappedByteBuffers which don't require java heap. On the other hand they maybe cause other problems as those reported by Carlos: > http://gis.19327.n8.nabble.com/mkgmap-r4025-implements-x-dem-poly-option-tp… > One work around would be to keep the unzipped data in temporary files (more I/O), another solution might be to further change the algorithm > so that it doesn't require full random access to all(!) hgt files "touched" by the overview map. I've already tried to change that with r4027 > so that only some rows are required, but that code is not very elegant. I hope I find a better solution for this. > @Frank: I think BuildDEMFile also has this problem, right? > -- The code needs further clean up , javadoc and unit tests. > > Thanks for the good feedback so far, I think this will be another big improvement if merged into trunk. > > Gerd > > ------------------------------ Thanks for the documentation Gerd. I look forward to creating my first map with the DEM included. You mention that some sources of .hgt files contain voids. Can you specify which ones you're aware of that should be avoided? Mark

2 1

mkgmap r4025 implements --x-dem-poly option
by Gerd Petermann 06 Jan '18

06 Jan '18

Hi all, see http://www.mkgmap.org.uk/websvn/revision.php?repname=mkgmap&rev=4025 this is useful if you create a map for a country extract like nepal (nepal-latest.osm.pbf) Here is a screenshot of a map created with the option --x-dem-poly=nepal.poly http://files.mkgmap.org.uk/download/378/nepal-with-poly.jpg The screenshot also shows some wrong reliefs, and I think I used files without many holes :-( So, next step is to find out what goes wrong with DEM in low resolutions. This might take longer, I hope Frank has an idea. It seems to be a good idea to use a rather high res for the overview map for now. For the screenshot I used --x-overview-dem-dist=276160, with --x-overview-dem-dist=88000 the problems are gone and DEM file is still only ~300KB. Gerd P.S. See also http://www.mkgmap.org.uk/websvn/revision.php?repname=mkgmap&path=%2F&rev=40… I've moved some messages into the logger, so there will be less noise from mkgmap.

5 18

Problem with --x-dem-poly
by Gerd Petermann 06 Jan '18

06 Jan '18

Hi all, I've noticed a few cases where DEM data contains "white lines" at tile borders. They are probably caused by rounding problems when calculating the intersection of the bounding polygon (--x-dem-poly) and the tile boundary. Working on it .... Gerd

3 7

some improvements with mkgmap r4027
by Gerd Petermann 05 Jan '18

05 Jan '18

Hi, see http://www.mkgmap.org.uk/websvn/revision.php?repname=mkgmap&rev=4027 It seems that there is no need for further options, so I'll document the dem options next. Gerd

1 0