r4033: new dem options are now documented
Hi all, see http://www.mkgmap.org.uk/doc/options I've added them in a chapter headed "Hill Shading (DEM) options" This also means that you no longer need the --x- prefix when using r4033 or newer. Please suggest improvements, esp. if you have found rules to determine "good" dem-dists values. If possible I'd like to do the calculations in mkgmap instead of asking each user to find out what is good. Maybe option --dem should be changed to --dem-source before merging to trunk? I think these are the other open problems now (no particular order): -- determine optimal block-size to avoid MapFailedExeption with large DEM files, if that is too complex we'll change the default from 512 to 2048. -- find possible encoding error for low resolutions, esp. in overview maps. Frank Stinner tries to find a solution for this :-) -- Problems with java heap when creating large or highly detailed maps: Zipped files require more heap memory because the current implementation has to keep a copy of the unzipped content while unzipped files are kept in MappedByteBuffers which don't require java heap. On the other hand they maybe cause other problems as those reported by Carlos: http://gis.19327.n8.nabble.com/mkgmap-r4025-implements-x-dem-poly-option-tp5... One work around would be to keep the unzipped data in temporary files (more I/O), another solution might be to further change the algorithm so that it doesn't require full random access to all(!) hgt files "touched" by the overview map. I've already tried to change that with r4027 so that only some rows are required, but that code is not very elegant. I hope I find a better solution for this. @Frank: I think BuildDEMFile also has this problem, right? -- The code needs further clean up , javadoc and unit tests. Thanks for the good feedback so far, I think this will be another big improvement if merged into trunk. Gerd
Gerd, I don't know why but mkgmap >v4025 does not work for my maps anymore. It is doing hours to compile one tile! My (experimental) DEM settings are dem-dists=3312,3312,3312,6628,9942 overview-dem-dist=16560 Map levels: levels = 0:24, 1:23, 2:22, 3:20, 4:18 overview-levels = 5:17, 6:15, 7:14 v4025 compiles ok, I updated my OFM Alps, Benelux and Germany recently. ________________________________ Van: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> namens Gerd Petermann <GPetermann_muenchen@hotmail.com> Verzonden: zaterdag 6 januari 2018 03:09:39 Aan: mkgmap-dev@lists.mkgmap.org.uk Onderwerp: [mkgmap-dev] r4033: new dem options are now documented Hi all, see http://www.mkgmap.org.uk/doc/options I've added them in a chapter headed "Hill Shading (DEM) options" This also means that you no longer need the --x- prefix when using r4033 or newer. Please suggest improvements, esp. if you have found rules to determine "good" dem-dists values. If possible I'd like to do the calculations in mkgmap instead of asking each user to find out what is good. Maybe option --dem should be changed to --dem-source before merging to trunk? I think these are the other open problems now (no particular order): -- determine optimal block-size to avoid MapFailedExeption with large DEM files, if that is too complex we'll change the default from 512 to 2048. -- find possible encoding error for low resolutions, esp. in overview maps. Frank Stinner tries to find a solution for this :-) -- Problems with java heap when creating large or highly detailed maps: Zipped files require more heap memory because the current implementation has to keep a copy of the unzipped content while unzipped files are kept in MappedByteBuffers which don't require java heap. On the other hand they maybe cause other problems as those reported by Carlos: http://gis.19327.n8.nabble.com/mkgmap-r4025-implements-x-dem-poly-option-tp5... One work around would be to keep the unzipped data in temporary files (more I/O), another solution might be to further change the algorithm so that it doesn't require full random access to all(!) hgt files "touched" by the overview map. I've already tried to change that with r4027 so that only some rows are required, but that code is not very elegant. I hope I find a better solution for this. @Frank: I think BuildDEMFile also has this problem, right? -- The code needs further clean up , javadoc and unit tests. Thanks for the good feedback so far, I think this will be another big improvement if merged into trunk. Gerd _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Minko, okay, your options look strange, but I cannot reproduce a run time problem here. Do you see this for all tiles? Maybe you have a tile with special boundaries. Please post the input file that takes so long. What is your value for JRE option -Xmx ? Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von lig fietser <ligfietser@hotmail.com> Gesendet: Samstag, 6. Januar 2018 14:30:04 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] r4033: new dem options are now documented Gerd, I don't know why but mkgmap >v4025 does not work for my maps anymore. It is doing hours to compile one tile! My (experimental) DEM settings are dem-dists=3312,3312,3312,6628,9942 overview-dem-dist=16560 Map levels: levels = 0:24, 1:23, 2:22, 3:20, 4:18 overview-levels = 5:17, 6:15, 7:14 v4025 compiles ok, I updated my OFM Alps, Benelux and Germany recently. ________________________________ Van: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> namens Gerd Petermann <GPetermann_muenchen@hotmail.com> Verzonden: zaterdag 6 januari 2018 03:09:39 Aan: mkgmap-dev@lists.mkgmap.org.uk Onderwerp: [mkgmap-dev] r4033: new dem options are now documented Hi all, see http://www.mkgmap.org.uk/doc/options I've added them in a chapter headed "Hill Shading (DEM) options" This also means that you no longer need the --x- prefix when using r4033 or newer. Please suggest improvements, esp. if you have found rules to determine "good" dem-dists values. If possible I'd like to do the calculations in mkgmap instead of asking each user to find out what is good. Maybe option --dem should be changed to --dem-source before merging to trunk? I think these are the other open problems now (no particular order): -- determine optimal block-size to avoid MapFailedExeption with large DEM files, if that is too complex we'll change the default from 512 to 2048. -- find possible encoding error for low resolutions, esp. in overview maps. Frank Stinner tries to find a solution for this :-) -- Problems with java heap when creating large or highly detailed maps: Zipped files require more heap memory because the current implementation has to keep a copy of the unzipped content while unzipped files are kept in MappedByteBuffers which don't require java heap. On the other hand they maybe cause other problems as those reported by Carlos: http://gis.19327.n8.nabble.com/mkgmap-r4025-implements-x-dem-poly-option-tp5... One work around would be to keep the unzipped data in temporary files (more I/O), another solution might be to further change the algorithm so that it doesn't require full random access to all(!) hgt files "touched" by the overview map. I've already tried to change that with r4027 so that only some rows are required, but that code is not very elegant. I hope I find a better solution for this. @Frank: I think BuildDEMFile also has this problem, right? -- The code needs further clean up , javadoc and unit tests. Thanks for the good feedback so far, I think this will be another big improvement if merged into trunk. Gerd _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Gerd, I use -Xmx4000m All tiles get stuck, doesnt matter which one. I have tried one of the Benelux, Germany, Canary Islands. I guess my splitter version is too old?? I have still r-321....(2014!). Strange because mkgmap 4025 performed well.
Hi Minko, I don't think that splitter is the problem. The major change in r4026 was for handling of zipped hgt files. Maybe you have an unexpected zip archive ? Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von lig fietser <ligfietser@hotmail.com> Gesendet: Samstag, 6. Januar 2018 15:47:06 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] r4033: new dem options are now documented Hi Gerd, I use -Xmx4000m All tiles get stuck, doesnt matter which one. I have tried one of the Benelux, Germany, Canary Islands. I guess my splitter version is too old?? I have still r-321....(2014!). Strange because mkgmap 4025 performed well.
How can I check that? And why does it go wrong with every random tile? BTW My max-nodes are 1600000 ________________________________ Van: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> namens Gerd Petermann <gpetermann_muenchen@hotmail.com> Verzonden: zaterdag 6 januari 2018 06:52:32 Aan: Development list for mkgmap Onderwerp: Re: [mkgmap-dev] r4033: new dem options are now documented Hi Minko, I don't think that splitter is the problem. The major change in r4026 was for handling of zipped hgt files. Maybe you have an unexpected zip archive ? Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von lig fietser <ligfietser@hotmail.com> Gesendet: Samstag, 6. Januar 2018 15:47:06 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] r4033: new dem options are now documented Hi Gerd, I use -Xmx4000m All tiles get stuck, doesnt matter which one. I have tried one of the Benelux, Germany, Canary Islands. I guess my splitter version is too old?? I have still r-321....(2014!). Strange because mkgmap 4025 performed well. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
BTW my DEM tiles for my Atlantic map are unzipped hgt's so that is not the case... ________________________________ Van: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> namens lig fietser <ligfietser@hotmail.com> Verzonden: zaterdag 6 januari 2018 06:54:09 Aan: Development list for mkgmap Onderwerp: Re: [mkgmap-dev] r4033: new dem options are now documented How can I check that? And why does it go wrong with every random tile? BTW My max-nodes are 1600000 ________________________________ Van: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> namens Gerd Petermann <gpetermann_muenchen@hotmail.com> Verzonden: zaterdag 6 januari 2018 06:52:32 Aan: Development list for mkgmap Onderwerp: Re: [mkgmap-dev] r4033: new dem options are now documented Hi Minko, I don't think that splitter is the problem. The major change in r4026 was for handling of zipped hgt files. Maybe you have an unexpected zip archive ? Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von lig fietser <ligfietser@hotmail.com> Gesendet: Samstag, 6. Januar 2018 15:47:06 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] r4033: new dem options are now documented Hi Gerd, I use -Xmx4000m All tiles get stuck, doesnt matter which one. I have tried one of the Benelux, Germany, Canary Islands. I guess my splitter version is too old?? I have still r-321....(2014!). Strange because mkgmap 4025 performed well. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Are you sure that r4026 was the first version that did not work for you? Do you use logging with -Dlog.config=... ? If yes, please try without. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von lig fietser <ligfietser@hotmail.com> Gesendet: Samstag, 6. Januar 2018 15:55:42 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] r4033: new dem options are now documented BTW my DEM tiles for my Atlantic map are unzipped hgt's so that is not the case... ________________________________ Van: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> namens lig fietser <ligfietser@hotmail.com> Verzonden: zaterdag 6 januari 2018 06:54:09 Aan: Development list for mkgmap Onderwerp: Re: [mkgmap-dev] r4033: new dem options are now documented How can I check that? And why does it go wrong with every random tile? BTW My max-nodes are 1600000 ________________________________ Van: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> namens Gerd Petermann <gpetermann_muenchen@hotmail.com> Verzonden: zaterdag 6 januari 2018 06:52:32 Aan: Development list for mkgmap Onderwerp: Re: [mkgmap-dev] r4033: new dem options are now documented Hi Minko, I don't think that splitter is the problem. The major change in r4026 was for handling of zipped hgt files. Maybe you have an unexpected zip archive ? Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von lig fietser <ligfietser@hotmail.com> Gesendet: Samstag, 6. Januar 2018 15:47:06 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] r4033: new dem options are now documented Hi Gerd, I use -Xmx4000m All tiles get stuck, doesnt matter which one. I have tried one of the Benelux, Germany, Canary Islands. I guess my splitter version is too old?? I have still r-321....(2014!). Strange because mkgmap 4025 performed well. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
I dont use logging Gerd. And I saw this behaviour in 4028 and 4033. Here are some of those o5m tiles, maybe they are too big too handle? http://mijndev.openstreetmap.nl/~ligfietser/mkgmap/ ________________________________ Van: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> namens Gerd Petermann <gpetermann_muenchen@hotmail.com> Verzonden: zaterdag 6 januari 2018 06:59 Aan: Development list for mkgmap Onderwerp: Re: [mkgmap-dev] r4033: new dem options are now documented Are you sure that r4026 was the first version that did not work for you? Do you use logging with -Dlog.config=... ? If yes, please try without. Gerd <http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev>
Hi Minko, can't reproduce the problem with your 10010001.o5m. Maybe you use a poly file with --dem-poly ? Please try to reproduce the problem with 10010001.o5m and a directory containing only N50E002.hgt (as this is the only file needed) and a small set of options (default style) If you can reproduce the problem please post the options and N50E002.hgt . Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von lig fietser <ligfietser@hotmail.com> Gesendet: Samstag, 6. Januar 2018 16:05:49 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] r4033: new dem options are now documented I dont use logging Gerd. And I saw this behaviour in 4028 and 4033. Here are some of those o5m tiles, maybe they are too big too handle? http://mijndev.openstreetmap.nl/~ligfietser/mkgmap/ ________________________________ Van: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> namens Gerd Petermann <gpetermann_muenchen@hotmail.com> Verzonden: zaterdag 6 januari 2018 06:59 Aan: Development list for mkgmap Onderwerp: Re: [mkgmap-dev] r4033: new dem options are now documented Are you sure that r4026 was the first version that did not work for you? Do you use logging with -Dlog.config=... ? If yes, please try without. Gerd <http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev>
Hi Gerd, I think you are right, it might be the zipped hgt. If I use it unzipped there seems no delay. I posted it here: http://mijndev.openstreetmap.nl/~ligfietser/mkgmap/ ________________________________ Van: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> namens Gerd Petermann <gpetermann_muenchen@hotmail.com> Verzonden: zaterdag 6 januari 2018 07:48:14 Aan: Development list for mkgmap Onderwerp: Re: [mkgmap-dev] r4033: new dem options are now documented Hi Minko, can't reproduce the problem with your 10010001.o5m. Maybe you use a poly file with --dem-poly ? Please try to reproduce the problem with 10010001.o5m and a directory containing only N50E002.hgt (as this is the only file needed) and a small set of options (default style) If you can reproduce the problem please post the options and N50E002.hgt . Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von lig fietser <ligfietser@hotmail.com> Gesendet: Samstag, 6. Januar 2018 16:05:49 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] r4033: new dem options are now documented I dont use logging Gerd. And I saw this behaviour in 4028 and 4033. Here are some of those o5m tiles, maybe they are too big too handle? http://mijndev.openstreetmap.nl/~ligfietser/mkgmap/ ________________________________ Van: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> namens Gerd Petermann <gpetermann_muenchen@hotmail.com> Verzonden: zaterdag 6 januari 2018 06:59 Aan: Development list for mkgmap Onderwerp: Re: [mkgmap-dev] r4033: new dem options are now documented Are you sure that r4026 was the first version that did not work for you? Do you use logging with -Dlog.config=... ? If yes, please try without. Gerd <http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev> _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Minko, yes, seems I've never tested this again since r4026 :-( Now I can reproduce the problem. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von lig fietser <ligfietser@hotmail.com> Gesendet: Samstag, 6. Januar 2018 17:37:36 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] r4033: new dem options are now documented Hi Gerd, I think you are right, it might be the zipped hgt. If I use it unzipped there seems no delay. I posted it here: http://mijndev.openstreetmap.nl/~ligfietser/mkgmap/ ________________________________ Van: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> namens Gerd Petermann <gpetermann_muenchen@hotmail.com> Verzonden: zaterdag 6 januari 2018 07:48:14 Aan: Development list for mkgmap Onderwerp: Re: [mkgmap-dev] r4033: new dem options are now documented Hi Minko, can't reproduce the problem with your 10010001.o5m. Maybe you use a poly file with --dem-poly ? Please try to reproduce the problem with 10010001.o5m and a directory containing only N50E002.hgt (as this is the only file needed) and a small set of options (default style) If you can reproduce the problem please post the options and N50E002.hgt . Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von lig fietser <ligfietser@hotmail.com> Gesendet: Samstag, 6. Januar 2018 16:05:49 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] r4033: new dem options are now documented I dont use logging Gerd. And I saw this behaviour in 4028 and 4033. Here are some of those o5m tiles, maybe they are too big too handle? http://mijndev.openstreetmap.nl/~ligfietser/mkgmap/ ________________________________ Van: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> namens Gerd Petermann <gpetermann_muenchen@hotmail.com> Verzonden: zaterdag 6 januari 2018 06:59 Aan: Development list for mkgmap Onderwerp: Re: [mkgmap-dev] r4033: new dem options are now documented Are you sure that r4026 was the first version that did not work for you? Do you use logging with -Dlog.config=... ? If yes, please try without. Gerd <http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev> _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Gerd, do you think using tif (as Frank pointed out even has better compression ratio) instead of zipped hgt will help regarding the RAM-footprint? Henning
Hi Henning, I don't think so. Did anybody try to store hgt files in a compressing file system (or a compressed folder)? Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Henning Scholland <osm@hscholland.de> Gesendet: Samstag, 6. Januar 2018 19:19:35 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] r4033: new dem options are now documented Hi Gerd, do you think using tif (as Frank pointed out even has better compression ratio) instead of zipped hgt will help regarding the RAM-footprint? Henning _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
participants (5)
-
Arndt Röhrig -
Gerd Petermann -
Gerd Petermann
-
Henning Scholland -
lig fietser