Hi all, I see no way to improve the handling of zipped files much more. We have two conflicting optimization goals here: 1) reduce heap memory by storing less data in buffers (risking additional rather slow unzip actions) 2) reduce run time be storing more data in buffers (risking OutOfMemoryException) The problem is that the DEM algo needs more or less random access to the data of one or more *.hgt.zip files, and I see no simple way to change that now. I did a few tests now with r4036 and a map for scandinavia and hgt files in 1'' resolution. The 380 unzipped hgt files require ~ 9.4 GB. When stored in a NTFS compressed folder the required disk space is ~ 37%, when stored in single zipped files (*.hgt.zip) the size is ~1.3 GB, means ~14%. I've used a simple poly file to cut scandinavia out of an older europe.o5m with osmconvert and used splitter with max-nodes=1000000. This gave me 174 tiles. I used the following command to create a map: java -Xmx6800m -jar d:\mkgmap\dist\mkgmap.jar -c d:\dbg\dev-addr.opt --output-dir=.\map --block-size=2048 --max-jobs=4 --overview-dem-dist=55000 --dem-poly=f:\osm\scandinavia.poly --dem=f:\srtm3_1_zip --dem-dists=3312,5846,8948,12646,20282 --gmapi -c e:\scand\tiles\template.args I repeated this command two times, once with the uncompressed files, once with the NTFS compressed files. All worked fine, java heap memory was no problem, but run time was different: 953 secs for the uncompressed files 1084 secs for the NTFS compressed files 1538 secs for the zipped files I think NTFS compressed folder is a good compromise if disk space matters. I hope the numbers are similar for compressing Linux file systems. I plan to implement a new dem-temp=[dir] option. If given, mkgmap could create temp files for the unzipped hgt data. Gerd
Hi Gerd,
2) reduce run time be storing more data in buffers
This is what I would chose. I think java x64 can use swap to increase heap size. And RAM memory is a relatively cheap upgrade.
I think NTFS compressed folder is a good compromise
I did it long time ago. Whole set of SRTM3 is about 38.3GB and takes 23.1G on HDD. It is even better for Viewfinder Panoramas files, where 44.3GB od HGT takes 13.8GB of drive space. Another question, does mkgmap support non-standard HGT size? With BuildDemFile I use HGT with raster 76x76 for overview maps. It is faster. -- Best regards, Andrzej
Hi Andrzej, thanks for the input. The current implemention with zip files fills a buffer for each hgt file when it is needed (when the first elevation data is needed). The buffer is freed if the DEM calculation for the tile is done or when mkgmap is sure that the buffer is no longer needed for the current tile. I did not try it but mkgmap should support 76x76 tiles, but I see no need to use that with mkgmap, at least not with unzipped hgt files. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Andrzej Popowski <popej@poczta.onet.pl> Gesendet: Montag, 8. Januar 2018 11:50:13 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] Performance with zipped hgt files Hi Gerd,
2) reduce run time be storing more data in buffers
This is what I would chose. I think java x64 can use swap to increase heap size. And RAM memory is a relatively cheap upgrade.
I think NTFS compressed folder is a good compromise
I did it long time ago. Whole set of SRTM3 is about 38.3GB and takes 23.1G on HDD. It is even better for Viewfinder Panoramas files, where 44.3GB od HGT takes 13.8GB of drive space. Another question, does mkgmap support non-standard HGT size? With BuildDemFile I use HGT with raster 76x76 for overview maps. It is faster. -- Best regards, Andrzej _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Gerd,
I did not try it but mkgmap should support 76x76 tiles, but I see no need to use that with mkgmap, at least not with unzipped hgt files.
Yes, but since I already have created small version, I can save several minutes of reading HGT files form drive, when creating a big overview map. The real use for non standard resolutions would be when dealing with data sources other than SRTM, like for example EU-DEM. -- Best regards, Andrzej
Hi Gerd, how about using a compressed temp-format for DEM information, which is more suitable for DEM-calculation? Or even store DEM-information in temp folder <output-dir>/DEM and just reuse them next time if possible. I think (at least for me) they don't change that often. So this will have a high impact and should be quite easy to implement. These DEM-precompiled data mkgmap could calculate in a separate run after user changed map areas. I guess whole DEM needs to be recalculated if areas change, am I right? That's why we can't store pre-calculated DEM-1°-Tiles. Henning On 08.01.2018 18:19, Gerd Petermann wrote:
Hi all,
I see no way to improve the handling of zipped files much more. We have two conflicting optimization goals here: 1) reduce heap memory by storing less data in buffers (risking additional rather slow unzip actions) 2) reduce run time be storing more data in buffers (risking OutOfMemoryException) The problem is that the DEM algo needs more or less random access to the data of one or more *.hgt.zip files, and I see no simple way to change that now.
I did a few tests now with r4036 and a map for scandinavia and hgt files in 1'' resolution. The 380 unzipped hgt files require ~ 9.4 GB. When stored in a NTFS compressed folder the required disk space is ~ 37%, when stored in single zipped files (*.hgt.zip) the size is ~1.3 GB, means ~14%.
I've used a simple poly file to cut scandinavia out of an older europe.o5m with osmconvert and used splitter with max-nodes=1000000. This gave me 174 tiles. I used the following command to create a map: java -Xmx6800m -jar d:\mkgmap\dist\mkgmap.jar -c d:\dbg\dev-addr.opt --output-dir=.\map --block-size=2048 --max-jobs=4 --overview-dem-dist=55000 --dem-poly=f:\osm\scandinavia.poly --dem=f:\srtm3_1_zip --dem-dists=3312,5846,8948,12646,20282 --gmapi -c e:\scand\tiles\template.args
I repeated this command two times, once with the uncompressed files, once with the NTFS compressed files. All worked fine, java heap memory was no problem, but run time was different: 953 secs for the uncompressed files 1084 secs for the NTFS compressed files 1538 secs for the zipped files
I think NTFS compressed folder is a good compromise if disk space matters. I hope the numbers are similar for compressing Linux file systems. I plan to implement a new dem-temp=[dir] option. If given, mkgmap could create temp files for the unzipped hgt data.
Gerd _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Henning, I also thought about that and I think it could work using a similar algo as for the DEM format in IMG files. We may have an index as with precomp sea data and rather small files compared to zip. On the other hand that format would not be used by any other application, so in the end you might need the data twice. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Henning Scholland <osm@hscholland.de> Gesendet: Montag, 8. Januar 2018 12:52:05 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] Performance with zipped hgt files Hi Gerd, how about using a compressed temp-format for DEM information, which is more suitable for DEM-calculation? Or even store DEM-information in temp folder <output-dir>/DEM and just reuse them next time if possible. I think (at least for me) they don't change that often. So this will have a high impact and should be quite easy to implement. These DEM-precompiled data mkgmap could calculate in a separate run after user changed map areas. I guess whole DEM needs to be recalculated if areas change, am I right? That's why we can't store pre-calculated DEM-1°-Tiles. Henning On 08.01.2018 18:19, Gerd Petermann wrote:
Hi all,
I see no way to improve the handling of zipped files much more. We have two conflicting optimization goals here: 1) reduce heap memory by storing less data in buffers (risking additional rather slow unzip actions) 2) reduce run time be storing more data in buffers (risking OutOfMemoryException) The problem is that the DEM algo needs more or less random access to the data of one or more *.hgt.zip files, and I see no simple way to change that now.
I did a few tests now with r4036 and a map for scandinavia and hgt files in 1'' resolution. The 380 unzipped hgt files require ~ 9.4 GB. When stored in a NTFS compressed folder the required disk space is ~ 37%, when stored in single zipped files (*.hgt.zip) the size is ~1.3 GB, means ~14%.
I've used a simple poly file to cut scandinavia out of an older europe.o5m with osmconvert and used splitter with max-nodes=1000000. This gave me 174 tiles. I used the following command to create a map: java -Xmx6800m -jar d:\mkgmap\dist\mkgmap.jar -c d:\dbg\dev-addr.opt --output-dir=.\map --block-size=2048 --max-jobs=4 --overview-dem-dist=55000 --dem-poly=f:\osm\scandinavia.poly --dem=f:\srtm3_1_zip --dem-dists=3312,5846,8948,12646,20282 --gmapi -c e:\scand\tiles\template.args
I repeated this command two times, once with the uncompressed files, once with the NTFS compressed files. All worked fine, java heap memory was no problem, but run time was different: 953 secs for the uncompressed files 1084 secs for the NTFS compressed files 1538 secs for the zipped files
I think NTFS compressed folder is a good compromise if disk space matters. I hope the numbers are similar for compressing Linux file systems. I plan to implement a new dem-temp=[dir] option. If given, mkgmap could create temp files for the unzipped hgt data.
Gerd _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Gerd, I was thinking of writing the 'final' DEM not only in img-container but also in a temp-file. This temp file can be used for all updates of map data without changes in areas.list. More or less the work flow can be: splitter calculate areas.list, mkgmap can calculate temporary DEM for areas.list. As long as I use the same areas.list for the map, I can reuse the finished DEM. If I need to change areas.list of course I need to let mkgmap calculate a new DEM. Overview-DEM will only change, if total map area is changing. So probably only needs to be calculated once per map. If we work like that at least for me compilation time wouldn't be that important. Basically the idea is to keep the static map-data and only update the dynamic part of the data. The other advantage would be, that we can use full amount of RAM during DEM-calculation, as there is no RAM used for map- data and so on... I don't know if it's really possible to store the 'final' DEM but at least some pre-compiled file should be possible I assume. I'm not sure, how img-files are written. Henning On 08.01.2018 20:42, Gerd Petermann wrote:
Hi Henning, I also thought about that and I think it could work using a similar algo as for the DEM format in IMG files. We may have an index as with precomp sea data and rather small files compared to zip. On the other hand that format would not be used by any other application, so in the end you might need the data twice.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Henning Scholland <osm@hscholland.de> Gesendet: Montag, 8. Januar 2018 12:52:05 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] Performance with zipped hgt files
Hi Gerd, how about using a compressed temp-format for DEM information, which is more suitable for DEM-calculation? Or even store DEM-information in temp folder <output-dir>/DEM and just reuse them next time if possible. I think (at least for me) they don't change that often. So this will have a high impact and should be quite easy to implement.
These DEM-precompiled data mkgmap could calculate in a separate run after user changed map areas.
I guess whole DEM needs to be recalculated if areas change, am I right? That's why we can't store pre-calculated DEM-1°-Tiles.
Henning
On 08.01.2018 18:19, Gerd Petermann wrote:
Hi all,
I see no way to improve the handling of zipped files much more. We have two conflicting optimization goals here: 1) reduce heap memory by storing less data in buffers (risking additional rather slow unzip actions) 2) reduce run time be storing more data in buffers (risking OutOfMemoryException) The problem is that the DEM algo needs more or less random access to the data of one or more *.hgt.zip files, and I see no simple way to change that now.
I did a few tests now with r4036 and a map for scandinavia and hgt files in 1'' resolution. The 380 unzipped hgt files require ~ 9.4 GB. When stored in a NTFS compressed folder the required disk space is ~ 37%, when stored in single zipped files (*.hgt.zip) the size is ~1.3 GB, means ~14%.
I've used a simple poly file to cut scandinavia out of an older europe.o5m with osmconvert and used splitter with max-nodes=1000000. This gave me 174 tiles. I used the following command to create a map: java -Xmx6800m -jar d:\mkgmap\dist\mkgmap.jar -c d:\dbg\dev-addr.opt --output-dir=.\map --block-size=2048 --max-jobs=4 --overview-dem-dist=55000 --dem-poly=f:\osm\scandinavia.poly --dem=f:\srtm3_1_zip --dem-dists=3312,5846,8948,12646,20282 --gmapi -c e:\scand\tiles\template.args
I repeated this command two times, once with the uncompressed files, once with the NTFS compressed files. All worked fine, java heap memory was no problem, but run time was different: 953 secs for the uncompressed files 1084 secs for the NTFS compressed files 1538 secs for the zipped files
I think NTFS compressed folder is a good compromise if disk space matters. I hope the numbers are similar for compressing Linux file systems. I plan to implement a new dem-temp=[dir] option. If given, mkgmap could create temp files for the unzipped hgt data.
Gerd _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Henning, I think that works already. I know for sure that there is no link between the other files (RGN,NOD etc) and DEM data, only the *.tdb contains a hint that the sub file exists. You can replace a *.DEM file by another with different resolutions , at least if they cover the same boundaries. So, if you use the gmap folder (--gmapi) you can already do what you want with a script. Example: Calculate the DEM with the wanted resolution, store the *.DEM files. Next time use options which calculate an almost DEM file (one level with very high dem-dist) and finally replace the *.DEM files with the stored ones. You probably need another tool to combine those files into a gmapsupp.img. So, possible workflow with r4037 looks like this: 1) run splitter with real OSM file to calculate areas.list 2) run splitter with this areas.list and an dummy osm file containing no data, this will produce osm.pbf files with the wanted boundaries 3) run mkgmap for those empty osm files with the wanted DEM options to create the good DEM files 4) run mkgmap for the files created in 1) with fake DEM options , e.g. --overview-dem-dist=276000 --dem-dists=50000 --dem=<empty folder> 5) replace *.DEM files created in 4) by *.DEM created in 3) 6) use a tool to combine the files into *.img Next time when you update the osm data you split with the areas.list created in step 1) and repeat steps 3-6. I've just tried that with a map for nepal and it worked fine (steps 1-5) , at least MapSource is happy with the resulting gmap folder. Don't know what tool and options you need to combine that folder back into img format. I've attached the dummy.osm file used in 2) Please let me know if that works for you. The only problem that I see is in combination with option --dem-poly. If you want to reuse tiles for different maps with different polygons (e.g. calculate DEM for Europe once and create maps for singles countries using specific *.poly files) you need a new method to filter the preprocessed DEM data. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Henning Scholland <osm@hscholland.de> Gesendet: Montag, 8. Januar 2018 14:16:31 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] Performance with zipped hgt files Hi Gerd, I was thinking of writing the 'final' DEM not only in img-container but also in a temp-file. This temp file can be used for all updates of map data without changes in areas.list. More or less the work flow can be: splitter calculate areas.list, mkgmap can calculate temporary DEM for areas.list. As long as I use the same areas.list for the map, I can reuse the finished DEM. If I need to change areas.list of course I need to let mkgmap calculate a new DEM. Overview-DEM will only change, if total map area is changing. So probably only needs to be calculated once per map. If we work like that at least for me compilation time wouldn't be that important. Basically the idea is to keep the static map-data and only update the dynamic part of the data. The other advantage would be, that we can use full amount of RAM during DEM-calculation, as there is no RAM used for map- data and so on... I don't know if it's really possible to store the 'final' DEM but at least some pre-compiled file should be possible I assume. I'm not sure, how img-files are written. Henning On 08.01.2018 20:42, Gerd Petermann wrote:
Hi Henning, I also thought about that and I think it could work using a similar algo as for the DEM format in IMG files. We may have an index as with precomp sea data and rather small files compared to zip. On the other hand that format would not be used by any other application, so in the end you might need the data twice.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Henning Scholland <osm@hscholland.de> Gesendet: Montag, 8. Januar 2018 12:52:05 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] Performance with zipped hgt files
Hi Gerd, how about using a compressed temp-format for DEM information, which is more suitable for DEM-calculation? Or even store DEM-information in temp folder <output-dir>/DEM and just reuse them next time if possible. I think (at least for me) they don't change that often. So this will have a high impact and should be quite easy to implement.
These DEM-precompiled data mkgmap could calculate in a separate run after user changed map areas.
I guess whole DEM needs to be recalculated if areas change, am I right? That's why we can't store pre-calculated DEM-1°-Tiles.
Henning
On 08.01.2018 18:19, Gerd Petermann wrote:
Hi all,
I see no way to improve the handling of zipped files much more. We have two conflicting optimization goals here: 1) reduce heap memory by storing less data in buffers (risking additional rather slow unzip actions) 2) reduce run time be storing more data in buffers (risking OutOfMemoryException) The problem is that the DEM algo needs more or less random access to the data of one or more *.hgt.zip files, and I see no simple way to change that now.
I did a few tests now with r4036 and a map for scandinavia and hgt files in 1'' resolution. The 380 unzipped hgt files require ~ 9.4 GB. When stored in a NTFS compressed folder the required disk space is ~ 37%, when stored in single zipped files (*.hgt.zip) the size is ~1.3 GB, means ~14%.
I've used a simple poly file to cut scandinavia out of an older europe.o5m with osmconvert and used splitter with max-nodes=1000000. This gave me 174 tiles. I used the following command to create a map: java -Xmx6800m -jar d:\mkgmap\dist\mkgmap.jar -c d:\dbg\dev-addr.opt --output-dir=.\map --block-size=2048 --max-jobs=4 --overview-dem-dist=55000 --dem-poly=f:\osm\scandinavia.poly --dem=f:\srtm3_1_zip --dem-dists=3312,5846,8948,12646,20282 --gmapi -c e:\scand\tiles\template.args
I repeated this command two times, once with the uncompressed files, once with the NTFS compressed files. All worked fine, java heap memory was no problem, but run time was different: 953 secs for the uncompressed files 1084 secs for the NTFS compressed files 1538 secs for the zipped files
I think NTFS compressed folder is a good compromise if disk space matters. I hope the numbers are similar for compressing Linux file systems. I plan to implement a new dem-temp=[dir] option. If given, mkgmap could create temp files for the unzipped hgt data.
Gerd _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Gerd, basically this describes my idea, just less quick&dirty ;-) I will give it a try next weekend. Of course for every map, you need a individual temp-DEM. So I don't seen any issues with poly-cutted-tiles. Henning
participants (4)
-
Andrzej Popowski -
Gerd Petermann -
Gerd Petermann
-
Henning Scholland