mkgmap doing excessive writing and converting to disk if used with --gmapi option
Quite a few people do not compile the contourlines each time they update a map, but reuse the old contourlines. This works fine if you insert them as say 1234*img (and they are named 12340000.img 12340001.img and so on). However this does not work when providing the --gmapi option. Actually mkgmap tries to rewrite all those files again/convert them. Could it please be possible for mkgmap to change this? There could be the status quo with --gmapi and there could be a new mode called --gmapiminimal that only converts files input as .o5m or .osm.pbf or .osm but not as .img --gpapiminimal would then work the same way as it does with .img files. This has two use cases. a) if you have to resplit a tile because some tiles were breaking - and directly gave the --gmapiminimal option - no need to redo the whole .gmap file. b) you symlink all gmap folders with the contourlines (or other layers you do not recreate on each update) into the gmap folder and mkgmap only writes out the new stuff. Yes on an nvme disk you barely notice the conversion - it's really quick. BUT it is not needed if you have the files and even more - it burns your NVME SSD disk. To my horror I noticed that one worldwide update of maps wrote 5TB of data to my disk (thanks to NVME SSD and fast processor this all took 28 hours). At that rate even datacenter NVME ssd disks are quickly turned into trash., consumer grade NVME disk will not make it half a year for me... In general maybe mkgmap could have an option to write temporary files to another disk? Then you use a RAMDisk for those files. Or mkgmap should keep them in memory. Modern servers often have 64 or 128GB of RAM so enough RAM for mkgmap not to waste writes to disk... Yes mkgmap uses quite a lot of RAM, but RAM is much cheaper than continous SSD writes.... Write now mkgmap crashes if you symlink already converted files to it: e.g. message: Number of MapFailedExceptions: 0 SEVERE (global): Error saving gmapi data java.nio.file.FileAlreadyExistsException: .\mtb_alp_08.09.2021.gmap\Product1\75280000 While if it would be allowed to write out the data it would overwrite existing files (which is fine or should do so - but same way as for .img files) For my worldwide maps 10m contourlines are 200GB, 20m contourlines are 100GB in size. Rendering the map in two styles and each time needing to convert them even though I already have them converted creates 600GB of useless writes per update (as well as useless time spend converting those 300GB at least twice) - and that is only if I use the --gmapi option as a separate step. If I give bot --tdb-file and --gmapi it gets worse if a tile failed to write - and I just parse it again. And that is only for the contourlines. As I need three sets of mapset files - once without contourlines, once with 10m contourlines, and once with 20m contourlines the actual map gmapi folders are written three times instead of once... Well over 1TB of useless conversion time and useless data written to disk (but I do not know another way to create the needed mapset_mdr.img, mapset.img and mapset.tdb files) I don't know how to change that myself, but I hope this is not much work to change. -- Felix Hartman - Openmtbmap.org & VeloMap.org
On Wed, Sep 08, 2021 at 02:10:52PM +0300, Felix Hartmann wrote:
Yes on an nvme disk you barely notice the conversion - it's really quick. BUT it is not needed if you have the files and even more - it burns your NVME SSD disk.
+1. I always use an old spinning rust disk when using mkgmap to save ssd write cycles, even without contours and such. It seems to be profligate in its use of disk cycles. I did try using RAM disk, but even with 16GB on a laptop, that was soon exhausted. ael
Well I could give it 20 GB ram disk, maybe 32 but then I need to render on less than 12 processes 64GB ram available). But that is not enough for Asia continent map, and I guess super tight for Europe... Mkgmap could definitely keep those .tmp files in memory. But the important bit is the gmap files not needeed to be written.... Also would save quite some CPU time. On Wed, 8 Sep 2021, 14:31 ael <witwall3@disroot.org> wrote:
On Wed, Sep 08, 2021 at 02:10:52PM +0300, Felix Hartmann wrote:
Yes on an nvme disk you barely notice the conversion - it's really quick. BUT it is not needed if you have the files and even more - it burns your NVME SSD disk.
+1. I always use an old spinning rust disk when using mkgmap to save ssd write cycles, even without contours and such. It seems to be profligate in its use of disk cycles. I did try using RAM disk, but even with 16GB on a laptop, that was soon exhausted.
ael
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Well I think the .tmp files are just building up - and the renamed. So they are not causing any actual excessive write. As for the gmap - it would be cool if there is a mode to not write them. Actually it would be great if mkgmap could write all in one go. Because the thing that takes so much time - is the address search - and that is always the same. The differences are tiny (just because MapInstall is crashing when files are missing) you need to compile them separately. but maybe there could be a mode where mkgamp writes all in one go. So family-name / family-name1 / family-name2 description / description1 / description2 input input1 input2 family-name.. show-profiles overview-mapname product-id (and maybe I missed some options are those that would need to be given for each set of input tiles. And then just an option where you tell mkgmap files starting with which first 4 numbers are relevant for address search. No need to analyze if those other supplied .img (e.g. buildings or contourlines) need to be added to address search.). I know coded around the problem of the gmap files causing excessive writes. But yeah that is actually really complicated be it on windows or linux... On Wed, 8 Sept 2021 at 17:07, Felix Hartmann <extremecarver@gmail.com> wrote:
Well I could give it 20 GB ram disk, maybe 32 but then I need to render on less than 12 processes 64GB ram available). But that is not enough for Asia continent map, and I guess super tight for Europe...
Mkgmap could definitely keep those .tmp files in memory. But the important bit is the gmap files not needeed to be written.... Also would save quite some CPU time.
On Wed, 8 Sep 2021, 14:31 ael <witwall3@disroot.org> wrote:
On Wed, Sep 08, 2021 at 02:10:52PM +0300, Felix Hartmann wrote:
Yes on an nvme disk you barely notice the conversion - it's really
quick.
BUT it is not needed if you have the files and even more - it burns your NVME SSD disk.
+1. I always use an old spinning rust disk when using mkgmap to save ssd write cycles, even without contours and such. It seems to be profligate in its use of disk cycles. I did try using RAM disk, but even with 16GB on a laptop, that was soon exhausted.
ael
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- Felix Hartman - Openmtbmap.org & VeloMap.org
Hi Felix, did not read all the posts in detail. I understand that mkgmap is neither burning SSDs nor doing any excessive writing unless you call it multiple times for the same input files. So the question is: Why do you do that? Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Felix Hartmann <extremecarver@gmail.com> Gesendet: Freitag, 10. September 2021 00:02 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option Well I think the .tmp files are just building up - and the renamed. So they are not causing any actual excessive write. As for the gmap - it would be cool if there is a mode to not write them. Actually it would be great if mkgmap could write all in one go. Because the thing that takes so much time - is the address search - and that is always the same. The differences are tiny (just because MapInstall is crashing when files are missing) you need to compile them separately. but maybe there could be a mode where mkgamp writes all in one go. So family-name / family-name1 / family-name2 description / description1 / description2 input input1 input2 family-name.. show-profiles overview-mapname product-id (and maybe I missed some options are those that would need to be given for each set of input tiles. And then just an option where you tell mkgmap files starting with which first 4 numbers are relevant for address search. No need to analyze if those other supplied .img (e.g. buildings or contourlines) need to be added to address search.). I know coded around the problem of the gmap files causing excessive writes. But yeah that is actually really complicated be it on windows or linux... On Wed, 8 Sept 2021 at 17:07, Felix Hartmann <extremecarver@gmail.com<mailto:extremecarver@gmail.com>> wrote: Well I could give it 20 GB ram disk, maybe 32 but then I need to render on less than 12 processes 64GB ram available). But that is not enough for Asia continent map, and I guess super tight for Europe... Mkgmap could definitely keep those .tmp files in memory. But the important bit is the gmap files not needeed to be written.... Also would save quite some CPU time. On Wed, 8 Sep 2021, 14:31 ael <witwall3@disroot.org<mailto:witwall3@disroot.org>> wrote: On Wed, Sep 08, 2021 at 02:10:52PM +0300, Felix Hartmann wrote:
Yes on an nvme disk you barely notice the conversion - it's really quick. BUT it is not needed if you have the files and even more - it burns your NVME SSD disk.
+1. I always use an old spinning rust disk when using mkgmap to save ssd write cycles, even without contours and such. It seems to be profligate in its use of disk cycles. I did try using RAM disk, but even with 16GB on a laptop, that was soon exhausted. ael _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev -- Felix Hartman - Openmtbmap.org & VeloMap.org
because you need to do it if you want to show contourlines. Now compiling the worldwide contourlines each time again - would burn SSD as well - besides taking longer than just compiling the maps. So everyone who offers maps for many countries as download puts the contourlines separate. If you want to offer the choice of showing contourlines or not - mkmgap will write the gmap contourlines once not needed, and once the maps not needed. If you don't want to offer that option - it's only the contourlines being written without need. Contourlines for all geofabrik extracts 20m equidistance are about 130GB, 10m would be 260GB. If you offer 10m and 20m it will be even more not needed writes. The only solution is to go for a Ramdisk instead - However you likely will need 128GB of RAM to do that - because for Asia or Europe you need a 64GB Ramdisk. Same for North America extract (but few people offer that and just have Canada and the US divided into the 4-5 zones). For other maps you will get by with a 15-20GB Ramdisk (which I have resorted to now for all but the windows installers because I don't want to let the server wait for ages for the single thread NSIS wrapping up the data and instead start the next country in parallel). And yes going RAMDISK already using my patch and symlinking those files so mkgmap cannot overwrite them (would still be faster if mkgmap didn't try in first place I think). If you want to write out the contourlines as well besides the additional time - for Asia continent you may then go for a 90GB Ramdisk minimum if offering windows and gmap installers. In this case gmapsupp.img donwnloads aren't possible anyhow due to the huge data amounts. For smaller countries I have them too.... I coded around this now by using symlinks - but yeah that will be quite a lot of work and is prone to break - while I guess it's only 10 lines or so to add to mkgmap code to have a mode that does not write them out if they are present - or if you tell on commandline they are present already. For .img files they aren't overwritten either... On Sun, 12 Sept 2021 at 18:40, Gerd Petermann < gpetermann_muenchen@hotmail.com> wrote:
Hi Felix,
did not read all the posts in detail. I understand that mkgmap is neither burning SSDs nor doing any excessive writing unless you call it multiple times for the same input files. So the question is: Why do you do that?
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Felix Hartmann <extremecarver@gmail.com> Gesendet: Freitag, 10. September 2021 00:02 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option
Well I think the .tmp files are just building up - and the renamed. So they are not causing any actual excessive write. As for the gmap - it would be cool if there is a mode to not write them.
Actually it would be great if mkgmap could write all in one go. Because the thing that takes so much time - is the address search - and that is always the same. The differences are tiny (just because MapInstall is crashing when files are missing) you need to compile them separately.
but maybe there could be a mode where mkgamp writes all in one go. So family-name / family-name1 / family-name2 description / description1 / description2 input input1 input2 family-name.. show-profiles overview-mapname product-id (and maybe I missed some options are those that would need to be given for each set of input tiles. And then just an option where you tell mkgmap files starting with which first 4 numbers are relevant for address search. No need to analyze if those other supplied .img (e.g. buildings or contourlines) need to be added to address search.). I know coded around the problem of the gmap files causing excessive writes. But yeah that is actually really complicated be it on windows or linux...
On Wed, 8 Sept 2021 at 17:07, Felix Hartmann <extremecarver@gmail.com <mailto:extremecarver@gmail.com>> wrote: Well I could give it 20 GB ram disk, maybe 32 but then I need to render on less than 12 processes 64GB ram available). But that is not enough for Asia continent map, and I guess super tight for Europe...
Mkgmap could definitely keep those .tmp files in memory. But the important bit is the gmap files not needeed to be written.... Also would save quite some CPU time.
On Wed, 8 Sep 2021, 14:31 ael <witwall3@disroot.org<mailto: witwall3@disroot.org>> wrote: On Wed, Sep 08, 2021 at 02:10:52PM +0300, Felix Hartmann wrote:
Yes on an nvme disk you barely notice the conversion - it's really quick. BUT it is not needed if you have the files and even more - it burns your NVME SSD disk.
+1. I always use an old spinning rust disk when using mkgmap to save ssd write cycles, even without contours and such. It seems to be profligate in its use of disk cycles. I did try using RAM disk, but even with 16GB on a laptop, that was soon exhausted.
ael
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- Felix Hartman - Openmtbmap.org & VeloMap.org
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- Felix Hartman - Openmtbmap.org & VeloMap.org
And well - it is burning SSD for the contourlines currently even if not calling multiple times. 130GB minimum per worldwide map update for all geofabrik extracts at 20m equidistance. 260GB at 10m. With calling multiple times and 10m and 20m I had about 2TB of useless writes per weekly map update. I've got rid of all of them with my uncommenting the line, plus saved about 500GB of writes by now doing all the gmapsupp.img and gmap stuff in Ramdisk So now instead of 4TB of disk writes per map update I'm down to 1TB.. (and yeah the resplitting of tiles added another 5TB of writes - but that could have been fixed easily by changing order of commands too). On Sun, 12 Sept 2021 at 20:04, Felix Hartmann <extremecarver@gmail.com> wrote:
because you need to do it if you want to show contourlines. Now compiling the worldwide contourlines each time again - would burn SSD as well - besides taking longer than just compiling the maps. So everyone who offers maps for many countries as download puts the contourlines separate. If you want to offer the choice of showing contourlines or not - mkmgap will write the gmap contourlines once not needed, and once the maps not needed. If you don't want to offer that option - it's only the contourlines being written without need. Contourlines for all geofabrik extracts 20m equidistance are about 130GB, 10m would be 260GB. If you offer 10m and 20m it will be even more not needed writes.
The only solution is to go for a Ramdisk instead - However you likely will need 128GB of RAM to do that - because for Asia or Europe you need a 64GB Ramdisk. Same for North America extract (but few people offer that and just have Canada and the US divided into the 4-5 zones). For other maps you will get by with a 15-20GB Ramdisk (which I have resorted to now for all but the windows installers because I don't want to let the server wait for ages for the single thread NSIS wrapping up the data and instead start the next country in parallel). And yes going RAMDISK already using my patch and symlinking those files so mkgmap cannot overwrite them (would still be faster if mkgmap didn't try in first place I think). If you want to write out the contourlines as well besides the additional time - for Asia continent you may then go for a 90GB Ramdisk minimum if offering windows and gmap installers. In this case gmapsupp.img donwnloads aren't possible anyhow due to the huge data amounts. For smaller countries I have them too....
I coded around this now by using symlinks - but yeah that will be quite a lot of work and is prone to break - while I guess it's only 10 lines or so to add to mkgmap code to have a mode that does not write them out if they are present - or if you tell on commandline they are present already. For .img files they aren't overwritten either...
On Sun, 12 Sept 2021 at 18:40, Gerd Petermann < gpetermann_muenchen@hotmail.com> wrote:
Hi Felix,
did not read all the posts in detail. I understand that mkgmap is neither burning SSDs nor doing any excessive writing unless you call it multiple times for the same input files. So the question is: Why do you do that?
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Felix Hartmann <extremecarver@gmail.com> Gesendet: Freitag, 10. September 2021 00:02 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option
Well I think the .tmp files are just building up - and the renamed. So they are not causing any actual excessive write. As for the gmap - it would be cool if there is a mode to not write them.
Actually it would be great if mkgmap could write all in one go. Because the thing that takes so much time - is the address search - and that is always the same. The differences are tiny (just because MapInstall is crashing when files are missing) you need to compile them separately.
but maybe there could be a mode where mkgamp writes all in one go. So family-name / family-name1 / family-name2 description / description1 / description2 input input1 input2 family-name.. show-profiles overview-mapname product-id (and maybe I missed some options are those that would need to be given for each set of input tiles. And then just an option where you tell mkgmap files starting with which first 4 numbers are relevant for address search. No need to analyze if those other supplied .img (e.g. buildings or contourlines) need to be added to address search.). I know coded around the problem of the gmap files causing excessive writes. But yeah that is actually really complicated be it on windows or linux...
On Wed, 8 Sept 2021 at 17:07, Felix Hartmann <extremecarver@gmail.com <mailto:extremecarver@gmail.com>> wrote: Well I could give it 20 GB ram disk, maybe 32 but then I need to render on less than 12 processes 64GB ram available). But that is not enough for Asia continent map, and I guess super tight for Europe...
Mkgmap could definitely keep those .tmp files in memory. But the important bit is the gmap files not needeed to be written.... Also would save quite some CPU time.
On Wed, 8 Sep 2021, 14:31 ael <witwall3@disroot.org<mailto: witwall3@disroot.org>> wrote: On Wed, Sep 08, 2021 at 02:10:52PM +0300, Felix Hartmann wrote:
Yes on an nvme disk you barely notice the conversion - it's really
quick.
BUT it is not needed if you have the files and even more - it burns your NVME SSD disk.
+1. I always use an old spinning rust disk when using mkgmap to save ssd write cycles, even without contours and such. It seems to be profligate in its use of disk cycles. I did try using RAM disk, but even with 16GB on a laptop, that was soon exhausted.
ael
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- Felix Hartman - Openmtbmap.org & VeloMap.org
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- Felix Hartman - Openmtbmap.org & VeloMap.org
-- Felix Hartman - Openmtbmap.org & VeloMap.org
Hi Felix, so you just want a new option so that mkgmap doesn't fail if it cannot overwrite (write-protected) files in the output directory, right? No need to verify the content of the exiting file(s)? Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Felix Hartmann <extremecarver@gmail.com> Gesendet: Sonntag, 12. September 2021 19:10 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option And well - it is burning SSD for the contourlines currently even if not calling multiple times. 130GB minimum per worldwide map update for all geofabrik extracts at 20m equidistance. 260GB at 10m. With calling multiple times and 10m and 20m I had about 2TB of useless writes per weekly map update. I've got rid of all of them with my uncommenting the line, plus saved about 500GB of writes by now doing all the gmapsupp.img and gmap stuff in Ramdisk So now instead of 4TB of disk writes per map update I'm down to 1TB.. (and yeah the resplitting of tiles added another 5TB of writes - but that could have been fixed easily by changing order of commands too). On Sun, 12 Sept 2021 at 20:04, Felix Hartmann <extremecarver@gmail.com<mailto:extremecarver@gmail.com>> wrote: because you need to do it if you want to show contourlines. Now compiling the worldwide contourlines each time again - would burn SSD as well - besides taking longer than just compiling the maps. So everyone who offers maps for many countries as download puts the contourlines separate. If you want to offer the choice of showing contourlines or not - mkmgap will write the gmap contourlines once not needed, and once the maps not needed. If you don't want to offer that option - it's only the contourlines being written without need. Contourlines for all geofabrik extracts 20m equidistance are about 130GB, 10m would be 260GB. If you offer 10m and 20m it will be even more not needed writes. The only solution is to go for a Ramdisk instead - However you likely will need 128GB of RAM to do that - because for Asia or Europe you need a 64GB Ramdisk. Same for North America extract (but few people offer that and just have Canada and the US divided into the 4-5 zones). For other maps you will get by with a 15-20GB Ramdisk (which I have resorted to now for all but the windows installers because I don't want to let the server wait for ages for the single thread NSIS wrapping up the data and instead start the next country in parallel). And yes going RAMDISK already using my patch and symlinking those files so mkgmap cannot overwrite them (would still be faster if mkgmap didn't try in first place I think). If you want to write out the contourlines as well besides the additional time - for Asia continent you may then go for a 90GB Ramdisk minimum if offering windows and gmap installers. In this case gmapsupp.img donwnloads aren't possible anyhow due to the huge data amounts. For smaller countries I have them too.... I coded around this now by using symlinks - but yeah that will be quite a lot of work and is prone to break - while I guess it's only 10 lines or so to add to mkgmap code to have a mode that does not write them out if they are present - or if you tell on commandline they are present already. For .img files they aren't overwritten either... On Sun, 12 Sept 2021 at 18:40, Gerd Petermann <gpetermann_muenchen@hotmail.com<mailto:gpetermann_muenchen@hotmail.com>> wrote: Hi Felix, did not read all the posts in detail. I understand that mkgmap is neither burning SSDs nor doing any excessive writing unless you call it multiple times for the same input files. So the question is: Why do you do that? Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk<mailto:mkgmap-dev-bounces@lists.mkgmap.org.uk>> im Auftrag von Felix Hartmann <extremecarver@gmail.com<mailto:extremecarver@gmail.com>> Gesendet: Freitag, 10. September 2021 00:02 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option Well I think the .tmp files are just building up - and the renamed. So they are not causing any actual excessive write. As for the gmap - it would be cool if there is a mode to not write them. Actually it would be great if mkgmap could write all in one go. Because the thing that takes so much time - is the address search - and that is always the same. The differences are tiny (just because MapInstall is crashing when files are missing) you need to compile them separately. but maybe there could be a mode where mkgamp writes all in one go. So family-name / family-name1 / family-name2 description / description1 / description2 input input1 input2 family-name.. show-profiles overview-mapname product-id (and maybe I missed some options are those that would need to be given for each set of input tiles. And then just an option where you tell mkgmap files starting with which first 4 numbers are relevant for address search. No need to analyze if those other supplied .img (e.g. buildings or contourlines) need to be added to address search.). I know coded around the problem of the gmap files causing excessive writes. But yeah that is actually really complicated be it on windows or linux... On Wed, 8 Sept 2021 at 17:07, Felix Hartmann <extremecarver@gmail.com<mailto:extremecarver@gmail.com><mailto:extremecarver@gmail.com<mailto:extremecarver@gmail.com>>> wrote: Well I could give it 20 GB ram disk, maybe 32 but then I need to render on less than 12 processes 64GB ram available). But that is not enough for Asia continent map, and I guess super tight for Europe... Mkgmap could definitely keep those .tmp files in memory. But the important bit is the gmap files not needeed to be written.... Also would save quite some CPU time. On Wed, 8 Sep 2021, 14:31 ael <witwall3@disroot.org<mailto:witwall3@disroot.org><mailto:witwall3@disroot.org<mailto:witwall3@disroot.org>>> wrote: On Wed, Sep 08, 2021 at 02:10:52PM +0300, Felix Hartmann wrote:
Yes on an nvme disk you barely notice the conversion - it's really quick. BUT it is not needed if you have the files and even more - it burns your NVME SSD disk.
+1. I always use an old spinning rust disk when using mkgmap to save ssd write cycles, even without contours and such. It seems to be profligate in its use of disk cycles. I did try using RAM disk, but even with 16GB on a laptop, that was soon exhausted. ael _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk><mailto:mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk>> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev -- Felix Hartman - Openmtbmap.org & VeloMap.org _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev -- Felix Hartman - Openmtbmap.org & VeloMap.org -- Felix Hartman - Openmtbmap.org & VeloMap.org
I think it's a good idea if mkgmap checks the required files are present. El 12/9/21 a las 21:16, Gerd Petermann escribió:
Hi Felix,
so you just want a new option so that mkgmap doesn't fail if it cannot overwrite (write-protected) files in the output directory, right? No need to verify the content of the exiting file(s)?
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Felix Hartmann <extremecarver@gmail.com> Gesendet: Sonntag, 12. September 2021 19:10 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option
And well - it is burning SSD for the contourlines currently even if not calling multiple times. 130GB minimum per worldwide map update for all geofabrik extracts at 20m equidistance. 260GB at 10m. With calling multiple times and 10m and 20m I had about 2TB of useless writes per weekly map update. I've got rid of all of them with my uncommenting the line, plus saved about 500GB of writes by now doing all the gmapsupp.img and gmap stuff in Ramdisk So now instead of 4TB of disk writes per map update I'm down to 1TB.. (and yeah the resplitting of tiles added another 5TB of writes - but that could have been fixed easily by changing order of commands too).
On Sun, 12 Sept 2021 at 20:04, Felix Hartmann <extremecarver@gmail.com<mailto:extremecarver@gmail.com>> wrote: because you need to do it if you want to show contourlines. Now compiling the worldwide contourlines each time again - would burn SSD as well - besides taking longer than just compiling the maps. So everyone who offers maps for many countries as download puts the contourlines separate. If you want to offer the choice of showing contourlines or not - mkmgap will write the gmap contourlines once not needed, and once the maps not needed. If you don't want to offer that option - it's only the contourlines being written without need. Contourlines for all geofabrik extracts 20m equidistance are about 130GB, 10m would be 260GB. If you offer 10m and 20m it will be even more not needed writes.
The only solution is to go for a Ramdisk instead - However you likely will need 128GB of RAM to do that - because for Asia or Europe you need a 64GB Ramdisk. Same for North America extract (but few people offer that and just have Canada and the US divided into the 4-5 zones). For other maps you will get by with a 15-20GB Ramdisk (which I have resorted to now for all but the windows installers because I don't want to let the server wait for ages for the single thread NSIS wrapping up the data and instead start the next country in parallel). And yes going RAMDISK already using my patch and symlinking those files so mkgmap cannot overwrite them (would still be faster if mkgmap didn't try in first place I think). If you want to write out the contourlines as well besides the additional time - for Asia continent you may then go for a 90GB Ramdisk minimum if offering windows and gmap installers. In this case gmapsupp.img donwnloads aren't possible anyhow due to the huge data amounts. For sm aller countries I have them too....
I coded around this now by using symlinks - but yeah that will be quite a lot of work and is prone to break - while I guess it's only 10 lines or so to add to mkgmap code to have a mode that does not write them out if they are present - or if you tell on commandline they are present already. For .img files they aren't overwritten either...
On Sun, 12 Sept 2021 at 18:40, Gerd Petermann <gpetermann_muenchen@hotmail.com<mailto:gpetermann_muenchen@hotmail.com>> wrote: Hi Felix,
did not read all the posts in detail. I understand that mkgmap is neither burning SSDs nor doing any excessive writing unless you call it multiple times for the same input files. So the question is: Why do you do that?
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk<mailto:mkgmap-dev-bounces@lists.mkgmap.org.uk>> im Auftrag von Felix Hartmann <extremecarver@gmail.com<mailto:extremecarver@gmail.com>> Gesendet: Freitag, 10. September 2021 00:02 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option
Well I think the .tmp files are just building up - and the renamed. So they are not causing any actual excessive write. As for the gmap - it would be cool if there is a mode to not write them.
Actually it would be great if mkgmap could write all in one go. Because the thing that takes so much time - is the address search - and that is always the same. The differences are tiny (just because MapInstall is crashing when files are missing) you need to compile them separately.
but maybe there could be a mode where mkgamp writes all in one go. So family-name / family-name1 / family-name2 description / description1 / description2 input input1 input2 family-name.. show-profiles overview-mapname product-id (and maybe I missed some options are those that would need to be given for each set of input tiles. And then just an option where you tell mkgmap files starting with which first 4 numbers are relevant for address search. No need to analyze if those other supplied .img (e.g. buildings or contourlines) need to be added to address search.). I know coded around the problem of the gmap files causing excessive writes. But yeah that is actually really complicated be it on windows or linux...
On Wed, 8 Sept 2021 at 17:07, Felix Hartmann <extremecarver@gmail.com<mailto:extremecarver@gmail.com><mailto:extremecarver@gmail.com<mailto:extremecarver@gmail.com>>> wrote: Well I could give it 20 GB ram disk, maybe 32 but then I need to render on less than 12 processes 64GB ram available). But that is not enough for Asia continent map, and I guess super tight for Europe...
Mkgmap could definitely keep those .tmp files in memory. But the important bit is the gmap files not needeed to be written.... Also would save quite some CPU time.
On Wed, 8 Sep 2021, 14:31 ael <witwall3@disroot.org<mailto:witwall3@disroot.org><mailto:witwall3@disroot.org<mailto:witwall3@disroot.org>>> wrote: On Wed, Sep 08, 2021 at 02:10:52PM +0300, Felix Hartmann wrote:
Yes on an nvme disk you barely notice the conversion - it's really quick. BUT it is not needed if you have the files and even more - it burns your NVME SSD disk. +1. I always use an old spinning rust disk when using mkgmap to save ssd write cycles, even without contours and such. It seems to be profligate in its use of disk cycles. I did try using RAM disk, but even with 16GB on a laptop, that was soon exhausted.
ael
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk><mailto:mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk>> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- Felix Hartman - Openmtbmap.org & VeloMap.org
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- Felix Hartman - Openmtbmap.org & VeloMap.org
-- Felix Hartman - Openmtbmap.org & VeloMap.org
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi all, attached is a quick patch that implements experimaltal option --x-gmapi-minimal. If used instead of --gmapi mkgmap will not fail if a write-protected output file exists in the gmapi output folder and mkgmap copies data from *.img. It should still crash when other files like Info.xml are written. BTW: no conversion is done when those files are written. I think mkgmap simply copies data from sub files in *.img into single files. So, the same sequence of Bytes exists two or more times on the Computer. Windows Server seems to support automatic data deduplication (1). I am sure Linux offers similar support. No idea how effective or reliable it is, but it might be worth trying. Gerd (1) https://docs.microsoft.com/en-us/windows-server/storage/data-deduplication/i... ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Carlos Dávila <carlos@alternativaslibres.org> Gesendet: Sonntag, 12. September 2021 22:45 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option I think it's a good idea if mkgmap checks the required files are present. El 12/9/21 a las 21:16, Gerd Petermann escribió:
Hi Felix,
so you just want a new option so that mkgmap doesn't fail if it cannot overwrite (write-protected) files in the output directory, right? No need to verify the content of the exiting file(s)?
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Felix Hartmann <extremecarver@gmail.com> Gesendet: Sonntag, 12. September 2021 19:10 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option
And well - it is burning SSD for the contourlines currently even if not calling multiple times. 130GB minimum per worldwide map update for all geofabrik extracts at 20m equidistance. 260GB at 10m. With calling multiple times and 10m and 20m I had about 2TB of useless writes per weekly map update. I've got rid of all of them with my uncommenting the line, plus saved about 500GB of writes by now doing all the gmapsupp.img and gmap stuff in Ramdisk So now instead of 4TB of disk writes per map update I'm down to 1TB.. (and yeah the resplitting of tiles added another 5TB of writes - but that could have been fixed easily by changing order of commands too).
On Sun, 12 Sept 2021 at 20:04, Felix Hartmann <extremecarver@gmail.com<mailto:extremecarver@gmail.com>> wrote: because you need to do it if you want to show contourlines. Now compiling the worldwide contourlines each time again - would burn SSD as well - besides taking longer than just compiling the maps. So everyone who offers maps for many countries as download puts the contourlines separate. If you want to offer the choice of showing contourlines or not - mkmgap will write the gmap contourlines once not needed, and once the maps not needed. If you don't want to offer that option - it's only the contourlines being written without need. Contourlines for all geofabrik extracts 20m equidistance are about 130GB, 10m would be 260GB. If you offer 10m and 20m it will be even more not needed writes.
The only solution is to go for a Ramdisk instead - However you likely will need 128GB of RAM to do that - because for Asia or Europe you need a 64GB Ramdisk. Same for North America extract (but few people offer that and just have Canada and the US divided into the 4-5 zones). For other maps you will get by with a 15-20GB Ramdisk (which I have resorted to now for all but the windows installers because I don't want to let the server wait for ages for the single thread NSIS wrapping up the data and instead start the next country in parallel). And yes going RAMDISK already using my patch and symlinking those files so mkgmap cannot overwrite them (would still be faster if mkgmap didn't try in first place I think). If you want to write out the contourlines as well besides the additional time - for Asia continent you may then go for a 90GB Ramdisk minimum if offering windows and gmap installers. In this case gmapsupp.img donwnloads aren't possible anyhow due to the huge data amounts. For sm aller countries I have them too....
I coded around this now by using symlinks - but yeah that will be quite a lot of work and is prone to break - while I guess it's only 10 lines or so to add to mkgmap code to have a mode that does not write them out if they are present - or if you tell on commandline they are present already. For .img files they aren't overwritten either...
On Sun, 12 Sept 2021 at 18:40, Gerd Petermann <gpetermann_muenchen@hotmail.com<mailto:gpetermann_muenchen@hotmail.com>> wrote: Hi Felix,
did not read all the posts in detail. I understand that mkgmap is neither burning SSDs nor doing any excessive writing unless you call it multiple times for the same input files. So the question is: Why do you do that?
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk<mailto:mkgmap-dev-bounces@lists.mkgmap.org.uk>> im Auftrag von Felix Hartmann <extremecarver@gmail.com<mailto:extremecarver@gmail.com>> Gesendet: Freitag, 10. September 2021 00:02 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option
Well I think the .tmp files are just building up - and the renamed. So they are not causing any actual excessive write. As for the gmap - it would be cool if there is a mode to not write them.
Actually it would be great if mkgmap could write all in one go. Because the thing that takes so much time - is the address search - and that is always the same. The differences are tiny (just because MapInstall is crashing when files are missing) you need to compile them separately.
but maybe there could be a mode where mkgamp writes all in one go. So family-name / family-name1 / family-name2 description / description1 / description2 input input1 input2 family-name.. show-profiles overview-mapname product-id (and maybe I missed some options are those that would need to be given for each set of input tiles. And then just an option where you tell mkgmap files starting with which first 4 numbers are relevant for address search. No need to analyze if those other supplied .img (e.g. buildings or contourlines) need to be added to address search.). I know coded around the problem of the gmap files causing excessive writes. But yeah that is actually really complicated be it on windows or linux...
On Wed, 8 Sept 2021 at 17:07, Felix Hartmann <extremecarver@gmail.com<mailto:extremecarver@gmail.com><mailto:extremecarver@gmail.com<mailto:extremecarver@gmail.com>>> wrote: Well I could give it 20 GB ram disk, maybe 32 but then I need to render on less than 12 processes 64GB ram available). But that is not enough for Asia continent map, and I guess super tight for Europe...
Mkgmap could definitely keep those .tmp files in memory. But the important bit is the gmap files not needeed to be written.... Also would save quite some CPU time.
On Wed, 8 Sep 2021, 14:31 ael <witwall3@disroot.org<mailto:witwall3@disroot.org><mailto:witwall3@disroot.org<mailto:witwall3@disroot.org>>> wrote: On Wed, Sep 08, 2021 at 02:10:52PM +0300, Felix Hartmann wrote:
Yes on an nvme disk you barely notice the conversion - it's really quick. BUT it is not needed if you have the files and even more - it burns your NVME SSD disk. +1. I always use an old spinning rust disk when using mkgmap to save ssd write cycles, even without contours and such. It seems to be profligate in its use of disk cycles. I did try using RAM disk, but even with 16GB on a laptop, that was soon exhausted.
ael
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk><mailto:mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk>> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- Felix Hartman - Openmtbmap.org & VeloMap.org
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- Felix Hartman - Openmtbmap.org & VeloMap.org
-- Felix Hartman - Openmtbmap.org & VeloMap.org
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Thanks Gerd, but that is just removing the warning if it cannot overwrite, correct? If it can overwrite the gmap file it will still overwrite it as I see.. So in essence just a bit more intelligent then my disabling that line. I think it should not overwrite at all if present and --x-gmapi-minimal (then you don't have to move away the files and link them back to the original folder). On Mon, 13 Sept 2021 at 10:43, Gerd Petermann < gpetermann_muenchen@hotmail.com> wrote:
Hi all,
attached is a quick patch that implements experimaltal option --x-gmapi-minimal.
If used instead of --gmapi mkgmap will not fail if a write-protected output file exists in the gmapi output folder and mkgmap copies data from *.img. It should still crash when other files like Info.xml are written.
BTW: no conversion is done when those files are written. I think mkgmap simply copies data from sub files in *.img into single files. So, the same sequence of Bytes exists two or more times on the Computer. Windows Server seems to support automatic data deduplication (1). I am sure Linux offers similar support. No idea how effective or reliable it is, but it might be worth trying.
Gerd (1)
https://docs.microsoft.com/en-us/windows-server/storage/data-deduplication/i...
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Carlos Dávila <carlos@alternativaslibres.org> Gesendet: Sonntag, 12. September 2021 22:45 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option
I think it's a good idea if mkgmap checks the required files are present.
El 12/9/21 a las 21:16, Gerd Petermann escribió:
Hi Felix,
so you just want a new option so that mkgmap doesn't fail if it cannot overwrite (write-protected) files in the output directory, right? No need to verify the content of the exiting file(s)?
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Felix Hartmann <extremecarver@gmail.com> Gesendet: Sonntag, 12. September 2021 19:10 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option
And well - it is burning SSD for the contourlines currently even if not calling multiple times. 130GB minimum per worldwide map update for all geofabrik extracts at 20m equidistance. 260GB at 10m. With calling multiple times and 10m and 20m I had about 2TB of useless writes per weekly map update. I've got rid of all of them with my uncommenting the line, plus saved about 500GB of writes by now doing all the gmapsupp.img and gmap stuff in Ramdisk So now instead of 4TB of disk writes per map update I'm down to 1TB.. (and yeah the resplitting of tiles added another 5TB of writes - but that could have been fixed easily by changing order of commands too).
On Sun, 12 Sept 2021 at 20:04, Felix Hartmann <extremecarver@gmail.com <mailto:extremecarver@gmail.com>> wrote: because you need to do it if you want to show contourlines. Now compiling the worldwide contourlines each time again - would burn SSD as well - besides taking longer than just compiling the maps. So everyone who offers maps for many countries as download puts the contourlines separate. If you want to offer the choice of showing contourlines or not - mkmgap will write the gmap contourlines once not needed, and once the maps not needed. If you don't want to offer that option - it's only the contourlines being written without need. Contourlines for all geofabrik extracts 20m equidistance are about 130GB, 10m would be 260GB. If you offer 10m and 20m it will be even more not needed writes.
The only solution is to go for a Ramdisk instead - However you likely will need 128GB of RAM to do that - because for Asia or Europe you need a 64GB Ramdisk. Same for North America extract (but few people offer that and just have Canada and the US divided into the 4-5 zones). For other maps you will get by with a 15-20GB Ramdisk (which I have resorted to now for all but the windows installers because I don't want to let the server wait for ages for the single thread NSIS wrapping up the data and instead start the next country in parallel). And yes going RAMDISK already using my patch and symlinking those files so mkgmap cannot overwrite them (would still be faster if mkgmap didn't try in first place I think). If you want to write out the contourlines as well besides the additional time - for Asia continent you may then go for a 90GB Ramdisk minimum if offering windows and gmap installers. In this case gmapsupp.img donwnloads aren't possible anyhow due to the huge data amounts. For sm aller countries I have them too....
I coded around this now by using symlinks - but yeah that will be quite a lot of work and is prone to break - while I guess it's only 10 lines or so to add to mkgmap code to have a mode that does not write them out if they are present - or if you tell on commandline they are present already. For .img files they aren't overwritten either...
On Sun, 12 Sept 2021 at 18:40, Gerd Petermann < gpetermann_muenchen@hotmail.com<mailto:gpetermann_muenchen@hotmail.com>> wrote: Hi Felix,
did not read all the posts in detail. I understand that mkgmap is neither burning SSDs nor doing any excessive writing unless you call it multiple times for the same input files. So the question is: Why do you do that?
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk<mailto: mkgmap-dev-bounces@lists.mkgmap.org.uk>> im Auftrag von Felix Hartmann < extremecarver@gmail.com<mailto:extremecarver@gmail.com>> Gesendet: Freitag, 10. September 2021 00:02 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option
Well I think the .tmp files are just building up - and the renamed. So they are not causing any actual excessive write. As for the gmap - it would be cool if there is a mode to not write them.
Actually it would be great if mkgmap could write all in one go. Because the thing that takes so much time - is the address search - and that is always the same. The differences are tiny (just because MapInstall is crashing when files are missing) you need to compile them separately.
but maybe there could be a mode where mkgamp writes all in one go. So family-name / family-name1 / family-name2 description / description1 / description2 input input1 input2 family-name.. show-profiles overview-mapname product-id (and maybe I missed some options are those that would need to be given for each set of input tiles. And then just an option where you tell mkgmap files starting with which first 4 numbers are relevant for address search. No need to analyze if those other supplied .img (e.g. buildings or contourlines) need to be added to address search.). I know coded around the problem of the gmap files causing excessive writes. But yeah that is actually really complicated be it on windows or linux...
On Wed, 8 Sept 2021 at 17:07, Felix Hartmann <extremecarver@gmail.com <mailto:extremecarver@gmail.com><mailto:extremecarver@gmail.com<mailto: extremecarver@gmail.com>>> wrote: Well I could give it 20 GB ram disk, maybe 32 but then I need to render on less than 12 processes 64GB ram available). But that is not enough for Asia continent map, and I guess super tight for Europe...
Mkgmap could definitely keep those .tmp files in memory. But the important bit is the gmap files not needeed to be written.... Also would save quite some CPU time.
On Wed, 8 Sep 2021, 14:31 ael <witwall3@disroot.org<mailto: witwall3@disroot.org><mailto:witwall3@disroot.org<mailto: witwall3@disroot.org>>> wrote: On Wed, Sep 08, 2021 at 02:10:52PM +0300, Felix Hartmann wrote:
Yes on an nvme disk you barely notice the conversion - it's really quick. BUT it is not needed if you have the files and even more - it burns your NVME SSD disk. +1. I always use an old spinning rust disk when using mkgmap to save ssd write cycles, even without contours and such. It seems to be profligate in its use of disk cycles. I did try using RAM disk, but even with 16GB on a laptop, that was soon exhausted.
ael
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk<mailto: mkgmap-dev@lists.mkgmap.org.uk>> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- Felix Hartman - Openmtbmap.org & VeloMap.org
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- Felix Hartman - Openmtbmap.org & VeloMap.org
-- Felix Hartman - Openmtbmap.org & VeloMap.org
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- Felix Hartman - Openmtbmap.org & VeloMap.org
Not sure if it makes it possible to use read only attribute instead of moving and mklink. Maybe yes - because that was not possible before. So it then would be set read only - instead of of move & mklink - and at the end remove read only instead of moving back. On Mon, 13 Sept 2021 at 12:59, Felix Hartmann <extremecarver@gmail.com> wrote:
Thanks Gerd,
but that is just removing the warning if it cannot overwrite, correct? If it can overwrite the gmap file it will still overwrite it as I see.. So in essence just a bit more intelligent then my disabling that line.
I think it should not overwrite at all if present and --x-gmapi-minimal (then you don't have to move away the files and link them back to the original folder).
On Mon, 13 Sept 2021 at 10:43, Gerd Petermann < gpetermann_muenchen@hotmail.com> wrote:
Hi all,
attached is a quick patch that implements experimaltal option --x-gmapi-minimal.
If used instead of --gmapi mkgmap will not fail if a write-protected output file exists in the gmapi output folder and mkgmap copies data from *.img. It should still crash when other files like Info.xml are written.
BTW: no conversion is done when those files are written. I think mkgmap simply copies data from sub files in *.img into single files. So, the same sequence of Bytes exists two or more times on the Computer. Windows Server seems to support automatic data deduplication (1). I am sure Linux offers similar support. No idea how effective or reliable it is, but it might be worth trying.
Gerd (1)
https://docs.microsoft.com/en-us/windows-server/storage/data-deduplication/i...
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Carlos Dávila <carlos@alternativaslibres.org> Gesendet: Sonntag, 12. September 2021 22:45 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option
I think it's a good idea if mkgmap checks the required files are present.
El 12/9/21 a las 21:16, Gerd Petermann escribió:
Hi Felix,
so you just want a new option so that mkgmap doesn't fail if it cannot overwrite (write-protected) files in the output directory, right? No need to verify the content of the exiting file(s)?
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Felix Hartmann <extremecarver@gmail.com> Gesendet: Sonntag, 12. September 2021 19:10 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option
And well - it is burning SSD for the contourlines currently even if not calling multiple times. 130GB minimum per worldwide map update for all geofabrik extracts at 20m equidistance. 260GB at 10m. With calling multiple times and 10m and 20m I had about 2TB of useless writes per weekly map update. I've got rid of all of them with my uncommenting the line, plus saved about 500GB of writes by now doing all the gmapsupp.img and gmap stuff in Ramdisk So now instead of 4TB of disk writes per map update I'm down to 1TB.. (and yeah the resplitting of tiles added another 5TB of writes - but that could have been fixed easily by changing order of commands too).
On Sun, 12 Sept 2021 at 20:04, Felix Hartmann <extremecarver@gmail.com <mailto:extremecarver@gmail.com>> wrote: because you need to do it if you want to show contourlines. Now compiling the worldwide contourlines each time again - would burn SSD as well - besides taking longer than just compiling the maps. So everyone who offers maps for many countries as download puts the contourlines separate. If you want to offer the choice of showing contourlines or not - mkmgap will write the gmap contourlines once not needed, and once the maps not needed. If you don't want to offer that option - it's only the contourlines being written without need. Contourlines for all geofabrik extracts 20m equidistance are about 130GB, 10m would be 260GB. If you offer 10m and 20m it will be even more not needed writes.
The only solution is to go for a Ramdisk instead - However you likely will need 128GB of RAM to do that - because for Asia or Europe you need a 64GB Ramdisk. Same for North America extract (but few people offer that and just have Canada and the US divided into the 4-5 zones). For other maps you will get by with a 15-20GB Ramdisk (which I have resorted to now for all but the windows installers because I don't want to let the server wait for ages for the single thread NSIS wrapping up the data and instead start the next country in parallel). And yes going RAMDISK already using my patch and symlinking those files so mkgmap cannot overwrite them (would still be faster if mkgmap didn't try in first place I think). If you want to write out the contourlines as well besides the additional time - for Asia continent you may then go for a 90GB Ramdisk minimum if offering windows and gmap installers. In this case gmapsupp.img donwnloads aren't possible anyhow due to the huge data amounts. For sm aller countries I have them too....
I coded around this now by using symlinks - but yeah that will be quite a lot of work and is prone to break - while I guess it's only 10 lines or so to add to mkgmap code to have a mode that does not write them out if they are present - or if you tell on commandline they are present already. For .img files they aren't overwritten either...
On Sun, 12 Sept 2021 at 18:40, Gerd Petermann < gpetermann_muenchen@hotmail.com<mailto:gpetermann_muenchen@hotmail.com>> wrote: Hi Felix,
did not read all the posts in detail. I understand that mkgmap is neither burning SSDs nor doing any excessive writing unless you call it multiple times for the same input files. So the question is: Why do you do that?
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk<mailto: mkgmap-dev-bounces@lists.mkgmap.org.uk>> im Auftrag von Felix Hartmann < extremecarver@gmail.com<mailto:extremecarver@gmail.com>> Gesendet: Freitag, 10. September 2021 00:02 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option
Well I think the .tmp files are just building up - and the renamed. So they are not causing any actual excessive write. As for the gmap - it would be cool if there is a mode to not write them.
Actually it would be great if mkgmap could write all in one go. Because the thing that takes so much time - is the address search - and that is always the same. The differences are tiny (just because MapInstall is crashing when files are missing) you need to compile them separately.
but maybe there could be a mode where mkgamp writes all in one go. So family-name / family-name1 / family-name2 description / description1 / description2 input input1 input2 family-name.. show-profiles overview-mapname product-id (and maybe I missed some options are those that would need to be given for each set of input tiles. And then just an option where you tell mkgmap files starting with which first 4 numbers are relevant for address search. No need to analyze if those other supplied .img (e.g. buildings or contourlines) need to be added to address search.). I know coded around the problem of the gmap files causing excessive writes. But yeah that is actually really complicated be it on windows or linux...
On Wed, 8 Sept 2021 at 17:07, Felix Hartmann <extremecarver@gmail.com <mailto:extremecarver@gmail.com><mailto:extremecarver@gmail.com<mailto: extremecarver@gmail.com>>> wrote: Well I could give it 20 GB ram disk, maybe 32 but then I need to render on less than 12 processes 64GB ram available). But that is not enough for Asia continent map, and I guess super tight for Europe...
Mkgmap could definitely keep those .tmp files in memory. But the important bit is the gmap files not needeed to be written.... Also would save quite some CPU time.
On Wed, 8 Sep 2021, 14:31 ael <witwall3@disroot.org<mailto: witwall3@disroot.org><mailto:witwall3@disroot.org<mailto: witwall3@disroot.org>>> wrote: On Wed, Sep 08, 2021 at 02:10:52PM +0300, Felix Hartmann wrote:
Yes on an nvme disk you barely notice the conversion - it's really quick. BUT it is not needed if you have the files and even more - it burns your NVME SSD disk. +1. I always use an old spinning rust disk when using mkgmap to save ssd write cycles, even without contours and such. It seems to be profligate in its use of disk cycles. I did try using RAM disk, but even with 16GB on a laptop, that was soon exhausted.
ael
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk<mailto: mkgmap-dev@lists.mkgmap.org.uk>> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- Felix Hartman - Openmtbmap.org & VeloMap.org
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- Felix Hartman - Openmtbmap.org & VeloMap.org
-- Felix Hartman - Openmtbmap.org & VeloMap.org
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- Felix Hartman - Openmtbmap.org & VeloMap.org
-- Felix Hartman - Openmtbmap.org & VeloMap.org
Oh - but data was certainly written - A rename will not show as data written in both Task manager on Windows, as well as in the smart data (I'm using Windows 10 pro not Windows server however - maybe that functionality is limited to windows server?) On Mon, 13 Sept 2021 at 13:06, Felix Hartmann <extremecarver@gmail.com> wrote:
Not sure if it makes it possible to use read only attribute instead of moving and mklink. Maybe yes - because that was not possible before. So it then would be set read only - instead of of move & mklink - and at the end remove read only instead of moving back.
On Mon, 13 Sept 2021 at 12:59, Felix Hartmann <extremecarver@gmail.com> wrote:
Thanks Gerd,
but that is just removing the warning if it cannot overwrite, correct? If it can overwrite the gmap file it will still overwrite it as I see.. So in essence just a bit more intelligent then my disabling that line.
I think it should not overwrite at all if present and --x-gmapi-minimal (then you don't have to move away the files and link them back to the original folder).
On Mon, 13 Sept 2021 at 10:43, Gerd Petermann < gpetermann_muenchen@hotmail.com> wrote:
Hi all,
attached is a quick patch that implements experimaltal option --x-gmapi-minimal.
If used instead of --gmapi mkgmap will not fail if a write-protected output file exists in the gmapi output folder and mkgmap copies data from *.img. It should still crash when other files like Info.xml are written.
BTW: no conversion is done when those files are written. I think mkgmap simply copies data from sub files in *.img into single files. So, the same sequence of Bytes exists two or more times on the Computer. Windows Server seems to support automatic data deduplication (1). I am sure Linux offers similar support. No idea how effective or reliable it is, but it might be worth trying.
Gerd (1)
https://docs.microsoft.com/en-us/windows-server/storage/data-deduplication/i...
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Carlos Dávila <carlos@alternativaslibres.org> Gesendet: Sonntag, 12. September 2021 22:45 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option
I think it's a good idea if mkgmap checks the required files are present.
El 12/9/21 a las 21:16, Gerd Petermann escribió:
Hi Felix,
so you just want a new option so that mkgmap doesn't fail if it cannot overwrite (write-protected) files in the output directory, right? No need to verify the content of the exiting file(s)?
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Felix Hartmann <extremecarver@gmail.com> Gesendet: Sonntag, 12. September 2021 19:10 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option
And well - it is burning SSD for the contourlines currently even if not calling multiple times. 130GB minimum per worldwide map update for all geofabrik extracts at 20m equidistance. 260GB at 10m. With calling multiple times and 10m and 20m I had about 2TB of useless writes per weekly map update. I've got rid of all of them with my uncommenting the line, plus saved about 500GB of writes by now doing all the gmapsupp.img and gmap stuff in Ramdisk So now instead of 4TB of disk writes per map update I'm down to 1TB.. (and yeah the resplitting of tiles added another 5TB of writes - but that could have been fixed easily by changing order of commands too).
On Sun, 12 Sept 2021 at 20:04, Felix Hartmann <extremecarver@gmail.com <mailto:extremecarver@gmail.com>> wrote: because you need to do it if you want to show contourlines. Now compiling the worldwide contourlines each time again - would burn SSD as well - besides taking longer than just compiling the maps. So everyone who offers maps for many countries as download puts the contourlines separate. If you want to offer the choice of showing contourlines or not - mkmgap will write the gmap contourlines once not needed, and once the maps not needed. If you don't want to offer that option - it's only the contourlines being written without need. Contourlines for all geofabrik extracts 20m equidistance are about 130GB, 10m would be 260GB. If you offer 10m and 20m it will be even more not needed writes.
The only solution is to go for a Ramdisk instead - However you likely will need 128GB of RAM to do that - because for Asia or Europe you need a 64GB Ramdisk. Same for North America extract (but few people offer that and just have Canada and the US divided into the 4-5 zones). For other maps you will get by with a 15-20GB Ramdisk (which I have resorted to now for all but the windows installers because I don't want to let the server wait for ages for the single thread NSIS wrapping up the data and instead start the next country in parallel). And yes going RAMDISK already using my patch and symlinking those files so mkgmap cannot overwrite them (would still be faster if mkgmap didn't try in first place I think). If you want to write out the contourlines as well besides the additional time - for Asia continent you may then go for a 90GB Ramdisk minimum if offering windows and gmap installers. In this case gmapsupp.img donwnloads aren't possible anyhow due to the huge data amounts. For sm aller countries I have them too....
I coded around this now by using symlinks - but yeah that will be quite a lot of work and is prone to break - while I guess it's only 10 lines or so to add to mkgmap code to have a mode that does not write them out if they are present - or if you tell on commandline they are present already. For .img files they aren't overwritten either...
On Sun, 12 Sept 2021 at 18:40, Gerd Petermann < gpetermann_muenchen@hotmail.com<mailto:gpetermann_muenchen@hotmail.com>> wrote: Hi Felix,
did not read all the posts in detail. I understand that mkgmap is neither burning SSDs nor doing any excessive writing unless you call it multiple times for the same input files. So the question is: Why do you do that?
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk<mailto: mkgmap-dev-bounces@lists.mkgmap.org.uk>> im Auftrag von Felix Hartmann < extremecarver@gmail.com<mailto:extremecarver@gmail.com>> Gesendet: Freitag, 10. September 2021 00:02 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option
Well I think the .tmp files are just building up - and the renamed. So they are not causing any actual excessive write. As for the gmap - it would be cool if there is a mode to not write them.
Actually it would be great if mkgmap could write all in one go. Because the thing that takes so much time - is the address search - and that is always the same. The differences are tiny (just because MapInstall is crashing when files are missing) you need to compile them separately.
but maybe there could be a mode where mkgamp writes all in one go. So family-name / family-name1 / family-name2 description / description1 / description2 input input1 input2 family-name.. show-profiles overview-mapname product-id (and maybe I missed some options are those that would need to be given for each set of input tiles. And then just an option where you tell mkgmap files starting with which first 4 numbers are relevant for address search. No need to analyze if those other supplied .img (e.g. buildings or contourlines) need to be added to address search.). I know coded around the problem of the gmap files causing excessive writes. But yeah that is actually really complicated be it on windows or linux...
On Wed, 8 Sept 2021 at 17:07, Felix Hartmann <extremecarver@gmail.com <mailto:extremecarver@gmail.com><mailto:extremecarver@gmail.com<mailto: extremecarver@gmail.com>>> wrote: Well I could give it 20 GB ram disk, maybe 32 but then I need to render on less than 12 processes 64GB ram available). But that is not enough for Asia continent map, and I guess super tight for Europe...
Mkgmap could definitely keep those .tmp files in memory. But the important bit is the gmap files not needeed to be written.... Also would save quite some CPU time.
On Wed, 8 Sep 2021, 14:31 ael <witwall3@disroot.org<mailto: witwall3@disroot.org><mailto:witwall3@disroot.org<mailto: witwall3@disroot.org>>> wrote: On Wed, Sep 08, 2021 at 02:10:52PM +0300, Felix Hartmann wrote:
Yes on an nvme disk you barely notice the conversion - it's really quick. BUT it is not needed if you have the files and even more - it burns your NVME SSD disk. +1. I always use an old spinning rust disk when using mkgmap to save ssd write cycles, even without contours and such. It seems to be profligate in its use of disk cycles. I did try using RAM disk, but even with 16GB on a laptop, that was soon exhausted.
ael
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk<mailto: mkgmap-dev@lists.mkgmap.org.uk>> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- Felix Hartman - Openmtbmap.org & VeloMap.org
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- Felix Hartman - Openmtbmap.org & VeloMap.org
-- Felix Hartman - Openmtbmap.org & VeloMap.org
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- Felix Hartman - Openmtbmap.org & VeloMap.org
-- Felix Hartman - Openmtbmap.org & VeloMap.org
-- Felix Hartman - Openmtbmap.org & VeloMap.org
Hi Felix, sorry, the Data Deduplication as implemented in Windows Server would not help here. It works after data was written. And yes, files which are not just copies of the *.img are written as before. My understanding is that you have different product directories in the gmapi folder and that you write protect those files which should be kept. If you have a script to zip the gmapi directory you have to exclude the unwanted folders. Didn't try it. Does it make sense? Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Felix Hartmann <extremecarver@gmail.com> Gesendet: Montag, 13. September 2021 12:09 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option Oh - but data was certainly written - A rename will not show as data written in both Task manager on Windows, as well as in the smart data (I'm using Windows 10 pro not Windows server however - maybe that functionality is limited to windows server?) On Mon, 13 Sept 2021 at 13:06, Felix Hartmann <extremecarver@gmail.com<mailto:extremecarver@gmail.com>> wrote: Not sure if it makes it possible to use read only attribute instead of moving and mklink. Maybe yes - because that was not possible before. So it then would be set read only - instead of of move & mklink - and at the end remove read only instead of moving back. On Mon, 13 Sept 2021 at 12:59, Felix Hartmann <extremecarver@gmail.com<mailto:extremecarver@gmail.com>> wrote: Thanks Gerd, but that is just removing the warning if it cannot overwrite, correct? If it can overwrite the gmap file it will still overwrite it as I see.. So in essence just a bit more intelligent then my disabling that line. I think it should not overwrite at all if present and --x-gmapi-minimal (then you don't have to move away the files and link them back to the original folder). On Mon, 13 Sept 2021 at 10:43, Gerd Petermann <gpetermann_muenchen@hotmail.com<mailto:gpetermann_muenchen@hotmail.com>> wrote: Hi all, attached is a quick patch that implements experimaltal option --x-gmapi-minimal. If used instead of --gmapi mkgmap will not fail if a write-protected output file exists in the gmapi output folder and mkgmap copies data from *.img. It should still crash when other files like Info.xml are written. BTW: no conversion is done when those files are written. I think mkgmap simply copies data from sub files in *.img into single files. So, the same sequence of Bytes exists two or more times on the Computer. Windows Server seems to support automatic data deduplication (1). I am sure Linux offers similar support. No idea how effective or reliable it is, but it might be worth trying. Gerd (1) https://docs.microsoft.com/en-us/windows-server/storage/data-deduplication/i... ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk<mailto:mkgmap-dev-bounces@lists.mkgmap.org.uk>> im Auftrag von Carlos Dávila <carlos@alternativaslibres.org<mailto:carlos@alternativaslibres.org>> Gesendet: Sonntag, 12. September 2021 22:45 An: mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk> Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option I think it's a good idea if mkgmap checks the required files are present. El 12/9/21 a las 21:16, Gerd Petermann escribió:
Hi Felix,
so you just want a new option so that mkgmap doesn't fail if it cannot overwrite (write-protected) files in the output directory, right? No need to verify the content of the exiting file(s)?
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk<mailto:mkgmap-dev-bounces@lists.mkgmap.org.uk>> im Auftrag von Felix Hartmann <extremecarver@gmail.com<mailto:extremecarver@gmail.com>> Gesendet: Sonntag, 12. September 2021 19:10 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option
And well - it is burning SSD for the contourlines currently even if not calling multiple times. 130GB minimum per worldwide map update for all geofabrik extracts at 20m equidistance. 260GB at 10m. With calling multiple times and 10m and 20m I had about 2TB of useless writes per weekly map update. I've got rid of all of them with my uncommenting the line, plus saved about 500GB of writes by now doing all the gmapsupp.img and gmap stuff in Ramdisk So now instead of 4TB of disk writes per map update I'm down to 1TB.. (and yeah the resplitting of tiles added another 5TB of writes - but that could have been fixed easily by changing order of commands too).
On Sun, 12 Sept 2021 at 20:04, Felix Hartmann <extremecarver@gmail.com<mailto:extremecarver@gmail.com><mailto:extremecarver@gmail.com<mailto:extremecarver@gmail.com>>> wrote: because you need to do it if you want to show contourlines. Now compiling the worldwide contourlines each time again - would burn SSD as well - besides taking longer than just compiling the maps. So everyone who offers maps for many countries as download puts the contourlines separate. If you want to offer the choice of showing contourlines or not - mkmgap will write the gmap contourlines once not needed, and once the maps not needed. If you don't want to offer that option - it's only the contourlines being written without need. Contourlines for all geofabrik extracts 20m equidistance are about 130GB, 10m would be 260GB. If you offer 10m and 20m it will be even more not needed writes.
The only solution is to go for a Ramdisk instead - However you likely will need 128GB of RAM to do that - because for Asia or Europe you need a 64GB Ramdisk. Same for North America extract (but few people offer that and just have Canada and the US divided into the 4-5 zones). For other maps you will get by with a 15-20GB Ramdisk (which I have resorted to now for all but the windows installers because I don't want to let the server wait for ages for the single thread NSIS wrapping up the data and instead start the next country in parallel). And yes going RAMDISK already using my patch and symlinking those files so mkgmap cannot overwrite them (would still be faster if mkgmap didn't try in first place I think). If you want to write out the contourlines as well besides the additional time - for Asia continent you may then go for a 90GB Ramdisk minimum if offering windows and gmap installers. In this case gmapsupp.img donwnloads aren't possible anyhow due to the huge data amounts. For sm aller countries I have them too....
I coded around this now by using symlinks - but yeah that will be quite a lot of work and is prone to break - while I guess it's only 10 lines or so to add to mkgmap code to have a mode that does not write them out if they are present - or if you tell on commandline they are present already. For .img files they aren't overwritten either...
On Sun, 12 Sept 2021 at 18:40, Gerd Petermann <gpetermann_muenchen@hotmail.com<mailto:gpetermann_muenchen@hotmail.com><mailto:gpetermann_muenchen@hotmail.com<mailto:gpetermann_muenchen@hotmail.com>>> wrote: Hi Felix,
did not read all the posts in detail. I understand that mkgmap is neither burning SSDs nor doing any excessive writing unless you call it multiple times for the same input files. So the question is: Why do you do that?
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk<mailto:mkgmap-dev-bounces@lists.mkgmap.org.uk><mailto:mkgmap-dev-bounces@lists.mkgmap.org.uk<mailto:mkgmap-dev-bounces@lists.mkgmap.org.uk>>> im Auftrag von Felix Hartmann <extremecarver@gmail.com<mailto:extremecarver@gmail.com><mailto:extremecarver@gmail.com<mailto:extremecarver@gmail.com>>> Gesendet: Freitag, 10. September 2021 00:02 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option
Well I think the .tmp files are just building up - and the renamed. So they are not causing any actual excessive write. As for the gmap - it would be cool if there is a mode to not write them.
Actually it would be great if mkgmap could write all in one go. Because the thing that takes so much time - is the address search - and that is always the same. The differences are tiny (just because MapInstall is crashing when files are missing) you need to compile them separately.
but maybe there could be a mode where mkgamp writes all in one go. So family-name / family-name1 / family-name2 description / description1 / description2 input input1 input2 family-name.. show-profiles overview-mapname product-id (and maybe I missed some options are those that would need to be given for each set of input tiles. And then just an option where you tell mkgmap files starting with which first 4 numbers are relevant for address search. No need to analyze if those other supplied .img (e.g. buildings or contourlines) need to be added to address search.). I know coded around the problem of the gmap files causing excessive writes. But yeah that is actually really complicated be it on windows or linux...
On Wed, 8 Sept 2021 at 17:07, Felix Hartmann <extremecarver@gmail.com<mailto:extremecarver@gmail.com><mailto:extremecarver@gmail.com<mailto:extremecarver@gmail.com>><mailto:extremecarver@gmail.com<mailto:extremecarver@gmail.com><mailto:extremecarver@gmail.com<mailto:extremecarver@gmail.com>>>> wrote: Well I could give it 20 GB ram disk, maybe 32 but then I need to render on less than 12 processes 64GB ram available). But that is not enough for Asia continent map, and I guess super tight for Europe...
Mkgmap could definitely keep those .tmp files in memory. But the important bit is the gmap files not needeed to be written.... Also would save quite some CPU time.
On Wed, 8 Sep 2021, 14:31 ael <witwall3@disroot.org<mailto:witwall3@disroot.org><mailto:witwall3@disroot.org<mailto:witwall3@disroot.org>><mailto:witwall3@disroot.org<mailto:witwall3@disroot.org><mailto:witwall3@disroot.org<mailto:witwall3@disroot.org>>>> wrote: On Wed, Sep 08, 2021 at 02:10:52PM +0300, Felix Hartmann wrote:
Yes on an nvme disk you barely notice the conversion - it's really quick. BUT it is not needed if you have the files and even more - it burns your NVME SSD disk. +1. I always use an old spinning rust disk when using mkgmap to save ssd write cycles, even without contours and such. It seems to be profligate in its use of disk cycles. I did try using RAM disk, but even with 16GB on a laptop, that was soon exhausted.
ael
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk><mailto:mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk>><mailto:mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk><mailto:mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk>>> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- Felix Hartman - Openmtbmap.org & VeloMap.org
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk><mailto:mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk>> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- Felix Hartman - Openmtbmap.org & VeloMap.org
-- Felix Hartman - Openmtbmap.org & VeloMap.org
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev -- Felix Hartman - Openmtbmap.org & VeloMap.org -- Felix Hartman - Openmtbmap.org & VeloMap.org -- Felix Hartman - Openmtbmap.org & VeloMap.org
I move away the info.xml - and use a different name for the mapset files and then just have mkgmap any file that is input in osm.pbf / osm / o5m format. Gmapi-minimal should not convert any file that is supplied as .img - as it can be assumed that those exist already (if they do not - then create them with --gmapi). That is in my opinion the best approach. So mkgmap does not even try to convert them. Afterwards I distribute a gmapi folder that includes all the data - and by replacing the info.xml you can enable or disable contourlines. For big countries the contourlines would be a separate download anyhow - so the user only needs to download the maps (including mapset files) but not redownload the contourlines. Same principle as in offering the contourlines as a separate gmapsupp.img file. so you have maps.img contourlines10m.img contourlines20m.img buildings.img .... On Mon, 13 Sept 2021 at 13:17, Gerd Petermann < gpetermann_muenchen@hotmail.com> wrote:
Hi Felix,
sorry, the Data Deduplication as implemented in Windows Server would not help here. It works after data was written.
And yes, files which are not just copies of the *.img are written as before. My understanding is that you have different product directories in the gmapi folder and that you write protect those files which should be kept. If you have a script to zip the gmapi directory you have to exclude the unwanted folders. Didn't try it. Does it make sense?
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Felix Hartmann <extremecarver@gmail.com> Gesendet: Montag, 13. September 2021 12:09 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option
Oh - but data was certainly written - A rename will not show as data written in both Task manager on Windows, as well as in the smart data (I'm using Windows 10 pro not Windows server however - maybe that functionality is limited to windows server?)
On Mon, 13 Sept 2021 at 13:06, Felix Hartmann <extremecarver@gmail.com <mailto:extremecarver@gmail.com>> wrote: Not sure if it makes it possible to use read only attribute instead of moving and mklink. Maybe yes - because that was not possible before. So it then would be set read only - instead of of move & mklink - and at the end remove read only instead of moving back.
On Mon, 13 Sept 2021 at 12:59, Felix Hartmann <extremecarver@gmail.com <mailto:extremecarver@gmail.com>> wrote: Thanks Gerd,
but that is just removing the warning if it cannot overwrite, correct? If it can overwrite the gmap file it will still overwrite it as I see.. So in essence just a bit more intelligent then my disabling that line.
I think it should not overwrite at all if present and --x-gmapi-minimal (then you don't have to move away the files and link them back to the original folder).
On Mon, 13 Sept 2021 at 10:43, Gerd Petermann < gpetermann_muenchen@hotmail.com<mailto:gpetermann_muenchen@hotmail.com>> wrote: Hi all,
attached is a quick patch that implements experimaltal option --x-gmapi-minimal.
If used instead of --gmapi mkgmap will not fail if a write-protected output file exists in the gmapi output folder and mkgmap copies data from *.img. It should still crash when other files like Info.xml are written.
BTW: no conversion is done when those files are written. I think mkgmap simply copies data from sub files in *.img into single files. So, the same sequence of Bytes exists two or more times on the Computer. Windows Server seems to support automatic data deduplication (1). I am sure Linux offers similar support. No idea how effective or reliable it is, but it might be worth trying.
Gerd (1)
https://docs.microsoft.com/en-us/windows-server/storage/data-deduplication/i...
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk<mailto: mkgmap-dev-bounces@lists.mkgmap.org.uk>> im Auftrag von Carlos Dávila < carlos@alternativaslibres.org<mailto:carlos@alternativaslibres.org>> Gesendet: Sonntag, 12. September 2021 22:45 An: mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk> Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option
I think it's a good idea if mkgmap checks the required files are present.
El 12/9/21 a las 21:16, Gerd Petermann escribió:
Hi Felix,
so you just want a new option so that mkgmap doesn't fail if it cannot overwrite (write-protected) files in the output directory, right? No need to verify the content of the exiting file(s)?
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk<mailto: mkgmap-dev-bounces@lists.mkgmap.org.uk>> im Auftrag von Felix Hartmann < extremecarver@gmail.com<mailto:extremecarver@gmail.com>> Gesendet: Sonntag, 12. September 2021 19:10 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option
And well - it is burning SSD for the contourlines currently even if not calling multiple times. 130GB minimum per worldwide map update for all geofabrik extracts at 20m equidistance. 260GB at 10m. With calling multiple times and 10m and 20m I had about 2TB of useless writes per weekly map update. I've got rid of all of them with my uncommenting the line, plus saved about 500GB of writes by now doing all the gmapsupp.img and gmap stuff in Ramdisk So now instead of 4TB of disk writes per map update I'm down to 1TB.. (and yeah the resplitting of tiles added another 5TB of writes - but that could have been fixed easily by changing order of commands too).
On Sun, 12 Sept 2021 at 20:04, Felix Hartmann <extremecarver@gmail.com <mailto:extremecarver@gmail.com><mailto:extremecarver@gmail.com<mailto: extremecarver@gmail.com>>> wrote: because you need to do it if you want to show contourlines. Now compiling the worldwide contourlines each time again - would burn SSD as well - besides taking longer than just compiling the maps. So everyone who offers maps for many countries as download puts the contourlines separate. If you want to offer the choice of showing contourlines or not - mkmgap will write the gmap contourlines once not needed, and once the maps not needed. If you don't want to offer that option - it's only the contourlines being written without need. Contourlines for all geofabrik extracts 20m equidistance are about 130GB, 10m would be 260GB. If you offer 10m and 20m it will be even more not needed writes.
The only solution is to go for a Ramdisk instead - However you likely will need 128GB of RAM to do that - because for Asia or Europe you need a 64GB Ramdisk. Same for North America extract (but few people offer that and just have Canada and the US divided into the 4-5 zones). For other maps you will get by with a 15-20GB Ramdisk (which I have resorted to now for all but the windows installers because I don't want to let the server wait for ages for the single thread NSIS wrapping up the data and instead start the next country in parallel). And yes going RAMDISK already using my patch and symlinking those files so mkgmap cannot overwrite them (would still be faster if mkgmap didn't try in first place I think). If you want to write out the contourlines as well besides the additional time - for Asia continent you may then go for a 90GB Ramdisk minimum if offering windows and gmap installers. In this case gmapsupp.img donwnloads aren't possible anyhow due to the huge data amounts. For sm aller countries I have them too....
I coded around this now by using symlinks - but yeah that will be quite a lot of work and is prone to break - while I guess it's only 10 lines or so to add to mkgmap code to have a mode that does not write them out if they are present - or if you tell on commandline they are present already. For .img files they aren't overwritten either...
On Sun, 12 Sept 2021 at 18:40, Gerd Petermann < gpetermann_muenchen@hotmail.com<mailto:gpetermann_muenchen@hotmail.com <mailto:gpetermann_muenchen@hotmail.com<mailto: gpetermann_muenchen@hotmail.com>>> wrote: Hi Felix,
did not read all the posts in detail. I understand that mkgmap is neither burning SSDs nor doing any excessive writing unless you call it multiple times for the same input files. So the question is: Why do you do that?
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk<mailto: mkgmap-dev-bounces@lists.mkgmap.org.uk><mailto: mkgmap-dev-bounces@lists.mkgmap.org.uk<mailto: mkgmap-dev-bounces@lists.mkgmap.org.uk>>> im Auftrag von Felix Hartmann < extremecarver@gmail.com<mailto:extremecarver@gmail.com><mailto: extremecarver@gmail.com<mailto:extremecarver@gmail.com>>> Gesendet: Freitag, 10. September 2021 00:02 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option
Well I think the .tmp files are just building up - and the renamed. So they are not causing any actual excessive write. As for the gmap - it would be cool if there is a mode to not write them.
Actually it would be great if mkgmap could write all in one go. Because the thing that takes so much time - is the address search - and that is always the same. The differences are tiny (just because MapInstall is crashing when files are missing) you need to compile them separately.
but maybe there could be a mode where mkgamp writes all in one go. So family-name / family-name1 / family-name2 description / description1 / description2 input input1 input2 family-name.. show-profiles overview-mapname product-id (and maybe I missed some options are those that would need to be given for each set of input tiles. And then just an option where you tell mkgmap files starting with which first 4 numbers are relevant for address search. No need to analyze if those other supplied .img (e.g. buildings or contourlines) need to be added to address search.). I know coded around the problem of the gmap files causing excessive writes. But yeah that is actually really complicated be it on windows or linux...
On Wed, 8 Sept 2021 at 17:07, Felix Hartmann <extremecarver@gmail.com <mailto:extremecarver@gmail.com><mailto:extremecarver@gmail.com<mailto: extremecarver@gmail.com>><mailto:extremecarver@gmail.com<mailto: extremecarver@gmail.com><mailto:extremecarver@gmail.com<mailto: extremecarver@gmail.com>>>> wrote: Well I could give it 20 GB ram disk, maybe 32 but then I need to render on less than 12 processes 64GB ram available). But that is not enough for Asia continent map, and I guess super tight for Europe...
Mkgmap could definitely keep those .tmp files in memory. But the important bit is the gmap files not needeed to be written.... Also would save quite some CPU time.
On Wed, 8 Sep 2021, 14:31 ael <witwall3@disroot.org<mailto: witwall3@disroot.org><mailto:witwall3@disroot.org<mailto: witwall3@disroot.org>><mailto:witwall3@disroot.org<mailto: witwall3@disroot.org><mailto:witwall3@disroot.org<mailto: witwall3@disroot.org>>>> wrote: On Wed, Sep 08, 2021 at 02:10:52PM +0300, Felix Hartmann wrote:
Yes on an nvme disk you barely notice the conversion - it's really quick. BUT it is not needed if you have the files and even more - it burns your NVME SSD disk. +1. I always use an old spinning rust disk when using mkgmap to save ssd write cycles, even without contours and such. It seems to be profligate in its use of disk cycles. I did try using RAM disk, but even with 16GB on a laptop, that was soon exhausted.
ael
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk<mailto: mkgmap-dev@lists.mkgmap.org.uk>><mailto:mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk><mailto: mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk>>> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- Felix Hartman - Openmtbmap.org & VeloMap.org
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk<mailto: mkgmap-dev@lists.mkgmap.org.uk>> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- Felix Hartman - Openmtbmap.org & VeloMap.org
-- Felix Hartman - Openmtbmap.org & VeloMap.org
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- Felix Hartman - Openmtbmap.org & VeloMap.org
-- Felix Hartman - Openmtbmap.org & VeloMap.org
-- Felix Hartman - Openmtbmap.org & VeloMap.org
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- Felix Hartman - Openmtbmap.org & VeloMap.org
Hi Felix, I have no clue how exactly your scripts work, how you manage to reuse *.img files and so on. Also, I want to find a solution that works for all users, not just you. So, I expect that you have one step that compiles frequently changing OSM data and other steps which are used to compile static data like contourlines or DEM. I don't know if you do the latter for each country / continent or once for planet and I don't care as long as it works for you. My understanding is that you have a large collection of *.img files at some stage and that you run mkgmap multiple times with different combinations of those *.img files as input to produce different zip-files with gmapi format or gmapsupp format. I think that's the normal way to do it, so I try to support that way. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Felix Hartmann <extremecarver@gmail.com> Gesendet: Montag, 13. September 2021 12:28 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option I move away the info.xml - and use a different name for the mapset files and then just have mkgmap any file that is input in osm.pbf / osm / o5m format. Gmapi-minimal should not convert any file that is supplied as .img - as it can be assumed that those exist already (if they do not - then create them with --gmapi). That is in my opinion the best approach. So mkgmap does not even try to convert them. Afterwards I distribute a gmapi folder that includes all the data - and by replacing the info.xml you can enable or disable contourlines. For big countries the contourlines would be a separate download anyhow - so the user only needs to download the maps (including mapset files) but not redownload the contourlines. Same principle as in offering the contourlines as a separate gmapsupp.img file. so you have maps.img contourlines10m.img contourlines20m.img buildings.img .... On Mon, 13 Sept 2021 at 13:17, Gerd Petermann <gpetermann_muenchen@hotmail.com<mailto:gpetermann_muenchen@hotmail.com>> wrote: Hi Felix, sorry, the Data Deduplication as implemented in Windows Server would not help here. It works after data was written. And yes, files which are not just copies of the *.img are written as before. My understanding is that you have different product directories in the gmapi folder and that you write protect those files which should be kept. If you have a script to zip the gmapi directory you have to exclude the unwanted folders. Didn't try it. Does it make sense? Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk<mailto:mkgmap-dev-bounces@lists.mkgmap.org.uk>> im Auftrag von Felix Hartmann <extremecarver@gmail.com<mailto:extremecarver@gmail.com>> Gesendet: Montag, 13. September 2021 12:09 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option Oh - but data was certainly written - A rename will not show as data written in both Task manager on Windows, as well as in the smart data (I'm using Windows 10 pro not Windows server however - maybe that functionality is limited to windows server?) On Mon, 13 Sept 2021 at 13:06, Felix Hartmann <extremecarver@gmail.com<mailto:extremecarver@gmail.com><mailto:extremecarver@gmail.com<mailto:extremecarver@gmail.com>>> wrote: Not sure if it makes it possible to use read only attribute instead of moving and mklink. Maybe yes - because that was not possible before. So it then would be set read only - instead of of move & mklink - and at the end remove read only instead of moving back. On Mon, 13 Sept 2021 at 12:59, Felix Hartmann <extremecarver@gmail.com<mailto:extremecarver@gmail.com><mailto:extremecarver@gmail.com<mailto:extremecarver@gmail.com>>> wrote: Thanks Gerd, but that is just removing the warning if it cannot overwrite, correct? If it can overwrite the gmap file it will still overwrite it as I see.. So in essence just a bit more intelligent then my disabling that line. I think it should not overwrite at all if present and --x-gmapi-minimal (then you don't have to move away the files and link them back to the original folder). On Mon, 13 Sept 2021 at 10:43, Gerd Petermann <gpetermann_muenchen@hotmail.com<mailto:gpetermann_muenchen@hotmail.com><mailto:gpetermann_muenchen@hotmail.com<mailto:gpetermann_muenchen@hotmail.com>>> wrote: Hi all, attached is a quick patch that implements experimaltal option --x-gmapi-minimal. If used instead of --gmapi mkgmap will not fail if a write-protected output file exists in the gmapi output folder and mkgmap copies data from *.img. It should still crash when other files like Info.xml are written. BTW: no conversion is done when those files are written. I think mkgmap simply copies data from sub files in *.img into single files. So, the same sequence of Bytes exists two or more times on the Computer. Windows Server seems to support automatic data deduplication (1). I am sure Linux offers similar support. No idea how effective or reliable it is, but it might be worth trying. Gerd (1) https://docs.microsoft.com/en-us/windows-server/storage/data-deduplication/i... ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk<mailto:mkgmap-dev-bounces@lists.mkgmap.org.uk><mailto:mkgmap-dev-bounces@lists.mkgmap.org.uk<mailto:mkgmap-dev-bounces@lists.mkgmap.org.uk>>> im Auftrag von Carlos Dávila <carlos@alternativaslibres.org<mailto:carlos@alternativaslibres.org><mailto:carlos@alternativaslibres.org<mailto:carlos@alternativaslibres.org>>> Gesendet: Sonntag, 12. September 2021 22:45 An: mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk><mailto:mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk>> Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option I think it's a good idea if mkgmap checks the required files are present. El 12/9/21 a las 21:16, Gerd Petermann escribió:
Hi Felix,
so you just want a new option so that mkgmap doesn't fail if it cannot overwrite (write-protected) files in the output directory, right? No need to verify the content of the exiting file(s)?
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk<mailto:mkgmap-dev-bounces@lists.mkgmap.org.uk><mailto:mkgmap-dev-bounces@lists.mkgmap.org.uk<mailto:mkgmap-dev-bounces@lists.mkgmap.org.uk>>> im Auftrag von Felix Hartmann <extremecarver@gmail.com<mailto:extremecarver@gmail.com><mailto:extremecarver@gmail.com<mailto:extremecarver@gmail.com>>> Gesendet: Sonntag, 12. September 2021 19:10 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option
And well - it is burning SSD for the contourlines currently even if not calling multiple times. 130GB minimum per worldwide map update for all geofabrik extracts at 20m equidistance. 260GB at 10m. With calling multiple times and 10m and 20m I had about 2TB of useless writes per weekly map update. I've got rid of all of them with my uncommenting the line, plus saved about 500GB of writes by now doing all the gmapsupp.img and gmap stuff in Ramdisk So now instead of 4TB of disk writes per map update I'm down to 1TB.. (and yeah the resplitting of tiles added another 5TB of writes - but that could have been fixed easily by changing order of commands too).
On Sun, 12 Sept 2021 at 20:04, Felix Hartmann <extremecarver@gmail.com<mailto:extremecarver@gmail.com><mailto:extremecarver@gmail.com<mailto:extremecarver@gmail.com>><mailto:extremecarver@gmail.com<mailto:extremecarver@gmail.com><mailto:extremecarver@gmail.com<mailto:extremecarver@gmail.com>>>> wrote: because you need to do it if you want to show contourlines. Now compiling the worldwide contourlines each time again - would burn SSD as well - besides taking longer than just compiling the maps. So everyone who offers maps for many countries as download puts the contourlines separate. If you want to offer the choice of showing contourlines or not - mkmgap will write the gmap contourlines once not needed, and once the maps not needed. If you don't want to offer that option - it's only the contourlines being written without need. Contourlines for all geofabrik extracts 20m equidistance are about 130GB, 10m would be 260GB. If you offer 10m and 20m it will be even more not needed writes.
The only solution is to go for a Ramdisk instead - However you likely will need 128GB of RAM to do that - because for Asia or Europe you need a 64GB Ramdisk. Same for North America extract (but few people offer that and just have Canada and the US divided into the 4-5 zones). For other maps you will get by with a 15-20GB Ramdisk (which I have resorted to now for all but the windows installers because I don't want to let the server wait for ages for the single thread NSIS wrapping up the data and instead start the next country in parallel). And yes going RAMDISK already using my patch and symlinking those files so mkgmap cannot overwrite them (would still be faster if mkgmap didn't try in first place I think). If you want to write out the contourlines as well besides the additional time - for Asia continent you may then go for a 90GB Ramdisk minimum if offering windows and gmap installers. In this case gmapsupp.img donwnloads aren't possible anyhow due to the huge data amounts. For sm aller countries I have them too....
I coded around this now by using symlinks - but yeah that will be quite a lot of work and is prone to break - while I guess it's only 10 lines or so to add to mkgmap code to have a mode that does not write them out if they are present - or if you tell on commandline they are present already. For .img files they aren't overwritten either...
On Sun, 12 Sept 2021 at 18:40, Gerd Petermann <gpetermann_muenchen@hotmail.com<mailto:gpetermann_muenchen@hotmail.com><mailto:gpetermann_muenchen@hotmail.com<mailto:gpetermann_muenchen@hotmail.com>><mailto:gpetermann_muenchen@hotmail.com<mailto:gpetermann_muenchen@hotmail.com><mailto:gpetermann_muenchen@hotmail.com<mailto:gpetermann_muenchen@hotmail.com>>>> wrote: Hi Felix,
did not read all the posts in detail. I understand that mkgmap is neither burning SSDs nor doing any excessive writing unless you call it multiple times for the same input files. So the question is: Why do you do that?
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk<mailto:mkgmap-dev-bounces@lists.mkgmap.org.uk><mailto:mkgmap-dev-bounces@lists.mkgmap.org.uk<mailto:mkgmap-dev-bounces@lists.mkgmap.org.uk>><mailto:mkgmap-dev-bounces@lists.mkgmap.org.uk<mailto:mkgmap-dev-bounces@lists.mkgmap.org.uk><mailto:mkgmap-dev-bounces@lists.mkgmap.org.uk<mailto:mkgmap-dev-bounces@lists.mkgmap.org.uk>>>> im Auftrag von Felix Hartmann <extremecarver@gmail.com<mailto:extremecarver@gmail.com><mailto:extremecarver@gmail.com<mailto:extremecarver@gmail.com>><mailto:extremecarver@gmail.com<mailto:extremecarver@gmail.com><mailto:extremecarver@gmail.com<mailto:extremecarver@gmail.com>>>> Gesendet: Freitag, 10. September 2021 00:02 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option
Well I think the .tmp files are just building up - and the renamed. So they are not causing any actual excessive write. As for the gmap - it would be cool if there is a mode to not write them.
Actually it would be great if mkgmap could write all in one go. Because the thing that takes so much time - is the address search - and that is always the same. The differences are tiny (just because MapInstall is crashing when files are missing) you need to compile them separately.
but maybe there could be a mode where mkgamp writes all in one go. So family-name / family-name1 / family-name2 description / description1 / description2 input input1 input2 family-name.. show-profiles overview-mapname product-id (and maybe I missed some options are those that would need to be given for each set of input tiles. And then just an option where you tell mkgmap files starting with which first 4 numbers are relevant for address search. No need to analyze if those other supplied .img (e.g. buildings or contourlines) need to be added to address search.). I know coded around the problem of the gmap files causing excessive writes. But yeah that is actually really complicated be it on windows or linux...
On Wed, 8 Sept 2021 at 17:07, Felix Hartmann <extremecarver@gmail.com<mailto:extremecarver@gmail.com><mailto:extremecarver@gmail.com<mailto:extremecarver@gmail.com>><mailto:extremecarver@gmail.com<mailto:extremecarver@gmail.com><mailto:extremecarver@gmail.com<mailto:extremecarver@gmail.com>>><mailto:extremecarver@gmail.com<mailto:extremecarver@gmail.com><mailto:extremecarver@gmail.com<mailto:extremecarver@gmail.com>><mailto:extremecarver@gmail.com<mailto:extremecarver@gmail.com><mailto:extremecarver@gmail.com<mailto:extremecarver@gmail.com>>>>> wrote: Well I could give it 20 GB ram disk, maybe 32 but then I need to render on less than 12 processes 64GB ram available). But that is not enough for Asia continent map, and I guess super tight for Europe...
Mkgmap could definitely keep those .tmp files in memory. But the important bit is the gmap files not needeed to be written.... Also would save quite some CPU time.
On Wed, 8 Sep 2021, 14:31 ael <witwall3@disroot.org<mailto:witwall3@disroot.org><mailto:witwall3@disroot.org<mailto:witwall3@disroot.org>><mailto:witwall3@disroot.org<mailto:witwall3@disroot.org><mailto:witwall3@disroot.org<mailto:witwall3@disroot.org>>><mailto:witwall3@disroot.org<mailto:witwall3@disroot.org><mailto:witwall3@disroot.org<mailto:witwall3@disroot.org>><mailto:witwall3@disroot.org<mailto:witwall3@disroot.org><mailto:witwall3@disroot.org<mailto:witwall3@disroot.org>>>>> wrote: On Wed, Sep 08, 2021 at 02:10:52PM +0300, Felix Hartmann wrote:
Yes on an nvme disk you barely notice the conversion - it's really quick. BUT it is not needed if you have the files and even more - it burns your NVME SSD disk. +1. I always use an old spinning rust disk when using mkgmap to save ssd write cycles, even without contours and such. It seems to be profligate in its use of disk cycles. I did try using RAM disk, but even with 16GB on a laptop, that was soon exhausted.
ael
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk><mailto:mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk>><mailto:mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk><mailto:mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk>>><mailto:mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk><mailto:mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk>><mailto:mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk><mailto:mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk>>>> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- Felix Hartman - Openmtbmap.org & VeloMap.org
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk><mailto:mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk>><mailto:mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk><mailto:mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk>>> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- Felix Hartman - Openmtbmap.org & VeloMap.org
-- Felix Hartman - Openmtbmap.org & VeloMap.org
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk><mailto:mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk>> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk><mailto:mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk>> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk><mailto:mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk>> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev -- Felix Hartman - Openmtbmap.org & VeloMap.org -- Felix Hartman - Openmtbmap.org & VeloMap.org -- Felix Hartman - Openmtbmap.org & VeloMap.org _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev -- Felix Hartman - Openmtbmap.org & VeloMap.org
Hi Gerd, yes exactly. I have a large collection of .img files of contourlines. When I realized that I trash my NVME disk very fast if I continue the way I did I got into thinking and analysing what happens. It was clear that it all comes down to the --gmapi option. I had believed that if I run several passes with .o5m and .img files as input - only the .o5m input is converted into new .gmapi folders - while the existing ones are not overwritten. Then I checked the created date/last modified date (because windows has in my eyes a bug that if you replace a file with a near identical file - it just shows a new modified date - but keeps the original created date - even though it was a full overwrite). That got me thinking that in order to save writes - I have to find a way to not only not recalculate the .img files - but also create a static set of .gmapi folders. Those I just mklink into the directory name that will be used on the next run of mkgmap.jar - I managed to do this by uncommenting this one line. Because I noticed that the symlinked (by mklink) files are not rewritten I changed my scripts to move them away and symlink them back. Then at the end delete all symlinks - and move the files back (or to the location that I will use for compressing). This step is a bit stupid if mkgmap could just have a --gmapi-minimal mode in which only those files are converted - that are also written out as .img files (if given --tdb-file option). I know that many people keep a static set of contourlines .img files (also containing DEM). You just add the show-profiles=1 option in case you include contourlines - and leave it out if not. But actually it does not matter if the contourlines files contain DEM or not. *I think the easiest way is the principle - --gmapi-minimal only converts those files, it would write out as .img files if --tdb-files option were given (or is given). --gmapi on the other hand should convert the all input files to .gmapi format.* Then mgkmap does not even need to test if those files are there or not. This not only saves a lot of writes - but also a lot of compile time. Because essentially if you only provide .img input files (including of course the ovm-img for the overview map if you want) you only create a new set of mapset files. The exception to this is the toolchain in which when a tile failed to compile - you resplit the input files - and parse them again with the same arguments so you have input of new o5m files and old .img files. But the principle stays the same. If on the rerun of the failed tiles now newy split with higher map-id you give --gmapi-minimal and tdb-file - only those new tiles are written - while the old .img and old ovm.img are supplied to create the correct mapset files. With this procedure you don't even need to put a symlink for the contourlines into the gmap folder. As the input data to create the contourlines rarely change - you can offer the contourlines as a separate download. Makes it much easier and faster. On the map compilation server you then do not even need to have a copy of the .gmapi contourlines files. You just need the new input data and the static contourlines .img files. Thomas Morgenstern for example had also not realized that he is writing the contourlines .gmapi files each time without any need. I think many/most providers of garmin maps who offer many regions/worldwide coverage put the contourlines separate. It's just a huge amount of compile time if you merge the contourlines in o5m format with the map data in o5m data each time before running mkgmap. The only actual advantage of this being that the contourlines do not overlap roads (as they are supplied as transparent .img files in another layer) - AND that when people select maps with a single click instead of drag in MapInstall/Mapsource they get all maps they think they get (though there is a different shading - each layer increases the darkness of the shading for selected). And of course that the very old Mapsource still has problems correctly showing the contourlines if they are in a separate layer. However nearly everyone moved on to Basecamp which is fully compatible with layered maps. The advantage of contourlines as separate layer is for the user he can switch them on / off independant of the maps and does not need to download and transfer them each time. So I think for most the advantages heavily outweigh the disadvantages - hence contourlines into a separate transparent layer. Same can be done of course for other things like buildings or vegetation - though the advantage here is much less on the compile side (while worldwide 10m coontourlines are 200GB of data - the same extracts are only 15GB in data of buildings - if buildings are shown only at resolution 24) On Mon, 13 Sept 2021 at 14:24, Gerd Petermann < gpetermann_muenchen@hotmail.com> wrote:
Hi Felix,
I have no clue how exactly your scripts work, how you manage to reuse *.img files and so on. Also, I want to find a solution that works for all users, not just you.
So, I expect that you have one step that compiles frequently changing OSM data and other steps which are used to compile static data like contourlines or DEM. I don't know if you do the latter for each country / continent or once for planet and I don't care as long as it works for you. My understanding is that you have a large collection of *.img files at some stage and that you run mkgmap multiple times with different combinations of those *.img files as input to produce different zip-files with gmapi format or gmapsupp format. I think that's the normal way to do it, so I try to support that way.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Felix Hartmann <extremecarver@gmail.com> Gesendet: Montag, 13. September 2021 12:28 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option
I move away the info.xml - and use a different name for the mapset files and then just have mkgmap any file that is input in osm.pbf / osm / o5m format. Gmapi-minimal should not convert any file that is supplied as .img - as it can be assumed that those exist already (if they do not - then create them with --gmapi). That is in my opinion the best approach. So mkgmap does not even try to convert them.
Afterwards I distribute a gmapi folder that includes all the data - and by replacing the info.xml you can enable or disable contourlines. For big countries the contourlines would be a separate download anyhow - so the user only needs to download the maps (including mapset files) but not redownload the contourlines. Same principle as in offering the contourlines as a separate gmapsupp.img file. so you have maps.img contourlines10m.img contourlines20m.img buildings.img ....
On Mon, 13 Sept 2021 at 13:17, Gerd Petermann < gpetermann_muenchen@hotmail.com<mailto:gpetermann_muenchen@hotmail.com>> wrote: Hi Felix,
sorry, the Data Deduplication as implemented in Windows Server would not help here. It works after data was written.
And yes, files which are not just copies of the *.img are written as before. My understanding is that you have different product directories in the gmapi folder and that you write protect those files which should be kept. If you have a script to zip the gmapi directory you have to exclude the unwanted folders. Didn't try it. Does it make sense?
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk<mailto: mkgmap-dev-bounces@lists.mkgmap.org.uk>> im Auftrag von Felix Hartmann < extremecarver@gmail.com<mailto:extremecarver@gmail.com>> Gesendet: Montag, 13. September 2021 12:09 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option
Oh - but data was certainly written - A rename will not show as data written in both Task manager on Windows, as well as in the smart data (I'm using Windows 10 pro not Windows server however - maybe that functionality is limited to windows server?)
On Mon, 13 Sept 2021 at 13:06, Felix Hartmann <extremecarver@gmail.com <mailto:extremecarver@gmail.com><mailto:extremecarver@gmail.com<mailto: extremecarver@gmail.com>>> wrote: Not sure if it makes it possible to use read only attribute instead of moving and mklink. Maybe yes - because that was not possible before. So it then would be set read only - instead of of move & mklink - and at the end remove read only instead of moving back.
On Mon, 13 Sept 2021 at 12:59, Felix Hartmann <extremecarver@gmail.com <mailto:extremecarver@gmail.com><mailto:extremecarver@gmail.com<mailto: extremecarver@gmail.com>>> wrote: Thanks Gerd,
but that is just removing the warning if it cannot overwrite, correct? If it can overwrite the gmap file it will still overwrite it as I see.. So in essence just a bit more intelligent then my disabling that line.
I think it should not overwrite at all if present and --x-gmapi-minimal (then you don't have to move away the files and link them back to the original folder).
On Mon, 13 Sept 2021 at 10:43, Gerd Petermann < gpetermann_muenchen@hotmail.com<mailto:gpetermann_muenchen@hotmail.com
<mailto:gpetermann_muenchen@hotmail.com<mailto: gpetermann_muenchen@hotmail.com>>> wrote: Hi all,
attached is a quick patch that implements experimaltal option --x-gmapi-minimal.
If used instead of --gmapi mkgmap will not fail if a write-protected output file exists in the gmapi output folder and mkgmap copies data from *.img. It should still crash when other files like Info.xml are written.
BTW: no conversion is done when those files are written. I think mkgmap simply copies data from sub files in *.img into single files. So, the same sequence of Bytes exists two or more times on the Computer. Windows Server seems to support automatic data deduplication (1). I am sure Linux offers similar support. No idea how effective or reliable it is, but it might be worth trying.
Gerd (1)
https://docs.microsoft.com/en-us/windows-server/storage/data-deduplication/i...
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk<mailto: mkgmap-dev-bounces@lists.mkgmap.org.uk><mailto: mkgmap-dev-bounces@lists.mkgmap.org.uk<mailto: mkgmap-dev-bounces@lists.mkgmap.org.uk>>> im Auftrag von Carlos Dávila < carlos@alternativaslibres.org<mailto:carlos@alternativaslibres.org
<mailto:carlos@alternativaslibres.org<mailto: carlos@alternativaslibres.org>>> Gesendet: Sonntag, 12. September 2021 22:45 An: mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk<mailto: mkgmap-dev@lists.mkgmap.org.uk>> Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option
I think it's a good idea if mkgmap checks the required files are present.
El 12/9/21 a las 21:16, Gerd Petermann escribió:
Hi Felix,
so you just want a new option so that mkgmap doesn't fail if it cannot overwrite (write-protected) files in the output directory, right? No need to verify the content of the exiting file(s)?
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk<mailto: mkgmap-dev-bounces@lists.mkgmap.org.uk><mailto: mkgmap-dev-bounces@lists.mkgmap.org.uk<mailto: mkgmap-dev-bounces@lists.mkgmap.org.uk>>> im Auftrag von Felix Hartmann < extremecarver@gmail.com<mailto:extremecarver@gmail.com><mailto: extremecarver@gmail.com<mailto:extremecarver@gmail.com>>> Gesendet: Sonntag, 12. September 2021 19:10 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option
And well - it is burning SSD for the contourlines currently even if not calling multiple times. 130GB minimum per worldwide map update for all geofabrik extracts at 20m equidistance. 260GB at 10m. With calling multiple times and 10m and 20m I had about 2TB of useless writes per weekly map update. I've got rid of all of them with my uncommenting the line, plus saved about 500GB of writes by now doing all the gmapsupp.img and gmap stuff in Ramdisk So now instead of 4TB of disk writes per map update I'm down to 1TB.. (and yeah the resplitting of tiles added another 5TB of writes - but that could have been fixed easily by changing order of commands too).
On Sun, 12 Sept 2021 at 20:04, Felix Hartmann <extremecarver@gmail.com <mailto:extremecarver@gmail.com><mailto:extremecarver@gmail.com<mailto: extremecarver@gmail.com>><mailto:extremecarver@gmail.com<mailto: extremecarver@gmail.com><mailto:extremecarver@gmail.com<mailto: extremecarver@gmail.com>>>> wrote: because you need to do it if you want to show contourlines. Now compiling the worldwide contourlines each time again - would burn SSD as well - besides taking longer than just compiling the maps. So everyone who offers maps for many countries as download puts the contourlines separate. If you want to offer the choice of showing contourlines or not - mkmgap will write the gmap contourlines once not needed, and once the maps not needed. If you don't want to offer that option - it's only the contourlines being written without need. Contourlines for all geofabrik extracts 20m equidistance are about 130GB, 10m would be 260GB. If you offer 10m and 20m it will be even more not needed writes.
The only solution is to go for a Ramdisk instead - However you likely will need 128GB of RAM to do that - because for Asia or Europe you need a 64GB Ramdisk. Same for North America extract (but few people offer that and just have Canada and the US divided into the 4-5 zones). For other maps you will get by with a 15-20GB Ramdisk (which I have resorted to now for all but the windows installers because I don't want to let the server wait for ages for the single thread NSIS wrapping up the data and instead start the next country in parallel). And yes going RAMDISK already using my patch and symlinking those files so mkgmap cannot overwrite them (would still be faster if mkgmap didn't try in first place I think). If you want to write out the contourlines as well besides the additional time - for Asia continent you may then go for a 90GB Ramdisk minimum if offering windows and gmap installers. In this case gmapsupp.img donwnloads aren't possible anyhow due to the huge data amounts. For sm aller countries I have them too....
I coded around this now by using symlinks - but yeah that will be quite a lot of work and is prone to break - while I guess it's only 10 lines or so to add to mkgmap code to have a mode that does not write them out if they are present - or if you tell on commandline they are present already. For .img files they aren't overwritten either...
On Sun, 12 Sept 2021 at 18:40, Gerd Petermann < gpetermann_muenchen@hotmail.com<mailto:gpetermann_muenchen@hotmail.com <mailto:gpetermann_muenchen@hotmail.com<mailto: gpetermann_muenchen@hotmail.com>><mailto:gpetermann_muenchen@hotmail.com <mailto:gpetermann_muenchen@hotmail.com><mailto: gpetermann_muenchen@hotmail.com<mailto:gpetermann_muenchen@hotmail.com>>>> wrote: Hi Felix,
did not read all the posts in detail. I understand that mkgmap is neither burning SSDs nor doing any excessive writing unless you call it multiple times for the same input files. So the question is: Why do you do that?
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk<mailto: mkgmap-dev-bounces@lists.mkgmap.org.uk><mailto: mkgmap-dev-bounces@lists.mkgmap.org.uk<mailto: mkgmap-dev-bounces@lists.mkgmap.org.uk>><mailto: mkgmap-dev-bounces@lists.mkgmap.org.uk<mailto: mkgmap-dev-bounces@lists.mkgmap.org.uk><mailto: mkgmap-dev-bounces@lists.mkgmap.org.uk<mailto: mkgmap-dev-bounces@lists.mkgmap.org.uk>>>> im Auftrag von Felix Hartmann < extremecarver@gmail.com<mailto:extremecarver@gmail.com><mailto: extremecarver@gmail.com<mailto:extremecarver@gmail.com>><mailto: extremecarver@gmail.com<mailto:extremecarver@gmail.com><mailto: extremecarver@gmail.com<mailto:extremecarver@gmail.com>>>> Gesendet: Freitag, 10. September 2021 00:02 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option
Well I think the .tmp files are just building up - and the renamed. So they are not causing any actual excessive write. As for the gmap - it would be cool if there is a mode to not write them.
Actually it would be great if mkgmap could write all in one go. Because the thing that takes so much time - is the address search - and that is always the same. The differences are tiny (just because MapInstall is crashing when files are missing) you need to compile them separately.
but maybe there could be a mode where mkgamp writes all in one go. So family-name / family-name1 / family-name2 description / description1 / description2 input input1 input2 family-name.. show-profiles overview-mapname product-id (and maybe I missed some options are those that would need to be given for each set of input tiles. And then just an option where you tell mkgmap files starting with which first 4 numbers are relevant for address search. No need to analyze if those other supplied .img (e.g. buildings or contourlines) need to be added to address search.). I know coded around the problem of the gmap files causing excessive writes. But yeah that is actually really complicated be it on windows or linux...
On Wed, 8 Sept 2021 at 17:07, Felix Hartmann <extremecarver@gmail.com <mailto:extremecarver@gmail.com><mailto:extremecarver@gmail.com<mailto: extremecarver@gmail.com>><mailto:extremecarver@gmail.com<mailto: extremecarver@gmail.com><mailto:extremecarver@gmail.com<mailto: extremecarver@gmail.com>>><mailto:extremecarver@gmail.com<mailto: extremecarver@gmail.com><mailto:extremecarver@gmail.com<mailto: extremecarver@gmail.com>><mailto:extremecarver@gmail.com<mailto: extremecarver@gmail.com><mailto:extremecarver@gmail.com<mailto: extremecarver@gmail.com>>>>> wrote: Well I could give it 20 GB ram disk, maybe 32 but then I need to render on less than 12 processes 64GB ram available). But that is not enough for Asia continent map, and I guess super tight for Europe...
Mkgmap could definitely keep those .tmp files in memory. But the important bit is the gmap files not needeed to be written.... Also would save quite some CPU time.
On Wed, 8 Sep 2021, 14:31 ael <witwall3@disroot.org<mailto: witwall3@disroot.org><mailto:witwall3@disroot.org<mailto: witwall3@disroot.org>><mailto:witwall3@disroot.org<mailto: witwall3@disroot.org><mailto:witwall3@disroot.org<mailto: witwall3@disroot.org>>><mailto:witwall3@disroot.org<mailto: witwall3@disroot.org><mailto:witwall3@disroot.org<mailto: witwall3@disroot.org>><mailto:witwall3@disroot.org<mailto: witwall3@disroot.org><mailto:witwall3@disroot.org<mailto: witwall3@disroot.org>>>>> wrote: On Wed, Sep 08, 2021 at 02:10:52PM +0300, Felix Hartmann wrote:
Yes on an nvme disk you barely notice the conversion - it's really quick. BUT it is not needed if you have the files and even more - it burns your NVME SSD disk. +1. I always use an old spinning rust disk when using mkgmap to save ssd write cycles, even without contours and such. It seems to be profligate in its use of disk cycles. I did try using RAM disk, but even with 16GB on a laptop, that was soon exhausted.
ael
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk<mailto: mkgmap-dev@lists.mkgmap.org.uk>><mailto:mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk><mailto: mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk
<mailto:mkgmap-dev@lists.mkgmap.org.uk<mailto: mkgmap-dev@lists.mkgmap.org.uk><mailto:mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk>><mailto: mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk<mailto: mkgmap-dev@lists.mkgmap.org.uk>>>> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- Felix Hartman - Openmtbmap.org & VeloMap.org
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk<mailto: mkgmap-dev@lists.mkgmap.org.uk>><mailto:mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk><mailto: mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk>>> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- Felix Hartman - Openmtbmap.org & VeloMap.org
-- Felix Hartman - Openmtbmap.org & VeloMap.org
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk<mailto: mkgmap-dev@lists.mkgmap.org.uk>> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk
<mailto:mkgmap-dev@lists.mkgmap.org.uk<mailto: mkgmap-dev@lists.mkgmap.org.uk>> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk
<mailto:mkgmap-dev@lists.mkgmap.org.uk<mailto: mkgmap-dev@lists.mkgmap.org.uk>> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- Felix Hartman - Openmtbmap.org & VeloMap.org
-- Felix Hartman - Openmtbmap.org & VeloMap.org
-- Felix Hartman - Openmtbmap.org & VeloMap.org
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- Felix Hartman - Openmtbmap.org & VeloMap.org
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- Felix Hartman - Openmtbmap.org & VeloMap.org
I share the same issue exposed by Felix, but not only for contour lines but also for DEM, which I have precompiled as *.img for many countries and just combine with the updated OSM tiles. El 12/9/21 a las 19:10, Felix Hartmann escribió:
And well - it is burning SSD for the contourlines currently even if not calling multiple times. 130GB minimum per worldwide map update for all geofabrik extracts at 20m equidistance. 260GB at 10m. With calling multiple times and 10m and 20m I had about 2TB of useless writes per weekly map update. I've got rid of all of them with my uncommenting the line, plus saved about 500GB of writes by now doing all the gmapsupp.img and gmap stuff in Ramdisk So now instead of 4TB of disk writes per map update I'm down to 1TB.. (and yeah the resplitting of tiles added another 5TB of writes - but that could have been fixed easily by changing order of commands too).
On Sun, 12 Sept 2021 at 20:04, Felix Hartmann <extremecarver@gmail.com <mailto:extremecarver@gmail.com>> wrote:
because you need to do it if you want to show contourlines. Now compiling the worldwide contourlines each time again - would burn SSD as well - besides taking longer than just compiling the maps. So everyone who offers maps for many countries as download puts the contourlines separate. If you want to offer the choice of showing contourlines or not - mkmgap will write the gmap contourlines once not needed, and once the maps not needed. If you don't want to offer that option - it's only the contourlines being written without need. Contourlines for all geofabrik extracts 20m equidistance are about 130GB, 10m would be 260GB. If you offer 10m and 20m it will be even more not needed writes.
The only solution is to go for a Ramdisk instead - However you likely will need 128GB of RAM to do that - because for Asia or Europe you need a 64GB Ramdisk. Same for North America extract (but few people offer that and just have Canada and the US divided into the 4-5 zones). For other maps you will get by with a 15-20GB Ramdisk (which I have resorted to now for all but the windows installers because I don't want to let the server wait for ages for the single thread NSIS wrapping up the data and instead start the next country in parallel). And yes going RAMDISK already using my patch and symlinking those files so mkgmap cannot overwrite them (would still be faster if mkgmap didn't try in first place I think). If you want to write out the contourlines as well besides the additional time - for Asia continent you may then go for a 90GB Ramdisk minimum if offering windows and gmap installers. In this case gmapsupp.img donwnloads aren't possible anyhow due to the huge data amounts. For smaller countries I have them too....
I coded around this now by using symlinks - but yeah that will be quite a lot of work and is prone to break - while I guess it's only 10 lines or so to add to mkgmap code to have a mode that does not write them out if they are present - or if you tell on commandline they are present already. For .img files they aren't overwritten either...
On Sun, 12 Sept 2021 at 18:40, Gerd Petermann <gpetermann_muenchen@hotmail.com <mailto:gpetermann_muenchen@hotmail.com>> wrote:
Hi Felix,
did not read all the posts in detail. I understand that mkgmap is neither burning SSDs nor doing any excessive writing unless you call it multiple times for the same input files. So the question is: Why do you do that?
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk <mailto:mkgmap-dev-bounces@lists.mkgmap.org.uk>> im Auftrag von Felix Hartmann <extremecarver@gmail.com <mailto:extremecarver@gmail.com>> Gesendet: Freitag, 10. September 2021 00:02 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option
Well I think the .tmp files are just building up - and the renamed. So they are not causing any actual excessive write. As for the gmap - it would be cool if there is a mode to not write them.
Actually it would be great if mkgmap could write all in one go. Because the thing that takes so much time - is the address search - and that is always the same. The differences are tiny (just because MapInstall is crashing when files are missing) you need to compile them separately.
but maybe there could be a mode where mkgamp writes all in one go. So family-name / family-name1 / family-name2 description / description1 / description2 input input1 input2 family-name.. show-profiles overview-mapname product-id (and maybe I missed some options are those that would need to be given for each set of input tiles. And then just an option where you tell mkgmap files starting with which first 4 numbers are relevant for address search. No need to analyze if those other supplied .img (e.g. buildings or contourlines) need to be added to address search.). I know coded around the problem of the gmap files causing excessive writes. But yeah that is actually really complicated be it on windows or linux...
On Wed, 8 Sept 2021 at 17:07, Felix Hartmann <extremecarver@gmail.com <mailto:extremecarver@gmail.com><mailto:extremecarver@gmail.com <mailto:extremecarver@gmail.com>>> wrote: Well I could give it 20 GB ram disk, maybe 32 but then I need to render on less than 12 processes 64GB ram available). But that is not enough for Asia continent map, and I guess super tight for Europe...
Mkgmap could definitely keep those .tmp files in memory. But the important bit is the gmap files not needeed to be written.... Also would save quite some CPU time.
On Wed, 8 Sep 2021, 14:31 ael <witwall3@disroot.org <mailto:witwall3@disroot.org><mailto:witwall3@disroot.org <mailto:witwall3@disroot.org>>> wrote: On Wed, Sep 08, 2021 at 02:10:52PM +0300, Felix Hartmann wrote: > > Yes on an nvme disk you barely notice the conversion - it's really quick. > BUT it is not needed if you have the files and even more - it burns your > NVME SSD disk.
+1. I always use an old spinning rust disk when using mkgmap to save ssd write cycles, even without contours and such. It seems to be profligate in its use of disk cycles. I did try using RAM disk, but even with 16GB on a laptop, that was soon exhausted.
ael
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk><mailto:mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk>> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev <https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev>
-- Felix Hartman - Openmtbmap.org & VeloMap.org
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev <https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev>
-- Felix Hartman - Openmtbmap.org & VeloMap.org
-- Felix Hartman - Openmtbmap.org & VeloMap.org
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi ael,
It seems to be profligate in its use of disk cycles. Any further prove for this despite the gmapi directory? While I do agree that the whole tool chain (download complete *.osm.pbf, maybe convert to *.o5m, maybe mix with contour data, splitter, and finally compile with mkgmap) is very I/O intensive I really don't see much room for improvements in mgmap. The program keeps most of the data in memory before writing the final result, only data for the global index files which are likely to grow very large are written to tempory files.
Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von ael <witwall3@disroot.org> Gesendet: Mittwoch, 8. September 2021 13:31 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option On Wed, Sep 08, 2021 at 02:10:52PM +0300, Felix Hartmann wrote:
Yes on an nvme disk you barely notice the conversion - it's really quick. BUT it is not needed if you have the files and even more - it burns your NVME SSD disk.
+1. I always use an old spinning rust disk when using mkgmap to save ssd write cycles, even without contours and such. It seems to be profligate in its use of disk cycles. I did try using RAM disk, but even with 16GB on a laptop, that was soon exhausted. ael _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
first of all I need all info.xml files - because I want different versions of a map possible. And yes second those files are tiny - just rename/Move them away before running mkgmap again. That is trivial on any system. (much easier than moving away all those folder and linking them back was was needed before this patch). On Sun, 19 Sept 2021 at 12:39, Gerd Petermann < gpetermann_muenchen@hotmail.com> wrote:
Hi ael,
It seems to be profligate in its use of disk cycles. Any further prove for this despite the gmapi directory? While I do agree that the whole tool chain (download complete *.osm.pbf, maybe convert to *.o5m, maybe mix with contour data, splitter, and finally compile with mkgmap) is very I/O intensive I really don't see much room for improvements in mgmap. The program keeps most of the data in memory before writing the final result, only data for the global index files which are likely to grow very large are written to tempory files.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von ael <witwall3@disroot.org> Gesendet: Mittwoch, 8. September 2021 13:31 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option
On Wed, Sep 08, 2021 at 02:10:52PM +0300, Felix Hartmann wrote:
Yes on an nvme disk you barely notice the conversion - it's really quick. BUT it is not needed if you have the files and even more - it burns your NVME SSD disk.
+1. I always use an old spinning rust disk when using mkgmap to save ssd write cycles, even without contours and such. It seems to be profligate in its use of disk cycles. I did try using RAM disk, but even with 16GB on a laptop, that was soon exhausted.
ael
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- Felix Hartman - Openmtbmap.org & VeloMap.org
The current version is perfect. Avoids all those unnecessary writes and no need to go crazy on symbolic links anymore for preventing them. Files that should be overwritten - are overwritten. On Sun, 19 Sept 2021 at 15:03, Felix Hartmann <extremecarver@gmail.com> wrote:
first of all I need all info.xml files - because I want different versions of a map possible. And yes second those files are tiny - just rename/Move them away before running mkgmap again. That is trivial on any system. (much easier than moving away all those folder and linking them back was was needed before this patch).
On Sun, 19 Sept 2021 at 12:39, Gerd Petermann < gpetermann_muenchen@hotmail.com> wrote:
Hi ael,
It seems to be profligate in its use of disk cycles. Any further prove for this despite the gmapi directory? While I do agree that the whole tool chain (download complete *.osm.pbf, maybe convert to *.o5m, maybe mix with contour data, splitter, and finally compile with mkgmap) is very I/O intensive I really don't see much room for improvements in mgmap. The program keeps most of the data in memory before writing the final result, only data for the global index files which are likely to grow very large are written to tempory files.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von ael <witwall3@disroot.org> Gesendet: Mittwoch, 8. September 2021 13:31 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to disk if used with --gmapi option
On Wed, Sep 08, 2021 at 02:10:52PM +0300, Felix Hartmann wrote:
Yes on an nvme disk you barely notice the conversion - it's really
quick.
BUT it is not needed if you have the files and even more - it burns your NVME SSD disk.
+1. I always use an old spinning rust disk when using mkgmap to save ssd write cycles, even without contours and such. It seems to be profligate in its use of disk cycles. I did try using RAM disk, but even with 16GB on a laptop, that was soon exhausted.
ael
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- Felix Hartman - Openmtbmap.org & VeloMap.org
-- Felix Hartman - Openmtbmap.org & VeloMap.org
Uncommenting line 102 from /combiners/gmapibuilder.java and then messing by making folder read only works (even though this is really like using crutches while being healthy)! And yeah it's great mkgmap code is so cleanly written that even someone at a loss with java can write a workaround. I really just believe no-one ever really noticed or minded enough to write here. There was a lot of optimization in making mkgmap faster or use less memory - but no developper so far cared for hdd/ssd writes. Actually I am pretty sure not writing those temp files or attempting to write the gmap files if they exist will be an easy speed up for mkgmap. Thanks Gerd and everyone else of the developper for writing such great software" as in the following example.
} catch (IOException e) { // throw new ExitException("Error saving gmapi data", e); }
So let's detail the writes on Alps step by step - I guess the best solution is to use a Ram Disk if mkgmap cannot be optimized - and then only write Asia and Europe Continent maps to NVME disk because I cannot create a big enough Ram Disk for them.... So the log is only in full GB - so here it what happens: start at: 63648GB written to C: *1.* 7*.img are the 20m contourlines. They and the 9*.img contourlines are already present in gmap format and in the relavant gmap folder.... mkgmap.jar is patched as above in order to not stop when it cannot write the 7* and 9* folders as they are read only. I'm thinking of making the 6* folder read only for the further steps? start /belownormal /b /wait java -jar -XX:+AggressiveHeap -XX:StringTableSize=1000003 -Xmx43000m C:\openmtbmap\mkgmap.jar --max-jobs=12 --order-by-decreasing-area "--generate-sea" --code-page=1252 "--precomp-sea=E:\OpenMTBMap\osmpbf_geofabrik\sea.zip" "--style-file=C:\openmtbmap\openmtbmap_style" --add-boundary-nodes-at-admin-boundaries --fix-roundabout-direction --improve-overview --drive-on=detect --allow-reverse-merge --line-types-with-direction=0x18,0x1f,0x10016,0x10101,0x10105,0x10401,0x10403,0x10405,0x10408,0x10409,0x1040b,0x1040d,0x1040e,0x10500,0x10501,0x10502,0x10503,0x10504,0x10505,0x10506,0x10507,0x10607,0x10609,0x1060b,0x10702,0x10703,0x10704,0x10705,0x10706,0x10707,0x10a00,0x10a02,0x10a04,0x10a05,0x10e1e,0x10e0f,0x10e0e,0x10e1e,0x10f0a,0x10f1c,0x10e1d --nsis --index --levels="0:24, 1:23, 2:22, 3:21, 4:20, 5:19, 6:18" --overview-levels="7:17, 8:16, 9:15, 10:14, 11:13" --add-pois-to-areas --pois-to-areas-placement=entrance=main;entrance=yes;building=entrance;barrier=entrance --simplify-lines=23:2.6,22:4.2,21:5.4,20:6,19:7,18:7.5,17:4.5,16:5,15:5,14:6 --simplify-polygons=23:3.6,22:7,21:6,20:9,17:5.4 --add-boundary-nodes-at-admin-boundaries=2 --poi-excl-index=0x6405,0x4316,0x2f00 --copyright-file="C:\openmtbmap\openmtbmap_svn\copyrightopm.txt" --license-file="C:\openmtbmap\openmtbmap_svn\licenseopm.txt" --cycle-map --ignore-fixme-values --housenumbers --road-name-config=C:\openmtbmap\openmtbmap_svn\roadNameConfig.txt --split-name-index --link-pois-to-ways --ignore-turn-restrictions --polygon-size-limits="24:12, 23:14, 22:14, 21:20, 20:20, 19:20, 18:20, 17:20, 16:20, 15:20, 14:20, 13:25" --description=omtb_alp --show-profiles=1 --location-autofill=bounds,is_in,nearest --bounds=E:\OpenMTBMap\osmpbf_geofabrik\bounds.zip --route --country-abbr=alp --country-name=alps --mapname=65280000 --family-id=6528 --product-id=1 --series-name=omtb_alps_08.09.2021 --family-name=mtb_alp_08.09.2021 --tdbfile --gmapi --overview-mapname=mapsetc --keep-going --area-name="alps_08.09.2021_omtb" -c D:\openmtbmap\maps\template.alps E:\OpenMTBMap\contourlines20\alps\7*.img typalp.TYP 1>NUL Mkgmap version 4806M Time started: Wed Sep 08 19:44:09 CEST 2021 ... Time taken 10 minutes Now written 63652GB While it has actually written 2.12GB of .img files and 2.1GB for mac OSx files. Everything going pretty smooth here. Yes there have been some temporary files which I do not see why they are written if the max memory used was 25GB with 43GB heap (and never less than 37GB of RAM available) *2.* C:\openmtbmap\maps>start /belownormal /b /wait java -jar -XX:+AggressiveHeap -XX:StringTableSize=1000003 -Xmx43000m C:\openmtbmap\mkgmap.jar --max-jobs=12 "--generate-sea" --code-page=1252 "--precomp-sea=E:\OpenMTBMap\osmpbf_geofabrik\sea.zip" "--style-file=C:\openmtbmap\openmtbmap_style" --add-boundary-nodes-at-admin-boundaries --fix-roundabout-direction --improve-overview --drive-on=detect --allow-reverse-merge --line-types-with-direction=0x18,0x1f,0x10016,0x10101,0x10105,0x10401,0x10403,0x10405,0x10408,0x10409,0x1040b,0x1040d,0x1040e,0x10500,0x10501,0x10502,0x10503,0x10504,0x10505,0x10506,0x10507,0x10607,0x10609,0x1060b,0x10702,0x10703,0x10704,0x10705,0x10706,0x10707,0x10a00,0x10a02,0x10a04,0x10a05,0x10e1e,0x10e0f,0x10e0e,0x10e1e,0x10f0a,0x10f1c,0x10e1d --nsis --index --levels="0:24, 1:23, 2:22, 3:21, 4:20, 5:19, 6:18" --overview-levels="7:17, 8:16, 9:15, 10:14, 11:13" --add-pois-to-areas --pois-to-areas-placement=entrance=main;entrance=yes;building=entrance;barrier=entrance --road-name-config=C:\openmtbmap\openmtbmap_svn\roadNameConfig.txt --split-name-index --housenumbers --description=omtb_alp --show-profiles=0 --location-autofill=bounds,is_in,nearest --bounds=E:\OpenMTBMap\osmpbf_geofabrik\bounds.zip --route --poi-excl-index=0x6405,0x4316,0x2f00 --cycle-map --copyright-file="C:\openmtbmap\openmtbmap_svn\copyrightopm.txt" --license-file="C:\openmtbmap\openmtbmap_svn\licenseopm.txt" --country-abbr=alp --country-name=alps --mapname=65280000 --family-id=6528 --product-id=1 --series-name=omtb_alps_08.09.2021 --family-name=mtb_alp_08.09.2021 --tdbfile --gmapi --overview-mapname=mapsetx --keep-going --area-name="alps_08.09.2021_omtb" 6528*.img typalp.TYP 1>NUL 2>NUL Time taken - 30 seconds or so Written 460MB (twice 230MB for each mapset_mdr file) However new counter: 63655GB Waste - minimum 2GB. maybe 3GB.. 3. C:\openmtbmap\maps>start /belownormal /b /wait java -jar -XX:+AggressiveHeap -XX:StringTableSize=1000003 -Xmx43000m C:\openmtbmap\mkgmap.jar --max-jobs=12 "--generate-sea" --code-page=1252 "--precomp-sea=E:\OpenMTBMap\osmpbf_geofabrik\sea.zip" "--style-file=C:\openmtbmap\openmtbmap_style" --add-boundary-nodes-at-admin-boundaries --fix-roundabout-direction --improve-overview --drive-on=detect --allow-reverse-merge --line-types-with-direction=0x18,0x1f,0x10016,0x10101,0x10105,0x10401,0x10403,0x10405,0x10408,0x10409,0x1040b,0x1040d,0x1040e,0x10500,0x10501,0x10502,0x10503,0x10504,0x10505,0x10506,0x10507,0x10607,0x10609,0x1060b,0x10702,0x10703,0x10704,0x10705,0x10706,0x10707,0x10a00,0x10a02,0x10a04,0x10a05,0x10e1e,0x10e0f,0x10e0e,0x10e1e,0x10f0a,0x10f1c,0x10e1d --nsis --index --levels="0:24, 1:23, 2:22, 3:21, 4:20, 5:19, 6:18" --overview-levels="7:17, 8:16, 9:15, 10:14, 11:13" --add-pois-to-areas --pois-to-areas-placement=entrance=main;entrance=yes;building=entrance;barrier=entrance --road-name-config=C:\openmtbmap\openmtbmap_svn\roadNameConfig.txt --split-name-index --housenumbers --description=omtb_alp --show-profiles=1 --location-autofill=bounds,is_in,nearest --bounds=E:\OpenMTBMap\osmpbf_geofabrik\bounds.zip --route --poi-excl-index=0x6405,0x4316,0x2f00 --cycle-map --copyright-file="C:\openmtbmap\openmtbmap_svn\copyrightopm.txt" --license-file="C:\openmtbmap\openmtbmap_svn\licenseopm.txt" --country-abbr=alp --country-name=alps --mapname=65280000 --family-id=6528 --product-id=1 --series-name=omtb_alps_08.09.2021 --family-name=mtb_alp_08.09.2021 --tdbfile --gmapi --overview-mapname=mapset10 --keep-going --area-name="alps_08.09.2021_omtb" 6528*.img typalp.TYP E:\OpenMTBMap\contourlines10\alps\9*.img 1>NUL 2>NUL Time taken 1 minute or so Written 460M (same as above) However new counter: 33657 GB (I guess the waste is the same as above - it cannot write the 9*.img - so each time around 2GB of wasted Write commands) 4. Let's create the gmapsupp.img - there is no more wasted writes here start /belownormal /b /wait java -jar -XX:+AggressiveHeap -XX:StringTableSize=1000003 -Xmx43000m C:\openmtbmap\mkgmap.jar --max-jobs=12 --hide-gmapsupp-on-pc --copyright-file="C:\openmtbmap\openmtbmap_svn\copyrightopm.txt" --road-name-config=C:\openmtbmap\openmtbmap_svn\roadNameConfig.txt --license-file="C:\openmtbmap\openmtbmap_svn\licenseopm.txt" --code-page=1252 --family-id=6528 --housenumbers --description="mtb_alp_08.09.2021" --series-name=mtbmap_alps_08.09.2021 --family-name=mtb_alp_08.09.2021 --product-id=1 --gmapsupp 6528*.img C:\openmtbmap\maps\widealp.TYP 1>NUL 2>NUL New counter: 33659 GB Overall summing it up: 2.57GB of .img mapdata supposed to write. 1.87GB for the gmapsupp.img , 2.53GB for the gmap folder containing everything written (i moved the mapset files and the info.xml into subdirectories). So 7GB of actual mapdata needed 11GB of writes. And I already saved 2GB of writes by making the contourline gmapi folder read only... Before this I had an additional 2GB of writes. -------------------------------- ------------------------------ So let's try something new - we move the gmap 6* folder after the first compile and put a hardlink (mklink) back so mkgmap cannot fuss around with them anymore. Yeah we're using croutches so to say. I will post again the results if I can improve by moving stuff and putting symlinks or by setting read only attribute. On Wed, 8 Sept 2021 at 14:10, Felix Hartmann <extremecarver@gmail.com> wrote:
Quite a few people do not compile the contourlines each time they update a map, but reuse the old contourlines. This works fine if you insert them as say 1234*img (and they are named 12340000.img 12340001.img and so on).
However this does not work when providing the --gmapi option. Actually mkgmap tries to rewrite all those files again/convert them.
Could it please be possible for mkgmap to change this? There could be the status quo with --gmapi and there could be a new mode called --gmapiminimal that only converts files input as .o5m or .osm.pbf or .osm but not as .img
--gpapiminimal would then work the same way as it does with .img files. This has two use cases.
a) if you have to resplit a tile because some tiles were breaking - and directly gave the --gmapiminimal option - no need to redo the whole .gmap file.
b) you symlink all gmap folders with the contourlines (or other layers you do not recreate on each update) into the gmap folder and mkgmap only writes out the new stuff.
Yes on an nvme disk you barely notice the conversion - it's really quick. BUT it is not needed if you have the files and even more - it burns your NVME SSD disk.
To my horror I noticed that one worldwide update of maps wrote 5TB of data to my disk (thanks to NVME SSD and fast processor this all took 28 hours). At that rate even datacenter NVME ssd disks are quickly turned into trash., consumer grade NVME disk will not make it half a year for me... In general maybe mkgmap could have an option to write temporary files to another disk? Then you use a RAMDisk for those files. Or mkgmap should keep them in memory. Modern servers often have 64 or 128GB of RAM so enough RAM for mkgmap not to waste writes to disk... Yes mkgmap uses quite a lot of RAM, but RAM is much cheaper than continous SSD writes....
Write now mkgmap crashes if you symlink already converted files to it: e.g. message: Number of MapFailedExceptions: 0 SEVERE (global): Error saving gmapi data java.nio.file.FileAlreadyExistsException: .\mtb_alp_08.09.2021.gmap\Product1\75280000
While if it would be allowed to write out the data it would overwrite existing files (which is fine or should do so - but same way as for .img files)
For my worldwide maps 10m contourlines are 200GB, 20m contourlines are 100GB in size. Rendering the map in two styles and each time needing to convert them even though I already have them converted creates 600GB of useless writes per update (as well as useless time spend converting those 300GB at least twice) - and that is only if I use the --gmapi option as a separate step. If I give bot --tdb-file and --gmapi it gets worse if a tile failed to write - and I just parse it again.
And that is only for the contourlines. As I need three sets of mapset files - once without contourlines, once with 10m contourlines, and once with 20m contourlines the actual map gmapi folders are written three times instead of once... Well over 1TB of useless conversion time and useless data written to disk (but I do not know another way to create the needed mapset_mdr.img, mapset.img and mapset.tdb files)
I don't know how to change that myself, but I hope this is not much work to change.
-- Felix Hartman - Openmtbmap.org & VeloMap.org
-- Felix Hartman - Openmtbmap.org & VeloMap.org
so continued.. 33659 GB After Step 1: new 33663 GB now set all files in Product1 folder to read only. Only applies to files not to the actual folder... attrib +r C:\openmtbmap\maps\mtb_%abr%_%date1%%IDT%.gmap\Product1\6* /s /d attrib -r C:\openmtbmap\maps\mtb_%abr%_%date1%%IDT%.gmap\Product1 /d Step2: Time started: Wed Sep 08 20:54:20 CEST 2021 Number of MapFailedExceptions: 0 SEVERE (global): Cannot write file java.nio.file.AccessDeniedException: .\mtb_alp_08.09.2021.gmap\Product1\65280000\65280000.RGN Number of ExitExceptions: 1 Time finished: Wed Sep 08 20:54:22 CEST 2021 Total time taken: 1 second so this is not working bad idea.... lets try moving the files to a temp directory on the same drive (so it's actually just a rename) and then run mkgmap again: Starting with 33663GB do Step 1 and move the files - and symlink them back... -- for /D %%f in (C:\openmtbmap\maps\mtb_%abr%_%date1%%IDT%.gmap\Product1\6*) do move %%f C:\openmtbmap\maps\temp >NUL SET SrcRoot=C:\openmtbmap\maps\temp\ SET TargetRoot=C:\openmtbmap\maps\mtb_%abr%_%date1%%IDT%.gmap\Product1\ FOR /D %%A IN ("%SrcRoot%\*") DO ( MKLINK /D "%TargetRoot%\%%~NA" "%%~A" ) 1>NUL 2>NUL FOR %%A IN ("%SrcRoot%\*") DO ( MKLINK "%TargetRoot%\%%~NXA" "%%~A" ) 1>NUL 2>NUL After Step 1 33668 GB After step 2 33669 GB After step 3 33669 GB After step 4: 33671 GB So now we're down to 8GB for actual 7GB of maps created. Bingo we have found a solution but it's really a lot of scripting to go there. I don't know why setting files to read only does not work - but creating symlinks does work with the tiny patch to mkgmap. I do think it would be in everyones interest to implement a better strategy for mkgmap... And with the loads of memory mkmap is taking up (25GB for me) - writing those .tmp files concerning the address search into memory instead of to disk would gain back the last ~1GB. The alternative being of course to write everything to RAMDISK - if you have loads of RAM - say 128GB this will even work for Europe or Asia continent map (and I guess in this scenario ECC RAM really makes sense?) - if you have 64GB you could try for all but Asia and Europe. Actually it will work for Asia using my above hacks of symlinking the contourlines. Because Asia alone is like 10GB contourlines on 20m. So if they are out of the way - the rest will do on the RAMDISK. Again would be much easier if those symlinks are not needed because mkgmap behaves SSD friendly and does not write those files in first place. I have come down from 14GB to 8GB of writes for the Alps. Maybe only 7.5GB (sound more plausible). On Wed, 8 Sept 2021 at 21:44, Felix Hartmann <extremecarver@gmail.com> wrote:
Uncommenting line 102 from /combiners/gmapibuilder.java and then messing by making folder read only works (even though this is really like using crutches while being healthy)! And yeah it's great mkgmap code is so cleanly written that even someone at a loss with java can write a workaround. I really just believe no-one ever really noticed or minded enough to write here. There was a lot of optimization in making mkgmap faster or use less memory - but no developper so far cared for hdd/ssd writes. Actually I am pretty sure not writing those temp files or attempting to write the gmap files if they exist will be an easy speed up for mkgmap. Thanks Gerd and everyone else of the developper for writing such great software"
as in the following example.
} catch (IOException e) { // throw new ExitException("Error saving gmapi data", e); }
So let's detail the writes on Alps step by step - I guess the best solution is to use a Ram Disk if mkgmap cannot be optimized - and then only write Asia and Europe Continent maps to NVME disk because I cannot create a big enough Ram Disk for them....
So the log is only in full GB - so here it what happens: start at: 63648GB written to C: *1.* 7*.img are the 20m contourlines. They and the 9*.img contourlines are already present in gmap format and in the relavant gmap folder.... mkgmap.jar is patched as above in order to not stop when it cannot write the 7* and 9* folders as they are read only. I'm thinking of making the 6* folder read only for the further steps? start /belownormal /b /wait java -jar -XX:+AggressiveHeap -XX:StringTableSize=1000003 -Xmx43000m C:\openmtbmap\mkgmap.jar --max-jobs=12 --order-by-decreasing-area "--generate-sea" --code-page=1252 "--precomp-sea=E:\OpenMTBMap\osmpbf_geofabrik\sea.zip" "--style-file=C:\openmtbmap\openmtbmap_style" --add-boundary-nodes-at-admin-boundaries --fix-roundabout-direction --improve-overview --drive-on=detect --allow-reverse-merge --line-types-with-direction=0x18,0x1f,0x10016,0x10101,0x10105,0x10401,0x10403,0x10405,0x10408,0x10409,0x1040b,0x1040d,0x1040e,0x10500,0x10501,0x10502,0x10503,0x10504,0x10505,0x10506,0x10507,0x10607,0x10609,0x1060b,0x10702,0x10703,0x10704,0x10705,0x10706,0x10707,0x10a00,0x10a02,0x10a04,0x10a05,0x10e1e,0x10e0f,0x10e0e,0x10e1e,0x10f0a,0x10f1c,0x10e1d --nsis --index --levels="0:24, 1:23, 2:22, 3:21, 4:20, 5:19, 6:18" --overview-levels="7:17, 8:16, 9:15, 10:14, 11:13" --add-pois-to-areas --pois-to-areas-placement=entrance=main;entrance=yes;building=entrance;barrier=entrance --simplify-lines=23:2.6,22:4.2,21:5.4,20:6,19:7,18:7.5,17:4.5,16:5,15:5,14:6 --simplify-polygons=23:3.6,22:7,21:6,20:9,17:5.4 --add-boundary-nodes-at-admin-boundaries=2 --poi-excl-index=0x6405,0x4316,0x2f00 --copyright-file="C:\openmtbmap\openmtbmap_svn\copyrightopm.txt" --license-file="C:\openmtbmap\openmtbmap_svn\licenseopm.txt" --cycle-map --ignore-fixme-values --housenumbers --road-name-config=C:\openmtbmap\openmtbmap_svn\roadNameConfig.txt --split-name-index --link-pois-to-ways --ignore-turn-restrictions --polygon-size-limits="24:12, 23:14, 22:14, 21:20, 20:20, 19:20, 18:20, 17:20, 16:20, 15:20, 14:20, 13:25" --description=omtb_alp --show-profiles=1 --location-autofill=bounds,is_in,nearest --bounds=E:\OpenMTBMap\osmpbf_geofabrik\bounds.zip --route --country-abbr=alp --country-name=alps --mapname=65280000 --family-id=6528 --product-id=1 --series-name=omtb_alps_08.09.2021 --family-name=mtb_alp_08.09.2021 --tdbfile --gmapi --overview-mapname=mapsetc --keep-going --area-name="alps_08.09.2021_omtb" -c D:\openmtbmap\maps\template.alps E:\OpenMTBMap\contourlines20\alps\7*.img typalp.TYP 1>NUL Mkgmap version 4806M Time started: Wed Sep 08 19:44:09 CEST 2021 ... Time taken 10 minutes
Now written 63652GB While it has actually written 2.12GB of .img files and 2.1GB for mac OSx files. Everything going pretty smooth here. Yes there have been some temporary files which I do not see why they are written if the max memory used was 25GB with 43GB heap (and never less than 37GB of RAM available)
*2.* C:\openmtbmap\maps>start /belownormal /b /wait java -jar -XX:+AggressiveHeap -XX:StringTableSize=1000003 -Xmx43000m C:\openmtbmap\mkgmap.jar --max-jobs=12 "--generate-sea" --code-page=1252 "--precomp-sea=E:\OpenMTBMap\osmpbf_geofabrik\sea.zip" "--style-file=C:\openmtbmap\openmtbmap_style" --add-boundary-nodes-at-admin-boundaries --fix-roundabout-direction --improve-overview --drive-on=detect --allow-reverse-merge --line-types-with-direction=0x18,0x1f,0x10016,0x10101,0x10105,0x10401,0x10403,0x10405,0x10408,0x10409,0x1040b,0x1040d,0x1040e,0x10500,0x10501,0x10502,0x10503,0x10504,0x10505,0x10506,0x10507,0x10607,0x10609,0x1060b,0x10702,0x10703,0x10704,0x10705,0x10706,0x10707,0x10a00,0x10a02,0x10a04,0x10a05,0x10e1e,0x10e0f,0x10e0e,0x10e1e,0x10f0a,0x10f1c,0x10e1d --nsis --index --levels="0:24, 1:23, 2:22, 3:21, 4:20, 5:19, 6:18" --overview-levels="7:17, 8:16, 9:15, 10:14, 11:13" --add-pois-to-areas --pois-to-areas-placement=entrance=main;entrance=yes;building=entrance;barrier=entrance --road-name-config=C:\openmtbmap\openmtbmap_svn\roadNameConfig.txt --split-name-index --housenumbers --description=omtb_alp --show-profiles=0 --location-autofill=bounds,is_in,nearest --bounds=E:\OpenMTBMap\osmpbf_geofabrik\bounds.zip --route --poi-excl-index=0x6405,0x4316,0x2f00 --cycle-map --copyright-file="C:\openmtbmap\openmtbmap_svn\copyrightopm.txt" --license-file="C:\openmtbmap\openmtbmap_svn\licenseopm.txt" --country-abbr=alp --country-name=alps --mapname=65280000 --family-id=6528 --product-id=1 --series-name=omtb_alps_08.09.2021 --family-name=mtb_alp_08.09.2021 --tdbfile --gmapi --overview-mapname=mapsetx --keep-going --area-name="alps_08.09.2021_omtb" 6528*.img typalp.TYP 1>NUL 2>NUL
Time taken - 30 seconds or so Written 460MB (twice 230MB for each mapset_mdr file) However new counter: 63655GB Waste - minimum 2GB. maybe 3GB..
3. C:\openmtbmap\maps>start /belownormal /b /wait java -jar -XX:+AggressiveHeap -XX:StringTableSize=1000003 -Xmx43000m C:\openmtbmap\mkgmap.jar --max-jobs=12 "--generate-sea" --code-page=1252 "--precomp-sea=E:\OpenMTBMap\osmpbf_geofabrik\sea.zip" "--style-file=C:\openmtbmap\openmtbmap_style" --add-boundary-nodes-at-admin-boundaries --fix-roundabout-direction --improve-overview --drive-on=detect --allow-reverse-merge --line-types-with-direction=0x18,0x1f,0x10016,0x10101,0x10105,0x10401,0x10403,0x10405,0x10408,0x10409,0x1040b,0x1040d,0x1040e,0x10500,0x10501,0x10502,0x10503,0x10504,0x10505,0x10506,0x10507,0x10607,0x10609,0x1060b,0x10702,0x10703,0x10704,0x10705,0x10706,0x10707,0x10a00,0x10a02,0x10a04,0x10a05,0x10e1e,0x10e0f,0x10e0e,0x10e1e,0x10f0a,0x10f1c,0x10e1d --nsis --index --levels="0:24, 1:23, 2:22, 3:21, 4:20, 5:19, 6:18" --overview-levels="7:17, 8:16, 9:15, 10:14, 11:13" --add-pois-to-areas --pois-to-areas-placement=entrance=main;entrance=yes;building=entrance;barrier=entrance --road-name-config=C:\openmtbmap\openmtbmap_svn\roadNameConfig.txt --split-name-index --housenumbers --description=omtb_alp --show-profiles=1 --location-autofill=bounds,is_in,nearest --bounds=E:\OpenMTBMap\osmpbf_geofabrik\bounds.zip --route --poi-excl-index=0x6405,0x4316,0x2f00 --cycle-map --copyright-file="C:\openmtbmap\openmtbmap_svn\copyrightopm.txt" --license-file="C:\openmtbmap\openmtbmap_svn\licenseopm.txt" --country-abbr=alp --country-name=alps --mapname=65280000 --family-id=6528 --product-id=1 --series-name=omtb_alps_08.09.2021 --family-name=mtb_alp_08.09.2021 --tdbfile --gmapi --overview-mapname=mapset10 --keep-going --area-name="alps_08.09.2021_omtb" 6528*.img typalp.TYP E:\OpenMTBMap\contourlines10\alps\9*.img 1>NUL 2>NUL
Time taken 1 minute or so Written 460M (same as above) However new counter: 33657 GB (I guess the waste is the same as above - it cannot write the 9*.img - so each time around 2GB of wasted Write commands)
4. Let's create the gmapsupp.img - there is no more wasted writes here start /belownormal /b /wait java -jar -XX:+AggressiveHeap -XX:StringTableSize=1000003 -Xmx43000m C:\openmtbmap\mkgmap.jar --max-jobs=12 --hide-gmapsupp-on-pc --copyright-file="C:\openmtbmap\openmtbmap_svn\copyrightopm.txt" --road-name-config=C:\openmtbmap\openmtbmap_svn\roadNameConfig.txt --license-file="C:\openmtbmap\openmtbmap_svn\licenseopm.txt" --code-page=1252 --family-id=6528 --housenumbers --description="mtb_alp_08.09.2021" --series-name=mtbmap_alps_08.09.2021 --family-name=mtb_alp_08.09.2021 --product-id=1 --gmapsupp 6528*.img C:\openmtbmap\maps\widealp.TYP 1>NUL 2>NUL New counter: 33659 GB
Overall summing it up: 2.57GB of .img mapdata supposed to write. 1.87GB for the gmapsupp.img , 2.53GB for the gmap folder containing everything written (i moved the mapset files and the info.xml into subdirectories). So 7GB of actual mapdata needed 11GB of writes. And I already saved 2GB of writes by making the contourline gmapi folder read only... Before this I had an additional 2GB of writes.
-------------------------------- ------------------------------
So let's try something new - we move the gmap 6* folder after the first compile and put a hardlink (mklink) back so mkgmap cannot fuss around with them anymore. Yeah we're using croutches so to say. I will post again the results if I can improve by moving stuff and putting symlinks or by setting read only attribute.
On Wed, 8 Sept 2021 at 14:10, Felix Hartmann <extremecarver@gmail.com> wrote:
Quite a few people do not compile the contourlines each time they update a map, but reuse the old contourlines. This works fine if you insert them as say 1234*img (and they are named 12340000.img 12340001.img and so on).
However this does not work when providing the --gmapi option. Actually mkgmap tries to rewrite all those files again/convert them.
Could it please be possible for mkgmap to change this? There could be the status quo with --gmapi and there could be a new mode called --gmapiminimal that only converts files input as .o5m or .osm.pbf or .osm but not as .img
--gpapiminimal would then work the same way as it does with .img files. This has two use cases.
a) if you have to resplit a tile because some tiles were breaking - and directly gave the --gmapiminimal option - no need to redo the whole .gmap file.
b) you symlink all gmap folders with the contourlines (or other layers you do not recreate on each update) into the gmap folder and mkgmap only writes out the new stuff.
Yes on an nvme disk you barely notice the conversion - it's really quick. BUT it is not needed if you have the files and even more - it burns your NVME SSD disk.
To my horror I noticed that one worldwide update of maps wrote 5TB of data to my disk (thanks to NVME SSD and fast processor this all took 28 hours). At that rate even datacenter NVME ssd disks are quickly turned into trash., consumer grade NVME disk will not make it half a year for me... In general maybe mkgmap could have an option to write temporary files to another disk? Then you use a RAMDisk for those files. Or mkgmap should keep them in memory. Modern servers often have 64 or 128GB of RAM so enough RAM for mkgmap not to waste writes to disk... Yes mkgmap uses quite a lot of RAM, but RAM is much cheaper than continous SSD writes....
Write now mkgmap crashes if you symlink already converted files to it: e.g. message: Number of MapFailedExceptions: 0 SEVERE (global): Error saving gmapi data java.nio.file.FileAlreadyExistsException: .\mtb_alp_08.09.2021.gmap\Product1\75280000
While if it would be allowed to write out the data it would overwrite existing files (which is fine or should do so - but same way as for .img files)
For my worldwide maps 10m contourlines are 200GB, 20m contourlines are 100GB in size. Rendering the map in two styles and each time needing to convert them even though I already have them converted creates 600GB of useless writes per update (as well as useless time spend converting those 300GB at least twice) - and that is only if I use the --gmapi option as a separate step. If I give bot --tdb-file and --gmapi it gets worse if a tile failed to write - and I just parse it again.
And that is only for the contourlines. As I need three sets of mapset files - once without contourlines, once with 10m contourlines, and once with 20m contourlines the actual map gmapi folders are written three times instead of once... Well over 1TB of useless conversion time and useless data written to disk (but I do not know another way to create the needed mapset_mdr.img, mapset.img and mapset.tdb files)
I don't know how to change that myself, but I hope this is not much work to change.
-- Felix Hartman - Openmtbmap.org & VeloMap.org
-- Felix Hartman - Openmtbmap.org & VeloMap.org
-- Felix Hartman - Openmtbmap.org & VeloMap.org
oh sorry - made a typo at the first number. (there was no error in the SMART data I just somehow wrote 3 instead of 6. Those 30TB of writes did not suddenly disappear). On Wed, 8 Sept 2021 at 22:36, Felix Hartmann <extremecarver@gmail.com> wrote:
so continued.. 33659 GB After Step 1: new 33663 GB now set all files in Product1 folder to read only. Only applies to files not to the actual folder... attrib +r C:\openmtbmap\maps\mtb_%abr%_%date1%%IDT%.gmap\Product1\6* /s /d attrib -r C:\openmtbmap\maps\mtb_%abr%_%date1%%IDT%.gmap\Product1 /d
Step2: Time started: Wed Sep 08 20:54:20 CEST 2021 Number of MapFailedExceptions: 0 SEVERE (global): Cannot write file java.nio.file.AccessDeniedException: .\mtb_alp_08.09.2021.gmap\Product1\65280000\65280000.RGN Number of ExitExceptions: 1 Time finished: Wed Sep 08 20:54:22 CEST 2021 Total time taken: 1 second
so this is not working bad idea.... lets try moving the files to a temp directory on the same drive (so it's actually just a rename) and then run mkgmap again: Starting with 33663GB do Step 1 and move the files - and symlink them back...
-- for /D %%f in (C:\openmtbmap\maps\mtb_%abr%_%date1%%IDT%.gmap\Product1\6*) do move %%f C:\openmtbmap\maps\temp >NUL SET SrcRoot=C:\openmtbmap\maps\temp\ SET TargetRoot=C:\openmtbmap\maps\mtb_%abr%_%date1%%IDT%.gmap\Product1\
FOR /D %%A IN ("%SrcRoot%\*") DO ( MKLINK /D "%TargetRoot%\%%~NA" "%%~A" ) 1>NUL 2>NUL FOR %%A IN ("%SrcRoot%\*") DO ( MKLINK "%TargetRoot%\%%~NXA" "%%~A" ) 1>NUL 2>NUL
After Step 1 33668 GB
After step 2 33669 GB
After step 3 33669 GB
After step 4: 33671 GB
So now we're down to 8GB for actual 7GB of maps created. Bingo we have found a solution but it's really a lot of scripting to go there. I don't know why setting files to read only does not work - but creating symlinks does work with the tiny patch to mkgmap. I do think it would be in everyones interest to implement a better strategy for mkgmap... And with the loads of memory mkmap is taking up (25GB for me) - writing those .tmp files concerning the address search into memory instead of to disk would gain back the last ~1GB.
The alternative being of course to write everything to RAMDISK - if you have loads of RAM - say 128GB this will even work for Europe or Asia continent map (and I guess in this scenario ECC RAM really makes sense?) - if you have 64GB you could try for all but Asia and Europe. Actually it will work for Asia using my above hacks of symlinking the contourlines. Because Asia alone is like 10GB contourlines on 20m. So if they are out of the way - the rest will do on the RAMDISK. Again would be much easier if those symlinks are not needed because mkgmap behaves SSD friendly and does not write those files in first place. I have come down from 14GB to 8GB of writes for the Alps. Maybe only 7.5GB (sound more plausible).
On Wed, 8 Sept 2021 at 21:44, Felix Hartmann <extremecarver@gmail.com> wrote:
Uncommenting line 102 from /combiners/gmapibuilder.java and then messing by making folder read only works (even though this is really like using crutches while being healthy)! And yeah it's great mkgmap code is so cleanly written that even someone at a loss with java can write a workaround. I really just believe no-one ever really noticed or minded enough to write here. There was a lot of optimization in making mkgmap faster or use less memory - but no developper so far cared for hdd/ssd writes. Actually I am pretty sure not writing those temp files or attempting to write the gmap files if they exist will be an easy speed up for mkgmap. Thanks Gerd and everyone else of the developper for writing such great software"
as in the following example.
} catch (IOException e) { // throw new ExitException("Error saving gmapi data", e); }
So let's detail the writes on Alps step by step - I guess the best solution is to use a Ram Disk if mkgmap cannot be optimized - and then only write Asia and Europe Continent maps to NVME disk because I cannot create a big enough Ram Disk for them....
So the log is only in full GB - so here it what happens: start at: 63648GB written to C: *1.* 7*.img are the 20m contourlines. They and the 9*.img contourlines are already present in gmap format and in the relavant gmap folder.... mkgmap.jar is patched as above in order to not stop when it cannot write the 7* and 9* folders as they are read only. I'm thinking of making the 6* folder read only for the further steps? start /belownormal /b /wait java -jar -XX:+AggressiveHeap -XX:StringTableSize=1000003 -Xmx43000m C:\openmtbmap\mkgmap.jar --max-jobs=12 --order-by-decreasing-area "--generate-sea" --code-page=1252 "--precomp-sea=E:\OpenMTBMap\osmpbf_geofabrik\sea.zip" "--style-file=C:\openmtbmap\openmtbmap_style" --add-boundary-nodes-at-admin-boundaries --fix-roundabout-direction --improve-overview --drive-on=detect --allow-reverse-merge --line-types-with-direction=0x18,0x1f,0x10016,0x10101,0x10105,0x10401,0x10403,0x10405,0x10408,0x10409,0x1040b,0x1040d,0x1040e,0x10500,0x10501,0x10502,0x10503,0x10504,0x10505,0x10506,0x10507,0x10607,0x10609,0x1060b,0x10702,0x10703,0x10704,0x10705,0x10706,0x10707,0x10a00,0x10a02,0x10a04,0x10a05,0x10e1e,0x10e0f,0x10e0e,0x10e1e,0x10f0a,0x10f1c,0x10e1d --nsis --index --levels="0:24, 1:23, 2:22, 3:21, 4:20, 5:19, 6:18" --overview-levels="7:17, 8:16, 9:15, 10:14, 11:13" --add-pois-to-areas --pois-to-areas-placement=entrance=main;entrance=yes;building=entrance;barrier=entrance --simplify-lines=23:2.6,22:4.2,21:5.4,20:6,19:7,18:7.5,17:4.5,16:5,15:5,14:6 --simplify-polygons=23:3.6,22:7,21:6,20:9,17:5.4 --add-boundary-nodes-at-admin-boundaries=2 --poi-excl-index=0x6405,0x4316,0x2f00 --copyright-file="C:\openmtbmap\openmtbmap_svn\copyrightopm.txt" --license-file="C:\openmtbmap\openmtbmap_svn\licenseopm.txt" --cycle-map --ignore-fixme-values --housenumbers --road-name-config=C:\openmtbmap\openmtbmap_svn\roadNameConfig.txt --split-name-index --link-pois-to-ways --ignore-turn-restrictions --polygon-size-limits="24:12, 23:14, 22:14, 21:20, 20:20, 19:20, 18:20, 17:20, 16:20, 15:20, 14:20, 13:25" --description=omtb_alp --show-profiles=1 --location-autofill=bounds,is_in,nearest --bounds=E:\OpenMTBMap\osmpbf_geofabrik\bounds.zip --route --country-abbr=alp --country-name=alps --mapname=65280000 --family-id=6528 --product-id=1 --series-name=omtb_alps_08.09.2021 --family-name=mtb_alp_08.09.2021 --tdbfile --gmapi --overview-mapname=mapsetc --keep-going --area-name="alps_08.09.2021_omtb" -c D:\openmtbmap\maps\template.alps E:\OpenMTBMap\contourlines20\alps\7*.img typalp.TYP 1>NUL Mkgmap version 4806M Time started: Wed Sep 08 19:44:09 CEST 2021 ... Time taken 10 minutes
Now written 63652GB While it has actually written 2.12GB of .img files and 2.1GB for mac OSx files. Everything going pretty smooth here. Yes there have been some temporary files which I do not see why they are written if the max memory used was 25GB with 43GB heap (and never less than 37GB of RAM available)
*2.* C:\openmtbmap\maps>start /belownormal /b /wait java -jar -XX:+AggressiveHeap -XX:StringTableSize=1000003 -Xmx43000m C:\openmtbmap\mkgmap.jar --max-jobs=12 "--generate-sea" --code-page=1252 "--precomp-sea=E:\OpenMTBMap\osmpbf_geofabrik\sea.zip" "--style-file=C:\openmtbmap\openmtbmap_style" --add-boundary-nodes-at-admin-boundaries --fix-roundabout-direction --improve-overview --drive-on=detect --allow-reverse-merge --line-types-with-direction=0x18,0x1f,0x10016,0x10101,0x10105,0x10401,0x10403,0x10405,0x10408,0x10409,0x1040b,0x1040d,0x1040e,0x10500,0x10501,0x10502,0x10503,0x10504,0x10505,0x10506,0x10507,0x10607,0x10609,0x1060b,0x10702,0x10703,0x10704,0x10705,0x10706,0x10707,0x10a00,0x10a02,0x10a04,0x10a05,0x10e1e,0x10e0f,0x10e0e,0x10e1e,0x10f0a,0x10f1c,0x10e1d --nsis --index --levels="0:24, 1:23, 2:22, 3:21, 4:20, 5:19, 6:18" --overview-levels="7:17, 8:16, 9:15, 10:14, 11:13" --add-pois-to-areas --pois-to-areas-placement=entrance=main;entrance=yes;building=entrance;barrier=entrance --road-name-config=C:\openmtbmap\openmtbmap_svn\roadNameConfig.txt --split-name-index --housenumbers --description=omtb_alp --show-profiles=0 --location-autofill=bounds,is_in,nearest --bounds=E:\OpenMTBMap\osmpbf_geofabrik\bounds.zip --route --poi-excl-index=0x6405,0x4316,0x2f00 --cycle-map --copyright-file="C:\openmtbmap\openmtbmap_svn\copyrightopm.txt" --license-file="C:\openmtbmap\openmtbmap_svn\licenseopm.txt" --country-abbr=alp --country-name=alps --mapname=65280000 --family-id=6528 --product-id=1 --series-name=omtb_alps_08.09.2021 --family-name=mtb_alp_08.09.2021 --tdbfile --gmapi --overview-mapname=mapsetx --keep-going --area-name="alps_08.09.2021_omtb" 6528*.img typalp.TYP 1>NUL 2>NUL
Time taken - 30 seconds or so Written 460MB (twice 230MB for each mapset_mdr file) However new counter: 63655GB Waste - minimum 2GB. maybe 3GB..
3. C:\openmtbmap\maps>start /belownormal /b /wait java -jar -XX:+AggressiveHeap -XX:StringTableSize=1000003 -Xmx43000m C:\openmtbmap\mkgmap.jar --max-jobs=12 "--generate-sea" --code-page=1252 "--precomp-sea=E:\OpenMTBMap\osmpbf_geofabrik\sea.zip" "--style-file=C:\openmtbmap\openmtbmap_style" --add-boundary-nodes-at-admin-boundaries --fix-roundabout-direction --improve-overview --drive-on=detect --allow-reverse-merge --line-types-with-direction=0x18,0x1f,0x10016,0x10101,0x10105,0x10401,0x10403,0x10405,0x10408,0x10409,0x1040b,0x1040d,0x1040e,0x10500,0x10501,0x10502,0x10503,0x10504,0x10505,0x10506,0x10507,0x10607,0x10609,0x1060b,0x10702,0x10703,0x10704,0x10705,0x10706,0x10707,0x10a00,0x10a02,0x10a04,0x10a05,0x10e1e,0x10e0f,0x10e0e,0x10e1e,0x10f0a,0x10f1c,0x10e1d --nsis --index --levels="0:24, 1:23, 2:22, 3:21, 4:20, 5:19, 6:18" --overview-levels="7:17, 8:16, 9:15, 10:14, 11:13" --add-pois-to-areas --pois-to-areas-placement=entrance=main;entrance=yes;building=entrance;barrier=entrance --road-name-config=C:\openmtbmap\openmtbmap_svn\roadNameConfig.txt --split-name-index --housenumbers --description=omtb_alp --show-profiles=1 --location-autofill=bounds,is_in,nearest --bounds=E:\OpenMTBMap\osmpbf_geofabrik\bounds.zip --route --poi-excl-index=0x6405,0x4316,0x2f00 --cycle-map --copyright-file="C:\openmtbmap\openmtbmap_svn\copyrightopm.txt" --license-file="C:\openmtbmap\openmtbmap_svn\licenseopm.txt" --country-abbr=alp --country-name=alps --mapname=65280000 --family-id=6528 --product-id=1 --series-name=omtb_alps_08.09.2021 --family-name=mtb_alp_08.09.2021 --tdbfile --gmapi --overview-mapname=mapset10 --keep-going --area-name="alps_08.09.2021_omtb" 6528*.img typalp.TYP E:\OpenMTBMap\contourlines10\alps\9*.img 1>NUL 2>NUL
Time taken 1 minute or so Written 460M (same as above) However new counter: 33657 GB (I guess the waste is the same as above - it cannot write the 9*.img - so each time around 2GB of wasted Write commands)
4. Let's create the gmapsupp.img - there is no more wasted writes here start /belownormal /b /wait java -jar -XX:+AggressiveHeap -XX:StringTableSize=1000003 -Xmx43000m C:\openmtbmap\mkgmap.jar --max-jobs=12 --hide-gmapsupp-on-pc --copyright-file="C:\openmtbmap\openmtbmap_svn\copyrightopm.txt" --road-name-config=C:\openmtbmap\openmtbmap_svn\roadNameConfig.txt --license-file="C:\openmtbmap\openmtbmap_svn\licenseopm.txt" --code-page=1252 --family-id=6528 --housenumbers --description="mtb_alp_08.09.2021" --series-name=mtbmap_alps_08.09.2021 --family-name=mtb_alp_08.09.2021 --product-id=1 --gmapsupp 6528*.img C:\openmtbmap\maps\widealp.TYP 1>NUL 2>NUL New counter: 33659 GB
Overall summing it up: 2.57GB of .img mapdata supposed to write. 1.87GB for the gmapsupp.img , 2.53GB for the gmap folder containing everything written (i moved the mapset files and the info.xml into subdirectories). So 7GB of actual mapdata needed 11GB of writes. And I already saved 2GB of writes by making the contourline gmapi folder read only... Before this I had an additional 2GB of writes.
-------------------------------- ------------------------------
So let's try something new - we move the gmap 6* folder after the first compile and put a hardlink (mklink) back so mkgmap cannot fuss around with them anymore. Yeah we're using croutches so to say. I will post again the results if I can improve by moving stuff and putting symlinks or by setting read only attribute.
On Wed, 8 Sept 2021 at 14:10, Felix Hartmann <extremecarver@gmail.com> wrote:
Quite a few people do not compile the contourlines each time they update a map, but reuse the old contourlines. This works fine if you insert them as say 1234*img (and they are named 12340000.img 12340001.img and so on).
However this does not work when providing the --gmapi option. Actually mkgmap tries to rewrite all those files again/convert them.
Could it please be possible for mkgmap to change this? There could be the status quo with --gmapi and there could be a new mode called --gmapiminimal that only converts files input as .o5m or .osm.pbf or .osm but not as .img
--gpapiminimal would then work the same way as it does with .img files. This has two use cases.
a) if you have to resplit a tile because some tiles were breaking - and directly gave the --gmapiminimal option - no need to redo the whole .gmap file.
b) you symlink all gmap folders with the contourlines (or other layers you do not recreate on each update) into the gmap folder and mkgmap only writes out the new stuff.
Yes on an nvme disk you barely notice the conversion - it's really quick. BUT it is not needed if you have the files and even more - it burns your NVME SSD disk.
To my horror I noticed that one worldwide update of maps wrote 5TB of data to my disk (thanks to NVME SSD and fast processor this all took 28 hours). At that rate even datacenter NVME ssd disks are quickly turned into trash., consumer grade NVME disk will not make it half a year for me... In general maybe mkgmap could have an option to write temporary files to another disk? Then you use a RAMDisk for those files. Or mkgmap should keep them in memory. Modern servers often have 64 or 128GB of RAM so enough RAM for mkgmap not to waste writes to disk... Yes mkgmap uses quite a lot of RAM, but RAM is much cheaper than continous SSD writes....
Write now mkgmap crashes if you symlink already converted files to it: e.g. message: Number of MapFailedExceptions: 0 SEVERE (global): Error saving gmapi data java.nio.file.FileAlreadyExistsException: .\mtb_alp_08.09.2021.gmap\Product1\75280000
While if it would be allowed to write out the data it would overwrite existing files (which is fine or should do so - but same way as for .img files)
For my worldwide maps 10m contourlines are 200GB, 20m contourlines are 100GB in size. Rendering the map in two styles and each time needing to convert them even though I already have them converted creates 600GB of useless writes per update (as well as useless time spend converting those 300GB at least twice) - and that is only if I use the --gmapi option as a separate step. If I give bot --tdb-file and --gmapi it gets worse if a tile failed to write - and I just parse it again.
And that is only for the contourlines. As I need three sets of mapset files - once without contourlines, once with 10m contourlines, and once with 20m contourlines the actual map gmapi folders are written three times instead of once... Well over 1TB of useless conversion time and useless data written to disk (but I do not know another way to create the needed mapset_mdr.img, mapset.img and mapset.tdb files)
I don't know how to change that myself, but I hope this is not much work to change.
-- Felix Hartman - Openmtbmap.org & VeloMap.org
-- Felix Hartman - Openmtbmap.org & VeloMap.org
-- Felix Hartman - Openmtbmap.org & VeloMap.org
-- Felix Hartman - Openmtbmap.org & VeloMap.org
participants (4)
-
ael -
Carlos Dávila -
Felix Hartmann -
Gerd Petermann