Uncommenting line 102  from /combiners/gmapibuilder.java and then messing by making folder read only works (even though this is really like using crutches while being healthy)! And yeah it's great mkgmap code is so cleanly written that even someone at a loss with java can write a workaround. I really just believe no-one ever really noticed or minded enough to write here. There was a lot of optimization in making mkgmap faster or use less memory - but no developper so far cared for hdd/ssd writes. Actually I am pretty sure not writing those temp files or attempting to write the gmap files if they exist will be an easy speed up for mkgmap. Thanks Gerd and everyone else of the developper for writing such great software"

as in the following example.

  } catch (IOException e) {
// throw new ExitException("Error saving gmapi data", e);
}

So let's detail the writes on Alps step by step - I guess the best solution is to use a Ram Disk if mkgmap cannot be optimized - and then only write Asia and Europe Continent maps to NVME disk because I cannot create a big enough Ram Disk for them....

So the log is only in full GB - so here it what happens:

start at: 63648GB written to C:

1. 7*.img are the 20m contourlines. They and the 9*.img contourlines are already present in gmap format and in the relavant gmap folder.... mkgmap.jar is patched as above in order to not stop when it cannot write the 7* and 9* folders as they are read only. I'm thinking of making the 6* folder read only for the further steps?

start /belownormal /b /wait java -jar -XX:+AggressiveHeap -XX:StringTableSize=1000003 -Xmx43000m C:\openmtbmap\mkgmap.jar --max-jobs=12 --order-by-decreasing-area  "--generate-sea" --code-page=1252 "--precomp-sea=E:\OpenMTBMap\osmpbf_geofabrik\sea.zip" "--style-file=C:\openmtbmap\openmtbmap_style" --add-boundary-nodes-at-admin-boundaries --fix-roundabout-direction --improve-overview --drive-on=detect --allow-reverse-merge --line-types-with-direction=0x18,0x1f,0x10016,0x10101,0x10105,0x10401,0x10403,0x10405,0x10408,0x10409,0x1040b,0x1040d,0x1040e,0x10500,0x10501,0x10502,0x10503,0x10504,0x10505,0x10506,0x10507,0x10607,0x10609,0x1060b,0x10702,0x10703,0x10704,0x10705,0x10706,0x10707,0x10a00,0x10a02,0x10a04,0x10a05,0x10e1e,0x10e0f,0x10e0e,0x10e1e,0x10f0a,0x10f1c,0x10e1d  --nsis --index --levels="0:24, 1:23, 2:22, 3:21, 4:20, 5:19, 6:18" --overview-levels="7:17, 8:16, 9:15, 10:14, 11:13" --add-pois-to-areas --pois-to-areas-placement=entrance=main;entrance=yes;building=entrance;barrier=entrance --simplify-lines=23:2.6,22:4.2,21:5.4,20:6,19:7,18:7.5,17:4.5,16:5,15:5,14:6  --simplify-polygons=23:3.6,22:7,21:6,20:9,17:5.4 --add-boundary-nodes-at-admin-boundaries=2 --poi-excl-index=0x6405,0x4316,0x2f00 --copyright-file="C:\openmtbmap\openmtbmap_svn\copyrightopm.txt" --license-file="C:\openmtbmap\openmtbmap_svn\licenseopm.txt" --cycle-map --ignore-fixme-values --housenumbers --road-name-config=C:\openmtbmap\openmtbmap_svn\roadNameConfig.txt --split-name-index --link-pois-to-ways --ignore-turn-restrictions --polygon-size-limits="24:12, 23:14, 22:14, 21:20, 20:20, 19:20, 18:20, 17:20, 16:20, 15:20, 14:20, 13:25" --description=omtb_alp --show-profiles=1  --location-autofill=bounds,is_in,nearest --bounds=E:\OpenMTBMap\osmpbf_geofabrik\bounds.zip  --route --country-abbr=alp --country-name=alps --mapname=65280000 --family-id=6528 --product-id=1 --series-name=omtb_alps_08.09.2021 --family-name=mtb_alp_08.09.2021 --tdbfile --gmapi --overview-mapname=mapsetc --keep-going --area-name="alps_08.09.2021_omtb" -c D:\openmtbmap\maps\template.alps E:\OpenMTBMap\contourlines20\alps\7*.img typalp.TYP  1>NUL
Mkgmap version 4806M
Time started: Wed Sep 08 19:44:09 CEST 2021

...

Time taken 10 minutes

Now written 

63652GB 

While it has actually written 2.12GB of .img files and 2.1GB for mac OSx files. Everything going pretty smooth here. Yes there have been some temporary files which I do not see why they are written if the max memory used was 25GB with 43GB heap (and never less than 37GB of RAM available)

2.

C:\openmtbmap\maps>start /belownormal /b /wait java -jar -XX:+AggressiveHeap -XX:StringTableSize=1000003 -Xmx43000m C:\openmtbmap\mkgmap.jar --max-jobs=12 "--generate-sea" --code-page=1252 "--precomp-sea=E:\OpenMTBMap\osmpbf_geofabrik\sea.zip" "--style-file=C:\openmtbmap\openmtbmap_style" --add-boundary-nodes-at-admin-boundaries --fix-roundabout-direction --improve-overview --drive-on=detect --allow-reverse-merge --line-types-with-direction=0x18,0x1f,0x10016,0x10101,0x10105,0x10401,0x10403,0x10405,0x10408,0x10409,0x1040b,0x1040d,0x1040e,0x10500,0x10501,0x10502,0x10503,0x10504,0x10505,0x10506,0x10507,0x10607,0x10609,0x1060b,0x10702,0x10703,0x10704,0x10705,0x10706,0x10707,0x10a00,0x10a02,0x10a04,0x10a05,0x10e1e,0x10e0f,0x10e0e,0x10e1e,0x10f0a,0x10f1c,0x10e1d  --nsis --index --levels="0:24, 1:23, 2:22, 3:21, 4:20, 5:19, 6:18" --overview-levels="7:17, 8:16, 9:15, 10:14, 11:13" --add-pois-to-areas --pois-to-areas-placement=entrance=main;entrance=yes;building=entrance;barrier=entrance --road-name-config=C:\openmtbmap\openmtbmap_svn\roadNameConfig.txt --split-name-index --housenumbers --description=omtb_alp --show-profiles=0 --location-autofill=bounds,is_in,nearest --bounds=E:\OpenMTBMap\osmpbf_geofabrik\bounds.zip --route --poi-excl-index=0x6405,0x4316,0x2f00 --cycle-map --copyright-file="C:\openmtbmap\openmtbmap_svn\copyrightopm.txt" --license-file="C:\openmtbmap\openmtbmap_svn\licenseopm.txt" --country-abbr=alp --country-name=alps --mapname=65280000 --family-id=6528 --product-id=1 --series-name=omtb_alps_08.09.2021 --family-name=mtb_alp_08.09.2021 --tdbfile --gmapi --overview-mapname=mapsetx --keep-going --area-name="alps_08.09.2021_omtb" 6528*.img typalp.TYP  1>NUL 2>NUL

Time taken - 30 seconds or so

Written 460MB (twice 230MB for each mapset_mdr file) 

However new counter: 

63655GB 

Waste - minimum 2GB. maybe 3GB..

3.

C:\openmtbmap\maps>start /belownormal /b /wait java -jar -XX:+AggressiveHeap -XX:StringTableSize=1000003 -Xmx43000m C:\openmtbmap\mkgmap.jar --max-jobs=12 "--generate-sea" --code-page=1252 "--precomp-sea=E:\OpenMTBMap\osmpbf_geofabrik\sea.zip" "--style-file=C:\openmtbmap\openmtbmap_style" --add-boundary-nodes-at-admin-boundaries --fix-roundabout-direction --improve-overview --drive-on=detect --allow-reverse-merge --line-types-with-direction=0x18,0x1f,0x10016,0x10101,0x10105,0x10401,0x10403,0x10405,0x10408,0x10409,0x1040b,0x1040d,0x1040e,0x10500,0x10501,0x10502,0x10503,0x10504,0x10505,0x10506,0x10507,0x10607,0x10609,0x1060b,0x10702,0x10703,0x10704,0x10705,0x10706,0x10707,0x10a00,0x10a02,0x10a04,0x10a05,0x10e1e,0x10e0f,0x10e0e,0x10e1e,0x10f0a,0x10f1c,0x10e1d  --nsis --index --levels="0:24, 1:23, 2:22, 3:21, 4:20, 5:19, 6:18" --overview-levels="7:17, 8:16, 9:15, 10:14, 11:13" --add-pois-to-areas --pois-to-areas-placement=entrance=main;entrance=yes;building=entrance;barrier=entrance --road-name-config=C:\openmtbmap\openmtbmap_svn\roadNameConfig.txt --split-name-index --housenumbers --description=omtb_alp --show-profiles=1 --location-autofill=bounds,is_in,nearest --bounds=E:\OpenMTBMap\osmpbf_geofabrik\bounds.zip --route --poi-excl-index=0x6405,0x4316,0x2f00 --cycle-map --copyright-file="C:\openmtbmap\openmtbmap_svn\copyrightopm.txt" --license-file="C:\openmtbmap\openmtbmap_svn\licenseopm.txt" --country-abbr=alp --country-name=alps --mapname=65280000 --family-id=6528 --product-id=1 --series-name=omtb_alps_08.09.2021 --family-name=mtb_alp_08.09.2021 --tdbfile --gmapi --overview-mapname=mapset10 --keep-going --area-name="alps_08.09.2021_omtb" 6528*.img typalp.TYP E:\OpenMTBMap\contourlines10\alps\9*.img  1>NUL 2>NUL

Time taken 1 minute or so

Written 460M (same as above)

However new counter:

33657 GB

(I guess the waste is the same as above - it cannot write the 9*.img - so each time around 2GB of wasted Write commands)

4.

Let's create the gmapsupp.img - there is no more wasted writes here

start /belownormal /b /wait java -jar -XX:+AggressiveHeap -XX:StringTableSize=1000003 -Xmx43000m C:\openmtbmap\mkgmap.jar --max-jobs=12 --hide-gmapsupp-on-pc --copyright-file="C:\openmtbmap\openmtbmap_svn\copyrightopm.txt" --road-name-config=C:\openmtbmap\openmtbmap_svn\roadNameConfig.txt --license-file="C:\openmtbmap\openmtbmap_svn\licenseopm.txt" --code-page=1252 --family-id=6528  --housenumbers --description="mtb_alp_08.09.2021" --series-name=mtbmap_alps_08.09.2021 --family-name=mtb_alp_08.09.2021 --product-id=1 --gmapsupp 6528*.img C:\openmtbmap\maps\widealp.TYP  1>NUL 2>NUL

New counter:

33659 GB

Overall summing it up:

2.57GB of .img mapdata supposed to write. 1.87GB for the gmapsupp.img , 2.53GB for the gmap folder containing everything written (i moved the mapset files and the info.xml into subdirectories). 

So 7GB of actual mapdata needed 11GB of writes. And I already saved 2GB of writes by making the contourline gmapi folder read only... Before this I had an additional 2GB of writes.

--------------------------------

------------------------------

So let's try something new - we move the gmap 6* folder after the first compile and put a hardlink (mklink) back so mkgmap cannot fuss around with them anymore. Yeah we're using croutches so to say. I will post again the results if I can improve by moving stuff and putting symlinks or by setting read only attribute.

On Wed, 8 Sept 2021 at 14:10, Felix Hartmann <extremecarver@gmail.com> wrote:

Quite a few people do not compile the contourlines each time they update a map, but reuse the old contourlines. This works fine if you insert them as say 1234*img (and they are named 12340000.img 12340001.img and so on).

However this does not work when providing the --gmapi option. Actually mkgmap tries to rewrite all those files again/convert them.

Could it please be possible for mkgmap to change this?

There could be the status quo with --gmapi

and there could be a new mode called

--gmapiminimal that only converts files input as .o5m or .osm.pbf or .osm but not as .img

--gpapiminimal would then work the same way as it does with .img files. This has two use cases.

a) if you have to resplit a tile because some tiles were breaking - and directly gave the --gmapiminimal option - no need to redo the whole .gmap file. 

b) you symlink all gmap folders with the contourlines (or other layers you do not recreate on each update) into the gmap folder and mkgmap only writes out the new stuff.

Yes on an nvme disk you barely notice the conversion - it's really quick. BUT it is not needed if you have the files and even more - it burns your NVME SSD disk. 

To my horror I noticed that one worldwide update of maps wrote 5TB of data to my disk (thanks to NVME SSD and fast processor this all took 28 hours). At that rate even datacenter NVME ssd disks are quickly turned into trash., consumer grade NVME disk will not make it half a year for me... In general maybe mkgmap could have an option to write temporary files to another disk? Then you use a RAMDisk for those files. Or mkgmap should keep them in memory. Modern servers often have 64 or 128GB of RAM so enough RAM for mkgmap not to waste writes to disk... 

Yes mkgmap uses quite a lot of RAM, but RAM is much cheaper than continous SSD writes....

Write now mkgmap crashes if you symlink already converted files to it:

e.g. message:

Number of MapFailedExceptions: 0
SEVERE (global): Error saving gmapi data
java.nio.file.FileAlreadyExistsException: .\mtb_alp_08.09.2021.gmap\Product1\75280000

While if it would be allowed to write out the data it would overwrite existing files (which is fine or should do so - but same way as for .img files)

For my worldwide maps 10m contourlines are 200GB, 20m contourlines are 100GB in size. Rendering the map in two styles and each time needing to convert them even though I already have them converted creates 600GB of useless writes per update (as well as useless time spend converting those 300GB at least twice) - and that is only if I use the --gmapi option as a separate step. If I give bot --tdb-file and --gmapi it gets worse if a tile failed to write - and I just parse it again.

And that is only for the contourlines. As I need three sets of mapset files - once without contourlines, once with 10m contourlines, and once with 20m contourlines the actual map gmapi folders are written three times instead of once... Well over 1TB of useless conversion time and useless data written to disk (but I do not know another way to create the needed mapset_mdr.img, mapset.img and mapset.tdb files)

I don't know how to change that myself, but I hope this is not much work to change.

--

Felix Hartman - Openmtbmap.org & VeloMap.org

--

Felix Hartman - Openmtbmap.org & VeloMap.org