- mkgmap-dev - The mkgmap lists

Re: [mkgmap-dev] max-jobs patch
by Henning Scholland 10 Feb '18

10 Feb '18

Hi Gerd, these are my runtimes of several compilations of my Turkey map, splitted with 1.500.000 nodes, compiled witout index, gmapi and gmappsupp, but including DEM and routing etc. RAM in GB Threads Time1 in ms Time2 in ms 12 8 328106 323605 8 8 323135 341242 4 8 378460 408714 4 7 364123 361501 4 6 349048 343866 4 4 371786 366581 4 1 845778 797376 2 4 527267 509698 2 1 804117 853805 It seems a lack of RAM slows more down if there are two many jobs. For 4GB Ram, 6 threads are faster than 8 threads, even using only 4 threads is faster than using 8. Henning On 10.02.2018 16:41, Gerd Petermann wrote: On 10.02.2018 16:41, Gerd Petermann wrote: > Hi Henning, > > thanks for the numbers. I assume that your JRE sets xmx=4G because you probably have 16G. > My machine has 4 cores and 8G and the JRE sets xmx=2G when I don't specify a value. > When you compile a large number of tiles peak memory is reached when building the index. > > Maybe you can do some more tests with a smaller set of tiles? > I'd like to know if the run time changes much when you compile e.g. 32 tiles with > a) -Xmx8GB and --max-jobs=8 > b) -Xmx4GB and --max-jobs=8 > c) -Xmx4GB and --max-jobs=7 > d) -Xmx4GB and --max-jobs=6 > > I suggest to do each test at least 2 times and start with one unmeasured test to reduce > IO caching effects. You should also not use --index, --gmapi or --gmapsupp for these tests. > > Gerd > > > ________________________________________ > Von: Henning Scholland <osm(a)hscholland.de> > Gesendet: Samstag, 10. Februar 2018 07:23 > An: Gerd Petermann; Development list for mkgmap > Betreff: Re: AW: [mkgmap-dev] max-jobs patch > > Hi Gerd, > Hi Mike, > > I gave it a try on all my maps (splitted with max-nodes=1.600.000). > Removed xmx and max jobs. mkgmap used all 8 logical cores and about 4GB > of memory. So it seems, Gerd was correct with the 500mb per job. So I > would suggest to limit max-jobs with 32bit java to 3 jobs and to > RAM/500mb on 64bit java. The used jobs should be minimum of both CPU and > RAM. > > Regarding to higher amount of RAM I was mentioning before, I was calling > mkgmap with xmx=12G, maybe java is not freeing the heap in that case. It > was during dem-branch. Maybe during that time mkgmap wasn't also that > efficient. > > Henning >

1 0

WG: AW: max-jobs patch
by Gerd Petermann 10 Feb '18

10 Feb '18

Sorry, answered to Henning instead of the list, see below... Gerd ________________________________________ Von: Gerd Petermann <gpetermann_muenchen(a)hotmail.com> Gesendet: Samstag, 10. Februar 2018 09:52 An: Henning Scholland Betreff: AW: AW: [mkgmap-dev] max-jobs patch Hi Henning, splitter uses multiple threads by default, but up to now only during the last phase that writes the tiles it can use more than 2. Not a big problem, most of the work is I/O bound. The workload in splitter is completely different to mkgmap. Peak memory depends on the --keep-complete option and the max-areas option, not so much on the number of parallel threads. In some cases I see slightly better throughput when I use --max-threads=6 on my 4 core machine, but only a few percent. The last time when I measured it I was quite happy with the defaults. Gerd ________________________________________ Von: Henning Scholland <osm(a)hscholland.de> Gesendet: Samstag, 10. Februar 2018 08:46 An: Gerd Petermann; Development list for mkgmap Betreff: Re: AW: [mkgmap-dev] max-jobs patch Hi Gerd, maybe splitter can be modified similar. Henning

1 0

Problem with gmapi-builder
by Carlos Dávila 09 Feb '18

09 Feb '18

Since mkgmap r4092 I get errors like "Missing part: 0 of GARMIN .RGN in IMG-file" or "Missing part: 0 of ~�?�.�� in IMG-file" when I use gmapi-builder.py to transform maps to gmap format. r4091 works fine with gmapi-builder. Resulting *.gmapi folder is almost empty with r4092 and above.

2 2

Commit r4106: Better detect errors in action command lists.
by svn commit 08 Feb '18

08 Feb '18

Version mkgmap-r4106 was committed by steve on Wed, 07 Feb 2018 Better detect errors in action command lists. It is no longer required to end set, add, setaccess and addaccess commands with a semicolon. http://www.mkgmap.org.uk/websvn/revision.php?repname=mkgmap&rev=4106

3 2

Commit r4107: Revert removal of semi-colons from lines file.
by svn commit 08 Feb '18

08 Feb '18

Version mkgmap-r4107 was committed by steve on Thu, 08 Feb 2018 Revert removal of semi-colons from lines file. They are note required, but it is probably clearer with them present. http://www.mkgmap.org.uk/websvn/revision.php?repname=mkgmap&rev=4107

1 0

Garmin uses overlapping tiles
by Gerd Petermann 07 Feb '18

07 Feb '18

Hi all, so far I've never seen the crash in MapSource using the Garmin maps, although they also have tiles with "invalid heights" and therefore the tile headers have the "Extra flag". my idea: - use mkgmap to create DEM data for exactly the same two areas as in a Garmin map [1] - Try to reproduce the crash with those two areas - replace Garmin DEM by mkgmap DEM and see what happens in MapSource The interesting point: Garmin tiles overlap. I've created this areas.list from the information in two Garmin TRE files: 00000029: 2174824,745652 to 2190360,768956 # 46,6667,16,0000 47,0000,16,5000 00000030: 2174824,768952 to 2190360,792260 # 46,6667,16,4999 47,0000,17,0001 Note that the first tile has eastern border at 768956 while the 2nd uses 768952 for the western border. When I use exactly those values I cannot reproduce the crash. When I move the boundary of one of the two tiles so that they are equal I can reproduce the crash, but not always with the same routes. I tried 768952, 768954, 768956. So, I tried to change mkgmap so that it calculates DEM as if the tile boundaries are overlapping, but with TRE where tiles have same 768956 --> crash possible. The next interesting point is that the 0x4a polygons of these two tiles do NOT overlap. They seem to share 768896 (!). So, I used a new areas.list with this value and still see the crash. My conclusion: We have to modify mkgmap to write a different TRE file, probably it has to cover the DEM area as well. Does that make sense? Gerd [1] TopoGuide Hungary 2.12c

4 11

Re: [mkgmap-dev] Garmin uses overlapping tiles
by Frank Stinner 06 Feb '18

06 Feb '18

Hi Gerd, i have only picked 1 IMG from the "Garmin TransAlpin v2" (Immenstadt im Allgau / 16596679.img). The TRE-borders are overlapping with 2 units with all the 4 neighbors. Perhaps a border-value is ever even (?). Frank

1 0

MapSource crash with DEM: Maybe tdb file needs changes?
by Gerd Petermann 05 Feb '18

05 Feb '18

Hi, Ive discussed with Frank if there might be a field in the tdb file that has to be changed. Maybe there is a value that tells MapSource something about the alignment of DEM data? I don't have a single original Garmin tdb file :-( It seems that mkgmap writes some constants that might be important, e.g. MapDetailBlock.java contains //01 c3 00 ff os.write4(0xff00c301); os.write(0); os.write(0); os.write(0); The wiki [1] describes these 7 bytes as "byte:0 has copyright text,byte 1: has 0x53 text : byte 2:if 1 then bytes 3_7: an unknown total (density) " The display tool seems to expect something similar to the wiki: d.rawValue(2, "01 01"); d.byteValue("intro flag"); d.intValue("intro %d"); Any idea what this "intro" or "unknown total(density) could be ? And what is the 0x53 text? [1]https://wiki.openstreetmap.org/wiki/OSM_Map_On_Garmin/TDB_File_Format Gerd

5 4

My findings about the crash in MapSource
by Gerd Petermann 05 Feb '18

05 Feb '18

Hi all, I found out that whenever mkgmap r4100 wrote a tile with DEM data that has the "extra flag" set to 2 (which means that the tile contains invalid height information which should be ignored) I was able to reproduce the crash in MapSource when clicking on "Show profile..." for a route which crosses the boundaries of this tile I found one error in the code for the --dem-poly option. This produced too many "invalid" values, even for tiles which are completely inside the bounding polygon. I fixed that with r4101. Still, the same problem occurs when a hgt file contains too many voids or the tile is in fact partly outside the polygon. It seems that Andrzejs patch dem-align-6.patch is part of the problem. When I disable parts of the code with the attached patch the problem seems to disappear. Gerd

5 12

Commit r4103: improve description of DEM options
by svn commit 05 Feb '18

05 Feb '18

Version mkgmap-r4103 was committed by gerd on Sat, 03 Feb 2018 improve description of DEM options dem-options2.patch by Carlos Dávila, changes in options.txt should be similar http://www.mkgmap.org.uk/websvn/revision.php?repname=mkgmap&rev=4103

2 1