Hi Gerd, please find attached v2 of the max-jobs patch. This version additionally catches out of heap memory exceptions and displays a message suggesting either the use of the Java -Xmx option to increase the available heap memory or the mkgmap --max-jobs option with a smaller value to reduce the memory requirement (the latter is suggested only if using more than one thread). How does that seem? Regards, Mike -----Original Message----- From: Gerd Petermann [mailto:gpetermann_muenchen@hotmail.com] Sent: 05 February 2018 07:28 To: Mike Baggaley <mike@tvage.co.uk>; 'Development list for mkgmap' <mkgmap-dev@lists.mkgmap.org.uk> Subject: Re: [mkgmap-dev] max-jobs patch Hi Mike, thought about this again. Maybe this change is too simple. With multiple jobs one also needs more heap (-Xmx JRE option). If you create rather large tiles with splitter (max-nodes=1600000) you need 0.5 - 1 GB for each job. Not sure what happens when a user creates a map with 10 tiles on an 8-core machine without any -Xmx option. I fear this will result in OutOfMemory exception, so better check the available heap as well. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Mike Baggaley <mike@tvage.co.uk> Gesendet: Sonntag, 4. Februar 2018 15:14 An: 'Development list for mkgmap' Betreff: [mkgmap-dev] max-jobs patch Hi Gerd, Please find attached a patch that amends the default behaviour if the --max-jobs option is not specified, to using a value equal to the number of CPU cores, as suggested in a previous post. The documentation is also amended to reflect the change. This halves the execution time of mkgmap for building a map of Staffordshire on my 8-core PC when --max-jobs is not specified (I didn't know about this option previously and was unaware the performance could have been improved). Cheers, Mike

I don't know, if Nordrhein-Westfalen is that kind of large tiles you meant. Splitter ended up with 43 files and i tried to make mapsource maps with and without --max-jobs option and mkgmap r4000. My computer does the job with 43 files about 12minutes faster than without this option. My laptop is a core i5 with 8Gb of memory. The command i use is like this line : java -Xmx7500m -XX:-UseGCOverheadLimit -ea -jar -Dlog.config=logging.properties C:\PROGRA~2\KartenTools\mkgmap\mkgmap.jar --location-autofill=is_in,nearest --mapname=70060001 --family-id=7006 .\styles\mystyle.txt --series-name="STYLE TEST" --family-name="OpenStreetmap" --country-name=STYLETEST --country-abbr=STY --area-name=STY --overview-mapname="overview" --latin1 --style-file=.\styles --style=myland --max-jobs --keep-going --check-roundabouts --drive-on=detect,right --output-dir=.\out\MYSTYLE --index --bounds=mybounds --route --name-tag-list=name,place_name,loc_name --housenumbers --x-split-name-index --add-pois-to-areas --precomp-sea=.\input\sea.zip  --tdbfile --draw-priority=10 --gmapsupp -c .\data\MYSTYLE\template.args --description=MYSTYLE --remove-ovm-work-files=true Hope this gave you some useful hints Am 07.02.2018 um 20:06 schrieb Gerd Petermann:

Hi Mike,

I did not yet try it. My understanding is that the Garbage Collection will try hard to avoid an out of heap exception. In other words: A user might see better throughput with fewer parallel jobs. What are your experiences? Did you try with large maps, say > 40 large tiles ?

Gerd

________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Mike Baggaley <mike@tvage.co.uk> Gesendet: Mittwoch, 7. Februar 2018 01:58 An: 'Development list for mkgmap' Betreff: Re: [mkgmap-dev] max-jobs patch

Hi Gerd, please find attached v2 of the max-jobs patch.

This version additionally catches out of heap memory exceptions and displays a message suggesting either the use of the Java -Xmx option to increase the available heap memory or the mkgmap --max-jobs option with a smaller value to reduce the memory requirement (the latter is suggested only if using more than one thread).

How does that seem? Regards, Mike

-----Original Message----- From: Gerd Petermann [mailto:gpetermann_muenchen@hotmail.com] Sent: 05 February 2018 07:28 To: Mike Baggaley <mike@tvage.co.uk>; 'Development list for mkgmap' <mkgmap-dev@lists.mkgmap.org.uk> Subject: Re: [mkgmap-dev] max-jobs patch

Hi Mike,

thought about this again. Maybe this change is too simple. With multiple jobs one also needs more heap (-Xmx JRE option). If you create rather large tiles with splitter (max-nodes=1600000) you need 0.5 - 1 GB for each job. Not sure what happens when a user creates a map with 10 tiles on an 8-core machine without any -Xmx option.

I fear this will result in OutOfMemory exception, so better check the available heap as well.

Gerd

________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Mike Baggaley <mike@tvage.co.uk> Gesendet: Sonntag, 4. Februar 2018 15:14 An: 'Development list for mkgmap' Betreff: [mkgmap-dev] max-jobs patch

Hi Gerd,

Please find attached a patch that amends the default behaviour if the --max-jobs option is not specified, to using a value equal to the number of CPU cores, as suggested in a previous post. The documentation is also amended to reflect the change. This halves the execution time of mkgmap for building a map of Staffordshire on my 8-core PC when --max-jobs is not specified (I didn't know about this option previously and was unaware the performance could have been improved).

Cheers, Mike

_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

-- Mit freundlichen Grüßen ##################################################### Manfred Haiduk, Zum Fischbach 9, 52393 Hürtgenwald e-mail mhaiduk@t-online.de #####################################################

Hi Gerd, Maybe my map is somehow special, but mkgmap definitely need more than 1gb heap per job. I will check on the weekend. Can you please post a patched mkgmap.jar? How about doing it the other way round and suggest the user to use max-job option if mkgmap think it is useful? Henning On 8 Feb 2018, 16:38, at 16:38, Gerd Petermann <gpetermann_muenchen@hotmail.com> wrote:

Hi Manfred,

thanks for testing. Just to make this clear: I always use --max-jobs, because I also always use -Xmx whenever I compile many tiles.

In this case a large tile is one that results in a large *.img (one with many MB). Typically this depends on the --max-nodes option for splitter and the style and the mkgmap options (routable or not, without or without DEM etc).

The max-jobs option allows to specify the number of parallel jobs, e.g. --max-jobs=5 would start 5 parallel jobs (threads). Whenever one input file is done, the next one is started until all are finished. The patch only changes the default from 1 to "number of cores". You can also use max-jobs=10 on a 2-core cpu, but this will slow down the run time.

What I tried to point out is that a user who doesn't know about the --max--jobs option might as well not know about the meaning of the -Xmx JRE option.

What happens if you start mkgmap on an 8-core machine with max-jobs=8 and without -Xmx option? I think it will crash, and with the 2nd version of the patch it will tell the user to look at the -Xmx option. So far so good. What happens on a 4 core machine with max-jobs=4 and no -Xmx option? What if the OS is 32 bit only? This would not allow much more than -Xmx1700m. What if the option is used but set to a rather low value, e.g. -Xmx=1000m?

My current thinking is this: If --max-jobs is not specified mkgmap should check the number of cores and the available heap and assume that one thread needs ~500M.

So, I would use something like num = Math.min(Runtime.getRuntime().maxMemory() / (512 * 1024 * 1024), Runtime.getRuntime().availableProcessors()) as default, maybe also for the case that one specifies --max-jobs .

How does that sound?

Gerd

________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Manfred Haiduk <mhaiduk@t-online.de> Gesendet: Mittwoch, 7. Februar 2018 22:28 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] max-jobs patch

I don't know, if Nordrhein-Westfalen is that kind of large tiles you meant. Splitter ended up with 43 files and i tried to make mapsource maps with and without --max-jobs option and mkgmap r4000. My computer does the job with 43 files about 12minutes faster than without this option. My laptop is a core i5 with 8Gb of memory. The command i use is like this line :

java -Xmx7500m -XX:-UseGCOverheadLimit -ea -jar -Dlog.config=logging.properties C:\PROGRA~2\KartenTools\mkgmap\mkgmap.jar --location-autofill=is_in,nearest --mapname=70060001 --family-id=7006 .\styles\mystyle.txt --series-name="STYLE TEST" --family-name="OpenStreetmap" --country-name=STYLETEST --country-abbr=STY --area-name=STY --overview-mapname="overview" --latin1 --style-file=.\styles --style=myland --max-jobs --keep-going --check-roundabouts --drive-on=detect,right --output-dir=.\out\MYSTYLE --index --bounds=mybounds --route --name-tag-list=name,place_name,loc_name --housenumbers --x-split-name-index --add-pois-to-areas --precomp-sea=.\input\sea.zip --tdbfile --draw-priority=10 --gmapsupp -c .\data\MYSTYLE\template.args --description=MYSTYLE --remove-ovm-work-files=true

Hope this gave you some useful hints

Am 07.02.2018 um 20:06 schrieb Gerd Petermann:

...

Hi Mike,

I did not yet try it. My understanding is that the Garbage Collection will try hard to avoid an out of heap exception. In other words: A user might see better throughput with fewer parallel jobs. What are your experiences? Did you try with large maps, say > 40 large tiles ?

Gerd

________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Mike Baggaley <mike@tvage.co.uk> Gesendet: Mittwoch, 7. Februar 2018 01:58 An: 'Development list for mkgmap' Betreff: Re: [mkgmap-dev] max-jobs patch

Hi Gerd, please find attached v2 of the max-jobs patch.

This version additionally catches out of heap memory exceptions and displays a message suggesting either the use of the Java -Xmx option to increase the available heap memory or the mkgmap --max-jobs option with a smaller value to reduce the memory requirement (the latter is suggested only if using more than one thread).

How does that seem? Regards, Mike

-----Original Message----- From: Gerd Petermann [mailto:gpetermann_muenchen@hotmail.com] Sent: 05 February 2018 07:28 To: Mike Baggaley <mike@tvage.co.uk>; 'Development list for mkgmap' <mkgmap-dev@lists.mkgmap.org.uk> Subject: Re: [mkgmap-dev] max-jobs patch

Hi Mike,

thought about this again. Maybe this change is too simple. With multiple jobs one also needs more heap (-Xmx JRE option). If you create rather large tiles with splitter (max-nodes=1600000) you need 0.5 - 1 GB for each job. Not sure what happens when a user creates a map with 10 tiles on an 8-core machine without any -Xmx option.

I fear this will result in OutOfMemory exception, so better check the available heap as well.

Gerd

________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Mike Baggaley <mike@tvage.co.uk> Gesendet: Sonntag, 4. Februar 2018 15:14 An: 'Development list for mkgmap' Betreff: [mkgmap-dev] max-jobs patch

Hi Gerd,

Please find attached a patch that amends the default behaviour if the --max-jobs option is not specified, to using a value equal to the number of CPU cores, as suggested in a previous post. The documentation is also amended to reflect the change. This halves the execution time of mkgmap for building a map of Staffordshire on my 8-core PC when --max-jobs is not specified (I didn't know about this option previously and was unaware the performance could have been improved).

Cheers, Mike

_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

--

Mit freundlichen Grüßen

##################################################### Manfred Haiduk, Zum Fischbach 9, 52393 Hürtgenwald e-mail mhaiduk@t-online.de #####################################################

_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Hi Gerd, Hi Mike, I gave it a try on all my maps (splitted with max-nodes=1.600.000). Removed xmx and max jobs. mkgmap used all 8 logical cores and about 4GB of memory. So it seems, Gerd was correct with the 500mb per job. So I would suggest to limit max-jobs with 32bit java to 3 jobs and to RAM/500mb on 64bit java. The used jobs should be minimum of both CPU and RAM. Regarding to higher amount of RAM I was mentioning before, I was calling mkgmap with xmx=12G, maybe java is not freeing the heap in that case. It was during dem-branch. Maybe during that time mkgmap wasn't also that efficient. Henning

Hi tested on my linux box, looks good Heap is 4.4 GB with 8 threads. With the unpatched version i'm using -Xmx8G for the heap Bernd Am Donnerstag, 8. Februar 2018, 15:34:23 CET schrieb Gerd Petermann:

binary is here: http://files.mkgmap.org.uk/download/415/mkgmap.jar

For those users who already used --max-jobs with or without a paramter before there should be no difference.

Hi Gerd the unpatched version uses up to 8.7 GB with the options for mkgmap I'm using maxnodes=512000 to create an area.list and use this file for every build. At the moment i see no problem with the patched version Bernd Am Samstag, 10. Februar 2018, 09:56:41 CET schrieb Gerd Petermann:

Hi Bernd,

okay, I think that confirms my estimate of ~500M per thread. See also my previous answer to Henning.

Gerd

________________________________________

Hi Gerd, I have put together the attached patch that should result in a reasonable value for max-jobs if it is defaulted. This version attempts to calculate a suitable value by processing one task, seeing how much memory it used, then dividing the available memory by the memory used to determine the value before running the remaining tasks. I have tried a variety of values of -Xmx on my PC, and it seems to work for both a county and the whole of the UK. I haven't tried it on a continent, as I don't have suitable data. I can't guarantee that every combination of memory and CPU will work, but if it does fail, you will still get a sensible error message telling you what to change. At the end of the run, if insufficient memory is allocated to the Java heap to allow all the CPU cores to be used, it will suggest a value for -Xmx. For my maps, it seems to suggest about 400Mb per thread. My build of the UK without specifying --max-jobs has reduced from about 3 hours to 45 minutes. I have also made a few further minor improvements to the documentation to make the form more consistent, and changed the logged execution time from milliseconds to a more useful hours, minutes and seconds (or milliseconds when less than 1 second). Regards, Mike -----Original Message----- From: Gerd Petermann [mailto:gpetermann_muenchen@hotmail.com] Sent: 07 February 2018 19:06 To: Development list for mkgmap <mkgmap-dev@lists.mkgmap.org.uk> Subject: Re: [mkgmap-dev] max-jobs patch Hi Mike, I did not yet try it. My understanding is that the Garbage Collection will try hard to avoid an out of heap exception. In other words: A user might see better throughput with fewer parallel jobs. What are your experiences? Did you try with large maps, say > 40 large tiles ? Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Mike Baggaley <mike@tvage.co.uk> Gesendet: Mittwoch, 7. Februar 2018 01:58 An: 'Development list for mkgmap' Betreff: Re: [mkgmap-dev] max-jobs patch Hi Gerd, please find attached v2 of the max-jobs patch. This version additionally catches out of heap memory exceptions and displays a message suggesting either the use of the Java -Xmx option to increase the available heap memory or the mkgmap --max-jobs option with a smaller value to reduce the memory requirement (the latter is suggested only if using more than one thread). How does that seem? Regards, Mike -----Original Message----- From: Gerd Petermann [mailto:gpetermann_muenchen@hotmail.com] Sent: 05 February 2018 07:28 To: Mike Baggaley <mike@tvage.co.uk>; 'Development list for mkgmap' <mkgmap-dev@lists.mkgmap.org.uk> Subject: Re: [mkgmap-dev] max-jobs patch Hi Mike, thought about this again. Maybe this change is too simple. With multiple jobs one also needs more heap (-Xmx JRE option). If you create rather large tiles with splitter (max-nodes=1600000) you need 0.5 - 1 GB for each job. Not sure what happens when a user creates a map with 10 tiles on an 8-core machine without any -Xmx option. I fear this will result in OutOfMemory exception, so better check the available heap as well. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Mike Baggaley <mike@tvage.co.uk> Gesendet: Sonntag, 4. Februar 2018 15:14 An: 'Development list for mkgmap' Betreff: [mkgmap-dev] max-jobs patch Hi Gerd, Please find attached a patch that amends the default behaviour if the --max-jobs option is not specified, to using a value equal to the number of CPU cores, as suggested in a previous post. The documentation is also amended to reflect the change. This halves the execution time of mkgmap for building a map of Staffordshire on my 8-core PC when --max-jobs is not specified (I didn't know about this option previously and was unaware the performance could have been improved). Cheers, Mike

Hi Gerd, I would expect the code to work on a MAC, because com.sun.management is part of the JDK, and defines the interface by which specific platforms can provide subinterfaces with further information (AFAIK there is only one for UNIX). However, I don't have a MAC, so can't test it on there. If it is a problem, I could take out that bit of code, which is just determining how much physical memory is installed, so that mkgmap won't suggest that the user increases the available heap too much. It isn't used in determining a value for maxjobs. Regards, Mike -----Original Message----- From: Gerd Petermann [mailto:gpetermann_muenchen@hotmail.com] Sent: 10 February 2018 15:45 To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] max-jobs patch Hi Mike, I've compiled r4107 with your patch: http://files.mkgmap.org.uk/download/418/mkgmap.jar I did not yet try it because I am still fighting with the crash in MapSource. I am not sure if the usage of classes like com.sun.management.OperatingSystemMXBean are supported with openjdk? Will this also work on a Mac? Gerd -- Sent from: http://gis.19327.n8.nabble.com/Mkgmap-Development-f5324443.html

I tried renaming the java library to force a failure and got a java error "Error occurred during initialization of VM" before any mkgmap code was executed, so to be on the safe side, I have removed that code and amended the message in the attached patch. Regards, Mike -----Original Message----- From: Steve Ratcliffe [mailto:steve@parabola.me.uk] Sent: 13 February 2018 23:01 To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] max-jobs patch Hi Mike

I could take out that bit of code, which is just determining how much physical memory is installed, so that mkgmap won't suggest that the user increases the available heap too much. It isn't used in determining a value for maxjobs.

Since it is just used for an informational message, you could just surround it with a try/catch block. Then if it doesn't exist on a particular it isn't a problem when using the downloaded pre-compiled distribution. Of course it would fail to compile on such a platform, but then hopefully someone will let us know and we can decide what to do then. Steve

Hi Mike, looks good to me. I am just not sure about the new stdout message "Setting max-jobs to 4" It appears when --max-jobs is not used (ok), but also when it is used without a parameter. In the latter case I would only print this message if the number of CPU cores is higher. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Mittwoch, 14. Februar 2018 19:38:06 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] max-jobs patch Hi Mike, okay, I'll try it tomorrow. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Mike Baggaley <mike@tvage.co.uk> Gesendet: Mittwoch, 14. Februar 2018 18:30:28 An: 'Steve Ratcliffe'; mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] max-jobs patch I tried renaming the java library to force a failure and got a java error "Error occurred during initialization of VM" before any mkgmap code was executed, so to be on the safe side, I have removed that code and amended the message in the attached patch. Regards, Mike -----Original Message----- From: Steve Ratcliffe [mailto:steve@parabola.me.uk] Sent: 13 February 2018 23:01 To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] max-jobs patch Hi Mike

I could take out that bit of code, which is just determining how much physical memory is installed, so that mkgmap won't suggest that the user increases the available heap too much. It isn't used in determining a value for maxjobs.

Since it is just used for an informational message, you could just surround it with a try/catch block. Then if it doesn't exist on a particular it isn't a problem when using the downloaded pre-compiled distribution. Of course it would fail to compile on such a platform, but then hopefully someone will let us know and we can decide what to do then. Steve _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev