[PATCH v1] make maps in parallel
OK, let it not be said that I don't listen to others! The attached patch provides support for making maps in parallel. By default, the behaviour is the same as before but if you specify --num-threads=N where N is greater than 1, it will process N maps at the same time and then combine the results (if required). Don't forget to increase the heap size appropriately. A quick test on the big box shows good speedup - specifying --num-threads=4 and 2GB VM size. I was seeing better than 380% utilisation with 8 cores in use. I suspect the performance limitation here will be VM size and memory system bandwidth. BTW - I don't think num-threads is actually the best name for the option, so please suggest alternatives. Cheers, Mark
FYI - a further test on the big box with all 16 cores in use showed a speedup of 600% when processing a tiled map using 8 threads. I was using a VM size of 1GB (not huge). Choosing the number of threads to be 1/2 of the available cores appears to make sense as the Java VM does use spare cores for other stuff (e.g. GC). Cheers, Mark
I used this patch on my 3core machine. I got about 75% improvement (using 2/3 cores instead of 1/3 without patch). No problems so would be great to integrate this into main branch (with switch to enable it). Mark Burton wrote:
FYI - a further test on the big box with all 16 cores in use showed a speedup of 600% when processing a tiled map using 8 threads. I was using a VM size of 1GB (not huge).
Choosing the number of threads to be 1/2 of the available cores appears to make sense as the Java VM does use spare cores for other stuff (e.g. GC).
Cheers,
Mark _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
__________ Information from ESET Smart Security, version of virus signature database 4063 (20090508) __________
The message was checked by ESET Smart Security.
__________ Information from ESET Smart Security, version of virus signature database 4065 (20090511) __________ The message was checked by ESET Smart Security. http://www.eset.com
I used this patch on my 3core machine. I got about 75% improvement (using 2/3 cores instead of 1/3 without patch). No problems so would be great to integrate this into main branch (with switch to enable it). This sounds cool, but I wonder if there are regression tests. So far I have not been put out by the mkgmap running time, but if it did the wrong thing that would be more troublesome.
Hi Greg,
I used this patch on my 3core machine. I got about 75% improvement (using 2/3 cores instead of 1/3 without patch). No problems so would be great to integrate this into main branch (with switch to enable it).
This sounds cool, but I wonder if there are regression tests. So far I have not been put out by the mkgmap running time, but if it did the wrong thing that would be more troublesome.
Well, this particular patch should not alter the order in which stuff is output so one could easily check that the same map is produced when using 1 thread or N threads. Cheers, Mark
participants (3)
-
Felix Hartmann -
Greg Troxel -
Mark Burton