Why does splitter limit the number of threads?
Hello list, the current implementation limits the number of threads to the number of available cores. I changed the code to allow +2 threads. This results in a much better throughput on my dual core machine (e.g. germany.osm.pbf is 100 seconds or 17% faster) with r190 + patchv7. What could be the reason for this limit? ciao, Gerd -- View this message in context: http://gis.638310.n2.nabble.com/Why-does-splitter-limit-the-number-of-thread... Sent from the Mkgmap Development mailing list archive at Nabble.com.
On Fri, Nov 25, GerdP wrote:
Hello list,
the current implementation limits the number of threads to the number of available cores. I changed the code to allow +2 threads. This results in a much better throughput on my dual core machine (e.g. germany.osm.pbf is 100 seconds or 17% faster) with r190 + patchv7. What could be the reason for this limit?
Let's say it in this way: In my daily work the experience is, that most of the time the limitating factor is not the number of cores, but the I/O bandwith of your harddisk. While in your example with a dual core machine, the I/O Bandwith should be enough, for some of our machines with a 2 digit number of cores, limitating the number of processes to number of cores/2 or even more makes it much faster. That's why I think the number of threads should be a config option which the user can adjust to his hardwre and not a hard number. Thorsten -- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg)
On 25/11/11 12:57, GerdP wrote:
Hello list,
the current implementation limits the number of threads to the number of available cores. I changed the code to allow +2 threads. This results in a much better throughput on my dual core machine (e.g. germany.osm.pbf is 100 seconds or 17% faster) with r190 + patchv7. What could be the reason for this limit?
I've no idea. The restriction should be removed in my opinion. ..Steve
OK, so it seems that we all think that the should be allowed to set higher values. The attached patch does that. It still uses the number of cores if --max-threads is not specified or set to "auto". http://gis.638310.n2.nabble.com/file/n7031578/max_threads.patch max_threads.patch ciao, Gerd -- View this message in context: http://gis.638310.n2.nabble.com/Why-does-splitter-limit-the-number-of-thread... Sent from the Mkgmap Development mailing list archive at Nabble.com.
Hello list,
the current implementation limits the number of threads to the number of available cores. I changed the code to allow +2 threads. This results in a much better throughput on my dual core machine (e.g. germany.osm.pbf is 100 seconds or 17% faster) with r190 + patchv7. What could be the reason for this limit?
ciao, Gerd
Hi, I think I have introduced this limitation a long time ago with the first multithreaded implementation of splitter. I remember that in this first multithreaded implementation it was very important that the reader thread was not dormant. The limitation of the max-threads parameter ensured that. But that's a long time ago and I am happy removing this limitation if it improves the performance (at least for someone). WanMil
participants (4)
-
GerdP -
Steve Ratcliffe -
Thorsten Kukuk -
WanMil