Re: [mkgmap-dev] [PATCH v9] make maps in parallel
Hi Greg,
Is the default for your patch to not become parallel? It seems that making mkgmap parallel is hard, and my personal utility function places correct behavior very high and speed not that much. Until the code is really shaken out, I don't want to run parallel. I don't know what others think, but just a thought.
At this time, it defaults to creating as many threads as there are cores. I don't have a big issue with making the default number of threads = 1 if that's what people want. How about this: 1 - with no option specified, the default number of jobs (aka threads) is 1. 2 - specifying --max-jobs without a value will maximise the number of jobs (i.e. create one thread per core). 3 - specifying --max-jobs=N will create N threads. Would that suit you? Cheers, Mark
Mark Burton <markb@ordern.com> writes:
Is the default for your patch to not become parallel? It seems that making mkgmap parallel is hard, and my personal utility function places correct behavior very high and speed not that much. Until the code is really shaken out, I don't want to run parallel. I don't know what others think, but just a thought.
At this time, it defaults to creating as many threads as there are cores. I don't have a big issue with making the default number of threads = 1 if that's what people want.
How about this:
1 - with no option specified, the default number of jobs (aka threads) is 1.
2 - specifying --max-jobs without a value will maximise the number of jobs (i.e. create one thread per core).
3 - specifying --max-jobs=N will create N threads.
Would that suit you?
That sounds great. I think it would be reasonable to change the default to #2 after enough time has passed that we are really sure that there are no problems from running in parallel. Perhaps there could be a regression test to compare the output from parallel and non-parallel runs. I'm not sure if they are meant to be bit-for-bit identical, or if not if you can articulate the allowable differences. This might require code to parse a .img into canonical form for semantic diffing, but that's probably a useful thing to have anyway.
On Mon, May 18, 2009 at 07:59:43AM -0400, Greg Troxel wrote:
How about this:
1 - with no option specified, the default number of jobs (aka threads) is 1.
2 - specifying --max-jobs without a value will maximise the number of jobs (i.e. create one thread per core).
3 - specifying --max-jobs=N will create N threads.
Would that suit you?
That sounds great. I think it would be reasonable to change the default to #2 after enough time has passed that we are really sure that there are no problems from running in parallel.
Perhaps there could be a regression test to compare the output from parallel and non-parallel runs. I'm not sure if they are meant to be bit-for-bit identical, or if not if you can articulate the allowable differences. This might require code to parse a .img into canonical form for semantic diffing, but that's probably a useful thing to have anyway.
+1. Adding some regression testing and pretty-printing infrastructure can never hurt. I haven't tested these patches, because my laptop is only 1.5 GiB and dual-core. I'm not sure if two tiles of Finland would fit in the memory at once. Marko
On Mon, 18 May 2009, Mark Burton wrote:
How about this:
1 - with no option specified, the default number of jobs (aka threads) is 1.
2 - specifying --max-jobs without a value will maximise the number of jobs (i.e. create one thread per core).
3 - specifying --max-jobs=N will create N threads.
Would that suit you?
This would be the best approach (and it's just like 'make -j'). Not everybody wants to clog their computer with a mkgmap run. If map generation really is time consuming it could also be done on one core in the background while the computer is responsive for other tasks. In addition, parallel mkgmap requires more memory which is not available on all computers. For instance the laptop I am writing this mail on has a dual core but only 1GB memory, that's scarce even for one mkgmap thread. -Wolfgang
participants (4)
-
Greg Troxel -
Mark Burton -
Marko Mäkelä -
Wolfgang v. Hansen