[PATCH v7] make maps in parallel
Now respects --num-jobs again (broken in last patch). --------- Now reports exceptions in the worker threads. --------- Here's a better fix than last night's effort for the problem where the mapname and description for each job were getting clobbered due to the way that the command args are processed. Each job now gets a "snapshot" of the command args so it doesn't matter if they subsequently get changed. --------- Whoops! fixed a bad bug whereby each map was being output to the same file. Not sure if the fix is very elegant but at least it's not being silly any more. Now limits the default value of max-jobs to 4 no matter how many cores you have as further testing shows that having more threads just burns CPU cycles but doesn't actually finish any quicker. I guess the memory system is limiting the performance and the CPUs are spinning waiting for access. Now showing a real speedup of around 240% (my earlier higher claim was based on CPU usage and I now realise that was erroneous, sorry). -------- Now defaults to creating a thread per core so without doing anything you should see a speedup on a SMP box when processing multiple maps. You can use --max-jobs=N to limit the concurrency - you may want to specify that if you can't increase the VM size to what is required. However, it occurs to me that if you can afford a box with more than 2 cores, then you can probably afford a reasonable amount of memory (otherwise, what's the point in having more cores?) Added help blurb. -------- OK, let it not be said that I don't listen to others! The attached patch provides support for making maps in parallel. By default, the behaviour is the same as before but if you specify --num-threads=N where N is greater than 1, it will process N maps at the same time and then combine the results (if required). Don't forget to increase the heap size appropriately. A quick test on the big box shows good speedup - specifying --num-threads=4 and 2GB VM size. I was seeing better than 380% utilisation with 8 cores in use. I suspect the performance limitation here will be VM size and memory system bandwidth. BTW - I don't think num-threads is actually the best name for the option, so please suggest alternatives. Cheers, Mark
Hi, Did some testing, using file: http://planet.openstreet.nl/planet-benelux-090513.osm.gz Split with: java -Xmx2048m -jar splitter.jar --mapid=31303001 --max-nodes=500000 planet-benelux-090513.osm.gz See outputs in the four directories at: http://www.amsk.nl/osmtest/ (slowish host, sorry). .../svn1035: mkgmap vanilla svn rev 1035 .../patchv7*: mkgmap 1035 + mb-parallel-maps-v7.patch + mb-fix-long-segments-v1.patch The mkgmap-invocations are listed in the "cmdline" files in the respective directories. Output and logging are in those directories as well. I've removed the input-files (except in the svn1035 directory; the others use the same input) and the gmapsupp.img-files (too large; let me know if you need them). Note especially the difference between "svn1035" and "patchv7_1thread" (not identical), and between "patchv7_4threads" and "patchv7_4threads.run2" (exact same inputs and commandline, different output). Hope this helps. If you need any other info/testing, let me know. Regards, Martin
Hi Martin, Thanks for the info. I am away now till Friday night so I am sorry that won't be able to look at it until the weekend. Cheers, Mark
Hi,
Did some testing, using file: http://planet.openstreet.nl/planet-benelux-090513.osm.gz
Split with: java -Xmx2048m -jar splitter.jar --mapid=31303001 --max-nodes=500000 planet-benelux-090513.osm.gz
See outputs in the four directories at: http://www.amsk.nl/osmtest/ (slowish host, sorry).
.../svn1035: mkgmap vanilla svn rev 1035 .../patchv7*: mkgmap 1035 + mb-parallel-maps-v7.patch + mb-fix-long-segments-v1.patch
The mkgmap-invocations are listed in the "cmdline" files in the respective directories. Output and logging are in those directories as well. I've removed the input-files (except in the svn1035 directory; the others use the same input) and the gmapsupp.img-files (too large; let me know if you need them).
Note especially the difference between "svn1035" and "patchv7_1thread" (not identical), and between "patchv7_4threads" and "patchv7_4threads.run2" (exact same inputs and commandline, different output).
Hope this helps. If you need any other info/testing, let me know.
Regards, Martin _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Mark I couldn't see anything wrong with your patch that might cause maps to be missed but it is hard to tell because of the way that CommandArgs works. So I've attached a patch that is based on your v7 patch that separates out CommandArgs into one class that reads them and another that contains the actual options. The options given to the processor are always copies so there is no need to further copy. There is also a tweek to make sure that the files are given to the combiners in the order they started. ..Steve
Hi Steve,
Hi Mark
I couldn't see anything wrong with your patch that might cause maps to be missed but it is hard to tell because of the way that CommandArgs works.
So I've attached a patch that is based on your v7 patch that separates out CommandArgs into one class that reads them and another that contains the actual options. The options given to the processor are always copies so there is no need to further copy.
There is also a tweek to make sure that the files are given to the combiners in the order they started.
Many thanks for the contribution, I shall look at it shortly. Unfortunately, I have just discovered another issue which is the static int nextPriority in GType - this needs to be changed to one instance per map but at the moment I can't work out what to use as a key to select the correct value for each map being processed concurrently. If you have any thoughts on that, please say. Cheers, Mark
Steve,
So I've attached a patch that is based on your v7 patch that separates out CommandArgs into one class that reads them and another that contains the actual options. The options given to the processor are always copies so there is no need to further copy.
If you are happy with the revamped CommandArgs stuff, why not commit it anyway rather than wait for the parallel maps issues to be sorted? Cheers, Mark
Hi
Unfortunately, I have just discovered another issue which is the static int nextPriority in GType - this needs to be changed to one instance per map but at the moment I can't work out what to use as a key to select the correct value for each map being processed concurrently. If you have any thoughts on that, please say.
I think it doesn't matter what the priority is, just that values that are obtained later are higher. So adding synchronized to getNextPriority() should do it. What is the nature of the problem you see? ..Steve
Hi Steve,
Hi
Unfortunately, I have just discovered another issue which is the static int nextPriority in GType - this needs to be changed to one instance per map but at the moment I can't work out what to use as a key to select the correct value for each map being processed concurrently. If you have any thoughts on that, please say.
I think it doesn't matter what the priority is, just that values that are obtained later are higher. So adding synchronized to getNextPriority() should do it. What is the nature of the problem you see?
I haven't yet seen any problems but just found the (potential) issue by searching for static data items that could cause problems when running stuff in parallel. As it happens, I have now coded around it by serialising the loading of the OSM data. It makes almost zero difference to the run-time so I am happy to leave GType as-is. Cheers, Mark
Hi Steve,
So I've attached a patch that is based on your v7 patch that separates out CommandArgs into one class that reads them and another that contains the actual options. The options given to the processor are always copies so there is no need to further copy.
I still have more work to do on the parallel code but before I even think of committing anything it would be nice to integrate your changes to the CommandArgs stuff. Can you please either commit those specific changes to the trunk or post a diff that only contains the changes to the options handling and not any of the parallel stuff because I would prefer to not squash those two changesets together. Cheers, Mark
participants (3)
-
Mark Burton -
Martin Marinus -
Steve Ratcliffe