May 2009 - mkgmap-dev - The mkgmap lists

[PATCH for Splitter] Reduce noise for too many area messages
by Andre Hinrichs 12 May '09

12 May '09

Hi! This is a small patch for the splitter to reduce the noise when a way/relation is in too many areas. Especially when max nodes value is low this messages can flood your screen. In my case this is especially the case for creating routable maps for the netherlands where max nodes is 500000. With this patch a message is printed only once. Hope, you like it... Regards Andre -- Neu: GMX FreeDSL Komplettanschluss mit DSL 6.000 Flatrate + Telefonanschluss für nur 17,95 Euro/mtl.!* http://dslspecial.gmx.de/freedsl-surfflat/?ac=OM.AD.PD003K11308T4569a

1 0

Re: Lower case option & special codes
by David 12 May '09

12 May '09

Hello Steve, Lower case letters are used by Etrex Vista HCx when you focuse an object with the cursor and Mapsource renders map with lower case letters when they are defined. I also confirm Etrex Vista HCx is able to display names in lower case letters on some POI types. Upper case letters seem to be vectorial letters for polylines. Does the option work with latin1 and codepage 1252 ? See lower case letters in action on the attached screen captures. On the capture "POI-lowercase-name", "Croghan 606m" is a pillar POI defined with the first part of name hidden (special code ~[0x1e]). You can see the full name on capture "POI-focus". Regards, David

3 2

Out of memory error in Nuvi
by maning sambale 12 May '09

12 May '09

Hi, This has been reported by several users of NUvi: Out of memory error when using 2nd or 3rd "go to" waypoint in Nuvi 200 and Nuvi 265WT The last data I distributed were date 20090325 using mkgmap trunk version dated 20090414. I'm still asking for more details (I suspect it's a data bug) to replicate the issue. Do others experience this? -- cheers, maning ------------------------------------------------------ "Freedom is still the most radical idea of all" -N.Branden wiki: http://esambale.wikispaces.com/ blog: http://epsg4253.wordpress.com/ ------------------------------------------------------

1 0

[PATCH v4] make maps in parallel
by Mark Burton 12 May '09

12 May '09

Whoops! fixed a bad bug whereby each map was being output to the same file. Not sure if the fix is very elegant but at least it's not being silly any more. Now limits the default value of max-jobs to 4 no matter how many cores you have as further testing shows that having more threads just burns CPU cycles but doesn't actually finish any quicker. I guess the memory system is limiting the performance and the CPUs are spinning waiting for access. Now showing a real speedup of around 240% (my earlier higher claim was based on CPU usage and I now realise that was erroneous, sorry). -------- Now defaults to creating a thread per core so without doing anything you should see a speedup on a SMP box when processing multiple maps. You can use --max-jobs=N to limit the concurrency - you may want to specify that if you can't increase the VM size to what is required. However, it occurs to me that if you can afford a box with more than 2 cores, then you can probably afford a reasonable amount of memory (otherwise, what's the point in having more cores?) Added help blurb. -------- OK, let it not be said that I don't listen to others! The attached patch provides support for making maps in parallel. By default, the behaviour is the same as before but if you specify --num-threads=N where N is greater than 1, it will process N maps at the same time and then combine the results (if required). Don't forget to increase the heap size appropriately. A quick test on the big box shows good speedup - specifying --num-threads=4 and 2GB VM size. I was seeing better than 380% utilisation with 8 cores in use. I suspect the performance limitation here will be VM size and memory system bandwidth. BTW - I don't think num-threads is actually the best name for the option, so please suggest alternatives. Cheers, Mark

1 0

[PATCH v3] make maps in parallel
by Mark Burton 11 May '09

11 May '09

Now defaults to creating a thread per core so without doing anything you should see a speedup on a SMP box when processing multiple maps. You can use --max-jobs=N to limit the concurrency - you may want to specify that if you can't increase the VM size to what is required. However, it occurs to me that if you can afford a box with more than 2 cores, then you can probably afford a reasonable amount of memory (otherwise, what's the point in having more cores?) Added help blurb. -------- OK, let it not be said that I don't listen to others! The attached patch provides support for making maps in parallel. By default, the behaviour is the same as before but if you specify --num-threads=N where N is greater than 1, it will process N maps at the same time and then combine the results (if required). Don't forget to increase the heap size appropriately. A quick test on the big box shows good speedup - specifying --num-threads=4 and 2GB VM size. I was seeing better than 380% utilisation with 8 cores in use. I suspect the performance limitation here will be VM size and memory system bandwidth. BTW - I don't think num-threads is actually the best name for the option, so please suggest alternatives. Cheers, Mark

1 0

[PATCH v2] make maps in parallel
by Mark Burton 11 May '09

11 May '09

Now defaults to creating a thread per core so without doing anything you should see a speedup on a SMP box when processing multiple maps. You can use --num-threads=N to limit the concurrency - you may want to specify that if you can't increase the VM size to what is required. However, it occurs to me that if you can afford a box with more than 2 cores, then you can probably afford a reasonable amount of memory (otherwise, what's the point in having more cores?) Added help blurb. -------- OK, let it not be said that I don't listen to others! The attached patch provides support for making maps in parallel. By default, the behaviour is the same as before but if you specify --num-threads=N where N is greater than 1, it will process N maps at the same time and then combine the results (if required). Don't forget to increase the heap size appropriately. A quick test on the big box shows good speedup - specifying --num-threads=4 and 2GB VM size. I was seeing better than 380% utilisation with 8 cores in use. I suspect the performance limitation here will be VM size and memory system bandwidth. BTW - I don't think num-threads is actually the best name for the option, so please suggest alternatives. Cheers, Mark

2 2

Commit: r1035: Avoid calling log.debug() when debug output is not enabled.
by svn commit 11 May '09

11 May '09

Version 1035 was commited by markb on 2009-05-11 22:19:55 +0100 (Mon, 11 May 2009) Avoid calling log.debug() when debug output is not enabled. This is especially important if the args are expensive to compute.

1 0

[PATCH v1] make maps in parallel
by Mark Burton 11 May '09

11 May '09

OK, let it not be said that I don't listen to others! The attached patch provides support for making maps in parallel. By default, the behaviour is the same as before but if you specify --num-threads=N where N is greater than 1, it will process N maps at the same time and then combine the results (if required). Don't forget to increase the heap size appropriately. A quick test on the big box shows good speedup - specifying --num-threads=4 and 2GB VM size. I was seeing better than 380% utilisation with 8 cores in use. I suspect the performance limitation here will be VM size and memory system bandwidth. BTW - I don't think num-threads is actually the best name for the option, so please suggest alternatives. Cheers, Mark

3 4

[PATCH v1] Experimental support for multi-threading
by Mark Burton 11 May '09

11 May '09

Hi Folks, I have recently gained access to a monster box containing 16 3GHz cores and 32 G of memory so I thought it would make an ideal machine to test a multi-threaded version of mkgmap. As a lot of the processing has to be done in sequence, the opportunities for using multiple threads are somewhat limited. However, the attached patch parallelises: 1 - processing of ways from OSM form to MapLine/MapRoad. 2 - processing of cities 3 - processing of POIs 4 - processing of polylines and polyshapes within each division At this time, the performance gain is useful but not stunning (typically, 25-30% speedup when using at least 4 cores). There is a small gain even when only using 2 cores. With the patch in place, it will default to using as many threads as you have cores. If you wish, you can explicitly specify the number of threads to use with the --num-threads=N option. You may specify more threads than you have cores but it won't make it any faster. Specifying less threads than cores is useful for testing purposes or disabling the parallel processing (by specifying just 1 thread). One side effect of the multi-threading is that the elements are written to the output file in a random order. If you specify the (existing) --preserve-element-order option, the order will be preserved but all of the threading will be disabled! The patch has received some testing using mapsource and I think I have removed the worst of the bugs but I would very much appreciate it if people could try it out and see if it causes any breakage. Obviously, I would also like to know how well it performs. As it has only been tested under Linux any reports from other platforms would be especially useful. Cheers, Mark

5 15

Way contains a semgment that is longer than 1683 (routing will fail for this way)
by Felix Hartmann 11 May '09

11 May '09

Creating maps for Europe I notice this warning more or less on any longer ferry connection. Could you try to artificially create in between points for such ways instead of breaking the routability? (off course this does not solve the more prominent problem of "relation/way XX in too many areas of the splitter - but at the latest once the splitter can/could handle ways in more than 4 areas then the problem would be much more prominent). Greetings, Felix

2 1