May 2009 - mkgmap-dev - The mkgmap lists

Re: [mkgmap-dev] [PATCH v9] make maps in parallel
by Mark Burton 19 May '09

19 May '09

Hi Greg, > Is the default for your patch to not become parallel? It seems that > making mkgmap parallel is hard, and my personal utility function places > correct behavior very high and speed not that much. Until the code is > really shaken out, I don't want to run parallel. I don't know what > others think, but just a thought. At this time, it defaults to creating as many threads as there are cores. I don't have a big issue with making the default number of threads = 1 if that's what people want. How about this: 1 - with no option specified, the default number of jobs (aka threads) is 1. 2 - specifying --max-jobs without a value will maximise the number of jobs (i.e. create one thread per core). 3 - specifying --max-jobs=N will create N threads. Would that suit you? Cheers, Mark

4 3

test files to figure out Garmin encodings?
by Greg Troxel 18 May '09

18 May '09

It seems there is a table of a lot of Garmin codepoints, and I wonder if havinga test file that has a POI with each possible codepoint, and similarly for roads, would help determine the behavior/meanings for the rest of them. Perhaps things are 100% figured out, but it feels like not. The basic idea is to take a lat/lon to start (to make the map 'near' someone working on this) and then to have a grid of points spaced 50m, roughly square, that has each possible codepoint in some range known to exist. Then a second grid with a N-S road and then every 50m E-W roads of every type on the left and on the right some normal road with every access combination. They could be be labeled with names that have the codepoint, and eventually the meaning, drawing from the database of meaning/codepoint mappings. Once this was written other people could more easily experiment to add knowledge.

5 6

Entry in style file leads to infinite loop
by Johann Gail 18 May '09

18 May '09

Hello, when I instert the following line into the style file, then mkgmap hangs with 100% cpu load after reading in some osm data. highway=* & access=agricultural {delete access; set motorcar = no;set motorcycle = no;} From looking at the log I assume, but can not be sure, that the freeze occurs at the first occurence of access=agricultural. Does anyone see, what I'm doing wrong, or is there a known bug in mkgmap? Any hint, how can I debug this case? Regards, Johann

3 4

Bug Report: First rule matches in style files ... not quite!
by Thilo Hannemann 18 May '09

18 May '09

It says in the documentation that the first rule in the style file that matches the elements is the one used. That is not quite true! The reason is, that the code handles the rules by putting them into baskets organized by the first condition of the rule (with some exceptions). There is for example a basket for all rules with the first condition "highway=residential". When deciding which rule to apply the code will iterate through all the tags of the osm element. It will determine whether there is a corresponding basket and then apply all the rules in that basket in sequence, until one fits. This does work only in simple cases. For example see a way with the following tags: highway=residential rcn=yes This is a residential way that is part of the regional cycle network. (Probably derived by using the "relations" style file to add the tag "rcn=yes" to all ways that are part of an rcn network.) The lines style file will look like -- rcn=yes [ ... some attributes for ways of the cycle network ... ] highway=residential [ ... some other attributes for generic residential roads ... ] -- If the documentation was right the way should always match the first rule (rcn=yes ...). But that is not the case, because it depends on which of the two tags is handled first. This in turn depends on which tag comes first in the osm data. If the tag "rcn=yes" is handled first, the first rule will match, because there is a fitting basket "rcn=yes". This would be according to the documentation. If the tag "highway=residential" is handled first, that rule will (unexpectedly) match, because there is also a fitting basket "highway=residential". To summarize the problem: The baskets are not ordered, therefore the application of the rules is only "until first hit" inside a basket. Between baskets it is "first tag in the osm data that fits". To make matters worse the code also tries to optimize the rules, so that the simple rule "first condition determines basket" does not hold. By inspecting the code it is possible to tweak the rules so that everything works ok, but if somebody can't read the code and tries to build complex style files he will run into serious trouble. Best regards Thilo

3 4

bad file format error
by frmas 18 May '09

18 May '09

Hi, During the run of mkgmap java -Xmx768M -jar mkgmap.jar --latin1 --mapname="09030406" cartes/paris.osm I got that message : Error at line 148424, col 84 Bad file format: cartes/paris.osm Looking at that line, it seems it's the "é" of "josé" that gives that error. <node id='94135255' lat='48.8761986' lon='2.423346' user='nitrix' xapi:users='josé,josÃ©m,Esperanza36,Charlie Echo,Hardangels,nitrix' timestamp='2009-05-17T09:53:48Z' uid='125299' version='28' changeset='1219222'> Francois --

4 11

[PATCH v8] make maps in parallel
by Mark Burton 18 May '09

18 May '09

Now serialises reading of style files and map source to avoid reentrancy issue in GType. Reworked top-level loop that waits for the parallel jobs to complete. Appears to use a lot less CPU and could possibly influence the weird problems some were reporting on Windows/Mac - please retest with this version. Steve, I haven't incorporated your changed options handling stuff yet but will do in the future if (a) you don't commit it separately and (b) we can fix the reliability issues with this parallelisation code. --------- Now respects --num-jobs again (broken in last patch). --------- Now reports exceptions in the worker threads. --------- Here's a better fix than last night's effort for the problem where the mapname and description for each job were getting clobbered due to the way that the command args are processed. Each job now gets a "snapshot" of the command args so it doesn't matter if they subsequently get changed. --------- Whoops! fixed a bad bug whereby each map was being output to the same file. Not sure if the fix is very elegant but at least it's not being silly any more. Now limits the default value of max-jobs to 4 no matter how many cores you have as further testing shows that having more threads just burns CPU cycles but doesn't actually finish any quicker. I guess the memory system is limiting the performance and the CPUs are spinning waiting for access. Now showing a real speedup of around 240% (my earlier higher claim was based on CPU usage and I now realise that was erroneous, sorry). -------- Now defaults to creating a thread per core so without doing anything you should see a speedup on a SMP box when processing multiple maps. You can use --max-jobs=N to limit the concurrency - you may want to specify that if you can't increase the VM size to what is required. However, it occurs to me that if you can afford a box with more than 2 cores, then you can probably afford a reasonable amount of memory (otherwise, what's the point in having more cores?) Added help blurb. -------- OK, let it not be said that I don't listen to others! The attached patch provides support for making maps in parallel. By default, the behaviour is the same as before but if you specify --num-threads=N where N is greater than 1, it will process N maps at the same time and then combine the results (if required). Don't forget to increase the heap size appropriately. A quick test on the big box shows good speedup - specifying --num-threads=4 and 2GB VM size. I was seeing better than 380% utilisation with 8 cores in use. I suspect the performance limitation here will be VM size and memory system bandwidth. BTW - I don't think num-threads is actually the best name for the option, so please suggest alternatives. Cheers, Mark

2 3

[PATCH v9] make maps in parallel
by Mark Burton 18 May '09

18 May '09

Now preserves order in which files are combined (thanks Steve for the tweak). --------- Now serialises reading of style files and map source to avoid reentrancy issue in GType. Reworked top-level loop that waits for the parallel jobs to complete. Appears to use a lot less CPU and could possibly influence the weird problems some were reporting on Windows/Mac - please retest with this version. Steve, I haven't incorporated your changed options handling stuff yet but will do in the future if (a) you don't commit it separately and (b) we can fix the reliability issues with this parallelisation code. --------- Now respects --num-jobs again (broken in last patch). --------- Now reports exceptions in the worker threads. --------- Here's a better fix than last night's effort for the problem where the mapname and description for each job were getting clobbered due to the way that the command args are processed. Each job now gets a "snapshot" of the command args so it doesn't matter if they subsequently get changed. --------- Whoops! fixed a bad bug whereby each map was being output to the same file. Not sure if the fix is very elegant but at least it's not being silly any more. Now limits the default value of max-jobs to 4 no matter how many cores you have as further testing shows that having more threads just burns CPU cycles but doesn't actually finish any quicker. I guess the memory system is limiting the performance and the CPUs are spinning waiting for access. Now showing a real speedup of around 240% (my earlier higher claim was based on CPU usage and I now realise that was erroneous, sorry). -------- Now defaults to creating a thread per core so without doing anything you should see a speedup on a SMP box when processing multiple maps. You can use --max-jobs=N to limit the concurrency - you may want to specify that if you can't increase the VM size to what is required. However, it occurs to me that if you can afford a box with more than 2 cores, then you can probably afford a reasonable amount of memory (otherwise, what's the point in having more cores?) Added help blurb. -------- OK, let it not be said that I don't listen to others! The attached patch provides support for making maps in parallel. By default, the behaviour is the same as before but if you specify --num-threads=N where N is greater than 1, it will process N maps at the same time and then combine the results (if required). Don't forget to increase the heap size appropriately. A quick test on the big box shows good speedup - specifying --num-threads=4 and 2GB VM size. I was seeing better than 380% utilisation with 8 cores in use. I suspect the performance limitation here will be VM size and memory system bandwidth. BTW - I don't think num-threads is actually the best name for the option, so please suggest alternatives. Cheers, Mark

1 0

Style-File not parsed from top to bottom - BUG:
by Felix Hartmann 18 May '09

18 May '09

Just to follow up the ignore-turn-restrictions discussion. I have done many more compilations with several different arguments to find out if there is any specific reason why this happens. But I couldn't find any. To me it seems now that the rules given in the stylefile (lines) are not really parsed like they should. This happened with or without the --not-sorted-roads argument (I first suspected that there might be an error, but this argument makes no changes). I assumed that if i give an argument like oneway=yes {set oneway=no} would only affect lines listed below the argument. This seems not to be the case. This is an abrigded lines style-file that does not work like it should (it should not touch the primary, secondary, tertiary oneway settings, but delete them from residential, living_street and : highway=primary [0x10 road_class=0 road_speed=2 resolution 19] highway=secondary [0x0d road_class=0 road_speed=2 resolution 19] highway=tertiary [0x06 road_class=1 road_speed=4 resolution 20] oneway=yes { set oneway=no } oneway=true { set oneway=no } oneway=1 { set oneway=no } oneway=-1 { set oneway=no } oneway=0 {set oneway=no} oneway=false {set oneway=no} highway=residential [0x07 road_class=1 road_speed=4 resolution 22] highway=living_street [0x07 road_class=1 road_speed=3 resolution 22] highway=unclassified [0x07 road_class=1 road_speed=4 resolution 22] Effectively what I get is that for some roads oneway attribute will be set to no, while on most roads the oneway attribute will be lost. Another error to verify: If I put oneway=* {set oneway=no} map generation takes literally ages (2 hours instead of 5 minutes). Using the alternatives as set above map rendering time does not increase significantly from the 5 minutes. the same happens if I put maxspeed=* {set maxspeed=no} into the beginning of the stylefile.

3 3

Splitting the UK
by Stuart Poulton 17 May '09

17 May '09

Hi All, I currently use the following to download and split the UK from a planet file using osmosis. In order to get routable maps, should I now be using the mkgmap splitter on an extract of the UK from the planet ? Does osmosis allow mkgmap routing across tiles ? Regards Stuart get -O - http://planet.openstreetmap.org/planet-080730.osm.gz| gunzip -c | ../Software/osmosis-0.24/bin/osmosis --rx /dev/stdin enableDateParsing="no" \ --log-progress \ --buffer bufferCapacity=1000 \ --tee 15 \ --bounding-box top="60.855" bottom="58.695" left="-3.505" right="-0.745" \ --bounding-box top="58.705" bottom="55.495" left="-8.005" right="-4.995" \ --bounding-box top="58.705" bottom="55.495" left="-5.005" right="-1.495" \ --bounding-box top="55.505" bottom="52.995" left="-2.005" right="0.405" \ --bounding-box top="55.505" bottom="52.995" left="-5.005" right="-1.995" \ --bounding-box top="55.505" bottom="52.995" left="-8.005" right="-4.995" \ --bounding-box top="55.505" bottom="52.995" left="-10.655" right="-7.995" \ --bounding-box top="53.005" bottom="49.845" left="-8.005" right="-4.995" \ --bounding-box top="53.005" bottom="51.295" left="-10.655" right="-7.995" \ --bounding-box top="53.005" bottom="50.095" left="-5.005" right="-1.995" \ --bounding-box top="53.005" bottom="50.495" left="-2.005" right="-0.495" \ --bounding-box top="53.005" bottom="50.645" left="-0.505" right="1.005" \ --bounding-box top="53.005" bottom="50.895" left="0.995" right="2.005" \ --bounding-box top="50.105" bottom="48.995" left="-2.755" right="-1.995" \ --bounding-box top="52.750" bottom="52.250" left="-2.450" right="-1.250" \ --write-xml file="B-Grid0.osm" \ --write-xml file="B-Grid1.osm" \ --write-xml file="B-Grid2.osm" \ --write-xml file="B-Grid3.osm" \ --write-xml file="B-Grid4.osm" \ --write-xml file="B-Grid5.osm" \ --write-xml file="B-Grid6.osm" \ --write-xml file="B-Grid7.osm" \ --write-xml file="B-Grid8.osm" \ --write-xml file="B-Grid9.osm" \ --write-xml file="B-Grid10.osm" \ --write-xml file="B-Grid11.osm" \ --write-xml file="B-Grid12.osm" \ --write-xml file="B-Grid13.osm" \ --write-xml file="Midland.osm"

3 4

--ignore-turn-restrictions
by Felix Hartmann 17 May '09

17 May '09

Currently this switch, disables oneway attributes too. Would it be possible to have several switches or change this to do what the name implies or otherwise rename to -ignore-restrictions? i.e. the most common: ignore-oneway ignore-turn-restrictions ignore-access-restrictions and as continuation of the current function ignore-restrictions to disable any kind of restriction (acces, turn, oneway, or other).

2 3