July 2009 - mkgmap-dev - The mkgmap lists

splitter - performance and memory
by Chris Miller 22 Jul '09

22 Jul '09

First off, let me say a quick thanks to everyone involved in this project. I've only just discovered it recently but wish I had found it much earlier, it really is incredible to see what has been done so far. I've downloaded the complete planet-090715.osm.bz2 file, and have been looking at splitting it. I read the description and limitations of the splitter.jar tool but decided to give it a go anyway since I have a 64 bit OS with 6GB ram. Unfortunately it still failed with a -Xmx5200m heap. I have a 16GB machine at work I could try it on that might work, however instead I decided to take a look at the source code to see if there's any possibility of reducing the memory requirements. I've only spent a short time looking at the code, but as far as I can tell the whole first step (computing the areas.list file) is using far more memory than it actually needs. The SplitIntMap (which is what takes up all the memory) isn't required here for two reasons. One is that the code never retrieves entries via .get(), rather it only uses an iterator so a list/array would suffice. Second, the node IDs aren't used in this stage so we don't even need to parse them let alone hold on to them. Assuming we replace the SplitIntMap with a wrapper around an int[] (or multiple int[] to mitigate the double-memory-on-copy problem), we'd be looking at memory savings of >50%. Does that make sense or have I missed something? If it sounds sensible I'd be happy to have a go at implementing it. Also, given the nature of the algorithm it wouldn't be too hard on performance if the lat+long values were written out to disk rather than held in memory which would mean splitting the whole dataset would be feasible even on a 32bit machine. I haven't yet looked at possibilities for tuning the second step but I assume that some sort of map/lookup is still required here. I figure there's a few options - perform multiple passes processing a subset of the splits at a time (limited by the total number of nodes we can hold in memory), optimise the existing data structures further, page some out to disk, etc. I was also thinking a little about performance. Given the enormous size of the full .osm file, I'd suggest a move away from SAX over to a pull parser (http://www.extreme.indiana.edu/xgws/xsoap/xpp/mxp1/index.html) It's even faster than SAX and uses very little memory. In my job we use it to parse many GB of XML daily with very good results. Another idea is to parallelise the code by running parts of the split on different threads to take advantage of multi-core CPUs. Possibly the best gain here would be when writing the files since gzip compression is fairly CPU intensive. What do people think? I'm happy to work on the above though I must confess up front that my spare time is quite limited so please don't expect too much too soon! Chris

3 11

Report: Mapsource display works again !
by David 21 Jul '09

21 Jul '09

Hello there, I don't know what you made between v1088 and v1095 but now Mapsource works again very well (my map is based on Ireland OSM data). There is no more problem with tiles cache generation. Now I have to understand why a line's type with a value on two bytes (ex : 0x040c) in OSM file is changed in a type defined with the least significant byte only (ex : 0x0c) in IMG. Thank you for your great work, David

1 0

Commit: r1095: Fix issues with routing nodes that are located on a tile's boundary.
by svn commit 21 Jul '09

21 Jul '09

Version 1095 was commited by markb on 2009-07-21 15:37:48 +0100 (Tue, 21 Jul 2009) Fix issues with routing nodes that are located on a tile's boundary. Previously, if a routing node was positioned exactly on a tile's boundary, the routing through that node was broken. Should now be fixed.

1 0

Commit: r1094: Introduce new method to check if all points are (really) inside the area.
by svn commit 21 Jul '09

21 Jul '09

Version 1094 was commited by markb on 2009-07-21 15:37:44 +0100 (Tue, 21 Jul 2009) Introduce new method to check if all points are (really) inside the area. Added new method insideBoundary() to check whether a given Coord lies inside the Area and doesn't touch the boundary. Renamed allInside() to allInsideBoundary() and make it use insideBoundary() rather than contains().

1 0

Commit: r1093: Bleat if zero length arc detected.
by svn commit 21 Jul '09

21 Jul '09

Version 1093 was commited by markb on 2009-07-21 15:37:41 +0100 (Tue, 21 Jul 2009) Bleat if zero length arc detected.

1 0

[PATCH v5] - fix clipping when node falls on tile boundary
by Mark Burton 21 Jul '09

21 Jul '09

v5 - Tweaked the test that catches zero-length segments - otherwise the same as before. Maning - I can't see why your example is failing, if you still have the data, please try with this version although I don't really think it will behave any differently. Also, does it just fail to route and give a straight line or does mapsource barf and give you the error box? Valentijn - This works well enough with your example - however, I did notice that mapsource fails to find a route that starts in one tile, goes into the second and then back to the first tile. See attached piccy. That shows a successful route. But if you move the destination point (102) to below the map join (visible as line across way not far below the destination point), mapsource gives up and draws a straight line. But there's nothing wrong with that way (Zuiddijk) because you can happily route across the boundary in both directions. Adding an intermediate point to the route in the upper tile, solves the problem. It's not implausible that this is a limitation of the Garmin routing algorithm. Unless we find any new problems, I intend to commit this stuff in the next day or so. Cheers, Mark

2 4

Commit: r1092: A space was not being stripped out of the highway reference.
by svn commit 21 Jul '09

21 Jul '09

Version 1092 was commited by steve on 2009-07-21 14:25:23 +0100 (Tue, 21 Jul 2009) A space was not being stripped out of the highway reference. - Clinton Gladstone

1 0

[PATCH] Properly remove spaces in highway shield labels
by Clinton Gladstone 21 Jul '09

21 Jul '09

Is anyone else using the highway shield feature? I found that highway names with spaces in them, such as "A 1" would be displayed in the shield as just "A". The attached patch fixes this. If this also fixes the problem for other people, and is not some weird peculiarity of my operating environment, I would appreciate it if the patch was committed. Cheers

2 1

[PATCH v4] - fix clipping when node falls on tile boundary
by Mark Burton 21 Jul '09

21 Jul '09

This version should do the same as v3 as far as the bug fix is concerned but I have cleaned things up a little so please test this version if possible. Thanks, Mark

2 8

[PATCH] Address values for area POIs
by Clinton Gladstone 20 Jul '09

20 Jul '09

The attached patch inelegantly copies address data to the POIs created when using the --add-pois-to-areas option. I have noticed that more and more map objects contain address information (street number, name, etc.). This was being skipped for POIs which were created from polygons. It would be nice if you could take a look and consider this patch, or a better solution, for commitment. Cheers.

1 0