Am 02.04.2010 06:16, schrieb Scott A Crosby:
On Thu, 01 Apr 2010 12:10:22 +0100, Steve Ratcliffe<steve@parabola.me.uk> writes:
On 31/03/10 14:56, Scott A Crosby wrote:
I noticed that mkgmap does not intern any strings. In particular, this
This is true. There is a pending patch that deals with excessive memory use in a slightly different way which is on the style-speed branch, with a pre-built one available at the bottom of the mkgmap download page.
I pulled it out of SVN@1569.
I drops all tags that are not used by the current style and as an extra feature it ensures that there is only copy of all the strings on the key side.
Could you try it out please. It may be worth also interning the values but when I was looking at it there was much less benefit for the maps I was looking at (but it might vary with the input). I'd be happy to apply a patch that interned the values too if there was a decent memory saving without a significant performance drop.
The style-speed branch builds the tile in question within 1gb of ram using the default style. I suspect that that style isn't using the problematic tags. A different style that did use the tags would likely still blow up. This seems fragile.
I benchmarked the problematic tile on the style-speed branch with and without interning all keys and values in the Tag() constructor using ordinary String.intern(). The 18 character change implementing interning seems to increase performance by about a second, from 66 to 65 seconds.
I'd say to go for it.
Scott
Scott, thanks for posting hard performance values! That's good news because it makes possible to use a simple intern() solution. Before I read your post I compared some german tiles. The single threaded SVN r1625 took 328s with the default style. Using the intern() patch it took 335s. That seems to be an unsignificant change. I am not able to post multithreaded comparisons because my geriatric CPU has one core only... I also used the VisualGC plugin of JVisualVM to view the GC. The permanent space (where all the interned Strings are stored) did not exceed 11mb. So interning all Strings will probably not exceed the permanent space. However we might add a note to the README file that in some special cases it is necessary to increase the permanent size with -XX:MaxPermSize=128m (or more) to avoid OutOfMemoryExceptions. WanMil