Hello WanMil, attached is the larger patch, together with a modified fastutil.jar. The patch must be applied to the memory_optimization branch. Added features compared to r183: - Allows Node Ids up to 2^37 - 1 I (hopefully) changed all places where node ids were stored in int or Integer to long or Long. I tested it with a modified OSM.gz file. - Is a bit faster when default methods are used and enough heap is available During the 1st phase, the highest used Node Id is saved, this value is used to allocate the needeed arrays. The previous version used ArrayLists, which were resized quite often because data from osm.pbf delivers the node ids in order. - With new parm optimize-mem it uses a hash map instead of the huge arrays to store chunk data. This allows to split files on machines with small available java heap, but is usually slower. remarks: a) as mentioned before I am a newbe to Java, so I am pretty sure that experts will find some nonsense in my code, esp. regarding coding style. b) Someone should review the way how I collect the information regarding the highest node id and the number of nodes c) Reg. the new parm: I would prefer to decide the best algorithm within the program, but found it very hard to do, so this is the simle solution. d) If the limit of 2^37-1 is a problem I'd suggest to change the code that stores the data in the chunks. I played with larger CHUNK_SIZE values combined with a (very simple) run length encoding to compress the chunks. I'll continue analysing that way because it might be faster as well. regards, Gerd
Date: Sun, 6 Nov 2011 14:13:18 +0100 From: wmgcnfg@web.de To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] [PATCH]splitter memory usage
Thanks for your analysis and your patch.
I do not have enough time to check if the patch produces the same valid results. But to me it sounds like a good aproach.
Therefore I created a branch memory_optimization including your patch so that your changes might be tested more easily by a bigger group of people. The compiled release of the branch should appear automatically at http://www.mkgmap.org.uk/splitter/. If that's not the case Steve might has to do some configuration stuff on the webserver?
WanMil