Hi, it is easy to change the HashMap<Long,Coord> coordMap in ElementSaver to use the OSMId2ObjectMap that I've implemented for splitter. This could reduce peak memory usage in mkgmap. A few numbers according to VisualVM: 2.248.404 coord instances managed in the HashMap req. 186.447.135 bytes, with OSMId2ObjectMap it is 57.194.684. This doesn't include the memory that is needed to store the Coord instances itself, (which I think is 2.248.404 * 32 = 71.948.928 bytes) it is just the overhead for the map. The disadvantage: OSMId2ObjectMap uses fastutil.jar, which is not used in mkgmap until now. I could try to change OSMId2ObjectMap so that it runs without fastutil.jar, but that would mean to invent the wheel again. Comments? Gerd -- View this message in context: http://gis.19327.n5.nabble.com/Possible-memory-reduction-in-mkgmap-tp5743567... Sent from the Mkgmap Development mailing list archive at Nabble.com.
Hi
The disadvantage: OSMId2ObjectMap uses fastutil.jar, which is not used in mkgmap until now. I could try to change OSMId2ObjectMap so that it runs without fastutil.jar,
I think if it make a good difference in execution time as well as memory then it will be well worth it. I want to be careful in selecting dependencies, but this is one that we already have in the wider project anyway. Its probably worth it for the memory saving alone since it is so large, but I suspect it will have some performance improvement anyway although it is hard to know for sure without trying it. ..Steve
Its probably worth it for the memory saving alone since it is so large, but I suspect it will have some performance improvement anyway although it is hard to know for sure without trying it.
Yes, it reduces runtime as well, maybe 5%. Attached is a patch for testing. You have to copy the fastutil.jar from latest splitter\lib to mkgmap\lib. Gerd
Hi,
it is easy to change the HashMap<Long,Coord> coordMap in ElementSaver to use the OSMId2ObjectMap that I've implemented for splitter. This could reduce peak memory usage in mkgmap.
A few numbers according to VisualVM: 2.248.404 coord instances managed in the HashMap req. 186.447.135 bytes, with OSMId2ObjectMap it is 57.194.684. This doesn't include the memory that is needed to store the Coord instances itself, (which I think is 2.248.404 * 32 = 71.948.928 bytes) it is just the overhead for the map.
The disadvantage: OSMId2ObjectMap uses fastutil.jar, which is not used in mkgmap until now. I could try to change OSMId2ObjectMap so that it runs without fastutil.jar, but that would mean to invent the wheel again.
Comments?
Gerd
Sounds good. I think it's ok to add the fastutil.jar to mkgmap. WanMil
WanMil wrote
The disadvantage: OSMId2ObjectMap uses fastutil.jar, which is not used in mkgmap until now. I could try to change OSMId2ObjectMap so that it runs without fastutil.jar, but that would mean to invent the wheel again.
Comments?
Gerd
Sounds good. I think it's ok to add the fastutil.jar to mkgmap.
Fine. I would prefer that Steve does that because I have no experience with the ivy configuration. I think about coding a LinkedOSMId2ObjectMap as well. This would allow to replace many instances of HashMap<Long,xyz> and LinkedHashMap<Long,xyz> I have to find out if it really helps. Gerd -- View this message in context: http://gis.19327.n5.nabble.com/Possible-memory-reduction-in-mkgmap-tp5743567... Sent from the Mkgmap Development mailing list archive at Nabble.com.
Hi, I looked at some heap dumps and I think the other maps are rather small, and they use eg. the values() method to iterate over all elements. This method can't be efficiently implemented in OSMId2Object, so, a lot of code would need changes if we also try to use a LinkedOSMId2ObjectMap to replace the LinkedHashMap<Long,xyz> maps. So, in short, we can use the patch as it is. @Steve: Do you want to manage fastutil.jar with ivy or should I add it to eg. mkgmap\lib? To avoid confusion with different versions I'd prefer to use exactly the same jar as in splitter, although mkgmap doesn't (yet) use the ShortArrayList or LongArrayList classes, so if you want to create the jar with autojar please make sure that it contains all classes needed by mkgmap and splitter. Gerd GerdP wrote
WanMil wrote
The disadvantage: OSMId2ObjectMap uses fastutil.jar, which is not used in mkgmap until now. I could try to change OSMId2ObjectMap so that it runs without fastutil.jar, but that would mean to invent the wheel again.
Comments?
Gerd
Sounds good. I think it's ok to add the fastutil.jar to mkgmap. Fine. I would prefer that Steve does that because I have no experience with the ivy configuration. I think about coding a LinkedOSMId2ObjectMap as well. This would allow to replace many instances of HashMap<Long,xyz> and LinkedHashMap<Long,xyz> I have to find out if it really helps.
Gerd
-- View this message in context: http://gis.19327.n5.nabble.com/Possible-memory-reduction-in-mkgmap-tp5743567... Sent from the Mkgmap Development mailing list archive at Nabble.com.
Steve Ratcliffe wrote
On 13/01/13 06:54, GerdP wrote:
@Steve: Do you want to manage fastutil.jar with ivy or should I add it to eg. mkgmap\lib?
It is now available with the attached patch to ivy.xml.
..Steve
Thanks, but I can't compile splitter with fastutil-6.5.1-mkg.1.jar. Is that intended? Gerd -- View this message in context: http://gis.19327.n5.nabble.com/Possible-memory-reduction-in-mkgmap-tp5743567... Sent from the Mkgmap Development mailing list archive at Nabble.com.
On 13/01/13 17:58, GerdP wrote:
Thanks, but I can't compile splitter with fastutil-6.5.1-mkg.1.jar. Is that intended?
Well it was unintentional, but splitter currently has its own version so I don't see what the problem is. I've built a new version and will install it once the index bug is fixed. ..Steve
Well it was unintentional, but splitter currently has its own version so I don't see what the problem is. I've built a new version and will install it once the index bug is fixed.
I'd prefer to have only one version of the library for both programs. On the other hand, you gave it a different name, so there is probably no chance that a user copies the wrong lib. Good luck with the error search! Ciao, Gerd
participants (4)
-
Gerd Petermann -
GerdP
-
Steve Ratcliffe -
WanMil