Hi Gerd allCities.trimToSize() is at the top of preWriteImpl(). cities is empty at this point. I should trim it at the end of genCitiesAndMdr20s or delay allocation so it can be made the same size as allCities. Is there any reason not to share "sort" between genCitiesAndMdr20s, calcMdr20/1/2SortPos() - I'll try it. Your patch wasn't attached. I had given up trying to understand the different combinations of sort keys for mrd20/1/2 and, until it runs out of memory again, I'd rather not touch it. I'll experiment a bit more and send another patch in a while. Just seen your change to show the stacktrace when OOM - good. Ticker On Tue, 2021-05-11 at 07:04 +0000, Gerd Petermann wrote:
Hi Ticker,
thanks, good findings!
the patch doesn't use trimToSize() on cities. Did you change your mind? The part about the scope is interesting. At first glance I thought this should make no difference but it probably helps GC to detect that this array cannot be garbage collected.
The SortKeys stuff is really eating memory and it would good to find a better solution. One approach is to use the cache as in attached patch but that only helps when memory is really the problem, it slows down processing for other situations. Maybe better would be to first collect all combinations of region+country, sort them, and use the position in that list to sort other objects?
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Montag, 10. Mai 2021 23:05 An: mkgmap development Betreff: [mkgmap-dev] MDR building out-of-memory
Hi Gerd
Since downloading loading britain-and-ireland-latest.osm.pbf I had been unable to build a gmapsupp because of running out of heap (my hardware is 32 bit, -Xmx1540M is largest value allowed)
My problem is mainly because I have 1731146 cities (along with 1046096 streets)
Looking at Mdr5 processing, I've changed it in 3 ways to improve memory usage and garbage collection.
1/ use trimToSize() after all the cities are loaded from the individual tile .img. I presume that the growth factor gradually increases as it runs out of allocated array space. I had to change the declaration from List<Mdr5Record> to ArrayList<Mdr5Record> to allow this, but I can't see any problem in this.
2/ Move the main part of preWriteImpl into its own method so the first sortKeys ArrayList and Sort can be freed before calcMdr20/1/2() each create another massive SortKeys and Sort.
3/ Move the scope of mdr20s to a class variable. This is referenced by all the Mdr5Records and the scope of where it was declared before seemed to to cause the garbage collector major problems - it churned for 5 mins using all the processors before running out of memory. After moving it, the whole of mdr is built in a couple of mins with cpu usage mostly < 125%.
Ticker _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev