Hi Ticker, I think the cache is not meant to improve run time, it is used to deduplicate and thus reduce memory. Maybe it would be better to use a smaller chunk size and no cache. No idea why I didn't use merged = new ArrayList<>(len); Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Dienstag, 11. Mai 2021 19:19 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] MDR building out-of-memory Hi Gerd I've looked at this, and if Mdr5 space becomes a problem again, I'll consider converting to it. A couple of comments: My map had 2909735 poi so the sort chunk size was ~727433. The cache sizes after each chunk were 563095, 603595, 597239 & 605718. Is the cache worth-while for this low hit-rate? Just running the Gmapsupp combiner on existing tiles (without --route, so no streets), I got a run time of 1 min 44 secs with the cache and 1 min 30 without! However most of this time is copying the tiles into gmapsupp.img so not an accurate statistic. You could pre-allocate List<> "merged" with the correct size. Ticker On Tue, 2021-05-11 at 14:40 +0000, Gerd Petermann wrote:
Hi Ticker,
I've committed the patch as is. I've not seen big changes in performance, but I've used a different (already existing) set of files which was created with my own style. For me, Mdr11.preWriteImpl() is the most problematic part reg. OOM errors.
Maybe look at the code which uses LargeListSorter.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ticker Berkin <rwb-mkgmap@jagit.co.uk> Gesendet: Dienstag, 11. Mai 2021 13:27 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] MDR building out-of-memory
Hi Gerd
Here is updated version of patch.
Changes from last:
Uses your cache code for region and country (in 2 places). For British Isles, there are 190 regions and 7 countries, so I don't think the extra memory will be a problem and there should be some performance benefit.
Delays allocating cities until it can use sortKeys.size() for initial allocation. For above map this is 0.07% too big, so I don't think trimToSize() is worthwhile.
Shares the Sort object between the 4 methods.
Ticker _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev