[Patch v3] reduce peak memory usage for global index
Hi all, the attached patch is a further optimization. Peak memory for --x-split-name-index is now ~ 57% of r3873, without it ~47 % compared to r3873. The run time is a bit better and the unit tests are passed again. I've removed the debug reports for now, so that this patch is ready for commit. A binary is here: http://files.mkgmap.org.uk/download/343/mkgmap.jar Worth to mention: The --x-split-name-index option requires a lot more memory in my test sample with only 51 tiles I see 250 MB instead of 101MB (437 instead of 214 with r3873), so the relative increase for the option is even higher with the patch. A lot of the memory is required to store so called sort keys. Those are created from the street name(s). A possible alternative would be to calculate the keys only only when required. This would increase run time for sorting but should keep memory much lower. I'll experiment with this later. Gerd
Hi all, something goes wrong with this patch. Please ignore it. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <GPetermann_muenchen@hotmail.com> Gesendet: Freitag, 31. März 2017 12:32:46 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: [mkgmap-dev] [Patch v3] reduce peak memory usage for global index Hi all, the attached patch is a further optimization. Peak memory for --x-split-name-index is now ~ 57% of r3873, without it ~47 % compared to r3873. The run time is a bit better and the unit tests are passed again. I've removed the debug reports for now, so that this patch is ready for commit. A binary is here: http://files.mkgmap.org.uk/download/343/mkgmap.jar Worth to mention: The --x-split-name-index option requires a lot more memory in my test sample with only 51 tiles I see 250 MB instead of 101MB (437 instead of 214 with r3873), so the relative increase for the option is even higher with the patch. A lot of the memory is required to store so called sort keys. Those are created from the street name(s). A possible alternative would be to calculate the keys only only when required. This would increase run time for sorting but should keep memory much lower. I'll experiment with this later. Gerd
participants (1)
-
Gerd Petermann