Re: [mkgmap-dev] Performance with zipped hgt files
Hi Gerd, at last you need the data uncompressed in a buffer, right? That's why i see no difference between using compressed or uncompressed hgt files. Compressed files need a little bit more time for uncompressing, but after that the data need the same size in memory. That's why i see no vantage for mkgmap if the data are stored in an other form. It can only save disc space, no heap space. What speaks against a limitation for the extent of a maptile? This limited the count of hgt's for one maptile too. You have the option --max-areas for splitter. We need only an additional option like --maxextent or whatever. Perhaps we have then maptiles without points, lines or areas (on oceans, deserts or so), but why not? Frank
Hi Frank, the current problem with zipped files is that I have to extract them into a buffer that is kept in Java Heap (managed by Garbage Collection) while uncompressed files are manged by class MappedByteBuffer. These are not managed as Java Heap, they are somewhat virtual and typically only require a small portion of the available memory, see https://howtodoinjava.com/java-7/nio/java-nio-2-0-memory-mapped-files-mapped... I don't yet see a need to change splitter like that but I can do that as well. I am now playing with code to extract zipped files into a temp dir. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Frank Stinner <frank.stinner@leipzig.de> Gesendet: Montag, 8. Januar 2018 15:30:08 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] Performance with zipped hgt files Hi Gerd, at last you need the data uncompressed in a buffer, right? That's why i see no difference between using compressed or uncompressed hgt files. Compressed files need a little bit more time for uncompressing, but after that the data need the same size in memory. That's why i see no vantage for mkgmap if the data are stored in an other form. It can only save disc space, no heap space. What speaks against a limitation for the extent of a maptile? This limited the count of hgt's for one maptile too. You have the option --max-areas for splitter. We need only an additional option like --maxextent or whatever. Perhaps we have then maptiles without points, lines or areas (on oceans, deserts or so), but why not? Frank
Hi Gerd,
I don't yet see a need to change splitter like that but I can do that as well.
I would use this option too. Whenever I split data for Europe, I get a tile which includes Azores and south shore of Island. I split it manually then. For maps with DEM I use an awk script, which process areas.list. I set maximum extent to 11.25 degree. -- Best regards, Andrzej
Hi Andrzej, okay, I'll add an option for that. At the moment, we have these hard coded limits in SplittableDensityArea: private static final int MAX_LAT_DEGREES = 85; private static final int MAX_LON_DEGREES = 90; I have to find out how a change in these limits affects the search for the best split, esp. when the splitting planet. I think one has to accept empty tiles with a max of 11.25 degrees. I think the last changes in the split algo were done to avoid this. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Andrzej Popowski <popej@poczta.onet.pl> Gesendet: Montag, 8. Januar 2018 18:43:33 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] Performance with zipped hgt files Hi Gerd,
I don't yet see a need to change splitter like that but I can do that as well.
I would use this option too. Whenever I split data for Europe, I get a tile which includes Azores and south shore of Island. I split it manually then. For maps with DEM I use an awk script, which process areas.list. I set maximum extent to 11.25 degree. -- Best regards, Andrzej _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Gerd, I decided for 11.25 (=360/32) because of memory limits in early BuildDemFile. Maybe this is not a problem for mkgmap? -- Best regards, Andrzej
Hi Andrzej, I think with unzipped hgt files the only limit is the memory that is needed to store the data for the tiles before they are written to img. So, with --max-jobs=4 or higher and all jobs with large tiles containing large amounts of data for RGN,NOD,NET etc. it is much more likely that the additional data for DEM causes trouble. I've not yet tried to reduce the memory for this temp. data, a possible alternative is to write it to temp files instead of storing it in buffers. This is esp. true for large tiles were DEM can be > 100MB while the other files cannot grow > 16MB. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Andrzej Popowski <popej@poczta.onet.pl> Gesendet: Montag, 8. Januar 2018 21:04:48 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] Performance with zipped hgt files Hi Gerd, I decided for 11.25 (=360/32) because of memory limits in early BuildDemFile. Maybe this is not a problem for mkgmap? -- Best regards, Andrzej _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Gerd, I can confirm, that mkgmap has no memory problems with DEM for big tiles. I can compile map without limiting tile sizes. -- Best regards, Andrzej
Hi all, not sure what happened the last time when I measured performance, probably another program installed updates or the NTFS files were still compressed in background :-( Maybe it was the change to build 152. Anyway, I cannot reproduce the poor performance with zipped hgt files, in fact I see times which are a slightly faster now compared to uncompressed files. So, as long as java heap is not a bottleneck there is probably no reason to avoid zipped files. I tried 4035, 4038 and 4040 and found nearly the same run times with zipped files. Please let me know if you see different results for your work loads. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Andrzej Popowski <popej@poczta.onet.pl> Gesendet: Dienstag, 9. Januar 2018 00:04:07 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] Performance with zipped hgt files Hi Gerd, I can confirm, that mkgmap has no memory problems with DEM for big tiles. I can compile map without limiting tile sizes. -- Best regards, Andrzej _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
participants (3)
-
Andrzej Popowski -
Frank Stinner -
Gerd Petermann