Hello list, thanks for the comments reqarding patchv5. Attached is one more patch to be applied against the branch. Changes: - Enhanced dictionary size to 65535 - Avoid allocating chunks (short[] objects) with a size that is not a multiple of 8 (r189 wasted 4 bytes for each chunk) - introduced special code for one-value-chunks - max-areas=255 is the default again, but maximum allowed is now 2048 I did not find a simple way to allow also "auto" for the parameter optimize-mem. Maybe Steve or WanMil can help? auto should set optimize-mem=true if Runtime.getRuntime().maxMemory() < 2GB Remarks regarding performance: On WinXP systems, use java -Xms1600m -Xmx1600m -jar splitter.jar ... On 64bit boxes use also -XX+UseCompressedOops This avoids a lot of GC work. I did not yet try that with very large files like europe.osm.pbf. http://gis.638310.n2.nabble.com/file/n7008720/memory_patch_v6.patch memory_patch_v6.patch Some numbers for splitting germany.osm.pbf: patchv6 with -Xms and optimize-mem=true: 588s patchv6 with -Xms and optimize-mem=false: 594s patchv6 without -Xms and optimize-mem=true: 651s patchv6 without -Xms and optimize-mem=false: 622s r189 without -Xms and optimize-mem=false: 639s -- View this message in context: http://gis.638310.n2.nabble.com/PATCH-v6-splitter-memory-usage-tp7008720p700... Sent from the Mkgmap Development mailing list archive at Nabble.com.
Thanks! just commited to r190. WanMil
Hello list,
thanks for the comments reqarding patchv5. Attached is one more patch to be applied against the branch. Changes: - Enhanced dictionary size to 65535 - Avoid allocating chunks (short[] objects) with a size that is not a multiple of 8 (r189 wasted 4 bytes for each chunk) - introduced special code for one-value-chunks - max-areas=255 is the default again, but maximum allowed is now 2048
I did not find a simple way to allow also "auto" for the parameter optimize-mem. Maybe Steve or WanMil can help? auto should set optimize-mem=true if Runtime.getRuntime().maxMemory()< 2GB
Remarks regarding performance: On WinXP systems, use java -Xms1600m -Xmx1600m -jar splitter.jar ... On 64bit boxes use also -XX+UseCompressedOops This avoids a lot of GC work. I did not yet try that with very large files like europe.osm.pbf. http://gis.638310.n2.nabble.com/file/n7008720/memory_patch_v6.patch memory_patch_v6.patch Some numbers for splitting germany.osm.pbf: patchv6 with -Xms and optimize-mem=true: 588s patchv6 with -Xms and optimize-mem=false: 594s patchv6 without -Xms and optimize-mem=true: 651s patchv6 without -Xms and optimize-mem=false: 622s
r189 without -Xms and optimize-mem=false: 639s
-- View this message in context: http://gis.638310.n2.nabble.com/PATCH-v6-splitter-memory-usage-tp7008720p700... Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Does adding -Xms also add performance on systems with abundant memory? On 18.11.2011 17:00, GerdP wrote:
Hello list,
thanks for the comments reqarding patchv5. Attached is one more patch to be applied against the branch. Changes: - Enhanced dictionary size to 65535 - Avoid allocating chunks (short[] objects) with a size that is not a multiple of 8 (r189 wasted 4 bytes for each chunk) - introduced special code for one-value-chunks - max-areas=255 is the default again, but maximum allowed is now 2048
I did not find a simple way to allow also "auto" for the parameter optimize-mem. Maybe Steve or WanMil can help? auto should set optimize-mem=true if Runtime.getRuntime().maxMemory()< 2GB
Remarks regarding performance: On WinXP systems, use java -Xms1600m -Xmx1600m -jar splitter.jar ... On 64bit boxes use also -XX+UseCompressedOops This avoids a lot of GC work. I did not yet try that with very large files like europe.osm.pbf. http://gis.638310.n2.nabble.com/file/n7008720/memory_patch_v6.patch memory_patch_v6.patch Some numbers for splitting germany.osm.pbf: patchv6 with -Xms and optimize-mem=true: 588s patchv6 with -Xms and optimize-mem=false: 594s patchv6 without -Xms and optimize-mem=true: 651s patchv6 without -Xms and optimize-mem=false: 622s
r189 without -Xms and optimize-mem=false: 639s
-- View this message in context: http://gis.638310.n2.nabble.com/PATCH-v6-splitter-memory-usage-tp7008720p700... Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hello Felix, if I got this right setting -Xms allows GC to do nothing until the heap reaches this limit. So, if you have 6GB free and don't need it for other programs, -Xms6000m should work better. I don't have such machine, so I can't test. Gerd -- View this message in context: http://gis.638310.n2.nabble.com/PATCH-v6-splitter-memory-usage-tp7008720p700... Sent from the Mkgmap Development mailing list archive at Nabble.com.
Okay I just tried it out. -Xms6000m did improve speed on r181, but on r190 it was no difference. Well I just use it, maybe sometimes it helps... On 18.11.2011 18:05, GerdP wrote:
Hello Felix,
if I got this right setting -Xms allows GC to do nothing until the heap reaches this limit. So, if you have 6GB free and don't need it for other programs, -Xms6000m should work better. I don't have such machine, so I can't test.
Gerd
-- View this message in context: http://gis.638310.n2.nabble.com/PATCH-v6-splitter-memory-usage-tp7008720p700... Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
participants (3)
-
Felix Hartmann -
GerdP -
WanMil