Hello, here is the final patch http://gis.638310.n2.nabble.com/file/n7001044/memory_patch_v5.patch memory_patch_v5.patch It has to be applied against the memory branch. Final notes: - the modified fastutil.jar is no longer needed, the trunk version version works fine - the program works almost always better when parameter --optimize-mem=true is used, so it might be better to change that to be the default. Some numbers: All values measured on a WinXP machine with 2.6Gh dual core AMD 5050e and with common parms: java -Xmx1600m -Xloggc:gclog.txt -verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -jar splitter.jar --status-freq=0 Times in seconds, max heap occupancy (calculated by garbagecat) * IDs in the range 1073742179 - 1107861564, parm --mixed is needed ** manually modified IDs to start with 3 (3073742179- 3107861564), parameer --split-file is used -- View this message in context: http://gis.638310.n2.nabble.com/PATCH-v5-splitter-memory-usage-tp7001044p700... Sent from the Mkgmap Development mailing list archive at Nabble.com.
Thanks Gerd, I have commited it to the branch. I will merge the branch back to trunk after getting some feedback on the list that the changes are worthy to merge. Unfortunately I do not have time to check that myself at the moment. Have fun! WanMil
Hello,
here is the final patch http://gis.638310.n2.nabble.com/file/n7001044/memory_patch_v5.patch memory_patch_v5.patch It has to be applied against the memory branch.
Final notes: - the modified fastutil.jar is no longer needed, the trunk version version works fine - the program works almost always better when parameter --optimize-mem=true is used, so it might be better to change that to be the default.
Some numbers: All values measured on a WinXP machine with 2.6Gh dual core AMD 5050e and with common parms: java -Xmx1600m -Xloggc:gclog.txt -verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -jar splitter.jar --status-freq=0
Times in seconds, max heap occupancy (calculated by garbagecat) * IDs in the range 1073742179 - 1107861564, parm --mixed is needed ** manually modified IDs to start with 3 (3073742179- 3107861564), parameer --split-file is used
-- View this message in context: http://gis.638310.n2.nabble.com/PATCH-v5-splitter-memory-usage-tp7001044p700... Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Seems to be still some fastutil.jar problem around? I tried to use the splitter.jar from the mkgmap download page WITHOUT the modified fastutil.jar (latest as of today): Exception in thread "main" java.lang.NoClassDefFoundError: it/unimi/dsi/fastutil/longs/Long2ShortFunction It does work however if I use the modified fastutil.jar. Just a quick speed comparison (i have 8GB RAM on my server, so RAM is abundant): splitter trunk Austria: 40 seconds (however with modified fastutil.jar, don't think it matters on speed however). splitter_patched with optimize-mem=33 seconds splitter _patched without optimime-mem=34 seconds I am right now doing some more tries on other countries (bigger ones, as it doesn't matter that much on too small countries where it is anyhow fast). Here is the complete log of it failing: c:\OpenMTBMap\maps>java -Xmx6600m -jar c:\openmtbmap\splitter.jar --max-nodes=1250000 --output=pbf --max-areas=128 --geonames-file=cities15000 --description=austria --cache=cache --mapid=63650000 aust ria.osm.pbf cache=cache description=austria geonames-file=cities15000 legacy-mode=false mapid=63650000 max-areas=128 max-nodes=1250000 max-threads=4 (auto) mixed=false no-trim=false optimize-mem=false output=pbf output-dir= overlap=2000 resolution=13 split-file= status-freq=120 write-kml= Elapsed time: 0s Memory: Current 122MB (1MB used, 121MB free) Max 5866MB Time started: Thu Nov 17 16:07:34 CET 2011 Map is being split for resolution 13: - area boundaries are aligned to 0x800 map units - areas are multiples of 0x1000 map units wide and high Processing austria.osm.pbf Bounding box 9.52678 46.36851 17.16273 49.02403 10'000'000 nodes processed... id=1174172078 1'000'000 ways processed... id=110462196 in 1 file Time: Thu Nov 17 16:07:42 CET 2011 Exact map coverage is (46.36850595474243,9.526777267456055) to (49.0240216255188,17.162725925445557) Trimmed and rounded map coverage is (46.40625,9.4921875) to (49.04296875,17.138671875) Splitting nodes into areas containing a maximum of 1'250'000 nodes each... Area (46.4501953125,9.4921875) to (47.2412109375,10.8984375) contains 540'283 nodes. DONE! Area (47.2412109375,9.4921875) to (47.7685546875,10.8984375) contains 737'377 nodes. DONE! Area (46.7138671875,10.8984375) to (47.2412109375,12.216796875) contains 525'664 nodes. DONE! Area (47.2412109375,10.8984375) to (48.1201171875,12.216796875) contains 1'125'411 nodes. DONE! Area (46.40625,12.216796875) to (47.4609375,13.0078125) contains 639'704 nodes. DONE! Area (46.40625,13.0078125) to (47.4609375,13.88671875) contains 1'106'723 nodes. DONE! Area (47.4609375,12.216796875) to (47.8125,13.88671875) contains 1'112'114 nodes. DONE! Area (47.8125,12.392578125) to (48.955078125,13.88671875) contains 1'058'689 nodes. DONE! Area (46.40625,13.88671875) to (47.021484375,15.205078125) contains 754'775 nodes. DONE! Area (47.021484375,13.88671875) to (47.8125,15.205078125) contains 801'876 nodes. DONE! Area (46.494140625,15.205078125) to (47.197265625,16.962890625) contains 933'138 nodes. DONE! Area (47.197265625,15.205078125) to (47.8125,15.908203125) contains 568'885 nodes. DONE! Area (47.197265625,15.908203125) to (47.8125,17.138671875) contains 756'400 nodes. DONE! Area (47.8125,13.88671875) to (48.1640625,15.205078125) contains 708'241 nodes. DONE! Area (48.1640625,13.88671875) to (48.33984375,15.205078125) contains 1'221'308 nodes. DONE! Area (48.33984375,13.88671875) to (49.04296875,15.205078125) contains 768'745 nodes. DONE! Area (47.8125,15.205078125) to (49.04296875,15.99609375) contains 1'244'722 nodes. DONE! Area (47.8125,15.99609375) to (48.1640625,17.138671875) contains 550'724 nodes. DONE! Area (48.1640625,15.99609375) to (49.04296875,17.138671875) contains 1'039'634 nodes. DONE! 19 areas: Area 63650000 covers (0x211000,0xad000) to (0x219000,0xc1000) AT-Graz Area 63650001 covers (0x220000,0xad000) to (0x22e000,0xb6000) AT-Sankt Polten Area 63650002 covers (0x219000,0xb5000) to (0x220000,0xc3000) HU-Szombathely Area 63650003 covers (0x220000,0x9e000) to (0x224000,0xad000) AT-Steyr Area 63650004 covers (0x213800,0x7c000) to (0x219800,0x8b000) IT-Bressanone Area 63650005 covers (0x210000,0x9e000) to (0x217000,0xad000) AT-Klagenfurt am Woerthersee Area 63650006 covers (0x21c000,0x8b000) to (0x220000,0x9e000) AT-Salzburg Area 63650007 covers (0x224000,0xb6000) to (0x22e000,0xc3000) AT-Vienna Area 63650008 covers (0x210800,0x6c000) to (0x219800,0x7c000) CH-Chur Area 63650009 covers (0x219000,0xad000) to (0x220000,0xb5000) AT-Kapfenberg Area 63650010 covers (0x210000,0x8b000) to (0x21c000,0x94000) AT-Saalfelden am Steinernen Meer Area 63650011 covers (0x226000,0x9e000) to (0x22e000,0xad000) CZ-Ceske Budejovice Area 63650012 covers (0x219800,0x7c000) to (0x223800,0x8b000) AT-Innsbruck Area 63650013 covers (0x217000,0x9e000) to (0x220000,0xad000) AT-Leoben Area 63650014 covers (0x220000,0x8d000) to (0x22d000,0x9e000) DE-Passau Area 63650015 covers (0x220000,0xb6000) to (0x224000,0xc3000) SK-Bratislava Area 63650016 covers (0x219800,0x6c000) to (0x21f800,0x7c000) DE-Kempten (Allgaeu) Area 63650017 covers (0x224000,0x9e000) to (0x226000,0xad000) AT-Linz Area 63650018 covers (0x210000,0x94000) to (0x21c000,0x9e000) AT-Villach Writing out split osm files Thu Nov 17 16:07:42 CET 2011 Processing 19 areas in a single pass Area 63650000 covers (0x211000,0xad000) to (0x219000,0xc1000) AT-Graz Area 63650001 covers (0x220000,0xad000) to (0x22e000,0xb6000) AT-Sankt Polten Area 63650002 covers (0x219000,0xb5000) to (0x220000,0xc3000) HU-Szombathely Area 63650003 covers (0x220000,0x9e000) to (0x224000,0xad000) AT-Steyr Area 63650004 covers (0x213800,0x7c000) to (0x219800,0x8b000) IT-Bressanone Area 63650005 covers (0x210000,0x9e000) to (0x217000,0xad000) AT-Klagenfurt am Woerthersee Area 63650006 covers (0x21c000,0x8b000) to (0x220000,0x9e000) AT-Salzburg Area 63650007 covers (0x224000,0xb6000) to (0x22e000,0xc3000) AT-Vienna Area 63650008 covers (0x210800,0x6c000) to (0x219800,0x7c000) CH-Chur Area 63650009 covers (0x219000,0xad000) to (0x220000,0xb5000) AT-Kapfenberg Area 63650010 covers (0x210000,0x8b000) to (0x21c000,0x94000) AT-Saalfelden am Steinernen Meer Area 63650011 covers (0x226000,0x9e000) to (0x22e000,0xad000) CZ-Ceske Budejovice Area 63650012 covers (0x219800,0x7c000) to (0x223800,0x8b000) AT-Innsbruck Area 63650013 covers (0x217000,0x9e000) to (0x220000,0xad000) AT-Leoben Area 63650014 covers (0x220000,0x8d000) to (0x22d000,0x9e000) DE-Passau Area 63650015 covers (0x220000,0xb6000) to (0x224000,0xc3000) SK-Bratislava Area 63650016 covers (0x219800,0x6c000) to (0x21f800,0x7c000) DE-Kempten (Allgaeu) Area 63650017 covers (0x224000,0x9e000) to (0x226000,0xad000) AT-Linz Area 63650018 covers (0x210000,0x94000) to (0x21c000,0x9e000) AT-Villach Writing out split osm files Thu Nov 17 16:07:42 CET 2011 Processing 19 areas in a single pass (46.494140625,15.205078125) to (47.197265625,16.962890625) (47.8125,15.205078125) to (49.04296875,15.99609375) (47.197265625,15.908203125) to (47.8125,17.138671875) (47.8125,13.88671875) to (48.1640625,15.205078125) (46.7138671875,10.8984375) to (47.2412109375,12.216796875) (46.40625,13.88671875) to (47.021484375,15.205078125) (47.4609375,12.216796875) to (47.8125,13.88671875) (48.1640625,15.99609375) to (49.04296875,17.138671875) (46.4501953125,9.4921875) to (47.2412109375,10.8984375) (47.197265625,15.205078125) to (47.8125,15.908203125) (46.40625,12.216796875) to (47.4609375,13.0078125) (48.33984375,13.88671875) to (49.04296875,15.205078125) (47.2412109375,10.8984375) to (48.1201171875,12.216796875) (47.021484375,13.88671875) to (47.8125,15.205078125) (47.8125,12.392578125) to (48.955078125,13.88671875) (47.8125,15.99609375) to (48.1640625,17.138671875) (47.2412109375,9.4921875) to (47.7685546875,10.8984375) (48.1640625,13.88671875) to (48.33984375,15.205078125) (46.40625,13.0078125) to (47.4609375,13.88671875) Starting pass 1 of 1, processing 19 areas (63650000 to 63650018) Exception in thread "main" java.lang.NoClassDefFoundError: it/unimi/dsi/fastutil/longs/Long2ShortFunction at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(Unknown Source) at java.security.SecureClassLoader.defineClass(Unknown Source) at java.net.URLClassLoader.defineClass(Unknown Source) at java.net.URLClassLoader.access$100(Unknown Source) at java.net.URLClassLoader$1.run(Unknown Source) at java.net.URLClassLoader$1.run(Unknown Source) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(Unknown Source) at java.lang.ClassLoader.loadClass(Unknown Source) at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source) at java.lang.ClassLoader.loadClass(Unknown Source) at uk.me.parabola.splitter.SplitProcessor.<init>(SplitProcessor.java:87) at uk.me.parabola.splitter.Main.writeAreas(Main.java:371) at uk.me.parabola.splitter.Main.split(Main.java:193) at uk.me.parabola.splitter.Main.start(Main.java:121) at uk.me.parabola.splitter.Main.main(Main.java:110) Caused by: java.lang.ClassNotFoundException: it.unimi.dsi.fastutil.longs.Long2ShortFunction at java.net.URLClassLoader$1.run(Unknown Source) at java.net.URLClassLoader$1.run(Unknown Source) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(Unknown Source) at java.lang.ClassLoader.loadClass(Unknown Source) at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source) at java.lang.ClassLoader.loadClass(Unknown Source) ... 17 more
some more speed observations: country germany.pbf from Geofabrik splitter_patched with optimize-mem: 3 min 6 seconds splitter_patched without optimize-mem: 3min 2 seconds old splitter: 3 min 42 seconds I am currently running against europe.pbf, will post the results later. But it seems that the patched mkgmap is consistently a bit faster. I think optimize-mem is better of by default though (most people using mkgmap/splitter have powerful computers anyhow). I'm interested if the 1024 max areas vs the old 255 and subsequent less runs make for a bigger speedup than on small(er) extracts.
Europe max-nodes 900.000 (resulting in 975 tiles, extract is about 4 weeks old already - hence 4 passes down to 1 pass ) old splitter: 52min 38 seconds (3158 seconds) splitter_patched: 29min 45 seconds (1785 seconds) splitter patched with optimize-mem: 30min 41seconds (1841 seconds) So I would support optimize-mem to stay off by default. It seems to produce worse results the more data it has to handle. The general improvement on splitting time is however great, especially on bigger files (mind mkgmap itself needs about 5x as much time as the patched splitter). Also the estimates taken by optimize-mem seem to be plain underestimated (end of the output): / MAP occupancy: 56'000'000, number of area dictionary entries: 10530 Chunk map details: used 1'447'751 (~ 4418%) of the estimated 32'768 entries. 56'000'000 ways processed... id=111312902 MAP occupancy: 57'000'000, number of area dictionary entries: 10596 Chunk map details: used 1'469'838 (~ 4486%) of the estimated 32'768 entries. 57'000'000 ways processed... id=112919265 MAP occupancy: 58'000'000, number of area dictionary entries: 10633 Chunk map details: used 1'493'839 (~ 4559%) of the estimated 32'768 entries. 58'000'000 ways processed... id=114728311 MAP occupancy: 59'000'000, number of area dictionary entries: 10660 Chunk map details: used 1'516'986 (~ 4629%) of the estimated 32'768 entries. 59'000'000 ways processed... id=116413634 MAP occupancy: 60'000'000, number of area dictionary entries: 10700 Chunk map details: used 1'541'052 (~ 4703%) of the estimated 32'768 entries. 60'000'000 ways processed... id=118147644 MAP occupancy: 61'000'000, number of area dictionary entries: 10750 Chunk map details: used 1'564'456 (~ 4774%) of the estimated 32'768 entries. 61'000'000 ways processed... id=119857968 MAP occupancy: 62'000'000, number of area dictionary entries: 10789 Chunk map details: used 1'589'073 (~ 4849%) of the estimated 32'768 entries. 62'000'000 ways processed... id=121660891 MAP occupancy: 63'000'000, number of area dictionary entries: 10828 Chunk map details: used 1'610'303 (~ 4914%) of the estimated 32'768 entries. 63'000'000 ways processed... id=123132157 MAP occupancy: 64'000'000, number of area dictionary entries: 10838 Chunk map details: used 1'632'875 (~ 4983%) of the estimated 32'768 entries. 64'000'000 ways processed... id=124780404 MAP occupancy: 65'000'000, number of area dictionary entries: 10847 Chunk map details: used 1'656'390 (~ 5055%) of the estimated 32'768 entries. 65'000'000 ways processed... id=126458865 MAP occupancy: 66'000'000, number of area dictionary entries: 10907 Chunk map details: used 1'679'991 (~ 5127%) of the estimated 32'768 entries. 66'000'000 ways processed... id=128180713 MAP occupancy: 67'000'000, number of area dictionary entries: 11004 Chunk map details: used 1'702'474 (~ 5196%) of the estimated 32'768 entries. 67'000'000 ways processed... id=129790909 MAP occupancy: 68'000'000, number of area dictionary entries: 11036 Chunk map details: used 1'724'908 (~ 5264%) of the estimated 32'768 entries. 68'000'000 ways processed... id=131429900 Writing relations Thu Nov 17 19:22:19 CET 2011 Elapsed time: 30m 6s Memory: Current 5574MB (4285MB used, 1289MB free) Max 6044MB 100'000 relations processed... id=190988 200'000 relations processed... id=349055 300'000 relations processed... id=485393 400'000 relations processed... id=952259 500'000 relations processed... id=1198900 600'000 relations processed... id=1376675 700'000 relations processed... id=1573951 800'000 relations processed... id=1743479 *********************************************************** Final statistics *********************************************************** coords occupancy MAP occupancy: 579'162'124, number of area dictionary entries: 6248 Chunk map details: used 15'886'784 (~ 48483%) of the estimated 32'768 entries. ways occupancy MAP occupancy: 68'744'281, number of area dictionary entries: 11058 Chunk map details: used 1'741'394 (~ 5314%) of the estimated 32'768 entries. Thread worker-1 has finished Thread worker-0 has finished Thread worker-2 has finished Time finished: Thu Nov 17 19:22:55 CET 2011 Total time taken: 1841s / On 17.11.2011 16:48, Felix Hartmann wrote:
some more speed observations: country germany.pbf from Geofabrik
splitter_patched with optimize-mem: 3 min 6 seconds splitter_patched without optimize-mem: 3min 2 seconds old splitter: 3 min 42 seconds
I am currently running against europe.pbf, will post the results later. But it seems that the patched mkgmap is consistently a bit faster. I think optimize-mem is better of by default though (most people using mkgmap/splitter have powerful computers anyhow). I'm interested if the 1024 max areas vs the old 255 and subsequent less runs make for a bigger speedup than on small(er) extracts.
Sorry, all of my my previous measurements were actually done with splitter r188. So also the fastutil.jar missing problem was of course caused by it. I have updated all tests with r189. Note that optimize-mem at least on my system (Core7 with 4 Cores available, 8GB Ram, Win 7 x64 server) is only faster on Europe.pbf. Else it is slower. r189 is in general faster or as fast as r188. However using optimize-mem on Germany.pbf it slowed down a bit (was faster without optimize-mem though). As most people will not split files as big as europe.pbf very often, I think optimize-mem=false should stay the default. Therefore I reran some tests 1 to complement old test: splitter r188 without optimize-mem and max-nodes 1 100 000 instead of 900 000 (807 areas instead of 975)): 1678 seconds (so 107 seconds gained over smaller max-nodes). splitter r189 without optimize-mem (max-nodes 1 100 000): 1635 seconds splitter r189 with optimize-mem (max-nodes 1 100 000): 1588 seconds Germany splitter r189 with optimize-mem: 190 seconds splitter r189 without optimize-mem 173 splitter r188 with optimize-mem: 186 splitter r188 without optimize-mem: 182 (just as comparison noted earlier: splitter 181: 225seconds) Austria splitter r189 with optimize-mem: 36 splitter r189 without optimize-mem 34 splitter r188 with optimize-mem: 34 splitter r188 without optimize-mem: 33 r181: 43 seconds Some Statistics on Europe with optimize-mem (without below) Final statistics *********************************************************** coords occupancy MAP occupancy: 579'162'289, number of area dictionary entries: 5596 Length-8 chunks: 5'243'245 (Bytes: 146'810'860) Length-12 chunks: 1'556'563 (Bytes: 56'036'268) Length-16 chunks: 873'095 (Bytes: 38'416'180) Length-20 chunks: 677'690 (Bytes: 35'239'880) Length-24 chunks: 604'329 (Bytes: 36'259'740) Length-28 chunks: 568'332 (Bytes: 38'646'576) Length-32 chunks: 564'759 (Bytes: 42'921'684) Length-36 chunks: 580'701 (Bytes: 48'778'884) Length-40 chunks: 580'432 (Bytes: 53'399'744) Length-44 chunks: 588'583 (Bytes: 58'858'300) Length-48 chunks: 605'760 (Bytes: 65'422'080) Length-52 chunks: 567'664 (Bytes: 65'849'024) Length-56 chunks: 493'862 (Bytes: 61'238'888) Length-60 chunks: 434'713 (Bytes: 57'382'116) Length-64 chunks: 465'204 (Bytes: 65'128'560) Length-68 chunks: 1'481'852 (Bytes: 219'314'096) RLE compresion info: compressed / uncompressed size / ratio: 449'530'736 / 664'184'788 / 33% Sparse chunk vector details: used 15'886'784 of 20'631'616 allocated entries (< 78%) ways occupancy MAP occupancy: 68'744'281, number of area dictionary entries: 8438 Length-8 chunks: 401'162 (Bytes: 11'232'536) Length-12 chunks: 188'797 (Bytes: 6'796'692) Length-16 chunks: 124'579 (Bytes: 5'481'476) Length-20 chunks: 103'673 (Bytes: 5'390'996) Length-24 chunks: 93'180 (Bytes: 5'590'800) Length-28 chunks: 86'282 (Bytes: 5'867'176) Length-32 chunks: 81'099 (Bytes: 6'163'524) Length-36 chunks: 76'133 (Bytes: 6'395'172) Length-40 chunks: 72'515 (Bytes: 6'671'380) Length-44 chunks: 69'720 (Bytes: 6'972'000) Length-48 chunks: 66'263 (Bytes: 7'156'404) Length-52 chunks: 61'777 (Bytes: 7'166'132) Length-56 chunks: 55'173 (Bytes: 6'841'452) Length-60 chunks: 47'582 (Bytes: 6'280'824) Length-64 chunks: 47'710 (Bytes: 6'679'400) Length-68 chunks: 165'749 (Bytes: 24'530'852) RLE compresion info: compressed / uncompressed size / ratio: 52'160'044 / 78'078'696 / 34% Sparse chunk vector details: used 1'741'394 of 2'005'472 allocated entries (< 88%) Thread worker-2 has finished Thread worker-0 has finished Thread worker-1 has finished Time finished: Thu Nov 17 21:09:39 CET 2011 Total time taken: 1588s Some Statistics on Europe (without optimize-mem): Final statistics *********************************************************** coords occupancy MAP occupancy: 579'162'289, number of area dictionary entries: 5596 Length-8 chunks: 5'243'245 (Bytes: 146'810'860) Length-12 chunks: 1'556'563 (Bytes: 56'036'268) Length-16 chunks: 873'095 (Bytes: 38'416'180) Length-20 chunks: 677'690 (Bytes: 35'239'880) Length-24 chunks: 604'329 (Bytes: 36'259'740) Length-28 chunks: 568'332 (Bytes: 38'646'576) Length-32 chunks: 564'759 (Bytes: 42'921'684) Length-36 chunks: 580'701 (Bytes: 48'778'884) Length-40 chunks: 580'432 (Bytes: 53'399'744) Length-44 chunks: 588'583 (Bytes: 58'858'300) Length-48 chunks: 605'760 (Bytes: 65'422'080) Length-52 chunks: 567'664 (Bytes: 65'849'024) Length-56 chunks: 493'862 (Bytes: 61'238'888) Length-60 chunks: 434'713 (Bytes: 57'382'116) Length-64 chunks: 465'204 (Bytes: 65'128'560) Length-68 chunks: 1'481'852 (Bytes: 219'314'096) RLE compresion info: compressed / uncompressed size / ratio: 449'530'736 / 664'184'788 / 33% Chunk vector details: used 15'886'784 of 33'554'432 allocated entries (< 48%) ways occupancy MAP occupancy: 68'744'281, number of area dictionary entries: 8438 Length-8 chunks: 401'162 (Bytes: 11'232'536) Length-12 chunks: 188'797 (Bytes: 6'796'692) Length-16 chunks: 124'579 (Bytes: 5'481'476) Length-20 chunks: 103'673 (Bytes: 5'390'996) Length-24 chunks: 93'180 (Bytes: 5'590'800) Length-28 chunks: 86'282 (Bytes: 5'867'176) Length-32 chunks: 81'099 (Bytes: 6'163'524) Length-36 chunks: 76'133 (Bytes: 6'395'172) Length-40 chunks: 72'515 (Bytes: 6'671'380) Length-44 chunks: 69'720 (Bytes: 6'972'000) Length-48 chunks: 66'263 (Bytes: 7'156'404) Length-52 chunks: 61'777 (Bytes: 7'166'132) Length-56 chunks: 55'173 (Bytes: 6'841'452) Length-60 chunks: 47'582 (Bytes: 6'280'824) Length-64 chunks: 47'710 (Bytes: 6'679'400) Length-68 chunks: 165'749 (Bytes: 24'530'852) RLE compresion info: compressed / uncompressed size / ratio: 52'160'044 / 78'078'696 / 34% Chunk vector details: used 1'741'394 of 33'554'432 allocated entries (< 6%) Thread worker-1 has finished Thread worker-0 has finished Thread worker-2 has finished Time finished: Thu Nov 17 20:37:40 CET 2011 Total time taken: 1635s 20:37:40 europe eu 6550 this is run-10 starting to compile map with mkgmap On 17.11.2011 19:30, Felix Hartmann wrote:
Europe max-nodes 900.000 (resulting in 975 tiles, extract is about 4 weeks old already - hence 4 passes down to 1 pass )
old splitter: 52min 38 seconds (3158 seconds) splitter_patched: 29min 45 seconds (1785 seconds) splitter patched with optimize-mem: 30min 41seconds (1841 seconds)
Hello Felix, okay, thanks again for these numbers. Just one remark regarding speed: My profiling programs says that ~ 3% of the time is now spent in the routines that store the data (Node2AreaMap and below). So, I can't get this much better. Most of the time is spent reading and writing pbf and converting the data between pbf format and internal format. The speed difference between max-nodes 900000 and 1100000 probably comes from the different amount of data written to the pbf files. The smaller the areas, the more data is written to more than one output file, and that slows down processing. ciao, Gerd -- View this message in context: http://gis.638310.n2.nabble.com/PATCH-v5-splitter-memory-usage-tp7001044p700... Sent from the Mkgmap Development mailing list archive at Nabble.com.
On 17.11.2011 22:38, GerdP wrote:
Hello Felix,
okay, thanks again for these numbers. Just one remark regarding speed: My profiling programs says that ~ 3% of the time is now spent in the routines that store the data (Node2AreaMap and below). So, I can't get this much better. Most of the time is spent reading and writing pbf and converting the data between pbf format and internal format. So the biggest speedup would then be a faster HDD or even better a SSD. Well maybe I try it out locally on my PC with SDD tomorrow (though my CPU is much slower, but nice to see where the bottleneck is)
The speed difference between max-nodes 900000 and 1100000 probably comes from the different amount of data written to the pbf files. The smaller the areas, the more data is written to more than one output file, and that slows down processing. Oh yes, that is well possible. Even more as I use --overlap=4000 (instead of default 2000).
participants (3)
-
Felix Hartmann -
GerdP -
WanMil