Re: [mkgmap-dev] Memory limits for mkgmap and splitter
Any feedback, questions or suggestions are welcome.
Chris
Hi Chris, thanks a lot for new splitter beta release. I tried your version with a small region (Bayern in Germany). Max nodes 800.000, 12 tiles: old splitter: 310 MB RAM new splitter: 240 MB RAM old splitter: 370 sec new splitter: 470 sec old splitter: problem with 31 relations in more than 4 tiles new splitter: no problem old splitter: problem with 2 ways in more than 4 tiles new splitter: problem with 2 ways in more than 4 tiles So this is basically good news: memory requirement decreased, and the problem with large relations disappeared. The performance is not as good as the old splitter version, but this is not a problem for me. Is it possible to improve your code so that large ways can be in more than 4 tiles? Cheers, Rudi ________________________________________________________________ Neu: WEB.DE Doppel-FLAT mit Internet-Flatrate + Telefon-Flatrate für nur 19,99 Euro/mtl.!* http://produkte.web.de/go/02/
thanks a lot for new splitter beta release. I tried your version with a small region (Bayern in Germany). Max nodes 800.000, 12 tiles:
old splitter: 310 MB RAM new splitter: 240 MB RAM old splitter: 370 sec new splitter: 470 sec
Hmm, I'm surprised the new version is taking longer actually. I didn't benchmark my most recent changes so perhaps I've done something silly. It should be the same speed or faster than the old. Thanks for the statistics, I'll take a look and see what I can do to get it back up to speed.
old splitter: problem with 31 relations in more than 4 tiles new splitter: no problem
That's in line with what I'd expect given my latest change.
old splitter: problem with 2 ways in more than 4 tiles new splitter: problem with 2 ways in more than 4 tiles
Again, this is in line with what I'd expect, I haven't done anything to solve this so far. Fixing ways that span more than 4 tiles is tricky unfortunately without hurting both memory usage and performance. I don't think I'll be able to address this limitation in the short term but I'm certainly aware it's a problem and have some ideas on how to tackle it down the line. Thanks for the feedback, much appreciated. Chris
old splitter: 370 sec new splitter: 470 sec
Hmm, I'm surprised the new version is taking longer actually. I didn't benchmark my most recent changes so perhaps I've done something silly. It should be the same speed or faster than the old. Thanks for the statistics, I'll take a look and see what I can do to get it back up to speed.
I've just run another benchmark against united_kingdom.osm.bz2 (170MB compressed, 2.1GB uncompressed, 9,500,000 nodes). When running against the compressed file I get the following results: old splitter: 590 sec new splitter: 536 sec This is more in line with what I'd expect. Almost all the time savings come from the first stage (calculating the areas.list file), while the second stage takes almost identical times in both splitters. Can you please tell me exactly what parameters you used to run both the old and the new splitters to get your results? Would it be possible to get hold of the osm file you used to run your test against? I'd be interested in replicating the poor performance so I can try to fix it. Regards, Chris
Can you please tell me exactly what parameters you used to run both the old and the new splitters to get your results? Would it be possible to get hold of the osm file you used to run your test against? I'd be interested in replicating the poor performance so I can try to fix it.
Hi Chris, I have run my tests on another machine (my first tests were made with a notebook with virus scanner, harddisk encryption ...): old splitter ( 51922 bytes): 196s new splitter (181951 bytes): 213s This is nearly the same performance (+8%). Windows XP SP2 32Bit, Java 1.6.0_15 Athlon 64 X2 4600+, 2 GB RAM No virus scanner, no harddisk encryption Data: http://download.geofabrik.de/osm/europe/germany/bayern.osm.bz2 I've unzipped the file and called the splitter with these parameters: java -Xmx1400M -ea -jar splitter.jar --mapid=1 --max-nodes=800000 bayern.osm I also made some tests with long ways in germany. With max-nodes 800000 I have 3 ways in more than 4 tiles, but they are not important (Oberpfalz polygons). With max-nodes 400000 I get 12 ways: 2 rivers (Rhein, Mosel), 5 power lines and some unimportant ways. So if I do not use low max-nodes values I have no urgent problem with long ways in germany. Thanks for solving the 4 tiles restriction of relations, this was more important! Rudi
Aha, thank you, this was very helpful! I've been doing most of my testing with compressed osm files but I just did a few runs with an uncompressed file instead and I see what you mean. The problem was due to some buffering I'd added when reading the osm file. It didn't make much difference for compressed files, but made the uncompressed reading a lot worse. Here are the times I get when I run the splitter using the same parameters and osm file as you: old splitter: 117 secs new splitter: 127 secs new splitter (fixed buffering): 103 secs I'm going to start getting this checked in to the codestore, but in the meantime you can grab a new copy from http://redyeti.net/splitter.jar Chris
I have run my tests on another machine (my first tests were made with a notebook with virus scanner, harddisk encryption ...): old splitter ( 51922 bytes): 196s new splitter (181951 bytes): 213s This is nearly the same performance (+8%).
Windows XP SP2 32Bit, Java 1.6.0_15 Athlon 64 X2 4600+, 2 GB RAM No virus scanner, no harddisk encryption Data: http://download.geofabrik.de/osm/europe/germany/bayern.osm.bz2 I've unzipped the file and called the splitter with these parameters:
java -Xmx1400M -ea -jar splitter.jar --mapid=1 --max-nodes=800000 bayern.osm
I also made some tests with long ways in germany. With max-nodes 800000 I have 3 ways in more than 4 tiles, but they are not important (Oberpfalz polygons). With max-nodes 400000 I get 12 ways: 2 rivers (Rhein, Mosel), 5 power lines and some unimportant ways.
So if I do not use low max-nodes values I have no urgent problem with long ways in germany. Thanks for solving the 4 tiles restriction of relations, this was more important!
Rudi
Thank you. Here are my final results: old splitter ( 51922 bytes): 196s (uncompressed) new splitter (175493 bytes): 181s (uncompressed) new splitter (175493 bytes): 197s (gzip) new splitter (175493 bytes): 611s (bzip2) Rudi
participants (3)
-
Chris Miller -
drahtesel42＠web.de -
Rudi