splitter performance with "osm.pbf" compared to "o5m"
Hi all, I noticed that splitter has a major performance lag on osm.pbf files compared to o5m. splitting off my DACH region in pbf format takes 15min 45sec, while splitting the same region in o5m format is finished after 5min 17sec. So it takes almost 3 times as long splitting pbf data. mkgmap on the other hand has almost no difference at all when building the DACH map from the pbf-tiles, compared to o5m-tiles. mkgmap with pbf takes 12min 28sec, with o5m 12min 09sec. here are my command lines: java -ea -XX:+AggressiveHeap -jar splitter-r597/splitter.jar > splitter.log --geonames-file=cities15000.zip --mapid=65000001 --output=pbf --precomp-sea=sea-latest.zip --write-kml=dach.kml --keep-complete --overlap=0 --max-nodes=1200000 dach.osm.pbf java -ea -XX:+AggressiveHeap -jar splitter-r597/splitter.jar > splitter.log --geonames-file=cities15000.zip --mapid=65000001 --output=o5m --precomp-sea=sea-latest.zip --write-kml=dach.kml --keep-complete --overlap=0 --max-nodes=1200000 dach.o5m Any ideas why pbf splitting is so slow compared to o5m splitting ? Ciao, Franco
Hi Franco, first see https://wiki.openstreetmap.org/wiki/Mkgmap/help/splitter#Tuning_for_best_per.... I think the difference should not be that big when splitter has enough memory, so maybe you should add the -Xmx JRE parameter. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Franco Bez <franco.bez@web.de> Gesendet: Samstag, 19. Dezember 2020 09:40 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: [mkgmap-dev] splitter performance with "osm.pbf" compared to "o5m" Hi all, I noticed that splitter has a major performance lag on osm.pbf files compared to o5m. splitting off my DACH region in pbf format takes 15min 45sec, while splitting the same region in o5m format is finished after 5min 17sec. So it takes almost 3 times as long splitting pbf data. mkgmap on the other hand has almost no difference at all when building the DACH map from the pbf-tiles, compared to o5m-tiles. mkgmap with pbf takes 12min 28sec, with o5m 12min 09sec. here are my command lines: java -ea -XX:+AggressiveHeap -jar splitter-r597/splitter.jar > splitter.log --geonames-file=cities15000.zip --mapid=65000001 --output=pbf --precomp-sea=sea-latest.zip --write-kml=dach.kml --keep-complete --overlap=0 --max-nodes=1200000 dach.osm.pbf java -ea -XX:+AggressiveHeap -jar splitter-r597/splitter.jar > splitter.log --geonames-file=cities15000.zip --mapid=65000001 --output=o5m --precomp-sea=sea-latest.zip --write-kml=dach.kml --keep-complete --overlap=0 --max-nodes=1200000 dach.o5m Any ideas why pbf splitting is so slow compared to o5m splitting ? Ciao, Franco
Hi Gerd, thanks for the fast reply. I tried "java -ea -Xmx16G -Xms16G" but this doen't make any difference on my machine, AMD Ryzen 7 3700X 32GB Ram 2 SSDs 1TB each see the log extracts below. Ciao, Franco ----O5M-------------------------------------------------------------------------------------------------------------------------------- Splitter version 597 compiled 2020-01-30T14:10:47+0000 boundary-tags=use-exclude-list cache= description= geonames-file=/home/franco/map_build/cities15000.zip handle-element-version=remove ignore-osm-bounds=false keep-complete=true mapid=65000001 max-areas=2048 max-nodes=1200000 max-threads=16 (auto) mixed=false no-trim=false num-tiles= output=o5m output-dir= overlap=0 polygon-desc-file= polygon-file= precomp-sea=/home/franco/map_build/precomp/sea-latest.zip problem-file= problem-report= resolution=13 route-rel-values= search-limit=200000 split-file= status-freq=120 stop-after=dist wanted-admin-level=5 write-kml=dach.kml Elapsed time: 0s Memory: Current 16384MB (19MB used, 16365MB free) Max 16384MB Time started: Sat Dec 19 10:10:21 CET 2020 Time finished: Sat Dec 19 10:15:44 CET 2020 Total time taken: 5 minutes 22 seconds ---PBF------------------------------------------------------------------------------------------------------------------------------ Splitter version 597 compiled 2020-01-30T14:10:47+0000 boundary-tags=use-exclude-list geonames-file=/home/franco/map_build/cities15000.zip handle-element-version=remove ignore-osm-bounds=false keep-complete=true mapid=65000001 max-areas=2048 max-nodes=1200000 max-threads=16 (auto) mixed=false no-trim=false num-tiles= output=pbf output-dir= overlap=0 polygon-desc-file= polygon-file= precomp-sea=/home/franco/map_build/precomp/sea-latest.zip problem-file= problem-report= resolution=13 route-rel-values= search-limit=200000 split-file= status-freq=120 stop-after=dist wanted-admin-level=5 write-kml=dach.kml Elapsed time: 0s Memory: Current 16384MB (19MB used, 16365MB free) Max 16384MB Time started: Sat Dec 19 10:33:28 CET 2020 Time finished: Sat Dec 19 10:51:12 CET 2020 Total time taken: 17 minutes 43 seconds
Hi Franco, OK, I guess 2048 areas should be enough and memory is for sure enough. The major reason that I added o5m support to splitter was that this format was much faster to read, esp. when the file is read multiple times some passes only need relations or only ways. The pbf format didn't support this as well, but I thought that I also improved handling of that format. If you like you can zip the two logs and upload them to http://files.mkgmap.org.uk . I'll do some tests on my own machine again. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Franco Bez <franco.bez@web.de> Gesendet: Samstag, 19. Dezember 2020 11:45 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] splitter performance with "osm.pbf" compared to "o5m" Hi Gerd, thanks for the fast reply. I tried "java -ea -Xmx16G -Xms16G" but this doen't make any difference on my machine, AMD Ryzen 7 3700X 32GB Ram 2 SSDs 1TB each see the log extracts below. Ciao, Franco ----O5M-------------------------------------------------------------------------------------------------------------------------------- Splitter version 597 compiled 2020-01-30T14:10:47+0000 boundary-tags=use-exclude-list cache= description= geonames-file=/home/franco/map_build/cities15000.zip handle-element-version=remove ignore-osm-bounds=false keep-complete=true mapid=65000001 max-areas=2048 max-nodes=1200000 max-threads=16 (auto) mixed=false no-trim=false num-tiles= output=o5m output-dir= overlap=0 polygon-desc-file= polygon-file= precomp-sea=/home/franco/map_build/precomp/sea-latest.zip problem-file= problem-report= resolution=13 route-rel-values= search-limit=200000 split-file= status-freq=120 stop-after=dist wanted-admin-level=5 write-kml=dach.kml Elapsed time: 0s Memory: Current 16384MB (19MB used, 16365MB free) Max 16384MB Time started: Sat Dec 19 10:10:21 CET 2020 Time finished: Sat Dec 19 10:15:44 CET 2020 Total time taken: 5 minutes 22 seconds ---PBF------------------------------------------------------------------------------------------------------------------------------ Splitter version 597 compiled 2020-01-30T14:10:47+0000 boundary-tags=use-exclude-list geonames-file=/home/franco/map_build/cities15000.zip handle-element-version=remove ignore-osm-bounds=false keep-complete=true mapid=65000001 max-areas=2048 max-nodes=1200000 max-threads=16 (auto) mixed=false no-trim=false num-tiles= output=pbf output-dir= overlap=0 polygon-desc-file= polygon-file= precomp-sea=/home/franco/map_build/precomp/sea-latest.zip problem-file= problem-report= resolution=13 route-rel-values= search-limit=200000 split-file= status-freq=120 stop-after=dist wanted-admin-level=5 write-kml=dach.kml Elapsed time: 0s Memory: Current 16384MB (19MB used, 16365MB free) Max 16384MB Time started: Sat Dec 19 10:33:28 CET 2020 Time finished: Sat Dec 19 10:51:12 CET 2020 Total time taken: 17 minutes 43 seconds
|Hi Gerd,| |here are the two log files. | |http://files.mkgmap.org.uk/download/491/logs.tgz| |Ciao,| |Franco | Am 19.12.20 um 12:09 schrieb Gerd Petermann:
Hi Franco,
OK, I guess 2048 areas should be enough and memory is for sure enough. The major reason that I added o5m support to splitter was that this format was much faster to read, esp. when the file is read multiple times some passes only need relations or only ways. The pbf format didn't support this as well, but I thought that I also improved handling of that format. If you like you can zip the two logs and upload them to http://files.mkgmap.org.uk .
I'll do some tests on my own machine again.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Franco Bez <franco.bez@web.de> Gesendet: Samstag, 19. Dezember 2020 11:45 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] splitter performance with "osm.pbf" compared to "o5m"
Hi Gerd,
thanks for the fast reply.
I tried "java -ea -Xmx16G -Xms16G" but this doen't make any difference on my machine, AMD Ryzen 7 3700X 32GB Ram 2 SSDs 1TB each
see the log extracts below.
Ciao,
Franco
Hi Franco, the differences in my numbers are not that drastical. I tried with Germany from geofabrik and got 5 minutes 21 seconds. for the o5m format and 12 min for pbf. Maybe the pbf reader could be improved to use multiple threads, but that's not my strength. BTW: Your logs show many warnings like "Sorry, way 4216363 is missing 29 node(s)." . Maybe you should check your tool chain if these ways are relavant for you. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Franco Bez <franco.bez@web.de> Gesendet: Samstag, 19. Dezember 2020 13:37 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] splitter performance with "osm.pbf" compared to "o5m" |Hi Gerd,| |here are the two log files. | |http://files.mkgmap.org.uk/download/491/logs.tgz| |Ciao,| |Franco | Am 19.12.20 um 12:09 schrieb Gerd Petermann:
Hi Franco,
OK, I guess 2048 areas should be enough and memory is for sure enough. The major reason that I added o5m support to splitter was that this format was much faster to read, esp. when the file is read multiple times some passes only need relations or only ways. The pbf format didn't support this as well, but I thought that I also improved handling of that format. If you like you can zip the two logs and upload them to http://files.mkgmap.org.uk .
I'll do some tests on my own machine again.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Franco Bez <franco.bez@web.de> Gesendet: Samstag, 19. Dezember 2020 11:45 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] splitter performance with "osm.pbf" compared to "o5m"
Hi Gerd,
thanks for the fast reply.
I tried "java -ea -Xmx16G -Xms16G" but this doen't make any difference on my machine, AMD Ryzen 7 3700X 32GB Ram 2 SSDs 1TB each
see the log extracts below.
Ciao,
Franco
Hi Gerd, thanks for your tests. On your machine it's still more than double, on my machine almost tripple the time for pbf compared to o5m. Thanks for the hint with "ways missing nodes", I'll check this. Ciao, Franco Am 22.12.20 um 10:51 schrieb Gerd Petermann:
Hi Franco,
the differences in my numbers are not that drastical. I tried with Germany from geofabrik and got 5 minutes 21 seconds. for the o5m format and 12 min for pbf. Maybe the pbf reader could be improved to use multiple threads, but that's not my strength.
BTW: Your logs show many warnings like "Sorry, way 4216363 is missing 29 node(s)." . Maybe you should check your tool chain if these ways are relavant for you.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Franco Bez <franco.bez@web.de> Gesendet: Samstag, 19. Dezember 2020 13:37 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] splitter performance with "osm.pbf" compared to "o5m"
|Hi Gerd,|
|here are the two log files. |
|http://files.mkgmap.org.uk/download/491/logs.tgz|
|Ciao,|
|Franco |
Am 19.12.20 um 12:09 schrieb Gerd Petermann:
Hi Franco,
OK, I guess 2048 areas should be enough and memory is for sure enough. The major reason that I added o5m support to splitter was that this format was much faster to read, esp. when the file is read multiple times some passes only need relations or only ways. The pbf format didn't support this as well, but I thought that I also improved handling of that format. If you like you can zip the two logs and upload them to http://files.mkgmap.org.uk .
I'll do some tests on my own machine again.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Franco Bez <franco.bez@web.de> Gesendet: Samstag, 19. Dezember 2020 11:45 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] splitter performance with "osm.pbf" compared to "o5m"
Hi Gerd,
thanks for the fast reply.
I tried "java -ea -Xmx16G -Xms16G" but this doen't make any difference on my machine, AMD Ryzen 7 3700X 32GB Ram 2 SSDs 1TB each
see the log extracts below.
Ciao,
Franco
Hi Franco, just noticed that I used o5m for output in both tests, so my numbers for pbf are probably a bit to small. Anyway, I think this just proves that it was a good idea to implement the o5m support ;) Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Franco Bez <franco.bez@web.de> Gesendet: Dienstag, 22. Dezember 2020 20:41 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] splitter performance with "osm.pbf" compared to "o5m" Hi Gerd, thanks for your tests. On your machine it's still more than double, on my machine almost tripple the time for pbf compared to o5m. Thanks for the hint with "ways missing nodes", I'll check this. Ciao, Franco Am 22.12.20 um 10:51 schrieb Gerd Petermann:
Hi Franco,
the differences in my numbers are not that drastical. I tried with Germany from geofabrik and got 5 minutes 21 seconds. for the o5m format and 12 min for pbf. Maybe the pbf reader could be improved to use multiple threads, but that's not my strength.
BTW: Your logs show many warnings like "Sorry, way 4216363 is missing 29 node(s)." . Maybe you should check your tool chain if these ways are relavant for you.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Franco Bez <franco.bez@web.de> Gesendet: Samstag, 19. Dezember 2020 13:37 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] splitter performance with "osm.pbf" compared to "o5m"
|Hi Gerd,|
|here are the two log files. |
|http://files.mkgmap.org.uk/download/491/logs.tgz|
|Ciao,|
|Franco |
Am 19.12.20 um 12:09 schrieb Gerd Petermann:
Hi Franco,
OK, I guess 2048 areas should be enough and memory is for sure enough. The major reason that I added o5m support to splitter was that this format was much faster to read, esp. when the file is read multiple times some passes only need relations or only ways. The pbf format didn't support this as well, but I thought that I also improved handling of that format. If you like you can zip the two logs and upload them to http://files.mkgmap.org.uk .
I'll do some tests on my own machine again.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Franco Bez <franco.bez@web.de> Gesendet: Samstag, 19. Dezember 2020 11:45 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] splitter performance with "osm.pbf" compared to "o5m"
Hi Gerd,
thanks for the fast reply.
I tried "java -ea -Xmx16G -Xms16G" but this doen't make any difference on my machine, AMD Ryzen 7 3700X 32GB Ram 2 SSDs 1TB each
see the log extracts below.
Ciao,
Franco
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
participants (2)
-
Franco Bez -
Gerd Petermann