splitter r325: improved split algo and new option
Hi all, I've applied num-tiles-v1.patch and improved the split algo, see http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5805165.html It is now less likely that splitter creates tiles with a low number of nodes, it is more likely that all tiles have nearly the same number of nodes, and typically you will see fewer tiles. Maybe this also means that you can increase the max-nodes value. I hope this also reduces the need for complex interactions between spltter and mkgmap. Gerd
Seems to be much better now. I don't think I can increase the max-nodes value though, but for most maps the new algo creates less tiles for the same max-nodes value (e.g. Austria from 43 down to 35 for me, with the smallest tile now around 5MB instead of 2.8, and the biggest 12MB instead of 11MB, for Asia I simultaneously increased max-nodes from 800k to 900k- so I'm down from 624 tiles to 493.... and size from 970KB-16MB to now ). So it still seems to depend on the country, but it's already a lot better... It's a bit slower (about 10% more time) On 06.05.2014 13:56, Gerd Petermann wrote:
Hi all,
I've applied num-tiles-v1.patch and improved the split algo, see http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5805165.html
It is now less likely that splitter creates tiles with a low number of nodes, it is more likely that all tiles have nearly the same number of nodes, and typically you will see fewer tiles. Maybe this also means that you can increase the max-nodes value.
I hope this also reduces the need for complex interactions between spltter and mkgmap.
Gerd
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- keep on biking and discovering new trails Felix openmtbmap.org & www.velomap.org
Hi Felix, fine :-) And yes, I meant max-nodes may have to be decreased, not increased. Sorry for the typo. Reg. speed: That's unexpected. It should not take much time to find the better split, (a few seconds) and typically safe some time to have fewer tiles. I'll check that tomorrow. Gerd Felix Hartmann-2 wrote
Seems to be much better now. I don't think I can increase the max-nodes value though, but for most maps the new algo creates less tiles for the same max-nodes value (e.g. Austria from 43 down to 35 for me, with the smallest tile now around 5MB instead of 2.8, and the biggest 12MB instead of 11MB, for Asia I simultaneously increased max-nodes from 800k to 900k- so I'm down from 624 tiles to 493.... and size from 970KB-16MB to now ). So it still seems to depend on the country, but it's already a lot better... It's a bit slower (about 10% more time)
On 06.05.2014 13:56, Gerd Petermann wrote:
Hi all,
I've applied num-tiles-v1.patch and improved the split algo, see http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5805165.html
It is now less likely that splitter creates tiles with a low number of nodes, it is more likely that all tiles have nearly the same number of nodes, and typically you will see fewer tiles. Maybe this also means that you can increase the max-nodes value.
I hope this also reduces the need for complex interactions between spltter and mkgmap.
Gerd
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
-- keep on biking and discovering new trails
Felix openmtbmap.org & www.velomap.org
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
-- View this message in context: http://gis.19327.n5.nabble.com/splitter-r325-improved-split-algo-and-new-opt... Sent from the Mkgmap Development mailing list archive at Nabble.com.
Hi Felix, reg. speed: I can't reproduce that. I compared a split of Germany, both versions (r321 and r325) are more or less running the same time. (I've executed both programs two times to make sure that disk caches are not causing big differences) Or did you mean the combination of splitter + mkgmap to process e.g. Asia? Gerd Date: Tue, 6 May 2014 18:22:00 +0200 From: extremecarver@gmail.com To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] splitter r325: improved split algo and new option Seems to be much better now. I don't think I can increase the max-nodes value though, but for most maps the new algo creates less tiles for the same max-nodes value (e.g. Austria from 43 down to 35 for me, with the smallest tile now around 5MB instead of 2.8, and the biggest 12MB instead of 11MB, for Asia I simultaneously increased max-nodes from 800k to 900k- so I'm down from 624 tiles to 493.... and size from 970KB-16MB to now ). So it still seems to depend on the country, but it's already a lot better... It's a bit slower (about 10% more time) On 06.05.2014 13:56, Gerd Petermann wrote: Hi all, I've applied num-tiles-v1.patch and improved the split algo, see http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5805165.html It is now less likely that splitter creates tiles with a low number of nodes, it is more likely that all tiles have nearly the same number of nodes, and typically you will see fewer tiles. Maybe this also means that you can increase the max-nodes value. I hope this also reduces the need for complex interactions between spltter and mkgmap. Gerd _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev -- keep on biking and discovering new trails Felix openmtbmap.org & www.velomap.org _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Well - I'll update all my maps on Thursday again, to recheck. Maybe it has to do with increasing-maxnodes? Though I thought the higher the max-nodes, the faster... And I only meant splitter. I upgraded mkgmap at the same time (now integrating performance branch changes) - so mkgmap by itself got faster (though it depends on the country - seems like well mapped countries profit a lot more (e.g. Austria like 30% time off), than countries where few continue commands will be in action cause their mapping is basic like Asia). I'm not using any pre-split files or cached files of any sort either... On 07.05.2014 06:49, Gerd Petermann wrote:
Hi Felix,
reg. speed: I can't reproduce that. I compared a split of Germany, both versions (r321 and r325) are more or less running the same time. (I've executed both programs two times to make sure that disk caches are not causing big differences)
Or did you mean the combination of splitter + mkgmap to process e.g. Asia?
Gerd
------------------------------------------------------------------------ Date: Tue, 6 May 2014 18:22:00 +0200 From: extremecarver@gmail.com To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] splitter r325: improved split algo and new option
Seems to be much better now. I don't think I can increase the max-nodes value though, but for most maps the new algo creates less tiles for the same max-nodes value (e.g. Austria from 43 down to 35 for me, with the smallest tile now around 5MB instead of 2.8, and the biggest 12MB instead of 11MB, for Asia I simultaneously increased max-nodes from 800k to 900k- so I'm down from 624 tiles to 493.... and size from 970KB-16MB to now ). So it still seems to depend on the country, but it's already a lot better... It's a bit slower (about 10% more time)
On 06.05.2014 13:56, Gerd Petermann wrote:
Hi all,
I've applied num-tiles-v1.patch and improved the split algo, see http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5805165.html
It is now less likely that splitter creates tiles with a low number of nodes, it is more likely that all tiles have nearly the same number of nodes, and typically you will see fewer tiles. Maybe this also means that you can increase the max-nodes value.
I hope this also reduces the need for complex interactions between spltter and mkgmap.
Gerd
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- keep on biking and discovering new trails
Felix openmtbmap.org &www.velomap.org <http://www.velomap.org>
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- keep on biking and discovering new trails Felix openmtbmap.org & www.velomap.org
Hi Felix, well, nowadays splitter performance mostly depends on I/O if you use o5m format for input and output and give enough heap. Reg. mkgmap performance improvements: yes, that's what I expected. In short, the branch improved the evaluation of tags and the creation of the NOD file. Gerd Date: Wed, 7 May 2014 11:29:10 +0200 From: extremecarver@gmail.com To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] splitter r325: improved split algo and new option Well - I'll update all my maps on Thursday again, to recheck. Maybe it has to do with increasing-maxnodes? Though I thought the higher the max-nodes, the faster... And I only meant splitter. I upgraded mkgmap at the same time (now integrating performance branch changes) - so mkgmap by itself got faster (though it depends on the country - seems like well mapped countries profit a lot more (e.g. Austria like 30% time off), than countries where few continue commands will be in action cause their mapping is basic like Asia). I'm not using any pre-split files or cached files of any sort either... On 07.05.2014 06:49, Gerd Petermann wrote: Hi Felix, reg. speed: I can't reproduce that. I compared a split of Germany, both versions (r321 and r325) are more or less running the same time. (I've executed both programs two times to make sure that disk caches are not causing big differences) Or did you mean the combination of splitter + mkgmap to process e.g. Asia? Gerd Date: Tue, 6 May 2014 18:22:00 +0200 From: extremecarver@gmail.com To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] splitter r325: improved split algo and new option Seems to be much better now. I don't think I can increase the max-nodes value though, but for most maps the new algo creates less tiles for the same max-nodes value (e.g. Austria from 43 down to 35 for me, with the smallest tile now around 5MB instead of 2.8, and the biggest 12MB instead of 11MB, for Asia I simultaneously increased max-nodes from 800k to 900k- so I'm down from 624 tiles to 493.... and size from 970KB-16MB to now ). So it still seems to depend on the country, but it's already a lot better... It's a bit slower (about 10% more time) On 06.05.2014 13:56, Gerd Petermann wrote: Hi all, I've applied num-tiles-v1.patch and improved the split algo, see http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5805165.html It is now less likely that splitter creates tiles with a low number of nodes, it is more likely that all tiles have nearly the same number of nodes, and typically you will see fewer tiles. Maybe this also means that you can increase the max-nodes value. I hope this also reduces the need for complex interactions between spltter and mkgmap. Gerd _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev -- keep on biking and discovering new trails Felix openmtbmap.org & www.velomap.org _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev -- keep on biking and discovering new trails Felix openmtbmap.org & www.velomap.org _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Well I still use pbf and not o5m. First pbf is smaller.. Second - Geofabrik only offers pbf - that's why I stayed with it. I don't think I can cut a lot of time by first converting to 05m, then hand it over to splitter... Actually I also let splitter output pbf... Maybe I could change that in future to 05m.. On 07.05.2014 11:36, Gerd Petermann wrote:
Hi Felix,
well, nowadays splitter performance mostly depends on I/O if you use o5m format for input and output and give enough heap.
Reg. mkgmap performance improvements: yes, that's what I expected. In short, the branch improved the evaluation of tags and the creation of the NOD file.
Gerd
------------------------------------------------------------------------ Date: Wed, 7 May 2014 11:29:10 +0200 From: extremecarver@gmail.com To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] splitter r325: improved split algo and new option
Well - I'll update all my maps on Thursday again, to recheck. Maybe it has to do with increasing-maxnodes? Though I thought the higher the max-nodes, the faster... And I only meant splitter. I upgraded mkgmap at the same time (now integrating performance branch changes) - so mkgmap by itself got faster (though it depends on the country - seems like well mapped countries profit a lot more (e.g. Austria like 30% time off), than countries where few continue commands will be in action cause their mapping is basic like Asia).
I'm not using any pre-split files or cached files of any sort either... On 07.05.2014 06:49, Gerd Petermann wrote:
Hi Felix,
reg. speed: I can't reproduce that. I compared a split of Germany, both versions (r321 and r325) are more or less running the same time. (I've executed both programs two times to make sure that disk caches are not causing big differences)
Or did you mean the combination of splitter + mkgmap to process e.g. Asia?
Gerd
------------------------------------------------------------------------ Date: Tue, 6 May 2014 18:22:00 +0200 From: extremecarver@gmail.com <mailto:extremecarver@gmail.com> To: mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> Subject: Re: [mkgmap-dev] splitter r325: improved split algo and new option
Seems to be much better now. I don't think I can increase the max-nodes value though, but for most maps the new algo creates less tiles for the same max-nodes value (e.g. Austria from 43 down to 35 for me, with the smallest tile now around 5MB instead of 2.8, and the biggest 12MB instead of 11MB, for Asia I simultaneously increased max-nodes from 800k to 900k- so I'm down from 624 tiles to 493.... and size from 970KB-16MB to now ). So it still seems to depend on the country, but it's already a lot better... It's a bit slower (about 10% more time)
On 06.05.2014 13:56, Gerd Petermann wrote:
Hi all,
I've applied num-tiles-v1.patch and improved the split algo, see http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5805165.html
It is now less likely that splitter creates tiles with a low number of nodes, it is more likely that all tiles have nearly the same number of nodes, and typically you will see fewer tiles. Maybe this also means that you can increase the max-nodes value.
I hope this also reduces the need for complex interactions between spltter and mkgmap.
Gerd
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- keep on biking and discovering new trails
Felix openmtbmap.org &www.velomap.org <http://www.velomap.org>
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- keep on biking and discovering new trails
Felix openmtbmap.org &www.velomap.org <http://www.velomap.org>
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- keep on biking and discovering new trails Felix openmtbmap.org & www.velomap.org
Hi Felix, try o5m for both input and output, it is much faster. The command osmcovert --drop-version europe-latest.osm.pbf -o=europe.o5m runs quite fast (~70 seconds on my machine), the o5m file is ~2.430 MB, the pbf file has 2.104 MB. Splitter is much faster reading from o5m when the keep-complete option is in use. (210 secs for the o5m, 441 for pbf) With --output=pbf both are slower, and mkgmap is also much slower. All times with I/O on one normal hard disk. Even better results if you have two different disks for reading and writing. Gerd Date: Wed, 7 May 2014 11:37:58 +0200 From: extremecarver@gmail.com To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] splitter r325: improved split algo and new option Well I still use pbf and not o5m. First pbf is smaller.. Second - Geofabrik only offers pbf - that's why I stayed with it. I don't think I can cut a lot of time by first converting to 05m, then hand it over to splitter... Actually I also let splitter output pbf... Maybe I could change that in future to 05m.. On 07.05.2014 11:36, Gerd Petermann wrote: Hi Felix, well, nowadays splitter performance mostly depends on I/O if you use o5m format for input and output and give enough heap. Reg. mkgmap performance improvements: yes, that's what I expected. In short, the branch improved the evaluation of tags and the creation of the NOD file. Gerd Date: Wed, 7 May 2014 11:29:10 +0200 From: extremecarver@gmail.com To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] splitter r325: improved split algo and new option Well - I'll update all my maps on Thursday again, to recheck. Maybe it has to do with increasing-maxnodes? Though I thought the higher the max-nodes, the faster... And I only meant splitter. I upgraded mkgmap at the same time (now integrating performance branch changes) - so mkgmap by itself got faster (though it depends on the country - seems like well mapped countries profit a lot more (e.g. Austria like 30% time off), than countries where few continue commands will be in action cause their mapping is basic like Asia). I'm not using any pre-split files or cached files of any sort either... On 07.05.2014 06:49, Gerd Petermann wrote: Hi Felix, reg. speed: I can't reproduce that. I compared a split of Germany, both versions (r321 and r325) are more or less running the same time. (I've executed both programs two times to make sure that disk caches are not causing big differences) Or did you mean the combination of splitter + mkgmap to process e.g. Asia? Gerd Date: Tue, 6 May 2014 18:22:00 +0200 From: extremecarver@gmail.com To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] splitter r325: improved split algo and new option Seems to be much better now. I don't think I can increase the max-nodes value though, but for most maps the new algo creates less tiles for the same max-nodes value (e.g. Austria from 43 down to 35 for me, with the smallest tile now around 5MB instead of 2.8, and the biggest 12MB instead of 11MB, for Asia I simultaneously increased max-nodes from 800k to 900k- so I'm down from 624 tiles to 493.... and size from 970KB-16MB to now ). So it still seems to depend on the country, but it's already a lot better... It's a bit slower (about 10% more time) On 06.05.2014 13:56, Gerd Petermann wrote: Hi all, I've applied num-tiles-v1.patch and improved the split algo, see http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5805165.html It is now less likely that splitter creates tiles with a low number of nodes, it is more likely that all tiles have nearly the same number of nodes, and typically you will see fewer tiles. Maybe this also means that you can increase the max-nodes value. I hope this also reduces the need for complex interactions between spltter and mkgmap. Gerd _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev -- keep on biking and discovering new trails Felix openmtbmap.org & www.velomap.org _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev -- keep on biking and discovering new trails Felix openmtbmap.org & www.velomap.org _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev -- keep on biking and discovering new trails Felix openmtbmap.org & www.velomap.org _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
You can also save your country.o5m files from one use to the next and then use osmupdate to update them. This way you don't have to download from Geofabrik and convert from pbf to o5m. El 07/05/14 11:59, Gerd Petermann escribió:
Hi Felix,
try o5m for both input and output, it is much faster. The command osmcovert --drop-version europe-latest.osm.pbf -o=europe.o5m runs quite fast (~70 seconds on my machine), the o5m file is ~2.430 MB, the pbf file has 2.104 MB.
Splitter is much faster reading from o5m when the keep-complete option is in use. (210 secs for the o5m, 441 for pbf)
With --output=pbf both are slower, and mkgmap is also much slower.
All times with I/O on one normal hard disk. Even better results if you have two different disks for reading and writing.
Gerd
------------------------------------------------------------------------ Date: Wed, 7 May 2014 11:37:58 +0200 From: extremecarver@gmail.com To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] splitter r325: improved split algo and new option
Well I still use pbf and not o5m. First pbf is smaller.. Second - Geofabrik only offers pbf - that's why I stayed with it.
I don't think I can cut a lot of time by first converting to 05m, then hand it over to splitter... Actually I also let splitter output pbf... Maybe I could change that in future to 05m.. On 07.05.2014 11:36, Gerd Petermann wrote:
Hi Felix,
well, nowadays splitter performance mostly depends on I/O if you use o5m format for input and output and give enough heap.
Reg. mkgmap performance improvements: yes, that's what I expected. In short, the branch improved the evaluation of tags and the creation of the NOD file.
Gerd
------------------------------------------------------------------------ Date: Wed, 7 May 2014 11:29:10 +0200 From: extremecarver@gmail.com <mailto:extremecarver@gmail.com> To: mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> Subject: Re: [mkgmap-dev] splitter r325: improved split algo and new option
Well - I'll update all my maps on Thursday again, to recheck. Maybe it has to do with increasing-maxnodes? Though I thought the higher the max-nodes, the faster... And I only meant splitter. I upgraded mkgmap at the same time (now integrating performance branch changes) - so mkgmap by itself got faster (though it depends on the country - seems like well mapped countries profit a lot more (e.g. Austria like 30% time off), than countries where few continue commands will be in action cause their mapping is basic like Asia).
I'm not using any pre-split files or cached files of any sort either... On 07.05.2014 06:49, Gerd Petermann wrote:
Hi Felix,
reg. speed: I can't reproduce that. I compared a split of Germany, both versions (r321 and r325) are more or less running the same time. (I've executed both programs two times to make sure that disk caches are not causing big differences)
Or did you mean the combination of splitter + mkgmap to process e.g. Asia?
Gerd
------------------------------------------------------------------------ Date: Tue, 6 May 2014 18:22:00 +0200 From: extremecarver@gmail.com <mailto:extremecarver@gmail.com> To: mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> Subject: Re: [mkgmap-dev] splitter r325: improved split algo and new option
Seems to be much better now. I don't think I can increase the max-nodes value though, but for most maps the new algo creates less tiles for the same max-nodes value (e.g. Austria from 43 down to 35 for me, with the smallest tile now around 5MB instead of 2.8, and the biggest 12MB instead of 11MB, for Asia I simultaneously increased max-nodes from 800k to 900k- so I'm down from 624 tiles to 493.... and size from 970KB-16MB to now ). So it still seems to depend on the country, but it's already a lot better... It's a bit slower (about 10% more time)
On 06.05.2014 13:56, Gerd Petermann wrote:
Hi all,
I've applied num-tiles-v1.patch and improved the split algo, see http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5805165.html
It is now less likely that splitter creates tiles with a low number of nodes, it is more likely that all tiles have nearly the same number of nodes, and typically you will see fewer tiles. Maybe this also means that you can increase the max-nodes value.
I hope this also reduces the need for complex interactions between spltter and mkgmap.
Gerd
Thanks Gerd! This is valuable information for those of us processing large areas of the planet. Unfortunately there is no additional speedup for me because I already use o5m because of osmupdate (to keep a local planet copy up-to-date). On 07/05/2014 11:59, Gerd Petermann wrote:
Hi Felix,
try o5m for both input and output, it is much faster. The command osmcovert --drop-version europe-latest.osm.pbf -o=europe.o5m runs quite fast (~70 seconds on my machine), the o5m file is ~2.430 MB, the pbf file has 2.104 MB.
Splitter is much faster reading from o5m when the keep-complete option is in use. (210 secs for the o5m, 441 for pbf)
With --output=pbf both are slower, and mkgmap is also much slower.
All times with I/O on one normal hard disk. Even better results if you have two different disks for reading and writing.
Gerd
------------------------------------------------------------------------ Date: Wed, 7 May 2014 11:37:58 +0200 From: extremecarver@gmail.com To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] splitter r325: improved split algo and new option
Well I still use pbf and not o5m. First pbf is smaller.. Second - Geofabrik only offers pbf - that's why I stayed with it.
I don't think I can cut a lot of time by first converting to 05m, then hand it over to splitter... Actually I also let splitter output pbf... Maybe I could change that in future to 05m.. On 07.05.2014 11:36, Gerd Petermann wrote:
Hi Felix,
well, nowadays splitter performance mostly depends on I/O if you use o5m format for input and output and give enough heap.
Reg. mkgmap performance improvements: yes, that's what I expected. In short, the branch improved the evaluation of tags and the creation of the NOD file.
Gerd
------------------------------------------------------------------------ Date: Wed, 7 May 2014 11:29:10 +0200 From: extremecarver@gmail.com <mailto:extremecarver@gmail.com> To: mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> Subject: Re: [mkgmap-dev] splitter r325: improved split algo and new option
Well - I'll update all my maps on Thursday again, to recheck. Maybe it has to do with increasing-maxnodes? Though I thought the higher the max-nodes, the faster... And I only meant splitter. I upgraded mkgmap at the same time (now integrating performance branch changes) - so mkgmap by itself got faster (though it depends on the country - seems like well mapped countries profit a lot more (e.g. Austria like 30% time off), than countries where few continue commands will be in action cause their mapping is basic like Asia).
I'm not using any pre-split files or cached files of any sort either... On 07.05.2014 06:49, Gerd Petermann wrote:
Hi Felix,
reg. speed: I can't reproduce that. I compared a split of Germany, both versions (r321 and r325) are more or less running the same time. (I've executed both programs two times to make sure that disk caches are not causing big differences)
Or did you mean the combination of splitter + mkgmap to process e.g. Asia?
Gerd
------------------------------------------------------------------------ Date: Tue, 6 May 2014 18:22:00 +0200 From: extremecarver@gmail.com <mailto:extremecarver@gmail.com> To: mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> Subject: Re: [mkgmap-dev] splitter r325: improved split algo and new option
Seems to be much better now. I don't think I can increase the max-nodes value though, but for most maps the new algo creates less tiles for the same max-nodes value (e.g. Austria from 43 down to 35 for me, with the smallest tile now around 5MB instead of 2.8, and the biggest 12MB instead of 11MB, for Asia I simultaneously increased max-nodes from 800k to 900k- so I'm down from 624 tiles to 493.... and size from 970KB-16MB to now ). So it still seems to depend on the country, but it's already a lot better... It's a bit slower (about 10% more time)
On 06.05.2014 13:56, Gerd Petermann wrote:
Hi all,
I've applied num-tiles-v1.patch and improved the split algo, see http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5805165.html
It is now less likely that splitter creates tiles with a low number of nodes, it is more likely that all tiles have nearly the same number of nodes, and typically you will see fewer tiles. Maybe this also means that you can increase the max-nodes value.
I hope this also reduces the need for complex interactions between spltter and mkgmap.
Gerd
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- keep on biking and discovering new trails
Felix openmtbmap.org &www.velomap.org <http://www.velomap.org>
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- keep on biking and discovering new trails
Felix openmtbmap.org &www.velomap.org <http://www.velomap.org>
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- keep on biking and discovering new trails
Felix openmtbmap.org &www.velomap.org <http://www.velomap.org>
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Lambertus, maybe also look at https://wiki.openstreetmap.org/wiki/Mkgmap/help/splitter#Tuning (the page needs some updates, but that part is correct) Gerd
Date: Thu, 8 May 2014 09:31:28 +0200 From: osm@na1400.info To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] splitter r325: improved split algo and new option
Thanks Gerd! This is valuable information for those of us processing large areas of the planet.
Unfortunately there is no additional speedup for me because I already use o5m because of osmupdate (to keep a local planet copy up-to-date).
On 07/05/2014 11:59, Gerd Petermann wrote:
Hi Felix,
try o5m for both input and output, it is much faster. The command osmcovert --drop-version europe-latest.osm.pbf -o=europe.o5m runs quite fast (~70 seconds on my machine), the o5m file is ~2.430 MB, the pbf file has 2.104 MB.
Splitter is much faster reading from o5m when the keep-complete option is in use. (210 secs for the o5m, 441 for pbf)
With --output=pbf both are slower, and mkgmap is also much slower.
All times with I/O on one normal hard disk. Even better results if you have two different disks for reading and writing.
Gerd
------------------------------------------------------------------------ Date: Wed, 7 May 2014 11:37:58 +0200 From: extremecarver@gmail.com To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] splitter r325: improved split algo and new option
Well I still use pbf and not o5m. First pbf is smaller.. Second - Geofabrik only offers pbf - that's why I stayed with it.
I don't think I can cut a lot of time by first converting to 05m, then hand it over to splitter... Actually I also let splitter output pbf... Maybe I could change that in future to 05m.. On 07.05.2014 11:36, Gerd Petermann wrote:
Hi Felix,
well, nowadays splitter performance mostly depends on I/O if you use o5m format for input and output and give enough heap.
Reg. mkgmap performance improvements: yes, that's what I expected. In short, the branch improved the evaluation of tags and the creation of the NOD file.
Gerd
------------------------------------------------------------------------ Date: Wed, 7 May 2014 11:29:10 +0200 From: extremecarver@gmail.com <mailto:extremecarver@gmail.com> To: mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> Subject: Re: [mkgmap-dev] splitter r325: improved split algo and new option
Well - I'll update all my maps on Thursday again, to recheck. Maybe it has to do with increasing-maxnodes? Though I thought the higher the max-nodes, the faster... And I only meant splitter. I upgraded mkgmap at the same time (now integrating performance branch changes) - so mkgmap by itself got faster (though it depends on the country - seems like well mapped countries profit a lot more (e.g. Austria like 30% time off), than countries where few continue commands will be in action cause their mapping is basic like Asia).
I'm not using any pre-split files or cached files of any sort either... On 07.05.2014 06:49, Gerd Petermann wrote:
Hi Felix,
reg. speed: I can't reproduce that. I compared a split of Germany, both versions (r321 and r325) are more or less running the same time. (I've executed both programs two times to make sure that disk caches are not causing big differences)
Or did you mean the combination of splitter + mkgmap to process e.g. Asia?
Gerd
------------------------------------------------------------------------ Date: Tue, 6 May 2014 18:22:00 +0200 From: extremecarver@gmail.com <mailto:extremecarver@gmail.com> To: mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> Subject: Re: [mkgmap-dev] splitter r325: improved split algo and new option
Seems to be much better now. I don't think I can increase the max-nodes value though, but for most maps the new algo creates less tiles for the same max-nodes value (e.g. Austria from 43 down to 35 for me, with the smallest tile now around 5MB instead of 2.8, and the biggest 12MB instead of 11MB, for Asia I simultaneously increased max-nodes from 800k to 900k- so I'm down from 624 tiles to 493.... and size from 970KB-16MB to now ). So it still seems to depend on the country, but it's already a lot better... It's a bit slower (about 10% more time)
On 06.05.2014 13:56, Gerd Petermann wrote:
Hi all,
I've applied num-tiles-v1.patch and improved the split algo, see http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5805165.html
It is now less likely that splitter creates tiles with a low number of nodes, it is more likely that all tiles have nearly the same number of nodes, and typically you will see fewer tiles. Maybe this also means that you can increase the max-nodes value.
I hope this also reduces the need for complex interactions between spltter and mkgmap.
Gerd
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- keep on biking and discovering new trails
Felix openmtbmap.org &www.velomap.org <http://www.velomap.org>
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- keep on biking and discovering new trails
Felix openmtbmap.org &www.velomap.org <http://www.velomap.org>
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- keep on biking and discovering new trails
Felix openmtbmap.org &www.velomap.org <http://www.velomap.org>
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Am Mittwoch, 7. Mai 2014, 11:37:58 schrieb Felix Hartmann:
Well I still use pbf and not o5m. First pbf is smaller.. Second - Geofabrik only offers pbf - that's why I stayed with it.
I don't think I can cut a lot of time by first converting to 05m, then hand it over to splitter... Actually I also let splitter output pbf... Maybe I could change that in future to 05m..
I download a planet.pbf, convert it to o5m, time ~50 mins,and cut the needed polies from that planet.o5m. an update of my germany.o5m, 3,2 GB. takes now 3 minutes. I update the planet once a month, my extracts everytime i need them. thats much faster then download a new pbf everyday or update this pbf. and with the local planet it is easier to create special maps for friends like bonn+100 or dach+ Bernd -- amarok2 now playing:
okay, well the numbers are convincing. I will change to o5m and osmupdate too - as soon as I find the time (I do think I need 5-10 hours changing my scripts and making sure everything is neatly cut, but right now I'm too timelimited to do so).. On 07.05.2014 12:15, Bernd Weigelt wrote:
Am Mittwoch, 7. Mai 2014, 11:37:58 schrieb Felix Hartmann:
Well I still use pbf and not o5m. First pbf is smaller.. Second - Geofabrik only offers pbf - that's why I stayed with it.
I don't think I can cut a lot of time by first converting to 05m, then hand it over to splitter... Actually I also let splitter output pbf... Maybe I could change that in future to 05m.. I download a planet.pbf, convert it to o5m, time ~50 mins,and cut the needed polies from that planet.o5m. an update of my germany.o5m, 3,2 GB. takes now 3 minutes. I update the planet once a month, my extracts everytime i need them.
thats much faster then download a new pbf everyday or update this pbf. and with the local planet it is easier to create special maps for friends like bonn+100 or dach+
Bernd
-- keep on biking and discovering new trails Felix openmtbmap.org & www.velomap.org
Well, I just ran splitter on the .osm.pbf file for Asia and austria. True the split is a bit faster to o5m - but the difference is not so big. Compiling from osm.pbf or o5m however - was more or less the same speed... on the other hand the o5m splitted files use much more space than osm.pbf (for Austria 293MB vs 450MB - that's about 50% more). I will do some further test with input o5m and output osm.pbf vs output o5m. However if there is also no significant time difference (right now it's like maybe 10% faster output to o5m instead of osm.pbf) - then I think I'm gonna stay splitter output at osm.pbf - simply because on my server I'm a bit harddisk bound (got a large network drive for saving stuff, but with about 20MB/s vs 150MB/s for the local drive, it's really only useful to save stuff there - not for working with anything on the network drive). Splitter input according to the above time measures however - really seems to profit from o5m... On 07.05.2014 12:56, Felix Hartmann wrote:
okay, well the numbers are convincing. I will change to o5m and osmupdate too - as soon as I find the time (I do think I need 5-10 hours changing my scripts and making sure everything is neatly cut, but right now I'm too timelimited to do so).. On 07.05.2014 12:15, Bernd Weigelt wrote:
Am Mittwoch, 7. Mai 2014, 11:37:58 schrieb Felix Hartmann:
Well I still use pbf and not o5m. First pbf is smaller.. Second - Geofabrik only offers pbf - that's why I stayed with it.
I don't think I can cut a lot of time by first converting to 05m, then hand it over to splitter... Actually I also let splitter output pbf... Maybe I could change that in future to 05m.. I download a planet.pbf, convert it to o5m, time ~50 mins,and cut the needed polies from that planet.o5m. an update of my germany.o5m, 3,2 GB. takes now 3 minutes. I update the planet once a month, my extracts everytime i need them.
thats much faster then download a new pbf everyday or update this pbf. and with the local planet it is easier to create special maps for friends like bonn+100 or dach+
Bernd
-- keep on biking and discovering new trails Felix openmtbmap.org & www.velomap.org
Hi Felix, yes, confirmed. The output files are much bigger because splitter never writes the version info (timestamp,uid,user,changeset, version). In mkgmap we read the file only once, not multiple times as in splitter, so the advantage is rather small. Gerd
Date: Wed, 7 May 2014 15:20:51 +0200 From: extremecarver@gmail.com To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] splitter r325: improved split algo and new option
Well, I just ran splitter on the .osm.pbf file for Asia and austria. True the split is a bit faster to o5m - but the difference is not so big.
Compiling from osm.pbf or o5m however - was more or less the same speed... on the other hand the o5m splitted files use much more space than osm.pbf (for Austria 293MB vs 450MB - that's about 50% more).
I will do some further test with input o5m and output osm.pbf vs output o5m. However if there is also no significant time difference (right now it's like maybe 10% faster output to o5m instead of osm.pbf) - then I think I'm gonna stay splitter output at osm.pbf - simply because on my server I'm a bit harddisk bound (got a large network drive for saving stuff, but with about 20MB/s vs 150MB/s for the local drive, it's really only useful to save stuff there - not for working with anything on the network drive). Splitter input according to the above time measures however - really seems to profit from o5m... On 07.05.2014 12:56, Felix Hartmann wrote:
okay, well the numbers are convincing. I will change to o5m and osmupdate too - as soon as I find the time (I do think I need 5-10 hours changing my scripts and making sure everything is neatly cut, but right now I'm too timelimited to do so).. On 07.05.2014 12:15, Bernd Weigelt wrote:
Am Mittwoch, 7. Mai 2014, 11:37:58 schrieb Felix Hartmann:
Well I still use pbf and not o5m. First pbf is smaller.. Second - Geofabrik only offers pbf - that's why I stayed with it.
I don't think I can cut a lot of time by first converting to 05m, then hand it over to splitter... Actually I also let splitter output pbf... Maybe I could change that in future to 05m.. I download a planet.pbf, convert it to o5m, time ~50 mins,and cut the needed polies from that planet.o5m. an update of my germany.o5m, 3,2 GB. takes now 3 minutes. I update the planet once a month, my extracts everytime i need them.
thats much faster then download a new pbf everyday or update this pbf. and with the local planet it is easier to create special maps for friends like bonn+100 or dach+
Bernd
-- keep on biking and discovering new trails
Felix openmtbmap.org & www.velomap.org
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Well, yes - I mainly tried on Austria now, so results may differ a bit by country - but for me the trend is clear. Starting with a .pbf file. Fastest is osmcovert it to o5m (about 15seconds) then splitter output to o5m. Total time to split 1:11 Second fastest os osmconvert it to o5m - then splitter output osm.pbf =1:14 Slowest is splitter osm.pbf to osm.pbf = 2:04min.. For mkgmap austria: osm.pbf (and creating mapset files twice): 3:32 o5m (and creating mapset files twice): 3:30... So for right now, I will output splitter to osm.pbf, but convert first to o5m with osmconvert. After successful split, I directly delete the o5m file. This is using a conventional hdd, maybe with an SSD the o5m advantage is a bit bigger (but then you might be more space bound on the SSD too) The output to osm.pbf by splitter is simply more effective - because o5m is 50% bigger. And for Austria the overall time it is faster is about 5 seconds... Of course osmconvert also needs some time (about 15seconds of the 1:11) - so If I started with an o5m file instead of osm.pbf - it would save that 15seconds... I only update my maps once a week (and europe continent only every ~6 weeks) - so even if I download the planet - I will have to see if I maybe cut it to osm.pbf instead of o5m files to save space... I don't have time right now to set up all that toolchain however. My server is on a 1000Mbit connection - so downloads are faster than the HDD if the source is also fast... For sure converting to o5m before feeding tiles to splitter - makes a lot of sense. Even if all else stays osm.pbf.. On 07.05.2014 15:28, Gerd Petermann wrote:
Hi Felix,
yes, confirmed. The output files are much bigger because splitter never writes the version info (timestamp,uid,user,changeset, version). In mkgmap we read the file only once, not multiple times as in splitter, so the advantage is rather small.
Gerd
Date: Wed, 7 May 2014 15:20:51 +0200 From: extremecarver@gmail.com To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] splitter r325: improved split algo and new option
Well, I just ran splitter on the .osm.pbf file for Asia and austria. True the split is a bit faster to o5m - but the difference is not so big.
Compiling from osm.pbf or o5m however - was more or less the same speed... on the other hand the o5m splitted files use much more space than osm.pbf (for Austria 293MB vs 450MB - that's about 50% more).
I will do some further test with input o5m and output osm.pbf vs output o5m. However if there is also no significant time difference (right now it's like maybe 10% faster output to o5m instead of osm.pbf) - then I think I'm gonna stay splitter output at osm.pbf - simply because on my server I'm a bit harddisk bound (got a large network drive for saving stuff, but with about 20MB/s vs 150MB/s for the local drive, it's really only useful to save stuff there - not for working with anything on the network drive). Splitter input according to the above time measures however - really seems to profit from o5m... On 07.05.2014 12:56, Felix Hartmann wrote:
okay, well the numbers are convincing. I will change to o5m and osmupdate too - as soon as I find the time (I do think I need 5-10 hours changing my scripts and making sure everything is neatly cut, but right now I'm too timelimited to do so).. On 07.05.2014 12:15, Bernd Weigelt wrote:
Am Mittwoch, 7. Mai 2014, 11:37:58 schrieb Felix Hartmann:
Well I still use pbf and not o5m. First pbf is smaller.. Second - Geofabrik only offers pbf - that's why I stayed with it.
I don't think I can cut a lot of time by first converting to 05m, then hand it over to splitter... Actually I also let splitter output pbf... Maybe I could change that in future to 05m.. I download a planet.pbf, convert it to o5m, time ~50 mins,and cut the needed polies from that planet.o5m. an update of my germany.o5m, 3,2 GB. takes now 3 minutes. I update the planet once a month, my extracts everytime i need them.
thats much faster then download a new pbf everyday or update this pbf. and with the local planet it is easier to create special maps for friends like bonn+100 or dach+
Bernd
-- keep on biking and discovering new trails
Felix openmtbmap.org & www.velomap.org
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- keep on biking and discovering new trails Felix openmtbmap.org & www.velomap.org
Hi Felix, I seem to remember that the disadvantage of pbf for output of splitter was higher, but maybe that was before I modified splitter to use a smaller "BatchLimit" in the pbf write routine: configBatchLimit(1000); The default value produced slightly better compression but required much more heap. The original code was optimized for one write process, while splitter may start many hundrets, and in some situations splitter and pbf writer were fighting for the heap, causing heavy work for GC. Of course this only happened when splitting large files like Europe or planet. Gerd Date: Wed, 7 May 2014 16:06:27 +0200 From: extremecarver@gmail.com To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] splitter r325: improved split algo and new option Well, yes - I mainly tried on Austria now, so results may differ a bit by country - but for me the trend is clear. Starting with a .pbf file. Fastest is osmcovert it to o5m (about 15seconds) then splitter output to o5m. Total time to split 1:11 Second fastest os osmconvert it to o5m - then splitter output osm.pbf =1:14 Slowest is splitter osm.pbf to osm.pbf = 2:04min.. For mkgmap austria: osm.pbf (and creating mapset files twice): 3:32 o5m (and creating mapset files twice): 3:30... So for right now, I will output splitter to osm.pbf, but convert first to o5m with osmconvert. After successful split, I directly delete the o5m file. This is using a conventional hdd, maybe with an SSD the o5m advantage is a bit bigger (but then you might be more space bound on the SSD too) The output to osm.pbf by splitter is simply more effective - because o5m is 50% bigger. And for Austria the overall time it is faster is about 5 seconds... Of course osmconvert also needs some time (about 15seconds of the 1:11) - so If I started with an o5m file instead of osm.pbf - it would save that 15seconds... I only update my maps once a week (and europe continent only every ~6 weeks) - so even if I download the planet - I will have to see if I maybe cut it to osm.pbf instead of o5m files to save space... I don't have time right now to set up all that toolchain however. My server is on a 1000Mbit connection - so downloads are faster than the HDD if the source is also fast... For sure converting to o5m before feeding tiles to splitter - makes a lot of sense. Even if all else stays osm.pbf.. On 07.05.2014 15:28, Gerd Petermann wrote: Hi Felix, yes, confirmed. The output files are much bigger because splitter never writes the version info (timestamp,uid,user,changeset, version). In mkgmap we read the file only once, not multiple times as in splitter, so the advantage is rather small. Gerd > Date: Wed, 7 May 2014 15:20:51 +0200 > From: extremecarver@gmail.com > To: mkgmap-dev@lists.mkgmap.org.uk > Subject: Re: [mkgmap-dev] splitter r325: improved split algo and new option > > Well, I just ran splitter on the .osm.pbf file for Asia and austria. > True the split is a bit faster to o5m - but the difference is not so big. > > Compiling from osm.pbf or o5m however - was more or less the same speed... > on the other hand the o5m splitted files use much more space than > osm.pbf (for Austria 293MB vs 450MB - that's about 50% more). > > I will do some further test with input o5m and output osm.pbf vs output > o5m. However if there is also no significant time difference (right now > it's like maybe 10% faster output to o5m instead of osm.pbf) - then I > think I'm gonna stay splitter output at osm.pbf - simply because on my > server I'm a bit harddisk bound (got a large network drive for saving > stuff, but with about 20MB/s vs 150MB/s for the local drive, it's really > only useful to save stuff there - not for working with anything on the > network drive). Splitter input according to the above time measures > however - really seems to profit from o5m... > On 07.05.2014 12:56, Felix Hartmann wrote: > > okay, well the numbers are convincing. I will change to o5m and > > osmupdate too - as soon as I find the time (I do think I need 5-10 > > hours changing my scripts and making sure everything is neatly cut, > > but right now I'm too timelimited to do so).. > > On 07.05.2014 12:15, Bernd Weigelt wrote: > >> Am Mittwoch, 7. Mai 2014, 11:37:58 schrieb Felix Hartmann: > >>> Well I still use pbf and not o5m. > >>> First pbf is smaller.. > >>> Second - Geofabrik only offers pbf - that's why I stayed with it. > >>> > >>> I don't think I can cut a lot of time by first converting to 05m, then > >>> hand it over to splitter... > >>> Actually I also let splitter output pbf... Maybe I could change that in > >>> future to 05m.. > >> I download a planet.pbf, convert it to o5m, time ~50 mins,and cut the > >> needed > >> polies from that planet.o5m. an update of my germany.o5m, 3,2 GB. > >> takes now 3 > >> minutes. I update the planet once a month, my extracts everytime i > >> need them. > >> > >> thats much faster then download a new pbf everyday or update this pbf. > >> and with the local planet it is easier to create special maps for > >> friends like > >> bonn+100 or dach+ > >> > >> Bernd > >> > >> > > > > -- > keep on biking and discovering new trails > > Felix > openmtbmap.org & www.velomap.org > > _______________________________________________ > mkgmap-dev mailing list > mkgmap-dev@lists.mkgmap.org.uk > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev -- keep on biking and discovering new trails Felix openmtbmap.org & www.velomap.org _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Am Dienstag, 6. Mai 2014, 13:56:42 schrieb Gerd Petermann:
It is now less likely that splitter creates tiles with a low number of nodes, it is more likely that all tiles have nearly the same number of nodes, and typically you will see fewer tiles. Maybe this also means that you can increase the max-nodes value.
some infos about splitter 321 vs. 325 Intel i5 QUAD @ 2.7 Ghz, 16 GB RAM ramsize = -Xmx6G maxnodes = 1600000 o5m as input and output rel tiles time use_areas sri lanka o5m size 17.1 M 321 1 2s yes 325 1 3s no 325 1 3s yes oal (Ost-Allgäu) o5m size 78.7 M 321 4 7s yes 325 3 8s no 325 3 7s yes china o5m size 261.1 M 321 16 18s yes 325 15 25s no 325 15 18s yes bonn +100km o5m size 739.9 M 321 35 53s yes 325 28 62s no 325 28 54s yes germany o5m size 3.2 G 321 152 250s yes 325 124 256s yes DACH+ o5m size 8.3 G 321 357 703s yes 325 392 866s no 325 392 733s yes europe (just for fun) o5m size 22.7 G 321 1038 4645s no 325 1046 5267s no 325 1046 4681s yes the whole planet o5m size 45.1 G not tested yet, maybe tonight -- amarok2 now playing:
participants (6)
-
Bernd Weigelt -
Carlos Dávila -
Felix Hartmann -
Gerd Petermann -
GerdP
-
Lambertus