HI Gerd, Using --search-limit=1000000 seems to help a lot for all extracts that were split horribly without a bondary file - it does not help at all for those that split badly using a polygon-file.. Using a bounds file still got better split for brazil or australia-oceania - but only by 2 respectively 3 tiles (while taking twice the time using a boundary file) I think I can drop the polygon-file and instead simply always use the 5x higher search-limit. That seems to be high enough. For Italy the split using a boundary file did not go well - even with higher search-limit. Same for Liechtenstein. Japan strangely split worse with the higher search-limit. Japan and Norway maybe are special cases - without a boundary file they do not split as well. increasing search-limit again by 5 times does not help either. Asia it is not possible to get a nice split without bounds file either - I tried with several search-value values and there is improvement for going to 10000000 but any higher does not help anymore. The split is a lot worse. Though much much better than with default search-value. Hope those debug files are of any use. Asia with search-limit=1000000 fives 1535 // 2489 search-limit=10000000 (10 times / 50 times default) gives: 1483 // 1691 increasing to search-limit=40000000 does not change anything anymore. Japan, Norway, Asia seem to need boundary file for good split. Italy and Liechtenstein have to be split without boundary file. For all other increasing the default --search-value by 5 times to 1000000 is very useful - and does not need much more time. Changing --max-nodes is not a good idea for an automated process.I use the highest for which out of experience mkgmap can still work without getting a too big tile. If mkgmap could itself call splitter to split again tiles that failed - and rewrite the template file, this would be okay. Otherwise I just stick to not optimal splits. Its just too complicated for me to change the build batches to split those tiles that did not compile again. So I am stuck with the max-nodes values that I am using. New higher search limit - 1000000 "for liechtenstein do not use polygon-file - cnt1 = 1 cnt0 = 6" "for australia-oceania use polygon-file - cnt1 = 111 cnt0 = 108" *"for italy do not use polygon-file - cnt1 = 154 cnt0 = 201"* "for brazil use polygon-file - cnt1 = 178 cnt0 = 175" *"for norway use polygon-file - cnt1 = 128 cnt0 = 115" * *"for japan use polygon-file - cnt1 = 194 cnt0 = 155" * "for africa use polygon-file - cnt1 = 665 cnt0 = 662" 5000000 (does not help at all) "for norway use polygon-file - cnt1 = 127 cnt0 = 115" "for japan use polygon-file - cnt1 = 194 cnt0 = 160" Hope you can make use of all those files/data. On Wed, 23 Jun 2021 at 17:26, Gerd Petermann < gpetermann_muenchen@hotmail.com> wrote:
Hi Felix, splitter reports when it's not happy with the split result. Sometimes it gives hints how to improve it.
If you find "Solution is not nice. " in the log you should try to find different options. Sometimes it helps to increase or decrease --max-nodes a little bit, sometimes it helps to use a polygon, sometimes it helps to use a large search-limit value = 10000000
In case of Brazil the higher search-limit helps. The effect of the polygon-file is that some nodes in the OSM file are ignored because they are outside of the polygon. So, just an edge case and the reported numbers are a bit misleading, as splitter really "solves" a split with fewer nodes. Will try to improve that...
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Felix Hartmann <extremecarver@gmail.com> Gesendet: Mittwoch, 23. Juni 2021 15:38 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] mkgmap splitter - Liechtenstein - horrible split if --polygon-file option used.
sorry - I thought after your previous reply that this behaviour is to be expected - but as the differences are so big I wanted to post it here so others are warned. I did not log those other files anywhere and usually use
NUL 2>NUL because I want to keep the terminal clean - if splitter crashes on a split there are usually 0byte files produced - so I just check from them. Same for mkgmap. All those error messages I only look at if there is actually something broken. Else they are simply too much when compiling 180 extracts in 4 versions every week.
On Wed, 23 Jun 2021 at 16:33, Gerd Petermann < gpetermann_muenchen@hotmail.com<mailto:gpetermann_muenchen@hotmail.com>> wrote: Hi Felix,
why don't you keep the extra files produced by splitter somewhere for debugging? They are rather small and make it much easier to understand what happened (at least for me).
Anyhow, no need to run something again for now.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk<mailto: mkgmap-dev-bounces@lists.mkgmap.org.uk>> im Auftrag von Felix Hartmann < extremecarver@gmail.com<mailto:extremecarver@gmail.com>> Gesendet: Mittwoch, 23. Juni 2021 15:25 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] mkgmap splitter - Liechtenstein - horrible split if --polygon-file option used.
No I did not log them - I only used a for loop to get the number of tiles. Brazil, Australia-Ocieania, Asia those differences are really so big that it is hard to believe mkgmap cannot do better. Options otherwise identical. Maxnodes sometimes a bit less than 140... If this is important I can write down for which countries they differed - but I think this happens at any maxnode value.
If you would like me to run all of them again and do a splitter.log I can. Will take 12 hours or so however. I am pretty sure all those splits which are vastly different in size have an error message. But I did not check that. Africa and Italy are quite stunning too. Why is it soo much less tiles not using a boundary file (I downloaded the extracts at the same time as the bound files - so they should be identical if geofabrik has no error).
On Wed, 23 Jun 2021 at 15:58, Gerd Petermann < gpetermann_muenchen@hotmail.com<mailto:gpetermann_muenchen@hotmail.com
<mailto:gpetermann_muenchen@hotmail.com<mailto: gpetermann_muenchen@hotmail.com>>> wrote: Hi Felix,
I can reproduce the surprising results for Brazil with --max-nodes=1400000. This really looks like an error in splitter because splitter should simply do the same after reporting "Warning: shape is too complex, using rectangle (-35.5078125,-74.091796875) to (5.537109375,-27.6416015625) with 213.392.248 nodes instead"
But the search for a good split shows slightly different numbers and the log without polygon file says "Solution is not nice. Can't find a better solution ..." I suggest to look for this message.
I'll try to find out why exactly splitter behaves like that.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk<mailto: mkgmap-dev-bounces@lists.mkgmap.org.uk><mailto: mkgmap-dev-bounces@lists.mkgmap.org.uk<mailto: mkgmap-dev-bounces@lists.mkgmap.org.uk>>> im Auftrag von Gerd Petermann < gpetermann_muenchen@hotmail.com<mailto:gpetermann_muenchen@hotmail.com
<mailto:gpetermann_muenchen@hotmail.com<mailto: gpetermann_muenchen@hotmail.com>>> Gesendet: Mittwoch, 23. Juni 2021 13:49 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] mkgmap splitter - Liechtenstein - horrible split if --polygon-file option used.
Hi Felix,
quite interesting. Do you have the log files from splitter and /or the densities-out.txt files? If yes, please share them. Did you always use the same splitter options?
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk<mailto: mkgmap-dev-bounces@lists.mkgmap.org.uk><mailto: mkgmap-dev-bounces@lists.mkgmap.org.uk<mailto: mkgmap-dev-bounces@lists.mkgmap.org.uk>>> im Auftrag von Felix Hartmann < extremecarver@gmail.com<mailto:extremecarver@gmail.com><mailto: extremecarver@gmail.com<mailto:extremecarver@gmail.com>>> Gesendet: Mittwoch, 23. Juni 2021 13:12 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] mkgmap splitter - Liechtenstein - horrible split if --polygon-file option used.
Okay - well I ran splitter against all geofrabrik extracts that I am having on my server (so nearly all - notably excluding DACH and Greater Britain extract) - and wrote a script comparing the splits. Overall because of Asia continent map it is better to use polygon-file. Seeing those huge differences on Asia, australia-oceania, Italy, Africa, Japan and Brazil I do feel splitter should be doing a bit better. Especially Asia continent is hard to understand the huge huge difference. In general for tiny countries - do not use it. Do not use it for German Bundesländer. Do not use polygon-file for Italy. But do use it on Australia-Oceania, Africa, Japan, Brazil, and Asia continent map.
Now I wonder - will this change in the future? So meaning I have to run this assessment once a year? Usually the additional tiles are mostly tiny - there is no reason for having them. If the split results in the same amount of tiles no matter using --polygon-file or not - it is not listed (overall 176 extracts). I put those countries with huge difference in bold. 26 splits are better not using --polygon-file. 30 splits are better using --polygon-file. 906 tiles less for Asia continent shows that splitting Asia without --polygon-file is a right mess. On the other hand for some tiny countries like Lichtenstein or German Bundesländer, and Africa using a --polygon-file from geofabrik created a way too many useless tiles. cnt1 = using --polygon-file, cnt0=not using polygon-file for splitting.
"for montenegro do not use polygon-file - cnt1 = 2 cnt0 = 7" "for france use polygon-file - cnt1 = 336 cnt0 = 332" "for netherlands do not use polygon-file - cnt1 = 91 cnt0 = 92" "for germany use polygon-file - cnt1 = 259 cnt0 = 258" "for liechtenstein do not use polygon-file - cnt1 = 1 cnt0 = 6" "for monaco do not use polygon-file - cnt1 = 1 cnt0 = 2" "for slovenia do not use polygon-file - cnt1 = 27 cnt0 = 28" "for ukraine use polygon-file - cnt1 = 62 cnt0 = 61" "for norway use polygon-file - cnt1 = 128 cnt0 = 115" "for switzerland do not use polygon-file - cnt1 = 29 cnt0 = 30" "for poland use polygon-file - cnt1 = 133 cnt0 = 127" "for sweden use polygon-file - cnt1 = 60 cnt0 = 54" "for finland do not use polygon-file - cnt1 = 66 cnt0 = 85" "for spain use polygon-file - cnt1 = 91 cnt0 = 89" "for italy do not use polygon-file - cnt1 = 155 cnt0 = 201" "for denmark use polygon-file - cnt1 = 33 cnt0 = 32" "for andorra do not use polygon-file - cnt1 = 1 cnt0 = 4" "for estonia use polygon-file - cnt1 = 9 cnt0 = 8" "for saarland do not use polygon-file - cnt1 = 4 cnt0 = 16" "for hamburg do not use polygon-file - cnt1 = 3 cnt0 = 12" "for hessen do not use polygon-file - cnt1 = 17 cnt0 = 18" "for niedersachsen do not use polygon-file - cnt1 = 26 cnt0 = 27" "for baden-wuerttemberg use polygon-file - cnt1 = 38 cnt0 = 36" "for nordrhein-westfalen do not use polygon-file - cnt1 = 56 cnt0 = 57" "for bayern do not use polygon-file - cnt1 = 47 cnt0 = 48" "for berlin do not use polygon-file - cnt1 = 5 cnt0 = 13" "for australia-oceania use polygon-file - cnt1 = 210 cnt0 = 110" "for south-america do not use polygon-file - cnt1 = 337 cnt0 = 339" "for africa do not use polygon-file - cnt1 = 673 cnt0 = 742" "for russia use polygon-file - cnt1 = 431 cnt0 = 409" "for central-america use polygon-file - cnt1 = 61 cnt0 = 58" "for morocco do not use polygon-file - cnt1 = 19 cnt0 = 32" "for azerbaijan do not use polygon-file - cnt1 = 3 cnt0 = 4" "for malaysia-singapore-brunei do not use polygon-file - cnt1 = 16 cnt0 = 17" "for china use polygon-file - cnt1 = 106 cnt0 = 100" "for india do not use polygon-file - cnt1 = 109 cnt0 = 110" "for indonesia do not use polygon-file - cnt1 = 193 cnt0 = 198" "for japan use polygon-file - cnt1 = 196 cnt0 = 154" "for kazakhstan use polygon-file - cnt1 = 14 cnt0 = 13" "for philippines use polygon-file - cnt1 = 53 cnt0 = 52" "for afghanistan do not use polygon-file - cnt1 = 12 cnt0 = 21" "for myanmar use polygon-file - cnt1 = 18 cnt0 = 17" "for australia use polygon-file - cnt1 = 68 cnt0 = 67" "for new-zealand do not use polygon-file - cnt1 = 27 cnt0 = 28" "for argentina use polygon-file - cnt1 = 33 cnt0 = 27" "for brazil use polygon-file - cnt1 = 265 cnt0 = 177" "for peru use polygon-file - cnt1 = 22 cnt0 = 21" "for canada use polygon-file - cnt1 = 299 cnt0 = 275" "for us-midwest use polygon-file - cnt1 = 155 cnt0 = 154" "for us-pacific use polygon-file - cnt1 = 21 cnt0 = 18" "for us-south use polygon-file - cnt1 = 265 cnt0 = 262" "for us-west use polygon-file - cnt1 = 240 cnt0 = 226" "for greenland use polygon-file - cnt1 = 4 cnt0 = 2" "for mexico use polygon-file - cnt1 = 44 cnt0 = 43" "for reunion do not use polygon-file - cnt1 = 3 cnt0 = 5" "for asia use polygon-file - cnt1 = 2489 cnt0 = 1505"
On Sun, 20 Jun 2021 at 19:12, Gerd Petermann < gpetermann_muenchen@hotmail.com<mailto:gpetermann_muenchen@hotmail.com
<mailto:gpetermann_muenchen@hotmail.com<mailto: gpetermann_muenchen@hotmail.com>><mailto:gpetermann_muenchen@hotmail.com <mailto:gpetermann_muenchen@hotmail.com><mailto: gpetermann_muenchen@hotmail.com<mailto:gpetermann_muenchen@hotmail.com>>>> wrote: Hi Felix,
it tries to reduce the area that is outside the polygon. I think for most countries there is no good reason to use a polygon. Candidates are those which cross 180° and maybe those which have lots of unrelateed coastline inside the bounding box.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk<mailto: mkgmap-dev-bounces@lists.mkgmap.org.uk><mailto: mkgmap-dev-bounces@lists.mkgmap.org.uk<mailto: mkgmap-dev-bounces@lists.mkgmap.org.uk>><mailto: mkgmap-dev-bounces@lists.mkgmap.org.uk<mailto: mkgmap-dev-bounces@lists.mkgmap.org.uk><mailto: mkgmap-dev-bounces@lists.mkgmap.org.uk<mailto: mkgmap-dev-bounces@lists.mkgmap.org.uk>>>> im Auftrag von Felix Hartmann < extremecarver@gmail.com<mailto:extremecarver@gmail.com><mailto: extremecarver@gmail.com<mailto:extremecarver@gmail.com>><mailto: extremecarver@gmail.com<mailto:extremecarver@gmail.com><mailto: extremecarver@gmail.com<mailto:extremecarver@gmail.com>>>> Gesendet: Sonntag, 20. Juni 2021 15:12 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] mkgmap splitter - Liechtenstein - horrible split if --polygon-file option used.
But why is it splitting up into so many parts? So that means you kinda have to check for every country individually if using poly or nor gives the split with less tiles? I would just prefer the split using less tiles so best I write a script to test that for every single country?
On Sun, 20 Jun 2021, 15:27 Gerd Petermann <gpetermann_muenchen@hotmail.com <mailto:gpetermann_muenchen@hotmail.com><mailto: gpetermann_muenchen@hotmail.com<mailto:gpetermann_muenchen@hotmail.com
<mailto:gpetermann_muenchen@hotmail.com<mailto: gpetermann_muenchen@hotmail.com><mailto:gpetermann_muenchen@hotmail.com <mailto:gpetermann_muenchen@hotmail.com>>><mailto: gpetermann_muenchen@hotmail.com<mailto:gpetermann_muenchen@hotmail.com <mailto:gpetermann_muenchen@hotmail.com<mailto: gpetermann_muenchen@hotmail.com>><mailto:gpetermann_muenchen@hotmail.com <mailto:gpetermann_muenchen@hotmail.com><mailto: gpetermann_muenchen@hotmail.com<mailto:gpetermann_muenchen@hotmail.com>>>>> wrote: Hi Felix,
the polygon-file tells splitter that you don't want the rectangular bounds given in the *.osm file. This means you don't want the areas outside of the polygon. If you are happy with the result without polygon file just don't use it
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk<mailto: mkgmap-dev-bounces@lists.mkgmap.org.uk><mailto: mkgmap-dev-bounces@lists.mkgmap.org.uk<mailto: mkgmap-dev-bounces@lists.mkgmap.org.uk>><mailto: mkgmap-dev-bounces@lists.mkgmap.org.uk<mailto: mkgmap-dev-bounces@lists.mkgmap.org.uk><mailto: mkgmap-dev-bounces@lists.mkgmap.org.uk<mailto: mkgmap-dev-bounces@lists.mkgmap.org.uk>>><mailto: mkgmap-dev-bounces@lists.mkgmap.org.uk<mailto: mkgmap-dev-bounces@lists.mkgmap.org.uk><mailto: mkgmap-dev-bounces@lists.mkgmap.org.uk<mailto: mkgmap-dev-bounces@lists.mkgmap.org.uk>><mailto: mkgmap-dev-bounces@lists.mkgmap.org.uk<mailto: mkgmap-dev-bounces@lists.mkgmap.org.uk><mailto: mkgmap-dev-bounces@lists.mkgmap.org.uk<mailto: mkgmap-dev-bounces@lists.mkgmap.org.uk>>>>> im Auftrag von Felix Hartmann <extremecarver@gmail.com<mailto:extremecarver@gmail.com><mailto: extremecarver@gmail.com<mailto:extremecarver@gmail.com>><mailto: extremecarver@gmail.com<mailto:extremecarver@gmail.com><mailto: extremecarver@gmail.com<mailto:extremecarver@gmail.com>>><mailto: extremecarver@gmail.com<mailto:extremecarver@gmail.com><mailto: extremecarver@gmail.com<mailto:extremecarver@gmail.com>><mailto: extremecarver@gmail.com<mailto:extremecarver@gmail.com><mailto: extremecarver@gmail.com<mailto:extremecarver@gmail.com>>>>> Gesendet: Sonntag, 20. Juni 2021 10:17 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] mkgmap splitter - Liechtenstein - horrible split if --polygon-file option used.
I think this applies to all "countries" / areas that are so small that they do not need a split or would split into very few areas. Another example is Monaco (2 tiles instead of 1 tile). For "countries" that are 10 tiles or more anyhow - I did not find this problem.
On Sun, 20 Jun 2021 at 11:08, Felix Hartmann <extremecarver@gmail.com <mailto:extremecarver@gmail.com><mailto:extremecarver@gmail.com<mailto: extremecarver@gmail.com>><mailto:extremecarver@gmail.com<mailto: extremecarver@gmail.com><mailto:extremecarver@gmail.com<mailto: extremecarver@gmail.com>>><mailto:extremecarver@gmail.com<mailto: extremecarver@gmail.com><mailto:extremecarver@gmail.com<mailto: extremecarver@gmail.com>><mailto:extremecarver@gmail.com<mailto: extremecarver@gmail.com><mailto:extremecarver@gmail.com<mailto: extremecarver@gmail.com>>>><mailto:extremecarver@gmail.com<mailto: extremecarver@gmail.com><mailto:extremecarver@gmail.com<mailto: extremecarver@gmail.com>><mailto:extremecarver@gmail.com<mailto: extremecarver@gmail.com><mailto:extremecarver@gmail.com<mailto: extremecarver@gmail.com>>><mailto:extremecarver@gmail.com<mailto: extremecarver@gmail.com><mailto:extremecarver@gmail.com<mailto: extremecarver@gmail.com>><mailto:extremecarver@gmail.com<mailto: extremecarver@gmail.com><mailto:extremecarver@gmail.com<mailto: extremecarver@gmail.com>>>>>> wrote: After last week looking at australia-oceania which was not splitted well without polygon-file - I now noticed that several small countries are split into way too many small tiles IF using .poly boundary file.
I guess there is some bug? This tile should not be split at all - as the node count fits into 1 tile!
Attached the splitter.log both with and without .poly file.
-- Felix Hartman - Openmtbmap.org & VeloMap.org
-- Felix Hartman - Openmtbmap.org & VeloMap.org
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk
<mailto:mkgmap-dev@lists.mkgmap.org.uk<mailto: mkgmap-dev@lists.mkgmap.org.uk>><mailto:mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk><mailto: mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk
<mailto:mkgmap-dev@lists.mkgmap.org.uk<mailto: mkgmap-dev@lists.mkgmap.org.uk><mailto:mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk>><mailto: mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk<mailto: mkgmap-dev@lists.mkgmap.org.uk>>>> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk
<mailto:mkgmap-dev@lists.mkgmap.org.uk<mailto: mkgmap-dev@lists.mkgmap.org.uk>><mailto:mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk><mailto: mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk>>> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- Felix Hartman - Openmtbmap.org & VeloMap.org
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk
<mailto:mkgmap-dev@lists.mkgmap.org.uk<mailto: mkgmap-dev@lists.mkgmap.org.uk>> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk
<mailto:mkgmap-dev@lists.mkgmap.org.uk<mailto: mkgmap-dev@lists.mkgmap.org.uk>> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- Felix Hartman - Openmtbmap.org & VeloMap.org
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- Felix Hartman - Openmtbmap.org & VeloMap.org
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- Felix Hartman - Openmtbmap.org & VeloMap.org