Hi all, the last months brought a lot of improvements, but there is still a lot to do: Before summer I only plan to improve readability/stability and maybe performance of code:use one internal representation for vehicles (mkgmap:car, mkgmap:bicycle,...) and only translate it to the img format when writingCreate a new class that combines a converted Way, the reference to the GType, and fields required for the creation of MapRoad objects. I think ConvertedWay is a good name for this. This should be used in StyledConverter, WrongAngleFixer, and RoadMerger.The RuleIndex might be a lot faster if we search for keys first and in a 2nd step for values. This would avoid millions of temporary StringBuilder objects. Other ideas that should not be forgotton: improve handling of motor_vehicle=* and vehicle=* in default style (WanMil and Mario Hantschke are working on that) manage drive-on-left/drive-on-right in resources\LocatorConfig.xmldetect sharp angles at road junctions and either change the heading values or add small arcsconditional includes in style files and/or if else statements error handling : set non-zero return code if the map (or at least one tile) is not usableimprove reader for polish input data: correct handling of mulitple DATAx lines rewrite MapSplitter so that overlaps are kept small. This should improve rendering speed in the devices. rewrite classes that hold information about map objects, esp. shapes. Idea: store information about outer way and holes as lists of points, similar to mp-relationsimplement methods that return the simplified object for a given resolution. If the area enclosed by the outer line is not visible at resolution x, ignore the object. If a hole is not visible at resolution x, ignore it. If both are visible, connect the simplified outer and inner lines so that holes are displayed in the map. This should all be doable without complex area operations.merge shapes before splitting the map into sub areasoptimize wrong angles in shapes with the same algo that is used for lines Expected result: fewer and less distorted shapes, smaller img file. The order of the 2nd list represents my assumption regarding complexity, not importance. I think the last 3 (bold) points can only be done together, but I'd like to be proven wrong. Is something important missing? @Steve: Maybe we should keep this list somewhere at http://www.mkgmap.org.uk/ and try to maintain it? Gerd
Hi Gerd, Bugfix: Internal oneway handling At the moment all oneway ways are normalized to onway=yes (they are reversed when they are tagged with oneway=-1). But all other tags are left untouched. So that will be something like http://josm.openstreetmap.de/browser/josm/trunk/src/org/openstreetmap/josm/c... RFE: Parameters for style functions maxspeedkmh() evaluates the maxspeed tag only. It might be useful to have something like kmh(${maxspeed:forward}). WanMil
Hi all,
the last months brought a lot of improvements, but there is still a lot to do: Before summer I only plan to improve readability/stability and maybe performance of code:
1. use one internal representation for vehicles (mkgmap:car, mkgmap:bicycle,...) and only translate it to the img format when writing 2. Create a new class that combines a converted Way, the reference to the GType, and fields required for the creation of MapRoad objects. I think ConvertedWay is a good name for this. This should be used in StyledConverter, WrongAngleFixer, and RoadMerger. 3. The RuleIndex might be a lot faster if we search for keys first and in a 2nd step for values. This would avoid millions of temporary StringBuilder objects.
Other ideas that should not be forgotton:
* improve handling of motor_vehicle=* and vehicle=* in default style (WanMil and Mario Hantschke are working on that) * manage drive-on-left/drive-on-right in resources\LocatorConfig.xml * detect sharp angles at road junctions and either change the heading values or add small arcs * conditional includes in style files and/or if else statements * error handling : set non-zero return code if the map (or at least one tile) is not usable * improve reader for polish input data: correct handling of mulitple DATAx lines * rewrite MapSplitter so that overlaps are kept small. This should improve rendering speed in the devices. * rewrite classes that hold information about map objects, esp. shapes. Idea: o store information about outer way and holes as lists of points, similar to mp-relations o implement methods that return the simplified object for a given resolution. If the area enclosed by the outer line is not visible at resolution x, ignore the object. If a hole is not visible at resolution x, ignore it. If both are visible, connect the simplified outer and inner lines so that holes are displayed in the map. This should all be doable without complex area operations. o merge shapes before splitting the map into sub areas o optimize wrong angles in shapes with the same algo that is used for lines Expected result: fewer and less distorted shapes, smaller img file.
The order of the 2nd list represents my assumption regarding complexity, not importance. I think the last 3 (bold) points can only be done together, but I'd like to be proven wrong.
Is something important missing?
@Steve: Maybe we should keep this list somewhere at http://www.mkgmap.org.uk/ and try to maintain it?
Gerd
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi WanMil, WanMil wrote
Bugfix: Internal oneway handling At the moment all oneway ways are normalized to onway=yes (they are reversed when they are tagged with oneway=-1). But all other tags are left untouched. So that will be something like http://josm.openstreetmap.de/browser/josm/trunk/src/org/openstreetmap/josm/c...
this normalisation happens after style processing, and I don't see code in mkgmap that interprets tags that depend on the direction, e.g. bicycle:backward. All this happens in the style. Why do you think that it is a bug? Gerd -- View this message in context: http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5803443.html Sent from the Mkgmap Development mailing list archive at Nabble.com.
Am 16.04.2014 22:01, schrieb GerdP:
Why do you think that it is a bug? Think about oneway=-1 and in style you add oneway-arrows (link mapnik does). In style you will handle -1 special, because arrows should drawn in oposite direction of way. If mkgmap now changes direction of the way, arrows should be in wrong direction, or am I wrong?
Henning
Hi Henning, okay. I think WanMil told me that before, but I forgot again ... A few details: mkgmap always reversed routable ways with oneway=-1 or oneway=reverse after style processing. With r2944 I've changed the code so that this happens before calling RoadMerger and also that overlaying lines are reversed. If I got that right, the style may use different types for overlaying lines, depending on the oneway tag, so if you use the same style for r2943 or older and for current version, you should see a difference for oneway=-1 ways. I don't know how difficult it is to change the style to produce correct results with current version again, but you have to remember that the way is reversed, so left will be right, forward will be backward. I see two alternatives: 1) mkgmap reverses the way before style processing. This requires also to reverse all tags like bicycle:left, car:forward etc, so it is not trivial. On the other hand, it might allow to merge more roads in RoadMerger. 2) mkgmap makes sure that overlaying lines are NOT reversed. I think this can be done with the attached patch. By the way: There are also tags on nodes which depend on the direction of the way, see http://gis.19327.n5.nabble.com/direction-forward-backward-on-nodes-tt5803072... I think we can ignore this because in the points file you can't detect the direction of the way. Gerd Date: Wed, 16 Apr 2014 23:00:35 +0200 From: osm@aighes.de To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] mkgmap ToDo list Am 16.04.2014 22:01, schrieb GerdP: Why do you think that it is a bug? Think about oneway=-1 and in style you add oneway-arrows (link mapnik does). In style you will handle -1 special, because arrows should drawn in oposite direction of way. If mkgmap now changes direction of the way, arrows should be in wrong direction, or am I wrong? Henning _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
I currently have basically no time - so I'm just reading on the list in skip through mode. Unable to test - but I'm pretty sure there is a big bug related to the oneway handling and reversing of ways. I'm also quite sure it wasn't there last year and got introduced with the change of the style-file getting all tags instead of internal handling. My style is a big mess right now (trying to adapt to the changes needed, but fucking up because no set mkgmap:access=yes/no and delete mkgmap:access shortcut exists anymore amongst other imcompatibilities compared to before) - but I'm pretty sure that the oneway handling - also related to oneway=yes or oneway=-1 set in the lines file somehow doesn't work correctly. I often reverse ways - in order to have depending on direction different classification of the way (regarding both layout and routing priority) and have noticed that lately I'm routed against the intended direction because mkgmap puts the way in the wrong direction. And I'm 95% sure that the error is on mkgmap not the style-file but didn't find the time to analyze... Being in China right now behind the great firewall - it's also cumbersome to access my SVN and other things as they are blocked - I won't be able to help much/at all before mid June in finding or specifying bugs... Alternative 1 sounds best - as it would be in line with the old behaviour if I'm not incorrectly understanding it here... Or Alternative 3 - don't do anything!! Leave it to the style developer - wasn't this the old behaviour? I don't know since when this is bugged - but I'm sure 1 year ago everything worked correctly if you assigned all ways to either oneway=yes or oneway=-1 (or was it reverse?). Only thing to normalize was true or 1 to yes and so on... On 17.04.2014 13:42, Gerd Petermann wrote:
Hi Henning,
okay. I think WanMil told me that before, but I forgot again ...
A few details:
mkgmap always reversed routable ways with oneway=-1 or oneway=reverse after style processing.
With r2944 I've changed the code so that this happens before calling RoadMerger and also that overlaying lines are reversed.
If I got that right, the style may use different types for overlaying lines, depending on the oneway tag, so if you use the same style for r2943 or older and for current version, you should see a difference for oneway=-1 ways. I don't know how difficult it is to change the style to produce correct results with current version again, but you have to remember that the way is reversed, so left will be right, forward will be backward.
I see two alternatives: 1) mkgmap reverses the way before style processing. This requires also to reverse all tags like bicycle:left, car:forward etc, so it is not trivial. On the other hand, it might allow to merge more roads in RoadMerger. 2) mkgmap makes sure that overlaying lines are NOT reversed. I think this can be done with the attached patch.
By the way: There are also tags on nodes which depend on the direction of the way, see http://gis.19327.n5.nabble.com/direction-forward-backward-on-nodes-tt5803072...
I think we can ignore this because in the points file you can't detect the direction of the way.
Gerd ------------------------------------------------------------------------ Date: Wed, 16 Apr 2014 23:00:35 +0200 From: osm@aighes.de To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] mkgmap ToDo list
Am 16.04.2014 22:01, schrieb GerdP:
Why do you think that it is a bug?
Think about oneway=-1 and in style you add oneway-arrows (link mapnik does). In style you will handle -1 special, because arrows should drawn in oposite direction of way. If mkgmap now changes direction of the way, arrows should be in wrong direction, or am I wrong?
Henning
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Felix, the old behaviour (before r2944 which was committed 2014-01-05) was that only the routable ways were reversed. The patch should bring back this behaviour. Gerd Date: Thu, 17 Apr 2014 13:54:39 +0800 From: extremecarver@gmail.com To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] mkgmap ToDo list I currently have basically no time - so I'm just reading on the list in skip through mode. Unable to test - but I'm pretty sure there is a big bug related to the oneway handling and reversing of ways. I'm also quite sure it wasn't there last year and got introduced with the change of the style-file getting all tags instead of internal handling. My style is a big mess right now (trying to adapt to the changes needed, but fucking up because no set mkgmap:access=yes/no and delete mkgmap:access shortcut exists anymore amongst other imcompatibilities compared to before) - but I'm pretty sure that the oneway handling - also related to oneway=yes or oneway=-1 set in the lines file somehow doesn't work correctly. I often reverse ways - in order to have depending on direction different classification of the way (regarding both layout and routing priority) and have noticed that lately I'm routed against the intended direction because mkgmap puts the way in the wrong direction. And I'm 95% sure that the error is on mkgmap not the style-file but didn't find the time to analyze... Being in China right now behind the great firewall - it's also cumbersome to access my SVN and other things as they are blocked - I won't be able to help much/at all before mid June in finding or specifying bugs... Alternative 1 sounds best - as it would be in line with the old behaviour if I'm not incorrectly understanding it here... Or Alternative 3 - don't do anything!! Leave it to the style developer - wasn't this the old behaviour? I don't know since when this is bugged - but I'm sure 1 year ago everything worked correctly if you assigned all ways to either oneway=yes or oneway=-1 (or was it reverse?). Only thing to normalize was true or 1 to yes and so on... On 17.04.2014 13:42, Gerd Petermann wrote: Hi Henning, okay. I think WanMil told me that before, but I forgot again ... A few details: mkgmap always reversed routable ways with oneway=-1 or oneway=reverse after style processing. With r2944 I've changed the code so that this happens before calling RoadMerger and also that overlaying lines are reversed. If I got that right, the style may use different types for overlaying lines, depending on the oneway tag, so if you use the same style for r2943 or older and for current version, you should see a difference for oneway=-1 ways. I don't know how difficult it is to change the style to produce correct results with current version again, but you have to remember that the way is reversed, so left will be right, forward will be backward. I see two alternatives: 1) mkgmap reverses the way before style processing. This requires also to reverse all tags like bicycle:left, car:forward etc, so it is not trivial. On the other hand, it might allow to merge more roads in RoadMerger. 2) mkgmap makes sure that overlaying lines are NOT reversed. I think this can be done with the attached patch. By the way: There are also tags on nodes which depend on the direction of the way, see http://gis.19327.n5.nabble.com/direction-forward-backward-on-nodes-tt5803072... I think we can ignore this because in the points file you can't detect the direction of the way. Gerd Date: Wed, 16 Apr 2014 23:00:35 +0200 From: osm@aighes.de To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] mkgmap ToDo list Am 16.04.2014 22:01, schrieb GerdP: Why do you think that it is a bug? Think about oneway=-1 and in style you add oneway-arrows (link mapnik does). In style you will handle -1 special, because arrows should drawn in oposite direction of way. If mkgmap now changes direction of the way, arrows should be in wrong direction, or am I wrong? Henning _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
On 16/04/14 07:08, Gerd Petermann wrote:
@Steve: Maybe we should keep this list somewhere at http://www.mkgmap.org.uk/ and try to maintain it?
Thanks, this is a good list. Some of the items may need some explanation for people that are not already very familiar with the code. I've put it at http://www.mkgmap.org.uk/dev/todo. ..Steve
Hi @ all, while thinking about creation of simple maps with less features I thought about an better parameter for splitter then just the number of OSM-nodes. Actually splitter writes density-files (but also based on OSM-nodes). You can imagine that the number of OSM-nodes may differ a lot in detailed mapped areas if you just want to generate a map of highways. Maybe it would be a great feature, that mkgmap can write a density-file (as splitter did) but also considering the used style file. Henning
Hi Henning, I don't understand. What would be the content of that file? What would you do with it? Gerd Henning Scholland wrote
Hi @ all,
while thinking about creation of simple maps with less features I thought about an better parameter for splitter then just the number of OSM-nodes. Actually splitter writes density-files (but also based on OSM-nodes). You can imagine that the number of OSM-nodes may differ a lot in detailed mapped areas if you just want to generate a map of highways. Maybe it would be a great feature, that mkgmap can write a density-file (as splitter did) but also considering the used style file.
Henning
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
-- View this message in context: http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5804174.html Sent from the Mkgmap Development mailing list archive at Nabble.com.
Hi Gerd, splitter is able to write a density file, which stores the density of OSM-nodes in the specific area. Afterwards the file is splitted and processed by mkgmap. mkgmap will use only the objects, which are addressed in style-file. Example: In a rectangle of 100x100m are 5 nodes belonging to a highway and 10 nodes belonging to POI, and polygones and style-file only contains highways, mkgmap writes only 5 nodes (and one line) to img-file. For an more effictive splitting of tiles I would imagine, that a density-file created by mkgmap based on the written data will be much better then the file written by splitter. Henning
Hi Henning, well, I don't know how well the number of nodes in a tile correlates with the size of the img file, but it seems to work for most users. My understanding is that this should work as well for a style which only processes a few details, only the ratio between number of nodes and tile size will be different, in other words, you have to find out how much higher the max-nodes value can be. Gerd
Date: Fri, 25 Apr 2014 19:27:09 +0200 From: osm@aighes.de To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] mkgmap ToDo list
Hi Gerd, splitter is able to write a density file, which stores the density of OSM-nodes in the specific area. Afterwards the file is splitted and processed by mkgmap. mkgmap will use only the objects, which are addressed in style-file.
Example: In a rectangle of 100x100m are 5 nodes belonging to a highway and 10 nodes belonging to POI, and polygones and style-file only contains highways, mkgmap writes only 5 nodes (and one line) to img-file.
For an more effictive splitting of tiles I would imagine, that a density-file created by mkgmap based on the written data will be much better then the file written by splitter.
Henning
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Gerd, finding a proper max-nodes value wont solve the problem globaly. Eg. then you compare the needed value from europe/north america to the value needed in the rest of the world. Henning Am 25.04.2014 19:35, schrieb Gerd Petermann:
Hi Henning,
well, I don't know how well the number of nodes in a tile correlates with the size of the img file, but it seems to work for most users. My understanding is that this should work as well for a style which only processes a few details, only the ratio between number of nodes and tile size will be different, in other words, you have to find out how much higher the max-nodes value can be.
Gerd
Date: Fri, 25 Apr 2014 19:27:09 +0200 From: osm@aighes.de To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] mkgmap ToDo list
Hi Gerd, splitter is able to write a density file, which stores the density of OSM-nodes in the specific area. Afterwards the file is splitted and processed by mkgmap. mkgmap will use only the objects, which are addressed in style-file.
Example: In a rectangle of 100x100m are 5 nodes belonging to a highway and 10 nodes belonging to POI, and polygones and style-file only contains highways, mkgmap writes only 5 nodes (and one line) to img-file.
For an more effictive splitting of tiles I would imagine, that a density-file created by mkgmap based on the written data will be much better then the file written by splitter.
Henning
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Henning, it seems I still don't understand. The max-nodes values influences the number of tiles created. I don't see a big difference between a map for US consisting of 100 or 250 tiles, esp. if the map doesn't contain routing info. I would try to find a value that allows to utilize all CPUs without reaching heap limits or img limits. Why do you think that you have to find an optimal value? Gerd Henning Scholland wrote
Hi Gerd,
finding a proper max-nodes value wont solve the problem globaly. Eg. then you compare the needed value from europe/north america to the value needed in the rest of the world.
Henning
Am 25.04.2014 19:35, schrieb Gerd Petermann:
Hi Henning,
well, I don't know how well the number of nodes in a tile correlates with the size of the img file, but it seems to work for most users. My understanding is that this should work as well for a style which only processes a few details, only the ratio between number of nodes and tile size will be different, in other words, you have to find out how much higher the max-nodes value can be.
Gerd
Date: Fri, 25 Apr 2014 19:27:09 +0200 From:
osm@
To:
mkgmap-dev@.org
Subject: Re: [mkgmap-dev] mkgmap ToDo list
Hi Gerd, splitter is able to write a density file, which stores the density of OSM-nodes in the specific area. Afterwards the file is splitted and processed by mkgmap. mkgmap will use only the objects, which are addressed in style-file.
Example: In a rectangle of 100x100m are 5 nodes belonging to a highway and 10 nodes belonging to POI, and polygones and style-file only contains highways, mkgmap writes only 5 nodes (and one line) to img-file.
For an more effictive splitting of tiles I would imagine, that a density-file created by mkgmap based on the written data will be much better then the file written by splitter.
Henning
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
-- View this message in context: http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5804188.html Sent from the Mkgmap Development mailing list archive at Nabble.com.
Hi Gerd, I would like to have img-tiles which have globally nearly the same filesize, so that they use the space of devices like eTrex 10. With my actual map I use globally the same value for max-nodes. But the size of the img-tiles differ more then factor 2. Eg. a tile in Germany is between 2 and 5 mb where a tile in China is about 10 mb. If I remove details, this difference will increase, because in Germany more objects will be removed from the img-tile then in China. Henning
Hi Henning, okay, got it. You mentioned the density file written by splitter. I have no idea how, but you may find a way to manipulate the values in that file. DensityMap.readMap() contains the code to read that file which simply represents the content of the density map (x and y index and number of nodes in each line) If splitter finds a file density.txt in its working directory, it will read that file instead of the input file(s) to calculate the areas.list. What you need now is a routine that calculates a weight factor for each field in the density map based on the number of bytes in the corresponding img file, which you can use to lower the number of nodes. The problem is: You have only one factor for many elements in the grid of the density map, and I see no way to calculate something more detailed in mkgmap. Does that help you somehow? Gerd Henning Scholland wrote
Hi Gerd,
I would like to have img-tiles which have globally nearly the same filesize, so that they use the space of devices like eTrex 10.
With my actual map I use globally the same value for max-nodes. But the size of the img-tiles differ more then factor 2. Eg. a tile in Germany is between 2 and 5 mb where a tile in China is about 10 mb. If I remove details, this difference will increase, because in Germany more objects will be removed from the img-tile then in China.
Henning
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
-- View this message in context: http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5804194.html Sent from the Mkgmap Development mailing list archive at Nabble.com.
Hi Henning, some more remarks: A factor of 2 is more or less the best value you can expect, and it is likely that the factor is bigger. Reason: splitter tries to create no tiles with a very small number of nodes, it will typically produce a few tiles with nearly 100% of the max-nodes value and other other with less than 50%. The log shows the values: ... Area 63240082 covers (48.8671875,8.701171875) to (49.21875,9.0966796875) and contains 837131 nodes (52 %) Area 63240083 covers (48.8671875,9.0966796875) to (49.04296875,9.4482421875) and contains 607941 nodes (37 %) Area 63240084 covers (49.04296875,9.0966796875) to (49.21875,9.4482421875) and contains 1035670 nodes (64 %) Area 63240085 covers (49.21875,8.4375) to (49.833984375,8.701171875) and contains 1541414 nodes (96 %) ... As long as we create rectangular tiles you should not expect much better results. Gerd
Date: Fri, 25 Apr 2014 21:54:05 +0200 From: osm@aighes.de To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] mkgmap ToDo list
Hi Gerd,
I would like to have img-tiles which have globally nearly the same filesize, so that they use the space of devices like eTrex 10.
With my actual map I use globally the same value for max-nodes. But the size of the img-tiles differ more then factor 2. Eg. a tile in Germany is between 2 and 5 mb where a tile in China is about 10 mb. If I remove details, this difference will increase, because in Germany more objects will be removed from the img-tile then in China.
Henning
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
While this possibly can be solved in Splitter or Mkgmap, it could also be solved by your build-script when you add a maximum tile size check and re-split (with a lower number of max-nodes) until you get two or more sub-tiles. Granted, this adds complexity to the script but it works well for me. On 25/04/2014 21:54, Henning Scholland wrote:
Hi Gerd,
I would like to have img-tiles which have globally nearly the same filesize, so that they use the space of devices like eTrex 10.
With my actual map I use globally the same value for max-nodes. But the size of the img-tiles differ more then factor 2. Eg. a tile in Germany is between 2 and 5 mb where a tile in China is about 10 mb. If I remove details, this difference will increase, because in Germany more objects will be removed from the img-tile then in China.
Henning
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Lambertus, that sounds like a possible change in splitter: Instead of specifying max-nodes you may specify --num-tiles=x and splitter will try to find a split that produces excactly x tiles which are not too narrow and have a node number which is not too far from the average (but still aligned to a multiple of map units as now). So, for your script that means you don't have to find the max-nodes value. I'll think about this again... Gerd
Date: Tue, 29 Apr 2014 14:59:36 +0200 From: osm@na1400.info To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] mkgmap ToDo list
While this possibly can be solved in Splitter or Mkgmap, it could also be solved by your build-script when you add a maximum tile size check and re-split (with a lower number of max-nodes) until you get two or more sub-tiles. Granted, this adds complexity to the script but it works well for me.
On 25/04/2014 21:54, Henning Scholland wrote:
Hi Gerd,
I would like to have img-tiles which have globally nearly the same filesize, so that they use the space of devices like eTrex 10.
With my actual map I use globally the same value for max-nodes. But the size of the img-tiles differ more then factor 2. Eg. a tile in Germany is between 2 and 5 mb where a tile in China is about 10 mb. If I remove details, this difference will increase, because in Germany more objects will be removed from the img-tile then in China.
Henning
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Num-tiles=x would indeed be better for this specific need. It is my experience that it regularly takes multiple calls to Splitter to get 2+ sub-tiles when you reduce the max-nodes by 100k for each sub-split attempt. This is what I currently do to get an optimum in tile-size vs total number of tiles. On 29/04/2014 15:09, Gerd Petermann wrote:
Hi Lambertus,
that sounds like a possible change in splitter: Instead of specifying max-nodes you may specify --num-tiles=x and splitter will try to find a split that produces excactly x tiles which are not too narrow and have a node number which is not too far from the average (but still aligned to a multiple of map units as now). So, for your script that means you don't have to find the max-nodes value.
I'll think about this again...
Gerd
Date: Tue, 29 Apr 2014 14:59:36 +0200 From: osm@na1400.info To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] mkgmap ToDo list
While this possibly can be solved in Splitter or Mkgmap, it could also be solved by your build-script when you add a maximum tile size check and re-split (with a lower number of max-nodes) until you get two or more sub-tiles. Granted, this adds complexity to the script but it works well for me.
On 25/04/2014 21:54, Henning Scholland wrote:
Hi Gerd,
I would like to have img-tiles which have globally nearly the same filesize, so that they use the space of devices like eTrex 10.
With my actual map I use globally the same value for max-nodes. But the size of the img-tiles differ more then factor 2. Eg. a tile in Germany is between 2 and 5 mb where a tile in China is about 10 mb. If I remove details, this difference will increase, because in Germany more objects will be removed from the img-tile then in China.
Henning
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Lambertus, and I guess that even after this optimization you will see a factor 3 or higher between the largest tile and the smallest. Can you confirm that? Gerd
Date: Tue, 29 Apr 2014 15:32:38 +0200 From: osm@na1400.info To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] mkgmap ToDo list
Num-tiles=x would indeed be better for this specific need.
It is my experience that it regularly takes multiple calls to Splitter to get 2+ sub-tiles when you reduce the max-nodes by 100k for each sub-split attempt. This is what I currently do to get an optimum in tile-size vs total number of tiles.
On 29/04/2014 15:09, Gerd Petermann wrote:
Hi Lambertus,
that sounds like a possible change in splitter: Instead of specifying max-nodes you may specify --num-tiles=x and splitter will try to find a split that produces excactly x tiles which are not too narrow and have a node number which is not too far from the average (but still aligned to a multiple of map units as now). So, for your script that means you don't have to find the max-nodes value.
I'll think about this again...
Gerd
Date: Tue, 29 Apr 2014 14:59:36 +0200 From: osm@na1400.info To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] mkgmap ToDo list
While this possibly can be solved in Splitter or Mkgmap, it could also be solved by your build-script when you add a maximum tile size check and re-split (with a lower number of max-nodes) until you get two or more sub-tiles. Granted, this adds complexity to the script but it works well for me.
On 25/04/2014 21:54, Henning Scholland wrote:
Hi Gerd,
I would like to have img-tiles which have globally nearly the same filesize, so that they use the space of devices like eTrex 10.
With my actual map I use globally the same value for max-nodes. But the size of the img-tiles differ more then factor 2. Eg. a tile in Germany is between 2 and 5 mb where a tile in China is about 10 mb. If I remove details, this difference will increase, because in Germany more objects will be removed from the img-tile then in China.
Henning
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Unfortunately I cannot confirm that. Below is a bit of logging from my script: Original: 97000020 (70551453), New: 0 (35684445), New: 1 (36852845) Original: 97000001 (74621042), New: 0 (37522992), New: 1 (37222739) Original: 97000002 (73391358), New: 0 (37679505), New: 1 (38098627) Original: 97000003 (77862567), New: 0 (39075311), New: 1 (39261197) The original files above contain contour data, the filesize is between brackets. As you can see both resulting file are approximately the same size. On 2014-04-29 15:39, Gerd Petermann wrote:
Hi Lambertus,
and I guess that even after this optimization you will see a factor 3 or higher between the largest tile and the smallest. Can you confirm that?
Gerd
Date: Tue, 29 Apr 2014 15:32:38 +0200 From: osm@na1400.info To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] mkgmap ToDo list
Num-tiles=x would indeed be better for this specific need.
It is my experience that it regularly takes multiple calls to Splitter to get 2+ sub-tiles when you reduce the max-nodes by 100k for each sub-split attempt. This is what I currently do to get an optimum in tile-size vs total number of tiles.
On 29/04/2014 15:09, Gerd Petermann wrote:
Hi Lambertus,
that sounds like a possible change in splitter: Instead of specifying max-nodes you may specify --num-tiles=x and splitter will try to find a split that produces excactly x tiles which are not too narrow and have a node number which is not too far from the average (but still aligned to a multiple of map units as now). So, for your script that means you don't have to find the max-nodes value.
I'll think about this again...
Gerd
Date: Tue, 29 Apr 2014 14:59:36 +0200 From: osm@na1400.info To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] mkgmap ToDo list
While this possibly can be solved in Splitter or Mkgmap, it could also be solved by your build-script when you add a maximum tile size check and re-split (with a lower number of max-nodes) until you get two or more sub-tiles. Granted, this adds complexity to the script but it works well for me.
On 25/04/2014 21:54, Henning Scholland wrote:
Hi Gerd,
I would like to have img-tiles which have globally nearly the same filesize, so that they use the space of devices like eTrex 10.
With my actual map I use globally the same value for max-nodes. But the size of the img-tiles differ more then factor 2. Eg. a tile in Germany is between 2 and 5 mb where a tile in China is about 10 mb. If I remove details, this difference will increase, because in Germany more objects will be removed from the img-tile then in China.
Henning
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Lambertus, that's interesting. Are these the img file sizes or the osm file sizes? Gerd Lambertus wrote
Unfortunately I cannot confirm that. Below is a bit of logging from my script: Original: 97000020 (70551453), New: 0 (35684445), New: 1 (36852845) Original: 97000001 (74621042), New: 0 (37522992), New: 1 (37222739) Original: 97000002 (73391358), New: 0 (37679505), New: 1 (38098627) Original: 97000003 (77862567), New: 0 (39075311), New: 1 (39261197)
The original files above contain contour data, the filesize is between brackets. As you can see both resulting file are approximately the same size.
On 2014-04-29 15:39, Gerd Petermann wrote:
Hi Lambertus,
and I guess that even after this optimization you will see a factor 3 or higher between the largest tile and the smallest. Can you confirm that?
Gerd
Date: Tue, 29 Apr 2014 15:32:38 +0200 From:
osm@
To:
mkgmap-dev@.org
Subject: Re: [mkgmap-dev] mkgmap ToDo list
Num-tiles=x would indeed be better for this specific need.
It is my experience that it regularly takes multiple calls to Splitter to get 2+ sub-tiles when you reduce the max-nodes by 100k for each sub-split attempt. This is what I currently do to get an optimum in tile-size vs total number of tiles.
On 29/04/2014 15:09, Gerd Petermann wrote:
Hi Lambertus,
that sounds like a possible change in splitter: Instead of specifying max-nodes you may specify --num-tiles=x and splitter will try to find a split that produces excactly x tiles which are not too narrow and have a node number which is not too far from the average (but still aligned to a multiple of map units as now). So, for your script that means you don't have to find the max-nodes value.
I'll think about this again...
Gerd
Date: Tue, 29 Apr 2014 14:59:36 +0200 From:
osm@
To:
mkgmap-dev@.org
Subject: Re: [mkgmap-dev] mkgmap ToDo list
While this possibly can be solved in Splitter or Mkgmap, it could also be solved by your build-script when you add a maximum tile size check and re-split (with a lower number of max-nodes) until you get two or more sub-tiles. Granted, this adds complexity to the script but it works well for me.
On 25/04/2014 21:54, Henning Scholland wrote:
Hi Gerd,
I would like to have img-tiles which have globally nearly the same filesize, so that they use the space of devices like eTrex 10.
With my actual map I use globally the same value for max-nodes. But the size of the img-tiles differ more then factor 2. Eg. a tile in Germany is between 2 and 5 mb where a tile in China is about 10 mb. If I remove details, this difference will increase, because in Germany more objects will be removed from the img-tile then in China.
Henning
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
mkgmap-dev mailing list
mkgmap-dev@.org
-- View this message in context: http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5804588.html Sent from the Mkgmap Development mailing list archive at Nabble.com.
These are the direct results from Splitter. The format is o5m, both input as output. Splitter version is: r321. For this test I split the original source with --max-nodes=8000000. Then I render the initial tiles, when the result is larger than 8MB it's subsplit again with --max-nodes=(8000000-(attempt*100000)). The initial source files are ~70MB (o5m) and after several subsplits the two *.img are < 8MB. During this process --max-nodes has been reduced to e.g. 7500000 and the source file is split up in two o5m files of about 37MB. I can upload an example source file and it's two subsplit siblings if you want. On 2014-04-29 19:38, GerdP wrote:
Hi Lambertus,
that's interesting. Are these the img file sizes or the osm file sizes?
Gerd
Lambertus wrote
Unfortunately I cannot confirm that. Below is a bit of logging from my script: Original: 97000020 (70551453), New: 0 (35684445), New: 1 (36852845) Original: 97000001 (74621042), New: 0 (37522992), New: 1 (37222739) Original: 97000002 (73391358), New: 0 (37679505), New: 1 (38098627) Original: 97000003 (77862567), New: 0 (39075311), New: 1 (39261197)
The original files above contain contour data, the filesize is between brackets. As you can see both resulting file are approximately the same size.
On 2014-04-29 15:39, Gerd Petermann wrote:
Hi Lambertus,
and I guess that even after this optimization you will see a factor 3 or higher between the largest tile and the smallest. Can you confirm that?
Gerd
Date: Tue, 29 Apr 2014 15:32:38 +0200 From:
osm@
To:
mkgmap-dev@.org
Subject: Re: [mkgmap-dev] mkgmap ToDo list
Num-tiles=x would indeed be better for this specific need.
It is my experience that it regularly takes multiple calls to Splitter to get 2+ sub-tiles when you reduce the max-nodes by 100k for each sub-split attempt. This is what I currently do to get an optimum in tile-size vs total number of tiles.
On 29/04/2014 15:09, Gerd Petermann wrote:
Hi Lambertus,
that sounds like a possible change in splitter: Instead of specifying max-nodes you may specify --num-tiles=x and splitter will try to find a split that produces excactly x tiles which are not too narrow and have a node number which is not too far from the average (but still aligned to a multiple of map units as now). So, for your script that means you don't have to find the max-nodes value.
I'll think about this again...
Gerd
Date: Tue, 29 Apr 2014 14:59:36 +0200 From:
osm@
To:
mkgmap-dev@.org
Subject: Re: [mkgmap-dev] mkgmap ToDo list
While this possibly can be solved in Splitter or Mkgmap, it could also be solved by your build-script when you add a maximum tile size check and re-split (with a lower number of max-nodes) until you get two or more sub-tiles. Granted, this adds complexity to the script but it works well for me.
On 25/04/2014 21:54, Henning Scholland wrote: > Hi Gerd, > > I would like to have img-tiles which have globally nearly the same > filesize, so that they use the space of devices like eTrex 10. > > With my actual map I use globally the same value for max-nodes. But the > size of the img-tiles differ more then factor 2. Eg. a tile in Germany > is between 2 and 5 mb where a tile in China is about 10 mb. If I remove > details, this difference will increase, because in Germany more objects > will be removed from the img-tile then in China. > > Henning > > _______________________________________________ > mkgmap-dev mailing list >
mkgmap-dev@.org
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
mkgmap-dev mailing list
mkgmap-dev@.org
-- View this message in context: http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5804588.html Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Lambertus, okay, if I got that right you finally get *.img files with a size near (but below) 8MB, so maybe Henning can use that script, too. If you do that for e.g. Germany, how small is tpically the smallest *.img file ? Is it probably near 4 MB? Gerd
To: mkgmap-dev@lists.mkgmap.org.uk Date: Tue, 29 Apr 2014 20:30:27 +0200 From: osm@na1400.info Subject: Re: [mkgmap-dev] mkgmap ToDo list
These are the direct results from Splitter. The format is o5m, both input as output. Splitter version is: r321.
For this test I split the original source with --max-nodes=8000000. Then I render the initial tiles, when the result is larger than 8MB it's subsplit again with --max-nodes=(8000000-(attempt*100000)). The initial source files are ~70MB (o5m) and after several subsplits the two *.img are < 8MB. During this process --max-nodes has been reduced to e.g. 7500000 and the source file is split up in two o5m files of about 37MB.
I can upload an example source file and it's two subsplit siblings if you want.
On 2014-04-29 19:38, GerdP wrote:
Hi Lambertus,
that's interesting. Are these the img file sizes or the osm file sizes?
Gerd
Lambertus wrote
Unfortunately I cannot confirm that. Below is a bit of logging from my script: Original: 97000020 (70551453), New: 0 (35684445), New: 1 (36852845) Original: 97000001 (74621042), New: 0 (37522992), New: 1 (37222739) Original: 97000002 (73391358), New: 0 (37679505), New: 1 (38098627) Original: 97000003 (77862567), New: 0 (39075311), New: 1 (39261197)
The original files above contain contour data, the filesize is between brackets. As you can see both resulting file are approximately the same size.
On 2014-04-29 15:39, Gerd Petermann wrote:
Hi Lambertus,
and I guess that even after this optimization you will see a factor 3 or higher between the largest tile and the smallest. Can you confirm that?
Gerd
Date: Tue, 29 Apr 2014 15:32:38 +0200 From:
osm@
To:
mkgmap-dev@.org
Subject: Re: [mkgmap-dev] mkgmap ToDo list
Num-tiles=x would indeed be better for this specific need.
It is my experience that it regularly takes multiple calls to Splitter to get 2+ sub-tiles when you reduce the max-nodes by 100k for each sub-split attempt. This is what I currently do to get an optimum in tile-size vs total number of tiles.
On 29/04/2014 15:09, Gerd Petermann wrote:
Hi Lambertus,
that sounds like a possible change in splitter: Instead of specifying max-nodes you may specify --num-tiles=x and splitter will try to find a split that produces excactly x tiles which are not too narrow and have a node number which is not too far from the average (but still aligned to a multiple of map units as now). So, for your script that means you don't have to find the max-nodes value.
I'll think about this again...
Gerd
> Date: Tue, 29 Apr 2014 14:59:36 +0200 > From:
osm@
> To:
mkgmap-dev@.org
> Subject: Re: [mkgmap-dev] mkgmap ToDo list > > While this possibly can be solved in Splitter or Mkgmap, it could also > be solved by your build-script when you add a maximum tile size check > and re-split (with a lower number of max-nodes) until you get two or > more sub-tiles. Granted, this adds complexity to the script but it works > well for me. > > On 25/04/2014 21:54, Henning Scholland wrote: > > Hi Gerd, > > > > I would like to have img-tiles which have globally nearly the same > > filesize, so that they use the space of devices like eTrex 10. > > > > With my actual map I use globally the same value for max-nodes. But the > > size of the img-tiles differ more then factor 2. Eg. a tile in Germany > > is between 2 and 5 mb where a tile in China is about 10 mb. If I remove > > details, this difference will increase, because in Germany more objects > > will be removed from the img-tile then in China. > > > > Henning > > > > _______________________________________________ > > mkgmap-dev mailing list > >
mkgmap-dev@.org
> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > _______________________________________________ > mkgmap-dev mailing list >
mkgmap-dev@.org
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
mkgmap-dev mailing list
mkgmap-dev@.org
-- View this message in context: http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5804588.html Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
I don't have the number for Germany, but perhaps the world suffices? The image size distribution of my generic map is: <2MB some 2-3MB 260 3-4MB 410 4-5MB 600 5-6MB 580 6-7MB 390 7-8MB 250 On 2014-04-29 20:37, Gerd Petermann wrote:
Hi Lambertus,
okay, if I got that right you finally get *.img files with a size near (but below) 8MB, so maybe Henning can use that script, too.
If you do that for e.g. Germany, how small is tpically the smallest *.img file ? Is it probably near 4 MB?
Gerd
To: mkgmap-dev@lists.mkgmap.org.uk Date: Tue, 29 Apr 2014 20:30:27 +0200 From: osm@na1400.info Subject: Re: [mkgmap-dev] mkgmap ToDo list
These are the direct results from Splitter. The format is o5m, both input as output. Splitter version is: r321.
For this test I split the original source with --max-nodes=8000000. Then I render the initial tiles, when the result is larger than 8MB it's subsplit again with --max-nodes=(8000000-(attempt*100000)). The initial source files are ~70MB (o5m) and after several subsplits the two *.img are < 8MB. During this process --max-nodes has been reduced to e.g. 7500000 and the source file is split up in two o5m files of about 37MB.
I can upload an example source file and it's two subsplit siblings if you want.
On 2014-04-29 19:38, GerdP wrote:
Hi Lambertus,
that's interesting. Are these the img file sizes or the osm file sizes?
Gerd
Lambertus wrote
Unfortunately I cannot confirm that. Below is a bit of logging from my script: Original: 97000020 (70551453), New: 0 (35684445), New: 1 (36852845) Original: 97000001 (74621042), New: 0 (37522992), New: 1 (37222739) Original: 97000002 (73391358), New: 0 (37679505), New: 1 (38098627) Original: 97000003 (77862567), New: 0 (39075311), New: 1 (39261197)
The original files above contain contour data, the filesize is between brackets. As you can see both resulting file are approximately the same size.
On 2014-04-29 15:39, Gerd Petermann wrote:
Hi Lambertus,
and I guess that even after this optimization you will see a factor 3 or higher between the largest tile and the smallest. Can you confirm that?
Gerd
Date: Tue, 29 Apr 2014 15:32:38 +0200 From:
osm@
To:
mkgmap-dev@.org
Subject: Re: [mkgmap-dev] mkgmap ToDo list
Num-tiles=x would indeed be better for this specific need.
It is my experience that it regularly takes multiple calls to Splitter to get 2+ sub-tiles when you reduce the max-nodes by 100k for each sub-split attempt. This is what I currently do to get an optimum in tile-size vs total number of tiles.
On 29/04/2014 15:09, Gerd Petermann wrote: > Hi Lambertus, > > that sounds like a possible change in splitter: > Instead of specifying max-nodes you may specify --num-tiles=x > and splitter will try to find a split that produces excactly x tiles > which are not too narrow and have a node number which is not > too far from the average (but still aligned to a multiple of map units > as now). > So, for your script that means you don't have to find the max-nodes > value. > > I'll think about this again... > > Gerd > > > Date: Tue, 29 Apr 2014 14:59:36 +0200 > > From:
osm@
> > To:
mkgmap-dev@.org
> > Subject: Re: [mkgmap-dev] mkgmap ToDo list > > > > While this possibly can be solved in Splitter or Mkgmap, it could also > > be solved by your build-script when you add a maximum tile size check > > and re-split (with a lower number of max-nodes) until you get two or > > more sub-tiles. Granted, this adds complexity to the script but it works > > well for me. > > > > On 25/04/2014 21:54, Henning Scholland wrote: > > > Hi Gerd, > > > > > > I would like to have img-tiles which have globally nearly the same > > > filesize, so that they use the space of devices like eTrex 10. > > > > > > With my actual map I use globally the same value for max-nodes. But the > > > size of the img-tiles differ more then factor 2. Eg. a tile in Germany > > > is between 2 and 5 mb where a tile in China is about 10 mb. If I remove > > > details, this difference will increase, because in Germany more objects > > > will be removed from the img-tile then in China. > > > > > > Henning > > > > > > _______________________________________________ > > > mkgmap-dev mailing list > > >
mkgmap-dev@.org
> > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > > _______________________________________________ > > mkgmap-dev mailing list > >
mkgmap-dev@.org
> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > _______________________________________________ > mkgmap-dev mailing list >
mkgmap-dev@.org
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev >
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
mkgmap-dev mailing list
mkgmap-dev@.org
-- View this message in context:
http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5804588.html
Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Lambertus, okay, so that's what I meant reg. the factor 3 between largest and smallest part. Gerd
To: mkgmap-dev@lists.mkgmap.org.uk Date: Tue, 29 Apr 2014 20:48:24 +0200 From: osm@na1400.info Subject: Re: [mkgmap-dev] mkgmap ToDo list
I don't have the number for Germany, but perhaps the world suffices?
The image size distribution of my generic map is: <2MB some 2-3MB 260 3-4MB 410 4-5MB 600 5-6MB 580 6-7MB 390 7-8MB 250
On 2014-04-29 20:37, Gerd Petermann wrote:
Hi Lambertus,
okay, if I got that right you finally get *.img files with a size near (but below) 8MB, so maybe Henning can use that script, too.
If you do that for e.g. Germany, how small is tpically the smallest *.img file ? Is it probably near 4 MB?
Gerd
To: mkgmap-dev@lists.mkgmap.org.uk Date: Tue, 29 Apr 2014 20:30:27 +0200 From: osm@na1400.info Subject: Re: [mkgmap-dev] mkgmap ToDo list
These are the direct results from Splitter. The format is o5m, both input as output. Splitter version is: r321.
For this test I split the original source with --max-nodes=8000000. Then I render the initial tiles, when the result is larger than 8MB it's subsplit again with --max-nodes=(8000000-(attempt*100000)). The initial source files are ~70MB (o5m) and after several subsplits the two *.img are < 8MB. During this process --max-nodes has been reduced to e.g. 7500000 and the source file is split up in two o5m files of about 37MB.
I can upload an example source file and it's two subsplit siblings if you want.
On 2014-04-29 19:38, GerdP wrote:
Hi Lambertus,
that's interesting. Are these the img file sizes or the osm file sizes?
Gerd
Lambertus wrote
Unfortunately I cannot confirm that. Below is a bit of logging from my script: Original: 97000020 (70551453), New: 0 (35684445), New: 1 (36852845) Original: 97000001 (74621042), New: 0 (37522992), New: 1 (37222739) Original: 97000002 (73391358), New: 0 (37679505), New: 1 (38098627) Original: 97000003 (77862567), New: 0 (39075311), New: 1 (39261197)
The original files above contain contour data, the filesize is between brackets. As you can see both resulting file are approximately the same size.
On 2014-04-29 15:39, Gerd Petermann wrote:
Hi Lambertus,
and I guess that even after this optimization you will see a factor 3 or higher between the largest tile and the smallest. Can you confirm that?
Gerd
> Date: Tue, 29 Apr 2014 15:32:38 +0200 > From:
osm@
> To:
mkgmap-dev@.org
> Subject: Re: [mkgmap-dev] mkgmap ToDo list > > Num-tiles=x would indeed be better for this specific need. > > It is my experience that it regularly takes multiple calls to Splitter > to get 2+ sub-tiles when you reduce the max-nodes by 100k for each > sub-split attempt. This is what I currently do to get an optimum in > tile-size vs total number of tiles. > > > > On 29/04/2014 15:09, Gerd Petermann wrote: > > Hi Lambertus, > > > > that sounds like a possible change in splitter: > > Instead of specifying max-nodes you may specify --num-tiles=x > > and splitter will try to find a split that produces excactly x tiles > > which are not too narrow and have a node number which is not > > too far from the average (but still aligned to a multiple of map units > > as now). > > So, for your script that means you don't have to find the max-nodes > > value. > > > > I'll think about this again... > > > > Gerd > > > > > Date: Tue, 29 Apr 2014 14:59:36 +0200 > > > From:
osm@
> > > To:
mkgmap-dev@.org
> > > Subject: Re: [mkgmap-dev] mkgmap ToDo list > > > > > > While this possibly can be solved in Splitter or Mkgmap, it could also > > > be solved by your build-script when you add a maximum tile size check > > > and re-split (with a lower number of max-nodes) until you get two or > > > more sub-tiles. Granted, this adds complexity to the script but it works > > > well for me. > > > > > > On 25/04/2014 21:54, Henning Scholland wrote: > > > > Hi Gerd, > > > > > > > > I would like to have img-tiles which have globally nearly the same > > > > filesize, so that they use the space of devices like eTrex 10. > > > > > > > > With my actual map I use globally the same value for max-nodes. But the > > > > size of the img-tiles differ more then factor 2. Eg. a tile in Germany > > > > is between 2 and 5 mb where a tile in China is about 10 mb. If I remove > > > > details, this difference will increase, because in Germany more objects > > > > will be removed from the img-tile then in China. > > > > > > > > Henning > > > > > > > > _______________________________________________ > > > > mkgmap-dev mailing list > > > >
mkgmap-dev@.org
> > > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > > > > _______________________________________________ > > > mkgmap-dev mailing list > > >
mkgmap-dev@.org
> > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > > > > _______________________________________________ > > mkgmap-dev mailing list > >
mkgmap-dev@.org
> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > > _______________________________________________ > mkgmap-dev mailing list >
mkgmap-dev@.org
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
mkgmap-dev mailing list
mkgmap-dev@.org
-- View this message in context:
http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5804588.html
Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Great that I could finally give you what you were looking for. Sorry for not understanding earlier. It would still be awesome to have a --min-files option in Splitter. Btw, now that I'm counting, and for comparison, below the distribution of my new generic map with a much larger max-file-size and --max-nodess=1200000: 1-2MB 37 2-3MB 210 3-4MB 330 4-5MB 350 5-6MB 324 6-7MB 260 7-8MB 200 8-9MB 160 9-10MB 100 10-11MB 70 11-12MB 40 12-13MB 40 13-14MB 20 14-15MB 10 15-16MB 10 16-17MB 2 17-18MB 3 18-19MB 2 19-20MB 3 20-21MB 2 And a Europe contour map with --max-nodes=8000000 without size restriction: 7-8MB 3 8-9MB 4 9-10MB 7 10-11MB 3 11-12MB 3 12-13MB 11 13-14MB 22 14-15MB 14 15-16MB 10 16-17MB 1 On 2014-04-29 20:51, Gerd Petermann wrote:
Hi Lambertus,
okay, so that's what I meant reg. the factor 3 between largest and smallest part.
Gerd
To: mkgmap-dev@lists.mkgmap.org.uk Date: Tue, 29 Apr 2014 20:48:24 +0200 From: osm@na1400.info Subject: Re: [mkgmap-dev] mkgmap ToDo list
I don't have the number for Germany, but perhaps the world suffices?
The image size distribution of my generic map is: <2MB some 2-3MB 260 3-4MB 410 4-5MB 600 5-6MB 580 6-7MB 390 7-8MB 250
On 2014-04-29 20:37, Gerd Petermann wrote:
Hi Lambertus,
okay, if I got that right you finally get *.img files with a size near (but below) 8MB, so maybe Henning can use that script, too.
If you do that for e.g. Germany, how small is tpically the smallest *.img file ? Is it probably near 4 MB?
Gerd
To: mkgmap-dev@lists.mkgmap.org.uk Date: Tue, 29 Apr 2014 20:30:27 +0200 From: osm@na1400.info Subject: Re: [mkgmap-dev] mkgmap ToDo list
These are the direct results from Splitter. The format is o5m, both input as output. Splitter version is: r321.
For this test I split the original source with --max-nodes=8000000. Then I render the initial tiles, when the result is larger than 8MB it's subsplit again with --max-nodes=(8000000-(attempt*100000)). The initial source files are ~70MB (o5m) and after several subsplits the two *.img are < 8MB. During this process --max-nodes has been reduced to e.g. 7500000 and the source file is split up in two o5m files of about 37MB.
I can upload an example source file and it's two subsplit siblings if you want.
On 2014-04-29 19:38, GerdP wrote:
Hi Lambertus,
that's interesting. Are these the img file sizes or the osm file sizes?
Gerd
Lambertus wrote
Unfortunately I cannot confirm that. Below is a bit of logging from my script: Original: 97000020 (70551453), New: 0 (35684445), New: 1 (36852845) Original: 97000001 (74621042), New: 0 (37522992), New: 1 (37222739) Original: 97000002 (73391358), New: 0 (37679505), New: 1 (38098627) Original: 97000003 (77862567), New: 0 (39075311), New: 1 (39261197)
The original files above contain contour data, the filesize is between brackets. As you can see both resulting file are approximately the same size.
On 2014-04-29 15:39, Gerd Petermann wrote: > Hi Lambertus, > > and I guess that even after this optimization you will > see a factor 3 or higher between the largest tile and the smallest. > Can you confirm that? > > Gerd > >> Date: Tue, 29 Apr 2014 15:32:38 +0200 >> From:
osm@
>> To:
mkgmap-dev@.org
>> Subject: Re: [mkgmap-dev] mkgmap ToDo list >> >> Num-tiles=x would indeed be better for this specific need. >> >> It is my experience that it regularly takes multiple calls to > Splitter >> to get 2+ sub-tiles when you reduce the max-nodes by 100k for each >> sub-split attempt. This is what I currently do to get an optimum in >> tile-size vs total number of tiles. >> >> >> >> On 29/04/2014 15:09, Gerd Petermann wrote: >> > Hi Lambertus, >> > >> > that sounds like a possible change in splitter: >> > Instead of specifying max-nodes you may specify --num-tiles=x >> > and splitter will try to find a split that produces excactly x > tiles >> > which are not too narrow and have a node number which is not >> > too far from the average (but still aligned to a multiple of map > units >> > as now). >> > So, for your script that means you don't have to find the > max-nodes >> > value. >> > >> > I'll think about this again... >> > >> > Gerd >> > >> > > Date: Tue, 29 Apr 2014 14:59:36 +0200 >> > > From:
osm@
>> > > To:
mkgmap-dev@.org
>> > > Subject: Re: [mkgmap-dev] mkgmap ToDo list >> > > >> > > While this possibly can be solved in Splitter or Mkgmap, it > could also >> > > be solved by your build-script when you add a maximum tile size > check >> > > and re-split (with a lower number of max-nodes) until you get > two or >> > > more sub-tiles. Granted, this adds complexity to the script but > it works >> > > well for me. >> > > >> > > On 25/04/2014 21:54, Henning Scholland wrote: >> > > > Hi Gerd, >> > > > >> > > > I would like to have img-tiles which have globally nearly the > same >> > > > filesize, so that they use the space of devices like eTrex 10. >> > > > >> > > > With my actual map I use globally the same value for > max-nodes. But the >> > > > size of the img-tiles differ more then factor 2. Eg. a tile in > Germany >> > > > is between 2 and 5 mb where a tile in China is about 10 mb. If > I remove >> > > > details, this difference will increase, because in Germany > more objects >> > > > will be removed from the img-tile then in China. >> > > > >> > > > Henning >> > > > >> > > > _______________________________________________ >> > > > mkgmap-dev mailing list >> > > >
mkgmap-dev@.org
>> > > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev >> > > >> > > _______________________________________________ >> > > mkgmap-dev mailing list >> > >
mkgmap-dev@.org
>> > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev >> > >> > >> > _______________________________________________ >> > mkgmap-dev mailing list >> >
mkgmap-dev@.org
>> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev >> > >> >> _______________________________________________ >> mkgmap-dev mailing list >>
mkgmap-dev@.org
>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > _______________________________________________ > mkgmap-dev mailing list >
mkgmap-dev@.org
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
-- View this message in context:
http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5804588.html
Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Lambertus, are these numbers the result of your script or the basic results of splitter? Henning Am 29.04.2014 20:48, schrieb osm@na1400.info:
I don't have the number for Germany, but perhaps the world suffices?
The image size distribution of my generic map is: <2MB some 2-3MB 260 3-4MB 410 4-5MB 600 5-6MB 580 6-7MB 390 7-8MB 250
Both actually. You'll notice that there is no image larger then 8MB, this is because of my script. The distribution below 8MB is purely the result of Splitter' behavior. On 2014-04-29 22:17, Henning Scholland wrote:
Hi Lambertus,
are these numbers the result of your script or the basic results of splitter?
Henning
Am 29.04.2014 20:48, schrieb osm@na1400.info:
I don't have the number for Germany, but perhaps the world suffices?
The image size distribution of my generic map is: <2MB some 2-3MB 260 3-4MB 410 4-5MB 600 5-6MB 580 6-7MB 390 7-8MB 250
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Oh, and ofcourse anyone interested can get my scripts, send an email. They'll be on Github someday anyway. On 2014-04-29 20:37, Gerd Petermann wrote:
Hi Lambertus,
okay, if I got that right you finally get *.img files with a size near (but below) 8MB, so maybe Henning can use that script, too.
If you do that for e.g. Germany, how small is tpically the smallest *.img file ? Is it probably near 4 MB?
Gerd
To: mkgmap-dev@lists.mkgmap.org.uk Date: Tue, 29 Apr 2014 20:30:27 +0200 From: osm@na1400.info Subject: Re: [mkgmap-dev] mkgmap ToDo list
These are the direct results from Splitter. The format is o5m, both input as output. Splitter version is: r321.
For this test I split the original source with --max-nodes=8000000. Then I render the initial tiles, when the result is larger than 8MB it's subsplit again with --max-nodes=(8000000-(attempt*100000)). The initial source files are ~70MB (o5m) and after several subsplits the two *.img are < 8MB. During this process --max-nodes has been reduced to e.g. 7500000 and the source file is split up in two o5m files of about 37MB.
I can upload an example source file and it's two subsplit siblings if you want.
On 2014-04-29 19:38, GerdP wrote:
Hi Lambertus,
that's interesting. Are these the img file sizes or the osm file sizes?
Gerd
Lambertus wrote
Unfortunately I cannot confirm that. Below is a bit of logging from my script: Original: 97000020 (70551453), New: 0 (35684445), New: 1 (36852845) Original: 97000001 (74621042), New: 0 (37522992), New: 1 (37222739) Original: 97000002 (73391358), New: 0 (37679505), New: 1 (38098627) Original: 97000003 (77862567), New: 0 (39075311), New: 1 (39261197)
The original files above contain contour data, the filesize is between brackets. As you can see both resulting file are approximately the same size.
On 2014-04-29 15:39, Gerd Petermann wrote:
Hi Lambertus,
and I guess that even after this optimization you will see a factor 3 or higher between the largest tile and the smallest. Can you confirm that?
Gerd
Date: Tue, 29 Apr 2014 15:32:38 +0200 From:
osm@
To:
mkgmap-dev@.org
Subject: Re: [mkgmap-dev] mkgmap ToDo list
Num-tiles=x would indeed be better for this specific need.
It is my experience that it regularly takes multiple calls to Splitter to get 2+ sub-tiles when you reduce the max-nodes by 100k for each sub-split attempt. This is what I currently do to get an optimum in tile-size vs total number of tiles.
On 29/04/2014 15:09, Gerd Petermann wrote: > Hi Lambertus, > > that sounds like a possible change in splitter: > Instead of specifying max-nodes you may specify --num-tiles=x > and splitter will try to find a split that produces excactly x tiles > which are not too narrow and have a node number which is not > too far from the average (but still aligned to a multiple of map units > as now). > So, for your script that means you don't have to find the max-nodes > value. > > I'll think about this again... > > Gerd > > > Date: Tue, 29 Apr 2014 14:59:36 +0200 > > From:
osm@
> > To:
mkgmap-dev@.org
> > Subject: Re: [mkgmap-dev] mkgmap ToDo list > > > > While this possibly can be solved in Splitter or Mkgmap, it could also > > be solved by your build-script when you add a maximum tile size check > > and re-split (with a lower number of max-nodes) until you get two or > > more sub-tiles. Granted, this adds complexity to the script but it works > > well for me. > > > > On 25/04/2014 21:54, Henning Scholland wrote: > > > Hi Gerd, > > > > > > I would like to have img-tiles which have globally nearly the same > > > filesize, so that they use the space of devices like eTrex 10. > > > > > > With my actual map I use globally the same value for max-nodes. But the > > > size of the img-tiles differ more then factor 2. Eg. a tile in Germany > > > is between 2 and 5 mb where a tile in China is about 10 mb. If I remove > > > details, this difference will increase, because in Germany more objects > > > will be removed from the img-tile then in China. > > > > > > Henning > > > > > > _______________________________________________ > > > mkgmap-dev mailing list > > >
mkgmap-dev@.org
> > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > > _______________________________________________ > > mkgmap-dev mailing list > >
mkgmap-dev@.org
> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > _______________________________________________ > mkgmap-dev mailing list >
mkgmap-dev@.org
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev >
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
mkgmap-dev mailing list
mkgmap-dev@.org
-- View this message in context:
http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5804588.html
Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
I would love if there was a possibility that you pass the used max-nodes value to mkgmap. When mkgmap is compiling the maps, then after the .img is created it should check a) did it crash due to too many max-nodes b) for me not important - but for others with very old GPS, etrex 10, ---> is tile bigger than X (usually 8) MB. if a) or b) true, then pass the file back to splitter and split with 60% of maxnodes - and compile the resulting .img files again. Should it fail again, use 40%, again 25%... Sometimes there are awful tiles, that need supersmall max-nodes till they compile, however lately (last 1-2 years) I never encountered them anymore. I think that happened rather due to a but in splitter/mgkmap that is fixed now. okay, you could also do this with a script, but it gets rather complicated to multithread it (you need to wait till mgkmap finished compiling all .img files - and run mkgmap first without address index to save time) and do some clever routines on making sure that the map id (e.g. 6340????.img) stay correct. Even more complicated to have consequent map id... For europe with a fixed max-node I get tiles from 1.9MB up to 18MB. That's factor 9 - so it's huge... If I could narrows that down easily to 8-18MB - without getting tiles crashing due to too high max-nodes values, that would be sweet. As for the scripts - would they run on windows too? - What programming language are they in? On 29.04.2014 21:39, osm@na1400.info wrote:
Oh, and ofcourse anyone interested can get my scripts, send an email. They'll be on Github someday anyway.
On 2014-04-29 20:37, Gerd Petermann wrote:
Hi Lambertus,
okay, if I got that right you finally get *.img files with a size near (but below) 8MB, so maybe Henning can use that script, too.
If you do that for e.g. Germany, how small is tpically the smallest *.img file ? Is it probably near 4 MB?
Gerd
To: mkgmap-dev@lists.mkgmap.org.uk Date: Tue, 29 Apr 2014 20:30:27 +0200 From: osm@na1400.info Subject: Re: [mkgmap-dev] mkgmap ToDo list
These are the direct results from Splitter. The format is o5m, both input as output. Splitter version is: r321.
For this test I split the original source with --max-nodes=8000000. Then I render the initial tiles, when the result is larger than 8MB it's subsplit again with --max-nodes=(8000000-(attempt*100000)). The initial source files are ~70MB (o5m) and after several subsplits the two *.img are < 8MB. During this process --max-nodes has been reduced to e.g. 7500000 and the source file is split up in two o5m files of about 37MB.
I can upload an example source file and it's two subsplit siblings if you want.
On 2014-04-29 19:38, GerdP wrote:
Hi Lambertus,
that's interesting. Are these the img file sizes or the osm file sizes?
Gerd
Lambertus wrote
Unfortunately I cannot confirm that. Below is a bit of logging from my script: Original: 97000020 (70551453), New: 0 (35684445), New: 1 (36852845) Original: 97000001 (74621042), New: 0 (37522992), New: 1 (37222739) Original: 97000002 (73391358), New: 0 (37679505), New: 1 (38098627) Original: 97000003 (77862567), New: 0 (39075311), New: 1 (39261197)
The original files above contain contour data, the filesize is between brackets. As you can see both resulting file are approximately the same size.
On 2014-04-29 15:39, Gerd Petermann wrote:
Hi Lambertus,
and I guess that even after this optimization you will see a factor 3 or higher between the largest tile and the smallest. Can you confirm that?
Gerd
> Date: Tue, 29 Apr 2014 15:32:38 +0200 > From:
osm@
> To:
mkgmap-dev@.org
> Subject: Re: [mkgmap-dev] mkgmap ToDo list > > Num-tiles=x would indeed be better for this specific need. > > It is my experience that it regularly takes multiple calls to Splitter > to get 2+ sub-tiles when you reduce the max-nodes by 100k for each > sub-split attempt. This is what I currently do to get an optimum in > tile-size vs total number of tiles. > > > > On 29/04/2014 15:09, Gerd Petermann wrote: > > Hi Lambertus, > > > > that sounds like a possible change in splitter: > > Instead of specifying max-nodes you may specify --num-tiles=x > > and splitter will try to find a split that produces excactly x tiles > > which are not too narrow and have a node number which is not > > too far from the average (but still aligned to a multiple of map units > > as now). > > So, for your script that means you don't have to find the max-nodes > > value. > > > > I'll think about this again... > > > > Gerd > > > > > Date: Tue, 29 Apr 2014 14:59:36 +0200 > > > From:
osm@
> > > To:
mkgmap-dev@.org
> > > Subject: Re: [mkgmap-dev] mkgmap ToDo list > > > > > > While this possibly can be solved in Splitter or Mkgmap, it could also > > > be solved by your build-script when you add a maximum tile size check > > > and re-split (with a lower number of max-nodes) until you get two or > > > more sub-tiles. Granted, this adds complexity to the script but it works > > > well for me. > > > > > > On 25/04/2014 21:54, Henning Scholland wrote: > > > > Hi Gerd, > > > > > > > > I would like to have img-tiles which have globally nearly the same > > > > filesize, so that they use the space of devices like eTrex 10. > > > > > > > > With my actual map I use globally the same value for max-nodes. But the > > > > size of the img-tiles differ more then factor 2. Eg. a tile in Germany > > > > is between 2 and 5 mb where a tile in China is about 10 mb. If I remove > > > > details, this difference will increase, because in Germany more objects > > > > will be removed from the img-tile then in China. > > > > > > > > Henning > > > > > > > > _______________________________________________ > > > > mkgmap-dev mailing list > > > >
mkgmap-dev@.org
> > > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > > > > _______________________________________________ > > > mkgmap-dev mailing list > > >
mkgmap-dev@.org
> > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > > > > _______________________________________________ > > mkgmap-dev mailing list > >
mkgmap-dev@.org
> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > > _______________________________________________ > mkgmap-dev mailing list >
mkgmap-dev@.org
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
mkgmap-dev mailing list
mkgmap-dev@.org
-- View this message in context:
http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5804588.html
Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- keep on biking and discovering new trails Felix openmtbmap.org & www.velomap.org
Multithreading the tile rendering for a single map is indeed a bit difficult and I gave it up because you need to keep track which image id's are already in use. Since I provide multiple maps the work-around is running a few scripts parallel, which is also a crude form of multithreading. The script language is PHP and it doesn't run on Windows without some changes ('/' vs '\' in paths, 'rm -rf', that sort of stuff). Never tried it. To get a better optimum in file size, using the process I described earlier, you could start off with a huge --max-nodes setting and then 'search' for the highest --max-nodes that works for each specific area. On 30/04/2014 11:49, Felix Hartmann wrote:
I would love if there was a possibility that you pass the used max-nodes value to mkgmap. When mkgmap is compiling the maps, then after the .img is created it should check a) did it crash due to too many max-nodes b) for me not important - but for others with very old GPS, etrex 10, ---> is tile bigger than X (usually 8) MB.
if a) or b) true, then pass the file back to splitter and split with 60% of maxnodes - and compile the resulting .img files again. Should it fail again, use 40%, again 25%... Sometimes there are awful tiles, that need supersmall max-nodes till they compile, however lately (last 1-2 years) I never encountered them anymore. I think that happened rather due to a but in splitter/mgkmap that is fixed now.
okay, you could also do this with a script, but it gets rather complicated to multithread it (you need to wait till mgkmap finished compiling all .img files - and run mkgmap first without address index to save time) and do some clever routines on making sure that the map id (e.g. 6340????.img) stay correct. Even more complicated to have consequent map id...
For europe with a fixed max-node I get tiles from 1.9MB up to 18MB. That's factor 9 - so it's huge... If I could narrows that down easily to 8-18MB - without getting tiles crashing due to too high max-nodes values, that would be sweet.
As for the scripts - would they run on windows too? - What programming language are they in?
On 29.04.2014 21:39, osm@na1400.info wrote:
Oh, and ofcourse anyone interested can get my scripts, send an email. They'll be on Github someday anyway.
On 2014-04-29 20:37, Gerd Petermann wrote:
Hi Lambertus,
okay, if I got that right you finally get *.img files with a size near (but below) 8MB, so maybe Henning can use that script, too.
If you do that for e.g. Germany, how small is tpically the smallest *.img file ? Is it probably near 4 MB?
Gerd
To: mkgmap-dev@lists.mkgmap.org.uk Date: Tue, 29 Apr 2014 20:30:27 +0200 From: osm@na1400.info Subject: Re: [mkgmap-dev] mkgmap ToDo list
These are the direct results from Splitter. The format is o5m, both input as output. Splitter version is: r321.
For this test I split the original source with --max-nodes=8000000. Then I render the initial tiles, when the result is larger than 8MB it's subsplit again with --max-nodes=(8000000-(attempt*100000)). The initial source files are ~70MB (o5m) and after several subsplits the two *.img are < 8MB. During this process --max-nodes has been reduced to e.g. 7500000 and the source file is split up in two o5m files of about 37MB.
I can upload an example source file and it's two subsplit siblings if you want.
On 2014-04-29 19:38, GerdP wrote:
Hi Lambertus,
that's interesting. Are these the img file sizes or the osm file sizes?
Gerd
Lambertus wrote
Unfortunately I cannot confirm that. Below is a bit of logging from my script: Original: 97000020 (70551453), New: 0 (35684445), New: 1 (36852845) Original: 97000001 (74621042), New: 0 (37522992), New: 1 (37222739) Original: 97000002 (73391358), New: 0 (37679505), New: 1 (38098627) Original: 97000003 (77862567), New: 0 (39075311), New: 1 (39261197)
The original files above contain contour data, the filesize is between brackets. As you can see both resulting file are approximately the same size.
On 2014-04-29 15:39, Gerd Petermann wrote: > Hi Lambertus, > > and I guess that even after this optimization you will > see a factor 3 or higher between the largest tile and the smallest. > Can you confirm that? > > Gerd > >> Date: Tue, 29 Apr 2014 15:32:38 +0200 >> From:
osm@
>> To:
mkgmap-dev@.org
>> Subject: Re: [mkgmap-dev] mkgmap ToDo list >> >> Num-tiles=x would indeed be better for this specific need. >> >> It is my experience that it regularly takes multiple calls to > Splitter >> to get 2+ sub-tiles when you reduce the max-nodes by 100k for each >> sub-split attempt. This is what I currently do to get an optimum in >> tile-size vs total number of tiles. >> >> >> >> On 29/04/2014 15:09, Gerd Petermann wrote: >> > Hi Lambertus, >> > >> > that sounds like a possible change in splitter: >> > Instead of specifying max-nodes you may specify --num-tiles=x >> > and splitter will try to find a split that produces excactly x > tiles >> > which are not too narrow and have a node number which is not >> > too far from the average (but still aligned to a multiple of map > units >> > as now). >> > So, for your script that means you don't have to find the > max-nodes >> > value. >> > >> > I'll think about this again... >> > >> > Gerd >> > >> > > Date: Tue, 29 Apr 2014 14:59:36 +0200 >> > > From:
osm@
>> > > To:
mkgmap-dev@.org
>> > > Subject: Re: [mkgmap-dev] mkgmap ToDo list >> > > >> > > While this possibly can be solved in Splitter or Mkgmap, it > could also >> > > be solved by your build-script when you add a maximum tile size > check >> > > and re-split (with a lower number of max-nodes) until you get > two or >> > > more sub-tiles. Granted, this adds complexity to the script but > it works >> > > well for me. >> > > >> > > On 25/04/2014 21:54, Henning Scholland wrote: >> > > > Hi Gerd, >> > > > >> > > > I would like to have img-tiles which have globally nearly the > same >> > > > filesize, so that they use the space of devices like eTrex 10. >> > > > >> > > > With my actual map I use globally the same value for > max-nodes. But the >> > > > size of the img-tiles differ more then factor 2. Eg. a tile in > Germany >> > > > is between 2 and 5 mb where a tile in China is about 10 mb. If > I remove >> > > > details, this difference will increase, because in Germany > more objects >> > > > will be removed from the img-tile then in China. >> > > > >> > > > Henning >> > > > >> > > > _______________________________________________ >> > > > mkgmap-dev mailing list >> > > >
mkgmap-dev@.org
>> > > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev >> > > >> > > _______________________________________________ >> > > mkgmap-dev mailing list >> > >
mkgmap-dev@.org
>> > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev >> > >> > >> > _______________________________________________ >> > mkgmap-dev mailing list >> >
mkgmap-dev@.org
>> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev >> > >> >> _______________________________________________ >> mkgmap-dev mailing list >>
mkgmap-dev@.org
>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > _______________________________________________ > mkgmap-dev mailing list >
mkgmap-dev@.org
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
-- View this message in context:
http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5804588.html
Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
On 30.04.2014 13:36, Lambertus wrote:
To get a better optimum in file size, using the process I described earlier, you could start off with a huge --max-nodes setting and then 'search' for the highest --max-nodes that works for each specific area.
Well that's basically what I do. I have for all areas a rather high max-nodes, and If I see it crashes on a tile, the next week I decrease it. However in reality I should also increase it sometimes - but that would be even more work. Some nice interaction between mkgmap calling splitter in order to keep track of crashing tiles due to too high max-nodes would be great. I cannot run multiple mkgmap.jar threads, as my server only has 12GB of RAM - and that works out to exactly running a single mkgmap.jar instance using max-jobs=8 --- plus multithreading the nsis.exe in the background as nsis packing is super slow... (damn, when will they ever update nsis to lzma2 and a rather recent .7z version - also getting rid of 2GB limit)..
You apparently do manually over a long timespan what my script essentially does on each map update run. Attached is my script (build.php), it will not run it but it might give you an example of how I implemented this. The interesting bits are the two functions render() and subsplit(). They call each other recursively until an image has been rendered successfully within the specified size limit or that subsplitting any further is not possible. Please excuse the sloppy code :p On 30/04/2014 13:47, Felix Hartmann wrote:
On 30.04.2014 13:36, Lambertus wrote:
To get a better optimum in file size, using the process I described earlier, you could start off with a huge --max-nodes setting and then 'search' for the highest --max-nodes that works for each specific area.
Well that's basically what I do. I have for all areas a rather high max-nodes, and If I see it crashes on a tile, the next week I decrease it. However in reality I should also increase it sometimes - but that would be even more work. Some nice interaction between mkgmap calling splitter in order to keep track of crashing tiles due to too high max-nodes would be great. I cannot run multiple mkgmap.jar threads, as my server only has 12GB of RAM - and that works out to exactly running a single mkgmap.jar instance using max-jobs=8 --- plus multithreading the nsis.exe in the background as nsis packing is super slow... (damn, when will they ever update nsis to lzma2 and a rather recent .7z version - also getting rid of 2GB limit).. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Perhaps you can use this parameter for makensis: -X'SetCompressor /FINAL bzip2' bzip2 compresses faster then lzma2 On 30/04/2014 13:47, Felix Hartmann wrote:
plus multithreading the nsis.exe in the background as nsis packing is super slow... (damn, when will they ever update nsis to lzma2 and a rather recent .7z version - also getting rid of 2GB limit).. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
On 30.04.2014 14:46, Lambertus wrote:
Perhaps you can use this parameter for makensis: -X'SetCompressor /FINAL bzip2'
bzip2 compresses faster then lzma2 Yes - but far far worse. Making the 2GB limit also 25% nearer...
On 30/04/2014 13:47, Felix Hartmann wrote:
plus multithreading the nsis.exe in the background as nsis packing is super slow... (damn, when will they ever update nsis to lzma2 and a rather recent .7z version - also getting rid of 2GB limit).. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- keep on biking and discovering new trails Felix openmtbmap.org & www.velomap.org
Hi, if I got that right the number of nodes is not highly correlated to the img size, so the max-nodes value is not a good estimate. I assume the reason is that nodes which belong to roads produce a lot more bytes in the img file compared to nodes which are parts of shapes or other non-routable ways, not talking about nodes which are simply ignored by the style. So, a possible solution in splitter could be to parse the ways before reading the nodes and save all nodeids which belong to ways with highway=*. If these nodes are refered by more than one way with highway=* we assume that they will be routing nodes. These special nodes could be counted e.g. 10 times to give a better estimate. Gerd
Date: Wed, 30 Apr 2014 13:36:19 +0200 From: osm@na1400.info To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] mkgmap ToDo list
Multithreading the tile rendering for a single map is indeed a bit difficult and I gave it up because you need to keep track which image id's are already in use. Since I provide multiple maps the work-around is running a few scripts parallel, which is also a crude form of multithreading.
The script language is PHP and it doesn't run on Windows without some changes ('/' vs '\' in paths, 'rm -rf', that sort of stuff). Never tried it.
To get a better optimum in file size, using the process I described earlier, you could start off with a huge --max-nodes setting and then 'search' for the highest --max-nodes that works for each specific area.
On 30/04/2014 11:49, Felix Hartmann wrote:
I would love if there was a possibility that you pass the used max-nodes value to mkgmap. When mkgmap is compiling the maps, then after the .img is created it should check a) did it crash due to too many max-nodes b) for me not important - but for others with very old GPS, etrex 10, ---> is tile bigger than X (usually 8) MB.
if a) or b) true, then pass the file back to splitter and split with 60% of maxnodes - and compile the resulting .img files again. Should it fail again, use 40%, again 25%... Sometimes there are awful tiles, that need supersmall max-nodes till they compile, however lately (last 1-2 years) I never encountered them anymore. I think that happened rather due to a but in splitter/mgkmap that is fixed now.
okay, you could also do this with a script, but it gets rather complicated to multithread it (you need to wait till mgkmap finished compiling all .img files - and run mkgmap first without address index to save time) and do some clever routines on making sure that the map id (e.g. 6340????.img) stay correct. Even more complicated to have consequent map id...
For europe with a fixed max-node I get tiles from 1.9MB up to 18MB. That's factor 9 - so it's huge... If I could narrows that down easily to 8-18MB - without getting tiles crashing due to too high max-nodes values, that would be sweet.
As for the scripts - would they run on windows too? - What programming language are they in?
On 29.04.2014 21:39, osm@na1400.info wrote:
Oh, and ofcourse anyone interested can get my scripts, send an email. They'll be on Github someday anyway.
On 2014-04-29 20:37, Gerd Petermann wrote:
Hi Lambertus,
okay, if I got that right you finally get *.img files with a size near (but below) 8MB, so maybe Henning can use that script, too.
If you do that for e.g. Germany, how small is tpically the smallest *.img file ? Is it probably near 4 MB?
Gerd
To: mkgmap-dev@lists.mkgmap.org.uk Date: Tue, 29 Apr 2014 20:30:27 +0200 From: osm@na1400.info Subject: Re: [mkgmap-dev] mkgmap ToDo list
These are the direct results from Splitter. The format is o5m, both input as output. Splitter version is: r321.
For this test I split the original source with --max-nodes=8000000. Then I render the initial tiles, when the result is larger than 8MB it's subsplit again with --max-nodes=(8000000-(attempt*100000)). The initial source files are ~70MB (o5m) and after several subsplits the two *.img are < 8MB. During this process --max-nodes has been reduced to e.g. 7500000 and the source file is split up in two o5m files of about 37MB.
I can upload an example source file and it's two subsplit siblings if you want.
On 2014-04-29 19:38, GerdP wrote:
Hi Lambertus,
that's interesting. Are these the img file sizes or the osm file sizes?
Gerd
Lambertus wrote > Unfortunately I cannot confirm that. Below is a bit of logging from my > script: > Original: 97000020 (70551453), New: 0 (35684445), New: 1 (36852845) > Original: 97000001 (74621042), New: 0 (37522992), New: 1 (37222739) > Original: 97000002 (73391358), New: 0 (37679505), New: 1 (38098627) > Original: 97000003 (77862567), New: 0 (39075311), New: 1 (39261197) > > The original files above contain contour data, the filesize is between > brackets. As you can see both resulting file are approximately the > same > size. > > On 2014-04-29 15:39, Gerd Petermann wrote: >> Hi Lambertus, >> >> and I guess that even after this optimization you will >> see a factor 3 or higher between the largest tile and the smallest. >> Can you confirm that? >> >> Gerd >> >>> Date: Tue, 29 Apr 2014 15:32:38 +0200 >>> From:
> osm@
>>> To:
> mkgmap-dev@.org
>>> Subject: Re: [mkgmap-dev] mkgmap ToDo list >>> >>> Num-tiles=x would indeed be better for this specific need. >>> >>> It is my experience that it regularly takes multiple calls to >> Splitter >>> to get 2+ sub-tiles when you reduce the max-nodes by 100k for each >>> sub-split attempt. This is what I currently do to get an optimum in >>> tile-size vs total number of tiles. >>> >>> >>> >>> On 29/04/2014 15:09, Gerd Petermann wrote: >>> > Hi Lambertus, >>> > >>> > that sounds like a possible change in splitter: >>> > Instead of specifying max-nodes you may specify --num-tiles=x >>> > and splitter will try to find a split that produces excactly x >> tiles >>> > which are not too narrow and have a node number which is not >>> > too far from the average (but still aligned to a multiple of map >> units >>> > as now). >>> > So, for your script that means you don't have to find the >> max-nodes >>> > value. >>> > >>> > I'll think about this again... >>> > >>> > Gerd >>> > >>> > > Date: Tue, 29 Apr 2014 14:59:36 +0200 >>> > > From:
> osm@
>>> > > To:
> mkgmap-dev@.org
>>> > > Subject: Re: [mkgmap-dev] mkgmap ToDo list >>> > > >>> > > While this possibly can be solved in Splitter or Mkgmap, it >> could also >>> > > be solved by your build-script when you add a maximum tile size >> check >>> > > and re-split (with a lower number of max-nodes) until you get >> two or >>> > > more sub-tiles. Granted, this adds complexity to the script but >> it works >>> > > well for me. >>> > > >>> > > On 25/04/2014 21:54, Henning Scholland wrote: >>> > > > Hi Gerd, >>> > > > >>> > > > I would like to have img-tiles which have globally nearly the >> same >>> > > > filesize, so that they use the space of devices like eTrex 10. >>> > > > >>> > > > With my actual map I use globally the same value for >> max-nodes. But the >>> > > > size of the img-tiles differ more then factor 2. Eg. a tile in >> Germany >>> > > > is between 2 and 5 mb where a tile in China is about 10 mb. If >> I remove >>> > > > details, this difference will increase, because in Germany >> more objects >>> > > > will be removed from the img-tile then in China. >>> > > > >>> > > > Henning >>> > > > >>> > > > _______________________________________________ >>> > > > mkgmap-dev mailing list >>> > > >
> mkgmap-dev@.org
>>> > > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev >>> > > >>> > > _______________________________________________ >>> > > mkgmap-dev mailing list >>> > >
> mkgmap-dev@.org
>>> > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev >>> > >>> > >>> > _______________________________________________ >>> > mkgmap-dev mailing list >>> >
> mkgmap-dev@.org
>>> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev >>> > >>> >>> _______________________________________________ >>> mkgmap-dev mailing list >>>
> mkgmap-dev@.org
>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev >> >> _______________________________________________ >> mkgmap-dev mailing list >>
> mkgmap-dev@.org
>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > _______________________________________________ > mkgmap-dev mailing list
> mkgmap-dev@.org
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- View this message in context:
http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5804588.html
Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
if that doesn't seriously (more than 30-40%) slow down the splitter, I assume it would be much better... On 30.04.2014 14:06, Gerd Petermann wrote:
Hi,
if I got that right the number of nodes is not highly correlated to the img size, so the max-nodes value is not a good estimate.
I assume the reason is that nodes which belong to roads produce a lot more bytes in the img file compared to nodes which are parts of shapes or other non-routable ways, not talking about nodes which are simply ignored by the style.
So, a possible solution in splitter could be to parse the ways before reading the nodes and save all nodeids which belong to ways with highway=*. If these nodes are refered by more than one way with highway=* we assume that they will be routing nodes. These special nodes could be counted e.g. 10 times to give a better estimate.
Gerd
Date: Wed, 30 Apr 2014 13:36:19 +0200 From: osm@na1400.info To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] mkgmap ToDo list
Multithreading the tile rendering for a single map is indeed a bit difficult and I gave it up because you need to keep track which image id's are already in use. Since I provide multiple maps the work-around is running a few scripts parallel, which is also a crude form of multithreading.
The script language is PHP and it doesn't run on Windows without some changes ('/' vs '\' in paths, 'rm -rf', that sort of stuff). Never tried it.
To get a better optimum in file size, using the process I described earlier, you could start off with a huge --max-nodes setting and then 'search' for the highest --max-nodes that works for each specific area.
On 30/04/2014 11:49, Felix Hartmann wrote:
I would love if there was a possibility that you pass the used max-nodes value to mkgmap. When mkgmap is compiling the maps, then after the .img is created it should check a) did it crash due to too many max-nodes b) for me not important - but for others with very old GPS, etrex 10, ---> is tile bigger than X (usually 8) MB.
if a) or b) true, then pass the file back to splitter and split with 60% of maxnodes - and compile the resulting .img files again. Should it fail again, use 40%, again 25%... Sometimes there are awful tiles, that need supersmall max-nodes till they compile, however lately (last 1-2 years) I never encountered them anymore. I think that happened rather due to a but in splitter/mgkmap that is fixed now.
okay, you could also do this with a script, but it gets rather complicated to multithread it (you need to wait till mgkmap finished compiling all .img files - and run mkgmap first without address index to save time) and do some clever routines on making sure that the map id (e.g. 6340????.img) stay correct. Even more complicated to have consequent map id...
For europe with a fixed max-node I get tiles from 1.9MB up to 18MB. That's factor 9 - so it's huge... If I could narrows that down easily to 8-18MB - without getting tiles crashing due to too high max-nodes values, that would be sweet.
As for the scripts - would they run on windows too? - What programming language are they in?
On 29.04.2014 21:39, osm@na1400.info wrote:
Oh, and ofcourse anyone interested can get my scripts, send an email. They'll be on Github someday anyway.
On 2014-04-29 20:37, Gerd Petermann wrote:
Hi Lambertus,
okay, if I got that right you finally get *.img files with a size near (but below) 8MB, so maybe Henning can use that script, too.
If you do that for e.g. Germany, how small is tpically the smallest *.img file ? Is it probably near 4 MB?
Gerd
To: mkgmap-dev@lists.mkgmap.org.uk Date: Tue, 29 Apr 2014 20:30:27 +0200 From: osm@na1400.info Subject: Re: [mkgmap-dev] mkgmap ToDo list
These are the direct results from Splitter. The format is o5m, both input as output. Splitter version is: r321.
For this test I split the original source with --max-nodes=8000000. Then I render the initial tiles, when the result is larger than 8MB it's subsplit again with --max-nodes=(8000000-(attempt*100000)). The initial source files are ~70MB (o5m) and after several subsplits the two *.img are < 8MB. During this process --max-nodes has been reduced to e.g. 7500000 and the source file is split up in two o5m files of about 37MB.
I can upload an example source file and it's two subsplit siblings if you want.
On 2014-04-29 19:38, GerdP wrote: > Hi Lambertus, > > that's interesting. Are these the img file sizes or the osm file sizes? > > Gerd > > > Lambertus wrote >> Unfortunately I cannot confirm that. Below is a bit of logging from my >> script: >> Original: 97000020 (70551453), New: 0 (35684445), New: 1 (36852845) >> Original: 97000001 (74621042), New: 0 (37522992), New: 1 (37222739) >> Original: 97000002 (73391358), New: 0 (37679505), New: 1 (38098627) >> Original: 97000003 (77862567), New: 0 (39075311), New: 1 (39261197) >> >> The original files above contain contour data, the filesize is between >> brackets. As you can see both resulting file are approximately the >> same >> size. >> >> On 2014-04-29 15:39, Gerd Petermann wrote: >>> Hi Lambertus, >>> >>> and I guess that even after this optimization you will >>> see a factor 3 or higher between the largest tile and the smallest. >>> Can you confirm that? >>> >>> Gerd >>> >>>> Date: Tue, 29 Apr 2014 15:32:38 +0200 >>>> From: > >> osm@ > >>>> To: > >> mkgmap-dev@.org > >>>> Subject: Re: [mkgmap-dev] mkgmap ToDo list >>>> >>>> Num-tiles=x would indeed be better for this specific need. >>>> >>>> It is my experience that it regularly takes multiple calls to >>> Splitter >>>> to get 2+ sub-tiles when you reduce the max-nodes by 100k for each >>>> sub-split attempt. This is what I currently do to get an optimum in >>>> tile-size vs total number of tiles. >>>> >>>> >>>> >>>> On 29/04/2014 15:09, Gerd Petermann wrote: >>>> > Hi Lambertus, >>>> > >>>> > that sounds like a possible change in splitter: >>>> > Instead of specifying max-nodes you may specify --num-tiles=x >>>> > and splitter will try to find a split that produces excactly x >>> tiles >>>> > which are not too narrow and have a node number which is not >>>> > too far from the average (but still aligned to a multiple of map >>> units >>>> > as now). >>>> > So, for your script that means you don't have to find the >>> max-nodes >>>> > value. >>>> > >>>> > I'll think about this again... >>>> > >>>> > Gerd >>>> > >>>> > > Date: Tue, 29 Apr 2014 14:59:36 +0200 >>>> > > From: > >> osm@ > >>>> > > To: > >> mkgmap-dev@.org > >>>> > > Subject: Re: [mkgmap-dev] mkgmap ToDo list >>>> > > >>>> > > While this possibly can be solved in Splitter or Mkgmap, it >>> could also >>>> > > be solved by your build-script when you add a maximum tile size >>> check >>>> > > and re-split (with a lower number of max-nodes) until you get >>> two or >>>> > > more sub-tiles. Granted, this adds complexity to the script but >>> it works >>>> > > well for me. >>>> > > >>>> > > On 25/04/2014 21:54, Henning Scholland wrote: >>>> > > > Hi Gerd, >>>> > > > >>>> > > > I would like to have img-tiles which have globally nearly the >>> same >>>> > > > filesize, so that they use the space of devices like eTrex 10. >>>> > > > >>>> > > > With my actual map I use globally the same value for >>> max-nodes. But the >>>> > > > size of the img-tiles differ more then factor 2. Eg. a tile in >>> Germany >>>> > > > is between 2 and 5 mb where a tile in China is about 10 mb. If >>> I remove >>>> > > > details, this difference will increase, because in Germany >>> more objects >>>> > > > will be removed from the img-tile then in China. >>>> > > > >>>> > > > Henning >>>> > > > >>>> > > > _______________________________________________ >>>> > > > mkgmap-dev mailing list >>>> > > > > >> mkgmap-dev@.org > >>>> > > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev >>>> > > >>>> > > _______________________________________________ >>>> > > mkgmap-dev mailing list >>>> > > > >> mkgmap-dev@.org > >>>> > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev >>>> > >>>> > >>>> > _______________________________________________ >>>> > mkgmap-dev mailing list >>>> > > >> mkgmap-dev@.org > >>>> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev >>>> > >>>> >>>> _______________________________________________ >>>> mkgmap-dev mailing list >>>> > >> mkgmap-dev@.org > >>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev >>> >>> _______________________________________________ >>> mkgmap-dev mailing list >>> > >> mkgmap-dev@.org > >>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev >> _______________________________________________ >> mkgmap-dev mailing list > >> mkgmap-dev@.org > >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > > > > -- > View this message in context: >
http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5804588.html
> Sent from the Mkgmap Development mailing list archive at Nabble.com. > _______________________________________________ > mkgmap-dev mailing list > mkgmap-dev@lists.mkgmap.org.uk > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- keep on biking and discovering new trails Felix openmtbmap.org & www.velomap.org
Hi all, I've coded a quick hack which seems to improve the ratios. On the other hand, I don't see these large differences between smallest and largest img file. What part of the world should I try? Do you use the precomp-sea parameter in splitter? Gerd Date: Wed, 30 Apr 2014 14:59:32 +0200 From: extremecarver@gmail.com To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] mkgmap ToDo list if that doesn't seriously (more than 30-40%) slow down the splitter, I assume it would be much better... On 30.04.2014 14:06, Gerd Petermann wrote: Hi, if I got that right the number of nodes is not highly correlated to the img size, so the max-nodes value is not a good estimate. I assume the reason is that nodes which belong to roads produce a lot more bytes in the img file compared to nodes which are parts of shapes or other non-routable ways, not talking about nodes which are simply ignored by the style. So, a possible solution in splitter could be to parse the ways before reading the nodes and save all nodeids which belong to ways with highway=*. If these nodes are refered by more than one way with highway=* we assume that they will be routing nodes. These special nodes could be counted e.g. 10 times to give a better estimate. Gerd > Date: Wed, 30 Apr 2014 13:36:19 +0200 > From: osm@na1400.info > To: mkgmap-dev@lists.mkgmap.org.uk > Subject: Re: [mkgmap-dev] mkgmap ToDo list > > Multithreading the tile rendering for a single map is indeed a bit > difficult and I gave it up because you need to keep track which image > id's are already in use. Since I provide multiple maps the work-around > is running a few scripts parallel, which is also a crude form of > multithreading. > > The script language is PHP and it doesn't run on Windows without some > changes ('/' vs '\' in paths, 'rm -rf', that sort of stuff). Never tried it. > > To get a better optimum in file size, using the process I described > earlier, you could start off with a huge --max-nodes setting and then > 'search' for the highest --max-nodes that works for each specific area. > > On 30/04/2014 11:49, Felix Hartmann wrote: > > I would love if there was a possibility that you pass the used max-nodes > > value to mkgmap. > > When mkgmap is compiling the maps, then after the .img is created it > > should check > > a) did it crash due to too many max-nodes > > b) for me not important - but for others with very old GPS, etrex 10, > > ---> is tile bigger than X (usually 8) MB. > > > > if a) or b) true, then pass the file back to splitter and split with 60% > > of maxnodes - and compile the resulting .img files again. Should it fail > > again, use 40%, again 25%... Sometimes there are awful tiles, that need > > supersmall max-nodes till they compile, however lately (last 1-2 years) > > I never encountered them anymore. I think that happened rather due to a > > but in splitter/mgkmap that is fixed now. > > > > okay, you could also do this with a script, but it gets rather > > complicated to multithread it (you need to wait till mgkmap finished > > compiling all .img files - and run mkgmap first without address index to > > save time) and do some clever routines on making sure that the map id > > (e.g. 6340????.img) stay correct. Even more complicated to have > > consequent map id... > > > > For europe with a fixed max-node I get tiles from 1.9MB up to 18MB. > > That's factor 9 - so it's huge... > > If I could narrows that down easily to 8-18MB - without getting tiles > > crashing due to too high max-nodes values, that would be sweet. > > > > > > > > As for the scripts - would they run on windows too? - What programming > > language are they in? > > > > > > On 29.04.2014 21:39, osm@na1400.info wrote: > >> Oh, and ofcourse anyone interested can get my scripts, send an email. > >> They'll be on Github someday anyway. > >> > >> > >> On 2014-04-29 20:37, Gerd Petermann wrote: > >>> Hi Lambertus, > >>> > >>> okay, if I got that right you finally get *.img files with a size > >>> near (but below) 8MB, so maybe Henning can use that script, too. > >>> > >>> If you do that for e.g. Germany, how small is tpically the smallest > >>> *.img file ? > >>> Is it probably near 4 MB? > >>> > >>> Gerd > >>> > >>>> To: mkgmap-dev@lists.mkgmap.org.uk > >>>> Date: Tue, 29 Apr 2014 20:30:27 +0200 > >>>> From: osm@na1400.info > >>>> Subject: Re: [mkgmap-dev] mkgmap ToDo list > >>>> > >>>> These are the direct results from Splitter. The format is o5m, both > >>>> input as output. Splitter version is: r321. > >>>> > >>>> For this test I split the original source with --max-nodes=8000000. > >>> Then > >>>> I render the initial tiles, when the result is larger than 8MB it's > >>>> subsplit again with --max-nodes=(8000000-(attempt*100000)). The > >>> initial > >>>> source files are ~70MB (o5m) and after several subsplits the two > >>> *.img > >>>> are < 8MB. During this process --max-nodes has been reduced to e.g. > >>>> 7500000 and the source file is split up in two o5m files of about > >>> 37MB. > >>>> > >>>> I can upload an example source file and it's two subsplit siblings > >>> if > >>>> you want. > >>>> > >>>> > >>>> On 2014-04-29 19:38, GerdP wrote: > >>>> > Hi Lambertus, > >>>> > > >>>> > that's interesting. Are these the img file sizes or the osm file > >>> sizes? > >>>> > > >>>> > Gerd > >>>> > > >>>> > > >>>> > Lambertus wrote > >>>> >> Unfortunately I cannot confirm that. Below is a bit of logging > >>> from my > >>>> >> script: > >>>> >> Original: 97000020 (70551453), New: 0 (35684445), New: 1 > >>> (36852845) > >>>> >> Original: 97000001 (74621042), New: 0 (37522992), New: 1 > >>> (37222739) > >>>> >> Original: 97000002 (73391358), New: 0 (37679505), New: 1 > >>> (38098627) > >>>> >> Original: 97000003 (77862567), New: 0 (39075311), New: 1 > >>> (39261197) > >>>> >> > >>>> >> The original files above contain contour data, the filesize is > >>> between > >>>> >> brackets. As you can see both resulting file are approximately > >>> the > >>>> >> same > >>>> >> size. > >>>> >> > >>>> >> On 2014-04-29 15:39, Gerd Petermann wrote: > >>>> >>> Hi Lambertus, > >>>> >>> > >>>> >>> and I guess that even after this optimization you will > >>>> >>> see a factor 3 or higher between the largest tile and the > >>> smallest. > >>>> >>> Can you confirm that? > >>>> >>> > >>>> >>> Gerd > >>>> >>> > >>>> >>>> Date: Tue, 29 Apr 2014 15:32:38 +0200 > >>>> >>>> From: > >>>> > > >>>> >> osm@ > >>>> > > >>>> >>>> To: > >>>> > > >>>> >> mkgmap-dev@.org > >>>> > > >>>> >>>> Subject: Re: [mkgmap-dev] mkgmap ToDo list > >>>> >>>> > >>>> >>>> Num-tiles=x would indeed be better for this specific need. > >>>> >>>> > >>>> >>>> It is my experience that it regularly takes multiple calls to > >>>> >>> Splitter > >>>> >>>> to get 2+ sub-tiles when you reduce the max-nodes by 100k for > >>> each > >>>> >>>> sub-split attempt. This is what I currently do to get an > >>> optimum in > >>>> >>>> tile-size vs total number of tiles. > >>>> >>>> > >>>> >>>> > >>>> >>>> > >>>> >>>> On 29/04/2014 15:09, Gerd Petermann wrote: > >>>> >>>> > Hi Lambertus, > >>>> >>>> > > >>>> >>>> > that sounds like a possible change in splitter: > >>>> >>>> > Instead of specifying max-nodes you may specify --num-tiles=x > >>>> >>>> > and splitter will try to find a split that produces excactly > >>> x > >>>> >>> tiles > >>>> >>>> > which are not too narrow and have a node number which is not > >>>> >>>> > too far from the average (but still aligned to a multiple of > >>> map > >>>> >>> units > >>>> >>>> > as now). > >>>> >>>> > So, for your script that means you don't have to find the > >>>> >>> max-nodes > >>>> >>>> > value. > >>>> >>>> > > >>>> >>>> > I'll think about this again... > >>>> >>>> > > >>>> >>>> > Gerd > >>>> >>>> > > >>>> >>>> > > Date: Tue, 29 Apr 2014 14:59:36 +0200 > >>>> >>>> > > From: > >>>> > > >>>> >> osm@ > >>>> > > >>>> >>>> > > To: > >>>> > > >>>> >> mkgmap-dev@.org > >>>> > > >>>> >>>> > > Subject: Re: [mkgmap-dev] mkgmap ToDo list > >>>> >>>> > > > >>>> >>>> > > While this possibly can be solved in Splitter or Mkgmap, it > >>>> >>> could also > >>>> >>>> > > be solved by your build-script when you add a maximum tile > >>> size > >>>> >>> check > >>>> >>>> > > and re-split (with a lower number of max-nodes) until you > >>> get > >>>> >>> two or > >>>> >>>> > > more sub-tiles. Granted, this adds complexity to the script > >>> but > >>>> >>> it works > >>>> >>>> > > well for me. > >>>> >>>> > > > >>>> >>>> > > On 25/04/2014 21:54, Henning Scholland wrote: > >>>> >>>> > > > Hi Gerd, > >>>> >>>> > > > > >>>> >>>> > > > I would like to have img-tiles which have globally nearly > >>> the > >>>> >>> same > >>>> >>>> > > > filesize, so that they use the space of devices like > >>> eTrex 10. > >>>> >>>> > > > > >>>> >>>> > > > With my actual map I use globally the same value for > >>>> >>> max-nodes. But the > >>>> >>>> > > > size of the img-tiles differ more then factor 2. Eg. a > >>> tile in > >>>> >>> Germany > >>>> >>>> > > > is between 2 and 5 mb where a tile in China is about 10 > >>> mb. If > >>>> >>> I remove > >>>> >>>> > > > details, this difference will increase, because in > >>> Germany > >>>> >>> more objects > >>>> >>>> > > > will be removed from the img-tile then in China. > >>>> >>>> > > > > >>>> >>>> > > > Henning > >>>> >>>> > > > > >>>> >>>> > > > _______________________________________________ > >>>> >>>> > > > mkgmap-dev mailing list > >>>> >>>> > > > > >>>> > > >>>> >> mkgmap-dev@.org > >>>> > > >>>> >>>> > > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >>>> > > > >>>> >>>> > > _______________________________________________ > >>>> >>>> > > mkgmap-dev mailing list > >>>> >>>> > > > >>>> > > >>>> >> mkgmap-dev@.org > >>>> > > >>>> >>>> > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >>>> > > >>>> >>>> > > >>>> >>>> > _______________________________________________ > >>>> >>>> > mkgmap-dev mailing list > >>>> >>>> > > >>>> > > >>>> >> mkgmap-dev@.org > >>>> > > >>>> >>>> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >>>> > > >>>> >>>> > >>>> >>>> _______________________________________________ > >>>> >>>> mkgmap-dev mailing list > >>>> >>>> > >>>> > > >>>> >> mkgmap-dev@.org > >>>> > > >>>> >>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >>> > >>>> >>> _______________________________________________ > >>>> >>> mkgmap-dev mailing list > >>>> >>> > >>>> > > >>>> >> mkgmap-dev@.org > >>>> > > >>>> >>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >> _______________________________________________ > >>>> >> mkgmap-dev mailing list > >>>> > > >>>> >> mkgmap-dev@.org > >>>> > > >>>> >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > -- > >>>> > View this message in context: > >>>> > > >>> http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5804588.html > >>>> > Sent from the Mkgmap Development mailing list archive at > >>> Nabble.com. > >>>> > _______________________________________________ > >>>> > mkgmap-dev mailing list > >>>> > mkgmap-dev@lists.mkgmap.org.uk > >>>> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> _______________________________________________ > >>>> mkgmap-dev mailing list > >>>> mkgmap-dev@lists.mkgmap.org.uk > >>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>> > >>> _______________________________________________ > >>> mkgmap-dev mailing list > >>> mkgmap-dev@lists.mkgmap.org.uk > >>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >> _______________________________________________ > >> mkgmap-dev mailing list > >> mkgmap-dev@lists.mkgmap.org.uk > >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > > _______________________________________________ > mkgmap-dev mailing list > mkgmap-dev@lists.mkgmap.org.uk > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev -- keep on biking and discovering new trails Felix openmtbmap.org & www.velomap.org _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
No - I use splitter like this - with maxnodes depending on area - I usually just reduced it after failures to compile a tile.: java -Xms4000m -Xmx10400m -jar c:\openmtbmap\splitter.jar --max-nodes=%maxnodes% --max-threads=8 --output=pbf --keep-complete --max-areas=1524 --geonames-file=cities5000 --description=%country% --mapid=%FID%0000 c:\OpenMTBMap\osmpbf_geofabrik\%country%-latest.osm.pbf would using the precomp-sea parameter for splitter improve results? It's not that tiles with loads of sea actually failed. On 03.05.2014 09:00, Gerd Petermann wrote:
Hi all,
I've coded a quick hack which seems to improve the ratios. On the other hand, I don't see these large differences between smallest and largest img file. What part of the world should I try? Do you use the precomp-sea parameter in splitter?
Gerd
------------------------------------------------------------------------ Date: Wed, 30 Apr 2014 14:59:32 +0200 From: extremecarver@gmail.com To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] mkgmap ToDo list
if that doesn't seriously (more than 30-40%) slow down the splitter, I assume it would be much better... On 30.04.2014 14:06, Gerd Petermann wrote:
Hi,
if I got that right the number of nodes is not highly correlated to the img size, so the max-nodes value is not a good estimate.
I assume the reason is that nodes which belong to roads produce a lot more bytes in the img file compared to nodes which are parts of shapes or other non-routable ways, not talking about nodes which are simply ignored by the style.
So, a possible solution in splitter could be to parse the ways before reading the nodes and save all nodeids which belong to ways with highway=*. If these nodes are refered by more than one way with highway=* we assume that they will be routing nodes. These special nodes could be counted e.g. 10 times to give a better estimate.
Gerd
> Date: Wed, 30 Apr 2014 13:36:19 +0200 > From: osm@na1400.info <mailto:osm@na1400.info> > To: mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> > Subject: Re: [mkgmap-dev] mkgmap ToDo list > > Multithreading the tile rendering for a single map is indeed a bit > difficult and I gave it up because you need to keep track which image > id's are already in use. Since I provide multiple maps the work-around > is running a few scripts parallel, which is also a crude form of > multithreading. > > The script language is PHP and it doesn't run on Windows without some > changes ('/' vs '\' in paths, 'rm -rf', that sort of stuff). Never tried it. > > To get a better optimum in file size, using the process I described > earlier, you could start off with a huge --max-nodes setting and then > 'search' for the highest --max-nodes that works for each specific area. > > On 30/04/2014 11:49, Felix Hartmann wrote: > > I would love if there was a possibility that you pass the used max-nodes > > value to mkgmap. > > When mkgmap is compiling the maps, then after the .img is created it > > should check > > a) did it crash due to too many max-nodes > > b) for me not important - but for others with very old GPS, etrex 10, > > ---> is tile bigger than X (usually 8) MB. > > > > if a) or b) true, then pass the file back to splitter and split with 60% > > of maxnodes - and compile the resulting .img files again. Should it fail > > again, use 40%, again 25%... Sometimes there are awful tiles, that need > > supersmall max-nodes till they compile, however lately (last 1-2 years) > > I never encountered them anymore. I think that happened rather due to a > > but in splitter/mgkmap that is fixed now. > > > > okay, you could also do this with a script, but it gets rather > > complicated to multithread it (you need to wait till mgkmap finished > > compiling all .img files - and run mkgmap first without address index to > > save time) and do some clever routines on making sure that the map id > > (e.g. 6340????.img) stay correct. Even more complicated to have > > consequent map id... > > > > For europe with a fixed max-node I get tiles from 1.9MB up to 18MB. > > That's factor 9 - so it's huge... > > If I could narrows that down easily to 8-18MB - without getting tiles > > crashing due to too high max-nodes values, that would be sweet. > > > > > > > > As for the scripts - would they run on windows too? - What programming > > language are they in? > > > > > > On 29.04.2014 21:39, osm@na1400.info <mailto:osm@na1400.info> wrote: > >> Oh, and ofcourse anyone interested can get my scripts, send an email. > >> They'll be on Github someday anyway. > >> > >> > >> On 2014-04-29 20:37, Gerd Petermann wrote: > >>> Hi Lambertus, > >>> > >>> okay, if I got that right you finally get *.img files with a size > >>> near (but below) 8MB, so maybe Henning can use that script, too. > >>> > >>> If you do that for e.g. Germany, how small is tpically the smallest > >>> *.img file ? > >>> Is it probably near 4 MB? > >>> > >>> Gerd > >>> > >>>> To: mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> > >>>> Date: Tue, 29 Apr 2014 20:30:27 +0200 > >>>> From: osm@na1400.info <mailto:osm@na1400.info> > >>>> Subject: Re: [mkgmap-dev] mkgmap ToDo list > >>>> > >>>> These are the direct results from Splitter. The format is o5m, both > >>>> input as output. Splitter version is: r321. > >>>> > >>>> For this test I split the original source with --max-nodes=8000000. > >>> Then > >>>> I render the initial tiles, when the result is larger than 8MB it's > >>>> subsplit again with --max-nodes=(8000000-(attempt*100000)). The > >>> initial > >>>> source files are ~70MB (o5m) and after several subsplits the two > >>> *.img > >>>> are < 8MB. During this process --max-nodes has been reduced to e.g. > >>>> 7500000 and the source file is split up in two o5m files of about > >>> 37MB. > >>>> > >>>> I can upload an example source file and it's two subsplit siblings > >>> if > >>>> you want. > >>>> > >>>> > >>>> On 2014-04-29 19:38, GerdP wrote: > >>>> > Hi Lambertus, > >>>> > > >>>> > that's interesting. Are these the img file sizes or the osm file > >>> sizes? > >>>> > > >>>> > Gerd > >>>> > > >>>> > > >>>> > Lambertus wrote > >>>> >> Unfortunately I cannot confirm that. Below is a bit of logging > >>> from my > >>>> >> script: > >>>> >> Original: 97000020 (70551453), New: 0 (35684445), New: 1 > >>> (36852845) > >>>> >> Original: 97000001 (74621042), New: 0 (37522992), New: 1 > >>> (37222739) > >>>> >> Original: 97000002 (73391358), New: 0 (37679505), New: 1 > >>> (38098627) > >>>> >> Original: 97000003 (77862567), New: 0 (39075311), New: 1 > >>> (39261197) > >>>> >> > >>>> >> The original files above contain contour data, the filesize is > >>> between > >>>> >> brackets. As you can see both resulting file are approximately > >>> the > >>>> >> same > >>>> >> size. > >>>> >> > >>>> >> On 2014-04-29 15:39, Gerd Petermann wrote: > >>>> >>> Hi Lambertus, > >>>> >>> > >>>> >>> and I guess that even after this optimization you will > >>>> >>> see a factor 3 or higher between the largest tile and the > >>> smallest. > >>>> >>> Can you confirm that? > >>>> >>> > >>>> >>> Gerd > >>>> >>> > >>>> >>>> Date: Tue, 29 Apr 2014 15:32:38 +0200 > >>>> >>>> From: > >>>> > > >>>> >> osm@ > >>>> > > >>>> >>>> To: > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >>>> Subject: Re: [mkgmap-dev] mkgmap ToDo list > >>>> >>>> > >>>> >>>> Num-tiles=x would indeed be better for this specific need. > >>>> >>>> > >>>> >>>> It is my experience that it regularly takes multiple calls to > >>>> >>> Splitter > >>>> >>>> to get 2+ sub-tiles when you reduce the max-nodes by 100k for > >>> each > >>>> >>>> sub-split attempt. This is what I currently do to get an > >>> optimum in > >>>> >>>> tile-size vs total number of tiles. > >>>> >>>> > >>>> >>>> > >>>> >>>> > >>>> >>>> On 29/04/2014 15:09, Gerd Petermann wrote: > >>>> >>>> > Hi Lambertus, > >>>> >>>> > > >>>> >>>> > that sounds like a possible change in splitter: > >>>> >>>> > Instead of specifying max-nodes you may specify --num-tiles=x > >>>> >>>> > and splitter will try to find a split that produces excactly > >>> x > >>>> >>> tiles > >>>> >>>> > which are not too narrow and have a node number which is not > >>>> >>>> > too far from the average (but still aligned to a multiple of > >>> map > >>>> >>> units > >>>> >>>> > as now). > >>>> >>>> > So, for your script that means you don't have to find the > >>>> >>> max-nodes > >>>> >>>> > value. > >>>> >>>> > > >>>> >>>> > I'll think about this again... > >>>> >>>> > > >>>> >>>> > Gerd > >>>> >>>> > > >>>> >>>> > > Date: Tue, 29 Apr 2014 14:59:36 +0200 > >>>> >>>> > > From: > >>>> > > >>>> >> osm@ > >>>> > > >>>> >>>> > > To: > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >>>> > > Subject: Re: [mkgmap-dev] mkgmap ToDo list > >>>> >>>> > > > >>>> >>>> > > While this possibly can be solved in Splitter or Mkgmap, it > >>>> >>> could also > >>>> >>>> > > be solved by your build-script when you add a maximum tile > >>> size > >>>> >>> check > >>>> >>>> > > and re-split (with a lower number of max-nodes) until you > >>> get > >>>> >>> two or > >>>> >>>> > > more sub-tiles. Granted, this adds complexity to the script > >>> but > >>>> >>> it works > >>>> >>>> > > well for me. > >>>> >>>> > > > >>>> >>>> > > On 25/04/2014 21:54, Henning Scholland wrote: > >>>> >>>> > > > Hi Gerd, > >>>> >>>> > > > > >>>> >>>> > > > I would like to have img-tiles which have globally nearly > >>> the > >>>> >>> same > >>>> >>>> > > > filesize, so that they use the space of devices like > >>> eTrex 10. > >>>> >>>> > > > > >>>> >>>> > > > With my actual map I use globally the same value for > >>>> >>> max-nodes. But the > >>>> >>>> > > > size of the img-tiles differ more then factor 2. Eg. a > >>> tile in > >>>> >>> Germany > >>>> >>>> > > > is between 2 and 5 mb where a tile in China is about 10 > >>> mb. If > >>>> >>> I remove > >>>> >>>> > > > details, this difference will increase, because in > >>> Germany > >>>> >>> more objects > >>>> >>>> > > > will be removed from the img-tile then in China. > >>>> >>>> > > > > >>>> >>>> > > > Henning > >>>> >>>> > > > > >>>> >>>> > > > _______________________________________________ > >>>> >>>> > > > mkgmap-dev mailing list > >>>> >>>> > > > > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >>>> > > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >>>> > > > >>>> >>>> > > _______________________________________________ > >>>> >>>> > > mkgmap-dev mailing list > >>>> >>>> > > > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >>>> > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >>>> > > >>>> >>>> > > >>>> >>>> > _______________________________________________ > >>>> >>>> > mkgmap-dev mailing list > >>>> >>>> > > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >>>> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >>>> > > >>>> >>>> > >>>> >>>> _______________________________________________ > >>>> >>>> mkgmap-dev mailing list > >>>> >>>> > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >>> > >>>> >>> _______________________________________________ > >>>> >>> mkgmap-dev mailing list > >>>> >>> > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >> _______________________________________________ > >>>> >> mkgmap-dev mailing list > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > -- > >>>> > View this message in context: > >>>> > > >>> http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5804588.html > >>>> > Sent from the Mkgmap Development mailing list archive at > >>> Nabble.com. > >>>> > _______________________________________________ > >>>> > mkgmap-dev mailing list > >>>> > mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> > >>>> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> _______________________________________________ > >>>> mkgmap-dev mailing list > >>>> mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> > >>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>> > >>> _______________________________________________ > >>> mkgmap-dev mailing list > >>> mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> > >>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >> _______________________________________________ > >> mkgmap-dev mailing list > >> mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> > >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > > _______________________________________________ > mkgmap-dev mailing list > mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- keep on biking and discovering new trails
Felix openmtbmap.org &www.velomap.org <http://www.velomap.org>
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- keep on biking and discovering new trails Felix openmtbmap.org & www.velomap.org
Hi Felix,
would using the precomp-sea parameter for splitter improve results? It's not that tiles with loads of sea actually failed.
good question. I've coded that parameter in splitter before we did some important changes in mkgmap. I think it is needed when you use a polygon file in combination with an imput file that doesn't cover the polygon. If the polygon covers a costline with a lot of islands, but the input file doesn't contain the data, splitter might create large tiles covering these "empty" areas. Later, in mkgmap, the empty areas are filled with the precompiled-sea data, and that can cause too large files. BTW: The quick hack turned out to be useless. It worked better in some areas and worse in others. I'll look again at this later next week. Gerd Date: Sun, 4 May 2014 21:21:59 +0200 From: extremecarver@gmail.com To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] mkgmap ToDo list No - I use splitter like this - with maxnodes depending on area - I usually just reduced it after failures to compile a tile.: java -Xms4000m -Xmx10400m -jar c:\openmtbmap\splitter.jar --max-nodes=%maxnodes% --max-threads=8 --output=pbf --keep-complete --max-areas=1524 --geonames-file=cities5000 --description=%country% --mapid=%FID%0000 c:\OpenMTBMap\osmpbf_geofabrik\%country%-latest.osm.pbf would using the precomp-sea parameter for splitter improve results? It's not that tiles with loads of sea actually failed. On 03.05.2014 09:00, Gerd Petermann wrote: Hi all, I've coded a quick hack which seems to improve the ratios. On the other hand, I don't see these large differences between smallest and largest img file. What part of the world should I try? Do you use the precomp-sea parameter in splitter? Gerd Date: Wed, 30 Apr 2014 14:59:32 +0200 From: extremecarver@gmail.com To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] mkgmap ToDo list if that doesn't seriously (more than 30-40%) slow down the splitter, I assume it would be much better... On 30.04.2014 14:06, Gerd Petermann wrote: Hi, if I got that right the number of nodes is not highly correlated to the img size, so the max-nodes value is not a good estimate. I assume the reason is that nodes which belong to roads produce a lot more bytes in the img file compared to nodes which are parts of shapes or other non-routable ways, not talking about nodes which are simply ignored by the style. So, a possible solution in splitter could be to parse the ways before reading the nodes and save all nodeids which belong to ways with highway=*. If these nodes are refered by more than one way with highway=* we assume that they will be routing nodes. These special nodes could be counted e.g. 10 times to give a better estimate. Gerd > Date: Wed, 30 Apr 2014 13:36:19 +0200 > From: osm@na1400.info > To: mkgmap-dev@lists.mkgmap.org.uk > Subject: Re: [mkgmap-dev] mkgmap ToDo list > > Multithreading the tile rendering for a single map is indeed a bit > difficult and I gave it up because you need to keep track which image > id's are already in use. Since I provide multiple maps the work-around > is running a few scripts parallel, which is also a crude form of > multithreading. > > The script language is PHP and it doesn't run on Windows without some > changes ('/' vs '\' in paths, 'rm -rf', that sort of stuff). Never tried it. > > To get a better optimum in file size, using the process I described > earlier, you could start off with a huge --max-nodes setting and then > 'search' for the highest --max-nodes that works for each specific area. > > On 30/04/2014 11:49, Felix Hartmann wrote: > > I would love if there was a possibility that you pass the used max-nodes > > value to mkgmap. > > When mkgmap is compiling the maps, then after the .img is created it > > should check > > a) did it crash due to too many max-nodes > > b) for me not important - but for others with very old GPS, etrex 10, > > ---> is tile bigger than X (usually 8) MB. > > > > if a) or b) true, then pass the file back to splitter and split with 60% > > of maxnodes - and compile the resulting .img files again. Should it fail > > again, use 40%, again 25%... Sometimes there are awful tiles, that need > > supersmall max-nodes till they compile, however lately (last 1-2 years) > > I never encountered them anymore. I think that happened rather due to a > > but in splitter/mgkmap that is fixed now. > > > > okay, you could also do this with a script, but it gets rather > > complicated to multithread it (you need to wait till mgkmap finished > > compiling all .img files - and run mkgmap first without address index to > > save time) and do some clever routines on making sure that the map id > > (e.g. 6340????.img) stay correct. Even more complicated to have > > consequent map id... > > > > For europe with a fixed max-node I get tiles from 1.9MB up to 18MB. > > That's factor 9 - so it's huge... > > If I could narrows that down easily to 8-18MB - without getting tiles > > crashing due to too high max-nodes values, that would be sweet. > > > > > > > > As for the scripts - would they run on windows too? - What programming > > language are they in? > > > > > > On 29.04.2014 21:39, osm@na1400.info wrote: > >> Oh, and ofcourse anyone interested can get my scripts, send an email. > >> They'll be on Github someday anyway. > >> > >> > >> On 2014-04-29 20:37, Gerd Petermann wrote: > >>> Hi Lambertus, > >>> > >>> okay, if I got that right you finally get *.img files with a size > >>> near (but below) 8MB, so maybe Henning can use that script, too. > >>> > >>> If you do that for e.g. Germany, how small is tpically the smallest > >>> *.img file ? > >>> Is it probably near 4 MB? > >>> > >>> Gerd > >>> > >>>> To: mkgmap-dev@lists.mkgmap.org.uk > >>>> Date: Tue, 29 Apr 2014 20:30:27 +0200 > >>>> From: osm@na1400.info > >>>> Subject: Re: [mkgmap-dev] mkgmap ToDo list > >>>> > >>>> These are the direct results from Splitter. The format is o5m, both > >>>> input as output. Splitter version is: r321. > >>>> > >>>> For this test I split the original source with --max-nodes=8000000. > >>> Then > >>>> I render the initial tiles, when the result is larger than 8MB it's > >>>> subsplit again with --max-nodes=(8000000-(attempt*100000)). The > >>> initial > >>>> source files are ~70MB (o5m) and after several subsplits the two > >>> *.img > >>>> are < 8MB. During this process --max-nodes has been reduced to e.g. > >>>> 7500000 and the source file is split up in two o5m files of about > >>> 37MB. > >>>> > >>>> I can upload an example source file and it's two subsplit siblings > >>> if > >>>> you want. > >>>> > >>>> > >>>> On 2014-04-29 19:38, GerdP wrote: > >>>> > Hi Lambertus, > >>>> > > >>>> > that's interesting. Are these the img file sizes or the osm file > >>> sizes? > >>>> > > >>>> > Gerd > >>>> > > >>>> > > >>>> > Lambertus wrote > >>>> >> Unfortunately I cannot confirm that. Below is a bit of logging > >>> from my > >>>> >> script: > >>>> >> Original: 97000020 (70551453), New: 0 (35684445), New: 1 > >>> (36852845) > >>>> >> Original: 97000001 (74621042), New: 0 (37522992), New: 1 > >>> (37222739) > >>>> >> Original: 97000002 (73391358), New: 0 (37679505), New: 1 > >>> (38098627) > >>>> >> Original: 97000003 (77862567), New: 0 (39075311), New: 1 > >>> (39261197) > >>>> >> > >>>> >> The original files above contain contour data, the filesize is > >>> between > >>>> >> brackets. As you can see both resulting file are approximately > >>> the > >>>> >> same > >>>> >> size. > >>>> >> > >>>> >> On 2014-04-29 15:39, Gerd Petermann wrote: > >>>> >>> Hi Lambertus, > >>>> >>> > >>>> >>> and I guess that even after this optimization you will > >>>> >>> see a factor 3 or higher between the largest tile and the > >>> smallest. > >>>> >>> Can you confirm that? > >>>> >>> > >>>> >>> Gerd > >>>> >>> > >>>> >>>> Date: Tue, 29 Apr 2014 15:32:38 +0200 > >>>> >>>> From: > >>>> > > >>>> >> osm@ > >>>> > > >>>> >>>> To: > >>>> > > >>>> >> mkgmap-dev@.org > >>>> > > >>>> >>>> Subject: Re: [mkgmap-dev] mkgmap ToDo list > >>>> >>>> > >>>> >>>> Num-tiles=x would indeed be better for this specific need. > >>>> >>>> > >>>> >>>> It is my experience that it regularly takes multiple calls to > >>>> >>> Splitter > >>>> >>>> to get 2+ sub-tiles when you reduce the max-nodes by 100k for > >>> each > >>>> >>>> sub-split attempt. This is what I currently do to get an > >>> optimum in > >>>> >>>> tile-size vs total number of tiles. > >>>> >>>> > >>>> >>>> > >>>> >>>> > >>>> >>>> On 29/04/2014 15:09, Gerd Petermann wrote: > >>>> >>>> > Hi Lambertus, > >>>> >>>> > > >>>> >>>> > that sounds like a possible change in splitter: > >>>> >>>> > Instead of specifying max-nodes you may specify --num-tiles=x > >>>> >>>> > and splitter will try to find a split that produces excactly > >>> x > >>>> >>> tiles > >>>> >>>> > which are not too narrow and have a node number which is not > >>>> >>>> > too far from the average (but still aligned to a multiple of > >>> map > >>>> >>> units > >>>> >>>> > as now). > >>>> >>>> > So, for your script that means you don't have to find the > >>>> >>> max-nodes > >>>> >>>> > value. > >>>> >>>> > > >>>> >>>> > I'll think about this again... > >>>> >>>> > > >>>> >>>> > Gerd > >>>> >>>> > > >>>> >>>> > > Date: Tue, 29 Apr 2014 14:59:36 +0200 > >>>> >>>> > > From: > >>>> > > >>>> >> osm@ > >>>> > > >>>> >>>> > > To: > >>>> > > >>>> >> mkgmap-dev@.org > >>>> > > >>>> >>>> > > Subject: Re: [mkgmap-dev] mkgmap ToDo list > >>>> >>>> > > > >>>> >>>> > > While this possibly can be solved in Splitter or Mkgmap, it > >>>> >>> could also > >>>> >>>> > > be solved by your build-script when you add a maximum tile > >>> size > >>>> >>> check > >>>> >>>> > > and re-split (with a lower number of max-nodes) until you > >>> get > >>>> >>> two or > >>>> >>>> > > more sub-tiles. Granted, this adds complexity to the script > >>> but > >>>> >>> it works > >>>> >>>> > > well for me. > >>>> >>>> > > > >>>> >>>> > > On 25/04/2014 21:54, Henning Scholland wrote: > >>>> >>>> > > > Hi Gerd, > >>>> >>>> > > > > >>>> >>>> > > > I would like to have img-tiles which have globally nearly > >>> the > >>>> >>> same > >>>> >>>> > > > filesize, so that they use the space of devices like > >>> eTrex 10. > >>>> >>>> > > > > >>>> >>>> > > > With my actual map I use globally the same value for > >>>> >>> max-nodes. But the > >>>> >>>> > > > size of the img-tiles differ more then factor 2. Eg. a > >>> tile in > >>>> >>> Germany > >>>> >>>> > > > is between 2 and 5 mb where a tile in China is about 10 > >>> mb. If > >>>> >>> I remove > >>>> >>>> > > > details, this difference will increase, because in > >>> Germany > >>>> >>> more objects > >>>> >>>> > > > will be removed from the img-tile then in China. > >>>> >>>> > > > > >>>> >>>> > > > Henning > >>>> >>>> > > > > >>>> >>>> > > > _______________________________________________ > >>>> >>>> > > > mkgmap-dev mailing list > >>>> >>>> > > > > >>>> > > >>>> >> mkgmap-dev@.org > >>>> > > >>>> >>>> > > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >>>> > > > >>>> >>>> > > _______________________________________________ > >>>> >>>> > > mkgmap-dev mailing list > >>>> >>>> > > > >>>> > > >>>> >> mkgmap-dev@.org > >>>> > > >>>> >>>> > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >>>> > > >>>> >>>> > > >>>> >>>> > _______________________________________________ > >>>> >>>> > mkgmap-dev mailing list > >>>> >>>> > > >>>> > > >>>> >> mkgmap-dev@.org > >>>> > > >>>> >>>> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >>>> > > >>>> >>>> > >>>> >>>> _______________________________________________ > >>>> >>>> mkgmap-dev mailing list > >>>> >>>> > >>>> > > >>>> >> mkgmap-dev@.org > >>>> > > >>>> >>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >>> > >>>> >>> _______________________________________________ > >>>> >>> mkgmap-dev mailing list > >>>> >>> > >>>> > > >>>> >> mkgmap-dev@.org > >>>> > > >>>> >>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >> _______________________________________________ > >>>> >> mkgmap-dev mailing list > >>>> > > >>>> >> mkgmap-dev@.org > >>>> > > >>>> >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > -- > >>>> > View this message in context: > >>>> > > >>> http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5804588.html > >>>> > Sent from the Mkgmap Development mailing list archive at > >>> Nabble.com. > >>>> > _______________________________________________ > >>>> > mkgmap-dev mailing list > >>>> > mkgmap-dev@lists.mkgmap.org.uk > >>>> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> _______________________________________________ > >>>> mkgmap-dev mailing list > >>>> mkgmap-dev@lists.mkgmap.org.uk > >>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>> > >>> _______________________________________________ > >>> mkgmap-dev mailing list > >>> mkgmap-dev@lists.mkgmap.org.uk > >>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >> _______________________________________________ > >> mkgmap-dev mailing list > >> mkgmap-dev@lists.mkgmap.org.uk > >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > > _______________________________________________ > mkgmap-dev mailing list > mkgmap-dev@lists.mkgmap.org.uk > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev -- keep on biking and discovering new trails Felix openmtbmap.org & www.velomap.org _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev -- keep on biking and discovering new trails Felix openmtbmap.org & www.velomap.org _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Is the 'just give me two tiles' option for splitter on the to-do list? I'd really appreciate such a function. On 05/04/2014 09:36 PM, Gerd Petermann wrote:
Hi Felix,
would using the precomp-sea parameter for splitter improve results? It's not that tiles with loads of sea actually failed.
good question. I've coded that parameter in splitter before we did some important changes in mkgmap. I think it is needed when you use a polygon file in combination with an imput file that doesn't cover the polygon. If the polygon covers a costline with a lot of islands, but the input file doesn't contain the data, splitter might create large tiles covering these "empty" areas. Later, in mkgmap, the empty areas are filled with the precompiled-sea data, and that can cause too large files.
BTW: The quick hack turned out to be useless. It worked better in some areas and worse in others. I'll look again at this later next week.
Gerd
------------------------------------------------------------------------ Date: Sun, 4 May 2014 21:21:59 +0200 From: extremecarver@gmail.com To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] mkgmap ToDo list
No - I use splitter like this - with maxnodes depending on area - I usually just reduced it after failures to compile a tile.: java -Xms4000m -Xmx10400m -jar c:\openmtbmap\splitter.jar --max-nodes=%maxnodes% --max-threads=8 --output=pbf --keep-complete --max-areas=1524 --geonames-file=cities5000 --description=%country% --mapid=%FID%0000 c:\OpenMTBMap\osmpbf_geofabrik\%country%-latest.osm.pbf would using the precomp-sea parameter for splitter improve results? It's not that tiles with loads of sea actually failed.
On 03.05.2014 09:00, Gerd Petermann wrote:
Hi all,
I've coded a quick hack which seems to improve the ratios. On the other hand, I don't see these large differences between smallest and largest img file. What part of the world should I try? Do you use the precomp-sea parameter in splitter?
Gerd
------------------------------------------------------------------------ Date: Wed, 30 Apr 2014 14:59:32 +0200 From: extremecarver@gmail.com <mailto:extremecarver@gmail.com> To: mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> Subject: Re: [mkgmap-dev] mkgmap ToDo list
if that doesn't seriously (more than 30-40%) slow down the splitter, I assume it would be much better... On 30.04.2014 14:06, Gerd Petermann wrote:
Hi,
if I got that right the number of nodes is not highly correlated to the img size, so the max-nodes value is not a good estimate.
I assume the reason is that nodes which belong to roads produce a lot more bytes in the img file compared to nodes which are parts of shapes or other non-routable ways, not talking about nodes which are simply ignored by the style.
So, a possible solution in splitter could be to parse the ways before reading the nodes and save all nodeids which belong to ways with highway=*. If these nodes are refered by more than one way with highway=* we assume that they will be routing nodes. These special nodes could be counted e.g. 10 times to give a better estimate.
Gerd
> Date: Wed, 30 Apr 2014 13:36:19 +0200 > From: osm@na1400.info <mailto:osm@na1400.info> > To: mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> > Subject: Re: [mkgmap-dev] mkgmap ToDo list > > Multithreading the tile rendering for a single map is indeed a bit > difficult and I gave it up because you need to keep track which image > id's are already in use. Since I provide multiple maps the work-around > is running a few scripts parallel, which is also a crude form of > multithreading. > > The script language is PHP and it doesn't run on Windows without some > changes ('/' vs '\' in paths, 'rm -rf', that sort of stuff). Never tried it. > > To get a better optimum in file size, using the process I described > earlier, you could start off with a huge --max-nodes setting and then > 'search' for the highest --max-nodes that works for each specific area. > > On 30/04/2014 11:49, Felix Hartmann wrote: > > I would love if there was a possibility that you pass the used max-nodes > > value to mkgmap. > > When mkgmap is compiling the maps, then after the .img is created it > > should check > > a) did it crash due to too many max-nodes > > b) for me not important - but for others with very old GPS, etrex 10, > > ---> is tile bigger than X (usually 8) MB. > > > > if a) or b) true, then pass the file back to splitter and split with 60% > > of maxnodes - and compile the resulting .img files again. Should it fail > > again, use 40%, again 25%... Sometimes there are awful tiles, that need > > supersmall max-nodes till they compile, however lately (last 1-2 years) > > I never encountered them anymore. I think that happened rather due to a > > but in splitter/mgkmap that is fixed now. > > > > okay, you could also do this with a script, but it gets rather > > complicated to multithread it (you need to wait till mgkmap finished > > compiling all .img files - and run mkgmap first without address index to > > save time) and do some clever routines on making sure that the map id > > (e.g. 6340????.img) stay correct. Even more complicated to have > > consequent map id... > > > > For europe with a fixed max-node I get tiles from 1.9MB up to 18MB. > > That's factor 9 - so it's huge... > > If I could narrows that down easily to 8-18MB - without getting tiles > > crashing due to too high max-nodes values, that would be sweet. > > > > > > > > As for the scripts - would they run on windows too? - What programming > > language are they in? > > > > > > On 29.04.2014 21:39, osm@na1400.info <mailto:osm@na1400.info> wrote: > >> Oh, and ofcourse anyone interested can get my scripts, send an email. > >> They'll be on Github someday anyway. > >> > >> > >> On 2014-04-29 20:37, Gerd Petermann wrote: > >>> Hi Lambertus, > >>> > >>> okay, if I got that right you finally get *.img files with a size > >>> near (but below) 8MB, so maybe Henning can use that script, too. > >>> > >>> If you do that for e.g. Germany, how small is tpically the smallest > >>> *.img file ? > >>> Is it probably near 4 MB? > >>> > >>> Gerd > >>> > >>>> To: mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> > >>>> Date: Tue, 29 Apr 2014 20:30:27 +0200 > >>>> From: osm@na1400.info <mailto:osm@na1400.info> > >>>> Subject: Re: [mkgmap-dev] mkgmap ToDo list > >>>> > >>>> These are the direct results from Splitter. The format is o5m, both > >>>> input as output. Splitter version is: r321. > >>>> > >>>> For this test I split the original source with --max-nodes=8000000. > >>> Then > >>>> I render the initial tiles, when the result is larger than 8MB it's > >>>> subsplit again with --max-nodes=(8000000-(attempt*100000)). The > >>> initial > >>>> source files are ~70MB (o5m) and after several subsplits the two > >>> *.img > >>>> are < 8MB. During this process --max-nodes has been reduced to e.g. > >>>> 7500000 and the source file is split up in two o5m files of about > >>> 37MB. > >>>> > >>>> I can upload an example source file and it's two subsplit siblings > >>> if > >>>> you want. > >>>> > >>>> > >>>> On 2014-04-29 19:38, GerdP wrote: > >>>> > Hi Lambertus, > >>>> > > >>>> > that's interesting. Are these the img file sizes or the osm file > >>> sizes? > >>>> > > >>>> > Gerd > >>>> > > >>>> > > >>>> > Lambertus wrote > >>>> >> Unfortunately I cannot confirm that. Below is a bit of logging > >>> from my > >>>> >> script: > >>>> >> Original: 97000020 (70551453), New: 0 (35684445), New: 1 > >>> (36852845) > >>>> >> Original: 97000001 (74621042), New: 0 (37522992), New: 1 > >>> (37222739) > >>>> >> Original: 97000002 (73391358), New: 0 (37679505), New: 1 > >>> (38098627) > >>>> >> Original: 97000003 (77862567), New: 0 (39075311), New: 1 > >>> (39261197) > >>>> >> > >>>> >> The original files above contain contour data, the filesize is > >>> between > >>>> >> brackets. As you can see both resulting file are approximately > >>> the > >>>> >> same > >>>> >> size. > >>>> >> > >>>> >> On 2014-04-29 15:39, Gerd Petermann wrote: > >>>> >>> Hi Lambertus, > >>>> >>> > >>>> >>> and I guess that even after this optimization you will > >>>> >>> see a factor 3 or higher between the largest tile and the > >>> smallest. > >>>> >>> Can you confirm that? > >>>> >>> > >>>> >>> Gerd > >>>> >>> > >>>> >>>> Date: Tue, 29 Apr 2014 15:32:38 +0200 > >>>> >>>> From: > >>>> > > >>>> >> osm@ > >>>> > > >>>> >>>> To: > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >>>> Subject: Re: [mkgmap-dev] mkgmap ToDo list > >>>> >>>> > >>>> >>>> Num-tiles=x would indeed be better for this specific need. > >>>> >>>> > >>>> >>>> It is my experience that it regularly takes multiple calls to > >>>> >>> Splitter > >>>> >>>> to get 2+ sub-tiles when you reduce the max-nodes by 100k for > >>> each > >>>> >>>> sub-split attempt. This is what I currently do to get an > >>> optimum in > >>>> >>>> tile-size vs total number of tiles. > >>>> >>>> > >>>> >>>> > >>>> >>>> > >>>> >>>> On 29/04/2014 15:09, Gerd Petermann wrote: > >>>> >>>> > Hi Lambertus, > >>>> >>>> > > >>>> >>>> > that sounds like a possible change in splitter: > >>>> >>>> > Instead of specifying max-nodes you may specify --num-tiles=x > >>>> >>>> > and splitter will try to find a split that produces excactly > >>> x > >>>> >>> tiles > >>>> >>>> > which are not too narrow and have a node number which is not > >>>> >>>> > too far from the average (but still aligned to a multiple of > >>> map > >>>> >>> units > >>>> >>>> > as now). > >>>> >>>> > So, for your script that means you don't have to find the > >>>> >>> max-nodes > >>>> >>>> > value. > >>>> >>>> > > >>>> >>>> > I'll think about this again... > >>>> >>>> > > >>>> >>>> > Gerd > >>>> >>>> > > >>>> >>>> > > Date: Tue, 29 Apr 2014 14:59:36 +0200 > >>>> >>>> > > From: > >>>> > > >>>> >> osm@ > >>>> > > >>>> >>>> > > To: > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >>>> > > Subject: Re: [mkgmap-dev] mkgmap ToDo list > >>>> >>>> > > > >>>> >>>> > > While this possibly can be solved in Splitter or Mkgmap, it > >>>> >>> could also > >>>> >>>> > > be solved by your build-script when you add a maximum tile > >>> size > >>>> >>> check > >>>> >>>> > > and re-split (with a lower number of max-nodes) until you > >>> get > >>>> >>> two or > >>>> >>>> > > more sub-tiles. Granted, this adds complexity to the script > >>> but > >>>> >>> it works > >>>> >>>> > > well for me. > >>>> >>>> > > > >>>> >>>> > > On 25/04/2014 21:54, Henning Scholland wrote: > >>>> >>>> > > > Hi Gerd, > >>>> >>>> > > > > >>>> >>>> > > > I would like to have img-tiles which have globally nearly > >>> the > >>>> >>> same > >>>> >>>> > > > filesize, so that they use the space of devices like > >>> eTrex 10. > >>>> >>>> > > > > >>>> >>>> > > > With my actual map I use globally the same value for > >>>> >>> max-nodes. But the > >>>> >>>> > > > size of the img-tiles differ more then factor 2. Eg. a > >>> tile in > >>>> >>> Germany > >>>> >>>> > > > is between 2 and 5 mb where a tile in China is about 10 > >>> mb. If > >>>> >>> I remove > >>>> >>>> > > > details, this difference will increase, because in > >>> Germany > >>>> >>> more objects > >>>> >>>> > > > will be removed from the img-tile then in China. > >>>> >>>> > > > > >>>> >>>> > > > Henning > >>>> >>>> > > > > >>>> >>>> > > > _______________________________________________ > >>>> >>>> > > > mkgmap-dev mailing list > >>>> >>>> > > > > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >>>> > > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >>>> > > > >>>> >>>> > > _______________________________________________ > >>>> >>>> > > mkgmap-dev mailing list > >>>> >>>> > > > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >>>> > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >>>> > > >>>> >>>> > > >>>> >>>> > _______________________________________________ > >>>> >>>> > mkgmap-dev mailing list > >>>> >>>> > > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >>>> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >>>> > > >>>> >>>> > >>>> >>>> _______________________________________________ > >>>> >>>> mkgmap-dev mailing list > >>>> >>>> > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >>> > >>>> >>> _______________________________________________ > >>>> >>> mkgmap-dev mailing list > >>>> >>> > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >> _______________________________________________ > >>>> >> mkgmap-dev mailing list > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > -- > >>>> > View this message in context: > >>>> > > >>> http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5804588.html > >>>> > Sent from the Mkgmap Development mailing list archive at > >>> Nabble.com. > >>>> > _______________________________________________ > >>>> > mkgmap-dev mailing list > >>>> > mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> > >>>> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> _______________________________________________ > >>>> mkgmap-dev mailing list > >>>> mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> > >>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>> > >>> _______________________________________________ > >>> mkgmap-dev mailing list > >>> mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> > >>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >> _______________________________________________ > >> mkgmap-dev mailing list > >> mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> > >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > > _______________________________________________ > mkgmap-dev mailing list > mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- keep on biking and discovering new trails
Felix openmtbmap.org &www.velomap.org <http://www.velomap.org>
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- keep on biking and discovering new trails
Felix openmtbmap.org &www.velomap.org <http://www.velomap.org>
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Lambertus, yes, I want to code that tomorrow. I prefer to make it "give me n tiles", but if that turns out to be difficult I try the simple version. I can think of scenarios where it is not possible to create exactly n tiles. Gerd Date: Sun, 4 May 2014 22:08:32 +0200 From: osm@na1400.info To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] mkgmap ToDo list Is the 'just give me two tiles' option for splitter on the to-do list? I'd really appreciate such a function. On 05/04/2014 09:36 PM, Gerd Petermann wrote: Hi Felix, > would using the precomp-sea parameter for splitter improve results? It's not that tiles with loads of sea actually failed. good question. I've coded that parameter in splitter before we did some important changes in mkgmap. I think it is needed when you use a polygon file in combination with an imput file that doesn't cover the polygon. If the polygon covers a costline with a lot of islands, but the input file doesn't contain the data, splitter might create large tiles covering these "empty" areas. Later, in mkgmap, the empty areas are filled with the precompiled-sea data, and that can cause too large files. BTW: The quick hack turned out to be useless. It worked better in some areas and worse in others. I'll look again at this later next week. Gerd Date: Sun, 4 May 2014 21:21:59 +0200 From: extremecarver@gmail.com To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] mkgmap ToDo list No - I use splitter like this - with maxnodes depending on area - I usually just reduced it after failures to compile a tile.: java -Xms4000m -Xmx10400m -jar c:\openmtbmap\splitter.jar --max-nodes=%maxnodes% --max-threads=8 --output=pbf --keep-complete --max-areas=1524 --geonames-file=cities5000 --description=%country% --mapid=%FID%0000 c:\OpenMTBMap\osmpbf_geofabrik\%country%-latest.osm.pbf would using the precomp-sea parameter for splitter improve results? It's not that tiles with loads of sea actually failed. On 03.05.2014 09:00, Gerd Petermann wrote: Hi all, I've coded a quick hack which seems to improve the ratios. On the other hand, I don't see these large differences between smallest and largest img file. What part of the world should I try? Do you use the precomp-sea parameter in splitter? Gerd Date: Wed, 30 Apr 2014 14:59:32 +0200 From: extremecarver@gmail.com To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] mkgmap ToDo list if that doesn't seriously (more than 30-40%) slow down the splitter, I assume it would be much better... On 30.04.2014 14:06, Gerd Petermann wrote: Hi, if I got that right the number of nodes is not highly correlated to the img size, so the max-nodes value is not a good estimate. I assume the reason is that nodes which belong to roads produce a lot more bytes in the img file compared to nodes which are parts of shapes or other non-routable ways, not talking about nodes which are simply ignored by the style. So, a possible solution in splitter could be to parse the ways before reading the nodes and save all nodeids which belong to ways with highway=*. If these nodes are refered by more than one way with highway=* we assume that they will be routing nodes. These special nodes could be counted e.g. 10 times to give a better estimate. Gerd > Date: Wed, 30 Apr 2014 13:36:19 +0200 > From: osm@na1400.info > To: mkgmap-dev@lists.mkgmap.org.uk > Subject: Re: [mkgmap-dev] mkgmap ToDo list > > Multithreading the tile rendering for a single map is indeed a bit > difficult and I gave it up because you need to keep track which image > id's are already in use. Since I provide multiple maps the work-around > is running a few scripts parallel, which is also a crude form of > multithreading. > > The script language is PHP and it doesn't run on Windows without some > changes ('/' vs '\' in paths, 'rm -rf', that sort of stuff). Never tried it. > > To get a better optimum in file size, using the process I described > earlier, you could start off with a huge --max-nodes setting and then > 'search' for the highest --max-nodes that works for each specific area. > > On 30/04/2014 11:49, Felix Hartmann wrote: > > I would love if there was a possibility that you pass the used max-nodes > > value to mkgmap. > > When mkgmap is compiling the maps, then after the .img is created it > > should check > > a) did it crash due to too many max-nodes > > b) for me not important - but for others with very old GPS, etrex 10, > > ---> is tile bigger than X (usually 8) MB. > > > > if a) or b) true, then pass the file back to splitter and split with 60% > > of maxnodes - and compile the resulting .img files again. Should it fail > > again, use 40%, again 25%... Sometimes there are awful tiles, that need > > supersmall max-nodes till they compile, however lately (last 1-2 years) > > I never encountered them anymore. I think that happened rather due to a > > but in splitter/mgkmap that is fixed now. > > > > okay, you could also do this with a script, but it gets rather > > complicated to multithread it (you need to wait till mgkmap finished > > compiling all .img files - and run mkgmap first without address index to > > save time) and do some clever routines on making sure that the map id > > (e.g. 6340????.img) stay correct. Even more complicated to have > > consequent map id... > > > > For europe with a fixed max-node I get tiles from 1.9MB up to 18MB. > > That's factor 9 - so it's huge... > > If I could narrows that down easily to 8-18MB - without getting tiles > > crashing due to too high max-nodes values, that would be sweet. > > > > > > > > As for the scripts - would they run on windows too? - What programming > > language are they in? > > > > > > On 29.04.2014 21:39, osm@na1400.info wrote: > >> Oh, and ofcourse anyone interested can get my scripts, send an email. > >> They'll be on Github someday anyway. > >> > >> > >> On 2014-04-29 20:37, Gerd Petermann wrote: > >>> Hi Lambertus, > >>> > >>> okay, if I got that right you finally get *.img files with a size > >>> near (but below) 8MB, so maybe Henning can use that script, too. > >>> > >>> If you do that for e.g. Germany, how small is tpically the smallest > >>> *.img file ? > >>> Is it probably near 4 MB? > >>> > >>> Gerd > >>> > >>>> To: mkgmap-dev@lists.mkgmap.org.uk > >>>> Date: Tue, 29 Apr 2014 20:30:27 +0200 > >>>> From: osm@na1400.info > >>>> Subject: Re: [mkgmap-dev] mkgmap ToDo list > >>>> > >>>> These are the direct results from Splitter. The format is o5m, both > >>>> input as output. Splitter version is: r321. > >>>> > >>>> For this test I split the original source with --max-nodes=8000000. > >>> Then > >>>> I render the initial tiles, when the result is larger than 8MB it's > >>>> subsplit again with --max-nodes=(8000000-(attempt*100000)). The > >>> initial > >>>> source files are ~70MB (o5m) and after several subsplits the two > >>> *.img > >>>> are < 8MB. During this process --max-nodes has been reduced to e.g. > >>>> 7500000 and the source file is split up in two o5m files of about > >>> 37MB. > >>>> > >>>> I can upload an example source file and it's two subsplit siblings > >>> if > >>>> you want. > >>>> > >>>> > >>>> On 2014-04-29 19:38, GerdP wrote: > >>>> > Hi Lambertus, > >>>> > > >>>> > that's interesting. Are these the img file sizes or the osm file > >>> sizes? > >>>> > > >>>> > Gerd > >>>> > > >>>> > > >>>> > Lambertus wrote > >>>> >> Unfortunately I cannot confirm that. Below is a bit of logging > >>> from my > >>>> >> script: > >>>> >> Original: 97000020 (70551453), New: 0 (35684445), New: 1 > >>> (36852845) > >>>> >> Original: 97000001 (74621042), New: 0 (37522992), New: 1 > >>> (37222739) > >>>> >> Original: 97000002 (73391358), New: 0 (37679505), New: 1 > >>> (38098627) > >>>> >> Original: 97000003 (77862567), New: 0 (39075311), New: 1 > >>> (39261197) > >>>> >> > >>>> >> The original files above contain contour data, the filesize is > >>> between > >>>> >> brackets. As you can see both resulting file are approximately > >>> the > >>>> >> same > >>>> >> size. > >>>> >> > >>>> >> On 2014-04-29 15:39, Gerd Petermann wrote: > >>>> >>> Hi Lambertus, > >>>> >>> > >>>> >>> and I guess that even after this optimization you will > >>>> >>> see a factor 3 or higher between the largest tile and the > >>> smallest. > >>>> >>> Can you confirm that? > >>>> >>> > >>>> >>> Gerd > >>>> >>> > >>>> >>>> Date: Tue, 29 Apr 2014 15:32:38 +0200 > >>>> >>>> From: > >>>> > > >>>> >> osm@ > >>>> > > >>>> >>>> To: > >>>> > > >>>> >> mkgmap-dev@.org > >>>> > > >>>> >>>> Subject: Re: [mkgmap-dev] mkgmap ToDo list > >>>> >>>> > >>>> >>>> Num-tiles=x would indeed be better for this specific need. > >>>> >>>> > >>>> >>>> It is my experience that it regularly takes multiple calls to > >>>> >>> Splitter > >>>> >>>> to get 2+ sub-tiles when you reduce the max-nodes by 100k for > >>> each > >>>> >>>> sub-split attempt. This is what I currently do to get an > >>> optimum in > >>>> >>>> tile-size vs total number of tiles. > >>>> >>>> > >>>> >>>> > >>>> >>>> > >>>> >>>> On 29/04/2014 15:09, Gerd Petermann wrote: > >>>> >>>> > Hi Lambertus, > >>>> >>>> > > >>>> >>>> > that sounds like a possible change in splitter: > >>>> >>>> > Instead of specifying max-nodes you may specify --num-tiles=x > >>>> >>>> > and splitter will try to find a split that produces excactly > >>> x > >>>> >>> tiles > >>>> >>>> > which are not too narrow and have a node number which is not > >>>> >>>> > too far from the average (but still aligned to a multiple of > >>> map > >>>> >>> units > >>>> >>>> > as now). > >>>> >>>> > So, for your script that means you don't have to find the > >>>> >>> max-nodes > >>>> >>>> > value. > >>>> >>>> > > >>>> >>>> > I'll think about this again... > >>>> >>>> > > >>>> >>>> > Gerd > >>>> >>>> > > >>>> >>>> > > Date: Tue, 29 Apr 2014 14:59:36 +0200 > >>>> >>>> > > From: > >>>> > > >>>> >> osm@ > >>>> > > >>>> >>>> > > To: > >>>> > > >>>> >> mkgmap-dev@.org > >>>> > > >>>> >>>> > > Subject: Re: [mkgmap-dev] mkgmap ToDo list > >>>> >>>> > > > >>>> >>>> > > While this possibly can be solved in Splitter or Mkgmap, it > >>>> >>> could also > >>>> >>>> > > be solved by your build-script when you add a maximum tile > >>> size > >>>> >>> check > >>>> >>>> > > and re-split (with a lower number of max-nodes) until you > >>> get > >>>> >>> two or > >>>> >>>> > > more sub-tiles. Granted, this adds complexity to the script > >>> but > >>>> >>> it works > >>>> >>>> > > well for me. > >>>> >>>> > > > >>>> >>>> > > On 25/04/2014 21:54, Henning Scholland wrote: > >>>> >>>> > > > Hi Gerd, > >>>> >>>> > > > > >>>> >>>> > > > I would like to have img-tiles which have globally nearly > >>> the > >>>> >>> same > >>>> >>>> > > > filesize, so that they use the space of devices like > >>> eTrex 10. > >>>> >>>> > > > > >>>> >>>> > > > With my actual map I use globally the same value for > >>>> >>> max-nodes. But the > >>>> >>>> > > > size of the img-tiles differ more then factor 2. Eg. a > >>> tile in > >>>> >>> Germany > >>>> >>>> > > > is between 2 and 5 mb where a tile in China is about 10 > >>> mb. If > >>>> >>> I remove > >>>> >>>> > > > details, this difference will increase, because in > >>> Germany > >>>> >>> more objects > >>>> >>>> > > > will be removed from the img-tile then in China. > >>>> >>>> > > > > >>>> >>>> > > > Henning > >>>> >>>> > > > > >>>> >>>> > > > _______________________________________________ > >>>> >>>> > > > mkgmap-dev mailing list > >>>> >>>> > > > > >>>> > > >>>> >> mkgmap-dev@.org > >>>> > > >>>> >>>> > > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >>>> > > > >>>> >>>> > > _______________________________________________ > >>>> >>>> > > mkgmap-dev mailing list > >>>> >>>> > > > >>>> > > >>>> >> mkgmap-dev@.org > >>>> > > >>>> >>>> > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >>>> > > >>>> >>>> > > >>>> >>>> > _______________________________________________ > >>>> >>>> > mkgmap-dev mailing list > >>>> >>>> > > >>>> > > >>>> >> mkgmap-dev@.org > >>>> > > >>>> >>>> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >>>> > > >>>> >>>> > >>>> >>>> _______________________________________________ > >>>> >>>> mkgmap-dev mailing list > >>>> >>>> > >>>> > > >>>> >> mkgmap-dev@.org > >>>> > > >>>> >>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >>> > >>>> >>> _______________________________________________ > >>>> >>> mkgmap-dev mailing list > >>>> >>> > >>>> > > >>>> >> mkgmap-dev@.org > >>>> > > >>>> >>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >> _______________________________________________ > >>>> >> mkgmap-dev mailing list > >>>> > > >>>> >> mkgmap-dev@.org > >>>> > > >>>> >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > -- > >>>> > View this message in context: > >>>> > > >>> http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5804588.html > >>>> > Sent from the Mkgmap Development mailing list archive at > >>> Nabble.com. > >>>> > _______________________________________________ > >>>> > mkgmap-dev mailing list > >>>> > mkgmap-dev@lists.mkgmap.org.uk > >>>> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> _______________________________________________ > >>>> mkgmap-dev mailing list > >>>> mkgmap-dev@lists.mkgmap.org.uk > >>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>> > >>> _______________________________________________ > >>> mkgmap-dev mailing list > >>> mkgmap-dev@lists.mkgmap.org.uk > >>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >> _______________________________________________ > >> mkgmap-dev mailing list > >> mkgmap-dev@lists.mkgmap.org.uk > >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > > _______________________________________________ > mkgmap-dev mailing list > mkgmap-dev@lists.mkgmap.org.uk > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev -- keep on biking and discovering new trails Felix openmtbmap.org & www.velomap.org _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev -- keep on biking and discovering new trails Felix openmtbmap.org & www.velomap.org _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Great, many thanks. I'm currently trying to create as large as (automatically) possible contour tiles and a lot of time is put in subsplitting a tile again and again until two subtiles appear. This new option would save much time. On 05/04/2014 10:13 PM, Gerd Petermann wrote:
Hi Lambertus,
yes, I want to code that tomorrow. I prefer to make it "give me n tiles", but if that turns out to be difficult I try the simple version. I can think of scenarios where it is not possible to create exactly n tiles.
Gerd
------------------------------------------------------------------------ Date: Sun, 4 May 2014 22:08:32 +0200 From: osm@na1400.info To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] mkgmap ToDo list
Is the 'just give me two tiles' option for splitter on the to-do list? I'd really appreciate such a function.
On 05/04/2014 09:36 PM, Gerd Petermann wrote:
Hi Felix,
> would using the precomp-sea parameter for splitter improve results? It's not that tiles with loads of sea actually failed.
good question. I've coded that parameter in splitter before we did some important changes in mkgmap. I think it is needed when you use a polygon file in combination with an imput file that doesn't cover the polygon. If the polygon covers a costline with a lot of islands, but the input file doesn't contain the data, splitter might create large tiles covering these "empty" areas. Later, in mkgmap, the empty areas are filled with the precompiled-sea data, and that can cause too large files.
BTW: The quick hack turned out to be useless. It worked better in some areas and worse in others. I'll look again at this later next week.
Gerd
------------------------------------------------------------------------ Date: Sun, 4 May 2014 21:21:59 +0200 From: extremecarver@gmail.com <mailto:extremecarver@gmail.com> To: mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> Subject: Re: [mkgmap-dev] mkgmap ToDo list
No - I use splitter like this - with maxnodes depending on area - I usually just reduced it after failures to compile a tile.: java -Xms4000m -Xmx10400m -jar c:\openmtbmap\splitter.jar --max-nodes=%maxnodes% --max-threads=8 --output=pbf --keep-complete --max-areas=1524 --geonames-file=cities5000 --description=%country% --mapid=%FID%0000 c:\OpenMTBMap\osmpbf_geofabrik\%country%-latest.osm.pbf would using the precomp-sea parameter for splitter improve results? It's not that tiles with loads of sea actually failed.
On 03.05.2014 09:00, Gerd Petermann wrote:
Hi all,
I've coded a quick hack which seems to improve the ratios. On the other hand, I don't see these large differences between smallest and largest img file. What part of the world should I try? Do you use the precomp-sea parameter in splitter?
Gerd
------------------------------------------------------------------------ Date: Wed, 30 Apr 2014 14:59:32 +0200 From: extremecarver@gmail.com <mailto:extremecarver@gmail.com> To: mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> Subject: Re: [mkgmap-dev] mkgmap ToDo list
if that doesn't seriously (more than 30-40%) slow down the splitter, I assume it would be much better... On 30.04.2014 14:06, Gerd Petermann wrote:
Hi,
if I got that right the number of nodes is not highly correlated to the img size, so the max-nodes value is not a good estimate.
I assume the reason is that nodes which belong to roads produce a lot more bytes in the img file compared to nodes which are parts of shapes or other non-routable ways, not talking about nodes which are simply ignored by the style.
So, a possible solution in splitter could be to parse the ways before reading the nodes and save all nodeids which belong to ways with highway=*. If these nodes are refered by more than one way with highway=* we assume that they will be routing nodes. These special nodes could be counted e.g. 10 times to give a better estimate.
Gerd
> Date: Wed, 30 Apr 2014 13:36:19 +0200 > From: osm@na1400.info <mailto:osm@na1400.info> > To: mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> > Subject: Re: [mkgmap-dev] mkgmap ToDo list > > Multithreading the tile rendering for a single map is indeed a bit > difficult and I gave it up because you need to keep track which image > id's are already in use. Since I provide multiple maps the work-around > is running a few scripts parallel, which is also a crude form of > multithreading. > > The script language is PHP and it doesn't run on Windows without some > changes ('/' vs '\' in paths, 'rm -rf', that sort of stuff). Never tried it. > > To get a better optimum in file size, using the process I described > earlier, you could start off with a huge --max-nodes setting and then > 'search' for the highest --max-nodes that works for each specific area. > > On 30/04/2014 11:49, Felix Hartmann wrote: > > I would love if there was a possibility that you pass the used max-nodes > > value to mkgmap. > > When mkgmap is compiling the maps, then after the .img is created it > > should check > > a) did it crash due to too many max-nodes > > b) for me not important - but for others with very old GPS, etrex 10, > > ---> is tile bigger than X (usually 8) MB. > > > > if a) or b) true, then pass the file back to splitter and split with 60% > > of maxnodes - and compile the resulting .img files again. Should it fail > > again, use 40%, again 25%... Sometimes there are awful tiles, that need > > supersmall max-nodes till they compile, however lately (last 1-2 years) > > I never encountered them anymore. I think that happened rather due to a > > but in splitter/mgkmap that is fixed now. > > > > okay, you could also do this with a script, but it gets rather > > complicated to multithread it (you need to wait till mgkmap finished > > compiling all .img files - and run mkgmap first without address index to > > save time) and do some clever routines on making sure that the map id > > (e.g. 6340????.img) stay correct. Even more complicated to have > > consequent map id... > > > > For europe with a fixed max-node I get tiles from 1.9MB up to 18MB. > > That's factor 9 - so it's huge... > > If I could narrows that down easily to 8-18MB - without getting tiles > > crashing due to too high max-nodes values, that would be sweet. > > > > > > > > As for the scripts - would they run on windows too? - What programming > > language are they in? > > > > > > On 29.04.2014 21:39, osm@na1400.info <mailto:osm@na1400.info> wrote: > >> Oh, and ofcourse anyone interested can get my scripts, send an email. > >> They'll be on Github someday anyway. > >> > >> > >> On 2014-04-29 20:37, Gerd Petermann wrote: > >>> Hi Lambertus, > >>> > >>> okay, if I got that right you finally get *.img files with a size > >>> near (but below) 8MB, so maybe Henning can use that script, too. > >>> > >>> If you do that for e.g. Germany, how small is tpically the smallest > >>> *.img file ? > >>> Is it probably near 4 MB? > >>> > >>> Gerd > >>> > >>>> To: mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> > >>>> Date: Tue, 29 Apr 2014 20:30:27 +0200 > >>>> From: osm@na1400.info <mailto:osm@na1400.info> > >>>> Subject: Re: [mkgmap-dev] mkgmap ToDo list > >>>> > >>>> These are the direct results from Splitter. The format is o5m, both > >>>> input as output. Splitter version is: r321. > >>>> > >>>> For this test I split the original source with --max-nodes=8000000. > >>> Then > >>>> I render the initial tiles, when the result is larger than 8MB it's > >>>> subsplit again with --max-nodes=(8000000-(attempt*100000)). The > >>> initial > >>>> source files are ~70MB (o5m) and after several subsplits the two > >>> *.img > >>>> are < 8MB. During this process --max-nodes has been reduced to e.g. > >>>> 7500000 and the source file is split up in two o5m files of about > >>> 37MB. > >>>> > >>>> I can upload an example source file and it's two subsplit siblings > >>> if > >>>> you want. > >>>> > >>>> > >>>> On 2014-04-29 19:38, GerdP wrote: > >>>> > Hi Lambertus, > >>>> > > >>>> > that's interesting. Are these the img file sizes or the osm file > >>> sizes? > >>>> > > >>>> > Gerd > >>>> > > >>>> > > >>>> > Lambertus wrote > >>>> >> Unfortunately I cannot confirm that. Below is a bit of logging > >>> from my > >>>> >> script: > >>>> >> Original: 97000020 (70551453), New: 0 (35684445), New: 1 > >>> (36852845) > >>>> >> Original: 97000001 (74621042), New: 0 (37522992), New: 1 > >>> (37222739) > >>>> >> Original: 97000002 (73391358), New: 0 (37679505), New: 1 > >>> (38098627) > >>>> >> Original: 97000003 (77862567), New: 0 (39075311), New: 1 > >>> (39261197) > >>>> >> > >>>> >> The original files above contain contour data, the filesize is > >>> between > >>>> >> brackets. As you can see both resulting file are approximately > >>> the > >>>> >> same > >>>> >> size. > >>>> >> > >>>> >> On 2014-04-29 15:39, Gerd Petermann wrote: > >>>> >>> Hi Lambertus, > >>>> >>> > >>>> >>> and I guess that even after this optimization you will > >>>> >>> see a factor 3 or higher between the largest tile and the > >>> smallest. > >>>> >>> Can you confirm that? > >>>> >>> > >>>> >>> Gerd > >>>> >>> > >>>> >>>> Date: Tue, 29 Apr 2014 15:32:38 +0200 > >>>> >>>> From: > >>>> > > >>>> >> osm@ > >>>> > > >>>> >>>> To: > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >>>> Subject: Re: [mkgmap-dev] mkgmap ToDo list > >>>> >>>> > >>>> >>>> Num-tiles=x would indeed be better for this specific need. > >>>> >>>> > >>>> >>>> It is my experience that it regularly takes multiple calls to > >>>> >>> Splitter > >>>> >>>> to get 2+ sub-tiles when you reduce the max-nodes by 100k for > >>> each > >>>> >>>> sub-split attempt. This is what I currently do to get an > >>> optimum in > >>>> >>>> tile-size vs total number of tiles. > >>>> >>>> > >>>> >>>> > >>>> >>>> > >>>> >>>> On 29/04/2014 15:09, Gerd Petermann wrote: > >>>> >>>> > Hi Lambertus, > >>>> >>>> > > >>>> >>>> > that sounds like a possible change in splitter: > >>>> >>>> > Instead of specifying max-nodes you may specify --num-tiles=x > >>>> >>>> > and splitter will try to find a split that produces excactly > >>> x > >>>> >>> tiles > >>>> >>>> > which are not too narrow and have a node number which is not > >>>> >>>> > too far from the average (but still aligned to a multiple of > >>> map > >>>> >>> units > >>>> >>>> > as now). > >>>> >>>> > So, for your script that means you don't have to find the > >>>> >>> max-nodes > >>>> >>>> > value. > >>>> >>>> > > >>>> >>>> > I'll think about this again... > >>>> >>>> > > >>>> >>>> > Gerd > >>>> >>>> > > >>>> >>>> > > Date: Tue, 29 Apr 2014 14:59:36 +0200 > >>>> >>>> > > From: > >>>> > > >>>> >> osm@ > >>>> > > >>>> >>>> > > To: > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >>>> > > Subject: Re: [mkgmap-dev] mkgmap ToDo list > >>>> >>>> > > > >>>> >>>> > > While this possibly can be solved in Splitter or Mkgmap, it > >>>> >>> could also > >>>> >>>> > > be solved by your build-script when you add a maximum tile > >>> size > >>>> >>> check > >>>> >>>> > > and re-split (with a lower number of max-nodes) until you > >>> get > >>>> >>> two or > >>>> >>>> > > more sub-tiles. Granted, this adds complexity to the script > >>> but > >>>> >>> it works > >>>> >>>> > > well for me. > >>>> >>>> > > > >>>> >>>> > > On 25/04/2014 21:54, Henning Scholland wrote: > >>>> >>>> > > > Hi Gerd, > >>>> >>>> > > > > >>>> >>>> > > > I would like to have img-tiles which have globally nearly > >>> the > >>>> >>> same > >>>> >>>> > > > filesize, so that they use the space of devices like > >>> eTrex 10. > >>>> >>>> > > > > >>>> >>>> > > > With my actual map I use globally the same value for > >>>> >>> max-nodes. But the > >>>> >>>> > > > size of the img-tiles differ more then factor 2. Eg. a > >>> tile in > >>>> >>> Germany > >>>> >>>> > > > is between 2 and 5 mb where a tile in China is about 10 > >>> mb. If > >>>> >>> I remove > >>>> >>>> > > > details, this difference will increase, because in > >>> Germany > >>>> >>> more objects > >>>> >>>> > > > will be removed from the img-tile then in China. > >>>> >>>> > > > > >>>> >>>> > > > Henning > >>>> >>>> > > > > >>>> >>>> > > > _______________________________________________ > >>>> >>>> > > > mkgmap-dev mailing list > >>>> >>>> > > > > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >>>> > > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >>>> > > > >>>> >>>> > > _______________________________________________ > >>>> >>>> > > mkgmap-dev mailing list > >>>> >>>> > > > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >>>> > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >>>> > > >>>> >>>> > > >>>> >>>> > _______________________________________________ > >>>> >>>> > mkgmap-dev mailing list > >>>> >>>> > > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >>>> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >>>> > > >>>> >>>> > >>>> >>>> _______________________________________________ > >>>> >>>> mkgmap-dev mailing list > >>>> >>>> > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >>> > >>>> >>> _______________________________________________ > >>>> >>> mkgmap-dev mailing list > >>>> >>> > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >> _______________________________________________ > >>>> >> mkgmap-dev mailing list > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > -- > >>>> > View this message in context: > >>>> > > >>> http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5804588.html > >>>> > Sent from the Mkgmap Development mailing list archive at > >>> Nabble.com. > >>>> > _______________________________________________ > >>>> > mkgmap-dev mailing list > >>>> > mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> > >>>> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> _______________________________________________ > >>>> mkgmap-dev mailing list > >>>> mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> > >>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>> > >>> _______________________________________________ > >>> mkgmap-dev mailing list > >>> mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> > >>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >> _______________________________________________ > >> mkgmap-dev mailing list > >> mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> > >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > > _______________________________________________ > mkgmap-dev mailing list > mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- keep on biking and discovering new trails
Felix openmtbmap.org &www.velomap.org <http://www.velomap.org>
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- keep on biking and discovering new trails
Felix openmtbmap.org &www.velomap.org <http://www.velomap.org>
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Lambertus, attached is a patch that implements the new option: --num-tiles A target value that is used when no split-file is given. Splitting is done so that the given number of tiles is produced. The max-nodes value is ignored if this option is given. A compiled binary is here: http://files.mkgmap.org.uk/download/206/splitter.jar Please let me know if it works as expected. Gerd Date: Sun, 4 May 2014 22:20:20 +0200 From: osm@na1400.info To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] mkgmap ToDo list Great, many thanks. I'm currently trying to create as large as (automatically) possible contour tiles and a lot of time is put in subsplitting a tile again and again until two subtiles appear. This new option would save much time. On 05/04/2014 10:13 PM, Gerd Petermann wrote: Hi Lambertus, yes, I want to code that tomorrow. I prefer to make it "give me n tiles", but if that turns out to be difficult I try the simple version. I can think of scenarios where it is not possible to create exactly n tiles. Gerd Date: Sun, 4 May 2014 22:08:32 +0200 From: osm@na1400.info To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] mkgmap ToDo list Is the 'just give me two tiles' option for splitter on the to-do list? I'd really appreciate such a function. On 05/04/2014 09:36 PM, Gerd Petermann wrote: Hi Felix, > would using the precomp-sea parameter for splitter improve results? It's not that tiles with loads of sea actually failed. good question. I've coded that parameter in splitter before we did some important changes in mkgmap. I think it is needed when you use a polygon file in combination with an imput file that doesn't cover the polygon. If the polygon covers a costline with a lot of islands, but the input file doesn't contain the data, splitter might create large tiles covering these "empty" areas. Later, in mkgmap, the empty areas are filled with the precompiled-sea data, and that can cause too large files. BTW: The quick hack turned out to be useless. It worked better in some areas and worse in others. I'll look again at this later next week. Gerd Date: Sun, 4 May 2014 21:21:59 +0200 From: extremecarver@gmail.com To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] mkgmap ToDo list No - I use splitter like this - with maxnodes depending on area - I usually just reduced it after failures to compile a tile.: java -Xms4000m -Xmx10400m -jar c:\openmtbmap\splitter.jar --max-nodes=%maxnodes% --max-threads=8 --output=pbf --keep-complete --max-areas=1524 --geonames-file=cities5000 --description=%country% --mapid=%FID%0000 c:\OpenMTBMap\osmpbf_geofabrik\%country%-latest.osm.pbf would using the precomp-sea parameter for splitter improve results? It's not that tiles with loads of sea actually failed. On 03.05.2014 09:00, Gerd Petermann wrote: Hi all, I've coded a quick hack which seems to improve the ratios. On the other hand, I don't see these large differences between smallest and largest img file. What part of the world should I try? Do you use the precomp-sea parameter in splitter? Gerd Date: Wed, 30 Apr 2014 14:59:32 +0200 From: extremecarver@gmail.com To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] mkgmap ToDo list if that doesn't seriously (more than 30-40%) slow down the splitter, I assume it would be much better... On 30.04.2014 14:06, Gerd Petermann wrote: Hi, if I got that right the number of nodes is not highly correlated to the img size, so the max-nodes value is not a good estimate. I assume the reason is that nodes which belong to roads produce a lot more bytes in the img file compared to nodes which are parts of shapes or other non-routable ways, not talking about nodes which are simply ignored by the style. So, a possible solution in splitter could be to parse the ways before reading the nodes and save all nodeids which belong to ways with highway=*. If these nodes are refered by more than one way with highway=* we assume that they will be routing nodes. These special nodes could be counted e.g. 10 times to give a better estimate. Gerd > Date: Wed, 30 Apr 2014 13:36:19 +0200 > From: osm@na1400.info > To: mkgmap-dev@lists.mkgmap.org.uk > Subject: Re: [mkgmap-dev] mkgmap ToDo list > > Multithreading the tile rendering for a single map is indeed a bit > difficult and I gave it up because you need to keep track which image > id's are already in use. Since I provide multiple maps the work-around > is running a few scripts parallel, which is also a crude form of > multithreading. > > The script language is PHP and it doesn't run on Windows without some > changes ('/' vs '\' in paths, 'rm -rf', that sort of stuff). Never tried it. > > To get a better optimum in file size, using the process I described > earlier, you could start off with a huge --max-nodes setting and then > 'search' for the highest --max-nodes that works for each specific area. > > On 30/04/2014 11:49, Felix Hartmann wrote: > > I would love if there was a possibility that you pass the used max-nodes > > value to mkgmap. > > When mkgmap is compiling the maps, then after the .img is created it > > should check > > a) did it crash due to too many max-nodes > > b) for me not important - but for others with very old GPS, etrex 10, > > ---> is tile bigger than X (usually 8) MB. > > > > if a) or b) true, then pass the file back to splitter and split with 60% > > of maxnodes - and compile the resulting .img files again. Should it fail > > again, use 40%, again 25%... Sometimes there are awful tiles, that need > > supersmall max-nodes till they compile, however lately (last 1-2 years) > > I never encountered them anymore. I think that happened rather due to a > > but in splitter/mgkmap that is fixed now. > > > > okay, you could also do this with a script, but it gets rather > > complicated to multithread it (you need to wait till mgkmap finished > > compiling all .img files - and run mkgmap first without address index to > > save time) and do some clever routines on making sure that the map id > > (e.g. 6340????.img) stay correct. Even more complicated to have > > consequent map id... > > > > For europe with a fixed max-node I get tiles from 1.9MB up to 18MB. > > That's factor 9 - so it's huge... > > If I could narrows that down easily to 8-18MB - without getting tiles > > crashing due to too high max-nodes values, that would be sweet. > > > > > > > > As for the scripts - would they run on windows too? - What programming > > language are they in? > > > > > > On 29.04.2014 21:39, osm@na1400.info wrote: > >> Oh, and ofcourse anyone interested can get my scripts, send an email. > >> They'll be on Github someday anyway. > >> > >> > >> On 2014-04-29 20:37, Gerd Petermann wrote: > >>> Hi Lambertus, > >>> > >>> okay, if I got that right you finally get *.img files with a size > >>> near (but below) 8MB, so maybe Henning can use that script, too. > >>> > >>> If you do that for e.g. Germany, how small is tpically the smallest > >>> *.img file ? > >>> Is it probably near 4 MB? > >>> > >>> Gerd > >>> > >>>> To: mkgmap-dev@lists.mkgmap.org.uk > >>>> Date: Tue, 29 Apr 2014 20:30:27 +0200 > >>>> From: osm@na1400.info > >>>> Subject: Re: [mkgmap-dev] mkgmap ToDo list > >>>> > >>>> These are the direct results from Splitter. The format is o5m, both > >>>> input as output. Splitter version is: r321. > >>>> > >>>> For this test I split the original source with --max-nodes=8000000. > >>> Then > >>>> I render the initial tiles, when the result is larger than 8MB it's > >>>> subsplit again with --max-nodes=(8000000-(attempt*100000)). The > >>> initial > >>>> source files are ~70MB (o5m) and after several subsplits the two > >>> *.img > >>>> are < 8MB. During this process --max-nodes has been reduced to e.g. > >>>> 7500000 and the source file is split up in two o5m files of about > >>> 37MB. > >>>> > >>>> I can upload an example source file and it's two subsplit siblings > >>> if > >>>> you want. > >>>> > >>>> > >>>> On 2014-04-29 19:38, GerdP wrote: > >>>> > Hi Lambertus, > >>>> > > >>>> > that's interesting. Are these the img file sizes or the osm file > >>> sizes? > >>>> > > >>>> > Gerd > >>>> > > >>>> > > >>>> > Lambertus wrote > >>>> >> Unfortunately I cannot confirm that. Below is a bit of logging > >>> from my > >>>> >> script: > >>>> >> Original: 97000020 (70551453), New: 0 (35684445), New: 1 > >>> (36852845) > >>>> >> Original: 97000001 (74621042), New: 0 (37522992), New: 1 > >>> (37222739) > >>>> >> Original: 97000002 (73391358), New: 0 (37679505), New: 1 > >>> (38098627) > >>>> >> Original: 97000003 (77862567), New: 0 (39075311), New: 1 > >>> (39261197) > >>>> >> > >>>> >> The original files above contain contour data, the filesize is > >>> between > >>>> >> brackets. As you can see both resulting file are approximately > >>> the > >>>> >> same > >>>> >> size. > >>>> >> > >>>> >> On 2014-04-29 15:39, Gerd Petermann wrote: > >>>> >>> Hi Lambertus, > >>>> >>> > >>>> >>> and I guess that even after this optimization you will > >>>> >>> see a factor 3 or higher between the largest tile and the > >>> smallest. > >>>> >>> Can you confirm that? > >>>> >>> > >>>> >>> Gerd > >>>> >>> > >>>> >>>> Date: Tue, 29 Apr 2014 15:32:38 +0200 > >>>> >>>> From: > >>>> > > >>>> >> osm@ > >>>> > > >>>> >>>> To: > >>>> > > >>>> >> mkgmap-dev@.org > >>>> > > >>>> >>>> Subject: Re: [mkgmap-dev] mkgmap ToDo list > >>>> >>>> > >>>> >>>> Num-tiles=x would indeed be better for this specific need. > >>>> >>>> > >>>> >>>> It is my experience that it regularly takes multiple calls to > >>>> >>> Splitter > >>>> >>>> to get 2+ sub-tiles when you reduce the max-nodes by 100k for > >>> each > >>>> >>>> sub-split attempt. This is what I currently do to get an > >>> optimum in > >>>> >>>> tile-size vs total number of tiles. > >>>> >>>> > >>>> >>>> > >>>> >>>> > >>>> >>>> On 29/04/2014 15:09, Gerd Petermann wrote: > >>>> >>>> > Hi Lambertus, > >>>> >>>> > > >>>> >>>> > that sounds like a possible change in splitter: > >>>> >>>> > Instead of specifying max-nodes you may specify --num-tiles=x > >>>> >>>> > and splitter will try to find a split that produces excactly > >>> x > >>>> >>> tiles > >>>> >>>> > which are not too narrow and have a node number which is not > >>>> >>>> > too far from the average (but still aligned to a multiple of > >>> map > >>>> >>> units > >>>> >>>> > as now). > >>>> >>>> > So, for your script that means you don't have to find the > >>>> >>> max-nodes > >>>> >>>> > value. > >>>> >>>> > > >>>> >>>> > I'll think about this again... > >>>> >>>> > > >>>> >>>> > Gerd > >>>> >>>> > > >>>> >>>> > > Date: Tue, 29 Apr 2014 14:59:36 +0200 > >>>> >>>> > > From: > >>>> > > >>>> >> osm@ > >>>> > > >>>> >>>> > > To: > >>>> > > >>>> >> mkgmap-dev@.org > >>>> > > >>>> >>>> > > Subject: Re: [mkgmap-dev] mkgmap ToDo list > >>>> >>>> > > > >>>> >>>> > > While this possibly can be solved in Splitter or Mkgmap, it > >>>> >>> could also > >>>> >>>> > > be solved by your build-script when you add a maximum tile > >>> size > >>>> >>> check > >>>> >>>> > > and re-split (with a lower number of max-nodes) until you > >>> get > >>>> >>> two or > >>>> >>>> > > more sub-tiles. Granted, this adds complexity to the script > >>> but > >>>> >>> it works > >>>> >>>> > > well for me. > >>>> >>>> > > > >>>> >>>> > > On 25/04/2014 21:54, Henning Scholland wrote: > >>>> >>>> > > > Hi Gerd, > >>>> >>>> > > > > >>>> >>>> > > > I would like to have img-tiles which have globally nearly > >>> the > >>>> >>> same > >>>> >>>> > > > filesize, so that they use the space of devices like > >>> eTrex 10. > >>>> >>>> > > > > >>>> >>>> > > > With my actual map I use globally the same value for > >>>> >>> max-nodes. But the > >>>> >>>> > > > size of the img-tiles differ more then factor 2. Eg. a > >>> tile in > >>>> >>> Germany > >>>> >>>> > > > is between 2 and 5 mb where a tile in China is about 10 > >>> mb. If > >>>> >>> I remove > >>>> >>>> > > > details, this difference will increase, because in > >>> Germany > >>>> >>> more objects > >>>> >>>> > > > will be removed from the img-tile then in China. > >>>> >>>> > > > > >>>> >>>> > > > Henning > >>>> >>>> > > > > >>>> >>>> > > > _______________________________________________ > >>>> >>>> > > > mkgmap-dev mailing list > >>>> >>>> > > > > >>>> > > >>>> >> mkgmap-dev@.org > >>>> > > >>>> >>>> > > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >>>> > > > >>>> >>>> > > _______________________________________________ > >>>> >>>> > > mkgmap-dev mailing list > >>>> >>>> > > > >>>> > > >>>> >> mkgmap-dev@.org > >>>> > > >>>> >>>> > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >>>> > > >>>> >>>> > > >>>> >>>> > _______________________________________________ > >>>> >>>> > mkgmap-dev mailing list > >>>> >>>> > > >>>> > > >>>> >> mkgmap-dev@.org > >>>> > > >>>> >>>> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >>>> > > >>>> >>>> > >>>> >>>> _______________________________________________ > >>>> >>>> mkgmap-dev mailing list > >>>> >>>> > >>>> > > >>>> >> mkgmap-dev@.org > >>>> > > >>>> >>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >>> > >>>> >>> _______________________________________________ > >>>> >>> mkgmap-dev mailing list > >>>> >>> > >>>> > > >>>> >> mkgmap-dev@.org > >>>> > > >>>> >>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >> _______________________________________________ > >>>> >> mkgmap-dev mailing list > >>>> > > >>>> >> mkgmap-dev@.org > >>>> > > >>>> >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > -- > >>>> > View this message in context: > >>>> > > >>> http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5804588.html > >>>> > Sent from the Mkgmap Development mailing list archive at > >>> Nabble.com. > >>>> > _______________________________________________ > >>>> > mkgmap-dev mailing list > >>>> > mkgmap-dev@lists.mkgmap.org.uk > >>>> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> _______________________________________________ > >>>> mkgmap-dev mailing list > >>>> mkgmap-dev@lists.mkgmap.org.uk > >>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>> > >>> _______________________________________________ > >>> mkgmap-dev mailing list > >>> mkgmap-dev@lists.mkgmap.org.uk > >>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >> _______________________________________________ > >> mkgmap-dev mailing list > >> mkgmap-dev@lists.mkgmap.org.uk > >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > > _______________________________________________ > mkgmap-dev mailing list > mkgmap-dev@lists.mkgmap.org.uk > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev -- keep on biking and discovering new trails Felix openmtbmap.org & www.velomap.org _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev -- keep on biking and discovering new trails Felix openmtbmap.org & www.velomap.org _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Well - I suppose that is only for continuing to split. Not like saying I want to split europe and --num-tiles=600 or? Anyhow that's very good in case mkgmap would pass back osm.pbf files to splitter that failed compiling. Because then it could pass forward - "split into 2 tiles" :-) Or better -there is no real change in the splitting algo behind I assume on counting max-nodes or whatever else.. On 05.05.2014 09:28, Gerd Petermann wrote:
Hi Lambertus,
attached is a patch that implements the new option: --num-tiles A target value that is used when no split-file is given. Splitting is done so that the given number of tiles is produced. The max-nodes value is ignored if this option is given. A compiled binary is here:
|http://files.mkgmap.org.uk/download/206/splitter.jar
Please let me know if it works as expected.
Gerd | ------------------------------------------------------------------------ Date: Sun, 4 May 2014 22:20:20 +0200 From: osm@na1400.info To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] mkgmap ToDo list
Great, many thanks. I'm currently trying to create as large as (automatically) possible contour tiles and a lot of time is put in subsplitting a tile again and again until two subtiles appear. This new option would save much time.
On 05/04/2014 10:13 PM, Gerd Petermann wrote:
Hi Lambertus,
yes, I want to code that tomorrow. I prefer to make it "give me n tiles", but if that turns out to be difficult I try the simple version. I can think of scenarios where it is not possible to create exactly n tiles.
Gerd
------------------------------------------------------------------------ Date: Sun, 4 May 2014 22:08:32 +0200 From: osm@na1400.info <mailto:osm@na1400.info> To: mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> Subject: Re: [mkgmap-dev] mkgmap ToDo list
Is the 'just give me two tiles' option for splitter on the to-do list? I'd really appreciate such a function.
On 05/04/2014 09:36 PM, Gerd Petermann wrote:
Hi Felix,
> would using the precomp-sea parameter for splitter improve results? It's not that tiles with loads of sea actually failed.
good question. I've coded that parameter in splitter before we did some important changes in mkgmap. I think it is needed when you use a polygon file in combination with an imput file that doesn't cover the polygon. If the polygon covers a costline with a lot of islands, but the input file doesn't contain the data, splitter might create large tiles covering these "empty" areas. Later, in mkgmap, the empty areas are filled with the precompiled-sea data, and that can cause too large files.
BTW: The quick hack turned out to be useless. It worked better in some areas and worse in others. I'll look again at this later next week.
Gerd
------------------------------------------------------------------------ Date: Sun, 4 May 2014 21:21:59 +0200 From: extremecarver@gmail.com <mailto:extremecarver@gmail.com> To: mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> Subject: Re: [mkgmap-dev] mkgmap ToDo list
No - I use splitter like this - with maxnodes depending on area - I usually just reduced it after failures to compile a tile.: java -Xms4000m -Xmx10400m -jar c:\openmtbmap\splitter.jar --max-nodes=%maxnodes% --max-threads=8 --output=pbf --keep-complete --max-areas=1524 --geonames-file=cities5000 --description=%country% --mapid=%FID%0000 c:\OpenMTBMap\osmpbf_geofabrik\%country%-latest.osm.pbf would using the precomp-sea parameter for splitter improve results? It's not that tiles with loads of sea actually failed.
On 03.05.2014 09:00, Gerd Petermann wrote:
Hi all,
I've coded a quick hack which seems to improve the ratios. On the other hand, I don't see these large differences between smallest and largest img file. What part of the world should I try? Do you use the precomp-sea parameter in splitter?
Gerd
------------------------------------------------------------------------ Date: Wed, 30 Apr 2014 14:59:32 +0200 From: extremecarver@gmail.com <mailto:extremecarver@gmail.com> To: mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> Subject: Re: [mkgmap-dev] mkgmap ToDo list
if that doesn't seriously (more than 30-40%) slow down the splitter, I assume it would be much better... On 30.04.2014 14:06, Gerd Petermann wrote:
Hi,
if I got that right the number of nodes is not highly correlated to the img size, so the max-nodes value is not a good estimate.
I assume the reason is that nodes which belong to roads produce a lot more bytes in the img file compared to nodes which are parts of shapes or other non-routable ways, not talking about nodes which are simply ignored by the style.
So, a possible solution in splitter could be to parse the ways before reading the nodes and save all nodeids which belong to ways with highway=*. If these nodes are refered by more than one way with highway=* we assume that they will be routing nodes. These special nodes could be counted e.g. 10 times to give a better estimate.
Gerd
> Date: Wed, 30 Apr 2014 13:36:19 +0200 > From: osm@na1400.info <mailto:osm@na1400.info> > To: mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> > Subject: Re: [mkgmap-dev] mkgmap ToDo list > > Multithreading the tile rendering for a single map is indeed a bit > difficult and I gave it up because you need to keep track which image > id's are already in use. Since I provide multiple maps the work-around > is running a few scripts parallel, which is also a crude form of > multithreading. > > The script language is PHP and it doesn't run on Windows without some > changes ('/' vs '\' in paths, 'rm -rf', that sort of stuff). Never tried it. > > To get a better optimum in file size, using the process I described > earlier, you could start off with a huge --max-nodes setting and then > 'search' for the highest --max-nodes that works for each specific area. > > On 30/04/2014 11:49, Felix Hartmann wrote: > > I would love if there was a possibility that you pass the used max-nodes > > value to mkgmap. > > When mkgmap is compiling the maps, then after the .img is created it > > should check > > a) did it crash due to too many max-nodes > > b) for me not important - but for others with very old GPS, etrex 10, > > ---> is tile bigger than X (usually 8) MB. > > > > if a) or b) true, then pass the file back to splitter and split with 60% > > of maxnodes - and compile the resulting .img files again. Should it fail > > again, use 40%, again 25%... Sometimes there are awful tiles, that need > > supersmall max-nodes till they compile, however lately (last 1-2 years) > > I never encountered them anymore. I think that happened rather due to a > > but in splitter/mgkmap that is fixed now. > > > > okay, you could also do this with a script, but it gets rather > > complicated to multithread it (you need to wait till mgkmap finished > > compiling all .img files - and run mkgmap first without address index to > > save time) and do some clever routines on making sure that the map id > > (e.g. 6340????.img) stay correct. Even more complicated to have > > consequent map id... > > > > For europe with a fixed max-node I get tiles from 1.9MB up to 18MB. > > That's factor 9 - so it's huge... > > If I could narrows that down easily to 8-18MB - without getting tiles > > crashing due to too high max-nodes values, that would be sweet. > > > > > > > > As for the scripts - would they run on windows too? - What programming > > language are they in? > > > > > > On 29.04.2014 21:39, osm@na1400.info <mailto:osm@na1400.info> wrote: > >> Oh, and ofcourse anyone interested can get my scripts, send an email. > >> They'll be on Github someday anyway. > >> > >> > >> On 2014-04-29 20:37, Gerd Petermann wrote: > >>> Hi Lambertus, > >>> > >>> okay, if I got that right you finally get *.img files with a size > >>> near (but below) 8MB, so maybe Henning can use that script, too. > >>> > >>> If you do that for e.g. Germany, how small is tpically the smallest > >>> *.img file ? > >>> Is it probably near 4 MB? > >>> > >>> Gerd > >>> > >>>> To: mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> > >>>> Date: Tue, 29 Apr 2014 20:30:27 +0200 > >>>> From: osm@na1400.info <mailto:osm@na1400.info> > >>>> Subject: Re: [mkgmap-dev] mkgmap ToDo list > >>>> > >>>> These are the direct results from Splitter. The format is o5m, both > >>>> input as output. Splitter version is: r321. > >>>> > >>>> For this test I split the original source with --max-nodes=8000000. > >>> Then > >>>> I render the initial tiles, when the result is larger than 8MB it's > >>>> subsplit again with --max-nodes=(8000000-(attempt*100000)). The > >>> initial > >>>> source files are ~70MB (o5m) and after several subsplits the two > >>> *.img > >>>> are < 8MB. During this process --max-nodes has been reduced to e.g. > >>>> 7500000 and the source file is split up in two o5m files of about > >>> 37MB. > >>>> > >>>> I can upload an example source file and it's two subsplit siblings > >>> if > >>>> you want. > >>>> > >>>> > >>>> On 2014-04-29 19:38, GerdP wrote: > >>>> > Hi Lambertus, > >>>> > > >>>> > that's interesting. Are these the img file sizes or the osm file > >>> sizes? > >>>> > > >>>> > Gerd > >>>> > > >>>> > > >>>> > Lambertus wrote > >>>> >> Unfortunately I cannot confirm that. Below is a bit of logging > >>> from my > >>>> >> script: > >>>> >> Original: 97000020 (70551453), New: 0 (35684445), New: 1 > >>> (36852845) > >>>> >> Original: 97000001 (74621042), New: 0 (37522992), New: 1 > >>> (37222739) > >>>> >> Original: 97000002 (73391358), New: 0 (37679505), New: 1 > >>> (38098627) > >>>> >> Original: 97000003 (77862567), New: 0 (39075311), New: 1 > >>> (39261197) > >>>> >> > >>>> >> The original files above contain contour data, the filesize is > >>> between > >>>> >> brackets. As you can see both resulting file are approximately > >>> the > >>>> >> same > >>>> >> size. > >>>> >> > >>>> >> On 2014-04-29 15:39, Gerd Petermann wrote: > >>>> >>> Hi Lambertus, > >>>> >>> > >>>> >>> and I guess that even after this optimization you will > >>>> >>> see a factor 3 or higher between the largest tile and the > >>> smallest. > >>>> >>> Can you confirm that? > >>>> >>> > >>>> >>> Gerd > >>>> >>> > >>>> >>>> Date: Tue, 29 Apr 2014 15:32:38 +0200 > >>>> >>>> From: > >>>> > > >>>> >> osm@ > >>>> > > >>>> >>>> To: > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >>>> Subject: Re: [mkgmap-dev] mkgmap ToDo list > >>>> >>>> > >>>> >>>> Num-tiles=x would indeed be better for this specific need. > >>>> >>>> > >>>> >>>> It is my experience that it regularly takes multiple calls to > >>>> >>> Splitter > >>>> >>>> to get 2+ sub-tiles when you reduce the max-nodes by 100k for > >>> each > >>>> >>>> sub-split attempt. This is what I currently do to get an > >>> optimum in > >>>> >>>> tile-size vs total number of tiles. > >>>> >>>> > >>>> >>>> > >>>> >>>> > >>>> >>>> On 29/04/2014 15:09, Gerd Petermann wrote: > >>>> >>>> > Hi Lambertus, > >>>> >>>> > > >>>> >>>> > that sounds like a possible change in splitter: > >>>> >>>> > Instead of specifying max-nodes you may specify --num-tiles=x > >>>> >>>> > and splitter will try to find a split that produces excactly > >>> x > >>>> >>> tiles > >>>> >>>> > which are not too narrow and have a node number which is not > >>>> >>>> > too far from the average (but still aligned to a multiple of > >>> map > >>>> >>> units > >>>> >>>> > as now). > >>>> >>>> > So, for your script that means you don't have to find the > >>>> >>> max-nodes > >>>> >>>> > value. > >>>> >>>> > > >>>> >>>> > I'll think about this again... > >>>> >>>> > > >>>> >>>> > Gerd > >>>> >>>> > > >>>> >>>> > > Date: Tue, 29 Apr 2014 14:59:36 +0200 > >>>> >>>> > > From: > >>>> > > >>>> >> osm@ > >>>> > > >>>> >>>> > > To: > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >>>> > > Subject: Re: [mkgmap-dev] mkgmap ToDo list > >>>> >>>> > > > >>>> >>>> > > While this possibly can be solved in Splitter or Mkgmap, it > >>>> >>> could also > >>>> >>>> > > be solved by your build-script when you add a maximum tile > >>> size > >>>> >>> check > >>>> >>>> > > and re-split (with a lower number of max-nodes) until you > >>> get > >>>> >>> two or > >>>> >>>> > > more sub-tiles. Granted, this adds complexity to the script > >>> but > >>>> >>> it works > >>>> >>>> > > well for me. > >>>> >>>> > > > >>>> >>>> > > On 25/04/2014 21:54, Henning Scholland wrote: > >>>> >>>> > > > Hi Gerd, > >>>> >>>> > > > > >>>> >>>> > > > I would like to have img-tiles which have globally nearly > >>> the > >>>> >>> same > >>>> >>>> > > > filesize, so that they use the space of devices like > >>> eTrex 10. > >>>> >>>> > > > > >>>> >>>> > > > With my actual map I use globally the same value for > >>>> >>> max-nodes. But the > >>>> >>>> > > > size of the img-tiles differ more then factor 2. Eg. a > >>> tile in > >>>> >>> Germany > >>>> >>>> > > > is between 2 and 5 mb where a tile in China is about 10 > >>> mb. If > >>>> >>> I remove > >>>> >>>> > > > details, this difference will increase, because in > >>> Germany > >>>> >>> more objects > >>>> >>>> > > > will be removed from the img-tile then in China. > >>>> >>>> > > > > >>>> >>>> > > > Henning > >>>> >>>> > > > > >>>> >>>> > > > _______________________________________________ > >>>> >>>> > > > mkgmap-dev mailing list > >>>> >>>> > > > > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >>>> > > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >>>> > > > >>>> >>>> > > _______________________________________________ > >>>> >>>> > > mkgmap-dev mailing list > >>>> >>>> > > > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >>>> > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >>>> > > >>>> >>>> > > >>>> >>>> > _______________________________________________ > >>>> >>>> > mkgmap-dev mailing list > >>>> >>>> > > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >>>> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >>>> > > >>>> >>>> > >>>> >>>> _______________________________________________ > >>>> >>>> mkgmap-dev mailing list > >>>> >>>> > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >>> > >>>> >>> _______________________________________________ > >>>> >>> mkgmap-dev mailing list > >>>> >>> > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >> _______________________________________________ > >>>> >> mkgmap-dev mailing list > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > -- > >>>> > View this message in context: > >>>> > > >>> http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5804588.html > >>>> > Sent from the Mkgmap Development mailing list archive at > >>> Nabble.com. > >>>> > _______________________________________________ > >>>> > mkgmap-dev mailing list > >>>> > mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> > >>>> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> _______________________________________________ > >>>> mkgmap-dev mailing list > >>>> mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> > >>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>> > >>> _______________________________________________ > >>> mkgmap-dev mailing list > >>> mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> > >>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >> _______________________________________________ > >> mkgmap-dev mailing list > >> mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> > >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > > _______________________________________________ > mkgmap-dev mailing list > mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- keep on biking and discovering new trails
Felix openmtbmap.org &www.velomap.org <http://www.velomap.org>
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- keep on biking and discovering new trails
Felix openmtbmap.org &www.velomap.org <http://www.velomap.org>
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- keep on biking and discovering new trails Felix openmtbmap.org & www.velomap.org
Hi Felix, I did not try with Europe, only with Finland, but I see no reason why it should not work. I'll see what I can do regarding the automatic call of splitter from mkgmap. Gerd Felix Hartmann-2 wrote
Well - I suppose that is only for continuing to split. Not like saying I want to split europe and --num-tiles=600 or? Anyhow that's very good in case mkgmap would pass back osm.pbf files to splitter that failed compiling. Because then it could pass forward - "split into 2 tiles" :-)
Or better -there is no real change in the splitting algo behind I assume on counting max-nodes or whatever else.. On 05.05.2014 09:28, Gerd Petermann wrote:
Hi Lambertus,
attached is a patch that implements the new option: --num-tiles A target value that is used when no split-file is given. Splitting is done so that the given number of tiles is produced. The max-nodes value is ignored if this option is given. A compiled binary is here:
|http://files.mkgmap.org.uk/download/206/splitter.jar
Please let me know if it works as expected.
Gerd | ------------------------------------------------------------------------ Date: Sun, 4 May 2014 22:20:20 +0200 From:
osm@
To:
mkgmap-dev@.org
Subject: Re: [mkgmap-dev] mkgmap ToDo list
Great, many thanks. I'm currently trying to create as large as (automatically) possible contour tiles and a lot of time is put in subsplitting a tile again and again until two subtiles appear. This new option would save much time.
On 05/04/2014 10:13 PM, Gerd Petermann wrote:
Hi Lambertus,
yes, I want to code that tomorrow. I prefer to make it "give me n tiles", but if that turns out to be difficult I try the simple version. I can think of scenarios where it is not possible to create exactly n tiles.
Gerd
------------------------------------------------------------------------ Date: Sun, 4 May 2014 22:08:32 +0200 From:
osm@
<mailto:
osm@
>
To:
mkgmap-dev@.org
<mailto:
mkgmap-dev@.org
>
Subject: Re: [mkgmap-dev] mkgmap ToDo list
Is the 'just give me two tiles' option for splitter on the to-do list? I'd really appreciate such a function.
On 05/04/2014 09:36 PM, Gerd Petermann wrote:
Hi Felix,
> would using the precomp-sea parameter for splitter improve results? It's not that tiles with loads of sea actually failed.
good question. I've coded that parameter in splitter before we did some important changes in mkgmap. I think it is needed when you use a polygon file in combination with an imput file that doesn't cover the polygon. If the polygon covers a costline with a lot of islands, but the input file doesn't contain the data, splitter might create large tiles covering these "empty" areas. Later, in mkgmap, the empty areas are filled with the precompiled-sea data, and that can cause too large files.
BTW: The quick hack turned out to be useless. It worked better in some areas and worse in others. I'll look again at this later next week.
Gerd
------------------------------------------------------------------------ Date: Sun, 4 May 2014 21:21:59 +0200 From:
extremecarver@
<mailto:
extremecarver@
>
To:
mkgmap-dev@.org
<mailto:
mkgmap-dev@.org
>
Subject: Re: [mkgmap-dev] mkgmap ToDo list
No - I use splitter like this - with maxnodes depending on area - I usually just reduced it after failures to compile a tile.: java -Xms4000m -Xmx10400m -jar c:\openmtbmap\splitter.jar --max-nodes=%maxnodes% --max-threads=8 --output=pbf --keep-complete --max-areas=1524 --geonames-file=cities5000 --description=%country% --mapid=%FID%0000 c:\OpenMTBMap\osmpbf_geofabrik\%country%-latest.osm.pbf would using the precomp-sea parameter for splitter improve results? It's not that tiles with loads of sea actually failed.
On 03.05.2014 09:00, Gerd Petermann wrote:
Hi all,
I've coded a quick hack which seems to improve the ratios. On the other hand, I don't see these large differences between smallest and largest img file. What part of the world should I try? Do you use the precomp-sea parameter in splitter?
Gerd
------------------------------------------------------------------------ Date: Wed, 30 Apr 2014 14:59:32 +0200 From:
extremecarver@
<mailto:
extremecarver@
>
To:
mkgmap-dev@.org
<mailto:
mkgmap-dev@.org
>
Subject: Re: [mkgmap-dev] mkgmap ToDo list
if that doesn't seriously (more than 30-40%) slow down the splitter, I assume it would be much better... On 30.04.2014 14:06, Gerd Petermann wrote:
Hi,
if I got that right the number of nodes is not highly correlated to the img size, so the max-nodes value is not a good estimate.
I assume the reason is that nodes which belong to roads produce a lot more bytes in the img file compared to nodes which are parts of shapes or other non-routable ways, not talking about nodes which are simply ignored by the style.
So, a possible solution in splitter could be to parse the ways before reading the nodes and save all nodeids which belong to ways with highway=*. If these nodes are refered by more than one way with highway=* we assume that they will be routing nodes. These special nodes could be counted e.g. 10 times to give a better estimate.
Gerd
> Date: Wed, 30 Apr 2014 13:36:19 +0200 > From:
osm@
<mailto:
osm@
>
> To:
mkgmap-dev@.org
<mailto:
mkgmap-dev@.org
>
> Subject: Re: [mkgmap-dev] mkgmap ToDo list > > Multithreading the tile rendering for a single map is indeed a bit > difficult and I gave it up because you need to keep track which image > id's are already in use. Since I provide multiple maps the work-around > is running a few scripts parallel, which is also a crude form of > multithreading. > > The script language is PHP and it doesn't run on Windows without some > changes ('/' vs '\' in paths, 'rm -rf', that sort of stuff). Never tried it. > > To get a better optimum in file size, using the process I described > earlier, you could start off with a huge --max-nodes setting and then > 'search' for the highest --max-nodes that works for each specific area. > > On 30/04/2014 11:49, Felix Hartmann wrote: > > I would love if there was a possibility that you pass the used max-nodes > > value to mkgmap. > > When mkgmap is compiling the maps, then after the .img is created it > > should check > > a) did it crash due to too many max-nodes > > b) for me not important - but for others with very old GPS, etrex 10, > > ---> is tile bigger than X (usually 8) MB. > > > > if a) or b) true, then pass the file back to splitter and split with 60% > > of maxnodes - and compile the resulting .img files again. Should it fail > > again, use 40%, again 25%... Sometimes there are awful tiles, that need > > supersmall max-nodes till they compile, however lately (last 1-2 years) > > I never encountered them anymore. I think that happened rather due to a > > but in splitter/mgkmap that is fixed now. > > > > okay, you could also do this with a script, but it gets rather > > complicated to multithread it (you need to wait till mgkmap finished > > compiling all .img files - and run mkgmap first without address index to > > save time) and do some clever routines on making sure that the map id > > (e.g. 6340????.img) stay correct. Even more complicated to have > > consequent map id... > > > > For europe with a fixed max-node I get tiles from 1.9MB up to 18MB. > > That's factor 9 - so it's huge... > > If I could narrows that down easily to 8-18MB - without getting tiles > > crashing due to too high max-nodes values, that would be sweet. > > > > > > > > As for the scripts - would they run on windows too? - What programming > > language are they in? > > > > > > On 29.04.2014 21:39,
osm@
<mailto:
osm@
> wrote:
> >> Oh, and ofcourse anyone interested can get my scripts, send an email. > >> They'll be on Github someday anyway. > >> > >> > >> On 2014-04-29 20:37, Gerd Petermann wrote: > >>> Hi Lambertus, > >>> > >>> okay, if I got that right you finally get *.img files with a size > >>> near (but below) 8MB, so maybe Henning can use that script, too. > >>> > >>> If you do that for e.g. Germany, how small is tpically the smallest > >>> *.img file ? > >>> Is it probably near 4 MB? > >>> > >>> Gerd > >>> > >>>> To:
mkgmap-dev@.org
<mailto:
mkgmap-dev@.org
>
> >>>> Date: Tue, 29 Apr 2014 20:30:27 +0200 > >>>> From:
osm@
<mailto:
osm@
>
> >>>> Subject: Re: [mkgmap-dev] mkgmap ToDo list > >>>> > >>>> These are the direct results from Splitter. The format is o5m, both > >>>> input as output. Splitter version is: r321. > >>>> > >>>> For this test I split the original source with --max-nodes=8000000. > >>> Then > >>>> I render the initial tiles, when the result is larger than 8MB it's > >>>> subsplit again with --max-nodes=(8000000-(attempt*100000)). The > >>> initial > >>>> source files are ~70MB (o5m) and after several subsplits the two > >>> *.img > >>>> are < 8MB. During this process --max-nodes has been reduced to e.g. > >>>> 7500000 and the source file is split up in two o5m files of about > >>> 37MB. > >>>> > >>>> I can upload an example source file and it's two subsplit siblings > >>> if > >>>> you want. > >>>> > >>>> > >>>> On 2014-04-29 19:38, GerdP wrote: > >>>> > Hi Lambertus, > >>>> > > >>>> > that's interesting. Are these the img file sizes or the osm file > >>> sizes? > >>>> > > >>>> > Gerd > >>>> > > >>>> > > >>>> > Lambertus wrote > >>>> >> Unfortunately I cannot confirm that. Below is a bit of logging > >>> from my > >>>> >> script: > >>>> >> Original: 97000020 (70551453), New: 0 (35684445), New: 1 > >>> (36852845) > >>>> >> Original: 97000001 (74621042), New: 0 (37522992), New: 1 > >>> (37222739) > >>>> >> Original: 97000002 (73391358), New: 0 (37679505), New: 1 > >>> (38098627) > >>>> >> Original: 97000003 (77862567), New: 0 (39075311), New: 1 > >>> (39261197) > >>>> >> > >>>> >> The original files above contain contour data, the filesize is > >>> between > >>>> >> brackets. As you can see both resulting file are approximately > >>> the > >>>> >> same > >>>> >> size. > >>>> >> > >>>> >> On 2014-04-29 15:39, Gerd Petermann wrote: > >>>> >>> Hi Lambertus, > >>>> >>> > >>>> >>> and I guess that even after this optimization you will > >>>> >>> see a factor 3 or higher between the largest tile and the > >>> smallest. > >>>> >>> Can you confirm that? > >>>> >>> > >>>> >>> Gerd > >>>> >>> > >>>> >>>> Date: Tue, 29 Apr 2014 15:32:38 +0200 > >>>> >>>> From: > >>>> > > >>>> >> osm@ > >>>> > > >>>> >>>> To: > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >>>> Subject: Re: [mkgmap-dev] mkgmap ToDo list > >>>> >>>> > >>>> >>>> Num-tiles=x would indeed be better for this specific need. > >>>> >>>> > >>>> >>>> It is my experience that it regularly takes multiple calls to > >>>> >>> Splitter > >>>> >>>> to get 2+ sub-tiles when you reduce the max-nodes by 100k for > >>> each > >>>> >>>> sub-split attempt. This is what I currently do to get an > >>> optimum in > >>>> >>>> tile-size vs total number of tiles. > >>>> >>>> > >>>> >>>> > >>>> >>>> > >>>> >>>> On 29/04/2014 15:09, Gerd Petermann wrote: > >>>> >>>> > Hi Lambertus, > >>>> >>>> > > >>>> >>>> > that sounds like a possible change in splitter: > >>>> >>>> > Instead of specifying max-nodes you may specify --num-tiles=x > >>>> >>>> > and splitter will try to find a split that produces excactly > >>> x > >>>> >>> tiles > >>>> >>>> > which are not too narrow and have a node number which is not > >>>> >>>> > too far from the average (but still aligned to a multiple of > >>> map > >>>> >>> units > >>>> >>>> > as now). > >>>> >>>> > So, for your script that means you don't have to find the > >>>> >>> max-nodes > >>>> >>>> > value. > >>>> >>>> > > >>>> >>>> > I'll think about this again... > >>>> >>>> > > >>>> >>>> > Gerd > >>>> >>>> > > >>>> >>>> > > Date: Tue, 29 Apr 2014 14:59:36 +0200 > >>>> >>>> > > From: > >>>> > > >>>> >> osm@ > >>>> > > >>>> >>>> > > To: > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >>>> > > Subject: Re: [mkgmap-dev] mkgmap ToDo list > >>>> >>>> > > > >>>> >>>> > > While this possibly can be solved in Splitter or Mkgmap, it > >>>> >>> could also > >>>> >>>> > > be solved by your build-script when you add a maximum tile > >>> size > >>>> >>> check > >>>> >>>> > > and re-split (with a lower number of max-nodes) until you > >>> get > >>>> >>> two or > >>>> >>>> > > more sub-tiles. Granted, this adds complexity to the script > >>> but > >>>> >>> it works > >>>> >>>> > > well for me. > >>>> >>>> > > > >>>> >>>> > > On 25/04/2014 21:54, Henning Scholland wrote: > >>>> >>>> > > > Hi Gerd, > >>>> >>>> > > > > >>>> >>>> > > > I would like to have img-tiles which have globally nearly > >>> the > >>>> >>> same > >>>> >>>> > > > filesize, so that they use the space of devices like > >>> eTrex 10. > >>>> >>>> > > > > >>>> >>>> > > > With my actual map I use globally the same value for > >>>> >>> max-nodes. But the > >>>> >>>> > > > size of the img-tiles differ more then factor 2. Eg. a > >>> tile in > >>>> >>> Germany > >>>> >>>> > > > is between 2 and 5 mb where a tile in China is about 10 > >>> mb. If > >>>> >>> I remove > >>>> >>>> > > > details, this difference will increase, because in > >>> Germany > >>>> >>> more objects > >>>> >>>> > > > will be removed from the img-tile then in China. > >>>> >>>> > > > > >>>> >>>> > > > Henning > >>>> >>>> > > > > >>>> >>>> > > > _______________________________________________ > >>>> >>>> > > > mkgmap-dev mailing list > >>>> >>>> > > > > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >>>> > > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >>>> > > > >>>> >>>> > > _______________________________________________ > >>>> >>>> > > mkgmap-dev mailing list > >>>> >>>> > > > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >>>> > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >>>> > > >>>> >>>> > > >>>> >>>> > _______________________________________________ > >>>> >>>> > mkgmap-dev mailing list > >>>> >>>> > > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >>>> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >>>> > > >>>> >>>> > >>>> >>>> _______________________________________________ > >>>> >>>> mkgmap-dev mailing list > >>>> >>>> > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >>> > >>>> >>> _______________________________________________ > >>>> >>> mkgmap-dev mailing list > >>>> >>> > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >> _______________________________________________ > >>>> >> mkgmap-dev mailing list > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > -- > >>>> > View this message in context: > >>>> > > >>>
http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5804588.html > >>>> > Sent from the Mkgmap Development mailing list archive at > >>> Nabble.com. > >>>> > _______________________________________________ > >>>> > mkgmap-dev mailing list > >>>> >
mkgmap-dev@.org
<mailto:
mkgmap-dev@.org
>
> >>>> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> _______________________________________________ > >>>> mkgmap-dev mailing list > >>>>
mkgmap-dev@.org
<mailto:
mkgmap-dev@.org
>
> >>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>> > >>> _______________________________________________ > >>> mkgmap-dev mailing list > >>>
mkgmap-dev@.org
<mailto:
mkgmap-dev@.org
>
> >>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >> _______________________________________________ > >> mkgmap-dev mailing list > >>
mkgmap-dev@.org
<mailto:
mkgmap-dev@.org
>
> >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > > _______________________________________________ > mkgmap-dev mailing list >
mkgmap-dev@.org
<mailto:
mkgmap-dev@.org
>
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
<mailto:
mkgmap-dev@.org
>
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- keep on biking and discovering new trails
Felix openmtbmap.org &www.velomap.org <http://www.velomap.org>;
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
<mailto:
mkgmap-dev@.org
>
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
<mailto:
mkgmap-dev@.org
>
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- keep on biking and discovering new trails
Felix openmtbmap.org &www.velomap.org <http://www.velomap.org>;
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
<mailto:
mkgmap-dev@.org
>
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
<mailto:
mkgmap-dev@.org
>
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
<mailto:
mkgmap-dev@.org
>
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
<mailto:
mkgmap-dev@.org
>
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
-- keep on biking and discovering new trails
Felix openmtbmap.org & www.velomap.org
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
-- View this message in context: http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5805221.html Sent from the Mkgmap Development mailing list archive at Nabble.com.
okay - I'll give it a try tomorrow (not much more time today). As for mkgmap calling splitter - I think the important bit is to hand over mkgmap the splitter "command" used. Maybe in a file that splitter writes after splitting? I don't think it's important that the splitted files are in perfect numeric order - but it would be great if mkgmap could keep numeric order without missing numbers.. About the tile limit: That's a tricky one - because there is no warning. The gps units simply stop reading in further map tiles after 2025 on most units, and 4000 on Oregon and some other modern units. So people wonder why the map doesn't show up even though it's activated - and the reason is then to be found in the tile limit... I'm not sure if only map display and routing is affected, or also POI search. The problematic thing is - it's at least for the user nearly impossible to know which tiles were read in, and which not (of course it's not random - but hard to find out the order that the device uses, so it's most likely many maps showing correctly, one partly, some not at all). It would be great if tiles were a bit bigger - but I would not use the mkgmap interaction to completly foregoe setting max-nodes or trying to get all maptiles 9-18MB in size. It would just be great if those very small tiles (1 to say 4MB for me) would not be so common anymore as like now where I need to work with very low max-nodes values. On 05.05.2014 13:49, GerdP wrote:
Hi Felix,
I did not try with Europe, only with Finland, but I see no reason why it should not work. I'll see what I can do regarding the automatic call of splitter from mkgmap.
Gerd
Felix Hartmann-2 wrote
Well - I suppose that is only for continuing to split. Not like saying I want to split europe and --num-tiles=600 or? Anyhow that's very good in case mkgmap would pass back osm.pbf files to splitter that failed compiling. Because then it could pass forward - "split into 2 tiles" :-)
Or better -there is no real change in the splitting algo behind I assume on counting max-nodes or whatever else.. On 05.05.2014 09:28, Gerd Petermann wrote:
Hi Lambertus,
attached is a patch that implements the new option: --num-tiles A target value that is used when no split-file is given. Splitting is done so that the given number of tiles is produced. The max-nodes value is ignored if this option is given. A compiled binary is here:
|http://files.mkgmap.org.uk/download/206/splitter.jar
Please let me know if it works as expected.
Gerd | ------------------------------------------------------------------------ Date: Sun, 4 May 2014 22:20:20 +0200 From: osm@ To: mkgmap-dev@.org Subject: Re: [mkgmap-dev] mkgmap ToDo list
Great, many thanks. I'm currently trying to create as large as (automatically) possible contour tiles and a lot of time is put in subsplitting a tile again and again until two subtiles appear. This new option would save much time.
On 05/04/2014 10:13 PM, Gerd Petermann wrote:
Hi Lambertus,
yes, I want to code that tomorrow. I prefer to make it "give me n tiles", but if that turns out to be difficult I try the simple version. I can think of scenarios where it is not possible to create exactly n tiles.
Gerd
------------------------------------------------------------------------ Date: Sun, 4 May 2014 22:08:32 +0200 From: osm@ <mailto: osm@ > To: mkgmap-dev@.org <mailto: mkgmap-dev@.org > Subject: Re: [mkgmap-dev] mkgmap ToDo list
Is the 'just give me two tiles' option for splitter on the to-do list? I'd really appreciate such a function.
On 05/04/2014 09:36 PM, Gerd Petermann wrote:
Hi Felix,
> would using the precomp-sea parameter for splitter improve results? It's not that tiles with loads of sea actually failed.
good question. I've coded that parameter in splitter before we did some important changes in mkgmap. I think it is needed when you use a polygon file in combination with an imput file that doesn't cover the polygon. If the polygon covers a costline with a lot of islands, but the input file doesn't contain the data, splitter might create large tiles covering these "empty" areas. Later, in mkgmap, the empty areas are filled with the precompiled-sea data, and that can cause too large files.
BTW: The quick hack turned out to be useless. It worked better in some areas and worse in others. I'll look again at this later next week.
Gerd
------------------------------------------------------------------------ Date: Sun, 4 May 2014 21:21:59 +0200 From: extremecarver@ <mailto: extremecarver@ > To: mkgmap-dev@.org <mailto: mkgmap-dev@.org > Subject: Re: [mkgmap-dev] mkgmap ToDo list
No - I use splitter like this - with maxnodes depending on area - I usually just reduced it after failures to compile a tile.: java -Xms4000m -Xmx10400m -jar c:\openmtbmap\splitter.jar --max-nodes=%maxnodes% --max-threads=8 --output=pbf --keep-complete --max-areas=1524 --geonames-file=cities5000 --description=%country% --mapid=%FID%0000 c:\OpenMTBMap\osmpbf_geofabrik\%country%-latest.osm.pbf would using the precomp-sea parameter for splitter improve results? It's not that tiles with loads of sea actually failed.
On 03.05.2014 09:00, Gerd Petermann wrote:
Hi all,
I've coded a quick hack which seems to improve the ratios. On the other hand, I don't see these large differences between smallest and largest img file. What part of the world should I try? Do you use the precomp-sea parameter in splitter?
Gerd
------------------------------------------------------------------------ Date: Wed, 30 Apr 2014 14:59:32 +0200 From: extremecarver@ <mailto: extremecarver@ > To: mkgmap-dev@.org <mailto: mkgmap-dev@.org > Subject: Re: [mkgmap-dev] mkgmap ToDo list
if that doesn't seriously (more than 30-40%) slow down the splitter, I assume it would be much better... On 30.04.2014 14:06, Gerd Petermann wrote:
Hi,
if I got that right the number of nodes is not highly correlated to the img size, so the max-nodes value is not a good estimate.
I assume the reason is that nodes which belong to roads produce a lot more bytes in the img file compared to nodes which are parts of shapes or other non-routable ways, not talking about nodes which are simply ignored by the style.
So, a possible solution in splitter could be to parse the ways before reading the nodes and save all nodeids which belong to ways with highway=*. If these nodes are refered by more than one way with highway=* we assume that they will be routing nodes. These special nodes could be counted e.g. 10 times to give a better estimate.
Gerd
> Date: Wed, 30 Apr 2014 13:36:19 +0200 > From: osm@ <mailto: osm@ > > To: mkgmap-dev@.org <mailto: mkgmap-dev@.org > > Subject: Re: [mkgmap-dev] mkgmap ToDo list > > Multithreading the tile rendering for a single map is indeed a bit > difficult and I gave it up because you need to keep track which image > id's are already in use. Since I provide multiple maps the work-around > is running a few scripts parallel, which is also a crude form of > multithreading. > > The script language is PHP and it doesn't run on Windows without some > changes ('/' vs '\' in paths, 'rm -rf', that sort of stuff). Never tried it. > > To get a better optimum in file size, using the process I described > earlier, you could start off with a huge --max-nodes setting and then > 'search' for the highest --max-nodes that works for each specific area. > > On 30/04/2014 11:49, Felix Hartmann wrote: > > I would love if there was a possibility that you pass the used max-nodes > > value to mkgmap. > > When mkgmap is compiling the maps, then after the .img is created it > > should check > > a) did it crash due to too many max-nodes > > b) for me not important - but for others with very old GPS, etrex 10, > > ---> is tile bigger than X (usually 8) MB. > > > > if a) or b) true, then pass the file back to splitter and split with 60% > > of maxnodes - and compile the resulting .img files again. Should it fail > > again, use 40%, again 25%... Sometimes there are awful tiles, that need > > supersmall max-nodes till they compile, however lately (last 1-2 years) > > I never encountered them anymore. I think that happened rather due to a > > but in splitter/mgkmap that is fixed now. > > > > okay, you could also do this with a script, but it gets rather > > complicated to multithread it (you need to wait till mgkmap finished > > compiling all .img files - and run mkgmap first without address index to > > save time) and do some clever routines on making sure that the map id > > (e.g. 6340????.img) stay correct. Even more complicated to have > > consequent map id... > > > > For europe with a fixed max-node I get tiles from 1.9MB up to 18MB. > > That's factor 9 - so it's huge... > > If I could narrows that down easily to 8-18MB - without getting tiles > > crashing due to too high max-nodes values, that would be sweet. > > > > > > > > As for the scripts - would they run on windows too? - What programming > > language are they in? > > > > > > On 29.04.2014 21:39, osm@ <mailto: osm@ > wrote: > >> Oh, and ofcourse anyone interested can get my scripts, send an email. > >> They'll be on Github someday anyway. > >> > >> > >> On 2014-04-29 20:37, Gerd Petermann wrote: > >>> Hi Lambertus, > >>> > >>> okay, if I got that right you finally get *.img files with a size > >>> near (but below) 8MB, so maybe Henning can use that script, too. > >>> > >>> If you do that for e.g. Germany, how small is tpically the smallest > >>> *.img file ? > >>> Is it probably near 4 MB? > >>> > >>> Gerd > >>> > >>>> To: mkgmap-dev@.org <mailto: mkgmap-dev@.org > > >>>> Date: Tue, 29 Apr 2014 20:30:27 +0200 > >>>> From: osm@ <mailto: osm@ > > >>>> Subject: Re: [mkgmap-dev] mkgmap ToDo list > >>>> > >>>> These are the direct results from Splitter. The format is o5m, both > >>>> input as output. Splitter version is: r321. > >>>> > >>>> For this test I split the original source with --max-nodes=8000000. > >>> Then > >>>> I render the initial tiles, when the result is larger than 8MB it's > >>>> subsplit again with --max-nodes=(8000000-(attempt*100000)). The > >>> initial > >>>> source files are ~70MB (o5m) and after several subsplits the two > >>> *.img > >>>> are < 8MB. During this process --max-nodes has been reduced to e.g. > >>>> 7500000 and the source file is split up in two o5m files of about > >>> 37MB. > >>>> > >>>> I can upload an example source file and it's two subsplit siblings > >>> if > >>>> you want. > >>>> > >>>> > >>>> On 2014-04-29 19:38, GerdP wrote: > >>>> > Hi Lambertus, > >>>> > > >>>> > that's interesting. Are these the img file sizes or the osm file > >>> sizes? > >>>> > > >>>> > Gerd > >>>> > > >>>> > > >>>> > Lambertus wrote > >>>> >> Unfortunately I cannot confirm that. Below is a bit of logging > >>> from my > >>>> >> script: > >>>> >> Original: 97000020 (70551453), New: 0 (35684445), New: 1 > >>> (36852845) > >>>> >> Original: 97000001 (74621042), New: 0 (37522992), New: 1 > >>> (37222739) > >>>> >> Original: 97000002 (73391358), New: 0 (37679505), New: 1 > >>> (38098627) > >>>> >> Original: 97000003 (77862567), New: 0 (39075311), New: 1 > >>> (39261197) > >>>> >> > >>>> >> The original files above contain contour data, the filesize is > >>> between > >>>> >> brackets. As you can see both resulting file are approximately > >>> the > >>>> >> same > >>>> >> size. > >>>> >> > >>>> >> On 2014-04-29 15:39, Gerd Petermann wrote: > >>>> >>> Hi Lambertus, > >>>> >>> > >>>> >>> and I guess that even after this optimization you will > >>>> >>> see a factor 3 or higher between the largest tile and the > >>> smallest. > >>>> >>> Can you confirm that? > >>>> >>> > >>>> >>> Gerd > >>>> >>> > >>>> >>>> Date: Tue, 29 Apr 2014 15:32:38 +0200 > >>>> >>>> From: > >>>> > > >>>> >> osm@ > >>>> > > >>>> >>>> To: > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >>>> Subject: Re: [mkgmap-dev] mkgmap ToDo list > >>>> >>>> > >>>> >>>> Num-tiles=x would indeed be better for this specific need. > >>>> >>>> > >>>> >>>> It is my experience that it regularly takes multiple calls to > >>>> >>> Splitter > >>>> >>>> to get 2+ sub-tiles when you reduce the max-nodes by 100k for > >>> each > >>>> >>>> sub-split attempt. This is what I currently do to get an > >>> optimum in > >>>> >>>> tile-size vs total number of tiles. > >>>> >>>> > >>>> >>>> > >>>> >>>> > >>>> >>>> On 29/04/2014 15:09, Gerd Petermann wrote: > >>>> >>>> > Hi Lambertus, > >>>> >>>> > > >>>> >>>> > that sounds like a possible change in splitter: > >>>> >>>> > Instead of specifying max-nodes you may specify --num-tiles=x > >>>> >>>> > and splitter will try to find a split that produces excactly > >>> x > >>>> >>> tiles > >>>> >>>> > which are not too narrow and have a node number which is not > >>>> >>>> > too far from the average (but still aligned to a multiple of > >>> map > >>>> >>> units > >>>> >>>> > as now). > >>>> >>>> > So, for your script that means you don't have to find the > >>>> >>> max-nodes > >>>> >>>> > value. > >>>> >>>> > > >>>> >>>> > I'll think about this again... > >>>> >>>> > > >>>> >>>> > Gerd > >>>> >>>> > > >>>> >>>> > > Date: Tue, 29 Apr 2014 14:59:36 +0200 > >>>> >>>> > > From: > >>>> > > >>>> >> osm@ > >>>> > > >>>> >>>> > > To: > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >>>> > > Subject: Re: [mkgmap-dev] mkgmap ToDo list > >>>> >>>> > > > >>>> >>>> > > While this possibly can be solved in Splitter or Mkgmap, it > >>>> >>> could also > >>>> >>>> > > be solved by your build-script when you add a maximum tile > >>> size > >>>> >>> check > >>>> >>>> > > and re-split (with a lower number of max-nodes) until you > >>> get > >>>> >>> two or > >>>> >>>> > > more sub-tiles. Granted, this adds complexity to the script > >>> but > >>>> >>> it works > >>>> >>>> > > well for me. > >>>> >>>> > > > >>>> >>>> > > On 25/04/2014 21:54, Henning Scholland wrote: > >>>> >>>> > > > Hi Gerd, > >>>> >>>> > > > > >>>> >>>> > > > I would like to have img-tiles which have globally nearly > >>> the > >>>> >>> same > >>>> >>>> > > > filesize, so that they use the space of devices like > >>> eTrex 10. > >>>> >>>> > > > > >>>> >>>> > > > With my actual map I use globally the same value for > >>>> >>> max-nodes. But the > >>>> >>>> > > > size of the img-tiles differ more then factor 2. Eg. a > >>> tile in > >>>> >>> Germany > >>>> >>>> > > > is between 2 and 5 mb where a tile in China is about 10 > >>> mb. If > >>>> >>> I remove > >>>> >>>> > > > details, this difference will increase, because in > >>> Germany > >>>> >>> more objects > >>>> >>>> > > > will be removed from the img-tile then in China. > >>>> >>>> > > > > >>>> >>>> > > > Henning > >>>> >>>> > > > > >>>> >>>> > > > _______________________________________________ > >>>> >>>> > > > mkgmap-dev mailing list > >>>> >>>> > > > > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >>>> > > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >>>> > > > >>>> >>>> > > _______________________________________________ > >>>> >>>> > > mkgmap-dev mailing list > >>>> >>>> > > > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >>>> > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >>>> > > >>>> >>>> > > >>>> >>>> > _______________________________________________ > >>>> >>>> > mkgmap-dev mailing list > >>>> >>>> > > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >>>> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >>>> > > >>>> >>>> > >>>> >>>> _______________________________________________ > >>>> >>>> mkgmap-dev mailing list > >>>> >>>> > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >>> > >>>> >>> _______________________________________________ > >>>> >>> mkgmap-dev mailing list > >>>> >>> > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >> _______________________________________________ > >>>> >> mkgmap-dev mailing list > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > -- > >>>> > View this message in context: > >>>> > > >>>
http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5804588.html > >>>> > Sent from the Mkgmap Development mailing list archive at > >>> Nabble.com. > >>>> > _______________________________________________ > >>>> > mkgmap-dev mailing list > >>>> > mkgmap-dev@.org <mailto: mkgmap-dev@.org > > >>>> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> _______________________________________________ > >>>> mkgmap-dev mailing list > >>>> mkgmap-dev@.org <mailto: mkgmap-dev@.org > > >>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>> > >>> _______________________________________________ > >>> mkgmap-dev mailing list > >>> mkgmap-dev@.org <mailto: mkgmap-dev@.org > > >>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >> _______________________________________________ > >> mkgmap-dev mailing list > >> mkgmap-dev@.org <mailto: mkgmap-dev@.org > > >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > > _______________________________________________ > mkgmap-dev mailing list > mkgmap-dev@.org <mailto: mkgmap-dev@.org > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org <mailto: mkgmap-dev@.org >
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- keep on biking and discovering new trails
Felix openmtbmap.org &www.velomap.org <http://www.velomap.org>;
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
<mailto:
mkgmap-dev@.org >
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org <mailto: mkgmap-dev@.org >
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- keep on biking and discovering new trails
Felix openmtbmap.org &www.velomap.org <http://www.velomap.org>;
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
<mailto:
mkgmap-dev@.org >
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org <mailto: mkgmap-dev@.org >
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
<mailto:
mkgmap-dev@.org >
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org <mailto: mkgmap-dev@.org >
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev -- keep on biking and discovering new trails
Felix openmtbmap.org & www.velomap.org
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@.org http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- View this message in context: http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5805221.html Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- keep on biking and discovering new trails Felix openmtbmap.org & www.velomap.org
Hi Felix, thanks for the info regarding tile limits. I think there is not much we can do in the program(s) to avoid that. Reg. splitter called from mkgmap: I think calling splitter from mkgmap is possible, but not simple. 1) We have to make sure that splitter.jar is available. 2) I can't simply call the Main routine of splitter, because the code contains many calls to System.exit() which would also end mkgmap. In other words, I have to call it creating a new JVM. 3) Reg. parameters for splitter: This is quite complex. We can't simply use the parameters of the initial splitter call, because e.g. no-trim , mapid, polygon-file,etc. have to be changed to make sure that the result still fits into the map. 4) Next problem: splitter output files like areas.list, *.kml, template.args What should happen with them ? In short: I fear this change, it has a lot of potential to mess up things and make the two programs more OS dependend :-( I still think the better alternative would be to collect some kind of statistic in mkgmap which is usable for splitter to weight the number of nodes in a given area. The process would then be: if no feedback file exits, create an empty one repeat - execute splitter with input file and feedback from mkgmap - execute mkgmap (which creates a new feedback file) until mkgmap finishes with rc 0 This is also not simple, as I don't know how to calculate the feedback. A first simple approach would be this: bbox + size of img file In the meantime I will try to find a better algo in splitter to avoid very small output files. The current algo is happy if the no tile has less then 1/3 of the --max-nodes value, if that can't be done, it accepts also smaller values. Gerd
Date: Mon, 5 May 2014 13:58:07 +0200 From: extremecarver@gmail.com To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] mkgmap ToDo list
okay - I'll give it a try tomorrow (not much more time today). As for mkgmap calling splitter - I think the important bit is to hand over mkgmap the splitter "command" used. Maybe in a file that splitter writes after splitting? I don't think it's important that the splitted files are in perfect numeric order - but it would be great if mkgmap could keep numeric order without missing numbers..
About the tile limit: That's a tricky one - because there is no warning. The gps units simply stop reading in further map tiles after 2025 on most units, and 4000 on Oregon and some other modern units. So people wonder why the map doesn't show up even though it's activated - and the reason is then to be found in the tile limit...
I'm not sure if only map display and routing is affected, or also POI search. The problematic thing is - it's at least for the user nearly impossible to know which tiles were read in, and which not (of course it's not random - but hard to find out the order that the device uses, so it's most likely many maps showing correctly, one partly, some not at all).
It would be great if tiles were a bit bigger - but I would not use the mkgmap interaction to completly foregoe setting max-nodes or trying to get all maptiles 9-18MB in size. It would just be great if those very small tiles (1 to say 4MB for me) would not be so common anymore as like now where I need to work with very low max-nodes values. On 05.05.2014 13:49, GerdP wrote:
Hi Felix,
I did not try with Europe, only with Finland, but I see no reason why it should not work. I'll see what I can do regarding the automatic call of splitter from mkgmap.
Gerd
Felix Hartmann-2 wrote
Well - I suppose that is only for continuing to split. Not like saying I want to split europe and --num-tiles=600 or? Anyhow that's very good in case mkgmap would pass back osm.pbf files to splitter that failed compiling. Because then it could pass forward - "split into 2 tiles" :-)
Or better -there is no real change in the splitting algo behind I assume on counting max-nodes or whatever else.. On 05.05.2014 09:28, Gerd Petermann wrote:
Hi Lambertus,
attached is a patch that implements the new option: --num-tiles A target value that is used when no split-file is given. Splitting is done so that the given number of tiles is produced. The max-nodes value is ignored if this option is given. A compiled binary is here:
|http://files.mkgmap.org.uk/download/206/splitter.jar
Please let me know if it works as expected.
Gerd | ------------------------------------------------------------------------ Date: Sun, 4 May 2014 22:20:20 +0200 From: osm@ To: mkgmap-dev@.org Subject: Re: [mkgmap-dev] mkgmap ToDo list
Great, many thanks. I'm currently trying to create as large as (automatically) possible contour tiles and a lot of time is put in subsplitting a tile again and again until two subtiles appear. This new option would save much time.
On 05/04/2014 10:13 PM, Gerd Petermann wrote:
Hi Lambertus,
yes, I want to code that tomorrow. I prefer to make it "give me n tiles", but if that turns out to be difficult I try the simple version. I can think of scenarios where it is not possible to create exactly n tiles.
Gerd
------------------------------------------------------------------------ Date: Sun, 4 May 2014 22:08:32 +0200 From: osm@ <mailto: osm@ > To: mkgmap-dev@.org <mailto: mkgmap-dev@.org > Subject: Re: [mkgmap-dev] mkgmap ToDo list
Is the 'just give me two tiles' option for splitter on the to-do list? I'd really appreciate such a function.
On 05/04/2014 09:36 PM, Gerd Petermann wrote:
Hi Felix,
> would using the precomp-sea parameter for splitter improve results? It's not that tiles with loads of sea actually failed.
good question. I've coded that parameter in splitter before we did some important changes in mkgmap. I think it is needed when you use a polygon file in combination with an imput file that doesn't cover the polygon. If the polygon covers a costline with a lot of islands, but the input file doesn't contain the data, splitter might create large tiles covering these "empty" areas. Later, in mkgmap, the empty areas are filled with the precompiled-sea data, and that can cause too large files.
BTW: The quick hack turned out to be useless. It worked better in some areas and worse in others. I'll look again at this later next week.
Gerd
------------------------------------------------------------------------ Date: Sun, 4 May 2014 21:21:59 +0200 From: extremecarver@ <mailto: extremecarver@ > To: mkgmap-dev@.org <mailto: mkgmap-dev@.org > Subject: Re: [mkgmap-dev] mkgmap ToDo list
No - I use splitter like this - with maxnodes depending on area - I usually just reduced it after failures to compile a tile.: java -Xms4000m -Xmx10400m -jar c:\openmtbmap\splitter.jar --max-nodes=%maxnodes% --max-threads=8 --output=pbf --keep-complete --max-areas=1524 --geonames-file=cities5000 --description=%country% --mapid=%FID%0000 c:\OpenMTBMap\osmpbf_geofabrik\%country%-latest.osm.pbf would using the precomp-sea parameter for splitter improve results? It's not that tiles with loads of sea actually failed.
On 03.05.2014 09:00, Gerd Petermann wrote:
Hi all,
I've coded a quick hack which seems to improve the ratios. On the other hand, I don't see these large differences between smallest and largest img file. What part of the world should I try? Do you use the precomp-sea parameter in splitter?
Gerd
------------------------------------------------------------------------ Date: Wed, 30 Apr 2014 14:59:32 +0200 From: extremecarver@ <mailto: extremecarver@ > To: mkgmap-dev@.org <mailto: mkgmap-dev@.org > Subject: Re: [mkgmap-dev] mkgmap ToDo list
if that doesn't seriously (more than 30-40%) slow down the splitter, I assume it would be much better... On 30.04.2014 14:06, Gerd Petermann wrote:
Hi,
if I got that right the number of nodes is not highly correlated to the img size, so the max-nodes value is not a good estimate.
I assume the reason is that nodes which belong to roads produce a lot more bytes in the img file compared to nodes which are parts of shapes or other non-routable ways, not talking about nodes which are simply ignored by the style.
So, a possible solution in splitter could be to parse the ways before reading the nodes and save all nodeids which belong to ways with highway=*. If these nodes are refered by more than one way with highway=* we assume that they will be routing nodes. These special nodes could be counted e.g. 10 times to give a better estimate.
Gerd
> Date: Wed, 30 Apr 2014 13:36:19 +0200 > From: osm@ <mailto: osm@ > > To: mkgmap-dev@.org <mailto: mkgmap-dev@.org > > Subject: Re: [mkgmap-dev] mkgmap ToDo list > > Multithreading the tile rendering for a single map is indeed a bit > difficult and I gave it up because you need to keep track which image > id's are already in use. Since I provide multiple maps the work-around > is running a few scripts parallel, which is also a crude form of > multithreading. > > The script language is PHP and it doesn't run on Windows without some > changes ('/' vs '\' in paths, 'rm -rf', that sort of stuff). Never tried it. > > To get a better optimum in file size, using the process I described > earlier, you could start off with a huge --max-nodes setting and then > 'search' for the highest --max-nodes that works for each specific area. > > On 30/04/2014 11:49, Felix Hartmann wrote: > > I would love if there was a possibility that you pass the used max-nodes > > value to mkgmap. > > When mkgmap is compiling the maps, then after the .img is created it > > should check > > a) did it crash due to too many max-nodes > > b) for me not important - but for others with very old GPS, etrex 10, > > ---> is tile bigger than X (usually 8) MB. > > > > if a) or b) true, then pass the file back to splitter and split with 60% > > of maxnodes - and compile the resulting .img files again. Should it fail > > again, use 40%, again 25%... Sometimes there are awful tiles, that need > > supersmall max-nodes till they compile, however lately (last 1-2 years) > > I never encountered them anymore. I think that happened rather due to a > > but in splitter/mgkmap that is fixed now. > > > > okay, you could also do this with a script, but it gets rather > > complicated to multithread it (you need to wait till mgkmap finished > > compiling all .img files - and run mkgmap first without address index to > > save time) and do some clever routines on making sure that the map id > > (e.g. 6340????.img) stay correct. Even more complicated to have > > consequent map id... > > > > For europe with a fixed max-node I get tiles from 1.9MB up to 18MB. > > That's factor 9 - so it's huge... > > If I could narrows that down easily to 8-18MB - without getting tiles > > crashing due to too high max-nodes values, that would be sweet. > > > > > > > > As for the scripts - would they run on windows too? - What programming > > language are they in? > > > > > > On 29.04.2014 21:39, osm@ <mailto: osm@ > wrote: > >> Oh, and ofcourse anyone interested can get my scripts, send an email. > >> They'll be on Github someday anyway. > >> > >> > >> On 2014-04-29 20:37, Gerd Petermann wrote: > >>> Hi Lambertus, > >>> > >>> okay, if I got that right you finally get *.img files with a size > >>> near (but below) 8MB, so maybe Henning can use that script, too. > >>> > >>> If you do that for e.g. Germany, how small is tpically the smallest > >>> *.img file ? > >>> Is it probably near 4 MB? > >>> > >>> Gerd > >>> > >>>> To: mkgmap-dev@.org <mailto: mkgmap-dev@.org > > >>>> Date: Tue, 29 Apr 2014 20:30:27 +0200 > >>>> From: osm@ <mailto: osm@ > > >>>> Subject: Re: [mkgmap-dev] mkgmap ToDo list > >>>> > >>>> These are the direct results from Splitter. The format is o5m, both > >>>> input as output. Splitter version is: r321. > >>>> > >>>> For this test I split the original source with --max-nodes=8000000. > >>> Then > >>>> I render the initial tiles, when the result is larger than 8MB it's > >>>> subsplit again with --max-nodes=(8000000-(attempt*100000)). The > >>> initial > >>>> source files are ~70MB (o5m) and after several subsplits the two > >>> *.img > >>>> are < 8MB. During this process --max-nodes has been reduced to e.g. > >>>> 7500000 and the source file is split up in two o5m files of about > >>> 37MB. > >>>> > >>>> I can upload an example source file and it's two subsplit siblings > >>> if > >>>> you want. > >>>> > >>>> > >>>> On 2014-04-29 19:38, GerdP wrote: > >>>> > Hi Lambertus, > >>>> > > >>>> > that's interesting. Are these the img file sizes or the osm file > >>> sizes? > >>>> > > >>>> > Gerd > >>>> > > >>>> > > >>>> > Lambertus wrote > >>>> >> Unfortunately I cannot confirm that. Below is a bit of logging > >>> from my > >>>> >> script: > >>>> >> Original: 97000020 (70551453), New: 0 (35684445), New: 1 > >>> (36852845) > >>>> >> Original: 97000001 (74621042), New: 0 (37522992), New: 1 > >>> (37222739) > >>>> >> Original: 97000002 (73391358), New: 0 (37679505), New: 1 > >>> (38098627) > >>>> >> Original: 97000003 (77862567), New: 0 (39075311), New: 1 > >>> (39261197) > >>>> >> > >>>> >> The original files above contain contour data, the filesize is > >>> between > >>>> >> brackets. As you can see both resulting file are approximately > >>> the > >>>> >> same > >>>> >> size. > >>>> >> > >>>> >> On 2014-04-29 15:39, Gerd Petermann wrote: > >>>> >>> Hi Lambertus, > >>>> >>> > >>>> >>> and I guess that even after this optimization you will > >>>> >>> see a factor 3 or higher between the largest tile and the > >>> smallest. > >>>> >>> Can you confirm that? > >>>> >>> > >>>> >>> Gerd > >>>> >>> > >>>> >>>> Date: Tue, 29 Apr 2014 15:32:38 +0200 > >>>> >>>> From: > >>>> > > >>>> >> osm@ > >>>> > > >>>> >>>> To: > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >>>> Subject: Re: [mkgmap-dev] mkgmap ToDo list > >>>> >>>> > >>>> >>>> Num-tiles=x would indeed be better for this specific need. > >>>> >>>> > >>>> >>>> It is my experience that it regularly takes multiple calls to > >>>> >>> Splitter > >>>> >>>> to get 2+ sub-tiles when you reduce the max-nodes by 100k for > >>> each > >>>> >>>> sub-split attempt. This is what I currently do to get an > >>> optimum in > >>>> >>>> tile-size vs total number of tiles. > >>>> >>>> > >>>> >>>> > >>>> >>>> > >>>> >>>> On 29/04/2014 15:09, Gerd Petermann wrote: > >>>> >>>> > Hi Lambertus, > >>>> >>>> > > >>>> >>>> > that sounds like a possible change in splitter: > >>>> >>>> > Instead of specifying max-nodes you may specify --num-tiles=x > >>>> >>>> > and splitter will try to find a split that produces excactly > >>> x > >>>> >>> tiles > >>>> >>>> > which are not too narrow and have a node number which is not > >>>> >>>> > too far from the average (but still aligned to a multiple of > >>> map > >>>> >>> units > >>>> >>>> > as now). > >>>> >>>> > So, for your script that means you don't have to find the > >>>> >>> max-nodes > >>>> >>>> > value. > >>>> >>>> > > >>>> >>>> > I'll think about this again... > >>>> >>>> > > >>>> >>>> > Gerd > >>>> >>>> > > >>>> >>>> > > Date: Tue, 29 Apr 2014 14:59:36 +0200 > >>>> >>>> > > From: > >>>> > > >>>> >> osm@ > >>>> > > >>>> >>>> > > To: > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >>>> > > Subject: Re: [mkgmap-dev] mkgmap ToDo list > >>>> >>>> > > > >>>> >>>> > > While this possibly can be solved in Splitter or Mkgmap, it > >>>> >>> could also > >>>> >>>> > > be solved by your build-script when you add a maximum tile > >>> size > >>>> >>> check > >>>> >>>> > > and re-split (with a lower number of max-nodes) until you > >>> get > >>>> >>> two or > >>>> >>>> > > more sub-tiles. Granted, this adds complexity to the script > >>> but > >>>> >>> it works > >>>> >>>> > > well for me. > >>>> >>>> > > > >>>> >>>> > > On 25/04/2014 21:54, Henning Scholland wrote: > >>>> >>>> > > > Hi Gerd, > >>>> >>>> > > > > >>>> >>>> > > > I would like to have img-tiles which have globally nearly > >>> the > >>>> >>> same > >>>> >>>> > > > filesize, so that they use the space of devices like > >>> eTrex 10. > >>>> >>>> > > > > >>>> >>>> > > > With my actual map I use globally the same value for > >>>> >>> max-nodes. But the > >>>> >>>> > > > size of the img-tiles differ more then factor 2. Eg. a > >>> tile in > >>>> >>> Germany > >>>> >>>> > > > is between 2 and 5 mb where a tile in China is about 10 > >>> mb. If > >>>> >>> I remove > >>>> >>>> > > > details, this difference will increase, because in > >>> Germany > >>>> >>> more objects > >>>> >>>> > > > will be removed from the img-tile then in China. > >>>> >>>> > > > > >>>> >>>> > > > Henning > >>>> >>>> > > > > >>>> >>>> > > > _______________________________________________ > >>>> >>>> > > > mkgmap-dev mailing list > >>>> >>>> > > > > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >>>> > > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >>>> > > > >>>> >>>> > > _______________________________________________ > >>>> >>>> > > mkgmap-dev mailing list > >>>> >>>> > > > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >>>> > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >>>> > > >>>> >>>> > > >>>> >>>> > _______________________________________________ > >>>> >>>> > mkgmap-dev mailing list > >>>> >>>> > > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >>>> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >>>> > > >>>> >>>> > >>>> >>>> _______________________________________________ > >>>> >>>> mkgmap-dev mailing list > >>>> >>>> > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >>> > >>>> >>> _______________________________________________ > >>>> >>> mkgmap-dev mailing list > >>>> >>> > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >> _______________________________________________ > >>>> >> mkgmap-dev mailing list > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > -- > >>>> > View this message in context: > >>>> > > >>>
http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5804588.html > >>>> > Sent from the Mkgmap Development mailing list archive at > >>> Nabble.com. > >>>> > _______________________________________________ > >>>> > mkgmap-dev mailing list > >>>> > mkgmap-dev@.org <mailto: mkgmap-dev@.org > > >>>> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> _______________________________________________ > >>>> mkgmap-dev mailing list > >>>> mkgmap-dev@.org <mailto: mkgmap-dev@.org > > >>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>> > >>> _______________________________________________ > >>> mkgmap-dev mailing list > >>> mkgmap-dev@.org <mailto: mkgmap-dev@.org > > >>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >> _______________________________________________ > >> mkgmap-dev mailing list > >> mkgmap-dev@.org <mailto: mkgmap-dev@.org > > >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > > _______________________________________________ > mkgmap-dev mailing list > mkgmap-dev@.org <mailto: mkgmap-dev@.org > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org <mailto: mkgmap-dev@.org >
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- keep on biking and discovering new trails
Felix openmtbmap.org &www.velomap.org <http://www.velomap.org>;
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
<mailto:
mkgmap-dev@.org >
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org <mailto: mkgmap-dev@.org >
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- keep on biking and discovering new trails
Felix openmtbmap.org &www.velomap.org <http://www.velomap.org>;
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
<mailto:
mkgmap-dev@.org >
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org <mailto: mkgmap-dev@.org >
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
<mailto:
mkgmap-dev@.org >
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org <mailto: mkgmap-dev@.org >
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev -- keep on biking and discovering new trails
Felix openmtbmap.org & www.velomap.org
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@.org http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- View this message in context: http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5805221.html Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- keep on biking and discovering new trails
Felix openmtbmap.org & www.velomap.org
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Thanks Gerd, seems to be working great! Two questions: - How does this interact with the --max-nodes setting? - Could it be used for an initial split of the entire planet? I.e., set the number of tiles I want and have splitter output equally sized tiles? (I know I could just try, but the servers are busy) On 2014-05-05 09:28, Gerd Petermann wrote:
Hi Lambertus,
attached is a patch that implements the new option: --num-tiles A target value that is used when no split-file is given. Splitting is done so that the given number of tiles is produced. The max-nodes value is ignored if this option is given. A compiled binary is here:
http://files.mkgmap.org.uk/download/206/splitter.jar [4]
Please let me know if it works as expected.
Gerd
------------------------- Date: Sun, 4 May 2014 22:20:20 +0200 From: osm@na1400.info To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] mkgmap ToDo list
Great, many thanks. I'm currently trying to create as large as (automatically) possible contour tiles and a lot of time is put in subsplitting a tile again and again until two subtiles appear. This new option would save much time.
On 05/04/2014 10:13 PM, Gerd Petermann wrote:
Hi Lambertus,
yes, I want to code that tomorrow. I prefer to make it "give me n tiles", but if that turns out to be difficult I try the simple version. I can think of scenarios where it is not possible to create exactly n tiles.
Gerd
------------------------- Date: Sun, 4 May 2014 22:08:32 +0200 From: osm@na1400.info To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] mkgmap ToDo list
Is the 'just give me two tiles' option for splitter on the to-do list? I'd really appreciate such a function.
On 05/04/2014 09:36 PM, Gerd Petermann wrote:
Hi Felix,
would using the precomp-sea parameter for splitter improve results? It's not that tiles with loads of sea actually failed.
good question. I've coded that parameter in splitter before we did some important changes in mkgmap. I think it is needed when you use a polygon file in combination with an imput file that doesn't cover the polygon. If the polygon covers a costline with a lot of islands, but the input file doesn't contain the data, splitter might create large tiles covering these "empty" areas. Later, in mkgmap, the empty areas are filled with the precompiled-sea data, and that can cause too large files.
BTW: The quick hack turned out to be useless. It worked better in some areas and worse in others. I'll look again at this later next week.
Gerd
------------------------- Date: Sun, 4 May 2014 21:21:59 +0200 From: extremecarver@gmail.com To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] mkgmap ToDo list
No - I use splitter like this - with maxnodes depending on area - I usually just reduced it after failures to compile a tile.: java -Xms4000m -Xmx10400m -jar c:openmtbmapsplitter.jar --max-nodes=%maxnodes% --max-threads=8 --output=pbf --keep-complete --max-areas=1524 --geonames-file=cities5000 --description=%country% --mapid=%FID%0000 c:OpenMTBMaposmpbf_geofabrik%country%-latest.osm.pbf would using the precomp-sea parameter for splitter improve results? It's not that tiles with loads of sea actually failed.
On 03.05.2014 09:00, Gerd Petermann wrote:
Hi all,
I've coded a quick hack which seems to improve the ratios. On the other hand, I don't see these large differences between smallest and largest img file. What part of the world should I try? Do you use the precomp-sea parameter in splitter?
Gerd
------------------------- Date: Wed, 30 Apr 2014 14:59:32 +0200 From: extremecarver@gmail.com To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] mkgmap ToDo list
if that doesn't seriously (more than 30-40%) slow down the splitter, I assume it would be much better...
On 30.04.2014 14:06, Gerd Petermann wrote:
Hi,
if I got that right the number of nodes is not highly correlated to the img size, so the max-nodes value is not a good estimate.
I assume the reason is that nodes which belong to roads produce a lot more bytes in the img file compared to nodes which are parts of shapes or other non-routable ways, not talking about nodes which are simply ignored by the style.
So, a possible solution in splitter could be to parse the ways before reading the nodes and save all nodeids which belong to ways with highway=*. If these nodes are refered by more than one way with highway=* we assume that they will be routing nodes. These special nodes could be counted e.g. 10 times to give a better estimate.
Gerd
Date: Wed, 30 Apr 2014 13:36:19 +0200 From: osm@na1400.info To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] mkgmap ToDo list
Multithreading the tile rendering for a single map is indeed a bit difficult and I gave it up because you need to keep track which image id's are already in use. Since I provide multiple maps the work-around is running a few scripts parallel, which is also a crude form of multithreading.
The script language is PHP and it doesn't run on Windows without some changes ('/' vs '' in paths, 'rm -rf', that sort of stuff). Never tried it.
To get a better optimum in file size, using the process I described earlier, you could start off with a huge --max-nodes setting and then 'search' for the highest --max-nodes that works for each specific area.
On 30/04/2014 11:49, Felix Hartmann wrote:
I would love if there was a possibility that you pass the used max-nodes value to mkgmap. When mkgmap is compiling the maps, then after the .img is created it should check a) did it crash due to too many max-nodes b) for me not important - but for others with very old GPS, etrex 10, ---> is tile bigger than X (usually 8) MB.
if a) or b) true, then pass the file back to splitter and split with 60% of maxnodes - and compile the resulting .img files again. Should it fail again, use 40%, again 25%... Sometimes there are awful tiles, that need supersmall max-nodes till they compile, however lately (last 1-2 years) I never encountered them anymore. I think that happened rather due to a but in splitter/mgkmap that is fixed now.
okay, you could also do this with a script, but it gets rather complicated to multithread it (you need to wait till mgkmap finished compiling all .img files - and run mkgmap first without address index to save time) and do some clever routines on making sure that the map id (e.g. 6340????.img) stay correct. Even more complicated to have consequent map id...
For europe with a fixed max-node I get tiles from 1.9MB up to 18MB. That's factor 9 - so it's huge... If I could narrows that down easily to 8-18MB - without getting tiles crashing due to too high max-nodes values, that would be sweet.
As for the scripts - would they run on windows too? - What programming language are they in?
On 29.04.2014 21:39, osm@na1400.info wrote:
Oh, and ofcourse anyone interested can get my scripts, send an email. They'll be on Github someday anyway.
On 2014-04-29 20:37, Gerd Petermann wrote:
Hi Lambertus,
okay, if I got that right you finally get *.img files with a size near (but below) 8MB, so maybe Henning can use that script, too.
If you do that for e.g. Germany, how small is tpically the smallest *.img file ? Is it probably near 4 MB?
Gerd
> To: mkgmap-dev@lists.mkgmap.org.uk > Date: Tue, 29 Apr 2014 20:30:27 +0200 > From: osm@na1400.info > Subject: Re: [mkgmap-dev] mkgmap ToDo list > > These are the direct results from Splitter. The format is o5m, both > input as output. Splitter version is: r321. > > For this test I split the original source with --max-nodes=8000000. Then > I render the initial tiles, when the result is larger than 8MB it's > subsplit again with --max-nodes=(8000000-(attempt*100000)). The initial > source files are ~70MB (o5m) and after several subsplits the two *.img > are < 8MB. During this process --max-nodes has been reduced to e.g. > 7500000 and the source file is split up in two o5m files of about 37MB. > > I can upload an example source file and it's two subsplit siblings if > you want. > > > On 2014-04-29 19:38, GerdP wrote: > > Hi Lambertus, > > > > that's interesting. Are these the img file sizes or the osm file sizes? > > > > Gerd > > > > > > Lambertus wrote > >> Unfortunately I cannot confirm that. Below is a bit of logging from my > >> script: > >> Original: 97000020 (70551453), New: 0 (35684445), New: 1 (36852845) > >> Original: 97000001 (74621042), New: 0 (37522992), New: 1 (37222739) > >> Original: 97000002 (73391358), New: 0 (37679505), New: 1 (38098627) > >> Original: 97000003 (77862567), New: 0 (39075311), New: 1 (39261197) > >> > >> The original files above contain contour data, the filesize is between > >> brackets. As you can see both resulting file are approximately the > >> same > >> size. > >> > >> On 2014-04-29 15:39, Gerd Petermann wrote: > >>> Hi Lambertus, > >>> > >>> and I guess that even after this optimization you will > >>> see a factor 3 or higher between the largest tile and the smallest. > >>> Can you confirm that? > >>> > >>> Gerd > >>> > >>>> Date: Tue, 29 Apr 2014 15:32:38 +0200 > >>>> From: > > > >> osm@ > > > >>>> To: > > > >> mkgmap-dev@.org > > > >>>> Subject: Re: [mkgmap-dev] mkgmap ToDo list > >>>> > >>>> Num-tiles=x would indeed be better for this specific need. > >>>> > >>>> It is my experience that it regularly takes multiple calls to > >>> Splitter > >>>> to get 2+ sub-tiles when you reduce the max-nodes by 100k for each > >>>> sub-split attempt. This is what I currently do to get an optimum in > >>>> tile-size vs total number of tiles. > >>>> > >>>> > >>>> > >>>> On 29/04/2014 15:09, Gerd Petermann wrote: > >>>> > Hi Lambertus, > >>>> > > >>>> > that sounds like a possible change in splitter: > >>>> > Instead of specifying max-nodes you may specify --num-tiles=x > >>>> > and splitter will try to find a split that produces excactly x > >>> tiles > >>>> > which are not too narrow and have a node number which is not > >>>> > too far from the average (but still aligned to a multiple of map > >>> units > >>>> > as now). > >>>> > So, for your script that means you don't have to find the > >>> max-nodes > >>>> > value. > >>>> > > >>>> > I'll think about this again... > >>>> > > >>>> > Gerd > >>>> > > >>>> > > Date: Tue, 29 Apr 2014 14:59:36 +0200 > >>>> > > From: > > > >> osm@ > > > >>>> > > To: > > > >> mkgmap-dev@.org > > > >>>> > > Subject: Re: [mkgmap-dev] mkgmap ToDo list > >>>> > > > >>>> > > While this possibly can be solved in Splitter or Mkgmap, it > >>> could also > >>>> > > be solved by your build-script when you add a maximum tile size > >>> check > >>>> > > and re-split (with a lower number of max-nodes) until you get > >>> two or > >>>> > > more sub-tiles. Granted, this adds complexity to the script but > >>> it works > >>>> > > well for me. > >>>> > > > >>>> > > On 25/04/2014 21:54, Henning Scholland wrote: > >>>> > > > Hi Gerd, > >>>> > > > > >>>> > > > I would like to have img-tiles which have globally nearly the > >>> same > >>>> > > > filesize, so that they use the space of devices like eTrex 10. > >>>> > > > > >>>> > > > With my actual map I use globally the same value for > >>> max-nodes. But the > >>>> > > > size of the img-tiles differ more then factor 2. Eg. a tile in > >>> Germany > >>>> > > > is between 2 and 5 mb where a tile in China is about 10 mb. If > >>> I remove > >>>> > > > details, this difference will increase, because in Germany > >>> more objects > >>>> > > > will be removed from the img-tile then in China. > >>>> > > > > >>>> > > > Henning > >>>> > > > > >>>> > > > _______________________________________________ > >>>> > > > mkgmap-dev mailing list > >>>> > > > > > > >> mkgmap-dev@.org > > > >>>> > > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1] > >>>> > > > >>>> > > _______________________________________________ > >>>> > > mkgmap-dev mailing list > >>>> > > > > > >> mkgmap-dev@.org > > > >>>> > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1] > >>>> > > >>>> > > >>>> > _______________________________________________ > >>>> > mkgmap-dev mailing list > >>>> > > > > >> mkgmap-dev@.org > > > >>>> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1] > >>>> > > >>>> > >>>> _______________________________________________ > >>>> mkgmap-dev mailing list > >>>> > > > >> mkgmap-dev@.org > > > >>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1] > >>> > >>> _______________________________________________ > >>> mkgmap-dev mailing list > >>> > > > >> mkgmap-dev@.org > > > >>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1] > >> _______________________________________________ > >> mkgmap-dev mailing list > > > >> mkgmap-dev@.org > > > >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1] > > > > > > > > > > > > -- > > View this message in context: > >
http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5804588.html
[3]
> > Sent from the Mkgmap Development mailing list archive at Nabble.com. > > _______________________________________________ > > mkgmap-dev mailing list > > mkgmap-dev@lists.mkgmap.org.uk > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1] > _______________________________________________ > mkgmap-dev mailing list > mkgmap-dev@lists.mkgmap.org.uk > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1]
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1]
mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1]
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1]
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1]
-- keep on biking and discovering new trails
Felix openmtbmap.org & www.velomap.org [2]
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1]
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1]
-- keep on biking and discovering new trails
Felix openmtbmap.org & www.velomap.org [2]
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1]
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1]
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1]
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1]
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Links: ------ [1] http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [2] http://www.velomap.org [3] http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5804588.html [4] http://files.mkgmap.org.uk/download/206/splitter.jar
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Lambertus, the algo is quite simple. The max-nodes parameter is ignored, splitter calculates an initial max-nodes value by dividing the total number of nodes by the wanted number of tiles. Then the normal split algo is used. If the result has too many tiles, the max-nodes value is increased, if it is too low, it is decreased. This is repeated until either a solution is found or no max-nodes value is found that results in the number of tiles. In other words: the tiles are as "equally sized" as before. Gerd Lambertus wrote
Thanks Gerd, seems to be working great!
Two questions: - How does this interact with the --max-nodes setting? - Could it be used for an initial split of the entire planet? I.e., set the number of tiles I want and have splitter output equally sized tiles? (I know I could just try, but the servers are busy)
On 2014-05-05 09:28, Gerd Petermann wrote:
Hi Lambertus,
attached is a patch that implements the new option: --num-tiles A target value that is used when no split-file is given. Splitting is done so that the given number of tiles is produced. The max-nodes value is ignored if this option is given. A compiled binary is here:
http://files.mkgmap.org.uk/download/206/splitter.jar [4]
Please let me know if it works as expected.
Gerd
------------------------- Date: Sun, 4 May 2014 22:20:20 +0200 From:
osm@
To:
mkgmap-dev@.org
Subject: Re: [mkgmap-dev] mkgmap ToDo list
Great, many thanks. I'm currently trying to create as large as (automatically) possible contour tiles and a lot of time is put in subsplitting a tile again and again until two subtiles appear. This new option would save much time.
On 05/04/2014 10:13 PM, Gerd Petermann wrote:
Hi Lambertus,
yes, I want to code that tomorrow. I prefer to make it "give me n tiles", but if that turns out to be difficult I try the simple version. I can think of scenarios where it is not possible to create exactly n tiles.
Gerd
------------------------- Date: Sun, 4 May 2014 22:08:32 +0200 From:
osm@
To:
mkgmap-dev@.org
Subject: Re: [mkgmap-dev] mkgmap ToDo list
Is the 'just give me two tiles' option for splitter on the to-do list? I'd really appreciate such a function.
On 05/04/2014 09:36 PM, Gerd Petermann wrote:
Hi Felix,
would using the precomp-sea parameter for splitter improve results? It's not that tiles with loads of sea actually failed.
good question. I've coded that parameter in splitter before we did some important changes in mkgmap. I think it is needed when you use a polygon file in combination with an imput file that doesn't cover the polygon. If the polygon covers a costline with a lot of islands, but the input file doesn't contain the data, splitter might create large tiles covering these "empty" areas. Later, in mkgmap, the empty areas are filled with the precompiled-sea data, and that can cause too large files.
BTW: The quick hack turned out to be useless. It worked better in some areas and worse in others. I'll look again at this later next week.
Gerd
------------------------- Date: Sun, 4 May 2014 21:21:59 +0200 From:
extremecarver@
To:
mkgmap-dev@.org
Subject: Re: [mkgmap-dev] mkgmap ToDo list
No - I use splitter like this - with maxnodes depending on area - I usually just reduced it after failures to compile a tile.: java -Xms4000m -Xmx10400m -jar c:openmtbmapsplitter.jar --max-nodes=%maxnodes% --max-threads=8 --output=pbf --keep-complete --max-areas=1524 --geonames-file=cities5000 --description=%country% --mapid=%FID%0000 c:OpenMTBMaposmpbf_geofabrik%country%-latest.osm.pbf would using the precomp-sea parameter for splitter improve results? It's not that tiles with loads of sea actually failed.
On 03.05.2014 09:00, Gerd Petermann wrote:
Hi all,
I've coded a quick hack which seems to improve the ratios. On the other hand, I don't see these large differences between smallest and largest img file. What part of the world should I try? Do you use the precomp-sea parameter in splitter?
Gerd
------------------------- Date: Wed, 30 Apr 2014 14:59:32 +0200 From:
extremecarver@
To:
mkgmap-dev@.org
Subject: Re: [mkgmap-dev] mkgmap ToDo list
if that doesn't seriously (more than 30-40%) slow down the splitter, I assume it would be much better...
On 30.04.2014 14:06, Gerd Petermann wrote:
Hi,
if I got that right the number of nodes is not highly correlated to the img size, so the max-nodes value is not a good estimate.
I assume the reason is that nodes which belong to roads produce a lot more bytes in the img file compared to nodes which are parts of shapes or other non-routable ways, not talking about nodes which are simply ignored by the style.
So, a possible solution in splitter could be to parse the ways before reading the nodes and save all nodeids which belong to ways with highway=*. If these nodes are refered by more than one way with highway=* we assume that they will be routing nodes. These special nodes could be counted e.g. 10 times to give a better estimate.
Gerd
Date: Wed, 30 Apr 2014 13:36:19 +0200 From:
osm@
To:
mkgmap-dev@.org
Subject: Re: [mkgmap-dev] mkgmap ToDo list
Multithreading the tile rendering for a single map is indeed a bit difficult and I gave it up because you need to keep track which image id's are already in use. Since I provide multiple maps the work-around is running a few scripts parallel, which is also a crude form of multithreading.
The script language is PHP and it doesn't run on Windows without some changes ('/' vs '' in paths, 'rm -rf', that sort of stuff). Never tried it.
To get a better optimum in file size, using the process I described earlier, you could start off with a huge --max-nodes setting and then 'search' for the highest --max-nodes that works for each specific area.
On 30/04/2014 11:49, Felix Hartmann wrote:
I would love if there was a possibility that you pass the used max-nodes value to mkgmap. When mkgmap is compiling the maps, then after the .img is created it should check a) did it crash due to too many max-nodes b) for me not important - but for others with very old GPS, etrex 10, ---> is tile bigger than X (usually 8) MB.
if a) or b) true, then pass the file back to splitter and split with 60% of maxnodes - and compile the resulting .img files again. Should it fail again, use 40%, again 25%... Sometimes there are awful tiles, that need supersmall max-nodes till they compile, however lately (last 1-2 years) I never encountered them anymore. I think that happened rather due to a but in splitter/mgkmap that is fixed now.
okay, you could also do this with a script, but it gets rather complicated to multithread it (you need to wait till mgkmap finished compiling all .img files - and run mkgmap first without address index to save time) and do some clever routines on making sure that the map id (e.g. 6340????.img) stay correct. Even more complicated to have consequent map id...
For europe with a fixed max-node I get tiles from 1.9MB up to 18MB. That's factor 9 - so it's huge... If I could narrows that down easily to 8-18MB - without getting tiles crashing due to too high max-nodes values, that would be sweet.
As for the scripts - would they run on windows too? - What programming language are they in?
On 29.04.2014 21:39,
osm@
wrote:
Oh, and ofcourse anyone interested can get my scripts, send an email. They'll be on Github someday anyway.
On 2014-04-29 20:37, Gerd Petermann wrote: > Hi Lambertus, > > okay, if I got that right you finally get *.img files with a size > near (but below) 8MB, so maybe Henning can use that script, too. > > If you do that for e.g. Germany, how small is tpically the smallest > *.img file ? > Is it probably near 4 MB? > > Gerd > >> To:
mkgmap-dev@.org
>> Date: Tue, 29 Apr 2014 20:30:27 +0200 >> From:
osm@
>> Subject: Re: [mkgmap-dev] mkgmap ToDo list >> >> These are the direct results from Splitter. The format is o5m, both >> input as output. Splitter version is: r321. >> >> For this test I split the original source with --max-nodes=8000000. > Then >> I render the initial tiles, when the result is larger than 8MB it's >> subsplit again with --max-nodes=(8000000-(attempt*100000)). The > initial >> source files are ~70MB (o5m) and after several subsplits the two > *.img >> are < 8MB. During this process --max-nodes has been reduced to e.g. >> 7500000 and the source file is split up in two o5m files of about > 37MB. >> >> I can upload an example source file and it's two subsplit siblings > if >> you want. >> >> >> On 2014-04-29 19:38, GerdP wrote: >> > Hi Lambertus, >> > >> > that's interesting. Are these the img file sizes or the osm file > sizes? >> > >> > Gerd >> > >> > >> > Lambertus wrote >> >> Unfortunately I cannot confirm that. Below is a bit of logging > from my >> >> script: >> >> Original: 97000020 (70551453), New: 0 (35684445), New: 1 > (36852845) >> >> Original: 97000001 (74621042), New: 0 (37522992), New: 1 > (37222739) >> >> Original: 97000002 (73391358), New: 0 (37679505), New: 1 > (38098627) >> >> Original: 97000003 (77862567), New: 0 (39075311), New: 1 > (39261197) >> >> >> >> The original files above contain contour data, the filesize is > between >> >> brackets. As you can see both resulting file are approximately > the >> >> same >> >> size. >> >> >> >> On 2014-04-29 15:39, Gerd Petermann wrote: >> >>> Hi Lambertus, >> >>> >> >>> and I guess that even after this optimization you will >> >>> see a factor 3 or higher between the largest tile and the > smallest. >> >>> Can you confirm that? >> >>> >> >>> Gerd >> >>> >> >>>> Date: Tue, 29 Apr 2014 15:32:38 +0200 >> >>>> From: >> > >> >> osm@ >> > >> >>>> To: >> > >> >> mkgmap-dev@.org >> > >> >>>> Subject: Re: [mkgmap-dev] mkgmap ToDo list >> >>>> >> >>>> Num-tiles=x would indeed be better for this specific need. >> >>>> >> >>>> It is my experience that it regularly takes multiple calls to >> >>> Splitter >> >>>> to get 2+ sub-tiles when you reduce the max-nodes by 100k for > each >> >>>> sub-split attempt. This is what I currently do to get an > optimum in >> >>>> tile-size vs total number of tiles. >> >>>> >> >>>> >> >>>> >> >>>> On 29/04/2014 15:09, Gerd Petermann wrote: >> >>>> > Hi Lambertus, >> >>>> > >> >>>> > that sounds like a possible change in splitter: >> >>>> > Instead of specifying max-nodes you may specify --num-tiles=x >> >>>> > and splitter will try to find a split that produces excactly > x >> >>> tiles >> >>>> > which are not too narrow and have a node number which is not >> >>>> > too far from the average (but still aligned to a multiple of > map >> >>> units >> >>>> > as now). >> >>>> > So, for your script that means you don't have to find the >> >>> max-nodes >> >>>> > value. >> >>>> > >> >>>> > I'll think about this again... >> >>>> > >> >>>> > Gerd >> >>>> > >> >>>> > > Date: Tue, 29 Apr 2014 14:59:36 +0200 >> >>>> > > From: >> > >> >> osm@ >> > >> >>>> > > To: >> > >> >> mkgmap-dev@.org >> > >> >>>> > > Subject: Re: [mkgmap-dev] mkgmap ToDo list >> >>>> > > >> >>>> > > While this possibly can be solved in Splitter or Mkgmap, it >> >>> could also >> >>>> > > be solved by your build-script when you add a maximum tile > size >> >>> check >> >>>> > > and re-split (with a lower number of max-nodes) until you > get >> >>> two or >> >>>> > > more sub-tiles. Granted, this adds complexity to the script > but >> >>> it works >> >>>> > > well for me. >> >>>> > > >> >>>> > > On 25/04/2014 21:54, Henning Scholland wrote: >> >>>> > > > Hi Gerd, >> >>>> > > > >> >>>> > > > I would like to have img-tiles which have globally nearly > the >> >>> same >> >>>> > > > filesize, so that they use the space of devices like > eTrex 10. >> >>>> > > > >> >>>> > > > With my actual map I use globally the same value for >> >>> max-nodes. But the >> >>>> > > > size of the img-tiles differ more then factor 2. Eg. a > tile in >> >>> Germany >> >>>> > > > is between 2 and 5 mb where a tile in China is about 10 > mb. If >> >>> I remove >> >>>> > > > details, this difference will increase, because in > Germany >> >>> more objects >> >>>> > > > will be removed from the img-tile then in China. >> >>>> > > > >> >>>> > > > Henning >> >>>> > > > >> >>>> > > > _______________________________________________ >> >>>> > > > mkgmap-dev mailing list >> >>>> > > > >> > >> >> mkgmap-dev@.org >> > >> >>>> > > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1] >> >>>> > > >> >>>> > > _______________________________________________ >> >>>> > > mkgmap-dev mailing list >> >>>> > > >> > >> >> mkgmap-dev@.org >> > >> >>>> > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1] >> >>>> > >> >>>> > >> >>>> > _______________________________________________ >> >>>> > mkgmap-dev mailing list >> >>>> > >> > >> >> mkgmap-dev@.org >> > >> >>>> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1] >> >>>> > >> >>>> >> >>>> _______________________________________________ >> >>>> mkgmap-dev mailing list >> >>>> >> > >> >> mkgmap-dev@.org >> > >> >>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1] >> >>> >> >>> _______________________________________________ >> >>> mkgmap-dev mailing list >> >>> >> > >> >> mkgmap-dev@.org >> > >> >>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1] >> >> _______________________________________________ >> >> mkgmap-dev mailing list >> > >> >> mkgmap-dev@.org >> > >> >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1] >> > >> > >> > >> > >> > >> > -- >> > View this message in context: >> > >
http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5804588.html
[3]
>> > Sent from the Mkgmap Development mailing list archive at > Nabble.com. >> > _______________________________________________ >> > mkgmap-dev mailing list >> >
mkgmap-dev@.org
>> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1] >> _______________________________________________ >> mkgmap-dev mailing list >>
mkgmap-dev@.org
>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1] > > _______________________________________________ > mkgmap-dev mailing list >
mkgmap-dev@.org
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1] _______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1]
-- keep on biking and discovering new trails
Felix openmtbmap.org & www.velomap.org [2]
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1]
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
-- keep on biking and discovering new trails
Felix openmtbmap.org & www.velomap.org [2]
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1]
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1]
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1]
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1]
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Links: ------ [1] http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [2] http://www.velomap.org [3] http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5804588.html [4] http://files.mkgmap.org.uk/download/206/splitter.jar
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
mkgmap-dev mailing list
mkgmap-dev@.org
-- View this message in context: http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5805281.html Sent from the Mkgmap Development mailing list archive at Nabble.com.
I just tried asia with and without the precomp-sea option. Without I even quicker run into broken tiles (1). Using it gives me 3 more tiles total (660 overall), and no failing tile. However - smallest tile is 970KB while biggest is 15MB (around 19MB is around the biggest I see from mkgmap without crash). So that's a more than 15 ratio. Long way to go I would say. Though it only really matters to me for continent maps, and Germany/China/Canada/USA/France maps - those are likely to push people into 2025 or 4000 tile limit on their devices if too many small tiles exist... Europe map is anyhow over the tops... (with my style-file - would need very reduced style for getting it <4GB). A range of 5-18MB would already be very nice... On 03.05.2014 09:00, Gerd Petermann wrote:
Hi all,
I've coded a quick hack which seems to improve the ratios. On the other hand, I don't see these large differences between smallest and largest img file. What part of the world should I try? Do you use the precomp-sea parameter in splitter?
Gerd
------------------------------------------------------------------------ Date: Wed, 30 Apr 2014 14:59:32 +0200 From: extremecarver@gmail.com To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] mkgmap ToDo list
if that doesn't seriously (more than 30-40%) slow down the splitter, I assume it would be much better... On 30.04.2014 14:06, Gerd Petermann wrote:
Hi,
if I got that right the number of nodes is not highly correlated to the img size, so the max-nodes value is not a good estimate.
I assume the reason is that nodes which belong to roads produce a lot more bytes in the img file compared to nodes which are parts of shapes or other non-routable ways, not talking about nodes which are simply ignored by the style.
So, a possible solution in splitter could be to parse the ways before reading the nodes and save all nodeids which belong to ways with highway=*. If these nodes are refered by more than one way with highway=* we assume that they will be routing nodes. These special nodes could be counted e.g. 10 times to give a better estimate.
Gerd
> Date: Wed, 30 Apr 2014 13:36:19 +0200 > From: osm@na1400.info <mailto:osm@na1400.info> > To: mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> > Subject: Re: [mkgmap-dev] mkgmap ToDo list > > Multithreading the tile rendering for a single map is indeed a bit > difficult and I gave it up because you need to keep track which image > id's are already in use. Since I provide multiple maps the work-around > is running a few scripts parallel, which is also a crude form of > multithreading. > > The script language is PHP and it doesn't run on Windows without some > changes ('/' vs '\' in paths, 'rm -rf', that sort of stuff). Never tried it. > > To get a better optimum in file size, using the process I described > earlier, you could start off with a huge --max-nodes setting and then > 'search' for the highest --max-nodes that works for each specific area. > > On 30/04/2014 11:49, Felix Hartmann wrote: > > I would love if there was a possibility that you pass the used max-nodes > > value to mkgmap. > > When mkgmap is compiling the maps, then after the .img is created it > > should check > > a) did it crash due to too many max-nodes > > b) for me not important - but for others with very old GPS, etrex 10, > > ---> is tile bigger than X (usually 8) MB. > > > > if a) or b) true, then pass the file back to splitter and split with 60% > > of maxnodes - and compile the resulting .img files again. Should it fail > > again, use 40%, again 25%... Sometimes there are awful tiles, that need > > supersmall max-nodes till they compile, however lately (last 1-2 years) > > I never encountered them anymore. I think that happened rather due to a > > but in splitter/mgkmap that is fixed now. > > > > okay, you could also do this with a script, but it gets rather > > complicated to multithread it (you need to wait till mgkmap finished > > compiling all .img files - and run mkgmap first without address index to > > save time) and do some clever routines on making sure that the map id > > (e.g. 6340????.img) stay correct. Even more complicated to have > > consequent map id... > > > > For europe with a fixed max-node I get tiles from 1.9MB up to 18MB. > > That's factor 9 - so it's huge... > > If I could narrows that down easily to 8-18MB - without getting tiles > > crashing due to too high max-nodes values, that would be sweet. > > > > > > > > As for the scripts - would they run on windows too? - What programming > > language are they in? > > > > > > On 29.04.2014 21:39, osm@na1400.info <mailto:osm@na1400.info> wrote: > >> Oh, and ofcourse anyone interested can get my scripts, send an email. > >> They'll be on Github someday anyway. > >> > >> > >> On 2014-04-29 20:37, Gerd Petermann wrote: > >>> Hi Lambertus, > >>> > >>> okay, if I got that right you finally get *.img files with a size > >>> near (but below) 8MB, so maybe Henning can use that script, too. > >>> > >>> If you do that for e.g. Germany, how small is tpically the smallest > >>> *.img file ? > >>> Is it probably near 4 MB? > >>> > >>> Gerd > >>> > >>>> To: mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> > >>>> Date: Tue, 29 Apr 2014 20:30:27 +0200 > >>>> From: osm@na1400.info <mailto:osm@na1400.info> > >>>> Subject: Re: [mkgmap-dev] mkgmap ToDo list > >>>> > >>>> These are the direct results from Splitter. The format is o5m, both > >>>> input as output. Splitter version is: r321. > >>>> > >>>> For this test I split the original source with --max-nodes=8000000. > >>> Then > >>>> I render the initial tiles, when the result is larger than 8MB it's > >>>> subsplit again with --max-nodes=(8000000-(attempt*100000)). The > >>> initial > >>>> source files are ~70MB (o5m) and after several subsplits the two > >>> *.img > >>>> are < 8MB. During this process --max-nodes has been reduced to e.g. > >>>> 7500000 and the source file is split up in two o5m files of about > >>> 37MB. > >>>> > >>>> I can upload an example source file and it's two subsplit siblings > >>> if > >>>> you want. > >>>> > >>>> > >>>> On 2014-04-29 19:38, GerdP wrote: > >>>> > Hi Lambertus, > >>>> > > >>>> > that's interesting. Are these the img file sizes or the osm file > >>> sizes? > >>>> > > >>>> > Gerd > >>>> > > >>>> > > >>>> > Lambertus wrote > >>>> >> Unfortunately I cannot confirm that. Below is a bit of logging > >>> from my > >>>> >> script: > >>>> >> Original: 97000020 (70551453), New: 0 (35684445), New: 1 > >>> (36852845) > >>>> >> Original: 97000001 (74621042), New: 0 (37522992), New: 1 > >>> (37222739) > >>>> >> Original: 97000002 (73391358), New: 0 (37679505), New: 1 > >>> (38098627) > >>>> >> Original: 97000003 (77862567), New: 0 (39075311), New: 1 > >>> (39261197) > >>>> >> > >>>> >> The original files above contain contour data, the filesize is > >>> between > >>>> >> brackets. As you can see both resulting file are approximately > >>> the > >>>> >> same > >>>> >> size. > >>>> >> > >>>> >> On 2014-04-29 15:39, Gerd Petermann wrote: > >>>> >>> Hi Lambertus, > >>>> >>> > >>>> >>> and I guess that even after this optimization you will > >>>> >>> see a factor 3 or higher between the largest tile and the > >>> smallest. > >>>> >>> Can you confirm that? > >>>> >>> > >>>> >>> Gerd > >>>> >>> > >>>> >>>> Date: Tue, 29 Apr 2014 15:32:38 +0200 > >>>> >>>> From: > >>>> > > >>>> >> osm@ > >>>> > > >>>> >>>> To: > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >>>> Subject: Re: [mkgmap-dev] mkgmap ToDo list > >>>> >>>> > >>>> >>>> Num-tiles=x would indeed be better for this specific need. > >>>> >>>> > >>>> >>>> It is my experience that it regularly takes multiple calls to > >>>> >>> Splitter > >>>> >>>> to get 2+ sub-tiles when you reduce the max-nodes by 100k for > >>> each > >>>> >>>> sub-split attempt. This is what I currently do to get an > >>> optimum in > >>>> >>>> tile-size vs total number of tiles. > >>>> >>>> > >>>> >>>> > >>>> >>>> > >>>> >>>> On 29/04/2014 15:09, Gerd Petermann wrote: > >>>> >>>> > Hi Lambertus, > >>>> >>>> > > >>>> >>>> > that sounds like a possible change in splitter: > >>>> >>>> > Instead of specifying max-nodes you may specify --num-tiles=x > >>>> >>>> > and splitter will try to find a split that produces excactly > >>> x > >>>> >>> tiles > >>>> >>>> > which are not too narrow and have a node number which is not > >>>> >>>> > too far from the average (but still aligned to a multiple of > >>> map > >>>> >>> units > >>>> >>>> > as now). > >>>> >>>> > So, for your script that means you don't have to find the > >>>> >>> max-nodes > >>>> >>>> > value. > >>>> >>>> > > >>>> >>>> > I'll think about this again... > >>>> >>>> > > >>>> >>>> > Gerd > >>>> >>>> > > >>>> >>>> > > Date: Tue, 29 Apr 2014 14:59:36 +0200 > >>>> >>>> > > From: > >>>> > > >>>> >> osm@ > >>>> > > >>>> >>>> > > To: > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >>>> > > Subject: Re: [mkgmap-dev] mkgmap ToDo list > >>>> >>>> > > > >>>> >>>> > > While this possibly can be solved in Splitter or Mkgmap, it > >>>> >>> could also > >>>> >>>> > > be solved by your build-script when you add a maximum tile > >>> size > >>>> >>> check > >>>> >>>> > > and re-split (with a lower number of max-nodes) until you > >>> get > >>>> >>> two or > >>>> >>>> > > more sub-tiles. Granted, this adds complexity to the script > >>> but > >>>> >>> it works > >>>> >>>> > > well for me. > >>>> >>>> > > > >>>> >>>> > > On 25/04/2014 21:54, Henning Scholland wrote: > >>>> >>>> > > > Hi Gerd, > >>>> >>>> > > > > >>>> >>>> > > > I would like to have img-tiles which have globally nearly > >>> the > >>>> >>> same > >>>> >>>> > > > filesize, so that they use the space of devices like > >>> eTrex 10. > >>>> >>>> > > > > >>>> >>>> > > > With my actual map I use globally the same value for > >>>> >>> max-nodes. But the > >>>> >>>> > > > size of the img-tiles differ more then factor 2. Eg. a > >>> tile in > >>>> >>> Germany > >>>> >>>> > > > is between 2 and 5 mb where a tile in China is about 10 > >>> mb. If > >>>> >>> I remove > >>>> >>>> > > > details, this difference will increase, because in > >>> Germany > >>>> >>> more objects > >>>> >>>> > > > will be removed from the img-tile then in China. > >>>> >>>> > > > > >>>> >>>> > > > Henning > >>>> >>>> > > > > >>>> >>>> > > > _______________________________________________ > >>>> >>>> > > > mkgmap-dev mailing list > >>>> >>>> > > > > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >>>> > > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >>>> > > > >>>> >>>> > > _______________________________________________ > >>>> >>>> > > mkgmap-dev mailing list > >>>> >>>> > > > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >>>> > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >>>> > > >>>> >>>> > > >>>> >>>> > _______________________________________________ > >>>> >>>> > mkgmap-dev mailing list > >>>> >>>> > > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >>>> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >>>> > > >>>> >>>> > >>>> >>>> _______________________________________________ > >>>> >>>> mkgmap-dev mailing list > >>>> >>>> > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >>> > >>>> >>> _______________________________________________ > >>>> >>> mkgmap-dev mailing list > >>>> >>> > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >> _______________________________________________ > >>>> >> mkgmap-dev mailing list > >>>> > > >>>> >> mkgmap-dev@.org <mailto:mkgmap-dev@.org> > >>>> > > >>>> >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > -- > >>>> > View this message in context: > >>>> > > >>> http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5804588.html > >>>> > Sent from the Mkgmap Development mailing list archive at > >>> Nabble.com. > >>>> > _______________________________________________ > >>>> > mkgmap-dev mailing list > >>>> > mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> > >>>> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> _______________________________________________ > >>>> mkgmap-dev mailing list > >>>> mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> > >>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>> > >>> _______________________________________________ > >>> mkgmap-dev mailing list > >>> mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> > >>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >> _______________________________________________ > >> mkgmap-dev mailing list > >> mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> > >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > > _______________________________________________ > mkgmap-dev mailing list > mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- keep on biking and discovering new trails
Felix openmtbmap.org &www.velomap.org <http://www.velomap.org>
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- keep on biking and discovering new trails Felix openmtbmap.org & www.velomap.org
Hi Felix, thanks for the info, I was not aware of any "number of tiles" limit. I have a new theory regarding "density of nodes" and "img file size". With the styles I use for testing, the largest tiles (img file size) are typically also large regarding the bbox. It seems that points in areas with rather low density are likely to produce more bytes compared to those in very dense areas. I see two possible reasons: 1) Many points in dense areas are ignored 2) For lines and shapes, the amount of points is not the only factor that determines the number of needed bytes, it also matters how big the largest delta values are (that is the lon/lat-delta between two consecutive points) Maybe we can find a simple rule to weight nodes in dense areas lower than those in rarely populated areas. Gerd Date: Mon, 5 May 2014 01:24:31 +0200 From: extremecarver@gmail.com To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] mkgmap ToDo list I just tried asia with and without the precomp-sea option. Without I even quicker run into broken tiles (1). Using it gives me 3 more tiles total (660 overall), and no failing tile. However - smallest tile is 970KB while biggest is 15MB (around 19MB is around the biggest I see from mkgmap without crash). So that's a more than 15 ratio. Long way to go I would say. Though it only really matters to me for continent maps, and Germany/China/Canada/USA/France maps - those are likely to push people into 2025 or 4000 tile limit on their devices if too many small tiles exist... Europe map is anyhow over the tops... (with my style-file - would need very reduced style for getting it <4GB). A range of 5-18MB would already be very nice... On 03.05.2014 09:00, Gerd Petermann wrote: Hi all, I've coded a quick hack which seems to improve the ratios. On the other hand, I don't see these large differences between smallest and largest img file. What part of the world should I try? Do you use the precomp-sea parameter in splitter? Gerd Date: Wed, 30 Apr 2014 14:59:32 +0200 From: extremecarver@gmail.com To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] mkgmap ToDo list if that doesn't seriously (more than 30-40%) slow down the splitter, I assume it would be much better... On 30.04.2014 14:06, Gerd Petermann wrote: Hi, if I got that right the number of nodes is not highly correlated to the img size, so the max-nodes value is not a good estimate. I assume the reason is that nodes which belong to roads produce a lot more bytes in the img file compared to nodes which are parts of shapes or other non-routable ways, not talking about nodes which are simply ignored by the style. So, a possible solution in splitter could be to parse the ways before reading the nodes and save all nodeids which belong to ways with highway=*. If these nodes are refered by more than one way with highway=* we assume that they will be routing nodes. These special nodes could be counted e.g. 10 times to give a better estimate. Gerd > Date: Wed, 30 Apr 2014 13:36:19 +0200 > From: osm@na1400.info > To: mkgmap-dev@lists.mkgmap.org.uk > Subject: Re: [mkgmap-dev] mkgmap ToDo list > > Multithreading the tile rendering for a single map is indeed a bit > difficult and I gave it up because you need to keep track which image > id's are already in use. Since I provide multiple maps the work-around > is running a few scripts parallel, which is also a crude form of > multithreading. > > The script language is PHP and it doesn't run on Windows without some > changes ('/' vs '\' in paths, 'rm -rf', that sort of stuff). Never tried it. > > To get a better optimum in file size, using the process I described > earlier, you could start off with a huge --max-nodes setting and then > 'search' for the highest --max-nodes that works for each specific area. > > On 30/04/2014 11:49, Felix Hartmann wrote: > > I would love if there was a possibility that you pass the used max-nodes > > value to mkgmap. > > When mkgmap is compiling the maps, then after the .img is created it > > should check > > a) did it crash due to too many max-nodes > > b) for me not important - but for others with very old GPS, etrex 10, > > ---> is tile bigger than X (usually 8) MB. > > > > if a) or b) true, then pass the file back to splitter and split with 60% > > of maxnodes - and compile the resulting .img files again. Should it fail > > again, use 40%, again 25%... Sometimes there are awful tiles, that need > > supersmall max-nodes till they compile, however lately (last 1-2 years) > > I never encountered them anymore. I think that happened rather due to a > > but in splitter/mgkmap that is fixed now. > > > > okay, you could also do this with a script, but it gets rather > > complicated to multithread it (you need to wait till mgkmap finished > > compiling all .img files - and run mkgmap first without address index to > > save time) and do some clever routines on making sure that the map id > > (e.g. 6340????.img) stay correct. Even more complicated to have > > consequent map id... > > > > For europe with a fixed max-node I get tiles from 1.9MB up to 18MB. > > That's factor 9 - so it's huge... > > If I could narrows that down easily to 8-18MB - without getting tiles > > crashing due to too high max-nodes values, that would be sweet. > > > > > > > > As for the scripts - would they run on windows too? - What programming > > language are they in? > > > > > > On 29.04.2014 21:39, osm@na1400.info wrote: > >> Oh, and ofcourse anyone interested can get my scripts, send an email. > >> They'll be on Github someday anyway. > >> > >> > >> On 2014-04-29 20:37, Gerd Petermann wrote: > >>> Hi Lambertus, > >>> > >>> okay, if I got that right you finally get *.img files with a size > >>> near (but below) 8MB, so maybe Henning can use that script, too. > >>> > >>> If you do that for e.g. Germany, how small is tpically the smallest > >>> *.img file ? > >>> Is it probably near 4 MB? > >>> > >>> Gerd > >>> > >>>> To: mkgmap-dev@lists.mkgmap.org.uk > >>>> Date: Tue, 29 Apr 2014 20:30:27 +0200 > >>>> From: osm@na1400.info > >>>> Subject: Re: [mkgmap-dev] mkgmap ToDo list > >>>> > >>>> These are the direct results from Splitter. The format is o5m, both > >>>> input as output. Splitter version is: r321. > >>>> > >>>> For this test I split the original source with --max-nodes=8000000. > >>> Then > >>>> I render the initial tiles, when the result is larger than 8MB it's > >>>> subsplit again with --max-nodes=(8000000-(attempt*100000)). The > >>> initial > >>>> source files are ~70MB (o5m) and after several subsplits the two > >>> *.img > >>>> are < 8MB. During this process --max-nodes has been reduced to e.g. > >>>> 7500000 and the source file is split up in two o5m files of about > >>> 37MB. > >>>> > >>>> I can upload an example source file and it's two subsplit siblings > >>> if > >>>> you want. > >>>> > >>>> > >>>> On 2014-04-29 19:38, GerdP wrote: > >>>> > Hi Lambertus, > >>>> > > >>>> > that's interesting. Are these the img file sizes or the osm file > >>> sizes? > >>>> > > >>>> > Gerd > >>>> > > >>>> > > >>>> > Lambertus wrote > >>>> >> Unfortunately I cannot confirm that. Below is a bit of logging > >>> from my > >>>> >> script: > >>>> >> Original: 97000020 (70551453), New: 0 (35684445), New: 1 > >>> (36852845) > >>>> >> Original: 97000001 (74621042), New: 0 (37522992), New: 1 > >>> (37222739) > >>>> >> Original: 97000002 (73391358), New: 0 (37679505), New: 1 > >>> (38098627) > >>>> >> Original: 97000003 (77862567), New: 0 (39075311), New: 1 > >>> (39261197) > >>>> >> > >>>> >> The original files above contain contour data, the filesize is > >>> between > >>>> >> brackets. As you can see both resulting file are approximately > >>> the > >>>> >> same > >>>> >> size. > >>>> >> > >>>> >> On 2014-04-29 15:39, Gerd Petermann wrote: > >>>> >>> Hi Lambertus, > >>>> >>> > >>>> >>> and I guess that even after this optimization you will > >>>> >>> see a factor 3 or higher between the largest tile and the > >>> smallest. > >>>> >>> Can you confirm that? > >>>> >>> > >>>> >>> Gerd > >>>> >>> > >>>> >>>> Date: Tue, 29 Apr 2014 15:32:38 +0200 > >>>> >>>> From: > >>>> > > >>>> >> osm@ > >>>> > > >>>> >>>> To: > >>>> > > >>>> >> mkgmap-dev@.org > >>>> > > >>>> >>>> Subject: Re: [mkgmap-dev] mkgmap ToDo list > >>>> >>>> > >>>> >>>> Num-tiles=x would indeed be better for this specific need. > >>>> >>>> > >>>> >>>> It is my experience that it regularly takes multiple calls to > >>>> >>> Splitter > >>>> >>>> to get 2+ sub-tiles when you reduce the max-nodes by 100k for > >>> each > >>>> >>>> sub-split attempt. This is what I currently do to get an > >>> optimum in > >>>> >>>> tile-size vs total number of tiles. > >>>> >>>> > >>>> >>>> > >>>> >>>> > >>>> >>>> On 29/04/2014 15:09, Gerd Petermann wrote: > >>>> >>>> > Hi Lambertus, > >>>> >>>> > > >>>> >>>> > that sounds like a possible change in splitter: > >>>> >>>> > Instead of specifying max-nodes you may specify --num-tiles=x > >>>> >>>> > and splitter will try to find a split that produces excactly > >>> x > >>>> >>> tiles > >>>> >>>> > which are not too narrow and have a node number which is not > >>>> >>>> > too far from the average (but still aligned to a multiple of > >>> map > >>>> >>> units > >>>> >>>> > as now). > >>>> >>>> > So, for your script that means you don't have to find the > >>>> >>> max-nodes > >>>> >>>> > value. > >>>> >>>> > > >>>> >>>> > I'll think about this again... > >>>> >>>> > > >>>> >>>> > Gerd > >>>> >>>> > > >>>> >>>> > > Date: Tue, 29 Apr 2014 14:59:36 +0200 > >>>> >>>> > > From: > >>>> > > >>>> >> osm@ > >>>> > > >>>> >>>> > > To: > >>>> > > >>>> >> mkgmap-dev@.org > >>>> > > >>>> >>>> > > Subject: Re: [mkgmap-dev] mkgmap ToDo list > >>>> >>>> > > > >>>> >>>> > > While this possibly can be solved in Splitter or Mkgmap, it > >>>> >>> could also > >>>> >>>> > > be solved by your build-script when you add a maximum tile > >>> size > >>>> >>> check > >>>> >>>> > > and re-split (with a lower number of max-nodes) until you > >>> get > >>>> >>> two or > >>>> >>>> > > more sub-tiles. Granted, this adds complexity to the script > >>> but > >>>> >>> it works > >>>> >>>> > > well for me. > >>>> >>>> > > > >>>> >>>> > > On 25/04/2014 21:54, Henning Scholland wrote: > >>>> >>>> > > > Hi Gerd, > >>>> >>>> > > > > >>>> >>>> > > > I would like to have img-tiles which have globally nearly > >>> the > >>>> >>> same > >>>> >>>> > > > filesize, so that they use the space of devices like > >>> eTrex 10. > >>>> >>>> > > > > >>>> >>>> > > > With my actual map I use globally the same value for > >>>> >>> max-nodes. But the > >>>> >>>> > > > size of the img-tiles differ more then factor 2. Eg. a > >>> tile in > >>>> >>> Germany > >>>> >>>> > > > is between 2 and 5 mb where a tile in China is about 10 > >>> mb. If > >>>> >>> I remove > >>>> >>>> > > > details, this difference will increase, because in > >>> Germany > >>>> >>> more objects > >>>> >>>> > > > will be removed from the img-tile then in China. > >>>> >>>> > > > > >>>> >>>> > > > Henning > >>>> >>>> > > > > >>>> >>>> > > > _______________________________________________ > >>>> >>>> > > > mkgmap-dev mailing list > >>>> >>>> > > > > >>>> > > >>>> >> mkgmap-dev@.org > >>>> > > >>>> >>>> > > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >>>> > > > >>>> >>>> > > _______________________________________________ > >>>> >>>> > > mkgmap-dev mailing list > >>>> >>>> > > > >>>> > > >>>> >> mkgmap-dev@.org > >>>> > > >>>> >>>> > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >>>> > > >>>> >>>> > > >>>> >>>> > _______________________________________________ > >>>> >>>> > mkgmap-dev mailing list > >>>> >>>> > > >>>> > > >>>> >> mkgmap-dev@.org > >>>> > > >>>> >>>> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >>>> > > >>>> >>>> > >>>> >>>> _______________________________________________ > >>>> >>>> mkgmap-dev mailing list > >>>> >>>> > >>>> > > >>>> >> mkgmap-dev@.org > >>>> > > >>>> >>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >>> > >>>> >>> _______________________________________________ > >>>> >>> mkgmap-dev mailing list > >>>> >>> > >>>> > > >>>> >> mkgmap-dev@.org > >>>> > > >>>> >>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> >> _______________________________________________ > >>>> >> mkgmap-dev mailing list > >>>> > > >>>> >> mkgmap-dev@.org > >>>> > > >>>> >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > -- > >>>> > View this message in context: > >>>> > > >>> http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5804588.html > >>>> > Sent from the Mkgmap Development mailing list archive at > >>> Nabble.com. > >>>> > _______________________________________________ > >>>> > mkgmap-dev mailing list > >>>> > mkgmap-dev@lists.mkgmap.org.uk > >>>> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>>> _______________________________________________ > >>>> mkgmap-dev mailing list > >>>> mkgmap-dev@lists.mkgmap.org.uk > >>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >>> > >>> _______________________________________________ > >>> mkgmap-dev mailing list > >>> mkgmap-dev@lists.mkgmap.org.uk > >>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >> _______________________________________________ > >> mkgmap-dev mailing list > >> mkgmap-dev@lists.mkgmap.org.uk > >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > > _______________________________________________ > mkgmap-dev mailing list > mkgmap-dev@lists.mkgmap.org.uk > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev -- keep on biking and discovering new trails Felix openmtbmap.org & www.velomap.org _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev -- keep on biking and discovering new trails Felix openmtbmap.org & www.velomap.org _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
participants (8)
-
Felix Hartmann -
Gerd Petermann -
GerdP
-
Henning Scholland -
Lambertus -
osm＠na1400.info
-
Steve Ratcliffe -
WanMil