Idea: map layers (multiple output tiles per input tile)
I got an idea today: Make the style language support layers, and allow the user to specify which layers to generate and in which output files. This would have the benefit that you could generate all output layers from a map in a single run, reducing the parsing and multipolygon processing overhead, and possibly allowing more parallel processing. The default layer (say, layer 0) would always be generated, to keep compatibility with the current behaviour. If we extended the default style so that it would generate overlay lines for bridges and tunnels on a non-default layer, these lines would be omitted by default. Someone with a fancy TYP file could invoke mkgmap with special options to generate these overlay lines, either in the same output tile, or in a separate output tile, so that the map user can enable or disable bridges and tunnels by selecting map sets on the device. Another application would be generating one map family per route relation type, so that you could choose which routes you want to see on the device: ski, bicycle, bus, road network, etc. My current --style=routes lumps all types of route relations together, not very useful. Let me illustrate this rough idea with an example. The syntax is not settled yet; I would appreciate some feedback on this. I think that I should be able to implement this, if there is no serious opposition to this idea. Consider the default style with the following addition to the beginning of the lines file: waterway=* [layer 1] boundary=* [layer 2] By default, all waterways and boundaries would be omitted from the map, because they are not in the default layer. Here is an example of such a default invocation, in the mkgmap.args format: family-id: 42 mapname: 63240001 description: City input-file: 63240001.osm.gz mapname: 63240002 description: Suburb input-file: 63240002.osm.gz Now, let us generate separate map sets for layer 1 and 2: family-id: 42 mapname: 63240001 description: City layer1-family-id: 1 layer1-mapname: 73240001 layer1-description: City Waters layer2-family-id: 2 layer2-mapname: 83240001 layer2-description: City Boundaries input-file: 63240001.osm.gz mapname: 63240002 description: Suburb layer1-family-id: 1 layer1-mapname: 73240002 layer1-description: Suburb Waters layer2-family-id: 2 layer2-mapname: 83240002 layer2-description: Suburb Boundaries input-file: 63240002.osm.gz We could also merge multiple layers to a single output tile by specifying layer3-merge: 2 or even ask mkgmap to merge everything to the default output tile: layer1-merge: default (or 0) layer2-merge: default What do you think? This scheme is only useable when it is acceptable to use the same map tiles for all map sets. If you need smaller or larger tiles for certain map layers, you would need to invoke mkgmap once for every set of input tiles. Best regards, Marko
not good. multiple layers show different top/bottom PC vs GPS . too much trouble to care for it. Use transparent asymetric objects instead. On 08.04.2011 22:19, Marko Mäkelä wrote:
I got an idea today: Make the style language support layers, and allow the user to specify which layers to generate and in which output files. This would have the benefit that you could generate all output layers from a map in a single run, reducing the parsing and multipolygon processing overhead, and possibly allowing more parallel processing.
The default layer (say, layer 0) would always be generated, to keep compatibility with the current behaviour.
If we extended the default style so that it would generate overlay lines for bridges and tunnels on a non-default layer, these lines would be omitted by default. Someone with a fancy TYP file could invoke mkgmap with special options to generate these overlay lines, either in the same output tile, or in a separate output tile, so that the map user can enable or disable bridges and tunnels by selecting map sets on the device.
Another application would be generating one map family per route relation type, so that you could choose which routes you want to see on the device: ski, bicycle, bus, road network, etc. My current --style=routes lumps all types of route relations together, not very useful.
Let me illustrate this rough idea with an example. The syntax is not settled yet; I would appreciate some feedback on this. I think that I should be able to implement this, if there is no serious opposition to this idea.
Consider the default style with the following addition to the beginning of the lines file:
waterway=* [layer 1] boundary=* [layer 2]
By default, all waterways and boundaries would be omitted from the map, because they are not in the default layer. Here is an example of such a default invocation, in the mkgmap.args format:
family-id: 42 mapname: 63240001 description: City input-file: 63240001.osm.gz mapname: 63240002 description: Suburb input-file: 63240002.osm.gz
Now, let us generate separate map sets for layer 1 and 2:
family-id: 42 mapname: 63240001 description: City layer1-family-id: 1 layer1-mapname: 73240001 layer1-description: City Waters layer2-family-id: 2 layer2-mapname: 83240001 layer2-description: City Boundaries input-file: 63240001.osm.gz mapname: 63240002 description: Suburb layer1-family-id: 1 layer1-mapname: 73240002 layer1-description: Suburb Waters layer2-family-id: 2 layer2-mapname: 83240002 layer2-description: Suburb Boundaries input-file: 63240002.osm.gz
We could also merge multiple layers to a single output tile by specifying
layer3-merge: 2
or even ask mkgmap to merge everything to the default output tile:
layer1-merge: default (or 0) layer2-merge: default
What do you think? This scheme is only useable when it is acceptable to use the same map tiles for all map sets. If you need smaller or larger tiles for certain map layers, you would need to invoke mkgmap once for every set of input tiles.
Best regards,
Marko
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Felix, On Sat, Apr 09, 2011 at 03:09:07AM +0200, Felix Hartmann wrote:
not good. multiple layers show different top/bottom PC vs GPS . too much trouble to care for it. Use transparent asymetric objects instead.
Note that the current way of doing things would not be going away. The multi-layer style could be based on the default style, just like the marine style currently is. (Heck, the marine style could be another layer in the new multi-layer style.) As layers could be assigned to map tiles when invoking mkgmap, you could merge all layers to one output tile if you wanted. My rationale for multiple layers is to allow the user to select what to see on the map. For example, bus, bicycle or ski routes are useless clutter when you are not interested in them. Typically, you would only want to see at most one type of routes at a time. A more radical example would be to be able to hide all ways that are not allowed for the currently selected mode of transportation. I am not sure how that would work; I guess that the routing network would have to be constructed from a union of all map layers then. Less clutter should mean faster display updates. For example, if landuse polygons were in a separate layer, maps would pan faster when you disable the landuse layer. I do not know about you, but I rarely need the landuse when using the street network. They can be nice for hikers or mountain bikers. When I take a shortcut through some forest paths, I could want to enable the landuse layer. How can you achieve this without using multiple layers, or without updating the map on the device? Marko
On 08.04.2011 22:19, Marko Mäkelä wrote:
I got an idea today: Make the style language support layers, and allow the user to specify which layers to generate and in which output files. This would have the benefit that you could generate all output layers from a map in a single run, reducing the parsing and multipolygon processing overhead, and possibly allowing more parallel processing.
The default layer (say, layer 0) would always be generated, to keep compatibility with the current behaviour.
If we extended the default style so that it would generate overlay lines for bridges and tunnels on a non-default layer, these lines would be omitted by default. Someone with a fancy TYP file could invoke mkgmap with special options to generate these overlay lines, either in the same output tile, or in a separate output tile, so that the map user can enable or disable bridges and tunnels by selecting map sets on the device.
Another application would be generating one map family per route relation type, so that you could choose which routes you want to see on the device: ski, bicycle, bus, road network, etc. My current --style=routes lumps all types of route relations together, not very useful.
Let me illustrate this rough idea with an example. The syntax is not settled yet; I would appreciate some feedback on this. I think that I should be able to implement this, if there is no serious opposition to this idea.
Consider the default style with the following addition to the beginning of the lines file:
waterway=* [layer 1] boundary=* [layer 2]
By default, all waterways and boundaries would be omitted from the map, because they are not in the default layer. Here is an example of such a default invocation, in the mkgmap.args format:
family-id: 42 mapname: 63240001 description: City input-file: 63240001.osm.gz mapname: 63240002 description: Suburb input-file: 63240002.osm.gz
Now, let us generate separate map sets for layer 1 and 2:
family-id: 42 mapname: 63240001 description: City layer1-family-id: 1 layer1-mapname: 73240001 layer1-description: City Waters layer2-family-id: 2 layer2-mapname: 83240001 layer2-description: City Boundaries input-file: 63240001.osm.gz mapname: 63240002 description: Suburb layer1-family-id: 1 layer1-mapname: 73240002 layer1-description: Suburb Waters layer2-family-id: 2 layer2-mapname: 83240002 layer2-description: Suburb Boundaries input-file: 63240002.osm.gz
We could also merge multiple layers to a single output tile by specifying
layer3-merge: 2
or even ask mkgmap to merge everything to the default output tile:
layer1-merge: default (or 0) layer2-merge: default
What do you think? This scheme is only useable when it is acceptable to use the same map tiles for all map sets. If you need smaller or larger tiles for certain map layers, you would need to invoke mkgmap once for every set of input tiles.
Best regards,
Marko
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
On 04/08/2011 03:19 PM, Marko Mäkelä wrote:
I got an idea today: Make the style language support layers, and allow the user to specify which layers to generate and in which output files.
You can already do this using the XAPI to pull the data you want for each layer you want to define in your own editor.
On Sat, Apr 09, 2011 at 01:14:31PM -0500, Paul Johnson wrote:
On 04/08/2011 03:19 PM, Marko Mäkelä wrote:
I got an idea today: Make the style language support layers, and allow the user to specify which layers to generate and in which output files.
You can already do this using the XAPI to pull the data you want for each layer you want to define in your own editor.
Sorry, I do not follow you. Sure, you can prepare all sorts of map extracts, and you may even filter map extracts with Osmosis. You can even filter stuff from the maps by using different mkgmap styles. Maybe I was being unclear. The use case that I have in mind is the generation of a map set that comprises several layers (families in Garmin terminology). These layers can be enabled or disabled in the map settings menu on the GPS receiver. Currently, this use case requires that mkgmap parses the input several times, once for every generated family ID, or every processed style. My proposed feature would allow mkgmap to generate multiple map tiles from the same source tile. That is, in a sense, process multiple styles in one go. The benefit should be reduced execution time. Best regards, Marko
2011/4/9 Marko Mäkelä <marko.makela@iki.fi>:
Maybe I was being unclear. The use case that I have in mind is the generation of a map set that comprises several layers (families in Garmin terminology). These layers can be enabled or disabled in the map settings menu on the GPS receiver.
Currently, this use case requires that mkgmap parses the input several times, once for every generated family ID, or every processed style.
My proposed feature would allow mkgmap to generate multiple map tiles from the same source tile. That is, in a sense, process multiple styles in one go. The benefit should be reduced execution time.
I think this is a great idea! I generate a map with seperate layers for -roads and small areas like buildings -landuse -a colored terrain model (to be able to switch between this and landuse representation) -speed limits as colored ways -hiking routes -elevation contours So today i need 4 mkgmap runs for the main osm data and 2 more for terrain and contours (I cound drop them because they don't change, but it doesn't take that long to generate those). If I could tell mkgmap "create a map from these input files using style1 and options1, style2 and options2, style3 and options3 and make a gmapsupp from the resulting files" I suspect this could save a lot of time and maybe prevent trouble with the adress index, as I would't have to use another rund to melt everything together. @Felix: I could'nt care less about mapsource not displaying stuff and don't know anyone really using it. QlandkarteGT is fine with my map, open source and runs well on all relevant platforms. -Martin
On Sun, Apr 10, 2011 at 11:26:58AM +0200, Martin Simon wrote:
I think this is a great idea! I generate a map with seperate layers for -roads and small areas like buildings -landuse -a colored terrain model (to be able to switch between this and landuse representation) -speed limits as colored ways -hiking routes -elevation contours
So today i need 4 mkgmap runs for the main osm data and 2 more for terrain and contours (I cound drop them because they don't change, but it doesn't take that long to generate those).
If I could tell mkgmap "create a map from these input files using style1 and options1, style2 and options2, style3 and options3 and make a gmapsupp from the resulting files" I suspect this could save a lot of time and maybe prevent trouble with the adress index, as I would't have to use another rund to melt everything together.
Good, now that I have a "customer" besides myself, I will start thinking about the details. For certain stuff, such as route relations, I believe that it would be easiest to generate all layers from one style. Otherwise you would need several almost-identical copies of the style files, such as "ski-route", "bus-route", "bicycle-route" and so on.
@Felix: I could'nt care less about mapsource not displaying stuff and don't know anyone really using it. QlandkarteGT is fine with my map, open source and runs well on all relevant platforms.
Same here. My only issue with QLandkarteGT is the inability to plan routes. Again, the old way of doing things would not be going away. This multi-layer stuff would remain compatible with existing styles and scripts. To take advantage of it, you would have to edit the styles and scripts. Marko
Same here. My only issue with QLandkarteGT is the inability to plan routes.
Again, the old way of doing things would not be going away. This multi-layer stuff would remain compatible with existing styles and scripts. To take advantage of it, you would have to edit the styles and scripts.
I'ld actually rather have the opposite. Get mkgmap to accept multiple input files and merge them into the "same layer". That way one could take 2 osm files, or one 1polish map, one osm, or one .img and one .osm (say one contourlines, one normal map) and join them into one map. Qlandkarte has gone a long way, but still Mapsource draws the maps better looking, quicker and autorouting and generall track planning is much quicker. Also currently Mapsource is needed for sending maps to GPS where you want to have address search. What you're proposing might be appealing to a very small minority, but for 99% of openstreetmap maps in Garmin format, they wouldn't even understand, and furthermore cause loads of support trouble. For it to work nice IMHO the following points would need to be done to Qlandkarte GT before it's really worthwile: 1. Get rid of the stupid yellow points instead of POI symbols (well that is a thing I should write to qlandkarte mailinglist) 2. implement autorouting for garmin .img maps 3. implement a WinGDB like route-/trackpoint adding mechanism so rerouting on GPS is fixed. 4. get a nice GUI to manage layers both for viewing and for selecting what to send to GPS 5. get qlandkarte GT to correctly create the address index for GPS Points 2-5 are all points that Oliver isn't interested in, so it's rather on the people wanting to have multilayered maps to implement it into Qlandkarte GT.
2011/4/10 Felix Hartmann <extremecarver@gmail.com>:
Same here. My only issue with QLandkarteGT is the inability to plan routes.
Again, the old way of doing things would not be going away. This multi-layer stuff would remain compatible with existing styles and scripts. To take advantage of it, you would have to edit the styles and scripts.
I'ld actually rather have the opposite. Get mkgmap to accept multiple input files and merge them into the "same layer". That way one could take 2 osm files, or one 1polish map, one osm, or one .img and one .osm (say one contourlines, one normal map) and join them into one map.
What's the benefit?
Qlandkarte has gone a long way, but still Mapsource draws the maps better looking, quicker and autorouting and generall track planning is much quicker.
On Linux? Serious question - when I get it to run on wine, it's dead slow and draws ugly, blurred tiles.
Also currently Mapsource is needed for sending maps to GPS where you want to have address search.
Right, but only few people take this workaround just to have half-working addresses on their devices, I guess.
What you're proposing might be appealing to a very small minority, but for 99% of openstreetmap maps in Garmin format, they wouldn't even understand, and furthermore cause loads of support trouble.
What are you talking about? Map users? map authors? What support? What kind of trouble? Cool statistics. My own guess: "99%" of users just want to throw a gmapsupp.img onto their device and use that. "99%" of the remaining 1% don't care if mapsource draws all layers correctly. ;-)
For it to work nice IMHO the following points would need to be done to Qlandkarte GT before it's really worthwile: 1. Get rid of the stupid yellow points instead of POI symbols (well that is a thing I should write to qlandkarte mailinglist)
No yellow points here. Symbols from the .TYP file are displayed correctly.
2. implement autorouting for garmin .img maps
Would be nice, but there's already openrouteservice.org integration for a start...
3. implement a WinGDB like route-/trackpoint adding mechanism so rerouting on GPS is fixed.
I don't understand that one...
4. get a nice GUI to manage layers both for viewing and for selecting what to send to GPS
Yes, would be nice, but not really necessary...
5. get qlandkarte GT to correctly create the address index for GPS
Would be better to have this in mkgmap, don't you think?
Points 2-5 are all points that Oliver isn't interested in, so it's rather on the people wanting to have multilayered maps to implement it into Qlandkarte GT.
It's on the people who want these features, I, personally, can live comfortably without them. I (and many others) could even live entirely without programs like mapsource and Qlandkarte - my usage of OSM Garmin maps for 99% takes place on the device, outdoors. Is it just me or do you try to prevent a feature just because *you* have no use for it? -Martin
On 10.04.2011 14:22, Martin Simon wrote:
2011/4/10 Felix Hartmann<extremecarver@gmail.com>:
Same here. My only issue with QLandkarteGT is the inability to plan routes.
Again, the old way of doing things would not be going away. This multi-layer stuff would remain compatible with existing styles and scripts. To take advantage of it, you would have to edit the styles and scripts.
I'ld actually rather have the opposite. Get mkgmap to accept multiple input files and merge them into the "same layer". That way one could take 2 osm files, or one 1polish map, one osm, or one .img and one .osm (say one contourlines, one normal map) and join them into one map. What's the benefit?
No need to use osmosis beforehand to join contourlines to .osm data.
Qlandkarte has gone a long way, but still Mapsource draws the maps better looking, quicker and autorouting and generall track planning is much quicker. On Linux? Serious question - when I get it to run on wine, it's dead slow and draws ugly, blurred tiles.
On both Ubuntu (not using Wine of course) and on Windows 7.
Also currently Mapsource is needed for sending maps to GPS where you want to have address search. Right, but only few people take this workaround just to have half-working addresses on their devices, I guess.
What you're proposing might be appealing to a very small minority, but for 99% of openstreetmap maps in Garmin format, they wouldn't even understand, and furthermore cause loads of support trouble. What are you talking about? Map users? map authors? What support? What kind of trouble?
Cool statistics. My own guess: "99%" of users just want to throw a gmapsupp.img onto their device and use that. "99%" of the remaining 1% don't care if mapsource draws all layers correctly. ;-)
I'ld say most users use mapsource to send maps to GPS. Especially for all old generation GPS you are forced to use a prog to assemble the map beforehand, as they only accept a single gmapsupp.img, and most users I know have several maps on their GPS. Also I think for most users tripplanning beforehand is more important than the actual map on GPS itself.
For it to work nice IMHO the following points would need to be done to Qlandkarte GT before it's really worthwile: 1. Get rid of the stupid yellow points instead of POI symbols (well that is a thing I should write to qlandkarte mailinglist) No yellow points here. Symbols from the .TYP file are displayed correctly. Strange, I always get yellow points until zooming in very close. I suppose this is a mechanism taken to get rid of POI from maps without .TYPfile, or as general far too big POI symbols to show up useful - but no good at all if the .TYP-file is well done. 2. implement autorouting for garmin .img maps Would be nice, but there's already openrouteservice.org integration for a start... Well and that is in no way as useful for outdoor activities. And for motorcar navigation no OSM map can compete with Garmin City Navigator maps (the problem is more in OSM data though than in mkgmap). 3. implement a WinGDB like route-/trackpoint adding mechanism so rerouting on GPS is fixed. I don't understand that one... Well than google WinGDB and read up on what it can do. For me for tripplanning it's essential. 4. get a nice GUI to manage layers both for viewing and for selecting what to send to GPS Yes, would be nice, but not really necessary... only as long as you don't offer maps publicly. I wouldn't want to support it as myriads of users don't read any instructions and many are not even able to do so (if it ain't a video on youtube, they are lost). 5. get qlandkarte GT to correctly create the address index for GPS Would be better to have this in mkgmap, don't you think? Well both should have it. Mkgmap for gmapsupp creation, Qlandkarte GT for creating index for sending maps to GPS. Points 2-5 are all points that Oliver isn't interested in, so it's rather on the people wanting to have multilayered maps to implement it into Qlandkarte GT. It's on the people who want these features, I, personally, can live comfortably without them. I (and many others) could even live entirely without programs like mapsource and Qlandkarte - my usage of OSM Garmin maps for 99% takes place on the device, outdoors.
Is it just me or do you try to prevent a feature just because *you* have no use for it? I'ld have use for it, when the problems I listed are solved. Before I think it only makes life more complicated. There are simply too many things users will do to completly break it). -Martin _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
El 10/04/11 14:37, Felix Hartmann escribió:
On 10.04.2011 14:22, Martin Simon wrote:
2011/4/10 Felix Hartmann<extremecarver@gmail.com>:
Qlandkarte has gone a long way, but still Mapsource draws the maps better looking, quicker and autorouting and generall track planning is much quicker.
On Linux? Serious question - when I get it to run on wine, it's dead slow and draws ugly, blurred tiles.
On both Ubuntu (not using Wine of course) and on Windows 7.
I use MapSource on debian+wine and don't find it to be slow or draw ugly.
2. implement autorouting for garmin .img maps
Would be nice, but there's already openrouteservice.org integration for a start...
Well and that is in no way as useful for outdoor activities. And for motorcar navigation no OSM map can compete with Garmin City Navigator maps (the problem is more in OSM data though than in mkgmap).
I completely disagree with you. City Navigator contains lots of errors in the data (that you can't fix as you can with OSM) and makes you travel longer/slower/wrong routes in many cases. I have used my OSM maps in Spain with better routing than when I used garmin CN maps in the past. I also used them in Switzerland, Portugal and Morocco without problems, provided the area you are traveling is enough covered by OSM (beforehand checking in MapSource is essential in that cases).
Am 10.04.2011 13:45, schrieb Felix Hartmann:
I'ld actually rather have the opposite. Get mkgmap to accept multiple input files and merge them into the "same layer". That way one could take 2 osm files, or one 1polish map, one osm, or one .img and one .osm (say one contourlines, one normal map) and join them into one map. It's already possible to combine *.img and osm-files in one step. I use this for processing a new map (input osm-tilesfrom splitter) with the old contour-tiles. Works fine. Result is a working map for MapSource and a working gmapsupp.img.
Henning
not good. multiple layers show different top/bottom PC vs GPS . too much trouble to care for it. Use transparent asymetric objects instead. If you don't follow the whack of needing europewide maps, but just export the region you really need to your GPS, rather install different maps onto your PC. Because multilayered maps are not properly workin on PC (dunno about Qlandkarte GT, I still primarily plan with Mapsource 6.16) Multilayered maps mean that if you plan with Basecamp/Mapsource, you need different maps for your GPS. On 08.04.2011 22:19, Marko Mäkelä wrote:
I got an idea today: Make the style language support layers, and allow the user to specify which layers to generate and in which output files. This would have the benefit that you could generate all output layers from a map in a single run, reducing the parsing and multipolygon processing overhead, and possibly allowing more parallel processing.
The default layer (say, layer 0) would always be generated, to keep compatibility with the current behaviour.
If we extended the default style so that it would generate overlay lines for bridges and tunnels on a non-default layer, these lines would be omitted by default. Someone with a fancy TYP file could invoke mkgmap with special options to generate these overlay lines, either in the same output tile, or in a separate output tile, so that the map user can enable or disable bridges and tunnels by selecting map sets on the device.
Another application would be generating one map family per route relation type, so that you could choose which routes you want to see on the device: ski, bicycle, bus, road network, etc. My current --style=routes lumps all types of route relations together, not very useful.
Let me illustrate this rough idea with an example. The syntax is not settled yet; I would appreciate some feedback on this. I think that I should be able to implement this, if there is no serious opposition to this idea.
Consider the default style with the following addition to the beginning of the lines file:
waterway=* [layer 1] boundary=* [layer 2]
By default, all waterways and boundaries would be omitted from the map, because they are not in the default layer. Here is an example of such a default invocation, in the mkgmap.args format:
family-id: 42 mapname: 63240001 description: City input-file: 63240001.osm.gz mapname: 63240002 description: Suburb input-file: 63240002.osm.gz
Now, let us generate separate map sets for layer 1 and 2:
family-id: 42 mapname: 63240001 description: City layer1-family-id: 1 layer1-mapname: 73240001 layer1-description: City Waters layer2-family-id: 2 layer2-mapname: 83240001 layer2-description: City Boundaries input-file: 63240001.osm.gz mapname: 63240002 description: Suburb layer1-family-id: 1 layer1-mapname: 73240002 layer1-description: Suburb Waters layer2-family-id: 2 layer2-mapname: 83240002 layer2-description: Suburb Boundaries input-file: 63240002.osm.gz
We could also merge multiple layers to a single output tile by specifying
layer3-merge: 2
or even ask mkgmap to merge everything to the default output tile:
layer1-merge: default (or 0) layer2-merge: default
What do you think? This scheme is only useable when it is acceptable to use the same map tiles for all map sets. If you need smaller or larger tiles for certain map layers, you would need to invoke mkgmap once for every set of input tiles.
Best regards,
Marko
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Moin, since my GPS-maps rely heavily on multiple layers, there would be some benefit, if producing of these layer could happen within a single mkgmap run, so that the common proccesing steps would be executed only once. Whether this can be achieved, depends on the time of the processing step. I would guess, that most steps are not performed on the whole OSM input data, but only on the output data of the styled converter (there is no need to process any data not used in the map). So if you want to use multiple styles in parallel, most of the processing must be done anyway multiple times. I don't like the proposal to put the multiple layer support into the style files, I think such a mechanism should be controlled via command line parameters, so that you could the same styles either for single layers or for a multiple layers mkgmap run. I would like an extension, which would allow such a usage of mkgmap: input.osm -> style-A, parameters-A -> outputA.img -> style-B, parameters-B -> outputB.img Gruss Torsten
Marko, I like the idea to use the same tile data in one mkgmap run to create multiple img files. At least it is worth to discuss it, if it makes mkgmap faster. Before starting to implement it some things have to be considered: * The advantage to create multiple img files in one mkgmap run is that parsing and preparing of the OSM data must happen once only. Do you have numbers how many percent of the time is used for these steps? Please be aware that mkgmap is optimized in such a way that it loads only the tags which are needed in the style file. * I think the layer concept will be too complicated. I would prefer to have multiple styles (as Torsten Leistikow proposed in a seperate mail). * From my point of view it won't be possible to run mkgmap with different options for the different styles (or layers in your concept). Will this remove the timing advantages? Thanks for your idea! WanMil
I got an idea today: Make the style language support layers, and allow the user to specify which layers to generate and in which output files. This would have the benefit that you could generate all output layers from a map in a single run, reducing the parsing and multipolygon processing overhead, and possibly allowing more parallel processing.
The default layer (say, layer 0) would always be generated, to keep compatibility with the current behaviour.
If we extended the default style so that it would generate overlay lines for bridges and tunnels on a non-default layer, these lines would be omitted by default. Someone with a fancy TYP file could invoke mkgmap with special options to generate these overlay lines, either in the same output tile, or in a separate output tile, so that the map user can enable or disable bridges and tunnels by selecting map sets on the device.
Another application would be generating one map family per route relation type, so that you could choose which routes you want to see on the device: ski, bicycle, bus, road network, etc. My current --style=routes lumps all types of route relations together, not very useful.
Let me illustrate this rough idea with an example. The syntax is not settled yet; I would appreciate some feedback on this. I think that I should be able to implement this, if there is no serious opposition to this idea.
Consider the default style with the following addition to the beginning of the lines file:
waterway=* [layer 1] boundary=* [layer 2]
By default, all waterways and boundaries would be omitted from the map, because they are not in the default layer. Here is an example of such a default invocation, in the mkgmap.args format:
family-id: 42 mapname: 63240001 description: City input-file: 63240001.osm.gz mapname: 63240002 description: Suburb input-file: 63240002.osm.gz
Now, let us generate separate map sets for layer 1 and 2:
family-id: 42 mapname: 63240001 description: City layer1-family-id: 1 layer1-mapname: 73240001 layer1-description: City Waters layer2-family-id: 2 layer2-mapname: 83240001 layer2-description: City Boundaries input-file: 63240001.osm.gz mapname: 63240002 description: Suburb layer1-family-id: 1 layer1-mapname: 73240002 layer1-description: Suburb Waters layer2-family-id: 2 layer2-mapname: 83240002 layer2-description: Suburb Boundaries input-file: 63240002.osm.gz
We could also merge multiple layers to a single output tile by specifying
layer3-merge: 2
or even ask mkgmap to merge everything to the default output tile:
layer1-merge: default (or 0) layer2-merge: default
What do you think? This scheme is only useable when it is acceptable to use the same map tiles for all map sets. If you need smaller or larger tiles for certain map layers, you would need to invoke mkgmap once for every set of input tiles.
Best regards,
Marko
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
On Sun, Apr 10, 2011 at 07:42:53PM +0200, WanMil wrote:
* The advantage to create multiple img files in one mkgmap run is that parsing and preparing of the OSM data must happen once only. Do you have numbers how many percent of the time is used for these steps?
No, I haven't collected any profiling data yet. Which tool would you recommend? For C and C++, which is what I have mainly been developing in, I have been using OProfile and before that, gprof. Which Java tools would come closest?
Please be aware that mkgmap is optimized in such a way that it loads only the tags which are needed in the style file.
Based on the log files, multipolygon relations seem to be processed even though the style does not contain any rules for polygons. Could you fix this?
* I think the layer concept will be too complicated. I would prefer to have multiple styles (as Torsten Leistikow proposed in a seperate mail).
Yes, doing it with styles would be better from the compatibility point of view. My example of "ski routes", "bus routes", "hiking routes", "mtb routes" could be implemented with a parameterized style, something like -style=routes --style-param route=bus.
* From my point of view it won't be possible to run mkgmap with different options for the different styles (or layers in your concept). Will this remove the timing advantages?
Why wouldn't it be possible to use style-specific options? I think that there could be a command line option for associating subsequent options with an output file. Something like this: java -jar mkgmap.jar \ --output-file 10000001.img -c family1.args \ --output-file 20000001.img -c family2.args \ --output-file 30000001.img -c family3.args \ --input-file 10000001.osm.gz \ --output-file 10000002.img -c family1.args \ --output-file 20000002.img -c family2.args \ --output-file 30000002.img -c family3.args \ --input-file 10000002.osm.gz \ and so on, for every input file. Basically, we would have a producer (input file parser) and several consumers (img file generators). Each img file generator would have its own set of options. Currently, another problem with multiple styles (apart from the alleged parsing overhead) is that the processing cannot be parallelized in an optimal way. For example, if I have an N-core machine, --max-jobs does the right thing for splitter and each mkgmap run, but my script would invoke mkgmap several times in succession, once for each output style. If I have M output tiles so that M is not an integer multiple of N, some cores would be sitting idle when a mkgmap run is close to completion. Best regards, Marko
On Sun, Apr 10, 2011 at 07:42:53PM +0200, WanMil wrote:
* The advantage to create multiple img files in one mkgmap run is that parsing and preparing of the OSM data must happen once only. Do you have numbers how many percent of the time is used for these steps?
No, I haven't collected any profiling data yet. Which tool would you recommend? For C and C++, which is what I have mainly been developing in, I have been using OProfile and before that, gprof. Which Java tools would come closest?
I would add some time logging to the mkgmap source code. There are profiles for Java but I think they are not useful for such a job.
Please be aware that mkgmap is optimized in such a way that it loads only the tags which are needed in the style file.
Based on the log files, multipolygon relations seem to be processed even though the style does not contain any rules for polygons. Could you fix this?
Yes that's an additional optimization. I can add an easy solution: Do not process a multipolygon if the mp does not contain any additional tag (other than type=multipolygon) and if no member way contains any tag.
* I think the layer concept will be too complicated. I would prefer to have multiple styles (as Torsten Leistikow proposed in a seperate mail).
Yes, doing it with styles would be better from the compatibility point of view. My example of "ski routes", "bus routes", "hiking routes", "mtb routes" could be implemented with a parameterized style, something like -style=routes --style-param route=bus.
* From my point of view it won't be possible to run mkgmap with different options for the different styles (or layers in your concept). Will this remove the timing advantages?
Why wouldn't it be possible to use style-specific options? I think that there could be a command line option for associating subsequent options with an output file. Something like this:
java -jar mkgmap.jar \ --output-file 10000001.img -c family1.args \ --output-file 20000001.img -c family2.args \ --output-file 30000001.img -c family3.args \ --input-file 10000001.osm.gz \ --output-file 10000002.img -c family1.args \ --output-file 20000002.img -c family2.args \ --output-file 30000002.img -c family3.args \ --input-file 10000002.osm.gz \ and so on, for every input file.
The problem arises with options that change the processing before the style come into play. Maybe that's a small number of options only: family1.args: generate-sea=multipolygon family2.args: generate-sea=polygon or family1.args: remove-short-arcs=3 family2.args: remove-short-arcs=-1 One need to check the complete list of options which will cause a problem.
Basically, we would have a producer (input file parser) and several consumers (img file generators). Each img file generator would have its own set of options.
Currently, another problem with multiple styles (apart from the alleged parsing overhead) is that the processing cannot be parallelized in an optimal way. For example, if I have an N-core machine, --max-jobs does the right thing for splitter and each mkgmap run, but my script would invoke mkgmap several times in succession, once for each output style. If I have M output tiles so that M is not an integer multiple of N, some cores would be sitting idle when a mkgmap run is close to completion.
I usually have more problems with memory limitations than CPU limitations (of course probably that's due to my hardware configuration). Your example with the N-core machine is not completely correct. You assume that each tile needs a fixed time T for compilation, so that you have some cores sitting idle at the end. But that's not a real example. Each tile needs a different time T(x) for compilation and so the CPU consumption decreases from 100% to 100%/N while compiling the last N tiles. After this the index and the other stuff (gmapsupp, nsis file etc.) is created with a single CPU only. I assume your new idea would not improve this point.
Best regards,
Marko
Have fun! WanMil
On Sun, Apr 10, 2011 at 09:18:00PM +0200, WanMil wrote:
On Sun, Apr 10, 2011 at 07:42:53PM +0200, WanMil wrote:
* The advantage to create multiple img files in one mkgmap run is that parsing and preparing of the OSM data must happen once only. Do you have numbers how many percent of the time is used for these steps?
No, I haven't collected any profiling data yet. Which tool would you recommend? For C and C++, which is what I have mainly been developing in, I have been using OProfile and before that, gprof. Which Java tools would come closest?
I would add some time logging to the mkgmap source code. There are profiles for Java but I think they are not useful for such a job.
I made an experiment. Generating the full map with the default style from 9 osm.gz tiles on a dual-core machine took some 7 minutes, with assertions enabled. With assertions disabled, generating a routes layer of the whole Finland takes about 2.5 minutes when parsing the whole country extract in osm.pbf (one input tile, thus one core used). I repeated this test for 5 layers, which I split from the currently rather useless --style=routes. These styles were routes-foot (route=foot|route=hiking), routes-bicycle, routes-bus, routes-rail (route=subway|route=train|route=tram), and routes-ferry. I did not generate routes-ski or routes-road. Each of these 5 minimal styles would spit out a number of multipolygon error messages, even though the styles are this simple: cat > relations << EOF type=route & route=ferry { add ref='${name}'; # if ref is missing, use name set ref='${network} ${ref}' | '${ref}' | '${network}'; apply { set mkgmap:route='$(mkgmap:route),${ref}' | '${ref}' } } EOF cat > lines << EOF mkgmap:route=* { name '${mkgmap:route}' } [0x1b resolution 20] EOF I think that 2.5 minutes for these simple layers is too much. Because the processing time per layer did not vary much, no matter how much data was generated (13312 bytes .img of ferry routes, 986624 bytes of bus routes), I would guess that parsing is dominating the execution time, and the processing time would reduce from N*2.5 minutes to about 2.5 minutes if we implemented a 'multiple output tiles from one input tile' feature. Do you have any hints where to add the "poor man's profiler" timestamp printouts to the mkgmap source code? Marko
On Sat, Apr 16, 2011 at 12:23:03AM +0300, Marko Mäkelä wrote:
I think that 2.5 minutes for these simple layers is too much. Because the processing time per layer did not vary much, no matter how much data was generated (13312 bytes .img of ferry routes, 986624 bytes of bus routes), I would guess that parsing is dominating the execution time, and the processing time would reduce from N*2.5 minutes to about 2.5 minutes if we implemented a 'multiple output tiles from one input tile' feature.
With --ignore-builtin-relations, the time reduces to N*2 minutes, give or take a couple of seconds per layer. I will next try how I could generate multiple type=route relation tiles at once (split by route=*), using a single parsing stage. Marko
Am 15.04.2011 23:23, schrieb Marko Mäkelä:
On Sun, Apr 10, 2011 at 09:18:00PM +0200, WanMil wrote:
On Sun, Apr 10, 2011 at 07:42:53PM +0200, WanMil wrote:
* The advantage to create multiple img files in one mkgmap run is that parsing and preparing of the OSM data must happen once only. Do you have numbers how many percent of the time is used for these steps?
No, I haven't collected any profiling data yet. Which tool would you recommend? For C and C++, which is what I have mainly been developing in, I have been using OProfile and before that, gprof. Which Java tools would come closest?
I would add some time logging to the mkgmap source code. There are profiles for Java but I think they are not useful for such a job.
I made an experiment. Generating the full map with the default style from 9 osm.gz tiles on a dual-core machine took some 7 minutes, with assertions enabled. With assertions disabled, generating a routes layer of the whole Finland takes about 2.5 minutes when parsing the whole country extract in osm.pbf (one input tile, thus one core used). I repeated this test for 5 layers, which I split from the currently rather useless --style=routes. These styles were routes-foot (route=foot|route=hiking), routes-bicycle, routes-bus, routes-rail (route=subway|route=train|route=tram), and routes-ferry. I did not generate routes-ski or routes-road.
Each of these 5 minimal styles would spit out a number of multipolygon error messages, even though the styles are this simple:
cat> relations<< EOF type=route& route=ferry { add ref='${name}'; # if ref is missing, use name set ref='${network} ${ref}' | '${ref}' | '${network}'; apply { set mkgmap:route='$(mkgmap:route),${ref}' | '${ref}' } } EOF cat> lines<< EOF mkgmap:route=* { name '${mkgmap:route}' } [0x1b resolution 20] EOF
I think that 2.5 minutes for these simple layers is too much. Because the processing time per layer did not vary much, no matter how much data was generated (13312 bytes .img of ferry routes, 986624 bytes of bus routes), I would guess that parsing is dominating the execution time, and the processing time would reduce from N*2.5 minutes to about 2.5 minutes if we implemented a 'multiple output tiles from one input tile' feature.
Do you have any hints where to add the "poor man's profiler" timestamp printouts to the mkgmap source code?
Marko
I would like to add some profiling code to mkgmap. My idea is, that it is possible to add a static class that collects profiling information from mkgmap classes. The classes can call: Profiler.start(String stepname) and Profiler.stop(String stepname). At the end of mkgmap it creates a statistic file how many time is spent in each step. Of course the profiler is active only if a specific option is set. This could be used to track the processing time of mkgmap with subsequent commits. Good idea? Bad idea? Any more ideas? WanMil
On Fri, Apr 08, 2011 at 11:19:17PM +0300, Marko Mäkelä wrote:
I got an idea today: Make the style language support layers, and allow the user to specify which layers to generate and in which output files. This would have the benefit that you could generate all output layers from a map in a single run, reducing the parsing and multipolygon processing overhead, and possibly allowing more parallel processing.
I did some research on this. It looks like the concept of map layer or output map file would have to be added to MapCollector, MapDetails and GType. Currently, I generate 5 nearly identical, very sparse layers from finland.osm.pbf. These use the styles routes-{foot,bicycle,bus,rail,ferry}. These styles could be merged into one if we had the concept of multiple output .img files per input tile. Each layer takes about 2.5 minutes to generate. The time is reduced to 2 minutes per tile if I patch the code to ignore multipolygon relations. Processing a "null style" (one with no action rules) takes about the same time as processing one of these route styles. This leads me to believe that the processing time of the 5 map layers could be reduced from 5*2.5 = 12.5 minutes to 2 minutes, or 1/6 of the original. Thoughts? Suggestions regarding the implementation? Marko
participants (8)
-
Carlos Dávila -
Felix Hartmann -
Henning Scholland -
Marko Mäkelä -
Martin Simon -
Paul Johnson -
Torsten Leistikow -
WanMil