Re: [mkgmap-dev] Missing ways part 1
Marko wrote:
If I remember previous discussions correctly, splitter does not split lines exactly at the tile border, but instead it includes all nodes that are not too much outside the tile.
That is also what I understood. Splitter includes in each tile, the nodes that fall within the tile, and also the nodes in an overlap region around the tile. The width of the overlap can be set with --overlap.
An almost-straight long way can be drawn with few nodes. You could help the situation by adding a point to the line close to the splitting border.
I would expect it to cause a problem, if a particular way does not have a node in the overlap region. The difficulty with your fix is that it would only work for me (and only if I use the same areas.list each time); it will not work for other users. Also, it will not work for me in the long term, because data is continuously added to OSM, so the tiles need to be made smaller as time goes on. As I understand it, splitter is intended to clip ways at the last included node. [This is done to avoid an additional pass through the input file.] If there is no node in the overlap region, I would expect the way to stop short of the bounding box. In the finished map, I would expect to see a gap in the way at the tile boundary. The maximum length of the gap would be one segment, between adjacent nodes. What I am actually seeing is that the way is entirely lost from both tiles. When inspecting the tiles in JOSM, I can see that all the nodes are present, including those from the missing way. But few of the nodes in the overlap region have ways attached. So it appears that splitter is not working as intended.
On Mon, Oct 18, 2010 at 3:38 PM, Adrian <ar2988-os@yahoo.co.uk> wrote:
Marko wrote:
If I remember previous discussions correctly, splitter does not split lines exactly at the tile border, but instead it includes all nodes that are not too much outside the tile.
That is also what I understood. Splitter includes in each tile, the nodes that fall within the tile, and also the nodes in an overlap region around the tile. The width of the overlap can be set with --overlap.
An almost-straight long way can be drawn with few nodes. You could help the situation by adding a point to the line close to the splitting border.
I would expect it to cause a problem, if a particular way does not have a node in the overlap region. The difficulty with your fix is that it would only work for me (and only if I use the same areas.list each time); it will not work for other users. Also, it will not work for me in the long term, because data is continuously added to OSM, so the tiles need to be made smaller as time goes on.
If the overlap region is always bigger than the largest internode spacing between two nodes in a way, then this shouldn't be a problem. Decide a distance, say, 20km, and then set the overlap to be that distance. Finding and fixing ways that violate the rule can be done by a bot. As a workaround, until the mkgmap problem is diagnosed, increase the overlap. Check out the crosby_integration splitter branch. It can correctly handle large overlaps. Scott
The only thing that seems to work for me on Australia if I use the default style is overlap of 50000. If I use any thing less I get lines of maritime boundaries drawing all over the place and have some missing ways. I even tried to add extra nodes on the maritime booundaries so I could decrease the overlap, but this didn't work. It's almost like if the tile size is larger than the overlap it will cause problems even though on the input file nodes are on the tile. Markus. _____ From: mkgmap-dev-bounces@lists.mkgmap.org.uk [mailto:mkgmap-dev-bounces@lists.mkgmap.org.uk] On Behalf Of Scott Crosby Sent: Saturday, 23 October 2010 5:39 AM To: ar2988-os@yahoo.co.uk; Development list for mkgmap Subject: Re: [mkgmap-dev] Missing ways part 1 On Mon, Oct 18, 2010 at 3:38 PM, Adrian <ar2988-os@yahoo.co.uk> wrote: Marko wrote:
If I remember previous discussions correctly, splitter does not split lines exactly at the tile border, but instead it includes all nodes that are not too much outside the tile.
That is also what I understood. Splitter includes in each tile, the nodes that fall within the tile, and also the nodes in an overlap region around the tile. The width of the overlap can be set with --overlap.
An almost-straight long way can be drawn with few nodes. You could help the situation by adding a point to the line close to the splitting border.
I would expect it to cause a problem, if a particular way does not have a node in the overlap region. The difficulty with your fix is that it would only work for me (and only if I use the same areas.list each time); it will not work for other users. Also, it will not work for me in the long term, because data is continuously added to OSM, so the tiles need to be made smaller as time goes on. If the overlap region is always bigger than the largest internode spacing between two nodes in a way, then this shouldn't be a problem. Decide a distance, say, 20km, and then set the overlap to be that distance. Finding and fixing ways that violate the rule can be done by a bot. As a workaround, until the mkgmap problem is diagnosed, increase the overlap. Check out the crosby_integration splitter branch. It can correctly handle large overlaps. Scott
On 23/10/10, Markus_g wrote:
The only thing that seems to work for me on Australia if I use the default style is overlap of 50000. If I use any thing less I get lines of maritime boundaries drawing all over the place and have some missing ways.
I've had a look at this. I followed the recipe in Markus_g's earlier message: http://www.mkgmap.org.uk/pipermail/mkgmap-dev/2010q3/009082.html I first looked at the coastline and did not find any problems. [While looking at the coastline, I did however find some multipolygons where I had doubts about the tagging or structure. In each case, one of the outer ways is a coastline way. I do not feel it is appropriate for me to edit Australian map data, so here is a list for anyone who may wish to take this up. rel. id coast way id way reversed 370821 47033944 548247 54777428 1216240 82155137 y 1231296 82163390 1237298 82378497 y 1240053 82488289 y] I then realised that Markus_g was talking about the line marking the 12-mile limit of territorial waters. Using the default overlap in splitter and the default style, I saw what Markus_g described. Apart from a small gap in the line, all the problems are in tile #3, in the vicinity of Tasmania. (See the attached areas.list) This is my analysis. The problems relate to four ways, all tagged boundary=administrative: way id admin_level rel. memberships role 62201729 2 Australia, Tasmania both outer 31509803 2 Australia, Tasmania both outer 31508204 2 Australia, Victoria both outer 62201730 2 Australia, Tasmania, 966803(timezone) all outer and three multipolygon relations, all tagged boundary=administrative: rel id admin_level name 80500 2 Australia 85104 4 Tasmania 80371 4 Victoria Tile #1 is to the east of tile #3 and the boundary between them is at 147.48°E. Way 62201729 starts and finishes in tile #3 but makes two excursions into tile #1, so it crosses the boundary four times, although one excursion does not go beyond the overlap region of tile #3. Way 31509803 starts and finishes in tile #3 and makes one excursion into tile #1, going beyond the overlap region of tile #3. Way 31508204 starts in tile #3 and finishes in tile #1. Way 62201730 is entirely within tile #3. There are two issues as Markus_g described, stray lines and missing ways; plus a small gap. Splitter is working as intended, at least with the default overlap. The stray lines could be changed to gaps by a modification to mkgmap. The stray lines and/or gaps are inherent to the way splitter works, but are readily fixed by a modest increase in the overlap, or by increasing the density of points in the map. Stray lines are caused by this combination of circumstances: a) A way leaves a tile, going beyond the overlap region, b) The way has no node in the overlap region, and c) The way re-enters the overlap region or the tile. The stray line is drawn from the last node in the tile before a), towards the first node after c), finishing at the tile boundary. E.g. for way 62201729, from node 352667234 towards node 352667223; and for way 31509803, from node 352667561 towards node 352650024. The two nodes are not consecutive nodes of the way. Mkgmap must have been intentionally coded to draw lines between non-consecutive nodes, passing over nodes whose lat-long is missing. Mkgmap would 'know' it has passed over a node, because mkgmap must work from the sequence of node ids, and the tile always contains the complete sequence for a way. Drawing the lines might be the best policy when both nodes are in the tile. But when one of the nodes is outside the tile, mkgmap could leave a gap rather than drawing a stray line. Mkgmap already 'knows' that one of the nodes is outside the tile, because it is finishing the line at the tile boundary. It just needs to detect the combination of a) passing over a node, and b) one node outside the tile. For the missing ways, I will describe exactly what is seen in the Garmin map. At zoom settings of 200mi or more, nothing is displayed. At zoom settings from 150mi to 10mi, ways with admin_level=2 are displayed. This is set in the style. At zoom settings of 7mi or less, ways with admin_level=2 or 4 are displayed. In some cases the ways have two labels. My interpretation of this is that there are two superimposed ways. In tiles except #3, it is like these ways in tile #1: way id label in zoom 150mi labels in zoom 7mi 62201729 "Australia" "Australia", "Tasmania" 31509803 "Australia" "Australia", "Tasmania" 31508204 "Australia" "Australia", "Victoria" In tile #3: way id label in zoom 150mi label(s) in zoom 7mi 62201729 [not rendered] "Tasmania" 31509803 [not rendered] "Tasmania" 31508204 "Victoria/Australia" "Victoria/Australia" 62201730 "Intl. Border" "Intl. Border", "Tasmania" There is another way, 31508201, which crosses from tile #3 into tile #10, and comes out the same as way 31508204. My conclusion from these observations is that multipolygon "Australia" is not rendered in tile #3, and this has different effects on different ways, depending on the number of times that the way crosses the tile boundary. Two or more crossings - the way is not rendered. One crossing - the way is rendered as an independent way, but acquires a label combining the names of both parent multipolygons. No crossings - the way is rendered as an independent way but receives a default label. It also appears that multipolygon "Victoria" is not rendered in tile #3. So I have broken down the interaction between a) the cutting of the map into tiles, b) the multipolygon processing, and c) the style. It will have to be for others more familiar with the software, to explain exactly why the observed behaviour is happening, and to say whether it is fixable. The tagging of boundaries in OSM is not straightforward and I have not studied the details. So I cannot say whether these Australian boundaries are correctly tagged. But in my opinion the results with mkgmap are satisfactory except in tile #3, so I would hope that it is fixable in mkgmap. Markus_g found that --overlap=50000 in splitter solved the problems. I believe this is simply the threshold, at which all of way 31509803 is included in the overlap region of tile #3. Presumably this results in multipolygon "Australia" being rendered, and that solves the problems. I have prepared an .osm file of Markus_g's bounding box, containing just the ways tagged admin_level=2 or 4, and the parent relations of those ways (but none of the other ways referred to in the relations). This reduced amount of data makes testing much easier because the file (and individual tiles) can be quickly inspected in JOSM, and it takes only a few seconds to make the Garmin map. The file is 14MB in .osm format and 0.4MB in .pbf format. If I am advised that this is a good thing to do, I can e-mail the .pbf to this list. I am also willing to e-mail the file to individuals on request (provided there are not too many!) [The observations above, are for the reduced data set. With the full data set, the results are slightly different. The difference arises from relation 966803, which is incomplete in the reduced data set because way 62201731 is omitted: rel 966803 tags type=multipolygon timezone=Australia/Currie members (all tagged boundary=administrative) way id admin_level role 62201730 2 outer 62201731 10 outer With the full data set, way 62201730 is not rendered as an independent way, but only as part of multipolygons "Tasmania" and 966803. It first shows up at zoom setting 7mi, labelled "Tasmania". At zoom 0.5mi, ways with admin_level=10 are displayed. At this zoom, multipolygon 966803 is rendered, with a default label "St./Prv. Border". It appears that the tags on way 62201731 have overridden the other tags. And when the relation is not complete, it appears either that no overriding takes place, or that the tags on way 62201730 win out over the tags on the relation. The OSM wiki says that tags on the multipolygon override tags on the outer ways, but the suggested algorithm implies that ways with line-type tags should also be rendered independently, if the tags differ from those on the multipolygon. Perhaps boundaries work differently.]
Hi Adrian, Thanks for looking at the problem and documenting it. Yes it is the 12-mile territorial waters I was referring to, sorry I didn't make that clear. I assume other parts of the world may have similar problems. The Australia map I sometimes download from http://garmin.na1400.info/routable.php always shows the problem. Regards, Markus_g -----Original Message----- From: mkgmap-dev-bounces@lists.mkgmap.org.uk [mailto:mkgmap-dev-bounces@lists.mkgmap.org.uk] On Behalf Of Adrian Sent: Saturday, 30 October 2010 2:11 AM To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] Missing ways part 1 On 23/10/10, Markus_g wrote:
The only thing that seems to work for me on Australia if I use the default style is overlap of 50000. If I use any thing less I get lines of maritime boundaries drawing all over the place and have some missing ways.
I've had a look at this. I followed the recipe in Markus_g's earlier message: http://www.mkgmap.org.uk/pipermail/mkgmap-dev/2010q3/009082.html I first looked at the coastline and did not find any problems. [While looking at the coastline, I did however find some multipolygons where I had doubts about the tagging or structure. In each case, one of the outer ways is a coastline way. I do not feel it is appropriate for me to edit Australian map data, so here is a list for anyone who may wish to take this up. rel. id coast way id way reversed 370821 47033944 548247 54777428 1216240 82155137 y 1231296 82163390 1237298 82378497 y 1240053 82488289 y] I then realised that Markus_g was talking about the line marking the 12-mile limit of territorial waters. Using the default overlap in splitter and the default style, I saw what Markus_g described. Apart from a small gap in the line, all the problems are in tile #3, in the vicinity of Tasmania. (See the attached areas.list) This is my analysis. The problems relate to four ways, all tagged boundary=administrative: way id admin_level rel. memberships role 62201729 2 Australia, Tasmania both outer 31509803 2 Australia, Tasmania both outer 31508204 2 Australia, Victoria both outer 62201730 2 Australia, Tasmania, 966803(timezone) all outer and three multipolygon relations, all tagged boundary=administrative: rel id admin_level name 80500 2 Australia 85104 4 Tasmania 80371 4 Victoria Tile #1 is to the east of tile #3 and the boundary between them is at 147.48°E. Way 62201729 starts and finishes in tile #3 but makes two excursions into tile #1, so it crosses the boundary four times, although one excursion does not go beyond the overlap region of tile #3. Way 31509803 starts and finishes in tile #3 and makes one excursion into tile #1, going beyond the overlap region of tile #3. Way 31508204 starts in tile #3 and finishes in tile #1. Way 62201730 is entirely within tile #3. There are two issues as Markus_g described, stray lines and missing ways; plus a small gap. Splitter is working as intended, at least with the default overlap. The stray lines could be changed to gaps by a modification to mkgmap. The stray lines and/or gaps are inherent to the way splitter works, but are readily fixed by a modest increase in the overlap, or by increasing the density of points in the map. Stray lines are caused by this combination of circumstances: a) A way leaves a tile, going beyond the overlap region, b) The way has no node in the overlap region, and c) The way re-enters the overlap region or the tile. The stray line is drawn from the last node in the tile before a), towards the first node after c), finishing at the tile boundary. E.g. for way 62201729, from node 352667234 towards node 352667223; and for way 31509803, from node 352667561 towards node 352650024. The two nodes are not consecutive nodes of the way. Mkgmap must have been intentionally coded to draw lines between non-consecutive nodes, passing over nodes whose lat-long is missing. Mkgmap would 'know' it has passed over a node, because mkgmap must work from the sequence of node ids, and the tile always contains the complete sequence for a way. Drawing the lines might be the best policy when both nodes are in the tile. But when one of the nodes is outside the tile, mkgmap could leave a gap rather than drawing a stray line. Mkgmap already 'knows' that one of the nodes is outside the tile, because it is finishing the line at the tile boundary. It just needs to detect the combination of a) passing over a node, and b) one node outside the tile. For the missing ways, I will describe exactly what is seen in the Garmin map. At zoom settings of 200mi or more, nothing is displayed. At zoom settings from 150mi to 10mi, ways with admin_level=2 are displayed. This is set in the style. At zoom settings of 7mi or less, ways with admin_level=2 or 4 are displayed. In some cases the ways have two labels. My interpretation of this is that there are two superimposed ways. In tiles except #3, it is like these ways in tile #1: way id label in zoom 150mi labels in zoom 7mi 62201729 "Australia" "Australia", "Tasmania" 31509803 "Australia" "Australia", "Tasmania" 31508204 "Australia" "Australia", "Victoria" In tile #3: way id label in zoom 150mi label(s) in zoom 7mi 62201729 [not rendered] "Tasmania" 31509803 [not rendered] "Tasmania" 31508204 "Victoria/Australia" "Victoria/Australia" 62201730 "Intl. Border" "Intl. Border", "Tasmania" There is another way, 31508201, which crosses from tile #3 into tile #10, and comes out the same as way 31508204. My conclusion from these observations is that multipolygon "Australia" is not rendered in tile #3, and this has different effects on different ways, depending on the number of times that the way crosses the tile boundary. Two or more crossings - the way is not rendered. One crossing - the way is rendered as an independent way, but acquires a label combining the names of both parent multipolygons. No crossings - the way is rendered as an independent way but receives a default label. It also appears that multipolygon "Victoria" is not rendered in tile #3. So I have broken down the interaction between a) the cutting of the map into tiles, b) the multipolygon processing, and c) the style. It will have to be for others more familiar with the software, to explain exactly why the observed behaviour is happening, and to say whether it is fixable. The tagging of boundaries in OSM is not straightforward and I have not studied the details. So I cannot say whether these Australian boundaries are correctly tagged. But in my opinion the results with mkgmap are satisfactory except in tile #3, so I would hope that it is fixable in mkgmap. Markus_g found that --overlap=50000 in splitter solved the problems. I believe this is simply the threshold, at which all of way 31509803 is included in the overlap region of tile #3. Presumably this results in multipolygon "Australia" being rendered, and that solves the problems. I have prepared an .osm file of Markus_g's bounding box, containing just the ways tagged admin_level=2 or 4, and the parent relations of those ways (but none of the other ways referred to in the relations). This reduced amount of data makes testing much easier because the file (and individual tiles) can be quickly inspected in JOSM, and it takes only a few seconds to make the Garmin map. The file is 14MB in .osm format and 0.4MB in .pbf format. If I am advised that this is a good thing to do, I can e-mail the .pbf to this list. I am also willing to e-mail the file to individuals on request (provided there are not too many!) [The observations above, are for the reduced data set. With the full data set, the results are slightly different. The difference arises from relation 966803, which is incomplete in the reduced data set because way 62201731 is omitted: rel 966803 tags type=multipolygon timezone=Australia/Currie members (all tagged boundary=administrative) way id admin_level role 62201730 2 outer 62201731 10 outer With the full data set, way 62201730 is not rendered as an independent way, but only as part of multipolygons "Tasmania" and 966803. It first shows up at zoom setting 7mi, labelled "Tasmania". At zoom 0.5mi, ways with admin_level=10 are displayed. At this zoom, multipolygon 966803 is rendered, with a default label "St./Prv. Border". It appears that the tags on way 62201731 have overridden the other tags. And when the relation is not complete, it appears either that no overriding takes place, or that the tags on way 62201730 win out over the tags on the relation. The OSM wiki says that tags on the multipolygon override tags on the outer ways, but the suggested algorithm implies that ways with line-type tags should also be rendered independently, if the tags differ from those on the multipolygon. Perhaps boundaries work differently.]
A new update is running with --overlap=50000 and --generate-sea=polygons,extend-sea-sectors,close-gaps=6000. Hopefully the problems will be gone. I do have a report from someone of a strange crash area in northern Tasmania. Where MapSource crashes if you zoom in on a specific spot (I tested different versions), and no routes across that section could be generated (neither in MapSource or on my GPS). I looked at the data in JOSM and couldn't find anything wrong though. Yesterday Mkgmap was updated to the latest version and a new map is being generated. Cross-fingers all is well this time :p On 2010-10-31 01:53, Markus_g wrote:
Hi Adrian,
Thanks for looking at the problem and documenting it. Yes it is the 12-mile territorial waters I was referring to, sorry I didn't make that clear.
I assume other parts of the world may have similar problems. The Australia map I sometimes download from http://garmin.na1400.info/routable.php always shows the problem.
Regards,
Markus_g
Op 01-11-10 09:42, Lambertus schreef:
A new update is running with --overlap=50000 and Well, that proved to be not working. Splitter has been crunching all day on the initial split and is constantly printing out messages like this:
Node 388947686 in too many areas. Already in areas 0x86665608, trying to add area 0x96 This is the command line I used: java -Xmx6000m -ea -jar ~/garmin/utils/splitter/splitter.jar --overlap=50000 --no-trim --max-areas=255 --cache=cache --mapid=6324000 --max-nodes=1200000 etc. So this might work in specific areas but not worldwide.
I agree it prob won't work world wide. I think I tried it on another country recently where it didn't work. Markus_g -----Original Message----- From: mkgmap-dev-bounces@lists.mkgmap.org.uk [mailto:mkgmap-dev-bounces@lists.mkgmap.org.uk] On Behalf Of Lambertus Sent: Tuesday, 2 November 2010 4:56 AM To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] Missing ways part 1 Op 01-11-10 09:42, Lambertus schreef:
A new update is running with --overlap=50000 and Well, that proved to be not working. Splitter has been crunching all day on the initial split and is constantly printing out messages like this:
Node 388947686 in too many areas. Already in areas 0x86665608, trying to add area 0x96 This is the command line I used: java -Xmx6000m -ea -jar ~/garmin/utils/splitter/splitter.jar --overlap=50000 --no-trim --max-areas=255 --cache=cache --mapid=6324000 --max-nodes=1200000 etc. So this might work in specific areas but not worldwide. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
On 2010-11-02 09:25, Markus_g wrote:
I agree it prob won't work world wide. I think I tried it on another country recently where it didn't work.
Well I'm not really sure but even --overlap=5000 seems to severely slow things down (although I'm not seeing the error messages this time).
-----Original Message----- From: mkgmap-dev-bounces@lists.mkgmap.org.uk [mailto:mkgmap-dev-bounces@lists.mkgmap.org.uk] On Behalf Of Lambertus Sent: Tuesday, 2 November 2010 4:56 AM To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] Missing ways part 1
Op 01-11-10 09:42, Lambertus schreef:
A new update is running with --overlap=50000 and Well, that proved to be not working. Splitter has been crunching all day on the initial split and is constantly printing out messages like this:
Node 388947686 in too many areas. Already in areas 0x86665608, trying to add area 0x96
This is the command line I used:
java -Xmx6000m -ea -jar ~/garmin/utils/splitter/splitter.jar --overlap=50000 --no-trim --max-areas=255 --cache=cache --mapid=6324000 --max-nodes=1200000 etc.
So this might work in specific areas but not worldwide.
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
participants (4)
-
Adrian -
Lambertus -
Markus_g -
Scott Crosby