Land polygon from sea generator
I have recently updated my setup to use a new version of mkgmap with improvements to sea generation and indexing. I would like to congratulate all involved for the great progress that has been made. I made a map of Great Britain and France. It occupies 1.1GB and comprises 184 tiles. I extracted the area I wanted from the Geofabrik Europe extract. The mkgmap run took 52 min (Core 2 Duo 2.66GHz, --max-jobs). I gave it 7GB of RAM and it used 4GB. [I am using Apple Java 1.6. Previous releases simply stopped with an error if they ran out of memory. The latest release appears to do something similar to virtual memory, so it does not stop when it runs out of memory, but it becomes extremely slow when it starts to swap to disk.] I used generate-sea:multipolygon,extend-sea-sectors with mkgmap-r1955 (trunk) and the sea all seems to have been generated correctly. There was just one place where a land polygon was missing, and so I got a yellow background instead of white. Tile details from areas.list: 63240143: 2381824,49152 to 2439168,131072 # : 51.108398,1.054688 to 52.338867,2.812500 This tile contained some white land and some yellow land. The yellow area was around 51.9N 1.1E. It is the area south of the Stour estuary including Clacton-on-Sea and Walton-on-the-Naze. I inspected the coastline data of this tile in JOSM and I could not see any problem with it. This is, however, a very complex coastline. The coastline ways and nodes account for about one fifth of the data of the tile. I think this coastline contains an excessive amount of detail. There appears to have been a large-scale import from the Ordnance Survey Vector Map and I suspect the data was not simplified before it was added to OSM. I have uploaded Ipswich.zip to files.mkgmap.org.uk/download. I would be grateful if someone could look at why the land polygon was not generated. The .zip contains: 143coast.osm.gz, which has just the coastline elements from the aforementioned tile, my commands, style and .TYP files, the details from areas.list, and the actual mkgmap command. I tried to use osmosis to extract the coastline elements from the tile, but osmosis could not touch the file produced by splitter, not even an old version of osmosis which supports API 0.5. Osmosis complained that the time stamps were missing. This is what I did instead: 1. Load the file produced by splitter, into JOSM. JOSM produces tens of thousands of warnings but it opens the file. (I believe JOSM was modified specifically to allow it to open splitter tiles.) 2. Use the search feature to select all elements tagged natural=coastline. 3. Create a new data layer. 4. Merge the selection into the new layer. 5. Save the new layer as 143coast.osm. This file lacks the bounds data, so... 6. Expand the file produced by splitter, to 63240143.osm. 7. Use a text editor to copy and paste the bounds element from 63240143.osm to 143coast.osm. (I was surprised that my text editor could open a 100MB file, but it did. It took 0.5GB of RAM.) During the run of mkgmap, three tiles produced errors like this: SEVERE (MapSplitter): 63240175.osm.gz: Area too small to split at http://www.openstreetmap.org/?mlat=52.03131&mlon=0.87899&zoom=17 (reduce the density of points, length of lines, etc.) With two of the tiles, I could not see any resulting defects in the map. For the third tile - 63240175: 2381824,32768 to 2439168,49152 # : 51.108398,0.703125 to 52.338867,1.054688 the only defect I could see was that at resolution 24, the land polygon was missing in the part of the tile to the north of 52.03131N, the latitude given in the error message. This run of mkgmap also prompted other questions which I might raise as separate threads.
Hi Adrian,
I have recently updated my setup to use a new version of mkgmap with improvements to sea generation and indexing. I would like to congratulate all involved for the great progress that has been made.
I made a map of Great Britain and France. It occupies 1.1GB and comprises 184 tiles. I extracted the area I wanted from the Geofabrik Europe extract. The mkgmap run took 52 min (Core 2 Duo 2.66GHz, --max-jobs). I gave it 7GB of RAM and it used 4GB. [I am using Apple Java 1.6. Previous releases simply stopped with an error if they ran out of memory. The latest release appears to do something similar to virtual memory, so it does not stop when it runs out of memory, but it becomes extremely slow when it starts to swap to disk.]
I used generate-sea:multipolygon,extend-sea-sectors with mkgmap-r1955 (trunk) and the sea all seems to have been generated correctly. There was just one place where a land polygon was missing, and so I got a yellow background instead of white. Tile details from areas.list: 63240143: 2381824,49152 to 2439168,131072 # : 51.108398,1.054688 to 52.338867,2.812500 This tile contained some white land and some yellow land. The yellow area was around 51.9N 1.1E. It is the area south of the Stour estuary including Clacton-on-Sea and Walton-on-the-Naze.
Can you please open the area with openstreetmap in your browser and send the permanent link? I want to avoid to have a look at the wrong area. (Debugging of the generate-sea parameter is quite laborious)
I inspected the coastline data of this tile in JOSM and I could not see any problem with it. This is, however, a very complex coastline. The coastline ways and nodes account for about one fifth of the data of the tile. I think this coastline contains an excessive amount of detail. There appears to have been a large-scale import from the Ordnance Survey Vector Map and I suspect the data was not simplified before it was added to OSM.
I have uploaded Ipswich.zip to files.mkgmap.org.uk/download. I would be grateful if someone could look at why the land polygon was not generated. The .zip contains: 143coast.osm.gz, which has just the coastline elements from the aforementioned tile, my commands, style and .TYP files, the details from areas.list, and the actual mkgmap command.
I tried to use osmosis to extract the coastline elements from the tile, but osmosis could not touch the file produced by splitter, not even an old version of osmosis which supports API 0.5. Osmosis complained that the time stamps were missing. This is what I did instead: 1. Load the file produced by splitter, into JOSM. JOSM produces tens of thousands of warnings but it opens the file. (I believe JOSM was modified specifically to allow it to open splitter tiles.) 2. Use the search feature to select all elements tagged natural=coastline. 3. Create a new data layer. 4. Merge the selection into the new layer. 5. Save the new layer as 143coast.osm. This file lacks the bounds data, so... 6. Expand the file produced by splitter, to 63240143.osm. 7. Use a text editor to copy and paste the bounds element from 63240143.osm to 143coast.osm. (I was surprised that my text editor could open a 100MB file, but it did. It took 0.5GB of RAM.)
During the run of mkgmap, three tiles produced errors like this: SEVERE (MapSplitter): 63240175.osm.gz: Area too small to split at http://www.openstreetmap.org/?mlat=52.03131&mlon=0.87899&zoom=17 (reduce the density of points, length of lines, etc.)
This is a known problem of mkgmap. There is much work to do to fix that. You might read http://www.mkgmap.org.uk/pipermail/mkgmap-dev/2011q1/009960.html and http://www.mkgmap.org.uk/pipermail/mkgmap-dev/2011q2/011379.html for more info.
With two of the tiles, I could not see any resulting defects in the map. For the third tile - 63240175: 2381824,32768 to 2439168,49152 # : 51.108398,0.703125 to 52.338867,1.054688 the only defect I could see was that at resolution 24, the land polygon was missing in the part of the tile to the north of 52.03131N, the latitude given in the error message.
This run of mkgmap also prompted other questions which I might raise as separate threads. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
WanMil, Thank you.
Can you please open the area with openstreetmap in your browser and send the permanent link?
http://www.openstreetmap.org/?lat=51.9001&lon=1.1&zoom=12&layers=M Also Clacton-on Sea http://www.openstreetmap.org/?lat=51.7881&lon=1.1533&zoom=12&layers=M and Walton-on-the-Naze http://www.openstreetmap.org/?lat=51.8481&lon=1.2732&zoom=12&layers=M
Adrian, the sea generation algorithm is working fine. The problem arises from the log messages: Area too small to split at http://www.openstreetmap.org/?mlat=51.87630&mlon=1.24341&zoom=17 (reduce the density of points, length of lines, etc.) Area too small to split at http://www.openstreetmap.org/?mlat=51.87630&mlon=1.24341&zoom=17 (reduce the density of points, length of lines, etc.) Area too small to split at http://www.openstreetmap.org/?mlat=51.87630&mlon=1.24341&zoom=17 (reduce the density of points, length of lines, etc.) This is a known bug of mkgmap. Maybe you can avoid that by moving the west border of the tile a bit to the east to reduce the number of nodes in the missing land part. WanMil
WanMil,
Thank you.
Can you please open the area with openstreetmap in your browser and send the permanent link?
http://www.openstreetmap.org/?lat=51.9001&lon=1.1&zoom=12&layers=M Also Clacton-on Sea http://www.openstreetmap.org/?lat=51.7881&lon=1.1533&zoom=12&layers=M and Walton-on-the-Naze http://www.openstreetmap.org/?lat=51.8481&lon=1.2732&zoom=12&layers=M _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
WanMil, It's strange. I don't get that error message
Area too small to split at http://www.openstreetmap.org/?mlat=51.87630&mlon=1.24341&zoom=17 (reduce the density of points, length of lines, etc.)
when I run mkgmap-r1955 on the filtered tile that I uploaded (3MB). I do get exactly that error message, three times, when I run mkgmap-r1955 on the complete tile (16MB). That's why I did not suspect that the subdivision splitting problem might be the cause of the missing land polygon. (It is missing in both cases.) Because of this discrepancy, I looked further at tile 63240175, which I described before. I produced a filtered version of this tile containing just the coastline elements. I ran mkgmap on it and got the error message five times, the same as with the complete tile.
WanMil,
It's strange. I don't get that error message
Area too small to split at http://www.openstreetmap.org/?mlat=51.87630&mlon=1.24341&zoom=17 (reduce the density of points, length of lines, etc.)
when I run mkgmap-r1955 on the filtered tile that I uploaded (3MB). I do get exactly that error message, three times, when I run mkgmap-r1955 on the complete tile (16MB). That's why I did not suspect that the subdivision splitting problem might be the cause of the missing land polygon. (It is missing in both cases.)
Because of this discrepancy, I looked further at tile 63240175, which I described before. I produced a filtered version of this tile containing just the coastline elements. I ran mkgmap on it and got the error message five times, the same as with the complete tile.
That's no surprise for me. *As far as I have understood* the problem is causes by single polygons with too many nodes (per area?). So no matter how much other data is contained in the tile the problem should occur always if the given single polygon is contained in the tile. But this is a quite complex thing so I could be wrong. WanMil
WanMil,
During the run of mkgmap, three tiles produced errors like this: SEVERE (MapSplitter): 63240175.osm.gz: Area too small to split at http://www.openstreetmap.org/?mlat=52.03131&mlon=0.87899&zoom=17 (reduce the density of points, length of lines, etc.)
This is a known problem of mkgmap. There is much work to do to fix that. You might read http://www.mkgmap.org.uk/pipermail/mkgmap-dev/2011q1/009960.html and http://www.mkgmap.org.uk/pipermail/mkgmap-dev/2011q2/011379.html for more info.
My main reason for reporting this, was as an example of this error having small consequences. This is in contrast to other contributors to this list, who have reported much bigger consequences.
WanMil,
During the run of mkgmap, three tiles produced errors like this: SEVERE (MapSplitter): 63240175.osm.gz: Area too small to split at http://www.openstreetmap.org/?mlat=52.03131&mlon=0.87899&zoom=17 (reduce the density of points, length of lines, etc.)
This is a known problem of mkgmap. There is much work to do to fix that. You might read http://www.mkgmap.org.uk/pipermail/mkgmap-dev/2011q1/009960.html and http://www.mkgmap.org.uk/pipermail/mkgmap-dev/2011q2/011379.html for more info.
My main reason for reporting this, was as an example of this error having small consequences. This is in contrast to other contributors to this list, who have reported much bigger consequences.
I am aware that this problem can lead to missing large polygons and I think it is also possible that large roads with very much points are missing. I expect that this problem will occur more often with an increasing level of detail of the OSM data. In the end there must be someone who has time to fix that. WanMil
On Mon, Jun 06, 2011 at 03:59:28PM +0100, Adrian wrote:
I inspected the coastline data of this tile in JOSM and I could not see any problem with it.
Did you invoke the JOSM Validator on it? The coastline checks are (or at least used to be) in the INFO level; you may have to crank up the chattiness. Note that JOSM and mkgmap do not detect the same class of coastline errors; there is some overlap. Last time I had to use my brain a little was when mkgmap emitted a warning but JOSM Validator thought it is fine. If I remember correctly, the problem was that the map data contained two clockwise (or counter-clockwise) natural=coastline ways so that they did not intersect, but one way fully enclosed the other. I would be surprised if mkgmap did not emit some coastline warnings for the map data. It is sometimes a bit challenging to decipher the messages and it might be impractical to keep the data tidy at all times. In Finland, coastline bugs occur rarely, maybe a few bugs once in a couple of weeks. I only use extend-sea-sectors for one tile, as a work-around because the coastline in the finland.osm.pbf in the north (Tornio/Haparanda) does not extend to as far west as my tile would like to have it. Before going PBF, I had a sed or perl filter that moved the coastline endpoint to the west in the XML data. Best regards, Marko
Marko, Thank you for the suggestion.
Did you invoke the JOSM Validator on it? The coastline checks are (or at least used to be) in the INFO level; you may have to crank up the chattiness.
I tried the JOSM validator on the filtered coastline data, with the informational level switched on. It found nothing except two keys that it did not recognise.
On Mon, Jun 06, 2011 at 09:14:54PM +0100, Adrian wrote:
I tried the JOSM validator on the filtered coastline data, with the informational level switched on. It found nothing except two keys that it did not recognise.
Did you enable mkgmap logging with java -Dlog.config=logging.properties ... -jar mkgmap.jar ...? That should produce a lot of data. I use the file http://www.polkupyoraily.net/osm/files/logging.properties and filter out the junk from mkgmap.log.0 with a custom filter, available at http://www.polkupyoraily.net/osm/. Best regards, Marko
participants (3)
-
Adrian -
Marko Mäkelä -
WanMil