splitter maximum tile size
Hi, regarding thread http://www.mkgmap.org.uk/pipermail/mkgmap-dev/2012q1/013611.html there is a maximum tile size that can be processed. Is this limit considered by splitter? WanMil
WanMil wrote
Hi,
regarding thread http://www.mkgmap.org.uk/pipermail/mkgmap-dev/2012q1/013611.html there is a maximum tile size that can be processed. Is this limit considered by splitter?
No, there is no limit for the maximum tiles size for now. Thanks for the hint. What is the limit? I guess it is not a constant, but depends on resolution and other values? Unfortunately the test data is gone and I didn't download it. Gerd -- View this message in context: http://gis.19327.n5.nabble.com/splitter-maximum-tile-size-tp5738382p5738388.... Sent from the Mkgmap Development mailing list archive at Nabble.com.
On Fri, Nov 30, GerdP wrote:
WanMil wrote
Hi,
regarding thread http://www.mkgmap.org.uk/pipermail/mkgmap-dev/2012q1/013611.html there is a maximum tile size that can be processed. Is this limit considered by splitter?
No, there is no limit for the maximum tiles size for now. Thanks for the hint. What is the limit? I guess it is not a constant, but depends on resolution and other values? Unfortunately the test data is gone and I didn't download it.
If you want to look at the problem, I can try to regenerate the data again. Thorsten -- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg)
Thorsten Kukuk wrote
No, there is no limit for the maximum tiles size for now. Thanks for the hint. What is the limit? I guess it is not a constant, but depends on resolution and other values? Unfortunately the test data is gone and I didn't download it.
If you want to look at the problem, I can try to regenerate the data again.
I am not sure if the problem has to be solved in splitter or in mkgmap. Anyway, please try to reproduce it. I also like to know why mkgmap has problems with tiles that contain only very few nodes. Or is it the gps device that has problems? Gerd -- View this message in context: http://gis.19327.n5.nabble.com/splitter-maximum-tile-size-tp5738382p5738401.... Sent from the Mkgmap Development mailing list archive at Nabble.com.
On 30.11.2012 12:08, GerdP wrote:
Thorsten Kukuk wrote
No, there is no limit for the maximum tiles size for now. Thanks for the hint. What is the limit? I guess it is not a constant, but depends on resolution and other values? Unfortunately the test data is gone and I didn't download it. If you want to look at the problem, I can try to regenerate the data again. I am not sure if the problem has to be solved in splitter or in mkgmap. Anyway, please try to reproduce it. I also like to know why mkgmap has problems with tiles that contain only very few nodes. Or is it the gps device that has problems?
Gerd
-- It's the GPS devices that have problems (BSOD). Maybe small tiles few nodes are okay. But big tiles, nearly no nodes were definitely a problem (as soon as two such tiles existed. A single tile didn't cause BSOD).
here is the archived thread about it: http://www.mkgmap.org.uk/pipermail/mkgmap-dev/2012q1/014023.html
Thorsten Kukuk wrote
No, there is no limit for the maximum tiles size for now. Thanks for the hint. What is the limit? I guess it is not a constant, but depends on resolution and other values? Unfortunately the test data is gone and I didn't download it.
If you want to look at the problem, I can try to regenerate the data again.
I am not sure if the problem has to be solved in splitter or in mkgmap. Anyway, please try to reproduce it. I also like to know why mkgmap has problems with tiles that contain only very few nodes. Or is it the gps device that has problems?
Gerd
I played a little bit with a small testfile how large a tile can be. The attached test file has the maximum size which works for me: <bounds minlat="0.0" minlon="0.0" maxlat="85.0" maxlon="179.98"/> Larger bounds cannot be put into a single tile. For large tiles mkgmap requires much memory problem if the bounds parameter is set. They need to be completely loaded with a high memory consumption. I don't know if there is a hard limit for the tile bounds size. Maybe using some limits in splitter for the maximum latitude and maximum longitude size would remove some problems, e.g. max lat = 20, max long = 20. WanMil
Hi WanMil, I think the maximum values should be as large as possible. When splitting planet, we have a huge tile covering an area in the pacific, this tile has almost no nodes. If splitter has to create tiles with max. 20 degrees width we probably see tiles without nodes. Gerd Date: Fri, 30 Nov 2012 18:18:02 +0100 From: wmgcnfg@web.de To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] splitter maximum tile size
Thorsten Kukuk wrote
No, there is no limit for the maximum tiles size for now. Thanks for the hint. What is the limit? I guess it is not a constant, but depends on resolution and other values? Unfortunately the test data is gone and I didn't download it.
If you want to look at the problem, I can try to regenerate the data again.
I am not sure if the problem has to be solved in splitter or in mkgmap. Anyway, please try to reproduce it. I also like to know why mkgmap has problems with tiles that contain only very few nodes. Or is it the gps device that has problems?
Gerd
I played a little bit with a small testfile how large a tile can be. The attached test file has the maximum size which works for me: <bounds minlat="0.0" minlon="0.0" maxlat="85.0" maxlon="179.98"/> Larger bounds cannot be put into a single tile. For large tiles mkgmap requires much memory problem if the bounds parameter is set. They need to be completely loaded with a high memory consumption. I don't know if there is a hard limit for the tile bounds size. Maybe using some limits in splitter for the maximum latitude and maximum longitude size would remove some problems, e.g. max lat = 20, max long = 20. WanMil _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Gerd, Ok, we should first investigate if there really is a limit for a tile size. Just for now it may be good if splitter issues a warning for tiles that exceeds a limit, e.g. "Tile xxxxx.osm.pbf covers more than n° in latitude. This may cause problems with mkgmap when using one of the options bounds, generate-sea and precomp-sea. <some hints what can be done to avoid that>". I think my testcase is not ideal. I have used a polygon that covered the whole tile. But this causes large (too large) img sizes because the polygon has to be split into many subpolygons when mkgmap splits into subdivisions. I think instead it is better to use one POI node in each corner of the bbox. I will try again and report about my findings. WanMil
Hi WanMil,
I think the maximum values should be as large as possible. When splitting planet, we have a huge tile covering an area in the pacific, this tile has almost no nodes. If splitter has to create tiles with max. 20 degrees width we probably see tiles without nodes.
Gerd
Date: Fri, 30 Nov 2012 18:18:02 +0100 From: wmgcnfg@web.de To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] splitter maximum tile size
Thorsten Kukuk wrote
No, there is no limit for the maximum tiles size for now. Thanks for the hint. What is the limit? I guess it is not a constant, but depends on resolution and other values? Unfortunately the test data is gone and I didn't download it.
If you want to look at the problem, I can try to regenerate the data again.
I am not sure if the problem has to be solved in splitter or in mkgmap. Anyway, please try to reproduce it. I also like to know why mkgmap has problems with tiles that contain only very few nodes. Or is it the gps device that has problems?
Gerd
I played a little bit with a small testfile how large a tile can be. The attached test file has the maximum size which works for me: <bounds minlat="0.0" minlon="0.0" maxlat="85.0" maxlon="179.98"/> Larger bounds cannot be put into a single tile.
For large tiles mkgmap requires much memory problem if the bounds parameter is set. They need to be completely loaded with a high memory consumption.
I don't know if there is a hard limit for the tile bounds size. Maybe using some limits in splitter for the maximum latitude and maximum longitude size would remove some problems, e.g. max lat = 20, max long = 20.
WanMil
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
I think my testcase is not ideal. I have used a polygon that covered the whole tile. But this causes large (too large) img sizes because the polygon has to be split into many subpolygons when mkgmap splits into subdivisions. I think instead it is better to use one POI node in each corner of the bbox. I will try again and report about my findings.
I've played around a bit more. mkgmap is able to compile a single tile that covers the whole planet wihtout any error messages or exception. But that works only if --transparent is used. If --transparent is not set mkgmap creates the background polygon that must be splitted in so many smaller parts so that the maximum tile size is exceeded. But such large tiles are not displayed by Mapsource and Basecamp. They display mkgmap generated tiles only if the longitude and latitude tile size is below 179.9x°. I didn't measured the exact x (179.9° works, 179.99° does not work). So a tile with the bounds <bounds minlat="-89.0" minlon="-10.0" maxlat="90.0" maxlon="169.9"/> can be compiled and displayed with MapSource and Basecamp. A tile with the bounds <bounds minlat="-89.0" minlon="-10.0" maxlat="90.0" maxlon="170.0"/> can be compiled with mkgmap but no data is displayed in MapSource and Basecamp. So splitter could use a max limit of 179° for lat and lon but I don't know if that makes sense at all. WanMil
Hi WanMil, WanMil wrote
So splitter could use a max limit of 179° for lat and lon but I don't know if that makes sense at all.
why not? It already turned out that a maximum tile size is a good way to avoid the situation that Thorsten has described for his USA extract. I'll code these limits and we'll see. Gerd -- View this message in context: http://gis.19327.n5.nabble.com/splitter-maximum-tile-size-tp5738382p5738941.... Sent from the Mkgmap Development mailing list archive at Nabble.com.
Hi WanMil,
WanMil wrote
So splitter could use a max limit of 179° for lat and lon but I don't know if that makes sense at all.
why not? It already turned out that a maximum tile size is a good way to avoid the situation that Thorsten has described for his USA extract.
I'll code these limits and we'll see.
Gerd
What happens with a tile that has a 179 longitude or latitude? mkgmap compiles it only if * --transparent is set * --bounds is not set * --precomp-sea and/or --generate-sea and/or --coastlinefile is not set So it is more a theoretical max limit. I guess for most cases one of the above rules will not be fulfilled and mkgmap will fail to compile it because the maximum tile size will exceed or mkgmap gets memory problems. WanMil
Am 04.12.2012 10:17, schrieb WanMil:
What happens with a tile that has a 179 longitude or latitude? mkgmap compiles it only if * --transparent is set * --bounds is not set * --precomp-sea and/or --generate-sea and/or --coastlinefile is not set I agree, that typical, at least one of the above things are set. I think best would be a parameter --max-tile-size=<max. degree-span of a tile>, which have to be smaller than 179°. In my maps I found a tile, which is a little bit larger than 20° x 20° in southern part of Patagonia. So I think the default should be larger.
Henning
On 03/12/2012 22:24, WanMil wrote:
But such large tiles are not displayed by Mapsource and Basecamp. They display mkgmap generated tiles only if the longitude and latitude tile size is below 179.9x°. I didn't measured the exact x (179.9° works, 179.99° does not work).
So a tile with the bounds <bounds minlat="-89.0" minlon="-10.0" maxlat="90.0" maxlon="169.9"/> can be compiled and displayed with MapSource and Basecamp.
A tile with the bounds <bounds minlat="-89.0" minlon="-10.0" maxlat="90.0" maxlon="170.0"/> can be compiled with mkgmap but no data is displayed in MapSource and Basecamp.
So splitter could use a max limit of 179° for lat and lon but I don't know if that makes sense at all.
Perhaps this explains why my map of New Zealand is not being shown at certain zoomlevels in MapSource when then 180° is in view. See: http://forum.openstreetmap.org/viewtopic.php?id=9809
Hi all, I am trying to find a decision what splitter should do in the following case: Assume you have created an extract from planet using the usa.poly from Thorsten. usa.poly <http://gis.19327.n5.nabble.com/file/n5739131/usa.poly> http://gis.19327.n5.nabble.com/splitter-maximum-tile-size-tp5738382p5738533.... This polygon file contains 10 distinct areas. The resulting osm dump contains a huge bounding box covering almost 50 % of planet, but it also contains huge areas without any nodes (e.g. the european and asian area) Without the parameter --polygon-file=usa.poly splitter will try to create tiles that cover the bounding box of the file, which results in huge tile(s) covering europe and russia. This is obviously not what the user wants. What should splitter do in such a case? I see two solutions: 1) stop with a message that the tiles are too big and ask for a polygon-file 2) guess what polygon was used. I'd say 1) is good enough. In other words: If splitter is not able to split a file without creating huge (almost empty) tiles, it will ask for a bounding-polygon. Is that okay? Gerd -- View this message in context: http://gis.19327.n5.nabble.com/splitter-maximum-tile-size-tp5738382p5739131.... Sent from the Mkgmap Development mailing list archive at Nabble.com.
Hi Gerd, what about a check, if left=~-180 or right=~180, if this is really the leftest|rightest part of the map, or if it continues on the other side on anti-meridian. If so, divide bounding-box at anti-meridian into two parts and calculate tiles for these two bounding-boxes. Henning
On Wed, Dec 05, Henning Scholland wrote:
Hi Gerd, what about a check, if left=~-180 or right=~180, if this is really the leftest|rightest part of the map, or if it continues on the other side on anti-meridian. If so, divide bounding-box at anti-meridian into two parts and calculate tiles for these two bounding-boxes.
Yes, I think the problem is if you cross 180. Maybe if the data is going from ~-180 to ~180, we should check what the latest node from the right is, and what the latest node from the left is, and exclude the empty middle? Thorsten -- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg)
Thorsten Kukuk wrote
what about a check, if left=~-180 or right=~180, if this is really the leftest|rightest part of the map, or if it continues on the other side on anti-meridian. If so, divide bounding-box at anti-meridian into two parts and calculate tiles for these two bounding-boxes.
Yes, I think the problem is if you cross 180. Maybe if the data is going from ~-180 to ~180, we should check what the latest node from the right is, and what the latest node from the left is, and exclude the empty middle?
I thought the same and coded that before, but the problem would be the same if you use a polygon containing e.g. all countries using french as the national language. I think it is not good enough to creae two distinct areas if the original polygon contained more than two, and guessing the original polygon is not trivial, esp. not if it contains small areas around islands. Gerd -- View this message in context: http://gis.19327.n5.nabble.com/splitter-maximum-tile-size-tp5738382p5739141.... Sent from the Mkgmap Development mailing list archive at Nabble.com.
So there are several cases: 1: several distinct areas in oneinput-file (eg Iceland and Germany or France and all there islands) -> splitter should stop and ask the user to use al polygon-file or create a polygon-file itself and warn user, that h did so 2: Map crossing 180/-180 -> split bounding-box at 180/-180 and calculate tiles for each of the two bounding-boxes 2a: Map crossing 180/-180, but covers hole planet as input -> use bounding-box left=-180 right=180 Henning
Henning Scholland wrote
So there are several cases:
1: several distinct areas in oneinput-file (eg Iceland and Germany or France and all there islands) -> splitter should stop and ask the user to use al polygon-file or create a polygon-file itself and warn user, that h did so
2: Map crossing 180/-180 -> split bounding-box at 180/-180 and calculate tiles for each of the two bounding-boxes
2a: Map crossing 180/-180, but covers hole planet as input -> use bounding-box left=-180 right=180
Yes, there are many cases, but it is not always easy for splitter to find out what the user did to create the input file. Splitter doesn't "know" that Europe exists and how many nodes it should find in that area. This could be changed if we always feed splitter with a file containing the densityMap for planet, but that wouldn't make it much easier to find out what tiles the user wants. Reg. "create a polygon-file" : A possible algorithm for that could try to find bounding boxes that are not too big and have nice aspect ratio. It would prefer areas with few nodes to areas that are very big. I am not sure if this is good enough, but I think it's worth trying. OK? Gerd Gerd -- View this message in context: http://gis.19327.n5.nabble.com/splitter-maximum-tile-size-tp5738382p5739146.... Sent from the Mkgmap Development mailing list archive at Nabble.com.
Am 05.12.2012 12:18, schrieb GerdP:
Henning Scholland wrote
So there are several cases:
1: several distinct areas in oneinput-file (eg Iceland and Germany or France and all there islands) -> splitter should stop and ask the user to use al polygon-file or create a polygon-file itself and warn user, that h did so
2: Map crossing 180/-180 -> split bounding-box at 180/-180 and calculate tiles for each of the two bounding-boxes
2a: Map crossing 180/-180, but covers hole planet as input -> use bounding-box left=-180 right=180 Yes, there are many cases, but it is not always easy for splitter to find out what the user did to create the input file. Splitter doesn't "know" that Europe exists and how many nodes it should find in that area. This could be changed if we always feed splitter with a file containing the densityMap for planet, but that wouldn't make it much easier to find out what tiles the user wants.
Reg. "create a polygon-file" : A possible algorithm for that could try to find bounding boxes that are not too big and have nice aspect ratio. It would prefer areas with few nodes to areas that are very big.
I am not sure if this is good enough, but I think it's worth trying. OK?
Hmm...for splitter my case 1 and 2 seems to be the same. He don't know what world continues on the other side of 180/-180. I think the density-map of splitter is like a table and each field contains the node-density for n° x n°. Then splitter adds them to eg. 1° x 1°-Tiles. If there is a density of zero, remove this tile. I don't know if 1° x 1° is a good value, it's just an example. It should be big enough, that also deserts have a density > 0. Splitter can now create a simple bounding polygon. This wont be perfect, but it's a starting-point. Also splitter should ask the user to improve the poly-file for better results. Henning
Henning Scholland wrote
Hmm...for splitter my case 1 and 2 seems to be the same. He don't know what world continues on the other side of 180/-180.
I think the density-map of splitter is like a table and each field contains the node-density for n° x n°. Then splitter adds them to eg. 1° x 1°-Tiles. If there is a density of zero, remove this tile. I don't know if 1° x 1° is a good value, it's just an example. It should be big enough, that also deserts have a density > 0.
Splitter can now create a simple bounding polygon. This wont be perfect, but it's a starting-point. Also splitter should ask the user to improve the poly-file for better results.
Yes, this is more or less what the trim function does in r202. I do also think that we need trimming in such a case, but I think we risk to create holes in areas that were covered by the unknown polygon. Maybe it will work if we allow to have holes that are bigger than - maybe - 1° x 1° (have to find a reasonable value). Gerd -- View this message in context: http://gis.19327.n5.nabble.com/splitter-maximum-tile-size-tp5738382p5739167.... Sent from the Mkgmap Development mailing list archive at Nabble.com.
On Wed, Dec 05, GerdP wrote:
Thorsten Kukuk wrote
Yes, I think the problem is if you cross 180. Maybe if the data is going from ~-180 to ~180, we should check what the latest node from the right is, and what the latest node from the left is, and exclude the empty middle?
I thought the same and coded that before, but the problem would be the same if you use a polygon containing e.g. all countries using french as the national language. I think it is not good enough to creae two distinct areas if the original polygon contained more than two, and guessing the original polygon is not trivial, esp. not if it contains small areas around islands.
Ok, but I can create the problem with only _one_ polygone, which contains only Alaska, too. The problem is if a polygone has to cross -180/180 longitude. Ok, due to the design, it will always be two polygons ... Thorsten -- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg)
Thorsten Kukuk wrote
Ok, but I can create the problem with only _one_ polygone, which contains only Alaska, too. The problem is if a polygone has to cross -180/180 longitude. Ok, due to the design, it will always be two polygons ...
Yes, that's why I like the polygon-file-format of osmosis :-) Gerd -- View this message in context: http://gis.19327.n5.nabble.com/splitter-maximum-tile-size-tp5738382p5739148.... Sent from the Mkgmap Development mailing list archive at Nabble.com.
On Fri, Nov 30, GerdP wrote:
Thorsten Kukuk wrote
No, there is no limit for the maximum tiles size for now. Thanks for the hint. What is the limit? I guess it is not a constant, but depends on resolution and other values? Unfortunately the test data is gone and I didn't download it.
If you want to look at the problem, I can try to regenerate the data again.
I am not sure if the problem has to be solved in splitter or in mkgmap. Anyway, please try to reproduce it.
Ok, here is the data: http://osm.thkukuk.de/tmp/usa.tar.bz2 It contains the tile 71500722.osm.pbf, the splitter data (areas.list, template.args, densities-out.txt) and the usa.poly with which you can cut out the input data from a planet file. The problem is the 180 longitude. The few data on the left on it should be added to the tiles on the right, not the very, very far away data on the left. I don't know if this is something we can solve better with splitter, but I doubt it's solveable with mkgmap. Thorsten -- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg)
Hello Thorsten, thanks, I think this is a problem that can be solved in splitter. It seems that r247 works fine if you use --polygon-file=usa.poly r248 and above will refuce the polygon because it contains long diagonal lines. See also http://gis.19327.n5.nabble.com/splitter-r250-and-open-questions-reg-bounding... Gerd
Date: Fri, 30 Nov 2012 22:04:32 +0100 From: kukuk@suse.de To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] splitter maximum tile size
On Fri, Nov 30, GerdP wrote:
Thorsten Kukuk wrote
No, there is no limit for the maximum tiles size for now. Thanks for the hint. What is the limit? I guess it is not a constant, but depends on resolution and other values? Unfortunately the test data is gone and I didn't download it.
If you want to look at the problem, I can try to regenerate the data again.
I am not sure if the problem has to be solved in splitter or in mkgmap. Anyway, please try to reproduce it.
Ok, here is the data:
http://osm.thkukuk.de/tmp/usa.tar.bz2
It contains the tile 71500722.osm.pbf, the splitter data (areas.list, template.args, densities-out.txt) and the usa.poly with which you can cut out the input data from a planet file.
The problem is the 180 longitude. The few data on the left on it should be added to the tiles on the right, not the very, very far away data on the left. I don't know if this is something we can solve better with splitter, but I doubt it's solveable with mkgmap.
Thorsten
-- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
On Fri, Nov 30, GerdP wrote:
Thorsten Kukuk wrote
No, there is no limit for the maximum tiles size for now. Thanks for the hint. What is the limit? I guess it is not a constant, but depends on resolution and other values? Unfortunately the test data is gone and I didn't download it.
If you want to look at the problem, I can try to regenerate the data again.
I am not sure if the problem has to be solved in splitter or in mkgmap. Anyway, please try to reproduce it.
Ok, here is the data:
http://osm.thkukuk.de/tmp/usa.tar.bz2
It contains the tile 71500722.osm.pbf, the splitter data (areas.list, template.args, densities-out.txt) and the usa.poly with which you can cut out the input data from a planet file.
The problem is the 180 longitude. The few data on the left on it should be added to the tiles on the right, not the very, very far away data on the left. I don't know if this is something we can solve better with splitter, but I doubt it's solveable with mkgmap.
Thorsten
Thanks for the data. It can be compiled with mkgmap without problems but you only if you don't use the bounds parameter. Otherwise loading of the bounds takes too much memory. WanMil
participants (7)
-
Felix Hartmann -
Gerd Petermann -
GerdP
-
Henning Scholland -
Lambertus -
Thorsten Kukuk -
WanMil