splitter: option for maximum tile area?
Hi all, currently a Java OutOfMemory exception prevents me from creating a map. I already use option --max-jobs=3 (the machine has 4 physical cores) and -Xmx5G (of 8 GB installed). Beyond OSM data, the map contains DEM and elevation contour lines. From the tiles finished and those with a new timestamp but about 0 bytes length, I can see that mkgmap was rendering tiles 47120005, 47120006, 47120007 at the time of crash. Tile 47120005 is extremely large by physical area - some 6° x 5.5° (see attached file), covering a lot of the south chinese sea, i.e. there are not many actual data in that area. I guess that the problem arises with that tile. I remember some case in the past where a single tile covering such a large area of mainly sea caused mkgmap to take an enormous amount of time for rendering - also here, mkgmap already spent about 1 hour before crashing. So I'd like to ask: is there some possibility for limiting the area of a tile among the splitter options? Kind regards, Bernhard
Hi Bernhard, there is no option for this. Do you use option --precomp-sea in splitter? Maybe you use --no-trim ? If that doesn't help you can try to change the values for private static final int MAX_LAT_DEGREES = 85; private static final int MAX_LON_DEGREES = 90; in SplittableDensityArea.java and compile your own version of splitter. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Bernhard Hiller <bhil@gmx.de> Gesendet: Mittwoch, 31. Oktober 2018 22:03 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: [mkgmap-dev] splitter: option for maximum tile area? Hi all, currently a Java OutOfMemory exception prevents me from creating a map. I already use option --max-jobs=3 (the machine has 4 physical cores) and -Xmx5G (of 8 GB installed). Beyond OSM data, the map contains DEM and elevation contour lines. From the tiles finished and those with a new timestamp but about 0 bytes length, I can see that mkgmap was rendering tiles 47120005, 47120006, 47120007 at the time of crash. Tile 47120005 is extremely large by physical area - some 6° x 5.5° (see attached file), covering a lot of the south chinese sea, i.e. there are not many actual data in that area. I guess that the problem arises with that tile. I remember some case in the past where a single tile covering such a large area of mainly sea caused mkgmap to take an enormous amount of time for rendering - also here, mkgmap already spent about 1 hour before crashing. So I'd like to ask: is there some possibility for limiting the area of a tile among the splitter options? Kind regards, Bernhard
Hi Bernhard, looked again at the splitter command in your last post. You also use a rather high max-nodes value. Such a high value means that you get rather large tiles and that mkgmap needs more memory for each tile compared to the default 1400000. Many users use a value near 1200000. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Mittwoch, 31. Oktober 2018 22:20 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] splitter: option for maximum tile area? Hi Bernhard, there is no option for this. Do you use option --precomp-sea in splitter? Maybe you use --no-trim ? If that doesn't help you can try to change the values for private static final int MAX_LAT_DEGREES = 85; private static final int MAX_LON_DEGREES = 90; in SplittableDensityArea.java and compile your own version of splitter. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Bernhard Hiller <bhil@gmx.de> Gesendet: Mittwoch, 31. Oktober 2018 22:03 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: [mkgmap-dev] splitter: option for maximum tile area? Hi all, currently a Java OutOfMemory exception prevents me from creating a map. I already use option --max-jobs=3 (the machine has 4 physical cores) and -Xmx5G (of 8 GB installed). Beyond OSM data, the map contains DEM and elevation contour lines. From the tiles finished and those with a new timestamp but about 0 bytes length, I can see that mkgmap was rendering tiles 47120005, 47120006, 47120007 at the time of crash. Tile 47120005 is extremely large by physical area - some 6° x 5.5° (see attached file), covering a lot of the south chinese sea, i.e. there are not many actual data in that area. I guess that the problem arises with that tile. I remember some case in the past where a single tile covering such a large area of mainly sea caused mkgmap to take an enormous amount of time for rendering - also here, mkgmap already spent about 1 hour before crashing. So I'd like to ask: is there some possibility for limiting the area of a tile among the splitter options? Kind regards, Bernhard _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Bernhard, I tried to reproduce the memory problems with tile 47120005. I don't think that the sea itself is a problem here, at least not when you use the --precomp-sea option. It took only 15 secs to process that tile with asia data from August with the default style and typical options for routing etc. My input file didn't contain SRTM data so I assume this is the reason. Maybe you have many contour lines with ele=0 in your data? Gerd ________________________________________ Von: Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Mittwoch, 31. Oktober 2018 22:49 An: Gerd Petermann; Development list for mkgmap Betreff: AW: [mkgmap-dev] splitter: option for maximum tile area? Hi Bernhard, looked again at the splitter command in your last post. You also use a rather high max-nodes value. Such a high value means that you get rather large tiles and that mkgmap needs more memory for each tile compared to the default 1400000. Many users use a value near 1200000. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Mittwoch, 31. Oktober 2018 22:20 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] splitter: option for maximum tile area? Hi Bernhard, there is no option for this. Do you use option --precomp-sea in splitter? Maybe you use --no-trim ? If that doesn't help you can try to change the values for private static final int MAX_LAT_DEGREES = 85; private static final int MAX_LON_DEGREES = 90; in SplittableDensityArea.java and compile your own version of splitter. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Bernhard Hiller <bhil@gmx.de> Gesendet: Mittwoch, 31. Oktober 2018 22:03 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: [mkgmap-dev] splitter: option for maximum tile area? Hi all, currently a Java OutOfMemory exception prevents me from creating a map. I already use option --max-jobs=3 (the machine has 4 physical cores) and -Xmx5G (of 8 GB installed). Beyond OSM data, the map contains DEM and elevation contour lines. From the tiles finished and those with a new timestamp but about 0 bytes length, I can see that mkgmap was rendering tiles 47120005, 47120006, 47120007 at the time of crash. Tile 47120005 is extremely large by physical area - some 6° x 5.5° (see attached file), covering a lot of the south chinese sea, i.e. there are not many actual data in that area. I guess that the problem arises with that tile. I remember some case in the past where a single tile covering such a large area of mainly sea caused mkgmap to take an enormous amount of time for rendering - also here, mkgmap already spent about 1 hour before crashing. So I'd like to ask: is there some possibility for limiting the area of a tile among the splitter options? Kind regards, Bernhard _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Gerd, thanks a lot for your many hints. The file contains contour lines with ele=0, but I do not know how many. Anyway, the style ignores contour lines at an elevation of 0. Currently, I do not use the --precomp-sea option, but "--generate-sea=multipolygon,extend-sea-sectors,floodblocker". Are the precompiled coast line files OK again (I remember that they were broken just recently)? Changing the source code of splitter is not an option for me: I have no Java IDE installed on my machine. I'll try now with a severely reduced max-nodes value. Kind regards, Bernhard Am 01.11.2018 um 09:32 schrieb Gerd Petermann:
Hi Bernhard,
I tried to reproduce the memory problems with tile 47120005. I don't think that the sea itself is a problem here, at least not when you use the --precomp-sea option. It took only 15 secs to process that tile with asia data from August with the default style and typical options for routing etc. My input file didn't contain SRTM data so I assume this is the reason. Maybe you have many contour lines with ele=in your data?
Gerd
________________________________________ Von: Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Mittwoch, 31. Oktober 2018 22:49 An: Gerd Petermann; Development list for mkgmap Betreff: AW: [mkgmap-dev] splitter: option for maximum tile area?
Hi Bernhard,
looked again at the splitter command in your last post. You also use a rather high max-nodes value. Such a high value means that you get rather large tiles and that mkgmap needs more memory for each tile compared to the default 1400000. Many users use a value near 1200000.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Mittwoch, 31. Oktober 2018 22:20 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] splitter: option for maximum tile area?
Hi Bernhard, there is no option for this. Do you use option --precomp-sea in splitter? Maybe you use --no-trim ? If that doesn't help you can try to change the values for private static final int MAX_LAT_DEGREES =5; private static final int MAX_LON_DEGREES =0; in SplittableDensityArea.java and compile your own version of splitter.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Bernhard Hiller <bhil@gmx.de> Gesendet: Mittwoch, 31. Oktober 2018 22:03 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: [mkgmap-dev] splitter: option for maximum tile area?
Hi all, currently a Java OutOfMemory exception prevents me from creating a map. I already use option --max-jobs=(the machine has 4 physical cores) and -Xmx5G (of 8 GB installed). Beyond OSM data, the map contains DEM and elevation contour lines. From the tiles finished and those with a new timestamp but about 0 bytes length, I can see that mkgmap was rendering tiles 47120005, 47120006, 47120007 at the time of crash. Tile 47120005 is extremely large by physical area - some 6° x 5.5° (see attached file), covering a lot of the south chinese sea, i.e. there are not many actual data in that area. I guess that the problem arises with that tile. I remember some case in the past where a single tile covering such a large area of mainly sea caused mkgmap to take an enormous amount of time for rendering - also here, mkgmap already spent about 1 hour before crashing. So I'd like to ask: is there some possibility for limiting the area of a tile among the splitter options? Kind regards, Bernhard _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Bernhard, my understanding is that you should never see broken precompiled sea data, because the data source is not directly OSM but http://openstreetmapdata.com/data/land-polygons where Jochen Topf invests time to check that the data is OK before updating it. So, sea.zip might not be up to date but should always be OK. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Bernhard Hiller <bhil@gmx.de> Gesendet: Donnerstag, 1. November 2018 11:23 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] splitter: option for maximum tile area? Hi Gerd, thanks a lot for your many hints. The file contains contour lines with ele=0, but I do not know how many. Anyway, the style ignores contour lines at an elevation of 0. Currently, I do not use the --precomp-sea option, but "--generate-sea=multipolygon,extend-sea-sectors,floodblocker". Are the precompiled coast line files OK again (I remember that they were broken just recently)? Changing the source code of splitter is not an option for me: I have no Java IDE installed on my machine. I'll try now with a severely reduced max-nodes value. Kind regards, Bernhard Am 01.11.2018 um 09:32 schrieb Gerd Petermann:
Hi Bernhard,
I tried to reproduce the memory problems with tile 47120005. I don't think that the sea itself is a problem here, at least not when you use the --precomp-sea option. It took only 15 secs to process that tile with asia data from August with the default style and typical options for routing etc. My input file didn't contain SRTM data so I assume this is the reason. Maybe you have many contour lines with ele=in your data?
Gerd
________________________________________ Von: Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Mittwoch, 31. Oktober 2018 22:49 An: Gerd Petermann; Development list for mkgmap Betreff: AW: [mkgmap-dev] splitter: option for maximum tile area?
Hi Bernhard,
looked again at the splitter command in your last post. You also use a rather high max-nodes value. Such a high value means that you get rather large tiles and that mkgmap needs more memory for each tile compared to the default 1400000. Many users use a value near 1200000.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Mittwoch, 31. Oktober 2018 22:20 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] splitter: option for maximum tile area?
Hi Bernhard, there is no option for this. Do you use option --precomp-sea in splitter? Maybe you use --no-trim ? If that doesn't help you can try to change the values for private static final int MAX_LAT_DEGREES =5; private static final int MAX_LON_DEGREES =0; in SplittableDensityArea.java and compile your own version of splitter.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Bernhard Hiller <bhil@gmx.de> Gesendet: Mittwoch, 31. Oktober 2018 22:03 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: [mkgmap-dev] splitter: option for maximum tile area?
Hi all, currently a Java OutOfMemory exception prevents me from creating a map. I already use option --max-jobs=(the machine has 4 physical cores) and -Xmx5G (of 8 GB installed). Beyond OSM data, the map contains DEM and elevation contour lines. From the tiles finished and those with a new timestamp but about 0 bytes length, I can see that mkgmap was rendering tiles 47120005, 47120006, 47120007 at the time of crash. Tile 47120005 is extremely large by physical area - some 6° x 5.5° (see attached file), covering a lot of the south chinese sea, i.e. there are not many actual data in that area. I guess that the problem arises with that tile. I remember some case in the past where a single tile covering such a large area of mainly sea caused mkgmap to take an enormous amount of time for rendering - also here, mkgmap already spent about 1 hour before crashing. So I'd like to ask: is there some possibility for limiting the area of a tile among the splitter options? Kind regards, Bernhard _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
On Thu, Nov 01, Gerd Petermann wrote:
Hi Bernhard,
my understanding is that you should never see broken precompiled sea data, because the data source is not directly OSM but http://openstreetmapdata.com/data/land-polygons where Jochen Topf invests time to check that the data is OK before updating it. So, sea.zip might not be up to date but should always be OK.
Not really, as he writes on his web page, his tools tries to repair the coastlines. And as I use them, too, I know that this automatic "repair" can break the result even more. I have added some additional post-precessing checks and patched osmcoastline to bail out on more errors to make sure that the result is for me really useable. Currently the coastlines are so heavy broken, that osmcoastline only seg.faults. So there will be no update with the broken costlines. I assume that why geofabrik does not report coastlines errors anymore, too. Thorsten -- Thorsten Kukuk, Distinguished Engineer, Senior Architect SLES & MicroOS SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nuernberg, Germany GF: Felix Imendoerffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nuernberg)
Hi Thorsten, thanks for the info. What is it that makes the correction so complicated? My approach would be this: 1) download planet 2) filter ways with natural=coastline 3) split the file with splitter into e.g. 100 parts 4) for each part: - open in JOSM - run validator - If an error is found, download that area to have current data and fix it This should allow to find and correct unconnected coastline ways within a few hours. Am I missing something? Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Thorsten Kukuk <kukuk@suse.de> Gesendet: Donnerstag, 1. November 2018 12:38 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] splitter: option for maximum tile area? On Thu, Nov 01, Gerd Petermann wrote:
Hi Bernhard,
my understanding is that you should never see broken precompiled sea data, because the data source is not directly OSM but http://openstreetmapdata.com/data/land-polygons where Jochen Topf invests time to check that the data is OK before updating it. So, sea.zip might not be up to date but should always be OK.
Not really, as he writes on his web page, his tools tries to repair the coastlines. And as I use them, too, I know that this automatic "repair" can break the result even more. I have added some additional post-precessing checks and patched osmcoastline to bail out on more errors to make sure that the result is for me really useable. Currently the coastlines are so heavy broken, that osmcoastline only seg.faults. So there will be no update with the broken costlines. I assume that why geofabrik does not report coastlines errors anymore, too. Thorsten -- Thorsten Kukuk, Distinguished Engineer, Senior Architect SLES & MicroOS SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nuernberg, Germany GF: Felix Imendoerffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nuernberg) _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi all, validator would not find ways like these two: https://www.openstreetmap.org/changeset/64081765 They are in the middle of South America. I found them only because they visually stuck out in JOSM. I would not find such an island when it is close to the real coastline but on land. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Donnerstag, 1. November 2018 15:15 An: Thorsten Kukuk; Development list for mkgmap Betreff: Re: [mkgmap-dev] splitter: option for maximum tile area? Hi Thorsten, thanks for the info. What is it that makes the correction so complicated? My approach would be this: 1) download planet 2) filter ways with natural=coastline 3) split the file with splitter into e.g. 100 parts 4) for each part: - open in JOSM - run validator - If an error is found, download that area to have current data and fix it This should allow to find and correct unconnected coastline ways within a few hours. Am I missing something? Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Thorsten Kukuk <kukuk@suse.de> Gesendet: Donnerstag, 1. November 2018 12:38 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] splitter: option for maximum tile area? On Thu, Nov 01, Gerd Petermann wrote:
Hi Bernhard,
my understanding is that you should never see broken precompiled sea data, because the data source is not directly OSM but http://openstreetmapdata.com/data/land-polygons where Jochen Topf invests time to check that the data is OK before updating it. So, sea.zip might not be up to date but should always be OK.
Not really, as he writes on his web page, his tools tries to repair the coastlines. And as I use them, too, I know that this automatic "repair" can break the result even more. I have added some additional post-precessing checks and patched osmcoastline to bail out on more errors to make sure that the result is for me really useable. Currently the coastlines are so heavy broken, that osmcoastline only seg.faults. So there will be no update with the broken costlines. I assume that why geofabrik does not report coastlines errors anymore, too. Thorsten -- Thorsten Kukuk, Distinguished Engineer, Senior Architect SLES & MicroOS SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nuernberg, Germany GF: Felix Imendoerffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nuernberg) _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi all, still struggling with that strange issue. But I guess I found some hint to the cause: inconsistent file sizes. - extracted OSM data: 400 MB pbf - elevation contour lines: 600 MB pbf - merged file: 1 GB pbf So far, sizes are consistent. When I run splitter on the OSM data only file, it produces many tiles summing up to some 400 MB. Consistent, too. When I run mkgmap on this output, I get a map within about half an hour, and it looks OK (tested with QLandkarte). When I run splitter on the merged file, the tiles sum up to some 9 GB. That is 9 times the size I'd expect. mkgmap can render several of those tiles (but very slowly); and then crashes on one of them with an OutOfMemory exception. So I think that the problem is somewhere in those contour lines, either when merged or alone. I'll try to create a contour lines only map as the next step to test this hypothesis. Kind regards, Bernhard Am 01.11.2018 um 09:32 schrieb Gerd Petermann:
Hi Bernhard,
I tried to reproduce the memory problems with tile 47120005. I don't think that the sea itself is a problem here, at least not when you use the --precomp-sea option. It took only 15 secs to process that tile with asia data from August with the default style and typical options for routing etc. My input file didn't contain SRTM data so I assume this is the reason. Maybe you have many contour lines with ele=in your data?
Gerd
________________________________________ Von: Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Mittwoch, 31. Oktober 2018 22:49 An: Gerd Petermann; Development list for mkgmap Betreff: AW: [mkgmap-dev] splitter: option for maximum tile area?
Hi Bernhard,
looked again at the splitter command in your last post. You also use a rather high max-nodes value. Such a high value means that you get rather large tiles and that mkgmap needs more memory for each tile compared to the default 1400000. Many users use a value near 1200000.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Mittwoch, 31. Oktober 2018 22:20 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] splitter: option for maximum tile area?
Hi Bernhard, there is no option for this. Do you use option --precomp-sea in splitter? Maybe you use --no-trim ? If that doesn't help you can try to change the values for private static final int MAX_LAT_DEGREES =5; private static final int MAX_LON_DEGREES =0; in SplittableDensityArea.java and compile your own version of splitter.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Bernhard Hiller <bhil@gmx.de> Gesendet: Mittwoch, 31. Oktober 2018 22:03 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: [mkgmap-dev] splitter: option for maximum tile area?
Hi all, currently a Java OutOfMemory exception prevents me from creating a map. I already use option --max-jobs=(the machine has 4 physical cores) and -Xmx5G (of 8 GB installed). Beyond OSM data, the map contains DEM and elevation contour lines. From the tiles finished and those with a new timestamp but about 0 bytes length, I can see that mkgmap was rendering tiles 47120005, 47120006, 47120007 at the time of crash. Tile 47120005 is extremely large by physical area - some 6° x 5.5° (see attached file), covering a lot of the south chinese sea, i.e. there are not many actual data in that area. I guess that the problem arises with that tile. I remember some case in the past where a single tile covering such a large area of mainly sea caused mkgmap to take an enormous amount of time for rendering - also here, mkgmap already spent about 1 hour before crashing. So I'd like to ask: is there some possibility for limiting the area of a tile among the splitter options? Kind regards, Bernhard _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Bernhard, what are your rules for the contour lines? Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Bernhard Hiller <bhil@gmx.de> Gesendet: Freitag, 2. November 2018 19:18 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] splitter: option for maximum tile area? Hi all, still struggling with that strange issue. But I guess I found some hint to the cause: inconsistent file sizes. - extracted OSM data: 400 MB pbf - elevation contour lines: 600 MB pbf - merged file: 1 GB pbf So far, sizes are consistent. When I run splitter on the OSM data only file, it produces many tiles summing up to some 400 MB. Consistent, too. When I run mkgmap on this output, I get a map within about half an hour, and it looks OK (tested with QLandkarte). When I run splitter on the merged file, the tiles sum up to some 9 GB. That is 9 times the size I'd expect. mkgmap can render several of those tiles (but very slowly); and then crashes on one of them with an OutOfMemory exception. So I think that the problem is somewhere in those contour lines, either when merged or alone. I'll try to create a contour lines only map as the next step to test this hypothesis. Kind regards, Bernhard Am 01.11.2018 um 09:32 schrieb Gerd Petermann:
Hi Bernhard,
I tried to reproduce the memory problems with tile 47120005. I don't think that the sea itself is a problem here, at least not when you use the --precomp-sea option. It took only 15 secs to process that tile with asia data from August with the default style and typical options for routing etc. My input file didn't contain SRTM data so I assume this is the reason. Maybe you have many contour lines with ele=in your data?
Gerd
________________________________________ Von: Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Mittwoch, 31. Oktober 2018 22:49 An: Gerd Petermann; Development list for mkgmap Betreff: AW: [mkgmap-dev] splitter: option for maximum tile area?
Hi Bernhard,
looked again at the splitter command in your last post. You also use a rather high max-nodes value. Such a high value means that you get rather large tiles and that mkgmap needs more memory for each tile compared to the default 1400000. Many users use a value near 1200000.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Mittwoch, 31. Oktober 2018 22:20 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] splitter: option for maximum tile area?
Hi Bernhard, there is no option for this. Do you use option --precomp-sea in splitter? Maybe you use --no-trim ? If that doesn't help you can try to change the values for private static final int MAX_LAT_DEGREES =5; private static final int MAX_LON_DEGREES =0; in SplittableDensityArea.java and compile your own version of splitter.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Bernhard Hiller <bhil@gmx.de> Gesendet: Mittwoch, 31. Oktober 2018 22:03 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: [mkgmap-dev] splitter: option for maximum tile area?
Hi all, currently a Java OutOfMemory exception prevents me from creating a map. I already use option --max-jobs=(the machine has 4 physical cores) and -Xmx5G (of 8 GB installed). Beyond OSM data, the map contains DEM and elevation contour lines. From the tiles finished and those with a new timestamp but about 0 bytes length, I can see that mkgmap was rendering tiles 47120005, 47120006, 47120007 at the time of crash. Tile 47120005 is extremely large by physical area - some 6° x 5.5° (see attached file), covering a lot of the south chinese sea, i.e. there are not many actual data in that area. I guess that the problem arises with that tile. I remember some case in the past where a single tile covering such a large area of mainly sea caused mkgmap to take an enormous amount of time for rendering - also here, mkgmap already spent about 1 hour before crashing. So I'd like to ask: is there some possibility for limiting the area of a tile among the splitter options? Kind regards, Bernhard _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Bernhard, just a guess: If you use option --add-pois-to-lines and you have a rule in the points file which processes ele to add a POI you may see something like that. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Freitag, 2. November 2018 19:24 An: Bernhard Hiller; Development list for mkgmap Betreff: Re: [mkgmap-dev] splitter: option for maximum tile area? Hi Bernhard, what are your rules for the contour lines? Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Bernhard Hiller <bhil@gmx.de> Gesendet: Freitag, 2. November 2018 19:18 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] splitter: option for maximum tile area? Hi all, still struggling with that strange issue. But I guess I found some hint to the cause: inconsistent file sizes. - extracted OSM data: 400 MB pbf - elevation contour lines: 600 MB pbf - merged file: 1 GB pbf So far, sizes are consistent. When I run splitter on the OSM data only file, it produces many tiles summing up to some 400 MB. Consistent, too. When I run mkgmap on this output, I get a map within about half an hour, and it looks OK (tested with QLandkarte). When I run splitter on the merged file, the tiles sum up to some 9 GB. That is 9 times the size I'd expect. mkgmap can render several of those tiles (but very slowly); and then crashes on one of them with an OutOfMemory exception. So I think that the problem is somewhere in those contour lines, either when merged or alone. I'll try to create a contour lines only map as the next step to test this hypothesis. Kind regards, Bernhard Am 01.11.2018 um 09:32 schrieb Gerd Petermann:
Hi Bernhard,
I tried to reproduce the memory problems with tile 47120005. I don't think that the sea itself is a problem here, at least not when you use the --precomp-sea option. It took only 15 secs to process that tile with asia data from August with the default style and typical options for routing etc. My input file didn't contain SRTM data so I assume this is the reason. Maybe you have many contour lines with ele=in your data?
Gerd
________________________________________ Von: Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Mittwoch, 31. Oktober 2018 22:49 An: Gerd Petermann; Development list for mkgmap Betreff: AW: [mkgmap-dev] splitter: option for maximum tile area?
Hi Bernhard,
looked again at the splitter command in your last post. You also use a rather high max-nodes value. Such a high value means that you get rather large tiles and that mkgmap needs more memory for each tile compared to the default 1400000. Many users use a value near 1200000.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Mittwoch, 31. Oktober 2018 22:20 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] splitter: option for maximum tile area?
Hi Bernhard, there is no option for this. Do you use option --precomp-sea in splitter? Maybe you use --no-trim ? If that doesn't help you can try to change the values for private static final int MAX_LAT_DEGREES =5; private static final int MAX_LON_DEGREES =0; in SplittableDensityArea.java and compile your own version of splitter.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Bernhard Hiller <bhil@gmx.de> Gesendet: Mittwoch, 31. Oktober 2018 22:03 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: [mkgmap-dev] splitter: option for maximum tile area?
Hi all, currently a Java OutOfMemory exception prevents me from creating a map. I already use option --max-jobs=(the machine has 4 physical cores) and -Xmx5G (of 8 GB installed). Beyond OSM data, the map contains DEM and elevation contour lines. From the tiles finished and those with a new timestamp but about 0 bytes length, I can see that mkgmap was rendering tiles 47120005, 47120006, 47120007 at the time of crash. Tile 47120005 is extremely large by physical area - some 6° x 5.5° (see attached file), covering a lot of the south chinese sea, i.e. there are not many actual data in that area. I guess that the problem arises with that tile. I remember some case in the past where a single tile covering such a large area of mainly sea caused mkgmap to take an enormous amount of time for rendering - also here, mkgmap already spent about 1 hour before crashing. So I'd like to ask: is there some possibility for limiting the area of a tile among the splitter options? Kind regards, Bernhard _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
On 11/02/2018 07:31 PM, Gerd Petermann wrote:
just a guess: If you use option --add-pois-to-lines and you have a rule in the points file which processes ele to add a POI you may see something like that.
I may have understood this wrong, but wasn't the file size comparison after the splitter stage? Splitter doesn't use any style rules, does it?
Hi Ralf, you got it wrong, it is about mkgmap because the problem is in this stage. The subject might be misleading ;-) Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Ralf Kleineisel <ralf@kleineisel.de> Gesendet: Freitag, 2. November 2018 21:48 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] splitter: option for maximum tile area? On 11/02/2018 07:31 PM, Gerd Petermann wrote:
just a guess: If you use option --add-pois-to-lines and you have a rule in the points file which processes ele to add a POI you may see something like that.
I may have understood this wrong, but wasn't the file size comparison after the splitter stage? Splitter doesn't use any style rules, does it? _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Gerd, contour lines were created with srtm2osm -step 25 -cat 500 100 25 -large with 3" elevation data downloaded from viewfinderpanoramas. The mkgmap parameters contain "--add-pois-to-areas", but no "--add-pois-to-lines" (the splitter parameters contain neither - as Ralf said). The style for contour lines is simple: contour=elevation & ele<=0 { delete contour; } contour=elevation & contour_ext=elevation_major { name '${ele|conv:m=>ft}'; } [0x22 level 4] contour=elevation & contour_ext=elevation_medium { name '${ele|conv:m=>ft}'; } [0x21 level 2] contour=elevation & contour_ext=elevation_minor { name '${ele|conv:m=>ft}'; } [0x20 level 0] There are no rules for contours in the points and polygons file. The options file defines the levels only. Running splitter on the contour lines file only, creates several pdb files summing up to 4.5 GB => the problem is with the contour lines. Next, I'll try to cut Laos from the contour lines file and create a contour lines map of Laos only... Kind regards, Bernhard Am 02.11.2018 um 19:31 schrieb Gerd Petermann:
Hi Bernhard,
just a guess: If you use option --add-pois-to-lines and you have a rule in the points file which processes ele to add a POI you may see something like that.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Freitag, 2. November 2018 19:24 An: Bernhard Hiller; Development list for mkgmap Betreff: Re: [mkgmap-dev] splitter: option for maximum tile area?
Hi Bernhard,
what are your rules for the contour lines?
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Bernhard Hiller <bhil@gmx.de> Gesendet: Freitag, 2. November 2018 19:18 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] splitter: option for maximum tile area?
Hi all, still struggling with that strange issue. But I guess I found some hint to the cause: inconsistent file sizes. - extracted OSM data: 400 MB pbf - elevation contour lines: 600 MB pbf - merged file: 1 GB pbf So far, sizes are consistent.
When I run splitter on the OSM data only file, it produces many tiles summing up to some 400 MB. Consistent, too. When I run mkgmap on this output, I get a map within about half an hour, and it looks OK (tested with QLandkarte).
When I run splitter on the merged file, the tiles sum up to some 9 GB. That is 9 times the size I'd expect. mkgmap can render several of those tiles (but very slowly); and then crashes on one of them with an OutOfMemory exception.
So I think that the problem is somewhere in those contour lines, either when merged or alone. I'll try to create a contour lines only map as the next step to test this hypothesis.
Kind regards, Bernhard
Am 01.11.2018 um 09:32 schrieb Gerd Petermann:
Hi Bernhard,
I tried to reproduce the memory problems with tile 47120005. I don't think that the sea itself is a problem here, at least not when you use the --precomp-sea option. It took only 15 secs to process that tile with asia data from August with the default style and typical options for routing etc. My input file didn't contain SRTM data so I assume this is the reason. Maybe you have many contour lines with ele= your data?
Gerd
________________________________________ Von: Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Mittwoch, 31. Oktober 2018 22:49 An: Gerd Petermann; Development list for mkgmap Betreff: AW: [mkgmap-dev] splitter: option for maximum tile area?
Hi Bernhard,
looked again at the splitter command in your last post. You also use a rather high max-nodes value. Such a high value means that you get rather large tiles and that mkgmap needs more memory for each tile compared to the default 1400000. Many users use a value near 1200000.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Mittwoch, 31. Oktober 2018 22:20 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] splitter: option for maximum tile area?
Hi Bernhard, there is no option for this. Do you use option --precomp-sea in splitter? Maybe you use --no-trim ? If that doesn't help you can try to change the values for private static final int MAX_LAT_DEGREES = private static final int MAX_LON_DEGREES = in SplittableDensityArea.java and compile your own version of splitter.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Bernhard Hiller <bhil@gmx.de> Gesendet: Mittwoch, 31. Oktober 2018 22:03 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: [mkgmap-dev] splitter: option for maximum tile area?
Hi all, currently a Java OutOfMemory exception prevents me from creating a map. I already use option --max-jobs=he machine has 4 physical cores) and -Xmx5G (of 8 GB installed). Beyond OSM data, the map contains DEM and elevation contour lines. From the tiles finished and those with a new timestamp but about 0 bytes length, I can see that mkgmap was rendering tiles 47120005, 47120006, 47120007 at the time of crash. Tile 47120005 is extremely large by physical area - some 6° x 5.5° (see attached file), covering a lot of the south chinese sea, i.e. there are not many actual data in that area. I guess that the problem arises with that tile. I remember some case in the past where a single tile covering such a large area of mainly sea caused mkgmap to take an enormous amount of time for rendering - also here, mkgmap already spent about 1 hour before crashing. So I'd like to ask: is there some possibility for limiting the area of a tile among the splitter options? Kind regards, Bernhard _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Bernhard, please post a link to one of the files produced by splitter. If the sum of file sizes is much larger than the input file that means that each file contains a lot of data that was written to keep ways complete. Maybe srtm2osm creates lots of very long ways ? Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Bernhard Hiller <bhil@gmx.de> Gesendet: Samstag, 3. November 2018 11:25 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] splitter: option for maximum tile area? Hi Gerd, contour lines were created with srtm2osm -step 25 -cat 500 100 25 -large with 3" elevation data downloaded from viewfinderpanoramas. The mkgmap parameters contain "--add-pois-to-areas", but no "--add-pois-to-lines" (the splitter parameters contain neither - as Ralf said). The style for contour lines is simple: contour=elevation & ele<=0 { delete contour; } contour=elevation & contour_ext=elevation_major { name '${ele|conv:m=>ft}'; } [0x22 level 4] contour=elevation & contour_ext=elevation_medium { name '${ele|conv:m=>ft}'; } [0x21 level 2] contour=elevation & contour_ext=elevation_minor { name '${ele|conv:m=>ft}'; } [0x20 level 0] There are no rules for contours in the points and polygons file. The options file defines the levels only. Running splitter on the contour lines file only, creates several pdb files summing up to 4.5 GB => the problem is with the contour lines. Next, I'll try to cut Laos from the contour lines file and create a contour lines map of Laos only... Kind regards, Bernhard Am 02.11.2018 um 19:31 schrieb Gerd Petermann:
Hi Bernhard,
just a guess: If you use option --add-pois-to-lines and you have a rule in the points file which processes ele to add a POI you may see something like that.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Freitag, 2. November 2018 19:24 An: Bernhard Hiller; Development list for mkgmap Betreff: Re: [mkgmap-dev] splitter: option for maximum tile area?
Hi Bernhard,
what are your rules for the contour lines?
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Bernhard Hiller <bhil@gmx.de> Gesendet: Freitag, 2. November 2018 19:18 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] splitter: option for maximum tile area?
Hi all, still struggling with that strange issue. But I guess I found some hint to the cause: inconsistent file sizes. - extracted OSM data: 400 MB pbf - elevation contour lines: 600 MB pbf - merged file: 1 GB pbf So far, sizes are consistent.
When I run splitter on the OSM data only file, it produces many tiles summing up to some 400 MB. Consistent, too. When I run mkgmap on this output, I get a map within about half an hour, and it looks OK (tested with QLandkarte).
When I run splitter on the merged file, the tiles sum up to some 9 GB. That is 9 times the size I'd expect. mkgmap can render several of those tiles (but very slowly); and then crashes on one of them with an OutOfMemory exception.
So I think that the problem is somewhere in those contour lines, either when merged or alone. I'll try to create a contour lines only map as the next step to test this hypothesis.
Kind regards, Bernhard
Am 01.11.2018 um 09:32 schrieb Gerd Petermann:
Hi Bernhard,
I tried to reproduce the memory problems with tile 47120005. I don't think that the sea itself is a problem here, at least not when you use the --precomp-sea option. It took only 15 secs to process that tile with asia data from August with the default style and typical options for routing etc. My input file didn't contain SRTM data so I assume this is the reason. Maybe you have many contour lines with ele= your data?
Gerd
________________________________________ Von: Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Mittwoch, 31. Oktober 2018 22:49 An: Gerd Petermann; Development list for mkgmap Betreff: AW: [mkgmap-dev] splitter: option for maximum tile area?
Hi Bernhard,
looked again at the splitter command in your last post. You also use a rather high max-nodes value. Such a high value means that you get rather large tiles and that mkgmap needs more memory for each tile compared to the default 1400000. Many users use a value near 1200000.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Mittwoch, 31. Oktober 2018 22:20 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] splitter: option for maximum tile area?
Hi Bernhard, there is no option for this. Do you use option --precomp-sea in splitter? Maybe you use --no-trim ? If that doesn't help you can try to change the values for private static final int MAX_LAT_DEGREES = private static final int MAX_LON_DEGREES = in SplittableDensityArea.java and compile your own version of splitter.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Bernhard Hiller <bhil@gmx.de> Gesendet: Mittwoch, 31. Oktober 2018 22:03 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: [mkgmap-dev] splitter: option for maximum tile area?
Hi all, currently a Java OutOfMemory exception prevents me from creating a map. I already use option --max-jobs=he machine has 4 physical cores) and -Xmx5G (of 8 GB installed). Beyond OSM data, the map contains DEM and elevation contour lines. From the tiles finished and those with a new timestamp but about 0 bytes length, I can see that mkgmap was rendering tiles 47120005, 47120006, 47120007 at the time of crash. Tile 47120005 is extremely large by physical area - some 6° x 5.5° (see attached file), covering a lot of the south chinese sea, i.e. there are not many actual data in that area. I guess that the problem arises with that tile. I remember some case in the past where a single tile covering such a large area of mainly sea caused mkgmap to take an enormous amount of time for rendering - also here, mkgmap already spent about 1 hour before crashing. So I'd like to ask: is there some possibility for limiting the area of a tile among the splitter options? Kind regards, Bernhard _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Gerd, I've uploaded some files: splitter log: https://drive.google.com/open?id=12wsnncNkfwKruEw-4UIgmvddK3E_TxtE kml: https://drive.google.com/open?id=1UOv_FTtl_pK_Nj126WDFnirhaswnvyfZ smallest tile: https://drive.google.com/open?id=1_yDKKpALKSfiRTiicCrEOYgH1vKmnm4w largest tile: https://drive.google.com/open?id=1aidryuJuhixrJIHHou3sibbBNBpRWJeN Kind regards, Bernhard Am 03.11.2018 um 11:38 schrieb Gerd Petermann:
Hi Bernhard,
please post a link to one of the files produced by splitter. If the sum of file sizes is much larger than the input file that means that each file contains a lot of data that was written to keep ways complete. Maybe srtm2osm creates lots of very long ways ?
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Bernhard Hiller <bhil@gmx.de> Gesendet: Samstag, 3. November 2018 11:25 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] splitter: option for maximum tile area?
Hi Gerd, contour lines were created with srtm2osm -step 25 -cat 500 100 25 -large with 3" elevation data downloaded from viewfinderpanoramas. The mkgmap parameters contain "--add-pois-to-areas", but no "--add-pois-to-lines" (the splitter parameters contain neither - as Ralf said). The style for contour lines is simple: contour=evation & ele<=0 { delete contour; } contour=evation & contour_ext=elevation_major { name '${ele|conv:m=t}'; } [0x22 level 4] contour=evation & contour_ext=elevation_medium { name '${ele|conv:m=t}'; } [0x21 level 2] contour=evation & contour_ext=elevation_minor { name '${ele|conv:m=t}'; } [0x20 level 0] There are no rules for contours in the points and polygons file. The options file defines the levels only.
Running splitter on the contour lines file only, creates several pdb files summing up to 4.5 GB =the problem is with the contour lines.
Next, I'll try to cut Laos from the contour lines file and create a contour lines map of Laos only...
Kind regards, Bernhard
Am 02.11.2018 um 19:31 schrieb Gerd Petermann:
Hi Bernhard,
just a guess: If you use option --add-pois-to-lines and you have a rule in the points file which processes ele to add a POI you may see something like that.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Freitag, 2. November 2018 19:24 An: Bernhard Hiller; Development list for mkgmap Betreff: Re: [mkgmap-dev] splitter: option for maximum tile area?
Hi Bernhard,
what are your rules for the contour lines?
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Bernhard Hiller <bhil@gmx.de> Gesendet: Freitag, 2. November 2018 19:18 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] splitter: option for maximum tile area?
Hi all, still struggling with that strange issue. But I guess I found some hint to the cause: inconsistent file sizes. - extracted OSM data: 400 MB pbf - elevation contour lines: 600 MB pbf - merged file: 1 GB pbf So far, sizes are consistent.
When I run splitter on the OSM data only file, it produces many tiles summing up to some 400 MB. Consistent, too. When I run mkgmap on this output, I get a map within about half an hour, and it looks OK (tested with QLandkarte).
When I run splitter on the merged file, the tiles sum up to some 9 GB. That is 9 times the size I'd expect. mkgmap can render several of those tiles (but very slowly); and then crashes on one of them with an OutOfMemory exception.
So I think that the problem is somewhere in those contour lines, either when merged or alone. I'll try to create a contour lines only map as the next step to test this hypothesis.
Kind regards, Bernhard
Am 01.11.2018 um 09:32 schrieb Gerd Petermann:
Hi Bernhard,
I tried to reproduce the memory problems with tile 47120005. I don't think that the sea itself is a problem here, at least not when you use the --precomp-sea option. It took only 15 secs to process that tile with asia data from August with the default style and typical options for routing etc. My input file didn't contain SRTM data so I assume this is the reason. Maybe you have many contour lines with ele=our data?
Gerd
________________________________________ Von: Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Mittwoch, 31. Oktober 2018 22:49 An: Gerd Petermann; Development list for mkgmap Betreff: AW: [mkgmap-dev] splitter: option for maximum tile area?
Hi Bernhard,
looked again at the splitter command in your last post. You also use a rather high max-nodes value. Such a high value means that you get rather large tiles and that mkgmap needs more memory for each tile compared to the default 1400000. Many users use a value near 1200000.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Mittwoch, 31. Oktober 2018 22:20 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] splitter: option for maximum tile area?
Hi Bernhard, there is no option for this. Do you use option --precomp-sea in splitter? Maybe you use --no-trim ? If that doesn't help you can try to change the values for private static final int MAX_LAT_DEGREES >> private static final int MAX_LON_DEGREES >> in SplittableDensityArea.java and compile your own version of splitter.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Bernhard Hiller <bhil@gmx.de> Gesendet: Mittwoch, 31. Oktober 2018 22:03 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: [mkgmap-dev] splitter: option for maximum tile area?
Hi all, currently a Java OutOfMemory exception prevents me from creating a map. I already use option --max-jobs= machine has 4 physical cores) and -Xmx5G (of 8 GB installed). Beyond OSM data, the map contains DEM and elevation contour lines. From the tiles finished and those with a new timestamp but about 0 bytes length, I can see that mkgmap was rendering tiles 47120005, 47120006, 47120007 at the time of crash. Tile 47120005 is extremely large by physical area - some 6° x 5.5° (see attached file), covering a lot of the south chinese sea, i.e. there are not many actual data in that area. I guess that the problem arises with that tile. I remember some case in the past where a single tile covering such a large area of mainly sea caused mkgmap to take an enormous amount of time for rendering - also here, mkgmap already spent about 1 hour before crashing. So I'd like to ask: is there some possibility for limiting the area of a tile among the splitter options? Kind regards, Bernhard _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hallo Bernhard, wolltest Du nicht max-nodes verringern? Jetzt ist es sogar 3.000.000 anstatt 1.400.000 Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Bernhard Hiller <bhil@gmx.de> Gesendet: Sonntag, 4. November 2018 11:19 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] splitter: option for maximum tile area? Hi Gerd, I've uploaded some files: splitter log: https://drive.google.com/open?id=12wsnncNkfwKruEw-4UIgmvddK3E_TxtE kml: https://drive.google.com/open?id=1UOv_FTtl_pK_Nj126WDFnirhaswnvyfZ smallest tile: https://drive.google.com/open?id=1_yDKKpALKSfiRTiicCrEOYgH1vKmnm4w largest tile: https://drive.google.com/open?id=1aidryuJuhixrJIHHou3sibbBNBpRWJeN Kind regards, Bernhard Am 03.11.2018 um 11:38 schrieb Gerd Petermann:
Hi Bernhard,
please post a link to one of the files produced by splitter. If the sum of file sizes is much larger than the input file that means that each file contains a lot of data that was written to keep ways complete. Maybe srtm2osm creates lots of very long ways ?
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Bernhard Hiller <bhil@gmx.de> Gesendet: Samstag, 3. November 2018 11:25 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] splitter: option for maximum tile area?
Hi Gerd, contour lines were created with srtm2osm -step 25 -cat 500 100 25 -large with 3" elevation data downloaded from viewfinderpanoramas. The mkgmap parameters contain "--add-pois-to-areas", but no "--add-pois-to-lines" (the splitter parameters contain neither - as Ralf said). The style for contour lines is simple: contour=evation & ele<=0 { delete contour; } contour=evation & contour_ext=elevation_major { name '${ele|conv:m=t}'; } [0x22 level 4] contour=evation & contour_ext=elevation_medium { name '${ele|conv:m=t}'; } [0x21 level 2] contour=evation & contour_ext=elevation_minor { name '${ele|conv:m=t}'; } [0x20 level 0] There are no rules for contours in the points and polygons file. The options file defines the levels only.
Running splitter on the contour lines file only, creates several pdb files summing up to 4.5 GB =the problem is with the contour lines.
Next, I'll try to cut Laos from the contour lines file and create a contour lines map of Laos only...
Kind regards, Bernhard
Am 02.11.2018 um 19:31 schrieb Gerd Petermann:
Hi Bernhard,
just a guess: If you use option --add-pois-to-lines and you have a rule in the points file which processes ele to add a POI you may see something like that.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Freitag, 2. November 2018 19:24 An: Bernhard Hiller; Development list for mkgmap Betreff: Re: [mkgmap-dev] splitter: option for maximum tile area?
Hi Bernhard,
what are your rules for the contour lines?
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Bernhard Hiller <bhil@gmx.de> Gesendet: Freitag, 2. November 2018 19:18 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] splitter: option for maximum tile area?
Hi all, still struggling with that strange issue. But I guess I found some hint to the cause: inconsistent file sizes. - extracted OSM data: 400 MB pbf - elevation contour lines: 600 MB pbf - merged file: 1 GB pbf So far, sizes are consistent.
When I run splitter on the OSM data only file, it produces many tiles summing up to some 400 MB. Consistent, too. When I run mkgmap on this output, I get a map within about half an hour, and it looks OK (tested with QLandkarte).
When I run splitter on the merged file, the tiles sum up to some 9 GB. That is 9 times the size I'd expect. mkgmap can render several of those tiles (but very slowly); and then crashes on one of them with an OutOfMemory exception.
So I think that the problem is somewhere in those contour lines, either when merged or alone. I'll try to create a contour lines only map as the next step to test this hypothesis.
Kind regards, Bernhard
Am 01.11.2018 um 09:32 schrieb Gerd Petermann:
Hi Bernhard,
I tried to reproduce the memory problems with tile 47120005. I don't think that the sea itself is a problem here, at least not when you use the --precomp-sea option. It took only 15 secs to process that tile with asia data from August with the default style and typical options for routing etc. My input file didn't contain SRTM data so I assume this is the reason. Maybe you have many contour lines with ele=our data?
Gerd
________________________________________ Von: Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Mittwoch, 31. Oktober 2018 22:49 An: Gerd Petermann; Development list for mkgmap Betreff: AW: [mkgmap-dev] splitter: option for maximum tile area?
Hi Bernhard,
looked again at the splitter command in your last post. You also use a rather high max-nodes value. Such a high value means that you get rather large tiles and that mkgmap needs more memory for each tile compared to the default 1400000. Many users use a value near 1200000.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Mittwoch, 31. Oktober 2018 22:20 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] splitter: option for maximum tile area?
Hi Bernhard, there is no option for this. Do you use option --precomp-sea in splitter? Maybe you use --no-trim ? If that doesn't help you can try to change the values for private static final int MAX_LAT_DEGREES >> private static final int MAX_LON_DEGREES >> in SplittableDensityArea.java and compile your own version of splitter.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Bernhard Hiller <bhil@gmx.de> Gesendet: Mittwoch, 31. Oktober 2018 22:03 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: [mkgmap-dev] splitter: option for maximum tile area?
Hi all, currently a Java OutOfMemory exception prevents me from creating a map. I already use option --max-jobs= machine has 4 physical cores) and -Xmx5G (of 8 GB installed). Beyond OSM data, the map contains DEM and elevation contour lines. From the tiles finished and those with a new timestamp but about 0 bytes length, I can see that mkgmap was rendering tiles 47120005, 47120006, 47120007 at the time of crash. Tile 47120005 is extremely large by physical area - some 6° x 5.5° (see attached file), covering a lot of the south chinese sea, i.e. there are not many actual data in that area. I guess that the problem arises with that tile. I remember some case in the past where a single tile covering such a large area of mainly sea caused mkgmap to take an enormous amount of time for rendering - also here, mkgmap already spent about 1 hour before crashing. So I'd like to ask: is there some possibility for limiting the area of a tile among the splitter options? Kind regards, Bernhard _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Ansonsten ist es so, wie ich vermutet habe. Srtm2OSM erzeugt elend lange Wege mit z.B. 184.962 Punkten. Die laufen dann natürlich weit über die Bounding Box hinaus und machen es wohl auch mkgmap schwer, weil die Wege erst recht spät abgeschnitten werden. Fazit: Das Problem ist in Srtm2OSM, versuch es mal mit der Option -maxwaynodes=5000 oder noch kleiner. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Sonntag, 4. November 2018 11:26 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] splitter: option for maximum tile area? Hallo Bernhard, wolltest Du nicht max-nodes verringern? Jetzt ist es sogar 3.000.000 anstatt 1.400.000 Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Bernhard Hiller <bhil@gmx.de> Gesendet: Sonntag, 4. November 2018 11:19 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] splitter: option for maximum tile area? Hi Gerd, I've uploaded some files: splitter log: https://drive.google.com/open?id=12wsnncNkfwKruEw-4UIgmvddK3E_TxtE kml: https://drive.google.com/open?id=1UOv_FTtl_pK_Nj126WDFnirhaswnvyfZ smallest tile: https://drive.google.com/open?id=1_yDKKpALKSfiRTiicCrEOYgH1vKmnm4w largest tile: https://drive.google.com/open?id=1aidryuJuhixrJIHHou3sibbBNBpRWJeN Kind regards, Bernhard Am 03.11.2018 um 11:38 schrieb Gerd Petermann:
Hi Bernhard,
please post a link to one of the files produced by splitter. If the sum of file sizes is much larger than the input file that means that each file contains a lot of data that was written to keep ways complete. Maybe srtm2osm creates lots of very long ways ?
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Bernhard Hiller <bhil@gmx.de> Gesendet: Samstag, 3. November 2018 11:25 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] splitter: option for maximum tile area?
Hi Gerd, contour lines were created with srtm2osm -step 25 -cat 500 100 25 -large with 3" elevation data downloaded from viewfinderpanoramas. The mkgmap parameters contain "--add-pois-to-areas", but no "--add-pois-to-lines" (the splitter parameters contain neither - as Ralf said). The style for contour lines is simple: contour=evation & ele<=0 { delete contour; } contour=evation & contour_ext=elevation_major { name '${ele|conv:m=t}'; } [0x22 level 4] contour=evation & contour_ext=elevation_medium { name '${ele|conv:m=t}'; } [0x21 level 2] contour=evation & contour_ext=elevation_minor { name '${ele|conv:m=t}'; } [0x20 level 0] There are no rules for contours in the points and polygons file. The options file defines the levels only.
Running splitter on the contour lines file only, creates several pdb files summing up to 4.5 GB =the problem is with the contour lines.
Next, I'll try to cut Laos from the contour lines file and create a contour lines map of Laos only...
Kind regards, Bernhard
Am 02.11.2018 um 19:31 schrieb Gerd Petermann:
Hi Bernhard,
just a guess: If you use option --add-pois-to-lines and you have a rule in the points file which processes ele to add a POI you may see something like that.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Freitag, 2. November 2018 19:24 An: Bernhard Hiller; Development list for mkgmap Betreff: Re: [mkgmap-dev] splitter: option for maximum tile area?
Hi Bernhard,
what are your rules for the contour lines?
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Bernhard Hiller <bhil@gmx.de> Gesendet: Freitag, 2. November 2018 19:18 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] splitter: option for maximum tile area?
Hi all, still struggling with that strange issue. But I guess I found some hint to the cause: inconsistent file sizes. - extracted OSM data: 400 MB pbf - elevation contour lines: 600 MB pbf - merged file: 1 GB pbf So far, sizes are consistent.
When I run splitter on the OSM data only file, it produces many tiles summing up to some 400 MB. Consistent, too. When I run mkgmap on this output, I get a map within about half an hour, and it looks OK (tested with QLandkarte).
When I run splitter on the merged file, the tiles sum up to some 9 GB. That is 9 times the size I'd expect. mkgmap can render several of those tiles (but very slowly); and then crashes on one of them with an OutOfMemory exception.
So I think that the problem is somewhere in those contour lines, either when merged or alone. I'll try to create a contour lines only map as the next step to test this hypothesis.
Kind regards, Bernhard
Am 01.11.2018 um 09:32 schrieb Gerd Petermann:
Hi Bernhard,
I tried to reproduce the memory problems with tile 47120005. I don't think that the sea itself is a problem here, at least not when you use the --precomp-sea option. It took only 15 secs to process that tile with asia data from August with the default style and typical options for routing etc. My input file didn't contain SRTM data so I assume this is the reason. Maybe you have many contour lines with ele=our data?
Gerd
________________________________________ Von: Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Mittwoch, 31. Oktober 2018 22:49 An: Gerd Petermann; Development list for mkgmap Betreff: AW: [mkgmap-dev] splitter: option for maximum tile area?
Hi Bernhard,
looked again at the splitter command in your last post. You also use a rather high max-nodes value. Such a high value means that you get rather large tiles and that mkgmap needs more memory for each tile compared to the default 1400000. Many users use a value near 1200000.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <gpetermann_muenchen@hotmail.com> Gesendet: Mittwoch, 31. Oktober 2018 22:20 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] splitter: option for maximum tile area?
Hi Bernhard, there is no option for this. Do you use option --precomp-sea in splitter? Maybe you use --no-trim ? If that doesn't help you can try to change the values for private static final int MAX_LAT_DEGREES >> private static final int MAX_LON_DEGREES >> in SplittableDensityArea.java and compile your own version of splitter.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Bernhard Hiller <bhil@gmx.de> Gesendet: Mittwoch, 31. Oktober 2018 22:03 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: [mkgmap-dev] splitter: option for maximum tile area?
Hi all, currently a Java OutOfMemory exception prevents me from creating a map. I already use option --max-jobs= machine has 4 physical cores) and -Xmx5G (of 8 GB installed). Beyond OSM data, the map contains DEM and elevation contour lines. From the tiles finished and those with a new timestamp but about 0 bytes length, I can see that mkgmap was rendering tiles 47120005, 47120006, 47120007 at the time of crash. Tile 47120005 is extremely large by physical area - some 6° x 5.5° (see attached file), covering a lot of the south chinese sea, i.e. there are not many actual data in that area. I guess that the problem arises with that tile. I remember some case in the past where a single tile covering such a large area of mainly sea caused mkgmap to take an enormous amount of time for rendering - also here, mkgmap already spent about 1 hour before crashing. So I'd like to ask: is there some possibility for limiting the area of a tile among the splitter options? Kind regards, Bernhard _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Thanks all, meanwhile I could generate a map which looks fine in QLandkarte and on my Oregon. I'd like to summarize the findings for those who try to create contour lines with srtm2osm and feed them into a Garmin map. I updated the wiki page https://wiki.openstreetmap.org/wiki/Srtm2Osm First of all, it is possible to use srtm2osm with elevation data downloaded from Viewfinderpanoramas and unzipped into its cache folder. In contrast to what one might expect, the folder where the *.hgt files must be placed is not the folder indicated with the -d option, but a folder "SrtmCache" below it (that's hard coded in the source code of srtm2osm). Next, the default start ids for nodes and ways generated by the program is long.MaxValue, which is a reserved value in splitter, thus causing an exception (see a previous thread: http://www.mkgmap.org.uk/pipermail/mkgmap-dev/2018q4/029073.html ). The options -firstnodeid 9023372036854775807 -firstwayid 9023372036854775807 (or other high values less than long.MaxValue) remove this issue. It is important to note that a misspelled option will not cause an error message by srtm2osm - that option is merely ignored. So do not feel surprised when a problem later on a step later on just happened again, though you thought that you changed something before... Another issue is the unlimited size of ways. In OSM data, ways longer than 2000 nodes do likely not exist because the API prevents the upload of such long ways. But srtm2osm produced ways with more than a hundred thousand nodes. splitter tries to keep ways complete. The long ways lead to a slow processing by splitter, and to an extremely inflated size of the tiles created by splitter (if the original file and the tiles have the same format, the sum of file sizes of the tiles should not be much more than the file size of the original file): upto 9 times the original size. There after, mkgmap crashes with an OutOfMemory error. The solution is to limit the file size with option -maxwaynodes 5000 (or an even lower value: the files were still inflated by about a half). Adding the option --precomp-sea to the splitter options is irrelevant for the OutOfMemory exception in mkgmap. Also, the value of the --max-nodes splitter option proofed irrelevant for the exception. Running mkgmap with --max-jobs=3 instead of 4 was also irrelevant. But running splitter with a high value of --max-nodes in combination with --precomp-sea lead to a different exception in mkgmap: (MapFailedException): 47120100.osm.pbf: (thrown in RoadDef.writeRgnOffsets()) Overflow of the NET1. The tile (47120100.osm.pbf) must be split so that there are fewer roads in it Then, that tile was left empty. I could see this issue with big cities (Bangkok, Kuala Lumpur) now, but never experienced it before (I created a map of Central Europe with a value of 3 million without precomp-sea - absolutely no problem). I hope this information won't be outdated when some else tries to follow these steps. Kind regards, Bernhard Am 31.10.2018 um 22:03 schrieb Bernhard Hiller:
Hi all, currently a Java OutOfMemory exception prevents me from creating a map. I already use option --max-jobs=3 (the machine has 4 physical cores) and -Xmx5G (of 8 GB installed). Beyond OSM data, the map contains DEM and elevation contour lines. From the tiles finished and those with a new timestamp but about 0 bytes length, I can see that mkgmap was rendering tiles 47120005, 47120006, 47120007 at the time of crash. Tile 47120005 is extremely large by physical area - some 6° x 5.5° (see attached file), covering a lot of the south chinese sea, i.e. there are not many actual data in that area. I guess that the problem arises with that tile. I remember some case in the past where a single tile covering such a large area of mainly sea caused mkgmap to take an enormous amount of time for rendering - also here, mkgmap already spent about 1 hour before crashing. So I'd like to ask: is there some possibility for limiting the area of a tile among the splitter options? Kind regards, Bernhard
Hi Bernhard, thanks for updating the Srtm2OSM wiki. I think it could still be improved, for example I did not find the -maxwaynodes option when I just looked for it, only found it using the search function of the browser. Reg. your other findings: I know these pros and cons for a max-nodes value different to the default: A larger --max-nodes value can help when you want to split a very large file (a large continent extract or planet) because - splitter might need less memory because it reduces the number of tiles and thus the number of ways crossing tile boundaries which are handled by the --keep-complete processing - the smaller number of tiles allows to avoid a limit in the gmapsupp, if I remember correctly max 4000 tiles are allowed in one mapset - the overall amount of data in the split files is smaller because less data is repeated in the areas outside the bounding boxes A lower --max-nodes value can help because - it avoids to reach several limits of the img format, esp. this one: "There is not enough room in a single garmin map for all the input data." - it reduces the amount of memory needed in mkgmap to process one tile. This can be important when multiple threads start to steal memory from each other and thus decrease throughput - it allows to open and check the tile in JOSM (presuming you have the pbf or o5m plugin installed) So, finding a perfet value depends on style and hardware and what mapset you plan to create. The expected effect of the --precomp-sea option in splitter is that it produces smaller tiles in areas with many islands because it will add extra weight to the coastline nodes. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Bernhard Hiller <bhil@gmx.de> Gesendet: Dienstag, 6. November 2018 14:54 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] splitter: option for maximum tile area? Thanks all, meanwhile I could generate a map which looks fine in QLandkarte and on my Oregon. I'd like to summarize the findings for those who try to create contour lines with srtm2osm and feed them into a Garmin map. I updated the wiki page https://wiki.openstreetmap.org/wiki/Srtm2Osm First of all, it is possible to use srtm2osm with elevation data downloaded from Viewfinderpanoramas and unzipped into its cache folder. In contrast to what one might expect, the folder where the *.hgt files must be placed is not the folder indicated with the -d option, but a folder "SrtmCache" below it (that's hard coded in the source code of srtm2osm). Next, the default start ids for nodes and ways generated by the program is long.MaxValue, which is a reserved value in splitter, thus causing an exception (see a previous thread: http://www.mkgmap.org.uk/pipermail/mkgmap-dev/2018q4/029073.html ). The options -firstnodeid 9023372036854775807 -firstwayid 9023372036854775807 (or other high values less than long.MaxValue) remove this issue. It is important to note that a misspelled option will not cause an error message by srtm2osm - that option is merely ignored. So do not feel surprised when a problem later on a step later on just happened again, though you thought that you changed something before... Another issue is the unlimited size of ways. In OSM data, ways longer than 2000 nodes do likely not exist because the API prevents the upload of such long ways. But srtm2osm produced ways with more than a hundred thousand nodes. splitter tries to keep ways complete. The long ways lead to a slow processing by splitter, and to an extremely inflated size of the tiles created by splitter (if the original file and the tiles have the same format, the sum of file sizes of the tiles should not be much more than the file size of the original file): upto 9 times the original size. There after, mkgmap crashes with an OutOfMemory error. The solution is to limit the file size with option -maxwaynodes 5000 (or an even lower value: the files were still inflated by about a half). Adding the option --precomp-sea to the splitter options is irrelevant for the OutOfMemory exception in mkgmap. Also, the value of the --max-nodes splitter option proofed irrelevant for the exception. Running mkgmap with --max-jobs=3 instead of 4 was also irrelevant. But running splitter with a high value of --max-nodes in combination with --precomp-sea lead to a different exception in mkgmap: (MapFailedException): 47120100.osm.pbf: (thrown in RoadDef.writeRgnOffsets()) Overflow of the NET1. The tile (47120100.osm.pbf) must be split so that there are fewer roads in it Then, that tile was left empty. I could see this issue with big cities (Bangkok, Kuala Lumpur) now, but never experienced it before (I created a map of Central Europe with a value of 3 million without precomp-sea - absolutely no problem). I hope this information won't be outdated when some else tries to follow these steps. Kind regards, Bernhard Am 31.10.2018 um 22:03 schrieb Bernhard Hiller:
Hi all, currently a Java OutOfMemory exception prevents me from creating a map. I already use option --max-jobs=3 (the machine has 4 physical cores) and -Xmx5G (of 8 GB installed). Beyond OSM data, the map contains DEM and elevation contour lines. From the tiles finished and those with a new timestamp but about 0 bytes length, I can see that mkgmap was rendering tiles 47120005, 47120006, 47120007 at the time of crash. Tile 47120005 is extremely large by physical area - some 6° x 5.5° (see attached file), covering a lot of the south chinese sea, i.e. there are not many actual data in that area. I guess that the problem arises with that tile. I remember some case in the past where a single tile covering such a large area of mainly sea caused mkgmap to take an enormous amount of time for rendering - also here, mkgmap already spent about 1 hour before crashing. So I'd like to ask: is there some possibility for limiting the area of a tile among the splitter options? Kind regards, Bernhard
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
participants (4)
-
Bernhard Hiller -
Gerd Petermann -
Ralf Kleineisel -
Thorsten Kukuk