mkgmap memory problem with big, nearly empty tiles
Hi, with my try to build a map for the whole US I run into the next problem: some of the US is on the other side of the 180 longitude. As result, splitter will create a really huge tile with nearly no data. mkgmap with --index --gmapsupp needs a really huge amount of memory for this (8 GB) and then prints: "There is not enough room in a single garmin map for all the input data The .osm file should be split into smaller pieces first." The bbox is really big: <bounds minlat='-0.65' minlon='-38.9' maxlat='71.851' maxlon='180.0'/> The data is only in a very small area of that tile. If I adjust the bounds area, I can build the tile without problems. Beside patching the tile after evey split, is there any way to better split the extract with tilesplitter? And why does mkgmap need so much memory for this tile? Is there any way to improve the mkgmap algorithm to work with this tile? I'm using mkgmap-r2169 and the command line is: env MKGMAP_MEM=8000M mkgmap --style-file=style --country-name=usa --country-abbr=US --family-name=TK-OSM-US --area-name=US --latin1 --license-file=TK-USA-Basemap_license.txt '--copyright-message=OpenStreetMap.org contributors. See: http://wiki.openstreetmap.org/index.php/Attribution. TK-USA-Basemap based on data from 2012-01-20.' --series-name=TK-USA-Basemap --bounds=bounds --location-autofill=bounds,nearest,is_in --add-pois-to-areas --reduce-point-density-polygon=8 --min-size-polygon=8 --make-opposite-cycleways --remove-short-arcs --adjust-turn-headings --route --net --generate-sea=extend-sea-sectors '--pois-to-areas-placement=entrance=main;entrance=yes;building=entrance;barrier=entrance' --index -c mkgmap.cfg --gmapsupp --input-file=verybigbb.osm --description=TK-USA-Basemap The sources can be found at: http://osm.thkukuk.de/tmp/bbox.tar.gz Thanks, Thorsten -- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg)
Hi Thorsten, please provide also the commands that you use to run splitter. I think you may get around the problem if you edit the areas.list created by splitter and use the modified version next time. Gerd
Date: Sun, 22 Jan 2012 12:30:41 +0100 From: kukuk@suse.de To: mkgmap-dev@lists.mkgmap.org.uk Subject: [mkgmap-dev] mkgmap memory problem with big, nearly empty tiles
Hi,
with my try to build a map for the whole US I run into the next problem: some of the US is on the other side of the 180 longitude.
As result, splitter will create a really huge tile with nearly no data. mkgmap with --index --gmapsupp needs a really huge amount of memory for this (8 GB) and then prints:
"There is not enough room in a single garmin map for all the input data The .osm file should be split into smaller pieces first."
The bbox is really big: <bounds minlat='-0.65' minlon='-38.9' maxlat='71.851' maxlon='180.0'/>
The data is only in a very small area of that tile. If I adjust the bounds area, I can build the tile without problems.
Beside patching the tile after evey split, is there any way to better split the extract with tilesplitter?
And why does mkgmap need so much memory for this tile? Is there any way to improve the mkgmap algorithm to work with this tile?
I'm using mkgmap-r2169 and the command line is: env MKGMAP_MEM=8000M mkgmap --style-file=style --country-name=usa --country-abbr=US --family-name=TK-OSM-US --area-name=US --latin1 --license-file=TK-USA-Basemap_license.txt '--copyright-message=OpenStreetMap.org contributors. See: http://wiki.openstreetmap.org/index.php/Attribution. TK-USA-Basemap based on data from 2012-01-20.' --series-name=TK-USA-Basemap --bounds=bounds --location-autofill=bounds,nearest,is_in --add-pois-to-areas --reduce-point-density-polygon=8 --min-size-polygon=8 --make-opposite-cycleways --remove-short-arcs --adjust-turn-headings --route --net --generate-sea=extend-sea-sectors '--pois-to-areas-placement=entrance=main;entrance=yes;building=entrance;barrier=entrance' --index -c mkgmap.cfg --gmapsupp --input-file=verybigbb.osm --description=TK-USA-Basemap
The sources can be found at: http://osm.thkukuk.de/tmp/bbox.tar.gz
Thanks, Thorsten -- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Gerd, On Sun, Jan 22, Gerd Petermann wrote:
Hi Thorsten,
please provide also the commands that you use to run splitter.
No problem: tilesplitter --mapid=$TILEID --max-nodes=700000 --overlap=4000 --no-trim --geonames-file=osmmaps/scripts/cities/${COUNTRY_ABBR}.txt --description="$DESC" --output=pbf --output-dir=$OUTPUTDIR $SRCDIR/$COUNTRY.$SUFFIX "tilesplitter" is only a shell script calling java splitter.jar, nothing more.
I think you may get around the problem if you edit the areas.list created by splitter and use the modified version next time.
Yes, beside patching the bounding box this is the second alternative, but I don't like to have a static areas.list file. OSM is increasing too fast and you need to adjust it too often. Thorsten
Gerd
Date: Sun, 22 Jan 2012 12:30:41 +0100 From: kukuk@suse.de To: mkgmap-dev@lists.mkgmap.org.uk Subject: [mkgmap-dev] mkgmap memory problem with big, nearly empty tiles
Hi,
with my try to build a map for the whole US I run into the next problem: some of the US is on the other side of the 180 longitude.
As result, splitter will create a really huge tile with nearly no data. mkgmap with --index --gmapsupp needs a really huge amount of memory for this (8 GB) and then prints:
"There is not enough room in a single garmin map for all the input data The .osm file should be split into smaller pieces first."
The bbox is really big: <bounds minlat='-0.65' minlon='-38.9' maxlat='71.851' maxlon='180.0'/>
The data is only in a very small area of that tile. If I adjust the bounds area, I can build the tile without problems.
Beside patching the tile after evey split, is there any way to better split the extract with tilesplitter?
And why does mkgmap need so much memory for this tile? Is there any way to improve the mkgmap algorithm to work with this tile?
I'm using mkgmap-r2169 and the command line is: env MKGMAP_MEM=8000M mkgmap --style-file=style --country-name=usa --country-abbr=US --family-name=TK-OSM-US --area-name=US --latin1 --license-file=TK-USA-Basemap_license.txt '--copyright-message=OpenStreetMap.org contributors. See: http://wiki.openstreetmap.org/index.php/Attribution. TK-USA-Basemap based on data from 2012-01-20.' --series-name=TK-USA-Basemap --bounds=bounds --location-autofill=bounds,nearest,is_in --add-pois-to-areas --reduce-point-density-polygon=8 --min-size-polygon=8 --make-opposite-cycleways --remove-short-arcs --adjust-turn-headings --route --net --generate-sea=extend-sea-sectors '--pois-to-areas-placement=entrance=main;entrance=yes;building=entrance;barrier=entrance' --index -c mkgmap.cfg --gmapsupp --input-file=verybigbb.osm --description=TK-USA-Basemap
The sources can be found at: http://osm.thkukuk.de/tmp/bbox.tar.gz
Thanks, Thorsten -- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg)
Hi Thorsten, I think you should not use the no-trim parameter. Did you try that? Gerd
Date: Sun, 22 Jan 2012 14:54:39 +0100 From: kukuk@suse.de To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] mkgmap memory problem with big, nearly empty tiles
Hi Gerd,
On Sun, Jan 22, Gerd Petermann wrote:
Hi Thorsten,
please provide also the commands that you use to run splitter.
No problem: tilesplitter --mapid=$TILEID --max-nodes=700000 --overlap=4000 --no-trim --geonames-file=osmmaps/scripts/cities/${COUNTRY_ABBR}.txt --description="$DESC" --output=pbf --output-dir=$OUTPUTDIR $SRCDIR/$COUNTRY.$SUFFIX
"tilesplitter" is only a shell script calling java splitter.jar, nothing more.
I think you may get around the problem if you edit the areas.list created by splitter and use the modified version next time.
Yes, beside patching the bounding box this is the second alternative, but I don't like to have a static areas.list file. OSM is increasing too fast and you need to adjust it too often.
Thorsten
Gerd
Date: Sun, 22 Jan 2012 12:30:41 +0100 From: kukuk@suse.de To: mkgmap-dev@lists.mkgmap.org.uk Subject: [mkgmap-dev] mkgmap memory problem with big, nearly empty tiles
Hi,
with my try to build a map for the whole US I run into the next problem: some of the US is on the other side of the 180 longitude.
As result, splitter will create a really huge tile with nearly no data. mkgmap with --index --gmapsupp needs a really huge amount of memory for this (8 GB) and then prints:
"There is not enough room in a single garmin map for all the input data The .osm file should be split into smaller pieces first."
The bbox is really big: <bounds minlat='-0.65' minlon='-38.9' maxlat='71.851' maxlon='180.0'/>
The data is only in a very small area of that tile. If I adjust the bounds area, I can build the tile without problems.
Beside patching the tile after evey split, is there any way to better split the extract with tilesplitter?
And why does mkgmap need so much memory for this tile? Is there any way to improve the mkgmap algorithm to work with this tile?
I'm using mkgmap-r2169 and the command line is: env MKGMAP_MEM=8000M mkgmap --style-file=style --country-name=usa --country-abbr=US --family-name=TK-OSM-US --area-name=US --latin1 --license-file=TK-USA-Basemap_license.txt '--copyright-message=OpenStreetMap.org contributors. See: http://wiki.openstreetmap.org/index.php/Attribution. TK-USA-Basemap based on data from 2012-01-20.' --series-name=TK-USA-Basemap --bounds=bounds --location-autofill=bounds,nearest,is_in --add-pois-to-areas --reduce-point-density-polygon=8 --min-size-polygon=8 --make-opposite-cycleways --remove-short-arcs --adjust-turn-headings --route --net --generate-sea=extend-sea-sectors '--pois-to-areas-placement=entrance=main;entrance=yes;building=entrance;barrier=entrance' --index -c mkgmap.cfg --gmapsupp --input-file=verybigbb.osm --description=TK-USA-Basemap
The sources can be found at: http://osm.thkukuk.de/tmp/bbox.tar.gz
Thanks, Thorsten -- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi, On Sun, Jan 22, Gerd Petermann wrote:
Hi Thorsten,
I think you should not use the no-trim parameter. Did you try that?
Tried it now, works. But this means other maps will now have again "land" or white space in the sea ... Thorsten
Date: Sun, 22 Jan 2012 14:54:39 +0100 From: kukuk@suse.de To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] mkgmap memory problem with big, nearly empty tiles
Hi Gerd,
On Sun, Jan 22, Gerd Petermann wrote:
Hi Thorsten,
please provide also the commands that you use to run splitter.
No problem: tilesplitter --mapid=$TILEID --max-nodes=700000 --overlap=4000 --no-trim --geonames-file=osmmaps/scripts/cities/${COUNTRY_ABBR}.txt --description="$DESC" --output=pbf --output-dir=$OUTPUTDIR $SRCDIR/$COUNTRY.$SUFFIX
"tilesplitter" is only a shell script calling java splitter.jar, nothing more.
I think you may get around the problem if you edit the areas.list created by splitter and use the modified version next time.
Yes, beside patching the bounding box this is the second alternative, but I don't like to have a static areas.list file. OSM is increasing too fast and you need to adjust it too often.
Thorsten
Gerd
Date: Sun, 22 Jan 2012 12:30:41 +0100 From: kukuk@suse.de To: mkgmap-dev@lists.mkgmap.org.uk Subject: [mkgmap-dev] mkgmap memory problem with big, nearly empty tiles
Hi,
with my try to build a map for the whole US I run into the next problem: some of the US is on the other side of the 180 longitude.
As result, splitter will create a really huge tile with nearly no data. mkgmap with --index --gmapsupp needs a really huge amount of memory for this (8 GB) and then prints:
"There is not enough room in a single garmin map for all the input data The .osm file should be split into smaller pieces first."
The bbox is really big: <bounds minlat='-0.65' minlon='-38.9' maxlat='71.851' maxlon='180.0'/>
The data is only in a very small area of that tile. If I adjust the bounds area, I can build the tile without problems.
Beside patching the tile after evey split, is there any way to better split the extract with tilesplitter?
And why does mkgmap need so much memory for this tile? Is there any way to improve the mkgmap algorithm to work with this tile?
I'm using mkgmap-r2169 and the command line is: env MKGMAP_MEM=8000M mkgmap --style-file=style --country-name=usa --country-abbr=US --family-name=TK-OSM-US --area-name=US --latin1 --license-file=TK-USA-Basemap_license.txt '--copyright-message=OpenStreetMap.org contributors. See: http://wiki.openstreetmap.org/index.php/Attribution. TK-USA-Basemap based on data from 2012-01-20.' --series-name=TK-USA-Basemap --bounds=bounds --location-autofill=bounds,nearest,is_in --add-pois-to-areas --reduce-point-density-polygon=8 --min-size-polygon=8 --make-opposite-cycleways --remove-short-arcs --adjust-turn-headings --route --net --generate-sea=extend-sea-sectors '--pois-to-areas-placement=entrance=main;entrance=yes;building=entrance;barrier=entrance' --index -c mkgmap.cfg --gmapsupp --input-file=verybigbb.osm --description=TK-USA-Basemap
The sources can be found at: http://osm.thkukuk.de/tmp/bbox.tar.gz
Thanks, Thorsten -- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg)
Hi
And why does mkgmap need so much memory for this tile? Is there any way to improve the mkgmap algorithm to work with this tile?
The area of the map is broken down into a number of sub areas, which have a maximum size and so there are a vast number required to cover that area. I believe that the algorithm could be amended to skip divisions that are empty and so the out of memory problem could be avoided. Its quite possible that the resulting map would not work though. But anyway in this case I would have thought that this is a splitter error? Even if it isn't, splitter needs to have a maximum tile size and I thought it did, or at least I thought it had at one time. ..Steve
Hi Steve,
Date: Sun, 22 Jan 2012 15:00:27 +0000 From: steve@parabola.me.uk To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] mkgmap memory problem with big, nearly empty tiles
Hi
And why does mkgmap need so much memory for this tile? Is there any way to improve the mkgmap algorithm to work with this tile?
The area of the map is broken down into a number of sub areas, which have a maximum size and so there are a vast number required to cover that area.
I believe that the algorithm could be amended to skip divisions that are empty and so the out of memory problem could be avoided. Its quite possible that the resulting map would not work though.
But anyway in this case I would have thought that this is a splitter error?
Thorsten used the no-trim parameter, so I think splitter result is correct.
Even if it isn't, splitter needs to have a maximum tile size and I thought it did, or at least I thought it had at one time.
The current split algorithm stops if the tiles are too small or if they have fewer than max-nodes coords. It should be easy to implement also a maximun-size criteria. What values do you suggest to use for that?
..Steve _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
participants (3)
-
Gerd Petermann -
Steve Ratcliffe -
Thorsten Kukuk