splitter exception with SRTM data for USA
Hi, I generated SRTM data for the USA with phygthmap and wanted to try to split it now for compiling with mkgmap. But while this worked fine for all other countries, in this case I get always the following exception: MAP occupancy: 2.350.000.000, number of area dictionary entries: 3994 of 65535 Map details: bytes/overhead 401.218.630 / 166.809.090, overhead includes 19 arrays with 8 MB 2.350.000.000 nodes processed... id=2526831732 MAP occupancy: 2.360.000.000, number of area dictionary entries: 4000 of 65535 Map details: bytes/overhead 402.878.606 / 166.808.202, overhead includes 19 arrays with 8 MB 2.360.000.000 nodes processed... id=2536831732 Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 524288 at uk.me.parabola.splitter.SparseLong2ShortMapInline.putChunk(SparseLong2ShortMapInline.java:448) at uk.me.parabola.splitter.SparseLong2ShortMapInline.saveCurrentChunk(SparseLong2ShortMapInline.java:191) at uk.me.parabola.splitter.SparseLong2ShortMapInline.put(SparseLong2ShortMapInline.java:218) at uk.me.parabola.splitter.ProblemListProcessor.processNode(ProblemListProcessor.java:167) at uk.me.parabola.splitter.BinaryMapParser.parseDense(BinaryMapParser.java:113) at crosby.binary.BinaryParser.parse(Unknown Source) at crosby.binary.BinaryParser.handleBlock(Unknown Source) at crosby.binary.file.FileBlock.process(Unknown Source) at crosby.binary.file.BlockInputStream.process(Unknown Source) at uk.me.parabola.splitter.Main.processMap(Main.java:792) at uk.me.parabola.splitter.Main.genProblemLists(Main.java:582) at uk.me.parabola.splitter.Main.partitionAreasForProblemListGenerator(Main.java:613) at uk.me.parabola.splitter.Main.split(Main.java:244) at uk.me.parabola.splitter.Main.start(Main.java:157) at uk.me.parabola.splitter.Main.main(Main.java:146) Any ideas? It's splitter r300. Splitter version unknown compiled 2013-04-10T21:31:05+0000 boundary-tags=use-exclude-list cache= description=TK-USA-Tile geonames-file=osmmaps/scripts/cities/USA.txt keep-complete=true mapid=71510001 max-areas=1024 max-nodes=5000000 max-threads=4 (auto) mixed=false no-trim=false output=pbf output-dir=build/usa/tiles-srtm overlap=0 polygon-file= precomp-sea=data/sea/current problem-file= problem-report= resolution=13 split-file= status-freq=120 stop-after=dist write-kml= -- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg)
Hi Thorsten, it seems that your data has reached a limit in the data structure used in splitter. This happens when you have a huge number of consecutive ids. Can you configure the way how the ids for the points are generated? A simple change would be to use 1 3 5 7 .. instead of 1 2 3 4 ... If that doesn't help: Can you tell me how to create the data or can you post a link? Gerd Thorsten Kukuk wrote
Hi,
I generated SRTM data for the USA with phygthmap and wanted to try to split it now for compiling with mkgmap. But while this worked fine for all other countries, in this case I get always the following exception:
MAP occupancy: 2.350.000.000, number of area dictionary entries: 3994 of 65535 Map details: bytes/overhead 401.218.630 / 166.809.090, overhead includes 19 arrays with 8 MB 2.350.000.000 nodes processed... id=2526831732 MAP occupancy: 2.360.000.000, number of area dictionary entries: 4000 of 65535 Map details: bytes/overhead 402.878.606 / 166.808.202, overhead includes 19 arrays with 8 MB 2.360.000.000 nodes processed... id=2536831732 Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 524288 at uk.me.parabola.splitter.SparseLong2ShortMapInline.putChunk(SparseLong2ShortMapInline.java:448) at uk.me.parabola.splitter.SparseLong2ShortMapInline.saveCurrentChunk(SparseLong2ShortMapInline.java:191) at uk.me.parabola.splitter.SparseLong2ShortMapInline.put(SparseLong2ShortMapInline.java:218) at uk.me.parabola.splitter.ProblemListProcessor.processNode(ProblemListProcessor.java:167) at uk.me.parabola.splitter.BinaryMapParser.parseDense(BinaryMapParser.java:113) at crosby.binary.BinaryParser.parse(Unknown Source) at crosby.binary.BinaryParser.handleBlock(Unknown Source) at crosby.binary.file.FileBlock.process(Unknown Source) at crosby.binary.file.BlockInputStream.process(Unknown Source) at uk.me.parabola.splitter.Main.processMap(Main.java:792) at uk.me.parabola.splitter.Main.genProblemLists(Main.java:582) at uk.me.parabola.splitter.Main.partitionAreasForProblemListGenerator(Main.java:613) at uk.me.parabola.splitter.Main.split(Main.java:244) at uk.me.parabola.splitter.Main.start(Main.java:157) at uk.me.parabola.splitter.Main.main(Main.java:146)
Any ideas? It's splitter r300.
Splitter version unknown compiled 2013-04-10T21:31:05+0000 boundary-tags=use-exclude-list cache= description=TK-USA-Tile geonames-file=osmmaps/scripts/cities/USA.txt keep-complete=true mapid=71510001 max-areas=1024 max-nodes=5000000 max-threads=4 (auto) mixed=false no-trim=false output=pbf output-dir=build/usa/tiles-srtm overlap=0 polygon-file= precomp-sea=data/sea/current problem-file= problem-report= resolution=13 split-file= status-freq=120 stop-after=dist write-kml=
-- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) _______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
-- View this message in context: http://gis.19327.n5.nabble.com/splitter-exception-with-SRTM-data-for-USA-tp5... Sent from the Mkgmap Development mailing list archive at Nabble.com.
Am 11.04.2013 15:19, schrieb GerdP:
Hi Thorsten,
it seems that your data has reached a limit in the data structure used in splitter. This happens when you have a huge number of consecutive ids. Can you configure the way how the ids for the points are generated? A simple change would be to use 1 3 5 7 .. instead of 1 2 3 4 ...
If that doesn't help: Can you tell me how to create the data or can you post a link? Typical its ongoing. You can set a start-value for way-id and node-id.
Henning
Hi Thorsten, another question: The wiki of phygthmap says that you don't need splitting: http://wiki.openstreetmap.org/wiki/Phyghtmap Why do you do it? Gerd
Date: Thu, 11 Apr 2013 14:06:30 +0200 From: kukuk@suse.de To: mkgmap-dev@lists.mkgmap.org.uk Subject: [mkgmap-dev] splitter exception with SRTM data for USA
Hi,
I generated SRTM data for the USA with phygthmap and wanted to try to split it now for compiling with mkgmap. But while this worked fine for all other countries, in this case I get always the following exception:
MAP occupancy: 2.350.000.000, number of area dictionary entries: 3994 of 65535 Map details: bytes/overhead 401.218.630 / 166.809.090, overhead includes 19 arrays with 8 MB 2.350.000.000 nodes processed... id=2526831732 MAP occupancy: 2.360.000.000, number of area dictionary entries: 4000 of 65535 Map details: bytes/overhead 402.878.606 / 166.808.202, overhead includes 19 arrays with 8 MB 2.360.000.000 nodes processed... id=2536831732 Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 524288 at uk.me.parabola.splitter.SparseLong2ShortMapInline.putChunk(SparseLong2ShortMapInline.java:448) at uk.me.parabola.splitter.SparseLong2ShortMapInline.saveCurrentChunk(SparseLong2ShortMapInline.java:191) at uk.me.parabola.splitter.SparseLong2ShortMapInline.put(SparseLong2ShortMapInline.java:218) at uk.me.parabola.splitter.ProblemListProcessor.processNode(ProblemListProcessor.java:167) at uk.me.parabola.splitter.BinaryMapParser.parseDense(BinaryMapParser.java:113) at crosby.binary.BinaryParser.parse(Unknown Source) at crosby.binary.BinaryParser.handleBlock(Unknown Source) at crosby.binary.file.FileBlock.process(Unknown Source) at crosby.binary.file.BlockInputStream.process(Unknown Source) at uk.me.parabola.splitter.Main.processMap(Main.java:792) at uk.me.parabola.splitter.Main.genProblemLists(Main.java:582) at uk.me.parabola.splitter.Main.partitionAreasForProblemListGenerator(Main.java:613) at uk.me.parabola.splitter.Main.split(Main.java:244) at uk.me.parabola.splitter.Main.start(Main.java:157) at uk.me.parabola.splitter.Main.main(Main.java:146)
Any ideas? It's splitter r300.
Splitter version unknown compiled 2013-04-10T21:31:05+0000 boundary-tags=use-exclude-list cache= description=TK-USA-Tile geonames-file=osmmaps/scripts/cities/USA.txt keep-complete=true mapid=71510001 max-areas=1024 max-nodes=5000000 max-threads=4 (auto) mixed=false no-trim=false output=pbf output-dir=build/usa/tiles-srtm overlap=0 polygon-file= precomp-sea=data/sea/current problem-file= problem-report= resolution=13 split-file= status-freq=120 stop-after=dist write-kml=
-- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Gerd, On Thu, Apr 11, Gerd Petermann wrote:
Hi Thorsten,
another question: The wiki of phygthmap says that you don't need splitting: http://wiki.openstreetmap.org/wiki/Phyghtmap
Why do you do it?
Because I create one big data file with phyghtmap and then extract the parts I need. That's much easier and simpler then to let phyghtmap create a big amount of files and try to find and cut the ones you need. Thorsten
Gerd
Date: Thu, 11 Apr 2013 14:06:30 +0200 From: kukuk@suse.de To: mkgmap-dev@lists.mkgmap.org.uk Subject: [mkgmap-dev] splitter exception with SRTM data for USA
Hi,
I generated SRTM data for the USA with phygthmap and wanted to try to split it now for compiling with mkgmap. But while this worked fine for all other countries, in this case I get always the following exception:
MAP occupancy: 2.350.000.000, number of area dictionary entries: 3994 of 65535 Map details: bytes/overhead 401.218.630 / 166.809.090, overhead includes 19 arrays with 8 MB 2.350.000.000 nodes processed... id=2526831732 MAP occupancy: 2.360.000.000, number of area dictionary entries: 4000 of 65535 Map details: bytes/overhead 402.878.606 / 166.808.202, overhead includes 19 arrays with 8 MB 2.360.000.000 nodes processed... id=2536831732 Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 524288 at uk.me.parabola.splitter.SparseLong2ShortMapInline.putChunk(SparseLong2ShortMapInline.java:448) at uk.me.parabola.splitter.SparseLong2ShortMapInline.saveCurrentChunk(SparseLong2ShortMapInline.java:191) at uk.me.parabola.splitter.SparseLong2ShortMapInline.put(SparseLong2ShortMapInline.java:218) at uk.me.parabola.splitter.ProblemListProcessor.processNode(ProblemListProcessor.java:167) at uk.me.parabola.splitter.BinaryMapParser.parseDense(BinaryMapParser.java:113) at crosby.binary.BinaryParser.parse(Unknown Source) at crosby.binary.BinaryParser.handleBlock(Unknown Source) at crosby.binary.file.FileBlock.process(Unknown Source) at crosby.binary.file.BlockInputStream.process(Unknown Source) at uk.me.parabola.splitter.Main.processMap(Main.java:792) at uk.me.parabola.splitter.Main.genProblemLists(Main.java:582) at uk.me.parabola.splitter.Main.partitionAreasForProblemListGenerator(Main.java:613) at uk.me.parabola.splitter.Main.split(Main.java:244) at uk.me.parabola.splitter.Main.start(Main.java:157) at uk.me.parabola.splitter.Main.main(Main.java:146)
Any ideas? It's splitter r300.
Splitter version unknown compiled 2013-04-10T21:31:05+0000 boundary-tags=use-exclude-list cache= description=TK-USA-Tile geonames-file=osmmaps/scripts/cities/USA.txt keep-complete=true mapid=71510001 max-areas=1024 max-nodes=5000000 max-threads=4 (auto) mixed=false no-trim=false output=pbf output-dir=build/usa/tiles-srtm overlap=0 polygon-file= precomp-sea=data/sea/current problem-file= problem-report= resolution=13 split-file= status-freq=120 stop-after=dist write-kml=
-- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg)
are you using the newest version of pyghtmap, newest splitter, and no older than 3-4 weeks mkgmap? I had a similar problem with Asia. mkgmap splitter wouldn't split the pyghtmap extract - no matter what I tried (splitter didn't split but simply output one huge file instead). (about 6 month ago) - 1 month ago I redid everything using newest versions, and it worked... On 11.04.2013 16:33, Thorsten Kukuk wrote:
Hi Gerd,
On Thu, Apr 11, Gerd Petermann wrote:
Hi Thorsten,
another question: The wiki of phygthmap says that you don't need splitting: http://wiki.openstreetmap.org/wiki/Phyghtmap
Why do you do it? Because I create one big data file with phyghtmap and then extract the parts I need. That's much easier and simpler then to let phyghtmap create a big amount of files and try to find and cut the ones you need.
Thorsten
Gerd
Date: Thu, 11 Apr 2013 14:06:30 +0200 From: kukuk@suse.de To: mkgmap-dev@lists.mkgmap.org.uk Subject: [mkgmap-dev] splitter exception with SRTM data for USA
Hi,
I generated SRTM data for the USA with phygthmap and wanted to try to split it now for compiling with mkgmap. But while this worked fine for all other countries, in this case I get always the following exception:
MAP occupancy: 2.350.000.000, number of area dictionary entries: 3994 of 65535 Map details: bytes/overhead 401.218.630 / 166.809.090, overhead includes 19 arrays with 8 MB 2.350.000.000 nodes processed... id=2526831732 MAP occupancy: 2.360.000.000, number of area dictionary entries: 4000 of 65535 Map details: bytes/overhead 402.878.606 / 166.808.202, overhead includes 19 arrays with 8 MB 2.360.000.000 nodes processed... id=2536831732 Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 524288 at uk.me.parabola.splitter.SparseLong2ShortMapInline.putChunk(SparseLong2ShortMapInline.java:448) at uk.me.parabola.splitter.SparseLong2ShortMapInline.saveCurrentChunk(SparseLong2ShortMapInline.java:191) at uk.me.parabola.splitter.SparseLong2ShortMapInline.put(SparseLong2ShortMapInline.java:218) at uk.me.parabola.splitter.ProblemListProcessor.processNode(ProblemListProcessor.java:167) at uk.me.parabola.splitter.BinaryMapParser.parseDense(BinaryMapParser.java:113) at crosby.binary.BinaryParser.parse(Unknown Source) at crosby.binary.BinaryParser.handleBlock(Unknown Source) at crosby.binary.file.FileBlock.process(Unknown Source) at crosby.binary.file.BlockInputStream.process(Unknown Source) at uk.me.parabola.splitter.Main.processMap(Main.java:792) at uk.me.parabola.splitter.Main.genProblemLists(Main.java:582) at uk.me.parabola.splitter.Main.partitionAreasForProblemListGenerator(Main.java:613) at uk.me.parabola.splitter.Main.split(Main.java:244) at uk.me.parabola.splitter.Main.start(Main.java:157) at uk.me.parabola.splitter.Main.main(Main.java:146)
Any ideas? It's splitter r300.
Splitter version unknown compiled 2013-04-10T21:31:05+0000 boundary-tags=use-exclude-list cache= description=TK-USA-Tile geonames-file=osmmaps/scripts/cities/USA.txt keep-complete=true mapid=71510001 max-areas=1024 max-nodes=5000000 max-threads=4 (auto) mixed=false no-trim=false output=pbf output-dir=build/usa/tiles-srtm overlap=0 polygon-file= precomp-sea=data/sea/current problem-file= problem-report= resolution=13 split-file= status-freq=120 stop-after=dist write-kml=
-- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- keep on biking and discovering new trails Felix openmtbmap.org & www.velomap.org
Hi, On Thu, Apr 11, Felix Hartmann wrote:
are you using the newest version of pyghtmap, newest splitter, and no older than 3-4 weeks mkgmap?
I had a similar problem with Asia. mkgmap splitter wouldn't split the pyghtmap extract - no matter what I tried (splitter didn't split but simply output one huge file instead). (about 6 month ago) - 1 month ago I redid everything using newest versions, and it worked...
Since I wrote a lot of the patches for the last phyghtmap versions, I use of course the newest one. And it is only with the US. Canada, but that's a little bit smaller, doesn't create any problems. Thorsten
On 11.04.2013 16:33, Thorsten Kukuk wrote:
Hi Gerd,
On Thu, Apr 11, Gerd Petermann wrote:
Hi Thorsten,
another question: The wiki of phygthmap says that you don't need splitting: http://wiki.openstreetmap.org/wiki/Phyghtmap
Why do you do it? Because I create one big data file with phyghtmap and then extract the parts I need. That's much easier and simpler then to let phyghtmap create a big amount of files and try to find and cut the ones you need.
Thorsten
Gerd
Date: Thu, 11 Apr 2013 14:06:30 +0200 From: kukuk@suse.de To: mkgmap-dev@lists.mkgmap.org.uk Subject: [mkgmap-dev] splitter exception with SRTM data for USA
Hi,
I generated SRTM data for the USA with phygthmap and wanted to try to split it now for compiling with mkgmap. But while this worked fine for all other countries, in this case I get always the following exception:
MAP occupancy: 2.350.000.000, number of area dictionary entries: 3994 of 65535 Map details: bytes/overhead 401.218.630 / 166.809.090, overhead includes 19 arrays with 8 MB 2.350.000.000 nodes processed... id=2526831732 MAP occupancy: 2.360.000.000, number of area dictionary entries: 4000 of 65535 Map details: bytes/overhead 402.878.606 / 166.808.202, overhead includes 19 arrays with 8 MB 2.360.000.000 nodes processed... id=2536831732 Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 524288 at uk.me.parabola.splitter.SparseLong2ShortMapInline.putChunk(SparseLong2ShortMapInline.java:448) at uk.me.parabola.splitter.SparseLong2ShortMapInline.saveCurrentChunk(SparseLong2ShortMapInline.java:191) at uk.me.parabola.splitter.SparseLong2ShortMapInline.put(SparseLong2ShortMapInline.java:218) at uk.me.parabola.splitter.ProblemListProcessor.processNode(ProblemListProcessor.java:167) at uk.me.parabola.splitter.BinaryMapParser.parseDense(BinaryMapParser.java:113) at crosby.binary.BinaryParser.parse(Unknown Source) at crosby.binary.BinaryParser.handleBlock(Unknown Source) at crosby.binary.file.FileBlock.process(Unknown Source) at crosby.binary.file.BlockInputStream.process(Unknown Source) at uk.me.parabola.splitter.Main.processMap(Main.java:792) at uk.me.parabola.splitter.Main.genProblemLists(Main.java:582) at uk.me.parabola.splitter.Main.partitionAreasForProblemListGenerator(Main.java:613) at uk.me.parabola.splitter.Main.split(Main.java:244) at uk.me.parabola.splitter.Main.start(Main.java:157) at uk.me.parabola.splitter.Main.main(Main.java:146)
Any ideas? It's splitter r300.
Splitter version unknown compiled 2013-04-10T21:31:05+0000 boundary-tags=use-exclude-list cache= description=TK-USA-Tile geonames-file=osmmaps/scripts/cities/USA.txt keep-complete=true mapid=71510001 max-areas=1024 max-nodes=5000000 max-threads=4 (auto) mixed=false no-trim=false output=pbf output-dir=build/usa/tiles-srtm overlap=0 polygon-file= precomp-sea=data/sea/current problem-file= problem-report= resolution=13 split-file= status-freq=120 stop-after=dist write-kml=
-- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- keep on biking and discovering new trails
Felix openmtbmap.org & www.velomap.org
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg)
Hi Thorsten, ok, so I assume that you can patch pyghtmap to generate the ids as I requested. In the mean time I will write a small generator which creates OSM data with a large number of nodes. Maybe it is a bug in splitter, not a limit. Gerd Thorsten Kukuk wrote
Hi,
On Thu, Apr 11, Felix Hartmann wrote:
are you using the newest version of pyghtmap, newest splitter, and no older than 3-4 weeks mkgmap?
I had a similar problem with Asia. mkgmap splitter wouldn't split the pyghtmap extract - no matter what I tried (splitter didn't split but simply output one huge file instead). (about 6 month ago) - 1 month ago I redid everything using newest versions, and it worked...
Since I wrote a lot of the patches for the last phyghtmap versions, I use of course the newest one.
And it is only with the US. Canada, but that's a little bit smaller, doesn't create any problems.
Thorsten
On 11.04.2013 16:33, Thorsten Kukuk wrote:
Hi Gerd,
On Thu, Apr 11, Gerd Petermann wrote:
Hi Thorsten,
another question: The wiki of phygthmap says that you don't need splitting: http://wiki.openstreetmap.org/wiki/Phyghtmap
Why do you do it? Because I create one big data file with phyghtmap and then extract the parts I need. That's much easier and simpler then to let phyghtmap create a big amount of files and try to find and cut the ones you need.
Thorsten
Gerd
Date: Thu, 11 Apr 2013 14:06:30 +0200 From:
kukuk@
To:
mkgmap-dev@.org
Subject: [mkgmap-dev] splitter exception with SRTM data for USA
Hi,
I generated SRTM data for the USA with phygthmap and wanted to try to split it now for compiling with mkgmap. But while this worked fine for all other countries, in this case I get always the following exception:
MAP occupancy: 2.350.000.000, number of area dictionary entries: 3994 of 65535 Map details: bytes/overhead 401.218.630 / 166.809.090, overhead includes 19 arrays with 8 MB 2.350.000.000 nodes processed... id=2526831732 MAP occupancy: 2.360.000.000, number of area dictionary entries: 4000 of 65535 Map details: bytes/overhead 402.878.606 / 166.808.202, overhead includes 19 arrays with 8 MB 2.360.000.000 nodes processed... id=2536831732 Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 524288 at uk.me.parabola.splitter.SparseLong2ShortMapInline.putChunk(SparseLong2ShortMapInline.java:448) at uk.me.parabola.splitter.SparseLong2ShortMapInline.saveCurrentChunk(SparseLong2ShortMapInline.java:191) at uk.me.parabola.splitter.SparseLong2ShortMapInline.put(SparseLong2ShortMapInline.java:218) at uk.me.parabola.splitter.ProblemListProcessor.processNode(ProblemListProcessor.java:167) at uk.me.parabola.splitter.BinaryMapParser.parseDense(BinaryMapParser.java:113) at crosby.binary.BinaryParser.parse(Unknown Source) at crosby.binary.BinaryParser.handleBlock(Unknown Source) at crosby.binary.file.FileBlock.process(Unknown Source) at crosby.binary.file.BlockInputStream.process(Unknown Source) at uk.me.parabola.splitter.Main.processMap(Main.java:792) at uk.me.parabola.splitter.Main.genProblemLists(Main.java:582) at uk.me.parabola.splitter.Main.partitionAreasForProblemListGenerator(Main.java:613) at uk.me.parabola.splitter.Main.split(Main.java:244) at uk.me.parabola.splitter.Main.start(Main.java:157) at uk.me.parabola.splitter.Main.main(Main.java:146)
Any ideas? It's splitter r300.
Splitter version unknown compiled 2013-04-10T21:31:05+0000 boundary-tags=use-exclude-list cache= description=TK-USA-Tile geonames-file=osmmaps/scripts/cities/USA.txt keep-complete=true mapid=71510001 max-areas=1024 max-nodes=5000000 max-threads=4 (auto) mixed=false no-trim=false output=pbf output-dir=build/usa/tiles-srtm overlap=0 polygon-file= precomp-sea=data/sea/current problem-file= problem-report= resolution=13 split-file= status-freq=120 stop-after=dist write-kml=
-- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) _______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
-- keep on biking and discovering new trails
Felix openmtbmap.org & www.velomap.org
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
-- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) _______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
-- View this message in context: http://gis.19327.n5.nabble.com/splitter-exception-with-SRTM-data-for-USA-tp5... Sent from the Mkgmap Development mailing list archive at Nabble.com.
Hi Gerd, On Thu, Apr 11, GerdP wrote:
Hi Thorsten,
ok, so I assume that you can patch pyghtmap to generate the ids as I requested.
Yes, I will try that over the weekend, generating such big data needs some time. Thorsten
In the mean time I will write a small generator which creates OSM data with a large number of nodes. Maybe it is a bug in splitter, not a limit.
Gerd
Thorsten Kukuk wrote
Hi,
On Thu, Apr 11, Felix Hartmann wrote:
are you using the newest version of pyghtmap, newest splitter, and no older than 3-4 weeks mkgmap?
I had a similar problem with Asia. mkgmap splitter wouldn't split the pyghtmap extract - no matter what I tried (splitter didn't split but simply output one huge file instead). (about 6 month ago) - 1 month ago I redid everything using newest versions, and it worked...
Since I wrote a lot of the patches for the last phyghtmap versions, I use of course the newest one.
And it is only with the US. Canada, but that's a little bit smaller, doesn't create any problems.
Thorsten
On 11.04.2013 16:33, Thorsten Kukuk wrote:
Hi Gerd,
On Thu, Apr 11, Gerd Petermann wrote:
Hi Thorsten,
another question: The wiki of phygthmap says that you don't need splitting: http://wiki.openstreetmap.org/wiki/Phyghtmap
Why do you do it? Because I create one big data file with phyghtmap and then extract the parts I need. That's much easier and simpler then to let phyghtmap create a big amount of files and try to find and cut the ones you need.
Thorsten
Gerd
Date: Thu, 11 Apr 2013 14:06:30 +0200 From:
kukuk@
To:
mkgmap-dev@.org
Subject: [mkgmap-dev] splitter exception with SRTM data for USA
Hi,
I generated SRTM data for the USA with phygthmap and wanted to try to split it now for compiling with mkgmap. But while this worked fine for all other countries, in this case I get always the following exception:
MAP occupancy: 2.350.000.000, number of area dictionary entries: 3994 of 65535 Map details: bytes/overhead 401.218.630 / 166.809.090, overhead includes 19 arrays with 8 MB 2.350.000.000 nodes processed... id=2526831732 MAP occupancy: 2.360.000.000, number of area dictionary entries: 4000 of 65535 Map details: bytes/overhead 402.878.606 / 166.808.202, overhead includes 19 arrays with 8 MB 2.360.000.000 nodes processed... id=2536831732 Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 524288 at uk.me.parabola.splitter.SparseLong2ShortMapInline.putChunk(SparseLong2ShortMapInline.java:448) at uk.me.parabola.splitter.SparseLong2ShortMapInline.saveCurrentChunk(SparseLong2ShortMapInline.java:191) at uk.me.parabola.splitter.SparseLong2ShortMapInline.put(SparseLong2ShortMapInline.java:218) at uk.me.parabola.splitter.ProblemListProcessor.processNode(ProblemListProcessor.java:167) at uk.me.parabola.splitter.BinaryMapParser.parseDense(BinaryMapParser.java:113) at crosby.binary.BinaryParser.parse(Unknown Source) at crosby.binary.BinaryParser.handleBlock(Unknown Source) at crosby.binary.file.FileBlock.process(Unknown Source) at crosby.binary.file.BlockInputStream.process(Unknown Source) at uk.me.parabola.splitter.Main.processMap(Main.java:792) at uk.me.parabola.splitter.Main.genProblemLists(Main.java:582) at uk.me.parabola.splitter.Main.partitionAreasForProblemListGenerator(Main.java:613) at uk.me.parabola.splitter.Main.split(Main.java:244) at uk.me.parabola.splitter.Main.start(Main.java:157) at uk.me.parabola.splitter.Main.main(Main.java:146)
Any ideas? It's splitter r300.
Splitter version unknown compiled 2013-04-10T21:31:05+0000 boundary-tags=use-exclude-list cache= description=TK-USA-Tile geonames-file=osmmaps/scripts/cities/USA.txt keep-complete=true mapid=71510001 max-areas=1024 max-nodes=5000000 max-threads=4 (auto) mixed=false no-trim=false output=pbf output-dir=build/usa/tiles-srtm overlap=0 polygon-file= precomp-sea=data/sea/current problem-file= problem-report= resolution=13 split-file= status-freq=120 stop-after=dist write-kml=
-- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) _______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
-- keep on biking and discovering new trails
Felix openmtbmap.org & www.velomap.org
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
-- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) _______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
-- View this message in context: http://gis.19327.n5.nabble.com/splitter-exception-with-SRTM-data-for-USA-tp5... Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev -- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg)
Hi Thorsten, I fear my idea regarding the numering of node ids will not help. The problem is the amount of nodes and the fact that the generated data probably is distributed in a way that the nodes with id x,x+1,x+2,..,x+n are very close together. This makes it very likely that splitter reaches a limit of 2.147.483.648. With normal OSM data, this is very unlikely. I fear I have to revert the latest optimizations in SparseLong2ShortMapInline to remove that limit. Gerd
Date: Thu, 11 Apr 2013 16:54:13 +0200 From: kukuk@suse.de To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] splitter exception with SRTM data for USA
Hi Gerd,
On Thu, Apr 11, GerdP wrote:
Hi Thorsten,
ok, so I assume that you can patch pyghtmap to generate the ids as I requested.
Yes, I will try that over the weekend, generating such big data needs some time.
Thorsten
In the mean time I will write a small generator which creates OSM data with a large number of nodes. Maybe it is a bug in splitter, not a limit.
Gerd
Thorsten Kukuk wrote
Hi,
On Thu, Apr 11, Felix Hartmann wrote:
are you using the newest version of pyghtmap, newest splitter, and no older than 3-4 weeks mkgmap?
I had a similar problem with Asia. mkgmap splitter wouldn't split the pyghtmap extract - no matter what I tried (splitter didn't split but simply output one huge file instead). (about 6 month ago) - 1 month ago I redid everything using newest versions, and it worked...
Since I wrote a lot of the patches for the last phyghtmap versions, I use of course the newest one.
And it is only with the US. Canada, but that's a little bit smaller, doesn't create any problems.
Thorsten
On 11.04.2013 16:33, Thorsten Kukuk wrote:
Hi Gerd,
On Thu, Apr 11, Gerd Petermann wrote:
Hi Thorsten,
another question: The wiki of phygthmap says that you don't need splitting: http://wiki.openstreetmap.org/wiki/Phyghtmap
Why do you do it? Because I create one big data file with phyghtmap and then extract the parts I need. That's much easier and simpler then to let phyghtmap create a big amount of files and try to find and cut the ones you need.
Thorsten
Gerd
> Date: Thu, 11 Apr 2013 14:06:30 +0200 > From:
kukuk@
> To:
mkgmap-dev@.org
> Subject: [mkgmap-dev] splitter exception with SRTM data for USA > > > Hi, > > I generated SRTM data for the USA with phygthmap and wanted to try > to split it now for compiling with mkgmap. But while this worked fine > for all other countries, in this case I get always the following > exception: > > MAP occupancy: 2.350.000.000, number of area dictionary entries: 3994 of 65535 > Map details: bytes/overhead 401.218.630 / 166.809.090, overhead includes 19 arrays with 8 MB > 2.350.000.000 nodes processed... id=2526831732 > MAP occupancy: 2.360.000.000, number of area dictionary entries: 4000 of 65535 > Map details: bytes/overhead 402.878.606 / 166.808.202, overhead includes 19 arrays with 8 MB > 2.360.000.000 nodes processed... id=2536831732 > Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 524288 > at uk.me.parabola.splitter.SparseLong2ShortMapInline.putChunk(SparseLong2ShortMapInline.java:448) > at uk.me.parabola.splitter.SparseLong2ShortMapInline.saveCurrentChunk(SparseLong2ShortMapInline.java:191) > at uk.me.parabola.splitter.SparseLong2ShortMapInline.put(SparseLong2ShortMapInline.java:218) > at uk.me.parabola.splitter.ProblemListProcessor.processNode(ProblemListProcessor.java:167) > at uk.me.parabola.splitter.BinaryMapParser.parseDense(BinaryMapParser.java:113) > at crosby.binary.BinaryParser.parse(Unknown Source) > at crosby.binary.BinaryParser.handleBlock(Unknown Source) > at crosby.binary.file.FileBlock.process(Unknown Source) > at crosby.binary.file.BlockInputStream.process(Unknown Source) > at uk.me.parabola.splitter.Main.processMap(Main.java:792) > at uk.me.parabola.splitter.Main.genProblemLists(Main.java:582) > at uk.me.parabola.splitter.Main.partitionAreasForProblemListGenerator(Main.java:613) > at uk.me.parabola.splitter.Main.split(Main.java:244) > at uk.me.parabola.splitter.Main.start(Main.java:157) > at uk.me.parabola.splitter.Main.main(Main.java:146) > > Any ideas? It's splitter r300. > > Splitter version unknown compiled 2013-04-10T21:31:05+0000 > boundary-tags=use-exclude-list > cache= > description=TK-USA-Tile > geonames-file=osmmaps/scripts/cities/USA.txt > keep-complete=true > mapid=71510001 > max-areas=1024 > max-nodes=5000000 > max-threads=4 (auto) > mixed=false > no-trim=false > output=pbf > output-dir=build/usa/tiles-srtm > overlap=0 > polygon-file= > precomp-sea=data/sea/current > problem-file= > problem-report= > resolution=13 > split-file= > status-freq=120 > stop-after=dist > write-kml= > > > -- > Thorsten Kukuk, Project Manager/Release Manager SLES > SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg > GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) > _______________________________________________ > mkgmap-dev mailing list >
mkgmap-dev@.org
> http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
-- keep on biking and discovering new trails
Felix openmtbmap.org & www.velomap.org
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
-- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) _______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
-- View this message in context: http://gis.19327.n5.nabble.com/splitter-exception-with-SRTM-data-for-USA-tp5... Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev -- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg)
mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi, On Thu, Apr 11, Gerd Petermann wrote:
Hi Thorsten,
I fear my idea regarding the numering of node ids will not help. The problem is the amount of nodes and the fact that the generated data probably is distributed in a way that the nodes with id x,x+1,x+2,..,x+n are very close together. This makes it very likely that splitter reaches a limit of 2.147.483.648. With normal OSM data, this is very unlikely.
I fear I have to revert the latest optimizations in SparseLong2ShortMapInline to remove that limit.
It looks like splitter will work if I lower the limit of max. nodes in a tile. I don't know which improvements your optimizations are giving, but I don't think that we need to care for this very special case, especially as it looks like I found a workaround. But with this big amount of data, testing will need some more time. Thorsten
Gerd
Date: Thu, 11 Apr 2013 16:54:13 +0200 From: kukuk@suse.de To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] splitter exception with SRTM data for USA
Hi Gerd,
On Thu, Apr 11, GerdP wrote:
Hi Thorsten,
ok, so I assume that you can patch pyghtmap to generate the ids as I requested.
Yes, I will try that over the weekend, generating such big data needs some time.
Thorsten
In the mean time I will write a small generator which creates OSM data with a large number of nodes. Maybe it is a bug in splitter, not a limit.
Gerd
Thorsten Kukuk wrote
Hi,
On Thu, Apr 11, Felix Hartmann wrote:
are you using the newest version of pyghtmap, newest splitter, and no older than 3-4 weeks mkgmap?
I had a similar problem with Asia. mkgmap splitter wouldn't split the pyghtmap extract - no matter what I tried (splitter didn't split but simply output one huge file instead). (about 6 month ago) - 1 month ago I redid everything using newest versions, and it worked...
Since I wrote a lot of the patches for the last phyghtmap versions, I use of course the newest one.
And it is only with the US. Canada, but that's a little bit smaller, doesn't create any problems.
Thorsten
On 11.04.2013 16:33, Thorsten Kukuk wrote:
Hi Gerd,
On Thu, Apr 11, Gerd Petermann wrote:
> Hi Thorsten, > > another question: The wiki of phygthmap says that you don't need splitting: > http://wiki.openstreetmap.org/wiki/Phyghtmap > > Why do you do it? Because I create one big data file with phyghtmap and then extract the parts I need. That's much easier and simpler then to let phyghtmap create a big amount of files and try to find and cut the ones you need.
Thorsten
> Gerd > >> Date: Thu, 11 Apr 2013 14:06:30 +0200 >> From:
kukuk@
>> To:
mkgmap-dev@.org
>> Subject: [mkgmap-dev] splitter exception with SRTM data for USA >> >> >> Hi, >> >> I generated SRTM data for the USA with phygthmap and wanted to try >> to split it now for compiling with mkgmap. But while this worked fine >> for all other countries, in this case I get always the following >> exception: >> >> MAP occupancy: 2.350.000.000, number of area dictionary entries: 3994 of 65535 >> Map details: bytes/overhead 401.218.630 / 166.809.090, overhead includes 19 arrays with 8 MB >> 2.350.000.000 nodes processed... id=2526831732 >> MAP occupancy: 2.360.000.000, number of area dictionary entries: 4000 of 65535 >> Map details: bytes/overhead 402.878.606 / 166.808.202, overhead includes 19 arrays with 8 MB >> 2.360.000.000 nodes processed... id=2536831732 >> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 524288 >> at uk.me.parabola.splitter.SparseLong2ShortMapInline.putChunk(SparseLong2ShortMapInline.java:448) >> at uk.me.parabola.splitter.SparseLong2ShortMapInline.saveCurrentChunk(SparseLong2ShortMapInline.java:191) >> at uk.me.parabola.splitter.SparseLong2ShortMapInline.put(SparseLong2ShortMapInline.java:218) >> at uk.me.parabola.splitter.ProblemListProcessor.processNode(ProblemListProcessor.java:167) >> at uk.me.parabola.splitter.BinaryMapParser.parseDense(BinaryMapParser.java:113) >> at crosby.binary.BinaryParser.parse(Unknown Source) >> at crosby.binary.BinaryParser.handleBlock(Unknown Source) >> at crosby.binary.file.FileBlock.process(Unknown Source) >> at crosby.binary.file.BlockInputStream.process(Unknown Source) >> at uk.me.parabola.splitter.Main.processMap(Main.java:792) >> at uk.me.parabola.splitter.Main.genProblemLists(Main.java:582) >> at uk.me.parabola.splitter.Main.partitionAreasForProblemListGenerator(Main.java:613) >> at uk.me.parabola.splitter.Main.split(Main.java:244) >> at uk.me.parabola.splitter.Main.start(Main.java:157) >> at uk.me.parabola.splitter.Main.main(Main.java:146) >> >> Any ideas? It's splitter r300. >> >> Splitter version unknown compiled 2013-04-10T21:31:05+0000 >> boundary-tags=use-exclude-list >> cache= >> description=TK-USA-Tile >> geonames-file=osmmaps/scripts/cities/USA.txt >> keep-complete=true >> mapid=71510001 >> max-areas=1024 >> max-nodes=5000000 >> max-threads=4 (auto) >> mixed=false >> no-trim=false >> output=pbf >> output-dir=build/usa/tiles-srtm >> overlap=0 >> polygon-file= >> precomp-sea=data/sea/current >> problem-file= >> problem-report= >> resolution=13 >> split-file= >> status-freq=120 >> stop-after=dist >> write-kml= >> >> >> -- >> Thorsten Kukuk, Project Manager/Release Manager SLES >> SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg >> GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) >> _______________________________________________ >> mkgmap-dev mailing list >>
mkgmap-dev@.org
>> http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > _______________________________________________ > mkgmap-dev mailing list >
mkgmap-dev@.org
-- keep on biking and discovering new trails
Felix openmtbmap.org & www.velomap.org
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
-- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) _______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
-- View this message in context: http://gis.19327.n5.nabble.com/splitter-exception-with-SRTM-data-for-USA-tp5... Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev -- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg)
mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg)
Hi Thorsten, good to know. Yes, the smaller the tiles, the less likely you hit the case were the limit is involved. It happens when many consecutive nodes lie in the same area, this gives a special pattern which is saved in one data structure that can only hold up to 2^31 items. I have to do some tests regarding the performance with the old code. I think it was ~ 10% lower memory usage, so I think this is not a good reason to use code that has known limitations and is a lot more complex. Gerd
Date: Thu, 11 Apr 2013 18:51:34 +0200 From: kukuk@suse.de To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] splitter exception with SRTM data for USA
Hi,
On Thu, Apr 11, Gerd Petermann wrote:
Hi Thorsten,
I fear my idea regarding the numering of node ids will not help. The problem is the amount of nodes and the fact that the generated data probably is distributed in a way that the nodes with id x,x+1,x+2,..,x+n are very close together. This makes it very likely that splitter reaches a limit of 2.147.483.648. With normal OSM data, this is very unlikely.
I fear I have to revert the latest optimizations in SparseLong2ShortMapInline to remove that limit.
It looks like splitter will work if I lower the limit of max. nodes in a tile. I don't know which improvements your optimizations are giving, but I don't think that we need to care for this very special case, especially as it looks like I found a workaround. But with this big amount of data, testing will need some more time.
Thorsten
Gerd
Date: Thu, 11 Apr 2013 16:54:13 +0200 From: kukuk@suse.de To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] splitter exception with SRTM data for USA
Hi Gerd,
On Thu, Apr 11, GerdP wrote:
Hi Thorsten,
ok, so I assume that you can patch pyghtmap to generate the ids as I requested.
Yes, I will try that over the weekend, generating such big data needs some time.
Thorsten
In the mean time I will write a small generator which creates OSM data with a large number of nodes. Maybe it is a bug in splitter, not a limit.
Gerd
Thorsten Kukuk wrote
Hi,
On Thu, Apr 11, Felix Hartmann wrote:
are you using the newest version of pyghtmap, newest splitter, and no older than 3-4 weeks mkgmap?
I had a similar problem with Asia. mkgmap splitter wouldn't split the pyghtmap extract - no matter what I tried (splitter didn't split but simply output one huge file instead). (about 6 month ago) - 1 month ago I redid everything using newest versions, and it worked...
Since I wrote a lot of the patches for the last phyghtmap versions, I use of course the newest one.
And it is only with the US. Canada, but that's a little bit smaller, doesn't create any problems.
Thorsten
On 11.04.2013 16:33, Thorsten Kukuk wrote: > Hi Gerd, > > On Thu, Apr 11, Gerd Petermann wrote: > >> Hi Thorsten, >> >> another question: The wiki of phygthmap says that you don't need splitting: >> http://wiki.openstreetmap.org/wiki/Phyghtmap >> >> Why do you do it? > Because I create one big data file with phyghtmap and then extract > the parts I need. That's much easier and simpler then to let phyghtmap > create a big amount of files and try to find and cut the ones you need. > > Thorsten > >> Gerd >> >>> Date: Thu, 11 Apr 2013 14:06:30 +0200 >>> From:
kukuk@
>>> To:
mkgmap-dev@.org
>>> Subject: [mkgmap-dev] splitter exception with SRTM data for USA >>> >>> >>> Hi, >>> >>> I generated SRTM data for the USA with phygthmap and wanted to try >>> to split it now for compiling with mkgmap. But while this worked fine >>> for all other countries, in this case I get always the following >>> exception: >>> >>> MAP occupancy: 2.350.000.000, number of area dictionary entries: 3994 of 65535 >>> Map details: bytes/overhead 401.218.630 / 166.809.090, overhead includes 19 arrays with 8 MB >>> 2.350.000.000 nodes processed... id=2526831732 >>> MAP occupancy: 2.360.000.000, number of area dictionary entries: 4000 of 65535 >>> Map details: bytes/overhead 402.878.606 / 166.808.202, overhead includes 19 arrays with 8 MB >>> 2.360.000.000 nodes processed... id=2536831732 >>> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 524288 >>> at uk.me.parabola.splitter.SparseLong2ShortMapInline.putChunk(SparseLong2ShortMapInline.java:448) >>> at uk.me.parabola.splitter.SparseLong2ShortMapInline.saveCurrentChunk(SparseLong2ShortMapInline.java:191) >>> at uk.me.parabola.splitter.SparseLong2ShortMapInline.put(SparseLong2ShortMapInline.java:218) >>> at uk.me.parabola.splitter.ProblemListProcessor.processNode(ProblemListProcessor.java:167) >>> at uk.me.parabola.splitter.BinaryMapParser.parseDense(BinaryMapParser.java:113) >>> at crosby.binary.BinaryParser.parse(Unknown Source) >>> at crosby.binary.BinaryParser.handleBlock(Unknown Source) >>> at crosby.binary.file.FileBlock.process(Unknown Source) >>> at crosby.binary.file.BlockInputStream.process(Unknown Source) >>> at uk.me.parabola.splitter.Main.processMap(Main.java:792) >>> at uk.me.parabola.splitter.Main.genProblemLists(Main.java:582) >>> at uk.me.parabola.splitter.Main.partitionAreasForProblemListGenerator(Main.java:613) >>> at uk.me.parabola.splitter.Main.split(Main.java:244) >>> at uk.me.parabola.splitter.Main.start(Main.java:157) >>> at uk.me.parabola.splitter.Main.main(Main.java:146) >>> >>> Any ideas? It's splitter r300. >>> >>> Splitter version unknown compiled 2013-04-10T21:31:05+0000 >>> boundary-tags=use-exclude-list >>> cache= >>> description=TK-USA-Tile >>> geonames-file=osmmaps/scripts/cities/USA.txt >>> keep-complete=true >>> mapid=71510001 >>> max-areas=1024 >>> max-nodes=5000000 >>> max-threads=4 (auto) >>> mixed=false >>> no-trim=false >>> output=pbf >>> output-dir=build/usa/tiles-srtm >>> overlap=0 >>> polygon-file= >>> precomp-sea=data/sea/current >>> problem-file= >>> problem-report= >>> resolution=13 >>> split-file= >>> status-freq=120 >>> stop-after=dist >>> write-kml= >>> >>> >>> -- >>> Thorsten Kukuk, Project Manager/Release Manager SLES >>> SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg >>> GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) >>> _______________________________________________ >>> mkgmap-dev mailing list >>>
mkgmap-dev@.org
>>> http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev >> >> _______________________________________________ >> mkgmap-dev mailing list >>
mkgmap-dev@.org
>> http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- keep on biking and discovering new trails
Felix openmtbmap.org & www.velomap.org
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
-- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) _______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
-- View this message in context: http://gis.19327.n5.nabble.com/splitter-exception-with-SRTM-data-for-USA-tp5... Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev -- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg)
mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Thorsten, the optimized data structure requires ~ 45% less memory (e.g. ~ 330MB instead of ~ 606MB for germany.osm.pbf) , so it would be great if we could keep this for a while. On the other hand, the limit will also be reached sooner or later when splitting a whole planet.osm.pbf, so I plan to implement a switch which allows to use the older (less optimized, but able to more data) version. My planet.osm.pbf is from 2012-11-07. The critical value is reported in the splitter log : ... Statistics for coords map: Length-1 chunks: 4.580.310, used Bytes including overhead: 47.376.048 ... If this value reaches 64 * 524288 = 33.554.432 you will see the error message Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 524288 I assume the numbers will not be much higher with a current planet, so we have probably not reached 20% of the critical value. Maybe you can change the parms for pyghtmap to generate fewer nodes? I assume that the most important value is the --step option, a slightly higher value should already produce much smaller number of nodes. I don't know if pyghtmap uses an algo like Douglas-Peucker to reduce points? Gerd
Date: Thu, 11 Apr 2013 18:51:34 +0200 From: kukuk@suse.de To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] splitter exception with SRTM data for USA
Hi,
On Thu, Apr 11, Gerd Petermann wrote:
Hi Thorsten,
I fear my idea regarding the numering of node ids will not help. The problem is the amount of nodes and the fact that the generated data probably is distributed in a way that the nodes with id x,x+1,x+2,..,x+n are very close together. This makes it very likely that splitter reaches a limit of 2.147.483.648. With normal OSM data, this is very unlikely.
I fear I have to revert the latest optimizations in SparseLong2ShortMapInline to remove that limit.
It looks like splitter will work if I lower the limit of max. nodes in a tile. I don't know which improvements your optimizations are giving, but I don't think that we need to care for this very special case, especially as it looks like I found a workaround. But with this big amount of data, testing will need some more time.
Thorsten
Gerd
Date: Thu, 11 Apr 2013 16:54:13 +0200 From: kukuk@suse.de To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] splitter exception with SRTM data for USA
Hi Gerd,
On Thu, Apr 11, GerdP wrote:
Hi Thorsten,
ok, so I assume that you can patch pyghtmap to generate the ids as I requested.
Yes, I will try that over the weekend, generating such big data needs some time.
Thorsten
In the mean time I will write a small generator which creates OSM data with a large number of nodes. Maybe it is a bug in splitter, not a limit.
Gerd
Thorsten Kukuk wrote
Hi,
On Thu, Apr 11, Felix Hartmann wrote:
are you using the newest version of pyghtmap, newest splitter, and no older than 3-4 weeks mkgmap?
I had a similar problem with Asia. mkgmap splitter wouldn't split the pyghtmap extract - no matter what I tried (splitter didn't split but simply output one huge file instead). (about 6 month ago) - 1 month ago I redid everything using newest versions, and it worked...
Since I wrote a lot of the patches for the last phyghtmap versions, I use of course the newest one.
And it is only with the US. Canada, but that's a little bit smaller, doesn't create any problems.
Thorsten
On 11.04.2013 16:33, Thorsten Kukuk wrote: > Hi Gerd, > > On Thu, Apr 11, Gerd Petermann wrote: > >> Hi Thorsten, >> >> another question: The wiki of phygthmap says that you don't need splitting: >> http://wiki.openstreetmap.org/wiki/Phyghtmap >> >> Why do you do it? > Because I create one big data file with phyghtmap and then extract > the parts I need. That's much easier and simpler then to let phyghtmap > create a big amount of files and try to find and cut the ones you need. > > Thorsten > >> Gerd >> >>> Date: Thu, 11 Apr 2013 14:06:30 +0200 >>> From:
kukuk@
>>> To:
mkgmap-dev@.org
>>> Subject: [mkgmap-dev] splitter exception with SRTM data for USA >>> >>> >>> Hi, >>> >>> I generated SRTM data for the USA with phygthmap and wanted to try >>> to split it now for compiling with mkgmap. But while this worked fine >>> for all other countries, in this case I get always the following >>> exception: >>> >>> MAP occupancy: 2.350.000.000, number of area dictionary entries: 3994 of 65535 >>> Map details: bytes/overhead 401.218.630 / 166.809.090, overhead includes 19 arrays with 8 MB >>> 2.350.000.000 nodes processed... id=2526831732 >>> MAP occupancy: 2.360.000.000, number of area dictionary entries: 4000 of 65535 >>> Map details: bytes/overhead 402.878.606 / 166.808.202, overhead includes 19 arrays with 8 MB >>> 2.360.000.000 nodes processed... id=2536831732 >>> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 524288 >>> at uk.me.parabola.splitter.SparseLong2ShortMapInline.putChunk(SparseLong2ShortMapInline.java:448) >>> at uk.me.parabola.splitter.SparseLong2ShortMapInline.saveCurrentChunk(SparseLong2ShortMapInline.java:191) >>> at uk.me.parabola.splitter.SparseLong2ShortMapInline.put(SparseLong2ShortMapInline.java:218) >>> at uk.me.parabola.splitter.ProblemListProcessor.processNode(ProblemListProcessor.java:167) >>> at uk.me.parabola.splitter.BinaryMapParser.parseDense(BinaryMapParser.java:113) >>> at crosby.binary.BinaryParser.parse(Unknown Source) >>> at crosby.binary.BinaryParser.handleBlock(Unknown Source) >>> at crosby.binary.file.FileBlock.process(Unknown Source) >>> at crosby.binary.file.BlockInputStream.process(Unknown Source) >>> at uk.me.parabola.splitter.Main.processMap(Main.java:792) >>> at uk.me.parabola.splitter.Main.genProblemLists(Main.java:582) >>> at uk.me.parabola.splitter.Main.partitionAreasForProblemListGenerator(Main.java:613) >>> at uk.me.parabola.splitter.Main.split(Main.java:244) >>> at uk.me.parabola.splitter.Main.start(Main.java:157) >>> at uk.me.parabola.splitter.Main.main(Main.java:146) >>> >>> Any ideas? It's splitter r300. >>> >>> Splitter version unknown compiled 2013-04-10T21:31:05+0000 >>> boundary-tags=use-exclude-list >>> cache= >>> description=TK-USA-Tile >>> geonames-file=osmmaps/scripts/cities/USA.txt >>> keep-complete=true >>> mapid=71510001 >>> max-areas=1024 >>> max-nodes=5000000 >>> max-threads=4 (auto) >>> mixed=false >>> no-trim=false >>> output=pbf >>> output-dir=build/usa/tiles-srtm >>> overlap=0 >>> polygon-file= >>> precomp-sea=data/sea/current >>> problem-file= >>> problem-report= >>> resolution=13 >>> split-file= >>> status-freq=120 >>> stop-after=dist >>> write-kml= >>> >>> >>> -- >>> Thorsten Kukuk, Project Manager/Release Manager SLES >>> SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg >>> GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) >>> _______________________________________________ >>> mkgmap-dev mailing list >>>
mkgmap-dev@.org
>>> http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev >> >> _______________________________________________ >> mkgmap-dev mailing list >>
mkgmap-dev@.org
>> http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- keep on biking and discovering new trails
Felix openmtbmap.org & www.velomap.org
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
-- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) _______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
-- View this message in context: http://gis.19327.n5.nabble.com/splitter-exception-with-SRTM-data-for-USA-tp5... Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev -- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg)
mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Am 12.04.2013 09:22, schrieb Gerd Petermann:
Statistics for *coords *map: *Length-1* chunks:*4.580.310*, used Bytes including overhead: 47.376.048 Planet with data of yesterday:
Length-1 chunks: 10.454.925, used Bytes including overhead: 107.695.080 If the rising continueslinear, the limit will be reached in next Summer. Henning
Hi Henning, that is more than expected. What value do you use for max-nodes? I used the default: 1600000. Besides that I just remember that this problem is also related to the max-areas value. If we reach the limit, it might be better to reduce max-areas so that splitter uses two passes. Gerd Date: Fri, 12 Apr 2013 09:30:15 +0200 From: osm@aighes.de To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] splitter exception with SRTM data for USA Am 12.04.2013 09:22, schrieb Gerd Petermann: Statistics for coords map: Length-1 chunks: 4.580.310, used Bytes including overhead: 47.376.048 Planet with data of yesterday: Length-1 chunks: 10.454.925, used Bytes including overhead: 107.695.080 If the rising continues linear, the limit will be reached in next Summer. Henning _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Gerd, my value for max-nodes is also 1600000. Henning Am 12.04.2013 09:36, schrieb Gerd Petermann:
Hi Henning,
that is more than expected. What value do you use for max-nodes? I used the default: 1600000.
Besides that I just remember that this problem is also related to the max-areas value. If we reach the limit, it might be better to reduce max-areas so that splitter uses two passes.
Gerd
------------------------------------------------------------------------ Date: Fri, 12 Apr 2013 09:30:15 +0200 From: osm@aighes.de To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] splitter exception with SRTM data for USA
Am 12.04.2013 09:22, schrieb Gerd Petermann:
Statistics for *coords *map: *Length-1* chunks:*4.580.310*, used Bytes including overhead: 47.376.048
Planet with data of yesterday:
Length-1 chunks: 10.454.925, used Bytes including overhead: 107.695.080
If the rising continueslinear, the limit will be reached in next Summer.
Henning
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Henning, please post a link to your splitter log. I want to run splitter with the same parms and the old planet to have better data. (e.g. --precomp-sea will also play a role) Gerd Date: Fri, 12 Apr 2013 09:43:48 +0200 From: osm@aighes.de To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] splitter exception with SRTM data for USA Hi Gerd, my value for max-nodes is also 1600000. Henning Am 12.04.2013 09:36, schrieb Gerd Petermann: Hi Henning, that is more than expected. What value do you use for max-nodes? I used the default: 1600000. Besides that I just remember that this problem is also related to the max-areas value. If we reach the limit, it might be better to reduce max-areas so that splitter uses two passes. Gerd Date: Fri, 12 Apr 2013 09:30:15 +0200 From: osm@aighes.de To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] splitter exception with SRTM data for USA Am 12.04.2013 09:22, schrieb Gerd Petermann: Statistics for coords map: Length-1 chunks: 4.580.310, used Bytes including overhead: 47.376.048 Planet with data of yesterday: Length-1 chunks: 10.454.925, used Bytes including overhead: 107.695.080 If the rising continues linear, the limit will be reached in next Summer. Henning _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Gerd, see: http://www.aighes.de/data/splitter.log If you also need areas.list: http://www.aighes.de/data/areas.list Henning Am 12.04.2013 09:50, schrieb Gerd Petermann:
Hi Henning,
please post a link to your splitter log. I want to run splitter with the same parms and the old planet to have better data. (e.g. --precomp-sea will also play a role)
Gerd
------------------------------------------------------------------------ Date: Fri, 12 Apr 2013 09:43:48 +0200 From: osm@aighes.de To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] splitter exception with SRTM data for USA
Hi Gerd, my value for max-nodes is also 1600000.
Henning
Am 12.04.2013 09:36, schrieb Gerd Petermann:
Hi Henning,
that is more than expected. What value do you use for max-nodes? I used the default: 1600000.
Besides that I just remember that this problem is also related to the max-areas value. If we reach the limit, it might be better to reduce max-areas so that splitter uses two passes.
Gerd
------------------------------------------------------------------------ Date: Fri, 12 Apr 2013 09:30:15 +0200 From: osm@aighes.de <mailto:osm@aighes.de> To: mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> Subject: Re: [mkgmap-dev] splitter exception with SRTM data for USA
Am 12.04.2013 09:22, schrieb Gerd Petermann:
Statistics for *coords *map: *Length-1* chunks:*4.580.310*, used Bytes including overhead: 47.376.048
Planet with data of yesterday:
Length-1 chunks: 10.454.925, used Bytes including overhead: 107.695.080
If the rising continueslinear, the limit will be reached in next Summer.
Henning
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk <mailto:mkgmap-dev@lists.mkgmap.org.uk> http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Henning, thanks, that explains the higher values. When you use --split-file, the max-nodes value is ignored. You create 804 areas, I create 1502. Gerd Date: Fri, 12 Apr 2013 09:56:29 +0200 From: osm@aighes.de To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] splitter exception with SRTM data for USA Hi Gerd, see: http://www.aighes.de/data/splitter.log If you also need areas.list: http://www.aighes.de/data/areas.list Henning Am 12.04.2013 09:50, schrieb Gerd Petermann: Hi Henning, please post a link to your splitter log. I want to run splitter with the same parms and the old planet to have better data. (e.g. --precomp-sea will also play a role) Gerd Date: Fri, 12 Apr 2013 09:43:48 +0200 From: osm@aighes.de To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] splitter exception with SRTM data for USA Hi Gerd, my value for max-nodes is also 1600000. Henning Am 12.04.2013 09:36, schrieb Gerd Petermann: Hi Henning, that is more than expected. What value do you use for max-nodes? I used the default: 1600000. Besides that I just remember that this problem is also related to the max-areas value. If we reach the limit, it might be better to reduce max-areas so that splitter uses two passes. Gerd Date: Fri, 12 Apr 2013 09:30:15 +0200 From: osm@aighes.de To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] splitter exception with SRTM data for USA Am 12.04.2013 09:22, schrieb Gerd Petermann: Statistics for coords map: Length-1 chunks: 4.580.310, used Bytes including overhead: 47.376.048 Planet with data of yesterday: Length-1 chunks: 10.454.925, used Bytes including overhead: 107.695.080 If the rising continues linear, the limit will be reached in next Summer. Henning _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Henning,
thanks, that explains the higher values. When you use --split-file, the max-nodes value is ignored. You create 804 areas, I create 1502.
Gerd Hi Gerd, you are covering with 1502 the hole planet, do you? I'm covering just a few parts. My areas.list was created on 22.12.2012 with a max-value of
Am 12.04.2013 10:06, schrieb Gerd Petermann: 1000000. So I started splitter now splitting the hole planet with a max-nodes of 1600000 and max-areas of 2048. Henning
Hi Henning, yes, whole planet, and I use keep-complete=true, overlap=0 and stop-after=gen-problem-list because that is enough to get the info. Gerd Henning Scholland wrote
Hi Henning,
thanks, that explains the higher values. When you use --split-file, the max-nodes value is ignored. You create 804 areas, I create 1502.
Gerd Hi Gerd, you are covering with 1502 the hole planet, do you? I'm covering just a few parts. My areas.list was created on 22.12.2012 with a max-value of
Am 12.04.2013 10:06, schrieb Gerd Petermann: 1000000.
So I started splitter now splitting the hole planet with a max-nodes of 1600000 and max-areas of 2048.
Henning
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
-- View this message in context: http://gis.19327.n5.nabble.com/splitter-exception-with-SRTM-data-for-USA-tp5... Sent from the Mkgmap Development mailing list archive at Nabble.com.
Am 12.04.2013 10:52, schrieb GerdP:
Hi Henning,
yes, whole planet, and I use keep-complete=true, overlap=0 and stop-after=gen-problem-list because that is enough to get the info.
Gerd With these parameters and splitting the complete planet I got the following values:
Length-1 chunks: 5.547.829, used Bytes including overhead: 57.051.176 Afterwards I generated a new areas.list file, containing all my (partly overlapping) maps also with max-nodes=1600000 and let splitter split them out of the planet. The result is again a value about 10.000.000. Henning
Hi Henning, Henning Scholland wrote
With these parameters and splitting the complete planet I got the following values:
Length-1 chunks: 5.547.829, used Bytes including overhead: 57.051.176
Afterwards I generated a new areas.list file, containing all my (partly overlapping) maps also with max-nodes=1600000 and let splitter split them out of the planet. The result is again a value about 10.000.000.
OK, these values make perfectly sense to me. When you split planet without split-file, the gen-problem-list pass distributes the data to > 1000 areas. That means that it is more likely that a list of nodes with ids x,x+1,x+3,x+7,x+8,...,x+63 fall into more than one different areas. If that is the case, the corresponding value is not stored in a Length-1 chunk. On the other hand, with your areas.list, splitter creates large "pseudo-areas" so that the whole planet is covered. With these large areas it is likely that a list of nodes falls into the same (pseudo-) area and thus is saved in the Length-1 chunk. By the way, with my old planet.osm.pbf and your old areas.list I get Statistics for coords map: Length-1 chunks: 8.757.219, used Bytes including overhead: 90.717.976 which means it will take a while to reach the limit, but it will not work for many years. And if you reduce your areas.list to a single small tile, you may already hit the problem. If I can't find a better solution in the current implementation I'll probably add a parm like --allow-huge-data which enables the older structure. Gerd -- View this message in context: http://gis.19327.n5.nabble.com/splitter-exception-with-SRTM-data-for-USA-tp5... Sent from the Mkgmap Development mailing list archive at Nabble.com.
Hi all, I see a few possible solutions for the problem: 1) simple but slow: use a smaller max-areas value so that splitter need one more pass. The only needed change in the code would be a test for the situation and an appropriate error message. 2) I can add an option like --allow-huge-data which tells splitter to use a different data structure that can hold much more data, but requires more memory to hold the same amount of data. 3) I can change splitter to always use this structure . I'd prefer option 3) Attached is a patch that implements the "huge-data" structure. huge_data_v1.patch <http://gis.19327.n5.nabble.com/file/n5756885/huge_data_v1.patch> The compiled binary is here: http://files.mkgmap.org.uk/download/104/splitter.jar Details: The "huge-data" structure uses arrays of longs (8 bytes) instead of ints (4 bytes) to store information. The number of these arrays depends on the distribution of (OSM) ids in the input file, not on the amount of nodes/ways etc. For most users, the effect of the change should be rather small, with current OSM data you will need ~180MB more memory in the "gen-problem-list" phase and in the "dist" phase. Worst case would be data with OSM ids that are generated with a random() function and a huge range of numbers, say between 1 and 2 ^63. So, I hope that no one codes such a generator ;-) With the "huge-data" structure, splitter uses 30 instead of 19 bits to address "chunk stores" of the same length,and there is room left to allow larger "chunk stores", so these limits should not be reached within many years. Gerd Thorsten Kukuk wrote
Hi,
I generated SRTM data for the USA with phygthmap and wanted to try to split it now for compiling with mkgmap. But while this worked fine for all other countries, in this case I get always the following exception:
MAP occupancy: 2.350.000.000, number of area dictionary entries: 3994 of 65535 Map details: bytes/overhead 401.218.630 / 166.809.090, overhead includes 19 arrays with 8 MB 2.350.000.000 nodes processed... id=2526831732 MAP occupancy: 2.360.000.000, number of area dictionary entries: 4000 of 65535 Map details: bytes/overhead 402.878.606 / 166.808.202, overhead includes 19 arrays with 8 MB 2.360.000.000 nodes processed... id=2536831732 Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 524288 at uk.me.parabola.splitter.SparseLong2ShortMapInline.putChunk(SparseLong2ShortMapInline.java:448) at uk.me.parabola.splitter.SparseLong2ShortMapInline.saveCurrentChunk(SparseLong2ShortMapInline.java:191) at uk.me.parabola.splitter.SparseLong2ShortMapInline.put(SparseLong2ShortMapInline.java:218) at uk.me.parabola.splitter.ProblemListProcessor.processNode(ProblemListProcessor.java:167) at uk.me.parabola.splitter.BinaryMapParser.parseDense(BinaryMapParser.java:113) at crosby.binary.BinaryParser.parse(Unknown Source) at crosby.binary.BinaryParser.handleBlock(Unknown Source) at crosby.binary.file.FileBlock.process(Unknown Source) at crosby.binary.file.BlockInputStream.process(Unknown Source) at uk.me.parabola.splitter.Main.processMap(Main.java:792) at uk.me.parabola.splitter.Main.genProblemLists(Main.java:582) at uk.me.parabola.splitter.Main.partitionAreasForProblemListGenerator(Main.java:613) at uk.me.parabola.splitter.Main.split(Main.java:244) at uk.me.parabola.splitter.Main.start(Main.java:157) at uk.me.parabola.splitter.Main.main(Main.java:146)
Any ideas? It's splitter r300.
Splitter version unknown compiled 2013-04-10T21:31:05+0000 boundary-tags=use-exclude-list cache= description=TK-USA-Tile geonames-file=osmmaps/scripts/cities/USA.txt keep-complete=true mapid=71510001 max-areas=1024 max-nodes=5000000 max-threads=4 (auto) mixed=false no-trim=false output=pbf output-dir=build/usa/tiles-srtm overlap=0 polygon-file= precomp-sea=data/sea/current problem-file= problem-report= resolution=13 split-file= status-freq=120 stop-after=dist write-kml=
-- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) _______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
-- View this message in context: http://gis.19327.n5.nabble.com/splitter-exception-with-SRTM-data-for-USA-tp5... Sent from the Mkgmap Development mailing list archive at Nabble.com.
Hi Gerd, I gave it a try, but it failed. The only output I got was: SparseLong2ShortMapInline: Allocating three-tier structure to save area info (HashMap->vector->chunkvector) Length-1 chunks: 93.749.999, used Bytes including overhead: 962.665.848 Length-2 chunks: 0, used Bytes including overhead: 0 Length-3 chunks: 0, used Bytes including overhead: 0 Length-4 chunks: 0, used Bytes including overhead: 0 Length-5 chunks: 0, used Bytes including overhead: 0 Length-6 chunks: 0, used Bytes including overhead: 0 Length-7 chunks: 0, used Bytes including overhead: 0 Length-8 chunks: 0, used Bytes including overhead: 0 Length-9 chunks: 0, used Bytes including overhead: 0 Length-10 chunks: 0, used Bytes including overhead: 0 Length-11 chunks: 0, used Bytes including overhead: 0 Length-12 chunks: 0, used Bytes including overhead: 0 Length-13 chunks: 0, used Bytes including overhead: 0 Length-14 chunks: 0, used Bytes including overhead: 0 Length-15 chunks: 0, used Bytes including overhead: 0 Length-16 chunks: 0, used Bytes including overhead: 0 Length-17 chunks: 0, used Bytes including overhead: 0 Length-18 chunks: 0, used Bytes including overhead: 0 Length-19 chunks: 0, used Bytes including overhead: 0 Length-20 chunks: 0, used Bytes including overhead: 0 Length-21 chunks: 0, used Bytes including overhead: 0 Length-22 chunks: 0, used Bytes including overhead: 0 Length-23 chunks: 0, used Bytes including overhead: 0 Length-24 chunks: 0, used Bytes including overhead: 0 Length-25 chunks: 0, used Bytes including overhead: 0 Length-26 chunks: 0, used Bytes including overhead: 0 Length-27 chunks: 0, used Bytes including overhead: 0 Length-28 chunks: 0, used Bytes including overhead: 0 Length-29 chunks: 0, used Bytes including overhead: 0 Length-30 chunks: 0, used Bytes including overhead: 0 Length-31 chunks: 0, used Bytes including overhead: 0 Length-32 chunks: 0, used Bytes including overhead: 0 Length-33 chunks: 0, used Bytes including overhead: 0 Length-34 chunks: 0, used Bytes including overhead: 0 Length-35 chunks: 0, used Bytes including overhead: 0 Length-36 chunks: 0, used Bytes including overhead: 0 Length-37 chunks: 0, used Bytes including overhead: 0 Length-38 chunks: 0, used Bytes including overhead: 0 Length-39 chunks: 0, used Bytes including overhead: 0 Length-40 chunks: 0, used Bytes including overhead: 0 Length-41 chunks: 0, used Bytes including overhead: 0 Length-42 chunks: 0, used Bytes including overhead: 0 Length-43 chunks: 0, used Bytes including overhead: 0 Length-44 chunks: 0, used Bytes including overhead: 0 Length-45 chunks: 0, used Bytes including overhead: 0 Length-46 chunks: 0, used Bytes including overhead: 0 Length-47 chunks: 0, used Bytes including overhead: 0 Length-48 chunks: 0, used Bytes including overhead: 0 Length-49 chunks: 0, used Bytes including overhead: 0 Length-50 chunks: 0, used Bytes including overhead: 0 Length-51 chunks: 0, used Bytes including overhead: 0 Length-52 chunks: 0, used Bytes including overhead: 0 Length-53 chunks: 0, used Bytes including overhead: 0 Length-54 chunks: 0, used Bytes including overhead: 0 Length-55 chunks: 0, used Bytes including overhead: 0 Length-56 chunks: 0, used Bytes including overhead: 0 Length-57 chunks: 0, used Bytes including overhead: 0 Length-58 chunks: 0, used Bytes including overhead: 0 Length-59 chunks: 0, used Bytes including overhead: 0 Length-60 chunks: 0, used Bytes including overhead: 0 Length-61 chunks: 0, used Bytes including overhead: 0 Length-62 chunks: 0, used Bytes including overhead: 0 Length-63 chunks: 0, used Bytes including overhead: 0 Length-64 chunks: 0, used Bytes including overhead: 0 Number of stored ids: 3.000.000.000 require ca. 0.57 bytes per pair. 93749999 chunks are used, the avg. number of values in one 64-chunk is 32. Map details: bytes/overhead 937.499.990 / 780.140.578, overhead includes 45 arrays with 16 MB RLE compresion info: compressed / uncompressed size / ratio: 93.749.999 / 2.999.999.968 / 97%
Hi Henning, stupid me! I've uploaded the wrong binary, here is the right one: http://files.mkgmap.org.uk/download/105/splitter.jar Gerd
Date: Sat, 13 Apr 2013 11:57:14 +0200 From: osm@aighes.de To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] splitter exception with SRTM data for USA
Hi Gerd, I gave it a try, but it failed. The only output I got was:
SparseLong2ShortMapInline: Allocating three-tier structure to save area info (HashMap->vector->chunkvector) Length-1 chunks: 93.749.999, used Bytes including overhead: 962.665.848 Length-2 chunks: 0, used Bytes including overhead: 0 Length-3 chunks: 0, used Bytes including overhead: 0 Length-4 chunks: 0, used Bytes including overhead: 0 Length-5 chunks: 0, used Bytes including overhead: 0 Length-6 chunks: 0, used Bytes including overhead: 0 Length-7 chunks: 0, used Bytes including overhead: 0 Length-8 chunks: 0, used Bytes including overhead: 0 Length-9 chunks: 0, used Bytes including overhead: 0 Length-10 chunks: 0, used Bytes including overhead: 0 Length-11 chunks: 0, used Bytes including overhead: 0 Length-12 chunks: 0, used Bytes including overhead: 0 Length-13 chunks: 0, used Bytes including overhead: 0 Length-14 chunks: 0, used Bytes including overhead: 0 Length-15 chunks: 0, used Bytes including overhead: 0 Length-16 chunks: 0, used Bytes including overhead: 0 Length-17 chunks: 0, used Bytes including overhead: 0 Length-18 chunks: 0, used Bytes including overhead: 0 Length-19 chunks: 0, used Bytes including overhead: 0 Length-20 chunks: 0, used Bytes including overhead: 0 Length-21 chunks: 0, used Bytes including overhead: 0 Length-22 chunks: 0, used Bytes including overhead: 0 Length-23 chunks: 0, used Bytes including overhead: 0 Length-24 chunks: 0, used Bytes including overhead: 0 Length-25 chunks: 0, used Bytes including overhead: 0 Length-26 chunks: 0, used Bytes including overhead: 0 Length-27 chunks: 0, used Bytes including overhead: 0 Length-28 chunks: 0, used Bytes including overhead: 0 Length-29 chunks: 0, used Bytes including overhead: 0 Length-30 chunks: 0, used Bytes including overhead: 0 Length-31 chunks: 0, used Bytes including overhead: 0 Length-32 chunks: 0, used Bytes including overhead: 0 Length-33 chunks: 0, used Bytes including overhead: 0 Length-34 chunks: 0, used Bytes including overhead: 0 Length-35 chunks: 0, used Bytes including overhead: 0 Length-36 chunks: 0, used Bytes including overhead: 0 Length-37 chunks: 0, used Bytes including overhead: 0 Length-38 chunks: 0, used Bytes including overhead: 0 Length-39 chunks: 0, used Bytes including overhead: 0 Length-40 chunks: 0, used Bytes including overhead: 0 Length-41 chunks: 0, used Bytes including overhead: 0 Length-42 chunks: 0, used Bytes including overhead: 0 Length-43 chunks: 0, used Bytes including overhead: 0 Length-44 chunks: 0, used Bytes including overhead: 0 Length-45 chunks: 0, used Bytes including overhead: 0 Length-46 chunks: 0, used Bytes including overhead: 0 Length-47 chunks: 0, used Bytes including overhead: 0 Length-48 chunks: 0, used Bytes including overhead: 0 Length-49 chunks: 0, used Bytes including overhead: 0 Length-50 chunks: 0, used Bytes including overhead: 0 Length-51 chunks: 0, used Bytes including overhead: 0 Length-52 chunks: 0, used Bytes including overhead: 0 Length-53 chunks: 0, used Bytes including overhead: 0 Length-54 chunks: 0, used Bytes including overhead: 0 Length-55 chunks: 0, used Bytes including overhead: 0 Length-56 chunks: 0, used Bytes including overhead: 0 Length-57 chunks: 0, used Bytes including overhead: 0 Length-58 chunks: 0, used Bytes including overhead: 0 Length-59 chunks: 0, used Bytes including overhead: 0 Length-60 chunks: 0, used Bytes including overhead: 0 Length-61 chunks: 0, used Bytes including overhead: 0 Length-62 chunks: 0, used Bytes including overhead: 0 Length-63 chunks: 0, used Bytes including overhead: 0 Length-64 chunks: 0, used Bytes including overhead: 0
Number of stored ids: 3.000.000.000 require ca. 0.57 bytes per pair. 93749999 chunks are used, the avg. number of values in one 64-chunk is 32. Map details: bytes/overhead 937.499.990 / 780.140.578, overhead includes 45 arrays with 16 MB RLE compresion info: compressed / uncompressed size / ratio: 93.749.999 / 2.999.999.968 / 97%
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Gerd, the second file works as normal. Log: http://www.aighes.de/data/splitter_new.log Henning
Hi Henning, yes, with large input file the difference is relative small. You may see problems when you run a small machine (e.g. a netbook), because then the additional 180MB are more important. Gerd Henning Scholland wrote
Hi Gerd, the second file works as normal. Log: http://www.aighes.de/data/splitter_new.log
Henning
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
-- View this message in context: http://gis.19327.n5.nabble.com/splitter-exception-with-SRTM-data-for-USA-tp5... Sent from the Mkgmap Development mailing list archive at Nabble.com.
Am 13.04.2013 15:36, schrieb GerdP:
Hi Henning,
yes, with large input file the difference is relative small. You may see problems when you run a small machine (e.g. a netbook), because then the additional 180MB are more important. Maybe you can implement a automatic switch between the two possibilities, depending on -Xmx parameter given to splitter. Maybe if the value is smaller then 1.5gb or so. But I don't know, if this is possible and useful.
Henning
Hi Henning, yes, that sounds like a good option to me, and it easy to do. Gerd Henning Scholland wrote
Am 13.04.2013 15:36, schrieb GerdP:
Hi Henning,
yes, with large input file the difference is relative small. You may see problems when you run a small machine (e.g. a netbook), because then the additional 180MB are more important. Maybe you can implement a automatic switch between the two possibilities, depending on -Xmx parameter given to splitter. Maybe if the value is smaller then 1.5gb or so. But I don't know, if this is possible and useful.
Henning
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
-- View this message in context: http://gis.19327.n5.nabble.com/splitter-exception-with-SRTM-data-for-USA-tp5... Sent from the Mkgmap Development mailing list archive at Nabble.com.
Hi all, I've committed r301 which implements what Henning suggested. The old structure is used when less than 2GB are configured. On my machine that means something like -Xmx2303M or less. Gerd Henning Scholland wrote
Maybe you can implement a automatic switch between the two possibilities, depending on -Xmx parameter given to splitter. Maybe if the value is smaller then 1.5gb or so. But I don't know, if this is possible and useful.
-- View this message in context: http://gis.19327.n5.nabble.com/splitter-exception-with-SRTM-data-for-USA-tp5... Sent from the Mkgmap Development mailing list archive at Nabble.com.
participants (5)
-
Felix Hartmann -
Gerd Petermann -
GerdP
-
Henning Scholland -
Thorsten Kukuk