Hi, On Thu, Apr 11, Gerd Petermann wrote:
Hi Thorsten,
I fear my idea regarding the numering of node ids will not help. The problem is the amount of nodes and the fact that the generated data probably is distributed in a way that the nodes with id x,x+1,x+2,..,x+n are very close together. This makes it very likely that splitter reaches a limit of 2.147.483.648. With normal OSM data, this is very unlikely.
I fear I have to revert the latest optimizations in SparseLong2ShortMapInline to remove that limit.
It looks like splitter will work if I lower the limit of max. nodes in a tile. I don't know which improvements your optimizations are giving, but I don't think that we need to care for this very special case, especially as it looks like I found a workaround. But with this big amount of data, testing will need some more time. Thorsten
Gerd
Date: Thu, 11 Apr 2013 16:54:13 +0200 From: kukuk@suse.de To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] splitter exception with SRTM data for USA
Hi Gerd,
On Thu, Apr 11, GerdP wrote:
Hi Thorsten,
ok, so I assume that you can patch pyghtmap to generate the ids as I requested.
Yes, I will try that over the weekend, generating such big data needs some time.
Thorsten
In the mean time I will write a small generator which creates OSM data with a large number of nodes. Maybe it is a bug in splitter, not a limit.
Gerd
Thorsten Kukuk wrote
Hi,
On Thu, Apr 11, Felix Hartmann wrote:
are you using the newest version of pyghtmap, newest splitter, and no older than 3-4 weeks mkgmap?
I had a similar problem with Asia. mkgmap splitter wouldn't split the pyghtmap extract - no matter what I tried (splitter didn't split but simply output one huge file instead). (about 6 month ago) - 1 month ago I redid everything using newest versions, and it worked...
Since I wrote a lot of the patches for the last phyghtmap versions, I use of course the newest one.
And it is only with the US. Canada, but that's a little bit smaller, doesn't create any problems.
Thorsten
On 11.04.2013 16:33, Thorsten Kukuk wrote:
Hi Gerd,
On Thu, Apr 11, Gerd Petermann wrote:
> Hi Thorsten, > > another question: The wiki of phygthmap says that you don't need splitting: > http://wiki.openstreetmap.org/wiki/Phyghtmap > > Why do you do it? Because I create one big data file with phyghtmap and then extract the parts I need. That's much easier and simpler then to let phyghtmap create a big amount of files and try to find and cut the ones you need.
Thorsten
> Gerd > >> Date: Thu, 11 Apr 2013 14:06:30 +0200 >> From:
kukuk@
>> To:
mkgmap-dev@.org
>> Subject: [mkgmap-dev] splitter exception with SRTM data for USA >> >> >> Hi, >> >> I generated SRTM data for the USA with phygthmap and wanted to try >> to split it now for compiling with mkgmap. But while this worked fine >> for all other countries, in this case I get always the following >> exception: >> >> MAP occupancy: 2.350.000.000, number of area dictionary entries: 3994 of 65535 >> Map details: bytes/overhead 401.218.630 / 166.809.090, overhead includes 19 arrays with 8 MB >> 2.350.000.000 nodes processed... id=2526831732 >> MAP occupancy: 2.360.000.000, number of area dictionary entries: 4000 of 65535 >> Map details: bytes/overhead 402.878.606 / 166.808.202, overhead includes 19 arrays with 8 MB >> 2.360.000.000 nodes processed... id=2536831732 >> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 524288 >> at uk.me.parabola.splitter.SparseLong2ShortMapInline.putChunk(SparseLong2ShortMapInline.java:448) >> at uk.me.parabola.splitter.SparseLong2ShortMapInline.saveCurrentChunk(SparseLong2ShortMapInline.java:191) >> at uk.me.parabola.splitter.SparseLong2ShortMapInline.put(SparseLong2ShortMapInline.java:218) >> at uk.me.parabola.splitter.ProblemListProcessor.processNode(ProblemListProcessor.java:167) >> at uk.me.parabola.splitter.BinaryMapParser.parseDense(BinaryMapParser.java:113) >> at crosby.binary.BinaryParser.parse(Unknown Source) >> at crosby.binary.BinaryParser.handleBlock(Unknown Source) >> at crosby.binary.file.FileBlock.process(Unknown Source) >> at crosby.binary.file.BlockInputStream.process(Unknown Source) >> at uk.me.parabola.splitter.Main.processMap(Main.java:792) >> at uk.me.parabola.splitter.Main.genProblemLists(Main.java:582) >> at uk.me.parabola.splitter.Main.partitionAreasForProblemListGenerator(Main.java:613) >> at uk.me.parabola.splitter.Main.split(Main.java:244) >> at uk.me.parabola.splitter.Main.start(Main.java:157) >> at uk.me.parabola.splitter.Main.main(Main.java:146) >> >> Any ideas? It's splitter r300. >> >> Splitter version unknown compiled 2013-04-10T21:31:05+0000 >> boundary-tags=use-exclude-list >> cache= >> description=TK-USA-Tile >> geonames-file=osmmaps/scripts/cities/USA.txt >> keep-complete=true >> mapid=71510001 >> max-areas=1024 >> max-nodes=5000000 >> max-threads=4 (auto) >> mixed=false >> no-trim=false >> output=pbf >> output-dir=build/usa/tiles-srtm >> overlap=0 >> polygon-file= >> precomp-sea=data/sea/current >> problem-file= >> problem-report= >> resolution=13 >> split-file= >> status-freq=120 >> stop-after=dist >> write-kml= >> >> >> -- >> Thorsten Kukuk, Project Manager/Release Manager SLES >> SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg >> GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) >> _______________________________________________ >> mkgmap-dev mailing list >>
mkgmap-dev@.org
>> http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > _______________________________________________ > mkgmap-dev mailing list >
mkgmap-dev@.org
-- keep on biking and discovering new trails
Felix openmtbmap.org & www.velomap.org
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
-- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) _______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
-- View this message in context: http://gis.19327.n5.nabble.com/splitter-exception-with-SRTM-data-for-USA-tp5... Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev -- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg)
mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg)