Hi all, I see a few possible solutions for the problem: 1) simple but slow: use a smaller max-areas value so that splitter need one more pass. The only needed change in the code would be a test for the situation and an appropriate error message. 2) I can add an option like --allow-huge-data which tells splitter to use a different data structure that can hold much more data, but requires more memory to hold the same amount of data. 3) I can change splitter to always use this structure . I'd prefer option 3) Attached is a patch that implements the "huge-data" structure. huge_data_v1.patch <http://gis.19327.n5.nabble.com/file/n5756885/huge_data_v1.patch> The compiled binary is here: http://files.mkgmap.org.uk/download/104/splitter.jar Details: The "huge-data" structure uses arrays of longs (8 bytes) instead of ints (4 bytes) to store information. The number of these arrays depends on the distribution of (OSM) ids in the input file, not on the amount of nodes/ways etc. For most users, the effect of the change should be rather small, with current OSM data you will need ~180MB more memory in the "gen-problem-list" phase and in the "dist" phase. Worst case would be data with OSM ids that are generated with a random() function and a huge range of numbers, say between 1 and 2 ^63. So, I hope that no one codes such a generator ;-) With the "huge-data" structure, splitter uses 30 instead of 19 bits to address "chunk stores" of the same length,and there is room left to allow larger "chunk stores", so these limits should not be reached within many years. Gerd Thorsten Kukuk wrote
Hi,
I generated SRTM data for the USA with phygthmap and wanted to try to split it now for compiling with mkgmap. But while this worked fine for all other countries, in this case I get always the following exception:
MAP occupancy: 2.350.000.000, number of area dictionary entries: 3994 of 65535 Map details: bytes/overhead 401.218.630 / 166.809.090, overhead includes 19 arrays with 8 MB 2.350.000.000 nodes processed... id=2526831732 MAP occupancy: 2.360.000.000, number of area dictionary entries: 4000 of 65535 Map details: bytes/overhead 402.878.606 / 166.808.202, overhead includes 19 arrays with 8 MB 2.360.000.000 nodes processed... id=2536831732 Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 524288 at uk.me.parabola.splitter.SparseLong2ShortMapInline.putChunk(SparseLong2ShortMapInline.java:448) at uk.me.parabola.splitter.SparseLong2ShortMapInline.saveCurrentChunk(SparseLong2ShortMapInline.java:191) at uk.me.parabola.splitter.SparseLong2ShortMapInline.put(SparseLong2ShortMapInline.java:218) at uk.me.parabola.splitter.ProblemListProcessor.processNode(ProblemListProcessor.java:167) at uk.me.parabola.splitter.BinaryMapParser.parseDense(BinaryMapParser.java:113) at crosby.binary.BinaryParser.parse(Unknown Source) at crosby.binary.BinaryParser.handleBlock(Unknown Source) at crosby.binary.file.FileBlock.process(Unknown Source) at crosby.binary.file.BlockInputStream.process(Unknown Source) at uk.me.parabola.splitter.Main.processMap(Main.java:792) at uk.me.parabola.splitter.Main.genProblemLists(Main.java:582) at uk.me.parabola.splitter.Main.partitionAreasForProblemListGenerator(Main.java:613) at uk.me.parabola.splitter.Main.split(Main.java:244) at uk.me.parabola.splitter.Main.start(Main.java:157) at uk.me.parabola.splitter.Main.main(Main.java:146)
Any ideas? It's splitter r300.
Splitter version unknown compiled 2013-04-10T21:31:05+0000 boundary-tags=use-exclude-list cache= description=TK-USA-Tile geonames-file=osmmaps/scripts/cities/USA.txt keep-complete=true mapid=71510001 max-areas=1024 max-nodes=5000000 max-threads=4 (auto) mixed=false no-trim=false output=pbf output-dir=build/usa/tiles-srtm overlap=0 polygon-file= precomp-sea=data/sea/current problem-file= problem-report= resolution=13 split-file= status-freq=120 stop-after=dist write-kml=
-- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) _______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
-- View this message in context: http://gis.19327.n5.nabble.com/splitter-exception-with-SRTM-data-for-USA-tp5... Sent from the Mkgmap Development mailing list archive at Nabble.com.