splitter OutOfMemoryError
I have run several times splitter on a 15.1 GB file (actually on cache from that file) obtained from Srtm2OSM on machine A (Athlon 64 3000 Ghz, 3GB RAM) using the following command: time java -Xmx2500M -jar splitter.jar --mapid=63240101 --mixed=true --cache=/160GB/cache_srtm/peninsula/ --max-nodes=4500000 --split-file=areas.list --description="SRTM Map" --overlap=1000 Running the same command on the same data on a new machine B (Athlon II 3000 Ghz dual core, 4GB RAM) I get an OutOfMemoryError. Even increasing to -Xmx3500M I get the same error, always at the same point. Running mkgmap on spain.osm from Geofabrik with my usual script I also had to increase memory. Machine A java -version: java version "1.6.0_16" Java(TM) SE Runtime Environment (build 1.6.0_16-b01) Java HotSpot(TM) Client VM (build 14.2-b01, mixed mode, sharing) Machine B java -version: java version "is 1.6.0_17" OpenJDK Runtime Environment (IcedTea6 1.7) (6b17-1.7-1) OpenJDK Server VM (build 16.0-b13, mixed mode) terminal output: ... 100.000.000 nodes processed... 102.500.000 nodes processed... Elapsed time: 10m 0s Memory: Current 2379MB (1539MB used, 840MB free) Max 2379MB Exception in thread "main" java.lang.OutOfMemoryError: Java heap space at uk.me.parabola.splitter.IntIntMap.ensureSpace(IntIntMap.java:121) at uk.me.parabola.splitter.IntIntMap.put(IntIntMap.java:73) at uk.me.parabola.splitter.SplitIntMap.put(SplitIntMap.java:49) at uk.me.parabola.splitter.SplitProcessor.writeNode(SplitProcessor.java:202) at uk.me.parabola.splitter.SplitProcessor.endNode(SplitProcessor.java:152) at uk.me.parabola.splitter.BinaryMapLoader.processNodes(BinaryMapLoader.java:96) at uk.me.parabola.splitter.BinaryMapLoader.load(BinaryMapLoader.java:78) at uk.me.parabola.splitter.Main.processMap(Main.java:377) at uk.me.parabola.splitter.Main.writeAreas(Main.java:366) at uk.me.parabola.splitter.Main.split(Main.java:183) at uk.me.parabola.splitter.Main.start(Main.java:109) at uk.me.parabola.splitter.Main.main(Main.java:98) Command exited with non-zero status 1 585.92user 11.65system 10:04.55elapsed 98%CPU (0avgtext+0avgdata 9424896maxresident)k 4919520inputs+1954184outputs (10major+597930minor)pagefaults 0swaps Error at line 2740921, col 57 Bad file format: ../Srtm2Osm/63240101.osm.gz Error at line 1627896, col 58 Bad file format: ../Srtm2Osm/63240102.osm.gz ... Any ideas about the problem? Carlos P.S. Does it make sense to give --overlap to splitter for contour lines or is it only necessary for routing?
Hi Carlos, I don't know the answer to your problem but I note that you are using a different VM on machine B. I would have thought it was worth trying the same VM as used on machine A. Cheers, Mark
Mark Burton escribió:
Hi Carlos,
I don't know the answer to your problem but I note that you are using a different VM on machine B. I would have thought it was worth trying the same VM as used on machine A. Using sun java as in machine A fixed the problem for splitter, although mkgmap still needs a slightly higher amount of memory. May it be due to --max-jobs? Cheers, Carlos
Carlos,
Using sun java as in machine A fixed the problem for splitter, although mkgmap still needs a slightly higher amount of memory. May it be due to --max-jobs?
Sure, if you have more cores available than before, --max-jobs will process that number of maps in parallel so it will take more memory. Cheers, Mark
The memory required by the splitter is roughly proportional to max-nodes multiplied by the number of areas being processed per pass. I see you're setting --max-nodes=4500000, which is rather high. To reduce the memory required you can set --max-areas to something lower than the default of 255. This will reduce the amount of memory required, but at the cost of additional passes over the data. Also, does machine B have a 64 bit VM by any chance? This will also increase the memory requirements somewhat, especially if compressed OOPS is disabled/unavailable. Chris
Chris Miller escribió:
The memory required by the splitter is roughly proportional to max-nodes multiplied by the number of areas being processed per pass. I see you're setting --max-nodes=4500000, which is rather high. To reduce the memory required you can set --max-areas to something lower than the default of 255. This will reduce the amount of memory required, but at the cost of additional passes over the data.
As I told before, using sun java I don't have memory problems, so I can leave it as is. The number of areas processed is around 50, so in case I need to reduce memory requirements I suppose I would have to set --max-areas<50.
Also, does machine B have a 64 bit VM by any chance? No, it doesn't. This will also increase the memory requirements somewhat, especially if compressed OOPS is disabled/unavailable.
What about the use of --overlap for contour lines? Carlos
As I told before, using sun java I don't have memory problems, so I can leave it as is. The number of areas processed is around 50, so in case I need to reduce memory requirements I suppose I would have to set --max-areas<50.
50 areas is pretty low so even with --max-nodes=4500000 it should be OK to split in a single pass with a 32bit VM. Sounds like that is indeed the case with Sun's VM for you. I have an idea why you're running into problems on the other machine though. Something I didn't notice the first time I read your email was this log message that was written not long before the memory ran out: Elapsed time: 10m 0s Memory: Current 2379MB (1539MB used, 840MB free) Max 2379MB That indicates to me that even though you're trying to allocate a 3.5GB heap, OpenJDK isn't able to do so. This is most likely because you don't have 3.5GB of *contiguous* memory available. As far as I'm aware, most JVMs still require unfragmented memory for their heap. This means the maximum heap size you can allocate is not limited by the amount of free memory you have but rather by the biggest single chunk of continuous free memory.
What about the use of --overlap for contour lines? Carlos
I'd recommend you still use some overlap. Even though you're not worried about routing issues, without any overlap you'll still end up with breaks in your contour lines near the edges of tiles which probably isn't what you want.
participants (3)
-
Carlos Dávila -
Chris Miller -
Mark Burton