Splitter output files are nearly empty
Recently I've upgraded upgraded all my Garmin map making tools to the latest versions: Osmosis 0.39 Splitter r171 Mkgmap r1926 Since then I'm having a problem while splitting the planet file: all output files are empty (~160 bytes). In de commandline output I notice two things for which I seek input: - What does "***** Full GC *****" mean? - Is input from stdin still functional/supported for splitter? Below is the commandline and output. Commandline: bunzip2 -d -c -k /home/lambertus/planet.openstreetmap.org/world.osm.bz2 | ~/garmin/utils/osmosis/bin/osmosis --rx - --tf reject-ways building=* --tf reject-nodes type=communication --tf reject-ways admin_level=8 --tf reject-ways admin_level=9 --tf reject-ways admin_level=10 --wx file='-' | java -Xmx7500m -ea -jar ~/garmin/utils/splitter/splitter.jar --no-trim --cache=cache --mapid=63240001 --max-nodes=1400000 --write-kml=world.kml --geonames-file=/home/lambertus/garmin/utils/cities15000.zip /dev/stdin Commandline output (where [...] are multiple similar lines removed for readability): cache=cache description= geonames-file=/home/lambertus/garmin/utils/cities15000.zip legacy-mode=false mapid=63240001 max-areas=255 max-nodes=1400000 max-threads=4 (auto) mixed=false no-trim=true output-dir= overlap=2000 resolution=13 split-file= status-freq=120 write-kml=world.kml Elapsed time: 0s Memory: Current 119MB (2MB used, 117MB free) Max 6666MB Time started: Mon May 02 23:00:41 CEST 2011 Map is being split for resolution 13: - area boundaries are aligned to 0x800 map units - areas are multiples of 0x1000 map units wide and high Processing /dev/stdin May 2, 2011 11:00:42 PM org.openstreetmap.osmosis.core.Osmosis run INFO: Osmosis Version 0.39 May 2, 2011 11:00:43 PM org.openstreetmap.osmosis.core.Osmosis run INFO: Preparing pipeline. May 2, 2011 11:00:43 PM org.openstreetmap.osmosis.core.Osmosis run INFO: Launching pipeline execution. May 2, 2011 11:00:43 PM org.openstreetmap.osmosis.core.Osmosis run INFO: Pipeline executing, waiting for completion. Elapsed time: 2m 0s Memory: Current 119MB (4MB used, 115MB free) Max 6666MB A <bounds/> tag was found. Area covered is (-90.0,-180.0) to (90.0,180.0) 2,500,000 nodes processed... [...] 117,500,000 nodes processed... ***** Full GC ***** Elapsed time: 18m 0s Memory: Current 135MB (106MB used, 29MB free) Max 6666MB [...] 267,500,000 nodes processed... ***** Full GC ***** Elapsed time: 38m 0s Memory: Current 152MB (121MB used, 31MB free) Max 6666MB [...] 412,500,000 nodes processed... ***** Full GC ***** Elapsed time: 58m 0s Memory: Current 154MB (122MB used, 32MB free) Max 6666MB [...] 567,500,000 nodes processed... ***** Full GC ***** Elapsed time: 1h 18m 0s Memory: Current 152MB (122MB used, 30MB free) Max 6666MB [...] 717,500,000 nodes processed... ***** Full GC ***** Elapsed time: 1h 38m 0s Memory: Current 156MB (122MB used, 34MB free) Max 6666MB [...] 875,000,000 nodes processed... ***** Full GC ***** Elapsed time: 1h 58m 0s Memory: Current 151MB (121MB used, 30MB free) Max 6666MB [...] 1,060,000,000 nodes processed... Elapsed time: 2h 22m 0s Memory: Current 152MB (128MB used, 24MB free) Max 6666MB 1,062,500,000 nodes processed... in 1 file Time: Tue May 03 01:22:59 CEST 2011 Exact map coverage is (-90.0,-180.0) to (90.0,180.0) Trimmed and rounded map coverage is (-84.990234375,-180.0) to (85.078125,180.0) Splitting nodes into areas containing a maximum of 1,400,000 nodes each... Area (-84.990234375,-180.0) to (16.083984375,-89.47265625) contains 748,264 nodes. DONE! [...] Area (50.009765625,132.1875) to (85.078125,180.0) contains 858,788 nodes. DONE! 1158 areas: Area 63240001 covers (0x201000,0xffbed000) to (0x20d000,0xffbff000) [...] Area 63241158 covers (0xffc39000,0xff800000) to (0xb7000,0xffc06000) GT-Guatemala City Writing KML file to ./world.kml Writing out split osm files Tue May 03 01:23:04 CEST 2011 Processing 1158 areas in 5 passes, 232 areas at a time Starting pass 1 of 5, processing 232 areas (63240001 to 63240232) Making SparseMultiMap Making SparseMultiMap Processing /dev/stdin org.xmlpull.v1.XmlPullParserException: only whitespace content allowed before start tag and not a (position: START_DOCUMENT seen a... @1:1) at org.xmlpull.mxp1.MXParser.parseProlog(MXParser.java:1519) at org.xmlpull.mxp1.MXParser.nextImpl(MXParser.java:1395) at org.xmlpull.mxp1.MXParser.next(MXParser.java:1093) at uk.me.parabola.splitter.AbstractXppParser.parse(AbstractXppParser.java:62) at uk.me.parabola.splitter.Main.processMap(Main.java:399) at uk.me.parabola.splitter.Main.writeAreas(Main.java:355) at uk.me.parabola.splitter.Main.split(Main.java:188) at uk.me.parabola.splitter.Main.start(Main.java:116) at uk.me.parabola.splitter.Main.main(Main.java:105) coords occupancy MAP occupancy: 0 ways occupancy MAP occupancy: 0 Thread worker-1 has finished Thread worker-2 has finished Thread worker-0 has finished Starting pass 2 of 5, processing 232 areas (63240233 to 63240464) Making SparseMultiMap Making SparseMultiMap Processing /dev/stdin org.xmlpull.v1.XmlPullParserException: only whitespace content allowed before start tag and not m (position: START_DOCUMENT seen m... @1:1) at org.xmlpull.mxp1.MXParser.parseProlog(MXParser.java:1519) at org.xmlpull.mxp1.MXParser.nextImpl(MXParser.java:1395) at org.xmlpull.mxp1.MXParser.next(MXParser.java:1093) at uk.me.parabola.splitter.AbstractXppParser.parse(AbstractXppParser.java:62) at uk.me.parabola.splitter.Main.processMap(Main.java:399) at uk.me.parabola.splitter.Main.writeAreas(Main.java:355) at uk.me.parabola.splitter.Main.split(Main.java:188) at uk.me.parabola.splitter.Main.start(Main.java:116) at uk.me.parabola.splitter.Main.main(Main.java:105) coords occupancy MAP occupancy: 0 ways occupancy MAP occupancy: 0 Thread worker-0 has finished Thread worker-1 has finished Thread worker-2 has finished Starting pass 3 of 5, processing 232 areas (63240465 to 63240696) Making SparseMultiMap Making SparseMultiMap Processing /dev/stdin org.xmlpull.v1.XmlPullParserException: start tag not allowed in epilog but got t (position: END_TAG seen ...<tag k="abutters" v="residential"/>\n <t... @2:7) at org.xmlpull.mxp1.MXParser.parseEpilog(MXParser.java:1588) at org.xmlpull.mxp1.MXParser.nextImpl(MXParser.java:1393) at org.xmlpull.mxp1.MXParser.next(MXParser.java:1093) at uk.me.parabola.splitter.AbstractXppParser.parse(AbstractXppParser.java:62) at uk.me.parabola.splitter.Main.processMap(Main.java:399) at uk.me.parabola.splitter.Main.writeAreas(Main.java:355) at uk.me.parabola.splitter.Main.split(Main.java:188) at uk.me.parabola.splitter.Main.start(Main.java:116) at uk.me.parabola.splitter.Main.main(Main.java:105) coords occupancy MAP occupancy: 0 ways occupancy MAP occupancy: 0 Thread worker-1 has finished Thread worker-2 has finished Thread worker-0 has finished Starting pass 4 of 5, processing 232 areas (63240697 to 63240928) Making SparseMultiMap Making SparseMultiMap Processing /dev/stdin org.xmlpull.v1.XmlPullParserException: start tag not allowed in epilog but got t (position: END_TAG seen ...<tag k="maintenance" v="gritting"/>\n <t... @2:7) at org.xmlpull.mxp1.MXParser.parseEpilog(MXParser.java:1588) at org.xmlpull.mxp1.MXParser.nextImpl(MXParser.java:1393) at org.xmlpull.mxp1.MXParser.next(MXParser.java:1093) at uk.me.parabola.splitter.AbstractXppParser.parse(AbstractXppParser.java:62) at uk.me.parabola.splitter.Main.processMap(Main.java:399) at uk.me.parabola.splitter.Main.writeAreas(Main.java:355) at uk.me.parabola.splitter.Main.split(Main.java:188) at uk.me.parabola.splitter.Main.start(Main.java:116) at uk.me.parabola.splitter.Main.main(Main.java:105) coords occupancy MAP occupancy: 0 ways occupancy MAP occupancy: 0 Thread worker-1 has finished Thread worker-0 has finished Thread worker-2 has finished Starting pass 5 of 5, processing 230 areas (63240929 to 63241158) Making SparseMultiMap Making SparseMultiMap Processing /dev/stdin org.xmlpull.v1.XmlPullParserException: only whitespace content allowed before start tag and not g (position: START_DOCUMENT seen g... @1:1) at org.xmlpull.mxp1.MXParser.parseProlog(MXParser.java:1519) at org.xmlpull.mxp1.MXParser.nextImpl(MXParser.java:1395) at org.xmlpull.mxp1.MXParser.next(MXParser.java:1093) at uk.me.parabola.splitter.AbstractXppParser.parse(AbstractXppParser.java:62) at uk.me.parabola.splitter.Main.processMap(Main.java:399) at uk.me.parabola.splitter.Main.writeAreas(Main.java:355) at uk.me.parabola.splitter.Main.split(Main.java:188) at uk.me.parabola.splitter.Main.start(Main.java:116) at uk.me.parabola.splitter.Main.main(Main.java:105) coords occupancy MAP occupancy: 0 ways occupancy MAP occupancy: 0 Thread worker-0 has finished Thread worker-1 has finished Thread worker-2 has finished Time finished: Tue May 03 01:23:04 CEST 2011 Total time taken: 8543s May 3, 2011 2:11:43 AM org.openstreetmap.osmosis.core.Osmosis run INFO: Pipeline complete. May 3, 2011 2:11:43 AM org.openstreetmap.osmosis.core.Osmosis run INFO: Total execution time: 11460683 milliseconds. lambertus@server:~/garmin/world$ ls -lh build total 5.3M -rw-r--r-- 1 lambertus lambertus 159 2011-05-03 01:23 63240001.osm.gz [...]
On Tue, May 03, 2011 at 05:48:55PM +0200, Lambertus wrote:
Recently I've upgraded upgraded all my Garmin map making tools to the latest versions: Osmosis 0.39 Splitter r171 Mkgmap r1926
Since then I'm having a problem while splitting the planet file: all output files are empty (~160 bytes).
It looks like a splitter problem.
In de commandline output I notice two things for which I seek input: - What does "***** Full GC *****" mean?
I assume that GC means garbage collection. Which JVM are you using? An educated guess is that the memory runs out and a "full garbage collection" cycle is started as a last resort.
- Is input from stdin still functional/supported for splitter?
I am not sure. I am using pbf output.
Below is the commandline and output.
Commandline: bunzip2 -d -c -k /home/lambertus/planet.openstreetmap.org/world.osm.bz2 | ~/garmin/utils/osmosis/bin/osmosis --rx - --tf reject-ways building=* --tf reject-nodes type=communication --tf reject-ways admin_level=8 --tf reject-ways admin_level=9 --tf reject-ways admin_level=10 --wx file='-'
Why not use osmosis --rb world.osm.pbf ... --wb filtered-world.osm.pbf and ... splitter.jar filtered-world.osm.pbf
| java -Xmx7500m -ea -jar ~/garmin/utils/splitter/splitter.jar --no-trim --cache=cache --mapid=63240001 --max-nodes=1400000 --write-kml=world.kml --geonames-file=/home/lambertus/garmin/utils/cities15000.zip /dev/stdin
I am not sure if the --cache option still works. It was sort of superceded or made unnecessary by the Protobuf input format support. Best regards, Marko
On 03-05-11 20:01, Marko Mäkelä wrote:
On Tue, May 03, 2011 at 05:48:55PM +0200, Lambertus wrote: I assume that GC means garbage collection. Which JVM are you using? An educated guess is that the memory runs out and a "full garbage collection" cycle is started as a last resort.
Ok, "Garbage Collection" sounds reasonable, but I gave Java 7.5 GB heap space while it used only a few hundred megabytes. I've got no idea why it would run out of memory. java version "1.6.0_24" Java(TM) SE Runtime Environment (build 1.6.0_24-b07) Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)
- Is input from stdin still functional/supported for splitter? I am not sure. I am using pbf output.
Yeah, I'm slow in moving to pbf, but is OSM XML really not functional/supported anymore? Splitter seems to accept the XML in the first phase without complaints. It's only in the step where it starts to write-out the tiles that is starts to complain. It would be better if Splitter rejects the input if it can't handle it properly later on?
Why not use
osmosis --rb world.osm.pbf ... --wb filtered-world.osm.pbf
and
... splitter.jar filtered-world.osm.pbf
Well I wanted to run multiple processes at the same time to utilize the quadcore CPU better and reduce processing time (but, then I shouldn't have used bz2 XML in the first place). Ok, I'll try this next.
I am not sure if the --cache option still works. It was sort of superceded or made unnecessary by the Protobuf input format support.
Oh, if it's not working anymore then I assume it will ignore the option. Or is this too simple thinking? Thanks for the response Marko. I'll start a new run using Osmosis pbf output.
On Tue, May 03, Lambertus wrote:
On 03-05-11 20:01, Marko Mäkelä wrote:
On Tue, May 03, 2011 at 05:48:55PM +0200, Lambertus wrote: I assume that GC means garbage collection. Which JVM are you using? An educated guess is that the memory runs out and a "full garbage collection" cycle is started as a last resort.
Ok, "Garbage Collection" sounds reasonable, but I gave Java 7.5 GB heap space while it used only a few hundred megabytes. I've got no idea why it would run out of memory.
Do you really have so much free memory available? If you tell Java it should use 7.5GB heap space, it will use that, even if not available, and fail. I run Java with 4GB heap space (machine had more), but since other processes allocated memory, too, it did fail sometimes with similar symptoms. After I reduced the memory to 2.5 GB heap space, it runs fine. -- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg)
On 03-05-11 23:25, Thorsten Kukuk wrote:
On Tue, May 03, Lambertus wrote:
Ok, "Garbage Collection" sounds reasonable, but I gave Java 7.5 GB heap space while it used only a few hundred megabytes. I've got no idea why it would run out of memory. Do you really have so much free memory available? If you tell Java it should use 7.5GB heap space, it will use that, even if not available, and fail. I run Java with 4GB heap space (machine had more), but since other processes allocated memory, too, it did fail sometimes with similar symptoms. After I reduced the memory to 2.5 GB heap space, it runs fine.
The machine's got 8 GB ram and 8 GB swap. There were other processes running as well, like bunzip and Osmosis but those don't use much ram. Even so, the logs say it's using only 250 MB ram, that should not cause problems right? If the logs said it was using more then 6GB then, ok, I could accept such an explanation, but 250 MB....
On Tue, May 03, 2011 at 11:12:25PM +0200, Lambertus wrote:
Well I wanted to run multiple processes at the same time to utilize the quadcore CPU better and reduce processing time
The processing time probably won't be reduced if the machine starts swapping. Like Thorsten pointed out, it is good to leave some "breathing room" for the computer.
I am not sure if the --cache option still works. It was sort of superceded or made unnecessary by the Protobuf input format support.
Oh, if it's not working anymore then I assume it will ignore the option. Or is this too simple thinking?
I do not know. I never used the --cache option myself. One more possibility (a wild guess) is that the format of the cache directory changed and you had some old-format stuff there that was being misinterpreted. Marko
On 2011-05-04 07:44, Marko Mäkelä wrote:
The processing time probably won't be reduced if the machine starts swapping. Like Thorsten pointed out, it is good to leave some "breathing room" for the computer.
Sure, but the machine has 8GB ram so it won't have to swap while doing multiple things at the same time and swapping did not take place (according to Munin). I'll run the script again with 5 or 6 GB heap, but I'm not convinced it will change the output or processing time significantly.
I am not sure if the --cache option still works. It was sort of superceded or made unnecessary by the Protobuf input format support.
Oh, if it's not working anymore then I assume it will ignore the option. Or is this too simple thinking?
I do not know. I never used the --cache option myself. One more possibility (a wild guess) is that the format of the cache directory changed and you had some old-format stuff there that was being misinterpreted.
The script does a full cleanup before starting the processing. There is no cache directory lingering around. But I try to remember if I saw a cache directory while processing and I don't think I did, so maybe the option is already ignored. Anyway, I've generated a pbf planet file from the bz2 XML last night, so can test if splitting pbf will be successful this evening.
Hi, I notice there is an extra - after /bin/osmosis --rx is that an error? Markus_g -----Original Message----- From: mkgmap-dev-bounces@lists.mkgmap.org.uk [mailto:mkgmap-dev-bounces@lists.mkgmap.org.uk] On Behalf Of Lambertus Sent: Wednesday, 4 May 2011 1:19 AM To: Development list for mkgmap Subject: [mkgmap-dev] Splitter output files are nearly empty Recently I've upgraded upgraded all my Garmin map making tools to the latest versions: Osmosis 0.39 Splitter r171 Mkgmap r1926 Since then I'm having a problem while splitting the planet file: all output files are empty (~160 bytes). In de commandline output I notice two things for which I seek input: - What does "***** Full GC *****" mean? - Is input from stdin still functional/supported for splitter? Below is the commandline and output. Commandline: bunzip2 -d -c -k /home/lambertus/planet.openstreetmap.org/world.osm.bz2 | ~/garmin/utils/osmosis/bin/osmosis --rx - --tf reject-ways building=* --tf reject-nodes type=communication --tf reject-ways admin_level=8 --tf reject-ways admin_level=9 --tf reject-ways admin_level=10 --wx file='-' | java -Xmx7500m -ea -jar ~/garmin/utils/splitter/splitter.jar --no-trim --cache=cache --mapid=63240001 --max-nodes=1400000 --write-kml=world.kml --geonames-file=/home/lambertus/garmin/utils/cities15000.zip /dev/stdin Commandline output (where [...] are multiple similar lines removed for readability): cache=cache description= geonames-file=/home/lambertus/garmin/utils/cities15000.zip legacy-mode=false mapid=63240001 max-areas=255 max-nodes=1400000 max-threads=4 (auto) mixed=false no-trim=true output-dir= overlap=2000 resolution=13 split-file= status-freq=120 write-kml=world.kml Elapsed time: 0s Memory: Current 119MB (2MB used, 117MB free) Max 6666MB Time started: Mon May 02 23:00:41 CEST 2011 Map is being split for resolution 13: - area boundaries are aligned to 0x800 map units - areas are multiples of 0x1000 map units wide and high Processing /dev/stdin May 2, 2011 11:00:42 PM org.openstreetmap.osmosis.core.Osmosis run INFO: Osmosis Version 0.39 May 2, 2011 11:00:43 PM org.openstreetmap.osmosis.core.Osmosis run INFO: Preparing pipeline. May 2, 2011 11:00:43 PM org.openstreetmap.osmosis.core.Osmosis run INFO: Launching pipeline execution. May 2, 2011 11:00:43 PM org.openstreetmap.osmosis.core.Osmosis run INFO: Pipeline executing, waiting for completion. Elapsed time: 2m 0s Memory: Current 119MB (4MB used, 115MB free) Max 6666MB A <bounds/> tag was found. Area covered is (-90.0,-180.0) to (90.0,180.0) 2,500,000 nodes processed... [...] 117,500,000 nodes processed... ***** Full GC ***** Elapsed time: 18m 0s Memory: Current 135MB (106MB used, 29MB free) Max 6666MB [...] 267,500,000 nodes processed... ***** Full GC ***** Elapsed time: 38m 0s Memory: Current 152MB (121MB used, 31MB free) Max 6666MB [...] 412,500,000 nodes processed... ***** Full GC ***** Elapsed time: 58m 0s Memory: Current 154MB (122MB used, 32MB free) Max 6666MB [...] 567,500,000 nodes processed... ***** Full GC ***** Elapsed time: 1h 18m 0s Memory: Current 152MB (122MB used, 30MB free) Max 6666MB [...] 717,500,000 nodes processed... ***** Full GC ***** Elapsed time: 1h 38m 0s Memory: Current 156MB (122MB used, 34MB free) Max 6666MB [...] 875,000,000 nodes processed... ***** Full GC ***** Elapsed time: 1h 58m 0s Memory: Current 151MB (121MB used, 30MB free) Max 6666MB [...] 1,060,000,000 nodes processed... Elapsed time: 2h 22m 0s Memory: Current 152MB (128MB used, 24MB free) Max 6666MB 1,062,500,000 nodes processed... in 1 file Time: Tue May 03 01:22:59 CEST 2011 Exact map coverage is (-90.0,-180.0) to (90.0,180.0) Trimmed and rounded map coverage is (-84.990234375,-180.0) to (85.078125,180.0) Splitting nodes into areas containing a maximum of 1,400,000 nodes each... Area (-84.990234375,-180.0) to (16.083984375,-89.47265625) contains 748,264 nodes. DONE! [...] Area (50.009765625,132.1875) to (85.078125,180.0) contains 858,788 nodes. DONE! 1158 areas: Area 63240001 covers (0x201000,0xffbed000) to (0x20d000,0xffbff000) [...] Area 63241158 covers (0xffc39000,0xff800000) to (0xb7000,0xffc06000) GT-Guatemala City Writing KML file to ./world.kml Writing out split osm files Tue May 03 01:23:04 CEST 2011 Processing 1158 areas in 5 passes, 232 areas at a time Starting pass 1 of 5, processing 232 areas (63240001 to 63240232) Making SparseMultiMap Making SparseMultiMap Processing /dev/stdin org.xmlpull.v1.XmlPullParserException: only whitespace content allowed before start tag and not a (position: START_DOCUMENT seen a... @1:1) at org.xmlpull.mxp1.MXParser.parseProlog(MXParser.java:1519) at org.xmlpull.mxp1.MXParser.nextImpl(MXParser.java:1395) at org.xmlpull.mxp1.MXParser.next(MXParser.java:1093) at uk.me.parabola.splitter.AbstractXppParser.parse(AbstractXppParser.java:62) at uk.me.parabola.splitter.Main.processMap(Main.java:399) at uk.me.parabola.splitter.Main.writeAreas(Main.java:355) at uk.me.parabola.splitter.Main.split(Main.java:188) at uk.me.parabola.splitter.Main.start(Main.java:116) at uk.me.parabola.splitter.Main.main(Main.java:105) coords occupancy MAP occupancy: 0 ways occupancy MAP occupancy: 0 Thread worker-1 has finished Thread worker-2 has finished Thread worker-0 has finished Starting pass 2 of 5, processing 232 areas (63240233 to 63240464) Making SparseMultiMap Making SparseMultiMap Processing /dev/stdin org.xmlpull.v1.XmlPullParserException: only whitespace content allowed before start tag and not m (position: START_DOCUMENT seen m... @1:1) at org.xmlpull.mxp1.MXParser.parseProlog(MXParser.java:1519) at org.xmlpull.mxp1.MXParser.nextImpl(MXParser.java:1395) at org.xmlpull.mxp1.MXParser.next(MXParser.java:1093) at uk.me.parabola.splitter.AbstractXppParser.parse(AbstractXppParser.java:62) at uk.me.parabola.splitter.Main.processMap(Main.java:399) at uk.me.parabola.splitter.Main.writeAreas(Main.java:355) at uk.me.parabola.splitter.Main.split(Main.java:188) at uk.me.parabola.splitter.Main.start(Main.java:116) at uk.me.parabola.splitter.Main.main(Main.java:105) coords occupancy MAP occupancy: 0 ways occupancy MAP occupancy: 0 Thread worker-0 has finished Thread worker-1 has finished Thread worker-2 has finished Starting pass 3 of 5, processing 232 areas (63240465 to 63240696) Making SparseMultiMap Making SparseMultiMap Processing /dev/stdin org.xmlpull.v1.XmlPullParserException: start tag not allowed in epilog but got t (position: END_TAG seen ...<tag k="abutters" v="residential"/>\n <t... @2:7) at org.xmlpull.mxp1.MXParser.parseEpilog(MXParser.java:1588) at org.xmlpull.mxp1.MXParser.nextImpl(MXParser.java:1393) at org.xmlpull.mxp1.MXParser.next(MXParser.java:1093) at uk.me.parabola.splitter.AbstractXppParser.parse(AbstractXppParser.java:62) at uk.me.parabola.splitter.Main.processMap(Main.java:399) at uk.me.parabola.splitter.Main.writeAreas(Main.java:355) at uk.me.parabola.splitter.Main.split(Main.java:188) at uk.me.parabola.splitter.Main.start(Main.java:116) at uk.me.parabola.splitter.Main.main(Main.java:105) coords occupancy MAP occupancy: 0 ways occupancy MAP occupancy: 0 Thread worker-1 has finished Thread worker-2 has finished Thread worker-0 has finished Starting pass 4 of 5, processing 232 areas (63240697 to 63240928) Making SparseMultiMap Making SparseMultiMap Processing /dev/stdin org.xmlpull.v1.XmlPullParserException: start tag not allowed in epilog but got t (position: END_TAG seen ...<tag k="maintenance" v="gritting"/>\n <t... @2:7) at org.xmlpull.mxp1.MXParser.parseEpilog(MXParser.java:1588) at org.xmlpull.mxp1.MXParser.nextImpl(MXParser.java:1393) at org.xmlpull.mxp1.MXParser.next(MXParser.java:1093) at uk.me.parabola.splitter.AbstractXppParser.parse(AbstractXppParser.java:62) at uk.me.parabola.splitter.Main.processMap(Main.java:399) at uk.me.parabola.splitter.Main.writeAreas(Main.java:355) at uk.me.parabola.splitter.Main.split(Main.java:188) at uk.me.parabola.splitter.Main.start(Main.java:116) at uk.me.parabola.splitter.Main.main(Main.java:105) coords occupancy MAP occupancy: 0 ways occupancy MAP occupancy: 0 Thread worker-1 has finished Thread worker-0 has finished Thread worker-2 has finished Starting pass 5 of 5, processing 230 areas (63240929 to 63241158) Making SparseMultiMap Making SparseMultiMap Processing /dev/stdin org.xmlpull.v1.XmlPullParserException: only whitespace content allowed before start tag and not g (position: START_DOCUMENT seen g... @1:1) at org.xmlpull.mxp1.MXParser.parseProlog(MXParser.java:1519) at org.xmlpull.mxp1.MXParser.nextImpl(MXParser.java:1395) at org.xmlpull.mxp1.MXParser.next(MXParser.java:1093) at uk.me.parabola.splitter.AbstractXppParser.parse(AbstractXppParser.java:62) at uk.me.parabola.splitter.Main.processMap(Main.java:399) at uk.me.parabola.splitter.Main.writeAreas(Main.java:355) at uk.me.parabola.splitter.Main.split(Main.java:188) at uk.me.parabola.splitter.Main.start(Main.java:116) at uk.me.parabola.splitter.Main.main(Main.java:105) coords occupancy MAP occupancy: 0 ways occupancy MAP occupancy: 0 Thread worker-0 has finished Thread worker-1 has finished Thread worker-2 has finished Time finished: Tue May 03 01:23:04 CEST 2011 Total time taken: 8543s May 3, 2011 2:11:43 AM org.openstreetmap.osmosis.core.Osmosis run INFO: Pipeline complete. May 3, 2011 2:11:43 AM org.openstreetmap.osmosis.core.Osmosis run INFO: Total execution time: 11460683 milliseconds. lambertus@server:~/garmin/world$ ls -lh build total 5.3M -rw-r--r-- 1 lambertus lambertus 159 2011-05-03 01:23 63240001.osm.gz [...] _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Ok. Thanks Markus_g -----Original Message----- From: mkgmap-dev-bounces@lists.mkgmap.org.uk [mailto:mkgmap-dev-bounces@lists.mkgmap.org.uk] On Behalf Of Marko Mäkelä Sent: Wednesday, 4 May 2011 6:28 PM To: Development list for mkgmap Subject: Re: [mkgmap-dev] Splitter output files are nearly empty On Wed, May 04, 2011 at 06:20:41PM +0930, Markus_g wrote:
I notice there is an extra - after /bin/osmosis --rx is that an error?
No, it tells osmosis to read from the standard input, which is being piped from bunzip2 -c. Marko _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hello
***** Full GC ***** Elapsed time: 18m 0s Memory: Current 135MB (106MB used, 29MB free) Max 6666MB
The 'Full GC' line is not a problem, it is a message printed by splitter every so often before it attempts to force a garbage collect. This is not usually a useful thing to do, but it can help make the printed memory usages more accurate, I suppose. Splitter can still read .osm.gz from stdin as long as there is a --split-file option, and as long as it doesn't have to use more than one pass through the data. You should just provide no file name on the command line though. (ie do not give /dev/stdin) However /dev/stdin should work as long as the input is uncompressed. Previously there was a --cache option which would copy and convert the input to a cache file, which would then be re-read if --split-file was not given or if more than one pass was needed. Since the newer pbf format is as or more efficient than the cache file format and osmosis is capable of producing it directly it seems best to just use osmosis to write it. Having splitter convert to pbf will not save any disk space and will probably be slower than overall than osmosis producing it directly. Best wishes ..Steve
On 2011-05-04 11:51, Steve Ratcliffe wrote:
***** Full GC ***** Elapsed time: 18m 0s Memory: Current 135MB (106MB used, 29MB free) Max 6666MB
The 'Full GC' line is not a problem, it is a message printed by splitter every so often before it attempts to force a garbage collect. This is not usually a useful thing to do, but it can help make the printed memory usages more accurate, I suppose.
Thanks for clearing that up Steve. The newer splitter apparently needs only a fraction of the memory it needed before. Great work devs!
Splitter can still read .osm.gz from stdin as long as there is a --split-file option, and as long as it doesn't have to use more than one pass through the data.
Alright. I don't think splitter is able to only use one pass on a planet file so it's another reason to move to the pbf format.
You should just provide no file name on the command line though. (ie do not give /dev/stdin) However /dev/stdin should work as long as the input is uncompressed.
Thanks, the /dev/stdin ended up there because of other problems I had and it worked. I can't confirm or deny that it works without it ;-)
Previously there was a --cache option which would copy and convert the input to a cache file, which would then be re-read if --split-file was not given or if more than one pass was needed.
Since the newer pbf format is as or more efficient than the cache file format and osmosis is capable of producing it directly it seems best to just use osmosis to write it. Having splitter convert to pbf will not save any disk space and will probably be slower than overall than osmosis producing it directly.
Again: thanks. It's clear I'll better migrate to pbf.
participants (5)
-
Lambertus -
Marko Mäkelä -
Markus_g -
Steve Ratcliffe -
Thorsten Kukuk