error splitting binary files
Hi, I have tried to use the splitter v161 on geofabriks europe.osm.pbf binary file with a custom areas.list file (link see below) but I get this error message (see attachment). Is it a bug? Only If I use it without a areas.list file it works ok. areas list file: http://mijndev.openstreetmap.nl/~ligfietser/openfietsmap/Scripts/areas_europ... Thanks, Minko PS I asked Steve, and he replied:
From the screen shot it looks like it is attempting to parse the file as XML and not as pbf, so as long as you are using the version that deals with pbf then it looks like a bug.
On Sun, 31 Oct 2010 15:39:31 -0700 Steve wrote: I've built a complete splitter distribution from the pbf branch and put it at http://files.mkgmap.org.uk/download/6/splitter-r161.zip It includes everything you need. File: splitter-r161.zip created on 31 oct 2010 URL: http://files.mkgmap.org.uk/download/6/splitter-r161.zip Description: Complete splitter distribution built from the .osm.pbf enabled branch.
Hi I know, it wont help you, but I haven't nay problems with pbf-input and modified area.list. I think the only think I do divernt is splitting the europe-file with osmosis to the needed size. aighes -- View this message in context: http://gis.638310.n2.nabble.com/error-splitting-binary-files-tp5758967p57590... Sent from the Mkgmap Development mailing list archive at Nabble.com.
I tried to split an extracted area with osmosis too. Same errors. I suppose you need to set osmosis to --write-pbf in order to get a binary abstract? aighes wrote: Hi I know, it wont help you, but I haven't nay problems with pbf-input and modified area.list. I think the only think I do divernt is splitting the europe-file with osmosis to the needed size. aighes
Hi Minko, I use --write-binary file=... omitmetadata=true usedense=false compress=none. aighes -- View this message in context: http://gis.638310.n2.nabble.com/error-splitting-binary-files-tp5758967p57598... Sent from the Mkgmap Development mailing list archive at Nabble.com.
On 11/20/2010 11:08 PM, aighes wrote:
I know, it wont help you, but I haven't nay problems with pbf-input and modified area.list. I think the only think I do divernt is splitting the europe-file with osmosis to the needed size.
I have splitted the europe pbf extract successfully without osmosis and with a pre-generated areas.list.
Hi, I tried it again, splitted then europe.pbf now without osmosis and without errors. So i don't know what went wrong the first time. The benelux took me now only 50 minutes (instead of many hours with the bzip2 file) Thanks, Minko Ralf wrote: I have splitted the europe pbf extract successfully without osmosis and with a pre-generated areas.list.
I'm having problems with splitting the latest europe.osm.pbf extracts from geofabrik (5 & 6 dec). After 25 sec the splitting seems done, normally it would take one hour. I have tried a smaller section too (Belgium.osm.pbf) but that looks fine. I'm using Splitter 161.
Hi I've the same problem with todays europe.osm.pbf. Extract of 02.12. works fine (was the last I used). aighes -- View this message in context: http://gis.638310.n2.nabble.com/error-splitting-binary-files-tp5758967p58089... Sent from the Mkgmap Development mailing list archive at Nabble.com.
Same with today's extract. Where do I sent a bug report to? Or is it a bug in the splitter? Can osmosis handle this file correctly?
It seems to be an error in extract-file. Osmosis has also problems Frederik is informed about it (talk-de) and will take a closer look to it. aighes -- View this message in context: http://gis.638310.n2.nabble.com/error-splitting-binary-files-tp5758967p58113... Sent from the Mkgmap Development mailing list archive at Nabble.com.
Thanks, I'm following the discussion on talk-de. I just splitted netherlands.osm.pbf (08-Dec-2010 02:40) and it seems all the relations are gone. Lines and polygons are ok. I also mailed Frederik about this. The splitted europe.osm.pbf (benelux area) from today showed only some pois. BTW im using windows. "aighes" wrote: It seems to be an error in extract-file. Osmosis has also problems Frederik is informed about it (talk-de) and will take a closer look to it.
Hi, Any news about this problem (which still exists)? "aighes" wrote: It seems to be an error in extract-file. Osmosis has also problems Frederik is informed about it (talk-de) and will take a closer look to it. aighes -- View this message in context: http://gis.638310.n2.nabble.com/error-splitting-binary-files-tp5758967p58113... Sent from the Mkgmap Development mailing list archive at Nabble.com.
Hi It was discussed on osmosis-dev ML. There was just a differenc in some bytes in the beginning of a working file and the not working file of Geofabrik. The part which includes the data was exactly the same. No one knew why one file doesn't work on windows. Regards, aighes
-----Ursprüngliche Nachricht----- Von: mkgmap-dev-bounces@lists.mkgmap.org.uk [mailto:mkgmap-dev- bounces@lists.mkgmap.org.uk] Im Auftrag von Minko Gesendet: Montag, 20. Dezember 2010 18:49 An: Development list for mkgmap Cc: frederik Betreff: Re: [mkgmap-dev] error splitting binary files
Hi,
Any news about this problem (which still exists)?
"aighes" wrote: It seems to be an error in extract-file. Osmosis has also problems Frederik is informed about it (talk-de) and will take a closer look to it.
aighes -- View this message in context: http://gis.638310.n2.nabble.com/error- splitting-binary-files-tp5758967p5811383.html Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Until 2 Dec. everything went ok with splitting the binary europe extract. Splitting took me about 1 hr on a modest laptop (windows vista). After 2 December there were some problems with the country extracts as well, but they were solved. It looked like they did some changes with the europe.osm.pbf because since that date I'm not able to split it anymore, nor with splitter nor with osmosis. When I try to split the europe.osm.pbf only pois are extracted in a few minutes time. Since the introduction of the pbf file something got changed with the europe.osm.bz2 extract as well. Normally it took me about 2 hrs to split the data, but since october they published the pbf file, splitting times of europe.osm.bz2 extract went up to >7 hours! See also http://www.mail-archive.com/mkgmap-dev@lists.mkgmap.org.uk/msg07071.html ----- Oorspronkelijk bericht ----- Van: "Henning Scholland" <h.scholland@googlemail.com> Aan: "Development list for mkgmap" <mkgmap-dev@lists.mkgmap.org.uk> Verzonden: Vrijdag 24 december 2010 11:37:32 Onderwerp: Re: [mkgmap-dev] error splitting binary files Hi It was discussed on osmosis-dev ML. There was just a differenc in some bytes in the beginning of a working file and the not working file of Geofabrik. The part which includes the data was exactly the same. No one knew why one file doesn't work on windows. Regards, aighes
Am 24.12.2010 11:37, schrieb Henning Scholland:
It was discussed on osmosis-dev ML. There was just a differenc in some bytes in the beginning of a working file and the not working file of Geofabrik. The part which includes the data was exactly the same.
You are sure that one of the two files was working on windows ? My guess was that it's some kind of 4 GB problem (only on windows) with pbf input files. Is splitter using the same pbf-code like osmosis ? Chris
Yes, the file generated under windows was working well. 4GB problem can't be, because of using 64bit java and also in the past I processed files larger then 4 GB and the windows-generated pbf-file is also larger than 4GB. Regards aighes
-----Ursprüngliche Nachricht----- Von: mkgmap-dev-bounces@lists.mkgmap.org.uk [mailto:mkgmap-dev- bounces@lists.mkgmap.org.uk] Im Auftrag von Chris66 Gesendet: Freitag, 24. Dezember 2010 20:44 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] error splitting binary files
Am 24.12.2010 11:37, schrieb Henning Scholland:
It was discussed on osmosis-dev ML. There was just a differenc in some bytes in the beginning of a working file and the not working file of Geofabrik. The part which includes the data was exactly the same.
You are sure that one of the two files was working on windows ?
My guess was that it's some kind of 4 GB problem (only on windows) with pbf input files.
Is splitter using the same pbf-code like osmosis ?
Chris
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
On Fri, Dec 24, 2010 at 3:40 PM, Henning Scholland <h.scholland@googlemail.com> wrote:
Yes, the file generated under windows was working well. 4GB problem can't be, because of using 64bit java and also in the past I processed files larger then 4 GB and the windows-generated pbf-file is also larger than 4GB.
Evidence seems to indicate that 64-bit java has a 4gb problem in windows that is encountered when handling PBF files. There is some code that is shared between osmosis and the splitter (osmbin.jar) for pbf file reading, and thus may both have this kind of bug with input files >4gb. Scott
participants (7)
-
aighes -
Chris66 -
Henning Scholland
-
Minko -
Ralf Kleineisel -
Scott Crosby -
Torsten Leistikow