New splitter build for pbf format
Hi Scott announced that he has made an important fix to the osmpbf library which fixes a bug only seen on Windows with large files. So I have rebuilt the pbf capable version of splitter with this newly released version of osmpbf.jar. There are no other differences so if you were not affected by the bug you do not need to re-download. The new download is at: URL: http://files.mkgmap.org.uk/download/7/splitter-r161-2.zip ..Steve
Hi Steve, I tried to compile the splitter myself, after copying in the libs from your splitter download. However it failed. After I overwrote the build.xml and /resources/manifest.mf it would compile without problems. Why are theese build.xml and manifest.mf files not committed? On 26.01.2011 12:31, Steve Ratcliffe wrote:
Hi
Scott announced that he has made an important fix to the osmpbf library which fixes a bug only seen on Windows with large files.
So I have rebuilt the pbf capable version of splitter with this newly released version of osmpbf.jar.
There are no other differences so if you were not affected by the bug you do not need to re-download.
The new download is at:
URL: http://files.mkgmap.org.uk/download/7/splitter-r161-2.zip
..Steve _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Felix
compile without problems. Why are theese build.xml and manifest.mf files not committed?
Because it is not my branch and I just wanted to get a build that was useful. All the source necessary to build itself is contained in the jar file. I do want to get a trunk version that works with pbf and I have contacted Scott and Chris to see what they want to proceed. ..Steve
Hi Steve, hi Scott, I tried the new splitter build but it's not working for me. I splitted the 4.6GB europe.osm.pbf using the default splitter arguments. The output was 64 osm.gz files with a total size of 881M which seems to contain nodes only. WanMil
Hi
Scott announced that he has made an important fix to the osmpbf library which fixes a bug only seen on Windows with large files.
So I have rebuilt the pbf capable version of splitter with this newly released version of osmpbf.jar.
There are no other differences so if you were not affected by the bug you do not need to re-download.
The new download is at:
URL: http://files.mkgmap.org.uk/download/7/splitter-r161-2.zip
..Steve _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi WanMil
I tried the new splitter build but it's not working for me. I splitted the 4.6GB europe.osm.pbf using the default splitter arguments. The output was 64 osm.gz files with a total size of 881M which seems to contain nodes only.
Although I am guilty of re-packaging it without testing further I have just run a split against germany.osm.pbf and it ran fine. I used a different max-nodes so I can't directly compare with my previous one though. ..Steve
Am 26.01.2011 13:54, schrieb WanMil:
Hi Steve, hi Scott,
I tried the new splitter build but it's not working for me. I splitted the 4.6GB europe.osm.pbf using the default splitter arguments. The output was 64 osm.gz files with a total size of 881M which seems to contain nodes only.
WanMil
Hi
Scott announced that he has made an important fix to the osmpbf library which fixes a bug only seen on Windows with large files.
So I have rebuilt the pbf capable version of splitter with this newly released version of osmpbf.jar.
There are no other differences so if you were not affected by the bug you do not need to re-download.
The new download is at:
URL: http://files.mkgmap.org.uk/download/7/splitter-r161-2.zip
..Steve _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Maybe Geofabrik must create the files also with new version. So it will take some days, til they do so. Cheers Henning
On Wed, Jan 26, 2011 at 7:54 AM, Henning Scholland <osm@aighes.de> wrote:
Am 26.01.2011 13:54, schrieb WanMil:
Hi Steve, hi Scott,
I tried the new splitter build but it's not working for me.
I splitted the 4.6GB europe.osm.pbf using the default splitter arguments. The output was 64 osm.gz files with a total size of 881M which seems to contain nodes only.
Thats my fault. I didn't actually push the new osmpbf.jar to osmosis's SVN, so Steve never incorporated the latest, fixed, version of the jar. I've just pushed it so hopefully Steve can rerelease a new version. I apologize for the error. The problem is entirely with the pbf reader --- a 32-bit cleanness bug that was only exposed on Windows made the reader erroneously sense end-of-file. Existing PBF files are fine and do not need to be regenerated. Scott
Hi
Thats my fault. I didn't actually push the new osmpbf.jar to osmosis's SVN, so Steve never incorporated the latest, fixed, version of the jar. I've just pushed it so hopefully Steve can rerelease a new version. I apologize for the error.
Ahh, so that was the bug in action. I've repackaged the splitter at URL: http://files.mkgmap.org.uk/download/8/splitter-r161-3.zip ..Steve
Hi
Thats my fault. I didn't actually push the new osmpbf.jar to osmosis's SVN, so Steve never incorporated the latest, fixed, version of the jar. I've just pushed it so hopefully Steve can rerelease a new version. I apologize for the error.
Ahh, so that was the bug in action. I've repackaged the splitter at URL: http://files.mkgmap.org.uk/download/8/splitter-r161-3.zip
..Steve
Thanks a lot, it's working now. It's the first time I tried the pbf splitter. The performance is great!! Anyhow here are my wishes to the ongoing splitter development (don't know if this is a good place?): * Apply the useful output-dir patch to the pbf branch * Output format pbf (most wanted - where can we start to implement?) * The output tiles should fill the complete bounding box of the input file. Sometimes it happens that areas without any data are not covered by the tile bounds. Example: http://dev.openstreetmap.de/aio/tiles/?zoom=7&lat=46.01815&lon=-3.47465&laye... shows the tiles of the All-in-One-Map. On the left, top and right of tile 66210071 there are free areas that are not covered by a tile. So in the generated map there is a hole instead of sea. WanMil
Is also the big file problem fixed? I think about the europe file splitting. Is this problem solved? Thanks, Marco Am 28.01.2011 18:00, schrieb WanMil:
Hi
Thats my fault. I didn't actually push the new osmpbf.jar to osmosis's SVN, so Steve never incorporated the latest, fixed, version of the jar. I've just pushed it so hopefully Steve can rerelease a new version. I apologize for the error. Ahh, so that was the bug in action. I've repackaged the splitter at URL: http://files.mkgmap.org.uk/download/8/splitter-r161-3.zip
..Steve Thanks a lot, it's working now. It's the first time I tried the pbf splitter. The performance is great!!
Anyhow here are my wishes to the ongoing splitter development (don't know if this is a good place?):
* Apply the useful output-dir patch to the pbf branch * Output format pbf (most wanted - where can we start to implement?) * The output tiles should fill the complete bounding box of the input file. Sometimes it happens that areas without any data are not covered by the tile bounds. Example: http://dev.openstreetmap.de/aio/tiles/?zoom=7&lat=46.01815&lon=-3.47465&laye... shows the tiles of the All-in-One-Map. On the left, top and right of tile 66210071 there are free areas that are not covered by a tile. So in the generated map there is a hole instead of sea.
WanMil _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
On 28.01.2011 19:24, dom Team OiD wrote:
Is also the big file problem fixed? I think about the europe file splitting. Is this problem solved? That was the only problem (windows only), and it is solved. Thanks, Marco
Am 28.01.2011 18:00, schrieb WanMil:
Hi
Thats my fault. I didn't actually push the new osmpbf.jar to osmosis's SVN, so Steve never incorporated the latest, fixed, version of the jar. I've just pushed it so hopefully Steve can rerelease a new version. I apologize for the error. Ahh, so that was the bug in action. I've repackaged the splitter at URL: http://files.mkgmap.org.uk/download/8/splitter-r161-3.zip
..Steve Thanks a lot, it's working now. It's the first time I tried the pbf splitter. The performance is great!!
Anyhow here are my wishes to the ongoing splitter development (don't know if this is a good place?):
* Apply the useful output-dir patch to the pbf branch * Output format pbf (most wanted - where can we start to implement?) * The output tiles should fill the complete bounding box of the input file. Sometimes it happens that areas without any data are not covered by the tile bounds. Example: http://dev.openstreetmap.de/aio/tiles/?zoom=7&lat=46.01815&lon=-3.47465&laye... shows the tiles of the All-in-One-Map. On the left, top and right of tile 66210071 there are free areas that are not covered by a tile. So in the generated map there is a hole instead of sea.
WanMil _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
On Fri, Jan 28, 2011 at 11:00 AM, WanMil <wmgcnfg@web.de> wrote:
Hi
Anyhow here are my wishes to the ongoing splitter development (don't know if this is a good place?):
I'm not likely to have time for splitter development for a while. I'm putting my OSM work into implementing a new planet format.
* Apply the useful output-dir patch to the pbf branch
If you send me a cleanly applying patch, I'll put it on the crosby_integration branch.
* Output format pbf (most wanted - where can we start to implement?)
If you're interested in doing it, I'd suggest modelling from the osmosis code. To my knowledge osmosis has the only implementation of a pbf writer.
* The output tiles should fill the complete bounding box of the input file. Sometimes it happens that areas without any data are not covered by the tile bounds. Example: http://dev.openstreetmap.de/aio/tiles/?zoom=7&lat=46.01815&lon=-3.47465&laye... shows the tiles of the All-in-One-Map. On the left, top and right of tile 66210071 there are free areas that are not covered by a tile. So in the generated map there is a hole instead of sea.
Ah, Try the --no-trim option. Scott
Am 27.01.2011 03:55, schrieb Scott Crosby:
The problem is entirely with the pbf reader --- a 32-bit cleanness bug that was only exposed on Windows made the reader erroneously sense end-of-file.
Hi Scott, is this a known java Bug? If not, has it been reported to oracle ? Chris
On Thu, Jan 27, 2011 at 7:31 AM, Chris66 <chris66nrw@gmx.de> wrote:
Am 27.01.2011 03:55, schrieb Scott Crosby:
The problem is entirely with the pbf reader --- a 32-bit cleanness bug that was only exposed on Windows made the reader erroneously sense end-of-file.
Hi Scott, is this a known java Bug?
Don't know. You could argue that there's a 64-bit cleanliness issue in the API. InputStream.available() returns 32-bit integer. I was formerly calling that function to detect EOF, but its not needed if I instead catch EOFException.
If not, has it been reported to oracle ?
I've not reported it. Scott
Am 27.01.2011 23:40, schrieb Scott Crosby:
The problem is entirely with the pbf reader --- a 32-bit cleanness bug that was only exposed on Windows made the reader erroneously sense end-of-file.
is this a known java Bug?
Don't know. You could argue that there's a 64-bit cleanliness issue in the API. InputStream.available() returns 32-bit integer.
I see. From the docs: --- InputStream.available : public int available() throws IOException Returns an *estimate* of the number of bytes that can be read (or skipped over) from this input stream without blocking --- So, this seems to be one of the java functions which one should not use... Chris
participants (9)
-
Chris66 -
dom Team OiD -
Felix Hartmann -
Henning Scholland -
Minko -
Scott Crosby -
Steve Ratcliffe -
Torsten Leistikow -
WanMil