[locator] Osmosis parameters required
I have the problem that some ways are missing in the osmosis filtered dump. Here is my osmosis call: osmosis.bat --rb europe.osm.pbf --tf accept-ways boundary=administrative --tf accept-relations boundary=administrative --used-way --used-node --wx europe-boundaries.osm.gz One example what is missing so far: Relation 7374 (http://www.openstreetmap.org/browse/relation/7374) Missing way 41119221 (http://www.openstreetmap.org/browse/way/41119221) The way is not tagged with boundary=administrative but is a member of the boundary=administrative tagged relation 7374. WanMil
El 30/04/11 00:59, WanMil escribió:
I have the problem that some ways are missing in the osmosis filtered dump.
Here is my osmosis call: osmosis.bat --rb europe.osm.pbf --tf accept-ways boundary=administrative --tf accept-relations boundary=administrative --used-way --used-node --wx europe-boundaries.osm.gz
One example what is missing so far: Relation 7374 (http://www.openstreetmap.org/browse/relation/7374) Missing way 41119221 (http://www.openstreetmap.org/browse/way/41119221) The way is not tagged with boundary=administrative but is a member of the boundary=administrative tagged relation 7374. Is that the only way missing in this relation? Ways 40404314 and 37208295 are not tagged boundary=administrative, are they present? I have the same problem with relation 342099: ways 85765542 and 85765550 are missing. I tried several ways of running osmosis but didn't succeed. I think the accept-ways part of the osmosis run is discarding those ways not tagged administrative, although they are part of a boundary=administrative mp. I tried to extract ways and mp in separate osmosis runs, but I get a java heap space error with any of the commands below: osmosis --rb spain.osm.pbf --tf accept-relations boundary=administrative --used-node --used-way --tf reject-ways --tf reject-nodes --write-xml file="spain-boundaries3.osm" osmosis --rb spain.osm.pbf --tf accept-relations boundary=administrative --used-node --used-way --write-xml file="spain-boundaries3.osm" Adding --used-way discards some ways tagged boundary=administrative which are not members of multipolygons. I'm currently checking those discarded from the spain-boundaries.osm (38) and some of them were wrongly tagged as administrative or islands missing in the continental admin_level=8 multipolygon, but others seem to be correct (e.g. ways 30108987, 32892880 and 31408486)
Try: osmosis --read-pbf europe.osm.pbf --tf accept-ways boundary=administrative --used-node --tf accept-relations boundary=administrative --used-way --write-xml boundary.osm I have't tried it, because I haven't now access to my desktop , but I think, this should catch all needed data. Henning
Am 30.04.2011 12:58, schrieb Henning Scholland:
osmosis --read-pbf europe.osm.pbf --tf accept-ways boundary=administrative --used-node --tf accept-relations boundary=administrative --used-way --write-xml boundary.osm Sorry, I've forgotten an --used-node
osmosis --read-pbf europe.osm.pbf --tf accept-ways boundary=administrative --used-node --tf accept-relations boundary=administrative --used-way --used-node --write-xml boundary.osm Henning
El 30/04/11 13:00, Henning Scholland escribió:
Am 30.04.2011 12:58, schrieb Henning Scholland:
osmosis --read-pbf europe.osm.pbf --tf accept-ways boundary=administrative --used-node --tf accept-relations boundary=administrative --used-way --write-xml boundary.osm Sorry, I've forgotten an --used-node
osmosis --read-pbf europe.osm.pbf --tf accept-ways boundary=administrative --used-node --tf accept-relations boundary=administrative --used-way --used-node --write-xml boundary.osm I get exactly the same file than without the extra --used-node in the ways --tf task.
Am 30.04.2011 14:19, schrieb Carlos Dávila:
El 30/04/11 13:00, Henning Scholland escribió:
Am 30.04.2011 12:58, schrieb Henning Scholland:
osmosis --read-pbf europe.osm.pbf --tf accept-ways boundary=administrative --used-node --tf accept-relations boundary=administrative --used-way --write-xml boundary.osm Sorry, I've forgotten an --used-node
osmosis --read-pbf europe.osm.pbf --tf accept-ways boundary=administrative --used-node --tf accept-relations boundary=administrative --used-way --used-node --write-xml boundary.osm I get exactly the same file than without the extra --used-node in the ways --tf task.
I've posted a question to the osmosis mailing list: http://lists.openstreetmap.org/pipermail/osmosis-dev/2011-April/001019.html
El 30/04/11 15:07, WanMil escribió:
I've posted a question to the osmosis mailing list: http://lists.openstreetmap.org/pipermail/osmosis-dev/2011-April/001019.html According to the answer in osmosis mailing list [1] I run osmosis --rb spain.osm.pbf --tf accept-ways boundary=administrative --used-node --tf reject-relations outPipe.0=ways --rb spain.osm.pbf --tf accept-relations boundary=administrative --used-node --used-way outPipe.0=relations --merge inPipe.0=ways inPipe.1=relations --write-xml file="spain-boundaries.osm" Although I get a file with all the desired ways and relations it doesn't seem to be the optimal osmosis call, as the output file is only ~2/3 of the uncompressed spain.osm (2.9 GB vs 4.5 GB), so a lot of unnecessary elements seems to be there. Anyway, now at first sight result is great: all streets have complete city, region, country information (and apparently correct) and some of the wrong State/Province places have gone (remaining ones could come from is_in information). Spain bounds are available at http://mapas.alternativaslibres.es/bounds.tar.gz [1] http://lists.openstreetmap.org/pipermail/osmosis-dev/2011-April/001020.html
Am 01.05.2011 00:41, schrieb Carlos Dávila:
osmosis --rb spain.osm.pbf --tf accept-ways boundary=administrative --used-node --tf reject-relations outPipe.0=ways --rb spain.osm.pbf --tf accept-relations boundary=administrative --used-node --used-way outPipe.0=relations --merge inPipe.0=ways inPipe.1=relations --write-xml file="spain-boundaries.osm"
Hi, I would suggest, that you have to reject in you first part also the nodes and in the second part nodes and ways. Henning
El 01/05/11 09:42, Henning Scholland escribió:
Am 01.05.2011 00:41, schrieb Carlos Dávila:
osmosis --rb spain.osm.pbf --tf accept-ways boundary=administrative --used-node --tf reject-relations outPipe.0=ways --rb spain.osm.pbf --tf accept-relations boundary=administrative --used-node --used-way outPipe.0=relations --merge inPipe.0=ways inPipe.1=relations --write-xml file="spain-boundaries.osm"
Hi, I would suggest, that you have to reject in you first part also the nodes and in the second part nodes and ways. I had already tried that, but with those parameters osmosis never finishes (I canceled it after 1:30 h while the above run completed in 5 minutes).
You could solve this with multiple runs of osmosis (but it might take some time). The idea is that on the first run, you retain just the relations tagged with boundary=administrative, the ways referred to in those relations, and the nodes used in those ways and relations. This gives you everything you want except for ways tagged with boundary=administrative, that are not referred to in any relation. So you do a second run on the Europe extract and retain ways tagged boundary=administrative. On the third run, you merge the two sets of results. This is similar to the advice from the osmosis mailing list, but I have filled in some details. Suggested commands (I haven't tried this): First create named pipes rel and way using mkfifo. You also need to create a file dummy.osm.pbf which contains the bounding box you want, to work round the bounding box problem in the merge function. See http://www.mkgmap.org.uk/pipermail/mkgmap-dev/2011q2/011229.html osmosis --rb europe.osm.pbf \ --tf accept-relations boundary=administrative \ --used-way idTrackerType=BitSet \ --used-node outPipe.0=rel idTrackerType=BitSet \ \ --rb europe.osm.pbf \ --tf accept-ways boundary=administrative \ --tf reject-relations \ --used-node outPipe.0=way idTrackerType=BitSet \ \ --rb dummy.osm.pbf \ --merge inPipe.1=rel \ --merge inPipe.1=way \ --wb boundary.osm.pbf An alternative that does not use named pipes (may be quicker because it only reads the input file once): osmosis --rb europe.osm.pbf \ --tee \ --tf accept-relations boundary=administrative \ --used-way idTrackerType=BitSet \ --used-node idTrackerType=BitSet \ --wb rel.osm.pbf \ --tf accept-ways boundary=administrative \ --tf reject-relations \ --used-node idTrackerType=BitSet \ --wb way.osm.pbf osmosis --rb dummy.osm.pbf \ --rb rel.osm.pbf \ --rb way.osm.pbf \ --merge \ --merge \ --wb boundary.osm.pbf
Am 30.04.2011 14:19, schrieb Carlos Dávila:
El 30/04/11 13:00, Henning Scholland escribió:
Am 30.04.2011 12:58, schrieb Henning Scholland:
osmosis --read-pbf europe.osm.pbf --tf accept-ways boundary=administrative --used-node --tf accept-relations boundary=administrative --used-way --write-xml boundary.osm Sorry, I've forgotten an --used-node
osmosis --read-pbf europe.osm.pbf --tf accept-ways boundary=administrative --used-node --tf accept-relations boundary=administrative --used-way --used-node --write-xml boundary.osm I get exactly the same file than without the extra --used-node in the ways --tf task.
I've posted a question to the osmosis mailing list: http://lists.openstreetmap.org/pipermail/osmosis-dev/2011-April/001019.html
The osmosis list has the answer: The osmosis call has to be split to 3 calls (it is possible to use the merge command and to make it in one call but the following works for sure): osmosis.bat --rb europe.osm.pbf --tf accept-ways boundary=administrative --used-node --wb bways.osm.pbf osmosis.bat --rb europe.osm.pbf --tf accept-relations boundary=administrative --used-way --used-node --wb brels.osm.pbf osmosis.bat --rb bways.osm.pbf outPipe.0=W --rb brels.osm.pbf outPipe.0=R --merge inPipe.0=W inPipe.1=R --wb europe-boundaries.osm.pbf Warning: This really needs a long time to finish. WanMil
El 01/05/11 19:44, WanMil escribió:
Am 30.04.2011 14:19, schrieb Carlos Dávila:
El 30/04/11 13:00, Henning Scholland escribió:
Am 30.04.2011 12:58, schrieb Henning Scholland:
osmosis --read-pbf europe.osm.pbf --tf accept-ways boundary=administrative --used-node --tf accept-relations boundary=administrative --used-way --write-xml boundary.osm Sorry, I've forgotten an --used-node
osmosis --read-pbf europe.osm.pbf --tf accept-ways boundary=administrative --used-node --tf accept-relations boundary=administrative --used-way --used-node --write-xml boundary.osm I get exactly the same file than without the extra --used-node in the ways --tf task.
I've posted a question to the osmosis mailing list: http://lists.openstreetmap.org/pipermail/osmosis-dev/2011-April/001019.html The osmosis list has the answer:
The osmosis call has to be split to 3 calls (it is possible to use the merge command and to make it in one call but the following works for sure):
osmosis.bat --rb europe.osm.pbf --tf accept-ways boundary=administrative --used-node --wb bways.osm.pbf You have to add --tf reject-relations, otherwise all relations are included in bways.osm.pbf (tested with a small *.osm.pbf file) osmosis.bat --rb europe.osm.pbf --tf accept-relations boundary=administrative --used-way --used-node --wb brels.osm.pbf
osmosis.bat --rb bways.osm.pbf outPipe.0=W --rb brels.osm.pbf outPipe.0=R --merge inPipe.0=W inPipe.1=R --wb europe-boundaries.osm.pbf
Warning: This really needs a long time to finish.
osmosis.bat --rb europe.osm.pbf --tf accept-ways boundary=administrative --used-node --wb bways.osm.pbf You have to add --tf reject-relations, otherwise all relations are included in bways.osm.pbf (tested with a small *.osm.pbf file)
Sounds good. I wil test it next time.
osmosis.bat --rb europe.osm.pbf --tf accept-relations boundary=administrative --used-way --used-node --wb brels.osm.pbf
osmosis.bat --rb bways.osm.pbf outPipe.0=W --rb brels.osm.pbf outPipe.0=R --merge inPipe.0=W inPipe.1=R --wb europe-boundaries.osm.pbf
Warning: This really needs a long time to finish.
El 01/05/11 19:44, WanMil escribió:
Am 30.04.2011 14:19, schrieb Carlos Dávila:
El 30/04/11 13:00, Henning Scholland escribió:
Am 30.04.2011 12:58, schrieb Henning Scholland:
osmosis --read-pbf europe.osm.pbf --tf accept-ways boundary=administrative --used-node --tf accept-relations boundary=administrative --used-way --write-xml boundary.osm Sorry, I've forgotten an --used-node
osmosis --read-pbf europe.osm.pbf --tf accept-ways boundary=administrative --used-node --tf accept-relations boundary=administrative --used-way --used-node --write-xml boundary.osm I get exactly the same file than without the extra --used-node in the ways --tf task.
I've posted a question to the osmosis mailing list: http://lists.openstreetmap.org/pipermail/osmosis-dev/2011-April/001019.html The osmosis list has the answer:
The osmosis call has to be split to 3 calls (it is possible to use the merge command and to make it in one call but the following works for sure):
osmosis.bat --rb europe.osm.pbf --tf accept-ways boundary=administrative --used-node --wb bways.osm.pbf
osmosis.bat --rb europe.osm.pbf --tf accept-relations boundary=administrative --used-way --used-node --wb brels.osm.pbf
osmosis.bat --rb bways.osm.pbf outPipe.0=W --rb brels.osm.pbf outPipe.0=R --merge inPipe.0=W inPipe.1=R --wb europe-boundaries.osm.pbf
Warning: This really needs a long time to finish.
WanMil Just to complete the information, this is the only osmosis call that worked for me to extract relations: osmosis --rb europe.osm.pbf --tf accept-relations boundary=administrative --used-node idTrackerType=BitSet --used-way idTrackerType=BitSet --wb boundary_relations.osm.pbf Without the idTrackerType=BitSet java heap space errors stop the process. It complains I should have used omitmetadata=true, but completed the task. I don't know if it is because I had previously extracted a bounding box from the original europe.osm.pbf
El 01/05/11 19:44, WanMil escribió:
Am 30.04.2011 14:19, schrieb Carlos Dávila:
El 30/04/11 13:00, Henning Scholland escribió:
Am 30.04.2011 12:58, schrieb Henning Scholland:
osmosis --read-pbf europe.osm.pbf --tf accept-ways boundary=administrative --used-node --tf accept-relations boundary=administrative --used-way --write-xml boundary.osm Sorry, I've forgotten an --used-node
osmosis --read-pbf europe.osm.pbf --tf accept-ways boundary=administrative --used-node --tf accept-relations boundary=administrative --used-way --used-node --write-xml boundary.osm I get exactly the same file than without the extra --used-node in the ways --tf task.
I've posted a question to the osmosis mailing list: http://lists.openstreetmap.org/pipermail/osmosis-dev/2011-April/001019.html The osmosis list has the answer:
The osmosis call has to be split to 3 calls (it is possible to use the merge command and to make it in one call but the following works for sure):
osmosis.bat --rb europe.osm.pbf --tf accept-ways boundary=administrative --used-node --wb bways.osm.pbf
osmosis.bat --rb europe.osm.pbf --tf accept-relations boundary=administrative --used-way --used-node --wb brels.osm.pbf
osmosis.bat --rb bways.osm.pbf outPipe.0=W --rb brels.osm.pbf outPipe.0=R --merge inPipe.0=W inPipe.1=R --wb europe-boundaries.osm.pbf
Warning: This really needs a long time to finish.
WanMil Just to complete the information, this is the only osmosis call that worked for me to extract relations: osmosis --rb europe.osm.pbf --tf accept-relations boundary=administrative --used-node idTrackerType=BitSet --used-way idTrackerType=BitSet --wb boundary_relations.osm.pbf Without the idTrackerType=BitSet java heap space errors stop the process. It complains I should have used omitmetadata=true, but completed the task. I don't know if it is because I had previously extracted a bounding box from the original europe.osm.pbf
The warning with omitmetadata=true seems to be a known osmosis warning without any effect. I have just ignored it. As it is documented the idTrackerType is an option that can be used to optimize the used-node and used-way implementation. If you give osmosis more memory it will definitely work without the option. But in some configurations it might be faster and less memory intensive. WanMil
participants (4)
-
Adrian -
Carlos Dávila -
Henning Scholland -
WanMil