Hi! I still have the problem that either lake Vänern in Sweden or lake Saimaa in Finland are partly rendered as land. The problem still exists using the latest precompiled sea from Wanmil. Using one of my older areas.list Vänern is OK but Saimaa is not. Letting the splitter create its own areas.list Saimaa is OK but then southern Vänern is dried out.
Am 17.09.2012 17:42, schrieb RheinSkipper:
The problem still exists using the latest precompiled sea from Wanmil.
This is completely independent, because they only create sea, no lakes.
Using one of my older areas.list Vänern is OK but Saimaa is not. Letting the splitter create its own areas.list Saimaa is OK but then southern Vänern is dried out.
You have to tweak your areas.list manually, so that the hole multipolygon is in one maptile. You could also take a look at my areas.list file (http://www.aighes.de/OSM/data/style.zip) (cc-by). It's in data-directory and starting with 1200. There are only two errors in Saimaa. Henning
Would it be difficult to have precompiled water rather than sea? It would save messing about with area files. Regards, Geoff. From: aighes Sent: Monday, September 17, 2012 5:08 PM To: Development list for mkgmap Subject: Re: [mkgmap-dev] Still problems with lakes Am 17.09.2012 17:42, schrieb RheinSkipper: The problem still exists using the latest precompiled sea from Wanmil. This is completely independent, because they only create sea, no lakes. Using one of my older areas.list Vänern is OK but Saimaa is not. Letting the splitter create its own areas.list Saimaa is OK but then southern Vänern is dried out. You have to tweak your areas.list manually, so that the hole multipolygon is in one maptile. You could also take a look at my areas.list file (http://www.aighes.de/OSM/data/style.zip) (cc-by). It's in data-directory and starting with 1200. There are only two errors in Saimaa. Henning -------------------------------------------------------------------------------- _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
It is not only large lakes that are affected, also large forest or other multipolygon areas are sometimes broken at the tile borders. This need to be fixed in the splitter. Sometimes it helps making the overlap higher (5000 or more). But this is just a workaround...
Would it be difficult to have precompiled water rather than sea?
It would save messing about with area files.
Regards, Geoff.
Hi, just to let everybody know: I began development on an improved splitter to handle this problem some months ago, but I got stuck after my system crashed when installing an SSD (my fault). After reinstalling around 10 times I was really pissed with computers and than I started planning (and doing) two longer bike rides. I am not sure if I find the energy to start again with splitter, my solution is somewhat half done. It turned out to be quite complex to collect all needed information without requiring huge amounts of memory or writing large temp files when splitting e.g. Europe . As a first work aproach I decided to implement multiple read passes, which is quite slow with pbf format, so I also implemented reading (and writing o5m format), which makes splitter much faster. The major open problem: Many polygons cross tiles, but most of them do not need special handling (think of all the houses near tile borders) You cannot decide that before you know all nodes (not just the ids, but the position) and tags of the polygon. It is impossible to collect all needed infomation for all polygons in heap, so one needs clever filters and effective data structures to filter only the really problematic polygons. I must confess that I did not find these clever filters after weeks of thinking, so maybe I don't see the forest because of the trees, maybe it is really too complex for me. A completely different approach came into my mind today: Instead of implementing a lot of logic to find the polygons, it might already help if we implement a function that allows to specify polygon ids that cause problems if they are incomplete, and splitter would simply make sure that those polygons are completely written to all related tiles? If someone is willing to help or continue, I can create a patch for a new branch, else maybe the winter will bring me back to the computer ;) Ciao, GerdP
Date: Tue, 18 Sep 2012 08:26:08 +0200 From: ligfietser@online.nl To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] Still problems with lakes
It is not only large lakes that are affected, also large forest or other multipolygon areas are sometimes broken at the tile borders. This need to be fixed in the splitter. Sometimes it helps making the overlap higher (5000 or more). But this is just a workaround...
Would it be difficult to have precompiled water rather than sea?
It would save messing about with area files.
Regards, Geoff.
mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
@GerdP: Some ideas concerning the "huge polygon problem": - huge polygons are rare - a broken polygon leads to an ugly (amateur like) map - it's hard to find all broken polygons in a map manually Some questions concerning a solution: - is it possible to implement a "huge polygon" parameter ? - the parameter acts as threshold and triggers splitter for extra processing - the parameter could be the area (e.g. 2000 sqkm) or circumference (e.g. 1000 km) of an "huge" polygon If the calculating of all polygons has a large negative performance impact: - is it possible to identify the huge polygons in an extra (on demand) splitter run ? - the result could be a list with all identified huge polygons (generated only once) - this list is than the input for the essential splitter run What do you think ? Regards Klaus PS: What is the exactly influence of the overlapping parameter concerning the splitting of (huge) polyons ? -- View this message in context: http://gis.19327.n5.nabble.com/Still-problems-with-lakes-tp5725668p5726209.h... Sent from the Mkgmap Development mailing list archive at Nabble.com.
Hi, I did never think about calculating the size of the area. My understanding of the problem is that 1) mkgmap sometimes fails to handle incomplete (closed) polygons 2) splitter sometimes causes incomplete polygons, esp. for large polygons that cross tile borders My approach to solve the problem: 1) detect polygons that cross tile borders (I call them multi-tile-polygons mtp). This is not so difficult and is already implemented in my version: you just have to note polygons that have points in more than one tile. 2) detect those tiles that are (or maybe) crossed or enclosed by an mtp, because mkgmap should see the complete data of them, but it doesn't with splitters (r200) implementation. This is were I got stuck. I tried to keep the precise coordinates of all points belonging to all mtp, but that is far too much data, so splitter would no longer work on 32 bit machines, and I wanted to avoid that. So, I started to implement filters for mtp that are not able to cause problems, which reduced the amount of data to maybe 50%, but still my algorithm is keeping too many false candidates and therefor requires too much heap. Just to make this clear: beause of the structure of the OSM data, splitter first sees all points, than all ways, than the relation definitions. Up to now, splitter can keep the information to what tiles a given point was written, but it can't save details about coordinates. It is not possible to calculate the size of an area without having the coordinates (no matter how precise), so this can't help. If you open the kml file produced by splitter (using the --write-kml parm) with e.g. google earth, you see the tiles. Any polygon which is near the border of two tiles is likely to be a mtp, but obviously most of them are houses or small areas, and only a very small amount is interesting for us. The problem is: You can't tell before you know the coordinates, because splitter just knows that a mtp contains points in two or more tiles, it doesn't know anything about the position of these points whithin the tiles. Even this bit of information requires a lot of heap (r200 keeps more or less the same data and is already highly optimized) So, what I have until now is a splitter program that doesn't write incomplete polygon data to the tiles (if the source data is complete), but it still doesn't write mtp data to tiles that do not contain any point of the polygon. If input is in o5m format, this version performs quite well and doesn't require much more heap memory, the values are almost equal if I remember correctly. Ciao, Gerd
Date: Wed, 19 Sep 2012 07:25:10 -0700 From: easyclasspage@googlemail.com To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] Still problems with lakes
@GerdP:
Some ideas concerning the "huge polygon problem": - huge polygons are rare - a broken polygon leads to an ugly (amateur like) map - it's hard to find all broken polygons in a map manually
Some questions concerning a solution: - is it possible to implement a "huge polygon" parameter ? - the parameter acts as threshold and triggers splitter for extra processing - the parameter could be the area (e.g. 2000 sqkm) or circumference (e.g. 1000 km) of an "huge" polygon
If the calculating of all polygons has a large negative performance impact: - is it possible to identify the huge polygons in an extra (on demand) splitter run ? - the result could be a list with all identified huge polygons (generated only once) - this list is than the input for the essential splitter run
What do you think ?
Regards Klaus
PS: What is the exactly influence of the overlapping parameter concerning the splitting of (huge) polyons ?
-- View this message in context: http://gis.19327.n5.nabble.com/Still-problems-with-lakes-tp5725668p5726209.h... Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi do you thought about filtering by a given tag-list? Mainly there is water- or forest-polygons causing visual problems. Buildings look mostly corrupt because of lower precision of Garmin-coords. About how much memory we are talking about for splitting a planet or Europe or Germany? It would be great, if user could decide, if splitter should filter polygons or not. Eg. if I would have a machine with 16gb RAM it doesn't harm if splitter need more RAM to generate better maps. Of course also it should be possible to split the planet with smaller machines. Henning
Hi Henning, yes, I think looking at the tags is needed and the most promising approach. This was one of the reasons why I stopped: Up to now, splitter doesn't contain much logic regarding the tags, most of that is only in mkgmap. The same goes for the handling of multi-polygon-relations. I did not want to double all the program code, and I did not want to invent completely new code. Splitter and mkgmap use totally different data structures, so it is not easy to share code. Regarding the memory needs: afaik splitter r200 is able to split europe on a 32 bit system (with a max. heap of about 1.5 GB) with three or four read passes (plus one for the calc. of the areas.list) . Planet will require about 10 or more passes. On 64 bit system with eg 15 GB available heap I guess that even planet can be done in two or three passes. I think these are already gigantic requirements, so I hate the idea of adding anything that could double these needs, not talking about a factor of 10. Ciao, Gerd
Date: Thu, 20 Sep 2012 10:45:34 +0200 From: osm@aighes.de To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] Still problems with lakes
Hi do you thought about filtering by a given tag-list? Mainly there is water- or forest-polygons causing visual problems. Buildings look mostly corrupt because of lower precision of Garmin-coords.
About how much memory we are talking about for splitting a planet or Europe or Germany? It would be great, if user could decide, if splitter should filter polygons or not. Eg. if I would have a machine with 16gb RAM it doesn't harm if splitter need more RAM to generate better maps. Of course also it should be possible to split the planet with smaller machines.
Henning
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
If a list with the IDs of all huge polygons is helpful, such a list could perhaps created by - user (manually) - mkgmap (automatic) Regards Klaus -- View this message in context: http://gis.19327.n5.nabble.com/Still-problems-with-lakes-tp5725668p5726700.h... Sent from the Mkgmap Development mailing list archive at Nabble.com.
Sonds like an easy workaround without to much data, which have to cached. A manual list would be a good thing, because mainly you as a user know, which polygons are problematic. Maybe mkgmap could throw out a csv-list of each polygon and multipolygon, which is causing problems. r<id>, <link to object > (for mp) w<id>, <link to object> (for p) So the user could have a view to the polygon and could start after checking the list a second run. Of course it won't help, if you don't use a fix areas.list Also wo could start a wiki-page and collect problematic objects. Henning Am 21.09.2012 14:59, schrieb toc-rox:
If a list with the IDs of all huge polygons is helpful, such a list could perhaps created by - user (manually) - mkgmap (automatic)
Regards Klaus
Hello Henning,
Sonds like an easy workaround without to much data, which have to cached. A manual list would be a good thing, because mainly you as a user know, which polygons are problematic.
I like the idea of a list, however it is produced. I wanted to create the list automatically, but it was too big because of false candidates. As long as no one finds an effecient and correct algorithm to create the list in splitter, we can at least use it to produce correct output in splitter and to verify that mkgmap produces good maps with it. The last point is quite important: mkgmap contains a lot of routines to handle incomplete or wrong or redundant data. That makes it very difficult to verify wether a change in splitter is producing better maps.
Maybe mkgmap could throw out a csv-list of each polygon and multipolygon, which is causing problems.
I am not sure if mkgmap really detects all ids that cause problems. I think it can't do that if the problem is a missing relation (in one tile), but maybe mkgmap sees the same relation id again in other tiles and reports it then.
r<id>, <link to object > (for mp) w<id>, <link to object> (for p)
So the user could have a view to the polygon and could start after checking the list a second run.
Well, dependent on the input I expect the list can contain thousands of ids, so I doubt that you will want to look at that.
Of course it won't help, if you don't use a fix areas.list
Hmm, as long as the input files dont change, splitter will always produce the same areas.list, so I see no big problem here. It doesn't harm if the list of problematic ids contains a few ids that are no longer problematic, as long as it is complete. The automatic algorithm that I implemented creates a complete list, but that list contains too many false entries (I guess the ratio is something like 1:100 (100 false entries for one really problematic id)
Also wo could start a wiki-page and collect problematic objects.
Henning
Thanks for the feedback. I'll try coding a few changes (e.g. filtering by tags), if that doesn't work out, I'll code the handling of a external list and we'll see what it helps. Gerd
Am 21.09.2012 14:59, schrieb toc-rox:
If a list with the IDs of all huge polygons is helpful, such a list could perhaps created by - user (manually) - mkgmap (automatic)
Regards Klaus
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi, here is a first try to fix this issue. splitter_problem_list.patch <http://gis.19327.n5.nabble.com/file/n5731258/splitter_problem_list.patch> A sample list of problem polygons: problem_polygons.txt <http://gis.19327.n5.nabble.com/file/n5731258/problem_polygons.txt> Specify it in the new parameter problem-file, e.g. --problem-file=./problem_polygons.txt It is recommended to use o5m format as input. To convert an existing osm.pbf file to o5m use e.g. osmconvert --drop-version --out-o5m europe.osm.pbf > europe.o5m Sorry, the new code is not well commented yet, but I think it works quite well now. Any feedback is welcome. Ciao, GerdP RheinSkipper wrote
Thanks for the feedback. I'll try coding a few changes (e.g. filtering by tags), if that doesn't work out, I'll code the handling of a external list and we'll see what it helps.
Gerd
Thank you so much!
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
-- View this message in context: http://gis.19327.n5.nabble.com/Still-problems-with-lakes-tp5725668p5731258.h... Sent from the Mkgmap Development mailing list archive at Nabble.com.
Od: GerdP <gpetermann_muenchen@hotmail.com>
Hi Gerd,
here is a first try to fix this issue. splitter_problem_list.patch <http://gis.19327.n5.nabble.com/file/n5731258/splitter_problem_list.patch>
I would like to try your patch as I have problem with glacier polygon when I build map of Greenland. I patched r200 splitter directory ... how to build splitter.jar file now?
A sample list of problem polygons: problem_polygons.txt <http://gis.19327.n5.nabble.com/file/n5731258/problem_polygons.txt>
Specify it in the new parameter problem-file, e.g. --problem-file=./problem_polygons.txt
It is recommended to use o5m format as input. To convert an existing osm.pbf file to o5m use e.g. osmconvert --drop-version --out-o5m europe.osm.pbf > europe.o5m
Why not osm.pbf format? Thanks for your work mira
Hi Gerd, is it possible to use a normal splitter r200 with o5m input-data? What about mkgmap? Henning Am 17.10.2012 16:36, schrieb GerdP:
Hi,
here is a first try to fix this issue. splitter_problem_list.patch <http://gis.19327.n5.nabble.com/file/n5731258/splitter_problem_list.patch>
A sample list of problem polygons: problem_polygons.txt <http://gis.19327.n5.nabble.com/file/n5731258/problem_polygons.txt>
Specify it in the new parameter problem-file, e.g. --problem-file=./problem_polygons.txt
It is recommended to use o5m format as input. To convert an existing osm.pbf file to o5m use e.g. osmconvert --drop-version --out-o5m europe.osm.pbf > europe.o5m
Sorry, the new code is not well commented yet, but I think it works quite well now. Any feedback is welcome.
Ciao, GerdP
Thanks for providing the patch ! I would like to verify the patch with a map of sweden (vänern sea). But I'm not familiar with pachtes ... how to apply the patch ? Or is it possible to provide a ready-to-run patched splitter ? Regards Klaus -- View this message in context: http://gis.19327.n5.nabble.com/Still-problems-with-lakes-tp5725668p5731380.h... Sent from the Mkgmap Development mailing list archive at Nabble.com.
+1
-----Ursprüngliche Nachricht----- Von: mkgmap-dev-bounces@lists.mkgmap.org.uk [mailto:mkgmap-dev- bounces@lists.mkgmap.org.uk] Im Auftrag von toc-rox Gesendet: Donnerstag, 18. Oktober 2012 10:09 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] [Patch V1]Re: Still problems with lakes
Thanks for providing the patch !
I would like to verify the patch with a map of sweden (vänern sea). But I'm not familiar with pachtes ... how to apply the patch ? Or is it possible to provide a ready-to-run patched splitter ?
Regards Klaus
-- View this message in context: http://gis.19327.n5.nabble.com/Still- problems-with-lakes-tp5725668p5731380.html Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Klaus wrote:
Or is it possible to provide a ready-to-run patched splitter ?
+1 If this patch will be implemented, will there be a list somewhere (wiki?) where we can add problematic multipolygons? Or maybe some script can detect those large mps automatically from the OSM database?
Hi Minko, Minko-2 wrote
If this patch will be implemented, will there be a list somewhere (wiki?) where we can add problematic multipolygons?
This is probably the biggest problem with this "solution". Therefore I consider any user written list of problem cases as a work-around. On the other hand, we may find out that most users find the same problem cases, which in turn would allow to distribute one list for all. Minko-2 wrote
Or maybe some script can detect those large mps automatically from the OSM database?
This would be great. As I've already written, I failed to produce the list within splitter. The algorithm that I've implemented calculates the bounding box of a way or relation to find out the tiles that should "know" the polygon. If this turns out to be good enough, and OSM api allows it, someone could provide a file that contains the info for each OSM way and relation (just the Id and the bbox info). Given that list (ordered by Id), splitter would not require much additional heap to write complete info to each tile, because we could simply merge the information while reading the normal input files. Ciao, Gerd -- View this message in context: http://gis.19327.n5.nabble.com/Still-problems-with-lakes-tp5725668p5731942.h... Sent from the Mkgmap Development mailing list archive at Nabble.com.
Am 18.10.2012 11:44, schrieb Minko:
If this patch will be implemented, will there be a list somewhere (wiki?) where we can add problematic multipolygons? I started a list here: http://wiki.openstreetmap.org/wiki/Mkgmap/help/problematic_polygons
Thanks Henning, I added a few more.
I started a list here: http://wiki.openstreetmap.org/wiki/Mkgmap/help/problematic_polygons
The patch works for my sweden map (vänern): Before: <http://gis.19327.n5.nabble.com/file/n5732113/Bildschirmfoto_2012-10-20_um_21.13.01.png> After: <http://gis.19327.n5.nabble.com/file/n5732113/Bildschirmfoto_2012-10-20_um_21.33.49.png> Grand ... Klaus -- View this message in context: http://gis.19327.n5.nabble.com/Still-problems-with-lakes-tp5725668p5732113.h... Sent from the Mkgmap Development mailing list archive at Nabble.com.
On Sun, Oct 21, toc-rox wrote:
The patch works for my sweden map (vänern):
Before: <http://gis.19327.n5.nabble.com/file/n5732113/Bildschirmfoto_2012-10-20_um_21.13.01.png>
After: <http://gis.19327.n5.nabble.com/file/n5732113/Bildschirmfoto_2012-10-20_um_21.33.49.png>
On both pictures you see some lines, where you see the background and not the polygon. Is this an artefact of the rounding issues discussed the last days here? Thorsten -- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg)
Hi Klaus, attached is r200 with my patch applied (the binary) splitter.jar <http://gis.19327.n5.nabble.com/file/n5731709/splitter.jar> Ciao, Gerd -- View this message in context: http://gis.19327.n5.nabble.com/Still-problems-with-lakes-tp5725668p5731709.h... Sent from the Mkgmap Development mailing list archive at Nabble.com.
Hello Henning, no, it is not (yet). I plan to add o5m support to mkgmap soon. With my patch you can use splitter with o5m, if you do not specify the problem-file parm, it should produce the same results as r200. Gerd
Date: Wed, 17 Oct 2012 22:58:48 +0200 From: osm@aighes.de To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] [Patch V1]Re: Still problems with lakes
Hi Gerd, is it possible to use a normal splitter r200 with o5m input-data? What about mkgmap?
Henning
Am 17.10.2012 16:36, schrieb GerdP:
Hi,
here is a first try to fix this issue. splitter_problem_list.patch <http://gis.19327.n5.nabble.com/file/n5731258/splitter_problem_list.patch>
A sample list of problem polygons: problem_polygons.txt <http://gis.19327.n5.nabble.com/file/n5731258/problem_polygons.txt>
Specify it in the new parameter problem-file, e.g. --problem-file=./problem_polygons.txt
It is recommended to use o5m format as input. To convert an existing osm.pbf file to o5m use e.g. osmconvert --drop-version --out-o5m europe.osm.pbf > europe.o5m
Sorry, the new code is not well commented yet, but I think it works quite well now. Any feedback is welcome.
Ciao, GerdP
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Steve, Steve Ratcliffe wrote
Hello Gerd
no, it is not (yet). I plan to add o5m support to mkgmap soon. With my patch you can use splitter
As an aside, what do you think it is about the o5m format that makes it quicker than pbf?
Well, not easy to say. I think it's a combination of many small points: 1) pbf uses (by default) compressied blocks, so you have to unzip a complete block before you can use any information in the block. 2) pbf read routines create a lot of temporary objects, this seems to stress GC 3) pbf doesn't allow to skip processing of node tags or way tags, but splitters' read passes often don't need them. So, with pbf we create lists of tags and return them to GC, with o5m we can simply skip them. To be fair, using the --drop-version parm in osmconvert removes a lot of info which is ignored by splitter and mkgmap. I did never try what effect is has to use pbf input that was created with this parm. When writing, o5m is probably only faster because it doesn't zip the data. As long as mkgmap doesn't understand o5m I see no benefit in using this. Maybe other computers show different results, esp. if the CPU is much faster than mine and the Disk access is slower. By the way: my patch also speeds up pbf reading a little bit. Ciao, Gerd -- View this message in context: http://gis.19327.n5.nabble.com/Still-problems-with-lakes-tp5725668p5731418.h... Sent from the Mkgmap Development mailing list archive at Nabble.com.
Hi Steve,
Steve Ratcliffe wrote
Hello Gerd
no, it is not (yet). I plan to add o5m support to mkgmap soon. With my patch you can use splitter
As an aside, what do you think it is about the o5m format that makes it quicker than pbf?
Well, not easy to say. I think it's a combination of many small points: 1) pbf uses (by default) compressied blocks, so you have to unzip a complete block before you can use any information in the block. 2) pbf read routines create a lot of temporary objects, this seems to stress GC 3) pbf doesn't allow to skip processing of node tags or way tags, but splitters' read passes often don't need them. So, with pbf we create lists of tags and return them to GC, with o5m we can simply skip them.
To be fair, using the --drop-version parm in osmconvert removes a lot of info which is ignored by splitter and mkgmap. I did never try what effect is has to use pbf input that was created with this parm.
When writing, o5m is probably only faster because it doesn't zip the data. As long as mkgmap doesn't understand o5m I see no benefit in using this.
Maybe other computers show different results, esp. if the CPU is much faster than mine and the Disk access is slower. By the way: my patch also speeds up pbf reading a little bit.
Ciao, Gerd
Hi Gerd I've done some tests with the latest splitter version r255. I have split the geofabrics europe extract in pbf and o5m format. As you pointed out o5m processing is much quicker (8528s vs. 12939s). I also observed that pbf seemed to use more memory than o5m and therefore I activated gc logging and checked it with garbagecat. The interesting values are Throughput o5m: 94% pbf: 61% So 3400m seems to be too small for pbf processing to workout the europe extract so that the GC runs permanently. Total Pause: o5m: 527816ms = 528s pbf: 5093916ms = 5094s Wow, so for pbf GC requires 4566s more time. Subtracting the GC time from the total processing time o5m and pbf need quite the same time: o5m: 8528s - 528s = 8000s pbf: 12939s - 5094s = 7845s Obviously a part of the difference in GC time can be explained with your thoughts (pbf must extract all parts and must read tags which are thrown away directly afterwards). But do you think that the whole difference can be explained with that? I will post my logfiles directly to you because they are too big to be posted on the mailing list. WanMil
Hi WanMil, I can't say for sure what the pbf reader is doing in detail, but it is for sure creating a lot more temporary objects which have to be GCed. In the o5m reader I tried to avoid that. In fact, in the current implementation, the o5m reader still reads and saves the tags to the internal string table, so that is similar to the pbf reader. I'll look at your logs soon. I am working on a tuning guide for splitter, because I found a lot of nonsense in the net searching for splitter. Ciio, Gerd
Date: Thu, 13 Dec 2012 15:27:07 +0100 From: wmgcnfg@web.de To: mkgmap-dev@lists.mkgmap.org.uk Subject: [mkgmap-dev] Splitter pbf vs o5m processing
Hi Steve,
Steve Ratcliffe wrote
Hello Gerd
no, it is not (yet). I plan to add o5m support to mkgmap soon. With my patch you can use splitter
As an aside, what do you think it is about the o5m format that makes it quicker than pbf?
Well, not easy to say. I think it's a combination of many small points: 1) pbf uses (by default) compressied blocks, so you have to unzip a complete block before you can use any information in the block. 2) pbf read routines create a lot of temporary objects, this seems to stress GC 3) pbf doesn't allow to skip processing of node tags or way tags, but splitters' read passes often don't need them. So, with pbf we create lists of tags and return them to GC, with o5m we can simply skip them.
To be fair, using the --drop-version parm in osmconvert removes a lot of info which is ignored by splitter and mkgmap. I did never try what effect is has to use pbf input that was created with this parm.
When writing, o5m is probably only faster because it doesn't zip the data. As long as mkgmap doesn't understand o5m I see no benefit in using this.
Maybe other computers show different results, esp. if the CPU is much faster than mine and the Disk access is slower. By the way: my patch also speeds up pbf reading a little bit.
Ciao, Gerd
Hi Gerd
I've done some tests with the latest splitter version r255. I have split the geofabrics europe extract in pbf and o5m format.
As you pointed out o5m processing is much quicker (8528s vs. 12939s). I also observed that pbf seemed to use more memory than o5m and therefore I activated gc logging and checked it with garbagecat.
The interesting values are Throughput o5m: 94% pbf: 61% So 3400m seems to be too small for pbf processing to workout the europe extract so that the GC runs permanently.
Total Pause: o5m: 527816ms = 528s pbf: 5093916ms = 5094s Wow, so for pbf GC requires 4566s more time.
Subtracting the GC time from the total processing time o5m and pbf need quite the same time: o5m: 8528s - 528s = 8000s pbf: 12939s - 5094s = 7845s
Obviously a part of the difference in GC time can be explained with your thoughts (pbf must extract all parts and must read tags which are thrown away directly afterwards). But do you think that the whole difference can be explained with that?
I will post my logfiles directly to you because they are too big to be posted on the mailing list.
WanMil _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Gerd, before adding severals boundarys and many ferry-ways in baltic sea just a question. Would it be possible to add wildcards like "all ways and relation with ferry=*" or "all ways and relations with admin_level=2" ? How do others thing about it? Are there more ways which have typical a larger distance between nodes? Henning
Hi Henning, I don't want to make the parser for this list too complex, but maybe I can copy some code from mkgmap. If you like, you can code some rules in the MultiTileProcessor, it contains a deactivated code in processRelation(), search for "experimental code:". The more heap you can offer to splitter, and the less big your input file is, the more likely splitter can handle more (or all) relations without running out of memory. Reg. admin_level=2: That's funny, I thought it might be useful to delete all boundary=administrative relations completely and only use the precompiled bounds. Gerd
Date: Sun, 21 Oct 2012 13:08:47 +0200 From: osm@aighes.de To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] [Patch V1]Re: Still problems with lakes
Hi Gerd, before adding severals boundarys and many ferry-ways in baltic sea just a question. Would it be possible to add wildcards like "all ways and relation with ferry=*" or "all ways and relations with admin_level=2" ?
How do others thing about it? Are there more ways which have typical a larger distance between nodes?
Henning
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
I have tested the patch but the southern half of Lake Geneva is still dry, Multipolygon: http://www.openstreetmap.org/browse/relation/332617 areas.list: 00000094: 2162688,278528 to 2183168,339968 # : 46.406250,5.976563 to 46.845703,7.294922 00000554: 2142208,290816 to 2162688,339968 # : 45.966797,6.240234 to 46.406250,7.294922 00001046: 2146304,258048 to 2162688,290816 # : 46.054688,5.537109 to 46.406250,6.240234 Tested on today's extract from http://download.geofabrik.de/openstreetmap/europe/alps.osm.pbf java -Xmx1400m -jar splitter-r200\splitter_patched.jar --write-kml=areas.kml --split-file=areas.list --no-trim --output=pbf --problem-file=problem_polygons.txt alps.osm.pbf
Hi Minko, Minko-2 wrote
I have tested the patch but the southern half of Lake Geneva is still dry,
Okay, I try to find out wether it is splitter that still writes incomplete data or is it mkgmap. Ciao, Gerd -- View this message in context: http://gis.19327.n5.nabble.com/Still-problems-with-lakes-tp5725668p5732180.h... Sent from the Mkgmap Development mailing list archive at Nabble.com.
Minko-2 wrote
I have tested the patch but the southern half of Lake Geneva is still dry, ... java -Xmx1400m -jar splitter-r200\splitter_patched.jar --write-kml=areas.kml --split-file=areas.list --no-trim --output=pbf --problem-file=problem_polygons.txt alps.osm.pbf ...
Hi Minko, you haven't specified a problem file ? E.g. something like this: java -Xmx6000M -jar /home/kto/Freizeitkarte-Entwicklung/tools/splitter/splitter-poly.jar --geonames-file=/home/kto/Freizeitkarte-Entwicklung/cities/cities15000.zip --problem-file=/home/kto/Freizeitkarte-Entwicklung/*problem_polygons.txt* --mixed --no-trim --overlap=10000 --mapid=58440001 --max-nodes=600000 --output=xml --output-dir=/home/kto/Freizeitkarte-Entwicklung/work/Freizeitkarte_Schweden /home/kto/Freizeitkarte-Entwicklung/source/Freizeitkarte_Schweden.o5m Regards Klaus -- View this message in context: http://gis.19327.n5.nabble.com/Still-problems-with-lakes-tp5725668p5732194.h... Sent from the Mkgmap Development mailing list archive at Nabble.com.
I did, Klaus: --problem-file=problem_polygons.txt I only didnt convert it to 05m format, but that shouldnt matter or does it?
java -Xmx1400m -jar splitter-r200\splitter_patched.jar --write-kml=areas.kml --split-file=areas.list --no-trim --output=pbf --problem-file=problem_polygons.txt alps.osm.pbf ...
Hi Minko,
you haven't specified a problem file ?
Hi Minko, Minko-2 wrote
I did, Klaus: --problem-file=problem_polygons.txt
I only didnt convert it to 05m format, but that shouldnt matter or does it?
No, that doesn't matter. I found the error, here is a new patch based on r202 (also containing the boundingBox.patch) + the new binary. Thanks for testing! splitter_problem_list_v2.patch <http://gis.19327.n5.nabble.com/file/n5732197/splitter_problem_list_v2.patch> splitter.jar <http://gis.19327.n5.nabble.com/file/n5732197/splitter.jar> Ciao, Gerd -- View this message in context: http://gis.19327.n5.nabble.com/Still-problems-with-lakes-tp5725668p5732197.h... Sent from the Mkgmap Development mailing list archive at Nabble.com.
Thanks Gerd, works fine now!
I only didnt convert it to 05m format, but that shouldnt matter or does it?
No, that doesn't matter. I found the error, here is a new patch based on r202 (also containing the boundingBox.patch) + the new binary.
Thanks for testing!
splitter_problem_list_v2.patch <http://gis.19327.n5.nabble.com/file/n5732197/splitter_problem_list_v2.patch>
splitter.jar <http://gis.19327.n5.nabble.com/file/n5732197/splitter.jar>
Ciao, Gerd
Od: GerdP <gpetermann_muenchen@hotmail.com>
splitter_problem_list_v2.patch <http://gis.19327.n5.nabble.com/file/n5732197/splitter_problem_list_v2.patch>
splitter.jar <http://gis.19327.n5.nabble.com/file/n5732197/splitter.jar>
I just want to confirm that this second patch fixing my issue with southern tip of Greenland's continental glacier which I reported before. Now it is totally perfect. Many thanks! mira
Minko-2 wrote
I did, Klaus: --problem-file=problem_polygons.txt
Oops - sorry - it seems I'm blind today. Regards Klaus -- View this message in context: http://gis.19327.n5.nabble.com/Still-problems-with-lakes-tp5725668p5732199.h... Sent from the Mkgmap Development mailing list archive at Nabble.com.
On Sun, Oct 21, Gerd Petermann wrote:
Reg. admin_level=2: That's funny, I thought it might be useful to delete all boundary=administrative relations completely and only use the precompiled bounds.
You cannot use precompiled bounds to draw borders on your map, can you? -- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg)
Hi Thorsten, well, believe it or not, I did never create a map to use it, I am just trying to solve the problems ;-) Your are probably right, afaik the precompiled bounds are only used to set the admin level tags, the data cannot be accessed to draw an object. Ciao, Gerd
Date: Sun, 21 Oct 2012 19:13:57 +0200 From: kukuk@suse.de To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] [Patch V1]Re: Still problems with lakes
On Sun, Oct 21, Gerd Petermann wrote:
Reg. admin_level=2: That's funny, I thought it might be useful to delete all boundary=administrative relations completely and only use the precompiled bounds.
You cannot use precompiled bounds to draw borders on your map, can you?
-- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Gerd, I found another problem. If a long way has no nodes in a tile but is crossing the tile, then this long way isn't copied into the crossing tile. Is it possible to fix this? Henning
Hi Henning, do you use the 2nd version of the patch? This should already work as you expect it. If yes, please let me know how to reproduce the error. By the way: I think I found a way to generate the problem list without too many false candidates. Ciao, Gerd
Date: Wed, 24 Oct 2012 16:07:45 +0200 From: osm@aighes.de To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] [Patch V1]Re: Still problems with lakes
Hi Gerd, I found another problem. If a long way has no nodes in a tile but is crossing the tile, then this long way isn't copied into the crossing tile. Is it possible to fix this?
Henning
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Gerd, I'm using v2 of your patch. areas.list: 16000005: 2320384,-153600 to 2369536,-55296 16000045: 2320384,-55296 to 2381824,26624 problematic way:67416703 This way has only nodes in 16000045 and is crossing 16000005. He isn't displayed with josm, if split the both tiles out of a planet.pbf but it is in the file (opened with texteditor). Also mkgmap doesn't render it. Similar problem: way:29911983 and way:150505423 have one node in 16000045 and are in the file but aren't shown in josm and mkgmap-generated map. Bu there are not all nodes of the ways in the tiles. Maybe this is problematic? Henning
Hi Henning, thanks, I can reproduce the problem and I am able to fix it. Splitter skips some nodes because they are not written to any tile using the normal algorithm. It has to check first whether the nodes belongs to a problem way or relation. Wait for a new version in the problem-list branch (r207 still contains this error) ciao, Gerd
Date: Thu, 25 Oct 2012 15:54:49 +0200 From: osm@aighes.de To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] [Patch V1]Re: Still problems with lakes
Hi Gerd, I'm using v2 of your patch.
areas.list: 16000005: 2320384,-153600 to 2369536,-55296 16000045: 2320384,-55296 to 2381824,26624
problematic way:67416703
This way has only nodes in 16000045 and is crossing 16000005. He isn't displayed with josm, if split the both tiles out of a planet.pbf but it is in the file (opened with texteditor). Also mkgmap doesn't render it.
Similar problem: way:29911983 and way:150505423 have one node in 16000045 and are in the file but aren't shown in josm and mkgmap-generated map.
Bu there are not all nodes of the ways in the tiles. Maybe this is problematic?
Henning
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
I managed to tweak my areas.list so both Saimaa and Vänern look good. In case anyone likes to use it (until there is a better solution), I attach it here. It works for the complete Geofabrik Europe extract.
This need to be fixed in the splitter. Sometimes it helps making the overlap higher (5000 or more). But this is just a workaround...
Regards, Geoff.
Even at an overlap of 12000 Vänern is still corrupted. Tweaking the areas.list is also not very satisfying as there may be more corrupted lakes and forests which I did not find, yet. And for Saimaa it would be a lot of work as there are very many tile borders to be shifted out of the lake. I really hope that GerdP or someone else can provide a new splitter to handle this problem automatically.
participants (12)
-
aighes -
Geoff Sherlock -
Gerd Petermann -
GerdP
-
Henning Scholland
-
Jaromír Mikeš -
Minko -
RheinSkipper -
Steve Ratcliffe -
Thorsten Kukuk -
toc-rox -
WanMil