Hi all, I've just committed r247. Changes: - added parameter --polygon-file to specify a bounding polygon. The polygon is only used when the tiles are calculated, it is NOT used when the real split process runs. - performance improvements in the new split algorithm The polygon file format is that of osmosis, I've simply added the osmosis sources: http://wiki.openstreetmap.org/wiki/Osmosis/Polygon_Filter_File_Format I hope that is the right way to do it. The polygon is only used when no split-file is given. It is used when all nodes were read. In this first pass, splitter fills a grid with node counts. Instead of filtering each node (which would take much too much cpu time), splitter simply uses the polygon to set those grid counts to 0 (zero) which do not intersect with the polygon. When that is done, splitter calculates the tiles. Gerd -- View this message in context: http://gis.19327.n5.nabble.com/splitter-r247-tp5737673.html Sent from the Mkgmap Development mailing list archive at Nabble.com.
Hi Gerd, r247 works not as expected. Let me explain it a little bit more. I would like to cut a area out of planet-osm. This osmconvert does. After this splitter should generate a areas.list out of this osm-extract (eg. dach.pbf, see dach.poly at the end of the mail). This areas.list should afterwards be splitted directly out of planet-file. splitter-r202 generates an areas.list for hole bounding box (max-top, max-right, min-left, min-bottom). So I have to tweak the generated areas.list manually, that it only covers the expected map-coverage. splitter-247 generates the same as r202. Henning dach.poly : http://www.aighes.de/data/dach.poly r202: http://www.aighes.de/data/r202.osm.gz manual tweaked: http://www.aighes.de/data/should_be.osm.gz
Henning Scholland wrote
Hi Gerd, r247 works not as expected.
Let me explain it a little bit more. I would like to cut a area out of planet-osm. This osmconvert does. After this splitter should generate a areas.list out of this osm-extract (eg. dach.pbf, see dach.poly at the end of the mail). This areas.list should afterwards be splitted directly out of planet-file.
splitter-r202 generates an areas.list for hole bounding box (max-top, max-right, min-left, min-bottom). So I have to tweak the generated areas.list manually, that it only covers the expected map-coverage.
splitter-247 generates the same as r202. o/mkgmap-dev
Well, okay, so I did not fully understand what you want to do. Just a question: when using r247 with --polygon-file=dach.poly, did you use no-trim=false? If not, please try that also. Without that parm, splitter always creates areas that cover the bounding box of the original file. Gerd -- View this message in context: http://gis.19327.n5.nabble.com/splitter-r247-tp5737673p5737715.html Sent from the Mkgmap Development mailing list archive at Nabble.com.
Some results for r247, I hope it is of any use: This version is able to split the planet (r246 could not) but needs more memory then r202. Using -Xmx4000m an OutOfMemoryException occurs but not with -Xmx6000m. Number of stored ids: 19,330,031 require ca. 2.04 bytes per pair. 1224726 chunks are used, the avg. number of values in one 64-chunk is 15. Map details: bytes/overhead 22,288,890 / 17,251,542, overhead includes 2 arrays with 8 MB RLE compresion info: compressed / uncompressed size / ratio: 6,245,541 / 19,330,024 / 68% JVM Memory Info: Current 5969MB (1422MB used, 4547MB free) Max 5969MB Full Node tests: 334,886,250 Quick Node tests: 114,029,611 Thread worker-0 has finished Thread worker-1 has finished Thread worker-2 has finished Distribution pass(es) took 15727941 ms (4h 20m) $ java -version java version "1.7.0_09" OpenJDK Runtime Environment (IcedTea7 2.3.3) (7u9-2.3.3-0ubuntu1~12.04.1) OpenJDK 64-Bit Server VM (build 23.2-b09, mixed mode)
Hi Lambertus, Lambertus wrote
Some results for r247, I hope it is of any use:
This version is able to split the planet (r246 could not) but needs more memory then r202. Using -Xmx4000m an OutOfMemoryException occurs but not with -Xmx6000m.
Number of stored ids: 19,330,031 require ca. 2.04 bytes per pair. 1224726 chunks are used, the avg. number of values in one 64-chunk is 15. Map details: bytes/overhead 22,288,890 / 17,251,542, overhead includes 2 arrays with 8 MB RLE compresion info: compressed / uncompressed size / ratio: 6,245,541 / 19,330,024 / 68% JVM Memory Info: Current 5969MB (1422MB used, 4547MB free) Max 5969MB Full Node tests: 334,886,250 Quick Node tests: 114,029,611 Thread worker-0 has finished Thread worker-1 has finished Thread worker-2 has finished Distribution pass(es) took 15727941 ms (4h 20m)
$ java -version java version "1.7.0_09" OpenJDK Runtime Environment (IcedTea7 2.3.3) (7u9-2.3.3-0ubuntu1~12.04.1) OpenJDK 64-Bit Server VM (build 23.2-b09, mixed mode)
Do you use the same parameters for both versions? In that case r247 should not require much more or less memory compared to r202. If you have the complete logs, please send them to me, I like to know in which phase it is likely to have memory problems. Gerd -- View this message in context: http://gis.19327.n5.nabble.com/splitter-r247-tp5737673p5738186.html Sent from the Mkgmap Development mailing list archive at Nabble.com.
On 29/11/2012 13:31, GerdP wrote:
Do you use the same parameters for both versions?
Not entirely. Compared to r202 I upped the -Xmx value and added for r247: --keep-complete --overlap=0 The rest of the commandline remained the same: java -Xmx6000m -ea -jar splitter.jar --output=xml --keep-complete --overlap=0 --no-trim --mapid=1 --max-nodes=1500000 --write-kml=initial.kml --geonames-file=cities15000.zip planet-latest.osm.pbf
In that case r247 should not require much more or less memory compared to r202.
Well, all I know for sure is that 4G was enough for r202 but not for r247. But the difference could be small if r202 was close to the 4G already.
If you have the complete logs, please send them to me, I like to know in which phase it is likely to have memory problems.
The log for r247 is attached. I'll run r202 again and send it's output as well.
Hi Lambertus, thanks for the input. Yes, the log shows that the keep-complete functions require more then 5GB. This is okay, splitter has to save more information for this compared to r202. If you have more available heap, e.g. -Xmx7500m, you could use try a higher max-areas value to reduce run time (e.g. max-areas=512 or higher) If you want to help me optimizing the heap consumption, please execute splitter r247 with java -agentlib:hprof=cpu=samples,depth=20 -XX:+PrintGCDetails -Xmx ....and send me both the splitter log and the generated java.hprof[.txt] ciao, Gerd Date: Thu, 29 Nov 2012 16:41:18 +0100 From: osm@na1400.info To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] splitter r247 On 29/11/2012 13:31, GerdP wrote:
Do you use the same parameters for both versions?
Not entirely. Compared to r202 I upped the -Xmx value and added for r247: --keep-complete --overlap=0 The rest of the commandline remained the same: java -Xmx6000m -ea -jar splitter.jar --output=xml --keep-complete --overlap=0 --no-trim --mapid=1 --max-nodes=1500000 --write-kml=initial.kml --geonames-file=cities15000.zip planet-latest.osm.pbf
In that case r247 should not require much more or less memory compared to r202.
Well, all I know for sure is that 4G was enough for r202 but not for r247. But the difference could be small if r202 was close to the 4G already.
If you have the complete logs, please send them to me, I like to know in which phase it is likely to have memory problems.
The log for r247 is attached. I'll run r202 again and send it's output as well. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
GerdP wrote
Hi Lambertus,
thanks for the input. Yes, the log shows that the keep-complete functions require more then 5GB. This is okay, splitter has to save more information for this compared to r202.
ciao, Gerd
Oh oh! I looked again at this logs and wondered why some numbers did not change during the different passes. I found out that the 7 passes in the ProblemListProcessor are more or less nonsense. The intention was to save memory by processing only a set of writers in each pass, but in fact all 7 passes store the node informations for all writers. In short: splitter requires either much less memory whhen this bug is corrected, or you can run everything in one pass if you specify --max-areas=2048 I hope I can fix this soon. Gerd -- View this message in context: http://gis.19327.n5.nabble.com/splitter-r247-tp5737673p5738287.html Sent from the Mkgmap Development mailing list archive at Nabble.com.
Great to see that a simple log can be so helpful. :) Would be nice for splitter to be much faster with only a single pass on a moderate pc (i.e. 8 gb ram), but I'm not complaining as it is... Running splitter r247 with the added logging parameters on my dev pc now, will send the results tomorrow. On 29-11-12 22:16, GerdP wrote:
Oh oh! I looked again at this logs and wondered why some numbers did not change during the different passes. I found out that the 7 passes in the ProblemListProcessor are more or less nonsense. The intention was to save memory by processing only a set of writers in each pass, but in fact all 7 passes store the node informations for all writers. In short: splitter requires either much less memory whhen this bug is corrected, or you can run everything in one pass if you specify --max-areas=2048 I hope I can fix this soon.
Gerd
-- View this message in context: http://gis.19327.n5.nabble.com/splitter-r247-tp5737673p5738287.html Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Lambertus wrote
Great to see that a simple log can be so helpful. :) Would be nice for splitter to be much faster with only a single pass on a moderate pc (i.e. 8 gb ram), but I'm not complaining as it is...
Running splitter r247 with the added logging parameters on my dev pc now, will send the results tomorrow.
Right now I don't even know if the result is correct. I think I will not need the log from r247 as I already know that it's wrong. Gerd -- View this message in context: http://gis.19327.n5.nabble.com/splitter-r247-tp5737673p5738296.html Sent from the Mkgmap Development mailing list archive at Nabble.com.
GerdP wrote
Lambertus wrote
Great to see that a simple log can be so helpful. :) Would be nice for splitter to be much faster with only a single pass on a moderate pc (i.e. 8 gb ram), but I'm not complaining as it is...
Running splitter r247 with the added logging parameters on my dev pc now, will send the results tomorrow. Right now I don't even know if the result is correct. I think I will not need the log from r247 as I already know that it's wrong.
Gerd
The results were ok, only runtime and memory usage was too high in your case. r250 corrects this. If you want to provide new logs, please run this version with java -agentlib:hprof=cpu=samples,depth=20 -XX:-HeapDumpOnOutOfMemoryError -XX:+PrintGCTimeStamps +PrintGCDetails -Xmx6000m -jar splitter.jar .... --max-areas=2048 This should result in max. 2 hours runtime. If you want to compare with the previous results, try also with --max-areas=255 Please send me the splitter log and the java.hprof . Gerd -- View this message in context: http://gis.19327.n5.nabble.com/splitter-r247-tp5737673p5738315.html Sent from the Mkgmap Development mailing list archive at Nabble.com.
Maybe because I forgot to change the -max-areas setting the split took a lot longer then expected. It even did not finish yet. CPU usage is around 1 to 4% now, this doesn't look good. I've attached the log. The commandline I used was: java -agentlib:hprof=cpu=samples,depth=20 -XX:-HeapDumpOnOutOfMemoryError -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -Xmx6000m -jar splitter.jar --output=xml --keep-complete --overlap=0 --no-trim --mapid=1 --max-nodes=1500000 --max-areas=512 --write-kml=initial.kml --geonames-file=cities15000.zip planet-latest.osm.pbf I'll start the split again using --max-areas=2048 On 30-11-12 01:53, GerdP wrote:
The results were ok, only runtime and memory usage was too high in your case. r250 corrects this.
If you want to provide new logs, please run this version with java -agentlib:hprof=cpu=samples,depth=20 -XX:-HeapDumpOnOutOfMemoryError -XX:+PrintGCTimeStamps +PrintGCDetails -Xmx6000m -jar splitter.jar .... --max-areas=2048
This should result in max. 2 hours runtime.
If you want to compare with the previous results, try also with --max-areas=255
Please send me the splitter log and the java.hprof .
Gerd
Hi Lambertus, okay, regarding performance I think it makes no sense to use a small max-areas value. The rule seems to be simple: The more passes, the longer the run time. On the other hand, it would be good to know that the results are exactly the same (same problem-list, same *.osm.gz, same areas.list). I've never tested that with planet, only with small areas. Gerd Date: Fri, 30 Nov 2012 19:57:47 +0100 From: osm@na1400.info To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] splitter r247 Maybe because I forgot to change the -max-areas setting the split took a lot longer then expected. It even did not finish yet. CPU usage is around 1 to 4% now, this doesn't look good. I've attached the log. The commandline I used was: java -agentlib:hprof=cpu=samples,depth=20 -XX:-HeapDumpOnOutOfMemoryError -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -Xmx6000m -jar splitter.jar --output=xml --keep-complete --overlap=0 --no-trim --mapid=1 --max-nodes=1500000 --max-areas=512 --write-kml=initial.kml --geonames-file=cities15000.zip planet-latest.osm.pbf I'll start the split again using --max-areas=2048 On 30-11-12 01:53, GerdP wrote: > > The results were ok, only runtime and memory usage was too high in > your case. r250 corrects this. > > If you want to provide new logs, please run this version with java > -agentlib:hprof=cpu=samples,depth=20 -XX:-HeapDumpOnOutOfMemoryError > -XX:+PrintGCTimeStamps +PrintGCDetails -Xmx6000m -jar splitter.jar > .... --max-areas=2048 > > This should result in max. 2 hours runtime. > > If you want to compare with the previous results, try also with > --max-areas=255 > > Please send me the splitter log and the java.hprof . > > Gerd > _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Please note that the message you replied to is a couple of days old and was stuck in the mailinglist server. On 02-12-12 12:08, Gerd Petermann wrote:
Hi Lambertus,
okay, regarding performance I think it makes no sense to use a small max-areas value. The rule seems to be simple: The more passes, the longer the run time. On the other hand, it would be good to know that the results are exactly the same (same problem-list, same *.osm.gz, same areas.list). I've never tested that with planet, only with small areas.
Gerd
------------------------------------------------------------------------ Date: Fri, 30 Nov 2012 19:57:47 +0100 From: osm@na1400.info To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] splitter r247
Maybe because I forgot to change the -max-areas setting the split took a lot longer then expected. It even did not finish yet. CPU usage is around 1 to 4% now, this doesn't look good. I've attached the log.
The commandline I used was:
java -agentlib:hprof=cpu=samples,depth=20 -XX:-HeapDumpOnOutOfMemoryError -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -Xmx6000m -jar splitter.jar --output=xml --keep-complete --overlap=0 --no-trim --mapid=1 --max-nodes=1500000 --max-areas=512 --write-kml=initial.kml --geonames-file=cities15000.zip planet-latest.osm.pbf
I'll start the split again using --max-areas=2048
On 30-11-12 01:53, GerdP wrote:
The results were ok, only runtime and memory usage was too high in your case. r250 corrects this.
If you want to provide new logs, please run this version with java -agentlib:hprof=cpu=samples,depth=20 -XX:-HeapDumpOnOutOfMemoryError -XX:+PrintGCTimeStamps +PrintGCDetails -Xmx6000m -jar splitter.jar .... --max-areas=2048
This should result in max. 2 hours runtime.
If you want to compare with the previous results, try also with --max-areas=255
Please send me the splitter log and the java.hprof .
Gerd
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
whoops, I noticed a bit late that there is an IndexOutOfBounds exception a bit earlier in the logfile. On 30-11-12 01:53, GerdP wrote:
GerdP wrote
Lambertus wrote
Great to see that a simple log can be so helpful. :) Would be nice for splitter to be much faster with only a single pass on a moderate pc (i.e. 8 gb ram), but I'm not complaining as it is...
Running splitter r247 with the added logging parameters on my dev pc now, will send the results tomorrow. Right now I don't even know if the result is correct. I think I will not need the log from r247 as I already know that it's wrong.
Gerd The results were ok, only runtime and memory usage was too high in your case. r250 corrects this.
If you want to provide new logs, please run this version with java -agentlib:hprof=cpu=samples,depth=20 -XX:-HeapDumpOnOutOfMemoryError -XX:+PrintGCTimeStamps +PrintGCDetails -Xmx6000m -jar splitter.jar .... --max-areas=2048
This should result in max. 2 hours runtime.
If you want to compare with the previous results, try also with --max-areas=255
Please send me the splitter log and the java.hprof .
Gerd
-- View this message in context: http://gis.19327.n5.nabble.com/splitter-r247-tp5737673p5738315.html Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Lambertus wrote
whoops, I noticed a bit late that there is an IndexOutOfBounds exception a bit earlier in the logfile.
Hmm, did you send something else? I can't see what you mean. Gerd -- View this message in context: http://gis.19327.n5.nabble.com/splitter-r247-tp5737673p5738575.html Sent from the Mkgmap Development mailing list archive at Nabble.com.
Yes, I sent an email with an attachment (zip file). Perhaps it's stuck in the spam bucket of this list? I haven't received it either. I'll send another mail with last-nights splitter run in a few minutes. On 01-12-12 08:16, GerdP wrote:
Lambertus wrote
whoops, I noticed a bit late that there is an IndexOutOfBounds exception a bit earlier in the logfile. Hmm, did you send something else? I can't see what you mean.
Gerd
-- View this message in context: http://gis.19327.n5.nabble.com/splitter-r247-tp5737673p5738575.html Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
El 26/11/12 15:45, GerdP escribió:
Hi all,
I've just committed r247. Changes: - added parameter --polygon-file to specify a bounding polygon. The polygon is only used when the tiles are calculated, it is NOT used when the real split process runs. - performance improvements in the new split algorithm
The polygon file format is that of osmosis, I've simply added the osmosis sources: http://wiki.openstreetmap.org/wiki/Osmosis/Polygon_Filter_File_Format
I hope that is the right way to do it.
The polygon is only used when no split-file is given. It is used when all nodes were read. In this first pass, splitter fills a grid with node counts. Instead of filtering each node (which would take much too much cpu time), splitter simply uses the polygon to set those grid counts to 0 (zero) which do not intersect with the polygon. When that is done, splitter calculates the tiles. I have just used r247 to produce a map of South America and noticed that although it results in an smaller number of tiles for the same max-nodes, tiles aspect ratio is not optimal (see screenshots). r247: java -Xmx1500M -jar splitter-pl.jar --keep-complete --overlap=0 --max-nodes=1000000 --no-trim --geonames-file=cities15000_AME.zip --mapid=55139001 south-america.osm.pbf 48 tiles, 474.3 MB resulting img's r202: java -Xmx1500M -jar splitter.jar --overlap=3000 --max-nodes=1000000 --no-trim --geonames-file=cities15000_AME.zip --mapid=55139001 south-america.osm.pbf 56 tiles, 475.3 MB resulting img's
Hi Carlos, Carlos Dávila-2 wrote
El 26/11/12 15:45, GerdP escribió: I have just used r247 to produce a map of South America and noticed that although it results in an smaller number of tiles for the same max-nodes, tiles aspect ratio is not optimal (see screenshots). r247: java -Xmx1500M -jar splitter-pl.jar --keep-complete --overlap=0 --max-nodes=1000000 --no-trim --geonames-file=cities15000_AME.zip --mapid=55139001 south-america.osm.pbf 48 tiles, 474.3 MB resulting img's r202: java -Xmx1500M -jar splitter.jar --overlap=3000 --max-nodes=1000000 --no-trim --geonames-file=cities15000_AME.zip --mapid=55139001 south-america.osm.pbf 56 tiles, 475.3 MB resulting img's
Please send me the densities-out.txt file produced by r247. Just a question: What is the reason for using --no-trim? I think it is not possible to create "nice" tiles when this option is used. Gerd -- View this message in context: http://gis.19327.n5.nabble.com/splitter-r247-tp5737673p5737719.html Sent from the Mkgmap Development mailing list archive at Nabble.com.
Am 26.11.2012 20:26, schrieb GerdP:
Hi Carlos,
Carlos Dávila-2 wrote
El 26/11/12 15:45, GerdP escribió: I have just used r247 to produce a map of South America and noticed that although it results in an smaller number of tiles for the same max-nodes, tiles aspect ratio is not optimal (see screenshots). r247: java -Xmx1500M -jar splitter-pl.jar --keep-complete --overlap=0 --max-nodes=1000000 --no-trim --geonames-file=cities15000_AME.zip --mapid=55139001 south-america.osm.pbf 48 tiles, 474.3 MB resulting img's r202: java -Xmx1500M -jar splitter.jar --overlap=3000 --max-nodes=1000000 --no-trim --geonames-file=cities15000_AME.zip --mapid=55139001 south-america.osm.pbf 56 tiles, 475.3 MB resulting img's Please send me the densities-out.txt file produced by r247. Just a question: What is the reason for using --no-trim? I think it is not possible to create "nice" tiles when this option is used. --no-trim results in a coverage without gaps. Without no-trim you'll get gaps in regions without data. Especially in water regions.
Henning
Henning Scholland wrote
--no-trim results in a coverage without gaps. Without no-trim you'll get gaps in regions without data. Especially in water regions.
OK, I understand. On the other hand, it gives the result that you want with your dach example, see attached kml files generated wth max-nodes=1200000. hdach.zip <http://gis.19327.n5.nabble.com/file/n5737724/hdach.zip> I think what we need is a special version of trimming. Let me think about this ... Gerd -- View this message in context: http://gis.19327.n5.nabble.com/splitter-r247-tp5737673p5737724.html Sent from the Mkgmap Development mailing list archive at Nabble.com.
GerdP wrote
Henning Scholland wrote
--no-trim results in a coverage without gaps. Without no-trim you'll get gaps in regions without data. Especially in water regions. OK, I understand. On the other hand, it gives the result that you want with your dach example, see attached kml files generated wth max-nodes=1200000. hdach.zip <http://gis.19327.n5.nabble.com/file/n5737724/hdach.zip>
I think what we need is a special version of trimming. Let me think about this ...
Gerd
I think what we need with polygons it this: - when setting the grid fields outside of the polygon to 0, splitter also must make sure that no grid field inside the polygon is 0 - having done that, splitter can activate trimming This is a simple change, the result is the splitter_auto_trim.kml in the attached zip hdach2.zip <http://gis.19327.n5.nabble.com/file/n5737726/hdach2.zip> To avoid holes, something similar could be done also if no polygon file is given. Wouldn't that be much better? Gerd -- View this message in context: http://gis.19327.n5.nabble.com/splitter-r247-tp5737673p5737726.html Sent from the Mkgmap Development mailing list archive at Nabble.com.
Am 26.11.2012 20:44, schrieb GerdP:
Henning Scholland wrote
--no-trim results in a coverage without gaps. Without no-trim you'll get gaps in regions without data. Especially in water regions. OK, I understand. On the other hand, it gives the result that you want with your dach example, see attached kml files generated wth max-nodes=1200000. hdach.zip <http://gis.19327.n5.nabble.com/file/n5737724/hdach.zip>
I think what we need is a special version of trimming. Let me think about this ...
Gerd Yes, then it works as it should be. Maybe your new algorithm is better as the older one.
Henning
Am 26.11.2012 21:08, schrieb Henning Scholland:
Am 26.11.2012 20:44, schrieb GerdP:
Henning Scholland wrote
--no-trim results in a coverage without gaps. Without no-trim you'll get gaps in regions without data. Especially in water regions. OK, I understand. On the other hand, it gives the result that you want with your dach example, see attached kml files generated wth max-nodes=1200000. hdach.zip <http://gis.19327.n5.nabble.com/file/n5737724/hdach.zip>
I think what we need is a special version of trimming. Let me think about this ...
Gerd Yes, then it works as it should be. Maybe your new algorithm is better as the older one. But not at all. If there are larger spaces with no data, no-trim=false will create "ugly" looking maps. As you can see here: http://www.aighes.de/data/scandinavia.png world.kml was created out of planet.osm with scandinavia.poly and no-trim=false
btw: Are there reasons to use overlap!= in combination with --keep-complete? Otherwise overlap=0 should be default if keep-complete=true. Henning
Henning Scholland wrote
But not at all. If there are larger spaces with no data, no-trim=false will create "ugly" looking maps. As you can see here: http://www.aighes.de/data/scandinavia.png world.kml was created out of planet.osm with scandinavia.poly and no-trim=false
btw: Are there reasons to use overlap!= in combination with --keep-complete? Otherwise overlap=0 should be default if keep-complete=true.
Just to make sure that I got this right, besides the holes the only ugly looking part is the big tile covering most of iceland, but also a large area outside of the polygon. Correct? Any other tile that is not acceptable? Reg. overlap: I agree that overlap should be ignored with --keep-complete. I just don't have enough experience to decide this alone. I wasnot sure whether overlap=0 should be forced with --keep-complete or whether it should be ignored only if user did not explicitely specify a value. I think forcing overlap=0 is the better solution. So, I think this adds three points to my todo list: - make sure that trimming doesn't generate holes within the polygon (easy) - force overlap=0 if keep-complete is used (easy) - make sure that gereated tiles do not overlap the polygon too much (not that easy) Gerd -- View this message in context: http://gis.19327.n5.nabble.com/splitter-r247-tp5737673p5737815.html Sent from the Mkgmap Development mailing list archive at Nabble.com.
Am 27.11.2012 12:47, schrieb GerdP:
Henning Scholland wrote
But not at all. If there are larger spaces with no data, no-trim=false will create "ugly" looking maps. As you can see here: http://www.aighes.de/data/scandinavia.png world.kml was created out of planet.osm with scandinavia.poly and no-trim=false
btw: Are there reasons to use overlap!= in combination with --keep-complete? Otherwise overlap=0 should be default if keep-complete=true. Just to make sure that I got this right, besides the holes the only ugly looking part is the big tile covering most of iceland, but also a large area outside of the polygon. Correct? Any other tile that is not acceptable? No the other tiles are ok.
Reg. overlap: I agree that overlap should be ignored with --keep-complete. I just don't have enough experience to decide this alone. I wasnot sure whether overlap=0 should be forced with --keep-complete or whether it should be ignored only if user did not explicitely specify a value. I think forcing overlap=0 is the better solution. +1
Henning
Reg. overlap: I agree that overlap should be ignored with --keep-complete. I just don't have enough experience to decide this alone. I wasnot sure whether overlap=0 should be forced with --keep-complete or whether it should be ignored only if user did not explicitely specify a value. I think forcing overlap=0 is the better solution.
There is a *small* exception where overlap=0 does not produce the same results like overlap>0 with keep-complete. The new process-destination option copies destination tags from links to motorways/trunks. In case the motorway intersects the bbox and the link does not the destination tag will not be copied to the motorway with overlap=0 because the link is missing in the tile data. So I propose to keep the overlap parameter but default it to 0 with keep-complete. The described case should be *very* seldom... WanMil
Hi all, GerdP wrote
So, I think this adds three points to my todo list: - make sure that trimming doesn't generate holes within the polygon (easy) - force overlap=0 if keep-complete is used (easy) - make sure that gereated tiles do not overlap the polygon too much (not that easy)
Regarding overlap I plan this: Default is -1, if keep-complete=true, the program sets it to 0, else to 2000. If user specifies a value > 0 together with keep-complete=true, the program prints a warning message to stderr: "Warning: --overlap is used in combination with --keep-complete=true" reg. overlapping polygons: I have a solution that works well for rectilinear polygons with only a few corners (like those that Henning uses), but it collapses when you use the polygons from geofabrik which contain diagonal lines. It's quite complicated to approximate such a polygon with rectangles , and I fear the result will never be near to what the user wants: Either it contains additional areas outside the original polygon or data is missing. So I plan to stop processing if such a polygon is used with a message like "Bounding polygon is too complex. Avoid diagonal lines!" Would that be okay for all? Gerd -- View this message in context: http://gis.19327.n5.nabble.com/splitter-r247-tp5737673p5738196.html Sent from the Mkgmap Development mailing list archive at Nabble.com.
Could splitter output a sensible polygon on runs without using one afterwards? Geofabrik doesn't change their polygons often, so if one runs it, splitter creates them, and one can reuse them for future runs, that would be the best I think... Maybe splitter could add 100m or 200m on generation, so small changes wouldn't matter (or does the polygon need to be exact?). On 29.11.2012 15:26, GerdP wrote:
" reg. overlapping polygons: I have a solution that works well for rectilinear polygons with only a few corners (like those that Henning uses), but it collapses when you use the polygons from geofabrik which contain diagonal lines. It's quite complicated to approximate such a polygon with rectangles , and I fear the result will never be near to what the user wants: Either it contains additional areas outside the original polygon or data is missing. So I plan to stop processing if such a polygon is used with a message like "Bounding polygon is too complex. Avoid diagonal lines!" Would that be okay for all? Gerd
Regarding overlap I plan this: Default is -1, if keep-complete=true, the program sets it to 0, else to 2000. If user specifies a value > 0 together with keep-complete=true, the program prints a warning message to stderr: "Warning: --overlap is used in combination with --keep-complete=true"
Hi Gerd, can you make the warning a bit more user friendly and add a hint, why this usually is not a good idea to use overlap>0 in combination with keep-complete=true? I think people that doesn't follow the dev list won't have an easy chance to find out why it is better to use overlap=0 with keep-complete=true. And there are many old threads of the time before keep-complete was introduced pointing out that overlap *must* be set. Thanks! WanMil
WanMil wrote
can you make the warning a bit more user friendly and add a hint, why this usually is not a good idea to use overlap>0 in combination with keep-complete=true? I think people that doesn't follow the dev list won't have an easy chance to find out why it is better to use overlap=0 with keep-complete=true. And there are many old threads of the time before keep-complete was introduced pointing out that overlap *must* be set.
I agree. In fact I hate parameters that are mutual exclusive. Whenever possible we should try to avoid that. So what about this (written to stderr) ? Warning: --overlap is used in combination with --keep-complete=true The option keep-complete should be used with overlap=0 because it is very unlikely that the overlap will add any important data. It will just cause a lot of additional output which has to be thrown away again in mkgmap. Gerd -- View this message in context: http://gis.19327.n5.nabble.com/splitter-r247-tp5737673p5738618.html Sent from the Mkgmap Development mailing list archive at Nabble.com.
WanMil wrote
can you make the warning a bit more user friendly and add a hint, why this usually is not a good idea to use overlap>0 in combination with keep-complete=true? I think people that doesn't follow the dev list won't have an easy chance to find out why it is better to use overlap=0 with keep-complete=true. And there are many old threads of the time before keep-complete was introduced pointing out that overlap *must* be set.
I agree. In fact I hate parameters that are mutual exclusive. Whenever possible we should try to avoid that. So what about this (written to stderr) ? Warning: --overlap is used in combination with --keep-complete=true The option keep-complete should be used with overlap=0 because it is very unlikely that the overlap will add any important data. It will just cause a lot of additional output which has to be thrown away again in mkgmap.
Gerd
Sounds good! WanMil
participants (7)
-
Carlos Dávila -
Felix Hartmann -
Gerd Petermann -
GerdP
-
Henning Scholland -
Lambertus -
WanMil