I would like a maxways parameter and associated code to limit the number of ways in a tile
Dear Sirs: I have been trying to build a World map for use in Garmin's Mapsource. I have encountered cases where tiles had too many ways (also polygons and relations) for Mapsource to handle, even though the tile had a sufficiently small numbers for Mapsource. This is because the area had a large number of ways with only begin and end nodes. I can constrain the number of ways, etc., by constraining the number of nodes severely. This causes many more tiles. Above 2025 tiles in a map, Mapsource appears to fail. This code should parallel the code for maxnodes. Randolph J. Herber
Hi Randolph, what do you mean by "ways"? Any line, road with an address, road used for routing? I don't remember if I ever spotted similar problem. On the other side, I think splitter can be better tuned for current mkgmap. I get impression, that splitter is designed to make a good work with simple maps and doesn't account for addresses, routing nodes od DEM data. It would be nice, if there were more options, than maxnodes. For my maps, I split OSM data in two stages. First I prepare artificial data, that I believe represent better actual size of compiled tiles. I extract addresses as points and highways as simple 2-points lines (I do it with modified osmfilter). Then I make splitter to calculate all tiles using as an input following data: - OSM source - 0-2 times extracted addresses - 6-10 times extracted highways - 1-2 times contour lines I believe, that additional points form address account for some NET size, points form highways for additional NOD size and duplicated contour lines for DEM size. It needs some tuning for different areas, but I think I get a bit more uniform sizes of img than with direct splitting. On second stage I use prepared areas.list to split real data. Basically I do something like this: osmfilter.exe data.o5m --keep= --keep-ways=highway=* -o=net.o5m osmfilter.exe data.o5m --keep= --keep-nodes=addr:housenumber=* -o=adr.o5m splitter.jar --stop-after=split --max-nodes=3000000 data.o5m net.o5m net.o5m net.o5m net.o5m net.o5m net.o5m net.o5m net.o5m adr.o5m adr.o5m contour.pbf contour.pbf splitter.jar --split-file=areas.list data.o5m contour.pbf It would be helpful, if I could do it in a single pass. Maybe splitter could filter internally specific nodes - these which would end as routing nodes or address information and add some multiplier when counting these nodes for splitting? Assuming this is easy to implement, otherwise my procedure is good enough. -- Best regards, Andrzej
Dear Sirs: My definition of "ways" was requested. Splitter first scans nodes (points of various kinds), then ways (forming lines, paths and polygons) and finally relations (container polygons for any or all of nodes, ways or relations). I was using "ways" in the same sense that I understand that splitter uses "ways" in its messages. I hope this helps. --maxnodes sets a soft upper limit on the number of nodes in a map tile. I am requesting a similar in concept --maxways to set a soft limit on the number of ways in a map tile. Randolph J. Herber On 5/21/2018 12:27 PM, Andrzej Popowski wrote:
Hi Randolph,
what do you mean by "ways"? Any line, road with an address, road used for routing? I don't remember if I ever spotted similar problem.
On the other side, I think splitter can be better tuned for current mkgmap. I get impression, that splitter is designed to make a good work with simple maps and doesn't account for addresses, routing nodes od DEM data. It would be nice, if there were more options, than maxnodes.
For my maps, I split OSM data in two stages. First I prepare artificial data, that I believe represent better actual size of compiled tiles.
I extract addresses as points and highways as simple 2-points lines (I do it with modified osmfilter). Then I make splitter to calculate all tiles using as an input following data: - OSM source - 0-2 times extracted addresses - 6-10 times extracted highways - 1-2 times contour lines
I believe, that additional points form address account for some NET size, points form highways for additional NOD size and duplicated contour lines for DEM size. It needs some tuning for different areas, but I think I get a bit more uniform sizes of img than with direct splitting.
On second stage I use prepared areas.list to split real data.
Basically I do something like this:
osmfilter.exe data.o5m --keep= --keep-ways=highway=* -o=net.o5m osmfilter.exe data.o5m --keep= --keep-nodes=addr:housenumber=* -o=adr.o5m
splitter.jar --stop-after=split --max-nodes=3000000 data.o5m net.o5m net.o5m net.o5m net.o5m net.o5m net.o5m net.o5m net.o5m adr.o5m adr.o5m contour.pbf contour.pbf
splitter.jar --split-file=areas.list data.o5m contour.pbf
It would be helpful, if I could do it in a single pass. Maybe splitter could filter internally specific nodes - these which would end as routing nodes or address information and add some multiplier when counting these nodes for splitting? Assuming this is easy to implement, otherwise my procedure is good enough.
Hi Randolph, I think it would be a lot of work to change splitter like that, and probably it would require a lot more resources. I'd be interested in the original error message from MapSource and the tile that produces it, maybe the error is somewhere in mkgmap. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Randolph J. Herber <army.bronze.star@gmail.com> Gesendet: Dienstag, 22. Mai 2018 00:45:49 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] I would like a maxways parameter and associated code to limit the number of ways in a tile Dear Sirs: My definition of "ways" was requested. Splitter first scans nodes (points of various kinds), then ways (forming lines, paths and polygons) and finally relations (container polygons for any or all of nodes, ways or relations). I was using "ways" in the same sense that I understand that splitter uses "ways" in its messages. I hope this helps. --maxnodes sets a soft upper limit on the number of nodes in a map tile. I am requesting a similar in concept --maxways to set a soft limit on the number of ways in a map tile. Randolph J. Herber On 5/21/2018 12:27 PM, Andrzej Popowski wrote: Hi Randolph, what do you mean by "ways"? Any line, road with an address, road used for routing? I don't remember if I ever spotted similar problem. On the other side, I think splitter can be better tuned for current mkgmap. I get impression, that splitter is designed to make a good work with simple maps and doesn't account for addresses, routing nodes od DEM data. It would be nice, if there were more options, than maxnodes. For my maps, I split OSM data in two stages. First I prepare artificial data, that I believe represent better actual size of compiled tiles. I extract addresses as points and highways as simple 2-points lines (I do it with modified osmfilter). Then I make splitter to calculate all tiles using as an input following data: - OSM source - 0-2 times extracted addresses - 6-10 times extracted highways - 1-2 times contour lines I believe, that additional points form address account for some NET size, points form highways for additional NOD size and duplicated contour lines for DEM size. It needs some tuning for different areas, but I think I get a bit more uniform sizes of img than with direct splitting. On second stage I use prepared areas.list to split real data. Basically I do something like this: osmfilter.exe data.o5m --keep= --keep-ways=highway=* -o=net.o5m osmfilter.exe data.o5m --keep= --keep-nodes=addr:housenumber=* -o=adr.o5m splitter.jar --stop-after=split --max-nodes=3000000 data.o5m net.o5m net.o5m net.o5m net.o5m net.o5m net.o5m net.o5m net.o5m adr.o5m adr.o5m contour.pbf contour.pbf splitter.jar --split-file=areas.list data.o5m contour.pbf It would be helpful, if I could do it in a single pass. Maybe splitter could filter internally specific nodes - these which would end as routing nodes or address information and add some multiplier when counting these nodes for splitting? Assuming this is easy to implement, otherwise my procedure is good enough.
Hi Andrzej, I think your goal is to create evenly sized tiles, e.g. around 8 MB. Most others are probably happy as long as the calculated tiles don't reach the hard size limits in the img format. The current algo in splitter simply counts the nodes (with an equal weight). Some nodes produce no data in the img files and others produce many bytes ( e.g. a POI ). I think the current algo in splitter assumes that there is a rather constant ratio when you look at enough nodes. This might not be the case, esp. in areas which are dominated by imports or by few mappers using special techniques to map something. My approach to handle this would be to compute a grid similar to the density map created by splitter, but this grid would contain a weigh factor for each grid element. So, when the node count is 12121 for a grid element and the weigh factor is 1.5 the used value for the area calculation would be 1.5 * 12121. The question is how to fill the weigh factor grid. Possible solution: Split planet file using the grid, run mkgmap with the desired options for each grid element, store the size of the img file, and normalize these values to a weigh factor. If one uses e.g. a country extract instead of planet we would use factor 1 for all grid elements not fully covered by the extract. I see these special cases: 1) In areas with rather few OSM nodes (deserts, large woods) the img size mainly depends on bytes for background polygons and even more on elevation / DEM data The actual amount of data depends on the profile of the area. 2) In areas with many small islands we have many bytes for background + sea polygons and the coastlines. Maybe that means we also need a constant value for each grid element. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Andrzej Popowski <popej@poczta.onet.pl> Gesendet: Montag, 21. Mai 2018 19:27:04 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] I would like a maxways parameter and associated code to limit the number of ways in a tile Hi Randolph, what do you mean by "ways"? Any line, road with an address, road used for routing? I don't remember if I ever spotted similar problem. On the other side, I think splitter can be better tuned for current mkgmap. I get impression, that splitter is designed to make a good work with simple maps and doesn't account for addresses, routing nodes od DEM data. It would be nice, if there were more options, than maxnodes. For my maps, I split OSM data in two stages. First I prepare artificial data, that I believe represent better actual size of compiled tiles. I extract addresses as points and highways as simple 2-points lines (I do it with modified osmfilter). Then I make splitter to calculate all tiles using as an input following data: - OSM source - 0-2 times extracted addresses - 6-10 times extracted highways - 1-2 times contour lines I believe, that additional points form address account for some NET size, points form highways for additional NOD size and duplicated contour lines for DEM size. It needs some tuning for different areas, but I think I get a bit more uniform sizes of img than with direct splitting. On second stage I use prepared areas.list to split real data. Basically I do something like this: osmfilter.exe data.o5m --keep= --keep-ways=highway=* -o=net.o5m osmfilter.exe data.o5m --keep= --keep-nodes=addr:housenumber=* -o=adr.o5m splitter.jar --stop-after=split --max-nodes=3000000 data.o5m net.o5m net.o5m net.o5m net.o5m net.o5m net.o5m net.o5m net.o5m adr.o5m adr.o5m contour.pbf contour.pbf splitter.jar --split-file=areas.list data.o5m contour.pbf It would be helpful, if I could do it in a single pass. Maybe splitter could filter internally specific nodes - these which would end as routing nodes or address information and add some multiplier when counting these nodes for splitting? Assuming this is easy to implement, otherwise my procedure is good enough. -- Best regards, Andrzej _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Gerd,
Split planet file using the grid, run mkgmap with the desired options for each grid element, store the size of the img file, and normalize these values to a weigh factor.
My suggestion: add an option to splitter, which shows a path to a compiled map. Then splitter can read all img from this path and get for each img size and area from TRE. This would give some kind of data to build weights for areas. I think this would be easy to use, in most cases we can present to splitter a previous compilation of the map. Or we can do a test compilation with small maxnodes count and use it for a second split. -- Best regards, Andrzej
Hi Andrzej, I think splitter needs a method to weigh the nodes in each grid element. A complete tile would already be too unprecise. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Andrzej Popowski <popej@poczta.onet.pl> Gesendet: Mittwoch, 30. Mai 2018 10:28:05 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] I would like a maxways parameter and associated code to limit the number of ways in a tile Hi Gerd,
Split planet file using the grid, run mkgmap with the desired options for each grid element, store the size of the img file, and normalize these values to a weigh factor.
My suggestion: add an option to splitter, which shows a path to a compiled map. Then splitter can read all img from this path and get for each img size and area from TRE. This would give some kind of data to build weights for areas. I think this would be easy to use, in most cases we can present to splitter a previous compilation of the map. Or we can do a test compilation with small maxnodes count and use it for a second split. -- Best regards, Andrzej _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Gerd,
A complete tile would already be too unprecise.
I agree, but I think it would be better than nothing and it looks like to be quite easy to implement. I like that this approach automatically takes account of used style and map properties like labels, routing or DEM settings. And maybe results will become better after several iteration? -- Best regards, Andrzej
Hi Andrzej, will think about alternatives during my coming cycling tour. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Andrzej Popowski <popej@poczta.onet.pl> Gesendet: Donnerstag, 31. Mai 2018 15:50:18 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] I would like a maxways parameter and associated code to limit the number of ways in a tile Hi Gerd,
A complete tile would already be too unprecise.
I agree, but I think it would be better than nothing and it looks like to be quite easy to implement. I like that this approach automatically takes account of used style and map properties like labels, routing or DEM settings. And maybe results will become better after several iteration? -- Best regards, Andrzej _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
participants (3)
-
Andrzej Popowski -
Gerd Petermann -
Randolph J. Herber