If you like I could add a parameter to the splitter that generated the KML file automatically so there's no need for the additional script. Maybe even http POST the resultant KML to your server and open the appropriate page in a browser for the user to view, though I suppose that would be open to abuse or loading down your server. Thoughts? What is the problem you're having with the red tiles? I think I saw you mention something about them elsewhere but I can't find a reference to it. Chris L> Some visualisation is done here: http;//garmin.na1400.info It uses L> the convertarealist script. L> L> I've thought about -and started working on it- making the areas L> editable L> through that site as well. But haven't got around completing the L> implementation. This could be made in such a way that you could point L> it L> to a local file perhaps so that I'm not the only one who's able to L> enjoy such functionality. L> L> If there is interest in me implementing it, then I can move it up the L> priority list. It would be very usefull for me as well to finally L> eliminate those red tiles... L>

L> Chris Miller wrote: L>

...

If you like I could add a parameter to the splitter that generated the KML file automatically so there's no need for the additional script.

L> That would be great to have. Could the KML file then also be used as L> an input to splitter?

OK I'll add KML output onto the todo list... Yes a KML file could be used as input to the splitter too, however that's a lot more work since KML files can be arbitrarily complex. I suspect the splitter would have to perform quite a lot of validation to ensure the areas described within were suitably rectangular, aligned etc, otherwise there'd be a lot of questions from people wondering why something went wrong with the splitter, mkgmap, or on their device. Of course there's no validation to speak of with areas.list files currently either, but if people are editing that by hand they generally be expected to have a better idea about the kinds of problems they might run in to. Having said that, I'm open to trying to deal with what's been given in a KML file and split accordingly, without regard for how sane the boundaries are. This would probably be provided as an experiemental/beta feature with suitable warnings about it being at your own risk. Over time as we get a better feel for how people are using it and what problems arise, we could look at providing better support. L> Most of the stuff should be done client side (OL) and the tiles L> originate from the OSM server, so there won't be much that bothers my L> server. But I don't know if you're willing to add a link to a page on L> a not-affiliated (not directly anyway to the splitter tool) server L> from within the code. It seems prone to failure some day ;) A link on L> the Mkgmap/splitter wiki page seems more appropriate to me. Steve's probably the best person to comment on linking to external sites. But wrt the chance of the url changing/breaking in the future, I was intending to make it configurable anyway. My idea was to be able to specify something like --launchKML, and a browser window would be launched once the subdivision had taken place, showing the user something like this: http://maps.google.com/maps?f=q&source=s_q&hl=en&geocode=&q=http:%2F%2Fwww.m... If you can think of a way to do that without having to serve the KML file from a server somewhere, I'd be interested to hear your ideas. L> the tiles rendered with Mkgmap. The red tiles in Denmark are caused L> by all the house numbers which cause the guesstimate mechanism L> (--max-nodes) of splitter to be fooled in thinking that Mkgmap could L> produce working maps for those areas. Splitter should be more L> intelligent about determining the max size of an area, ultimately. Ahh yes, that rings a bell now that you mention it. I'm open to suggestions on how this could be improved, unfortunately I don't have enough experience with the limits yet to know what might be the best approach. At this stage I'm thinking about building up a low-res "density map" (for lack of a better term), which would divide the world into MxN areas (probably with their boundaries aligned to suit the map resolution) and then tally up the number of nodes/ways/relations in each area. Those aggregations could then be used to decide how to group areas together into larger tiles. I can see a lot of advantages to this, for example: - The nodes/ways/rels could contribute different (easily customisable) weightings, so we can experiment to find the best balance. - The algorithm that decides how to group the areas into tiles would be easy to adjust and could even be made pluggable. - High density areas could be identified and placed in the middle of a tile, rather than cut through the middle as is the tendency with the current splitter (see eg London). - The density map could be paged out to disk and the subdivision performed on a portion at a time to save memory without much of a performance hit. - The density map could be handed off to a tile editor tool so users could take tile complexity into account. I haven't yet sat down and figured out how it will affect performance and memory of the current splitter, but I doubt it'll be a deal breaker and will probably even have some benefits. What do people think (assuming you can even understand my admittedly poor description!)? L> But, as things are now, I'm manually editing the areas.list to get L> rid of those failed (red) tiles. Unfortunately, doing the splitting L> of the bboxes by hand is cumbersome work. I also expect mass imports L> and just sheer manual labour will cause many red tiles to appear in L> the future. These are the reasons why I was working towards a L> webbased UI that can do the splitting for me with a simple mouse L> click. How about splitting those red areas in half again (taking care with the alignment), rather than just getting rid of the areas completely? L> There are also quite a few people who would like to be able to split L> tiles because the tiles would otherwise include bits of data from L> neighboring countries e.g. across large bodies of water (look at e.g. L> France/England/Spain, Zweden/Poland etc). I have no problem with this L> type of splitting result, but some apparently do. Makes sense. I notice the real Garmin maps have tiles that must have at least partially been crafted by hand. Eg London is covered very nicely by a single tile, various tiles have edges along national borders, bodies of water are tiled appropriately and so on. I guess no algorithm's ever going to be able to match that :( Chris

CF> This sounds like an excellent idea - the density map would ideally CF> be a planet map and live in server-land getting updated as OSM gets CF> updated. Users could then download the latest map (or a portion of CF> it) when they want to do some manual editing of areas.list. CF> CF> Exactly: the ideal process (imho) would work as follows: CF> 1) User opens TileGUI and specifies the region they're interested in CF> (preferably, by selecting a rectangular area on a map), or opens a CF> pre-specified area that they have previously saved CF> 2) TileGUI downloads the appropriate portion of the current density CF> map and automatically generates a set of suggested tiles based on CF> data density CF> 3) User manually edits these tiles (e.g. to avoid overlapping CF> countries or tiles with huge areas of empty sea, per his/her CF> preference) CF> 4a) TileGUI alerts user if they create a tile which is likely to be CF> too large CF> 4b) TileGUI alerts user if the user's tiles do not completely cover CF> the region specified in 1) CF> 5) User presses button and tileGUI spits out an areas.list file Sounds about right. The only thing I'm not sure of is whether it'll be viable to serve the density map from a server - users may need to generate it themselves, particularly if they're using relatively high target resolutions. My current thinking is that a full planet density map (uncompressed) would require roughly 2048*2048*8 bytes = 32MB at the default resolution of 13, but each extra step in resolution increases the space required by a factor of 4 (ie resolution 10 = 2GB). It should however be possible to reduce that requirement using some sort of sparse matrix (since the oceans and remote areas are empty), quadtree partitioning, or other compression techniques. I'll give it some thought. While we're on the topic of improving the tile generation, how much work would be required to add support for non-rectangular tiles in mkgmap? Chris

On Tue, 25 Aug 2009 15:18:16 +0000 (UTC), Chris Miller wrote:

L> Most of the stuff should be done client side (OL) and the tiles L> originate from the OSM server, so there won't be much that bothers my L> server. But I don't know if you're willing to add a link to a page on L> a not-affiliated (not directly anyway to the splitter tool) server L> from within the code. It seems prone to failure some day ;) A link on L> the Mkgmap/splitter wiki page seems more appropriate to me.

Steve's probably the best person to comment on linking to external sites.

But wrt the chance of the url changing/breaking in the future, I was intending to make it configurable anyway. My idea was to be able to specify something

like --launchKML, and a browser window would be launched once the subdivision had taken place, showing the user something like this:

http://maps.google.com/maps?f=q&source=s_q&hl=en&geocode=&q=http:%2F%2Fwww.m...

Sorry for not responding earlier, I missed this one. I don't see the big advantage of having splitter start the browser with a KML argument. It is a simple task for a person to launch Google (or another) site and pass the kml file manually. So I think you can spare yourself the trouble of implementing this.

If you can think of a way to do that without having to serve the KML file

from a server somewhere, I'd be interested to hear your ideas.

I can't think of any other way either.

L> the tiles rendered with Mkgmap. The red tiles in Denmark are caused L> by all the house numbers which cause the guesstimate mechanism L> (--max-nodes) of splitter to be fooled in thinking that Mkgmap could L> produce working maps for those areas. Splitter should be more L> intelligent about determining the max size of an area, ultimately.

Ahh yes, that rings a bell now that you mention it. I'm open to

suggestions

on how this could be improved, unfortunately I don't have enough

experience

with the limits yet to know what might be the best approach. At this

stage

I'm thinking about building up a low-res "density map" (for lack of a better term), which would divide the world into MxN areas (probably with their boundaries aligned to suit the map resolution) and then tally up the number of nodes/ways/relations in each area. Those aggregations could then be used to decide how to group areas together into larger tiles. I can see a lot of advantages to this, for example:

- The nodes/ways/rels could contribute different (easily customisable) weightings, so we can experiment to find the best balance. - The algorithm that decides how to group the areas into tiles would be easy to adjust and could even be made pluggable. - High density areas could be identified and placed in the middle of a tile, rather than cut through the middle as is the tendency with the current splitter (see eg London). - The density map could be paged out to disk and the subdivision performed on a portion at a time to save memory without much of a performance hit. - The density map could be handed off to a tile editor tool so users could take tile complexity into account.

This sound like a lot of work for you and for algorithms while humans can perform this task with ease. So perhaps the current crude (I don't mean in an insulting way!) method might just be fine for the time being and people can fix these kind of problems easily by hand. Especially when working on areas.list files can be simplified through a (webbased perhaps) tool.

I haven't yet sat down and figured out how it will affect performance and

memory of the current splitter, but I doubt it'll be a deal breaker and will probably even have some benefits.

What do people think (assuming you can even understand my admittedly poor

description!)?

L> But, as things are now, I'm manually editing the areas.list to get L> rid of those failed (red) tiles. Unfortunately, doing the splitting L> of the bboxes by hand is cumbersome work. I also expect mass imports L> and just sheer manual labour will cause many red tiles to appear in L> the future. These are the reasons why I was working towards a L> webbased UI that can do the splitting for me with a simple mouse L> click.

How about splitting those red areas in half again (taking care with the alignment), rather than just getting rid of the areas completely?

That is what I'm doing manually using an Excel file to do the dividing math, but without the alignment rules (which I heard of not so long ago). But I hate doing that kind of work so I just haven't done it for every problem area yet (Not true actually: I've split those red tiles in Denmark twice already but it still wasn't enough. Then I gave up for the time being). Or do you mean to have splitter split those problem areas again by feeding *only* that problem area and a very low --max-nodes parameter? Because that is a solution that I did not think about before. I just run splitter once a week on 2 halves of the planet and if tiles fail then that's it...

L> There are also quite a few people who would like to be able to split L> tiles because the tiles would otherwise include bits of data from L> neighboring countries e.g. across large bodies of water (look at e.g. L> France/England/Spain, Zweden/Poland etc). I have no problem with this L> type of splitting result, but some apparently do.

Makes sense. I notice the real Garmin maps have tiles that must have at least partially been crafted by hand. Eg London is covered very nicely by a single tile, various tiles have edges along national borders, bodies of water

are

tiled appropriately and so on. I guess no algorithm's ever going to be able

to match that :(

I guess pure math algorithms would have a hard time, but we can feed them the required information because we have the national border (and city) polygons, so if Mkgmap is going to support polygons instead of bboxes then there is a way... In the mean time when there is a GUI then people can optimise the areas.list quite easily. But, so far, I reckoned that having maps available is far better then beautifully crafted tile coordinates but no maps :)

On Fri, 28 Aug 2009 17:40:54 +0000 (UTC), Chris Miller wrote:

L> This sound like a lot of work for you and for algorithms while humans L> can perform this task with ease. So perhaps the current crude (I L> don't mean in an insulting way!) method might just be fine for the L> time being and people can fix these kind of problems easily by hand. L> Especially when working on areas.list files can be simplified through L> a (webbased perhaps) tool.

I agree with everything you say here, and a user tool will definitely give the best and most flexible results. I'd still like to try and improve the

automated split though for a couple of reasons. I think plenty of people (based on a sample size of one - myself ;)) aren't too concerned about where the borders are, but would appreciate tiles that hold as much information

as possible without breaking mkgmap or introducing other problems. Currently the tiles vary a lot in what they contain, plus figuring out the right --max-nodes threshold involves a bit of trial and error. The better the automated split, the less post-processing that is required. Plus, it's an interesting challenge to work on :)

Agreed, it would be totally awesome to have a density map thing and 1st time correct tile size determination.

...

...

How about splitting those red areas in half again (taking care with the

alignment), rather than just getting rid of the areas completely?

L> Or do you mean to have splitter split those problem areas again by L> feeding *only* that problem area and a very low --max-nodes L> parameter? Because that is a solution that I did not think about L> before. I just run splitter once a week on 2 halves of the planet and L> if tiles fail then that's it...

That would certainly do what you want - it would give you a more balanced

split than Excel because it takes node distribution into account, plus it

would handle the alignment problem for you. It would even be possible to automate this though I'm not sure it's worth the extra complexity of the splitter in terms of both code and command line parameters.

I'm building the maps using a CLI php script so I can use that to call splitter again to redo the failed areas. Shouldn't be a big deal, it's just something that I never though about.

Great, thanks a lot! Can I be so bold and grab go for the jackpot and ask if you plan on making the reverse working as well (feed KML into Splitter)? Personally, I don't mind if there aren't many sanity check on reading the KML. If things bark because of an ill formed KML then that's the users error... On Wed, 26 Aug 2009 20:28:35 +0000 (UTC), Chris Miller <chris.miller@kbcfp.com> wrote:

...

...

If you like I could add a parameter to the splitter that generated the KML file automatically so there's no need for the additional script.

L> That would be great to have.

...and now you have it :) I've just checked in an update that gives you a --write-kml=<filename> parameter. Pretty much does what you'd expect, writes out a KML file to the specified filename in roughly the same format as that

python script. The only real difference is I've named the areas "Map 63240001" etc, instead of "Area 1", "Area 2", ... Hopefully you'll agree that's a bit more useful.

Chris

_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

On Wed, 26 Aug 2009 21:11:31 +0000 (UTC), Chris Miller wrote:

If I limit it to only handling KML that's in the same format as the output then it's not too difficult to implement so yes I'll see what I can do.

Very fine by me :)

One thing with the KML the splitter generates - I wasn't sure if I should

call the areas in the KML I write out "Map 63240001", "Area 63240001", "Tile 63240001" or just "63240001". Does anyone have any strong preferences/opinions? Regardless of which of those is chosen, when reading in a KML file the splitter should look for an 8-digit number in the KML name and use that as the area's map ID. I'm not sure if that should be in preference to any supplied --mapid parameter though, or if the --mapid parameter should override any IDs found

in the KML names instead. Thoughts?

Yes, thoughts all over. I seem to remember a haiku that was brought up just before SOTM2 and was a mantra during last SOTM which was something like "Mappas mappas mappas". :p What I mean is that it might be best to just stick to the number and not include any language. I'm fine with EN or EN_US but perhaps others would rather have it something else. You can't do much wrong sticking to numbers only...

If anyone has any more suggestions/ideas in this area it would be good to

hear them. I'll hopefully find time to start implementing this on the weekend.

I will commit myself to making a webinterface for editing areas.kml files now that you have addressed my wishes and comments so swiftly :) Another nice way of putting OpenLayers to work and perhaps make a useful tool for others as well. Thank you very much Chris for your work!

On Fri, 28 Aug 2009 17:50:22 +0000 (UTC), Chris Miller wrote:

L> I will commit myself to making a webinterface for editing areas.kml L> files now that you have addressed my wishes and comments so swiftly L> :)

Haha I haven't implemented the KML parsing yet, don't get too excited ;)

I'm counting on you will :p

L> Thank you very much Chris for your work!

No problem at all. Thanks to you for your feedback and of course for all the effort that's gone into your website. Your site was one of the reasons I got interested in this project in the first place.

Cool. I sounds like this project is almost a chicken or egg situation. Perhaps we should be glad that it exists at all despite the odds. /me is sentimental after having a beer, haha ;)

Chris Miller schrieb:

...and now you have it :) I've just checked in an update that gives you a --write-kml=<filename> parameter. Pretty much does what you'd expect, writes out a KML file to the specified filename in roughly the same format as that python script. The only real difference is I've named the areas "Map 63240001" etc, instead of "Area 1", "Area 2", ... Hopefully you'll agree that's a bit more useful.

Hi, Don't works for me in Google Earth. The areas are not shown as squares, but as funny shapes spread around the world. <coordinates> 3,295898,45,791016 Looks like a decimal point vs. comma problem. Question for manually adjusting the values in areas.list: 1: 2134016,153600 to 2228224,256000 # : 45.791016,3.295898 to 47.812500,5.493164 The values (eg 2134016) have to be a multiple of 1024 ? Chris

C> Hi, C> Don't works for me in Google Earth. The areas are not shown as C> squares, C> but as funny shapes spread around the world. C> <coordinates> C> 3,295898,45,791016 C> Looks like a decimal point vs. comma problem. Thanks for the report. As Mark points out, it's because the code is using the default locale rather than forcing it to English. I'll check in a fix in the next day or so. C> Question for manually adjusting the values in areas.list: C> C> 1: 2134016,153600 to 2228224,256000 C> # : 45.791016,3.295898 to 47.812500,5.493164 C> The values (eg 2134016) have to be a multiple of 1024 ? The alignment depends on the value you specify for --resolution. You need to make the borders multiples of 2 ^ (24 - resolution), and the tiles themselves must be multiples of 2 ^ (25 - resolution). So for the default resolution of 13, tile boundaries must be multiples of 2048 (0x800 hex) and the tiles themselves must have widths and heights that are multiples of 4096 (0x1000 hex). When you run the splitter, you'll see a message along the lines of the following that tells you the alignment you need: Map is being split for resolution 13: - area boundaries are aligned to 0x800 map units - areas are multiples of 0x1000 map units wide and high Hope that helps Chris

Hmm, this one had me stumped for a bit but I think I know what it is. The "mapid = 1" is the giveaway... It looks like the code that parses areas.list currently looks for an 8 digit number - if it doesn't find one it'll fail with the exception you've shown. Because you specified mapid = 1, the ID was written out as a single digit (rather than "00000001"). Try editing the mapids in the file to give the maps 8 digit numbers, it will then hopefully work OK. I should probably fix the splitter so it zero-pads out the map ID when writing the areas.list file. Though I don't know much about the importance of the map ID - do mkgmap, mapsource, devices etc still work OK if the ID is less than 10000000? Chris C> And the --split-file=<file> option is also not working for me on XP. C> C> Maybe also a locale problem? C> C> mapid = 1 C> split-file = newareas.list C> max-nodes = 1200000 C> Exception in thread "main" java.lang.IllegalStateException: No match C> found C> at java.util.regex.Matcher.group(Unknown Source) C> at uk.me.parabola.splitter.AreaList.read(AreaList.java:163) C> at uk.me.parabola.splitter.Main.readArgs(Main.java:203) C> at uk.me.parabola.splitter.Main.split(Main.java:113) C> at uk.me.parabola.splitter.Main.main(Main.java:99) C> Drücken Sie eine beliebige Taste . . . C> Command was: C> C> java -Xmx%heap% -jar ..\tools\splitter.jar --mapid=1 C> --split-file=newareas.list --max-nodes=%max_nodes% C> ..\osmdata\data.osm C> C> newareas.list is a unmodified copy of areas.list. C> C> Chris

Chris Miller schrieb:

C> And the --split-file=<file> option is also not working for me

Hmm, this one had me stumped for a bit but I think I know what it is. The "mapid = 1" is the giveaway... It looks like the code that parses areas.list currently looks for an 8 digit number - if it doesn't find one it'll fail with the exception you've shown. Because you specified mapid = 1, the ID was written out as a single digit (rather than "00000001").

Try editing the mapids in the file to give the maps 8 digit numbers, it will then hopefully work OK. I should probably fix the splitter so it zero-pads out the map ID when writing the areas.list file. Though I don't know much about the importance of the map ID - do mkgmap, mapsource, devices etc still work OK if the ID is less than 10000000?

Don't know. For mkgmap I use a 8 digit mapid, only for the splitter I used the "1". Chris