On Thu, Sep 16, 2010 at 12:33 PM, WanMil <wmgcnfg@web.de> wrote:

Hi Steve,

that was a quick implementation!

I haven't tried so far the new binary format. I haven't found a download for the osmprotobuf.jar. Is there any? Download links should be added to the documentation and the website if the changes are merged back.

I am still in release candidate status as far as jar is concerned. At present, I haven't set up hosting for it anyplace. The best place to get it right now is from osmosis trunk ('osmbin.jar'), or build it from my git repository on github.

I had some thoughts if the splitter is still needed with this new format. The website of the new protocol tells me that the operation getNodesbyBoundingBox(north,south,east,west) is supported. So it should be possible to start mkgmap with the bbox definition of all tiles (like the areas.list file of the tile splitter) and a planet dump file?  The splitter work could be reduced to generate a reasonable areas.list file. Are there any objections on this?

This could be done, but the data has to be geographically sorted. I've described how to do that, but the code implementing it hasn't been written. I've got a question though, why can't mkgmap generate different areas in parallel? It seems like it should be possible and it would make it a lot faster to render maps. Scott

Ok, that means we cannot use the common planet dumps for this and a separate step (geographical sorting) is needed which maybe eats up the advantage to skip the tile splitter step. Just keep the idea in mind and give it a try when the geographical sorting is available.

I designed the format to support that feature, but I have no idea when, or if, I'll have time to program it. One advantage is that the sorting process only has to be done once.

...

I've got a question though, why can't mkgmap generate different areas in parallel? It seems like it should be possible and it would make it a lot faster to render maps.

mkgmap already works in parallel if you specifiy multiple osm files which are usually generated by the tile splitter (see parameter --max-jobs). The advantage of my idea would be to skip the tile splitter step.

Are you sure? If I try it on a quad-core hyperthreading CPU, I only see about 2-3 cores used and 5 idle. (Invoked with '-c template.args and --max-jobs=10') There's also this code in MapMaker.java: /** * Load up from the file. It is not necessary for the map reader to completely * read the whole file in at once, it could pull in map-features as needed. */ private LoadableMapDataSource loadFromFile(CommandArgs args, String name) throws FileNotFoundException, FormatException { LoadableMapDataSource src; // work around non-reentrancy of GType priority stuff // by serialising the map reading synchronized(MapMaker.class) { src = MapReader.createMapReader(name); src.config(args.getProperties()); log.info("Started loading " + name); src.load(name); log.info("Finished loading " + name); } return src; }

On Thu, Sep 16, 2010 at 01:34:20PM -0500, Scott Crosby wrote:

I've got a question though, why can't mkgmap generate different areas in parallel? It seems like it should be possible and it would make it a lot faster to render maps.

I thought that it already does, when you specify the parameter max-jobs (it defaults to the number of processor cores). Another question that I have been asking is, why cannot mkgmap generate multiple maps (map layers) from a single map tile. Perhaps the binary format reduces the parsing overhead, but there still must be some. Here is one use case: Have several map layers built from route relations, which can be enabled or disabled on the device as needed. (When you want to see bus routes, you usually don't want to see hiking or bicycle routes and vice versa.) This would need a minor change to the style language (say, a directive identifying the layer where to generate the object) and some mapping between layer IDs in the style language and filenames on the command line. Marko