2009/8/15 Chris Miller <chris.miller@kbcfp.com>:

Hi Paul,

This is because the .osm file you are splitting has a node with no 'lat' attribute. As far as I'm aware this shouldn't happen; something is probably wrong with your .osm file? I downloaded the europe.osm file a few hours after you posted your message and it processes without a problem. My europe.osm file is 28,626,280,448 bytes in size.

My file was about 200KB smaller...

Anyway I've put in a check for this since in this case the change is simple and it doesn't have much affect on performance. The splitter will now output details of the problem, ignore the node and carry on with the split. Generally speaking though it's not such a good idea to put too much validation of the XML into the splitter because it will just complicate the code and slow things down. I guess if we any further problems like this we'll have to decide what's best on a case-by-case basis.

I think that putting additional validation in the splitter for the case of reporting and ignoring bad data is useful. I can understand the problem you mentioned about performance -- it is almost always the case of making compromise between robustness and speed. The alternative would be creating additional app that would filter the bad data ("bad" as defined in splitter, not necessarily OSM) and write valid xml to file for processing with splitter. It is almost always the case you cannot assume that data coming from outside of your private framework is valid. The use case I could imagine would be processing data with splitter, if it fails, then preprocessing it with "cleaner" application and starting splitter once again on the cleaned data. It is just an idea. If we come across significant speed reduction because of necessary robustness then splitting validation/cleaning and processing might be a good way to go. Thanks for the fix :) Paul -- Don't take life too seriously; you will never get out of it alive. -- Elbert Hubbard

That doesn't really buy us much unfortunately, given the way the parsing works the checks need to be scattered about in various places (otherwise a separate validation pass would be required, taking even longer. I think a separate validation step is outside the scope of the splitter). Experience in dealing with huge volumes of unreliable external data feeds at my current job has taught me there's only so much you can do to tollerate and recover from bad input. If the source data is corrupt/unexpected then it's often better to fail-fast and let a human deal with the problem rather than make assumptions about what is wrong with the data and carry on since that can just introduce further problems downstream. And something else I've learned is that no matter how many checks you have in place, something will always slip through the cracks anyway! That aside, I don't think we have anything to worry about here. The splitter is already tollerant of unrecognised XML tags and attributes and for the most part this is all we really need. As I said in a previous post, any further problems that we hit of this nature are best dealt with on a case-by-case basis. Chris F> cant you put in the check as an option. With the -check_for_bad_date F> all data will be past in the splitter for a growing number of know F> xml bugs and without the check .. you check will be done an the app F> will be faster ? F> F> lg Dirk

JO> Is there a XML schema definition for the 0.6 XML dumps? A bit of JO> googling showed one for the 0.5 api but I didn't see anything for JO> 0.6. JO> If we had an XML schema definition a separate XML parser/validator JO> could be used to check inputs. I can't say I've looked for one though I'd hope there was a schema/DTD somewhere. As far as the splitter's concerned it's only interested in a limited subset of the information anyway so strict conformance to the schema isn't essential. It certainly wouldn't do any harm for people to validate their XML one way or another before passing it to the splitter though.