Out of heap space: Too many tags?
I just tried to compile a relatively small tile and got an out of memory error from Java! Normally when the tiles get too large, there are error messages from mkgmap. So I investigated a little bit and found out that the tile in question contained a huge number of <tag>s. The number of tags is ten times the number of nodes in the file. Is it possible that all these tags filled up my memory? Here's a list of the most frequently occuring tags: k='kms:parish_name' k='kms:parish_no' k='kms:city_name' k='kms:county_name' k='kms:county_no' k='kms:house_no' k='kms:last_updated' k='kms:municipality_name' k='kms:municipality_no' k='kms:street_name' k='kms:street_no' k='kms:zip_name' k='kms:zip_no' k='addr:housenumber' k='addr:street' k='addr:city' k='addr:postcode' k='source' And here's the Exception: Exception in thread "main" java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOfRange(Arrays.java:3209) at java.lang.String.<init>(String.java:216) at com.sun.org.apache.xerces.internal.xni.XMLString.toString(XMLString.j ava:185) at com.sun.org.apache.xerces.internal.util.XMLAttributesImpl.getValue(XM LAttributesImpl.java:557) at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser$Attribut esProxy.getValue(AbstractSAXParser.java:2323) at uk.me.parabola.mkgmap.reader.osm.xml.Osm5XmlHandler.startInNode(Osm5X mlHandler.java:178) at uk.me.parabola.mkgmap.reader.osm.xml.Osm5XmlHandler.startElement(Osm5 XmlHandler.java:136) at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.startEle ment(AbstractSAXParser.java:501) at com.sun.org.apache.xerces.internal.parsers.AbstractXMLDocumentParser. emptyElement(AbstractXMLDocumentParser.java:179) at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImp l.scanStartElement(XMLDocumentFragmentScannerImpl.java:1339) at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImp l$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2747) at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(X MLDocumentScannerImpl.java:648) at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImp l.scanDocument(XMLDocumentFragmentScannerImpl.java:510) at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(X ML11Configuration.java:807) at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(X ML11Configuration.java:737) at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser. java:107) at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(Ab stractSAXParser.java:1205) at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.p arse(SAXParserImpl.java:522) at javax.xml.parsers.SAXParser.parse(SAXParser.java:395) at javax.xml.parsers.SAXParser.parse(SAXParser.java:198) at uk.me.parabola.mkgmap.reader.osm.xml.Osm5MapDataSource.load(Osm5MapDa taSource.java:78) at uk.me.parabola.mkgmap.main.MapMaker.loadFromFile(MapMaker.java:134) at uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:51) at uk.me.parabola.mkgmap.main.Main.processFilename(Main.java:150) at uk.me.parabola.mkgmap.CommandArgs$Filename.processArg(CommandArgs.jav a:329) at uk.me.parabola.mkgmap.CommandArgs.readArgs(CommandArgs.java:119) at uk.me.parabola.mkgmap.main.Main.main(Main.java:91) -Wolfgang
On Tue, Feb 24, 2009 at 10:17:47PM +0100, Wolfgang v. Hansen wrote:
I just tried to compile a relatively small tile and got an out of memory error from Java! Normally when the tiles get too large, there are error messages from mkgmap. So I investigated a little bit and found out that the tile in question contained a huge number of <tag>s. The number of tags is ten times the number of nodes in the file.
The typical ratio in the UK seems to be about the same number or slightly more tags as nodes. So ten times is a lot. In places where there have been data imports it is probably more normal to have large numbers of tags though. Normally the number of nodes determines the amount of memory needed, but if there are so many more tags then they will indeed require even more memory. If there are tags on each node that will also bloat the memory required dramatically too. ..Steve
On Tue, 24 Feb 2009, Steve Ratcliffe wrote:
On Tue, Feb 24, 2009 at 10:17:47PM +0100, Wolfgang v. Hansen wrote:
I just tried to compile a relatively small tile and got an out of memory error from Java! [...] The number of tags is ten times the number of nodes in the file.
The typical ratio in the UK seems to be about the same number or slightly more tags as nodes. So ten times is a lot. In places where there have been data imports it is probably more normal to have large numbers of tags though.
Normally the number of nodes determines the amount of memory needed, but if there are so many more tags then they will indeed require even more memory.
So I was right. Obviously these were all addresses so it might well be possible that tags might dominate the planet.osm in the future. In addition, as tags are the free-style way to add information, many of them will not be recognized by a particular software such as mkgmap. Is it feasible to filter the tags while reading the file? One could ingnore unknown tags entirely and convert the known ones to a (compressed) internal representation. -Wolfgang
participants (2)
-
Steve Ratcliffe -
Wolfgang v. Hansen