Binary OSM file support?
Now that Osmosis is getting support for a binary format, is there anyone working on getting binary support into the splitter and mkgmap? I could take a look, but I'm not much of a Java hacker... -- Jeff Ollie
Hi Jeff, Scott Crosby posted a message to this list back in June asking for comment on his binary format. My thoughts at the time were that it looked rather promising and adding support for it to the splitter would be a good idea, though I was reluctant to use it other than internally if the file format wasn't relatively stable because of the maintenance problems that it might introduce. I haven't been following the discussions on the Osmosis side (and indeed, have had hardly any time to dedicate to the splitter lately either), but it sounds like this file format is now at release candidate status in Osmosis? If so I think that's great news! I can't speak for the other developers on splitter or mkgmap, but personally at least I'd really like to see a revamped splitter and mkgmap that can work with the format directly - there are clearly some huge benefits to be had from going that route. Perhaps Scott, Steve and co can chime in with their thoughts on the topic and where they see things heading from here? Chris
Now that Osmosis is getting support for a binary format, is there anyone working on getting binary support into the splitter and mkgmap? I could take a look, but I'm not much of a Java hacker...
some huge benefits to be had from going that route. Perhaps Scott, Steve and co can chime in with their thoughts on the topic and where they see things heading from here?
I am entirely in favour of supporting the format. Most importantly, it seems to me, in splitter, but in mkgmap too. I'll have more time to spend on development for a while too so, so I would like to get it added soon. ..Steve
On Wed, Sep 8, 2010 at 2:27 PM, Chris Miller <chris_overseas@hotmail.com> wrote:
Hi Jeff,
Scott Crosby posted a message to this list back in June asking for comment on his binary format. My thoughts at the time were that it looked rather promising and adding support for it to the splitter would be a good idea, though I was reluctant to use it other than internally if the file format wasn't relatively stable because of the maintenance problems that it might introduce.
I haven't been following the discussions on the Osmosis side (and indeed, have had hardly any time to dedicate to the splitter lately either), but it sounds like this file format is now at release candidate status in Osmosis?
The format is stable, but I want to release one more RC, with a full validation before I declare it stable. I expect no incompatible changes.
If so I think that's great news! I can't speak for the other developers on splitter or mkgmap, but personally at least I'd really like to see a revamped splitter and mkgmap that can work with the format directly - there are clearly some huge benefits to be had from going that route. Perhaps Scott, Steve and co can chime in with their thoughts on the topic and where they see things heading from here?
I have a version of the splitter that reads the binary file format sitting in my local git repository, last used a few days ago. I've been keeping that branch up-to-date with respect to the changes I've made to the binary format. However, that branch split off last June and longer applies cleanly due to the various input handling changes that have happened in the meantime. My repo has a bunch of other stuff that needs to be cleaned up, rebased to trunk, tested, and submitted. I'm not sure when I'll have time, but my repo is public on http://github.com/scrosby/OSM-splitter Scott
Here's a quick rebasing of the patches in Scott's splitter repository to the current splitter trunk. My Java dev system is down for some upgrades at the moment so I haven't even compiled these yet but hopefully I didn't do too bad of a job of resolving the conflicts.
From: Scott Crosby <scrosby@cs.rice.edu> --- src/uk/me/parabola/splitter/BinaryMapLoader.java | 160 ----------- .../me/parabola/splitter/CachingMapProcessor.java | 242 ----------------- src/uk/me/parabola/splitter/Main.java | 103 +------- .../splitter/disk/AbstractStoreReader.java | 75 ------ .../splitter/disk/AbstractStoreWriter.java | 58 ---- .../me/parabola/splitter/disk/CacheVerifier.java | 183 ------------- .../me/parabola/splitter/disk/KeyLookupReader.java | 58 ---- .../me/parabola/splitter/disk/KeyLookupWriter.java | 50 ---- .../splitter/disk/LengthPrefixInputStream.java | 278 -------------------- .../splitter/disk/LengthPrefixOutputStream.java | 194 -------------- src/uk/me/parabola/splitter/disk/Member.java | 40 --- src/uk/me/parabola/splitter/disk/MemberType.java | 21 -- .../me/parabola/splitter/disk/NodeStoreReader.java | 48 ---- .../me/parabola/splitter/disk/NodeStoreWriter.java | 37 --- .../splitter/disk/RelationStoreReader.java | 54 ---- .../splitter/disk/RelationStoreWriter.java | 53 ---- .../me/parabola/splitter/disk/WayStoreReader.java | 54 ---- .../me/parabola/splitter/disk/WayStoreWriter.java | 43 --- .../me/parabola/splitter/disk/TestKeyLookups.java | 51 ---- .../splitter/disk/TestLengthPrefixStreams.java | 87 ------ test/uk/me/parabola/splitter/disk/TestStores.java | 216 --------------- 21 files changed, 12 insertions(+), 2093 deletions(-) delete mode 100644 src/uk/me/parabola/splitter/BinaryMapLoader.java delete mode 100644 src/uk/me/parabola/splitter/CachingMapProcessor.java delete mode 100644 src/uk/me/parabola/splitter/disk/AbstractStoreReader.java delete mode 100644 src/uk/me/parabola/splitter/disk/AbstractStoreWriter.java delete mode 100644 src/uk/me/parabola/splitter/disk/CacheVerifier.java delete mode 100644 src/uk/me/parabola/splitter/disk/KeyLookupReader.java delete mode 100644 src/uk/me/parabola/splitter/disk/KeyLookupWriter.java delete mode 100644 src/uk/me/parabola/splitter/disk/LengthPrefixInputStream.java delete mode 100644 src/uk/me/parabola/splitter/disk/LengthPrefixOutputStream.java delete mode 100644 src/uk/me/parabola/splitter/disk/Member.java delete mode 100644 src/uk/me/parabola/splitter/disk/MemberType.java delete mode 100644 src/uk/me/parabola/splitter/disk/NodeStoreReader.java delete mode 100644 src/uk/me/parabola/splitter/disk/NodeStoreWriter.java delete mode 100644 src/uk/me/parabola/splitter/disk/RelationStoreReader.java delete mode 100644 src/uk/me/parabola/splitter/disk/RelationStoreWriter.java delete mode 100644 src/uk/me/parabola/splitter/disk/WayStoreReader.java delete mode 100644 src/uk/me/parabola/splitter/disk/WayStoreWriter.java delete mode 100644 test/uk/me/parabola/splitter/disk/TestKeyLookups.java delete mode 100644 test/uk/me/parabola/splitter/disk/TestLengthPrefixStreams.java delete mode 100644 test/uk/me/parabola/splitter/disk/TestStores.java diff --git a/src/uk/me/parabola/splitter/BinaryMapLoader.java b/src/uk/me/parabola/splitter/BinaryMapLoader.java deleted file mode 100644 index c7b5717..0000000 --- a/src/uk/me/parabola/splitter/BinaryMapLoader.java +++ /dev/null @@ -1,160 +0,0 @@ -/* - * Copyright (c) 2009. - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 3 as - * published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - */ -package uk.me.parabola.splitter; - -import java.io.File; -import java.io.IOException; -import java.util.Map; - -import uk.me.parabola.splitter.disk.Member; -import uk.me.parabola.splitter.disk.NodeStoreReader; -import uk.me.parabola.splitter.disk.RelationStoreReader; -import uk.me.parabola.splitter.disk.WayStoreReader; - -/** - * Loads binary map files, calling the appropriate methods on a - * {@code MapProcessor} as it progresses. - */ -class BinaryMapLoader implements MapReader { - - // How many elements to process before displaying a status update - private static final int NODE_STATUS_UPDATE_THRESHOLD = 2500000; - private static final int WAY_STATUS_UPDATE_THRESHOLD = 500000; - private static final int RELATION_STATUS_UPDATE_THRESHOLD = 50000; - - private final String path; - private final MapProcessor processor; - - private boolean startNodeOnly; - - private long nodeCount; - private long wayCount; - private long relationCount; - private int minNodeId = Integer.MAX_VALUE; - private int maxNodeId = Integer.MIN_VALUE; - - BinaryMapLoader(String path, MapProcessor processor) { - this.path = path; - this.processor = processor; - this.startNodeOnly = processor.isStartNodeOnly(); - } - - @Override - public long getNodeCount() { - return nodeCount; - } - - @Override - public long getWayCount() { - return wayCount; - } - - @Override - public long getRelationCount() { - return relationCount; - } - - @Override - public int getMinNodeId() { - return minNodeId; - } - - @Override - public int getMaxNodeId() { - return maxNodeId; - } - - public void load() throws IOException { - processNodes(); - if (!startNodeOnly) { - processWays(); - processRelations(); - } - processor.endMap(); - } - - private void processNodes() throws IOException { - System.out.println("Loading and processing nodes"); - NodeStoreReader reader = new NodeStoreReader(path + File.separatorChar + "nodes.bin"); - while (reader.next()) { - int id = reader.getId(); - processor.startNode(id, reader.getLat(), reader.getLon()); - if (!startNodeOnly) { - for (Map.Entry<String, String> entry : reader.getTags().entrySet()) { - processor.nodeTag(entry.getKey(), entry.getValue()); - } - processor.endNode(); - } - - if (id < minNodeId) { - minNodeId = id; - } - if (id > maxNodeId) { - maxNodeId = id; - } - - nodeCount++; - if (nodeCount % NODE_STATUS_UPDATE_THRESHOLD == 0) { - System.out.println(Utils.format(nodeCount) + " nodes processed..."); - } - } - reader.close(); - } - - private void processWays() throws IOException { - System.out.println("Loading and processing ways"); - WayStoreReader reader = new WayStoreReader(path + File.separatorChar + "ways.bin"); - while (reader.next()) { - processor.startWay(reader.getId()); - for (int nodeId : reader.getNodeIds()) { - processor.wayNode(nodeId); - } - for (Map.Entry<String, String> entry : reader.getTags().entrySet()) { - processor.wayTag(entry.getKey(), entry.getValue()); - } - processor.endWay(); - wayCount++; - if (wayCount % WAY_STATUS_UPDATE_THRESHOLD == 0) { - System.out.println(Utils.format(wayCount) + " ways processed..."); - } - } - reader.close(); - } - - private void processRelations() throws IOException { - System.out.println("Loading and processing relations"); - RelationStoreReader reader = new RelationStoreReader(path + File.separatorChar + "relations.bin"); - while (reader.next()) { - processor.startRelation(reader.getId()); - for (Map.Entry<String, String> entry : reader.getTags().entrySet()) { - processor.relationTag(entry.getKey(), entry.getValue()); - } - for (Member member : reader.getMembers()) { - switch (member.getType()) { - case Node: - processor.relationNode(member.getId(), member.getRole()); - break; - case Way: - processor.relationWay(member.getId(), member.getRole()); - break; - } - } - processor.endRelation(); - relationCount++; - if (relationCount % RELATION_STATUS_UPDATE_THRESHOLD == 0) { - System.out.println(Utils.format(relationCount) + " relations processed..."); - } - } - reader.close(); - } -} \ No newline at end of file diff --git a/src/uk/me/parabola/splitter/CachingMapProcessor.java b/src/uk/me/parabola/splitter/CachingMapProcessor.java deleted file mode 100644 index 5b8b799..0000000 --- a/src/uk/me/parabola/splitter/CachingMapProcessor.java +++ /dev/null @@ -1,242 +0,0 @@ -/* - * Copyright (c) 2009. - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 3 as - * published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - */ - -package uk.me.parabola.splitter; - -import java.io.File; -import java.io.IOException; - -import uk.me.parabola.splitter.disk.CacheVerifier; -import uk.me.parabola.splitter.disk.MemberType; -import uk.me.parabola.splitter.disk.NodeStoreWriter; -import uk.me.parabola.splitter.disk.RelationStoreWriter; -import uk.me.parabola.splitter.disk.WayStoreWriter; - -/** - * Writes the map out in a binary format to a disk cache and then - * delegates calls on to another {@link MapProcessor}. - * - * @author Chris Miller - */ -public class CachingMapProcessor implements MapProcessor { - - private NodeStoreWriter nodeWriter; - private WayStoreWriter wayWriter; - private RelationStoreWriter relationWriter; - - private MapProcessor delegate; - CacheVerifier verifier; - private int currentNode; - private int currentWay; - private int currentRel; - private boolean startedWayTags; - private boolean startedRelTags; - - public CachingMapProcessor(String outputDir, CacheVerifier verifier, MapProcessor delegate) throws IOException { - this.delegate = delegate; - this.verifier = verifier; - verifier.clearEntries(); - nodeWriter = new NodeStoreWriter(outputDir + File.separatorChar + "nodes.bin"); - wayWriter = new WayStoreWriter(outputDir + File.separatorChar + "ways.bin"); - relationWriter = new RelationStoreWriter(outputDir + File.separatorChar + "relations.bin"); - } - - @Override - public boolean isStartNodeOnly() { - return false; - } - - @Override - public void boundTag(Area bounds) { - // todo: write the bounds out to the cache - delegate.boundTag(bounds); - } - - @Override - public void startNode(int id, double lat, double lon) { - currentNode = id; - try { - nodeWriter.write(id, lat, lon); - } catch (IOException e) { - System.out.println("Unable to write node " + id + ". Reason: " + e.getMessage()); - throw new RuntimeException(e); - } - delegate.startNode(id, lat, lon); - } - - @Override - public void startWay(int id) { - currentWay = id; - startedWayTags = false; - try { - wayWriter.write(id); - } catch (IOException e) { - System.out.println("Unable to write way " + id + ". Reason: " + e.getMessage()); - throw new RuntimeException(e); - } - if (!delegate.isStartNodeOnly()) - delegate.startWay(id); - } - - @Override - public void startRelation(int id) { - currentRel = id; - startedRelTags = false; - try { - relationWriter.write(id); - } catch (IOException e) { - System.out.println("Unable to write relation " + id + ". Reason: " + e.getMessage()); - throw new RuntimeException(e); - } - if (!delegate.isStartNodeOnly()) - delegate.startRelation(id); - } - - @Override - public void nodeTag(String key, String value) { - try { - nodeWriter.writeTag(key, value); - } catch (IOException e) { - System.out.println("Unable to write tag for node " + currentNode + ". key=" + key + ", value=" + value + ". Reason: " + e.getMessage()); - throw new RuntimeException(e); - } - if (!delegate.isStartNodeOnly()) - delegate.nodeTag(key, value); - } - - @Override - public void wayTag(String key, String value) { - try { - if (!startedWayTags) { - startedWayTags = true; - wayWriter.closeNodeRefs(); - } - wayWriter.writeTag(key, value); - } catch (IOException e) { - System.out.println("Unable to write tag for way " + currentWay + ". key=" + key + ", value=" + value + ". Reason: " + e.getMessage()); - throw new RuntimeException(e); - } - if (!delegate.isStartNodeOnly()) - delegate.wayTag(key, value); - } - - @Override - public void relationTag(String key, String value) { - try { - if (!startedRelTags) { - startedRelTags = true; - relationWriter.closeMembers(); - } - relationWriter.writeTag(key, value); - } catch (IOException e) { - System.out.println("Unable to write tag for relation " + currentRel + ". key=" + key + ", value=" + value + ". Reason: " + e.getMessage()); - throw new RuntimeException(e); - } - if (!delegate.isStartNodeOnly()) - delegate.relationTag(key, value); - } - - @Override - public void wayNode(int nodeId) { - try { - wayWriter.writeNodeRef(nodeId); - } catch (IOException e) { - System.out.println("Unable to write node reference for way " + currentWay + ", nodeId=" + nodeId + ". Reason: " + e.getMessage()); - throw new RuntimeException(e); - } - if (!delegate.isStartNodeOnly()) - delegate.wayNode(nodeId); - } - - @Override - public void relationNode(int nodeId, String role) { - try { - relationWriter.writeMember(MemberType.Node, nodeId, role); - } catch (IOException e) { - System.out.println("Unable to write node member for relation " + currentRel + ", nodeId=" + nodeId + ", role='" + role + "'. Reason: " + e.getMessage()); - throw new RuntimeException(e); - } - if (!delegate.isStartNodeOnly()) - delegate.relationNode(nodeId, role); - } - - @Override - public void relationWay(int wayId, String role) { - try { - relationWriter.writeMember(MemberType.Way, wayId, role); - } catch (IOException e) { - System.out.println("Unable to write way member for relation " + currentRel + ", wayId=" + wayId + ", role='" + role + "'. Reason: " + e.getMessage()); - throw new RuntimeException(e); - } - if (!delegate.isStartNodeOnly()) - delegate.relationWay(wayId, role); - } - - @Override - public void endNode() { - try { - nodeWriter.closeTags(); - nodeWriter.next(); - } catch (IOException e) { - System.out.println("Unable to finish writing node " + currentNode + ". Reason: " + e.getMessage()); - throw new RuntimeException(e); - } - if (!delegate.isStartNodeOnly()) - delegate.endNode(); - } - - @Override - public void endWay() { - try { - if (!startedWayTags) - wayWriter.closeNodeRefs(); - wayWriter.closeTags(); - wayWriter.next(); - } catch (IOException e) { - System.out.println("Unable to finish writing way " + currentWay + ". Reason: " + e.getMessage()); - throw new RuntimeException(e); - } - if (!delegate.isStartNodeOnly()) - delegate.endWay(); - } - - @Override - public void endRelation() { - try { - if (!startedRelTags) - relationWriter.closeMembers(); - relationWriter.closeTags(); - relationWriter.next(); - } catch (IOException e) { - System.out.println("Unable to finish writing relation " + currentRel + ". Reason: " + e.getMessage()); - throw new RuntimeException(e); - } - if (!delegate.isStartNodeOnly()) - delegate.endRelation(); - } - - @Override - public void endMap() { - try { - nodeWriter.close(); - wayWriter.close(); - relationWriter.close(); - verifier.saveEntries(); - } catch (IOException e) { - System.out.println("Unable to close node/way/relation writer(s). Reason: " + e.getMessage()); - throw new RuntimeException(e); - } - if (!delegate.isStartNodeOnly()) - delegate.endMap(); - } -} diff --git a/src/uk/me/parabola/splitter/Main.java b/src/uk/me/parabola/splitter/Main.java index 37dfc77..84e5dd7 100644 --- a/src/uk/me/parabola/splitter/Main.java +++ b/src/uk/me/parabola/splitter/Main.java @@ -13,7 +13,6 @@ package uk.me.parabola.splitter; -import java.io.File; import java.io.FileWriter; import java.io.IOException; import java.io.InputStreamReader; @@ -27,7 +26,6 @@ import java.util.Set; import uk.me.parabola.splitter.args.ParamParser; import uk.me.parabola.splitter.args.SplitterParams; -import uk.me.parabola.splitter.disk.CacheVerifier; import uk.me.parabola.splitter.geo.City; import uk.me.parabola.splitter.geo.CityFinder; import uk.me.parabola.splitter.geo.CityLoader; @@ -84,12 +82,6 @@ public class Main { // Whether or not the source OSM file(s) contain strictly nodes first, then ways, then rels, // or they're all mixed up. Running with mixed enabled takes longer. private boolean mixed; - // The path to the disk cache. If this is null, no cache will be generated or used. - private String diskCachePath; - // Whether or not a new cache needs to be generated. - private boolean generateCache; - // Used to verify whether an existing cache is valid or not. - private CacheVerifier verifier; // The path where the results are written out to. private String fileOutputDir; // A GeoNames file to use for naming the tiles. @@ -147,53 +139,15 @@ public class Main { fileOutputDir = DEFAULT_DIR; } - if (diskCachePath != null) { - File cacheDir = new File(diskCachePath); - if (!cacheDir.exists()) { - System.out.println("Cache directory not found. Creating directory '" + cacheDir + "' and generating cache"); - if (cacheDir.mkdirs()) { - generateCache = true; - } else { - System.err.println("Unable to create cache directory! Disk cache disabled"); - diskCachePath = null; - } - } else if (!cacheDir.isDirectory()) { - System.err.println("The --cache parameter must specify a directory. The --cache parameter is being ignored, disk cache is disabled."); - diskCachePath = null; - } - if (diskCachePath != null) { - verifier = new CacheVerifier(diskCachePath, filenames); - try { - if (!generateCache) { - System.out.println("Checking for an existing cache and verifying contents..."); - } - if (verifier.validateCache()) { - System.out.println("A suitable cache was found. All data will be loaded from cache rather than any .osm file(s) or stdin"); - } else if (filenames.isEmpty()) { - System.out.println("No .osm files were supplied and the --cache isn't populated so osm data will be read from stdin"); - useStdIn = true; - generateCache = true; - } else { - System.out.println("No suitable cache was found. A new cache will be created to speed up the splitting stage"); - generateCache = true; - } - } catch (IOException e) { - System.err.println("Unable to verify cache content - regenerating cache. Reason: " + e.getMessage()); - e.printStackTrace(); - generateCache = true; - } - } - } - - if (diskCachePath == null && filenames.isEmpty()) { + if (filenames.isEmpty()) { if (areaList == null) { - throw new IllegalArgumentException("No .osm files were supplied so at least one of --cache or --split-file must be specified"); + throw new IllegalArgumentException("No .osm files were supplied so --split-file must be specified"); } else { int areaCount = areaList.getAreas().size(); int passes = getAreasPerPass(areaCount); if (passes > 1) { - throw new IllegalArgumentException("No .osm files or --cache parameter were supplied, but stdin cannot be used because " + passes - + " passes are required to write out the areas. Either provide --cache or increase --max-areas to match the number of areas (" + areaCount + ')'); + throw new IllegalArgumentException("No .osm files supplied, but stdin cannot be used because " + passes + + " passes are required to write out the areas. Increase --max-areas to match the number of areas (" + areaCount + ')'); } useStdIn = true; } @@ -270,7 +224,6 @@ public class Main { } mixed = params.isMixed(); statusFreq = params.getStatusFreq(); - diskCachePath = params.getCache(); fileOutputDir = params.getOutputDir(); if (fileOutputDir == null) { fileOutputDir = DEFAULT_DIR; @@ -311,27 +264,12 @@ public class Main { MapCollector nodes = densityMap ? new DensityMapCollector(trim, resolution) : new NodeCollector(); MapProcessor processor = nodes; - boolean loadFromCache = false; - if (diskCachePath == null) { - System.out.println("The input osm file(s) will be re-parsed during the split (slower) because no --cache parameter was specified"); - } else { - if (generateCache) { - processor = new CachingMapProcessor(diskCachePath, verifier, processor); - generateCache = false; - } else { - loadFromCache = true; - } - } - - MapReader mapReader = processMap(processor, loadFromCache); + MapReader mapReader = processMap(processor); System.out.print("A total of " + Utils.format(mapReader.getNodeCount()) + " nodes, " + Utils.format(mapReader.getWayCount()) + " ways and " + Utils.format(mapReader.getRelationCount()) + " relations were processed "); - if (loadFromCache) { - System.out.println("from the disk cache"); - } else { - System.out.println("in " + filenames.size() + (filenames.size() == 1 ? " file" : " files")); - } + + System.out.println("in " + filenames.size() + (filenames.size() == 1 ? " file" : " files")); System.out.println("Min node ID = " + mapReader.getMinNodeId()); System.out.println("Max node ID = " + mapReader.getMaxNodeId()); @@ -404,18 +342,7 @@ public class Main { areas.get(i * areasPerPass + currentWriters.length - 1).getMapId() + ')'); MapProcessor processor = new SplitProcessor(currentWriters, maxThreads); - if (generateCache) { - if (numPasses == 1) { - System.out.println("No cache will be generated since only one pass is required"); - generateCache = false; - diskCachePath = null; - } else { - System.out.println("No valid existing cache found. A cache will be generated on this pass"); - processor = new CachingMapProcessor(diskCachePath, verifier, processor); - } - } - MapReader mapReader = processMap(processor, !generateCache && diskCachePath != null); - generateCache = false; // Make sure the cache isn't generated more than once! + MapReader mapReader = processMap(processor); System.out.println("Wrote " + Utils.format(mapReader.getNodeCount()) + " nodes, " + Utils.format(mapReader.getWayCount()) + " ways, " + Utils.format(mapReader.getRelationCount()) + " relations"); @@ -426,16 +353,10 @@ public class Main { return (int) Math.ceil((double) areaCount / (double) maxAreasPerPass); } - private MapReader processMap(MapProcessor processor, boolean useCache) throws XmlPullParserException, IOException { - if (useCache) { - BinaryMapLoader loader = new BinaryMapLoader(diskCachePath, processor); - loader.load(); - return loader; - } else { - OSMParser parser = new OSMParser(processor, mixed); - processOsmFiles(parser); - return parser; - } + private MapReader processMap(MapProcessor processor) throws XmlPullParserException, IOException { + OSMParser parser = new OSMParser(processor, mixed); + processOsmFiles(parser); + return parser; } private void processOsmFiles(OSMParser parser) throws IOException, XmlPullParserException { diff --git a/src/uk/me/parabola/splitter/disk/AbstractStoreReader.java b/src/uk/me/parabola/splitter/disk/AbstractStoreReader.java deleted file mode 100644 index cb0b863..0000000 --- a/src/uk/me/parabola/splitter/disk/AbstractStoreReader.java +++ /dev/null @@ -1,75 +0,0 @@ -/* - * Copyright (c) 2009. - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 3 as - * published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - */ -package uk.me.parabola.splitter.disk; - -import java.io.BufferedInputStream; -import java.io.EOFException; -import java.io.IOException; -import java.io.InputStream; -import java.util.HashMap; -import java.util.Map; - -/** - * Abstract base class for reading in binary cache files - */ -public abstract class AbstractStoreReader { - protected final LengthPrefixInputStream in; - protected final KeyLookupReader keys; - protected int id; - protected Map<String, String> tags = new HashMap<String, String>(20); - - public AbstractStoreReader(InputStream in, KeyLookupReader keys) throws IOException { - this.in = new LengthPrefixInputStream(new BufferedInputStream(in, 16384)); - this.keys = keys; - } - - public int getId() { - return id; - } - - public Map<String, String> getTags() { - return tags; - } - - public boolean next() throws IOException { - try { - if (!in.next()) { - return false; - } - id = in.readInt(); - readHeader(); - readTags(); - return true; - } catch (EOFException e) { - return false; - } - } - - protected LengthPrefixInputStream getIn() { - return in; - } - - protected abstract void readHeader() throws IOException; - - private void readTags() throws IOException { - short keyId; - tags.clear(); - while ((keyId = in.readShort()) != 0) { - tags.put(keys.get(keyId), in.readUTF()); - } - } - - public void close() throws IOException { - in.close(); - } -} diff --git a/src/uk/me/parabola/splitter/disk/AbstractStoreWriter.java b/src/uk/me/parabola/splitter/disk/AbstractStoreWriter.java deleted file mode 100644 index d7bad04..0000000 --- a/src/uk/me/parabola/splitter/disk/AbstractStoreWriter.java +++ /dev/null @@ -1,58 +0,0 @@ -/* - * Copyright (c) 2009. - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 3 as - * published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - */ -package uk.me.parabola.splitter.disk; - -import java.io.IOException; -import java.io.OutputStream; - -/** - * Provides base functionality for writing elements out to disk in binary format - */ -public abstract class AbstractStoreWriter { - - private final LengthPrefixOutputStream out; - private final KeyLookupWriter keys; - - public AbstractStoreWriter(OutputStream out, KeyLookupWriter keys) { - this.out = new LengthPrefixOutputStream(out); - this.keys = keys; - } - - protected LengthPrefixOutputStream getOut() { - return out; - } - - public void writeTag(CharSequence key, CharSequence value) throws IOException { - out.writeShort(keys.set(key)); - out.writeUTF(value); - } - - public void closeTags() throws IOException { - out.writeShort(0); - } - - /** - * Inserts the length header and writes the element through to - * the underlying output stream. - * - * @throws IOException - */ - public void next() throws IOException { - out.next(); - } - - public void close() throws IOException { - out.close(); - keys.close(); - } -} diff --git a/src/uk/me/parabola/splitter/disk/CacheVerifier.java b/src/uk/me/parabola/splitter/disk/CacheVerifier.java deleted file mode 100644 index b99d401..0000000 --- a/src/uk/me/parabola/splitter/disk/CacheVerifier.java +++ /dev/null @@ -1,183 +0,0 @@ -/* - * Copyright (c) 2009. - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 3 as - * published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - */ - -package uk.me.parabola.splitter.disk; - -import java.io.BufferedOutputStream; -import java.io.BufferedReader; -import java.io.File; -import java.io.FileInputStream; -import java.io.FileOutputStream; -import java.io.IOException; -import java.io.InputStreamReader; -import java.io.OutputStreamWriter; -import java.io.Writer; -import java.util.ArrayList; -import java.util.Collections; -import java.util.List; - -/** - * Maintains a persistent list of the .osm file(s) used to build the cache. Also verifies this - * list against a list of files that are to be processed, so a decision can be made about - * whether to flush the cache or not. - * - * @author Chris Miller - */ -public class CacheVerifier { - private final String cacheDirectory; // the cache location - private final File entriesFile; // the file containing a list of the cache entries - private final List<String> filenames; // the .osm files we expect to have been cached - - public CacheVerifier(String cacheDirectory, List<String> filenames) { - this.cacheDirectory = cacheDirectory; - entriesFile = new File(cacheDirectory, "cache.entries"); - this.filenames = filenames; - } - - /** - * Checks to see if the cache already contains suitable data. It does this by comparing - * filenames and timestamps from the cache.entries file against the files that are - * being processed by this run of the splitter. - * - * @param filenames the files to process. - * @return {@code true} if there is an existing cache and it is valid for this run. - */ - public boolean validateCache() throws IOException { - List<CacheEntry> existingEntries = loadEntries(); - - // If no cache.entries file was found, we have to regenerate the cache - if (existingEntries == null) - return false; - - // See if there are any cache files - boolean nodesExist = new File(cacheDirectory, "nodes.bin").exists(); - - // If no filenames were provided we use the cache (if it exists) - if (filenames.isEmpty()) - return nodesExist; - - // Compare the existing cache entries with the .osm filenames provided. If they match we can use the cache - List<CacheEntry> entries = new ArrayList<CacheEntry>(filenames.size()); - for (String filename : filenames) { - entries.add(new CacheEntry(filename)); - } - Collections.sort(existingEntries); - Collections.sort(entries); - - return existingEntries.equals(entries) && nodesExist; - } - - public void saveEntries() throws IOException { - Writer out = new OutputStreamWriter(new BufferedOutputStream(new FileOutputStream(entriesFile), 4096), "UTF-8"); - try { - for (String filename : filenames) { - File file = new File(filename); - if (file.exists()) { - long size = file.length(); - long timestamp = file.lastModified(); - out.write(String.valueOf(size)); - out.write(','); - out.write(String.valueOf(timestamp)); - out.write(','); - out.write(file.getCanonicalPath()); - out.write('\n'); - } - } - } finally { - out.close(); - } - } - - protected List<CacheEntry> loadEntries() throws IOException { - if (!entriesFile.exists()) { - return null; - } - BufferedReader in = new BufferedReader(new InputStreamReader(new FileInputStream(entriesFile), "UTF-8")); - List<CacheEntry> result = new ArrayList<CacheEntry>(); - int count = 0; - String line; - try { - while ((line = in.readLine()) != null) { - count++; - String[] parts = line.split(",", 3); - if (parts.length < 3) { - System.err.println("Invalid cache.entries file on line " + count + ". The cache will be rebuilt"); - return null; - } - long size = Long.valueOf(parts[0]); - long timestamp = Long.valueOf(parts[1]); - result.add(new CacheEntry(parts[2], size, timestamp)); - } - } finally { - in.close(); - } - return result; - } - - public boolean clearEntries() { - return entriesFile.delete(); - } - - protected static class CacheEntry implements Comparable<CacheEntry> { - private final String filename; - private final long size; - private final long timestamp; - - public CacheEntry(String filename, long size, long timestamp) { - this.filename = filename; - this.size = size; - this.timestamp = timestamp; - } - - protected CacheEntry(String filename) throws IOException { - File file = new File(filename); - this.filename = file.getCanonicalPath(); - size = file.length(); - timestamp = file.lastModified(); - } - - @Override - public boolean equals(Object o) { - if (this == o) return true; - if (o == null || getClass() != o.getClass()) return false; - - CacheEntry that = (CacheEntry) o; - - if (size != that.size) return false; - if (timestamp != that.timestamp) return false; - if (!filename.equals(that.filename)) return false; - - return true; - } - - @Override - public int hashCode() { - int result = filename.hashCode(); - result = 31 * result + (int) (size ^ (size >>> 32)); - result = 31 * result + (int) (timestamp ^ (timestamp >>> 32)); - return result; - } - - @Override - public int compareTo(CacheEntry o) { - int result = filename.compareTo(o.filename); - if (result == 0) { - result = size < o.size ? -1 : size > o.size ? 1 : 0; - } - if (result == 0) { - result = timestamp < o.timestamp ? -1 : timestamp > o.timestamp ? 1 : 0; - } - return result; - } - } -} diff --git a/src/uk/me/parabola/splitter/disk/KeyLookupReader.java b/src/uk/me/parabola/splitter/disk/KeyLookupReader.java deleted file mode 100644 index d983679..0000000 --- a/src/uk/me/parabola/splitter/disk/KeyLookupReader.java +++ /dev/null @@ -1,58 +0,0 @@ -/* - * Copyright (c) 2009. - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 3 as - * published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - */ -package uk.me.parabola.splitter.disk; - -import java.io.EOFException; -import java.io.File; -import java.io.FileInputStream; -import java.io.IOException; -import java.io.InputStream; - -import uk.me.parabola.splitter.IntObjMap; - -/** - * Reads in a set of strings to binary format so they can be indexed by ID - */ -public class KeyLookupReader { - private final IntObjMap<String> lookup = new IntObjMap<String>(); - - public KeyLookupReader(String keyFilename) throws IOException { - File keyFile = new File(keyFilename); - LengthPrefixInputStream is = new LengthPrefixInputStream(new FileInputStream(keyFile), 16384); - load(is); - } - - public KeyLookupReader(InputStream in) throws IOException { - LengthPrefixInputStream is = new LengthPrefixInputStream(in, 16384); - load(is); - } - - private void load(LengthPrefixInputStream is) throws IOException { - try { - int s = 0; - while (is.next()) { - try { - lookup.put(++s, is.readUTF()); - } catch (EOFException e) { - break; - } - } - } finally { - is.close(); - } - } - - public String get(int i) { - return lookup.get(i); - } -} diff --git a/src/uk/me/parabola/splitter/disk/KeyLookupWriter.java b/src/uk/me/parabola/splitter/disk/KeyLookupWriter.java deleted file mode 100644 index 536a219..0000000 --- a/src/uk/me/parabola/splitter/disk/KeyLookupWriter.java +++ /dev/null @@ -1,50 +0,0 @@ -/* - * Copyright (c) 2009. - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 3 as - * published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - */ -package uk.me.parabola.splitter.disk; - -import java.io.FileOutputStream; -import java.io.IOException; -import java.io.OutputStream; -import java.util.HashMap; -import java.util.Map; - -/** - * Writes out a set of strings to binary format so they can be indexed by ID - */ -public class KeyLookupWriter { - private final LengthPrefixOutputStream out; - private final Map<CharSequence, Short> lookup = new HashMap<CharSequence, Short>(20); - - public KeyLookupWriter(String keyFilename) throws IOException { - out = new LengthPrefixOutputStream(new FileOutputStream(keyFilename)); - } - - public KeyLookupWriter(OutputStream out) throws IOException { - this.out = new LengthPrefixOutputStream(out); - } - - public short set(CharSequence key) throws IOException { - Short result = lookup.get(key); - if (result == null) { - result = Short.valueOf((short) (lookup.size() + 1)); - lookup.put(key, result); - out.writeUTF(key); - out.next(); - } - return result; - } - - public void close() throws IOException { - out.close(); - } -} \ No newline at end of file diff --git a/src/uk/me/parabola/splitter/disk/LengthPrefixInputStream.java b/src/uk/me/parabola/splitter/disk/LengthPrefixInputStream.java deleted file mode 100644 index 7f41c5f..0000000 --- a/src/uk/me/parabola/splitter/disk/LengthPrefixInputStream.java +++ /dev/null @@ -1,278 +0,0 @@ -/* - * Copyright (c) 2009. - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 3 as - * published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - */ - -package uk.me.parabola.splitter.disk; - -import java.io.IOException; -import java.io.InputStream; -import java.io.UTFDataFormatException; - -import uk.me.parabola.splitter.Utils; - -/** - * Reads in a 'unit' of data at a time into a buffer every time next() is called. - * Each unit has a two-byte prefix indicating how many bytes long the unit is. - * Every time next() is called, the next unit will be read in. - * <p/> - * This is similar to a {@code DataInputStream} but offers better - * performance (more specialised buffering, no synchronisation). - * - * @author Chris Miller - */ -public class LengthPrefixInputStream extends InputStream { - - private InputStream in; - private byte[] buf; - private int index; - private int maxLen; - private long bytesRead; - - public LengthPrefixInputStream(InputStream in) { - this(in, 4096); - } - - public LengthPrefixInputStream(InputStream in, int bufferSize) { - this.in = in; - buf = new byte[bufferSize]; - } - - public int remaining() { - return maxLen - index; - } - - public byte readByte() throws IOException { - ensureData(1); - return buf[index++]; - } - - public int readUnsignedByte() throws IOException { - ensureData(1); - return buf[index++] & 0xff; - } - - public short readShort() throws IOException { - ensureData(2); - return (short) ((buf[index++] << 8) | (buf[index++] & 0xff)); - } - - public int readUnsignedShort() throws IOException { - ensureData(2); - return (buf[index++] & 0xff) << 8 | (buf[index++] & 0xff); - } - - public int readInt() throws IOException { - ensureData(4); - return buf[index++] << 24 | (buf[index++] & 0xff) << 16 | (buf[index++] & 0xff) << 8 | (buf[index++] & 0xff); - } - - public long readLong() throws IOException { - ensureData(8); - return (long) buf[index++] << 56 | - ((long) buf[index++] & 0xff) << 48 | - ((long) buf[index++] & 0xff) << 40 | - ((long) buf[index++] & 0xff) << 32 | - ((long) buf[index++] & 0xff) << 24 | - ((long) buf[index++] & 0xff) << 16 | - ((long) buf[index++] & 0xff) << 8 | - buf[index++] & 0xff; - } - - public double readDouble() throws IOException { - return Double.longBitsToDouble(readLong()); - } - - /** - * This has been lifted from DataInputStream. We inline it here rather than - * delegate to DIS because that way we can avoid creating a temporary buffer - * for the UTF-8 chars. - */ - public String readUTF() throws IOException { - int utflen = readUnsignedShort(); - ensureData(utflen); - char[] chararr = new char[utflen]; - - int c, char2, char3; - int count = 0; - int chararr_count = 0; - - while (count < utflen) { - c = (int) buf[index + count] & 0xff; - if (c > 127) break; - count++; - chararr[chararr_count++] = (char) c; - } - - while (count < utflen) { - c = (int) buf[index + count] & 0xff; - switch (c >> 4) { - case 0: - case 1: - case 2: - case 3: - case 4: - case 5: - case 6: - case 7: - /* 0xxxxxxx*/ - count++; - chararr[chararr_count++] = (char) c; - break; - case 12: - case 13: - /* 110x xxxx 10xx xxxx*/ - count += 2; - if (count > utflen) - throw new UTFDataFormatException("malformed input: partial character at end"); - char2 = (int) buf[index + count - 1]; - if ((char2 & 0xC0) != 0x80) - throw new UTFDataFormatException("malformed input around byte " + count); - chararr[chararr_count++] = (char) (((c & 0x1F) << 6) | (char2 & 0x3F)); - break; - case 14: - /* 1110 xxxx 10xx xxxx 10xx xxxx */ - count += 3; - if (count > utflen) - throw new UTFDataFormatException("malformed input: partial character at end"); - char2 = (int) buf[index + count - 2]; - char3 = (int) buf[index + count - 1]; - if (((char2 & 0xC0) != 0x80) || ((char3 & 0xC0) != 0x80)) - throw new UTFDataFormatException("malformed input around byte " + (count - 1)); - chararr[chararr_count++] = (char) ((c & 0x0F) << 12 | (char2 & 0x3F) << 6 | (char3 & 0x3F)); - break; - default: - /* 10xx xxxx, 1111 xxxx */ - throw new UTFDataFormatException("malformed input around byte " + count); - } - } - index += count; - // The number of chars produced may be less than utflen - return new String(chararr, 0, chararr_count); - } - - private byte[] lengthBuf = new byte[2]; - - public boolean next() throws IOException { - // Read the length prefix - int len = 0; - while (len < 2) { - int read = in.read(lengthBuf, len, 2 - len); - if (read < 0) { - if (len > 0) { - displayByteBuffer(); - System.out.println("Unexpected EOF reached while reading segment length. Only " + len + " of 2 bytes read. Total bytes read " + Utils.format(bytesRead)); - } - return false; - } - len += read; - bytesRead += read; - } - maxLen = (lengthBuf[0] & 0xFF) << 8 | (lengthBuf[1] & 0xFF); - if (maxLen == 0xFFFF) { - // we've hit a marker that indicates the length is held in an int rather than a short - byte[] temp = new byte[4]; - len = 0; - while (len < 4) { - int read = in.read(temp, len, 4 - len); - if (read < 0) { - displayByteBuffer(); - System.out.println("Unexpected EOF reached while reading segment length. Only " + len + " of 4 bytes read. Total bytes read " + Utils.format(bytesRead)); - return false; - } - len += read; - bytesRead += read; - } - maxLen = temp[0] << 24 | (temp[1] & 0xff) << 16 | (temp[2] & 0xff) << 8 | (temp[3] & 0xff); - } - index = 0; - - // Make sure the buffer has enough room for this segment of data - if (buf.length < index + maxLen) { - buf = new byte[index + maxLen]; - } - // Read in the whole buffer - len = 0; - while (len < maxLen) { - int read = in.read(buf, index + len, maxLen - len); - if (read < 0) { - // We've hit the EOF - this shouldn't happen! - System.out.println("Unexpected EOF reached while loading segment. Expected " + maxLen + " bytes, only read " + len + ". Total bytes read " + Utils.format(bytesRead)); - System.out.println("Here's the buffer that was read:"); - maxLen = len; - displayByteBuffer(); - return false; - } - len += read; - bytesRead += read; - } - return true; - } - - private void ensureData(int len) throws IOException { - if (index + len > maxLen) { - displayByteBuffer(); // useful for debugging - throw new IOException("Attempt was made to read " + len + " bytes when there are only " + (maxLen - index) + " remaining (from " + maxLen + " total in this segment)"); - } - } - - @Override - public int read(byte[] b) throws IOException { - return read(b, 0, b.length); - } - - @Override - public int read(byte[] b, int off, int len) throws IOException { - int result = Math.min(len, maxLen - index); - System.arraycopy(buf, index, b, off, result); - index += result; - return result; - } - - @Override - public int read() throws IOException { - return in.read(); - } - - @Override - public void close() throws IOException { - in.close(); - } - - private static final char[] HEX_CHARS = new char[] {'0','1','2','3','4','5','6','7','8','9','A','B','C','D','E','F'}; - private static final int BYTES_PER_ROW = 32; - - private void displayByteBuffer() { - char[] ascii = new char[BYTES_PER_ROW]; - for (int i = 0; i < maxLen; i++) { - System.out.print(HEX_CHARS[(buf[i] & 0xFF) >> 4]); - System.out.print(HEX_CHARS[buf[i] & 0x0F]); - if (i % 8 == 7) - System.out.print(" "); - else - System.out.print(' '); - - char c = (char) buf[i]; - if (c < 32 || c == 127) - c = '.'; - ascii[i % BYTES_PER_ROW] = c; - - if (i % BYTES_PER_ROW == BYTES_PER_ROW - 1) - System.out.println(ascii); - } - int missing = (BYTES_PER_ROW - (maxLen % BYTES_PER_ROW) % BYTES_PER_ROW); - for (int i = 0; i < missing; i++) - System.out.print(" "); - for (int i = 0; i < missing / 8; i++) - System.out.print(' '); - System.out.println(ascii); - } -} \ No newline at end of file diff --git a/src/uk/me/parabola/splitter/disk/LengthPrefixOutputStream.java b/src/uk/me/parabola/splitter/disk/LengthPrefixOutputStream.java deleted file mode 100644 index ff97315..0000000 --- a/src/uk/me/parabola/splitter/disk/LengthPrefixOutputStream.java +++ /dev/null @@ -1,194 +0,0 @@ -/* - * Copyright (c) 2009. - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 3 as - * published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - */ - -package uk.me.parabola.splitter.disk; - -import java.io.BufferedOutputStream; -import java.io.IOException; -import java.io.OutputStream; -import java.io.UTFDataFormatException; - -/** - * Buffers the output then writes it all at once with a 2-byte length prefix - * every time next() is called. - * <p/> - * This is similar to a {@code DataOutputStream} but offers better - * performance (more specialised buffering, no synchronisation). - * - * @author Chris Miller - */ -public class LengthPrefixOutputStream extends OutputStream { - - private OutputStream out; - private byte[] buf; - private int index = 2; - - public LengthPrefixOutputStream(OutputStream out) { - this(out, 4096); - } - - public LengthPrefixOutputStream(OutputStream out, int bufferSize) { - this.out = new BufferedOutputStream(out, 8192); - buf = new byte[bufferSize]; - } - - public void writeByte(int b) throws IOException { - ensureCapacity(1); - buf[index++] = (byte) b; - } - - public final void writeShort(int v) throws IOException { - ensureCapacity(2); - buf[index++] = (byte) (v >>> 8); - buf[index++] = (byte) v; - } - - public final void writeInt(int v) throws IOException { - ensureCapacity(4); - buf[index++] = (byte) (v >>> 24); - buf[index++] = (byte) (v >>> 16); - buf[index++] = (byte) (v >>> 8); - buf[index++] = (byte) v; - } - - public final void writeLong(long v) throws IOException { - ensureCapacity(8); - buf[index++] = (byte) (v >>> 56); - buf[index++] = (byte) (v >>> 48); - buf[index++] = (byte) (v >>> 40); - buf[index++] = (byte) (v >>> 32); - buf[index++] = (byte) (v >>> 24); - buf[index++] = (byte) (v >>> 16); - buf[index++] = (byte) (v >>> 8); - buf[index++] = (byte) v; - } - - public final void writeDouble(double v) throws IOException { - writeLong(Double.doubleToLongBits(v)); - } - - /** - * This has been lifted from DataOutputStream. We inline it here rather than - * delegate to DOS because that way we can avoid creating a temporary buffer - * for the UTF-8 chars. - */ - public void writeUTF(CharSequence str) throws IOException { - int strlen = str.length(); - int utflen = 0; - int c; - - for (int i = 0; i < strlen; i++) { - c = str.charAt(i); - if ((c >= 0x0001) && (c <= 0x007F)) { - utflen++; - } else if (c > 0x07FF) { - utflen += 3; - } else { - utflen += 2; - } - } - - if (utflen > 65535) - throw new UTFDataFormatException("encoded string too long: " + utflen + " bytes"); - - ensureCapacity(utflen + 2); - buf[index++] = (byte) (utflen >>> 8); - buf[index++] = (byte) utflen; - - int i; - for (i = 0; i < strlen; i++) { - c = str.charAt(i); - if (!((c >= 0x0001) && (c <= 0x007F))) - break; - buf[index++] = (byte) c; - } - - for (; i < strlen; i++) { - c = str.charAt(i); - if ((c >= 0x0001) && (c <= 0x007F)) { - buf[index++] = (byte) c; - - } else if (c > 0x07FF) { - buf[index++] = (byte) (0xE0 | ((c >> 12) & 0x0F)); - buf[index++] = (byte) (0x80 | ((c >> 6) & 0x3F)); - buf[index++] = (byte) (0x80 | c & 0x3F); - } else { - buf[index++] = (byte) (0xC0 | ((c >> 6) & 0x1F)); - buf[index++] = (byte) (0x80 | c & 0x3F); - } - } - } - - @Override - public void write(int b) throws IOException { - ensureCapacity(1); - buf[index++] = (byte) b; - } - - @Override - public void write(byte[] b) throws IOException { - ensureCapacity(b.length); - System.arraycopy(b, 0, buf, index, b.length); - index += b.length; - } - - @Override - public void write(byte[] b, int off, int len) throws IOException { - ensureCapacity(len); - System.arraycopy(b, off, buf, this.index, len); - this.index += len; - } - - public void next() throws IOException { - // Write the length prefix - index -= 2; - - if (index > 0xFFFF) { - // Write a marker so we know the length is held in an int rather than a short - byte[] temp = new byte[4]; - temp[0] = (byte) 0xFF; - temp[1] = (byte) 0xFF; - temp[2] = (byte) (index >>> 24); - temp[3] = (byte) (index >>> 16); - out.write(temp); - buf[0] = (byte) (index >>> 8); - buf[1] = (byte) index; - } else { - buf[0] = (byte) (index >>> 8); - buf[1] = (byte) index; - } - // Dump out the whole buffer - out.write(buf, 0, index + 2); - // Reset the length - index = 2; - } - - @Override - public void flush() throws IOException { - out.flush(); - } - - private void ensureCapacity(int len) { - if (buf.length < this.index + len) { - // We need to grow our buffer - byte[] temp = new byte[(this.index + len) * 3 / 2]; - System.arraycopy(buf, 0, temp, 0, buf.length); - buf = temp; - } - } - - @Override - public void close() throws IOException { - out.close(); - } -} diff --git a/src/uk/me/parabola/splitter/disk/Member.java b/src/uk/me/parabola/splitter/disk/Member.java deleted file mode 100644 index 34cd086..0000000 --- a/src/uk/me/parabola/splitter/disk/Member.java +++ /dev/null @@ -1,40 +0,0 @@ -/* - * Copyright (c) 2009. - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 3 as - * published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - */ -package uk.me.parabola.splitter.disk; - -/** - * Represents a member of a relation - */ -public class Member { - private MemberType type; - private int id; - private String role; - - public Member(MemberType type, int id, String role) { - this.type = type; - this.id = id; - this.role = role; - } - - public MemberType getType() { - return type; - } - - public int getId() { - return id; - } - - public String getRole() { - return role; - } -} diff --git a/src/uk/me/parabola/splitter/disk/MemberType.java b/src/uk/me/parabola/splitter/disk/MemberType.java deleted file mode 100644 index 0f37659..0000000 --- a/src/uk/me/parabola/splitter/disk/MemberType.java +++ /dev/null @@ -1,21 +0,0 @@ -/* - * Copyright (c) 2009. - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 3 as - * published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - */ -package uk.me.parabola.splitter.disk; - -/** - * Represents the type of a relation's member - */ -public enum MemberType { - Node, Way -} - diff --git a/src/uk/me/parabola/splitter/disk/NodeStoreReader.java b/src/uk/me/parabola/splitter/disk/NodeStoreReader.java deleted file mode 100644 index 3a73c85..0000000 --- a/src/uk/me/parabola/splitter/disk/NodeStoreReader.java +++ /dev/null @@ -1,48 +0,0 @@ -/* - * Copyright (c) 2009. - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 3 as - * published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - */ -package uk.me.parabola.splitter.disk; - -import java.io.File; -import java.io.FileInputStream; -import java.io.IOException; -import java.io.InputStream; - -/** - * Reads in nodes from a binary format - */ -public class NodeStoreReader extends AbstractStoreReader { - private double lat; - private double lon; - - public NodeStoreReader(String filename) throws IOException { - this(new FileInputStream(new File(filename)), new KeyLookupReader(filename + ".keys")); - } - - public NodeStoreReader(InputStream in, KeyLookupReader keys) throws IOException { - super(in, keys); - } - - public double getLat() { - return lat; - } - - public double getLon() { - return lon; - } - - @Override - protected void readHeader() throws IOException { - lat = getIn().readDouble(); - lon = getIn().readDouble(); - } -} \ No newline at end of file diff --git a/src/uk/me/parabola/splitter/disk/NodeStoreWriter.java b/src/uk/me/parabola/splitter/disk/NodeStoreWriter.java deleted file mode 100644 index 0be262c..0000000 --- a/src/uk/me/parabola/splitter/disk/NodeStoreWriter.java +++ /dev/null @@ -1,37 +0,0 @@ -/* - * Copyright (c) 2009. - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 3 as - * published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - */ -package uk.me.parabola.splitter.disk; - -import java.io.FileOutputStream; -import java.io.IOException; -import java.io.OutputStream; - -/** - * Persists a collection of nodes to binary format on disk for later retrieval - */ -public class NodeStoreWriter extends AbstractStoreWriter { - - public NodeStoreWriter(String filename) throws IOException { - this(new FileOutputStream(filename), new KeyLookupWriter(filename + ".keys")); - } - - public NodeStoreWriter(OutputStream out, KeyLookupWriter keys) { - super(out, keys); - } - - public void write(int nodeId, double lat, double lon) throws IOException { - getOut().writeInt(nodeId); - getOut().writeDouble(lat); - getOut().writeDouble(lon); - } -} \ No newline at end of file diff --git a/src/uk/me/parabola/splitter/disk/RelationStoreReader.java b/src/uk/me/parabola/splitter/disk/RelationStoreReader.java deleted file mode 100644 index 748ef1d..0000000 --- a/src/uk/me/parabola/splitter/disk/RelationStoreReader.java +++ /dev/null @@ -1,54 +0,0 @@ -/* - * Copyright (c) 2009. - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 3 as - * published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - */ -package uk.me.parabola.splitter.disk; - -import java.io.File; -import java.io.FileInputStream; -import java.io.IOException; -import java.io.InputStream; -import java.util.ArrayList; -import java.util.List; - -/** - * Reads in relations from a binary format - */ -public class RelationStoreReader extends AbstractStoreReader { - private List<Member> members = new ArrayList<Member>(10); - private final KeyLookupReader roles; - - public RelationStoreReader(String filename) throws IOException { - this(new FileInputStream(new File(filename)), new KeyLookupReader(filename + ".keys"), new KeyLookupReader(filename + ".roles")); - } - - public RelationStoreReader(InputStream in, KeyLookupReader keys, KeyLookupReader roles) throws IOException { - super(in, keys); - this.roles = roles; - } - - public List<Member> getMembers() { - return members; - } - - @Override - protected void readHeader() throws IOException { - members.clear(); - byte typeByte; - while ((typeByte = getIn().readByte()) != 0) { - MemberType type = MemberType.values()[typeByte - 1]; - int id = getIn().readInt(); - short roleId = getIn().readShort(); - String role = roles.get(roleId); - members.add(new Member(type, id, role)); - } - } -} \ No newline at end of file diff --git a/src/uk/me/parabola/splitter/disk/RelationStoreWriter.java b/src/uk/me/parabola/splitter/disk/RelationStoreWriter.java deleted file mode 100644 index 5e513b9..0000000 --- a/src/uk/me/parabola/splitter/disk/RelationStoreWriter.java +++ /dev/null @@ -1,53 +0,0 @@ -/* - * Copyright (c) 2009. - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 3 as - * published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - */ -package uk.me.parabola.splitter.disk; - -import java.io.FileOutputStream; -import java.io.IOException; -import java.io.OutputStream; - -/** - * Persists a collection of relations to binary format on disk for later retrieval - */ -public class RelationStoreWriter extends AbstractStoreWriter { - private final KeyLookupWriter roles; - - public RelationStoreWriter(String filename) throws IOException { - this(new FileOutputStream(filename), new KeyLookupWriter(filename + ".keys"), new KeyLookupWriter(filename + ".roles")); - } - - public RelationStoreWriter(OutputStream out, KeyLookupWriter keys, KeyLookupWriter roles) { - super(out, keys); - this.roles = roles; - } - - public void write(int id) throws IOException { - getOut().writeInt(id); - } - - public void writeMember(MemberType type, int id, CharSequence role) throws IOException { - getOut().writeByte(type.ordinal() + 1); - getOut().writeInt(id); - getOut().writeShort(roles.set(role)); - } - - public void closeMembers() throws IOException { - getOut().writeByte(0); - } - - @Override - public void close() throws IOException { - super.close(); - roles.close(); - } -} \ No newline at end of file diff --git a/src/uk/me/parabola/splitter/disk/WayStoreReader.java b/src/uk/me/parabola/splitter/disk/WayStoreReader.java deleted file mode 100644 index 05a659e..0000000 --- a/src/uk/me/parabola/splitter/disk/WayStoreReader.java +++ /dev/null @@ -1,54 +0,0 @@ -/* - * Copyright (c) 2009. - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 3 as - * published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - */ -package uk.me.parabola.splitter.disk; - -import java.io.File; -import java.io.FileInputStream; -import java.io.IOException; -import java.io.InputStream; - -/** - * Reads in ways from a binary file format - */ -public class WayStoreReader extends AbstractStoreReader { - private int[] buf = new int[256]; - private int[] nodeIds; - - public WayStoreReader(String filename) throws IOException { - this(new FileInputStream(new File(filename)), new KeyLookupReader(filename + ".keys")); - } - - public WayStoreReader(InputStream in, KeyLookupReader keys) throws IOException { - super(in, keys); - } - - public int[] getNodeIds() { - return nodeIds; - } - - @Override - protected void readHeader() throws IOException { - int i = 0; - int id; - while ((id = getIn().readInt()) != 0) { - if (buf.length <= i) { - int[] temp = new int[(buf.length * 3) / 2 + 1]; - System.arraycopy(buf, 0, temp, 0, buf.length); - buf = temp; - } - buf[i++] = id; - } - nodeIds = new int[i]; - System.arraycopy(buf, 0, nodeIds, 0, i); - } -} \ No newline at end of file diff --git a/src/uk/me/parabola/splitter/disk/WayStoreWriter.java b/src/uk/me/parabola/splitter/disk/WayStoreWriter.java deleted file mode 100644 index c17d7d4..0000000 --- a/src/uk/me/parabola/splitter/disk/WayStoreWriter.java +++ /dev/null @@ -1,43 +0,0 @@ -/* - * Copyright (c) 2009. - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 3 as - * published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - */ -package uk.me.parabola.splitter.disk; - -import java.io.FileOutputStream; -import java.io.IOException; -import java.io.OutputStream; - -/** - * Writes out a sequence of ways to binary format for later retrieval - */ -public class WayStoreWriter extends AbstractStoreWriter { - - public WayStoreWriter(String filename) throws IOException { - this(new FileOutputStream(filename), new KeyLookupWriter(filename + ".keys")); - } - - public WayStoreWriter(OutputStream out, KeyLookupWriter keys) { - super(out, keys); - } - - public void write(int id) throws IOException { - getOut().writeInt(id); - } - - public void writeNodeRef(int id) throws IOException { - getOut().writeInt(id); - } - - public void closeNodeRefs() throws IOException { - getOut().writeInt(0); - } -} \ No newline at end of file diff --git a/test/uk/me/parabola/splitter/disk/TestKeyLookups.java b/test/uk/me/parabola/splitter/disk/TestKeyLookups.java deleted file mode 100644 index 782471c..0000000 --- a/test/uk/me/parabola/splitter/disk/TestKeyLookups.java +++ /dev/null @@ -1,51 +0,0 @@ -/* - * Copyright (c) 2009. - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 3 as - * published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - */ - -package uk.me.parabola.splitter.disk; - -import java.io.ByteArrayInputStream; -import java.io.ByteArrayOutputStream; -import java.io.IOException; -import java.io.InputStream; - -import org.testng.Assert; -import org.testng.annotations.Test; - -/** - * - */ -public class TestKeyLookups { - @Test - public void test() throws IOException { - ByteArrayOutputStream result = new ByteArrayOutputStream(); - KeyLookupWriter writer = new KeyLookupWriter(result); - writer.set("Testing1"); - writer.set("Testing2"); - writer.set("Testing3"); - writer.set("Testing4"); - writer.set("Euro: \u20AC Pound: \u00A3"); - writer.close(); - - byte[] data = result.toByteArray(); - InputStream in = new ByteArrayInputStream(data); - KeyLookupReader reader = new KeyLookupReader(in); - - Assert.assertEquals(reader.get(0), null); - Assert.assertEquals(reader.get(1), "Testing1"); - Assert.assertEquals(reader.get(2), "Testing2"); - Assert.assertEquals(reader.get(3), "Testing3"); - Assert.assertEquals(reader.get(4), "Testing4"); - Assert.assertEquals(reader.get(5), "Euro: \u20AC Pound: \u00A3"); - Assert.assertEquals(reader.get(6), null); - } -} \ No newline at end of file diff --git a/test/uk/me/parabola/splitter/disk/TestLengthPrefixStreams.java b/test/uk/me/parabola/splitter/disk/TestLengthPrefixStreams.java deleted file mode 100644 index b50e6bc..0000000 --- a/test/uk/me/parabola/splitter/disk/TestLengthPrefixStreams.java +++ /dev/null @@ -1,87 +0,0 @@ -/* - * Copyright (c) 2009. - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 3 as - * published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - */ - -package uk.me.parabola.splitter.disk; - -import java.io.ByteArrayInputStream; -import java.io.ByteArrayOutputStream; -import java.io.IOException; - -import org.testng.Assert; -import org.testng.annotations.Test; - -/** - * - */ -public class TestLengthPrefixStreams { - @Test - public void testOutput() throws IOException { - ByteArrayOutputStream result = new ByteArrayOutputStream(); - LengthPrefixOutputStream os = new LengthPrefixOutputStream(result, 2); // 2 so we test the buffer expansion - os.writeShort(0xABCD); - os.writeInt(0x12345678); - os.writeInt(0xAABBCCDD); - os.writeUTF("Testing"); - os.next(); - os.writeShort(0x9988); - os.writeByte(240); - os.next(); - os.close(); - - byte[] bytes = result.toByteArray(); - - // Test to see if the byte array contains what we'd expect - Assert.assertEquals(bytes, new byte[] { - 0x00, 0x13, // length prefix - (byte) 0xAB, (byte) 0xCD, // 0xABCD - 0x12, 0x34, 0x56, 0x78, // 0x12345678 - (byte) 0xAA, (byte) 0xBB, (byte) 0xCC, (byte) 0xDD, // 0xAABBCCDD - 0x00, 0x07, 'T', 'e', 's', 't', 'i', 'n', 'g', // "Testing" - 0x00, 0x03, // length prefix - (byte) 0x99, (byte) 0x88, // 0x9988 - (byte) 240, // 240 - }, - "Resultant byte array does not match expected result"); - } - - @Test - public void testinput() throws IOException { - byte[] buf = new byte[] { - 0x00, 0x13, // length prefix - (byte) 0xAB, (byte) 0xCD, // 0xABCD - 0x12, 0x34, 0x56, 0x78, // 0x12345678 - (byte) 0xAA, (byte) 0xBB, (byte) 0xCC, (byte) 0xDD, // 0xAABBCCDD - 0x00, 0x07, 'T', 'e', 's', 't', 'i', 'n', 'g', // "Testing" - 0x00, 0x04, // length prefix - (byte) 0x99, (byte) 0x88, // 0x9988 - (byte) 240, // 240 - (byte) 240, // 240 - }; - ByteArrayInputStream result = new ByteArrayInputStream(buf); - LengthPrefixInputStream is = new LengthPrefixInputStream(result, 2); // 2 so we test the buffer expansion - - Assert.assertEquals(is.next(), true, "Premature end of stream reached"); - Assert.assertEquals(is.remaining(), 0x13, "Unexpected number of bytes remaining in this segment"); - Assert.assertEquals(is.readUnsignedShort(), 0xABCD); - Assert.assertEquals(is.readInt(), 0x12345678); - Assert.assertEquals(is.readInt(), 0xAABBCCDD); - Assert.assertEquals(is.readUTF(), "Testing"); - Assert.assertEquals(is.remaining(), 0x00, "Unexpected number of bytes remaining in this segment"); - Assert.assertEquals(is.next(), true, "Premature end of stream reached"); - Assert.assertEquals(is.readShort(), (short) 0x9988); - Assert.assertEquals(is.readUnsignedByte(), 240); - Assert.assertEquals(is.readByte(), (byte) 240); - Assert.assertEquals(is.remaining(), 0x00, "Unexpected number of bytes remaining in this segment"); - Assert.assertEquals(is.next(), false, "Expected to reach the end of stream but didn't"); - } -} \ No newline at end of file diff --git a/test/uk/me/parabola/splitter/disk/TestStores.java b/test/uk/me/parabola/splitter/disk/TestStores.java deleted file mode 100644 index 9f1bbe5..0000000 --- a/test/uk/me/parabola/splitter/disk/TestStores.java +++ /dev/null @@ -1,216 +0,0 @@ -/* - * Copyright (c) 2009. - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 3 as - * published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - */ - -package uk.me.parabola.splitter.disk; - -import java.io.ByteArrayInputStream; -import java.io.ByteArrayOutputStream; -import java.io.IOException; -import java.util.List; -import java.util.Map; - -import org.testng.Assert; -import org.testng.annotations.Test; - -/** - * - */ -public class TestStores { - @Test - public void testNodeStore() throws IOException { - ByteArrayOutputStream nodeOut = new ByteArrayOutputStream(); - ByteArrayOutputStream keyOut = new ByteArrayOutputStream(); - KeyLookupWriter keyWriter = new KeyLookupWriter(keyOut); - - NodeStoreWriter nodeWriter = new NodeStoreWriter(nodeOut, keyWriter); - nodeWriter.write(1, 0.1234d, 99.125d); - nodeWriter.writeTag("key1", "value1"); - nodeWriter.writeTag("key2", "value2"); - nodeWriter.writeTag("key3", "Euro: \u20AC Pound: \u00A3"); - nodeWriter.closeTags(); - nodeWriter.next(); - nodeWriter.write(2, 5432.12d, 48484.48484d); - nodeWriter.writeTag("key2", "Euro: \u20AC"); - nodeWriter.writeTag("key4", "Pound: \u00A3"); - nodeWriter.closeTags(); - nodeWriter.next(); - - nodeWriter.close(); - - byte[] nodeBuf = nodeOut.toByteArray(); - byte[] keyBuf = keyOut.toByteArray(); - KeyLookupReader keyReader = new KeyLookupReader(new ByteArrayInputStream(keyBuf)); - ByteArrayInputStream nodeIn = new ByteArrayInputStream(nodeBuf); - NodeStoreReader nodeReader = new NodeStoreReader(nodeIn, keyReader); - - Assert.assertTrue(nodeReader.next()); - Assert.assertEquals(nodeReader.getId(), 1); - Assert.assertEquals(nodeReader.getLat(), 0.1234d); - Assert.assertEquals(nodeReader.getLon(), 99.125d); - Map<String, String> tags = nodeReader.getTags(); - Assert.assertEquals(tags.size(), 3); - Assert.assertEquals(tags.get("key1"), "value1"); - Assert.assertEquals(tags.get("key2"), "value2"); - Assert.assertEquals(tags.get("key3"), "Euro: \u20AC Pound: \u00A3"); - Assert.assertNull(tags.get("key4")); - - Assert.assertTrue(nodeReader.next()); - Assert.assertEquals(nodeReader.getId(), 2); - Assert.assertEquals(nodeReader.getLat(), 5432.12d); - Assert.assertEquals(nodeReader.getLon(), 48484.48484d); - tags = nodeReader.getTags(); - Assert.assertEquals(tags.size(), 2); - Assert.assertNull(tags.get("key1")); - Assert.assertEquals(tags.get("key2"), "Euro: \u20AC"); - Assert.assertNull(tags.get("key3")); - Assert.assertEquals(tags.get("key4"), "Pound: \u00A3"); - - Assert.assertFalse(nodeReader.next()); - } - - @Test - public void testWayStore() throws IOException { - ByteArrayOutputStream nodeOut = new ByteArrayOutputStream(); - ByteArrayOutputStream keyOut = new ByteArrayOutputStream(); - KeyLookupWriter keyWriter = new KeyLookupWriter(keyOut); - - WayStoreWriter wayWriter = new WayStoreWriter(nodeOut, keyWriter); - wayWriter.write(1); - wayWriter.writeNodeRef(10); - wayWriter.writeNodeRef(11); - wayWriter.writeNodeRef(12); - wayWriter.closeNodeRefs(); - wayWriter.writeTag("key1", "value1"); - wayWriter.writeTag("key2", "value2"); - wayWriter.writeTag("key3", "Euro: \u20AC Pound: \u00A3"); - wayWriter.closeTags(); - wayWriter.next(); - wayWriter.write(2); - wayWriter.closeNodeRefs(); - wayWriter.writeTag("key2", "Euro: \u20AC"); - wayWriter.writeTag("key4", "Pound: \u00A3"); - wayWriter.closeTags(); - wayWriter.next(); - - wayWriter.close(); - - byte[] wayBuf = nodeOut.toByteArray(); - byte[] keyBuf = keyOut.toByteArray(); - KeyLookupReader keyReader = new KeyLookupReader(new ByteArrayInputStream(keyBuf)); - ByteArrayInputStream wayIn = new ByteArrayInputStream(wayBuf); - WayStoreReader wayReader = new WayStoreReader(wayIn, keyReader); - - Assert.assertTrue(wayReader.next()); - Assert.assertEquals(wayReader.getId(), 1); - int[] nodeIds = wayReader.getNodeIds(); - Assert.assertEquals(nodeIds.length, 3); - Assert.assertEquals(nodeIds[0], 10); - Assert.assertEquals(nodeIds[1], 11); - Assert.assertEquals(nodeIds[2], 12); - Map<String, String> tags = wayReader.getTags(); - Assert.assertEquals(tags.size(), 3); - Assert.assertEquals(tags.get("key1"), "value1"); - Assert.assertEquals(tags.get("key2"), "value2"); - Assert.assertEquals(tags.get("key3"), "Euro: \u20AC Pound: \u00A3"); - Assert.assertNull(tags.get("key4")); - - Assert.assertTrue(wayReader.next()); - Assert.assertEquals(wayReader.getId(), 2); - nodeIds = wayReader.getNodeIds(); - Assert.assertEquals(nodeIds.length, 0); - tags = wayReader.getTags(); - Assert.assertEquals(tags.size(), 2); - Assert.assertNull(tags.get("key1")); - Assert.assertEquals(tags.get("key2"), "Euro: \u20AC"); - Assert.assertNull(tags.get("key3")); - Assert.assertEquals(tags.get("key4"), "Pound: \u00A3"); - - Assert.assertFalse(wayReader.next()); - } - - @Test - public void testRelStore() throws IOException { - ByteArrayOutputStream relOut = new ByteArrayOutputStream(); - ByteArrayOutputStream keyOut = new ByteArrayOutputStream(); - KeyLookupWriter keyWriter = new KeyLookupWriter(keyOut); - ByteArrayOutputStream roleOut = new ByteArrayOutputStream(); - KeyLookupWriter roleWriter = new KeyLookupWriter(roleOut); - - RelationStoreWriter relWriter = new RelationStoreWriter(relOut, keyWriter, roleWriter); - relWriter.write(1); - relWriter.writeMember(MemberType.Node, 5, "testRole1"); - relWriter.writeMember(MemberType.Node, 4, "testRole2"); - relWriter.writeMember(MemberType.Node, 3, "Euro: \u20AC"); - relWriter.writeMember(MemberType.Way, 3, "Pound: \u00A3"); - relWriter.closeMembers(); - relWriter.writeTag("key1", "value1"); - relWriter.writeTag("key2", "value2"); - relWriter.writeTag("key3", "Euro: \u20AC Pound: \u00A3"); - relWriter.closeTags(); - relWriter.next(); - relWriter.write(2); - relWriter.closeMembers(); - relWriter.writeTag("key2", "Euro: \u20AC"); - relWriter.writeTag("key4", "Pound: \u00A3"); - relWriter.closeTags(); - relWriter.next(); - - relWriter.close(); - - - byte[] relBuf = relOut.toByteArray(); - byte[] keyBuf = keyOut.toByteArray(); - KeyLookupReader keyReader = new KeyLookupReader(new ByteArrayInputStream(keyBuf)); - byte[] roleBuf = roleOut.toByteArray(); - KeyLookupReader roleReader = new KeyLookupReader(new ByteArrayInputStream(roleBuf)); - - ByteArrayInputStream relIn = new ByteArrayInputStream(relBuf); - RelationStoreReader relReader = new RelationStoreReader(relIn, keyReader, roleReader); - - Assert.assertTrue(relReader.next()); - Assert.assertEquals(relReader.getId(), 1); - List<Member> members = relReader.getMembers(); - Assert.assertEquals(members.size(), 4); - Assert.assertEquals(members.get(0).getType(), MemberType.Node); - Assert.assertEquals(members.get(0).getId(), 5); - Assert.assertEquals(members.get(0).getRole(), "testRole1"); - Assert.assertEquals(members.get(1).getType(), MemberType.Node); - Assert.assertEquals(members.get(1).getId(), 4); - Assert.assertEquals(members.get(1).getRole(), "testRole2"); - Assert.assertEquals(members.get(2).getType(), MemberType.Node); - Assert.assertEquals(members.get(2).getId(), 3); - Assert.assertEquals(members.get(2).getRole(), "Euro: \u20AC"); - Assert.assertEquals(members.get(3).getType(), MemberType.Way); - Assert.assertEquals(members.get(3).getId(), 3); - Assert.assertEquals(members.get(3).getRole(), "Pound: \u00A3"); - Map<String, String> tags = relReader.getTags(); - Assert.assertEquals(tags.size(), 3); - Assert.assertEquals(tags.get("key1"), "value1"); - Assert.assertEquals(tags.get("key2"), "value2"); - Assert.assertEquals(tags.get("key3"), "Euro: \u20AC Pound: \u00A3"); - Assert.assertNull(tags.get("key4")); - - Assert.assertTrue(relReader.next()); - Assert.assertEquals(relReader.getId(), 2); - members = relReader.getMembers(); - Assert.assertEquals(members.size(), 0); - tags = relReader.getTags(); - Assert.assertEquals(tags.size(), 2); - Assert.assertNull(tags.get("key1")); - Assert.assertEquals(tags.get("key2"), "Euro: \u20AC"); - Assert.assertNull(tags.get("key3")); - Assert.assertEquals(tags.get("key4"), "Pound: \u00A3"); - - Assert.assertFalse(relReader.next()); - } -} \ No newline at end of file -- 1.7.2.3
From: Scott Crosby <scrosby@cs.rice.edu> --- src/uk/me/parabola/splitter/IntList.java | 9 +++++++++ 1 files changed, 9 insertions(+), 0 deletions(-) diff --git a/src/uk/me/parabola/splitter/IntList.java b/src/uk/me/parabola/splitter/IntList.java index 9d2b86a..39918a8 100644 --- a/src/uk/me/parabola/splitter/IntList.java +++ b/src/uk/me/parabola/splitter/IntList.java @@ -16,6 +16,10 @@ */ package uk.me.parabola.splitter; +import java.util.Arrays; + +import com.sun.xml.internal.bind.v2.runtime.unmarshaller.XsiNilLoader.Array; + /** * Maintains a list of int primitives. */ @@ -57,4 +61,9 @@ public class IntList { System.arraycopy(temp, 0, data, 0, size); } } + + /** Get as a read-only array of integers */ + int []asArray() { + return Arrays.copyOf(data,size); + } } \ No newline at end of file -- 1.7.2.3
From: Scott Crosby <scrosby@cs.rice.edu> --- .../me/parabola/splitter/DensityMapCollector.java | 11 ++------ src/uk/me/parabola/splitter/MapProcessor.java | 26 +++---------------- src/uk/me/parabola/splitter/NodeCollector.java | 12 ++------- src/uk/me/parabola/splitter/OSMParser.java | 10 ++++--- src/uk/me/parabola/splitter/SplitProcessor.java | 20 +++------------ 5 files changed, 20 insertions(+), 59 deletions(-) diff --git a/src/uk/me/parabola/splitter/DensityMapCollector.java b/src/uk/me/parabola/splitter/DensityMapCollector.java index e77e3d4..ef13a01 100644 --- a/src/uk/me/parabola/splitter/DensityMapCollector.java +++ b/src/uk/me/parabola/splitter/DensityMapCollector.java @@ -48,9 +48,9 @@ class DensityMapCollector implements MapCollector { } @Override - public void startNode(int id, double lat, double lon) { - int glat = Utils.toMapUnit(lat); - int glon = Utils.toMapUnit(lon); + public void processNode(Node n) { + int glat = Utils.toMapUnit(n.getLat()); + int glon = Utils.toMapUnit(n.getLon()); densityMap.addNode(glat, glon); details.addToBounds(glat, glon); } @@ -61,8 +61,6 @@ class DensityMapCollector implements MapCollector { @Override public void startRelation(int id) {} - @Override - public void nodeTag(String key, String value) {} @Override public void wayTag(String key, String value) {} @@ -80,9 +78,6 @@ class DensityMapCollector implements MapCollector { public void relationWay(int wayId, String role) {} @Override - public void endNode() {} - - @Override public void endWay() {} @Override diff --git a/src/uk/me/parabola/splitter/MapProcessor.java b/src/uk/me/parabola/splitter/MapProcessor.java index ce3e3a1..536bcb1 100644 --- a/src/uk/me/parabola/splitter/MapProcessor.java +++ b/src/uk/me/parabola/splitter/MapProcessor.java @@ -34,14 +34,6 @@ public interface MapProcessor { void boundTag(Area bounds); /** - * Called when a node is encountered. - * @param id the node's ID. - * @param lat the node's latitude. - * @param lon the node's longitude. - */ - void startNode(int id, double lat, double lon); - - /** * Called when a way is encountered. * @param id the way's ID. */ @@ -53,14 +45,11 @@ public interface MapProcessor { */ void startRelation(int id); + /** - * Called when a tag is encountered on a node. This method will be - * called for every tag associated with the node that was specified - * in the most recent call to {@link #startNode(int, double, double)}. - * @param key the tag's key. - * @param value the tag's value. - */ - void nodeTag(String key, String value); + * Called when a whole node has been processed. + */ + void processNode(Node n); /** * Called when a tag is encountered on a way. This method will be @@ -105,13 +94,6 @@ public interface MapProcessor { void relationWay(int wayId, String role); /** - * Called when processing is complete for a node. This method will be called once - * there is no further data available for the node specified in the most recent - * call to {@link #startNode(int, double, double)}. - */ - void endNode(); - - /** * Called when processing is complete for a way. This method will be called once * there is no further data available for the way specified in the most recent * call to {@link #startWay(int)}. diff --git a/src/uk/me/parabola/splitter/NodeCollector.java b/src/uk/me/parabola/splitter/NodeCollector.java index 4cc3d71..6556ee6 100644 --- a/src/uk/me/parabola/splitter/NodeCollector.java +++ b/src/uk/me/parabola/splitter/NodeCollector.java @@ -36,12 +36,12 @@ class NodeCollector implements MapCollector { } @Override - public void startNode(int id, double lat, double lon) { + public void processNode(Node n) { // Since we are rounding areas to fit on a low zoom boundary we // can drop the bottom 8 bits of the lat and lon and then fit // the whole lot into a single int. - int glat = Utils.toMapUnit(lat); - int glon = Utils.toMapUnit(lon); + int glat = Utils.toMapUnit(n.getLat()); + int glon = Utils.toMapUnit(n.getLon()); int coord = ((glat << 8) & 0xffff0000) + ((glon >> 8) & 0xffff); coords.add(coord); @@ -55,9 +55,6 @@ class NodeCollector implements MapCollector { public void startRelation(int id) {} @Override - public void nodeTag(String key, String value) {} - - @Override public void wayTag(String key, String value) {} @Override @@ -73,9 +70,6 @@ class NodeCollector implements MapCollector { public void relationWay(int wayId, String role) {} @Override - public void endNode() {} - - @Override public void endWay() {} @Override diff --git a/src/uk/me/parabola/splitter/OSMParser.java b/src/uk/me/parabola/splitter/OSMParser.java index c902909..80f0c29 100644 --- a/src/uk/me/parabola/splitter/OSMParser.java +++ b/src/uk/me/parabola/splitter/OSMParser.java @@ -29,6 +29,8 @@ class OSMParser extends AbstractXppParser implements MapReader { Node, Way, Relation, None } + private Node currentNode = new Node(); + private final MapProcessor processor; // There are mixed nodes and ways in the file @@ -139,7 +141,8 @@ class OSMParser extends AbstractXppParser implements MapReader { maxNodeId = id; } - processor.startNode(id, lat, lon); + currentNode = new Node(); + currentNode.set(id, lat, lon); state = State.Node; } @@ -155,7 +158,7 @@ class OSMParser extends AbstractXppParser implements MapReader { private void processNode(CharSequence name) { if (name.equals("tag")) { - processor.nodeTag(getAttr("k"), getAttr("v")); + currentNode.addTag(getAttr("k"), getAttr("v")); } } @@ -235,8 +238,7 @@ class OSMParser extends AbstractXppParser implements MapReader { public void endElement(String name) { if (state == State.Node) { if (name.equals("node")) { - if (!startNodeOnly) - processor.endNode(); + processor.processNode(currentNode); state = State.None; nodeCount++; if (nodeCount % NODE_STATUS_UPDATE_THRESHOLD == 0) { diff --git a/src/uk/me/parabola/splitter/SplitProcessor.java b/src/uk/me/parabola/splitter/SplitProcessor.java index 92578df..f02c1af 100644 --- a/src/uk/me/parabola/splitter/SplitProcessor.java +++ b/src/uk/me/parabola/splitter/SplitProcessor.java @@ -33,7 +33,6 @@ class SplitProcessor implements MapProcessor { private final BlockingQueue<InputQueueInfo> writerInputQueue; private final ArrayList<Thread> workerThreads; - private Node currentNode = new Node(); private int currentNodeAreaSet; private Way currentWay = new Way(); @@ -77,11 +76,6 @@ class SplitProcessor implements MapProcessor { } @Override - public void startNode(int id, double lat, double lon) { - currentNode.set(id, lat, lon); - } - - @Override public void startWay(int id) { currentWay.set(id); } @@ -92,11 +86,6 @@ class SplitProcessor implements MapProcessor { } @Override - public void nodeTag(String key, String value) { - currentNode.addTag(key, value); - } - - @Override public void wayTag(String key, String value) { currentWay.addTag(key, value); } @@ -172,13 +161,12 @@ class SplitProcessor implements MapProcessor { } @Override - public void endNode() { + public void processNode(Node n) { try { - writeNode(); - currentNode = new Node(); + writeNode(n); currentNodeAreaSet = 0; } catch (IOException e) { - throw new RuntimeException("failed to write node " + currentNode.getId(), e); + throw new RuntimeException("failed to write node " + n.getId(), e); } } @@ -225,7 +213,7 @@ class SplitProcessor implements MapProcessor { } } - private void writeNode() throws IOException { + private void writeNode(Node currentNode) throws IOException { for (int n = 0; n < writers.length; n++) { boolean found = writers[n].nodeBelongsToThisArea(currentNode); if (found) { -- 1.7.2.3
From: Scott Crosby <scrosby@cs.rice.edu> --- .../me/parabola/splitter/DensityMapCollector.java | 12 +---- src/uk/me/parabola/splitter/MapProcessor.java | 32 +---------- src/uk/me/parabola/splitter/NodeCollector.java | 11 +---- src/uk/me/parabola/splitter/OSMParser.java | 10 ++-- src/uk/me/parabola/splitter/SplitProcessor.java | 58 ++++++++------------ 5 files changed, 32 insertions(+), 91 deletions(-) diff --git a/src/uk/me/parabola/splitter/DensityMapCollector.java b/src/uk/me/parabola/splitter/DensityMapCollector.java index ef13a01..b4a5953 100644 --- a/src/uk/me/parabola/splitter/DensityMapCollector.java +++ b/src/uk/me/parabola/splitter/DensityMapCollector.java @@ -56,29 +56,19 @@ class DensityMapCollector implements MapCollector { } @Override - public void startWay(int id) {} - - @Override public void startRelation(int id) {} - - @Override - public void wayTag(String key, String value) {} - @Override public void relationTag(String key, String value) {} @Override - public void wayNode(int nodeId) {} - - @Override public void relationNode(int nodeId, String role) {} @Override public void relationWay(int wayId, String role) {} @Override - public void endWay() {} + public void processWay(Way w) {} @Override public void endRelation() {} diff --git a/src/uk/me/parabola/splitter/MapProcessor.java b/src/uk/me/parabola/splitter/MapProcessor.java index 536bcb1..b179d85 100644 --- a/src/uk/me/parabola/splitter/MapProcessor.java +++ b/src/uk/me/parabola/splitter/MapProcessor.java @@ -34,12 +34,6 @@ public interface MapProcessor { void boundTag(Area bounds); /** - * Called when a way is encountered. - * @param id the way's ID. - */ - void startWay(int id); - - /** * Called when a relation is encountered. * @param id the relation's ID. */ @@ -52,15 +46,6 @@ public interface MapProcessor { void processNode(Node n); /** - * Called when a tag is encountered on a way. This method will be - * called for every tag associated with the way that was specified - * in the most recent call to {@link #startWay(int)}. - * @param key the tag's key. - * @param value the tag's value. - */ - void wayTag(String key, String value); - - /** * Called when a tag is encountered on a relation. This method will * be called for every tag associated with the relation that was * specified in the most recent call to {@link #startRelation(int)}. @@ -70,14 +55,6 @@ public interface MapProcessor { void relationTag(String key, String value); /** - * Called when a reference to a node is encountered within a way. This - * method will be called for every node associated with the way that was - * specified in the most recent call to {@link #startWay(int)} . - * @param nodeId the ID of the node. - */ - void wayNode(int nodeId); - - /** * Called when a reference to a node is encountered within a relation. * This method will be called for every node that is associated with the * relation that was specified in the most recent call to {@link #startRelation(int)} . @@ -93,13 +70,8 @@ public interface MapProcessor { */ void relationWay(int wayId, String role); - /** - * Called when processing is complete for a way. This method will be called once - * there is no further data available for the way specified in the most recent - * call to {@link #startWay(int)}. - */ - void endWay(); - + void processWay(Way w); + /** * Called when processing is complete for a relation. This method will be called once * there is no further data available for the relation specified in the most recent diff --git a/src/uk/me/parabola/splitter/NodeCollector.java b/src/uk/me/parabola/splitter/NodeCollector.java index 6556ee6..2607d18 100644 --- a/src/uk/me/parabola/splitter/NodeCollector.java +++ b/src/uk/me/parabola/splitter/NodeCollector.java @@ -49,28 +49,19 @@ class NodeCollector implements MapCollector { } @Override - public void startWay(int id) {} - - @Override public void startRelation(int id) {} @Override - public void wayTag(String key, String value) {} - - @Override public void relationTag(String key, String value) {} @Override - public void wayNode(int nodeId) {} - - @Override public void relationNode(int nodeId, String role) {} @Override public void relationWay(int wayId, String role) {} @Override - public void endWay() {} + public void processWay(Way w) {} @Override public void endRelation() {} diff --git a/src/uk/me/parabola/splitter/OSMParser.java b/src/uk/me/parabola/splitter/OSMParser.java index 80f0c29..081bc83 100644 --- a/src/uk/me/parabola/splitter/OSMParser.java +++ b/src/uk/me/parabola/splitter/OSMParser.java @@ -30,6 +30,7 @@ class OSMParser extends AbstractXppParser implements MapReader { } private Node currentNode = new Node(); + private Way currentWay = new Way(); private final MapProcessor processor; @@ -147,7 +148,8 @@ class OSMParser extends AbstractXppParser implements MapReader { } private void startWay() { - processor.startWay(getIntAttr("id")); + currentWay = new Way(); + currentWay.set(getIntAttr("id")); state = State.Way; } @@ -164,9 +166,9 @@ class OSMParser extends AbstractXppParser implements MapReader { private void processWay(CharSequence name) { if (name.equals("nd")) { - processor.wayNode(getIntAttr("ref")); + currentWay.addRef(getIntAttr("ref")); } else if (name.equals("tag")) { - processor.wayTag(getAttr("k"), getAttr("v")); + currentWay.addTag(getAttr("k"), getAttr("v")); } } @@ -248,7 +250,7 @@ class OSMParser extends AbstractXppParser implements MapReader { } else if (state == State.Way) { if (name.equals("way")) { if (!startNodeOnly) - processor.endWay(); + processor.processWay(currentWay); state = State.None; wayCount++; if (wayCount % WAY_STATUS_UPDATE_THRESHOLD == 0) { diff --git a/src/uk/me/parabola/splitter/SplitProcessor.java b/src/uk/me/parabola/splitter/SplitProcessor.java index f02c1af..693ebc5 100644 --- a/src/uk/me/parabola/splitter/SplitProcessor.java +++ b/src/uk/me/parabola/splitter/SplitProcessor.java @@ -34,8 +34,6 @@ class SplitProcessor implements MapProcessor { private final ArrayList<Thread> workerThreads; private int currentNodeAreaSet; - - private Way currentWay = new Way(); private BitSet currentWayAreaSet; private Relation currentRelation = new Relation(); @@ -76,45 +74,16 @@ class SplitProcessor implements MapProcessor { } @Override - public void startWay(int id) { - currentWay.set(id); - } - - @Override public void startRelation(int id) { currentRelation.set(id); } @Override - public void wayTag(String key, String value) { - currentWay.addTag(key, value); - } - - @Override public void relationTag(String key, String value) { currentRelation.addTag(key, value); } @Override - public void wayNode(int id) { - // Get the list of areas that the node is in. A node may be in - // more than one area because of overlap. - int set = coords.get(id); - - // add the list of areas to the currentWayAreaSet - if (set != 0) { - int mask = 0xff; - for (int slot = 0; slot < 4; slot++, mask <<= 8) { - int val = (set & mask) >>> (slot * 8); - if (val == 0) - break; - currentWayAreaSet.set(val - 1); - } - } - currentWay.addRef(id); - } - - @Override public void relationNode(int id, String role) { { currentRelation.addMember("node", id, role); @@ -171,13 +140,30 @@ class SplitProcessor implements MapProcessor { } @Override - public void endWay() { + public void processWay(Way w) { + + for (int id: w.getRefs().asArray()) { + // Get the list of areas that the node is in. A node may be in + // more than one area because of overlap. + int set = coords.get(id); + + // add the list of areas to the currentWayAreaSet + if (set != 0) { + int mask = 0xff; + for (int slot = 0; slot < 4; slot++, mask <<= 8) { + int val = (set & mask) >>> (slot * 8); + if (val == 0) + break; + currentWayAreaSet.set(val - 1); + } + } + } + try { - writeWay(); - currentWay = new Way(); + writeWay(w); currentWayAreaSet.clear(); } catch (IOException e) { - throw new RuntimeException("failed to write way " + currentWay.getId(), e); + throw new RuntimeException("failed to write way " + w.getId(), e); } } @@ -237,7 +223,7 @@ class SplitProcessor implements MapProcessor { private boolean seenWay; - private void writeWay() throws IOException { + private void writeWay(Way currentWay) throws IOException { if (!seenWay) { seenWay = true; System.out.println("Writing ways " + new Date()); -- 1.7.2.3
From: Scott Crosby <scrosby@cs.rice.edu> --- .../me/parabola/splitter/DensityMapCollector.java | 14 +------ src/uk/me/parabola/splitter/MapProcessor.java | 39 +------------------ src/uk/me/parabola/splitter/NodeCollector.java | 14 +------ src/uk/me/parabola/splitter/OSMParser.java | 14 +++--- src/uk/me/parabola/splitter/SplitProcessor.java | 41 +++++++++----------- 5 files changed, 29 insertions(+), 93 deletions(-) diff --git a/src/uk/me/parabola/splitter/DensityMapCollector.java b/src/uk/me/parabola/splitter/DensityMapCollector.java index b4a5953..b94eda8 100644 --- a/src/uk/me/parabola/splitter/DensityMapCollector.java +++ b/src/uk/me/parabola/splitter/DensityMapCollector.java @@ -56,22 +56,10 @@ class DensityMapCollector implements MapCollector { } @Override - public void startRelation(int id) {} - - @Override - public void relationTag(String key, String value) {} - - @Override - public void relationNode(int nodeId, String role) {} - - @Override - public void relationWay(int wayId, String role) {} - - @Override public void processWay(Way w) {} @Override - public void endRelation() {} + public void processRelation(Relation r) {} @Override public void endMap() {} diff --git a/src/uk/me/parabola/splitter/MapProcessor.java b/src/uk/me/parabola/splitter/MapProcessor.java index b179d85..2bd020f 100644 --- a/src/uk/me/parabola/splitter/MapProcessor.java +++ b/src/uk/me/parabola/splitter/MapProcessor.java @@ -33,51 +33,16 @@ public interface MapProcessor { */ void boundTag(Area bounds); - /** - * Called when a relation is encountered. - * @param id the relation's ID. - */ - void startRelation(int id); - /** * Called when a whole node has been processed. */ void processNode(Node n); - /** - * Called when a tag is encountered on a relation. This method will - * be called for every tag associated with the relation that was - * specified in the most recent call to {@link #startRelation(int)}. - * @param key the tag's key. - * @param value the tag's value. - */ - void relationTag(String key, String value); - - /** - * Called when a reference to a node is encountered within a relation. - * This method will be called for every node that is associated with the - * relation that was specified in the most recent call to {@link #startRelation(int)} . - * @param nodeId the ID of the node. - */ - void relationNode(int nodeId, String role); - - /** - * Called when a reference to a way is encountered within a relation. - * This method will be called for every way that is associated with the relation - * that was specified in the most recent call to {@link #startRelation(int)} . - * @param nodeId the ID of the node. - */ - void relationWay(int wayId, String role); - void processWay(Way w); - /** - * Called when processing is complete for a relation. This method will be called once - * there is no further data available for the relation specified in the most recent - * call to {@link #startRelation(int)}. - */ - void endRelation(); + void processRelation(Relation w); + /** * Called once the entire map has finished processing. diff --git a/src/uk/me/parabola/splitter/NodeCollector.java b/src/uk/me/parabola/splitter/NodeCollector.java index 2607d18..409545f 100644 --- a/src/uk/me/parabola/splitter/NodeCollector.java +++ b/src/uk/me/parabola/splitter/NodeCollector.java @@ -49,22 +49,10 @@ class NodeCollector implements MapCollector { } @Override - public void startRelation(int id) {} - - @Override - public void relationTag(String key, String value) {} - - @Override - public void relationNode(int nodeId, String role) {} - - @Override - public void relationWay(int wayId, String role) {} - - @Override public void processWay(Way w) {} @Override - public void endRelation() {} + public void processRelation(Relation r) {} @Override public void endMap() {} diff --git a/src/uk/me/parabola/splitter/OSMParser.java b/src/uk/me/parabola/splitter/OSMParser.java index 081bc83..50569be 100644 --- a/src/uk/me/parabola/splitter/OSMParser.java +++ b/src/uk/me/parabola/splitter/OSMParser.java @@ -31,6 +31,7 @@ class OSMParser extends AbstractXppParser implements MapReader { private Node currentNode = new Node(); private Way currentWay = new Way(); + private Relation currentRelation = new Relation(); private final MapProcessor processor; @@ -154,7 +155,8 @@ class OSMParser extends AbstractXppParser implements MapReader { } private void startRelation() { - processor.startRelation(getIntAttr("id")); + currentRelation = new Relation(); + currentRelation.set(getIntAttr("id")); state = State.Relation; } @@ -174,15 +176,13 @@ class OSMParser extends AbstractXppParser implements MapReader { private void processRelation(CharSequence name) { if (name.equals("tag")) { - processor.relationTag(getAttr("k"), getAttr("v")); + currentRelation.addTag(getAttr("k"), getAttr("v")); } else if (name.equals("member")) { String type = getAttr("type"); int id = getIntAttr("ref"); String role = getAttr("role"); - if ("node".equals(type)) { - processor.relationNode(id, role); - } else if ("way".equals(type)) { - processor.relationWay(id, role); + if ("node".equals(type) || "way".equals(type)) { + currentRelation.addMember(type, id, role); } } } @@ -260,7 +260,7 @@ class OSMParser extends AbstractXppParser implements MapReader { } else if (state == State.Relation) { if (name.equals("relation")) { if (!startNodeOnly) - processor.endRelation(); + processor.processRelation(currentRelation); state = State.None; relationCount++; if (relationCount % RELATION_STATUS_UPDATE_THRESHOLD == 0) { diff --git a/src/uk/me/parabola/splitter/SplitProcessor.java b/src/uk/me/parabola/splitter/SplitProcessor.java index 693ebc5..24d23db 100644 --- a/src/uk/me/parabola/splitter/SplitProcessor.java +++ b/src/uk/me/parabola/splitter/SplitProcessor.java @@ -19,6 +19,8 @@ import java.util.Date; import java.util.concurrent.ArrayBlockingQueue; import java.util.concurrent.BlockingQueue; +import uk.me.parabola.splitter.Relation.Member; + /** * Splits a map into multiple areas. */ @@ -35,8 +37,6 @@ class SplitProcessor implements MapProcessor { private int currentNodeAreaSet; private BitSet currentWayAreaSet; - - private Relation currentRelation = new Relation(); private BitSet currentRelAreaSet; private final int maxThreads; @@ -73,20 +73,8 @@ class SplitProcessor implements MapProcessor { public void boundTag(Area bounds) { } - @Override - public void startRelation(int id) { - currentRelation.set(id); - } - - @Override - public void relationTag(String key, String value) { - currentRelation.addTag(key, value); - } - - @Override - public void relationNode(int id, String role) { + private void relationNode(int id, String role) { { - currentRelation.addMember("node", id, role); int set = coords.get(id); if (set != 0) { int mask = 0xff; @@ -101,11 +89,9 @@ class SplitProcessor implements MapProcessor { } } - @Override - public void relationWay(int id, String role) { + private void relationWay(int id, String role) { { long[] bigSet; - currentRelation.addMember("way", id, role); int set = ways.get(id); if (set != 0) { int mask = 0xff; @@ -168,13 +154,22 @@ class SplitProcessor implements MapProcessor { } @Override - public void endRelation() { + public void processRelation(Relation r) { try { - writeRelation(); - currentRelation = new Relation(); + for (Member mem : r.getMembers()) { + String role = mem.getRole(); + int id = mem.getRef(); + if (mem.getType().equals("node")) { + relationNode(id,role); + } else if (mem.getType().equals("way")){ + relationWay(id,role); + } + } + + writeRelation(r); currentRelAreaSet.clear(); } catch (IOException e) { - throw new RuntimeException("failed to write relation " + currentRelation.getId(), e); + throw new RuntimeException("failed to write relation " + r.getId(), e); } } @@ -262,7 +257,7 @@ class SplitProcessor implements MapProcessor { private boolean seenRel; - private void writeRelation() throws IOException { + private void writeRelation(Relation currentRelation) throws IOException { if (!seenRel) { seenRel = true; System.out.println("Writing relations " + new Date()); -- 1.7.2.3
From: Scott Crosby <scrosby@cs.rice.edu> --- src/uk/me/parabola/splitter/BinaryMapParser.java | 167 ++++++++++++++++++++++ src/uk/me/parabola/splitter/Main.java | 56 ++++++-- src/uk/me/parabola/splitter/MapReader.java | 10 +- src/uk/me/parabola/splitter/OSMParser.java | 6 +- 4 files changed, 216 insertions(+), 23 deletions(-) create mode 100644 src/uk/me/parabola/splitter/BinaryMapParser.java diff --git a/src/uk/me/parabola/splitter/BinaryMapParser.java b/src/uk/me/parabola/splitter/BinaryMapParser.java new file mode 100644 index 0000000..4d76b01 --- /dev/null +++ b/src/uk/me/parabola/splitter/BinaryMapParser.java @@ -0,0 +1,167 @@ +package uk.me.parabola.splitter; + +import java.util.List; + + +import crosby.binary.BinaryParser; +import crosby.binary.Osmformat; + +public class BinaryMapParser extends BinaryParser { + // How many elements to process before displaying a status update + private static final int NODE_STATUS_UPDATE_THRESHOLD = 2500000; + private static final int WAY_STATUS_UPDATE_THRESHOLD = 500000; + private static final int RELATION_STATUS_UPDATE_THRESHOLD = 50000; + + + private long nodeCount; + private long wayCount; + private long relationCount; + + BinaryMapParser(MapProcessor processor) { + this.processor = processor; + } + MapProcessor processor; + + public void complete() { + // End of map is sent when all input files are processed. + // So do nothing. + } + + // Per-block state for parsing, set when processing the header of a block; + protected void parseDense(Osmformat.DenseNodes nodes) { + long last_id = 0, last_lat = 0, last_lon = 0; + int j = 0; + for (int i=0 ; i < nodes.getIdCount(); i++) { + Node tmp = new Node(); + long lat = nodes.getLat(i)+last_lat; last_lat = lat; + long lon = nodes.getLon(i)+last_lon; last_lon = lon; + long id = nodes.getId(i)+last_id; last_id = id; + double latf = parseLat(lat), lonf = parseLon(lon); + tmp = new Node(); + tmp.set((int)id, latf, lonf); + if (nodes.getKeysValsCount() > 0) { + while (nodes.getKeysVals(j) != 0) { + int keyid = nodes.getKeysVals(j++); + int valid = nodes.getKeysVals(j++); + tmp.addTag(getStringById(keyid),getStringById(valid)); + } + j++; // Skip over the '0' delimiter. + } + processor.processNode(tmp); + processNodes(); + } + } + + protected void parseNodes(List<Osmformat.Node> nodes) { + for (Osmformat.Node i : nodes) { + Node tmp = new Node(); + for (int j=0 ; j < i.getKeysCount(); j++) + tmp.addTag(getStringById(i.getKeys(j)),getStringById(i.getVals(j))); + long id = i.getId(); + double latf = parseLat(i.getLat()), lonf = parseLon(i.getLon()); + + tmp.set((int)id, latf, lonf); + + processor.processNode(tmp); + processNodes(); + } + } + + + protected void parseWays(List<Osmformat.Way> ways) { + for (Osmformat.Way i : ways) { + Way tmp = new Way(); + for (int j=0 ; j < i.getKeysCount(); j++) + tmp.addTag(getStringById(i.getKeys(j)),getStringById(i.getVals(j))); + + long last_id=0; + for (long j : i.getRefsList()) { + tmp.addRef((int)(j+last_id)); + last_id = j+last_id; + } + + long id = i.getId(); + tmp.set((int)id); + + processor.processWay(tmp); + processWays(); + } + } + protected void parseRelations(List<Osmformat.Relation> rels) { + for (Osmformat.Relation i : rels) { + Relation tmp = new Relation(); + for (int j=0 ; j < i.getKeysCount(); j++) + tmp.addTag(getStringById(i.getKeys(j)),getStringById(i.getVals(j))); + + long id = i.getId(); + tmp.set((int)id); + + long last_mid=0; + for (int j =0; j < i.getMemidsCount() ; j++) { + long mid = last_mid + i.getMemids(j); + last_mid = mid; + String role = getStringById(i.getRolesSid(j)); + String etype=null; + + if (i.getTypes(j) == Osmformat.Relation.MemberType.NODE) + etype = "node"; + else if (i.getTypes(j) == Osmformat.Relation.MemberType.WAY) + etype = "way"; + else if (i.getTypes(j) == Osmformat.Relation.MemberType.RELATION) + continue; + else + assert false; // TODO; Illegal file? + + tmp.addMember(etype,(int)mid,role); + } + processor.processRelation(tmp); + processRelations(); + } + } + + public void parse(Osmformat.HeaderBlock block) { + double multiplier = .000000001; + double rightf = block.getBbox().getRight() * multiplier; + double leftf = block.getBbox().getLeft() * multiplier; + double topf = block.getBbox().getTop() * multiplier; + double bottomf = block.getBbox().getBottom() * multiplier; + + for (String s : block.getRequiredFeaturesList()) { + if (s.equals("OsmSchema-V0.6")) continue; // OK. + if (s.equals("DenseNodes")) continue; // OK. + throw new Error("File requires unknown feature: " + s); + } + + System.out.println("Bounding box "+leftf+" "+bottomf+" "+rightf+" "+topf); + + Area area = new Area( + Utils.toMapUnit(bottomf), + Utils.toMapUnit(leftf), + Utils.toMapUnit(topf), + Utils.toMapUnit(rightf)); + processor.boundTag(area); + } + + + private void processNodes() { + nodeCount++; + if (nodeCount % NODE_STATUS_UPDATE_THRESHOLD == 0) { + System.out.println(Utils.format(nodeCount) + " nodes processed..."); + } + +} + +private void processWays() { + wayCount++; + if (wayCount % WAY_STATUS_UPDATE_THRESHOLD == 0) { + System.out.println(Utils.format(wayCount) + " ways processed..."); + } +} +private void processRelations() { + relationCount++; + if (relationCount % RELATION_STATUS_UPDATE_THRESHOLD == 0) { + System.out.println(Utils.format(relationCount) + " ways processed..."); + } +} + +} diff --git a/src/uk/me/parabola/splitter/Main.java b/src/uk/me/parabola/splitter/Main.java index 84e5dd7..b5469c6 100644 --- a/src/uk/me/parabola/splitter/Main.java +++ b/src/uk/me/parabola/splitter/Main.java @@ -13,6 +13,9 @@ package uk.me.parabola.splitter; +import java.io.File; +import java.io.FileInputStream; +import java.io.FileNotFoundException; import java.io.FileWriter; import java.io.IOException; import java.io.InputStreamReader; @@ -34,6 +37,8 @@ import uk.me.parabola.splitter.geo.DummyCityFinder; import org.xmlpull.v1.XmlPullParserException; +import crosby.binary.file.BlockInputStream; + /** * Splitter for OSM files with the purpose of providing input files for mkgmap. * <p/> @@ -264,14 +269,15 @@ public class Main { MapCollector nodes = densityMap ? new DensityMapCollector(trim, resolution) : new NodeCollector(); MapProcessor processor = nodes; - MapReader mapReader = processMap(processor); - System.out.print("A total of " + Utils.format(mapReader.getNodeCount()) + " nodes, " + - Utils.format(mapReader.getWayCount()) + " ways and " + - Utils.format(mapReader.getRelationCount()) + " relations were processed "); + //MapReader mapReader = + processMap(processor); + //System.out.print("A total of " + Utils.format(mapReader.getNodeCount()) + " nodes, " + + // Utils.format(mapReader.getWayCount()) + " ways and " + + // Utils.format(mapReader.getRelationCount()) + " relations were processed "); - System.out.println("in " + filenames.size() + (filenames.size() == 1 ? " file" : " files")); - System.out.println("Min node ID = " + mapReader.getMinNodeId()); - System.out.println("Max node ID = " + mapReader.getMaxNodeId()); + //System.out.println("in " + filenames.size() + (filenames.size() == 1 ? " file" : " files")); + //System.out.println("Min node ID = " + mapReader.getMinNodeId()); + //System.out.println("Max node ID = " + mapReader.getMaxNodeId()); System.out.println("Time: " + new Date()); @@ -342,10 +348,10 @@ public class Main { areas.get(i * areasPerPass + currentWriters.length - 1).getMapId() + ')'); MapProcessor processor = new SplitProcessor(currentWriters, maxThreads); - MapReader mapReader = processMap(processor); - System.out.println("Wrote " + Utils.format(mapReader.getNodeCount()) + " nodes, " + - Utils.format(mapReader.getWayCount()) + " ways, " + - Utils.format(mapReader.getRelationCount()) + " relations"); + processMap(processor); + //System.out.println("Wrote " + Utils.format(mapReader.getNodeCount()) + " nodes, " + + // Utils.format(mapReader.getWayCount()) + " ways, " + + // Utils.format(mapReader.getRelationCount()) + " relations"); } } @@ -366,12 +372,32 @@ public class Main { parse(parser, reader); } else { for (String filename : filenames) { - System.out.println("Processing " + filename + "..."); - Reader reader = Utils.openFile(filename, maxThreads > 1); - parse(parser, reader); + try { + if (filename.endsWith(".bin")) { + // Is it a binary file? + File file = new File(filename); + BlockInputStream blockinput = (new BlockInputStream( + new FileInputStream(file), new BinaryMapParser(processor))); + try { + blockinput.process(); + } finally { + blockinput.close(); + } + } else { + // No, try XML. + Reader reader = Utils.openFile(filename, maxThreads > 1); + parse(parser, reader); + } + } catch (FileNotFoundException e) { + e.printStackTrace(); + } catch (XmlPullParserException e) { + e.printStackTrace(); + } catch (IOException e) { + e.printStackTrace(); + } } } - parser.endMap(); + processor.endMap(); } private void parse(OSMParser parser, Reader reader) throws IOException, XmlPullParserException { diff --git a/src/uk/me/parabola/splitter/MapReader.java b/src/uk/me/parabola/splitter/MapReader.java index 8afa12b..d8a8a21 100644 --- a/src/uk/me/parabola/splitter/MapReader.java +++ b/src/uk/me/parabola/splitter/MapReader.java @@ -14,13 +14,13 @@ package uk.me.parabola.splitter; public interface MapReader { - long getNodeCount(); + //long getNodeCount(); - long getWayCount(); + //long getWayCount(); - long getRelationCount(); + //long getRelationCount(); - int getMinNodeId(); + //int getMinNodeId(); - int getMaxNodeId(); + //int getMaxNodeId(); } diff --git a/src/uk/me/parabola/splitter/OSMParser.java b/src/uk/me/parabola/splitter/OSMParser.java index 50569be..1388261 100644 --- a/src/uk/me/parabola/splitter/OSMParser.java +++ b/src/uk/me/parabola/splitter/OSMParser.java @@ -51,7 +51,7 @@ class OSMParser extends AbstractXppParser implements MapReader { this.startNodeOnly = processor.isStartNodeOnly(); this.mixed = mixed; } - + /* @Override public long getNodeCount() { return nodeCount; @@ -76,11 +76,11 @@ class OSMParser extends AbstractXppParser implements MapReader { public int getMaxNodeId() { return maxNodeId; } - + */ public void endMap() { processor.endMap(); } - + /** * Receive notification of the start of an element. */ -- 1.7.2.3
From: Scott Crosby <scrosby@cs.rice.edu> --- src/uk/me/parabola/splitter/OSMWriter.java | 4 ++ src/uk/me/parabola/splitter/SplitProcessor.java | 36 +++++++++++++++++++++-- 2 files changed, 37 insertions(+), 3 deletions(-) diff --git a/src/uk/me/parabola/splitter/OSMWriter.java b/src/uk/me/parabola/splitter/OSMWriter.java index a0d140c..701a639 100644 --- a/src/uk/me/parabola/splitter/OSMWriter.java +++ b/src/uk/me/parabola/splitter/OSMWriter.java @@ -37,6 +37,10 @@ public class OSMWriter { this.outputDir = outputDir; } + public Area getExtendedBounds() { + return extendedBounds; + } + public void initForWrite(int mapId, int extra) { extendedBounds = new Area(bounds.getMinLat() - extra, bounds.getMinLong() - extra, diff --git a/src/uk/me/parabola/splitter/SplitProcessor.java b/src/uk/me/parabola/splitter/SplitProcessor.java index 24d23db..716c50b 100644 --- a/src/uk/me/parabola/splitter/SplitProcessor.java +++ b/src/uk/me/parabola/splitter/SplitProcessor.java @@ -16,6 +16,9 @@ import java.io.IOException; import java.util.ArrayList; import java.util.BitSet; import java.util.Date; +import java.util.HashMap; +import java.util.Map.Entry; +import java.util.TreeMap; import java.util.concurrent.ArrayBlockingQueue; import java.util.concurrent.BlockingQueue; @@ -41,8 +44,31 @@ class SplitProcessor implements MapProcessor { private final int maxThreads; + TreeMap<Integer,ArrayList<OSMWriter>> writermap = new TreeMap<Integer,ArrayList<OSMWriter>>(); + HashMap<OSMWriter,Integer> writerToID = new HashMap<OSMWriter,Integer>(); + + void makeWriterMap() { + for (int i = 0 ; i < writers.length ; i++) { + writerToID.put(writers[i],i); + } + + for (OSMWriter w : writers) { + writermap.put(w.getExtendedBounds().getMinLat(),new ArrayList<OSMWriter>()); + writermap.put(w.getExtendedBounds().getMaxLat(),new ArrayList<OSMWriter>()); + } + for (OSMWriter w: writers) { + int minlat = w.getExtendedBounds().getMinLat(); + int maxlat = w.getExtendedBounds().getMaxLat(); + for (Integer i = minlat ; i != null && i <= maxlat; i = writermap.higherKey(i)) { + writermap.get(i).add(w); + } + } + + } + SplitProcessor(OSMWriter[] writers, int maxThreads) { this.writers = writers; + makeWriterMap(); this.maxThreads = maxThreads; this.writerInputQueue = new ArrayBlockingQueue<InputQueueInfo>(writers.length); this.writerInputQueues = new BlockingQueue[writers.length]; @@ -195,13 +221,17 @@ class SplitProcessor implements MapProcessor { } private void writeNode(Node currentNode) throws IOException { - for (int n = 0; n < writers.length; n++) { - boolean found = writers[n].nodeBelongsToThisArea(currentNode); + Entry<Integer, ArrayList<OSMWriter>> entry = writermap.floorEntry(currentNode.getMapLat()); + if (entry == null) + return; + for (OSMWriter w : entry.getValue()) { + int n = writerToID.get(w); + boolean found = w.nodeBelongsToThisArea(currentNode); if (found) { if (maxThreads > 1) { addToWorkingQueue(n, currentNode); } else { - writers[n].write(currentNode); + w.write(currentNode); } if (currentNodeAreaSet == 0) { currentNodeAreaSet = n + 1; -- 1.7.2.3
From: Scott Crosby <scrosby@cs.rice.edu> --- src/uk/me/parabola/splitter/SplitProcessor.java | 33 ++++++++++++++++------ 1 files changed, 24 insertions(+), 9 deletions(-) diff --git a/src/uk/me/parabola/splitter/SplitProcessor.java b/src/uk/me/parabola/splitter/SplitProcessor.java index 716c50b..97b014e 100644 --- a/src/uk/me/parabola/splitter/SplitProcessor.java +++ b/src/uk/me/parabola/splitter/SplitProcessor.java @@ -14,7 +14,9 @@ package uk.me.parabola.splitter; import java.io.IOException; import java.util.ArrayList; +import java.util.Arrays; import java.util.BitSet; +import java.util.Collections; import java.util.Date; import java.util.HashMap; import java.util.Map.Entry; @@ -39,7 +41,7 @@ class SplitProcessor implements MapProcessor { private final ArrayList<Thread> workerThreads; private int currentNodeAreaSet; - private BitSet currentWayAreaSet; + private ArrayList<Integer> currentWayAreaSet, tmpWayAreaSet; private BitSet currentRelAreaSet; private final int maxThreads; @@ -76,8 +78,8 @@ class SplitProcessor implements MapProcessor { writerInputQueues[i] = new ArrayBlockingQueue<Element>(NO_ELEMENTS); writerInputQueue.add(new InputQueueInfo(this.writers[i], writerInputQueues[i])); } - - currentWayAreaSet = new BitSet(writers.length); + tmpWayAreaSet = new ArrayList<Integer>(10); + currentWayAreaSet = new ArrayList<Integer>(10); currentRelAreaSet = new BitSet(writers.length); int noOfWorkerThreads = this.maxThreads - 1; @@ -154,11 +156,12 @@ class SplitProcessor implements MapProcessor { @Override public void processWay(Way w) { + int last_val = 0; for (int id: w.getRefs().asArray()) { // Get the list of areas that the node is in. A node may be in // more than one area because of overlap. int set = coords.get(id); - + // add the list of areas to the currentWayAreaSet if (set != 0) { int mask = 0xff; @@ -166,7 +169,9 @@ class SplitProcessor implements MapProcessor { int val = (set & mask) >>> (slot * 8); if (val == 0) break; - currentWayAreaSet.set(val - 1); + if (val != last_val) + currentWayAreaSet.add(val - 1); + last_val = val; } } } @@ -254,10 +259,20 @@ class SplitProcessor implements MapProcessor { System.out.println("Writing ways " + new Date()); } if (!currentWayAreaSet.isEmpty()) { - if (currentWayAreaSet.cardinality() <= 4) { + Collections.sort(currentWayAreaSet); + int last = -1; + tmpWayAreaSet.clear(); + for (Integer n : currentWayAreaSet) { + if (n.equals(last)) + continue; + tmpWayAreaSet.add(n); + last = n; + } + + if (tmpWayAreaSet.size() <= 4) { // this way falls into 4 or less areas (the normal case). Store these areas in the ways map int set = 0; - for (int n = currentWayAreaSet.nextSetBit(0); n >= 0; n = currentWayAreaSet.nextSetBit(n + 1)) { + for (Integer n : tmpWayAreaSet) { if (maxThreads > 1) { addToWorkingQueue(n, currentWay); } else { @@ -271,8 +286,8 @@ class SplitProcessor implements MapProcessor { } else { // this way falls into 5 or more areas. Convert the currentWayAreaSet into a long[] and store // these areas in the bigWays map - long[] set = new long[currentWayAreaSet.size() / 64]; - for (int n = currentWayAreaSet.nextSetBit(0); n >= 0; n = currentWayAreaSet.nextSetBit(n + 1)) { + long[] set = new long[(writers.length+63) / 64]; + for (Integer n : tmpWayAreaSet) { if (maxThreads > 1) { addToWorkingQueue(n, currentWay); } else { -- 1.7.2.3
From: Scott Crosby <scrosby@cs.rice.edu> --- src/uk/me/parabola/splitter/SplitProcessor.java | 28 ++++++++++++++++++----- 1 files changed, 22 insertions(+), 6 deletions(-) diff --git a/src/uk/me/parabola/splitter/SplitProcessor.java b/src/uk/me/parabola/splitter/SplitProcessor.java index 97b014e..4f7220e 100644 --- a/src/uk/me/parabola/splitter/SplitProcessor.java +++ b/src/uk/me/parabola/splitter/SplitProcessor.java @@ -46,10 +46,13 @@ class SplitProcessor implements MapProcessor { private final int maxThreads; - TreeMap<Integer,ArrayList<OSMWriter>> writermap = new TreeMap<Integer,ArrayList<OSMWriter>>(); + int lats[]; + ArrayList<OSMWriter> writersets[]; + HashMap<OSMWriter,Integer> writerToID = new HashMap<OSMWriter,Integer>(); void makeWriterMap() { + TreeMap<Integer,ArrayList<OSMWriter>> writermap = new TreeMap<Integer,ArrayList<OSMWriter>>(); for (int i = 0 ; i < writers.length ; i++) { writerToID.put(writers[i],i); } @@ -58,6 +61,10 @@ class SplitProcessor implements MapProcessor { writermap.put(w.getExtendedBounds().getMinLat(),new ArrayList<OSMWriter>()); writermap.put(w.getExtendedBounds().getMaxLat(),new ArrayList<OSMWriter>()); } + // Sentinel keys + writermap.put(Integer.MIN_VALUE,new ArrayList<OSMWriter>()); + writermap.put(Integer.MAX_VALUE,new ArrayList<OSMWriter>()); + for (OSMWriter w: writers) { int minlat = w.getExtendedBounds().getMinLat(); int maxlat = w.getExtendedBounds().getMaxLat(); @@ -65,7 +72,14 @@ class SplitProcessor implements MapProcessor { writermap.get(i).add(w); } } - + lats = new int[writermap.size()]; + writersets = new ArrayList[writermap.size()]; + int i = 0; + for (Entry<Integer, ArrayList<OSMWriter>> e : writermap.entrySet()) { + lats[i] = e.getKey(); + writersets[i] = e.getValue(); + i++; + } } SplitProcessor(OSMWriter[] writers, int maxThreads) { @@ -226,10 +240,12 @@ class SplitProcessor implements MapProcessor { } private void writeNode(Node currentNode) throws IOException { - Entry<Integer, ArrayList<OSMWriter>> entry = writermap.floorEntry(currentNode.getMapLat()); - if (entry == null) - return; - for (OSMWriter w : entry.getValue()) { + int index = Arrays.binarySearch(lats, currentNode.getMapLat()); + if (index < 0) + index = -index-1; + + //System.out.println("Send to "+entry.getValue().size()); + for (OSMWriter w : writersets[index]) { int n = writerToID.get(w); boolean found = w.nodeBelongsToThisArea(currentNode); if (found) { -- 1.7.2.3
From: Scott Crosby <scrosby@cs.rice.edu> --- src/uk/me/parabola/splitter/Element.java | 57 +++++++++++++--------------- src/uk/me/parabola/splitter/OSMWriter.java | 4 +- 2 files changed, 28 insertions(+), 33 deletions(-) diff --git a/src/uk/me/parabola/splitter/Element.java b/src/uk/me/parabola/splitter/Element.java index 94a3265..d3f4a57 100644 --- a/src/uk/me/parabola/splitter/Element.java +++ b/src/uk/me/parabola/splitter/Element.java @@ -12,18 +12,15 @@ */ package uk.me.parabola.splitter; -import java.util.HashMap; +import java.util.ArrayList; +import java.util.Collections; import java.util.Iterator; -import java.util.Map; -import java.util.NoSuchElementException; /** * @author Steve Ratcliffe */ public class Element { - private static final Iterator<Map.Entry<String, String>> EMPTY_ITERATOR = new EmptyIterator(); - - private Map<String, String> tags; + private ArrayList<Tag> tags; private int id; protected void setId(int id) { @@ -36,43 +33,41 @@ public class Element { public void reset() { this.id = 0; - tags = null; + tags.clear(); } + class Tag { + public Tag(String key,String value) { + this.key = key; + this.value = value; + } + public String getKey() { + return key; + } + public String getValue() { + return value; + } + final public String key,value; + } + public void addTag(String key, String value) { if (key.equals("created_by")) return; // Most elements are nodes. Most nodes have no tags. Create the tag table lazily - if (tags == null) { - tags = new HashMap<String, String>(4); - } - tags.put(key, value); + if (tags == null) + tags = new ArrayList<Tag>(4); + + tags.add(new Tag(key, value)); } public boolean hasTags() { return tags != null; } - public Iterator<Map.Entry<String, String>> tagsIterator() { - if (tags == null) { - return EMPTY_ITERATOR; - } - return tags.entrySet().iterator(); - } - - private static class EmptyIterator implements Iterator<Map.Entry<String, String>> - { - public boolean hasNext() - { - return false; - } + public Iterator<Tag> tagsIterator() { + if (tags == null) + return Collections.EMPTY_LIST.iterator(); - public Map.Entry<String, String> next() { - throw new NoSuchElementException(); - } - - public void remove() { - throw new UnsupportedOperationException(); - } + return tags.iterator(); } } diff --git a/src/uk/me/parabola/splitter/OSMWriter.java b/src/uk/me/parabola/splitter/OSMWriter.java index 701a639..8cd0740 100644 --- a/src/uk/me/parabola/splitter/OSMWriter.java +++ b/src/uk/me/parabola/splitter/OSMWriter.java @@ -145,9 +145,9 @@ public class OSMWriter { } private void writeTags(Element element) throws IOException { - Iterator<Map.Entry<String, String>> it = element.tagsIterator(); + Iterator<Element.Tag> it = element.tagsIterator(); while (it.hasNext()) { - Map.Entry<String, String> entry = it.next(); + Element.Tag entry = it.next(); writeString("<tag k='"); writeAttribute(entry.getKey()); writeString("' v='"); -- 1.7.2.3
From: Scott Crosby <scrosby@cs.rice.edu> --- src/uk/me/parabola/splitter/OSMWriter.java | 29 +++++++++++++++++++++++---- 1 files changed, 24 insertions(+), 5 deletions(-) diff --git a/src/uk/me/parabola/splitter/OSMWriter.java b/src/uk/me/parabola/splitter/OSMWriter.java index 8cd0740..03c7d52 100644 --- a/src/uk/me/parabola/splitter/OSMWriter.java +++ b/src/uk/me/parabola/splitter/OSMWriter.java @@ -18,6 +18,8 @@ import java.io.IOException; import java.io.OutputStream; import java.io.OutputStreamWriter; import java.io.Writer; +import java.text.DecimalFormat; +import java.text.DecimalFormatSymbols; import java.util.Formatter; import java.util.Iterator; import java.util.List; @@ -26,6 +28,11 @@ import java.util.Map; import java.util.zip.GZIPOutputStream; public class OSMWriter { + static final DecimalFormat numberFormat = new DecimalFormat( + "0.#######;-0.#######", + new DecimalFormatSymbols(Locale.US) + ); + private final Area bounds; private Writer writer; private Area extendedBounds; @@ -64,13 +71,13 @@ public class OSMWriter { writeString("<osm version='0.5' generator='splitter'>\n"); writeString("<bounds minlat='"); - writeDouble(Utils.toDegrees(bounds.getMinLat())); + writeLongDouble(Utils.toDegrees(bounds.getMinLat())); writeString("' minlon='"); - writeDouble(Utils.toDegrees(bounds.getMinLong())); + writeLongDouble(Utils.toDegrees(bounds.getMinLong())); writeString("' maxlat='"); - writeDouble(Utils.toDegrees(bounds.getMaxLat())); + writeLongDouble(Utils.toDegrees(bounds.getMaxLat())); writeString("' maxlon='"); - writeDouble(Utils.toDegrees(bounds.getMaxLong())); + writeLongDouble(Utils.toDegrees(bounds.getMaxLong())); writeString("'/>\n"); } @@ -203,9 +210,21 @@ public class OSMWriter { index += end - start; } + /** Write a double to full precision */ + private void writeLongDouble(double value) throws IOException { + checkFlush(22); + writeString(Double.toString(value)); + } + + /** Write a double truncated to OSM's 7 digits of precision + * + * TODO: Optimize. Responsible for >30% of the runtime after other using binary + * format and improved hash table. + */ private void writeDouble(double value) throws IOException { checkFlush(22); - writeString(Double.toString(value)); + writeString(numberFormat.format(value)); + return; } private void writeInt(int value) throws IOException { -- 1.7.2.3
From: Scott Crosby <scrosby@cs.rice.edu> Result is identical numbers but not binary-identical because of trailing 0 digits in the output. Speedup is cumulative with binary format and fixed hash table. --- src/uk/me/parabola/splitter/OSMWriter.java | 19 +++++++++++++++---- 1 files changed, 15 insertions(+), 4 deletions(-) diff --git a/src/uk/me/parabola/splitter/OSMWriter.java b/src/uk/me/parabola/splitter/OSMWriter.java index 03c7d52..c42ce52 100644 --- a/src/uk/me/parabola/splitter/OSMWriter.java +++ b/src/uk/me/parabola/splitter/OSMWriter.java @@ -215,7 +215,6 @@ public class OSMWriter { checkFlush(22); writeString(Double.toString(value)); } - /** Write a double truncated to OSM's 7 digits of precision * * TODO: Optimize. Responsible for >30% of the runtime after other using binary @@ -223,10 +222,22 @@ public class OSMWriter { */ private void writeDouble(double value) throws IOException { checkFlush(22); - writeString(numberFormat.format(value)); - return; + // Punt on some annoying specialcases + if (value < -200 || value > 200 || (value > -1 && value < 1)) + writeString(numberFormat.format(value)); + else { + if (value < 0) { + writeChar('-'); + value = -value; + } + + int val = (int)Math.round(value*10000000); + StringBuilder s = new StringBuilder(Integer.toString(val)); + s.insert(s.length()-7, '.'); + writeString(s.toString()); + } } - + private void writeInt(int value) throws IOException { checkFlush(11); index += Convert.intToString(value, charBuf, index); -- 1.7.2.3
From: Scott Crosby <scrosby@cs.rice.edu> --- src/uk/me/parabola/splitter/OSMWriter.java | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/src/uk/me/parabola/splitter/OSMWriter.java b/src/uk/me/parabola/splitter/OSMWriter.java index c42ce52..350661c 100644 --- a/src/uk/me/parabola/splitter/OSMWriter.java +++ b/src/uk/me/parabola/splitter/OSMWriter.java @@ -227,7 +227,7 @@ public class OSMWriter { writeString(numberFormat.format(value)); else { if (value < 0) { - writeChar('-'); + charBuf[index++] = '-'; // Write directly. value = -value; } -- 1.7.2.3
From: Scott Crosby <scrosby@cs.rice.edu> --- src/uk/me/parabola/splitter/SplitProcessor.java | 132 ++++++++++------------- 1 files changed, 56 insertions(+), 76 deletions(-) diff --git a/src/uk/me/parabola/splitter/SplitProcessor.java b/src/uk/me/parabola/splitter/SplitProcessor.java index 4f7220e..71957d9 100644 --- a/src/uk/me/parabola/splitter/SplitProcessor.java +++ b/src/uk/me/parabola/splitter/SplitProcessor.java @@ -18,9 +18,7 @@ import java.util.Arrays; import java.util.BitSet; import java.util.Collections; import java.util.Date; -import java.util.HashMap; -import java.util.Map.Entry; -import java.util.TreeMap; +import java.util.List; import java.util.concurrent.ArrayBlockingQueue; import java.util.concurrent.BlockingQueue; @@ -30,15 +28,18 @@ import uk.me.parabola.splitter.Relation.Member; * Splits a map into multiple areas. */ class SplitProcessor implements MapProcessor { - + public static final int NO_ELEMENTS = 10; + public static final int BUNDLE_SIZE = 2000; + + private final SplitIntMap coords = new SplitIntMap(); private final SplitIntMap ways = new SplitIntMap(); private final IntObjMap<long[]> bigWays = new IntObjMap<long[]>(); private final OSMWriter[] writers; - private final BlockingQueue<Element>[] writerInputQueues; - private final BlockingQueue<InputQueueInfo> writerInputQueue; - private final ArrayList<Thread> workerThreads; + private final BlockingQueue<List<Element>>[] writerInputQueues; + private final List<Element>[] bundlingQueues; + Thread threads[]; private int currentNodeAreaSet; private ArrayList<Integer> currentWayAreaSet, tmpWayAreaSet; @@ -86,24 +87,18 @@ class SplitProcessor implements MapProcessor { this.writers = writers; makeWriterMap(); this.maxThreads = maxThreads; - this.writerInputQueue = new ArrayBlockingQueue<InputQueueInfo>(writers.length); this.writerInputQueues = new BlockingQueue[writers.length]; + this.bundlingQueues = new ArrayList[writers.length]; + this.threads = new Thread[writers.length]; for (int i = 0; i < writerInputQueues.length;i++) { - writerInputQueues[i] = new ArrayBlockingQueue<Element>(NO_ELEMENTS); - writerInputQueue.add(new InputQueueInfo(this.writers[i], writerInputQueues[i])); + writerInputQueues[i] = new ArrayBlockingQueue<List<Element>>(NO_ELEMENTS); + bundlingQueues[i] = new ArrayList<Element>(BUNDLE_SIZE); + threads[i] = new Thread(new OSMWriterWorker(writers[i],writerInputQueues[i])); + threads[i].start(); } tmpWayAreaSet = new ArrayList<Integer>(10); currentWayAreaSet = new ArrayList<Integer>(10); currentRelAreaSet = new BitSet(writers.length); - - int noOfWorkerThreads = this.maxThreads - 1; - workerThreads = new ArrayList<Thread>(noOfWorkerThreads); - for (int i = 0; i < noOfWorkerThreads; i++) { - Thread worker = new Thread(new OSMWriterWorker()); - worker.setName("worker-" + i); - workerThreads.add(worker); - worker.start(); - } } @Override @@ -220,20 +215,18 @@ class SplitProcessor implements MapProcessor { @Override public void endMap() { - for (int i = 0; i < writerInputQueues.length; i++) { - try { - writerInputQueues[i].put(STOP_ELEMENT); - } catch (InterruptedException e) { - throw new RuntimeException("Failed to add the stop element for worker thread " + i, e); - } - } - for (Thread workerThread : workerThreads) { - try { - workerThread.join(); + try { + // Push the stop element into every queue. + for (int i = 0 ; i < threads.length ; i++ ) + addToWorkingQueue(i,STOP_ELEMENT); + // Wait for them to all exit. + for (int i = 0 ; i < threads.length ; i++ ) + threads[i].join(); } catch (InterruptedException e) { - throw new RuntimeException("Failed to join for thread " + workerThread.getName(), e); + // TODO Auto-generated catch block + e.printStackTrace(); } - } + for (OSMWriter writer : writers) { writer.finishWrite(); } @@ -350,29 +343,34 @@ class SplitProcessor implements MapProcessor { } private void addToWorkingQueue(int writerNumber, Element element) { + List<Element> bundle=bundlingQueues[writerNumber]; + bundle.add(element); + if (bundle.size() < BUNDLE_SIZE && element != STOP_ELEMENT) + return; try { - writerInputQueues[writerNumber].put(element); + BlockingQueue<List<Element>> queue = writerInputQueues[writerNumber]; + queue.put(bundle); + bundlingQueues[writerNumber] = new ArrayList<Element>(BUNDLE_SIZE); } catch (InterruptedException e) { throw new RuntimeException("Failed to write node " + element.getId() + " to worker thread " + writerNumber, e); } } - private static class InputQueueInfo { - private final OSMWriter writer; - private final BlockingQueue<Element> inputQueue; - - public InputQueueInfo(OSMWriter writer, BlockingQueue<Element> inputQueue) { - this.writer = writer; - this.inputQueue = inputQueue; - } - } private static final Element STOP_ELEMENT = new Element(); - public static final int NO_ELEMENTS = 1000; private class OSMWriterWorker implements Runnable { + + private OSMWriter writer; + private BlockingQueue<List<Element>> queue; + + public OSMWriterWorker(OSMWriter writer, BlockingQueue<List<Element>> queue) { + this.writer = writer; + this.queue = queue; + } + public void processElement(Element element, OSMWriter writer) throws IOException { if (element instanceof Node) { writer.write((Node) element); @@ -385,41 +383,23 @@ class SplitProcessor implements MapProcessor { @Override public void run() { - boolean finished = false; - while (!finished) { - InputQueueInfo workPackage = writerInputQueue.poll(); - if (workPackage==null) { - finished=true; - } else { - while (!workPackage.inputQueue.isEmpty()) { - Element element =null; - try { - element = workPackage.inputQueue.poll(); - if (element == null) { - writerInputQueue.put(workPackage); - workPackage=null; - break; - } else if (element == STOP_ELEMENT) { - workPackage=null; - break; - } else { - processElement(element, workPackage.writer); - } - - } catch (InterruptedException e) { - throw new RuntimeException("Thread " + Thread.currentThread().getName() + " failed to get next element", e); - } catch (IOException e) { - throw new RuntimeException("Thread " + Thread.currentThread().getName() + " failed to write element " + element.getId() + '(' + element.getClass().getSimpleName() + ')', e); - } - } - if (workPackage != null) { - try { - writerInputQueue.put(workPackage); - } catch (InterruptedException e) { - throw new RuntimeException("Thread " + Thread.currentThread().getName() + " failed to return work package", e); - } - } + while (true) { + //System.out.println("Doing loop"); + try { + + List<Element> elements = queue.take(); + for (Element element : elements) + if (element == STOP_ELEMENT) + return; + else + processElement(element, writer); + + } catch (InterruptedException e) { + throw new RuntimeException("Thread " + Thread.currentThread().getName() + " failed to get next element", e); + } catch (IOException e) { + throw new RuntimeException("Thread " + Thread.currentThread().getName() + " failed to write element ",e); } + Thread.yield(); } System.out.println("Thread " + Thread.currentThread().getName() + " has finished"); } -- 1.7.2.3
Hello Jeffrey, Thanks very much for this. It's likely to be a while before I get a chance to look at and apply these sorry, but hopefully will find some time next weekend. Perhaps in the meantime if you have a current build of the splitter that includes these patches you could make it available for others to test and provide some initial feedback on? The one thing I'm not sure about is the last one with the new thread design - I know it provides a decent performance boost, but I'm a little uneasy with the way it achieves this by using more threads than CPU cores. AFAIK that patch is incidental to the rest of the binary file support though so shouldn't affect this transition. Cheers, Chris
Here's a quick rebasing of the patches in Scott's splitter repository to the current splitter trunk. My Java dev system is down for some upgrades at the moment so I haven't even compiled these yet but hopefully I didn't do too bad of a job of resolving the conflicts.
My internal repository has several independent development threads. Only one of them is the binary format. The other development threads included patches that are already in and some have not been benchmarked/tested thoroughly. * The binary format. * Improved double writing for XML output. * Alternate threading design. * Avoid sending each node to each osmwriter to do a bbox check. (Depends on some of the binary format refactoring.) * too-many-areas. (already in) * Tag representation (partially in; not benchmarked) Git knows which patch is in which thread. The two critical ones for the binary format are the binary format patches and the ULP patch in the double-writing thread. The patches where I avoid sending each node to each OSM writer improved the Big-O from O(n) to O(sqrt(n))), but the benchmarks seem to be the identical performance. I think things are bottlenecked updating the large shared arrays, which may also be why there's no need to worry about the threading design making one thread per writer; they're all idle waiting for the bottleneck to dribble out stuff to write. Scott On Sun, Sep 12, 2010 at 8:25 AM, Chris Miller <chris_overseas@hotmail.com> wrote:
Hello Jeffrey,
Thanks very much for this. It's likely to be a while before I get a chance to look at and apply these sorry, but hopefully will find some time next weekend. Perhaps in the meantime if you have a current build of the splitter that includes these patches you could make it available for others to test and provide some initial feedback on?
The one thing I'm not sure about is the last one with the new thread design - I know it provides a decent performance boost, but I'm a little uneasy with the way it achieves this by using more threads than CPU cores. AFAIK that patch is incidental to the rest of the binary file support though so shouldn't affect this transition.
Cheers, Chris
Here's a quick rebasing of the patches in Scott's splitter repository to the current splitter trunk. My Java dev system is down for some upgrades at the moment so I haven't even compiled these yet but hopefully I didn't do too bad of a job of resolving the conflicts.
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Scott I am adding support for your binary format to mkgmap itself (not the splitter). Now I've done enough refactoring of the XML reader to allow me to start I have a few questions. 1. Is there an final name for the format and the jar file? 2. Is there an 'offical' download location for a pre-built jar file? 3. How do I recognise a file in your format. Is there a conventional file extension for it? Are the OSMHeader and OSMData blocks required file block types? 4. Does the osmosis conversion from XML to binary keep the order of the elements the same as they were in the XML file? ..Steve
On Mon, Sep 13, 2010 at 6:06 AM, Steve Ratcliffe <steve@parabola.me.uk> wrote:
Hi Scott
I am adding support for your binary format to mkgmap itself (not the splitter).
Excellent! Thank you.
Now I've done enough refactoring of the XML reader to allow me to start I have a few questions.
1. Is there an final name for the format and the jar file?
Not yet. There are too many 'osm binary formats'. I asked for suggestions on osm-dev yesterday and got 'protobuf binary format'. Do you have any ideas?
2. Is there an 'offical' download location for a pre-built jar file?
No.
3. How do I recognise a file in your format. Is there a conventional file extension for it?
Those are good questions; I wish someone had asked them earlier. I have not put in error checking to detect illegal/wrong inputs. I have fixes in my local tree that throw exceptions on very ill-formed inputs. They will be out in my next RC. I have not thought about how one might detect the format. How important is this? For now, try feeding the data to the parser and see if there is an exception? I can add a more robust check with magic at the start of the file, but I won't have time to implement it for a while. I designed a concatenable and streamable format. Magic at the start of a file needs to also be a legal fileblock. I can specify and define such a magic, as the static serialized contents of a '__Magic' fileblock but implementing this may take a little while. I have used *.bin as an extension, but I am open to suggestions.
Are the OSMHeader and OSMData blocks required file block types?
I am not sure what you are asking, but yes, both are required. OSMHeader contains HeaderBlock::required_features, which must be examined to confirm that your implementation can parse the file. You may also want the contents of HeaderBlock::bbox.
4. Does the osmosis conversion from XML to binary keep the order of the elements the same as they were in the XML file?
Yes. Absolutely. Scott
I have used *.bin as an extension, but I am open to suggestions.
I find *.bin a little general. Most of files are bin. I would suggest instead *.osm.bin A abbreviation of the phrase 'protobuf binary format' would give *.pbf. Both from 'ProtoBuF' and from 'Protobuf Bin Format'. The more I think about it, the more *.osm.pbf sounds good to me.
On 13/09/10 17:55, Scott Crosby wrote:
Not yet. There are too many 'osm binary formats'. I asked for suggestions on osm-dev yesterday and got 'protobuf binary format'. Do you have any ideas?
Not really, I'm calling it osmprotobuf at the moment...
I have not thought about how one might detect the format. How important is this?
Its needed because mkgmap takes all kinds of input files and I don't want the user to have to say what the file format is, I just want mkgmap to work out the file format and use the correct reader. I do this by extension or by reading the beginning of the file and looking for something distinctive. It doesn't have to be fool proof just good enough to tell genuine files apart. Hence why I was asking if OSMHeader and or OSMData would always be present near the beginning of the file, I could look for them.
4. Does the osmosis conversion from XML to binary keep the order of the elements the same as they were in the XML file?
Yes. Absolutely.
Great, should be easy to test it is working by comparing the resulting files. Thanks ..Steve
On Mon, Sep 13, 2010 at 3:36 PM, Steve Ratcliffe <steve@parabola.me.uk> wrote:
Its needed because mkgmap takes all kinds of input files and I don't want the user to have to say what the file format is, I just want mkgmap to work out the file format and use the correct reader.
I do this by extension or by reading the beginning of the file and looking for something distinctive. It doesn't have to be fool proof just good enough to tell genuine files apart.
Hence why I was asking if OSMHeader and or OSMData would always be present near the beginning of the file, I could look for them.
Yes, I believe OSMHeader should occur in the first 16 bytes or so. (Just looked at a hexdump. Yes, confirmed, byte offset [6,14]. Furthermore, I believe that byte offsets [4,15] will always be static when the first block is an OSMHeader, however, I cannot guarantee it because I cannot guarantee that a given protocol buffer will always serialize to the same output in the future. Scott
On Sep 9, 2010, at 19:55, Scott Crosby wrote:
The format is stable, but I want to release one more RC, with a full validation before I declare it stable. I expect no incompatible changes.
Is this the binary format in discussion: http://wiki.openstreetmap.org/wiki/OSMbin(file_format) I had trouble finding the right description. There seem to be a few other "binary" format projects, apparently primarily intended for mobile applications. (And the link in Scot's original e-mail to this list is sadly broken.) Cheers.
On Sun, Sep 12, 2010 at 6:39 AM, Clinton Gladstone <clinton.gladstone@googlemail.com> wrote:
On Sep 9, 2010, at 19:55, Scott Crosby wrote:
The format is stable, but I want to release one more RC, with a full validation before I declare it stable. I expect no incompatible changes.
Is this the binary format in discussion:
The original email is: http://www.mail-archive.com/dev@openstreetmap.org/msg11392.html I also just posted up some text on the Wiki with a complete description. http://wiki.openstreetmap.org/wiki/APIbin Scott
participants (7)
-
Chris Miller -
Clinton Gladstone -
Jeffrey C. Ollie -
Jeffrey Ollie
-
Johann Gail -
Scott Crosby -
Steve Ratcliffe