[PATCH V4] boundary preparer with quadtree
Hi WanMil, attached is the new patch for the performance branch. Its quite big because I've rewritten many methods to work with the quadtree instead of List<boundary> Major changes: - new file format for *.bnd files. It stores first the boundary tags, then the areas with float precision. The file size is not much different from the old format, but when zipped it is much smaller. The time to load the file into the quadtree is shorter than the time to load the old format. The old format is still supported, but requires much more time in the LocationHook (probably it is still faster than trunk, but not much) - the bounds parameter allows now to specify a zip file with the *.bnd files. The zip file can have a directory structure, but should not contain duplicate *.bnd files. - BoundaryPreparer uses multiple processors for the workoutBoundaryRelations() part when --max-jobs parameter allows it. When called as stand-alone program, it starts one thread for each processor. - the utilities BoundaryDiff, BoundaryMerger, BoundaryFile2Gpx, BoundaryLister were rewritten to use the quadtree, most of them also allow *.zip as input. BoundaryLister lists only the OSM tags, not the information created for the quadtree. Can be changed if needed. Open problem: The BoundaryMerger creates a result that contains a few more small holes than the trunk version. I'm going to analyse that during the next days. Todo: Javadoc Ciao, Gerd http://gis.19327.n5.nabble.com/file/n5495676/boundary_prep_quadtree_v4.patch boundary_prep_quadtree_v4.patch -- View this message in context: http://gis.19327.n5.nabble.com/PATCH-V4-boundary-preparer-with-quadtree-tp54... Sent from the Mkgmap Development mailing list archive at Nabble.com.
Hi Gerd, thanks for that patch. It's a big step forward :-) I've comitted it and also merged all changes from the trunk so that the branch does not diverge too much. I tried to compile boundaries for an old planet file but got an exception in the BoundaryMerger: Exception in thread "main" java.lang.NullPointerException at uk.me.parabola.mkgmap.reader.osm.boundary.BoundaryMerger.merge(BoundaryMerger.java:93) at uk.me.parabola.mkgmap.reader.osm.boundary.BoundaryMerger.main(BoundaryMerger.java:144) Can you have a look on it? Thanks! WanMil
Hi WanMil,
attached is the new patch for the performance branch. Its quite big because I've rewritten many methods to work with the quadtree instead of List<boundary>
Major changes: - new file format for *.bnd files. It stores first the boundary tags, then the areas with float precision. The file size is not much different from the old format, but when zipped it is much smaller. The time to load the file into the quadtree is shorter than the time to load the old format. The old format is still supported, but requires much more time in the LocationHook (probably it is still faster than trunk, but not much)
- the bounds parameter allows now to specify a zip file with the *.bnd files. The zip file can have a directory structure, but should not contain duplicate *.bnd files. - BoundaryPreparer uses multiple processors for the workoutBoundaryRelations() part when --max-jobs parameter allows it. When called as stand-alone program, it starts one thread for each processor.
- the utilities BoundaryDiff, BoundaryMerger, BoundaryFile2Gpx, BoundaryLister were rewritten to use the quadtree, most of them also allow *.zip as input. BoundaryLister lists only the OSM tags, not the information created for the quadtree. Can be changed if needed.
Open problem: The BoundaryMerger creates a result that contains a few more small holes than the trunk version. I'm going to analyse that during the next days. Todo: Javadoc
Ciao, Gerd
http://gis.19327.n5.nabble.com/file/n5495676/boundary_prep_quadtree_v4.patch boundary_prep_quadtree_v4.patch
-- View this message in context: http://gis.19327.n5.nabble.com/PATCH-V4-boundary-preparer-with-quadtree-tp54... Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi WanMil, thanks. Reg. the error: I have to add some tests to make sure that the BoundaryQuadTree object was created. Did the program already process some bnd files or did that happen right after the program start? Gerd WanMil wrote
Hi Gerd,
thanks for that patch. It's a big step forward :-) I've comitted it and also merged all changes from the trunk so that the branch does not diverge too much.
I tried to compile boundaries for an old planet file but got an exception in the BoundaryMerger: Exception in thread "main" java.lang.NullPointerException at uk.me.parabola.mkgmap.reader.osm.boundary.BoundaryMerger.merge(BoundaryMerger.java:93) at uk.me.parabola.mkgmap.reader.osm.boundary.BoundaryMerger.main(BoundaryMerger.java:144)
Can you have a look on it? Thanks!
WanMil
Hi WanMil,
attached is the new patch for the performance branch. Its quite big because I've rewritten many methods to work with the quadtree instead of List<boundary>
Major changes: - new file format for *.bnd files. It stores first the boundary tags, then the areas with float precision. The file size is not much different from the old format, but when zipped it is much smaller. The time to load the file into the quadtree is shorter than the time to load the old format. The old format is still supported, but requires much more time in the LocationHook (probably it is still faster than trunk, but not much)
- the bounds parameter allows now to specify a zip file with the *.bnd files. The zip file can have a directory structure, but should not contain duplicate *.bnd files. - BoundaryPreparer uses multiple processors for the workoutBoundaryRelations() part when --max-jobs parameter allows it. When called as stand-alone program, it starts one thread for each processor.
- the utilities BoundaryDiff, BoundaryMerger, BoundaryFile2Gpx, BoundaryLister were rewritten to use the quadtree, most of them also allow *.zip as input. BoundaryLister lists only the OSM tags, not the information created for the quadtree. Can be changed if needed.
Open problem: The BoundaryMerger creates a result that contains a few more small holes than the trunk version. I'm going to analyse that during the next days. Todo: Javadoc
Ciao, Gerd
http://gis.19327.n5.nabble.com/file/n5495676/boundary_prep_quadtree_v4.patch boundary_prep_quadtree_v4.patch
-- View this message in context: http://gis.19327.n5.nabble.com/PATCH-V4-boundary-preparer-with-quadtree-tp54... Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@.org http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@.org http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- View this message in context: http://gis.19327.n5.nabble.com/PATCH-V4-boundary-preparer-with-quadtree-tp54... Sent from the Mkgmap Development mailing list archive at Nabble.com.
Hi Gerd, merging of africa, australia-oceania did work (but I think they have no bnd file in common). Merging of asia to them failed at the first file. WanMil
Hi WanMil,
thanks. Reg. the error: I have to add some tests to make sure that the BoundaryQuadTree object was created. Did the program already process some bnd files or did that happen right after the program start?
Gerd
WanMil wrote
Hi Gerd,
thanks for that patch. It's a big step forward :-) I've comitted it and also merged all changes from the trunk so that the branch does not diverge too much.
I tried to compile boundaries for an old planet file but got an exception in the BoundaryMerger: Exception in thread "main" java.lang.NullPointerException at uk.me.parabola.mkgmap.reader.osm.boundary.BoundaryMerger.merge(BoundaryMerger.java:93) at uk.me.parabola.mkgmap.reader.osm.boundary.BoundaryMerger.main(BoundaryMerger.java:144)
Can you have a look on it? Thanks!
WanMil
Hi WanMil,
attached is the new patch for the performance branch. Its quite big because I've rewritten many methods to work with the quadtree instead of List<boundary>
Major changes: - new file format for *.bnd files. It stores first the boundary tags, then the areas with float precision. The file size is not much different from the old format, but when zipped it is much smaller. The time to load the file into the quadtree is shorter than the time to load the old format. The old format is still supported, but requires much more time in the LocationHook (probably it is still faster than trunk, but not much)
- the bounds parameter allows now to specify a zip file with the *.bnd files. The zip file can have a directory structure, but should not contain duplicate *.bnd files. - BoundaryPreparer uses multiple processors for the workoutBoundaryRelations() part when --max-jobs parameter allows it. When called as stand-alone program, it starts one thread for each processor.
- the utilities BoundaryDiff, BoundaryMerger, BoundaryFile2Gpx, BoundaryLister were rewritten to use the quadtree, most of them also allow *.zip as input. BoundaryLister lists only the OSM tags, not the information created for the quadtree. Can be changed if needed.
Open problem: The BoundaryMerger creates a result that contains a few more small holes than the trunk version. I'm going to analyse that during the next days. Todo: Javadoc
Ciao, Gerd
http://gis.19327.n5.nabble.com/file/n5495676/boundary_prep_quadtree_v4.patch boundary_prep_quadtree_v4.patch
-- View this message in context: http://gis.19327.n5.nabble.com/PATCH-V4-boundary-preparer-with-quadtree-tp54... Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@.org http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@.org http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- View this message in context: http://gis.19327.n5.nabble.com/PATCH-V4-boundary-preparer-with-quadtree-tp54... Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi WanMil, do you see messages in the log, e.g. "Cannot load boundary file" ? If yes, please provide the content of the two *.bnd files. I think in such a case the merger should stop with some speaking error message. Do you agree? Gerd
Date: Sun, 19 Feb 2012 13:20:03 +0100 From: wmgcnfg@web.de To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] [PATCH V4] boundary preparer with quadtree
Hi Gerd,
merging of africa, australia-oceania did work (but I think they have no bnd file in common). Merging of asia to them failed at the first file.
WanMil
Hi WanMil,
thanks. Reg. the error: I have to add some tests to make sure that the BoundaryQuadTree object was created. Did the program already process some bnd files or did that happen right after the program start?
Gerd
WanMil wrote
Hi Gerd,
thanks for that patch. It's a big step forward :-) I've comitted it and also merged all changes from the trunk so that the branch does not diverge too much.
I tried to compile boundaries for an old planet file but got an exception in the BoundaryMerger: Exception in thread "main" java.lang.NullPointerException at uk.me.parabola.mkgmap.reader.osm.boundary.BoundaryMerger.merge(BoundaryMerger.java:93) at uk.me.parabola.mkgmap.reader.osm.boundary.BoundaryMerger.main(BoundaryMerger.java:144)
Can you have a look on it? Thanks!
WanMil
Hi WanMil,
attached is the new patch for the performance branch. Its quite big because I've rewritten many methods to work with the quadtree instead of List<boundary>
Major changes: - new file format for *.bnd files. It stores first the boundary tags, then the areas with float precision. The file size is not much different from the old format, but when zipped it is much smaller. The time to load the file into the quadtree is shorter than the time to load the old format. The old format is still supported, but requires much more time in the LocationHook (probably it is still faster than trunk, but not much)
- the bounds parameter allows now to specify a zip file with the *.bnd files. The zip file can have a directory structure, but should not contain duplicate *.bnd files. - BoundaryPreparer uses multiple processors for the workoutBoundaryRelations() part when --max-jobs parameter allows it. When called as stand-alone program, it starts one thread for each processor.
- the utilities BoundaryDiff, BoundaryMerger, BoundaryFile2Gpx, BoundaryLister were rewritten to use the quadtree, most of them also allow *.zip as input. BoundaryLister lists only the OSM tags, not the information created for the quadtree. Can be changed if needed.
Open problem: The BoundaryMerger creates a result that contains a few more small holes than the trunk version. I'm going to analyse that during the next days. Todo: Javadoc
Ciao, Gerd
http://gis.19327.n5.nabble.com/file/n5495676/boundary_prep_quadtree_v4.patch boundary_prep_quadtree_v4.patch
-- View this message in context: http://gis.19327.n5.nabble.com/PATCH-V4-boundary-preparer-with-quadtree-tp54... Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@.org http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@.org http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- View this message in context: http://gis.19327.n5.nabble.com/PATCH-V4-boundary-preparer-with-quadtree-tp54... Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Gerd, the only message I have in the log file is: 2012/02/19 14:08:34 Schwerwiegend (BoundaryUtil): Cannot read asia 2012/02/19 14:08:35 Schwerwiegend (BoundaryUtil): Cannot read tomerge WanMil
Hi WanMil,
do you see messages in the log, e.g. "Cannot load boundary file" ? If yes, please provide the content of the two *.bnd files. I think in such a case the merger should stop with some speaking error message. Do you agree?
Gerd
Date: Sun, 19 Feb 2012 13:20:03 +0100 From: wmgcnfg@web.de To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] [PATCH V4] boundary preparer with quadtree
Hi Gerd,
merging of africa, australia-oceania did work (but I think they have no bnd file in common). Merging of asia to them failed at the first file.
WanMil
Hi WanMil,
thanks. Reg. the error: I have to add some tests to make sure that the BoundaryQuadTree object was created. Did the program already process some bnd files or did that happen right after the program start?
Gerd
WanMil wrote
Hi Gerd,
thanks for that patch. It's a big step forward :-) I've comitted it and also merged all changes from the trunk so
that the
branch does not diverge too much.
I tried to compile boundaries for an old planet file but got an exception in the BoundaryMerger: Exception in thread "main" java.lang.NullPointerException at
uk.me.parabola.mkgmap.reader.osm.boundary.BoundaryMerger.merge(BoundaryMerger.java:93)
at
uk.me.parabola.mkgmap.reader.osm.boundary.BoundaryMerger.main(BoundaryMerger.java:144)
Can you have a look on it? Thanks!
WanMil
Hi WanMil,
attached is the new patch for the performance branch. Its quite big because I've rewritten many methods to work with the quadtree instead of List<boundary>
Major changes: - new file format for *.bnd files. It stores first the boundary tags, then the areas with float precision. The file size is not much different from the old format, but when
zipped
it is much smaller. The time to load the file into the quadtree is shorter than the time to load the old format. The old format is still supported, but requires much more time in the LocationHook (probably it is still faster than trunk, but not much)
- the bounds parameter allows now to specify a zip file with the *.bnd files. The zip file can have a directory structure, but should not contain duplicate *.bnd files. - BoundaryPreparer uses multiple processors for the workoutBoundaryRelations() part when --max-jobs parameter allows it. When called as stand-alone program, it starts one thread for each processor.
- the utilities BoundaryDiff, BoundaryMerger, BoundaryFile2Gpx, BoundaryLister were rewritten to use the quadtree, most of them also allow *.zip as input. BoundaryLister lists only the OSM tags, not the information created for the quadtree. Can be changed if needed.
Open problem: The BoundaryMerger creates a result that contains a few more small holes than the trunk version. I'm going to analyse that during the next days. Todo: Javadoc
Ciao, Gerd
http://gis.19327.n5.nabble.com/file/n5495676/boundary_prep_quadtree_v4.patch
boundary_prep_quadtree_v4.patch
-- View this message in context:
http://gis.19327.n5.nabble.com/PATCH-V4-boundary-preparer-with-quadtree-tp54...
Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@.org http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@.org http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- View this message in context: http://gis.19327.n5.nabble.com/PATCH-V4-boundary-preparer-with-quadtree-tp54... Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi WanMil, I think the problem was caused by mixed usage of dir1.getName() and dir1.getAbsolutePath(). Please try the attached patch. I think the BoundaryMerger doesn't work yet, when you specify the same directory for dir1 and dir2, it sometimes creates files that are much larger than the input. I'm going to look at this tomorrow. Gerd http://gis.19327.n5.nabble.com/file/n5496926/boundary_prep_quadtree_v5.patch boundary_prep_quadtree_v5.patch WanMil wrote
Hi Gerd,
the only message I have in the log file is: 2012/02/19 14:08:34 Schwerwiegend (BoundaryUtil): Cannot read asia 2012/02/19 14:08:35 Schwerwiegend (BoundaryUtil): Cannot read tomerge
WanMil
Hi WanMil,
do you see messages in the log, e.g. "Cannot load boundary file" ? If yes, please provide the content of the two *.bnd files. I think in such a case the merger should stop with some speaking error message. Do you agree?
Gerd
Date: Sun, 19 Feb 2012 13:20:03 +0100 From: wmgcnfg@ To: mkgmap-dev@.org Subject: Re: [mkgmap-dev] [PATCH V4] boundary preparer with quadtree
Hi Gerd,
merging of africa, australia-oceania did work (but I think they have no bnd file in common). Merging of asia to them failed at the first file.
WanMil
Hi WanMil,
thanks. Reg. the error: I have to add some tests to make sure that the BoundaryQuadTree object was created. Did the program already process some bnd files or did that happen right after the program start?
Gerd
WanMil wrote
Hi Gerd,
thanks for that patch. It's a big step forward :-) I've comitted it and also merged all changes from the trunk so
that the
branch does not diverge too much.
I tried to compile boundaries for an old planet file but got an exception in the BoundaryMerger: Exception in thread "main" java.lang.NullPointerException at
uk.me.parabola.mkgmap.reader.osm.boundary.BoundaryMerger.merge(BoundaryMerger.java:93)
at
uk.me.parabola.mkgmap.reader.osm.boundary.BoundaryMerger.main(BoundaryMerger.java:144)
Can you have a look on it? Thanks!
WanMil
Hi WanMil,
attached is the new patch for the performance branch. Its quite big because I've rewritten many methods to work with the quadtree instead of List<boundary>
Major changes: - new file format for *.bnd files. It stores first the boundary
tags,
then the areas with float precision. The file size is not much different from the old format, but when zipped it is much smaller. The time to load the file into the quadtree is shorter than the time to load the old format. The old format is still supported, but requires much more time in the LocationHook (probably it is still faster than trunk, but not much)
- the bounds parameter allows now to specify a zip file with the *.bnd files. The zip file can have a directory structure, but should not contain duplicate *.bnd files. - BoundaryPreparer uses multiple processors for the workoutBoundaryRelations() part when --max-jobs parameter allows it. When called as stand-alone program, it starts one thread for each processor.
- the utilities BoundaryDiff, BoundaryMerger, BoundaryFile2Gpx, BoundaryLister were rewritten to use the quadtree, most of them also allow *.zip as input. BoundaryLister lists only the OSM tags, not the information created for the quadtree. Can be changed if needed.
Open problem: The BoundaryMerger creates a result that contains a few more small holes than the trunk version. I'm going to analyse that during the next days. Todo: Javadoc
Ciao, Gerd
http://gis.19327.n5.nabble.com/file/n5495676/boundary_prep_quadtree_v4.patch
boundary_prep_quadtree_v4.patch
-- View this message in context:
http://gis.19327.n5.nabble.com/PATCH-V4-boundary-preparer-with-quadtree-tp54...
Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@.org http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@.org http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- View this message in context: http://gis.19327.n5.nabble.com/PATCH-V4-boundary-preparer-with-quadtree-tp54... Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@.org http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@.org http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@.org http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@.org http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- View this message in context: http://gis.19327.n5.nabble.com/PATCH-V4-boundary-preparer-with-quadtree-tp54... Sent from the Mkgmap Development mailing list archive at Nabble.com.
Hi Gerd, thanks. It's working now. I think it does not make any sense to start the BoundaryMerger with identical dir1 and dir2. There would be nothing to merge. So this can be checked and either an error message is issued and the program exits or dir1 is just copied to the merge directory. WanMil
Hi WanMil,
I think the problem was caused by mixed usage of dir1.getName() and dir1.getAbsolutePath().
Please try the attached patch.
I think the BoundaryMerger doesn't work yet, when you specify the same directory for dir1 and dir2, it sometimes creates files that are much larger than the input. I'm going to look at this tomorrow.
Gerd
http://gis.19327.n5.nabble.com/file/n5496926/boundary_prep_quadtree_v5.patch boundary_prep_quadtree_v5.patch
WanMil wrote
Hi Gerd,
the only message I have in the log file is: 2012/02/19 14:08:34 Schwerwiegend (BoundaryUtil): Cannot read asia 2012/02/19 14:08:35 Schwerwiegend (BoundaryUtil): Cannot read tomerge
WanMil
Hi WanMil,
do you see messages in the log, e.g. "Cannot load boundary file" ? If yes, please provide the content of the two *.bnd files. I think in such a case the merger should stop with some speaking error message. Do you agree?
Gerd
Date: Sun, 19 Feb 2012 13:20:03 +0100 From: wmgcnfg@ To: mkgmap-dev@.org Subject: Re: [mkgmap-dev] [PATCH V4] boundary preparer with quadtree
Hi Gerd,
merging of africa, australia-oceania did work (but I think they have no bnd file in common). Merging of asia to them failed at the first file.
WanMil
Hi WanMil,
thanks. Reg. the error: I have to add some tests to make sure that the BoundaryQuadTree object was created. Did the program already process some bnd files or did that happen right after the program start?
Gerd
WanMil wrote
Hi Gerd,
thanks for that patch. It's a big step forward :-) I've comitted it and also merged all changes from the trunk so
that the
branch does not diverge too much.
I tried to compile boundaries for an old planet file but got an exception in the BoundaryMerger: Exception in thread "main" java.lang.NullPointerException at
uk.me.parabola.mkgmap.reader.osm.boundary.BoundaryMerger.merge(BoundaryMerger.java:93)
at
uk.me.parabola.mkgmap.reader.osm.boundary.BoundaryMerger.main(BoundaryMerger.java:144)
Can you have a look on it? Thanks!
WanMil
> Hi WanMil, > > attached is the new patch for the performance branch. > Its quite big because I've rewritten many methods to work with the > quadtree > instead of > List<boundary> > > Major changes: > - new file format for *.bnd files. It stores first the boundary
tags,
> then > the areas with float precision. > The file size is not much different from the old format, but when zipped > it > is much smaller. > The time to load the file into the quadtree is shorter than the time to > load > the old format. > The old format is still supported, but requires much more time in the > LocationHook (probably it is still > faster than trunk, but not much) > > - the bounds parameter allows now to specify a zip file with the *.bnd > files. The zip file can have a directory structure, but should not > contain > duplicate *.bnd files. > - BoundaryPreparer uses multiple processors for the > workoutBoundaryRelations() part when --max-jobs parameter allows it. When > called as stand-alone program, it starts one thread for each processor. > > - the utilities BoundaryDiff, BoundaryMerger, BoundaryFile2Gpx, > BoundaryLister were rewritten to use the quadtree, most of them also > allow > *.zip as input. BoundaryLister lists only the OSM tags, not the > information created for the quadtree. Can be changed if needed. > > Open problem: > The BoundaryMerger creates a result that contains a few more small holes > than the trunk version. I'm going to analyse that during the next days. > Todo: Javadoc > > Ciao, > Gerd > > > http://gis.19327.n5.nabble.com/file/n5495676/boundary_prep_quadtree_v4.patch > boundary_prep_quadtree_v4.patch > > -- > View this message in context: > http://gis.19327.n5.nabble.com/PATCH-V4-boundary-preparer-with-quadtree-tp54... > Sent from the Mkgmap Development mailing list archive at Nabble.com. > _______________________________________________ > mkgmap-dev mailing list > mkgmap-dev@.org > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@.org http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- View this message in context: http://gis.19327.n5.nabble.com/PATCH-V4-boundary-preparer-with-quadtree-tp54... Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@.org http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@.org http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@.org http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@.org http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- View this message in context: http://gis.19327.n5.nabble.com/PATCH-V4-boundary-preparer-with-quadtree-tp54... Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi WanMil, fine. Yes, it makes no sense, but I see no reason why it should not produce a result that is more or less identical to the input. By the way, today I learned that Path2D is serializable, do you thing we should prefer the writeObject() method to store the shape info? Gerd WanMil wrote
Hi Gerd,
thanks. It's working now.
I think it does not make any sense to start the BoundaryMerger with identical dir1 and dir2. There would be nothing to merge. So this can be checked and either an error message is issued and the program exits or dir1 is just copied to the merge directory.
WanMil
Hi WanMil,
I think the problem was caused by mixed usage of dir1.getName() and dir1.getAbsolutePath().
Please try the attached patch.
I think the BoundaryMerger doesn't work yet, when you specify the same directory for dir1 and dir2, it sometimes creates files that are much larger than the input. I'm going to look at this tomorrow.
Gerd
http://gis.19327.n5.nabble.com/file/n5496926/boundary_prep_quadtree_v5.patch boundary_prep_quadtree_v5.patch
WanMil wrote
Hi Gerd,
the only message I have in the log file is: 2012/02/19 14:08:34 Schwerwiegend (BoundaryUtil): Cannot read asia 2012/02/19 14:08:35 Schwerwiegend (BoundaryUtil): Cannot read tomerge
WanMil
Hi WanMil,
do you see messages in the log, e.g. "Cannot load boundary file" ? If yes, please provide the content of the two *.bnd files. I think in such a case the merger should stop with some speaking error message. Do you agree?
Gerd
Date: Sun, 19 Feb 2012 13:20:03 +0100 From: wmgcnfg@ To: mkgmap-dev@.org Subject: Re: [mkgmap-dev] [PATCH V4] boundary preparer with quadtree
Hi Gerd,
merging of africa, australia-oceania did work (but I think they have no bnd file in common). Merging of asia to them failed at the first file.
WanMil
Hi WanMil,
thanks. Reg. the error: I have to add some tests to make sure that the BoundaryQuadTree object was created. Did the program already process some bnd files or did that happen right after the program start?
Gerd
WanMil wrote > > Hi Gerd, > > thanks for that patch. It's a big step forward :-) > I've comitted it and also merged all changes from the trunk so that the > branch does not diverge too much. > > I tried to compile boundaries for an old planet file but got an > exception in the BoundaryMerger: > Exception in thread "main" java.lang.NullPointerException > at > uk.me.parabola.mkgmap.reader.osm.boundary.BoundaryMerger.merge(BoundaryMerger.java:93) > at > uk.me.parabola.mkgmap.reader.osm.boundary.BoundaryMerger.main(BoundaryMerger.java:144) > > Can you have a look on it? > Thanks! > > WanMil > >> Hi WanMil, >> >> attached is the new patch for the performance branch. >> Its quite big because I've rewritten many methods to work with the >> quadtree >> instead of >> List<boundary> >> >> Major changes: >> - new file format for *.bnd files. It stores first the boundary tags, >> then >> the areas with float precision. >> The file size is not much different from the old format, but when zipped >> it >> is much smaller. >> The time to load the file into the quadtree is shorter than the time to >> load >> the old format. >> The old format is still supported, but requires much more time in the >> LocationHook (probably it is still >> faster than trunk, but not much) >> >> - the bounds parameter allows now to specify a zip file with the *.bnd >> files. The zip file can have a directory structure, but should not >> contain >> duplicate *.bnd files. >> - BoundaryPreparer uses multiple processors for the >> workoutBoundaryRelations() part when --max-jobs parameter allows it. When >> called as stand-alone program, it starts one thread for each processor. >> >> - the utilities BoundaryDiff, BoundaryMerger, BoundaryFile2Gpx, >> BoundaryLister were rewritten to use the quadtree, most of them also >> allow >> *.zip as input. BoundaryLister lists only the OSM tags, not the >> information created for the quadtree. Can be changed if needed. >> >> Open problem: >> The BoundaryMerger creates a result that contains a few more small holes >> than the trunk version. I'm going to analyse that during the next days. >> Todo: Javadoc >> >> Ciao, >> Gerd >> >> >> http://gis.19327.n5.nabble.com/file/n5495676/boundary_prep_quadtree_v4.patch >> boundary_prep_quadtree_v4.patch >> >> -- >> View this message in context: >> http://gis.19327.n5.nabble.com/PATCH-V4-boundary-preparer-with-quadtree-tp54... >> Sent from the Mkgmap Development mailing list archive at Nabble.com. >> _______________________________________________ >> mkgmap-dev mailing list >> mkgmap-dev@.org >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > _______________________________________________ > mkgmap-dev mailing list > mkgmap-dev@.org > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev >
-- View this message in context: http://gis.19327.n5.nabble.com/PATCH-V4-boundary-preparer-with-quadtree-tp54... Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@.org http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@.org http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@.org http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@.org http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- View this message in context: http://gis.19327.n5.nabble.com/PATCH-V4-boundary-preparer-with-quadtree-tp54... Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@.org http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@.org http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- View this message in context: http://gis.19327.n5.nabble.com/PATCH-V4-boundary-preparer-with-quadtree-tp54... Sent from the Mkgmap Development mailing list archive at Nabble.com.
Hi Gerd, using the serialization means that you will always have to use the Path2D class to read it. You cannot use another lib with better functionality without changing the format. Also other programming languages do not have a real chance to read the files. So unless there is a big advantage regarding size or speed I would not use the Java serialization. WanMil
Hi WanMil,
fine. Yes, it makes no sense, but I see no reason why it should not produce a result that is more or less identical to the input. By the way, today I learned that Path2D is serializable, do you thing we should prefer the writeObject() method to store the shape info?
Gerd
WanMil wrote
Hi Gerd,
thanks. It's working now.
I think it does not make any sense to start the BoundaryMerger with identical dir1 and dir2. There would be nothing to merge. So this can be checked and either an error message is issued and the program exits or dir1 is just copied to the merge directory.
WanMil
Hi WanMil,
I think the problem was caused by mixed usage of dir1.getName() and dir1.getAbsolutePath().
Please try the attached patch.
I think the BoundaryMerger doesn't work yet, when you specify the same directory for dir1 and dir2, it sometimes creates files that are much larger than the input. I'm going to look at this tomorrow.
Gerd
http://gis.19327.n5.nabble.com/file/n5496926/boundary_prep_quadtree_v5.patch boundary_prep_quadtree_v5.patch
WanMil wrote
Hi Gerd,
the only message I have in the log file is: 2012/02/19 14:08:34 Schwerwiegend (BoundaryUtil): Cannot read asia 2012/02/19 14:08:35 Schwerwiegend (BoundaryUtil): Cannot read tomerge
WanMil
Hi WanMil,
do you see messages in the log, e.g. "Cannot load boundary file" ? If yes, please provide the content of the two *.bnd files. I think in such a case the merger should stop with some speaking error message. Do you agree?
Gerd
Date: Sun, 19 Feb 2012 13:20:03 +0100 From: wmgcnfg@ To: mkgmap-dev@.org Subject: Re: [mkgmap-dev] [PATCH V4] boundary preparer with quadtree
Hi Gerd,
merging of africa, australia-oceania did work (but I think they have no bnd file in common). Merging of asia to them failed at the first file.
WanMil
> Hi WanMil, > > thanks. Reg. the error: I have to add some tests to make sure that the > BoundaryQuadTree object was created. > Did the program already process some bnd files or did that happen right > after the program start? > > Gerd > > > > WanMil wrote >> >> Hi Gerd, >> >> thanks for that patch. It's a big step forward :-) >> I've comitted it and also merged all changes from the trunk so that the >> branch does not diverge too much. >> >> I tried to compile boundaries for an old planet file but got an >> exception in the BoundaryMerger: >> Exception in thread "main" java.lang.NullPointerException >> at >> uk.me.parabola.mkgmap.reader.osm.boundary.BoundaryMerger.merge(BoundaryMerger.java:93) >> at >> uk.me.parabola.mkgmap.reader.osm.boundary.BoundaryMerger.main(BoundaryMerger.java:144) >> >> Can you have a look on it? >> Thanks! >> >> WanMil >> >>> Hi WanMil, >>> >>> attached is the new patch for the performance branch. >>> Its quite big because I've rewritten many methods to work with the >>> quadtree >>> instead of >>> List<boundary> >>> >>> Major changes: >>> - new file format for *.bnd files. It stores first the boundary tags, >>> then >>> the areas with float precision. >>> The file size is not much different from the old format, but when zipped >>> it >>> is much smaller. >>> The time to load the file into the quadtree is shorter than the time to >>> load >>> the old format. >>> The old format is still supported, but requires much more time in the >>> LocationHook (probably it is still >>> faster than trunk, but not much) >>> >>> - the bounds parameter allows now to specify a zip file with the *.bnd >>> files. The zip file can have a directory structure, but should not >>> contain >>> duplicate *.bnd files. >>> - BoundaryPreparer uses multiple processors for the >>> workoutBoundaryRelations() part when --max-jobs parameter allows it. When >>> called as stand-alone program, it starts one thread for each processor. >>> >>> - the utilities BoundaryDiff, BoundaryMerger, BoundaryFile2Gpx, >>> BoundaryLister were rewritten to use the quadtree, most of them also >>> allow >>> *.zip as input. BoundaryLister lists only the OSM tags, not the >>> information created for the quadtree. Can be changed if needed. >>> >>> Open problem: >>> The BoundaryMerger creates a result that contains a few more small holes >>> than the trunk version. I'm going to analyse that during the next days. >>> Todo: Javadoc >>> >>> Ciao, >>> Gerd >>> >>> >>> http://gis.19327.n5.nabble.com/file/n5495676/boundary_prep_quadtree_v4.patch >>> boundary_prep_quadtree_v4.patch >>> >>> -- >>> View this message in context: >>> http://gis.19327.n5.nabble.com/PATCH-V4-boundary-preparer-with-quadtree-tp54... >>> Sent from the Mkgmap Development mailing list archive at Nabble.com. >>> _______________________________________________ >>> mkgmap-dev mailing list >>> mkgmap-dev@.org >>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev >> >> _______________________________________________ >> mkgmap-dev mailing list >> mkgmap-dev@.org >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev >> > > > -- > View this message in context: http://gis.19327.n5.nabble.com/PATCH-V4-boundary-preparer-with-quadtree-tp54... > Sent from the Mkgmap Development mailing list archive at Nabble.com. > _______________________________________________ > mkgmap-dev mailing list > mkgmap-dev@.org > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@.org http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@.org http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@.org http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- View this message in context: http://gis.19327.n5.nabble.com/PATCH-V4-boundary-preparer-with-quadtree-tp54... Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@.org http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@.org http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- View this message in context: http://gis.19327.n5.nabble.com/PATCH-V4-boundary-preparer-with-quadtree-tp54... Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hello WanMil, Fine, I think exactly the same way. I did not even try it after looking into the source code of Path2D.java I just wanted to let you know because you told me once that Area doesn't allow serialization and so I thought that you may like to use it. Gerd WanMil wrote
Hi Gerd,
using the serialization means that you will always have to use the Path2D class to read it. You cannot use another lib with better functionality without changing the format. Also other programming languages do not have a real chance to read the files. So unless there is a big advantage regarding size or speed I would not use the Java serialization.
WanMil
Hi WanMil,
fine. Yes, it makes no sense, but I see no reason why it should not produce a result that is more or less identical to the input. By the way, today I learned that Path2D is serializable, do you thing we should prefer the writeObject() method to store the shape info?
Gerd
WanMil wrote
Hi Gerd,
thanks. It's working now.
I think it does not make any sense to start the BoundaryMerger with identical dir1 and dir2. There would be nothing to merge. So this can be checked and either an error message is issued and the program exits or dir1 is just copied to the merge directory.
WanMil
Hi WanMil,
I think the problem was caused by mixed usage of dir1.getName() and dir1.getAbsolutePath().
Please try the attached patch.
I think the BoundaryMerger doesn't work yet, when you specify the same directory for dir1 and dir2, it sometimes creates files that are much larger than the input. I'm going to look at this tomorrow.
Gerd
http://gis.19327.n5.nabble.com/file/n5496926/boundary_prep_quadtree_v5.patch boundary_prep_quadtree_v5.patch
WanMil wrote
Hi Gerd,
the only message I have in the log file is: 2012/02/19 14:08:34 Schwerwiegend (BoundaryUtil): Cannot read asia 2012/02/19 14:08:35 Schwerwiegend (BoundaryUtil): Cannot read tomerge
WanMil
Hi WanMil,
do you see messages in the log, e.g. "Cannot load boundary file" ? If yes, please provide the content of the two *.bnd files. I think in such a case the merger should stop with some speaking error message. Do you agree?
Gerd
> Date: Sun, 19 Feb 2012 13:20:03 +0100 > From: wmgcnfg@ > To: mkgmap-dev@.org > Subject: Re: [mkgmap-dev] [PATCH V4] boundary preparer with quadtree > > Hi Gerd, > > merging of africa, australia-oceania did work (but I think they have no > bnd file in common). Merging of asia to them failed at the first file. > > WanMil > > > Hi WanMil, > > > > thanks. Reg. the error: I have to add some tests to make sure that the > > BoundaryQuadTree object was created. > > Did the program already process some bnd files or did that happen right > > after the program start? > > > > Gerd > > > > > > > > WanMil wrote > >> > >> Hi Gerd, > >> > >> thanks for that patch. It's a big step forward :-) > >> I've comitted it and also merged all changes from the trunk so that the > >> branch does not diverge too much. > >> > >> I tried to compile boundaries for an old planet file but got an > >> exception in the BoundaryMerger: > >> Exception in thread "main" java.lang.NullPointerException > >> at > >> uk.me.parabola.mkgmap.reader.osm.boundary.BoundaryMerger.merge(BoundaryMerger.java:93) > >> at > >> uk.me.parabola.mkgmap.reader.osm.boundary.BoundaryMerger.main(BoundaryMerger.java:144) > >> > >> Can you have a look on it? > >> Thanks! > >> > >> WanMil > >> > >>> Hi WanMil, > >>> > >>> attached is the new patch for the performance branch. > >>> Its quite big because I've rewritten many methods to work with the > >>> quadtree > >>> instead of > >>> List<boundary> > >>> > >>> Major changes: > >>> - new file format for *.bnd files. It stores first the boundary tags, > >>> then > >>> the areas with float precision. > >>> The file size is not much different from the old format, but when zipped > >>> it > >>> is much smaller. > >>> The time to load the file into the quadtree is shorter than the time to > >>> load > >>> the old format. > >>> The old format is still supported, but requires much more time in the > >>> LocationHook (probably it is still > >>> faster than trunk, but not much) > >>> > >>> - the bounds parameter allows now to specify a zip file with the *.bnd > >>> files. The zip file can have a directory structure, but should not > >>> contain > >>> duplicate *.bnd files. > >>> - BoundaryPreparer uses multiple processors for the > >>> workoutBoundaryRelations() part when --max-jobs parameter allows it. When > >>> called as stand-alone program, it starts one thread for each processor. > >>> > >>> - the utilities BoundaryDiff, BoundaryMerger, BoundaryFile2Gpx, > >>> BoundaryLister were rewritten to use the quadtree, most of them also > >>> allow > >>> *.zip as input. BoundaryLister lists only the OSM tags, not the > >>> information created for the quadtree. Can be changed if needed. > >>> > >>> Open problem: > >>> The BoundaryMerger creates a result that contains a few more small holes > >>> than the trunk version. I'm going to analyse that during the next days. > >>> Todo: Javadoc > >>> > >>> Ciao, > >>> Gerd > >>> > >>> > >>> http://gis.19327.n5.nabble.com/file/n5495676/boundary_prep_quadtree_v4.patch > >>> boundary_prep_quadtree_v4.patch > >>> > >>> -- > >>> View this message in context: > >>> http://gis.19327.n5.nabble.com/PATCH-V4-boundary-preparer-with-quadtree-tp54... > >>> Sent from the Mkgmap Development mailing list archive at Nabble.com. > >>> _______________________________________________ > >>> mkgmap-dev mailing list > >>> mkgmap-dev@.org > >>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >> > >> _______________________________________________ > >> mkgmap-dev mailing list > >> mkgmap-dev@.org > >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > >> > > > > > > -- > > View this message in context: http://gis.19327.n5.nabble.com/PATCH-V4-boundary-preparer-with-quadtree-tp54... > > Sent from the Mkgmap Development mailing list archive at Nabble.com. > > _______________________________________________ > > mkgmap-dev mailing list > > mkgmap-dev@.org > > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > _______________________________________________ > mkgmap-dev mailing list > mkgmap-dev@.org > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@.org http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@.org http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- View this message in context: http://gis.19327.n5.nabble.com/PATCH-V4-boundary-preparer-with-quadtree-tp54... Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@.org http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@.org http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- View this message in context: http://gis.19327.n5.nabble.com/PATCH-V4-boundary-preparer-with-quadtree-tp54... Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@.org http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@.org http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- View this message in context: http://gis.19327.n5.nabble.com/PATCH-V4-boundary-preparer-with-quadtree-tp54... Sent from the Mkgmap Development mailing list archive at Nabble.com.
Hi Gerd, I've just started to look through the sources. I haven't read all so maybe some of my comments are just wrong... * class BoundaryLevelCollator in BoundaryLocationPreparer: I think you have copied that from the LocationHook. But now you are using it in a different context which might cause problems. The collator retrieves the name of an admin_level and compares them. In the LocationHook it can use the final name-tag-list but during the preparation step the preparer does not know anything about the name-tag-list (it is possible that I prepare the bounds with "name:de,name:it" and you are using the prepared bounds with "name:zh"). The chances are not very high that this causes a problem but I think during preparation you should check all tags matching "*name*" to be safe. * Good idea to use the max-jobs parameter for the preparation. Maybe you can use the threadPool ExecutorService from the Main class instead of creating your own thread environment? Implementing the preparer as a thread was not my final idea but it was easy to realize so I didn't invest the time to change that. * FindBugs found two warnings: BoundaryQuadTree line 1048: if (nodes.get(i).boundaryId == nodes.get(i-1).boundaryId){ line 1271: if (o1 == o2) { Strings should be compared with equals instead of == That's it for now. WanMil
Hi Gerd,
thanks for that patch. It's a big step forward :-) I've comitted it and also merged all changes from the trunk so that the branch does not diverge too much.
I tried to compile boundaries for an old planet file but got an exception in the BoundaryMerger: Exception in thread "main" java.lang.NullPointerException at uk.me.parabola.mkgmap.reader.osm.boundary.BoundaryMerger.merge(BoundaryMerger.java:93) at uk.me.parabola.mkgmap.reader.osm.boundary.BoundaryMerger.main(BoundaryMerger.java:144)
Can you have a look on it? Thanks!
WanMil
Hi WanMil,
attached is the new patch for the performance branch. Its quite big because I've rewritten many methods to work with the quadtree instead of List<boundary>
Major changes: - new file format for *.bnd files. It stores first the boundary tags, then the areas with float precision. The file size is not much different from the old format, but when zipped it is much smaller. The time to load the file into the quadtree is shorter than the time to load the old format. The old format is still supported, but requires much more time in the LocationHook (probably it is still faster than trunk, but not much)
- the bounds parameter allows now to specify a zip file with the *.bnd files. The zip file can have a directory structure, but should not contain duplicate *.bnd files. - BoundaryPreparer uses multiple processors for the workoutBoundaryRelations() part when --max-jobs parameter allows it. When called as stand-alone program, it starts one thread for each processor.
- the utilities BoundaryDiff, BoundaryMerger, BoundaryFile2Gpx, BoundaryLister were rewritten to use the quadtree, most of them also allow *.zip as input. BoundaryLister lists only the OSM tags, not the information created for the quadtree. Can be changed if needed.
Open problem: The BoundaryMerger creates a result that contains a few more small holes than the trunk version. I'm going to analyse that during the next days. Todo: Javadoc
Ciao, Gerd
http://gis.19327.n5.nabble.com/file/n5495676/boundary_prep_quadtree_v4.patch boundary_prep_quadtree_v4.patch
-- View this message in context: http://gis.19327.n5.nabble.com/PATCH-V4-boundary-preparer-with-quadtree-tp54... Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi WanMil, thanks for the quick response.
I've just started to look through the sources. I haven't read all so maybe some of my comments are just wrong...
I think all are okay...
* class BoundaryLevelCollator in BoundaryLocationPreparer: I think you have copied that from the LocationHook. But now you are using it in a different context which might cause problems. The collator retrieves the name of an admin_level and compares them. In the LocationHook it can use the final name-tag-list but during the preparation step the preparer does not know anything about the name-tag-list (it is possible that I prepare the bounds with "name:de,name:it" and you are using the prepared bounds with "name:zh"). The chances are not very high that this causes a problem but I think during preparation you should check all tags matching "*name*" to be safe.
I did already think about this as well, and in fact today I've removed that part from BoundaryLocationPreparer and changed BoundaryQuadTree to use always the AdminLevelCollator() method and sort the input with that. This one ignores the name, I think. That solved some of the problems in BoundaryMerger. Regarding the name tag: The method BoundaryElementSaver.isBoundary() can remove bounaries that do not have the name tag. This might also be a problem.
* Good idea to use the max-jobs parameter for the preparation. Maybe you can use the threadPool ExecutorService from the Main class instead of creating your own thread environment? Implementing the preparer as a thread was not my final idea but it was easy to realize so I didn't invest the time to change that.
Yes, I think this can be done. We have to split the preparer, so that the 1st pass is executed as a single thread.
* FindBugs found two warnings: BoundaryQuadTree line 1048: if (nodes.get(i).boundaryId == nodes.get(i-1).boundaryId){ line 1271: if (o1 == o2) {
Strings should be compared with equals instead of ==
OK, I will change that. The problem in the BoundaryMerger is solved, it did not produce a wrong result, it just created some deeper sub-trees and therefor had to write more areas. The new version does the oposite: It typically creates smaller trees and smaller output files, so you can think of it as an optimizer :-) I'll post a new patch today. Gerd
That's it for now.
WanMil
Hi Gerd,
thanks for that patch. It's a big step forward :-) I've comitted it and also merged all changes from the trunk so that the branch does not diverge too much.
I tried to compile boundaries for an old planet file but got an exception in the BoundaryMerger: Exception in thread "main" java.lang.NullPointerException at uk.me.parabola.mkgmap.reader.osm.boundary.BoundaryMerger.merge(BoundaryMerger.java:93) at uk.me.parabola.mkgmap.reader.osm.boundary.BoundaryMerger.main(BoundaryMerger.java:144)
Can you have a look on it? Thanks!
WanMil
Hi WanMil,
attached is the new patch for the performance branch. Its quite big because I've rewritten many methods to work with the quadtree instead of List<boundary>
Major changes: - new file format for *.bnd files. It stores first the boundary tags, then the areas with float precision. The file size is not much different from the old format, but when zipped it is much smaller. The time to load the file into the quadtree is shorter than the time to load the old format. The old format is still supported, but requires much more time in the LocationHook (probably it is still faster than trunk, but not much)
- the bounds parameter allows now to specify a zip file with the *.bnd files. The zip file can have a directory structure, but should not contain duplicate *.bnd files. - BoundaryPreparer uses multiple processors for the workoutBoundaryRelations() part when --max-jobs parameter allows it. When called as stand-alone program, it starts one thread for each processor.
- the utilities BoundaryDiff, BoundaryMerger, BoundaryFile2Gpx, BoundaryLister were rewritten to use the quadtree, most of them also allow *.zip as input. BoundaryLister lists only the OSM tags, not the information created for the quadtree. Can be changed if needed.
Open problem: The BoundaryMerger creates a result that contains a few more small holes than the trunk version. I'm going to analyse that during the next days. Todo: Javadoc
Ciao, Gerd
http://gis.19327.n5.nabble.com/file/n5495676/boundary_prep_quadtree_v4.patch boundary_prep_quadtree_v4.patch
-- View this message in context: http://gis.19327.n5.nabble.com/PATCH-V4-boundary-preparer-with-quadtree-tp54... Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
participants (3)
-
Gerd Petermann -
GerdP
-
WanMil