[PATCH] Use Apache Ivy to fetch dependencies, put them in dist
Hi all, Attached is a first attempt at tweaking the build system to automatically fetch the pbf, junit, and macker dependencies as needed. The pbf dependencies are placed in dist/lib so that users can simply run dist/mkgmap.jar after building. Much of the code was adapted from osmosis's build system. It uses Apache Ivy to resolve and download dependencies. Unfortunately, osmpbf.jar has not yet been published to the Maven Central Repository, so it can't be automatically downloaded (easily). This patch assumes that the jar is checked in to the mkgmap svn repository. Any feedback would be appreciated. Thanks, Richard
Hi Richard, I have problems to apply the patch on r2179 using tortoiseSVN on windows. What tool do you suggest to do that? ciao, Gerd Richard Hansen wrote
Hi all,
Attached is a first attempt at tweaking the build system to automatically fetch the pbf, junit, and macker dependencies as needed. The pbf dependencies are placed in dist/lib so that users can simply run dist/mkgmap.jar after building. Much of the code was adapted from osmosis's build system. It uses Apache Ivy to resolve and download dependencies.
Unfortunately, osmpbf.jar has not yet been published to the Maven Central Repository, so it can't be automatically downloaded (easily). This patch assumes that the jar is checked in to the mkgmap svn repository.
Any feedback would be appreciated.
Thanks, Richard
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@.org http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- View this message in context: http://gis.638310.n2.nabble.com/PATCH-Use-Apache-Ivy-to-fetch-dependencies-p... Sent from the Mkgmap Development mailing list archive at Nabble.com.
On 2012-01-22 03:23, GerdP wrote:
Hi Richard,
I have problems to apply the patch on r2179 using tortoiseSVN on windows. What tool do you suggest to do that?
Apologies -- I prefer Git for development so that's what I used to generate the patch. Subversion must not have liked the osmpbf.jar (binary file) section. Attached is the same patch but generated by svn, so tortoiseSVN should be able to apply it. Because svn doesn't support binary diffs, you'll have to manually download osmpbf.jar from <https://github.com/openstreetmap/osmosis/raw/master/build-support/repo/crosby/osmpbf/1.1.1-754a33af/jars/osmpbf.jar> and put it in build-support/repo/crosby/osmpbf/1.1.1-754a33af/jars. Subversion also doesn't support embedding the commit message in patches, so here's the commit message I used in case someone want to reuse some of it: ------------------------------------------------------------------- Use Apache Ivy to fetch dependencies, put them in dist Much of the code was borrowed from osmosis. Notes: * pbf support is no longer conditionally compiled -- it is always enabled * osmpbf has not yet been published to the public Maven Central Repository, so a copy of it is included locally * junit is only fetched if the 'test' target is run * macker and its dependencies are only fetched if the 'macker' target is run * The Ivy publish task needs the project version, so build.xml now sets a property called 'project.version'. The value is currently hard-coded to "svn"; this should probably be replaced with a programmatic method for determining the current svn version. * build.xml now creates the version.properties file with svn.version set to the value of the project.version property. This ensures that the version used by the Ivy publish task matches the version reported by the '--version' argument. * build.xml now sets the Implementation-Version property in the manifest file to the value of the project.version property * the organization setting in the ivy.xml file is set to "uk.me.parabola"; maybe this should be "uk.org.mkgmap"
Hi Richard, I have no idea why, but tortoisesvn still wasn't happy with that patch. Anyway, I tried the build branch and the only thing to do was the manual copying of the pbf jars. Nice! Gerd Date: Sun, 22 Jan 2012 14:34:11 -0500 From: rhansen@bbn.com To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] [PATCH] Use Apache Ivy to fetch dependencies, put them in dist On 2012-01-22 03:23, GerdP wrote:
Hi Richard,
I have problems to apply the patch on r2179 using tortoiseSVN on windows. What tool do you suggest to do that?
Apologies -- I prefer Git for development so that's what I used to generate the patch. Subversion must not have liked the osmpbf.jar (binary file) section. Attached is the same patch but generated by svn, so tortoiseSVN should be able to apply it. Because svn doesn't support binary diffs, you'll have to manually download osmpbf.jar from <https://github.com/openstreetmap/osmosis/raw/master/build-support/repo/crosby/osmpbf/1.1.1-754a33af/jars/osmpbf.jar> and put it in build-support/repo/crosby/osmpbf/1.1.1-754a33af/jars. Subversion also doesn't support embedding the commit message in patches, so here's the commit message I used in case someone want to reuse some of it: ------------------------------------------------------------------- Use Apache Ivy to fetch dependencies, put them in dist Much of the code was borrowed from osmosis. Notes: * pbf support is no longer conditionally compiled -- it is always enabled * osmpbf has not yet been published to the public Maven Central Repository, so a copy of it is included locally * junit is only fetched if the 'test' target is run * macker and its dependencies are only fetched if the 'macker' target is run * The Ivy publish task needs the project version, so build.xml now sets a property called 'project.version'. The value is currently hard-coded to "svn"; this should probably be replaced with a programmatic method for determining the current svn version. * build.xml now creates the version.properties file with svn.version set to the value of the project.version property. This ensures that the version used by the Ivy publish task matches the version reported by the '--version' argument. * build.xml now sets the Implementation-Version property in the manifest file to the value of the project.version property * the organization setting in the ivy.xml file is set to "uk.me.parabola"; maybe this should be "uk.org.mkgmap" _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi
Attached is a first attempt at tweaking the build system to automatically fetch the pbf, junit, and macker dependencies as needed.
I have created a branch 'build' and applied this patch to it. You will have to manually copy the osmpbf.jar file from splitter to build-support/repo/crosby/osmpbf/1.1.1-754a33af/jars/osmpbf.jar if you want to try it out. I have a few comments, but I'm going to play with it a bit more first. There are also add-on files under the 'extra' directory which also need third party jar files, so I think this would be very useful for them. ..Steve
On 2012-01-22 16:25, Steve Ratcliffe wrote:
Hi
Attached is a first attempt at tweaking the build system to automatically fetch the pbf, junit, and macker dependencies as needed.
I have created a branch 'build' and applied this patch to it.
Excellent, thank you.
You will have to manually copy the osmpbf.jar file from splitter to build-support/repo/crosby/osmpbf/1.1.1-754a33af/jars/osmpbf.jar if you want to try it out.
I emailed Scott Crosby to ask if he'd be willing to publish osmpbf to the Maven Central Repository. If it is published, ivy can download it and we can get rid of the build-support/repo directory entirely. Until that happens, is the jar too big to check in to the mkgmap svn repository (~200KB)? It's lgpl, so we'd have to make sure we have access to the source code in case anyone asks. Alternatively, we can configure ivy to download the jar from osmosis's files at GitHub. I skimmed GitHub's terms of service and didn't see anything prohibiting such usage, but we would probably want to ask permission anyway. osmpbf has also been published at <http://repo.opengeo.org/crosby/binary/osmpbf/>, but I am unfamiliar with opengeo.org and their usage policies. Also, the POM's sha1 and md5 checksum files don't match the POM's contents so Ivy refuses to download it (unless we turn off checksum checks, which seems risky).
I have a few comments, but I'm going to play with it a bit more first.
There are also add-on files under the 'extra' directory which also need third party jar files, so I think this would be very useful for them.
We could perhaps add an 'extras' ant target that downloads those dependencies and builds those sources. Or they could be moved into the main build. Thanks, Richard
Hi
I emailed Scott Crosby to ask if he'd be willing to publish osmpbf to the Maven Central Repository. If it is published, ivy can download it and we can get rid of the build-support/repo directory entirely.
OK lets see how that works out. It is already checked into the splitter repo, so maybe I will just link to that for now.
Alternatively, we can configure ivy to download the jar from osmosis's files at GitHub. I skimmed GitHub's terms of service and didn't see anything prohibiting such usage, but we would probably want to ask permission anyway.
Presumably I can create an ivy repo on mkgmap.org.uk too?
We could perhaps add an 'extras' ant target that downloads those dependencies and builds those sources. Or they could be moved into the main build.
Yes it seems like a good oportunity to make that stuff build easily. ..Steve
On 2012-01-23 07:52, Steve Ratcliffe wrote:
Presumably I can create an ivy repo on mkgmap.org.uk too?
Now why didn't I think of that? How embarrassing! :) The files can be placed anywhere, but I recommend this layout: ${mkgmap.ivy.repo}/crosby/osmpbf/1.1.1-754a33af/ivys/ivy.xml ${mkgmap.ivy.repo}/crosby/osmpbf/1.1.1-754a33af/ivys/ivy.xml.md5 ${mkgmap.ivy.repo}/crosby/osmpbf/1.1.1-754a33af/ivys/ivy.xml.sha1 ${mkgmap.ivy.repo}/crosby/osmpbf/1.1.1-754a33af/jars/osmpbf.jar ${mkgmap.ivy.repo}/crosby/osmpbf/1.1.1-754a33af/jars/osmpbf.jar.md5 ${mkgmap.ivy.repo}/crosby/osmpbf/1.1.1-754a33af/jars/osmpbf.jar.sha1 With that layout, you can use the attached patches to tell ivy to fetch osmpbf.jar from the mkgmap.org.uk repository instead of looking in the svn working directory. The ivyconfig.xml file configures ivy to first try the standard locations for stuff, then check the mkgmap.org.uk repository. If splitter's build system is adapted to use ivy, it can download osmpbf.jar from the same place. Ivy caches the files it downloads, so the jar would only need to be downloaded once to build both projects. -Richard
Is ivy a new build requirement, or is that already present with openjdk and ant? (Right now I think mkgmap is documented to only need a jdk and ant.)
On 2012-01-23 21:24, Greg Troxel wrote:
Is ivy a new build requirement, or is that already present with openjdk and ant? (Right now I think mkgmap is documented to only need a jdk and ant.)
Ivy is automatically downloaded using ant's built-in http fetch capability (on the build branch; the changes haven't been merged to master). -Richard
Hi
With that layout, you can use the attached patches to tell ivy to fetch osmpbf.jar from the mkgmap.org.uk repository instead of looking in the svn working directory. The ivyconfig.xml file configures ivy to first try the standard locations for stuff, then check the mkgmap.org.uk repository.
OK, thanks for the patch, I've applied it and followed your layout with the repo located of http://ivy.mkgmap.org.uk/repo I had to remove the ivy.shared.default.root setting to make it work. I'm guessing that this was used to load from svn checked-in jar files, but I would appreciate it if you took a look. The other thing I noticed was that if there are no md5/sha1 files, it does not complain, is it possible to require them to be present? ..Steve
On 2012-01-24 08:53, Steve Ratcliffe wrote:
OK, thanks for the patch, I've applied it and followed your layout with the repo located of http://ivy.mkgmap.org.uk/repo
That's great news, thanks. I tested it out and it seems to work fine.
I had to remove the ivy.shared.default.root setting to make it work. I'm guessing that this was used to load from svn checked-in jar files, but I would appreciate it if you took a look.
Yes, that setting was used to tell ivy to look in the svn working directory. Because you removed the build-support/repo directory entirely, that setting had to be removed to keep ivy happy. It's good they were removed -- an svn working directory repo is unnecessary now that there is an online repo you control. Attached is a patch that restores a comment that appears to have been accidentally altered.
The other thing I noticed was that if there are no md5/sha1 files, it does not complain, is it possible to require them to be present?
Unfortunately I don't see an option for forcing the check. Thanks, Richard
Forgot to mention... On 22/01/12 00:01, Richard Hansen wrote:
* the organization setting in the ivy.xml file is set to "uk.me.parabola"; maybe this should be "uk.org.mkgmap"
Yes, uk.org.mkgmap would be better. I also own org.mkgmap so that would be possible too. Thanks, ..Steve
On 2012-01-23 08:04, Steve Ratcliffe wrote:
On 22/01/12 00:01, Richard Hansen wrote:
* the organization setting in the ivy.xml file is set to "uk.me.parabola"; maybe this should be "uk.org.mkgmap"
Yes, uk.org.mkgmap would be better. I also own org.mkgmap so that would be possible too.
I forgot about fixing this. Attached is a patch. Now that you've had a bit of time to play with the build system changes, what are your thoughts? Do you feel comfortable merging the build branch to trunk? Also, would you like me to work on modifying splitter's build system to match? Thanks, Richard
Hi, Now that you've had a bit of time to play with the build system changes, what are your thoughts? Do you feel comfortable merging the build branch to trunk? For me it is ok, so I'd like to see it merged. Thanks for your work! Ciao, Gerd
Hi
Now that you've had a bit of time to play with the build system changes, what are your thoughts? Do you feel comfortable merging the build branch to trunk?
I think that this will be really good, and I've been playing around with it quite a lot over the last day or so. There are a few more jars needed to set up a development environment so that everything can be compiled. I added the following dependancies to my working copy: For the mkgmap ant task: <dependency org="org.apache.ant" name="ant" rev="1.8.2" conf="compile->compile(*)"/> Development only (internationalisation): <dependency org="com.ibm.icu" name="icu4j" rev="4.8" conf="compile->compile(*)"/> There is the optional DEM reader which needs the java JAI libraries. I had trouble finding a good repository for this. The best I found is: <dependency org="javax.media.jai" name="com.springsource.javax.media.jai.codec" rev="1.1.3" conf="*->compile" /> with the repository definition: <url name="com.springsource.repository.bundles.external"> <ivy pattern="http://repository.springsource.com/ivy/bundles/external/[organisation]/[module]/[revision]/[artifact]-[revision].[ext]" /> <artifact pattern="http://repository.springsource.com/ivy/bundles/external/[organisation]/[module]/[revision]/[artifact]-[revision].[ext]" /> </url> I think we should have three configurations, default/master for the normal build, "optional" for things that are not part of the main distribution but which people might like to use (including anything where we cannot distribute the jars, which may be the case with JAI - need more investigation) and "dev" to cover macker, icu4j etc. The idea being that resolving dev would bring in everything needed to get going. I'd also like to remove the build-support directory, and just move the scripts to the top level and combine them into one or even add them to the build.xml. The ivy jar file can go under lib/build or something.
Also, would you like me to work on modifying splitter's build system to match?
Yes that would be great. Thanks Steve
On 2012-01-30 19:13, Steve Ratcliffe wrote:
There are a few more jars needed to set up a development environment so that everything can be compiled. I added the following dependancies to my working copy:
For the mkgmap ant task: <dependency org="org.apache.ant" name="ant" rev="1.8.2" conf="compile->compile(*)"/>
Development only (internationalisation): <dependency org="com.ibm.icu" name="icu4j" rev="4.8" conf="compile->compile(*)"/>
These two are for the files in extra, correct? I don't see an ant target for the extras, only an entry in mkgmap.iml. Do you want to add a special ant target for these? Or simply always include them in the main build?
There is the optional DEM reader which needs the java JAI libraries. I had trouble finding a good repository for this. The best I found is:
<dependency org="javax.media.jai" name="com.springsource.javax.media.jai.codec" rev="1.1.3" conf="*->compile" />
with the repository definition:
<url name="com.springsource.repository.bundles.external"> <ivy pattern="http://repository.springsource.com/ivy/bundles/external/[organisation]/[module]/[revision]/[artifact]-[revision].[ext]" /> <artifact pattern="http://repository.springsource.com/ivy/bundles/external/[organisation]/[module]/[revision]/[artifact]-[revision].[ext]" /> </url>
That seems like a perfectly fine repository to add to the ivyconfig.xml.
I think we should have three configurations, default/master for the normal build, "optional" for things that are not part of the main distribution but which people might like to use (including anything where we cannot distribute the jars, which may be the case with JAI - need more investigation) and "dev" to cover macker, icu4j etc. The idea being that resolving dev would bring in everything needed to get going.
I think it's better to think of it from a target-centric perspective. In other words, a jar should only be downloaded if it is needed to successfully complete the target. For example, macker is only needed if you run 'ant macker', so there's no reason to download it otherwise. If there are multiple flavors of the standard build (e.g., full vs lite, with vs without DEM support), then things get slightly tricky. We could either have multiple targets (one per flavor), or we could have a single target that is controlled by a user-settable property. I'm not sure which approach is better.
I'd also like to remove the build-support directory, and just move the scripts to the top level and combine them into one or even add them to the build.xml. The ivy jar file can go under lib/build or something.
I agree; I'll make a patch.
Also, would you like me to work on modifying splitter's build system to match?
Yes that would be great.
OK, I'll work on it over the weekend. Thanks, Richard
On 2012-01-31 00:46, Richard Hansen wrote:
On 2012-01-30 19:13, Steve Ratcliffe wrote:
I'd also like to remove the build-support directory, and just move the scripts to the top level and combine them into one or even add them to the build.xml. The ivy jar file can go under lib/build or something.
I agree; I'll make a patch.
Attached is a series of patches that moves the contents of build-support/scripts/*.xml into build.xml, derives the project version from the svn revision, and fixes some minor issues. Thanks, Richard
Hi Richard
For the mkgmap ant task: <dependency org="org.apache.ant" name="ant" rev="1.8.2" conf="compile->compile(*)"/>
Development only (internationalisation): <dependency org="com.ibm.icu" name="icu4j" rev="4.8" conf="compile->compile(*)"/>
These two are for the files in extra, correct?
The first one (ant) is, the other is in fact in the main src tree, but is excluded from the compilation in the build.xml file.
I don't see an ant target for the extras, only an entry in mkgmap.iml. Do you want to add a special ant target for these? Or simply always include them in the main build?
Thats right there isn't, I don't know if anyone uses the code or even if it still works. So I don't see this as something that needs to be addressed before we move over to the new system.
I think it's better to think of it from a target-centric perspective. In other words, a jar should only be downloaded if it is needed to successfully complete the target. For example, macker is only needed if you run 'ant macker', so there's no reason to download it otherwise.
Thats fine about the targets, but there are really only two situations in my mind: (1) build the standard distribution and (2) get everything needed for development, so that nothing shows up as red in an IDE and you can check that nothing is broken including things that are not distributed. Lets ignore optional stuff at the moment. Also aren't a lot of the configurations that *are* there, not used. ..Steve
I've attached a few more patches: * #1, #2 are repeats that appear to have slipped through the cracks * #3 adds support for deriving the project version from the git revision (in case the svn repository is ever converted to git, and for those like myself who like to work in git) * #4 updates the .gitignore file.
I don't see an ant target for the extras, only an entry in mkgmap.iml. Do you want to add a special ant target for these? Or simply always include them in the main build?
Thats right there isn't, I don't know if anyone uses the code or even if it still works. So I don't see this as something that needs to be addressed before we move over to the new system.
I agree; maybe we can create a new branch after build is reintegrated into trunk.
Thats fine about the targets, but there are really only two situations in my mind: (1) build the standard distribution and (2) get everything needed for development, so that nothing shows up as red in an IDE and you can check that nothing is broken including things that are not distributed.
You can run 'ant resolve' to get all dependencies. Is that sufficient? (The term "resolve" is not my favorite, but it's what osmosis uses so I kept it.)
Lets ignore optional stuff at the moment.
Agreed. -Richard
On 05/02/12 21:35, Richard Hansen wrote:
I've attached a few more patches: * #1, #2 are repeats that appear to have slipped through the cracks
I don't think that #2 actually clarifies anything. If I see a name like ivy.compile.classpath I'd think it was a path that was only for compiling ivy or something like that. How the files on the classpath are obtained doesn't seem relevant to its use to me. ..Steve
Hi [Forgot about this bit.]
You can run 'ant resolve' to get all dependencies. Is that sufficient? (The term "resolve" is not my favorite, but it's what osmosis uses so I kept it.)
That is fine, as long as the jars that are development-only do not get into dist/lib ..Steve
participants (5)
-
Gerd Petermann -
GerdP
-
Greg Troxel -
Richard Hansen -
Steve Ratcliffe