Hi all, Java 1.7 is no longer maintained, so I guess that most users are already using Java 1.8 I'd like to change the build of splitter and mkgmap to use 1.8 instead of 1.7 soon, JOSM already requires 1.8. I needed a newer ant version, 1.8.4 did not work with JDK 1.8, but 1.9.7 seems to work fine. @Steve : I think we don't need a warning message for users with an older JRE, but I don't know if you need time to change the build environment? Gerd
On 07/31/2016 07:03 PM, Gerd Petermann wrote:
JOSM already requires 1.8.
JOSM is not entirely there yet. The next JOSM tested snapshot will be the first to require Java 8, the current tested snapshot still uses Java 7. https://josm.openstreetmap.de/ticket/11390 Kind Regards, Bas -- GPG Key ID: 4096R/6750F10AE88D4AF1 Fingerprint: 8182 DE41 7056 408D 6146 50D1 6750 F10A E88D 4AF1
I already use Java 8 nearly since the beginning and have checked out the workspace for splitter and mkgmap into eclipse Neon without problems. My local project is modified to produce Java 8 output. With this change it would be possible to replace a lot of list operations with the new streaming interface for the collections of Java 8 and thereby a significant performance boost is possible. So i'm all in for that! Best regards, Uli -- View this message in context: http://gis.19327.n5.nabble.com/Move-to-Java-1-8-tp5879546p5879555.html Sent from the Mkgmap Development mailing list archive at Nabble.com.
I think most users switched a long time ago... On 31 July 2016 at 22:10, UliBaer <ulibaer@gmail.com> wrote:
I already use Java 8 nearly since the beginning and have checked out the workspace for splitter and mkgmap into eclipse Neon without problems. My local project is modified to produce Java 8 output. With this change it would be possible to replace a lot of list operations with the new streaming interface for the collections of Java 8 and thereby a significant performance boost is possible. So i'm all in for that!
Best regards, Uli
-- View this message in context: http://gis.19327.n5.nabble.com/Move-to-Java-1-8-tp5879546p5879555.html Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- Felix Hartman - Openmtbmap.org & VeloMap.org Schusterbergweg 32/8 6020 Innsbruck Austria - Österreich
On 07/31/2016 10:13 PM, Felix Hartmann wrote:
I think most users switched a long time ago...
All except Linux users where distributions have not switched the default to Java 8 yet, as can be seen in the JOSM statistics. Debian stable still has JDK 7 as default, but has JDK 8 available in backports for example. Those users still mostly use Java 7 unless they explicitly chose to use the Java 8 backport (or a backported application that requires it). On Windows & OSX the Oracle JRE tends to be used instead of what comes with the OS, those users have mostly all switched to Java 8. There should be no blockers to move mkgmap to require Java 8, except on really old Linux distro releases which don't include OpenJDK 8, e.g. Debian wheezy & Ubuntu trusty. Those Debian users can build their own JDK packages using make-jpkg from java-package [0], and those Ubuntu users can use the oracle-java8-installer packages from the webup8team PPA [1]. [0] https://tracker.debian.org/pkg/java-package [1] https://launchpad.net/~webupd8team/+archive/ubuntu/java/+packages Kind Regards, Bas -- GPG Key ID: 4096R/6750F10AE88D4AF1 Fingerprint: 8182 DE41 7056 408D 6146 50D1 6750 F10A E88D 4AF1
Hi Uli, do you have an example for me? Gerd ________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von UliBaer <ulibaer@gmail.com> Gesendet: Sonntag, 31. Juli 2016 22:10:56 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] Move to Java 1.8 I already use Java 8 nearly since the beginning and have checked out the workspace for splitter and mkgmap into eclipse Neon without problems. My local project is modified to produce Java 8 output. With this change it would be possible to replace a lot of list operations with the new streaming interface for the collections of Java 8 and thereby a significant performance boost is possible. So i'm all in for that! Best regards, Uli -- View this message in context: http://gis.19327.n5.nabble.com/Move-to-Java-1-8-tp5879546p5879555.html Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Gerd Petermann wrote
Hi Uli,
do you have an example for me?
Gerd
Hi Gerd, i'll have to look deeper into the source code, but i assume, mkgmap also uses list operations in the calculations? I had major performance gains in a commercial project, migrating bigger and longish list operations from standard api to Java 8 streaming api, especially by using the parallel streams on multi core machines. Best regards, Uli -- View this message in context: http://gis.19327.n5.nabble.com/Move-to-Java-1-8-tp5879546p5879580.html Sent from the Mkgmap Development mailing list archive at Nabble.com.
Hi Uli, thanks for the hint. I'll experiment with this in future. At the moment I don't know of any bottlenecks which could be solved by this. Gerd ________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von UliBaer <ulibaer@gmail.com> Gesendet: Montag, 1. August 2016 13:23:50 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] Move to Java 1.8 Gerd Petermann wrote
Hi Uli,
do you have an example for me?
Gerd
Hi Gerd, i'll have to look deeper into the source code, but i assume, mkgmap also uses list operations in the calculations? I had major performance gains in a commercial project, migrating bigger and longish list operations from standard api to Java 8 streaming api, especially by using the parallel streams on multi core machines. Best regards, Uli -- View this message in context: http://gis.19327.n5.nabble.com/Move-to-Java-1-8-tp5879546p5879580.html Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Gerd Petermann wrote
Hi Uli,
thanks for the hint. I'll experiment with this in future. At the moment I don't know of any bottlenecks which could be solved by this.
Gerd
Hi Gerd, yesterday evening i played a little bit with list streaming and found an (albeit small) speed increase in modifying some source code. Unfortunately the loop code is mainly not written in a "streaming friendly" way, e. g. with lots of local variable access inside the loop. I have attached a patch out of eclipse against version 3688. If i find more time, i'll dig deeper into that. Best regards, Uli Attached patch: patch.txt <http://gis.19327.n5.nabble.com/file/n5879637/patch.txt> -- View this message in context: http://gis.19327.n5.nabble.com/Move-to-Java-1-8-tp5879546p5879637.html Sent from the Mkgmap Development mailing list archive at Nabble.com.
Hi Uli, thanks for the patch. I tried it and found no significant change in run time or memory usage. My test case: Compile 9 tiles (a part of the Britisch Isles) (with --max-jobs on 4 cores) In what scenario do you think that the stream methods should save time? My understanding is that the list streaming is better as it allows automatic use of multiple cores, so I'd expect improvements in e.g. the gmapsupp builder which uses a single thread, but not in the code which compiles the tiles. On the other hand, the combiners which run single threaded are probably more I/O bound. Gerd ________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von UliBaer <ulibaer@gmail.com> Gesendet: Dienstag, 2. August 2016 10:53:15 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] Move to Java 1.8 Gerd Petermann wrote
Hi Uli,
thanks for the hint. I'll experiment with this in future. At the moment I don't know of any bottlenecks which could be solved by this.
Gerd
Hi Gerd, yesterday evening i played a little bit with list streaming and found an (albeit small) speed increase in modifying some source code. Unfortunately the loop code is mainly not written in a "streaming friendly" way, e. g. with lots of local variable access inside the loop. I have attached a patch out of eclipse against version 3688. If i find more time, i'll dig deeper into that. Best regards, Uli Attached patch: patch.txt <http://gis.19327.n5.nabble.com/file/n5879637/patch.txt> -- View this message in context: http://gis.19327.n5.nabble.com/Move-to-Java-1-8-tp5879546p5879637.html Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Gerd Petermann wrote
Hi Uli,
thanks for the patch. I tried it and found no significant change in run time or memory usage. My test case: Compile 9 tiles (a part of the Britisch Isles) (with --max-jobs on 4 cores) In what scenario do you think that the stream methods should save time?
My understanding is that the list streaming is better as it allows automatic use of multiple cores, so I'd expect improvements in e.g. the gmapsupp builder which uses a single thread, but not in the code which compiles the tiles. On the other hand, the combiners which run single threaded are probably more I/O bound.
Gerd
Hi Gerd, the patch only used the sequential .stream(). method. If you want to use multiple cores, simply replace the .stream(). with .parallelStream(). and the jvm tries to work parallel pipelines with multiple threads. You only have to be aware, that the order of processing is completely arbitrary in that case, so the atomic operations have to be independent of each other and not depend on the order of execution. Also the order of elements in the resulting list is arbitrary. This can be worked around by using a SortedList for the collection or by using .sorted((o1, o2) -> o1.getField().compareTo(o2.getField())). near the end of the streaming pipeline. Best regards, Uli -- View this message in context: http://gis.19327.n5.nabble.com/Move-to-Java-1-8-tp5879546p5879702.html Sent from the Mkgmap Development mailing list archive at Nabble.com.
Hi Uli + Steve, reg. config files: I think the file .classpath should also be changed to state 1-8, see attached patch. Gerd ________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von UliBaer <ulibaer@gmail.com> Gesendet: Dienstag, 2. August 2016 18:11:34 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] Move to Java 1.8 Gerd Petermann wrote
Hi Uli,
thanks for the patch. I tried it and found no significant change in run time or memory usage. My test case: Compile 9 tiles (a part of the Britisch Isles) (with --max-jobs on 4 cores) In what scenario do you think that the stream methods should save time?
My understanding is that the list streaming is better as it allows automatic use of multiple cores, so I'd expect improvements in e.g. the gmapsupp builder which uses a single thread, but not in the code which compiles the tiles. On the other hand, the combiners which run single threaded are probably more I/O bound.
Gerd
Hi Gerd, the patch only used the sequential .stream(). method. If you want to use multiple cores, simply replace the .stream(). with .parallelStream(). and the jvm tries to work parallel pipelines with multiple threads. You only have to be aware, that the order of processing is completely arbitrary in that case, so the atomic operations have to be independent of each other and not depend on the order of execution. Also the order of elements in the resulting list is arbitrary. This can be worked around by using a SortedList for the collection or by using .sorted((o1, o2) -> o1.getField().compareTo(o2.getField())). near the end of the streaming pipeline. Best regards, Uli -- View this message in context: http://gis.19327.n5.nabble.com/Move-to-Java-1-8-tp5879546p5879702.html Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi all, reg. fastutil: I did a few tests with splitter and mkgmap using the 6.5.15 and the 7.0.11 version, found no change in output (which is good) but also no significant change in memory usage or run time, although the changelog for 6.6.0 mentions "significant improvements" . Maybe the reason is that we already use custom classes for all collections which are critical, maybe one has to change the source to see these improvements. The changelog states that some old-standing bugs were fixed. I did not find any place in the sources of splitter or mkgmap which uses the bogus routines. So, in short, there is no strong need to move to 7.0.11 now. The advantage would be that we are "up to date" , the disadvantage is a possible trouble for users when they mix old and new jars, which happens often when you compare the result of an older mkgmap version with that of the current version. My conclusion: Let's stay with the old fastuitl libs. Gerd ________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <GPetermann_muenchen@hotmail.com> Gesendet: Mittwoch, 3. August 2016 07:47:44 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] Move to Java 1.8 Hi Uli + Steve, reg. config files: I think the file .classpath should also be changed to state 1-8, see attached patch. Gerd ________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von UliBaer <ulibaer@gmail.com> Gesendet: Dienstag, 2. August 2016 18:11:34 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] Move to Java 1.8 Gerd Petermann wrote
Hi Uli,
thanks for the patch. I tried it and found no significant change in run time or memory usage. My test case: Compile 9 tiles (a part of the Britisch Isles) (with --max-jobs on 4 cores) In what scenario do you think that the stream methods should save time?
My understanding is that the list streaming is better as it allows automatic use of multiple cores, so I'd expect improvements in e.g. the gmapsupp builder which uses a single thread, but not in the code which compiles the tiles. On the other hand, the combiners which run single threaded are probably more I/O bound.
Gerd
Hi Gerd, the patch only used the sequential .stream(). method. If you want to use multiple cores, simply replace the .stream(). with .parallelStream(). and the jvm tries to work parallel pipelines with multiple threads. You only have to be aware, that the order of processing is completely arbitrary in that case, so the atomic operations have to be independent of each other and not depend on the order of execution. Also the order of elements in the resulting list is arbitrary. This can be worked around by using a SortedList for the collection or by using .sorted((o1, o2) -> o1.getField().compareTo(o2.getField())). near the end of the streaming pipeline. Best regards, Uli -- View this message in context: http://gis.19327.n5.nabble.com/Move-to-Java-1-8-tp5879546p5879702.html Mkgmap Development - Move to Java 1.8<http://gis.19327.n5.nabble.com/Move-to-Java-1-8-tp5879546p5879702.html> gis.19327.n5.nabble.com Move to Java 1.8. Hi all, Java 1.7 is no longer maintained, so I guess that most users are already using Java 1.8 I'd like to change the build of splitter and mkgmap to use 1.8 instead of 1.7... Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Uli, sorry for the delay. In the meantime I coded some patches for JOSM. I do not yet see the advantage of using the .stream() method, maybe because I am not used to lambda expressions. Why do you favor them in the code snippets touched by the patch? Gerd ________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von UliBaer <ulibaer@gmail.com> Gesendet: Dienstag, 2. August 2016 18:11:34 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] Move to Java 1.8 Gerd Petermann wrote
Hi Uli,
thanks for the patch. I tried it and found no significant change in run time or memory usage. My test case: Compile 9 tiles (a part of the Britisch Isles) (with --max-jobs on 4 cores) In what scenario do you think that the stream methods should save time?
My understanding is that the list streaming is better as it allows automatic use of multiple cores, so I'd expect improvements in e.g. the gmapsupp builder which uses a single thread, but not in the code which compiles the tiles. On the other hand, the combiners which run single threaded are probably more I/O bound.
Gerd
Hi Gerd, the patch only used the sequential .stream(). method. If you want to use multiple cores, simply replace the .stream(). with .parallelStream(). and the jvm tries to work parallel pipelines with multiple threads. You only have to be aware, that the order of processing is completely arbitrary in that case, so the atomic operations have to be independent of each other and not depend on the order of execution. Also the order of elements in the resulting list is arbitrary. This can be worked around by using a SortedList for the collection or by using .sorted((o1, o2) -> o1.getField().compareTo(o2.getField())). near the end of the streaming pipeline. Best regards, Uli -- View this message in context: http://gis.19327.n5.nabble.com/Move-to-Java-1-8-tp5879546p5879702.html Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Gerd
@Steve : I think we don't need a warning message for users with an older JRE,
but I don't know if you need time to change the build environment?
Can change whenever you want. It is already built and tested on jdk8 (and openjdk8), just need to change the default over and then remove the jdk7 build/test. ..Steve
Hi Steve, I think there is no reason to wait, so go ahead. Not sure what to do with the libraries, I noticed that fastutil was changed heavily, so I'd first try to find out if those changes have a positive impact or not. Gerd ________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Steve Ratcliffe <steve@parabola.me.uk> Gesendet: Montag, 1. August 2016 00:05:38 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] Move to Java 1.8 Hi Gerd
@Steve : I think we don't need a warning message for users with an older JRE,
but I don't know if you need time to change the build environment?
Can change whenever you want. It is already built and tested on jdk8 (and openjdk8), just need to change the default over and then remove the jdk7 build/test. ..Steve _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Gerd
I think there is no reason to wait, so go ahead.
OK thats changed so any new build of mkgmap/splitter will be built with java 8. I will remove the "1.7" lines in the build.xml and commit to create the first java 8 build.
Not sure what to do with the libraries, I noticed that fastutil was changed
Are you looking to use 7.0.x of fastutil? If so I will build a trimmed jar of it. ..Steve
heavily, so I'd first try to find out if those changes have a positive impact
or not.
Gerd
------------------------------------------------------------------------ *Von:* mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Steve Ratcliffe <steve@parabola.me.uk> *Gesendet:* Montag, 1. August 2016 00:05:38 *An:* Development list for mkgmap *Betreff:* Re: [mkgmap-dev] Move to Java 1.8
Hi Gerd
@Steve : I think we don't need a warning message for users with an older JRE,
but I don't know if you need time to change the build environment?
Can change whenever you want. It is already built and tested on jdk8 (and openjdk8), just need to change the default over and then remove the jdk7 build/test.
..Steve
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Steve, okay, reg. fastutil: I want to compare 6.5.15 with 6.6.x and 7.*both with splitter and mkgmap. I'll try to gen the java sources on my machine, if that doesn't work I'll let you know. As a first step I think you should only compile the existing libs for 1.8. I assume it is better to give those new versions a different name? Gerd ________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Steve Ratcliffe <steve@parabola.me.uk> Gesendet: Montag, 1. August 2016 12:19:09 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] Move to Java 1.8 Hi Gerd
I think there is no reason to wait, so go ahead.
OK thats changed so any new build of mkgmap/splitter will be built with java 8. I will remove the "1.7" lines in the build.xml and commit to create the first java 8 build.
Not sure what to do with the libraries, I noticed that fastutil was changed
Are you looking to use 7.0.x of fastutil? If so I will build a trimmed jar of it. ..Steve
heavily, so I'd first try to find out if those changes have a positive impact
or not.
Gerd
------------------------------------------------------------------------ *Von:* mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Steve Ratcliffe <steve@parabola.me.uk> *Gesendet:* Montag, 1. August 2016 00:05:38 *An:* Development list for mkgmap *Betreff:* Re: [mkgmap-dev] Move to Java 1.8
Hi Gerd
@Steve : I think we don't need a warning message for users with an older JRE,
but I don't know if you need time to change the build environment?
Can change whenever you want. It is already built and tested on jdk8 (and openjdk8), just need to change the default over and then remove the jdk7 build/test.
..Steve
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
participants (5)
-
Felix Hartmann -
Gerd Petermann -
Sebastiaan Couwenberg -
Steve Ratcliffe -
UliBaer