Re: [mkgmap-dev] Error in memory calculation
Hi Manfred, mkgmap contains code to estimate a suitable value for max-jobs if this parameter isn't set. This code fails and the result is that it believes that a first thread needed all available memory. This doesn't happen when the -Xms parameter in my sample call is removed, and it also doesn't happen with JRE 1.8 (both from OpenJDK and Oracle). I've not yet started to look at the details returned in class java.lang.management.MemoryPoolMXBean Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von DD8KQ <dd8kq@gmx.de> Gesendet: Mittwoch, 5. Februar 2020 21:16 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] Error in memory calculation Hi Gerd i'm not sure what you are looking for. In my build environment i'm using java 1.8.0_241 (64 bit) and as my laptop have 32 Gb of memory availiable my command line to build is like this : java -Xmx30000m -XX:-UseGCOverheadLimit -ea -jar -Dlog.config=logging.properties "c:\Program Files (x86)\KartenTools\mkgmap\mkgmap.jar" .... and depending on which map i want to build, the occupied memory runs up nearly to 30Gb Am 05.02.2020 um 14:05 schrieb Ticker Berkin:
Hi Gerd
I don't have a 64bit environment or that much memory. In my normal environment, I get:
$ java -Xms1500M -Xmx1540M -jar ../mkgmap.trunk/mkgmap.jar \ dummy.osm dummy.osm dummy.osm Time started: Wed Feb 05 12:56:28 GMT 2020 Setting max-jobs to 4 Number of MapFailedExceptions: 0 Number of ExitExceptions: 0 Time finished: Wed Feb 05 12:56:29 GMT 2020 Total time taken: 1 second
-Xmx1540M is the maximum value that allows Java to start up on my machine.
With a newer version I loaded in October, but it didn't have any obvious advantages:
$ /opt/jre1.8.0_231/bin/java -Xms1500M -Xmx1540M \ -jar ../mkgmap.trunk/mkgmap.jar \ dummy.osm dummy.osm dummy.osm Time started: Wed Feb 05 12:58:29 GMT 2020 Setting max-jobs to 4 Number of MapFailedExceptions: 0 Number of ExitExceptions: 0 Time finished: Wed Feb 05 12:58:30 GMT 2020 Total time taken: 1 second
Ticker
On Fri, 2020-01-31 at 16:15 +0000, Gerd Petermann wrote:
please try this command with openjdk 11 or 13 64Bit: java -Xms4G -Xmx10G -jar mkgmap.jar f:\osm\dummy.osm f:\osm\dummy.osm f:\osm\dummy.osm This is what I get: Time started: Fri Jan 31 17:12:53 CET 2020 Setting max-jobs to 1 Number of MapFailedExceptions: 0 To reduce the run time, consider increasing the amnount of memory available for use by mkgmap by using the Java -Xmx flag to set the memory to more than 41000 MB, providing this is less than the amount of physical memory installed. Number of ExitExceptions: 0 Time finished: Fri Jan 31 17:12:54 CET 2020 Total time taken: 1 second
mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- ##################################################### Viele Grüße und 73 de Manfred Haiduk, DD8KQ e-mail mhaiduk@t-online.de dd8kq@gmx.de ##################################################### _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Gerd just did a small map job but on the console there is no print about number of max jobs. Is there a special log flag to get this print out ? What i've seen in the task manager while building the map, all 8 cpu's are used, but i don't know anything about the number of jobs ... Am 05.02.2020 um 21:35 schrieb Gerd Petermann:
Hi Manfred,
mkgmap contains code to estimate a suitable value for max-jobs if this parameter isn't set. This code fails and the result is that it believes that a first thread needed all available memory. This doesn't happen when the -Xms parameter in my sample call is removed, and it also doesn't happen with JRE 1.8 (both from OpenJDK and Oracle). I've not yet started to look at the details returned in class java.lang.management.MemoryPoolMXBean
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von DD8KQ <dd8kq@gmx.de> Gesendet: Mittwoch, 5. Februar 2020 21:16 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] Error in memory calculation
Hi Gerd
i'm not sure what you are looking for. In my build environment i'm using java 1.8.0_241 (64 bit) and as my laptop have 32 Gb of memory availiable my command line to build is like this :
java -Xmx30000m -XX:-UseGCOverheadLimit -ea -jar -Dlog.config=logging.properties "c:\Program Files (x86)\KartenTools\mkgmap\mkgmap.jar" ....
and depending on which map i want to build, the occupied memory runs up nearly to 30Gb
Am 05.02.2020 um 14:05 schrieb Ticker Berkin:
Hi Gerd
I don't have a 64bit environment or that much memory. In my normal environment, I get:
$ java -Xms1500M -Xmx1540M -jar ../mkgmap.trunk/mkgmap.jar \ dummy.osm dummy.osm dummy.osm Time started: Wed Feb 05 12:56:28 GMT 2020 Setting max-jobs to 4 Number of MapFailedExceptions: 0 Number of ExitExceptions: 0 Time finished: Wed Feb 05 12:56:29 GMT 2020 Total time taken: 1 second
-Xmx1540M is the maximum value that allows Java to start up on my machine.
With a newer version I loaded in October, but it didn't have any obvious advantages:
$ /opt/jre1.8.0_231/bin/java -Xms1500M -Xmx1540M \ -jar ../mkgmap.trunk/mkgmap.jar \ dummy.osm dummy.osm dummy.osm Time started: Wed Feb 05 12:58:29 GMT 2020 Setting max-jobs to 4 Number of MapFailedExceptions: 0 Number of ExitExceptions: 0 Time finished: Wed Feb 05 12:58:30 GMT 2020 Total time taken: 1 second
Ticker
On Fri, 2020-01-31 at 16:15 +0000, Gerd Petermann wrote:
please try this command with openjdk 11 or 13 64Bit: java -Xms4G -Xmx10G -jar mkgmap.jar f:\osm\dummy.osm f:\osm\dummy.osm f:\osm\dummy.osm This is what I get: Time started: Fri Jan 31 17:12:53 CET 2020 Setting max-jobs to 1 Number of MapFailedExceptions: 0 To reduce the run time, consider increasing the amnount of memory available for use by mkgmap by using the Java -Xmx flag to set the memory to more than 41000 MB, providing this is less than the amount of physical memory installed. Number of ExitExceptions: 0 Time finished: Fri Jan 31 17:12:54 CET 2020 Total time taken: 1 second
mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev --
#####################################################
Viele Grüße und 73 de Manfred Haiduk, DD8KQ e-mail mhaiduk@t-online.de dd8kq@gmx.de
#####################################################
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- ##################################################### Viele Grüße und 73 de Manfred Haiduk, DD8KQ e-mail mhaiduk@t-online.de dd8kq@gmx.de #####################################################
Hi Manfred, the calculation is only done when you don't use the --max-jobs option and when more than two input files are processed. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von DD8KQ <dd8kq@gmx.de> Gesendet: Mittwoch, 5. Februar 2020 21:49 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] Error in memory calculation Hi Gerd just did a small map job but on the console there is no print about number of max jobs. Is there a special log flag to get this print out ? What i've seen in the task manager while building the map, all 8 cpu's are used, but i don't know anything about the number of jobs ... Am 05.02.2020 um 21:35 schrieb Gerd Petermann:
Hi Manfred,
mkgmap contains code to estimate a suitable value for max-jobs if this parameter isn't set. This code fails and the result is that it believes that a first thread needed all available memory. This doesn't happen when the -Xms parameter in my sample call is removed, and it also doesn't happen with JRE 1.8 (both from OpenJDK and Oracle). I've not yet started to look at the details returned in class java.lang.management.MemoryPoolMXBean
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von DD8KQ <dd8kq@gmx.de> Gesendet: Mittwoch, 5. Februar 2020 21:16 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] Error in memory calculation
Hi Gerd
i'm not sure what you are looking for. In my build environment i'm using java 1.8.0_241 (64 bit) and as my laptop have 32 Gb of memory availiable my command line to build is like this :
java -Xmx30000m -XX:-UseGCOverheadLimit -ea -jar -Dlog.config=logging.properties "c:\Program Files (x86)\KartenTools\mkgmap\mkgmap.jar" ....
and depending on which map i want to build, the occupied memory runs up nearly to 30Gb
Am 05.02.2020 um 14:05 schrieb Ticker Berkin:
Hi Gerd
I don't have a 64bit environment or that much memory. In my normal environment, I get:
$ java -Xms1500M -Xmx1540M -jar ../mkgmap.trunk/mkgmap.jar \ dummy.osm dummy.osm dummy.osm Time started: Wed Feb 05 12:56:28 GMT 2020 Setting max-jobs to 4 Number of MapFailedExceptions: 0 Number of ExitExceptions: 0 Time finished: Wed Feb 05 12:56:29 GMT 2020 Total time taken: 1 second
-Xmx1540M is the maximum value that allows Java to start up on my machine.
With a newer version I loaded in October, but it didn't have any obvious advantages:
$ /opt/jre1.8.0_231/bin/java -Xms1500M -Xmx1540M \ -jar ../mkgmap.trunk/mkgmap.jar \ dummy.osm dummy.osm dummy.osm Time started: Wed Feb 05 12:58:29 GMT 2020 Setting max-jobs to 4 Number of MapFailedExceptions: 0 Number of ExitExceptions: 0 Time finished: Wed Feb 05 12:58:30 GMT 2020 Total time taken: 1 second
Ticker
On Fri, 2020-01-31 at 16:15 +0000, Gerd Petermann wrote:
please try this command with openjdk 11 or 13 64Bit: java -Xms4G -Xmx10G -jar mkgmap.jar f:\osm\dummy.osm f:\osm\dummy.osm f:\osm\dummy.osm This is what I get: Time started: Fri Jan 31 17:12:53 CET 2020 Setting max-jobs to 1 Number of MapFailedExceptions: 0 To reduce the run time, consider increasing the amnount of memory available for use by mkgmap by using the Java -Xmx flag to set the memory to more than 41000 MB, providing this is less than the amount of physical memory installed. Number of ExitExceptions: 0 Time finished: Fri Jan 31 17:12:54 CET 2020 Total time taken: 1 second
mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev --
#####################################################
Viele Grüße und 73 de Manfred Haiduk, DD8KQ e-mail mhaiduk@t-online.de dd8kq@gmx.de
#####################################################
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- ##################################################### Viele Grüße und 73 de Manfred Haiduk, DD8KQ e-mail mhaiduk@t-online.de dd8kq@gmx.de ##################################################### _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Ah, now i understand .... Am 05.02.2020 um 22:33 schrieb Gerd Petermann:
Hi Manfred,
the calculation is only done when you don't use the --max-jobs option and when more than two input files are processed.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von DD8KQ <dd8kq@gmx.de> Gesendet: Mittwoch, 5. Februar 2020 21:49 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] Error in memory calculation
Hi Gerd
just did a small map job but on the console there is no print about number of max jobs. Is there a special log flag to get this print out ? What i've seen in the task manager while building the map, all 8 cpu's are used, but i don't know anything about the number of jobs ...
Am 05.02.2020 um 21:35 schrieb Gerd Petermann:
Hi Manfred,
mkgmap contains code to estimate a suitable value for max-jobs if this parameter isn't set. This code fails and the result is that it believes that a first thread needed all available memory. This doesn't happen when the -Xms parameter in my sample call is removed, and it also doesn't happen with JRE 1.8 (both from OpenJDK and Oracle). I've not yet started to look at the details returned in class java.lang.management.MemoryPoolMXBean
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von DD8KQ <dd8kq@gmx.de> Gesendet: Mittwoch, 5. Februar 2020 21:16 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] Error in memory calculation
Hi Gerd
i'm not sure what you are looking for. In my build environment i'm using java 1.8.0_241 (64 bit) and as my laptop have 32 Gb of memory availiable my command line to build is like this :
java -Xmx30000m -XX:-UseGCOverheadLimit -ea -jar -Dlog.config=logging.properties "c:\Program Files (x86)\KartenTools\mkgmap\mkgmap.jar" ....
and depending on which map i want to build, the occupied memory runs up nearly to 30Gb
Am 05.02.2020 um 14:05 schrieb Ticker Berkin:
Hi Gerd
I don't have a 64bit environment or that much memory. In my normal environment, I get:
$ java -Xms1500M -Xmx1540M -jar ../mkgmap.trunk/mkgmap.jar \ dummy.osm dummy.osm dummy.osm Time started: Wed Feb 05 12:56:28 GMT 2020 Setting max-jobs to 4 Number of MapFailedExceptions: 0 Number of ExitExceptions: 0 Time finished: Wed Feb 05 12:56:29 GMT 2020 Total time taken: 1 second
-Xmx1540M is the maximum value that allows Java to start up on my machine.
With a newer version I loaded in October, but it didn't have any obvious advantages:
$ /opt/jre1.8.0_231/bin/java -Xms1500M -Xmx1540M \ -jar ../mkgmap.trunk/mkgmap.jar \ dummy.osm dummy.osm dummy.osm Time started: Wed Feb 05 12:58:29 GMT 2020 Setting max-jobs to 4 Number of MapFailedExceptions: 0 Number of ExitExceptions: 0 Time finished: Wed Feb 05 12:58:30 GMT 2020 Total time taken: 1 second
Ticker
On Fri, 2020-01-31 at 16:15 +0000, Gerd Petermann wrote:
please try this command with openjdk 11 or 13 64Bit: java -Xms4G -Xmx10G -jar mkgmap.jar f:\osm\dummy.osm f:\osm\dummy.osm f:\osm\dummy.osm This is what I get: Time started: Fri Jan 31 17:12:53 CET 2020 Setting max-jobs to 1 Number of MapFailedExceptions: 0 To reduce the run time, consider increasing the amnount of memory available for use by mkgmap by using the Java -Xmx flag to set the memory to more than 41000 MB, providing this is less than the amount of physical memory installed. Number of ExitExceptions: 0 Time finished: Fri Jan 31 17:12:54 CET 2020 Total time taken: 1 second
mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev --
#####################################################
Viele Grüße und 73 de Manfred Haiduk, DD8KQ e-mail mhaiduk@t-online.de dd8kq@gmx.de
#####################################################
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev --
#####################################################
Viele Grüße und 73 de Manfred Haiduk, DD8KQ e-mail mhaiduk@t-online.de dd8kq@gmx.de
#####################################################
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- ##################################################### Viele Grüße und 73 de Manfred Haiduk, DD8KQ e-mail mhaiduk@t-online.de dd8kq@gmx.de #####################################################
participants (3)
-
DD8KQ -
Gerd Petermann -
Gerd Petermann