Performance of mkgmap.jar and splitter.jar
I think during the last weeks the increased feature set of mkgmap is really nice, not so nice is the performance though..... I made some tests - see below. I think with the current versions performance really has gone by the wayside. It takes 4times as long against austria.osm compared to rev 937. I will check against some other versions to find out if there was one specific patch that fucked up the performance or if it increased slowly.... Compiling austria.osm (from geofabrik) with my custom style-files (6layers) takes 788 seconds with rev. 937, but 3519 seconds with rev 988. 1. Performance Tile Splitter. Split austria.osm with already established areas.list (4 areas) takes 120sec, the same but austria.osm.bz2 took astonishing 312 seconds. The tile splitter works significantly faster on unzipped sources. 2. Performance mkgmap rev 988 .gz zipped: The bigger the tile size you try to get done in one step, the better the performance. For tryout style1 calculating the 4 tiles took 6 minutes (about 1:30 minute for tiles 1-3, and *3 minutes* for the last and biggest tile (which still works fine in Mapsource though). The same tiles trying to calculate with a bit more heavyweight (only lines though, no Polygons or POI) style2 - the first 2 tiles took a bit less than 1 minute each, the third (and only second biggest tile) took *15 minutes,* and the forth took 2:30 minutes. I don't know what causes mkgmap to fuck up here on the third tile. If someone wants to have a look at the exact configuration, I can send the configuration files (including batch to run it through). In total it took from *04:45 for the first style, but 20:21 *for the second style. 2b) the same again this time from unzipped sources: The first 3 tiles on stylefile1 got done in about 2 minutes, the fourth one took again 3 minutes. That's 04:42 compared to 04:45 zipped, so no significant difference). The first 2 tiles on stylefile2 got done in about 1:20, the third now took 17 minutes and the fourth 2 minutes.(so even worse than before; thats 20:35 unzipped compared to 20:21 zipped) The filesize of the resulting tiles with style1 is: 2.54MB 4.70MB 4.31MB 8.03MB The filesize of the resulting tiles with style 2 is: 2.78MB 5.87MB 5.90MB 8.96MB 2c) trying to run the two styles on the whole austria.osm without splitting results in: Stylfile1: 3:36:48 (so thats about a full minute quicker than the the country split into 4 tiles). Stylefile2: 16:48 (that's about 4 minutes quicker, 4 tiles took 20:35). 2d) Speed of processing Austria.osm with mkgmap rev 937 Stylefile1: 2:52 Stylefile2: 3:17 *My whole mapset: 788 seconds (compared to 3519 seconds with rev 988)* P.S. on a related note, in case the style-file is not found there should be an error message, and not a defaulting to the default mkgmap style. P.P.S: see stylefile2, it's really short an simple - nothing else (no POI or Polygons), I have no clue why it takes so long on Austria.osm respectively tile three (having run splitter jar with 25000000)..... mtb:scale:uphill=0 [0x1d resolution 22] mtb:scale:uphill=1 [0x1e resolution 22] mtb:scale:uphill=2 [0x26 resolution 22] mtb:scale:uphill=3 [0x29 resolution 22] mtb:scale:uphill=4 [0x2b resolution 22] mtb:scale:uphill=5 [0x1b resolution 22] highway=primary [0x04 road_class=0 road_speed=3 resolution 19] highway=primary_link [0x04 road_class=0 road_speed=2 resolution 19] highway=secondary [0x05 road_class=1 road_speed=3 resolution 16] highway=tertiary [0x06 road_class=1 road_speed=4 resolution 20] highway=residential [0x07 road_class=1 road_speed=3 resolution 22] highway=living_street [0x07 road_class=1 road_speed=3 resolution 22] highway=unclassified [0x07 road_class=1 road_speed=4 resolution 22] P.P.P.S - the commandline for mkgmap: "start java -jar -Xmx1184M d:\garmin\mkgmap_svn\dist\mkgmap.jar --description=02_Austria --style-file=2 --route --net --country-name=AUSTRIA --country-abbr=AUT --country-name=AUSTRIA --ignore-turn-restrictions --mapname=63260001 austria.osm"
Hi Felix,
I think during the last weeks the increased feature set of mkgmap is really nice, not so nice is the performance though..... I made some tests - see below. I think with the current versions performance really has gone by the wayside. It takes 4times as long against austria.osm compared to rev 937.
Please add the following to your java options and post the resulting java.hprof.txt file to the list or to me directly. Then we will know where the time is going. -agentlib:hprof=cpu=samples,depth=20 Cheers, Mark
Will do. Until rev 984 performance is good (o.k got a bit slower, but that's o.k.), 988 got intolerable and 991 seems to need twice as much time as 988. Im still waiting for 991 to run through and then test on with -agentlib. I will continue testing and update you on the results in a few hours. Mark Burton wrote:
Hi Felix,
I think during the last weeks the increased feature set of mkgmap is really nice, not so nice is the performance though..... I made some tests - see below. I think with the current versions performance really has gone by the wayside. It takes 4times as long against austria.osm compared to rev 937.
Please add the following to your java options and post the resulting java.hprof.txt file to the list or to me directly. Then we will know where the time is going.
-agentlib:hprof=cpu=samples,depth=20
Cheers,
Mark _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Felix,
Will do. Until rev 984 performance is good (o.k got a bit slower, but that's o.k.), 988 got intolerable and 991 seems to need twice as much time as 988. Im still waiting for 991 to run through and then test on with -agentlib. I will continue testing and update you on the results in a few hours.
988 introduced the sorted road data so it's probably the road name sorting that's impacted the performance. I will look into that when the hprof data becomes available. It's always going to have some hit because you have to sort all of the road names in a tile. The way the sort is done needs looking at to see if it can be made more efficient. I make no claims that it is optimal at the moment. One thought that occurred to me a while back was that converting labels to their encoded form may be happening too early now. i.e. it would be more efficient to keep the labels as strings and then convert to the encoded form on output to the img file. That way, any operations on the labels (like sorting) could use the standard string functions. Also, correct me if I am wrong, but I don't think we handle multibyte characters yet (do we?) so if the labels are using some funky 16 bit character set, the sorting code as it is now would not work right anyway. As to why 991 is even slower than 988, I don't have a clue at this time. I am out this afternoon and this evening but if you post the hprof results I will look at them as soon as I can. Cheers, Mark
Will do. 991 is now calculation on my stylefile2 since 43 minutes (so time to run has more than doubled). I will stop it now. Get hprof data from 988, then again use 991 and let it run while going outside for some mountainbiking. I hope that command appends data and does not start the file from scratch when running mkgmap again. In case you can't significantly speed up the changes, a parameter to turn that functions on/off is in my eyes desperately needed! Cheers, Felix Mark Burton wrote:
Hi Felix,
Will do. Until rev 984 performance is good (o.k got a bit slower, but that's o.k.), 988 got intolerable and 991 seems to need twice as much time as 988. Im still waiting for 991 to run through and then test on with -agentlib. I will continue testing and update you on the results in a few hours.
988 introduced the sorted road data so it's probably the road name sorting that's impacted the performance. I will look into that when the hprof data becomes available. It's always going to have some hit because you have to sort all of the road names in a tile. The way the sort is done needs looking at to see if it can be made more efficient. I make no claims that it is optimal at the moment.
One thought that occurred to me a while back was that converting labels to their encoded form may be happening too early now. i.e. it would be more efficient to keep the labels as strings and then convert to the encoded form on output to the img file. That way, any operations on the labels (like sorting) could use the standard string functions. Also, correct me if I am wrong, but I don't think we handle multibyte characters yet (do we?) so if the labels are using some funky 16 bit character set, the sorting code as it is now would not work right anyway.
As to why 991 is even slower than 988, I don't have a clue at this time.
I am out this afternoon and this evening but if you post the hprof results I will look at them as soon as I can.
Cheers,
Mark _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hey mark - where do I have to put that argument? I get no real output. Here is my cmd line: start /low /b /wait java -jar -agentlib:hprof=cpu=samples,depth=20 -Xmx1200M d:\garmin\mkgmap988\mkgmap.jar --description=00_Austria --style-file=ystandard --route --net --country-name=AUSTRIA --country-abbr=AUT --country-name=AUSTRIA --ignore-turn-restrictions --mapname=63260000 austria.osm Any mistakes? P.S. Sorry my previously mentioned runtimes were for Rev 991 not rev 988. However I compiled 991 myself. If I take 991 or 988 from the server then map compilation times skyrocket (Win XP SP3 for me, Pentium M 2.26Ghz, 3GB Ram, No Pagefile) Here is the content of java.hprof.txt, in the cmd window the line: dumping to jvm.hprof.txt is shown - I can't find that file though: JAVA PROFILE 1.0.1, created Sat Apr 04 13:17:45 2009 Header for -agentlib:hprof (or -Xrunhprof) ASCII Output (JDK 5.0 JVMTI based) @(#)jvm.hprof.txt 1.5 06/01/28 Copyright (c) 2006 Sun Microsystems, Inc. All Rights Reserved. WARNING! This file format is under development, and is subject to change without notice. This file contains the following types of records: THREAD START THREAD END mark the lifetime of Java threads TRACE represents a Java stack trace. Each trace consists of a series of stack frames. Other records refer to TRACEs to identify (1) where object allocations have taken place, (2) the frames in which GC roots were found, and (3) frequently executed methods. HEAP DUMP is a complete snapshot of all live objects in the Java heap. Following distinctions are made: ROOT root set as determined by GC CLS classes OBJ instances ARR arrays SITES is a sorted list of allocation sites. This identifies the most heavily allocated object types, and the TRACE at which those allocations occurred. CPU SAMPLES is a statistical profile of program execution. The VM periodically samples all running threads, and assigns a quantum to active TRACEs in those threads. Entries in this record are TRACEs ranked by the percentage of total quanta they consumed; top-ranked TRACEs are typically hot spots in the program. CPU TIME is a profile of program execution obtained by measuring the time spent in individual methods (excluding the time spent in callees), as well as by counting the number of times each method is called. Entries in this record are TRACEs ranked by the percentage of total CPU time. The "count" field indicates the number of times each TRACE is invoked. MONITOR TIME is a profile of monitor contention obtained by measuring the time spent by a thread waiting to enter a monitor. Entries in this record are TRACEs ranked by the percentage of total monitor contention time and a brief description of the monitor. The "count" field indicates the number of times the monitor was contended at that TRACE. MONITOR DUMP is a complete snapshot of all the monitors and threads in the System. HEAP DUMP, SITES, CPU SAMPLES|TIME and MONITOR DUMP|TIME records are generated at program exit. They can also be obtained during program execution by typing Ctrl-\ (on Solaris) or by typing Ctrl-Break (on Win32). -------- Mark Burton wrote:
Hi Felix,
I think during the last weeks the increased feature set of mkgmap is really nice, not so nice is the performance though..... I made some tests - see below. I think with the current versions performance really has gone by the wayside. It takes 4times as long against austria.osm compared to rev 937.
Please add the following to your java options and post the resulting java.hprof.txt file to the list or to me directly. Then we will know where the time is going.
-agentlib:hprof=cpu=samples,depth=20
Cheers,
Mark _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Felix,
Hey mark - where do I have to put that argument? I get no real output.
Here is my cmd line: start /low /b /wait java -jar -agentlib:hprof=cpu=samples,depth=20 -Xmx1200M d:\garmin\mkgmap988\mkgmap.jar --description=00_Austria --style-file=ystandard --route --net --country-name=AUSTRIA --country-abbr=AUT --country-name=AUSTRIA --ignore-turn-restrictions --mapname=63260000 austria.osm Any mistakes?
I don't use windows so I can't comment on that aspect but, otherwise, I think it looks OK. The data gets output when the program terminates. Try running on a smaller osm file and see if you get some output. Cheers, Mark
Oh I got it. Because I used that command subsequently - on the next layer it deleted the output file. Do you know how to specify that the file is appended or how to specify the output file of the dump? Mark Burton wrote:
Hi Felix,
Hey mark - where do I have to put that argument? I get no real output.
Here is my cmd line: start /low /b /wait java -jar -agentlib:hprof=cpu=samples,depth=20 -Xmx1200M d:\garmin\mkgmap988\mkgmap.jar --description=00_Austria --style-file=ystandard --route --net --country-name=AUSTRIA --country-abbr=AUT --country-name=AUSTRIA --ignore-turn-restrictions --mapname=63260000 austria.osm Any mistakes?
I don't use windows so I can't comment on that aspect but, otherwise, I think it looks OK.
The data gets output when the program terminates. Try running on a smaller osm file and see if you get some output.
Cheers,
Mark _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Felix,
Oh I got it. Because I used that command subsequently - on the next layer it deleted the output file. Do you know how to specify that the file is appended or how to specify the output file of the dump?
I think that force=n should do what you want. See http://java.sun.com/developer/technicalArticles/Programming/HPROF.html or http://publib.boulder.ibm.com/infocenter/javasdk/v5r0/index.jsp?topic=/com.i... Cheers, Mark
1. Performance Tile Splitter. Split austria.osm with already established areas.list (4 areas) takes 120sec, the same but austria.osm.bz2 took astonishing 312 seconds. The tile splitter works significantly faster on unzipped sources.
I haven't stopped by stopwatch, but on my system by feeling it is roughly the same time at b2zipped and unzipped files. I don't know any significant slowdown on my system. But it is fact, that the bzip2 algorithm needs a lot of cpu resources, you can't change it. Do you save time, if you unzip the bz2 data before running splitter and add that needed time to the splitter running time?
Yes, unzipping with 7-zip takes 57 seconds for austria.osm.bz2 Johann Gail wrote:
1. Performance Tile Splitter. Split austria.osm with already established areas.list (4 areas) takes 120sec, the same but austria.osm.bz2 took astonishing 312 seconds. The tile splitter works significantly faster on unzipped sources.
I haven't stopped by stopwatch, but on my system by feeling it is roughly the same time at b2zipped and unzipped files. I don't know any significant slowdown on my system. But it is fact, that the bzip2 algorithm needs a lot of cpu resources, you can't change it. Do you save time, if you unzip the bz2 data before running splitter and add that needed time to the splitter running time?
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Felix, Can you please try this little patch and see if helps the performance when generating the sorted road data. Cheers, Mark ------------------ diff --git a/src/uk/me/parabola/imgfmt/app/trergn/Polyline.java b/src/uk/me/parabola/imgfmt/app/trergn/Polyline.java index cc5204b..d71f1f9 100644 --- a/src/uk/me/parabola/imgfmt/app/trergn/Polyline.java +++ b/src/uk/me/parabola/imgfmt/app/trergn/Polyline.java @@ -183,9 +183,10 @@ public class Polyline extends MapObject { if(s0.equals(e1) || e0.equals(s1) || s0.equals(s1) || e0.equals(e1)) return true; for(Coord p1 : points) - for(Coord p2 : other.points) - if(p1.equals(p2)) - return true; + if(p1.getId() != 0) + for(Coord p2 : other.points) + if(p2.getId() != 0 && p1.equals(p2)) + return true; return false; }
Felix, I have now committed an improved version of this fix + the no-sorted-road option to trunk. Please test and see if the efficiency has improved. Cheers, Mark
Thanks for the fix. Working great. Efficiency of rev 993 is not as good as rev 984 (my mapset took 845 seconds) but with 939 seconds pretty good again. Mark Burton wrote:
Felix,
I have now committed an improved version of this fix + the no-sorted-road option to trunk. Please test and see if the efficiency has improved.
Cheers,
Mark
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
participants (3)
-
Felix Hartmann -
Johann Gail -
Mark Burton