Slowness processing specific tile
I noticed a big hang/delay processing one tile in NC. Here's the generated IMG files for each tile: 05/19/2011 09:33 AM 4,230,144 63240001.img 05/19/2011 09:33 AM 2,450,432 63240002.img 05/19/2011 09:33 AM 2,893,824 63240003.img 05/19/2011 09:33 AM 2,170,880 63240004.img 05/19/2011 09:33 AM 2,173,952 63240005.img 05/19/2011 09:34 AM 2,518,016 63240006.img 05/19/2011 09:34 AM 2,553,856 63240007.img 05/19/2011 09:34 AM 1,730,560 63240008.img 05/19/2011 09:34 AM 2,041,856 63240009.img 05/19/2011 09:34 AM 3,491,328 63240010.img 05/19/2011 09:34 AM 6,999,552 63240011.img 05/19/2011 09:35 AM 4,713,984 63240012.img 05/19/2011 09:35 AM 2,435,072 63240013.img 05/19/2011 09:35 AM 3,771,392 63240014.img 05/19/2011 09:35 AM 6,385,152 63240015.img 05/19/2011 09:35 AM 3,237,376 63240016.img 05/19/2011 09:35 AM 3,420,160 63240017.img 05/19/2011 09:35 AM 2,976,768 63240018.img 05/19/2011 09:35 AM 3,629,056 63240019.img 05/19/2011 09:36 AM 3,863,552 63240020.img 05/19/2011 09:36 AM 3,583,488 63240021.img 05/19/2011 09:36 AM 1,553,408 63240022.img 05/19/2011 09:36 AM 3,411,456 63240023.img 05/19/2011 09:36 AM 3,038,208 63240024.img 05/19/2011 09:36 AM 7,901,696 63240025.img 05/19/2011 09:36 AM 3,146,752 63240026.img 05/19/2011 09:36 AM 4,070,400 63240027.img 05/19/2011 09:37 AM 2,481,664 63240028.img 05/19/2011 09:37 AM 3,462,656 63240029.img 05/19/2011 09:37 AM 4,033,536 63240030.img 05/19/2011 09:47 AM 8,536,576 63240031.img 05/19/2011 09:47 AM 1,745,408 63240032.img 05/19/2011 09:47 AM 4,779,008 63240033.img 05/19/2011 09:47 AM 3,905,024 63240034.img 05/19/2011 09:47 AM 5,238,272 63240035.img 05/19/2011 09:47 AM 3,605,504 63240036.img You can see that tile 31 took about 10 minutes to be generated while the rest was much quicker. Anyone seen anything similar? Here's the tile boundaries from the splitter output: 63240031: 1576960,-3932160 to 1597440,-3891200 # : 33.837891,-84.375000 to 34.277344,-83.496094 Just curious. Not a big deal but I thought I had caused it somehow with the PBF patch. Francisco
On 19.05.2011 20:20, Francisco Moraes wrote:
I noticed a big hang/delay processing one tile in NC. Here's the generated IMG files for each tile:
05/19/2011 09:33 AM 4,230,144 63240001.img 05/19/2011 09:33 AM 2,450,432 63240002.img 05/19/2011 09:33 AM 2,893,824 63240003.img 05/19/2011 09:33 AM 2,170,880 63240004.img 05/19/2011 09:33 AM 2,173,952 63240005.img 05/19/2011 09:34 AM 2,518,016 63240006.img 05/19/2011 09:34 AM 2,553,856 63240007.img 05/19/2011 09:34 AM 1,730,560 63240008.img 05/19/2011 09:34 AM 2,041,856 63240009.img 05/19/2011 09:34 AM 3,491,328 63240010.img 05/19/2011 09:34 AM 6,999,552 63240011.img 05/19/2011 09:35 AM 4,713,984 63240012.img 05/19/2011 09:35 AM 2,435,072 63240013.img 05/19/2011 09:35 AM 3,771,392 63240014.img 05/19/2011 09:35 AM 6,385,152 63240015.img 05/19/2011 09:35 AM 3,237,376 63240016.img 05/19/2011 09:35 AM 3,420,160 63240017.img 05/19/2011 09:35 AM 2,976,768 63240018.img 05/19/2011 09:35 AM 3,629,056 63240019.img 05/19/2011 09:36 AM 3,863,552 63240020.img 05/19/2011 09:36 AM 3,583,488 63240021.img 05/19/2011 09:36 AM 1,553,408 63240022.img 05/19/2011 09:36 AM 3,411,456 63240023.img 05/19/2011 09:36 AM 3,038,208 63240024.img 05/19/2011 09:36 AM 7,901,696 63240025.img 05/19/2011 09:36 AM 3,146,752 63240026.img 05/19/2011 09:36 AM 4,070,400 63240027.img 05/19/2011 09:37 AM 2,481,664 63240028.img 05/19/2011 09:37 AM 3,462,656 63240029.img 05/19/2011 09:37 AM 4,033,536 63240030.img 05/19/2011 09:47 AM 8,536,576 63240031.img 05/19/2011 09:47 AM 1,745,408 63240032.img 05/19/2011 09:47 AM 4,779,008 63240033.img 05/19/2011 09:47 AM 3,905,024 63240034.img 05/19/2011 09:47 AM 5,238,272 63240035.img 05/19/2011 09:47 AM 3,605,504 63240036.img
You can see that tile 31 took about 10 minutes to be generated while the rest was much quicker. Anyone seen anything similar? Here's the tile boundaries from the splitter output:
63240031: 1576960,-3932160 to 1597440,-3891200 # : 33.837891,-84.375000 to 34.277344,-83.496094
Just curious. Not a big deal but I thought I had caused it somehow with the PBF patch. Noticed the same thing on Turkey (11min for one tile). I have to recheck if on Iceland and Serbia the same problem appears - and mkgmap is not completly choking. This happened also before the pbf patches/ or without locator branch, but now more often?? Francisco _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
I noticed a big hang/delay processing one tile in NC. Here's the generated IMG files for each tile:
05/19/2011 09:33 AM 4,230,144 63240001.img 05/19/2011 09:33 AM 2,450,432 63240002.img 05/19/2011 09:33 AM 2,893,824 63240003.img 05/19/2011 09:33 AM 2,170,880 63240004.img 05/19/2011 09:33 AM 2,173,952 63240005.img 05/19/2011 09:34 AM 2,518,016 63240006.img 05/19/2011 09:34 AM 2,553,856 63240007.img 05/19/2011 09:34 AM 1,730,560 63240008.img 05/19/2011 09:34 AM 2,041,856 63240009.img 05/19/2011 09:34 AM 3,491,328 63240010.img 05/19/2011 09:34 AM 6,999,552 63240011.img 05/19/2011 09:35 AM 4,713,984 63240012.img 05/19/2011 09:35 AM 2,435,072 63240013.img 05/19/2011 09:35 AM 3,771,392 63240014.img 05/19/2011 09:35 AM 6,385,152 63240015.img 05/19/2011 09:35 AM 3,237,376 63240016.img 05/19/2011 09:35 AM 3,420,160 63240017.img 05/19/2011 09:35 AM 2,976,768 63240018.img 05/19/2011 09:35 AM 3,629,056 63240019.img 05/19/2011 09:36 AM 3,863,552 63240020.img 05/19/2011 09:36 AM 3,583,488 63240021.img 05/19/2011 09:36 AM 1,553,408 63240022.img 05/19/2011 09:36 AM 3,411,456 63240023.img 05/19/2011 09:36 AM 3,038,208 63240024.img 05/19/2011 09:36 AM 7,901,696 63240025.img 05/19/2011 09:36 AM 3,146,752 63240026.img 05/19/2011 09:36 AM 4,070,400 63240027.img 05/19/2011 09:37 AM 2,481,664 63240028.img 05/19/2011 09:37 AM 3,462,656 63240029.img 05/19/2011 09:37 AM 4,033,536 63240030.img 05/19/2011 09:47 AM 8,536,576 63240031.img 05/19/2011 09:47 AM 1,745,408 63240032.img 05/19/2011 09:47 AM 4,779,008 63240033.img 05/19/2011 09:47 AM 3,905,024 63240034.img 05/19/2011 09:47 AM 5,238,272 63240035.img 05/19/2011 09:47 AM 3,605,504 63240036.img
You can see that tile 31 took about 10 minutes to be generated while the rest was much quicker. Anyone seen anything similar? Here's the tile boundaries from the splitter output:
63240031: 1576960,-3932160 to 1597440,-3891200 # : 33.837891,-84.375000 to 34.277344,-83.496094
Just curious. Not a big deal but I thought I had caused it somehow with the PBF patch. Noticed the same thing on Turkey (11min for one tile). I have to recheck if on Iceland and Serbia the same problem appears - and mkgmap is not completly choking. This happened also before the pbf patches/ or without locator branch, but now more often?? Francisco
Don't know if that's the reason for your case but you might check. If Java has too few memory the garbage collector needs so much time that it seems as if mkgmap blocks. I have seen such cases. You can check that either by connecting with jvisualvm and watch the CPU time used by the garbage collector or you can add -verbose:gc as java parameter: java -verbose:gc -jar mkgmap.jar ... In this case you see a log message each time the gc is running. In case mkgmap seems to block and low memory is the reason you should get tons of such messages. By the way: I suppose the mkgmap code still has some potential to reduce the memory footprint. All nodes, ways and relations are stored in HashMaps which might be changed to a less memory consuming implementation. It would be great if anyone tries to patch that. WanMil
On 19.05.2011 21:38, WanMil wrote:
I noticed a big hang/delay processing one tile in NC. Here's the generated IMG files for each tile:
05/19/2011 09:33 AM 4,230,144 63240001.img 05/19/2011 09:33 AM 2,450,432 63240002.img 05/19/2011 09:33 AM 2,893,824 63240003.img 05/19/2011 09:33 AM 2,170,880 63240004.img 05/19/2011 09:33 AM 2,173,952 63240005.img 05/19/2011 09:34 AM 2,518,016 63240006.img 05/19/2011 09:34 AM 2,553,856 63240007.img 05/19/2011 09:34 AM 1,730,560 63240008.img 05/19/2011 09:34 AM 2,041,856 63240009.img 05/19/2011 09:34 AM 3,491,328 63240010.img 05/19/2011 09:34 AM 6,999,552 63240011.img 05/19/2011 09:35 AM 4,713,984 63240012.img 05/19/2011 09:35 AM 2,435,072 63240013.img 05/19/2011 09:35 AM 3,771,392 63240014.img 05/19/2011 09:35 AM 6,385,152 63240015.img 05/19/2011 09:35 AM 3,237,376 63240016.img 05/19/2011 09:35 AM 3,420,160 63240017.img 05/19/2011 09:35 AM 2,976,768 63240018.img 05/19/2011 09:35 AM 3,629,056 63240019.img 05/19/2011 09:36 AM 3,863,552 63240020.img 05/19/2011 09:36 AM 3,583,488 63240021.img 05/19/2011 09:36 AM 1,553,408 63240022.img 05/19/2011 09:36 AM 3,411,456 63240023.img 05/19/2011 09:36 AM 3,038,208 63240024.img 05/19/2011 09:36 AM 7,901,696 63240025.img 05/19/2011 09:36 AM 3,146,752 63240026.img 05/19/2011 09:36 AM 4,070,400 63240027.img 05/19/2011 09:37 AM 2,481,664 63240028.img 05/19/2011 09:37 AM 3,462,656 63240029.img 05/19/2011 09:37 AM 4,033,536 63240030.img 05/19/2011 09:47 AM 8,536,576 63240031.img 05/19/2011 09:47 AM 1,745,408 63240032.img 05/19/2011 09:47 AM 4,779,008 63240033.img 05/19/2011 09:47 AM 3,905,024 63240034.img 05/19/2011 09:47 AM 5,238,272 63240035.img 05/19/2011 09:47 AM 3,605,504 63240036.img
You can see that tile 31 took about 10 minutes to be generated while the rest was much quicker. Anyone seen anything similar? Here's the tile boundaries from the splitter output:
63240031: 1576960,-3932160 to 1597440,-3891200 # : 33.837891,-84.375000 to 34.277344,-83.496094
Just curious. Not a big deal but I thought I had caused it somehow with the PBF patch. Noticed the same thing on Turkey (11min for one tile). I have to recheck if on Iceland and Serbia the same problem appears - and mkgmap is not completly choking. This happened also before the pbf patches/ or without locator branch, but now more often?? Francisco
Don't know if that's the reason for your case but you might check. If Java has too few memory the garbage collector needs so much time that it seems as if mkgmap blocks. I have seen such cases.
You can check that either by connecting with jvisualvm and watch the CPU time used by the garbage collector or you can add -verbose:gc as java parameter: java -verbose:gc -jar mkgmap.jar ... In this case you see a log message each time the gc is running. In case mkgmap seems to block and low memory is the reason you should get tons of such messages.
By the way: I suppose the mkgmap code still has some potential to reduce the memory footprint. All nodes, ways and relations are stored in HashMaps which might be changed to a less memory consuming implementation. It would be great if anyone tries to patch that.
WanMil
In my case with Turkey that is definitely not the problem. It happens on small maps with 8GB RAM attributed (the server got 12GB, so RAM shortage is highly unlikely). I'll try jvisualvm on Saturday or Sunday.
I noticed a big hang/delay processing one tile in NC. Here's the generated IMG files for each tile:
05/19/2011 09:33 AM 4,230,144 63240001.img 05/19/2011 09:33 AM 2,450,432 63240002.img 05/19/2011 09:33 AM 2,893,824 63240003.img 05/19/2011 09:33 AM 2,170,880 63240004.img 05/19/2011 09:33 AM 2,173,952 63240005.img 05/19/2011 09:34 AM 2,518,016 63240006.img 05/19/2011 09:34 AM 2,553,856 63240007.img 05/19/2011 09:34 AM 1,730,560 63240008.img 05/19/2011 09:34 AM 2,041,856 63240009.img 05/19/2011 09:34 AM 3,491,328 63240010.img 05/19/2011 09:34 AM 6,999,552 63240011.img 05/19/2011 09:35 AM 4,713,984 63240012.img 05/19/2011 09:35 AM 2,435,072 63240013.img 05/19/2011 09:35 AM 3,771,392 63240014.img 05/19/2011 09:35 AM 6,385,152 63240015.img 05/19/2011 09:35 AM 3,237,376 63240016.img 05/19/2011 09:35 AM 3,420,160 63240017.img 05/19/2011 09:35 AM 2,976,768 63240018.img 05/19/2011 09:35 AM 3,629,056 63240019.img 05/19/2011 09:36 AM 3,863,552 63240020.img 05/19/2011 09:36 AM 3,583,488 63240021.img 05/19/2011 09:36 AM 1,553,408 63240022.img 05/19/2011 09:36 AM 3,411,456 63240023.img 05/19/2011 09:36 AM 3,038,208 63240024.img 05/19/2011 09:36 AM 7,901,696 63240025.img 05/19/2011 09:36 AM 3,146,752 63240026.img 05/19/2011 09:36 AM 4,070,400 63240027.img 05/19/2011 09:37 AM 2,481,664 63240028.img 05/19/2011 09:37 AM 3,462,656 63240029.img 05/19/2011 09:37 AM 4,033,536 63240030.img 05/19/2011 09:47 AM 8,536,576 63240031.img 05/19/2011 09:47 AM 1,745,408 63240032.img 05/19/2011 09:47 AM 4,779,008 63240033.img 05/19/2011 09:47 AM 3,905,024 63240034.img 05/19/2011 09:47 AM 5,238,272 63240035.img 05/19/2011 09:47 AM 3,605,504 63240036.img
You can see that tile 31 took about 10 minutes to be generated while the rest was much quicker. Anyone seen anything similar? Here's the tile boundaries from the splitter output:
63240031: 1576960,-3932160 to 1597440,-3891200 # : 33.837891,-84.375000 to 34.277344,-83.496094
Just curious. Not a big deal but I thought I had caused it somehow with the PBF patch. Noticed the same thing on Turkey (11min for one tile). I have to recheck if on Iceland and Serbia the same problem appears - and mkgmap is not completly choking. This happened also before the pbf patches/ or without locator branch, but now more often?? Francisco
Don't know if that's the reason for your case but you might check. If Java has too few memory the garbage collector needs so much time that it seems as if mkgmap blocks. I have seen such cases.
You can check that either by connecting with jvisualvm and watch the CPU time used by the garbage collector or you can add -verbose:gc as java parameter: java -verbose:gc -jar mkgmap.jar ... In this case you see a log message each time the gc is running. In case mkgmap seems to block and low memory is the reason you should get tons of such messages.
By the way: I suppose the mkgmap code still has some potential to reduce the memory footprint. All nodes, ways and relations are stored in HashMaps which might be changed to a less memory consuming implementation. It would be great if anyone tries to patch that.
WanMil
In my case with Turkey that is definitely not the problem. It happens on small maps with 8GB RAM attributed (the server got 12GB, so RAM shortage is highly unlikely). I'll try jvisualvm on Saturday or Sunday.
The tile that took so long is the biggest one of all. Important for memory considerations is the size of tiles and the number of threads. So 8GB *might* also be too low if you use big tiles and a high number of threads.
On 19.05.2011 21:52, WanMil wrote:
I noticed a big hang/delay processing one tile in NC. Here's the generated IMG files for each tile:
05/19/2011 09:33 AM 4,230,144 63240001.img 05/19/2011 09:33 AM 2,450,432 63240002.img 05/19/2011 09:33 AM 2,893,824 63240003.img 05/19/2011 09:33 AM 2,170,880 63240004.img 05/19/2011 09:33 AM 2,173,952 63240005.img 05/19/2011 09:34 AM 2,518,016 63240006.img 05/19/2011 09:34 AM 2,553,856 63240007.img 05/19/2011 09:34 AM 1,730,560 63240008.img 05/19/2011 09:34 AM 2,041,856 63240009.img 05/19/2011 09:34 AM 3,491,328 63240010.img 05/19/2011 09:34 AM 6,999,552 63240011.img 05/19/2011 09:35 AM 4,713,984 63240012.img 05/19/2011 09:35 AM 2,435,072 63240013.img 05/19/2011 09:35 AM 3,771,392 63240014.img 05/19/2011 09:35 AM 6,385,152 63240015.img 05/19/2011 09:35 AM 3,237,376 63240016.img 05/19/2011 09:35 AM 3,420,160 63240017.img 05/19/2011 09:35 AM 2,976,768 63240018.img 05/19/2011 09:35 AM 3,629,056 63240019.img 05/19/2011 09:36 AM 3,863,552 63240020.img 05/19/2011 09:36 AM 3,583,488 63240021.img 05/19/2011 09:36 AM 1,553,408 63240022.img 05/19/2011 09:36 AM 3,411,456 63240023.img 05/19/2011 09:36 AM 3,038,208 63240024.img 05/19/2011 09:36 AM 7,901,696 63240025.img 05/19/2011 09:36 AM 3,146,752 63240026.img 05/19/2011 09:36 AM 4,070,400 63240027.img 05/19/2011 09:37 AM 2,481,664 63240028.img 05/19/2011 09:37 AM 3,462,656 63240029.img 05/19/2011 09:37 AM 4,033,536 63240030.img 05/19/2011 09:47 AM 8,536,576 63240031.img 05/19/2011 09:47 AM 1,745,408 63240032.img 05/19/2011 09:47 AM 4,779,008 63240033.img 05/19/2011 09:47 AM 3,905,024 63240034.img 05/19/2011 09:47 AM 5,238,272 63240035.img 05/19/2011 09:47 AM 3,605,504 63240036.img
You can see that tile 31 took about 10 minutes to be generated while the rest was much quicker. Anyone seen anything similar? Here's the tile boundaries from the splitter output:
63240031: 1576960,-3932160 to 1597440,-3891200 # : 33.837891,-84.375000 to 34.277344,-83.496094
Just curious. Not a big deal but I thought I had caused it somehow with the PBF patch. Noticed the same thing on Turkey (11min for one tile). I have to recheck if on Iceland and Serbia the same problem appears - and mkgmap is not completly choking. This happened also before the pbf patches/ or without locator branch, but now more often?? Francisco
Don't know if that's the reason for your case but you might check. If Java has too few memory the garbage collector needs so much time that it seems as if mkgmap blocks. I have seen such cases.
You can check that either by connecting with jvisualvm and watch the CPU time used by the garbage collector or you can add -verbose:gc as java parameter: java -verbose:gc -jar mkgmap.jar ... In this case you see a log message each time the gc is running. In case mkgmap seems to block and low memory is the reason you should get tons of such messages.
By the way: I suppose the mkgmap code still has some potential to reduce the memory footprint. All nodes, ways and relations are stored in HashMaps which might be changed to a less memory consuming implementation. It would be great if anyone tries to patch that.
WanMil
In my case with Turkey that is definitely not the problem. It happens on small maps with 8GB RAM attributed (the server got 12GB, so RAM shortage is highly unlikely). I'll try jvisualvm on Saturday or Sunday.
The tile that took so long is the biggest one of all. Important for memory considerations is the size of tiles and the number of threads. So 8GB *might* also be too low if you use big tiles and a high number of threads. Well I use small tiles (max-nodes 900.000 on Turkey) plus only 4 threads. (I could use 8 due to Hyperthreading, but kept it at 4). I even have this problem on single tiles where the osm.gz is like 10MB. There shouldn't be a slow down due to memory in that case, should there? (I cannot check for memory use, as my user rights are too low).
Le 19/05/2011 22:05, Felix Hartmann a écrit :
On 19.05.2011 21:52, WanMil wrote:
I noticed a big hang/delay processing one tile in NC. Here's the generated IMG files for each tile:
Do not know whether or not that the same pb, but two or three months ago, when I run splitter and mkgmap to create a gmapsupp.img file, against france.osm data, it took 4 hours to create it. On the same machine, with the same scripts, it takes now 17 hours. My main pb is with splitter as for example, it began to run yesterday at 11pm and end its run today at 11.56am. 13 hours to split, but the machine is really not a big one :-) Francois --
On 19.05.2011 22:46, frmas wrote:
Le 19/05/2011 22:05, Felix Hartmann a écrit :
On 19.05.2011 21:52, WanMil wrote:
> I noticed a big hang/delay processing one tile in NC. Here's the > generated IMG files for each tile:
Do not know whether or not that the same pb, but two or three months ago, when I run splitter and mkgmap to create a gmapsupp.img file, against france.osm data, it took 4 hours to create it. On the same machine, with the same scripts, it takes now 17 hours. My main pb is with splitter as for example, it began to run yesterday at 11pm and end its run today at 11.56am. 13 hours to split, but the machine is really not a big one :-) Francois
Well in that case your machine is the culprit. France splits in around 16 minutes for me. And takes bout 60minutes to compile (12GB Ram, Intel Core somthing (4 cores).
Le 19/05/2011 22:51, Felix Hartmann a écrit :
Do not know whether or not that the same pb, but two or three months ago, when I run splitter and mkgmap to create a gmapsupp.img file, against france.osm data, it took 4 hours to create it. On the same machine, with the same scripts, it takes now 17 hours. My main pb is with splitter as for example, it began to run yesterday at 11pm and end its run today at 11.56am. 13 hours to split, but the machine is really not a big one :-) Francois
Well in that case your machine is the culprit. France splits in around 16 minutes for me. And takes bout 60minutes to compile (12GB Ram, Intel Core somthing (4 cores).
Maybe, maybe, but tonight, using a backup of a splitter.jar file I compiled on february 19th, it took one hour to split France. The latest splitter code I compiled yesterday, took 17 hours to split France. Same data (osm data), same machine. ls splitter.jar* -rw-r--r-- 1 xx xxxx 286292 2011-02-19 22:59 splitter.jar.old -rw-r--r-- 1 xx xxxx 272603 2011-05-19 23:02 splitter.jar Francois --
* also be too low if you use big tiles and a high number
of threads. Well I use small tiles (max-nodes 900.000 on Turkey) plus only 4 threads. (I could use 8 due to Hyperthreading, but kept it at 4). I even have this problem on single tiles where the osm.gz is like 10MB. There shouldn't be a slow down due to memory in that case, should there? (I cannot check for memory use, as my user rights are too low).
What happens if you only use one thread?
Okay, some more tries to find out why mkgmap locks up on iceland (but not on most countries, so the thing is still really strange): I'll upload all needed files to recreate the problem in another mail, cause that will be easier than e-mailing forth/back to find out why the problem happens. First using my normal commandline: start /low /b /wait java -ea -jar -Xmx6500M c:\openmtbmap\mkgmap.jar %style-file% --max-jobs=4 %generate-sea% --reduce-point-density=4 --nsis --index --transparent --adjust-turn-headings --add-pois-to-areas --ignore-maxspeeds --x-reduce-point-density-polygon=8 --link-pois-to-ways --ignore-turn-restrictions --min-size-polygon=15 --remove-short-arcs=4 --description=openmtbmap_%abr% --merge-lines --location-autofill=1 --route --country-abbr=%abr% --country-name=%country% %profile% --mapname=%FID%0000 --family-id=%FID% --product-id=1 --series-name=openmtbmap_%country%_%date% --family-name=mtbmap_%abr%_%date% --tdbfile --overview-mapname=mapset --keep-going --area-name="%country%_%date%_openmtbmap.org" -c c:\openmtbmap\maps\template.%country% Country is Iceland. I hope the error message on 2. can provide the needed information why the locator branch gets stuck indefinitely?? Note, I'm not using pbf as input files, but standard osm.gz, therefore I find it strange that including pbf support without any other changes, creates an Fatal Error, plus adds about 18% of time surplus to render the map. I clearly have a problem with a lib, but have no clue how to solve this. Therefore please, include the libs into mkgmap svn, everything else will cause faults. I clearly have the libs that mkgmap wants, but probably got a wrong version that mkgmap does not expect at some point. However also the mkgmap_locator_r1492.jar downloaded from mkgmap.org.uk gets stuck for me indefinitely, so there definitely is a problem. Here is the output on building mkgmap.jar, it looks fine to me: d:\Garmin\mkgmap_svn_trunk>start /b /wait ant dist Buildfile: build.xml prepare: [mkdir] Created dir: d:\Garmin\mkgmap_svn_trunk\build\classes compile: [javac] Compiling 378 source files to d:\Garmin\mkgmap_svn_trunk\build\classes [javac] Note: Some input files use unchecked or unsafe operations. [javac] Note: Recompile with -Xlint:unchecked for details. compile-pbf: [echo] Protobuf binary format support [javac] Compiling 3 source files to d:\Garmin\mkgmap_svn_trunk\build\classes build: [copy] Copying 422 files to d:\Garmin\mkgmap_svn_trunk\build\classes dist: [jar] Building jar: d:\Garmin\mkgmap_svn_trunk\dist\mkgmap.jar *1. mkgmap_trunk without pbf support (including my patches):* 44 seconds No error message. *2. mkgmap_trunk with pbf support (including my patches)*: 50 seconds plus error message (but map created): [Fatal Error] :66:2: The content of elements must consist of well-formed character data or markup. org.xml.sax.SAXParseException: The content of elements must consist of well-formed character data or markup. at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(Unknown Source) at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(Unknown Source) at javax.xml.parsers.DocumentBuilder.parse(Unknown Source) at uk.me.parabola.mkgmap.build.LocatorConfig.loadConfig(LocatorConfig.java:70) at uk.me.parabola.mkgmap.build.LocatorConfig.<init>(LocatorConfig.java:47) at uk.me.parabola.mkgmap.build.Locator.<init>(Locator.java:68) at uk.me.parabola.mkgmap.build.MapBuilder.<init>(MapBuilder.java:103) at uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:97) at uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:65) at uk.me.parabola.mkgmap.main.Main$1.call(Main.java:224) at uk.me.parabola.mkgmap.main.Main$1.call(Main.java:221) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) [Fatal Error] :66:2: The content of elements must consist of well-formed character data or markup. org.xml.sax.SAXParseException: The content of elements must consist of well-formed character data or markup. at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(Unknown Source) at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(Unknown Source) at javax.xml.parsers.DocumentBuilder.parse(Unknown Source) at uk.me.parabola.mkgmap.build.LocatorConfig.loadConfig(LocatorConfig.java:70) at uk.me.parabola.mkgmap.build.LocatorConfig.<init>(LocatorConfig.java:47) at uk.me.parabola.mkgmap.build.Locator.<init>(Locator.java:68) at uk.me.parabola.mkgmap.build.MapBuilder.<init>(MapBuilder.java:103) at uk.me.parabola.mkgmap.combiners.TdbBuilder.writeOverviewMap(TdbBuilder.java:246) at uk.me.parabola.mkgmap.combiners.TdbBuilder.onFinish(TdbBuilder.java:239) at uk.me.parabola.mkgmap.main.Main.endOptions(Main.java:419) at uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.java:126) at uk.me.parabola.mkgmap.main.Main.main(Main.java:129) 2a) as above but using mkgmap.jar trunk rev 1949 as downloaded from http://www.mkgmap.org.uk/snapshots/mkgmap-r1949.zip No errors, but also I don't know if pbf support is icluded here?? *3. mkgmap_locator with my patches including pbf support (without using bound option with newest bound files as published here on the list):* No map created but 0bit file only, and mkgmap stuck for 20minutes. SCHWERWIEGEND (LocationHook): c:\openmtbmap\maps\63990000.osm.gz: Element lists created after 627 ms SCHWERWIEGEND (LocationHook): c:\openmtbmap\maps\63990000.osm.gz: Quadtree created after 3194 ms SCHWERWIEGEND (LocationHook): c:\openmtbmap\maps\63990000.osm.gz: Location hook finished in 3823 ms *4. mkgmap_locator as above, but adding --createboundsfile=c:\openmtbmap\maps\boundaries" to the input options:* No map created but 0bit file only, and mkgmap stuck for 20minutes. SCHWERWIEGEND (LocationHook): c:\openmtbmap\maps\63990000.osm.gz: Element lists created after 657 ms SCHWERWIEGEND (LocationHook): c:\openmtbmap\maps\63990000.osm.gz: Quadtree created after 3021 ms SCHWERWIEGEND (LocationHook): c:\openmtbmap\maps\63990000.osm.gz: Location hook finished in 3569 ms *5. Same as 3. But using --max-jobs=1:* No map created, but 0bit file only, mkgmap stuck for 20 minutes. 21:55:40 iceland is 6399 this is run6 start compilation 21:55:52 iceland SCHWERWIEGEND (LocationHook): c:\openmtbmap\maps\63990000.osm.gz: Element lists created after 350 ms SCHWERWIEGEND (LocationHook): c:\openmtbmap\maps\63990000.osm.gz: Quadtree created after 1660 ms SCHWERWIEGEND (LocationHook): c:\openmtbmap\maps\63990000.osm.gz: Location hook finished in 2277 ms *6. Same as 3. but using mkgmap_locator_r1952 downloaded from mkgmap.co.uk.:* mkgmap stuck for 20 minutes .... no map created, only 0bit file. 22:11:48 iceland is 6399 this is run6 start compilation 22:12:00 iceland SCHWERWIEGEND (LocationHook): c:\openmtbmap\maps\63990000.osm.gz: Element lists created after 232 ms SCHWERWIEGEND (LocationHook): c:\openmtbmap\maps\63990000.osm.gz: Quadtree created after 2587 ms SCHWERWIEGEND (LocationHook): c:\openmtbmap\maps\63990000.osm.gz: Location hook finished in 3158 ms On 20.05.2011 16:00, Nakor Osm wrote:
* also be too low if you use big tiles and a high number
> of threads. Well I use small tiles (max-nodes 900.000 on Turkey) plus only 4 threads. (I could use 8 due to Hyperthreading, but kept it at 4). I even have this problem on single tiles where the osm.gz is like 10MB. There shouldn't be a slow down due to memory in that case, should there? (I cannot check for memory use, as my user rights are too low).
What happens if you only use one thread?
Okay, some more tries to find out why mkgmap locks up on iceland (but not on most countries, so the thing is still really strange):
I'll upload all needed files to recreate the problem in another mail, cause that will be easier than e-mailing forth/back to find out why the problem happens.
First using my normal commandline: start /low /b /wait java -ea -jar -Xmx6500M c:\openmtbmap\mkgmap.jar %style-file% --max-jobs=4 %generate-sea% --reduce-point-density=4 --nsis --index --transparent --adjust-turn-headings --add-pois-to-areas --ignore-maxspeeds --x-reduce-point-density-polygon=8 --link-pois-to-ways --ignore-turn-restrictions --min-size-polygon=15 --remove-short-arcs=4 --description=openmtbmap_%abr% --merge-lines --location-autofill=1 --route --country-abbr=%abr% --country-name=%country% %profile% --mapname=%FID%0000 --family-id=%FID% --product-id=1 --series-name=openmtbmap_%country%_%date% --family-name=mtbmap_%abr%_%date% --tdbfile --overview-mapname=mapset --keep-going --area-name="%country%_%date%_openmtbmap.org" -c c:\openmtbmap\maps\template.%country%
Country is Iceland. I hope the error message on 2. can provide the needed information why the locator branch gets stuck indefinitely?? Note, I'm not using pbf as input files, but standard osm.gz, therefore I find it strange that including pbf support without any other changes, creates an Fatal Error, plus adds about 18% of time surplus to render the map.
I clearly have a problem with a lib, but have no clue how to solve this. Therefore please, include the libs into mkgmap svn, everything else will cause faults. I clearly have the libs that mkgmap wants, but probably got a wrong version that mkgmap does not expect at some point. However also the mkgmap_locator_r1492.jar downloaded from mkgmap.org.uk gets stuck for me indefinitely, so there definitely is a problem.
Here is the output on building mkgmap.jar, it looks fine to me: d:\Garmin\mkgmap_svn_trunk>start /b /wait ant dist Buildfile: build.xml
prepare: [mkdir] Created dir: d:\Garmin\mkgmap_svn_trunk\build\classes compile: [javac] Compiling 378 source files to d:\Garmin\mkgmap_svn_trunk\build\classes [javac] Note: Some input files use unchecked or unsafe operations. [javac] Note: Recompile with -Xlint:unchecked for details. compile-pbf: [echo] Protobuf binary format support [javac] Compiling 3 source files to d:\Garmin\mkgmap_svn_trunk\build\classes build: [copy] Copying 422 files to d:\Garmin\mkgmap_svn_trunk\build\classes dist: [jar] Building jar: d:\Garmin\mkgmap_svn_trunk\dist\mkgmap.jar
*1. mkgmap_trunk without pbf support (including my patches):* 44 seconds No error message.
*2. mkgmap_trunk with pbf support (including my patches)*: 50 seconds plus error message (but map created): [Fatal Error] :66:2: The content of elements must consist of well-formed character data or markup. org.xml.sax.SAXParseException: The content of elements must consist of well-formed character data or markup. at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(Unknown Source) at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(Unknown Source) at javax.xml.parsers.DocumentBuilder.parse(Unknown Source) at uk.me.parabola.mkgmap.build.LocatorConfig.loadConfig(LocatorConfig.java:70) at uk.me.parabola.mkgmap.build.LocatorConfig.<init>(LocatorConfig.java:47) at uk.me.parabola.mkgmap.build.Locator.<init>(Locator.java:68) at uk.me.parabola.mkgmap.build.MapBuilder.<init>(MapBuilder.java:103) at uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:97) at uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:65) at uk.me.parabola.mkgmap.main.Main$1.call(Main.java:224) at uk.me.parabola.mkgmap.main.Main$1.call(Main.java:221) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) [Fatal Error] :66:2: The content of elements must consist of well-formed character data or markup. org.xml.sax.SAXParseException: The content of elements must consist of well-formed character data or markup. at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(Unknown Source) at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(Unknown Source) at javax.xml.parsers.DocumentBuilder.parse(Unknown Source) at uk.me.parabola.mkgmap.build.LocatorConfig.loadConfig(LocatorConfig.java:70) at uk.me.parabola.mkgmap.build.LocatorConfig.<init>(LocatorConfig.java:47) at uk.me.parabola.mkgmap.build.Locator.<init>(Locator.java:68) at uk.me.parabola.mkgmap.build.MapBuilder.<init>(MapBuilder.java:103) at uk.me.parabola.mkgmap.combiners.TdbBuilder.writeOverviewMap(TdbBuilder.java:246) at uk.me.parabola.mkgmap.combiners.TdbBuilder.onFinish(TdbBuilder.java:239) at uk.me.parabola.mkgmap.main.Main.endOptions(Main.java:419) at uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.java:126) at uk.me.parabola.mkgmap.main.Main.main(Main.java:129)
This error complains about a wrong format of you LocatorConfig.xml. The XML parser is not able to parse it.
2a) as above but using mkgmap.jar trunk rev 1949 as downloaded from http://www.mkgmap.org.uk/snapshots/mkgmap-r1949.zip No errors, but also I don't know if pbf support is icluded here??
*3. mkgmap_locator with my patches including pbf support (without using bound option with newest bound files as published here on the list):* No map created but 0bit file only, and mkgmap stuck for 20minutes. SCHWERWIEGEND (LocationHook): c:\openmtbmap\maps\63990000.osm.gz: Element lists created after 627 ms SCHWERWIEGEND (LocationHook): c:\openmtbmap\maps\63990000.osm.gz: Quadtree created after 3194 ms SCHWERWIEGEND (LocationHook): c:\openmtbmap\maps\63990000.osm.gz: Location hook finished in 3823 ms
*4. mkgmap_locator as above, but adding --createboundsfile=c:\openmtbmap\maps\boundaries" to the input options:* No map created but 0bit file only, and mkgmap stuck for 20minutes. SCHWERWIEGEND (LocationHook): c:\openmtbmap\maps\63990000.osm.gz: Element lists created after 657 ms SCHWERWIEGEND (LocationHook): c:\openmtbmap\maps\63990000.osm.gz: Quadtree created after 3021 ms SCHWERWIEGEND (LocationHook): c:\openmtbmap\maps\63990000.osm.gz: Location hook finished in 3569 ms
*5. Same as 3. But using --max-jobs=1:* No map created, but 0bit file only, mkgmap stuck for 20 minutes. 21:55:40 iceland is 6399 this is run6 start compilation 21:55:52 iceland SCHWERWIEGEND (LocationHook): c:\openmtbmap\maps\63990000.osm.gz: Element lists created after 350 ms SCHWERWIEGEND (LocationHook): c:\openmtbmap\maps\63990000.osm.gz: Quadtree created after 1660 ms SCHWERWIEGEND (LocationHook): c:\openmtbmap\maps\63990000.osm.gz: Location hook finished in 2277 ms
*6. Same as 3. but using mkgmap_locator_r1952 downloaded from mkgmap.co.uk.:* mkgmap stuck for 20 minutes .... no map created, only 0bit file. 22:11:48 iceland is 6399 this is run6 start compilation 22:12:00 iceland SCHWERWIEGEND (LocationHook): c:\openmtbmap\maps\63990000.osm.gz: Element lists created after 232 ms SCHWERWIEGEND (LocationHook): c:\openmtbmap\maps\63990000.osm.gz: Quadtree created after 2587 ms SCHWERWIEGEND (LocationHook): c:\openmtbmap\maps\63990000.osm.gz: Location hook finished in 3158 ms
Felix, can you please post a stack trace of the situation when mkgmap hangs? WanMil
On 22.05.2011 22:35, WanMil wrote:
Okay, some more tries to find out why mkgmap locks up on iceland (but not on most countries, so the thing is still really strange):
I'll upload all needed files to recreate the problem in another mail, cause that will be easier than e-mailing forth/back to find out why the problem happens.
First using my normal commandline: start /low /b /wait java -ea -jar -Xmx6500M c:\openmtbmap\mkgmap.jar %style-file% --max-jobs=4 %generate-sea% --reduce-point-density=4 --nsis --index --transparent --adjust-turn-headings --add-pois-to-areas --ignore-maxspeeds --x-reduce-point-density-polygon=8 --link-pois-to-ways --ignore-turn-restrictions --min-size-polygon=15 --remove-short-arcs=4 --description=openmtbmap_%abr% --merge-lines --location-autofill=1 --route --country-abbr=%abr% --country-name=%country% %profile% --mapname=%FID%0000 --family-id=%FID% --product-id=1 --series-name=openmtbmap_%country%_%date% --family-name=mtbmap_%abr%_%date% --tdbfile --overview-mapname=mapset --keep-going --area-name="%country%_%date%_openmtbmap.org" -c c:\openmtbmap\maps\template.%country%
Country is Iceland. I hope the error message on 2. can provide the needed information why the locator branch gets stuck indefinitely?? Note, I'm not using pbf as input files, but standard osm.gz, therefore I find it strange that including pbf support without any other changes, creates an Fatal Error, plus adds about 18% of time surplus to render the map.
I clearly have a problem with a lib, but have no clue how to solve this. Therefore please, include the libs into mkgmap svn, everything else will cause faults. I clearly have the libs that mkgmap wants, but probably got a wrong version that mkgmap does not expect at some point. However also the mkgmap_locator_r1492.jar downloaded from mkgmap.org.uk gets stuck for me indefinitely, so there definitely is a problem.
Here is the output on building mkgmap.jar, it looks fine to me: d:\Garmin\mkgmap_svn_trunk>start /b /wait ant dist Buildfile: build.xml
prepare: [mkdir] Created dir: d:\Garmin\mkgmap_svn_trunk\build\classes compile: [javac] Compiling 378 source files to d:\Garmin\mkgmap_svn_trunk\build\classes [javac] Note: Some input files use unchecked or unsafe operations. [javac] Note: Recompile with -Xlint:unchecked for details. compile-pbf: [echo] Protobuf binary format support [javac] Compiling 3 source files to d:\Garmin\mkgmap_svn_trunk\build\classes build: [copy] Copying 422 files to d:\Garmin\mkgmap_svn_trunk\build\classes dist: [jar] Building jar: d:\Garmin\mkgmap_svn_trunk\dist\mkgmap.jar
*1. mkgmap_trunk without pbf support (including my patches):* 44 seconds No error message.
*2. mkgmap_trunk with pbf support (including my patches)*: 50 seconds plus error message (but map created): [Fatal Error] :66:2: The content of elements must consist of well-formed character data or markup. org.xml.sax.SAXParseException: The content of elements must consist of well-formed character data or markup. at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(Unknown Source) at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(Unknown Source) at javax.xml.parsers.DocumentBuilder.parse(Unknown Source) at uk.me.parabola.mkgmap.build.LocatorConfig.loadConfig(LocatorConfig.java:70) at uk.me.parabola.mkgmap.build.LocatorConfig.<init>(LocatorConfig.java:47) at uk.me.parabola.mkgmap.build.Locator.<init>(Locator.java:68) at uk.me.parabola.mkgmap.build.MapBuilder.<init>(MapBuilder.java:103) at uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:97) at uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:65) at uk.me.parabola.mkgmap.main.Main$1.call(Main.java:224) at uk.me.parabola.mkgmap.main.Main$1.call(Main.java:221) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) [Fatal Error] :66:2: The content of elements must consist of well-formed character data or markup. org.xml.sax.SAXParseException: The content of elements must consist of well-formed character data or markup. at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(Unknown Source) at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(Unknown Source) at javax.xml.parsers.DocumentBuilder.parse(Unknown Source) at uk.me.parabola.mkgmap.build.LocatorConfig.loadConfig(LocatorConfig.java:70) at uk.me.parabola.mkgmap.build.LocatorConfig.<init>(LocatorConfig.java:47) at uk.me.parabola.mkgmap.build.Locator.<init>(Locator.java:68) at uk.me.parabola.mkgmap.build.MapBuilder.<init>(MapBuilder.java:103) at uk.me.parabola.mkgmap.combiners.TdbBuilder.writeOverviewMap(TdbBuilder.java:246) at uk.me.parabola.mkgmap.combiners.TdbBuilder.onFinish(TdbBuilder.java:239) at uk.me.parabola.mkgmap.main.Main.endOptions(Main.java:419) at uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.java:126) at uk.me.parabola.mkgmap.main.Main.main(Main.java:129) This error complains about a wrong format of you LocatorConfig.xml. The XML parser is not able to parse it. How can this happen, if without pbf support, LocatorConfig.xml can be parsed without problems?? 2a) as above but using mkgmap.jar trunk rev 1949 as downloaded from http://www.mkgmap.org.uk/snapshots/mkgmap-r1949.zip No errors, but also I don't know if pbf support is icluded here??
*3. mkgmap_locator with my patches including pbf support (without using bound option with newest bound files as published here on the list):* No map created but 0bit file only, and mkgmap stuck for 20minutes. SCHWERWIEGEND (LocationHook): c:\openmtbmap\maps\63990000.osm.gz: Element lists created after 627 ms SCHWERWIEGEND (LocationHook): c:\openmtbmap\maps\63990000.osm.gz: Quadtree created after 3194 ms SCHWERWIEGEND (LocationHook): c:\openmtbmap\maps\63990000.osm.gz: Location hook finished in 3823 ms
*4. mkgmap_locator as above, but adding --createboundsfile=c:\openmtbmap\maps\boundaries" to the input options:* No map created but 0bit file only, and mkgmap stuck for 20minutes. SCHWERWIEGEND (LocationHook): c:\openmtbmap\maps\63990000.osm.gz: Element lists created after 657 ms SCHWERWIEGEND (LocationHook): c:\openmtbmap\maps\63990000.osm.gz: Quadtree created after 3021 ms SCHWERWIEGEND (LocationHook): c:\openmtbmap\maps\63990000.osm.gz: Location hook finished in 3569 ms
*5. Same as 3. But using --max-jobs=1:* No map created, but 0bit file only, mkgmap stuck for 20 minutes. 21:55:40 iceland is 6399 this is run6 start compilation 21:55:52 iceland SCHWERWIEGEND (LocationHook): c:\openmtbmap\maps\63990000.osm.gz: Element lists created after 350 ms SCHWERWIEGEND (LocationHook): c:\openmtbmap\maps\63990000.osm.gz: Quadtree created after 1660 ms SCHWERWIEGEND (LocationHook): c:\openmtbmap\maps\63990000.osm.gz: Location hook finished in 2277 ms
*6. Same as 3. but using mkgmap_locator_r1952 downloaded from mkgmap.co.uk.:* mkgmap stuck for 20 minutes .... no map created, only 0bit file. 22:11:48 iceland is 6399 this is run6 start compilation 22:12:00 iceland SCHWERWIEGEND (LocationHook): c:\openmtbmap\maps\63990000.osm.gz: Element lists created after 232 ms SCHWERWIEGEND (LocationHook): c:\openmtbmap\maps\63990000.osm.gz: Quadtree created after 2587 ms SCHWERWIEGEND (LocationHook): c:\openmtbmap\maps\63990000.osm.gz: Location hook finished in 3158 ms
Felix,
can you please post a stack trace of the situation when mkgmap hangs? I don't know how to get a stack trace. Is there a howto for this somewhere? I'm right now trying to cut down the options to find out, which switch is breaking it. without any switches the locator branch does compile fine -- I'll put that in the announced 2.nd mail. WanMil _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
On 22.05.2011 22:49, Felix Hartmann wrote:
How can this happen, if without pbf support, LocatorConfig.xml can be parsed without problems??
okay, found this problem, I had an update turn wrong on me and some lines appeared twice therefore. So pbf libs seem okay, only problems with locator branch for now. (separate thread for more exact description)
I don't know how to get a stack trace. Is there a howto for this somewhere?
Your JDK installation contains a program named jvisualvm. jvisualvm gives you a list of running programs. Double click the blocking mkgmap instance. Then you see different tabs, one of it is "Threads" containing a button "Thread Dump". Hit this button and you get a full stack trace including some warnings if a deadlock exists. WanMil
On 23.05.2011 17:58, WanMil wrote:
I don't know how to get a stack trace. Is there a howto for this somewhere?
Your JDK installation contains a program named jvisualvm. jvisualvm gives you a list of running programs. Double click the blocking mkgmap instance. Then you see different tabs, one of it is "Threads" containing a button "Thread Dump". Hit this button and you get a full stack trace including some warnings if a deadlock exists.
WanMil Okay, here is the dump - I don't know what to do with it.: (just some notes before, the used heap is fluctuating between 500MB and 1.6GB of around 2.8GB heap size that got reduced to around 2.2GB and 8.5GB max-heap set (it definitely not runs out of memory). I have max-jobs=4, but only one CPU core is maxed out.
2011-05-23 21:38:06 Full thread dump Java HotSpot(TM) 64-Bit Server VM (17.0-b16 mixed mode): "RMI TCP Connection(2)-88.198.53.163" daemon prio=6 tid=0x0000000008a01000 nid=0xd24 runnable [0x000000000d98f000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(Unknown Source) at java.io.BufferedInputStream.fill(Unknown Source) at java.io.BufferedInputStream.read(Unknown Source) - locked <0x00000001e0ec8bd8> (a java.io.BufferedInputStream) at java.io.FilterInputStream.read(Unknown Source) at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - <0x00000001db78c950> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) "JMX server connection timeout 14" daemon prio=6 tid=0x00000000093e3000 nid=0xd7c in Object.wait() [0x000000000d88f000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000001db7803d0> (a [I) at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(Unknown Source) - locked <0x00000001db7803d0> (a [I) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "RMI Scheduler(0)" daemon prio=6 tid=0x000000000ada4800 nid=0x19dc waiting on condition [0x000000000d78f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000001db6d09f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source) at java.util.concurrent.DelayQueue.take(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "RMI TCP Connection(1)-88.198.53.163" daemon prio=6 tid=0x00000000089b5000 nid=0xcdc runnable [0x000000000d68f000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(Unknown Source) at java.io.BufferedInputStream.fill(Unknown Source) at java.io.BufferedInputStream.read(Unknown Source) - locked <0x00000001e0eee9e0> (a java.io.BufferedInputStream) at java.io.FilterInputStream.read(Unknown Source) at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - <0x00000001db78cc10> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) "RMI TCP Accept-0" daemon prio=6 tid=0x00000000099b4800 nid=0x1c8c runnable [0x000000000d58f000] java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(Unknown Source) - locked <0x00000001db7805c8> (a java.net.SocksSocketImpl) at java.net.ServerSocket.implAccept(Unknown Source) at java.net.ServerSocket.accept(Unknown Source) at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(Unknown Source) at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(Unknown Source) at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-1-thread-1" prio=6 tid=0x0000000007961800 nid=0x1de8 runnable [0x000000000846f000] java.lang.Thread.State: RUNNABLE at java.util.LinkedList.entry(Unknown Source) at java.util.LinkedList.get(Unknown Source) at uk.me.parabola.imgfmt.app.net.NETFile.simplifySortedRoads(NETFile.java:140) at uk.me.parabola.imgfmt.app.net.NETFile.writePost(NETFile.java:84) at uk.me.parabola.mkgmap.build.MapBuilder.makeMap(MapBuilder.java:228) at uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:101) at uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:65) at uk.me.parabola.mkgmap.main.Main$1.call(Main.java:222) at uk.me.parabola.mkgmap.main.Main$1.call(Main.java:219) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - <0x0000000086234090> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) "Low Memory Detector" daemon prio=6 tid=0x00000000077cd800 nid=0x1f34 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "CompilerThread1" daemon prio=10 tid=0x00000000077cb800 nid=0x19d4 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "CompilerThread0" daemon prio=10 tid=0x00000000077c4800 nid=0x17d8 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "Attach Listener" daemon prio=10 tid=0x00000000077c1000 nid=0x196c waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "Signal Dispatcher" daemon prio=10 tid=0x00000000077bc000 nid=0x1264 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "Finalizer" daemon prio=8 tid=0x0000000000522800 nid=0x1dc8 in Object.wait() [0x000000000776f000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x0000000086224688> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(Unknown Source) - locked <0x0000000086224688> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(Unknown Source) at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source) Locked ownable synchronizers: - None "Reference Handler" daemon prio=10 tid=0x0000000000521000 nid=0x900 in Object.wait() [0x000000000766f000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x0000000086224640> (a java.lang.ref.Reference$Lock) at java.lang.Object.wait(Object.java:485) at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source) - locked <0x0000000086224640> (a java.lang.ref.Reference$Lock) Locked ownable synchronizers: - None "main" prio=6 tid=0x00000000002bb800 nid=0x768 waiting on condition [0x00000000024ff000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at uk.me.parabola.mkgmap.main.Main.endOptions(Main.java:383) at uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.java:126) at uk.me.parabola.mkgmap.main.Main.main(Main.java:132) Locked ownable synchronizers: - None "VM Thread" prio=10 tid=0x000000000051d000 nid=0xccc runnable "GC task thread#0 (ParallelGC)" prio=6 tid=0x0000000000476800 nid=0x1e68 runnable "GC task thread#1 (ParallelGC)" prio=6 tid=0x0000000000479000 nid=0x1f0c runnable "GC task thread#2 (ParallelGC)" prio=6 tid=0x000000000047b000 nid=0x394 runnable "GC task thread#3 (ParallelGC)" prio=6 tid=0x000000000047d000 nid=0x14d8 runnable "VM Periodic Task Thread" prio=10 tid=0x00000000077d9000 nid=0xdc4 waiting on condition JNI global references: 1134
participants (5)
-
Felix Hartmann -
Francisco Moraes -
frmas -
Nakor Osm -
WanMil