Hi, I was trying to update my Benelux map but the splitter refused to finish the process. Normally, splitter is finished in 2 minutes, now it gets in an infinite loop with Full GC. What can be the cause for this, a corrupt download of the europe extract?
Am Samstag, 8. März 2014, 10:44:16 schrieb Minko:
I was trying to update my Benelux map but the splitter refused to finish the process. Normally, splitter is finished in 2 minutes, now it gets in an infinite loop with Full GC. What can be the cause for this, a corrupt download of the europe extract?
I have seen this error last week, too. I had to recreate my dach extract. Bernd -- amarok2 now playing:
Hi Minko, please keep the files that produce this problem. I'd like to have them to be able to reproduce it. Gerd
Date: Sat, 8 Mar 2014 11:09:03 +0100 From: ligfietser@online.nl To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] Infinite loop splitter
What have you done to solve this Bernd, should I just wait for another extract tomorrow?
I have seen this error last week, too.
I had to recreate my dach extract.
Bernd
mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Gerd, I always mix the osm data with address nodes for Vlaanderen and the Netherlands with osmconvert. This part of my batch file looks like this: :osmconvert %DIR%\Tools\osmconvert --drop-version %DIR%\%date%\Splitter\%extract% %DIR%\resources\adres.o5m %DIR%\resources\vlaanderen-adres.o5m -B=%DIR%\resources\bnl_01.poly -o=%DIR%\%date%\Splitter\benelux.o5m %extract% is europe.osm.pbf from today NL adres.o5m: lon min: 3.3590433 lon max: 7.2259690 lat min: 50.7510383 lat max: 53.4972667 nodes: 8551053 ways: 0 relations: 0 node id min: 100000000000 node id max: 100008551052 keyval pairs max: 2 keyval pairs max object: node 100000000000 Vlaanderen vlaanderen-adres.o5m lon min: 2.3066895 lon max: 5.8980601 lat min: 49.2933351 lat max: 51.5047683 nodes: 2449726 ways: 0 relations: 0 node id min: 200000000001 node id max: 200002449726 keyval pairs max: 4 keyval pairs max object: node 200000000001 After this I split the resulting file as follows :Splitter java -Xmx8000m -XX:+UseParallelGC -XX:+HeapDumpOnOutOfMemoryError -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -jar %SPLITTER% --output-dir=%date%\Splitter --max-threads=8 --problem-file=%DIR%\resources\problem_areas.txt --precomp-sea=%seafile% --keep-complete --overlap=0 --mapid=10010001 --polygon-file=%DIR%\resources\benelux.poly --max-nodes=1750000 --write-kml=%areas%.kml --output=o5m --geonames-file=%DIR%\resources\cities5000.txt --description=OFM_BNL %DIR%\%date%\Splitter\benelux.o5m > splitter2.log 2>err2.log Normally it all goes well until this morning. I've tried the latest splitter version too but it didnt help. I also tried to make the first polygon bnl_01.poly much bigger ( it is a little bigger than the second benelux.poly in the splitter step 2). This also didnt help. Now if I skip the first step and split it without merging the address nodes it goes well without problems. Could it be that the splitter creates fake ID's that overlap my address data nodes? Or is there an overlap in osm id's with the fake address nodes?
BTW Here are the links to the files in case you need them: https://www.dropbox.com/sh/d3ssenipw2vx6qs/ucrj2cNiFM/adres.o5m https://www.dropbox.com/sh/d3ssenipw2vx6qs/bgWpOCpfwK/vlaanderen-adres.o5m https://www.dropbox.com/sh/d3ssenipw2vx6qs/zqzksFWZcp/bnl_01.poly https://www.dropbox.com/sh/d3ssenipw2vx6qs/2N8CdpLXQc/benelux.poly
Hi Minko, ligfietser wrote
Now if I skip the first step and split it without merging the address nodes it goes well without problems. Could it be that the splitter creates fake ID's that overlap my address data nodes? Or is there an overlap in osm id's with the fake address nodes?
I have no idea. Download of Europe will take a while... Gerd -- View this message in context: http://gis.19327.n5.nabble.com/Infinite-loop-splitter-tp5799076p5799084.html Sent from the Mkgmap Development mailing list archive at Nabble.com.
Hi Minko, is there a reason for merging the files before splitting? Did you try to just pass all the files to splitter? I'm doing so with osm and srtm-data. Don't know, if this helps in this case, but in general I think it should be much faster. Henning
Hi Henning, I didnt know that you could feed multiple files to the splitter? How do you do this?
Hi Minko, is there a reason for merging the files before splitting? Did you try to just pass all the files to splitter? I'm doing so with osm and srtm-data. Don't know, if this helps in this case, but in general I think it should be much faster.
Henning
Hi Minko, I've committed r320 to fix the loop problem. You can specify multiplie input files, but you have to make sure that each file contains ids which are higher than all previous ones. I think this is the case with your address files. The reason for the loop was a way in a mp-relation for which no nodes were found. I guess this is caused by using osmconvert without the parameters --complete-ways --complex-ways I am not sure what's faster: Using osmconvert with polygon first or only splitter with multiple input files. When all files are in o5m format, single run with splitter should be quite fast. Gerd -- View this message in context: http://gis.19327.n5.nabble.com/Infinite-loop-splitter-tp5799076p5799103.html Sent from the Mkgmap Development mailing list archive at Nabble.com.
Hi Gerd, Thanks for the fix, I will test it out. I tried Hennings approach but splitting seemed to last much longer because it analyses the whole EU extract and I only need the Benelux plus a small area of France and Germany. Splitting wasn't completed and threw out this error: warning: --keep-complete is only used for the first input file. New way id 195314 is not higher than last id 264955883 Maybe the IDs are not sorted. This is not supported with keep-complete=true or --problem-list I'll try now r320.
I've committed r320 to fix the loop problem.
You can specify multiplie input files, but you have to make sure that each file contains ids which are higher than all previous ones. I think this is the case with your address files.
The reason for the loop was a way in a mp-relation for which no nodes were found. I guess this is caused by using osmconvert without the parameters --complete-ways --complex-ways
I am not sure what's faster: Using osmconvert with polygon first or only splitter with multiple input files. When all files are in o5m format, single run with splitter should be quite fast.
Gerd
Gerd wrote:
The reason for the loop was a way in a mp-relation for which no nodes were found. I guess this is caused by using osmconvert without the parameters --complete-ways --complex-ways
If I use --complete-ways or --complex-ways with osmconvert, I didnt get any output at all so I always skip this.
Hi Minko, the order of input files would be java ... splitter.jar [options] europe.o5m adres.o5m vlaanderen-adres.o5m Your approach with osmconvert is probably faster because you have to use osmconvert anyway to convert to o5m. Gerd ligfietser wrote
Hi Gerd, Thanks for the fix, I will test it out. I tried Hennings approach but splitting seemed to last much longer because it analyses the whole EU extract and I only need the Benelux plus a small area of France and Germany.
Splitting wasn't completed and threw out this error:
warning: --keep-complete is only used for the first input file. New way id 195314 is not higher than last id 264955883 Maybe the IDs are not sorted. This is not supported with keep-complete=true or --problem-list
I'll try now r320.
I've committed r320 to fix the loop problem.
You can specify multiplie input files, but you have to make sure that each file contains ids which are higher than all previous ones. I think this is the case with your address files.
The reason for the loop was a way in a mp-relation for which no nodes were found. I guess this is caused by using osmconvert without the parameters --complete-ways --complex-ways
I am not sure what's faster: Using osmconvert with polygon first or only splitter with multiple input files. When all files are in o5m format, single run with splitter should be quite fast.
Gerd
mkgmap-dev mailing list
mkgmap-dev@.org
-- View this message in context: http://gis.19327.n5.nabble.com/Infinite-loop-splitter-tp5799076p5799107.html Sent from the Mkgmap Development mailing list archive at Nabble.com.
Hi Minko, fine:-) I think the only good reason to use splitter with an input file like europe.o5m or planet.o5m in combination with a poly file that extracts a small part is that it is able to keep ferry lines complete. osmconvert --complete-ways --complex-ways -B=bnl_01.poly europe.o5m -o=benelux.o5m fails for me as well. I'll ask the author about it. Gerd ligfietser wrote
Gerd wrote:
the order of input files would be java ... splitter.jar [options] europe.o5m adres.o5m vlaanderen-adres.o5m
Yes I did that too. Anyway, with r320 it seems to work, mkgmap is running now thanks!
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
-- View this message in context: http://gis.19327.n5.nabble.com/Infinite-loop-splitter-tp5799076p5799112.html Sent from the Mkgmap Development mailing list archive at Nabble.com.
Hi all, I've contacted Markus Weber, the author of osmconvert and osmfilter. It seems that the problem with complex-ways occurs only in the Windows version, maybe because something goes wrong with files > 4GB. Markus doesn't have access to a Win64 system. I use a mingw-64 installation to compile my own binary for 64 bit, maybe an update will help, but I doubt it, and unfortunately I forgot how the installation works, I just remember it wasn't simple. So, if you use osmconvert/osmfilter to update your files it is probably better to use a 64 bit Linux installation. Gerd GerdP wrote
Hi Minko,
fine:-)
I think the only good reason to use splitter with an input file like europe.o5m or planet.o5m in combination with a poly file that extracts a small part is that it is able to keep ferry lines complete.
osmconvert --complete-ways --complex-ways -B=bnl_01.poly europe.o5m -o=benelux.o5m fails for me as well. I'll ask the author about it.
Gerd ligfietser wrote
Gerd wrote:
the order of input files would be java ... splitter.jar [options] europe.o5m adres.o5m vlaanderen-adres.o5m
Yes I did that too. Anyway, with r320 it seems to work, mkgmap is running now thanks!
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
-- View this message in context: http://gis.19327.n5.nabble.com/Infinite-loop-splitter-tp5799076p5799142.html Sent from the Mkgmap Development mailing list archive at Nabble.com.
R320 certainly seems to have fixed it for me. If there is still a problem with osmconvert etc then the new splitter appears to cope with it somehow. I will keep an eye on the maps coming out of mkgmap in case there any anomalies, but so far it's looking good. Thanks for the fix! Colin On 2014-03-09 08:49, GerdP wrote:
Hi all,
I've contacted Markus Weber, the author of osmconvert and osmfilter. It seems that the problem with complex-ways occurs only in the Windows version, maybe because something goes wrong with files > 4GB. Markus doesn't have access to a Win64 system. I use a mingw-64 installation to compile my own binary for 64 bit, maybe an update will help, but I doubt it, and unfortunately I forgot how the installation works, I just remember it wasn't simple.
So, if you use osmconvert/osmfilter to update your files it is probably better to use a 64 bit Linux installation.
Gerd
GerdP wrote Hi Minko, fine:-) I think the only good reason to use splitter with an input file like europe.o5m or planet.o5m in combination with a poly file that extracts a small part is that it is able to keep ferry lines complete. osmconvert --complete-ways --complex-ways -B=bnl_01.poly europe.o5m -o=benelux.o5m fails for me as well. I'll ask the author about it. Gerd ligfietser wrote Gerd wrote: the order of input files would be java ... splitter.jar [options] europe.o5m adres.o5m vlaanderen-adres.o5m Yes I did that too. Anyway, with r320 it seems to work, mkgmap is running now thanks! _______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
-- View this message in context: http://gis.19327.n5.nabble.com/Infinite-loop-splitter-tp5799076p5799142.html [2] Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [1] Links: ------ [1] http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev [2] http://gis.19327.n5.nabble.com/Infinite-loop-splitter-tp5799076p5799142.html
Hi Colin, thanks for the feedback. It was an obvious bug (once identified), but did not occur for me as I always used downloads from geofabrik. I don't say you can't use osmconvert, but I would be careful using the Windows version in combinaton with osmupdate and files > 4GB. Gerd Date: Sun, 9 Mar 2014 16:17:58 +0100 From: colin.smale@xs4all.nl To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] Infinite loop splitter R320 certainly seems to have fixed it for me. If there is still a problem with osmconvert etc then the new splitter appears to cope with it somehow. I will keep an eye on the maps coming out of mkgmap in case there any anomalies, but so far it's looking good. Thanks for the fix! Colin On 2014-03-09 08:49, GerdP wrote: Hi all, I've contacted Markus Weber, the author of osmconvert and osmfilter. It seems that the problem with complex-ways occurs only in the Windows version, maybe because something goes wrong with files > 4GB. Markus doesn't have access to a Win64 system. I use a mingw-64 installation to compile my own binary for 64 bit, maybe an update will help, but I doubt it, and unfortunately I forgot how the installation works, I just remember it wasn't simple. So, if you use osmconvert/osmfilter to update your files it is probably better to use a 64 bit Linux installation. Gerd GerdP wrote Hi Minko, fine:-) I think the only good reason to use splitter with an input file like europe.o5m or planet.o5m in combination with a poly file that extracts a small part is that it is able to keep ferry lines complete. osmconvert --complete-ways --complex-ways -B=bnl_01.poly europe.o5m -o=benelux.o5m fails for me as well. I'll ask the author about it. Gerd ligfietser wrote Gerd wrote: the order of input files would be java ... splitter.jar [options] europe.o5m adres.o5m vlaanderen-adres.o5m Yes I did that too. Anyway, with r320 it seems to work, mkgmap is running now thanks! _______________________________________________ mkgmap-dev mailing list mkgmap-dev@.org http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev -- View this message in context: http://gis.19327.n5.nabble.com/Infinite-loop-splitter-tp5799076p5799142.html Sent from the Mkgmap Development mailing list archive at Nabble.com. _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Am Samstag, 8. März 2014, 23:49:18 schrieb GerdP:
So, if you use osmconvert/osmfilter to update your files it is probably better to use a 64 bit Linux installation.
BTW Hmmh, i saw this error on a 64 bit linux system with 16 GiB RAM and i use osmupdate to update an extract from a local planet This is the splitter.log from march, 1st 2014 splitter version 317 compiled 2014-02-27T07:09:08+0000 boundary-tags=use-exclude-list cache= .... Elapsed time: 10m 0s Memory: Current 2278MB (1176MB used, 1102MB free) Max 5461MB Elapsed time: 12m 0s Memory: Current 2278MB (1178MB used, 1100MB free) Max 5461MB Elapsed time: 14m 0s Memory: Current 2278MB (1178MB used, 1100MB free) Max 5461MB Elapsed time: 16m 0s Memory: Current 2278MB (1178MB used, 1100MB free) Max 5461MB ***** Full GC ***** .... Elapsed time: 11h 4m 7s Memory: Current 2386MB (743MB used, 1643MB free) Max 5461MB Elapsed time: 11h 6m 7s Memory: Current 2386MB (743MB used, 1643MB free) Max 5461MB Elapsed time: 11h 8m 7s Memory: Current 2386MB (743MB used, 1643MB free) Max 5461MB Elapsed time: 11h 10m 7s Memory: Current 2386MB (743MB used, 1643MB free) Max 5461MB Elapsed time: 11h 12m 7s Memory: Current 2386MB (743MB used, 1643MB free) Max 5461MB Elapsed time: 11h 14m 7s Memory: Current 2386MB (743MB used, 1643MB free) Max 5461MB Elapsed time: 11h 16m 7s Memory: Current 2386MB (743MB used, 1643MB free) Max 5461MB Bernd -- amarok2 now playing: artist: Sade title: The Sweetest Taboo album: The Best of Sade
Hi Bernd, the error is splitter is not dependent on the System. I don't know if osmupdate tries to make sure that ways are kept complete if you use it with a bbox or polygon file. Gerd
From: weigelt.bernd@web.de To: mkgmap-dev@lists.mkgmap.org.uk Date: Mon, 10 Mar 2014 16:08:19 +0100 Subject: Re: [mkgmap-dev] Infinite loop splitter
Am Samstag, 8. März 2014, 23:49:18 schrieb GerdP:
So, if you use osmconvert/osmfilter to update your files it is probably better to use a 64 bit Linux installation.
BTW Hmmh, i saw this error on a 64 bit linux system with 16 GiB RAM and i use osmupdate to update an extract from a local planet
This is the splitter.log from march, 1st 2014
splitter version 317 compiled 2014-02-27T07:09:08+0000 boundary-tags=use-exclude-list cache= .... Elapsed time: 10m 0s Memory: Current 2278MB (1176MB used, 1102MB free) Max 5461MB Elapsed time: 12m 0s Memory: Current 2278MB (1178MB used, 1100MB free) Max 5461MB Elapsed time: 14m 0s Memory: Current 2278MB (1178MB used, 1100MB free) Max 5461MB Elapsed time: 16m 0s Memory: Current 2278MB (1178MB used, 1100MB free) Max 5461MB ***** Full GC ***** .... Elapsed time: 11h 4m 7s Memory: Current 2386MB (743MB used, 1643MB free) Max 5461MB Elapsed time: 11h 6m 7s Memory: Current 2386MB (743MB used, 1643MB free) Max 5461MB Elapsed time: 11h 8m 7s Memory: Current 2386MB (743MB used, 1643MB free) Max 5461MB Elapsed time: 11h 10m 7s Memory: Current 2386MB (743MB used, 1643MB free) Max 5461MB Elapsed time: 11h 12m 7s Memory: Current 2386MB (743MB used, 1643MB free) Max 5461MB Elapsed time: 11h 14m 7s Memory: Current 2386MB (743MB used, 1643MB free) Max 5461MB Elapsed time: 11h 16m 7s Memory: Current 2386MB (743MB used, 1643MB free) Max 5461MB
Bernd
-- amarok2 now playing: artist: Sade title: The Sweetest Taboo album: The Best of Sade
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
I see this very often as well, when I use osmupdate on an earlier download. The only solution I have found is to download a new pbf, then it works again. I suspect therefore that osmupdate or one of the underlying tools is corrupting the data. Colin On 8 March 2014 10:44:16 CET, Minko <ligfietser@online.nl> wrote:
Hi, I was trying to update my Benelux map but the splitter refused to finish the process. Normally, splitter is finished in 2 minutes, now it gets in an infinite loop with Full GC. What can be the cause for this, a corrupt download of the europe extract?
------------------------------------------------------------------------
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Colin, thanks for the hint. No matter what other tools do: infinite loop in splitter is the worst case as a reaction on errors ;-) Gerd Colin Smale wrote
I see this very often as well, when I use osmupdate on an earlier download. The only solution I have found is to download a new pbf, then it works again. I suspect therefore that osmupdate or one of the underlying tools is corrupting the data.
Colin
On 8 March 2014 10:44:16 CET, Minko <
ligfietser@
> wrote:
Hi, I was trying to update my Benelux map but the splitter refused to finish the process. Normally, splitter is finished in 2 minutes, now it gets in an infinite loop with Full GC. What can be the cause for this, a corrupt download of the europe extract?
------------------------------------------------------------------------
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
-- View this message in context: http://gis.19327.n5.nabble.com/Infinite-loop-splitter-tp5799076p5799086.html Sent from the Mkgmap Development mailing list archive at Nabble.com.
Gerd, If I skip merging the Vlaanderen address data then splitting goes well. I have uploaded my 'corrupt' Benelux data which goes wrong (incl the Vlaanderen addresses) here: http://mijndev.openstreetmap.nl/~ligfietser/test/benelux.o5m
Hi Minko, thanks, I can reproduce the problem with this file. Looking for a fix now. Gerd ligfietser wrote
Gerd, If I skip merging the Vlaanderen address data then splitting goes well. I have uploaded my 'corrupt' Benelux data which goes wrong (incl the Vlaanderen addresses) here: http://mijndev.openstreetmap.nl/~ligfietser/test/benelux.o5m
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
-- View this message in context: http://gis.19327.n5.nabble.com/Infinite-loop-splitter-tp5799076p5799099.html Sent from the Mkgmap Development mailing list archive at Nabble.com.
participants (6)
-
Bernd Weigelt -
Colin Smale -
Gerd Petermann -
GerdP
-
Henning Scholland -
Minko