Error when running splitter with several input files, could there be keep-complete=single mode?
I split france.osm.pbf with D:\openmtbmap\maps>if france == legend start /belownormal /b /wait java -jar -XX:+AggressiveHeap -Xms1000m -Xmx3800m C:\openmtbmap\splitter.jar --max-nodes=9600000 --max-threads=12 --search-limit=1000000 --output=pbf "--keep-complete" --max-areas=1024 --geonames-file=D:\openmtbmap\osmpbf_geofabrik\cities5000.zip --description=france --mapid=83910000 france.osm 1>NUL Then I compiled the map using mgkamp.jar - and want to resplit all remaining files that did not compile with smaller max-nodes value. However splitter.jar chokes if I pass several files... Is there any way to have splitter just working with several files without trying to merge them? I just need the files split to the new max-nodes value and incrementing the map id. If I pass only one file at a time, it is a PITA to script this (is already hard enough figuring out the new map-id and creating a list of the tiles). D:\openmtbmap\maps>if yes EQU yes start /belownormal /b /wait java -XX:+AggressiveHeap -Xms5000M -Xmx54000M -jar C:\openmtbmap\splitter.jar --max-nodes=4800000 --max-threads=12 --search-limit=1000000 --output=pbf --geonames-file=D:\openmtbmap\osmpbf_geofabrik\cities5000.zip --description=france --mapid=83910046 "83910015.osm.pbf" "83910025.osm.pbf" "83910026.osm.pbf" "83910042.osm.pbf" 1>NUL Warning: --keep-complete is only used for the first input file. Further files must use higher ids. Error: Node ids are not sorted. Use e.g. osmosis to sort the input data. This is not supported with keep-complete=true or --problem-list uk.me.parabola.splitter.SplitFailedException: Node ids are not sorted at uk.me.parabola.splitter.MultiTileProcessor.storeCoord(MultiTileProcessor.java:497) at uk.me.parabola.splitter.MultiTileProcessor.processNode(MultiTileProcessor.java:126) at uk.me.parabola.splitter.AbstractMapProcessor.consume(AbstractMapProcessor.java:82) at uk.me.parabola.splitter.OSMFileHandler.execute(OSMFileHandler.java:157) at uk.me.parabola.splitter.ProblemLists.calcMultiTileElements(ProblemLists.java:255) at uk.me.parabola.splitter.Main.useProblemLists(Main.java:503) at uk.me.parabola.splitter.Main.start(Main.java:127) at uk.me.parabola.splitter.Main.main(Main.java:81) The option to first join the files with osmconvert - is a PITA - as you cannot parse several osm.pbf files to osmconvert (it only supports joining osm or o5m files). The option to supply splitter.jar with each file on it's own - is also a PITA that would require horrendous for loops because after each split you need to check which new map-id to use for the next file! So it would be great to have a keep-complete option that for each input file, but not for all input files, uses keep-complete. Actually I think this should be the default mode when supplying several input files. Even if mkgmap actually cannot run this in parallel / multithreading it would be way better than parsing each file on it's own... Or am I dumbstruck in finding a way to resplit too big input files? -- Felix Hartman - Openmtbmap.org & VeloMap.org
okay - well the only working solution to this problem is to use o5m output format with mkgmap, and then use osmconvert to join the files before splitting them again. Would it be possible for mkgmap to have a mode to split the files one by one without the detour via osmconvert? o5m uses a lot more hdd space, and mkgmap.jar is only marginally faster with 05m input vs osm.pbf (splitter works quite a lot faster on big o5m files vs osm.pbf - but small input files make not much time difference)... On Mon, 30 Aug 2021 at 02:25, Felix Hartmann <extremecarver@gmail.com> wrote:
I split france.osm.pbf with D:\openmtbmap\maps>if france == legend start /belownormal /b /wait java -jar -XX:+AggressiveHeap -Xms1000m -Xmx3800m C:\openmtbmap\splitter.jar --max-nodes=9600000 --max-threads=12 --search-limit=1000000 --output=pbf "--keep-complete" --max-areas=1024 --geonames-file=D:\openmtbmap\osmpbf_geofabrik\cities5000.zip --description=france --mapid=83910000 france.osm 1>NUL
Then I compiled the map using mgkamp.jar - and want to resplit all remaining files that did not compile with smaller max-nodes value. However splitter.jar chokes if I pass several files... Is there any way to have splitter just working with several files without trying to merge them? I just need the files split to the new max-nodes value and incrementing the map id. If I pass only one file at a time, it is a PITA to script this (is already hard enough figuring out the new map-id and creating a list of the tiles).
D:\openmtbmap\maps>if yes EQU yes start /belownormal /b /wait java -XX:+AggressiveHeap -Xms5000M -Xmx54000M -jar C:\openmtbmap\splitter.jar --max-nodes=4800000 --max-threads=12 --search-limit=1000000 --output=pbf --geonames-file=D:\openmtbmap\osmpbf_geofabrik\cities5000.zip --description=france --mapid=83910046 "83910015.osm.pbf" "83910025.osm.pbf" "83910026.osm.pbf" "83910042.osm.pbf" 1>NUL Warning: --keep-complete is only used for the first input file. Further files must use higher ids. Error: Node ids are not sorted. Use e.g. osmosis to sort the input data. This is not supported with keep-complete=true or --problem-list uk.me.parabola.splitter.SplitFailedException: Node ids are not sorted at uk.me.parabola.splitter.MultiTileProcessor.storeCoord(MultiTileProcessor.java:497) at uk.me.parabola.splitter.MultiTileProcessor.processNode(MultiTileProcessor.java:126) at uk.me.parabola.splitter.AbstractMapProcessor.consume(AbstractMapProcessor.java:82) at uk.me.parabola.splitter.OSMFileHandler.execute(OSMFileHandler.java:157) at uk.me.parabola.splitter.ProblemLists.calcMultiTileElements(ProblemLists.java:255) at uk.me.parabola.splitter.Main.useProblemLists(Main.java:503) at uk.me.parabola.splitter.Main.start(Main.java:127) at uk.me.parabola.splitter.Main.main(Main.java:81)
The option to first join the files with osmconvert - is a PITA - as you cannot parse several osm.pbf files to osmconvert (it only supports joining osm or o5m files). The option to supply splitter.jar with each file on it's own - is also a PITA that would require horrendous for loops because after each split you need to check which new map-id to use for the next file!
So it would be great to have a keep-complete option that for each input file, but not for all input files, uses keep-complete. Actually I think this should be the default mode when supplying several input files. Even if mkgmap actually cannot run this in parallel / multithreading it would be way better than parsing each file on it's own... Or am I dumbstruck in finding a way to resplit too big input files?
-- Felix Hartman - Openmtbmap.org & VeloMap.org
-- Felix Hartman - Openmtbmap.org & VeloMap.org
Well I'm not fully sure this works as intended - though at first glance I cannot find a problem - but running mkgmap on the split files gives an: C:\openmtbmap\maps>start /belownormal /b /wait java -jar -XX:+AggressiveHeap -XX:StringTableSize=1000003 -Xms5000M -Xmx43000M C:\openmtbmap\mkgmap.jar --max-jobs=12 --order-by-decreasing-area --code-page=1252 "--style-file=C:\openmtbmap\buildings_style" --levels="0:24, 1:23, 2:22, 3:21" --ignore-osm-bounds --ignore-turn-restrictions --merge-lines --allow-reverse-merge --transparent --draw-priority=28 --add-pois-to-areas --simplify-polygons=23:4,22:7,21:8 --copyright-file="C:\openmtbmap\openmtbmap_svn\copyrightopm.txt" --license-file="C:\openmtbmap\openmtbmap_svn\licenseopm.txt" --add-boundary-nodes-at-admin-boundaries=2 --poi-excl-index=0x6405,0x4316,0x2f00 --ignore-fixme-values --polygon-size-limits="24:16, 23:14, 22:12, 21:11" --ignore-turn-restrictions --description=buildings_fr --country-abbr=fr --country-name=france --mapname=83910046 --family-id=8391 --product-id=1 --series-name="buildings_france_30.08.2021" --keep-going --family-name="buildings_fr_30.08.2021" --tdbfile --gmapi --gmapsupp --overview-mapname=mapsetb --area-name="france_30.08.2021_buildings" -c D:\openmtbmap\maps\template.franceb1 8391*.img buildfr.typ 1>NUL Mkgmap version 4806M Time started: Mon Aug 30 12:34:48 CEST 2021 Number of MapFailedExceptions: 0 WARNING (global): Could not copy 83910046.RGN uk.me.parabola.imgfmt.FileExistsException: File 83910046.RGN already exists WARNING (global): Could not copy 83910046.TRE uk.me.parabola.imgfmt.FileExistsException: File 83910046.TRE already exists WARNING (global): Could not copy 83910046.LBL uk.me.parabola.imgfmt.FileExistsException: File 83910046.LBL already exists Number of ExitExceptions: 0 Time finished: Mon Aug 30 12:35:28 CEST 2021 Total time taken: 39 seconds Now of course there has been no file at all called 83910046* in the directory except 83910046.o5m (referenced via D:\openmtbmap\maps\template.franceb1 input file option). Google spits out no mention at all concerning this error. I would guess there is an error in the --gmapi file? On Mon, 30 Aug 2021 at 12:38, Felix Hartmann <extremecarver@gmail.com> wrote:
okay - well the only working solution to this problem is to use o5m output format with mkgmap, and then use osmconvert to join the files before splitting them again. Would it be possible for mkgmap to have a mode to split the files one by one without the detour via osmconvert?
o5m uses a lot more hdd space, and mkgmap.jar is only marginally faster with 05m input vs osm.pbf (splitter works quite a lot faster on big o5m files vs osm.pbf - but small input files make not much time difference)...
On Mon, 30 Aug 2021 at 02:25, Felix Hartmann <extremecarver@gmail.com> wrote:
I split france.osm.pbf with D:\openmtbmap\maps>if france == legend start /belownormal /b /wait java -jar -XX:+AggressiveHeap -Xms1000m -Xmx3800m C:\openmtbmap\splitter.jar --max-nodes=9600000 --max-threads=12 --search-limit=1000000 --output=pbf "--keep-complete" --max-areas=1024 --geonames-file=D:\openmtbmap\osmpbf_geofabrik\cities5000.zip --description=france --mapid=83910000 france.osm 1>NUL
Then I compiled the map using mgkamp.jar - and want to resplit all remaining files that did not compile with smaller max-nodes value. However splitter.jar chokes if I pass several files... Is there any way to have splitter just working with several files without trying to merge them? I just need the files split to the new max-nodes value and incrementing the map id. If I pass only one file at a time, it is a PITA to script this (is already hard enough figuring out the new map-id and creating a list of the tiles).
D:\openmtbmap\maps>if yes EQU yes start /belownormal /b /wait java -XX:+AggressiveHeap -Xms5000M -Xmx54000M -jar C:\openmtbmap\splitter.jar --max-nodes=4800000 --max-threads=12 --search-limit=1000000 --output=pbf --geonames-file=D:\openmtbmap\osmpbf_geofabrik\cities5000.zip --description=france --mapid=83910046 "83910015.osm.pbf" "83910025.osm.pbf" "83910026.osm.pbf" "83910042.osm.pbf" 1>NUL Warning: --keep-complete is only used for the first input file. Further files must use higher ids. Error: Node ids are not sorted. Use e.g. osmosis to sort the input data. This is not supported with keep-complete=true or --problem-list uk.me.parabola.splitter.SplitFailedException: Node ids are not sorted at uk.me.parabola.splitter.MultiTileProcessor.storeCoord(MultiTileProcessor.java:497) at uk.me.parabola.splitter.MultiTileProcessor.processNode(MultiTileProcessor.java:126) at uk.me.parabola.splitter.AbstractMapProcessor.consume(AbstractMapProcessor.java:82) at uk.me.parabola.splitter.OSMFileHandler.execute(OSMFileHandler.java:157) at uk.me.parabola.splitter.ProblemLists.calcMultiTileElements(ProblemLists.java:255) at uk.me.parabola.splitter.Main.useProblemLists(Main.java:503) at uk.me.parabola.splitter.Main.start(Main.java:127) at uk.me.parabola.splitter.Main.main(Main.java:81)
The option to first join the files with osmconvert - is a PITA - as you cannot parse several osm.pbf files to osmconvert (it only supports joining osm or o5m files). The option to supply splitter.jar with each file on it's own - is also a PITA that would require horrendous for loops because after each split you need to check which new map-id to use for the next file!
So it would be great to have a keep-complete option that for each input file, but not for all input files, uses keep-complete. Actually I think this should be the default mode when supplying several input files. Even if mkgmap actually cannot run this in parallel / multithreading it would be way better than parsing each file on it's own... Or am I dumbstruck in finding a way to resplit too big input files?
-- Felix Hartman - Openmtbmap.org & VeloMap.org
-- Felix Hartman - Openmtbmap.org & VeloMap.org
-- Felix Hartman - Openmtbmap.org & VeloMap.org
Sorry - the last error was a bug in my script. I actually chose the mapid for splitter too low and while mkgmap did overwrite the .img file without warning, it did not overwrite the gmapi files without warning (due to a bug in finding out the highest number of tiles created). So with o5m input this all works fine. Seems I have to beef up my server by another NVME disk to accommodate for the larger o5m vs osm.pbf files. I need to store all splitter created files - because I run mkgmap several times... Just convenience. On Mon, 30 Aug 2021 at 12:50, Felix Hartmann <extremecarver@gmail.com> wrote:
Well I'm not fully sure this works as intended - though at first glance I cannot find a problem - but running mkgmap on the split files gives an:
C:\openmtbmap\maps>start /belownormal /b /wait java -jar -XX:+AggressiveHeap -XX:StringTableSize=1000003 -Xms5000M -Xmx43000M C:\openmtbmap\mkgmap.jar --max-jobs=12 --order-by-decreasing-area --code-page=1252 "--style-file=C:\openmtbmap\buildings_style" --levels="0:24, 1:23, 2:22, 3:21" --ignore-osm-bounds --ignore-turn-restrictions --merge-lines --allow-reverse-merge --transparent --draw-priority=28 --add-pois-to-areas --simplify-polygons=23:4,22:7,21:8 --copyright-file="C:\openmtbmap\openmtbmap_svn\copyrightopm.txt" --license-file="C:\openmtbmap\openmtbmap_svn\licenseopm.txt" --add-boundary-nodes-at-admin-boundaries=2 --poi-excl-index=0x6405,0x4316,0x2f00 --ignore-fixme-values --polygon-size-limits="24:16, 23:14, 22:12, 21:11" --ignore-turn-restrictions --description=buildings_fr --country-abbr=fr --country-name=france --mapname=83910046 --family-id=8391 --product-id=1 --series-name="buildings_france_30.08.2021" --keep-going --family-name="buildings_fr_30.08.2021" --tdbfile --gmapi --gmapsupp --overview-mapname=mapsetb --area-name="france_30.08.2021_buildings" -c D:\openmtbmap\maps\template.franceb1 8391*.img buildfr.typ 1>NUL Mkgmap version 4806M Time started: Mon Aug 30 12:34:48 CEST 2021 Number of MapFailedExceptions: 0 WARNING (global): Could not copy 83910046.RGN uk.me.parabola.imgfmt.FileExistsException: File 83910046.RGN already exists WARNING (global): Could not copy 83910046.TRE uk.me.parabola.imgfmt.FileExistsException: File 83910046.TRE already exists WARNING (global): Could not copy 83910046.LBL uk.me.parabola.imgfmt.FileExistsException: File 83910046.LBL already exists Number of ExitExceptions: 0 Time finished: Mon Aug 30 12:35:28 CEST 2021 Total time taken: 39 seconds
Now of course there has been no file at all called 83910046* in the directory except 83910046.o5m (referenced via D:\openmtbmap\maps\template.franceb1 input file option). Google spits out no mention at all concerning this error. I would guess there is an error in the --gmapi file?
On Mon, 30 Aug 2021 at 12:38, Felix Hartmann <extremecarver@gmail.com> wrote:
okay - well the only working solution to this problem is to use o5m output format with mkgmap, and then use osmconvert to join the files before splitting them again. Would it be possible for mkgmap to have a mode to split the files one by one without the detour via osmconvert?
o5m uses a lot more hdd space, and mkgmap.jar is only marginally faster with 05m input vs osm.pbf (splitter works quite a lot faster on big o5m files vs osm.pbf - but small input files make not much time difference)...
On Mon, 30 Aug 2021 at 02:25, Felix Hartmann <extremecarver@gmail.com> wrote:
I split france.osm.pbf with D:\openmtbmap\maps>if france == legend start /belownormal /b /wait java -jar -XX:+AggressiveHeap -Xms1000m -Xmx3800m C:\openmtbmap\splitter.jar --max-nodes=9600000 --max-threads=12 --search-limit=1000000 --output=pbf "--keep-complete" --max-areas=1024 --geonames-file=D:\openmtbmap\osmpbf_geofabrik\cities5000.zip --description=france --mapid=83910000 france.osm 1>NUL
Then I compiled the map using mgkamp.jar - and want to resplit all remaining files that did not compile with smaller max-nodes value. However splitter.jar chokes if I pass several files... Is there any way to have splitter just working with several files without trying to merge them? I just need the files split to the new max-nodes value and incrementing the map id. If I pass only one file at a time, it is a PITA to script this (is already hard enough figuring out the new map-id and creating a list of the tiles).
D:\openmtbmap\maps>if yes EQU yes start /belownormal /b /wait java -XX:+AggressiveHeap -Xms5000M -Xmx54000M -jar C:\openmtbmap\splitter.jar --max-nodes=4800000 --max-threads=12 --search-limit=1000000 --output=pbf --geonames-file=D:\openmtbmap\osmpbf_geofabrik\cities5000.zip --description=france --mapid=83910046 "83910015.osm.pbf" "83910025.osm.pbf" "83910026.osm.pbf" "83910042.osm.pbf" 1>NUL Warning: --keep-complete is only used for the first input file. Further files must use higher ids. Error: Node ids are not sorted. Use e.g. osmosis to sort the input data. This is not supported with keep-complete=true or --problem-list uk.me.parabola.splitter.SplitFailedException: Node ids are not sorted at uk.me.parabola.splitter.MultiTileProcessor.storeCoord(MultiTileProcessor.java:497) at uk.me.parabola.splitter.MultiTileProcessor.processNode(MultiTileProcessor.java:126) at uk.me.parabola.splitter.AbstractMapProcessor.consume(AbstractMapProcessor.java:82) at uk.me.parabola.splitter.OSMFileHandler.execute(OSMFileHandler.java:157) at uk.me.parabola.splitter.ProblemLists.calcMultiTileElements(ProblemLists.java:255) at uk.me.parabola.splitter.Main.useProblemLists(Main.java:503) at uk.me.parabola.splitter.Main.start(Main.java:127) at uk.me.parabola.splitter.Main.main(Main.java:81)
The option to first join the files with osmconvert - is a PITA - as you cannot parse several osm.pbf files to osmconvert (it only supports joining osm or o5m files). The option to supply splitter.jar with each file on it's own - is also a PITA that would require horrendous for loops because after each split you need to check which new map-id to use for the next file!
So it would be great to have a keep-complete option that for each input file, but not for all input files, uses keep-complete. Actually I think this should be the default mode when supplying several input files. Even if mkgmap actually cannot run this in parallel / multithreading it would be way better than parsing each file on it's own... Or am I dumbstruck in finding a way to resplit too big input files?
-- Felix Hartman - Openmtbmap.org & VeloMap.org
-- Felix Hartman - Openmtbmap.org & VeloMap.org
-- Felix Hartman - Openmtbmap.org & VeloMap.org
-- Felix Hartman - Openmtbmap.org & VeloMap.org
Hi Felix, I did not yet try to unterstand what your goal is but it sounds wrong to merge files produced by spltter. This can only work well when the merged tiles form a rectangular area without gaps. Gerd ---- Felix Hartmann schrieb ---- Sorry - the last error was a bug in my script. I actually chose the mapid for splitter too low and while mkgmap did overwrite the .img file without warning, it did not overwrite the gmapi files without warning (due to a bug in finding out the highest number of tiles created). So with o5m input this all works fine. Seems I have to beef up my server by another NVME disk to accommodate for the larger o5m vs osm.pbf files. I need to store all splitter created files - because I run mkgmap several times... Just convenience. On Mon, 30 Aug 2021 at 12:50, Felix Hartmann <extremecarver@gmail.com<mailto:extremecarver@gmail.com>> wrote: Well I'm not fully sure this works as intended - though at first glance I cannot find a problem - but running mkgmap on the split files gives an: C:\openmtbmap\maps>start /belownormal /b /wait java -jar -XX:+AggressiveHeap -XX:StringTableSize=1000003 -Xms5000M -Xmx43000M C:\openmtbmap\mkgmap.jar --max-jobs=12 --order-by-decreasing-area --code-page=1252 "--style-file=C:\openmtbmap\buildings_style" --levels="0:24, 1:23, 2:22, 3:21" --ignore-osm-bounds --ignore-turn-restrictions --merge-lines --allow-reverse-merge --transparent --draw-priority=28 --add-pois-to-areas --simplify-polygons=23:4,22:7,21:8 --copyright-file="C:\openmtbmap\openmtbmap_svn\copyrightopm.txt" --license-file="C:\openmtbmap\openmtbmap_svn\licenseopm.txt" --add-boundary-nodes-at-admin-boundaries=2 --poi-excl-index=0x6405,0x4316,0x2f00 --ignore-fixme-values --polygon-size-limits="24:16, 23:14, 22:12, 21:11" --ignore-turn-restrictions --description=buildings_fr --country-abbr=fr --country-name=france --mapname=83910046 --family-id=8391 --product-id=1 --series-name="buildings_france_30.08.2021" --keep-going --family-name="buildings_fr_30.08.2021" --tdbfile --gmapi --gmapsupp --overview-mapname=mapsetb --area-name="france_30.08.2021_buildings" -c D:\openmtbmap\maps\template.franceb1 8391*.img buildfr.typ 1>NUL Mkgmap version 4806M Time started: Mon Aug 30 12:34:48 CEST 2021 Number of MapFailedExceptions: 0 WARNING (global): Could not copy 83910046.RGN uk.me.parabola.imgfmt.FileExistsException: File 83910046.RGN already exists WARNING (global): Could not copy 83910046.TRE uk.me.parabola.imgfmt.FileExistsException: File 83910046.TRE already exists WARNING (global): Could not copy 83910046.LBL uk.me.parabola.imgfmt.FileExistsException: File 83910046.LBL already exists Number of ExitExceptions: 0 Time finished: Mon Aug 30 12:35:28 CEST 2021 Total time taken: 39 seconds Now of course there has been no file at all called 83910046* in the directory except 83910046.o5m (referenced via D:\openmtbmap\maps\template.franceb1 input file option). Google spits out no mention at all concerning this error. I would guess there is an error in the --gmapi file? On Mon, 30 Aug 2021 at 12:38, Felix Hartmann <extremecarver@gmail.com<mailto:extremecarver@gmail.com>> wrote: okay - well the only working solution to this problem is to use o5m output format with mkgmap, and then use osmconvert to join the files before splitting them again. Would it be possible for mkgmap to have a mode to split the files one by one without the detour via osmconvert? o5m uses a lot more hdd space, and mkgmap.jar is only marginally faster with 05m input vs osm.pbf (splitter works quite a lot faster on big o5m files vs osm.pbf - but small input files make not much time difference)... On Mon, 30 Aug 2021 at 02:25, Felix Hartmann <extremecarver@gmail.com<mailto:extremecarver@gmail.com>> wrote: I split france.osm.pbf with D:\openmtbmap\maps>if france == legend start /belownormal /b /wait java -jar -XX:+AggressiveHeap -Xms1000m -Xmx3800m C:\openmtbmap\splitter.jar --max-nodes=9600000 --max-threads=12 --search-limit=1000000 --output=pbf "--keep-complete" --max-areas=1024 --geonames-file=D:\openmtbmap\osmpbf_geofabrik\cities5000.zip --description=france --mapid=83910000 france.osm 1>NUL Then I compiled the map using mgkamp.jar - and want to resplit all remaining files that did not compile with smaller max-nodes value. However splitter.jar chokes if I pass several files... Is there any way to have splitter just working with several files without trying to merge them? I just need the files split to the new max-nodes value and incrementing the map id. If I pass only one file at a time, it is a PITA to script this (is already hard enough figuring out the new map-id and creating a list of the tiles). D:\openmtbmap\maps>if yes EQU yes start /belownormal /b /wait java -XX:+AggressiveHeap -Xms5000M -Xmx54000M -jar C:\openmtbmap\splitter.jar --max-nodes=4800000 --max-threads=12 --search-limit=1000000 --output=pbf --geonames-file=D:\openmtbmap\osmpbf_geofabrik\cities5000.zip --description=france --mapid=83910046 "83910015.osm.pbf" "83910025.osm.pbf" "83910026.osm.pbf" "83910042.osm.pbf" 1>NUL Warning: --keep-complete is only used for the first input file. Further files must use higher ids. Error: Node ids are not sorted. Use e.g. osmosis to sort the input data. This is not supported with keep-complete=true or --problem-list uk.me.parabola.splitter.SplitFailedException: Node ids are not sorted at uk.me.parabola.splitter.MultiTileProcessor.storeCoord(MultiTileProcessor.java:497) at uk.me.parabola.splitter.MultiTileProcessor.processNode(MultiTileProcessor.java:126) at uk.me.parabola.splitter.AbstractMapProcessor.consume(AbstractMapProcessor.java:82) at uk.me.parabola.splitter.OSMFileHandler.execute(OSMFileHandler.java:157) at uk.me.parabola.splitter.ProblemLists.calcMultiTileElements(ProblemLists.java:255) at uk.me.parabola.splitter.Main.useProblemLists(Main.java:503) at uk.me.parabola.splitter.Main.start(Main.java:127) at uk.me.parabola.splitter.Main.main(Main.java:81) The option to first join the files with osmconvert - is a PITA - as you cannot parse several osm.pbf files to osmconvert (it only supports joining osm or o5m files). The option to supply splitter.jar with each file on it's own - is also a PITA that would require horrendous for loops because after each split you need to check which new map-id to use for the next file! So it would be great to have a keep-complete option that for each input file, but not for all input files, uses keep-complete. Actually I think this should be the default mode when supplying several input files. Even if mkgmap actually cannot run this in parallel / multithreading it would be way better than parsing each file on it's own... Or am I dumbstruck in finding a way to resplit too big input files? -- Felix Hartman - Openmtbmap.org & VeloMap.org -- Felix Hartman - Openmtbmap.org & VeloMap.org -- Felix Hartman - Openmtbmap.org & VeloMap.org -- Felix Hartman - Openmtbmap.org & VeloMap.org
Hi My understanding of Felix's problem and a possible enhancement to solve it: Splitting a country... into a number of .osm.pfb files. Almost all of these are processable by mkgmap to produce tiles, but one or two exceed some .img limits. Rather that resplitting the whole country with a lower --max-nodes and trying again: Run the splitter again using the --split-file=area.list from the previous run (maybe with the good tiles deleted) and a new option that forces generation of 2 .osm.pbf files for each area. Probably just split in half, with some appropriate mapname augmentation. mkgmap should now be able to produce pairs of tiles to fill in the holes where the first run failed. Ticker
yes - exactly what Ticker wrote is my intention. It seems fine using o5m and merging with osmconvert however for right now.... If it's only a single tile no problems, but if its 2-X files - then you can only use keep-complete=false - or instead of using osm.pbf output in splitter o5m, and then merge the failed o5m tiles into a new input2.o5m and process that one again with splitter and lower max-nodes value. Instead if several input files are given and keep-complete=true (or default which is true) - then use keep-complete for each input file only. I do run into another problem there on the huge tiles, will have to check on that later. Seems to be general about maps with very large tiles and not related to splitter. On Tue, 31 Aug 2021 at 09:52, Ticker Berkin <rwb-mkgmap@jagit.co.uk> wrote:
Hi
My understanding of Felix's problem and a possible enhancement to solve it:
Splitting a country... into a number of .osm.pfb files. Almost all of these are processable by mkgmap to produce tiles, but one or two exceed some .img limits.
Rather that resplitting the whole country with a lower --max-nodes and trying again:
Run the splitter again using the --split-file=area.list from the previous run (maybe with the good tiles deleted) and a new option that forces generation of 2 .osm.pbf files for each area. Probably just split in half, with some appropriate mapname augmentation. mkgmap should now be able to produce pairs of tiles to fill in the holes where the first run failed.
Ticker
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- Felix Hartman - Openmtbmap.org & VeloMap.org
Also do not understand and on vacation now, so my options for understanding are very limited. But fwiw... Mkgmap should read files collection from contents files. One o5m file missing or 1 file extra (having different FID) is a show stopper. (Missing files is not tested, assumption only). I create 2-6 different maps each overnight from same dataset by simply: - Rename all o5m files by FID - Rename Reference in two content files by filename (changed by FID). This procedure takes seconds to run. All scripts (power management, splitter, OsmConvert, mkgmap, NSI) are executed by shell.run from within a "wrapper application". VB exe is: flexible, configurable, unlimited procedures and language elements. App UI checkboxes will select which datasets to download. App UI checkboxes will select which maps to be created. UI settings are saved in iNI file for Unattended execution. App will decide which datasets need to be merged by OsmConvert. App will check preconditions. App will run preprocess procedures (f.i. Rename by FID). App will wait for recent "latest" dataset. App will retry a failed download for n-times. App will decide what to do if a dataset is missing or outdated. App will verify each step to be successful (or not). App will install correctly created maps (read log files) into BaseCamp. Installer exe files and log files will be renamed by date (history, back-up and review options). App will run by Windows Scheduled Tasks. Recreating a specific map (because of style change) is as simple as clicking checkboxes. Time investment for mkgmap and Mapillary: one weekend development. Payback period in 2021 is already passed. Not by lead time (although Rename is faster than Merge) but most impressive by needed attention time. Don't shut down PC and tomorrow my new maps have been created somewhere between 06:00 and 09:00 AM or they failed. A one time problem is resolved by a new procedure for next occurrence. One time problems don't exist 😝 AnkEric (Eric) On Tue, Aug 31, 2021, 10:15 Felix Hartmann <extremecarver@gmail.com> wrote:
yes - exactly what Ticker wrote is my intention. It seems fine using o5m and merging with osmconvert however for right now.... If it's only a single tile no problems, but if its 2-X files - then you can only use keep-complete=false - or instead of using osm.pbf output in splitter o5m, and then merge the failed o5m tiles into a new input2.o5m and process that one again with splitter and lower max-nodes value. Instead if several input files are given and keep-complete=true (or default which is true) - then use keep-complete for each input file only.
I do run into another problem there on the huge tiles, will have to check on that later. Seems to be general about maps with very large tiles and not related to splitter.
On Tue, 31 Aug 2021 at 09:52, Ticker Berkin <rwb-mkgmap@jagit.co.uk> wrote:
Hi
My understanding of Felix's problem and a possible enhancement to solve it:
Splitting a country... into a number of .osm.pfb files. Almost all of these are processable by mkgmap to produce tiles, but one or two exceed some .img limits.
Rather that resplitting the whole country with a lower --max-nodes and trying again:
Run the splitter again using the --split-file=area.list from the previous run (maybe with the good tiles deleted) and a new option that forces generation of 2 .osm.pbf files for each area. Probably just split in half, with some appropriate mapname augmentation. mkgmap should now be able to produce pairs of tiles to fill in the holes where the first run failed.
Ticker
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- Felix Hartman - Openmtbmap.org & VeloMap.org
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
That step is very easy. You feed %FID% (if you use fid for first 4 numbers of mapid) and the new template.args to mkgmap.jar. No need to recreate the .img files that are okay. The broken ones - all 0byte ones - just delete them. If you wanna run mkgmap again on the same o5m input files - you join the template.args file from both runs - I always replace mapname with #notused so the template.args only contains description and the actual input file. It throws a warning that input files are missing - but that is allright. Of course you need to add in your script to delete those input files that were resplit. I don't know what your "app" does - but the process is pretty simple. As long as splitter can handle the several input files with --keep-complete (and just runs keep-complete on each input file but not on the totality of input files, plus does not try to join data from two files). And yes as said - if you want to run splitter on each input file separately, that creates loads of work - I don't know how to script that efficiently. Oh yeah - and your script needs to read the highest number of the maps created - add 1, and run mkgmap again with that number. But yeah - scripting this takes some time. But it is understandable that mkgmap cannot call splitter.jar on it's own....The main thing in this is getting to run splitter on several input files with keep-complete valid for each tile on it's own. But yeah I think osmconvert 1000000.o5m 100001.o5m 100010.o5m -o=inputfile.o5m and then calling mkgmap with the higher mapid, and *already_created.img inputfile.o5m would suffice. So far I cannot see problems (as long as using keep-complete=true) - you can even do a third pass. You cannot do a third pass if you use keep-complete=false overlap=2000 because the overlap would create a huge mess (or better objects ending up twice in your map) On Tue, 31 Aug 2021 at 11:33, Eric Internet <ankeric.osm@gmail.com> wrote:
Also do not understand and on vacation now, so my options for understanding are very limited.
But fwiw...
Mkgmap should read files collection from contents files. One o5m file missing or 1 file extra (having different FID) is a show stopper. (Missing files is not tested, assumption only).
I create 2-6 different maps each overnight from same dataset by simply: - Rename all o5m files by FID - Rename Reference in two content files by filename (changed by FID). This procedure takes seconds to run.
All scripts (power management, splitter, OsmConvert, mkgmap, NSI) are executed by shell.run from within a "wrapper application". VB exe is: flexible, configurable, unlimited procedures and language elements.
App UI checkboxes will select which datasets to download. App UI checkboxes will select which maps to be created. UI settings are saved in iNI file for Unattended execution. App will decide which datasets need to be merged by OsmConvert. App will check preconditions. App will run preprocess procedures (f.i. Rename by FID). App will wait for recent "latest" dataset. App will retry a failed download for n-times. App will decide what to do if a dataset is missing or outdated. App will verify each step to be successful (or not). App will install correctly created maps (read log files) into BaseCamp. Installer exe files and log files will be renamed by date (history, back-up and review options). App will run by Windows Scheduled Tasks.
Recreating a specific map (because of style change) is as simple as clicking checkboxes.
Time investment for mkgmap and Mapillary: one weekend development. Payback period in 2021 is already passed. Not by lead time (although Rename is faster than Merge) but most impressive by needed attention time. Don't shut down PC and tomorrow my new maps have been created somewhere between 06:00 and 09:00 AM or they failed.
A one time problem is resolved by a new procedure for next occurrence. One time problems don't exist 😝
AnkEric (Eric)
On Tue, Aug 31, 2021, 10:15 Felix Hartmann <extremecarver@gmail.com> wrote:
yes - exactly what Ticker wrote is my intention. It seems fine using o5m and merging with osmconvert however for right now.... If it's only a single tile no problems, but if its 2-X files - then you can only use keep-complete=false - or instead of using osm.pbf output in splitter o5m, and then merge the failed o5m tiles into a new input2.o5m and process that one again with splitter and lower max-nodes value. Instead if several input files are given and keep-complete=true (or default which is true) - then use keep-complete for each input file only.
I do run into another problem there on the huge tiles, will have to check on that later. Seems to be general about maps with very large tiles and not related to splitter.
On Tue, 31 Aug 2021 at 09:52, Ticker Berkin <rwb-mkgmap@jagit.co.uk> wrote:
Hi
My understanding of Felix's problem and a possible enhancement to solve it:
Splitting a country... into a number of .osm.pfb files. Almost all of these are processable by mkgmap to produce tiles, but one or two exceed some .img limits.
Rather that resplitting the whole country with a lower --max-nodes and trying again:
Run the splitter again using the --split-file=area.list from the previous run (maybe with the good tiles deleted) and a new option that forces generation of 2 .osm.pbf files for each area. Probably just split in half, with some appropriate mapname augmentation. mkgmap should now be able to produce pairs of tiles to fill in the holes where the first run failed.
Ticker
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- Felix Hartman - Openmtbmap.org & VeloMap.org
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- Felix Hartman - Openmtbmap.org & VeloMap.org
In general I now got it fully working - the problem is if those merged tiles are split - and not split in "pairs" but say from two tiles that failed - which are not adjacent - 3 tiles are created (or 5) there sometimes will be a tile spanning another tile. This works as far as I can see in Basecamp/MapInstall/Mapsource - but gives strange impression when selecting the tiles in MapInstall/Mapsource. Have not yet tested it on device. So if there is no possibility for splitter to implement a multiple single input tile mode - I will have to resort to writing a big for loop calling splitter separately for each crashed tile on mkgmap.jar processing. It often can be prevented by using exactly 50% maxnodes size on resplit - but that is not reliable of course. Basically for each tile that is 0 byte, you need to split the original o5m, remove the mention to the input-file from the template.args file, after each run check for a new highest mapid (or if splitting with 50% maxnodes just add 3), always delete all 0byte files - and after all tiles are split run mkgmap.jar again (either on the existing .img files plus the new templates.args file (from joining all the templates.args now newly created - also required quite a bit of logic). It's scriptable - but really requires a lot of work. Would be great if splitter.jar has a multiple tile input method (even more if it could multithread that one as usual). If not the multithreading of course can be achieved by piping each of these broken tiles into a separate process for splitting. On Tue, 31 Aug 2021 at 12:23, Felix Hartmann <extremecarver@gmail.com> wrote:
That step is very easy.
You feed %FID% (if you use fid for first 4 numbers of mapid) and the new template.args to mkgmap.jar. No need to recreate the .img files that are okay. The broken ones - all 0byte ones - just delete them. If you wanna run mkgmap again on the same o5m input files - you join the template.args file from both runs - I always replace mapname with #notused so the template.args only contains description and the actual input file. It throws a warning that input files are missing - but that is allright. Of course you need to add in your script to delete those input files that were resplit.
I don't know what your "app" does - but the process is pretty simple. As long as splitter can handle the several input files with --keep-complete (and just runs keep-complete on each input file but not on the totality of input files, plus does not try to join data from two files). And yes as said - if you want to run splitter on each input file separately, that creates loads of work - I don't know how to script that efficiently. Oh yeah - and your script needs to read the highest number of the maps created - add 1, and run mkgmap again with that number. But yeah - scripting this takes some time. But it is understandable that mkgmap cannot call splitter.jar on it's own....The main thing in this is getting to run splitter on several input files with keep-complete valid for each tile on it's own. But yeah I think osmconvert 1000000.o5m 100001.o5m 100010.o5m -o=inputfile.o5m and then calling mkgmap with the higher mapid, and *already_created.img inputfile.o5m would suffice. So far I cannot see problems (as long as using keep-complete=true) - you can even do a third pass. You cannot do a third pass if you use keep-complete=false overlap=2000 because the overlap would create a huge mess (or better objects ending up twice in your map)
On Tue, 31 Aug 2021 at 11:33, Eric Internet <ankeric.osm@gmail.com> wrote:
Also do not understand and on vacation now, so my options for understanding are very limited.
But fwiw...
Mkgmap should read files collection from contents files. One o5m file missing or 1 file extra (having different FID) is a show stopper. (Missing files is not tested, assumption only).
I create 2-6 different maps each overnight from same dataset by simply: - Rename all o5m files by FID - Rename Reference in two content files by filename (changed by FID). This procedure takes seconds to run.
All scripts (power management, splitter, OsmConvert, mkgmap, NSI) are executed by shell.run from within a "wrapper application". VB exe is: flexible, configurable, unlimited procedures and language elements.
App UI checkboxes will select which datasets to download. App UI checkboxes will select which maps to be created. UI settings are saved in iNI file for Unattended execution. App will decide which datasets need to be merged by OsmConvert. App will check preconditions. App will run preprocess procedures (f.i. Rename by FID). App will wait for recent "latest" dataset. App will retry a failed download for n-times. App will decide what to do if a dataset is missing or outdated. App will verify each step to be successful (or not). App will install correctly created maps (read log files) into BaseCamp. Installer exe files and log files will be renamed by date (history, back-up and review options). App will run by Windows Scheduled Tasks.
Recreating a specific map (because of style change) is as simple as clicking checkboxes.
Time investment for mkgmap and Mapillary: one weekend development. Payback period in 2021 is already passed. Not by lead time (although Rename is faster than Merge) but most impressive by needed attention time. Don't shut down PC and tomorrow my new maps have been created somewhere between 06:00 and 09:00 AM or they failed.
A one time problem is resolved by a new procedure for next occurrence. One time problems don't exist 😝
AnkEric (Eric)
On Tue, Aug 31, 2021, 10:15 Felix Hartmann <extremecarver@gmail.com> wrote:
yes - exactly what Ticker wrote is my intention. It seems fine using o5m and merging with osmconvert however for right now.... If it's only a single tile no problems, but if its 2-X files - then you can only use keep-complete=false - or instead of using osm.pbf output in splitter o5m, and then merge the failed o5m tiles into a new input2.o5m and process that one again with splitter and lower max-nodes value. Instead if several input files are given and keep-complete=true (or default which is true) - then use keep-complete for each input file only.
I do run into another problem there on the huge tiles, will have to check on that later. Seems to be general about maps with very large tiles and not related to splitter.
On Tue, 31 Aug 2021 at 09:52, Ticker Berkin <rwb-mkgmap@jagit.co.uk> wrote:
Hi
My understanding of Felix's problem and a possible enhancement to solve it:
Splitting a country... into a number of .osm.pfb files. Almost all of these are processable by mkgmap to produce tiles, but one or two exceed some .img limits.
Rather that resplitting the whole country with a lower --max-nodes and trying again:
Run the splitter again using the --split-file=area.list from the previous run (maybe with the good tiles deleted) and a new option that forces generation of 2 .osm.pbf files for each area. Probably just split in half, with some appropriate mapname augmentation. mkgmap should now be able to produce pairs of tiles to fill in the holes where the first run failed.
Ticker
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- Felix Hartman - Openmtbmap.org & VeloMap.org
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- Felix Hartman - Openmtbmap.org & VeloMap.org
-- Felix Hartman - Openmtbmap.org & VeloMap.org
participants (4)
-
Eric Internet -
Felix Hartmann -
Gerd Petermann -
Ticker Berkin