Hi Gerd, thanks I assume there are no ways - and phyghtmap does not work correctly either on such a huge area - has maybe a 64GB limit which would be broken? or does not work on the poly file with two areas. I will run it again now against both single areas (one has finished already and looks okay - but that was the tiny area). The other area will need 15 days or so as phyghtmap on ubuntu is using 1 thread only (can use 2 threads - but that is only maybe 1% faster. if more threads just wasting memory but not being any faster). So I will wait and see in 2 weeks... osmconvert also needed like 60 minutes or so to put out: timestamp min: 2021-05-13T08:41:54Z timestamp max: 2021-05-13T08:41:54Z lon min: 44.9999999 lon max: 179.9991666 lat min: -11.0025000 lat max: 81.8573611 nodes: 26761028462 ways: 0 relations: 0 node id min: 10000000 node id max: 26771065386 so clearly no ways. On Sat, 29 May 2021 at 08:47, Gerd Petermann < gpetermann_muenchen@hotmail.com> wrote:
Hi Felix, the log shows "way Map is empty" so splitter doesn't find any ways in asia10m_.osm.pbf.
So, either the file doesn't contain ways or splitter fails to find them. Maybe use osmconvert --out-statistics asia10m_.osm.pbf to find out more.
Reg. performance when splitting huge files with --keep-complete use o5m format and --max-areas=4096 (or higher)
I'll have to check why splitter needs so much time in the "Problem-list-generator" passes. Without ways or relations there is no need to even execute that.
The asia.poly contains two rings. I think splitter can copy with that but maybe phyghtmap cannot? Try with two separate *.poly files
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Felix Hartmann <extremecarver@gmail.com> Gesendet: Freitag, 28. Mai 2021 19:29 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] Splitter only writes points, not ways on huge file?
Attached the full splitter.log - looks fine. Is it most likely that the error is on phyghtmap?
On Fri, 28 May 2021 at 13:12, Felix Hartmann <extremecarver@gmail.com <mailto:extremecarver@gmail.com>> wrote: running now again with resolution 13, I had tried if changing resolution to 12 would solve the problem - but is does not.
On Fri, 28 May 2021 at 13:11, Felix Hartmann <extremecarver@gmail.com <mailto:extremecarver@gmail.com>> wrote: Asia.poly is attached - I did not save the splitter.log - doing this right now (but will take some hours - splitting this huge file takes a lot of time)
The log starts with: Splitter version 597 compiled 2020-01-30T14:10:47+0000 boundary-tags=use-exclude-list cache= description= geonames-file= handle-element-version=remove ignore-osm-bounds=false keep-complete=true mapid=75400000 max-areas=2048 max-nodes=12000000 max-threads=7 mixed=false no-trim=true num-tiles= output=pbf output-dir= overlap=auto polygon-desc-file= polygon-file= precomp-sea= problem-file= problem-report= resolution=13 route-rel-values= search-limit=200000 split-file= status-freq=120 stop-after=dist wanted-admin-level=5 write-kml= Elapsed time: 0s Memory: Current 1012MB (7MB used, 1005MB free) Max 25000MB Time started: Fri May 28 12:06:59 CEST 2021 Map is being split for resolution 13: - area boundaries are aligned to 0x800 map units (0.0439453125 degrees) - areas are multiples of 0x800 map units wide and high Processing asia10m_.osm.pbf Bounding box -179.99990000000003 -13.011650000000001 179.99990000000003 84.52666 10,000,000 nodes parsed... id=19999999 20,000,000 nodes parsed... id=29999999 30,000,000 nodes parsed... id=39999999 40,000,000 nodes parsed... id=49999999 50,000,000 nodes parsed... id=59999999 60,000,000 nodes parsed... id=69999999 70,000,000 nodes parsed... id=79999999 80,000,000 nodes parsed... id=89999999 90,000,000 nodes parsed... id=99999999 100,000,000 nodes parsed... id=109999999
On Fri, 28 May 2021 at 12:25, Gerd Petermann < gpetermann_muenchen@hotmail.com<mailto:gpetermann_muenchen@hotmail.com>> wrote: Hi Felix,
the pbf file really contains only nodes. Can you attach the splitter log file and the asia.poly, please?
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk<mailto: mkgmap-dev-bounces@lists.mkgmap.org.uk>> im Auftrag von Felix Hartmann < extremecarver@gmail.com<mailto:extremecarver@gmail.com>> Gesendet: Freitag, 28. Mai 2021 11:18 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] Splitter only writes points, not ways on huge file?
Hi Gerd, sorry - fixed. I attached the file. I could also put the whole 60GB file for analysis. But do not know if you could download such a huge file or analyse it better. https://openmtbmap.org/75400407.osm.pbf
Felix
On Fri, 28 May 2021 at 09:03, Gerd Petermann < gpetermann_muenchen@hotmail.com<mailto:gpetermann_muenchen@hotmail.com
<mailto:gpetermann_muenchen@hotmail.com<mailto: gpetermann_muenchen@hotmail.com>>> wrote: Hi Felix,
the link to the pbf file seems to be wrong? I get an 404
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk<mailto: mkgmap-dev-bounces@lists.mkgmap.org.uk><mailto: mkgmap-dev-bounces@lists.mkgmap.org.uk<mailto: mkgmap-dev-bounces@lists.mkgmap.org.uk>>> im Auftrag von Felix Hartmann < extremecarver@gmail.com<mailto:extremecarver@gmail.com><mailto: extremecarver@gmail.com<mailto:extremecarver@gmail.com>>> Gesendet: Freitag, 28. Mai 2021 07:56 An: Development list for mkgmap Betreff: [mkgmap-dev] Splitter only writes points, not ways on huge file?
I used phyghtmap to create a huge file (60GB) of 10m equidistance osm.pbf contourlines data.
phyghtmap --jobs=2 --polygon=/home/contourlines/bounds/asia.poly --step=10 --no-zero-contour --void-range-max=-420 --output-prefix="Asia10m" --line-cat=500,100--start-node-id=10000000 --start-way-id=10000000 --source=view1,srtm1,view3,srtm3 --max-nodes-per-way=230 --max-nodes-per-tile=0 --pbf
I am not fully sure that this pbf file is correct, but I created it exactly the same way as smaller countries. Maybe its also phyghtmap going wrong. But I doubt it with phyghtmaps incremental approach of just constantly writing out data.
When splitting it with splitter - and opening up one of the osm.pbf files in JOSM - I noticed it contains only the contourline nodes, no lines. I think the Asia continent file spans 360 degrees - so maybe that causes problems with splitter?
I have uploaded a splitted file here: https://openmtbmap.org/75400407.osm.pbf If you load it in Josm, you will notice it is alll the points but no lines...
-- Felix Hartman - Openmtbmap.org & VeloMap.org
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk
<mailto:mkgmap-dev@lists.mkgmap.org.uk<mailto: mkgmap-dev@lists.mkgmap.org.uk>> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- Felix Hartman - Openmtbmap.org & VeloMap.org
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- Felix Hartman - Openmtbmap.org & VeloMap.org
-- Felix Hartman - Openmtbmap.org & VeloMap.org
-- Felix Hartman - Openmtbmap.org & VeloMap.org
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
-- Felix Hartman - Openmtbmap.org & VeloMap.org