Usage of DEM files within zip
Hi I struggled today with the DEM/hgt files in a zip. After downloading some extra and checking/rezipping finally my maps did not work anymore. I do not have the exact error anymore but it was clear: DEM HGT files not found Now it looks like if the ZIP does not have a folder but only the HGT files on the root of the zip then mkgmap does not recognize them If I rezip everything again and put the files within a subfolder first, it is oke. For me its fine, I solved 'my' issue, but if this is true it could better be in the manual or solved in code. Looks like this does not work: world-hgt.zip\N00E006.hgt world-hgt.zip\N00E007.hgt world-hgt.zip\N00E008.hgt And if I do it like this its oke world-hgt.zip\world-hgt\N00E006.hgt world-hgt.zip\world-hgt\N00E007.hgt world-hgt.zip\world-hgt\N00E008.hgt Kind regards, Joris Bo
Hi Joris, I cannot reproduce the problem. I've created a srtm3.zip containing two files N52W003.hgt and N53W003.hgt and --dem=c:\temp\srtm3.zip works fine. My assumption: Your world-hgt.zip file really contained all hgt files and got larger than maybe 2G and the zipfile class in java cannot process it. mkgmap tries to support the file formats which are provided by the various DEM providers. I think I once tested performance with a large zip containing many hgt files and it was worse compared to single zipped files. IIRC the better alternative was to compress the folder with the hgt files (not using zip but the OS methods) or to have each hgt file zipped. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Joris Bo <jorisbo@hotmail.com> Gesendet: Sonntag, 7. Juni 2020 20:45 An: Development list for mkgmap Betreff: [mkgmap-dev] Usage of DEM files within zip Hi I struggled today with the DEM/hgt files in a zip. After downloading some extra and checking/rezipping finally my maps did not work anymore. I do not have the exact error anymore but it was clear: DEM HGT files not found Now it looks like if the ZIP does not have a folder but only the HGT files on the root of the zip then mkgmap does not recognize them If I rezip everything again and put the files within a subfolder first, it is oke. For me its fine, I solved ‘my’ issue, but if this is true it could better be in the manual or solved in code. Looks like this does not work: world-hgt.zip\N00E006.hgt world-hgt.zip\N00E007.hgt world-hgt.zip\N00E008.hgt And if I do it like this its oke world-hgt.zip\world-hgt\N00E006.hgt world-hgt.zip\world-hgt\N00E007.hgt world-hgt.zip\world-hgt\N00E008.hgt Kind regards, Joris Bo
Thanks, i'll do some more testing. It was indeed a single zip with 17Gb hgt. Strange thing is that it for sure works with a 17Gigabyte single zip containing 26.000 mixed 1arc/3arc hgt files. (Windows 10) It still finds all files from New Sealand, to Zambia and Europe as long as it contains a subfolder in the zip. But don't bother, I did not take performance into account. And my tests where not that scientifically correct. Because of the huge sizes... I only tried only until it worked again. Kind regards Joris -----Oorspronkelijk bericht----- Van: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> Namens Gerd Petermann Verzonden: maandag 8 juni 2020 07:57 Aan: Development list for mkgmap <mkgmap-dev@lists.mkgmap.org.uk> Onderwerp: Re: [mkgmap-dev] Usage of DEM files within zip Hi Joris, I cannot reproduce the problem. I've created a srtm3.zip containing two files N52W003.hgt and N53W003.hgt and --dem=c:\temp\srtm3.zip works fine. My assumption: Your world-hgt.zip file really contained all hgt files and got larger than maybe 2G and the zipfile class in java cannot process it. mkgmap tries to support the file formats which are provided by the various DEM providers. I think I once tested performance with a large zip containing many hgt files and it was worse compared to single zipped files. IIRC the better alternative was to compress the folder with the hgt files (not using zip but the OS methods) or to have each hgt file zipped. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Joris Bo <jorisbo@hotmail.com> Gesendet: Sonntag, 7. Juni 2020 20:45 An: Development list for mkgmap Betreff: [mkgmap-dev] Usage of DEM files within zip Hi I struggled today with the DEM/hgt files in a zip. After downloading some extra and checking/rezipping finally my maps did not work anymore. I do not have the exact error anymore but it was clear: DEM HGT files not found Now it looks like if the ZIP does not have a folder but only the HGT files on the root of the zip then mkgmap does not recognize them If I rezip everything again and put the files within a subfolder first, it is oke. For me its fine, I solved 'my' issue, but if this is true it could better be in the manual or solved in code. Looks like this does not work: world-hgt.zip\N00E006.hgt world-hgt.zip\N00E007.hgt world-hgt.zip\N00E008.hgt And if I do it like this its oke world-hgt.zip\world-hgt\N00E006.hgt world-hgt.zip\world-hgt\N00E007.hgt world-hgt.zip\world-hgt\N00E008.hgt Kind regards, Joris Bo _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Joris, it probably depends on the tool that is used to create / update the zip file. Many programs cannot handle large zip files. I hope mkgmap / java has no such problems, else we may run into problems with precompiled bounds or sea. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Joris Bo <jorisbo@hotmail.com> Gesendet: Montag, 8. Juni 2020 11:21 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] Usage of DEM files within zip Thanks, i'll do some more testing. It was indeed a single zip with 17Gb hgt. Strange thing is that it for sure works with a 17Gigabyte single zip containing 26.000 mixed 1arc/3arc hgt files. (Windows 10) It still finds all files from New Sealand, to Zambia and Europe as long as it contains a subfolder in the zip. But don't bother, I did not take performance into account. And my tests where not that scientifically correct. Because of the huge sizes... I only tried only until it worked again. Kind regards Joris -----Oorspronkelijk bericht----- Van: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> Namens Gerd Petermann Verzonden: maandag 8 juni 2020 07:57 Aan: Development list for mkgmap <mkgmap-dev@lists.mkgmap.org.uk> Onderwerp: Re: [mkgmap-dev] Usage of DEM files within zip Hi Joris, I cannot reproduce the problem. I've created a srtm3.zip containing two files N52W003.hgt and N53W003.hgt and --dem=c:\temp\srtm3.zip works fine. My assumption: Your world-hgt.zip file really contained all hgt files and got larger than maybe 2G and the zipfile class in java cannot process it. mkgmap tries to support the file formats which are provided by the various DEM providers. I think I once tested performance with a large zip containing many hgt files and it was worse compared to single zipped files. IIRC the better alternative was to compress the folder with the hgt files (not using zip but the OS methods) or to have each hgt file zipped. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Joris Bo <jorisbo@hotmail.com> Gesendet: Sonntag, 7. Juni 2020 20:45 An: Development list for mkgmap Betreff: [mkgmap-dev] Usage of DEM files within zip Hi I struggled today with the DEM/hgt files in a zip. After downloading some extra and checking/rezipping finally my maps did not work anymore. I do not have the exact error anymore but it was clear: DEM HGT files not found Now it looks like if the ZIP does not have a folder but only the HGT files on the root of the zip then mkgmap does not recognize them If I rezip everything again and put the files within a subfolder first, it is oke. For me its fine, I solved 'my' issue, but if this is true it could better be in the manual or solved in code. Looks like this does not work: world-hgt.zip\N00E006.hgt world-hgt.zip\N00E007.hgt world-hgt.zip\N00E008.hgt And if I do it like this its oke world-hgt.zip\world-hgt\N00E006.hgt world-hgt.zip\world-hgt\N00E007.hgt world-hgt.zip\world-hgt\N00E008.hgt Kind regards, Joris Bo _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi I finished some testing on the DEM files in zip not found by mkgmap. It has something to do with the zip-tool used. When internal Windows 10 zip is used through explorer or vbscript, you can create ZIP’s of 17GB but only zips up to 2Gb can be read “by mkgmap”. Even when removing files from such big zip until it drops under the 2Gb does not help. As soon as it is re-created from scratch and smaller then 2Gb its oke again. Also tested with winzip 11 (pretty ancient version which I had licence for) and the usage of 7zip, and both work fine above 2Gb. Winzip was tested up to the 2,3Gb and 7zip up to 17Gb Pretty useless info, but then you know 😊 Hgt Files Tool Result Remark Zipped size 2 Windows 10 ok 2 7-zip 18.05 extension .7z not found 2 7-zip 18.05 extension .zip ok 2 VBA Script Windows 10 ok 1154 VBA Script Windows 10 not found From scratch 2,24 GB (2.407.959.231 bytes) 1154 Windows 10 not found From scratch 2,24 GB (2.407.959.231 bytes) 1094 Windows 10 not found Internal delete 1,93 GB (2.078.032.075 bytes) 1079 Windows 10 not found Internal delete 1,90 GB (2.047.599.717 bytes) 1063 Windows 10 not found Internal delete 1,87 GB (2.013.685.658 bytes) 1059 Windows 10 not found Internal delete 1,86 GB (2.007.054.183 bytes) 1052 Windows 10 not found Internal delete 1,84 GB (1.978.252.178 bytes) 1047 Windows 10 not found Internal delete 1,82 GB (1.960.539.426 bytes) 1019 Windows 10 not found Internal delete 1,73 GB (1.862.120.309 bytes) 1000 Windows 10 not found Internal delete 1,69 GB (1.819.411.992 bytes) 1000 Windows 10 ok Above unzipped then from scratch 1,76 GB (1.894.852.116 bytes) 1154 Winrar Warning before start > too big for zip 1154 Winzip 11 2006 ok 2,26 GB (2.436.489.540 bytes) -----Oorspronkelijk bericht----- Van: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> Namens Joris Bo Verzonden: maandag 8 juni 2020 11:21 Aan: Development list for mkgmap <mkgmap-dev@lists.mkgmap.org.uk> Onderwerp: Re: [mkgmap-dev] Usage of DEM files within zip Thanks, i'll do some more testing. It was indeed a single zip with 17Gb hgt. Strange thing is that it for sure works with a 17Gigabyte single zip containing 26.000 mixed 1arc/3arc hgt files. (Windows 10) It still finds all files from New Sealand, to Zambia and Europe as long as it contains a subfolder in the zip. But don't bother, I did not take performance into account. And my tests where not that scientifically correct. Because of the huge sizes... I only tried only until it worked again. Kind regards Joris -----Oorspronkelijk bericht----- Van: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk<mailto:mkgmap-dev-bounces@lists.mkgmap.org.uk>> Namens Gerd Petermann Verzonden: maandag 8 juni 2020 07:57 Aan: Development list for mkgmap <mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk>> Onderwerp: Re: [mkgmap-dev] Usage of DEM files within zip Hi Joris, I cannot reproduce the problem. I've created a srtm3.zip containing two files N52W003.hgt and N53W003.hgt and --dem=c:\temp\srtm3.zip works fine. My assumption: Your world-hgt.zip file really contained all hgt files and got larger than maybe 2G and the zipfile class in java cannot process it. mkgmap tries to support the file formats which are provided by the various DEM providers. I think I once tested performance with a large zip containing many hgt files and it was worse compared to single zipped files. IIRC the better alternative was to compress the folder with the hgt files (not using zip but the OS methods) or to have each hgt file zipped. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk<mailto:mkgmap-dev-bounces@lists.mkgmap.org.uk>> im Auftrag von Joris Bo <jorisbo@hotmail.com<mailto:jorisbo@hotmail.com>> Gesendet: Sonntag, 7. Juni 2020 20:45 An: Development list for mkgmap Betreff: [mkgmap-dev] Usage of DEM files within zip Hi I struggled today with the DEM/hgt files in a zip. After downloading some extra and checking/rezipping finally my maps did not work anymore. I do not have the exact error anymore but it was clear: DEM HGT files not found Now it looks like if the ZIP does not have a folder but only the HGT files on the root of the zip then mkgmap does not recognize them If I rezip everything again and put the files within a subfolder first, it is oke. For me its fine, I solved 'my' issue, but if this is true it could better be in the manual or solved in code. Looks like this does not work: world-hgt.zip\N00E006.hgt world-hgt.zip\N00E007.hgt world-hgt.zip\N00E008.hgt And if I do it like this its oke world-hgt.zip\world-hgt\N00E006.hgt world-hgt.zip\world-hgt\N00E007.hgt world-hgt.zip\world-hgt\N00E008.hgt Kind regards, Joris Bo _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
participants (2)
-
Gerd Petermann -
Joris Bo