Issues generating DEM
I have done some tests with BuildDEMFile (thanks Frank for providing it!), but with some problems. I have followed the procedure described by Gerd at [1]. What I have done is the following: 1-Try to generate a DEM for my Western-Sahara map with command wine ../../../../DEM/BuildDEMFile.exe --usedummydata=true -d 55132001/55132001.DEM --hgtpath=../../../../DEM/hgt/ --tre=55132001/55132001.TRE --dlon=0.00027761 Result: Daten aus der TRE-Datei '55132001/55132001.TRE' gelesen Höhen für -18° .. -17° / 20° .. 21° eingelesen ... 1 Zoomlevel erzeuge 31594 x 25790 interpolierte Höhenwerte für den Abstand 0,00027761° x 0,00027761°... Out of memory BuildDEMFile was using about 1 GB RAM when it stopped, while there was ~2 GB free available on system. How can I allocate more memory for BuildDEMFile? Western Sahara is very low mapped, so that a large area fits in a single img, but it covers 80 hgt files. Is there a size limit for DEM files and thus large areas should be split for img generation? 2-Try to generate a DEM for my region (Extremadura, Spain) with command wine ../../../../DEM/BuildDEMFile.exe --usedummydata=true -d 55113002/55113002.DEM --hgtpath=../../../../DEM/hgt/ --lastcolstd --tre=55113002/55113002.TRE --dlon=0.00027761 Result: Daten aus der TRE-Datei '55113002/55113002.TRE' gelesen Höhen für -8° .. -7° / 37° .. 38° eingelesen ... 1 Zoomlevel erzeuge 10560 x 9339 interpolierte Höhenwerte für den Abstand 0,00027761° x 0,00027761°... Daten 10560 x 9339, Min. 70, Max. 2577 für Zoomlevel 0 Value cannot be null. Parameter name: destinationArray What does it mean and how to fix it? 3-Try to generate a DEM for an small portion of my region (only 4 hgt files required) with the same command as previous Result: DEM created. [1] http://www.mkgmap.org.uk/pipermail/mkgmap-dev/2017q4/027176.html
I suppose, the process can't manage more memory. It runs on a Win32-machine? When you can tell me the size of the area (see areas.list), i will try to run it on my machine. If the memory is the reason, then you can only split this IMG area in smaller IMG's. The 2. issue seems to be an error in program. I need the area size to debug the program. Please look at the file areas.list (generated from splitter). Frank --- Diese E-Mail wurde von Avast Antivirus-Software auf Viren geprüft. https://www.avast.com/antivirus
El 01/11/17 a las 21:14, Frank Stinner escribió:
I suppose, the process can't manage more memory. It runs on a Win32-machine? It's a 32 Linux machine+wine. I think it works as a win XP machine. When you can tell me the size of the area (see areas.list), i will try to run it on my machine. 63240001: 960512,-808960 to 1292288,-401408 # : 20.610352,-17.358398 to 27.729492,-8.613281
If the memory is the reason, then you can only split this IMG area in smaller IMG's.
The 2. issue seems to be an error in program. I need the area size to debug the program. Please look at the file areas.list (generated from splitter).
63240001: 1765376,-354304 to 1888256,-215040 # : 37.880859,-7.602539 to 40.517578,-4.614258 Note in both cases it's a "fake" splitter run, as both maps fit in a single img. If it helps, you can get gmap folders from http://alternativaslibres.org/tmp/Extremadura.gmap.zip and http://alternativaslibres.org/tmp/Western-Sahara.gmap.zip
Frank
Sorry, there is a intern limitation of .NET. The maximal size of a "object" on a 64-bit-machine/windows is near by 2 GByte. That means in this case, we can have maximal about 532.000.000 height values. 31503 x 25646 = 807.925.938 is too much. You have to split the area in 2 tiles. This all is only valid, if your machine have enough memory. I'm not shure, but i believe on a 32-bit-machine/windows is the object-size only 1 GByte. By the way, a .NET-program should be run with linux/mono. I cannot test that, but a command-line program should be easy enough. Can you try that? At second, i have used your TRE file with this output: BuildDEMFile, Version vom 30.10.2017, Copyright © FSofT 2017 Daten aus der TRE-Datei '55113002.TRE' gelesen Höhen für -8° .. -7° / 37° .. 38° eingelesen .... Höhen für -5° .. -4° / 40° .. 41° eingelesen 1 Zoomlevel erzeuge 10560 x 9339 interpolierte Höhenwerte für den Abstand 0,00027761° x 0,00027761°... Daten 10560 x 9339, Min. 70, Max. 2531 für Zoomlevel 0 Kachelanzahl 165 x 146 (rechte Spalte 64 breit, untere Zeile 59 hoch) Rand links -7,56333589553833°, oben 40,498673915863° Pixelgröße 0,00027761° x 0,00027761° .... 24090 Kacheln encodiert It is mysterious. Frank --- Diese E-Mail wurde von Avast Antivirus-Software auf Viren geprüft. https://www.avast.com/antivirus
El 02/11/17 a las 18:01, Frank Stinner escribió:
Sorry, there is a intern limitation of .NET. The maximal size of a "object" on a 64-bit-machine/windows is near by 2 GByte. That means in this case, we can have maximal about 532.000.000 height values. 31503 x 25646 = 807.925.938 is too much. You have to split the area in 2 tiles. This all is only valid, if your machine have enough memory. I'm not shure, but i believe on a 32-bit-machine/windows is the object-size only 1 GByte. I have tried other maps and nearly reached 2 GB, so the limit seems to be the same as for 64-bit machines.
By the way, a .NET-program should be run with linux/mono. I cannot test that, but a command-line program should be easy enough. Can you try that? I don't have mono installed on my Linux machine, but wine seems to suffice by now. I tried BuildDEMFile.exe in a win XP virtual machine, but it refuses to work with "BuildDEMFile.exe is not a valid win32 application". Does it have any dependency I should install on the XP machine?
At second, i have used your TRE file with this output:
BuildDEMFile, Version vom 30.10.2017, Copyright © FSofT 2017 Daten aus der TRE-Datei '55113002.TRE' gelesen Höhen für -8° .. -7° / 37° .. 38° eingelesen .... Höhen für -5° .. -4° / 40° .. 41° eingelesen 1 Zoomlevel erzeuge 10560 x 9339 interpolierte Höhenwerte für den Abstand 0,00027761° x 0,00027761°... Daten 10560 x 9339, Min. 70, Max. 2531 für Zoomlevel 0 Kachelanzahl 165 x 146 (rechte Spalte 64 breit, untere Zeile 59 hoch) Rand links -7,56333589553833°, oben 40,498673915863° Pixelgröße 0,00027761° x 0,00027761° .... 24090 Kacheln encodiert
It is mysterious.
Frank
"BuildDEMFile.exe is not a valid win32 application"? Yes and no. It's a .NET program and need the .NET-library >= 4.5.2. The .NET-library ist part of a "normal" Windows. Perhaps the library ist not part of your virtual machine? Frank --- Diese E-Mail wurde von Avast Antivirus-Software auf Viren geprüft. https://www.avast.com/antivirus
Hi Frank, I have successfully build several maps with DEM, thanks again for your tool. I have tripped on out-of-memory problems too. One case is, when tile is too big. This I can deal with, simply preparing smaller tiles. I think the limit is about 200 degree square. Other problem is with preview maps. I can't make it smaller. What I can do is to make hgt files smaller, since there is no need for full resolution for preview. Unfortunately your program assumes, that dummy tile is always 1201x1201. So regardless of hgt size, memory is full anyway, since my preview maps include a lot of sea. Side note: isn't predefined size of dummy tile an error? What does happen, when I use 1-second hgt with size 3601x3601? It has worked for me, but maybe I was lucky? But if it works, then maybe use 3x3 size for dummy? I have recompiled BuildDEMFile, adding option for hgt line length (actually I have reused usedummydata for this purpose). So now I can use hgt files downsized to 76x76. This allow me to create very big preview maps, like for a full continent. I think BuildDEMFile could downsize hgt files on the fly, when there is big difference in DEM pixel size compared to hgt pixel. I hope you could consider addition of that kind of functionality. -- Best regards, Andrzej
Hi Andrzej, there is no predefined size for HGT's. I use 1201-HGT's mixed with 3601'HGT's. It should not to be a problem to use any other size. The only restriction is: height==widht. The memory consumption is a problem. A HGT with 1201x1201 need in memory 1201*1201*4 Byte = 5,5 MB (.NET use for short also 4 byte). 200 HGT's need 1,1 GB. Hard to estimate is the size for DEM-Data. The prog is compiled as "any cpu" and "prefer 32-bit". If "prefer 32-bit" not set, run the prog in 64-bit-win as 64-bit-prog and can use more memory. But it runs slower. If you self compiled the prog, unset the option and try to use a xml-file BuildDEMFile.exe.config like this: <?xml version="1.0" encoding="utf-8" ?> <configuration> <runtime> <gcAllowVeryLargeObjects enabled="true" /> </runtime> </configuration> But you are allright, downsizesing the HGT's "on the fly" should not to be a problem. Let me a little bit think about that. Frank --- Diese E-Mail wurde von Avast Antivirus-Software auf Viren geprüft. https://www.avast.com/antivirus
Hi Frank,
there is no predefined size for HGT's.
I mean this code in HGTReader.cs: if (!dummydataonerror) ... else { Minimum = Maximum = NOVALUE; data = new Int16[1201 * 1201]; for (int i = 0; i < data.Length; i++) data[i] = NOVALUE; NotValid = data.Length; } This is the part, that I have modified, so the size of "data" depends on input option. -- Best regards, Andrzej
Hi Andrzej, mea culpa. I have overlooked that. It's changed. There is a new option --changehgtsize=x. You can for special cases give a new size x. After reading a HGT runs a interpolation and set a new size. Hope that helps. Frank --- Diese E-Mail wurde von Avast Antivirus-Software auf Viren geprüft. https://www.avast.com/antivirus
Hi Frank, BildDEmFile from 26.11.2017 works without memory problems, but the DEM output is wrong, map doesn't display correctly in Mapsource or BaseCamp. I get wrong img, even without using added features, just creating a standard img from SRTM3 data. See attached picture of single tile in BaseCamp. -- Best regards, Andrzej
Hi Andrzej, is this a overviewmap? Can you send me TRE or the exact boundary for this tile and the your command line? I will see what's going wrong in debugger. Frank --- Diese E-Mail wurde von Avast Antivirus-Software auf Viren geprüft. https://www.avast.com/antivirus
Hi Frank Andrzej is correct I get the same for ANY map - I am not using --changehgtsize r Nick On 26/11/2017 17:18, Frank Stinner wrote:
Hi Andrzej,
is this a overviewmap? Can you send me TRE or the exact boundary for this tile and the your command line? I will see what's going wrong in debugger.
Frank
--- Diese E-Mail wurde von Avast Antivirus-Software auf Viren geprüft. https://www.avast.com/antivirus
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi all, a short test show me, that there is a change in DEM's since the version from 23.11. I will look for the error. Frank --- Diese E-Mail wurde von Avast Antivirus-Software auf Viren geprüft. https://www.avast.com/antivirus
participants (5)
-
Andrzej Popowski -
Carlos Dávila -
Carlos Dávila -
Frank Stinner -
osm＠pinns