Re: [mkgmap-dev] DEM subfile
A friend and I spend some time on research the DEM subfile too. While still not knowing the real data encoding, I think we found some additional information about the structure. In general a DEM subfile consists of a common header, a specific header (so far as all subfiles), a block of two data sections per zoom level and a block of descriptions for each zoom level: Header - Common - Specific Zoom level data block - Block 1 -- Data 1 -- Data 2 - Block 2 -- Data 1 -- Data 2 .... Zoom level description block - Description 1 - Description 2 There exist at least to versions of the specific header (0x25 byte - old version? - and 0x29 byte). The headers differ only slightly, the additional bytes are (always?) zero. Offset Length 0x00 0x15 common header 0x15 1 different values (MapSource crashes if second bit is set) 0x16 3 all 0 for 0x29 version, different for 0x25 version 0x19 2 number of zoom levels 0x1B 4 0 0x1F 2 size of single description (seen only 3c00 = 60 byte) 0x21 4 Offset of first description block (0x25) 4 All zero The single description blocks describe the structure and position of Data blocks. Offset Length 0x00 2 Index of description block, starting with 0000. This differs between both versions: While 0x25 version counts in high byte (0001), 0x29 version increments lower byte (0100). Maybe this has different meaning. 0x02 4 Unknown (seems to be always 40000000) 0x06 4 Unknown (seems to be always 40000000) 0x0A 4 Values in latitude dimension (vy) 0x0E 4 Values in longitude dimension (vx) 0x12 2 Unknown (always 0) 0x14 2 Block size in data section 1 (add 1) 0x16 2 Unknown (always 0) 0x18 2 Block count in data section 1 (add 1) 0x1A 4 Unknown value (Last byte not zero) 0x1E 2 Record size in data section 1 0x20 4 Offset of data section 1 0x24 4 Offset of data section 2 0x28 4 western boundary (I know this is unusual but non of the bytes has to be zero and the result still fits) 0x2C 4 northern boundary 0x30 4 Pixel resolution latitude (py) 0x34 4 Pixel resolution longitude (px) 0x38 4 Unknown Using these values the size of data section could be calculated as Block size * Block count * Record size Data section 1 could be read as having Block count blocks. Each of them containing Block size records of Record size. In our opinion the records don't hold offset to data section 2: They are often zero and not continuosly growing, which I would expect them to do. For the two lowest zoom levels the bounding box/size for the covered area could be calculated by (vx * px, vy * py) We don't have any glue what to do with data section 2. But this has to be some picture format, because through removing some layers you will get a lower resolution elevation shading in MapSource. Not knowing much about picture formats, data section 2 (in combination with data section 2) may be a header less GeoTiff or Jpeg (or whatever) Hope there's someone out with good ideas Ronny
Ronny Klier schrieb:
A friend and I spend some time on research the DEM subfile too. While still not knowing the real data encoding, I think we found some additional information about the structure.
thank you very much for sharing your results!
In general a DEM subfile consists of a common header, a specific header (so far as all subfiles), a block of two data sections per zoom level and a block of descriptions for each zoom level:
Header - Common - Specific Zoom level data block - Block 1 -- Data 1 -- Data 2 - Block 2 -- Data 1 -- Data 2 .... Zoom level description block - Description 1 - Description 2
There exist at least to versions of the specific header (0x25 byte - old version? - and 0x29 byte). The headers differ only slightly, the additional bytes are (always?) zero.
Offset Length 0x00 0x15 common header 0x15 1 different values (MapSource crashes if second bit is set) 0x16 3 all 0 for 0x29 version, different for 0x25 version 0x19 2 number of zoom levels 0x1B 4 0 0x1F 2 size of single description (seen only 3c00 = 60 byte) 0x21 4 Offset of first description block (0x25) 4 All zero
The single description blocks describe the structure and position of Data blocks.
Offset Length 0x00 2 Index of description block, starting with 0000. This differs between both versions: While 0x25 version counts in high byte (0001), 0x29 version increments lower byte (0100). Maybe this has different meaning. 0x02 4 Unknown (seems to be always 40000000) 0x06 4 Unknown (seems to be always 40000000) 0x0A 4 Values in latitude dimension (vy) 0x0E 4 Values in longitude dimension (vx) 0x12 2 Unknown (always 0) 0x14 2 Block size in data section 1 (add 1) 0x16 2 Unknown (always 0) 0x18 2 Block count in data section 1 (add 1) 0x1A 4 Unknown value (Last byte not zero) 0x1E 2 Record size in data section 1 0x20 4 Offset of data section 1 0x24 4 Offset of data section 2 0x28 4 western boundary (I know this is unusual but non of the bytes has to be zero and the result still fits)
In the files I analyzed the western/northern boundaries are coded with 3 bytes (as usual) and the first byte of these fields look like a separate field to me. Or are you saying the map unit is 360/232 in the DEM file?
0x2C 4 northern boundary 0x30 4 Pixel resolution latitude (py) 0x34 4 Pixel resolution longitude (px)
Are you sure these values are four bytes long? 0x38 2 Minimum height in feet? (0x8e18=-29160 for files with sea/-8888m) 0x3A 2 Maximum height in feet? (always larger than the previous fields)
Using these values the size of data section could be calculated as Block size * Block count * Record size
Data section 1 could be read as having Block count blocks. Each of them containing Block size records of Record size. In our opinion the records don't hold offset to data section 2: They are often zero and not continuosly growing, which I would expect them to do.
I analyzed mostly tiles with much sea (shown with elevation -8888m in Basecamp) and I think the offset zero is used for "empty" block (with no corresponding data in section 2). If you ignore the zeros, the values are indeed continuously growing. Did you see record sizes other than 7 or 8? The variation in size me be due to the fact that for smaller files the offset into section 2 fits into 2 bytes. My guess is that the first 2 (size 7) or 3 (size 8) bytes of the records are an offset and the next two bytes are a "base height".
For the two lowest zoom levels the bounding box/size for the covered area could be calculated by (vx * px, vy * py)
For the "empty" DEM files generated by GMapTool this is correct (px and py just contain the width/height in map units). However, for other files I analyzed the values of px/py are too large.
We don't have any glue what to do with data section 2. But this has to be some picture format, because through removing some layers you will get a lower resolution elevation shading in MapSource. Not knowing much about picture formats, data section 2 (in combination with data section 2) may be a header less GeoTiff or Jpeg (or whatever)
I don't think the data is stored in a "standard" image format, but I don't have any clue yet.
Hope there's someone out with good ideas Ronny
Am 24.02.2010 20:53, schrieb Christian Gawron:
Ronny Klier schrieb:
The single description blocks describe the structure and position of Data blocks.
Offset Length 0x00 2 Index of description block, starting with 0000. This differs between both versions: While 0x25 version counts in high byte (0001), 0x29 version increments lower byte (0100). Maybe this has different meaning. 0x02 4 Unknown (seems to be always 40000000) 0x06 4 Unknown (seems to be always 40000000) 0x0A 4 Values in latitude dimension (vy) 0x0E 4 Values in longitude dimension (vx)
Not sure about this anymore.
0x12 2 Unknown (always 0) 0x14 2 Block size in data section 1 (add 1) 0x16 2 Unknown (always 0) 0x18 2 Block count in data section 1 (add 1)
I saw you wrote on the wiki page that 0x14 is the number of tiles in longitude direction and 0x18 is the number of tiles in latitude direction. This seems to be true: Moving a "line" of records results in moving the elevation shape in MapSource.
0x1A 4 Unknown value (Last byte not zero) 0x1E 2 Record size in data section 1 0x20 4 Offset of data section 1 0x24 4 Offset of data section 2 0x28 4 western boundary (I know this is unusual but non of the bytes has to be zero and the result still fits)
In the files I analyzed the western/northern boundaries are coded with 3 bytes (as usual) and the first byte of these fields look like a separate field to me. Or are you saying the map unit is 360/232 in the DEM file?
Yes, it seems so. One extra byte does not matter. You will get the same result if the extra byte is zero.
0x2C 4 northern boundary 0x30 4 Pixel resolution latitude (py) 0x34 4 Pixel resolution longitude (px)
Are you sure these values are four bytes long?
Yes, I’ve seen maps where the pixel resolution is always 4 times the resolution of higher level. Last value needs at least 3 bytes. (0x0670 -> 0x19E0 -> 0x6790 -> 0x019E40)
0x38 2 Minimum height in feet? (0x8e18=-29160 for files with sea/-8888m) 0x3A 2 Maximum height in feet? (always larger than the previous fields)
Unlikely, 0x38 < 0x3A only if values are read as signed. But why should negative values occur (beside 0x8e18)? Perhaps one of them is the minimum/average/maximum and the other defines the difference between min and max?
Using these values the size of data section could be calculated as Block size * Block count * Record size
Data section 1 could be read as having Block count blocks. Each of them containing Block size records of Record size. In our opinion the records don't hold offset to data section 2: They are often zero and not continuosly growing, which I would expect them to do.
I analyzed mostly tiles with much sea (shown with elevation -8888m in Basecamp) and I think the offset zero is used for "empty" block (with no corresponding data in section 2). If you ignore the zeros, the values are indeed continuously growing.
Did you see record sizes other than 7 or 8? The variation in size me be due to the fact that for smaller files the offset into section 2 fits into 2 bytes.
My guess is that the first 2 (size 7) or 3 (size 8) bytes of the records are an offset and the next two bytes are a "base height".
Seen every record size from 3 to 8. And you are right about the first 2 or 3 bytes being an offset to data 2 (I just read in wrong byte order and got confused by zero offsets). I think the zero offsets maybe reused data. If you have a map with an island surrounded by water it is possible to use offset zero for all water tiles? I saw different size schemes for offset: Record size Offset 7 3 6 2 5 1 But also: 6 3 5 2 4 2 3 1 May be the offset size is coded in lowest 2 bits of byte 0x1C. This bits where always "offset size -1". Ronny
participants (2)
-
Christian Gawron -
Ronny Klier