Corrupt img directory on large image files
Hello, building my own maps with mkgmap I found a strange behavior, probably a bug. Phenomenon: I built an full-blown img from a single 46Mb source, let's say 63240090.osm.gz. No errors in mkgmap so far. Got the 63240090.img and the gmapsupp.img. The gmapsupp.img is empty - no maps in it. The 63240090.img seems fine but is invalid. I know that I could increase the block size to get rid of the large source but this is not the point. Analysis: The img starts with 0x400 bytes header. Then, due to the size of the source 2 header directory blocks follow. The normal directory blocks start subsequently at 0x800 with 63240090RGN. Unfortunately this first block at 0x800 contains size 0 and starts with sequence number 0x01. So I assume that the missing first block has been overwritten by the 2nd header block. Suggested code changes: In package uk.me.parabola.imgfmt.sys in file directory.java in function void sync() replace: int forHeader = (blocks + (blockSize - 1)) / blockSize; by: int forHeader = (blocks + DirectoryEntry.SLOTS_PER_ENTRY - 1)/DirectoryEntry.SLOTS_PER_ENTRY; The calculation of forHeader is obviously wrong what in turn causes the position needed for other blocks to be wrong too. 2 lines later there's an assert to ensure a forHeader of 1. This could be removed. The rest of the code seems to be aware of multiple header blocks. If someone tells me a place were I could store the sample source I will do so. I've no experience in respect of the code change procedures, so could one of you guys check-in the changes? By the way - I wonder why nobody else remarked this before. cheers, Steffen
Hello
Phenomenon: I built an full-blown img from a single 46Mb source, let's say
This is rather large for a single map tile.
Suggested code changes: In package uk.me.parabola.imgfmt.sys in file directory.java in function void sync() replace: int forHeader = (blocks + (blockSize - 1)) / blockSize; by: int forHeader = (blocks + DirectoryEntry.SLOTS_PER_ENTRY - 1)/DirectoryEntry.SLOTS_PER_ENTRY; The calculation of forHeader is obviously wrong what in turn causes the position needed for other blocks to be wrong too.
Yes that does appear to be wrong. Anyway the real bug is that this should have been caught earlier on. There are a couple of places where sizes are checked, my first impression is that the one in BlockManager needs changing.
2 lines later there's an assert to ensure a forHeader of 1. This could be removed. The rest of the code seems to be aware of multiple header blocks.
I'm not sure that more than one block is allowed as I've not seen any files that have more than one. It is also not very useful as you will hit the total number of blocks limit after only a few entries in the second block. I think that another changes would be needed as we only leave room for one block in the calculations of how big the header is I think. The gmapsupp code is able to automatically adjust the block size so that the directory does not overflow. This was never done for single .img files, although it would be good for completeness. The code should really be combined, so there are not two different implementations.
If someone tells me a place were I could store the sample source I will do so.
I've no experience in respect of the code change procedures, so could one of you guys check-in the changes?
If you can create a patch you can post it to the list, otherwise zip up the affected source files and upload to files.mkgmap.org.uk or anywhere else you have access to.
By the way - I wonder why nobody else remarked this before.
Usually you hit another limit first before the block limit is reached. And if the file was somewhat bigger then the check in BlockManager would have been hit, but it is a bit strange that this hasn't been reported before. ..Steve
Phenomenon: I built an full-blown img from a single 46Mb source, let's say
This is rather large for a single map tile. Yes, it is. But should work properly or terminate with an error.
Anyway the real bug is that this should have been caught earlier on. There are a couple of places where sizes are checked, my first impression is that the one in BlockManager needs changing. Where should it be caught? The BlockManger has a single limitation and this is maxBlock. As long as you not hit it, everything is fine. The base value in all setMaxBlock() calls is params.getReservedDirectoryBlocks(). And this seems to be always 2.
2 lines later there's an assert to ensure a forHeader of 1. This could be removed. The rest of the code seems to be aware of multiple header blocks.
I'm not sure that more than one block is allowed as I've not seen any files that have more than one. It is also not very useful as you
It's probably out of spec. I don't know.
will hit the total number of blocks limit after only a few entries in the second block. Yes, after 34 entries.
I think that another changes would be needed as we only leave room for one block in the calculations of how big the header is I think.
I tried out different constellations and it seems to work. The position of other directory blocks is based on the number of header blocks calculated above (the wrong formula). In contrast - the calculation of data block position seems to be based on the correct formula in Map.close(). So I suggest to take-over my approach as quick fix.
The gmapsupp code is able to automatically adjust the block size so that the directory does not overflow. This was never done for single .img files, although it would be good for completeness. The code should really be combined, so there are not two different implementations.
Oh yes, this could be the 'all singing all dancing' solution but for the time being a quick fix should work. The essential question is: Is it compatible to MapSource?
If you can create a patch you can post it to the list, otherwise zip up the affected source files and upload to files.mkgmap.org.uk or anywhere else you have access to. Done. Please find it attached.
best Steffen ___________________________________________________________ WEB.DE DSL Doppel-Flat ab 19,99 €/mtl.! Jetzt auch mit gratis Notebook-Flat! http://produkte.web.de/go/DSL_Doppel_Flatrate/2
Hi
There are a couple of places where sizes are checked, my first impression is that the one in BlockManager needs changing. Where should it be caught? The BlockManger has a single limitation and this is maxBlock. As long as you not hit it, everything is fine. The base value in all setMaxBlock() calls is params.getReservedDirectoryBlocks(). And this seems to be always 2.
I was thinking that maxBlock is 240*240 when the block size is 512 and not 0xfffe. Sure, the logic to set that may be better elsewhere.
The essential question is: Is it compatible to MapSource?
Have you not tried it? A gmapsupp should be fine since it adjusts the block size so that there is only one header file directory block required. The Mdr index just uses a 4k block size always and so there is a maximum size for it. I believe that qlandkarte mostly ignores the block numbers and so might not be affected.
Done. Please find it attached.
OK, but we need to find out if the directory file can really have more than one entry. ..Steve
The essential question is: Is it compatible to MapSource?
Have you not tried it? I normally don't use MapSource (*). But I've installed it - just to try out and it shows the tile with the 2 directory blocks. So everything seems to be fine.
bye Steffen (*) I have my own experimental renderer. Play around with it if you like. Download: http://www.litotes.de/osm/FancyRoute.exe Map setup: Menu-Options-Preferences-Garmin Map: Insert the path to your img-folder or the path to a single img. Some features (ungraded): draw/edit waypoints/tracks/routes, typ-support, track autorouting, gpx-in/out, map print, Garmin GPS track/route/wp up/download, POI search and many more. Use mouse wheel to move(press) or zoom(up/down) map. Check preferences to get an impression about all features. Use menu help to get more info. The renderer is very fast. But keep in mind: its experimental. ___________________________________________________________ GRATIS! Movie-FLAT mit über 300 Videos. Jetzt freischalten unter http://movieflat.web.de
participants (2)
-
Steffen Neumann -
Steve Ratcliffe