DEM file and the block-size option
Hi all, the current default block-size is 512. This is used for most *.img files and gives a maximum file size of 65534*512 ~ 32MB. For large DEM files you might need a --block-size=1024 or even 2048. If I got that right you may run into the next problem when using this option as the mdr file will be also created with this value, and if no value is given it uses 16384 (which means the mdr file can grow close to 1GB) This disadvantage of a larger block-size is that your img files are typically a bit larger. @Steve: If I got that right we should be able to calculate the optimal block-size as we do know the number of bytes that we will write to the img-file before anything is actually written to the disk. Wouldn't that be much better? Gerd
Hi Gerd It would be a lot better and that is what happens for the gmapsupp file. I should be able to make the code work for all IMG files. Steve
@Steve: If I got that right we should be able to calculate the optimal block-size as we do know the number of bytes that we will write to the img-file before anything is actually written to the disk. Wouldn't that be much better?
Hi Gerd,
This disadvantage of a larger block-size is that your img files are typically a bit larger.
For small img yes. For bigger img larger block-size results in smaller img size, since FAT contains less records and is smaller. I think you simply could set block-size 2048 as default. -- Best regards, Andrzej
Andrzej, Steve, not sure what to do here. I agree that 2048 looks like a good default, at least for DEM tiles. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Andrzej Popowski <popej@poczta.onet.pl> Gesendet: Dienstag, 26. Dezember 2017 20:03:33 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] DEM file and the block-size option Hi Gerd,
This disadvantage of a larger block-size is that your img files are typically a bit larger.
For small img yes. For bigger img larger block-size results in smaller img size, since FAT contains less records and is smaller. I think you simply could set block-size 2048 as default. -- Best regards, Andrzej _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Gerd, I have tested some img, changing block size with GMapTool. These are results, first original file size, then size delta for blocks 1k, 2k and 4k: 14.8k -- +3.6k +9.9k +22.0k 109k -- +2.6k +7.7k +16.9k 1M -- +0.0k +5.2k +17.7k 2M -- -1.6k -0.5k +10.7k 3M -- -3.6k -3.6k +2.6k 4M -- -5.7k -4.7k -0.5k 8M -- -14.3k -18.4k -16.4k Looks like 2MB is a tile size, above which block size 2048 can be better. I would change default. You could think about optimization, if you are interested, but gain won't be big. Side notes: Block size for overview map is probably always 512, can be problematic for very big maps with DEM. There is something wrong with FAT in MDR. FAT shouldn't be bigger than about 140kB. Mkgmap creates MDR with FAT of 3.2MB, almost all empty. -- Best regards, Andrzej
Hi Andrzej, for MDR mkgmap sets the blocksize to 16384 if not set otherwise with the block-size option: params.setBlockSize(args.get("block-size", 16384)); Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Andrzej Popowski <popej@poczta.onet.pl> Gesendet: Samstag, 6. Januar 2018 20:39:12 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] DEM file and the block-size option Hi Gerd, I have tested some img, changing block size with GMapTool. These are results, first original file size, then size delta for blocks 1k, 2k and 4k: 14.8k -- +3.6k +9.9k +22.0k 109k -- +2.6k +7.7k +16.9k 1M -- +0.0k +5.2k +17.7k 2M -- -1.6k -0.5k +10.7k 3M -- -3.6k -3.6k +2.6k 4M -- -5.7k -4.7k -0.5k 8M -- -14.3k -18.4k -16.4k Looks like 2MB is a tile size, above which block size 2048 can be better. I would change default. You could think about optimization, if you are interested, but gain won't be big. Side notes: Block size for overview map is probably always 512, can be problematic for very big maps with DEM. There is something wrong with FAT in MDR. FAT shouldn't be bigger than about 140kB. Mkgmap creates MDR with FAT of 3.2MB, almost all empty. -- Best regards, Andrzej _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Gerd,
for MDR mkgmap sets the blocksize to 16384 if not set otherwise with the block-size option: params.setBlockSize(args.get("block-size", 16384));
I have seen this code. I think it is wrong to use the same option for tiles and index. Index can be very big regardless of tile size. I noticed other problem. Please look at any *_mdr.img with block size 16k. You will find a 3MB of zeroes between FAT and map data. I haven't found code, which calculates offset from beginning of file to start of MDR header, I think it is calculated wrong. It looks like offset increases with block size, which makes no sens - FAT records remains the same, regardless of block size. -- Best regards, Andrzej
Hi Gerd
not sure what to do here. I agree that 2048 looks like a good default, at least for DEM tiles.
I am looking at this. I am going to remove all the existing code that writes the FAT tables as that needs to the block size before you start. The new code will calculate the best block size. ..Steve
Hi Steve, thanks, sounds great. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Steve Ratcliffe <steve@parabola.me.uk> Gesendet: Samstag, 6. Januar 2018 22:58:46 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] DEM file and the block-size option Hi Gerd
not sure what to do here. I agree that 2048 looks like a good default, at least for DEM tiles.
I am looking at this. I am going to remove all the existing code that writes the FAT tables as that needs to the block size before you start. The new code will calculate the best block size. ..Steve _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi, here is attached temporary fix for block-size. List of changes: * Default block size for img set to 1k. This allows for img up to 64MB. User can change this value with option --block-size. * Block size for overview map fixed at 1k. Not changeable by user. * Block size for MDR fixed at 16k. Not changeable by user. * Corrected number of reserved directory blocks for MDR, reduces size of MDR by 3MB. I have added a comment about calculation of reserved directory blocks, I'm not sure if code is wrong, but it does not exactly what comments describe. -- Best regards, Andrzej
Hi Andrzej
- // There are 2 slots for the header itself. + // There are 2 slots for the header itself. (slots or blocks? code adds blocks) int blocksRequired = 2 + headerSlotsRequired * 512 / blockSize;
It should be slots, the code is wrong. Also it should be 1 and not 2. The mistake is harmless apart from the file being a little bigger than necessary. ..Steve
Hi all, I hesitate to commit this patch because I think that some users create overview maps > 64MB. Please check! @Steve: I see you are making progress in the auto-block branch. Please stop me if this patch causes trouble when merging. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Steve Ratcliffe <steve@parabola.me.uk> Gesendet: Samstag, 13. Januar 2018 15:21:18 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] DEM file and the block-size option Hi Andrzej
- // There are 2 slots for the header itself. + // There are 2 slots for the header itself. (slots or blocks? code adds blocks) int blocksRequired = 2 + headerSlotsRequired * 512 / blockSize;
It should be slots, the code is wrong. Also it should be 1 and not 2. The mistake is harmless apart from the file being a little bigger than necessary. ..Steve _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi, I got once overview bigger than 32M and I had to decrease DEM resolution. 64MB seems quite big for an overview. Anyway, I have provided this patch, because I have finally found the reason for big MDR sizes. I hope at least this one correction can be moved to trunk. -- Best regards, Andrzej
Hi Gerd
@Steve: I see you are making progress in the auto-block branch. Please stop me if this patch causes trouble when merging.
It is not a problem for me. It may be a while before the auto-block code is fully working and tested, so the patch will be useful in the mean time. ..Steve
For overview-map it would be fine for me, but for detailed map I need a block-size of 2k Henning On 13.01.2018 22:57, Gerd Petermann wrote:
Hi all,
I hesitate to commit this patch because I think that some users create overview maps > 64MB. Please check!
@Steve: I see you are making progress in the auto-block branch. Please stop me if this patch causes trouble when merging.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Steve Ratcliffe <steve@parabola.me.uk> Gesendet: Samstag, 13. Januar 2018 15:21:18 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] DEM file and the block-size option
Hi Andrzej
- // There are 2 slots for the header itself. + // There are 2 slots for the header itself. (slots or blocks? code adds blocks) int blocksRequired = 2 + headerSlotsRequired * 512 / blockSize; It should be slots, the code is wrong. Also it should be 1 and not 2.
The mistake is harmless apart from the file being a little bigger than necessary.
..Steve
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
OK, I've now committed this patch, see http://www.mkgmap.org.uk/websvn/revision.php?repname=mkgmap&rev=4054 ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Henning Scholland <osm@hscholland.de> Gesendet: Sonntag, 14. Januar 2018 03:22:23 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] DEM file and the block-size option For overview-map it would be fine for me, but for detailed map I need a block-size of 2k Henning On 13.01.2018 22:57, Gerd Petermann wrote:
Hi all,
I hesitate to commit this patch because I think that some users create overview maps > 64MB. Please check!
@Steve: I see you are making progress in the auto-block branch. Please stop me if this patch causes trouble when merging.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Steve Ratcliffe <steve@parabola.me.uk> Gesendet: Samstag, 13. Januar 2018 15:21:18 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] DEM file and the block-size option
Hi Andrzej
- // There are 2 slots for the header itself. + // There are 2 slots for the header itself. (slots or blocks? code adds blocks) int blocksRequired = 2 + headerSlotsRequired * 512 / blockSize; It should be slots, the code is wrong. Also it should be 1 and not 2.
The mistake is harmless apart from the file being a little bigger than necessary.
..Steve
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
participants (5)
-
Andrzej Popowski -
Gerd Petermann -
Gerd Petermann
-
Henning Scholland -
Steve Ratcliffe