Hi Steve, I hope you can help me out here. I've implemented a new option --x-dem-reuse which allows to copy the DEM file from an older map. My idea is that the option specifies the path to either 1) a gmapsupp or 2) a gmap folder or 3) a directory containing *.img files from a previous run with similar dem-dist option and tile boundaries. If given, mkgmap tries to find the matching *.DEM in any of these formats, checks if the content matches the wanted boundary and dem-dist option and - if ok - uses the existing file instead of doing the DEM calculations again. At the moment I have some working quick+dirty code (see attached patch). I assume the new code in MapBuilder should go somewhere in package uk.me.parabola.imgfmt.app, maybe a new class ImgFinder or so? Or do we already have such code in mkgmap? What I also don't like is that it copies the data from the existing file into memory. Is there already an easy way to avoid that? Gerd
Hi Gerd I would use a new implementation of FileSystem, named GmapFS or similar, which would go in package .imgfmt.sys Then add the trait FileSystem.openFS() which will return the correct ImgFS or GmapFS. Then your code would be roughly: fs = FileSystem.openFS(demReuseDir); // Will return ImgFS or GmapFS chan = fs.open(demFilename, "r") // read DEM header to check if it can be used. To copy without having to read it into memory, you can use FileCopier. I could implement GmapFS as I was already thinking of doing that so that we would be able to convert gmapi format to gmapsupp. Steve
Hi Steve, I hope you can help me out here.
I've implemented a new option --x-dem-reuse which allows to copy the DEM file from an older map. My idea is that the option specifies the path to either 1) a gmapsupp or 2) a gmap folder or 3) a directory containing *.img files from a previous run with similar dem-dist option and tile boundaries.
If given, mkgmap tries to find the matching *.DEM in any of these formats, checks if the content matches the wanted boundary and dem-dist option and - if ok - uses the existing file instead of doing the DEM calculations again.
At the moment I have some working quick+dirty code (see attached patch). I assume the new code in MapBuilder should go somewhere in package uk.me.parabola.imgfmt.app, maybe a new class ImgFinder or so? Or do we already have such code in mkgmap? What I also don't like is that it copies the data from the existing file into memory. Is there already an easy way to avoid that?
Gerd
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Steve, A new fs sounds good to me, would be great if you could do that, I am always unsure where to call close(). Reg. FileCopier: I have to read the DEM file to make sure that it matches, I just think that it would be better to store a reference to the existing file instead of copying it into memory. The code that writes the new img file would somehow notice that the data is not in memory but on disk. No idea if this would really improve anything, maybe the partial reading of DEM already requires a full copy in memory. I'll have a closer look at this now. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Steve Ratcliffe <steve@parabola.me.uk> Gesendet: Sonntag, 28. Januar 2018 22:41 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] gmapi reader? Hi Gerd I would use a new implementation of FileSystem, named GmapFS or similar, which would go in package .imgfmt.sys Then add the trait FileSystem.openFS() which will return the correct ImgFS or GmapFS. Then your code would be roughly: fs = FileSystem.openFS(demReuseDir); // Will return ImgFS or GmapFS chan = fs.open(demFilename, "r") // read DEM header to check if it can be used. To copy without having to read it into memory, you can use FileCopier. I could implement GmapFS as I was already thinking of doing that so that we would be able to convert gmapi format to gmapsupp. Steve
Hi Steve, I hope you can help me out here.
I've implemented a new option --x-dem-reuse which allows to copy the DEM file from an older map. My idea is that the option specifies the path to either 1) a gmapsupp or 2) a gmap folder or 3) a directory containing *.img files from a previous run with similar dem-dist option and tile boundaries.
If given, mkgmap tries to find the matching *.DEM in any of these formats, checks if the content matches the wanted boundary and dem-dist option and - if ok - uses the existing file instead of doing the DEM calculations again.
At the moment I have some working quick+dirty code (see attached patch). I assume the new code in MapBuilder should go somewhere in package uk.me.parabola.imgfmt.app, maybe a new class ImgFinder or so? Or do we already have such code in mkgmap? What I also don't like is that it copies the data from the existing file into memory. Is there already an easy way to avoid that?
Gerd
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Gerd
A new fs sounds good to me, would be great if you could do that, I am always unsure where to call close(). OK
Reg. FileCopier: I have to read the DEM file to make sure that it matches, I just
If I didn't miss anything you just need to read the header to do the check. I would do the check outside of DEMFile. FileCopier acts like a placeholder to the existing DEM in another file and when the IMG file is closed the data will be copied directly to the output. If you always want DEM files to be written to a temporary file instead of to memory (like the .MDR file is) then you can use FileBackedImgWriter instead of BufferedImgWriter in DEM file.
think that it would be better to store a reference to the existing file instead of copying it into memory. The code that writes the new img file would somehow notice that the data is not in memory but on disk. No idea if this would really improve anything, maybe the partial reading of DEM already requires a full copy in memory. I'll have a closer look at this now.
Hi Steve, the DEM header structures are quite special, the headers of the sections are written at the very end of the DEM file, and they point to data that follows the main header. This is probably not mandantory but it's the way Garmin does it, and also DemDisplay uses it to determine the length of the last bitstream. So, to do the needed checks we have to read main header and the DEM Section headers at the end of the file. Not sure where to place this if not in DEMFile. Where would you implement that? reg. FileCopier: I don't see how I can use that class outside of GmapsuppBuilder. Maybe you can extract it? I've already thought about using FileBackedImgWriter. I'll do some tests to find out how it influences performance. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Steve Ratcliffe <steve@parabola.me.uk> Gesendet: Montag, 29. Januar 2018 10:27 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] gmapi reader? Hi Gerd
A new fs sounds good to me, would be great if you could do that, I am always unsure where to call close(). OK
Reg. FileCopier: I have to read the DEM file to make sure that it matches, I just
If I didn't miss anything you just need to read the header to do the check. I would do the check outside of DEMFile. FileCopier acts like a placeholder to the existing DEM in another file and when the IMG file is closed the data will be copied directly to the output. If you always want DEM files to be written to a temporary file instead of to memory (like the .MDR file is) then you can use FileBackedImgWriter instead of BufferedImgWriter in DEM file.
think that it would be better to store a reference to the existing file instead of copying it into memory. The code that writes the new img file would somehow notice that the data is not in memory but on disk. No idea if this would really improve anything, maybe the partial reading of DEM already requires a full copy in memory. I'll have a closer look at this now.
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Gerd Is this something that needs to be done before the merge?
the DEM header structures are quite special, the headers of the sections are written at the very end of the DEM file, and they point to data that follows the main header. This is probably not mandantory but it's the way Garmin does it, and also DemDisplay uses it to determine the length of the last bitstream. So, to do the needed checks we have to read main header and the DEM Section headers at the end of the file.
OK so that is more than I thought, but still not the whole file? If you do have to read the whole file, then you might as well use it and the following may not make sense.
Not sure where to place this if not in DEMFile. Where would you implement that?
I meant to move everything in DEMFile.tryCopy(). I guess it goes back to Map.addDem(). If possible you want to do things there or in the code that calls addDem(). You may not have all the required information there, in which case I don't know how to proceed. Currently addDem() is: demFile = new DEMFile(fileSystem.create(mapId + ".DEM"), true); But you want either a DEMFile or a FileCopier, you only want the demFile when you need to create it. handle = fileSystem.create(mapId + ".DEM"); if (checkExistingDem()) { // Can copy existing one // Just a rough idea, many details to be filled in... fc = FileCopier(fileContainingDem); cl = fc.add(demFilename, handle) handle.link(() -> demSize, cl); // Probably would create a new constructor: // fc = FileCopier(handle, fileContainingDem); // fc.add(demFilename); // When the img file is closed then the copy will happen // automatically. // demFile = null } else { // Need to create it demFile = new DEMFile(handle, true); } Hope that makes some kind of sense. There are probably a lot of details missing.
reg. FileCopier: I don't see how I can use that class outside of GmapsuppBuilder. Maybe you can extract it?
I've already thought about using FileBackedImgWriter. I'll do some tests to find out how it influences performance.
Yes it will need to be moved to a top level class and maybe some changes as well. ..Steve
Hi Steve, I'd like to merge first and do any optimisation later. I've done a few tests reg. memory. I think we should not care much. It would be easy to free the resources allocated in DEMTile directly after the data was written. I think peak memory is reached when processing the style, not at this stage. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Steve Ratcliffe <steve@parabola.me.uk> Gesendet: Montag, 29. Januar 2018 12:30 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] gmapi reader? Hi Gerd Is this something that needs to be done before the merge?
the DEM header structures are quite special, the headers of the sections are written at the very end of the DEM file, and they point to data that follows the main header. This is probably not mandantory but it's the way Garmin does it, and also DemDisplay uses it to determine the length of the last bitstream. So, to do the needed checks we have to read main header and the DEM Section headers at the end of the file.
OK so that is more than I thought, but still not the whole file? If you do have to read the whole file, then you might as well use it and the following may not make sense.
Not sure where to place this if not in DEMFile. Where would you implement that?
I meant to move everything in DEMFile.tryCopy(). I guess it goes back to Map.addDem(). If possible you want to do things there or in the code that calls addDem(). You may not have all the required information there, in which case I don't know how to proceed. Currently addDem() is: demFile = new DEMFile(fileSystem.create(mapId + ".DEM"), true); But you want either a DEMFile or a FileCopier, you only want the demFile when you need to create it. handle = fileSystem.create(mapId + ".DEM"); if (checkExistingDem()) { // Can copy existing one // Just a rough idea, many details to be filled in... fc = FileCopier(fileContainingDem); cl = fc.add(demFilename, handle) handle.link(() -> demSize, cl); // Probably would create a new constructor: // fc = FileCopier(handle, fileContainingDem); // fc.add(demFilename); // When the img file is closed then the copy will happen // automatically. // demFile = null } else { // Need to create it demFile = new DEMFile(handle, true); } Hope that makes some kind of sense. There are probably a lot of details missing.
reg. FileCopier: I don't see how I can use that class outside of GmapsuppBuilder. Maybe you can extract it?
I've already thought about using FileBackedImgWriter. I'll do some tests to find out how it influences performance.
Yes it will need to be moved to a top level class and maybe some changes as well. ..Steve _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Gerd
I'd like to merge first and do any optimisation later.
OK, that's good.
I've done a few tests reg. memory. I think we should not care much. It would be easy to free the resources allocated in DEMTile directly after the data was written. I think peak memory is reached when processing the style, not at this stage.
Steve
participants (3)
-
Gerd Petermann -
Gerd Petermann
-
Steve Ratcliffe