FW: AW: RE: computing power mdx/mdr
Hi all, Arndt confirmed that his memory problem is solved with the Mdr7.patch . If I hear no complains I'll commit it to trunk tomorrow. Gerd Hi Gerd,funktioniert ☺ ich werde zukünftig mehrere Durchläufe mit mkgmap machen.Kacheln - mdx/mdr - gmapsupp -tdb/img (mit den Höhenlinien). Für Europa muss ich mal in richtige Hardware investieren. Vielen Dank für deine Hilfe!!! GrußArndt
Hi Steve, I think a better solution reg. memory would be not to store the byte array for the partial name at all. Since it is a sub string of the name, I assume it should be sufficient to calc the offset and length in the byte array and store these values? Gerd Steve Ratcliffe wrote
Hi Gerd
Arndt confirmed that his memory problem is solved with the Mdr7.patch . If I hear no complains I'll commit it to trunk tomorrow.
Great, thanks for fixing this.
..Steve _______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
-- View this message in context: http://gis.19327.n5.nabble.com/FW-AW-RE-computing-power-mdx-mdr-tp5834885p58... Sent from the Mkgmap Development mailing list archive at Nabble.com.
On 25/02/15 15:45, GerdP wrote:
I think a better solution reg. memory would be not to store the byte array for the partial name at all. Since it is a sub string of the name, I assume it should be sufficient to calc the offset and length in the byte array and store these values?
Where do you mean? A call to substring() shares the byte array of the original string, so there is no extra array. Also I don't think it is stored anywhere as the sort key is a new array. We just do not need the MultiSortKey when the partialName is the same as the name so we can eliminate using the partialName altogether. As in attached patch (warning: untested). ..Steve
Hi Steve, your patch (which BTW contains a lot of experimental code) tries to avoid the additional memory for the MultiSortKey. That is probably also a good idea in case the compareTo() routines can handle different types properly. My idea was to optimize the cases where partial name and name are different. If I got it right, we use this to handle the highway shields. I wonder if it would be enough to move the byte(s) for the highway shield to the end of the name that is used for sorting? Gerd Steve Ratcliffe wrote
On 25/02/15 15:45, GerdP wrote:
I think a better solution reg. memory would be not to store the byte array for the partial name at all. Since it is a sub string of the name, I assume it should be sufficient to calc the offset and length in the byte array and store these values?
Where do you mean? A call to substring() shares the byte array of the original string, so there is no extra array. Also I don't think it is stored anywhere as the sort key is a new array.
We just do not need the MultiSortKey when the partialName is the same as the name so we can eliminate using the partialName altogether.
As in attached patch (warning: untested).
..Steve
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
single-key-for-full-name.patch (3K) <http://gis.19327.n5.nabble.com/attachment/5835052/0/single-key-for-full-name...;
-- View this message in context: http://gis.19327.n5.nabble.com/FW-AW-RE-computing-power-mdx-mdr-tp5834885p58... Sent from the Mkgmap Development mailing list archive at Nabble.com.
Hi Gerd
My idea was to optimize the cases where partial name and name are different. If I got it right, we use this to handle the highway shields. I wonder if it would be enough to move the byte(s) for the highway shield to the end of the name that is used for sorting?
Yes, we can completely remove the MultiSortKey and treat all cases the same. Just need to append any initial part of the name at the end. As in attached (untested) patch. ..Steve
Hi Steve, yes, looks much better. I did not test it, but I think it will give a different result for those cases where name contains a suffix (0x1e) . I remember these as lead in/lead out sequences for japanese characters (double byte) characters in ASCII strings? Gerd Steve Ratcliffe wrote
Hi Gerd
My idea was to optimize the cases where partial name and name are different. If I got it right, we use this to handle the highway shields. I wonder if it would be enough to move the byte(s) for the highway shield to the end of the name that is used for sorting?
Yes, we can completely remove the MultiSortKey and treat all cases the same. Just need to append any initial part of the name at the end.
As in attached (untested) patch.
..Steve
_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@.org
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
sort-save-mem.patch (1K) <http://gis.19327.n5.nabble.com/attachment/5835073/0/sort-save-mem.patch>;
-- View this message in context: http://gis.19327.n5.nabble.com/FW-AW-RE-computing-power-mdx-mdr-tp5834885p58... Sent from the Mkgmap Development mailing list archive at Nabble.com.
Hi Gerd
yes, looks much better. I did not test it, but I think it will give a different result for those cases where name contains a suffix (0x1e) . I remember these as lead in/lead out sequences for japanese characters (double byte) characters in ASCII strings?
Japanese almost certainly doesn't work with the global index anyway. I don't see that there would be a difference however - the MultiSortKey is effectively just sorting on the concatenation of the partial and its prefix. ..Steve
Hi Steve, I think in the old algo the bytes of the suffix were compared at last, with the patch they are compared before the prefix. Gerd
Date: Thu, 26 Feb 2015 22:47:42 +0000 From: steve@parabola.me.uk To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] FW: AW: RE: computing power mdx/mdr
Hi Gerd
yes, looks much better. I did not test it, but I think it will give a different result for those cases where name contains a suffix (0x1e) . I remember these as lead in/lead out sequences for japanese characters (double byte) characters in ASCII strings?
Japanese almost certainly doesn't work with the global index anyway. I don't see that there would be a difference however - the MultiSortKey is effectively just sorting on the concatenation of the partial and its prefix.
..Steve _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Gerd
I think in the old algo the bytes of the suffix were compared at last, with the patch they are compared before the prefix.
Ah, right. I think that in both cases all streets could be found and it is just the order that they are presented to the user that will differ. Since we currently don't use suffixes, then I am not sure what will work the best. ..Steve
Hi Steve, okay, in that case I think you should commit this patch with the corrected method name. I've tested the performance with a set of files created with Arndts style, and peak memory decreased from 208 MB to 175 MB. Gerd
Date: Fri, 27 Feb 2015 23:57:35 +0000 From: steve@parabola.me.uk To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] FW: AW: RE: computing power mdx/mdr
Hi Gerd
I think in the old algo the bytes of the suffix were compared at last, with the patch they are compared before the prefix.
Ah, right. I think that in both cases all streets could be found and it is just the order that they are presented to the user that will differ. Since we currently don't use suffixes, then I am not sure what will work the best.
..Steve _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
On Thu, Feb 26, 2015 at 10:47:42PM +0000, Steve Ratcliffe wrote:
Japanese almost certainly doesn't work with the global index anyway.
Right. For Japanese, MeCab has a different approach for fulltext search, applying morphological analysis. I wonder if Garmin supports anything like that on its devices, or if this type of indexing makes sense for the street naming schemes that are used in Japan. If not, there is not much that a map generator can do. BTW, in the patch there was a typo in a method name: getInitalPart() should be getInitialPart(). Marko
On 27/02/15 12:11, Marko Mäkelä wrote:
Right. For Japanese, MeCab has a different approach for fulltext search, applying morphological analysis.
Ahh, that would be something that I was not at all aware of :) I was only referring to the cp932 code page and the index. I doubt that it works at all even for whole name matches. But I haven't tried it so maybe it does! ..Steve
participants (4)
-
Gerd Petermann -
GerdP
-
Marko Mäkelä -
Steve Ratcliffe