mkgmap-r3906 (optimize-index)
Hi all, with the help of Steve I fixed some problems with the index, esp. sorting of road names with different speliing of Straße caused a lot of problems, like Ahornstraße (Germany) and Ahornstrasse (Switzerland). I think address / road search works very well now, at least with west european languages. I tried various combinations of options like --latin1 / --unicode, --lower-case, --x-split-name-index both in MapSource and on my Oregon 600 and always got what I expected. IMPORTANT: If you try this version, please make sure that you also compile the img files with this version so that the changes in the sort are used everywhere. The branch also reduces peak memory compared to trunk and because of that it is faster when creating large indexes, but speed is probably not so important here. The created index is a bit smaller although it now also contains roads with an empty string as first label. If memory is still an issue for you when compiling the index for large maps I can try to implement a merge sort which would only create the - heap consuming - sort keys for a rather small number (e.g. 100.000) roads and sort those and finally merge the parts. If you know special cases which don't work with r3890 trunk please try the branch and let me know if something might be improved. I think it is a big step forward, but there may still be special cases with other languages. I used a small set of only 4 tiles to test functionality and compiled index for Europe (compiled with default style) (>1600 tiles) with -Xmx6800m and --x-split-name-index Gerd
Hi all, as a follow up: In r3907 and r3908 I have coded the merge sort for roads and pois. Now -Xmx4000 was easily enough to create the index for Europe. r3906 failed with OutOfMemoryError even with -Xmx5000 . So I think memory is no longer a problem unless you want to create an index for planet ;-) No other changes, means r3908 can be used to create index for *.img files created with r3906. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <GPetermann_muenchen@hotmail.com> Gesendet: Montag, 17. April 2017 20:05:15 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: [mkgmap-dev] mkgmap-r3906 (optimize-index) Hi all, with the help of Steve I fixed some problems with the index, esp. sorting of road names with different speliing of Straße caused a lot of problems, like Ahornstraße (Germany) and Ahornstrasse (Switzerland). I think address / road search works very well now, at least with west european languages. I tried various combinations of options like --latin1 / --unicode, --lower-case, --x-split-name-index both in MapSource and on my Oregon 600 and always got what I expected. IMPORTANT: If you try this version, please make sure that you also compile the img files with this version so that the changes in the sort are used everywhere. The branch also reduces peak memory compared to trunk and because of that it is faster when creating large indexes, but speed is probably not so important here. The created index is a bit smaller although it now also contains roads with an empty string as first label. If memory is still an issue for you when compiling the index for large maps I can try to implement a merge sort which would only create the - heap consuming - sort keys for a rather small number (e.g. 100.000) roads and sort those and finally merge the parts. If you know special cases which don't work with r3890 trunk please try the branch and let me know if something might be improved. I think it is a big step forward, but there may still be special cases with other languages. I used a small set of only 4 tiles to test functionality and compiled index for Europe (compiled with default style) (>1600 tiles) with -Xmx6800m and --x-split-name-index Gerd _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Arndt, thanks for the flowers ;-) I am still surprised that your style requires so much more memory compared to the default style. Did you use the scripts in SpeicheFabrik_Steuerdateien170419.zip for this? Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Arndt Röhrig <arndt@speichenkarte.de> Gesendet: Mittwoch, 19. April 2017 17:04:43 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] mkgmap-r3906 (optimize-index) Thank you for that great work! Now my old nostalgia PC is able to create the index for Speiche_Europa! My PC has only 6GB RAM. -Xmx5300M -> fails with "java heap space error" -Xmx10500M -> fails with "overflowed directory with max block". - Xmx10500M and Option --block-size=65536 -> That works! Its better to run mkgmap 2 times. First step builds the maptiles with Xmx3000M, so that java not use the harddisk to swap, cause that makes the PC very slow. The ovm-work files may not be deleted for the overview map. The second step with -Xmx10500M builds the index and the overview map. The taskmanager show maximal ~9GB in use. The Speiche_Europa map has ~20GB. Special thanks to Gerd, who show me many things to improve my map building procedere! Best regards Arndt speichenkarte.de . Gerd Petermann <GPetermann_muenchen@hotmail.com> hat am 18. April 2017 um 16:41 geschrieben: Hi all, as a follow up: In r3907 and r3908 I have coded the merge sort for roads and pois. Now -Xmx4000 was easily enough to create the index for Europe. r3906 failed with OutOfMemoryError even with -Xmx5000 . So I think memory is no longer a problem unless you want to create an index for planet ;-) No other changes, means r3908 can be used to create index for *.img files created with r3906. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <GPetermann_muenchen@hotmail.com> Gesendet: Montag, 17. April 2017 20:05:15 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: [mkgmap-dev] mkgmap-r3906 (optimize-index) Hi all, with the help of Steve I fixed some problems with the index, esp. sorting of road names with different speliing of Straße caused a lot of problems, like Ahornstraße (Germany) and Ahornstrasse (Switzerland). I think address / road search works very well now, at least with west european languages. I tried various combinations of options like --latin1 / --unicode, --lower-case, --x-split-name-index both in MapSource and on my Oregon 600 and always got what I expected. IMPORTANT: If you try this version, please make sure that you also compile the img files with this version so that the changes in the sort are used everywhere. The branch also reduces peak memory compared to trunk and because of that it is faster when creating large indexes, but speed is probably not so important here. The created index is a bit smaller although it now also contains roads with an empty string as first label. If memory is still an issue for you when compiling the index for large maps I can try to implement a merge sort which would only create the - heap consuming - sort keys for a rather small number (e.g. 100.000) roads and sort those and finally merge the parts. If you know special cases which don't work with r3890 trunk please try the branch and let me know if something might be improved. I think it is a big step forward, but there may still be special cases with other languages. I used a small set of only 4 tiles to test functionality and compiled index for Europe (compiled with default style) (>1600 tiles) with -Xmx6800m and --x-split-name-index Gerd _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Arndt, strange, the index itself is ~ 1GB, that's very close to the size for the default style in r3908. Did you use r3908 or an older version? Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Arndt Röhrig <arndt@speichenkarte.de> Gesendet: Freitag, 21. April 2017 10:28:25 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] mkgmap-r3906 (optimize-index) Hi Gerd, yes, i use 170419. The OSM-Data is downloaded by "Geofabrik" (europe). To be added 5GB SRTM data. osm and srtm data merged with the splitter. Arndt Gerd Petermann <GPetermann_muenchen@hotmail.com> hat am 21. April 2017 um 07:34 geschrieben: Hi Arndt, thanks for the flowers ;-) I am still surprised that your style requires so much more memory compared to the default style. Did you use the scripts in SpeicheFabrik_Steuerdateien170419.zip for this? Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Arndt Röhrig <arndt@speichenkarte.de> Gesendet: Mittwoch, 19. April 2017 17:04:43 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] mkgmap-r3906 (optimize-index) Thank you for that great work! Now my old nostalgia PC is able to create the index for Speiche_Europa! My PC has only 6GB RAM. -Xmx5300M -> fails with "java heap space error" -Xmx10500M -> fails with "overflowed directory with max block". * Xmx10500M and Option --block-size=65536 -> That works! Its better to run mkgmap 2 times. First step builds the maptiles with Xmx3000M, so that java not use the harddisk to swap, cause that makes the PC very slow. The ovm-work files may not be deleted for the overview map. The second step with -Xmx10500M builds the index and the overview map. The taskmanager show maximal ~9GB in use. The Speiche_Europa map has ~20GB. Special thanks to Gerd, who show me many things to improve my map building procedere! Best regards Arndt speichenkarte.de . Gerd Petermann <GPetermann_muenchen@hotmail.com> hat am 18. April 2017 um 16:41 geschrieben: Hi all, as a follow up: In r3907 and r3908 I have coded the merge sort for roads and pois. Now -Xmx4000 was easily enough to create the index for Europe. r3906 failed with OutOfMemoryError even with -Xmx5000 . So I think memory is no longer a problem unless you want to create an index for planet ;-) No other changes, means r3908 can be used to create index for *.img files created with r3906. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <GPetermann_muenchen@hotmail.com> Gesendet: Montag, 17. April 2017 20:05:15 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: [mkgmap-dev] mkgmap-r3906 (optimize-index) Hi all, with the help of Steve I fixed some problems with the index, esp. sorting of road names with different speliing of Straße caused a lot of problems, like Ahornstraße (Germany) and Ahornstrasse (Switzerland). I think address / road search works very well now, at least with west european languages. I tried various combinations of options like --latin1 / --unicode, --lower-case, --x-split-name-index both in MapSource and on my Oregon 600 and always got what I expected. IMPORTANT: If you try this version, please make sure that you also compile the img files with this version so that the changes in the sort are used everywhere. The branch also reduces peak memory compared to trunk and because of that it is faster when creating large indexes, but speed is probably not so important here. The created index is a bit smaller although it now also contains roads with an empty string as first label. If memory is still an issue for you when compiling the index for large maps I can try to implement a merge sort which would only create the - heap consuming - sort keys for a rather small number (e.g. 100.000) roads and sort those and finally merge the parts. If you know special cases which don't work with r3890 trunk please try the branch and let me know if something might be improved. I think it is a big step forward, but there may still be special cases with other languages. I used a small set of only 4 tiles to test functionality and compiled index for Europe (compiled with default style) (>1600 tiles) with -Xmx6800m and --x-split-name-index Gerd _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Arndt, seems that -Xmx5300 was just a little too less, on my machine it worked fine with -Xmx6000m and also crashed with -Xmx5300m. I've downloaded your map and used MapReverseConverter from http://www.javawa.nl/cne2009_multilang.html to convert the .gmap format back to *.img files so that I could use mkgmap to re-calculate the index. java -Xmx6000m -jar d:\mkgmap\dist\mkgmap_r3910.jar -c mkgmap_options_mdrmdx.args --block-size=65536 ..\Speiche_Europa_170419\88*.img (r3910 is exactly like r3908 for this purpose) I am working on some improvements and corrections for the prefix/suffix support now, maybe this will require less memory, maybe a bit more. Gerd ________________________________________ Von: Arndt Röhrig <arndt@speichenkarte.de> Gesendet: Freitag, 21. April 2017 10:40:48 An: Development list for mkgmap; Gerd Petermann Betreff: Re: [mkgmap-dev] mkgmap-r3906 (optimize-index) Hi Gerd, the map tiles from Speiche_Europa 190419 was build with mkgmap-r3906. The index with mkgmap-r3908. (I start it before 3908 was available) Arndt
Gerd Petermann <GPetermann_muenchen@hotmail.com> hat am 21. April 2017 um 10:32 geschrieben:
Hi Arndt,
strange, the index itself is ~ 1GB, that's very close to the size for the default style in r3908. Did you use r3908 or an older version?
Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Arndt Röhrig <arndt@speichenkarte.de> Gesendet: Freitag, 21. April 2017 10:28:25 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] mkgmap-r3906 (optimize-index)
Hi Gerd,
yes, i use 170419.
The OSM-Data is downloaded by "Geofabrik" (europe). To be added 5GB SRTM data. osm and srtm data merged with the splitter.
Arndt
Gerd Petermann <GPetermann_muenchen@hotmail.com> hat am 21. April 2017 um 07:34 geschrieben:
Hi Arndt,
thanks for the flowers ;-) I am still surprised that your style requires so much more memory compared to the default style. Did you use the scripts in SpeicheFabrik_Steuerdateien170419.zip for this?
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Arndt Röhrig <arndt@speichenkarte.de> Gesendet: Mittwoch, 19. April 2017 17:04:43 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] mkgmap-r3906 (optimize-index)
Thank you for that great work!
Now my old nostalgia PC is able to create the index for Speiche_Europa!
My PC has only 6GB RAM.
-Xmx5300M -> fails with "java heap space error"
-Xmx10500M -> fails with "overflowed directory with max block".
* Xmx10500M and Option --block-size=65536 -> That works!
Its better to run mkgmap 2 times. First step builds the maptiles with Xmx3000M, so that java not use the harddisk to swap, cause that makes the PC very slow. The ovm-work files may not be deleted for the overview map.
The second step with -Xmx10500M builds the index and the overview map. The taskmanager show maximal ~9GB in use. The Speiche_Europa map has ~20GB.
Special thanks to Gerd, who show me many things to improve my map building procedere!
Best regards
Arndt
speichenkarte.de
.
Gerd Petermann <GPetermann_muenchen@hotmail.com> hat am 18. April 2017 um 16:41 geschrieben:
Hi all,
as a follow up: In r3907 and r3908 I have coded the merge sort for roads and pois. Now -Xmx4000 was easily enough to create the index for Europe. r3906 failed with OutOfMemoryError even with -Xmx5000 . So I think memory is no longer a problem unless you want to create an index for planet ;-) No other changes, means r3908 can be used to create index for *.img files created with r3906.
Gerd
________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Gerd Petermann <GPetermann_muenchen@hotmail.com> Gesendet: Montag, 17. April 2017 20:05:15 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: [mkgmap-dev] mkgmap-r3906 (optimize-index)
Hi all,
with the help of Steve I fixed some problems with the index, esp. sorting of road names with different speliing of Straße caused a lot of problems, like Ahornstraße (Germany) and Ahornstrasse (Switzerland).
I think address / road search works very well now, at least with west european languages. I tried various combinations of options like --latin1 / --unicode, --lower-case, --x-split-name-index both in MapSource and on my Oregon 600 and always got what I expected.
IMPORTANT: If you try this version, please make sure that you also compile the img files with this version so that the changes in the sort are used everywhere.
The branch also reduces peak memory compared to trunk and because of that it is faster when creating large indexes, but speed is probably not so important here. The created index is a bit smaller although it now also contains roads with an empty string as first label.
If memory is still an issue for you when compiling the index for large maps I can try to implement a merge sort which would only create the - heap consuming - sort keys for a rather small number (e.g. 100.000) roads and sort those and finally merge the parts.
If you know special cases which don't work with r3890 trunk please try the branch and let me know if something might be improved. I think it is a big step forward, but there may still be special cases with other languages.
I used a small set of only 4 tiles to test functionality and compiled index for Europe (compiled with default style) (>1600 tiles) with -Xmx6800m and --x-split-name-index
Gerd
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
El 17/04/17 a las 20:05, Gerd Petermann escribió:
Hi all,
with the help of Steve I fixed some problems with the index, esp. sorting of road names with different speliing of Straße caused a lot of problems, like Ahornstraße (Germany) and Ahornstrasse (Switzerland).
I think address / road search works very well now, at least with west european languages. I tried various combinations of options like --latin1 / --unicode, --lower-case, --x-split-name-index both in MapSource and on my Oregon 600 and always got what I expected.
IMPORTANT: If you try this version, please make sure that you also compile the img files with this version so that the changes in the sort are used everywhere.
The branch also reduces peak memory compared to trunk and because of that it is faster when creating large indexes, but speed is probably not so important here. The created index is a bit smaller although it now also contains roads with an empty string as first label.
If memory is still an issue for you when compiling the index for large maps I can try to implement a merge sort which would only create the - heap consuming - sort keys for a rather small number (e.g. 100.000) roads and sort those and finally merge the parts.
If you know special cases which don't work with r3890 trunk please try the branch and let me know if something might be improved. I think it is a big step forward, but there may still be special cases with other languages.
I used a small set of only 4 tiles to test functionality and compiled index for Europe (compiled with default style) (>1600 tiles) with -Xmx6800m and --x-split-name-index
Gerd I could not yet test it in deep, but first few searches seem to work pretty well both on MapSource and nuvi
Hi Carlos, thanks for testing. I am now looking again at the details for the prefix/suffix support. Gerd ________________________________________ Von: mkgmap-dev <mkgmap-dev-bounces@lists.mkgmap.org.uk> im Auftrag von Carlos Dávila <cdavilam@orangecorreo.es> Gesendet: Dienstag, 18. April 2017 23:29:37 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] mkgmap-r3906 (optimize-index) El 17/04/17 a las 20:05, Gerd Petermann escribió:
Hi all,
with the help of Steve I fixed some problems with the index, esp. sorting of road names with different speliing of Straße caused a lot of problems, like Ahornstraße (Germany) and Ahornstrasse (Switzerland).
I think address / road search works very well now, at least with west european languages. I tried various combinations of options like --latin1 / --unicode, --lower-case, --x-split-name-index both in MapSource and on my Oregon 600 and always got what I expected.
IMPORTANT: If you try this version, please make sure that you also compile the img files with this version so that the changes in the sort are used everywhere.
The branch also reduces peak memory compared to trunk and because of that it is faster when creating large indexes, but speed is probably not so important here. The created index is a bit smaller although it now also contains roads with an empty string as first label.
If memory is still an issue for you when compiling the index for large maps I can try to implement a merge sort which would only create the - heap consuming - sort keys for a rather small number (e.g. 100.000) roads and sort those and finally merge the parts.
If you know special cases which don't work with r3890 trunk please try the branch and let me know if something might be improved. I think it is a big step forward, but there may still be special cases with other languages.
I used a small set of only 4 tiles to test functionality and compiled index for Europe (compiled with default style) (>1600 tiles) with -Xmx6800m and --x-split-name-index
Gerd I could not yet test it in deep, but first few searches seem to work pretty well both on MapSource and nuvi
mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
participants (3)
-
Arndt Röhrig -
Carlos Dávila -
Gerd Petermann