Memory Consumption on Index Creation
1. I noticed that (at least for my style-file) 7500MB of RAM available does not suffice to compile a map of Europe including index! However if I recreate the index in another go, then it is no problem. How comes? The index creation seems to be completly independent of the maptiles creation. Is mkgmap leaving stuff inside the memory, that is in reality not needed anymore? 2. Also as running out of memory primarily happens on index creation, it would be great if mkgmap writes out the index under a temporary name, and once it has passed correctly, renames the file. Else it is not possible to easily assess in a bash/batch script whether mkgmap completed the map correctly or not (and hence one uploads broken maps if everything automated). Or at least I don't see any way to find out whether it passed correctly or not.
1. I noticed that (at least for my style-file) 7500MB of RAM available does not suffice to compile a map of Europe including index! However if I recreate the index in another go, then it is no problem.
How comes? The index creation seems to be completly independent of the maptiles creation. Is mkgmap leaving stuff inside the memory, that is in reality not needed anymore?
I think you have listed your style and your mkgmap parameters on your website. Can you give a link to them? That makes it easier to find the problem. And important: Do you use any patches? WanMil
2. Also as running out of memory primarily happens on index creation, it would be great if mkgmap writes out the index under a temporary name, and once it has passed correctly, renames the file. Else it is not possible to easily assess in a bash/batch script whether mkgmap completed the map correctly or not (and hence one uploads broken maps if everything automated). Or at least I don't see any way to find out whether it passed correctly or not.
On 29.08.2011 20:52, WanMil wrote:
1. I noticed that (at least for my style-file) 7500MB of RAM available does not suffice to compile a map of Europe including index! However if I recreate the index in another go, then it is no problem.
How comes? The index creation seems to be completly independent of the maptiles creation. Is mkgmap leaving stuff inside the memory, that is in reality not needed anymore?
I think you have listed your style and your mkgmap parameters on your website. Can you give a link to them? That makes it easier to find the problem.
Subversion * Path: https://svn.origo.ethz.ch/openmtbmap/ * Web SVN: http://svn.origo.ethz.ch/wsvn/openmtbmap/ * ViewVC: http://svn.origo.ethz.ch/viewvc/openmtbmap/ * Nightly SVN Dump: http://svn.origo.ethz.ch/dump/openmtbmap/latest.tar.gz And here is the specific options on full creation (crashes low on java heap space, passes if Xmx11000M is given, but I only have 12GB RAM on a testmachine on my build sever there is only about 7800M before it seriously slows down -- this is running against Europe extract from geofabrik): c:\OpenMTBMap\maps>start /low /b /wait java -ea -jar -Xmx7500M c:\openmtbmap\mkgmap.jar --max-jobs=4 --generate-sea=extend-sea-sectors,close-gaps=6000,floodblocker,fbgap=60,fbthres=200,fbratio=0.6 --latin1 "--style-file=c:\openmtbmap\new4" --reduce-point-density=4 --nsis --index --transparent --adjust-turn-headings --add-pois-to-areas --ignore-maxspeeds --x-reduce-point-density-polygon=8 --link-pois-to-ways --ignore-turn-restrictions --min-size-polygon=15 --remove-short-arcs=4 --description=openmtbmap_eu --merge-l ines --location-autofill=1 --route --country-abbr=eu --country-name=europe --mapname=65500000 --family-id=6550 --product-id=1 --series-name=openmtbmap_europe_29.08.2011 --family-name=mtbmap_eu_29.08.2011 --tdbfile --overview-mapname=mapset --keep-going --area-name="europe_29.08.2011_openmtbmap.org" -c c:\openmtbmap\maps\template.europe 1>NUL versus the options for index only creation (works well, I can even reduce Xmx to about 4500M): c:\OpenMTBMap\maps>start /low /b /wait java -ea -jar -Xmx7500M c:\openmtbmap\mkgmap.jar --nsis --index --transparent --description=openmtbmap_eu --location-autofill=1 --country-abbr=eu --country-name=europe --mapname=65500000 --family-id=6550 --product-id=1 --series-name=openmtbmap_europe_29.08.2011 --family-name=mtbmap_eu_29.08.2011 --tdbfile --overview-mapname=mapset --keep-going --area-name="europe_29.08.2011_openmtbmap.org" 6550*.img
And important: Do you use any patches?
yes listed here: http://svn.origo.ethz.ch/wsvn/openmtbmap/mkgmap_patches.patch but they shouldn't affect the index creation. Maybe it's again an issue of assigning names like ft to unnamed footways. However there must still be some memory leak if creating the index standalone it needs only about 50% of memory. Actually it would be really great if the index creation could be lighter on memory, maybe using a temporary file for buffer?
WanMil
2. Also as running out of memory primarily happens on index creation, it would be great if mkgmap writes out the index under a temporary name, and once it has passed correctly, renames the file. Else it is not possible to easily assess in a bash/batch script whether mkgmap completed the map correctly or not (and hence one uploads broken maps if everything automated). Or at least I don't see any way to find out whether it passed correctly or not.
On 29/08/11 08:31, Felix Hartmann wrote:
1. I noticed that (at least for my style-file) 7500MB of RAM available does not suffice to compile a map of Europe including index! However if I recreate the index in another go, then it is no problem.
How comes? The index creation seems to be completly independent of the maptiles creation. Is mkgmap leaving stuff inside the memory, that is in reality not needed anymore?
Yes, the index is completely independent of the map tile creation phase and all the memory used for a single map tile is released after it is finished. Profiling confirmed in my tests that all the memory was indeed released. Also if the memory was not released you would see gradually increasing memory usage the more tiles that were compiled at once. Do you see an evidence of that happening? So I can't reproduce this at the moment. I do have some ideas for some reduction in the memory use by the index generation. ..Steve
On 30.08.2011 17:23, Steve Ratcliffe wrote:
On 29/08/11 08:31, Felix Hartmann wrote:
1. I noticed that (at least for my style-file) 7500MB of RAM available does not suffice to compile a map of Europe including index! However if I recreate the index in another go, then it is no problem.
How comes? The index creation seems to be completly independent of the maptiles creation. Is mkgmap leaving stuff inside the memory, that is in reality not needed anymore?
Yes, the index is completely independent of the map tile creation phase and all the memory used for a single map tile is released after it is finished. could it be that the index starts to be worked one, when the first of the 4 worker threads (to compile the maps) is finished, with 3 still working and hence overshooting the memory limit?? I really cannot see why it would have failed me 4 times in a row on a single call, but single index creation allways worked...
I have the problem that on my local machine, it ain't powerful enough to compile all of Europe, so I cannot test really what happens (like decreasing the Xmx value to values that crash the index creation alone, are so low, that I cannot compile the maps anyhow). On the server I use to render the map, however I only have a limited account, and can not see actual memory consumption.
Profiling confirmed in my tests that all the memory was indeed released.
Also if the memory was not released you would see gradually increasing memory usage the more tiles that were compiled at once. Do you see an evidence of that happening?
So I can't reproduce this at the moment.
I do have some ideas for some reduction in the memory use by the index generation.
Well that would be great. It is currently using much more memory than any other tool in the chain and growing faster than any other memory consumption (for running mkgmap with 4 threads, about 2.5GB is enough, for splitting Europe 3GB is enough, however for index creation below 4.5GB it is a no go). Best would even be if on single index creation, one could specify an mdr file for existing maps, and just add stuff for maps that you want to add to the compilation (use case, user has a map of Germany in several .img with mdr/mdx and wants to add contourlines into the mapset, it would be great if the index of the Germany map could be given, and the contourline maps just added to the mdr without needing to regenerate the full mdr -- don't know the format well enough, but would hope that this is possible..). Sorry for taking so long to respond, I wanted to test it out locally, and didn't have a working build environment anymore, hence I took some time. Regarding the -XX:+UseCompressedOops option I'll add it to my commandline.... Felix
On 06/09/11 00:17, Felix Hartmann wrote:
could it be that the index starts to be worked one, when the first of the 4 worker threads (to compile the maps) is finished, with 3 still working and hence overshooting the memory limit?? I really cannot see why it would have failed me 4 times in a row on a single call, but single index creation allways worked...
Maybe - if something else is failing while tiles are being generated. If you apply the attached patch and run, it will print out how much memory is available, what threads are running and version information just before it starts the combining step. On my machine it looks like this: About to start combine steps Thread[Signal Dispatcher,9,system] RUNNABLE Thread[Reference Handler,10,system] WAITING at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:485) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116) Thread[main,5,main] RUNNABLE at java.lang.Thread.dumpThreads(Native Method) at java.lang.Thread.getAllStackTraces(Thread.java:1530) at uk.me.parabola.mkgmap.main.Main.endOptions(Main.java:423) at uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.java:126) at uk.me.parabola.mkgmap.main.Main.main(Main.java:132) Thread[Finalizer,8,system] WAITING at java.lang.Object.wait(Native Method) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159) Memory 2M 105M 881M Java Version 1.6.0_22 Sun Microsystems Inc., on Linux 2.6.40.3-0.fc15.x86_64
I do have some ideas for some reduction in the memory use by the index generation.
Well that would be great. It is currently using much more memory than
I looked into this a bit more and found that the simple ideas wouldn't save much memory unfortunately.
Best would even be if on single index creation, one could specify an mdr file for existing maps, and just add stuff for maps that you want to add to the compilation (use case, user has a map of Germany in several .img with mdr/mdx and wants to add contourlines into the mapset, it would be great if the index of the Germany map could be given, and the contourline maps just added to the mdr without needing to regenerate the full mdr -- don't know the format well enough, but would hope that this is possible..).
It would be possible, but the existing file would have to be read so that the new information could be merged in the correct sorted order. It would be a fair amount of work and would use a similar amount of memory as doing the whole lot. ..Steve
Hi steve, Okay strange, tonight it went through without problems (255M reserved when starting). Maybe something else was wrong then... However could you change mkgmap to delete broken mdr files if it crashes on low memory?? That would be a real troublesaver!!!! I don't know how to find out using a batch/bash if the mdr file is okay or not otherwise! (I can't simply look at the size, as for the Europe file when it crashed it was always around 3MB, while simple maps a working mdr is much smaller). Felix /About to start combine steps/ /Thread[Reference Handler,10,system] WAITING/ / at java.lang.Object.wait(Native Method)/ / at java.lang.Object.wait(Object.java:485)/ / at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)/ /Thread[Finalizer,8,system] WAITING/ / at java.lang.Object.wait(Native Method)/ / at java.lang.ref.ReferenceQueue.remove(Unknown Source)/ / at java.lang.ref.ReferenceQueue.remove(Unknown Source)/ / at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)/ /Thread[Attach Listener,5,system] RUNNABLE/ /Thread[main,5,main] RUNNABLE/ / at java.lang.Thread.dumpThreads(Native Method)/ / at java.lang.Thread.getAllStackTraces(Unknown Source)/ / at uk.me.parabola.mkgmap.main.Main.endOptions(Main.java:423)/ / at uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.java:126)/ / at uk.me.parabola.mkgmap.main.Main.main(Main.java:132)/ /Thread[Signal Dispatcher,9,system] RUNNABLE/ /Memory 255M 7569M 7569M/ /Java Version 1.6.0_21 Sun Microsystems Inc., on Windows Server 2008 R2 6.1/ On 06.09.2011 10:53, Steve Ratcliffe wrote:
On 06/09/11 00:17, Felix Hartmann wrote:
could it be that the index starts to be worked one, when the first of the 4 worker threads (to compile the maps) is finished, with 3 still working and hence overshooting the memory limit?? I really cannot see why it would have failed me 4 times in a row on a single call, but single index creation allways worked...
Maybe - if something else is failing while tiles are being generated.
If you apply the attached patch and run, it will print out how much memory is available, what threads are running and version information just before it starts the combining step.
On my machine it looks like this:
About to start combine steps Thread[Signal Dispatcher,9,system] RUNNABLE Thread[Reference Handler,10,system] WAITING at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:485) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116) Thread[main,5,main] RUNNABLE at java.lang.Thread.dumpThreads(Native Method) at java.lang.Thread.getAllStackTraces(Thread.java:1530) at uk.me.parabola.mkgmap.main.Main.endOptions(Main.java:423) at uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.java:126) at uk.me.parabola.mkgmap.main.Main.main(Main.java:132) Thread[Finalizer,8,system] WAITING at java.lang.Object.wait(Native Method) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159) Memory 2M 105M 881M Java Version 1.6.0_22 Sun Microsystems Inc., on Linux 2.6.40.3-0.fc15.x86_64
I do have some ideas for some reduction in the memory use by the index generation.
Well that would be great. It is currently using much more memory than
I looked into this a bit more and found that the simple ideas wouldn't save much memory unfortunately.
Best would even be if on single index creation, one could specify an mdr file for existing maps, and just add stuff for maps that you want to add to the compilation (use case, user has a map of Germany in several .img with mdr/mdx and wants to add contourlines into the mapset, it would be great if the index of the Germany map could be given, and the contourline maps just added to the mdr without needing to regenerate the full mdr -- don't know the format well enough, but would hope that this is possible..).
It would be possible, but the existing file would have to be read so that the new information could be merged in the correct sorted order. It would be a fair amount of work and would use a similar amount of memory as doing the whole lot.
..Steve
OK. 255 sounds a bit high but shouldn't be a problem all the same. You may have noticed that I've started a index-memory branch. Currently this does little more than print out memory use at various point during the process. It would be useful to run it, particularly if it fails again. ..Steve
Hi Felix
However could you change mkgmap to delete broken mdr files if it crashes on low memory?? That would be a real troublesaver!!!!
I've implemented this idea on the index-memory branch. The mdr file is not written unless the process completes successfully, so you should never see a broken file due to an out of memory or any other fatal error. On the other hand if there is an existing mdr file it will not be touched, so make sure you remove any pre-existing file. Cheers ..Steve
Hello list, sometimes some streets are not findable, when you use the city to find the streets. When you leave the city out and search in every city, you can find them. The funny thing is, when you use only the tile which include the street, you can find the street also with the city-search. Also if you surround this tile with some other tiles you can find the street after entering the city. Maybe somebody can reproduce this error. # List of areas # Generated Thu Sep 08 09:54:31 CEST 2011 # Named 5 63240345: 2381824,305152 to 2410496,321536 # : 51.108398,6.547852 to 51.723633,6.899414 # Named 6 63240346: 2357248,346112 to 2385920,387072 # : 50.581055,7.426758 to 51.196289,8.305664 # Named 7 63240347: 2394112,321536 to 2398208,346112 # : 51.372070,6.899414 to 51.459961,7.426758 # Named 8 63240348: 2385920,346112 to 2398208,387072 # : 51.196289,7.426758 to 51.459961,8.305664 # Named 9 63240349: 2365440,309248 to 2381824,346112 # : 50.756836,6.635742 to 51.108398,7.426758 # Named x 63240350: 2381824,321536 to 2394112,346112 # : 51.108398,6.899414 to 51.372070,7.426758 The street we are searching is the "Engelberthusstraße" in Wipperfürth (this problem also occur with other streets like the "Pennstraße" in München) I gave the tiles different numbers. The tile which include the Engelbertusstraße (Wipperfürth) is named x an used in every map. http://img198.imageshack.us/img198/1058/garminmap.png -: Street couldn't be found (only be found when you search in every city) +:Street could be found (with city-search) 69: - 5789: + 579: + 569: - 59: + 6789: - 678: + 5678: + 56789: - As you can see, when tile 6 and 9 is included, the street is only found when you search in every city. You can find a zip-file including all this files here: http://snailrun.de/wipperfuerth.zip Cheers, Martin
What happens happens if you change Straße Tor Strasse in the original data On Sep 24, 2011 4:11 PM, "Martin" <mkmap@snailrun.de> wrote:
Hello list,
sometimes some streets are not findable, when you use the city to find the
streets. When you leave the city out and search in every city, you can find them. The funny thing is, when you use only the tile which include the street, you can find the street also with the city-search. Also if you surround this tile with some other tiles you can find the street after entering the city. Maybe somebody can reproduce this error.
# List of areas # Generated Thu Sep 08 09:54:31 CEST 2011 # Named 5 63240345: 2381824,305152 to 2410496,321536 # : 51.108398,6.547852 to 51.723633,6.899414 # Named 6 63240346: 2357248,346112 to 2385920,387072 # : 50.581055,7.426758 to 51.196289,8.305664 # Named 7 63240347: 2394112,321536 to 2398208,346112 # : 51.372070,6.899414 to 51.459961,7.426758 # Named 8 63240348: 2385920,346112 to 2398208,387072 # : 51.196289,7.426758 to 51.459961,8.305664 # Named 9 63240349: 2365440,309248 to 2381824,346112 # : 50.756836,6.635742 to 51.108398,7.426758 # Named x 63240350: 2381824,321536 to 2394112,346112 # : 51.108398,6.899414 to 51.372070,7.426758
The street we are searching is the "Engelberthusstraße" in Wipperfürth
(this problem also occur with other streets like the "Pennstraße" in München)
I gave the tiles different numbers. The tile which include the Engelbertusstraße (Wipperfürth) is named x an used in every map. http://img198.imageshack.us/img198/1058/garminmap.png -: Street couldn't be found (only be found when you search in every city) +:Street could be found (with city-search)
69: - 5789: + 579: + 569: - 59: + 6789: - 678: + 5678: + 56789: -
As you can see, when tile 6 and 9 is included, the street is only found when you search in every city. You can find a zip-file including all this files here: http://snailrun.de/wipperfuerth.zip
Cheers, Martin
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
On the Openfietsmap Germany (see http://www.openfietsmap.nl) I have no problems finding the "Engelberthusstraße" in Wipperfürth and "Pennstraße" in München. However, I noticed the same problem with other streets, it looks like it depends how the tiles are splitted. ---------- "Martin" wrote: The street we are searching is the "Engelberthusstraße" in Wipperfürth (this problem also occur with other streets like the "Pennstraße" in München)
Minko, thanks for your hint. This problem comes indeed from the tile border. I shifted the lower and right tile-border and now I can find the Engelbertusstraße, even with all surrounding tiles. Any idea how we can fix this? Cheers Martin # List of areas # Generated Thu Sep 08 09:54:31 CEST 2011 # 63240345: 2379083,305152 to 2410496,321536 # : 51.0499968,6.547852 to 51.723633,6.899414 63240346: 2357248,347658 to 2385920,387072 # : 50.581055,7.459991 to 51.196289,8.305664 63240347: 2394112,321536 to 2398208,347658 # : 51.372070,6.899414 to 51.459961,7.459991 63240348: 2385920,347658 to 2398208,387072 # : 51.196289,7.459991 to 51.459961,8.305664 63240349: 2365440,309248 to 2379083,347658 # : 50.756836,6.635742 to 51.0499968,7.459991 63240350: 2379083,321536 to 2394112,347658 # : 51.0499968,6.899414 to 51.372070,7.459991 Am 24.09.2011 um 17:58 schrieb Minko:
On the Openfietsmap Germany (see http://www.openfietsmap.nl) I have no problems finding the "Engelberthusstraße" in Wipperfürth and "Pennstraße" in München. However, I noticed the same problem with other streets, it looks like it depends how the tiles are splitted.
---------- "Martin" wrote: The street we are searching is the "Engelberthusstraße" in Wipperfürth (this problem also occur with other streets like the "Pennstraße" in München) _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi
As you can see, when tile 6 and 9 is included, the street is only found when you search in every city. You can find a zip-file including all this files here: http://snailrun.de/wipperfuerth.zip
I've an interesting difference between the ones that work and the ones that don't. I need the original MDR file to look into it further though, would it be possible for you to upload it? In all the cases that work MdrCheck shows one empty entry in mdr20, eg: mdr20 62252 city=WIPPERFÜRTH r= c=, from=62252 to=62252 empty mdr20 62252 city=WIPPERFÜRTH r= c=, from=62252 to=62669 and the ones that don't work have two empty entries, for example: mdr20 51677 city=WIPPERFÜRTH r= c=, from=51677 to=51677 empty mdr20 51677 city=WIPPERFÜRTH r= c=, from=51677 to=51677 empty mdr20 51677 city=WIPPERFÜRTH r= c=, from=51677 to=52259 without the original mdr file though, I can't tell what is wrong. ..Steve
Hello Steve, unfortunately I have deleted all the files, but I have created them again with the geofabrik-datas from today. Search for the Engelbertusstraße works: http://snailrun.de/59.zip Didn't work: http://snailrun.de/56789.zip All zip-files include the gmapsupp-image and the mdr-file etc. How can I read the the different mdr-tables? Thanks Martin Am 25.09.2011 um 21:18 schrieb Steve Ratcliffe:
Hi
As you can see, when tile 6 and 9 is included, the street is only found when you search in every city. You can find a zip-file including all this files here: http://snailrun.de/wipperfuerth.zip
I've an interesting difference between the ones that work and the ones that don't. I need the original MDR file to look into it further though, would it be possible for you to upload it?
In all the cases that work MdrCheck shows one empty entry in mdr20, eg:
mdr20 62252 city=WIPPERFÜRTH r= c=, from=62252 to=62252 empty mdr20 62252 city=WIPPERFÜRTH r= c=, from=62252 to=62669
and the ones that don't work have two empty entries, for example:
mdr20 51677 city=WIPPERFÜRTH r= c=, from=51677 to=51677 empty mdr20 51677 city=WIPPERFÜRTH r= c=, from=51677 to=51677 empty mdr20 51677 city=WIPPERFÜRTH r= c=, from=51677 to=52259
without the original mdr file though, I can't tell what is wrong.
..Steve _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Martin
unfortunately I have deleted all the files, but I have created them again with the geofabrik-datas from today. Search for the Engelbertusstraße works: http://snailrun.de/59.zip Didn't work: http://snailrun.de/56789.zip
Thanks very much. I believe I found the problem (or at least a problem). Could you try out the attached patch please? In the mdr20 section the streets were being sorted not just by city name, but also by map number. So it was looking like this: APPLE STREET ZOO ROAD BEE ROAD ZEBRA ROAD with Apple and Bee in the same city but different maps, instead of like this: APPLE STREET BEE ROAD ZEBRA ROAD ZOO ROAD I previously suggested that the city being in three tiles rather than two might have been a problem, but I believe that was not particularly relevant. I think that some streets will be un-findable whenever a city falls across more than one tile.
How can I read the the different mdr-tables?
In the display project (code at http://svn.mkgmap.org.uk/display/trunk) there are a number of utilities that display various aspects of the file. test.display.MdrDisplay displays the structure of an MDR file This is reliable since it prints out exactly what is in the file, but the interpretation of mdr files that are contained within gmapsup.img files may be incomplete or incorrect. Pass the name of the file that is or contains an MDR file. Followed by numbers if you want to limit the sections that are printed. test.display.MdrCheck reads an MDR file and the associated .img files that it was created from. It cross checks the data between the index and the img files and prints in a form that shows the meaning rather than the literal layout of the file. It can be mis-leading since I just add to it when needed and some changes I have made to mkgmap are not reflected in this code. So some of the errors it prints are not errors. Pass the name of a file (an MDR file, xxx_mdr.img or gmapsupp.img) Followed by options: --print 5,7 to just print sections 5 and 7 --errors to just print the errors. ..Steve
Hi Steve, thank you very much for this patch. Now I can find the Engelbertusstraße (and some more). I know, it's just a small step, but I think now are more streets findable. I'm making a map of whole germany and test it for a while. Afterwards I will give you a feedback. But please give me 1-2 days for this. Cheers, Martin P.S.: If you would like to take a look at the mdr-table you can download the map and the other files under: http://snailrun.de/56789_patch.zip Am 26.09.2011 um 12:33 schrieb Steve Ratcliffe:
Hi Martin
unfortunately I have deleted all the files, but I have created them again with the geofabrik-datas from today. Search for the Engelbertusstraße works: http://snailrun.de/59.zip Didn't work: http://snailrun.de/56789.zip
Thanks very much. I believe I found the problem (or at least a problem). Could you try out the attached patch please?
In the mdr20 section the streets were being sorted not just by city name, but also by map number. So it was looking like this:
APPLE STREET ZOO ROAD BEE ROAD ZEBRA ROAD
with Apple and Bee in the same city but different maps, instead of like this:
APPLE STREET BEE ROAD ZEBRA ROAD ZOO ROAD
I previously suggested that the city being in three tiles rather than two might have been a problem, but I believe that was not particularly relevant. I think that some streets will be un-findable whenever a city falls across more than one tile.
How can I read the the different mdr-tables?
In the display project (code at http://svn.mkgmap.org.uk/display/trunk) there are a number of utilities that display various aspects of the file.
test.display.MdrDisplay displays the structure of an MDR file This is reliable since it prints out exactly what is in the file, but the interpretation of mdr files that are contained within gmapsup.img files may be incomplete or incorrect.
Pass the name of the file that is or contains an MDR file. Followed by numbers if you want to limit the sections that are printed.
test.display.MdrCheck reads an MDR file and the associated .img files that it was created from. It cross checks the data between the index and the img files and prints in a form that shows the meaning rather than the literal layout of the file. It can be mis-leading since I just add to it when needed and some changes I have made to mkgmap are not reflected in this code. So some of the errors it prints are not errors.
Pass the name of a file (an MDR file, xxx_mdr.img or gmapsupp.img) Followed by options: --print 5,7 to just print sections 5 and 7 --errors to just print the errors.
..Steve <mdr20_sort.patch>_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi Martin
unfortunately I have deleted all the files, but I have created them again with the geofabrik-datas from today. Search for the Engelbertusstraße works: http://snailrun.de/59.zip Didn't work: http://snailrun.de/56789.zip
Thanks very much. I believe I found the problem (or at least a problem). Could you try out the attached patch please?
In the mdr20 section the streets were being sorted not just by city name, but also by map number. So it was looking like this:
APPLE STREET ZOO ROAD BEE ROAD ZEBRA ROAD
with Apple and Bee in the same city but different maps, instead of like this:
APPLE STREET BEE ROAD ZEBRA ROAD ZOO ROAD
I previously suggested that the city being in three tiles rather than two might have been a problem, but I believe that was not particularly relevant. I think that some streets will be un-findable whenever a city falls across more than one tile.
How can I read the the different mdr-tables?
In the display project (code at http://svn.mkgmap.org.uk/display/trunk) there are a number of utilities that display various aspects of the file.
test.display.MdrDisplay displays the structure of an MDR file This is reliable since it prints out exactly what is in the file, but the interpretation of mdr files that are contained within gmapsup.img files may be incomplete or incorrect.
Pass the name of the file that is or contains an MDR file. Followed by numbers if you want to limit the sections that are printed.
test.display.MdrCheck reads an MDR file and the associated .img files that it was created from. It cross checks the data between the index and the img files and prints in a form that shows the meaning rather than the literal layout of the file. It can be mis-leading since I just add to it when needed and some changes I have made to mkgmap are not reflected in this code. So some of the errors it prints are not errors.
Pass the name of a file (an MDR file, xxx_mdr.img or gmapsupp.img) Followed by options: --print 5,7 to just print sections 5 and 7 --errors to just print the errors.
..Steve
The patch works great! I tested Germany and found all my testcases which didn't work before. I also tried to compile a complete europe map but 3g memory wasn't enough to create the mdr file. I've seen that you committed a low memory variant in the index-memory branch but that ended with a 227mb big img9136486325470304255.tmp temporary file and without any error message. WanMil
Hello Steve, your patch works perfect. I can find a lot of more streets now and didn't find new problems. I would suggest to commit this patch to the trunk. Cheers, Martin Am 29.09.2011 um 23:32 schrieb WanMil:
Hi Martin
unfortunately I have deleted all the files, but I have created them again with the geofabrik-datas from today. Search for the Engelbertusstraße works: http://snailrun.de/59.zip Didn't work: http://snailrun.de/56789.zip
Thanks very much. I believe I found the problem (or at least a problem). Could you try out the attached patch please?
In the mdr20 section the streets were being sorted not just by city name, but also by map number. So it was looking like this:
APPLE STREET ZOO ROAD BEE ROAD ZEBRA ROAD
with Apple and Bee in the same city but different maps, instead of like this:
APPLE STREET BEE ROAD ZEBRA ROAD ZOO ROAD
I previously suggested that the city being in three tiles rather than two might have been a problem, but I believe that was not particularly relevant. I think that some streets will be un-findable whenever a city falls across more than one tile.
How can I read the the different mdr-tables?
In the display project (code at http://svn.mkgmap.org.uk/display/trunk) there are a number of utilities that display various aspects of the file.
test.display.MdrDisplay displays the structure of an MDR file This is reliable since it prints out exactly what is in the file, but the interpretation of mdr files that are contained within gmapsup.img files may be incomplete or incorrect.
Pass the name of the file that is or contains an MDR file. Followed by numbers if you want to limit the sections that are printed.
test.display.MdrCheck reads an MDR file and the associated .img files that it was created from. It cross checks the data between the index and the img files and prints in a form that shows the meaning rather than the literal layout of the file. It can be mis-leading since I just add to it when needed and some changes I have made to mkgmap are not reflected in this code. So some of the errors it prints are not errors.
Pass the name of a file (an MDR file, xxx_mdr.img or gmapsupp.img) Followed by options: --print 5,7 to just print sections 5 and 7 --errors to just print the errors.
..Steve
The patch works great! I tested Germany and found all my testcases which didn't work before.
I also tried to compile a complete europe map but 3g memory wasn't enough to create the mdr file. I've seen that you committed a low memory variant in the index-memory branch but that ended with a 227mb big img9136486325470304255.tmp temporary file and without any error message.
WanMil _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
On 29/09/11 22:40, Martin wrote:
Hello Steve,
your patch works perfect. I can find a lot of more streets now and didn't find new problems. I would suggest to commit this patch to the trunk.
Good. Thanks for the very clear test case -- I'd probably never have found the problem without it! ..Steve
Hi
The patch works great! I tested Germany and found all my testcases which didn't work before.
Great!
I also tried to compile a complete europe map but 3g memory wasn't enough to create the mdr file. I've seen that you committed a low memory variant in the index-memory branch but that ended with a 227mb big img9136486325470304255.tmp temporary file and without any error message.
Is this on Windows? Is the .tmp file a complete osmmap_mdr.img file -- if you rename it, does it work? ..Steve
Hi
The patch works great! I tested Germany and found all my testcases which didn't work before.
Great!
I also tried to compile a complete europe map but 3g memory wasn't enough to create the mdr file. I've seen that you committed a low memory variant in the index-memory branch but that ended with a 227mb big img9136486325470304255.tmp temporary file and without any error message.
Is this on Windows? Is the .tmp file a complete osmmap_mdr.img file -- if you rename it, does it work?
..Steve
It's Windows 7 64bit. I noted that some of the tmp files are created in the start directory of mkgmap whereas the mdrNNNN.tmp file is created in the target img directory. When renaming the imgXXXX.tmp file to osmmap_mdr.mdr the find menu item is disabled. So that's not the whole solution. WanMil
Hi
It's Windows 7 64bit.
Thanks, I've made some changes for Windows and it works for me on Vista (32 bit, but don't think that is important), I was not surprised that there were problems there!
I noted that some of the tmp files are created in the start directory of mkgmap whereas the mdrNNNN.tmp file is created in the target img directory.
Yes, I intend to fix this before merging back to trunk. They should all be in the target directory ..Steve
Hi
It's Windows 7 64bit.
Thanks, I've made some changes for Windows and it works for me on Vista (32 bit, but don't think that is important), I was not surprised that there were problems there!
I noted that some of the tmp files are created in the start directory of mkgmap whereas the mdrNNNN.tmp file is created in the target img directory.
Yes, I intend to fix this before merging back to trunk. They should all be in the target directory
..Steve
That's works now. +1 for merging back to trunk! Good work! WanMil
Hello Steve, don't panic, everything works fine with your patch. I had some sparetime and would like to test the display-tool you recommended below. But I have trouble to compile it. Is there anywhere a documentation for compiling this tool? I'm not a java-developer, more C++-"beginner". I have attached the error.log. Cheers, Martin Am 2011-09-26 12:33, schrieb Steve Ratcliffe:
Hi Martin
unfortunately I have deleted all the files, but I have created them again with the geofabrik-datas from today. Search for the Engelbertusstraße works: http://snailrun.de/59.zip Didn't work: http://snailrun.de/56789.zip
Thanks very much. I believe I found the problem (or at least a problem). Could you try out the attached patch please?
In the mdr20 section the streets were being sorted not just by city name, but also by map number. So it was looking like this:
APPLE STREET ZOO ROAD BEE ROAD ZEBRA ROAD
with Apple and Bee in the same city but different maps, instead of like this:
APPLE STREET BEE ROAD ZEBRA ROAD ZOO ROAD
I previously suggested that the city being in three tiles rather than two might have been a problem, but I believe that was not particularly relevant. I think that some streets will be un-findable whenever a city falls across more than one tile.
How can I read the the different mdr-tables?
In the display project (code at http://svn.mkgmap.org.uk/display/trunk) there are a number of utilities that display various aspects of the file.
test.display.MdrDisplay displays the structure of an MDR file This is reliable since it prints out exactly what is in the file, but the interpretation of mdr files that are contained within gmapsup.img files may be incomplete or incorrect.
Pass the name of the file that is or contains an MDR file. Followed by numbers if you want to limit the sections that are printed.
test.display.MdrCheck reads an MDR file and the associated .img files that it was created from. It cross checks the data between the index and the img files and prints in a form that shows the meaning rather than the literal layout of the file. It can be mis-leading since I just add to it when needed and some changes I have made to mkgmap are not reflected in this code. So some of the errors it prints are not errors.
Pass the name of a file (an MDR file, xxx_mdr.img or gmapsupp.img) Followed by options: --print 5,7 to just print sections 5 and 7 --errors to just print the errors.
..Steve
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
On 14/10/11 08:41, Martin wrote: Hi Martin
don't panic, everything works fine with your patch. Great!
I had some sparetime and would like to test the display-tool you recommended below. But I have trouble to compile it. Is there anywhere a documentation for compiling this tool? I'm not a java-developer, more C++-"beginner". I have attached the error.log.
You need the mkgmap classes too. You can do this in two ways. 1. put a mkgmap.jar in the top directory of display. (/home/me/App/display/mkgmap.jar in your case) 2. have a built mkgmap directory alongside the display directory. eg in your case: /home/me/App/display /home/me/App/mkgmap When you come to run it you also need both sets of classes so the correct command will be something like the following depending on where you put things: java -cp /home/me/App/display/dist/display.jar:/home/me/App/mkgmap/dist/mkgmap.jar test.display.MdrDisplay <filename..> ..Steve
Hi One tip to reduce the memory use when using 64-bit java with a heap size less than 32G is to use the -XX:+UseCompressedOops option. This is a java option so put it along with the -Xmx option. I've still had no luck reproducing the problem you are seeing. ..Steve
participants (6)
-
Felix Hartmann -
Martin -
Martin -
Minko -
Steve Ratcliffe -
WanMil