<div dir="ltr"><div><div><div><div><div>Hello again Burkhard,<br><br></div>Thank you for your persistence in this issue. After looking at it again today, I realized that there must be a memory leak in how we try to map and synchronize ImageJ 1.x ImagePluses with ImageJ2 data structures.<br>
<br></div>Indeed, that was exactly what I found... however, the solution is non-trivial.<br><br></div>I have filed a new issue for this here: <a href="https://github.com/imagej/imagej-legacy/issues/67">https://github.com/imagej/imagej-legacy/issues/67</a>. It is something I would like to resolve quickly, but because it is an involved issue that requires additional investigation and discussion, it's at a lower priority than some necessary grant work this week. <br>
<br></div>Thanks again for the report - it is, at least, good to know where the problem is now.<br></div>- Mark<br></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Jul 14, 2014 at 1:39 PM, Burkhard Höckendorf <span dir="ltr"><<a href="mailto:burkhard.hoeckendorf@web.de" target="_blank">burkhard.hoeckendorf@web.de</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Mark,<br>
<br>
Thanks for the speedy response. I've rewritten my macro to perhaps illustrate my point more clearly. It now opens the file and immediately closes it again, for 50 iterations.<br>
<br>
With scifio enabled I do not get an out of memory error, but it still accumulates memory to the point that it can not open the entire image anymore but instead opens virtual stacks. Without scifio this does not occur.<br>
<br>
Also note that reading the full image into memory without scifio is performance-wise much closer to opening a scifio virtual image than reading the full image with scifio.<br>
<br>
Below is the macro and its output.<br>
Best,<br>
Burkhard<br>
<br>
nOpenTimes = 50;<br>
file = "/path/to/image";<br>
<br>
for (i=1; i<=nOpenTimes; i+=1) {<br>
msToOpen = getTime();<br>
open(file);<br>
msToOpen = getTime() - msToOpen;<br>
<br>
msg = ""+ i +": "+ IJ.currentMemory() +" mem, "+ msToOpen +" ms";<br>
if (is("Virtual Stack"))<br>
msg += " (virtual)";<br>
print(msg);<br>
<br>
close();<br>
}<br>
<br>
Using SCIFIO:<br>
1: 3822648536 mem, 15984 ms<br>
2: <a href="tel:7572956280" value="+17572956280" target="_blank">7572956280</a> mem, 14986 ms<br>
3: 9998931400 mem, 14821 ms<br>
4: 13784264296 mem, 14895 ms<br>
5: 16885256128 mem, 15019 ms<br>
6: 19656427040 mem, 15036 ms<br>
7: 24079096808 mem, 15212 ms<br>
8: 27510832552 mem, 15179 ms<br>
9: 32077394688 mem, 15427 ms<br>
10: 34409304040 mem, 15482 ms<br>
11: 36859610432 mem, 15227 ms<br>
12: 43061552320 mem, 15607 ms<br>
13: 42104714576 mem, 15359 ms<br>
14: 47910156296 mem, 15073 ms<br>
15: 48044651592 mem, 3053 ms (virtual)<br>
16: 48170004824 mem, 2502 ms (virtual)<br>
17: 48295208256 mem, 2522 ms (virtual)<br>
18: 48422832064 mem, 2495 ms (virtual)<br>
19: 48550863920 mem, 2504 ms (virtual)<br>
20: 48228290696 mem, 2615 ms (virtual)<br>
21: 48343976264 mem, 2482 ms (virtual)<br>
22: 48461224352 mem, 2495 ms (virtual)<br>
23: 48577275224 mem, 2482 ms (virtual)<br>
24: 48696021320 mem, 2493 ms (virtual)<br>
25: 48815302152 mem, 2513 ms (virtual)<br>
26: 48512671680 mem, 2682 ms (virtual)<br>
27: 48624380048 mem, 2493 ms (virtual)<br>
28: 48734321624 mem, 2487 ms (virtual)<br>
29: 48847280096 mem, 2504 ms (virtual)<br>
30: 48958189928 mem, 2491 ms (virtual)<br>
31: 49069385672 mem, 2499 ms (virtual)<br>
32: 48787137328 mem, 2614 ms (virtual)<br>
33: 48896061816 mem, 2500 ms (virtual)<br>
34: 49004784912 mem, 2488 ms (virtual)<br>
35: 49116572664 mem, 2516 ms (virtual)<br>
36: 49226035440 mem, 2497 ms (virtual)<br>
37: 49334418896 mem, 2500 ms (virtual)<br>
38: 49074728512 mem, 2660 ms (virtual)<br>
39: 49183254768 mem, 2506 ms (virtual)<br>
40: 49290891152 mem, 2491 ms (virtual)<br>
41: 49399382336 mem, 2499 ms (virtual)<br>
42: 49505843008 mem, 2487 ms (virtual)<br>
43: 49613438808 mem, 2487 ms (virtual)<br>
44: 49721198184 mem, 2481 ms (virtual)<br>
45: 49441266864 mem, 2709 ms (virtual)<br>
46: 49548934656 mem, 2485 ms (virtual)<br>
47: 49656785920 mem, 2501 ms (virtual)<br>
48: 49764127512 mem, 2480 ms (virtual)<br>
49: 49872069552 mem, 2495 ms (virtual)<br>
50: 49979814216 mem, 2482 ms (virtual)<br>
<br>
Not using SCIFIO:<br>
1: <a href="tel:3107094592" value="+13107094592" target="_blank">3107094592</a> mem, 3921 ms<br>
2: <a href="tel:3109153032" value="+13109153032" target="_blank">3109153032</a> mem, 3837 ms<br>
3: <a href="tel:3109836568" value="+13109836568" target="_blank">3109836568</a> mem, 3631 ms<br>
4: <a href="tel:3109512264" value="+13109512264" target="_blank">3109512264</a> mem, 3581 ms<br>
5: <a href="tel:3109478728" value="+13109478728" target="_blank">3109478728</a> mem, 3633 ms<br>
6: <a href="tel:3123692656" value="+13123692656" target="_blank">3123692656</a> mem, 4145 ms<br>
7: 3112714624 mem, 3301 ms<br>
8: <a href="tel:3128373632" value="+13128373632" target="_blank">3128373632</a> mem, 3202 ms<br>
9: 3117798216 mem, 3235 ms<br>
10: <a href="tel:3133595872" value="+13133595872" target="_blank">3133595872</a> mem, 3190 ms<br>
11: <a href="tel:3123048664" value="+13123048664" target="_blank">3123048664</a> mem, 3229 ms<br>
12: <a href="tel:5754899128" value="+15754899128" target="_blank">5754899128</a> mem, 3404 ms<br>
13: <a href="tel:3128308704" value="+13128308704" target="_blank">3128308704</a> mem, 3307 ms<br>
14: 3117759024 mem, 3219 ms<br>
15: 5910573440 mem, 3112 ms<br>
16: 5488969832 mem, 3222 ms<br>
17: 3112824736 mem, 3273 ms<br>
18: <a href="tel:3128647888" value="+13128647888" target="_blank">3128647888</a> mem, 3147 ms<br>
19: 3118094280 mem, 3208 ms<br>
20: 5902184440 mem, 3281 ms<br>
21: <a href="tel:3123372736" value="+13123372736" target="_blank">3123372736</a> mem, 3224 ms<br>
22: 3112825408 mem, 3259 ms<br>
23: <a href="tel:3128648960" value="+13128648960" target="_blank">3128648960</a> mem, 3146 ms<br>
24: 3118100760 mem, 3230 ms<br>
25: <a href="tel:3133941888" value="+13133941888" target="_blank">3133941888</a> mem, 3119 ms<br>
26: 5488982136 mem, 3225 ms<br>
27: 3112820248 mem, 3249 ms<br>
28: <a href="tel:3128655064" value="+13128655064" target="_blank">3128655064</a> mem, 3154 ms<br>
29: 3118110296 mem, 3241 ms<br>
30: <a href="tel:3133956392" value="+13133956392" target="_blank">3133956392</a> mem, 3176 ms<br>
31: <a href="tel:3123381976" value="+13123381976" target="_blank">3123381976</a> mem, 3220 ms<br>
32: 3112831584 mem, 3264 ms<br>
33: <a href="tel:3128664576" value="+13128664576" target="_blank">3128664576</a> mem, 3170 ms<br>
34: 3118103400 mem, 3209 ms<br>
35: <a href="tel:3133956744" value="+13133956744" target="_blank">3133956744</a> mem, 3163 ms<br>
36: <a href="tel:3123387312" value="+13123387312" target="_blank">3123387312</a> mem, 3233 ms<br>
37: 3112827304 mem, 3279 ms<br>
38: <a href="tel:3128664688" value="+13128664688" target="_blank">3128664688</a> mem, 3158 ms<br>
39: 3118115152 mem, 3256 ms<br>
40: <a href="tel:3133956432" value="+13133956432" target="_blank">3133956432</a> mem, 3128 ms<br>
41: 5488994224 mem, 3214 ms<br>
42: <a href="tel:5746873888" value="+15746873888" target="_blank">5746873888</a> mem, 3266 ms<br>
43: <a href="tel:3128666592" value="+13128666592" target="_blank">3128666592</a> mem, 3174 ms<br>
44: 3118109984 mem, 3215 ms<br>
45: <a href="tel:3133949360" value="+13133949360" target="_blank">3133949360</a> mem, 3131 ms<br>
46: <a href="tel:3123388320" value="+13123388320" target="_blank">3123388320</a> mem, 3207 ms<br>
47: 3112843792 mem, 3269 ms<br>
48: <a href="tel:3128683960" value="+13128683960" target="_blank">3128683960</a> mem, 3144 ms<br>
49: <a href="tel:5617941896" value="+15617941896" target="_blank">5617941896</a> mem, 3201 ms<br>
50: 5902216016 mem, 3131 ms<br>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Date: Mon, 14 Jul 2014 08:34:45 -0500<br>
From: Mark Hiner <<a href="mailto:hinerm@gmail.com" target="_blank">hinerm@gmail.com</a>><br>
To: Burkhard H?ckendorf <<a href="mailto:burkhard.hoeckendorf@web.de" target="_blank">burkhard.hoeckendorf@web.de</a>><br>
Cc: "<a href="mailto:scifio@scif.io" target="_blank">scifio@scif.io</a>" <<a href="mailto:scifio@scif.io" target="_blank">scifio@scif.io</a>><br>
Subject: Re: [SCIFIO] SCIFIO does not seem to release memory of closed<br>
images<br>
Message-ID:<br>
<CA+B=<a href="mailto:mGqx47SkUzM_FHGkoyAy3vbpf8%2BNMMbURUOSJ65UfeCf3w@mail.gmail.com" target="_blank">mGqx47SkUzM_<u></u>FHGkoyAy3vbpf8+<u></u>NMMbURUOSJ65UfeCf3w@mail.<u></u>gmail.com</a>><br>
Content-Type: text/plain; charset="utf-8"<br>
<br>
Hi Burkhard,<br>
<br>
> run("Collect Garbage");<br>
<br>
This just calls System.gc().. twice, actually. There are many articles and<br>
StackOverflow posts about why System.gc() is unreliable<br>
<<a href="http://stackoverflow.com/questions/2414105/why-is-it-a-bad-practice-to-call-system-gc" target="_blank">http://stackoverflow.com/<u></u>questions/2414105/why-is-it-a-<u></u>bad-practice-to-call-system-gc</a><u></u>>.<br>
I personally have seen endlessly repeated System.gc() calls effectively do<br>
nothing, necessitating the creation of large objects to force memory to be<br>
released.<br>
<br>
Also, the JVM should automatically attempt to run the garbage collector<br>
when memory is requested for allocation in excess of the JVM's memory<br>
limits. So the best way to test if SCIFIO has a memory leak would be:<br>
<br>
1) Open datasets that occupy slightly less than the max memory allocated to<br>
your Fiji<br>
2) Close these datasets<br>
3) Try opening more datasets (of a size such that, if the original datasets<br>
not released, the total memory used would exceed the max allocation)<br>
<br>
If you see OutOfMemory errors then there could quite possibly be a leak.<br>
<br>
Note also that we have unit tests in place designed to catch these leaks:<br>
<a href="https://github.com/scifio/scifio/blob/master/src/test/java/io/scif/img/cell/cache/utests/CacheServiceTest.java" target="_blank">https://github.com/scifio/<u></u>scifio/blob/master/src/test/<u></u>java/io/scif/img/cell/cache/<u></u>utests/CacheServiceTest.java</a>.<br>
An excellent way to demonstrate the memory error would be to come up with<br>
another test in this class that fails.<br>
<br>
Thanks,<br>
Mark<br>
<br>
<br>
On Sat, Jul 12, 2014 at 1:54 PM, Burkhard H?ckendorf <<br>
<a href="mailto:burkhard.hoeckendorf@web.de" target="_blank">burkhard.hoeckendorf@web.de</a>> wrote:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Dear List,<br>
<br>
First of all thanks for developing scifio and doing that open source. I<br>
have started to use it with ImageJ, since Fiji now offers the option to use<br>
it. Having said that, I may also have stumbled upon an issue. I'm unsure<br>
when (or whether) scifio releases the corresponding memory after images are<br>
closed. To look at this in a somewhat systematic way, I ran a macro (see<br>
below) to repeatedly open and close a 3GB single file uncompressed TIFF<br>
stack (2048x2048x367x16bits) and record some performance stats along the<br>
way. I ran this using a new download of Fiji.<br>
<br>
Here are the results with scifio disabled:<br>
11450 ms to open files<br>
28578184 mem before opening files<br>
<a href="tel:9282667288" value="+19282667288" target="_blank">9282667288</a> mem after opening files<br>
9272732560 mem after opening files & GC<br>
9272780080 mem after closing files<br>
24220936 mem after closing files & GC<br>
<br>
And here with scifio enabled:<br>
43715 ms to open files<br>
26765224 mem before opening files<br>
10517954400 mem after opening files<br>
9327459480 mem after opening files & GC<br>
9327506440 mem after closing files<br>
<a href="tel:9314822896" value="+19314822896" target="_blank">9314822896</a> mem after closing files & GC<br>
<br>
I have also checked, whether memory is freed a couple of minutes after<br>
closing the images, which does not appear to be the case.<br>
<br>
Note that scifio is also somewhat slower, although I'm a bit hesitant to<br>
bring this up, because it is still relatively young and I can see a few<br>
"improve ... performance" tickets in the issue tracker.<br>
<br>
Below is the macro.<br>
Kind regards,<br>
Burkhard<br>
<br>
<br>
nOpenTimes = 3;<br>
file = "/path/to/stack";<br>
<br>
run("Collect Garbage");<br>
memBefore = IJ.currentMemory();<br>
<br>
msToOpen = getTime();<br>
for (i=0; i<nOpenTimes; i+=1) {<br>
open(file);<br>
}<br>
msToOpen = getTime() - msToOpen;<br>
<br>
memFilesOpen = IJ.currentMemory();<br>
run("Collect Garbage");<br>
memFilesOpenGC = IJ.currentMemory();<br>
<br>
for (i=0; i<nOpenTimes; i+=1) {<br>
close();<br>
}<br>
memFilesClosed = IJ.currentMemory();<br>
run("Collect Garbage");<br>
memFilesClosedGC = IJ.currentMemory();<br>
<br>
print(""+ msToOpen + " ms to open files");<br>
print(""+ memBefore + " mem before opening files");<br>
print(""+ memFilesOpen + " mem after opening files");<br>
print(""+ memFilesOpenGC + " mem after opening files & GC");<br>
print(""+ memFilesClosed + " mem after closing files");<br>
print(""+ memFilesClosedGC + " mem after closing files & GC");<br>
<br>
______________________________<u></u>_________________<br>
SCIFIO mailing list<br>
<a href="mailto:SCIFIO@scif.io" target="_blank">SCIFIO@scif.io</a><br>
<a href="http://scif.io/mailman/listinfo/scifio" target="_blank">http://scif.io/mailman/<u></u>listinfo/scifio</a><br>
<br>
</blockquote>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="http://scif.io/pipermail/scifio/attachments/20140714/0e7f8b63/attachment-0001.html" target="_blank">http://scif.io/pipermail/<u></u>scifio/attachments/20140714/<u></u>0e7f8b63/attachment-0001.html</a>><br>
<br>
------------------------------<br>
<br>
______________________________<u></u>_________________<br>
SCIFIO mailing list<br>
<a href="mailto:SCIFIO@scif.io" target="_blank">SCIFIO@scif.io</a><br>
<a href="http://scif.io/mailman/listinfo/scifio" target="_blank">http://scif.io/mailman/<u></u>listinfo/scifio</a><br>
<br>
<br>
End of SCIFIO Digest, Vol 11, Issue 5<br>
******************************<u></u>*******<br>
<br>
</blockquote>
<br>
______________________________<u></u>_________________<br>
SCIFIO mailing list<br>
<a href="mailto:SCIFIO@scif.io" target="_blank">SCIFIO@scif.io</a><br>
<a href="http://scif.io/mailman/listinfo/scifio" target="_blank">http://scif.io/mailman/<u></u>listinfo/scifio</a><br>
</blockquote></div><br></div>