[SCIFIO] SCIFIO does not seem to release memory of closed images

[Mark Hiner]

Hi Burkhard,

 > run("Collect Garbage");

 This just calls System.gc().. twice, actually. There are many articles and
StackOverflow posts about why System.gc() is unreliable
<http://stackoverflow.com/questions/2414105/why-is-it-a-bad-practice-to-call-system-gc>.
I personally have seen endlessly repeated System.gc() calls effectively do
nothing, necessitating the creation of large objects to force memory to be
released.

 Also, the JVM should automatically attempt to run the garbage collector
when memory is requested for allocation in excess of the JVM's memory
limits. So the best way to test if SCIFIO has a memory leak would be:

1) Open datasets that occupy slightly less than the max memory allocated to
your Fiji
2) Close these datasets
3) Try opening more datasets (of a size such that, if the original datasets
not released, the total memory used would exceed the max allocation)

If you see OutOfMemory errors then there could quite possibly be a leak.

Note also that we have unit tests in place designed to catch these leaks:
https://github.com/scifio/scifio/blob/master/src/test/java/io/scif/img/cell/cache/utests/CacheServiceTest.java.
An excellent way to demonstrate the memory error would be to come up with
another test in this class that fails.

Thanks,
Mark


On Sat, Jul 12, 2014 at 1:54 PM, Burkhard Höckendorf <
burkhard.hoeckendorf at web.de> wrote:

> Dear List,
>
> First of all thanks for developing scifio and doing that open source. I
> have started to use it with ImageJ, since Fiji now offers the option to use
> it. Having said that, I may also have stumbled upon an issue. I'm unsure
> when (or whether) scifio releases the corresponding memory after images are
> closed. To look at this in a somewhat systematic way, I ran a macro (see
> below) to repeatedly open and close a 3GB single file uncompressed TIFF
> stack (2048x2048x367x16bits) and record some performance stats along the
> way. I ran this using a new download of Fiji.
>
> Here are the results with scifio disabled:
>      11450 ms to open files
>   28578184 mem before opening files
> 9282667288 mem after opening files
> 9272732560 mem after opening files & GC
> 9272780080 mem after closing files
>   24220936 mem after closing files & GC
>
> And here with scifio enabled:
>       43715 ms to open files
>    26765224 mem before opening files
> 10517954400 mem after opening files
>  9327459480 mem after opening files & GC
>  9327506440 mem after closing files
>  9314822896 mem after closing files & GC
>
> I have also checked, whether memory is freed a couple of minutes after
> closing the images, which does not appear to be the case.
>
> Note that scifio is also somewhat slower, although I'm a bit hesitant to
> bring this up, because it is still relatively young and I can see a few
> "improve ... performance" tickets in the issue tracker.
>
> Below is the macro.
> Kind regards,
> Burkhard
>
>
> nOpenTimes = 3;
> file = "/path/to/stack";
>
> run("Collect Garbage");
> memBefore = IJ.currentMemory();
>
> msToOpen = getTime();
> for (i=0; i<nOpenTimes; i+=1) {
>         open(file);
> }
> msToOpen = getTime() - msToOpen;
>
> memFilesOpen = IJ.currentMemory();
> run("Collect Garbage");
> memFilesOpenGC = IJ.currentMemory();
>
> for (i=0; i<nOpenTimes; i+=1) {
>         close();
> }
> memFilesClosed = IJ.currentMemory();
> run("Collect Garbage");
> memFilesClosedGC = IJ.currentMemory();
>
> print(""+ msToOpen + " ms to open files");
> print(""+ memBefore + " mem before opening files");
> print(""+ memFilesOpen + " mem after opening files");
> print(""+ memFilesOpenGC + " mem after opening files & GC");
> print(""+ memFilesClosed + " mem after closing files");
> print(""+ memFilesClosedGC + " mem after closing files & GC");
>
> _______________________________________________
> SCIFIO mailing list
> SCIFIO at scif.io
> http://scif.io/mailman/listinfo/scifio
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://scif.io/pipermail/scifio/attachments/20140714/0e7f8b63/attachment-0002.html>