[SCIFIO] How to fix bug: BigTiff not handled well by Bio-Formats or SCIFIO

[Curtis Rueden]

Hi Adam,

[CCing the SCIFIO and OME users mailing lists, as this discussion will be
of interest to others.]

> This bug:
> https://groups.google.com/forum/#!topic/fiji-users/7qLsi-V-jvs
> (Bio-Formats does not export a tiff using the BigTIFF library
> automatically when a file will be bigger than 4GB) is still a bug in
> the latest version of FIJI and the latest build of Bio-Formats
> installed separately in ImageJ, though you hoped to fix it in 2012.

Right. My apologies that that problem has continued to exist for so long.
IMO, Bio-Formats should auto-detect whether the output file might be large
enough to be need BigTIFF, and act accordingly. But the current 5.0.1
release does not.

> You mention in late 2013 on stackoverflow that SCIFIO should solve
> this problem:
>
http://stackoverflow.com/questions/18986368/java-how-to-write-a-very-large-20-000x20-000-px-or-larger-tif-image/19034370#19034370
> but that does not yet seem to be the case as FIJI, for which SCIFIO
> claims to do the I/O, still has this bug when trying to write out a
> tiff >4GB.

Actually, that SO question is about a different issue: huge planes. The
BigTIFF format is actually needed even for more modest size planes, when
you simply have many of them, which result in a TIFF file of more than 4GB
in size. Regular TIFF uses 32-bit offsets, so as soon as the TIFF file gets
over 4GB, there is no way to indicate that an offset should go deeper into
the file than that. BigTIFF solves this problem by using 64-bit offsets.

> Changing Line 132 in TIffWriter.JAVA to say: isBigTiff = true; would
> simply force writing BigTIFFs all the time, which would solve the
> problem for me (although I realize cause other problems for others!)

Well, there is also an API method in TiffWriter: setBigTiff(true). But of
course you have to write a little program in order to use it; there is no
built-in method in the ImageJ user interface to enable that option. And it
is off by default.

> However, I cannot find a way to change this java file and propagate
> the change to within ImageJ/FIJI.  When I compile the new
> TiffWriter.java in ImageJ I get hundreds of errors (see attached log
> from ImageJ).

I am not sure why you are seeing those errors. I would need to know more
about how you were trying to build the system. The best ways to build
Bio-Formats are with either Ant ("ant jars" from top level works) or Maven
(e.g., "mvn" from top level).

> What am I missing and/or is there any simple way to solve this problem
> and/or fix the bug?

Earlier today, Mark Hiner released a new version of SCIFIO that
automatically enables BigTIFF as needed:
    http://scif.io/pipermail/scifio/2014-April/000067.html
    http://maven.imagej.net/content/groups/public/io/scif/scifio/0.11.0/

We are close to having Fiji use SCIFIO for its image I/O by default; once
that happens, opening and saving large images should "just work" even when
they are larger than 4GB in size.

Regards,
Curtis


On Wed, Apr 16, 2014 at 4:03 PM, Adam Packer <a.packer at ucl.ac.uk> wrote:

> Dear Curtis
>
> This bug:
> https://groups.google.com/forum/#!topic/fiji-users/7qLsi-V-jvs
>  (Bio-Formats does not export a tiff using the BigTIFF library
> automatically when a file will be bigger than 4GB) is still a bug in the
> latest version of FIJI and the latest build of Bio-Formats installed
> separately in ImageJ, though you hoped to fix it in 2012.
>
> You mention in late 2013 on stackoverflow that SCIFIO should solve this
> problem:
>
> http://stackoverflow.com/questions/18986368/java-how-to-write-a-very-large-20-000x20-000-px-or-larger-tif-image/19034370#19034370
> but that does not yet seem to be the case as FIJI, for which SCIFIO claims
> to do the I/O, still has this bug when trying to write out a tiff >4GB.
>
> Changing Line 132 in TIffWriter.JAVA to say:
> isBigTiff = true;
> would simply force writing BigTIFFs all the time, which would solve the
> problem for me (although I realize cause other problems for others!)
>
> However, I cannot find a way to change this java file and propagate the
> change to within ImageJ/FIJI.  When I compile the new TiffWriter.java in
> ImageJ I get hundreds of errors (see attached log from ImageJ).
>
> What am I missing and/or is there any simple way to solve this problem
> and/or fix the bug?
>
> All the best,
> Adam Packer
> University College London
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://scif.io/pipermail/scifio/attachments/20140421/d8e7c68d/attachment-0002.html>