[SCIFIO] DICOM format detection

[Mark Hiner]

Hi Burkhard,

>I apologize in case I am posting this mail twice. The first attempt at the
end of last week came with an attachment and I assume it was probably
scrubbed (and rightly so).

No worries - there actually was a technical issue with the mailing list
that's now fixed. Sorry about that!

>As it turns out, even before looking at its file name extension, SCIFIO
opened the file and tried to detect whether its header is in DICOM format

DICOM is an unfortunate example that has some history in Bio-Formats
(extensionless DICOM images, DICOM images with no magic string) that
carried over to the SCIFIO implementation with potential for false
positives. However, I think the best solution right now is to set your
format to a higher priority. Formats are checked in plugin priority order.

For example, see a sample format that uses a higher priority[1]. DICOM uses
the normal priority (0), so you can use any of the higher constants[2] or
set it manually[3] to a value > 0 for your format to be detected first.

>More generally though, this experience has led me to wonder whether the
order in which file formats are detected could be tweaked such that the
ones requiring the most work are only attempted when the easier ones fail.

Agreed completely. Plugin priority was a starting point, but we have
started discussing how exactly to do things better[4] with some ideas very
similar to what you outline.

Thanks for the feedback, let us know if you have any more suggestions or
questions.

Best,
Mark

[1]
https://github.com/scifio/scifio-bf-compat/blob/scifio-bf-compat-1.11.0/src/main/java/io/scif/bf/BioFormatsFormat.java#L79-80
[2]
https://github.com/scijava/scijava-common/blob/scijava-common-2.40.0/src/main/java/org/scijava/Priority.java#L48-55
[3]
https://github.com/scijava/scijava-common/blob/scijava-common-2.40.0/src/main/java/org/scijava/plugin/Plugin.java#L108-129
[4] https://github.com/scifio/scifio/issues/39

On Mon, Apr 13, 2015 at 12:59 PM, Burkhard Hoeckendorf <
burkhard.hoeckendorf at web.de> wrote:

> Dear List,
>
> I apologize in case I am posting this mail twice. The first attempt at the
> end of last week came with an attachment and I assume it was probably
> scrubbed (and rightly so).
>
> We recently ran into an issue with DICOM format detection. We use a custom
> file format that is completely unrelated to DICOM and that is implemented
> for SCIFIO such that the file name extension alone is necessary and
> sufficient to recognize it. This works well, except for a single file which
> we couldn't open in Fiji. Reading it by other means (C++, MATLAB) worked
> fine.
>
> As it turns out, even before looking at its file name extension, SCIFIO
> opened the file and tried to detect whether its header is in DICOM format.
> I am unfamiliar with the DICOM standard and not a professional Programmer,
> but from the SCIFIO implementation (io.scif.formats.DICOMFormat) it appears
> to me that the magic 'DICM' string in the header is treated as optional. If
> it is not found, SCIFIO tries to read a single header field, and if
> successful considers the file a DICOM file. In our case, we got unlucky and
> by accident SCIFIO retrieved a valid-enough result when reading the
> corresponding position in the non-DICOM file. Hence, it was wrongly
> detected as a DICOM file.
>
> For the time being, we are now running our own, locally modified SCIFIO
> that does not try harder when the 'DICM' string is missing, and this solved
> our problem. For reference, we modified line 1079 in
> io.scif.formats.DICOMFormat. I am also happy to send this as a pull
> request, but I am not a DICOM user, so can not comment whether or not this
> is actually a workable solution.
>
> More generally though, this experience has led me to wonder whether the
> order in which file formats are detected could be tweaked such that the
> ones requiring the most work are only attempted when the easier ones fail.
> DICOM format detection would thus only occur when the file name extension
> does not match that of any format where the extension alone is necessary
> and sufficient. This may help prevent similar conflicts in the future and
> may additionally contribute a little bit to a much needed performance
> increase.
>
> Cheers,
> Burkhard
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://scif.io/pipermail/scifio/attachments/20150413/a89ccb8e/attachment.html>