[SCIFIO] DICOM format detection

[Burkhard Hoeckendorf]

Dear Mark,

Thanks a lot for your help. I'll try playing with the format priority. 
That being said, I've tried this strategy (in an unrelated project) to 
override a format implementation in SCIFIO with our own implementation. 
Back then (a few months ago), it did not work reliably in my hands.

Another question is whether there is an option to turn off the automatic 
min-max computation. Thanks to SCIFIO, we've found a nice way to 
integrate our format with ImageJ, but using this comes at the price of a 
pretty substantial IO performance hit that frankly is a source of 
constant dissatisfaction. I suspect deactivating the min-max computation 
may be an easy way to improve things, presumably without breaking 
anything else (?)

Best,
Burkhard



On 4/13/2015 2:37 PM, Mark Hiner wrote:
> Hi Burkhard,
>
>  >I apologize in case I am posting this mail twice. The first attempt at
> the end of last week came with an attachment and I assume it was
> probably scrubbed (and rightly so).
>
> No worries - there actually was a technical issue with the mailing list
> that's now fixed. Sorry about that!
>
>  >As it turns out, even before looking at its file name extension,
> SCIFIO opened the file and tried to detect whether its header is in
> DICOM format
>
> DICOM is an unfortunate example that has some history in Bio-Formats
> (extensionless DICOM images, DICOM images with no magic string) that
> carried over to the SCIFIO implementation with potential for false
> positives. However, I think the best solution right now is to set your
> format to a higher priority. Formats are checked in plugin priority order.
>
> For example, see a sample format that uses a higher priority[1]. DICOM
> uses the normal priority (0), so you can use any of the higher
> constants[2] or set it manually[3] to a value > 0 for your format to be
> detected first.
>
>  >More generally though, this experience has led me to wonder whether
> the order in which file formats are detected could be tweaked such that
> the ones requiring the most work are only attempted when the easier ones
> fail.
>
> Agreed completely. Plugin priority was a starting point, but we have
> started discussing how exactly to do things better[4] with some ideas
> very similar to what you outline.
>
> Thanks for the feedback, let us know if you have any more suggestions or
> questions.
>
> Best,
> Mark
>
> [1]
> https://github.com/scifio/scifio-bf-compat/blob/scifio-bf-compat-1.11.0/src/main/java/io/scif/bf/BioFormatsFormat.java#L79-80
> [2]
> https://github.com/scijava/scijava-common/blob/scijava-common-2.40.0/src/main/java/org/scijava/Priority.java#L48-55
> [3]
> https://github.com/scijava/scijava-common/blob/scijava-common-2.40.0/src/main/java/org/scijava/plugin/Plugin.java#L108-129
> [4] https://github.com/scifio/scifio/issues/39
>
> On Mon, Apr 13, 2015 at 12:59 PM, Burkhard Hoeckendorf
> <burkhard.hoeckendorf at web.de <mailto:burkhard.hoeckendorf at web.de>> wrote:
>
>     Dear List,
>
>     I apologize in case I am posting this mail twice. The first attempt
>     at the end of last week came with an attachment and I assume it was
>     probably scrubbed (and rightly so).
>
>     We recently ran into an issue with DICOM format detection. We use a
>     custom file format that is completely unrelated to DICOM and that is
>     implemented for SCIFIO such that the file name extension alone is
>     necessary and sufficient to recognize it. This works well, except
>     for a single file which we couldn't open in Fiji. Reading it by
>     other means (C++, MATLAB) worked fine.
>
>     As it turns out, even before looking at its file name extension,
>     SCIFIO opened the file and tried to detect whether its header is in
>     DICOM format. I am unfamiliar with the DICOM standard and not a
>     professional Programmer, but from the SCIFIO implementation
>     (io.scif.formats.DICOMFormat) it appears to me that the magic 'DICM'
>     string in the header is treated as optional. If it is not found,
>     SCIFIO tries to read a single header field, and if successful
>     considers the file a DICOM file. In our case, we got unlucky and by
>     accident SCIFIO retrieved a valid-enough result when reading the
>     corresponding position in the non-DICOM file. Hence, it was wrongly
>     detected as a DICOM file.
>
>     For the time being, we are now running our own, locally modified
>     SCIFIO that does not try harder when the 'DICM' string is missing,
>     and this solved our problem. For reference, we modified line 1079 in
>     io.scif.formats.DICOMFormat. I am also happy to send this as a pull
>     request, but I am not a DICOM user, so can not comment whether or
>     not this is actually a workable solution.
>
>     More generally though, this experience has led me to wonder whether
>     the order in which file formats are detected could be tweaked such
>     that the ones requiring the most work are only attempted when the
>     easier ones fail. DICOM format detection would thus only occur when
>     the file name extension does not match that of any format where the
>     extension alone is necessary and sufficient. This may help prevent
>     similar conflicts in the future and may additionally contribute a
>     little bit to a much needed performance increase.
>
>     Cheers,
>     Burkhard
>
>
>