Update TiffWriter to automatically switch to BigTIFF

[mlinkert]

This takes the burden off of the user to call setBigTiff(true) when needed.

The easiest thing to do is probably to compute the expected number of bytes in the file before anything else happens in setId(...). If the byte count is close enough to 4 GB (3.8 GB? 3.9 GB?) then set the BigTIFF flag.

I don't think it makes sense to remove the 'setBigTiff' method, though, as it does allow the user to force small files to be written as BigTIFFs.

[rleigh]

Writing out a large (4.9GB) stack using the LOCI exporter plugin:

loci.formats.FormatException: File is too large; call setBigTiff(true)
	at loci.formats.out.TiffWriter.prepareToWriteImage(TiffWriter.java:294)
	at loci.formats.out.TiffWriter.saveBytes(TiffWriter.java:184)
	at loci.formats.out.OMETiffWriter.saveBytes(OMETiffWriter.java:201)
	at loci.formats.out.OMETiffWriter.saveBytes(OMETiffWriter.java:187)
	at loci.formats.FormatWriter.saveBytes(FormatWriter.java:136)
	at loci.plugins.out.Exporter.run(Exporter.java:571)
	at loci.plugins.LociExporter.run(LociExporter.java:77)
	at ij.plugin.filter.PlugInFilterRunner.processOneImage(PlugInFilterRunner.java:262)
	at ij.plugin.filter.PlugInFilterRunner.<init>(PlugInFilterRunner.java:111)
	at ij.IJ.runUserPlugIn(IJ.java:186)
	at ij.IJ.runPlugIn(IJ.java:151)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at ij.Command.runPlugIn(Command.java:148)
	at ij.Command.runCommand(Command.java:97)
	at ij.Executer.run(Executer.java:64)
	at java.lang.Thread.run(Thread.java:680)

Using current dev_4_4 loci_tools.jar and FiJi?. There doesn't look to be a way of enabling BigTIFF in the UI. The exporter writes the TIFF up to the 4GB mark (4291159820 bytes) and then fails. It's incomplete and lacks any OME-XML metadata.

[rleigh]

If writing a non-big TIFF fails with the above exception, would it be possible to re-try writing out as a big TIFF? This would work as a fallback in the case that an up-front size estimation heuristic is wrong, since e.g. compression could complicate the estimate. Maybe switch to big at a much lower threshold e.g. 2GB as well?

[mtbcarroll]

Don't know why the Version set itself, bah.

[rleigh]

Some additional thoughts.

There are semi-official tf2/tf8/btf extensions for "big" tiff images. Could the TIFF writer automatically set the isBigTiff flag if any of these extensions are used? This would provide a means for the user to request big tiff output without having to use the API to call setBigTiff().

As an extension to this, could the OME-TIFF writer make use of .ome.tf2, .ome.tf8 and .ome.btf extensions to also permit the same behaviour when writing OME-TIFF files? The reader would also need updating to accept these extensions without any other code changes.

This would seem to be simple to implement, minimally invasive and provides a simple, reliable and accessible means for an end user to generate big tiffs, e.g. from imagej, bfconvert, etc. without any break with backward compatibility when using the .tif or .tiff extensions which would continue to default to 32-bit offsets.

[rleigh]

For the above suggestion, it's completely orthogonal to the original proposal of calculating the expected file size. Both can be added independently.

[mlinkert]

Moving to 5.1.1 for triage.

[mlinkert]

Fixed with ​https://github.com/openmicroscopy/bioformats/pull/1744, with the caveat that automatic switching will not happen when compression is used and a BigTIFF extension is not present.

[Melissa Linkert <melissa@…>]

(In [6decd24c801c753183a156f2d716ccd3d69fdc49/bioformats.git] on branch develop) TIFF writer: automatically switch to BigTIFF when possible

If a BigTIFF extension is used, or we know that more than 4GB of data
will be written, then BigTIFF is used automatically. This does not
attempt to switch to BigTIFF when compression is used and a BigTIFF
extension is not present, as we can't easily predict the actual size
of the output file in that case.

Fixes #6589.