Support .ndpi files with width or height > 65500

[mlinkert]

Many .ndpi files contain JPEG streams that have a width and/or height larger than 65500 (the maximum size supported by Java's JPEG decoder. The maximum size allowed in the JPEG specification is 65535, and it is not uncommon to have .ndpi files that contain much larger images.

After some investigation, it looks like Hamamatsu has written their very own JPEG encoder/decoder to work around the limitations in the JPEG specification, as no other decoder that I can find actually supports this.
A lot of investigation has already gone into this ticket, and given the scope of work required I don't see it happening for 4.3.

When we finally do have time to sort this out, the plan is roughly:

1) See if there is a way to strip off the dimension information and just read all of the tiles using Java's built-in JPEG support. We can then re-pack the tiles using the dimension information stored in the .ndpi file.

2) Failing (1), see if we can write our own Java interface to the JPEG native libraries included with the JRE that would accomplish the same thing as (1).

3) If neither (1) nor (2) work, we'll need to write our own baseline JPEG decoder that allows the image dimensions to be overridden.

[Melissa Linkert <melissa@…>]

(In [4d346e2b279677ec57d6d443833762f6abeb4fe7/bioformats.git]) Prevent errors for .ndpi files that use a DNL marker.

See #5092. The problem now is what to do about images which have a
width or height > 65535 (the maximum width and height allowed in a JPEG
stream).

[Melissa Linkert <melissa@…>]

(In [de977aa84dffe5a14583689bfbd5d57703192290/bioformats.git]) Prevent errors for .ndpi files that use a DNL marker.

See #5092. The problem now is what to do about images which have a
width or height > 65535 (the maximum width and height allowed in a JPEG
stream).

[Melissa Linkert <melissa@…>]

(In [de977aa84dffe5a14583689bfbd5d57703192290/bioformats.git]) Prevent errors for .ndpi files that use a DNL marker.

See #5092. The problem now is what to do about images which have a
width or height > 65535 (the maximum width and height allowed in a JPEG
stream).

[Melissa Linkert <melissa@…>]

(In [200b08be344e4969c259fad783737e254fcfe86b/bioformats.git]) Throw an exception if the dimensions are unsupported.

...instead of silently returning incorrect pixels. See #5092.

[Melissa Linkert <melissa@…>]

(In [200b08be344e4969c259fad783737e254fcfe86b/bioformats.git]) Throw an exception if the dimensions are unsupported.

...instead of silently returning incorrect pixels. See #5092.

[Melissa Linkert <melissa@…>]

(In [4dcd4aee06124aad6ef6ed4004d0906f662f8a94/bioformats.git]) Throw an exception if the dimensions are unsupported.

...instead of silently returning incorrect pixels. See #5092.
(cherry picked from commit 200b08be344e4969c259fad783737e254fcfe86b)

[Melissa Linkert <melissa@…>]

(In [4dcd4aee06124aad6ef6ed4004d0906f662f8a94/bioformats.git]) Throw an exception if the dimensions are unsupported.

...instead of silently returning incorrect pixels. See #5092.
(cherry picked from commit 200b08be344e4969c259fad783737e254fcfe86b)

[mlinkert]

Updated outline of what needs to happen and the target completion date:

• native library bundling [7 September]
  • create a JAR file that contains the libjpegturbo native libraries
  • adopt a solution for loading native libraries from a JAR file
• underlying JPEG infrastructure [15 September]
  • update existing JPEG readers to use libjpegturbo when available
  • update JPEG support to allow tile access
• NDPI-specific updates [30 September]
  • make any necessary updates to libjpegturbo and/or the tile accessors to account for dimensions > 65500
  • update the NDPI reader to respect recent large-image API changes
• testing, with assistance from the rest of the team [5 October]

The idea is then to have this ready for the late fall release (4.5?).

[mlinkert]

Fixed with merging of ​https://github.com/openmicroscopy/bioformats/pull/159

Builds for testing are available here: ​http://hudson.openmicroscopy.org.uk/view/Bio-Formats/job/BIOFORMATS-trunk/
There is not yet a stable release with this functionality.