BUG: Cellworx Screen-- directory scanning never stops

[jrswedlow]

Using .insight for import. Pick directory with .HTD file, scan starts and never stops, so import never occurs

1. directory scanning does not work, or scans all file, and doe snot have smarts to figure out that directory is a screen. So it imports all TIFs, PNGs, and also the screen, but as a PDI structure.

2. A workaround might be to use the filtering that you have in dialog, except that if you choose that filtering, then displaying a large directory takes forever.

You might also limit the screens to showing ONLY the metadata file you ant people to select.

[jmoore]

Jason, can you give me a specific example of such a mixed directory?

[jburel <j.burel@…>]

(In [d5b05c94941eb4c82281a23b55ccb8b9d9da2e8d/ome.git] on branch develop) Improve support form screening data import. (see #5585, close #5578)

[jmoore]

The ones I could find on our system are:

jmoore@necromancer /ome/data_repo $ find . -iname "*.htd"
./test_images_good/cellworx/hms/plate1/TIFF/plate1.HTD
./test_images_good/cellworx/hms/plate1/plate1.HTD
./from_skyking/samples/ome/011B1_Morph.HTD
./96-well-sample-dataset/GF1/PNL_Representative_Data_Set_96_Well.HTD

[jburel]

Good ones to test import candidates.

[jburel <j.burel@…>]

(In [7ef9ebf536aa663b24ce2bd0e3fa61400e247c12/ome.git] on branch develop) Improve folder import e.g. folder containing a mix of folders and files. Files can end up
being hcs data or not (see #5585)

[jburel]

Improve to support the following scenario:

• Select Folder1
• Folder is as follow
  • /Folder/Flex? (folder containing screening data)
  • /Folder/Flex1 (folder containing screening data)
  • /Folder/image.dv (or whatever non screening data)
• Import Folder
• 2 plates imported and one image.

[jmoore]

Melissa, the particular case of 96-well-sample-dataset is because of the data being copied as TIFFs, right? In which case, the only thing we could do is use a readers.txt file without the tiff reader in it (or clean up the directory).

2011-05-27 10:31:39,178 678003     [      main] DEBUG                loci.formats.FormatHandler  - loci.formats.in.TiffReader.initFile(/ome/data_repo/96-well-sample-dataset/./GF1/TIFF/PNL_Representative_Data_Set_96_Well_F11_2_w685.tif)
2011-05-27 10:31:39,180 678005     [      main] INFO          loci.formats.in.MinimalTiffReader  - Reading IFDs
2011-05-27 10:31:39,182 678007     [      main] INFO          loci.formats.in.MinimalTiffReader  - Populating metadata
2011-05-27 10:31:39,182 678007     [      main] INFO                 loci.formats.in.TiffReader  - Checking comment style
2011-05-27 10:31:39,212 678037     [      main] INFO             loci.formats.in.BaseTiffReader  - Populating OME metadata
2011-05-27 10:31:39,212 678037     [      main] WARN             loci.formats.in.BaseTiffReader  - unknown creation date format: 2010/06/10 11:09:25
2011-05-27 10:31:39,216 678041     [      main] DEBUG              loci.formats.tiff.TiffParser  - Reading tile Length 2097152 Offset 8
2011-05-27 10:31:39,287 678112     [      main] DEBUG     ome.formats.importer.cli.ErrorHandler  - SCANNING: Depth:2 Num: 1600 Tot: 1732 File: ell_C08_scan.log
2011-05-27 10:31:39,289 678114     [      main] DEBUG     ome.formats.importer.cli.ErrorHandler  - SCANNING: Depth:2 Num: 1700 Tot: 1732 File: ell_F12_scan.log
2011-05-27 10:31:39,289 678114     [      main] DEBUG     ome.formats.importer.cli.ErrorHandler  - SCANNING: Depth:0 Num: 1732 Tot: 1732 File: .
2011-05-27 10:31:42,345 681170     [      main] INFO      ome.formats.importer.ImportCandidates  - 1732 file(s) parsed into 1537 group(s) with 1537 call(s) to setId in 675272ms. (680727ms total) [0 unknowns]
#======================================
# Group: /ome/data_repo/96-well-sample-dataset/./GF1/PNL_Representative_Data_Set_96_Well.HTD SPW: true Reader: CellWorx
/ome/data_repo/96-well-sample-dataset/./GF1/PNL_Representative_Data_Set_96_Well.HTD
/ome/data_repo/96-well-sample-dataset/./GF1/PNL_Representative_Data_Set_96_Well_scan.log
/ome/data_repo/96-well-sample-dataset/./GF1/z_focus_map.pnl
/ome/data_repo/96-well-sample-dataset/./GF1/PNL_Representative_Data_Set_96_Well_A01_scan.log
/ome/data_repo/96-well-sample-dataset/./GF1/PNL_Representative_Data_Set_96_Well_A01.pnl
/ome/data_repo/96-well-sample-dataset/./GF1/PNL_Representative_Data_Set_96_Well_A02_scan.log
...
#======================================
# Group: /ome/data_repo/96-well-sample-dataset/./GF1/TIFF/PNL_Representative_Data_Set_96_Well_C06_2_w685.tif SPW: false Reader: Tagged Image File Format
/ome/data_repo/96-well-sample-dataset/./GF1/TIFF/PNL_Representative_Data_Set_96_Well_C06_2_w685.tif
#======================================
# Group: /ome/data_repo/96-well-sample-dataset/./GF1/TIFF/PNL_Representative_Data_Set_96_Well_E07_4_w530.tif SPW: false Reader: Tagged Image File Format
/ome/data_repo/96-well-sample-dataset/./GF1/TIFF/PNL_Representative_Data_Set_96_Well_E07_4_w530.tif
#======================================
...

[mlinkert]

Replying to jmoore:

Melissa, the particular case of 96-well-sample-dataset is because of the data being copied as TIFFs, right? In which case, the only thing we could do is use a readers.txt file without the tiff reader in it (or clean up the directory).

Right, it's the 'TIFF' sub-directory that causes the problem. For testing, you could just copy the actual CellWorX dataset (*.HTD, *.log, and *.pnl) to test_images_good/cellworx/hms/96-well-sample-dataset/, that way the TIFFs are not lost.

[jmoore]

from Jason: /ome/data_repo/test_images_good/cellworx/hms/plate1

[jburel]

Moved from sprint 2011-06-02 (13)

[jmoore]

The directory from Jason has the same issue of having the TIFFs duplicated:

...

#======================================
# Group: /ome/data_repo/test_images_good/cellworx/hms/plate1/TIFF/plate1_E09_1_w460.tif SPW: false Reader: Tagged Image File Format
/ome/data_repo/test_images_good/cellworx/hms/plate1/TIFF/plate1_E09_1_w460.tif
#======================================
# Group: /ome/data_repo/test_images_good/cellworx/hms/plate1/TIFF/plate1_E07_2_w460.tif SPW: false Reader: Tagged Image File Format
/ome/data_repo/test_images_good/cellworx/hms/plate1/TIFF/plate1_E07_2_w460.tif
#======================================
# Group: /ome/data_repo/test_images_good/cellworx/hms/plate1/TIFF/plate1_D05_3_w460.tif SPW: false Reader: Tagged Image File Format
/ome/data_repo/test_images_good/cellworx/hms/plate1/TIFF/plate1_D05_3_w460.tif

real    3m43.016s
user    1m17.181s
sys     0m58.372s

However, scanning only took 3 minutes. Jean-Marie, do you think that Insight's of opening the TiffReader internally is doubling the time taken here? Passing to you for consideration. If that's not happening, then probably we should turn this in a ticket to cleanup the data repository.

[jmoore]

Here's the command from my import above:

bin/omero import -f /ome/data_repo/test_images_good/cellworx/hms/plate1

[jburel]

I figured out the problem in insight related to the scanning. It now takes <2mins to scan.
ImportCandidate.getPaths() returns the HTD file and all the tiff as separate images.
so they all end up being imported: I can find a workaround if necessary.

[jburel <j.burel@…>]

(In [ffa6139a876c9f94b8d536bc9d041aab1cf60507/ome.git] on branch develop) Fix the scanning directory issue when Cellworx (see #5585)

[jburel]

Scanning problem is now fixed, we can review the workflow in a 4.3.1 i.e. identifies the htd files and indicate
that a folder with tiff (or others) and decided to include/exclude it to the import.
Currently the tiff will be imported correctly.

The htd file is imported as a plate. Downgrading the ticket

[jburel]

Discussion Jason, Chris, J-M:

• add documentation on website indicating how to import such file.

Closing ticket