Task #12007 (closed)
Opened 10 years ago
Closed 10 years ago
BUG: Pyramids not being built
Reported by: | pwalczysko | Owned by: | bramalingam |
---|---|---|---|
Priority: | critical | Milestone: | 5.0.2 |
Component: | Services | Version: | 5.0.0-rc3 |
Keywords: | n.a. | Cc: | ux@…, jamoore, sbesson, java@…, khgillen |
Resources: | n.a. | Referenced By: | n.a. |
References: | n.a. | Remaining Time: | n.a. |
Sprint: | n.a. |
Description (last modified by jamoore)
On Gretzky, with the merge build post RC4.
http://trac.openmicroscopy.org.uk/ome/ticket/11869
Hamamatsu-vms/CMU-2 was imported by @dlindner in the morning hours as well as gatan and other large images. The server was working on the pyramids for couple of hours, showing the clock placeholders for 2 gatans and the hamamatsu-vms file. In between, there was a rebuild and restart of Gretzky, at around 1 pm. After this, although the web client was open and the images were showing in full viewer "Pyramids being calculated", nothing was really happening in Pixel log.
Attachments (3)
Change History (17)
Changed 10 years ago by pwalczysko
comment:1 Changed 10 years ago by pwalczysko
Changed 10 years ago by pwalczysko
Changed 10 years ago by pwalczysko
comment:2 Changed 10 years ago by jamoore
- Cc java@… added
- Priority changed from major to critical
comment:3 Changed 10 years ago by jamoore
- Description modified (diff)
- Milestone changed from 5.0.1 to 5.0.2
comment:4 Changed 10 years ago by jamoore
- Owner set to jamoore
Balaji reported:
I just now checked the import on dogfish : 5.0.0. (test_images_good). The
images are imported correctly, except for one image, where the import
seems to have worked but when I try to open the image, it says invalid
image.
I looked at the image (http://dogfish.openmicroscopy.org/webclient/?show=image-3157) and saw the behavior Petr mentions. The PixelData? process has completely fallen over. I'm going to first adjust the memory settings and restart OMERO on dogfish. Hopefully then the pyramid will be generated. If not, I'll investigate further.
comment:5 Changed 10 years ago by jamoore
- Owner changed from jamoore to bramalingam
On restart, got "error instantiating /OMERO/Pixels/...*pyramid, so was created corrupt! Can also be seen in OMEROweb.log:
serverExceptionClass = ome.conditions.ResourceError message = Error instantiating pixel buffer: /repositories/omero_reference/Pixels/Dir-003/3157_pyramid }
Balaji or Petr: can one of you as admin run the following:
[omero_reference@ome-c6220-1-2 OMERO-CURRENT]$ bin/omero admin fixpyramids --dry-run /repositories/omero_reference/
comment:6 Changed 10 years ago by pwalczysko
@jamoore: Hmm, looks like this is not what I can do :
pwalczysko@ome-c6220-1-2 /]$ find / -type d -name "OMERO-CURRENT" find: `/proc/tty/driver': Permission denied find: `/proc/1/task/1/fd': Permission denied find: `/proc/1/task/1/fdinfo': Permission denied find: `/proc/1/task/1/ns': Permission denied find: `/proc/1/fd': Permission denied find: `/proc/1/fdinfo': Permission denied find: `/proc/1/ns': Permission denied
Maybe Balaji can.
comment:7 Changed 10 years ago by pwalczysko
[pwalczysko@ome-c6220-1-2 OMERO-CURRENT]$ bin/omero admin fixpyramids --dry-run /repositories/omero_reference/ /opt/omero/OMERO.server-5.0.1-ice34-b21/etc/grid/config.xml read-only FATAL: Cannot access /opt/omero/OMERO.server-5.0.1-ice34-b21/var, a required file/directory for OMERO [pwalczysko@ome-c6220-1-2 OMERO-CURRENT]$
comment:8 Changed 10 years ago by jamoore
- Cc khgillen added
Sorry, you'd have to be logged in as the omero_reference user, who starts the server. Not sure that's supported, so I've CC'd Kenny. (Note: we're still waiting on postgres tuning on dogfish before pyramids will be generated)
comment:9 Changed 10 years ago by pwalczysko
Okay, this explains it - this will be then for @bramalingam to execute, I have never logged in as omero_reference user.
comment:10 Changed 10 years ago by bramalingam
[inplace_user@ome-c6220-1-2 bin]$ ./omero admin fixpyramids --dry-run /repositories/omero_reference/ /opt/omero/OMERO.server-5.0.1-ice34-b21/etc/grid/config.xml read-only Previous session expired for jamoore on localhost:4064 Server: [localhost] Username: [jamoore]bramalingam Password: Created session 46a4dcb9-ab5f-4a54-ae82-ae0a9909fd1e (bramalingam@localhost:4064). Idle timeout: 10.0 min. Current group: data_repo Would remove 592_pyramid Would remove 591_pyramid Would remove 590_pyramid Would remove 230_pyramid Would remove 1046_pyramid Would remove 1354_pyramid Would remove 1290_pyramid Would remove 1797_pyramid Would remove 1796_pyramid Would remove 1818_pyramid Would remove 1490_pyramid Would remove 1045_pyramid Would remove 1285_pyramid Would remove 1286_pyramid Would remove 1283_pyramid Would remove 1284_pyramid Would remove 2495_pyramid Would remove 2489_pyramid Would remove 2695_pyramid Would remove 3746_pyramid Would remove 3069_pyramid Would remove 3740_pyramid Would remove 3946_pyramid Would remove 3501_pyramid Would remove 3502_pyramid Would remove 3810_pyramid Would remove 3739_pyramid Would remove 3160_pyramid Would remove 3494_pyramid Would remove 3741_pyramid Would remove 3945_pyramid Would remove 3742_pyramid Would remove 3048_pyramid
comment:11 Changed 10 years ago by jamoore
Thanks guys. So that looks reasonable from the length. Once the pyramid process is corrected, we can run this (without --dry-run), delete each of those files, and then things *should* be back to normal.
comment:12 Changed 10 years ago by jamoore
btw, process seems to be running. Perhaps we test this tomorrow?
comment:13 Changed 10 years ago by jamoore
Ping.
comment:14 Changed 10 years ago by jamoore
- Resolution set to fixed
- Status changed from new to closed
Cleaned up the blocking pyramids:
[omero_reference@ome-c6220-1-2 OMERO-CURRENT]$ bin/omero admin fixpyramids /repositories/omero_reference/ Using session bfa05f71-7178-4b14-88c3-397caa3ce48f (jamoore@localhost:4064). Idle timeout: 10.0 min. Current group: data_repo Removing 592_pyramid Removing 591_pyramid Removing 590_pyramid Removing 230_pyramid Removing 88832_pyramid Removing 88829_pyramid Removing 1046_pyramid Removing 1354_pyramid Removing 1290_pyramid Removing 1797_pyramid Removing 1796_pyramid Removing 1818_pyramid Removing 1490_pyramid Removing 1045_pyramid Removing 1285_pyramid Removing 1286_pyramid Removing 1283_pyramid Removing 1284_pyramid Removing 78027_pyramid Removing 78026_pyramid Removing 78024_pyramid Removing 2495_pyramid Removing 2489_pyramid Removing 2695_pyramid Removing 3746_pyramid Removing 3069_pyramid Removing 3740_pyramid Removing 3946_pyramid Removing 3501_pyramid Removing 3502_pyramid Removing 3810_pyramid Removing 3739_pyramid Removing 3160_pyramid Removing 3494_pyramid Removing 3741_pyramid Removing 3945_pyramid Removing 3742_pyramid Removing 3048_pyramid
With that, I assume that this is taken care of. If anything else arises, re-open or file a new ticket.
Note that the "…-vms full resolution" file did not show any message, but neither did it show anything in full viewer (see screenshot).