Warning: Can't synchronize with repository "(default)" (/home/git/ome.git does not appear to be a Git repository.). Look in the Trac log for more information.
Notice: In order to edit this ticket you need to be either: a Product Owner, The owner or the reporter of the ticket, or, in case of a Task not yet assigned, a team_member"

Task #12007 (closed)

Opened 10 years ago

Closed 10 years ago

BUG: Pyramids not being built

Reported by: pwalczysko Owned by: bramalingam
Priority: critical Milestone: 5.0.2
Component: Services Version: 5.0.0-rc3
Keywords: n.a. Cc: ux@…, jamoore, sbesson, java@…, khgillen
Resources: n.a. Referenced By: n.a.
References: n.a. Remaining Time: n.a.
Sprint: n.a.

Description (last modified by jamoore)

On Gretzky, with the merge build post RC4.
http://trac.openmicroscopy.org.uk/ome/ticket/11869
Hamamatsu-vms/CMU-2 was imported by @dlindner in the morning hours as well as gatan and other large images. The server was working on the pyramids for couple of hours, showing the clock placeholders for 2 gatans and the hamamatsu-vms file. In between, there was a rebuild and restart of Gretzky, at around 1 pm. After this, although the web client was open and the images were showing in full viewer "Pyramids being calculated", nothing was really happening in Pixel log.

Attachments (3)

Screen Shot 2014-02-21 at 17.57.08.png (157.1 KB) - added by pwalczysko 10 years ago.
Blitz-0.log (8.9 MB) - added by pwalczysko 10 years ago.
PixelData-0.log (8.1 KB) - added by pwalczysko 10 years ago.

Change History (17)

Changed 10 years ago by pwalczysko

comment:1 Changed 10 years ago by pwalczysko

Note that the "…-vms full resolution" file did not show any message, but neither did it show anything in full viewer (see screenshot).

Changed 10 years ago by pwalczysko

Changed 10 years ago by pwalczysko

comment:2 Changed 10 years ago by jamoore

  • Cc java@… added
  • Priority changed from major to critical

comment:3 Changed 10 years ago by jamoore

  • Description modified (diff)
  • Milestone changed from 5.0.1 to 5.0.2

comment:4 Changed 10 years ago by jamoore

  • Owner set to jamoore

Balaji reported:

I just now checked the import on dogfish : 5.0.0. (test_images_good). The
images are imported correctly, except for one image, where the import
seems to have worked but when I try to open the image, it says invalid
image.

I looked at the image (http://dogfish.openmicroscopy.org/webclient/?show=image-3157) and saw the behavior Petr mentions. The PixelData? process has completely fallen over. I'm going to first adjust the memory settings and restart OMERO on dogfish. Hopefully then the pyramid will be generated. If not, I'll investigate further.

comment:5 Changed 10 years ago by jamoore

  • Owner changed from jamoore to bramalingam

On restart, got "error instantiating /OMERO/Pixels/...*pyramid, so was created corrupt! Can also be seen in OMEROweb.log:

   serverExceptionClass = ome.conditions.ResourceError
    message = Error instantiating pixel buffer: /repositories/omero_reference/Pixels/Dir-003/3157_pyramid
}

Balaji or Petr: can one of you as admin run the following:

[omero_reference@ome-c6220-1-2 OMERO-CURRENT]$ bin/omero admin fixpyramids --dry-run /repositories/omero_reference/

comment:6 Changed 10 years ago by pwalczysko

@jamoore: Hmm, looks like this is not what I can do :

pwalczysko@ome-c6220-1-2 /]$ find / -type d -name "OMERO-CURRENT"
find: `/proc/tty/driver': Permission denied
find: `/proc/1/task/1/fd': Permission denied
find: `/proc/1/task/1/fdinfo': Permission denied
find: `/proc/1/task/1/ns': Permission denied
find: `/proc/1/fd': Permission denied
find: `/proc/1/fdinfo': Permission denied
find: `/proc/1/ns': Permission denied

Maybe Balaji can.

comment:7 Changed 10 years ago by pwalczysko

[pwalczysko@ome-c6220-1-2 OMERO-CURRENT]$ bin/omero admin fixpyramids --dry-run /repositories/omero_reference/
/opt/omero/OMERO.server-5.0.1-ice34-b21/etc/grid/config.xml read-only
FATAL: Cannot access /opt/omero/OMERO.server-5.0.1-ice34-b21/var, a required file/directory for OMERO
[pwalczysko@ome-c6220-1-2 OMERO-CURRENT]$ 

comment:8 Changed 10 years ago by jamoore

  • Cc khgillen added

Sorry, you'd have to be logged in as the omero_reference user, who starts the server. Not sure that's supported, so I've CC'd Kenny. (Note: we're still waiting on postgres tuning on dogfish before pyramids will be generated)

comment:9 Changed 10 years ago by pwalczysko

Okay, this explains it - this will be then for @bramalingam to execute, I have never logged in as omero_reference user.

comment:10 Changed 10 years ago by bramalingam

[inplace_user@ome-c6220-1-2 bin]$ ./omero admin fixpyramids --dry-run /repositories/omero_reference/
/opt/omero/OMERO.server-5.0.1-ice34-b21/etc/grid/config.xml read-only
Previous session expired for jamoore on localhost:4064
Server: [localhost]
Username: [jamoore]bramalingam
Password:
Created session 46a4dcb9-ab5f-4a54-ae82-ae0a9909fd1e (bramalingam@localhost:4064). Idle timeout: 10.0 min. Current group: data_repo
Would remove 592_pyramid
Would remove 591_pyramid
Would remove 590_pyramid
Would remove 230_pyramid
Would remove 1046_pyramid
Would remove 1354_pyramid
Would remove 1290_pyramid
Would remove 1797_pyramid
Would remove 1796_pyramid
Would remove 1818_pyramid
Would remove 1490_pyramid
Would remove 1045_pyramid
Would remove 1285_pyramid
Would remove 1286_pyramid
Would remove 1283_pyramid
Would remove 1284_pyramid
Would remove 2495_pyramid
Would remove 2489_pyramid
Would remove 2695_pyramid
Would remove 3746_pyramid
Would remove 3069_pyramid
Would remove 3740_pyramid
Would remove 3946_pyramid
Would remove 3501_pyramid
Would remove 3502_pyramid
Would remove 3810_pyramid
Would remove 3739_pyramid
Would remove 3160_pyramid
Would remove 3494_pyramid
Would remove 3741_pyramid
Would remove 3945_pyramid
Would remove 3742_pyramid
Would remove 3048_pyramid

comment:11 Changed 10 years ago by jamoore

Thanks guys. So that looks reasonable from the length. Once the pyramid process is corrected, we can run this (without --dry-run), delete each of those files, and then things *should* be back to normal.

comment:12 Changed 10 years ago by jamoore

btw, process seems to be running. Perhaps we test this tomorrow?

comment:13 Changed 10 years ago by jamoore

Ping.

comment:14 Changed 10 years ago by jamoore

  • Resolution set to fixed
  • Status changed from new to closed

Cleaned up the blocking pyramids:

[omero_reference@ome-c6220-1-2 OMERO-CURRENT]$ bin/omero admin fixpyramids /repositories/omero_reference/                                   
Using session bfa05f71-7178-4b14-88c3-397caa3ce48f (jamoore@localhost:4064). Idle timeout: 10.0 min. Current group: data_repo
Removing 592_pyramid
Removing 591_pyramid
Removing 590_pyramid
Removing 230_pyramid
Removing 88832_pyramid
Removing 88829_pyramid
Removing 1046_pyramid
Removing 1354_pyramid
Removing 1290_pyramid
Removing 1797_pyramid
Removing 1796_pyramid
Removing 1818_pyramid
Removing 1490_pyramid
Removing 1045_pyramid
Removing 1285_pyramid
Removing 1286_pyramid
Removing 1283_pyramid
Removing 1284_pyramid
Removing 78027_pyramid
Removing 78026_pyramid
Removing 78024_pyramid
Removing 2495_pyramid
Removing 2489_pyramid
Removing 2695_pyramid
Removing 3746_pyramid
Removing 3069_pyramid
Removing 3740_pyramid
Removing 3946_pyramid
Removing 3501_pyramid
Removing 3502_pyramid
Removing 3810_pyramid
Removing 3739_pyramid
Removing 3160_pyramid
Removing 3494_pyramid
Removing 3741_pyramid
Removing 3945_pyramid
Removing 3742_pyramid
Removing 3048_pyramid

With that, I assume that this is taken care of. If anything else arises, re-open or file a new ticket.

Note: See TracTickets for help on using tickets. You may also have a look at Agilo extensions to the ticket.

1.3.13-PRO © 2008-2011 Agilo Software all rights reserved (this page was served in: 0.67396 sec.)

We're Hiring!