Task #10116 (closed)
Opened 11 years ago
Closed 10 years ago
Bug: Insight CPU @ 300% (4.4+)
| Reported by: | bpindelski | Owned by: | jburel |
|---|---|---|---|
| Priority: | blocker | Milestone: | 5.0.2 |
| Component: | Insight | Version: | 4.4.8 |
| Keywords: | n.a. | Cc: | ux@… |
| Resources: | n.a. | Referenced By: | n.a. |
| References: | n.a. | Remaining Time: | n.a. |
| Sprint: | n.a. |
Description (last modified by pwalczysko)
After import, a folder for the images is created, but when trying to browse, CPU usages jumps up to 300% and Insight hangs. On other occasions the thumbnails aren't present.
Server: gretzky
Build: OMERO.insight-4.4.4-766-7e18246-dirty-ice33-b111-mac
also
Server: howe
Build: 4.4.5-602-2ea0777-dirty-ice33-b124 (webstart)
also
Server: Howe
Build: OMERO.insight-4.4.5-701-5eceada-ice33-b138-mac (NOT webstart)
Attachments (21)
Change History (83)
comment:1 Changed 11 years ago by bpindelski
- Component changed from General to Insight
- Owner set to jburel
comment:2 Changed 11 years ago by jburel
comment:3 Changed 11 years ago by bpindelski
No, nothing in the log J-M. I had to force-quit it, so it didn't manage to log anything besides the moment when I logged in.
comment:4 Changed 11 years ago by rkferguson
I had same issue when trying to expand dataset in tree after importing a variety of files.
Had spinning progress wheel top right of Insight and fans went crazy.
Had to force quit.
Log extract and screenshot attached.
Changed 11 years ago by rkferguson
Changed 11 years ago by rkferguson
comment:5 Changed 11 years ago by sylittlewood
I had the same Socket exception as Gus as well at times. As well as the below.
My log file has the following entries around the time it occurred on my machine. The sheer volume of entries over 1 second intervals might be an indication.
I was performing Imports of several file types within the standalone OMERO.Importer while trying to view and verify other imported images in OMERO.Insight.
comment:6 Changed 11 years ago by jburel
- Cc ux@… added
- Milestone changed from Unscheduled to OMERO-4.5
comment:7 Changed 11 years ago by jmoore
- Cc fs@… added
- Owner changed from jburel to bpindelski
Blazej, could you try to reproduce, and while Insight is spinning, take a stack dump?
Added: fs@ to go along with the "Fs" keyword.
comment:8 Changed 11 years ago by bpindelski
Josh, tried reproducing today, but to no avail. Cellsense image set imported with errors for frame_t_0.ets (Cannot create thumbnail). Browsing the dataset doesn't upset Inisght. The thumbnails are black, but images can be viewed. I can't open Image_01_V76 K2 11_lung.overview.jpg and frame_t_0.ets. For other images, the pyramids are incorrect (i.e. the zoom level and what is rendered).
comment:9 Changed 11 years ago by spli
Importing multiple files (testing files from imacon through to iplab). Tried to browse some imports midway. Saw a dialog pop up for maybe half a second and close again- might've been a network connection warning but it was too fast, I didn't notice any other signs of network glitches. Screenshot and jstack output attached.
Using Insight webstart OMERO.web 4.4.4-823-b02a128-dirty-ice33-b116
Changed 11 years ago by spli
Changed 11 years ago by spli
comment:10 Changed 11 years ago by jmoore
Nothing particularly obvious in the stack trace. Did you kill the client? Did it stop consuming the CPU on its own?
comment:11 Changed 11 years ago by mtbcarroll
For what it's worth, I've had a couple of very brief glimpses of network connection warnings.
comment:12 Changed 11 years ago by pwalczysko
I have the same experience here. After import of one-tiffs, I have started checking them in full viewer. When I came to some eight image, couple of things happpened. First, the bug with grey preview panel appeared (was sorted on Howe)j After further clicking around (all imports were finished successfully by that time), an intermittent message "The Network is down...". Then some thumbnails which were okay already before went grey. Nothing opens in full viewer. Also the CPU cooling is going crazy. Attaching insight.Log
Changed 11 years ago by pwalczysko
comment:13 Changed 11 years ago by jmoore
Gus saw something possibly related with Olympus ScanR see #10180.
comment:14 Changed 11 years ago by jmoore
From Andrew's log:
28 "pool-1-thread-1" prio=5 tid=1109de800 nid=0x1149c1000 runnable [1149c0000] 27 java.lang.Thread.State: RUNNABLE 26 at java.util.Arrays.sort(Arrays.java:1210) 25 at java.util.Collections.sort(Collections.java:157) 24 at IceInternal.RoutableReference.filterEndpoints(RoutableReference.java:747) 23 at IceInternal.RoutableReference.createConnection(RoutableReference.java:831) 22 at IceInternal.RoutableReference$1.setEndpoints(RoutableReference.java:484) 21 at IceInternal.RouterInfo.getClientEndpoints(RouterInfo.java:92) 20 at IceInternal.RoutableReference.getConnection(RoutableReference.java:476) 19 at IceInternal.ConnectRequestHandler.connect(ConnectRequestHandler.java:41) 18 at Ice._ObjectDelM.setup(_ObjectDelM.java:275) 17 at Ice.ObjectPrxHelperBase.createDelegate(ObjectPrxHelperBase.java:1007) 16 at Ice.ObjectPrxHelperBase.__getDelegate(ObjectPrxHelperBase.java:946) 15 - locked <7be5c2930> (a omero.api.ServiceFactoryPrxHelper) 14 at omero.api.ServiceFactoryPrxHelper.keepAllAlive(ServiceFactoryPrxHelper.java:1648) 13 at omero.api.ServiceFactoryPrxHelper.keepAllAlive(ServiceFactoryPrxHelper.java:1622) 12 at org.openmicroscopy.shoola.env.data.Connector.keepSessionAlive(Connector.java:708) 11 at org.openmicroscopy.shoola.env.data.OMEROGateway.keepSessionAlive(OMEROGateway.java:5542) 10 at org.openmicroscopy.shoola.env.data.KeepClientAlive.run(KeepClientAlive.java:63)
That's not one I've ever seen before. I wonder if it's happening more frequently than is common for some reason.
comment:15 Changed 11 years ago by ajpatterson
Hung after import when I tried to open first dataset.
comment:16 Changed 11 years ago by bpindelski
Another round of testing on 16/01/2013. Cellsens doesn't import at all and after a while Insight crashes with:
java.lang.Exception: org.openmicroscopy.shoola.env.data.DSOutOfServiceException: Cannot access the Metadata service. at org.openmicroscopy.shoola.env.data.OMEROGateway.getMetadataService(OMEROGateway.java:1559) at org.openmicroscopy.shoola.env.data.OMEROGateway.loadAnnotations(OMEROGateway.java:2737) at org.openmicroscopy.shoola.env.data.OmeroMetadataServiceImpl.loadStructuredAnnotations(OmeroMetadataServiceImpl.java:1022) at org.openmicroscopy.shoola.env.data.OmeroMetadataServiceImpl.loadStructuredData(OmeroMetadataServiceImpl.java:707) at org.openmicroscopy.shoola.env.data.OmeroMetadataServiceImpl.loadStructuredData(OmeroMetadataServiceImpl.java:811) at org.openmicroscopy.shoola.env.data.views.calls.StructuredAnnotationLoader$6.doCall(StructuredAnnotationLoader.java:211) at org.openmicroscopy.shoola.env.data.views.BatchCall.doStep(BatchCall.java:144) at org.openmicroscopy.shoola.util.concur.tasks.CompositeTask.doStep(CompositeTask.java:226) at org.openmicroscopy.shoola.env.data.views.CompositeBatchCall.doStep(CompositeBatchCall.java:126) at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.exec(ExecCommand.java:165) at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.run(ExecCommand.java:276) at org.openmicroscopy.shoola.util.concur.tasks.AsyncProcessor$Runner.run(AsyncProcessor.java:91) at java.lang.Thread.run(Unknown Source) at org.openmicroscopy.shoola.env.ui.UserNotifierImpl.showErrorDialog(UserNotifierImpl.java:189) at org.openmicroscopy.shoola.env.ui.UserNotifierImpl.notifyError(UserNotifierImpl.java:287) at org.openmicroscopy.shoola.env.ui.UserNotifierImpl.notifyError(UserNotifierImpl.java:257) at org.openmicroscopy.shoola.agents.metadata.MetadataLoader.handleException(MetadataLoader.java:169) at org.openmicroscopy.shoola.env.data.events.DSCallAdapter.eventFired(DSCallAdapter.java:84) at org.openmicroscopy.shoola.env.data.views.BatchCallMonitor$1.run(BatchCallMonitor.java:124) at java.awt.event.InvocationEvent.dispatch(Unknown Source) at java.awt.EventQueue.dispatchEventImpl(Unknown Source) at java.awt.EventQueue.access$200(Unknown Source) at java.awt.EventQueue$3.run(Unknown Source) at java.awt.EventQueue$3.run(Unknown Source) at java.security.AccessController.doPrivileged(Native Method) at java.security.ProtectionDomain$1.doIntersectionPrivilege(Unknown Source) at java.awt.EventQueue.dispatchEvent(Unknown Source) at java.awt.EventDispatchThread.pumpOneEventForFilters(Unknown Source) at java.awt.EventDispatchThread.pumpEventsForFilter(Unknown Source) at java.awt.EventDispatchThread.pumpEventsForFilter(Unknown Source) at java.awt.WaitDispatchSupport$2.run(Unknown Source) at java.awt.event.InvocationEvent.dispatch(Unknown Source) at java.awt.EventQueue.dispatchEventImpl(Unknown Source) at java.awt.EventQueue.access$200(Unknown Source) at java.awt.EventQueue$3.run(Unknown Source) at java.awt.EventQueue$3.run(Unknown Source) at java.security.AccessController.doPrivileged(Native Method) at java.security.ProtectionDomain$1.doIntersectionPrivilege(Unknown Source) at java.awt.EventQueue.dispatchEvent(Unknown Source) at java.awt.EventDispatchThread.pumpOneEventForFilters(Unknown Source) at java.awt.EventDispatchThread.pumpEventsForFilter(Unknown Source) at java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown Source) at java.awt.EventDispatchThread.pumpEvents(Unknown Source) at java.awt.EventDispatchThread.pumpEvents(Unknown Source) at java.awt.EventDispatchThread.run(Unknown Source)
insight.log also reports a lost connection.
comment:17 Changed 11 years ago by pwalczysko
Partial hang again, after import of openlab-liff. The images do not open in full viewer, but otherwise Insight reacts. The fan is going turbo again.
From the Insight.log:
{[{
2013-01-16 11:20:57,789 INFO [ ome.formats.importer.ImportConfig?] (Thread-181) OMERO Version: 4.4.4-823-b02a128-dirty-ice33-b116
2013-01-16 11:20:57,789 INFO [ ome.formats.importer.ImportConfig?] (Thread-181) Bioformats version: 4.5-DEV revision: fb46c88 date: 16 January 2013
2013-01-16 11:20:57,844 INFO [ ome.formats.importer.ImportConfig?] (Thread-181) OMERO Version: 4.4.4-823-b02a128-dirty-ice33-b116
2013-01-16 11:20:57,844 INFO [ ome.formats.importer.ImportConfig?] (Thread-181) Bioformats version: 4.5-DEV revision: fb46c88 date: 16 January 2013
2013-01-16 11:21:13,445 ERROR [py.shoola.env.data.OmeroImageServiceImpl?] (Thread-187) Thumbnail service null for pixelsID: [819, 821, 822]
2013-01-16 11:25:49,298 INFO [ola.agents.metadata.StructuredDataLoader?] (entQueue-0) The data retrieval has been cancelled.
}}}
Did Force Quit. Fan is quiet again.
comment:18 Changed 11 years ago by pwalczysko
The same problems with perkinelmer-operetta (Plate). The problems started when trying to see the wells in full viewer. Flash of "Network is down..." window, then hang, and loud fan. The error identical with the one already reported for one-tiff above.
2013-01-16 13:20:23,900 DEBUG [copy.shoola.env.data.DataServicesFactory] (Thread-319) Connection ErrorIce.SocketException
error = 0
at IceInternal.Selector.setInterrupt(Selector.java:347)
at IceInternal.Selector.remove(Selector.java:107)
at IceInternal.ThreadPool.finish(ThreadPool.java:183)
at Ice.ConnectionI.setState(ConnectionI.java:1454)
at Ice.ConnectionI.setState(ConnectionI.java:1351)
at Ice.ConnectionI.sendRequest(ConnectionI.java:383)
at IceInternal.ConnectionRequestHandler.sendRequest(ConnectionRequestHandler.java:36)
at IceInternal.Outgoing.invoke(Outgoing.java:66)
at omero.api._IMetadataDelM.loadAnnotations(_IMetadataDelM.java:260)
at omero.api.IMetadataPrxHelper.loadAnnotations(IMetadataPrxHelper.java:320)
at omero.api.IMetadataPrxHelper.loadAnnotations(IMetadataPrxHelper.java:292)
at org.openmicroscopy.shoola.env.data.OMEROGateway.loadAnnotations(OMEROGateway.java:2739)
at org.openmicroscopy.shoola.env.data.OmeroMetadataServiceImpl.loadStructuredAnnotations(OmeroMetadataServiceImpl.java:1022)
at org.openmicroscopy.shoola.env.data.OmeroMetadataServiceImpl.loadStructuredData(OmeroMetadataServiceImpl.java:707)
at org.openmicroscopy.shoola.env.data.OmeroMetadataServiceImpl.loadStructuredData(OmeroMetadataServiceImpl.java:811)
at org.openmicroscopy.shoola.env.data.views.calls.StructuredAnnotationLoader$6.doCall(StructuredAnnotationLoader.java:211)
at org.openmicroscopy.shoola.env.data.views.BatchCall.doStep(BatchCall.java:144)
at org.openmicroscopy.shoola.util.concur.tasks.CompositeTask.doStep(CompositeTask.java:226)
at org.openmicroscopy.shoola.env.data.views.CompositeBatchCall.doStep(CompositeBatchCall.java:126)
at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.exec(ExecCommand.java:165)
at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.run(ExecCommand.java:276)
at org.openmicroscopy.shoola.util.concur.tasks.AsyncProcessor$Runner.run(AsyncProcessor.java:91)
at java.lang.Thread.run(Thread.java:680)
Caused by: java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184)
at sun.nio.ch.SinkChannelImpl.write(SinkChannelImpl.java:153)
at IceInternal.Selector.setInterrupt(Selector.java:343)
... 22 more
Connection ErrorIce.SocketException
error = 0
at IceInternal.Selector.setInterrupt(Selector.java:347)
at IceInternal.Selector.remove(Selector.java:107)
at IceInternal.ThreadPool.finish(ThreadPool.java:183)
at Ice.ConnectionI.setState(ConnectionI.java:1454)
at Ice.ConnectionI.setState(ConnectionI.java:1351)
at Ice.ConnectionI.sendRequest(ConnectionI.java:383)
at IceInternal.ConnectionRequestHandler.sendRequest(ConnectionRequestHandler.java:36)
at IceInternal.Outgoing.invoke(Outgoing.java:66)
at omero.api._IMetadataDelM.loadAnnotations(_IMetadataDelM.java:260)
at omero.api.IMetadataPrxHelper.loadAnnotations(IMetadataPrxHelper.java:320)
at omero.api.IMetadataPrxHelper.loadAnnotations(IMetadataPrxHelper.java:292)
at org.openmicroscopy.shoola.env.data.OMEROGateway.loadAnnotations(OMEROGateway.java:2739)
at org.openmicroscopy.shoola.env.data.OmeroMetadataServiceImpl.loadStructuredAnnotations(OmeroMetadataServiceImpl.java:1022)
at org.openmicroscopy.shoola.env.data.OmeroMetadataServiceImpl.loadStructuredData(OmeroMetadataServiceImpl.java:707)
at org.openmicroscopy.shoola.env.data.OmeroMetadataServiceImpl.loadStructuredData(OmeroMetadataServiceImpl.java:811)
at org.openmicroscopy.shoola.env.data.views.calls.StructuredAnnotationLoader$6.doCall(StructuredAnnotationLoader.java:211)
at org.openmicroscopy.shoola.env.data.views.BatchCall.doStep(BatchCall.java:144)
at org.openmicroscopy.shoola.util.concur.tasks.CompositeTask.doStep(CompositeTask.java:226)
at org.openmicroscopy.shoola.env.data.views.CompositeBatchCall.doStep(CompositeBatchCall.java:126)
at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.exec(ExecCommand.java:165)
at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.run(ExecCommand.java:276)
at org.openmicroscopy.shoola.util.concur.tasks.AsyncProcessor$Runner.run(AsyncProcessor.java:91)
at java.lang.Thread.run(Thread.java:680)
Caused by: java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184)
at sun.nio.ch.SinkChannelImpl.write(SinkChannelImpl.java:153)
at IceInternal.Selector.setInterrupt(Selector.java:343)
... 22 more
Exception in thread "Thread-319"
comment:19 follow-up: ↓ 21 Changed 11 years ago by jburel
that's probably b/c gretzky was restarted
comment:20 Changed 11 years ago by jmoore
Jean-Marie: no, the gretzky restart is unrelated. Something very weird is going on with the CPU and these sockets. 4-5 different people saw the same behavior while testing this morning.
comment:21 in reply to: ↑ 19 Changed 11 years ago by pwalczysko
Replying to jburel:
that's probably b/c gretzky was restarted
But the error and the event are almost identical with the one-tiff case described above - at that time Gretzky was not being restarted.
comment:22 Changed 11 years ago by sylittlewood
Just had another 300% CPU freeze. I was performing an import while attempting to verify Thumbnails and viewer functionality for another already imported file type. My logs show tht a connection error occurred, this then resulted in the CPU spike.
2013-01-16 17:01:06,403 DEBUG [ ome.scifio.io.Location] ( Thread-48) Location.mapFile: embedded-stream.raw -> null
2013-01-16 17:01:06,405 DEBUG [ ome.scifio.io.Location] ( Thread-48) Location.mapFile: embedded-stream.raw -> null
2013-01-16 17:01:06,405 DEBUG [ ome.scifio.io.Location] ( Thread-48) Location.mapFile: embedded-stream.raw -> null
2013-01-16 17:01:06,408 DEBUG [ ome.scifio.io.Location] ( Thread-48) Location.mapFile: embedded-stream.raw -> null
2013-01-16 17:01:06,408 DEBUG [ ome.scifio.io.Location] ( Thread-48) Location.mapFile: embedded-stream.raw -> null
2013-01-16 17:01:06,410 DEBUG [ ome.scifio.io.Location] ( Thread-48) Location.mapFile: embedded-stream.raw -> null
2013-01-16 17:01:06,410 DEBUG [ ome.scifio.io.Location] ( Thread-48) Location.mapFile: embedded-stream.raw -> null
2013-01-16 17:01:06,413 DEBUG [ ome.scifio.io.Location] ( Thread-48) Location.mapFile: embedded-stream.raw -> null
2013-01-16 17:01:06,413 DEBUG [ ome.scifio.io.Location] ( Thread-48) Location.mapFile: embedded-stream.raw -> null
2013-01-16 17:01:06,450 DEBUG [ ome.scifio.io.Location] ( Thread-48) Location.mapFile: embedded-stream.raw -> null
2013-01-16 17:01:06,450 DEBUG [ ome.scifio.io.Location] ( Thread-48) Location.mapFile: embedded-stream.raw -> null
2013-01-16 17:01:06,463 DEBUG [ ome.scifio.io.Location] ( Thread-48) Location.mapFile: embedded-stream.raw -> null
2013-01-16 17:01:06,464 DEBUG [ ome.scifio.io.Location] ( Thread-48) Location.mapFile: embedded-stream.raw -> null
2013-01-16 17:01:06,466 DEBUG [ ome.scifio.io.Location] ( Thread-48) Location.mapFile: embedded-stream.raw -> null
2013-01-16 17:01:06,466 DEBUG [ ome.scifio.io.Location] ( Thread-48) Location.mapFile: embedded-stream.raw -> null
2013-01-16 17:01:06,469 DEBUG [ ome.scifio.io.Location] ( Thread-48) Location.mapFile: embedded-stream.raw -> null
2013-01-16 17:01:06,469 DEBUG [ ome.scifio.io.Location] ( Thread-48) Location.mapFile: embedded-stream.raw -> null
2013-01-16 17:01:06,471 DEBUG [ ome.scifio.io.Location] ( Thread-48) Location.mapFile: embedded-stream.raw -> null
2013-01-16 17:01:06,471 DEBUG [ ome.scifio.io.Location] ( Thread-48) Location.mapFile: embedded-stream.raw -> null
2013-01-16 17:01:06,485 DEBUG [ ome.scifio.io.Location] ( Thread-48) Location.mapFile: embedded-stream.raw -> null
2013-01-16 17:01:06,485 DEBUG [ ome.scifio.io.Location] ( Thread-48) Location.mapFile: embedded-stream.raw -> null
2013-01-16 17:01:06,687 DEBUG [ ome.scifio.io.Location] ( Thread-48) Location.mapFile: embedded-stream.raw -> null
2013-01-16 17:01:06,690 INFO [ ome.formats.importer.ImportCandidates] ( Thread-48) 4437 file(s) parsed into 4 group(s) with 4090 call(s) to setId in 41188ms. (41992ms total) [4086 unknowns]
2013-01-16 17:01:06,746 INFO [ ome.formats.importer.ImportConfig] ( Thread-48) OMERO Version: 4.4.4-824-61182a3-dirty-ice33-b117
2013-01-16 17:01:06,746 INFO [ ome.formats.importer.ImportConfig] ( Thread-48) Bioformats version: 4.5-DEV revision: 1595a7b date: 16 January 2013
2013-01-16 17:01:06,747 INFO [ ome.formats.importer.ImportCandidates] ( Thread-48) Depth: 4 Metadata Level: MINIMUM
2013-01-16 17:01:06,747 DEBUG [ ome.scifio.io.Location] ( Thread-48) Location.mapFile: embedded-stream.raw -> null
2013-01-16 17:01:06,751 DEBUG [ ome.scifio.io.Location] ( Thread-48) Location.mapFile: embedded-stream.raw -> null
2013-01-16 17:01:06,761 DEBUG [ ome.scifio.io.Location] ( Thread-48) Location.mapFile: embedded-stream.raw -> null
2013-01-16 17:01:06,762 INFO [ ome.formats.importer.ImportCandidates] ( Thread-48) 1 file(s) parsed into 1 group(s) with 1 call(s) to setId in 14ms. (15ms total) [0 unknowns]
2013-01-16 17:01:06,762 INFO [ ome.formats.importer.ImportConfig] ( Thread-48) OMERO Version: 4.4.4-824-61182a3-dirty-ice33-b117
2013-01-16 17:01:06,762 INFO [ ome.formats.importer.ImportConfig] ( Thread-48) Bioformats version: 4.5-DEV revision: 1595a7b date: 16 January 2013
2013-01-16 17:01:06,824 INFO [ ome.formats.importer.ImportConfig] ( Thread-48) OMERO Version: 4.4.4-824-61182a3-dirty-ice33-b117
2013-01-16 17:01:06,824 INFO [ ome.formats.importer.ImportConfig] ( Thread-48) Bioformats version: 4.5-DEV revision: 1595a7b date: 16 January 2013
2013-01-16 17:01:21,617 INFO [ ome.formats.importer.ImportConfig] ( Thread-48) OMERO Version: 4.4.4-824-61182a3-dirty-ice33-b117
2013-01-16 17:01:21,617 INFO [ ome.formats.importer.ImportConfig] ( Thread-48) Bioformats version: 4.5-DEV revision: 1595a7b date: 16 January 2013
2013-01-16 17:01:21,619 INFO [ ome.formats.importer.ImportCandidates] ( Thread-48) Depth: 4 Metadata Level: MINIMUM
2013-01-16 17:01:21,619 DEBUG [ ome.scifio.io.Location] ( Thread-48) Location.mapFile: embedded-stream.raw -> null
2013-01-16 17:01:21,781 DEBUG [ ome.scifio.io.Location] ( Thread-48) Location.mapFile: embedded-stream.raw -> null
2013-01-16 17:01:21,782 INFO [ ome.formats.importer.ImportCandidates] ( Thread-48) 1 file(s) parsed into 1 group(s) with 1 call(s) to setId in 162ms. (164ms total) [0 unknowns]
2013-01-16 17:01:21,839 INFO [ ome.formats.importer.ImportConfig] ( Thread-48) OMERO Version: 4.4.4-824-61182a3-dirty-ice33-b117
2013-01-16 17:01:21,839 INFO [ ome.formats.importer.ImportConfig] ( Thread-48) Bioformats version: 4.5-DEV revision: 1595a7b date: 16 January 2013
2013-01-16 17:01:21,853 INFO [ ome.formats.importer.ImportConfig] ( Thread-48) OMERO Version: 4.4.4-824-61182a3-dirty-ice33-b117
2013-01-16 17:01:21,854 INFO [ ome.formats.importer.ImportConfig] ( Thread-48) Bioformats version: 4.5-DEV revision: 1595a7b date: 16 January 2013
2013-01-16 17:01:24,624 INFO [ ome.formats.importer.ImportConfig] ( Thread-48) OMERO Version: 4.4.4-824-61182a3-dirty-ice33-b117
2013-01-16 17:01:24,624 INFO [ ome.formats.importer.ImportConfig] ( Thread-48) Bioformats version: 4.5-DEV revision: 1595a7b date: 16 January 2013
2013-01-16 17:01:24,624 INFO [ ome.formats.importer.ImportCandidates] ( Thread-48) Depth: 4 Metadata Level: MINIMUM
2013-01-16 17:01:24,624 DEBUG [ ome.scifio.io.Location] ( Thread-48) Location.mapFile: embedded-stream.raw -> null
2013-01-16 17:01:24,655 DEBUG [ ome.scifio.io.Location] ( Thread-48) Location.mapFile: embedded-stream.raw -> null
2013-01-16 17:01:25,034 DEBUG [ ome.scifio.io.Location] ( Thread-48) Location.mapFile: embedded-stream.raw -> null
2013-01-16 17:01:25,036 INFO [ ome.formats.importer.ImportCandidates] ( Thread-48) 1 file(s) parsed into 1 group(s) with 1 call(s) to setId in 410ms. (412ms total) [0 unknowns]
2013-01-16 17:01:25,037 INFO [ ome.formats.importer.ImportConfig] ( Thread-48) OMERO Version: 4.4.4-824-61182a3-dirty-ice33-b117
2013-01-16 17:01:25,037 INFO [ ome.formats.importer.ImportConfig] ( Thread-48) Bioformats version: 4.5-DEV revision: 1595a7b date: 16 January 2013
2013-01-16 17:01:25,051 INFO [ ome.formats.importer.ImportConfig] ( Thread-48) OMERO Version: 4.4.4-824-61182a3-dirty-ice33-b117
2013-01-16 17:01:25,051 INFO [ ome.formats.importer.ImportConfig] ( Thread-48) Bioformats version: 4.5-DEV revision: 1595a7b date: 16 January 2013
2013-01-16 17:03:34,120 ERROR [py.shoola.env.data.OmeroImageServiceImpl] ( Thread-53) Thumbnail service null for pixelsID: [1346]
2013-01-16 17:05:06,212 DEBUG [copy.shoola.env.data.DataServicesFactory] ( Thread-66) Connection ErrorIce.SocketException
error = 0
at IceInternal.Selector.setInterrupt(Selector.java:347)
at IceInternal.Selector.remove(Selector.java:107)
at IceInternal.ThreadPool.finish(ThreadPool.java:183)
at Ice.ConnectionI.setState(ConnectionI.java:1454)
at Ice.ConnectionI.setState(ConnectionI.java:1351)
at Ice.ConnectionI.sendRequest(ConnectionI.java:383)
at IceInternal.ConnectionRequestHandler.sendRequest(ConnectionRequestHandler.java:36)
at IceInternal.Outgoing.invoke(Outgoing.java:66)
at omero.api._IMetadataDelM.loadAnnotations(_IMetadataDelM.java:260)
at omero.api.IMetadataPrxHelper.loadAnnotations(IMetadataPrxHelper.java:320)
at omero.api.IMetadataPrxHelper.loadAnnotations(IMetadataPrxHelper.java:292)
at org.openmicroscopy.shoola.env.data.OMEROGateway.loadAnnotations(OMEROGateway.java:2739)
at org.openmicroscopy.shoola.env.data.OmeroMetadataServiceImpl.loadStructuredAnnotations(OmeroMetadataServiceImpl.java:1022)
at org.openmicroscopy.shoola.env.data.OmeroMetadataServiceImpl.loadStructuredData(OmeroMetadataServiceImpl.java:707)
at org.openmicroscopy.shoola.env.data.OmeroMetadataServiceImpl.loadStructuredData(OmeroMetadataServiceImpl.java:811)
at org.openmicroscopy.shoola.env.data.views.calls.StructuredAnnotationLoader$6.doCall(StructuredAnnotationLoader.java:211)
at org.openmicroscopy.shoola.env.data.views.BatchCall.doStep(BatchCall.java:144)
at org.openmicroscopy.shoola.util.concur.tasks.CompositeTask.doStep(CompositeTask.java:226)
at org.openmicroscopy.shoola.env.data.views.CompositeBatchCall.doStep(CompositeBatchCall.java:126)
at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.exec(ExecCommand.java:165)
at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.run(ExecCommand.java:276)
at org.openmicroscopy.shoola.util.concur.tasks.AsyncProcessor$Runner.run(AsyncProcessor.java:91)
at java.lang.Thread.run(Thread.java:680)
Caused by: java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184)
at sun.nio.ch.SinkChannelImpl.write(SinkChannelImpl.java:153)
at IceInternal.Selector.setInterrupt(Selector.java:343)
... 22 more
Connection ErrorIce.SocketException
error = 0
at IceInternal.Selector.setInterrupt(Selector.java:347)
at IceInternal.Selector.remove(Selector.java:107)
at IceInternal.ThreadPool.finish(ThreadPool.java:183)
at Ice.ConnectionI.setState(ConnectionI.java:1454)
at Ice.ConnectionI.setState(ConnectionI.java:1351)
at Ice.ConnectionI.sendRequest(ConnectionI.java:383)
at IceInternal.ConnectionRequestHandler.sendRequest(ConnectionRequestHandler.java:36)
at IceInternal.Outgoing.invoke(Outgoing.java:66)
at omero.api._IMetadataDelM.loadAnnotations(_IMetadataDelM.java:260)
at omero.api.IMetadataPrxHelper.loadAnnotations(IMetadataPrxHelper.java:320)
at omero.api.IMetadataPrxHelper.loadAnnotations(IMetadataPrxHelper.java:292)
at org.openmicroscopy.shoola.env.data.OMEROGateway.loadAnnotations(OMEROGateway.java:2739)
at org.openmicroscopy.shoola.env.data.OmeroMetadataServiceImpl.loadStructuredAnnotations(OmeroMetadataServiceImpl.java:1022)
at org.openmicroscopy.shoola.env.data.OmeroMetadataServiceImpl.loadStructuredData(OmeroMetadataServiceImpl.java:707)
at org.openmicroscopy.shoola.env.data.OmeroMetadataServiceImpl.loadStructuredData(OmeroMetadataServiceImpl.java:811)
at org.openmicroscopy.shoola.env.data.views.calls.StructuredAnnotationLoader$6.doCall(StructuredAnnotationLoader.java:211)
at org.openmicroscopy.shoola.env.data.views.BatchCall.doStep(BatchCall.java:144)
at org.openmicroscopy.shoola.util.concur.tasks.CompositeTask.doStep(CompositeTask.java:226)
at org.openmicroscopy.shoola.env.data.views.CompositeBatchCall.doStep(CompositeBatchCall.java:126)
at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.exec(ExecCommand.java:165)
at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.run(ExecCommand.java:276)
at org.openmicroscopy.shoola.util.concur.tasks.AsyncProcessor$Runner.run(AsyncProcessor.java:91)
at java.lang.Thread.run(Thread.java:680)
Caused by: java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184)
at sun.nio.ch.SinkChannelImpl.write(SinkChannelImpl.java:153)
at IceInternal.Selector.setInterrupt(Selector.java:343)
... 22 more
Exception in thread "Thread-66"
comment:23 Changed 11 years ago by spli
I should probably add I've seen the 300% CPU issue a couple of times just when viewing images (seems to only occur with ones which haven't imported properly, so are showing blank/black thumbnails). No imports at all were being done.
comment:24 Changed 11 years ago by ajpatterson
When I had the hand above I was trying to view thumbnails for a image that has imported fine. I can view it in web ok.
comment:25 Changed 11 years ago by pwalczysko
When trying to open the auto-pre-imported lei files on gretzky in user-2 group read-only-1, in Orphaned images.
All thumbnails appeared grey. When double-clicking on one of them, the full viewer never came up, just waiting for minutes. Cancelled the full viewer. Error appeared after a while.
Fan on laptop went turbo.
Force quit Insight.
When logged in again and coming to the dataset for the second time, it looked like the thumbnails needed ages to appear properly and were finishing to do so from the previous time- there were still the last two which were grey, which improved within ~1 min. After this, the thumbnails were okay, the images were opening okay as well.
Fan was quiet.
java.lang.Exception: java.lang.StackOverflowError at java.util.Arrays.sort(Arrays.java:1210) at java.util.Collections.sort(Collections.java:157) at IceInternal.RoutableReference.filterEndpoints(RoutableReference.java:747) at IceInternal.RoutableReference.createConnection(RoutableReference.java:831) at IceInternal.RoutableReference.getConnectionNoRouterInfo(RoutableReference.java:510) at IceInternal.RoutableReference.getConnection(RoutableReference.java:501) at IceInternal.ConnectRequestHandler.connect(ConnectRequestHandler.java:41) at Ice._ObjectDelM.setup(_ObjectDelM.java:275) at Ice.ObjectPrxHelperBase.createDelegate(ObjectPrxHelperBase.java:1007) at Ice.ObjectPrxHelperBase.__getDelegate(ObjectPrxHelperBase.java:946) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:298) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at IceInternal.OutgoingAsync.__send(OutgoingAsync.java:303) at Ice.ObjectPrxHelperBase.__handleExceptionWrapperRelaxed(ObjectPrxHelperBase.java:911) at IceInternal.OutgoingAsync.handleException(OutgoingAsync.java:406) at org.openmicroscopy.shoola.env.ui.UserNotifierImpl.showErrorDialog(UserNotifierImpl.java:189) at org.openmicroscopy.shoola.env.ui.UserNotifierImpl.notifyError(UserNotifierImpl.java:287) at org.openmicroscopy.shoola.env.ui.UserNotifierImpl.notifyError(UserNotifierImpl.java:257) at org.openmicroscopy.shoola.agents.metadata.EditorLoader.handleException(EditorLoader.java:159) at org.openmicroscopy.shoola.env.data.events.DSCallAdapter.eventFired(DSCallAdapter.java:84) at org.openmicroscopy.shoola.env.data.views.BatchCallMonitor$1.run(BatchCallMonitor.java:124) at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:209) at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:702) at java.awt.EventQueue.access$400(EventQueue.java:82) at java.awt.EventQueue$2.run(EventQueue.java:663) at java.awt.EventQueue$2.run(EventQueue.java:661) at java.security.AccessController.doPrivileged(Native Method) at java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:87) at java.awt.EventQueue.dispatchEvent(EventQueue.java:672) at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:296) at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:211) at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:205) at java.awt.Dialog$1.run(Dialog.java:1044) at java.awt.Dialog$3.run(Dialog.java:1096) at java.security.AccessController.doPrivileged(Native Method) at java.awt.Dialog.show(Dialog.java:1094) at java.awt.Component.show(Component.java:1591) at java.awt.Component.setVisible(Component.java:1543) at java.awt.Window.setVisible(Window.java:843) at java.awt.Dialog.setVisible(Dialog.java:984) at org.openmicroscopy.shoola.util.ui.UIUtilities.centerAndShow(UIUtilities.java:557) at org.openmicroscopy.shoola.env.ui.UserNotifierImpl.showErrorDialog(UserNotifierImpl.java:197) at org.openmicroscopy.shoola.env.ui.UserNotifierImpl.notifyError(UserNotifierImpl.java:287) at org.openmicroscopy.shoola.env.ui.UserNotifierImpl.notifyError(UserNotifierImpl.java:257) at org.openmicroscopy.shoola.agents.metadata.EditorLoader.handleException(EditorLoader.java:159) at org.openmicroscopy.shoola.env.data.events.DSCallAdapter.eventFired(DSCallAdapter.java:84) at org.openmicroscopy.shoola.env.data.views.BatchCallMonitor$1.run(BatchCallMonitor.java:124) at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:209) at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:702) at java.awt.EventQueue.access$400(EventQueue.java:82) at java.awt.EventQueue$2.run(EventQueue.java:663) at java.awt.EventQueue$2.run(EventQueue.java:661) at java.security.AccessController.doPrivileged(Native Method) at java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:87) at java.awt.EventQueue.dispatchEvent(EventQueue.java:672) at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:296) at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:211) at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:205) at java.awt.Dialog$1.run(Dialog.java:1044) at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:209) at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:702) at java.awt.EventQueue.access$400(EventQueue.java:82) at java.awt.EventQueue$2.run(EventQueue.java:663) at java.awt.EventQueue$2.run(EventQueue.java:661) at java.security.AccessController.doPrivileged(Native Method) at java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:87) at java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:98) at java.awt.EventQueue$3.run(EventQueue.java:677) at java.awt.EventQueue$3.run(EventQueue.java:675) at java.security.AccessController.doPrivileged(Native Method) at java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:87) at java.awt.EventQueue.dispatchEvent(EventQueue.java:674) at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:296) at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:211) at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:201) at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:196) at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:188) at java.awt.EventDispatchThread.run(EventDispatchThread.java:122)
Changed 11 years ago by pwalczysko
comment:26 Changed 11 years ago by spli
Just seen the 300% cpu issue with Howe (jstack-l-30188.txt). Imported some screens successfully, whilst browsing them a network dialog momentarily popped up and closed again, and the CPU hit 300%.
comment:27 Changed 11 years ago by jmoore
- Cc fs@… removed
- Description modified (diff)
- Keywords fs removed
- Priority changed from major to blocker
- Summary changed from Bug: Cellsense crashes Insight to Bug: Insight CPU @ 300% (4.4+)
Removing "FS" keyword and upping this since it occurs on dev_4_4. Again, all the threads are hanging on isSessionAlive. One particularly interesting stack:
68 "Thread-194" prio=5 tid=0x00007f8046832000 nid=0x2fc0f runnable [0x000000016e277000] 69 java.lang.Thread.State: RUNNABLE 70 at java.security.AccessController.getStackAccessControlContext(Native Method) 71 at java.security.AccessController.checkPermission(AccessController.java:534) 72 at java.lang.SecurityManager.checkPermission(SecurityManager.java:549) 73 at java.lang.SecurityManager.checkPropertyAccess(SecurityManager.java:1302) 74 at java.lang.System.getProperty(System.java:706) 75 at Ice.PropertiesI.getPropertyAsIntWithDefault(PropertiesI.java:95) 76 - locked <0x0000000116885908> (a Ice.PropertiesI) 77 at IceInternal.ConnectRequestHandler.<init>(ConnectRequestHandler.java:329) 78 at Ice._ObjectDelM.setup(_ObjectDelM.java:274) 79 at Ice.ObjectPrxHelperBase.createDelegate(ObjectPrxHelperBase.java:1007) 80 at Ice.ObjectPrxHelperBase.__getDelegate(ObjectPrxHelperBase.java:946) 81 - locked <0x00000001168ebf20> (a omero.api.ServiceFactoryPrxHelper) 82 at omero.api.ServiceFactoryPrxHelper.keepAlive(ServiceFactoryPrxHelper.java:1603)
comment:28 Changed 11 years ago by pwalczysko
The new local build from J_M' from today, on squig, OMERO.insight-4.4.5-DEV-ice33-mac. The issues described above for Howe by spli do not happen in this build. This build also solves the #10244 ticket.
comment:29 Changed 11 years ago by spli
Imported a screen successfully (I think it was cellworx), browsed it, all was fine. Attempted to quit, CPU shot up to 400%.
Changed 11 years ago by spli
insight log corresponding to the previous stacktrace (note this was for insight webstart)
comment:30 Changed 11 years ago by jmoore
- Owner changed from bpindelski to jburel
Another duplicate: #10269
comment:31 Changed 11 years ago by pwalczysko
Using Webstart Insight on howe. Freshly upgraded my java to Java 7.
Went to OMERO.web 4.4.5-701-5eceada-ice33-b138. (webclient) and did Insight Webstart.
Manipulating big plates (54 wells, 4 fields) (test_images_good/cellworx/hms/plate 1/) ). CPU usage was around 50 % every time when doing "Apply to all" actions.
Did "simultaneously"
change channel name - apply to all,
render to different intensity - apply to all,
change fields,
2 imports of this plate still running
The actions were started quickly one after each other.
The last action was
Switch to another plate (the other actions still unfinished)
CPU is at 390%. Insight partially hangs (importer okay, full viewer okay, but Data Manater does not react.)
After a while it reaches 400%.
Insight completely stopped reacting.
Doing Force Quit.
Stack dump and Insight log attached.
Changed 11 years ago by pwalczysko
Changed 11 years ago by pwalczysko
comment:32 Changed 11 years ago by pwalczysko
Just repeated the bug on normal (NOT Webstart) Insight build on Howe (OMERO.insight-4.4.5-701-5eceada-ice33-b138-mac). Workflow identical with the one described above from today. Insight log and Stack Dump attached.
Changed 11 years ago by pwalczysko
Changed 11 years ago by pwalczysko
comment:33 Changed 11 years ago by pwalczysko
- Description modified (diff)
comment:34 Changed 11 years ago by jmoore
Just reproduced 300% CPU with todays build (OMERO.clients-4.4.5-709-db1d27a-ice33-b141.mac)
I connected as root and was looking at dataset user-6's testimages_256x256 dataset (in group read-annotate-1). There are a lot of red numbered images. I clicked on them as quickly as I could. While doing so, I received the "Connection down" message, and then processing went up to 300%. My assumption is that an exception is being thrown somewhere, that exception handling is interrupting as "network down" but that that's not really the case. Then, the connection is being actively killed, which is leaving us with a weird Ice bug. (I.e. it's a swallowed OMERO exception followed by Ice weirdness). See below.
I'll try to reproduce with the production 4.4.5 client now.
Exception in thread "AWT-EventQueue-0"
2013-02-04 20:50:30,865 DEBUG [copy.shoola.env.data.DataServicesFactory] (Thread-193) Connection ErrorIce.SocketException
error = 0
at IceInternal.Selector.setInterrupt(Selector.java:347)
at IceInternal.Selector.remove(Selector.java:107)
at IceInternal.ThreadPool.finish(ThreadPool.java:183)
at Ice.ConnectionI.setState(ConnectionI.java:1454)
at Ice.ConnectionI.setState(ConnectionI.java:1351)
at Ice.ConnectionI.sendRequest(ConnectionI.java:383)
at IceInternal.ConnectionRequestHandler.sendRequest(ConnectionRequestHandler.java:36)
at IceInternal.Outgoing.invoke(Outgoing.java:66)
at omero.api._IMetadataDelM.loadAnnotations(_IMetadataDelM.java:260)
at omero.api.IMetadataPrxHelper.loadAnnotations(IMetadataPrxHelper.java:320)
at omero.api.IMetadataPrxHelper.loadAnnotations(IMetadataPrxHelper.java:292)
at org.openmicroscopy.shoola.env.data.OMEROGateway.loadAnnotations(OMEROGateway.java:2740)
at org.openmicroscopy.shoola.env.data.OmeroMetadataServiceImpl.loadStructuredAnnotations(OmeroMetadataServiceImpl.java:1046)
at org.openmicroscopy.shoola.env.data.OmeroMetadataServiceImpl.loadStructuredData(OmeroMetadataServiceImpl.java:731)
at org.openmicroscopy.shoola.env.data.OmeroMetadataServiceImpl.loadStructuredData(OmeroMetadataServiceImpl.java:835)
at org.openmicroscopy.shoola.env.data.views.calls.StructuredAnnotationLoader$6.doCall(StructuredAnnotationLoader.java:211)
at org.openmicroscopy.shoola.env.data.views.BatchCall.doStep(BatchCall.java:144)
at org.openmicroscopy.shoola.util.concur.tasks.CompositeTask.doStep(CompositeTask.java:226)
at org.openmicroscopy.shoola.env.data.views.CompositeBatchCall.doStep(CompositeBatchCall.java:126)
at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.exec(ExecCommand.java:165)
at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.run(ExecCommand.java:276)
at org.openmicroscopy.shoola.util.concur.tasks.AsyncProcessor$Runner.run(AsyncProcessor.java:91)
at java.lang.Thread.run(Thread.java:680)
Caused by: java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184)
at sun.nio.ch.SinkChannelImpl.write(SinkChannelImpl.java:153)
at IceInternal.Selector.setInterrupt(Selector.java:343)
... 22 more
Connection ErrorIce.SocketException
error = 0
at IceInternal.Selector.setInterrupt(Selector.java:347)
at IceInternal.Selector.remove(Selector.java:107)
at IceInternal.ThreadPool.finish(ThreadPool.java:183)
at Ice.ConnectionI.setState(ConnectionI.java:1454)
at Ice.ConnectionI.setState(ConnectionI.java:1351)
at Ice.ConnectionI.sendRequest(ConnectionI.java:383)
at IceInternal.ConnectionRequestHandler.sendRequest(ConnectionRequestHandler.java:36)
at IceInternal.Outgoing.invoke(Outgoing.java:66)
at omero.api._IMetadataDelM.loadAnnotations(_IMetadataDelM.java:260)
at omero.api.IMetadataPrxHelper.loadAnnotations(IMetadataPrxHelper.java:320)
at omero.api.IMetadataPrxHelper.loadAnnotations(IMetadataPrxHelper.java:292)
at org.openmicroscopy.shoola.env.data.OMEROGateway.loadAnnotations(OMEROGateway.java:2740)
at org.openmicroscopy.shoola.env.data.OmeroMetadataServiceImpl.loadStructuredAnnotations(OmeroMetadataServiceImpl.java:1046)
at org.openmicroscopy.shoola.env.data.OmeroMetadataServiceImpl.loadStructuredData(OmeroMetadataServiceImpl.java:731)
at org.openmicroscopy.shoola.env.data.OmeroMetadataServiceImpl.loadStructuredData(OmeroMetadataServiceImpl.java:835)
at org.openmicroscopy.shoola.env.data.views.calls.StructuredAnnotationLoader$6.doCall(StructuredAnnotationLoader.java:211)
at org.openmicroscopy.shoola.env.data.views.BatchCall.doStep(BatchCall.java:144)
at org.openmicroscopy.shoola.util.concur.tasks.CompositeTask.doStep(CompositeTask.java:226)
at org.openmicroscopy.shoola.env.data.views.CompositeBatchCall.doStep(CompositeBatchCall.java:126)
at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.exec(ExecCommand.java:165)
at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.run(ExecCommand.java:276)
at org.openmicroscopy.shoola.util.concur.tasks.AsyncProcessor$Runner.run(AsyncProcessor.java:91)
at java.lang.Thread.run(Thread.java:680)
Caused by: java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184)
at sun.nio.ch.SinkChannelImpl.write(SinkChannelImpl.java:153)
at IceInternal.Selector.setInterrupt(Selector.java:343)
... 22 more
Exception in thread "Thread-193"
comment:35 Changed 11 years ago by jmoore
Note, in a stack dump the above is hanging on
"Thread-184" prio=6 tid=10eb5d800 nid=0x129662000 waiting for monitor entry [129661000] java.lang.Thread.State: BLOCKED (on object monitor) at org.openmicroscopy.shoola.env.data.OMEROGateway.isSessionAlive(OMEROGateway.java:432) - waiting to lock <7c03b4a18> (a org.openmicroscopy.shoola.env.data.OMEROGateway) at org.openmicroscopy.shoola.env.data.OMEROGateway.getEnumerations(OMEROGateway.java:5957) at org.openmicroscopy.shoola.env.data.OmeroMetadataServiceImpl.getEnumeration(OmeroMetadataServiceImpl.java:1749) at org.openmicroscopy.shoola.env.data.views.calls.EnumerationLoader$1.doCall(EnumerationLoader.java:91) at org.openmicroscopy.shoola.env.data.views.BatchCall.doStep(BatchCall.java:144) at org.openmicroscopy.shoola.util.concur.tasks.CompositeTask.doStep(CompositeTask.java:226) at org.openmicroscopy.shoola.env.data.views.CompositeBatchCall.doStep(CompositeBatchCall.java:126) at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.exec(ExecCommand.java:165) at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.run(ExecCommand.java:276) at org.openmicroscopy.shoola.util.concur.tasks.AsyncProcessor$Runner.run(AsyncProcessor.java:91) at java.lang.Thread.run(Thread.java:680)
and
"Thread-186" prio=6 tid=10eb09000 nid=0x129868000 in Object.wait() [129867000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:485) at IceInternal.Outgoing.invoke(Outgoing.java:109) - locked <7c34a5a10> (a IceInternal.Outgoing) at omero.api._RenderingEngineDelM.requiresPixelsPyramid(_RenderingEngineDelM.java:156) at omero.api.RenderingEnginePrxHelper.requiresPixelsPyramid(RenderingEnginePrxHelper.java:252) at omero.api.RenderingEnginePrxHelper.requiresPixelsPyramid(RenderingEnginePrxHelper.java:224) at org.openmicroscopy.shoola.env.rnd.RenderingControlProxy.isBigImage(RenderingControlProxy.java:1994) at org.openmicroscopy.shoola.env.rnd.RenderingControlProxy.invalidateCache(RenderingControlProxy.java:267) at org.openmicroscopy.shoola.env.rnd.RenderingControlProxy.setQuantizationMap(RenderingControlProxy.java:1081) at org.openmicroscopy.shoola.env.rnd.RenderingControlProxy.tmpSolutionForNoiseReduction(RenderingControlProxy.java:465) at org.openmicroscopy.shoola.env.rnd.RenderingControlProxy.<init>(RenderingControlProxy.java:827) at org.openmicroscopy.shoola.env.rnd.PixelsServicesFactory.makeNew(PixelsServicesFactory.java:656) at org.openmicroscopy.shoola.env.rnd.PixelsServicesFactory.createRenderingControl(PixelsServicesFactory.java:252) at org.openmicroscopy.shoola.env.data.OmeroImageServiceImpl.loadRenderingControl(OmeroImageServiceImpl.java:588) at org.openmicroscopy.shoola.env.data.views.calls.RenderingControlLoader$1.doCall(RenderingControlLoader.java:97) at org.openmicroscopy.shoola.env.data.views.BatchCall.doStep(BatchCall.java:144) at org.openmicroscopy.shoola.util.concur.tasks.CompositeTask.doStep(CompositeTask.java:226) at org.openmicroscopy.shoola.env.data.views.CompositeBatchCall.doStep(CompositeBatchCall.java:126) at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.exec(ExecCommand.java:165) at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.run(ExecCommand.java:276) at org.openmicroscopy.shoola.util.concur.tasks.AsyncProcessor$Runner.run(AsyncProcessor.java:91) at java.lang.Thread.run(Thread.java:680)
Etc.
comment:36 Changed 11 years ago by jburel
If this is the case then we are blocked by the
omero.client.keepAlive(null)
comment:37 Changed 11 years ago by jburel
Error constantly returned when clicking quickly on thumbnails
Caused by: Ice.SocketException
error = 0
at IceInternal.Selector.setInterrupt(Selector.java:347)
at IceInternal.Selector.remove(Selector.java:107)
at IceInternal.ThreadPool.finish(ThreadPool.java:183)
at Ice.ConnectionI.setState(ConnectionI.java:1454)
at Ice.ConnectionI.setState(ConnectionI.java:1351)
at Ice.ConnectionI.sendRequest(ConnectionI.java:383)
at IceInternal.ConnectionRequestHandler.sendRequest(ConnectionRequestHandler.java:36)
at IceInternal.Outgoing.invoke(Outgoing.java:66)
at omero.api._IMetadataDelM.loadAnnotations(_IMetadataDelM.java:260)
at omero.api.IMetadataPrxHelper.loadAnnotations(IMetadataPrxHelper.java:320)
at omero.api.IMetadataPrxHelper.loadAnnotations(IMetadataPrxHelper.java:292)
at org.openmicroscopy.shoola.env.data.OMEROGateway.loadAnnotations(OMEROGateway.java:2742)
... 11 more
Caused by: java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184)
at sun.nio.ch.SinkChannelImpl.write(SinkChannelImpl.java:153)
at IceInternal.Selector.setInterrupt(Selector.java:343)
... 22 more
comment:38 Changed 11 years ago by jburel
I suspect the problem existed with previous version.
comment:39 Changed 11 years ago by jburel
Found on the Ice forum http://www.zeroc.com/forums/help-center/5057-ice-java-exception-ice-thread-pool-client.html
As a general rule, Ice for Java does not work well in the presence of thread interrupts.
Changed 11 years ago by jburel
comment:40 Changed 11 years ago by jburel
This time I managed to get an error before the Socket exception
IceInternal.OutgoingConnectionFactory.findConnectionByEndpoint(line 457) a stackOverflowError is sent
comment:41 Changed 11 years ago by jburel
This time I managed to get an error before the Socket exception
IceInternal.OutgoingConnectionFactory.findConnectionByEndpoint(line 457) a stackOverflowError is sent
comment:42 Changed 11 years ago by jburel
see also #3259
comment:43 Changed 11 years ago by jmoore
I just tried reproducing with 4.4.5 GA (build 20) and could not reproduce. From the same dataset as above (i.e. against today's howe server), I opened 2-3 dozen images. They opened slowly, but the client remained stable. Trying again with yesterday's howe client, after clicking on 4 thumbnails I got:
java.lang.Exception: org.openmicroscopy.shoola.env.data.DSAccessException: Cannot access data.
Cannot find annotations for class pojos.ImageData.
at org.openmicroscopy.shoola.env.data.OMEROGateway.handleException(OMEROGateway.java:890)
at org.openmicroscopy.shoola.env.data.OMEROGateway.loadAnnotations(OMEROGateway.java:2744)
...
at org.openmicroscopy.shoola.util.concur.tasks.AsyncProcessor$Runner.run(AsyncProcessor.java:91)
at java.lang.Thread.run(Thread.java:680)
Caused by: Ice.SocketException
error = 0
at IceInternal.Selector.setInterrupt(Selector.java:347)
...
at omero.api.IMetadataPrxHelper.loadAnnotations(IMetadataPrxHelper.java:292)
at org.openmicroscopy.shoola.env.data.OMEROGateway.loadAnnotations(OMEROGateway.java:2740)
... 11 more
Caused by: java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184)
at sun.nio.ch.SinkChannelImpl.write(SinkChannelImpl.java:153)
at IceInternal.Selector.setInterrupt(Selector.java:343)
... 22 more
at org.openmicroscopy.shoola.env.ui.UserNotifierImpl.showErrorDialog(UserNotifierImpl.java:189)
...
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:188)
at java.awt.EventDispatchThread.run(EventDispatchThread.java:122)
NB: CPU usage jumped to 300% while the error dialog was still open.
comment:44 Changed 11 years ago by jmoore
On today's build (OMERO.clients-4.4.5-603-c654df5-dirty-ice33-b142.mac.zip), I:
- Logged in as user-6
- Added the read-annotate-1 group
- Opened it, the "No archive" project and the "testimages_256x256" dataset and waited until insight was doing nothing. (via jstack)
- Then double clicked on the red images (in the middle panel) 1, 2, 3, 4 and while clicking on 5 the network down message came.
- The error messages are all as previously, CPU usage, etc. I don't see any methods hanging on the server.
Jean-Marie, likely you're right about the Ice behavior on Thread.interrupt. Are you calling that anywhere? Or do you suspect something else is calling it?
comment:45 Changed 11 years ago by jmoore
#10282 marked as duplicate.
comment:46 Changed 11 years ago by sylittlewood
Just experienced 100% cpu on windows testing 4.4.6. Stack trace is attached: https://trac.openmicroscopy.org.uk/ome/attachment/ticket/10116/cpu_100_percent_windows_jstack_trace_4_4_6.txt
Changed 11 years ago by bpindelski
comment:47 Changed 11 years ago by bpindelski
Also experienced the 100% CPU issue on Windows: https://trac.openmicroscopy.org.uk/ome/attachment/ticket/10116/insight_windows_stacktrace.txt
comment:48 Changed 11 years ago by jmoore
Took a screenshot of the call to Thread.interrupt in insight, which seems to occur if the id of an activity is null.
comment:49 Changed 11 years ago by jburel
as discussed briefly this morning, that could be the source of the problem
various loaders are cancelled, in different part of the client. So will need to review that carefully
comment:50 Changed 11 years ago by jmoore
See https://github.com/joshmoore/openmicroscopy/tree/10116-thread.interrupt
I made 3 sets of changes (cumulative):
- Refactored HashMap out into a synchronized object (failure still occurred)
- Replaced Integer loaderID with AtomicInteger loaderID (failure still occurred)
- Added sychronized to MetadataViewerComponent.setRootObject (No failure).
I don't know the implications of this last step.
comment:51 Changed 11 years ago by jburel
The synchronized will work but that will lead to other type of problems e.g. out of synch with slow connection. between selected object and annotated object.
I am reluctant to follow that path.
I have commented out for now the executor.interrupt()
b/c the problem could occur elsewhere not only for the metadata.
comment:52 Changed 11 years ago by jmoore
I agree that the synchronized is a good way forward, but the way that the loaderID is incremented non-thread-safe, then passed to the instance, then passed to a map, definitely leads a *lot* of room for race conditions. This is almost certainly why fast clicking causes it. If we can avoid calling Thread.interrupt then we'll fix the CPU@300% issue, but there will still be a race condition that needs to get fixed. Should I file a second ticket?
comment:53 Changed 11 years ago by jburel
I agree, for now since I do not want to delay the release, I have commented it out the interrupt only used to call for loading data from server.
That should fix the issue
comment:54 Changed 11 years ago by jburel
Until we move away from ice. probably no need to cancel => loader id and map can be removed.
comment:55 Changed 11 years ago by jburel
possibility:
- we do not recycle the component and only have reference to the required loaders.
comment:56 Changed 11 years ago by jmoore
My 1-2-3-4-5 test from yesterday does *not* reproduce today. If no other similar situations show up, I'd assume this is the best we can do for 4.4.6.
However, if I click even more wildly, opening projects/dataset, then clicking on many images, and then changing from Preview to Metadata and back, etc., I do start to see errors. I receive an application error of :
java.lang.Exception: org.openmicroscopy.shoola.env.data.DSAccessException: Cannot access data. Cannot load the plane info for pixels: 6070 at org.openmicroscopy.shoola.env.data.OMEROGateway.handleException(OMEROGateway.java:891) at org.openmicroscopy.shoola.env.data.OMEROGateway.loadPlaneInfo(OMEROGateway.java:6077) at org.openmicroscopy.shoola.env.data.OmeroImageServiceImpl.loadPlaneInfo(OmeroImageServiceImpl.java:1005) at org.openmicroscopy.shoola.env.data.views.calls.PlaneInfoLoader$1.doCall(PlaneInfoLoader.java:78) at org.openmicroscopy.shoola.env.data.views.BatchCall.doStep(BatchCall.java:144) at org.openmicroscopy.shoola.util.concur.tasks.CompositeTask.doStep(CompositeTask.java:226) at org.openmicroscopy.shoola.env.data.views.CompositeBatchCall.doStep(CompositeBatchCall.java:126) at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.exec(ExecCommand.java:165) at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.run(ExecCommand.java:276) at org.openmicroscopy.shoola.util.concur.tasks.AsyncProcessor$Runner.run(AsyncProcessor.java:91) at java.lang.Thread.run(Thread.java:680) Caused by: Ice.CommunicatorDestroyedException at IceInternal.Instance.objectAdapterFactory(Instance.java:122) at Ice.ObjectPrxHelperBase.createDelegate(ObjectPrxHelperBase.java:997) at Ice.ObjectPrxHelperBase.__getDelegate(ObjectPrxHelperBase.java:946) at omero.api.IQueryPrxHelper.findAllByQuery(IQueryPrxHelper.java:318) at omero.api.IQueryPrxHelper.findAllByQuery(IQueryPrxHelper.java:292) at org.openmicroscopy.shoola.env.data.OMEROGateway.loadPlaneInfo(OMEROGateway.java:6075) ... 9 more at org.openmicroscopy.shoola.env.ui.UserNotifierImpl.showErrorDialog(UserNotifierImpl.java:189) ... at org.openmicroscopy.shoola.env.ui.UserNotifierImpl.notifyError(UserNotifierImpl.java:257) at org.openmicroscopy.shoola.agents.metadata.EditorLoader.handleException(EditorLoader.java:159) at org.openmicroscopy.shoola.env.data.events.DSCallAdapter.eventFired(DSCallAdapter.java:84) at org.openmicroscopy.shoola.env.data.views.BatchCallMonitor$1.run(BatchCallMonitor.java:124) at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:209) ... at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:188) at java.awt.EventDispatchThread.run(EventDispatchThread.java:122)
and the log has similar messages (to be attached). I have 2 pop ups with "The connection has been destroyed. The application will now exit." and one with "A failure occurred while attempting to reconnect. The application will now exit". I'd assume that removing cancel can at best be considered a workaround.
comment:57 Changed 11 years ago by jburel <j.burel@…>
(In [239dc7fe9b000e6ceb0457b30a87569ec0619cfc/ome.git] on branch develop) Do not interrupt thread (see #10116)
As a general rule, Ice for Java does not work well in the
presence of thread interrupts.
see
http://www.zeroc.com/forums/help-center/
5057-ice-java-exception-ice-thread-pool-client.html
comment:58 Changed 11 years ago by jamoore
- Milestone changed from OMERO-4.4.7 to OMERO-4.4.8
comment:59 Changed 11 years ago by jburel
- Milestone changed from OMERO-4.4.9 to OMERO-4.4.x
- Version set to 4.4.8
comment:60 Changed 11 years ago by jamoore
Note: this may actually now be a non-issue with the rebased connector work (https://github.com/openmicroscopy/openmicroscopy/pull/1321)
comment:61 Changed 10 years ago by jamoore
- Milestone changed from 5.x to 5.0.2
Moving all remaining blockers to 5.0.2 for re-evaluation.
comment:62 Changed 10 years ago by jamoore
- Resolution set to fixed
- Status changed from new to closed
There has been no further significant hangs based on timeouts. Closing this in favor of smaller, more specific bug tickets.
Note: See
TracTickets for help on using
tickets.
You may also have a look at Agilo extensions to the ticket.
anything in the log?