StackOverflowException on opening a large dataset

[jmoore]

From Jean-Marie:

That's the error message returned when Michael opens a dataset with more than 15000 images. Have you seen the overflow error before? I tried to increase memory in JAVA_OPTS before running server but still same problem?

java.lang.RuntimeException: java.lang.StackOverflowError
         at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed 
(InvocationContextImpl.java:174)
         at ome.tools.spring.AOPAdapter.invokeJoinpoint 
(AOPAdapter.java:53)
         at  
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed 
(ReflectiveMethodInvocation.java:149)
         at ome.security.basic.EventHandler.invoke(EventHandler.java: 
112)
         at  
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed 
(ReflectiveMethodInvocation.java:171)
         at  
org.springframework.orm.hibernate3.HibernateInterceptor.invoke 
(HibernateInterceptor.java:111)
         at  
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed 
(ReflectiveMethodInvocation.java:171)
         at  
org.springframework.transaction.interceptor.TransactionInterceptor.invok 
e(TransactionInterceptor.java:106)
         at  
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed 
(ReflectiveMethodInvocation.java:171)
         at ome.tools.hibernate.ProxyCleanupFilter$Interceptor.invoke 
(ProxyCleanupFilter.java:169)
         at  
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed 
(ReflectiveMethodInvocation.java:171)
         at ome.services.util.ServiceHandler.invoke 
(ServiceHandler.java:92)
         at  
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed 
(ReflectiveMethodInvocation.java:171)
         at  
org.springframework.aop.interceptor.AbstractTraceInterceptor.invoke 
(AbstractTraceInterceptor.java:113)
         at  
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed 
(ReflectiveMethodInvocation.java:171)
         at ome.security.basic.BasicSecurityWiring.invoke 
(BasicSecurityWiring.java:79)
         at  
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed 
(ReflectiveMethodInvocation.java:171)
         at ome.services.util.OmeroAroundInvoke.call 
(OmeroAroundInvoke.java:161)
         at ome.services.util.OmeroAroundInvoke.loginAndSpringWrap 
(OmeroAroundInvoke.java:134)
         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
         at sun.reflect.NativeMethodAccessorImpl.invoke 
(NativeMethodAccessorImpl.java:39)
         at sun.reflect.DelegatingMethodAccessorImpl.invoke 
(DelegatingMethodAccessorImpl.java:25)
         at java.lang.reflect.Method.invoke(Method.java:585)
         at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed 
(InvocationContextImpl.java:118)
         at  
org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke 
(EJB3InterceptorsInterceptor.java:63)
         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext 
(MethodInvocation.java:101)
         at  
org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke 
(TransactionScopedEntityManagerInterceptor.java:54)
         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext 
(MethodInvocation.java:101)
         at org.jboss.ejb3.AllowedOperationsInterceptor.invoke 
(AllowedOperationsInterceptor.java:47)
         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext 
(MethodInvocation.java:101)
         at org.jboss.ejb3.tx.BMTInterceptor.handleStateless 
(BMTInterceptor.java:71)
         at org.jboss.ejb3.tx.BMTInterceptor.invoke 
(BMTInterceptor.java:131)
         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext 
(MethodInvocation.java:101)
         at org.jboss.aspects.tx.TxPropagationInterceptor.invoke 
(TxPropagationInterceptor.java:76)
         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext 
(MethodInvocation.java:101)
         at  
org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke 
(StatelessInstanceInterceptor.java:62)
         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext 
(MethodInvocation.java:101)
         at  
org.jboss.aspects.security.AuthenticationInterceptor.invoke 
(AuthenticationInterceptor.java:77)
         at  
org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke 
(Ejb3AuthenticationInterceptor.java:106)
         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext 
(MethodInvocation.java:101)
         at org.jboss.ejb3.ENCPropagationInterceptor.invoke 
(ENCPropagationInterceptor.java:46)
         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext 
(MethodInvocation.java:101)
         at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke 
(AsynchronousInterceptor.java:106)
         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext 
(MethodInvocation.java:101)
         at org.jboss.ejb3.stateless.StatelessContainer.dynamicInvoke 
(StatelessContainer.java:278)
         at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:106)
         at  
org.jboss.aspects.remoting.AOPRemotingInvocationHandler.invoke 
(AOPRemotingInvocationHandler.java:82)
         at org.jboss.remoting.ServerInvoker.invoke 
(ServerInvoker.java:734)
         at  
org.jboss.remoting.transport.socket.ServerThread.processInvocation 
(ServerThread.java:560)
         at org.jboss.remoting.transport.socket.ServerThread.dorun 
(ServerThread.java:383)
         at org.jboss.remoting.transport.socket.ServerThread.run 
(ServerThread.java:165)
Caused by: java.lang.StackOverflowError
         at org.hibernate.hql.ast.util.NodeTraverser.visitDepthFirst 
(NodeTraverser.java:40)

[jmoore]

Caused by: java.lang.StackOverflowError
         at org.hibernate.hql.ast.util.NodeTraverser.visitDepthFirst 
(NodeTraverser.java:40)

This bit would imply that it's due to the HQL. Could you give me that?

[jmoore]

Perhaps: HHH-1985?

[jburel]

method call is loadContainerHierarchy with root node Dataset.class, passing the dataset id and requesting the leaves

[jmoore]

From Chris:

This is really a programming error rather than a memory allocation one, the stack size can be set with -Xss (defaults to 256K on most environments I think). Not sure the best way to handle this J-M, we can always increase the stack size but then 20,000 images will fail, or 25,000. The only way I can think of doing this "properly" is to request the images in batches and not all at one time.

Do we add auto-batching on the server-side? Or throw an exception when idSet.size() > 5000 (or similar) ? This goes along with all the throttling tickets floating around:

• #248
• #486
• #619
• #864

[callan]

Just been reading http://gregluck.com/blog/archives/2004/12/limits_to_threa_1.html and playing with thread-test.jar which is from that blog post. It would appear that the Mac OS X JVM really sucks in these cases (256KB default stack size as opposed to Linux's 512KB) and we'd probably never see them on our Linux machines.

Not sure what to say other than: if you've got lots of images in the same dataset don't run your server on Mac OS X. :)

[callan]

JVM thread tester.

[callan]

Been doing tests with:

$ java -cp thread-test.jar com.wotif.jaguar.threading.ThreadManualTest 20000

[jburel]

Need the possibility in PojosOptions? to retrieve group of images e.g. 0-500 then 501- x etc.

[jmoore]

r2563, r2574 and r2575 permit pagination on getImages. Jean-Marie, try that out, and let's open new tickets for any other methods.

Note: I was never able to reproduce the StackOverflowException even with datasets as large as 25,000 images on my Mac 10.4. So it's very fickle. We'll have to take this into account when we start to implement throttling. Not sure how I would catch this particular issue before the StackOverflowException other than to check the size of each collection manually.