• Views
  • Iteration Report
  • My Iteration Report
  •  
OMERO.server
  • Login
  • Help/Guide
  • About Trac
  • Preferences
  • Wiki
  • Timeline
  • Roadmap
  • Browse Source
  • View Tickets
  • Search

Context Navigation

  • ← Previous Ticket
  • Next Ticket →

Ticket #834 (assigned defect)

Opened 12 months ago

Last modified 11 months ago

JBoss PermGen Errors

Reported by: callan Owned by: callan
Priority: critical Milestone:
Component: ApplicationServer Version:
Keywords: Cc: atarkowska

Description

JBoss is throwing java.lang.OutOfMemoryError: PermGen space exceptions: 

15:12:51,890 ERROR [[jsp]] Servlet.service() for servlet jsp threw exception
java.lang.NoClassDefFoundError: org/eclipse/jdt/internal/compiler/codegen/CodeStream
        at org.eclipse.jdt.internal.compiler.ast.TypeDeclaration.generateCode(TypeDeclaration.java:508)
        at org.eclipse.jdt.internal.compiler.ast.TypeDeclaration.generateCode(TypeDeclaration.java:584)
        at org.eclipse.jdt.internal.compiler.ast.CompilationUnitDeclaration.generateCode(CompilationUnitDeclaration.java:214)
        at org.eclipse.jdt.internal.compiler.Compiler.process(Compiler.java:602)
        at org.eclipse.jdt.internal.compiler.Compiler.compile(Compiler.java:411)
        at org.apache.jasper.compiler.JDTCompiler.generateClass(JDTCompiler.java:405)
        at org.apache.jasper.compiler.Compiler.compile(Compiler.java:308)
        at org.apache.jasper.compiler.Compiler.compile(Compiler.java:286)
        at org.apache.jasper.compiler.Compiler.compile(Compiler.java:273)
        at org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:566)
        at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:311)
        at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:320)
        at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:266)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
        at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:179)
        at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
        at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:241)
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:580)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
        at java.lang.Thread.run(Thread.java:619)
15:12:55,949 ERROR [[jsp]] Servlet.service() for servlet jsp threw exception
java.lang.NoClassDefFoundError: org/eclipse/jdt/internal/compiler/codegen/CodeStream
        at org.eclipse.jdt.internal.compiler.ast.TypeDeclaration.generateCode(TypeDeclaration.java:508)
        at org.eclipse.jdt.internal.compiler.ast.TypeDeclaration.generateCode(TypeDeclaration.java:584)
        at org.eclipse.jdt.internal.compiler.ast.CompilationUnitDeclaration.generateCode(CompilationUnitDeclaration.java:214)
        at org.eclipse.jdt.internal.compiler.Compiler.process(Compiler.java:602)
        at org.eclipse.jdt.internal.compiler.Compiler.compile(Compiler.java:411)
        at org.apache.jasper.compiler.JDTCompiler.generateClass(JDTCompiler.java:405)
        at org.apache.jasper.compiler.Compiler.compile(Compiler.java:308)
        at org.apache.jasper.compiler.Compiler.compile(Compiler.java:286)
        at org.apache.jasper.compiler.Compiler.compile(Compiler.java:273)
        at org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:566)
        at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:311)
        at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:320)
        at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:266)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
        at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:179)
        at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
        at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:241)
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:580)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
        at java.lang.Thread.run(Thread.java:619)
15:51:43,148 ERROR [STDERR] Exception in thread "RMI TCP Connection(idle)"
15:51:43,151 ERROR [STDERR] java.lang.OutOfMemoryError: PermGen space
15:51:57,866 WARN  [RunnableTaskWrapper] Unhandled throwable for runnable: org.jnp.server.Main$BootstrapRequestHandler@723e94a6
java.lang.OutOfMemoryError: PermGen space
16:21:54,278 WARN  [RunnableTaskWrapper] Unhandled throwable for runnable: org.jnp.server.Main$BootstrapRequestHandler@53b69ca8
java.lang.OutOfMemoryError: PermGen space
16:22:24,906 WARN  [RunnableTaskWrapper] Unhandled throwable for runnable: org.jnp.server.Main$BootstrapRequestHandler@6264c005
java.lang.OutOfMemoryError: PermGen space
16:22:48,294 ERROR [STDERR] Exception in thread "WorkerThread#0[10.34.2.145:57702]"
16:22:48,294 ERROR [STDERR] java.lang.OutOfMemoryError: PermGen space

Change History

Changed 12 months ago by callan

  • cc josh removed

Changed 12 months ago by callan

Context from the JBoss server.log: 

2007-11-12 16:18:56,248 DEBUG [com.arjuna.ats.arjuna.logging.arjLogger] Periodic recovery - first pass <Mon, 12 Nov 2007 16:18:56>
2007-11-12 16:18:56,248 DEBUG [com.arjuna.ats.arjuna.logging.arjLogger] StatusModule: first pass 
2007-11-12 16:18:56,248 DEBUG [com.arjuna.ats.txoj.logging.txojLoggerI18N] [com.arjuna.ats.internal.txoj.recovery.TORecoveryModule_3] - TORecoveryModule - first pass
2007-11-12 16:18:56,248 DEBUG [com.arjuna.ats.jta.logging.loggerI18N] [com.arjuna.ats.internal.jta.recovery.info.firstpass] Local XARecoveryModule - first pass
2007-11-12 16:19:06,253 DEBUG [com.arjuna.ats.arjuna.logging.arjLogger] Periodic recovery - second pass <Mon, 12 Nov 2007 16:19:06>
2007-11-12 16:19:06,253 DEBUG [com.arjuna.ats.arjuna.logging.arjLogger] AtomicActionRecoveryModule: Second pass 
2007-11-12 16:19:06,253 DEBUG [com.arjuna.ats.txoj.logging.txojLoggerI18N] [com.arjuna.ats.internal.txoj.recovery.TORecoveryModule_6] - TORecoveryModule - second pass
2007-11-12 16:19:06,253 DEBUG [com.arjuna.ats.jta.logging.loggerI18N] [com.arjuna.ats.internal.jta.recovery.info.secondpass] Local XARecoveryModule - second pass
2007-11-12 16:21:06,274 DEBUG [com.arjuna.ats.arjuna.logging.arjLogger] Periodic recovery - first pass <Mon, 12 Nov 2007 16:21:06>
2007-11-12 16:21:06,274 DEBUG [com.arjuna.ats.arjuna.logging.arjLogger] StatusModule: first pass 
2007-11-12 16:21:06,275 DEBUG [com.arjuna.ats.txoj.logging.txojLoggerI18N] [com.arjuna.ats.internal.txoj.recovery.TORecoveryModule_3] - TORecoveryModule - first pass
2007-11-12 16:21:06,275 DEBUG [com.arjuna.ats.jta.logging.loggerI18N] [com.arjuna.ats.internal.jta.recovery.info.firstpass] Local XARecoveryModule - first pass
2007-11-12 16:21:16,279 DEBUG [com.arjuna.ats.arjuna.logging.arjLogger] Periodic recovery - second pass <Mon, 12 Nov 2007 16:21:16>
2007-11-12 16:21:16,280 DEBUG [com.arjuna.ats.arjuna.logging.arjLogger] AtomicActionRecoveryModule: Second pass 
2007-11-12 16:21:16,280 DEBUG [com.arjuna.ats.txoj.logging.txojLoggerI18N] [com.arjuna.ats.internal.txoj.recovery.TORecoveryModule_6] - TORecoveryModule - second pass
2007-11-12 16:21:16,280 DEBUG [com.arjuna.ats.jta.logging.loggerI18N] [com.arjuna.ats.internal.jta.recovery.info.secondpass] Local XARecoveryModule - second pass
2007-11-12 16:21:54,278 WARN  [org.jboss.util.threadpool.RunnableTaskWrapper] Unhandled throwable for runnable: org.jnp.server.Main$BootstrapRequestHandler@53b69ca8
java.lang.OutOfMemoryError: PermGen space
2007-11-12 16:22:24,906 WARN  [org.jboss.util.threadpool.RunnableTaskWrapper] Unhandled throwable for runnable: org.jnp.server.Main$BootstrapRequestHandler@6264c005
java.lang.OutOfMemoryError: PermGen space
2007-11-12 16:22:45,285 DEBUG [org.jboss.remoting.transport.socket.ClientSocketWrapper] reset timeout: 60000
2007-11-12 16:22:48,294 ERROR [STDERR] Exception in thread "WorkerThread#0[10.34.2.145:57702]" 
2007-11-12 16:22:48,294 ERROR [STDERR] java.lang.OutOfMemoryError: PermGen space
2007-11-12 16:23:16,299 DEBUG [com.arjuna.ats.arjuna.logging.arjLogger] Periodic recovery - first pass <Mon, 12 Nov 2007 16:23:16>
2007-11-12 16:23:16,299 DEBUG [com.arjuna.ats.arjuna.logging.arjLogger] StatusModule: first pass 
2007-11-12 16:23:16,299 DEBUG [com.arjuna.ats.txoj.logging.txojLoggerI18N] [com.arjuna.ats.internal.txoj.recovery.TORecoveryModule_3] - TORecoveryModule - first pass
2007-11-12 16:23:16,299 DEBUG [com.arjuna.ats.jta.logging.loggerI18N] [com.arjuna.ats.internal.jta.recovery.info.firstpass] Local XARecoveryModule - first pass
2007-11-12 16:23:26,304 DEBUG [com.arjuna.ats.arjuna.logging.arjLogger] Periodic recovery - second pass <Mon, 12 Nov 2007 16:23:26>
2007-11-12 16:23:26,304 DEBUG [com.arjuna.ats.arjuna.logging.arjLogger] AtomicActionRecoveryModule: Second pass 
2007-11-12 16:23:26,304 DEBUG [com.arjuna.ats.txoj.logging.txojLoggerI18N] [com.arjuna.ats.internal.txoj.recovery.TORecoveryModule_6] - TORecoveryModule - second pass
2007-11-12 16:23:26,304 DEBUG [com.arjuna.ats.jta.logging.loggerI18N] [com.arjuna.ats.internal.jta.recovery.info.secondpass] Local XARecoveryModule - second pass
2007-11-12 16:25:00,446 DEBUG [org.jboss.resource.connectionmanager.IdleRemover] run: IdleRemover notifying pools, interval: 450000
2007-11-12 16:25:26,324 DEBUG [com.arjuna.ats.arjuna.logging.arjLogger] Periodic recovery - first pass <Mon, 12 Nov 2007 16:25:26>
2007-11-12 16:25:26,324 DEBUG [com.arjuna.ats.arjuna.logging.arjLogger] StatusModule: first pass 
2007-11-12 16:25:26,325 DEBUG [com.arjuna.ats.txoj.logging.txojLoggerI18N] [com.arjuna.ats.internal.txoj.recovery.TORecoveryModule_3] - TORecoveryModule - first pass
2007-11-12 16:25:26,325 DEBUG [com.arjuna.ats.jta.logging.loggerI18N] [com.arjuna.ats.internal.jta.recovery.info.firstpass] Local XARecoveryModule - first pass
2007-11-12 16:25:36,329 DEBUG [com.arjuna.ats.arjuna.logging.arjLogger] Periodic recovery - second pass <Mon, 12 Nov 2007 16:25:36>

Changed 12 months ago by jmoore

For anyone experiencing this, it would probably be useful to run JBoss with the following: 

JAVA_OPTS=-XX:-HeapDumpOnOutOfMemoryError $JBOSS_HOME/bin/run.sh

Changed 12 months ago by callan

  • cc atarkowska added
  • owner changed from jmoore to callan
  • status changed from new to assigned

Looking at the JVM's permanent generation usage (explanation on what the columns represent is available here (java.sun.com)): 

$ jstat -gcutil 13242
 S0     S1     E      O      P     YGC     YGCT    FGC    FGCT     GCT   
 0.00  97.83  47.87  82.41  99.12    327    4.487    33   25.707   30.194

The important column for us is P which tells us the permanent generation space utilization is at 99.12%. Not exactly ideal and just begging to be pushed beyond the maximum allowable space.

Checking the actual numbers on permanent generation maximum allowable space we achieve by examining further output from jstat: 

$ bin/jstat -gcpermcapacity 13242
 PGCMN      PGCMX       PGC         PC      YGC   FGC    FGCT     GCT   
 21248.0    86016.0    82240.0    82240.0   328    33   25.707   30.207

So, it would appear that our solution lies in increasing the size of the permanent generation using the -XX:MaxPermSize JVM arguments as documented here (java.sun.com).

The default permanent generation size for Java 5.0 and newer VMs is:

  • 64MB on 32-bit systems
  • 64MB * 30% (83.20MB) on 64-bit systems

Changed 12 months ago by callan

Now testing with the following: 

JAVA_OPTS="-XX:MaxPermSize=128M" $JBOSS_HOME/bin/run.sh

Changed 11 months ago by callan

  • milestone set to 3.0-Beta2.3
Note: See TracTickets for help on using tickets.

Download in other formats:

  • Comma-delimited Text
  • Tab-delimited Text
  • RSS Feed
Trac Powered

Powered by Trac 0.11
By Edgewall Software.

Visit the Trac open source project at
http://trac.edgewall.org/