Urgent Help - exception trashing live service

I am getting the following exception periodically, seems to occur after a publication (but not every publication). It causes a page not found error to be generated for all pages on the site, including the home page. I need urgent advice on this issue.

11 Feb 2010 05:28:46.367 [ERROR] [http-80-5] [org.infoglue.deliver.applications.actions.ViewPageAction] - An error occurred so we should not complete the transaction:null
Original URL: http://www.XXXXXXXXXXXX.com/styles/css/
Referer: null
UserAgent: Mozilla/4.0 (compatible

User IP: 192.254.1.7
ClientAbortException: java.io.IOException
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:35

at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:354)
at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:381)
at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:370)
at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:89)
at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:83)
at org.infoglue.deliver.invokers.PageInvoker.deliverPage(PageInvoker.java:426)
at org.infoglue.deliver.applications.actions.ViewPageAction.doExecute(ViewPageAction.java:352)
at org.infoglue.cms.applications.common.actions.WebworkAbstractAction.execute(WebworkAbstractAction.java:113)
at webwork.dispatcher.GenericDispatcher.executeAction(GenericDispatcher.java:131)
at org.infoglue.deliver.portal.dispatcher.DeliveryServletDispatcher.service(DeliveryServletDispatcher.java:92)
at org.infoglue.deliver.portal.dispatcher.PortalServletDispatcher.service(PortalServletDispatcher.java:176)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:646)
at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:436)
at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:374)
at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:302)
at org.infoglue.deliver.applications.filters.ViewPageFilter.doFilter(ViewPageFilter.java:292)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.infoglue.deliver.applications.filters.PortalParameterFilter.doFilter(PortalParameterFilter.java:90)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.infoglue.cms.util.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:132)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.infoglue.deliver.applications.filters.CacheEvictionFilter.doFilter(CacheEvictionFilter.java:92)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:465)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:29

at org.apache.coyote.http11.Http11AprProcessor.process(Http11AprProcessor.java:859)
at org.apache.coyote.http11.Http11AprProtocol$Http11ConnectionHandler.process(Http11AprProtocol.java:579)
at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1555)
at java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException
at org.apache.coyote.http11.InternalAprOutputBuffer.flushBuffer(InternalAprOutputBuffer.java:696)
at org.apache.coyote.http11.InternalAprOutputBuffer$SocketOutputBuffer.doWrite(InternalAprOutputBuffer.java:726)
at org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:126)
at org.apache.coyote.http11.InternalAprOutputBuffer.doWrite(InternalAprOutputBuffer.java:532)
at org.apache.coyote.Response.doWrite(Response.java:560)
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:353)
... 41 more
11 Feb 2010 05:28:46.367 [ERROR] [http-80-5] [org.infoglue.cms.applications.common.actions.WebworkAbstractAction] - Throwable java.lang.IllegalStateException: Cannot forward after response has been committed
java.lang.Exception: java.lang.IllegalStateException: Cannot forward after response has been committed
at org.infoglue.cms.applications.common.actions.WebworkAbstractAction.execute(WebworkAbstractAction.java:155)
at webwork.dispatcher.GenericDispatcher.executeAction(GenericDispatcher.java:131)
at org.infoglue.deliver.portal.dispatcher.DeliveryServletDispatcher.service(DeliveryServletDispatcher.java:92)
at org.infoglue.deliver.portal.dispatcher.PortalServletDispatcher.service(PortalServletDispatcher.java:176)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:646)
at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:436)
at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:374)
at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:302)
at org.infoglue.deliver.applications.filters.ViewPageFilter.doFilter(ViewPageFilter.java:292)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.infoglue.deliver.applications.filters.PortalParameterFilter.doFilter(PortalParameterFilter.java:90)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.infoglue.cms.util.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:132)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.infoglue.deliver.applications.filters.CacheEvictionFilter.doFilter(CacheEvictionFilter.java:92)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:465)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:29

at org.apache.coyote.http11.Http11AprProcessor.process(Http11AprProcessor.java:859)
at org.apache.coyote.http11.Http11AprProtocol$Http11ConnectionHandler.process(Http11AprProtocol.java:579)
at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1555)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.IllegalStateException: Cannot forward after response has been committed
at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:312)
at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:302)
at org.infoglue.deliver.applications.actions.ViewPageAction.doExecute(ViewPageAction.java:416)
at org.infoglue.cms.applications.common.actions.WebworkAbstractAction.execute(WebworkAbstractAction.java:113)
... 33 more

Something seems to do a forward in your page ... are you doing a special treatment on your response into your components ?
It's like some HTML was writing into the response stream and then you're trying to do a forward (or maybe changing response's headers, ...) which cause "Cannot forward after response has been committed".
I can't help you more without looking some of your code =/

Another guess is that you might send some wrong header that makes the Client Browser to terminate the stream (ClientAbortException)
--> which makes IG to redirect ...

I am not sure, but you should try with different browsers and try to review/manipulate your headers.

DJD

The code mentioned in the stacktrace is my CSS file. In that file the only 'complicated' piece of coding is loading images as follows:

I'm now in the process of changing my code to JSP as follows:

This may be a more stable implementation, any thoughts.

Mark.

We have narrowed this issue down to garbage collection/memory and a misconfiguration in Tomcat.

The person who installed Tomcat had it pointing to a client JVM instead of a server JVM, and they also hadn't configured JAVA_HOME.

The garbage collector for tomcat was not using the default garbage collector so we changed it back.

We increased the -Xmx and -Xms memory from 1024M to 4096M, the server has 8G of RAM.

At the minute this seems to have settled the issue down. I'll keep you posted if anything changes.

I haven't posted on this issue in a long time, but the problem is still present.

When I first install infoglue into the live environment I installed infoglueCMS, preview, working and live all under one Tomcat instance. This mirrored the development environment, where there are no problems (at least not like this).

In the early hours of the this morning I split the CMS, preview and working onto a completely different server and configured the CMS so that it could publish to live on the original server. I believed that this would clear up the issue.

It might be worth noting now that I have a custom error page configured.

This morning when I split infoglue up I did extensive testing, i.e. making sure I could publish, etc. This went smoothly without error.

My client has today attempted a published, nothing major just a few templates. As soon as this was done the live server stopped serving infoglue pages and only served the custom error page (the custom error page has reference to image assets and css within the site, these are not being served, simply the plain HTML of the custom error page). The only way to resolve this situation is to restart Tomcat. When the site when down today I did not get a chance to see if the CMS or Working were serving pages and whether it was just the live server that was down.

The log file entry is as follows:
javax.servlet.ServletException: File "/custom_error.jsp" not found
at org.apache.jasper.servlet.JspServlet.handleMissingResource(JspServlet.java:335)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:301)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:646)
at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:551)
at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:48

at org.infoglue.deliver.applications.actions.ErrorPageAction.doExecute(ErrorPageAction.java:166)
at org.infoglue.cms.applications.common.actions.WebworkAbstractAction.execute(WebworkAbstractAction.java:113)
at webwork.dispatcher.GenericDispatcher.executeAction(GenericDispatcher.java:131)
at webwork.dispatcher.ServletDispatcher.service(ServletDispatcher.java:174)
at org.infoglue.deliver.portal.dispatcher.DeliveryServletDispatcher.service(DeliveryServletDispatcher.java:66)
at org.infoglue.deliver.portal.dispatcher.PortalServletDispatcher.service(PortalServletDispatcher.java:176)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:646)
at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:43

at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:374)
at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:302)
at org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.java:415)
at org.apache.catalina.core.StandardHostValve.status(StandardHostValve.java:342)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:29

at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:852)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:58

at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
at java.lang.Thread.run(Unknown Source)

Any help in this area would be much appreciated.

Thanks,

Mark.

Just to confirm, I have placed my custom_error.jsp file in the tomcat/webapp/ROOT folder and not within Infoglue. Is this correct?

Thanks,

Mark.

Hi,

Your custom_error.jsp should be placed in your Infoglue deliver directory.

e.g in tomcat/webapp/infoglueDeliverLive/

Best regards

Karl,

I have reconfigured Tomcat/InfoglueCMS so that infoglueDeliverLive is served through the ROOT web app, running on port 80. This way user can simply go to http://www.mydomain.com/ and see the live web site.

Thanks,

Mark.