NPE creating a new session on WebLogic Server 9.2 - WebLogic Server - Clustering

We're getting the following stack trace from WebLogic Server 9.2 when a page gets hit.
It appears to be a clustering error when creating a new session.
When there is only a single server left running, the pages are served correctly.
Any thoughts appreciated.
2009-01-27 11:13:47,423 [ExecuteThread: '12' for queue: 'weblogic.kernel.Default'] ERROR com.vignette.portal.website.enduser.filters - java.lang.NullPointerException:
java.lang.NullPointerException:
at weblogic.rjvm.ResponseImpl.unmarshalReturn(ResponseImpl.java:211)
at weblogic.rmi.internal.BasicRemoteRef.invoke(BasicRemoteRef.java:224)
at weblogic.cluster.replication.ReplicationManager_922_WLStub.create(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at weblogic.cluster.replication.SecureReplicationInvocationHandler$ReplicationServicesInvocationAction.run(SecureReplicationInvocationHandler.java:184)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)
at weblogic.cluster.replication.SecureReplicationInvocationHandler.invoke(SecureReplicationInvocationHandler.java:154)
at $Proxy46.create(Unknown Source)
at weblogic.cluster.replication.ReplicationManager.trySecondary(ReplicationManager.java:666)
at weblogic.cluster.replication.ReplicationManager.createSecondary(ReplicationManager.java:638)
at weblogic.cluster.replication.ReplicationManager.add(ReplicationManager.java:359)
at weblogic.cluster.replication.ReplicationManager.register(ReplicationManager.java:352)
at weblogic.servlet.internal.session.ReplicatedSessionData.registerOrAdd(ReplicatedSessionData.java:108)
at weblogic.servlet.internal.session.ReplicatedSessionData.<init>(ReplicatedSessionData.java:78)
at weblogic.servlet.internal.session.ReplicatedSessionData.<init>(ReplicatedSessionData.java:64)
at weblogic.servlet.internal.session.ReplicatedSessionContext.getNewSession(ReplicatedSessionContext.java:161)
at weblogic.servlet.internal.ServletRequestImpl$SessionHelper.getNewSession(ServletRequestImpl.java:2526)
at weblogic.servlet.internal.ServletRequestImpl$SessionHelper.getSessionInternal(ServletRequestImpl.java:2111)
at weblogic.servlet.internal.ServletRequestImpl$SessionHelper.getSession(ServletRequestImpl.java:2075)
at weblogic.servlet.internal.ServletRequestImpl.getSession(ServletRequestImpl.java:1207)
at com.vignette.portal.website.internal.DefaultHttpRequestWrapper.getSession(DefaultHttpRequestWrapper.java:65)
at com.vignette.portal.website.internal.DefaultHttpRequestWrapper.getSession(DefaultHttpRequestWrapper.java:41)
at com.epicentric.common.website.InitUtils.initSession(InitUtils.java:73)
at com.vignette.portal.website.enduser.filters.AuthenticationFilter.initSession(AuthenticationFilter.java:47)
at com.vignette.portal.website.enduser.filters.AuthenticationFilter.doFilter(AuthenticationFilter.java:41)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
at com.vignette.portal.website.admin.internal.control.InitFrameworkFilter.httpDoFilter(InitFrameworkFilter.java:35)
at com.vignette.portal.website.admin.internal.control.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:61)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
at com.vignette.portal.website.admin.internal.control.MultipartFilter.httpDoFilter(MultipartFilter.java:33)
at com.vignette.portal.website.admin.internal.control.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:61)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
at com.vignette.portal.website.internal.StartupProtectionFilter.doFilterSingleInvocation(StartupProtectionFilter.java:100)
at com.vignette.portal.website.internal.SingleInvocationFilter.doFilter(SingleInvocationFilter.java:52)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
at com.vignette.portal.website.internal.EnvironmentalWrapperFilter.doFilterSingleInvocation(EnvironmentalWrapperFilter.java:44)
at com.vignette.portal.website.internal.SingleInvocationFilter.doFilter(SingleInvocationFilter.java:52)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3229)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)
at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2002)
at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:1908)
at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1362)
at weblogic.work.ExecuteRequestAdapter.execute(ExecuteRequestAdapter.java:21)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)
Caused by: java.lang.NullPointerException:
at weblogic.server.channels.ServerChannelImpl.equals(ServerChannelImpl.java:103)
at weblogic.cluster.replication.ReplicationManagerServerRef.checkPriviledges(ReplicationManagerServerRef.java:102)
at weblogic.rmi.internal.BasicServerRef.dispatch(BasicServerRef.java:302)
at weblogic.rmi.internal.BasicServerRef.dispatch(BasicServerRef.java:877)
at weblogic.rjvm.RJVMImpl.dispatchRequest(RJVMImpl.java:1084)
at weblogic.rjvm.RJVMImpl.dispatch(RJVMImpl.java:1001)
at weblogic.rjvm.ConnectionManagerServer.handleRJVM(ConnectionManagerServer.java:240)
at weblogic.rjvm.ConnectionManager.dispatch(ConnectionManager.java:881)
at weblogic.rjvm.MsgAbbrevJVMConnection.dispatch(MsgAbbrevJVMConnection.java:446)
at weblogic.rjvm.t3.MuxableSocketT3.dispatch(MuxableSocketT3.java:368)
at weblogic.socket.AbstractMuxableSocket.dispatch(AbstractMuxableSocket.java:378)
at weblogic.socket.SSLFilter.dispatch(SSLFilter.java:258)
at weblogic.socket.SocketMuxer.readReadySocketOnce(SocketMuxer.java:856)
at weblogic.socket.SocketMuxer.readReadySocket(SocketMuxer.java:802)
at weblogic.socket.EPollSocketMuxer.dataReceived(EPollSocketMuxer.java:192)
at weblogic.socket.EPollSocketMuxer.processSockets(EPollSocketMuxer.java:174)
at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)
at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)
... 2 more 

I think that this error means NPE when serializing or deserializing session data from different node. If this is the case, then you should look into what types of data are placed into HTTP-session in your application. 

Thanks for your response. The strange thing is though, that we have the same application running on about 8 other clustered environments without issue, its just this one that it fails on.
It could be a configuration issue for the application, but I'd like to believe its something in the app server :) 

Have you checked differences in configuration between those clusters? 

It's a sensible question :)
We haven't yet, as they're standard builds, and should all be the same, but it's worth checking. 

Hi Scott,
Did you get to the bottom of this?
We're experiencing exactly the same symptoms, except we're upgrading 8.1 -> 10.3
App works fine on a single server, as soon as a second server is brought into the cluster we get the same exception.
I'm checking the configs now for anything suspicious... 

No - there were some minor configuration changes made to the servers (things like the max open files limit in Linux), and I was waiting for some restarts to test again afterwards, but am no longer on the project, so I'm not sure if that has helped. 

user10800410 did you manage to find the cause of this error? 

The error codes posted clearly show replication problems in Clustered environment. If some configuration values were posted, it would be easier to track root cause.

Related

OSB problem posting >5MB body to a business service endpoint

Hello,
I have an https endpoint they results in a Broken pipe exception after running for about 10mins if I send a SOAP body greater than 5MB. If I send the same request to this endpoint using SOAPUI (no OSB involvement) it works fine and completes in <1min, it only fails through OSB. Also if the request is less than 5MB it works fine through OSB and again completes in <1min.
I've tried calling a proxy service (through test console and SOAPUI) and the business service (through test console) and all fail with the same broken pipe exception. Below is the constant state of the thread during this 10min period.
"[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=3 tid=0x02fa6400 nid=0x16 runnable [0xb19fd000..0xb19ffaf0]
java.lang.Thread.State: RUNNABLE
     at java.net.SocketOutputStream.socketWrite0(Native Method)
     at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
     at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
     at com.certicom.io.OutputSSLIOStream.write(Unknown Source)
     at com.certicom.tls.record.WriteHandler.flushOutput(Unknown Source)
     at com.certicom.tls.record.WriteHandler.write(Unknown Source)
     at com.certicom.io.OutputSSLIOStreamWrapper.write(Unknown Source)
     at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
     - locked <0xe70af568> (a java.io.BufferedOutputStream)
     at weblogic.net.http.HttpOutputStream.write(HttpOutputStream.java:22)
     at weblogic.utils.io.UnsyncByteArrayOutputStream.writeTo(UnsyncByteArrayOutputStream.java:104)
     at weblogic.net.http.HttpURLConnection.writeRequests(HttpURLConnection.java:162)
     - locked <0xcefd08f8> (a weblogic.net.http.SOAPHttpsURLConnection)
     at weblogic.net.http.HttpURLConnection.writeRequestForAsyncResponse(HttpURLConnection.java:492)
     at weblogic.net.http.AsyncResponseHandler.writeRequestAndRegister(AsyncResponseHandler.java:190)
     at weblogic.net.http.AsyncResponseHandler.writeRequestAndRegister(AsyncResponseHandler.java:152)
     at com.bea.wli.sb.transports.http.HttpOutboundMessageContext.send(HttpOutboundMessageContext.java:313)
     at com.bea.wli.sb.transports.http.HttpTransportProvider.sendMessageAsync(HttpTransportProvider.java:564)
     at sun.reflect.GeneratedMethodAccessor310.invoke(Unknown Source)
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
     at java.lang.reflect.Method.invoke(Method.java:597)
     at com.bea.wli.sb.transports.Util$1.invoke(Util.java:82)
............
I am running OSB 10.3.0 on SPARC Solaris using Sun JVM.
Any ideas/suggestions?
Many thanks,
Mike. 
A quick search on google shows that broken pipe exception are thrown if the tcp connection is closed from some one other than the client. Could the network be the bottleneck..I think you are running OSB from the data center, while SOAPUI is deployed in your local env. Could the network path taken by the two being different be the reason for the difference you are observing ? Could there be any active firewalls between OSB and endpoint which stops connection after 10 mins ? 
Hello,
thank you for your response but I don't believe the network is the problem. I'm running the tests from the same network and go through the same firewalls. The other factor that makes me think it is NOT a network issue is that the request succeeds if the body is less than <5MB (a 4.5MB body will work absolutely fine).
Thanks,
Mike. 
Can you try using the streaming option on the proxy service?
I was able to send messages of like 50mb to http endpoints (with streaming though)
It could be some timeout setting in wls.
http://forums.adobe.com/thread/584822 (http post)
http://stackoverflow.com/questions/1307154/weblogic-transaction-timeout-how-to-set-in-admin-console-in-weblogic-as-8-1 (jta)
Can you play a bit with these kind of settings to see what they do for you?
Any other stacktraces in the wls logging ?
Edited by: Eric Elzinga (IT-Eye) on Mar 29, 2010 11:42 PM 
Hello,
I've tried content streaming but it made no difference. Maybe I've misunderstood content streaming but I thought you use this when body variable in the pipeline pair of a proxy is too big to manipulate in memory. In my scenario this is not a problem (I can report on the body fine just before I route to the business service) the only issue occurs when I call the business service with a request greater than 5MB.
I've already adjusted the WLS timeouts/message sizes as follows:
Max Post Size = -1 (although I think this is only used for POST to WLS rather than the other way around)
Max Message Size = 50000000
JTA timeout = 60
The only stack trace is for the broken pipe:
####<30-Mar-2010 08:35:57 o'clock BST> <Debug> <AlsbTransports> <int-app-04> <RSPCA_ESB_Server1> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <> <1269934557943> <BEA-000000> <LoadBalanceFailoverListener.sendMessageToServiceAsync
com.bea.wli.sb.transports.TransportException: Broken pipe
at com.bea.wli.sb.transports.TransportException.newInstance(TransportException.java:195)
at com.bea.wli.sb.transports.http.HttpOutboundMessageContext.send(HttpOutboundMessageContext.java:370)
at com.bea.wli.sb.transports.http.HttpTransportProvider.sendMessageAsync(HttpTransportProvider.java:564)
at sun.reflect.GeneratedMethodAccessor327.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.bea.wli.sb.transports.Util$1.invoke(Util.java:82)
at $Proxy60.sendMessageAsync(Unknown Source)
at com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageAsync(LoadBalanceFailoverListener.java:148)
at com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageToServiceAsync(LoadBalanceFailoverListener.java:543)
at com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageToService(LoadBalanceFailoverListener.java:478)
at com.bea.wli.sb.transports.TransportManagerImpl.sendMessageToService(TransportManagerImpl.java:544)
at com.bea.wli.sb.transports.TransportManagerImpl.sendMessageAsync(TransportManagerImpl.java:422)
at com.bea.wli.sb.pipeline.PipelineContextImpl.doDispatch(PipelineContextImpl.java:583)
at com.bea.wli.sb.pipeline.PipelineContextImpl.dispatch(PipelineContextImpl.java:498)
at stages.routing.runtime.RouteRuntimeStep.processMessage(RouteRuntimeStep.java:128)
at com.bea.wli.sb.pipeline.StatisticUpdaterRuntimeStep.processMessage(StatisticUpdaterRuntimeStep.java:41)
at com.bea.wli.sb.stages.StageMetadataImpl$WrapperRuntimeStep.processMessage(StageMetadataImpl.java:339)
at com.bea.wli.sb.pipeline.RouteNode.doRequest(RouteNode.java:106)
at com.bea.wli.sb.pipeline.Node.processMessage(Node.java:67)
at com.bea.wli.sb.pipeline.PipelineContextImpl.execute(PipelineContextImpl.java:866)
at com.bea.wli.sb.pipeline.Router.processMessage(Router.java:191)
at com.bea.wli.sb.pipeline.MessageProcessor.processRequest(MessageProcessor.java:75)
at com.bea.wli.sb.pipeline.RouterManager$1.run(RouterManager.java:508)
at com.bea.wli.sb.pipeline.RouterManager$1.run(RouterManager.java:506)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
at weblogic.security.service.SecurityManager.runAs(Unknown Source)
at com.bea.wli.sb.security.WLSSecurityContextService.runAs(WLSSecurityContextService.java:55)
at com.bea.wli.sb.pipeline.RouterManager.processMessage(RouterManager.java:505)
at com.bea.wli.sb.transports.TransportManagerImpl.receiveMessage(TransportManagerImpl.java:371)
at com.bea.wli.sb.transports.http.HttpTransportServlet$RequestHelper$1.run(HttpTransportServlet.java:279)
at com.bea.wli.sb.transports.http.HttpTransportServlet$RequestHelper$1.run(HttpTransportServlet.java:277)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
at weblogic.security.service.SecurityManager.runAs(Unknown Source)
at com.bea.wli.sb.transports.http.HttpTransportServlet$RequestHelper.securedInvoke(HttpTransportServlet.java:276)
at com.bea.wli.sb.transports.http.HttpTransportServlet$RequestHelper.service(HttpTransportServlet.java:237)
at com.bea.wli.sb.transports.http.HttpTransportServlet.service(HttpTransportServlet.java:133)
at weblogic.servlet.FutureResponseServlet.service(FutureResponseServlet.java:24)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:292)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:175)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3498)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic.security.service.SecurityManager.runAs(Unknown Source)
at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2180)
at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2086)
at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1406)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at com.certicom.io.OutputSSLIOStream.write(Unknown Source)
at com.certicom.tls.record.WriteHandler.flushOutput(Unknown Source)
at com.certicom.tls.record.WriteHandler.write(Unknown Source)
at com.certicom.io.OutputSSLIOStreamWrapper.write(Unknown Source)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
at weblogic.net.http.HttpOutputStream.write(HttpOutputStream.java:22)
at weblogic.utils.io.UnsyncByteArrayOutputStream.writeTo(UnsyncByteArrayOutputStream.java:104)
at weblogic.net.http.HttpURLConnection.writeRequests(HttpURLConnection.java:162)
at weblogic.net.http.HttpURLConnection.writeRequestForAsyncResponse(HttpURLConnection.java:492)
at weblogic.net.http.AsyncResponseHandler.writeRequestAndRegister(AsyncResponseHandler.java:190)
at weblogic.net.http.AsyncResponseHandler.writeRequestAndRegister(AsyncResponseHandler.java:152)
at com.bea.wli.sb.transports.http.HttpOutboundMessageContext.send(HttpOutboundMessageContext.java:313)
at com.bea.wli.sb.transports.http.HttpTransportProvider.sendMessageAsync(HttpTransportProvider.java:564)
at sun.reflect.GeneratedMethodAccessor327.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.bea.wli.sb.transports.Util$1.invoke(Util.java:82)
at $Proxy60.sendMessageAsync(Unknown Source)
at com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageAsync(LoadBalanceFailoverListener.java:148)
at com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageToServiceAsync(LoadBalanceFailoverListener.java:543)
at com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageToService(LoadBalanceFailoverListener.java:478)
at com.bea.wli.sb.transports.TransportManagerImpl.sendMessageToService(TransportManagerImpl.java:544)
at com.bea.wli.sb.transports.TransportManagerImpl.sendMessageAsync(TransportManagerImpl.java:422)
at com.bea.wli.sb.pipeline.PipelineContextImpl.doDispatch(PipelineContextImpl.java:583)
at com.bea.wli.sb.pipeline.PipelineContextImpl.dispatch(PipelineContextImpl.java:498)
at stages.routing.runtime.RouteRuntimeStep.processMessage(RouteRuntimeStep.java:128)
at com.bea.wli.sb.pipeline.StatisticUpdaterRuntimeStep.processMessage(StatisticUpdaterRuntimeStep.java:41)
at com.bea.wli.sb.stages.StageMetadataImpl$WrapperRuntimeStep.processMessage(StageMetadataImpl.java:339)
at com.bea.wli.sb.pipeline.RouteNode.doRequest(RouteNode.java:106)
at com.bea.wli.sb.pipeline.Node.processMessage(Node.java:67)
at com.bea.wli.sb.pipeline.PipelineContextImpl.execute(PipelineContextImpl.java:866)
at com.bea.wli.sb.pipeline.Router.processMessage(Router.java:191)
at com.bea.wli.sb.pipeline.MessageProcessor.processRequest(MessageProcessor.java:75)
at com.bea.wli.sb.pipeline.RouterManager$1.run(RouterManager.java:508)
at com.bea.wli.sb.pipeline.RouterManager$1.run(RouterManager.java:506)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
at weblogic.security.service.SecurityManager.runAs(Unknown Source)
at com.bea.wli.sb.security.WLSSecurityContextService.runAs(WLSSecurityContextService.java:55)
at com.bea.wli.sb.pipeline.RouterManager.processMessage(RouterManager.java:505)
at com.bea.wli.sb.transports.TransportManagerImpl.receiveMessage(TransportManagerImpl.java:371)
at com.bea.wli.sb.transports.http.HttpTransportServlet$RequestHelper$1.run(HttpTransportServlet.java:279)
at com.bea.wli.sb.transports.http.HttpTransportServlet$RequestHelper$1.run(HttpTransportServlet.java:277)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
at weblogic.security.service.SecurityManager.runAs(Unknown Source)
at com.bea.wli.sb.transports.http.HttpTransportServlet$RequestHelper.securedInvoke(HttpTransportServlet.java:276)
at com.bea.wli.sb.transports.http.HttpTransportServlet$RequestHelper.service(HttpTransportServlet.java:237)
at com.bea.wli.sb.transports.http.HttpTransportServlet.service(HttpTransportServlet.java:133)
at weblogic.servlet.FutureResponseServlet.service(FutureResponseServlet.java:24)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:292)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:175)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3498)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic.security.service.SecurityManager.runAs(Unknown Source)
at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2180)
at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2086)
at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1406)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
Thanks,
Mike. 
Can you try turning on network snoop to see the packets exchanged? In solaris i think you can use the snoop command . Could give you some indication why it is taking more than 10 mins for the transfer of 5 MB . Pipe broken error seems to occur when the other end of the tcp connection closes the connection. So seems like there could be some setting on the end system which is breaking the connection after 10 mins ... 
Hello,
Snoop has highlighting some interesting goings on. It seems when sending a payload > 5MB after about 10secs I get ZeroWindow (need to Google what this is in a minute) warning and then the destination sends a RST. At this point OSB re-establishes the the connections and sends the data again. This keeps repeating itself again and again until it eventually gives up.
Thanks,
Mike. 
Looks like the problem is at target app which is receiving the message so slowly which eventually results in filling of the tcp buffer at their side and making the available window size to 0 
I think it might be the other way around, OSB receive buffer is full but not processing what's in it for some reason. Sorry I didn't give you all the details before, the zerowindow message is reported by the OSB server and then the destination server keeps sending zerowindowprobes until it eventually gives up and resets the connection.
Mike. 
I've just setup OSB 10.3.0 on Windows using the Sun JVM and JRockit and experience the same thing. Maybe a JVM bug? I'll try updating to the latest at see what happens.
Mike.
Edited by: mikerobins on Apr 1, 2010 3:36 AM 
Hi Mike,
Try disabling "Use Chunked Streaming Mode" in your Business Service.
I had some issues when running performance test with this option enabled. After disabling it the issue was sorted out.
Please let me know if this helps.
Fabio Douek 
Thanks for the suggestion but this is already off and I don't think it can even be enabled for https endpoints.
Mike. 
Hi Mike,
I would be interested in trying to reproduce the issue. If you want to export your project and send me via email, I can try to reproduce (just remove any confidential information and unnecessary resources).
Just some points to check before:
1) You mentioned that if you send the requests for about 10 minutes with message size>5mb, the following requests will fail.
1.1) Only the requests with message size greater than 5MB or any smaller requests from OSB?
1.2) How many messages did you send during this 10 minutes period? How many parallel threads?
1.3) Is there any hogging threads in WLS?
1.4) At this stage (after the 10 minutes), if you see the socket connections to your backend from your unix shell, do you see any connection? How many? In what state?
1.5) What kind of backend are you hitting?
1.6) Are you using a single server for this test? How much memory do you have allocated to the JVM (min/max)?
Regards,
Fabio Douek. 
Hi Fabio,
1.1) Any message sent to this business service that is around 5MB or greater (I cannot be 100% on the exact tipping point) will return an broken pipe exception after 10mins or so. Smaller messages e.g. a 4.5MB body will complete successfully in ~30secs.
1.2) I only invoke the business service once but as I mentioned earlier, a packet analyser shows the target reseting the TCP connection and then OSB trying to resend the entire message. Again this only happens with a larger message, smaller ones go through fine with no TCP resets and resends.
1.3) Only the one thread is considered stuck whilst trying to write to the socket (see first post for the detail) but cpu utilisation is basically zero when I call this business service.
1.4) Will need to check this but I think only one with a state of ESTABLISHED (will update this message once I've checked)
1.5) Just an .Net service on IIS - think its running on Windows 2003 VM machine (again will update this message once I've checked). I'd love to blame this but the fact messages of all shapes and sizes work fine from SOAPUI means it's probably an OSB/WLS/JVM issue.
1.6) I've tried a clustered and single server setup. Clustered setup has 2GB heap per managed server (min and max fixed to 2GB). Single server had 1GB (min 512MB and max 1GB)
I've also tried taking my Windows install of OSB outside our corporate network (just direct internet connection to the endpoint) to rule out any firewall, NAT etc issues on our network and got the same problem :(
Thanks,
Mike. 
Hi Mike,
You mentioned that the reason for tcp connection reset is due to a zero window condition. Can you check out which side of the connection gets its buffer full. Since OSB is sending data to the endpoint , data is flowing into the endpoint only. Apart from the ACK's for the packet send by OSB, nothing should be coming back to OSB. So its hard to understand why OSB is reporting its side buffer is full even though no real data is flowing towards it.
The packet analyser should also tell the available window size on either side of the connections. So if you track through all the packets exchanged in an interaction you would better understand where the bottlenck is.
As a next step of your troubleshooting may be you can use Wireshark in your windows install and see the TCP interactions for the 2 scenarios:
1. Through OSB - where you get the pipe broken error
2. Through SOAPUI - where the request is working successful.
This would help you understand the difference. Analyse the ACK flows in 2 scenarios, the available window size on 2 sides etc.
If you feel this could be an OSB/WLS issue, I think it is time to contact the support.

OutofMemory at deployment

Hello
I wonder if anyone has a similar to the one I am experiencing.
When I deploy and ear file on production environment the ejb deploys but when
the war file tries to get deployed I get this error .
Start server side stack trace:
java.lang.OutOfMemoryError
     at java.util.zip.ZipFile.open(Native Method)
     at java.util.zip.ZipFile.<init>(ZipFile.java:105)
     at java.util.jar.JarFile.<init>(JarFile.java:110)
     at java.util.jar.JarFile.<init>(JarFile.java:52)
     at weblogic.servlet.internal.WebAppServletContext.getRoot(WebAppServletContext.java:1088)
     at weblogic.servlet.internal.WebAppServletContext.setActive(WebAppServletContext.java:4705)
     at weblogic.servlet.internal.WebAppModule.activate(WebAppModule.java:510)
     at weblogic.j2ee.J2EEApplicationContainer.activateModule(J2EEApplicationContainer.java:1655)
     at weblogic.j2ee.J2EEApplicationContainer.activate(J2EEApplicationContainer.java:1092)
     at weblogic.j2ee.J2EEApplicationContainer.activate(J2EEApplicationContainer.java:1070)
     at weblogic.management.deploy.slave.SlaveDeployer.commitUpdate(SlaveDeployer.java:1514)
     at weblogic.drs.internal.SlaveCallbackHandler$2.execute(SlaveCallbackHandler.java:34)
     at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:153)
     at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:134)
End server side stack trace
Even when I restart the server I still get the same problem .I delete the .wlndelete
directory and restart . Sometimes this resolves the problem but not all the time
. It can take atleast three attempts to restart the server before the war file
is deployed . This only happens in the production environment . I have other environments
with the same configuration but I do not encounter any problems in the other environments
. The only difference between the production environment and the others is the
fact there are a few hundred users in production .
Any ideas how I can resolve this problem ?
Also I am looking for a memory profiler I have tried to use OptimizeIt and JProbe
which really slowed the system down to the point of no use.
thanks
Sarnie
Very difficult to say without more information. I'd start by running
the jvm with -verbose:gc. I'd first want to know if you were exhausing
the java heap.
If the heap space looks fine, I'd probably next try to bump of the perm
space and see if that helps.
-- Rob
sarnie wrote:
Hello
I wonder if anyone has a similar to the one I am experiencing.
When I deploy and ear file on production environment the ejb deploys but when
the war file tries to get deployed I get this error .
Start server side stack trace:
java.lang.OutOfMemoryError
     at java.util.zip.ZipFile.open(Native Method)
     at java.util.zip.ZipFile.<init>(ZipFile.java:105)
     at java.util.jar.JarFile.<init>(JarFile.java:110)
     at java.util.jar.JarFile.<init>(JarFile.java:52)
     at weblogic.servlet.internal.WebAppServletContext.getRoot(WebAppServletContext.java:1088)
     at weblogic.servlet.internal.WebAppServletContext.setActive(WebAppServletContext.java:4705)
     at weblogic.servlet.internal.WebAppModule.activate(WebAppModule.java:510)
     at weblogic.j2ee.J2EEApplicationContainer.activateModule(J2EEApplicationContainer.java:1655)
     at weblogic.j2ee.J2EEApplicationContainer.activate(J2EEApplicationContainer.java:1092)
     at weblogic.j2ee.J2EEApplicationContainer.activate(J2EEApplicationContainer.java:1070)
     at weblogic.management.deploy.slave.SlaveDeployer.commitUpdate(SlaveDeployer.java:1514)
     at weblogic.drs.internal.SlaveCallbackHandler$2.execute(SlaveCallbackHandler.java:34)
     at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:153)
     at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:134)
End server side stack trace
Even when I restart the server I still get the same problem .I delete the .wlndelete
directory and restart . Sometimes this resolves the problem but not all the time
It can take atleast three attempts to restart the server before the war file
is deployed . This only happens in the production environment . I have other environments
with the same configuration but I do not encounter any problems in the other environments
The only difference between the production environment and the others is the
fact there are a few hundred users in production .
Any ideas how I can resolve this problem ?
Also I am looking for a memory profiler I have tried to use OptimizeIt and JProbe
which really slowed the system down to the point of no use.
thanks
Sarnie

Exception after using connection pool for first time.

Hello!
I am running a web application on WebLogic 10.0
The Managed server has a connection pool and everything seems to be working fine except an exception that appears right after I logged in.
The stack dump is here:
<Jun 25, 2009 2:52:25 PM EDT> <Warning> <JDBC> <BEA-001153> <Forcibly releasing inactive connection
"weblogic.jdbc.wrapper.PoolConnection_oracle_jdbc_driver_T4CConnection#3" back into the connection pool "myDataSource", currently reserved by: java.lang.Exception
at weblogic.jdbc.common.internal.ConnectionEnv.setup(ConnectionEnv.java:291)
at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:314)
at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:292)
at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:425)
at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:316)
at weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:93)
at weblogic.jdbc.common.internal.RmiDataSource.getPoolConnection(RmiDataSource.java:326)
at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:344)
at obnet.common.CacheManager.getConnection(CacheManager.java:377)
at obnet.common.DatabaseManager.<init>(DatabaseManager.java:76)
at obnet.common.OBNetUtility.encryptPassword(OBNetUtility.java:4446)
at obnet.login.LoginVerificationsb.validateUser(LoginVerificationsb.java:545)
at obnet.login.LoginVerificationsb_rxjfa8_EOImpl.validateUser(LoginVerificationsb_rxjfa8_EOImpl.java:415)
at obnet.login.LoginVerificationHandler.doPerform(LoginVerificationHandler.java:194)
at obnet.common.OBNetBaseAction.perform(OBNetBaseAction.java:578)
at org.apache.struts.action.ActionServlet.processActionPerform(ActionServlet.java:1787)
at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1586)
at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:510)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:226)
at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:124)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:283)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:175)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3395)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic.security.service.SecurityManager.runAs(Unknown Source)
at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2140)
at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2046)
at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1366)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:200)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:172)
.>
I have no idea how to correct it. The application is runnign normal after that.
I know it is just a Warning, but I would like to remove it and I would like to know the source of the exception.
Thank you! 
Do you have access to the code in these routines?
at obnet.common.CacheManager.getConnection(CacheManager.java:377)
at obnet.common.DatabaseManager.<init>(DatabaseManager.java:76)
at obnet.common.OBNetUtility.encryptPassword(OBNetUtility.java:4446)
at obnet.login.LoginVerificationsb.validateUser(LoginVerificationsb.java:545)
at obnet.login.LoginVerificationsb_rxjfa8_EOImpl.validateUser(LoginVerificationsb_rxjfa8_EOImpl.java:415)
The issue is that this code is obtaining a JDBC connection from the WLS datasource,
then not using it and not closing it, for longer than your datasource/pool is configured
to allow. 
I am going to examine the source code in this direction. I will let you know the result.
Thank you! 
Yes, I have found in source code the connection that has not been properly closed.
After modifying teh source code the exception has disappeared!
Thank you again!

connecting to remote weblogic jms server

hi all,
i am trying to get jms connection factory and jms queue from remote weblogic server instance.
my client is also weblogic instance from different domain.
at first, i am trying to lookup remote jndi directly. but i get no connection found exception.
So, i use Foreign JMS Providers and lookup from my local weblogic instance.
this time i get the following exception.
so what is the best way to use remote jms destination. i am noob with jms.
java.lang.SecurityException: [Security:090398]Invalid Subject: principals=[weblogic, Administrators]
at weblogic.rjvm.ResponseImpl.unmarshalReturn(ResponseImpl.java:234)
at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:348)
at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:259)
at weblogic.jms.frontend.FEConnectionFactoryImpl_1032_WLStub.connectionCreateRequest(Unknown Source)
at weblogic.jms.client.JMSConnectionFactory.setupJMSConnection(JMSConnectionFactory.java:224)
at weblogic.jms.client.JMSConnectionFactory.createConnectionInternal(JMSConnectionFactory.java:285)
at weblogic.jms.client.JMSConnectionFactory.createQueueConnection(JMSConnectionFactory.java:157)
at web.MessageSenderManagedBean.sendMessage(MessageSenderManagedBean.java:55)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.sun.el.parser.AstValue.invoke(AstValue.java:157)
at com.sun.el.MethodExpressionImpl.invoke(MethodExpressionImpl.java:283)
at org.apache.myfaces.trinidad.component.MethodExpressionMethodBinding.invoke(MethodExpressionMethodBinding.java
:46)
at com.sun.faces.application.ActionListenerImpl.processAction(ActionListenerImpl.java:102)
at org.apache.myfaces.trinidad.component.UIXCommand.broadcast(UIXCommand.java:190)
at javax.faces.component.UIViewRoot.broadcastEvents(UIViewRoot.java:475)
at javax.faces.component.UIViewRoot.processApplication(UIViewRoot.java:756)
at oracle.adfinternal.view.faces.lifecycle.LifecycleImpl._invokeApplication(LifecycleImpl.java:698)
at oracle.adfinternal.view.faces.lifecycle.LifecycleImpl._executePhase(LifecycleImpl.java:285)
at oracle.adfinternal.view.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:177)
at javax.faces.webapp.FacesServlet.service(FacesServlet.java:265)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:292)
at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
at oracle.adf.share.http.ServletADFFilter.doFilter(ServletADFFilter.java:62)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
at oracle.adfinternal.view.faces.webapp.rich.RegistrationFilter.doFilter(RegistrationFilter.java:97)
at org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl$FilterListChain.doFilter(TrinidadFilterImpl.jav
a:420)
at oracle.adfinternal.view.faces.activedata.AdsFilter.doFilter(AdsFilter.java:60)
at org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl$FilterListChain.doFilter(TrinidadFilterImpl.jav
a:420)
at org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl._doFilterImpl(TrinidadFilterImpl.java:247)
at org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl.doFilter(TrinidadFilterImpl.java:157)
at org.apache.myfaces.trinidad.webapp.TrinidadFilter.doFilter(TrinidadFilter.java:92)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
at oracle.dms.wls.DMSServletFilter.doFilter(DMSServletFilter.java:326)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
at weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:27)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3592)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)
at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2202)
at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2108)
at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1432)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
Caused by: java.lang.SecurityException: [Security:090398]Invalid Subject: principals=[weblogic, Administrators]
at weblogic.security.service.SecurityServiceManager.seal(SecurityServiceManager.java:835)
at weblogic.security.service.SecurityServiceManager.getSealedSubjectFromWire(SecurityServiceManager.java:524)
at weblogic.rjvm.MsgAbbrevInputStream.getSubject(MsgAbbrevInputStream.java:315)
at weblogic.rmi.internal.BasicServerRef.acceptRequest(BasicServerRef.java:875)
at weblogic.rmi.internal.BasicServerRef.dispatch(BasicServerRef.java:310)
at weblogic.rmi.cluster.ClusterableServerRef.dispatch(ClusterableServerRef.java:242)
at weblogic.rjvm.RJVMImpl.dispatchRequest(RJVMImpl.java:1138)
at weblogic.rjvm.RJVMImpl.dispatch(RJVMImpl.java:1020)
at weblogic.rjvm.ConnectionManagerServer.handleRJVM(ConnectionManagerServer.java:240)
at weblogic.rjvm.ConnectionManager.dispatch(ConnectionManager.java:882)
at weblogic.rjvm.MsgAbbrevJVMConnection.dispatch(MsgAbbrevJVMConnection.java:453)
at weblogic.rjvm.t3.MuxableSocketT3.dispatch(MuxableSocketT3.java:322)
at weblogic.socket.BaseAbstractMuxableSocket.dispatch(BaseAbstractMuxableSocket.java:298)
at weblogic.socket.NTSocketMuxer.processSockets(NTSocketMuxer.java:105)
at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)
at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)
With Regards,
WP 
Hi,
You need to enable Domain Trust between Both of your WLS Domains....as described in the following Link: http://download-llnw.oracle.com/docs/cd/E13222_01/wls/docs81/secmanage/domain.html#1171534
The above link belongs to WLS8.1 but the steps mentioned there is almost same for Latest version of WLS as welll.
.
.
Thanks
Jay SenSharma
http://jaysensharma.wordpress.com (WebLogic Wonders Are Here) 
hi,
I try this according to http://download.oracle.com/docs/cd/E12839_01/web.1111/e13707/domain.htm#i1175685
but i still have same exception.
With Regards,
WP 
Hi,
Could you discribe exactly what you have done , i mean the exact configurations ?
I might help you if I have a clear idea ..
Thanks. 
hi,
here is my exact configuration
Configuring Cross-Domain Security
Click the name of the domain in the Domain Configuration panel.
Open the Security: General tab in the console.
Check Cross Domain Security Enabled.
Configuring a Cross-Domain User
Click the name of your security realm
On the Credential Mappings > Default tab, click New.
On the Creating the Remote Resource for the Security Credential Mapping:
I fill remote domain name.
Click Next.
On the Create a New Security Credential Map Entry page, enter the following:
Local User: cross-domain
Remote User: weblogic
Password: weblogic1
Click Finish.
then i add weblogic user into CrossDomainConnectors group.
I apply this configuration for both weblogic instances.
With Regards,
WP 
It looks like that you are encountering a known issue in using Foreign JMS Server in a cross-domain configuration. You may want to try to really enable domain trust, as described in the following section of the document.
http://download.oracle.com/docs/cd/E13222_01/wls/docs92/ConsoleHelp/taskhelp/security/EnableTrustBetweenDomains.html 
currently i am using weblogic 11g.
i don't see any configuration like from ur reference.
With Regards,
WP

Problem with the portal page render - ArrayIndexOutOfBoundsException

Hi,
I'm getting a problem with Weblogic Portal 10.3, that happen with a load test scenario, accessing some specific pages. Some times I have the "Error 500--Internal Server Error" when a page is called, and sometimes it render the page well. I already disable my cluster and with only one managed server it still happen, and the stack trace that I have points to a problem on a Portal Core Class "com/bea/netuix/nf/factories/MetaPortlet".
####<09/06/2010 14h31min00s BRT> <Error> <HTTP> <naenia.customer.com.br> <Portal01StbyMgd03> <[ACTIVE] ExecuteThread: '31' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1276104660065> <BEA-101020> <[weblogic.servlet.internal.WebAppServletContext#86ea675 - appName: '_PortalEAR', name: '_Portal', context-path: '/_Portal', spec-version: '2.5'] Servlet failed with Exception
java.lang.ArrayIndexOutOfBoundsException: 8
at java.util.ArrayList.add(ArrayList.java:352)
at com.bea.netuix.nf.factories.MetaPortlet.init(MetaPortlet.java:90)
at com.bea.netuix.nf.factories.MetaUIControl.initNode(MetaUIControl.java:123)
at com.bea.netuix.nf.factories.MetaUIControl.initNode(MetaUIControl.java:129)
at com.bea.netuix.nf.factories.MetaUIControl.initNode(MetaUIControl.java:129)
at com.bea.netuix.nf.factories.MetaUIControl.initNode(MetaUIControl.java:129)
at com.bea.netuix.nf.factories.MetaUIControl.initNode(MetaUIControl.java:129)
at com.bea.netuix.nf.factories.MetaUIControl.initNode(MetaUIControl.java:129)
at com.bea.netuix.nf.factories.MetaUIControl.initNode(MetaUIControl.java:129)
at com.bea.netuix.nf.factories.MetaUIControl.initNode(MetaUIControl.java:129)
at com.bea.netuix.nf.factories.MetaUIControl.init(MetaUIControl.java:119)
at com.bea.netuix.nf.factories.PartialUIControlTreeCreator.init(PartialUIControlTreeCreator.java:147)
at com.bea.netuix.nf.factories.ControlTreeFactory.init(ControlTreeFactory.java:73)
at com.bea.netuix.servlets.manager.PortalServlet.createPortalAccessData(PortalServlet.java:1421)
at com.bea.netuix.servlets.manager.PortalServlet.getCustomizedPortalAccessData(PortalServlet.java:1367)
at com.bea.netuix.servlets.manager.PortalServlet.getPortalAccessDataForDesktop(PortalServlet.java:1248)
at com.bea.netuix.servlets.manager.PortalServlet.getPortalAccessData(PortalServlet.java:1049)
at com.bea.netuix.servlets.manager.PortalServlet.service(PortalServlet.java:900)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:292)
at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
at com.bea.portal.tools.servlet.http.HttpContextFilter.doFilter(HttpContextFilter.java:60)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
at com.bea.p13n.servlets.PortalServletFilter.doFilter(PortalServletFilter.java:336)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3502)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic.security.service.SecurityManager.runAs(Unknown Source)
at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2186)
at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2092)
at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1406)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
>
I suppose that problem is related with some kind of race condition while the page is rendering, but I don't know where it can be. If I activate the portalControlTreeCache on the PAT (Portal Administrative Console), in place of have this stack trace, I get a page partially rendered, missing header or some other piece of the HTML. On this scenario, all subsequent access to the page return the same bad-html, until the cache is flushed.
I'm looking for a "little smell" about how to find where this problem is happening.
Any help is welcome.
Regards,
Luis Amaral.
Edited by: luis.amaral on Jun 10, 2010 4:54 PM 
Hello Luis,
Have you looked at the <servlet-reload-check-secs> setting in your webapp's WEB-INF/weblogic.xml file? This sort of error may be caused by the server churning too much under load acting as a development server (looking for changes to configuration files and portlets) in a production (load-testing) environment. At least that would be my first guess.
Kevin 
I should point out that setting the servlet-reload-check-secs to "-1" should turn that off completely, where it loads configuration files and servlets only once per webapp deployment.
Kevin

Categories

Resources