Thursday, February 7, 2013

RJVM has already been shutdown


 ###<Feb 7, 2013 5:59:50 AM GMT> <Info> <JMS> <kkkkk> <bo-server03513> <[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1360216790795> <BEA-040507> <The JMS SAF forwarder failed to connect to the remote destination "t3://abc.xyz:7003/jms/EJournalImport", because of javax.naming.CommunicationException [Root exception is java.rmi.UnmarshalException: RJVM has already been shutdown; nested exception is:
 java.io.IOException: RJVM has already been shutdown]
 at weblogic.jndi.internal.ExceptionTranslator.toNamingException(ExceptionTranslator.java:74)
 at weblogic.jndi.internal.WLContextImpl.translateException(WLContextImpl.java:470)
 at weblogic.jndi.internal.WLContextImpl.lookup(WLContextImpl.java:426)
 at weblogic.jndi.internal.WLContextImpl.lookup(WLContextImpl.java:411)
 at weblogic.jms.forwarder.Forwarder.connectTarget(Forwarder.java:416)
 at weblogic.jms.forwarder.Forwarder.reconnect(Forwarder.java:246)
 at weblogic.jms.forwarder.Forwarder.timerExpired(Forwarder.java:311)
 at weblogic.timers.internal.TimerImpl.run(TimerImpl.java:273)
 at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:528)
 at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
 at weblogic.work.ExecuteThread.run(ExecuteThread.java:178)
Caused by: java.rmi.UnmarshalException: RJVM has already been shutdown; nested exception is:
 java.io.IOException: RJVM has already been shutdown
 at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:360)
 at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:259)
 at weblogic.jndi.internal.ServerNamingNode_1035_WLStub.lookup(Unknown Source)
 at weblogic.jndi.internal.WLContextImpl.lookup(WLContextImpl.java:423)
 ... 8 more
Caused by: java.io.IOException: RJVM has already been shutdown
 at weblogic.rjvm.ResponseImpl.unmarshalReturn(ResponseImpl.java:234)
 at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:348)
 ... 11 more
Caused by: java.io.IOException: RJVM has already been shutdown
 at weblogic.rjvm.RJVMImpl.addPendingResponse(RJVMImpl.java:513)
 at weblogic.rjvm.RJVMImpl.send(RJVMImpl.java:900)
 at weblogic.rjvm.MsgAbbrevOutputStream.flushAndSend(MsgAbbrevOutputStream.java:394)
 at weblogic.rjvm.MsgAbbrevOutputStream.sendRecv(MsgAbbrevOutputStream.java:404)
 at weblogic.rjvm.BasicOutboundRequest.sendReceive(BasicOutboundRequest.java:109)
 at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:345)
 ... 11 more


The solution was strange and found on the internet.


The problem is probably a bad or slow DNS lookup on the IP address 192.100.88.1Indeed, the weblogic server tries to connect to the admin ListenAddress and did a double resolution 
Name => Address
Address => name
To avoid this problem, it is better not to name fqdn in ListenAddress, but to put the listening IP address.We changed hostname to ip from admin console and bounced managed server.

1 comment:

  1. cant you say that in a more simple way.... BTW thanks a lot.

    ReplyDelete