###<Feb 7, 2013 5:59:50 AM GMT> <Info> <JMS> <kkkkk> <bo-server03513> <[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1360216790795> <BEA-040507> <The JMS SAF forwarder failed to connect to the remote destination "t3://abc.xyz:7003/jms/EJournalImport", because of javax.naming.CommunicationException [Root exception is java.rmi.UnmarshalException: RJVM has already been shutdown; nested exception is:
java.io.IOException: RJVM has already been shutdown]
at weblogic.jndi.internal.ExceptionTranslator.toNamingException(ExceptionTranslator.java:74)
at weblogic.jndi.internal.WLContextImpl.translateException(WLContextImpl.java:470)
at weblogic.jndi.internal.WLContextImpl.lookup(WLContextImpl.java:426)
at weblogic.jndi.internal.WLContextImpl.lookup(WLContextImpl.java:411)
at weblogic.jms.forwarder.Forwarder.connectTarget(Forwarder.java:416)
at weblogic.jms.forwarder.Forwarder.reconnect(Forwarder.java:246)
at weblogic.jms.forwarder.Forwarder.timerExpired(Forwarder.java:311)
at weblogic.timers.internal.TimerImpl.run(TimerImpl.java:273)
at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:528)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:178)
Caused by: java.rmi.UnmarshalException: RJVM has already been shutdown; nested exception is:
java.io.IOException: RJVM has already been shutdown
at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:360)
at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:259)
at weblogic.jndi.internal.ServerNamingNode_1035_WLStub.lookup(Unknown Source)
at weblogic.jndi.internal.WLContextImpl.lookup(WLContextImpl.java:423)
... 8 more
Caused by: java.io.IOException: RJVM has already been shutdown
at weblogic.rjvm.ResponseImpl.unmarshalReturn(ResponseImpl.java:234)
at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:348)
... 11 more
Caused by: java.io.IOException: RJVM has already been shutdown
at weblogic.rjvm.RJVMImpl.addPendingResponse(RJVMImpl.java:513)
at weblogic.rjvm.RJVMImpl.send(RJVMImpl.java:900)
at weblogic.rjvm.MsgAbbrevOutputStream.flushAndSend(MsgAbbrevOutputStream.java:394)
at weblogic.rjvm.MsgAbbrevOutputStream.sendRecv(MsgAbbrevOutputStream.java:404)
at weblogic.rjvm.BasicOutboundRequest.sendReceive(BasicOutboundRequest.java:109)
at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:345)
... 11 more
The solution was strange and found on the internet.
The problem is probably a bad or slow DNS lookup on the IP address 192.100.88.1. Indeed, the weblogic server tries to connect to the admin ListenAddress and did a double resolution
Name => Address
Address => name
To avoid this problem, it is better not to name fqdn in ListenAddress, but to put the listening IP address.We changed hostname to ip from admin console and bounced managed server.
