<Aug 14, 2014 5:45:38 AM UTC> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FAILED>
<Aug 14, 2014 5:45:38 AM UTC> <Error> <WebLogicServer> <BEA-000383> <A critical service failed. The server will shut itself down>
<Aug 14, 2014 5:45:38 AM UTC> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FORCE_SHUTTING_DOWN>
<Aug 14, 2014 5:45:38 AM UTC> <Critical> <Health> <BEA-310006> <Critical Subsystem PersistentStore._WLS_wls_osb2 has failed. Setting server state to FAILED.
Reason: weblogic.store.PersistentStoreFatalException: [Store:280105]The persistent file store "_WLS_wls_osb2" cannot open file _WLS_WLS_OSB2000000.DAT.>
<Aug 14, 2014 5:45:39 AM UTC> <Critical> <WebLogicServer> <BEA-000385> <Server health failed. Reason: health of critical service 'PersistentStore._WLS_wls_osb2' failed>
<Aug 14, 2014 5:45:39 AM UTC> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FAILED>
<Aug 14, 2014 5:45:39 AM UTC> <Error> <> <BEA-000000> <cannot dump threads on this VM:
<Aug 14, 2014 2:58:43 AM UTC> <Error> <Store> <BEA-280061> <The persistent store "_WLS_wls_soa1" could not be deployed: weblogic.store.PersistentStoreFatalException: [Store:280105]The persistent file store "_WLS_wls_soa1" cannot open file _WLS_WLS_SOA1000000.DAT.
weblogic.store.PersistentStoreFatalException: [Store:280105]The persistent file store "_WLS_wls_soa1" cannot open file _WLS_WLS_SOA1000000.DAT.
We were getting these errors on a 4 node cluster for both SOA and OSB, i checked config.xml and couldnot find any file store with this name!! Also we already tried after renaming tmp/cache/data folder.Seems like nothing was working.But i finally found the root cause! It was the files in cluster folder.
/prodo/fmw/soa_domain/osb_cluster/tlogs
-bash-4.1$ ls
_WLS_WLS_OSB1000000.DAT _WLS_WLS_OSB2000000.DAT
<Aug 14, 2014 5:45:38 AM UTC> <Error> <WebLogicServer> <BEA-000383> <A critical service failed. The server will shut itself down>
<Aug 14, 2014 5:45:38 AM UTC> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FORCE_SHUTTING_DOWN>
<Aug 14, 2014 5:45:38 AM UTC> <Critical> <Health> <BEA-310006> <Critical Subsystem PersistentStore._WLS_wls_osb2 has failed. Setting server state to FAILED.
Reason: weblogic.store.PersistentStoreFatalException: [Store:280105]The persistent file store "_WLS_wls_osb2" cannot open file _WLS_WLS_OSB2000000.DAT.>
<Aug 14, 2014 5:45:39 AM UTC> <Critical> <WebLogicServer> <BEA-000385> <Server health failed. Reason: health of critical service 'PersistentStore._WLS_wls_osb2' failed>
<Aug 14, 2014 5:45:39 AM UTC> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FAILED>
<Aug 14, 2014 5:45:39 AM UTC> <Error> <> <BEA-000000> <cannot dump threads on this VM:
<Aug 14, 2014 2:58:43 AM UTC> <Error> <Store> <BEA-280061> <The persistent store "_WLS_wls_soa1" could not be deployed: weblogic.store.PersistentStoreFatalException: [Store:280105]The persistent file store "_WLS_wls_soa1" cannot open file _WLS_WLS_SOA1000000.DAT.
weblogic.store.PersistentStoreFatalException: [Store:280105]The persistent file store "_WLS_wls_soa1" cannot open file _WLS_WLS_SOA1000000.DAT.
We were getting these errors on a 4 node cluster for both SOA and OSB, i checked config.xml and couldnot find any file store with this name!! Also we already tried after renaming tmp/cache/data folder.Seems like nothing was working.But i finally found the root cause! It was the files in cluster folder.
/prodo/fmw/soa_domain/osb_cluster/tlogs
-bash-4.1$ ls
_WLS_WLS_OSB1000000.DAT _WLS_WLS_OSB2000000.DAT
/prodo/fmw/soa_domain/soa_cluster/tlogs
-bash-4.1$ ls
_WLS_WLS_SOA1000000.DAT _WLS_WLS_SOA2000000.DAT
-bash-4.1$
-bash-4.1$ ls
_WLS_WLS_SOA1000000.DAT _WLS_WLS_SOA2000000.DAT
-bash-4.1$
Just renamed these are tried starting , then we got local filestore errors, in which case we had to rename every *.DAT file(file store folders are in domain home).After that the server came up
