Error while applying BSU patch

 bash-3.2$ ./bsu.sh -install -patch_download_dir=/prod/oracle/111/utils/bsu/cache_dir/ -patchlist=T5F1 -prod_dir=/prod/oracle/111/wlserver_10.3
Exception in thread "Thread-0" java.lang.NullPointerException
        at com.bea.plateng.patch.PatchTargetHelper.loadPatchProfiles(PatchTargetHelper.java:447)
        at com.bea.plateng.patch.PatchTarget.<init>(PatchTarget.java:268)
        at com.bea.plateng.patch.PatchTargetFactory.create(PatchTargetFactory.java:30)
        at com.bea.plateng.patch.ProductAliasTarget.constructPatchTargetList(ProductAliasTarget.java:88)
        at com.bea.plateng.patch.ProductAliasTarget.<init>(ProductAliasTarget.java:46)
        at com.bea.plateng.patch.ProductAliasTargetHelper.getProdAliasTargetList(ProductAliasTargetHelper.java:55)        at com.bea.plateng.patch.ProductAliasTargetHelper.getAllHomeToProdAliasesTargetMap(ProductAliasTargetHelper.java:32)        at com.bea.plateng.patch.ProductAliasTargetHelper.checkProfilesInProductAliases(ProductAliasTargetHelper.java:133)        at com.bea.plateng.patch.Patch$1.run(Patch.java:376)
        at java.lang.Thread.run(Thread.java:662)
null

 I thought this is related to previous error posted in my blog, i.e.patch_backup.xml file having wrong MW home.But this one was different.

The directory  /prod/oracle/111/patch_ocp371/ had wrong owner hence we were getting issues.After changing to correct owner issue was fixed.Make sure the directories inside it also have correct owner and permissions..(/prod/oracle/111/patch_ocp371/profiles/default)

BEA-000109

Error:
====
<Sep 24, 2014 10:48:26 PM CDT> <Error> <Cluster> <BEA-000109> <An error occurred while sending multicast message: java.io.IOException: Invalid argument
java.io.IOException: Invalid argument
        at java.net.PlainDatagramSocketImpl.send(Native Method)
        at java.net.DatagramSocket.send(DatagramSocket.java:676)
        at weblogic.cluster.MulticastFragmentSocket.sendThrottled(MulticastFragmentSocket.java:206)
        at weblogic.cluster.MulticastFragmentSocket.send(MulticastFragmentSocket.java:158)
        at weblogic.cluster.FragmentSocketWrapper.send(FragmentSocketWrapper.java:91)
        Truncated. see log file for complete stacktrace


We faced this issue on a customer with a cluster and using multicast for some reason.When we tried starting managed server we got this error.

Fix:

Refer:Doc ID 1360498.1

Even if both protocols are enabled, it is possible to force a specific protocol for multicast using the property java.net.preferIPv4Stack. Adding this property to JAVA_OPTIONS in the startup script, like setDominEnv.sh, resolves the issue: export JAVA_OPTIONS="${JAVA_OPTIONS} -Djava.net.preferIPv4Stack=true" This fixed the issue.

Store:280105 : Error with file store

 <Aug 14, 2014 5:45:38 AM UTC> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FAILED>
<Aug 14, 2014 5:45:38 AM UTC> <Error> <WebLogicServer> <BEA-000383> <A critical service failed. The server will shut itself down>
<Aug 14, 2014 5:45:38 AM UTC> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FORCE_SHUTTING_DOWN>
<Aug 14, 2014 5:45:38 AM UTC> <Critical> <Health> <BEA-310006> <Critical Subsystem PersistentStore._WLS_wls_osb2 has failed. Setting server state to FAILED.
Reason: weblogic.store.PersistentStoreFatalException: [Store:280105]The persistent file store "_WLS_wls_osb2" cannot open file _WLS_WLS_OSB2000000.DAT.>
<Aug 14, 2014 5:45:39 AM UTC> <Critical> <WebLogicServer> <BEA-000385> <Server health failed. Reason: health of critical service 'PersistentStore._WLS_wls_osb2' failed>
<Aug 14, 2014 5:45:39 AM UTC> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FAILED>
<Aug 14, 2014 5:45:39 AM UTC> <Error> <> <BEA-000000> <cannot dump threads on this VM:

<Aug 14, 2014 2:58:43 AM UTC> <Error> <Store> <BEA-280061> <The persistent store "_WLS_wls_soa1" could not be deployed: weblogic.store.PersistentStoreFatalException: [Store:280105]The persistent file store "_WLS_wls_soa1" cannot open file _WLS_WLS_SOA1000000.DAT.
weblogic.store.PersistentStoreFatalException: [Store:280105]The persistent file store "_WLS_wls_soa1" cannot open file _WLS_WLS_SOA1000000.DAT.


We were getting these errors on a 4 node cluster for both SOA and OSB, i checked config.xml and couldnot find any file store with this name!! Also we already tried after renaming tmp/cache/data folder.Seems like nothing was working.But i finally found the root cause! It was the files in cluster folder.

/prodo/fmw/soa_domain/osb_cluster/tlogs
-bash-4.1$ ls
_WLS_WLS_OSB1000000.DAT  _WLS_WLS_OSB2000000.DAT

/prodo/fmw/soa_domain/soa_cluster/tlogs
-bash-4.1$ ls
_WLS_WLS_SOA1000000.DAT    _WLS_WLS_SOA2000000.DAT  
-bash-4.1$

Just renamed these are tried starting , then we got local filestore errors, in which case we had to rename every *.DAT file(file store folders are in domain home).After that the server came up

'NameError' when Trying to use the sca_exportComposite

I was unable to export a composite from the em console, so though of doing it through wlst.

 wls:/offline> sca_exportComposite('http://host:7001','none', '/tmp/sca_TestWS.jar', 'TestWS', '1.0',partition='test')
Traceback (innermost last):
  File "<console>", line 1, in ?
NameError: sca_exportComposite
wls:/offline>

We tried using the wlst within Oracle_common and weblogic home.But finally realized that as per Doc ID 1484174.1 it has to be from SOA Oracle Home.

http://docs.oracle.com/cd/E12839_01/core.1111/e10105/getstart.htm#ASADM10692

Then again we need to give SOA managed server port and not the admin port.

wls:/offline> sca_exportComposite('http://host:7011','none', '/tmp/sca_TestWS.jar', 'TestWS', '1.0', partition='test')
serverURL = http://host:7011
sarFile = /tmp/sca_TestWS.jar
partition= test
compositeName = TestWS
revision = 1.0
user = None
compositeDN = test/TestWS!1.0
INFO: Creating HTTP connection to host:host, port:7011
Enter username and password for realm 'default' on host host:7011
Authentication Scheme: Basic
Username: ohsadmin
Password:
INFO: Received HTTP response from the server, response code=200
Write response into file:/tmp/sca_TestWS.jar
---->Export composite success.
wls:/offline>

OAM Audting Data is not populated

We enabled OAM auditing through ,

http://docs.oracle.com/cd/E37115_01/admin.1112/e27239/audit.htm#BABHAJCB

In the service instance specified in the domain file ($DOMAIN_HOME/config/fmwconfig/jps-config.xml), enable database auditing by changing the value of the property audit.loader.repositoryType to DB

Infact we can use OAM 11g : How To Configure OAM Audit Configuration (Doc ID 1492681.1)

In spite of all these settings IAU_* tables are not populated with any data.

I found these in logs.

 [ecid: 597cbd9ac221c087:2466da57:1459486cc99:-8000-00000000000008d0,0] [APP: oam_server#11.1.2.0.0] Read configuration parameter Port = 11,598.
[2014-04-24T10:18:11.169-06:00] [wls_oam1] [NOTIFICATION] [OAM-04003] [oracle.oam.proxy.oam] [tid: [ACTIVE].ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: <anonymous>] [ecid: 597cbd9ac221c087:2466da57:1459486cc99:-8000-00000000000008d0,0] [APP: oam_server#11.1.2.0.0] OAM proxy started. Listening on port 11598.

<Apr 29, 2014 12:15:48 PM MDT> <Error> <oracle.security.audit.logger> <BEA-000000> <IAU:Unable to write Audit Event for Type: OAM>
<Apr 29, 2014 12:15:48 PM MDT> <Error> <oracle.security.audit.logger> <BEA-000000> <IAU:Unable to prepare statement:INSERT INTO IAU_BASE COLUMNS ( IAU_ID, IAU_ComponentType, IAU_InstanceId, IAU_HostId, IAU_HostNwaddr, IAU_ModuleId, IAU_ProcessId, IAU_OracleHome, IAU_HomeInstance, IAU_ECID, IAU_RID, IAU_ContextFields, IAU_SessionId, IAU_TargetComponentType, IAU_ApplicationName, IAU_EventType, IAU_EventCategory, IAU_EventStatus, IAU_TstzOriginating, IAU_ThreadId, IAU_ComponentName, IAU_Initiator, IAU_MessageText, IAU_FailureCode, IAU_RemoteIP, IAU_Target, IAU_Resource, IAU_Roles, IAU_DomainName, IAU_ComponentData, IAU_AuditUser, IAU_TenantId, IAU_TransactionId, IAU_UserTenantId, IAU_AuthenticationMethod ) VALUES ( ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ? ) >

Also the OAM version was,

    <Setting Name="ProductRelease" Type="xsd:string">11.1.2.1.0</Setting>

The issue is due to a known bug.

Bug 18120234 : OAM 11.1.2.1.0 : IAU:UNABLE TO WRITE AUDIT EVENT FOR TYPE: OAM

The fix is,

SQL> alter table iau_base add (IAU_USERTENANTID VARCHAR2(255));

Table altered.

SQL> alter table iau_base add (IAU_TENANTID VARCHAR2(255));

Table altered.

SQL> alter table iau_base add (IAU_AUDITUSER VARCHAR2(255));

Table altered.

The table structure doesnot have these columns,

Bsu error

Sometimes we get strange error which has no head or tail and no hit on google or oracle support, this was one of them :)

When applying a PSU on weblogic , we need to remove the previously applied PSU.We tried and got this error.

$ ./bsu.sh -prod_dir=/app/111/wlserver_10.3 -patchlist=MYFD -log=test.log -verbose -remove -log_priority=trace
Checking for conflicts..
No conflict(s) detected
Starting removal of Patch ID: MYFD
Restoring /app1/111/modules/com.bea.core.bea.opensaml_1.0.0.0_6-2-0-0.jar from /app1/fmw/product/111/patch_wls1036/backup/backup.jar
Result: Failure
Failure condition follows:
An error occured while removing patches - run with logging to obtain detailed information
java.io.FileNotFoundException: /app1/111/modules/com.bea.core.bea.opensaml_1.0.0.0_6-2-0-0.jar (No such file or directory)
Result: Failure
Failure condition follows:
An error occured while removing patches - run with logging to obtain detailed information
java.io.FileNotFoundException: /app1/111/modules/com.bea.core.bea.opensaml_1.0.0.0_6-2-0-0.jar (No such file or directory)


Trace error
========

2014-01-15 02:09:21,317 DEBUG [Main Thread] com.bea.plateng.common.util.ResourceBundleManager - Retrieved (An unexpected error was encountered removing patch %s.  That resource may be busy.) under key (patchCommandLine.remove.error.message1) from namespace <public>.

2014-01-15 02:09:21,331 DEBUG [Main Thread] com.bea.plateng.common.util.ResourceBundleManager - Retrieved (The patch was not removed, and has been restored to its original state.  For further information, enable logging and run patch removal again.) under key (patchCommandLine.remove.error.message2) from namespace <public>.

2014-01-15 02:09:21,332 DEBUG [Main Thread] com.bea.plateng.patch.CommandLinePatchInstaller - An unexpected error was encountered removing patch MYFD.  That resource may be busy.

The patch was not removed, and has been restored to its original state.  For further information, enable logging and run patch removal again.

com.bea.plateng.patch.PatchRemovalException: java.io.FileNotFoundException: /app1/111/modules/com.bea.core.bea.opensaml_1.0.0.0_6-2-0-0.jar (No such file or directory)        at com.bea.plateng.patch.FilePatchRemover.removePatch(FilePatchRemover.java:210)

Notice its picking wrong path app1 instead of app, in our case app1 was not even on same host and didnot even exist.I was looking where was it located.Registry.xml as correct.The culprit was in 

<MW_HOME>/patch_wls1036/registry/patch-backup.xml 

Fixed the correct name here and was able to apply and remove patch.

BAM-00404: Authentication failed. User is marked inactive.

Customer said unable to login to BAM


 ####<Jan 8, 2014 12:47:27 AM CST> <Notice> <Diagnostics> <host> <bam_server1> <[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (sel
f-tuning)'> <<WLS Kernel>> <> <350d4dae856e0ef9:2ce414fe:14355f34b38:-8000-000000000000bd06> <1389163647160> <BEA-320068> <Watch 'UncheckedException' with severity 'Notice' on server 'bam_server1' has triggered at Jan 8, 2014 12:47:27 AM CST. Notification details:
WatchRuleType: Log
WatchRule: (SEVERITY = 'Error') AND ((MSGID = 'WL-101020') OR (MSGID = 'WL-101017') OR (MSGID = 'WL-000802') OR (MSGID = 'BEA-101020') OR (MSGID = 'BEA-101017') OR (MSGID = 'BEA
-000802'))
WatchData: DATE = Jan 8, 2014 12:47:27 AM CST SERVER = bam_server1 MESSAGE = [ServletContext@345087967[app:oracle-bam module:OracleBAM path:/OracleBAM spec-version:2.5 version:1
1.1.1]] Servlet failed with Exception oracle.bam.common.security.authentication.AuthenticationException: BAM-00404: Authentication failed. User is marked inactive.
        at oracle.bam.adc.security.authentication.Authenticator.synchronizeUser(Authenticator.java:1147)
        at oracle.bam.adc.security.authentication.Authenticator.setUserContext(Authenticator.java:806)
        at oracle.bam.adc.kernel.server.DataStoreServer.setUserContext(DataStoreServer.java:476)
        at oracle.bam.adc.ejb.BamAdcServerBean.interceptor(BamAdcServerBean.java:261)
        at sun.reflect.GeneratedMethodAccessor544.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:310)
        at com.oracle.pitchfork.intercept.JeeInterceptorInterceptor.invoke(JeeInterceptorInterceptor.java:68)

solution:
======
There was a note which my colleague found , it fixed the issue.

 1. Run the following script to reset the user login table.This needs to be done on BAM schema.

Do a check,

SQL> select GUID from Dev_ORABAM."SysIterUser";

GUID
--------------------------------------------------------------------------------
E4112BD05D4611E19F345DE83E5A62E5

UPDATE "SysIterUser" SET "SysIterUser"."GUID" = NULL,
"SysIterUser"."Inactive" = NULL;

After,
SQL> select GUID from DKENDO_ORABAM."SysIterUser";

GUID
--------------------------------------------------------------------------------


2. Reorder authentication providers and make the LDAP provider on the top of the list in WebLogic Server Console.

The second step was not needed in our case.