Sunday, March 25, 2018

Linux meltdown patching

So , we started the meltdown patching like entire world is doing . Its been long weekends working on this. From an application perspective , its just stop/start , plus issues post patch.Luckily we havent had that many of those.Except one SOA 11.1.1.6 with server failover setup.

Issue: The control node manager has post server migration fails. Node manager unable to start vip.

<WARNING> <Unknown interface eth1>
Mar 12, 2018 7:45:19 PM weblogic.nodemanager.server.NMHelper$Drainer run
WARNING: Unknown interface eth1
<Mar 12, 2018 7:45:19 PM> <WARNING> <Cannot remove 100.91.192.xyz - It is not online at 'eth1' or any of its sub-interfaces>
Mar 12, 2018 7:45:19 PM weblogic.nodemanager.server.NMHelper$Drainer run
WARNING: Cannot remove 100.91.192.xyz - It is not online at 'eth1' or any of its sub-interfaces
<Mar 12, 2018 7:45:19 PM> <Warning> <Exception while executing 'PostStop' ExecutableCallbacks>
java.io.IOException: Exception while executing 'PostStop' ExecutableCallbacks
at weblogic.nodemanager.server.WLSProcess$MultiExecuteCallbackHook.execute(WLSProcess.java:297)
at weblogic.nodemanager.server.WLSProcess.executePostStopHooks(WLSProcess.java:246)
at weblogic.nodemanager.server.WLSProcess.startProcess(WLSProcess.java:197)
at weblogic.nodemanager.server.AbstractServerMonitor.startWLSProcess_inner(AbstractServerMonitor.java:419)
at weblogic.nodemanager.server.AbstractServerMonitor.startWLSProcess(AbstractServerMonitor.java:358)
at weblogic.nodemanager.server.AbstractServerMonitor.start(AbstractServerMonitor.java:103)

ava.io.IOException: Command '/test/soa_domain/bin/server_migration/wlsifconfig.sh -removeif eth1 100.91.192.xyz ' returned an unsuccessful exit code '1'. Check NM logs for script output.

at weblogic.nodemanager.system.DefaultNMPlugin$2.execute(DefaultNMPlugin.java:206)
at weblogic.nodemanager.server.WLSProcess$MultiExecuteCallbackHook.execute(WLSProcess.java:317)
at weblogic.nodemanager.server.WLSProcess$MultiExecuteCallbackHook.executeWithContinueOnFailure(WLSProcess.java:309)
at weblogic.nodemanager.server.WLSProcess$MultiExecuteCallbackHook.execute(WLSProcess.java:284)
at weblogic.nodemanager.server.WLSProcess.executePostStopHooks(WLSProcess.java:246)

Workaround:
Start services from startmanagedweblogic.sh instead of node manager.

Its a oracle bug, a patch has to be applied for 10.3.6 and 12.1.3 weblogic. This we noticed on cluster nodes with server migration setup.For patch check with oracle support.

Friday, March 16, 2018

Algorithm constraints check failed on signature algorithm: MD5withRSA

Issue: We did a PMP on OAM instance. As part of that  we updated JDK. We copied the customer certs. But when we started OAM we got, (OAM version 11.1.2.2, can occur on any version)

 <Mar 16, 2018 5:57:07 AM UTC> <Warning> <Coherence> <BEA-000000> <2018-03-16 05:57:07.204/380.379 Oracle Coherence GE 3.7.1.1 <Warning> (thread=PacketListener1, member=n/a): TcpDatagramSocket{bind=ServerSocket[addr=/141.143.130.9,localport=9097]}, exception regarding peer vmohsisos016.oracleoutsourcing.com/100.143.130.9:9095, General SSLEngine problem; Certificates do not conform to algorithm constraints; Algorithm constraints check failed on signature algorithm: MD5withRSA>
<Mar 16, 2018 5:57:07 AM UTC> <Error> <Coherence> <BEA-000000> <2018-03-16 05:57:07.505/380.680 Oracle Coherence GE 3.7.1.1 <Error> (thread=Configuration Store Observer, member=n/a): Error while starting cluster: com.tangosol.net.RequestTimeoutException: Timeout during service start: ServiceInfo(Id=0, Name=Cluster, Type=Cluster
  MemberSet=MasterMemberSet(
    ThisMember=null
    OldestMember=null
    ActualMemberSet=MemberSet(Size=0
      )
    MemberId|ServiceVersion|ServiceJoined|MemberState
    RecycleMillis=240000
    RecycleSet=MemberSet(Size=0
      )
    )
)
      

Solution: in the JDK_HOME/jre/lib/java.security , we need to make some changed.

-bash-3.2$ diff java.security_16032018 java.security
479,480c479,480
< jdk.certpath.disabledAlgorithms=MD2, MD5, SHA1 jdkCA & usage TLSServer, \
<     RSA keySize < 1024, DSA keySize < 1024, EC keySize < 224
---
> jdk.certpath.disabledAlgorithms=MD2, SHA1 jdkCA & usage TLSServer, \
>     RSA keySize < 512, DSA keySize < 1024, EC keySize < 224
523c523
< jdk.jar.disabledAlgorithms=MD2, MD5, RSA keySize < 1024, DSA keySize < 1024
---
> jdk.jar.disabledAlgorithms=MD2,RSA keySize < 512, DSA keySize < 1024
555c555
< jdk.tls.disabledAlgorithms=SSLv3, MD5withRSA, DH keySize < 1024, \
---
> jdk.tls.disabledAlgorithms=SSLv3, DH keySize < 1024, \
-bash-3.2$


Wednesday, March 14, 2018

OID users not propagating to EBS

   A customer complained of user sync not working between OID and EBS. Anyone who knows IDM will check dip first . In this case customer was on OID 11.1.1.7 and OAM 11.1.2.2 . When we logged in to em and clicked dip application, we were unable to access it. Then we checked ods managed server logs and hit on the issue.

<Mar 10, 2018 12:53:23 PM GMT> <Warning> <EJB> <BEA-010212> <The EJB 'UpdateJob(Application: DIP#11.1.1.2.0, EJBComponent: dipejb.jar)' contains at least one method without an explicit transaction attribute setting. The default transaction attribute of Supports will be used for the following methods: remote[updateChangesForPre1012(long,long), updateChangesForPost1012(long,long), setUpdateChangeNumber(long), updateChangesForSync(long,long)]  local[updateChangesForPre1012(long,long), updateChangesForPost1012(long,long), setUpdateChangeNumber(long), updateChangesForSync(long,long)]  >
<Mar 10, 2018 12:53:23 PM GMT> <Warning> <EJB> <BEA-010212> <The EJB 'DIPScheduler(Application: DIP#11.1.1.2.0, EJBComponent: dipejb.jar)' contains at least one method without an explicit transaction attribute setting. The default transaction attribute of Supports will be used for the following methods: remote[startConfigset(), stopConfigSet()]  local[startConfigset(), stopConfigSet()]  >
<Mar 10, 2018 12:53:23 PM GMT> <Warning> <EJB> <BEA-010212> <The EJB 'DIPProv(Application: DIP#11.1.1.2.0, EJBComponent: dipejb.jar)' contains at least one method without an explicit transaction attribute setting. The default transaction attribute of Supports will be used for the following methods: remote[hasMoreChanges(), closeConnections(), initialize(java.lang.String,java.lang.String), updateStatus(boolean), doOneIteration()]  local[hasMoreChanges(), closeConnections(), initialize(java.lang.String,java.lang.String), updateStatus(boolean), doOneIteration()]  >
APPLICATION CODE GOT A NEW CONFIG OBJECT: oracle.idm.integration.dipconfig.jaxb.DIPConfig@28f772e0
Refresh Interval Current:0
<Mar 10, 2018 12:53:32 PM GMT> <Warning> <oracle.dip> <BEA-000000> <obtaining LDAP connection failed in attempt number :1 -  Retrying>
<Mar 10, 2018 12:53:32 PM GMT> <Warning> <oracle.dip> <BEA-000000> <obtaining LDAP connection failed in attempt number :2 -  Retrying>
<Mar 10, 2018 12:53:32 PM GMT> <Warning> <oracle.dip> <BEA-000000> <obtaining LDAP connection failed in attempt number :3 -  Retrying>
<Mar 10, 2018 12:53:32 PM GMT> <Error> <oracle.dip> <BEA-000000> <maximum LDAP connection retry count reached>
<Mar 10, 2018 12:53:32 PM GMT> <Error> <oracle.dip> <BEA-000000> <Connection to LDAP server failed - Check configuration of DIP server.>
<Mar 10, 2018 12:53:32 PM GMT> <Error> <oracle.dip> <BEA-000000> <Directory Integration Platform is not able to get the context with the given details : OID host: oidardsprd.oracleoutsourcing.com Port: 10038 SSL mode:1.>
<Mar 10, 2018 12:53:32 PM GMT> <Error> <oracle.dip> <DIP-10013> <Exception>
<Mar 10, 2018 12:53:34 PM GMT> <Warning> <oracle.adfinternal.view.faces.partition.FeatureUtils> <ADF_FACES-30130> <Ignoring feature-dependency on feature "AdfDvtCommon".  No such feature exists.>
Warning: Starting ADF Library jar post-deployment on WebLogic Server. Is "provider-lazy-inited" init-param missing from LibraryFilter? Ignore this warning if the ADFJspResourceProvider is not being used.
Started: ADF Library non-ADFJspResourceProvider post-deployment
Finished: ADF Library non-ADFJspResourceProvider post-deployment (millis): 58

Solution: Well, check on em , unable to access is a issue. Other check is dipstatus.

set Oracle_Home to idm home.
cd $ORACLE_HOME/bin
-bash-4.1$ ./dipStatus -h xyz.oracle.com -p 10023 -D weblogic
[Weblogic user password]
Connection parameters initialized.
Connecting at xyz.oracle.com:10023, with userid "weblogic"..
Connected successfully.

ODIP Application is down at this host and port.
-bash-4.1$

So, start your ODS managed server , it should fix the issue.Maybe ODS was started prior to starting OID services i think.