CRS-6706

# ./crsctl start cluster -all
CRS-6706: Oracle Clusterware Release patch level ('459864538') does not match Software patch level ('1175331221'). Oracle Clusterware cannot be started.
CRS-4000: Command Start failed, or completed with errors.

Case 1: When the CRS nodes have different patches or missing patch on one of the node. Then you need to apply the missing patch.

case2: <<our case above>>
Even despite having same patchset, the opatch lsinventory showed different patch set.This was on 12.1.0.2

Execute"<GI_HOME>/crs/install/rootcrs.sh -patch" as root 

This fixed the issue.

The policy domain "OIMDomain" does not exist

Issue: Was applying the OIM Jan patch and looks like same issue while applying APril 2018 patch.

Error : While running post patch step.Had updated the file patch_oim_wls.profile   and ran sh patch_oim_wls.sh

Failed with,
/sid/oracle/oim/Middleware/Oracle_IDM1/server/setup/deploy-files/automation.xml:303: The following error occurred while executing this line:
/sid/oracle/oim/Middleware/Oracle_IDM1/server/setup/deploy-files/automation.xml:106: oracle.security.jps.service.policystore.PolicyObjectNotFoundException: The policy domain "OIMDomain" does not exist

Solution: 

Doc ID 2409764.1, below are actual steps.

xlconfig.xml in OIM 11g

So, those who have worked with old version of OIM will wonder , where is xlconfig.xml.Below two are handy,

Option 1:
=========

1) Using the Enterprise Manager: http://hostname:<WL_Admin_port>/em. Login using weblogic credentials.
2) Go to Identity And Access and right click on OIM server and select System Mbean Browser.
3) Once it loads, in the System Mbean Browser on the right pane, scroll to the "Application Defined MBeans" >oracle.iam-> oim_server1->application:oim->XML config->config. This holds the config properties in Mbeans.

Option 2:
========

1) Open the weblogic.properties file in OIM_ORACLE_HOME\server\bin and set the wls_servername, application_name and metadata_to_loc tovalues based your environment.
2) Set the metadata_files=/db/oim-config.xml
3) Then run the WeblogicExportMetaData.bat/sh to get the xlconfig in the location specified at metadata_to_loc.

Oracle Big data cloud

 I got a demo account finally setup, so explored few cloud services.The Oracle Big data cloud service makes setting up bigdata very simple.Below are few screenshots.

 Big Data Appliance, Oracle provides you not only the hardware, but the software for your Big Data needs.BDCS, you have that same capability that you have with your Big Data Appliance, but taking advantage of cloud services.

Big data appliance, on premise
BDCC - > Orcale Bigdata cloud at customer (Oracle will do it , but on premise)
BDCS - > Oracle Bigdata cloud service.


Here i am exploring BDCS. Software components of BDCS are,

Linux 8
Java 8
Cloudera enterprise datahub edition(5.x)
  - Cloudera distribution including Apache hadoop(CDH)
- Cloudera Impala
 - Hbase
 - Cloudera search
 - Apache spark
Oracle bigdata spacial and graph
Oracle R distribution

There are also connectors to integrate with relational database.Oracle Bigdata sql cloud service provides ability to write sql to connect to both Bigdata and relational database.It also has the ability of smart scan i.e data driven parallel processing.

Oracle OCIC and OCI

 I think everyone who worked on Oracle cloud was initially confused on what is OCI and OCIC. How is it different.

OCIC

OCIC has 3 types:
Dedicated compute classic
Classic compute
Sparc compute

OCIC is based on Nimbula director (Hypervisor). OCIC is virtual machine only.

OCI

OCI has 2 Type:
Bare metal instances (Dedicated physical hosts with no hypervisor.Also, customer has complete control over resources)
Virtual machine

Basically , Oracle re-named bare metal cloud services as OCI.  

Undeploy composite when soa-infra down

cd /<middleware home>/oracle_common/common/bin
./wlst.sh
wls:/offline> connect()
Please enter your username :ohsadmin
Please enter your password :
Please enter your server URL [t3://localhost:7001] :t3://xyz.143.10.42:7001
Connecting to t3://xyz.143.10.42:7001 with userid weblogic ...
deploy('ShareSoaInfraPartition','/tmp/max/ShareSoaInfraPartition.ear',upload='true')

once done

exportMetadata(application='ShareSoaInfraPartition',server='AdminServer',toLocation='<Domain_Home>',docs='/deployed-composites/deployed-composites.xml')

this created /<DOMAIN_HOME>/deployed-composites directory

edited the deployed-composites.xml
removed the ElectronicInvoice  entries from the xml.

then executed

importMetadata(application='ShareSoaInfraPartition',server='AdminServer',fromLocation='<Domain_home>',docs='/deployed-composites/deployed-composites.xml')

restarted soa_server

Linux meltdown patching

So , we started the meltdown patching like entire world is doing . Its been long weekends working on this. From an application perspective , its just stop/start , plus issues post patch.Luckily we havent had that many of those.Except one SOA 11.1.1.6 with server failover setup.

Issue: The control node manager has post server migration fails. Node manager unable to start vip.

<WARNING> <Unknown interface eth1>
Mar 12, 2018 7:45:19 PM weblogic.nodemanager.server.NMHelper$Drainer run
WARNING: Unknown interface eth1
<Mar 12, 2018 7:45:19 PM> <WARNING> <Cannot remove 100.91.192.xyz - It is not online at 'eth1' or any of its sub-interfaces>
Mar 12, 2018 7:45:19 PM weblogic.nodemanager.server.NMHelper$Drainer run
WARNING: Cannot remove 100.91.192.xyz - It is not online at 'eth1' or any of its sub-interfaces
<Mar 12, 2018 7:45:19 PM> <Warning> <Exception while executing 'PostStop' ExecutableCallbacks>
java.io.IOException: Exception while executing 'PostStop' ExecutableCallbacks
at weblogic.nodemanager.server.WLSProcess$MultiExecuteCallbackHook.execute(WLSProcess.java:297)
at weblogic.nodemanager.server.WLSProcess.executePostStopHooks(WLSProcess.java:246)
at weblogic.nodemanager.server.WLSProcess.startProcess(WLSProcess.java:197)
at weblogic.nodemanager.server.AbstractServerMonitor.startWLSProcess_inner(AbstractServerMonitor.java:419)
at weblogic.nodemanager.server.AbstractServerMonitor.startWLSProcess(AbstractServerMonitor.java:358)
at weblogic.nodemanager.server.AbstractServerMonitor.start(AbstractServerMonitor.java:103)

ava.io.IOException: Command '/test/soa_domain/bin/server_migration/wlsifconfig.sh -removeif eth1 100.91.192.xyz ' returned an unsuccessful exit code '1'. Check NM logs for script output.
at weblogic.nodemanager.system.DefaultNMPlugin$2.execute(DefaultNMPlugin.java:206)
at weblogic.nodemanager.server.WLSProcess$MultiExecuteCallbackHook.execute(WLSProcess.java:317)
at weblogic.nodemanager.server.WLSProcess$MultiExecuteCallbackHook.executeWithContinueOnFailure(WLSProcess.java:309)
at weblogic.nodemanager.server.WLSProcess$MultiExecuteCallbackHook.execute(WLSProcess.java:284)
at weblogic.nodemanager.server.WLSProcess.executePostStopHooks(WLSProcess.java:246)

Workaround:
Start services from startmanagedweblogic.sh instead of node manager.

Its a oracle bug, a patch has to be applied for 10.3.6 and 12.1.3 weblogic. This we noticed on cluster nodes with server migration setup.For patch check with oracle support.

Algorithm constraints check failed on signature algorithm: MD5withRSA

Issue: We did a PMP on OAM instance. As part of that  we updated JDK. We copied the customer certs. But when we started OAM we got, (OAM version 11.1.2.2, can occur on any version)

 <Mar 16, 2018 5:57:07 AM UTC> <Warning> <Coherence> <BEA-000000> <2018-03-16 05:57:07.204/380.379 Oracle Coherence GE 3.7.1.1 <Warning> (thread=PacketListener1, member=n/a): TcpDatagramSocket{bind=ServerSocket[addr=/141.143.130.9,localport=9097]}, exception regarding peer vmohsisos016.oracleoutsourcing.com/100.143.130.9:9095, General SSLEngine problem; Certificates do not conform to algorithm constraints; Algorithm constraints check failed on signature algorithm: MD5withRSA>
<Mar 16, 2018 5:57:07 AM UTC> <Error> <Coherence> <BEA-000000> <2018-03-16 05:57:07.505/380.680 Oracle Coherence GE 3.7.1.1 <Error> (thread=Configuration Store Observer, member=n/a): Error while starting cluster: com.tangosol.net.RequestTimeoutException: Timeout during service start: ServiceInfo(Id=0, Name=Cluster, Type=Cluster
  MemberSet=MasterMemberSet(
    ThisMember=null
    OldestMember=null
    ActualMemberSet=MemberSet(Size=0
      )
    MemberId|ServiceVersion|ServiceJoined|MemberState
    RecycleMillis=240000
    RecycleSet=MemberSet(Size=0
      )
    )
)
      

Solution: in the JDK_HOME/jre/lib/java.security , we need to make some changed.

-bash-3.2$ diff java.security_16032018 java.security
479,480c479,480
< jdk.certpath.disabledAlgorithms=MD2, MD5, SHA1 jdkCA & usage TLSServer, \
<     RSA keySize < 1024, DSA keySize < 1024, EC keySize < 224
---
> jdk.certpath.disabledAlgorithms=MD2, SHA1 jdkCA & usage TLSServer, \
>     RSA keySize < 512, DSA keySize < 1024, EC keySize < 224
523c523
< jdk.jar.disabledAlgorithms=MD2, MD5, RSA keySize < 1024, DSA keySize < 1024
---
> jdk.jar.disabledAlgorithms=MD2,RSA keySize < 512, DSA keySize < 1024
555c555
< jdk.tls.disabledAlgorithms=SSLv3, MD5withRSA, DH keySize < 1024, \
---
> jdk.tls.disabledAlgorithms=SSLv3, DH keySize < 1024, \
-bash-3.2$

OID users not propagating to EBS

   A customer complained of user sync not working between OID and EBS. Anyone who knows IDM will check dip first . In this case customer was on OID 11.1.1.7 and OAM 11.1.2.2 . When we logged in to em and clicked dip application, we were unable to access it. Then we checked ods managed server logs and hit on the issue.

<Mar 10, 2018 12:53:23 PM GMT> <Warning> <EJB> <BEA-010212> <The EJB 'UpdateJob(Application: DIP#11.1.1.2.0, EJBComponent: dipejb.jar)' contains at least one method without an explicit transaction attribute setting. The default transaction attribute of Supports will be used for the following methods: remote[updateChangesForPre1012(long,long), updateChangesForPost1012(long,long), setUpdateChangeNumber(long), updateChangesForSync(long,long)]  local[updateChangesForPre1012(long,long), updateChangesForPost1012(long,long), setUpdateChangeNumber(long), updateChangesForSync(long,long)]  >
<Mar 10, 2018 12:53:23 PM GMT> <Warning> <EJB> <BEA-010212> <The EJB 'DIPScheduler(Application: DIP#11.1.1.2.0, EJBComponent: dipejb.jar)' contains at least one method without an explicit transaction attribute setting. The default transaction attribute of Supports will be used for the following methods: remote[startConfigset(), stopConfigSet()]  local[startConfigset(), stopConfigSet()]  >
<Mar 10, 2018 12:53:23 PM GMT> <Warning> <EJB> <BEA-010212> <The EJB 'DIPProv(Application: DIP#11.1.1.2.0, EJBComponent: dipejb.jar)' contains at least one method without an explicit transaction attribute setting. The default transaction attribute of Supports will be used for the following methods: remote[hasMoreChanges(), closeConnections(), initialize(java.lang.String,java.lang.String), updateStatus(boolean), doOneIteration()]  local[hasMoreChanges(), closeConnections(), initialize(java.lang.String,java.lang.String), updateStatus(boolean), doOneIteration()]  >
APPLICATION CODE GOT A NEW CONFIG OBJECT: oracle.idm.integration.dipconfig.jaxb.DIPConfig@28f772e0
Refresh Interval Current:0
<Mar 10, 2018 12:53:32 PM GMT> <Warning> <oracle.dip> <BEA-000000> <obtaining LDAP connection failed in attempt number :1 -  Retrying>
<Mar 10, 2018 12:53:32 PM GMT> <Warning> <oracle.dip> <BEA-000000> <obtaining LDAP connection failed in attempt number :2 -  Retrying>
<Mar 10, 2018 12:53:32 PM GMT> <Warning> <oracle.dip> <BEA-000000> <obtaining LDAP connection failed in attempt number :3 -  Retrying>
<Mar 10, 2018 12:53:32 PM GMT> <Error> <oracle.dip> <BEA-000000> <maximum LDAP connection retry count reached>
<Mar 10, 2018 12:53:32 PM GMT> <Error> <oracle.dip> <BEA-000000> <Connection to LDAP server failed - Check configuration of DIP server.>
<Mar 10, 2018 12:53:32 PM GMT> <Error> <oracle.dip> <BEA-000000> <Directory Integration Platform is not able to get the context with the given details : OID host: oidardsprd.oracleoutsourcing.com Port: 10038 SSL mode:1.>
<Mar 10, 2018 12:53:32 PM GMT> <Error> <oracle.dip> <DIP-10013> <Exception>
<Mar 10, 2018 12:53:34 PM GMT> <Warning> <oracle.adfinternal.view.faces.partition.FeatureUtils> <ADF_FACES-30130> <Ignoring feature-dependency on feature "AdfDvtCommon".  No such feature exists.>
Warning: Starting ADF Library jar post-deployment on WebLogic Server. Is "provider-lazy-inited" init-param missing from LibraryFilter? Ignore this warning if the ADFJspResourceProvider is not being used.
Started: ADF Library non-ADFJspResourceProvider post-deployment
Finished: ADF Library non-ADFJspResourceProvider post-deployment (millis): 58

Solution: Well, check on em , unable to access is a issue. Other check is dipstatus.

set Oracle_Home to idm home.
cd $ORACLE_HOME/bin
-bash-4.1$ ./dipStatus -h xyz.oracle.com -p 10023 -D weblogic
[Weblogic user password]
Connection parameters initialized.
Connecting at xyz.oracle.com:10023, with userid "weblogic"..
Connected successfully.

ODIP Application is down at this host and port.
-bash-4.1$

So, start your ODS managed server , it should fix the issue.Maybe ODS was started prior to starting OID services i think.

AWS cloud services for a Oracle hardcore techie!

       Recently , i have commuting long distances which take a lot of time. I try and read something beyond usual work related articles.  I developed a interest on AWS , in comparison with Oracle cloud. With a basic knowledge on it, i thought of buying a course on Udemy. I did buy AWS Certified Solutions Architect - Associate 2018, by a cloud guru. This was to get a structured knowledge . It has basic fundamentals. I am trying to complete it, purely to get insight on various clouds in the world!

Renamed the blog

 Now, i think i will start writing on various topics of interest i read about. These days knowledge is power. So from fusion11g its now middlewaredairy. So, i will venture and start having some knowledge on middleware apart from oracle :) . Though Oracle will remain my favorite, but we need to know other players too! 

Oracle Cloud Infrastructure Classic 2018 Associate Architect 1Z0-337

 I was on a 6 month maternity break last year. I wanted to utilize this time by doing some certification. One of the things i wanted to do was 1z0-337. I had already started working on cloud i.e Oracle IAAS and PAAS. So , i already had an understanding of cloud concepts. But i also went through the oracle subscription based trainings (Its free for employees). I also went through ,

Oracle Cloud IaaS: Compute and Storage Fundamentals

This helped me in understanding some more storage and compute related terms. When i did this, it was called Oracle Infrastructure as a Service Cloud 2017 Certified Implementation Specialist .

The classic version was fairly simple, due to it being mostly one word answers type of questions. They have now launched 1Z0-932, which appears to be slightly advanced topics. I am wondering to give this new exam of target the IAM which i am preparing.

OID 12c and EM

I was just checking a newly provisioned oid 12c for a customer. A team member pointed that post  OID 12.2.1.3 ,it doesnot appear on the FMW em console. It is a product feature not a bug. Apparently since oid 12c is not controlled by opmn , as in previous versions. I could find lot of bugs created , but for now its a expected feature. 

SEVERE: [FATAL] [INS-13013]

Pre requisite failed on database:

INFO: Using default Validator configured in the Action class oracle.install.ivw.db.action.PrereqAction
INFO: ------------------List of failed Tasks------------------
INFO: *********************************************
INFO: Clock Synchronization: This test checks the Oracle Cluster Time Synchronization Services across the cluster nodes.
INFO: Severity:CRITICAL
INFO: OverallStatus:VERIFICATION_FAILED
INFO: *********************************************
INFO: Network Time Protocol (NTP): This task verifies cluster time synchronization on clusters that use Network Time Protocol (NTP).
INFO: Severity:CRITICAL
INFO: OverallStatus:VERIFICATION_FAILED
INFO: *********************************************
INFO: NTP daemon command line for slewing option "-x":
INFO: Severity:CRITICAL
INFO: OverallStatus:VERIFICATION_FAILED
INFO: *********************************************
INFO: NTP daemon's boot time configuration, in file "/etc/sysconfig/ntpd", for slewing option "-x":
INFO: Severity:CRITICAL
INFO: OverallStatus:VERIFICATION_FAILED
INFO: -----------------End of failed Tasks List----------------
INFO: Adding ExitStatus PREREQUISITES_NOT_MET to the exit status set
SEVERE: [FATAL] [INS-13013] Target environment does not meet some mandatory requirements.
   CAUSE: Some of the mandatory prerequisites are not met. See logs for details. /oracrs/oraInventory/logs/installActions2018-02-28_01-05-59AM.log

Solution: /etc/sysconfig/ntpd <<take a backup first>>

diff ntpd ntpd_bkp

2c2

< OPTIONS="-u ntp:ntp -x -p /var/run/ntpd.pid -g"

---

> OPTIONS="-u ntp:ntp -p /var/run/ntpd.pid -g"