Saturday, February 15, 2020

OMCAGNT-2024]: ORACLE_HOSTNAME[oc-abc-1xyz-ph-zz.compute.oraclecloud.com] cannot be evaluated to Fully Qualified Domain Name (FQDN)

So a different team got this error while installing OMC agent.

Solution :
IGNORE_VALIDATIONS=true when used while installing helped fixing the issue.

Friday, December 6, 2019

T2P ERROR - CLONE-20275

We were running t2p for a customer for migrating to OCI. Even when /tmp has space we were running into issues.

Command for T2P:
sh $MW_HOME/oracle_common/bin/copyBinary.sh \
-javaHome /oracle/fmw/jrockit-jdk1.6.0_211 \
-archiveLoc /oracle/ias/ocimig/my_copy.jar \
-sourceMWHomeLoc /oracle/fmw/product/111 \
-invPtrLoc /oracle/fmw/product/111/oracle_common/oraInst.loc
-logDirLoc /oracle/ias/tmp/logs \
-silent false \
-ignoreDiskWarning false

Error:
INFO : Dec 6, 2019 04:42:56 - CLONE-21166   Addition of the Middleware Home /pbmboo/fmw/product/111 to the archive started ...
SEVERE : Dec 6, 2019 04:43:11 - ERROR - CLONE-20275   Insufficient space to create /tmp/CLONINGCLIENT-8961367960413781108.
SEVERE : Dec 6, 2019 04:43:11 - CAUSE - CLONE-20275   Minimum required space was at least "4,318" MB, but only "4,177" MB was available for use.
SEVERE : Dec 6, 2019 04:43:11 - ACTION - CLONE-20275   Make sure that the minimum required space is available for use.

There was quite a bit of space in /tmp.

Solution:
export T2P_JAVA_OPTIONS="-Djava.io.tmpdir=<dir>/tmp" 

Then rerun the clone.If you inspect the <dir>/tmp directory while the script is running you'll see that a CLONINGCLIENT202653551162129417 file is created.
That file will be removed when the copyBinary.sh script has ended.
Now, the copyBinary.sh script should end without errors.


Friday, October 4, 2019

JCS backup failing

Another case of failed backup on JCS,

Oct 2, 2019 12:17:29 AM UTC Activity Submitted
Oct 2, 2019 12:17:29 AM UTC Activity Started
Oct 2, 2019 12:17:47 AM UTC PSM-BKP-50095: Operation execution was abborted or it has crashed possibly due to script execution error or lack of space for capturing script output. If the problem persists, contact Oracle Support Services.
Oct 2, 2019 12:19:58 AM UTC PSM-BKP-50106: Operation failed and will be retried.
Oct 2, 2019 12:25:18 AM UTC PSM-BKP-50095: Operation execution was abborted or it has crashed possibly due to script execution error or lack of space for capturing script output. If the problem persists, contact Oracle Support Services.
Oct 2, 2019 12:27:29 AM UTC PSM-BKP-50106: Operation failed and will be retried.
Oct 2, 2019 12:32:48 AM UTC PSM-BKP-50095: Operation execution was abborted or it has crashed possibly due to script execution error or lack of space for capturing script output. If the problem persists, contact Oracle Support Services.
Oct 2, 2019 12:32:48 AM UTC Activity Ended

The reason was file system was full!

Dbaas credential update failed

So, we got a new customer and some consultant used his email/password to setup dbaas backup. So, we got a standard backup user created. When i tried updating from IDD console i got,

Sep 26, 2019 6:21:36 AM UTC    Activity Submitted
Sep 26, 2019 6:21:41 AM UTC    Activity Started
Sep 26, 2019 6:21:41 AM UTC    Resetting service credentials, updating the cloud storage password for Service Type DBaaS and Service id : ADB-DATAMART-TEST-DBCS
Sep 26, 2019 6:21:41 AM UTC    Initialization of Backup Config update operation.
Sep 26, 2019 6:21:41 AM UTC    Submitting update config command for vm instance : [ADB-DATAMART-TEST-DBCS]
Sep 26, 2019 6:21:42 AM UTC    DBaaS Tools version in Service Instance [ADB-DATAMART-TEST-DBCS] is an old version [18.2.3.1.0], please consider updating it.
Sep 26, 2019 6:21:44 AM UTC    Backup Config update operation in progress.
Sep 26, 2019 6:36:56 AM UTC    Backup Config update failed due to timeout within [15] minutes.
Sep 26, 2019 6:36:59 AM UTC    Failed to reset cloud storage credential, try again! Error message: Failed cloud storage credential reset for DBaaS service : ADB-DATAMART-TEST-DBCS
Sep 26, 2019 6:36:59 AM UTC    Activity Ended


I used this to update,
https://docs.oracle.com/en/cloud/paas/database-dbaas-cloud/csdbi/update-storage-container-password.html


# rpm -qa|grep -i dbaastools
dbaastools-version_number-release_number

dbaascli patch tools list

dbaascli patch tools apply --patchid LATEST 

# /var/opt/oracle/ocde/assistants/bkup/bkup


Thursday, September 12, 2019

WSM-00254 : The message is expired

So my weekend started with a major escalation. Customer said he cant process payload and it was working prior to the CPU patching of system. He was using TLS 1.0 (supressed by new jdk), We had already disabed algorithms in java.security.

Issue:<OSB 12.2.1.2>
when try to consume a osb service:https://abc.oracle.com/xyxxxx/Customers/ContactsSvc/v2?wsdl

We are getting a 500 server error at any pharmacy,

Traige: To look for actual issue , we found that this was connecting to a SAAS application. So, i looked at public OTD logs considering it must be on the internet. I found http 200(success ) in the logs. Then i grep'ed for that string in SOA logs.

Errors:
oracle.wsm.security.SecurityException: WSM-00254 : The message is expired. Check the timestamp element in the message.
The current server time is "September 7, 2019 11:30:07 PM CDT", incoming message creation time is "September 7, 2019 5:30:00 PM CDT", configured agent expiry is 300 seconds, incoming message expiry is 120 seconds, effective message expiry (minimum of agent expiry and incoming message expiry) is 120 seconds, configured clock skew is 360 seconds. The acceptable time range is "September 7, 2019 5:24:00 PM CDT" to "September 7, 2019 5:38:00 PM CDT". The incoming message is outside the valid range as allowed by clock skew and expiry times.

Solution:
This is just a temp fix.
Set the parameters "clock skew" and "Message expiration time" as follows to high value

Enterprise Manager -> expand "WebLogic Domain" in the left pane -> right click on the domain name -> web services --> WSM Domain Configuration --> Message Security tab -> Under "Security Settings", set below values.

- Clock Skew -> 23400000msecs (6.5hrs)
- Message Expiration Time -> 23400000msecs (6.5hrs)

Tuesday, July 30, 2019

OCI: APEX does not show up in DBA_REGISTRY

We had a case where moving a customer from OCIC to OCI with apex caused issues. We could not find dba_registry on OCI.

Follow this,
https://cloud.oracle.com/iaas/whitepapers/oracle_apex_on_oci_database.pdf

[gsdsiConnect] ORA-28040, ORA-28040: No matching authentication protocol

We had upgraded the OID database from 12.1 to 12.2 , post that when starting OID we got the below error.

[2019-07-29T04:24:20.342724-06:00] [OID] [NOTIFICATION:16] [] [OIDMON] [host: aXXXXXXX] [pid: 40240] [tid: 0] Guardian: [gsdsiConnect] ORA-28040, ORA-28040: No matching authentication protocol
[2019-07-29T04:24:20.343169-06:00] [OID] [NOTIFICATION:16] [] [OIDMON] [host: aXXXXXXX] [pid: 40240] [tid: 0] Guardian: [oidmon]: Unable to connect to database,will retry again after 10 sec

So, this is known error when we move from 11g to 12c database. Then we just have to make sec_case_sensitive_logon   parameter to false and it used to work. But in this case, the parameter didnot help. So, i created a sqlnet.ora updated below,

SQLNET.ALLOWED_LOGON_VERSION=11

This fixed the issue and OID was started.

Thursday, May 2, 2019

DIP issue after JDK update

There are issues like DIP shows as down in em after JDK update. The below is the fix for this.

setDomainEnv.sh files:
    EXTRA_JAVA_PROPERTIES="${EXTRA_JAVA_PROPERTIES} -Dcom.sun.jndi.ldap.object.disableEndpointIdentification=true"
    export EXTRA_JAVA_PROPERTIES

Importing SSL certificate in SOA 11g/12c

This article will describe the method to import certificate in SOA 11g and 12c. In 11g its stored in java key tool, while in 12c it is in OPSS (i.e database).Below are applicable for both 11g and 12c.

Steps for 11g:
 In the weblogic console in SOA/OSB managed server check for keystore tab.You will see that demo identity and demo trust will be enabled.Now login to linux server where SOA managed server is running.
ps -ef|grep -i SSL
You will find a process like 

              “-Djavax.net.ssl.trustStore=/app/fmw/111/wlserver_10.3/server/lib/DemoTrust.jks”
Take a backup of this DemoTrust.jks file.

keytool -import -trustcacerts -alias <Alias Name> -file <Certificate File > -keystore <Trust Store Location> -storepass <passwd>

The services will need restart after this.

Steps for 12c:
As seen 12c , the OPSS based keystore is used.Below are the steps.
- Remove Djavax.net.ssl.trustStore=${WL_HOME}/server/lib/DemoTrust.jks from setDomainEnv.sh
The follow below steps.

1. Log in to Fusion Middleware Control (EM).

2. From the navigation pane, locate the domain i.e "SOA Domain"
3. Navigate to Security, then Keystore. The Keystore page appears.
4. Expand the stripe in which the keystore resides and  Select the row corresponding to the keystore. For this case system -> trust
We will use Trustore to place the certificate to call the external SSL partner link.
5. Click Manage.
6. If the keystore is password-protected, you are prompted for a password. Enter the keystore password and click OK.
7. The Manage Certificates page appears. Click Import.
8. The Import Certificate dialog appears.
9. Select the certificate type, either Certificate or Trusted Certificate, from the drop-down. For this case use "Trusted Certificate"
10. Provide an alias i.e "testTrust"
11. Specify the certificate source. If using the Paste option, copy and paste the certificate directly into the text box. If using the Select a file option, click Browse to select the file from the operating system.
12. Click OK. The imported certificate or trusted certificate appears in the list of certificates.
13. Click OK.
14. Bounce the managed server.
 

OIM timezone issue

We had a case where OIM was ignoring the timezone set and adding EST .Even though the JDK , OS and application timezones were set correctly.

Its a bug and Oracle has released patches for it.I would suggest raise a support SR for details.But workaround is as below,

- Add to startweblogic.sh
- Add  the TZ parameter like -Duser.timezone=Asia/Calcutta

if [ "${WLS_REDIRECT_LOG}" = "" ] ; then
        echo "Starting WLS with line:"
        echo "${JAVA_HOME}/bin/java ${JAVA_VM} ${MEM_ARGS} ${LAUNCH_ARGS} -Dweblogic.Name=${SERVER_NAME} -Djava.security.policy=${WLS_POLICY_FILE} ${JAVA_OPTIONS} ${PROXY_SETTINGS} ${SERVER_CLASS}"
        ${JAVA_HOME}/bin/java ${JAVA_VM} ${MEM_ARGS} ${LAUNCH_ARGS} -Dweblogic.Name=${SERVER_NAME} -Duser.timezone=Asia/Calcutta -Djava.security.policy=${WLS_POLICY_FILE} ${JAVA_OPTIONS} ${PROXY_SETTINGS} ${SERVER_CLASS}
else
        echo "Redirecting output from WLS window to ${WLS_REDIRECT_LOG}"

        ${JAVA_HOME}/bin/java ${JAVA_VM} ${MEM_ARGS} ${LAUNCH_ARGS} -Dweblogic.Name=${SERVER_NAME} -Duser.timezone=Asia/Calcutta -Djava.security.policy=${WLS_POLICY_FILE} ${JAVA_OPTIONS} ${PROXY_SETTINGS} ${SERVER_CLASS}  >"${WLS_REDIRECT_LOG}" 2>&1
fi

Note that location is before SERVER_NAME

Tuesday, April 23, 2019

Failed to create RCU schemas when creating JCS

When i was creating a JCS instance in past. I had faced an issues where it failed at schema creation on DBCS.

Failed to create RCU schemas. Verify Database connectivity to XXXXX:1521:PDB1.YYYYY.oraclecloud.internal with sys -dbRole sysdba and make sure the Database has enough available space>

First thing is to check if database and listener are up. If they are then check the security rules.

- In the service console go to the network tab.
-  Search <servicename>/db_1/ora_p2_dblistener using the word "ora_p2_dblistener"
- Select update
- Check the status if its disabled , enable it.

Now you should be able to proceed to create the JCS instance.

The backup failed due to an object store connectivity issue

These days i have had exposure to OCI database. Its similar to OCIC dbaas which i had worked on from sometime. Though had stopped due to some organizational restructure. But i had faced this for one of the early customer i provisioned. As the infra team was also trying to understand OCI when i was launched. I like OCI as its quite robust and has many features compatible to AWS.  The above error is due to object storage connectivity error.



        We did refer the pre-reqs,

https://docs.cloud.oracle.com/iaas/Content/Database/Tasks/backingupOS.htm#Prerequi



However, we could either create a service gateway or a internet gateway and create appropriate routes to the object store.Basically , service gateway is created when you dont want traffic to go over the internet.

Wednesday, August 29, 2018

CRS-6706

# ./crsctl start cluster -all
CRS-6706: Oracle Clusterware Release patch level ('459864538') does not match Software patch level ('1175331221'). Oracle Clusterware cannot be started.
CRS-4000: Command Start failed, or completed with errors.

Case 1: When the CRS nodes have different patches or missing patch on one of the node. Then you need to apply the missing patch.

case2: <<our case above>>
Even despite having same patchset, the opatch lsinventory showed different patch set.This was on 12.1.0.2

Execute"<GI_HOME>/crs/install/rootcrs.sh -patch" as root 

This fixed the issue.

Monday, June 25, 2018

The policy domain "OIMDomain" does not exist

Issue: Was applying the OIM Jan patch and looks like same issue while applying APril 2018 patch.

Error : While running post patch step.Had updated the file patch_oim_wls.profile   and ran sh patch_oim_wls.sh

Failed with,
/sid/oracle/oim/Middleware/Oracle_IDM1/server/setup/deploy-files/automation.xml:303: The following error occurred while executing this line:
/sid/oracle/oim/Middleware/Oracle_IDM1/server/setup/deploy-files/automation.xml:106: oracle.security.jps.service.policystore.PolicyObjectNotFoundException: The policy domain "OIMDomain" does not exist

Solution: 
Doc ID 2409764.1, below are actual steps.


Tuesday, May 22, 2018

xlconfig.xml in OIM 11g

So, those who have worked with old version of OIM will wonder , where is xlconfig.xml.Below two are handy,

Option 1:
=========

1) Using the Enterprise Manager: http://hostname:<WL_Admin_port>/em. Login using weblogic credentials.
2) Go to Identity And Access and right click on OIM server and select System Mbean Browser.
3) Once it loads, in the System Mbean Browser on the right pane, scroll to the "Application Defined MBeans" >oracle.iam-> oim_server1->application:oim->XML config->config. This holds the config properties in Mbeans.

Option 2:
========

1) Open the weblogic.properties file in OIM_ORACLE_HOME\server\bin and set the wls_servername, application_name and metadata_to_loc tovalues based your environment.
2) Set the metadata_files=/db/oim-config.xml
3) Then run the WeblogicExportMetaData.bat/sh to get the xlconfig in the location specified at metadata_to_loc.

Friday, May 11, 2018

Oracle Big data cloud

 I got a demo account finally setup, so explored few cloud services.The Oracle Big data cloud service makes setting up bigdata very simple.Below are few screenshots.

 Big Data Appliance, Oracle provides you not only the hardware, but the software for your Big Data needs.BDCS, you have that same capability that you have with your Big Data Appliance, but taking advantage of cloud services.

Big data appliance, on premise
BDCC - > Orcale Bigdata cloud at customer (Oracle will do it , but on premise)
BDCS - > Oracle Bigdata cloud service.


Here i am exploring BDCS. Software components of BDCS are,
Linux 8
Java 8
Cloudera enterprise datahub edition(5.x)
  - Cloudera distribution including Apache hadoop(CDH)
- Cloudera Impala
 - Hbase
 - Cloudera search
 - Apache spark
Oracle bigdata spacial and graph
Oracle R distribution

There are also connectors to integrate with relational database.Oracle Bigdata sql cloud service provides ability to write sql to connect to both Bigdata and relational database.It also has the ability of smart scan i.e data driven parallel processing.




Oracle OCIC and OCI

 I think everyone who worked on Oracle cloud was initially confused on what is OCI and OCIC. How is it different.

OCIC

OCIC has 3 types:
Dedicated compute classic
Classic compute
Sparc compute

OCIC is based on Nimbula director (Hypervisor). OCIC is virtual machine only.

OCI

OCI has 2 Type:
Bare metal instances (Dedicated physical hosts with no hypervisor.Also, customer has complete control over resources)
Virtual machine

Basically , Oracle re-named bare metal cloud services as OCI.  

Monday, May 7, 2018

Undeploy composite when soa-infra down

cd /<middleware home>/oracle_common/common/bin
./wlst.sh
wls:/offline> connect()
Please enter your username :ohsadmin
Please enter your password :
Please enter your server URL [t3://localhost:7001] :t3://xyz.143.10.42:7001
Connecting to t3://xyz.143.10.42:7001 with userid weblogic ...
deploy('ShareSoaInfraPartition','/tmp/max/ShareSoaInfraPartition.ear',upload='true')

once done

exportMetadata(application='ShareSoaInfraPartition',server='AdminServer',toLocation='<Domain_Home>',docs='/deployed-composites/deployed-composites.xml')

this created /<DOMAIN_HOME>/deployed-composites directory

edited the deployed-composites.xml
removed the ElectronicInvoice  entries from the xml.

then executed

importMetadata(application='ShareSoaInfraPartition',server='AdminServer',fromLocation='<Domain_home>',docs='/deployed-composites/deployed-composites.xml')

restarted soa_server

Sunday, March 25, 2018

Linux meltdown patching

So , we started the meltdown patching like entire world is doing . Its been long weekends working on this. From an application perspective , its just stop/start , plus issues post patch.Luckily we havent had that many of those.Except one SOA 11.1.1.6 with server failover setup.

Issue: The control node manager has post server migration fails. Node manager unable to start vip.

<WARNING> <Unknown interface eth1>
Mar 12, 2018 7:45:19 PM weblogic.nodemanager.server.NMHelper$Drainer run
WARNING: Unknown interface eth1
<Mar 12, 2018 7:45:19 PM> <WARNING> <Cannot remove 100.91.192.xyz - It is not online at 'eth1' or any of its sub-interfaces>
Mar 12, 2018 7:45:19 PM weblogic.nodemanager.server.NMHelper$Drainer run
WARNING: Cannot remove 100.91.192.xyz - It is not online at 'eth1' or any of its sub-interfaces
<Mar 12, 2018 7:45:19 PM> <Warning> <Exception while executing 'PostStop' ExecutableCallbacks>
java.io.IOException: Exception while executing 'PostStop' ExecutableCallbacks
at weblogic.nodemanager.server.WLSProcess$MultiExecuteCallbackHook.execute(WLSProcess.java:297)
at weblogic.nodemanager.server.WLSProcess.executePostStopHooks(WLSProcess.java:246)
at weblogic.nodemanager.server.WLSProcess.startProcess(WLSProcess.java:197)
at weblogic.nodemanager.server.AbstractServerMonitor.startWLSProcess_inner(AbstractServerMonitor.java:419)
at weblogic.nodemanager.server.AbstractServerMonitor.startWLSProcess(AbstractServerMonitor.java:358)
at weblogic.nodemanager.server.AbstractServerMonitor.start(AbstractServerMonitor.java:103)

ava.io.IOException: Command '/test/soa_domain/bin/server_migration/wlsifconfig.sh -removeif eth1 100.91.192.xyz ' returned an unsuccessful exit code '1'. Check NM logs for script output.

at weblogic.nodemanager.system.DefaultNMPlugin$2.execute(DefaultNMPlugin.java:206)
at weblogic.nodemanager.server.WLSProcess$MultiExecuteCallbackHook.execute(WLSProcess.java:317)
at weblogic.nodemanager.server.WLSProcess$MultiExecuteCallbackHook.executeWithContinueOnFailure(WLSProcess.java:309)
at weblogic.nodemanager.server.WLSProcess$MultiExecuteCallbackHook.execute(WLSProcess.java:284)
at weblogic.nodemanager.server.WLSProcess.executePostStopHooks(WLSProcess.java:246)

Workaround:
Start services from startmanagedweblogic.sh instead of node manager.

Its a oracle bug, a patch has to be applied for 10.3.6 and 12.1.3 weblogic. This we noticed on cluster nodes with server migration setup.For patch check with oracle support.

Friday, March 16, 2018

Algorithm constraints check failed on signature algorithm: MD5withRSA

Issue: We did a PMP on OAM instance. As part of that  we updated JDK. We copied the customer certs. But when we started OAM we got, (OAM version 11.1.2.2, can occur on any version)

 <Mar 16, 2018 5:57:07 AM UTC> <Warning> <Coherence> <BEA-000000> <2018-03-16 05:57:07.204/380.379 Oracle Coherence GE 3.7.1.1 <Warning> (thread=PacketListener1, member=n/a): TcpDatagramSocket{bind=ServerSocket[addr=/141.143.130.9,localport=9097]}, exception regarding peer vmohsisos016.oracleoutsourcing.com/100.143.130.9:9095, General SSLEngine problem; Certificates do not conform to algorithm constraints; Algorithm constraints check failed on signature algorithm: MD5withRSA>
<Mar 16, 2018 5:57:07 AM UTC> <Error> <Coherence> <BEA-000000> <2018-03-16 05:57:07.505/380.680 Oracle Coherence GE 3.7.1.1 <Error> (thread=Configuration Store Observer, member=n/a): Error while starting cluster: com.tangosol.net.RequestTimeoutException: Timeout during service start: ServiceInfo(Id=0, Name=Cluster, Type=Cluster
  MemberSet=MasterMemberSet(
    ThisMember=null
    OldestMember=null
    ActualMemberSet=MemberSet(Size=0
      )
    MemberId|ServiceVersion|ServiceJoined|MemberState
    RecycleMillis=240000
    RecycleSet=MemberSet(Size=0
      )
    )
)
      

Solution: in the JDK_HOME/jre/lib/java.security , we need to make some changed.

-bash-3.2$ diff java.security_16032018 java.security
479,480c479,480
< jdk.certpath.disabledAlgorithms=MD2, MD5, SHA1 jdkCA & usage TLSServer, \
<     RSA keySize < 1024, DSA keySize < 1024, EC keySize < 224
---
> jdk.certpath.disabledAlgorithms=MD2, SHA1 jdkCA & usage TLSServer, \
>     RSA keySize < 512, DSA keySize < 1024, EC keySize < 224
523c523
< jdk.jar.disabledAlgorithms=MD2, MD5, RSA keySize < 1024, DSA keySize < 1024
---
> jdk.jar.disabledAlgorithms=MD2,RSA keySize < 512, DSA keySize < 1024
555c555
< jdk.tls.disabledAlgorithms=SSLv3, MD5withRSA, DH keySize < 1024, \
---
> jdk.tls.disabledAlgorithms=SSLv3, DH keySize < 1024, \
-bash-3.2$