T2P ERROR - CLONE-20275

We were running t2p for a customer for migrating to OCI. Even when /tmp has space we were running into issues.

Command for T2P:
sh $MW_HOME/oracle_common/bin/copyBinary.sh \
-javaHome /oracle/fmw/jrockit-jdk1.6.0_211 \
-archiveLoc /oracle/ias/ocimig/my_copy.jar \
-sourceMWHomeLoc /oracle/fmw/product/111 \
-invPtrLoc /oracle/fmw/product/111/oracle_common/oraInst.loc
-logDirLoc /oracle/ias/tmp/logs \
-silent false \
-ignoreDiskWarning false

Error:

INFO : Dec 6, 2019 04:42:56 - CLONE-21166   Addition of the Middleware Home /pbmboo/fmw/product/111 to the archive started ...
SEVERE : Dec 6, 2019 04:43:11 - ERROR - CLONE-20275   Insufficient space to create /tmp/CLONINGCLIENT-8961367960413781108.
SEVERE : Dec 6, 2019 04:43:11 - CAUSE - CLONE-20275   Minimum required space was at least "4,318" MB, but only "4,177" MB was available for use.
SEVERE : Dec 6, 2019 04:43:11 - ACTION - CLONE-20275   Make sure that the minimum required space is available for use.

There was quite a bit of space in /tmp.

Solution:
export T2P_JAVA_OPTIONS="-Djava.io.tmpdir=<dir>/tmp" 

Then rerun the clone.If you inspect the <dir>/tmp directory while the script is running you'll see that a CLONINGCLIENT202653551162129417 file is created.
That file will be removed when the copyBinary.sh script has ended.
Now, the copyBinary.sh script should end without errors.

4:33 PM

JCS backup failing

Another case of failed backup on JCS,

Oct 2, 2019 12:17:29 AM UTC Activity Submitted
Oct 2, 2019 12:17:29 AM UTC Activity Started
Oct 2, 2019 12:17:47 AM UTC PSM-BKP-50095: Operation execution was abborted or it has crashed possibly due to script execution error or lack of space for capturing script output. If the problem persists, contact Oracle Support Services.
Oct 2, 2019 12:19:58 AM UTC PSM-BKP-50106: Operation failed and will be retried.
Oct 2, 2019 12:25:18 AM UTC PSM-BKP-50095: Operation execution was abborted or it has crashed possibly due to script execution error or lack of space for capturing script output. If the problem persists, contact Oracle Support Services.
Oct 2, 2019 12:27:29 AM UTC PSM-BKP-50106: Operation failed and will be retried.
Oct 2, 2019 12:32:48 AM UTC PSM-BKP-50095: Operation execution was abborted or it has crashed possibly due to script execution error or lack of space for capturing script output. If the problem persists, contact Oracle Support Services.
Oct 2, 2019 12:32:48 AM UTC Activity Ended

The reason was file system was full!

Dbaas credential update failed

So, we got a new customer and some consultant used his email/password to setup dbaas backup. So, we got a standard backup user created. When i tried updating from IDD console i got,

Sep 26, 2019 6:21:36 AM UTC    Activity Submitted

Sep 26, 2019 6:21:41 AM UTC    Activity Started
Sep 26, 2019 6:21:41 AM UTC    Resetting service credentials, updating the cloud storage password for Service Type DBaaS and Service id : ADB-DATAMART-TEST-DBCS
Sep 26, 2019 6:21:41 AM UTC    Initialization of Backup Config update operation.
Sep 26, 2019 6:21:41 AM UTC    Submitting update config command for vm instance : [ADB-DATAMART-TEST-DBCS]
Sep 26, 2019 6:21:42 AM UTC    DBaaS Tools version in Service Instance [ADB-DATAMART-TEST-DBCS] is an old version [18.2.3.1.0], please consider updating it.
Sep 26, 2019 6:21:44 AM UTC    Backup Config update operation in progress.
Sep 26, 2019 6:36:56 AM UTC    Backup Config update failed due to timeout within [15] minutes.
Sep 26, 2019 6:36:59 AM UTC    Failed to reset cloud storage credential, try again! Error message: Failed cloud storage credential reset for DBaaS service : ADB-DATAMART-TEST-DBCS
Sep 26, 2019 6:36:59 AM UTC    Activity Ended

I used this to update,
https://docs.oracle.com/en/cloud/paas/database-dbaas-cloud/csdbi/update-storage-container-password.html

# rpm -qa|grep -i dbaastools
dbaastools-version_number-release_number

# dbaascli patch tools list

# dbaascli patch tools apply --patchid LATEST 

# /var/opt/oracle/ocde/assistants/bkup/bkup

WSM-00254 : The message is expired

So my weekend started with a major escalation. Customer said he cant process payload and it was working prior to the CPU patching of system. He was using TLS 1.0 (supressed by new jdk), We had already disabed algorithms in java.security.

Issue:<OSB 12.2.1.2>
when try to consume a osb service:https://abc.oracle.com/xyxxxx/Customers/ContactsSvc/v2?wsdl

We are getting a 500 server error at any pharmacy,

Traige: To look for actual issue , we found that this was connecting to a SAAS application. So, i looked at public OTD logs considering it must be on the internet. I found http 200(success ) in the logs. Then i grep'ed for that string in SOA logs.

Errors:
oracle.wsm.security.SecurityException: WSM-00254 : The message is expired. Check the timestamp element in the message.
The current server time is "September 7, 2019 11:30:07 PM CDT", incoming message creation time is "September 7, 2019 5:30:00 PM CDT", configured agent expiry is 300 seconds, incoming message expiry is 120 seconds, effective message expiry (minimum of agent expiry and incoming message expiry) is 120 seconds, configured clock skew is 360 seconds. The acceptable time range is "September 7, 2019 5:24:00 PM CDT" to "September 7, 2019 5:38:00 PM CDT". The incoming message is outside the valid range as allowed by clock skew and expiry times.

Solution:
This is just a temp fix.
Set the parameters "clock skew" and "Message expiration time" as follows to high value

Enterprise Manager -> expand "WebLogic Domain" in the left pane -> right click on the domain name -> web services --> WSM Domain Configuration --> Message Security tab -> Under "Security Settings", set below values.

- Clock Skew -> 23400000msecs (6.5hrs)
- Message Expiration Time -> 23400000msecs (6.5hrs)

OCI: APEX does not show up in DBA_REGISTRY

We had a case where moving a customer from OCIC to OCI with apex caused issues. We could not find dba_registry on OCI.

Follow this,
https://cloud.oracle.com/iaas/whitepapers/oracle_apex_on_oci_database.pdf

[gsdsiConnect] ORA-28040, ORA-28040: No matching authentication protocol

We had upgraded the OID database from 12.1 to 12.2 , post that when starting OID we got the below error.

[2019-07-29T04:24:20.342724-06:00] [OID] [NOTIFICATION:16] [] [OIDMON] [host: aXXXXXXX] [pid: 40240] [tid: 0] Guardian: [gsdsiConnect] ORA-28040, ORA-28040: No matching authentication protocol
[2019-07-29T04:24:20.343169-06:00] [OID] [NOTIFICATION:16] [] [OIDMON] [host: aXXXXXXX] [pid: 40240] [tid: 0] Guardian: [oidmon]: Unable to connect to database,will retry again after 10 sec

So, this is known error when we move from 11g to 12c database. Then we just have to make sec_case_sensitive_logon   parameter to false and it used to work. But in this case, the parameter didnot help. So, i created a sqlnet.ora updated below,

SQLNET.ALLOWED_LOGON_VERSION=11

This fixed the issue and OID was started.

DIP issue after JDK update

There are issues like DIP shows as down in em after JDK update. The below is the fix for this.

setDomainEnv.sh files:
    EXTRA_JAVA_PROPERTIES="${EXTRA_JAVA_PROPERTIES} -Dcom.sun.jndi.ldap.object.disableEndpointIdentification=true"
    export EXTRA_JAVA_PROPERTIES

Importing SSL certificate in SOA 11g/12c

This article will describe the method to import certificate in SOA 11g and 12c. In 11g its stored in java key tool, while in 12c it is in OPSS (i.e database).Below are applicable for both 11g and 12c.

Steps for 11g:
 In the weblogic console in SOA/OSB managed server check for keystore tab.You will see that demo identity and demo trust will be enabled.Now login to linux server where SOA managed server is running.
ps -ef|grep -i SSL
You will find a process like 

              “-Djavax.net.ssl.trustStore=/app/fmw/111/wlserver_10.3/server/lib/DemoTrust.jks”

Take a backup of this DemoTrust.jks file.

keytool -import -trustcacerts -alias <Alias Name> -file <Certificate File > -keystore <Trust Store Location> -storepass <passwd>

The services will need restart after this.

Steps for 12c:

As seen 12c , the OPSS based keystore is used.Below are the steps.
- Remove Djavax.net.ssl.trustStore=${WL_HOME}/server/lib/DemoTrust.jks from setDomainEnv.sh
The follow below steps.

1. Log in to Fusion Middleware Control (EM).

2. From the navigation pane, locate the domain i.e "SOA Domain"
3. Navigate to Security, then Keystore. The Keystore page appears.
4. Expand the stripe in which the keystore resides and  Select the row corresponding to the keystore. For this case system -> trust
We will use Trustore to place the certificate to call the external SSL partner link.
5. Click Manage.
6. If the keystore is password-protected, you are prompted for a password. Enter the keystore password and click OK.
7. The Manage Certificates page appears. Click Import.
8. The Import Certificate dialog appears.
9. Select the certificate type, either Certificate or Trusted Certificate, from the drop-down. For this case use "Trusted Certificate"
10. Provide an alias i.e "testTrust"
11. Specify the certificate source. If using the Paste option, copy and paste the certificate directly into the text box. If using the Select a file option, click Browse to select the file from the operating system.
12. Click OK. The imported certificate or trusted certificate appears in the list of certificates.
13. Click OK.
14. Bounce the managed server.
 

OIM timezone issue

We had a case where OIM was ignoring the timezone set and adding EST .Even though the JDK , OS and application timezones were set correctly.

Its a bug and Oracle has released patches for it.I would suggest raise a support SR for details.But workaround is as below,

- Add to startweblogic.sh
- Add  the TZ parameter like -Duser.timezone=Asia/Calcutta

if [ "${WLS_REDIRECT_LOG}" = "" ] ; then
        echo "Starting WLS with line:"
        echo "${JAVA_HOME}/bin/java ${JAVA_VM} ${MEM_ARGS} ${LAUNCH_ARGS} -Dweblogic.Name=${SERVER_NAME} -Djava.security.policy=${WLS_POLICY_FILE} ${JAVA_OPTIONS} ${PROXY_SETTINGS} ${SERVER_CLASS}"
        ${JAVA_HOME}/bin/java ${JAVA_VM} ${MEM_ARGS} ${LAUNCH_ARGS} -Dweblogic.Name=${SERVER_NAME} -Duser.timezone=Asia/Calcutta -Djava.security.policy=${WLS_POLICY_FILE} ${JAVA_OPTIONS} ${PROXY_SETTINGS} ${SERVER_CLASS}
else
        echo "Redirecting output from WLS window to ${WLS_REDIRECT_LOG}"

        ${JAVA_HOME}/bin/java ${JAVA_VM} ${MEM_ARGS} ${LAUNCH_ARGS} -Dweblogic.Name=${SERVER_NAME} -Duser.timezone=Asia/Calcutta -Djava.security.policy=${WLS_POLICY_FILE} ${JAVA_OPTIONS} ${PROXY_SETTINGS} ${SERVER_CLASS}  >"${WLS_REDIRECT_LOG}" 2>&1
fi

Note that location is before SERVER_NAME

Failed to create RCU schemas when creating JCS

When i was creating a JCS instance in past. I had faced an issues where it failed at schema creation on DBCS.

Failed to create RCU schemas. Verify Database connectivity to XXXXX:1521:PDB1.YYYYY.oraclecloud.internal with sys -dbRole sysdba and make sure the Database has enough available space>

First thing is to check if database and listener are up. If they are then check the security rules.

- In the service console go to the network tab.
-  Search <servicename>/db_1/ora_p2_dblistener using the word "ora_p2_dblistener"
- Select update
- Check the status if its disabled , enable it.

Now you should be able to proceed to create the JCS instance.

The backup failed due to an object store connectivity issue

These days i have had exposure to OCI database. Its similar to OCIC dbaas which i had worked on from sometime. Though had stopped due to some organizational restructure. But i had faced this for one of the early customer i provisioned. As the infra team was also trying to understand OCI when i was launched. I like OCI as its quite robust and has many features compatible to AWS.  The above error is due to object storage connectivity error.



        We did refer the pre-reqs,

https://docs.cloud.oracle.com/iaas/Content/Database/Tasks/backingupOS.htm#Prerequi



However, we could either create a service gateway or a internet gateway and create appropriate routes to the object store.Basically , service gateway is created when you dont want traffic to go over the internet.