OMCAGNT-2024]: ORACLE_HOSTNAME[oc-abc-1xyz-ph-zz.compute.oraclecloud.com] cannot be evaluated to Fully Qualified Domain Name (FQDN)

So a different team got this error while installing OMC agent.

Solution :
IGNORE_VALIDATIONS=true when used while installing helped fixing the issue.

T2P ERROR - CLONE-20275

We were running t2p for a customer for migrating to OCI. Even when /tmp has space we were running into issues.

Command for T2P:
sh $MW_HOME/oracle_common/bin/copyBinary.sh \
-javaHome /oracle/fmw/jrockit-jdk1.6.0_211 \
-archiveLoc /oracle/ias/ocimig/my_copy.jar \
-sourceMWHomeLoc /oracle/fmw/product/111 \
-invPtrLoc /oracle/fmw/product/111/oracle_common/oraInst.loc
-logDirLoc /oracle/ias/tmp/logs \
-silent false \
-ignoreDiskWarning false

Error:

INFO : Dec 6, 2019 04:42:56 - CLONE-21166   Addition of the Middleware Home /pbmboo/fmw/product/111 to the archive started ...
SEVERE : Dec 6, 2019 04:43:11 - ERROR - CLONE-20275   Insufficient space to create /tmp/CLONINGCLIENT-8961367960413781108.
SEVERE : Dec 6, 2019 04:43:11 - CAUSE - CLONE-20275   Minimum required space was at least "4,318" MB, but only "4,177" MB was available for use.
SEVERE : Dec 6, 2019 04:43:11 - ACTION - CLONE-20275   Make sure that the minimum required space is available for use.

There was quite a bit of space in /tmp.

Solution:
export T2P_JAVA_OPTIONS="-Djava.io.tmpdir=<dir>/tmp" 

Then rerun the clone.If you inspect the <dir>/tmp directory while the script is running you'll see that a CLONINGCLIENT202653551162129417 file is created.
That file will be removed when the copyBinary.sh script has ended.
Now, the copyBinary.sh script should end without errors.

4:33 PM

JCS backup failing

Another case of failed backup on JCS,

Oct 2, 2019 12:17:29 AM UTC Activity Submitted
Oct 2, 2019 12:17:29 AM UTC Activity Started
Oct 2, 2019 12:17:47 AM UTC PSM-BKP-50095: Operation execution was abborted or it has crashed possibly due to script execution error or lack of space for capturing script output. If the problem persists, contact Oracle Support Services.
Oct 2, 2019 12:19:58 AM UTC PSM-BKP-50106: Operation failed and will be retried.
Oct 2, 2019 12:25:18 AM UTC PSM-BKP-50095: Operation execution was abborted or it has crashed possibly due to script execution error or lack of space for capturing script output. If the problem persists, contact Oracle Support Services.
Oct 2, 2019 12:27:29 AM UTC PSM-BKP-50106: Operation failed and will be retried.
Oct 2, 2019 12:32:48 AM UTC PSM-BKP-50095: Operation execution was abborted or it has crashed possibly due to script execution error or lack of space for capturing script output. If the problem persists, contact Oracle Support Services.
Oct 2, 2019 12:32:48 AM UTC Activity Ended

The reason was file system was full!

Dbaas credential update failed

So, we got a new customer and some consultant used his email/password to setup dbaas backup. So, we got a standard backup user created. When i tried updating from IDD console i got,

Sep 26, 2019 6:21:36 AM UTC    Activity Submitted

Sep 26, 2019 6:21:41 AM UTC    Activity Started
Sep 26, 2019 6:21:41 AM UTC    Resetting service credentials, updating the cloud storage password for Service Type DBaaS and Service id : ADB-DATAMART-TEST-DBCS
Sep 26, 2019 6:21:41 AM UTC    Initialization of Backup Config update operation.
Sep 26, 2019 6:21:41 AM UTC    Submitting update config command for vm instance : [ADB-DATAMART-TEST-DBCS]
Sep 26, 2019 6:21:42 AM UTC    DBaaS Tools version in Service Instance [ADB-DATAMART-TEST-DBCS] is an old version [18.2.3.1.0], please consider updating it.
Sep 26, 2019 6:21:44 AM UTC    Backup Config update operation in progress.
Sep 26, 2019 6:36:56 AM UTC    Backup Config update failed due to timeout within [15] minutes.
Sep 26, 2019 6:36:59 AM UTC    Failed to reset cloud storage credential, try again! Error message: Failed cloud storage credential reset for DBaaS service : ADB-DATAMART-TEST-DBCS
Sep 26, 2019 6:36:59 AM UTC    Activity Ended

I used this to update,
https://docs.oracle.com/en/cloud/paas/database-dbaas-cloud/csdbi/update-storage-container-password.html

# rpm -qa|grep -i dbaastools
dbaastools-version_number-release_number

# dbaascli patch tools list

# dbaascli patch tools apply --patchid LATEST 

# /var/opt/oracle/ocde/assistants/bkup/bkup

WSM-00254 : The message is expired

So my weekend started with a major escalation. Customer said he cant process payload and it was working prior to the CPU patching of system. He was using TLS 1.0 (supressed by new jdk), We had already disabed algorithms in java.security.

Issue:<OSB 12.2.1.2>
when try to consume a osb service:https://abc.oracle.com/xyxxxx/Customers/ContactsSvc/v2?wsdl

We are getting a 500 server error at any pharmacy,

Traige: To look for actual issue , we found that this was connecting to a SAAS application. So, i looked at public OTD logs considering it must be on the internet. I found http 200(success ) in the logs. Then i grep'ed for that string in SOA logs.

Errors:
oracle.wsm.security.SecurityException: WSM-00254 : The message is expired. Check the timestamp element in the message.
The current server time is "September 7, 2019 11:30:07 PM CDT", incoming message creation time is "September 7, 2019 5:30:00 PM CDT", configured agent expiry is 300 seconds, incoming message expiry is 120 seconds, effective message expiry (minimum of agent expiry and incoming message expiry) is 120 seconds, configured clock skew is 360 seconds. The acceptable time range is "September 7, 2019 5:24:00 PM CDT" to "September 7, 2019 5:38:00 PM CDT". The incoming message is outside the valid range as allowed by clock skew and expiry times.

Solution:
This is just a temp fix.
Set the parameters "clock skew" and "Message expiration time" as follows to high value

Enterprise Manager -> expand "WebLogic Domain" in the left pane -> right click on the domain name -> web services --> WSM Domain Configuration --> Message Security tab -> Under "Security Settings", set below values.

- Clock Skew -> 23400000msecs (6.5hrs)
- Message Expiration Time -> 23400000msecs (6.5hrs)