ERP on DB: Rebuild R12.2 Central Inventory

- The problem: adop failures on a server which has multiple EBS instances

ADOP reported unexpected errors on an UAT instance. "adop -validate" also failed on running script $AD_TOP/patch/115/bin/txkADOPEvalSrvStatus.pl

AdopValidate failed or are incomplete on node(s): Node2name

[UNEXPECTED]Unable to continue processing on other available nodes: Node1name

[UNEXPECTED]Error running "AdopValidate" on node(s): Node2name.

[UNEXPECTED]Remote action failed.

Seems the problem was on the 2nd node/server Node2name. Logs on Node2name gave more details on the error:

$ cd $NE_BASE/EBSapps/log/adop/4/20220425_174046/validate/Node2name

$ egrep -i 'erro|fail' *.*

AdopValidate.log: [PROCEDURE] [START 2022/04/25 17:42:49] Validating failed hotpatch cycle

AdopValidate.log: [PROCEDURE] [END 2022/04/25 17:42:49] Validating failed hotpatch cycle

AdopValidate.log: [UNEXPECTED]Error calling TXK validations procedure

ValidationResults.log: Validate failed hotpatch cycle

ValidationResults.log:SUCCESS: No failed hotpatch cycle present for this Node2name node.

ValidationResults.log: ERROR: The value of inventory_loc in $FMW_HOME/oracle_common/oraInst.loc is not the same as /u03/app/oraQA2Inventory present in other files

ValidationResults.log: ERROR: The value of inventory_loc in $FMW_HOME/webtier/oraInst.loc is not the same as /u03/app/oraQA2Inventory present in other files

ValidationResults.log: ERROR: The value of inventory_loc in $FMW_HOME/Oracle_EBS-app1/oraInst.loc is not the same as /u03/app/oraQA2Inventory present in other files

It indicates there was a confusion on the Central Inventory location. After checked pointer file /etc/oraInst.loc, I realized that when Node2name was added to the instance by clone, a new central inventory was not created for it and cloning used the inventory for QA2 which is a different instance on the host.

"opatch apply" also reported a similar error if file /etc/oraInst.loc does not point its Central Inventory location to the one for QA2.

The Oracle Home /u02/app/EBSUAT/fs1/EBSapps/10.1.2 is not registered with the Central Inventory. OPatch was not able to get details of the home from the inventory.

ERROR: OPatch failed because of Inventory problem.

I saw file /u03/app/oraQA2Inventory/ContentsXML/inventory.xml file has many entries for UAT instance. That confirms Inventory for ORACLE HOMEs was messed up.

- Re-build central inventory for ORACLE HOMEs

The only permanent fix is to follow Doc ID 1586607.1 (R12.2 How To Re-attach Oracle Homes To The Central Inventory) to build/create its Central Inventory in below steps:

$ $FMW_HOME/oracle_common/oui/bin/runInstaller -silent -attachHome -invPtrLoc /etc/oraInst.loc ORACLE_HOME="/u02/app/EBSUAT/fs1/FMW_Home/oracle_common" ORACLE_HOME_NAME="fs1_FMW_common" CLUSTER_NODES="{}"

Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB. Actual 14566 MB Passed

Preparing to launch Oracle Universal Installer from /tmp/OraInstall2022-04-26_12-58-12PM. Please wait ...

$ The inventory pointer is located at /etc/oraInst.loc

The inventory is located at /u02/app/oraEBSUATinventory

'AttachHome' was successful.

$ $FMW_HOME/oracle_common/oui/bin/runInstaller -silent -attachHome -invPtrLoc /etc/oraInst.loc ORACLE_HOME="/u02/app/EBSUAT/fs2/FMW_Home/oracle_common" ORACLE_HOME_NAME="fs2_FMW_common" CLUSTER_NODES="{}"

$ $FMW_HOME/oracle_common/oui/bin/runInstaller -silent -attachHome -invPtrLoc /etc/oraInst.loc ORACLE_HOME="/u02/app/EBSUAT/fs1/FMW_Home/webtier" ORACLE_HOME_NAME="fs1_FMW_webtier" CLUSTER_NODES="{}"

$ $FMW_HOME/oracle_common/oui/bin/runInstaller -silent -attachHome -invPtrLoc /etc/oraInst.loc ORACLE_HOME="/u02/app/EBSUAT/fs2/FMW_Home/webtier" ORACLE_HOME_NAME="fs2_FMW_webtier" CLUSTER_NODES="{}"

$ $FMW_HOME/oracle_common/oui/bin/runInstaller -silent -attachHome -invPtrLoc /etc/oraInst.loc ORACLE_HOME="/u02/app/EBSUAT/fs1/FMW_Home/Oracle_EBS-app1" ORACLE_HOME_NAME="fs1_FMW_app1" CLUSTER_NODES="{}"

$ $FMW_HOME/oracle_common/oui/bin/runInstaller -silent -attachHome -invPtrLoc /etc/oraInst.loc ORACLE_HOME="/u02/app/EBSUAT/fs2/FMW_Home/Oracle_EBS-app1" ORACLE_HOME_NAME="fs2_FMW_app1" CLUSTER_NODES="{}"

$ $RUN_BASE/EBSapps/10.1.2/oui/bin/runInstaller -silent -attachHome -invPtrLoc /etc/oraInst.loc ORACLE_HOME="/u02/app/EBSUAT/fs1/EBSapps/10.1.2" ORACLE_HOME_NAME="fs1_tool_EBSapps_10_1_2" CLUSTER_NODES="{}"

Starting Oracle Universal Installer...

No pre-requisite checks found in oraparam.ini, no system pre-requisite checks will be executed.

Preparing to launch Oracle Universal Installer from /tmp/OraInstall2022-04-26_01-13-13PM. Please wait ...

$ Java HotSpot(TM) Server VM warning: You have loaded library /tmp/OraInstall2022-04-26_01-13-13PM/oui/lib/linux/liboraInstaller.so which might have disabled stack guard. The VM will try to fix the stack guard now.

It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.

'AttachHome' was successful.

$ $PATCH_BASE/EBSapps/10.1.2/oui/bin/runInstaller -silent -attachHome -invPtrLoc /etc/oraInst.loc ORACLE_HOME="/u02/app/EBSUAT/fs2/EBSapps/10.1.2" ORACLE_HOME_NAME="fs2_tool_EBSapps_10_1_2" CLUSTER_NODES="{}"

$ cd /u02/apps/oraEBSUATinventory/ContentsXML

$ more inventory.xml

<?xml version="1.0" standalone="yes" ?>

<COMPOSITEHOME_LIST>

</COMPOSITEHOME_LIST>

<VERSION_INFO>

<SAVED_WITH>10.1.0.6.0</SAVED_WITH>

<MINIMUM_VER>2.1.0.6.0</MINIMUM_VER>

</VERSION_INFO>

<HOME_LIST>

<NODE_LIST>

</NODE_LIST>

</HOME>

<NODE_LIST>

</NODE_LIST>

</HOME>

</HOME_LIST>

</INVENTORY>

After Central Inventory was rebuilt, ADOP and opatch worked on the node using the new & correct Central Inventory.

ERP on DB

Friday, April 29, 2022

Rebuild R12.2 Central Inventory

No comments: