opatch and Oracle Home inventory

There are two inventory locations. One is the central inventory defined by file /etc/oraInst.loc (in Linux host). For each ORACLE HOME on the server, there is a local inventory at $ORACLE_HOME/inventory.

opatch lists the information first when applying a patch:

Oracle Home                : /path/to/apps/tech_st/10.1.2
Oracle Home Inventory  : $ORACLE_HOME/inventory
Central Inventory           : /path/to/oraEbsInventory
   from                          : /etc/oraInst.loc

Because the central inventory /path/to/oraEbsInventory may be shared by multiple ORACLE HOMEs on the server, it is more vulnerable to error or corruption. If opatch throws out some error, troubleshoot central inventory first.

If there is a problem with local inventory, it may turn out you have to re-install it. So, do not try to change anything without backing it up.

When I apply a small patch to 10.1.2 ORACLE HOME, it failed with below messages in log file $ORACLE_HOME/.patch_storage/8551790/xxx.log. "opatch lsinventory" also did not show the patch was applied.

Patch 8551790 has been applied successfully
... ... ...

verifying patch
  verifying that patch ID is in the Oracle Home inventory
INVENTORY PROBLEM: Patch 8551790 is not present in Oracle Home inventory.
  Verifying copy files.
Comparing "/path/to/8551790/files/procbuilder/lib/cus_procbuilder.mk" and "$ORACLE_HOME/procbuilder/lib/cus_procbuilder.mk"
... ... ...
OPATCH_JAVA_ERROR: Patch was not successfully applied.

Verification of the patch failed.
ERROR: OPatch failed as verification of the patch failed.

The real message is "INVENTORY PROBLEM ...".  Doc ID 438067.1 (Updating the Inventory Is Failing During Patching) provides a solution to it.

Fix:
I renamed the existing Contents folder and created an empty folder Contents under oraEbsInventory. After I ran "opatch apply" again, the issue was fixed.

The same fix may address error "Inventory check failed: Patch ID is NOT registered in Oracle Home inventory." or "Verification of patch failed: Patch is not found in the Inventory." as well.

That Oracle document also says "opatch lsinventory -detail" accesses the inventory stored in text format in central inventory while "opatch lsinventory" gets inventory info from binary inventory (under oraEbsInventory/Contents ?).

R12.1 log locations and log collection

To troubleshoot a technology stack (Web issue or Forms error),  Oracle Support asks a lot of logs, before they take a look on the service request (SR) or even they may not know what they are doing.  Recently I followed Doc ID 1942889.1 (SRDC - Data Collection Request for EBS: IAS) to collect diagnostic data/logs.

1. follow Doc ID 2026081.1 to run 5 sql scripts to generate 5 .txt files and save them to /tmp.

2. run below line to zip all needed files
$ more zip1942889_1.sh
zip -r /tmp/$TWO_TASK'_'`uname -n`_`date +%m%d%y`_Techstack_info.zip \
/tmp/apps_version_info.txt \
/tmp/fndnodes_oss.txt \
/tmp/web_urls.txt \
/tmp/atg_ver_patches.txt \
/tmp/login_profiles.txt \
$OA_HTML/bin/appsweb* \
$CONTEXT_FILE \
$FND_SECURE/* \
$ORA_CONFIG_HOME/10.1.3/Apache/* \
$ORA_CONFIG_HOME/10.1.3/j2ee/* \
$ORA_CONFIG_HOME/10.1.3/javacache/* \
$ORA_CONFIG_HOME/10.1.3/opmn/*

3. run below line from Doc  (Doc ID 601736.1 to get techstack Component Versions (Forms, Http Server, JDK, Framework)

$ cd $FND_TOP/patch/115/bin   
$ $ADPERLPRG $FND_TOP/patch/115/bin/TXKScript.pl \
-script=$FND_TOP/patch/115/bin/txkInventory.pl -txktop=$APPLTMP \
-contextfile=$CONTEXT_FILE \
-appspass=appsPWD \
-outfile=/tmp/Report_App_Inventory.html
*** ALL THE FOLLOWING FILES ARE REQUIRED FOR RESOLVING RUNTIME ERRORS
*** STDOUT   = /$LOG_HOME/appl/rgf/TXK/txkInventory_Thu_Jan_14_10_21_05_2016_stdout.log
Reportfile /tmp/Report_App_Inventory.html generated successfully.

Summary on R12.1 log locations. Note env variable $LOG_HOME and $INST_TOP/logs are the same.

-----------------------------------------------------------------------------
-- Startup/Shutdown services (by $ADMIN_SCRIPTS_HOME scripts) log files:

$LOG_HOME/appl/admin/log

----------------------------------------------------------------------------
-- Apache, OC4J and OPMN (see Doc ID 454178.1): runtime

$LOG_HOME/ora/10.1.3/Apache

$LOG_HOME/ora/10.1.3/opmn
$LOG_HOME/ora/10.1.3/j2ee (each sub-folder /forms and /oacore has files application.log and server.log)

 $LOG_HOME/ora/10.1.3/opmn/opmn.log
 $LOG_HOME/ora/10.1.3/opmn/oacore_default_group_X/oacorestd.err   (X depends on s_oacore_nprocs)
 $LOG_HOME/ora/10.1.3/opmn/default_group~oacore~default_group~X.log
 $LOG_HOME/ora/10.1.3/j2ee/oacore/oacore_default_group_X/application.log
 $LOG_HOME/ora/10.1.3/j2ee/forms/forms_default_group_X/application.log    X depends on s_forms_nprocs

$APPLRGF/javacache.log

- user's DB connection log (and trace if enabled)
 $LOG_HOME/ora/10.1.2/network

--------------------------------------------------------------------------
-- Concurrent program log & out

$APPLCSF/log
$APPLCSF/out
($INST_TOP/logs/appl/conc/log  <== default)

SQL> SELECT logfile_name, outfile_name, outfile_node_name, last_update_date
FROM apps.FND_CONCURRENT_REQUESTS
WHERE REQUEST_ID = &requestID;

- temporary directories
 $APPLPTMP  
 $APPLTMP     <== temporary files for EBS forms.
 $INST_TOP/logs/ora/10.1.2/forms  ($FORMS_TRACE_DR)  <== holds forms runtime files
- Forms Runtime Diagnostics (FRD) tracing AND logging files (Doc ID 438652.1)

----------------------------------------------------------------------------
-- Autoconfig log file:
--
Apps:
$INST_TOP/admin/log/MMDDHHMM/adconfig.log
($INST_TOP/admin/out/MMDDHHMM has run-time files)

DB tier:
$ORACLE_HOME/appsutil/log/$CONTEXT_NAME/<MMDDHHMM>/

----------------------------------------------------------------------------
-- adadmin, adpatch and adctrl

$APPL_TOP/admin/${TWO_TASK}/log

----------------------------------------------------------------------------
-- Patching related log files

i)   $APPL_TOP/admin/${TWO_TASK}/log/ <== Application Tier -- adpatch log
ii)  $ORACLE_HOME/.patch_storage            <== Forms (Forms & Reports 10.1.2/Developer) patch
iii) $IAS_ORACLE_HOME/.patch_storage    <== Apache (Web Server) patch

----------------------------------------------------------------------------
-- Clone logs:

Pre-clone log files in source instance

Apps: => $INST_TOP/admin/log/StageAppsTier_MMDDHHMM.log
DB:    => $ORACLE_HOME/appsutil/log/$CONTEXT_NAME/StageDBTier_MMDDHHMM.log

Clone log files in target instance

Apps : => $INST_TOP/admin/log/ApplyAppsTier_<time>.log
DB:     => $ORACLE_HOME/appsutil/log/$CONTEXT_NAME/ApplyDBTier_<time>.log

---------------------------------------------------------------------------
-- Output Post Processor (OPP) log from the Application

Getting OPP Log from the application itself
a. System Administrator > Concurrent > Manager > Administer
b. Search for 'Output Post Processor'
c. Click the 'Processes' button
d. Click the Manager Log button. This will open the 'OPP'