Attach 12.1.2 ORACLE HOME after clone

After an instance was cloned, 12.1.2 ORACLE HOME does not exist. I had to run a command to attach it.

$ echo $ORACLE_HOME
/u05/app/EBS/apps/tech_st/10.1.2

$ opatch lsinventory

Oracle Interim Patch Installer version 1.0.0.0.63
Copyright (c) 2009 Oracle Corporation. All Rights Reserved..

Oracle recommends you to use the latest OPatch version
and read the OPatch documentation available in the OPatch/docs
directory for usage. For information about the latest OPatch and
other support-related issues, refer to document ID 293369.1
available on My Oracle Support (https://myoraclesupport.oracle.com)

Oracle Home                : /u05/app/EBS/apps/tech_st/10.1.2
Oracle Home Inventory : /u05/app/EBS/apps/tech_st/10.1.2/inventory
Central Inventory          : /u05/app/oraEbsInventory
   from                          : /etc/oraInst.loc
OUI location                : /u05/app/EBS/apps/tech_st/10.1.2/oui
OUI shared library       : /u05/app/EBS/apps/tech_st/10.1.2/oui/lib/linux/liboraInstaller.so
Java location                : /u05/app/EBS/apps/tech_st/10.1.2/jre/1.4.2/bin/java
Log file location           : /u05/app/EBS/apps/tech_st/10.1.2/.patch_storage/<patch ID>/*.log

Creating log file "/u05/app/EBS/apps/tech_st/10.1.2/.patch_storage/LsInventory__06-26-2015_11-21-30.log"

Result:
LsInventory: OPatch Exception while accessing O2O

OPATCH_JAVA_ERROR  : An exception of type "OPatchException" has occurred:
OPatch Exception:  OUI found no such ORACLE_HOME set in the environment
  Can not get details for given Oracle Home
  An exception occurs
  null

There is no Interim Patch

OPATCH_JAVA_ERROR  : An exception of type "OPatchException" has occurred:
Can not get a list of inventory on this home.

ERROR: OPatch failed because of Inventory problem.

Check inventory.xml and also see the 10.1.2 ORACLE HOME has no entry in the file:
$ more /ebsu05/app/oraEbsInventory/ContentsXML/inventory.xml
<?xml version="1.0" standalone="yes" ?>
<!-- Copyright (c) 2002 Oracle Corporation. All rights Reserved -->
<!-- Do not modify the contents of this file by hand. -->
<INVENTORY>
<VERSION_INFO>
   <SAVED_WITH>10.1.0.6.0</SAVED_WITH>
   <MINIMUM_VER>2.1.0.6.0</MINIMUM_VER>
</VERSION_INFO>
<HOME_LIST>
<HOME NAME="EBS_WEBOH__u05_app_EBS_apps_tech_st_10_1_3" LOC="/u05/app/EBS/apps/tech_st/10.1.3" TYPE="O" IDX="1"/>
</HOME_LIST>
</INVENTORY>

Fix: run ./runInstaller to create ORACLE_HOME

First of all, make sure inventory location is correct:

$ vi /etc/oraInst.loc

inventory_loc=/u05/app/oraEbsInventory

$ cd $ORACLE_HOME/oui/bin

$ echo $ORACLE_HOME
 /u05/app/EBS/apps/tech_st/10.1.2
$ ./runInstaller -silent -attachHome ORACLE_HOME=/u05/app/EBS/apps/tech_st/10.1.2 ORACLE_HOME_NAME=EBS_1012_home

Starting Oracle Universal Installer...
No pre-requisite checks found in oraparam.ini, no system pre-requisite checks will be executed.
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2015-06-26_11-39-13AM. Please wait ...
$ 'AttachHome' was successful.
$

Now, check file /u05/app/oraEbsInventory/ContentsXML/inventory.xml again, it should have two entries. And also "$ opatch lsinventory" works!

Similarly steps can attach a 10.1.3 ORACLE_HOME in EBS R12.1:

$ cd $INST_TOP/ora/10.1.3
$ ls

$ . TWO_TASK_nodename.env
$ echo $ORACLE_HOME

/u05/app/EBS/apps/tech_st/10.1.3

$ cd $ORACLE_HOME/oui/bin
$ ./runInstaller -silent -attachHome ORACLE_HOME=/u05/app/EBS/apps/tech_st/10.1.3 ORACLE_HOME_NAME=EBS_WEBOH_10_1_3_HOME

Starting Oracle Universal Installer...
No pre-requisite checks found in oraparam.ini, no system pre-requisite checks will be executed.
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2020-11-07_11-42-25AM. Please wait ...
$ 'AttachHome' was successful.

NOTES:
1. if you get a error, like "Please wait ...The Java RunTime Environment was not found at /tmp/OraInstall2015-06-26_11-23-23AM/jre/bin/java", modify one line in oraparam.ini

  $ vi oraparam.ini   (under $ORACLE_HOME/oui  10.1.2 OH)
        # JRE_LOCATION=%JRE_LOCATION%
        JRE_LOCATION=../../jdk/jre                           #  <-- actual path is $ORACLE_HOME/jdk/jre

2. I tried " ouicli.pl " recommended in "How To Create, Update or Rebuild The Central Inventory For Oracle Applications E-Business Suite (Doc ID 742477.1)". But it failed to add one entry to inventory.xml and so does not attach 10.1.2 Oracle Home for unknown reason.

3. If need to detach an ORACLE_HOME from central inventory, see Don ID 417694.1:
./runInstaller -silent -detachHome ORACLE_HOME="<path_to_Oracle_Home>"
If above line fails, you "have to consider building a separate central inventory for this Oracle Home".

Enable TLS1.x in EBS R12.1 secured network

Transport Layer Security (TLS) is a newer cryptographic protocol. We started to work on enabling TLS1.x and disabling SSLv3 with EBS R12.1 (Doc ID 1937646.1) in February 2015. At that time, two requirements are necessary:
- OracleAS 10g must be 10.1.3.5 in EBS R12. If not, use Doc ID 454811.1 to upgrade it.
- Apply latest CPU patch. The latest EBS apps patch was October 2014 (Doc ID 1923805.1), which points 10GiAS 10.1.3.5 CPU patch to 16802901: CPUJUL2013 TRACKING BUG FOR APPLICATION SERVER 10.1.3.5 UNIX. Find and then download it through a link in document 1923805.1, not by patch number. Without it, EBS login page may not work after TLS1.x is enabled.

1. Verify Apache version:  $IAS_ORACLE_HOME/Apache/Apache/bin/httpd -v

Server version: Oracle-Application-Server-10g/10.1.3.5.0 Oracle-HTTP-Server
Server built:   Jul 21 2009 11:12:22

2. download / copy file p16802901_101350_LINUX.zip
    and unzip it to 1680291 folder.

3. vi /etc/oraInst.loc
    to make sure the central Oracle inventory is right.

4. shutdown apps services

5. $ cd $INST_TOP/ora/10.1.3
    then,  source the .env file.

6. make sure $ORACLE_HOME points to 10.1.3
    $ echo $ORACLE_HOME

7. $ "opatch lsinventory -detail"   to verify most patches are NOT installed
    $ cd 16802901
    $ opatch napply          Notes: it will rollback some patches.
    $ opatch lsinventory   to verify the patches are installed.

    $ sh remove_demo.sh   (optional)

    Notes: after opatch completed, "opatch lsinventory" shows 16802900, but not 16802901, was installed.

8. start a new Linux session to get EBS normal env variables.

9. modify  $FND_TOP/admin/template/ssl_conf_1013.tmp (per Doc ID 1937646.1)

SSLProtocol    all +TLSv1 -SSLv2 -SSLv3
SSLCipherSuite HIGH:MEDIUM:!aNULL:+SHA1:+MD5:+HIGH:+MEDIUM:!SSLv3:!SSLv2

10. run autoconfig
Then, verify file $ORA_CONFIG_HOME/10.1.3/Apache/Apache/conf/ssl.conf

11. start apps services

12. go to an EBS webpage, right click ==> Properties. The Properties box shall show "Connection    TLS 1.0 AES ...". The same message may be in Apache log as well.

13. If the site URL is https://siteName.domain.com:4439, below line will display certificate info if TLS1 is enabled:
$ openssl s_client -connect siteName.domain.com:4439 -tls1
$ openssl s_client -connect siteName.domain.com:4439 -tls1 | grep Cipher
(try "openssl s_client -connect siteName.domain.com:4439 -ssl3" to check if ssl3 is still enabled)

UPDATES in January 2016:
1) 1680291 is a very old patch. If patch 19568561 or 21845960 (CPU Oct2015 patch) was installed, 1680291 is not needed. CPU Oct2015 patch 21845960 works better (and it rollback patch 1680291 during installation). Doc ID 1937646.1 Change Log on 02-Nov-2015 shows patch 21845960 was also added to its "Release-Specific Requirements" section.
2) As of today, EBS12.1 only supports TLS1.0 and does not support TLS1.1 or TLS1.2.
3) If TLS1.0 is not enabled, EBS site may not work with Chrome and Firefox because they do not support SSLv3 now. 

Google Chrome 49.0.2623.108
This site can’t provide a secure connection
sitename.domain.com sent an invalid response.
ERR_SSL_PROTOCOL_ERROR

Firefox 44.0.2
Secure Connection Failed
An error occurred during a connection to sitename.domain.com. SSL peer rejected a handshake message for unacceptable content. (Error code: ssl_error_illegal_parameter_alert)

How to turn on Debug log on user's session

During troubleshooting on data issue, use below steps in Doc ID 1320736.1 to turn on FND debug log on user's forms session or concurrent requests.

Steps for On-Line processes
  1) From a System Administrator responsibility
       Navigation path: Profile -> System
           a) Set the following profile option at SITE level:
                  FND: Diagnostics -> Yes
           b) Set the following profile options at USER level:
                  FND: Debug Log Module -> %
                  FND: Debug Log Enabled -> Yes
                  FND: Debug Log Level -> Statement

  2) Run the following query to get the starting log sequence(seq1)
           --- ---
           SELECT MAX(log_sequence)
           FROM FND_LOG_MESSAGES;
           --- ---
  3) Go to the Payables (or any) responsibility
       Open the involved Forms or run the concurrent job and reproduce the error.

  4) Run the following query again to get the finishing log sequence(seq2)
           --- ---
           SELECT MAX(log_sequence)
           FROM FND_LOG_MESSAGES;
           --- ---
  5) Run the following query to generate the log file:
           --- ---
           SELECT module, message_text
           FROM fnd_log_messages
           WHERE log_sequence between &seq1 and &seq2
           ORDER BY log_sequence;
           --- ---
  6) Provide the results in an EXCEL spreadsheet with column headings for readability.

  Notes: If table fnd_log_messages becomes huge, it's safe to just truncate it.
              SQL> truncate table APPLSYS.FND_LOG_MESSAGES;

adoacorectl.sh: exiting with status 150

when the services are not brought down gracefully, or shutdown scripts got errors:

$ADMIN_SCRIPTS_HOME/adoafmctl.sh stop
script returned:
****************************************************
ERROR : Timed out( 100000 ): Interrupted Exception
You are running adoafmctl.sh version 120.8
Stopping OPMN managed OAFM OC4J instance ...
****************************************************

Executing service control script:
$ADMIN_SCRIPTS_HOME/adformsctl.sh stop
script returned:
****************************************************
ERROR : Timed out( 100000 ): Interrupted Exception
You are running adformsctl.sh  version 120.16.12010000.3
Stopping OPMN managed FORMS OC4J instance  ...
****************************************************

Executing service control script:
$ADMIN_SCRIPTS_HOME/adoacorectl.sh stop
script returned:
****************************************************
ERROR : Timed out( 100000 ): Interrupted Exception
You are running adoacorectl.sh version 120.13
Stopping OPMN managed OACORE OC4J instance ...

then in that case the lock file or other files will be still present and they will create problem when we try to bring up the environment. For example

$ADMIN_SCRIPTS_HOME/adoacorectl.sh start
script returned:
adoacorectl.sh: exiting with status 150
or
adoacorectl.sh: exiting with status 204

  - check startup log at $LOG_HOME/appl/admin/log/ adoacorectl.txt, adoafmctl.txt, adoaformsctl.txt
  - check log at $LOG_HOME/ora/10.1.3/opmn/opmn.log
  - check Apache logs. Location is "$LOG_HOME/ora/10.1.3/Apache". The latest ones error_log and access_log.

Fix:

stop all R12 processes

rm -rf $ORA_CONFIG_HOME/10.1.3/j2ee/oacore/persistence/*
rm -rf $ORA_CONFIG_HOME/10.1.3/j2ee/oafm/persistence/*
rm -rf $ORA_CONFIG_HOME/10.1.3/j2ee/forms/persistence/*

start all service again

Use TAR to backup a folder

Use right options of TAR to backup entire folder EBSDEV under /u01/app to a different location in Linux. It also shows all errors/warnings during the file copy.

$ which tar
/bin/tar
$ cd /u01/app       NOTE: do not have /u01/app in front of folder name EBSDEV below
$ tar -zcvf /u04/app/backup/m3d_DEV_022015_mt.tar.gz EBSDEV >> /u04/app/ebsde v_backup/backup022015.log

To avoid interruption and keep tar running on the background, use nohup in Linux:

$ nohup tar -zcvf /u04/app/backup/m3d_DEV_022015_mt.tar.gz EBSDEV &

It will create log file nohup.out in current directory and keep grabbing everything thru folder EBSDEV. Use ps to check the status.

$ ps -ef | grep backup

Potential errors:
tar: EBSDEV/apps/apps_st/appl/aagl/bin/.sasa.swp: Cannot open: Permission denied
tar: EBSDEV/apps/apps_st/appl/docs/install/.vi_v_pkb.sql.swp: Cannot open: Permission denied
tar: EBSDEV/apps/apps_st/appl/aafs/bin/fssd01_old: Cannot stat: No such file or directory
tar: EBSDEV/apps/apps_st/appl/aafs/bin/text.txt: Cannot stat: No such file or directory
tar: Error exit delayed from previous errors

To get detail on above errors, use "ls -al" to find that /u01/app/EBSDEV/apps/apps_st/appl/aagl/bin/.sasa.swp is owned by a different OS user, and find file link is missing /u01/app/EBSDEV/apps/apps_st/appl/aafs/bin/fssd01_old -> /bin/fndcpesr, etc.

NOTES:

1. If "-h" is used (e.g. tar -zhcvf  ...), it ditches the symlinks and copies/brings in the actual files they pointed to. Without '-h", it retains the soft links.
2. It can back up multiple folders to one file. It is useful in EBS R12.2.

$ cd /u04/app/EBSDEV

$ tar -zcvf /u04/app/backup/m3d_DEV_beforeSSL.tar.gz  fs1 fs2 fs_ne EBSapps.env >> /u04/app/backup/backup1005.log

To decompress/untar/restore backed-up folder EBSDEV in file /path/to/m3d_DEV_022015_mt.tar.gz (or copy it to a new location first):

$ mkdir temp
$ cd temp
$ tar vzxfp /path/to/m3d_DEV_022015_mt.tar.gz
It will create folder EBSDEV under current folder /temp/

The options stand for verbose, extract, pass thru gzip first, specify file, preserve permissions. "p" is important when trying to restore a directory.

Now, "find EBSDEV -type f | wc -l" shall match the counts on source and target folders. With "-type f", find does not count symlinks (symbolic link).

BTW, zip also can be used to backup a (smaller) folder:

$ zip -r flie_name.zip folder_name
"-r" is to get all subfolders.

gzip can be used to compress a big file (but it is slow):

$ ls -al java*.*

-rw-------  1 user group 5842435627 Jan 18 12:07 java_pid1583.hpro

$ gzip java_pid1583.hprof

 $ ls -al java*.*

-rw-------  1 user group 697002093 Jan 18 12:07 java_pid1583.hprof.gz

$ which gzip

/usr/bin/gzip