Sunday, August 30, 2015

R12.1 patching for Internet Explorer 11 (IE 11)

R12.1 is certified with Internet Explorer 11. But patches are necessary (Doc ID 389422.1).

1. Merge and apply 4 patches to R12.1.3:
18936791:R12.FWK.B.delta.4
17932167:R12.TXK.B.delta.3
19273341:R12.BNE.B.delta.4
19559960:R12.FRM.B.Delta.4
Notes: when CRM module is not used, applying CRM patch will create a lot invalid objects in DB)

2. Apply patch 16983427 to fix a bug on Payables after above 4 patches:
'ViewObject attribute is null' Error Trying To Submit Payment Process Request (Doc ID 1481246.1)

3. Clean some cache files:
$ cd $OA_HTML
$ rm -f cabo/images/cache/*
$ rm -f cabo/styles/cache/*

4. Update profile option "BNE Redirect Portal URL" to empty (Bug: Doc ID 1965891.1).

5. Users need to use some workarounds to make ADI Excel import/export work.

Sunday, August 23, 2015

EBS Reports errors as concurrent jobs

In R12, below command can check if Report Manager is running or not on CM node:

$ ps -ef | grep rwrun

But, it there is no report in Running status in concurrent manager, that command will return nothing.

Troubleshooting on Reports:
 
1. Reports finish, but give error at the end: "REP-57054: In-process job terminated:Finished successfully but output is voided". The behavior matches "Intermittent Oracle Reports REP-0069: REP-57054: Error (Doc ID 1237834.1)".

Workaround:
(1) create a file -  $FND_TOP/admin/template/custom/rwbuilder_conf_1012.tmp
 use the idea in Doc ID 1237834.1 to prevent Autoconfig from clobbering the changes.
(2) modify the file changing one line to <property name="cacheSize" value="50"/>
(3) run autoconfig
(4) verify $INST_TOP/ora/10.1.2/reports/conf/rwbuilder.conf have 50 as the cacheSize.
(5) schedule cron jobs to delete log files.
05 23 * * * /usr/bin/find $INST_TOP/logs/ora/10.1.2/reports/cache/*.txt -mtime +3 -exec rm -rf {} \;
05 05 * * * /usr/bin/find $INST_TOP/logs/ora/10.1.2/reports/cache/*.xml -mtime +7 -exec rm -rf {} \;
05 10 * * * /usr/bin/find $INST_TOP/logs/ora/10.1.2/reports/cache/*.txt -mtime +7 -exec rm -rf {} \;
(replace $INST_TOP with the real path)

2. After submit a CM job/report, the job stays there forever.

It looks like it has difficulties to hit the database. But the real fix is to delete file  $ORACLE_HOME/reports/conf/rwnetwork.conf. 
See "Concurrent Processing - R12 Concurrent Requests Run Forever, rwrun Errors with REP-50125 (Doc ID 737445.1)".

3.

Attach 12.1.2 ORACLE HOME after clone

After an instance was cloned, 12.1.2 ORACLE HOME does not exist. I had to run a command to attach it.

$ echo $ORACLE_HOME
/u05/app/EBS/apps/tech_st/10.1.2

$ opatch lsinventory

Oracle Interim Patch Installer version 1.0.0.0.63
Copyright (c) 2009 Oracle Corporation. All Rights Reserved..

Oracle recommends you to use the latest OPatch version
and read the OPatch documentation available in the OPatch/docs
directory for usage. For information about the latest OPatch and
other support-related issues, refer to document ID 293369.1
available on My Oracle Support (https://myoraclesupport.oracle.com)

Oracle Home                : /u05/app/EBS/apps/tech_st/10.1.2
Oracle Home Inventory : /u05/app/EBS/apps/tech_st/10.1.2/inventory
Central Inventory          : /u05/app/oraEbsInventory
   from                          : /etc/oraInst.loc
OUI location                : /u05/app/EBS/apps/tech_st/10.1.2/oui
OUI shared library       : /u05/app/EBS/apps/tech_st/10.1.2/oui/lib/linux/liboraInstaller.so
Java location                : /u05/app/EBS/apps/tech_st/10.1.2/jre/1.4.2/bin/java
Log file location           : /u05/app/EBS/apps/tech_st/10.1.2/.patch_storage/<patch ID>/*.log

Creating log file "/u05/app/EBS/apps/tech_st/10.1.2/.patch_storage/LsInventory__06-26-2015_11-21-30.log"

Result:
LsInventory: OPatch Exception while accessing O2O

OPATCH_JAVA_ERROR  : An exception of type "OPatchException" has occurred:
OPatch Exception:  OUI found no such ORACLE_HOME set in the environment
  Can not get details for given Oracle Home
  An exception occurs
  null

There is no Interim Patch

OPATCH_JAVA_ERROR  : An exception of type "OPatchException" has occurred:
Can not get a list of inventory on this home.

ERROR: OPatch failed because of Inventory problem.

Check inventory.xml and also see the 10.1.2 ORACLE HOME has no entry in the file:
$ more /ebsu05/app/oraEbsInventory/ContentsXML/inventory.xml
<?xml version="1.0" standalone="yes" ?>
<!-- Copyright (c) 2002 Oracle Corporation. All rights Reserved -->
<!-- Do not modify the contents of this file by hand. -->
<INVENTORY>
<VERSION_INFO>
   <SAVED_WITH>10.1.0.6.0</SAVED_WITH>
   <MINIMUM_VER>2.1.0.6.0</MINIMUM_VER>
</VERSION_INFO>
<HOME_LIST>
<HOME NAME="EBS_WEBOH__u05_app_EBS_apps_tech_st_10_1_3" LOC="/u05/app/EBS/apps/tech_st/10.1.3" TYPE="O" IDX="1"/>
</HOME_LIST>
</INVENTORY>

Fix: run ./runInstaller to create ORACLE_HOME

First of all, make sure inventory location is correct:
$ vi /etc/oraInst.loc
inventory_loc=/u05/app/oraEbsInventory

$ cd $ORACLE_HOME/oui/bin
$ echo $ORACLE_HOME
 /u05/app/EBS/apps/tech_st/10.1.2
$ ./runInstaller -silent -attachHome ORACLE_HOME=/u05/app/EBS/apps/tech_st/10.1.2 ORACLE_HOME_NAME=EBS_1012_home

Starting Oracle Universal Installer...
No pre-requisite checks found in oraparam.ini, no system pre-requisite checks will be executed.
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2015-06-26_11-39-13AM. Please wait ...
$ 'AttachHome' was successful.
$

Now, check file /u05/app/oraEbsInventory/ContentsXML/inventory.xml again, it should have two entries. And also "$ opatch lsinventory" works!

Similarly steps can attach a 10.1.3 ORACLE_HOME in EBS R12.1:

$ cd $INST_TOP/ora/10.1.3
$ ls
$ . TWO_TASK_nodename.env
$ echo $ORACLE_HOME
/u05/app/EBS/apps/tech_st/10.1.3
$ cd $ORACLE_HOME/oui/bin
$ ./runInstaller -silent -attachHome ORACLE_HOME=/u05/app/EBS/apps/tech_st/10.1.3 ORACLE_HOME_NAME=EBS_WEBOH_10_1_3_HOME
Starting Oracle Universal Installer...
No pre-requisite checks found in oraparam.ini, no system pre-requisite checks will be executed.
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2020-11-07_11-42-25AM. Please wait ...
$ 'AttachHome' was successful.


NOTES:
1. if you get a error, like "Please wait ...The Java RunTime Environment was not found at /tmp/OraInstall2015-06-26_11-23-23AM/jre/bin/java", modify one line in oraparam.ini

  $ vi oraparam.ini   (under $ORACLE_HOME/oui  10.1.2 OH)
        # JRE_LOCATION=%JRE_LOCATION%
        JRE_LOCATION=../../jdk/jre                           #  <-- actual path is $ORACLE_HOME/jdk/jre

2. I tried " ouicli.pl " recommended in "How To Create, Update or Rebuild The Central Inventory For Oracle Applications E-Business Suite (Doc ID 742477.1)". But it failed to add one entry to inventory.xml and so does not attach 10.1.2 Oracle Home for unknown reason.

3. If need to detach an ORACLE_HOME from central inventory, see Don ID 417694.1:
./runInstaller -silent -detachHome ORACLE_HOME="<path_to_Oracle_Home>"
If above line fails, you "have to consider building a separate central inventory for this Oracle Home".

Enable TLS1.x in EBS R12.1 secured network

Transport Layer Security (TLS) is a newer cryptographic protocol. We started to work on enabling TLS1.x and disabling SSLv3 with EBS R12.1 (Doc ID 1937646.1) in February 2015. At that time, two requirements are necessary:
- OracleAS 10g must be 10.1.3.5 in EBS R12. If not, use Doc ID 454811.1 to upgrade it.
- Apply latest CPU patch. The latest EBS apps patch was October 2014 (Doc ID 1923805.1), which points 10GiAS 10.1.3.5 CPU patch to 16802901: CPUJUL2013 TRACKING BUG FOR APPLICATION SERVER 10.1.3.5 UNIX. Find and then download it through a link in document 1923805.1, not by patch number. Without it, EBS login page may not work after TLS1.x is enabled.

1. Verify Apache version:  $IAS_ORACLE_HOME/Apache/Apache/bin/httpd -v

Server version: Oracle-Application-Server-10g/10.1.3.5.0 Oracle-HTTP-Server
Server built:   Jul 21 2009 11:12:22

2. download / copy file p16802901_101350_LINUX.zip
    and unzip it to 1680291 folder.

3. vi /etc/oraInst.loc
    to make sure the central Oracle inventory is right.

4. shutdown apps services

5. $ cd $INST_TOP/ora/10.1.3
    then,  source the .env file.

6. make sure $ORACLE_HOME points to 10.1.3
    $ echo $ORACLE_HOME

7. $ "opatch lsinventory -detail"   to verify most patches are NOT installed
    $ cd 16802901
    $ opatch napply          Notes: it will rollback some patches.
    $ opatch lsinventory   to verify the patches are installed.

    $ sh remove_demo.sh   (optional)

    Notes: after opatch completed, "opatch lsinventory" shows 16802900, but not 16802901, was installed.

8. start a new Linux session to get EBS normal env variables.

9. modify  $FND_TOP/admin/template/ssl_conf_1013.tmp (per Doc ID 1937646.1)

SSLProtocol    all +TLSv1 -SSLv2 -SSLv3
SSLCipherSuite HIGH:MEDIUM:!aNULL:+SHA1:+MD5:+HIGH:+MEDIUM:!SSLv3:!SSLv2

10. run autoconfig
Then, verify file $ORA_CONFIG_HOME/10.1.3/Apache/Apache/conf/ssl.conf

11. start apps services

12. go to an EBS webpage, right click ==> Properties. The Properties box shall show "Connection    TLS 1.0 AES ...". The same message may be in Apache log as well.

13. If the site URL is https://siteName.domain.com:4439, below line will display certificate info if TLS1 is enabled:
$ openssl s_client -connect siteName.domain.com:4439 -tls1
$ openssl s_client -connect siteName.domain.com:4439 -tls1 | grep Cipher
(try "openssl s_client -connect siteName.domain.com:4439 -ssl3" to check if ssl3 is still enabled)

UPDATES in January 2016:
1) 1680291 is a very old patch. If patch 19568561 or 21845960 (CPU Oct2015 patch) was installed, 1680291 is not needed. CPU Oct2015 patch 21845960 works better (and it rollback patch 1680291 during installation). Doc ID 1937646.1 Change Log on 02-Nov-2015 shows patch 21845960 was also added to its "Release-Specific Requirements" section.
2) As of today, EBS12.1 only supports TLS1.0 and does not support TLS1.1 or TLS1.2.
3) If TLS1.0 is not enabled, EBS site may not work with Chrome and Firefox because they do not support SSLv3 now. 

Google Chrome 49.0.2623.108
This site can’t provide a secure connection
sitename.domain.com sent an invalid response.
ERR_SSL_PROTOCOL_ERROR


Firefox 44.0.2
Secure Connection Failed
An error occurred during a connection to sitename.domain.com. SSL peer rejected a handshake message for unacceptable content. (Error code: ssl_error_illegal_parameter_alert)