Monday, September 7, 2015

Upgrade database of EBS R12.1.3 to Oracle 12c (12.1.0.2)

Steps to upgrade EBS database from 11.2.0.2 to 12.1.0.2:

0. Make sure pre-requisites for patches of Step 2 were applied:
select * from ad_bugs where bug_number in (
'8502056',    -- R12.AD.B.DELTA.2.            for 12923944
'7651091',    -- R12.ATG_PF.B.DELTA.2.   for 12923944
'6636108',    -- AutoConfig patch (Doc ID 387859.1, Section 3.2)
'8919489',    -- AutoConfig patch (Doc ID 387859.1, Section 3.1)
'9239089',    -- R12.AD.B.DELTA.3        for 16289505
'8919491'     -- R12.ATG_PF.B.delta.3   for 16289505
-- '18039691'  -- AFTER DB upgrade.    for 8796558 on WMS
);

1. Pre-step - Apply patch 11791600:R12.AR.B to address error " Failed on compiling library ARXRWAPP.pll " from applying below 5 patches in Step 2. This could be optional to some environments.

2.  Apply 5 required patches by "Interoperability Notes EBS 12.0 or 12.1 with RDBMS 12cR1 (Doc ID 1524398.1)".
 p12923944_R12.FND.B_R12_LINUX.zip
 p13473483_R12.IBE.B_R12_GENERIC.zip  (for iStore. no pre-requisite)
 p16289505_12.1.0_R12_LINUX.zip
 p19558309_R12.AR.B_R12_LINUX.zip
 p8796558_R12.WMS.B_R12_LINUX.zip

NOTES:
(1) Verified that instance has all pre-requisites, including RPC patches for R12.1.3. In my instance, no RPC patch was applied before and so patch 18843706 is not needed.
SQL> select * from ad_bugs where bug_number in (
  '17774755', --    RPC1
  '19030202', --    RPC2
  '20203366'  --    RPC3);
"For the 12.1.1 instance, there is no reason to apply that patch since you can not apply a RPC patch unless you are on 12.1.3". See Doc ID 1563799.1 (FAQ: E-Business Suite Financials Recommended Patch Collection - RPC ) for more on RPC patches.
UPDATES in April 2018:
      21236633 -- R12.1.3+ RPC4. See Doc ID 2053709.1
      22644544 -- R12.1.3+ RPC5. See Doc ID 2152266.1
(2) Also verified ORACLE_HOME patch was applied before.
In my instance, patch 6400501 was applied to 10.1.2 Oracle Home on apps server.
(3) It works fine for me to merge above 6 patches together and then apply it at one time. 
(4) After 6 patches are applied and before doing database 12c upgrade in next Step 3, start Apps services to do a quick check-out. All shall work.

3. DBAs upgrade database from 11.2.0.2 to 12.1.0.2 (see patch list in Doc ID 761570.1 appendix A) and also apply July2015 PSU patch (Doc ID 21150768.8).
      Patch           Conflict patch
     17694377   12.1.0.2 Patchset (files p17694377_121020_platform_1of8.zip,
                                                          p17694377_121020_platform_2of8.zip,
                                                          p17694377_121020_platform_7of8.zip)
     21150768   Combo of 12.1.0.2.4 OJVM PSU and 12.1.0.2.4 DB PSU (Jul 2015)
     6880880     Latest OPatch
X 19382851     Already included in 12.1.0.2.4 PSU
X 19393542     Already included in 12.1.0.2.4 PSU
X 19627012     Already included in 12.1.0.2.4 PSU
X 19649152     Already included in 12.1.0.2.4 PSU
    20204035    12.1.0.2.0 version. No conflicts

After all, it includes the patch numbers for the 12.1.0.2 release itself, the one-offs required for R12.1.3 EBS and ones that are included in the July PSU

$ $ORACLE_HOME/OPatch/opatch lsinventory | grep applied
 Patch 21068507 : applied on Wed Jul 29 11:22:52 EDT 2015 (part of patch 21150768)
 Patch 20204035 : applied on Wed Jul 29 10:40:05 EDT 2015
 Patch 20831110 : applied on Tue Jul 28 16:04:35 EDT 2015  (part of patch 21150768)

After dbupgdiag.sql script as part of pre 12c upgrade steps, it may report invalid objects in SYS and SYSTEM schemas and utlrp.sql did not compile them.
Invalid views V$LOCKS_WITH_COLLISIONS ,V$LOCK_ACTIVITY & V$LOCK_ELEMENT are only for Oracle 6 compatibility and can be dropped if invalid after migration. Follow the below document for dropping them: Doc ID 837667.1 (Invalid SYS Objects Found On 9.2 Before Upgrade To 10.2.0.1). For invalid package system.OSS, follow Doc ID 1087534.1 (Package Body OSS and OSSADMIN In SYSTEM Schema Are Invalid After Database Upgrade).

Also, dropped MGDSYS schema in 11.2.0.3 as pre upgrade step. After post-upgrade step adconfig.sh on database server, created MGDSYS which helped compile package WMS_EPC_PVT body from invalid to valid. (Before MGDSYS was re-created, compiling WMS_EPC_PVT got same error as described in Doc ID 2198041.1. But after re-creation of MGDSYS, the compiling worked without EBS patch 19666945 even WMS is Inactive status, not installed, in the instance).

Invalid packages QPR_MAINTAIN_AW (Doc ID 1136003.1) and ZPB_BUILD_METADATA (Doc ID 1951646.1) can be ignored, and sys.V$KXSBD can be dropped in 12c if it exists.

NOTES: Patch Set Updates (PSU) are cumulative patches that include both the security fixes and priority fixes.  The key with PSUs is they are minor version upgrades (e.g., 12.1.0.2.0 to 12.1.0.2.4). Once a PSU is applied, only PSUs can be applied in future quarters until the database is upgraded to a new base version.

4. Follow Doc ID 1524398.1 to do the next steps

Get scripts/files ready for DBA tasks:
 (step 21)  cp -p $APPL_TOP/admin/adgrants.sql $APPLPTMP/.
 (step 22)  cp -p $AD_TOP/patch/115/sql/adctxprv.sql $APPLPTMP/.
 (step 25)  cp -p $APPL_TOP/admin/adstats.sql $APPLPTMP/.
 (step 27)  Generate new file appsutil.zip for DB server   (Doc ID 387859.1, Step 3.1 B)
                $ cd $AD_TOP/bin
                $ perl admkappsutil.pl
                $ cp $INST_TOP/admin/out/appsutil.zip $APPLPTMP/.

 Step 24: Validate Workflow ruleset
       $ cd  $FND_TOP/patch/115/sql
       $ sqlplus apps/appPWD @wfaqupfix.sql APPLSYS APPS

 Step 26.  Deregister the current database server.
       Note: it is not necessary, if " select * from fnd_nodes; " output is fine!
       SQL> exec fnd_conc_clone.setup_clean;

Step 27. Run adconfig.sh on database server

Step 29: Apply post-upgrade WMS patch 18039691   -- Seems optional (if WMS module is not used)

 Step 32: Re-create grants and synonyms
       $ adadmin
       --> 4 --> 2.

Step 35  Log on to Oracle E-Business Suite with the "System Administrator" responsibility.
    Click Requests > Run > Single Request and the OK button. Enter the following parameters:
    Request Name = Workflow Directory Services User/Role Validation
    Batch Size = 10000
    Fix dangling users = Yes
    Add missing user/role assignments = Yes
    Update WHO columns in WF tables = No
Click "OK" and "Submit".
Notes: it may take more than one hour! 

NOTE: After all above patches and steps, it got error with db login issue on running concurrent reports cross database link (defined by ILM archiving & purging tool). I had to apply apps patch16946854 to resolve the errors:
REP-0167: Database connect strings are not permitted in USERID parameter. Use TNSnames alias to specify the database.
REP-0501: Unable to connect to the specified database.
ORA-01017: invalid username/password; logon denied


Extra step: e-Business Suite CPU patches:
Apply e-business suite July 2015 CPU patch 20953340 (Doc ID 2013117.1):

1. apply pre-requisite patch 19273341 R12.BNE.B.delta.4
2. run SQL> @adgrants.sql apps
as SYSDBA  (newer version is /20953340/admin/adgrants.sql)
3. apply CPU patch 20953340
Note: for this patch, "adpatch" will run autoconfig during the patching process. All shall be smooth.

I merged 19273341, 20953340 and 16946854 (for db login fix), and then applied it successfully on EBS instances, after the database 12c upgrade and database Jul 2015 PSU patching.

UPDATES in February 2016:
1) During the busiest time in the month, 12.1.0.2 database got hang and no login was allowed.  There were many processes waiting on “LGWR worker group ordering”. It did not give ORA- errors described in Doc ID 1957710.1 (but DEADLOCK might be seen if a system state dump was performed). We applied the workaround "_use_single_log_writer"=TRUE to disable a performance feature in 12c for using multiple log writers.

2) After database upgrade to 12.1.0.2, " AR XLAACCUP module: Accounting Program " concurrent job failed very quick with  error:
An internal error occurred.  Please inform your system administrator or support representative that:
An internal error has occurred in the program xla_ae_lines_pkg.AccountingReversal.  ORA-01555: snapshot too old: rollback segment number 28

The fix is EBS patch 19591608.  We tried workaround in 2009636.1 to set temp_undo_enabled = FALSE. But it did not fix the problem. Then set it back to TRUE before applying patch 19591608.

3) If 12.1.0.2 database for EBS R12.1 gets below UTL_HTTP error, apply 10gAS Oct2015 CUP patch 21845960 to 10.1.3.5 ORACLE_HOME (for Apache) as the fix (see more detail).
SQL> SELECT utl_http.request('https://sitename.domain.com:443', NULL ,'file:/path/to/wallet', NULL) from dual;
SELECT utl_http.request('https://sitename.domain.com:443', NULL ,'file:/path/to/wallet', NULL) from dual
       *
ERROR at line 1:
ORA-29273: HTTP request failed
ORA-29048: Unexpected message received.
ORA-06512: at "SYS.UTL_HTTP", line 1491
ORA-06512: at line 1

See here for setting up Wallet file on database host

4) We also applied R12.1 patch 21950151 to address the performance issue on " ARRGTA: Journal Entries Report ". Note this patch seems having impact on Process Lockboxes report (ARLPLB).

5) After database was upgraded to 12.1.0.2, and database PSU and apps CPU were applied, EBS R12.1.3 got an Import Rules issue. Patch  21149920, which changes only 3 XLA packages that are related to Import functionality, was applied. It fixed the issue.

Sunday, August 30, 2015

R12.1 patching for Internet Explorer 11 (IE 11)

R12.1 is certified with Internet Explorer 11. But patches are necessary (Doc ID 389422.1).

1. Merge and apply 4 patches to R12.1.3:
18936791:R12.FWK.B.delta.4
17932167:R12.TXK.B.delta.3
19273341:R12.BNE.B.delta.4
19559960:R12.FRM.B.Delta.4
Notes: when CRM module is not used, applying CRM patch will create a lot invalid objects in DB)

2. Apply patch 16983427 to fix a bug on Payables after above 4 patches:
'ViewObject attribute is null' Error Trying To Submit Payment Process Request (Doc ID 1481246.1)

3. Clean some cache files:
$ cd $OA_HTML
$ rm -f cabo/images/cache/*
$ rm -f cabo/styles/cache/*

4. Update profile option "BNE Redirect Portal URL" to empty (Bug: Doc ID 1965891.1).

5. Users need to use some workarounds to make ADI Excel import/export work.

Sunday, August 23, 2015

EBS Reports errors as concurrent jobs

In R12, below command can check if Report Manager is running or not on CM node:

$ ps -ef | grep rwrun

But, it there is no report in Running status in concurrent manager, that command will return nothing.

Troubleshooting on Reports:
 
1. Reports finish, but give error at the end: "REP-57054: In-process job terminated:Finished successfully but output is voided". The behavior matches "Intermittent Oracle Reports REP-0069: REP-57054: Error (Doc ID 1237834.1)".

Workaround:
(1) create a file -  $FND_TOP/admin/template/custom/rwbuilder_conf_1012.tmp
 use the idea in Doc ID 1237834.1 to prevent Autoconfig from clobbering the changes.
(2) modify the file changing one line to <property name="cacheSize" value="50"/>
(3) run autoconfig
(4) verify $INST_TOP/ora/10.1.2/reports/conf/rwbuilder.conf have 50 as the cacheSize.
(5) schedule cron jobs to delete log files.
05 23 * * * /usr/bin/find $INST_TOP/logs/ora/10.1.2/reports/cache/*.txt -mtime +3 -exec rm -rf {} \;
05 05 * * * /usr/bin/find $INST_TOP/logs/ora/10.1.2/reports/cache/*.xml -mtime +7 -exec rm -rf {} \;
05 10 * * * /usr/bin/find $INST_TOP/logs/ora/10.1.2/reports/cache/*.txt -mtime +7 -exec rm -rf {} \;
(replace $INST_TOP with the real path)

2. After submit a CM job/report, the job stays there forever.

It looks like it has difficulties to hit the database. But the real fix is to delete file  $ORACLE_HOME/reports/conf/rwnetwork.conf. 
See "Concurrent Processing - R12 Concurrent Requests Run Forever, rwrun Errors with REP-50125 (Doc ID 737445.1)".

3.

Attach 12.1.2 ORACLE HOME after clone

After an instance was cloned, 12.1.2 ORACLE HOME does not exist. I had to run a command to attach it.

$ echo $ORACLE_HOME
/u05/app/EBS/apps/tech_st/10.1.2

$ opatch lsinventory

Oracle Interim Patch Installer version 1.0.0.0.63
Copyright (c) 2009 Oracle Corporation. All Rights Reserved..

Oracle recommends you to use the latest OPatch version
and read the OPatch documentation available in the OPatch/docs
directory for usage. For information about the latest OPatch and
other support-related issues, refer to document ID 293369.1
available on My Oracle Support (https://myoraclesupport.oracle.com)

Oracle Home                : /u05/app/EBS/apps/tech_st/10.1.2
Oracle Home Inventory : /u05/app/EBS/apps/tech_st/10.1.2/inventory
Central Inventory          : /u05/app/oraEbsInventory
   from                          : /etc/oraInst.loc
OUI location                : /u05/app/EBS/apps/tech_st/10.1.2/oui
OUI shared library       : /u05/app/EBS/apps/tech_st/10.1.2/oui/lib/linux/liboraInstaller.so
Java location                : /u05/app/EBS/apps/tech_st/10.1.2/jre/1.4.2/bin/java
Log file location           : /u05/app/EBS/apps/tech_st/10.1.2/.patch_storage/<patch ID>/*.log

Creating log file "/u05/app/EBS/apps/tech_st/10.1.2/.patch_storage/LsInventory__06-26-2015_11-21-30.log"

Result:
LsInventory: OPatch Exception while accessing O2O

OPATCH_JAVA_ERROR  : An exception of type "OPatchException" has occurred:
OPatch Exception:  OUI found no such ORACLE_HOME set in the environment
  Can not get details for given Oracle Home
  An exception occurs
  null

There is no Interim Patch

OPATCH_JAVA_ERROR  : An exception of type "OPatchException" has occurred:
Can not get a list of inventory on this home.

ERROR: OPatch failed because of Inventory problem.

Check inventory.xml and also see the 10.1.2 ORACLE HOME has no entry in the file:
$ more /ebsu05/app/oraEbsInventory/ContentsXML/inventory.xml
<?xml version="1.0" standalone="yes" ?>
<!-- Copyright (c) 2002 Oracle Corporation. All rights Reserved -->
<!-- Do not modify the contents of this file by hand. -->
<INVENTORY>
<VERSION_INFO>
   <SAVED_WITH>10.1.0.6.0</SAVED_WITH>
   <MINIMUM_VER>2.1.0.6.0</MINIMUM_VER>
</VERSION_INFO>
<HOME_LIST>
<HOME NAME="EBS_WEBOH__u05_app_EBS_apps_tech_st_10_1_3" LOC="/u05/app/EBS/apps/tech_st/10.1.3" TYPE="O" IDX="1"/>
</HOME_LIST>
</INVENTORY>

Fix: run ./runInstaller to create ORACLE_HOME

First of all, make sure inventory location is correct:
$ vi /etc/oraInst.loc
inventory_loc=/u05/app/oraEbsInventory

$ cd $ORACLE_HOME/oui/bin
$ echo $ORACLE_HOME
 /u05/app/EBS/apps/tech_st/10.1.2
$ ./runInstaller -silent -attachHome ORACLE_HOME=/u05/app/EBS/apps/tech_st/10.1.2 ORACLE_HOME_NAME=EBS_1012_home

Starting Oracle Universal Installer...
No pre-requisite checks found in oraparam.ini, no system pre-requisite checks will be executed.
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2015-06-26_11-39-13AM. Please wait ...
$ 'AttachHome' was successful.
$

Now, check file /u05/app/oraEbsInventory/ContentsXML/inventory.xml again, it should have two entries. And also "$ opatch lsinventory" works!

Similarly steps can attach a 10.1.3 ORACLE_HOME in EBS R12.1:

$ cd $INST_TOP/ora/10.1.3
$ ls
$ . TWO_TASK_nodename.env
$ echo $ORACLE_HOME
/u05/app/EBS/apps/tech_st/10.1.3
$ cd $ORACLE_HOME/oui/bin
$ ./runInstaller -silent -attachHome ORACLE_HOME=/u05/app/EBS/apps/tech_st/10.1.3 ORACLE_HOME_NAME=EBS_WEBOH_10_1_3_HOME
Starting Oracle Universal Installer...
No pre-requisite checks found in oraparam.ini, no system pre-requisite checks will be executed.
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2020-11-07_11-42-25AM. Please wait ...
$ 'AttachHome' was successful.


NOTES:
1. if you get a error, like "Please wait ...The Java RunTime Environment was not found at /tmp/OraInstall2015-06-26_11-23-23AM/jre/bin/java", modify one line in oraparam.ini

  $ vi oraparam.ini   (under $ORACLE_HOME/oui  10.1.2 OH)
        # JRE_LOCATION=%JRE_LOCATION%
        JRE_LOCATION=../../jdk/jre                           #  <-- actual path is $ORACLE_HOME/jdk/jre

2. I tried " ouicli.pl " recommended in "How To Create, Update or Rebuild The Central Inventory For Oracle Applications E-Business Suite (Doc ID 742477.1)". But it failed to add one entry to inventory.xml and so does not attach 10.1.2 Oracle Home for unknown reason.

3. If need to detach an ORACLE_HOME from central inventory, see Don ID 417694.1:
./runInstaller -silent -detachHome ORACLE_HOME="<path_to_Oracle_Home>"
If above line fails, you "have to consider building a separate central inventory for this Oracle Home".

Enable TLS1.x in EBS R12.1 secured network

Transport Layer Security (TLS) is a newer cryptographic protocol. We started to work on enabling TLS1.x and disabling SSLv3 with EBS R12.1 (Doc ID 1937646.1) in February 2015. At that time, two requirements are necessary:
- OracleAS 10g must be 10.1.3.5 in EBS R12. If not, use Doc ID 454811.1 to upgrade it.
- Apply latest CPU patch. The latest EBS apps patch was October 2014 (Doc ID 1923805.1), which points 10GiAS 10.1.3.5 CPU patch to 16802901: CPUJUL2013 TRACKING BUG FOR APPLICATION SERVER 10.1.3.5 UNIX. Find and then download it through a link in document 1923805.1, not by patch number. Without it, EBS login page may not work after TLS1.x is enabled.

1. Verify Apache version:  $IAS_ORACLE_HOME/Apache/Apache/bin/httpd -v

Server version: Oracle-Application-Server-10g/10.1.3.5.0 Oracle-HTTP-Server
Server built:   Jul 21 2009 11:12:22

2. download / copy file p16802901_101350_LINUX.zip
    and unzip it to 1680291 folder.

3. vi /etc/oraInst.loc
    to make sure the central Oracle inventory is right.

4. shutdown apps services

5. $ cd $INST_TOP/ora/10.1.3
    then,  source the .env file.

6. make sure $ORACLE_HOME points to 10.1.3
    $ echo $ORACLE_HOME

7. $ "opatch lsinventory -detail"   to verify most patches are NOT installed
    $ cd 16802901
    $ opatch napply          Notes: it will rollback some patches.
    $ opatch lsinventory   to verify the patches are installed.

    $ sh remove_demo.sh   (optional)

    Notes: after opatch completed, "opatch lsinventory" shows 16802900, but not 16802901, was installed.

8. start a new Linux session to get EBS normal env variables.

9. modify  $FND_TOP/admin/template/ssl_conf_1013.tmp (per Doc ID 1937646.1)

SSLProtocol    all +TLSv1 -SSLv2 -SSLv3
SSLCipherSuite HIGH:MEDIUM:!aNULL:+SHA1:+MD5:+HIGH:+MEDIUM:!SSLv3:!SSLv2

10. run autoconfig
Then, verify file $ORA_CONFIG_HOME/10.1.3/Apache/Apache/conf/ssl.conf

11. start apps services

12. go to an EBS webpage, right click ==> Properties. The Properties box shall show "Connection    TLS 1.0 AES ...". The same message may be in Apache log as well.

13. If the site URL is https://siteName.domain.com:4439, below line will display certificate info if TLS1 is enabled:
$ openssl s_client -connect siteName.domain.com:4439 -tls1
$ openssl s_client -connect siteName.domain.com:4439 -tls1 | grep Cipher
(try "openssl s_client -connect siteName.domain.com:4439 -ssl3" to check if ssl3 is still enabled)

UPDATES in January 2016:
1) 1680291 is a very old patch. If patch 19568561 or 21845960 (CPU Oct2015 patch) was installed, 1680291 is not needed. CPU Oct2015 patch 21845960 works better (and it rollback patch 1680291 during installation). Doc ID 1937646.1 Change Log on 02-Nov-2015 shows patch 21845960 was also added to its "Release-Specific Requirements" section.
2) As of today, EBS12.1 only supports TLS1.0 and does not support TLS1.1 or TLS1.2.
3) If TLS1.0 is not enabled, EBS site may not work with Chrome and Firefox because they do not support SSLv3 now. 

Google Chrome 49.0.2623.108
This site can’t provide a secure connection
sitename.domain.com sent an invalid response.
ERR_SSL_PROTOCOL_ERROR


Firefox 44.0.2
Secure Connection Failed
An error occurred during a connection to sitename.domain.com. SSL peer rejected a handshake message for unacceptable content. (Error code: ssl_error_illegal_parameter_alert)