Sunday, October 25, 2020

Pre steps and Rapid Install for creating R12.2 upgrade file system

R12.2 Rapid Install performs two functions during an upgrade. This post is for the first function that specifies the configuration values for Rapid Install to lay down a new file system and installs the new technology stack.

- Pre-steps

1. Make sure OS meets the requirements for installation (See my post). And R12.2 installation needs about additional 200GB diskspace.
$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.9 (Maipo)
2. Activate all Apps services to only one master node if R12.1 is a multi-node environment.
3. Reduce number of concurrent Managers so that it will start less processes on OS level from starting EBS services during the upgrade.
4. Put some scheduled CM jobs on hold.
5. Disable cron jobs (if any).
 
- Database preparation (pre-steps in Release 12.1.0.2.0):

1. Collect stats by apps
SQL> conn apps/apps_PWD
SQL> exec apps.fnd_stats.gather_schema_statistics('ALL',estimate_percent=>100);   
PL/SQL procedure successfully completed.    
                      <== This takes long time, can do so via concurrent by System Admin.

SQL> SELECT package_name,object_name,argument_name,data_type
FROM all_arguments WHERE (package_name,object_name)
  IN (SELECT call_package,call_procedure FROM hr_api_hook_calls) 
AND data_type = 'LONG';

Ignore package PSB_HR_POPULATE_DATA_PVT because it is obsolete in R12.2 (Doc ID 1969202.1).

SQL> spool invalid_R12_1.txt
SQL> col owner for a27
SQL> col object_name for a45;
SQL> col object_typr for a27;
SQL> col status for a9;
SQL> set lines 222 pages 123;

SQL> select owner, count(*) from dba_objects where status = 'INVALID' 
group by owner order by owner desc;
SQL> select owner,object_type,count(*) from dba_objects where status <> 'VALID' 
group by owner,object_type order by owner;
SQL> select owner, substr(object_name,1,30) object_name, object_type, status 
from dba_objects 
where status = 'INVALID' and owner ='APPS'
order by object_name, object_type, object_name; 

SQL> select fnd_web_sec.validate_login('GUEST','ORACLE') valid from dual;
VALID
-------------------------------
Y
If 'N' is returned, then query the error message for more info:

SQL> select fnd_message.get from dual;

2. Create xml file and run autoconfig (if it was run in long time ago).

3. Init parameters (Oracle Doc ID 396009.1 gives recommendations on init parameters). Some of them shall be set to:
ALTER SYSTEM SET job_queue_processes=60 scope=both;    (SYSTEM can do it)
ALTER SYSTEM SET aq_tm_processes=1;                       (default: scope=memory?)
ALTER SYSTEM SET undo_retention=900;
ALTER SYSTEM SET plsql_optimize_level=2;
ALTER SYSTEM SET enable_ddl_logging=FALSE;
ALTER SYSTEM SET sec_case_sensitive_logon=FALSE;
ALTER SYSTEM SET resource_manager_plan=null;
ALTER SYSTEM SET recyclebin=OFF DEFERRED;       (it takes effect in new sessions)
ALTER SYSTEM SET "_system_trig_enabled"=TURE;  (for online patching enablement)

And, disable unneeded hidden parameters (if see any).

Notes: re-visit them after R12.2 upgrade.

4. Additional steps:

SQL> show user
USER is "SYSTEM"
SQL> show parameter service_names
NAME                                 TYPE        VALUE
---------------------------------- ----------- ------------------------------
service_names                   string      EBSDEV

SQL> alter system set service_names='EBSDEV','ebs_patch' scope=both ;
SQL> select SERVICE_ID,NAME from dba_services where name='ebs_patch';
SERVICE_ID NAME
----------------- ------------
                  6  ebs_patch

SQL> show parameter local_listener
NAME               TYPE  VALUE
------------------- -------- ------------------------------
local_listener   string    EBSDEV_LOCAL

(or, SQL> alter system set local_listener='EBSDEV_LOCAL' scope=both;)

Disable AOL Audit Trail (conditional):
SQL> select distinct enabled from dba_audit_policies;
no rows selected

Fixed object and dictionary statistics: about 10 minutes

SQL> EXECUTE DBMS_STATS.GATHER_FIXED_OBJECTS_STATS;
SQL> EXEC dbms_stats.gather_dictionary_stats;

Collect SYS stats as sysdba (SYSTEM can also do it)
SQL> begin
   dbms_stats.gather_schema_stats (
      'SYS',
      options=>'GATHER STALE',
      estimate_percent => DBMS_STATS.AUTO_SAMPLE_SIZE,
      method_opt => 'FOR ALL COLUMNS SIZE AUTO',
      cascade => TRUE);
end;

Get the ORACLE_HOME path if necessary:
SQL> var OH varchar2(200);
SQL> EXEC dbms_system.get_env('ORACLE_HOME', :OH) ;
PL/SQL procedure successfully completed.
SQL> PRINT OH
OH
--------------------------------------------------------------------------------
/ohome/u01/app/oracle/product/12.0.0.0/db_ebsdev

5. Disable triggers in custom schemas (Seems optional?):
col TRIGGER_NAME for a36;
col TABLE_OWNER for a27;
col TABLE_NAME for a36;
col STATUS for a12;
set lines 333 pages 123;
SQL> select owner,trigger_name,status,table_owner from dba_triggers 
where status='ENABLED' and owner not in ('APPS','SYS','MDSYS');

SQL> select owner,trigger_name,status,table_owner from dba_triggers where owner in
(select username from dba_users where default_tablespace not in ('APPS_TS_TX_DATA','SYSAUX'));

SQL> select 'alter trigger '||owner||'.'||trigger_name||' disable;' from dba_triggers 
where owner like 'XX%';

Note: not sure how to use $ALR_TOP/patch/115/sql/alrdtrig.sql

6. Important step: modify database profiles to avoid errors during installations (assume APPS uses TRUSTED_ID_NO_EXPIRE profile and Installer creates new users of R12.2, such as GHG, to use DEFAULT profile).
alter profile default limit PASSWORD_VERIFY_FUNCTION null;
alter profile default limit FAILED_LOGIN_ATTEMPTS UNLIMITED;
alter profile TRUSTED_ID_NO_EXPIRE limit FAILED_LOGIN_ATTEMPTS UNLIMITED;
alter profile TRUSTED_ID_NO_EXPIRE limit PASSWORD_REUSE_TIME UNLIMITED;
alter profile TRUSTED_ID_NO_EXPIRE limit PASSWORD_REUSE_MAX UNLIMITED;
alter profile TRUSTED_ID_NO_EXPIRE limit PASSWORD_VERIFY_FUNCTION null;

col profile for a27;
col resource_name for a36;
col limit for a27;
SQL> select profile,resource_name,limit from dba_profiles
where profile in ('DEFAULT','TRUSTED_ID_NO_EXPIRE')
and  resource_type = 'PASSWORD'  order by 1,2;

PROFILE                                 RESOURCE_NAME                           LIMIT
------------------------------------ ---------------------------------------------  -----------------
DEFAULT                                FAILED_LOGIN_ATTEMPTS          UNLIMITED
DEFAULT                                PASSWORD_GRACE_TIME            7
DEFAULT                                PASSWORD_LIFE_TIME                 360
DEFAULT                                PASSWORD_LOCK_TIME               UNLIMITED
DEFAULT                                PASSWORD_REUSE_MAX              24
DEFAULT                                PASSWORD_REUSE_TIME             365
DEFAULT                                PASSWORD_VERIFY_FUNCTION NULL

TRUSTED_ID_NO_EXPIRE FAILED_LOGIN_ATTEMPTS           UNLIMITED
TRUSTED_ID_NO_EXPIRE PASSWORD_GRACE_TIME             UNLIMITED
TRUSTED_ID_NO_EXPIRE PASSWORD_LIFE_TIME                  UNLIMITED
TRUSTED_ID_NO_EXPIRE PASSWORD_LOCK_TIME               1
TRUSTED_ID_NO_EXPIRE PASSWORD_REUSE_MAX              UNLIMITED
TRUSTED_ID_NO_EXPIRE PASSWORD_REUSE_TIME             UNLIMITED
TRUSTED_ID_NO_EXPIRE PASSWORD_VERIFY_FUNCTION NULL

SQL> select profile, resource_name, limit from dba_profiles where profile in 
(select profile from dba_users where username = 'APPS')
and resource_type = 'PASSWORD' order by resource_name;

7. ETCC: The EBS Technology Code Level Checker (ETCC) (See Doc ID 2008451.1)
Download patch 17537119, and unzip p17537119_R12_GENERIC.zip to a folder.

DBA can run checkDBpatch.sh, also called DBCC, on database server. It uses opatch to check installed patches in database and may come out about 20+ db patches required. But checkMTpatch.sh will not work on Apps nodes until Rapidwiz completes the installation.

$ ./checkDBpatch.sh contextfile= xxxxx.xml

Using context file from command line argument: xxxxx.xml
Starting Database EBS Technology Codelevel Checker, Version 120.81
Tue Oct 20 12:56:16 EDT 2020
Log file for this session : /u01/db_ebsdev/appsutil/etcc/log/checkDBpatch_65979616.log
Identifying database release.
Database release set to 12.1.0.2.
Connecting to database.
Database connection successful.
Identifying APPS and APPLSYS schema names.
 - APPS schema : APPS
 - APPLSYS schema : APPLSYS
Checking for DB-ETCC results table.
Creating DB-ETCC results table.
Created the table to store DB-ETCC results.
Checking if InMemory option is enabled.
InMemory option is not enabled in the database.
Bugfix file ./db/onprem/txk_R1220_DB_base_bugs.xml : 120.0.12020000.56
This file will be used for identifying missing bugfixes.
Mapping file ./db/onprem/txk_R1220_DB_mappings.xml : 120.0.12020000.31
This file will be used for mapping bugfixes to patches.
[WARNING] DB-ETCC: Bugfix XML file (./db/onprem/txk_R1220_DB_base_bugs.xml) is more than 30 days old.
 Check if a newer version is available in patch 17537119.
Identified RDBMS DST version 19.
Checking Bugfix XML file for 12.1.0.2_PSU
Obtained list of bugfixes to be applied and the list to be rolled back.
Now checking Database ORACLE_HOME.
DB-ETCC is compatible with this opatch version.
Found patch records in the inventory.
Checking Mapping XML file for 12.1.0.2.200714
  Missing Bugfix: 8975044  ->  Patch 8975044
  Missing Bugfix: 14666816  ->  Patch 30614876
  Missing Bugfix: 15894842  ->  Patch 15894842
  Missing Bugfix: 18485835  ->  Patch 30216977
  Missing Bugfix: 18689530  ->  Patch 22098146
  Missing Bugfix: 18881811  ->  Patch 25599890
  Missing Bugfix: 19239846  ->  Patch 31469752
  Missing Bugfix: 19472320  ->  Patch 19472320
  Missing Bugfix: 19509982  ->  Patch 30216977
  Missing Bugfix: 20123899  ->  Patch 20123899
  Missing Bugfix: 20181016  ->  Patch 20181016
  Missing Bugfix: 20476776  ->  Patch 30614876
  Missing Bugfix: 20887355  ->  Patch 20887355
  Missing Bugfix: 21028698  ->  Patch 22098146
  Missing Bugfix: 21321429  ->  Patch 21321429
  Missing Bugfix: 21614112  ->  Patch 30216977
  Missing Bugfix: 21751519  ->  Patch 30614876
  Missing Bugfix: 21813400  ->  Patch 31469752
  Missing Bugfix: 21864513  ->  Patch 21864513
  Missing Bugfix: 21904072  ->  Patch 21904072
  Missing Bugfix: 21967332  ->  Patch 21967332
  Missing Bugfix: 22229581  ->  Patch 30614876
  Missing Bugfix: 22452653  ->  Patch 30614876
  Missing Bugfix: 22666802  ->  Patch 31469752
  Missing Bugfix: 22737974  ->  Patch 30614876
  Missing Bugfix: 23604553  ->  Patch 23604553
  Missing Bugfix: 23632545  ->  Patch 31469752
  Missing Bugfix: 23645516  ->  Patch 31469752
  Missing Bugfix: 25139545  ->  Patch 25139545
  Missing Bugfix: 25305405  ->  Patch 25305405
  Missing Bugfix: 25906117  ->  Patch 25906117
  Missing Bugfix: 25971286  ->  Patch 31469752
  Missing Bugfix: 26742604  ->  Patch 30614876
  Missing Bugfix: 26798411  ->  Patch 26798411
  Missing Bugfix: 29125374  ->  Patch 30216977
Generating Patch Recommendation Summary.
========================================================
PATCH RECOMMENDATION SUMMARY
========================================================
The default patch recommendations to install these missing bugfixes are:
-------------------------------------------------------------------------------
Oracle Database Release 12.1.0.2  (PATCHSET UPDATE 12.1.0.2.200714)
-------------------------------------------------------------------------------
  Patch 8975044 [12.1.0.2.200414 version]
    - Filename: p8975044_12102200414_AIX64-5L.zip
  Patch 30614876 [12.1.0.2.171017 version]
    - Filename: p30614876_12102171017_AIX64-5L.zip
  Patch 15894842 [12.1.0.2.200714 version]
    - Filename: p15894842_12102200714_Generic.zip
  Patch 30216977 [12.1.0.2.191015 version]
    - Filename: p30216977_12102191015_AIX64-5L.zip
  Patch 22098146
    - Filename: p22098146_121020_AIX64-5L.zip
  Patch 25599890
    - Filename: p25599890_121020_Generic.zip
  Patch 31469752 [12.1.0.2.200714 version]
    - Filename: p31469752_12102200714_AIX64-5L.zip
  Patch 19472320
    - Filename: p19472320_121020_AIX64-5L.zip
  Patch 20123899 [12.1.0.2.5 version]
    - Filename: p20123899_121025_Generic.zip
  Patch 20181016
    - Filename: p20181016_121020_AIX64-5L.zip
  Patch 20887355
    - Filename: p20887355_121020_Generic.zip
  Patch 21321429 [12.1.0.2.170117 version]
    - Filename: p21321429_12102170117_Generic.zip
  Patch 21864513
    - Filename: p21864513_121020_AIX64-5L.zip
  Patch 21904072
    - Filename: p21904072_121020_AIX64-5L.zip
  Patch 21967332 [12.1.0.2.160419 version]
    - Filename: p21967332_12102160419_AIX64-5L.zip
  Patch 23604553 [12.1.0.2.200414 version]
    - Filename: p23604553_12102200414_Generic.zip
  Patch 25139545 [12.1.0.2.190416 version]
    - Filename: p25139545_12102190416_Generic.zip
  Patch 25305405 [12.1.0.2.160119 version]
    - Filename: p25305405_12102160119_AIX64-5L.zip
  Patch 25906117
    - Filename: p25906117_121020_AIX64-5L.zip
  Patch 26798411 [12.1.0.2.190716 version]
    - Filename: p26798411_12102190716_AIX64-5L.zip

Apply the required patches and rerun this script
You should check the patch READMEs for minimum opatch version requirements.
The latest opatch is available from My Oracle Support via Patch 6880880.
+-----------------------------------------------------------------------------+
A consolidated zip file with the required patches for Database release
12.1.0.2.200714 is available on My Oracle Support via:
  Patch 31381618 [12.1.0.2.200714 version]
    - EBS RELEASE 12.2 CONSOLIDATED DATABASE FIXES FOR JUL 2020
Note: This zip does not include database Release Updates, bundles or PSUs.
+-----------------------------------------------------------------------------+
See Doc ID 1147107.1 for any special instructions for these patches.
Note: Footnotes in Doc ID 1594274.1 also apply to corresponding overlay patches.
Stored Technology Codelevel Checker results in the database EBSDEV successfully.
Finished checking fixes for Oracle Database: Tue Oct 20 12:56:49 EDT 2020
Log file for this session: /u01/db_ebsdev/appsutil/etcc/log/checkDBpatch_65979616.log

DBA has to apply patches reported from checkDBpatch.sh and run the script repeatedly until the list is cleared.

Database patches applied.
ETCC ran successfully
The opatch utility is version 12.2.0.1.21.
DB-ETCC is compatible with this opatch version.
Found patch records in the inventory.
Checking Mapping XML file for 12.1.0.2.200714
All the required one-off bugfixes are present in Database ORACLE_HOME.


8. If database is upgraded to 12c before EBS R12.2 upgrade, complete steps in Doc ID 1524398.1 (Interoperability Notes EBS 12.0 or 12.1 with RDBMS 12cR1).

Below steps might not be executed previously. It is necessary run them now to avoid error from ADZDPREP.sql in applying R12.2 online patch 13543062.

Post DB upgrade steps:
   # add SQLNET.ALLOWED_LOGON_VERSION_SERVER=8 in sqlnet.ora
   #. run scripts: It seems it does not run on AEBSGOLD/AEBSDEV2
                sqlplus / as sysdba
                SQL> @?/rdbms/admin/dbmsxdbschmig.sql
                SQL> @?/rdbms/admin/prvtxdbschmig.plb
                SQL> @?/rdbms/admin/utlrp.sql

- Run Rapid Installer to lay down R12.2.0 software ("Create Upgrade File System")

1. Backup APPS file system and database
2. Make sure R12.1 works and then stop all services
$ ps -ef | grep $LOGNAME
3. If pre-installation failed, you have to delete all related files in /tmp, and folders fs1, fs2, fs_ne. (See Doc ID 2091880.1: 12.2 E-Business Suite - How To Clean Up a Rapid Install Failure and Restart a Fresh Install).
4. If previous installation worked successfully and you want to re-run rapidwiz from the beginning, you have to delete the inventory folder defined in oraInst.loc file in addition to step 2 above to avoid below strange errors. Then, re-create the folder and attach two R12.1 ORACLE_HOMEs to it.
Fatal Error: TXK Install Service
oracle.apps.fnd.txk.config.ProcessStateException: OPatch process failed : Exit=160
make[1]: *** [/path/to/fs2/EBSapps/10.1.2/sysman/lib/libnmemso.so] Error 1

4. Launch the installer
$ vi .profile   <== and re-login (NOT to use current R12.1 env file)
$ env             <== make sure no Oracle related and strange env values
$ env | grep OPATCH_PLATFORM_ID
Make sure nothing is set for this to avoid Rapidwiz fatal error with patch 6275529 oracle.apps.fnd.txk.config.ProcessStateException (Doc ID 2336585.1)
$ more /etc/oraInst.loc
inventory_loc=/path/to/oraEBSdevInventory
S mv /path/to/oraEBSdevInventory /path/to/oraEBSdevInventory_R12.1
$ mkdir /path/to/oraEBSdevInventory    
  Notes: R12.1 file folders can be renamed after R12.2.10 is good and up.
$ export DISPLAY=xxx.xx.xxx:0.0     <== assume Exceed is installed on the desktop
$ cd startCD/Disk1/rapidwiz

$ ./rapidwiz
choose "Upgrade to Oracle E-Business Suite Release 12.2.0"
            "Create Upgrade File System"
enter 41 & 42 (if database uses port 1562) in port pool
enter database info such as $ORACLE_BASE & $ORACLE_HOME from database node, and 
choose "Use Existing Oracle Home (No new Database Home install)"
System check marks shall all be green, before continue to "Next".

The installation GUI stays at 0% (for about 30 minutes), 40% for a while and takes totally about 3 hours. It will post screen messages telling installation log location $INST_TOP/logs
Configuration file written to: /…/EBSDEV/fs1/inst/apps/<CONTEXT_NAME>/conf_EBSEV.txt
Configuration file written to: /…/EBSDEV/fs2/inst/apps/<CONTEXT_NAME>/conf_EBSDEV.txt
Second File System logfile - /…/EBSDEV/fs2/inst/apps/<CONTEXT_NAME>/logs/10212202.log
First File System logfile - /…/EBSDEV/fs1/inst/apps/<CONTEXT_NAME>/logs/10212202.log


The .log file size shall be about 30k (under /fs1/) and the end of log file has entries (replace R12.2 env variables):
… … 
Stopping the Service - /bin/sh $ADMIN_SCRIPTS_HOME/adnodemgrctl.sh  stop
... ...
You are running adnodemgrctl.sh version 120.11.12020000.4
Enter the WebLogic Admin password:
Either the Node Manager is not up or the credentials are incorrect.
adnodemgrctl.sh: exiting with status 2
adnodemgrctl.sh: check the logfile $INST_TOP/logs/appl/admin/log/adnodemgrctl.txt for more information ...
stty: standard input: Inappropriate ioctl for device
stty: standard input: Inappropriate ioctl for device
Upgrade File
-------------
   APPL_TOP environment file passes instantiated variables test:
      File = $APPL_TOP/<CONTEXT_NAME>.env
   ADOVARS environment file passes instantiated variables test:
      File = $APPL_TOP/admin/adovars.env
   APPSCONFIG passes instantiated variables test:
      File = $APPL_TOP/admin/adconfig.txt
DoInstallPanel - Summary Text
The Rapid Install Wizard will now install the following:

If go to Apps Base Directory specified on the installation GUI, 3 new folders are created there:
$ du -sh fs*
28G     fs1
28G     fs2
32K     fs_ne

$ ps -ef | grep $LOGNAME
Nothing shall run.

Choose the RUN file system by run the .env file under /fs1/EBSapps/appl (or /fs2/EBSapps/appl) to set up R12.2 environment for continuing next steps. 

$ cd <Apps Base Directory>/fs1/EBSapps/appl
$ . ./APPS<TWO_TASK>_<node_name>.env
$ echo $FILE_EDITION
run
$ echo $APPL_TOP
<Apps Base Directory>/fs1/EBSapps/appl

Now, The installation is completed.