Sunday, October 25, 2020

Pre steps and Rapid Install for creating R12.2 upgrade file system

R12.2 Rapid Install performs two functions during an upgrade. This post is for the first function that specifies the configuration values for Rapid Install to lay down a new file system and installs the new technology stack.

- Pre-steps

1. Make sure OS meets the requirements for installation (See my post). And R12.2 installation needs about additional 200GB diskspace.
$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.9 (Maipo)
2. Activate all Apps services to only one master node if R12.1 is a multi-node environment.
3. Reduce number of concurrent Managers so that it will start less processes on OS level from starting EBS services during the upgrade.
4. Put some scheduled CM jobs on hold.
5. Disable cron jobs (if any).
 
- Database preparation (pre-steps in Release 12.1.0.2.0):

1. Collect stats by apps
SQL> conn apps/apps_PWD
SQL> exec apps.fnd_stats.gather_schema_statistics('ALL',estimate_percent=>100);   
PL/SQL procedure successfully completed.    
                      <== This takes long time, can do so via concurrent by System Admin.

SQL> SELECT package_name,object_name,argument_name,data_type
FROM all_arguments WHERE (package_name,object_name)
  IN (SELECT call_package,call_procedure FROM hr_api_hook_calls) 
AND data_type = 'LONG';

Ignore package PSB_HR_POPULATE_DATA_PVT because it is obsolete in R12.2 (Doc ID 1969202.1).

SQL> spool invalid_R12_1.txt
SQL> col owner for a27
SQL> col object_name for a45;
SQL> col object_typr for a27;
SQL> col status for a9;
SQL> set lines 222 pages 123;

SQL> select owner, count(*) from dba_objects where status = 'INVALID' 
group by owner order by owner desc;
SQL> select owner,object_type,count(*) from dba_objects where status <> 'VALID' 
group by owner,object_type order by owner;
SQL> select owner, substr(object_name,1,30) object_name, object_type, status 
from dba_objects 
where status = 'INVALID' and owner ='APPS'
order by object_name, object_type, object_name; 

SQL> select fnd_web_sec.validate_login('GUEST','ORACLE') valid from dual;
VALID
-------------------------------
Y
If 'N' is returned, then query the error message for more info:

SQL> select fnd_message.get from dual;

2. Create xml file and run autoconfig (if it was run in long time ago).

3. Init parameters (Oracle Doc ID 396009.1 gives recommendations on init parameters). Some of them shall be set to:
ALTER SYSTEM SET job_queue_processes=60 scope=both;    (SYSTEM can do it)
ALTER SYSTEM SET aq_tm_processes=1;                       (default: scope=memory?)
ALTER SYSTEM SET undo_retention=900;
ALTER SYSTEM SET plsql_optimize_level=2;
ALTER SYSTEM SET enable_ddl_logging=FALSE;
ALTER SYSTEM SET sec_case_sensitive_logon=FALSE;
ALTER SYSTEM SET resource_manager_plan=null;
ALTER SYSTEM SET recyclebin=OFF DEFERRED;       (it takes effect in new sessions)
ALTER SYSTEM SET "_system_trig_enabled"=TURE;  (for online patching enablement)

And, disable unneeded hidden parameters (if see any).

Notes: re-visit them after R12.2 upgrade.

4. Additional steps:

SQL> show user
USER is "SYSTEM"
SQL> show parameter service_names
NAME                                 TYPE        VALUE
---------------------------------- ----------- ------------------------------
service_names                   string      EBSDEV

SQL> alter system set service_names='EBSDEV','ebs_patch' scope=both ;
SQL> select SERVICE_ID,NAME from dba_services where name='ebs_patch';
SERVICE_ID NAME
----------------- ------------
                  6  ebs_patch

SQL> show parameter local_listener
NAME               TYPE  VALUE
------------------- -------- ------------------------------
local_listener   string    EBSDEV_LOCAL

(or, SQL> alter system set local_listener='EBSDEV_LOCAL' scope=both;)

Disable AOL Audit Trail (conditional):
SQL> select distinct enabled from dba_audit_policies;
no rows selected

Fixed object and dictionary statistics: about 10 minutes

SQL> EXECUTE DBMS_STATS.GATHER_FIXED_OBJECTS_STATS;
SQL> EXEC dbms_stats.gather_dictionary_stats;

Collect SYS stats as sysdba (SYSTEM can also do it)
SQL> begin
   dbms_stats.gather_schema_stats (
      'SYS',
      options=>'GATHER STALE',
      estimate_percent => DBMS_STATS.AUTO_SAMPLE_SIZE,
      method_opt => 'FOR ALL COLUMNS SIZE AUTO',
      cascade => TRUE);
end;

Get the ORACLE_HOME path if necessary:
SQL> var OH varchar2(200);
SQL> EXEC dbms_system.get_env('ORACLE_HOME', :OH) ;
PL/SQL procedure successfully completed.
SQL> PRINT OH
OH
--------------------------------------------------------------------------------
/ohome/u01/app/oracle/product/12.0.0.0/db_ebsdev

5. Disable triggers in custom schemas (Seems optional?):
col TRIGGER_NAME for a36;
col TABLE_OWNER for a27;
col TABLE_NAME for a36;
col STATUS for a12;
set lines 333 pages 123;
SQL> select owner,trigger_name,status,table_owner from dba_triggers 
where status='ENABLED' and owner not in ('APPS','SYS','MDSYS');

SQL> select owner,trigger_name,status,table_owner from dba_triggers where owner in
(select username from dba_users where default_tablespace not in ('APPS_TS_TX_DATA','SYSAUX'));

SQL> select 'alter trigger '||owner||'.'||trigger_name||' disable;' from dba_triggers 
where owner like 'XX%';

Note: not sure how to use $ALR_TOP/patch/115/sql/alrdtrig.sql

6. Important step: modify database profiles to avoid errors during installations (assume APPS uses TRUSTED_ID_NO_EXPIRE profile and Installer creates new users of R12.2, such as GHG, to use DEFAULT profile).
alter profile default limit PASSWORD_VERIFY_FUNCTION null;
alter profile default limit FAILED_LOGIN_ATTEMPTS UNLIMITED;
alter profile TRUSTED_ID_NO_EXPIRE limit FAILED_LOGIN_ATTEMPTS UNLIMITED;
alter profile TRUSTED_ID_NO_EXPIRE limit PASSWORD_REUSE_TIME UNLIMITED;
alter profile TRUSTED_ID_NO_EXPIRE limit PASSWORD_REUSE_MAX UNLIMITED;
alter profile TRUSTED_ID_NO_EXPIRE limit PASSWORD_VERIFY_FUNCTION null;

col profile for a27;
col resource_name for a36;
col limit for a27;
SQL> select profile,resource_name,limit from dba_profiles
where profile in ('DEFAULT','TRUSTED_ID_NO_EXPIRE')
and  resource_type = 'PASSWORD'  order by 1,2;

PROFILE                                 RESOURCE_NAME                           LIMIT
------------------------------------ ---------------------------------------------  -----------------
DEFAULT                                FAILED_LOGIN_ATTEMPTS          UNLIMITED
DEFAULT                                PASSWORD_GRACE_TIME            7
DEFAULT                                PASSWORD_LIFE_TIME                 360
DEFAULT                                PASSWORD_LOCK_TIME               UNLIMITED
DEFAULT                                PASSWORD_REUSE_MAX              24
DEFAULT                                PASSWORD_REUSE_TIME             365
DEFAULT                                PASSWORD_VERIFY_FUNCTION NULL

TRUSTED_ID_NO_EXPIRE FAILED_LOGIN_ATTEMPTS           UNLIMITED
TRUSTED_ID_NO_EXPIRE PASSWORD_GRACE_TIME             UNLIMITED
TRUSTED_ID_NO_EXPIRE PASSWORD_LIFE_TIME                  UNLIMITED
TRUSTED_ID_NO_EXPIRE PASSWORD_LOCK_TIME               1
TRUSTED_ID_NO_EXPIRE PASSWORD_REUSE_MAX              UNLIMITED
TRUSTED_ID_NO_EXPIRE PASSWORD_REUSE_TIME             UNLIMITED
TRUSTED_ID_NO_EXPIRE PASSWORD_VERIFY_FUNCTION NULL

SQL> select profile, resource_name, limit from dba_profiles where profile in 
(select profile from dba_users where username = 'APPS')
and resource_type = 'PASSWORD' order by resource_name;

7. ETCC: The EBS Technology Code Level Checker (ETCC) (See Doc ID 2008451.1)
Download patch 17537119, and unzip p17537119_R12_GENERIC.zip to a folder.

DBA can run checkDBpatch.sh, also called DBCC, on database server. It uses opatch to check installed patches in database and may come out about 20+ db patches required. But checkMTpatch.sh will not work on Apps nodes until Rapidwiz completes the installation.

$ ./checkDBpatch.sh contextfile= xxxxx.xml

Using context file from command line argument: xxxxx.xml
Starting Database EBS Technology Codelevel Checker, Version 120.81
Tue Oct 20 12:56:16 EDT 2020
Log file for this session : /u01/db_ebsdev/appsutil/etcc/log/checkDBpatch_65979616.log
Identifying database release.
Database release set to 12.1.0.2.
Connecting to database.
Database connection successful.
Identifying APPS and APPLSYS schema names.
 - APPS schema : APPS
 - APPLSYS schema : APPLSYS
Checking for DB-ETCC results table.
Creating DB-ETCC results table.
Created the table to store DB-ETCC results.
Checking if InMemory option is enabled.
InMemory option is not enabled in the database.
Bugfix file ./db/onprem/txk_R1220_DB_base_bugs.xml : 120.0.12020000.56
This file will be used for identifying missing bugfixes.
Mapping file ./db/onprem/txk_R1220_DB_mappings.xml : 120.0.12020000.31
This file will be used for mapping bugfixes to patches.
[WARNING] DB-ETCC: Bugfix XML file (./db/onprem/txk_R1220_DB_base_bugs.xml) is more than 30 days old.
 Check if a newer version is available in patch 17537119.
Identified RDBMS DST version 19.
Checking Bugfix XML file for 12.1.0.2_PSU
Obtained list of bugfixes to be applied and the list to be rolled back.
Now checking Database ORACLE_HOME.
DB-ETCC is compatible with this opatch version.
Found patch records in the inventory.
Checking Mapping XML file for 12.1.0.2.200714
  Missing Bugfix: 8975044  ->  Patch 8975044
  Missing Bugfix: 14666816  ->  Patch 30614876
  Missing Bugfix: 15894842  ->  Patch 15894842
  Missing Bugfix: 18485835  ->  Patch 30216977
  Missing Bugfix: 18689530  ->  Patch 22098146
  Missing Bugfix: 18881811  ->  Patch 25599890
  Missing Bugfix: 19239846  ->  Patch 31469752
  Missing Bugfix: 19472320  ->  Patch 19472320
  Missing Bugfix: 19509982  ->  Patch 30216977
  Missing Bugfix: 20123899  ->  Patch 20123899
  Missing Bugfix: 20181016  ->  Patch 20181016
  Missing Bugfix: 20476776  ->  Patch 30614876
  Missing Bugfix: 20887355  ->  Patch 20887355
  Missing Bugfix: 21028698  ->  Patch 22098146
  Missing Bugfix: 21321429  ->  Patch 21321429
  Missing Bugfix: 21614112  ->  Patch 30216977
  Missing Bugfix: 21751519  ->  Patch 30614876
  Missing Bugfix: 21813400  ->  Patch 31469752
  Missing Bugfix: 21864513  ->  Patch 21864513
  Missing Bugfix: 21904072  ->  Patch 21904072
  Missing Bugfix: 21967332  ->  Patch 21967332
  Missing Bugfix: 22229581  ->  Patch 30614876
  Missing Bugfix: 22452653  ->  Patch 30614876
  Missing Bugfix: 22666802  ->  Patch 31469752
  Missing Bugfix: 22737974  ->  Patch 30614876
  Missing Bugfix: 23604553  ->  Patch 23604553
  Missing Bugfix: 23632545  ->  Patch 31469752
  Missing Bugfix: 23645516  ->  Patch 31469752
  Missing Bugfix: 25139545  ->  Patch 25139545
  Missing Bugfix: 25305405  ->  Patch 25305405
  Missing Bugfix: 25906117  ->  Patch 25906117
  Missing Bugfix: 25971286  ->  Patch 31469752
  Missing Bugfix: 26742604  ->  Patch 30614876
  Missing Bugfix: 26798411  ->  Patch 26798411
  Missing Bugfix: 29125374  ->  Patch 30216977
Generating Patch Recommendation Summary.
========================================================
PATCH RECOMMENDATION SUMMARY
========================================================
The default patch recommendations to install these missing bugfixes are:
-------------------------------------------------------------------------------
Oracle Database Release 12.1.0.2  (PATCHSET UPDATE 12.1.0.2.200714)
-------------------------------------------------------------------------------
  Patch 8975044 [12.1.0.2.200414 version]
    - Filename: p8975044_12102200414_AIX64-5L.zip
  Patch 30614876 [12.1.0.2.171017 version]
    - Filename: p30614876_12102171017_AIX64-5L.zip
  Patch 15894842 [12.1.0.2.200714 version]
    - Filename: p15894842_12102200714_Generic.zip
  Patch 30216977 [12.1.0.2.191015 version]
    - Filename: p30216977_12102191015_AIX64-5L.zip
  Patch 22098146
    - Filename: p22098146_121020_AIX64-5L.zip
  Patch 25599890
    - Filename: p25599890_121020_Generic.zip
  Patch 31469752 [12.1.0.2.200714 version]
    - Filename: p31469752_12102200714_AIX64-5L.zip
  Patch 19472320
    - Filename: p19472320_121020_AIX64-5L.zip
  Patch 20123899 [12.1.0.2.5 version]
    - Filename: p20123899_121025_Generic.zip
  Patch 20181016
    - Filename: p20181016_121020_AIX64-5L.zip
  Patch 20887355
    - Filename: p20887355_121020_Generic.zip
  Patch 21321429 [12.1.0.2.170117 version]
    - Filename: p21321429_12102170117_Generic.zip
  Patch 21864513
    - Filename: p21864513_121020_AIX64-5L.zip
  Patch 21904072
    - Filename: p21904072_121020_AIX64-5L.zip
  Patch 21967332 [12.1.0.2.160419 version]
    - Filename: p21967332_12102160419_AIX64-5L.zip
  Patch 23604553 [12.1.0.2.200414 version]
    - Filename: p23604553_12102200414_Generic.zip
  Patch 25139545 [12.1.0.2.190416 version]
    - Filename: p25139545_12102190416_Generic.zip
  Patch 25305405 [12.1.0.2.160119 version]
    - Filename: p25305405_12102160119_AIX64-5L.zip
  Patch 25906117
    - Filename: p25906117_121020_AIX64-5L.zip
  Patch 26798411 [12.1.0.2.190716 version]
    - Filename: p26798411_12102190716_AIX64-5L.zip

Apply the required patches and rerun this script
You should check the patch READMEs for minimum opatch version requirements.
The latest opatch is available from My Oracle Support via Patch 6880880.
+-----------------------------------------------------------------------------+
A consolidated zip file with the required patches for Database release
12.1.0.2.200714 is available on My Oracle Support via:
  Patch 31381618 [12.1.0.2.200714 version]
    - EBS RELEASE 12.2 CONSOLIDATED DATABASE FIXES FOR JUL 2020
Note: This zip does not include database Release Updates, bundles or PSUs.
+-----------------------------------------------------------------------------+
See Doc ID 1147107.1 for any special instructions for these patches.
Note: Footnotes in Doc ID 1594274.1 also apply to corresponding overlay patches.
Stored Technology Codelevel Checker results in the database EBSDEV successfully.
Finished checking fixes for Oracle Database: Tue Oct 20 12:56:49 EDT 2020
Log file for this session: /u01/db_ebsdev/appsutil/etcc/log/checkDBpatch_65979616.log

DBA has to apply patches reported from checkDBpatch.sh and run the script repeatedly until the list is cleared.

Database patches applied.
ETCC ran successfully
The opatch utility is version 12.2.0.1.21.
DB-ETCC is compatible with this opatch version.
Found patch records in the inventory.
Checking Mapping XML file for 12.1.0.2.200714
All the required one-off bugfixes are present in Database ORACLE_HOME.


8. If database is upgraded to 12c before EBS R12.2 upgrade, complete steps in Doc ID 1524398.1 (Interoperability Notes EBS 12.0 or 12.1 with RDBMS 12cR1).

Below steps might not be executed previously. It is necessary run them now to avoid error from ADZDPREP.sql in applying R12.2 online patch 13543062.

Post DB upgrade steps:
   # add SQLNET.ALLOWED_LOGON_VERSION_SERVER=8 in sqlnet.ora
   #. run scripts: It seems it does not run on AEBSGOLD/AEBSDEV2
                sqlplus / as sysdba
                SQL> @?/rdbms/admin/dbmsxdbschmig.sql
                SQL> @?/rdbms/admin/prvtxdbschmig.plb
                SQL> @?/rdbms/admin/utlrp.sql

- Run Rapid Installer to lay down R12.2.0 software ("Create Upgrade File System")

1. Backup APPS file system and database
2. Make sure R12.1 works and then stop all services
$ ps -ef | grep $LOGNAME
3. If pre-installation failed, you have to delete all related files in /tmp, and folders fs1, fs2, fs_ne. (See Doc ID 2091880.1: 12.2 E-Business Suite - How To Clean Up a Rapid Install Failure and Restart a Fresh Install).
4. If previous installation worked successfully and you want to re-run rapidwiz from the beginning, you have to delete the inventory folder defined in oraInst.loc file in addition to step 2 above to avoid below strange errors. Then, re-create the folder and attach two R12.1 ORACLE_HOMEs to it.
Fatal Error: TXK Install Service
oracle.apps.fnd.txk.config.ProcessStateException: OPatch process failed : Exit=160
make[1]: *** [/path/to/fs2/EBSapps/10.1.2/sysman/lib/libnmemso.so] Error 1

4. Launch the installer
$ vi .profile   <== and re-login (NOT to use current R12.1 env file)
$ env             <== make sure no Oracle related and strange env values
$ env | grep OPATCH_PLATFORM_ID
Make sure nothing is set for this to avoid Rapidwiz fatal error with patch 6275529 oracle.apps.fnd.txk.config.ProcessStateException (Doc ID 2336585.1)
$ more /etc/oraInst.loc
inventory_loc=/path/to/oraEBSdevInventory
S mv /path/to/oraEBSdevInventory /path/to/oraEBSdevInventory_R12.1
$ mkdir /path/to/oraEBSdevInventory    
  Notes: R12.1 file folders can be renamed after R12.2.10 is good and up.
$ export DISPLAY=xxx.xx.xxx:0.0     <== assume Exceed is installed on the desktop
$ cd startCD/Disk1/rapidwiz

$ ./rapidwiz
choose "Upgrade to Oracle E-Business Suite Release 12.2.0"
            "Create Upgrade File System"
enter 41 & 42 (if database uses port 1562) in port pool
enter database info such as $ORACLE_BASE & $ORACLE_HOME from database node, and 
choose "Use Existing Oracle Home (No new Database Home install)"
System check marks shall all be green, before continue to "Next".

The installation GUI stays at 0% (for about 30 minutes), 40% for a while and takes totally about 3 hours. It will post screen messages telling installation log location $INST_TOP/logs
Configuration file written to: /…/EBSDEV/fs1/inst/apps/<CONTEXT_NAME>/conf_EBSEV.txt
Configuration file written to: /…/EBSDEV/fs2/inst/apps/<CONTEXT_NAME>/conf_EBSDEV.txt
Second File System logfile - /…/EBSDEV/fs2/inst/apps/<CONTEXT_NAME>/logs/10212202.log
First File System logfile - /…/EBSDEV/fs1/inst/apps/<CONTEXT_NAME>/logs/10212202.log


The .log file size shall be about 30k (under /fs1/) and the end of log file has entries (replace R12.2 env variables):
… … 
Stopping the Service - /bin/sh $ADMIN_SCRIPTS_HOME/adnodemgrctl.sh  stop
... ...
You are running adnodemgrctl.sh version 120.11.12020000.4
Enter the WebLogic Admin password:
Either the Node Manager is not up or the credentials are incorrect.
adnodemgrctl.sh: exiting with status 2
adnodemgrctl.sh: check the logfile $INST_TOP/logs/appl/admin/log/adnodemgrctl.txt for more information ...
stty: standard input: Inappropriate ioctl for device
stty: standard input: Inappropriate ioctl for device
Upgrade File
-------------
   APPL_TOP environment file passes instantiated variables test:
      File = $APPL_TOP/<CONTEXT_NAME>.env
   ADOVARS environment file passes instantiated variables test:
      File = $APPL_TOP/admin/adovars.env
   APPSCONFIG passes instantiated variables test:
      File = $APPL_TOP/admin/adconfig.txt
DoInstallPanel - Summary Text
The Rapid Install Wizard will now install the following:

If go to Apps Base Directory specified on the installation GUI, 3 new folders are created there:
$ du -sh fs*
28G     fs1
28G     fs2
32K     fs_ne

$ ps -ef | grep $LOGNAME
Nothing shall run.

Choose the RUN file system by run the .env file under /fs1/EBSapps/appl (or /fs2/EBSapps/appl) to set up R12.2 environment for continuing next steps. 

$ cd <Apps Base Directory>/fs1/EBSapps/appl
$ . ./APPS<TWO_TASK>_<node_name>.env
$ echo $FILE_EDITION
run
$ echo $APPL_TOP
<Apps Base Directory>/fs1/EBSapps/appl

Now, The installation is completed.

Wednesday, September 30, 2020

Run RDA for health check before Oracle software installation

When I run R12.2 rapidwiz to install R12.2 software, the pre-install test reports failures. "Web Server Install Prerequisites" log gives message:

Check Name:CertifiedVersions
Check Description:This is a prerequisite condition to test whether the Oracle software is certified on the current O/S or not.
Check complete. 
The overall result of this check is: Not executed
... ...
Summary : 0 requirements failed, 5 requirements to be verified.

It does not tell the details on what OS requirements are not met, maybe due to failure on multiple requirements. RDA (Remote Diagnostic Agent) is an Oracle tool for health check and reports more details. Doc ID 250262.1 (RDA - Health Check / Validation Engine Guide) is a long document. But really there are just a few steps to run it.

1. Download  from Doc ID 314422.1 (Remote Diagnostic Agent - Getting Started):
select Linux x86 (64-bit) bundle to get ZIP file p21769913_20220421_Linux-x86-64.zip.
Note: the patch number is 21769913, and at this time 20.2.20.4.21 is the latest release. Oracle updates it from time to time.

2. unzip  p21769913_20220421_Linux-x86-64.zip    <= It will create /rda directory
3. Verify RDA works
$ ./rda.sh -cv
It shall say "No issues found"
$ perl -V
Summary of my perl5 (revision 5.0 version 8 subversion 3) configuration:
… … a long list of info
4. Run it
$ perl rda.pl -T hcve
Processing HCVE tests ...
Available Pre-Installation Rule Sets:
   1.  Oracle Database 10g R1 (10.1.0) Preinstall (Linux)
   2.  Oracle Database 10g R2 (10.2.0) Preinstall (Linux)
   3.  Oracle Database 11g R1 (11.1) Preinstall (Linux)
   4.  Oracle Database 11g R2 (11.2.0) Preinstall (Linux)
   5.  Oracle Database 12c R1 (12.1.0) Preinstallation (Linux)
   6.  Oracle Database 12c R2 (12.2.0) Preinstallation (Linux)
   7.  Oracle Database 18c Preinstallation (Linux)
   8.  Oracle Database 19c Preinstallation (Linux)
   9.  Oracle Identity and Access Management PreInstall Check: Oracle Identity
       and Access Management 11g Release 2 (11.1.2) Linux
  10.  Oracle JDeveloper PreInstall Check: Oracle JDeveloper 11g Release 2
       (11.1.2.4) Linux
  11.  Oracle JDeveloper PreInstall Check: Oracle JDeveloper 12c (12.1.3)
       Linux
  12.  OAS PreInstall Check: Application Server 10g R2 (10.1.2) Linux
  13.  OAS PreInstall Check: Application Server 10g R3 (10.1.3) Linux
  14.  OFM PreInstall Check: Oracle Fusion Middleware 11g R1 (11.1.1) Linux
  15.  OFM PreInstall Check: Oracle Fusion Middleware 12c (12.1.3) Linux
  16.  OFM PreInstall Check: Oracle Fusion Middleware 12c (12.2.1.3.0) Linux
  17.  Oracle Forms and Reports PreInstall Check: Oracle Forms and Reports 11g
       Release 2 (11.1.2) Linux
  18.  Portal PreInstall Check: Oracle Portal Generic
  19.  IDM PreInstall Check: Identity Management 10g (10.1.4) Linux
  20.  BIEE PreInstall Check: Business Intelligence Enterprise Edition 11g
       (11.1.1) Linux
  21.  EPM PreInstall Check: Enterprise Performance Management Server (11.1.2)
       Generic
  22.  Oracle Enterprise Manager Cloud Control PreInstall Check: Oracle
       Enterprise Manager Cloud Control 12c Release 4 (12.1.0.4) Linux
  23.  Oracle E-Business Suite Release 11i (11.5.10) Preinstall (Linux x86 and
       x86_64)
  24.  Oracle E-Business Suite Release 12 (12.1.1) Preinstall (Linux x86 and
       x86_64)
  25.  Oracle E-Business Suite Release 12 (12.2.0) Preinstall (Linux x86_64)
Available Post-Installation Rule Sets:
  26.  RAC 10G DB and OS Best Practices (Linux)
  27.  Data Guard Postinstall (Generic)
  28.  WLS PostInstall Check: WebLogic Server 11g (10.3.x) Generic
  29.  WLS PostInstall Check: WebLogic Server 12c (12.x) Generic
  30.  Portal PostInstall Check: Oracle Portal Generic
  31.  OC4J PostInstall Check: Oracle Containers for J2EE 10g (10.1.x) Generic
  32.  SOA PostInstall Check: Service-Oriented Architecture 11g and Later Generic
  33.  OSB PostInstall Check: Service Bus 11g and Later Generic
  34.  Oracle Forms 11g Post Installation (Generic)
  35.  Oracle Enterprise Manager Agent 12c Post Installation (Generic)
  36.  Oracle Management Server 12c Post Installation (Generic)
  37.  Network Charging and Control Database Post Installation (Generic)
Enter the HCVE rule set number or 0 to cancel the test
Press Return to accept the default (0)
> 25

Performing HCVE checks ...
Enter value for < Planned application tier installation directory >
> /u04/app/EBSDEV

Test "Oracle E-Business Suite Release 12 (12.2.0) Preinstall (Linux x86_64)" executed at 18-Aug-2020 00:44:48
Test Results
~~~~~~~~~~~~
ID     NAME                 RESULT  VALUE
====== ========= ======= ==========================================
A00100 OS Type                        RECORD  RH7 64
A00200 OS Certified?                PASSED  Adequate
A01010 ApplTierDirectory         RECORD  /u04/app/EBSDEV
A01020 A_T Valid?                    PASSED  ATexists
A01030 A_T Permissions OK?  PASSED  CorrectPerms
A01040 A_T Disk Space            PASSED  OK
A01400 Got Software Tools?     PASSED  tools_found
A02030 Limit Processes             PASSED  Adequate
A02050 Limit Descriptors          PASSED  Adequate
A02100 ENV Variable Unset     SKIPPED Not SuSE Linux Enterprise 10 or SuSE ...
A02210 Kernel Params OK?      FAILED  AIOMAXTooSmall FILEMAXTooSmall WMEM_M...
A02240 NPTL Selected?            PASSED  NPTL is default
A03010 Space in tmp                 PASSED  Available
A03050 Swap Space (MB)         RECORD  8191.99609375
A03060 Swap Space?                 FAILED  Need at least 16 GB
A03510 IP Address                     RECORD  157.1xx.xx.xx
A03530 Domain Name               RECORD  domain.com
A03540 /etc/hosts format            PASSED  Adequate IPv4 entry
A03550 DNS Lookup                 PASSED  Host correctly registered in DNS
A03560 Net Service Access?      PASSED  PermOk
A03570 Port 6000                       FAILED  Occupied
A03580 Port Range OK?            FAILED  LowValue HighValue
A03590 DNS Settings                 FAILED  ATTEMPTSUndef TIMEOUTTooSmall
A03600 SysNetw File                 FAILED  Missing host.domain
A03610 NoNetwProf File          PASSED  OK
A04301 RPM OL5/64 OK?       SKIPPED Not Oracle Linux 5 64-bit
A04302 RPM OL6/64 OK?       SKIPPED Not Oracle Linux 6 64-bit
A04303 RPM OL7/64 OK?       SKIPPED Not Oracle Linux 7 64-bit
A04311 RPM RH5/64 OK?       SKIPPED Not Red Hat Enterprise Linux 5 64-bit
A04312 RPM RH6/64 OK?       SKIPPED Not Red Hat Enterprise Linux 6 64-bit
A04313 RPM RH7/64 OK?       FAILED  [compat-libstdc++-296(i686)] not inst...
A04321 RPM SLES10/64 OK?    SKIPPED Not SuSE Linux Enterprise 10 64-bit
A04322 RPM SLES11/64 OK?    SKIPPED Not SuSE Linux Enterprise 11 64-bit
Result file: output/collect/APPS_HCVE_A_EBS122_lin_res.htm

Notes: above is just summary on the screen. You have to open file output/collect/APPS_HCVE_A_EBS122_lin_res.htm in browser to see all details.

After Linux Admin installs more packages or makes some changed on OS settings, R12.2 rapidwiz shall pass the verification and gave all green checkmarks even rda reports some failures in a case:

Test Results
~~~~~~~~~~~~
ID     NAME                 RESULT  VALUE
====== ========= ======= ========================================
A00100 OS Type                        RECORD  RH7 64
A00200 OS Certified?                PASSED  Adequate
A01010 ApplTierDirectory         RECORD  /u04/app/EBSDEV
A01020 A_T Valid?                     PASSED  ATexists
A01030 A_T Permissions OK?   PASSED  CorrectPerms
A01040 A_T Disk Space             PASSED  OK
A01400 Got Software Tools?      PASSED  tools_found
A02030 Limit Processes              PASSED  Adequate
A02050 Limit Descriptors           PASSED  Adequate
A02100 ENV Variable Unset      SKIPPED Not SuSE Linux Enterprise 10 or SuSE ...
A02210 Kernel Params OK?       FAILED  WMEM_MAXTooSmall
A02240 NPTL Selected?             PASSED  NPTL is default
A03010 Space in tmp                  PASSED  Available
A03050 Swap Space (MB)         RECORD  16383.9921875
A03060 Swap Space?                 FAILED  Need at least 16 GB
A03510 IP Address                     RECORD  157.1xx.xx.xx
A03530 Domain Name               RECORD  domain.com
A03540 /etc/hosts format            PASSED  Adequate IPv4 entry
A03550 DNS Lookup                  PASSED  Host correctly registered in DNS
A03560 Net Service Access?      PASSED  PermOk
A03570 Port 6000                       PASSED  Free
A03580 Port Range OK?            FAILED  LowValue HighValue
A03590 DNS Settings                 FAILED  ATTEMPTSUndef TIMEOUTTooSmall
A03600 SysNetw File                 FAILED  Missing host.domain
A03610 NoNetwProf File           PASSED  OK
A04301 RPM OL5/64 OK?        SKIPPED Not Oracle Linux 5 64-bit
A04302 RPM OL6/64 OK?        SKIPPED Not Oracle Linux 6 64-bit
A04303 RPM OL7/64 OK?        SKIPPED Not Oracle Linux 7 64-bit
A04311 RPM RH5/64 OK?        SKIPPED Not Red Hat Enterprise Linux 5 64-bit
A04312 RPM RH6/64 OK?        SKIPPED Not Red Hat Enterprise Linux 6 64-bit
A04313 RPM RH7/64 OK?        FAILED  [compat-libstdc++-296(i686)] not inst...
A04321 RPM SLES10/64 OK?   SKIPPED Not SuSE Linux Enterprise 10 64-bit
A04322 RPM SLES11/64 OK?   SKIPPED Not SuSE Linux Enterprise 11 64-bit
Result file: output/collect/APPS_HCVE_A_EBS122_lin_res.htm

In one instance, after package redhat-lsb was installed, rapidwiz passed verification check from previous failure:
$ rpm -qa redhat-lsb
redhat-lsb-4.1-27.el7.x86_64
redhat-lsb-4.1-27.el7.i686

With below packages are installed, seems compat-libstdc++-296(i686) is not necessary.
$ rpm -qa | grep libstdc
libstdc++-devel-4.8.5-39.el7.x86_64
libstdc++-devel-4.8.5-39.el7.i686
libstdc++-4.8.5-39.el7.x86_64
libstdc++-4.8.5-39.el7.i686
compat-libstdc++-33-3.2.3-72.el7.x86_64
compat-libstdc++-33-3.2.3-72.el7.i686

$ rpm -qa | grep libaio
libaio-0.3.109-13.el7.x86_64
libaio-0.3.109-13.el7.i686

$ rpm -qa | grep libaio-dev
libaio-devel-0.3.109-13.el7.x86_64     <== it is REQUIRED on my Linux 64 bits

Most Kernel parameters are in a file under folder /etc/sysctl.d. Here is a custom file:
$ more /etc/sysctl.d/90-oracle.conf
fs.aio-max-nr = 1048576
fs.file-max = 6815744
kernel.shmall = 2097152
kernel.shmmax = 4294967295
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576

$ more /proc/sys/fs/file-max
6815744

Explanation on port 6000 conflict (See Doc ID 761566.1):

rapidwiz uses port 6000, because the OPMN service of the Application Server technology stack listens on port 6000 by default when started up during Rapid InstallThis could conflict with the X11 port used for the graphics console on Linux servers and prevent Rapid Install from completing.

To address this issue, perform the following steps before installing or upgrading:
. Check if the port 6000 is being used "netstat -a | grep 6000" or "netstat -tunap | grep -i 6000"
. If so, you should disable the graphics console login process on the Linux server by logging in as root and issuing the following commands:
 - Find the process that uses port 6000 by the command "lsof -i TCP:6000".
 - Use "kill -9" to abort the process found in the previous step
 - Confirm that nothing is listening at port 6000 ("netstat -a | grep 6000")
 - Modify /etc/inittab by changing "id:5:initdefault:" to "id:3:initdefault:"
 - Run "/sbin/init 3" to restart dtlogin

Wednesday, September 23, 2020

R12.2 software and StartCD 51 for repaid installation

The first task in upgrading EBS to R12.2 is to get the software ready. But it is not straight and easy. Here are steps to get started.

1. Create folder /path/to/R122, and add at lease 100GB disk space to /path/to partition.

2. Download ZIP files to /path/to/R122
See 12.2 E-Business Suite Applications Installation Rapid Install Steps To Create the Stage Directories In Preparation For A 12.2 Installation (Doc ID 1596433.1)

https://edelivery.oracle.com/osdc/faces/SoftwareDelivery
Click Popular Downloads => REL: Oracle E-Business Suite 12.2.0 Current => 23 ZIP files
Click on Download to get ORACLE DOWNLOAD MANAGER (then, enter your login info).

Oracle E-Business Suite 12.2.0 Current for Linux x86-64:
V35803-01_1of3.zip EBS R 12.2.0 for Linux x86-64 Rapid Install APPL_TOP - Disk 1, 1.2 GB
V35803-01_2of3.zip EBS R 12.2.0 for Linux x86-64 Rapid Install APPL_TOP - Disk 1, 1022.8 MB
V35803-01_3of3.zip EBS R 12.2.0 for Linux x86-64 Rapid Install APPL_TOP - Disk 1, 1.3 GB
V35804-01_1of2.zip EBS R 12.2.0 for Linux x86-64 Rapid Install APPL_TOP - Disk 2, 1.2 GB
V35804-01_2of2.zip EBS R 12.2.0 for Linux x86-64 Rapid Install APPL_TOP - Disk 2, 1.6 GB
V35802-01.zip  EBS R 12.2.0 for Linux x86-64 Rapid Install AS10.1.2, 711.6 MB
V100060-01_1of3.zip EBS R 12.2.0 for Linux x86-64 Rapid Install DBs PROD - Disk 2, 1.6 GB
V100060-01_2of3.zip EBS R 12.2.0 for Linux x86-64 Rapid Install DBs PROD - Disk 2, 1.1 GB
V100060-01_3of3.zip EBS R 12.2.0 for Linux x86-64 Rapid Install DBs PROD - Disk 2, 1.0 GB
V100061-01.zip  EBS R 12.2.0 for Linux x86-64 Rapid Install Technology one-off Patches, 839.9 MB
V100054-01.zip  EBS R 12.2.0 for Linux x86-64 Rapid Install Databases VISION - Disk 2, 3.0 GB
V100055-01.zip  EBS R 12.2.0 for Linux x86-64 Rapid Install Databases VISION - Disk 3, 2.8 GB
V100053-01_1of2.zip EBS R 12.2.0 for Linux x86-64 Rapid Install DBs VISION - Disk 1, 52.0 MB
V100053-01_2of2.zip EBS R 12.2.0 for Linux x86-64 Rapid Install DBs VISION - Disk 1, 3.4 GB
V100058-01.zip  EBS R 12.2.0 for Linux x86-64 Rapid Install DBs VISION - Disk 6, 2.0 GB
V100059-01_1of2.zip EBS R 12.2.0 for Linux x86-64 Rapid Install DBs PROD - Disk 1, 1007.0 MB
V100059-01_2of2.zip EBS R 12.2.0 for Linux x86-64 Rapid Install DBs PROD - Disk 1, 1.6 GB
V100056-01.zip  EBS R 12.2.0 for Linux x86-64 Rapid Install Databases VISION - Disk 4, 2.6 GB
V100057-01.zip  EBS R 12.2.0 for Linux x86-64 Rapid Install Databases VISION - Disk 5, 2.4 GB
V100102-01.zip  Oracle DB 12c Release 1 (12.1.0.2.0) EXAMPLES for Linux x86-64, 476.1 MB

If patch file p22066363_R12_GENERIC.zip is downloaded, below three files are not needed:

V100052-01_1of3.zip EBS R 12.2.0 Rapid Install Start Here, 725.3 MB
V100052-01_2of3.zip EBS R 12.2.0 Rapid Install Start Here, 81.6 MB
V100052-01_3of3.zip EBS R 12.2.0 Rapid Install Start Here, 97.3 MB

Go to https://www.oracle.com/database/technologies/database12c-linux-downloads.html
to download Oracle Database 12c Release 1 (12.1.0.2.0) for Linux x86-64 (and then rename them to V46095-01_xxxx.zip)
V46095-01_1of2.zip (linuxamd64_12102_database_1of2.zip)
V46095-01_2of2.zip  (linuxamd64_12102_database_2of2.zip)

Then, follow instruction to download four ZIP files for Oracle Coherence 3.7.1.0.0 and one ZIP file for Oracle Web Tier 11.1.1.9.0
V29856-01.zip Oracle WebLogic Server 10.3.6 (11gR1) Generic with Coherence 3.7.1, 1018.5 MB
V27954-01.zip Oracle Coherence 3.7.1.0.0 for C++ for Linux x86-64, 21.1 MB
V27953-01.zip Oracle Coherence 3.7.1.0.0 for the Microsoft .NET Framework, 17.0 MB
V28274-01.zip Oracle Coherence 3.7.1.0.0, 15.7 MB
V75792-01.zip Oracle Web Tier 11.1.1.9.0 Utilities (11g Patch Set 7) - Oracle Fusion Middleware for Linux x86-64, 2.0 GB

Totally, 30 ZIP files.

3. Create folder /path/to/R122/rapidSTARTCD, and download p22066363_R12_GENERIC.zip (Patch 22066363: RAPID INSTALL STARTCD 12.2.0.51, StartCD 51) to this new folder.

4. Unzip p22066363_R12_GENERIC.zip creates folder startCD.
NOTE: Do not manually unzip the other stage files.  After unzipping the startCD, buildStage.sh script should be run to unzip the rest of the installation software.  

5. Build stage area  -- it needs about 45GB to unzip files
$ cd startCD/Disk1/rapidwiz/bin
(or, /path/to/stageR122/rapidSTARTCD/startCD/Disk1/rapidwiz/bin)
$ ./buildStage.sh

Two actions are necessary.

                      Build Stage Menu
   ------------------------------------------------------
   1.     Create new stage area
   2.     Copy patches to existing stage area

   Enter your choice [4]: 1
… …
Specify the directory containing the zipped installation media: /path/to/stageR122

Verifying stage area...
Directory /path/to/stageR122/rapidSTARTCD/TechInstallMedia is valid.
Directory /path/to/stageR122/rapidSTARTCD/TechPatches/DB is valid.
Directory /path/to/stageR122/rapidSTARTCD/TechPatches/MiddleTier is valid.
Directory /path/to/stageR122/rapidSTARTCD/EBSInstallMedia/AppDB is valid.
Directory /path/to/stageR122/rapidSTARTCD/EBSInstallMedia/Apps is valid.
Directory /path/to/stageR122/rapidSTARTCD/EBSInstallMedia/AS10.1.2 is valid.
Directory /path/to/stageR122/rapidSTARTCD/TechInstallMedia/database is valid.
Directory /path/to/stageR122/rapidSTARTCD/TechInstallMedia/ohs11119 is valid.
Directory
/path/to/stageR122/rapidSTARTCD/TechInstallMedia/wls1036_generic is valid.

If all worked, you shall see "Stage area verified."  by the end.

It unzip files to /path/to/stageR122/rapidSTARTCD and creates 3 sub-folders EBSInstallMedia, TechInstallMedia, TechPatches

6. Download patch 25525148 p25525148_R12_GENERIC.zip, the consolidated fixes on top of Patch 22066363, to /path/to/stageR122/rapidSTARTCD
(If database 19c is used, it needs another patch 29963728. See Oracle E-Business Suite Release Notes, Release 12.2 (Doc ID 1320300.1) ).

$ unzip p25525148_R12_GENERIC.zip
$ cd 25525148

$ ./patchRIStage.sh
Enter the location of the Rapid Install Stage area: /path/to/stageR122/rapidSTARTCD

$ cd ../
$ cd startCD/Disk1/rapidwiz/bin
$ ./buildStage.sh
                    Build Stage Menu
   ------------------------------------------------------
   1.     Create new stage area
   2.     Copy patches to existing stage area
   3.     List files in TechPatches directory
   4.     Exit menu
   Enter your choice [4]: 2
… … 
Copying latest one-off patches to stage area...

Finished copying additional patches.

7. Now, Rapid Install CDs are ready, you can go to folder /path/to/stageR122 to tar folder rapidSTARTCD up to a file and copy it to other servers for installation if needed.

8. Test rapidwiz 

Stay at /path/to/stageR122/rapidSTARTCD/startCD/Disk1/rapidwiz

$ vi /etc/oraInst.loc
$ export DISPLAY=xx.xx.xxx.xxx:0.0
$ ./rapidwiz -version
Rapid Install Wizard is validating your file system......
Rapid Install Wizard will now launch the Java Interface.....
Oracle E-Business Suite Rapid Install Wizard
Version 12.2.0.51           <=
startCD 51
(c) Copyright 2000-2011 Oracle Corporation.  All rights reserved.

Now, R12.2 software is ready for Rapidwiz.  Rapid Install will validate the system configuration and tell what pre-steps on OS and database are required for the installation.

$ ./rapidwiz -help
Rapid Install Wizard is validating your file system......
CMDDIR=/path/to/stageR122/rapidSTARTCD/startCD/Disk1/rapidwiz
Rapid Install Wizard will now launch the Java Interface.....