ERP on DB: September 2020

Wednesday, September 30, 2020

Run RDA for health check before Oracle software installation

When I run R12.2 rapidwiz to install R12.2 software, the pre-install test reports failures. "Web Server Install Prerequisites" log gives message:

Check Name:CertifiedVersions
Check Description:This is a prerequisite condition to test whether the Oracle software is certified on the current O/S or not.
Check complete. The overall result of this check is: Not executed

... ...

Summary : 0 requirements failed, 5 requirements to be verified.

It does not tell the details on what OS requirements are not met, maybe due to failure on multiple requirements. RDA (Remote Diagnostic Agent) is an Oracle tool for health check and reports more details. Doc ID 250262.1 (RDA - Health Check / Validation Engine Guide) is a long document. But really there are just a few steps to run it.

1. Download from Doc ID 314422.1 (Remote Diagnostic Agent - Getting Started):

select Linux x86 (64-bit) bundle to get ZIP file p21769913_20220421_Linux-x86-64.zip.

Note: the patch number is 21769913, and at this time 20.2.20.4.21 is the latest release. Oracle updates it from time to time.

2. unzip p21769913_20220421_Linux-x86-64.zip    <= It will create /rda directory
3. Verify RDA works
$ ./rda.sh -cv
It shall say "No issues found"
$ perl -V
Summary of my perl5 (revision 5.0 version 8 subversion 3) configuration:
… … a long list of info
4. Run it
$ perl rda.pl -T hcve
Processing HCVE tests ...
Available Pre-Installation Rule Sets:
   1. Oracle Database 10g R1 (10.1.0) Preinstall (Linux)
   2. Oracle Database 10g R2 (10.2.0) Preinstall (Linux)
   3. Oracle Database 11g R1 (11.1) Preinstall (Linux)
   4. Oracle Database 11g R2 (11.2.0) Preinstall (Linux)
   5. Oracle Database 12c R1 (12.1.0) Preinstallation (Linux)
   6. Oracle Database 12c R2 (12.2.0) Preinstallation (Linux)
   7. Oracle Database 18c Preinstallation (Linux)
   8. Oracle Database 19c Preinstallation (Linux)
   9. Oracle Identity and Access Management PreInstall Check: Oracle Identity
       and Access Management 11g Release 2 (11.1.2) Linux
10. Oracle JDeveloper PreInstall Check: Oracle JDeveloper 11g Release 2
       (11.1.2.4) Linux
11. Oracle JDeveloper PreInstall Check: Oracle JDeveloper 12c (12.1.3)
       Linux
12. OAS PreInstall Check: Application Server 10g R2 (10.1.2) Linux
13. OAS PreInstall Check: Application Server 10g R3 (10.1.3) Linux
14. OFM PreInstall Check: Oracle Fusion Middleware 11g R1 (11.1.1) Linux
15. OFM PreInstall Check: Oracle Fusion Middleware 12c (12.1.3) Linux
16. OFM PreInstall Check: Oracle Fusion Middleware 12c (12.2.1.3.0) Linux
17. Oracle Forms and Reports PreInstall Check: Oracle Forms and Reports 11g
       Release 2 (11.1.2) Linux
18. Portal PreInstall Check: Oracle Portal Generic
19. IDM PreInstall Check: Identity Management 10g (10.1.4) Linux
20. BIEE PreInstall Check: Business Intelligence Enterprise Edition 11g
       (11.1.1) Linux
21. EPM PreInstall Check: Enterprise Performance Management Server (11.1.2)
       Generic
22. Oracle Enterprise Manager Cloud Control PreInstall Check: Oracle
       Enterprise Manager Cloud Control 12c Release 4 (12.1.0.4) Linux
23. Oracle E-Business Suite Release 11i (11.5.10) Preinstall (Linux x86 and
       x86_64)
24. Oracle E-Business Suite Release 12 (12.1.1) Preinstall (Linux x86 and
       x86_64)
25. Oracle E-Business Suite Release 12 (12.2.0) Preinstall (Linux x86_64)
Available Post-Installation Rule Sets:
26. RAC 10G DB and OS Best Practices (Linux)
27. Data Guard Postinstall (Generic)
28. WLS PostInstall Check: WebLogic Server 11g (10.3.x) Generic
29. WLS PostInstall Check: WebLogic Server 12c (12.x) Generic
30. Portal PostInstall Check: Oracle Portal Generic
31. OC4J PostInstall Check: Oracle Containers for J2EE 10g (10.1.x) Generic
32. SOA PostInstall Check: Service-Oriented Architecture 11g and Later Generic
33. OSB PostInstall Check: Service Bus 11g and Later Generic
34. Oracle Forms 11g Post Installation (Generic)
35. Oracle Enterprise Manager Agent 12c Post Installation (Generic)
36. Oracle Management Server 12c Post Installation (Generic)
37. Network Charging and Control Database Post Installation (Generic)
Enter the HCVE rule set number or 0 to cancel the test
Press Return to accept the default (0)
> 25

Performing HCVE checks ...
Enter value for < Planned application tier installation directory >
> /u04/app/EBSDEV

Test "Oracle E-Business Suite Release 12 (12.2.0) Preinstall (Linux x86_64)" executed at 18-Aug-2020 00:44:48
Test Results
~~~~~~~~~~~~
ID     NAME                 RESULT VALUE
====== ========= ======= ==========================================
A00100 OS Type    RECORD RH7 64
A00200 OS Certified?    PASSED Adequate
A01010 ApplTierDirectory RECORD /u04/app/EBSDEV
A01020 A_T Valid?    PASSED ATexists
A01030 A_T Permissions OK? PASSED CorrectPerms
A01040 A_T Disk Space    PASSED OK
A01400 Got Software Tools? PASSED tools_found
A02030 Limit Processes PASSED Adequate
A02050 Limit Descriptors    PASSED Adequate
A02100 ENV Variable Unset SKIPPED Not SuSE Linux Enterprise 10 or SuSE ...
A02210 Kernel Params OK?    FAILED AIOMAXTooSmall FILEMAXTooSmall WMEM_M...
A02240 NPTL Selected?    PASSED NPTL is default
A03010 Space in tmp PASSED Available
A03050 Swap Space (MB) RECORD 8191.99609375
A03060 Swap Space? FAILED Need at least 16 GB
A03510 IP Address RECORD 157.1xx.xx.xx
A03530 Domain Name RECORD domain.com
A03540 /etc/hosts format    PASSED Adequate IPv4 entry
A03550 DNS Lookup PASSED Host correctly registered in DNS
A03560 Net Service Access?    PASSED PermOk
A03570 Port 6000 FAILED Occupied
A03580 Port Range OK?    FAILED LowValue HighValue
A03590 DNS Settings FAILED ATTEMPTSUndef TIMEOUTTooSmall
A03600 SysNetw File FAILED Missing host.domain
A03610 NoNetwProf File    PASSED OK
A04301 RPM OL5/64 OK?       SKIPPED Not Oracle Linux 5 64-bit
A04302 RPM OL6/64 OK?       SKIPPED Not Oracle Linux 6 64-bit
A04303 RPM OL7/64 OK?       SKIPPED Not Oracle Linux 7 64-bit
A04311 RPM RH5/64 OK?       SKIPPED Not Red Hat Enterprise Linux 5 64-bit
A04312 RPM RH6/64 OK?       SKIPPED Not Red Hat Enterprise Linux 6 64-bit
A04313 RPM RH7/64 OK?       FAILED [compat-libstdc++-296(i686)] not inst...
A04321 RPM SLES10/64 OK?    SKIPPED Not SuSE Linux Enterprise 10 64-bit
A04322 RPM SLES11/64 OK?    SKIPPED Not SuSE Linux Enterprise 11 64-bit
Result file: output/collect/APPS_HCVE_A_EBS122_lin_res.htm

Notes: above is just summary on the screen. You have to open file output/collect/APPS_HCVE_A_EBS122_lin_res.htm in browser to see all details.

After Linux Admin installs more packages or makes some changed on OS settings, R12.2 rapidwiz shall pass the verification and gave all green checkmarks even rda reports some failures in a case:

Test Results
~~~~~~~~~~~~
ID     NAME                 RESULT VALUE
====== ========= ======= ========================================
A00100 OS Type    RECORD RH7 64
A00200 OS Certified?    PASSED Adequate
A01010 ApplTierDirectory RECORD /u04/app/EBSDEV
A01020 A_T Valid? PASSED ATexists
A01030 A_T Permissions OK? PASSED CorrectPerms
A01040 A_T Disk Space PASSED OK
A01400 Got Software Tools?    PASSED tools_found
A02030 Limit Processes    PASSED Adequate
A02050 Limit Descriptors PASSED Adequate
A02100 ENV Variable Unset    SKIPPED Not SuSE Linux Enterprise 10 or SuSE ...
A02210 Kernel Params OK? FAILED WMEM_MAXTooSmall
A02240 NPTL Selected? PASSED NPTL is default
A03010 Space in tmp    PASSED Available
A03050 Swap Space (MB) RECORD 16383.9921875
A03060 Swap Space? FAILED Need at least 16 GB
A03510 IP Address RECORD 157.1xx.xx.xx
A03530 Domain Name RECORD domain.com
A03540 /etc/hosts format    PASSED Adequate IPv4 entry
A03550 DNS Lookup    PASSED Host correctly registered in DNS
A03560 Net Service Access?    PASSED PermOk
A03570 Port 6000 PASSED Free
A03580 Port Range OK?    FAILED LowValue HighValue
A03590 DNS Settings FAILED ATTEMPTSUndef TIMEOUTTooSmall
A03600 SysNetw File FAILED Missing host.domain
A03610 NoNetwProf File PASSED OK
A04301 RPM OL5/64 OK?    SKIPPED Not Oracle Linux 5 64-bit
A04302 RPM OL6/64 OK?    SKIPPED Not Oracle Linux 6 64-bit
A04303 RPM OL7/64 OK?    SKIPPED Not Oracle Linux 7 64-bit
A04311 RPM RH5/64 OK?    SKIPPED Not Red Hat Enterprise Linux 5 64-bit
A04312 RPM RH6/64 OK?    SKIPPED Not Red Hat Enterprise Linux 6 64-bit
A04313 RPM RH7/64 OK?    FAILED [compat-libstdc++-296(i686)] not inst...
A04321 RPM SLES10/64 OK? SKIPPED Not SuSE Linux Enterprise 10 64-bit
A04322 RPM SLES11/64 OK? SKIPPED Not SuSE Linux Enterprise 11 64-bit
Result file: output/collect/APPS_HCVE_A_EBS122_lin_res.htm

In one instance, after package redhat-lsb was installed, rapidwiz passed verification check from previous failure:

$ rpm -qa redhat-lsb

redhat-lsb-4.1-27.el7.x86_64

redhat-lsb-4.1-27.el7.i686

With below packages are installed, seems compat-libstdc++-296(i686) is not necessary.

$ rpm -qa | grep libstdc
libstdc++-devel-4.8.5-39.el7.x86_64
libstdc++-devel-4.8.5-39.el7.i686
libstdc++-4.8.5-39.el7.x86_64
libstdc++-4.8.5-39.el7.i686
compat-libstdc++-33-3.2.3-72.el7.x86_64
compat-libstdc++-33-3.2.3-72.el7.i686

$ rpm -qa | grep libaio
libaio-0.3.109-13.el7.x86_64

libaio-0.3.109-13.el7.i686

$ rpm -qa | grep libaio-dev
libaio-devel-0.3.109-13.el7.x86_64 <== it is REQUIRED on my Linux 64 bits

Most Kernel parameters are in a file under folder /etc/sysctl.d. Here is a custom file:

$ more /etc/sysctl.d/90-oracle.conf
fs.aio-max-nr = 1048576
fs.file-max = 6815744
kernel.shmall = 2097152
kernel.shmmax = 4294967295
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576

$ more /proc/sys/fs/file-max
6815744

Explanation on port 6000 conflict (See Doc ID 761566.1):

rapidwiz uses port 6000, because the OPMN service of the Application Server technology stack listens on port 6000 by default when started up during Rapid Install. This could conflict with the X11 port used for the graphics console on Linux servers and prevent Rapid Install from completing.

To address this issue, perform the following steps before installing or upgrading:
. Check if the port 6000 is being used "netstat -a | grep 6000" or "netstat -tunap | grep -i 6000"
. If so, you should disable the graphics console login process on the Linux server by logging in as root and issuing the following commands:
- Find the process that uses port 6000 by the command "lsof -i TCP:6000".
- Use "kill -9" to abort the process found in the previous step
- Confirm that nothing is listening at port 6000 ("netstat -a | grep 6000")
- Modify /etc/inittab by changing "id:5:initdefault:" to "id:3:initdefault:"
- Run "/sbin/init 3" to restart dtlogin

Wednesday, September 23, 2020

R12.2 software and StartCD 51 for repaid installation

The first task in upgrading EBS to R12.2 is to get the software ready. But it is not straight and easy. Here are steps to get started.

1. Create folder /path/to/R122, and add at lease 100GB disk space to /path/to partition.

2. Download ZIP files to /path/to/R122
See 12.2 E-Business Suite Applications Installation Rapid Install Steps To Create the Stage Directories In Preparation For A 12.2 Installation (Doc ID 1596433.1)

https://edelivery.oracle.com/osdc/faces/SoftwareDelivery
Click Popular Downloads => REL: Oracle E-Business Suite 12.2.0 Current => 23 ZIP files
Click on Download to get ORACLE DOWNLOAD MANAGER (then, enter your login info).

Oracle E-Business Suite 12.2.0 Current for Linux x86-64:
V35803-01_1of3.zip EBS R 12.2.0 for Linux x86-64 Rapid Install APPL_TOP - Disk 1, 1.2 GB
V35803-01_2of3.zip EBS R 12.2.0 for Linux x86-64 Rapid Install APPL_TOP - Disk 1, 1022.8 MB
V35803-01_3of3.zip EBS R 12.2.0 for Linux x86-64 Rapid Install APPL_TOP - Disk 1, 1.3 GB
V35804-01_1of2.zip EBS R 12.2.0 for Linux x86-64 Rapid Install APPL_TOP - Disk 2, 1.2 GB
V35804-01_2of2.zip EBS R 12.2.0 for Linux x86-64 Rapid Install APPL_TOP - Disk 2, 1.6 GB
V35802-01.zip  EBS R 12.2.0 for Linux x86-64 Rapid Install AS10.1.2, 711.6 MB
V100060-01_1of3.zip EBS R 12.2.0 for Linux x86-64 Rapid Install DBs PROD - Disk 2, 1.6 GB
V100060-01_2of3.zip EBS R 12.2.0 for Linux x86-64 Rapid Install DBs PROD - Disk 2, 1.1 GB
V100060-01_3of3.zip EBS R 12.2.0 for Linux x86-64 Rapid Install DBs PROD - Disk 2, 1.0 GB
V100061-01.zip  EBS R 12.2.0 for Linux x86-64 Rapid Install Technology one-off Patches, 839.9 MB
V100054-01.zip  EBS R 12.2.0 for Linux x86-64 Rapid Install Databases VISION - Disk 2, 3.0 GB
V100055-01.zip  EBS R 12.2.0 for Linux x86-64 Rapid Install Databases VISION - Disk 3, 2.8 GB
V100053-01_1of2.zip EBS R 12.2.0 for Linux x86-64 Rapid Install DBs VISION - Disk 1, 52.0 MB
V100053-01_2of2.zip EBS R 12.2.0 for Linux x86-64 Rapid Install DBs VISION - Disk 1, 3.4 GB
V100058-01.zip  EBS R 12.2.0 for Linux x86-64 Rapid Install DBs VISION - Disk 6, 2.0 GB
V100059-01_1of2.zip EBS R 12.2.0 for Linux x86-64 Rapid Install DBs PROD - Disk 1, 1007.0 MB
V100059-01_2of2.zip EBS R 12.2.0 for Linux x86-64 Rapid Install DBs PROD - Disk 1, 1.6 GB
V100056-01.zip  EBS R 12.2.0 for Linux x86-64 Rapid Install Databases VISION - Disk 4, 2.6 GB
V100057-01.zip  EBS R 12.2.0 for Linux x86-64 Rapid Install Databases VISION - Disk 5, 2.4 GB
V100102-01.zip  Oracle DB 12c Release 1 (12.1.0.2.0) EXAMPLES for Linux x86-64, 476.1 MB

If patch file p22066363_R12_GENERIC.zip is downloaded, below three files are not needed:

V100052-01_1of3.zip EBS R 12.2.0 Rapid Install Start Here, 725.3 MB
V100052-01_2of3.zip EBS R 12.2.0 Rapid Install Start Here, 81.6 MB
V100052-01_3of3.zip EBS R 12.2.0 Rapid Install Start Here, 97.3 MB

Go to https://www.oracle.com/database/technologies/database12c-linux-downloads.html
to download Oracle Database 12c Release 1 (12.1.0.2.0) for Linux x86-64 (and then rename them to V46095-01_xxxx.zip)
V46095-01_1of2.zip (linuxamd64_12102_database_1of2.zip)
V46095-01_2of2.zip (linuxamd64_12102_database_2of2.zip)

Then, follow instruction to download four ZIP files for Oracle Coherence 3.7.1.0.0 and one ZIP file for Oracle Web Tier 11.1.1.9.0
V29856-01.zip Oracle WebLogic Server 10.3.6 (11gR1) Generic with Coherence 3.7.1, 1018.5 MB
V27954-01.zip Oracle Coherence 3.7.1.0.0 for C++ for Linux x86-64, 21.1 MB
V27953-01.zip Oracle Coherence 3.7.1.0.0 for the Microsoft .NET Framework, 17.0 MB
V28274-01.zip Oracle Coherence 3.7.1.0.0, 15.7 MB
V75792-01.zip Oracle Web Tier 11.1.1.9.0 Utilities (11g Patch Set 7) - Oracle Fusion Middleware for Linux x86-64, 2.0 GB

Totally, 30 ZIP files.

3. Create folder /path/to/R122/rapidSTARTCD, and download p22066363_R12_GENERIC.zip (Patch 22066363: RAPID INSTALL STARTCD 12.2.0.51, StartCD 51) to this new folder.

4. Unzip p22066363_R12_GENERIC.zip creates folder startCD.
NOTE: Do not manually unzip the other stage files. After unzipping the startCD, buildStage.sh script should be run to unzip the rest of the installation software.

5. Build stage area -- it needs about 45GB to unzip files
$ cd startCD/Disk1/rapidwiz/bin
(or, /path/to/stageR122/rapidSTARTCD/startCD/Disk1/rapidwiz/bin)
$ ./buildStage.sh

Two actions are necessary.

                      Build Stage Menu
   ------------------------------------------------------
   1.     Create new stage area
   2.     Copy patches to existing stage area

   Enter your choice [4]: 1
… …
Specify the directory containing the zipped installation media: /path/to/stageR122

Verifying stage area...
Directory /path/to/stageR122/rapidSTARTCD/TechInstallMedia is valid.
Directory /path/to/stageR122/rapidSTARTCD/TechPatches/DB is valid.
Directory /path/to/stageR122/rapidSTARTCD/TechPatches/MiddleTier is valid.
Directory /path/to/stageR122/rapidSTARTCD/EBSInstallMedia/AppDB is valid.
Directory /path/to/stageR122/rapidSTARTCD/EBSInstallMedia/Apps is valid.
Directory /path/to/stageR122/rapidSTARTCD/EBSInstallMedia/AS10.1.2 is valid.
Directory /path/to/stageR122/rapidSTARTCD/TechInstallMedia/database is valid.
Directory /path/to/stageR122/rapidSTARTCD/TechInstallMedia/ohs11119 is valid.
Directory /path/to/stageR122/rapidSTARTCD/TechInstallMedia/wls1036_generic is valid.

If all worked, you shall see "Stage area verified." by the end.

It unzip files to /path/to/stageR122/rapidSTARTCD and creates 3 sub-folders EBSInstallMedia, TechInstallMedia, TechPatches

6. Download patch 25525148 p25525148_R12_GENERIC.zip, the consolidated fixes on top of Patch 22066363, to /path/to/stageR122/rapidSTARTCD
(If database 19c is used, it needs another patch 29963728. See Oracle E-Business Suite Release Notes, Release 12.2 (Doc ID 1320300.1) ).

$ unzip p25525148_R12_GENERIC.zip
$ cd 25525148
$ ./patchRIStage.sh
Enter the location of the Rapid Install Stage area: /path/to/stageR122/rapidSTARTCD

$ cd ../
$ cd startCD/Disk1/rapidwiz/bin
$ ./buildStage.sh
                    Build Stage Menu
   ------------------------------------------------------
   1.     Create new stage area
   2.     Copy patches to existing stage area
   3.     List files in TechPatches directory
   4.     Exit menu
   Enter your choice [4]: 2

… …

Copying latest one-off patches to stage area...

Finished copying additional patches.

7. Now, Rapid Install CDs are ready, you can go to folder /path/to/stageR122 to tar folder rapidSTARTCD up to a file and copy it to other servers for installation if needed.

8. Test rapidwiz

Stay at /path/to/stageR122/rapidSTARTCD/startCD/Disk1/rapidwiz

$ vi /etc/oraInst.loc

$ export DISPLAY=xx.xx.xxx.xxx:0.0
$ ./rapidwiz -version

Rapid Install Wizard is validating your file system......
Rapid Install Wizard will now launch the Java Interface.....

Now, R12.2 software is ready for Rapidwiz. Rapid Install will validate the system configuration and tell what pre-steps on OS and database are required for the installation.

$ ./rapidwiz -help
Rapid Install Wizard is validating your file system......
CMDDIR=/path/to/stageR122/rapidSTARTCD/startCD/Disk1/rapidwiz
Rapid Install Wizard will now launch the Java Interface.....