Wednesday, November 22, 2023

How to run AutoConfig on PATCH file system in R12.2

 Steps for running AutoConfig on PATCH file system:

1) Disable a trigger
SQL> conn system/systemPWD
Connected.
SQL> alter trigger ebs_logon disable;
Trigger altered.

2) Set PATCH env and connect to database
$ . /<EBS_HOME_BASE>/EBSapps.env patch

$ echo $TWO_TASK
EBSDEV_patch
$ sqlplus apps/appsPWD  
    -- If it got error, compare tnsnames.ora in both RUN and PATCH file systems &
    -- modifying tnsnames.ora in PATCH file system may be needed in Oracle 19c db. 
SQL> show user
USER is "APPS"
SQL> exit

3) Run AutoConfig in PATCH file system
$ cd $ADMIN_SCRIPTS_HOME
$ echo $FILE_EDITION
patch
$ ./adautocfg.sh
Enter the APPS user password:

The log file for this session is located at: $INST_TOP/admin/log/MMDDHHMI/adconfig.log
... ...
AutoConfig completed successfully.

4). Enable the trigger
SQL> conn system/systemPWD
Connected.
SQL> alter trigger ebs_logon enable;
Trigger altered.

Friday, November 10, 2023

Port conflict during R12.2 apps clone

If Target mid-tier and Source mid-tier are on the same server, EBS clone script adcfgclone.pl may fail with port conflict error. 

Prots used for an EBS instance is listed in file $INST_TOP/admin/out/portpool.lst. Depending on which port has conflict on the server, the cloning error message will be different.  

- If the port for httpd is busy and used by another environment on the server, the cloning script will stop on Target instance with errors:

ERROR: Failed to configure the target system,
please check the logfile in : $RUN_BASE/inst/apps/$CONTEXT_NAME/admin/log/clone

CLONE-26003   Error in validating listen host and port. 
CLONE-26176  In config group httpd.conf , the value of "Listen" config property was xxxx. xxxx was not free

Log message shows the problem is from file httpd.conf. When I checked the port number and compare it in file $CONTEXT_FILE and in file $FMW_HOME/webtier/instances/EBS_web_<SID>_OHS1/config/OHS/EBS_web_<SID>/httpd.conf. 
$ grep Listen httpd.conf
$ grep s_http_listen_parameter $CONTEXT_FILE

I saw they are different in Source instance. Even they are not the same, Apache still works fine on Source instance and httpd.worker processes occupy the port number in httpd.conf. After the clone script copied file httpd.conf to Target instance, Apache failed to start because OHS was running on the same port as the other instance

Apparently httpd.conf of Source instance was edited by picking up randomly a port number. httpd.conf can be edited and updated manually as AutoConfig in R12.2 does not update it. For more details on modifying port values for OHS, see Doc ID 1905593.1 (Managing Configuration of Oracle HTTP Server and Web Application Services in Oracle E-Business Suite Release 12.2).

- While cloning script uses a temporary port but it was not available, the error could be 
CLONE-20372 Server port validation failed.
The fix could be just shutdown apps services of Source instance to let the cloning complete. See Doc ID 2002613.1

- Doc ID 2437111.1 gives a fix on port for s_ohs_adminport. 

On Linux server, use command to find if a port is used or not:

$ netstat -tuanp | grep 6230
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp    0 0 167.69.109.82:6230 0.0.0.0:*   LISTEN 

Then the process owner (or root) can use below line to find what process is using the port: 

$ lsof -i :6230 
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
opmn 31876 ebsdev 9u IPv4 42886593 0t0 TCP server3d.domain.com:6230 (LISTEN)

After adcfgclone.pl failed, you have to remove/rename folders BEFORE re-run clone script adcfgclone.pl 
- Remove two new folders under $RUN_BASE (or only folder $RUN_BASE/FMW_Home, depending on the failure stage) to avoid error "Exiting cloning as FMW Home already exists".
- Also remove the folder defined by inventory_loc in file /etc/oraInst.loc to avoid error "Oracle Homes are already registered in the inventory". If the inventory is just for one environment, do not try to detach a home because it may give error:
$ ./runInstaller -detachhome $FMW_HOMEe/oracle_common
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB.   Actual 13519 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at ... ...
The operation failed as it was called without name of the Oracle Home being attached.
- Since the script failed before the completion, there is not need to remove info in the database.

Friday, August 11, 2023

Apply R12.2 July 2023 CPU patches

Oracle E-Business Suite Release 12.2 Critical Patch Update Availability Document (July 2023) (Doc ID 2953580.1) is the document for July 2023 CPU patches.

Check to find ATG_PF.C.delta.9 is the latest ATG (Application Technology Family) patch in my instances.

SQL> select * from ad_bugs where bug_number in (31856779, 30399994, 33527666);

30399994 Patch R12.ATG_PF.C.delta.9 (Application Technology Family)
31856779 Patch R12.ATG_PF.C.delta.10 
33527666 Patch R12.ATG_PF.C.Delta.11 
                 (Released on Jan 9 2023. See Doc ID 2876714.1 on how to apply it)

1. Pre-requisites: R12.AD.C.Delta.14 and R12.TXK.C.Delta.14 patches

Follow steps in Apply AD-TXK Delta 14 RUPs to apply them. After all are done, 8 patches were applied:

SQL> SELECT adb.bug_number, aas.name appl_top_name, adb.creation_date, adb.language,
decode(ad_patch.is_patch_applied('R12',aas.appl_top_id,adb.bug_number,adb.language),'EXPLICIT','APPLIED','NOT_APPLIED','NOT APPLIED') status
FROM ad_bugs adb,
(select distinct appltop_id appl_top_id, node_name name from ad_adop_sessions 
           where node_name in (select node_name from ADOP_VALID_NODES) ) aas
where adb.bug_number in (
'33600809',  -- R12.AD.delta.C.14
'34668508',
'34669333',
'34681299',
'35280947',
'33602997',  -- R12.TXK.delta.C.14
'34708635',
'34654260'
) order by creation_date desc, adb.bug_number,aas.name,adb.language;

2. Apply July 2023 CPU patch 35385938. It takes about 30 minutes in the multi-node.

First of all, download all EBS patches to a shared location /a46/CPUs/CPU_July2023

$ echo $FILE_EDITION
run

$ adop phase=apply apply_mode=downtime patches=35385938 patchtop=/a46/CPUs/CPU_July2023/CPU

3. Run autoconfig
$ perl $AD_TOP/bin/admkappsutil.pl
$ cp -p $INST_TOP/admin/out/appsutil.zip $APPLPTMP

Autoconfig on db node first
Autoconfig on all apps nodes

NOTES: After CPU patching, custom .tmp file in $FND_TOP/admin/template/custom may be out-of-date and needs an update if existing to avoid autoconfig fails on apps nodes.

4. Post patches:  It turns out that it needs to apply only 5 post patches in my instances.

$ adop phase=apply apply_mode=downtime patches=34212478,34556525,34291981,35181823,35355008 patchtop=/a46/CPUs/CPU_July2023/posts

After that, verify all 11 patches were applied:

SQL> SELECT adb.bug_number, aas.name appl_top_name, adb.creation_date, adb.language,
decode(ad_patch.is_patch_applied('R12',aas.appl_top_id,adb.bug_number,adb.language),'EXPLICIT','APPLIED','NOT_APPLIED','NOT APPLIED') status
FROM ad_bugs adb,
(select distinct appltop_id appl_top_id, node_name name from ad_adop_sessions 
           where node_name in (select node_name from ADOP_VALID_NODES) ) aas
where adb.bug_number in (
'35181823',    -- post patch   (Table 1)
'35355008',    -- post patch   (Table 1)
-- '33207251',  -- Post-CPU EBS Security Patches  (Table 2)
-- '32980025',  -- Post-CPU EBS Security Patches
-- '33286000',  -- Post-CPU EBS Security Patches
'34212478',     -- Post-CPU EBS Security Patches
'34556525',     -- Post-CPU EBS Security Patches
'34291981'      -- Post-CPU EBS Security Patches
-- '33625264',  -- Post-CPU EBS Security Patches
-- '33862025',  -- needed for AD.C.Delta.13 and TXK.Delta.13. Comes after CPU 35385938
-- '33568131'   -- needed for AD.C.Delta.13 and TXK.Delta.13. Comes after CPU 35385938
) order by creation_date desc, adb.bug_number,aas.name,adb.language;

Optional: start apps services to verify all work.

6. Run ETCC script checkMTpatch.sh to get the list of patches asked by ETCC (patch 17537119). But, for WebLogic, Table 4 of Doc ID 2953580.1 asks to apply Oracle WebLogic Server 10.3.6.0 patch 34900723 (WLS PSU 10.3.6.0.230418). 

Download directly patch file p34900723_1036_Generic.zip and copy it to $FMW_HOME/utils/bsu/cache_dir.

$ cd $FMW_HOME/utils/bsu/cache_dir

$ unzip -o p34900723_1036_Generic.zip
Archive:  p34900723_1036_Generic.zip
 extracting: P8S7.jar
  inflating: patch-catalog_27964.xml
  inflating: README.html
  inflating: README.txt

$ cd $FMW_HOME/utils/bsu
$ ./bsu.sh -install -patch_download_dir=$FMW_HOME/utils/bsu/cache_dir -patchlist=P8S7 -prod_dir=$FMW_HOME/wlserver_10.3
Checking for conflicts..
Conflict(s) detected - resolve conflict condition and execute patch installation again
Conflict condition details follow:
Patch P8S7 is mutually exclusive and cannot coexist with patch(es): HYG5,TNS1,IJC1,7GCA

$ ./bsu.sh -remove -patchlist=IJC1,7GCA,TNS1 -prod_dir=$FMW_HOME/wlserver_10.3
Checking for conflicts..
No conflict(s) detected

Removing Patch ID: IJC1.
Result: Success

Removing Patch ID: 7GCA..
Result: Success

Removing Patch ID: TNS1..
Result: Success

$ ./bsu.sh -remove -patchlist=HYG5 -prod_dir=$FMW_HOME/wlserver_10.3
Checking for conflicts..
No conflict(s) detected

Removing Patch ID: HYG5..
Result: Success

$ ./bsu.sh -install -patch_download_dir=$FMW_HOME/utils/bsu/cache_dir -patchlist=P8S7 -prod_dir=$FMW_HOME/wlserver_10.3
Checking for conflicts..
No conflict(s) detected

Installing Patch ID: P8S7..
Result: Success

-- verify it
$ ./bsu.sh -prod_dir=$FMW_HOME/wlserver_10.3 -status=applied -verbose -view | grep P8S7
Patch ID:          P8S7
PatchContainer:    P8S7.jar

6. Run ETCC script checkMTpatch.sh to identify missing technology patches. 

Starting Application Tier Technology Codelevel Checker
Version: 120.0.12020000.65

Bugfix XML file version: 120.0.12020000.61
This file will be used for identifying missing bugfixes.

Mapping XML file version: 120.0.12020000.49
This file will be used for mapping bugfixes to patches.
... ...
The default patch recommendations to install these missing bugfixes are:
-------------------------------------------------------------------------------
Oracle Fusion Middleware (FMW) - Web Tier 11.1.1.9.0
-------------------------------------------------------------------------------
  Patch 35540062
    - Filename: p35540062_111190_Linux-x86-64.zip
  Patch 34067016
    - Filename: p34067016_11119210420OSS_Linux-x86-64.zip

-------------------------------------------------------------------------------
Oracle Fusion Middleware (FMW) - oracle_common 11.1.1.9.0
-------------------------------------------------------------------------------
  Patch 34330735
    - Filename: p34330735_111190_Generic.zip

-------------------------------------------------------------------------------
Oracle WebLogic Server (WLS) 10.3.6.0.230418
-------------------------------------------------------------------------------
  Patch 13845626 [SU Patch [AMGE]]
    - Filename: p13845626_10360230418_Generic.zip
  Patch 34856303 [SU Patch [ENUU]]
    - Filename: p34856303_1036_Generic.zip
  Patch 31042881 [SU Patch [7BIA]]
    - Filename: p31042881_1036_Generic.zip

7. Continue to apply WLS patches (after 3 patches files were downloaded and copied to $FMW_HOME/utils/bsu/cache_dir)

$ ./bsu.sh -install -patch_download_dir=$FMW_HOME/utils/bsu/cache_dir -patchlist=AMGE -prod_dir=$FMW_HOME/wlserver_10.3
Checking for conflicts..
No conflict(s) detected

Installing Patch ID: AMGE.
Result: Success

$ ./bsu.sh -install -patch_download_dir=$FMW_HOME/utils/bsu/cache_dir -patchlist=ENUU -prod_dir=$FMW_HOME/wlserver_10.3
Checking for conflicts..
Conflict(s) detected - resolve conflict condition and execute patch installation again
Conflict condition details follow:
Patch ENUU is mutually exclusive and cannot coexist with patch(es): DN1F

$ ./bsu.sh -remove -patchlist=DN1F -prod_dir=$FMW_HOME/wlserver_10.3
Checking for conflicts..
No conflict(s) detected

Removing Patch ID: DN1F.
Result: Success

$ ./bsu.sh -install -patch_download_dir=$FMW_HOME/utils/bsu/cache_dir -patchlist=ENUU -prod_dir=$FMW_HOME/wlserver_10.3
Checking for conflicts..
No conflict(s) detected

Installing Patch ID: ENUU.
Result: Success

$ ./bsu.sh -install -patch_download_dir=$FMW_HOME/utils/bsu/cache_dir -patchlist=7BIA -prod_dir=$FMW_HOME/wlserver_10.3
Checking for conflicts..
No conflict(s) detected

Installing Patch ID: 7BIA.
Result: Success

8. Apply patches from ETCC list to FMW homes.
Apply 35540062 and 34067016 to Oracle Fusion Middleware (FMW) - Web Tier
Note: 35540062 (OHS Security Patch Update JulCPU2023 Patch) will roll back 34867865 (JanCPU2023 Patch) if it was applied before.

$ echo $IAS_ORACLE_HOME
$ export ORACLE_HOME=$IAS_ORACLE_HOME
$ echo $ORACLE_HOME          <== $FMW_Home/webtier
$ export PATH=$IAS_ORACLE_HOME/OPatch:$PATH
$ which opatch

$ echo $FILE_EDITION
run

$ opatch lsinventory | grep 31304503

$ opatch lsinventory
Interim patches (17)     <== depending on patching history

$ cd /a46/CPUs/CPU_July2023/tech

$ unzip p34067016_11119210420OSS_Linux-x86-64.zip
$ cd 34067016
$ opatch apply
... ...
Patching component oracle.ldap.rsf, 11.1.1.9.0...
Patching component oracle.rdbms.rsf, 11.1.0.7.0...

Verifying the update...
Patch 34067016 successfully applied

$ cd ..
$ rm -rf 34067016

$ cd 35540062
$ opatch apply

Patching component oracle.ohs2, 11.1.1.9.0...
RollbackSession removing interim patch '33311587' from inventory
OPatch back to application of the patch '35540062' after auto-rollback.
Patching component oracle.ohs2, 11.1.1.9.0...
Verifying the update...
Patch 35540062 successfully applied

$ cd ..
$ rm -rf 35540062

$ opatch lsinventory
Interim patches (18) 
.. ...

9. Apply patch 34330735 to Oracle Fusion Middleware (FMW) - Common

$ export ORACLE_HOME=$FMW_HOME/oracle_common
$ export PATH=$ORACLE_HOME/OPatch:$PATH

$ echo $ORACLE_HOME
.../FMW_Home/oracle_common
$ which opatch
.../FMW_Home/oracle_common/OPatch/opatch

$ cd /a46/CPUs/CPU_July2023/tech
$ opatch lsinventory
Interim patches (14)

$ unzip p34330735_111190_Generic.zip
$ cd 34330735/oui
$ ls
$ opatch apply
... ...
RollbackSession removing interim patch '31985571' from inventory
OPatch back to application of the patch '34330735' after auto-rollback.
Patching component oracle.jrf.adfrt, 11.1.1.9.0...

Verifying the update...
Patch 34330735 successfully applied

cd ..
rm -rp 34330735

$ opatch lsinventory
Interim patches (14)

By now ETCC script shows all patches were applied.

10. Upgrade JDK to 1.7.0_391

See Upgrade JDK 7 in EBS R12.2 for detail on upgrading JDK. Table 3 in CPU release document 2953580.1 shows JDK 1.7.0_391 is the latest JDK for EBS. Use the link in Doc ID 2957260.1 (Oracle Critical Patch Update (CPU) Jul 2023 for Oracle Java SE) to download patch 35374159 files  p35374159_170391_LINUX.zip & p35374159_170391_Linux-x86-64.zip and save them to /a46/CPUs/CPU_July2023/jdk. 

$ unzip p35374159_170391_Linux-x86-64.zip
$ tar vzxfp jdk-7u391-linux-x64.tar.gz
$ mv jdk1.7.0_391 jdk_391_64

$ unzip p35374159_170391_LINUX.zip
$ tar vzxfp jdk-7u391-linux-i586.tar.gz
$ mv jdk1.7.0_391 jdk_391_32

$ echo $FILE_EDITION
run

JDKfolder=/a46/CPUs/CPU_July2023/jdk

cd $COMMON_TOP/util
tar -cvzf jdk64_BK.tar.gz jdk64
tar -cvzf jdk32_BK.tar.gz jdk32
rm -fr jdk64
rm -fr jdk32
cp -rp $JDKfolder/jdk_391_64 jdk64
cp -rp $JDKfolder/jdk_391_32 jdk32

cd $FMW_HOME/webtier
tar -cvzf jdk64_BK.tar.gz jdk
rm -fr jdk
cp -rp $JDKfolder/jdk_391_64 jdk

cd $ORACLE_HOME
ls
tar -cvzf jdk32_BK.tar.gz jdk
ls -al
rm -fr jdk
cp -rp $JDKfolder/jdk_391_32 jdk

$ADJVAPRG -version
$AFJVAPRG -version

cd $ORACLE_HOME/forms/lib
make -f ins_forms.mk sharedlib install
cd $ORACLE_HOME/reports/lib
make -f ins_reports.mk install

11. Upgrade JRE (Java Run-time Environment)

See 4) in Enable JWS with EBS R12.2 for JRE upgrade. Doc ID 2957260.1 also gives a link to download JRE 8 Update 381 (patch 35374151). I used file p35374151_180381_WINNT.zip (for Windows 32-bit).

$ cd /a46/CPUs/CPU_July2023/jre
$ unzip p35374151_180381_WINNT.zip
Archive:  p35374151_180381_WINNT.zip
  inflating: jre-8u381-windows-i586.exe
  inflating: jre-8u381-windows-i586.tar.gz
  inflating: readme.txt

$ cp jre-8u381-windows-i586.exe $COMMON_TOP/webapps/oacore/util/javaplugin/j2se18381.exe
$ cd $COMMON_TOP/webapps/oacore/util/javaplugin
$ ls -al
$ echo $FILE_EDITION
run

$ $FND_TOP/bin/txkSetPlugin.sh 18381
 
$ grep sun $CONTEXT_FILE
         <sun_plugin_ver oa_var="s_sun_plugin_ver">1.8.0_381</sun_plugin_ver>
         <sun_plugin_type oa_var="s_sun_plugin_type">jdk</sun_plugin_type>
         <sun_clsid oa_var="s_sun_clsid">CAFEEFAC-0018-0000-0381-ABCDEFFEDCBA</sun_clsid>

12. Optional: Re-generate and sign JAR files.
$ adadmin   (1 => 4 => Yes)
13. Start Apps services for test
$ adstrtal.sh apps/xxxx
14. adop phase=fs_clone  (sync RUN file system to PATCH file system)

Saturday, August 5, 2023

vulnerabilities by Qualys report

Qualys Scan reports huge number of vulnerabilities in our EBS R12.2.10 servers where JDK was upgraded to 1.7.0_331. Vulnerability issue became a priority task for us due to corporate security requirement.

$ $ADJVAPRG -version
java version "1.7.0_331"
Java(TM) SE Runtime Environment (build 1.7.0_331-b06)
Java HotSpot(TM) Server VM (build 24.331-b06, mixed mode)

$ $AFJVAPRG -version
java version "1.7.0_331"
Java(TM) SE Runtime Environment (build 1.7.0_331-b06)
Java HotSpot(TM) 64-Bit Server VM (build 24.331-b06, mixed mode)

Some vulnerabilities can be addressed by applying latest CPU patches, upgrading JDK and JRE (see details). 

Vulnerabilities also reported:

$COMMON_TOP/clone/FMW/t2pjdk/bin/java 1.7.0_85-b15 Enhanced
$COMMON_TOP/clone/FMW/t2pjdk/jre/bin/java 1.7.0_85-b15 Enhanced
(It is safe to delete above folders because they will be recreated by an adpreclone.pl run)
$COMMON_TOP/clone/jre/bin/java 1.7.0_331-b06 Enhanced  
(It is not in JDK folders and so a JDK upgrade will not fix it. But it may be recreated by an adpreclone.pl run)
$COMMON_TOP/adopclone_nodeName/FMW/t2pjdk/jre/bin/java 1.7.0_85-b15 Enhanced
$COMMON_TOP/adopclone_nodeName/FMW/t2pjdk/bin/java 1.7.0_85-b15 Enhanced
$COMMON_TOP/adopclone_nodeName/jre/bin/java 1.7.0_85-b15 Enhanced
($COMMON_TOP/adopclone_nodeName may get created/updated from ADOP run, such as fs_clone. nodeName is the EBS host, where ADOP (or adpreclone.pl?) ran. )
$APPL_TOP/msc/12.0.0/bin/SNO/scp/12.2/sno/installer_jre/bin/java 1.6.0_12-b04 Enhanced
$APPL_TOP/msc/12.0.0/bin/PS/scp/12.2/ps/jre/bin/java 1.8.0_92-b14 Enhanced

Oracle support confirmed that they can be deleted and some of them will be recreated next time adpreclone.pl is run. I wrote a script to clean them on RUN file system (and on PATCH file system or run FS_CLONE). 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# delete Java files for vulnerabilities by Qualys report

curr=`pwd`
echo "current path: $curr"
cd $COMMON_TOP/clone/FMW/t2pjdk/bin
pwd
ls -al java
rm -f java

cd $COMMON_TOP/clone/FMW/t2pjdk/jre/bin
pwd
ls -al java
rm -f java

cd $COMMON_TOP/clone/jre/bin
pwd
ls -al java
rm -f java

cd $APPL_TOP/msc/12.0.0/bin/SNO/scp/12.2/sno/installer_jre/bin
pwd
ls -al java
rm -f java

cd $APPL_TOP/msc/12.0.0/bin/PS/scp/12.2/ps/jre/bin
pwd
ls -al java
rm -f java

# loop folder(s) $COMMON_TOP/adopclone_*
for x in `ls -ald $COMMON_TOP/adopclone_* |egrep "^drw" |awk '{print $9}'`
do
echo Under path: $x
cd $x/jre/bin
pwd
ls -al java
rm -f java
cd $x/FMW/t2pjdk/jre/bin
pwd
ls -al java
rm -f java
cd $x/FMW/t2pjdk/bin
pwd
ls -al java
rm -f java
done

cd $curr
exit

Saturday, July 8, 2023

SuperPuTTY and WinSCP

Software tools on Windows PC for routine work.

SuperPuTTY

Steps to install and set up SuperPutty

1. Install Putty. Version 0.76 is installed in a folder under C:\Program Files\PuTTY (vs. just one file putty.exe).
PuTTY stores setup data (saved sessions, SSH host keys) in the Registry. The location is
HKEY_CURRENT_USER\Software\SimonTatham\PuTTY

To configure PuTTY, such as log file location and font size (under Appearance), select and save them to "Default Settings".
 
2. Download SuperPuTTY from https://github.com/jimradford/superputty/releases (Release 1.5.0.0 is available in 2023).
3. Unzip the .zip file to a folder (e.g. c:\Putty\SuperPuTTY-1.5.0.0)
4. Create a shortcut on desktop pointing to file SuperPutty.exe

c:\Putty\SuperPuTTY-1.5.0.0>dir
 ... ...
01/29/2023  08:32 AM           44,464 License.rtf
01/29/2023  08:31 AM             1,129 License.txt
07/24/2022  11:40 PM          270,336 log4net.dll
01/29/2023  08:30 AM           20,623 ReleaseNotes.txt
05/29/2022  04:51 AM         804,352 Renci.SshNet.dll
01/29/2023  08:36 AM      1,090,560 SuperPutty.exe
01/29/2023  06:57 AM           10,378 SuperPutty.exe.config
01/29/2023  08:36 AM         794,112 SuperPutty.pdb
08/24/2023  06:54 PM    <DIR>        themes
08/27/2021  04:24 PM          244,736 WeifenLuo.WinFormsUI.Docking.dll
08/27/2021  04:24 PM            90,112 WeifenLuo.WinFormsUI.Docking.ThemeVS2005.dll

Initial setup info of SuperPutty is stored in file C:\Users\User_ID\SuperPuTTY.settings:
Tells where Putty is installed:
      <PuttyExe>C:\Program Files\Putty\putty.exe</PuttyExe>
Specifies a folder to hold server listing and session info:
      <SettingsFolder>c:\Putty\SuperPuTTY</SettingsFolder>
  
If needed, setup info can by changed by Tool -> Options.  File SuperPutty.exe.config holds configuration info.
 
To use the passwords on the command line or in a session, you must check the box 
"Allow plain text passwords on putty command line" in Tools -> Options -> Gui under security

You can disable the update check (Tools -> Options -> Advanced -> Uncheck Automatically check for updates) to avoid message in version 1.5.0.0:

Error during update check
There was an error while checking for updates. Please try again later.

Need to backup folder C:\Putty\SuperPuTTY from time to time (specially when a new server session is added to the list!). When a SuperPutty is installed on a new PC, you can restore the backup folder to the new PC. Then, point SuperPutty to this folder to transfer all connection info to the new PC. 

WinSCP 

On its login box, it has a Tools dropdown for Import/Restore Configuration and Export/Backup Configuration. 

I have WinSCP version 5.19.5 installed in C:\Program Files (x86)\WinSCP. Its configuration is stored in Windows registry HKEY_CURRENT_USER\SOFTWARE\Martin Prikryl\WinSCP 2

On my old laptop, I ran potable version 5.5.6 in a folder by unzipping file winscp556.zip. File WinSCP.ini in the folder has all configuration info. When a newer WinSCP version was installed on a new laptop, I copied the .ini file to the new computer and use Import to transfer the info to the newer version.

Exceed

Exceed is a tool for bringing X Window (e.g. GUI forms) from Linux/Unix session to your PC. It is owned by Open Text and is installed in folder C:\Program Files\Hummingbird\Connectivity. OpenText website says "In October 2006, OpenText™ acquired Hummingbird. Hummingbird’s enterprise software solutions, including Exceed, enabled the management of the life cycle of enterprise content."

Monday, July 3, 2023

Oracle SQL Developer and JDK

After Oracle SQL Developer was installed (without including JDK in the pack) on my new laptop, its first launch creates file C:\Users\user_ID\AppData\Roaming\sqldeveloper\product.conf, which specifies and saves the path of Java. In the first run, it brings up a popup box says "Please specify the path to the Java JDK home (e.g. ... ...)". The correct path has to be entered here for SQL Developer to fire up. On my laptop, after I click on OK to accept default path C:\Program Files (x86)\Java\jdk18, the popup box goes away. Then nothing shows up and it seems at a dead end.

During troubleshooting, I learned my laptop has two Java installation locations, which leads some confusion. The one in C:\Program Files\Java\jdk18 is a 64-bit that was requested and installed for being used by SQL Developer. SQL Developer installed is a 64-bit application and needs 64-bit Java. After I entered C:\Program Files\Java\jdk18 in the popup box, SQL Developer 21.4.3 worked and the Properties info can be viewed by Help -> About.

It the first launch, it asks if you want to import preference from previous version. I copied over folder C:\Users\user_ID\AppData\Roaming\SQL Developer\system20.4.1.407.0006 from my old laptop, and let import copy SQL Developer setups from old computer. It keeps all database connections (ID & password). This is exactly what I want.

SQL Developer connection info is saved in location C:\Users\user_ID\AppData\Roaming\SQL Developer\systemX.X.X.X.X\o.jdeveloper.db.connection\connections. It is a good practice to back it up from time to time.

-- How to fine JAVA HOME and Java info on Windows?

On the Windows machine, use "cmd" window. 

Microsoft Windows [Version 10.0.19044.3324]
(c) Microsoft Corporation. All rights reserved.

c:\> echo %JAVA_HOME%
C:\Program Files\Java\jdk18;C:\Program Files (x86)\Java\jdk18

It shows two Java installations. The one in C:\Program Files\Java\jdk18 is a 64-bit,

c:\>cd \Program Files\Java
c:\Program Files\Java>dir
 ... ...
08/14/2023  09:21 AM    <DIR>          jdk18
08/14/2023  09:12 AM    <DIR>          jre8

c:\Program Files\Java>cd jdk18
c:\Program Files\Java\jdk18>dir /A:D
... ...
08/14/2023  09:21 AM    <DIR>          bin
08/14/2023  09:21 AM    <DIR>          include
08/14/2023  09:21 AM    <DIR>          jre
08/14/2023  09:21 AM    <DIR>          legal
08/14/2023  09:21 AM    <DIR>          lib

c:\Program Files\Java\jdk18>cd bin
c:\Program Files\Java\jdk18\bin>java.exe -version
java version "1.8.0_281"
Java(TM) SE Runtime Environment (build 1.8.0_281-b09)
Java HotSpot(TM) 64-Bit Server VM (build 25.281-b09, mixed mode)

The JDK in C:\Program Files (x86)\Java\jdk18 is 32-bit.

c:\Program Files\Java\jdk18>cd \Program Files (x86)\Java\jdk18
c:\Program Files (x86)\Java\jdk18>cd bin
c:\Program Files (x86)\Java\jdk18\bin>java -version
java version "1.8.0_281"
Java(TM) SE Runtime Environment (build 1.8.0_281-b09)
Java HotSpot(TM) Client VM (build 25.281-b09, mixed mode)

-- How to find out the Windows is 64-bit or 32-bit?

Go to Start > Settings (a wheel icon) > System > About

System Type: 64-bit operating system, x64-based processor

Friday, June 9, 2023

How to allow only specific users to access EBS site

During some special time periods, Business management wants to restrict users to access R12.2 EBS website but allow only one or two users to complete some dedicated tasks, such as month-end process. 

Oracle EBS has a way to do that. See Oracle Doc ID 605538.1 (How To Lock Users Out Of E-Business Suite And Allow Specific Users). This only works if EBS website does not have multiple web nodes which may use load balancing (such as F5) in the middle.

Our users use VPN to log onto company's network and PC. After I added below lines to file $IAS_ORACLE_HOME/instances/EBS_web_OHS1/config/OHS/EBS_web/custom.conf and bounced apps services, only users use the two IP addresses can access our EBS site and all other users get "Forbidden oops." message on the login page. 

<Location ~ "/OA_HTML">
Order deny,allow
Deny from all
Allow from 10.55.xxx.92
Allow from 10.53.xxx.134
ErrorDocument 403 "Forbidden oops."
Allow from localhost
</Location>

Use "grep ohs_inst $CONTEXT_FILE" to identify the OHS# (i.e. EBS_web_OHS1) and the location.

If load balancing is used, above method may not work because end-user's IP may not reach EBS server. Other ways can be used to accomplish this but additional work is needed.
1. Create a special Responsibility. Then, disable all other Responsibilities. Only users in the new Responsibility will be able to log into EBS site to do the work.
2. You can disable/enable users in bulk using API. But this will change last_update_date and last_updated_by of table fnd_user, and may become a security auditing concern (specially when we have 57,000+ active users in the system. Huge worry is users' password would still work).
    fnd_user_pkg.disableuser('<username>');
    fnd_user_pkg.enableuser('<username>');
3. Ask internal firewall team to restrict access to the EBS Load Balancer URL to specific IP addresses.
4. Change the Load Balancer port to a temporary port that only will be given to the authorized users, and after the archive process is complete, the port can be switched back to the original port so it becomes available to all users.

An alternative solution is keep running a query to monitor who is using EBS site.

Friday, May 12, 2023

How to capture HTTP Header information from the client browser

Steps to generate the HAR file for Chrome (Version 112.0.5615.140). It is part of Oracle Doc ID 815734.1 (How to Gather HTTP Header Information From a Client PC)

1.    Open Google Chrome and go to the page (such as EBS login page) where the issue is occurring.
2.    Bring the Chrome menu by clicking on the upper right > More tools > Developer Tools.
3.    From the panel opened at the bottom of your screen, select the Network tab.
4.    Look for a round Record button (Record button) in the upper left corner of the Network tab, and make sure it is red. If it is grey, click it once to start recording.
5.    Check the box next to Preserve log .
6.    Click the Clear button ( Clear button ) to clear out any existing logs from the Network tab.

7.    Now try to reproduce the issue that you were experiencing before, while the network requests are being recorded.
8.    Once you have reproduced the issue, right click anywhere on the grid of network requests, select Save as HAR with Content, and save the file to your computer.
9.    Upload your HAR file to your ticket or attach it to your email so that we may analyze it.


Somehow, during the capture, I do see some menu items are missing from System Administrator responsibility.  

 

Monday, May 1, 2023

Which EBS server did a user session log into

Oracle Doc ID 364439.1 (Tips and Queries for Troubleshooting Advanced Topologies) provides a few useful queries. One of them is to tell what server a user logged into.

A query to show and monitor all users who logged to the system within the last hour.

col server_name format a12
col user_name format a30
select
  to_char(first_connect, 'HH24:MI:SS') "TIME",
  user_name,
  decode
   (a.node_id,
   a.node_id,
   (select node_name  
   from fnd_nodes n  
   where a.node_id=n.node_id),a.node_id) server_name
from
  icx_sessions a,
  fnd_user b,
  fnd_nodes svr
where
  first_connect > (sysdate-1/24)
  and (a.user_id=b.user_id
       and a.node_id=svr.node_id)
  and disabled_flag='N'
order by first_connect;

TIME      USER_NAME         SERVER_NAME
----------- --------------------------- ------------
20:11:21 SYSADMIN          HOSTBE
20:17:53 USER                   HOSTINT
20:19:16 OPERATIONS     HOSTEMT
20:39:21 GUEST                HOSTPC8  (did not login, hit "forgot password")

Tuesday, April 25, 2023

Upgrade AD and TXK from Delta 12 to Delta 14 RUPs in R12.2

AD.C.Delta.14 (patch 33600809) and TXK.C.Delta.14 (patch 33602997) are pre-requisite for January 2023 CPU patches (see Doc ID 2916871.1). We have to apply them before that CPU patch set can be applied to R12.2 instance.

Oracle document ID 1617461.1 (Applying the Latest AD and TXK Release Update Packs to Oracle E-Business Suite Release 12.2) gives details on how to apply the latest Oracle Applications DBA (AD) and Oracle E-Business Suite Technology Stack (TXK) release update packs (RUPs) to EBS R12.2. I performed below steps to upgrade them to delta 14 from delta 12 in my instances where existing package levels are:
R12.AD.C.Delta.12
R12.TXK.C.Delta.12
R12.ATG_PF.C.delta.9 (30399994)

1. DBA runs the latest version of ETCC (via Patch 17537119), and ensure all required database objects by the Delta 14 RUP installation process will be found. One of recent database PSU patches will help to achieve this.

The following optimizer parameter should always be set to a value of TRUE:
_disable_actualization_for_grant=true

2. Pre-step: download all .zip files and unzip them to $PATCH_TOP (or an alternation folder) of all nodes. Then, DBA runs $PATCH_TOP/34669333/admin/adgrants.sql as SYSDBA

AD critical patch 34669333 has the latest adgrants.sql, as of now. Please read the instruction in the file before run it.

$ grep Header $PATCH_TOP/34669333/admin/adgrants.sql
REM $Header: adgrants.sql 120.67.12020000.123 2022/09/30 21:13:17 jwsmith ship $

3. Verify new db account ebs_system. 
adgrants.sql creates new database account ebs_system. Unlock it and change its password to be the same as SYSTEM's password (if it has not been done yet). 
$ sqlplus system/systemPWD
SQL> alter user ebs_system account unlock;
SQL> alter user ebs_system identified by systemPWD;
SQL> conn ebs_system/systemPWD

4. Run the validation script
$ perl $PATCH_TOP/33600809/ad/bin/adValidateEbssystemSchema.pl

Enter the APPS password:
Enter the SYSTEM password:
Validating APPS credentials...validated successfully
Validating SYSTEM credentials...validated successfully
Validating EBS_SYSTEM user
ebssys entity is not registered
All required validations completed successfully

5. Optional: Validate adop, and stop all services (if use downtime mode to apply the patch)
$ adop -validate
$ adop -status

$ cd $ADMIN_SCRIPTS_HOME
$ ./adadminsrvctl.sh stop
$ ./adnodemgrctl.sh stop

$ ps -ef | grep $LOGNAME

6. Apply AD.C.Delta.14 patch, plus 3 critical patches, by running below line on primary mode. I used downtime mode to apply them.

$ adop phase=apply apply_mode=downtime patches=33600809,34668508,34669333,34681299 merge=yes patchtop=$PATCH_TOP

Enter the APPS password:
Enter the SYSTEM password:
Enter the WLSADMIN password:

If patching gets error, most likely there is a problem with something else. Confirm 4 patches were applied to all nodes:
SQL> SELECT adb.bug_number, aas.name appl_top_name, adb.creation_date, adb.language,
decode(ad_patch.is_patch_applied('R12',aas.appl_top_id,adb.bug_number,adb.language),'EXPLICIT','APPLIED','NOT_APPLIED','NOT APPLIED') status
FROM ad_bugs adb,
(select distinct appltop_id appl_top_id, node_name name from ad_adop_sessions 
           where node_name in (select node_name from ADOP_VALID_NODES) ) aas
where adb.bug_number in (
'33600809',
'34668508',
'34669333',
'34681299'
) order by creation_date desc, adb.bug_number,aas.name,adb.language;

UPDATE in July 2023: patch 35280947 was added as a AD critical patch in the document. When they were applied to an instance with one single node, the adop log is

$ adop phase=apply apply_mode=downtime patches=33600809,34668508,34669333,34681299,35280947 merge=yes patchtop=$PATCH_TOP
... ...
Copying files...
5% complete. Copied 26 files of 519...
10% complete. Copied 52 files of 519...
15% complete. Copied 78 files of 519...
20% complete. Copied 104 files of 519...
25% complete. Copied 130 files of 519...
30% complete. Copied 156 files of 519...
35% complete. Copied 182 files of 519...
40% complete. Copied 208 files of 519...
45% complete. Copied 234 files of 519...
50% complete. Copied 260 files of 519...
55% complete. Copied 286 files of 519...
60% complete. Copied 312 files of 519...
65% complete. Copied 338 files of 519...
70% complete. Copied 364 files of 519...
75% complete. Copied 390 files of 519...
80% complete. Copied 416 files of 519...
85% complete. Copied 442 files of 519...
90% complete. Copied 468 files of 519...
95% complete. Copied 494 files of 519...
100% complete. Copied 519 files of 519...

Character-set converting files...
  5 unified drivers merged.
Patch merge completed successfully

Please check the log file at $ADOP_LOG_HOME/8/20230804_163029/apply/$EBS_HOSTNAME/admrgpch.log.

Applying patch ADOP_MRG_20230804_1691181079:u_ad_3366500462.drv.
    Log: $ADOP_LOG_HOME/8/20230804_163029/apply/$EBS_HOSTNAME/ADOP_MRG_20230804_1691181079/log/u_ad_3366500462.log

Running finalize actions for the patches being applied.
    Log: @ADZDSHOWLOG.sql "2023/08/04 16:36:47"

Running cutover actions for the patches being applied.
    Creating workers to process cutover DDL in parallel
    Log: $ADOP_LOG_HOME/8/20230804_163029/apply/$EBS_HOSTNAME/log/cutover.log
    Performing database cutover in Quick mode

Generating post apply reports.

Generating log report.
    Output: $ADOP_LOG_HOME/8/20230804_163029/apply/$EBS_HOSTNAME/adzdshowlog.out

The apply phase completed successfully.
adop exiting with status = 0 (Success)

7. Apply TXK.C.Delta.14 patch, plus 2 critical patches, by below line on primary mode. Note it now asks for EBS_SYSTEM password.

$ adop phase=apply apply_mode=downtime patches=33602997,34708635,34654260 merge=yes patchtop=$PATCH_TOP

Enter the APPS password:
Enter the EBS_SYSTEM password:
Enter the WLSADMIN password:

Confirm 3 patches were applied successfully:
SQL> SELECT adb.bug_number, aas.name appl_top_name, adb.creation_date, adb.language,
decode(ad_patch.is_patch_applied('R12',aas.appl_top_id,adb.bug_number,adb.language),'EXPLICIT','APPLIED','NOT_APPLIED','NOT APPLIED') status
FROM ad_bugs adb,
(select distinct appltop_id appl_top_id, node_name name from ad_adop_sessions 
           where node_name in (select node_name from ADOP_VALID_NODES) ) aas
where adb.bug_number in (
'33602997',
'34708635',
'34654260'
) order by creation_date desc, adb.bug_number,aas.name,adb.language;

8. Post steps
$ perl $AD_TOP/bin/admkappsutil.pl
$ cp -p $INST_TOP/admin/out/appsutil.zip $APPLPTMP

$ Autoconfig on db node
$ Autoconfig on all apps nodes

9. Start Apps services
$ ./adstrtal.sh apps/aapsPWD -mode=allnodes

10. After the instance was verified and checked out, run FS_CLONE to sync the file systems.

NOTES: adgrants.sql from patch 34669333 makes important changes in database. During above patching, it is copied to $APPL_TOP/admin as the current one. Below is its log by SYSDBA: 

Current user is SYS
------------------------------------------------------------
--- adgrants.sql started at 2023-04-20 11:15:11 ---
------------------------------------------------------------
Removing logs from prior runs of adgrants.sql
-
Start granting from SYS to EBS_SYSTEM
-
End granting from SYS to EBS_SYSTEM
Completed granting and checking privileges
Generating list of ERRORS and WARNINGS to print out
PRINT_ERROR
---------------------
:current_user

CURRENT_USER
-----------------------------------------------------------------------------------------
SYS user ERRORS and WARNINGS will report at end of script
to print out
CURRENT_USER
-----------------------------------------------------------------------------------------
SYS user ERRORS and WARNINGS will report at end of script
Creating PL/SQL profiler objects.
---------------------------------------------------------------
--- profload.sql started at 2023-04-25 11:15:14 ---

In 12.2c and beyond, the Oracle-supplied profload.sql script is a verification script, not an installation script.
Testing for correct installation
SYS.DBMS_PROFILER successfully loaded.
-----------------------------------------------------
--- profload.sql completed at 2023-04-25 11:15:16 ---
--------------------------------------------------
--- proftab.sql started at 2023-04-25 11:15:16 ---
-----------------------------------------------------
--- profltab.sql completed at 2023-04-25 11:15:18 ---

Installing Hierarchical Profiler.
-
Loading Stylesheets if missing
Begin creating the AD_ZD_SYS package
End creating the AD_ZD_SYS package
Executing PURGE DBA_RECYCLEBIN.
The following ERRORS and WARNINGS have been encountered during this adgrants session:
Grants given by this script have been written to the ad_zd_logs table.
You can run $AD_TOP/sql/ADZDSHOWLOG.sql to produce a report showing these grants.

Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.18.0.0.0
Started at 2023-04-25 11:15:14 ---

In 12.2c and beyond, the Oracle-supplied profload.sql script is a verification script, not an installation script.
Testing for correct installation
SYS.DBMS_PROFILER successfully loaded.
-----------------------------------------------------
--- profload.sql completed at 2023-04-25 11:15:16 ---
--------------------------------------------------
--- proftab.sql started at 2023-04-25 11:15:16 ---
-----------------------------------------------------
--- profltab.sql completed at 2023-04-25 11:15:18 ---

Installing Hierarchical Profiler.

HPTAB_SCRIPT
---------------------------
?/rdbms/admin/nothing.sql
-
Loading Stylesheets if missing
PACKAGE_NAME
------------------------
AD_ZD_SYS
Begin creating the AD_ZD_SYS package
End creating the AD_ZD_SYS package
Executing PURGE DBA_RECYCLEBIN.
The following ERRORS and WARNINGS have been encountered during this adgrants session:
Grants given by this script have been written to the ad_zd_logs table.
You can run $AD_TOP/sql/ADZDSHOWLOG.sql to produce a report showing these grants.

Friday, March 3, 2023

JVM version in an Oracle database

Two ways can be used to check/confirm the JVM (Java Virtual Machine) version in a 19c database.

1. Set env for database 

$ echo $ORACLE_HOME
/u01/app/oracle/product/19.0.0.0/db_ebsdev
$ opatch lsinventory|grep "Patch description"
Patch description:  "INCREASE _LM_RES_HASH_BUCKET AND BACK OUT CHANGES FROM THE BUG 29416368 FIX"
Patch description:  "ORA-600 [KTSL_ALLOCATE_DISP KCBZ_OBJDCHK] ORA-600 [KCBZIB_6] SECUREFILE DOUBLE ALLOCATION"
Patch description:  "MERGE ON DATABASE RU 19.15.0.0.0 OF 28318139 28357349 28555193 28771564 29276889"
Patch description:  "DBMS_STATS.IMPORT_DATABASE_STATS FAILING WITH ERROR OPERATION NOT ALLOWED FROM WITHIN A PLUGGABLE DATABASE"
Patch description:  "IMPDP TAKES LONG TIME WHEN IMPORTING BASELINES"
Patch description:  "OJVM RELEASE UPDATE: 19.15.0.0.220419 (33808367)"
Patch description:  "Database Release Update : 19.15.0.0.220419 (33806152)"
Patch description:  "OCW RELEASE UPDATE 19.9.0.0.0 (31772784)"
$
$ opatch lspatches
29780459;INCREASE _LM_RES_HASH_BUCKET AND BACK OUT CHANGES FROM THE BUG 29416368 FIX
32455516;ORA-600 [KTSL_ALLOCATE_DISP KCBZ_OBJDCHK] ORA-600 [KCBZIB_6] SECUREFILE DOUBLE ALLOCATION
33956157;MERGE ON DATABASE RU 19.15.0.0.0 OF 28318139 28357349 28555193 28771564 29276889
33405295;DBMS_STATS.IMPORT_DATABASE_STATS FAILING WITH ERROR OPERATION NOT ALLOWED FROM WITHIN A PLUGGABLE DATABASE
31665107;IMPDP TAKES LONG TIME WHEN IMPORTING BASELINES
33808367;OJVM RELEASE UPDATE: 19.15.0.0.220419 (33808367)
33806152;Database Release Update : 19.15.0.0.220419 (33806152)
31772784;OCW RELEASE UPDATE 19.9.0.0.0 (31772784)

OPatch succeeded.

2.  Login to database and run below 

SET PAGESIZE 60
SET LINESIZE 300
SET VERIFY OFF
COL comp_name FOR a44 HEA 'Component'
COL version FOR a17 HEA 'Version'
COL VERSION_FULL FOR a17 HEA 'VERSION_FULL'
COL status FOR a17 HEA 'Status'
SELECT comp_name, version,VERSION_FULL, status FROM dba_registry;

Component                                 Version           VERSION_FULL   Status
-------------------------------------------- ----------------- ----------------- -----------------
JServer JAVA Virtual Machine    19.0.0.0.0       9.15.0.0.0            VALID
... ...

Thursday, January 19, 2023

How to setup a custom env variable in R12.2

In R12.2, custom env variables can be defined in file $INST_TOP/appl/admin/custom$CONTEXT_NAME.env (after it is created), which is called by $APPL_TOP/APPS$CONTEXT_NAME.env (and so, by the main env file EBSapps.env).  For example, use two lines in custom$CONTEXT_NAME.env to define a custom variable:
$ cd $INST_TOP/appl/admin
$ more custom$CONTEXT_NAME.env
FORMS_RECORD_GROUP_MAX=30000
export FORMS_RECORD_GROUP_MAX
$ echo $FORMS_RECORD_GROUP_MAX
30000

Another way is to use the "custom" folder under $AD_TOP/admin/template. Oracle Doc ID 745711.1 gives steps on defining a Forms env variable:

a) Go to the autoconfig Template folder:
$ cd $AD_TOP/admin/template
b) Create new directory named (custom)
$ mkdir custom
c) Make sure that new directory has same file permissions as ($AD_TOP/admin/template)
d) Copy the following autoconfig template to the new custom directory:
$ cp $AD_TOP/admin/template/APPLSYS_ux.env $AD_TOP/admin/template/custom/APPLSYS_ux.env
e) Edit the file copied file under custom directory and add the following 2 lines at the end of section:

####################################
# Oracle Forms environment variables
####################################
FORMS_RECORD_GROUP_MAX=30000
export FORMS_RECORD_GROUP_MAX

f) Save and exit from the file.
g) Next time autoconfig run, it will read the custom directory and check for any customizations there.

After AutoConfig, above two lines will be copied to file $APPL_TOP/$CONTEXT_NAME.env (and other AD files?).

Notes: "FORMS_RECORD_GROUP_MAX=30000" can be equivalently added to Forms file $INST_TOP/ora/10.1.2/forms/server/default.env. But that file will be overwritten by AutoConfig. See Oracle Doc ID 745711.1.

Sunday, January 15, 2023

EBS session status in the database & on OS

When an OS process uses high CPU or other OS resources, use below line to find its start time:
$  ps -eo pid,lstart,cmd | grep 21846
74063 Tue Dec 19 12:29:31 2022 grep --color=auto 21486
21486 Tue Dec 19 07:39:01 2022 frmweb server webfile=HTTP-0,0,1,default

If a forms process (frmweb) exists on OS level but does not have a responding session in the database, it is a run-away and it can be terminated.  

Below SQL statement will tell and monitor what an EBS session is doing in the database after its process ID is identified (e.g. 21846) on OS level of EBS apps server. 

SQL> SELECT to_char(S.logon_time, 'DD-MON-RRRR HH24:MI:SS') "logon_time", to_char(S.PREV_EXEC_START, 'DD-MON-RRRR HH24:MI:SS') "last op started", 
    client_identifier, module, action, status, machine, round(last_call_et/60,1) "Minutes in current status",
    sql_id, blocking_session, process, sid, serial#,
    STATE,
    ROUND((S.WAIT_TIME_MICRO/1000000)/60/60,1) "total wait hrs",
    DECODE(S.TIME_REMAINING_MICRO,'-1', 'indefinite', '0','Wait timed out',NULL,'session not waiting') "remaining wait time", 
    DECODE(S.TIME_SINCE_LAST_WAIT_MICRO, '0','Still waiting') "current wait status",
    S.time_since_last_wait_micro
   FROM v$session s
WHERE process = '21846' 
-- order by client_identifier
;
  
  BLOCKING_SESSION - Tell if it is blocked by another session.
  STATE - The state of the wait event.
  WAIT_TIME_MICRO - The wait time of the wait event in microseconds.
  TIME_REMAINING_MICRO - The time remaining in microseconds before the wait event times out.
  TIME_SINCE_LAST_WAIT_MICRO - The amount of time in microseconds elapsed from the last wait to the current wait.

Normally, a session for a running concurrent request will show the SQL_ID.

An oacore process may have many sessions in the database. To find which process ID is for oacore, try "$ ps -ef | grep oacore " on OS level of apps server, or run below line in the database (if XX is part of a CUSTOM top name).

SQL> select distinct process, machine from v$session 
where action like '%XX%' order by machine;

When an OACORE process taking high CPU, find the process ID and run following commands :

1) ps -eLo pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:14,comm | grep PID
==> which thread is taking CPU.  Most of them should show 0.0 (which is the CPU column).

I did see an OACORE process (e.g. 13030) supports 125 JVM threads in one of my instances. So, it is not good to simply kill an oacore process.
$ ps -eLo pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:14,comm | grep 13030 | wc -l
125

2) jmap -histo <jvm_process_id> > /tmp/ <jvm_process_id>.histo
==> will dump heap usage details.  

3) jmap -dump:format=b,file= <jvm_process_id>.jmap <jvm_process_id>
==> will generate an actual heap dump, binary format

4) kill -3 <jvm_process_id>
==> output goes to 12.2 oacore_server1.out. It will generate stack trace, run twice a minute apart.
kill -3 <jvm_process_id>

5) lsof -p <jvm_process_id> > <jvm_process_id>.log
==> List of open ports/files

6) SELECT audsid, module, last_call_et, action from gv$session where process = '&jvm_process_id';
==> corresponding DB sessions

7) log / incident files from that oacore :

$EBS_DOMAIN_HOME/servers/oa*/logs/*
$EBS_DOMAIN_HOME/servers/oa*/adr/diag/ofm/EBS_domain*/oacore*/incident/incdir*/*

Tuesday, January 10, 2023

SQL statement to find blocking session

Use SQL to identify and monitor the database session that blocks other session:

SQL> SELECT DECODE(request,0,'Holder: ','Waiter: ') status, sid sess_ID, id1, id2, lmode, request, type
FROM    V$LOCK
WHERE (id1, id2, type) IN
(SELECT id1, id2, type FROM V$LOCK WHERE request > 0)
ORDER BY id1, request;

Use below to confirm the blocking status in the database:

SQL> SELECT blocking_session block_holder,
  sid, serial#, process, machine, client_identifier, wait_class, seconds_in_wait
FROM v$session
WHERE blocking_session IS NOT NULL
ORDER BY blocking_session; 

Wednesday, January 4, 2023

Purge R12.2 log files in Linux OS

Folder structure in R12.2 file system is more complicated than in R12.1, because RUN and PATCH may switch their location. After an adop cutover, the path to log files will be changed to a different one. Below are folders that have a growing number of log files. "ls" can be used to list the log files but "cd" do not work to reach the folder, while the path uses "*" in it.

$IAS_ORACLE_HOME/instances/*/diagnostics/logs/OHS/*/*.log*
$EBS_DOMAIN_HOME/servers/oacore*/logs/*.log*
$EBS_DOMAIN_HOME/servers/form*/logs/*.log*
$EBS_DOMAIN_HOME/servers/AdminServer/logs/*.log*

Fortunately, "find" works with path having "*" in it. Below lines work in cron to purge/delete old log files (on web/forms node).

$ crontab -l
05 05 * * * . /u07/app/.profile; /usr/bin/find $IAS_ORACLE_HOME/instances/*/diagnostics/logs/OHS/*/*log* -type f -mtime +30 -exec rm -f {} \;
05 15 * * * . /u07/app/.profile; /usr/bin/find $EBS_DOMAIN_HOME/servers/oacore*/logs/*log* -type f -mtime +30 -exec rm -f {} \;

Other logs or their locations:

$ADOP_LOG_HOME/*                   <== each ADOP session ID has a sub-folder
$INST_TOP/admin/log/MMDDHHMM/adconfig.log
$LOG_HOME/appl/admin/log/*      <== adstrtal.sh & adstpall.sh 
$LOG_HOME/appl/rgf/javacache.log 
$IAS_ORACLE_HOME/instances/*/diagnostics/logs/OPMN/opmn/* 
$EBS_DOMAIN_HOME/sysman/log/* 
$EBS_DOMAIN_HOME/servers/oac*/adr/diag/ofm/EBS_domain*/oac*/incident/* 
$EBS_DOMAIN_HOME/servers/forms_s*/adr/diag/ofm/EBS_domain*/forms_s*/incident/*
$INST_TOP/logs/ora/10.1.2/reports/cache/*.*

Locations or files for configuration and setups:

$IAS_ORACLE_HOME/instances/EBS_web*/config/OHS/EBS_web* 
$FMW_HOME/webtier/instances/EBS_web*/config/OPMN/opmn/* 
$CONTEXT_FILE
$INST_TOP/appl/admin/*.properties
$INST_TOP/ora/10.1.2/forms/server/default.env    (AutoConfig overwrites it)
$FND_TOP/fndenv.env
$EBS_DOMAIN_HOME/config/config.xml
$EBS_APPS_DEPLOYMENT_DIR/*/APP-INF/node_info.txt

.profile file (for account runs the crontab jobs on RHEL)
$ more .profile
PATH=/bin:/usr/bin:/usr/local/bin
export PATH

MANPATH=/usr/share/man:/usr/local/share/man:/usr/X11R6/man
export MANPATH

EDITOR=/bin/vi
export EDITOR

. /u07/app/EBSDEV/EBSapps.env RUN

isMaster="no"
if [ ! -z $APPS_VERSION ] && [ ${APPS_VERSION:0:4} == "12.2" ]
then
s_status=`cat $CONTEXT_FILE | grep -i s_adminserverstatus`
isMaster="${s_status:60:7}"
fi

if [ $isMaster == "enabled" ]
then

if [ `echo -n ${HOSTNAME%%.*} | tail -c -1` != "p" ]
then
PS1=$'
\e[0;31m$USER@${HOSTNAME%%.*}[$TWO_TASK]\e[m$PWD
-->$ '
else
PS1=$'
\e[0;31m$USER@${HOSTNAME%%.*}[$TWO_TASK]\e[m\E[32m$PWD \E[0m
-->$ '
fi

else

if [ `echo -n ${HOSTNAME%%.*} | tail -c -1` != "p" ]
then
PS1='
$USER@${HOSTNAME%%.*}[$TWO_TASK]$PWD
-->$ '
else
PS1=$'
$USER@${HOSTNAME%%.*}[$TWO_TASK]\E[32m$PWD \E[0m
-->$ '
fi

fi

alias rm='rm -i'
stty erase ^?