ERP on DB

Friday, November 29, 2024

Apply R12.2 October 2024 CPU patches

The document for October 2024 CPU patches is Doc ID 3037725.1 (Oracle E-Business Suite Release 12.2 Critical Patch Update Availability Document - October 2024). We have to follow it to apply this CPU patches.

1. First of all, specify a location for holding all patch files. All zip file shall be downloaded/saved in sub-folders of this location

$ patch_folder=/path/to/CPU_Patches/Oct2024

2. Run EJCPUC script on database server to check the Java versions. A new utility, Oracle E-Business Suite Java Critical Patch Update Checker (EJCPUC, Patch 37171025), was released with the Oct 2024 CPU. EJCPUC Output:

$ bash ejcpuc.sh

#############################################################

## Checking DB tier Java for CPU 2024.10 on Platform IBM_AIX

#############################################################

## Check Database Version

#############################################################

Your database version is 19.25.0.0.0

ORACLE_HOME $ORACLE_HOME

ORACLE_SID EBSDEV

ORACLE_UNQNAME

## Check Java Version of OJVM, Database JDK and EBS's appsutil JRE

#############################################################

Latest Version action Your Version bitness Java Location

-------------- -------- ------------ ------- ---------------

1.8.0_431 o) _.__.101034000 64-bit OJVM In database

1.8.0_411 1.8.0_421 64-bit $ORACLE_HOME/jdk/bin/java

1.8.0_411 u) 1.8.0_271 64-bit $ORACLE_HOME/appsutil/jre/bin/java

o) Apply the Database Release Update (DBRU) recommended by ETCC which will update the DB OJVM version to the latest

u) When the DB JDK version is updated to the latest - then follow section 3 of 1530033.1 to update this JRE

3. Download ETCC (patch 17537119). Make sure both database and apps use the same release of ETCC

$ cd $patch_folder/ETCC

$ grep 120 *.sh

checkDBpatch.sh:# $Header: checkDBpatch.sh 120.127 2024/11/12 15:02:31 chrhill noship $

checkMTpatch.sh:# $Header: checkMTpatch.sh 120.0.12020000.68 2024/04/12 16:35:27 chrhill noship

3. DBAs apply database patches (after EBS services are stopped) to the database and make sure all requirements are met. ETCC Output :

$ ./checkDBpatch.sh

+==========================================+

| Oracle E-Business Suite Release 12.2 |

| Database EBS Technology Codelevel Checker |

+==========================================+

Database environment not set, going to check for GridHome.

Oracle Grid Infrastructure not identified.

Database environment not set and no Grid home found, so context file must be specified.

Enter full path to database context file: $ORACLE_HOME/appsutil/<CONTEXT_NAME>.xml

Validating context file: $ORACLE_HOME/appsutil/<CONTEXT_NAME>.xml

Using context file from user input:

$ORACLE_HOME/appsutil/<CONTEXT_NAME>.xml

Starting Database EBS Technology Codelevel Checker, Version 120.127

Mon Nov 25 16:41:09 EST 2024

Log file for this session: $ORACLE_HOME/appsutil/etcc/log/checkDBpatch_45744600.log

Identifying database release.

Database release set to 19.25.0.0.

Multitenant identified.

- Container database (CDB) identified via s_cdb_name is CEBSDEV

- Pluggable database (PDB) identified via s_pdb_name is EBSDEV

Connecting to database.

Database connection successful.

Database EBSDEV is in READ WRITE mode.

Identifying APPS and APPLSYS schema names.

- APPS schema: APPS

- APPLSYS schema: APPLSYS

Checking for existence DB-ETCC results table.

Table to store DB-ETCC results already exists in the database.

Bugfix file ./db/onprem/txk_R1220_DB_base_bugs.xml: 120.0.12020000.85

This file will be used for identifying missing bugfixes.

Mapping file ./db/onprem/txk_R1220_DB_mappings.xml: 120.0.12020000.61

This file will be used for mapping bugfixes to patches.

+---------------------------------------------------------------------------------------+

Always use the latest version of ETCC available in patch 17537119,

as new bugfixes will not be checked by older versions of the utility.

You should apply the latest recommended RU, BP, or PSU as appropriate.

+---------------------------------------------------------------------------------------+

Identified database DST version: 32

Checking Bugfix XML file for tag 19.25.0.0_RU.

Obtained list of bugfixes to be applied and list to be rolled back.

Validating OPatch version:

The OPatch utility is version 12.2.0.1.44.

DB-ETCC is compatible with this OPatch version.

Checking for applied patch history:

Found patch history in the inventory.

Checking mapping XML file for tag 19.25.0.0.241015DBRU.

All the required one-off bugfixes are present in database ORACLE_HOME.

5. Proceed with Apps patches

- Back up Apps file systems

- Start Apps services to make sure EBS works after database patching.

- Stop Apps services

6. ECPUC script (from patch p35583866) to identify what EBS CPU patches and security fixes are needed.

$ cd $patch_folder/ECPUC

$ sqlplus apps/appsPWD

SQL> @ECPUC.sql

... ...

E-Business Suite Critical Patch Update Checker (ECPUC)

ECPUC.sql may be run on any EBS 12.2 environment to identify missing patches that are in the latest EBS CPU.

You can download the latest version of ECPUC via Patch 35583866.

Refer to the README.txt in Patch 35583866 for instructions for running ECPUC and information regarding the generated ECPUC.lst report.

The checker generates the report ECPUC_YYYY-MM-DD_HH24-MI.lst that lists recommended EBS CPU patches and security fixes for your environment per Table 1 'CPU Patches for Oracle E-Business Suite' and Table 2 'Additional Patches Required' documented in the quarterly EBS CPU MOS Note.

Each quarterly EBS CPU MOS Note ID is unique. Refer to My Oracle Support (MOS) Knowledge Document 2484000.1, 'Identifying the Latest Critical Patch Update for Oracle E-Business Suite Release 12' which includes a link to the current EBS CPU MOS document.

============================================

SECTION-1 ECPUC Version

============================================

EBS CPU Checker Version

-----------------------

2024.1

============================================

SECTION-2 Oracle E-Business Suite (EBS): Instance Information

============================================

***********************************************************

Instance Summary

***********************************************************

EBS Release EBS CPU Level

----------- -------------

12.2.10 2023.10

Instance Name Database Database Version

---------------- ---------------- ------------------

CEBSDEV EBSDEV 19.25.0.0.0

ATG Product Product Name Code Level

------------ ----------------------------------------------- ----------

ad Applications DBA C.14

txk Oracle Applications Technology Stack C.14

atg_pf Oracle Applications Technology Family C.9

fwk Oracle Applications Framework C.9

===========================================

SECTION-3 Required EBS CPU and Security Fixes

===========================================

The following output is a list of required patches and security fixes that are missing in your environment.

It is strongly recommended that you apply all of the listed patches as soon as possible.

If no patches (no rows) are listed then no additional action is required at this time as your environment includes all patches for this EBS CPU.

**********************************************************

The following patches are required for this EBS CPU

--------------------------------------------------------------

36944346:12.2.0

37078855:R12.ATG_PF.C

37078915:R12.SCM_PF.C

37120482:R12.CC_PF.C

35362524:R12.IGI.C

34979060:R12.MSC.C

33457157:R12.HXT.C

30448458:R12.HXT.C

8 rows selected.

7. Apply two EBS patches. After consulting with Developers and Business users, we will apply only first two patches because we do not use the EBS modules on the list.

36944346:12.2.0

37078855:R12.ATG_PF.C

Make sure all Apps services are down

$ ps -ef | grep $LOGNAME

$ vi /etc/oraInst.loc

$ adop -status

If FS_CLONE has not been run, run it. This will help to screen out errors in applying new patches.

$ adop phase=fs_clone

-- Apply patch to multiple nodes

$ adop phase=apply apply_mode=downtime patches=36944346 patchtop=$patch_folder/Oct24_CPU

$ adop phase=apply apply_mode=downtime patches=37078855 patchtop=$patch_folder/Post

Run query to confirm Oct 2024 CPU in the instance. See Doc ID 2484000.1 (Identifying the Latest Critical Patch Update for Oracle E-Business Suite Release 12.2)

SQL> col CPU format a9

SQL> select max(CODELEVEL) "CPU" from ad_trackable_entities where abbreviation in ('ebscpu');

CPU

---------

2024.10

8. Run autoconfig on database server

$ perl $AD_TOP/bin/admkappsutil.pl

Starting the generation of appsutil.zip

Log file located at $INST_TOP/admin/log/MakeAppsUtil_11260913.log

output located at $INST_TOP/admin/out/appsutil.zip

MakeAppsUtil completed successfully.

$ cp -p $INST_TOP/admin/out/appsutil.zip $APPLPTMP

$ chmod 777 $APPLPTMP/appsutil.zip

DBA run the below steps on EBSDEV database.

1. cp /u04/shared/utl_dir/appsutil.zip $ORACLE_HOME/

2. cd $ORACLE_HOME; unzip -o appsutil.zip

3. Set env and then run autoconfig on the database.

9. Run ETCC

$ cd $patch_folder/ETCC

$ ./checkMTpatch.sh

... ...

Starting Application Tier EBS Technology Codelevel Checker, Version 120.0.12020000.68.

... ...

=========================================

PATCH RECOMMENDATION SUMMARY

=========================================

One or more products have bugfixes missing.

The default patch recommendations to install these missing bugfixes are:

-------------------------------------------------------------------------------

Oracle Fusion Middleware (FMW) - Oracle Common 11.1.1.9.0

-------------------------------------------------------------------------------

Patch 34714760

- Filename: p34714760_111190_Generic.zip

-------------------------------------------------------------------------------

WLS 10.3.6.0.231017

-------------------------------------------------------------------------------

Patch 35476084 [SU Patch [KMHV]]

- Filename: p35476084_1036_Linux-x86-64.zip

Apply the required patches and rerun this script.

+-----------------------------------------------------------------------------+

A consolidated zip file with the required application tier patches is

available on My Oracle Support via:

Patch 36616672

10. Apply WLS patches: KMHV, WY44

$ echo $APPL_TOP

$ echo $ORACLE_HOME

$RUN_BASE/EBSapps/10.1.2

$ echo $FMW_HOME

$RUN_BASE/FMW_Home

$ cd $FMW_HOME/utils/bsu

-- Check if patch p13845626 was applied:

$ ./bsu.sh -prod_dir=$FMW_HOME/wlserver_10.3 -status=applied -verbose -view | egrep -i 'WY44'

Patch ID: WY44

PatchContainer: WY44.jar

$ cd $FMW_HOME/utils/bsu/cache_dir

$ cp -p $patch_folder/WLS/35476084/p35476084_1036_Linux-x86-64.zip . # (KMHV)

$ cp -p $patch_folder/WLS/13845626/p13845626_10360231017_Generic.zip .

# (WY44. Asked by Doc ID 3037725.1)

$ unzip p35476084_1036_Linux-x86-64.zip

Archive: p35476084_1036_Linux-x86-64.zip

replace README.txt? [y]es, [n]o, [A]ll, [N]one, [r]ename: A

inflating: README.txt

inflating: patch-catalog_27986.xml

inflating: KMHV.jar

$ cd ..

$ ./bsu.sh -install -patch_download_dir=$FMW_HOME/utils/bsu/cache_dir -patchlist=KMHV -prod_dir=$FMW_HOME/wlserver_10.3

Checking for conflicts..

Conflict(s) detected - resolve conflict condition and execute patch installation again

Conflict condition details follow:

Patch KMHV is mutually exclusive and cannot coexist with patch(es): CW7X

$ ./bsu.sh -remove -patchlist=CW7X -prod_dir=$FMW_HOME/wlserver_10.3

Checking for conflicts..

No conflict(s) detected

Removing Patch ID: CW7X..

Result: Success

$ ./bsu.sh -install -patch_download_dir=$FMW_HOME/utils/bsu/cache_dir -patchlist=KMHV -prod_dir=$FMW_HOME/wlserver_10.3

Checking for conflicts..

No conflict(s) detected

Installing Patch ID: KMHV..

Result: Success

$ ./bsu.sh -prod_dir=$FMW_HOME/wlserver_10.3 -status=applied -verbose -view | egrep -i 'KMHV'

Patch ID: KMHV

PatchContainer: KMHV.jar

-- if patch p13845626 was NOT applied:

$ ./bsu.sh -install -patch_download_dir=$FMW_HOME/utils/bsu/cache_dir -patchlist=WY44 -prod_dir=$FMW_HOME/wlserver_10.3

11. Apply patch 34714760 to Oracle Fusion Middleware (FMW) - Common

$ export ORACLE_HOME=$FMW_HOME/oracle_common

$ export PATH=$ORACLE_HOME/OPatch:$PATH

$ which opatch

$ cd /tmpshrs/ebsarupgradeTEMP/CPU_Patches/Oct2024/FMW_OHS_Orcl_Comn

$ ls -al

$ mv 34714760 34714760_Feb19

$ unzip p34714760_111190_Generic.zip

$ cd 34714760

$ opatch apply

... ...

Patching component oracle.jrf.opss, 11.1.1.9.0...

... ...

-- Check if 4 patches were applied

$ opatch lsinventory | egrep -i '34330735|33974106|33960746|34714760'

-- if patch 33960746 was not applied

$ cd /tmpshrs/ebsarupgradeTEMP/CPU_Patches/Oct2024/FMW_OHS_Orcl_Comn

$ unzip p33960746_111190_Generic.zip (required by Doc ID 3037725.1)

$ cd 33960746

$ opatch apply

... ...

Patching component oracle.sysman.common, 10.2.0.5.6...

Patching component oracle.sysman.oms.core, 11.1.1.9.0...

Patching component oracle.sysman.plugin.ai.main.oms, 11.1.1.9.0...

... ...

Note: if a patch was applied previously, "opatch apply" again will give below message:

$ opatch apply

Applying interim patch '33960746' to OH '$FMW_HOME/oracle_common'

Verifying environment and performing prerequisite checks...

The following patch(es) are duplicate patches with patches installed in the Oracle Home.

[ 33960746]

You have already installed same patch(es) with same UPI(s) or same version(s).

These patch(es) will be skipped.

... ...

OPatch stopped on request.

$ opatch lsinventory | egrep -i '34330735|33974106|33960746|34714760'

Patch 33960746 : applied on Thu Dec 05 12:03:32 EST 2024

33960746

Patch 34714760 : applied on Thu Dec 05 11:48:27 EST 2024

34714760

Patch 33974106 : applied on Fri Nov 24 13:31:49 EST 2023

Patch 34330735 : applied on Wed Aug 17 21:59:30 EDT 2022

12. Run ETCC again

Start a new OS session to get the correct env, and make sure no more patches are needed.

$ cd $patch_folder/ETCC

$ ./checkMTpatch.sh

==========================================

All required one-offs are confirmed as present.

Finished checking prerequisite patches for file edition: run.

13. Upgrade JDK

$ cd $patch_folder/EJCPUC

$ ls

ejcpuc.cmd ejcpuc.sh p37171025_R12_GENERIC.zip Readme.txt

$ sh ejcpuc.sh

############################################################

## Checking Apptier Java 7 for CPU 2024.10 on Platform Linux_x64 - need 1.7.0_441

############################################################

2024.10 action Your Version bitness Java Location

------------ ------ ------------ ------- ---------------

1.7.0_441 UPDATE 1.7.0_391 32-bit $ORACLE_HOME/jdk/bin/java

1.7.0_441 UPDATE 1.7.0_391 32-bit $COMMON_TOP/util/jdk32/bin/java

1.7.0_441 UPDATE 1.7.0_391 64-bit $COMMON_TOP/util/jdk64/bin/java

1.7.0_441 UPDATE 1.7.0_391 64-bit $FMW_HOME/webtier/jdk/bin/java

Follow 1530033.1 to update the JDK(s). Your application tier JDK 7 is lower than the 1.7.0_441 update released in CPU 2024.10.

$ cd $patch_folder/JDK_1_7_441

Assume 2 JDK files and my shell script for JDK upgrade exist in this folder:

$ ls

jdk-7u441-linux-i586.tar.gz

jdk-7u441-linux-x64.tar.gz

JDK_upgrade441.sh

$ ./JDK_upgrade441.sh

Verify:

$ cd $COMMON_TOP/util/

$ ls -al

$ cd $patch_folder/EJCPUC

$ sh ejcpuc.sh

###########################################################

## Checking Apptier Java 7 for CPU 2024.10 on Platform Linux_x64 - need 1.7.0_441

############################################################

2024.10 action Your Version bitness Java Location

------------ ------ ------------ ------- ---------------

1.7.0_441 OK 1.7.0_441 32-bit $ORACLE_HOME/jdk/bin/java

1.7.0_441 OK 1.7.0_441 32-bit $COMMON_TOP/util/jdk32/bin/java

1.7.0_441 OK 1.7.0_441 64-bit $COMMON_TOP/util/jdk64/bin/java

1.7.0_441 OK 1.7.0_441 64-bit $FMW_HOME/webtier/jdk/bin/java

14. Upgrade JRE

$ cd $COMMON_TOP/webapps/oacore/util/javaplugin

# Assume below file is downloaded and unzipped in folder

# $patch_folder/JRE_8_431

# p37063177_180431_WINNT.zip

$ cp $patch_folder/JRE_8_431/jre-8u431-windows-i586.exe j2se18431.exe

$ ls -altr

$ $FND_TOP/bin/txkSetPlugin.sh 18431

$ grep sun_plugin_ver $CONTEXT_FILE

<sun_plugin_ver oa_var="s_sun_plugin_ver">1.8.0_431</sun_plugin_ver>

$ grep s_forms_launch_method $CONTEXT_FILE

<config_option type="techstack" oa_var="s_forms_launch_method">jws</config_option>

15. Finalize

optional: Re-sign JAR files. It takes time: "Creating and signing every jar file can take about thirty minutes ..."

$ adadmin

option 1 => 4 => yes

$ autoconfig on all Apps nodes

Start all services

$ adop phase=fs_clone

=== script JDK_upgrade441.sh for upgrading JDK =====

$ more JDK_upgrade441.sh

# Oracle JDK 7 Update 441 Patch Patch 37063192

DT=`date +"%h_%Y"`

curr=`pwd`

echo $curr

JDKfolder=$patch_folder/Oct2024/JDK_1_7_441

# Assume two JDK files (from patch 37063192) are saved in the folder:

# jdk-7u441-linux-i586.tar.gz

# jdk-7u441-linux-x64.tar.gz

ls -al $JDKfolder

echo "Current JDK version:"

$ADJVAPRG -version

$AFJVAPRG -version

# --

echo "$COMMON_TOP/util"

cd $COMMON_TOP/util

tar -czf jdk64_BK_$DT.tar.gz jdk64 # do not use -v (to turn off output)

tar -czf jdk32_BK_$DT.tar.gz jdk32

rm -fr jdk64

rm -fr jdk32

cp -p $JDKfolder/*.tar.gz .

tar -xzf jdk-7u441-linux-i586.tar.gz

mv jdk1.7.0_441 jdk32

tar -xzf jdk-7u441-linux-x64.tar.gz

mv jdk1.7.0_441 jdk64

ls -al jdk*

pwd

sleep 5

# --

echo "$FMW_HOME/webtier"

cd $FMW_HOME/webtier

tar -czf jdk64_BK_$DT.tar.gz jdk

rm -fr jdk

cp -p $JDKfolder/jdk-7u441-linux-x64.tar.gz .

tar -xzf jdk-7u441-linux-x64.tar.gz

mv jdk1.7.0_441 jdk

ls -al jdk*

pwd

sleep 5

# --

echo "$ORACLE_HOME"

cd $ORACLE_HOME

tar -czf jdk32_BK_$DT.tar.gz jdk

rm -fr jdk

cp -p $JDKfolder/jdk-7u441-linux-i586.tar.gz .

tar -xzf jdk-7u441-linux-i586.tar.gz

mv jdk1.7.0_441 jdk

ls -al jdk*

pwd

sleep 5

echo "New JDK version:"

$ADJVAPRG -version

$AFJVAPRG -version

echo "Compiling:"

cd $ORACLE_HOME/forms/lib

make -f ins_forms.mk sharedlib install

cd $ORACLE_HOME/reports/lib

make -f ins_reports.mk install

cd $curr

echo "Done"

Saturday, September 21, 2024

Script to check if a space partition is full

We can use "df" to check if a Linux server space is full or not. Script below will send an email out when when it reaches the threshold.

#Script to check if a space partition is 80% full

threshold=80

shareLoc='/path/to/partition' # such as the one for $APPLCSF of Oracle EBS

spacesize=`df -h | grep $shareLoc | awk '{print $5}' | tr -cd '[[:digit:]]'`

spacepercent=`df -h | grep /tmpshrs/ebsarupgradeTEMP | awk '{print $5}'`

freesize=`df -h | grep /tmpshrs/ebsarupgradeTEMP | awk '{print $4}'`

if [ $spacesize -gt $threshold ]; then

echo "$shareLoc is too large" | mailx -s "Warning: $shareLoc is ${spacepercent} full" email@address.com

echo "Email sent."

else

echo "$shareLoc has $freesize free. Used $spacepercent "

Friday, August 9, 2024

Finding concurrent programs that trace is enabled

EBS can enable trace on concurrent program level. Navigation: Concurrent => Program => Define. Enter "Short Name" (concurrent_program_name below) to check the checkbox field on Enable Trace.

After trace is enabled, the job may take more resources on database server. SQL statement to find all concurrent programs that trace is enabled.

SQL> select fp.concurrent_program_name, fct.user_concurrent_program_name, fct.last_update_date, fct.last_updated_by, fu.description

from applsys.fnd_concurrent_programs fp, applsys.fnd_concurrent_programs_tl fct, fnd_user fu

where fp.concurrent_program_id = fct.concurrent_program_id

and fct.last_updated_by = fu.user_id and enable_trace <> 'N'

order by fct.last_update_date asc;

Saturday, June 15, 2024

fnd_web_sec.change_password in R12.2

fnd_web_sec.change_password still works in R12.2.10. It is recommended to use it only in some special/urgent needs because it may (or may not) ignore the restrictions by EBS Profile options 'Signon%' (see Oracle Doc ID 1350776.1 on ORA-14552).

Before change the password for troubleshooting, verify if the EBS account is disabled/inactive or not:

SQL> SELECT fu.user_name, fu.description, fu.start_date, fu.end_date,

CASE

WHEN fu.end_date IS NOT NULL and fu.end_date < SYSDATE THEN 'Inactive' ELSE 'Active' END AS account_status

FROM fnd_user fu

WHERE fu.user_name = 'EBS_userID'

-- and fu.end_date IS NOT NULL

-- and fu.end_date < SYSDATE

ORDER BY fu.user_name;

If it is inactive, run below statement to enable it if needed. And then even try to retrieve the passowrd.

SQL> exec apps.fnd_user_pkg.enableuser('EBS_userID');

If it becomes necessary, below statement by APPS will change EBS_userID password:

SQL> SELECT fnd_web_sec.change_password('EBS_userID','newPwd4U') FROM dual;

FND_WEB_SEC.CHANGE_PASSWORD('EBS_USERID','NEWPWD4U')

-------------------------------------------------------------------------------------

You can use below line to confirm a password:

SQL> select fnd_web_sec.validate_login('EBS_userID', 'newPwd4U') from dual;

FND_WEB_SEC.VALIDATE_LOGIN('EBS_USERID','NEWPWD4U')

--------------------------------------------------------------------------------

SQL> select fnd_web_sec.validate_login('EBS_userID', 'myPWD_01') from dual;

FND_WEB_SEC.VALIDATE_LOGIN('EBS_USERID','MYPWD_01')

--------------------------------------------------------------------------------

SQL> select fnd_message.get from dual;

GET

--------------------------------------------------------------------------------

PASSWORD_INVALID

One day, EBS users cannot log onto EBS site. The login webpage shows up but does not allow any users in. Since there is no error on EBS apps side, we do not know it is a security/password problem or other problems. I used below queries to show it is a database problem

SQL> show user

USER is "APPS"

SQL> select HOST_NAME, INSTANCE_NAME from v$instance;

HOST_NAME INSTANCE_NAME

------------------ -------------------------

ebsdb1q CEBSQA

SQL> select fnd_web_sec.validate_login('EBS_userID', 'XXXXxxx') from dual;

ERROR at line 1:

ORA-03113: end-of-file on communication channel

SQL> select sysdate from dual;

ERROR:

ORA-03114: not connected to ORACLE

NOTES 1: FND_WEB_SEC.validate_password( ) is aonther function.

SQL> select fnd_web_sec.validate_password('EBS_userID', 'newPwd4U') from dual;

FND_WEB_SEC.VALIDATE_PASSWORD('EBS_USERID','NEWPWD4U')

--------------------------------------------------------------------------------

SQL> select fnd_message.get from dual;

GET

--------------------------------------------------------------------------------

Must not reuse a recently used password. Please supply a different password.

NOTES 2: fnd_message.get can be used sometimes to get useful information. For example, after a Java load errored out in Sql*Plus, below line gives some details:

SQL> select fnd_message.get from dual;

GET

--------------------------------------------------------------------------------

Unable to load Java class oracle.apps.xxfnd.custom.security.PasswordValidation specified in profile option SIGNON_PASSWORD_CUSTOM. Please verify that the class exists and that it implements the Java interface oracle.apps.fnd.security.PasswordValidation.

Saturday, May 25, 2024

EBS forms failed by CrowdStrike

EBS Forms in our financial applications suddenly does not work. The message on the webpage is

Failure of Web Server bridge:

No backend server available for connection: timed out after 10 seconds or idempotent set to OFF or method not idempotent.

The error message does not tell the true problem. When checking into services on OS level, I saw Oracle EBS Forms service was not running and also saw errors from startup script $ADMIN_SCRIPTS_HOME/adstrtal.sh:

Forms service failed to start.

The Node Manager is already up.

ERROR: Unable to start up the managed server forms_server1

Server specific logs are located at $EBS_DOMAIN_HOME/servers/forms_server1/logs

05/13/24-20:56:26 :: admanagedsrvctl.sh: exiting with status 1

Java error exists in Forms log file $EBS_DOMAIN_HOME/servers/forms_server1/logs/forms_server1.out

<May 13, 2024 8:56:25 PM EDT> <Emergency> <Store> <BEA-280060> <The persistent store "_WLS_forms_server1" encountered a fatal error, and it must be shut down: weblogic.store.PersistentStoreFatalException: [Store:280020]There was an error while reading from the log file

weblogic.store.PersistentStoreFatalException: [Store:280020]There was an error while reading from the log file

at weblogic.store.io.file.FileStoreIO.open(FileStoreIO.java:128)

at weblogic.store.internal.PersistentStoreImpl.recoverStoreConnections(PersistentStoreImpl.java:435)

at weblogic.store.internal.PersistentStoreImpl.open(PersistentStoreImpl.java:423)

at weblogic.store.admin.AdminHandler.activate(AdminHandler.java:126)

at weblogic.store.admin.FileAdminHandler.activate(FileAdminHandler.java:207)

Truncated.

Caused By: java.io.EOFException: premature EOF: expected=512, actual=126

at weblogic.store.io.file.StoreFile.readBulk(StoreFile.java:316)

at weblogic.store.io.file.Heap.readStoreFile(Heap.java:1142)

at weblogic.store.io.file.Heap.getNextRecoveryFile(Heap.java:1226)

at weblogic.store.io.file.Heap.open(Heap.java:373)

at weblogic.store.io.file.FileStoreIO.open(FileStoreIO.java:117)

Truncated.

Seems WebLogic failed to open a file, but the log did not say which file. I knew that Linux Admins just did server maintenance and rebooted server after they applied monthly patches and Security updates on OS level. That was the only change in the application environment recently.

After searching around, I found the Java errors match the description in Oracle Doc ID 3017110.1 ( Managed Forms Server Fails To Start - Displaying Message: FAILED_NOT_RESTARTABLE - ERROR: <BEA-280061> The persistent store "_WLS_forms_server1" could not be deployed: weblogic.store.PersistentStoreFatalException [Store:280020] ).

The document points out the problem is caused by CrowdStrike, which locks a Forms file in $EBS_DOMAIN_HOME/servers/forms_server#/data/store/default.

CrowdStrike is installed in /opt/CrowdStrike. It is owned by root, and it is running constantly on the Linux server.

$ ps -ef | grep falcon-sensor

root 1081 1079 0 May13 ? 00:22:23 falcon-sensor

The problem can be fixed temporarily by a workaround:

1. Delete/rename below .DAT file (I guess CrowdStrike does not like the file name and so locks it)

$ cd $EBS_DOMAIN_HOME/servers/forms_server1/data/store/default

$ ls -altr

total 1028

drwxr-xr-x 4 user group 40 Sep 13 2023 ..

-rw-r--r-- 1 user group 1049088 May 13 20:51 _WLS_FORMS_SERVER1000000.DAT

drwxr-xr-x 2 user group 42 May 13 20:56 .

$ rm _WLS_FORMS_SERVER1000000.DAT

2. Re-start services cleanly by

$ADMIN_SCRIPTS_HOME/adstrtal.sh

$ADMIN_SCRIPTS_HOME/admanagedsrvctl.sh start forms_server1

The permanent fix is that the Secure team rolls back the CrowdStrike change (and applies it again until CrowdStrike fixes the problem), because its new update touches an Oracle Forms data file wrongly during its scan.