Finding concurrent programs that trace is enabled

EBS can enable trace on concurrent program level. Navigation: Concurrent => Program => Define. Enter "Short Name" (concurrent_program_name below) to check the checkbox field on Enable Trace.

After trace is enabled, the job may take more resources on database server. SQL statement to find all concurrent programs that trace is enabled. 

SQL> select fp.concurrent_program_name, fct.user_concurrent_program_name, fct.last_update_date, fct.last_updated_by, fu.description

 from applsys.fnd_concurrent_programs fp, applsys.fnd_concurrent_programs_tl fct, fnd_user fu

 where fp.concurrent_program_id = fct.concurrent_program_id

     and fct.last_updated_by = fu.user_id and enable_trace <> 'N'

  order by fct.last_update_date asc;

fnd_web_sec.change_password in R12.2

fnd_web_sec.change_password still works in R12.2.10. It is recommended to use it only in some special/urgent needs because it may (or may not) ignore the restrictions by EBS Profile options 'Signon%' (see Oracle Doc ID 1350776.1 on ORA-14552).

Before change the password for troubleshooting, verify if the EBS account is disabled/inactive or not:

SQL> SELECT fu.user_name, fu.description, fu.start_date, fu.end_date,

CASE

WHEN fu.end_date IS NOT NULL and fu.end_date < SYSDATE THEN 'Inactive' ELSE 'Active' END AS account_status

FROM fnd_user fu

WHERE fu.user_name = 'EBS_userID'

-- and fu.end_date IS NOT NULL

-- and fu.end_date < SYSDATE

ORDER BY fu.user_name;

If it is inactive, run below statement to enable it if needed. And then even try to retrieve the passowrd.

SQL> exec apps.fnd_user_pkg.enableuser('EBS_userID');

If it becomes necessary, below statement by APPS will change EBS_userID password:

SQL> SELECT fnd_web_sec.change_password('EBS_userID','newPwd4U') FROM dual;

FND_WEB_SEC.CHANGE_PASSWORD('EBS_USERID','NEWPWD4U')

-------------------------------------------------------------------------------------

Y

You can use below line to confirm a password:

SQL> select fnd_web_sec.validate_login('EBS_userID', 'newPwd4U') from dual;

FND_WEB_SEC.VALIDATE_LOGIN('EBS_USERID','NEWPWD4U')

--------------------------------------------------------------------------------

Y

SQL> select fnd_web_sec.validate_login('EBS_userID', 'myPWD_01') from dual;

FND_WEB_SEC.VALIDATE_LOGIN('EBS_USERID','MYPWD_01')

--------------------------------------------------------------------------------

N

SQL> select fnd_message.get from dual;

GET

--------------------------------------------------------------------------------

PASSWORD_INVALID

One day, EBS users cannot log onto EBS site. The login webpage shows up but does not allow any users in. Since there is no error on EBS apps side, we do not know it is a security/password problem or other problems.  I used below queries to show it is a database problem 

SQL> show user

USER is "APPS"

SQL> select HOST_NAME, INSTANCE_NAME from v$instance;

HOST_NAME   INSTANCE_NAME

------------------  -------------------------

ebsdb1q             CEBSQA

SQL> select fnd_web_sec.validate_login('EBS_userID', 'XXXXxxx') from dual;   

ERROR at line 1:

ORA-03113: end-of-file on communication channel

SQL> select sysdate from dual;

ERROR:

ORA-03114: not connected to ORACLE

NOTES 1: FND_WEB_SEC.validate_password( ) is aonther function.

SQL> select fnd_web_sec.validate_password('EBS_userID', 'newPwd4U') from dual;

FND_WEB_SEC.VALIDATE_PASSWORD('EBS_USERID','NEWPWD4U')

--------------------------------------------------------------------------------

N

SQL> select fnd_message.get from dual;

GET

--------------------------------------------------------------------------------

Must not reuse a recently used password. Please supply a different password.

NOTES 2: fnd_message.get can be used sometimes to get useful information. For example, after a Java load errored out in Sql*Plus, below line gives some details:

SQL> select fnd_message.get from dual;

GET

--------------------------------------------------------------------------------

Unable to load Java class oracle.apps.xxfnd.custom.security.PasswordValidation specified in profile option SIGNON_PASSWORD_CUSTOM.  Please verify that the class exists and that it implements the Java interface oracle.apps.fnd.security.PasswordValidation.

EBS forms failed by CrowdStrike

EBS Forms in our financial applications suddenly does not work. The message on the webpage is

Failure of Web Server bridge:

No backend server available for connection: timed out after 10 seconds or idempotent set to OFF or method not idempotent. 

 

The error message does not tell the true problem. When checking into services on OS level, I saw Oracle EBS Forms service was not running and also saw errors from startup script $ADMIN_SCRIPTS_HOME/adstrtal.sh:

Forms service failed to start. 

The Node Manager is already up.

ERROR: Unable to start up the managed server forms_server1

Server specific logs are located at $EBS_DOMAIN_HOME/servers/forms_server1/logs

05/13/24-20:56:26 :: admanagedsrvctl.sh: exiting with status 1

Java error exists in Forms log file $EBS_DOMAIN_HOME/servers/forms_server1/logs/forms_server1.out

<May 13, 2024 8:56:25 PM EDT> <Emergency> <Store> <BEA-280060> <The persistent store "_WLS_forms_server1" encountered a fatal error, and it must be shut down: weblogic.store.PersistentStoreFatalException: [Store:280020]There was an error while reading from the log file

weblogic.store.PersistentStoreFatalException: [Store:280020]There was an error while reading from the log file

        at weblogic.store.io.file.FileStoreIO.open(FileStoreIO.java:128)

        at weblogic.store.internal.PersistentStoreImpl.recoverStoreConnections(PersistentStoreImpl.java:435)

        at weblogic.store.internal.PersistentStoreImpl.open(PersistentStoreImpl.java:423)

        at weblogic.store.admin.AdminHandler.activate(AdminHandler.java:126)

        at weblogic.store.admin.FileAdminHandler.activate(FileAdminHandler.java:207)

        Truncated.

Caused By: java.io.EOFException: premature EOF: expected=512, actual=126

        at weblogic.store.io.file.StoreFile.readBulk(StoreFile.java:316)

        at weblogic.store.io.file.Heap.readStoreFile(Heap.java:1142)

        at weblogic.store.io.file.Heap.getNextRecoveryFile(Heap.java:1226)

        at weblogic.store.io.file.Heap.open(Heap.java:373)

        at weblogic.store.io.file.FileStoreIO.open(FileStoreIO.java:117)

        Truncated.

Seems WebLogic failed to open a file, but the log did not say which file. I knew that Linux Admins just did server maintenance and rebooted server after they applied monthly patches and Security updates on OS level. That was the only change in the application environment recently.

After searching around, I found the Java errors match the description in Oracle Doc ID 3017110.1 ( Managed Forms Server Fails To Start - Displaying Message: FAILED_NOT_RESTARTABLE - ERROR: <BEA-280061> The persistent store "_WLS_forms_server1" could not be deployed: weblogic.store.PersistentStoreFatalException [Store:280020] ). 

The document points out the problem is caused by CrowdStrike, which locks a Forms file in $EBS_DOMAIN_HOME/servers/forms_server#/data/store/default.

CrowdStrike is installed in /opt/CrowdStrike. It is owned by root, and it is running constantly on the Linux server.

$ ps -ef | grep falcon-sensor

root      1081  1079  0 May13 ?        00:22:23 falcon-sensor

The problem can be fixed temporarily by a workaround:

1. Delete/rename below .DAT file (I guess CrowdStrike does not like the file name and so locks it)

$ cd $EBS_DOMAIN_HOME/servers/forms_server1/data/store/default

$ ls -altr

total 1028

drwxr-xr-x 4 user group      40 Sep 13  2023 ..

-rw-r--r-- 1   user group  1049088 May 13 20:51 _WLS_FORMS_SERVER1000000.DAT

drwxr-xr-x 2 user group     42 May 13 20:56 .

$ rm _WLS_FORMS_SERVER1000000.DAT

2. Re-start services cleanly by

$ADMIN_SCRIPTS_HOME/adstrtal.sh

Or

$ADMIN_SCRIPTS_HOME/admanagedsrvctl.sh start forms_server1

The permanent fix is that the Secure team rolls back the CrowdStrike change (and applies it again until CrowdStrike fixes the problem), because its new update touches an Oracle Forms data file wrongly during its scan. 

Scripts for start & stop EBS services

When server reboots for maintenance or unexpected downtime, we want it to bring EBS services down and up automatically. Sometimes, we also want to schedule an EBS downtime by cron job. I wrote two shell scripts by calling EBS Admin scripts for automation. They also generate log files for tracing the scripts' last run.

Assume that a solid $HOME/.profile for setting up R12.2 environment variables and a file $HOME/xxx_scripts/.EBSpassenv holding key passwords exist on the server. 

$ more .EBSpassenv

export APPS_PWD=apps#@PWD

export SYSTEM_PWD=system%_PWD

export WLS_ADMIN=wls$%^PWD

root can create a short script in directory /etc/init.d (or /etc/rc.d/init.d) which will be called during server reboot to execute both auto_stopall.sh for automatic shutdown and anto_startall.sh for automatic startup.

 

============ script auto_startall.sh ============

# Start all EBS services 

DT=date +"%h %d, %y %H:%M"

RUNLOG="$HOME/xxx_scripts/reboot_scripts/reboot_start.log"

RUNLOG_ERR="$HOME/xxx_scripts/reboot_scripts/reboot_start_Error.log"

if [ -f $RUNLOG ]; then

mv $RUNLOG ${RUNLOG}_old

fi

if [ -f $RUNLOG_ERR ]; then

mv $RUNLOG_ERR ${RUNLOG_ERR}_old

fi

exec 1>$RUNLOG

exec 2>$RUNLOG_ERR

sleep 2

echo "Running at $DT"

. $HOME/.profile

. $HOME/xxx_scripts/.EBSpassenv

ps -ef | grep $LOGNAME           # check current status of EBS services

# for R12.1

# $ADMIN_SCRIPTS_HOME/adstrtal.sh apps/$APPS_PWD@$TWO_TASK

# for R12.2

if [ $isMaster == "enabled" ]; then      ## $isMaster is defined in .profile

{ echo apps ; echo $AAPS_PWD ; echo $WLS_ADMIN ; } | $ADMIN_SCRIPTS_HOME/adstrtal.sh @ -mode=allnodes -nopromptmsg

else

{ echo apps ; echo $AAPS_PWD ; echo $WLS_ADMIN ; } | $ADMIN_SCRIPTS_HOME/adstrtal.sh @ -msimode -nopromptmsg

fi

echo 'sleep 10 seconds'

sleep 10

exit 0

============= end ============

NOTES for concurrent (CM) server/node:

1. Even WLS AdminServer are not started and running on Primary node, adstrtal.sh will fully start concurrent managers on a CM node (where only s_batch_status is "enabled" in CONTEXT_FILE).

2. If CM node crashed and services were not stopped gracefully, adstrtal.sh may not be able to start concurrent managers on CM node next time. Instead,  all FNDLUBR processes may start and run on the Primary node (where even  s_batch_status is "disabled"). If that happens, you have to stop services on all nodes, and then run adstrtal.sh to start concurrent services on CM node first to correct the issue. " adcmctl.sh start " by its own will not do much in R12.2.

3. If adstpall.sh has troubles in stopping CM processes, run " adcmctl.sh stop " may help. 

========== script auto_stopall.sh =========

# Stop all EBS services. It may take 10 minutes for all apps processes shutdown.

DT=date +"%h %d, %y %H:%M"

RUNLOG="$HOME/xxx_scripts/reboot_scripts/reboot_stop.log"

RUNLOG_ERR="$HOME/xxx_scripts/reboot_scripts/reboot_stop_Error.log"

if [ -f $RUNLOG ]; then

mv $RUNLOG ${RUNLOG}_old

fi

if [ -f $RUNLOG_ERR ]; then

mv $RUNLOG_ERR ${RUNLOG_ERR}_old

fi

exec 1>$RUNLOG

exec 2>$RUNLOG_ERR

echo "Running at $DT"

. $HOME/.profile

. $HOME/xxx_scripts/.EBSpassenv

ps -ef | grep $LOGNAME

echo "shutting down ..."

# for R12.1

# $ADMIN_SCRIPTS_HOME/adstpall.sh apps/$APPS_PWD

{ echo apps ; echo $APPS_PWD ; echo $WLS_PWD ; } | $ADMIN_SCRIPTS_HOME/adstpall.sh @ -nopromptmsg

echo 'sleep 20 seconds'

sleep 20

PNUM=ps -ef | grep $LOGNAME | egrep -i 'FNDLIB|FNDSM' | wc -l 

if [ $PNUM -gt 1 ]; then

echo 'sleep 90 seconds more...'

sleep 90

fi

# only check upper case and assume $TWO_TASK is in the $ORACLE_HOME path

PNUM=ps -ef | grep -w $LOGNAME | egrep 'FNDLIB|FNDSM|'$TWO_TASK | wc -l 

if [ $PNUM -gt 1 ]; then

echo 'sleep 30 seconds'

sleep 30

fi

PNUM=ps -ef | grep -w $LOGNAME | egrep 'FNDLIB|FNDSM|'$TWO_TASK | wc -l 

if [ $PNUM -gt 1 ]; then

echo 'sleep 30 seconds more ...'

sleep 30

fi

PNUM=ps -ef | grep -w $LOGNAME | egrep 'FNDLIB|FNDSM|'$TWO_TASK | wc -l 

if [ $PNUM -gt 1 ]; then

echo 'sleep 15 seconds more ...'

sleep 15

fi

ps -ef | grep $LOGNAME

exit 0

============= end ============

Use .profile in Linux to customize the shell prompt

When you have many EBS instances in a multi-nodes environment, it will be very useful to let the Linux prompt display current user ID, server name and the path location. A custom .profile saved under $HOME works for me very well. Its colors tell if you are in a Admin node or not, and if you are in a production environment or not (assume the last character of production server's name is "p").

For a Linux account, environment variable $HOME is defined by file /etc/passwd. But, if the account was created by AD (Active Directory), the default value of $HOME is defined in "Home Directory" section of AD and the defined value of $HOME could be exported in file ~applMgr/.profile.

Our EBS applMgr accounts use Korn shell which uses two startup files under $HOME, the .profile and the .kshrc. During a session start, .profile is first read once, then .kshrc (if it exists) is read by each new ksh. e.g. :

$ echo $SHELL

/bin/ksh

$ echo $0

-ksh

$ which ksh

/usr/bin/ksh

$ more .kshrc

alias ftp="print 'Reminder: Use sftp instead of \\\ftp'"

echo "This is .kshrc"

$ ksh

This is .kshrc

$ ftp

Reminder: Use sftp instead of \ftp

============= $HOME/.profile =============

PATH=/bin:/usr/bin:/usr/local/bin

export PATH

MANPATH=/usr/share/man:/usr/local/share/man

export MANPATH      # for man manual 

EDITOR=/bin/vi

export EDITOR

# ENV=$HOME/.kshrc

# export ENV

. /u02/app/EBSPROD/EBSapps.env RUN     # R12.2 env file

. /u02/app/xxx_scripts/.EBSpassenv              # password file (custom)

isMaster="no"

if [ ! -z $APPS_VERSION ] && [ ${APPS_VERSION:0:4} == "12.2" ]

then

s_status=cat $CONTEXT_FILE | grep -i s_adminserverstatus

isMaster="${s_status:60:7}"

fi

if [ $isMaster == "enabled" ]   # on admin/primary node

then

if [ echo -n ${HOSTNAME%%.*} | tail -c -1 != "p" ]   

             # last character of server name is not "p" => non-production server

then

PS1=$'

\e[0;31m$USER@${HOSTNAME%%.}[$TWO_TASK]\e[m$PWD

-->$ '  

else       # on production server: Red, and Green color on PWD

PS1=$'

\e[0;31m$USER@${HOSTNAME%%.}[$TWO_TASK]\e[m\E[32m$PWD \E[0m

-->$ '

fi

else                                          # on other node(s)

if [ echo -n ${HOSTNAME%%.*} | tail -c -1 != "p" ]   

            # on non-production server

then

PS1='

$USER@${HOSTNAME%%.}[$TWO_TASK]$PWD

-->$ '

else      # on production server

PS1=$'

$USER@${HOSTNAME%%.}[$TWO_TASK]\E[32m$PWD \E[0m

-->$ '

fi

fi

alias rm='rm -i'

stty erase ^?

umask u=rwx,g=rwx,o=rx

================ end =================

On an Admin node in production env, the prompt looks like this:

applMgr@server_1p[EBSPROD]/u02/app

-->$

applMgr@server_1p[EBSPROD]/u02/app

-->$ echo $USER

applMgr

applMgr@server_1p[EBSPROD]/u02/app

-->$ echo $TWO_TASK

EBSPROD

applMgr@server_1p[EBSPROD]/u02/app

-->$ cd $TWO_TASK

applMgr@server_1p[EBSPROD]/u02/app/EBSPROD

-->$ echo $HOME

/u02/app

applMgr@server_1p[EBSPROD]/u02/app/EBSPROD

-->$ ls

EBSapps.env   fs1   fs2   fs_ne