Saturday, April 20, 2024

Scripts for start & stop EBS services

When server reboots for maintenance or unexpected downtime, we want it to bring EBS services down and up automatically. Sometimes, we also want to schedule an EBS downtime by cron job. I wrote two shell scripts by calling EBS Admin scripts for automation. They also generate log files for tracing the scripts' last run.

Assume that a solid $HOME/.profile for setting up R12.2 environment variables and a file $HOME/xxx_scripts/.EBSpassenv holding key passwords exist on the server. 
$ more .EBSpassenv
export APPS_PWD=apps#@PWD
export SYSTEM_PWD=system%_PWD
export WLS_ADMIN=wls$%^PWD

root can create a short script in directory /etc/init.d (or /etc/rc.d/init.d) which will be called during server reboot to execute both auto_stopall.sh for automatic shutdown and anto_startall.sh for automatic startup.
 
============ script auto_startall.sh ============
# Start all EBS services 
DT=date +"%h %d, %y %H:%M"
RUNLOG="$HOME/xxx_scripts/reboot_scripts/reboot_start.log"
RUNLOG_ERR="$HOME/xxx_scripts/reboot_scripts/reboot_start_Error.log"
if [ -f $RUNLOG ]; then
mv $RUNLOG ${RUNLOG}_old
fi
if [ -f $RUNLOG_ERR ]; then
mv $RUNLOG_ERR ${RUNLOG_ERR}_old
fi
exec 1>$RUNLOG
exec 2>$RUNLOG_ERR
sleep 2
echo "Running at $DT"
. $HOME/.profile
. $HOME/xxx_scripts/.EBSpassenv
ps -ef | grep $LOGNAME           # check current status of EBS services

# for R12.1
# $ADMIN_SCRIPTS_HOME/adstrtal.sh apps/$APPS_PWD@$TWO_TASK

# for R12.2
if [ $isMaster == "enabled" ]; then      ## $isMaster is defined in .profile
{ echo apps ; echo $AAPS_PWD ; echo $WLS_ADMIN ; } | $ADMIN_SCRIPTS_HOME/adstrtal.sh @ -mode=allnodes -nopromptmsg
else
{ echo apps ; echo $AAPS_PWD ; echo $WLS_ADMIN ; } | $ADMIN_SCRIPTS_HOME/adstrtal.sh @ -msimode -nopromptmsg
fi
echo 'sleep 10 seconds'
sleep 10
exit 0
============= end ============
NOTES for concurrent (CM) server/node:
1. Even WLS AdminServer are not started and running on Primary node, adstrtal.sh will fully start concurrent managers on a CM node (where only s_batch_status is "enabled" in CONTEXT_FILE).
2. If CM node crashed and services were not stopped gracefully, adstrtal.sh may not be able to start concurrent managers on CM node next time. Instead,  all FNDLUBR processes may start and run on the Primary node (where even  s_batch_status is "disabled"). If that happens, you have to stop services on all nodes, and then run adstrtal.sh to start concurrent services on CM node first to correct the issue. " adcmctl.sh start " by its own will not do much in R12.2.
3. If adstpall.sh has troubles in stopping CM processes, run " adcmctl.sh stop " may help. 

========== script auto_stopall.sh =========
# Stop all EBS services. It may take 10 minutes for all apps processes shutdown.
DT=date +"%h %d, %y %H:%M"
RUNLOG="$HOME/xxx_scripts/reboot_scripts/reboot_stop.log"
RUNLOG_ERR="$HOME/xxx_scripts/reboot_scripts/reboot_stop_Error.log"
if [ -f $RUNLOG ]; then
mv $RUNLOG ${RUNLOG}_old
fi
if [ -f $RUNLOG_ERR ]; then
mv $RUNLOG_ERR ${RUNLOG_ERR}_old
fi
exec 1>$RUNLOG
exec 2>$RUNLOG_ERR
echo "Running at $DT"
. $HOME/.profile
. $HOME/xxx_scripts/.EBSpassenv
ps -ef | grep $LOGNAME
echo "shutting down ..."
# for R12.1
# $ADMIN_SCRIPTS_HOME/adstpall.sh apps/$APPS_PWD
{ echo apps ; echo $APPS_PWD ; echo $WLS_PWD ; } | $ADMIN_SCRIPTS_HOME/adstpall.sh @ -nopromptmsg
echo 'sleep 20 seconds'
sleep 20
PNUM=ps -ef | grep $LOGNAME | egrep -i 'FNDLIB|FNDSM' | wc -l 
if [ $PNUM -gt 1 ]; then
echo 'sleep 90 seconds more...'
sleep 90
fi
# only check upper case and assume $TWO_TASK is in the $ORACLE_HOME path
PNUM=ps -ef | grep -w $LOGNAME | egrep 'FNDLIB|FNDSM|'$TWO_TASK | wc -l 
if [ $PNUM -gt 1 ]; then
echo 'sleep 30 seconds'
sleep 30
fi
PNUM=ps -ef | grep -w $LOGNAME | egrep 'FNDLIB|FNDSM|'$TWO_TASK | wc -l 
if [ $PNUM -gt 1 ]; then
echo 'sleep 30 seconds more ...'
sleep 30
fi
PNUM=ps -ef | grep -w $LOGNAME | egrep 'FNDLIB|FNDSM|'$TWO_TASK | wc -l 
if [ $PNUM -gt 1 ]; then
echo 'sleep 15 seconds more ...'
sleep 15
fi
ps -ef | grep $LOGNAME
exit 0
============= end ============

Saturday, April 6, 2024

Use .profile in Linux to customize the shell prompt

When you have many EBS instances in a multi-nodes environment, it will be very useful to let the Linux prompt display current user ID, server name and the path location. A custom .profile saved under $HOME works for me very well. Its colors tell if you are in a Admin node or not, and if you are in a production environment or not (assume the last character of production server's name is "p").

For a Linux account, environment variable $HOME is defined by file /etc/passwd. But, if the account was created by AD (Active Directory), the default value of $HOME is defined in "Home Directory" section of AD and the defined value of $HOME could be exported in file ~applMgr/.profile.

Our EBS applMgr accounts use Korn shell which uses two startup files under $HOME, the .profile and the .kshrc. During a session start, .profile is first read once, then .kshrc (if it exists) is read by each new ksh. e.g. :

$ echo $SHELL
/bin/ksh
$ echo $0
-ksh
$ which ksh
/usr/bin/ksh
$ more .kshrc
alias ftp="print 'Reminder: Use sftp instead of \\\ftp'"
echo "This is .kshrc"
$ ksh
This is .kshrc
$ ftp
Reminder: Use sftp instead of \ftp

============= $HOME/.profile =============
PATH=/bin:/usr/bin:/usr/local/bin
export PATH
MANPATH=/usr/share/man:/usr/local/share/man
export MANPATH      # for man manual 
EDITOR=/bin/vi
export EDITOR
# ENV=$HOME/.kshrc
# export ENV
. /u02/app/EBSPROD/EBSapps.env RUN     # R12.2 env file
. /u02/app/xxx_scripts/.EBSpassenv              # password file (custom)
isMaster="no"
if [ ! -z $APPS_VERSION ] && [ ${APPS_VERSION:0:4} == "12.2" ]
then
s_status=cat $CONTEXT_FILE | grep -i s_adminserverstatus
isMaster="${s_status:60:7}"
fi
if [ $isMaster == "enabled" ]   # on admin/primary node
then
if [ echo -n ${HOSTNAME%%.*} | tail -c -1 != "p" ]   
             # last character of server name is not "p" => non-production server
then
PS1=$'
\e[0;31m$USER@${HOSTNAME%%.}[$TWO_TASK]\e[m$PWD
-->$ '  
else       # on production server: Red, and Green color on PWD
PS1=$'
\e[0;31m$USER@${HOSTNAME%%.}[$TWO_TASK]\e[m\E[32m$PWD \E[0m
-->$ '
fi
else                                          # on other node(s)
if [ echo -n ${HOSTNAME%%.*} | tail -c -1 != "p" ]   
            # on non-production server
then
PS1='
$USER@${HOSTNAME%%.}[$TWO_TASK]$PWD
-->$ '
else      # on production server
PS1=$'
$USER@${HOSTNAME%%.}[$TWO_TASK]\E[32m$PWD \E[0m
-->$ '
fi
fi

alias rm='rm -i'
stty erase ^?
umask u=rwx,g=rwx,o=rx
================ end =================

On an Admin node in production env, the prompt looks like this:
applMgr@server_1p[EBSPROD]/u02/app
-->$

applMgr@server_1p[EBSPROD]/u02/app
-->$ echo $USER
applMgr
applMgr@server_1p[EBSPROD]/u02/app
-->$ echo $TWO_TASK
EBSPROD
applMgr@server_1p[EBSPROD]/u02/app
-->$ cd $TWO_TASK
applMgr@server_1p[EBSPROD]/u02/app/EBSPROD
-->$ echo $HOME
/u02/app
applMgr@server_1p[EBSPROD]/u02/app/EBSPROD
-->$ ls
EBSapps.env   fs1   fs2   fs_ne