Scripts for start & stop EBS services

When server reboots for maintenance or unexpected downtime, we want it to bring EBS services down and up automatically. Sometimes, we also want to schedule an EBS downtime by cron job. I wrote two shell scripts by calling EBS Admin scripts for automation. They also generate log files for tracing the scripts' last run.

Assume that a solid $HOME/.profile for setting up R12.2 environment variables and a file $HOME/xxx_scripts/.EBSpassenv holding key passwords exist on the server. 

$ more .EBSpassenv

export APPS_PWD=apps#@PWD

export SYSTEM_PWD=system%_PWD

export WLS_ADMIN=wls$%^PWD

root can create a short script in directory /etc/init.d (or /etc/rc.d/init.d) which will be called during server reboot to execute both auto_stopall.sh for automatic shutdown and anto_startall.sh for automatic startup.

 

============ script auto_startall.sh ============

# Start all EBS services 

DT=date +"%h %d, %y %H:%M"

RUNLOG="$HOME/xxx_scripts/reboot_scripts/reboot_start.log"

RUNLOG_ERR="$HOME/xxx_scripts/reboot_scripts/reboot_start_Error.log"

if [ -f $RUNLOG ]; then

mv $RUNLOG ${RUNLOG}_old

fi

if [ -f $RUNLOG_ERR ]; then

mv $RUNLOG_ERR ${RUNLOG_ERR}_old

fi

exec 1>$RUNLOG

exec 2>$RUNLOG_ERR

sleep 2

echo "Running at $DT"

. $HOME/.profile

. $HOME/xxx_scripts/.EBSpassenv

ps -ef | grep $LOGNAME           # check current status of EBS services

# for R12.1

# $ADMIN_SCRIPTS_HOME/adstrtal.sh apps/$APPS_PWD@$TWO_TASK

# for R12.2

if [ $isMaster == "enabled" ]; then      ## $isMaster is defined in .profile

{ echo apps ; echo $AAPS_PWD ; echo $WLS_ADMIN ; } | $ADMIN_SCRIPTS_HOME/adstrtal.sh @ -mode=allnodes -nopromptmsg

else

{ echo apps ; echo $AAPS_PWD ; echo $WLS_ADMIN ; } | $ADMIN_SCRIPTS_HOME/adstrtal.sh @ -msimode -nopromptmsg

fi

echo 'sleep 10 seconds'

sleep 10

exit 0

============= end ============

NOTES for concurrent (CM) server/node:

1. Even WLS AdminServer are not started and running on Primary node, adstrtal.sh will fully start concurrent managers on a CM node (where only s_batch_status is "enabled" in CONTEXT_FILE).

2. If CM node crashed and services were not stopped gracefully, adstrtal.sh may not be able to start concurrent managers on CM node next time. Instead,  all FNDLUBR processes may start and run on the Primary node (where even  s_batch_status is "disabled"). If that happens, you have to stop services on all nodes, and then run adstrtal.sh to start concurrent services on CM node first to correct the issue. " adcmctl.sh start " by its own will not do much in R12.2.

3. If adstpall.sh has troubles in stopping CM processes, run " adcmctl.sh stop " may help. 

========== script auto_stopall.sh =========

# Stop all EBS services. It may take 10 minutes for all apps processes shutdown.

DT=date +"%h %d, %y %H:%M"

RUNLOG="$HOME/xxx_scripts/reboot_scripts/reboot_stop.log"

RUNLOG_ERR="$HOME/xxx_scripts/reboot_scripts/reboot_stop_Error.log"

if [ -f $RUNLOG ]; then

mv $RUNLOG ${RUNLOG}_old

fi

if [ -f $RUNLOG_ERR ]; then

mv $RUNLOG_ERR ${RUNLOG_ERR}_old

fi

exec 1>$RUNLOG

exec 2>$RUNLOG_ERR

echo "Running at $DT"

. $HOME/.profile

. $HOME/xxx_scripts/.EBSpassenv

ps -ef | grep $LOGNAME

echo "shutting down ..."

# for R12.1

# $ADMIN_SCRIPTS_HOME/adstpall.sh apps/$APPS_PWD

{ echo apps ; echo $APPS_PWD ; echo $WLS_PWD ; } | $ADMIN_SCRIPTS_HOME/adstpall.sh @ -nopromptmsg

echo 'sleep 20 seconds'

sleep 20

PNUM=ps -ef | grep $LOGNAME | egrep -i 'FNDLIB|FNDSM' | wc -l 

if [ $PNUM -gt 1 ]; then

echo 'sleep 90 seconds more...'

sleep 90

fi

# only check upper case and assume $TWO_TASK is in the $ORACLE_HOME path

PNUM=ps -ef | grep -w $LOGNAME | egrep 'FNDLIB|FNDSM|'$TWO_TASK | wc -l 

if [ $PNUM -gt 1 ]; then

echo 'sleep 30 seconds'

sleep 30

fi

PNUM=ps -ef | grep -w $LOGNAME | egrep 'FNDLIB|FNDSM|'$TWO_TASK | wc -l 

if [ $PNUM -gt 1 ]; then

echo 'sleep 30 seconds more ...'

sleep 30

fi

PNUM=ps -ef | grep -w $LOGNAME | egrep 'FNDLIB|FNDSM|'$TWO_TASK | wc -l 

if [ $PNUM -gt 1 ]; then

echo 'sleep 15 seconds more ...'

sleep 15

fi

ps -ef | grep $LOGNAME

exit 0

============= end ============

Use .profile in Linux to customize the shell prompt

When you have many EBS instances in a multi-nodes environment, it will be very useful to let the Linux prompt display current user ID, server name and the path location. A custom .profile saved under $HOME works for me very well. Its colors tell if you are in a Admin node or not, and if you are in a production environment or not (assume the last character of production server's name is "p").

For a Linux account, environment variable $HOME is defined by file /etc/passwd. But, if the account was created by AD (Active Directory), the default value of $HOME is defined in "Home Directory" section of AD and the defined value of $HOME could be exported in file ~applMgr/.profile.

Our EBS applMgr accounts use Korn shell which uses two startup files under $HOME, the .profile and the .kshrc. During a session start, .profile is first read once, then .kshrc (if it exists) is read by each new ksh. e.g. :

$ echo $SHELL

/bin/ksh

$ echo $0

-ksh

$ which ksh

/usr/bin/ksh

$ more .kshrc

alias ftp="print 'Reminder: Use sftp instead of \\\ftp'"

echo "This is .kshrc"

$ ksh

This is .kshrc

$ ftp

Reminder: Use sftp instead of \ftp

============= $HOME/.profile =============

PATH=/bin:/usr/bin:/usr/local/bin

export PATH

MANPATH=/usr/share/man:/usr/local/share/man

export MANPATH      # for man manual 

EDITOR=/bin/vi

export EDITOR

# ENV=$HOME/.kshrc

# export ENV

. /u02/app/EBSPROD/EBSapps.env RUN     # R12.2 env file

. /u02/app/xxx_scripts/.EBSpassenv              # password file (custom)

isMaster="no"

if [ ! -z $APPS_VERSION ] && [ ${APPS_VERSION:0:4} == "12.2" ]

then

s_status=cat $CONTEXT_FILE | grep -i s_adminserverstatus

isMaster="${s_status:60:7}"

fi

if [ $isMaster == "enabled" ]   # on admin/primary node

then

if [ echo -n ${HOSTNAME%%.*} | tail -c -1 != "p" ]   

             # last character of server name is not "p" => non-production server

then

PS1=$'

\e[0;31m$USER@${HOSTNAME%%.}[$TWO_TASK]\e[m$PWD

-->$ '  

else       # on production server: Red, and Green color on PWD

PS1=$'

\e[0;31m$USER@${HOSTNAME%%.}[$TWO_TASK]\e[m\E[32m$PWD \E[0m

-->$ '

fi

else                                          # on other node(s)

if [ echo -n ${HOSTNAME%%.*} | tail -c -1 != "p" ]   

            # on non-production server

then

PS1='

$USER@${HOSTNAME%%.}[$TWO_TASK]$PWD

-->$ '

else      # on production server

PS1=$'

$USER@${HOSTNAME%%.}[$TWO_TASK]\E[32m$PWD \E[0m

-->$ '

fi

fi

alias rm='rm -i'

stty erase ^?

umask u=rwx,g=rwx,o=rx

================ end =================

On an Admin node in production env, the prompt looks like this:

applMgr@server_1p[EBSPROD]/u02/app

-->$

applMgr@server_1p[EBSPROD]/u02/app

-->$ echo $USER

applMgr

applMgr@server_1p[EBSPROD]/u02/app

-->$ echo $TWO_TASK

EBSPROD

applMgr@server_1p[EBSPROD]/u02/app

-->$ cd $TWO_TASK

applMgr@server_1p[EBSPROD]/u02/app/EBSPROD

-->$ echo $HOME

/u02/app

applMgr@server_1p[EBSPROD]/u02/app/EBSPROD

-->$ ls

EBSapps.env   fs1   fs2   fs_ne