Refresh from EBSPROD (under /u06/app/EBSPROD) to EBSQA (under /u01/app/EBSQA) on a different host in Linux OS.
1. On source nodes, run PERL script.
$ cd $ADMIN_SCRIPTS_HOME
$ perl adpreclone.pl appsTier <-- No password is needed
NOTE: Pre-clone log file located at $INST_TOP/admin/log/StageAppsTier_MMDDHHMM.log
$ cd $COMMON_TOP/clone <-- to verify new folders are created
$ ls -altr $COMMON_TOP/clone
total 40
drwxr-xr-x 2 applmgr appsuser 4096 Jan 22 2008 html
drwxr-xr-x 3 applmgr appsuser 4096 Jan 22 2008 context
drwxr-xr-x 2 applmgr appsuser 4096 Jan 22 2008 bin
drwxr-xr-x 9 applmgr appsuser 4096 Sep 15 2013 ..
drwxr-xr-x 5 applmgr appsuser 4096 Dec 1 10:06 appsts
drwx------ 7 applmgr appsuser 4096 Dec 1 10:07 jre
drwxr-xr-x 3 applmgr appsuser 4096 Dec 1 10:07 oui
drwxr-xr-x 4 applmgr appsuser 4096 Dec 1 10:07 jlib
drwxr-xr-x 5 applmgr appsuser 4096 Dec 1 10:07 appl
drwxr-xr-x 10 applmgr appsuser 4096 Dec 1 10:07 .
Do the same on other source hosts.
2. On target nodes, backup filess
Create a folder, such as /u01/app/admin_scripts/backups, to hold backed-up files. Then run the script backup_4refresh.sh to back up <SID_HOST> specific files.
$ cd admin_scripts/backups
$ ./backup_4refresh.sh
Confirm necessary files are just backed up:
$ ls -al /u01/app/admin_scripts/backups
$ ls -al /u01/app/admin_scripts/backups/admin
$ ls -al /u01/app/admin_scripts/backups/Apache
Optional: Backup the entire folder to aother partition
$ cd /u01/app
$ tar -zhcvf /u02/app/$CONTEXT_NAME_backup_MMDDYY.tar.gz EBSQA >> backup_MMDD.log
Do the same on other hosts.
3. Make sure all apps services were stopped and database refresh was completed.
Now, remove directories
$ cd /u01/app
$ rm -rf EBSQA
$ rm -rf oraEBSQAinventory; mkdir oraEBSQAinventory
4. Copy directories from source node.
NOTE: copy CM host files to CM host, web/forms host to web/forms host.
$ cd /u01/app
$ rsync --progress -avze ssh applmgr@source_host:/u06/app/EBSPROD /u01/app
enter password
sent 12619428 bytes received 9649138852 bytes 1312649.72 bytes/sec
total size is 33018009660 speedup is 3.42
It may give warning message:
rsync warning: some files vanished before they could be transferred (code 24) at main.c(1532) [generator=3.0.6]
5. Rename the folder and verify the size
$ cd /u01/app
$ mv EBSPROD EBSQA
$ ls -al EBSQAtotal 16
drwxr-xr-x 4 applmgr appsuser 4096 May 7 2013 .
drwxr-xr-x 16 applmgr appsuser 4096 Apr 15 11:27 ..
drwxr-xr-x 4 applmgr appsuser 4096 May 2 2013 apps
drwxr-xr-x 3 applmgr appsuser 4096 May 2 2013 inst
$cd apps
$ du -ks
31640672 .
$ cd ../inst
$ du -ks
2502420 .
6. Edit /etc/oraInst.loc file to make sure below line is used:
inventory_loc=/u01/app/oraEBSQAInventory
7. Start a new OS session (without running the .env file)
If you do not modify the .profile, re-login will get message:
$APPL_TOP/${CONTEXT_NAME}.env: cannot open [No such file or directory]
Note: by this time, the .env file was deleted when old folder EBSQA was removed in Step 5.
So env vairables do not exist
8. Go to /u01/app/EBSQA/apps/apps_st/comn/clone/bin ($COMMON_TOP/clone/bin)
option: delete old files in /u01/app/EBSQA/apps/apps_st/comn/clone
$ cd /u01/app/EBS/apps/apps_st/comn/clone
$ rm -rf jre_BAK_*
Now, get below answers ready -
Database Server Node => dbhost1q.domain.com
database SID => EBSQA
apps => appsPWD
Base Directory => /u01/app/EBSQA
PORT => 1542
port pool: 21 (1542 -1521)
9. Run clone perl script and answer questions
$ perl adcfgclone.pl appsTier <<-- run this in CM node first!!
++++++++++++++++ answers on CM node ++++++++++++++++
Target System Root Service [disabled] : <-- for CM node. WEB/Form node will be opposite
Target System Web Entry Point Services [disabled] :
Target System Web Application Services [disabled] :
Target System Batch Processing Services [enabled] :
Target System Other Services [enabled] : <-- for CM node
Do you want to preserve the Display [y] (y/n) : n
Target System Display [host1q:0.0] : host1q.domain.com:21.0 <-- OAM will use it for display
Do you want the the target system to have the same port values as the source system (y/n) [y] ? : n
Target System Port Pool [0-99] : 21
++++++++++++++++ answers on Form/Web node ++++++++++++++++
Target System Root Service [enabled] : <-- for Form node
Target System Web Entry Point Services [enabled] :
Target System Web Application Services [enabled] :
Target System Batch Processing Services [enabled] : disabled <-- for Form node
Target System Other Services [enabled] : disabled
Do you want to preserve the Display [xifsapm3q:0.0] (y/n) : n
Target System Display [host2q:0.0] : host2q.domain.com:21.0 <-- OAM will use it for display
Do you want the the target system to have the same port values as the source system (y/n) [y] ? : n
Target System Port Pool [0-99] : 21
The script may remain silent for a few minutes (or even 40 minutes) on executing $INST_TOP/admin/install/txkWfClone.sql !
(only statement SELECT * from v$session where upper(machine) like 'HOSTNAME%'; can tell if the sql script is running).
Answer "no" - Dot not start apps services after autoconfig completed.
Check the log file. Any more lines than below need your attention or fix:
$ egrep -i 'fail|error|ora-' $INST_TOP/admin/log/ApplyAppsTier_04151536.log
Error while running adlnktools.sh.
ERRORCODE = 0 ERRORCODE_END
ERRORCODE = 0 ERRORCODE_END
AC-50480: Internal error occurred: java.lang.Exception: Error while generating listener.ora.
Error generating tnsnames.ora from the database, temporary tnsnames.ora will be generated using templates
ERRORCODE = 2 ERRORCODE_END
Result : FAILED
ERRORCODE = 0 ERRORCODE_END
Unique constraint error (00001) is OK if key already exists
ERRORCODE = 0 ERRORCODE_END
ERRORCODE = 0 ERRORCODE_END
. . . . . . . . .
10. Start a new OS session. If necessary, modify the .profile to run .env
$ . ./.profile
11. Remove obsolete files
$ cd $APPLPTMP <-- make sure $APPLPTMP points to the right folder!
$ rm -f *.tmp
Note: seems the clone script cleans the folder $APPLTMP automatically.
12. Generate autoconfig file for DBA (only do this on ONE node)
$ cd $AD_TOP/bin
$ ls -al ad*mk*
$ perl admkappsutil.pl
Starting the generation of appsutil.zip
Log file located at $INST_TOP/admin/log/MakeAppsUtil_04151607.log
output located at $INST_TOP/admin/out/appsutil.zip
MakeAppsUtil completed successfully.
$ cp -p $INST_TOP/admin/out/appsutil.zip $APPLPTMP
13. Verify apps password before email DBA with the password. And clean CM tables
$ sqlplus apps/passwd
SQL> @cmclean.sql
SQL> EXEC FND_CONC_CLONE.SETUP_CLEAN;
SQL> select * from fnd_nodes;
(This SELECT shell get 0 row returned. After DBA runs adconfig.sh below, it shall have one row.)
Notes: cmclean.sql is an old file but it still works for me in my R12.1.3 instances
REM $Id: cmclean.sql,v 1.4 2001/04/07 15:55:07 pferguso Exp $
14. Ask DBA to run ADCONFIG
$ adconfig.sh contextfile=$ORACLE_HOME/db_ebsqa/appsutil/$CONTEXT_NAME.xml
NOTE: if .xml does not exist on a new instance, generate it first. here is how to generate the .xml file after unzip appsutil.zip on database server :
$ export PERL5LIB=$ORACLE_HOME/appsutil/perl
$ export LD_LIBRARY_PATH=$ORACLE_HOME/lib:$OARCLE_HOME/ctx/lib
$ perl $ORACLE_HOME/appsutil/bin/adbldxml.pl
<enter apps password when prompted>
<enter hostname when prompted>
<enter listener port when prompted>
<enter SID name when prompted>
Output should indicate:
The context file has been created at: $ORACLE_HOME/db_ebsdev/appsutil/EBSDEV_ebsdb1d.xml
(or $ perl adbldxml.pl tier=db appsuser=<APPSuser> )
15. Run script to restore files from backup
$ cd admin_scripts/backups
$ ./resotore_refresh.sh
15. Verify files and make sure two files for site custom
$FND_TOP/fndenv.env
$INST_TOP/ora/10.1.2/forms/server/default.env
Note: two files shall have a section in below format (Otherwise it will be overwritten by autoconfig). Or, modify them after adautocfg.
#Begin Customizations
custom tops
#End Customizations
$ ls -al $INST_TOP/certs/Apache
cwallet.sso ewallet.p12
17. Update $CONTEXT_FILE
- APPLCSF & APPLPTMP on ALL nodes !
- Below ones are ONLY on Form/Web nodes:
webport
activewebport
login
external_url
webentryhost
webentryurlprotocol
Notes: For real port (not as 443 in F5) when SSL is enabled, make sure the port (such as 4472) for s_active_webport match the port in file ssl.conf and s_webssl_port in $CONTEXT_FILE .
18. Run AD autocfg : always make CM node work first (after DBA is done with adconfig!)
$ cd $ADMIN_SCRIPTS_HOME
$ ./adautocfg.sh
Optional: change APPS password using FNDCPASS here, and then run adautocfg.sh again on all nodes.
19. Source .profile to take new env variables after autocfg (or re-login)
$ . ./.profile
20. Start services
21. Login as XML Publisher Admin --> Administration --> General --> replace the path with $APPLPTMP.
22. Update profile "Site%name%"
Check profile option: BNE Debug Log Directory, BNE Upload Staging Directory (then, bounce services)
RRA: Service Prefix (this should be BLANK. Not hard coded !!)
RRA: Enabled (someones say this must be set to YES. Seems not necessary)
Viewer: Text (this should be blank or browser)
23. Modify files for customization. Maybe only on the web node. Such as:
- TEMPLATE FILE : $FND_TOP/admin/template/httpd_conf_1013.tmp
add a section to it
- TEMPLATE FILE : $FND_TOP/admin/template/oracle_apache_conf_1013.tmp
#include "%s_ora_config_home%/10.1.3/Apache/modplsql/conf/plsql.conf"
include "%s_weboh_oh%/Apache/modplsql/conf/plsql.conf" <<-- add this line
- Add APPS password to $IAS_ORACLE_HOME/Apache/modplsql/conf/dads.conf
24. Re-create database links, check DBA_DIRECTORIES (DBA task)
25. Re-link some folders for $CUSTOM_TOP:
$ cd $CUSTOM_TOP/ftp
$ rm in
$ rm out
$ ln -s /path/to/ftp/in in
$ ln -s /path/to/ftp/out out
$ chmod 777 in
$ chmod 777 out
26. Optional: disable production users
-- Doc. ID: 1064798.1. Disable EBS users
DECLARE
-- get user list which should be disabled
cc number :=0;
cursor c1 is
select user_name
from fnd_user
where (end_date is null or end_date > sysdate)
-- access all users that has no end-date or end-date is future
and (user_name like 'B%' or user_name like 'M%')
and user_name not in ('SYSADMIN', 'USERVIP1', 'USERVIP2', 'A999999'); -- excluding VIPs
BEGIN
for c in c1 loop
-- disable user
fnd_user_pkg.disableuser(c.user_name);
cc := cc+1;
If ( MOD(cc, 50) = 0) then
commit;
end if;
end loop;
dbms_output.put_line('Totla disabled ' || cc);
commit; --commit changes
END;
/
++++++++ Troubleshooting +++++++
1. After CM node is up in the first time, run a CM job got error when opening the log
An error occurred while attempting to establish an Applications File Server connection with the
node FNDFS_CMHOST. There may be a network configuration problem, or the TNS listener
on node FNDFS_CMHOST may not be running. Please contact your system administrator.
you can confirm the problem by running below command on Web Host:
$ tnsping FNDFS_CMHOST
TNS-12547: TNS:lost contact
after host names were deleted by "EXEC FND_CONC_CLONE.SETUP_CLEAN" from database tables and when autoconfig was executed first on Concurrent Host CMhost, web host names are not saved to database tables yet and so tnsnames.ora file on CMhost does not have entries for web hosts.
Fix: two ways to fix it
1). on CM host
(a). Replace tnsnames.ora and sqlnet.ora in $TNS_ADMIN with files from the backup before refresh.
(b). adalnctl.sh stop
(c). adalnctl.sh start
Now, on the Web Host server, "tnsping FNDFS_CMHOST" returns "OK"
2). Run autoconfig AGAIN on all hosts, starting with CM host first.
2. When stopping CM services (to run autoconfig again for the new APPS password), 5 processes on
OS level did not stop. See all five of them in the Concurrent Manager:
CONCURRENT MANAGER Actual Target
Output Post Processor 2 0
Workflow Agent Listener Service 1 0
Workflow Mailer Service 1 0
Workflow Document Web Services Service 1 0
I clicked "Terminate" on them, but that did not kill the OS processes. I had to kill them by OS command:
$ kill -12 15524 15529 15544 15548 15555
After I ran autoconfig and started CM services, they showed "Terminted" in GUI Concurrent Manager.
I had to clicked each of them to "Activate" them. They did get started after the clicks !!
Should not try to terminate them in the GUI !!
3. When I ran the autoconfig last on CM node, the login page did not work!!
So, run autoconfig on web/forms node last.
4. After clone script adcfgclone.pl worked, Autoconfig ended with below error in adconfig.log & ApplyAppsTier_11011132.log:
[CVM Error Report]
The following report lists errors encountered during CVM Phase
<filename> <return code where appropriate>
/u03/app/EBSUAT/apps/apps_st/appl/ad/12.0.0/bin/adgentns.pl 2
Updating s_tnsmode to 'generateTNS'
UpdateContext exited with status: 0
AC-50480: Internal error occurred: java.lang.Exception: Error while generating listener.ora.
Error generating tnsnames.ora from the database, temporary tnsnames.ora will be generated using templates
Instantiating Tools tnsnames.ora
Tools tnsnames.ora instantiated
Web tnsnames.ora instantiated
adgentns.pl exiting with status 2
Fix: run below line AGAIN
SQL> EXEC FND_CONC_CLONE.SETUP_CLEAN;
Then, run adconfig.sh on DB host. After that, make sure the DB host name shows up in below query before run adautocfg.sh on Apps tiers:
SQL> select * from fnd_nodes;
~~~~~~~~~~~~~~~~~~ script: backup_4refresh.sh ~~~~~~~~~~~~~~~~~~
#!/bin/ksh
# backup files before the refresh Dec. 2015
cd $HOME/admin_scripts/backups
DT=`date +"%h_%d_%y"` # backup date
echo It will delete old backup files. Continue ? yes or no
read answer
case $answer in
[yY][eE][sS]|[yY])
if [ -n "$CONTEXT_FILE" ]; then
echo 'running'
else
echo 'environment variable $CONTEXT_FILE does not exit. Exit ...'
exit 1;
fi;
if [ -f $CONTEXT_FILE ]; then
echo 'starting copy ...'
else
echo 'CONTEXT file $CONTEXT_FILE does not exist. Exit ...'
exit 1;
fi
file1=`basename $CONTEXT_FILE`
if [ -f $file1 ]; then
mv $file1 $file1'_'$DT # backup date
fi
file2=custom`basename $CONTEXT_FILE .xml`'.env'
if [ -f $APPL_TOP/$file2 ]; then
cp -p $APPL_TOP/$file2 .
fi
rm -rf Apache admin
cp -p $CONTEXT_FILE .
cp -rp $INST_TOP/certs/Apache .
cp -rp $TNS_ADMIN .
cp -p $FND_TOP/fndenv.env .
cp -p $INST_TOP/ora/*2/forms/server/default.env .
cp -p $APPL_TOP/admin/adkeystore.dat .
cp -p $APPL_TOP/admin/adsign.txt .
## more optional files. Need to manually copy them back if needed
cp -p $FND_TOP/admin/template/custom/rwbuilder_conf_1012.tmp .
cp -p $FND_TOP/admin/template/ssl_conf_1013.tmp .
cp -p $FND_TOP/admin/template/httpd_conf_1013.tmp .
cp -p $FND_TOP/admin/template/oracle_apache_conf_1013.tmp .
cp -p $IAS_ORACLE_HOME/Apache/modplsql/conf/dads.conf .
echo 'Done with backing up files'
ls -altr $HOME/admin_scripts/backups
exit 0
;;
[nN][oO]|[nN])
echo "No"
;;
*)
echo "Invalid input..."
exit 1
;;
esac
~~~~~~~~~~~~~~~~~~ scripts: restore_refresh.sh ~~~~~~~~~~~~~~~~~~~
#!/bin/ksh
# restore files AFTER clone script during the refresh Dec. 2015
cd $HOME/admin_scripts/backups
DT=`date +"%h_%d_%y"`
echo It will overwrite some files. Continue? yes or no
read answer
case $answer in
[yY][eE][sS]|[yY])
if [ -n "$CONTEXT_FILE" ]; then
echo 'running'
else
echo 'environment variable $CONTEXT_FILE does not exit. Exit ...'
exit 1;
fi;
file2=custom`basename $CONTEXT_FILE .xml`'.env'
if [ -f $file2 ]; then
cp -p $file2 $APPL_TOP/.
fi
mv $FND_TOP/fndenv.env $FND_TOP/fndenv.env_clone
cp -p fndenv.env $FND_TOP/.
mv $INST_TOP/ora/10.1.2/forms/server/default.env $INST_TOP/ora/10.1.2/forms/server/default.env_clone
cp -p default.env $INST_TOP/ora/10.1.2/forms/server/.
mv $INST_TOP/certs/Apache $INST_TOP/certs/Apache'_'$DT
cp -pr Apache $INST_TOP/certs/.
## files for JRE 1.8. They will be used if Java signing is re-ran.
mv $APPL_TOP/admin/adkeystore.dat $APPL_TOP/admin/adkeystore.dat_clone
cp -p adkeystore.dat $APPL_TOP/admin/.
mv $APPL_TOP/admin/adsign.txt $APPL_TOP/admin/adsign.txt_clone
cp -p adsign.txt $APPL_TOP/admin/.
echo 'Done with restoring files.'
;;
[nN][oO]|[nN])
echo "No"
;;
*)
echo "Invalid input..."
exit 1
;;
esac
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The first task of EBS refresh is to refresh the database. Here are the steps:
0. Stop EBS apps
1. In the source database EBSPROD (in 10G and 11G)
1.1 Identify the SCN nmuber for the RMAN script. Check source database backup log to find the sequence.
1.2 SQL> select first_change#, next_change#, first_time from v$log_history
where sequence# = 11047;
2. In the target database EBSQA
2.1 Shutdown the target database
SQL> alter database backup controlfile to trace;
SQL> shutdown immediate;
2.2 Drop the databae
SQL> startup mount restrict;
SQL> drop database;
2.3 Get ready for the RMAN restore
SQL> startup nomount;
2.4 Make sure host has enough disk space
2.5 Assume the source database was backed up by RMAN. Run the script
$ cat db_refresh.sh
ORACLE_SID=targetDB_ID
export ORACLE_SID
export TNS_ADMIN=$ORACLE_HOME/network/admin
ORACLE_USER=oracle
NLS_LANG=AMERICAN_AMERICA.AL32UTF8
NLS_DATE_FORMAT='YYYY-MM-DD hh24:mi:ss'
export NLS_LANG
export NLS_DATE_FORMAT
NB_ORA_CLASS=sourceDBserver
export NB_ORA_CLASS
DATE=`date +"%h%d%y_%H%M"`
rman catalog rman/rmanPWD@rmanDB_ID target sys/PSWD@sourceDB_ID auxiliary / msglog db_refresh.${DATE}.log << EOF
run {
ALLOCATE AUXILIARY CHANNEL ch00 TYPE 'SBT_TAPE';
ALLOCATE AUXILIARY CHANNEL ch01 TYPE 'SBT_TAPE';
ALLOCATE AUXILIARY CHANNEL ch02 TYPE 'SBT_TAPE';
SEND 'NB_ORA_CLIENT=sourceDBserver';
SET NEWNAME FOR TEMPFILE 1 to '/path/to/temp01.dbf';
SET NEWNAME FOR TEMPFILE 2 to '/path/to/temp02.dbf';
duplicate target database to targetDB_ID
UNTIL scn = 5968220384735;
}
EOF
$ ./db_refresh.sh &
3. Bring the database up.
Do NOT change any password (except SYSTEM).
No comments:
Post a Comment