To clone a R12.2 instance is similar to add a node to an existing instance. The document for this task is Doc ID 1383621.1 (Cloning Oracle E-Business Suite Release 12.2 with Rapid Clone).
1. Clone the database in steps:
a) Clone the DB
b) Clean out node info (EXEC FND_CONC_CLONE.SETUP_CLEAN)
c) Run autocfg
d) Recompile invalids
e) Run ETCC script
... ...
Found patch records in the inventory.
Checking Mapping XML file for 12.1.0.2.210420
All the required one-off bugfixes are present in Database ORACLE_HOME.
Stored Technology Codelevel Checker results in the database IFSUAT successfully.
Finished checking fixes for Oracle Database: Fri ... ...
Notes: Per Doc ID 2008451.1, results of ETCC execution are stored in a database table (APPLSYS.TXK_TCC_RESULTS). The table may still hold obsolete data after DB clone, and needs at least one row for current db server to let ADOP work through and avoid error:
[ERROR] ETCC not run in the database node ebsdb1d
The EBS Technology Codelevel Checker needs to be run on the database node.
It is available as Patch 17537119
f) Confirm listeners are up.
For 19c database, CDB name => CEBSDEV & PDB name => EBSDEV
$ lsnrctl status CEBSDEV
2. Preparation on Source instance (usually on its primary node)
Doc ID 1383621.1 (Cloning Oracle E-Business Suite Release 12.2 with Rapid Clone) says:
Before cloning a system with Rapid Clone, be sure to allow any active online patching cycles to run all the way through the final (cleanup) phase. In case patches are applied in hotpatch or downtime mode, then you must run cleanup phase of adop.
Then run fs_clone to synchronize with the other file system, to avoid the need for synchronization to be performed in the next patching cycle.
Most of time, running FS_CLONE is good enough, if there is no pending/open patching cycle (e.g. ADOP phases were not used to apply patch in online mode). If FS_CLONE status was completed long time ago (and downtime mode was not used to apply patches since then), it is still preferable to run FS_CLONE again to synchronize PATCH file system with all configuration changes, such as oacore addition, ssl cert renewal, etc.
$ grep adminserverstatus $CONTEXT_FILE
<oa_service_status oa_var="s_adminserverstatus">enabled</oa_service_status>
$ adop -status
If an ADOP step is not completed successfully, "perl adpreclone.pl appsTier" may fail.
3. Run pre-clone script in both RUN and PATCH file systems on Source node
$ echo $FILE_EDITION
run
$ cd $ADMIN_SCRIPTS_HOME
$ perl adpreclone.pl appsTier
Enter the APPS User Password:
Enter the Weblogic AdminServer password :
... ...
Completed Stage...
$ ls -altr $COMMON_TOP/clone <== 6 folders have new timestamp
$ . $HOME/$TWO_TASK/EBSapps.env PATCH
$ echo $FILE_EDITION
patch
$ cd $ADMIN_SCRIPTS_HOME
$ sh adadminsrvctl.sh start forcepatchfs
$ perl adpreclone.pl appsTier
$ ls -altr $COMMON_TOP/clone <== 9 folders have new timestamp
$ adadminsrvctl.sh stop forcepatchfs=yes
$ adnodemgrctl.sh stop
4. Copy files from Source node to Target node
Key step: before dropping old folders on Target node, save some key config files for the refresh.
Also, write down the result "echo $TWO_TASK":
$ echo $TWO_TASK
EBSDEV
$ cd /u07/app/EBSDEV
$ mkdir fs1
$ mkdir fs2
$ cd fs1
$ nohup rsync --progress -avze ssh applmgr@master_node:/u02/app/EBSQA/fs1/EBSapps .
$ cd ../fs2
$ nohup rsync --progress -avze ssh applmgr@master_node:/u02/app/EBSQA/fs2/EBSapps .
5. Make sure database is running and connection works.
Make sure listeners ebsdev, cebsdev are up and running on database server:
$ lsnrctl status CEBSDEV
Use Oracle client on a laptop or a terminal to connect to database, and verify some info:
$ sqlplus apps/appsPWD@ebsdev
SQL> show parameter local_listener
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
local_listener string ebdb1d:1537
SQL> show parameter service
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
service_names string CEBSDEV
SQL> select * from fnd_nodes; <= one row of db node info
SQL> select * from fnd_oam_context_files; <= 3 rows on database node
SQL> select * from ADOP_VALID_NODES; <= it will get updated after clone
If they have some unexpected rows, the clone script may clean them.
Then, use tnsping to get the database host name and port number. Below setting depends on entries in tnsnames.ora file on your local client:
$ tnsping EBSDEV
Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL= TCP)(Host=ebdb1d.domain.com)(Port= 1537)) (CONNECT_DATA = (SERVICE_NAME = EBSDEV)))
OK (220 msec)
Or
$ tnsping EBSDEV
Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL= TCP)(Host=ebdb1d.domain.com)(Port= 1537)) (CONNECT_DATA = (SID = EBSDEV)))
OK (200 msec)
6. Run the clone script for both file systems (with dualfs option) on Target server
a) env
b) vi /etc/oraInst.loc
c) rm -rf oraEBSDEVInventory; mkdir oraEBSDEVInventory
d) cd EBSDEV/fs1/EBSapps/comn/clone/bin <== assume fs1 is the RUN on Source instance
UPDATES:
If Target mid-tier and Source mid-tier are on the same server, adcfgclone.pl may fail with port conflict error. To prevent that happens, stop apps services of Source instance to let the cloning complete, and then start apps services to find if any issues. See Port conflict during clone.
$ perl adcfgclone.pl appsTier dualfs
Copyright (c) 2002, 2015 Oracle Corporation
Redwood Shores, California, USA
Oracle E-Business Suite Rapid Clone
Version 12.2
adcfgclone Version 120.63.12020000.65
Enter the APPS password :
Enter the Weblogic AdminServer password :
<== Will be new WLS password for Target instance
Do you want to add a node (yes/no) [no] : no
Running: Context clone...
Log file located at $COMMON_TOP/clone/bin/CloneContext_0209154645.log
Provide the values required for creation of the new APPL_TOP Context file.
Target System Hostname (virtual or normal) [nodeName] :
Target System Database SID : EBSDEV <== same for previous $TWO_TASK
Target System Database Server Node [nodeName] : ebdb1d.domain.com
Target System Base Directory : /u07/app/EBSDEV
Target System Base Directory set to /u07/app/EBSDEV
Target System Current File System Base set to /u07/app/EBSDEV/fs1
Target System Other File System Base set to /u07/app/EBSDEV/fs2
Target System Fusion Middleware Home set to /u07/app/EBSDEVfs1/FMW_Home
Target System Other File System Fusion Middleware Home set to /u07/app/EBSDEV/fs2/FMW_Home
Target System Web Oracle Home set to /u07/app/EBSDEV/fs1/FMW_Home/webtier
Target System Other File System Web Oracle Home set to /u07/app/EBSDEV/fs2/FMW_Home/webtier
Target System Appl TOP set to /u07/app/EBSDEV/fs1/EBSapps/appl
Target System Other File System Appl TOP set to /u07/app/EBSDEV/fs2/EBSapps/appl
Target System COMMON TOP set to /u07/app/EBSDEV/fs1/EBSapps/comn
Target System Other File System COMMON TOP set to /u07/app/EBSDEV/fs2/EBSapps/comn
Target System Instance Home Directory [/u07/app/EBSDEV] :
Target System Current File System Instance Top set to /u07/app/EBSDEV/fs1/inst/apps/EBSDEV_nodeName
Do you want to preserve the Display [NodeName:0.0] (y/n) : n
Target System Display [nodeName:0.0] : nodeName.domain.com:16.0
Target System Root Service [enabled] :
Target System Web Entry Point Services [enabled] : <== assume this is a web/form node
Target System Web Application Services [enabled] :
Target System Batch Processing Services [disabled] :
Target System Other Services [disabled] :
Do you want the target system to have the same port values as the source system (y/n) [y] ? : n
Target System Port Pool [0-99] : 16
Checking the port pool 16
done: Port Pool 16 is free
Report file located at $INST_TOP/admin/out/portpool.lst
Target System proxy port [80] :
UTL_FILE_DIR on database tier consists of the following directories.
1. /network/to/EBSDEV/utl_dir <== get this path from database
2. ... ...
Choose a value which will be set as APPLPTMP value on the target node [1] : 1
The new APPL_TOP context file has been created :
$INST_TOP/appl/admin/$CONTEXT_NAME.xml
Check Clone Context logfile $COMMON_TOP/clone/bin/CloneContext_0209154645.log for details.
Creating Patch file system context file.....
Log file located at $COMMON_TOP/clone/bin/CloneContextPatch_0209155146.log
Target System Other File System Instance Top set to /u07/app/EBSDEV/fs2/inst/apps/$CONTEXT_NAME
Target System Port Pool [0-99] : 16
Patch file system should have different port pool than Run file system..
Target System Port Pool [0-99] : 17
Checking the port pool 17
done: Port Pool 17 is free
Report file located at /u07/app/EBSDEV/fs2/inst/apps/$CONTEXT_NAME/admin/out/portpool.lst
The new APPL_TOP context file has been created : ... ... (on fs2)
Check Clone Context logfile $CONMMON_TOP/clone/bin/CloneContextPatch_0209155146.log for details.
FMW Pre-requisite check log file location : $COMMON_TOP/clone/FMW/logs/prereqcheck.log
Running: FMW pre-req check...
Configuring: Run file system....
LogFile located at $INST_TOP/admin/log/clone/run/RCloneApplyAppstier_02091555.log
<Feb 9, 2022 4:17:08 PM EST> <Warning> <JNDI> <BEA-050001> <WLContext.close() was called in a different thread than the one in which it was created.> <== not sure why got them?
<Feb 9, 2022 4:22:57 PM EST> <Warning> <JNDI> <BEA-050001> <WLContext.close() was called in a different thread than the one in which it was created.>
Configuring: Patch file system....
LogFile located at $INST_TOP/admin/log/clone/patch/RCloneApplyAppstier_02091623.log
<Feb 9, 2022 4:50:51 PM EST> <Warning> <JNDI> <BEA-050001> <WLContext.close() was called in a different thread than the one in which it was created.>
Do you want to startup the Application Services for EBSDEV? (y/n) [n] : n
Services not started
NOTES: Cloning logs are in folders run and patch under $INST_TOP/admin/log/clone. But they may be removed after ADOP runs. Seems some log files are consolidated to sub-folders under $INST_TOP/admin/log/clone. All .error files under $INST_TOP/admin/log/clone/xxxT2PApply shall be empty.
7. Prepare the Target instance
a) add below line to .profile (on Linux node) and then start a new OS session
. /u02/app/EBSDEV/EBSapps.env RUN
$ echo $FILE_EDITION
$ echo $APPL_TOP
$ echo $TWO_TASK
EBSDEV
$ grep adminserverstatus $CONTEXT_FILE
<oa_service_status oa_var="s_adminserverstatus">enabled</oa_service_status>
b) Connect to database
$ sqlplus apps/appsPWD
set lines 333;
col NODE_NAME for a18;
col NODENAME for a18;
SQL> select * from fnd_nodes; <== 3 rows
SQL> select * from ADOP_VALID_NODES; <== 1 row
SQL> select * from AD_NODES_CONFIG_STATUS; <== empty
SQL> select * from fnd_oam_context_files;
SQL> select * from fnd_oam_context_files
where node_name = 'nodeName' ; <== shall have rows for RUN and PATCH
SQL> select instance_name from v$instance;
INSTANCE_NAME
----------------
CEBSDEV
In 19c database, the clone script create tnsames.ora maybe with below entries, by pulling info from database setups:
$ tnsping ebsdev
... ...
Used TNSNAMES adapter to resolve the alias
Attempting to contact (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp)(HOST=aebsw1d.aetna.com)(PORT=1537)) (CONNECT_DATA= (SERVICE_NAME=ebs_EBSDEV) (INSTANCE_NAME=CDBEDEV)))
OK (0 msec)
Do not run "EXEC FND_CONC_CLONE.SETUP_CLEAN;" again, because it will delete all data in some tables and autoconfig may not get them fully back to table fnd_oam_context_files.
8. Enable SSL TLS1.2 by using cert file in $NE_BASE/inst/$CONTEXT_NAME/certs and copy Java signing files to $NE_BASE/EBSapps/appl/ad/admin/. See Enable TLS1.2 in R12.2 Steps 1 - 4. Note This step can be skipped on the concurrent node. But if TLS 1.2 was enabled, it is necessary to get the later cert file to avoid potential FS_CLONE failure.
Edit SSL config files. Do not replace files by backed-up files from prior config, because s_ohs_instance or s_wls_domain_name (and so WLS file paths) could be different from prior one.
9. Generate .zip file for autoconfig on database node
$ cd $AD_TOP/bin
$ ls -al ad*mk*
$ perl admkappsutil.pl
Copy $INST_TOP/admin/out/appsutil.zip to database node and run AutoConfig on database node.
10. Edit $CONTEXT_FILE and restore files
applcsf, applptmp, s_forms_tmpdir, sitename, s_smtphost, s_javamailer_reply_to, s_sesstimeout, s_url_protocol, s_local_url_protocol, s_webentryurlprotocol, s_active_webport, s_https_listen_parameter, s_login_page, s_external_url, s_webentryhost, forms_catchterm
Restore files from backup:
$APPL_TOP_NE/ad/admin/adkeystore.dat
$APPL_TOP_NE/ad/custom/adop_sync.drv
Verify files/folders:
$APPL_TOP_NE/ad/admin/adsign.txt
$FND_TOP/fndenv.env
$INST_TOP/ora/10.1.2/forms/server/default.env
$AD_TOP/admin/template/custom (if exists)
$FND_TOP/admin/template/custom
Apply a workaround to make the Weblogic console accessible after started by modifying a file:
$ iPath=`tr < $CONTEXT_FILE '<>' ' ' | awk '/"s_wls_domain_name/ {print $(NF-1)}'`
$ echo $iPath
$ cd $FMW_HOME/user_projects/domains/$iPath/config
$ cp -p config.xml config.xml_BK_clone
$ diff config.xml config.xml_BK_clone
< <connection-filter-rule>0.0.0.0/0 * * allow</connection-filter-rule>
---
> <connection-filter-rule>0.0.0.0/0 * * deny</connection-filter-rule>
11. Run configuration scripts
- adadmin (option: 1==> 4 Yes)
Run it to sign Java files to avoid GUI forms error: Application Blocked by Java Security
Your security settings have blocked an application signed with an expired or not-yet-valid certificate from running.
- autoconfig on nodeName
12. Start all services
$ cd $ADMIN_SCRIPTS_HOME
$ ./adstrtal.sh apps/apps_PWD
Then, test the login and GUI forms. Also make sure Weblogic console works.
$ grep s_wls_adminport $CONTEXT_FILE
13. Change APPS password in two steps (if needed)
Notes: Doc ID 2360475.1 states APPS password is not saved in any files but stored in database.
1) Stop Apps services
2) $ FNDCPASS apps/old_appsPWS 0 Y system/'systemPWD' SYSTEM APPLSYS new_AppsPWD
3) autoconfig
4) start Admin services
./adadminsrvctl.sh start
5) Manually update the password in WLS DataSource for starting oacore, etc.
See Doc ID 1674462.1 (R12.2: Steps to Change the APPS, APPLSYS, and APPS_NE Password Using FNDCPASS or AFPASSWD)
a. Log in to WLS Administration Console.
b. Click Lock & Edit in Change Center.
c. In the Domain Structure tree, expand Services, then select Data Sources.
d. On the "Summary of JDBC Data Sources" page, select EBSDataSource.
e. On the "Settings for EBSDataSource" page, select the Connection Pool tab.
f. Enter the new password in the "Password" field.
g. Enter the new password in the "Confirm Password" field.
h. Click Save.
i. Click Activate Changes in Change Center.
6) Start all Apps services
14. Sanity check on other things, such as
$ adop -status <== Nothing with adop yet.
Enter the APPS password:
Connected.
======================================================
ADOP (C.Delta.12)
Session Id: 8
Command: status
Output: $NE_BASE/EBSapps/log/adop/8/.../adzdshowstatus.out
======================================================
File System Synchronization Type: None
adop exiting with status = 0 (Success)
15. Add additional node(s) to the new instance
Copy files from this new instance to add more nodes to the instance. In this way, all ports are kept the same in all nodes. See Add a R12.2 node.
16. After all are done, run FS_CLONE to synchronize up edited settings to PATCH file system.
Without this step, context variable s_active_webport on RUN and PATCH file systems is different if TLS1.2 was enabled in Step 8.
UPDATES on troubleshooting
A) In one case I got below error from "adop phase=cutover" and then I had to run "adop phase=abort" to abort the patching session. Seem to me that the issue came while fs_clone was not executed and "adop phase=prepare" was unable to screen out the port difference and sync it:
$NE_BASE/EBSapps/log/adop/5/.../cutover/validate/node1Name/ADOPValidations_detailed.log:
-------------------------------------------------------------------------------------------------------------------
Lines #(51-55):
Checking the value of s_active_webport...
ERROR: The value of s_active_webport are different on RUN & PATCH Context files.
The Value present in RUN Context file = 4484
The Value present in PATCH Context file = 8041
Per Doc ID 2199585.1, s_active_webport (and s_login_page) in Context File on both RUN and PATCH File Systems shall be the same.
The fix is to run FS_CLONE first and then run the ADOP cycle.
B) "perl adcfgclone.pl appsTier dualfs" failed in the half way of script running with error:
[ Config tool : $FND_TOP/patch/115/bin/txkManageDBConnectionPool.pl ]
Command $FMW_HOME/webtier/perl/bin/perl -I $FMW_HOME/webtier/perl/lib/5.10.0 ... ...
Action : execute
SCRIPT RETURNED:
.end std out.
stty: standard input: Inappropriate ioctl for device
stty: standard input: Inappropriate ioctl for device
*******FATAL ERROR*******
PROGRAM : ($FND_TOP/patch/115/bin/txkManageDBConnectionPool.pl)
TIME : Wed Mar 29 16:15:31 2022
FUNCTION: main::validateAppsSchemaCredentials [ Level 1 ]
ERRORMSG: Invalid APPS database user credentials. No updates Performed
.end err out.
The password was correct because I saw new entries were entered into database tables by the clone script. Then I tried to use "perl $AD_TOP/patch/115/bin/adProvisionEBS.pl ebs-delete-node ..." to clean it from database and got message saying primary node can not be deleted.
The fix is to run AutoConfig on database node, then just run "perl adcfgclone.pl appsTier dualfs" again.
No comments:
Post a Comment