Key steps for adding a node to R12.2 instance are in Oracle Doc ID 1383621.1 (Cloning Oracle E-Business Suite Release 12.2 with Rapid Clone, Section 5.3). From the Oracle document, seems the Source system has to be the primary node of the same existing system.
I used below steps to add node(s) in R12.2.10 as part of R12.2 upgrade (from R12.1.3). To save time, node may not be added to the system until post-upgrade steps, such as enabling TLS1.2, Enabling JWS, business & hot-fix patches, CPU patches and 3rd party tool installation, have been all completed.
1. Prepare the Source node
Make sure ADOP is in good status on Source/Master node:
$ adop -status
Enter the APPS password:
Connected.
====================================================
ADOP (C.Delta.12)
Session Id: 5
Command: status
Output: $NE_BASE/EBSapps/log/adop/5/.../adzdshowstatus.out
====================================================
Node Name Node Type Phase Status Started Finished Elapsed
--------------- ------------- --------------- ------------------- -------------------------- -------------------- ------------
node_name master FS_CLONE COMPLETED 2021/0X/14 13:15:26 2021/0X/14 14:53:11 1:37:45
Or, ADOP patch cycle never started:
$ adop -status
Enter the APPS password:
Connected.
================================================
ADOP (C.Delta.12)
Session Id: 3
Command: status
Output: $NE_BASE/EBSapps/log/adop/3/.../adzdshowstatus.out
================================================
File System Synchronization Type: Full
adop exiting with status = 0 (Success)
a) Run pre-clone script on RUN file system
First, check WLS Admin Server shall be up and be accessible from browser or a remote location
$ echo $FILE_EDITION
run
$ echo $APPL_TOP <= write down /fs2/ or /fs1/
$ cd $ADMIN_SCRIPTS_HOME
$ ./adadminsrvctl.sh status
You are running adadminsrvctl.sh version 120.10.12020000.11
Enter the WebLogic Admin password:
Enter the APPS Schema password:
The AdminServer is running
$ grep s_wls_adminport $CONTEXT_FILE
<wls_adminport oa_var="s_wls_adminport" oa_type="PORT" base="7001" step="1" range="-1" label="WLS Admin Server Port">7033</wls_adminport>
<wls_adminport oa_var="s_wls_adminport" oa_type="PORT" base="7001" step="1" range="-1" label="WLS Admin Server Port">7033</wls_adminport>
$ wget http://node_name.domain.com:7033/console
… … etc
HTTP request sent, awaiting response... 200 OK
Length: 3162 (3.1K) [text/html]
Saving to: ‘console’
100%[======================>] 3,162 --.-K/s in 0s
Length: 3162 (3.1K) [text/html]
Saving to: ‘console’
100%[======================>] 3,162 --.-K/s in 0s
2021-0X-19 11:06:29 (343 MB/s) - ‘console’ saved [3162/3162]
$ perl adpreclone.pl appsTier
Copyright (c) 2011, 2014 Oracle Corporation
Redwood Shores, California, USA
Redwood Shores, California, USA
Oracle E-Business Suite Rapid Clone
Version 12.2
adpreclone Version 120.31.12020000.25
Enter the APPS User Password:
Enter the Weblogic AdminServer password :
Enter the Weblogic AdminServer password :
Checking the status of the Oracle WebLogic Administration Server....
Running perl $AD_TOP/patch/115/bin/adProvisionEBS.pl ebs-get-serverstatus -contextfile=$CONTEXT_FILE -servername=AdminServer -promptmsg=hide
The Oracle WebLogic Administration Server is up.
wlsDomainName: EBS_domain_${TWO_TASK}
WLS Domain Name is VALID.
WLS Domain Name is VALID.
Running:
perl $AD_TOP/bin/adclone.pl java=$COMMON_TOP/util/jdk64 mode=stage stage=$COMMON_TOP/clone component=appsTier method= appctx=$CONTEXT_FILE showProgress
perl $AD_TOP/bin/adclone.pl java=$COMMON_TOP/util/jdk64 mode=stage stage=$COMMON_TOP/clone component=appsTier method= appctx=$CONTEXT_FILE showProgress
Setting the wls environment
Beginning application tier Stage - Sun Feb 28 19:12:36 2021
$COMMON_TOP/util/jdk64/bin/java -Xmx600M -Doracle.jdbc.autoCommitSpecCompliant=false -DCONTEXT_VALIDATED=false -Doracle.installer.oui_loc=/oui -classpath $FMW_HOME/webtier/lib/xmlparserv2.jar:$FMW_HOME/webtier/jdbc/lib/ojdbc6.jar:$COMMON_TOP/java/classes:$FMW_HOME/webtier/oui/jlib/OraInstaller.jar:$FMW_HOME/webtier/oui/jlib/ewt3.jar:$FMW_HOME/webtier/oui/jlib/share.jar:$FMW_HOME/webtier/../Oracle_EBS-app1/oui/jlib/srvm.jar:$FMW_HOME/webtier/jlib/ojmisc.jar:$FMW_HOME/wlserver_10.3/server/lib/weblogic.jar:$FMW_HOME/oracle_common/jlib/obfuscatepassword.jar oracle.apps.ad.clone.StageAppsTier -e $CONTEXT_FILE -stage $COMMON_TOP/clone -tmp /tmp -method CUSTOM -showProgress -nopromptmsg
Log file located at $INST_TOP/admin/log/clone/StageAppsTier_02281912.log
| 20% completed
Completed Stage...
Mon Xxx 29 19:19:28 2021
Mon Xxx 29 19:19:28 2021
$ ls -altr $COMMON_TOP/clone
<= show 6 folders are newly created.
b) Run adpreclone.pl on PATCH file system as well. Start Admin Server first.
$ . $HOME/$TWO_TASK/EBSapps.env PATCH
$ echo $TWO_TASK
EBSDEV_patch
$ sqlplus /nolog
SQL> conn apps/appsPWD@EBSDEV_patch
ERROR:
ORA-00604: error occurred at recursive SQL level 1
ORA-20099: E-Business Suite Patch Edition does not exist.
ORA-06512: at line 29
SQL> conn apps/appsPWD@EBSDEV
Connected.
SQL> select count(*) from fnd_nodes;
COUNT(*)
----------
3
(Return info for database node and Master node)
$ echo $FILE_EDITION
patch
patch
$ cd $FMW_HOME/user_projects/domains/EBS_domain/config
$ grep deny config.xml
if it has "deny", you shall replace it with "allow" before starting Admin Server.
$ cd $ADMIN_SCRIPTS_HOME
Start WebLogic Server Administration Console on the patch file system
$ sh adadminsrvctl.sh start forcepatchfs
You are running adadminsrvctl.sh version 120.10.12020000.11
Enter the WebLogic Admin password:
Enter the APPS Schema password:
Starting WLS Admin Server...
Enter the APPS Schema password:
Starting WLS Admin Server...
… … etc
adadminsrvctl.sh: exiting with status 0
adadminsrvctl.sh: check the logfile $INST_TOP/logs/appl/admin/log/adadminsrvctl.txt for more information ...
$ grep s_wls_adminport $CONTEXT_FILE
<wls_adminport oa_var="s_wls_adminport" oa_type="PORT" base="7001" step="1" range="-1" label="WLS Admin Server Port">7032</wls_adminport>
$ wget http://node_name.domain.com:7032/console
… … etc
HTTP request sent, awaiting response... 200 OK
Length: 3162 (3.1K) [text/html]
Saving to: ‘console’
Saving to: ‘console’
100%[===================>] 3,162 --.-K/s in 0s
2021-0X-19 11:24:00 (354 MB/s) - ‘console’ saved [3162/3162]
$ perl adpreclone.pl appsTier
Copyright (c) 2011, 2014 Oracle Corporation
Redwood Shores, California, USA
Redwood Shores, California, USA
Oracle E-Business Suite Rapid Clone
Version 12.2
adpreclone Version 120.31.12020000.25
Enter the APPS User Password:
Enter the Weblogic AdminServer password :
Enter the Weblogic AdminServer password :
Checking the status of the Oracle WebLogic Administration Server....
… … etc
| 20% completed
Completed Stage...
Mon Xxx 29 20:40:00 2021
Mon Xxx 29 20:40:00 2021
$ ls -altr $COMMON_TOP/clone <== 9 folders have new timestamp.
c) Write down database info and other info
$ tnsping ebsdev
Database Server Node= ebsdb1d.domain.com
database SID = EBSDEV
PORT= 1562 (1562 - 1521 = 41)
apps apps_PWD
weblogic wls_PWD
Base Directory= /u02/app/EBSDEV
2. On TARGET node (2nd apps node. See Section 5.3.2):
a) Make sure clean env and new inventory folder
$ env <= to make sure NO any Oracle environment variables show up
$ vi /etc/oraInst.loc
make sure line "inventory_loc=/u02/app/oraInventory" is in the file
$ hostname <= make sure it is the right node!
$ cd /u02/app
$ rm -rf oraInventory; mkdir oraInventory
b) Create folders and copy files from Source node. Folder copay may take 1 to 2 hours.
$ cd /u02/app/EBSDEV
$ mkdir fs1
$ mkdir fs2
$ cd fs1
$ nohup rsync --progress -avze ssh applmgr@master_node:/u02/app/EBSDEV/fs1/EBSapps .
$ cd ../fs2
$ nohup rsync --progress -avze ssh applmgr@master_node:/u02/app/EBSDEV/fs2/EBSapps .
Notes: only copy EBSapps folders to avoid error:
ERROR: Failed to configure the target system
See Doc ID 1900583.1: RCloneApplyAppstier - Exiting cloning as FMW Home already exists
c) Make sure below two commands works on Target node, or Admin Server webpages can be accessed from browsers.
$ wget http://master_node.domain.com:7033/console
$ wget http://master_node.domain.com:7032/console
If you hit this error, you have to fix it on both file systems on Source node:
$ wget http://node_name.domain.com:7033/console
--2021-04-19 13:11:44-- http://node_name.domain.com:7033/console
wget: /u02/app/.netrc:4: unknown token "GROUP/APPLMGR"
Resolving node_name.domain.com (node_name.domain.com)... 367.69.67.89
Connecting to node_name.domain.com (node_name.domain.com)|367.69.67.89|:7033... connected.
HTTP request sent, awaiting response... 403 Forbidden
2021-04-19 13:11:44 ERROR 403: Forbidden.
d) Run the clone script
Assume this is to add a node only for concurrent jobs and database listens to port 1562
$ cd /u02/app/EBSDEV/fs1/EBSapps/comn/clone/bin
<= make sure $RUN_BASE on Source node is also on fs1 !
$ perl adcfgclone.pl appsTier dualfs
Copyright (c) 2002, 2015 Oracle Corporation <= it takes 30 to 60 minutes
Redwood Shores, California, USA
Redwood Shores, California, USA
Oracle E-Business Suite Rapid Clone
Version 12.2
adcfgclone Version 120.63.12020000.65
Enter the APPS password :
Enter the Weblogic AdminServer password :
Do you want to add a node (yes/no) [no] : yes
Verifying: Run file system AdminServer is running
Verifying: Patch file system AdminServer is running
Running: Context clone...
Log file located at $COMMON_TOP/clone/bin/CloneContext_0301100358.log
Target System Hostname (virtual or normal) [node_name] :
Target System Base Directory set to /u02/app/EBSDEV
Target System Current File System Base set to /u02/app/EBSDEV/fs1
Target System Other File System Base set to /u02/app/EBSDEV/fs2
Target System Fusion Middleware Home set to /u02/app/EBSDEV/fs1/FMW_Home
Target System Other File System Fusion Middleware Home set to /u02/app/EBSDEV/fs2/FMW_Home
Target System Other File System Fusion Middleware Home set to /u02/app/EBSDEV/fs2/FMW_Home
Target System Web Oracle Home set to /u02/app/EBSDEV/fs1/FMW_Home/webtier
Target System Other File System Web Oracle Home set to /u02/app/ESBDEV/fs2/FMW_Home/webtier
Target System Other File System Web Oracle Home set to /u02/app/ESBDEV/fs2/FMW_Home/webtier
Target System Appl TOP set to /u02/app/EBSDEV/fs1/EBSapps/appl
Target System Other File System Appl TOP set to /u02/app/EBSDEV/fs2/EBSapps/appl
Target System Other File System Appl TOP set to /u02/app/EBSDEV/fs2/EBSapps/appl
Target System COMMON TOP set to /u02/app/EBSDEV/fs1/EBSapps/comn
Target System Other File System COMMON TOP set to /u02/app/EBSDEV/fs2/EBSapps/comn
Target System Other File System COMMON TOP set to /u02/app/EBSDEV/fs2/EBSapps/comn
Target System Instance Home Directory [/u02/app/EBSDEV] :
Target System Current File System Instance Top set to /u02/app/EBSDEV/fs1/inst/apps/$CONTEXT_NAME
Do you want to preserve the Display [master_node:0.0] (y/n) : n
Target System Display [xifsapm3d:0.0] : node_name.domian.com:41.0
Target System Root Service [enabled] : disabled
Target System Web Entry Point Services [enabled] : disabled
Target System Web Application Services [enabled] : disabled
Target System Batch Processing Services [enabled] :
Target System Other Services [disabled] : disabled
Do you want the target system to have the same port values as the source system (y/n) [y] ? : y
Validating if the source port numbers are available on the target system..
Complete port information available at /u02/app/EBSDEV/fs1/EBSapps/comn/clone/bin/out/$CONTEXT_NAME/portpool.lst
Validating if the source port numbers are available on the target system..
Complete port information available at /u02/app/EBSDEV/fs1/EBSapps/comn/clone/bin/out/$CONTEXT_NAME/portpool.lst
Target System proxy port [80] :
UTL_FILE_DIR on database tier consists of the following directories.
1. /path/to/utl_dir
Choose a value which will be set as APPLPTMP value on the target node [1] :
Choose a value which will be set as APPLPTMP value on the target node [1] :
The new APPL_TOP context file has been created :
/u02/app/EBSDEV/fs1/inst/apps/$CONTEXT_NAME/appl/admin/$CONTEXT_NAME.xml
Check Clone Context logfile $COMMON_TOP/clone/bin/CloneContext_0301100358.log for details.Creating Patch file system context file.....
Log file located at $COMMON_TOP/clone/bin/CloneContextPatch_0301100504.log
Target System Other File System Instance Top set to /u02/app/EBSDEV/fs2/inst/apps/$CONTEXT_NAME
Validating if the source port numbers are available on the target system..
Complete port information available at $COMMON_TOP/clone/bin/out/$CONTEXT_NAME/portpool.lst
Validating if the source port numbers are available on the target system..
Complete port information available at $COMMON_TOP/clone/bin/out/$CONTEXT_NAME/portpool.lst
Target System proxy port [80] :
The new APPL_TOP context file has been created :
/u02/app/EBSDEV/fs2/inst/apps/$CONTEXT_NAME/appl/admin/$CONTEXT_NAME.xml
Check Clone Context logfile $COMMON_TOP/clone/bin/CloneContextPatch_0301100504.log for details.FMW Pre-requisite check log file location : $COMMON_TOP/clone/FMW/logs/prereqcheck.log
Running: FMW pre-req check...
Running: FMW pre-req check...
Configuring: Run file system....
LogFile located at $INST_TOP/admin/log/clone/run/RCloneApplyAppstier_03011005.log
LogFile located at $INST_TOP/admin/log/clone/run/RCloneApplyAppstier_03011005.log
Configuring: Patch file system....
LogFile located at $INST_TOP/admin/log/clone/patch/RCloneApplyAppstier_03011020.log
LogFile located at $INST_TOP/admin/log/clone/patch/RCloneApplyAppstier_03011020.log
Do you want to startup the Application Services for EBSDEV? (y/n) [n] : n
Services not started
Notes: if it is to add a web/forms node, use below choices:
Target System Root Service [enabled] : enabled
Target System Web Entry Point Services [enabled] : enabled
Target System Web Application Services [enabled] : enabled
Target System Batch Processing Services [enabled] : disabled
Target System Other Services [disabled] : disabled
Now, verify database and diskspace:
By now, file EBSapps.env is created in /u02/app/EBSDEV.
$ . /u02/app/EBSDEV/EBSapps.env RUN (or, add it to .profile on Linux)
$ echo $APPL_TOP
/u02/app/EBSDEV/fs1/EBSapps/appl
SQL> select * from fnd_nodes;
SQL> select * from ADOP_VALID_NODES;
SQL> select * from fnd_oam_context_files;
$ cd /u02/app/EBSDEV
$ du -sh fs*
41G fs1 <= depending on how many patches were applied on master node
42G fs2
80K fs_ne
e) Edit $CONTEXT_FILE on Target node
$ vi $CONTEXT_FILE
R12.2 generates context file from source node and does not need to edit much. Verify below variables:
s_local_url_protocol, s_webentryurlprotocol, s_active_webport, s_https_listen_parameter, s_login_page, s_external_url, s_webentryhost
Edit variables:
applcsf, applptmp, s_forms_tmpdir, sitename, s_smtphost, s_javamailer_reply_to, s_sesstimeout
s_forms_catchterm, s_concstatus => disabled (to prevent concurrent manager from start on web node? Doc ID 417437.1. But seems this does nothing.)
f) On the newly added node, TLS 1.2 is not enabled by clone script. It is necessary to enable it manually. See Enable TLS1.2 in R12.2.
Also, need to copy two files from old R12.1 to $NE_BASE/EBSapps/appl/ad/admin/ for Java signing:
adkeystore.dat
adsign.txt
g) Stop apps services on Source/Master node. Do not do this yet if you have more nodes to add.
In the RUN session, stop all services first. NOTE: if R12.2 services are running, do not edit $CONTEXT_FILE before that because adstpall.sh reads it to decide what will be stopped
$ cd $ADMIN_SCRIPTS_HOME
$ sh adstpall.sh apps/appsPWD
Notes: if FND processes are not stopped by above line, use "./adcmctl.sh stop" to stop them
If all service groups were enabled on the Source node, edit $CONTEXT_FILE to disable two groups (because the newly added node is for concurrent jobs).
$ vi $CONTEXT_FILE
For web/forms node:
Root Service Group : s_root_status => enabled
Web Entry Point Services : s_web_entry_status => enabled
Web Application Services : s_web_applications_status => enabled
Web Application Services : s_web_applications_status => enabled
Batch Processing Services : s_batch_status => disabled
Other Service Group : s_other_service_group_status => disabled
In a new OS session, stop Admin Server on PATCH file system
$ . /u02/app/EBSDEV//EBSapps.env patch
$ echo $FILE_EDITION
patch
$ cd $ADMIN_SCRIPTS_HOME
$ adadminsrvctl.sh stop forcepatchfs=yes
$ adnodemgrctl.sh stop
$ ps -ef | grep $LOGNAME | grep fs2
$ exit
h) Run autoconfig on database server (if Master node was newly cloned from other instance, this step seems optional)
$ cd $AD_TOP/bin
$ ls -al ad*mk*
$ perl admkappsutil.pl
Starting the generation of appsutil.zip
Log file located at $INST_TOP/admin/log/MakeAppsUtil_04151607.log
output located at $INST_TOP/admin/out/appsutil.zip
MakeAppsUtil completed successfully.
Send file appsutil.zip to DBA for running adconfig.sh.
i) Run autoconfig on all app-tier nodes
In sequence: database node => 2nd web/forms => CM node => Master
$ cd $ADMIN_SCRIPTS_HOME
$ ps -ef | grep $LOGNAME
$ ./adautocfg.sh
j) Start apps services on all nodes
$ ps -ef | grep $LOGNAME
$ sh adstrtal.sh apps/appsPWD
Optional:
$ lsnrctl reload APPS_${TWO_TASK}
Connecting to (ADDRESS=(PROTOCOL=TCP)(Host=$EBS_HOSTNAME)(Port=1562))
The command completed successfully
Now, test R12.2 works on all nodes.
k) Steps to make SCP works in Master node without asking a password. This is required for ADOP in multi-nodes instance to avoid error
[UNEXPECTED]ssh is not enabled.
I did try txkRunSSHSetup.pl for this and could not resolve some errors. More in Doc ID 2461229.1 (ERROR During Perl $AD_TOP/patch/115/bin/txkRunSSHSetup.pl Enablessh).
m) Validate ADOP. After ssh can connect to remote nodes with entering a password, ADOP validation worked:
$ adop -status
Notes: It may show only the Master node (with FS_CLONE completed). But after adop is used to apply patch to both nodes, "adop -status" will show two nodes.
$ adop -validate
... ...
Checking for existing patching cycle.
No existing patching cycle exists
Verifying SSH connection to all nodes.
Log: $LOG_HOME/appl/rgf/TXK/verifyssh.log
Output: $LOG_HOME/appl/rgf/TXK/out.xml
Remote execution is operational.
Running adop validations on Admin node: [node1].
Log: node1:$NE_BASE/EBSapps/log/adop/5/.../validate/node1
Output: $NE_BASE/EBSapps/log/adop/5/.../validate/remote_execution_result_level1.xml
txkADOPEvalSrvStatus.pl returned SUCCESS
Running adop validations on node(s): [node2,node3 and ].
Output: $NE_BASE/EBSapps/log/adop/5/.../validate/remote_execution_result_level2.xml
txkADOPEvalSrvStatus.pl returned SUCCESS
adop exiting with status = 0 (Success)
If it gets error on Public synonym, it is safe to drop it because R12.2 does not use Public synonym.
3. More notes.
a) If concurrent processes do not start or concurrent manager log gives below error, most likely it can be fixed by running autoconfig again.
Unable to contact Applications TNS Listener for FNDSM_<CONTEXT_NAME> on $NODE. The Listener may be down or the TNS address is not properly configured for this name. (NODE=node_name)
An error occured in client-side routine afpsmckp for Service Manager FNDSM_<CONTEXT_FILE>. The routine returned code 144.
b) You have to start apps services on Master node first to avoid errors from adstrtal.sh:
ERROR: Skipping startup of oacore_server1 since the AdminServer is down. AdminServer needs to be up before starting/stopping the managed servers.
... ...
c) Start apps services on web/forms node (non-Master node) with option"adstrtal.sh -msimode" (see Doc ID 2052792.1) to avoid errors:
ERROR: Skipping startup of oacore_server2 since the AdminServer is down. AdminServer needs to be up before starting/stopping the managed servers.
d) Below line tells it is the Master node:
$ grep adminserverstatus $CONTEXT_FILE
<oa_service_status oa_var="s_adminserverstatus">enabled</oa_service_status>
<oa_service_status oa_var="s_adminserverstatus">enabled</oa_service_status>
e) SQL statements to check more on status: all nodes shall have record in 4 tables below
set lines 333;
col NODE_NAME for a18;
col NODENAME for a18;
SQL> select * from ADOP_VALID_NODES;
SQL> select * from AD_NODES_CONFIG_STATUS;
SQL> select * from fnd_oam_context_files;
SQL> select * from fnd_oam_context_files
where node_name = 'target_node_name' ;
SQL> select NODE_NAME "Node_Name",
decode(STATUS,'Y','ACTIVE','INACTIVE') "Status",
decode(SUPPORT_CP,'Y', 'Y','') "Conc",
decode(SUPPORT_FORMS,'Y','Y', '') "Forms",
decode(SUPPORT_WEB,'Y','Y', '') "Web",
decode(SUPPORT_ADMIN, 'Y','Y', '') "Admin",
decode(SUPPORT_DB, 'Y','Y', '') "DB",
NODE_MODE "Mode",
HOST||'.'||DOMAIN "Server.Domain"
from fnd_nodes where node_name != 'AUTHENTICATION'
order by node_name;
If a patch status on the new added node shows NOT APPLIED, it indicates its snapshot needs a complete update.
SQL> SELECT adb.bug_number, aat.name appl_top_name, adb.creation_date,
decode(ad_patch.is_patch_applied('R12',aat.appl_top_id,adb.bug_number,adb.language),'EXPLICIT','APPLIED','NOT_APPLIED','NOT APPLIED') status
FROM ad_bugs adb,
(select aat.name, aat.appl_top_id
from applsys.ad_appl_tops aat,
(select distinct fat.name from applsys.fnd_appl_tops fat) fat
where aat.name=fat.name ) aat
where adb.bug_number in ('30399994')
order by adb.bug_number,aat.name,adb.language;
BUB_NUMBER APPL_TOP_NAME CREATION_DATE STATUS
30399994 AT_master_node 9-MAR-21 APPLIED
30399994 new node 9-MAR-21 NOT APPLIED
- TROUBLESHOOTING:
1. If "perl adcfgclone.pl appsTier dualfs" was executed from a wrong file system (such /fs2/), it would work successfully. But after you realize that and run the .pl clone script again, it will give errors:
ERROR: node_name entry is there in fnd_oam_context_files
ERROR: node_name entry is there in ADOP_VALID_NODES
ERROR: node_name entry is there in fnd_nodes
ERROR: node_name already registered in AutoConfig tables
ERROR: Some old configuration footprints still exists
Run ebs-delete-node before re-running add node
If adcfgclone started in the same session where previous environment (R12.1) was set, it will fail with errors:
[opmn] [ERROR:1] [] [internal] lxlinit - NLS boot file not found or invalid
-- default linked-in boot block used
[opmn] [ERROR:1] [105] [internal] XML parser init: character set initialization failed.
... ...
ERROR: Unable to create OHS instance.
ERROR: OHS instance creation and configuration failed
In both situations, try below steps to make the clone script work from scratch:
a) ps -ef | grep $LOGNAME
to make sure nothing is running
b) Do NOT delete any folders/files yet, and go to the RUN file system created by the clone script on Target node:
$ . /u02/app/EBSDEV/EBSapps.env RUN
$ echo $FILE_EDITION
run
$ perl $AD_TOP/patch/115/bin/adProvisionEBS.pl ebs-delete-node \
-contextfile=$CONTEXT_FILE -hostname=target_node_name -logfile=del_node.log )
Enter the APPS Schema password:
Enter the WebLogic AdminServer password:
Node deleted successfully.
Also see How To Delete an Application Tier Node (Doc ID 2315133.1)
(You may get ERROR: An error has occurred. Refer to the log file delete_node.log for details.
The log file show a lot of Java errors. But, It does delete data from db tables FND_NODES, etc.)
SQL> select * from ADOP_VALID_NODES;
SQL> select * from AD_NODES_CONFIG_STATUS;
SQL> select * from FND_NODES;
SQL> select * from fnd_oam_context_files
where node_name = 'target_node_name' ;
c) Remove/rename folders fs1, fs2, fs_ne under /u02/app/EBSDEV
d) Remove and then re-create folder oraInventory defined in /etc/oraInst.loc
e) Copy two folders from master node (Source node) again.
f) Run "perl adcfgclone.pl appsTier dualfs" from the directory matching RUN file system in Source node.
2. If you get below error, install RPM libaio-devel-0.3.109-13.el7.x86_64
FMW Pre-requisite check log file location : /u02/app/EBSDEV/fs1/EBSapps/comn/clone/FMW/logs/prereqcheck.log
Running: FMW pre-req check...
ERROR: Failed to execute FMW pre-requisite check
Log file has error matching Doc ID 2502181.1 (Adcfgclone.pl Error "START: Creating FMW Home"):
3. "perl adcfgclone.pl appsTier dualfs" gives error:
PATCH Weblogic AdminServer Connection attempt failed with
Connection rejected, filter blocked Socket, weblogic.security.net.FilterException:
Please ensure ConnectionFilter Rule exist for node_name.domain.com
In this case, wget and browsers get errors from accessing WLS Admin Server page:
$ wget http://node_name.domain.com:7033/console
--2021-0X-19 13:11:44-- http://node_name.domain.com:7033/console
wget: /u02/app/.netrc:4: unknown token "GROUP/APPLMGR"
Resolving node_name.domain.com (node_name.domain.com)... 367.69.67.89
Connecting to node_name.domain.com (node_name.domain.com)|367.69.67.89|:7033... connected.
HTTP request sent, awaiting response... 403 Forbidden
2021-04-19 13:11:44 ERROR 403: Forbidden.
Socket:000445]Connection rejected, filter blocked Socket, weblogic.security.net.FilterException: [Security:090220]
You have to fix it on both file editions on Source node:
$ cd $FMW_HOME/user_projects/domains/EBS_domain_"$TWO_TASK"/config
$ diff config.xml config.xml_BK_clone
< <connection-filter-rule>0.0.0.0/0 * * allow</connection-filter-rule>
---
> <connection-filter-rule>0.0.0.0/0 * * deny</connection-filter-rule>
Seems the permanent fix is to modify config.xml twice by replacing "deny" and each time is followed by an autoconfig (which will re-generate config.xml file).
4. Failure due to ADOP status
First of all, R12.2 cloning has requirement on ADOP status before the instance can be cloned. Anytime after "adop phase=prepare" completed or failed, the pre-clone script will not run on Source instance:
$ cd $ADMIN_SCRIPTS_HOME
$ perl adpreclone.pl appsTier
Copyright (c) 2011, 2014 Oracle Corporation
Redwood Shores, California, USA
Redwood Shores, California, USA
Oracle E-Business Suite Rapid Clone
Version 12.2
adpreclone Version 120.31.12020000.25
Enter the APPS User Password:
Enter the Weblogic AdminServer password :
Enter the Weblogic AdminServer password :
Checking the status of the Oracle WebLogic Administration Server....
Running perl $AD_TOP/patch/115/bin/adProvisionEBS.pl ebs-get-serverstatus -contextfile=$CONTEXT_FILE -servername=AdminServer -promptmsg=hide
The Oracle WebLogic Administration Server is up.
wlsDomainName: EBS_domain_$TWO_TASK
WLS Domain Name is VALID.
WLS Domain Name is VALID.
There is already an ACTIVE ADOP CYCLE with session id : 5
adpreclone cannot be run with pending ADOP session
adpreclone cannot be run with pending ADOP session
$ adop -status
Enter the APPS password:
Connected.
==============================================================
ADOP (C.Delta.12)
Session Id: 5
Command: status
Output: $NE_BASE/EBSapps/log/adop/5/.../adzdshowstatus.out
===============================================================
Node Name Node Type Phase Status Started Finished Elapsed
--------------- ---------- --------------- --------------- -------------------- -------------------- ------------
xifsapw3d master PREPARE COMPLETED 2021/02/25 16:06:14 2021/02/25 19:05:25 2:59:11
APPLY NOT STARTED
FINALIZE NOT STARTED
CUTOVER NOT STARTED
CLEANUP NOT STARTED
Connected.
==============================================================
ADOP (C.Delta.12)
Session Id: 5
Command: status
Output: $NE_BASE/EBSapps/log/adop/5/.../adzdshowstatus.out
===============================================================
Node Name Node Type Phase Status Started Finished Elapsed
--------------- ---------- --------------- --------------- -------------------- -------------------- ------------
xifsapw3d master PREPARE COMPLETED 2021/02/25 16:06:14 2021/02/25 19:05:25 2:59:11
APPLY NOT STARTED
FINALIZE NOT STARTED
CUTOVER NOT STARTED
CLEANUP NOT STARTED
File System Synchronization Type: Full
adop exiting with status = 0 (Success)
I had to Abort it (by "adop phase=abort" which takes hours) before I could do the clone.
$ adop -status -detail
Enter the APPS password:
Connected.
==============================================================
ADOP (C.Delta.12)
Session Id: 5
Command: status
Output: $NE_BASE/EBSapps/log/adop/5/.../adzdshowstatus.out
===============================================================
Node Name Node Type Phase Status Started Finished Elapsed
--------------- ---------- --------------- --------------- -------------------- -------------------- ------------
xifsapw3d master PREPARE SESSION ABORTED 2021/02/25 16:06:14 2021/02/25 19:05:25 2:59:11
APPLY SESSION ABORTED
FINALIZE SESSION ABORTED 2021/02/28 10:21:12 2021/02/28 10:29:15 0:08:03
CUTOVER SESSION ABORTED
CLEANUP COMPLETED 2021/02/28 19:00:07 2021/02/28 19:03:57 0:03:50
Connected.
==============================================================
ADOP (C.Delta.12)
Session Id: 5
Command: status
Output: $NE_BASE/EBSapps/log/adop/5/.../adzdshowstatus.out
===============================================================
Node Name Node Type Phase Status Started Finished Elapsed
--------------- ---------- --------------- --------------- -------------------- -------------------- ------------
xifsapw3d master PREPARE SESSION ABORTED 2021/02/25 16:06:14 2021/02/25 19:05:25 2:59:11
APPLY SESSION ABORTED
FINALIZE SESSION ABORTED 2021/02/28 10:21:12 2021/02/28 10:29:15 0:08:03
CUTOVER SESSION ABORTED
CLEANUP COMPLETED 2021/02/28 19:00:07 2021/02/28 19:03:57 0:03:50
File System Synchronization Type: Full
INFORMATION: Patching cycle aborted, so fs_clone will run automatically on node_name node in prepare phase of next patching cycle.
Generating full adop status report: $NE_BASE/EBSapps/log/adop/5/.../adzdshowstatus.out
adop exiting with status = 0 (Success)
5. After services started on 2nd node, I see the file systems are not identical.
On the 2nd node for concurrent jobs, this folder does not have "servers" folder:
$ ls -altr $FMW_HOME/user_projects/domains/EBS_domain
drwxr-xr-x 2 user group 37 Mar 1 20:00 security
drwxr-xr-x 4 user group 55 Mar 1 20:00 bin
drwxr-xr-x 3 user group 56 Mar 1 20:03 config
drwxr-xr-x 2 user group 37 Mar 1 20:00 security
drwxr-xr-x 4 user group 55 Mar 1 20:00 bin
drwxr-xr-x 3 user group 56 Mar 1 20:03 config
But, on the 1st node (master), the same folder has more sub-folders and "servers" folder has all service names:
$ ls -altr $FMW_HOME/user_projects/domains/EBS_domain
drwxr-x--- 2 user group 24 Mar 1 19:25 autodeploy
drwxr-x--- 8 user group 83 Mar 1 19:25 sysman
drwxr-x--- 2 user group 24 Mar 1 19:25 lib
drwxr-x--- 2 user group 24 Mar 1 19:25 console-ext
drwxr-x--- 5 user group 230 Mar 1 19:25 bin
drwxr-x--- 2 user group 143 Mar 1 19:25 security
-rwxr-x--- 1 user group 282 Mar 1 19:25 startWebLogic.sh
-rw-r----- 1 user group 243 Mar 1 19:30 shutdown.py
drwxr----- 9 user group 174 Mar 1 19:33 servers
drwxr-x--- 2 user group 6 Mar 1 19:34 pending
drwxr-x--- 2 user group 6 Mar 1 19:34 tmp
-rw-r----- 1 user group 32 Mar 1 20:00 edit.lok
drwxr-x--- 4 user group 66 Mar 1 20:01 opmn
drwxr-x---11 user group 4096 Mar 3 09:06 config
drwxr-x--- 2 user group 211 Mar 3 09:06 init-info
-rw-r----- 1 user group 495 Mar 3 09:06 fileRealm.properties
-rw-r--r-- 1 user group 808 Mar 3 09:06 startManagedWebLogic_readme.txt
drwxr-x--- 8 user group 83 Mar 1 19:25 sysman
drwxr-x--- 2 user group 24 Mar 1 19:25 lib
drwxr-x--- 2 user group 24 Mar 1 19:25 console-ext
drwxr-x--- 5 user group 230 Mar 1 19:25 bin
drwxr-x--- 2 user group 143 Mar 1 19:25 security
-rwxr-x--- 1 user group 282 Mar 1 19:25 startWebLogic.sh
-rw-r----- 1 user group 243 Mar 1 19:30 shutdown.py
drwxr----- 9 user group 174 Mar 1 19:33 servers
drwxr-x--- 2 user group 6 Mar 1 19:34 pending
drwxr-x--- 2 user group 6 Mar 1 19:34 tmp
-rw-r----- 1 user group 32 Mar 1 20:00 edit.lok
drwxr-x--- 4 user group 66 Mar 1 20:01 opmn
drwxr-x---11 user group 4096 Mar 3 09:06 config
drwxr-x--- 2 user group 211 Mar 3 09:06 init-info
-rw-r----- 1 user group 495 Mar 3 09:06 fileRealm.properties
-rw-r--r-- 1 user group 808 Mar 3 09:06 startManagedWebLogic_readme.txt
$ cd $FMW_HOME/user_projects/domains/EBS_domain/servers
$ ls -altr
drwxr----- 10 user group 109 Mar 1 19:26 AdminServer
drwxr-xr-x 3 user group 22 Mar 1 19:33 forms-c4ws_server1
drwxr-xr-x 3 user group 22 Mar 1 19:33 oaea_server1
drwxr-x--- 3 user group 25 Mar 1 19:34 domain_bak
drwxr-xr-x 9 user group 95 Mar 2 16:12 forms_server1
drwxr-xr-x 9 user group 95 Mar 2 16:12 oafm_server1
drwxr-xr-x 9 user group 95 Mar 2 16:12 oacore_server1
-rw-r----- 1 user group 14 Mar 10 13:47 AdminServerTag
drwxr----- 10 user group 109 Mar 1 19:26 AdminServer
drwxr-xr-x 3 user group 22 Mar 1 19:33 forms-c4ws_server1
drwxr-xr-x 3 user group 22 Mar 1 19:33 oaea_server1
drwxr-x--- 3 user group 25 Mar 1 19:34 domain_bak
drwxr-xr-x 9 user group 95 Mar 2 16:12 forms_server1
drwxr-xr-x 9 user group 95 Mar 2 16:12 oafm_server1
drwxr-xr-x 9 user group 95 Mar 2 16:12 oacore_server1
-rw-r----- 1 user group 14 Mar 10 13:47 AdminServerTag
6. MORE on Multi-node setups
Doc ID 415385.1 (On a R12 Multi-Node Install -- Why Do All Middle Tiers Run All Services?) gives details on deciding what shall run on each node:
You have 2 Application Tiers (XXX and YYY). XXX is running Admin and Concurrent Manager Services, and YYY is running Forms and Web Services.
During the Rapid Install, you select the above configuration via the "Edit Services" button as follows :
XXX : enable "Root Service Group", "Batch Processing Services"
YYY : enable "Root Service Group", "Web Entry Point Services", "Web Application Services", "Other Service Group"
YYY : enable "Root Service Group", "Web Entry Point Services", "Web Application Services", "Other Service Group"
The logic behind these service classification in R12.1 is as follows :
a) Root Services Group (Runs services on AS, 10.1.3 OH) : s_root_status
Oracle Process Manager: s_opmnstatus (adopmnctl.sh)
b) Web Entry Point Services (Runs services on AS, 10.1.3 OH) : s_web_entry_status
Oracle HTTP Server: s_oacorestatus (adapcctl.sh)
Oracle HTTP Server: s_oacorestatus (adapcctl.sh)
c) Web Application Services (Runs services on AS, 10.1.3 OH) : s_web_applications_status
OACORE OC4J: s_oacorectrl (adoacorectl.sh)
FORMS OC4J: s_formsstatus (adformsctl.sh)
OAFM OC4J: s_oafmstatus (adoafmctl.sh)
OACORE OC4J: s_oacorectrl (adoacorectl.sh)
FORMS OC4J: s_formsstatus (adformsctl.sh)
OAFM OC4J: s_oafmstatus (adoafmctl.sh)
d) Batch Processing Services (Concurrent Managers and Apps Listener): s_batch_status
OracleTNSListenerAPPS: s_tnsstatus (adalnctl.sh)
OracleConcMgr: s_concstatus (adcmctl.sh)
Oracle ICSM: s_icsmctrl (ieoicsm.sh)
Oracle Fulfillment Server: s_jtffsstatus (jtffmctl.sh)
OracleTNSListenerAPPS: s_tnsstatus (adalnctl.sh)
OracleConcMgr: s_concstatus (adcmctl.sh)
Oracle ICSM: s_icsmctrl (ieoicsm.sh)
Oracle Fulfillment Server: s_jtffsstatus (jtffmctl.sh)
e) Other Service Group (Runs services for Forms on 10.1.2) : s_other_service_group_status
OracleFormsServer: s_formsserver_status (adformsrvctl.sh)
Oracle Metrics Client: s_metcstatus (adfmcctl.sh)
Oracle Metrics Server: s_metsstatus (adfmsctl.sh)
Oracle MWA Service: s_mwastatus (mwactlwrpr.sh)
OracleFormsServer: s_formsserver_status (adformsrvctl.sh)
Oracle Metrics Client: s_metcstatus (adfmcctl.sh)
Oracle Metrics Server: s_metsstatus (adfmsctl.sh)
Oracle MWA Service: s_mwastatus (mwactlwrpr.sh)
De-attach (deattach) Oracle Homes
De-register the Oracle Homes on both Run Edition and Patch Edition file system:
Verify if Oracle Inventory contains the following Oracle Home entries for the Run Edition or Patch Edition file system:
<FMW_HOME>/oracle_common
<FMW_HOME>/webtier
<FMW_HOME>/Oracle_EBS-app1
<FMW_HOME>/oracle_common
<FMW_HOME>/webtier
<FMW_HOME>/Oracle_EBS-app1
If any of the above Oracle Home entries are already registered in Oracle Inventory, you can run the following command to de-register or detach that Oracle Home:
$ ./runInstaller -detachhome \
ORACLE_HOME=<Oracle Home Location> [-invPtrLoc <s_invPtrLoc>]
ORACLE_HOME=<Oracle Home Location> [-invPtrLoc <s_invPtrLoc>]
Here,
-invPtrLoc argument needs to be specified only if an 'EBS Installation Central' inventory is being used.
s_invPtrLoc is the context variable that stores the inventory pointer location.
-invPtrLoc argument needs to be specified only if an 'EBS Installation Central' inventory is being used.
s_invPtrLoc is the context variable that stores the inventory pointer location.
For example:
$ cd /u02/app/fs1/FMW_Home/oracle_common/oui/bin
$ ./runInstaller -detachhome \
ORACLE_HOME=/s0/r122/at1/FMW_Home/oracle_common
If the FMW_HOME directory structure exists, delete it as follows:
$ rm -rf <FMW_HOME>
$ cd /u02/app/fs1/FMW_Home/oracle_common/oui/bin
$ ./runInstaller -detachhome \
ORACLE_HOME=/s0/r122/at1/FMW_Home/oracle_common
If the FMW_HOME directory structure exists, delete it as follows:
$ rm -rf <FMW_HOME>
Related documents:
- Re-attach Oracle Homes To R12.2 Central Inventory (Doc ID 1586607.1)
- Sharing The Application Tier File System in Oracle E-Business Suite Release 12.2 ( Doc ID 1375769.1 )
