ERP on DB: Add/clone an apps-tier node to R12.2.10

Key steps for adding a node to R12.2 instance are in Oracle Doc ID 1383621.1 (Cloning Oracle E-Business Suite Release 12.2 with Rapid Clone, Section 5.3). From the Oracle document, seems the Source system has to be the primary node of the same existing system.

I used below steps to add node(s) in R12.2.10 as part of R12.2 upgrade (from R12.1.3). To save time, node may not be added to the system until post-upgrade steps, such as enabling TLS1.2, Enabling JWS, business & hot-fix patches, CPU patches and 3rd party tool installation, have been all completed.

1. Prepare the Source node

Make sure ADOP is in good status on Source/Master node:

$ adop -status

Enter the APPS password:

Connected.

====================================================

ADOP (C.Delta.12)

Session Id: 5

Command: status

Output: $NE_BASE/EBSapps/log/adop/5/.../adzdshowstatus.out

====================================================

Node Name Node Type Phase Status Started Finished Elapsed

--------------- ------------- --------------- ------------------- -------------------------- -------------------- ------------

node_name master FS_CLONE COMPLETED 2021/0X/14 13:15:26 2021/0X/14 14:53:11 1:37:45

Or, ADOP patch cycle never started:

$ adop -status

Enter the APPS password:

Connected.

================================================

ADOP (C.Delta.12)

Session Id: 3

Command: status

Output: $NE_BASE/EBSapps/log/adop/3/.../adzdshowstatus.out

================================================

File System Synchronization Type: Full

adop exiting with status = 0 (Success)

a) Run pre-clone script on RUN file system

First, check WLS Admin Server shall be up and be accessible from browser or a remote location

$ echo $FILE_EDITION

run

$ echo $APPL_TOP <= write down /fs2/ or /fs1/

$ cd $ADMIN_SCRIPTS_HOME

$ ./adadminsrvctl.sh status

You are running adadminsrvctl.sh version 120.10.12020000.11

Enter the WebLogic Admin password:

Enter the APPS Schema password:

The AdminServer is running

$ grep s_wls_adminport $CONTEXT_FILE
<wls_adminport oa_var="s_wls_adminport" oa_type="PORT" base="7001" step="1" range="-1" label="WLS Admin Server Port">7033</wls_adminport>

$ wget http://node_name.domain.com:7033/console
… … etc

HTTP request sent, awaiting response... 200 OK
Length: 3162 (3.1K) [text/html]
Saving to: ‘console’
100%[======================>] 3,162 --.-K/s in 0s

2021-0X-19 11:06:29 (343 MB/s) - ‘console’ saved [3162/3162]

$ perl adpreclone.pl appsTier

Oracle E-Business Suite Rapid Clone

Version 12.2

adpreclone Version 120.31.12020000.25

Enter the APPS User Password:
Enter the Weblogic AdminServer password :

Checking the status of the Oracle WebLogic Administration Server....

Running perl $AD_TOP/patch/115/bin/adProvisionEBS.pl ebs-get-serverstatus -contextfile=$CONTEXT_FILE -servername=AdminServer -promptmsg=hide

The Oracle WebLogic Administration Server is up.

wlsDomainName: EBS_domain_${TWO_TASK}
WLS Domain Name is VALID.

Running:
perl $AD_TOP/bin/adclone.pl java=$COMMON_TOP/util/jdk64 mode=stage stage=$COMMON_TOP/clone component=appsTier method= appctx=$CONTEXT_FILE showProgress

Setting the wls environment

Beginning application tier Stage - Sun Feb 28 19:12:36 2021

$COMMON_TOP/util/jdk64/bin/java -Xmx600M -Doracle.jdbc.autoCommitSpecCompliant=false -DCONTEXT_VALIDATED=false -Doracle.installer.oui_loc=/oui -classpath $FMW_HOME/webtier/lib/xmlparserv2.jar:$FMW_HOME/webtier/jdbc/lib/ojdbc6.jar:$COMMON_TOP/java/classes:$FMW_HOME/webtier/oui/jlib/OraInstaller.jar:$FMW_HOME/webtier/oui/jlib/ewt3.jar:$FMW_HOME/webtier/oui/jlib/share.jar:$FMW_HOME/webtier/../Oracle_EBS-app1/oui/jlib/srvm.jar:$FMW_HOME/webtier/jlib/ojmisc.jar:$FMW_HOME/wlserver_10.3/server/lib/weblogic.jar:$FMW_HOME/oracle_common/jlib/obfuscatepassword.jar oracle.apps.ad.clone.StageAppsTier -e $CONTEXT_FILE -stage $COMMON_TOP/clone -tmp /tmp -method CUSTOM -showProgress -nopromptmsg

Log file located at $INST_TOP/admin/log/clone/StageAppsTier_02281912.log

| 20% completed

Completed Stage...
Mon Xxx 29 19:19:28 2021

$ ls -altr $COMMON_TOP/clone

<= show 6 folders are newly created.

b) Run adpreclone.pl on PATCH file system as well. Start Admin Server first.

$ . $HOME/$TWO_TASK/EBSapps.env PATCH

$ echo $TWO_TASK

EBSDEV_patch

$ sqlplus /nolog

SQL> conn apps/appsPWD@EBSDEV_patch

ERROR:

ORA-00604: error occurred at recursive SQL level 1

ORA-20099: E-Business Suite Patch Edition does not exist.

ORA-06512: at line 29

SQL> conn apps/appsPWD@EBSDEV

Connected.

SQL> select count(*) from fnd_nodes;

COUNT(*)

----------

(Return info for database node and Master node)

$ echo $FILE_EDITION
patch

$ cd $FMW_HOME/user_projects/domains/EBS_domain/config

$ grep deny config.xml

if it has "deny", you shall replace it with "allow" before starting Admin Server.

$ cd $ADMIN_SCRIPTS_HOME

Start WebLogic Server Administration Console on the patch file system

$ sh adadminsrvctl.sh start forcepatchfs

You are running adadminsrvctl.sh version 120.10.12020000.11

Enter the WebLogic Admin password:
Enter the APPS Schema password:
Starting WLS Admin Server...

… … etc

adadminsrvctl.sh: exiting with status 0

adadminsrvctl.sh: check the logfile $INST_TOP/logs/appl/admin/log/adadminsrvctl.txt for more information ...

$ grep s_wls_adminport $CONTEXT_FILE
<wls_adminport oa_var="s_wls_adminport" oa_type="PORT" base="7001" step="1" range="-1" label="WLS Admin Server Port">7032</wls_adminport>

$ wget http://node_name.domain.com:7032/console

… … etc

HTTP request sent, awaiting response... 200 OK

Length: 3162 (3.1K) [text/html]
Saving to: ‘console’

100%[===================>] 3,162 --.-K/s in 0s

2021-0X-19 11:24:00 (354 MB/s) - ‘console’ saved [3162/3162]

$ perl adpreclone.pl appsTier

Oracle E-Business Suite Rapid Clone

Version 12.2

adpreclone Version 120.31.12020000.25

Enter the APPS User Password:
Enter the Weblogic AdminServer password :

Checking the status of the Oracle WebLogic Administration Server....

… … etc

| 20% completed

Completed Stage...
Mon Xxx 29 20:40:00 2021

$ ls -altr $COMMON_TOP/clone <== 9 folders have new timestamp.

c) Write down database info and other info

$ tnsping ebsdev

Database Server Node= ebsdb1d.domain.com

database SID = EBSDEV

PORT= 1562 (1562 - 1521 = 41)

apps apps_PWD

weblogic wls_PWD

Base Directory= /u02/app/EBSDEV

2. On TARGET node (2nd apps node. See Section 5.3.2):

a) Make sure clean env and new inventory folder

$ env <= to make sure NO any Oracle environment variables show up

$ vi /etc/oraInst.loc

make sure line "inventory_loc=/u02/app/oraInventory" is in the file

$ hostname <= make sure it is the right node!

$ cd /u02/app

$ rm -rf oraInventory; mkdir oraInventory

b) Create folders and copy files from Source node. Folder copay may take 1 to 2 hours.

$ cd /u02/app/EBSDEV

$ mkdir fs1

$ mkdir fs2

$ cd fs1

$ nohup rsync --progress -avze ssh applmgr@master_node:/u02/app/EBSDEV/fs1/EBSapps .

$ cd ../fs2

$ nohup rsync --progress -avze ssh applmgr@master_node:/u02/app/EBSDEV/fs2/EBSapps .

Notes: only copy EBSapps folders to avoid error:

ERROR: Failed to configure the target system

See Doc ID 1900583.1: RCloneApplyAppstier - Exiting cloning as FMW Home already exists

c) Make sure below two commands works on Target node, or Admin Server webpages can be accessed from browsers.

$ wget http://master_node.domain.com:7033/console

$ wget http://master_node.domain.com:7032/console

If you hit this error, you have to fix it on both file systems on Source node:

$ wget http://node_name.domain.com:7033/console

--2021-04-19 13:11:44-- http://node_name.domain.com:7033/console

wget: /u02/app/.netrc:4: unknown token "GROUP/APPLMGR"

Resolving node_name.domain.com (node_name.domain.com)... 367.69.67.89

Connecting to node_name.domain.com (node_name.domain.com)|367.69.67.89|:7033... connected.

HTTP request sent, awaiting response... 403 Forbidden

2021-04-19 13:11:44 ERROR 403: Forbidden.

d) Run the clone script

Assume this is to add a node only for concurrent jobs and database listens to port 1562

$ cd /u02/app/EBSDEV/fs1/EBSapps/comn/clone/bin

<= make sure $RUN_BASE on Source node is also on fs1 !

$ perl adcfgclone.pl appsTier dualfs

Oracle E-Business Suite Rapid Clone

Version 12.2

adcfgclone Version 120.63.12020000.65

Enter the APPS password :

Enter the Weblogic AdminServer password :

Do you want to add a node (yes/no) [no] : yes

Verifying: Run file system AdminServer is running
Verifying: Patch file system AdminServer is running
Running: Context clone...

Log file located at $COMMON_TOP/clone/bin/CloneContext_0301100358.log

Target System Hostname (virtual or normal) [node_name] :

Target System Base Directory set to /u02/app/EBSDEV

Target System Current File System Base set to /u02/app/EBSDEV/fs1

Target System Other File System Base set to /u02/app/EBSDEV/fs2

Target System Fusion Middleware Home set to /u02/app/EBSDEV/fs1/FMW_Home
Target System Other File System Fusion Middleware Home set to /u02/app/EBSDEV/fs2/FMW_Home

Target System Web Oracle Home set to /u02/app/EBSDEV/fs1/FMW_Home/webtier
Target System Other File System Web Oracle Home set to /u02/app/ESBDEV/fs2/FMW_Home/webtier

Target System Appl TOP set to /u02/app/EBSDEV/fs1/EBSapps/appl
Target System Other File System Appl TOP set to /u02/app/EBSDEV/fs2/EBSapps/appl

Target System COMMON TOP set to /u02/app/EBSDEV/fs1/EBSapps/comn
Target System Other File System COMMON TOP set to /u02/app/EBSDEV/fs2/EBSapps/comn

Target System Instance Home Directory [/u02/app/EBSDEV] :

Target System Current File System Instance Top set to /u02/app/EBSDEV/fs1/inst/apps/$CONTEXT_NAME

Do you want to preserve the Display [master_node:0.0] (y/n) : n

Target System Display [xifsapm3d:0.0] : node_name.domian.com:41.0

Target System Root Service [enabled] : disabled

Target System Web Entry Point Services [enabled] : disabled

Target System Web Application Services [enabled] : disabled

Target System Batch Processing Services [enabled] :

Target System Other Services [disabled] : disabled

Do you want the target system to have the same port values as the source system (y/n) [y] ? : y
Validating if the source port numbers are available on the target system..
Complete port information available at /u02/app/EBSDEV/fs1/EBSapps/comn/clone/bin/out/$CONTEXT_NAME/portpool.lst

Target System proxy port [80] :

UTL_FILE_DIR on database tier consists of the following directories.

1. /path/to/utl_dir
Choose a value which will be set as APPLPTMP value on the target node [1] :

The new APPL_TOP context file has been created :

/u02/app/EBSDEV/fs1/inst/apps/$CONTEXT_NAME/appl/admin/$CONTEXT_NAME.xml

Check Clone Context logfile $COMMON_TOP/clone/bin/CloneContext_0301100358.log for details.

Creating Patch file system context file.....

Log file located at $COMMON_TOP/clone/bin/CloneContextPatch_0301100504.log

Target System Other File System Instance Top set to /u02/app/EBSDEV/fs2/inst/apps/$CONTEXT_NAME
Validating if the source port numbers are available on the target system..
Complete port information available at $COMMON_TOP/clone/bin/out/$CONTEXT_NAME/portpool.lst

Target System proxy port [80] :

The new APPL_TOP context file has been created :

/u02/app/EBSDEV/fs2/inst/apps/$CONTEXT_NAME/appl/admin/$CONTEXT_NAME.xml

Check Clone Context logfile $COMMON_TOP/clone/bin/CloneContextPatch_0301100504.log for details.

FMW Pre-requisite check log file location : $COMMON_TOP/clone/FMW/logs/prereqcheck.log
Running: FMW pre-req check...

Configuring: Run file system....
LogFile located at $INST_TOP/admin/log/clone/run/RCloneApplyAppstier_03011005.log

Configuring: Patch file system....
LogFile located at $INST_TOP/admin/log/clone/patch/RCloneApplyAppstier_03011020.log

Do you want to startup the Application Services for EBSDEV? (y/n) [n] : n

Services not started

Notes: if it is to add a web/forms node, use below choices:

Target System Root Service [enabled] : enabled

Target System Web Entry Point Services [enabled] : enabled

Target System Web Application Services [enabled] : enabled

Target System Batch Processing Services [enabled] : disabled

Target System Other Services [disabled] : disabled

Now, verify database and diskspace:

By now, file EBSapps.env is created in /u02/app/EBSDEV.

$ . /u02/app/EBSDEV/EBSapps.env RUN (or, add it to .profile on Linux)

$ echo $APPL_TOP

/u02/app/EBSDEV/fs1/EBSapps/appl

SQL> select * from fnd_nodes;

SQL> select * from ADOP_VALID_NODES;

SQL> select * from fnd_oam_context_files;

$ cd /u02/app/EBSDEV

$ du -sh fs*

41G fs1 <= depending on how many patches were applied on master node

42G fs2

80K fs_ne

e) Edit $CONTEXT_FILE on Target node

$ vi $CONTEXT_FILE

R12.2 generates context file from source node and does not need to edit much. Verify below variables:

s_local_url_protocol, s_webentryurlprotocol, s_active_webport, s_https_listen_parameter, s_login_page, s_external_url, s_webentryhost

Edit variables:

applcsf, applptmp, s_forms_tmpdir, sitename, s_smtphost, s_javamailer_reply_to, s_sesstimeout

s_forms_catchterm, s_concstatus => disabled (to prevent concurrent manager from start on web node? Doc ID 417437.1. But seems this does nothing.)

f) On the newly added node, TLS 1.2 is not enabled by clone script. It is necessary to enable it manually. See Enable TLS1.2 in R12.2.

Also, need to copy two files from old R12.1 to $NE_BASE/EBSapps/appl/ad/admin/ for Java signing:

adkeystore.dat

adsign.txt

g) Stop apps services on Source/Master node. Do not do this yet if you have more nodes to add.

In the RUN session, stop all services first. NOTE: if R12.2 services are running, do not edit $CONTEXT_FILE before that because adstpall.sh reads it to decide what will be stopped

$ cd $ADMIN_SCRIPTS_HOME

$ sh adstpall.sh apps/appsPWD

Notes: if FND processes are not stopped by above line, use "./adcmctl.sh stop" to stop them

If all service groups were enabled on the Source node, edit $CONTEXT_FILE to disable two groups (because the newly added node is for concurrent jobs).

$ vi $CONTEXT_FILE

For web/forms node:

Root Service Group : s_root_status => enabled

Web Entry Point Services : s_web_entry_status => enabled
Web Application Services : s_web_applications_status => enabled

Batch Processing Services : s_batch_status => disabled

Other Service Group : s_other_service_group_status => disabled

In a new OS session, stop Admin Server on PATCH file system

$ . /u02/app/EBSDEV//EBSapps.env patch

$ echo $FILE_EDITION

patch

$ cd $ADMIN_SCRIPTS_HOME

$ adadminsrvctl.sh stop forcepatchfs=yes

$ adnodemgrctl.sh stop

$ ps -ef | grep $LOGNAME | grep fs2

$ exit

h) Run autoconfig on database server (if Master node was newly cloned from other instance, this step seems optional)

$ cd $AD_TOP/bin

$ ls -al ad*mk*

$ perl admkappsutil.pl

Starting the generation of appsutil.zip

Log file located at $INST_TOP/admin/log/MakeAppsUtil_04151607.log

output located at $INST_TOP/admin/out/appsutil.zip

MakeAppsUtil completed successfully.

Send file appsutil.zip to DBA for running adconfig.sh.

i) Run autoconfig on all app-tier nodes

In sequence: database node => 2nd web/forms => CM node => Master

$ cd $ADMIN_SCRIPTS_HOME

$ ps -ef | grep $LOGNAME

$ ./adautocfg.sh

j) Start apps services on all nodes

$ ps -ef | grep $LOGNAME

$ sh adstrtal.sh apps/appsPWD

Optional:

$ lsnrctl reload APPS_${TWO_TASK}

Connecting to (ADDRESS=(PROTOCOL=TCP)(Host=$EBS_HOSTNAME)(Port=1562))

The command completed successfully

Now, test R12.2 works on all nodes.

k) Steps to make SCP works in Master node without asking a password. This is required for ADOP in multi-nodes instance to avoid error

[UNEXPECTED]ssh is not enabled.

I did try txkRunSSHSetup.pl for this and could not resolve some errors. More in Doc ID 2461229.1 (ERROR During Perl $AD_TOP/patch/115/bin/txkRunSSHSetup.pl Enablessh).

m) Validate ADOP. After ssh can connect to remote nodes with entering a password, ADOP validation worked:

$ adop -status

Notes: It may show only the Master node (with FS_CLONE completed). But after adop is used to apply patch to both nodes, "adop -status" will show two nodes.

$ adop -validate

... ...

Checking for existing patching cycle.

No existing patching cycle exists

Verifying SSH connection to all nodes.

Log: $LOG_HOME/appl/rgf/TXK/verifyssh.log

Output: $LOG_HOME/appl/rgf/TXK/out.xml

Remote execution is operational.

Running adop validations on Admin node: [node1].

Log: node1:$NE_BASE/EBSapps/log/adop/5/.../validate/node1

Output: $NE_BASE/EBSapps/log/adop/5/.../validate/remote_execution_result_level1.xml

txkADOPEvalSrvStatus.pl returned SUCCESS

Running adop validations on node(s): [node2,node3 and ].

Output: $NE_BASE/EBSapps/log/adop/5/.../validate/remote_execution_result_level2.xml

txkADOPEvalSrvStatus.pl returned SUCCESS

adop exiting with status = 0 (Success)

If it gets error on Public synonym, it is safe to drop it because R12.2 does not use Public synonym.

3. More notes.

a) If concurrent processes do not start or concurrent manager log gives below error, most likely it can be fixed by running autoconfig again.

Unable to contact Applications TNS Listener for FNDSM_<CONTEXT_NAME> on $NODE. The Listener may be down or the TNS address is not properly configured for this name. (NODE=node_name)

An error occured in client-side routine afpsmckp for Service Manager FNDSM_<CONTEXT_FILE>. The routine returned code 144.

b) You have to start apps services on Master node first to avoid errors from adstrtal.sh:

ERROR: Skipping startup of oacore_server1 since the AdminServer is down. AdminServer needs to be up before starting/stopping the managed servers.

... ...

c) Start apps services on web/forms node (non-Master node) with option"adstrtal.sh -msimode" (see Doc ID 2052792.1) to avoid errors:

ERROR: Skipping startup of oacore_server2 since the AdminServer is down. AdminServer needs to be up before starting/stopping the managed servers.

d) Below line tells it is the Master node:

$ grep adminserverstatus $CONTEXT_FILE
<oa_service_status oa_var="s_adminserverstatus">enabled</oa_service_status>

e) SQL statements to check more on status: all nodes shall have record in 4 tables below

set lines 333;

col NODE_NAME for a18;

col NODENAME for a18;

SQL> select * from ADOP_VALID_NODES;

SQL> select * from AD_NODES_CONFIG_STATUS;

SQL> select * from fnd_oam_context_files;

SQL> select * from fnd_oam_context_files

where node_name = 'target_node_name' ;

SQL> select NODE_NAME "Node_Name",

decode(STATUS,'Y','ACTIVE','INACTIVE') "Status",

decode(SUPPORT_CP,'Y', 'Y','') "Conc",

decode(SUPPORT_FORMS,'Y','Y', '') "Forms",

decode(SUPPORT_WEB,'Y','Y', '') "Web",

decode(SUPPORT_ADMIN, 'Y','Y', '') "Admin",

decode(SUPPORT_DB, 'Y','Y', '') "DB",

NODE_MODE "Mode",

HOST||'.'||DOMAIN "Server.Domain"

from fnd_nodes where node_name != 'AUTHENTICATION'

order by node_name;

If a patch status on the new added node shows NOT APPLIED, it indicates its snapshot needs a complete update.

SQL> SELECT adb.bug_number, aat.name appl_top_name, adb.creation_date,

decode(ad_patch.is_patch_applied('R12',aat.appl_top_id,adb.bug_number,adb.language),'EXPLICIT','APPLIED','NOT_APPLIED','NOT APPLIED') status

FROM ad_bugs adb,

(select aat.name, aat.appl_top_id

from applsys.ad_appl_tops aat,

(select distinct fat.name from applsys.fnd_appl_tops fat) fat

where aat.name=fat.name ) aat

where adb.bug_number in ('30399994')

order by adb.bug_number,aat.name,adb.language;

BUB_NUMBER APPL_TOP_NAME CREATION_DATE STATUS

30399994 AT_master_node 9-MAR-21 APPLIED

30399994 new node 9-MAR-21 NOT APPLIED

- TROUBLESHOOTING:

1. If "perl adcfgclone.pl appsTier dualfs" was executed from a wrong file system (such /fs2/), it would work successfully. But after you realize that and run the .pl clone script again, it will give errors:

ERROR: node_name entry is there in fnd_oam_context_files

ERROR: node_name entry is there in ADOP_VALID_NODES

ERROR: node_name entry is there in fnd_nodes

ERROR: node_name already registered in AutoConfig tables

ERROR: Some old configuration footprints still exists

Run ebs-delete-node before re-running add node

If adcfgclone started in the same session where previous environment (R12.1) was set, it will fail with errors:

[opmn] [ERROR:1] [] [internal] lxlinit - NLS boot file not found or invalid

-- default linked-in boot block used

[opmn] [ERROR:1] [105] [internal] XML parser init: character set initialization failed.

... ...

ERROR: Unable to create OHS instance.

ERROR: OHS instance creation and configuration failed

In both situations, try below steps to make the clone script work from scratch:

a) ps -ef | grep $LOGNAME

to make sure nothing is running

b) Do NOT delete any folders/files yet, and go to the RUN file system created by the clone script on Target node:

$ . /u02/app/EBSDEV/EBSapps.env RUN

$ echo $FILE_EDITION

run

$ perl $AD_TOP/patch/115/bin/adProvisionEBS.pl ebs-delete-node \

-contextfile=$CONTEXT_FILE -hostname=target_node_name -logfile=del_node.log )

Enter the APPS Schema password:

Enter the WebLogic AdminServer password:

Node deleted successfully.

Also see How To Delete an Application Tier Node (Doc ID 2315133.1)

(You may get ERROR: An error has occurred. Refer to the log file delete_node.log for details.

The log file show a lot of Java errors. But, It does delete data from db tables FND_NODES, etc.)

SQL> select * from ADOP_VALID_NODES;

SQL> select * from AD_NODES_CONFIG_STATUS;

SQL> select * from FND_NODES;

SQL> select * from fnd_oam_context_files

where node_name = 'target_node_name' ;

c) Remove/rename folders fs1, fs2, fs_ne under /u02/app/EBSDEV

d) Remove and then re-create folder oraInventory defined in /etc/oraInst.loc

e) Copy two folders from master node (Source node) again.

f) Run "perl adcfgclone.pl appsTier dualfs" from the directory matching RUN file system in Source node.

2. If you get below error, install RPM libaio-devel-0.3.109-13.el7.x86_64

FMW Pre-requisite check log file location : /u02/app/EBSDEV/fs1/EBSapps/comn/clone/FMW/logs/prereqcheck.log

Running: FMW pre-req check...

ERROR: Failed to execute FMW pre-requisite check

Log file has error matching Doc ID 2502181.1 (Adcfgclone.pl Error "START: Creating FMW Home"):

3. "perl adcfgclone.pl appsTier dualfs" gives error:

PATCH Weblogic AdminServer Connection attempt failed with

Connection rejected, filter blocked Socket, weblogic.security.net.FilterException:

Please ensure ConnectionFilter Rule exist for node_name.domain.com

In this case, wget and browsers get errors from accessing WLS Admin Server page:

$ wget http://node_name.domain.com:7033/console

--2021-0X-19 13:11:44-- http://node_name.domain.com:7033/console

wget: /u02/app/.netrc:4: unknown token "GROUP/APPLMGR"

Resolving node_name.domain.com (node_name.domain.com)... 367.69.67.89

Connecting to node_name.domain.com (node_name.domain.com)|367.69.67.89|:7033... connected.

HTTP request sent, awaiting response... 403 Forbidden

2021-04-19 13:11:44 ERROR 403: Forbidden.

Socket:000445]Connection rejected, filter blocked Socket, weblogic.security.net.FilterException: [Security:090220]

You have to fix it on both file editions on Source node:

$ cd $FMW_HOME/user_projects/domains/EBS_domain_"$TWO_TASK"/config

$ diff config.xml config.xml_BK_clone

< <connection-filter-rule>0.0.0.0/0 * * allow</connection-filter-rule>

---

> <connection-filter-rule>0.0.0.0/0 * * deny</connection-filter-rule>

Seems the permanent fix is to modify config.xml twice by replacing "deny" and each time is followed by an autoconfig (which will re-generate config.xml file).

4. Failure due to ADOP status

First of all, R12.2 cloning has requirement on ADOP status before the instance can be cloned. Anytime after "adop phase=prepare" completed or failed, the pre-clone script will not run on Source instance:

$ cd $ADMIN_SCRIPTS_HOME

$ perl adpreclone.pl appsTier

Oracle E-Business Suite Rapid Clone

Version 12.2

adpreclone Version 120.31.12020000.25

Enter the APPS User Password:
Enter the Weblogic AdminServer password :

Checking the status of the Oracle WebLogic Administration Server....

Running perl $AD_TOP/patch/115/bin/adProvisionEBS.pl ebs-get-serverstatus -contextfile=$CONTEXT_FILE -servername=AdminServer -promptmsg=hide

The Oracle WebLogic Administration Server is up.

wlsDomainName: EBS_domain_$TWO_TASK
WLS Domain Name is VALID.

There is already an ACTIVE ADOP CYCLE with session id : 5
adpreclone cannot be run with pending ADOP session

$ adop -status

Enter the APPS password:
Connected.
==============================================================
ADOP (C.Delta.12)
Session Id: 5
Command: status
Output: $NE_BASE/EBSapps/log/adop/5/.../adzdshowstatus.out
===============================================================
Node Name       Node Type Phase           Status          Started              Finished             Elapsed
--------------- ---------- --------------- --------------- -------------------- -------------------- ------------
xifsapw3d       master     PREPARE    COMPLETED       2021/02/25 16:06:14 2021/02/25 19:05:25 2:59:11
                                      APPLY    NOT STARTED
                                      FINALIZE        NOT STARTED
                                      CUTOVER NOT STARTED
                                      CLEANUP NOT STARTED

File System Synchronization Type: Full

adop exiting with status = 0 (Success)

I had to Abort it (by "adop phase=abort" which takes hours) before I could do the clone.

$ adop -status -detail

Enter the APPS password:
Connected.
==============================================================
ADOP (C.Delta.12)
Session Id: 5
Command: status
Output: $NE_BASE/EBSapps/log/adop/5/.../adzdshowstatus.out
===============================================================
Node Name       Node Type Phase           Status          Started              Finished             Elapsed
--------------- ---------- --------------- --------------- -------------------- -------------------- ------------
xifsapw3d       master     PREPARE    SESSION ABORTED 2021/02/25 16:06:14 2021/02/25 19:05:25 2:59:11
   APPLY SESSION ABORTED
   FINALIZE SESSION ABORTED 2021/02/28 10:21:12 2021/02/28 10:29:15 0:08:03
   CUTOVER    SESSION ABORTED
   CLEANUP COMPLETED    2021/02/28 19:00:07 2021/02/28 19:03:57 0:03:50

File System Synchronization Type: Full

INFORMATION: Patching cycle aborted, so fs_clone will run automatically on node_name node in prepare phase of next patching cycle.

Generating full adop status report: $NE_BASE/EBSapps/log/adop/5/.../adzdshowstatus.out

adop exiting with status = 0 (Success)

5. After services started on 2nd node, I see the file systems are not identical.

On the 2nd node for concurrent jobs, this folder does not have "servers" folder:

$ ls -altr $FMW_HOME/user_projects/domains/EBS_domain

drwxr-xr-x 2 user group 37 Mar 1 20:00 security
drwxr-xr-x 4 user group 55 Mar 1 20:00 bin
drwxr-xr-x 3 user group 56 Mar 1 20:03 config

But, on the 1st node (master), the same folder has more sub-folders and "servers" folder has all service names:

$ ls -altr $FMW_HOME/user_projects/domains/EBS_domain

drwxr-x--- 2 user  group 24    Mar 1 19:25 autodeploy
drwxr-x--- 8 user group   83    Mar 1 19:25 sysman
drwxr-x--- 2 user group   24    Mar 1 19:25 lib
drwxr-x--- 2 user group   24    Mar 1 19:25 console-ext
drwxr-x--- 5 user group 230 Mar 1 19:25 bin
drwxr-x--- 2 user group 143 Mar 1 19:25 security
-rwxr-x--- 1 user group 282 Mar 1 19:25 startWebLogic.sh
-rw-r----- 1 user group 243 Mar 1 19:30 shutdown.py
drwxr----- 9 user group 174 Mar 1 19:33 servers
drwxr-x--- 2 user group    6 Mar 1 19:34 pending
drwxr-x--- 2 user group    6 Mar 1 19:34 tmp
-rw-r----- 1 user group   32    Mar 1 20:00 edit.lok
drwxr-x--- 4 user group   66    Mar 1 20:01 opmn
drwxr-x---11 user group 4096 Mar 3 09:06 config
drwxr-x--- 2 user group 211 Mar 3 09:06 init-info
-rw-r----- 1 user group 495 Mar 3 09:06 fileRealm.properties
-rw-r--r-- 1 user group  808 Mar 3 09:06 startManagedWebLogic_readme.txt

$ cd $FMW_HOME/user_projects/domains/EBS_domain/servers

$ ls -altr
drwxr----- 10 user group 109 Mar 1 19:26 AdminServer
drwxr-xr-x 3 user group 22 Mar 1 19:33 forms-c4ws_server1
drwxr-xr-x 3 user group 22 Mar 1 19:33 oaea_server1
drwxr-x--- 3 user group 25 Mar 1 19:34 domain_bak
drwxr-xr-x 9 user group 95 Mar 2 16:12 forms_server1
drwxr-xr-x 9 user group 95 Mar 2 16:12 oafm_server1
drwxr-xr-x 9 user group 95 Mar 2 16:12 oacore_server1
-rw-r----- 1 user group 14 Mar 10 13:47 AdminServerTag

6. MORE on Multi-node setups

Doc ID 415385.1 (On a R12 Multi-Node Install -- Why Do All Middle Tiers Run All Services?) gives details on deciding what shall run on each node:

You have 2 Application Tiers (XXX and YYY). XXX is running Admin and Concurrent Manager Services, and YYY is running Forms and Web Services.

During the Rapid Install, you select the above configuration via the "Edit Services" button as follows :

XXX : enable "Root Service Group", "Batch Processing Services"
YYY : enable "Root Service Group", "Web Entry Point Services", "Web Application Services", "Other Service Group"

The logic behind these service classification in R12.1 is as follows :

a) Root Services Group (Runs services on AS, 10.1.3 OH) : s_root_status

Oracle Process Manager: s_opmnstatus (adopmnctl.sh)

b) Web Entry Point Services (Runs services on AS, 10.1.3 OH) : s_web_entry_status
Oracle HTTP Server: s_oacorestatus (adapcctl.sh)

c) Web Application Services (Runs services on AS, 10.1.3 OH) : s_web_applications_status
OACORE OC4J: s_oacorectrl (adoacorectl.sh)
FORMS OC4J: s_formsstatus (adformsctl.sh)
OAFM OC4J: s_oafmstatus (adoafmctl.sh)

d) Batch Processing Services (Concurrent Managers and Apps Listener): s_batch_status
OracleTNSListenerAPPS: s_tnsstatus (adalnctl.sh)
OracleConcMgr: s_concstatus (adcmctl.sh)
Oracle ICSM: s_icsmctrl (ieoicsm.sh)
Oracle Fulfillment Server: s_jtffsstatus (jtffmctl.sh)

e) Other Service Group (Runs services for Forms on 10.1.2) : s_other_service_group_status
OracleFormsServer: s_formsserver_status (adformsrvctl.sh)
Oracle Metrics Client: s_metcstatus (adfmcctl.sh)
Oracle Metrics Server: s_metsstatus (adfmsctl.sh)
Oracle MWA Service: s_mwastatus (mwactlwrpr.sh)

De-attach (deattach) Oracle Homes

De-register the Oracle Homes on both Run Edition and Patch Edition file system:

Verify if Oracle Inventory contains the following Oracle Home entries for the Run Edition or Patch Edition file system:
<FMW_HOME>/oracle_common
<FMW_HOME>/webtier
<FMW_HOME>/Oracle_EBS-app1

If any of the above Oracle Home entries are already registered in Oracle Inventory, you can run the following command to de-register or detach that Oracle Home:

$ ./runInstaller -detachhome \
ORACLE_HOME=<Oracle Home Location> [-invPtrLoc <s_invPtrLoc>]

Here,
-invPtrLoc argument needs to be specified only if an 'EBS Installation Central' inventory is being used.
s_invPtrLoc is the context variable that stores the inventory pointer location.

For example:
$ cd /u02/app/fs1/FMW_Home/oracle_common/oui/bin
$ ./runInstaller -detachhome \
ORACLE_HOME=/s0/r122/at1/FMW_Home/oracle_common
If the FMW_HOME directory structure exists, delete it as follows:
$ rm -rf <FMW_HOME>

ERP on DB

Tuesday, May 25, 2021

Add/clone an apps-tier node to R12.2.10

No comments: