Clone EBS R12.2 instance

To clone a R12.2 instance is similar to add a node to an existing instance. The document for this task is Doc ID 1383621.1 (Cloning Oracle E-Business Suite Release 12.2 with Rapid Clone). "SQL> SELECT name, creation_date FROM apps.fnd_apps_system;" tells the last date the environment was cloned. 

1. Clone the database in steps:

a) Clone the DB

b) Clean out node info (EXEC FND_CONC_CLONE.SETUP_CLEAN)

c) Run autocfg

d) Recompile invalids

e) Run ETCC script

    ... ...

     Found patch records in the inventory.

     Checking Mapping XML file for 12.1.0.2.210420

     All the required one-off bugfixes are present in Database ORACLE_HOME.

     Stored Technology Codelevel Checker results in the database EBSDEV successfully.

     Finished checking fixes for Oracle Database: Fri ... ...

Notes: Per Doc ID 2008451.1, results of ETCC execution are stored in a database table (APPLSYS.TXK_TCC_RESULTS). The table may still hold obsolete data after DB clone, and needs at least one row for current db server to let ADOP work through and avoid error:

    [ERROR]     ETCC not run in the database node ebsdb1d

                The EBS Technology Codelevel Checker needs to be run on the database node.

                It is available as Patch 17537119

f) Confirm listeners are up. 

For 19c database, CDB name => CEBSDEV & PDB name => EBSDEV

$ lsnrctl status CEBSDEV

2. Preparation on Source instance (usually on its primary node)

Doc ID 1383621.1 (Cloning Oracle E-Business Suite Release 12.2 with Rapid Clone) says:

Before cloning a system with Rapid Clone, be sure to allow any active online patching cycles to run all the way through the final (cleanup) phase. In case patches are applied in hotpatch or downtime mode, then you must run cleanup phase of adop. 

Then run fs_clone to synchronize with the other file system, to avoid the need for synchronization to be performed in the next patching cycle. 

Most of time, running FS_CLONE is good enough, if there is no pending/open patching cycle (e.g. ADOP phases were not used to apply patch in online mode). If FS_CLONE status was completed long time ago (and downtime mode was not used to apply patches since then), it is still preferable to run FS_CLONE again to synchronize PATCH file system with all configuration changes, such as oacore addition, ssl cert renewal, etc.

$ grep adminserverstatus $CONTEXT_FILE

            <oa_service_status oa_var="s_adminserverstatus">enabled</oa_service_status>

$ adop -status 

If an ADOP step is not completed successfully, "perl adpreclone.pl appsTier" may fail. 

3. Run pre-clone script in both RUN and PATCH file systems on Source node

$ echo $FILE_EDITION

run

$ cd $ADMIN_SCRIPTS_HOME

$ perl adpreclone.pl appsTier

Enter the APPS User Password:

Enter the Weblogic AdminServer password :

... ...

Completed Stage...

$ ls -altr $COMMON_TOP/clone                  <== 6 folders have new timestamp

$ . $HOME/$TWO_TASK/EBSapps.env PATCH

$ echo $FILE_EDITION

patch

$ cd $ADMIN_SCRIPTS_HOME

$ sh adadminsrvctl.sh start forcepatchfs

$ perl adpreclone.pl appsTier

$ ls -altr $COMMON_TOP/clone                  <== 9 folders have new timestamp

$ adadminsrvctl.sh stop forcepatchfs=yes

$ adnodemgrctl.sh stop

4. Copy files from Source node to Target node

Key step: before dropping old folders on Target node, save (back up!) some key config files for the refresh.

Also, write down the result "echo $TWO_TASK":

$ echo $TWO_TASK

EBSDEV

$ echo $RUN_BASE     <= for comparison 

$ cd /u07/app/EBSDEV

$ mkdir fs1

$ mkdir fs2

$ cd fs1

$ nohup rsync --progress -avze ssh applmgr@master_node:/u02/app/EBSQA/fs1/EBSapps .

$ cd ../fs2

$ nohup rsync --progress -avze ssh applmgr@master_node:/u02/app/EBSQA/fs2/EBSapps .

5. Make sure database is running and connection works.

Make sure listeners ebsdev, cebsdev are up and running on database server:

$ lsnrctl status CEBSDEV

Use Oracle client on a laptop or a terminal to connect to database, and verify some info:

$ sqlplus apps/appsPWD@ebsdev 

    

SQL> show parameter local_listener

NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

local_listener                       string       ebdb1d:1537

SQL> show parameter service

NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

service_names                        string      CEBSDEV

 

SQL> select * from fnd_nodes;                         <= one row of db node info

SQL> select * from fnd_oam_context_files;     <= 3 rows on database node

SQL> select * from ADOP_VALID_NODES;  <= it will get updated after clone

If they have some unexpected rows, the clone script may clean them.

Then, use tnsping to get the database host name and port number. Below setting depends on entries in tnsnames.ora file on your local client:

$ tnsping EBSDEV

Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL= TCP)(Host=ebdb1d.domain.com)(Port= 1537)) (CONNECT_DATA = (SERVICE_NAME = EBSDEV)))

OK (220 msec)

Or

$ tnsping EBSDEV

Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL= TCP)(Host=ebdb1d.domain.com)(Port= 1537)) (CONNECT_DATA = (SID = EBSDEV)))

OK (200 msec)

6. Run the clone script for both file systems (with dualfs option) on Target server

a) env

b) vi /etc/oraInst.loc

c) rm -rf oraEBSDEVInventory; mkdir oraEBSDEVInventory

d) cd EBSDEV/fs1/EBSapps/comn/clone/bin      <== assume fs1 is the RUN on Source instance

UPDATES: 

If Target mid-tier and Source mid-tier are on the same server, adcfgclone.pl may fail with port conflict error. To prevent that happens, stop apps services of Source instance to let the cloning complete, and then start apps services to find if any issues. See Port conflict during clone.

$ perl adcfgclone.pl appsTier dualfs

                     Copyright (c) 2002, 2015 Oracle Corporation

                        Redwood Shores, California, USA

                        Oracle E-Business Suite Rapid Clone

                                 Version 12.2

                      adcfgclone Version 120.63.12020000.65

Enter the APPS password :

Enter the Weblogic AdminServer password : 

                                                                <== Will be new WLS password for Target instance

Do you want to add a node (yes/no) [no] : no

Running: Context clone...

Log file located at $COMMON_TOP/clone/bin/CloneContext_0209154645.log

Provide the values required for creation of the new APPL_TOP Context file.

Target System Hostname (virtual or normal) [nodeName] :

Target System Database SID : EBSDEV       <== same for previous $TWO_TASK

Target System Database Server Node [nodeName] : ebdb1d.domain.com

Target System Base Directory : /u07/app/EBSDEV

Target System Base Directory set to /u07/app/EBSDEV

Target System Current File System Base set to /u07/app/EBSDEV/fs1

Target System Other File System Base set to /u07/app/EBSDEV/fs2

Target System Fusion Middleware Home set to /u07/app/EBSDEVfs1/FMW_Home

Target System Other File System Fusion Middleware Home set to /u07/app/EBSDEV/fs2/FMW_Home

Target System Web Oracle Home set to /u07/app/EBSDEV/fs1/FMW_Home/webtier

Target System Other File System Web Oracle Home set to /u07/app/EBSDEV/fs2/FMW_Home/webtier

Target System Appl TOP set to /u07/app/EBSDEV/fs1/EBSapps/appl

Target System Other File System Appl TOP set to /u07/app/EBSDEV/fs2/EBSapps/appl

Target System COMMON TOP set to /u07/app/EBSDEV/fs1/EBSapps/comn

Target System Other File System COMMON TOP set to /u07/app/EBSDEV/fs2/EBSapps/comn

Target System Instance Home Directory [/u07/app/EBSDEV] :

Target System Current File System Instance Top set to /u07/app/EBSDEV/fs1/inst/apps/EBSDEV_nodeName

Do you want to preserve the Display [NodeName:0.0] (y/n)  : n

Target System Display [nodeName:0.0] : nodeName.domain.com:16.0

Target System Root Service [enabled] :

Target System Web Entry Point Services [enabled] :             <== assume this is a web/form node

Target System Web Application Services [enabled] :

Target System Batch Processing Services [disabled] :

Target System Other Services [disabled] :

Do you want the target system to have the same port values as the source system (y/n) [y] ? : n

Target System Port Pool [0-99] : 16

Checking the port pool 16

done: Port Pool 16 is free

Report file located at $INST_TOP/admin/out/portpool.lst

Target System proxy port [80] :

UTL_FILE_DIR on database tier consists of the following directories.

1. /network/to/EBSDEV/utl_dir       <== get this path from database

2. ... ...

Choose a value which will be set as APPLPTMP value on the target node [1] : 1

The new APPL_TOP context file has been created :

 $INST_TOP/appl/admin/$CONTEXT_NAME.xml

Check Clone Context logfile $COMMON_TOP/clone/bin/CloneContext_0209154645.log for details.

Creating Patch file system context file.....

Log file located at $COMMON_TOP/clone/bin/CloneContextPatch_0209155146.log

Target System Other File System Instance Top set to /u07/app/EBSDEV/fs2/inst/apps/$CONTEXT_NAME

Target System Port Pool [0-99] : 16

Patch file system should have different port pool than Run file system..

Target System Port Pool [0-99] : 17

Checking the port pool 17

done: Port Pool 17 is free

Report file located at /u07/app/EBSDEV/fs2/inst/apps/$CONTEXT_NAME/admin/out/portpool.lst

The new APPL_TOP context file has been created : ... ... (on fs2)

Check Clone Context logfile $CONMMON_TOP/clone/bin/CloneContextPatch_0209155146.log for details.

FMW Pre-requisite check log file location : $COMMON_TOP/clone/FMW/logs/prereqcheck.log

Running: FMW pre-req check...

Configuring: Run file system....

LogFile located at $INST_TOP/admin/log/clone/run/RCloneApplyAppstier_02091555.log

<Feb 9, 2022 4:17:08 PM EST> <Warning> <JNDI> <BEA-050001> <WLContext.close() was called in a different thread than the one in which it was created.>          <== not sure why got them?

<Feb 9, 2022 4:22:57 PM EST> <Warning> <JNDI> <BEA-050001> <WLContext.close() was called in a different thread than the one in which it was created.>

Configuring: Patch file system....

LogFile located at $INST_TOP/admin/log/clone/patch/RCloneApplyAppstier_02091623.log

<Feb 9, 2022 4:50:51 PM EST> <Warning> <JNDI> <BEA-050001> <WLContext.close() was called in a different thread than the one in which it was created.>

Do you want to startup the Application Services for EBSDEV? (y/n) [n] : n

Services not started

NOTES: Cloning logs are in folders run and patch under $INST_TOP/admin/log/clone. But they may be removed after ADOP runs. Seems some log files are consolidated to sub-folders under $INST_TOP/admin/log/clone. All .error files under $INST_TOP/admin/log/clone/xxxT2PApply shall be empty.  

7. Prepare the Target instance

a) add below line to .profile (on Linux node) and then start a new OS session

. /u02/app/EBSDEV/EBSapps.env RUN 

$ echo $FILE_EDITION

$ echo $APPL_TOP

$ echo $TWO_TASK

EBSDEV

$ grep adminserverstatus $CONTEXT_FILE

            <oa_service_status oa_var="s_adminserverstatus">enabled</oa_service_status>

b) Connect to database

$ sqlplus apps/appsPWD

set lines 333;

col NODE_NAME for a18;

col NODENAME for a18;

SQL> select * from fnd_nodes;                                      <== 3 rows

SQL> select * from ADOP_VALID_NODES;                  <== 1 row

SQL> select * from AD_NODES_CONFIG_STATUS;    <== empty

SQL> select * from fnd_oam_context_files;

SQL> select * from fnd_oam_context_files 

          where node_name = 'nodeName' ;       <== shall have rows for RUN and PATCH

SQL> select instance_name from v$instance;

INSTANCE_NAME

----------------

CEBSDEV

In 19c database, the clone script create tnsames.ora maybe with below entries, by pulling info from database setups:

$ tnsping ebsdev

... ...

Used TNSNAMES adapter to resolve the alias

Attempting to contact (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp)(HOST=ebdb1d.domain.com)(PORT=1537)) (CONNECT_DATA= (SERVICE_NAME=ebs_EBSDEV) (INSTANCE_NAME=CDBEDEV)))

OK (0 msec)

Do not run "EXEC FND_CONC_CLONE.SETUP_CLEAN;" again, because it will delete all data in some tables and autoconfig may not get them fully back to table fnd_oam_context_files.

8. Enable SSL TLS1.2 by using cert file in $NE_BASE/inst/$CONTEXT_NAME/certs and copy Java signing files to $NE_BASE/EBSapps/appl/ad/admin/. See Enable TLS1.2 in R12.2 Steps 1 - 4. Note This step can be skipped on the concurrent node. But if TLS 1.2 was enabled, it is necessary to get the later cert file to avoid potential FS_CLONE failure.

Edit SSL config files. Do not replace files by backed-up files from prior config, because s_ohs_instance or s_wls_domain_name (and so WLS file paths) could be different from prior one. 

9. Generate .zip file for autoconfig on database node 

$ cd $AD_TOP/bin

$ ls -al ad*mk*

$ perl admkappsutil.pl

Copy $INST_TOP/admin/out/appsutil.zip to database node and run AutoConfig on database node.

10. Edit $CONTEXT_FILE and restore files

applcsf, applptmp, s_forms_tmpdir, sitename, s_smtphost, s_javamailer_reply_to, s_sesstimeout, s_url_protocol, s_local_url_protocol, s_webentryurlprotocol, s_active_webport, s_https_listen_parameter,  s_login_page, s_external_url, s_webentryhost, forms_catchterm

Restore files from backup:

$APPL_TOP_NE/ad/admin/adkeystore.dat

$APPL_TOP_NE/ad/custom/adop_sync.drv

Verify files/folders:

$APPL_TOP_NE/ad/admin/adsign.txt

$FND_TOP/fndenv.env

$INST_TOP/ora/10.1.2/forms/server/default.env

$AD_TOP/admin/template/custom    (if exists)

$FND_TOP/admin/template/custom

Apply a workaround to make the Weblogic console accessible after started by modifying a file: 

$ iPath=`tr < $CONTEXT_FILE '<>' '  ' | awk '/"s_wls_domain_name/ {print $(NF-1)}'`

$ echo $iPath

$ cd $FMW_HOME/user_projects/domains/$iPath/config

$ cp -p config.xml config.xml_BK_clone

$  diff config.xml config.xml_BK_clone

<     <connection-filter-rule>0.0.0.0/0 * * allow</connection-filter-rule>

---

>     <connection-filter-rule>0.0.0.0/0 * * deny</connection-filter-rule>

11. Run configuration scripts

-  adadmin  (option: 1==> 4  Yes)

Run it to sign Java files to avoid GUI forms error:  Application Blocked by Java Security

Your security settings have blocked an application signed with an expired or not-yet-valid certificate from running.

- autoconfig on nodeName 

12. Start all services

$ cd $ADMIN_SCRIPTS_HOME

$ ./adstrtal.sh apps/apps_PWD

Then, test the login and GUI forms. Also make sure Weblogic console works.

$ grep s_wls_adminport $CONTEXT_FILE

13. Change APPS password in two steps (if needed)

Notes: Doc ID 2360475.1 states APPS password is not saved in any files but stored in database.

1) Stop Apps services

2) $ FNDCPASS apps/old_appsPWS 0 Y system/'systemPWD' SYSTEM APPLSYS new_AppsPWD

3) autoconfig

4) start Admin services

./adadminsrvctl.sh start

5) Manually update the password in WLS DataSource for starting oacore, etc.

See Doc ID 1674462.1 (R12.2: Steps to Change the APPS, APPLSYS, and APPS_NE Password Using FNDCPASS or AFPASSWD)

a. Log in to WLS Administration Console.

b. Click Lock & Edit in Change Center.

c. In the Domain Structure tree, expand Services, then select Data Sources.

d. On the "Summary of JDBC Data Sources" page, select EBSDataSource.

e. On the "Settings for EBSDataSource" page, select the Connection Pool tab.

f. Enter the new password in the "Password" field.

g. Enter the new password in the "Confirm Password" field.

h. Click Save.

i. Click Activate Changes in Change Center.

6) Start all Apps services

14. Sanity check on other things, such as

$ adop -status         <== Nothing with adop yet.

Enter the APPS password:

Connected.

======================================================

ADOP (C.Delta.12)

Session Id: 8

Command: status

Output: $NE_BASE/EBSapps/log/adop/8/.../adzdshowstatus.out

======================================================

File System Synchronization Type: None

adop exiting with status = 0 (Success)

15. Add additional node(s) to the new instance

Copy files from this new instance to add more nodes to the instance. In this way, all ports are kept the same in all nodes. See Add a R12.2 node.

16. After all are done, run FS_CLONE to synchronize up edited settings to PATCH file system. 

Without this step, context variable s_active_webport on RUN and PATCH file systems is different if TLS1.2 was enabled in Step 8.

UPDATES on troubleshooting

A)  In one case I got below error from "adop phase=cutover" and then I had to run "adop phase=abort" to abort the patching session. Seem to me that the issue came while fs_clone was not executed and "adop phase=prepare" was unable to screen out the port difference and sync it:

$NE_BASE/EBSapps/log/adop/5/.../cutover/validate/node1Name/ADOPValidations_detailed.log: 

-------------------------------------------------------------------------------------------------------------------

Lines #(51-55):

Checking the value of s_active_webport...

ERROR: The value of s_active_webport are different on RUN & PATCH Context files.

The Value present in RUN Context file = 4484

The Value present in PATCH Context file = 8041

Per Doc ID 2199585.1, s_active_webport (and s_login_page) in Context File on both RUN and PATCH File Systems shall be the same.

The fix is to run FS_CLONE first and then run the ADOP cycle.

B)  "perl adcfgclone.pl appsTier dualfs" failed in the half way of script running with error:  

[ Config tool : $FND_TOP/patch/115/bin/txkManageDBConnectionPool.pl ]

   Command      $FMW_HOME/webtier/perl/bin/perl -I $FMW_HOME/webtier/perl/lib/5.10.0 ... ...

   Action             : execute

SCRIPT RETURNED:

.end std out.

stty: standard input: Inappropriate ioctl for device

stty: standard input: Inappropriate ioctl for device

*******FATAL ERROR*******

PROGRAM : ($FND_TOP/patch/115/bin/txkManageDBConnectionPool.pl)

TIME    : Wed Mar 29 16:15:31 2022

FUNCTION: main::validateAppsSchemaCredentials [ Level 1 ]

ERRORMSG: Invalid APPS database user credentials. No updates Performed

.end err out.

The password was correct because I saw new entries were entered into database tables by the clone script. Then I tried to use "perl $AD_TOP/patch/115/bin/adProvisionEBS.pl ebs-delete-node ..." to clean it from database and got message saying primary node can not be deleted.

The fix is to run AutoConfig on database node, then just run  "perl adcfgclone.pl appsTier dualfs" again. 

R12.2 adautocfg.sh fails with AC-00444: Unable to write to file

Suddenly, R12.2 autoconfig failed with below message in log file $INST_TOP/admin/log/0X131006/adconfig.log:

   [INSTANTIATE PHASE]

  AutoConfig could not successfully instantiate the following files:

    Directory: $INST_TOP/admin/install

          adadmprf.sh             INSTE8_PRF

          adapperms.sh           INSTE8_APPLY

          adgendbc.sh             INSTE8_SETUP

  adgenjky.sh              INSTE8_SETUP

  admkpltfrm.sh          INSTE8_PRF

  admkxdir.sh             INSTE8_SETUP

  adsvcm.sh                INSTE8_APPLY

  afadmprf.sh             INSTE8_PRF

  afcpctx.sh                INSTE8_PRF

  afcpgsm.sh              INSTE8_PRF

  afcpnode.sh             INSTE8_SETUP

  afgcsreg.sh              INSTE8_SETUP

  ibywebprf.sh           INSTE8_PRF

  igccmprf.sh             INSTE8_PRF

  java.sh                    INSTE8

  jtfictx.sh                  INSTE8_PRF

  jtfsvfm.sh                INSTE8_APPLY

  okladmprf.sh           INSTE8_PRF

  shttimezone.sh        INSTE8_SETUP

  txkJavaMailerCfg.sh     INSTE8_PRF

  txkWebServicescfg.sh    INSTE8_PRF

    Directory: $INST_TOP/admin/scripts/msc

          mscmatlab.sh            INSTE8_PRF  

Seems those .sh files are randomly picked up. Log file has also message:

AC-00444: Unable to write to file: java.io.FileNotFoundException: $INST_TOP/admin/install/adadmprf.sh (Text file busy)

Raised by oracle.apps.ad.autoconfig.InstantiateProcess

After trying different ways to troubleshoot it, I suspected it is caused by something on the OS level. 

Finally, Linux Admin found it is CrowdStrike falcon-sensor, a security tool for monitoring.

$ ps -ef | grep falcon

root      1027        1  0 May1 ?        00:00:00 /opt/CrowdStrike/falcond

root      1028  1027  0 May1 ?        00:00:17 falcon-sensor

After disabled it and rebooted the server, adautocfg.sh worked successfully. 

We found it also made EBS Rapid Install for R12.2 upgrade fail with strange errors. I was told the bug in CrowdStrike was fixed later.

Post actions after R12.2 upgrade worked

After EBS instance is upgraded to R12.2.10, patches and other custom steps are necessary for it to meet business needs. Before adding more nodes to the instance, I did followings in my site.

1. Apply required R12.2 apps patches for database 19c upgrade (Section 2.1, Doc ID 2552181.1 - Interoperability Notes: Oracle E-Business Suite Release 12.2 with Oracle Database 19c).

The following 9 patches must exist in the environment before database is upgraded to 19c.

SELECT adb.bug_number, aat.name appl_top_name, adb.language, adb.creation_date,
decode(ad_patch.is_patch_applied('R12',aat.appl_top_id,adb.bug_number,adb.language),'EXPLICIT','APPLIED','NOT_APPLIED','NOT APPLIED') status
 FROM ad_bugs adb,
(select aat.name, aat.appl_top_id
from applsys.ad_appl_tops aat,
(select distinct fat.name from applsys.fnd_appl_tops fat) fat
where aat.name=fat.name ) aat
where adb.bug_number in (
'25452805', -- Applied
'26052406', -- Applied
'26521736', -- Need (19c interoperability patch for Release 12.2)
'30433124', -- Applied
'31088182', -- Need to apply
'31349591', -- Need to apply
'31800803', -- Need to apply
'30611319', -- Applied (only if using Integrated SOA Gateway (ISG))
'28732161'  -- Need (Release 12.2.6 or later versions)
) order by adb.bug_number,aat.name,adb.language;

In my instance, 4 patches were applied. Use downtime mode to apply 5 patches:

26521736
31088182
31349591

31800803
28732161

DOWNTIME mode will apply the patch to RUN file system.  It is not necessary to use downtime mode to apply them, but it saves time to run adop cut-over.

$ sh $ADMIN_SCRIPTS_HOME/adstpall.sh apps/apps_PWD

$ ps -ef | grep $LOGNAME

$ echo $FILE_EDITION

run

$ echo $PATCH_TOP 

$NE_BASE/EBSapps/patch

$ cd $PATCH_TOP

$ unzip p26521736_12.2.0_R12_LINUX.zip

$ unzip p31088182_R12.FND.C_R12_GENERIC.zip

$ unzip p31349591_R12.OTA.C_R12_GENERIC.zip

$ unzip p31800803_R12.TXK.C_R12_GENERIC.zip

$ unzip p28732161_R12.FRM.C_R12_GENERIC.zip

$ adop phase=apply apply_mode=downtime patches=26521736,31088182,31349591,31800803,28732161

Run the SQL statement again to confirm all 9 patches were applied by now. 

Before completing the patching cycle, which is required by R12.2 Clone script, I applied 5 more patches to the instance:

2. Two more patches:

Patch 31405457 (See Doc ID 2554156.1 for 19c database Export/Import):

$ unzip p31405457_R12_GENERIC.zip

Patch 32357755 to fix issue on changing sysadmin password (Doc ID 2743640.1 - FRM-40735: WHEN-VALIDATE-ITEM Trigger Raised Unhandled Exception ORA-29261 )

$ unzip p32357755_R12.FND.C_R12_LINUX.zip

$ adop phase=apply apply_mode=downtime patches=31405457,32357755

2. Four patches for 1099 tax asked by Business:

$ unzip p32072819_R12.AP.C_R12_GENERIC.zip

$ unzip p32157516_R12.AP.C_R12_GENERIC.zip
$ unzip p32181474_R12.AP.C_R12_GENERIC.zip

$ unzip p32266274_R12.AP.C_R12_GENERIC.zip

$ adop phase=apply apply_mode=downtime patches=32072819,32157516,32181474,32266274 

Confirm 6 patches were applied:

SQL> SELECT adb.bug_number, aat.name appl_top_name, adb.language, adb.creation_date,
decode(ad_patch.is_patch_applied('R12',aat.appl_top_id,adb.bug_number,adb.language),'EXPLICIT','APPLIED','NOT_APPLIED','NOT APPLIED') status
 FROM ad_bugs adb,
(select aat.name, aat.appl_top_id
from applsys.ad_appl_tops aat,
(select distinct fat.name from applsys.fnd_appl_tops fat) fat
where aat.name=fat.name ) aat
where adb.bug_number in (
'31405457',

'32357755',
'32072819',
'32157516',
'32181474',
'32266274'

) order by adb.bug_number,aat.name,adb.language;

3. Apply EBS CPU patches

Oracle E-Business Suite Release 12 Critical Patch Update Knowledge Document (January 2021) (Doc ID 2737201.1)

4. Install 3rd party application (such as Excel4app)

5. If a custom top is not necessary but an env variable is needed as a pointer to a folder under $APPL_TOP, modify file $FND_TOP/fndenv.env by adding lines:

#Begin Customizations

XXEDI_TOP=${APPL_TOP}/xxedi/12.0.0; export XXEDI_TOP

#End Customizations

Start a new OS session to see $XXEDI_TOP

$ echo $XXEDI_TOP

Since ADSplice adds lines to Forms config file $INST_TOP/ora/10.1.2/forms/server/default.env when creating custom top, manual modification on file default.env is not necessary in R12.2:

#Begin Customizations

XXEDI_TOP=/path/to/appl/xxedi/12.0.0

#End Customizations

6. Copy Custom Top folders to R12.2. Then re-link folders & executables, and compile custom forms. 

7. Custom OAF page development

Modify $FND_TOP/admin/template/custom/ebsProductManifest_xml.tmp to add the entry below for customprod.jar (after cu

After EBS instance is upgraded to R12.2.10, Patches and other custom steps are necessary for it to meet the business needs. I did followings in my site.

1. . Apply required R12.2 apps patches for database 19c upgrade (Section 2.1, Doc ID 2552181.1 - Interoperability Notes: Oracle E-Business Suite Release 12.2 with Oracle Database 19c).

The following 9 patches must exist in the environment before database is upgraded to 19c.

SELECT adb.bug_number, aat.name appl_top_name, adb.language, adb.creation_date,
decode(ad_patch.is_patch_applied('R12',aat.appl_top_id,adb.bug_number,adb.language),'EXPLICIT','APPLIED','NOT_APPLIED','NOT APPLIED') status
 FROM ad_bugs adb,
(select aat.name, aat.appl_top_id
from applsys.ad_appl_tops aat,
(select distinct fat.name from applsys.fnd_appl_tops fat) fat
where aat.name=fat.name ) aat
where adb.bug_number in (
'25452805', -- Applied
'26052406', -- Applied
'26521736', -- Need (19c interoperability patch for Release 12.2)
'30433124', -- Applied
'31088182', -- Need to apply
'31349591', -- Need to apply
'31800803', -- Need to apply
'30611319', -- Applied (only if using Integrated SOA Gateway (ISG))
'28732161'  -- Need (Release 12.2.6 or later versions)
) order by adb.bug_number,aat.name,adb.language;

In my instance, 4 patches were applied. Use downtime mode to apply 5 patches:

26521736
31088182
31349591

31800803
28732161

DOWNTIME mode will apply the patch to RUN file system.

 

* It is not necessary to use downtime mode to apply them, if FS_CLONE was executed after R12.2.10 patch.

$ sh $ADMIN_SCRIPTS_HOME/adstpall.sh apps/apps_PWD

$ ps -ef | grep $LOGNAME

$ echo $FILE_EDITION

run

$ echo $PATCH_TOP 

$NE_BASE/EBSapps/patch

$ cd $PATCH_TOP

$ unzip p26521736_12.2.0_R12_LINUX.zip

$ unzip p31088182_R12.FND.C_R12_GENERIC.zip

$ unzip p31349591_R12.OTA.C_R12_GENERIC.zip

$ unzip p31800803_R12.TXK.C_R12_GENERIC.zip

$ unzip p28732161_R12.FRM.C_R12_GENERIC.zip

$ adop phase=apply apply_mode=downtime patches=26521736,31088182,31349591,31800803,28732161

Run the SQL statement again to confirm all 9 patches were applied by now. 

Before completing the patching cycle, which is required by R12.2 Clone script, I applied 5 more patches to the instance:

2. Two more patches:

Patch 31405457 (See Doc ID 2554156.1 for 19c database Export/Import):

$ unzip p31405457_R12_GENERIC.zip

Patch 32357755 to fix issue on changing sysadmin password (Doc ID 2743640.1 - FRM-40735: WHEN-VALIDATE-ITEM Trigger Raised Unhandled Exception ORA-29261 )

$ unzip p32357755_R12.FND.C_R12_LINUX.zip

$ adop phase=apply apply_mode=downtime patches=31405457,32357755

2. Four patches for 1099 tax asked by Business:

$ unzip p32072819_R12.AP.C_R12_GENERIC.zip

$ unzip p32157516_R12.AP.C_R12_GENERIC.zip
$ unzip p32181474_R12.AP.C_R12_GENERIC.zip

$ unzip p32266274_R12.AP.C_R12_GENERIC.zip

$ adop phase=apply apply_mode=downtime patches=32072819,32157516,32181474,32266274 

Confirm 6 patches were applied:

SQL> SELECT adb.bug_number, aat.name appl_top_name, adb.language, adb.creation_date,
decode(ad_patch.is_patch_applied('R12',aat.appl_top_id,adb.bug_number,adb.language),'EXPLICIT','APPLIED','NOT_APPLIED','NOT APPLIED') status
 FROM ad_bugs adb,
(select aat.name, aat.appl_top_id
from applsys.ad_appl_tops aat,
(select distinct fat.name from applsys.fnd_appl_tops fat) fat
where aat.name=fat.name ) aat
where adb.bug_number in (
'31405457',

'32357755',
'32072819',
'32157516',
'32181474',
'32266274'

) order by adb.bug_number,aat.name,adb.language;

3. Apply 2021 January CPU patches

Oracle E-Business Suite Release 12 Critical Patch Update Knowledge Document (January 2021) (Doc ID 2737201.1)

(a) Apply EBS patch 32071646

SQL> select ad_patch.is_patch_applied('R12',-1,32071646) from dual;

AD_PATCH.IS_PATCH_APPLIED('R12',-1,32071646)

-------------------------------------------------------------------

NOT_APPLIED

SQL> SELECT adb.bug_number, aat.name appl_top_name, adb.language, adb.creation_date,

decode(ad_patch.is_patch_applied('R12',aat.appl_top_id,adb.bug_number,adb.language),'EXPLICIT','APPLIED','NOT_APPLIED','NOT APPLIED') status

 FROM ad_bugs adb,

(select aat.name, aat.appl_top_id

from applsys.ad_appl_tops aat,

(select distinct fat.name from applsys.fnd_appl_tops fat) fat

where aat.name=fat.name ) aat

where adb.bug_number in (

'32071646',

'32163187',

'32004048'

) order by adb.bug_number,aat.name,adb.language;

no rows selected

$ echo APPL_TOP

$ cd $PATCH_TOP

$ unzip p32071646_12.2.0_R12_LINUX.zip

$ adop phase=apply apply_mode=downtime patches=32071646

$ unzip p32163187_R12.FWK.C_R12_GENERIC.zip

$ unzip p32004048_R12.OKC.C_R12_GENERIC.zip

$ adop phase=apply apply_mode=downtime patches=32163187,32004048

... ...

Applying patch 32163187.

    Log:  $NE_BASE/EBSapps/log/adop/4/.../32163187/log/u32163187.log

Applying patch 32004048.

    Log: $NE_BASE/EBSapps/log/adop/4/.../32004048/log/u32004048.log

Running finalize actions for the patches being applied.

    Log: @ADZDSHOWLOG.sql "2021/07/13 12:05:43"

Running cutover actions for the patches being applied.

    Creating workers to process cutover DDL in parallel

... ... 

The apply phase completed successfully.

adop exiting with status = 0 (Success)

Run SQL statement again to confirm 3 patches are applied.

(b) WebLogic PSU patch 32052267 was applied as a ETCC requirement.

See Doc ID 1306505.1 for more on Oracle WebLogic Server PSUs (Patch Set Updates).

(c) Oracle Fusion Middleware 11.1.1.9 OSS - Web Tier Home

Patch 31304503 (OSS Security Patch Update CPUJul2020): applied

$ export ORACLE_HOME=$IAS_ORACLE_HOME

$ export PATH=$IAS_ORACLE_HOME/OPatch:$PATH

$  opatch lsinventory | grep 31304503

Patch  31304503     : applied on Sat Jun 26 12:07:37 EDT 2021

(d) Oracle Fusion Middleware 11.1.1.9 OHS - Web Tier Home

Patch 31047338 (OHS Security Patch Update CPUApr2020)

$ opatch lsinventory | grep -i 31047338

Patch  31047338     : applied on Sat Jun 26 12:20:11 EDT 2021

(e) Oracle Fusion Middleware 11.1.1.9 - Oracle Common Home

Patch 30368663 (Security Patch Update CPUOct2019)

Note: Patch 31985571 is a superset of patch 30368663 in Oct 2020

$ export ORACLE_HOME=$FMW_HOME/oracle_common

$ export PATH=$ORACLE_HOME/OPatch:$PATH

$ opatch lsinventory | grep 30368663

Patch  30368663     : applied on Sat Jun 26 12:31:37 EDT 2021

$ adop -status

Enter the APPS password:

Connected.

==================================================

ADOP (C.Delta.12)

Session Id: 4

Command: status

Output: $NE_BASE/EBSapps/log/adop/4/20210713_130234/adzdshowstatus.out

==================================================

Node Name       Node Type  Phase           Status          Started              Finished             Elapsed

--------------- ---------- --------------- --------------- -------------------- -------------------- ------------

node_name     master     APPLY           ACTIVE      2021/0X/12 10:50:40  2021/0X/13 12:55:03  26:04:23

                                       CLEANUP     NOT STARTED

4. Install 3rd party application (such as Excel4app)

5. If a custom top is not necessary but an env variable is needed as a pointer to a folder under $APPL_TOP, modify file $FND_TOP/fndenv.env by adding lines:

#Begin Customizations

XXEDI_TOP=${APPL_TOP}/xxedi/12.0.0; export XXEDI_TOP

#End Customizations

Start a new OS session to see $XXEDI_TOP

$ echo $XXEDI_TOP

Since ADSplice adds lines to Forms config file $INST_TOP/ora/10.1.2/forms/server/default.env when creating custom top, manual modification on file default.env is not necessary in R12.2:

#Begin Customizations

XXEDI_TOP=/path/to/appl/xxedi/12.0.0

#End Customizations

6. Copy Custom Top folders to R12.2. Then re-link folders & executables, and compile custom forms. 

7. Custom OAF page development

Notes:  key entry is saved in file $FND_TOP/admin/template/custom/ebsProductManifest_xml.tmp (after customall.jar):

8. Apply 3 AP patch to fix AP issues

$ export FORMS_PATH=$FORMS_PATH:$AU_TOP/forms/US:AU_TOP/resource

$ frmcmp_batch userid=apps/apps_PWD module=$AU_TOP/resource/IBYFDOBJ.pll output_file=$AU_TOP/resource/IBYFDOBJ.plx module_type=library batch=no compile_all=special

Forms 10.1 (Form Compiler) Version 10.1.2.3.0 (Production)

Forms 10.1 (Form Compiler): Release  - Production

Copyright (c) 1982, 2005, Oracle.  All rights reserved.

Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production

     With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options

PL/SQL Version 10.1.0.5.0 (Production)

Oracle Procedure Builder V10.1.2.3.0 - Production

Oracle Virtual Graphics System Version 10.1.2.0.0 (Production)

Oracle Multimedia Version 10.1.2.0.2 (Production)

Oracle Tools Integration Version 10.1.2.0.2 (Production)

Oracle Tools Common Area Version 10.1.2.0.2

Oracle CORE     10.1.0.5.0      Production

Compiling library IBYFDOBJ...

  Invalidating Package Spec IBY_PMTATTRIB......

  Invalidating Package Body IBY_PMTATTRIB......

  Compiling Package Spec IBY_PMTATTRIB......

  Compiling Package Body IBY_PMTATTRIB......

 Done.

NOTES: after compiled by frmcmp_batch, IBYFDOBJ.plx got newer file:

$ ls -al $AU_TOP/resource/IBYFDOBJ.plx

-rw-r--r-- 1 user group 40960 Mmm 20 14:01 IBYFDOBJ.plx

$ adident Header $AU_TOP/resource/IBYFDOBJ.plx

$Header IBYFDOBJ.pld 120.7.12020000.2 2012/07/12 15:13:21 sgogula ship $

$ adident Header $AU_TOP/resource/IBYFDOBJ.pll

$Header IBYFDOBJ.pld 120.7.12020000.2 2012/07/12 15:13:21 sgogula ship $

$ sqlplus  apps/apps_PWD

SQL> alter package IBY_DISBURSEMENT_COMP_PUB compile body;

Package body altered.

SQL> alter package IBY_DISBURSEMENT_COMP_PUB compile;

Package altered.

SQL> show parameter optimizer_capture_sql_plan_baselines

NAME                                                   TYPE        VALUE

----------------------------------------------- ----------- ------------------

optimizer_capture_sql_plan_baselines boolean     FALSE

$ unzip p32385168_R12.AP.C_R12_GENERIC.zip

$ adop phase=apply apply_mode=downtime patches=32385168

$ strings -a $AP_TOP/patch/115/sql/apaprvlb.pls |grep '$Header'

/* $Header: apaprvlb.pls 120.231.12020000.57 2021/02/24 09:40:39 anankitg ship $ */

$ unzip p32768426_R12.AP.C_R12_GENERIC.zip

$ unzip p31211521_R12.AP.C_R12_GENERIC.zip

$ adop phase=apply apply_mode=downtime patches=32768426,31211521

Confirm they are applied:

SQL> SELECT adb.bug_number, aat.name appl_top_name, adb.language, adb.creation_date,

decode(ad_patch.is_patch_applied('R12',aat.appl_top_id,adb.bug_number,adb.language),'EXPLICIT','APPLIED','NOT_APPLIED','NOT APPLIED') status

 FROM ad_bugs adb,

(select aat.name, aat.appl_top_id

from applsys.ad_appl_tops aat,

(select distinct fat.name from applsys.fnd_appl_tops fat) fat

where aat.name=fat.name ) aat

where adb.bug_number in (

'32385168',

'32768426',

'31211521'

) order by adb.bug_number,aat.name,adb.language;

32768426:R12.AP.C

---------------------------------

$AU_TOP/forms/US/APXINWKB.fmb 120.983.12020000.181

$AP_TOP/forms/US/APXINWKB.fmx 120.983.12020000.181

31211521:R12.AP.C

---------------------------------

$AP_TOP/patch/115/sql/apetxsrb.pls 120.223.12020000.38

Follow-up steps asked by Oracle:

1) Validate APPS schema using adadmin.

a. start adadmin

b. Pick Option 4 Maintain Applications Database Entities menu

c. Pick Validate APPS schema using adadmin (1).

2) Re-create grants and synonyms for APPS schema using adadmin.

a. Pick Option 4 Maintain Applications Database Entities menu

b. Pick Re-create grants and synonyms for APPS schema (2)

3) Generate product JAR files using adadmin.

a. Pick Option 1 Generate Applications Files menu

b. Pick Generate product JAR files (4, and answer Yes)

4) Reload JAR files to database using adadmin.

a. Pick Option 3 Compile/Reload Applications Database Entities menu

b. Pick Reload JAR files to database (4)

5) Compile all the INVALIDS using adadmin.

a. Pick Option 3 Compile/Reload Applications Database Entities menu

b. Pick Option 1 Compile APPS schema

6) Compile all the menus.

a. Pick Option 3 Compile/Reload Applications Database Entities menu

b. Pick Option 2 Compile menu information 

7) Compile all the ZX, IBY, XLA, PO, AP forms including libraries using adadmin.

a. Pick Option 1 Generate Applications Files menu

b. Pick Option 2 Generate form files

8)* Bounce the Database (Shutdown & Startup, optional)

9)* Start all the application services (adstrtal.sh apps/<apps password>).

10)* Submit the concurrent request "Gather Schema Statistics" with Schema Name: ALL and Estimate Percent: 20% Atleast and wait for the request to complete.

11)* Retest the issue.

* They can be deferred till configuration on all nodes is completed.

9. Verify database parameters for ADOP:

SQL> show parameter local_listener

NAME                       TYPE      VALUE

--------------------- ---- ----------- ------------------------------

local_listener             string       EBSDEV_LOCAL

SQL> show parameter service_names

NAME                       TYPE      VALUE

-------------------------- ----------- ------------------------------

service_names            string       EBSDEV, ebs_patch,EBSDEV_ebs_patch

SQL> select SERVICE_ID,NAME from dba_services where name='ebs_patch';

SERVICE_ID  NAME

-----------------  ----------------------------------------------------------------

                    6   ebs_patch

Notes: This line may got lost somehow. If so, add it (by "alter system ...").

SQL> select * from dba_editions;

EDITION_NAME PARENT_EDITION_NAME USABLE

---------------------- ------------------------------------ ------------

ORA$BASE                                                             YES

$ adop -status

Enter the APPS password:

Connected.

=======================================================

ADOP (C.Delta.12)

Session Id: 4

Command: status

Output: $NE_BASE/EBSapps/log/adop/4/20210X14_130108/adzdshowstatus.out

========================================================

Node Name       Node Type  Phase           Status          Started              Finished             Elapsed

--------------- ---------- --------------- --------------- -------------------- -------------------- ------------

$EBS_HOSTNAME        master     APPLY           ACTIVE          2021/0X/12 10:50:40  2021/0X/13 12:55:03  26:04:23

                                                          CLEANUP    NOT STARTED

10. Run FS_CLONE (when EBS Services are running) before adding/cloning 2 node

$ adop phase=cleanup   (or, adop phase=cleanup cleanup_mode=full )

Enter the APPS password:

Enter the SYSTEM password:

Enter the WLSADMIN password:

Validating credentials.

... ...

    Patch file system free space: 49.76 GB

Validating system setup.

    Node registry is valid.

Checking for existing adop sessions.

===============================================

ADOP (C.Delta.12)

Session ID: 4

Node: $EBS_HOSTNAME

Phase: cleanup

Log: $NE_BASE/EBSapps/log/adop/4/20210X14_130309/adop.log

===============================================

Processing cleanup actions in parallel.

    Log: $NE_BASE/EBSapps/log/adop/4/20210X14_130309/cleanup/$EBS_HOSTNAME/log/cleanup.log

Generating report of queued DDL actions.

    Output: $NE_BASE/EBSapps/log/adop/4/20210X14_130309/cleanup/$EBS_HOSTNAME/adzdallddls.out

Running cleanup in STANDARD mode.

    Log: @ADZDSHOWLOG.sql "2021/0X/14 13:05:32"

Generating log report.

    Output: $NE_BASE/EBSapps/log/adop/4/20210X14_130309/cleanup/$EBS_HOSTNAME/adzdshowlog.out

The cleanup phase completed successfully.

adop exiting with status = 0 (Success)

$ adop -status

Enter the APPS password:

Connected.

========================================================

ADOP (C.Delta.12)

Session Id: 4

Command: status

Output: $NE_BASE/EBSapps/log/adop/4/20210X14_131226/adzdshowstatus.out

========================================================

Node Name               Node Type  Phase           Status          Started              Finished             Elapsed

-------------------------- --------------- --------------- -------------------- -------------------- ------------

$EBS_HOSTNAME    master     APPLY           COMPLETED     2021/0X/12 10:50:40  2021/0X/13 12:55:03  26:04:23

                                                   CLEANUP     COMPLETED     2021/0X/14 13:04:04  2021/0X/14 13:05:54  0:01:50

$ echo $FILE_EDITION

run

NOTES: fs_clone will delete folders from fs2 (PATCH file system) and then copy folders from fs1 (RUN file system) to replace them. If it failed in the middle, some folders or files may be missed on fs2.

$ adop phase=fs_clone    (or, adop phase=fs_clone force=yes)

                                   <== it may take up to 2 hours

Enter the APPS password:

Enter the SYSTEM password:

Enter the WLSADMIN password:

Validating credentials.

Initializing.

... ...

    Patch file system free space: 49.75 GB

Validating system setup.

    Node registry is valid.

Checking for existing adop sessions.

    No pending session exists.

    Starting new adop session.

==========================================================

ADOP (C.Delta.12)

Session ID: 5

Node: $EBS_HOSTNAME

Phase: fs_clone

Log: $NE_BASE/EBSapps/log/adop/5/20210X14_131409/adop.log

==========================================================

Validating configuration on node: [$EBS_HOSTNAME].

    Log: $NE_BASE/EBSapps/log/adop/5/20210X14_131409/fs_clone/validate/$EBS_HOSTNAME

        [WARNING]: There could be issues while validating the ports used for E-Business Suite instance against ports used in /etc/services. Refer the log file for more details.

Checking for pending cleanup actions.

    No pending cleanup actions found.

Blocking managed server ports.

    Log: $NE_BASE/EBSapps/log/adop/5/20210X14_131409/fs_clone/$EBS_HOSTNAME/txkCloneAcquirePort.log

Performing CLONE steps.

    Log: $NE_BASE/EBSapps/log/adop/5/20210X14_131409/fs_clone/$EBS_HOSTNAME

Beginning application tier FSCloneStage - Wed Mmm 14 13:17:31 2021

$COMMON_TOP/util/jdk32/bin/java -Xmx600M -Doracle.jdbc.autoCommitSpecCompliant=false -DCONTEXT_VALIDATED=false -Doracle.installer.oui_loc=/oui -classpath ... ...

Log file located at $INST_TOP/admin/log/clone/FSCloneStageAppsTier_0X141317.log

Completed FSCloneStage...

Wed Jul 14 13:24:27 2021

Beginning application tier FSCloneApply - Wed Mmm 14 13:26:59 2021

... ...

Completed FSCloneApply...

Wed Mmm 14 13:52:59 2021

Resetting FARM name...

runDomainName: EBS_domain_$TWO_TASK

patchDomainName: EBS_domain

targets_xml_loc: $FMW_HOME/user_projects/domains/EBS_domain/sysman/state/targets.xml

FARM name reset is successful.

Releasing managed server ports.

    Log: $NE_BASE/EBSapps/log/adop/5/20210X14_131409/fs_clone/$EBS_HOSTNAME/txkCloneAcquirePort.log

Synchronizing snapshots.

Stopping services on patch file system.

    Stopping admin server.

You are running adadminsrvctl.sh version 120.10.12020000.11

Stopping WLS Admin Server...

Refer $PATCH_BASE/inst/apps/$CONTEXT_NAME/logs/appl/admin/log/adadminsrvctl.txt for details

AdminServer logs are located at $PATCH_BASE/FMW_Home/user_projects/domains/EBS_domain/servers/AdminServer/logs

adadminsrvctl.sh: exiting with status 0

adadminsrvctl.sh: check the logfile $PARCH_BASE/inst/apps/$CONTEXT_NAME/logs/appl/admin/log/adadminsrvctl.txt for more information ...  

    Stopping node manager.

You are running adnodemgrctl.sh version 120.11.12020000.12

The Node Manager is already shutdown

NodeManager log is located at $PATCH_BASE/FMW_Home/wlserver_10.3/common/nodemanager/nmHome1

adnodemgrctl.sh: exiting with status 2

adnodemgrctl.sh: check the logfile $LOG_HOME/appl/admin/log/adnodemgrctl.txt for more information ...  

Generating log report.

    Output: $NE_BASE/EBSapps/log/adop/5/20210X14_131409/fs_clone/$EBS_HOSTNAME/adzdshowlog.out

The fs_clone phase completed successfully.

adop exiting with status = 0 (Success)

$ adop -status

Enter the APPS password:

Connected.

=========================================================

ADOP (C.Delta.12)

Session Id: 5

Command: status

Output: $NE_BASE/EBSapps/log/adop/5/20210X14_140211/adzdshowstatus.out

=========================================================

Node Name              Node Type  Phase           Status                    Started                          Finished             Elapsed

------------------------ -------------- ----------------- ------------------ --------------------- ------------

$EBS_HOSTNAME    master     FS_CLONE   COMPLETED   2021/0X/14 13:15:26  2021/0X/14 14:53:11  1:37:45

File System Synchronization Type: Full

adop exiting with status = 0 (Success)

Now, it can be cloned to 2nd node.