FND_OAM_CONTEXT_FILES misses a row of context file

I got below error from using ADOP to apply patches:

*******FATAL ERROR*******

PROGRAM : ($AD_TOP/patch/115/bin/txkADOPEvalSrvStatus.pl)

TIME    : Thu Feb 24 12:26:26 2022

FUNCTION: TXK::XML::load_doc [ Level 1 ]

MESSAGES:

error = Cannot open XML file for load

errorno = No such file or directory

file = $NE_BASE/EBSapps/log/adop/.../apply/node_name/TXK_EVAL_apply_.../ctx_files/$CONTEXT_NAME.xml

STACK TRACE

 at $AU_TOP/perl/TXK/Error.pm line 168

        TXK::Error::abort('TXK::Error', 'HASH(0x3d523b0)') called at $AU_TOP/perl/TXK/XML.pm line 280

        TXK::XML::load_doc('TXK::XML=HASH(0x3d49840)', 'HASH(0x3d1c698)') called at $AU_TOP/perl/TXK/XML.pm line 266

        TXK::XML::loadDocument('TXK::XML=HASH(0x3d49840)', 'HASH(0x3d1c698)') called at $AD_TOP/patch/115/bin/txkADOPEvalSrvStatus.pl line 1238

... ...        

The error and logs do not tell what is the problem. But when I ran ADOP validation, it gave me direction to find the root cause.

$ adop -validate

... ... 

    [UNEXPECTED] Could not find patch context file from database

    [UNEXPECTED]Remote action failed.

... ...

"adop phase=fs_clone" can hit the same error:

   Verifying existence of context files in database.

       [UNEXPECTED] Could not find patch context file from database

       UNEXPECTED]Patch edition context file not found in database for host node_NAME

1. What was the problem?

Most likely "EXEC FND_CONC_CLONE.SETUP_CLEAN;" was executed during a clone or somewhere, which deleted rows in tables:

SQL> select * from fnd_nodes;

SQL> select * from fnd_oam_context_files;

I used query to confirm the row for PATCH context file of the newly added node was not in table FND_OAM_CONTEXT_FILES:

SQL> select distinct(PATH) from FND_OAM_CONTEXT_FILES 

where NAME not in ('TEMPLATE','METADATA','config.txt') and CTX_TYPE='A' 

and (status is null or upper(status) in ('S','F')) and EXTRACTVALUE(XMLType(TEXT),'//file_edition_type') = 'patch';

2. Fix: run AutoConfig on PATCH file system

a) Disable a trigger

SQL> conn system/systemPWD

Connected.

SQL> alter trigger ebs_logon disable;

Trigger altered.

b) Set PATCH env and run autoConfig

$ . /<EBS_HOME_BASE>/EBSapps.env patch

$ echo $TWO_TASK

EBSDEV_patch

$ sqlplus apps/appsPWD   -- modify tnsnames.ora if it failed

SQL> show user

USER is "APPS"

SQL> exit

$ cd $ADMIN_SCRIPTS_HOME

$ echo $FILE_EDITION

patch

$ ./adautocfg.sh

Enter the APPS user password:

The log file for this session is located at: $INST_TOP/admin/log/MMDDHHMI/adconfig.log

... ...

AutoConfig completed successfully.

c). Enable the trigger

SQL> conn system/systemPWD

Connected.

SQL> alter trigger ebs_logon enable;

Trigger altered.

d). Confirm two rows in FND_OAM_CONTEXT_FILES for patch filesystem now

SQL> connect apps/appsPWD

SQL> select distinct(PATH) from FND_OAM_CONTEXT_FILES 

where NAME not in ('TEMPLATE','METADATA','config.txt') and CTX_TYPE='A' 

and (status is null or upper(status) in ('S','F')) 

and EXTRACTVALUE(XMLType(TEXT),'//file_edition_type') = 'patch';

AutoConfig also inserted one row to table applsys.FND_APPL_TOPS.

3. Run ADOP again from the beginning. All shall work.

Start a new OS session 

$ echo $FILE_EDITION

run

$  adop -validate

Enter the APPS password:

Enter the SYSTEM password:

Enter the WLSADMIN password:

.. ...

adop exiting with status = 0 (Success)

Notes: 

1. After AutoConfig on patch filesystem, it may bring a wrong login settings to the site. You will have to run AutoConfig in RUN filesystem to overwrite them.

2. If this problem was from "adop phase=fs_clone", run it again by "adop phase=fs_clone force=yes" to start it from the beginning to avoid the Apache failure (because some wrong info was saved in adop from previous run).

Unable to initialize SSL environment, nzos call nzosSetCredential returned 28791

Another possible way to fix the problem:

$ . ./EBSapps.env patch

$ echo $FILE_EDITION

patch

$ echo $PATCH_BASE

<EBS_HOME_BASE>/fs2

$ $ADJVAPRG oracle.apps.ad.autoconfig.oam.CtxSynchronizer \

        action=upload \ contextfile=/path on fs2/to/$CONTEXT_FILE

        logfile=/path/to/patchupload.log

REFERENCE:  Doc ID 2090393.1 (Error "Use of uninitialized value $result in split" In File txkADOPValidations.error When Running fs_clone)