Monday, March 22, 2021

12.2.10 patch 30399999 failed on POS_SUPPLIER_SEARCH_INDEX.sql

After all pre-steps were completed and conditions met the requirement, I downloaded zip files for 12.2.10 patch 30399999 and unzipped to folder /path/to/R12210_Patches. Command line to apply it:

$ adop phase=apply apply_mode=downtime patches=30399999 patchtop=/path/to/R12210_Patches 

ADOP failed and exited with error after ran for hours:

You should check the file
$NE_BASE/EBSapps/log/adop/.../apply/node_name/30399999/log/u30399999.log
for errors.
 [UNEXPECTED]Error occurred executing "adpatch workers=16 options=hotpatch console=no interactive=no defaultsfile=/path/to/R12210_Patches//30399999 driver=u30399999.drv logfile=u30399999.log"
[UNEXPECTED]Refer to the log files for more information.
[UNEXPECTED]Apply phase has failed.
[STATEMENT] Please run adopscanlog utility, using the command

"adopscanlog -latest=yes"
to get the list of the log files along with snippet of the error message corresponding to each log file.
adop exiting with status = 1 (Fail)

Check adwork001.log file and see the real error:

sqlplus -s APPS/***** @$AD_TOP/patch/115/sql/adsqlwrapper.sql '$POS_TOP/patch/115/sql/POS_SUPPLIER_SEARCH_INDEX.sql '
Connected.
PL/SQL procedure successfully completed.

DECLARE
*
ERROR at line 1:
ORA-20000: possearchindex.sql(500): ORA-20000: Exception at
POS_SUPPLIER_SEARCH_INDEX_PKG.create_index(1800): ORA-20000: Exception at
POS_SUPPLIER_SEARCH_INDEX_PKG.create_index(1800): ORA-29855: error occurred in
the execution of ODCIINDEXCREATE routine
ORA-20000: Oracle Text error:
DRG-10754: Size (in bytes) must be between 1024 and 52428800
ORA-06512: at line 56

Time when worker failed: Fri Sep 24 2021 13:56:21

SQL> select index_name, table_name, index_type, status, domidx_opstatus 
from all_indexes where domidx_opstatus != 'VALID';

        Control
Worker  Code      Context            Filename                    Status
------  --------  -----------------  --------------------------  --------------
     1  Done      AutoPatch R120     POS_SUPPLIER_SEARCH_INDEX.  FAILED


At this stage, we do not want to make any changes to the instance. The only option is to skip this SQL code and then to re-start the adop session. 

Fortunately adctrl still shows the workers' status, while ADOP session already exited by the error. I choose option #8 to skip the failed worker.

After that, I ran below line to re-start from where it failed and it completed:
$ adop phase=apply apply_mode=downtime patches=30399999 patchtop=/path/to/R12210_Patches restart=yes

$ vi u30399999.log
... ...
Worker process 15 started.
Worker process 16 started.

Checking if all jobs have their actual and symbolic arguments in sync....
Done.
Reading jobs from FND_INSTALL_PROCESSES table ...
    Fixed: file POS_SUPPLIER_SEARCH_INDEX.sql on worker  1 for product pos username POS.
Time is: Fri Sep 24 2021 16:22:18

Done reading jobs from FND_INSTALL_PROCESSES table ...
Telling workers to read 'todo' restart file.
Done.
Completed: file POS_SUPPLIER_SEARCH_INDEX.sql on worker  1 for product pos username POS.
Time is: Fri Sep 24 2021 16:22:18
Connecting to APPS......Connected successfully.

Even u30399999.log says file POS_SUPPLIER_SEARCH_INDEX.sql was completed. But the worker log shows it was skipped. Index table still shows a failure as well.
$ vi adwork001.log
... ...
Ready to run jobs.
Setting parallel context: PARALLEL AUTOPATCH AT R120
Restarting job that failed - will skip a single step.
Time when worker restarted job: Fri Sep 24 2021 16:22:18
Skipping step POS_SUPPLIER_SEARCH_INDEX.sql.
Step skipped at time: Fri Sep 24 2021 16:22:18

Time when worker completed job: Fri Sep 24 2021 16:22:18

SQL> select index_name, table_name, index_type, status, domidx_opstatus 
from all_indexes where domidx_opstatus != 'VALID';
INDEX_NAME                                 TABLE_NAME                            INDEX_TYPE  STATUS   DOMIDX_OPSTATUS
---------------------------------------------  --------------------------------------------   ------------------  ------------   ----------------------------
POS_SUPPLIER_SEARCH_INDEX POS_SUPPLIER_ENTITY_DATA   DOMAIN       VALID      FAILED

I believe our instance does not use this index and just ignored it. I do not know exactly what causes the failure on creating the index. I believe it relates to the 0 value on two database init parameters:
SQL> show parameter PGA_AGGREGATE_TARGET;
NAME                             TYPE        VALUE
----------------------------- ----------- ------------------------------
pga_aggregate_target     big integer     0          (Note: 0 means unlimited ) 

SQL> show parameter SGA_TARGET;
NAME                                 TYPE        VALUE
------------------------------ ----------- ------------------------------
sga_target                        big integer     0


No comments: