Use zip to address Apache log4j vulnerabilities

It was from an Oracle document by using zip/unzip to remove Java class from a JAR file. I put steps to a script for a quick execution. Notes: Oracle soon updated the document to use a patch to address this vulnerability, instead. 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

# CVE-2021-44228 Advisory for Oracle E-Business Suite (Apache log4j Vulnerabilities)

# (Doc ID 2827804.1)

# In R12.2.10 instances only

#

jars="$FND_TOP/java/3rdparty/stdalone/log4j_core.jar $COMMON_TOP/java/lib/log4j_core.jar"

echo "Before change"

# check the JAR files exist

for jar in $jars ;do ls -l $jar ;done

# verify whether the log4j JARs contain the affected JndiLookup class:

for jar in $jars ;do unzip -l $jar org/apache/logging/log4j/core/lookup/JndiLookup.class ;done

# backup file

mv $FND_TOP/java/3rdparty/stdalone/log4j_core.jar $FND_TOP/java/3rdparty/stdalone/log4j_core.jar.bak

cp $FND_TOP/java/3rdparty/stdalone/log4j_core.jar.bak $FND_TOP/java/3rdparty/stdalone/log4j_core.jar

echo "After backup"

ls -al $FND_TOP/java/3rdparty/stdalone/log4j_core.jar*

echo "After change"

# remove the JndiLookup class from the JAR files

for jar in $jars ;do zip -q -d $jar org/apache/logging/log4j/core/lookup/JndiLookup.class ;done

# verify that the JAR files have been recently modified and that their size has become smaller:

for jar in $jars ;do ls -l $jar ;done

# verify that the JndiLookup classes are no longer present:

for jar in $jars ;do unzip -l -q $jar org/apache/logging/log4j/core/lookup/JndiLookup.class ;done

echo DONE 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Check the result:

$ cd $FND_TOP/java/3rdparty/stdalone

$ ls -altr log4j_core.jar*

-rwxr-xr-x 1 users group 3887706 Apr 30  2021 log4j_core.jar.bak

-rwxr-xr-x 1 users group         624 Dec 21  2021 log4j_core.jar

Notes: unzip can be used to list classes of a .jar file:

$ unzip -l log4j_core.jar

Archive:  log4j_core.jar

$Header: log4j_core.jar 120.0.12020000.2 2021/12/14 22:42  srkumma $

  Length     Date   Time    Name

 --------    ----   ----    ----

      114  12-14-21 22:22   META-INF/JRIMETA.DAT

      186  12-14-21 22:02   README_log4j_core.txt

 --------                   -------

      300                   2 files

How to find the expiration date of certificate in EBS R12.2

Steps to find the expiration date of certificate file $NE_BASE/inst/$CONTEXT_NAME/certs/Apache/cwallet.sso

$ cd $NE_BASE/inst/$CONTEXT_NAME/certs/Apache

$ alias orapki=$FMW_HOME/oracle_common/bin/orapki

$ orapki wallet display -wallet ./cwallet.sso     

                                                          -- Note: file cwallet.sso does not ask for the password

Requested Certificates:

User Certificates:

Subject:        CN=siteName.domian.com,O=CompanyName,L=cityName,ST=AZ,C=US

Trusted Certificates:

Subject:        CN=DigiCert High Assurance EV Root CA,OU=www.digicert.com,O=DigiCert Inc,C=US

Subject:        CN=CompanyName Secure CA2,O=ComanyName,C=US

$ orapki wallet export -wallet ./cwallet.sso -dn "CN=siteName.domian.com,O=CompanyName,L=cityName,ST=AZ,C=U" -cert siteName_certs.cer

$ orapki cert display -cert ./siteName_certs.cer -summary

Subject:       CN=siteName.domian.com,O=CompanyName,L=cityName,ST=AZ,C=US  

Issuer:         CN=CompanyName Secure CA2,O=ComanyName,C=US

Valid Until:    Fri Jul 16 19:59:59 EDT 2021

How to start a R12.2 OHS (Apache) after ssl certificate expired

When ssl certificate expired (or something is wrong in the certificate .sso file),  Apache will not start by adapcctl.sh in R12.2. Possible error message in adapcctl.txt (or adopmnctl.txt) under $LOG_HOME/appl/admin/log:

[opmn] [ERROR:1] [] [internal] $FMW_HOME/webtier/opmn/bin/opmn: unexpected exit: status 4200

opmnctl start: opmn failed to start.

ias-component/process-type/process-set:  EBS_web/OHS/OHS/

Error

--> Process (index=1,uid=1246640827,pid=29336)

  failed to start a managed process after the maximum retry limit

  Log: $FMW_HOME/webtier/instances/EBS_web_OHS1/diagnostics/logs/OHS/EBS_web/console~OHS~1.log

... ... :: adapcctl.sh: exiting with status 204

$FMW_HOME/webtier/instances/EBS_web_OHS1/diagnostics/logs/OHS/EBS_web/EBS_web.log may give errors from starting Apache:

[OHS] [ERROR:32] [] [core.c] [host_id: node_name.domain.com] [host_addr: 167.69.xx.xx] [pid: 1851] [tid: 139696124196736] [user: applmgr] [VirtualHost: site_name.domain.com:0] Init: (site_name.domain.com:443) Unable to initialize SSL environment, nzos call nzosSetCredential returned 28791

[OHS] [ERROR:32] [] [core.c] [host_id: node_name.domain.com] [host_addr: 167.69.xx.xx] [pid: 1851] [tid: 139696124196736] [user: applmgr] [VirtualHost: site_name.domain.com:0] NZ Library Error: Unknown error

"adopmnctl.sh start" can be used to get more error message in $FMW_HOME/webtier/instances/EBS_web_OHS1/diagnostics/logs/OPMN/opmn/opmn.log, such as  errors (that matches Doc ID 2676628.1):

[opmn] [ERROR:1] [] [ons-secure] Connection server SSL set credentials failed (28791)

[opmn] [ERROR:1] [222] [ons-secure] SSL initialization failed

Note: If cert file .../config/OPMN/opmn/wallet/cwallet.sso is a wrong file, "adapcctl.sh start" will fail quickly and may give misleading error in adapcctl.txt:

[opmn] [ERROR:1] [] [internal] $FMW_HOME/webtier/opmn/bin/opmn: unexpected exit: status 4200

opmnctl start: opmn failed to start.

So its first webpage and so the entire EBS site is not accessible. The error messages in the log do not point out the real problem. Most likely (but not 100% true), the cert expired. 

I tried to renew the cert file. But if the new one did not make Apache start, it is impossible to tell whether CA software webpage (such as Venafi) gave me a valid cert file. At that situation, there is no good way to test the cert renewal. It became an urgent problem. 

The solution is to create a temporary cert to bring the site up.  Doc ID 2555355.1 (Prerequisite Steps to Configure Oracle Fusion Middleware 11.1.1.9 Components for Oracle E-Business Suite Release 12.2 Before Applying the July 2019 and Later FMW OSS Security Patch) gives steps for creating a temporary cert file. I had to keep Admin Server ("adadminsrvctl.sh start") up during this process in R12.2.10.

First of all, make sure to use the right orapki:

$ alias orapki=$FMW_HOME/oracle_common/bin/orapki

$ cd /u01/app/temp

$ mkdir ss

$ cd ss

Create a new wallet with an acceptable self-signed certificate in /u01/app/temp/ss:

$ orapki wallet create -wallet ./ -auto_login_only

Oracle PKI Tool : Version 11.1.1.9.0

Copyright (c) 2004, 2015, Oracle and/or its affiliates. All rights reserved.

$ orapki wallet add -wallet . -dn "CN=FMWSmallCircleOfTrust" -asym_alg RSA -keysize 2048 -sign_alg sha256 -self_signed -validity 3652 -auto_login_only

Oracle PKI Tool : Version 11.1.1.9.0

Copyright (c) 2004, 2015, Oracle and/or its affiliates. All rights reserved.

$ orapki wallet display -wallet .               <== to verify/see the new wallet

Oracle PKI Tool : Version 11.1.1.9.0

Copyright (c) 2004, 2015, Oracle and/or its affiliates. All rights reserved.

Requested Certificates:

User Certificates:

Subject:        CN=FMWSmallCircleOfTrust

Trusted Certificates:

Subject:        CN=FMWSmallCircleOfTrust

Define useful OS variable $iName for next steps:

$ tr < $CONTEXT_FILE '<>' '  ' | awk '/"s_ohs_instance"/ {print $(NF-1)}'

EBS_web_OHS1

$ iName=$(tr < $CONTEXT_FILE '<>' '  ' | awk '/"s_ohs_instance"/ {print $(NF-1)}' )

$ pwd

/u01/app/temp/ss

$ cd $FMW_HOME/webtier/instances/$iName

$ pwd

$FMW_HOME/webtier/instances/EBS_web_OHS1

$ find . -name cwallet.sso                   <== to find cwallet.sso is used in 3 locations

./config/OPMN/opmn/wallet_ORIG/cwallet.sso

./config/OPMN/opmn/wallet/cwallet.sso

./config/OHS/EBS_web/keystores/default/cwallet.sso

./config/OHS/EBS_web/proxy-wallet/cwallet.sso

Back up the existing cwallet.sso and replace it by the temporary cert file in 3 locations.

$ find . -name cwallet.sso | fgrep -v /webgate/ | while read w ; do echo $w; cp -p /u01/app/temp/ss/cwallet.sso $w ; done

./config/OPMN/opmn/wallet/cwallet.sso

./config/OHS/EBS_web/keystores/default/cwallet.sso

./config/OHS/EBS_web/proxy-wallet/cwallet.sso

$ ls -al ./config/OPMN/opmn/wallet

total 12

-rw------- 1 user group 3853 Oct  1 16:44 cwallet.sso

-rw------- 1 user group 4365 May 28 15:53 cwallet.sso_BK_1001

-rw------- 1 user group    0 May 28 15:53 cwallet.sso.lck

$ ls -al

drwx------ 3 user group 17 May 28 15:54 auditlogs

drwx------ 2 user group 21 May 28 15:53 bin

drwx------ 4 user group 29 May 28 15:53 config

drwx------ 3 user group 18 May 28 15:53 diagnostics

drwx------ 3 user group 21 May 28 15:53 OHS

drwx------ 3 user group 23 Jun 14 03:24 tmp

$ find . -name cwallet.sso | fgrep -v /webgate/ | while read w ; do echo -e "\n$w"; orapki wallet display -nologo -wallet $w ; done

./config/OPMN/opmn/wallet_ORIG/cwallet.sso

Requested Certificates:

User Certificates:

Subject:        CN=Self-Signed Certificate for EBS_web_OHS1\20,OU=OAS,O=ORACLE,L=REDWOODSHORES,ST=CA,C=US

Trusted Certificates:

Subject:        CN=Self-Signed Certificate for EBS_web_OHS1\20,OU=OAS,O=ORACLE,L=REDWOODSHORES,ST=CA,C=US

./config/OPMN/opmn/wallet/cwallet.sso

Requested Certificates:

User Certificates:

Subject:        CN=FMWSmallCircleOfTrust

Trusted Certificates:

Subject:        CN=FMWSmallCircleOfTrust

./config/OHS/EBS_web/keystores/default/cwallet.sso

Requested Certificates:

User Certificates:

Subject:        CN=FMWSmallCircleOfTrust

Trusted Certificates:

Subject:        CN=FMWSmallCircleOfTrust

./config/OHS/EBS_web/proxy-wallet/cwallet.sso

Requested Certificates:

User Certificates:

Subject:        CN=FMWSmallCircleOfTrust

Trusted Certificates:

Subject:        CN=FMWSmallCircleOfTrust

Re-register OHS and its new certificate with Fusion Middleware Control. Seems to me this step is necessary (while I do not know what it really does).

$ aHost=$( tr < $CONTEXT_FILE '<>' '  ' | awk '/"s_wls_admin_host"/ {print $(NF-1)}' )

$ aPort=$( tr < $CONTEXT_FILE '<>' '  ' | awk '/"s_wls_adminport"/ {print $(NF-1)}' )

$ aUser=$( tr < $CONTEXT_FILE '<>' '  ' | awk '/"s_wls_admin_user"/ {print $(NF-1)}' )

$ echo $aHost

node_name

$ echo $aPort

7032

$ echo $aUser

weblogic

$ cd $FMW_HOME/webtier/instances/$iName/bin

$ ./opmnctl unregisterinstance -adminHost $aHost -adminPort $aPort -adminUsername $aUser -instanceName $iName

Command requires login to weblogic admin server (node_name):

  Username: weblogic

  Password:

Unregistering instance

Command succeeded.

$ ./opmnctl registerinstance -adminHost $aHost -adminPort $aPort -adminUsername $aUser

Command requires login to weblogic admin server (node_name ):

  Username: weblogic

  Password:

Registering instance

Command succeeded.

I logged onto EM site at http://node_name.domain.com:7032/em (vs. Console) and saw OHS was still down (Somehow, my EM always shows Web Tier sites are down), and adopmnctl.sh reported OHS in Down status.

Now, when I ran adapcctl.sh, it started Apache successfully and the webpage worked in "unsafe" mode!  Then, I shutdown everything and ran an autoConfig before started all EBS services. 

$ cd $ADMIN_SCRIPTS_HOME

$ ./adapcctl.sh start

$ ./adopmnctl.sh status

You are running adopmnctl.sh version 120.0.12020000.2

Checking status of OPMN managed processes...

Processes in Instance: EBS_web_OHS1

----------------------+--------------------+---------+---------

ias-component | process-type | pid | status

----------------------+--------------------+---------+---------

EBS_web        | OHS              | 1166 | Alive

With that, I had the R12.2.10 site available to test the certificate renewal and got the expired cert renewed after replacing the temporary cert.

TROUBLESHOOTING

If Apache does still not start, check console~OHS~1.log under  $FMW_HOME/webtier/instances/$iName/diagnostics/logs/OHS/EBS_xxx

If Apache started in Alive status, but the login webpage is still not available, below line shall return "connected"

$ wget http://node_name.domain.com:s_webport

If the login page shows ERR_SSL_PROTOCOL_ERROR, most likely some parameter in .xml file(s) for enabling TLS 1.2 is wrong. 

If the login page shows ERR_CONNECTION_RESET, one of the possibilities is F5 listens to a wrong port if F5 is used in company network. It shall listen to s_webssl_port. Also check Oracle Doc ID 2771703.1 for other possible causes.

Notes: ADOP will not automatically copy cert file cwallet.sso from RUN file system to PATCH file system. You have to modify adop_sync.drv located under $APPL_TOP_NE/ad/custom to include the followings:

#Oracle HTTP Server Wallet - cwallet.sso

rsync -zr %s_current_base%/FMW_Home/webtier/instances/%s_ohs_instance%/config/OHS/%s_ohs_component%/keystores/default/cwallet.sso %s_other_base%/FMW_Home/webtier/instances/%s_ohs_instance%/config/OHS/%s_ohs_component%/keystores/default/cwallet.sso

#OPMN Wallet - cwallet.sso

rsync -zr %s_current_base%/FMW_Home/webtier/instances/%s_ohs_instance%/config/OPMN/opmn/wallet/cwallet.sso %s_other_base%/FMW_Home/webtier/instances/%s_ohs_instance%/config/OPMN/opmn/wallet/cwallet.sso

rsync -zr %s_current_base%/FMW_Home/webtier/instances/%s_ohs_instance%/config/OHS/%s_ohs_component%/proxy-wallet/cwallet.sso %s_other_base%/FMW_Home/webtier/instances/%s_ohs_instance%/config/OHS/%s_ohs_component%/proxy-wallet/cwallet.sso

2021 January CPU patches for R12.2

Below are steps for applying R12.2 CPU patches to a R12.2.10 instance. This instance was newly upgraded from R12.1 and all technology patches were already applied during the upgrade as requirement by ETCC script. The document for this CPU patch set is Oracle E-Business Suite Release 12 Critical Patch Update Knowledge Document (January 2021) (Doc ID 2737201.1)

(a) Apply EBS patch 32071646

SQL> select ad_patch.is_patch_applied('R12',-1,32071646) from dual;

AD_PATCH.IS_PATCH_APPLIED('R12',-1,32071646)

-------------------------------------------------------------------

NOT_APPLIED

SQL> SELECT adb.bug_number, aat.name appl_top_name, adb.language, adb.creation_date,

decode(ad_patch.is_patch_applied('R12',aat.appl_top_id,adb.bug_number,adb.language),'EXPLICIT','APPLIED','NOT_APPLIED','NOT APPLIED') status

 FROM ad_bugs adb,

(select aat.name, aat.appl_top_id

from applsys.ad_appl_tops aat,

(select distinct fat.name from applsys.fnd_appl_tops fat) fat

where aat.name=fat.name ) aat

where adb.bug_number in (

'32071646',

'32163187',

'32004048'

) order by adb.bug_number,aat.name,adb.language;

no rows selected

$ echo APPL_TOP

$ cd $PATCH_TOP

$ unzip p32071646_12.2.0_R12_LINUX.zip

$ adop phase=apply apply_mode=downtime patches=32071646

$ unzip p32163187_R12.FWK.C_R12_GENERIC.zip

$ unzip p32004048_R12.OKC.C_R12_GENERIC.zip

$ adop phase=apply apply_mode=downtime patches=32163187,32004048

... ...

Applying patch 32163187.

    Log:  $NE_BASE/EBSapps/log/adop/4/.../32163187/log/u32163187.log

Applying patch 32004048.

    Log: $NE_BASE/EBSapps/log/adop/4/.../32004048/log/u32004048.log

Running finalize actions for the patches being applied.

    Log: @ADZDSHOWLOG.sql "2021/07/13 12:05:43"

Running cutover actions for the patches being applied.

    Creating workers to process cutover DDL in parallel

... ... 

The apply phase completed successfully.

adop exiting with status = 0 (Success)

Run SQL statement again to confirm 3 patches are applied.

(b) WebLogic PSU patch 32052267 was applied as a ETCC requirement.

See Doc ID 1306505.1 for more on Oracle WebLogic Server PSUs (Patch Set Updates).

(c) Oracle Fusion Middleware 11.1.1.9 OSS - Web Tier Home

Patch 31304503 (OSS Security Patch Update CPUJul2020): applied

$ export ORACLE_HOME=$IAS_ORACLE_HOME

$ export PATH=$IAS_ORACLE_HOME/OPatch:$PATH

$  opatch lsinventory | grep 31304503

Patch  31304503     : applied on Sat Jun 26 12:07:37 EDT 2021

(d) Oracle Fusion Middleware 11.1.1.9 OHS - Web Tier Home

Patch 31047338 (OHS Security Patch Update CPUApr2020)

$ opatch lsinventory | grep -i 31047338

Patch  31047338     : applied on Sat Jun 26 12:20:11 EDT 2021

(e) Oracle Fusion Middleware 11.1.1.9 - Oracle Common Home

Patch 30368663 (Security Patch Update CPUOct2019)

Note: Patch 31985571 is a superset of patch 30368663 in Oct 2020

$ export ORACLE_HOME=$FMW_HOME/oracle_common

$ export PATH=$ORACLE_HOME/OPatch:$PATH

$ opatch lsinventory | grep 30368663

Patch  30368663     : applied on Sat Jun 26 12:31:37 EDT 2021

$ adop -status

Enter the APPS password:

Connected.

==================================================

ADOP (C.Delta.12)

Session Id: 4

Command: status

Output: $NE_BASE/EBSapps/log/adop/4/20210X13_130234/adzdshowstatus.out

==================================================

Node Name       Node Type  Phase           Status          Started                        Finished             Elapsed

--------------- ---------- --------------- --------------- ------------------------------- -------------------- ------------

node_name     master     APPLY           ACTIVE      2021/0X/12 10:50:40  2021/0X/13 12:55:03  26:04:23

                                       CLEANUP     NOT STARTED

Apply R12.2 patch using ADOP

Adop downtime mode does not start an online patching cycle. It applies a patch to RUN file system and completes more quickly than in online mode, but at the cost of increased system downtime. There is no option to abort a failed patch and return to the existing RUN filesystem in downtime mode.

"hotpatch=yes" means while applying the patches, applications services are up and running. Same like hotpatch in R12.1 adpatch. It applies patch to RUN filesystem directly as well. Next time, when Online Patching mode is used, first run the "adop phase=prepare" command. At that time, adop config change detector will find that the RUN filesystem has had patches applied in hotpatch mode and will sync the PATCH filesystem as part of the prepare phase (Doc ID 1928798.1). Oracle support engineer says never try with hotpatch mode unless it is specified in the patch readme or any document. For example applying 33600809 (R12.AD.C.Delta.14) in hotpatch mode will fail and cause problem.

1. Pre steps

To make sure ADOP works on a multi-node instance, run "adop -validate" first, which includes execution of below line:

$ perl $AD_TOP/patch/115/bin/txkRunSSHSetup.pl verifyssh -contextfile=$CONTEXT_FILE -hosts=2nd_node

$ vi /etc/oraInst.loc

$ adop -validate

... ...

========================================

ADOP (C.Delta.12)

Node: master_node

Command: validate

Log: $NE_BASE/EBSapps/log/adop/16/.../validate/adopConsole.log

========================================

Checking for existing patching cycle.

    No existing patching cycle exists

Verifying SSH connection to all nodes.

    Log: $LOG_HOME/appl/rgf/TXK/verifyssh.log

    Output: $LOG_HOME/appl/rgf/TXK/out.xml

    Remote execution is operational.

Running adop validations on Admin node: [master_node].

    Log: master_node:$NE_BASE/EBSapps/log/adop/.../validate/master_node

    Output: $NE_BASE/EBSapps/log/adop/.../validate/remote_execution_result_level1.xml

        txkADOPEvalSrvStatus.pl returned SUCCESS

Running adop validations on node(s): [2nd_node and ].

    Output: $NE_BASE/EBSapps/log/adop/.../validate/remote_execution_result_level2.xml

        txkADOPEvalSrvStatus.pl returned SUCCESS

adop exiting with status = 0 (Success)

NOTES: it will starts WLS Admin server

$ adop -status

Enter the APPS password:

Connected.

=======================================================

ADOP (C.Delta.12)

Session Id: 16

Command: status

Output: $NE_BASE/EBSapps/log/adop/.../adzdshowstatus.out

=======================================================

Node Name       Node Type  Phase           Status          Started              Finished             Elapsed

--------------- ---------- --------------- --------------- -------------------- -------------------- ------------

Master_node    master   APPLY           ACTIVE         2021/0X/02 17:53:08  2021/0X/09 15:04:17  65:11:09

                                      CLEANUP     NOT STARTED

2nd_node         slave     APPLY           ACTIVE         2021/0X/02 18:22:12  2021/0X/09 15:22:16  65:00:04

                                      CLEANUP     NOT STARTED

File System Synchronization Type: None

adop exiting with status = 0 (Success)

2. Apply patch to multiple nodes in one command line

Below line on Master node shall apply AP patch 32385168, for example, to all nodes even $APPL_TOP file structure is not shared:

$ echo $FILE_EDITION

run

$ cd $PATCH_TOP

$ unzip p32385168_R12.AP.C_R12_GENERIC.zip

NOTES: Copy and run "unzip p32385168_R12.AP.C_R12_GENERIC.zip" on each node to avoid ADOP failure on the node.

     

$ adop phase=apply apply_mode=downtime patches=32385168 patchtop=/path/to/$PATCH_TOP

... ...

===========================================================

ADOP (C.Delta.12)

Session ID: 16

Node: master_node

Phase: apply

Log: $NE_BASE/EBSapps/log/adop/16/2021xxxx_142226/adop.log

============================================================

Verifying existence of context files in database.

Checking for failed nodes in the configuration.

Checking if adop can continue with available nodes in the configuration.

    Log: $NE_BASE/EBSapps/log/adop/16/.../apply/master_node

        txkADOPEvalSrvStatus.pl returned SUCCESS

Applying <32385168> patch(es) on admin node: [master_node].

    Output: $NE_BASE/EBSapps/log/adop/16/.../apply/remote_execution_result_level1.xml

    Log: $NE_BASE/EBSapps/log/adop/16/.../apply/master_node

        txkADOPEvalSrvStatus.pl returned SUCCESS

Applying <32385168> patch(es) on node(s): [2nd_node].

Running in Serial

    Output: $NE_BASE/EBSapps/log/adop/16/.../apply/remote_execution_result_level2.xml

    Log: $NE_BASE/EBSapps/log/adop/16/.../apply/master_node

        txkADOPEvalSrvStatus.pl returned SUCCESS

Summary report for current adop session:

     Node 2nd_node: Completed successfully

          - Apply status: Completed successfully

     Node master_node: Completed successfully

          - Apply status: Completed successfully

     For more details, run the command: adop -status -detail

Use SQL statement to confirm the patch was just applied to ALL nodes:

SQL> select ADOP_SESSION_ID, BUG_NUMBER, STATUS, APPLIED_FILE_SYSTEM_BASE, ADPATCH_OPTIONS, round((end_date-start_date)*24*60, 1) EXEC_TIME, AUTOCONFIG_STATUS,  DRIVER_FILE_NAME, NODE_NAME, END_DATE, CLONE_STATUS

from ad_adop_session_patches

order by end_date desc;

Seems specifying "patchtop=" is important. Otherwise, ADOP may fail on applying the patch to remote node. 

If patch's folder (from .zip file) did not exist in $PATCH_TOP of 2nd node or the patch was applied already, ADOP may give misleading error:

    [ERROR]     adop phase=apply failed on Node: "2nd_node"

    Log: $NE_BASE/EBSapps/log/adop/.../apply/2nd_node

        --------------------------------

        Summary of unavailable services:

        --------------------------------

        Group Name:

                Batch Processing Services

        Individual Services enabled in the group:

                OracleTNSListenerAPPS_EBSDEV_2nd_node

                OracleConcMgrEBSDEV_2nd_node

                Oracle Fulfillment Server EBSDEV_2nd_node

After this failure, SQL statement shows the patch was applied to just one node. After made a fix, I applied it to 2nd node only (with allnodes=no action=nodb options). 

In a situation, the patch failed on one of three nodes (but, the SQL may show it was not applied to any node).

    Summary report for current adop session:

          Node node1Name: Completed successfully

                - Apply status:      Completed successfully

          Node node2Name: Completed successfully

               - Apply status:      Completed successfully

          Node node3Name: Failed

               - Apply status:      Failed

         For more details, run the command: adop -status -detail

I ran "adop phase=apply apply_mode=downtime patches=33xxxxxx patchtop=$NE_BASE/EBSapps/patch restart=yes" on Master node to re-apply it.  adop will figure out and only apply the patch to Failed node node3Name.

Because patch can be applied to multiple nodes by one command line, I believe "passwordless ssh" setups are good between nodes in my multi-node instance. But somehow when I tried to set ssh up by running Oracle script, I always got errors (which I ignored):

$ perl $AD_TOP/patch/115/bin/txkRunSSHSetup.pl enablessh -contextfile=$CONTEXT_FILE -hosts=master_node,2nd_node

Enter SSH User password for the OS user applmgr:

Log: $LOG_HOME/appl/rgf/TXK/enablessh.log

Error in setting up ssh equivalence

FAILED: enableSSH

SEVERE: com.jcraft.jsch.JSchException: Algorithm negotiation fail

at com.jcraft.jsch.Session.receive_kexinit(Session.java:510)

at com.jcraft.jsch.Session.connect(Session.java:285)

at com.jcraft.jsch.Session.connect(Session.java:149)

at oracle.sysman.prov.ssh.RunCommand.runCommand(RunCommand.java:134)

at oracle.sysman.prov.ssh.SSHSetup.runCommandHelper(SSHSetup.java:2350)

at oracle.sysman.prov.ssh.SSHSetup.validateRemoteScp(SSHSetup.java:643)

at oracle.sysman.prov.ssh.SSHConnectivity.startSetup(SSHConnectivity.java:201)

at oracle.sysman.prov.ssh.SSHConnectivity.main(SSHConnectivity.java:360)

... ...

I am not sure what is the real problem or the root cause. It may be because the instance is in non-shared $APPL_TOP (s_shared_file_system => false).

3.  Apply patches on each node separately. I used below steps to apply two AP patches to two nodes.

On Master node: 

$ echo $FILE_EDITION

run

$ cd $PATCH_TOP

$ unzip p32768426_R12.AP.C_R12_GENERIC.zip

$ unzip p31211521_R12.AP.C_R12_GENERIC.zip

$ adop phase=apply apply_mode=downtime patches=32768426,31211521 allnodes=no action=db

(if it failed in somewhere, you have to fix the error and then try it again by command:

$ adop phase=apply apply_mode=downtime patches=32768426,31211521 allnodes=no action=db restart=yes)

On 2nd node:

$ cd $PATCH_TOP

$ adop phase=apply apply_mode=downtime patches=32768426,31211521 allnodes=no action=nodb

(Or, line also works: $ adop phase=apply apply_mode=downtime patches=32768426,31211521 allnodes=no action=nodb options=nocompiledb,nocompilejsp,nogenerateportion )

... ...

Applying patch 32768426.

    Log: /path/to/2nd_node/32768426/log/u32768426.log

Applying patch 31211521.

    Log: /path/to/2nd_node/31211521/log/u31211521.log

Generating post apply reports.

Generating log report.

    Output: /path/to/2nd_node/adzdshowlog.out

The apply phase completed successfully.

adop exiting with status = 0 (Success)

Then, SQL statement verifies patches were applied to both nodes.

4.  If you have to apply R12.2 patches online (or without "downtime") mode to multi-node instance and you have to run ADOP separately, more picky steps and more time are needed:

PREPARE Phase: Run this on both the servers

==================================

on Master Node:  $ adop phase=prepare allnodes=no action=db 

on 2nd Node:       $ adop phase=prepare allnodes=no action=nodb

Apply Phase: Run this on both the servers

=================================

on Master Node:  $ adop phase=apply patches=<Patch Number> allnodes=no action=db 

on 2nd Node:       $ adop phase=apply patches=<Patch Number> allnodes=no action=nodb options=nocompiledb,nocompilejsp,nogenerateportion 

Finalize:  Run only on Master node

==================================

$ adop phase=finalize allnodes=no action=db

Cutover: Run this on both the servers

==================================

on Master Node:  $ adop phase=cutover allnodes=no action=db mtrestart=no

Notes: You will see the some comments saying that it is waiting for the second node, then you need to run the adop command on the second node then only it will complete the cutover session.

on 2nd Node:       $ adop phase=cutover allnodes=no action=nodb mtrestart=no

Before running cleanup command/phase, start a new OS session or source the environment variable on the Primary server and then run the below cleanup command.

Cleanup: Run on Master node

================================

$ adop phase=cleanup cleanup_mode=full

Start and Stop R12.2 services and their logs

EBS R12.2 scripts have more options to start and stop its services

1.  Start /stop services on all nodes by running command on Master node:

First, use below line to check if it is a Master node:

$ grep s_adminserverstatus $CONTEXT_FILE

            <oa_service_status oa_var="s_adminserverstatus">enabled</oa_service_status>

$ { echo apps ; echo appsPWD ; echo wlsPWD ; } | adstpall.sh @ -mode=allnodes -nopromptmsg

... ... 

All enabled services for this node are started.

adstrtal.sh: Exiting with status 0

adstrtal.sh: check the logfile $LOG_HOME/appl/admin/log/adstrtal.log for more information ...

or 

$ ./adstrtal.sh apps/appsPWD -mode=allnodes

You are running adstrtal.sh version 120.24.12020000.11

Enter the WebLogic Server password: 

Running command in node node_name

All enabled services for this node are started.

adstrtal.sh: Exiting with status 0

adstrtal.sh: check the logfile $LOG_HOME/appl/admin/log/adstrtal.log for more information ...

$ adstpall.sh apps/appsPWD -mode=allnodes

You are running adstpall.sh version 120.22.12020000.7

Enter the WebLogic Server password:

Running command in node node_name

Running command in node node_name

All enabled services on this node are stopped.

adstpall.sh:Exiting with status 0

adstpall.sh: check the logfile $LOG_HOME/appl/admin/log/adstpall.log for more information ...

2. Start services on any one node (Doc ID 1902776.1):

{ echo apps ; echo appsPWD ; echo wlsPWD ; } | adstpall.sh @ -nopromptmsg

You may have to start app service on Master node first to avoid errors:

ERROR: Skipping startup of oacore_server1 since the AdminServer is down. AdminServer needs to be up before starting/stopping the managed servers.

Doc ID 2053781.1 (R12.2. Adadminsrvctl.sh Failed to Start WLS Adminserver When Using -secureapps Option)

3. Quicker stop (and so, quicker start)

./adstpall.sh apps/appsPWD -skipNM -skipAdmin

For help:

$ ./adstpall.sh -help

$ ./adstrtal.sh -help

4. Start services on Secondary node (when Primary node is not fully up)

{ echo apps ; echo appsPWD ; echo wlsPWD ; } | adstrtal.sh @ -msimode -nopromptmsg

Using -msimode option to avoid error 

ERROR: Skipping startup of oacore_server2 since the AdminServer is down. AdminServer needs to be up before starting/stopping the managed servers.

Doc ID 2052792.1 (EBS 12.2 Service Startup On Secondary Node Fails When Primary Node is Down)

5. For troubleshooting, start/stop individual component

For example, if login page is not up and available,  run

$ adopmnctl.sh status

You are running adopmnctl.sh version 120.0.12020000.2

Checking status of OPMN managed processes...

Processes in Instance: EBS_web_OHS1

---------------------------------------+--------------------+---------+---------

ias-component                     | process-type       |     pid    | status

---------------------------------------+--------------------+---------+---------

EBS_web                             | OHS                    |   30358 | NONE

It reads file in $IAS_ORACLE_HOME/instances/EBS_web_OHS1/config/OHS/EBS_web

To start Apache, run 

$ ./adapcctl.sh start

Timeout specified in context file: 100 second(s)

script returned:

****************************************************

You are running adapcctl.sh version 120.0.12020000.6

Starting OPMN managed Oracle HTTP Server (OHS) instance ...

adapcctl.sh: exiting with status 0

adapcctl.sh: check the logfile $LOG_HOME/adapcctl.txt for more information ...

In R12,2, Opmn only manages the Apache server.

WLS (Weblogic server) manages the oacore, forms and oafm services. To check a component's status:

$ sh $ADMIN_SCRIPTS_HOME/admanagedsrvctl.sh status oacore_server1

You are running admanagedsrvctl.sh version 120.14.12020000.12

Enter the WebLogic Admin password:

oacore_server1 is running.

Server specific logs are located at $FMW_HOME/user_projects/domains/...

$ sh $ADMIN_SCRIPTS_HOME/admanagedsrvctl.sh status oafm_server1

You are running admanagedsrvctl.sh version 120.14.12020000.12

Enter the WebLogic Admin password:

oafm_server1 is running.

Command                                          (Component)

$ adnodemgrctl.sh start                         (Node Manager)

$ adadminsrvctl.sh start                         (Weblogic Admin Server only on Primary node)

$ adalnctl.sh start                                       (Application Listener)

$ adopmnctl.sh start                                   (Oracle Process Manager)

$ adapcctl.sh start                                       (Apache Services)

$ admanagedsrvctl.sh start oacore_server1  (Managed Server for OACORE Services)

$ admanagedsrvctl.sh start forms_server1   (Managed Server for FormsServices)

$ admanagedsrvctl.sh start oafm_server1   (Managed Server for Fusion MiddleWare Services)

$ admanagedsrvctl.sh start forms-c4ws_server1  (Managed Server for Forms web Services)

$ adcmctl.sh start apps/apps                                (Concurrent Manager Service)

$ jtffmctl.sh start                                                  (Fullfillment Serer Services)

6. Log files

./adstrtal.sh 

You are running adstrtal.sh version 120.24.12020000.11

Enter the APPS username: apps

Enter the APPS password:
Enter the WebLogic Server password:

The logfile for this session is located at $INST_TOP/logs/appl/admin/log/adstrtal.log

Executing service control script: $INST_TOP/admin/scripts/jtffmctl.sh start
Timeout specified in context file: 100 second(s)

script returned:
*************************

You are running jtffmctl.sh version 120.3.12020000.4

Validating Fulfillment patch level via $RUN_BASE/EBSapps/comn/java/classes
Fulfillment patch level validated.
Starting Fulfillment Server for $TWO_TASK on port 9341 ...

jtffmctl.sh: exiting with status 0

.end std out.

.end err out.

****************************************************

Executing service control script: $INST_TOP/admin/scripts/adopmnctl.sh start
Timeout specified in context file: 100 second(s)

script returned:
***********************

You are running adopmnctl.sh version 120.0.12020000.2

Starting Oracle Process Manager (OPMN) ...

adopmnctl.sh: exiting with status 0

adopmnctl.sh: check the logfile $INST_TOP/logs/appl/admin/log/adopmnctl.txt for more information ...

.end std out.

.end err out.

****************************************************

Executing service control script: $INST_TOP/admin/scripts/adapcctl.sh start
Timeout specified in context file: 100 second(s)

script returned:
************************

You are running adapcctl.sh version 120.0.12020000.6

Starting OPMN managed Oracle HTTP Server (OHS) instance ...

adapcctl.sh: exiting with status 0

adapcctl.sh: check the logfile $INST_TOP/logs/appl/admin/log/adapcctl.txt for more information ...

.end std out.

.end err out.

****************************************************

Executing service control script: $INST_TOP/admin/scripts/adnodemgrctl.sh start -nopromptmsg
Timeout specified in context file: -1 second(s)

script returned:
***********************

You are running adnodemgrctl.sh version 120.11.12020000.12

Calling txkChkEBSDependecies.pl to perform dependency checks for ALL MANAGED SERVERS
Perl script txkChkEBSDependecies.pl got executed successfully

Starting the Node Manager...
Refer $INST_TOP/logs/appl/admin/log/adnodemgrctl.txt for details

NodeManager log is located at $FMW_HOME/wlserver_10.3/common/nodemanager/nmHome1

adnodemgrctl.sh: exiting with status 0

adnodemgrctl.sh: check the logfile $INST_TOP/logs/appl/admin/log/adnodemgrctl.txt for more information ...

.end std out.
*** ALL THE FOLLOWING FILES ARE REQUIRED FOR RESOLVING RUNTIME ERRORS
*** Log File = $INST_TOP/logs/appl/rgf/TXK/txkChkEBSDependecies_Tue_Feb_16_21_51_31_2021/txkChkEBSDependecies_Tue_Feb_16_21_51_31_2021.log

.end err out.

****************************************************

Executing service control script: $INST_TOP/admin/scripts/adalnctl.sh start
Timeout specified in context file: 100 second(s)

script returned:
**********************

adalnctl.sh version 120.3.12020000.4

Checking for FNDFS executable.
Starting listener process APPS_${TWO_TASK}.

adalnctl.sh: exiting with status 0

adalnctl.sh: check the logfile $INST_TOP/logs/appl/admin/log/adalnctl.txt for more information ...

.end std out.

.end err out.

****************************************************

Executing service control script: $INST_TOP/admin/scripts/adcmctl.sh start
Timeout specified in context file: 1000 second(s)

script returned:
**********************

You are running adcmctl.sh {TWO_TASK} version 120.19.12020000.7

Starting concurrent manager for {TWO_TASK}...
Starting ${TWO_TASK}_0413@${TWO_TASK} Internal Concurrent Manager
Default printer is noprint

adcmctl.sh: exiting with status 0

adcmctl.sh: check the logfile $INST_TOP/logs/appl/admin/log/adcmctl.txt for more information ...

.end std out.

.end err out.

****************************************************

Then, it may stay here for a while from running "adadminsrvctl.sh start -nopromptmsg":

Executing service control script: $INST_TOP/admin/scripts/adadminsrvctl.sh start -nopromptmsg
Timeout specified in context file: -1 second(s)

script returned:
************************

You are running adadminsrvctl.sh version 120.10.12020000.11

Starting WLS Admin Server...
Refer $INST_TOP/logs/appl/admin/log/adadminsrvctl.txt for details

AdminServer logs are located at $FMW_HOME/user_projects/domains/EBS_domain_${TWO_TASK}/servers/AdminServer/logs

adadminsrvctl.sh: exiting with status 0

adadminsrvctl.sh: check the logfile $INST_TOP/logs/appl/admin/log/adadminsrvctl.txt for more information ...

.end std out.

.end err out.

****************************************************

Executing service control script:
$INST_TOP/admin/scripts/admanagedsrvctl.sh start forms_server1 -nopromptmsg
Timeout specified in context file: -1 second(s)

script returned:
***********************

You are running admanagedsrvctl.sh version 120.14.12020000.12

Starting forms_server1...

Server specific logs are located at $FMW_HOME/user_projects/domains/EBS_domain_${TWO_TASK}/servers/forms_server1/logs

admanagedsrvctl.sh: exiting with status 0

admanagedsrvctl.sh: check the logfile $INST_TOP/logs/appl/admin/log/adformsctl.txt for more information ...

.end std out.

.end err out.

****************************************************

Executing service control script:
$INST_TOP/admin/scripts/admanagedsrvctl.sh start oafm_server1 -nopromptmsg
Timeout specified in context file: -1 second(s)

script returned:
*********************

You are running admanagedsrvctl.sh version 120.14.12020000.12

Starting oafm_server1...

Server specific logs are located at $FMW_HOME/user_projects/domains/EBS_domain_${TWO_TASK}/servers/oafm_server1/logs

admanagedsrvctl.sh: exiting with status 0

admanagedsrvctl.sh: check the logfile $INST_TOP/logs/appl/admin/log/adoafmctl.txt for more information ...

.end std out.

.end err out.

****************************************************

Executing service control script:
$INST_TOP/admin/scripts/admanagedsrvctl.sh start oacore_server1 -nopromptmsg
Timeout specified in context file: -1 second(s)

script returned:
***********************

You are running admanagedsrvctl.sh version 120.14.12020000.12

Starting oacore_server1...

Server specific logs are located at $FMW_HOME/user_projects/domains/EBS_domain_${TWO_TASK}/servers/oacore_server1/logs

admanagedsrvctl.sh: exiting with status 0

admanagedsrvctl.sh: check the logfile $INST_TOP/logs/appl/admin/log/adoacorectl.txt for more information ...

.end std out.

.end err out.

****************************************************

All enabled services for this node are started.

adstrtal.sh: Exiting with status 0

adstrtal.sh: check the logfile $INST_TOP/logs/appl/admin/log/adstrtal.log for more information ...

$ ./adstpall.sh apps/apps_PWD

You are running adstpall.sh version 120.22.12020000.7

Enter the WebLogic Server password:

The logfile for this session is located at $LOG_HOME/appl/admin/log/adstpall.log

Executing service control script: $ADMIN_SCRIPTS_HOME/adapcctl.sh stop
Timeout specified in context file: 100 second(s)

script returned:
****************************************************

You are running adapcctl.sh version 120.0.12020000.6

Stopping OPMN managed Oracle HTTP Server (OHS) instance ...

adapcctl.sh: exiting with status 0

adapcctl.sh: check the $LOG_HOME/appl/admin/log/adapcctl.txt for more information ...

.end std out.

.end err out.

****************************************************

Executing service control script: $ADMIN_SCRIPTS_HOME/adcmctl.sh stop
Timeout specified in context file: 1000 second(s)

script returned:
****************************************************

You are running adcmctl.sh version 120.19.12020000.7

Shutting down concurrent managers for EBSDEV ...
ORACLE Password:
Submitted request 43081823 for CONCURRENT FND SHUTDOWN

adcmctl.sh: exiting with status 0

adcmctl.sh: check the logfile $LOG_HOME/appl/admin/log/adcmctl.txt for more information ...

.end std out.

.end err out.

****************************************************

Executing service control script: $ADMIN_SCRIPTS_HOME/jtffmctl.sh stop
Timeout specified in context file: 100 second(s)

script returned:
****************************************************

You are running jtffmctl.sh version 120.3.12020000.4

Shutting down Fulfillment Server for IFSDEV on port 9331 ...

jtffmctl.sh: exiting with status 0

.end std out.

.end err out.

****************************************************

Executing service control script: $ADMIN_SCRIPTS_HOME/adopmnctl.sh stop
Timeout specified in context file: 100 second(s)

script returned:
****************************************************

You are running adopmnctl.sh version 120.0.12020000.2

Stopping Oracle Process Manager (OPMN)  and the managed processes ...
opmnctl stopall: stopping opmn and all managed processes...

adopmnctl.sh: exiting with status 0

adopmnctl.sh: check the logfile $LOG_HOME/appl/admin/log/adopmnctl.txt for more information ...

.end std out.

.end err out.

****************************************************

Executing service control script: $ADMIN_SCRIPTS_HOME/admanagedsrvctl.sh abort forms_server1 -nopromptmsg
Timeout specified in context file: -1 second(s)

script returned:
****************************************************

You are running admanagedsrvctl.sh version 120.14.12020000.12

Stopping forms_server1...
Refer $LOG_HOME/appl/admin/log/adformsctl.txt for details

Server specific logs are located at $FMW_HOME/user_projects/domains/EBS_domain_${TWO_TASK}/servers/forms_server1/logs

admanagedsrvctl.sh: exiting with status 0

admanagedsrvctl.sh: check the logfile $LOG_HOME/appl/admin/log/adformsctl.txt for more information ...

.end std out.
WARNING - Unable to create log file <$FMW_HOME/modules/../logs/wlst_20210416093507.log>.

.end err out.

****************************************************

Executing service control script: $ADMIN_SCRIPTS_HOME/admanagedsrvctl.sh abort oacore_server1 -nopromptmsg
Timeout specified in context file: -1 second(s)

script returned:
****************************************************

You are running admanagedsrvctl.sh version 120.14.12020000.12

Stopping oacore_server1...
Refer $LOG_HOME/appl/admin/log/adoacorectl.txt for details

Server specific logs are located at $FMW_HOME/user_projects/domains/EBS_domain_${TWO_TASK}/servers/oacore_server1/logs

admanagedsrvctl.sh: exiting with status 0

admanagedsrvctl.sh: check the logfile $LOG_HOME/appl/admin/log/adoacorectl.txt for more information ...

.end std out.

.end err out.

****************************************************

Executing service control script: $ADMIN_SCRIPTS_HOME/admanagedsrvctl.sh abort oafm_server1 -nopromptmsg
Timeout specified in context file: -1 second(s)

script returned:
****************************************************

You are running admanagedsrvctl.sh version 120.14.12020000.12

Stopping oafm_server1...
Refer $LOG_HOME/appl/admin/log/adoafmctl.txt for details

Server specific logs are located at $FMW_HOME/user_projects/domains/EBS_domain_${TWO_TASK}/servers/oafm_server1/logs

admanagedsrvctl.sh: exiting with status 0

admanagedsrvctl.sh: check the logfile $LOG_HOME/appl/admin/log/adoafmctl.txt for more information ...

.end std out.

.end err out.

****************************************************

Executing service control script: $ADMIN_SCRIPTS_HOME/adalnctl.sh stop
Timeout specified in context file: 100 second(s)

script returned:
****************************************************

adalnctl.sh version 120.3.12020000.4

Shutting down listener process APPS_IFSDEV.

adalnctl.sh: exiting with status 0

adalnctl.sh: check the logfile $LOG_HOME/appl/admin/log/adalnctl.txt for more information ...

.end std out.

.end err out.

****************************************************

Executing service control script: $ADMIN_SCRIPTS_HOME/adadminsrvctl.sh stop -nopromptmsg
Timeout specified in context file: -1 second(s)

script returned:
****************************************************

You are running adadminsrvctl.sh version 120.10.12020000.11

Stopping WLS Admin Server...
Refer $LOG_HOME/appl/admin/log/adadminsrvctl.txt for details

AdminServer logs are located at $FMW_HOME/user_projects/domains/EBS_domain_${TWO_TASK}/servers/AdminServer/logs

adadminsrvctl.sh: exiting with status 0

adadminsrvctl.sh: check the logfile $LOG_HOME/appl/admin/log/adadminsrvctl.txt for more information ...

.end std out.

.end err out.

****************************************************

Executing service control script: $ADMIN_SCRIPTS_HOME/adnodemgrctl.sh stop -nopromptmsg
Timeout specified in context file: -1 second(s)

script returned:
****************************************************

You are running adnodemgrctl.sh version 120.11.12020000.12

NodeManager log is located at $FMW_HOME/wlserver_10.3/common/nodemanager/nmHome1

adnodemgrctl.sh: exiting with status 0

adnodemgrctl.sh: check the logfile $LOG_HOME/appl/admin/log/adnodemgrctl.txt for more information ...

.end std out.

.end err out.

****************************************************

All enabled services on this node are stopped.

adstpall.sh:Exiting with status 0

adstpall.sh: check the logfile $LOG_HOME/appl/admin/log/adstpall.log for more information ...

7. All options

$ ./adstrtal.sh -help

USAGE:  adstrtal.sh <appsusername/appspassword> [-nothreading]

        adstrtal.sh <applications_username/applications_password> -secureapps

        adstrtal.sh -nodbchk

        adstrtal.sh -mode=allnodes

        adstrtal.sh -msimode

$ ./adstpall.sh -help

USAGE:  adstpall.sh <appsusername/appspassword> [-skipNM] [-skipAdmin] [-nothreading]

        adstpall.sh <applications_username/applications_password> -secureapps [-skipNM] [-skipAdmin]

        adstpall.sh -nodbchk [-skipNM] [-skipAdmin]

        adstpall.sh -mode=allnodes