umask and default file permission in Linux

You can find the umask value in your Linux account by typing "umask" on the command line:

$ umask

0077

The value on umask in a Linux server level is defined in file /etc/login.defs (or maybe in /etc/profile, /etc/bashrc or /etc/cshrc). Note contents of those files (and so umask value) may be changed in RHEL8 when OS got upgraded from RHEL7.

$ more /etc/login.defs

MAIL_DIR                /var/spool/mail

PASS_MAX_DAYS   90

PASS_MIN_DAYS    7 

PASS_WARN_AGE  7

PASS_MIN_LEN     8

UID_MIN               1000

UID_MAX              60000

GID_MIN              1000

GID_MAX             60000

CREATE_HOME     yes

UMASK                  077

USERGROUPS_ENAB yes

ENCRYPT_METHOD   sha512

$ egerp -i umask /etc/bashrc

Under 0077, any file you created will have "-rw-------" permission, which means only yourself can read and write it.

$ touch test1.del

$ ls -al test1.del

-rw-------. 1 userID Group 0 Feb 02 13:25 test1.del

But you can set up your own umask in .profile for your account. Sometimes, it is necessary for other users to read or modify a file created by a service account or you want others to read your files. To change the default from server level, add one line to the account's .profile:

$ vi $HOME/.profile

umask u=rwx,g=rwx,o=rx

or

umask  0002

After re-login, umask will change to 0002 in the account. Then, any file created by that account will get "-rw-rw-r--" permission. 

$ umask

0002

$ touch test2.del

$ ls -al test2.del

-rw-rw-r--. 1 userID Group 0 Feb 02 13:55 test2.del

Now, other users can read it.

If you put "umask u=rwx,g=rwx,o=rwx" or "umask  0000" in .profile, any new file will get "-rw-rw-rw-" permission (666). 

$ vi $HOME/.profile

umask u=rwx,g=rwx,o=rwx

$ umask

0000

$ touch test3.del

$ ls -al test3.del

-rw-rw-rw-. 1 userID Group 0 Feb 02 15:55 test3.del

If you have "umask  0022" in the profile, new file will get permission "-rw-r--r--".

Note that "x" in the .profile only applies to new folder creation. Linux allows only manually to grant executable to a file.

Connect to a remote server using the private key credential

When a 3rd party tool, such as PPM (Project and Portfolio Management) and Venafi (ssl cert tool), needs to access Oracle EBS server, we usually have to share the applMgr password to other teams. The good and efficient way is to provide them with the private key for them to log onto EBS server without entering the password. Steps to accomplish that on RHEL8 servers:

On Oracle EBS server ebs2d (local server):

1. Generate a pair of key files

$ hostname

ebs2d

$ echo $USER

applmgr

$ ssh-keygen -t rsa -b 2048

Generating public/private rsa key pair.

Enter file in which to save the key (/u06/app/.ssh/id_rsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /u06/app/.ssh/id_rsa.

Your public key has been saved in /u06/app/.ssh/id_rsa.pub.

The key fingerprint is:

SHA256:3y1+M95Js+4k383juI/qSsxxxxxxx   applmgr@ebs2d.domain.com

The key's randomart image is:

+---[RSA 2048]----+

|      .   .                   |

|     E +S+               |

|    o o =... .             |

|   + B  .o.oo..+       |

|  o.=++o. o. .@+  |

| .o+=++  .oooB@*|

+----[SHA256]-----+ 

$ cd .ssh

$ ls -alZ

-rw-------.  1 applmgr grp unconfined_u:object_r:unlabeled_t:s0 1843 Jun  4 20:27 id_rsa

-rw-------.  1 applmgr grp unconfined_u:object_r:unlabeled_t:s0  409 Jun  4 20:27 id_rsa.pub

-rw-------.  1 applmgr grp system_u:object_r:unlabeled_t:s0     3563 Jan 16 11:35 known_hosts

2. Make file authorized_keys as a copy of public key file id_rsa.pub (or, add the key to file authorized_keys)

$ cat id_rsa.pub >> authorized_keys

$ chmod 600 authorized_keys       # <= right permission is important

3. Change the labels on file authorized_keys (in RHEL8 OS)

$ chcon -u system_u authorized_keys

$ chcon -t user_home_t authorized_keys

$ ls -alZ

-rw-------.  1 applmgr grp system_u:object_r:user_home_t:s0 409 Jun  4 20:27 authorized_keys

-rw-------.  1 applmgr grp unconfined_u:object_r:unlabeled_t:s0 1843 Jun  4 20:27 id_rsa

-rw-------.  1 applmgr grp unconfined_u:object_r:unlabeled_t:s0  409 Jun  4 20:27 id_rsa.pub

-rw-------.  1 applmgr grp system_u:object_r:unlabeled_t:s0     3563 Jan 16 11:35 known_hosts

4. Copy private key id_rsa to remote server ppm1p and name it meaningfully. 

Or, send file id_rsa to other trusted teams.

$ scp -p id_rsa usr@ppm1p:/path/to/applmgr_ebs2d_PPM_key

Password:

id_rsa                                                              100% 1843   903.2KB/s   00:00

On remote server ppm1p (host of 3rd party tool):  

After received the private key, other team can set up their server to run scripts against Oracle EBS server without intervention.

$ cd /path/to

$ ls -alZ

-rw-------. 1 usr group unconfined_u:object_r:unlabeled_t:s0 1843 Jun  4 20:27 applmgr_ebs2d_PPM_key

Now, connect to EBS server using the private key credential (without entering applmgr's password!):

$ hostname 

ppm1p

$ echo $USER

usr

$ ssh -i /path/to/applmgr_ebs1d_PPM_key applmgr@ebs2d

Connected!

$ hostname

ebs2d

$ echo $USER

applmgr

My old post has more details on running ssh, sftp, scp between servers without a pasword.

Apply R12.2 October 2024 CPU patches

The document for October 2024 CPU patches is Doc ID 3037725.1 (Oracle E-Business Suite Release 12.2 Critical Patch Update Availability Document - October 2024). We have to follow it to apply this CPU patches.

1. First of all, specify a location for holding all patch files. All zip file shall be downloaded/saved in sub-folders of this location

$ patch_folder=/path/to/CPU_Patches/Oct2024

2. Run EJCPUC script on database server to check the Java versions. A new utility, Oracle E-Business Suite Java Critical Patch Update Checker (EJCPUC, Patch 37171025), was released with the Oct 2024 CPU. EJCPUC Output:

$ bash ejcpuc.sh

#############################################################

## Checking DB tier Java for CPU 2024.10 on Platform IBM_AIX

#############################################################

## Check Database Version

#############################################################

Your database version is 19.25.0.0.0

         ORACLE_HOME     $ORACLE_HOME

         ORACLE_SID          EBSDEV

         ORACLE_UNQNAME

## Check Java Version of OJVM, Database JDK and EBS's appsutil JRE

#############################################################

 Latest Version  action  Your Version  bitness Java Location

 -------------- -------- ------------  ------- ---------------

 1.8.0_431       o)    _.__.101034000   64-bit   OJVM In database

 1.8.0_411               1.8.0_421             64-bit   $ORACLE_HOME/jdk/bin/java

 1.8.0_411        u)   1.8.0_271             64-bit   $ORACLE_HOME/appsutil/jre/bin/java

o) Apply the Database Release Update (DBRU) recommended by ETCC which will update the DB OJVM version to the latest

u) When the DB JDK version is updated to the latest - then follow section 3 of 1530033.1 to update this JRE

3. Download ETCC (patch 17537119). Make sure both database and apps use the same release of ETCC

$ cd $patch_folder/ETCC

$ grep 120 *.sh

checkDBpatch.sh:# $Header: checkDBpatch.sh 120.127 2024/11/12 15:02:31 chrhill noship $

checkMTpatch.sh:# $Header: checkMTpatch.sh 120.0.12020000.68 2024/04/12 16:35:27 chrhill noship 

3. DBAs apply database patches (after EBS services are stopped) to the database and make sure all requirements are met. ETCC Output :

$ ./checkDBpatch.sh

 +==========================================+

 |    Copyright (c) 2005, 2024 Oracle and/or its affiliates.        |

 |                     All rights reserved.                                            |

 |             Oracle E-Business Suite Release 12.2                     |

 |          Database EBS Technology Codelevel Checker           |

 +==========================================+

Database environment not set, going to check for GridHome.

Oracle Grid Infrastructure not identified.

Database environment not set and no Grid home found, so context file must be specified.

Enter full path to database context file: $ORACLE_HOME/appsutil/<CONTEXT_NAME>.xml

Validating context file: $ORACLE_HOME/appsutil/<CONTEXT_NAME>.xml

Using context file from user input:

$ORACLE_HOME/appsutil/<CONTEXT_NAME>.xml

Starting Database EBS Technology Codelevel Checker, Version 120.127

Mon Nov 25 16:41:09 EST 2024

Log file for this session: $ORACLE_HOME/appsutil/etcc/log/checkDBpatch_45744600.log

Identifying database release.

Database release set to 19.25.0.0.

Multitenant identified.

 - Container database (CDB) identified via s_cdb_name is CEBSDEV

 - Pluggable database (PDB) identified via s_pdb_name is EBSDEV

Connecting to database.

Database connection successful.

Database EBSDEV is in READ WRITE mode.

Identifying APPS and APPLSYS schema names.

 - APPS schema:   APPS

 - APPLSYS schema: APPLSYS

Checking for existence DB-ETCC results table.

Table to store DB-ETCC results already exists in the database.

Bugfix file ./db/onprem/txk_R1220_DB_base_bugs.xml: 120.0.12020000.85

This file will be used for identifying missing bugfixes.

Mapping file ./db/onprem/txk_R1220_DB_mappings.xml: 120.0.12020000.61

This file will be used for mapping bugfixes to patches.

+---------------------------------------------------------------------------------------+

  Always use the latest version of ETCC available in patch 17537119,

  as new bugfixes will not be checked by older versions of the utility.

  You should apply the latest recommended RU, BP, or PSU as appropriate.

+---------------------------------------------------------------------------------------+

Identified database DST version: 32

Checking Bugfix XML file for tag 19.25.0.0_RU.

Obtained list of bugfixes to be applied and list to be rolled back.

Validating OPatch version:

The OPatch utility is version 12.2.0.1.44.

DB-ETCC is compatible with this OPatch version.

Checking for applied patch history:

Found patch history in the inventory.

Checking mapping XML file for tag 19.25.0.0.241015DBRU.

All the required one-off bugfixes are present in database ORACLE_HOME.

5. Proceed with Apps patches

- Back up Apps file systems

- Start Apps services to make sure EBS works after database patching. 

- Stop Apps services 

6. ECPUC script (from patch p35583866) to identify what EBS CPU patches and security fixes are needed.

$ cd $patch_folder/ECPUC

$ sqlplus apps/appsPWD

SQL> @ECPUC.sql

... ...

E-Business Suite Critical Patch Update Checker (ECPUC)

ECPUC.sql may be run on any EBS 12.2 environment to identify missing patches that are in the latest EBS CPU.

You can download the latest version of ECPUC via Patch 35583866.

Refer to the README.txt in Patch 35583866 for instructions for running ECPUC and information regarding the generated ECPUC.lst report.

The checker generates the report ECPUC_YYYY-MM-DD_HH24-MI.lst that lists recommended EBS CPU patches and security fixes for your environment per Table 1 'CPU Patches for Oracle E-Business Suite' and Table 2 'Additional Patches Required' documented in the quarterly EBS CPU MOS Note.

Each quarterly EBS CPU MOS Note ID is unique. Refer to My Oracle Support (MOS) Knowledge Document 2484000.1, 'Identifying the Latest Critical Patch Update for Oracle E-Business Suite Release 12' which includes a link to the current EBS CPU MOS document.

============================================

SECTION-1 ECPUC Version

============================================

EBS CPU Checker Version

-----------------------

                 2024.1

============================================

SECTION-2 Oracle E-Business Suite (EBS): Instance Information

============================================

***********************************************************

Instance Summary

***********************************************************

EBS Release EBS CPU Level

----------- -------------

12.2.10     2023.10

Instance Name    Database         Database Version

---------------- ---------------- ------------------

CEBSDEV           EBSDEV            19.25.0.0.0

ATG Product  Product Name                             Code Level

------------ ----------------------------------------------- ----------

ad           Applications DBA                                    C.14

txk         Oracle Applications Technology Stack     C.14

atg_pf    Oracle Applications Technology Family  C.9

fwk        Oracle Applications Framework               C.9

===========================================

SECTION-3 Required EBS CPU and Security Fixes

===========================================

The following output is a list of required patches and security fixes that are missing in your environment.

It is strongly recommended that you apply all of the listed patches as soon as possible.

If no patches (no rows) are listed then no additional action is required at this time as your environment includes all patches for this EBS CPU.

**********************************************************

The following patches are required for this EBS CPU

--------------------------------------------------------------

36944346:12.2.0

37078855:R12.ATG_PF.C

37078915:R12.SCM_PF.C

37120482:R12.CC_PF.C

35362524:R12.IGI.C

34979060:R12.MSC.C

33457157:R12.HXT.C

30448458:R12.HXT.C

8 rows selected.

7. Apply two EBS patches. After consulting with Developers and Business users, we will apply only first two patches because we do not use the EBS modules on the list.

36944346:12.2.0

37078855:R12.ATG_PF.C

Make sure all Apps services are down

$ ps -ef | grep $LOGNAME

$ vi /etc/oraInst.loc

$ adop -status

If FS_CLONE has not been run, run it. This will help to screen out errors in applying new patches.

$ adop phase=fs_clone 

-- Apply patch to multiple nodes

$ adop phase=apply apply_mode=downtime patches=36944346 patchtop=$patch_folder/Oct24_CPU

$ adop phase=apply apply_mode=downtime patches=37078855 patchtop=$patch_folder/Post

Run query to confirm Oct 2024 CPU in the instance. See Doc ID 2484000.1 (Identifying the Latest Critical Patch Update for Oracle E-Business Suite Release 12.2)

SQL> col CPU format a9

SQL> select max(CODELEVEL) "CPU" from ad_trackable_entities where abbreviation in ('ebscpu');

CPU

---------

2024.10

8. Run autoconfig on database server

$ perl $AD_TOP/bin/admkappsutil.pl

Starting the generation of appsutil.zip

Log file located at $INST_TOP/admin/log/MakeAppsUtil_11260913.log

output located at $INST_TOP/admin/out/appsutil.zip

MakeAppsUtil completed successfully.

$ cp -p $INST_TOP/admin/out/appsutil.zip $APPLPTMP

$ chmod 777 $APPLPTMP/appsutil.zip

DBA run the below steps on EBSDEV database.

1. cp /u04/shared/utl_dir/appsutil.zip $ORACLE_HOME/

2. cd $ORACLE_HOME; unzip -o appsutil.zip

3. Set env and then run autoconfig on the database.

9. Run ETCC

$ cd $patch_folder/ETCC

$ ./checkMTpatch.sh

... ...

Starting Application Tier EBS Technology Codelevel Checker, Version 120.0.12020000.68.

... ...

=========================================

PATCH RECOMMENDATION SUMMARY

=========================================

One or more products have bugfixes missing.

The default patch recommendations to install these missing bugfixes are:

-------------------------------------------------------------------------------

Oracle Fusion Middleware (FMW) - Oracle Common 11.1.1.9.0

-------------------------------------------------------------------------------

  Patch 34714760

    - Filename: p34714760_111190_Generic.zip

-------------------------------------------------------------------------------

WLS 10.3.6.0.231017

-------------------------------------------------------------------------------

  Patch 35476084 [SU Patch [KMHV]]

    - Filename: p35476084_1036_Linux-x86-64.zip

Apply the required patches and rerun this script.

+-----------------------------------------------------------------------------+

A consolidated zip file with the required application tier patches is

available on My Oracle Support via:

  Patch 36616672

10.  Apply WLS patches: KMHV, WY44

$ echo $APPL_TOP 

$ echo $ORACLE_HOME

$RUN_BASE/EBSapps/10.1.2

$ echo $FMW_HOME

$RUN_BASE/FMW_Home

$ cd $FMW_HOME/utils/bsu

-- Check if patch p13845626 was applied:

$ ./bsu.sh -prod_dir=$FMW_HOME/wlserver_10.3 -status=applied -verbose -view | egrep -i 'WY44'

Patch ID:            WY44

PatchContainer:    WY44.jar

$ cd $FMW_HOME/utils/bsu/cache_dir

$ cp -p $patch_folder/WLS/35476084/p35476084_1036_Linux-x86-64.zip .   # (KMHV)

$ cp -p $patch_folder/WLS/13845626/p13845626_10360231017_Generic.zip . 

# (WY44. Asked by Doc ID 3037725.1)

$ unzip p35476084_1036_Linux-x86-64.zip

Archive:  p35476084_1036_Linux-x86-64.zip

replace README.txt? [y]es, [n]o, [A]ll, [N]one, [r]ename: A

  inflating: README.txt

  inflating: patch-catalog_27986.xml

  inflating: KMHV.jar

$ cd ..

$ ./bsu.sh -install -patch_download_dir=$FMW_HOME/utils/bsu/cache_dir -patchlist=KMHV -prod_dir=$FMW_HOME/wlserver_10.3

Checking for conflicts..

Conflict(s) detected - resolve conflict condition and execute patch installation again

Conflict condition details follow:

Patch KMHV is mutually exclusive and cannot coexist with patch(es): CW7X

$ ./bsu.sh -remove -patchlist=CW7X -prod_dir=$FMW_HOME/wlserver_10.3

Checking for conflicts..

No conflict(s) detected

Removing Patch ID: CW7X..

Result: Success

$ ./bsu.sh -install -patch_download_dir=$FMW_HOME/utils/bsu/cache_dir -patchlist=KMHV -prod_dir=$FMW_HOME/wlserver_10.3

Checking for conflicts..

No conflict(s) detected

Installing Patch ID: KMHV..

Result: Success

$ ./bsu.sh -prod_dir=$FMW_HOME/wlserver_10.3 -status=applied -verbose -view | egrep -i 'KMHV'

Patch ID:          KMHV

PatchContainer:    KMHV.jar

-- if patch p13845626 was NOT applied:

$ ./bsu.sh -install -patch_download_dir=$FMW_HOME/utils/bsu/cache_dir -patchlist=WY44 -prod_dir=$FMW_HOME/wlserver_10.3

11. Apply patch 34714760 to Oracle Fusion Middleware (FMW) - Common

$ export ORACLE_HOME=$FMW_HOME/oracle_common

$ export PATH=$ORACLE_HOME/OPatch:$PATH

$ which opatch

$ cd /tmpshrs/ebsarupgradeTEMP/CPU_Patches/Oct2024/FMW_OHS_Orcl_Comn

$ ls -al

$ mv 34714760 34714760_Feb19

$ unzip p34714760_111190_Generic.zip

$ cd 34714760

$ opatch apply

... ...

Patching component oracle.jrf.opss, 11.1.1.9.0...

... ...

-- Check if 4 patches were applied

$ opatch lsinventory | egrep -i '34330735|33974106|33960746|34714760'

-- if patch 33960746 was not applied

$ cd /tmpshrs/ebsarupgradeTEMP/CPU_Patches/Oct2024/FMW_OHS_Orcl_Comn

$ unzip p33960746_111190_Generic.zip    (required by Doc ID 3037725.1)

$ cd 33960746

$ opatch apply

... ...

Patching component oracle.sysman.common, 10.2.0.5.6...

Patching component oracle.sysman.oms.core, 11.1.1.9.0...

Patching component oracle.sysman.plugin.ai.main.oms, 11.1.1.9.0...

... ...

Note: if a patch was applied previously, "opatch apply" again will give below message: 

$ opatch apply

Applying interim patch '33960746' to OH '$FMW_HOME/oracle_common'

Verifying environment and performing prerequisite checks...

The following patch(es) are duplicate patches with patches installed in the Oracle Home.

 [ 33960746]

You have already installed same patch(es) with same UPI(s) or same version(s).

These patch(es) will be skipped.

... ...

OPatch stopped on request.

 

$ opatch lsinventory | egrep -i '34330735|33974106|33960746|34714760'

Patch  33960746     : applied on Thu Dec 05 12:03:32 EST 2024

     33960746

Patch  34714760     : applied on Thu Dec 05 11:48:27 EST 2024

     34714760

Patch  33974106     : applied on Fri Nov 24 13:31:49 EST 2023

Patch  34330735     : applied on Wed Aug 17 21:59:30 EDT 2022

12. Run ETCC again

Start a new OS session to get the correct env, and make sure no more patches are needed.

$ cd $patch_folder/ETCC

$ ./checkMTpatch.sh

==========================================

All required one-offs are confirmed as present.

Finished checking prerequisite patches for file edition: run.

13. Upgrade JDK

$ cd $patch_folder/EJCPUC

$ ls

ejcpuc.cmd  ejcpuc.sh  p37171025_R12_GENERIC.zip  Readme.txt

$ sh ejcpuc.sh

############################################################

## Checking Apptier Java 7 for CPU 2024.10 on Platform Linux_x64 - need 1.7.0_441

############################################################

 2024.10        action  Your Version    bitness Java Location

 ------------   ------  ------------    ------- ---------------

 1.7.0_441      UPDATE   1.7.0_391    32-bit  $ORACLE_HOME/jdk/bin/java

 1.7.0_441      UPDATE   1.7.0_391    32-bit  $COMMON_TOP/util/jdk32/bin/java

 1.7.0_441      UPDATE   1.7.0_391    64-bit  $COMMON_TOP/util/jdk64/bin/java

 1.7.0_441      UPDATE   1.7.0_391    64-bit  $FMW_HOME/webtier/jdk/bin/java

Follow 1530033.1 to update the JDK(s). Your application tier JDK 7 is lower than the 1.7.0_441 update released in CPU 2024.10.

$ cd $patch_folder/JDK_1_7_441

Assume 2 JDK files and my shell script for JDK upgrade exist in this folder:

$ ls

jdk-7u441-linux-i586.tar.gz

jdk-7u441-linux-x64.tar.gz

JDK_upgrade441.sh

$ ./JDK_upgrade441.sh

Verify:

$ cd $COMMON_TOP/util/

$ ls -al

$ cd $patch_folder/EJCPUC

$ sh ejcpuc.sh

###########################################################

## Checking Apptier Java 7 for CPU 2024.10 on Platform Linux_x64 - need 1.7.0_441

############################################################

 2024.10        action  Your Version    bitness Java Location

 ------------   ------  ------------    ------- ---------------

 1.7.0_441      OK      1.7.0_441       32-bit   $ORACLE_HOME/jdk/bin/java

 1.7.0_441      OK      1.7.0_441       32-bit   $COMMON_TOP/util/jdk32/bin/java

 1.7.0_441      OK      1.7.0_441       64-bit   $COMMON_TOP/util/jdk64/bin/java

 1.7.0_441      OK      1.7.0_441       64-bit   $FMW_HOME/webtier/jdk/bin/java

14. Upgrade JRE

$ cd $COMMON_TOP/webapps/oacore/util/javaplugin

# Assume below file is downloaded and unzipped in folder

# $patch_folder/JRE_8_431

# p37063177_180431_WINNT.zip

$ cp $patch_folder/JRE_8_431/jre-8u431-windows-i586.exe j2se18431.exe

$ ls -altr

$ $FND_TOP/bin/txkSetPlugin.sh 18431

$ grep sun_plugin_ver $CONTEXT_FILE

         <sun_plugin_ver oa_var="s_sun_plugin_ver">1.8.0_431</sun_plugin_ver>

$ grep s_forms_launch_method $CONTEXT_FILE

         <config_option type="techstack" oa_var="s_forms_launch_method">jws</config_option>

15. Finalize 

optional: Re-sign JAR files. It takes time: "Creating and signing every jar file can take about thirty minutes ..."

$ adadmin

option 1 => 4 => yes

$ autoconfig on all Apps nodes

Start all services

$ adop phase=fs_clone

=== script JDK_upgrade441.sh for upgrading JDK =====

$ more JDK_upgrade441.sh

# Oracle JDK 7 Update 441 Patch Patch 37063192

DT=`date +"%h_%Y"`

curr=`pwd`

echo $curr

JDKfolder=$patch_folder/Oct2024/JDK_1_7_441

# Assume two JDK files (from patch 37063192) are saved in the folder:

# jdk-7u441-linux-i586.tar.gz

# jdk-7u441-linux-x64.tar.gz

ls -al $JDKfolder

echo "Current JDK version:"

$ADJVAPRG -version

$AFJVAPRG -version

# --

echo "$COMMON_TOP/util"

cd $COMMON_TOP/util

tar -czf jdk64_BK_$DT.tar.gz jdk64  # do not use -v (to turn off output)

tar -czf jdk32_BK_$DT.tar.gz jdk32

rm -fr jdk64

rm -fr jdk32

cp -p $JDKfolder/*.tar.gz .

tar -xzf jdk-7u441-linux-i586.tar.gz

mv jdk1.7.0_441 jdk32

tar -xzf jdk-7u441-linux-x64.tar.gz

mv jdk1.7.0_441 jdk64

ls -al jdk*

pwd

sleep 5

# --

echo "$FMW_HOME/webtier"

cd $FMW_HOME/webtier

tar -czf jdk64_BK_$DT.tar.gz jdk

rm -fr jdk

cp -p $JDKfolder/jdk-7u441-linux-x64.tar.gz .

tar -xzf jdk-7u441-linux-x64.tar.gz

mv jdk1.7.0_441 jdk

ls -al jdk*

pwd

sleep 5

# --

echo "$ORACLE_HOME"

cd $ORACLE_HOME

tar -czf jdk32_BK_$DT.tar.gz jdk

rm -fr jdk

cp -p $JDKfolder/jdk-7u441-linux-i586.tar.gz .

tar -xzf jdk-7u441-linux-i586.tar.gz

mv jdk1.7.0_441 jdk

ls -al jdk*

pwd

sleep 5

echo "New JDK version:"

$ADJVAPRG -version

$AFJVAPRG -version

echo "Compiling:"

cd $ORACLE_HOME/forms/lib

make -f ins_forms.mk sharedlib install

cd $ORACLE_HOME/reports/lib

make -f ins_reports.mk install

cd $curr

echo "Done"

Finding concurrent programs that trace is enabled

EBS can enable trace on concurrent program level. Navigation: Concurrent => Program => Define. Enter "Short Name" (concurrent_program_name below) to check the checkbox field on Enable Trace.

After trace is enabled, the job may take more resources on database server. SQL statement to find all concurrent programs that trace is enabled. 

SQL> select fp.concurrent_program_name, fct.user_concurrent_program_name, fct.last_update_date, fct.last_updated_by, fu.description

 from applsys.fnd_concurrent_programs fp, applsys.fnd_concurrent_programs_tl fct, fnd_user fu

 where fp.concurrent_program_id = fct.concurrent_program_id

     and fct.last_updated_by = fu.user_id and enable_trace <> 'N'

  order by fct.last_update_date asc;

fnd_web_sec.change_password in R12.2

fnd_web_sec.change_password still works in R12.2.10. It is recommended to use it only in some special/urgent needs because it ignores the restrictions by EBS Profile options 'Signon%'.

Below statement by APPS will change EBS_userID password:

SQL> select fnd_web_sec.change_password('EBS_userID','newPwd4U') from dual;

FND_WEB_SEC.CHANGE_PASSWORD('EBS_USERID','NEWPWD4U')

-------------------------------------------------------------------------------------

Y

You can use below line to confirm a password:

SQL> select fnd_web_sec.validate_login('EBS_userID', 'newPwd4U') from dual;

FND_WEB_SEC.VALIDATE_LOGIN('EBS_USERID','NEWPWD4U')

--------------------------------------------------------------------------------

Y

SQL> select fnd_web_sec.validate_login('EBS_userID', 'myPWD_01') from dual;

FND_WEB_SEC.VALIDATE_LOGIN('EBS_USERID','MYPWD_01')

--------------------------------------------------------------------------------

N

SQL> select fnd_message.get from dual;

GET

--------------------------------------------------------------------------------

PASSWORD_INVALID

One day, EBS users cannot log onto EBS site. The login webpage shows up but does not allow any users in. Since there is no error on EBS apps side, we do not know it is a security/password problem or other problems.  I used below queries to show it is a database problem 

SQL> show user

USER is "APPS"

SQL> select HOST_NAME, INSTANCE_NAME from v$instance;

HOST_NAME   INSTANCE_NAME

------------------  -------------------------

ebsdb1q             CEBSQA

SQL> select fnd_web_sec.validate_login('EBS_userID', 'XXXXxxx') from dual;   

ERROR at line 1:

ORA-03113: end-of-file on communication channel

SQL> select sysdate from dual;

ERROR:

ORA-03114: not connected to ORACLE

NOTE: FND_WEB_SEC.validate_password( ) is aonther function.

SQL> select fnd_web_sec.validate_password('EBS_userID', 'newPwd4U') from dual;

FND_WEB_SEC.VALIDATE_PASSWORD('EBS_USERID','NEWPWD4U')

--------------------------------------------------------------------------------

N

SQL> select fnd_message.get from dual;

GET

--------------------------------------------------------------------------------

Must not reuse a recently used password. Please supply a different password.

NOTE: fnd_message.get can be used sometimes to get useful information. For example, after a Java load errored out in Sql*Plus, below line gives some details:

SQL> select fnd_message.get from dual;

GET

--------------------------------------------------------------------------------

Unable to load Java class oracle.apps.xxfnd.custom.security.PasswordValidation specified in profile option SIGNON_PASSWORD_CUSTOM.  Please verify that the class exists and that it implements the Java interface oracle.apps.fnd.security.PasswordValidation.

EBS forms failed by CrowdStrike

EBS Forms in our financial applications suddenly does not work. The message on the webpage is

Failure of Web Server bridge:

No backend server available for connection: timed out after 10 seconds or idempotent set to OFF or method not idempotent. 

 

The error message does not tell the true problem. When checking into services on OS level, I saw Oracle EBS Forms service was not running and also saw errors from startup script $ADMIN_SCRIPTS_HOME/adstrtal.sh:

Forms service failed to start. 

The Node Manager is already up.

ERROR: Unable to start up the managed server forms_server1

Server specific logs are located at $EBS_DOMAIN_HOME/servers/forms_server1/logs

05/13/24-20:56:26 :: admanagedsrvctl.sh: exiting with status 1

Java error exists in Forms log file $EBS_DOMAIN_HOME/servers/forms_server1/logs/forms_server1.out

<May 13, 2024 8:56:25 PM EDT> <Emergency> <Store> <BEA-280060> <The persistent store "_WLS_forms_server1" encountered a fatal error, and it must be shut down: weblogic.store.PersistentStoreFatalException: [Store:280020]There was an error while reading from the log file

weblogic.store.PersistentStoreFatalException: [Store:280020]There was an error while reading from the log file

        at weblogic.store.io.file.FileStoreIO.open(FileStoreIO.java:128)

        at weblogic.store.internal.PersistentStoreImpl.recoverStoreConnections(PersistentStoreImpl.java:435)

        at weblogic.store.internal.PersistentStoreImpl.open(PersistentStoreImpl.java:423)

        at weblogic.store.admin.AdminHandler.activate(AdminHandler.java:126)

        at weblogic.store.admin.FileAdminHandler.activate(FileAdminHandler.java:207)

        Truncated.

Caused By: java.io.EOFException: premature EOF: expected=512, actual=126

        at weblogic.store.io.file.StoreFile.readBulk(StoreFile.java:316)

        at weblogic.store.io.file.Heap.readStoreFile(Heap.java:1142)

        at weblogic.store.io.file.Heap.getNextRecoveryFile(Heap.java:1226)

        at weblogic.store.io.file.Heap.open(Heap.java:373)

        at weblogic.store.io.file.FileStoreIO.open(FileStoreIO.java:117)

        Truncated.

Seems WebLogic failed to open a file, but the log did not say which file. I knew that Linux Admins just did server maintenance and rebooted server after they applied monthly patches and Security updates on OS level. That was the only change in the application environment recently.

After searching around, I found the Java errors match the description in Oracle Doc ID 3017110.1 ( Managed Forms Server Fails To Start - Displaying Message: FAILED_NOT_RESTARTABLE - ERROR: <BEA-280061> The persistent store "_WLS_forms_server1" could not be deployed: weblogic.store.PersistentStoreFatalException [Store:280020] ). 

The document points out the problem is caused by CrowdStrike, which locks a Forms file in $EBS_DOMAIN_HOME/servers/forms_server#/data/store/default.

CrowdStrike is installed in /opt/CrowdStrike. It is owned by root, and it is running constantly on the Linux server.

$ ps -ef | grep falcon-sensor

root      1081  1079  0 May13 ?        00:22:23 falcon-sensor

The problem can be fixed temporarily by a workaround:

1. Delete/rename below .DAT file (I guess CrowdStrike does not like the file name and so locks it)

$ cd $EBS_DOMAIN_HOME/servers/forms_server1/data/store/default

$ ls -altr

total 1028

drwxr-xr-x 4 user group      40 Sep 13  2023 ..

-rw-r--r-- 1   user group  1049088 May 13 20:51 _WLS_FORMS_SERVER1000000.DAT

drwxr-xr-x 2 user group     42 May 13 20:56 .

$ rm _WLS_FORMS_SERVER1000000.DAT

2. Re-start services cleanly by

$ADMIN_SCRIPTS_HOME/adstrtal.sh

Or

$ADMIN_SCRIPTS_HOME/admanagedsrvctl.sh start forms_server1

The permanent fix is that the Secure team rolls back the CrowdStrike change (and applies it again until CrowdStrike fixes the problem), because its new update touches an Oracle Forms data file wrongly during its scan. 

Scripts for start & stop EBS services

When server reboots for maintenance or unexpected downtime, we want it to bring EBS services down and up automatically. Sometimes, we also want to schedule EBS downtime by a cron job. Shell scripts are necessary for accomplishing those.

Two scripts assume that a solid $HOME/.profile and a file $HOME/xxx_scripts/.EBSpassenv holding key passwords exist on the server. They will generate log files to show when the scripts last ran.

$ more .EBSpassenv

export APPS_PWD=apps#@PWD

export SYSTEM_PWD=system%_PWD

export WLS_ADMIN=wls$%^PWD

============ script auto_startall.sh ============

# Start all EBS services 

DT=date +"%h %d, %y %H:%M"

RUNLOG="$HOME/xxx_scripts/reboot_scripts/reboot_start.log"

RUNLOG_ERR="$HOME/xxx_scripts/reboot_scripts/reboot_start_Error.log"

if [ -f $RUNLOG ]; then

mv $RUNLOG ${RUNLOG}_old

fi

if [ -f $RUNLOG_ERR ]; then

mv $RUNLOG_ERR ${RUNLOG_ERR}_old

fi

exec 1>$RUNLOG

exec 2>$RUNLOG_ERR

sleep 2

echo "Running at $DT"

. $HOME/.profile

. $HOME/xxx_scripts/.EBSpassenv

ps -ef | grep $LOGNAME           # check current status of EBS services

# for R12.1

# $ADMIN_SCRIPTS_HOME/adstrtal.sh apps/$APPS_PWD@$TWO_TASK

# for R12.2

if [ $isMaster == "enabled" ]then      ## $isMaster is defined in .profile

{ echo apps ; echo $AAPS_PWD ; echo $WLS_ADMIN ; } | $ADMIN_SCRIPTS_HOME/adstrtal.sh @ -mode=allnodes -nopromptmsg

else

{ echo apps ; echo $AAPS_PWD ; echo $WLS_ADMIN ; } | $ADMIN_SCRIPTS_HOME/adstrtal.sh @ -msimode -nopromptmsg

fi

echo 'sleep 10 seconds'

sleep 10

exit 0

============= end ============

========== script auto_stopall.sh =========

# Stop all EBS services. It may take 3 minutes for all apps processes shutdown.

DT=date +"%h %d, %y %H:%M"

RUNLOG="$HOME/xxx_scripts/reboot_scripts/reboot_stop.log"

RUNLOG_ERR="$HOME/xxx_scripts/reboot_scripts/reboot_stop_Error.log"

if [ -f $RUNLOG ]; then

mv $RUNLOG ${RUNLOG}_old

fi

if [ -f $RUNLOG_ERR ]; then

mv $RUNLOG_ERR ${RUNLOG_ERR}_old

fi

exec 1>$RUNLOG

exec 2>$RUNLOG_ERR

echo "Running at $DT"

. $HOME/.profile

. $HOME/xxx_scripts/.EBSpassenv

ps -ef | grep $LOGNAME

echo "shutting down ..."

# for R12.1

# $ADMIN_SCRIPTS_HOME/adstpall.sh apps/$APPS_PWD

{ echo apps ; echo $APPS_PWD ; echo $WLS_PWD ; } | $ADMIN_SCRIPTS_HOME/adstpall.sh @ -nopromptmsg

echo 'sleep 20 seconds'

sleep 20

PNUM=ps -ef | grep $LOGNAME | egrep -i 'FNDLIB|FNDSM' | wc -l 

if [ $PNUM -gt 1 ]; then

echo 'sleep 90 seconds more...'

sleep 90

fi

# only check upper case and assume $TWO_TASK is in the $ORACLE_HOME path

PNUM=ps -ef | grep -w $LOGNAME | egrep 'FNDLIB|FNDSM|'$TWO_TASK | wc -l 

if [ $PNUM -gt 1 ]; then

echo 'sleep 30 seconds'

sleep 30

fi

PNUM=ps -ef | grep -w $LOGNAME | egrep 'FNDLIB|FNDSM|'$TWO_TASK | wc -l 

if [ $PNUM -gt 1 ]; then

echo 'sleep 30 seconds more ...'

sleep 30

fi

PNUM=ps -ef | grep -w $LOGNAME | egrep 'FNDLIB|FNDSM|'$TWO_TASK | wc -l 

if [ $PNUM -gt 1 ]; then

echo 'sleep 15 seconds more ...'

sleep 15

fi

ps -ef | grep $LOGNAME

exit 0

============= end ============

Use .profile in Linux to customize the shell prompt

When you have many EBS instances in a multi-nodes environment, it will be very useful to let the Linux prompt display current user ID, server name and the path location. A custom .profile saved under $HOME works for me very well. Its colors tell if you are in a Admin node or not, and if you are in a production environment or not (assume the last character of production server's name is "p").

For a Linux account, environment variable $HOME is defined by file /etc/passwd. But, if the account was created by AD (Active Directory), $HOME is defined in AD by "Home Directory".

Our EBS applMgr accounts use Korn shell which uses two startup files under $HOME, the .profile and the .kshrc. During a session start, .profile is first read once, then .kshrc (if it exists) is read by each new ksh. e.g.:

$ echo $SHELL

/bin/ksh

$ echo $0

-ksh

$ which ksh

/usr/bin/ksh

$ more .kshrc

alias ftp="print 'Reminder: Use sftp instead of \\\ftp'"

echo "This is .kshrc"

$ ksh

This is .kshrc

$ ftp

Reminder: Use sftp instead of \ftp

============= $HOME/.profile =============

PATH=/bin:/usr/bin:/usr/local/bin

export PATH

MANPATH=/usr/share/man:/usr/local/share/man

export MANPATH      # for man manual 

EDITOR=/bin/vi

export EDITOR

# ENV=$HOME/.kshrc

# export ENV

. /u02/app/EBSPROD/EBSapps.env RUN     # R12.2 env file

. /u02/app/xxx_scripts/.EBSpassenv              # password file (custom)

isMaster="no"

if [ ! -z $APPS_VERSION ] && [ ${APPS_VERSION:0:4} == "12.2" ]

then

s_status=cat $CONTEXT_FILE | grep -i s_adminserverstatus

isMaster="${s_status:60:7}"

fi

if [ $isMaster == "enabled" ]   # on admin/primary node

then

if [ echo -n ${HOSTNAME%%.*} | tail -c -1 != "p" ]   

             # last character of server name is not "p" => non-production server

then

PS1=$'

\e[0;31m$USER@${HOSTNAME%%.}[$TWO_TASK]\e[m$PWD

-->$ '  

else       # on production server: Red, and Green color on PWD

PS1=$'

\e[0;31m$USER@${HOSTNAME%%.}[$TWO_TASK]\e[m\E[32m$PWD \E[0m

-->$ '

fi

else                                          # on other node(s)

if [ echo -n ${HOSTNAME%%.*} | tail -c -1 != "p" ]   

            # on non-production server

then

PS1='

$USER@${HOSTNAME%%.}[$TWO_TASK]$PWD

-->$ '

else      # on production server

PS1=$'

$USER@${HOSTNAME%%.}[$TWO_TASK]\E[32m$PWD \E[0m

-->$ '

fi

fi

alias rm='rm -i'

stty erase ^?

umask u=rwx,g=rwx,o=rx

================ end =================

On an Admin node in production env, the prompt looks like this:

userID@server_1p[EBSPROD]/u02/app

-->$

userID@server_1p[EBSPROD]/u02/app

-->$ echo $USER

userID

userID@server_1p[EBSPROD]/u02/app

-->$ echo $TWO_TASK

EBSPROD

userID@server_1p[EBSPROD]/u02/app

-->$ cd $TWO_TASK

userID@server_1p[EBSPROD]/u02/app/EBSPROD

-->$ echo $HOME

/u02/app

userID@server_1p[EBSPROD]/u02/app/EBSPROD

-->$ ls

EBSapps.env   fs1   fs2   fs_ne

script to check if a password is expiring

The environment variable $HOME for a Linux account is defined by file /etc/passwd in almost all servers' setups. Each account has an entry line in file /etc/passwd. For example, I can get my account's password expiration date by: 

$ echo $HOME

/u02/app

$ whoami

userID

$ grep userID /etc/passwd

userID:x:50378:102:Oracle EBS ID - J Y:/u02/app:/bin/ksh

$ expstr=$( chage -l $(whoami) | grep "^Password expires" | awk -F: '{ print $(NF) }' | sed -e 's/^ *//g; s/ *$//g;' )

$ echo "password for account `whoami` will expire on $expstr"

password for account userID will expire on Jul 30, 2025

But, if the account was created by AD (Active Directory), the variable $HOME is defined in AD by "Home Directory". ADHelp search page may show info:

    Unix Account

Home Directory:   /users/userID

Login Shell:          /bin/ksh

In that case, "chage" will give a different result:

$ echo $HOME

/users/userID

$ expstr=$( chage -l $(whoami) | grep "^Password expires" | awk -F: '{ print $(NF) }' | sed -e 's/^ *//g; s/ *$//g;' )

chage: user 'userID' does not exist in /etc/passwd

For an important account created in Linux, I wrote a script to email warning out before its password expires if it is a Linux account (vs. an AD account). It can be run by a cron job, such as

30 12 * * * /path/to/xxxx_scripts/checkPWDexpire.sh 2>&1

============= script checkPWDexpire.sh =============

let secs_per_day=606024

nowtime=$( date +%s )

expstr=$( chage -l $(whoami) | grep "^Password expires" | awk -F: '{ print $(NF) }' | sed -e 's/^ *//g; s/ *$//g;' )

echo "DEBUG: expstr is $expstr"

if [ "$expstr" == "never" ]; then

echo "Password never expires.";

exit 0;

fi

exptime=$( date --date "$expstr" +%s )

if [ "$exptime" -lt 1 ];        then

echo "Something is wrong.";

exit 255;   # Or, email a message out

fi

if [ "$exptime" -lt "$nowtime" ]; then

echo "Password already expired.";

exit 1;      # Or, email a message out

fi

secs_til_exp=$(expr $exptime - $nowtime)

days_til_exp=$(expr $secs_til_exp / $secs_per_day)

echo "Password expires in $days_til_exp days."

if [ "$days_til_exp" -lt 6 ]; then

# send email out

echo "Please reset manually and update PPM & Venafi environments." | mailx -s "`whoami` on `uname -n`  will expire in $days_til_exp days" me@email.com

# or 

# mailx -s "`whoami` on `uname -n`  will expire in $days_til_exp days" me@email.com < aFile.log

else

echo "All is fine.";

exit ;

fi

============== end =====================

Shell script for renewing ssl certificate

My post Re-new R12.2 ssl certificate has details on how to renew a certificate. A shell script helps a lot when there are many EBS instances waiting for renewal. I wrote below script which takes only one minute to renew the cert on each node after the certificate is renewed on Venafi website and downloaded/copied to Linux server. 

As of today, we still have difficulties using .yaml file to extract certificate from Venafi server to Linux server automatically. We tried to set up a "push" way on Venafi website to do the automation. But if the password is changed on the Linux account, the push will fail. 

============= Script renew_cert.sh ============

# Script for renewing ssl certificate after new cert file is saved to Linux server

walletpwd='putPWDhere'

# walletpwd='tttest'

walletloc=$HOME/xxx/Certs_Renew   # path where the Venafi cert file is saved

walletname='ewallet.p12'                # Must name Venafi cert file to this name

certname='cwallet.sso'

echo "cert at: $walletloc"

echo "cert name: $walletname"

echo $walletpwd

cd $walletloc

errorC=`env| grep RUN_BASE | wc -l`

if [ $errorC -lt 1 ]; then

  echo "No R12.2 environment"

  exit 1

  # . $HOME/EBSQA/EBSapps.env RUN

fi

alias orapki=$FMW_HOME/oracle_common/bin/orapki

orapki wallet display -wallet $walletloc/$walletname -pwd $walletpwd > viewCert.log

errorC=`egrep -i 'PKI-' viewCert.log | wc -l`

echo "Error: $errorC"

if [ $errorC -gt 0 ]; then

   echo "The password is incorrect or the Venafi cert file is incorrect."

   exit 2

fi

DT=`date +"%h_%d_%y_%H%M"`

if [ -f $certname ]; then

   mv $certname ${certname}_${DT}

fi

orapki wallet create -wallet $walletloc/$walletname -pwd $walletpwd -auto_login

if [ ! -f $certname ]; then

   echo "Failure in getting new cert file. Exiting."

   exit 3

fi

echo " "

echo "Copy cert file to directories ..."

cd $NE_BASE/inst/$CONTEXT_NAME/certs    # save a copy in this folder

if [ -d Apache ]; then

mv Apache Apache_${DT}

fi

mkdir Apache

cd Apache

pwd

cp -p $walletloc/$walletname ${walletname}

cp -p $walletloc/$certname ${certname}

iName=$(tr < $CONTEXT_FILE '<>' '  ' | awk '/"s_ohs_instance"/ {print $(NF-1)}' )

SUBiName=${iName%?????}

cd $FMW_HOME/webtier/instances/$iName/config/OPMN/opmn/wallet

pwd

if [ -f $certname ]; then

   mv $certname ${certname}_${DT}

fi

cp -p $walletloc/$certname ${certname}

cd $FMW_HOME/webtier/instances/$iName/config/OHS/$SUBiName/keystores/default

pwd

if [ -f $certname ]; then

   mv $certname ${certname}_${DT}

fi

cp -p $walletloc/$certname ${certname}

cd $FMW_HOME/webtier/instances/$iName/config/OHS/$SUBiName/proxy-wallet

pwd

if [ -f $certname ]; then

   mv $certname ${certname}_${DT}

fi

cp -p $walletloc/$certname ${certname}

echo " "

echo "Recycle Apache service..."

cd $ADMIN_SCRIPTS_HOME

./adopmnctl.sh stop

sleep 10

./adopmnctl.sh status

./adapcctl.sh start

./adopmnctl.sh status

echo "Paths for log files:"

echo $FMW_HOME/webtier/instances/$iName/diagnostics/logs/OHS/$SUBiName

echo $FMW_HOME/webtier/instances/$iName/diagnostics/logs/OPMN/opmn

cd

============ End ==========

Run the script to renew certificate on each node:

$ ./renew_cert.sh

cert at: $HOME/temp/Certs_Renew

cert name: ewallet.p12

putPWDhere

Error: 0

Oracle PKI Tool : Version 11.1.1.9.0

Copyright (c) 2004, 2015, Oracle and/or its affiliates. All rights reserved.

Copy cert file to directories ...

/u04/app/EBSQA/fs_ne/inst/EBSQA_nodeName/certs/Apache

$FMW_HOME/webtier/instances/EBS_web_EBSQA_OHS1/config/OPMN/opmn/wallet

$FMW_HOME/webtier/instances/EBS_web_EBSQA_OHS1/config/OHS/EBS_web_EBSQA/keystores/default

$FMW_HOME/webtier/instances/EBS_web_EBSQA_OHS1/config/OHS/EBS_web_EBSQA/proxy-wallet

Recycle Apache service ...

You are running adopmnctl.sh version 120.0.12020000.2

Stopping Oracle Process Manager (OPMN)  and the managed processes ...

opmnctl stopall: stopping opmn and all managed processes...

adopmnctl.sh: exiting with status 0

adopmnctl.sh: check the logfile $LOG_HOME/appl/admin/log/adopmnctl.txt for more information ...  

You are running adopmnctl.sh version 120.0.12020000.2

Checking status of OPMN managed processes...

opmnctl status: opmn is not running.

adopmnctl.sh: exiting with status 0

adopmnctl.sh: check the logfile $LOG_HOME/appl/admin/log/adopmnctl.txt for more information ...  

You are running adapcctl.sh version 120.0.12020000.6

Starting OPMN managed Oracle HTTP Server (OHS) instance ...

adapcctl.sh: exiting with status 0

adapcctl.sh: check the logfile $LOG_HOME/appl/admin/log/adapcctl.txt for more information ...  

You are running adopmnctl.sh version 120.0.12020000.2

Checking status of OPMN managed processes...

Processes in Instance: EBS_web_ARQA_OHS1

--------------------------------+--------------------+---------+---------

ias-component                    | process-type       |     pid    | status  

--------------------------------+--------------------+---------+---------

EBS_web_EBSQA             | OHS                   |   14542 | Alive   

adopmnctl.sh: exiting with status 0

adopmnctl.sh: check the logfile $LOG_HOME/appl/admin/log/adopmnctl.txt for more information ...  

Paths for log files:

$FMW_HOME/webtier/instances/EBS_web_EBSQA_OHS1/diagnostics/logs/OHS/EBS_web_EBSQA

$FMW_HOME/webtier/instances/EBS_web_EBSQA_OHS1/diagnostics/logs/OPMN/opmn

Check files in folder $HOME/xxx/Certs_Renew:

$ ls 

renew_cert.sh

ewallet.p12

ewallet.p12.lck

cwallet.sso.lck

viewCert.log

cwallet.sso

NOTES: there is a cert file in $EBS_DOMAIN_HOME/opmn/EBS_web_EBSQA_OHS1/wallet and $EBS_DOMAIN_HOME/opmn/EBS_web_EBSQA_OHS1/EBS_web/wallet. But I do not know what uses them.