EBS forms failed by CrowdStrike

EBS Forms in our financial applications suddenly does not work. The message on the webpage is

Failure of Web Server bridge:

No backend server available for connection: timed out after 10 seconds or idempotent set to OFF or method not idempotent. 

 

The error message does not tell the true problem. When checking into services on OS level, I saw Oracle EBS Forms service was not running and also saw errors from startup script $ADMIN_SCRIPTS_HOME/adstrtal.sh:

Forms service failed to start. 

The Node Manager is already up.

ERROR: Unable to start up the managed server forms_server1

Server specific logs are located at $EBS_DOMAIN_HOME/servers/forms_server1/logs

05/13/24-20:56:26 :: admanagedsrvctl.sh: exiting with status 1

Java error exists in Forms log file $EBS_DOMAIN_HOME/servers/forms_server1/logs/forms_server1.out

<May 13, 2024 8:56:25 PM EDT> <Emergency> <Store> <BEA-280060> <The persistent store "_WLS_forms_server1" encountered a fatal error, and it must be shut down: weblogic.store.PersistentStoreFatalException: [Store:280020]There was an error while reading from the log file

weblogic.store.PersistentStoreFatalException: [Store:280020]There was an error while reading from the log file

        at weblogic.store.io.file.FileStoreIO.open(FileStoreIO.java:128)

        at weblogic.store.internal.PersistentStoreImpl.recoverStoreConnections(PersistentStoreImpl.java:435)

        at weblogic.store.internal.PersistentStoreImpl.open(PersistentStoreImpl.java:423)

        at weblogic.store.admin.AdminHandler.activate(AdminHandler.java:126)

        at weblogic.store.admin.FileAdminHandler.activate(FileAdminHandler.java:207)

        Truncated.

Caused By: java.io.EOFException: premature EOF: expected=512, actual=126

        at weblogic.store.io.file.StoreFile.readBulk(StoreFile.java:316)

        at weblogic.store.io.file.Heap.readStoreFile(Heap.java:1142)

        at weblogic.store.io.file.Heap.getNextRecoveryFile(Heap.java:1226)

        at weblogic.store.io.file.Heap.open(Heap.java:373)

        at weblogic.store.io.file.FileStoreIO.open(FileStoreIO.java:117)

        Truncated.

Seems WebLogic failed to open a file, but the log did not say which file. I knew that Linux Admins just did server maintenance and rebooted server after they applied monthly patches and Security updates on OS level. That was the only change in the application environment recently.

After searching around, I found the Java errors match the description in Oracle Doc ID 3017110.1 ( Managed Forms Server Fails To Start - Displaying Message: FAILED_NOT_RESTARTABLE - ERROR: <BEA-280061> The persistent store "_WLS_forms_server1" could not be deployed: weblogic.store.PersistentStoreFatalException [Store:280020] ). 

The document points out the problem is caused by CrowdStrike, which locks a Forms file in $EBS_DOMAIN_HOME/servers/forms_server#/data/store/default.

CrowdStrike is installed in /opt/CrowdStrike. It is owned by root, and it is running constantly on the Linux server.

$ ps -ef | grep falcon-sensor

root      1081  1079  0 May13 ?        00:22:23 falcon-sensor

The problem can be fixed temporarily by a workaround:

1. Delete/rename below .DAT file (I guess CrowdStrike does not like the file name and so locks it)

$ cd $EBS_DOMAIN_HOME/servers/forms_server1/data/store/default

$ ls -altr

total 1028

drwxr-xr-x 4 user group      40 Sep 13  2023 ..

-rw-r--r-- 1   user group  1049088 May 13 20:51 _WLS_FORMS_SERVER1000000.DAT

drwxr-xr-x 2 user group     42 May 13 20:56 .

$ rm _WLS_FORMS_SERVER1000000.DAT

2. Re-start services cleanly by

$ADMIN_SCRIPTS_HOME/adstrtal.sh

Or

$ADMIN_SCRIPTS_HOME/admanagedsrvctl.sh start forms_server1

The permanent fix is that the Secure team rolls back the CrowdStrike change (and applies it again until CrowdStrike fixes the problem), because its new update touches an Oracle Forms data file wrongly during its scan. 

How to run AutoConfig on PATCH file system in R12.2

 Steps for running AutoConfig on PATCH file system:

1) Disable a trigger

SQL> conn system/systemPWD

Connected.

SQL> alter trigger ebs_logon disable;

Trigger altered.

2) Set PATCH env and connect to database

$ . /<EBS_HOME_BASE>/EBSapps.env patch

$ echo $TWO_TASK

EBSDEV_patch

$ sqlplus apps/appsPWD  

    -- If it got error, compare tnsnames.ora in both RUN and PATCH file systems &

    -- modifying tnsnames.ora in PATCH file system may be needed in Oracle 19c db. 

SQL> show user

USER is "APPS"

SQL> exit

3) Run AutoConfig in PATCH file system

$ cd $ADMIN_SCRIPTS_HOME

$ echo $FILE_EDITION

patch

$ ./adautocfg.sh

Enter the APPS user password:

The log file for this session is located at: $INST_TOP/admin/log/MMDDHHMI/adconfig.log

... ...

AutoConfig completed successfully.

4). Enable the trigger

SQL> conn system/systemPWD

Connected.

SQL> alter trigger ebs_logon enable;

Trigger altered.

Port conflict during R12.2 apps clone

If Target mid-tier and Source mid-tier are on the same server, EBS clone script adcfgclone.pl may fail with port conflict error. 

Prots used for an EBS instance is listed in file $INST_TOP/admin/out/portpool.lst. Depending on which port has conflict on the server, the cloning error message will be different.  

- If the port for httpd is busy and used by another environment on the server, the cloning script will stop on Target instance with errors:

ERROR: Failed to configure the target system,

please check the logfile in : $RUN_BASE/inst/apps/$CONTEXT_NAME/admin/log/clone

CLONE-26003   Error in validating listen host and port. 

CLONE-26176  In config group httpd.conf , the value of "Listen" config property was xxxx. xxxx was not free. 

Log message shows the problem is from file httpd.conf. When I checked the port number and compare it in file $CONTEXT_FILE and in file $FMW_HOME/webtier/instances/EBS_web_<SID>_OHS1/config/OHS/EBS_web_<SID>/httpd.conf. 

$ grep Listen httpd.conf

$ grep s_http_listen_parameter $CONTEXT_FILE

I saw they are different in Source instance. Even they are not the same, Apache still works fine on Source instance and httpd.worker processes occupy the port number in httpd.conf. After the clone script copied file httpd.conf to Target instance, Apache failed to start because OHS was running on the same port as the other instance

Apparently httpd.conf of Source instance was edited by picking up randomly a port number. httpd.conf can be edited and updated manually as AutoConfig in R12.2 does not update it. For more details on modifying port values for OHS, see Doc ID 1905593.1 (Managing Configuration of Oracle HTTP Server and Web Application Services in Oracle E-Business Suite Release 12.2).

- While cloning script uses a temporary port but it was not available, the error could be 

CLONE-20372 Server port validation failed.

The fix could be just shutdown apps services of Source instance to let the cloning complete. See Doc ID 2002613.1

- Doc ID 2437111.1 gives a fix on port for s_ohs_adminport. 

On Linux server, use command to find if a port is used or not:

$ netstat -tuanp | grep 6230

(Not all processes could be identified, non-owned process info

will not be shown, you would have to be root to see it all.)

tcp    0 0 167.69.109.82:6230 0.0.0.0:*   LISTEN 

Then the process owner (or root) can use below line to find what process is using the port: 

$ lsof -i :6230 

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME

opmn 31876 ebsdev 9u IPv4 42886593 0t0 TCP server3d.domain.com:6230 (LISTEN)

After adcfgclone.pl failed, you have to remove/rename folders BEFORE re-run clone script adcfgclone.pl 

- Remove two new folders under $RUN_BASE (or only folder $RUN_BASE/FMW_Home, depending on the failure stage) to avoid error "Exiting cloning as FMW Home already exists".

- Also remove the folder defined by inventory_loc in file /etc/oraInst.loc to avoid error "Oracle Homes are already registered in the inventory". If the inventory is just for one environment, do not try to detach a home because it may give error:

$ ./runInstaller -detachhome $FMW_HOMEe/oracle_common

Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 13519 MB    Passed

The inventory pointer is located at /etc/oraInst.loc

The inventory is located at ... ...

The operation failed as it was called without name of the Oracle Home being attached.

- Since the script failed before the completion, there is not need to remove info in the database.

Apply R12.2 July 2023 CPU patches

Oracle E-Business Suite Release 12.2 Critical Patch Update Availability Document (July 2023) (Doc ID 2953580.1) is the document for July 2023 CPU patches.

Check to find ATG_PF.C.delta.9 is the latest ATG (Application Technology Family) patch in my instances.

SQL> select * from ad_bugs where bug_number in (31856779, 30399994, 33527666);

30399994 Patch R12.ATG_PF.C.delta.9 (Application Technology Family)

31856779 Patch R12.ATG_PF.C.delta.10 

33527666 Patch R12.ATG_PF.C.Delta.11 

                 (Released on Jan 9 2023. See Doc ID 2876714.1 on how to apply it)

1. Pre-requisites: R12.AD.C.Delta.14 and R12.TXK.C.Delta.14 patches

Follow steps in Apply AD-TXK Delta 14 RUPs to apply them. After all are done, 8 patches were applied:

SQL> SELECT adb.bug_number, aas.name appl_top_name, adb.creation_date, adb.language,

decode(ad_patch.is_patch_applied('R12',aas.appl_top_id,adb.bug_number,adb.language),'EXPLICIT','APPLIED','NOT_APPLIED','NOT APPLIED') status

FROM ad_bugs adb,

(select distinct appltop_id appl_top_id, node_name name from ad_adop_sessions 

           where node_name in (select node_name from ADOP_VALID_NODES) ) aas

where adb.bug_number in (

'33600809',  -- R12.AD.delta.C.14

'34668508',

'34669333',

'34681299',

'35280947',

'33602997',  -- R12.TXK.delta.C.14

'34708635',

'34654260'

) order by creation_date desc, adb.bug_number,aas.name,adb.language;

2. Apply July 2023 CPU patch 35385938. It takes about 30 minutes in the multi-node.

First of all, download all EBS patches to a shared location /a46/CPUs/CPU_July2023

$ echo $FILE_EDITION

run

$ adop phase=apply apply_mode=downtime patches=35385938 patchtop=/a46/CPUs/CPU_July2023/CPU

3. Run autoconfig

$ perl $AD_TOP/bin/admkappsutil.pl

$ cp -p $INST_TOP/admin/out/appsutil.zip $APPLPTMP

Autoconfig on db node first

Autoconfig on all apps nodes

NOTES: After CPU patching, custom .tmp file in $FND_TOP/admin/template/custom may be out-of-date and needs an update if existing to avoid autoconfig fails on apps nodes.

4. Post patches:  It turns out that it needs to apply only 5 post patches in my instances.

$ adop phase=apply apply_mode=downtime patches=34212478,34556525,34291981,35181823,35355008 patchtop=/a46/CPUs/CPU_July2023/posts

After that, verify all 11 patches were applied:

SQL> SELECT adb.bug_number, aas.name appl_top_name, adb.creation_date, adb.language,

decode(ad_patch.is_patch_applied('R12',aas.appl_top_id,adb.bug_number,adb.language),'EXPLICIT','APPLIED','NOT_APPLIED','NOT APPLIED') status

FROM ad_bugs adb,

(select distinct appltop_id appl_top_id, node_name name from ad_adop_sessions 

           where node_name in (select node_name from ADOP_VALID_NODES) ) aas

where adb.bug_number in (

'35181823',    -- post patch   (Table 1)

'35355008',    -- post patch   (Table 1)

-- '33207251',  -- Post-CPU EBS Security Patches  (Table 2)

-- '32980025',  -- Post-CPU EBS Security Patches

-- '33286000',  -- Post-CPU EBS Security Patches

'34212478',     -- Post-CPU EBS Security Patches

'34556525',     -- Post-CPU EBS Security Patches

'34291981'      -- Post-CPU EBS Security Patches

-- '33625264',  -- Post-CPU EBS Security Patches

-- '33862025',  -- needed for AD.C.Delta.13 and TXK.Delta.13. Comes after CPU 35385938

-- '33568131'   -- needed for AD.C.Delta.13 and TXK.Delta.13. Comes after CPU 35385938

) order by creation_date desc, adb.bug_number,aas.name,adb.language;

Optional: start apps services to verify all work.

6. Run ETCC script checkMTpatch.sh to get the list of patches asked by ETCC (patch 17537119). But, for WebLogic, Table 4 of Doc ID 2953580.1 asks to apply Oracle WebLogic Server 10.3.6.0 patch 34900723 (WLS PSU 10.3.6.0.230418). 

Download directly patch file p34900723_1036_Generic.zip and copy it to $FMW_HOME/utils/bsu/cache_dir.

$ cd $FMW_HOME/utils/bsu/cache_dir

$ unzip -o p34900723_1036_Generic.zip

Archive:  p34900723_1036_Generic.zip

 extracting: P8S7.jar

  inflating: patch-catalog_27964.xml

  inflating: README.html

  inflating: README.txt

$ cd $FMW_HOME/utils/bsu

$ ./bsu.sh -install -patch_download_dir=$FMW_HOME/utils/bsu/cache_dir -patchlist=P8S7 -prod_dir=$FMW_HOME/wlserver_10.3

Checking for conflicts..

Conflict(s) detected - resolve conflict condition and execute patch installation again

Conflict condition details follow:

Patch P8S7 is mutually exclusive and cannot coexist with patch(es): HYG5,TNS1,IJC1,7GCA

$ ./bsu.sh -remove -patchlist=IJC1,7GCA,TNS1 -prod_dir=$FMW_HOME/wlserver_10.3

Checking for conflicts..

No conflict(s) detected

Removing Patch ID: IJC1.

Result: Success

Removing Patch ID: 7GCA..

Result: Success

Removing Patch ID: TNS1..

Result: Success

$ ./bsu.sh -remove -patchlist=HYG5 -prod_dir=$FMW_HOME/wlserver_10.3

Checking for conflicts..

No conflict(s) detected

Removing Patch ID: HYG5..

Result: Success

$ ./bsu.sh -install -patch_download_dir=$FMW_HOME/utils/bsu/cache_dir -patchlist=P8S7 -prod_dir=$FMW_HOME/wlserver_10.3

Checking for conflicts..

No conflict(s) detected

Installing Patch ID: P8S7..

Result: Success

-- verify it

$ ./bsu.sh -prod_dir=$FMW_HOME/wlserver_10.3 -status=applied -verbose -view | grep P8S7

Patch ID:          P8S7

PatchContainer:    P8S7.jar

6. Run ETCC script checkMTpatch.sh to identify missing technology patches. 

Starting Application Tier Technology Codelevel Checker

Version: 120.0.12020000.65

Bugfix XML file version: 120.0.12020000.61

This file will be used for identifying missing bugfixes.

Mapping XML file version: 120.0.12020000.49

This file will be used for mapping bugfixes to patches.

... ...

The default patch recommendations to install these missing bugfixes are:

-------------------------------------------------------------------------------

Oracle Fusion Middleware (FMW) - Web Tier 11.1.1.9.0

-------------------------------------------------------------------------------

  Patch 35540062

    - Filename: p35540062_111190_Linux-x86-64.zip

  Patch 34067016

    - Filename: p34067016_11119210420OSS_Linux-x86-64.zip

-------------------------------------------------------------------------------

Oracle Fusion Middleware (FMW) - oracle_common 11.1.1.9.0

-------------------------------------------------------------------------------

  Patch 34330735

    - Filename: p34330735_111190_Generic.zip

-------------------------------------------------------------------------------

Oracle WebLogic Server (WLS) 10.3.6.0.230418

-------------------------------------------------------------------------------

  Patch 13845626 [SU Patch [AMGE]]

    - Filename: p13845626_10360230418_Generic.zip

  Patch 34856303 [SU Patch [ENUU]]

    - Filename: p34856303_1036_Generic.zip

  Patch 31042881 [SU Patch [7BIA]]

    - Filename: p31042881_1036_Generic.zip

7. Continue to apply WLS patches (after 3 patches files were downloaded and copied to $FMW_HOME/utils/bsu/cache_dir)

$ ./bsu.sh -install -patch_download_dir=$FMW_HOME/utils/bsu/cache_dir -patchlist=AMGE -prod_dir=$FMW_HOME/wlserver_10.3

Checking for conflicts..

No conflict(s) detected

Installing Patch ID: AMGE.

Result: Success

$ ./bsu.sh -install -patch_download_dir=$FMW_HOME/utils/bsu/cache_dir -patchlist=ENUU -prod_dir=$FMW_HOME/wlserver_10.3

Checking for conflicts..

Conflict(s) detected - resolve conflict condition and execute patch installation again

Conflict condition details follow:

Patch ENUU is mutually exclusive and cannot coexist with patch(es): DN1F

$ ./bsu.sh -remove -patchlist=DN1F -prod_dir=$FMW_HOME/wlserver_10.3

Checking for conflicts..

No conflict(s) detected

Removing Patch ID: DN1F.

Result: Success

$ ./bsu.sh -install -patch_download_dir=$FMW_HOME/utils/bsu/cache_dir -patchlist=ENUU -prod_dir=$FMW_HOME/wlserver_10.3

Checking for conflicts..

No conflict(s) detected

Installing Patch ID: ENUU.

Result: Success

$ ./bsu.sh -install -patch_download_dir=$FMW_HOME/utils/bsu/cache_dir -patchlist=7BIA -prod_dir=$FMW_HOME/wlserver_10.3

Checking for conflicts..

No conflict(s) detected

Installing Patch ID: 7BIA.

Result: Success

8. Apply patches from ETCC list to FMW homes.

Apply 35540062 and 34067016 to Oracle Fusion Middleware (FMW) - Web Tier

Note: 35540062 (OHS Security Patch Update JulCPU2023 Patch) will roll back 34867865 (JanCPU2023 Patch) if it was applied before.

$ echo $IAS_ORACLE_HOME

$ export ORACLE_HOME=$IAS_ORACLE_HOME

$ echo $ORACLE_HOME          <== $FMW_Home/webtier

$ export PATH=$IAS_ORACLE_HOME/OPatch:$PATH

$ which opatch

$ echo $FILE_EDITION

run

$ opatch lsinventory | grep 31304503

$ opatch lsinventory

Interim patches (17)     <== depending on patching history

$ cd /a46/CPUs/CPU_July2023/tech

$ unzip p34067016_11119210420OSS_Linux-x86-64.zip

$ cd 34067016

$ opatch apply

... ...

Patching component oracle.ldap.rsf, 11.1.1.9.0...

Patching component oracle.rdbms.rsf, 11.1.0.7.0...

Verifying the update...

Patch 34067016 successfully applied

$ cd ..

$ rm -rf 34067016

$ cd 35540062

$ opatch apply

Patching component oracle.ohs2, 11.1.1.9.0...

RollbackSession removing interim patch '33311587' from inventory

OPatch back to application of the patch '35540062' after auto-rollback.

Patching component oracle.ohs2, 11.1.1.9.0...

Verifying the update...

Patch 35540062 successfully applied

$ cd ..

$ rm -rf 35540062

$ opatch lsinventory

Interim patches (18) 

.. ...

9. Apply patch 34330735 to Oracle Fusion Middleware (FMW) - Common

$ export ORACLE_HOME=$FMW_HOME/oracle_common

$ export PATH=$ORACLE_HOME/OPatch:$PATH

$ echo $ORACLE_HOME

.../FMW_Home/oracle_common

$ which opatch

.../FMW_Home/oracle_common/OPatch/opatch

$ cd /a46/CPUs/CPU_July2023/tech

$ opatch lsinventory

Interim patches (14)

$ unzip p34330735_111190_Generic.zip

$ cd 34330735/oui

$ ls

$ opatch apply

... ...

RollbackSession removing interim patch '31985571' from inventory

OPatch back to application of the patch '34330735' after auto-rollback.

Patching component oracle.jrf.adfrt, 11.1.1.9.0...

Verifying the update...

Patch 34330735 successfully applied

cd ..

rm -rp 34330735

$ opatch lsinventory

Interim patches (14)

By now ETCC script shows all patches were applied.

10. Upgrade JDK to 1.7.0_391

See Upgrade JDK 7 in EBS R12.2 for detail on upgrading JDK. Table 3 in CPU release document 2953580.1 shows JDK 1.7.0_391 is the latest JDK for EBS. Use the link in Doc ID 2957260.1 (Oracle Critical Patch Update (CPU) Jul 2023 for Oracle Java SE) to download patch 35374159 files  p35374159_170391_LINUX.zip & p35374159_170391_Linux-x86-64.zip and save them to /a46/CPUs/CPU_July2023/jdk. 

$ unzip p35374159_170391_Linux-x86-64.zip

$ tar vzxfp jdk-7u391-linux-x64.tar.gz

$ mv jdk1.7.0_391 jdk_391_64

$ unzip p35374159_170391_LINUX.zip

$ tar vzxfp jdk-7u391-linux-i586.tar.gz

$ mv jdk1.7.0_391 jdk_391_32

$ echo $FILE_EDITION

run

JDKfolder=/a46/CPUs/CPU_July2023/jdk

cd $COMMON_TOP/util

tar -cvzf jdk64_BK.tar.gz jdk64

tar -cvzf jdk32_BK.tar.gz jdk32

rm -fr jdk64

rm -fr jdk32

cp -rp $JDKfolder/jdk_391_64 jdk64

cp -rp $JDKfolder/jdk_391_32 jdk32

cd $FMW_HOME/webtier

tar -cvzf jdk64_BK.tar.gz jdk

rm -fr jdk

cp -rp $JDKfolder/jdk_391_64 jdk

cd $ORACLE_HOME

ls

tar -cvzf jdk32_BK.tar.gz jdk

ls -al

rm -fr jdk

cp -rp $JDKfolder/jdk_391_32 jdk

$ADJVAPRG -version

$AFJVAPRG -version

cd $ORACLE_HOME/forms/lib

make -f ins_forms.mk sharedlib install

cd $ORACLE_HOME/reports/lib

make -f ins_reports.mk install

11. Upgrade JRE (Java Run-time Environment)

See 4) in Enable JWS with EBS R12.2 for JRE upgrade. Doc ID 2957260.1 also gives a link to download JRE 8 Update 381 (patch 35374151). I used file p35374151_180381_WINNT.zip (for Windows 32-bit).

$ cd /a46/CPUs/CPU_July2023/jre

$ unzip p35374151_180381_WINNT.zip

Archive:  p35374151_180381_WINNT.zip

  inflating: jre-8u381-windows-i586.exe

  inflating: jre-8u381-windows-i586.tar.gz

  inflating: readme.txt

$ cp jre-8u381-windows-i586.exe $COMMON_TOP/webapps/oacore/util/javaplugin/j2se18381.exe

$ cd $COMMON_TOP/webapps/oacore/util/javaplugin

$ ls -al

$ echo $FILE_EDITION

run

$ $FND_TOP/bin/txkSetPlugin.sh 18381

 

$ grep sun $CONTEXT_FILE

         <sun_plugin_ver oa_var="s_sun_plugin_ver">1.8.0_381</sun_plugin_ver>

         <sun_plugin_type oa_var="s_sun_plugin_type">jdk</sun_plugin_type>

         <sun_clsid oa_var="s_sun_clsid">CAFEEFAC-0018-0000-0381-ABCDEFFEDCBA</sun_clsid>

12. Optional: Re-generate and sign JAR files.

$ adadmin   (1 => 4 => Yes)

13. Start Apps services for test

$ adstrtal.sh apps/xxxx

14. adop phase=fs_clone  (sync RUN file system to PATCH file system)

vulnerabilities by Qualys report

Qualys Scan reports huge number of vulnerabilities in our EBS R12.2.10 servers where JDK was upgraded to 1.7.0_331. Vulnerability issue became a priority task for us due to corporate security requirement.

$ $ADJVAPRG -version

java version "1.7.0_331"

Java(TM) SE Runtime Environment (build 1.7.0_331-b06)

Java HotSpot(TM) Server VM (build 24.331-b06, mixed mode)

$ $AFJVAPRG -version

java version "1.7.0_331"

Java(TM) SE Runtime Environment (build 1.7.0_331-b06)

Java HotSpot(TM) 64-Bit Server VM (build 24.331-b06, mixed mode)

Some vulnerabilities can be addressed by applying latest CPU patches, upgrading JDK and JRE (see details). 

Vulnerabilities also reported:

$COMMON_TOP/clone/FMW/t2pjdk/bin/java 1.7.0_85-b15 Enhanced

$COMMON_TOP/clone/FMW/t2pjdk/jre/bin/java 1.7.0_85-b15 Enhanced

(It is safe to delete above folders because they will be recreated by an adpreclone.pl run)

$COMMON_TOP/clone/jre/bin/java 1.7.0_331-b06 Enhanced  

(It is not in JDK folders and so a JDK upgrade will not fix it. But it may be recreated by an adpreclone.pl run)

$COMMON_TOP/adopclone_nodeName/FMW/t2pjdk/jre/bin/java 1.7.0_85-b15 Enhanced

$COMMON_TOP/adopclone_nodeName/FMW/t2pjdk/bin/java 1.7.0_85-b15 Enhanced

$COMMON_TOP/adopclone_nodeName/jre/bin/java 1.7.0_85-b15 Enhanced

($COMMON_TOP/adopclone_nodeName may get created/updated from ADOP run, such as fs_clone. nodeName is the EBS host, where ADOP (or adpreclone.pl?) ran. )

$APPL_TOP/msc/12.0.0/bin/SNO/scp/12.2/sno/installer_jre/bin/java 1.6.0_12-b04 Enhanced

$APPL_TOP/msc/12.0.0/bin/PS/scp/12.2/ps/jre/bin/java 1.8.0_92-b14 Enhanced

Oracle support confirmed that they can be deleted and some of them will be recreated next time adpreclone.pl is run. I wrote a script to clean them on RUN file system (and on PATCH file system or run FS_CLONE). 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

# delete Java files for vulnerabilities by Qualys report

curr=`pwd`

echo "current path: $curr"

cd $COMMON_TOP/clone/FMW/t2pjdk/bin

pwd

ls -al java

rm -f java

cd $COMMON_TOP/clone/FMW/t2pjdk/jre/bin

pwd

ls -al java

rm -f java

cd $COMMON_TOP/clone/jre/bin

pwd

ls -al java

rm -f java

cd $APPL_TOP/msc/12.0.0/bin/SNO/scp/12.2/sno/installer_jre/bin

pwd

ls -al java

rm -f java

cd $APPL_TOP/msc/12.0.0/bin/PS/scp/12.2/ps/jre/bin

pwd

ls -al java

rm -f java

# loop folder(s) $COMMON_TOP/adopclone_*

for x in `ls -ald $COMMON_TOP/adopclone_* |egrep "^drw" |awk '{print $9}'`

do

echo Under path: $x

cd $x/jre/bin

pwd

ls -al java

rm -f java

cd $x/FMW/t2pjdk/jre/bin

pwd

ls -al java

rm -f java

cd $x/FMW/t2pjdk/bin

pwd

ls -al java

rm -f java

done

cd $curr

exit