R12.2 software and StartCD 51 for repaid installation

The first task in upgrading EBS to R12.2 is to get the software ready. But it is not straight and easy. Here are steps to get started.

1. Create folder /path/to/R122, and add at lease 100GB disk space to /path/to partition.

2. Download ZIP files to /path/to/R122
See 12.2 E-Business Suite Applications Installation Rapid Install Steps To Create the Stage Directories In Preparation For A 12.2 Installation (Doc ID 1596433.1)

https://edelivery.oracle.com/osdc/faces/SoftwareDelivery
Click Popular Downloads => REL: Oracle E-Business Suite 12.2.0 Current => 23 ZIP files
Click on Download to get ORACLE DOWNLOAD MANAGER (then, enter your login info).

Oracle E-Business Suite 12.2.0 Current for Linux x86-64:
V35803-01_1of3.zip EBS R 12.2.0 for Linux x86-64 Rapid Install APPL_TOP - Disk 1, 1.2 GB
V35803-01_2of3.zip EBS R 12.2.0 for Linux x86-64 Rapid Install APPL_TOP - Disk 1, 1022.8 MB
V35803-01_3of3.zip EBS R 12.2.0 for Linux x86-64 Rapid Install APPL_TOP - Disk 1, 1.3 GB
V35804-01_1of2.zip EBS R 12.2.0 for Linux x86-64 Rapid Install APPL_TOP - Disk 2, 1.2 GB
V35804-01_2of2.zip EBS R 12.2.0 for Linux x86-64 Rapid Install APPL_TOP - Disk 2, 1.6 GB
V35802-01.zip  EBS R 12.2.0 for Linux x86-64 Rapid Install AS10.1.2, 711.6 MB
V100060-01_1of3.zip EBS R 12.2.0 for Linux x86-64 Rapid Install DBs PROD - Disk 2, 1.6 GB
V100060-01_2of3.zip EBS R 12.2.0 for Linux x86-64 Rapid Install DBs PROD - Disk 2, 1.1 GB
V100060-01_3of3.zip EBS R 12.2.0 for Linux x86-64 Rapid Install DBs PROD - Disk 2, 1.0 GB
V100061-01.zip  EBS R 12.2.0 for Linux x86-64 Rapid Install Technology one-off Patches, 839.9 MB
V100054-01.zip  EBS R 12.2.0 for Linux x86-64 Rapid Install Databases VISION - Disk 2, 3.0 GB
V100055-01.zip  EBS R 12.2.0 for Linux x86-64 Rapid Install Databases VISION - Disk 3, 2.8 GB
V100053-01_1of2.zip EBS R 12.2.0 for Linux x86-64 Rapid Install DBs VISION - Disk 1, 52.0 MB
V100053-01_2of2.zip EBS R 12.2.0 for Linux x86-64 Rapid Install DBs VISION - Disk 1, 3.4 GB
V100058-01.zip  EBS R 12.2.0 for Linux x86-64 Rapid Install DBs VISION - Disk 6, 2.0 GB
V100059-01_1of2.zip EBS R 12.2.0 for Linux x86-64 Rapid Install DBs PROD - Disk 1, 1007.0 MB
V100059-01_2of2.zip EBS R 12.2.0 for Linux x86-64 Rapid Install DBs PROD - Disk 1, 1.6 GB
V100056-01.zip  EBS R 12.2.0 for Linux x86-64 Rapid Install Databases VISION - Disk 4, 2.6 GB
V100057-01.zip  EBS R 12.2.0 for Linux x86-64 Rapid Install Databases VISION - Disk 5, 2.4 GB
V100102-01.zip  Oracle DB 12c Release 1 (12.1.0.2.0) EXAMPLES for Linux x86-64, 476.1 MB

If patch file p22066363_R12_GENERIC.zip is downloaded, below three files are not needed:

V100052-01_1of3.zip EBS R 12.2.0 Rapid Install Start Here, 725.3 MB
V100052-01_2of3.zip EBS R 12.2.0 Rapid Install Start Here, 81.6 MB
V100052-01_3of3.zip EBS R 12.2.0 Rapid Install Start Here, 97.3 MB

Go to https://www.oracle.com/database/technologies/database12c-linux-downloads.html
to download Oracle Database 12c Release 1 (12.1.0.2.0) for Linux x86-64 (and then rename them to V46095-01_xxxx.zip)
V46095-01_1of2.zip (linuxamd64_12102_database_1of2.zip)
V46095-01_2of2.zip  (linuxamd64_12102_database_2of2.zip)

Then, follow instruction to download four ZIP files for Oracle Coherence 3.7.1.0.0 and one ZIP file for Oracle Web Tier 11.1.1.9.0
V29856-01.zip Oracle WebLogic Server 10.3.6 (11gR1) Generic with Coherence 3.7.1, 1018.5 MB
V27954-01.zip Oracle Coherence 3.7.1.0.0 for C++ for Linux x86-64, 21.1 MB
V27953-01.zip Oracle Coherence 3.7.1.0.0 for the Microsoft .NET Framework, 17.0 MB
V28274-01.zip Oracle Coherence 3.7.1.0.0, 15.7 MB
V75792-01.zip Oracle Web Tier 11.1.1.9.0 Utilities (11g Patch Set 7) - Oracle Fusion Middleware for Linux x86-64, 2.0 GB

Totally, 30 ZIP files.

3. Create folder /path/to/R122/rapidSTARTCD, and download p22066363_R12_GENERIC.zip (Patch 22066363: RAPID INSTALL STARTCD 12.2.0.51, StartCD 51) to this new folder.

4. Unzip p22066363_R12_GENERIC.zip creates folder startCD.
NOTE: Do not manually unzip the other stage files.  After unzipping the startCD, buildStage.sh script should be run to unzip the rest of the installation software.  

5. Build stage area  -- it needs about 45GB to unzip files
$ cd startCD/Disk1/rapidwiz/bin
(or, /path/to/stageR122/rapidSTARTCD/startCD/Disk1/rapidwiz/bin)
$ ./buildStage.sh

Two actions are necessary.

                      Build Stage Menu
   ------------------------------------------------------
   1.     Create new stage area
   2.     Copy patches to existing stage area

   Enter your choice [4]: 1
… …
Specify the directory containing the zipped installation media: /path/to/stageR122

Verifying stage area...
Directory /path/to/stageR122/rapidSTARTCD/TechInstallMedia is valid.
Directory /path/to/stageR122/rapidSTARTCD/TechPatches/DB is valid.
Directory /path/to/stageR122/rapidSTARTCD/TechPatches/MiddleTier is valid.
Directory /path/to/stageR122/rapidSTARTCD/EBSInstallMedia/AppDB is valid.
Directory /path/to/stageR122/rapidSTARTCD/EBSInstallMedia/Apps is valid.
Directory /path/to/stageR122/rapidSTARTCD/EBSInstallMedia/AS10.1.2 is valid.
Directory /path/to/stageR122/rapidSTARTCD/TechInstallMedia/database is valid.
Directory /path/to/stageR122/rapidSTARTCD/TechInstallMedia/ohs11119 is valid.
Directory /path/to/stageR122/rapidSTARTCD/TechInstallMedia/wls1036_generic is valid.

If all worked, you shall see "Stage area verified."  by the end.

It unzip files to /path/to/stageR122/rapidSTARTCD and creates 3 sub-folders EBSInstallMedia, TechInstallMedia, TechPatches

6. Download patch 25525148 p25525148_R12_GENERIC.zip, the consolidated fixes on top of Patch 22066363, to /path/to/stageR122/rapidSTARTCD
(If database 19c is used, it needs another patch 29963728. See Oracle E-Business Suite Release Notes, Release 12.2 (Doc ID 1320300.1) ).

$ unzip p25525148_R12_GENERIC.zip
$ cd 25525148
$ ./patchRIStage.sh
Enter the location of the Rapid Install Stage area: /path/to/stageR122/rapidSTARTCD

$ cd ../
$ cd startCD/Disk1/rapidwiz/bin
$ ./buildStage.sh
                    Build Stage Menu
   ------------------------------------------------------
   1.     Create new stage area
   2.     Copy patches to existing stage area
   3.     List files in TechPatches directory
   4.     Exit menu
   Enter your choice [4]: 2

… … 

Copying latest one-off patches to stage area...

Finished copying additional patches.

7. Now, Rapid Install CDs are ready, you can go to folder /path/to/stageR122 to tar folder rapidSTARTCD up to a file and copy it to other servers for installation if needed.

8. Test rapidwiz 

Stay at /path/to/stageR122/rapidSTARTCD/startCD/Disk1/rapidwiz

$ vi /etc/oraInst.loc

$ export DISPLAY=xx.xx.xxx.xxx:0.0
$ ./rapidwiz -version

Rapid Install Wizard is validating your file system......
Rapid Install Wizard will now launch the Java Interface.....

Oracle E-Business Suite Rapid Install Wizard
Version 12.2.0.51           <= startCD 51
(c) Copyright 2000-2011 Oracle Corporation.  All rights reserved.

Now, R12.2 software is ready for Rapidwiz.  Rapid Install will validate the system configuration and tell what pre-steps on OS and database are required for the installation.

$ ./rapidwiz -help
Rapid Install Wizard is validating your file system......
CMDDIR=/path/to/stageR122/rapidSTARTCD/startCD/Disk1/rapidwiz
Rapid Install Wizard will now launch the Java Interface.....

Use Oracle guarantee restore point (GRP) to restore database

Here are the steps to use Oracle guarantee restore point with in Oracle Flashback to roll back changes in 12.1.0.2 database when backout from Apps changes becomes necessary.

Pre-steps:
1. Run full level 0 backup through OEM (or by script) on EBS_DB
2. Setup recovery directory for EBS_DB if you have not yet. Here is example to do that. You need to find location & space for flashback.
SQL> alter system set db_recovery_file_dest_size='100G' scope=both;
SQL> alter system set db_recovery_file_dest='/path/to/EBS_DB/flashback' scope=both;
3. Send email to all parties to confirm it is complete

Steps
1. Checking App Admin before run archivelog backup
2. Setup GRP:
SQL> create restore point JDK_BACKOUT_GRP guarantee flashback database;
3. Send out email for confirmation

4. After all changes worked and are confirmed, drop GRP. Please make sure to drop GRP, otherwise database might get frozen if flashback area is full.
SQL> drop restore point JDK_BACKOUT_GRP;

Backout steps
If things went bad and decision to roll all changes back is made, you need to restore database EBS_DB by following steps
SQL> shutdown immediate;
SQL> startup mount;
SQL> flashback database to restore point JDK_BACKOUT_GRP;
SQL> alter database open resetlogs;
SQL> drop restore point JDK_BACKOUT_GRP;

After that, database will be restored back to the time point when JDK_BACKOUT_GRP was created. You may do a database backup before run Backout steps, in case you need it to troubleshoot problem later.

FRM-92101 error when attempting to launch Forms

After EBS R12 was copied/moved to a new server due to Linux OS upgrade, autoconfig worked and all EBS services started without any issue. But after successful login, EBS R12 forms failed to open during startup. The error is

FRM-92101 : There was a failure in the Forms Server during startup.
This could happen due to invalid configuration. Please look into the web-server log file for details.

Click on Detail to see
Java Exception:
oracle.forms.net.ConnectionException: Forms session <2> failed during startup: no response from runtime process
at oracle.forms.net.ConnectionException.createConnectionException(Unknown Source)
at oracle.forms.net.HTTPNStream.getResponse(Unknown Source)
at oracle.forms.net.HTTPNStream.doFlush(Unknown Source)
at oracle.forms.net.HTTPNStream.flush(Unknown Source)
at java.io.DataOutputStream.flush(Unknown Source)
at oracle.forms.net.HTTPConnection.connect(Unknown Source)
at oracle.forms.engine.FormsDispatcher.initConnection(Unknown Source)
at oracle.forms.engine.FormsDispatcher.init(Unknown Source)
at oracle.forms.engine.Runform.initConnection(Unknown Source)
at oracle.forms.engine.Runform.startRunform(Unknown Source)
at oracle.forms.engine.Main.createRunform(Unknown Source)
at oracle.forms.engine.Main.start(Unknown Source)
… ...

Look into log file application.log in folder $LOG_HOME/ora/10.1.3/j2ee/forms/forms_default_group_1, see luckily a message:

runtime process failed during startup with errors $ORACLE_HOME/bin/frmweb: error while loading shared libraries: libXm.so.2: cannot open shared object file: No such file or directory

The problem was caused by missing Linux package openmotif21 when the new server was configured. EBS forms function works after install the right openmotif rpm and bounce EBS services.

$ rpm -qa | grep openmotif
openmotif21-2.1.30-11.EL6.i686

UPDATES in 2024:

The same problem can happen to R12.2 if openmofit does not exist on the server or it was removed by a Linux patching (e.g. lesstif-0.95.2-8.el7.x86_64 may replace it).

The error message in R12.2 is at $EBS_DOMAIN_HOME/servers/forms_server#/logs

(# could be 1, 2, etc)

The fix is to make sure BOTH openmotif and the link exist on the server

$ rpm -qa | grep openmotif

openmotif21-2.1.30-11.EL6.i686

$ ls -l /usr/X11R6/lib/libXm.so.2

lrwxrwxrwx 1 root root 12 May 19  2020 /usr/X11R6/lib/libXm.so.2 -> libXm.so.2.1

UPDATES in 2025

If R12.2 gets FRM-92101 error on Linux RHEL8, the solution may be different. RHEL8 has different rpm packages and may not have /usr/X11R6/lib directory. See Doc ID 3052209.1 (FRM-92101: There Was A Failure In The Forms Server During Startup, After Linux 7 to Linux 8 Upgrade).

Move EBS from Linux 6 to Linux 7

1. Build new server with a temporary name and install RHEL7 on it (by Linux Admin)
It is very important to rename it later to the same server name for EBS apps to work properly.

Linux Admin has to follow Oracle Doc ID 761566.1 (Oracle E-Business Suite Installation and Upgrade Notes Release 12 (12.1.1) for Linux x86-64) to install all required RHEL7 (Red Hat Enterprise Linux 7) components.

$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.7 (Maipo)

$ cd /usr/bin
$ ls -altr java*
-rwxr-xr-x 1 root root    5530 Jul 22  2019 javaws.itweb
lrwxrwxrwx 1 root root   24 Feb 13 09:24 javaws -> /etc/alternatives/javaws
lrwxrwxrwx 1 root root   27 Feb 18 19:16 javac -> /usr/java/default/bin/javac
lrwxrwxrwx 1 root root   29 Feb 18 19:16 javadoc -> /usr/java/default/bin/javadoc
lrwxrwxrwx 1 root root   22 Feb 18 19:45 java -> /etc/alternatives/java

$ java -version
openjdk version "1.8.0_232"
OpenJDK Runtime Environment (build 1.8.0_232-b09)
OpenJDK 64-Bit Server VM (build 25.232-b09, mixed mode)
$ javac -version
javac 1.7.0_131

3. Verify the new RHEL7 server

1) Make sure /etc/oraInst.loc is copied over and has right permission.

2) Check all files under /etc/init.d are copied over for auto stop/start EBS services by server's reboot. For example, below two shell scripts stop/start EBS services of two instances.

$ cd /etc/init.d
$ ls -l ebs*
-rwxr-xr-x 1 root root 545 Jun 21  2016 ebs1
-rwxr-xr-x 1 root root 549 Jun 21  2016 ebs2

3) It is good to have /usr/bin/xclock, ftp, sftp, gzip installed.

4) A soft link for libdb-4.3.so under /usr/lib on RHEL7 server is necessary. Otherwise, Apache service will not start on RHEL7 sever by error

httpd: error while loading shared libraries: libdb-4.3.so: cannot open shared object file: No such file or directory

$ cd /usr/lib

$ ls -al libdb*.*
lrwxrwxrwx 1 root root        12    Feb  4 11:30 libdb-4.3.so -> libdb-5.3.so
-rwxr-xr-x 1 root root 1868348    Jan 30  2019 libdb-5.3.so
lrwxrwxrwx 1 root root        12   Dec 10 15:21 libdb-5.so -> libdb-5.3.so
-rw-r--r-- 1 root root         5825   Dec 11 10:52 libdb.so.2

Note: This is different from RHEL6 server, and Apache worked fine.

$ cat /etc/redhat-release

Red Hat Enterprise Linux Server release 6.10 (Santiago)

$ ls -al /usr/lib/libdb*.*

-rw-r--r-- 1 root root 5825 Nov 11  2016 /usr/lib/libdb.so.2

4. Upgrade EBS JDK from JDK 6 to JDK 7 on RHEL6 host. This step is an important pre-step for RHEL7 upgrade, and is not necessary if JDK 7 was used within EBS already. See blow on how to upgrade JDK:

https://erpondb.blogspot.com/2019/12/upgrade-jdk-to-jdk-7-in-ebs-r121.html

$ $ADJVAPRG -version
java version "1.7.0_231"
Java(TM) SE Runtime Environment (build 1.7.0_231-b08)
Java HotSpot(TM) Client VM (build 24.231-b08, mixed mode)

$ echo $IAS_ORACLE_HOME
/path/to/apps/tech_st/10.1.3
$ $IAS_ORACLE_HOME/appsutil/jdk/jre/bin/java -fullversion
java full version "1.7.0_231-b08"

$ echo $ORACLE_HOME
/path/to/apps/tech_st/10.1.2
$ $ORACLE_HOME/jdk/bin/java -fullversion
java full version "1.7.0_231-b08

5. Make final switch

1) Stop EBS apps services on RHEL6 servers
2) Copy all file partitions from RHEL6 servers to RHEL7 servers (using rsync by Linux Admin)
3) Rename new RHEL7 server back to the old server name
(Huge extra work and cloning process are required if EBS server name does not keep the same)
4) Change network DNS with the new IP address for RHEL7 server name
5) Replace old IPs in F5 configuration with the new IP addresses of new EBS hosts
6) Run autoconfig on all EBS nodes on RHEL7 servers
7) Start EBS apps services on RHEL7 servers
8) Make sure file system backup is activated on new servers

Doc ID 761566.1 mentioned two patches. But I did not apply them. I think they may be required if clone process is used to move EBS to new servers.
- 19671435:R12.WIP.B
- 19863797:R12.ENG.C

Re-new Java Signing Key

The server certificate for JAR files expires after some years, for example, in each two years. That is the reason we have to renew it periodically. After run renewal, I got better understanding on the process than in my post Keystore & Java Signing in 2016.

1. Find the current keystore password and key password embedded in EBS R12.1 instance

SQL> set serveroutput on
SQL> declare
spass varchar2(30);
kpass varchar2(30);
begin
ad_jar.get_jripasswords(spass, kpass);
dbms_output.put_line(spass);
dbms_output.put_line(kpass);
end; 
/
puneet
myxuan

If you want to them, use "$ adjkey -storepasswd" and "$ adjkey -keypasswd" to do that.

2. $APPL_TOP/admin/adkeystore.dat is a binary file, which includes, in my file, one key entry and 3 certificates. Use keytool to view its contents:

$ echo $TWO_TASK
EBSDEV
$ hostname
devserver1d

$ keytool -list -keystore adkeystore.dat
Enter keystore password:
Keystore type: JKS
Keystore provider: SUN
Your keystore contains 1 entry
ebsdev_devserver1d, Apr 15, 2019, PrivateKeyEntry,
Certificate fingerprint (SHA1): A7:01:6E:76:D7:2D:F5: … ...

$ keytool -list -v -keystore adkeystore.dat    <= View details
Enter keystore password:
Keystore type: JKS
Keystore provider: SUN
Your keystore contains 1 entry
Alias name: ebsdev_devserver1d
Creation date: Apr 15, 2019
Entry type: PrivateKeyEntry
Certificate chain length: 3
Certificate[1]:
Owner: CN=Company name, O=Company name, L=City, ST=State, C=US
Issuer: CN=DigiCert SHA2 Assured ID Code Signing CA, OU=www.digicert.com, O=DigiCert Inc, C=US
Serial number: f7d45adf32f16e ….
Valid from: Thu Jan 03 19:00:00 EST 2019 until: Wed Jan 12 07:00:00 EST 2022
Certificate fingerprints:
         MD5:  17:68:2F:67:CC:F0: … …
         SHA1: A7:01:6E:76:D7:2D: … …
         Signature algorithm name: SHA256withRSA
         Version: 3

Extensions:

#1:

…  … …
Certificate[2]:
Owner: CN=DigiCert SHA2 Assured ID Code Signing CA, OU=www.digicert.com, O=DigiCert Inc, C=US
Issuer: CN=DigiCert Assured ID Root CA, OU=www.digicert.com, O=DigiCert Inc, C=US
Serial number: 409181b5fd5bb667 ......
Valid from: Tue Oct 22 08:00:00 EDT 2013 until: Sun Oct 22 08:00:00 EDT 2028
Certificate fingerprints:
         MD5:  B6:56:37:6C:3D:2A: … …
         SHA1: 92:C1:58:8E:85:AF: … …
         Signature algorithm name: SHA256withRSA
         Version: 3
Extensions:
#1:
…  … …
Certificate[3]:
Owner: CN=DigiCert Assured ID Root CA, OU=www.digicert.com, O=DigiCert Inc, C=US
Issuer: CN=DigiCert Assured ID Root CA, OU=www.digicert.com, O=DigiCert Inc, C=US
Serial number: ce7e0e517d8 … …
Valid from: Thu Nov 09 19:00:00 EST 2006 until: Sun Nov 09 19:00:00 EST 2031
Certificate fingerprints:
         MD5:  87:CE:0B:7B:2A: … …
         SHA1: 05:63:B8:63:0D: … …
         Signature algorithm name: SHA1withRSA
         Version: 3
Extensions:
#1: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
  DigitalSignature
  Key_CertSign
  Crl_Sign
]
#2: ObjectId: 2.5.29.19 Criticality=true
BasicConstraints:[
  CA:true
  PathLen:2147483647
]
#3: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 45 EB  9C A7 A7 21 9D  E.......1-Q...!.
0010: F3 6D C8 0F                                        .m..
]
]
#4: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: 45 EB 8B A7 A7 21 9D  E.......1-Q...!.
0010: F3 6D C8 0F                                        .m..
]
]

3. Before cert's expiration date (01/12/2022 in above keystore), you have to renew it. First, you make request to Security team for a new keystore file that will work for one EBS instance. Then, run adadmin to re-generate JAR files to that instance (as in Step 5). 
I do not know what tool Security team uses to create file adkeystore.dat. If the password in new adkeystore.dat do not match EBS instance's password in Step 1, adadmin in Step 5 will fail with error - jarsigner: unable to recover key from keystore. Commands to change keystore password and key password:

$ keytool -storepasswd -keystore /path/to/temp/adkeystore.dat
Enter keystore password:                <= storePwd
New keystore password:                 <= puneet
Re-enter new keystore password:   <= puneet

$ keytool -keypasswd -keystore /path/to/temp/adkeystore.dat -alias ebsdev_devserver1d
Enter keystore password:                                                   <= puneet
Enter key password for <ebsdev_devserver1d>                <= keyPWD
New key password for <ebsdev_devserver1d>:                <= myxuan
Re-enter new key password for <ebsdev_devserver1d>:  <= myxuan

4. Make adkeystore.dat to work in other EBS instances. 

The original adkeystore.dat is only for instance EBSDEV. If you have multiple instances, below command worked for me to make it work for QA instance EBSQA (on different servers):

$ cp adkeystore.dat $APPL_TOP/admin
$ cd $APPL_TOP/admin

$ keytool -changealias -alias ebsdev_devserver1d -destalias EBSQA_qaserver1q -keystore adkeystore.dat
Enter keystore password:
Enter key password for <ebsdev_devserver1d>

$ keytool -list -keystore adkeystore.dat
Enter keystore password:
Keystore type: JKS
Keystore provider: SUN
Your keystore contains 1 entry
ebsqa_qaserver1q, Apr 20, 2019, PrivateKeyEntry,
Certificate fingerprint (SHA1): A7:01:6E:76:D7:2D:F5: … ...

5. Generate JAR using adadmin utility which will take the new certificates in adkeystore.dat

$ echo $TWO_TASK

EBSQA
$ adstpall.sh
$ adadmin
    ==> 1 Select Generate Applications Files
    ==> 4 Generate Product JAR Files    Yes.  (Do force the regeneration of all JAR files.)

(adadmin can be used to re-load JAR files, which is not required, by 
    ==> 3 Compile/Reload Applications Database Entities menu 
    ==> 3 Reload JAR Files to Database. Seems it uploads some .jar files, but not all .jar files)

NOTES: After JDK was upgraded to JDK 7, adadmin gives warning on more files when generating JAR files. Depending on what EBS components are used, usually it's safe to ignore them.
adearea();
Warning=> unable to locate $FND_TOP/java/3rdparty/stdalone/odi-sdk-invocation10g.zip file.
Warning=> unable to locate $FND_TOP/java/3rdparty/stdalone/ucmscript.zip file.
Warning=> unable to locate $FND_TOP/java/3rdparty/stdalone/oracle-ucm-ridc-12_1_2_0_0.zip file.
Warning=> unable to locate $FND_TOP/java/3rdparty/stdalone/gcm-server.zip file.
Warning=> unable to locate $FND_TOP/java/3rdparty/stdalone/json_simple-1-1.zip file.

6. Verify file's new timestamp and new expiration date

$ ls -al $AD_TOP/java/jar/adxlib.jar      <= pick up a JAR file
$ jarsigner -verify -verbose -certs $AD_TOP/java/jar/adxlib.jar
 ... ... ...
sm      7732 Wed Jan 02 05:02:02 EST 2002 oracle/apps/ad/util/zip/ZipOutputStream.class

      X.509, CN=Company name, O=Company name, L=City, ST=State, C=US
      [certificate is valid from 1/3/19 7:00 PM to 1/12/22 7:00 AM]
      X.509, CN=DigiCert SHA2 Assured ID Code Signing CA, OU=www.digicert.com, O=DigiCert Inc, C=US
      [certificate is valid from 10/22/13 8:00 AM to 10/22/28 8:00 AM]
      X.509, CN=DigiCert Assured ID Root CA, OU=www.digicert.com, O=DigiCert Inc, C=US
      [certificate is valid from 11/9/06 7:00 PM to 11/9/31 7:00 PM]
... ... ...
  
- Signed by "CN=Company name, O=Company name, L=City, ST=State, C=US"
    Digest algorithm: SHA1
    Signature algorithm: SHA1withRSA, 2048-bit key

jar verified.

Warning:

This jar contains signatures that does not include a timestamp. Without a timestamp, users may not
be able to validate this jar after the signer certificate's expiration date (2019-06-05) or after
any future revocation date.

(Not sure why getting this warning)

NOTE: keytool is part of JDK

$ which keytool

$IAS_ORACLE_HOME/appsutil/jdk/jre/bin/keytool