EBS hits ORA-28040: No matching authentication protocol

After database was upgraded to 12.1.0.2, clone script to refresh a R12.1.3 instance got a error:

$ perl adcfgclone.pl appsTier
... ...
Target System Port Pool [0-99] : 27
Checking the port pool 27
done: Port Pool 27 is free
Report file located at $INST_TOP/admin/out/portpool.lst
Complete port information available at $INST_TOP/admin/out/portpool.lst
RC-40201: Unable to connect to Database EBSQA.

The detail error messages in log file are

---------------------------------------------------------------
                   ADX Database Utility
---------------------------------------------------------------
getConnectionUsingAppsJDBCConnector() -->
    APPS_JDBC_URL=''
    Trying to get connection using SID based connect descriptor
getConnection() -->
    sDbHost    : ebsdb1q
    sDbDomain  : domain.com
    sDbPort    : 1527
    sDbSid     : EBSQA
    sDbUser    : APPS
    Trying to connect using SID...
getConnectionUsingSID() -->
    JDBC URL: jdbc:oracle:thin:@ebsdb1q.domain.com:1527:EBSQA
    Exception occurred: java.sql.SQLException: ORA-28040: No matching authentication protocol
    Trying to connect using SID as ServiceName
getConnectionUsingServiceName() -->
    JDBC URL: jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ebsdb1q.domain.com)(PORT=1527))(CONNECT_DATA=(SERVICE_NAME=EBSQA)))
    Exception occurred: java.sql.SQLException: ORA-28040: No matching authentication protocol
    Trying to connect using SID as ServiceName.DomainName
getConnectionUsingServiceName() -->
    JDBC URL: jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ebsdb1q.domain.com)(PORT=1527))(CONNECT_DATA=(SERVICE_NAME=EBSQA.domain.com)))
    Exception occurred: java.sql.SQLException: Listener refused the connection with the following error:
ORA-12514, TNS:listener does not currently know of service requested in connect descriptor
The Connection descriptor used by the client was:
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ebsdb1q.domain.com)(PORT=1527))(CONNECT_DATA=(SERVICE_NAME=EBSQA.domain.com)))
    Connection could not be obtained; returning null
-------------------ADX Database Utility Finished---------------
Exception occurred while preseeding variables in the context file: java.sql.SQLException: Could not get connection to the database

StackTrace:
java.sql.SQLException: Could not get connection to the database

Because of the error, it failed to generate TNS files:

##########################################################################
                   Generate Tns Names
##########################################################################
Logfile:  $INST_TOP/admin/log/NetServiceHandler.log
Classpath: $COMMON_TOP/java/lib/appsborg2.zip:$COMMON_TOP/java/classes
Updating s_tnsmode to 'generateTNS'
UpdateContext exited with status: 0
AC-50480: Internal error occurred: java.lang.Exception: Error while generating listener.ora.
Error generating tnsnames.ora from the database, temporary tnsnames.ora will be generated using templates
Instantiating Tools tnsnames.ora
Tools tnsnames.ora instantiated
Web tnsnames.ora instantiated
adgentns.pl exiting with status 2
ERRORCODE = 2 ERRORCODE_END
.end std out.
.end err out.
Result             : FAILED

It seems next steps in the clone script use Sqlplus to make database connection and make this error ignorable. So, even with the error, the cloned instance worked fine.

To avoid this JDBC error, before running the clone script, add two lines to file $TNS_ADMIN/sqlnet.ora on database server and then bounce database listener (Doc. 2125856.1):
SQLNET.ALLOWED_LOGON_VERSION_CLIENT=8
SQLNET.ALLOWED_LOGON_VERSION_SERVER=8

The setting 8 (or 10, 11) has nothing to do with the Oracle database version, it is authentication level used by the JDBC. But, it is a temporary solution, because it leads to use old and unsupported Oracle drivers. The best solution though is to upgrade the Oracle driver on client side.

One solution after the clone worked is to customize file $ORA_CONFIG_HOME/10.1.3/j2ee/forms/config/server.xml with using two .jar files in $COMMON_TOP/java/lib.

I did not test the solution by Doc. 2228803.1 which will modify Perl script to use two newer .jar files.

If clone script does not get error, the log should like this:

---------------------------------------------------------------
                   ADX Database Utility
---------------------------------------------------------------
getConnection() -->
    sDbHost    : ebsdb1q
    sDbDomain  : domain.com
    sDbPort    : 1547
    sDbSid     : EBSQA
    sDbUser    : APPS
    Trying to connect using SID...
getConnectionUsingSID() -->
    JDBC URL: jdbc:oracle:thin:@ebsdb1q.domain.com:1547:EBSQA
    Connection obtained
-------------------ADX Database Utility Finished---------------

Every EBS Forms got a popup FRM-40735 error

After I apply January 2018 CPU patch to R12.1.3, every single forms gives a popup window with error:

ORA-01403: no data found
FRM-40735 WHEN-NEW-ITEM-INSTANCE trigger raised unhandled exception ORA-06502

I compiled below items using adadmin. But the problem is still there.

Forms PL/SQL library (.pll)
Forms menu
Generate message files
Generate product JAR files
Forms executable

I also tried "Help > Diagnostics > Trace > Trace with Binds and Waits" to enable forms trace, but did not find any ORA- error in the trace log.

The problem was that one file CUSTOM.pll in $AU_TOP/resource was not owned by appl_mgr, but was owned by a developer who copied it to the server. Because of that, adpatch did not compile it during patching and later adadmin refused to compile it also when I ran adadmin to compile all library files.

After I manually compile it, the popup window does not show up on any forms! 

$ export FORMS_PATH=$FORMS_PATH:$AU_TOP/forms/US:$AU_TOP/resource
$ cd $AU_TOP/resource
$ frmcmp_batch.sh module=CUSTOM.pll userid=apps/appsPWD module_type=library compile_all=special

CUSTOM.pll timestamp will change and its file size may increase after compilation. Use command to check and compare file contents:

$ adident Header $AU_TOP/resource/CUSTOM.pll