Saturday, April 23, 2016

Manually deploy EAR file

After a patch (such as JE patch 21950151) is applied or adadmin generates new JAR (Java Archive) file, the apps startup script adformsctl.sh may deploy forms EAR file. You may see a Warning (with status 0) in log $LOG_HOME/appl/admin/log/adstrtal.log.

Starting OPMN managed FORMS OC4J instance  ...
Calling txkChkFormsDeployment.pl to check whether latest FORMSAPP.EAR is deployed...
Program : $FND_TOP/patch/115/bin/txkChkFormsDeployment.pl started @ Mon Apr 18 1
6:58:03 2016
*** Log File = $LOG_HOME/appl/rgf/TXK/txkChkFormsDeployment_Mon_Apr_18_16_58_02_2016/txkChkFormsDeployment_Mon_Apr_18_16_58_02_2016.log

File "$IAS_ORACLE_HOME/j2ee/forms/applications/forms/formsweb/WEB-INF/lib/frmsrv.jar" exists. Proceeding to check the size...
=================================================
*** Latest formsapp.ear has NOT been deployed ***
Deploying the latest EAR file...
=================================================
Program : $FND_TOP/patch/115/bin/txkChkFormsDeployment.pl completed @ Mon Apr 18 16:59:04 2016
*****************************************
*             W A R N I N G             *
*****************************************
Error while executing the perl script txkChkFormsDeployment.pl

We have determined that you need to redeploy Forms (using txkChkFormsDeployment.pl).
But could not do it automatically for you, due to some issues.
For details refer the log files.
Follow Note: 397174.1 to redeploy Forms manually.
*****************************************
adformsctl.sh: exiting with status 0


I followed steps in  Doc ID 397174.1 to get over this. Without this steps, some forms may not open up. 
1)  back up file $INST_TOP/ora/10.1.3/j2ee/forms/config/system-jazn-data.xml (for deploying the formsapp.ear file). Note this file's timestamp always gets updated.
2)  when the oc4jadmin Password is unknown, search keyword oc4jadmin for pattern in file system-jazn-data.xml , and replace line
  <description>OC4J Administrator</description>
  <credentials>{903}9VrhYTuhd7DyBJf7J/4KwbSEIlETQJOO</credentials>
with
  <description>OC4J Administrator</description>
  <credentials>!welcome123</credentials>

Optional: after above change and bounce apps services, below line can verify the password is correct :
$ $AF_JRE_TOP/bin/java -jar $IAS_ORACLE_HOME/j2ee/home/admin_client.jar deployer:oc4j:opmn://hostname.domain.com:6530/forms oc4jadmin welcome123 -validateURI
URI deployer:oc4j:opmn://hostname.domain.com:6530/forms is valid and connected
-- where to get the port number
$ grep 6530 $ORA_CONFIG_HOME/10.1.3/opmn/conf/opmn.xml
<port local="6130" remote="6230" request="6530"/>
-- one more validation
$ cd $ORA_CONFIG_HOME/10.1.3/opmn/bin
$ ./opmnctl validate
opmnctl: opmn validation succeeded.

3)  stop apps services, and run the Perl script on each Forms tier to deploy the EAR file
$ $FND_TOP/bin/txkrun.pl -script=CfgOC4JApp
*** ALL THE FOLLOWING FILES ARE REQUIRED FOR RESOLVING RUNTIME ERRORS
*** Log File = $LOG_HOME/appl/rgf/TXK/txkCfgOC4JApp_Mon_Apr_18_17_17_52_2016.log
Program : $FND_TOP/patch/115/bin/txkCfgOC4JApp.pl started @ Mon Apr 18 17:17:52 2016

*** Log File = $LOG_HOME/appl/rgf/TXK/txkCfgOC4JApp_Mon_Apr_18_17_17_52_2016.log

Enter Application name for re-deployment ? forms
Enter Oc4j Instance  password for re-deployment ? welcome123
Run Autoconfig <Yes/No> ? No

*****************************************************
Required values for starting OC4J instance "forms":
====================================================
s_formsstatus = enabled
s_forms_nprocs = 1 (value should be greater than 0)
Existing values from the context file:
======================================
s_formsstatus = enabled
s_forms_nprocs = 2
----------------------------------------------
*** Values for context variables are VALID ***
----------------------------------------------
*****************************************************
Stopping all OPMN processes.
OPMN stopped.
OPMN started.
Deplolying Application : "forms" onto OC4J instance: "forms"
Application deployed successfully.
Stopping and starting OC4J instances.
Started OC4J instances.
Binding webApp : "forms" with webmodule : "formsweb" for OC4J instance: "forms"
Web application bound successfully.
Stopping OPMN.
OPMN stopped.

Program :
$FND_TOP/patch/115/bin/txkCfgOC4JApp.pl completed @ Mon Apr 18 17:42:21 2016
End of
$FND_TOP/patch/115/bin/txkCfgOC4JApp.pl : No Errors encountered 

4)  restore the original system-jazn-data.xml file.
5)  run AutoConfig on each Forms server. (Without this step, the forms may not pop up after click due to failure in back-end authentication.)
$ADMIN_SCRIPTS_HOME/adautocfg.sh

Now, you shall see new timestamp on $IAS_ORACLE_HOME/j2ee/forms/applications/forms.ear file. And, adformsctl.sh will start forms successfully.

___________________________________________________________________
~~~~~~~ Additional notes (Real paths are replaced with env variables) ~~~~~~~
Manually deploying EAR file may not always work. Here is what I experienced when execution "$FND_TOP/bin/txkrun.pl -script=CfgOC4JApp" gets failure in instances where adstpall.sh (to stop apps) get Time out error from "adoafmctl.sh stop" and "adformsctl.sh stop".

1. I first saw error in startup log adstrtal.log in a situation when adstrtal.sh called the Perl script to deploy EAR file. 
Executing service control script:
$ADMIN_SCRIPTS_HOME/adformsctl.sh start
Timeout specified in context file: 100 second(s)

script returned:
****************************************************
ERROR : Timed out( 100000 ): Interrupted Exception

You are running adformsctl.sh  version 120.16.12010000.3

Starting OPMN managed FORMS OC4J instance  ...
Calling txkChkFormsDeployment.pl to check whether latest FORMSAPP.EAR is deployed...
Program : $FND_TOP/patch/115/bin/txkChkFormsDeployment.pl started @ Wed Jul 
14 15:47:39 2016

*** Log File = $LOG_HOME/appl/rgf/TXK/

txkChkFormsDeployment_Thu_Jul_14_15_47_39_2016/txkChkFormsDeployment_Thu_Jul_14_15_47_39_2016.log

File "$IAS_ORACLE_HOME/j2ee/forms/applications/forms/formsweb/WEB-INF/lib/frmsrv.jar" exists. Proceeding to check the size...

=================================================
*** Latest formsapp.ear has NOT been deployed ***
Deploying the latest EAR file...
=================================================

*** ALL THE FOLLOWING FILES ARE REQUIRED FOR RESOLVING RUNTIME ERRORS
*** Log File = $LOG_HOME/appl/rgf/TXK/

txkChkFormsDeployment_Thu_Jul_14_15_47_39_2016/txkChkFormsDeployment_Thu_Jul_14_15_47_39_2016.log
****************************************************


2. Because of the error, adopmnctl.sh may show OC4J: forms is DOWN. When I looked into further, I saw errors in the EAR deployment log:

 $ more $LOG_HOME/appl/rgf/TXK/txkChkFormsDeployment_Thu_Jul_14_15_47_39_2016/txkChkFormsDeployment_Thu_Jul_14_15_47_39_2016.log

- - -  - - -  - - -
Application deployed successfully.        <== this line (and above) show deployment went good 
===========================
Inside manageFormsOc4j()...
===========================

*** OPERATION is stopproc ***
Execute SYSTEM command : $IAS_ORACLE_HOME/opmn/bin/opmnctl stopproc instancename=forms
opmnctl: stopping opmn managed processes...
opmn id=hostname.domin.com:6230
0 of 1 processes stopped. 

-------------------------------------------------------------------------------- 
ias-component/process-type/process-set:
    default_group/forms/default_group/

Error
--> Process (index=1,uid=2087060759,pid=14495)
    time out while waiting for a managed process to stop
    Log:
   $LOG_HOME/ora/10.1.3/opmn/default_group~forms~default_group~1.log

===========================
Inside manageFormsOc4j()...
===========================
*** OPERATION is startproc ***
Execute SYSTEM command : $IAS_ORACLE_HOME/opmn/bin/opmnctl startproc instancename=forms
ERROR DESCRIPTION:
(*******FATAL ERROR*******
PROGRAM : $FND_TOP/patch/115/bin/txkChkFormsDeployment.pl($FND_TOP/bin/txkrun.pl)
TIME    : Thu Jul 14 15:50:07 2016
FUNCTION: TXK::Process::run [ Level 3 ]
MESSAGES:
Command error: <rc> = 38400, <command> = $IAS_ORACLE_HOME/opmn/bin/opmnctl startproc instancename=forms
Command error: <rc> = 52224, <command> = $IAS_ORACLE_HOME/opmn/bin/opmnctl stopproc instancename=forms
Command error: <rc> = 38400, <command> = $IAS_ORACLE_HOME/opmn/bin/opmnctl stopproc instancename=forms

STACK TRACE
        TXK::Error::abort('TXK::Error','HASH(0x8942208)') called at $AU_TOP/perl/TXK/Common.pm line 299
        TXK::Common::doError('TXK::Process=HASH(0x89eec34)','Command error: <rc> = 38400, <command> = /path/to/EBSQA/...','undef')
 called at $AU_TOP/perl/TXK/Common.pm line 314
... ... ... ...

 3. Seems the errors are at the last stage of EAR deployment. When I shutdown apps service and started it again by adstrtal.sh, it did not try to deploy the EAR file again and just showed all in status 0. But, when I checked out the login and forms, I could not launch forms.

4. When I tried to deploy the EAR file manually by following the steps above, I got similar errors. Seems the error is from the last stage.

. . .  . . .  . . .
Stopping all OPMN processes.
OPMN stopped.
OPMN started.
Deplolying Application : "forms" onto OC4J instance: "forms"
Application deployed successfully.


Stopping and starting OC4J instances.
Errors encountered running $FND_TOP/patch/115/bin/txkCfgOC4JApp.pl
*******FATAL ERROR*******
PROGRAM :
$FND_TOP/patch/115/bin/txkCfgOC4JApp.pl($FND_TOP/bin/txkrun.pl)
TIME    : Wed Jul  6 17:11:06 2016
FUNCTION: TXK::Process::run [ Level 3 ]
MESSAGES:
Command error: <rc> = 52224, <command> =
$IAS_ORACLE_HOME/opmn/bin/opmnctl stopproc  instancename=forms

STACK TRACE
        TXK::Error::abort('TXK::Error','HASH(0x91118f4)') called at $AU_TOP/perl/TXK/Common.pm line 299
        TXK::Common::doError('TXK::Process=HASH(0x9a86c10)','Command error: <rc> = 52224, <command> = 
/path/to/EBSQA/a...','undef') called at $AU_TOP/perl/TXK/Common.pm line 314
. . .  . . .  . . .

5. I did try things around and made forms work before I ran Step 6 below.

6. The deployment errors match the issue in Doc ID 1399491.1, which recommends patch 12965674 as the fix. After I applied it to 10.1.3.5 ORACLE_HOME (for Apache), it not only fixed the errors in EAR deployment but also fixed below Time out error in adstpall.log during apps services shutdown !

You are running adstpall.sh version 120.10.12010000.4

Executing service control script:
$ADMIN_SCRIPTS_HOME/adoafmctl.sh stop
Timeout specified in context file: 100 second(s)
script returned:
****************************************************
ERROR : Timed out( 100000 ): Interrupted Exception

You are running adoafmctl.sh version 120.8
Stopping OPMN managed OAFM OC4J instance ...

****************************************************
Executing service control script:
$ADMIN_SCRIPTS_HOME/adformsctl.sh stop
Timeout specified in context file: 100 second(s)
script returned:
****************************************************
ERROR : Timed out( 100000 ): Interrupted Exception

You are running adformsctl.sh  version 120.16.12010000.3
Stopping OPMN managed FORMS OC4J instance  ...


NOTES:  Oracle gives a document with a easier way to manually deploy R12 Ear file $ORACLE_HOME/forms/j2ee/formsapp.ear. See Doc ID 1961805.1

Monday, April 4, 2016

Custom EBS password complexity

When I tried to create a new user in a non-prod R12.1 instance, I got below error:

Unable to load java class oracle.apps.custom.security.XXXXXPasswordValidation specified profile option SIGNON_PASSWORD_CUSTOM. Please verify that the class exists and that it implements the java interface oracle.apps.custom.security.XXXXXPasswordValidation

I checked profile option Signon Password Custom which had "oracle.apps.custom.security.XXXXXPasswordValidation" on site level. I ran below query and did not see the custom Java class in the database. So that error makes sense because it was not uploaded to the database.

SQL> SELECT dbms_java.longname(object_name), status
 FROM user_objects
 WHERE object_type = 'JAVA CLASS'
 AND dbms_java.longname(object_name) like '%PasswordValidation';

 DBMS_JAVA.LONGNAME(OBJECT_NAME)        STATUS
 oracle/apps/fnd/security/PasswordValidation           VALID
 oracle/apps/fnd/security/AppsPasswordValidation   VALID 


Fortunately I found the Java files are still saved in the file system. So I uploaded it to the database:

$ cd $JAVA_TOP/oracle/apps/custom/security
$ ls -al
-rwxrwxrwx 1 trainer users 1927 Nov  8  2013 XXXXXPasswordValidation.class
-rwxrwxrwx 1 trainer users 2629 Feb  6  2012 XXXXXPasswordValidation.java

$ loadjava -user apps/appsPWD -verbose -resolve -force XXXXXPasswordValidation.class
arguments: '-user' 'apps/appsPWD' '-verbose' '-resolve' '-force' 'XXXXXPasswordValidation.class'
creating : class oracle/apps/custom/security/XXXXXPasswordValidation
loading  : class oracle/apps/custom/security/XXXXXPasswordValidation
resolving: class oracle/apps/custom/security/XXXXXPasswordValidation


SQL> SELECT dbms_java.longname(object_name), status
 FROM user_objects
 WHERE object_type = 'JAVA CLASS'
 AND dbms_java.longname(object_name) like '%PasswordValidation';

DBMS_JAVA.LONGNAME(OBJECT_NAME)                STATUS
oracle/apps/custom/security/XXXXXPasswordValidation  VALID
oracle/apps/fnd/security/PasswordValidation                    VALID
oracle/apps/fnd/security/AppsPasswordValidation            VALID


After those steps, I was able to create EBS user if the password meets the complexity defined in the custom Java file.

Since oracle/apps/fnd/security/PasswordValidation exists in the database, I assumed it is an Oracle standard class and so I changed profile option Signon Password Custom from  "oracle.apps.custom.security.XXXXXPasswordValidation" to "oracle.apps.fnd.security.PasswordValidation".  But when I tried to create a user now , I got error:

Unable to load Java class oracle.apps.fnd.security.PasswordValidation specified in profile option SIGNON_PASSWORD_CUSTOM. Please verify that the class exists and that it implements the Java interface oracle.apps.fnd.security.PasswordValidation.

As an exercise, I re-uploaded the Java class to the database:

$ cd $JAVA_TOP/oracle/apps/fnd/security
$ ls -al PasswordValidation.class
-rwxrwxrwx 1 trainer users 673 Jan  3  2007 PasswordValidation.class

$ loadjava -user apps/appsPWD -verbose -resolve -force PasswordValidation.class
arguments: '-user' 'apps/appsPWD' '-verbose' '-resolve' '-force' 'PasswordValidation.class'
creating : class oracle/apps/fnd/security/PasswordValidation
loading  : class oracle/apps/fnd/security/PasswordValidation
resolving: class oracle/apps/fnd/security/PasswordValidation

Interestingly, that made it and other Java classes INVALID in the database.I had to run adadmin to compile APPS schema to make them VALID again. Seems that oracle.apps.fnd.security.PasswordValidation can not be used or it is a wrong value to put in profile option Signon Password Custom.

The custom Java code looks like: 
$ view XXXXXPasswordValidation.java

package oracle.apps.custom.security;

import oracle.apps.fnd.common.VersionInfo;
import oracle.apps.fnd.security.*;
import java.*;

// Referenced classes of package oracle.apps.fnd.security:
// PasswordValidation

public class XXXXXPasswordValidation implements PasswordValidation
{

due to copy right, no detail here...

}

Updates:
Profile option "Signon Password Failure Limit" defines maximum number of tries allowed before user account becomes locked.
Profile option "Signon Password No Reuse" defines minimum number of days that a user must wait before being allowed to reuse a password.