Before Linux 5 support becomes expired, we have to upgrade it. But, RHEL5 does not have an option to "upgrade" it on the same server. We had to build a fresh new Linux 6 server and then migrate Oracle EBS R12.1.3 to the new RHEL6 server.
1. Server requirements by Oracle
Linux Admin followed Oracle E-Business Suite Installation and Upgrade Notes Release 12 (12.1.1) for Linux x86-64 (Doc ID 761566.1) to install all required components.
- list of RPMs
To make openmotif21 installation work, follow Doc ID 2057494.1 (Oracle Linux: Dependency Resolution Error "Package oracle-ebs-server-R12-preinstall requires openmotif21 and that has been obsoleted by lesstif") to delete "lesstif" package from Linux server first. Some of the installed RPMs:
$ rpm -q openmotif
package openmotif is not installed
$ rpm -q openmotif21
openmotif21-2.1.30-11.EL6.i686
$ rpm -q xorg-x11-libs-compat
xorg-x11-libs-compat-6.8.2-1.EL.33.0.1.i386
$ rpm -q libXrender
libXrender-0.9.8-2.1.el6_8.1.x86_64
libXrender-0.9.8-2.1.el6_8.1.i686
... ... ... ...
Also, download and install Oracle patch 6078836 (p6078836_101330_LINUX.ZIP) to Linux.
$ ls -al /usr/lib/libdb.so.2
-rw-r--r-- 1 root root 5825 Sep 6 21:12 /usr/lib/libdb.so.2
Without it, starting Apache will fail with error:
$IAS_ORACLE_HOME/Apache/Apache/bin/httpd: error while loading shared libraries: libdb.so.2: cannot open shared object file: No such file or directory
- maintenance tools:
ar, gcc, g++, ksh, ld, linux32, make, XDisplay Server
- kernel settings (in /ect/sysctl.conf)
- Number of Open File Descriptors (/etc/security/limits.conf)
* hard nofile 65535
* soft nofile 4096
2. Steps during the migration
2.1 Stop EBS apps clearly on old hosts and run a file counts
$ find EBS_DIR -type f | wc -l
2.2 Wait Linux Admin to rename the new server name to the "old" server name, and do a sync on file systems including user accounts, cron jobs, etc.
2.3 On the new host, make sure it connects to the right database
$ sqlpuse apps/appsPWD
SQL> select instance_name from v$instance
2.4 Run autoconfig on each nodes
$ cd $ADMIN_SCRIPTS_HOME
$ ./adautocfg.sh
2.5. To avoid "adoafmctl.sh (&adoacorectl.sh): exiting with status 150"
$ mv $ORA_CONFIG_HOME/10.1.3/j2ee/oacore/persistence $ORA_CONFIG_HOME/10.1.3/j2ee/oacore/persistence_DELETE
$ mkdir $ORA_CONFIG_HOME/10.1.3/j2ee/oacore/persistence
$ mv $ORA_CONFIG_HOME/10.1.3/j2ee/oafm/persistence $ORA_CONFIG_HOME/10.1.3/j2ee/oafm/persistence_DELETE
$ mkdir $ORA_CONFIG_HOME/10.1.3/j2ee/oafm/persistence
$ mv $ORA_CONFIG_HOME/10.1.3/j2ee/forms/persistence $ORA_CONFIG_HOME/10.1.3/j2ee/forms/persistence_DELETE
$ mkdir $ORA_CONFIG_HOME/10.1.3/j2ee/forms/persistence
If still getting "status 150", stop all services cleanly and start them again.
2.6 Open a F5 ticket to change old IP address to new one in F5.
2.7 Start apps services
With above steps in many instances I did not get any problem in starting EBS services.
2.8. Copy file libgcc_s-2.3.2-stub.so from patch 12415211 ( see Doc ID 761566.1)
3. Troubleshooting
3.1 scp and sftp failed with error:
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: POSSIBLE DNS SPOOFING DETECTED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
The RSA host key for host_name.domain.com has changed,
and the key for the according IP address 177.321.89.56
is unchanged. This could either mean that
DNS SPOOFING is happening or the IP address for the host
and its host key have changed at the same time.
Offending key for IP in /users/userID/.ssh/known_hosts:28
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
e4:5d:1c:12:e5:ab:f5:03:b4:cd:d8:cd:eb:f3:a7:23.
Please contact your system administrator.
Add correct host key in /users/userID/.ssh/known_hosts to get rid of this message.
Offending key in /users/userID/.ssh/known_hosts:7
RSA host key for host_name.domain.com has changed and you have requested strict checking.
Host key verification failed.
lost connection
Fix: $ ssh-keygen -R host_name.domain.com
3.2 Environment variable $PATH does not have the path to its home (such as /users/userID) included
Symptom: Scheduled jobs does not run because scheduler account logs in but does not "see" the script at all.
Fix: add "export PATH=$PATH:$HOME" to file .bashrc
3.3 Shell script can see env variable $HOME.
Symptom: command, such as "cd $HOME", failed.
Fix: hard code the path in the script. (not a good solution).
3.4 When a directory was removed and the old path still exists in the PATH variable, executing a file may get below misleading error. I had to remove the obsolete path from the PATH string.
$ . run_environment
-ksh: . /users/userID/run_environment: not found [No such file or directory]
3.5 RedHat RHEL6 changes mailx on using -a to email attachment. The right syntax:
$ echo "test_body" | mailx -s test-email -a test_attachment.txt emailid@youemail.com
NOTES on Linux commands:
1) Linux version
$ uname -a
Linux host_name.domain.com 2.6.32-642.3.1.el6.x86_64 #1 SMP Sun Jun 26 18:16:44 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux
$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.8 (Santiago)
2) Linux command to find CUP info
$ cat /proc/cpuinfo
$ cat /proc/cpuinfo | grep processor
processor : 0
processor : 1
3) Linux command to find total memory
$ cat /proc/meminfo
MemTotal: 20586472 kB
MemFree: 1406228 kB
Buffers: 873268 kB
Cached: 8353596 kB
SwapCached: 11332 kB
Active: 15413552 kB
Inactive: 2228904 kB
Active(anon): 7687868 kB
Inactive(anon): 729196 kB
Active(file): 7725684 kB
Inactive(file): 1499708 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 8388604 kB
SwapFree: 8288296 kB
... ...
$ free
total used free shared buffers cached
Mem: 20586472 19607488 978984 1504 876480 8510684
-/+ buffers/cache: 10220324 10366148
Swap: 8388604 100308 8288296
(note this instance has more then 350 EBS apps sessions in normal business hours)
Friday, December 2, 2016
Thursday, November 17, 2016
FNDCPASS and APP-FND-02704
When I tried to change apps password by FNDCPASS in R12.1.3, it hang for a while and then the log file shows APP-FND-02704 error.
$ FNDCPASS apps/oldAppsPWD 0 Y system/system_PWD SYSTEM APPLSYS N3WAqt_$EBS
APP-FND-02704: Unable to alter user APPS to change password.
The error does not tell the true cause. The problem in my case is the new password does not meet new rules added by DBA to APPS' profile for password security or single quotes are necessary to make FNDCPASS work with special character in the password.
$ FNDCPASS apps/oldAppsPWD 0 Y system/system_PWD SYSTEM APPLSYS 'N3WAqt_$EBS'
If the system_PWD has special character with it, single quotes are necessary. Otherwise FNDCPASS may not throw out an meaningful error but just does not change apps password.
Also when the password includes $ character, single quotes around it are needed to make sqlplus connection on Linux prompt work. Without it, it will give ORA-01017 error.
$ sqlplus apps/N3WAqt_$EBS
SQL*Plus: Release 10.1.0.5.0 - Production on Thu Nov 17 18:57:12 2016
Copyright (c) 1982, 2005, Oracle. All rights reserved.
ERROR:
ORA-01017: invalid username/password; logon denied
Enter user-name:
$ sqlplus apps/'N3WAqt_$EBS'
Connected to: ....
Or, get into sqlplus first and then connect to apps
$ sqlplus /nolog
SQL> conn apps/N3WAqt_$EBS
Connected.
If the password does not include $ character, the single quotes are not necessary for sqlplus:
$ FNDCPASS apps/'N3WAqt_$EBS' 0 Y system/system_PWD SYSTEM APPLSYS 'Cr8ze#p0Wd'
$ sqlplus apps/Cr8ze#p0Wd
Connected to: ....
Craze $
Additional notes:
$ FNDCPASS apps/oldAppsPWD 0 Y system/system_PWD SYSTEM APPLSYS N3WAqt_$EBS
APP-FND-02704: Unable to alter user APPS to change password.
The error does not tell the true cause. The problem in my case is the new password does not meet new rules added by DBA to APPS' profile for password security or single quotes are necessary to make FNDCPASS work with special character in the password.
Below line changes apps password to N3WAqt_$EBS. Note the single quotes.
$ FNDCPASS apps/oldAppsPWD 0 Y system/system_PWD SYSTEM APPLSYS 'N3WAqt_$EBS'
If the system_PWD has special character with it, single quotes are necessary. Otherwise FNDCPASS may not throw out an meaningful error but just does not change apps password.
Also when the password includes $ character, single quotes around it are needed to make sqlplus connection on Linux prompt work. Without it, it will give ORA-01017 error.
$ sqlplus apps/N3WAqt_$EBS
SQL*Plus: Release 10.1.0.5.0 - Production on Thu Nov 17 18:57:12 2016
Copyright (c) 1982, 2005, Oracle. All rights reserved.
ERROR:
ORA-01017: invalid username/password; logon denied
Enter user-name:
$ sqlplus apps/'N3WAqt_$EBS'
Connected to: ....
Or, get into sqlplus first and then connect to apps
$ sqlplus /nolog
SQL> conn apps/N3WAqt_$EBS
Connected.
If the password does not include $ character, the single quotes are not necessary for sqlplus:
$ FNDCPASS apps/'N3WAqt_$EBS' 0 Y system/system_PWD SYSTEM APPLSYS 'Cr8ze#p0Wd'
$ sqlplus apps/Cr8ze#p0Wd
Connected to: ....
Craze $
Additional notes:
1. FNDCPASS may also give misleading error from changing password when db parameter sec_case_sensitive_logon is set to TRUE.
2. If the new password does not meet the complexity requirements by Oracle database, FNDCPASS will not change APPS password and does not tell what is the real problem. It only says
Working...
APP-FND-02704: Unable to alter user APPS to change password.
Oracle error 28003: has been detected in alterpassword2.
You have to have more complexity in the new password to make it work!
3. When I used FNDCPASS to change APPS password, it refused to do so and give strange error:
FNDCPASS was not able to decrypt password for user 'ABCD1' during applsys password change.
FNDCPASS was not able to decrypt password for user 'ABCD2' during applsys password change.
FNDCPASS was not able to decrypt password for user 'ABCD3' during applsys password change.
ABCD1, ABCD2 and ABCD3 are not database account at all. But they are EBS users in inactive status for many years. Apparently FNDCPASS does not like their EBS password or security. The fix is to change their EBS password first, and then FNDCPASS is able to change APPS password.
Tuesday, November 15, 2016
EBS Mailer status and troubleshooting
Some times, users do not get emails/alerts from EBS R12.1.3 even workflow Mailer is up and running. Table wf_notifications may give different status:
RECIPIENT_ROLE MESSAGE_TYPE STATUS MAIL_STATUS TO_USER
APINVGR APINVAPR OPEN FAILED AP GROUP
APINVGR APINVAPR OPEN MAIL AP GROUP
APINVGR APINVAPR OPEN SENT AP GROUP
APINVGR APINVAPR CLOSED SENT AP GROUP
Seems to me that only mails with MAIL_STATUS "SENT" were sent out successfully. If it stays in "MAIL", the Mailer may get stuck somewhere and may need a re-start.
Below query checks the email status:
SELECT notification_id N_ID, to_char(begin_date, 'mm-dd-yyyy hh24:mi:ss') beginDate, recipient_role, message_type, message_name, status, mail_status, context, to_user, subject
FROM wf_notifications
WHERE begin_date > sysdate - 5 -- List only emails within 5 days
ORDER by notification_id desc;
Here is a notification with FAILED status. Note: the record values could be dynamically changed with approval workflow's move.
N_ID RECIPIENT_ROLE MESSAGE_TYPE STATUS MAIL_STATUS CONTEXT TO_USER
15331513 APINVGR APINVAPR OPEN FAILED APINVAPR:11345495_1:116020 AP GROUP
To troubleshoot the failed notification, find and check the Workflow status first to see if any error.
You shall verify user APINVGR status and email address are correct.
Also try the Mailer logs in $APPLCSF/$APPLLOG to see if you are lucky to get some errors.
$ ls -lrt $APPLCSF/$APPLLOG/FNDCP*.txt
$ grep ":ERROR:" $APPLCSF/$APPLLOG/FNDCPGSC*.txt > mlrerr.log
$ grep "Exception:" $APPLCSF/$APPLLOG/FNDCPGSC*.txt > mlrexc.log
$ grep ":UNEXPECTED:" $APPLCSF/$APPLLOG/FNDCPGSC*.txt > mlrunexp.log
If the logs do not tell much, Doc ID 1051421.1 is a good document to follow on enabling debug for Mailer Service Component :
- In OAM : Site Map (or, WF Manager) > Notification Mailer > Edit > Advanced > Go to Step 2 > Change "Log Level" to "Statement" > Apply
- Bounce the Mailer container to generate a fresh log file
In OAM: Site Map > Generic Services > under Status Overview > Next (twice) > Generic Service Component Container > select Workflow Mailer Service > Restart (from the dropdown on the top).
(or, Dashboard > dropdown "Application Services" Go > Next ... to find Generic Service Component Container > ... )
Note it may take 5 minutes or so for a re-start cycle.
Even Workflow Mailer Service is running, Notification Mailer could be down if two $CONTEXT_FILE parameters are not right, because autoconfig will overwrite them if company's SMTP server name, e.g. mail.comany_name.com, is only entered under Outpound EMail Account section on OAM webpage. The correct way is to define them in $CONTEXT_FILE:
<hostname oa_var="s_smtphost">mail</hostname>
<domain oa_var="s_smtpdomainname">company_name.com</domain>
</oa_smtp_server>
If individual users do not receive notifications or their mails have Failed status, check two tables to see if their preference is turned off (disabled)which could happen when the Workflow Mailer attempts to send a notification to an invalid email address or when email server crashed and caused an outage of hours (Doc ID 360541.1).
SQL> select * from FND_USER_PREFERENCES where preference_value = 'DISABLED';
SQL> select * from WF_LOCAL_ROLES where notification_preference = 'DISABLED';
To enable it in FND_USER_PREFERENCES: log onto EBS as user who has the issue > click on Preferences > scroll down to Notifications > Email Style > dropdown "HTML mail with attachments"
To populate above change to WF_LOCAL_ROLES and WF_ROLES:
as System Administrator Responsibility > Define User form> query up user with issue:
a. Remove the Person Field > save.
b. Re-query the same user > add back the Person Field > Save
c. This effort does an internal synchronization for the particular user.
Notes on synchronizing workflow tables:
1. In my R12.1.3 instances, above actions synchronizes Status column in tables FND_USER and WF_LOCAL_ROLES. Doc 1213304.1 has info on three sync concurrent programs. In 11i, the synchronization seems more difficult. See Doc 728331.1 and Doc 364647.1.
2. Two steps are necessary to change a user's email address and synchronize it to workflow tables. Prior to about steps on Define User form, go to Setup (mostly under a Purchase/PO responsibility) > Personal > Employee form > update user's email address first.
3. If the Description column is out-of-date in table wf_local_roles, most likely newer data in table PER_ALL_PEOPLE_F is not populated to WF tables. The solution is to run concurrent program 'Synchronize WF LOCAL tables' with parameter FND_USR. That synchronized WF tables in my instances.
After this, run concurrent program Resend Failed/Error Workflow Notifications to resend those FAILED notifications.
This program should send FAILED records out from table wf_notifications. In my case, account APINVGR uses a service email address and each time after Mailer sends a notification to it, its notification_preference becomes DISABLED for unknown reason. After I replace that email address with a real user's email address, the problem is resolved.
If Workflow Mailer stopped, workflow notification will not be sent. Below query shows Workflow Mailer is not running, it needs a fix:
SQL> SELECT component_name, component_status, component_status_info
FROM fnd_svc_components_v
WHERE component_name like 'Workflow%';
component_name component_status component_status_info
------------------------------------------------------------- ----------------------- --------------
Workflow Deferred Agent Listener RUNNING
Workflow Deferred Notification Agent Listener RUNNING
Workflow Error Agent Listener RUNNING
Workflow Inbound JMS Agent Listener STOPPED
Workflow Inbound Notifications Agent Listener RUNNING
Workflow Java Deferred Agent Listener RUNNING
Workflow Java Error Agent Listener RUNNING
Workflow Notification Mailer DEACTIVATED_SYSTEM xxxxx
Doc ID 562551.1 has useful queries. If you want to check into an individual notification, get the NOTIFICATION_ID from above query result and run below quest to see its status from tables WF_NOTIFICATIONS, WF_DEFERRED and WF_NOTIFICATION_OUT.
NOTE: a Failed notification will not reach WF_NOTIFICATION_OUT table
SQL> SELECT
n.begin_date, n.status, n.mail_status, n.recipient_role, de.def_enq_time, de.def_deq_time,
de.def_state, ou.out_enq_time, ou.out_deq_time, ou.out_state
FROM applsys.wf_notifications n,
(SELECT d.enq_time def_enq_time,
d.deq_time def_deq_time,
TO_NUMBER((SELECT VALUE
FROM TABLE(d.user_data.parameter_list)
WHERE NAME = 'NOTIFICATION_ID')) d_notification_id,
msg_state def_state
FROM applsys.aq$wf_deferred d
WHERE d.corr_id = 'APPS:oracle.apps.wf.notification.send') de,
(SELECT o.deq_time out_deq_time,
o.enq_time out_enq_time,
TO_NUMBER((SELECT str_value
FROM TABLE(o.user_data.header.properties)
WHERE NAME = 'NOTIFICATION_ID')) o_notification_id,
msg_state out_state
FROM applsys.aq$wf_notification_out o) ou
WHERE n.notification_id = 15331600
AND n.notification_id = de.d_notification_id(+)
AND n.notification_id = ou.o_notification_id(+) ;
Check if there are stuck messages in The PROCESS Folder:
SQL> SELECT trunc(win.enq_time) PROCESSED_TIME, wfn.status NOTIFICATION_STATUS, COUNT(*) NO_EMAILS
FROM (select wi.msgid, wi.corrid, wi.enq_time, wi.state, wi.sender_name,
(select str_value from table(wi.user_data.header.properties) where name = 'NOTIFICATION_ID') notification_id
from apps.WF_NOTIFICATION_IN wi) win, apps.WF_ERROR werr, apps.wf_notifications wfn
where win.notification_id = wfn.notification_id
and win.msgid = werr.msgid (+)
--and win.enq_time >= sysdate - 4/24 -- In the Process Folder for more than 4 hours
and wfn.status = 'OPEN'
group by trunc(win.enq_time), wfn.status;
RECIPIENT_ROLE MESSAGE_TYPE STATUS MAIL_STATUS TO_USER
APINVGR APINVAPR OPEN FAILED AP GROUP
APINVGR APINVAPR OPEN MAIL AP GROUP
APINVGR APINVAPR OPEN SENT AP GROUP
APINVGR APINVAPR CLOSED SENT AP GROUP
Seems to me that only mails with MAIL_STATUS "SENT" were sent out successfully. If it stays in "MAIL", the Mailer may get stuck somewhere and may need a re-start.
Below query checks the email status:
SELECT notification_id N_ID, to_char(begin_date, 'mm-dd-yyyy hh24:mi:ss') beginDate, recipient_role, message_type, message_name, status, mail_status, context, to_user, subject
FROM wf_notifications
WHERE begin_date > sysdate - 5 -- List only emails within 5 days
ORDER by notification_id desc;
Here is a notification with FAILED status. Note: the record values could be dynamically changed with approval workflow's move.
N_ID RECIPIENT_ROLE MESSAGE_TYPE STATUS MAIL_STATUS CONTEXT TO_USER
15331513 APINVGR APINVAPR OPEN FAILED APINVAPR:11345495_1:116020 AP GROUP
To troubleshoot the failed notification, find and check the Workflow status first to see if any error.
You shall verify user APINVGR status and email address are correct.
Also try the Mailer logs in $APPLCSF/$APPLLOG to see if you are lucky to get some errors.
$ ls -lrt $APPLCSF/$APPLLOG/FNDCP*.txt
$ grep ":ERROR:" $APPLCSF/$APPLLOG/FNDCPGSC*.txt > mlrerr.log
$ grep "Exception:" $APPLCSF/$APPLLOG/FNDCPGSC*.txt > mlrexc.log
$ grep ":UNEXPECTED:" $APPLCSF/$APPLLOG/FNDCPGSC*.txt > mlrunexp.log
If the logs do not tell much, Doc ID 1051421.1 is a good document to follow on enabling debug for Mailer Service Component :
- In OAM : Site Map (or, WF Manager) > Notification Mailer > Edit > Advanced > Go to Step 2 > Change "Log Level" to "Statement" > Apply
- Bounce the Mailer container to generate a fresh log file
In OAM: Site Map > Generic Services > under Status Overview > Next (twice) > Generic Service Component Container > select Workflow Mailer Service > Restart (from the dropdown on the top).
(or, Dashboard > dropdown "Application Services" Go > Next ... to find Generic Service Component Container > ... )
Note it may take 5 minutes or so for a re-start cycle.
Even Workflow Mailer Service is running, Notification Mailer could be down if two $CONTEXT_FILE parameters are not right, because autoconfig will overwrite them if company's SMTP server name, e.g. mail.comany_name.com, is only entered under Outpound EMail Account section on OAM webpage. The correct way is to define them in $CONTEXT_FILE:
<hostname oa_var="s_smtphost">mail</hostname>
<domain oa_var="s_smtpdomainname">company_name.com</domain>
</oa_smtp_server>
If individual users do not receive notifications or their mails have Failed status, check two tables to see if their preference is turned off (disabled)which could happen when the Workflow Mailer attempts to send a notification to an invalid email address or when email server crashed and caused an outage of hours (Doc ID 360541.1).
SQL> select * from FND_USER_PREFERENCES where preference_value = 'DISABLED';
SQL> select * from WF_LOCAL_ROLES where notification_preference = 'DISABLED';
To enable it in FND_USER_PREFERENCES: log onto EBS as user who has the issue > click on Preferences > scroll down to Notifications > Email Style > dropdown "HTML mail with attachments"
To populate above change to WF_LOCAL_ROLES and WF_ROLES:
as System Administrator Responsibility > Define User form> query up user with issue:
a. Remove the Person Field > save.
b. Re-query the same user > add back the Person Field > Save
c. This effort does an internal synchronization for the particular user.
Notes on synchronizing workflow tables:
1. In my R12.1.3 instances, above actions synchronizes Status column in tables FND_USER and WF_LOCAL_ROLES. Doc 1213304.1 has info on three sync concurrent programs. In 11i, the synchronization seems more difficult. See Doc 728331.1 and Doc 364647.1.
2. Two steps are necessary to change a user's email address and synchronize it to workflow tables. Prior to about steps on Define User form, go to Setup (mostly under a Purchase/PO responsibility) > Personal > Employee form > update user's email address first.
3. If the Description column is out-of-date in table wf_local_roles, most likely newer data in table PER_ALL_PEOPLE_F is not populated to WF tables. The solution is to run concurrent program 'Synchronize WF LOCAL tables' with parameter FND_USR. That synchronized WF tables in my instances.
After this, run concurrent program Resend Failed/Error Workflow Notifications to resend those FAILED notifications.
If Workflow Mailer stopped, workflow notification will not be sent. Below query shows Workflow Mailer is not running, it needs a fix:
SQL> SELECT component_name, component_status, component_status_info
FROM fnd_svc_components_v
WHERE component_name like 'Workflow%';
component_name component_status component_status_info
------------------------------------------------------------- ----------------------- --------------
Workflow Deferred Agent Listener RUNNING
Workflow Deferred Notification Agent Listener RUNNING
Workflow Error Agent Listener RUNNING
Workflow Inbound JMS Agent Listener STOPPED
Workflow Inbound Notifications Agent Listener RUNNING
Workflow Java Deferred Agent Listener RUNNING
Workflow Java Error Agent Listener RUNNING
Workflow Notification Mailer DEACTIVATED_SYSTEM xxxxx
Doc ID 562551.1 has useful queries. If you want to check into an individual notification, get the NOTIFICATION_ID from above query result and run below quest to see its status from tables WF_NOTIFICATIONS, WF_DEFERRED and WF_NOTIFICATION_OUT.
NOTE: a Failed notification will not reach WF_NOTIFICATION_OUT table
SQL> SELECT
n.begin_date, n.status, n.mail_status, n.recipient_role, de.def_enq_time, de.def_deq_time,
de.def_state, ou.out_enq_time, ou.out_deq_time, ou.out_state
FROM applsys.wf_notifications n,
(SELECT d.enq_time def_enq_time,
d.deq_time def_deq_time,
TO_NUMBER((SELECT VALUE
FROM TABLE(d.user_data.parameter_list)
WHERE NAME = 'NOTIFICATION_ID')) d_notification_id,
msg_state def_state
FROM applsys.aq$wf_deferred d
WHERE d.corr_id = 'APPS:oracle.apps.wf.notification.send') de,
(SELECT o.deq_time out_deq_time,
o.enq_time out_enq_time,
TO_NUMBER((SELECT str_value
FROM TABLE(o.user_data.header.properties)
WHERE NAME = 'NOTIFICATION_ID')) o_notification_id,
msg_state out_state
FROM applsys.aq$wf_notification_out o) ou
WHERE n.notification_id = 15331600
AND n.notification_id = de.d_notification_id(+)
AND n.notification_id = ou.o_notification_id(+) ;
Check if there are stuck messages in The PROCESS Folder:
SQL> SELECT trunc(win.enq_time) PROCESSED_TIME, wfn.status NOTIFICATION_STATUS, COUNT(*) NO_EMAILS
FROM (select wi.msgid, wi.corrid, wi.enq_time, wi.state, wi.sender_name,
(select str_value from table(wi.user_data.header.properties) where name = 'NOTIFICATION_ID') notification_id
from apps.WF_NOTIFICATION_IN wi) win, apps.WF_ERROR werr, apps.wf_notifications wfn
where win.notification_id = wfn.notification_id
and win.msgid = werr.msgid (+)
--and win.enq_time >= sysdate - 4/24 -- In the Process Folder for more than 4 hours
and wfn.status = 'OPEN'
group by trunc(win.enq_time), wfn.status;
Wednesday, October 19, 2016
FTP and SFTP
1. Run ftp using a specific account interactively
$ ftp -inv remote_host.domain.com
Connected to remote_host.domain.com (10.133.67.38).
220 Microsoft FTP Service
Remote system type is Windows_NT.
ftp> user ftpuser
331 Password required for ftpuser.
Password:
230 User logged in.
ftp>
Note below line does not work:
$ ftp ftpuser@remote_host.domain.com
ftp: ftpuser@remote_host.domain.com: Name or service not known
ftp> exit
2. Run FTP in a shell script
Assume FTP login info is saved in file $HOME/.netpw in format:
put $LOCAL_PATH/output.txt ouput1.txt
bye
EndFTP
}
3. Try "man ftp" for help.
4. SFTP options:
$ sftp -oport=2222 ftpuser@remote_host.domain.com
Connecting to remote_host.domain.com...
WARNING: Logon attempts are audited. Access and use allowed for authorized purposes only. Violators will be prosecuted
$ sftp -oIdentityFile=/path/to/sftp_key/id_dsa ftpuser@remote_host.domain.com
Also see https://erpondb.blogspot.com/2015/11/run-scp-or-sftp-without-password.html
$ ftp -inv remote_host.domain.com
Connected to remote_host.domain.com (10.133.67.38).
220 Microsoft FTP Service
Remote system type is Windows_NT.
ftp> user ftpuser
331 Password required for ftpuser.
Password:
230 User logged in.
ftp>
Note below line does not work:
$ ftp ftpuser@remote_host.domain.com
ftp: ftpuser@remote_host.domain.com: Name or service not known
ftp> exit
2. Run FTP in a shell script
Assume FTP login info is saved in file $HOME/.netpw in format:
machine WintEdi.domain.COM login ANET/userID password userPWD
REMOTE_PATH=......
LOCAL_PATH=......
REMOTE_MACHINE=WintEdi.domain.COM
REMOTE_PATH=......
LOCAL_PATH=......
REMOTE_MACHINE=WintEdi.domain.COM
function ftp_to_remote
{
ftpuserid=`grep $REMOTE_MACHINE $HOME/.netpw|cut -f4 -d' '`
ftpuserpwd=`grep $REMOTE_MACHINE $HOME/.netpw|cut -f6 -d' '`
ftp -inv <<EndFTP
open $REMOTE_MACHINE
user $ftpuserid $ftpuserpwd
cd $REMOTE_PATHput $LOCAL_PATH/output.txt ouput1.txt
bye
EndFTP
}
3. Try "man ftp" for help.
4. SFTP options:
$ sftp -oport=2222 ftpuser@remote_host.domain.com
Connecting to remote_host.domain.com...
WARNING: Logon attempts are audited. Access and use allowed for authorized purposes only. Violators will be prosecuted
$ sftp -oIdentityFile=/path/to/sftp_key/id_dsa ftpuser@remote_host.domain.com
Also see https://erpondb.blogspot.com/2015/11/run-scp-or-sftp-without-password.html
Troubleshhot UTL_FILE error
Ran a PL/SQL code and got error message:
ORA-06512: at
"SYS.UTL_FILE", line 536
ORA-29283: invalid file ope
declare
*
ERROR at line 1:
ORA-29283: invalid file operation
ORA-06512: at "SYS.UTL_FILE", line 536
ORA-29283: invalid file operation
ORA-06512: at line 90
Two steps to troubleshoot the error:
A. Find which directory it tries to write the file to, and verify it is defined in the database:
SQL> select * from dba_directories where directory_name='FTPOUT';
The directory could be owned by SYS. But APPS should be granted READ and WRITE.
SQL> SELECT * FROM dba_tab_privs WHERE table_name = 'FTPOUT';
B. Run a short code as APPS to test it. If it works, file test_UTL.txt shall create in the directory.
DECLARE
fileHandler UTL_FILE.FILE_TYPE;
v_error_msg varchar2(1000);
BEGIN
fileHandler := UTL_FILE.FOPEN('FTPOUT', 'test_UTL.txt', 'W');
UTL_FILE.PUT_LINE(fileHandler, 'Writing TO a test file\n');
UTL_FILE.PUTF(fileHandler, 'Writing 2nd line to test file\n');
UTL_FILE.FCLOSE(fileHandler);
EXCEPTION
WHEN utl_file.invalid_path THEN
raise_application_error(-20001, 'ERROR: Invalid PATH for the file.');
when others then
v_error_msg := substr(sqlerrm,1,100);
utl_file.fclose(fileHandler);
dbms_output.put_line('Error message is - ' || v_error_msg);
raise;
END;
/
A MAX linesize written into file test_UTL.txt can be specified in the call:
UTL_FILE.FOPEN('FTPOUT', 'test_UTL.txt', 'W', 32767)
In EBS environment, the path for $APPLPTMP is not necessary to be in DBA_DIRECTORIES.
UPDATES in 2018:
During a database 12c upgrade, somehow the OS env variable ORA_NLS10 was unset. That made " utl_file.fcopy ('FTPOUT','test.txt','FTPOUT','test_copy.txt'); ", which is used by Export of AAD rules using Oracle Forms Lock/Unlock option, generate file test_copy.txt 0 in size and UTL_FILE.PUT_LINE fail if output file test_UTL.txt is larger than 1 MB with below errors:
declare
*
ERROR at line 1:
ORA-29282: invalid file ID
ORA-06512: at "SYS.UTL_FILE", line 77
ORA-06512: at "SYS.UTL_FILE", line 690
ORA-06512: at line 256
ORA-29285: file write error
or
Also check the init parameter file and value for
utl_file_dir in v$parameter table.
declare
*
ERROR at line 1:
ORA-29285: file write error
ORA-06512: at line 258
The fix is to set env variable ORA_NLS10 on database server to $ORACLE_HOME/nls/data/9idata
ORA-06512: at
"SYS.UTL_FILE", line 536
ORA-29283: invalid file ope
declare
*
ERROR at line 1:
ORA-29283: invalid file operation
ORA-06512: at "SYS.UTL_FILE", line 536
ORA-29283: invalid file operation
ORA-06512: at line 90
Two steps to troubleshoot the error:
A. Find which directory it tries to write the file to, and verify it is defined in the database:
SQL> select * from dba_directories where directory_name='FTPOUT';
The directory could be owned by SYS. But APPS should be granted READ and WRITE.
SQL> SELECT * FROM dba_tab_privs WHERE table_name = 'FTPOUT';
B. Run a short code as APPS to test it. If it works, file test_UTL.txt shall create in the directory.
DECLARE
fileHandler UTL_FILE.FILE_TYPE;
v_error_msg varchar2(1000);
BEGIN
fileHandler := UTL_FILE.FOPEN('FTPOUT', 'test_UTL.txt', 'W');
UTL_FILE.PUT_LINE(fileHandler, 'Writing TO a test file\n');
UTL_FILE.PUTF(fileHandler, 'Writing 2nd line to test file\n');
UTL_FILE.FCLOSE(fileHandler);
EXCEPTION
WHEN utl_file.invalid_path THEN
raise_application_error(-20001, 'ERROR: Invalid PATH for the file.');
when others then
v_error_msg := substr(sqlerrm,1,100);
utl_file.fclose(fileHandler);
dbms_output.put_line('Error message is - ' || v_error_msg);
raise;
END;
/
A MAX linesize written into file test_UTL.txt can be specified in the call:
UTL_FILE.FOPEN('FTPOUT', 'test_UTL.txt', 'W', 32767)
In EBS environment, the path for $APPLPTMP is not necessary to be in DBA_DIRECTORIES.
UPDATES in 2018:
During a database 12c upgrade, somehow the OS env variable ORA_NLS10 was unset. That made " utl_file.fcopy ('FTPOUT','test.txt','FTPOUT','test_copy.txt'); ", which is used by Export of AAD rules using Oracle Forms Lock/Unlock option, generate file test_copy.txt 0 in size and UTL_FILE.PUT_LINE fail if output file test_UTL.txt is larger than 1 MB with below errors:
declare
*
ERROR at line 1:
ORA-29282: invalid file ID
ORA-06512: at "SYS.UTL_FILE", line 77
ORA-06512: at "SYS.UTL_FILE", line 690
ORA-06512: at line 256
ORA-29285: file write error
or
Also check the init parameter file and value for
utl_file_dir in v$parameter table.
declare
*
ERROR at line 1:
ORA-29285: file write error
ORA-06512: at line 258
The fix is to set env variable ORA_NLS10 on database server to $ORACLE_HOME/nls/data/9idata
By the way, the quick way to test $APPLPTMP and UTL_FILE_DIR in EBS is to run one line:
SQL> exec FND_FILE.PUT_LINE(fnd_file.log, 'Hello World!');
A new file with a name like l0009966.tmp shall be created in the first directory specified in the db parameter utl_file_dir, containing 'Hello World!'.
Friday, September 9, 2016
EBS login page get frozen and then timed out
A R12 EBS web site stayed frozen and did not re-direct to the login page. It seems it was waiting something or was trying to gain an access, until timed-out with Internal error. The message in Apache log:
[Thu Sep 8 07:17:03 2016] [warn] [client 172.xxx.xx.xxx] oc4j_socket_recvfull timed out
[Thu Sep 8 07:17:03 2016] [error] [client 172.xxx.xx.xxx] [ecid: xxx] mod_oc4j: request to OC4J hostname.domain.com:21700 failed: Connect failed
I checked around and saw errors in below log file:
<jcache_port oa_var="s_java_object_cache_port" oa_type="PORT" base="12345" step="1" range="-1" label="Java Object Cache Port">12385</jcache_port>
I verified port 12385 was in use:
$ netstat -an | grep 12385
tcp 1480 0 ::ffff:157.121.53.0:12385 ::ffff:167.69.38.184:1571 ESTABLISHED
Then, I asked System Admin to run below lines as root to get the detail:
# lsof -i :12385
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 7135 ebsdev2 194u IPv6 156352 0t0 TCP hostname.domain.com:12385->dbHost.domian.com:rdb-dbs-disp (ESTABLISHED)
# ls -l /proc/7135/exe
lrwxrwxrwx 1 applmgr2 users 0 Sep 7 13:31 /proc/7135/exe -> /path/to/apps/tech_st/10.1.3/appsutil/jdk/bin/java
It appears port 12385 was used by another EBS instance EBSDEV2 on the same server.
The fix is easy. I stopped all EBS services for instance EBSDEV2, and then started the ones that was frozen first. All worked fine.
Additional Notes:
- In my R12.1.3 instances, I experienced EBS site is inaccessible with below errors in EBS first webpage. The cause could be a change in network switch or some interruption on host/database. The fix in my cases is to make sure apps account in database is not locked and recycle EBS services.
Unable to generate forwarding URL. Exception: oracle.apps.fnd.common.AppsException: oracle.apps.jtf.base.resources.FrameworkException
Unable to generate forwarding URL. Exception: java.lang.NullPointerException
- Same fix for intermittent and sporadic login issue with error
You have encountered an unexpected error. Please contact the System Administrator for assistance.
- Below error in EBS first webpage could be caused by database error (e.g. database is down or a tablespace is full):
Unable to generate forwarding URL. Exception: oracle.apps.fnd.common.AppsException: oracle.apps.fnd.common.PoolException: Exception creating new Poolable object.
- Got below error on 2nd page (login page) from re-directing 1st webpage on IE browser. The problem went away after deleted IE cache files.
Unable to authenticate session
- After Apps nodes rebooted unexpectedly, EBS site webpage does not work even Apache started fine without any error. The problem was related to some kind of cache issue. The final fix is to bounce the database (surprisingly).
[Thu Sep 8 07:17:03 2016] [warn] [client 172.xxx.xx.xxx] oc4j_socket_recvfull timed out
[Thu Sep 8 07:17:03 2016] [error] [client 172.xxx.xx.xxx] [ecid: xxx] mod_oc4j: request to OC4J hostname.domain.com:21700 failed: Connect failed
I checked around and saw errors in below log file:
$LOG_HOME/ora/10.1.3/j2ee/oacore/oacore_default_group_1/application.log
16/09/07 16:01:54.978 10.1.3.5.0 Started
16/09/07 16:02:03.327 html: 10.1.3.5.0 Started
16/09/07 16:15:40.920 html: Error initializing servlet
java.lang.NoClassDefFoundError: Could not initialize class oracle.apps.fnd.profiles.Profiles at oracle.apps.fnd.sso.AppsLoginRedirect.AppsSetting(AppsLoginRedirect.java:239)
at oracle.apps.fnd.sso.AppsLoginRedirect.init(AppsLoginRedirect.java:287)
at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].server.http.HttpApplication.loadServlet(HttpApplication.java:2529)
at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].server.http.HttpApplication.findServlet(HttpApplication.java:5008)
at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].server.http.HttpApplication.findServlet(HttpApplication.java:4932)
at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].server.http.HttpApplication.getRequestDispatcher(HttpApplication.java:3140)
at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].server.http.HttpRequestHandler.doProcessRequest(HttpRequestHandler.java:775)
at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:458)
at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].server.http.AJPRequestHandler.run(AJPRequestHandler.java:313)
at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].server.http.AJPRequestHandler.run(AJPRequestHandler.java:199)
at oracle.oc4j.network.ServerSocketReadHandler$SafeRunnable.run(ServerSocketReadHandler.java:260)
at oracle.oc4j.network.ServerSocketAcceptHandler.procClientSocket(ServerSocketAcceptHandler.java:234)
at oracle.oc4j.network.ServerSocketAcceptHandler.access$700(ServerSocketAcceptHandler.java:29)
at oracle.oc4j.network.ServerSocketAcceptHandler$AcceptHandlerHorse.run(ServerSocketAcceptHandler.java:879)
at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.java:303)
at java.lang.Thread.run(Thread.java:619)
16/09/07 16:02:03.327 html: 10.1.3.5.0 Started
16/09/07 16:15:40.920 html: Error initializing servlet
java.lang.NoClassDefFoundError: Could not initialize class oracle.apps.fnd.profiles.Profiles at oracle.apps.fnd.sso.AppsLoginRedirect.AppsSetting(AppsLoginRedirect.java:239)
at oracle.apps.fnd.sso.AppsLoginRedirect.init(AppsLoginRedirect.java:287)
at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].server.http.HttpApplication.loadServlet(HttpApplication.java:2529)
at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].server.http.HttpApplication.findServlet(HttpApplication.java:5008)
at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].server.http.HttpApplication.findServlet(HttpApplication.java:4932)
at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].server.http.HttpApplication.getRequestDispatcher(HttpApplication.java:3140)
at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].server.http.HttpRequestHandler.doProcessRequest(HttpRequestHandler.java:775)
at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:458)
at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].server.http.AJPRequestHandler.run(AJPRequestHandler.java:313)
at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].server.http.AJPRequestHandler.run(AJPRequestHandler.java:199)
at oracle.oc4j.network.ServerSocketReadHandler$SafeRunnable.run(ServerSocketReadHandler.java:260)
at oracle.oc4j.network.ServerSocketAcceptHandler.procClientSocket(ServerSocketAcceptHandler.java:234)
at oracle.oc4j.network.ServerSocketAcceptHandler.access$700(ServerSocketAcceptHandler.java:29)
at oracle.oc4j.network.ServerSocketAcceptHandler$AcceptHandlerHorse.run(ServerSocketAcceptHandler.java:879)
at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.java:303)
at java.lang.Thread.run(Thread.java:619)
This error could be port for s_java_object_cache_port is not available.
$ grep s_java_object_cache_port $CONTEXT_FILE<jcache_port oa_var="s_java_object_cache_port" oa_type="PORT" base="12345" step="1" range="-1" label="Java Object Cache Port">12385</jcache_port>
I verified port 12385 was in use:
$ netstat -an | grep 12385
tcp 1480 0 ::ffff:157.121.53.0:12385 ::ffff:167.69.38.184:1571 ESTABLISHED
Then, I asked System Admin to run below lines as root to get the detail:
# lsof -i :12385
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 7135 ebsdev2 194u IPv6 156352 0t0 TCP hostname.domain.com:12385->dbHost.domian.com:rdb-dbs-disp (ESTABLISHED)
# ls -l /proc/7135/exe
lrwxrwxrwx 1 applmgr2 users 0 Sep 7 13:31 /proc/7135/exe -> /path/to/apps/tech_st/10.1.3/appsutil/jdk/bin/java
It appears port 12385 was used by another EBS instance EBSDEV2 on the same server.
The fix is easy. I stopped all EBS services for instance EBSDEV2, and then started the ones that was frozen first. All worked fine.
Additional Notes:
- In my R12.1.3 instances, I experienced EBS site is inaccessible with below errors in EBS first webpage. The cause could be a change in network switch or some interruption on host/database. The fix in my cases is to make sure apps account in database is not locked and recycle EBS services.
Unable to generate forwarding URL. Exception: oracle.apps.fnd.common.AppsException: oracle.apps.jtf.base.resources.FrameworkException
Unable to generate forwarding URL. Exception: java.lang.NullPointerException
- Same fix for intermittent and sporadic login issue with error
You have encountered an unexpected error. Please contact the System Administrator for assistance.
- Below error in EBS first webpage could be caused by database error (e.g. database is down or a tablespace is full):
Unable to generate forwarding URL. Exception: oracle.apps.fnd.common.AppsException: oracle.apps.fnd.common.PoolException: Exception creating new Poolable object.
- Got below error on 2nd page (login page) from re-directing 1st webpage on IE browser. The problem went away after deleted IE cache files.
Unable to authenticate session
- After Apps nodes rebooted unexpectedly, EBS site webpage does not work even Apache started fine without any error. The problem was related to some kind of cache issue. The final fix is to bounce the database (surprisingly).
- After DBA applied patches and made changes to Java on database server, EBS site re-directs to login page that is not coming up, instead a blank page is being displayed. I check around and get below result:
SQL> select fnd_web_sec.validate_password('GUEST','oracle') from dual;
FND_WEB_SEC.VALIDATE_PASSWORD('GUEST','ORACLE')
--------------------------------------------------------------------------------
N
--------------------------------------------------------------------------------
N
SQL> select fnd_message.get from dual;
GET
--------------------------------------------------------------------------------
Oracle error -29548: ORA-29548: Java system class reported: release of Java system classes in the database (12.1.0.2.200714 1.6) does not match that of the oracle executable (12.1.0.2.171017 1.6) has been detected in FND_WEB_SEC.VALIDATE_PASSWORD.
--------------------------------------------------------------------------------
Oracle error -29548: ORA-29548: Java system class reported: release of Java system classes in the database (12.1.0.2.200714 1.6) does not match that of the oracle executable (12.1.0.2.171017 1.6) has been detected in FND_WEB_SEC.VALIDATE_PASSWORD.
It matches exact same error as in Doc ID 1673030.1, which says "If it shows ORA-29548 errors, then issue is with the database OJVM. Rebuild it using Note 2149019.1 How to Install, Remove, Reload, Validate and Repair the JVM Component in an Oracle Database". DBA had to fix the problem.
Thursday, September 8, 2016
Run concurrent jobs on EBS host in a different account
Users need to run shell scripts to submit concurrent jobs and then do other things. If you do not want to share the Applmgr password to other users, you can set up a separate OS account for users to run their scripts. Assume Oracle EBS is installed under /u01/app/ by OS user Applmgr.
$ hostname
hostname1q
$ echo $APPL_TOP
/u01/app/EBSQA/apps/apps_st/appl
$ echo $ORACLE_HOME
/u01/app/EBSQA/apps/tech_st/10.1.2
$ echo $TWO_TASK
EBSQA
1. create user batchuser on same host hostname1q
2. install Oracle client on the same host but under a different location
$ whoami
batchuser
$ echo $ORACLE_HOME
/batchu01/app/product/11.2.0/EBIZ
$ echo ORACLE_SID
EBSQA
3. create a file to setup the environment variables
$ whoami
batchuser
$ more setenv.ebsqa
export ORACLE_HOME=/batchu01/app/product/11.2.0/EBIZ
export ORACLE_SID=EBSQA
export TNS_ADMIN=/batchu01/app/product/11.2.0/EBIZ/network/admin
export LD_LIBRARY_PATH=/batchu01/app/product/11.2.0/EBIZ/lib
export PATH=/batchu01/app/product/11.2.0/EBIZ/bin:/usr/bin:/usr/local/bin:/etc
. /u01/app/EBSQA/apps/apps_st/appl/EBSQA_hostname1q.env
export PATH=$PATH:/batchu01/app/product/11.2.0/EBIZ/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH
echo `which CONCSUB`
echo `which sqlldr`
$ . ./setenv.ebsqa
/u01/app/EBSQA/apps/apps_st/appl/fnd/12.0.0/bin/CONCSUB
/batchu01/app/product/11.2.0/EBIZ/bin/sqlldr
4. now, submit a concurrent job by $FND_TOP/bin/CONCSUB
$ CONCSUB APPS/appsPWD SYSADMIN "System Administrator" EBSadmin WAIT=Y CONCURRENT FND FNDSCURS PROGRAM_NAME='"Active Users"'
Submitted request 4654627 for CONCURRENT FND FNDSCURS PROGRAM_NAME="Active Users"
Normal completion
$ more $APPLCSF/log/l4654627.req
+---------------------------------------------------------------------------+
Application Object Library: Version : 12.0.0
Copyright (c) 1979, 1999, Oracle Corporation. All rights reserved.
FNDSCURS module: Active Users
+---------------------------------------------------------------------------+
Current system time is 28-JUL-2016 14:08:59
.. ... ...
5. Troubleshooting: if you get below error, the library file may be wrong or bad. Copy the file from EBS installation folder /u01/app/EBSQA/apps/tech_st/10.1.2/lib to current library folder /batchu01/app/product/11.2.0/EBIZ/lib.
CONCSUB: error while loading shared libraries: libclntsh.so.10.1: wrong ELF class: ELFCLASS64
CONCSUB: error while loading shared libraries: libnnz10.so: cannot open shared object file: No such file or directory
~~~~~~~~~~~~~~~~~~~~ A sample script ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
#!/bin/ksh
# Some variables
CONC_APPUSR=EBSBATCH # an EBS user who runs this cm job
CONC_LOGUSR=apps
MYFA_RESP_NAME=MY_FA_Resp
MYFA_SHORTNAME=MYFA
CONTROL_LOG=$APPLPTMP/log/${EBS_SYSCODE}.log
SECU_HOME=$HOME/ebsbatch # <- folder holds file orauser.pwd in which has apps password.
# That means who has permission to run this script will know apps password
# So, in production, put it a scheduler server with high security
date >> $CONTROL_LOG
print "CONCSUB $CONC_LOGUSR $MYFA_SHORTNAME $MYFA_RESP_NAME $CONC_APPUSR WAIT=Y CONCURRENT $MYFA_SHORTNAME FAB2RPT1" >> $CONTROL_LOG
## CONCSUB apps MYFA MY_FA_Resp EBSBATCH WAIT=Y CONCURRENT MYFA FAB2RPT1
submission=$(CONCSUB $CONC_LOGUSR $MYFA_SHORTNAME \
${MYFA_RESP_NAME} $CONC_APPUSR WAIT=Y CONCURRENT \
$MYFA_SHORTNAME FABLD2RPT1 "BEFORE" < $SECU_HOME/orauser.pwd)
if (( $? != 0 ))
then
print "Customization : CONCSUB invoke failed" >> $CONTROL_LOG
exit 99
fi
print "Customization: Invoking CM for invoice lines from AP/PO report" >> $CONTROL_LOG
#-------------------------------------------------------------------------
# Parse out request number from concsub output. Password prompt
# may be included depending on invoke style, so trim up to key words.
#-------------------------------------------------------------------------
reqno=$( echo ${submission#*Submitted request}|awk {'print $1'} )
#-------------------------------------------------------------------------
# Query Concurrent Mgr table selecting status code for this reqno
#-------------------------------------------------------------------------
MY_MASTER_PWD=apps/appsPWD@EBSQA #Or, use Oracle Wallet to hide the password
completion_code=$(sqlplus -s $MY_MASTER_PWD <<-EOF
set heading off
set verify off
select fcr.status_code
from applsys.fnd_concurrent_requests fcr
where fcr.request_id = $reqno
/
exit
EOF)
# Trim off white space that sometimes occurs
completion_code=$( echo $completion_code|awk {'print $1'} )
#-------------------------------------------------------------------------
# Exit based on completion code
#-------------------------------------------------------------------------
if [[ $completion_code = C ]]
then
print "Concurrent Manager job "$reqno" completed succesfully" >> $CONTROL_LOG
exit 0
else
print "Concurrent Manager job #"$reqno" did not complete succesfully" >> $CONTROL_LOG
cat $APPLCSF/$APPLLOG/l${reqno}.req $APPLCSF/$APPLOUT/o${reqno}.out >> $CONTROL_LOG
exit 1
fi
exit 0
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
$ hostname
hostname1q
$ echo $APPL_TOP
/u01/app/EBSQA/apps/apps_st/appl
$ echo $ORACLE_HOME
/u01/app/EBSQA/apps/tech_st/10.1.2
$ echo $TWO_TASK
EBSQA
1. create user batchuser on same host hostname1q
2. install Oracle client on the same host but under a different location
$ whoami
batchuser
$ echo $ORACLE_HOME
/batchu01/app/product/11.2.0/EBIZ
$ echo ORACLE_SID
EBSQA
3. create a file to setup the environment variables
$ whoami
batchuser
$ more setenv.ebsqa
export ORACLE_HOME=/batchu01/app/product/11.2.0/EBIZ
export ORACLE_SID=EBSQA
export TNS_ADMIN=/batchu01/app/product/11.2.0/EBIZ/network/admin
export LD_LIBRARY_PATH=/batchu01/app/product/11.2.0/EBIZ/lib
export PATH=/batchu01/app/product/11.2.0/EBIZ/bin:/usr/bin:/usr/local/bin:/etc
. /u01/app/EBSQA/apps/apps_st/appl/EBSQA_hostname1q.env
export PATH=$PATH:/batchu01/app/product/11.2.0/EBIZ/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH
echo `which CONCSUB`
echo `which sqlldr`
$ . ./setenv.ebsqa
/u01/app/EBSQA/apps/apps_st/appl/fnd/12.0.0/bin/CONCSUB
/batchu01/app/product/11.2.0/EBIZ/bin/sqlldr
4. now, submit a concurrent job by $FND_TOP/bin/CONCSUB
$ CONCSUB APPS/appsPWD SYSADMIN "System Administrator" EBSadmin WAIT=Y CONCURRENT FND FNDSCURS PROGRAM_NAME='"Active Users"'
Submitted request 4654627 for CONCURRENT FND FNDSCURS PROGRAM_NAME="Active Users"
Normal completion
$ more $APPLCSF/log/l4654627.req
+---------------------------------------------------------------------------+
Application Object Library: Version : 12.0.0
Copyright (c) 1979, 1999, Oracle Corporation. All rights reserved.
FNDSCURS module: Active Users
+---------------------------------------------------------------------------+
Current system time is 28-JUL-2016 14:08:59
.. ... ...
5. Troubleshooting: if you get below error, the library file may be wrong or bad. Copy the file from EBS installation folder /u01/app/EBSQA/apps/tech_st/10.1.2/lib to current library folder /batchu01/app/product/11.2.0/EBIZ/lib.
CONCSUB: error while loading shared libraries: libclntsh.so.10.1: wrong ELF class: ELFCLASS64
CONCSUB: error while loading shared libraries: libnnz10.so: cannot open shared object file: No such file or directory
~~~~~~~~~~~~~~~~~~~~ A sample script ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
#!/bin/ksh
# Some variables
CONC_APPUSR=EBSBATCH # an EBS user who runs this cm job
CONC_LOGUSR=apps
MYFA_RESP_NAME=MY_FA_Resp
MYFA_SHORTNAME=MYFA
CONTROL_LOG=$APPLPTMP/log/${EBS_SYSCODE}.log
SECU_HOME=$HOME/ebsbatch # <- folder holds file orauser.pwd in which has apps password.
# That means who has permission to run this script will know apps password
# So, in production, put it a scheduler server with high security
date >> $CONTROL_LOG
print "CONCSUB $CONC_LOGUSR $MYFA_SHORTNAME $MYFA_RESP_NAME $CONC_APPUSR WAIT=Y CONCURRENT $MYFA_SHORTNAME FAB2RPT1" >> $CONTROL_LOG
## CONCSUB apps MYFA MY_FA_Resp EBSBATCH WAIT=Y CONCURRENT MYFA FAB2RPT1
submission=$(CONCSUB $CONC_LOGUSR $MYFA_SHORTNAME \
${MYFA_RESP_NAME} $CONC_APPUSR WAIT=Y CONCURRENT \
$MYFA_SHORTNAME FABLD2RPT1 "BEFORE" < $SECU_HOME/orauser.pwd)
if (( $? != 0 ))
then
print "Customization : CONCSUB invoke failed" >> $CONTROL_LOG
exit 99
fi
print "Customization: Invoking CM for invoice lines from AP/PO report" >> $CONTROL_LOG
#-------------------------------------------------------------------------
# Parse out request number from concsub output. Password prompt
# may be included depending on invoke style, so trim up to key words.
#-------------------------------------------------------------------------
reqno=$( echo ${submission#*Submitted request}|awk {'print $1'} )
#-------------------------------------------------------------------------
# Query Concurrent Mgr table selecting status code for this reqno
#-------------------------------------------------------------------------
MY_MASTER_PWD=apps/appsPWD@EBSQA #Or, use Oracle Wallet to hide the password
completion_code=$(sqlplus -s $MY_MASTER_PWD <<-EOF
set heading off
set verify off
select fcr.status_code
from applsys.fnd_concurrent_requests fcr
where fcr.request_id = $reqno
/
exit
EOF)
# Trim off white space that sometimes occurs
completion_code=$( echo $completion_code|awk {'print $1'} )
#-------------------------------------------------------------------------
# Exit based on completion code
#-------------------------------------------------------------------------
if [[ $completion_code = C ]]
then
print "Concurrent Manager job "$reqno" completed succesfully" >> $CONTROL_LOG
exit 0
else
print "Concurrent Manager job #"$reqno" did not complete succesfully" >> $CONTROL_LOG
cat $APPLCSF/$APPLLOG/l${reqno}.req $APPLCSF/$APPLOUT/o${reqno}.out >> $CONTROL_LOG
exit 1
fi
exit 0
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Thursday, August 4, 2016
Profile option FND_INIT_SQL may cause error
When switching Responsibility, forms gave error:
Oracle error -6550: ORA-06550 : line 1, column 7:
PLS-00306: wrong number or types of arguments in call to 'INIT'
ORA-06550 : line 1, column 7:
PL/SQL: Statement ignored
has been detected in fnd_global.initialize[fnd_init_sql].
I checked profile 'Initialization SQL Statement - Custom' as suggested by Doc ID 577753.1 and did see anything on Site level. But when I ran below code, I sew one at Application level (by ID 10002).
SQL> select PROFILE_OPTION_ID, LEVEL_ID, LEVEL_VALUE, PROFILE_OPTION_VALUE
from apps.fnd_profile_option_values
where profile_option_id = (
select profile_option_id from apps.fnd_profile_options
where profile_option_name = 'FND_INIT_SQL' );
PROFILE_OPTION_ID LEVEL_ID LEVEL_VALUE PROFILE_OPTION_VALUE
--------------------------- --------------- -------------------- --------------------------------
3157 10002 20004 begin my_global.init; end;
I ran a script in check R12 profile options to got the custom Application name. After I changed profile 'Initialization SQL Statement - Custom' on that Application level to null, which deletes that row from table apps.fnd_profile_option_values, the Oracle errors on forms were fixed.
Oracle error -6550: ORA-06550 : line 1, column 7:
PLS-00306: wrong number or types of arguments in call to 'INIT'
ORA-06550 : line 1, column 7:
PL/SQL: Statement ignored
has been detected in fnd_global.initialize[fnd_init_sql].
I checked profile 'Initialization SQL Statement - Custom' as suggested by Doc ID 577753.1 and did see anything on Site level. But when I ran below code, I sew one at Application level (by ID 10002).
SQL> select PROFILE_OPTION_ID, LEVEL_ID, LEVEL_VALUE, PROFILE_OPTION_VALUE
from apps.fnd_profile_option_values
where profile_option_id = (
select profile_option_id from apps.fnd_profile_options
where profile_option_name = 'FND_INIT_SQL' );
PROFILE_OPTION_ID LEVEL_ID LEVEL_VALUE PROFILE_OPTION_VALUE
--------------------------- --------------- -------------------- --------------------------------
3157 10002 20004 begin my_global.init; end;
I ran a script in check R12 profile options to got the custom Application name. After I changed profile 'Initialization SQL Statement - Custom' on that Application level to null, which deletes that row from table apps.fnd_profile_option_values, the Oracle errors on forms were fixed.
Tuesday, August 2, 2016
Run FTP and submit concurrent job by SQL
This is a SQL script to FTP get a file from a remote Windows server to database server, and then to submit an EBS concurrent job which will read that file. The database version is Oracle 11.2.0.4.
/** pre setups:
SQL> select fnd_profile.value('WIN_UPS_FILE_INFO') from dual;
FND_PROFILE.VALUE('WIN_UPS_FILE_INFO')
--------------------------------------------------------------------------------
PUL/NT_USERID passWD WIN_HOSTNAME.domain.com
SQL> select * from all_directories where directory_name = 'CUSTOM_FTPIN';
OWNER DIRECTORY_NAME DIRECTORY_PATH
--------- ----------------------- -----------------------------
SYS CUSTOM_FTPIN /path/to/ftp/in
**/
SQL> set serveroutput on
SQL> DECLARE
-- l_utl_file_handler UTL_FILE.FILE_TYPE;
l_conn UTL_TCP.connection;
l_chr_file_line varchar2(200);
l_chr_user_id varchar2(20);
l_chr_password varchar2(20);
l_chr_directory_path varchar2(200);
l_chr_server_name varchar2(40);
l_chr_file_name varchar2(20) := 'WIN07-24-2016.csv';
-- l_chr_file_name varchar2(20) := '&1';
l_request_id Number;
BEGIN
l_chr_file_line := fnd_profile.value('WIN_UPS_FILE_INFO');
select substr(l_chr_file_line,1,12) into l_chr_user_id from dual;
select substr(l_chr_file_line,14,7) into l_chr_password from dual;
select substr(l_chr_file_line,22,30) into l_chr_server_name from dual;
l_conn := ftp.login(l_chr_server_name, '21',l_chr_user_id,l_chr_password);
ftp.ascii(p_conn => l_conn);
ftp.get(p_conn => l_conn,
p_from_file => l_chr_file_name,
p_to_dir => 'CUSTOM_FTPIN',
p_to_file => l_chr_file_name);
ftp.logout(l_conn);
utl_tcp.close_all_connections;
/** commend this out for testing FTP
l_request_id := FND_REQUEST.SUBMIT_REQUEST('MYAP','MYAPUFVW','','',FALSE,''||l_chr_file_name||'',
chr(0), '', '','','',
'', '', '', '', '', '', '', '', '', '',
'', '', '', '', '', '', '', '', '', '',
'', '', '', '', '', '', '', '', '', '',
'', '', '', '', '', '', '', '', '', '',
'', '', '', '', '', '', '', '', '', '',
'', '', '', '', '', '', '', '', '', '',
'', '', '', '', '', '', '', '', '', '',
'', '', '', '', '', '', '', '', '', '',
'', '', '', '', '', '', '', '', '');
**/
commit;
END;
/
SQL>
220 Microsoft FTP Service
331 Password required for PUL/NT_USERID.
230 User logged in.
200 Type set to A.
227 Entering Passive Mode (177,79,176,170,278,170).
125 Data connection already open; Transfer starting.
226 Transfer complete.
221 Goodbye.
PL/SQL procedure successfully completed.
SQL> exit
/** pre setups:
SQL> select fnd_profile.value('WIN_UPS_FILE_INFO') from dual;
FND_PROFILE.VALUE('WIN_UPS_FILE_INFO')
--------------------------------------------------------------------------------
PUL/NT_USERID passWD WIN_HOSTNAME.domain.com
SQL> select * from all_directories where directory_name = 'CUSTOM_FTPIN';
OWNER DIRECTORY_NAME DIRECTORY_PATH
--------- ----------------------- -----------------------------
SYS CUSTOM_FTPIN /path/to/ftp/in
**/
SQL> set serveroutput on
SQL> DECLARE
-- l_utl_file_handler UTL_FILE.FILE_TYPE;
l_conn UTL_TCP.connection;
l_chr_file_line varchar2(200);
l_chr_user_id varchar2(20);
l_chr_password varchar2(20);
l_chr_directory_path varchar2(200);
l_chr_server_name varchar2(40);
l_chr_file_name varchar2(20) := 'WIN07-24-2016.csv';
-- l_chr_file_name varchar2(20) := '&1';
l_request_id Number;
BEGIN
l_chr_file_line := fnd_profile.value('WIN_UPS_FILE_INFO');
select substr(l_chr_file_line,1,12) into l_chr_user_id from dual;
select substr(l_chr_file_line,14,7) into l_chr_password from dual;
select substr(l_chr_file_line,22,30) into l_chr_server_name from dual;
l_conn := ftp.login(l_chr_server_name, '21',l_chr_user_id,l_chr_password);
ftp.ascii(p_conn => l_conn);
ftp.get(p_conn => l_conn,
p_from_file => l_chr_file_name,
p_to_dir => 'CUSTOM_FTPIN',
p_to_file => l_chr_file_name);
ftp.logout(l_conn);
utl_tcp.close_all_connections;
/** commend this out for testing FTP
l_request_id := FND_REQUEST.SUBMIT_REQUEST('MYAP','MYAPUFVW','','',FALSE,''||l_chr_file_name||'',
chr(0), '', '','','',
'', '', '', '', '', '', '', '', '', '',
'', '', '', '', '', '', '', '', '', '',
'', '', '', '', '', '', '', '', '', '',
'', '', '', '', '', '', '', '', '', '',
'', '', '', '', '', '', '', '', '', '',
'', '', '', '', '', '', '', '', '', '',
'', '', '', '', '', '', '', '', '', '',
'', '', '', '', '', '', '', '', '', '',
'', '', '', '', '', '', '', '', '');
**/
commit;
END;
/
SQL>
220 Microsoft FTP Service
331 Password required for PUL/NT_USERID.
230 User logged in.
200 Type set to A.
227 Entering Passive Mode (177,79,176,170,278,170).
125 Data connection already open; Transfer starting.
226 Transfer complete.
221 Goodbye.
PL/SQL procedure successfully completed.
SQL> exit
Tuesday, July 26, 2016
Network error FRM-92102 and F5
A R12.1.3 EBS site has one concurrent server and two web/forms servers which are monitored by F5 for load balance. If I started Apache services on two web hosts, forms got network error very quick. But if I started services just on one host, this problem would not happen. The error is
FRM-92102: A network error has occurred. The Forms Client has attempted to reestablish its
connection to the server 5 times(s) without success. Please check the network connection and try again later.
The Java Console log showed " oracle.forms.net.ConnectionException: 500 " error.
I believed this was something in the network, not in the EBS apps config. I worked with F5 admin on this and they found Persistence profile in F5 setups was null for this site somehow. After they set the value to two hours, this problem does not happen again in this site.
I had similar problem in another R12 site before that, when forms got FRM-92102 errors after user opened forms for a few minutes. That problem was caused by a wrong timeout value for Persistence profile in F5.
connection to the server 5 times(s) without success. Please check the network connection and try again later.
The Java Console log showed " oracle.forms.net.ConnectionException: 500 " error.
I believed this was something in the network, not in the EBS apps config. I worked with F5 admin on this and they found Persistence profile in F5 setups was null for this site somehow. After they set the value to two hours, this problem does not happen again in this site.
I had similar problem in another R12 site before that, when forms got FRM-92102 errors after user opened forms for a few minutes. That problem was caused by a wrong timeout value for Persistence profile in F5.
Friday, July 1, 2016
JSP files for OAF pages
Our developers coded a custom OAF (Oracle Application Framework) apps. After code files were saved to folders under $JAVA_TOP/oracle/apps/custom/Rebate/webui and .xml files were loaded into the database, webpage https://sitename.domain.com/OA_HTML/OA.jsp?page=/oracle/apps/custom/Rebate/webui/RebatePG&language_code=US&.... got error:
Error Page You have encountered an unexpected error. Please contact the System Administrator for assistance
I enabled Profile option "FND: Diagnostics" to see more detailed exception.
Exception Details.
oracle.apps.fnd.framework.OAException: oracle.adf.mds.exception.MDSRuntimeException:
Unable to find component with absolute reference = /oracle/apps/aear/Rebate/webui/SearchPage, XML Path = null.
Please verify that the reference is valid and the definition of the component exists either on the File System or in the MDS Repository.
at oracle.apps.fnd.framework.OAException.wrapperException(OAException.java:912)
... ... ...
We verified file $JAVA_TOP/oracle/apps/custom/Rebate/webui/SearchPage.xml does exist on in the OS folder, and then we ran a short script to upload it to the database
APPS_USER_PARAM=$1
APPS_PASS_PARAM=$2
HOST_NAME_PARAM=$3
DB_SID_PARAM=$4
DB_PORT_PARAM=$5
LOG_CONTROL=$LOG_TOP/log/SearchPage`date '+%Y%m%d%H%M'`.log
... ... ....
java oracle.jrad.tools.xml.importer.XMLImporter $JAVA_TOP/oracle/apps/custom/Rebate/webui/SearchPage.xml \
-username $APPS_USER_PARAM -password $APPS_PASS_PARAM -rootdir . \
-dbconnection "(DESCRIPTION= (ADDRESS=(PROTOCOL=tcp)(HOST=$HOST_NAME_PARAM)(PORT=$DB_PORT_PARAM)) (CONNECT_DATA= (SID=$DB_SID_PARAM)))"
After the xml is imported successfully, below code should display its definition stored in MDS repository.
SQL> set serveroutput on
SQL> Begin
SQL> apps.jdr_utils.printdocument('/oracle/apps/custom/Rebate/webui/SearchPage');
SQL> End;
Now, after we click on the OAF page again, it works! The problem was fixed.
Additional notes:
1. all .java files in a folder can be compiled to .class file by javac. For example,
$ cd $JAVA_TOP/oracle/apps/custom/Rebate/webui
$ javac *.java
2. For JSP apps, Java .class file is not loaded into the database. All compiled JSP files are stored in $COMMON/_pages. In R12, the jsp files does not get compiled automatically. If OC4J doesn’t find the .class file in the _pages directory, it will just display a blank webpage and will not even attempt to compile the JSP. This is different from 11i. So, do not modify or delete files in folder $COMMON/_pages in R12 if you do not know how to compile JSP files.
3. The Perl script to compile JSP file is $FND_TOP/patch/115/bin/ojspCompile.pl. Run below line will clear cache and force compile all jsp pages
$ perl -x $FND_TOP/patch/115/bin/ojspCompile.pl --compile --flush
Good documents on compiling JSP files:
JSP Pages Hanging in R12 After Removing Cached Class Files in _pages (Doc ID 433386.1)
How to Enable Automatic Compilation of JSP pages in R12 Environment (Doc ID 458338.1)
UPDATES: How jsp file works in R12.2.10?
Error Page You have encountered an unexpected error. Please contact the System Administrator for assistance
I enabled Profile option "FND: Diagnostics" to see more detailed exception.
Exception Details.
oracle.apps.fnd.framework.OAException: oracle.adf.mds.exception.MDSRuntimeException:
Unable to find component with absolute reference = /oracle/apps/aear/Rebate/webui/SearchPage, XML Path = null.
Please verify that the reference is valid and the definition of the component exists either on the File System or in the MDS Repository.
at oracle.apps.fnd.framework.OAException.wrapperException(OAException.java:912)
... ... ...
We verified file $JAVA_TOP/oracle/apps/custom/Rebate/webui/SearchPage.xml does exist on in the OS folder, and then we ran a short script to upload it to the database
APPS_USER_PARAM=$1
APPS_PASS_PARAM=$2
HOST_NAME_PARAM=$3
DB_SID_PARAM=$4
DB_PORT_PARAM=$5
LOG_CONTROL=$LOG_TOP/log/SearchPage`date '+%Y%m%d%H%M'`.log
... ... ....
java oracle.jrad.tools.xml.importer.XMLImporter $JAVA_TOP/oracle/apps/custom/Rebate/webui/SearchPage.xml \
-username $APPS_USER_PARAM -password $APPS_PASS_PARAM -rootdir . \
-dbconnection "(DESCRIPTION= (ADDRESS=(PROTOCOL=tcp)(HOST=$HOST_NAME_PARAM)(PORT=$DB_PORT_PARAM)) (CONNECT_DATA= (SID=$DB_SID_PARAM)))"
After the xml is imported successfully, below code should display its definition stored in MDS repository.
SQL> set serveroutput on
SQL> Begin
SQL> apps.jdr_utils.printdocument('/oracle/apps/custom/Rebate/webui/SearchPage');
SQL> End;
Now, after we click on the OAF page again, it works! The problem was fixed.
Additional notes:
1. all .java files in a folder can be compiled to .class file by javac. For example,
$ cd $JAVA_TOP/oracle/apps/custom/Rebate/webui
$ javac *.java
2. For JSP apps, Java .class file is not loaded into the database. All compiled JSP files are stored in $COMMON/_pages. In R12, the jsp files does not get compiled automatically. If OC4J doesn’t find the .class file in the _pages directory, it will just display a blank webpage and will not even attempt to compile the JSP. This is different from 11i. So, do not modify or delete files in folder $COMMON/_pages in R12 if you do not know how to compile JSP files.
3. The Perl script to compile JSP file is $FND_TOP/patch/115/bin/ojspCompile.pl. Run below line will clear cache and force compile all jsp pages
$ perl -x $FND_TOP/patch/115/bin/ojspCompile.pl --compile --flush
Good documents on compiling JSP files:
JSP Pages Hanging in R12 After Removing Cached Class Files in _pages (Doc ID 433386.1)
How to Enable Automatic Compilation of JSP pages in R12 Environment (Doc ID 458338.1)
UPDATES: How jsp file works in R12.2.10?
For R12.2, JSP file is saved in $OA_HTML (same as $EBS_APPS_DEPLOYMENT_DIR/oacore/html). They are not uploaded to database. You can use same command to compile a single custom .jsp file, such as
$ cd $OA_HTML
$ more JDKtest.jsp
The JDK version is: <%= System.getProperty("java.version") %>
$ perl -x $FND_TOP/patch/115/bin/ojspCompile.pl --compile -s 'JDKtest.jsp' --flush
After compilation, file jdktest.class is saved in $OA_HTML/WEB-INF/classes/_pages. But it may give error on page https://[web node]:[port]/OA_HTML/JDKtest.jsp :
logfile set: $LOG_HOME/appl/rgf/ojsp/ojspc_error.log
starting...(compiling delta)
using 10i internal ojsp ver: 10.3.6.0
synchronizing dependency file:
loading deplist...7874
enumerating jsps...7875
updating dependency...7875
parsing jsp...7875
writing deplist...7875
initializing compilation:
files to compile...1
eliminating children...1 (-0)
searching uncompiled...1
translating and compiling:
searching untranslated...1
translating jsps...1/1 in 23s
compiling jsps...1/1 in 3s
Finished!
Requested resource or page is not allowed in this site
This feature is controlled by new profile options in R12.2. To resolve the issue, change "Security: Allowed Resources" to ALL (and may also change "Allow Unrestricted JSP Access [FND_SEC_ALLOW_JSP_UNRESTRICTED_ACCESS] to Yes).
Then, bounce all Apps services. Now page http://[web node]:[port]/OA_HTML/JDKtest.jsp shall say "The JDK version is: 1.7.0_xxx".
Then, bounce all Apps services. Now page http://[web node]:[port]/OA_HTML/JDKtest.jsp shall say "The JDK version is: 1.7.0_xxx".
Sunday, June 26, 2016
s_oacore_nprocs and s_forms_nprocs
An EBS R12.1.1 instance had a lot of database sessions kept in INACTIVE for a long time (15+ days)
and never got closed. So, the number of db sessions increased daily and easily
exceeded database parameter PROCESSES (which is set to 3000). I had to recycle
Apps services each two weeks to kill the idle DB sessions. Most idle sessions were from
"e::bes:oracle.apps.icx.security.session.created" by JDBC Thin Client. Modules AR and GL are used most in this
instance.
After the instance was upgraded to R12.1.3 and its database was upgraded to 12c, the idle session problem went away surprisingly. Before that, I tried below setting parameters and did NOT get much luck.
Two documents were used to tune CONTEXT variables:
- How To Prevent Inactive JDBC Connections In Oracle Applications 11i and R12 ( Doc ID 427759.1 )
- JVM: Guidelines to setup the Java Virtual Machine in Apps Ebusiness Suite 11i and R12 ( Doc ID 362851.1 )
The number of jvms (oc4j in R12) is configured by the autoconfig variables s_oacore_nprocs, s_forms_nprocs, s_disco_nprocs, and s_xmlsvcs_nprocs. They can be updated in the $CONTEXT_FILE (using the autoconfig editor from OAM).
The document suggests use 1 JVM per CPU for 100 active connected users to OACoreGroup. Use the script to determine "active users" for OACoreGroup :
REM
REM SQL to count number of Apps 11i users
REM Run as APPS user
REM
select 'Number of user sessions : ' || count( distinct session_id) How_many_user_sessions
from icx_sessions icx
where disabled_flag != 'Y' and PSEUDO_FLAG = 'N'
and (last_connect + decode(FND_PROFILE.VALUE('ICX_SESSION_TIMEOUT'), NULL,limit_time, 0,limit_time, FND_PROFILE.VALUE('ICX_SESSION_TIMEOUT')/60)/24) > sysdate and counter < limit_connects;
REM
REM END OF SQL
REM
I got "Number of user sessions : 372" in my instance. So, I set s_oacore_nprocs to 4 to handle the volume ( assuming that 372 is peak level), and set s_forms_nprocs to 4 as well.
s_oacore_nprocs will decide how the number of sub-folders under $LOG_HOME/ora/10.1.3/j2ee/oacore
I set Heap configuration (4 parameters forms_jvm_start_options, oacore_jvm_start_options, forms_jvm_stop_options, oacore_jvm_stop_options in $CONTEXT_FILE) to
-Xmx1024M -Xms512M -XX:MaxPermSize=256M -XX:NewRatio=2 -XX:+PrintGCTimeStamps
I also added the following parameter to the DBC file:
JDBC\:oracle.jdbc.maxCachedBufferSize=262144
Changed the DBC file settings for dbc file under $FND_SECURE directory:
FND_JDBC_BUFFER_DECAY_INTERVAL=120
FND_JDBC_BUFFER_MIN=0
FND_JDBC_BUFFER_MAX=0
FND_MAX_JDBC_CONNECTIONS=256
FND_JDBC_USABLE_CHECK=true
FND_JDBC_BUFFER_DECAY_SIZE=5
But all above changes did not help much :(
UPDATE in November 2016:
After we moved EBS R12.1.3 instance to a new Linux 6 server (from Linux 5), adstrtal.sh got error when starting services.
Executing service control script:
$ADMIN_SCRIPTS_HOME/adoacorectl.sh start
Timeout specified in context file: 100 second(s)
script returned:
****************************************************
ERROR : Timed out( 100000 ): Interrupted Exception
You are running adoacorectl.sh version 120.13
Starting OPMN managed OACORE OC4J instance ...
****************************************************
Executing service control script:$ADMIN_SCRIPTS_HOME/adformsctl.sh start
Timeout specified in context file: 100 second(s)
script returned:
****************************************************
ERROR : Timed out( 100000 ): Interrupted Exception
You are running adformsctl.sh version 120.16.12010000.3
Starting OPMN managed FORMS OC4J instance ...
But I had no problem in log onto the EBS site and launching Forms.
The error (or warning) could be fixed by increasing CONTEXT FILE parameters for "Timeout specified in context file: 100 second(s)". Since parameters s_oacore_nprocs=4 and s_forms_nprocs=4 did not help, I just re-set them back to 2 (and fnd_jdbc_usable_check to false) in $CONTEXT_FILE and ran adautocfg.sh. After that, the adstrtal.sh Timed-out error did not show up.
After the instance was upgraded to R12.1.3 and its database was upgraded to 12c, the idle session problem went away surprisingly. Before that, I tried below setting parameters and did NOT get much luck.
Two documents were used to tune CONTEXT variables:
- How To Prevent Inactive JDBC Connections In Oracle Applications 11i and R12 ( Doc ID 427759.1 )
- JVM: Guidelines to setup the Java Virtual Machine in Apps Ebusiness Suite 11i and R12 ( Doc ID 362851.1 )
The number of jvms (oc4j in R12) is configured by the autoconfig variables s_oacore_nprocs, s_forms_nprocs, s_disco_nprocs, and s_xmlsvcs_nprocs. They can be updated in the $CONTEXT_FILE (using the autoconfig editor from OAM).
The document suggests use 1 JVM per CPU for 100 active connected users to OACoreGroup. Use the script to determine "active users" for OACoreGroup :
REM
REM SQL to count number of Apps 11i users
REM Run as APPS user
REM
select 'Number of user sessions : ' || count( distinct session_id) How_many_user_sessions
from icx_sessions icx
where disabled_flag != 'Y' and PSEUDO_FLAG = 'N'
and (last_connect + decode(FND_PROFILE.VALUE('ICX_SESSION_TIMEOUT'), NULL,limit_time, 0,limit_time, FND_PROFILE.VALUE('ICX_SESSION_TIMEOUT')/60)/24) > sysdate and counter < limit_connects;
REM
REM END OF SQL
REM
I got "Number of user sessions : 372" in my instance. So, I set s_oacore_nprocs to 4 to handle the volume ( assuming that 372 is peak level), and set s_forms_nprocs to 4 as well.
s_oacore_nprocs will decide how the number of sub-folders under $LOG_HOME/ora/10.1.3/j2ee/oacore
I set Heap configuration (4 parameters forms_jvm_start_options, oacore_jvm_start_options, forms_jvm_stop_options, oacore_jvm_stop_options in $CONTEXT_FILE) to
-Xmx1024M -Xms512M -XX:MaxPermSize=256M -XX:NewRatio=2 -XX:+PrintGCTimeStamps
I also added the following parameter to the DBC file:
JDBC\:oracle.jdbc.maxCachedBufferSize=262144
Changed the DBC file settings for dbc file under $FND_SECURE directory:
FND_JDBC_BUFFER_DECAY_INTERVAL=120
FND_JDBC_BUFFER_MIN=0
FND_JDBC_BUFFER_MAX=0
FND_MAX_JDBC_CONNECTIONS=256
FND_JDBC_USABLE_CHECK=true
FND_JDBC_BUFFER_DECAY_SIZE=5
But all above changes did not help much :(
UPDATE in November 2016:
After we moved EBS R12.1.3 instance to a new Linux 6 server (from Linux 5), adstrtal.sh got error when starting services.
Executing service control script:
$ADMIN_SCRIPTS_HOME/adoacorectl.sh start
Timeout specified in context file: 100 second(s)
script returned:
****************************************************
ERROR : Timed out( 100000 ): Interrupted Exception
You are running adoacorectl.sh version 120.13
Starting OPMN managed OACORE OC4J instance ...
****************************************************
Executing service control script:$ADMIN_SCRIPTS_HOME/adformsctl.sh start
Timeout specified in context file: 100 second(s)
script returned:
****************************************************
ERROR : Timed out( 100000 ): Interrupted Exception
You are running adformsctl.sh version 120.16.12010000.3
Starting OPMN managed FORMS OC4J instance ...
But I had no problem in log onto the EBS site and launching Forms.
The error (or warning) could be fixed by increasing CONTEXT FILE parameters for "Timeout specified in context file: 100 second(s)". Since parameters s_oacore_nprocs=4 and s_forms_nprocs=4 did not help, I just re-set them back to 2 (and fnd_jdbc_usable_check to false) in $CONTEXT_FILE and ran adautocfg.sh. After that, the adstrtal.sh Timed-out error did not show up.
Wednesday, June 15, 2016
Output page cannot be displayed
Concurrent job completed successfully. But the Output can not be opened due to its size.
Error: Page Cannot Be Displayed, Cannot View Output of Large concurrent reports
I tried below actions in R12.1, which may help or may not help. Seems to me it also depends on the resources of client PC. VMware mostly has this trouble when the Output file size is huge.
1. Follow Doc ID 845841.1 (without applying patches)
a) Set "ICX: Session Timeout" profile option value (120 minutes)
Note: s_oc4j_sesstimeout shall match this value
b) Set "OC4J Session Timeout" in orion-web.xml
Note: two orion-web.xml files. One for OC4J, and one for OAFM (which may not be important to the time-out)
$ORA_CONFIG_HOME/10.1.3/j2ee/oacore/application-deployments/oacore/html/orion-web.xml
$ORA_CONFIG_HOME/10.1.3/j2ee/oafm/application-deployments/oafm/webservices/orion-web.xml
increase <session-timeout> from 30 to 120
c) Set Timeout in $ORA_CONFIG_HOME/10.1.3/Apache/Apache/conf/httpd.conf
Increase Timeout from 300 to 900 (seconds)
2. Doc ID 780081.1 Java Heap Errors when Running An FSG
Section: To Increase Heap Size of oacore process:
In file $ORA_CONFIG_HOME/10.1.3/opmn/conf/opmn.xml
<process-type id="oacore" module-id="OC4J" status="enabled" working-dir="$ORACLE_HOME/j2ee/home">
<module-data>
<category id="start-parameters">
<data id="java-options" value="-server -verbose:gc -Xmx1024M -Xms256M -XX:MaxPermSize=160M
oacore java-options meaning:
-Xmx1024M -- Specifies maximum heap memory.(1024 MB)
-Xms256M -- Initial heap memory.(256 MB)
3. Bounce the Concurrent Manager, Apache Server (and maybe database listener).
Error: Page Cannot Be Displayed, Cannot View Output of Large concurrent reports
I tried below actions in R12.1, which may help or may not help. Seems to me it also depends on the resources of client PC. VMware mostly has this trouble when the Output file size is huge.
1. Follow Doc ID 845841.1 (without applying patches)
a) Set "ICX: Session Timeout" profile option value (120 minutes)
Note: s_oc4j_sesstimeout shall match this value
b) Set "OC4J Session Timeout" in orion-web.xml
Note: two orion-web.xml files. One for OC4J, and one for OAFM (which may not be important to the time-out)
$ORA_CONFIG_HOME/10.1.3/j2ee/oacore/application-deployments/oacore/html/orion-web.xml
$ORA_CONFIG_HOME/10.1.3/j2ee/oafm/application-deployments/oafm/webservices/orion-web.xml
increase <session-timeout> from 30 to 120
c) Set Timeout in $ORA_CONFIG_HOME/10.1.3/Apache/Apache/conf/httpd.conf
Increase Timeout from 300 to 900 (seconds)
2. Doc ID 780081.1 Java Heap Errors when Running An FSG
Section: To Increase Heap Size of oacore process:
In file $ORA_CONFIG_HOME/10.1.3/opmn/conf/opmn.xml
<process-type id="oacore" module-id="OC4J" status="enabled" working-dir="$ORACLE_HOME/j2ee/home">
<module-data>
<category id="start-parameters">
<data id="java-options" value="-server -verbose:gc -Xmx1024M -Xms256M -XX:MaxPermSize=160M
oacore java-options meaning:
-Xmx1024M -- Specifies maximum heap memory.(1024 MB)
-Xms256M -- Initial heap memory.(256 MB)
3. Bounce the Concurrent Manager, Apache Server (and maybe database listener).
Wednesday, May 25, 2016
opatch and Oracle Home inventory
There are two inventory locations. One is the central inventory defined by file /etc/oraInst.loc (in Linux host). For each ORACLE HOME on the server, there is a local inventory at $ORACLE_HOME/inventory.
opatch lists the information first when applying a patch:
Oracle Home : /path/to/apps/tech_st/10.1.2
Oracle Home Inventory : $ORACLE_HOME/inventory
Central Inventory : /path/to/oraEbsInventory
from : /etc/oraInst.loc
Because the central inventory /path/to/oraEbsInventory may be shared by multiple ORACLE HOMEs on the server, it is more vulnerable to error or corruption. If opatch throws out some error, troubleshoot central inventory first.
If there is a problem with local inventory, it may turn out you have to re-install it. So, do not try to change anything without backing it up.
When I apply a small patch to 10.1.2 ORACLE HOME, it failed with below messages in log file $ORACLE_HOME/.patch_storage/8551790/xxx.log. "opatch lsinventory" also did not show the patch was applied.
Patch 8551790 has been applied successfully
... ... ...
verifying patch
verifying that patch ID is in the Oracle Home inventory
INVENTORY PROBLEM: Patch 8551790 is not present in Oracle Home inventory.
Verifying copy files.
Comparing "/path/to/8551790/files/procbuilder/lib/cus_procbuilder.mk" and "$ORACLE_HOME/procbuilder/lib/cus_procbuilder.mk"
... ... ...
OPATCH_JAVA_ERROR: Patch was not successfully applied.
Verification of the patch failed.
ERROR: OPatch failed as verification of the patch failed.
The real message is "INVENTORY PROBLEM ...". Doc ID 438067.1 (Updating the Inventory Is Failing During Patching) provides a solution to it.
Fix:
I renamed the existing Contents folder and created an empty folder Contents under oraEbsInventory. After I ran "opatch apply" again, the issue was fixed.
The same fix may address error "Inventory check failed: Patch ID is NOT registered in Oracle Home inventory." or "Verification of patch failed: Patch is not found in the Inventory." as well.
That Oracle document also says "opatch lsinventory -detail" accesses the inventory stored in text format in central inventory while "opatch lsinventory" gets inventory info from binary inventory (under oraEbsInventory/Contents ?).
opatch lists the information first when applying a patch:
Oracle Home : /path/to/apps/tech_st/10.1.2
Oracle Home Inventory : $ORACLE_HOME/inventory
Central Inventory : /path/to/oraEbsInventory
from : /etc/oraInst.loc
Because the central inventory /path/to/oraEbsInventory may be shared by multiple ORACLE HOMEs on the server, it is more vulnerable to error or corruption. If opatch throws out some error, troubleshoot central inventory first.
If there is a problem with local inventory, it may turn out you have to re-install it. So, do not try to change anything without backing it up.
When I apply a small patch to 10.1.2 ORACLE HOME, it failed with below messages in log file $ORACLE_HOME/.patch_storage/8551790/xxx.log. "opatch lsinventory" also did not show the patch was applied.
Patch 8551790 has been applied successfully
... ... ...
verifying patch
verifying that patch ID is in the Oracle Home inventory
INVENTORY PROBLEM: Patch 8551790 is not present in Oracle Home inventory.
Verifying copy files.
Comparing "/path/to/8551790/files/procbuilder/lib/cus_procbuilder.mk" and "$ORACLE_HOME/procbuilder/lib/cus_procbuilder.mk"
... ... ...
OPATCH_JAVA_ERROR: Patch was not successfully applied.
Verification of the patch failed.
ERROR: OPatch failed as verification of the patch failed.
The real message is "INVENTORY PROBLEM ...". Doc ID 438067.1 (Updating the Inventory Is Failing During Patching) provides a solution to it.
Fix:
I renamed the existing Contents folder and created an empty folder Contents under oraEbsInventory. After I ran "opatch apply" again, the issue was fixed.
The same fix may address error "Inventory check failed: Patch ID is NOT registered in Oracle Home inventory." or "Verification of patch failed: Patch is not found in the Inventory." as well.
That Oracle document also says "opatch lsinventory -detail" accesses the inventory stored in text format in central inventory while "opatch lsinventory" gets inventory info from binary inventory (under oraEbsInventory/Contents ?).
Sunday, May 22, 2016
R12.1 log locations and log collection
To troubleshoot a technology stack (Web issue or Forms error), Oracle Support asks a lot of logs, before they take a look on the service request (SR) or even they may not know what they are doing. Recently I followed Doc ID 1942889.1 (SRDC - Data Collection Request for EBS: IAS) to collect diagnostic data/logs.
1. follow Doc ID 2026081.1 to run 5 sql scripts to generate 5 .txt files and save them to /tmp.
2. run below line to zip all needed files
$ more zip1942889_1.sh
zip -r /tmp/$TWO_TASK'_'`uname -n`_`date +%m%d%y`_Techstack_info.zip \
/tmp/apps_version_info.txt \
/tmp/fndnodes_oss.txt \
/tmp/web_urls.txt \
/tmp/atg_ver_patches.txt \
/tmp/login_profiles.txt \
$OA_HTML/bin/appsweb* \
$CONTEXT_FILE \
$FND_SECURE/* \
$ORA_CONFIG_HOME/10.1.3/Apache/* \
$ORA_CONFIG_HOME/10.1.3/j2ee/* \
$ORA_CONFIG_HOME/10.1.3/javacache/* \
$ORA_CONFIG_HOME/10.1.3/opmn/*
3. run below line from Doc (Doc ID 601736.1 to get techstack Component Versions (Forms, Http Server, JDK, Framework)
$ cd $FND_TOP/patch/115/bin
$ $ADPERLPRG $FND_TOP/patch/115/bin/TXKScript.pl \
-script=$FND_TOP/patch/115/bin/txkInventory.pl -txktop=$APPLTMP \
-contextfile=$CONTEXT_FILE \
-appspass=appsPWD \
-outfile=/tmp/Report_App_Inventory.html
*** ALL THE FOLLOWING FILES ARE REQUIRED FOR RESOLVING RUNTIME ERRORS
*** STDOUT = /$LOG_HOME/appl/rgf/TXK/txkInventory_Thu_Jan_14_10_21_05_2016_stdout.log
Reportfile /tmp/Report_App_Inventory.html generated successfully.
Summary on R12.1 log locations. Note env variable $LOG_HOME and $INST_TOP/logs are the same.
-----------------------------------------------------------------------------
-- Startup/Shutdown services (by $ADMIN_SCRIPTS_HOME scripts) log files:
$LOG_HOME/appl/admin/log
----------------------------------------------------------------------------
-- Apache, OC4J and OPMN (see Doc ID 454178.1): runtime
$LOG_HOME/ora/10.1.3/Apache
$LOG_HOME/ora/10.1.3/opmn
$LOG_HOME/ora/10.1.3/j2ee (each sub-folder /forms and /oacore has files application.log and server.log)
$LOG_HOME/ora/10.1.3/opmn/opmn.log
$LOG_HOME/ora/10.1.3/opmn/oacore_default_group_X/oacorestd.err (X depends on s_oacore_nprocs)
$LOG_HOME/ora/10.1.3/opmn/default_group~oacore~default_group~X.log
$LOG_HOME/ora/10.1.3/j2ee/oacore/oacore_default_group_X/application.log
$LOG_HOME/ora/10.1.3/j2ee/forms/forms_default_group_X/application.log X depends on s_forms_nprocs
$APPLRGF/javacache.log
- user's DB connection log (and trace if enabled)
$LOG_HOME/ora/10.1.2/network
--------------------------------------------------------------------------
-- Concurrent program log & out
$APPLCSF/log
$APPLCSF/out
($INST_TOP/logs/appl/conc/log <== default)
SQL> SELECT logfile_name, outfile_name, outfile_node_name, last_update_date
FROM apps.FND_CONCURRENT_REQUESTS
WHERE REQUEST_ID = &requestID;
- temporary directories
$APPLPTMP
$APPLTMP <== temporary files for EBS forms.
$INST_TOP/logs/ora/10.1.2/forms ($FORMS_TRACE_DR) <== holds forms runtime files
- Forms Runtime Diagnostics (FRD) tracing AND logging files (Doc ID 438652.1)
----------------------------------------------------------------------------
-- Autoconfig log file:
--
Apps:
$INST_TOP/admin/log/MMDDHHMM/adconfig.log
($INST_TOP/admin/out/MMDDHHMM has run-time files)
DB tier:
$ORACLE_HOME/appsutil/log/$CONTEXT_NAME/<MMDDHHMM>/
----------------------------------------------------------------------------
-- adadmin, adpatch and adctrl
$APPL_TOP/admin/${TWO_TASK}/log
----------------------------------------------------------------------------
-- Patching related log files
i) $APPL_TOP/admin/${TWO_TASK}/log/ <== Application Tier -- adpatch log
ii) $ORACLE_HOME/.patch_storage <== Forms (Forms & Reports 10.1.2/Developer) patch
iii) $IAS_ORACLE_HOME/.patch_storage <== Apache (Web Server) patch
----------------------------------------------------------------------------
-- Clone logs:
Pre-clone log files in source instance
Apps: => $INST_TOP/admin/log/StageAppsTier_MMDDHHMM.log
DB: => $ORACLE_HOME/appsutil/log/$CONTEXT_NAME/StageDBTier_MMDDHHMM.log
Clone log files in target instance
Apps : => $INST_TOP/admin/log/ApplyAppsTier_<time>.log
DB: => $ORACLE_HOME/appsutil/log/$CONTEXT_NAME/ApplyDBTier_<time>.log
---------------------------------------------------------------------------
-- Output Post Processor (OPP) log from the Application
Getting OPP Log from the application itself
a. System Administrator > Concurrent > Manager > Administer
b. Search for 'Output Post Processor'
c. Click the 'Processes' button
d. Click the Manager Log button. This will open the 'OPP'
1. follow Doc ID 2026081.1 to run 5 sql scripts to generate 5 .txt files and save them to /tmp.
2. run below line to zip all needed files
$ more zip1942889_1.sh
zip -r /tmp/$TWO_TASK'_'`uname -n`_`date +%m%d%y`_Techstack_info.zip \
/tmp/apps_version_info.txt \
/tmp/fndnodes_oss.txt \
/tmp/web_urls.txt \
/tmp/atg_ver_patches.txt \
/tmp/login_profiles.txt \
$OA_HTML/bin/appsweb* \
$CONTEXT_FILE \
$FND_SECURE/* \
$ORA_CONFIG_HOME/10.1.3/Apache/* \
$ORA_CONFIG_HOME/10.1.3/j2ee/* \
$ORA_CONFIG_HOME/10.1.3/javacache/* \
$ORA_CONFIG_HOME/10.1.3/opmn/*
3. run below line from Doc (Doc ID 601736.1 to get techstack Component Versions (Forms, Http Server, JDK, Framework)
$ cd $FND_TOP/patch/115/bin
$ $ADPERLPRG $FND_TOP/patch/115/bin/TXKScript.pl \
-script=$FND_TOP/patch/115/bin/txkInventory.pl -txktop=$APPLTMP \
-contextfile=$CONTEXT_FILE \
-appspass=appsPWD \
-outfile=/tmp/Report_App_Inventory.html
*** ALL THE FOLLOWING FILES ARE REQUIRED FOR RESOLVING RUNTIME ERRORS
*** STDOUT = /$LOG_HOME/appl/rgf/TXK/txkInventory_Thu_Jan_14_10_21_05_2016_stdout.log
Reportfile /tmp/Report_App_Inventory.html generated successfully.
Summary on R12.1 log locations. Note env variable $LOG_HOME and $INST_TOP/logs are the same.
-----------------------------------------------------------------------------
-- Startup/Shutdown services (by $ADMIN_SCRIPTS_HOME scripts) log files:
$LOG_HOME/appl/admin/log
----------------------------------------------------------------------------
-- Apache, OC4J and OPMN (see Doc ID 454178.1): runtime
$LOG_HOME/ora/10.1.3/Apache
$LOG_HOME/ora/10.1.3/opmn
$LOG_HOME/ora/10.1.3/j2ee (each sub-folder /forms and /oacore has files application.log and server.log)
$LOG_HOME/ora/10.1.3/opmn/opmn.log
$LOG_HOME/ora/10.1.3/opmn/oacore_default_group_X/oacorestd.err (X depends on s_oacore_nprocs)
$LOG_HOME/ora/10.1.3/opmn/default_group~oacore~default_group~X.log
$LOG_HOME/ora/10.1.3/j2ee/oacore/oacore_default_group_X/application.log
$LOG_HOME/ora/10.1.3/j2ee/forms/forms_default_group_X/application.log X depends on s_forms_nprocs
$APPLRGF/javacache.log
- user's DB connection log (and trace if enabled)
$LOG_HOME/ora/10.1.2/network
--------------------------------------------------------------------------
-- Concurrent program log & out
$APPLCSF/log
$APPLCSF/out
($INST_TOP/logs/appl/conc/log <== default)
SQL> SELECT logfile_name, outfile_name, outfile_node_name, last_update_date
FROM apps.FND_CONCURRENT_REQUESTS
WHERE REQUEST_ID = &requestID;
- temporary directories
$APPLPTMP
$APPLTMP <== temporary files for EBS forms.
$INST_TOP/logs/ora/10.1.2/forms ($FORMS_TRACE_DR) <== holds forms runtime files
- Forms Runtime Diagnostics (FRD) tracing AND logging files (Doc ID 438652.1)
----------------------------------------------------------------------------
-- Autoconfig log file:
--
Apps:
$INST_TOP/admin/log/MMDDHHMM/adconfig.log
($INST_TOP/admin/out/MMDDHHMM has run-time files)
DB tier:
$ORACLE_HOME/appsutil/log/$CONTEXT_NAME/<MMDDHHMM>/
----------------------------------------------------------------------------
-- adadmin, adpatch and adctrl
$APPL_TOP/admin/${TWO_TASK}/log
----------------------------------------------------------------------------
-- Patching related log files
i) $APPL_TOP/admin/${TWO_TASK}/log/ <== Application Tier -- adpatch log
ii) $ORACLE_HOME/.patch_storage <== Forms (Forms & Reports 10.1.2/Developer) patch
iii) $IAS_ORACLE_HOME/.patch_storage <== Apache (Web Server) patch
----------------------------------------------------------------------------
-- Clone logs:
Pre-clone log files in source instance
Apps: => $INST_TOP/admin/log/StageAppsTier_MMDDHHMM.log
DB: => $ORACLE_HOME/appsutil/log/$CONTEXT_NAME/StageDBTier_MMDDHHMM.log
Clone log files in target instance
Apps : => $INST_TOP/admin/log/ApplyAppsTier_<time>.log
DB: => $ORACLE_HOME/appsutil/log/$CONTEXT_NAME/ApplyDBTier_<time>.log
---------------------------------------------------------------------------
-- Output Post Processor (OPP) log from the Application
Getting OPP Log from the application itself
a. System Administrator > Concurrent > Manager > Administer
b. Search for 'Output Post Processor'
c. Click the 'Processes' button
d. Click the Manager Log button. This will open the 'OPP'
Saturday, April 23, 2016
Manually deploy EAR file
After a patch (such as JE patch 21950151) is applied or adadmin generates new JAR (Java Archive) file, the apps startup script adformsctl.sh may deploy forms EAR file. You may see a Warning (with status 0) in log $LOG_HOME/appl/admin/log/adstrtal.log.
Starting OPMN managed FORMS OC4J instance ...
Calling txkChkFormsDeployment.pl to check whether latest FORMSAPP.EAR is deployed...
Program : $FND_TOP/patch/115/bin/txkChkFormsDeployment.pl started @ Mon Apr 18 1
6:58:03 2016
*** Log File = $LOG_HOME/appl/rgf/TXK/txkChkFormsDeployment_Mon_Apr_18_16_58_02_2016/txkChkFormsDeployment_Mon_Apr_18_16_58_02_2016.log
File "$IAS_ORACLE_HOME/j2ee/forms/applications/forms/formsweb/WEB-INF/lib/frmsrv.jar" exists. Proceeding to check the size...
=================================================
*** Latest formsapp.ear has NOT been deployed ***
Deploying the latest EAR file...
=================================================
Program : $FND_TOP/patch/115/bin/txkChkFormsDeployment.pl completed @ Mon Apr 18 16:59:04 2016
*****************************************
* W A R N I N G *
*****************************************
Error while executing the perl script txkChkFormsDeployment.pl
We have determined that you need to redeploy Forms (using txkChkFormsDeployment.pl).
But could not do it automatically for you, due to some issues.
For details refer the log files.
Follow Note: 397174.1 to redeploy Forms manually.
*****************************************
adformsctl.sh: exiting with status 0
I followed steps in Doc ID 397174.1 to get over this. Without this steps, some forms may not open up.
1) back up file $INST_TOP/ora/10.1.3/j2ee/forms/config/system-jazn-data.xml (for deploying the formsapp.ear file). Note this file's timestamp always gets updated.
2) when the oc4jadmin Password is unknown, search keyword oc4jadmin for pattern in file system-jazn-data.xml , and replace line
<description>OC4J Administrator</description>
<credentials>{903}9VrhYTuhd7DyBJf7J/4KwbSEIlETQJOO</credentials>
with
<description>OC4J Administrator</description>
<credentials>!welcome123</credentials>
Optional: after above change and bounce apps services, below line can verify the password is correct :
$ $AF_JRE_TOP/bin/java -jar $IAS_ORACLE_HOME/j2ee/home/admin_client.jar deployer:oc4j:opmn://hostname.domain.com:6530/forms oc4jadmin welcome123 -validateURI
URI deployer:oc4j:opmn://hostname.domain.com:6530/forms is valid and connected
-- where to get the port number
$ grep 6530 $ORA_CONFIG_HOME/10.1.3/opmn/conf/opmn.xml
<port local="6130" remote="6230" request="6530"/>
-- one more validation
$ cd $ORA_CONFIG_HOME/10.1.3/opmn/bin
$ ./opmnctl validate
opmnctl: opmn validation succeeded.
3) stop apps services, and run the Perl script on each Forms tier to deploy the EAR file
$ $FND_TOP/bin/txkrun.pl -script=CfgOC4JApp
*** ALL THE FOLLOWING FILES ARE REQUIRED FOR RESOLVING RUNTIME ERRORS
*** Log File = $LOG_HOME/appl/rgf/TXK/txkCfgOC4JApp_Mon_Apr_18_17_17_52_2016.log
Program : $FND_TOP/patch/115/bin/txkCfgOC4JApp.pl started @ Mon Apr 18 17:17:52 2016
*** Log File = $LOG_HOME/appl/rgf/TXK/txkCfgOC4JApp_Mon_Apr_18_17_17_52_2016.log
Enter Application name for re-deployment ? forms
Enter Oc4j Instance password for re-deployment ? welcome123
Run Autoconfig <Yes/No> ? No
*****************************************************
Required values for starting OC4J instance "forms":
====================================================
s_formsstatus = enabled
s_forms_nprocs = 1 (value should be greater than 0)
Existing values from the context file:
======================================
s_formsstatus = enabled
s_forms_nprocs = 2
----------------------------------------------
*** Values for context variables are VALID ***
----------------------------------------------
*****************************************************
Stopping all OPMN processes.
OPMN stopped.
OPMN started.
Deplolying Application : "forms" onto OC4J instance: "forms"
Application deployed successfully.
Stopping and starting OC4J instances.
Started OC4J instances.
Binding webApp : "forms" with webmodule : "formsweb" for OC4J instance: "forms"
Web application bound successfully.
Stopping OPMN.
OPMN stopped.
Program : $FND_TOP/patch/115/bin/txkCfgOC4JApp.pl completed @ Mon Apr 18 17:42:21 2016
End of $FND_TOP/patch/115/bin/txkCfgOC4JApp.pl : No Errors encountered
4) restore the original system-jazn-data.xml file.
5) run AutoConfig on each Forms server. (Without this step, the forms may not pop up after click due to failure in back-end authentication.)
$ADMIN_SCRIPTS_HOME/adautocfg.sh
Now, you shall see new timestamp on $IAS_ORACLE_HOME/j2ee/forms/applications/forms.ear file. And, adformsctl.sh will start forms successfully.
___________________________________________________________________
~~~~~~~ Additional notes (Real paths are replaced with env variables) ~~~~~~~
Manually deploying EAR file may not always work. Here is what I experienced when execution "$FND_TOP/bin/txkrun.pl -script=CfgOC4JApp" gets failure in instances where adstpall.sh (to stop apps) get Time out error from "adoafmctl.sh stop" and "adformsctl.sh stop".
1. I first saw error in startup log adstrtal.log in a situation when adstrtal.sh called the Perl script to deploy EAR file.
Executing service control script:
$ADMIN_SCRIPTS_HOME/adformsctl.sh start
Timeout specified in context file: 100 second(s)
script returned:
****************************************************
ERROR : Timed out( 100000 ): Interrupted Exception
You are running adformsctl.sh version 120.16.12010000.3
Starting OPMN managed FORMS OC4J instance ...
Calling txkChkFormsDeployment.pl to check whether latest FORMSAPP.EAR is deployed...
Program : $FND_TOP/patch/115/bin/txkChkFormsDeployment.pl started @ Wed Jul 14 15:47:39 2016
*** Log File = $LOG_HOME/appl/rgf/TXK/
txkChkFormsDeployment_Thu_Jul_14_15_47_39_2016/txkChkFormsDeployment_Thu_Jul_14_15_47_39_2016.log
File "$IAS_ORACLE_HOME/j2ee/forms/applications/forms/formsweb/WEB-INF/lib/frmsrv.jar" exists. Proceeding to check the size...
=================================================
*** Latest formsapp.ear has NOT been deployed ***
Deploying the latest EAR file...
=================================================
*** ALL THE FOLLOWING FILES ARE REQUIRED FOR RESOLVING RUNTIME ERRORS
*** Log File = $LOG_HOME/appl/rgf/TXK/
txkChkFormsDeployment_Thu_Jul_14_15_47_39_2016/txkChkFormsDeployment_Thu_Jul_14_15_47_39_2016.log
****************************************************
2. Because of the error, adopmnctl.sh may show OC4J: forms is DOWN. When I looked into further, I saw errors in the EAR deployment log:
$ more $LOG_HOME/appl/rgf/TXK/txkChkFormsDeployment_Thu_Jul_14_15_47_39_2016/txkChkFormsDeployment_Thu_Jul_14_15_47_39_2016.log
- - - - - - - - -
Application deployed successfully. <== this line (and above) show deployment went good
===========================
Inside manageFormsOc4j()...
===========================
*** OPERATION is stopproc ***
Execute SYSTEM command : $IAS_ORACLE_HOME/opmn/bin/opmnctl stopproc instancename=forms
opmnctl: stopping opmn managed processes...
opmn id=hostname.domin.com:6230
0 of 1 processes stopped.
--------------------------------------------------------------------------------
ias-component/process-type/process-set:
default_group/forms/default_group/
Error
--> Process (index=1,uid=2087060759,pid=14495)
time out while waiting for a managed process to stop
Log:
$LOG_HOME/ora/10.1.3/opmn/default_group~forms~default_group~1.log
===========================
Inside manageFormsOc4j()...
===========================
*** OPERATION is startproc ***
Execute SYSTEM command : $IAS_ORACLE_HOME/opmn/bin/opmnctl startproc instancename=forms
ERROR DESCRIPTION:
(*******FATAL ERROR*******
PROGRAM : $FND_TOP/patch/115/bin/txkChkFormsDeployment.pl($FND_TOP/bin/txkrun.pl)
TIME : Thu Jul 14 15:50:07 2016
FUNCTION: TXK::Process::run [ Level 3 ]
MESSAGES:
Command error: <rc> = 38400, <command> = $IAS_ORACLE_HOME/opmn/bin/opmnctl startproc instancename=forms
Command error: <rc> = 52224, <command> = $IAS_ORACLE_HOME/opmn/bin/opmnctl stopproc instancename=forms
Command error: <rc> = 38400, <command> = $IAS_ORACLE_HOME/opmn/bin/opmnctl stopproc instancename=forms
STACK TRACE
TXK::Error::abort('TXK::Error','HASH(0x8942208)') called at $AU_TOP/perl/TXK/Common.pm line 299
TXK::Common::doError('TXK::Process=HASH(0x89eec34)','Command error: <rc> = 38400, <command> = /path/to/EBSQA/...','undef')
called at $AU_TOP/perl/TXK/Common.pm line 314
... ... ... ...
3. Seems the errors are at the last stage of EAR deployment. When I shutdown apps service and started it again by adstrtal.sh, it did not try to deploy the EAR file again and just showed all in status 0. But, when I checked out the login and forms, I could not launch forms.
4. When I tried to deploy the EAR file manually by following the steps above, I got similar errors. Seems the error is from the last stage.
. . . . . . . . .
Stopping all OPMN processes.
OPMN stopped.
OPMN started.
Deplolying Application : "forms" onto OC4J instance: "forms"
Application deployed successfully.
Stopping and starting OC4J instances.
Errors encountered running $FND_TOP/patch/115/bin/txkCfgOC4JApp.pl
*******FATAL ERROR*******
PROGRAM : $FND_TOP/patch/115/bin/txkCfgOC4JApp.pl($FND_TOP/bin/txkrun.pl)
TIME : Wed Jul 6 17:11:06 2016
FUNCTION: TXK::Process::run [ Level 3 ]
MESSAGES:
Command error: <rc> = 52224, <command> =$IAS_ORACLE_HOME/opmn/bin/opmnctl stopproc instancename=forms
STACK TRACE
TXK::Error::abort('TXK::Error','HASH(0x91118f4)') called at $AU_TOP/perl/TXK/Common.pm line 299
TXK::Common::doError('TXK::Process=HASH(0x9a86c10)','Command error: <rc> = 52224, <command> = /path/to/EBSQA/a...','undef') called at $AU_TOP/perl/TXK/Common.pm line 314
. . . . . . . . .
5. I did try things around and made forms work before I ran Step 6 below.
6. The deployment errors match the issue in Doc ID 1399491.1, which recommends patch 12965674 as the fix. After I applied it to 10.1.3.5 ORACLE_HOME (for Apache), it not only fixed the errors in EAR deployment but also fixed below Time out error in adstpall.log during apps services shutdown !
You are running adstpall.sh version 120.10.12010000.4
Executing service control script:
$ADMIN_SCRIPTS_HOME/adoafmctl.sh stop
Timeout specified in context file: 100 second(s)
script returned:
****************************************************
ERROR : Timed out( 100000 ): Interrupted Exception
You are running adoafmctl.sh version 120.8
Stopping OPMN managed OAFM OC4J instance ...
****************************************************
Executing service control script:
$ADMIN_SCRIPTS_HOME/adformsctl.sh stop
Timeout specified in context file: 100 second(s)
script returned:
****************************************************
ERROR : Timed out( 100000 ): Interrupted Exception
You are running adformsctl.sh version 120.16.12010000.3
Stopping OPMN managed FORMS OC4J instance ...
NOTES: Oracle gives a document with a easier way to manually deploy R12 Ear file $ORACLE_HOME/forms/j2ee/formsapp.ear. See Doc ID 1961805.1
Starting OPMN managed FORMS OC4J instance ...
Calling txkChkFormsDeployment.pl to check whether latest FORMSAPP.EAR is deployed...
Program : $FND_TOP/patch/115/bin/txkChkFormsDeployment.pl started @ Mon Apr 18 1
6:58:03 2016
*** Log File = $LOG_HOME/appl/rgf/TXK/txkChkFormsDeployment_Mon_Apr_18_16_58_02_2016/txkChkFormsDeployment_Mon_Apr_18_16_58_02_2016.log
File "$IAS_ORACLE_HOME/j2ee/forms/applications/forms/formsweb/WEB-INF/lib/frmsrv.jar" exists. Proceeding to check the size...
=================================================
*** Latest formsapp.ear has NOT been deployed ***
Deploying the latest EAR file...
=================================================
Program : $FND_TOP/patch/115/bin/txkChkFormsDeployment.pl completed @ Mon Apr 18 16:59:04 2016
*****************************************
* W A R N I N G *
*****************************************
Error while executing the perl script txkChkFormsDeployment.pl
We have determined that you need to redeploy Forms (using txkChkFormsDeployment.pl).
But could not do it automatically for you, due to some issues.
For details refer the log files.
Follow Note: 397174.1 to redeploy Forms manually.
*****************************************
adformsctl.sh: exiting with status 0
I followed steps in Doc ID 397174.1 to get over this. Without this steps, some forms may not open up.
1) back up file $INST_TOP/ora/10.1.3/j2ee/forms/config/system-jazn-data.xml (for deploying the formsapp.ear file). Note this file's timestamp always gets updated.
2) when the oc4jadmin Password is unknown, search keyword oc4jadmin for pattern in file system-jazn-data.xml , and replace line
<description>OC4J Administrator</description>
<credentials>{903}9VrhYTuhd7DyBJf7J/4KwbSEIlETQJOO</credentials>
with
<description>OC4J Administrator</description>
<credentials>!welcome123</credentials>
Optional: after above change and bounce apps services, below line can verify the password is correct :
$ $AF_JRE_TOP/bin/java -jar $IAS_ORACLE_HOME/j2ee/home/admin_client.jar deployer:oc4j:opmn://hostname.domain.com:6530/forms oc4jadmin welcome123 -validateURI
URI deployer:oc4j:opmn://hostname.domain.com:6530/forms is valid and connected
-- where to get the port number
$ grep 6530 $ORA_CONFIG_HOME/10.1.3/opmn/conf/opmn.xml
<port local="6130" remote="6230" request="6530"/>
-- one more validation
$ cd $ORA_CONFIG_HOME/10.1.3/opmn/bin
$ ./opmnctl validate
opmnctl: opmn validation succeeded.
3) stop apps services, and run the Perl script on each Forms tier to deploy the EAR file
$ $FND_TOP/bin/txkrun.pl -script=CfgOC4JApp
*** ALL THE FOLLOWING FILES ARE REQUIRED FOR RESOLVING RUNTIME ERRORS
*** Log File = $LOG_HOME/appl/rgf/TXK/txkCfgOC4JApp_Mon_Apr_18_17_17_52_2016.log
Program : $FND_TOP/patch/115/bin/txkCfgOC4JApp.pl started @ Mon Apr 18 17:17:52 2016
*** Log File = $LOG_HOME/appl/rgf/TXK/txkCfgOC4JApp_Mon_Apr_18_17_17_52_2016.log
Enter Application name for re-deployment ? forms
Enter Oc4j Instance password for re-deployment ? welcome123
Run Autoconfig <Yes/No> ? No
*****************************************************
Required values for starting OC4J instance "forms":
====================================================
s_formsstatus = enabled
s_forms_nprocs = 1 (value should be greater than 0)
Existing values from the context file:
======================================
s_formsstatus = enabled
s_forms_nprocs = 2
----------------------------------------------
*** Values for context variables are VALID ***
----------------------------------------------
*****************************************************
Stopping all OPMN processes.
OPMN stopped.
OPMN started.
Deplolying Application : "forms" onto OC4J instance: "forms"
Application deployed successfully.
Stopping and starting OC4J instances.
Started OC4J instances.
Binding webApp : "forms" with webmodule : "formsweb" for OC4J instance: "forms"
Web application bound successfully.
Stopping OPMN.
OPMN stopped.
Program : $FND_TOP/patch/115/bin/txkCfgOC4JApp.pl completed @ Mon Apr 18 17:42:21 2016
End of $FND_TOP/patch/115/bin/txkCfgOC4JApp.pl : No Errors encountered
4) restore the original system-jazn-data.xml file.
5) run AutoConfig on each Forms server. (Without this step, the forms may not pop up after click due to failure in back-end authentication.)
$ADMIN_SCRIPTS_HOME/adautocfg.sh
Now, you shall see new timestamp on $IAS_ORACLE_HOME/j2ee/forms/applications/forms.ear file. And, adformsctl.sh will start forms successfully.
___________________________________________________________________
~~~~~~~ Additional notes (Real paths are replaced with env variables) ~~~~~~~
Manually deploying EAR file may not always work. Here is what I experienced when execution "$FND_TOP/bin/txkrun.pl -script=CfgOC4JApp" gets failure in instances where adstpall.sh (to stop apps) get Time out error from "adoafmctl.sh stop" and "adformsctl.sh stop".
1. I first saw error in startup log adstrtal.log in a situation when adstrtal.sh called the Perl script to deploy EAR file.
Executing service control script:
$ADMIN_SCRIPTS_HOME/adformsctl.sh start
Timeout specified in context file: 100 second(s)
script returned:
****************************************************
ERROR : Timed out( 100000 ): Interrupted Exception
You are running adformsctl.sh version 120.16.12010000.3
Starting OPMN managed FORMS OC4J instance ...
Calling txkChkFormsDeployment.pl to check whether latest FORMSAPP.EAR is deployed...
Program : $FND_TOP/patch/115/bin/txkChkFormsDeployment.pl started @ Wed Jul 14 15:47:39 2016
*** Log File = $LOG_HOME/appl/rgf/TXK/
txkChkFormsDeployment_Thu_Jul_14_15_47_39_2016/txkChkFormsDeployment_Thu_Jul_14_15_47_39_2016.log
File "$IAS_ORACLE_HOME/j2ee/forms/applications/forms/formsweb/WEB-INF/lib/frmsrv.jar" exists. Proceeding to check the size...
=================================================
*** Latest formsapp.ear has NOT been deployed ***
Deploying the latest EAR file...
=================================================
*** ALL THE FOLLOWING FILES ARE REQUIRED FOR RESOLVING RUNTIME ERRORS
*** Log File = $LOG_HOME/appl/rgf/TXK/
txkChkFormsDeployment_Thu_Jul_14_15_47_39_2016/txkChkFormsDeployment_Thu_Jul_14_15_47_39_2016.log
****************************************************
2. Because of the error, adopmnctl.sh may show OC4J: forms is DOWN. When I looked into further, I saw errors in the EAR deployment log:
$ more $LOG_HOME/appl/rgf/TXK/txkChkFormsDeployment_Thu_Jul_14_15_47_39_2016/txkChkFormsDeployment_Thu_Jul_14_15_47_39_2016.log
- - - - - - - - -
Application deployed successfully. <== this line (and above) show deployment went good
===========================
Inside manageFormsOc4j()...
===========================
*** OPERATION is stopproc ***
Execute SYSTEM command : $IAS_ORACLE_HOME/opmn/bin/opmnctl stopproc instancename=forms
opmnctl: stopping opmn managed processes...
opmn id=hostname.domin.com:6230
0 of 1 processes stopped.
--------------------------------------------------------------------------------
ias-component/process-type/process-set:
default_group/forms/default_group/
Error
--> Process (index=1,uid=2087060759,pid=14495)
time out while waiting for a managed process to stop
Log:
$LOG_HOME/ora/10.1.3/opmn/default_group~forms~default_group~1.log
===========================
Inside manageFormsOc4j()...
===========================
*** OPERATION is startproc ***
Execute SYSTEM command : $IAS_ORACLE_HOME/opmn/bin/opmnctl startproc instancename=forms
ERROR DESCRIPTION:
(*******FATAL ERROR*******
PROGRAM : $FND_TOP/patch/115/bin/txkChkFormsDeployment.pl($FND_TOP/bin/txkrun.pl)
TIME : Thu Jul 14 15:50:07 2016
FUNCTION: TXK::Process::run [ Level 3 ]
MESSAGES:
Command error: <rc> = 38400, <command> = $IAS_ORACLE_HOME/opmn/bin/opmnctl startproc instancename=forms
Command error: <rc> = 52224, <command> = $IAS_ORACLE_HOME/opmn/bin/opmnctl stopproc instancename=forms
Command error: <rc> = 38400, <command> = $IAS_ORACLE_HOME/opmn/bin/opmnctl stopproc instancename=forms
STACK TRACE
TXK::Error::abort('TXK::Error','HASH(0x8942208)') called at $AU_TOP/perl/TXK/Common.pm line 299
TXK::Common::doError('TXK::Process=HASH(0x89eec34)','Command error: <rc> = 38400, <command> = /path/to/EBSQA/...','undef')
called at $AU_TOP/perl/TXK/Common.pm line 314
... ... ... ...
3. Seems the errors are at the last stage of EAR deployment. When I shutdown apps service and started it again by adstrtal.sh, it did not try to deploy the EAR file again and just showed all in status 0. But, when I checked out the login and forms, I could not launch forms.
4. When I tried to deploy the EAR file manually by following the steps above, I got similar errors. Seems the error is from the last stage.
. . . . . . . . .
Stopping all OPMN processes.
OPMN stopped.
OPMN started.
Deplolying Application : "forms" onto OC4J instance: "forms"
Application deployed successfully.
Stopping and starting OC4J instances.
Errors encountered running $FND_TOP/patch/115/bin/txkCfgOC4JApp.pl
*******FATAL ERROR*******
PROGRAM : $FND_TOP/patch/115/bin/txkCfgOC4JApp.pl($FND_TOP/bin/txkrun.pl)
TIME : Wed Jul 6 17:11:06 2016
FUNCTION: TXK::Process::run [ Level 3 ]
MESSAGES:
Command error: <rc> = 52224, <command> =$IAS_ORACLE_HOME/opmn/bin/opmnctl stopproc instancename=forms
STACK TRACE
TXK::Error::abort('TXK::Error','HASH(0x91118f4)') called at $AU_TOP/perl/TXK/Common.pm line 299
TXK::Common::doError('TXK::Process=HASH(0x9a86c10)','Command error: <rc> = 52224, <command> = /path/to/EBSQA/a...','undef') called at $AU_TOP/perl/TXK/Common.pm line 314
. . . . . . . . .
5. I did try things around and made forms work before I ran Step 6 below.
6. The deployment errors match the issue in Doc ID 1399491.1, which recommends patch 12965674 as the fix. After I applied it to 10.1.3.5 ORACLE_HOME (for Apache), it not only fixed the errors in EAR deployment but also fixed below Time out error in adstpall.log during apps services shutdown !
You are running adstpall.sh version 120.10.12010000.4
Executing service control script:
$ADMIN_SCRIPTS_HOME/adoafmctl.sh stop
Timeout specified in context file: 100 second(s)
script returned:
****************************************************
ERROR : Timed out( 100000 ): Interrupted Exception
You are running adoafmctl.sh version 120.8
Stopping OPMN managed OAFM OC4J instance ...
****************************************************
Executing service control script:
$ADMIN_SCRIPTS_HOME/adformsctl.sh stop
Timeout specified in context file: 100 second(s)
script returned:
****************************************************
ERROR : Timed out( 100000 ): Interrupted Exception
You are running adformsctl.sh version 120.16.12010000.3
Stopping OPMN managed FORMS OC4J instance ...
NOTES: Oracle gives a document with a easier way to manually deploy R12 Ear file $ORACLE_HOME/forms/j2ee/formsapp.ear. See Doc ID 1961805.1
Subscribe to:
Posts (Atom)