oacore OC4J logfile application.log

$LOG_HOME/ora/10.1.3/j2ee/oacore/oacore_default_group_1/application.log got 8GB in size due to 10.1.3.5.0 Container error entries (bug 10126440 ? ).

$ find . -type f -size +500000 -exec ls -alh {} \;
-rw-r----- 1 ebsuser users 8.1G April 17 13:55
./EBSP/inst/apps/EBSP_note1p/logs/ora/10.1.3/j2ee/oacore/oacore_default_group_1/application.log
I tried to rename application.log, but oacore process kept writing entries to the re-named file. I also deleted it and found oacore process did not create a new one while the space was still getting filled up and "find" command did not find which file it write the log entries to.

The fix is to re-name/delete application.log file and then bounce OACORE process which will create a new application.log file.

$ ps -ef | grep oacore| wc -l
4
(in my another instance, it gets 6)

$ cd $ADMIN_SCRIPTS_HOME

$ ./adoacorectl.sh stop

$ ./adopmnctl.sh status
You are running adopmnctl.sh version 120.6.12010000.5
Checking status of OPMN managed processes...
Processes in Instance: xxxx.com
---------------------------------+--------------------+---------+---------
ias-component                    | process-type       |     pid | status
---------------------------------+--------------------+---------+---------
OC4JGroup:default_group          | OC4J:oafm          |   20423 | Alive
OC4JGroup:default_group          | OC4J:forms         |   20354 | Alive
OC4JGroup:default_group          | OC4J:oacore       |     N/A | Down
HTTP_Server                                | HTTP_Server      |   20151 | Alive

adopmnctl.sh: exiting with status 0
adopmnctl.sh: check the logfile /u02/app/EBSPROD/inst/apps/<CONTEXT_NAME>/logs/appl/admin/log/adopmnctl.txt for more information ...

$ ps -ef | grep oacore| wc -l
3

$ cd $LOG_HOME/ora/10.1.3/j2ee/oacore/oacore_default_group_1

$ mv application.log application.log_Delete

$ cd $ADMIN_SCRIPTS_HOME

$ ./adoacorectl.sh start
You are running adoacorectl.sh version 120.13
Starting OPMN managed OACORE OC4J instance  ...
adoacorectl.sh: exiting with status 0
adoacorectl.sh: check the logfile $LOG_HOME/appl/admin/log/adoacorectl.txt for more information ...

$ ./adopmnctl.sh status
---------------------------------+--------------------+---------+---------
ias-component                    | process-type       |     pid | status
---------------------------------+--------------------+---------+---------
OC4JGroup:default_group          | OC4J:oafm          |   20423 | Alive
OC4JGroup:default_group          | OC4J:forms         |   20354 | Alive
OC4JGroup:default_group          | OC4J:oacore       |   23513 | Alive
HTTP_Server                              | HTTP_Server      |   20151 | Alive

adopmnctl.sh: exiting with status 0
adopmnctl.sh: $LOG_HOME/appl/admin/log/adopmnctl.txt for more information ...

$ ps -ef | grep oacore| wc -l
4

$ cd $LOG_HOME/ora/10.1.3/j2ee/oacore/oacore_default_group_1

$ more application.log
17/04/19 23:38:50.411 10.1.3.5.0 Started
17/04/19 23:38:51.979 html: 10.1.3.5.0 Started

Apache does not start due to bad group name

When I logged onto a server to clone an EBS instance from production, it gave a message

id: cannot find name for group ID 148.

"id" command shows group name is missing.
$ id
uid=1172(ebsdev) gid=148() groups=148(ebs)

I ignored the message and completed the clone. After that, Apache service did not start and even did not create any log files in $LOG_HOME/ora/10.1.3/Apache folder. The only error is in $LOG_HOME/ora/10.1.3/opmn/HTTP_Server~1.log :

apachectl startssl: execing httpd
httpd: bad group name 148

I realized where the problem came from and asked Linux Admin to fix the Group name issue.

$ id
uid=1172(ebsdev) gid=148(ebs) groups=148(ebs)

Then, I replaced 148 with the group name in file $ORA_CONFIG_HOME/10.1.3/Apache/Apache/conf/httpd.conf

User ebsdev
Group ebs

I also verified file /etc/group is READable. After that, I started Apache without problem.

Concurrent job out of memory issue

Create Accounting (XLAACCPB) Program in one of our instances gave below error in log file:

Calling XDO Data Engine...
****Warning!!! Due to high volume of data, got out of memory exception...***
****Please retry with scalable option or modify the Data template to run in scalable mode...***

Oracle Support recommended below action plan in a SR. That fixed the issue.

Action Plan:
 ==================
 1. In  Profile, please change following setting at ALL levels in System option:
 FND: Debug Log Enabled => NO
 AR: Enable Debug Message Output => NO
 SLA: Enable Diagnostics => No

2. Doc ID 1350719.1
 Access the "System Administrator" responsibility.

 Navigate to Concurrent > Program > Define.
 Search for the report (e.g., Create Accounting)
 Click on Parameters button.
 Move to the last parameter.

 Add the Scalable Flag as follows -

 Parameter- P_SCALABLE_FLAG
 Description - ScalableFlag
 Enabled - Check
 Value Set - XLA_SRS_NO_VALIDATION
 Default Type - Constant
 Default Value - Y
 Prompt - P_SCALABLE_FLAG
 Token - ScalableFlag (Case sensitive. Give this exact value)

 P_SCALABLE_FLAG - has a Default Value of 'Y'

3.
 A>  Increase the memory for Java used by the concurrent program.
Query the concurrent program in
 System Administrator> Concurrent> Program> Define >
 Query for the Create Accounting concurrent program >
 Enter value for Options field with : -Xmx2048M  (Note: Don't miss the dash before the X) .

 B> Time out parameters (Profile)
 Concurrent:OPP Response Timeout : 240
 Concurrent:OPP Process Timeout :   10800 sec (3 hours)

Additional Notes: Oracle Doc ID 838810.1 provides a similar solution to fix below error and it worked in my instance.

Subledger Accounting: Version : 12.0.0
XLAPEXRPT module: Subledger Period Close Exceptions Report
Calling XDO Data Engine...
****Warning!!! Due to high volume of data, got out of memory exception...***
****Please retry with scalable option or modify the Data template to run in scalable mode...***

Set the following parameters in the concurrent program definition:
Under Concurrent > Program > Define, query for concurrent program
short name = XLAPEXRPT and application = Subledger Accounting.

1. In the 'Options' field enter -Xmx2048M
2. Add a parameter to the concurrent token = ScalableFlag, default value set to 'Y'. Same way
you add DebugFlag
In the application developer responsibility
-> Concurrent -> Program -> Define
-> Query for short name= XLAPEXRPT
-> Click on Parameters
->define a new parameter (if not existing)
parameter name=P_DEBUG_FLAG, attach any valueset that is YES_NO valueset.
Give Default value=Y
Token=DebugFlag
Save
3. Submit the request with scalable mode Yes.

Shell scripts to back up and retore files for EBS refresh or clone

Some setup and configuration files works specifically for the EBS instance. It takes a long time or may be impossible to re-create them. It is are very important to back them up before the instance refresh or clone, and then restore the files after clone script completes. Below two scripts work for EBS R12.1.

~~~~~~~~~~~~~~~~~ Backup ~~~~~~~~~~~~~~~~~~

# backup files before the refresh    

cd $HOME/admin_scripts/backups  # where to save the backed-up files
DT=`date +"%h_%d_%y"`               # backup date

echo It will delete old backup files. Continue ? yes or no
read  answer

case $answer in
  [yY][eE][sS]|[yY])

    if [ -n "$CONTEXT_FILE" ]; then
      echo 'running'
    else
      echo 'environment variable $CONTEXT_FILE does not exit. Exit ...'
      exit 1;
    fi;

    if [ -f $CONTEXT_FILE ]; then
      echo 'starting copy ...'
    else
      echo 'CONTEXT file $CONTEXT_FILE does not exist. Exit ...'
      exit 1;
    fi

    file1=`basename $CONTEXT_FILE`
    if [ -f $file1 ]; then
      mv $file1 $file1'_'$DT      # adding the backup date
    fi
    file2=custom`basename $CONTEXT_FILE .xml`'.env'
    if [ -f $APPL_TOP/$file2 ]; then
      cp -p $APPL_TOP/$file2 .
    fi

    rm -rf Apache admin    # remove two previous backup folders 

    cp -p $CONTEXT_FILE .
    cp -rp $INST_TOP/certs/Apache .      # SSL certs
    cp -rp $TNS_ADMIN .                       # TNS names
    cp -p $FND_TOP/fndenv.env .
    cp -p $INST_TOP/ora/*2/forms/server/default.env .
    cp -p $APPL_TOP/admin/adkeystore.dat .
    cp -p $APPL_TOP/admin/adsign.txt .

    echo 'Done with backing up files'
    ls -altr $HOME/admin_scripts/backups
    exit 0

   ;;

  [nN][oO]|[nN])
      echo "No"
   ;;

   *)
      echo "Invalid input..."
      exit 1
   ;;
esac

~~~~~~~~~~~~~~~~~~~ Restore ~~~~~~~~~~~~~~~~~~~
#!/bin/ksh
# restore files AFTER clone script completes during the refresh  

cd $HOME/admin_scripts/backups
DT=`date +"%h_%d_%y"`

echo It will overwrite some files. Continue? yes or no
read  answer

case $answer in
   [yY][eE][sS]|[yY])

    if [ -n "$CONTEXT_FILE" ]; then
      echo 'running'
    else
      echo 'environment variable $CONTEXT_FILE does not exit. Exit ...'
      exit 1;
    fi;

    file2=custom`basename $CONTEXT_FILE .xml`'.env'
    if [ -f $file2 ]; then
      cp -p $file2 $APPL_TOP/.
    fi

    mv $FND_TOP/fndenv.env $FND_TOP/fndenv.env_clone
    cp -p fndenv.env $FND_TOP/.

    mv $INST_TOP/ora/10.1.2/forms/server/default.env $INST_TOP/ora/10.1.2/forms/server/default.env_clone
    cp -p default.env $INST_TOP/ora/10.1.2/forms/server/.

    mv $INST_TOP/certs/Apache  $INST_TOP/certs/Apache'_'$DT
    cp -pr Apache $INST_TOP/certs/.

    ## files for JRE 1.8. They will be used if Java signing is re-ran. 
    mv $APPL_TOP/admin/adkeystore.dat $APPL_TOP/admin/adkeystore.dat_clone
    cp -p adkeystore.dat $APPL_TOP/admin/.

    mv $APPL_TOP/admin/adsign.txt $APPL_TOP/admin/adsign.txt_clone
    cp -p adsign.txt $APPL_TOP/admin/.

    echo 'Done with restoring files.'

    ;;

  [nN][oO]|[nN])
    echo "No"
    ;;

  *)
    echo "Invalid input..."
    exit 1
    ;;
esac

Stop Old Outbound Workflow Notification Emails after Clone

I followed below document to cancel/purge R12.1.3 notifications:
How To Purge E-Mail Notifications From The Workflow Queue So The E-Mail Is Not Sent (Doc ID 372933.1)

It gives a query getting notifications that would be sent, and are waiting to be e-mailed when the Mailer get started:

SQL> select notification_id, begin_date, recipient_role, message_type, message_name, status, mail_status
 from wf_notifications
where Status In ('OPEN', 'CANCELED')
   and Mail_Status In ('MAIL', 'INVALID')
   and begin_date < sysdate-30              -- List only emails older than 30 days ago
order by notification_id;

SQL> select count(*) from wf_notifications
         where status in ('OPEN', 'CANCELED')
         And Mail_Status In ('MAIL', 'INVALID')
         order by notification_id;
  COUNT(*)
----------
    271022
SQL> select count(*) from wf_notifications
    where status in ('OPEN', 'CANCELED')
    and Mail_Status In ('MAIL', 'INVALID')
    and begin_date < sysdate-30;    -- List only emails older than 30 days ago
  COUNT(*)
----------
    270991
SQL> create table apps.wf_notifications_BK_120816 as
    select * from wf_notifications
    where status in ('OPEN', 'CANCELED')
    and Mail_Status In ('MAIL', 'INVALID')
    and begin_date < sysdate-30;
Table created.

SQL> update WF_NOTIFICATIONS set mail_status = 'SENT'
    where status in ('OPEN', 'CANCELED')
    and Mail_Status In ('MAIL', 'INVALID')
    and begin_date < sysdate-30;
270998 rows updated.
SQL> commit;
Commit complete.

SQL> @$FND_TOP/patch/115/sql/wfntfqup.sql apps apps_PWD applsys
PL/SQL procedure successfully completed.
Elapsed: 00:00:00.76
Commit complete.
Elapsed: 00:00:00.00
**** TEMPORARY TABLES / AQs created ****
PL/SQL procedure successfully completed.
Elapsed: 00:00:00.27
**** Messages backed up ****
Commit complete.
Elapsed: 00:00:00.00
**** wfaqback.sql completed,  executing SQLs from wfntfqup
PL/SQL procedure successfully completed.
Elapsed: 00:00:04.60
***  invoking wfaqrenq.sql ******
*** Re-enqueing messages
PL/SQL procedure successfully completed.
Elapsed: 00:00:00.02
Commit complete.
Elapsed: 00:00:00.01
***** Re-enqueue OF Alerts completed *****

I tried to use an email address to hold the old notifications after SMTP works fine on the CM server. Seems to me that the "Set Override Address" will not work until the Mailer was started, because prior to that, it would not send the Verification Code out to the email address.

References: Doc ID 562551.1 includes two old documents:

- How to Cancel Email Notifications for Particular Workflow Type (Doc ID 736508.1 for 11i)
- How To Stop Old Outbound Workflow Notification Email Messages During Clone Activity (Doc ID 828812.1 for 11.5.9 to 12.0.0)