How to capture a HEAP dump for oacore process in EBS Weblogic

When users could not log onto EBS site (with error code 502 for URL /OA_JAVA/oracle/apps/fnd/jar/fndewt.jar), we saw oacore processes used very high CPU and WLS Admin site show oacore1 status is unknown (blank). We had to shut oacore1 down and then start it up in WLS Admin Console.

After checking all EBS logs, Oracle Support did not find anything that might cause the problem and so asked us to capture a HEAP dump when the issue occurs. 

Use the following steps to customize the managed server configuration (of each OACORE) via the WebLogic Server Administration Console.

1. Log on to the WebLogic Server Administration Console.

2. Click on the 'Servers' link. This link takes you to a page containing a summary of the WebLogic Administration Server and all managed servers.

3. Click on the managed server whose configuration needs to be updated. A page containing various tabs for the settings of the managed server appears (e.g. oacore_server1).

4. Navigate to Configuration Tab > Server Start Tab > You can see the Arguments Section.

5. Click on Lock and Edit button in the 'Change Center' panel.

6. Update the Heap setting to append

-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/u02/app/temp/HEAP_trace

(add above line to the end of the long string with a single space in between. No service bounce is needed. /u02/app/temp/HEAP_trace can be changed to a different location.)

7. Click the 'Save' button to save the configuration changes.

8. Once the customizations are complete and saved, click the 'Activate Changes' button in the 'Change Center' panel to activate the changes.

9. Next time Out of Memory error occurs (or service bounce), a *.hprof dump file will be generated in /u02/app/temp/HEAP_trace.  Compressed it by gzip and upload it to Oracle SR.

$ grep hprof $EBS_DOMAIN_HOME/servers/oacore_server1/logs/oacore_server1.out00033

Dumping heap to /u02/app/temp/HEAP_trace/java_pid21982.hprof

More details in the log file:

java.lang.OutOfMemoryError: Java heap space

Dumping heap to /ifsu02/app/temp/HEAP_trace/java_pid21982.hprof ...

Heap dump file created [5475772387 bytes in 33.025 secs]

java.lang.OutOfMemoryError: Java heap space

        at org.apache.xmlbeans.impl.store.Locale.weakCur(Locale.java:2711)

        ... ...