FRM-40735 on some custom Forms & Forms trace

Some users (but not all users) can not open EBS forms by a pop-up error:

FRM-40735: ON-ERROR trigger raised unhandled exception ORA-06508.

Before that happened, we got alerts the diskspace for holding Forms temp file (defined by forms_tmpdir) was full briefly when "deleted" files keep staying in memory or somewhere. 

Log from Forms process are at $FMW_HOME/user_projects/domains/EBS_domain_${TWO_TASK}/servers/forms_server1/logs. But individual forms' error may not be written to it. The only way to get a direction on finding the root cause is to turn on FRD trace. See Oracle Doc ID 2796573.1 (EBS:FRD Trace in EBS 12.2).

1. Log into EBS > profile > system

 select "user" > give the username (who is going to reproduce the issue)

 Profile : ICX: Forms Launcher > click on find

 Set the profile for the user as https://hostname:[port_number]/forms/frmservlet?record=collect

 logout of EBS

2. Go to control panel > java > advanced

enable logging and show console

Now login to EBS with the username (who is going to reproduce the issue)

(in the Java Console search for "record=collect" to confirm it is being used)

Open forms > reproduce the issue

 

3. Log into EBS server as OS owner of the application (putty session), get the file from the trace path

 

$ echo $FORMS_TRACE_DIR

$ cd $FORMS_TRACE_DIR

$ ls -lrt *collect*

In my case, I see messages in one of "collect" files:

Opened file: $CUSTOM_TOP/12.0.0/forms/US/XXARXTWM.fmx

Error Message: FRM-40039: Cannot attach library ARXCOQIT while opening form ARXTWMAI.

ON-ERROR Trigger Fired:

Form: ARXTWMAI

State Delta:

ARXTWMAI, 21, Trigger, Entry, 758240456, ON-ERROR

ARXTWMAI, 22, Prog Unit, Entry, 758567456, /ARXTWMAI-6/P120_26_SEP_202102_21_28

Unhandled Exception ORA-06508

State Delta:

Error Message: FRM-40735: ON-ERROR trigger raised unhandled exception ORA-06508.

ARXTWMAI, 22, Trigger, Exit, 765768456, ON-ERROR

# 16 - ARXTWMAI:<null>.<null>.1648228229694425000

Nothing showed the true problem. But when I look around the folder, I see file $AU_TOP/resource/ARXCWWIN.plx had a new timestamp and 0 bytes. After I copied the same file from another node to replace it, all forms errors went away.

Very strange thing is why that file became 0 bytes! One possibility is when the folder for holding the temp forms file was full and Linux Admin removed some temp files in memory by file ID, some unexpected thing happened.