"deleted" files still take disk space

We received alerts on disk space used by EBS is 97% full. When I use "du" to check the space, it shows only 60% of the space is used. But "df -h" shows it is 97% full. The problem is that some files deleted by EBS process will not really release the space back yet until the process is stopped. See https://access.redhat.com/solutions/2316

Below line will find if there are "deleted" files that still hold space, assuming the partition is /u02/app:

$ lsof | grep /u02/ | grep app | grep delete | more
rwrun     18759   applMgr   13u      REG              253,3 3140558848    3082366 $INST_TOP/temp/dat000418759 (deleted)
rwrun     18759   applMgr   15u      REG              253,3  745730048    3082368 $INST_TOP/temp/idx000518759 (deleted)
rwrun     18759   applMgr   17u      REG              253,3          0    3082375 $INST_TOP/temp/000718759 (deleted)

3140558848 is the size by the file. You may see the size is increasing if process 18759 keeps running. But you will not see the file by "ls" command.

$ ps -ef | grep 18759
$ORACLE_HOME/bin/rwrun mode=character ARRAYSIZE=5 P_CONC_REQUEST_ID=21334937 P_MAIL_INDICATOR='ALL' report=$APPL_TOP/aear/reports/US/XXXXRREG.rdf userid=APPS batch=yes destype=file desname=$APPLCSF/out/o21334937.out desformat=$FND_TOP/reports/PD pagesize=2050x66

We found concurrent report "XXXX Receipt Register" in request ID 21334937 ran for 15 hours. After it is cancelled, the "hidden" space is released back.

The same problem happens to /tmp file partition, when parameter forms_tmpdir points to /tmp.

$ grep forms_tmpdir $CONTEXT_FILE

         <forms_tmpdir oa_var="s_forms_tmpdir" osd="UNIX">/tmp</forms_tmpdir>

$ lsof | grep /tmp | grep delete   <== it takes longer

$ lsof /tmp | grep delete             <== it runs quick

frmweb      398              applMgr    86u      REG              253,5    5303826        365 /tmp/filer6nHW3.TMP (deleted)

frmweb      570              s044150   86u      REG              253,5   41307666        144 /tmp/file4FWO2C.TMP (deleted)

$ ps -ef | grep 570

applMgr     570 31251  0 Mar15 ?        00:07:39 frmweb server webfile=HTTP-0,0,1,default

It is better to change forms_tmpdir to $APPLTMP, $INST_TOP/temp or $NE_BASE/EBSapps/log. /tmp is for server logs and if it is full, the server may crash and go down.