Recovery area is full

Database was hung with error in alert log file of instance pdb1:

ORA-19815: WARNING: db_recovery_file_dest_size of 605590388736 bytes is 100.00% used, and has 0 remaining bytes available.

1. Re-mount the database pdb1

After I stopped the database, I had difficulties to mount it maybe because the ORA error, and so I had to work around to mount the database.

SQL> startup nomount;
ORACLE instance started.

Total System Global Area 4294967296 bytes
Fixed Size 2078328 bytes
Variable Size 687868296 bytes
Database Buffers 3590324224 bytes
Redo Buffers 14696448 bytes

-- Temporarily to increase the size to make room for mounting the db
SQL> alter system set DB_RECOVERY_FILE_DEST_SIZE = 700g scope=memory;
System altered.

SQL> alter database mount;
alter database mount
*
ERROR at line 1:
ORA-01105: mount is incompatible with mounts by other instances
ORA-19808: recovery destination parameter mismatch

--
-- Note: It's a RAC. I had to shut down the instance pdb2 on another
-- node before I can mount this one because the parameter change.
--

SQL> alter database mount;
Database altered.

2. Run RMAN to back up recovery area after connect to the catalog and the target pdb1 where the regular RMAN backup job runs. Note that the database has to be in mount for the below RMAN command to run:

run {
ALLOCATE CHANNEL ch00 TYPE 'SBT_TAPE';
ALLOCATE CHANNEL ch01 TYPE 'SBT_TAPE';
ALLOCATE CHANNEL ch02 TYPE 'SBT_TAPE';
ALLOCATE CHANNEL ch03 TYPE 'SBT_TAPE';
backup recovery area;
RELEASE CHANNEL ch00;
RELEASE CHANNEL ch01;
RELEASE CHANNEL ch02;
RELEASE CHANNEL ch03;
}

3. Check the database view to make sure that RMAN gets the space back as well:

SQL> SELECT substr(name, 1, 30) name, space_limit AS quota, space_used AS used, space_reclaimable AS reclaimable, number_of_files AS files
FROM v$recovery_file_dest;

NAME QUOTA USED RECLAIMABLE FILES
------ ---------- ---------- ----------- ----------
+FRA 7.5162E+11 6.0577E+11 6.0576E+11 88

4. Shut down the mounted instance:

$ srvctl status db -d pdb
Instance pdb1 is running on node db1s
Instance pdb2 is not running on node db2s

-- shut it down
SQL> shutdown immediate;

-- re-check
$ srvctl status db -d pdb
Instance pdb1 is not running on node db1s
Instance pdb2 is not running on node db2s

$ srvctl status service -d pdb
Service pdb_taf is not running.

5. Use srvctl to start the instances:

$ srvctl start service -d pdb -s "pdb_taf" -o open

$ srvctl status db -d pdb
Instance pdb1 is running on node db1s
Instance pdb2 is running on node db2s

$ srvctl status service -d pdb
Service pdb_taf is running on instance(s) pdb1, pdb2

6. Check the parameter in the database. It's far too big!

SQL> show parameter db_recovery_file

NAME TYPE VALUE
----------------------------- ----------- ------
db_recovery_file_dest string +FRA
db_recovery_file_dest_size big integer 564G

Now, after the instances started, the alert log says:

db_recovery_file_dest_size of 577536 MB is 0.00% used. This is a user-specified limit on the amount of space that will be used by this database for recovery-related files, and does not reflect the amount of space available in the underlying filesystem or ASM diskgroup.

Metalink note 315098.1 says that this action will not delete the files from the FRA. The files would be deleted whenever any space is required for any new operation. RMAN may just mark those archive logs as deletion cadidates as they were backed up to the tape. It is the safest way to fix the issue.