Trick on Restoring Cold Backup from RMAN to ASM

User made mistakes by deleteing objects owned by SYS on 2/20/08 afternoon, so I tried to restore the database from RMAN cold backup of 2/19/08 22:03.

After the restore, I got "ORA-01113: file 1 needs media recovery" error when using "alter database open;" to open database. Then, I ran followings to open the database:

SQL> recover database;
Media recovery complete.
SQL> alter database open;
Database altered.

The above actions uses redo logs to recover the database. After the database was up, we found the mistakes that users made on 2/20/08 were still in the database. Even the alert log says all data files were restored I was not sure if the redo logs brought the mistakes in, or if the RMAN restore failed or not because there are no timestamps on files in ASM and there are no additional files created in ASM by the restore job.

I had to restore the database again using a backup on an earlier date, and used "recover database until cancel using backup controlfile;" to recover the database after reading Metalink Note 272321.1. Below are what I did to restore the database successfully in 2nd run.

-- Find the backup
RMAN> list backup of database completed between 'sysdate-9' and 'sysdate-5';

-- Preview
RMAN> run {
2> ALLOCATE CHANNEL ch00 TYPE 'SBT_TAPE';
3> ALLOCATE CHANNEL ch01 TYPE 'SBT_TAPE';
4> ALLOCATE CHANNEL ch02 TYPE 'SBT_TAPE';
5> ALLOCATE CHANNEL ch03 TYPE 'SBT_TAPE';
6> restore database from tag=TAG20080216T233131 preview;
7> RELEASE CHANNEL ch00;
8> RELEASE CHANNEL ch01;
9> RELEASE CHANNEL ch02;
10> RELEASE CHANNEL ch03;
11> }

allocated channel: ch00
channel ch00: sid=317 devtype=SBT_TAPE
channel ch00: VERITAS NetBackup for Oracle - Release 6.0 (2006110304)

allocated channel: ch01
channel ch01: sid=316 devtype=SBT_TAPE
channel ch01: VERITAS NetBackup for Oracle - Release 6.0 (2006110304)

allocated channel: ch02
channel ch02: sid=315 devtype=SBT_TAPE
channel ch02: VERITAS NetBackup for Oracle - Release 6.0 (2006110304)

allocated channel: ch03
channel ch03: sid=314 devtype=SBT_TAPE
channel ch03: VERITAS NetBackup for Oracle - Release 6.0 (2006110304)

Starting restore at 25-FEB-08

List of Backup Sets
===================

BS Key Type LV Size Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ ---------------
2239936 Incr 0 39.01G SBT_TAPE 00:31:57 16-FEB-08
BP Key: 2239943 Status: AVAILABLE Compressed: NO Tag: TAG20080216T233131
Handle: bk_cpj8t1uq_1_1_646875098 Media: PH0313
List of Datafiles in backup set 2239936
File LV Type Ckp SCN Ckp Time Name
---- -- ---- ---------- --------- ----
1 0 Incr 32529107158 16-FEB-08 +DATA/dbrep/datafile/system.270.630660291
5 0 Incr 32529107158 16-FEB-08 +DATA/dbrep/datafile/cis_data_1
7 0 Incr 32529107158 16-FEB-08 +DATA/dbrep/datafile/dev_user_1
13 0 Incr 32529107158 16-FEB-08 +DATA/dbrep/datafile/dss_data_3
14 0 Incr 32529107158 16-FEB-08 +DATA/dbrep/datafile/dss_idx_1

BS Key Type LV Size Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ ---------------
2239937 Incr 0 44.25G SBT_TAPE 00:33:54 16-FEB-08
BP Key: 2239944 Status: AVAILABLE Compressed: NO Tag: TAG20080216T233131
Handle: bk_cnj8t1uq_1_1_646875098 Media: X228L2
List of Datafiles in backup set 2239937
File LV Type Ckp SCN Ckp Time Name
---- -- ---- ---------- --------- ----
6 0 Incr 32529107158 16-FEB-08 +DATA/dbrep/datafile/cis_idx_1
10 0 Incr 32529107158 16-FEB-08 +DATA/dbrep/datafile/law_data_1
11 0 Incr 32529107158 16-FEB-08 +DATA/dbrep/datafile/dss_data_1
16 0 Incr 32529107158 16-FEB-08 +DATA/dbrep/datafile/dss_idx_3

BS Key Type LV Size Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ ---------------
2239938 Incr 0 43.63G SBT_TAPE 00:35:33 16-FEB-08
BP Key: 2239945 Status: AVAILABLE Compressed: NO Tag: TAG20080216T233131
Handle: bk_coj8t1uq_1_1_646875098 Media: EX0213
List of Datafiles in backup set 2239938
File LV Type Ckp SCN Ckp Time Name
---- -- ---- ---------- --------- ----
3 0 Incr 32529107158 16-FEB-08 +DATA/dbrep/datafile/sysaux.272.630660299
9 0 Incr 32529107158 16-FEB-08 +DATA/dbrep/datafile/pdw_idx_1
12 0 Incr 32529107158 16-FEB-08 +DATA/dbrep/datafile/dss_data_2
15 0 Incr 32529107158 16-FEB-08 +DATA/dbrep/datafile/dss_idx_2
18 0 Incr 32529107158 16-FEB-08 +DATA/dbrep/datafile/symantec_i3_orcl.dbf

BS Key Type LV Size Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ ---------------
2239939 Incr 0 45.18G SBT_TAPE 00:36:11 16-FEB-08
BP Key: 2239946 Status: AVAILABLE Compressed: NO Tag: TAG20080216T233131
Handle: bk_cmj8t1uq_1_1_646875098 Media: X441L2
List of Datafiles in backup set 2239939
File LV Type Ckp SCN Ckp Time Name
---- -- ---- ---------- --------- ----
2 0 Incr 32529107158 16-FEB-08 +DATA/dbrep/datafile/undotbs1.271.630660297
4 0 Incr 32529107158 16-FEB-08 +DATA/dbrep/datafile/users.274.630660303
8 0 Incr 32529107158 16-FEB-08 +DATA/dbrep/datafile/pdw_data_1
17 0 Incr 32529107158 16-FEB-08 +DATA/dbrep/datafile/dss_data_4
19 0 Incr 32529107158 16-FEB-08 +DATA/dbrep/datafile/dss_idx_4

archive logs generated after SCN 32529107158 not found in repository
Media recovery start SCN is 32529107158
Recovery must be done beyond SCN 32529107158 to clear data files fuzziness
Finished restore at 25-FEB-08

released channel: ch00
released channel: ch01
released channel: ch02
released channel: ch03

-- Restore
RMAN> run {
2> ALLOCATE CHANNEL ch00 TYPE 'SBT_TAPE';
3> ALLOCATE CHANNEL ch01 TYPE 'SBT_TAPE';
4> ALLOCATE CHANNEL ch02 TYPE 'SBT_TAPE';
5> ALLOCATE CHANNEL ch03 TYPE 'SBT_TAPE';
6> restore database from tag=TAG20080216T233131;
7> RELEASE CHANNEL ch00;
8> RELEASE CHANNEL ch01;
9> RELEASE CHANNEL ch02;
10> RELEASE CHANNEL ch03;
11> }

allocated channel: ch00
channel ch00: sid=317 devtype=SBT_TAPE
channel ch00: VERITAS NetBackup for Oracle - Release 6.0 (2006110304)

allocated channel: ch01
channel ch01: sid=316 devtype=SBT_TAPE
channel ch01: VERITAS NetBackup for Oracle - Release 6.0 (2006110304)

allocated channel: ch02
channel ch02: sid=315 devtype=SBT_TAPE
channel ch02: VERITAS NetBackup for Oracle - Release 6.0 (2006110304)

allocated channel: ch03
channel ch03: sid=314 devtype=SBT_TAPE
channel ch03: VERITAS NetBackup for Oracle - Release 6.0 (2006110304)

Starting restore at 25-FEB-08

channel ch00: starting datafile backupset restore
channel ch00: specifying datafile(s) to restore from backup set
restoring datafile 00001 to +DATA/dbrep/datafile/system.270.630660291
restoring datafile 00005 to +DATA/dbrep/datafile/cis_data_1
restoring datafile 00007 to +DATA/dbrep/datafile/dev_user_1
restoring datafile 00013 to +DATA/dbrep/datafile/dss_data_3
restoring datafile 00014 to +DATA/dbrep/datafile/dss_idx_1
channel ch00: reading from backup piece bk_cpj8t1uq_1_1_646875098
channel ch01: starting datafile backupset restore
channel ch01: specifying datafile(s) to restore from backup set
restoring datafile 00006 to +DATA/dbrep/datafile/cis_idx_1
restoring datafile 00010 to +DATA/dbrep/datafile/law_data_1
restoring datafile 00011 to +DATA/dbrep/datafile/dss_data_1
restoring datafile 00016 to +DATA/dbrep/datafile/dss_idx_3
channel ch01: reading from backup piece bk_cnj8t1uq_1_1_646875098
channel ch02: starting datafile backupset restore
channel ch02: specifying datafile(s) to restore from backup set
restoring datafile 00003 to +DATA/dbrep/datafile/sysaux.272.630660299
restoring datafile 00009 to +DATA/dbrep/datafile/pdw_idx_1
restoring datafile 00012 to +DATA/dbrep/datafile/dss_data_2
restoring datafile 00015 to +DATA/dbrep/datafile/dss_idx_2
restoring datafile 00018 to +DATA/dbrep/datafile/symantec_i3_orcl.dbf
channel ch02: reading from backup piece bk_coj8t1uq_1_1_646875098
channel ch03: starting datafile backupset restore
channel ch03: specifying datafile(s) to restore from backup set
restoring datafile 00002 to +DATA/dbrep/datafile/undotbs1.271.630660297
restoring datafile 00004 to +DATA/dbrep/datafile/users.274.630660303
restoring datafile 00008 to +DATA/dbrep/datafile/pdw_data_1
restoring datafile 00017 to +DATA/dbrep/datafile/dss_data_4
restoring datafile 00019 to +DATA/dbrep/datafile/dss_idx_4
channel ch03: reading from backup piece bk_cmj8t1uq_1_1_646875098
channel ch00: restored backup piece 1
piece handle=bk_cpj8t1uq_1_1_646875098 tag=TAG20080216T233131
channel ch00: restore complete, elapsed time: 00:21:57
channel ch01: restored backup piece 1
piece handle=bk_cnj8t1uq_1_1_646875098 tag=TAG20080216T233131
channel ch01: restore complete, elapsed time: 00:26:32
channel ch02: restored backup piece 1
piece handle=bk_coj8t1uq_1_1_646875098 tag=TAG20080216T233131
channel ch02: restore complete, elapsed time: 00:31:57
channel ch03: restored backup piece 1
piece handle=bk_cmj8t1uq_1_1_646875098 tag=TAG20080216T233131
channel ch03: restore complete, elapsed time: 00:32:22
Finished restore at 25-FEB-08

released channel: ch00
released channel: ch01
released channel: ch02
released channel: ch03
RMAN>

-- Bring the database up in SQL*Plus
SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-01113: file 1 needs media recovery
ORA-01110: data file 1: '+DATA/dbrep/datafile/system.270.630660291'

SQL> recover database until cancel using backup controlfile;
ORA-00279: change 32529107158 generated at 02/16/2008 23:30:33 needed for thread 1
ORA-00289: suggestion :
/u01/app/oracle/product/10.2.0/db_1/dbs/arch1_490_630660282.dbf
ORA-00280: change 32529107158 for thread 1 is in sequence #490

Specify log: {=suggested | filename | AUTO | CANCEL}
cancel
Media recovery cancelled.
SQL> alter database open resetlogs;

Database altered.

srvctl commands

1. Stop the database

- Shut down all services in the Oracle home on each node that might be accessing a database:
$ srvctl stop service -d pdb -s pdb_taf (stop pdb_taf only, not stop the instances)
$ srvctl stop service -d pdb (stop all services accessing pdb?, including pdb_taf)

In general:
$ srvctl stop service -d db_name [-s service_name_list [-i inst_name]]

- Shut down all Oracle RAC instances on all cluster nodes by entering the following command where db_name is the name of the database:
$ srvctl stop database -d pdb
(or $ srvctl stop db -d pdb )
$ srvctl stop database -d pdb -o immediate

In general:
$ srvctl stop database -d db_name

- Shut down Automatic Storage Management (ASM) instances on all cluster nodes by entering the following command (replace node_name with each node):

$ srvctl stop asm -n node_name

- To stop the listener running on a node, where the listener is running:

$ srvctl stop listener -n node_name [-l listenername]

- Stop all node applications (gsd, ons, listener, and vip) on all cluster nodes by entering the following command as the root user, where node_name is the name of the node where the applications are running:

# srvctl stop nodeapps -n node_name

- Shut down the Oracle Clusterware processes by entering the following command on all nodes as the root user:

# CRS_home/bin/crsctl stop crs

2. Start the database

- Start all instances (database)
$ srvctl start database -d db_name
- Start an instance
$ srvctl start instance -d db_name -i inst_name
- Start a services pdb_taf:
$ srvctl start service -d pdb -s pdb_taf
(If related instances are down, it will start all instances as well)

In general:
$ srvctl start service -d db_name -s service_name
$ srvctl start service -d pdb (start all services?)

- Start listener on each node of the cluster:
$ srvctl start listener -n node

3. A way to run RAC in a single instance:
- In instance PDB1 on PDB1P, issue
SQL> alter system cluster_database=false scope=spfile;

- Stop all services and instances:
pdb2p:$ srvctl stop service -d pdb
pdb2p:$ srvctl stop database -d pdb -o immediate

- Restart all instances and services:
pdb2p:$ srvctl start database -d pdb
pdb2p:$ srvctl start service -d pdb -s pdb_taf (only need this one?)

- Check the status:
pdb2p:$ srvctl status service -d pdb
Service pdb_taf is running on instance(s) pdb2
pdb2p:$ srvctl status db -d pdb
Instance pdb1 is not running on node pdb1p
Instance pdb2 is running on node pdb2p

- alter_pdb1.log shows the message:

Sun Feb 17 15:52:09 2008
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Interface type 1 en3 192.168.10.0 configured from OCR for use as a cluster interconnect
Interface type 1 en0 10.1.0.0 configured from OCR for use as a public interface
Picked latch-free SCN scheme 3
WARNING: db_recovery_file_dest is same as db_create_online_log_dest_2
Autotune of undo retention is turned on.
IMODE=BR
ILAT =121
LICENSE_MAX_USERS = 0
SYS auditing is disabled
ksdpec: called for event 13740 prior to event group initialization
Starting up ORACLE RDBMS Version: 10.2.0.3.0.
System parameters with non-default values:
processes = 1000
sessions = 1105
sga_max_size = 4294967296
__shared_pool_size = 570425344
__large_pool_size = 33554432
__java_pool_size = 16777216
__streams_pool_size = 0
spfile = +PDATA/pdb/spfilepdb.ora
sga_target = 4294967296
control_files = +PDATA/pdb/controlfile/current.384.636108915, +PFRA/pdb/controlfile/current.272.636108915
db_block_size = 8192
__db_cache_size = 3657433088
compatible = 10.2.0.3.0
log_archive_dest_1 = LOCATION=+PFRA
db_file_multiblock_read_count= 8
cluster_database = FALSE
cluster_database_instances= 1
db_create_file_dest = +PDATA
db_create_online_log_dest_1= +PDATA
db_create_online_log_dest_2= +PFRA
db_recovery_file_dest = +PFRA
db_recovery_file_dest_size= 605590388736
thread = 1
instance_number = 1
undo_management = AUTO
undo_tablespace = UNDOTBS1
remote_login_passwordfile= EXCLUSIVE
db_domain =
dispatchers = (PROTOCOL=TCP) (SERVICE=pdwaXDB)
remote_listener = LISTENERS_PDB
job_queue_processes = 10
parallel_max_servers = 40
parallel_execution_message_size= 16384
background_dump_dest = /u01/oracle/admin/pdb/bdump
user_dump_dest = /u01/oracle/admin/pdb/udump
core_dump_dest = /u01/oracle/admin/pdb/cdump
audit_file_dest = /u01/oracle/admin/pdb/adump
db_name = pdb
open_cursors = 300
pga_aggregate_target = 536870912
Cluster communication is configured to use the following interface(s) for this instance
192.168.10.101
Sun Feb 17 15:52:09 2008
cluster interconnect IPC version:Oracle UDP/IP (generic)
IPC Vendor 1 proto 2
PMON started with pid=2, OS id=1200584
DIAG started with pid=3, OS id=831956
PSP0 started with pid=4, OS id=1409044
LMON started with pid=5, OS id=1179846
LMD0 started with pid=6, OS id=1290462
MMAN started with pid=7, OS id=1241146
DBW0 started with pid=8, OS id=1405370
LGWR started with pid=9, OS id=729280
CKPT started with pid=10, OS id=880810
SMON started with pid=11, OS id=926172
RECO started with pid=12, OS id=684094
CJQ0 started with pid=13, OS id=589840
MMON started with pid=14, OS id=1417592
Sun Feb 17 15:52:10 2008
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
MMNL started with pid=15, OS id=917530
Sun Feb 17 15:52:10 2008
starting up 1 shared server(s) ...
Sun Feb 17 15:52:10 2008
lmon registered with NM - instance id 1 (internal mem no 0)
cluster_database_instances parameter mismatch!
Instance 1 = 2, mine = 1
This instance has not mounted yet. Exiting!
Sun Feb 17 15:52:11 2008
USER: terminating instance due to error 29707
Instance terminated by USER, pid = 1474652

Run RAC on one single node

Sometimes, we want to run RAC on a single node, instead. Here is what I did to make that happen.

-- Stop the TAF service
phdb2p:/localhome/oracle$srvctl stop service -d pdb
phdb2p:/localhome/oracle$srvctl status service -d pdb
Service pdb_taf is not running.

-- Stop the instances
phdb2p:/localhome/oracle$srvctl stop db -d pdb
phdb2p:/localhome/oracle$srvctl status db -d pdb
Instance pdb1 is not running on node phdb1p
Instance pdb2 is not running on node phdb2p

-- Make sure the neighor on the same box is not affected
phdb2p:/localhome/oracle$srvctl status service -d pdba
Service pdba_taf is running on instance(s) pdba2, pdba1

phdb2p:/localhome/oracle$srvctl status db -d pdba
Instance pdba1 is running on node phdb1p
Instance pdba2 is running on node phdb2p

phdb2p:/localhome/oracle$export ORACLE_SID=pdb2
phdb2p:/localhome/oracle$sqlplus /nolog

SQL*Plus: Release 10.2.0.3.0 - Production on Mon Feb 18 09:19:07 2008
Copyright (c) 1982, 2006, Oracle. All Rights Reserved.

SQL> conn / as sysdba
Connected to an idle instance.
SQL> startup mount;
ORACLE instance started.

Total System Global Area 4294967296 bytes
Fixed Size 2078368 bytes
Variable Size 671091040 bytes
Database Buffers 3607101440 bytes
Redo Buffers 14696448 bytes
Database mounted.

SQL> alter system set CLUSTER_DATABASE=false scope=memory;
alter system set CLUSTER_DATABASE=false scope=memory
*
ERROR at line 1:
ORA-02095: specified initialization parameter cannot be modified

SQL> create pfile='/u01/oracle/admin/pdwa/pfile/pdb2Init.ora' from spfile;

File created.

SQL> shutdown immediate;
ORA-01109: database not open

Database dismounted.
ORACLE instance shut down.

-- Edit the file by changing the line to "CLUSTER_DATABASE=false"
-- Then, start the instance. In this way, the instance will use the
-- original spfile for the next startup without any manual changes.
--
SQL> startup mount pfile=/u01/oracle/admin/pdb/pfile/pdb2Init.ora;
ORACLE instance started.

Total System Global Area 4294967296 bytes
Fixed Size 2078368 bytes
Variable Size 671091040 bytes
Database Buffers 3607101440 bytes
Redo Buffers 14696448 bytes
Database mounted.
SQL> alter database open;

Database altered.

SQL> archive log list;
Database log mode No Archive Mode
Automatic archival Disabled
Archive destination +PFRA
Oldest online log sequence 26801
Current log sequence 26802

SQL> select instance_name from v$instance;

INSTANCE_NAME
----------------
pdb2

SQL> select name from v$database;

NAME
---------
PDB

SQL>

-- After that, user needs to make sure the tnsnames.ora has an
-- entry for the instance pdb2:
PDB2 =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = phdb2p-vip)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = pdb)
(INSTANCE_NAME = pdb2)
)
)

CONTROL_FILE_RECORD_KEEP_TIME and RMAN

Our financial database has about 5 terabytes (TB) data. In the upgrade project, we maintein more than 5 instances (QA, Staging, Conversion, Development, Static, etc) running. That creates big challenge in backing up all instances.

During a period of days, the Veritas backup server ran very slow and took more than 24 hours to back up one instance. When we used one backup to refresh (restore) another instance, we kept geeting "missing files" or "expired files" from the backup set, such as:

MAN-00571: ==================================
RMAN-00569: == ERROR MESSAGE STACK FOLLOWS ==
RMAN-00571: =================================
RMAN-03002: failure of restore command at 01/24/2008 11:35:15
RMAN-06026: some targets not found - aborting restore
RMAN-06023: no backup or copy of datafile 1381 found to restore
RMAN-06023: no backup or copy of datafile 1382 found to restore

But, the Veritas log and command "RMAN> list backup of datafile 1381 ; " clearly indicate that the data file was included in a backup piece.

After I cheked the Verita log carefully, I found that some backupieces that were just created by the script were deleted at the end of same backup script.

The reason is that RMAN backup uses target control file instead of recovery catalog. The parameter "CONTROL_FILE_RECORD_KEEP_TIME = 1" in the init file makes RMAN delete all backuppieces that was backed up 24 hours (1 day) ago, even "RMAN retention policy is set to redundancy 3;" in the RMAN configuration.

Be careful when you use CONTROL_FILE_RECORD_KEEP_TIME!

Recovery area is full

Database was hung with error in alert log file of instance pdb1:

ORA-19815: WARNING: db_recovery_file_dest_size of 605590388736 bytes is 100.00% used, and has 0 remaining bytes available.

1. Re-mount the database pdb1

After I stopped the database, I had difficulties to mount it maybe because the ORA error, and so I had to work around to mount the database.

SQL> startup nomount;
ORACLE instance started.

Total System Global Area 4294967296 bytes
Fixed Size 2078328 bytes
Variable Size 687868296 bytes
Database Buffers 3590324224 bytes
Redo Buffers 14696448 bytes

-- Temporarily to increase the size to make room for mounting the db
SQL> alter system set DB_RECOVERY_FILE_DEST_SIZE = 700g scope=memory;
System altered.

SQL> alter database mount;
alter database mount
*
ERROR at line 1:
ORA-01105: mount is incompatible with mounts by other instances
ORA-19808: recovery destination parameter mismatch

--
-- Note: It's a RAC. I had to shut down the instance pdb2 on another
-- node before I can mount this one because the parameter change.
--

SQL> alter database mount;
Database altered.

2. Run RMAN to back up recovery area after connect to the catalog and the target pdb1 where the regular RMAN backup job runs. Note that the database has to be in mount for the below RMAN command to run:

run {
ALLOCATE CHANNEL ch00 TYPE 'SBT_TAPE';
ALLOCATE CHANNEL ch01 TYPE 'SBT_TAPE';
ALLOCATE CHANNEL ch02 TYPE 'SBT_TAPE';
ALLOCATE CHANNEL ch03 TYPE 'SBT_TAPE';
backup recovery area;
RELEASE CHANNEL ch00;
RELEASE CHANNEL ch01;
RELEASE CHANNEL ch02;
RELEASE CHANNEL ch03;
}

3. Check the database view to make sure that RMAN gets the space back as well:

SQL> SELECT substr(name, 1, 30) name, space_limit AS quota, space_used AS used, space_reclaimable AS reclaimable, number_of_files AS files
FROM v$recovery_file_dest;

NAME QUOTA USED RECLAIMABLE FILES
------ ---------- ---------- ----------- ----------
+FRA 7.5162E+11 6.0577E+11 6.0576E+11 88

4. Shut down the mounted instance:

$ srvctl status db -d pdb
Instance pdb1 is running on node db1s
Instance pdb2 is not running on node db2s

-- shut it down
SQL> shutdown immediate;

-- re-check
$ srvctl status db -d pdb
Instance pdb1 is not running on node db1s
Instance pdb2 is not running on node db2s

$ srvctl status service -d pdb
Service pdb_taf is not running.

5. Use srvctl to start the instances:

$ srvctl start service -d pdb -s "pdb_taf" -o open

$ srvctl status db -d pdb
Instance pdb1 is running on node db1s
Instance pdb2 is running on node db2s

$ srvctl status service -d pdb
Service pdb_taf is running on instance(s) pdb1, pdb2

6. Check the parameter in the database. It's far too big!

SQL> show parameter db_recovery_file

NAME TYPE VALUE
----------------------------- ----------- ------
db_recovery_file_dest string +FRA
db_recovery_file_dest_size big integer 564G

Now, after the instances started, the alert log says:

db_recovery_file_dest_size of 577536 MB is 0.00% used. This is a user-specified limit on the amount of space that will be used by this database for recovery-related files, and does not reflect the amount of space available in the underlying filesystem or ASM diskgroup.

Metalink note 315098.1 says that this action will not delete the files from the FRA. The files would be deleted whenever any space is required for any new operation. RMAN may just mark those archive logs as deletion cadidates as they were backed up to the tape. It is the safest way to fix the issue.