Place a blackout in Grid Control

There are two ways to place a blackout in Grid Control.
- From the Grid Control UI (Console), using Setup to create a blackout or using the icon on each Target page.
- From the command line at the agent home.

Here are the commands for creating a blackout:

- Place a blackout at host level. In $AGENT_HOME/bin
$ ./emctl start blackout test_dev3 -nodelevel -d 12:00
Oracle Enterprise Manager 10g Release 10.2.0.1.0.
Copyright (c) 1996, 2005 Oracle Corporation. All rights reserved.
Blackout test_dev3 added successfully
EMD reload completed successfully

$./emctl status blackout
Oracle Enterprise Manager 10g Release 10.2.0.1.0.
Copyright (c) 1996, 2005 Oracle Corporation. All rights reserved.
Blackoutname = Blackout-Sep 11 2008 11:47:59 PM
Targets = (bidev.corpnet.com:oracle_database,)
Time = ({2008-9-11|20:50:2|3600 Sec,|} )
Expired = True

Blackoutname = test_dev3(NODE_LEVEL)
Targets = (dbdev:host,)
Time = ({2008-10-01|11:09:33|720 Min,|} )
Expired = False

- Place a blackout on a target / database. In $AGENT_HOME/bin
$ ./emctl start blackout test_dev3 bidev.corpnet.com:oracle_database -d 03:00
Oracle Enterprise Manager 10g Release 10.2.0.1.0.
Copyright (c) 1996, 2005 Oracle Corporation. All rights reserved.
Blackout test_dev3 added successfully
EMD reload completed successfully

./emctl status blackout
Oracle Enterprise Manager 10g Release 10.2.0.1.0.
Copyright (c) 1996, 2005 Oracle Corporation. All rights reserved.
Blackoutname = Blackout-Sep 11 2008 11:47:59 PM
Targets = (bidev.corpnet.com:oracle_database,)
Time = ({2008-9-11|20:50:2|3600 Sec,|} )
Expired = True

Blackoutname = test_dev3
Targets = (bidev.corpnet.com:oracle_database,)
Time = ({2008-10-01|12:50:49|180 Min,|} )
Expired = False

- To stop the blackout, run
$ ./emctl stop blackout test_dev3
Oracle Enterprise Manager 10g Release 10.2.0.1.0.
Copyright (c) 1996, 2005 Oracle Corporation. All rights reserved.
Blackout test_dev3 stopped successfully
EMD reload completed successfully

- Additional notes
1. After "emctl start blackout test_dev3 ..." is executed, it will add an entry in $AGENT_HOME/sysman/emd/blackouts.xml. When "emctl stop blackout test_dev3 ..." is run, the entry is removed from the file. It seems if the blackout expires before the stop command is run, the entry will stay in the file for ever.
2. If you do not use the "-d" option in running "emctl start blackout ..." command, the GC console (Setup --> Blackouts) will list the blackout with "Ended" status and "Indefinite" duration immediately. It seems the blackout without "-d" option stops out quickly. It may be a bug in Grid Control 10.2.0.1.0.
3. Checking the blackout status from the repository table MGMT_BLACKOUTS.
SQL> select * from sysman.MGMT_BLACKOUTS
where BLACKOUT_NAME like 'test_dev3%';

CRS related commands

Oracle provides useful commands to manage CRS (Cluster Ready Services):

crs_stat -t --> Shows HA resource / service status (hard to read)
crsstat --> Ouptut of crs_stat -t formatted nicely (see Metalink note 259301.1)
crs_stop -all --> Stops all registered resources (but keeps CRS running!)
crs_start -all --> Starts all registered resources
crsctl check crs --> Verifies CSS,CRS,EVM functioning
crsctl stop crs --> Stops crs and all other services
crsctl start crs --> Starts crs and all other services

Two commands
crsctl disable crs --> Prevents CRS from starting on reboot
crsctl enable crs --> Enables CRS start on reboot
will update the file /etc/oracle/scls_scr/Node_name/root/crsstart which contains the string “enable” or “disable” as appropriate.

Command "ps -ef | grep d.bin" will check three main background processes. They are normally started by init during the operating system boot process. They can be started and stopped manually by issuing the command /etc/init.d/init.crs {start | stop | enable | disable} (or /etc/init.crs {start | stop | enable | disable})

oracle 2498 2091 0 Aug 18 - 8:18 /u01/crs/oracle/product/crs/bin/evmd.bin
root 2580 1927 0 Aug 18 - 705:21 /u01/crs/oracle/product/crs/bin/crsd.bin reboot
oracle 2662 2542 0 Aug 18 - 557:13 /u01/crs/oracle/product/crs/bin/ocssd.bin
root 2785 2951 0 Aug 18 - 1:59 /u01/crs/oracle/product/crs/bin/oprocd.bin run -t 1000 -m 500 -f

Here is a short description of each of the CRS daemon processes (Note 259301.1):

CRSD:
- Engine for HA operation
- Manages 'application resources'
- Starts, stops, and fails 'application resources' over
- Spawns separate 'actions' to start/stop/check application resources
- Maintains configuration profiles in the OCR
- Stores current known state in the OCR.
- Runs as root
- Is restarted automatically on failure
OCSSD:
- OCSSD is part of RAC and Single Instance with ASM
- Provides access to node membership
- Provides group services
- Provides basic cluster locking
- Integrates with existing vendor clusteware, when present
- Can also runs without integration to vendor clustware
- Runs as Oracle.
- Failure exit causes machine reboot. --> This is a feature to prevent data corruption in event of a split brain.
EVMD:
- Generates events when things happen
- Spawns a permanent child evmlogger
- Evmlogger, on demand, spawns children
- Scans callout directory and invokes callouts.
- Runs as Oracle.
- Restarted automatically on failure

Once the above processes are running, they will automatically start the following services in the following order if they are enabled.

- The nodeapps (gsd, VIP, ons, listener) are brought online.
- The ASM instances are brought online.
- The database instances are brought online.
- Any defined services are brought online.

Outputs from running some commands:

[root:/dssdb1]# crsctl stop crs
Stopping resources.
This could take several minutes.
Successfully stopped CRS resources.
Stopping CSSD.Shutting down CSS daemon.
Shutdown request successfully issued.
[root:/dssdb1]#

[root:/dssdb1]# crsctl start crs
Attempting to start CRS stack
The CRS stack will be started shortly
[root:/dssdb1]#

[oracle:/dssdb1]$ crsctl check crs
CSS appears healthy
CRS appears healthy
EVM appears healthy
[oracle:/dssdb1]$

If you see message other than above lines, you may use "ps -ef | grep d.bin" and "crs_stat -t" to check the status on each daemon and service.

Oracle provides an utility CLUVFY in $ORACLE_HOME/bin to verify the CRS as well:
[oracle:/dssdb1]$ cluvfy stage -post crsinst -n all -verbose

Database views for backuppiece info

1. On the RMAN catalog database
SELECT bp_key backuppiece_key, bs_key backupset_key, recid, set_count, incremental_level, handle, media, completion_time, status,
decode(backup_type, 'D', 'FULL', 'I', 'INCREMENTAL', 'L', 'LOGS') type
FROM rc_backup_piece
WHERE db_key = (select max(db_key) FROM rc_database WHERE name = 'PDWS')
-- and recid=2876
-- and status = 'X'
ORDER BY completion_time desc
;

After you get the backupset_key from this query, you can use "RMAN> list backupset backupset_key" or "RMAN> list backuppiece backuppiece_key" to see its contents (data files and/or archivelogs).

Once a backup piece has been deleted by "delete expired backup" and "delete obsolete", its record will be removed from this view and so all information on the backup piece in RMAN will be gone.

View RC_BACKUP_SET gives similar information, but is less useful.

2. On the target database
SELECT recid, set_count, handle, status, tag, media, completion_time, deleted
FROM v$BACKUP_PIECE
-- where recid=2876
order by completion_time desc;

This view keeps more records (maybe three months). But, after a backup piece has been deleted by "delete expired backup" and "delete obsolete", the HANDLE column of its record will become blank. So it is only useful if you want to find the tape ID from the MEDIA column.

Catalog the RMAN backuppieces

After the RMAN records are already deleted (by "delete expired backup" or "delete obsolete"), you need to catalog the RMAN backuppieces once you have the tape containing the RMAN backup of those backup pieces.

Same on the disk. After a file is deleted by RMAN, the file will no longer exist on disk. If somehow you saved a copy of the backup and later want to know what is in it, you have to use the RMAN "catalog" command to have RMAN review the file header, place in the controlfile the details about the backup. Then you run the "list backup" command to get details on the backup.

I believe Oracle is more confident on getting them back into RMAN if the backup pieces are on the disk than on the tape. If the backup is on the disk, see Note 727655.1 on getting it back. Here is an unpublished document (Note 550082.1) by Oracle on how to catalog tape backup pieces. I have not got chance to test it.

~~~~~~~~~~~~~~~~~~~~~~~~
Applies to:

Oracle Server - Enterprise Edition - Version: 10.1 to 11.1
Information in this document applies to any platform.
Applies to databases release 10g and further

Goal

Starting with 10g, it's possible to use the rman CATALOG command to add backuppieces stored in disk to the rman repository.
You may need to catalog backup pieces in the following situations:
. You copy or move a backup piece with an operating system utility and want it to be usable by RMAN.
. The RMAN metadata for the backup piece was removed, but the backup piece still exists. This situation can occur if you ran the DELETE command on a backup piece that was only temporarily unavailable.
. You make a NOCATALOG backup on one database host in a Data Guard environment and move the backup piece to the same location on a different database host. In this case, the recovery catalog has no record of the original backup piece.
. You do not use a recovery catalog and must re-create the control file, thereby losing all RMAN repository data. Cataloging your backups makes them available again.
. When control file autobackup is disabled, you back up the control file and then back up the archived redo logs. You can restore and mount the control file, but must catalog the backup pieces containing the archived redo logs backed up after the control file.

But it's not possible to use the CATALOG command for backup pieces stored in TAPE. This note explains how to add backuppieces stored in TAPE to the repository

Solution

From 10.1, there is an undocumented command that allows to catalog tape backup pieces:
CATALOG DEVICE TYPE 'SBT_TAPE' BACKUPPIECE '';

* Prerequisites
1. Use automatic channel configuration. It's mandatory to configure one sbt_tape device channel in your rman automatic configuration parameters;
CONFIGURE CHANNEL DEVICE TYPE 'SBT_TAPE' PARMS ''
2. It's necessary to know the backup piece file name in the tape and the backup piece file needs to be available and accessible.

* How to
Once there is a tape channel configured for accessing to the tape, the rman CATALOG command can be used to insert in RMAN catalog the tape backup piece:
CATALOG DEVICE TYPE 'SBT_TAPE' BACKUPPIECE '';

* Examples
- This is an example using Oracle Secure Backup (OSB):
1. Define a tape channel in the RMAN automatic configuration:
RMAN> CONFIGURE CHANNEL DEVICE TYPE 'SBT_TAPE' PARMS 'SBT_LIBRARY=/usr/local/oracle/backup/lib/libobk.so,ENV=(OB_MEDIA_FAMILY=RMAN-DEFAULT)';
2. Check that channel configuration is correct
RMAN> show all;
....
CONFIGURE CHANNEL DEVICE TYPE 'SBT_TAPE' PARMS 'SBT_LIBRARY=/usr/local/oracle/backup/lib/libobk.so,ENV=(OB_MEDIA_FAMILY=RMAN-DEFAULT)';
....
3. Catalog the backup piece
RMAN> CATALOG DEVICE TYPE 'SBT_TAPE' BACKUPPIECE '0pivagf8_1_1';

- The following will catalog a backuppiece on netbackup:
RMAN> CONFIGURE CHANNEL DEVICE TYPE 'SBT_TAPE' PARMS 'ENV=(NB_ORA_CLASS=oraclebkup, SBT_LIBRARY=/usr/openv/netbackup/bin/libobk.so';
RMAN> CATALOG DEVICE TYPE 'SBT_TAPE' BACKUPPIECE 'lij1qaa3_1_1';

Crosscheck archivelog all

The RMAN commands "crosscheck archivelog all" and "change archivelog all crosscheck" will check the archivelogs whether they are physically available on disk. If the archivelogs are no longer on disk, then the status in the RMAN catalog and controlfile will be marked from"A" for available to "X" for expired.

The RMAN command "delete noprompt expired archivelog all" will only delete entries on expired archivelogs from the RMAN catalog and controlfile (not from the disk bacause "X" means the archivelog has been removed from OS).

To see the list of expired archivelogs, run "delete expired archivelog all" and then answer "no" on confirmation.

Two ways to get archivelog space back from disk:
1. If the archivelogs have never been backed up to tape, back them up and delete them in one command:
RMAN> backup archivelog until logseq delete all input;

2. If you know the archivelogs have a good backup on the tape, you can remove them at OS level (Note 249452.1):
a) delete unwanted archive log files from disk ( rm /del )
b) connect to rman
c) RMAN> crosscheck archivelog all;
d) RMAN> delete expired archivelog all;

Note that "crosscheck backup of archivelog all" means crosschecking backups of archivelog files on the tape, and "delete expired backup of archivelog all" deletes the backup pieces from the MMD database. (??)