DBVERIFY

DBVERIFY is an Oracle utility for verifying the structure in data files when you experiences ORA-01578 / ORA-08103 or any other kind of corrupt messages.

During a database recovery, I gor error from creating control file:

ORA-01503: CREATE CONTROLFILE failed
ORA-01565: error in identifying file
'/t10/app/oracle/oradata/lawtest/lawts_accons_ptn_idx_7_1.dbf'
ORA-27041: unable to open file
IBM AIX RISC System/6000 Error: 5: I/O error
Additional information: 2

DBVERIFY provided great evidence to System Admin that the copy of some datafiles is a bad copy by OS command, bacause DBVERIFY showed no error on other files:

$ dbv file=lawts_accons_ptn_idx_7_1.dbf blocksize=16384
DBVERIFY: Release 9.2.0.6.0 - Production on Sun Apr 27 15:03:36 2008
Copyright (c) 1982, 2002, Oracle Corporation. All rights reserved.
DBV-00100: Specified FILE (lawts_accons_ptn_idx_7_1.dbf) not accessible

$ dbv file=lawts_acbudhdr_ptn_idx_max_1.dbf blocksize=16384
DBVERIFY: Release 9.2.0.6.0 - Production on Sun Apr 27 15:04:11 2008
Copyright (c) 1982, 2002, Oracle Corporation. All rights reserved.
DBVERIFY - Verification starting : FILE = lawts_acbudhdr_ptn_idx_max_1.dbf
DBVERIFY - Verification complete

Total Pages Examined : 16384
Total Pages Processed (Data) : 0
Total Pages Failing (Data) : 0
Total Pages Processed (Index): 9996
Total Pages Failing (Index): 0
Total Pages Processed (Other): 120
Total Pages Processed (Seg) : 0
Total Pages Failing (Seg) : 0
Total Pages Empty : 6268
Total Pages Marked Corrupt : 0
Total Pages Influx : 0
Highest block SCN : 36435995208 (8.2076256840)

Blocksize is specified by parameter DB_BLOCK_SIZE and is necessary in running DBV.

If ASM storage is used, we need to specify USERID to get authenticated on ASM Instance. The below script will get lines for running DBV on all datafiles of a database on ASM:
select 'dbv file=' || name || ' blocksize='|| block_size || ' USERID=sys/&SYS_PASSWORD logfile=' ||
substr(name, instr(name, '/', -1, 1) +1) || '.' || file# || '.log'
from v$datafile;

Place a blackout in Grid Control

There are two ways to place a blackout in Grid Control.
- From the Grid Control UI (Console), using Setup to create a blackout or using the icon on each Target page.
- From the command line at the agent home.

Here are the commands for creating a blackout:

- Place a blackout at host level. In $AGENT_HOME/bin
$ ./emctl start blackout test_dev3 -nodelevel -d 12:00
Oracle Enterprise Manager 10g Release 10.2.0.1.0.
Copyright (c) 1996, 2005 Oracle Corporation. All rights reserved.
Blackout test_dev3 added successfully
EMD reload completed successfully

$./emctl status blackout
Oracle Enterprise Manager 10g Release 10.2.0.1.0.
Copyright (c) 1996, 2005 Oracle Corporation. All rights reserved.
Blackoutname = Blackout-Sep 11 2008 11:47:59 PM
Targets = (bidev.corpnet.com:oracle_database,)
Time = ({2008-9-11|20:50:2|3600 Sec,|} )
Expired = True

Blackoutname = test_dev3(NODE_LEVEL)
Targets = (dbdev:host,)
Time = ({2008-10-01|11:09:33|720 Min,|} )
Expired = False

- Place a blackout on a target / database. In $AGENT_HOME/bin
$ ./emctl start blackout test_dev3 bidev.corpnet.com:oracle_database -d 03:00
Oracle Enterprise Manager 10g Release 10.2.0.1.0.
Copyright (c) 1996, 2005 Oracle Corporation. All rights reserved.
Blackout test_dev3 added successfully
EMD reload completed successfully

./emctl status blackout
Oracle Enterprise Manager 10g Release 10.2.0.1.0.
Copyright (c) 1996, 2005 Oracle Corporation. All rights reserved.
Blackoutname = Blackout-Sep 11 2008 11:47:59 PM
Targets = (bidev.corpnet.com:oracle_database,)
Time = ({2008-9-11|20:50:2|3600 Sec,|} )
Expired = True

Blackoutname = test_dev3
Targets = (bidev.corpnet.com:oracle_database,)
Time = ({2008-10-01|12:50:49|180 Min,|} )
Expired = False

- To stop the blackout, run
$ ./emctl stop blackout test_dev3
Oracle Enterprise Manager 10g Release 10.2.0.1.0.
Copyright (c) 1996, 2005 Oracle Corporation. All rights reserved.
Blackout test_dev3 stopped successfully
EMD reload completed successfully

- Additional notes
1. After "emctl start blackout test_dev3 ..." is executed, it will add an entry in $AGENT_HOME/sysman/emd/blackouts.xml. When "emctl stop blackout test_dev3 ..." is run, the entry is removed from the file. It seems if the blackout expires before the stop command is run, the entry will stay in the file for ever.
2. If you do not use the "-d" option in running "emctl start blackout ..." command, the GC console (Setup --> Blackouts) will list the blackout with "Ended" status and "Indefinite" duration immediately. It seems the blackout without "-d" option stops out quickly. It may be a bug in Grid Control 10.2.0.1.0.
3. Checking the blackout status from the repository table MGMT_BLACKOUTS.
SQL> select * from sysman.MGMT_BLACKOUTS
where BLACKOUT_NAME like 'test_dev3%';