CRS related commands

Oracle provides useful commands to manage CRS (Cluster Ready Services):

crs_stat -t --> Shows HA resource / service status (hard to read)
crsstat --> Ouptut of crs_stat -t formatted nicely (see Metalink note 259301.1)
crs_stop -all --> Stops all registered resources (but keeps CRS running!)
crs_start -all --> Starts all registered resources
crsctl check crs --> Verifies CSS,CRS,EVM functioning
crsctl stop crs --> Stops crs and all other services
crsctl start crs --> Starts crs and all other services

Two commands
crsctl disable crs --> Prevents CRS from starting on reboot
crsctl enable crs --> Enables CRS start on reboot
will update the file /etc/oracle/scls_scr/Node_name/root/crsstart which contains the string “enable” or “disable” as appropriate.

Command "ps -ef | grep d.bin" will check three main background processes. They are normally started by init during the operating system boot process. They can be started and stopped manually by issuing the command /etc/init.d/init.crs {start | stop | enable | disable} (or /etc/init.crs {start | stop | enable | disable})

oracle 2498 2091 0 Aug 18 - 8:18 /u01/crs/oracle/product/crs/bin/evmd.bin
root 2580 1927 0 Aug 18 - 705:21 /u01/crs/oracle/product/crs/bin/crsd.bin reboot
oracle 2662 2542 0 Aug 18 - 557:13 /u01/crs/oracle/product/crs/bin/ocssd.bin
root 2785 2951 0 Aug 18 - 1:59 /u01/crs/oracle/product/crs/bin/oprocd.bin run -t 1000 -m 500 -f

Here is a short description of each of the CRS daemon processes (Note 259301.1):

CRSD:
- Engine for HA operation
- Manages 'application resources'
- Starts, stops, and fails 'application resources' over
- Spawns separate 'actions' to start/stop/check application resources
- Maintains configuration profiles in the OCR
- Stores current known state in the OCR.
- Runs as root
- Is restarted automatically on failure
OCSSD:
- OCSSD is part of RAC and Single Instance with ASM
- Provides access to node membership
- Provides group services
- Provides basic cluster locking
- Integrates with existing vendor clusteware, when present
- Can also runs without integration to vendor clustware
- Runs as Oracle.
- Failure exit causes machine reboot. --> This is a feature to prevent data corruption in event of a split brain.
EVMD:
- Generates events when things happen
- Spawns a permanent child evmlogger
- Evmlogger, on demand, spawns children
- Scans callout directory and invokes callouts.
- Runs as Oracle.
- Restarted automatically on failure

Once the above processes are running, they will automatically start the following services in the following order if they are enabled.

- The nodeapps (gsd, VIP, ons, listener) are brought online.
- The ASM instances are brought online.
- The database instances are brought online.
- Any defined services are brought online.

Outputs from running some commands:

[root:/dssdb1]# crsctl stop crs
Stopping resources.
This could take several minutes.
Successfully stopped CRS resources.
Stopping CSSD.Shutting down CSS daemon.
Shutdown request successfully issued.
[root:/dssdb1]#

[root:/dssdb1]# crsctl start crs
Attempting to start CRS stack
The CRS stack will be started shortly
[root:/dssdb1]#

[oracle:/dssdb1]$ crsctl check crs
CSS appears healthy
CRS appears healthy
EVM appears healthy
[oracle:/dssdb1]$

If you see message other than above lines, you may use "ps -ef | grep d.bin" and "crs_stat -t" to check the status on each daemon and service.

Oracle provides an utility CLUVFY in $ORACLE_HOME/bin to verify the CRS as well:
[oracle:/dssdb1]$ cluvfy stage -post crsinst -n all -verbose