Monday, April 7, 2014

Apply Oracle database 11G Security Patch Update (SPU) on AIX

January 2014 Database SPU Installation steps:

1.    Check/install opatch:
Above patches require opatch version 11.2.0.3. Check current version of opatch and install correct version if required. To check:
$ cd /app/oracle/product/OPatch
$ ./opatch version
To install opatch newer version 11.2.0.3, download patch 6880880 and unzip it:
$ cd /app/oracle/product
$ rm -rf OPatch

$ unzip opatch_p6880880_112000_AIX64-5L.zip
$ rm opatch_p6880880_112000_AIX64-5L.zip
$ export PATH=/app/oracle/product/OPatch:$PATH
$ opatch version        <= to verify opatch version. It should be 11.2.0.3

2.    FTP/unzip SPU patch:
$ mkdir $ORACLE_BASE/patches/spu_jan14_11203
$ cd  $ORACLE_BASE/patches/spu_jan14_11203
$ unzip p17478415_112030_AIX64-5L.zip
$ rm p17478415_112030_AIX64-5L.zip

3.    Check/set environment:

- Set correct ORACLE_HOME
Set ORACLE_BASE
  $ export ORACLE_BASE=/app/oracle

Set LIBPATH
  $ export LIBPATH=$ORACLE_HOME/lib

Set LD_LIBRARY_PATH
  $ export LD_LIBRARY_PATH=$ORACLE_HOME/lib

- Ensure that the $PATH has the following executables: make, ar, ld, and nm.
$ which make
$ which ar
$ which ld
$ which nm

The location of these executables depends on your operating system. On many operating systems, they are located in /usr/ccs/bin, in which case you can set your PATH as follows:

      $ export PATH=/usr/ccs/bin:$PATH

- Check jdk version: It should be version 1.4.2 or higher.

$ java -version

If java version is lower, check if version 1.4.2 or higher is available on the host.
      $ lslpp -L | grep Java
              (or)
    $ lslpp -L | grep Java14
    $ export PATH=/usr/java14/bin:$PATH

If available, adjust PATH accordingly and check again.

If java version 1.4.2 or higher is not available, work with System Admin to have it installed.

- Check location of oracle inventory in /etc/oraInst.loc
- Run oracle inventory

    $ cd /app/oracle/product/OPatch
    $ ./opatch lsinventory
    Make sure that Oracle version is 11.2.0.3.

- Check XDB: XDB must be VALID.
SQL> select status from dba_registry where comp_id='XDB'

If XDB is invalid, re-install DB – reference Oracle Support note 1292089.1

4.    List invalid objects:

For each database on the server, save a list of invalid objects for comparison after SPU patch.
  SQL> select owner, object_name, object_type from dba_objects
             where status= 'INVALID';

5.    Stop database replication tool if any.

6.    Stop database monitoring tools, such as Grid Control agent.

7.    Shutdown databases for the Oracle Home being patched and stop listener.

8.    Check processes:
Make sure that there are no oracle processes running except aioserver process.
  $ ps -fu'oracle’ | grep -v aioserver
  $ ps -ef | grep sqlplus

Make sure that there are no oraggs processes running except aioserver process.
  $ ps -fu'oragg’

9.    Check shared file in use:
Following commands should not return any file name:
  $ genld -l | grep $ORACLE_HOME
  $ genkld | grep $ORACLE_HOME

If these return any file name, have system administrator run following command
  $ /usr/sbin/slibclean

Check again.

10.    Apply patch

$ cd $ORACLE_BASE/patches/spu_jan14_11203/17478415
$ /app/oracle/product/OPatch/opatch napply -skip_subset -skip_duplicate

An observation: During installation on one host, it might as for Oracle Support login ID / password to make an internet connection to Oracle Support website for any latest information relating patch. Just press return.

Patch will possible complete with warnings, as listed below.  Per Oracle these can safely be ignored:
ld: 0711-319 WARNING: Exported symbol not defined: ****
Symbol xxxxxx is already exported.
ld: 0711-773 WARNING: Object ***, imported symbol ***
Symbol was expected to be local.
ld: 0711-783 WARNING: TOC overflow. TOC size: Maximum size:
0711-224 WARNING: Duplicate symbol

Check the following log files in $ORACLE_BASE/cfgtoollogs/catbundle for any errors:
    catbundle_CPU_<database SID>_APPLY_<TIMESTAMP>.log
    catbundle_CPU_<database SID>_GENERATE_<TIMESTAMP>.log

11.    Check inventory again:
$ /app/oracle/product/OPatch/opatch lsinventory

12.    Load modified SQL files into databases:

For each database instance running on the Oracle home being patched, connect to the database using SQL*Plus. Connect as SYSDBA and run the catbundle.sql script as follows:
$ cd $ORACLE_HOME/rdbms/admin
$ sqlplus /nolog
SQL> CONNECT / AS SYSDBA
SQL> STARTUP
SQL> @catbundle.sql cpu apply
SQL> @utlrp.sql
SQL> QUIT

Note: Start the LISTENER

For information about the catbundle.sql script, see My Oracle Support Note 605795.1 Introduction to Oracle Database catbundle.sql.

Check the following log files in $ORACLE_BASE/cfgtoollogs/catbundle for any errors:
catbundle_CPU_<database SID>_APPLY_<TIMESTAMP>.log
catbundle_CPU_<database SID>_GENERATE_<TIMESTAMP>.log

13.    Invalid objects check
For each database on the server, make a list of invalid objects and compare with the list saved before patching. Take appropriate action if there are new invalid objects

14.    Check XDB
If XDB is invalid, there is a possibility that LIBPATH has not been set correctly.

To check status of XDB:
  SQL> select status from DBA_REGISTRY where comp_id='XDB'

If status is INVALID:
  $ export LIBPATH=$ORACLE_HOME/lib
  Shutdown database
  Startup database
      Run utlrp

If XDB is still invalid, re-install DB – reference Oracle Support note 1292089.1

15.    Final check on registry:
$ sqlplus /nolog
SQL> CONNECT / AS SYSDBA
SQL> SELECT * FROM sys.registry$history where comments='CPUJan2014';
    It should return one line for CPUJan2014.