Sunday, December 13, 2015

Configure printer to work with EBS

This summary is not available. Please click here to view the post.

Saturday, December 12, 2015

EBS Refresh

Refresh from EBSPROD (under /u06/app/EBSPROD) to EBSQA (under /u01/app/EBSQA) on a different host in Linux OS.

1. On source nodes, run PERL script. 
$ cd $ADMIN_SCRIPTS_HOME
$ perl adpreclone.pl appsTier        <-- No password is needed
   NOTE: Pre-clone log file located at $INST_TOP/admin/log/StageAppsTier_MMDDHHMM.log

$ cd $COMMON_TOP/clone      <-- to verify new folders are created
$ ls -altr $COMMON_TOP/clone
total 40
drwxr-xr-x  2 applmgr appsuser 4096 Jan 22  2008 html
drwxr-xr-x  3 applmgr appsuser 4096 Jan 22  2008 context
drwxr-xr-x  2 applmgr appsuser 4096 Jan 22  2008 bin
drwxr-xr-x  9 applmgr appsuser 4096 Sep 15  2013 ..
drwxr-xr-x  5 applmgr appsuser  4096 Dec  1 10:06 appsts
drwx------  7 applmgr appsuser  4096 Dec  1 10:07 jre
drwxr-xr-x  3 applmgr appsuser  4096 Dec  1 10:07 oui
drwxr-xr-x  4 applmgr appsuser  4096 Dec  1 10:07 jlib
drwxr-xr-x  5 applmgr appsuser  4096 Dec  1 10:07 appl
drwxr-xr-x 10 applmgr appsuser 4096 Dec  1 10:07 .

Do the same on other source hosts.

2. On target nodes, backup filess
Create a folder, such as /u01/app/admin_scripts/backups, to hold backed-up files. Then run the script backup_4refresh.sh to back up <SID_HOST> specific files.

$ cd admin_scripts/backups
$ ./backup_4refresh.sh

Confirm necessary files are just backed up:
$ ls -al /u01/app/admin_scripts/backups
$ ls -al /u01/app/admin_scripts/backups/admin
$ ls -al /u01/app/admin_scripts/backups/Apache

Optional:  Backup the entire folder to aother partition
$ cd /u01/app
$ tar -zhcvf /u02/app/$CONTEXT_NAME_backup_MMDDYY.tar.gz EBSQA >> backup_MMDD.log

Do the same on other hosts.

3. Make sure all apps services were stopped and database refresh was completed.
Now, remove directories

$ cd /u01/app
$ rm -rf EBSQA
$ rm -rf  oraEBSQAinventory; mkdir  oraEBSQAinventory

4. Copy directories from source node.
    NOTE: copy CM host files to CM host, web/forms host to web/forms host.

$ cd /u01/app
$ rsync --progress -avze ssh applmgr@source_host:/u06/app/EBSPROD /u01/app
   enter password

   sent 12619428 bytes  received 9649138852 bytes  1312649.72 bytes/sec
   total size is 33018009660  speedup is 3.42

It may give warning message:
rsync warning: some files vanished before they could be transferred (code 24) at main.c(1532) [generator=3.0.6]

5. Rename the folder and verify the size
$ cd /u01/app
$ mv EBSPROD EBSQA

$ ls -al EBSQAtotal 16
drwxr-xr-x  4 applmgr appsuser 4096 May  7  2013 .
drwxr-xr-x 16 applmgr appsuser 4096 Apr 15 11:27 ..
drwxr-xr-x  4 applmgr appsuser 4096 May  2  2013 apps
drwxr-xr-x  3 applmgr appsuser 4096 May  2  2013 inst

$cd apps
$ du -ks
31640672       .

$ cd ../inst
$ du -ks
2502420 .

6. Edit /etc/oraInst.loc file to make sure below line is used:

inventory_loc=/u01/app/oraEBSQAInventory

7. Start a new OS session (without running the .env file)

    If you do not modify the .profile, re-login will get message:
    $APPL_TOP/${CONTEXT_NAME}.env: cannot open [No such file or directory]
    Note: by this time, the .env file was deleted when old folder EBSQA was removed in Step 5.
             So env vairables do not exist

8. Go to /u01/app/EBSQA/apps/apps_st/comn/clone/bin    ($COMMON_TOP/clone/bin)
    option: delete old files in /u01/app/EBSQA/apps/apps_st/comn/clone
                $ cd /u01/app/EBS/apps/apps_st/comn/clone
                $ rm -rf jre_BAK_*

   Now, get below answers ready -
   Database Server Node => dbhost1q.domain.com
   database SID => EBSQA
   apps   => appsPWD
   Base Directory =>  /u01/app/EBSQA
   PORT => 1542
   port pool:  21  (1542 -1521)
 
9. Run clone perl script and answer questions
   $ perl adcfgclone.pl appsTier      <<-- run this in CM node first!!
  
++++++++++++++++ answers on CM node ++++++++++++++++
Target System Root Service [disabled] :                <-- for CM node. WEB/Form node will be opposite
Target System Web Entry Point Services [disabled] :
Target System Web Application Services [disabled] :
Target System Batch Processing Services [enabled] :
Target System Other Services [enabled] :               <-- for CM node
Do you want to preserve the Display [y] (y/n)  : n
Target System Display [host1q:0.0] : host1q.domain.com:21.0      <-- OAM will use it for display
Do you want the the target system to have the same port values as the source system (y/n) [y] ? : n
Target System Port Pool [0-99] : 21

++++++++++++++++ answers on Form/Web node ++++++++++++++++
Target System Root Service [enabled] :                       <= for Form node
Target System Web Entry Point Services [enabled] :
Target System Web Application Services [enabled] :
Target System Batch Processing Services [enabled] : disabled  <= for Form node
Target System Other Services [enabled] : disabled
Do you want to preserve the Display [xxx_name:0.0] (y/n)  : n
Target System Display [host2q:0.0] : host2q.domain.com:21.0   <= OAM will use it for display
Do you want the the target system to have the same port values as the source system (y/n) [y] ? : n
Target System Port Pool [0-99] : 21

The script may remain silent for a few minutes (or even 40 minutes) on executing $INST_TOP/admin/install/txkWfClone.sql !
(only statement SELECT * from v$session where upper(machine) like 'HOSTNAME%'; can tell if the sql script is running).

Answer "no" - Dot not start apps services after autoconfig completed.

Check the log file. Any more lines than below need your attention or fix:
$ egrep -i 'fail|error|ora-' $INST_TOP/admin/log/ApplyAppsTier_04151536.log
Error while running adlnktools.sh.
ERRORCODE = 0 ERRORCODE_END
ERRORCODE = 0 ERRORCODE_END
AC-50480: Internal error occurred: java.lang.Exception: Error while generating listener.ora.
Error generating tnsnames.ora from the database, temporary tnsnames.ora will be generated using templates
ERRORCODE = 2 ERRORCODE_END
   Result             : FAILED
ERRORCODE = 0 ERRORCODE_END
Unique constraint error (00001) is OK if key already exists
ERRORCODE = 0 ERRORCODE_END

ERRORCODE = 0 ERRORCODE_END
. . .  . . .  . . .

10. Start a new OS session. If necessary, modify the .profile to run .env 
$ . ./.profile

11. Remove obsolete files
$ cd $APPLPTMP    <-- make sure $APPLPTMP points to the right folder!
$ rm -f *.tmp

Note: seems the clone script cleans the folder $APPLTMP automatically.

12. Generate autoconfig file for DBA    (only do this on ONE node)
$ cd $AD_TOP/bin
$ ls -al ad*mk*

$ perl admkappsutil.pl
Starting the generation of appsutil.zip
Log file located at $INST_TOP/admin/log/MakeAppsUtil_04151607.log
output located at $INST_TOP/admin/out/appsutil.zip
MakeAppsUtil completed successfully.

$ cp -p $INST_TOP/admin/out/appsutil.zip $APPLPTMP

13. Verify apps password before email DBA with the password. And clean CM tables
$ sqlplus apps/passwd
SQL> @cmclean.sql
SQL> EXEC FND_CONC_CLONE.SETUP_CLEAN;
SQL> select * from fnd_nodes;    
(This SELECT shell get 0 row returned. After DBA runs adconfig.sh below, it shall have one row.)

Notes: cmclean.sql is an old file but it still works for me in my R12.1.3 instances
REM $Id: cmclean.sql,v 1.4 2001/04/07 15:55:07 pferguso Exp $

14. Ask DBA to run ADCONFIG
$ adconfig.sh contextfile=$ORACLE_HOME/db_ebsqa/appsutil/$CONTEXT_NAME.xml

NOTE: if .xml does not exist on a new instance, generate it first. here is how to generate the .xml file after unzip appsutil.zip on database server :
$ export PERL5LIB=$ORACLE_HOME/appsutil/perl
$ export LD_LIBRARY_PATH=$ORACLE_HOME/lib:$OARCLE_HOME/ctx/lib
$ perl $ORACLE_HOME/appsutil/bin/adbldxml.pl
<enter apps password when prompted>
<enter hostname when prompted>
<enter listener port when prompted>
<enter SID name when prompted>

Output should indicate:
The context file has been created at: $ORACLE_HOME/db_ebsdev/appsutil/EBSDEV_ebsdb1d.xml
(or $ perl adbldxml.pl tier=db appsuser=<APPSuser> )

15. Run script to restore files from backup

$ cd admin_scripts/backups
$ ./resotore_refresh.sh

15. Verify files and make sure two files for site custom
$FND_TOP/fndenv.env
$INST_TOP/ora/10.1.2/forms/server/default.env

Note: two files shall have a section in below format (Otherwise it will be overwritten by autoconfig). Or, modify them after adautocfg.
#Begin Customizations
custom tops
#End Customizations

$ ls -al $INST_TOP/certs/Apache
cwallet.sso  ewallet.p12

17. Update $CONTEXT_FILE

- APPLCSF & APPLPTMP on ALL nodes !
- Below ones are ONLY on Form/Web nodes:
webport
activewebport
login
external_url
webentryhost
webentryurlprotocol

Notes: For real port (not as 443 in F5) when SSL is enabled, make sure the port (such as 4472) for s_active_webport match the port in file ssl.conf and s_webssl_port in $CONTEXT_FILE .

18. Run AD autocfg :  always make CM node work first (after DBA is done with adconfig!)
$ cd $ADMIN_SCRIPTS_HOME
$ ./adautocfg.sh

Optional:  change APPS password using FNDCPASS here, and then run adautocfg.sh again on all nodes.

19. Source .profile to take new env variables after autocfg (or re-login)
$ . ./.profile

20. Start services

21. Login as XML Publisher Admin => Administration => General => replace the path with $APPLPTMP.

22. Update profile "Site%name%"
Check profile option: BNE Debug Log Directory, BNE Upload Staging Directory (then, bounce services)
                                 RRA: Service Prefix (this should be BLANK. Not  hard coded !!)
                                 RRA: Enabled (someones say this must be set to YES. Seems not necessary)
                                 Viewer: Text  (this should be blank or browser)

23. Modify files for customization. Maybe only on the web node. Such as:

- TEMPLATE FILE   : $FND_TOP/admin/template/httpd_conf_1013.tmp
    add a section to it
- TEMPLATE FILE   : $FND_TOP/admin/template/oracle_apache_conf_1013.tmp
   #include "%s_ora_config_home%/10.1.3/Apache/modplsql/conf/plsql.conf"
     include "%s_weboh_oh%/Apache/modplsql/conf/plsql.conf"            <<-- add this line
- Add APPS password to $IAS_ORACLE_HOME/Apache/modplsql/conf/dads.conf

24. Re-create database links, check DBA_DIRECTORIES   (DBA task)

25. Re-link some folders for $CUSTOM_TOP:
$ cd $CUSTOM_TOP/ftp
$ rm in
$ rm out
$ ln -s /path/to/ftp/in in
$ ln -s /path/to/ftp/out out
$ chmod 777 in
$ chmod 777 out

26. Optional: disable production users
-- Doc. ID: 1064798.1.  Disable EBS users   
DECLARE
  -- get user list which should be disabled
  cc number :=0;
  cursor c1 is
  select user_name
     from fnd_user
    where (end_date is null or end_date > sysdate)
                -- access all users that has no end-date or end-date is future
      and (user_name like 'B%' or user_name like 'M%')
      and user_name not in ('SYSADMIN', 'USERVIP1', 'USERVIP2', 'A999999'); -- excluding VIPs
BEGIN
  for c in c1 loop
    -- disable user
    fnd_user_pkg.disableuser(c.user_name);
    cc := cc+1;
    If ( MOD(cc, 50) = 0) then
      commit;
    end if;
  end loop;
  dbms_output.put_line('Totla disabled ' || cc);

  commit;    --commit changes
END;
/

++++++++ Troubleshooting +++++++
1. After CM node is up in the first time, run a CM job got error when opening the log
An error occurred while attempting to establish an Applications File Server connection with the
node FNDFS_CMHOST. There may be a network configuration problem, or the TNS listener
on node FNDFS_CMHOST may not be running. Please contact your system administrator.

you can confirm the problem by running below command on Web Host:

$ tnsping FNDFS_CMHOST
TNS-12547: TNS:lost contact

after host names were deleted by "EXEC FND_CONC_CLONE.SETUP_CLEAN" from database tables and when autoconfig was executed first on Concurrent Host CMhost, web host names are not saved to database tables yet and so tnsnames.ora file on CMhost does not have entries for web hosts.

Fix: two ways to fix it
1). on CM host
(a). Replace tnsnames.ora and sqlnet.ora in $TNS_ADMIN with files from the backup before refresh.
(b). adalnctl.sh stop
(c). adalnctl.sh start
Now, on the Web Host server, "tnsping FNDFS_CMHOST" returns "OK"

2). Run autoconfig AGAIN on all hosts, starting with CM host first.

2. When stopping CM services (to run autoconfig again for the new APPS password), 5 processes on
OS level did not stop. See all five of them in the Concurrent Manager:

CONCURRENT MANAGER                        Actual     Target
Output Post Processor                                     2              0
Workflow Agent Listener Service                     1              0
Workflow Mailer Service                                 1              0
Workflow Document Web Services Service     1              0

I clicked "Terminate" on them, but that did not kill the OS processes. I had to kill them by OS command:

$ kill -12 15524 15529 15544 15548 15555

After I ran autoconfig and started CM services, they showed "Terminted" in GUI Concurrent Manager.
I had to clicked each of them to "Activate" them. They did get started after the clicks !!
Should not try to terminate them in the GUI !!

3. When I ran the autoconfig last on CM node, the login page did not work!!
    So, run autoconfig on web/forms node last.

4. After clone script adcfgclone.pl worked, Autoconfig ended with below error in adconfig.log & ApplyAppsTier_11011132.log:
[CVM Error Report]
The following report lists errors encountered during CVM Phase
      <filename>  <return code where appropriate>
  /u03/app/EBSUAT/apps/apps_st/appl/ad/12.0.0/bin/adgentns.pl  2



Updating s_tnsmode to 'generateTNS'
UpdateContext exited with status: 0
AC-50480: Internal error occurred: java.lang.Exception: Error while generating listener.ora.
Error generating tnsnames.ora from the database, temporary tnsnames.ora will be generated using templates
Instantiating Tools tnsnames.ora
Tools tnsnames.ora instantiated
Web tnsnames.ora instantiated

adgentns.pl exiting with status 2


Fix:  run below line AGAIN
SQL> EXEC FND_CONC_CLONE.SETUP_CLEAN;
Then, run adconfig.sh on DB host. After that, make sure the DB host name shows up in below query before run adautocfg.sh on Apps tiers:
SQL> select * from fnd_nodes;

~~~~~~~~~~~~~~~~~~ script: backup_4refresh.sh ~~~~~~~~~~~~~~~~~~
#!/bin/ksh
# backup files before the refresh   Dec. 2015
cd $HOME/admin_scripts/backups
DT=`date +"%h_%d_%y"`   # backup date

echo It will delete old backup files. Continue ? yes or no
read  answer

case $answer in
  [yY][eE][sS]|[yY])

    if [ -n "$CONTEXT_FILE" ]; then
      echo 'running'
    else
      echo 'environment variable $CONTEXT_FILE does not exit. Exit ...'
      exit 1;
    fi;

    if [ -f $CONTEXT_FILE ]; then
      echo 'starting copy ...'
    else
      echo 'CONTEXT file $CONTEXT_FILE does not exist. Exit ...'
      exit 1;
    fi

    file1=`basename $CONTEXT_FILE`
    if [ -f $file1 ]; then
      mv $file1 $file1'_'$DT   # backup date
    fi
    file2=custom`basename $CONTEXT_FILE .xml`'.env'
    if [ -f $APPL_TOP/$file2 ]; then
      cp -p $APPL_TOP/$file2 .
    fi

    rm -rf Apache admin

    cp -p $CONTEXT_FILE .
    cp -rp $INST_TOP/certs/Apache .
    cp -rp $TNS_ADMIN .
    cp -p $FND_TOP/fndenv.env .
    cp -p $INST_TOP/ora/*2/forms/server/default.env .
    cp -p $APPL_TOP/admin/adkeystore.dat .
    cp -p $APPL_TOP/admin/adsign.txt .
    ## more optional files. Need to manually copy them back if needed
    cp -p $FND_TOP/admin/template/custom/rwbuilder_conf_1012.tmp .
    cp -p $FND_TOP/admin/template/ssl_conf_1013.tmp .
    cp -p $FND_TOP/admin/template/httpd_conf_1013.tmp .
    cp -p $FND_TOP/admin/template/oracle_apache_conf_1013.tmp .
    cp -p $IAS_ORACLE_HOME/Apache/modplsql/conf/dads.conf .
    echo 'Done with backing up files'
    ls -altr $HOME/admin_scripts/backups
    exit 0

   ;;

  [nN][oO]|[nN])
      echo "No"
   ;;

   *)
      echo "Invalid input..."
      exit 1
   ;;
esac
~~~~~~~~~~~~~~~~~~ scripts: restore_refresh.sh ~~~~~~~~~~~~~~~~~~~
#!/bin/ksh
# restore files AFTER clone script during the refresh   Dec. 2015
cd $HOME/admin_scripts/backups
DT=`date +"%h_%d_%y"`

echo It will overwrite some files. Continue? yes or no
read  answer

case $answer in
   [yY][eE][sS]|[yY])

    if [ -n "$CONTEXT_FILE" ]; then
      echo 'running'
    else
      echo 'environment variable $CONTEXT_FILE does not exit. Exit ...'
      exit 1;
    fi;

    file2=custom`basename $CONTEXT_FILE .xml`'.env'
    if [ -f $file2 ]; then
      cp -p $file2 $APPL_TOP/.
    fi

    mv $FND_TOP/fndenv.env $FND_TOP/fndenv.env_clone
    cp -p fndenv.env $FND_TOP/.

    mv $INST_TOP/ora/10.1.2/forms/server/default.env $INST_TOP/ora/10.1.2/forms/server/default.env_clone
    cp -p default.env $INST_TOP/ora/10.1.2/forms/server/.

    mv $INST_TOP/certs/Apache  $INST_TOP/certs/Apache'_'$DT
    cp -pr Apache $INST_TOP/certs/.

    ## files for JRE 1.8. They will be used if Java signing is re-ran. 
    mv $APPL_TOP/admin/adkeystore.dat $APPL_TOP/admin/adkeystore.dat_clone
    cp -p adkeystore.dat $APPL_TOP/admin/.

    mv $APPL_TOP/admin/adsign.txt $APPL_TOP/admin/adsign.txt_clone
    cp -p adsign.txt $APPL_TOP/admin/.

    echo 'Done with restoring files.'

    ;;

  [nN][oO]|[nN])
    echo "No"
    ;;

  *)
    echo "Invalid input..."
    exit 1
    ;;
esac
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The first task of EBS refresh is to refresh the database. Here are the steps:

0. Stop EBS apps
1. In the source database EBSPROD (in 10G and 11G)
1.1 Identify the SCN nmuber for the RMAN script. Check source database backup log to find the sequence.
1.2 SQL> select first_change#, next_change#, first_time from v$log_history
where sequence# = 11047;
2. In the target database EBSQA
2.1 Shutdown the target database
      SQL> alter database backup controlfile to trace;
      SQL> shutdown immediate;
2.2 Drop the databae
      SQL> startup mount restrict;
      SQL> drop database;
2.3 Get ready for the RMAN restore
      SQL> startup nomount;
2.4 Make sure host has enough disk space
2.5 Assume the source database was backed up by RMAN. Run the script
     $ cat db_refresh.sh
      ORACLE_SID=targetDB_ID
      export ORACLE_SID
      export TNS_ADMIN=$ORACLE_HOME/network/admin
      ORACLE_USER=oracle
      NLS_LANG=AMERICAN_AMERICA.AL32UTF8
      NLS_DATE_FORMAT='YYYY-MM-DD hh24:mi:ss'
      export NLS_LANG
      export NLS_DATE_FORMAT
      NB_ORA_CLASS=sourceDBserver
      export NB_ORA_CLASS
      DATE=`date +"%h%d%y_%H%M"`
      rman catalog rman/rmanPWD@rmanDB_ID target sys/PSWD@sourceDB_ID auxiliary / msglog db_refresh.${DATE}.log << EOF
      run {
      ALLOCATE AUXILIARY CHANNEL ch00 TYPE 'SBT_TAPE';
      ALLOCATE AUXILIARY CHANNEL ch01 TYPE 'SBT_TAPE';
      ALLOCATE AUXILIARY CHANNEL ch02 TYPE 'SBT_TAPE';
      SEND 'NB_ORA_CLIENT=sourceDBserver';
      SET NEWNAME FOR TEMPFILE 1 to '/path/to/temp01.dbf';
      SET NEWNAME FOR TEMPFILE 2 to '/path/to/temp02.dbf';
      duplicate target database to targetDB_ID
      UNTIL scn = 5968220384735;
      }
      EOF

      $ ./db_refresh.sh &

3. Bring the database up.
    Do NOT change any password (except SYSTEM).

EBS apps RDA

Two types of RDA on Oracle EBS apps.

1. Data collection test

- Run it for R12.1 from browser (Doc ID 732091.1)
  1. Log onto Oracle E-Business Suite
  2. Click on responsibility Application Diagnostics
  3. Select the Diagnose menu option
  4. Click button Select Application and select an appropriate Application, such as payables (SQLAP), receivables (AR)
  5. Scroll down to group "System Snapshot"
  6. Select test name "RDA"
  7. Input Parameters (* required)
    • Responsibility Id (LOV) *      (seems System Administrator works)
    • Application Shortname (LOV) *  (such as AR, XLA)
    • Mask sensitive data (LOV) *  (seems optional)
    • APPS Schema Username  
    • APPS Schema Password
- Run it by command line (Doc ID 427278.1)

$ cd $IZU_TOP/bin
$ ./rda.sh -vdCRP -e APPL _SHORT='AR' ACT

If getting error "RDA-00001: Cannot change to the output directory", rename the setup.cfg file in the directory and try it again.

2. Download file p20603454_8715310_Linux-x86-64.zip from Oracle support site and install it from unzip. Then
$ cd rda
$ ./rda.sh -T hcve

Monday, November 23, 2015

Run scp or sftp Without a Password

To automate a job for exchanging data files between two servers, we want to run it without entering a password by human being.  Assume there are two servers local2d (with users batchdev and batchmgr) and remote1d (with user user01).  

A pair of key files is required: id_rsa resides on the local server, id_rsa.pub is imported to remote server.

A. if a passphase was not entered for "ssh-keygen"
1. Generate the key files on the local server local2d

batchdev@local2d: /u06/app
$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/u06/app/.ssh/id_rsa): <Hit enter>   <== option: another path/name
Enter passphrase (empty for no passphrase): <Hit enter>           <== leave it empty
Enter same passphrase again: <Hit enter>
Your identification has been saved in /u06/app/.ssh/id_rsa.
Your public key has been saved in /u06/app/.ssh/id_rsa.pub.
The key fingerprint is:
66:38:f8:65:74:5d:10:88:f9:9a:3e:89:4a:d5:0c:49 batchdev@local2d.domain.com
The key's randomart image is:
+--[ RSA 2048]----+
|      E  o .oo.  |
|     . .o .. .   |
|      o ... .    |
|     . * ..      |
|    . + So       |
|     o *o        |
|    . .o .       |
|   .  . +        |
|    ..   .       |
+-----------------+
NOTES: If run "$ ssh-keygen -t dsa", it will generate two files with names id_dsa and id_dsa.phb.  More options, such as:
$ ssh-keygen -f $HOME/.ssh/ppm -t rsa -b 4096 -P "" -C "PPM Key"

batchdev@locald2d: /u06/app
$ cd .ssh
$ ls -al
-rw-------  1 batchdev users 1675 Oct 13 11:46 id_rsa
-rw-r--r--  1 batchdev users  410  Oct 13 11:46 id_rsa.pub
-rw-r--r--  1 batchdev users 1622 Aug  6 10:50 known_hosts

2. On the remote server remote1d
Task: Import the key from file id_rsa.pub of local machine to file authorized_keys of the remove machine.

$ hostname
remote1d.domain.com
$ whoami
user01

## copy the pub key to remote1d (from server local2d).
$ cd /u04/apps/.ssh
$ scp -p batchdev@local2d:/u06/app/.ssh/id_rsa.pub local2d.key
batchdev@local2d's password: xxxxx

$ mv local2d.key authorized_keys                <== do not overwrite the file!
         ## Note if the file exists, run "$ cat local2d.key >> authorized_keys"
$ chmod 600 authorized_keys

$ ls -al /u04/app/.ssh
-rw-------  1 user01 users  410    Oct 13 11:46 authorized_keys  
-rw-r--r--  1 user01 users  2087  Mar 30  2015 known_hosts

3. Test SFTP and SSH on local server local2d

batchdev@local2d: /u06/app/  
$ sftp remote1d                      ## atchdev can not get onto it. sftp asks the password
Connecting to remote1d...
Password:

$ sftp user01@remote1d    ## But user01 can login to remote1d w/o a password!
Connecting to remote1d...
sftp> dir -all
-rw-r--r--    1 user01  users     481 Jun 22  2010 .profile
-rw-------    1 user01  users     933 Oct 13 12:35 .sh_history
drwx------   2 user01  users     096 Oct 13 12:27 .ssh

sftp> quit

In a script, use SFTP syntax to connect to remote server without a password after file id_rsa is created (on local server):
$ sftp -oidentityFile=/path/to/id_rsa userID@remoteServer.domain.com

Use "ssh -i /path/to/id_rsa ... " and "sftp -oidentityFile=/path/to/id_rsa ... " on local server to make connect to a remote server if private key id_rsa is saved in folder /path/to/ (vs. the default location $HOME/.ssh) of the local server.  Key in id_rsa.pub is still needed by the remote server.

SSH to remote1s as user01 also works:

batchdev@local2d: /u06/app  
$ ssh user01@remote1d         
$ hostname
remote1d.domain.com
$ exit

batchdev@local2d: /u06/app
$ scp user01@remote1d:/u04/app/cert.txt .   ## even SCP does not ask for the pwd
cert.txt                                                   100% 1383     1.4KB/s   00:00 

TROUBLESHOOTING: permission on /u06/app and /u04/app should be 755 or 700. 777 will make passwordless authentication not work. Needs more restrictive.

If it gives below warnings:
$ ssh user01@remote1d
The authenticity of host 'remote1d (167.xx.xx.xxx)' can't be established.
ECDSA key fingerprint is SHA256:r4ofQUP1F8ebW5hWCcCsK7ah...A.
ECDSA key fingerprint is MD5:49:86:cd:21:a3:a4:22:05:68:4a:0e:...:35.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'remote1d' (ECDSA) to the list of known hosts.
Warning: the ECDSA host key for 'remote1d' differs from the key for the IP address '167.xx.xx.xxx'
Offending key for IP in /u06/app/.ssh/known_hosts:20
Are you sure you want to continue connecting (yes/no)? yes

First, answer Yes, and delete line 20 from file /u06/app/.ssh/known_hosts (after doing a backup) on local2d. 
Then, try ssh to login twice. The warning will go away in the 2nd try.

$ ssh user01@remote1d
Warning: Permanently added the ECDSA host key for IP address '167.xx.xx.xxx' to the list of known hosts.

$ ssh user01@remote1d
Now, no more warnings

B. If there is a 2nd user batchMgr on server local2d wants to run sftp into remote1d, additional steps are necessary:
1. Login to local2d server with batchMgr account
2. Create a new directory apps_sftp_key and do a chmod 700 on this directory
3. Copy files id_rsa and id_rsa.pub from ~batchdev/.ssh to apps_sftp_key directory and do a chmod 600 on id_rsa

$ hostname
local2d
$ whoami
batchmgr
$ pwd
/home/batchmgr/apps_sftp_key
$ ls -al
total 16
drwx------  2 batchmgr 1211 4096 Sep 16 14:52 .
drwxrwxr-x 12 batchmgr 1211 4096 Sep 19 12:23 ..
-rw-------  1 batchmgr users 1675 Oct 13 11:46 id_rsa
-rw-r--r--  1 batchmgr users  410  Oct 13 11:46 id_rsa.pub 

One line to run SFTP without interactive:

batchmgr@locald2d: $ sftp -oidentityFile=~batchmgr/apps_sftp_key/id_rsa user01@remote1d

Or, more actions in one line:
batchmgr@locald2d: $ sftp -oport=22 -b sftp.ctl -oidentityFile=~batchmgr/apps_sftp_key/id_rsa user01@remote1d:/apps/local/ftp/in 
Changing to: /apps/local/ftp/in
sftp> put /home/batchmgr/temp/test.del test.del
Uploading /home/batchmgr/temp/test.del to /apps/local/ftp/in/test.del
sftp> version

SFTP protocol 3
sftp> quit

Here, sftp.ctl is a command file in the current directory:
$ more sftp.ctl
put /home/batchmgr/temp/test.del test.del
version
quit 

C. If a passphrase was entered for "ssh-keygen"

$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/u06/app/.ssh/id_rsa): <Hit enter>
Enter passphrase (empty for no passphrase):    <== "welcome01" works, but "welcome" not
Enter same passphrase again:<Enter your passphrase again>   <== welcome01

After complete steps in part A, logging onto remote machine by ssh will have to enter the passphrase:

$ ssh user01@remote1d    
Enter passphrase for key "/u06/app/.ssh/id_rsa':  welcome01
                                                                           <= Use file id_rsa on local machine
To suppress that, additional steps on ssh agent are needed on the local2d:

batchdev@local2d: /u06/app
$ echo $SHELL
/usr/bin/ksh
$ ssh-agent $SHELL
$ ps -ef | grep ssh-agent
  batchdev 118621 174327   0 17:32:17       -   0:00 ssh-agent /usr/bin/ksh
  batchdev 161219 174327   0 17:32:28  pts/0  0:00 grep ssh-agent

$ ssh-add
Enter passphrase for /u06/app/.ssh/id_rsa: welcome01
Identity added: /u06/app/.ssh/id_rsa (/u06/app/.ssh/id_rsa)

Now, sftp, ssh and scp shall work without a password or a passphrase.

UPDATES in 2024:
Linux RHEL8, "sestatus" (or file /etc/selinux/config) can be used to check if SELinux label is enabled or not. If it is enabled, above passwordless setups may become not working. The cause is the labels on file authorized_keys in RHEL 8. Use " ls -alZ " to check it. Or, check my new post on this topic.

Saturday, November 7, 2015

EBS R12.1 patching for JRE 1.8

To make EBS R12.1.3 work with JRE (Java Run-time Environment) 1.8, patches on apps and both ORACLE_HOMEs are needed. The main document is ID 393931.1 "Deploying JRE (Native Plug-in) for Windows Clients in Oracle E-Business Suite Release 12". I do not see this activity requires any database patches. My databases are in 11.2.0.2 or above.

1. Verify the versions and conditions
  (1) $ $IAS_ORACLE_HOME/Apache/Apache/bin/httpd -v
  Server version: Oracle-Application-Server-10g/10.1.3.5.0 Oracle-HTTP-Server
  Server built:   May 16 2013 15:47:41

  (2) $ $ORACLE_HOME/bin/frmcmp_batch|grep Forms| grep Version
  Forms 10.1 (Form Compiler) Version 10.1.2.3.0 (Production)
  (Because 10.1.2.3.0 patch 5983622 was was installed by Installer, how to confirm it is installed?)

  (3) Confirm first 3 patches were installed
  select * from ad_bugs where bug_number in
  ('4377566',  -- Step 2.1, 393931.1
  '8919489',   -- Step 2.3.2, 437878.1
  '14837539', -- Step 2.1.1 & Step 3.8, 437878.1
  '11776182' ); -- Step 2.1.2 Footnote7, 437878.1. very old? Skip it

2. Patch in 10.1.3 ORACLE_HOME ( Section 1)

Shutdown apps services (adstpall.sh apps/appsPWD) on all nodes and complete a backup.

Apply patch 19568561 (required):
  $ cd $INST_TOP/ora/10.1.3
  $ . xxxx_XXX.env               <== change the ORACLE_HOME to 10.1.3
  $ echo $ORACLE_HOME 
     /u02/app/EBSDev/apps/tech_st/10.1.3        <==  make sure it is 10.1.3!

  $ export OPATCH_PLATFORM_ID=46
     (to avoid error on 64-bit Linux:
     OPatch detects your platform as 226 while this patch 19568561 supports platforms: 46 Linux Intel)
  $ cd 19568561
  $ opatch apply

Update on 12/19/2015: Document 393931.1 was updated recently with recommendation of replacing this patch 19568561 with October 2015 CUP patch 21845960. I found patch 21845960 worked better and fixes issue with Firefox (version 43.0+).

3. Patches in 10.1.2 ORACLE_HOME (OracleAS 10g Patches. Follow Doc ID 437878.1
Start a new OS session, or $ . .profile
$ echo $ORACLE_HOME
   /u02/app/EBSDev/apps/tech_st/10.1.2      <== make sure it is 10.1.2

(1) p14825718_10105_LINUX.zip     (MLR patch or Bundle Patch. It is pre-requisite for patch 21103001 )
     Verify conditions are met to apply Patch 14825718 (Step 3.2 notes of Document 437878.1):
     12881480, 13808590, 14041415, 14262118, 14614795, 14577216, 9593176 were NOT applied
     while 6995251, 7121788 were APPLIED

$ cd 14825718
$ opatch apply

NOTES: It will roll back some patches that were previously applied. Accept that and do NOT stop it.
The patch's README is not applicable to E-Business Suite. Step 5 in below covers it after all Forms patches are applied. 

Install any patches that miss from the list of Step 2.1.2 in 10.1.2 ORACLE_HOME. I had to catch up 14 patches (well, even the document 437878.1 does not say "required").

(2) p6640838_10106_Linux-x86-64.zip     <== pre-requisite for patch 21103001
This patch 6640838 only adds missing components to OUI 10.1.0.6.0, and does not install a new OUI.
Installation steps with screenshots: How to patch OUI for installing overlay patches on top of Forms Bundle Patch - 9593176 (Doc ID 1301320.1)

$ mv $ORACLE_HOME/OPatch $ORACLE_HOME/OPatch.pre_6640838
to reserve the higher version of opatch.
$ export DISPLAY=xxx.xx.xxxx:0.0

$ cd cd/Disk1/install

$ Modify file oraparam.ini to include Linux 5     <== need this workaround on RHEL5
                    Linux=redhat-2.1AS,redhat-3,redhat-4,redhat-5
$ ./runInstaller
or
$ ./runInstaller -ignoreSysPrereqs   ==> follow Doc ID 1301320.1 to the finish line!

$ cd $ORACLE_HOME
$ mv OPatch OPatch_delete
$ mv OPatch.pre_6640838 OPatch

(3) p8551790_10123_LINUX.zip            <== pre-requisite for patch 21103001

$ cd 8551790
$ opatch apply

(4) p21103001_101232_LINUX.zip
 This patch replaces 19434967
 Pre-requisite: OUI componetes (patch 6640838) and patch 8551790.  Also references:    
        Post Steps For Patch 19434967 Files Do Not Exist (Doc ID 1945012.1)
        Font Changes In EBS Forms After Application Of CPU Patches (Doc ID 2005998.1)
Notes: if opatch version is lower than 1.0.0.0.63, it may give a misleading error: the patch directory 21103301 doesn't match the patch id.

Updates in June 2016: patch 21103001 is replaced by MLR patch 22698265 after JRE 1.8.0_92 was released (see Change Log of Doc 437878.1 and Doc. 393931.1). During the installation of patch 22698265, it will rollback patch 21103001 if it was applied. It has the same pre-requisites as 21103001 does.

$ cd 21103001
$ opatch apply

(5) p10152652_10123_LINUX.zip

pre-requisite: patch 7121788. Yes
verify: $ chmod +x $ORACLE_HOME/bin/genshlib

$ cd 10152652
$ opatch apply
$ sh ./patch.sh
... ... ... ... 
I skipped RSF (required support files) patches in Step 3.4 (and Step 3.5), because my database is in 11.2.0.2 and those patches seem for database 10g.

4. Apply EBS R12.1 Forms Interoperability Patch (Step 2.1.1, Step 3.8 of Doc ID 437878.1)

 SQL> select * from ad_bugs where bug_number = '14837539';  -- to confirm it was not installed yet

 $ adadmin    to enable maintenance mode
 $ cd 14837539
 $ adpatch

5. Post AS10g patching steps (Section 4 of Doc ID 437878.1)

$ cd $ORACLE_HOME/forms/lib
$ make -f ins_forms.mk install

$ cd $ORACLE_HOME/reports/lib
$ make -f ins_reports.mk install

$ adadmin
    ==> 1 Select Generate Applications Files
    ==> 4 Generate Product JAR Files  (No  - Do not opt to force the regeneration of all JAR files)

optional: verify the versions.
$ export DISPLAY=XXX.XXX.XXX:0.0
$ ORACLE_HOME/bin/frmcmp help=y
$ ORACLE_HOME/bin/rwrun ?|grep Release

UPDATES:
I highly recommend to run "adstrtal.sh" to start all services and check out web login and forms. At this time point, the startup script may deploy new EAR file. If any failure, you know it is not caused by new JRE version of next steps.

I had one instance that failed to launch forms after Step 10. There was a Time out error in adstrtal.log on "adformsctl.sh start" to deploy formsapp.ear file. After I recycled all services, adopmnctl.sh showed "OC4J: forms" status got changed from DOWN to ALIVE. But, GUI forms was still not popup. It took me many hours to find that the problem was not from the new JRE version. Actually, after I re-ran Step 10, I was able to launch EBS forms even though the errors from EAR deployment never got fixed.

6.  Apply the JRE Interoperability Patch 21624242:R12.TXK.B (Step 2.1). Now follow Doc ID 393931.1
SQL> select * from ad_bugs where bug_number = '4377566'; --confirm pre-patch 4377566 was applied

$ mv $FND_TOP/bin/txkSetPlugin.sh $FND_TOP/bin/txkSetPlugin-4377566.sh
$ cd 21624242
$ adpatch

7. AD Patch 17191279 - Enhanced Jar Signing for Oracle E-Business Suite (Doc ID 1591073.1)

(as sysdba) SQL> @adgrants.sql apps
Note: This script may give errors from dropping some objects. Ignore them.
If a newer version of adgrants.sql (e.g. the one by Oct 2015 CPU patch) has been executed before, this step can be skipped.

$ cd 17191279
$ adpatch

8. AD Patch 18312333 (see Doc ID 1591073.1)

$ cd 18312333
$ adpatch

9. Download JRE file (Step 2.2)
   Click on the hype link " JRE Parameter Settings " (in Step 2.2) to get the information table in
   Appendix D: Reference Information of this document.
  
   Patch 21045690: Oracle JDK 8 Update 51
     selected and downloaded this Window one for both Windows (32-bit) and Windows x64 (64-bit),
     although my EBS hosts use Linux OS:
     jre-8u51-windows-i586.exe     <== In Oct. 2015 I downloaded this latest one (32-bit)

   (or, http://www.oracle.com/technetwork/java/javase/downloads/jre8-downloads-2133155.html)
 
$ cp -p jre-8u51-windows-i586.exe $COMMON_TOP/webapps/oacore/util/jinitiator/j2se18051.exe
  
10. Run script on web tier only (Section 3 of Doc ID 393931.1 )
The document says "Run the $FND_TOP/bin/txkSetPlugin.sh script against the web node". But I did not meet problem after running it on CM node also.

 $ $FND_TOP/bin/txkSetPlugin.sh 18051
... ... ...
Updating XML context file with new J2SE parameters...
Updated J2SE plugin to jdk
Updated J2SE Version to 1.8.0_051
Updated J2SE Classid to CAFEEFAC-0018-0000-0051-ABCDEFFEDCBA

AutoConfig is configuring the Applications environment ... ...

If success, two variables shall get new values in $CONTEXT_FILE:
$ grep sun $CONTEXT_FILE
    <sun_plugin_ver oa_var="s_sun_plugin_ver">1.8.0_051</sun_plugin_ver>
    <sun_plugin_type oa_var="s_sun_plugin_type">jdk</sun_plugin_type>
    <sun_clsid oa_var="s_sun_clsid">CAFEEFAC-0018-0000-0051-ABCDEFFEDCBA</sun_clsid>

$ cat $FORMS_WEB_CONFIG_FILE|grep sun_plugin_version| cut -c 1-35      (see Doc. 468311.1)
sun_plugin_version=1.8.0_051

UPDATES in January 2016: after newer JRE 1.8.0_66 version became available, I downloaded it from patch 22286087 and ran below lines, instead. All worked the same as JRE 1.8.0_51 on both server and user's client.
$ cp -p jre-8u66-windows-i586.exe $COMMON_TOP/webapps/oacore/util/jinitiator/j2se18066.exe
$ $FND_TOP/bin/txkSetPlugin.sh 18066

11. Start apps services
 $ adadmin    to disable maintenance mode
 $ adstrtal.sh apps/PW
12. On client workstation's Java Console, add the URL (https://sitename.domain.com) to Exception list on Security tab. Without this, forms will get " Application Blocked by Java Security " popup.

Now, the EBS Forms shall fire up on a workstation where JRE 1.8.0_51 was installed (with 2 or 3 annoying popups).

TROUBLESHOOTING:

If the forms link has no response or Forms do not show up after clicks, clean Java cache and IE cache on the client machine before re-trying it. You may try it in Firefox and if you get " 404 Not Found  The requested URL /forms/frmservlet was not found on this server. " error, there might be a problem with EAR file deployment. Scary part! Workarounds: (a) Stop all services cleanly and try startup adstrtal.sh again. (b) In some of my cases, I had to re-run Step 5 or Step 10 to get forms fired up. (c) Manually deploy EAR file.

Normally, just keep all default options under Java Console. If Java Console log shows timed out or failure error, test some options under Advanced tab in Java. Also, on IE Tools => Manage add-ons, click the dropdown under Show (on the left) to select "All adds-on", then make sure Java add-ons are "Enabled". You shall see below similar two (or more):






NOTES:
(1). In some environment, users may not have permission to install software on their workstation. JRE 1.8.0_51+ (32-bit) shall have been installed on PC by Windows Admin before users can launch EBS Forms.
(2). Make sure TLS1.x is enabled. If TLS1.x is not enabled, enable it first. Otherwise, EBS Forms may not launch after JRE1.8 patches are applied.
(3). If users' client has higher than 1.8.0_51 (such as JRE 1.8.0_66) installed, EBS Forms will still work with the higher JRE class (with a couple of popups for confirmation). But it will not work with JRE 1.9.0_xx (a higher JRE family in the future).

Post steps:
Follow Doc ID 1591073.1 on Java signing (to get rid of Step 12 above).