Monday, November 23, 2015

Run scp or sftp Without a Password

To automate a job for exchanging data files between two servers, it is asked to run it without entering a password by human being.  Assume there are two servers local2d (with users batchdev and batchmgr) and remote1d (with user user01).  Below steps worked on RHEL5. 

A. if a passphase was not entered for "ssh-keygen"
1. on the local server local2d

batchdev@local2d: /u06/app
$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/u06/app/.ssh/id_rsa): <Hit enter>   <== option: another path/name
Enter passphrase (empty for no passphrase): <Hit enter>           <== leave it empty
Enter same passphrase again: <Hit enter>
Your identification has been saved in /u06/app/.ssh/id_rsa.
Your public key has been saved in /u06/app/.ssh/id_rsa.pub.
The key fingerprint is:
66:38:f8:65:74:5d:10:88:f9:9a:3e:89:4a:d5:0c:49 batchdev@local2d.domain.com
The key's randomart image is:
+--[ RSA 2048]----+
|      E  o .oo.  |
|     . .o .. .   |
|      o ... .    |
|     . * ..      |
|    . + So       |
|     o *o        |
|    . .o .       |
|   .  . +        |
|    ..   .       |
+-----------------+
Note: If run "$ ssh-keygen -t dsa", it will generate two files with names id_dsa and id_dsa.phb

batchdev@locald2d: /u06/app
$ cd .ssh
$ ls -al
-rw-------  1 batchdev users 1675 Oct 13 11:46 id_rsa
-rw-r--r--  1 batchdev users  410  Oct 13 11:46 id_rsa.pub
-rw-r--r--  1 batchdev users 1622 Aug  6 10:50 known_hosts

2. On the remote server remote1d
$ hostname
remote1d.domain.com
$ whoami
user01

## copy the pub key to remote1d (from server local2d).
$ cd /u04/apps/.ssh
$ scp -p batchdev@local2d:/u06/app/.ssh/id_rsa.pub local2d.key
batchdev@local2d's password: xxxxx

$ mv local2d.key authorized_keys                <== do not overwrite the file!
## Note if the file exists, run "$ cat local2d.key >> authorized_keys"
$ chmod 600 authorized_keys

$ ls -al /u04/app/.ssh
-rw-------  1 user01 users  410    Oct 13 11:46 authorized_keys  
-rw-r--r--  1 user01 users  2087  Mar 30  2015 known_hosts

3. Test SFTP and SSH on local server local2d

batchdev@local2d: /u06/app/  
$ sftp remote1d                             ## batchdev can not get onto it. sftp asks the password.
Connecting to remote1d...
Password:

$ sftp user01@remote1d      ## But login to remote1d as user01 works without a password!
Connecting to remote1d...
sftp> dir -all
-rw-r--r--    1 user01  users     481 Jun 22  2010 .profile
-rw-------    1 user01  users     933 Oct 13 12:35 .sh_history
drwx------   2 user01  users     096 Oct 13 12:27 .ssh
sftp> quit

batchdev@local2d: /u06/app  
$ ssh user01@remote1d            ## ssh to remote1d as user01 also works!
$ hostname
remote1d.domain.com
$ exit

batchdev@local2d: /u06/app
$ scp user01@remote1d:/u04/app/cert.txt .   ## even SCP does not ask for the password
cert.txt                                                   100% 1383     1.4KB/s   00:00 

TROUBLESHOOTING: permission on /u06/app and /u04/app should be 755 or 700. 777 will make passwordless authentication not work. Needs more restrictive.

If it gives below warnings:
$ ssh user01@remote1d
The authenticity of host 'remote1d (167.xx.xx.xxx)' can't be established.
ECDSA key fingerprint is SHA256:r4ofQUP1F8ebW5hWCcCsK7ah...A.
ECDSA key fingerprint is MD5:49:86:cd:21:a3:a4:22:05:68:4a:0e:...:35.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'remote1d' (ECDSA) to the list of known hosts.
Warning: the ECDSA host key for 'remote1d' differs from the key for the IP address '167.xx.xx.xxx'
Offending key for IP in /u06/app/.ssh/known_hosts:20
Are you sure you want to continue connecting (yes/no)? yes

First, answer Yes, and delete line 20 from file /u06/app/.ssh/known_hosts (after doing a backup) on local2d. 
Then, try ssh to login twice. The warning will go away in the 2nd try.

$ ssh user01@remote1d
Warning: Permanently added the ECDSA host key for IP address '167.xx.xx.xxx' to the list of known hosts.

$ ssh user01@remote1d
Now, no more warnings

B. If there is a 2nd user batchMgr on server local2d wants to run sftp into remote1d, additional steps are necessary:
1. Login to local2d server with batchMgr account
2. Create a new directory apps_sftp_key and do a chmod 700 on this directory
3. Copy files id_rsa and id_rsa.pub from ~batchdev/.ssh to apps_sftp_key directory and do a chmod 600 on id_rsa

$ hostname
local2d
$ whoami
batchmgr
$ pwd
/home/batchmgr/apps_sftp_key
$ ls -al
total 16
drwx------  2 batchmgr 1211 4096 Sep 16 14:52 .
drwxrwxr-x 12 batchmgr 1211 4096 Sep 19 12:23 ..
-rw-------  1 batchmgr users 1675 Oct 13 11:46 id_rsa
-rw-r--r--  1 batchmgr users  410  Oct 13 11:46 id_rsa.pub 

One line to run SFTP without interactive:

batchmgr@locald2d: $ sftp -oport=22 -b sftp.ctl -oIdentityFile=~batchmgr/apps_sftp_key/id_rsa user01@remote1d:/apps/local/ftp/in 
Changing to: /apps/local/ftp/in
sftp> put /home/batchmgr/temp/test.del test.del
Uploading /home/batchmgr/temp/test.del to /apps/local/ftp/in/test.del
sftp> version

SFTP protocol 3
sftp> quit

Here, sftp.ctl is a command file in the current directory:
$ more sftp.ctl
put /home/batchmgr/temp/test.del test.del
version
quit 

C. If a passphrase was entered for "ssh-keygen"

$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/u06/app/.ssh/id_rsa): <Hit enter>
Enter passphrase (empty for no passphrase):    <== "welcome01" works, but "welcome" not
Enter same passphrase again:<Enter your passphrase again>   <== welcome01

Additional steps on ssh agent are needed on the local2d to make it work:

batchdev@local2d: /u06/app
$ echo $SHELL
/usr/bin/ksh
$ ssh-agent $SHELL
$ ps -ef | grep ssh-agent
  batchdev 118621 174327   0 17:32:17       -   0:00 ssh-agent /usr/bin/ksh
  batchdev 161219 174327   0 17:32:28  pts/0  0:00 grep ssh-agent

$ ssh-add
Enter passphrase for /u06/app/.ssh/id_rsa: welcome01
Identity added: /u06/app/.ssh/id_rsa (/u06/app/.ssh/id_rsa)

Now, sftp, ssh and scp shall work without a password or a passphase.

Saturday, November 7, 2015

EBS R12.1 patching for JRE 1.8

To make EBS R12.1.3 work with JRE (Java Run-time Environment) 1.8, patches on apps and both ORACLE_HOMEs are needed. The main document is ID 393931.1 "Deploying JRE (Native Plug-in) for Windows Clients in Oracle E-Business Suite Release 12". I do not see this activity requires any database patches. My databases are in 11.2.0.2 or above.

1. Verify the versions and conditions
  (1) $ $IAS_ORACLE_HOME/Apache/Apache/bin/httpd -v
  Server version: Oracle-Application-Server-10g/10.1.3.5.0 Oracle-HTTP-Server
  Server built:   May 16 2013 15:47:41

  (2) $ $ORACLE_HOME/bin/frmcmp_batch|grep Forms| grep Version
  Forms 10.1 (Form Compiler) Version 10.1.2.3.0 (Production)
  (Because 10.1.2.3.0 patch 5983622 was was installed by Installer, how to confirm it is installed?)

  (3) Confirm first 3 patches were installed
  select * from ad_bugs where bug_number in
  ('4377566',  -- Step 2.1, 393931.1
  '8919489',   -- Step 2.3.2, 437878.1
  '14837539', -- Step 2.1.1 & Step 3.8, 437878.1
  '11776182' ); -- Step 2.1.2 Footnote7, 437878.1. very old? Skip it

2. Patch in 10.1.3 ORACLE_HOME ( Section 1)

Shutdown apps services (adstpall.sh apps/appsPWD) on all nodes and complete a backup.

Apply patch 19568561 (required):
  $ cd $INST_TOP/ora/10.1.3
  $ . xxxx_XXX.env               <== change the ORACLE_HOME to 10.1.3
  $ echo $ORACLE_HOME 
     /u02/app/EBSDev/apps/tech_st/10.1.3        <==  make sure it is 10.1.3!

  $ export OPATCH_PLATFORM_ID=46
     (to avoid error on 64-bit Linux:
     OPatch detects your platform as 226 while this patch 19568561 supports platforms: 46 Linux Intel)
  $ cd 19568561
  $ opatch apply

Update on 12/19/2015: Document 393931.1 was updated recently with recommendation of replacing this patch 19568561 with October 2015 CUP patch 21845960. I found patch 21845960 worked better and fixes issue with Firefox (version 43.0+).

3. Patches in 10.1.2 ORACLE_HOME (OracleAS 10g Patches. Follow Doc ID 437878.1
Start a new OS session, or $ . .profile
$ echo $ORACLE_HOME
   /u02/app/EBSDev/apps/tech_st/10.1.2      <== make sure it is 10.1.2

(1) p14825718_10105_LINUX.zip     (MLR patch or Bundle Patch. It is pre-requisite for patch 21103001 )
     Verify conditions are met to apply Patch 14825718 (Step 3.2 notes of Document 437878.1):
     12881480, 13808590, 14041415, 14262118, 14614795, 14577216, 9593176 were NOT applied
     while 6995251, 7121788 were APPLIED

$ cd 14825718
$ opatch apply

NOTES: It will roll back some patches that were previously applied. Accept that and do NOT stop it.
The patch's README is not applicable to E-Business Suite. Step 5 in below covers it after all Forms patches are applied. 

Install any patches that miss from the list of Step 2.1.2 in 10.1.2 ORACLE_HOME. I had to catch up 14 patches (well, even the document 437878.1 does not say "required").

(2) p6640838_10106_Linux-x86-64.zip     <== pre-requisite for patch 21103001
This patch 6640838 only adds missing components to OUI 10.1.0.6.0, and does not install a new OUI.
Installation steps with screenshots: How to patch OUI for installing overlay patches on top of Forms Bundle Patch - 9593176 (Doc ID 1301320.1)

$ mv $ORACLE_HOME/OPatch $ORACLE_HOME/OPatch.pre_6640838
to reserve the higher version of opatch.
$ export DISPLAY=xxx.xx.xxxx:0.0

$ cd cd/Disk1/install

$ Modify file oraparam.ini to include Linux 5     <== need this workaround on RHEL5
                    Linux=redhat-2.1AS,redhat-3,redhat-4,redhat-5
$ ./runInstaller
or
$ ./runInstaller -ignoreSysPrereqs   ==> follow Doc ID 1301320.1 to the finish line!

$ cd $ORACLE_HOME
$ mv OPatch OPatch_delete
$ mv OPatch.pre_6640838 OPatch

(3) p8551790_10123_LINUX.zip            <== pre-requisite for patch 21103001

$ cd 8551790
$ opatch apply

(4) p21103001_101232_LINUX.zip
 This patch replaces 19434967
 Pre-requisite: OUI componetes (patch 6640838) and patch 8551790.  Also references:    
        Post Steps For Patch 19434967 Files Do Not Exist (Doc ID 1945012.1)
        Font Changes In EBS Forms After Application Of CPU Patches (Doc ID 2005998.1)
Notes: if opatch version is lower than 1.0.0.0.63, it may give a misleading error: the patch directory 21103301 doesn't match the patch id.

Updates in June 2016: patch 21103001 is replaced by MLR patch 22698265 after JRE 1.8.0_92 was released (see Change Log of Doc 437878.1 and Doc. 393931.1). During the installation of patch 22698265, it will rollback patch 21103001 if it was applied. It has the same pre-requisites as 21103001 does.

$ cd 21103001
$ opatch apply

(5) p10152652_10123_LINUX.zip

pre-requisite: patch 7121788. Yes
verify: $ chmod +x $ORACLE_HOME/bin/genshlib

$ cd 10152652
$ opatch apply
$ sh ./patch.sh
... ... ... ... 
I skipped RSF (required support files) patches in Step 3.4 (and Step 3.5), because my database is in 11.2.0.2 and those patches seem for database 10g.

4. Apply EBS R12.1 Forms Interoperability Patch (Step 2.1.1, Step 3.8 of Doc ID 437878.1)

 SQL> select * from ad_bugs where bug_number = '14837539';  -- to confirm it was not installed yet

 $ adadmin    to enable maintenance mode
 $ cd 14837539
 $ adpatch

5. Post AS10g patching steps (Section 4 of Doc ID 437878.1)

$ cd $ORACLE_HOME/forms/lib
$ make -f ins_forms.mk install

$ cd $ORACLE_HOME/reports/lib
$ make -f ins_reports.mk install

$ adadmin
    ==> 1 Select Generate Applications Files
    ==> 4 Generate Product JAR Files  (No  - Do not opt to force the regeneration of all JAR files)

optional: verify the versions.
$ export DISPLAY=XXX.XXX.XXX:0.0
$ ORACLE_HOME/bin/frmcmp help=y
$ ORACLE_HOME/bin/rwrun ?|grep Release

UPDATES:
I highly recommend to run "adstrtal.sh" to start all services and check out web login and forms. At this time point, the startup script may deploy new EAR file. If any failure, you know it is not caused by new JRE version of next steps.

I had one instance that failed to launch forms after Step 10. There was a Time out error in adstrtal.log on "adformsctl.sh start" to deploy formsapp.ear file. After I recycled all services, adopmnctl.sh showed "OC4J: forms" status got changed from DOWN to ALIVE. But, GUI forms was still not popup. It took me many hours to find that the problem was not from the new JRE version. Actually, after I re-ran Step 10, I was able to launch EBS forms even though the errors from EAR deployment never got fixed.

6.  Apply the JRE Interoperability Patch 21624242:R12.TXK.B (Step 2.1). Now follow Doc ID 393931.1
SQL> select * from ad_bugs where bug_number = '4377566'; --confirm pre-patch 4377566 was applied

$ mv $FND_TOP/bin/txkSetPlugin.sh $FND_TOP/bin/txkSetPlugin-4377566.sh
$ cd 21624242
$ adpatch

7. AD Patch 17191279 - Enhanced Jar Signing for Oracle E-Business Suite (Doc ID 1591073.1)

(as sysdba) SQL> @adgrants.sql apps
Note: This script may give errors from dropping some objects. Ignore them.
If a newer version of adgrants.sql (e.g. the one by Oct 2015 CPU patch) has been executed before, this step can be skipped.

$ cd 17191279
$ adpatch

8. AD Patch 18312333 (see Doc ID 1591073.1)

$ cd 18312333
$ adpatch

9. Download JRE file (Step 2.2)
   Click on the hype link " JRE Parameter Settings " (in Step 2.2) to get the information table in
   Appendix D: Reference Information of this document.
  
   Patch 21045690: Oracle JDK 8 Update 51
     selected and downloaded this Window one for both Windows (32-bit) and Windows x64 (64-bit),
     although my EBS hosts use Linux OS:
     jre-8u51-windows-i586.exe     <== In Oct. 2015 I downloaded this latest one (32-bit)

   (or, http://www.oracle.com/technetwork/java/javase/downloads/jre8-downloads-2133155.html)
 
$ cp -p jre-8u51-windows-i586.exe $COMMON_TOP/webapps/oacore/util/jinitiator/j2se18051.exe
  
10. Run script on web tier only (Section 3 of Doc ID 393931.1 )
The document says "Run the $FND_TOP/bin/txkSetPlugin.sh script against the web node". But I did not meet problem after running it on CM node also.

 $ $FND_TOP/bin/txkSetPlugin.sh 18051
... ... ...
Updating XML context file with new J2SE parameters...
Updated J2SE plugin to jdk
Updated J2SE Version to 1.8.0_051
Updated J2SE Classid to CAFEEFAC-0018-0000-0051-ABCDEFFEDCBA

AutoConfig is configuring the Applications environment ... ...

If success, two variables shall get new values in $CONTEXT_FILE:
$ grep sun $CONTEXT_FILE
    <sun_plugin_ver oa_var="s_sun_plugin_ver">1.8.0_051</sun_plugin_ver>
    <sun_plugin_type oa_var="s_sun_plugin_type">jdk</sun_plugin_type>
    <sun_clsid oa_var="s_sun_clsid">CAFEEFAC-0018-0000-0051-ABCDEFFEDCBA</sun_clsid>

$ cat $FORMS_WEB_CONFIG_FILE|grep sun_plugin_version| cut -c 1-35      (see Doc. 468311.1)
sun_plugin_version=1.8.0_051

UPDATES in January 2016: after newer JRE 1.8.0_66 version became available, I downloaded it from patch 22286087 and ran below lines, instead. All worked the same as JRE 1.8.0_51 on both server and user's client.
$ cp -p jre-8u66-windows-i586.exe $COMMON_TOP/webapps/oacore/util/jinitiator/j2se18066.exe
$ $FND_TOP/bin/txkSetPlugin.sh 18066

11. Start apps services
 $ adadmin    to disable maintenance mode
 $ adstrtal.sh apps/PW
12. On client workstation's Java Console, add the URL (https://sitename.domain.com) to Exception list on Security tab. Without this, forms will get " Application Blocked by Java Security " popup.

Now, the EBS Forms shall fire up on a workstation where JRE 1.8.0_51 was installed (with 2 or 3 annoying popups).

TROUBLESHOOTING:

If the forms link has no response or Forms do not show up after clicks, clean Java cache and IE cache on the client machine before re-trying it. You may try it in Firefox and if you get " 404 Not Found  The requested URL /forms/frmservlet was not found on this server. " error, there might be a problem with EAR file deployment. Scary part! Workarounds: (a) Stop all services cleanly and try startup adstrtal.sh again. (b) In some of my cases, I had to re-run Step 5 or Step 10 to get forms fired up. (c) Manually deploy EAR file.

Normally, just keep all default options under Java Console. If Java Console log shows timed out or failure error, test some options under Advanced tab in Java. Also, on IE Tools => Manage add-ons, click the dropdown under Show (on the left) to select "All adds-on", then make sure Java add-ons are "Enabled". You shall see below similar two (or more):






NOTES:
(1). In some environment, users may not have permission to install software on their workstation. JRE 1.8.0_51+ (32-bit) shall have been installed on PC by Windows Admin before users can launch EBS Forms.
(2). Make sure TLS1.x is enabled. If TLS1.x is not enabled, enable it first. Otherwise, EBS Forms may not launch after JRE1.8 patches are applied.
(3). If users' client has higher than 1.8.0_51 (such as JRE 1.8.0_66) installed, EBS Forms will still work with the higher JRE class (with a couple of popups for confirmation). But it will not work with JRE 1.9.0_xx (a higher JRE family in the future).

Post steps:
Follow Doc ID 1591073.1 on Java signing (to get rid of Step 12 above).


Sunday, November 1, 2015

Find the version of executable file and database package

1. When see below lines in request log, SQL statement could be used to find the executable name.
    -----------------------------------------------------------------------------
    Copyright (c) 1979, 1999, Oracle Corporation. All rights reserved.
    MSCPLD module: Loader Worker With Direct Load Option
    +---------------------------------------------------------------------------+

Step1
SQL> select EXECUTABLE_NAME,EXECUTION_FILE_NAME,EXECUTION_FILE_PATH
 from fnd_executables
 where EXECUTABLE_NAME like '%MSCPLD%';

EXECUTABLE_NAME    EXECUTION_FILE_NAME        EXECUTION_FILE_PATH
MSCPLD                          MSCPLD.sh   

Step 2. Then, use "locate" to find the file:
$ locate MSCPLD.sh
 /u02/app/EBSDEV/apps/apps_st/appl/msc/12.0.0/bin/MSCPLD.sh

2. How to get the header file versions of all the header files for an executable in Unix

Example 1
$ cd $AP_TOP/bin
$ strings -a APXXTR |grep Header

Example 2, ARRGTA file:
$ strings -a $AR_TOP/bin/ARRGTA | grep '$Header'

SELECT /* $Header: fdffvs.lc 120.17.12010000.8 2010/05/07 22:42:10 tebarnes ship $ */ event_code, user_exit FROM fnd_flex_validation_events WHERE flex_value_set_id = :id ORDER BY event_code
SELECT /* $Header: fdffvs.lc 120.17.12010000.8 2010/05/07 22:42:10 tebarnes ship $ */ additional_where_clause FROM fnd_flex_validation_tables WHERE flex_value_set_id = :id
$Header: arrgta.opc 120.9 2006/06/27 17:16:49 rkader ship $
$Header: arjbal.lpc 120.2 2005/10/24 14:13:01 srivasud ship $
$Header: arjigl.lpc 120.18.12010000.6 2009/10/05 20:43:34 mraymond ship $
 .......

Notes: "strings" can also get Java file's version:
$ strings -a $JAVA_TOP/oracle/apps/frm/xdo/bne/utilities/FrmUploadModule.class |grep '$Header'
S$Header: FrmUploadModule.java 120.2.12010000.16 2014/05/09 05:25:46 jsiripur ship $

3. Use below statement to get database package version applied to the database:
SQL> select * from dba_source where name='XXXXX' and line=2;
Example:
select text from dba_source where name='AP_ACCTG_DATA_FIX_PKG' and line=2;
TEXT
/* $Header: apgdfals.pls 120.1.12010000.21 2013/09/05 13:06:08 pshivara ship $ */
/* $Header: apgdfalb.pls 120.1.12010000.68 2013/10/24 11:27:22 rseeta ship $ */

To check the code file version on file system, use
$ grep -i Header $AP_TOP/patch/115/sql/apgdfalb.pls

NOTE1: EBS utility adident (in unix, windows and other OS) can also be used to provide the file versions.
NOTE2: Before applying a patch, you may check if there a higher version of file in other patches:
How to find the Latest/Newest Version of an Oracle Receivables File (Doc ID 1584999.1)

    Blank log or error "Function ncrorou returned error code 3010"

    When trying to open the log or output files from EBS R12.1 Forms, suddenly get a totally blank html page or below error:

    An error occurred while attempting to receive the output arguments of the remote procedure. Function ncrorou returned error code 3010. The Applications File Server process may have exited abnormally. Please contact your system administrator

    I found the cause can be bad Temp tablespace in the database, over-size log file, or no disk space left in somewhere on the server.