Monday, November 23, 2015

Run scp or sftp Without a Password

To automate a job for exchanging data files between two servers, we want to run it without entering a password by human being.  Assume there are two servers local2d (with users batchdev and batchmgr) and remote1d (with user user01).  

A pair of key files is required: id_rsa resides on the local server, id_rsa.pub is imported to remote server.

A. if a passphase was not entered for "ssh-keygen"
1. Generate the key files on the local server local2d

batchdev@local2d: /u06/app
$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/u06/app/.ssh/id_rsa): <Hit enter>   <== option: another path/name
Enter passphrase (empty for no passphrase): <Hit enter>           <== leave it empty
Enter same passphrase again: <Hit enter>
Your identification has been saved in /u06/app/.ssh/id_rsa.
Your public key has been saved in /u06/app/.ssh/id_rsa.pub.
The key fingerprint is:
66:38:f8:65:74:5d:10:88:f9:9a:3e:89:4a:d5:0c:49 batchdev@local2d.domain.com
The key's randomart image is:
+--[ RSA 2048]----+
|      E  o .oo.  |
|     . .o .. .   |
|      o ... .    |
|     . * ..      |
|    . + So       |
|     o *o        |
|    . .o .       |
|   .  . +        |
|    ..   .       |
+-----------------+
NOTES: If run "$ ssh-keygen -t dsa", it will generate two files with names id_dsa and id_dsa.phb.  More options, such as:
$ ssh-keygen -f $HOME/.ssh/ppm -t rsa -b 4096 -P "" -C "PPM Key"

batchdev@locald2d: /u06/app
$ cd .ssh
$ ls -al
-rw-------  1 batchdev users 1675 Oct 13 11:46 id_rsa
-rw-r--r--  1 batchdev users  410  Oct 13 11:46 id_rsa.pub
-rw-r--r--  1 batchdev users 1622 Aug  6 10:50 known_hosts

2. On the remote server remote1d
Task: Import the key from file id_rsa.pub of local machine to file authorized_keys of the remove machine.

$ hostname
remote1d.domain.com
$ whoami
user01

## copy the pub key to remote1d (from server local2d).
$ cd /u04/apps/.ssh
$ scp -p batchdev@local2d:/u06/app/.ssh/id_rsa.pub local2d.key
batchdev@local2d's password: xxxxx

$ mv local2d.key authorized_keys                <== do not overwrite the file!
         ## Note if the file exists, run "$ cat local2d.key >> authorized_keys"
$ chmod 600 authorized_keys

$ ls -al /u04/app/.ssh
-rw-------  1 user01 users  410    Oct 13 11:46 authorized_keys  
-rw-r--r--  1 user01 users  2087  Mar 30  2015 known_hosts

3. Test SFTP and SSH on local server local2d

batchdev@local2d: /u06/app/  
$ sftp remote1d                             ## batchdev can not get onto it. sftp asks the password.
Connecting to remote1d...
Password:

$ sftp user01@remote1d    ## But user01 can login to remote1d w/o a password!
Connecting to remote1d...
sftp> dir -all
-rw-r--r--    1 user01  users     481 Jun 22  2010 .profile
-rw-------    1 user01  users     933 Oct 13 12:35 .sh_history
drwx------   2 user01  users     096 Oct 13 12:27 .ssh

sftp> quit

In a script, use SFTP syntax to connect to remote server without a password after file id_rsa is created (on local server):
$ sftp -oidentityFile=/path/to/id_rsa userID@remoteServer.domain.com

Use "ssh -i /path/to/id_rsa ... " and "sftp -oidentityFile=/path/to/id_rsa ... " on local server to make connect to a remote server if private key id_rsa is saved in folder /path/to/ (vs. the default location $HOME/.ssh) of the local server.  Key in id_rsa.pub is still needed by the remote server.

SSH to remote1s as user01 also works:

batchdev@local2d: /u06/app  
$ ssh user01@remote1d         
$ hostname
remote1d.domain.com
$ exit

batchdev@local2d: /u06/app
$ scp user01@remote1d:/u04/app/cert.txt .   ## even SCP does not ask for the pwd
cert.txt                                                   100% 1383     1.4KB/s   00:00 

TROUBLESHOOTING: permission on /u06/app and /u04/app should be 755 or 700. 777 will make passwordless authentication not work. Needs more restrictive.

If it gives below warnings:
$ ssh user01@remote1d
The authenticity of host 'remote1d (167.xx.xx.xxx)' can't be established.
ECDSA key fingerprint is SHA256:r4ofQUP1F8ebW5hWCcCsK7ah...A.
ECDSA key fingerprint is MD5:49:86:cd:21:a3:a4:22:05:68:4a:0e:...:35.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'remote1d' (ECDSA) to the list of known hosts.
Warning: the ECDSA host key for 'remote1d' differs from the key for the IP address '167.xx.xx.xxx'
Offending key for IP in /u06/app/.ssh/known_hosts:20
Are you sure you want to continue connecting (yes/no)? yes

First, answer Yes, and delete line 20 from file /u06/app/.ssh/known_hosts (after doing a backup) on local2d. 
Then, try ssh to login twice. The warning will go away in the 2nd try.

$ ssh user01@remote1d
Warning: Permanently added the ECDSA host key for IP address '167.xx.xx.xxx' to the list of known hosts.

$ ssh user01@remote1d
Now, no more warnings

B. If there is a 2nd user batchMgr on server local2d wants to run sftp into remote1d, additional steps are necessary:
1. Login to local2d server with batchMgr account
2. Create a new directory apps_sftp_key and do a chmod 700 on this directory
3. Copy files id_rsa and id_rsa.pub from ~batchdev/.ssh to apps_sftp_key directory and do a chmod 600 on id_rsa

$ hostname
local2d
$ whoami
batchmgr
$ pwd
/home/batchmgr/apps_sftp_key
$ ls -al
total 16
drwx------  2 batchmgr 1211 4096 Sep 16 14:52 .
drwxrwxr-x 12 batchmgr 1211 4096 Sep 19 12:23 ..
-rw-------  1 batchmgr users 1675 Oct 13 11:46 id_rsa
-rw-r--r--  1 batchmgr users  410  Oct 13 11:46 id_rsa.pub 

One line to run SFTP without interactive:

batchmgr@locald2d: $ sftp -oidentityFile=~batchmgr/apps_sftp_key/id_rsa user01@remote1d

Or, more actions in one line:
batchmgr@locald2d: $ sftp -oport=22 -b sftp.ctl -oidentityFile=~batchmgr/apps_sftp_key/id_rsa user01@remote1d:/apps/local/ftp/in 
Changing to: /apps/local/ftp/in
sftp> put /home/batchmgr/temp/test.del test.del
Uploading /home/batchmgr/temp/test.del to /apps/local/ftp/in/test.del
sftp> version

SFTP protocol 3
sftp> quit

Here, sftp.ctl is a command file in the current directory:
$ more sftp.ctl
put /home/batchmgr/temp/test.del test.del
version
quit 

C. If a passphrase was entered for "ssh-keygen"

$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/u06/app/.ssh/id_rsa): <Hit enter>
Enter passphrase (empty for no passphrase):    <== "welcome01" works, but "welcome" not
Enter same passphrase again:<Enter your passphrase again>   <== welcome01

After complete steps in part A, logging onto remote machine by ssh will have to enter the passphrase:

$ ssh user01@remote1d    
Enter passphrase for key "/u06/app/.ssh/id_rsa':  welcome01
                                                                           <= Use file id_rsa on local machine
To suppress that, additional steps on ssh agent are needed on the local2d:

batchdev@local2d: /u06/app
$ echo $SHELL
/usr/bin/ksh
$ ssh-agent $SHELL
$ ps -ef | grep ssh-agent
  batchdev 118621 174327   0 17:32:17       -   0:00 ssh-agent /usr/bin/ksh
  batchdev 161219 174327   0 17:32:28  pts/0  0:00 grep ssh-agent

$ ssh-add
Enter passphrase for /u06/app/.ssh/id_rsa: welcome01
Identity added: /u06/app/.ssh/id_rsa (/u06/app/.ssh/id_rsa)

Now, sftp, ssh and scp shall work without a password or a passphrase.

UPDATES in 2024:
Linux RHEL8, "sestatus" can be used to check if SELinux label is enabled or not. If it is enabled, above passwordless setups may become not working. The cause is the labels on file authorized_keys in RHEL 8. Use " ls -alZ " to check it. Or, check my new post on this topic.

No comments: