When I logged onto a server to clone an EBS instance from production, it gave a message
id: cannot find name for group ID 148.
"id" command shows group name is missing.
$ id
uid=1172(ebsdev) gid=148() groups=148(ebs)
I ignored the message and completed the clone. After that, Apache service did not start and even did not create any log files in $LOG_HOME/ora/10.1.3/Apache folder. The only error is in $LOG_HOME/ora/10.1.3/opmn/HTTP_Server~1.log :
apachectl startssl: execing httpd
httpd: bad group name 148
I realized where the problem came from and asked Linux Admin to fix the Group name issue.
$ id
uid=1172(ebsdev) gid=148(ebs) groups=148(ebs)
Then, I replaced 148 with the group name in file $ORA_CONFIG_HOME/10.1.3/Apache/Apache/conf/httpd.conf
User ebsdev
Group ebs
I also verified file /etc/group is READable. After that, I started Apache without problem.
Tuesday, April 11, 2017
Concurrent job out of memory issue
Create Accounting (XLAACCPB) Program in one of our instances gave below error in log file:
Calling XDO Data Engine...
****Warning!!! Due to high volume of data, got out of memory exception...***
****Please retry with scalable option or modify the Data template to run in scalable mode...***
Oracle Support recommended below action plan in a SR. That fixed the issue.
Action Plan:
==================
1. In Profile, please change following setting at ALL levels in System option:
FND: Debug Log Enabled => NO
AR: Enable Debug Message Output => NO
SLA: Enable Diagnostics => No
2. Doc ID 1350719.1
Access the "System Administrator" responsibility.
Navigate to Concurrent > Program > Define.
Search for the report (e.g., Create Accounting)
Click on Parameters button.
Move to the last parameter.
Add the Scalable Flag as follows -
Parameter- P_SCALABLE_FLAG
Description - ScalableFlag
Enabled - Check
Value Set - XLA_SRS_NO_VALIDATION
Default Type - Constant
Default Value - Y
Prompt - P_SCALABLE_FLAG
Token - ScalableFlag (Case sensitive. Give this exact value)
P_SCALABLE_FLAG - has a Default Value of 'Y'
3.
A> Increase the memory for Java used by the concurrent program.
Query the concurrent program in
System Administrator> Concurrent> Program> Define >
Query for the Create Accounting concurrent program >
Enter value for Options field with : -Xmx2048M (Note: Don't miss the dash before the X) .
B> Time out parameters (Profile)
Concurrent:OPP Response Timeout : 240
Concurrent:OPP Process Timeout : 10800 sec (3 hours)
Additional Notes: Oracle Doc ID 838810.1 provides a similar solution to fix below error and it worked in my instance.
Subledger Accounting: Version : 12.0.0
XLAPEXRPT module: Subledger Period Close Exceptions Report
Calling XDO Data Engine...
****Warning!!! Due to high volume of data, got out of memory exception...***
****Please retry with scalable option or modify the Data template to run in scalable mode...***
Set the following parameters in the concurrent program definition:
Under Concurrent > Program > Define, query for concurrent program
short name = XLAPEXRPT and application = Subledger Accounting.
1. In the 'Options' field enter -Xmx2048M
2. Add a parameter to the concurrent token = ScalableFlag, default value set to 'Y'. Same way
you add DebugFlag
In the application developer responsibility
-> Concurrent -> Program -> Define
-> Query for short name= XLAPEXRPT
-> Click on Parameters
->define a new parameter (if not existing)
parameter name=P_DEBUG_FLAG, attach any valueset that is YES_NO valueset.
Give Default value=Y
Token=DebugFlag
Save
3. Submit the request with scalable mode Yes.
Calling XDO Data Engine...
****Warning!!! Due to high volume of data, got out of memory exception...***
****Please retry with scalable option or modify the Data template to run in scalable mode...***
Oracle Support recommended below action plan in a SR. That fixed the issue.
Action Plan:
==================
1. In Profile, please change following setting at ALL levels in System option:
FND: Debug Log Enabled => NO
AR: Enable Debug Message Output => NO
SLA: Enable Diagnostics => No
2. Doc ID 1350719.1
Access the "System Administrator" responsibility.
Navigate to Concurrent > Program > Define.
Search for the report (e.g., Create Accounting)
Click on Parameters button.
Move to the last parameter.
Add the Scalable Flag as follows -
Parameter- P_SCALABLE_FLAG
Description - ScalableFlag
Enabled - Check
Value Set - XLA_SRS_NO_VALIDATION
Default Type - Constant
Default Value - Y
Prompt - P_SCALABLE_FLAG
Token - ScalableFlag (Case sensitive. Give this exact value)
P_SCALABLE_FLAG - has a Default Value of 'Y'
3.
A> Increase the memory for Java used by the concurrent program.
Query the concurrent program in
System Administrator> Concurrent> Program> Define >
Query for the Create Accounting concurrent program >
Enter value for Options field with : -Xmx2048M (Note: Don't miss the dash before the X) .
B> Time out parameters (Profile)
Concurrent:OPP Response Timeout : 240
Concurrent:OPP Process Timeout : 10800 sec (3 hours)
Additional Notes: Oracle Doc ID 838810.1 provides a similar solution to fix below error and it worked in my instance.
Subledger Accounting: Version : 12.0.0
XLAPEXRPT module: Subledger Period Close Exceptions Report
Calling XDO Data Engine...
****Warning!!! Due to high volume of data, got out of memory exception...***
****Please retry with scalable option or modify the Data template to run in scalable mode...***
Set the following parameters in the concurrent program definition:
Under Concurrent > Program > Define, query for concurrent program
short name = XLAPEXRPT and application = Subledger Accounting.
1. In the 'Options' field enter -Xmx2048M
2. Add a parameter to the concurrent token = ScalableFlag, default value set to 'Y'. Same way
you add DebugFlag
In the application developer responsibility
-> Concurrent -> Program -> Define
-> Query for short name= XLAPEXRPT
-> Click on Parameters
->define a new parameter (if not existing)
parameter name=P_DEBUG_FLAG, attach any valueset that is YES_NO valueset.
Give Default value=Y
Token=DebugFlag
Save
3. Submit the request with scalable mode Yes.
Monday, March 6, 2017
Shell scripts to back up and retore files for EBS refresh or clone
Some setup and configuration files works specifically for the EBS instance. It takes a long time or may be impossible to re-create them. It is are very important to back them up before the instance refresh or clone, and then restore the files after clone script completes. Below two scripts work for EBS R12.1.
~~~~~~~~~~~~~~~~~ Backup ~~~~~~~~~~~~~~~~~~
# backup files before the refresh
cd $HOME/admin_scripts/backups # where to save the backed-up files
DT=`date +"%h_%d_%y"` # backup date
echo It will delete old backup files. Continue ? yes or no
read answer
case $answer in
[yY][eE][sS]|[yY])
if [ -n "$CONTEXT_FILE" ]; then
echo 'running'
else
echo 'environment variable $CONTEXT_FILE does not exit. Exit ...'
exit 1;
fi;
if [ -f $CONTEXT_FILE ]; then
echo 'starting copy ...'
else
echo 'CONTEXT file $CONTEXT_FILE does not exist. Exit ...'
exit 1;
fi
file1=`basename $CONTEXT_FILE`
if [ -f $file1 ]; then
mv $file1 $file1'_'$DT # adding the backup date
fi
file2=custom`basename $CONTEXT_FILE .xml`'.env'
if [ -f $APPL_TOP/$file2 ]; then
cp -p $APPL_TOP/$file2 .
fi
rm -rf Apache admin # remove two previous backup folders
cp -p $CONTEXT_FILE .
cp -rp $INST_TOP/certs/Apache . # SSL certs
cp -rp $TNS_ADMIN . # TNS names
cp -p $FND_TOP/fndenv.env .
cp -p $INST_TOP/ora/*2/forms/server/default.env .
cp -p $APPL_TOP/admin/adkeystore.dat .
cp -p $APPL_TOP/admin/adsign.txt .
echo 'Done with backing up files'
ls -altr $HOME/admin_scripts/backups
exit 0
;;
[nN][oO]|[nN])
echo "No"
;;
*)
echo "Invalid input..."
exit 1
;;
esac
DT=`date +"%h_%d_%y"` # backup date
echo It will delete old backup files. Continue ? yes or no
read answer
case $answer in
[yY][eE][sS]|[yY])
if [ -n "$CONTEXT_FILE" ]; then
echo 'running'
else
echo 'environment variable $CONTEXT_FILE does not exit. Exit ...'
exit 1;
fi;
if [ -f $CONTEXT_FILE ]; then
echo 'starting copy ...'
else
echo 'CONTEXT file $CONTEXT_FILE does not exist. Exit ...'
exit 1;
fi
file1=`basename $CONTEXT_FILE`
if [ -f $file1 ]; then
mv $file1 $file1'_'$DT # adding the backup date
fi
file2=custom`basename $CONTEXT_FILE .xml`'.env'
if [ -f $APPL_TOP/$file2 ]; then
cp -p $APPL_TOP/$file2 .
fi
rm -rf Apache admin # remove two previous backup folders
cp -p $CONTEXT_FILE .
cp -rp $INST_TOP/certs/Apache . # SSL certs
cp -rp $TNS_ADMIN . # TNS names
cp -p $FND_TOP/fndenv.env .
cp -p $INST_TOP/ora/*2/forms/server/default.env .
cp -p $APPL_TOP/admin/adkeystore.dat .
cp -p $APPL_TOP/admin/adsign.txt .
echo 'Done with backing up files'
ls -altr $HOME/admin_scripts/backups
exit 0
;;
[nN][oO]|[nN])
echo "No"
;;
*)
echo "Invalid input..."
exit 1
;;
esac
~~~~~~~~~~~~~~~~~~~ Restore ~~~~~~~~~~~~~~~~~~~
#!/bin/ksh
# restore files AFTER clone script completes during the refresh
#!/bin/ksh
# restore files AFTER clone script completes during the refresh
cd $HOME/admin_scripts/backups
DT=`date +"%h_%d_%y"`
echo It will overwrite some files. Continue? yes or no
read answer
case $answer in
[yY][eE][sS]|[yY])
if [ -n "$CONTEXT_FILE" ]; then
echo 'running'
else
echo 'environment variable $CONTEXT_FILE does not exit. Exit ...'
exit 1;
fi;
file2=custom`basename $CONTEXT_FILE .xml`'.env'
if [ -f $file2 ]; then
cp -p $file2 $APPL_TOP/.
fi
mv $FND_TOP/fndenv.env $FND_TOP/fndenv.env_clone
cp -p fndenv.env $FND_TOP/.
mv $INST_TOP/ora/10.1.2/forms/server/default.env $INST_TOP/ora/10.1.2/forms/server/default.env_clone
cp -p default.env $INST_TOP/ora/10.1.2/forms/server/.
mv $INST_TOP/certs/Apache $INST_TOP/certs/Apache'_'$DT
cp -pr Apache $INST_TOP/certs/.
## files for JRE 1.8. They will be used if Java signing is re-ran.
mv $APPL_TOP/admin/adkeystore.dat $APPL_TOP/admin/adkeystore.dat_clone
cp -p adkeystore.dat $APPL_TOP/admin/.
mv $APPL_TOP/admin/adsign.txt $APPL_TOP/admin/adsign.txt_clone
cp -p adsign.txt $APPL_TOP/admin/.
echo 'Done with restoring files.'
;;
[nN][oO]|[nN])
echo "No"
;;
*)
echo "Invalid input..."
exit 1
;;
esac
DT=`date +"%h_%d_%y"`
echo It will overwrite some files. Continue? yes or no
read answer
case $answer in
[yY][eE][sS]|[yY])
if [ -n "$CONTEXT_FILE" ]; then
echo 'running'
else
echo 'environment variable $CONTEXT_FILE does not exit. Exit ...'
exit 1;
fi;
file2=custom`basename $CONTEXT_FILE .xml`'.env'
if [ -f $file2 ]; then
cp -p $file2 $APPL_TOP/.
fi
mv $FND_TOP/fndenv.env $FND_TOP/fndenv.env_clone
cp -p fndenv.env $FND_TOP/.
mv $INST_TOP/ora/10.1.2/forms/server/default.env $INST_TOP/ora/10.1.2/forms/server/default.env_clone
cp -p default.env $INST_TOP/ora/10.1.2/forms/server/.
mv $INST_TOP/certs/Apache $INST_TOP/certs/Apache'_'$DT
cp -pr Apache $INST_TOP/certs/.
## files for JRE 1.8. They will be used if Java signing is re-ran.
mv $APPL_TOP/admin/adkeystore.dat $APPL_TOP/admin/adkeystore.dat_clone
cp -p adkeystore.dat $APPL_TOP/admin/.
mv $APPL_TOP/admin/adsign.txt $APPL_TOP/admin/adsign.txt_clone
cp -p adsign.txt $APPL_TOP/admin/.
echo 'Done with restoring files.'
;;
[nN][oO]|[nN])
echo "No"
;;
*)
echo "Invalid input..."
exit 1
;;
esac
Sunday, February 19, 2017
Stop Old Outbound Workflow Notification Emails after Clone
I followed below document to cancel/purge R12.1.3 notifications:
How To Purge E-Mail Notifications From The Workflow Queue So The E-Mail Is Not Sent (Doc ID 372933.1)
It gives a query getting notifications that would be sent, and are waiting to be e-mailed when the Mailer get started:
SQL> select notification_id, begin_date, recipient_role, message_type, message_name, status, mail_status
from wf_notifications
where Status In ('OPEN', 'CANCELED')
and Mail_Status In ('MAIL', 'INVALID')
and begin_date < sysdate-30 -- List only emails older than 30 days ago
order by notification_id;
SQL> select count(*) from wf_notifications
where status in ('OPEN', 'CANCELED')
And Mail_Status In ('MAIL', 'INVALID')
order by notification_id;
COUNT(*)
----------
271022
SQL> select count(*) from wf_notifications
where status in ('OPEN', 'CANCELED')
and Mail_Status In ('MAIL', 'INVALID')
and begin_date < sysdate-30; -- List only emails older than 30 days ago
COUNT(*)
----------
270991
SQL> create table apps.wf_notifications_BK_120816 as
select * from wf_notifications
where status in ('OPEN', 'CANCELED')
and Mail_Status In ('MAIL', 'INVALID')
and begin_date < sysdate-30;
Table created.
SQL> update WF_NOTIFICATIONS set mail_status = 'SENT'
where status in ('OPEN', 'CANCELED')
and Mail_Status In ('MAIL', 'INVALID')
and begin_date < sysdate-30;
270998 rows updated.
SQL> commit;
Commit complete.
SQL> @$FND_TOP/patch/115/sql/wfntfqup.sql apps apps_PWD applsys
PL/SQL procedure successfully completed.
Elapsed: 00:00:00.76
Commit complete.
Elapsed: 00:00:00.00
**** TEMPORARY TABLES / AQs created ****
PL/SQL procedure successfully completed.
Elapsed: 00:00:00.27
**** Messages backed up ****
Commit complete.
Elapsed: 00:00:00.00
**** wfaqback.sql completed, executing SQLs from wfntfqup
PL/SQL procedure successfully completed.
Elapsed: 00:00:04.60
*** invoking wfaqrenq.sql ******
*** Re-enqueing messages
PL/SQL procedure successfully completed.
Elapsed: 00:00:00.02
Commit complete.
Elapsed: 00:00:00.01
***** Re-enqueue OF Alerts completed *****
I tried to use an email address to hold the old notifications after SMTP works fine on the CM server. Seems to me that the "Set Override Address" will not work until the Mailer was started, because prior to that, it would not send the Verification Code out to the email address.
References: Doc ID 562551.1 includes two old documents:
- How to Cancel Email Notifications for Particular Workflow Type (Doc ID 736508.1 for 11i)
- How To Stop Old Outbound Workflow Notification Email Messages During Clone Activity (Doc ID 828812.1 for 11.5.9 to 12.0.0)
How To Purge E-Mail Notifications From The Workflow Queue So The E-Mail Is Not Sent (Doc ID 372933.1)
It gives a query getting notifications that would be sent, and are waiting to be e-mailed when the Mailer get started:
SQL> select notification_id, begin_date, recipient_role, message_type, message_name, status, mail_status
from wf_notifications
where Status In ('OPEN', 'CANCELED')
and Mail_Status In ('MAIL', 'INVALID')
and begin_date < sysdate-30 -- List only emails older than 30 days ago
order by notification_id;
SQL> select count(*) from wf_notifications
where status in ('OPEN', 'CANCELED')
And Mail_Status In ('MAIL', 'INVALID')
order by notification_id;
COUNT(*)
----------
271022
SQL> select count(*) from wf_notifications
where status in ('OPEN', 'CANCELED')
and Mail_Status In ('MAIL', 'INVALID')
and begin_date < sysdate-30; -- List only emails older than 30 days ago
COUNT(*)
----------
270991
SQL> create table apps.wf_notifications_BK_120816 as
select * from wf_notifications
where status in ('OPEN', 'CANCELED')
and Mail_Status In ('MAIL', 'INVALID')
and begin_date < sysdate-30;
Table created.
SQL> update WF_NOTIFICATIONS set mail_status = 'SENT'
where status in ('OPEN', 'CANCELED')
and Mail_Status In ('MAIL', 'INVALID')
and begin_date < sysdate-30;
270998 rows updated.
SQL> commit;
Commit complete.
SQL> @$FND_TOP/patch/115/sql/wfntfqup.sql apps apps_PWD applsys
PL/SQL procedure successfully completed.
Elapsed: 00:00:00.76
Commit complete.
Elapsed: 00:00:00.00
**** TEMPORARY TABLES / AQs created ****
PL/SQL procedure successfully completed.
Elapsed: 00:00:00.27
**** Messages backed up ****
Commit complete.
Elapsed: 00:00:00.00
**** wfaqback.sql completed, executing SQLs from wfntfqup
PL/SQL procedure successfully completed.
Elapsed: 00:00:04.60
*** invoking wfaqrenq.sql ******
*** Re-enqueing messages
PL/SQL procedure successfully completed.
Elapsed: 00:00:00.02
Commit complete.
Elapsed: 00:00:00.01
***** Re-enqueue OF Alerts completed *****
I tried to use an email address to hold the old notifications after SMTP works fine on the CM server. Seems to me that the "Set Override Address" will not work until the Mailer was started, because prior to that, it would not send the Verification Code out to the email address.
References: Doc ID 562551.1 includes two old documents:
- How to Cancel Email Notifications for Particular Workflow Type (Doc ID 736508.1 for 11i)
- How To Stop Old Outbound Workflow Notification Email Messages During Clone Activity (Doc ID 828812.1 for 11.5.9 to 12.0.0)
Friday, December 2, 2016
Move EBS host from Linux 5 to Linux 6
Before Linux 5 support becomes expired, we have to upgrade it. But, RHEL5 does not have an option to "upgrade" it on the same server. We had to build a fresh new Linux 6 server and then migrate Oracle EBS R12.1.3 to the new RHEL6 server.
1. Server requirements by Oracle
Linux Admin followed Oracle E-Business Suite Installation and Upgrade Notes Release 12 (12.1.1) for Linux x86-64 (Doc ID 761566.1) to install all required components.
- list of RPMs
To make openmotif21 installation work, follow Doc ID 2057494.1 (Oracle Linux: Dependency Resolution Error "Package oracle-ebs-server-R12-preinstall requires openmotif21 and that has been obsoleted by lesstif") to delete "lesstif" package from Linux server first. Some of the installed RPMs:
$ rpm -q openmotif
package openmotif is not installed
$ rpm -q openmotif21
openmotif21-2.1.30-11.EL6.i686
$ rpm -q xorg-x11-libs-compat
xorg-x11-libs-compat-6.8.2-1.EL.33.0.1.i386
$ rpm -q libXrender
libXrender-0.9.8-2.1.el6_8.1.x86_64
libXrender-0.9.8-2.1.el6_8.1.i686
... ... ... ...
Also, download and install Oracle patch 6078836 (p6078836_101330_LINUX.ZIP) to Linux.
$ ls -al /usr/lib/libdb.so.2
-rw-r--r-- 1 root root 5825 Sep 6 21:12 /usr/lib/libdb.so.2
Without it, starting Apache will fail with error:
$IAS_ORACLE_HOME/Apache/Apache/bin/httpd: error while loading shared libraries: libdb.so.2: cannot open shared object file: No such file or directory
- maintenance tools:
ar, gcc, g++, ksh, ld, linux32, make, XDisplay Server
- kernel settings (in /ect/sysctl.conf)
- Number of Open File Descriptors (/etc/security/limits.conf)
* hard nofile 65535
* soft nofile 4096
2. Steps during the migration
2.1 Stop EBS apps clearly on old hosts and run a file counts
$ find EBS_DIR -type f | wc -l
2.2 Wait Linux Admin to rename the new server name to the "old" server name, and do a sync on file systems including user accounts, cron jobs, etc.
2.3 On the new host, make sure it connects to the right database
$ sqlpuse apps/appsPWD
SQL> select instance_name from v$instance
2.4 Run autoconfig on each nodes
$ cd $ADMIN_SCRIPTS_HOME
$ ./adautocfg.sh
2.5. To avoid "adoafmctl.sh (&adoacorectl.sh): exiting with status 150"
$ mv $ORA_CONFIG_HOME/10.1.3/j2ee/oacore/persistence $ORA_CONFIG_HOME/10.1.3/j2ee/oacore/persistence_DELETE
$ mkdir $ORA_CONFIG_HOME/10.1.3/j2ee/oacore/persistence
$ mv $ORA_CONFIG_HOME/10.1.3/j2ee/oafm/persistence $ORA_CONFIG_HOME/10.1.3/j2ee/oafm/persistence_DELETE
$ mkdir $ORA_CONFIG_HOME/10.1.3/j2ee/oafm/persistence
$ mv $ORA_CONFIG_HOME/10.1.3/j2ee/forms/persistence $ORA_CONFIG_HOME/10.1.3/j2ee/forms/persistence_DELETE
$ mkdir $ORA_CONFIG_HOME/10.1.3/j2ee/forms/persistence
If still getting "status 150", stop all services cleanly and start them again.
2.6 Open a F5 ticket to change old IP address to new one in F5.
2.7 Start apps services
With above steps in many instances I did not get any problem in starting EBS services.
2.8. Copy file libgcc_s-2.3.2-stub.so from patch 12415211 ( see Doc ID 761566.1)
3. Troubleshooting
3.1 scp and sftp failed with error:
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: POSSIBLE DNS SPOOFING DETECTED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
The RSA host key for host_name.domain.com has changed,
and the key for the according IP address 177.321.89.56
is unchanged. This could either mean that
DNS SPOOFING is happening or the IP address for the host
and its host key have changed at the same time.
Offending key for IP in /users/userID/.ssh/known_hosts:28
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
e4:5d:1c:12:e5:ab:f5:03:b4:cd:d8:cd:eb:f3:a7:23.
Please contact your system administrator.
Add correct host key in /users/userID/.ssh/known_hosts to get rid of this message.
Offending key in /users/userID/.ssh/known_hosts:7
RSA host key for host_name.domain.com has changed and you have requested strict checking.
Host key verification failed.
lost connection
Fix: $ ssh-keygen -R host_name.domain.com
3.2 Environment variable $PATH does not have the path to its home (such as /users/userID) included
Symptom: Scheduled jobs does not run because scheduler account logs in but does not "see" the script at all.
Fix: add "export PATH=$PATH:$HOME" to file .bashrc
3.3 Shell script can see env variable $HOME.
Symptom: command, such as "cd $HOME", failed.
Fix: hard code the path in the script. (not a good solution).
3.4 When a directory was removed and the old path still exists in the PATH variable, executing a file may get below misleading error. I had to remove the obsolete path from the PATH string.
$ . run_environment
-ksh: . /users/userID/run_environment: not found [No such file or directory]
3.5 RedHat RHEL6 changes mailx on using -a to email attachment. The right syntax:
$ echo "test_body" | mailx -s test-email -a test_attachment.txt emailid@youemail.com
NOTES on Linux commands:
1) Linux version
$ uname -a
Linux host_name.domain.com 2.6.32-642.3.1.el6.x86_64 #1 SMP Sun Jun 26 18:16:44 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux
$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.8 (Santiago)
2) Linux command to find CUP info
$ cat /proc/cpuinfo
$ cat /proc/cpuinfo | grep processor
processor : 0
processor : 1
3) Linux command to find total memory
$ cat /proc/meminfo
MemTotal: 20586472 kB
MemFree: 1406228 kB
Buffers: 873268 kB
Cached: 8353596 kB
SwapCached: 11332 kB
Active: 15413552 kB
Inactive: 2228904 kB
Active(anon): 7687868 kB
Inactive(anon): 729196 kB
Active(file): 7725684 kB
Inactive(file): 1499708 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 8388604 kB
SwapFree: 8288296 kB
... ...
$ free
total used free shared buffers cached
Mem: 20586472 19607488 978984 1504 876480 8510684
-/+ buffers/cache: 10220324 10366148
Swap: 8388604 100308 8288296
(note this instance has more then 350 EBS apps sessions in normal business hours)
1. Server requirements by Oracle
Linux Admin followed Oracle E-Business Suite Installation and Upgrade Notes Release 12 (12.1.1) for Linux x86-64 (Doc ID 761566.1) to install all required components.
- list of RPMs
To make openmotif21 installation work, follow Doc ID 2057494.1 (Oracle Linux: Dependency Resolution Error "Package oracle-ebs-server-R12-preinstall requires openmotif21 and that has been obsoleted by lesstif") to delete "lesstif" package from Linux server first. Some of the installed RPMs:
$ rpm -q openmotif
package openmotif is not installed
$ rpm -q openmotif21
openmotif21-2.1.30-11.EL6.i686
$ rpm -q xorg-x11-libs-compat
xorg-x11-libs-compat-6.8.2-1.EL.33.0.1.i386
$ rpm -q libXrender
libXrender-0.9.8-2.1.el6_8.1.x86_64
libXrender-0.9.8-2.1.el6_8.1.i686
... ... ... ...
Also, download and install Oracle patch 6078836 (p6078836_101330_LINUX.ZIP) to Linux.
$ ls -al /usr/lib/libdb.so.2
-rw-r--r-- 1 root root 5825 Sep 6 21:12 /usr/lib/libdb.so.2
Without it, starting Apache will fail with error:
$IAS_ORACLE_HOME/Apache/Apache/bin/httpd: error while loading shared libraries: libdb.so.2: cannot open shared object file: No such file or directory
- maintenance tools:
ar, gcc, g++, ksh, ld, linux32, make, XDisplay Server
- kernel settings (in /ect/sysctl.conf)
- Number of Open File Descriptors (/etc/security/limits.conf)
* hard nofile 65535
* soft nofile 4096
2. Steps during the migration
2.1 Stop EBS apps clearly on old hosts and run a file counts
$ find EBS_DIR -type f | wc -l
2.2 Wait Linux Admin to rename the new server name to the "old" server name, and do a sync on file systems including user accounts, cron jobs, etc.
2.3 On the new host, make sure it connects to the right database
$ sqlpuse apps/appsPWD
SQL> select instance_name from v$instance
2.4 Run autoconfig on each nodes
$ cd $ADMIN_SCRIPTS_HOME
$ ./adautocfg.sh
2.5. To avoid "adoafmctl.sh (&adoacorectl.sh): exiting with status 150"
$ mv $ORA_CONFIG_HOME/10.1.3/j2ee/oacore/persistence $ORA_CONFIG_HOME/10.1.3/j2ee/oacore/persistence_DELETE
$ mkdir $ORA_CONFIG_HOME/10.1.3/j2ee/oacore/persistence
$ mv $ORA_CONFIG_HOME/10.1.3/j2ee/oafm/persistence $ORA_CONFIG_HOME/10.1.3/j2ee/oafm/persistence_DELETE
$ mkdir $ORA_CONFIG_HOME/10.1.3/j2ee/oafm/persistence
$ mv $ORA_CONFIG_HOME/10.1.3/j2ee/forms/persistence $ORA_CONFIG_HOME/10.1.3/j2ee/forms/persistence_DELETE
$ mkdir $ORA_CONFIG_HOME/10.1.3/j2ee/forms/persistence
If still getting "status 150", stop all services cleanly and start them again.
2.6 Open a F5 ticket to change old IP address to new one in F5.
2.7 Start apps services
With above steps in many instances I did not get any problem in starting EBS services.
2.8. Copy file libgcc_s-2.3.2-stub.so from patch 12415211 ( see Doc ID 761566.1)
3. Troubleshooting
3.1 scp and sftp failed with error:
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: POSSIBLE DNS SPOOFING DETECTED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
The RSA host key for host_name.domain.com has changed,
and the key for the according IP address 177.321.89.56
is unchanged. This could either mean that
DNS SPOOFING is happening or the IP address for the host
and its host key have changed at the same time.
Offending key for IP in /users/userID/.ssh/known_hosts:28
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
e4:5d:1c:12:e5:ab:f5:03:b4:cd:d8:cd:eb:f3:a7:23.
Please contact your system administrator.
Add correct host key in /users/userID/.ssh/known_hosts to get rid of this message.
Offending key in /users/userID/.ssh/known_hosts:7
RSA host key for host_name.domain.com has changed and you have requested strict checking.
Host key verification failed.
lost connection
Fix: $ ssh-keygen -R host_name.domain.com
3.2 Environment variable $PATH does not have the path to its home (such as /users/userID) included
Symptom: Scheduled jobs does not run because scheduler account logs in but does not "see" the script at all.
Fix: add "export PATH=$PATH:$HOME" to file .bashrc
3.3 Shell script can see env variable $HOME.
Symptom: command, such as "cd $HOME", failed.
Fix: hard code the path in the script. (not a good solution).
3.4 When a directory was removed and the old path still exists in the PATH variable, executing a file may get below misleading error. I had to remove the obsolete path from the PATH string.
$ . run_environment
-ksh: . /users/userID/run_environment: not found [No such file or directory]
3.5 RedHat RHEL6 changes mailx on using -a to email attachment. The right syntax:
$ echo "test_body" | mailx -s test-email -a test_attachment.txt emailid@youemail.com
NOTES on Linux commands:
1) Linux version
$ uname -a
Linux host_name.domain.com 2.6.32-642.3.1.el6.x86_64 #1 SMP Sun Jun 26 18:16:44 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux
$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.8 (Santiago)
2) Linux command to find CUP info
$ cat /proc/cpuinfo
$ cat /proc/cpuinfo | grep processor
processor : 0
processor : 1
3) Linux command to find total memory
$ cat /proc/meminfo
MemTotal: 20586472 kB
MemFree: 1406228 kB
Buffers: 873268 kB
Cached: 8353596 kB
SwapCached: 11332 kB
Active: 15413552 kB
Inactive: 2228904 kB
Active(anon): 7687868 kB
Inactive(anon): 729196 kB
Active(file): 7725684 kB
Inactive(file): 1499708 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 8388604 kB
SwapFree: 8288296 kB
... ...
$ free
total used free shared buffers cached
Mem: 20586472 19607488 978984 1504 876480 8510684
-/+ buffers/cache: 10220324 10366148
Swap: 8388604 100308 8288296
(note this instance has more then 350 EBS apps sessions in normal business hours)
Subscribe to:
Comments (Atom)