Saturday, October 3, 2015

SQL scripts show R12 Profile Options

Below query will find the value on Profile option(s). It is modified from a script in Doc ID 1159313.1 "Unable To Authenticate Session Error When Login to Different R12 Instance in Same IE Session". Exact profile name is in table fnd_profile_options_tl .

SELECT p.profile_option_name, fpot.user_profile_option_name NAME,
decode(v.level_id,
             10001,'SITE',
             10002, (select 'App:'||a.application_short_name
                            from fnd_application a
                          where a.application_id = v.level_value),
             10003, (select 'Resp:'||f.RESPONSIBILITY_KEY
                            from fnd_responsibility f
                          where f.responsibility_id = v.level_value),
             10004, (select 'User:'||u.user_name
                            from fnd_user u
                          where u.user_id = v.level_value),
             10005, (select 'Server:'||n.node_name
                            from fnd_nodes n
                          where n.node_id = v.level_value),
             10006, (select 'Org:'|| o.name
                            from hr_operating_units o
                          where o.organization_id = v.level_value),
             'NOT SET') PROF_LEVEL,
              nvl(v.profile_option_value,'NOT SET') profile_option_value, fpot.description
FROM
  fnd_profile_options p,
  fnd_profile_option_values v,
  fnd_profile_options_tl fpot
WHERE
p.profile_option_id = v.profile_option_id (+)
and p.application_id = v.application_id (+)
and p.profile_option_name = fpot.profile_option_name
-- and p.profile_option_name = 'FND_DIAGNOSTICS'
-- and fpot.user_profile_option_name like 'FND: D%'
and fpot.user_profile_option_name like 'Signon%'
order by 1, v.level_id;

A query to find ALL profile Option values (modified from a script in Doc ID 2026081.1):

select n.user_profile_option_name NAME,
       decode(v.level_id,
               10001, 'Site',
               10002, 'Application',
               10003, 'Responsibility',
               10004, 'User',
               10005, 'Server',
               10006, 'Organization',
               10007, 'ServResp',
               'UnDef') LEVEL_SET,
       decode(to_char(v.level_id),
              '10001', '',
              '10002', app.application_short_name,
              '10003', rsp.responsibility_key,
              '10004', usr.user_name,
              '10005', svr.node_name,
              '10006', org.name,
              '10007', 'depends=',
              v.level_id) "CONTEXT",
       v.profile_option_value VALUE,
       (select n.node_name
          from
               fnd_nodes n
         where
               n.node_id=level_value2) Server,
               decode(v.LEVEL_VALUE,
               -1, 'Default',
               rsp.responsibility_key) Resp,
               decode(LEVEL_VALUE_APPLICATION_ID,
               -1, 'Default',
               app.application_short_name) Application
  from fnd_profile_options p,
       fnd_profile_option_values v,
       fnd_profile_options_tl n,
       fnd_user usr,
       fnd_application app,
       fnd_responsibility rsp,
       fnd_nodes svr,
       hr_operating_units org
 where p.profile_option_id = v.profile_option_id (+)
   and p.profile_option_name = n.profile_option_name
  -- and (upper(v.profile_option_value) like '%HTTP%')   -- if you want to find the value, enable this line
  -- and p.profile_option_name like 'APPLICATIONS_HOME%'   -- if you want to find the name
   and    usr.user_id (+) = v.level_value
   and    rsp.application_id (+) = v.level_value_application_id
   and    rsp.responsibility_id (+) = v.level_value
   and    app.application_id (+) = v.level_value
   and    svr.node_id (+) = v.level_value
   and    org.organization_id (+) = v.level_value
 order by name, level_set;

A change in Profile option can make the EBS run very differently. Below script will find Profile options that were changed within one day.

 SELECT '''' || a.user_profile_option_name
       || ''' Was Updated with value '
       || b.profile_option_value mesg,
       (SELECT user_name
             FROM apps.fnd_user u
            WHERE u.user_id = b.last_updated_by) who , b.last_update_date, b.level_value
  FROM apps.fnd_profile_options_vl a,
       apps.fnd_profile_option_values b,
       apps.fnd_user c
 WHERE a.profile_option_id = b.profile_option_id
   AND b.last_updated_by = c.user_id
   AND (   b.last_update_date > SYSDATE - 1   -- <- put a different number here if needed
        OR b.creation_date > SYSDATE - 1
       );
NOTE: Not sure why, but this query is not accurate sometimes (when the value was changed to null).

NOTES: If you have to change a Profile option but the EBS webpage is not available, try FND_PROFILE package (for example):

declare value Boolean;
begin
value := FND_PROFILE.SAVE('SIGNON_PASSWORD_CASE','1','SITE');
end;
/

Clean Java cache

Sometimes, changes to Profile options do not take effect until Apache server is bounced. This delay may be due to Java Caching.  Java caching provides a powerful, flexible, easy to use mechanism for storing database results and other Java objects in-memory for repeated usage, thereby improving application performance.

Steps for cleaning Java cache (do NOT do it in a busy production system):

1. Log into EBS with the Functional Administrator responsibility
2. Choose the Core Services Tab => Caching Framework Sub-Menu
3. Proceed to choose 'Global Configuration' from the left hand side menu that appears
4. In the far right choose 'Clear all Cache' button

Oracle documents on EBS Caching:

1. Note 759038.1 How To Clear The Cache Using Functional Administrator?
2. Note 455194.1 Diagnosing database invalidation issues with Java Cache for eBusiness Suite
3. Note 742107.1 How To Clear Caches (Apache/iAS, Modplsql, Browser, Jinitiator, Java, Portal, WebADI) for E-Business Suite?
4. Note 275879.1 Oracle Applications Java Caching Framework Developer’s Guide Release 11i

Friday, September 25, 2015

Error messages on EBS Forms

1. "XXXX is not a valid responsibility for the current user. Please contact your System Administrator."

- First possibility is that this responsibility has an End date for this user. Check user's account.

- After adding a responsibility to a user, the user may receive above message. And the attached responsibility may not appear in the Home Page until the Apache is bounced.

One solution could be to clear the cache:
Navigate to Functional administrator > Core Service > Caching Framework > Global Configuration > Clear All Cache

Another place to check:
Search for profile 'Applications Start Page'. Delete the value set at user level (if any).

Also make sure Workflow Service Components are up and running (OAM -> Workflow Manager (dropdown) -> Service Components):
   . Workflow Deferred Agent Listener
   . Workflow Java Deferred Agent Listener

2. "Function not available to this responsibility. Change responsibilities or contact System Administrator"

- If it is on a custom form, make sure its path CUSTOM_TOP is added to $INST_TOP/ora/10.1.2/forms/server/default.env file.

- Use submenu item Help --> Diagnostics --> Examine.  Then, click the "Examine Fields and variable values" to troubleshoot. Reference Doc ID 1079951.1.

- If this message shows in R12.1.3 while trying to access Help --> Diagnostics, change the profile "Utilities:Diagnostics" to Yes at user or responsibility level. Please see Doc ID 1200743.1.

3.  If  R12.1 "adformsctl.sh start" gets error, check logs:

$LOG_HOME/ora/10.1.3/j2ee/forms/forms_default_group_1/application.log
$LOG_HOME/ora/10.1.3/j2ee/oacore/oacore_default_group_1/application.log

When the Form service is starting, it checks whether latest formsapp.ear is deployed. If it finds a newer version, Form service will take longer to start and say message:
==========================================
*** Latest formsapp.ear has NOT been deployed ***
Deploying the latest EAR file...
==========================================
See DocID 1210660.1 (Forms Startup Script adformsctl.sh Shows Deployment Message While Starting Forms Services).

4.  How to deploy Forms EAR (Enterprise ARchive) file, such as formsapp.ear (under $ORACLE_HOME/forms/j2ee) ?

DocID 397174.1 (Deploying a New EAR File in Oracle Applications Release 12) gives two key steps:
1) Edit file (with a temp password) - $INST_TOP/ora/10.1.3/j2ee/forms/config/system-jazn-data.xml
2) Run deployment - $FND_TOP/bin/txkrun.pl -script=CfgOC4JApp
    (if all work, it shall say "No error encountered." at the end)

DocID 1945012.1 also gives a way to manually deploy formsapp.ear (for Oracle Forms ?? vs Oracle EBS R12). But I never try it:
$ dcmctl redeployApplication -f $ORACLE_HOME/forms/j2ee/formsapp.ear -a formsapp -co OC4J_BI_Forms

Use adctrl to stop a worker of adpatch

1.  Skip a script when it has been "Running" status for too long

During patching, a program, such as cstpostimportaad.sql, stays in "Running" status forever (Doc ID 1453612.1). To stop and skip it (and you may re-run the program after adpatch finishes):

$ adctrl
==> 4. Tell manager that a worker failed its job
==> 3. Tell worker to quit
==> 6. Restart a worker on the current machine. (Otherwise, worker keeps staying in "Assigned")

2.  A worker stuck in "Failed" status.  How to stop the worker and skip this program?

Worker  Code      Context             Filename           Status
----------  --------  ----------------------  -------------------  --------------
       3  Done   AutoPatch R120     EGOSILDU.sql   Failed

$ adctrl
==> 8    (Note: 8 is NOT listed on the menu)
 
       Now, the status becomes:
       3  Done  AutoPatch R120     EGOSILDU.sql    Skip & restart

==> 6. Restart a worker on the current machine

In another case, after I manually uploaded a failed workflow file to database (Doc ID 2581597.1), I used adcrtl to continue the adpatch session:

Worker  Code      Context             Filename           Status
----------  --------  ----------------------  -------------------  --------------
     2  Run       AutoPatch R120     POSSPAPM.wft     FAILED

==> 8
Enter the worker number(s)/range(s) or 'all' for all workers,
or press [Return] to go back to the menu : 2

Status changed to 'Skip & restart' for worker 2.
Review the messages above, then press [Return] to continue.


Once above steps in adctrl were done, adpatch automatically continued to next jobs.

3.  How to re-start "adpatch" from scratch?

Sometimes, you have to start adpatch again to clean all memories kept in previous run, such as change of a schema password in the middle of adpatch session. Even using "3. Tell worker to quit" on all workers, adpatch will still run on OS level. You have to use "kill -9" to stop it.

Sunday, September 13, 2015

Re-new SSL certificate file & How to enable SSL

In my environment, two files ewallet.p12 and cwallet.sso are saved in $INST_TOP/certs/Apache. The path is defined in ssl.conf file. Autoconfig will add cwallet.sso as the keystore variable to $CONTEXT_FILE.
 
Seems to me that certificate file works for the host name or site name. If one server hosts multiple EBS instances, same certificate file can be used for all Apache instances on the host. Certificate expires after a period of time.

Steps to renew certificate files (from .p12 file):

1. Generate the new cert file (renew the file) from certificate tool
Depending on the tool provided by CA (certificate authority) of the company for generating the certificate file, select to Renew cert for the site or host with key size of 2048 kb.
2. Download the .p12 file in my company's tool
Check Yes to include Root Chain and Private key (in PKCS12 format, depending on the tool). Also enter a password to protect the file.
3. Copy the new cert file to $INST_TOP/certs/Apache and name it ewallet.p12 (after backing up the old one).
4. Go to $INST_TOP/ora/10.1.3 directory, and source the .env file
Make sure $ORACLE_HOME point to 10.1.3 HOME ( for all infrastructure changes).
5. $ owm (run Oracle Wallet Manager after starting Exceed for XWindow).
NOTES: if run owm from 10.1.2 Oracle Home, it may give an error, such as
owm: line 155: /d2/R12/ab/apps/R1211XB9/apps/tech_st/10.1.2/jdk/jre//bin/java: No such file or directory
6. In OWM GUI, Wallet => Open,  navigate to folder $INST_TOP/certs/Apache, click on the path => Open
7. Enter the password (the one entered when re-new and download certificate file) and then click Wallet => check "Auto Login" to enable it, then => Save.

Note in February 2016:  If new certificate standard SHA2 (DigiCert) replaces SHA1 (GeoTrust) on generating new .p12 file, OWM may not open file ewallet.p12 with invalid password error. The fix is to apply patch 21845960 (October 2015 CPU Patch) to 10.1.3 ORACLE HOME (for Apache).

8. verify file cwallet.sso gets created in the same directory $INST_TOP/certs/Apache.
9. Copy the two files to other nodes of same SITE (if site uses multiple nodes for network load balance).
10. Open a new session to take regular .env file and then run autoconfig in web nodes.
11. Start apps services.  Click on the lock icon in the URL, you shall see the new date in certificate file.
12. Copy the two files to other instance(s) on the same node (if applicable).

How to enable SSL?

1. Create new certificate file for site ebssitename.domain.com by the certificate tool and get the two files ready in $INST_TOP/certs/Apache (similar to Steps 1 to 9 above).
2. Update 5 entries in Context file on web tiers: active web port, login, external url, webentryhost, webentryurlprotocol
  - active web port (s_active_webport) : 80 --> 443
    Usually F5 listens to 443, the HTTPS default, and redirects connections to real ssl port, such as 4472. For real port (without F5), makes sure the port (such as 4472) for s_active_webport and s_webssl_port in $CONTEXT_FILE are the same and matches the port in $INST_TOP/ora/10.1.3/Apache/Apache/conf/ssl.conf.
  - login (s_login_page): https://ebssitename.domain.com:s_active_webport/OA_HTML/AppsLogin
  - external url (s_external_url): https://ebssitename.domain.com:s_active_webport
  - webentryhost (s_webentryhost): ebssitename (no change)
  - webentryurlprotocol (s_webentryurlprotocol): http --> https
3. It may be necessary to modify ssl.conf in $INST_TOP/ora/10.1.3/Apache/Apache/conf to enable TLS1.x by editing file $FND_TOP/admin/template/ssl_conf_1013.tmp
4. Run auotconfig on all web nodes (this step will update Port number in httpd.conf, etc).

How to disable SSL?

1. Change above 5 variables in $CONTEXT_FILE. or
2. After shurdown apps tier, run
$ txkrun.pl -script=SetAdvCfg \-appsuser=apps -appspass=<Apps Password> -disable=SSL \-s_webport=<Web port>(Replace your Web Port Number)
3. "Running Configuration Wizards from the Command Line in Oracle E-Business Suite Release 12 [ID 1364368.1]", under "Disabling SSL" section.

Reference:
Doc ID 376700.1 - Enabling SSL or TLS in Oracle E-Business Suite Release 12

UPDATES in 2020:  After TLS1.2 is enabled (see https://erpondb.blogspot.com/2020/03/enable-tls12-in-ebs-r121.html), more steps are needed in re-newing ssl certificate because more configuration files need and use new keys from .p12 file.  I write a script to extract keys.

#!/bin/bash
# To get new files server.key and opmn.crt, from new cert file ewallet.p12, used by template files
# in $FND_TOP/admin/template/custom
# Assumpation: TLS1.2 was enabled. And, intermediate.crt & ca.crt are company-wide
#                     and are unchanged from year to year.
# Steps: 1. go to $INST_TOP/ora/10.1.3 to make 10.1.3 the ORACLE_HOME, and then modify the .p12 file by owm
#        2. place this file in $INST_TOP/certs/Apache, and then run it.
#
p12PWD='N0Pe'
echo -n "p12 cert password > "
read p12PWD
if [ $p12PWD != 'N0Pe' ]; then
 echo "$p12PWD"
 CURRPWD=$PWD
 echo $CURRPWD
echo "working on cert files ..."
# Assume two certs ca.crt and intermediate.crt are universal within the company.
# certs files
export PATH=$IAS_ORACLE_HOME/Apache/open_ssl/bin:$PATH
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$IAS_ORACLE_HOME/lib
export OPENSSL_CONF=$IAS_ORACLE_HOME/Apache/open_ssl/bin/openssl.cnf
cd $IAS_ORACLE_HOME/Apache/open_ssl/bin
chmod 755 openssl
which openssl
cd $INST_TOP/certs/Apache
if [ -f 'jyy.key' ]; then
   mv jyy.key jyy.key_OLD
fi
if [ -f 'server.key' ]; then
   mv server.key server.key_OLD
fi
if [ -f 'server.crt' ]; then
   mv server.crt server.crt_OLD
fi
if [ -f 'opmn.crt' ]; then
   mv opmn.crt opmn.crt_OLD
fi
if [ -f 'allcerts.jyy' ]; then
   mv allcerts.jyy allcerts.jyy_OLD
fi

openssl pkcs12 -in ewallet.p12 -out jyy.key -nocerts -passout pass:'change1t' -password pass:$p12PWD
openssl rsa -in jyy.key -out server.key -passin pass:'change1t'
# below line assumes the SERVER cert is the 1st position in ewallet.p12.
# If it is not, need to manually copy it from allcerts.jyy by next command line.
openssl pkcs12 -in ewallet.p12 -clcerts -nokeys -password pass:$p12PWD | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p;/-END CERTIFICATE-/q' > server.crt
# Use this to get details of 3 certs. Note allcerts.jyy is just for
# troubleshooting/comparison purpose when needed.
openssl pkcs12 -in ewallet.p12 -out allcerts.jyy -nokeys -password pass:$p12PWD
#
cat server.crt intermediate.crt ca.crt > opmn.crt
#
# Not sure if the order in below output file is always correct
openssl pkcs12 -in ewallet.p12 -clcerts -nokeys -password pass:$p12PWD | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > opmn.crt_2nd
# opmn.crt and opmn.crt_2nd shall be identical except the key order in the files
# echo "Use below line to check difference. should have nothing"
# diff opmn.crt opmn.cer_2nd
ls -altr
echo $INST_TOP/certs/Apache
cd $CURRPWD
else
  echo "password seems wrong."
  exit 1
fi

UPDATES:
I tried to use orapki to get server.crt file from ewallet.p12 (in R12.1.3). It was very close to make it work.

$ cd $INST_TOP/ora/10.1.3
$ ls
$ . $CONTEXT_NAME.env
$ which orapki             <= 10.1.3
$ cd /path/to/Apache
$ mv server.crt server.crt_BK_good
$ orapki wallet display -wallet /path/to/Apache/ewallet.p12
Enter wallet password:   xxxxxx
Requested Certificates:
Subject:        CN=site_name.domian.com,O=company_name,L=city,ST=AZ,C=US
User Certificates:
Trusted Certificates:
Subject:        CN=site_name.domian.com,O=company_name,L=city,ST=AZ,C=US
Subject:        CN=company_name Secure CA2,O=company_name,C=US
Subject:        CN=DigiCert High Assurance EV Root CA,OU=www.digicert.com,O=DigiCert Inc,C=US
$ orapki wallet export -wallet /path/to/Apache/ewallet.p12 -dn 'CN=site_name.domian.com,O=company_name,L=city,ST=AZ,C=US' -cert server.crt
$ chmod +r server.crt
$ more server.crt

$ diff server.crt server.crt_BK_good
... ...                                                  <== the length of lines is different
\ No newline at end of file                <== also this message

I used openssl to view their contents:

$ export PATH=$IAS_ORACLE_HOME/Apache/open_ssl/bin:$PATH
$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$IAS_ORACLE_HOME/lib
$ export OPENSSL_CONF=$IAS_ORACLE_HOME/Apache/open_ssl/bin/openssl.cnf
$ which openssl
~/$TWO_TASK/tech_st/10.1.3/Apache/open_ssl/bin/openssl
$ openssl version
OpenSSL 1.0.2q  20 Nov 2018

$ openssl x509 -in server.crt -noout -text  > server.txt
$ openssl x509 -in server.crt_BK_good -noout -text > server.txt_good
$ diff  server.txt server.txt_good    <== two text files are identical!

$ openssl x509 -noout -modulus -in server.crt | openssl md5
(stdin)= 637615c9fcc4c8945817b4941bbff708
$ openssl rsa -noout -modulus -in server.key | openssl md5
(stdin)= 637615c9fcc4c8945817b4941bbff708 

But when using this server.crt to start Apache, it gets error (which is misleading). :

$ ./adapcctl.sh start
You are running adapcctl.sh version 120.7.12010000.2

Starting OPMN managed Oracle HTTP Server (OHS) instance ...
sslSecureInit: SSL_CTX_use_certificate_chain_file($INST_TOP/certs/Apache/server.key): failed
4148582144:error:0906D066:PEM routines:PEM_read_bio:bad end line:pem_lib.c:805:
4148582144:error:140DC009:SSL routines:SSL_CTX_use_certificate_chain_file:PEM lib:ssl_rsa.c:708:
opmnctl: opmn start failed.
opmnctl: opmn is not running.

adapcctl.sh: exiting with status 0

The errors match Doc ID 2721500.1 (OPMN Start Fails with Error : SslSecureInit: SSL_CTX_use_certificate_chain_file Failed). But the solutions do not work. It may have a format or "new line" issue. Or, orapki only works well after version 11.1.1.7.0 (in R12.2)