Modified 21-APR-2010 Type BULLETIN Status ARCHIVED
PURPOSE
-------
This is a comprehensive guide aimed to explain the interlinked working
architecture of different files under $IAS_ORACLE_HOME/Apache directory and also
suggestions to follow when things are not working as they should.
Pre-Amble
---------
For easy navigation and reference let us first set the following variables in
the environment of the user that owns the Apache directory structure:
Variable Physical Path
COMMON_TOP ..../[sid]comn
APACHE_TOP $IAS_TOP/Apache
SCRIPT_TOP $COMMON_TOP/admin/scripts
Note: The environment variables will not be available to the Apache listener
runtime. In addition, you should not add these variables to the Applications
environment file. Keep your environment for administering; starting and stopping
Apache free of the Applications environment variables to avoid potential problems.
Startup
=======
The core of the Technology Stack is built around Apache Web Server.
So let that be our starting point.
First thing to be determined is which file Apache is using httpd.conf or httpds.conf? to
get its startup information.
This can be determined by examining the script $APACHE_TOP/Apache/bin/apachectl
which is ultimately called by the script $SCRIPT_TOP/adapcctl.sh when you start
and stop the web server.
adapcctl.sh => apachectl
In the script apachectl, there is a variable HTTPD defined points to either the
httpd or the httpds executable.
Example: $ grep HTTPD= apachectl
HTTPD=/apps/oracle/visora/iAS/Apache/Apache/bin/httpd
If the HTTPD variable in the apachectl script calls httpd, Apache configuration
depends on httpd.conf. If the apachectl script calls httpds, Apache configuration
depends on httpds.conf.
For Release 11i Applications, the Apache Web Server depends on the full
JDK 1.3 to be installed under (or symbolically linked from) $APACHE_TOP/jdk.
The latest rapidWiz (11.5.10) installs JDK 1.4 on its own.
Note: The following type of error will occur if you have pointed $APACHE_TOP/jdk
to a directory containing only the JRE, and try to execute a jsp through browser:
JSP Error:
Request URI:/html/jtflogin.jsp
Exception: java.lang.NoClassDefFoundError:sun/tools/javac/Main
If the installation process for your platform installs the JDK to a predefined
directory, symbolically link $APACHE_TOP/jdk to the actual location.
Here is an example (assuming that the JDK installed to /usr/local/jdk118_v1):
$ln -s /usr/local/jdk118_v1 $APACHE_TOP/jdk
To determine the version of the JDK that is in use by Apache,
use the following script as noted in Note:309769.1 (not AIX specific):
sh -c "`awk -F= '$1 ~ /^JSERVJAVA.*$/ {print $2}' \
$ORACLE_HOME/../iAS/Apache/Apache/bin/java.sh` -version;"
Files of Importance
===================
The files that are involved in making Apache work with JServ are the following:
I $APACHE_TOP/Apache/conf/httpd.conf
(or httpds.conf, depending on your platform)
II $APACHE_TOP/Jserv/etc/jserv.conf
III $APACHE_TOP/Jserv/etc/jserv.properties
IV $APACHE_TOP/Jserv/etc/zone.properties
V $APACHE_TOP/Apache/conf/oracle_apache.conf
VI $APACHE_TOP/Ojsp/conf/ojsp.conf
IMPORTANT NOTE
-.-.-.-.-.-.-.
All these files are controlled by AutoConfig and it is not supported to make
manual changes to these files. If there are any changes that you require for
your system to function that cannot be made using AutoConfig then raise a
Service Request with Oracle Support
I httpd(s).conf ( "master" file for Apache )
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Observe the include lines for the files jserv.conf and oracle_apache.conf.
# Include the configuration files needed for jserv
include "/u08/app/oracle/viscomn/util/apache/1.3.9/Apache/Jserv/etc/jserv.conf"
# Include the Oracle configuration file for custom settings
include "/u08/app/oracle/viscomn/util/apache/1.3.9/Apache/Apache/conf/oracle_apache.conf"
The file ojsp.conf is indirectly referenced for Apache's configuration using the
following line in oracle_apache.conf:
include "/u08/app/oracle/viscomn/util/apache/1.3.9/Apache/Ojsp/conf/ojsp.conf"
A line that begins with the word Alias maps a directory name viewable on the
browser URL to a physical directory on the file system. Alias directives must
have a trailing slash for both virtual and physical paths.
Check to make sure the trailing slashes are there. The aliases will not work
without them. The following aliases must be defined.
Some are redundant, but must be defined nonetheless:
Alias Physical Path
/OA_HTML/ $OA_HTML
/OA_MEDIA/ $OA_MEDIA
/OA_JAVA/ $OA_JAVA
/html/ $OA_HTML
/media/ $OA_MEDIA
The DocumentRoot directive defines the directory where the Apache listener will
initially take users when no virtual path is specified in the URL. This should
be defined as your $COMMON_TOP/portal directory.
Just below the DocumentRoot directive, the default directory permissions are
defined. Make sure to also check that the line that allows users to view the
DocumentRoot directory is set to $COMMON_TOP/portal.
DocumentRoot "[$COMMON_TOP]/portal"
Options FollowSymLinks
AllowOverride None
DocumentRoot "/apps/oracle/viscomn/portal/VIS_incq093r"
httpd.conf ----> oracle_apache.conf => ojsp.conf
----> Jserv.conf => jserv.properties => zone.properties
Make sure ServerName is the fully qualified host name.
$ grep ServerName httpd.conf
ServerName myHost.oracle.com
The following portion shows that Apache is listening through port 3333 for
oprocmgr .(From FND.H there is a separate oprocmgr.conf file).
Here it denies the access from anywhere except if you are in the same host.
So, comment out the line: Deny from all to make it accessible from anywhere.
Listen 3333
ProcNode myHost.oracle.com 3333
Port 3333
Order Deny,Allow
#Deny from all
Allow from localhost
Allow from sweetnsour.idc.oracle.com
Check the way the /etc/hosts file is set up.
It is required that you have your machine.domain (Fully Qualified Domain Name)
name before the entry with just the hostname.
i.e. If it is:
127.0.0.1 localhost
168.142.16.5 myHost myHost.oracle.com loghost
Then change it to:
127.0.0.1 localhost
168.142.16.5 myHost.oracle.com myHost loghost
Trouble Shooting Checks
=======================
1. Are their any existing Apache processes?
If so, shut them down or use a kill command to remove them.
2. Test for existing processes: Unix: ps ef | grep Apache
3. Check for Port Conflicts: $IAS_TOP/Apache/Apache/conf/
In Files httpd.conf httpd_pls.conf (Port)
mod_oprocmgr (Listen / Port )
$IAS_TOP/Jserv/etc/jserv.properties
4. In case you are using the same Port Number for Jserv as the one you are
using for Apache then this will cause a Port Conflict on the Web Server.
The Jserv must use a different Port that is not used on the Server.
(The port for Jserv is the one that Jserv will use to comunicate with the
Apache Server, so it must be different than the one that Apache uses for the Web Pages)
For example Change the line :
# Set the port Apache JServ listens to.
# Syntax: port=[1024,65535] (int)
# Default: 8007
port=2030
Using AutoConfig Change the port=2030 for a Port NOT used on the Server
Save jserv.properties -> Stop All the applications processes
Start ONLY Apache using : adapcctl.sh start
5. In case Apache is not starting up Disable GSM /OAM in the following way
mv $FND_TOP/bin/FNDSVCRG $FND_TOP/bin/FNDSVCRG_ori
6. Try to Restart Apache using adapcctl.sh in $OAD_TOP/admin/scripts/
Check if it starts without using the GSM? If yes that makes sure this is not a WebServer issue. Pass the tar to 99/GSM.
7. Apply Patch 3351007 or comment out the rotatelogs (transferlog) directives.
Also uncomment the following line as well
#CustomLog /usr/oracle/visora/iAS/Apache/Apache/logs/access_log common
8. Check whether you are no on iAS 1.0.2.2.2? If yes have you applied iAS rollup 4?
(How to check, “httpd –v”)
A rare error on error_log is :
script not found or unable to stat:
/opt/oracle/product/OAD2/806/tools/web60/cgi/jinit
This indicates that the Apache web server has not even gotten to the part of
actually calling any Jinitiator. So troubleshoot Apache not Jinitiator.
Also check that the cgi directory has only f60cgi and rwcgi60.
Performance Tuning
===================
Now that you have taken care to startup Apache tune it up so that processes are
not started up or shutdown heavily. The load on the web server governs this
tuning. one commonly practised rule of thumb is set :
MaxRequestsPerChild = high number [This is set to 0 (unlimited) by default]
MinSpareServers = low number (e.g. 0 or 1)
This configuration ensures that Apache processes are very rarely shutdown,
actually only when they reach the MaxRequestsPerChild value.
MaxClients can be set to a maximum of 1024 in iAS 1.0.2
Followings are some detail possibilities for tuning
1. Limit the number of processes (HTTPD on Unix; THREAD on NT) to avoid spawning
too many HTTPD processes (which is memory consuming).
2. Set the following directive in the Oracle HTTP Listener configuration file
httpd.conf: KeepAlive Off
If you must use KeepAlive On (for example, for another application), make sure
that KeepAliveTimeout is set to a low number (for example, 15 seconds, which
is the default).
3. Set the maxClient directive to a high value.
The best is to let the HTTP Listener control to create more HTTPD daemons.
Therefore set the maxClient directive to a high value in the configuration
file (httpd.conf). However, you need to consider the memory available on the
system when setting this parameter.
MaxClient=512(by default) means that the listener can create up to 512 HTTPD
processes to handle concurrent requests.
4. MinSpareServers / MaxSpareServers
If your HTTP requests come in bursts, and you want to reduce the time to start
the necessary HTTPD processes, you can set MinSpareServers and MaxSpareServers
within httpd.conf) to have an appropriate number of processes ready.
However, the default values of 5 and 10 respectively are sufficient for most sites.
5. The MaxRequestsPerChild directive sets the limit on the number of requests that
an individual child server process will handle. After MaxRequestsPerChild
requests, the child process will die.
If MaxRequestsPerChild is 0, then the process will never expire.
This directive sets the maximum configured value for ThreadsPerChild for the
lifetime of the Apache process. Any attempts to change this directive during a
restart will be ignored, but ThreadsPerChild can be modified during a
restart up to the value of this directive.
6. ThreadsPerChild
This directive sets the number of threads created by each child process.
The child creates these threads at startup and never creates more.
II $APACHE_TOP/Jserv/etc/jserv.conf
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The file jserv.conf references jserv.properties using the ApJServProperties
directive. The file should exist in the same directory ($APACHE_TOP/Jserv/etc)
as jserv.conf. Make sure the path is defined correctly and the file exists.
ApJServProperties /u08/app/oracle/viscomn/util/apache/1.3.9/Apache/Jserv/etc/jserv.properties
jserv.conf => jserv.properties
Trouble Shooting Checks
========================
Check if you are hitting the following issue documented in Oracle 9iAS release
notes Release 1 (v1.0.2.2.2) for Sun SPARC Solaris Part Number A95823-01
1.5.3.2.20 HTTP Server Infrastructure Watcher May Suspend JVM
The infrastructure contains a watcher component that monitors the automatically
started JServ processes. If a JServ process stops responding, the watcher terminates
it and starts another process to replace it.
If the JVM is heavily loaded or performing garbage collection, the JVM will also
be suspended.
To prevent the JVM from terminating, increase its timeout value (ApJServVMTimeout)
in the jserv.conf file, or decrease the JVM heap size to reduce garbage collection time.
Default is 90 seconds timeout and 3 retries. If that is not enough,
you have got some other problem that needs to be sorted.
To address this issue you need to first determine if Apache or Jserv is hanging
since in some cases Apache maybe the problem and not the JVM.
The easiest way to do that is to get thread dumps for the JVM when the hang happens.
This will show if the JVM is the problem and why.
Another test you can do during the hang is to access a static html page
(not cached yet in browser, you can cleanup your cache to make sure of that)
and a non OA based jsp page (such as isItWorking servlet).
III $APACHE_TOP/Jserv/etc/jserv.properties
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This file is organized in following 3 sections:
1: Execution Parameters 2: Servlet Zones parameters
3: CZ Mem & URL Parameters
Find the line that starts with wrapper.bin.
This should point to the full path to the java executable from the JDK 1.1.8
which you ensured was properly installed earlier. This should point to
$APACHE_TOP/jdk/bin/java.sh
wrapper.bin=/u08/app/oracle/viscomn/util/apache/1.3.9/Apache/jdk/bin/java
Go to the "CZ Mem & URL Parameters" section.
There are 3 that need to be added at the end of this section:
wrapper.bin.parameters=-DJTFDBCFILE=/u08/app/oracle/visappl/fnd/11.5.0/secure/otclnx1_vis.dbc
wrapper.bin.parameters=-Dservice.Logging.common.filename=/tmp/ibe.log
wrapper.bin.parameters=-Dframework.Logging.system.filename=/tmp/fwsys.log
Go to the "Servlet Zones parameters" section.
jserv.properties => zone.properties
The file jserv.properties references zone.properties using the root.properties directive.
Check the entry for bindaddress:
# Syntax: bindaddress=[ipaddress] or [localhost] or [*]
# Default: localhost
bindaddress=myHost.oracle.com
If you notice that the bindaddress is set to the IP address or the hostname.domain
of the host machine, try replacing it with localhost:
For example: bindaddress=localhost
In order for this to work, there must be an entry in the /etc/hosts file for
that machine:
127.0.0.1 localhost loghost
Trouble Shooting Checks
=======================
Where to look when the going gets Tough
-- Apache is quite verbose in logging errors when they occur.
While Apache is down, rename or delete the following files so that you can
troubleshoot with clean copies to check the log files for the cause of the
problem.
$APACHE_TOP/Apache/logs/error_log
$APACHE_TOP/Apache/logs/access_log (not necessary to rename if rotatelogs is being used)
$APACHE_TOP/Jserv/logs/mod_jserv.log
$APACHE_TOP/Jserv/logs/jserv.log
After that Edit the httpd.conf/httpds.conf file to enable debug for apache.
Set the "LogLevel" parameter to debug
Edit the $IAS_TOP/Apache/Jserv/etc/jserv.conf to enable debug for the jserv module.
Set the "ApJServLogLevel" parameter to debug
Ensure that entry log=true is present in jserv.properties.
Check that the necessary libraries of class files exist on your server:
$APACHE_TOP/Jserv/libexec/ApacheJServ.jar
$APACHE_TOP/Jserv/libexec/ApacheJServ.jar
$APACHE_TOP/Jsdk/lib/jsdk.jar
$APACHE_TOP/Ojsp/lib/ojsp.jar
$APACHE_TOP/Ojsp/lib/servlet.jar
$APACHE_TOP/Ojsp/lib/xmlparserv2.jar
$APACHE_TOP/Ojsp/lib/translator.zip
$JAVA_TOP/apps.zip
$JAVA_TOP/jdbc111.zip
Check that the .dbc file in $FND_TOP/secure specifies that the THIN JDBC driver
should be used by Oracle Applications. The GUEST_USER_PWD should be a valid
Applications username and password.
Open the file in a text editor and verify that the following lines appear in it:
#DB Settings
#Sun Aug 06 10:31:21 EDT 2000
APPS_JDBC_DRIVER_TYPE=THIN
APPL_SERVER_ID=6E2ECF457C39DE9FE030028AE99E70823172354937397097845740928337FA42
TWO_TASK=VIS
GUEST_USER_PWD=GUEST/GUEST
DB_HOST=myHost.oracle.com
FNDNAM=apps
GWYUID=APPLSYSPUB/PUB
DB_PORT=11523
Keep in mind that JVMs often could not start in the first place due to the
failure to bind to socket. In such a scenario the OPM (Oracle Procedure Manager)
cannot find any of them. Since they are all dead, Apache also cannot mount the
oa_servlets directory to the JServ module and therefore it tries to interpret
/oa_servlets/ as a virtual directory (which it is not).
As a result client browser request cannot find Jsp pages when actually there is
nothing wrong with any of them.
If you observe in log file Jserv is not being able to create a socket on any of
the ports selected so far be sure every other problem in the logs is based upon
this initial one. Most of the time this may be a problem with the clone,
but is more likely a problem at the operating system level.
Always check does "netstat -a" show that ports range specified in
jserv.properties available? You can also try to expand this range of ports and
let the system pick through them one by one until you find an open port yourself.
Verify the JDK Settings for SUN following Note: 130091.1
Set jserv.properties parameter security.maxConnections higher than 256
** If you still have a issue at hand suggest customers to apply the following
patches.
a> Apply AOL/J Function Security Patch 3119955
Confirm by doing:
SQL> select * from ad_bugs where bug_number='3119955';
b> Apply TXK D for JDBC Drivers.
Confirm by doing:
SQL> select * from ad_bugs where bug_number='2902755';
IV $APACHE_TOP/Jserv/etc/zone.properties
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Here the repositories directive defines a comma-delimited list of the locations of
class files Jserv considers as "root". By default, the directive should point to
the $APACHE_TOP/jserv/servlets directory (which contains classes to test Jserv).
repositories=/apps/oracle/visora/iAS/Apache/Jserv/servlets,/apps/oracle/viscomn/java/apps.zip,/apps/oracle/viscomn/java
servlet.oracle.jsp.JspServlet.initArgs=translate_params=true,developer_mode=false,page_repository_root=/apps/oracle/viscomn/_pages
zone.properties <-- Jserv/servlets + apps.zip + _pages Points to Remember ================== + The page_repository_root is an alternative root directory for the Oracle JSP container to use in loading and generating JSP pages as long as it is not set as one of the wrapper.classpath entry in jserv.properties. + The System Class Path is shared by all servlets zones and is defined using the "wrapper.classpath" directive in the jserv.properties file when JServ startup mode is set to either automatic or auto. + Due to performance issues, class reloading is non-functional. Nevertheless, that's one of the reasons why Apache Jserv has been deprecated by the Apache foundation in favour of tomcat. There is no way Oracle should recommend class reloading in a production environment since the performance constraints that will introduce, might render the application unusable. + If you are working on a customization and really need class reloading (which is not good and I will explain why later on), it would be a good idea to use either Tomcat as a standalone development environment, or better yet, use Jdeveloper with the OA extensions that will mimic most of the functionality found in java layer on Oracle Applications. + Classes defined in the System Class Path are not automatically reloaded by JServ when they are modified and JServ needs to be restarted in order for new classes to be loaded. On the other hand, classes defined in the Servlet Zone Class Path are automatically loaded by the Zone Class Loader when they are modified. + The repositories setting is for the classes you want to be reloaded upon modification. + Classes must be in the JSP classpath, not the system classpath, to be dynamically reloaded. If they are in the system classpath as well, the system class loader may take precedence in some circumstances, possibly interfering with JSP automatic-reloading functionality" + If there are classes you want to be loaded by the Oracle JSP class loader instead of the system class loader, then use the classpath configuration parameter, or place the classes in the Oracle JSP default classpath. From Note 211918.1 that says the following: Usually in normal servlets you can take advantage of class/es autoreloading by specifying those classes in the repositories section (in zone.properties) and setting autoreload.classes property to true. But Configurator servlet can not take advantage of this autoreload mechanism because of the Configurator engine library. Because of this dependency, servlets does not know how to autoreload the modified classes. So, the only option is to bounce the Apache to see the changes in Functional companion. And currently, there is no way to bounce the Apache from the web browser. Add "unsafe_reload=true" to the initArgs, like below servlet.oracle.jsp.JspServlet.initArgs=translate_params=true,developer_mode=true, unsafe_reload=true, page_repository_root=/apps/testora/iAS/testcache with the above settings actually JSP pages will get refreshed without relogin after modifying them. But still for class modification, you will have to bounce the Apache Web Server + Class reloading is not a good idea on apache jserv because there is no "concept" of a Web Application. Class reloading occurs in any J2EE java servlet/jsp container where the classes pertaining to a particular web application are reloaded via the container where the application as a whole is reloaded. In our deployment for Oracle Applications, we didn't have any other choice (OC4J was a promise back then) than going with apache jserv, so everything is deployed as a big application. + Individual class reloading might cause several issues (heap corruption, memory leaks) in a massive java application as Oracle Apps. Every time a class is reloaded, all the instantiated classes need to be reloaded as well (jdbc connection pool,session, cache engine) and should not be recommended under any circumstance. From an official Oracle's standpoint, extending Oracle Applications should be done using Framework methodologies (see www-apps.us.oracle.com/fwk/), which implies using jdeveloper as a development environment. Anything else should be considered a hack. V $APACHE_TOP/Apache/conf/oracle_apache.conf ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Note: On some platforms the oracle_apache.conf file may be missing. The oracle_apache.conf file is included in httpd.conf or httpds.conf for the sole purpose of referencing additional configuration files that are Oracle-specific. In addition to the configuration file for Oracle jsp's it will be used for configuring Oracle's implementation of xml and plsql. Check/define include line for the files ojsp.conf include "[$APACHE_TOP]/Ojsp/conf/ojsp.conf" include "/u08/app/oracle/viscomn/util/apache/1.3.9/Apache/modplsql/cfg/plsql.conf" include "/u08/app/oracle/viscomn/util/apache/1.3.9/Apache/Ojsp/conf/ojsp.conf" include "/u08/app/oracle/viscomn/util/apache/1.3.9/Apache/xdk/admin/xml.conf" oracle_apache.conf --> plsql.conf
--> ojsp.conf
--> xml.conf
VI $APACHE_TOP/Ojsp/conf/ojsp.conf
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Note: On some platforms the conf subdirectory of $APACHE_TOP/Ojsp and the
ojsp.conf file may be missing.
The 2 types of directives in this file are Alias directives and ApJServAction
directives. The ApJservAction directives map a file extension to a class file
that defines how Jserv will execute it.
Check/define the Alias for jspsamples and jspdocs.
Make sure that the trailing slash appears for both the alias and the physical
directory.
# Set aliases for Oracle JSP
Alias /jspdocs/ "/u08/app/oracle/viscomn/util/apache/1.3.9/Apache/Ojsp/"
Alias /jspsample08/app/oracle/viscomn/util/apache/1.3.9/Apache/Ojsp/webroot/"
Check/define the ApJServAction directives for .jsp and .sqljsp files
# Associate OJSP extension with OJSP class to be called
ApJServAction .jsp /servlets/oracle.jsp.JspServlet
ApJServAction .sqljsp /servlets/oracle.jsp.JspServlet
Comprehensive Performance Tuning for the Entite Middle tier
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Whenever you are into some performance tuning issue with respect to Applications
find out the following information :
+ What version of Apache are you running?
Determine the current Apache version and patch level as follows:
%IAS_ORACLE_HOME%\Apache\Apache\Apache.exe -version
+ Check whether the database can still be accessed. (SQL*Plus)
( Confirms that number of connections has been exceeded. )
+ Implement Dead Connection Detection (DCD) in the following manner.
On the server side add in SQLNET.ORA:
SQLNET.EXPIRE_TIME= <# of minutes>
# of minutes is the time between successive probe packets sent from server to
the client to check if the connection is still alive.
( A time of 10 minutes is probably optimum for most applications. )
+ Check if stopping the concurrent manager makes a difference.
+ Following statements will give you a fair idea about the sessions in use
set linesize 132
set pagesize 100
col machine format a15
col OSuser format a12
col program format a30
SQL> select count(*) from v$session;
SQL> select count(*) from v$session where status='INACTIVE';
SQL> select count(*) from v$session where status='ACTIVE';
SQL> select machine, osuser, program, count(*) from v$session
group by machine, osuser, program order by 4 desc;
SQL> select count(status) Count, status, machine, program from v$session
where program like '%JDBC%' group by status, machine, program;
SQL> select count(status) Count, status, machine, module from v$session
where program = 'JDBC Thin Client' group by status, machine, module;
SQL> SELECT serial#, substr(program,1,20) program, status, to_char(logon_time,'DD-MON-YY HH24:SS') Login_Time,
to_char(sysdate-last_call_et/86400,'DD-MON-YY HH24:SS') Last_Activity FROM v$session
WHERE program like 'JDBC%' order by 4;
+ If you are particularly concerned about the inactive sessions, keep in mind there
will always be a certain number of Inactive sessions. They will get recreated
once Apache is bounced. Now the forms processes are a different matter, and
should be getting cleaned up, especially if they are abnormally terminated
forms sessions. If this is not working, this could go back to the current
forms version you are on.
+ Apache version 1.3.12 is notorious for not cleaning up after itself.
The recommendation is always to upgrade Apache to 1.3.19 (9iAS 1.0.2.2.2) per
Note.146468.1 at the earliest opportunity.
Between upgrading Apache and upgrading the JDBC driver, the latter would obviously
be of least impact to the system, but there is no guarantee that the latest JDBC
driver alone will resolve any issue with inactive sessions.
+ How many JVM's have you defined?
+ Have your appied iAS rollup patch 4 { 3072811}?
Development strongly recommends applying 3072811 as it is the latest iAS rollup patch.
+ What modules are you using in applications?
+ If you are not using ICSM, please check that parameter s_icsmstatus = 'disabled' in
your Applications Context (XML) file for Autoconfig.
+ What the profile option "Self Service Personal Home Page mode" is set to when
you get performance issues. In a general sense, the performance of the framework
style interface is much better than the PHP (plsql based) version. Most of all
development going forward is tied to framework and development had considered
desupporting PHP until customer demand insisted on keeping it.
The old sytle is desupported from 11.5.10 CU2 and has been strongly
recommended since FND.G
+ Have you applied the latest TXK patch at the time of writing is
4104924 TXK (FND) AUTOCONFIG TEMPLATE ROLLUP PATCH K (July 2005) for Autoconfig
and roll up patch 5 for 9iAS 1.0.2.2.2.
+ Have you applied Patch 3902820 - New CONSOLIDATED FIXES - POST AOLJ PATCH 3313291
+ When you bounce the Apache server, does the memory consumption continue to increase
from that point?
+ Development has stated if performance problems are shown with Jserv processes
in 11i, JDK 1.4 should be implement. The JDK is the engine for the Jserv.
The steps also move the JDBC driver to the current version. It would recommend
to test the JDK 1.4. (246105.1)
Secondly, if you run the following command, it will give you the number of
JDBC connections you have on the instance.
select program, count(*) from v$session group by program;
+ When memory usage is high, what is the result of the following query?:
select count(*) from v$session where program like 'JDBC%';
The above output may show the memory is being consumed by large counts of JDBC
connections. The JDBC Pool in e-Business Suite is implemented through the core
java technology framework for the entire suite (named AOL/J). AOL/J uses the
dbc file in order to obtain the parameters needed to create a database connection.
AOL/J technology uses JDBC, mostly and more intensively, in web applications.
When starting iAS HTTPD it launches the ApacheJserv engine.
During the startup of ApacheJerv is when the pool is built, based on the
parameters specified by the DBC file.
When using the default settings in a heavily used system, it is possible to
encounter situations where the size of the pool grows, leaving the database
running out of processes or out of connections.
+ If this condition surfaces, the best way to deal with this situation is to
re-configure the AOL/J database connection pool parameters.
The values in the DBC file will override any values set in SSP_INIT.txt,
but you should not put these values in ssp_init.txt anymore. The file
'ssp_init.txt' is now only used for iProcurement specific settings
I would like to expound on the explanation above to clear the doubt on how the
JDBC pool monitor will behalve with the following parameters set:
FND_MAX_JDBC_CONNECTIONS=500
FND_JDBC_BUFFER_MIN=5
FND_JDBC_BUFFER_MAX=25%
FND_JDBC_BUFFER_DECAY_INTERVAL=60
FND_JDBC_BUFFER_DECAY_SIZE=1
FND_JDBC_USABLE_CHECK=true
FND_JDBC_CONTEXT_CHECK=true
FND_JDBC_PLSQL_RESET=false
Note We are changing the default values for the parameter FND_JDBC_MAX_CONNECTIONS
to 100 in Bug#3186367.
When apache starts, it will create a buffer of 5 JDBC connections (FND_JDBC_BUFFER_MIN=5)
plus a few more as java dependant products initialize. As users login, those 5
connections will get used and more connections will be created to replenish the
buffer. These connections will increase as needed up to the limit of 500
(FND_MAX_JDBC_CONNECTIONS), which is probably too high for user load to hit.
If the buffer decay interval is too high then the pool cleanup thread will not
kick in often enough. The default value is 60 seconds. If the interval is too
high and the decay size is too small then insufficient number of connections
will be cleaned each time the cleanup thread sweeps the pool.
The default for decay size is 1.
You may want to up the decay size and lower the interval to make the pool cleanup
thread more aggressive. I suggest you the following:
FND_JDBC_BUFFER_DECAY_INTERVAL = 60
FND_JDBC_BUFFER_DECAY_SIZE = 1
As users log off, the JDBC connections will die off at a rate of 1 every 60 seconds
(FND_JDBC_BUFFER_DECAY_SIZE=1, FND_JDBC_BUFFER_DECAY_INTERVAL=60), but this
decay won't start until you have reached 125 JDBC connections (25% of 500 where
FND_JDBC_BUFFER_MAX=25% and FND_MAX_JDBC_CONNECTIONS=500).
The decay will slowly reduce the number of inactive JDBC connections down to
FND_JDBC_BUFFER_MIN=5, with the least recently used ones being removed first.
Note that each of these parameters are PER JVM.
If your jserv.conf has 3 groups of 1 JVM each (OACoreGroup, DiscoGroup,
XmlSvcsGrp) for a total of 3 JVM's, each with a FND_JDBC_BUFFER_MAX of 125 for
a total of 375.
Make a back up of the DBC file(s) in $FND_TOP/secure.
You may have two dbc files instance_sid.domain and instance_sid. You can tune
these as you see fit, such as increasing the decay rate and decreasing the
FND_JDBC_BUFFER_MAX, but whatever you do they MUST BE SPECIFIED in the DBC file.
In order to see the results it would be best to do this at a time when you can
bounce the database as well as the web server.
Look at the AOL/J pool status monitor to verify the changes Over a period of
time take snapshots of the pool monitor and then your connections at the database
level.
Point to Remember
=================
If multiple JVMs are implemented (ie: Jserv load balancing -multiple JVMs in a
single box-, HTTP Load balancing -Multiple Web Servers connecting to the same
instance- or a combination of both), it is important to note that each JVM will
create its own JDBC Pool.
Under these circumstances,it is very difficult to troubleshoot what the problem
is, since some users might be getting errors when trying to obtain connections
from the pool, while when trying to use the diagnostics tools provided to
troubleshoot the problem, a different JVMs could be accessed, obtaining the
wrong results for the tests.
+ Currently there are 8 user-configurable pool parameters.
4 govern pool size, 2 control the thread used to maintain the pool, and
3 control the types of safety checks the pool performs on connections.
All connection pool parameters are configured by setting the corresponding
environment variable in the dbc file.
+ The fundamental purpose of a web server is to deliver files, and when the
JInitiator needs jar files to build the client side of a forms connection the
Apache web server is the logical choice for sending them. The forms process does
not have a mechanism for this.
+ All of the Apache web server controlled JVMs use the DBC and ssp_init.txt files
on the forms/web tier and they are the bulk of the JDBC connections.
One exception that I know of is the concurrent request "OAM Dashboard Collector"
that runs as a java program and makes up to 2 JDBC connections everytime it runs.
It is typically scheduled to run every 10 minutes and on slower systems these
requests tend to back up and stack up the JDBC connections.
The "OAM Dashboad Collector" is the concurrent request that updates the statuses
of everything for the Oracle Applications Manager. If you cancel this request,
then the Oracle Applications Manager console will have stale data.
Ref: Item 11 in Note:226826.1 under "Troubleshooting".
If you don't use the Oracle Applications Manager this probably will not matter
to you, but if you do find more than 1 of these requests running or in pending
status you may want to cancel them temporarily and see if that alleviates your
JDBC session problem.
How to Disable the "OAM Application Dashboard Collection"
1. Bring up applications.
2. To disable the "OAM Application Dashboard Collection" agent, we are
going to do this from the concurrent managers:
2a. Login as someone who has the System Administrator Responsibility
(Do not confuse with System Administration - OSSWA)
2b. Pick System Adminstrator
2c. Navigation Steps:
Concurrent >> Program >> Define
Under Short Name Query up 'FNDOAMCOL'
This will bring up "OAM Application Dashboard Collection"
2d. Uncheck the "Enabled" check box.
2e. Save the record.
3. Bounce applications. (Specifically the Concurrent Managers)
Note: The reason why we have to bounce apps is that the OAM Dashboard Collection is scheduled via OAM.
Prior to "OAM Rollup Patch A (3119911)" this was defaulted to be turned on.
We want to disable this, shutdown the concurrent managers, and then kill any scheduled requests from
trying to launch again.
4. Log back into System Administrator Responsiblity
Go to Concurrent >> Requests
Choose "All My Requests" >> Find
If you see any "OAM Application Dashboard Collection" Pending Scheduled.
Cancel these requests.
Still if you see a CPU spike by JAVA process, then run
$ps -aef |grep java --followed by this SQL statement:
SELECT s.username||' '||s.osuser||' ('|| DECODE(s.terminal,'unknown','?',
'UNKNOWN','?',s.terminal) ||')' userinfo, s.sid||','|| s.serial# ||'
('|| s.audsid ||')' "sid (audsid)", p.spid, w.event , w.p1
FROM v$session s, v$process p ,v$session_wait w
WHERE p.addr = s.paddr AND s.sid = w.sid AND s.process = '&java_process_id'
ORDER BY s.osuser, s.terminal ;
Try to associate the corresponding database processes:
SELECT sid, username, module, program, status FROM v$session WHERE status='INACTIVE' ;
In case you see a corresponding database process, then gather more information
about that session. Use : Note 169630.1: bde_session.sql
- Expensive SQL and resources utilization for given Session ID.
+ Did you make the change to the heap size because you suspected that GC
activity is causing the hang?
If yes, how does the activity compare before and after the change.
With a smaller heap you will be spending more time in GC since it will
happen more frequently.
Another thing to mention here is that when the JVM stops responding to the ping
coming from OPMN, it will be bounced rather than suspended.
To take a thread dump on Unix/Linux, you need to issue the following:
kill -3 jvmpid
where jvmpid is the jvm process id on the OS.
The output will go to $iAS/Apache/Jserv/logs/jvm/OACoreGroup.*.stdout file.
Run the command for all JVMs during the hang.
You should run it 3 times per JVM spaced out by 10 seconds each to establish a
pattern.
The thread dump will show all threads within a particular JVM along with state
and stack for each one of those threads. So for example, if the reason for the
hang is a lock that all threads is waiting for then we will be able to see that.
In other words, it will show what the JVM is doing when it is not responding to
requests. When you take multiple thread dumps for every JVM we will be able to
establish a pattern of what every thread is doing and approximately for how long.
Please also capture the GC activity with timestamps (not natively available if
you are on 1.3.1 Unix/Linux or Windows) and the actual clock time when the JVM
was started and when it went into the hanging phase.
This will be needed to find out if the JVM at the same time of the hang was busy
doing long or excessive GC activity.
Run the following script to determine your patchset level
REM =====================
REM "BEGIN CHECK PATCHES"
REM =====================
set pagesize 1000;
set linesize 141;
spool /tmp/supp_patch.txt;
select bug_number "BUG",
decode(bug_number,
3594604, 'TXK Rollup Patch I',
4068109, 'POST ADX.E[.1] CONSOLIDATED UPDATE',
3072811, 'MLR FOR IAS1.0.2.2 ROLLUP4',
4090504, 'REHOST ORACLE 9iR2 JDBC DRIVERS (v9.2.0.5) WITH ORACLE APPS 11i',
3214685, 'APPS NAV PORTLET: NEED TO RELEASE THE AM FROM THE NAVIGATE PAGE',
3543254, 'Rollup for JTT Session Management',
3005778, 'INACTIVE JDBC CONNECTIONS PERSIST AFTER IPROCURMENT TERMINATED',
3075636, 'INACTIVE JDBC CONNECTIONS PERSIST AFTER IPROCURMENT TERMINATED 2',
3551500, 'UNRELEASED JDBC CONNECTIONS IN IPROCUREMENT 11.5.9',
3137283, 'JVM memory leak',
3298118, 'Extra connections being used under load and connection not released',
3308185, 'Navigates from receiving and connection is not released',
3315942, 'JDBC Connection Leak when navigate to iProcurement Home Page',
3334973, 'APPSPERF: FND: Server side profile cache misses',
3308185, 'Navigates from receiving and connection is not released',
3549568, 'Fixes for connection leaks',
3819096, 'FND_PROFILE.VALUE_SPECIFIC returns Site Value if no context passed',
3968216, 'Set ORACLE.JDBC.FREEMEMORYONENTERIMPLICITCACHE to TRUE') "Patches Applied",
b.LAST_UPDATE_DATE "APPLIED"
from AD_BUGS b where b.BUG_NUMBER in
('4068109','3072811','4090504','3214685','3543254','3005778','3075636','3551500','3137283','3298118','3308185','3315942',
'3334973','3549568','3819096','3308185' ,'3968216');
spool off;
REM ===================
REM "END CHECK PATCHES"
REM ===================
Follow these steps to clean up fnd_nodes table :
1. Make a backup copy of fnd_nodes
2. EXEC FND_CONC_CLONE.SETUP_CLEAN;
COMMIT;
EXIT;
3. Source the iAS env.
4. Then run adlnkiAS.sh from $iAS_Oracle_home/bin.
5. Clear OS memory ( Contact your O/S administrator )
6. Stop all services.
7. Retest the issue using the following login URL:
http://
<
No comments:
Post a Comment