taming the oacore jvm session id - oatug

Session ID:

Prepared by:

Remember to complete your evaluation for this session within the app!

10886Taming the OACore JVM

April 10, 2019

Ahmed Alomari

Cybernoor Corporation

cybernoor.com

Agenda

• Architecture

• JVM Tuning

• Java Object Cache (JOC)

• Applications Connection Pool

• OA Framework

• Thread Dumps

• Heap Dumps

• Q & A

Architecture

R12.2 Architecture

R12.2 Architecture

• R12.2 uses Oracle Fusion Middleware (11gR1)

• WebLogic Server (WLS) -10.1.3.6

• WebLogic JSP Compiler and Engine 11.1.1.9

• Dual File System (FS1 & FS2) – Run and Patch Filesystems

• ADOP (online Patching) utility

• 64 bit Java Support

– Can deploy Managed servers with larger heaps such as 8GB

– Useful for components with large memory footprints

• Oracle Configurator

• Reporting

• Large number of online users

• New Middle Tier Code level checker (MT –ETCC)

Terminology

• JVM – Java Virtual Machine

• JRE – Java Runtime Environment

• JDK – Java Development Kit

• Java SE – Java Standard Edition

• Java EE – Java Enterprise Edition

• Java ME – Java Micro Edition

• Hotspot – Oracle (Sun’s) VM Implementation

➢Server VM and Client VM

• JNI - Java Native Interface (Native Applications)

Terminology

• JSP - Java Server Pages

➢Web pages which can embed Java, JavaScript, and/or HTML. Usually compiled into a servlet.

➢Located in $OA_HTML

• JDBC – Java Database Connectivity

➢ Interface for SQL execution for Java Applications.

• Servlets – Server side Java code used to interact with Web based applications.

➢Configurator

➢ iPayment

➢AppsLogin

Terminology

• Applet – Client side Java code such as Java code which runs inside a browser [JRE Plug-in or Java Web Start (JWS)]

• Oracle WebForms

• javac – Compiler used to compile java source code files into class files (i.e. byte code).

• JAR – Java Archive

➢Typically used to package an Application including class files, configuration files, properties, manifest, etc..

• Manifest – Descriptor file

• JAR Signing

➢Attach a digital signature to the JAR file in order to improve security.

Terminology

• Class file

➢Well-defined format of compiled Java source (i.e. byte code).

• Package

➢Allows class files to be organized into a hierarchy

• java.lang

• java.util

• oracle.apps.fnd.common

• Class hierarchy

➢oracle.apps.fnd.common.WebAppsContext →

• $JAVA_TOP/oracle/apps/fnd/common/WebAppsContext.class

Terminology

• JDK Vendors

➢Oracle – Linux, Solaris, Windows, macOS

➢ IBM – AIX, Linux, z/OS, Windows

➢HP – HP UX

➢JRockit - Linux, Solaris, Windows

➢Extended Support ended December 2018

➢Oracle JVM – DB Server platforms.

Terminology

• Web Container

➢A Java Application Server which provides

deployment and runtime services

➢JSP engine

➢Pooling Services

➢Thread Pool

➢JDBC Connection Pool

➢Security

➢Logging

➢Transaction Management

➢Cluster Services

Terminology

• Web Container

➢Oracle Containers for J2EE (OC4J)

➢Oracle WebLogic

➢ IBM WebSphere

➢Apache Tomcat

➢RedHat JBoss

➢GlassFish

Architecture

Architecture• HotSpot VM / Mixed-Mode VM

➢ Compiles “hot” classes using thresholds and other criteria into native machine code.

• Oracle HotSpot VM provides Client and Server compilers.

• -server• -client

➢ Uses background thread (CompilerThread0)• To perform compilation.

➢ Switch -XX:+PrintCompilation can be used to report compilation activity including method names.

• java.lang.String::charAt (33 bytes)• java.lang.String::hashCode (60 bytes)

Architecture

Architecture

.

• Java Object Life Cycle

Architecture

• Finalizers

➢ Object has been declared unreachable.

➢ The finalize() method on the object is invoked.

Architecture

• Leaks

try {

webAppsCtx = new WebAppsContext(l_dbc);

requestType = httpservletrequest.getParameter("requestType");

String jvm_url =

webAppsCtx.getProfileStore().getProfile("APPS_FRAMEWORK_AGENT");

webAppsCtx.createAnonymousSession();

connection = webAppsCtx.getJDBCConnection();

. . . . . . .

} catch (Exception e)

{

MessageErrorHandler(e);

}

Architecture

• Garbage Collection

➢ Process of re-claiming memory for objects which are

either no longer reachable (i.e. strong reference cleared)

or object reference type permits collection.

➢ Memory leaks in Java are due to references which are not

cleared such as adding objects to a collection without

removing the entry or setting the reference to null.

• Connection leaks

• Cursor leaks

• Data Structure leaks (hash tables, hash maps, other

collection types, etc..)

➢ For example, cursor leaks are fixed by calling the close()

method on the statement object (i.e. reference is cleared

therefore making the object collectable).

Architecture

• Garbage Collection

➢ The JVM provides various collectors based on the heap

being collected as well as the JVM configuration options.

➢ Garbage Collection can be monitored via the following

switches:

• -verbosegc

• Enables GC messages

• -XX:+PrintGCDetails

• Records heap level details of GC event

• -Xloggc

• Redirects GC messages to a dedicated log file

• Can use tools to open the GC log file and plot the data

points.

Architecture

• Collectors➢ Serial Collector (-XX:+UseSerialGC)

➢ Parallel New Space Collector

• -XX:+UseParallelGC / -XX:ParallelGCThreads

➢ Concurrent Collector (CMS)

• -XX:+UseConcMarkSweepGC

• Tries to minimize GC Pause times for stop the world collections.

• CMS optimal for old generation collections as well as Applications which require a large permanent footprint (i.e. large data caches).

• Much more efficient than the conventional Full GC.

➢ Garbage First (G1) Garbage Collector – 1.7+

• Successor to CMS

• -XX:+UseG1GC

• Utilizes compaction

• Divides heap into regions rather than by generation.

Architecture

• Java Heaps (-Xmx/-Xms)➢ Xmx (maximum heap size)

➢ -Xms (minimum heap size at startup)

• Heap segment is memory mapped or a shared memory segment is

created if large pages is used.

➢ Young Generation

• Eden (Nursery)

• NewRatio or NewSize/MaxNewSize

• Survivor Space

• Can be tuned via SurvivorRatio

➢ Tenured Generation (Old Generation)

➢ Meta Space heap (Permanent Generation)

• MaxMetaspaceSize (formerly PermSize and MaxPermSize)

• -XX:MetaspaceSize / -XX:MaxMetaspaceSize

• Code Cache (classes)

• Constants

• Intern strings

Architecture

• Java Heaps (-Xmx/-Xms)

➢ Total OS memory size will be the sum of the value of:

• Xmx + MaxMetaspaceSize + Thread Stacks + Control

Structures

➢ Gather and review verbose GC data to optimally size the

JVM heaps.

• Need to understand the permanent memory footprint (i.e.

tenured generation footprint after caches are loaded and

steady state achieved).

• Need to understand object creation and allocation rates.

• Need to understand lifespan of transient objects.

• Need to understand user level concurrency traffic as well as

end-user response time SLAs.

• Choose optimal collector based on above data points.

Infrastructure Tuning


• Ensure kernel/shell limits are set sufficiently as per

the load.

• Low settings can result in JVM errors, hangs, or

OutOfMemoryError(s).

• /etc/security/limits.confaebizprd hard core unlimited

aebizprd hard nofile 131072

aebizprd hard nproc 131072

aebizprd soft core unlimited

aebizprd soft nofile 131072

aebizprd soft nproc 131072


• Ensure optimal performance of NAS device and the

shared application filesystem (NFS)

• OHS and JVM logs and output files (runtime)

• JVM performance is impacted due to NAS latency.

• Utilize NFS V4 for application file system mounts

• Ensure NAS device supports V4

• nas-app-prod:/app01 on /app01 type nfs4

(rw,relatime,vers=4.0,rsize=64k,wsize=64k,namlen=2

55,soft,nordirplus, local_lock=none)


• Monitor CPU and memory utilization on the

application tier hosts.

• Top processes (by CPU and memory)

• Leverage EM grid or install OSWatcher.

• Monitor Entropy pool to ensure sufficient availabilityIf Entropy values are low or exhausted, the JVM can hang

with threads blocked waiting on random key generation.

applmgr> cat /proc/sys/kernel/random/entropy_avail

3520

applmgr> cat /proc/sys/kernel/random/entropy_avail

118

Might need to switch to /dev/urandom or use options to increase

Entropy availability.

JVM Tuning

Application Tier - JVM

• Ensure you are running a current version of the JDK as

well as current patch set.

JDK

Version

Current

Patchset

MetaLink Note Applications

Version

6.0 Update 211

(28414647)

2244851.1

1459546.1 (12.2)

Extended Support ended 12/2018

7.0 Update 211

(28916517)

1467892.1 (12.1)

1530033.1 (12.2)

12.1.3 /12.2.X

8.0 Update 202

(28916765)

393931.1 (R12) JRE


• JDK 7.0 New Features

➢ Invokedynamic instruction for dynamic languages.

➢ Garbage-First (G1) Garbage Collector.

➢ Improved Class loading

➢ Parallel ClassLoader

➢ try with block

➢ java.lang.AutoCloseable

➢ Tiered Compilation

➢ -XX:+TieredCompilation

➢ Compressed Ordinary Object Pointers

➢ -XX:+UseCompressedOops

➢ NUMA Aware Collectors

➢ -XX:+UseNUMA

➢ Switch statement can reference String Objects as expressions.

➢ ThreadLocalRandom

➢ Improve performance of random number generators.


• JDK 1.5 JVM monitoring tools

➢ jps

➢ jstat

➢ jconsole

➢ jmap (utility to obtain heap information)

• JDK 1.6

➢ jhat (Memory Analysis tool)

➢ jmap enhanced to specify heap dump file name/path.

➢ JDK 1.7

➢ jcmd (diagnostic commands utility)

➢ jvisualvm (JVM monitoring tool)

➢ JDK 1.8

➢ jmc (Java Mission Control)


➢ jvisualvm (JVM monitoring tool)


➢ jmc (Java Mission Control)


➢ jmc (Java Mission Control) – Flight Recorder


• Set initial and max. heap sizes appropriately. Sizes

below are starting points and should be adjusted as per

the load and footprint.

Usage Model Recommended Starting Heap

Size

Small Xms/Xmx=1G

Medium Xms/Xmx=3-4G

Large Xms/Xmx=6-8G


• Add sufficient managed servers across the app-tier nodes to handle the load.

• Minimum two OACore managed servers per app tier host.

• Ensure sufficient CPU and memory exist to support two managed servers plus other processes.

• Use a hardware load balancer (VIP) to load balance the requests across the OACore managed servers.

• Refer to MOS 1905593.1.

perl <AD_TOP>/patch/115/bin/adProvisionEBS.pl \

ebs-create-managedserver -contextfile=<CONTEXT_FILE> \

-managedsrvname=oacore_server4 -servicetype=oacore \

-managedsrvport=21205 -logfile=<APPLRGF>/TXK/addMS_oacoreserver4.log

Application Tier – Logging

• Set logging level to Notice (Severity Levels)• WebLogic Admin Console for oacore_server<n>

• Logging tab

Application Tier – Logging

• Set logging level to Notice (Severity Levels)

➢ Review JVM log files for exceptions or errors

• 12.2: $EBS_DOMAIN_HOME/servers/oa*/logs/

• oacore_server2.out

• oacore_server2.log

• OPMN logs:• $IAS_ORACLE_HOME/instances/*/diagnostics/logs/OPMN/opmn

• 12.1: $INST_TOP/logs/ora/10.1.3/opmn/*oacore*• oacore_default_group_<n>

• oacorestd.out

• oacorestd.err• log4j:

• log4j.rootLogger=error, stdout, R

Application Tier – JVM

• Enable verbose GC

➢ $INST_TOP/ora/10.1.3/opmn/conf/opmn.xml (12.1)

• -verbose:gc

• -XX:+PrintGCTimeStamps

• -XX:+PrintGCDetails

➢ Direct verbosegc output to a specific file

• 12.1: $INST_TOP/admin/scripts/java.sh

• -Xloggc:$JVMLOGDIR/oacore.gc.$$

• 12.2: WebLogic Admin Console

• Configuration->Server Start->Arguments


263888.839: [GC [PSYoungGen: 378131K->9061K(393280K)] 1175648K->807594K(1212480K), 0.0249310 secs] [Times:

user=0.19 sys=0.01, real=0.02 secs]



264062.982: [Full GC[Unloading class sun.reflect.GeneratedMethodAccessor443]

[PSYoungGen: 2107K->0K(392768K)] [PSOldGen: 805666K->247375K(819200K)] 807774K->247375K(1211968K) [PSPermGen:

54677K->53972K(71680K)], 2.6600860 secs] [Times: user=2.63 sys=0.03, real=2.66 secs]






New Space

GC

Elapsed Time

Size Before

GC Size After

GC

Perm. Gen

Type of GC

• Review verbose GC data

Application Tier – JVM

• Jstat• S0C Current survivor space 0 capacity (KB).

• S1C Current survivor space 1 capacity (KB).

• S0U Survivor space 0 utilization (KB).

• S1U Survivor space 1 utilization (KB).

• EC Current eden space capacity (KB).

• EU Eden space utilization (KB).

• OC Current old space capacity (KB).

• OU Old space utilization (KB).

• PC Current permanent space capacity (KB).

• PU Permanent space utilization (KB).

• YGC Number of young generation GC Events.

• YGCT Young generation garbage collection time.

• FGC Number of full GC events.

• FGCT Full garbage collection time.

• GCT Total garbage collection time.

• jstat -gc 298291; jstat -gcutil 298291

• S0C S1C S0U S1U EC EU OC OU PC PU YGC YGCT FGC

FGCT GCT

• 294400.0 303616.0 0.0 141953.0 6370816.0 1796791.0 13981696.0 2371051.0 1048576.0 194275.0 49

8.497 0 0.000 8.497

• S0 S1 E O P YGC YGCT FGC FGCT GCT

• 0.00 46.75 28.20 16.96 18.53 49 8.497 0 0.000 8.497

Application Tier – Session Timeout

• Set the served session timeout to 30 minutes

➢12.2: Set the profile “ICX:Session Timeout”

➢12.1: orion-web.xml

• <session-timeout>30</session-timeout>

• AutoConfig: s_oc4j_sesstimeout

• WebLogic: timeout-secs (1800)

➢Larger timeout values increase overall JVM

memory footprint.

Application Tier – Web Applications

• Ensure the users are trained to use the Logout or Home global buttons

when completing their transactions.

• Ensure users do not use the browser close (“x”) link.

• Logging out gracefully releases the memory and corresponding

resources (i.e. connections, etc..)

• Avoids memory leaks and timeout based invalidation.

Java Object Cache (JOC)

Application Tier – JOC

• Java Object Cache (JOC) is an Middleware Component

which provides a caching framework for Java based

applications.

• E-Business Suite uses JOC for many of the core caches:

• Page Metadata Cache (MDS)

• FND Caches

• Menus

• Responsibilities

• Profiles

• Function Security

Application Tier - JOC

• JOC Provides local caching (i.e. per JVM) as well as

distributed caching (i.e. all JVMs)

• Java System Property LONG_RUNNING_JVM

specifies the caching behavior

• TRUE – Distributed Caching

• FALSE – Local Caching

• LONG_RUNNING_JVM is set in oacore properties file

via the AutConfig context variable

s_long_running_jvm


• JOC Distributed Caching

– Endpoint Receiver thread is created on all OACore

JVMs, GSM JVMs, and XML Services JVMs

– Receives cache update messages

– Listens on Cache Port

• s_java_object_cache_port (AutoConfig)

• Profile: JTF_DIST_CACHE_PORT

• Creates several threads in each JVM

– EndPointConnection

– Ack Processor

– Packet Sender

– CL Packet Processor


• JOC Distributed Caching

• Endpoint Receiver thread receives cache update messages.

"EndPointConnection CL-65 to DI-3 at [10.16.240.16:22128]" daemon prio=10

tid=0x00007fd691dcf000 nid=0x3fe5 runnable [0x00007fd677dfc000]

java.lang.Thread.State: RUNNABLE

at java.net.SocketInputStream.socketRead0(Native Method)

at java.net.SocketInputStream.read(SocketInputStream.java:153)

at java.net.SocketInputStream.read(SocketInputStream.java:122)

at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)

at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)

at java.io.BufferedInputStream.read(BufferedInputStream.java:334)

- locked <0x00000007f55b4600> (a java.io.BufferedInputStream)

at oracle.ias.cache.groupv2.HttpHandler.read(HttpHandler.java:167)

- locked <0x00000007f55b4648> (a oracle.ias.cache.groupv2.HttpHandler)

at

oracle.ias.cache.groupv2.EndPointConnection.receive(EndPointConnection.java:383)

at oracle.ias.cache.groupv2.EndPointConnection.run(EndPointConnection.java:214)


• Monitor JOC log file for exceptions/errors

➢ $APPLRGF/javacache.log*

➢ Memory leaks and JVM instability can occur if JOC loses

contact with other JVMs.

➢ NoClassDef errors involving the Profiles class can occur

during user login if JOC errors occur.

• Check firewall configuration/rules to account for the Cache port

as well as external nodes (DMZ).


• Java Object Cache Full Exception:

oracle.ias.cache.CacheFullException: J2EE JOC-017 The cache is full.

at oracle.ias.cache.CacheHandle.findObject(CacheHandle.java:1682)

at oracle.ias.cache.CacheHandle.locateObject(CacheHandle.java:1120)

at oracle.ias.cache.CacheAccess.get(CacheAccess.java:877)

at oracle.apps.jtf.cache.IASCacheProvider.get(IASCacheProvider.java:771)

at oracle.apps.jtf.cache.CacheManager.getInternal(CacheManager.java:4802)

at oracle.apps.jtf.cache.CacheManager.get(CacheManager.java:4624)

at oracle.apps.fnd.cache.AppsCache.get(Unknown Source)

at

oracle.apps.fnd.profiles.Profiles.getProfileOptionValue(Profiles.java:1714)

• Increase JOC Cache sizes via javacache.xml underneath

$EBS_DOMAIN_HOME

• Change From: max-objects="5000" max-size="10“

• Change To: max-size=”1000″ max-objects=”5000000″

Apps Connection Pool

Application Tier – Connection Pool

• Apps Connection Pool (dbc configuration)

➢ Enable sanity checks in production environments

• FND_JDBC_USABLE_CHECK=true

➢ Start with a reasonable default such as 300.

• FND_JDBC_MAX_CONNECTIONS=300

➢ Tune FND_JDBC_MAX_CONNECTIONS as per the amount of user concurrency (per JVM).

➢ Apps automatically decays idle connections and resizes the pool


• Apps Connection Pool – Enable Test Connection• WebLogic->Configuration->EBSDataSource->Connection Pool-

>Advanced


• Apps Connection Pool (dbc configuration)

➢ Apps Supports the use of Services for RAC load balancing or

failover.

• dbc parameter APPS_JDBC_URL

APPS_JDBC_URL=jdbc\:oracle\:thin\:@(DESCRIPTION\=(LOAD_BALANCE\=YES)(FA

ILOVER\=YES)(ADDRESS_LIST\=(ADDRESS\=(PROTOCOL\=tcp)(HOST\=db1host)(POR

T\=24601))(ADDRESS\=(PROTOCOL\=tcp)(HOST\=db2host)(PORT\=24601)))(CONNE

CT_DATA\=(SERVICE_NAME\= ebizprodOA)))


• Apps Connection Pool Monitoring (12.1)

➢ Use the AOL/J Database Connection Pool

Status Page to monitor the connection pool

status.

• System Administration Responsibility

• AOL/J Database Connection Pool Status

(Function)


• Apps Connection Pool Monitoring (12.1)

➢ Use the AOL/J Database Connection Pool

Status Page to monitor the connection pool

status.

• Lists DBC configuration parameter settings.

• Connection Statistics

• Locked Connections

• Available Connections

• Closed Connections

• Allows drill-downs to determine source of the

connection.


• AOL/J Database Connection Pool Status Page (12.1)


• WebLogic Connection Pool Monitoring (12.2)

• WebLogic Console->Data Source->Monitoring-

>Statistics


• Process Identification (GV$SESSION)

• Allows you to map the JDBC session from v$session to a particular JVM

process.

SID MACHINE PROCESS MODULE LOGON

---- ------------------------- ------------ ------------------ -----------------

41 aptier1.us.oracle.com 23806 JDBC Thin Client 04/08/19 17:26:43




. . . . . . .

. . . . . . .

aptier1{apps_a}-> ps -ef | grep 23806

apps_a 23806 23804 53 17:26:40 pts/20 8:55 /R12/fs1/EBSapps/comn/util/jdk64/bin/java -

Dweblogic.Name=oacore_server2 -DCLIENT_PROCESSID=24314 -server -verbose:gc . . . . . . .


• Monitor the number of JDBC connections for both

APPS and APPLSYSPUB to check for leaks or

excessive connection usage.

• General rule of thumb is that the total number of

connections should not exceed 2*<peak number of

users>.

➢ Keep in mind that each JVM will create ~10 connections for

background threads and bootstrapping at JVM startup time.

• Query GV$SESSION and group by module to

determine which modules are consuming the

connections.


• Ensure Inactive Connection Timeout is set to zero (12.2).

• Only enable if debugging connection leaks, and reset to

zero after collecting debug logs.

• A non-zero values can actually create additional connection

leaks due to the WebLogic Shrink activity.


• Ensure Deadman detection is enabled in sqlnet.ora for

the DB listener

• $TNS_ADMIN/sqlnet.ora

• SQLNET.EXPIRE_TIME= 10

• If not enabled, connections can hang due to TCP timeouts,

firewall rules, abnormal connection terminations, etc..

• Ensure sqlnet.ora entry is present for all relevant listeners

used by the application:

• DB listener (AutoConfig instance listener directory)

• GI/ASM listeners

• Scan listener


• 12.2 Connection Leaks

• Check MOS for latest patches

• Shipping: Patch 27508480

• WebADI: MOS 2141923.1

• Payroll: Patch 22684327

• FND: Patch 26599059 and 28831943

• Ping Database checks: Patches 25969099 and 20141119

OA Framework

68

Database

OA Framework

JDBC

Statement Cache

Apps

Connection Pool

MDS

Metadata Cache

BC4J

Metadata Cache

Data Cache

UIX

HTML Generation

JSP Engine

OA.jsp / OAP.jsp

Module Code

Controller Object

http

OA Framework Architecture


• Java Server Page Source (JSPs)

➢$OA_HTML

• Java Server Page Cache

➢$COMMON_TOP/_pages

• Java Classes

➢$JAVA_TOP

Java Server Pages (JSP)

• Java Server Page (JSP) Execution

➢Upon JSP invocation, the JSP page cache is

examined.

• If a cache entry exists, the JSP class is executed.

• If a cache entry does not exist, the JSP is

processed as follows:

• Translation (.jsp -> .java)

• Compilation (.java -> .class)

Java Server Pages (JSP)

• OA.jsp

➢Centralized JSP which executes OA FWK based

pages such as the Home Page.

➢Uses redirection to render the relevant page.

• RF.jsp

➢Centralized JSP which is used to execute standard

web functions such as:

• Launching Oracle Forms

• Launching Certain Web pages


• User navigates to the OA FWK Page such as iExpenses

or iProcurement

➢Page Metadata is read and loaded into the MDS

Cache

➢Controller Page Object is invoked

➢Controller Page Object invokes server side

components such as the Entity Object or View Object.

• About this Page link can be used to display page

components.

Application Tier – About this Page

• About this Page link


• About this Page link (page definition details)


• About this Page link (personalization details)


• About this Page link – Technology Components

Pool Monitor

Application Tier – Pool Monitor

• Pool Monitor can be used to monitor the Framework Applications as well as the JVM utilization

• Application Module Pool

• Memory Utilization

• Servlet Sessions

• Navigation Path: Home>Settings->Diagnostics->Show Pool Monitor.


• Pool Monitor Home Page


• Pool Monitor – Application Module Pool

Application Tier – AM Pooling

• Application Module (AM) Pool

➢AM Pooling can be enabled/disabled via the profile

“FND: Application Module Pool Enabled”

➢ If AM Pooling is disabled, the values for Creations

and Removals on the AM Pool Page of Pool Monitor

should be close.

➢ If AM Pooling is enabled, monitor Check-ins and

Check-Outs as well as the Available and Unavailable

counters to ensure AM instances are not being

leaked.

Application Tier – WebLogic Console

• Managed Server Performance page

➢Monitor JVM metrics

➢Heap Utilization

➢Garbage Collections

➢Thread information

➢Stuck Threads

Application Server – WebLogic Console

WebLogic Tuning

• Monitor Thread and Connection Pools

• Thread Count

• Maximum number of connections

• JDBC Ping Check Tuning

• Consider disabling if connections are stable and not hard-closed due to

firewall or inactivity termination.• Test Connections On Reserve

• Test Frequency

• Enable JDBC Statement Caching• Test thoroughly and watch for cursor leaks

• Utilize Hardware Load Balancers in place of Web Server clustering

• Monitor Heap utilization and GC traffic

WebLogic Tuning

• Monitor Long Running and Stuck Threads• Stuck Thread Max Time

• Stuck Thread Timer Interval

Java Thread Dumps

Application Tier – Thread Dumps

➢ Thread dumps show the stack trace for each thread

that is alive.

➢ To generate a thread dump, identify the process id of

the JVM process.

• Then, execute the following:

-kill –QUIT <pid>

➢ Thread dumps show deadlocks which can cause the

JVM to stop responding to user requests. If that’s the

case, the JVM should be restarted.

➢ Helps identify deadlocks, infinite loops, long running

queries submitted by the user, etc..


➢ Thread dumps show the stack trace for each thread

that is alive.

➢ To generate a thread dump, identify the oacore

Managed Server name and click on the Dump Thread

Stacks button.

Application Tier – Thread Dumps2019-04-09 14:58:31

Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.201-b11 mixed mode):

"Worker-5" daemon prio=10 tid=0x00007f17d402b000 nid=0x5e751 in Object.wait() [0x00007f1712bfd000]

java.lang.Thread.State: WAITING (on object monitor)

at java.lang.Object.wait(Native Method)

at oracle.ias.cache.TaskQ.waitForWork(TaskQ.java:255)

- locked <0x0000000312f2f250> (a oracle.ias.cache.TaskQ)

at oracle.ias.cache.TaskQ.getTask(TaskQ.java:138)


at oracle.ias.cache.WorkerThread.run(ThreadPool.java:303)

"Worker-4" daemon prio=10 tid=0x00007f16f8023000 nid=0x5e750 in Object.wait() [0x00007f17397fc000]








"Worker-2" daemon prio=10 tid=0x00007f175d85a000 nid=0x5e6cb in Object.wait() [0x00007f1713ffe000]









Found one Java-level deadlock:

=============================

"Thread-65103":

waiting to lock monitor 0x0813e334 (object 0xb4006318, a

oracle.jbo.common.ampool.SessionCookieImpl$SessionCookieLock),

which is held by "Thread-65088"

"Thread-65088":

waiting to lock monitor 0x0813e374 (object 0xb3fa1418, a org.apache.jserv.JServSession),

which is held by "Thread-65103"

Java stack information for the threads listed above:

===================================================

"Thread-65103":

at oracle.apps.fnd.framework.webui.OAHttpSessionCookieImpl.timeout(OAHttpSessionCookieImpl.java:607)

- waiting to lock <0xb4006318> (a oracle.jbo.common.ampool.SessionCookieImpl$SessionCookieLock)

at oracle.jbo.http.HttpSessionCookieImpl.timeout(HttpSessionCookieImpl.java:146)

at oracle.jbo.http.HttpContainer.fireTimeout(HttpContainer.java:512)

at oracle.jbo.http.HttpContainer.timeout(HttpContainer.java:232)

at oracle.jbo.http.HttpContainer.valueUnbound(HttpContainer.java:300)

at org.apache.jserv.JServSession.invalidate(JServSession.java:240)

- locked <0xb3fa1418> (a org.apache.jserv.JServSession)

. . . . . . . .

"Thread-65088":

at org.apache.jserv.JServSession.getValue(JServSession.java:293)

- waiting to lock <0xb3fa1418> (a org.apache.jserv.JServSession)

at oracle.jsp.provider.JspUniversalHttpRequest.getSession(JspUniversalHttpRequest.java:408)

at oracle.jsp.provider.JspRDRequest.getSession(JspRDRequest.java:144)





. . . . . . . .


• Java Thread dump output (standard out)

➢12.2:

$EBS_DOMAIN_HOME/servers/oacore_server<n>/lo

gs/oacore_server<n>.out

➢12.1:

• $INST_TOP/logs/ora/10.1.3/opmn/default_group~o

acore~default_group*

• What should you look for?

➢Deadlocked threads.

➢Threads stuck in an infinite loop.

• Check to see if the stack is changing.

Java Heap Dumps

Application Tier – Heap Dumps

• Java Heap Dumps

➢Useful when trying to debug memory leaks,

excessive footprint/garbage collection, and/or

OutofMemoryErrors.

➢There are two types of heap dumps:

• Summary (histogram of all class instances and

shallow size)

• Full heap dump (complete dump of the entire

Java heap) which includes old generation and

permanent generation.

-Can result in very large files depending on the

heap sizes.


• Java Heap Dumps – Summary Heap Dump

➢Summary (histogram of all class instances and

shallow size)

• Can be generated using jmap

• jmap -histo:live <pid>

• Can be generated using a thread dump (kill -3) if

the JVM switch (-XX:+PrintClassHistogram) is

present. The summary histogram will be rewritten

to the oacore*.stdout log files (by default).


• Java Heap Dumps - Summary Heap Dumpnum #instances #bytes class name

----------------------------------------------

1: 21437694 686006208 com.sun.java.util.collections.HashMap$Entry

2: 5553282 646005184 [C

3: 19853550 476485200 oracle.apps.cz.dio.FieldCache

4: 615172 218915592 [Lcom.sun.java.util.collections.HashMap$Entry;

5: 315197 143264864 [B

6: 5539474 132947376 java.lang.String

7: 1453342 81990664 [Ljava.lang.Object;

8: 1545022 61800880 java.math.BigDecimal

9: 341887 54284456 <constMethodKlass>

10: 2119195 50860680 java.lang.Long

11: 341887 43778960 <methodKlass>

12: 283528 40828032 oracle.apps.cz.dio.model.DbBomOptionClass

13: 29342 37480752 <constantPoolKlass>

14: 615172 34449632 com.sun.java.util.collections.HashMap

15: 1004869 32155808 java.sql.Timestamp

16: 1124467 26987208 com.sun.java.util.collections.ArrayList

17: 1120206 26884944 java.lang.Double

18: 839012 26848384 EDU.oswego.cs.dl.util.concurrent.ConcurrentHashMap$Entry

19: 600378 24015120 oracle.apps.cz.utilities.LongHashMap$Entry

20: 29342 23976200 <instanceKlassKlass>


• Java Heap Dumps – Summary Histogram

– Quick to generate (several seconds)

• -XX:+PrintClassHistogram (and kill -3) is much faster

than jmap because it uses the native signal handler to

dump the heap.

– Good place to start when debugging leaks as it can help

you narrow down which objects are growing and which

objects consume the most memory.

• Once you have narrowed down the suspect objects, a

full heap dump can be generated to identify the root

cause and examine the reference graph.


• Java Heap Dumps – Full Heap Dump– Dumps all live objects to a heap dump file usually in hprof

format.

– Can be generated using jmap• jmap -heap:format=b <pid>

– Trigger a heap dump when an OutofMemoryError occurs

• -XX:+HeapDumpOnOutOfMemoryError

• -XX:HeapDumpPath=<path>

– Use Memory Profiling tools to analyze the heap dump:

• jhat (part of JDK)

– Can also compare two heap dumps which will help narrow down the objects which continue to grow

• Eclipse Memory Analyzer (MAT)

• NetBeans IDE (Memory Profiler)

References

• https://docs.oracle.com/javase/8/docs

• http://www.oracle.com/technetwork/tutorials/tutorials

-1876574.html

• http://www.oracle.com/technetwork/java/index.html

• http://bugs.java.com/bugdatabase

• http://www.oracle.com/technetwork/java/javase/docu

mentation/index.html

• http://www.oracle.com/technetwork/java/javaseprod

ucts/mission-control/java-mission-control-

1998576.html

https://docs.oracle.com/javase/8/docs

http://www.oracle.com/technetwork/tutorials/tutorials-1876574.html

http://www.oracle.com/technetwork/java/index.html

http://bugs.java.com/bugdatabase

http://www.oracle.com/technetwork/java/javase/documentation/index.html

http://www.oracle.com/technetwork/java/javaseproducts/mission-control/java-mission-control-1998576.html

Session ID:

Remember to complete your evaluation for this session within the app!

Taming the OACore JVM

10886

[email protected]

taming the oacore jvm session id - oatug

Documents