How to define WorkManager in Weblogic

WebLogic Server prioritizes work and allocates threads based on an execution model that takes into account administrator-defined parameters actual run-time performance throughput.

Administrators can configure a set of scheduling guidelines and associate them with one or more applications, or with particular application components.
For example, you can associate one set of scheduling guidelines for one application, and another set of guidelines for other applications. At run time, WebLogic Server uses these guidelines to assign pending work and enqueued requests to execution threads.

To manage work in your applications, you define one or more of the following Work Manager components:

• Fair Share Request Class
• Response Time Request Class
• Min Threads Constraint
• Max Threads Constraint
• Capacity Constraint
• Context Request Class

You can use any of these Work Manager components to control the performance of your application by referencing the name of the component in the application deployment descriptor

In addition, you may define a Work Manager that encapsulates all of the above components (except Context Request Class and reference the name of the Work Manager in your application's deployment descriptor

You can configure Work Managers at the domain level, application level, and module level in one of the following configuration files, or by using the WebLogic Server Administration Console:

• config.xml—Work Managers specified in config.xml can be assigned to any application, or application component, in the domain.
• weblogic-application.xml—Work Managers specified at the application level can be assigned to that application, or any component of that application.
• weblogic-ejb-jar.xml or weblogic.xml—Work Managers specified at the component level can be assigned to that component.
• weblogic.xml—Work Managers specified for a Web application.

Work Manager Stanza

Referencing the Work Manager in a Web Application

Br

play with WLST

Deleting JMS messages using WLST

At time you will see that your environment ends up with n numbers of messages getting piled up, and it is not always a good option to remove the Filestore, instead another option is to purge/delete the messages from queues.

do the following to achieve this

# set WLS environment
$ ORACLE_HOME/wlserver/server/bin/./setWLSenv.sh

# launch WLST
$ ORACLE_HOME/oracle_common/common/bin/./wlst.sh

$ java weblogic.WLST

wls:/offline>connect('weblogic','weblogic1','t3://192.0.0.123:7001')
wls:/domain/serverConfig>serverRuntime()
wls:/domain/serverRuntime>cd('JMSRuntime')
wls:/domain/serverRuntime/JMSRuntime>:ls()
wls:/domain/serverRuntime/JMSRuntime>:cd('server.jms')
wls:/domain/serverRuntime/JMSRuntime/server.jms> cd ('JMSServers')
wls:/domain/serverRuntime/JMSRuntime/server.jms/JMSServers>cd (JMSServer-1')

**
wls:/WLST_domain/serverRuntime>cd('JMSRuntime/managed1_wls.jms/JMSServers/JMSServer-1/Destinations/SystemModule-0!com/fixstream/benifits/dr/out/dq')
wls:/WLST_domain/serverRuntime/JMSRuntime/managed1_wls.jms/JMSServers/JMSServer-0/Destinations/SystemModule-0!Queue-0> ls()
wls:/WLST_domain/serverRuntime/JMSRuntime/managed1_wls.jms/JMSServers/JMSServer4/Destinations/SystemModule-0!Queue-0> cmo.deleteMessages('')
**

do the necessary changes and that's it.

App Un-deployment using WLST

# set WLS environment
$ ORACLE_HOME/wlserver/server/bin/./setWLSenv.sh

# launch WLST
$ ORACLE_HOME/oracle_common/common/bin/./wlst.sh

$ java weblogic.WLST

wls:/offline>connect('weblogic','weblogic1','t3://192.0.0.123:7001')
wls:/domain/serverConfig>listApplications()
wls:/domain/serverConfig>edit()
wls:/domain/serverConfig/startEdit()
wls:/domain/serverConfig/stopApplication('myapp')
wls:/domain/serverConfig/undeploy('myapp')
wls:/domain/serverConfig/save()
wls:/domain/serverConfig/activate()
wls:/domain/serverConfig/exit()

# use below command to Deploy application
deploy('myapp','/app/deliv/myapp.war',targets='managed1_mydom01') 

--

WLS domain extension to include JRF template

Extend a domain (/wls_domains/base_domain ) by including JRF restricted template & it works fine
without the use of any database. (in order to have JRF features in plain WLS domain)

Template to use:(/MW_HOME/oracle_common/common/template/wls/Oracle_jrf_restricted.jar)

Alternatively, you can use the following JAVA_argument to include in config.sh which allows you to create a compact domain and will use Embedded Database (JAVADB)

CONFIG_JVM_ARGS  to -Dcom.oracle.cie.config.showProfile=true

But when OIM 11.1.1.9 is being configured on top of it, it is overwriting the configuration files from MW_HOME and not allowing the domain to come up

(compatibility challenge here)

**

Exception in thread "main" java.lang.NoSuchMethodError: weblogic.i18n.logging.CatalogMessage.(Ljava/lang/String;I[Ljava/lang/Object;Lweblogic/i18n/Localizer;)V

        at weblogic.security.SecurityLogger.logDisallowingCryptoJDefaultJCEVerification(SecurityLogger.java:15099)

        at weblogic.security.utils.SecurityUtils.turnOffCryptoJDefaultJCEVerification(SecurityUtils.java:98)

**

Also when we try to set up OVD using 11g installer it is expecting to have wlserver10.3, but we have /12.2.1.2/wlserver, so not allowed to proceed further.

The only possible way is to have 2 WLS domains running on 11g & 12c to serve their functionality respectively.

What Oracle has to say on this - 

There is no exception from the certification matrix. OVD is not (and I do not have information that will be in the future) certified with Weblogic 12c.
However, please note that you can have Weblogic 12c and Weblogic 11g (10.3.6) on the same server running on different ports.
So you may create a OVD domain in Weblogic 11g (10.3.6) and keep weblogic 12c for your current purposes.

Some known Weblogic exceptions

java.lang.IllegalStateException

If you fails to access em console ( http://localhost:port/em) and you see below error trace in your server logs while deploying em.war archive do not panic and try to implement below provided solution

----------error---------------

500 Internal Server Error

Servlet error: An exception occurred. The current application deployment descriptors do not allow

for including it in this response. Please consult the application log for details.

java.lang.IllegalStateException: Environment vairable ORACLE_HOME is null.

at oracle.sysman.emSDK.conf.ConfigManager.init(ConfigManager.java:159)

at oracle.sysman.emSDK.svc.ServiceUtil.initService(ServiceUtil.java:73)

----------/error---------------

Solution:
-------------------------------

modify your setDomainEnv.sh from you domain/bin to add em.oracle.home setting

APPLICATIONS_DIRECTORY="/wls_domains/emdomain/applications/emdomain"

export APPLICATIONS_DIRECTORY

EXTRA_JAVA_PROPERTIES="${EXTRA_JAVA_PROPERTIES} -Dem.oracle.home=/software/bea/Middleware/wls/10.3.6/oracle_common -Djava.awt.headless=true"

export EXTRA_JAVA_PROPERTIES

NameAlreadyUnboundException

if you see below error trace in you server logs, do not panic instead try to fix it using below provided solution

----------error---------------
<[ServletContext@628023324[app:bea_wls_cluster_internal module:bea_wls_cluster_internal.war path:/bea_wls_cluster_internal spec-version:null]] Servlet failed with IOException
weblogic.jndi.internal.NameAlreadyUnboundException: Object already partially unbound in cluster.
        at weblogic.rmi.cluster.ReplicaAwareRemoteObject.writeExternal(ReplicaAwareRemoteObject.java:179)
        at weblogic.rmi.cluster.ClusterableRemoteObject.writeExternal(ClusterableRemoteObject.java:307)
        at java.io.ObjectOutputStream.writeExternalData(ObjectOutputStream.java:1442)
        at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1411)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1159)
        Truncated. see log file for complete stacktrace
----------/error---------------

Reason to the Problem:
--------------------------------
public class NameAlreadyBoundException
extends NamingException https://docs.oracle.com/javase/7/docs/api/javax/naming/NamingException.html

This exception is thrown by methods to indicate that a binding cannot be added because the name is already bound to another object.
You have same interface in two jar files and also both these jars exists in this environment.Only one JNDI can exists. You have to remove one of them so that other one can work as expected.
If you can't do this please change the JNDI name in one of the jar and redeploy.

Solution:
--------------------------------
change code from:
@Stateless(mappedName = "management.service")
to
@Stateless(name = "management.service")

Issue: While activating the changes in the WebLogic admin console

An error occurred during activation of changes, please see the log for details. [Management:141191]The prepare phase of the configuration update failed with an exception: Provider org.apache.xerces.jaxp.DocumentBuilderFactoryImpl not found

Solution:
--------------------------------

1. Stop the AdminServer & ManagedServers.

2. On your AdminServer cd to %WEBLOGIC_HOME%\user_projects\domains\base_domain.

3. Delete the files under pending and servers/domain_bak directory.

4. Repeat steps 2 & 3 one each ManagedServers in the domain.

5. Start the AdminServer & ManagedServers.

6. Login the AdminConsole and click on ReleaseConfiguration button.

7. Perform the same changes now.

Installing WebLogic 12.1.3 without GUI

There are multiple ways to install and configure WebLogic, one of them is doing in Silent mode
Here I tried to document the steps with v12.1.3, but can be followed with any release of 12c by simply changing the product version.

    STEP 1 - Download the generic jar and java

•      fmw_12.1.3.0.0_wls.jar
•      jdk-7u45-linux-x64.tar.gz

    STEP 2 - Create the response file as below
    wls.rsp

[ENGINE]
#DO NOT CHANGE THIS.
Response File Version=1.0.0.0.0

[GENERIC]
#The oracle home location. This can be an existing Oracle Home or a new Oracle Home
ORACLE_HOME=/apps/weblogic/12.1.3

#Set this variable value to the Installation Type selected. e.g. WebLogic Server, Coherence, Complete with Examples.
INSTALL_TYPE=WebLogic Server

#Provide the My Oracle Support Username. If you wish to ignore Oracle Configuration Manager configuration provide empty string for user name.
MYORACLESUPPORT_USERNAME=

#Provide the My Oracle Support Password
MYORACLESUPPORT_PASSWORD=

#Set this to true if you wish to decline the security updates. 
#Setting this to true and providing empty string for My Oracle Support username will ignore the Oracle Configuration Manager configuration
DECLINE_SECURITY_UPDATES=true

#Set this to true if My Oracle Support Password is specified
SECURITY_UPDATES_VIA_MYORACLESUPPORT=false

#Provide the Proxy Host
PROXY_HOST=

#Provide the Proxy Port
PROXY_PORT=

#Provide the Proxy Username
PROXY_USER=

#Provide the Proxy Password
PROXY_PWD=

#Type String (URL format) Indicates the OCM Repeater URL which should be of the format [scheme[Http/Https]]://[repeater host]:[repeater port]
COLLECTOR_SUPPORTHUB_URL=

STEP 3 - Create ora Installation as below

oraInst.loc

inventory_loc=/apps/weblogic/
inst_group=inetsupp

STEP 4 -  Execute the below command accordingly

$ $JAVA_HOME/bin/java -jar fmw_12.1.3.0.0_wls.jar -silent -responseFile /apps/weblogic/wls.rsp -invPtrLoc /apps/weblogic/oraInst.loc -debug
Launcher log file is /tmp/OraInstall2015-09-03_11-06-10PM/launcher2015-09-03_11-06-10PM.log.
Extracting files.............
Starting Oracle Universal Installer 
. 
. 
Preparing to launch the Oracle Universal Installer from /tmp/OraInstall2015-09-03_11-06-10PM 
Command line arguments:
Arg:0:/apps/java/jdk1.7.0_45/bin/java 
. 
. 
Verifying data......
Copying Files...
You can find the log of this install session at:
/tmp/OraInstall2015-09-03_11-06-10PM/install2015-09-03_11-06-10PM.log
-----------20%----------40%----------60%----------80%--------100%
The installation of Oracle Fusion Middleware 12c WebLogic Server and Coherence 12.1.3.0.0 completed successfully.
Logs successfully copied to /apps/weblogic/logs. 
you will get your weblogic installation in /apps/weblogic/12.1.3/..

STEP 5 - Creating a domain using wlst script

$ cd /apps/weblogic/12.1.3/oracle_common/common/bin/
$ ./wlst.sh
wls:/offline>help()
wls:/offline>help('createDomain')
wls:/offline> createDomain('/apps/weblogic/12.1.3/wlserver/common/templates/wls/wls.jar','/apps/weblogic/domain/base_domain','punit','punit123')
Br,

Punit

Weblogic SSL Renewal Steps

When you wish to install the security certificate on your WebLogic server, it's a 3 step procedure 

STEP 1: Creating a Keystore (.jks file ) 

(this is the prime entity that stores your certificates)

 # Generate a public key 
 $ keytool -genkey -alias punit -keyalg RSA -keysize 2048 -keystore identity_keystore.jks -storepass weblogic1
 
 it will prompt you for following questions:

 What is your first and last name?
 [Unknown]: abc.com
 What is the name of your organizational unit?
 [Unknown]: MW
 What is the name of your organization?
 [Unknown]: CTS
 What is the name of your City or Locality?
 [Unknown]:
 What is the name of your State or Province?
 [Unknown]:
 What is the two-letter country code for this unit?
 [Unknown]: IN 

 it will prompt for confirmation so give > yes

 this will create a file named > identityKeystore.jks
 

STEP 2: Generating CSR (Certificate Signing Request )

 $ keytool -certreq -alias punit -file csr.txt -keystore identity_keystore.jks -storepass weblogic1

 this will create a file named > csr.txt

copy the content of csr.txt and send it to the signing authority, they will encrypt their private key into the CSR's and send three files (root.pem, interim.pem & server.pem)

STEP 3: Importing the Certificates:

open all the three certificates and copy & paste the content into certificate_chain.pem in order
server > intermediate > root
now import this certificate_chain.pem into identity keystore (identity_keystore.jks) using below command

 
 $ keytool -v -import -alias punit -file certificate_chain.pem -keystore identity_keystore.jks -storepass weblogic1
 or
 $ keytool -importcert -file certificate.cer -keystore keystore.jks -alias "Alias" 

(use alias & password defined by you while requesting the certificate)

it will prompt you for Yes/No  > Yes

this will import the certificate chain (Root, Interim, Server) into identity_keystore.jks

if required import (Root, Interim) into trust_keystore.jks as well (optional/Depends)

(now define the ssl & keystore properties from Admin console for identity & trust keystore and bounce the servers)

**
Select keystore type as: Custom Identity and Command-Line Trust
Define attributes for the Identity keystore
Custom Identity Keystore File Name - The fully qualified path to the Identity keystore
Keystore type - The type of the keystore. Generally, this attribute is jks
Keystore PassPhrase—The password defined when creating the keystore
-Enable SSL port of server (from server SSL tab)
-click continue
-click finish
-Reboot weblogic server

**

NOTE: for SHA2 certs we need to enable JSSE ssl options corresponding to the servers & sometimes if required add below java_options also in server starts or in nodemanger.properties files if node manager is configured.

 
 -Dweblogic.security.SSL.enableJSSE=true    (client)

 -Dweblogic.ssl.JSSEEnabled=true            (server)

OPatch

OPatch is a Java-based utility built by Oracle that runs on all supported operating systems and requires the installation of the Oracle Universal Installer. It is used to apply patches to Oracle software.

Oracle has recently implemented a new patch nomenclature for Oracle products.

In previous releases of Oracle WebLogic Server, you can patch your WebLogic server software using a utility called Bea Smart Update (bsu). Smart Update is a standalone Java application that you use to upgrade the software installations quickly and easily with maintenance patches and maintenance packs.

For Oracle Fusion Middleware 12c (12.1.2), BSU is no longer supported. instead, WebLogic server users can use OPatch to apply patches for both WebLogic Server and for Oracle Fusion Middleware.

OPatch offers many of the same features as Smart Update, but it has a different set of commands and command options.

OPatch can be found in the following location after you install any Oracle Fusion Middleware product:

ORACLE_HOME/OPatch

To run OPatch, simply run the opatch command in this directory.

For example, to view the list of commands available for OPatch on a UNIX system, enter the following:

$ ./opatch –help

The multiple commands that can be used with opatch utility are 

Command: = apply      Example: opatch -help 

Checkapplicable      opatch apply -help
Compare
lsinventory             Use: to check the number of patches applied
lspatches               List out the number of patches
Napply                  to apply multiple patches in one go
Nrollback               to rollback multiple patch in one go // opatch nrollback -id 15343893,14342349

Rollback                to rollback a patch you applied //Syntax: opatch rollback –id 15343893
Query
version                 to check the version of opatch
prereq
util

To verify that a patch can be applied to a specific Oracle home, use the -report argument to the OPatch apply command, as follows: 

$ opatch apply <path_to_patch_directory> -report

To apply a patch

$ ./opatch apply /tmp/15941858/

if opatch fails to find oraInst.loc or central Inventory

$ ./opatch lsinventory  -invPtrLoc ../oraInst.loc

Possible causes of opatch commands getting failed:

• Dependency on another component
• Corrupted patch zip file download
• Downloaded or extracted the patch zip file under $ORACLE_HOME/inventory/oneoffs
• The OPatch version is too old
• The patch was unzipped for installation in a remote location ( NFS)
• OPatch detects wrong Middleware Home in inventory.xml

which can be fixed, after correcting HOME NAME in inventory.xml which can be found at:
             /software/bea/Middleware/product/oraInventory_ver/ContentsXML/inventory.xml

<HOME_LIST>
<HOME NAME="OracleHome1" LOC="/software/bea/Middleware/product/version/" TYPE="O" IDX="1"/>

</HOME_LIST>

• Prerequisite check "CheckActiveFilesAndExecutables" failed.

                    The details are:

                    Exception occurred : fuser could not be located:

                    UtilSession failed: Prerequisite check "CheckActiveFilesAndExecutables" failed.

The OPatch utility uses "fuser" on UNIX systems to check for active Oracle instances which is unavailable on the host so do the following:

    Set the environment variable OPATCH_NO_FUSER=true
    Shut down the WebLogic instances
    Run the OPatch utility

• Opatch uplift : Doc ID 224346.1

- download p28186730_139424_Generic.zip for 12c

- unzip p28186730_139424_Generic.zip 

you will get a patch folder named - 6880880
$ cd $ORACLE_HOME/OPatch/
$ ./opatch version
if ask Include OPatch in PATH  export PATH=$ORACLE_HOME/OPatch:$PATH 

and run below command to uplift

$ <javaHome>/bin/java -jar 6880880/opatch_generic.jar -silent oracle_home=​<ORACLE_HOME>

ORACLE_HOME will be the absolute path till your WebLogic installation.
e.g /software/bea/wls/12.2.1.3/

if Opatch Fails With Error Code 255

There are possible reasons for this error code to name a few :-

- Your installer doesn't want an opatch uplift and you need to rollback this opatch to the supported version

- The ORACLE_HOME variable you are using is incorrect

OutOfMemoryError issues

You can encounter the "java.lang.OutOfMemoryError: Unable to create new native thread"

• whenever the JVM asks for a new thread from the OS
• Whenever the underlying OS cannot allocate a new native thread
• The exact limit for native threads is very platform-dependent.

It is recommended to find out those limits by running a test similar to the below.

In general, the situation causing java.lang.OutOfMemoryError: Unable to create new native thread goes through the following phases:

1. A new Java thread is requested by an application running inside the JVM
2. JVM native code proxies the request to create a new native thread to the OS
3. The OS tries to create a new native thread that requires memory to be allocated to the thread
4. The OS will refuse native memory allocation either because the 32-bit Java process size has depleted its memory address space – e.g. (2-4) GB process size limit has been hit – or the virtual memory of the OS has been fully exhausted.

 Solution:
1) Reduce the amount of memory allocated to each thread through the use of the -Xss JVM option
reducing the stack size using -Xss (e.g “-Xss64kb” ) start with 64k, try the application, then if it doesn’t work (it will fail with a Java.lang.StackOverFlowError), increase the stack to 128k, then 256k, and so on.
2) Determine the problem with the application that requires that so many threads be opened and redesigned to prevent so many threads from being spawned.
use "ulimit -a" to check the "max user processes" and "open files" and change the value to a bigger value.

Reference: Oracle Doc ID 2232666.1

OOM due to -XXtlasize (Local Thread Area)

java.lang.OutOfMemoryError: getNewTla.
java.lang.OutOfMemoryError: getNewTla
at com.sun.jmx.remote.internal.ArrayNotificationBuffer.fetchNotifications(ArrayNotificationBuffer.java:459)
at com.sun.jmx.remote.internal.ArrayNotificationBuffer$ShareBuffer.fetchNotifications(ArrayNotificationBuffer.java:209)
at com.sun.jmx.remote.internal.ServerNotifForwarder.fetchNotifs(ServerNotifForwarder.java:258)
at javax.management.remote.rmi.RMIConnectionImpl$2.run(RMIConnectionImpl.java:1227)
at javax.management.remote.rmi.RMIConnectionImpl$2.run(RMIConnectionImpl.java:1225)
Truncated. see log file for complete stacktrace
>

------------------------------

The standard approach I can suggest to this kind of problem is to configure the JVM to take a heap dump when it OOME's, and analyze it with jhat or some other tool.
OR
If you are totally stumped, back out the changes to the last known working version and reapply them one at a time in a lower environment first.

Reason for the Problem:
------------------------------
The TLA is the space reserved on the heap for use exclusively by a thread. Running out of memory means that there is no more space to use in the heap. The solution is to increase the available heap space (with Xmx) and not just to play around with the XXtlaSize

-XXtlaSize Sets the thread-local area size.

To increase performance JRockit JVM uses thread-local areas (TLA) for object allocation. This option can be used to tune the size of the thread-local areas, which may affect performance.

Use this option with caution, as changing the thread-local area size can have a severe impact on the performance.

The default value for the preferred size depends on the heap size

current settings you may be using:

 -Xms:512m -Xmx:784m -Xns:300m
 -XXtlaSize:min=16k,preferred=32k

When you set the minimum and/or the preferred TLA size, the large object limit (set with -XXlargeObjectLimit) and the minimum block size (set with -XXminBlockSize) may be adjusted automatically by JRockit JVM if necessary.