Java Mission Control (jmc) Crashing: SIGSEGV at C [libsoup-2.4.so.1+0x6dab1] soup_session_feature_detac

I was checking around the new JMC, that comes bundle with the jdk1.7_45; this tool is like a mix of Jrockit Mission Control and jvisualvm. As for now, does not have as many tools as offers the JRMC and the connection is done through JMX, just like you would do while connecting with jvisualvm. 

But, to spoil my fun while running the ./jmc I got :



A fatal error has been detected by the Java Runtime Environment:

#

# SIGSEGV (0xb) at pc=0x0000003119a6dab1, pid=9245, tid=140539766241024

#

# JRE version: Java(TM) SE Runtime Environment (7.0_45-b18) (build 1.7.0_45-b18)

# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.45-b08 mixed mode linux-amd64 compressed oops)

# Problematic frame:

# C [libsoup-2.4.so.1+0x6dab1] soup_session_feature_detach+0x11

#

# Core dump written. Default location: /usr/java/jdk1.7.0_45/bin/core or core.9245

Yes, its a nice coredump, and with a filedump in which I got the header above. By looking at the filedump, I could check that this .so lib was being use by an or.eclipse.swt.internal.webkit:

Stack: [0x00007fd1f6c7f000,0x00007fd1f6d80000], sp=0x00007fd1f6d7cb20, free space=1014kNative frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)C [libsoup-2.4.so.1+0x6dab1] soup_session_feature_detach+0x11
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)j org.eclipse.swt.internal.webkit.WebKitGTK._soup_session_feature_detach(JJ)V+0j org.eclipse.swt.internal.webkit.WebKitGTK.soup_session_feature_detach(JJ)V+9j org.eclipse.swt.browser.WebKit.create(Lorg/eclipse/swt/widgets/Composite;I)V+920j org.eclipse.swt.browser.Browser.<init>(Lorg/eclipse/swt/widgets/Composite;I)V+81j org.eclipse.ui.internal.intro.impl.presentations.BrowserIntroPartImplementation.createPartControl(Lorg/eclipse/swt/widgets/Composite;)V+19j org.eclipse.ui.internal.intro.impl.model.IntroPartPresentation.createPartControl(Lorg/eclipse/swt/widgets/Composite;)V+74j org.eclipse.ui.intro.config.CustomizableIntroPart.createPartControl(Lorg/eclipse/swt/widgets/Composite;)V+64j org.eclipse.ui.internal.ViewIntroAdapterPart.createPartControl(Lorg/eclipse/swt/widgets/Composite;)V+9j org.eclipse.ui.internal.ViewReference.createPartHelper()Lorg/eclipse/ui/IWorkbenchPart;+406j org.eclipse.ui.internal.ViewReference.createPart()Lorg/eclipse/ui/IWorkbenchPart;+5j org.eclipse.ui.internal.WorkbenchPartReference.getPart(Z)Lorg/eclipse/ui/IWorkbenchPart;+65j org.eclipse.ui.internal.Perspective.showView(Ljava/lang/String;Ljava/lang/String;)Lorg/eclipse/ui/IViewPart;+16j org.eclipse.ui.internal.WorkbenchPage.busyShowView(Ljava/lang/String;Ljava/lang/String;I)Lorg/eclipse/ui/IViewPart;+59j org.eclipse.ui.internal.WorkbenchPage$20.run()V+21

Investigating the issue with eclipse bugs, I came across this Bug 404776; Which nicely provided a simple workaround by just adding two new parameters:

-Dorg.eclipse.swt.browser.DefaultType=mozilla
-Dorg.eclipse.swt.browser.XULRunnerPath=/urs/lib64/xulrunner

The XULRunnerPath should be set to the actual xulrunner on your OS. At this case, mine is a 64bits Fedora 19...

To Java Mission Control take these parameters, you need to edit the following file:

/usr/java/jdk1.7.0_45/lib/missioncontrol/configuration/config.ini

Just need to append those two parameters at the end of config.ini file and the Java Mission Control should start without a problem.

#echo org.eclipse.swt.browser.DefaultType=mozilla >> /usr/java/jdk1.7.0_45/lib/missioncontrol/configuration/config.ini

#echo org.eclipse.swt.browser.XULRunnerPath=/urs/lib64/xulrunner/ >> /usr/java/jdk1.7.0_45/lib/missioncontrol/configuration/config.ini

# --> root user... 

Watching and taking Thread Dumps with WLST

The hanging or looping behaviour trouble shooting can be some troublesome if you do not know how to read this snapshot. Specially if you did not get the snapshot in the correct time and also on the intervals needed to identify and inculpate a specific thread and your respective class stack.

One of the first steps to investigate is to identify how frequent and how does behave the hanging. Usually we must considered a very broad approach by asking:

• Why do we use this program for?
• Does it have a process peak?

Then we should ask our selves:

• Does the hanging behaviour frequency relate with any of the questions asked previously?

Hopefully with these questions answered will help us determine when to actually wait to collect data for analysis. Some behaviour can help us identify what actually what we are looking for, if the hanging consumes a lot CPU, means that we are dealing with a possible loop, a while(true) or even recursive methods loop or recursive Architecture with bad handling exceptions loops; If the behaviour is just hanging and not consuming much CPU resources, this means is possible a deadlock or livelock. But some times the frequency and behaviour are very random, thanks to parallelism, that makes even more difficult to collect good useful data.

To analyse thread dumps if very difficult and we usually need more then just one, I would say that 10 to 20 thread dumps are good amount for investigating and the frequency between thread dumps depends how fast does the contention happens. There are many visual tools such as samurai and TDA that might help in investigating the hanging. But going back to my original mental thread (talking about parallelism ;), is actually to collect the useful data at the right time which holds the key.

On WLS, I have write a simple WLST/script which does the thread-dumping for me. That way I can spend my time on real useful things like updating my facebook or reading Dilbert strips.

How does it work:

1. Select one of the many Managed-Servers, and set the following parameters:

goto: Managed Server:Configuration:Tuning and set 

• Stuck Thread Max Time: 15 sec (needs restart)
• Stuck Thread Timer Interval: 10 sec (needs restart)

*This parameters have different behaviours and can be set as you like, please look for oracle WLS documentation for more details.

2. Set your domain Environment by running setDomainEnv.sh from your <DOMAIN>/bin directory:

$ . setDomainEnv.sh

3. Create a python script  in which will watch the health of the server:

<code:>

import java.lang
import os
import string
import time

def serverRuntimeNavegate():
    serverRuntime()
    cd("/")
    cd("ThreadPoolRuntime/ThreadPoolRuntime")

def runtimeNavegate():
    runtime()
    cd("/")
    cd("JVMRuntime/" + serverName)

def checkHealthOfServer(serverName):
    print 'Checking : ' + serverName
    os.system("echo Starting")
    serverRuntimeNavegate()
    x = 0
    while true: 
        state = str(cmo.getHealthState())
        check = string.find(state,"HEALTH_WARN")
        if  check != -1:
          print "Warning State"
          threadDump(writeToFile="true",fileName= "ServerDump" + str(x))
          serverRuntimeNavegate()
          Thread.sleep(20000)
          x += 1
        else: 
          print 'Its all good...'
          Thread.sleep(5000)

connect("weblogic","weblogic1","localhost:7001")
checkHealthOfServer(serverName)


4. All you need to do on this script is to change the username, password and server URL:PORT on the connect() command from the script, at the next to last script. Then you can call the WLST to run the script:

$ java weblogic.WLST <scriptName>.py

Basically this Jython/WLST connects to any server and check on the health, if returns OK, all it prints is a message. Soon as the WLS engine decides that exist a long running thread, the script start taking thread dumps and writes the output on files generated on the same location where you called the script. 

Managing The CLASSPATH

The classpath in java and during Runtime, searches for classes and other zip type resources files to load into JVM memory to make the program run. classpath simply maps locations of classes to be use (load) as needed during application's life cycle; There's different ways of setting a java classpath:

• Use the -classpath or -cp to set your classpath;
• Use environment commands to set the classpath;
• Using the class-path tag, on the application's manifest;

Examples:

1. sdktool -[classpath,cp] class1[;|:]class2
   * ';' is use for Windows, ':' Unix like. sdktool is java, javac, jhat, etc..
2. set classpath="class1;class2"

You can only add the following file types '.jar', '.zip' or '.class' into a classpath; Directories or the wildcard '*' can be used to add resources that eventually will be mapped during runtime. Compress files like '.jar' and '.zip' must contains '.class' files, other wise will be ignored.
While using the wildcard '*', the order of files loaded to the JVM vary from platform to platform and also from moment to moment on the same HW machine. Supposedly a well build application does not actually depend on a particular order, but if needed you must explicitly order files into the classpath. In which reads from left to write, and the first file is the one that will be use in case of similar jars with same full class name.
Classpath also can be set on the application's Manifest, using the class-path tag, but this tag does not honour the wildcard. While setting directories, java will load files in the order they appear in the path variables;

WLS

Uses delegation model when loading class, and to improve performance do the following steps:

1. Checks its class load cache (to improve performance);
2. While loading, search for the parent class; therefore goes reading from top to bottom on the family tree of the class, in case does not find the parent it does not attempt to load the class;
3. If both child and parent exist, the parent is loaded first;

Weblogic comes with some feature such as hot-deployment and hot-redeployment. Allows to deploy and changes to newer versions of application module while server is running state, in which helps on avoiding downtime. On the java classloather do not support any standard for hot-deployment nor hot-redeployment; In this case you would need to restart the entire server in case of adding new libs or modifying your application.
Each application deployed on Weblogic Server has a separate hierarchy of classloaders from the system classloader; Also with the bool prefer-web-inf-classes child of container-descriptor tag on the weblogic.xml allow applications to use different libs, base on versioning details description. This is very useful when installing thirdparty libs or upgrading them, by default this tag is set to false.
Ex:
<?xml version="1.0" encoding="UTF-8"?>

<weblogic-web-app xmlns="http://xmlns.oracle.com/weblogic/weblogic-web-app">
  <container-descriptor>
    <prefer-web-inf-classes>true</prefer-web-inf-classes>
    <prefer-application-packages>
      <package-name>javax.faces.*</package-name>
      <package-name>com.sun.faces.*</package-name>
      <package-name>com.bea.faces.*</package-name>
    </prefer-application-packages>

    <prefer-application-resources>
      <resource-name>javax.faces.*</resource-name>
      <resource-name>com.sun.faces.*</resource-name>
      <resource-name>com.bea.faces.*</resource-name>
      <resource-name>META-INF/services/javax.servlet.ServletContainerInitializer</resource-name>
      </prefer-application-resources>
  </container-descriptor>
</weblogic-web-app>


  As you may notice you need to inform which packages or resource to be loaded, therefore we use the list prefer-application-packages and prefer-application-resources tags; There is no need to use both at the same time.   ref:

• Java Class Loading
• WLS class Loading

Fatal Error, no space left on device WLS

I have been changing linux distros and I've faced with an issue in which is simple to resolve but does not happen very often on Debian's distros. Currently I am using Fedora, to get more experience with RH like OS.

While building from scratch my environment, installing the WLS for testing purpose I have faced the following FE (Fatal Error, I call it fatality):

The fatality is clear text, during extraction there is no more space left; But where???

To find out on linux, just open an terminal and run the df command.

$df

On my bellow screenshot, my /tmp directory is almost full.

• As you can see my /tmp is 2G and is 79% full. There is not enough space for a WLS installation which takes a lot tmp space... I believe is about 1.2G of space, and setting a small 2G tmp like I did is a bad business.

• But why did I not have such issue using Debian distro? Well, a friend with more RH experience answered me: "Fedora does not clean tmp directory!!!" (Vaaapaaa, in your face response...)
• Well, after watching over the tmp I came with a possible theory. OS are not responsible for cleaning up tmp, but the programs that use this directory is, and Debians distro just want to be nice with us. 

I have listed a few commands that might be useful for this type of situation:

1. From the ehow blog I found the following commands in which needs to be run as root: 

a. finds and delete any file older than 10 days:
#find /tmp -mtime +10 |xargs rm -f 

b. Delete all empty directory from tmp:
#find /tmp -type d | xargs rmdir

*you can modify this commands to delete needed files, but this is a bad thing because some running programs might be using files from tmp directory; So be advise to be careful with what you delete. 

2. This is more elegant solution in which there no risk. If you have notice I am running a java command, therefore we will use a java solution for this issue without having to delete any files from tmp directory. 

a. create a tmp directory at your home user directory:  

$mkdir tmp
$ls
/home/"YourUser"/tmp

b. run the WLS jar installer with the followind java flag:  

java -Djava.io.tmpdir=/home/"yourUser"/tmp -jar wls1035_generic.jar

Either solution will work but I would rather use the second one; The screen shot taken bellow was when testing the commands from ehow blog.

ref:  How to Clear TMP in Fedora