Java Thread Pools and their Thread Dumps

When analysing a thead dump, if the thread is created with a custom thread name, we can easily trace it to where the thread pool is created by the unique thread name.

Otherwise, we will have to guess which type of thread pool is created from the stack trace, and then search the usage of creation methods like newCachedThreadPool, newSingleThreadScheduledExecutor, newScheduledThreadPool, or newCachedThreadPool.

When using Executors.newCachedThreadPool on JDK 6:

"pool-1-thread-2" prio=5 tid=7ffb30143000 nid=0x117fde000 waiting on condition [117fdd000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <7f310c388> (a java.util.concurrent.SynchronousQueue$TransferStack)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
    at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
    at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
    at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
    at java.lang.Thread.run(Thread.java:680)

When using Executors.newCachedThreadPool on JDK 7:

"pool-1-thread-2" prio=5 tid=0x00007f90b38c9800 nid=0x5603 waiting on condition [0x000000016cb92000]
   java.lang.Thread.State: TIMED_WAITING (parking)
 at sun.misc.Unsafe.park(Native Method)
 - parking to wait for  <0x000000014c55bbd8> (a java.util.concurrent.SynchronousQueue$TransferStack)
 at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
 at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
 at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
 at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
 at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
 at java.lang.Thread.run(Thread.java:722)

When using Executors.newScheduledThreadPool(numOfCoreThreads) on JDK 6:

"pool-3-thread-1" prio=6 tid=0x4a5cc000 nid=0xc810 waiting on condition [0x4c96f000]
   java.lang.Thread.State: TIMED_WAITING (parking)
 at sun.misc.Unsafe.park(Native Method)
 - parking to wait for  <0x0934ccc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
 at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
 at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
 at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
 at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
 at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
 at java.lang.Thread.run(Thread.java:662)

When using Executors.newScheduledThreadPool(numOfCoreThreads) on JDK 7:

"pool-1-thread-1" prio=5 tid=0x00007f8075802000 nid=0x5503 waiting on condition [0x00000001610ed000]
   java.lang.Thread.State: WAITING (parking)
 at sun.misc.Unsafe.park(Native Method)
 - parking to wait for  <0x0000000140bbc108> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
 at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
 at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
 at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
 at java.lang.Thread.run(Thread.java:722)

When using Executors.newSingleThreadScheduledExecutor on JDK 6:

"pool-1-thread-1" prio=5 tid=7ff0fe159800 nid=0x11442e000 waiting on condition [11442d000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <7f310c8c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
    at java.lang.Thread.run(Thread.java:680)

When using Executors.newSingleThreadScheduledExecutor on JDK 7:

"pool-1-thread-1" prio=5 tid=0x00007ff7fb0aa000 nid=0x5503 waiting on condition [0x0000000169c4d000]
   java.lang.Thread.State: WAITING (parking)
 at sun.misc.Unsafe.park(Native Method)
 - parking to wait for  <0x000000014971c208> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
 at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
 at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
 at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
 at java.lang.Thread.run(Thread.java:722)

When using Executors.newFixedThreadPool on JDK 6:

"pool-1-thread-2" prio=5 tid=7ff7c4854000 nid=0x118682000 waiting on condition [118681000]
   java.lang.Thread.State: WAITING (parking)
 at sun.misc.Unsafe.park(Native Method)
 - parking to wait for  <7f44c0048> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
 at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
 at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
 at java.lang.Thread.run(Thread.java:680)

When using Executors.newFixedThreadPool on JDK 7:

"pool-1-thread-1" prio=5 tid=0x00007fb3b4070000 nid=0x5503 waiting on condition [0x000000016c6bf000]
   java.lang.Thread.State: WAITING (parking)
 at sun.misc.Unsafe.park(Native Method)
 - parking to wait for  <0x000000014c18b888> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
 at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
 at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
 at java.lang.Thread.run(Thread.java:722)

Java Applicatioin Process Hangs on Windows and Cached Thread Pool

The following java test app runs and terminates normally on Unix but hangs on Windows.

import java.util.concurrent.*;

public class ThreadPoolTest {
    private static ExecutorService es = Executors.newCachedThreadPool();

    public static void main(String args[]) { 
        es.submit(new Task("## running task 1..."));
        es.submit(new Task("## running task 2..."));
        System.out.println("## after tasks, in main");
    }

    private static class Task implements Runnable {
        private String msg;

        private Task(String s) {
            this.msg = s;
        }

        @Override public void run() {
            System.out.println(msg);
        }
    }
}

The direct reason for hanging is the 2 worker threads were returned to the cached thread pool after finishing the printing task, and stayed alive and idle indefinitely. Pooled threads by default are non-daemon threads. A Java program will terminate only after all non-daemon threads have terminated. Although the main thread in the above test app have finished its job, the Java process will not terminate due to the 2 live worker threads.

A couple of solutions and their pros and cons:

1, So shall we make the pooled threads daemon threads to fix it? Be careful here, since the Java process may just terminate prematurely, immediately after the main thread is done, but before daemon workers complete their work. Since they are daemon threads, their task status is totally ignored in exiting JVM.

If you really want to take the daemon approach, the application will need some logic to poll the stask status, and wait for their completion before exiting the main thread.

2, How about explicitly call ExecutorService.shutdown() method? After all, a task implementing app logic is not a daemon thread, and should be marked as such. Shutting down the thread pool (the actual type of ExecutorService in our example) makes more sense. But pay attention to all possible exit paths and make sure shutdown guaranteed in all paths, such as normal completion, throwable.

If the thread pool is used by various parts of a large application, how do you coordinate them such that shutdown is called only after all clients are finished? Compared to approach 1, additional coordiation (e.g., with wait/notify or CountDownLatch) is needed. To fix the hang using shutdown():

import java.util.concurrent.*;

public class ThreadPoolTest {
    private static ExecutorService es = Executors.newScheduledThreadPool(2);

    public static void main(String args[]) throws InterruptedException {
        es.submit(new Task("# running task 1..."));
        es.submit(new Task("# running task 2..."));
        es.shutdown();

        //block here for all tasks to finish before proceeding in the main thead.
        //if you want to enforce the execution order. optional.
        es.awaitTermination(1, TimeUnit.DAYS);
        System.out.println("# after tasks, in main");
    }

    private static class Task implements Runnable {
        private String msg;

        private Task(String s) {
            this.msg = s;
        }

        @Override public void run() {
            try {
                Thread.sleep(1000*30);
            } catch (InterruptedException e) {
            }
            System.out.println(msg);
        }
    }
}

awaitTermination is totally optional. With it, you will see 2 task output before main thread output:

# running task 2...
# running task 1...
# after tasks, in main

Without it, main thread output comes before task output:

# after tasks, in main
# running task 2...
# running task 1...

3, Is there a default idle timeout for worker threads, so applications don't have to manage the shutdown? Yes, there is a default idle timeout 60 seconds for non-core threads. When creating a thread pool, you can specify the number of core threads, along with other parameters. By default core threads do not time out after becoming idle. But you can certainly make core threads subject to idle timeout by calling threadPool.allowCoreThreadsTimeout(true).

How does this apply to our test app? We created the thread pool by calling newCachedThreadPool() without passing any parameters. By default, the pool is created with 0 core threads and 60-second idle timeout. So any worker threads in the test app should be non-core and should automatically time out after 60 seconds. Yes, that is what happened on Mac OS or Linux. But on windows 7, the same program just hangs.

Do you still want to rely on its default idle timeout, which seems sensitive to OS?

4, Use Thread and Runnable class directly for simple use cases.  When you only need threads, but don't need to maintain a pool, just directly use threads.

To rewrite the above program directly using java.lang.Thread:

public class ThreadTest {
    public static void main(String args[]) throws InterruptedException {
        Thread t1 = new Thread(new Task("## running task 1..."));
        Thread t2 = new Thread(new Task("## running task 2..."));
        t1.start();
        t2.start();
        t1.join();
        t2.join();
        System.out.println("## after tasks, in main");
    }

    private static class Task implements Runnable {
        private String msg;

        private Task(String s) {
            this.msg = s;
        }

        @Override public void run() {
            System.out.println(msg);
        }
    }
}

The 2 join calls are added to wait for the 2 child threads to finish processing before proceeding to the main thread execution.

How to Configure hprof in GlassFish 3.x

These are the steps to configure hprof profiler in GlassFish 3.x:

1, Identify the target JVM to profile.  In most cases, it's the domain administration (DAS) JVM, but it can be other JVM such as another standalone server instance, or a cluster server instance.

2, Edit $GLASSFISH_HOME/config/osgi.properties, locate org.osgi.framework.bootdelegation property, and append hprof classes to it, using , (comma) as the package separator.  Do not forget to add a comma at the end of the existing value.  The resulting property should look like this:

org.osgi.framework.bootdelegation=${eclipselink.bootdelegation}, \
                                  com.sun.btrace, com.sun.btrace.*, \
                                  org.netbeans.lib.profiler, org.netbeans.lib.profiler.*, \
                                  sun.tools.hprof,com.sun.tools.hat.internal.parser,com.sun.demo.jvmti.hprof

3, Start the domain, and go to admin console to add the hprof profiler:

asadmin start-domain
http://localhost:4848

On the left, choose Configurations | server-config | JVM Settings, on the rigth content panel, choose Profiler tab,

Name: hprof
Status: enabled yes
Classpath: not needed
Native library path: no needed
add a JVM Option:

-agentlib:hprof=file=log.txt,thread=y,depth=3

or using the old non-standard option:

-Xrunhprof:file=log.txt,thread=y,depth=3

Create this profiler and restart the domain from the command line.  You will see the following elements are added to $GLASSFISH_HOME/domains/domain1/config/domain.xml,

<profiler name="hprof" native-library-path="" classpath="">
  <jvm-options>-Xrunhprof:file=log.txt,thread=y,depth=3</jvm-options>
</profiler>

So you could also directly edit domain.xml (not recommended) to save you some clicks, if you know where to add this new element.  For server-config, which corresponds to DAS, it is usually after the first group of jvm-options elements.

There is also asadmin CLI commands to create and delete profiler configuration:

asadmin create-profiler hprof

asadmin delete-profiler

But create-profiler command doesn't allow you to specify hprof jvm options, so you would still need to come back to domain.xml to add jvm options.  However, it takes a --target param, where you can specify where to apply the profiler configuration.

4, After starting and stopping the domain, the hprof data file is created in $GLASSFISH_HOME/domains/domain1/config/ directory,  which is the ${user.dir} for GlassFish server process.

Another related post: How to Configure hprof in JBoss AS7

My Intellij Notes

To search for a command, setting, shortcut, etc, and execute it:
Command + Shift + A

To maximize or full screen (useful when reading long lines).  You can also use the same shortcut to go full screen in FireFox, Safari, Chrome, Preview, iTunes, etc.
Command + Ctrl + F

To see javadoc for the method in focus:
Ctrl + J

To see which method/class the current line is in (with super long method body):
Ctrl + Shift + Q

To paste from clipboard history (including copied content from external apps):
Command + Shift + V

To copy the file path of a class:
Command + Shift + C, while inside the editor of that class, not even need to highlight the class name.

To copy the fully qualified name of a class (FQN):
Command + Alt + Shift + C

To delete a line:
Command + Y (y for yank)

To go to a line:
Command + G

To open a class:
Command + N (press it again to include non-project classes like JDK classes)

To organize imports:
Command + Alt + O

To bring up Version Control System popup (git, svn, etc):
Ctrl + V

To view all methods of the current class (press F12 again to include inherited methods):
Command + F12

To view type hierarchy (super-types, sub-types):
Ctrl + H

To jump to source file of another class:
Command + Click, or F4
Command + Shift + I, for a quick view in pop-up without openning it in editor

To go to next error/warning:
F2
Command + F1, to display the error details
Alt + Enter, to display recommended actions

To complete a statement (addthe semi-colon, going to the next line, etc):
Command + Shift + Enter

To auto-complete with text match (as opposed to code completion):
Alt + / (upward search), Alt + Shift + / (downward search)

To hide a tool window:
Shift + Escape
Command + window-number

To format the whole file:
Command + Alt + L

To format a selection:
Command + W to make a selection, then Command + Alt + L

To view and search for recent files:
Command + E, and type the first few letters to filter it in the pop-up
Command + Shift + E, for recent changed files

To switch between open files and tool windows:
Ctrl + Tab, or Ctrl + Shift + Tab, while holding Ctrl, repeatedly press Tab to change selection

To find usage of a type/class/method/variable:
Alt + F7

To generate constructor, getter, setter, toString, hashcode, equals, override method, inside editor window:
Ctrl + N
Command + O (select methods to override)
Command + I  (select methods to implement from interfaces)

To attach to a remote debugger: F9
To continue the debugger, skip all breakpoints: F9
To step over (go to next line): F8
To step into: F7
To set/unset a breakpoint: Command + F8
To view breakpoints: Command + Shift + F8

To increase indent of the current line:
Command + W, then Tab.  If pressing Tab without selection, it will just insert a tab at the cursor, which is not what you want unless the cursor is at the beginning of the line.

To decrease indent of the current line:
Shift + Tab.  No need to make a selection

To auto-format the current line:
Command + W, then Command + Alt + L.  Without a selection, it will just auto-indent the whole file

To auto-indent the current line:
Command + W, then Command + Alt + I.

To join next line into the current line:
Ctrl + Shift + J.  Useful when you need to get rid of the next few blank lines.  Command + Y will also delete a line, but you will need to move cursor to the blank line first.

To go to the beginning and end of the file:
Command + fn + Left
Command + fn + right
3 finger move on touch pad won't work

To go to the beginning and end of the screen:
Command + fn + Up
Command + fn + Down

To go to the beginning and end of the line:
Command + Left
Command + Right

To move the content up and down, while keeping the cursor in the current line:
Command + Up
Command + Down
2 fingers move on touch pad

Jump to navigation bar:
fn + Alt + Left