java -XX:+PrintFlagsFinal -version| grep -i Pre
     intx AllocatePrefetchDistance                  = 192             {product}
     intx AllocatePrefetchInstr                     = 0               {product}
     intx AllocatePrefetchLines                     = 4               {product}
     intx AllocatePrefetchStepSize                  = 64              {product}
     intx AllocatePrefetchStyle                     = 1               {product}
     bool AlwaysPreTouch                            = false           {product}
    uintx CMSAbortablePrecleanMinWorkPerIteration   = 100             {product}
     intx CMSAbortablePrecleanWaitMillis            = 100             {manageable}
    uintx CMSMaxAbortablePrecleanLoops              = 0               {product}
     intx CMSMaxAbortablePrecleanTime               = 5000            {product}
     bool CMSPermGenPrecleaningEnabled              = true            {product}
    uintx CMSPrecleanDenominator                    = 3               {product}
    uintx CMSPrecleanIter                           = 3               {product}
    uintx CMSPrecleanNumerator                      = 2               {product}
     bool CMSPrecleanRefLists1                      = true            {product}
     bool CMSPrecleanRefLists2                      = false           {product}
     bool CMSPrecleanSurvivors1                     = false           {product}
     bool CMSPrecleanSurvivors2                     = true            {product}
    uintx CMSPrecleanThreshold                      = 1000            {product}
     bool CMSPrecleaningEnabled                     = true            {product}
     bool CompilerThreadHintNoPreempt               = true            {product}
    uintx G1HeapRegionSize                          = 0               {product}
     intx InterpreterProfilePercentage              = 33              {product}
     intx PreBlockSpin                              = 10              {product}
     intx PreInflateSpin                            = 10              {pd product}

Rationale

Sampling tools like oprofile or dtrace’s profile provider don’t really provide methods to see what [multithreaded] programs are blocking on – only where they spend CPU time. Though there exist advanced techniques (such as systemtap and dtrace call level probes), it is overkill to build upon that. Poor man doesn’t have time. Poor man needs food.

Method

For a poor developer to understand what a program is doing, he needs to see stacks. Once upon a time (back in Linux 2.4) there was a ‘pstack’ tool for that, Solaris has it too.

Modern Linux systems though do not have such facilities, and one needs to improvise, like.. use debuggers – they can walk threads and provide stacks.

Technology

Getting stacks:

gdb -ex "set pagination 0" -ex "thread apply all bt" \
  --batch -p $(pidof mysqld)

Or for version-impaired (gdb 6.3 and older):

(echo "set pagination 0";
 echo "thread apply all bt";
 echo "quit"; cat /dev/zero ) | gdb -p $(pidof mysqld)

Collapsing traces (awk!):

BEGIN { s = ""; }
/Thread/ { print s; s = ""; }
/^\#/ { if (s != "" ) { s = s "," $4} else { s = $4 } }
END { print s }

Full technology demonstration:

#!/bin/bash
nsamples=1
sleeptime=0
pid=$(pidof mysqld)

for x in $(seq 1 $nsamples)
  do
    gdb -ex "set pagination 0" -ex "thread apply all bt" -batch -p $pid
    sleep $sleeptime
  done | \
awk '
  BEGIN { s = ""; }
  /Thread/ { print s; s = ""; }
  /^\#/ { if (s != "" ) { s = s "," $4} else { s = $4 } }
  END { print s }' | \
sort | uniq -c | sort -r -n -k 1,1

Output

291 pthread_cond_wait@@GLIBC_2.3.2,one_thread_per_connection_end,handle_one_connection
 57 read,my_real_read,my_net_read,do_command,handle_one_connection,start_thread
 26 pthread_cond_wait@@GLIBC_2.3.2,os_event_wait_low,os_aio_simulated_handle,fil_aio_wait,io_handler_thread,start_thread
  3 pthread_cond_wait@@GLIBC_2.3.2,os_event_wait_low,srv_purge_worker_thread
  1 select,os_thread_sleep,srv_purge_thread
  1 select,os_thread_sleep,srv_master_thread
  1 select,os_thread_sleep,srv_lock_timeout_and_monitor_thread
  1 select,os_thread_sleep,srv_error_monitor_thread
  1 select,handle_connections_sockets,main,select
  1 read,vio_read_buff,my_real_read,my_net_read,cli_safe_read,handle_slave_io
  1 pthread_cond_wait@@GLIBC_2.3.2,os_event_wait_low,sync_array_wait_event,rw_lock_s_lock_spin,buf_page_get_gen,btr_cur_search_to_nth_level,row_search_for_mysql,ha_innodb::index_read,handler::index_read_idx_map,join_read_const,join_read_const_table,make_join_statistics,JOIN::optimize,mysql_select,handle_select,execute_sqlcom_select,mysql_execute_command,mysql_parse,dispatch_command,do_command,handle_one_connection
  1 pread64,os_file_pread,os_file_read,fil_io,buf_read_page_low,buf_read_page,buf_page_get_gen,btr_cur_search_to_nth_level,row_search_index_entry,row_upd_step,row_update_for_mysql,ha_innodb::delete_row,handler::ha_delete_row,mysql_delete,mysql_execute_command,mysql_parse,Query_log_event::do_apply_event,apply_event_and_update_pos,handle_slave_sql
  1 pread64,os_file_pread,os_file_read,fil_io,buf_read_page_low,buf_read_page,buf_page_get_gen,btr_cur_search_to_nth_level,row_search_for_mysql,ha_innodb::index_read,handler::index_read_idx_map,join_read_const,join_read_const_table,make_join_statistics,JOIN::optimize,mysql_select,handle_select,execute_sqlcom_select,mysql_execute_command,mysql_parse,dispatch_command,do_command,handle_one_connection
  1 do_sigwait,sigwait,signal_hand

Success stories and references

We hear this technology has been used by performance engineers at Google, Facebook, Wikipedia, Intel, Sun Microsystems and other places.

• Baron Schwartz
• Mikael Ronstrom
• Mark Callaghan

Credits

Originally this technology was released as a collaborative effort by Mark Callaghan and Domas Mituzas at this blog post.

Extract RPM file using rpm2cpio and cpio command:

$ rpm2cpio any_rpm_package.x86_64.rpm | cpio -idmv




Hung, Deadlocked, or Looping Process

• Print thread stack for all Java threads:
  • Control-\
  • kill -QUIT pid
  • jstack pid (or jstack -F pid if jstack pid does not respond)
• Detect deadlocks:
  • Request deadlock detection: JConsole tool, Threads tab
  • Print information on deadlocked threads: Control-\
  • Print list of concurrent locks owned by each thread: -XX:+PrintConcurrentLocks set, then Control-\
  • Print lock information for a process: jstack -l pid
• Get a heap histogram for a process:
  • Start Java process with -XX:+PrintClassHistogram, then Control-\
  • jmap -histo pid (with -F option if pid does not respond)
• Dump Java heap for a process in binary format to file:
  • jmap -dump:format=b,file=filename pid (with -F option if pid does not respond)
• Print shared object mappings for a process:
  • jmap pid
• Print heap summary for a process:
  • Control-\
  • jmap -heap pid
• Print finalization information for a process:
  • jmap -finalizerinfo pid
• Attach the command-line debugger to a process:
  • jdb -connect sun.jvm.hotspot.jdi.SAPIDAttachingConnector:pid=pid

Post-mortem Diagnostics, Memory Leaks

• Examine the fatal error log file. Default file name is hs_err_pidpid.log in the working-directory.
• Create a heap dump:
  • Start the application with HPROF enabled: java -agentlib:hprof=file=file,format=bapplication; then Control-\
  • Start the application with HPROF enabled: java -agentlib:hprof=heap=dump application
  • JConsole tool, MBeans tab
  • Start VM with -XX:+HeapDumpOnOutOfMemoryError; if OutOfMemoryError is thrown, VM generates a heap dump.
• Browse Java heap dump:
  • jhat heap-dump-file
• Dump Java heap from core file in binary format to a file:
  • jmap -dump:format=b,file=filename corefile
• Get a heap histogram for a process:
  • Start Java process with -XX:+PrintClassHistogram, then Control-\
  • jmap -histo pid (with -F option if pid does not respond)
• Get a heap histogram from a core file:
  • jmap -histo corefile
• Print shared object mappings from a core file:
  • jmap corefile
• Print heap summary from a core file:
  • jmap -heap corefile
• Print finalization information from a core file:
  • jmap -finalizerinfo corefile
• Print Java configuration information from a core file:
  • jinfo corefile
• Print thread trace from a core file:
  • jstack corefile
• Print lock information from a core file:
  • jstack -l corefile
• Attach the command-line debugger to a core file on the same machine:
  • jdb -connect sun.jvm.hotspot.jdi.SACoreAttachingConnector:javaExecutable=path,core=corefile
• Attach the command-line debugger to a core file on a different machine:
  • On the machine with the core file: jsadebugd path corefile
    and on the machine with the debugger: jdb -connect sun.jvm.hotspot.jdi.SADebugServerAttachingConnector:debugServerName=machine
• libumem can be used to debug memory leaks.

PermGen holds the metadata about classes that have been loaded/created. This information is garbage collected like the other parts of the heap, however there are rough edges that can prevent this from happening, class loaders in particular. Generally, the amount of PermGen space needed is small in relation to the rest of the heap.

How do I know what classes are being loaded or unloaded? Use the command line options

-XX:+TraceClassLoading and -XX:+TraceClassUnloading .

If you suspect that that classloader isn’t the issue, use -verbose:class to investigate classes which have been loaded. This, as it suggests, will be very verbose.

How do you increase PermGen space?

Increase using the -XX:MaxPermSize option.

Conversion Table

Data Types and Data Structures

The new 5.5 is now available with “Semisynchronous Replication”. This comes as an addition to the built-in asynchronous replication. MySQL replication is asynchronous by default. Events written to the binary logs on the Master server being retrieved by the slave server(s). Unfortunately, the Master server has no knowledge of when the slave has retrieved or processed these events. As a result, when Master crashes, transactions committed on Master, might have not have been committed on the slave server(s). In other words, there is no guarantee that any event will ever reach any slave.

In case of “Semisynchronous Replication”, the Master server blockes the transaction commit until at least one of the slave servers acknowledges the it has received all the events for that transaction. Obviously, this is great for consistency, however it brings up questions about replication over the WWW which isn’t an uncommon practice. In case of hitting the timeout limit (in case of no response from any of the slave servers), the Master server reverts back to asynchronous replication.

From MySQL support:

To understand what the “semi” in “semisynchronous replication” means, compare it with asynchronous and fully synchronous replication:

* With asynchronous replication, the master writes events to its binary log and slaves request them when they are ready. There is no guarantee that any event will ever reach any slave.
* With fully synchronous replication, when a master commits a transaction, all slaves also will have committed the transaction before the master returns to the session that performed the transaction. The drawback of this is that there might be a lot of delay to complete a transaction.
* Semisynchronous replication falls between asynchronous and fully synchronous replication. The master waits after commit only until at least one slave has received and logged the events. It does not wait for all slaves to acknowledge receipt, and it requires only receipt, not that the events have been fully executed and committed on the slave side.