java -XX:+PrintFlagsFinal -version| grep -i Pre
     intx AllocatePrefetchDistance                  = 192             {product}
     intx AllocatePrefetchInstr                     = 0               {product}
     intx AllocatePrefetchLines                     = 4               {product}
     intx AllocatePrefetchStepSize                  = 64              {product}
     intx AllocatePrefetchStyle                     = 1               {product}
     bool AlwaysPreTouch                            = false           {product}
    uintx CMSAbortablePrecleanMinWorkPerIteration   = 100             {product}
     intx CMSAbortablePrecleanWaitMillis            = 100             {manageable}
    uintx CMSMaxAbortablePrecleanLoops              = 0               {product}
     intx CMSMaxAbortablePrecleanTime               = 5000            {product}
     bool CMSPermGenPrecleaningEnabled              = true            {product}
    uintx CMSPrecleanDenominator                    = 3               {product}
    uintx CMSPrecleanIter                           = 3               {product}
    uintx CMSPrecleanNumerator                      = 2               {product}
     bool CMSPrecleanRefLists1                      = true            {product}
     bool CMSPrecleanRefLists2                      = false           {product}
     bool CMSPrecleanSurvivors1                     = false           {product}
     bool CMSPrecleanSurvivors2                     = true            {product}
    uintx CMSPrecleanThreshold                      = 1000            {product}
     bool CMSPrecleaningEnabled                     = true            {product}
     bool CompilerThreadHintNoPreempt               = true            {product}
    uintx G1HeapRegionSize                          = 0               {product}
     intx InterpreterProfilePercentage              = 33              {product}
     intx PreBlockSpin                              = 10              {product}
     intx PreInflateSpin                            = 10              {pd product}

The query to find the total number of databases can be found by executing the following query:


select count(*) from information_schema.SCHEMATA where schema_name not in
('mysql','information_schema');

Rationale

Sampling tools like oprofile or dtrace’s profile provider don’t really provide methods to see what [multithreaded] programs are blocking on – only where they spend CPU time. Though there exist advanced techniques (such as systemtap and dtrace call level probes), it is overkill to build upon that. Poor man doesn’t have time. Poor man needs food.

Method

For a poor developer to understand what a program is doing, he needs to see stacks. Once upon a time (back in Linux 2.4) there was a ‘pstack’ tool for that, Solaris has it too.

Modern Linux systems though do not have such facilities, and one needs to improvise, like.. use debuggers – they can walk threads and provide stacks.

Technology

Getting stacks:

gdb -ex "set pagination 0" -ex "thread apply all bt" \
  --batch -p $(pidof mysqld)

Or for version-impaired (gdb 6.3 and older):

(echo "set pagination 0";
 echo "thread apply all bt";
 echo "quit"; cat /dev/zero ) | gdb -p $(pidof mysqld)

Collapsing traces (awk!):

BEGIN { s = ""; }
/Thread/ { print s; s = ""; }
/^\#/ { if (s != "" ) { s = s "," $4} else { s = $4 } }
END { print s }

Full technology demonstration:

#!/bin/bash
nsamples=1
sleeptime=0
pid=$(pidof mysqld)

for x in $(seq 1 $nsamples)
  do
    gdb -ex "set pagination 0" -ex "thread apply all bt" -batch -p $pid
    sleep $sleeptime
  done | \
awk '
  BEGIN { s = ""; }
  /Thread/ { print s; s = ""; }
  /^\#/ { if (s != "" ) { s = s "," $4} else { s = $4 } }
  END { print s }' | \
sort | uniq -c | sort -r -n -k 1,1

Output

291 pthread_cond_wait@@GLIBC_2.3.2,one_thread_per_connection_end,handle_one_connection
 57 read,my_real_read,my_net_read,do_command,handle_one_connection,start_thread
 26 pthread_cond_wait@@GLIBC_2.3.2,os_event_wait_low,os_aio_simulated_handle,fil_aio_wait,io_handler_thread,start_thread
  3 pthread_cond_wait@@GLIBC_2.3.2,os_event_wait_low,srv_purge_worker_thread
  1 select,os_thread_sleep,srv_purge_thread
  1 select,os_thread_sleep,srv_master_thread
  1 select,os_thread_sleep,srv_lock_timeout_and_monitor_thread
  1 select,os_thread_sleep,srv_error_monitor_thread
  1 select,handle_connections_sockets,main,select
  1 read,vio_read_buff,my_real_read,my_net_read,cli_safe_read,handle_slave_io
  1 pthread_cond_wait@@GLIBC_2.3.2,os_event_wait_low,sync_array_wait_event,rw_lock_s_lock_spin,buf_page_get_gen,btr_cur_search_to_nth_level,row_search_for_mysql,ha_innodb::index_read,handler::index_read_idx_map,join_read_const,join_read_const_table,make_join_statistics,JOIN::optimize,mysql_select,handle_select,execute_sqlcom_select,mysql_execute_command,mysql_parse,dispatch_command,do_command,handle_one_connection
  1 pread64,os_file_pread,os_file_read,fil_io,buf_read_page_low,buf_read_page,buf_page_get_gen,btr_cur_search_to_nth_level,row_search_index_entry,row_upd_step,row_update_for_mysql,ha_innodb::delete_row,handler::ha_delete_row,mysql_delete,mysql_execute_command,mysql_parse,Query_log_event::do_apply_event,apply_event_and_update_pos,handle_slave_sql
  1 pread64,os_file_pread,os_file_read,fil_io,buf_read_page_low,buf_read_page,buf_page_get_gen,btr_cur_search_to_nth_level,row_search_for_mysql,ha_innodb::index_read,handler::index_read_idx_map,join_read_const,join_read_const_table,make_join_statistics,JOIN::optimize,mysql_select,handle_select,execute_sqlcom_select,mysql_execute_command,mysql_parse,dispatch_command,do_command,handle_one_connection
  1 do_sigwait,sigwait,signal_hand

Success stories and references

We hear this technology has been used by performance engineers at Google, Facebook, Wikipedia, Intel, Sun Microsystems and other places.

• Baron Schwartz
• Mikael Ronstrom
• Mark Callaghan

Credits

Originally this technology was released as a collaborative effort by Mark Callaghan and Domas Mituzas at this blog post.

Extract RPM file using rpm2cpio and cpio command:

$ rpm2cpio any_rpm_package.x86_64.rpm | cpio -idmv


How do I check RAM speed and type (line DDR or DDR2) without opening my computer? I need to purchase RAM and I need to know the exact speed and type installed. How do I find out ram information from a shell prompt?

$ sudo dmidecode --type 17


dmidecode is a tool for dumping a computer’s DMI (some say SMBIOS) table contents in a human-readable format. This table contains a description of the system’s hardware components, as well as other useful pieces of information such as serial numbers and BIOS revision. Thanks to this table, you can retrieve this information without having to probe for the actual hardware. While this is a good point in terms of report speed and safeness, this also makes the presented information possibly unreliable.

G1 collector is concurrent and parallel with more predictability and better useability. It splits the heap into regions and young and old generational spaces are organized in sets of regions. This eliminates the need to fine tune “NewSize/MaxNewSize”. G1 also provides incremental compaction in those regions by “evacuating” the regions that are full and fragmented. To try this G1 collector, you’ll need the aforementioned Fix Pack/Update and the following JVM options:

-XX:+UnlockExperimentalVMOptions -XX:+UseG1GC

In addition, the following options can be used to adjust G1′s characteristics:

To set the max GC pause time goal in milliseconds that G1 will attempt to meet:

-XX:MaxGCPauseMillis

To set the time interval over which GC pauses totaling up to MaxGCPauseMillis may take place:

-XX:GCPauseIntervalMillis

For more information about G1 (how it operates, more options, etc.) see Java HotSpot Garbage Collection.



Hung, Deadlocked, or Looping Process

• Print thread stack for all Java threads:
  • Control-\
  • kill -QUIT pid
  • jstack pid (or jstack -F pid if jstack pid does not respond)
• Detect deadlocks:
  • Request deadlock detection: JConsole tool, Threads tab
  • Print information on deadlocked threads: Control-\
  • Print list of concurrent locks owned by each thread: -XX:+PrintConcurrentLocks set, then Control-\
  • Print lock information for a process: jstack -l pid
• Get a heap histogram for a process:
  • Start Java process with -XX:+PrintClassHistogram, then Control-\
  • jmap -histo pid (with -F option if pid does not respond)
• Dump Java heap for a process in binary format to file:
  • jmap -dump:format=b,file=filename pid (with -F option if pid does not respond)
• Print shared object mappings for a process:
  • jmap pid
• Print heap summary for a process:
  • Control-\
  • jmap -heap pid
• Print finalization information for a process:
  • jmap -finalizerinfo pid
• Attach the command-line debugger to a process:
  • jdb -connect sun.jvm.hotspot.jdi.SAPIDAttachingConnector:pid=pid

Post-mortem Diagnostics, Memory Leaks

• Examine the fatal error log file. Default file name is hs_err_pidpid.log in the working-directory.
• Create a heap dump:
  • Start the application with HPROF enabled: java -agentlib:hprof=file=file,format=bapplication; then Control-\
  • Start the application with HPROF enabled: java -agentlib:hprof=heap=dump application
  • JConsole tool, MBeans tab
  • Start VM with -XX:+HeapDumpOnOutOfMemoryError; if OutOfMemoryError is thrown, VM generates a heap dump.
• Browse Java heap dump:
  • jhat heap-dump-file
• Dump Java heap from core file in binary format to a file:
  • jmap -dump:format=b,file=filename corefile
• Get a heap histogram for a process:
  • Start Java process with -XX:+PrintClassHistogram, then Control-\
  • jmap -histo pid (with -F option if pid does not respond)
• Get a heap histogram from a core file:
  • jmap -histo corefile
• Print shared object mappings from a core file:
  • jmap corefile
• Print heap summary from a core file:
  • jmap -heap corefile
• Print finalization information from a core file:
  • jmap -finalizerinfo corefile
• Print Java configuration information from a core file:
  • jinfo corefile
• Print thread trace from a core file:
  • jstack corefile
• Print lock information from a core file:
  • jstack -l corefile
• Attach the command-line debugger to a core file on the same machine:
  • jdb -connect sun.jvm.hotspot.jdi.SACoreAttachingConnector:javaExecutable=path,core=corefile
• Attach the command-line debugger to a core file on a different machine:
  • On the machine with the core file: jsadebugd path corefile
    and on the machine with the debugger: jdb -connect sun.jvm.hotspot.jdi.SADebugServerAttachingConnector:debugServerName=machine
• libumem can be used to debug memory leaks.

The Java Virtual Machine (JVM)* has the following types of memory: heap, non-heap, and native.

Heap memory is the runtime data area from which memory for all class instances and arrays is allocated. Non-heap memory includes the method area and memory required for the internal processing or optimization of the JVM. It stores per-class structures such as a runtime constant pool, field and method data, and the code for methods and constructors. Native memory is the virtual memory managed by the operating system. When the memory is insufficient for an application to allocate, a java.lang.OutOfMemoryError will be thrown.

Following are the possible error messages for OutOfMemoryErrors in each type of memory:

Heap memory error: When an application creates a new object but the heap does not have sufficient space and cannot be expanded further, an OutOfMemoryError will be thrown with the following error message:

java.lang.OutOfMemoryError: Java heap space

Non-heap memory error: The permanent generation is a non-heap memory area in the HotSpot VM implementation that stores per-class structures as well as interned strings. When the permanent generation is full, the application will fail to load a class or to allocate an interned string, and an OutOfMemoryError will be thrown with the following error message:

java.lang.OutOfMemoryError: PermGen space

Native memory error: The Java Native Interface (JNI) code or the native library of an application and the JVM implementation allocate memory from the native heap. An OutOfMemoryError will be thrown when an allocation in the native heap fails. For example, the following error message indicates insufficient swap space, which could be caused by a configuration issue in the operating system or by another process in the system that is consuming much of the memory:

java.lang.OutOfMemoryError: request bytes for .
Out of swap space?

An insufficient memory problem could be due either to a problem with the configuration — the application really needs that much memory — or to a performance problem in the application that requires you to profile and optimize to reduce the memory use.

You can find more information at http://java.sun.com/developer/technicalArticles/J2SE/monitoring/

PermGen holds the metadata about classes that have been loaded/created. This information is garbage collected like the other parts of the heap, however there are rough edges that can prevent this from happening, class loaders in particular. Generally, the amount of PermGen space needed is small in relation to the rest of the heap.

How do I know what classes are being loaded or unloaded? Use the command line options

-XX:+TraceClassLoading and -XX:+TraceClassUnloading .

If you suspect that that classloader isn’t the issue, use -verbose:class to investigate classes which have been loaded. This, as it suggests, will be very verbose.

How do you increase PermGen space?

Increase using the -XX:MaxPermSize option.

ConcurrentHashMap is hash table supporting full concurrency of retrievals and adjustable expected concurrency for updates. I recently came across this code during testing, and one part really got my attention. To generate the hash, ConcurrentHashMap uses an algorithm based on bitshifting and bitwise operations.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

========================================
Variant of single-word Wang/Jenkins hash
========================================
private static int hash(int h) {
 
// Spread bits to regularize both segment and index locations,
// using variant of single-word Wang/Jenkins hash.
h += (h <<  15) ^ 0xffffcd7d;
h ^= (h >>> 10);
h += (h <<   3);
h ^= (h >>>  6);
h += (h <<   2) + (h << 14);
 
return h ^ (h >>> 16);
}

According to the comment in the code, this method applies a supplemental hash function to a given hashCode, which defends against poor quality hash functions.

Good hash functions are important as a hash table effectively turns from a map to a linked list, in the worst case, all keys in the same bucket. There are also other considerations that come into play such as the performance of hash calculation and the number of buckets. Dr. Heinz M. Kabutz explains the power of “power-of-two number of buckets” which gives us some good starting point to understand what is really going on here.

Let’s look at the code above and see how things change, line-by-line. To make things simple, I use int 1 to perform all the operations.

In Java, the int data type is a 32-bit signed two’s complement integer. To represent int 1 in binary code, we have the following:

1

h=1 > 0000-0000-0000-0000-0000-0000-0001

Now, let’s dissect the following line:

1

h += (h << 15) ^ 0xffffcd7d

First, let’s re-write this into an easier-to-read format.. at least for me .

1
2
3
4

h1 = h << 15      =  0000-0000-0000-0000-1000-0000-0000-0000
hex = 0xffffcd7d  =  1111-1111-1111-1111-1100-1101-0111-1101
h2 = h1 ^ hex     =  1111-1111-1111-1111-0100-1101-0111-1101
h2 + h            =  1111-1111-1111-1111-0100-1101-0111-1110

Using the same thought processing and applying it to each line, we end-up with:

1
2
3
4
5
6

h += (h << 15) ^ 0xffffcd7d = 1111-1111-1111-1111-0100-1101-0111-1110
h ^= (h >>> 10)	            = 1111-1111-1100-0000-1011-0010-1010-1101
h += (h << 3)		    = 1111-1101-1100-0110-0100-1000-0001-0101
h ^= (h >>> 6)              = 1111-1110-0011-0001-0101-0001-0011-0101
h += (h << 2) + (h << 14)   = 0100-1011-0100-0011-1101-0110-0000-1001
h ^= (h >>> 16)             = 0100-1011-0100-0011-1001-1101-0100-1010

Result:
Bin = 0100-1011-0100-0011-1001-1101-0100-1010
Decimal = 1,262,722,378