The query to find the total number of databases can be found by executing the following query:
select count(*) from information_schema.SCHEMATA where schema_name not in
('mysql','information_schema');
Sampling tools like oprofile or dtrace’s profile provider don’t really provide methods to see what [multithreaded] programs are blocking on – only where they spend CPU time. Though there exist advanced techniques (such as systemtap and dtrace call level probes), it is overkill to build upon that. Poor man doesn’t have time. Poor man needs food.
For a poor developer to understand what a program is doing, he needs to see stacks. Once upon a time (back in Linux 2.4) there was a ‘pstack’ tool for that, Solaris has it too.
Modern Linux systems though do not have such facilities, and one needs to improvise, like.. use debuggers – they can walk threads and provide stacks.
Getting stacks:
gdb -ex "set pagination 0" -ex "thread apply all bt" \ --batch -p $(pidof mysqld)
Or for version-impaired (gdb 6.3 and older):
(echo "set pagination 0"; echo "thread apply all bt"; echo "quit"; cat /dev/zero ) | gdb -p $(pidof mysqld)
Collapsing traces (awk!):
BEGIN { s = ""; }
/Thread/ { print s; s = ""; }
/^\#/ { if (s != "" ) { s = s "," $4} else { s = $4 } }
END { print s }
Full technology demonstration:
#!/bin/bash
nsamples=1
sleeptime=0
pid=$(pidof mysqld)
for x in $(seq 1 $nsamples)
do
gdb -ex "set pagination 0" -ex "thread apply all bt" -batch -p $pid
sleep $sleeptime
done | \
awk '
BEGIN { s = ""; }
/Thread/ { print s; s = ""; }
/^\#/ { if (s != "" ) { s = s "," $4} else { s = $4 } }
END { print s }' | \
sort | uniq -c | sort -r -n -k 1,1
291 pthread_cond_wait@@GLIBC_2.3.2,one_thread_per_connection_end,handle_one_connection 57 read,my_real_read,my_net_read,do_command,handle_one_connection,start_thread 26 pthread_cond_wait@@GLIBC_2.3.2,os_event_wait_low,os_aio_simulated_handle,fil_aio_wait,io_handler_thread,start_thread 3 pthread_cond_wait@@GLIBC_2.3.2,os_event_wait_low,srv_purge_worker_thread 1 select,os_thread_sleep,srv_purge_thread 1 select,os_thread_sleep,srv_master_thread 1 select,os_thread_sleep,srv_lock_timeout_and_monitor_thread 1 select,os_thread_sleep,srv_error_monitor_thread 1 select,handle_connections_sockets,main,select 1 read,vio_read_buff,my_real_read,my_net_read,cli_safe_read,handle_slave_io 1 pthread_cond_wait@@GLIBC_2.3.2,os_event_wait_low,sync_array_wait_event,rw_lock_s_lock_spin,buf_page_get_gen,btr_cur_search_to_nth_level,row_search_for_mysql,ha_innodb::index_read,handler::index_read_idx_map,join_read_const,join_read_const_table,make_join_statistics,JOIN::optimize,mysql_select,handle_select,execute_sqlcom_select,mysql_execute_command,mysql_parse,dispatch_command,do_command,handle_one_connection 1 pread64,os_file_pread,os_file_read,fil_io,buf_read_page_low,buf_read_page,buf_page_get_gen,btr_cur_search_to_nth_level,row_search_index_entry,row_upd_step,row_update_for_mysql,ha_innodb::delete_row,handler::ha_delete_row,mysql_delete,mysql_execute_command,mysql_parse,Query_log_event::do_apply_event,apply_event_and_update_pos,handle_slave_sql 1 pread64,os_file_pread,os_file_read,fil_io,buf_read_page_low,buf_read_page,buf_page_get_gen,btr_cur_search_to_nth_level,row_search_for_mysql,ha_innodb::index_read,handler::index_read_idx_map,join_read_const,join_read_const_table,make_join_statistics,JOIN::optimize,mysql_select,handle_select,execute_sqlcom_select,mysql_execute_command,mysql_parse,dispatch_command,do_command,handle_one_connection 1 do_sigwait,sigwait,signal_hand
We hear this technology has been used by performance engineers at Google, Facebook, Wikipedia, Intel, Sun Microsystems and other places.
Originally this technology was released as a collaborative effort by Mark Callaghan and Domas Mituzas at this blog post.
Extract RPM file using rpm2cpio and cpio command:
$ rpm2cpio any_rpm_package.x86_64.rpm | cpio -idmv
How do I check RAM speed and type (line DDR or DDR2) without opening my computer? I need to purchase RAM and I need to know the exact speed and type installed. How do I find out ram information from a shell prompt?
$ sudo dmidecode --type 17
dmidecode is a tool for dumping a computer’s DMI (some say SMBIOS) table contents in a human-readable format. This table contains a description of the system’s hardware components, as well as other useful pieces of information such as serial numbers and BIOS revision. Thanks to this table, you can retrieve this information without having to probe for the actual hardware. While this is a good point in terms of report speed and safeness, this also makes the presented information possibly unreliable.
G1 collector is concurrent and parallel with more predictability and better useability. It splits the heap into regions and young and old generational spaces are organized in sets of regions. This eliminates the need to fine tune “NewSize/MaxNewSize”. G1 also provides incremental compaction in those regions by “evacuating” the regions that are full and fragmented. To try this G1 collector, you’ll need the aforementioned Fix Pack/Update and the following JVM options:
-XX:+UnlockExperimentalVMOptions -XX:+UseG1GC
In addition, the following options can be used to adjust G1′s characteristics:
To set the max GC pause time goal in milliseconds that G1 will attempt to meet:
-XX:MaxGCPauseMillis
To set the time interval over which GC pauses totaling up to MaxGCPauseMillis may take place:
-XX:GCPauseIntervalMillis
For more information about G1 (how it operates, more options, etc.) see Java HotSpot Garbage Collection.
The Java Virtual Machine (JVM)* has the following types of memory: heap, non-heap, and native.
Heap memory is the runtime data area from which memory for all class instances and arrays is allocated. Non-heap memory includes the method area and memory required for the internal processing or optimization of the JVM. It stores per-class structures such as a runtime constant pool, field and method data, and the code for methods and constructors. Native memory is the virtual memory managed by the operating system. When the memory is insufficient for an application to allocate, a java.lang.OutOfMemoryError will be thrown.
Following are the possible error messages for OutOfMemoryErrors in each type of memory:
Heap memory error: When an application creates a new object but the heap does not have sufficient space and cannot be expanded further, an OutOfMemoryError will be thrown with the following error message:
java.lang.OutOfMemoryError: Java heap space
Non-heap memory error: The permanent generation is a non-heap memory area in the HotSpot VM implementation that stores per-class structures as well as interned strings. When the permanent generation is full, the application will fail to load a class or to allocate an interned string, and an OutOfMemoryError will be thrown with the following error message:
java.lang.OutOfMemoryError: PermGen space
Native memory error: The Java Native Interface (JNI) code or the native library of an application and the JVM implementation allocate memory from the native heap. An OutOfMemoryError will be thrown when an allocation in the native heap fails. For example, the following error message indicates insufficient swap space, which could be caused by a configuration issue in the operating system or by another process in the system that is consuming much of the memory:
java.lang.OutOfMemoryError: request
Out of swap space?
An insufficient memory problem could be due either to a problem with the configuration — the application really needs that much memory — or to a performance problem in the application that requires you to profile and optimize to reduce the memory use.
You can find more information at http://java.sun.com/developer/technicalArticles/J2SE/monitoring/
PermGen holds the metadata about classes that have been loaded/created. This information is garbage collected like the other parts of the heap, however there are rough edges that can prevent this from happening, class loaders in particular. Generally, the amount of PermGen space needed is small in relation to the rest of the heap.
How do I know what classes are being loaded or unloaded? Use the command line options
-XX:+TraceClassLoading and -XX:+TraceClassUnloading .
If you suspect that that classloader isn’t the issue, use -verbose:class to investigate classes which have been loaded. This, as it suggests, will be very verbose.
How do you increase PermGen space?
Increase using the -XX:MaxPermSize option.
ConcurrentHashMap is hash table supporting full concurrency of retrievals and adjustable expected concurrency for updates. I recently came across this code during testing, and one part really got my attention. To generate the hash, ConcurrentHashMap uses an algorithm based on bitshifting and bitwise operations.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | ======================================== Variant of single-word Wang/Jenkins hash ======================================== private static int hash(int h) { // Spread bits to regularize both segment and index locations, // using variant of single-word Wang/Jenkins hash. h += (h << 15) ^ 0xffffcd7d; h ^= (h >>> 10); h += (h << 3); h ^= (h >>> 6); h += (h << 2) + (h << 14); return h ^ (h >>> 16); } |
According to the comment in the code, this method applies a supplemental hash function to a given hashCode, which defends against poor quality hash functions.
Good hash functions are important as a hash table effectively turns from a map to a linked list, in the worst case, all keys in the same bucket. There are also other considerations that come into play such as the performance of hash calculation and the number of buckets. Dr. Heinz M. Kabutz explains the power of “power-of-two number of buckets” which gives us some good starting point to understand what is really going on here.
Let’s look at the code above and see how things change, line-by-line. To make things simple, I use int 1 to perform all the operations.
In Java, the int data type is a 32-bit signed two’s complement integer. To represent int 1 in binary code, we have the following:
1 | h=1 > 0000-0000-0000-0000-0000-0000-0001 |
Now, let’s dissect the following line:
1 | h += (h << 15) ^ 0xffffcd7d |
First, let’s re-write this into an easier-to-read format.. at least for me 
.
1 2 3 4 | h1 = h << 15 = 0000-0000-0000-0000-1000-0000-0000-0000 hex = 0xffffcd7d = 1111-1111-1111-1111-1100-1101-0111-1101 h2 = h1 ^ hex = 1111-1111-1111-1111-0100-1101-0111-1101 h2 + h = 1111-1111-1111-1111-0100-1101-0111-1110 |
Using the same thought processing and applying it to each line, we end-up with:
1 2 3 4 5 6 | h += (h << 15) ^ 0xffffcd7d = 1111-1111-1111-1111-0100-1101-0111-1110 h ^= (h >>> 10) = 1111-1111-1100-0000-1011-0010-1010-1101 h += (h << 3) = 1111-1101-1100-0110-0100-1000-0001-0101 h ^= (h >>> 6) = 1111-1110-0011-0001-0101-0001-0011-0101 h += (h << 2) + (h << 14) = 0100-1011-0100-0011-1101-0110-0000-1001 h ^= (h >>> 16) = 0100-1011-0100-0011-1001-1101-0100-1010 |
Result:
Bin = 0100-1011-0100-0011-1001-1101-0100-1010
Decimal = 1,262,722,378
Conversion Table
| HEX | DECIMAL | BINARY |
| 0 | 0 = 0+0+0+0 | 0000 |
| 1 | 1 = 0+0+0+1 | 0001 |
| 2 | 2 = 0+0+2+0 | 0010 |
| 3 | 3 = 0+0+2+1 | 0011 |
| 4 | 4 = 0+2+0+0 | 0100 |
| 5 | 5 = 0+4+0+1 | 0101 |
| 6 | 6 = 0+4+2+0 | 0110 |
| 7 | 7 = 0+4+2+1 | 0111 |
| 8 | 8 = 8+0+0+0 | 1000 |
| 9 | 9 = 8+0+0+1 | 1001 |
| A | 10 = 8+0+2+0 | 1010 |
| B | 11 = 8+0+0+1 | 1011 |
| C | 12 = 8+4+0+0 | 1100 |
| D | 13 = 8+4+0+1 | 1101 |
| E | 14 = 8+4+2+0 | 1110 |
| F | 15 = 8+4+2+1 | 1111 |
Data Types and Data Structures
| Primitive Type | Size | Minimum Value | Maximum Value |
| char | 16-bit | Unicode 0 | Unicode 216-1 |
| byte | 8-bit | -128 | +127 |
| short | 16-bit | -215 (-32,768) |
+215-1 (32,767) |
| int | 32-bit | -231 (-2,147,483,648) |
+231-1 (2,147,483,647) |
| long | 64-bit | -263 (-9,223,372,036,854,775,808) |
+263-1 (9,223,372,036,854,775,807) |
| float | 32-bit | 32-bit IEEE 754 floating-point numbers | |
| double | 64-bit | 64-bit IEEE 754 floating-point numbers | |
| boolean | 1-bit | true or false | |
| void | —– | —– | Void |