HBase Memstore Flush - Part 1

Aim:

Aim of this blog is to discuss various scenarios which will lead to memstore flushes in HBase.

Any put operation to HBase goes to memstore (in memory). It is also written to WAL by default. There is one memstore per column family per region per regionserver per HBase table. When certain threshold is reached memstore is flushed.

The threshold can be mainly categorized into two:
[A] Size based
[B] Time based

This blog focuses on size based memstore flushes. My next blog (HBase Memstore Flush - Part 2) discusses about time based memstore flushes.

Size based memstore flushes:

Since memstore is in memory and is part of regionserver memory, it is flushed when it reaches a certain threshold.

The threshold is controlled by following parameters:

[a] hbase.hregion.memstore.flush.size (specified in bytes)

Each memstore is checked for this threshold periodically (determined by 'hbase.server.thread.wakefrequency'). If the memstore hits this limit, it will be flushed. Please note that every memstore flush creates one HFile per CF per region.

[b] Regionserver might have many regions managed by it. Since memstore uses heap memory of regionserver, we also need to control the total heap memory used by all the memstores. This is controlled by following parameters:

(1) hbase.regionserver.global.memstore.size.lower.limit 
Maximum size of all memstores in a region server before flushes are forced. Defaults to 95% of hbase.regionserver.global.memstore.size (0.95). hbase.regionserver.global.memstore.lowerLimit is old property for the same. It will be honored if specified.

(2) hbase.regionserver.global.memstore.size
Maximum size of all memstores in a region server before new updates are blocked and flushes are forced. Defaults to 40% of heap (0.4). Updates are blocked and flushes are forced until size of all memstores in a region server hits hbase.regionserver.global.memstore.size.lower.limit. hbase.regionserver.global.memstore.upperLimit is the old property for the same. It will be honored if specified.


Few other parameters that needs to be taken care are the following:
* hbase.hregion.memstore.block.multiplier - Updates are blocked if memstore has hbase.hregion.memstore.block.multiplier times hbase.hregion.memstore.flush.size bytes. The default value is 4.

* hbase.hregion.percolumnfamilyflush.size.lower.bound - If FlushLargeStoresPolicy is used, then every time that we hit the total memstore limit, we find out all the column families whose memstores exceed this value, and only flush them, while retaining the others whose memstores are lower than this limit. If none of the families have their memstore size more than this, all the memstores will be flushed (just as usual). This value should be less than half of the total memstore threshold (hbase.hregion.memstore.flush.size). (https://issues.apache.org/jira/browse/HBASE-10201). To restore the old behavior of flushes writing out all column families, set hbase.regionserver.flush.policy to org.apache.hadoop.hbase.regionserver.FlushAllStoresPolicy either in hbase-default.xml or on a per-table basis by setting the policy to use with HTableDescriptor.getFlushPolicyClassName().