Remote mirroring between two secured MapR cluster

AIM

To setup remote mirroring between two secured MapR cluster. (In the example given below both source and destination cluster has single node.)

In source Cluster:

Cluster name: ajames

Securing the source cluster

/opt/mapr/server/configure.sh -C <source_cluster> -Z <source_cluster> -N ajames -secure -genkeys

Check the cluster conf to verify cluster is secured

[root@c51-4 ~]# cat /opt/mapr/conf/mapr-clusters.conf

ajames secure=true <source_cluster>:7222

Change the permission of ‘ssl_keystore’ to 600

chmod 600 ssl_keystore

Start the zookeeper and warden services

service mapr-zookeeper start

service mapr-warden start

Login as mapr user

su mapr

Generate the maprlogin password

maprlogin password

maprcli node list -columns svc

[mapr@c51-4 conf]$ maprcli node list -columns svc

service                              hostname  ip

webserver,cldb,fileserver,hoststats  c51-4     <source_cluster> 

In destination Cluster:

Cluster name: ajames2

Securing the destination cluster

/opt/mapr/server/configure.sh -C <destination_cluster> -Z <destination_cluster> -N ajames2 -secure -genkeys

Check the cluster conf to verify cluster is secured

[root@c51-4 ~]# cat /opt/mapr/conf/mapr-clusters.conf

ajames2 secure=true <destination_cluster>:7222

Change the permission of ‘ssl_keystore’ to 600

chmod 600 ssl_keystore

Start the zookeeper and warden services

service mapr-zookeeper start

service mapr-warden start

Login as mapr user

su mapr

Generate the maprlogin password

maprlogin password

[mapr@c51-5 root]$  maprcli node list -columns svc

service                              hostname  ip

webserver,cldb,fileserver,hoststats  c51-5     <destination_cluster>

In source cluster:

Merging ssl_truststore

Copy the ssl_truststore from source cluster to ‘/tmp’ of destination cluster.

scp /opt/mapr/conf/ssl_truststore root@<destination_cluster>:/tmp/

In destination cluster

As ‘root’ user

cd /tmp

chown mapr:mapr ssl_truststore

As ‘mapr’ user

/opt/mapr/server/manageSSLKeys.sh merge /tmp/ssl_truststore /opt/mapr/conf/ssl_truststore

cd /opt/mapr/conf

Add ‘ajames secure=true <source_cluster>:7222’ (details of source cluster) in

vim mapr-clusters.conf

cat mapr-clusters.conf

ajames2 secure=true <destination_cluster>:7222

ajames secure=true <source_cluster>:7222

Restart the ‘webserver’

maprcli node services -action restart -name webserver -nodes <destination_cluster>

In the source cluster

cd /opt/mapr/conf

Add ‘ajames2 secure=true <destination_cluster>:7222’ (details of source cluster) in

vim mapr-clusters.conf

cat mapr-clusters.conf

ajames secure=true <source_cluster>:7222

ajames2 secure=true <destination_cluster>:7222

Restart the ‘webserver’

maprcli node services -action restart -name webserver -nodes <source_cluster>

In source cluster:

Create the ‘crosscluster’ ticket

maprcli security getmaprclusterticket -clusterusername mapr -inmaprserverticketfile /opt/mapr/conf/maprserverticket -ticketfile /opt/mapr/conf/maprclusterticket

In destination cluster:

Copy and append the ticket to the destination cluster in maprserverticket

maprlogin password -cluster ajames

Check whether both clusters are accessible from the destination cluster

hadoop fs -ls /mapr/ajames

[mapr@c51-5 root]$ hadoop fs -ls /mapr/ajames

Found 8 items

drwxr-xr-x   - mapr mapr          0 2016-07-25 19:28 /mapr/ajames/apps

drwxr-xr-x   - mapr mapr          0 2016-07-25 19:27 /mapr/ajames/hbase

drwxr-xr-x   - mapr mapr          0 2016-07-25 19:29 /mapr/ajames/opt

drwxrwxrwx   - mapr mapr          0 2016-07-25 19:27 /mapr/ajames/tmp

drwxr-xr-x   - mapr mapr          0 2016-07-25 19:29 /mapr/ajames/user

drwxr-xr-x   - mapr mapr          1 2016-07-25 19:28 /mapr/ajames/var

hadoop fs -ls /mapr/ajames2

[mapr@c51-5 root]$ hadoop fs -ls /mapr/ajames2

Found 9 items

drwxr-xr-x   - mapr mapr          0 2016-07-25 21:09 /mapr/ajames2/apps

drwxr-xr-x   - mapr mapr          0 2016-07-25 21:09 /mapr/ajames2/hbase

drwxr-xr-x   - mapr mapr          0 2016-07-25 21:11 /mapr/ajames2/opt

drwxrwxrwx   - mapr mapr          0 2016-07-25 21:09 /mapr/ajames2/tmp

drwxr-xr-x   - mapr mapr          0 2016-07-25 21:11 /mapr/ajames2/user

drwxr-xr-x   - mapr mapr          1 2016-07-25 21:10 /mapr/ajames2/var

In source cluster:

Create a volume at source cluster ‘test1’

Create a file in the volume

hadoop fs -touchz /test/abc.txt

 

In destination cluster:

From destination cluster create the remote mirror from CLI

Example: maprcli volume create -name testmirror -path /testmirror -type mirror -source test@ajames -quota 50G -advisoryquota 40G -schedule 3 -mirrorschedule 3 -topology /data

Login to MCS and start mirroring to test the same.

Sqoop – Tibero Integration

Version:

Sqoop  - 1.4.1

Tibero - 6

Tibero installation in UNIX system:

Prepare the following two files. (You can get trial edition in http://www.tmaxsoft.com/)

·        Binary executable file (tar.gz)

·        License file (license.xml)

Extract the tar:

tar -xvf tibero5-bin-5_rel_FS05-linux64-127068-opt-tested.tar.gz

Create a folder ‘license’ in the tibero home and copy the license.xml

Execute the following:

./Tibero_6_FS04_linux6420160418.bin

Check that the environment variables are properly set in the environment configuration files (.bashrc, .bash_profile, .profile, etc.) for each user account for each OS. Sample given below:

#Created by Tibero Installer. You should manually delete below variables for uninstall.

export TB_HOME=/root/TmaxData/tibero6

export TB_SID=tibero

export PATH=/usr/xpg4/bin:/usr/xpg4/bin:/sbin:/bin:/usr/sbin:/usr/bin:/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:/root/TmaxData/tibero6/bin:/root/TmaxData/tibero6/client/bin:/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:/root/TmaxData/tibero6/bin:/root/TmaxData/tibero6/client/bin

export LD_LIBRARY_PATH=/root/TmaxData/tibero6/lib:/root/TmaxData/tibero6/client/lib

###############################################################################

 

Execute the following command in $TB_HOME/config.

./gen_tip.sh

 

Start the database in ‘NOMOUNT’ mode from $TB_HOME:

bin/tbboot nomount

Create a database using the CREATE DATABASE statement. (Very basic)

create database "tibero"

  user sys identified by tibero;

Restart Tibero using the tbboot command.

Execute system.sh command in $TB_HOME/scripts to execute the SQL file in the current directory. (The default passwords for the sys and syscat account are tibero and syscat, respectively.)

Start the tibero database client:

client/bin/tbsql sys/tibero

 

Checking the tibero services

[root@c51-3 ~]# ps -ef | grep tbsvr

root     28167     1  0 Jul18 ?        00:04:10 tbsvr          -t NORMAL -SVR_SID tibero

root     28169 28167  0 Jul18 ?        00:00:00 tbsvr_TBMP     -t NORMAL -SVR_SID tibero

root     28170 28167  0 Jul18 ?        00:00:00 tbsvr_WP000    -t NORMAL -SVR_SID tibero

root     28171 28167  0 Jul18 ?        00:00:00 tbsvr_WP001    -t NORMAL -SVR_SID tibero

root     28172 28167  0 Jul18 ?        00:39:37 tbsvr_AGNT     -t NORMAL -SVR_SID tibero

root     28173 28167  0 Jul18 ?        00:10:02 tbsvr_DBWR     -t NORMAL -SVR_SID tibero

root     28174 28167  0 Jul18 ?        00:00:00 tbsvr_RECO     -t NORMAL -SVR_SID tibero

root     31398 17165  0 22:39 pts/1    00:00:00 grep tbsvr

Integrating Sqoop

Copy tibero6-jdbc-14.jar to /opt/mapr/sqoop/sqoop-1.4.6/lib/

Importing data from Tibero:

sqoop import --driver com.tmax.tibero.jdbc.TbDriver --connect jdbc:tibero:thin:@localhost:8629:tibero --username sys --password tibero --table emp -m 1;

Kafka Tips

• Distributed publish-subscribe messaging system
• The messages are maintained in ‘topics’
• Messages are coming as ‘byte arrays’.
• ‘Producers’ write data to ‘topics’
• ‘Consumers’ pull data from ‘topics’
• ‘Topics’ are partitioned (partitions) and distributed across nodes
• Each ‘partition’ will have a single ‘leader’ and zero or more servers that act as ‘followers’
• ‘Producers’ can attach each message with a ‘key’ so that message with same key goes into the same partition, or it can simply write in ‘round robin’ fashion just for load balancing
• ‘Consumer Group’ – group of consumers
• They subscribe to a topic
• Each consumer in the group will consume from unique subset of partitions
• Each consumer will have all messages related to the same in key
• Each topic partition is considered to be ‘log’. (an ordered set of messages)
• Messages are tracked by offset
• Consumers are responsible for tracking the offset
• Messages will be deleted after a certain retention period
• Retention period is normally set for all topics together, however it can be set specific to a topic
• Kafka maintains order within a topic, but not across topics
• Kafka does not track acknowledgements, it also does not track messages per consumer
• It can handle batch consumers – process that wake up in a period of time and consumes all the messages
• Kafka uses ‘Zookeeper’ for cluster membership
• ‘Broker’ is a daemon running in each node of Kafka cluster
• ‘Broker ID’ is unique and permanent name for each node in the cluster – must be non-negative
• ‘isr’ is in-sync replicas
• There is a ‘max lag time ‘ parameter, and if the ‘followers’ do not send a fetch request within this period of time, then, the leader will remove the ‘follower’ from the list of ‘isr’ (Also, the ‘follower’ will treated as dead, if it falls more a set number of messages behind the leader)
• Communication happens using TCP protocol
• The log for a topic partition will be stored as a directory of segment files. (log.segment.bytes – controls the size of each segment. The ‘log’ will be rolled forcefully by Kafka in a certain time period even if it does not reach the size. The default period is 1 week)
• The default replication factor for automatically created topics is 1.
• Kafka is written in Scala

Some important properties: (kafka.utils.VerifiableProperties)

• log.retention.check.interval.ms - The frequency in milliseconds that the log cleaner checks whether any log segment is eligible for deletion to meet the retention policies
• log.retention.hours – The number of hours to keep the log segment before deleting it
• zookeeper.connect – The details of zookeeper. <hostname:port>, normally 2181. To allow connecting through other zookeeper nodes when the host is down, we can specify the details in a comma separated list
• Num.partitions – The default number of partitions per topic
• group.id – To identify the group of consumers. All the consumer process having the same id will belong to same group

Kafka Mirroring

• Feature to maintain replica of an existing Kafka cluster
• This is done via ‘MirrorMaker’ tool
• The tool uses Kafka Consumer to consume data from the source Kafka cluster and Kafka Producer to publish it to the target Kafka cluster
• ‘- - whitelist’  specifies the list of topics to be mirrored from the source cluster and ‘- - blacklist’ specifies the list of topics that should not be mirrored
• Check to see if mirror is keeping up – The ‘consumer offset checker’ tool can be used to measure how well our mirror is keeping up.

Basic Kafka Operations

Adding, Modifying and Removing Topics

• Topics are created either manually or automatically
• Manually topics can be added using :
• Example: bin/kafka-topics.sh --zookeeper zk_host:port/chroot --create --topic my_topic_name --partitions 20 --replication-factor 3 --config x=y
• Topics can be modified using : 
• Example: bin/kafka-topics.sh --zookeeper zk_host:port/chroot --alter --topic my_topic_name --partitions 40
• Kafka does not currently support reducing the number of partitions for a topic or changing the replication factor

Graceful shutdown

• Kafka will automatically detect any broker shutdown or failure and elect new leaders for the partitions on that machine.
• Advantages of graceful shutdown:
• Sync all the logs avoiding any need for log recovery
• Migrate all partitions the server is leader for to other replicas prior shutdown
• controlled.shutdown.enable=true should be set for graceful shutdown

Balancing leadership

• Whenever broker stops/crashes leadership for that broker’s partitions transfers to other replicas. This means, when the broker restarts it will be only the follower rather than being the leader. This might lead to imbalance in leadership. Hence, to avoid this Kafka has a concept of ‘preferred replica’. 
• Suppose we have a partition replicated in 3,5,6 and if 3 is the preferred replica, then we can have Kafka cluster restore its leadership by running the command:
• bin/kafka-preferred-replica-election.sh --zookeeper zk_host:port/chroot
• Or we can set the property in configuration:
• auto.leader.rebalance.enable=true

Expanding cluster

• Add a unique broker id
• Startup Kafka on new servers
• New servers will not be automatically be assigned any data partitions, so need to migrate data
• ‘Partition Reassignment’ tool can be used to move partitions across brokers

Enhancements in Java 8 – Java Programming

A.      Lambda Expression

Aka – anonymous function – not bound to an identifier

1.       Used as arguments to higher-order functions

2.       Used to construct the result of a higher-order function that needs to return a function

Functionality that need to be used only for short term – e.g.: ‘closures’ and ‘currying’

a = ['house', 'car', 'bike']

a.sort (lambda x, y: cmp (len(x), len(y)))

print (a)

['car', 'bike', 'house']

B.      Default Methods

Aka – defender function – allows new methods to be added to interfaces without breaking the functionality of existing interface.

Allows interface to use as default in the situation where a concrete class fails to provide implementation for that method.

public interface oldInterface {

    public void existingMethod();

        default public void newDefaultMethod() {

        System.out.println("New default method"

              " is added in interface");

    }

}

How we can use default methods?

ü  Not override the default method and will inherit the default method.

ü  Override the default method similar to other methods we override in subclass.

ü  Redeclare default method as abstract, which force subclass to override it.

C.      Repeating Annotations

Allows same annotations to be used more than once to same declaration or type use. Prior to Java 8, to have repeated annotations, we need to group them into an annotation container.

        @Manufactures({

        @Manufacturer(name =”BMW”),

        @Manufacturer(name = “Range Rover”)

        })

        public class Car{

        //code goes in here

        }

Now with Java 8, for repeating annotations, we need not specify an annotation container.

        @Manufacturer(name = “BMW”)

        @Manufacturer(name= “Range Rover”)

        public class Car{

        //code goes in here

}

D.      Type Annotation

Java 7 allowed annotations to be written only on method formal parameters and declarations of packages, classes, methods, fields and local variables. Type annotations are annotations that can be used anywhere we use a type.

Examples:

@NotNull String str1 = ...

@Email String str2 = ...

@NotNull @NotBlank String str3 = ...

Java annotations are used to provide meta data for your Java code. Annotations are used for compiler instructions, build-time instructions and runtime instructions.

Type annotations are provided for stronger type checking.

Java 8 introduces annotations to ‘use’ of types.

·         Class instance creation expression:

new @Intered MyObject();

·         Type cast

mystring = (@NonNull String) str;

·         implements clause

class UnmodifiableList<T> implements

        @Readonly List<@Readonly T> { ... }

·         Thrown exception declaration

void monitorTemperature() throws

        @Critical TemperatureException { ... }

E.       Method References

Shortcuts that can be used anywhere we use lambdas. They are compact and more readable   form of a lambda expression for already written methods.  “::” operator is used for method reference.

Example:

interface IsReferable {

                public void referenceDemo();

                }

class ReferenceDemo {

public static void commonMethod()

{

System.out.println("This method is already defined.");

}

public void implement()

{ // Anonymous class.

IsReferable demoOne = new IsReferable() {

@Override

public void referenceDemo() {

ReferenceDemo.commonMethod();

}

};

demoOne.referenceDemo();

// Lambda implementaion.

IsReferable demo = () -> ReferenceDemo.commonMethod();

demo.referenceDemo();

// Method reference.

IsReferable demoTwo = ReferenceDemo::commonMethod;

demoTwo.referenceDemo();

}

}

F.       Type Interface - Generics

It is possible to infer the generic types from method signatures when passing a constructor as a parameter of a method. Java 7 does not use ‘target typing’, but Java 8 does. Target typing is a language feature wherein the type of the variable in which a result is to be stored influences the type of the computation.

Target typing example:

long MICRO_SECONDS_IN_DAY = 24 × 60 × 60 × 1000 × 1000; à Java 7 does not return what is        expected.

Type interface example:

public class TypeInference {

  public int getDictionarySize(Map<String, String> theDict) {

    return theDict.size();

  }

}

@Test

public void emptyDictionarySizeShouldBeZero() {

  TypeInference testObj = new TypeInference();

  Map<String, String> dict = new HashMap<>();

  int expected = 0;

  int actual = testObj.getDictionarySize(dict);

  assertEquals("Size is incorrect!", expected, actual);

}

@Test

public void emptyDictionarySizeShouldBeZero2() {

  TypeInference testObj = new TypeInference();

  int expected = 0;

  int actual = testObj.getDictionarySize(new HashMap<>());  //Java 7: Compile Error.

                                                            //Java 8: Better Type Inference

  assertEquals("Size is incorrect!", expected, actual);

}

G.     Method parameter reflection

We can obtain the names of the formal parameters of any method constructor with the method java.lang.reflect.Executable.getParameters. However, .class files do not store formal parameter names by default. To store formal parameter names in a particular .class file, and thus enable the Reflection API to retrieve formal parameter names, compile the source file with the -parameters option of the javac compiler.

H.     Collections – The new ‘java.util.stream’ package

Classes in the new java.util.stream package provide a Stream API to support functional-style operations on streams of elements. The Stream API is integrated into the Collections API, which enables bulk operations on collections, such as sequential or parallel map-reduce transformations.

Example:

                    int sum = widgets.stream()

                      .filter(b -> b.getColor() == RED)

                      .mapToInt(b -> b.getWeight())

                      .sum();

Here we use widgets, a Collection<Widget>, as a source for a stream, and then perform a filter-map-reduce on the stream to obtain the sum of the weights of the red widgets.

The key abstraction introduced in this package is stream. The classes Stream, IntStream, LongStream, and DoubleStream are streams over objects and the primitive int, long and double types. Streams differ from collections in several ways:

No storage. A stream is not a data structure that stores elements; instead, it conveys elements from a source such as a data structure, an array, a generator function, or an I/O channel, through a pipeline of computational operations.

Functional in nature. An operation on a stream produces a result, but does not modify its source.

Laziness-seeking. Many stream operations, such as filtering, mapping, or duplicate removal, can be implemented lazily, exposing opportunities for optimization.

Possibly unbounded. While collections have a finite size, streams need not.

Consumable. The elements of a stream are only visited once during the life of a stream. Like an Iterator, a new stream must be generated to revisit the same elements of the source.

Stream operations are divided into intermediate and terminal operations, and are combined to form stream pipelines. A stream pipeline consists of a source (such as a Collection, an array, a generator function, or an I/O channel); followed by zero or more intermediate operations such as Stream.filter or Stream.map; and a terminal operation such asStream.forEach or Stream.reduce.

I.        Collections - Performance Improvement for HashMaps with Key Collisions

Hash bins containing a large number of colliding keys improve performance by storing their entries in a balanced tree instead of a linked list. This JDK 8 change applies only to HashMap, LinkedHashMap, and ConcurrentHashMap.

In rare situations, this change could introduce a change to the iteration order of HashMap and HashSet. A particular iteration order is not specified for HashMap objects - any code that depends on iteration order should be fixed.

Java 8 is on average 20% faster than Java 7 in simple HashMap.get(). When a bucket becomes too big (currently: TREEIFY_THRESHOLD = 8), HashMap dynamically replaces it with an ad-hoc implementation of tree map.

References:

1.       http://www.coderanch.com/t/621359/java/java/Difference-Java-Java-respect-Annotations

2.       https://docs.oracle.com/javase/tutorial/java/annotations/basics.html

3.       https://docs.oracle.com/javase/tutorial/java/annotations/predefined.html

4.       http://www.oracle.com/technetwork/java/javase/8-whats-new-2157071.html

5.       https://docs.oracle.com/javase/8/docs/technotes/guides/reflection/enhancements.html

6.       https://docs.oracle.com/javase/tutorial/reflect/member/methodparameterreflection.html

7.       http://blog.jooq.org/2013/11/25/a-lesser-known-java-8-feature-generalized-target-type-inference/

8.       http://docs.oracle.com/javase/8/docs/technotes/guides/language/enhancements.html

9.       http://www.jroller.com/rezaghp/entry/target_typing_in_java

10.   http://blog.decaresystems.ie/2014/04/09/better-type-inference-in-java-8-generics/

11.   https://docs.oracle.com/javase/8/docs/api/java/util/stream/package-summary.html

12.   http://java.dzone.com/articles/hashmap-performance

13.   https://docs.oracle.com/javase/8/docs/technotes/guides/collections/changes8.html

Apache HBase – Support for Medium Objects (MOBs)

Use case:

It is quite useful to save the binary data like images, documents into the HBase. The traditional database has the ability to save the MOB, for example, Oracle Database.

Apache HBase is technically designed to handle binary objects up to 10MB size. However, it is designed for storing data with <10K in each cell with low latency reads and writes. However, the performance can degrade when we use moderately sized objects (medium objects) 100K – 10MB. This is because of increasing I/O pressure created by compactions.

After Effects:

The increase in I/O pressure will lead to slower compactions, which eventually blocks memstore flushing and hence blocking updates. This will also increase the frequency of region splits reducing the availability of affected regions.

Characteristics of MOBs:

•        Write intensive
•        Data size is quite big
•        Seldom deletes and updates
•        Infrequent read (MOB data are accessed much less than the corresponding meta data)
•        Stored along with metadata

MOB Design:

The key is to treat MOBs as a separate region. This separates the MOBs from normal region splits and compactions thus decreasing the I/O pressure.

The idea is HBase + HDFS in managing the data. The memstore caches the MOB files before they are flushed onto the disk. The MOBs are written to HFile called ‘MOB’ file which may contain multiple MOB objects. The meta data is stored in HBase and there is a reference column that links to MOB file. The meta data and MOB are stored in different column families.

In order to take advantage of HBase consistency feature we need to use the memstore flushing. If we save the MOBs directly into sequence file, then it will make compaction difficult and add load to HBase when updating pointers. The MOB data will not take part in split and compaction in HBase.

The actions that take place while writing the MOB data

The MOB data is written into KeyValue of MOB column. When the memstore is full, then MOB data is flushed to MOB files in the format of HFiles and metadata are flushed to StoreFiles. The values of MOB keyvalues are replaced by the path of MOB files. The MOB KeyValue in StoreFile have a tag that is a reference which links to MOB files.

The file path is /rootPath/tableName/.mob/columnFamilyName/${filename}

The chain of the random read against the MOB data

       1.       Find the metadata, and the path of the HFile in the metadata

       2.       Find the HFile by the path

       3.       Seek the KeyValue with the keyrowkey,columnFamily:column,ts in this HFile, and            retrieve it

MOB Files Cleaner and Sweep Tool

·         The MOB file cleaner cleans the expired MOB files.

·         The sweep tool uses a MapReduce job to clean the unused MOB data. This tool also sweeps the small MOB files to larger files.

How to take advantage of MOBs?

In order take advantage of MOB feature, we need to use HFile version 3. We need to edit the ‘hbase-site.xml’ and restart RegionServer. Changes will take place after major compaction.

<property>

  <name>hfile.format.version</name>

  <value>3</value>

</property>

Configuring Columns to Store MOBs

The properties to be set for handling MOB data are ‘IS_MOB’ and ‘MOB_THRESHOLD’. ‘IS_MOB’ specifies whether a column can store MOB data. The ‘MOB_THRESHOLD’ specifies the size of data to be considered as a MOB. Default is 100 bytes.

HBase Shell command:

hbase> create 'sample', {NAME => 'm', IS_MOB => true, MOB_THRESHOLD => 10500}

hbase> alter 'sample', {NAME => 'm', IS_MOB => true, MOB_THRESHOLD => 10500}

References:

[1] http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/admin_hbase_mob.html

[2] http://blog.cloudera.com/blog/2015/06/inside-apache-hbases-new-support-for-mobs/

[3] https://issues.apache.org/jira/browse/HBASE-11643

[4] https://issues.apache.org/jira/browse/HBASE-11339

[5] https://github.com/intel-hadoop/HBase-LOB

[6] https://github.com/Cloudera-Intel-QA-Transition/test-cases/wiki/HBase-MOB-Test-Plan