HBase - Identifying and moving corrupted HFiles

Identifying corrupted files:

HBase provides a utility to check for any corrupted files in HBase.

hbase hbck -checkCorruptHFiles

The above command will check the entire HBase tables for any corrupted files.

To check corrupted in specific table use the following command:

hbase hbck -checkCorruptHFiles <table_name>

If any files are corrupted, then we will have something similar to following in the output log:

Checked 17 hfile for corruption

  HFiles corrupted:                  2

    HFiles moved while checking:     0

Summary: CORRUPTED

The logs also contain details about which files are corrupted.

Another way of checking whether a file is corrupted or not is using the following command:

hbase org.apache.hadoop.hbase.io.hfile.HFile -f <path_to_hfile> 

Sidelining corrupted files:

HBase provides a utility sidelining corrupted files in HBase.

hbase hbck -sidelineCorruptHFiles

The above command will check the entire HBase tables.

To apply on specific table use the following command:

hbase hbck -sidelineCorruptHFiles <table_name>

You will see similar information in the command output log:

Checked 17 hfile for corruption

  HFiles corrupted:                  2

    HFiles successfully quarantined: 2

      maprfs:/hbase/corrupt/hcrt/1090c602c005a4ca76fda4ec7bd2865c/f/97fcf7fee25c469a81e7a0aa567a4627

      maprfs:/hbase/corrupt/hcrt/1090c602c005a4ca76fda4ec7bd2865c/f/97fcf7fee25c469a81e7a0aa567a4628

    HFiles failed quarantine:        0

    HFiles moved while checking:     0

Summary: CORRUPTED => OK

The corrupted files are moved to '/hbase/corrupt' folder.

HBase Compaction - Part 2 

Following is a study of parameters that control minor and major compaction in HBase.

Minor compaction 

The default minor compaction algorithm depend on the following parameters:

[1] hbase.hstore.compaction.min – Minimum number of StoreFiles to be selected for a compaction to occur. Defaults to 3 Hbase 1.1.8.

StoreFiles – files produced by memstore flush

StoreFile contains many metadata information, some of the important ones are:

MAX_SEQ_ID_KEY – maximum sequence ID in FileInfo

MAJOR_COMPACTION_KEY – Major compaction flag info

EXCLUDE_FROM_MINOR_COMPACTION_KEY  – Major compaction flag info

HFILE_NAME_REGEX - HFiles are uuid ([0-9a-z]+). Bulk loaded hfiles has (_SeqId_[0-9]+_) has suffix.

[2] hbase.hstore.compaction.max – maximum number of storefiles to be compacted in each minor compaction. Defaults to 10.

[3] hbase.hstore.compaction.min.size – any file that is smaller that this will be a candidate for compaction.

[4] hbase.hstore.compaction.max.size – any file greater than this is automatically excluded from compaction - (by default it is LONG.MAX_VALUE)

[5] hbase.store.compaction.ratio – default is 1.2f

The simple formula for selection of a file for minor compaction is :

selects a file for compaction when the file size <= sum(smaller_files_size) * hbase.hstore.compaction.ratio.

Example from HBase official documentation:

The following StoreFiles exist: 100, 50, 23, 12, and 12 bytes apiece (oldest to newest). With the above parameters, the files that would be selected for minor compaction are 23, 12, and 12.

Why?

Remember the logic

selects a file for compaction when the file size <= sum(smaller_files_size) * hbase.hstore.compaction.ratio.

    100 --> No, because sum(50, 23, 12, 12) * 1.0 = 97.

    50 --> No, because sum(23, 12, 12) * 1.0 = 47.

    23 --> Yes, because sum(12, 12) * 1.0 = 24.

    12 --> Yes, because the previous file has been included, and because this does not exceed the the max-file limit of 5

    12 --> Yes, because the previous file had been included, and because this does not exceed the the max-file limit of 5.

Following log snippet shows HBase regionserver logs during minor compaction or shortcompactions:

It provide following information:

[1] Shows which table, which column family, which region is undergoing compaction.

[2] Number of files compacted.

[3] Total size of file for compaction. Sum of individual files undergoing compaction.

[4] Shows total file size after compaction is completed.

[5] Time taken for minor compaction.

Minor compaction logs: (Table name is 'hb' with column family 'c') 

2017-07-10 17:08:13,967 INFO  [regionserver/vm52/10.10.XX.XX:16020-shortCompactions-1499720893966] regionserver.HRegion: Starting compaction on c in region hb,,1499720284228.0f0486e029334542705e66f401fa698b.

2017-07-10 17:08:13,968 INFO  [regionserver/vm52/10.10.XX.XX:16020-shortCompactions-1499720893966] regionserver.HStore: Starting compaction of 3 file(s) in c of hb,,1499720284228.0f0486e029334542705e66f401fa698b. into tmpdir=maprfs:/hbase/data/default/hb/0f0486e029334542705e66f401fa698b/.tmp, totalSize=14.7 K

2017-07-10 17:08:13,980 INFO  [regionserver/vm52/10.10.XX.XX:16020-shortCompactions-1499720893966] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=2, currentSize=1291688, freeSize=1249607128, maxSize=1250898816, heapSize=1291688, minSize=1188353920, minFactor=0.95, multiSize=594176960, multiFactor=0.5, singleSize=297088480, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false

2017-07-10 17:08:14,109 INFO  [regionserver/vm52/10.10.XX.XX:16020-shortCompactions-1499720893966] regionserver.HStore: Completed compaction of 3 (all) file(s) in c of hb,,1499720284228.0f0486e029334542705e66f401fa698b. into 4cc6a50eb38d4ef2844a3339bcdfe11d(size=5.0 K), total size for store is 5.0 K. This selection was in queue for 0sec, and took 0sec to execute.

2017-07-10 17:08:14,113 INFO  [regionserver/vm52/10.10.XX.XX:16020-shortCompactions-1499720893966] regionserver.CompactSplitThread: Completed compaction: Request = regionName=hb,,1499720284228.0f0486e029334542705e66f401fa698b., storeName=c, fileCount=3, fileSize=14.7 K, priority=1, time=4222857501044128; duration=0sec

Major Compaction:

Major compaction depend on the following parameters:

[1] hbase.hregion.majorcompaction - Default value is 604800000 (7 days)

The time interval between each major compaction. Setting this to 0 will disable time based major compaction. 

Sometimes, minor compactions can be promoted to major compaction.

[2] Off peak hour compactions:

Identifying peak hour of your cluster will help in notifying HBase not to do heavy minor compactions during the busy hours. 

For this from HBase 1.2+ onward, there are following parameters:

[a] hbase.hstore.compaction.max.size.offpeak – sets a value for the largest file that can be used for compaction

[b] hbase.offpeak.start.hour= 0..23 (specify start hour)

[c] hbase.offpeak.end.hour= 0..23 (specify end hour)

The hstore compaction ratio is by default 1.2 for peak hours. For offpeak hours, it is 5.

Both the values can be adjusted using the following parameters:

[a] hbase.hstore.compaction.ratio

[b] hbase.hstore.compaction.ratio.offpeak

[3] hbase.hregion.majorcompaction.jitter

Compactions are carried out by regionservers. Inorder to make sure that all regionserver does not do major compaction at the same time, we have this jitter parameter. 

By default the value is 0.5. 0.5 is the maximum value of outer bound. hbase.hregion.majorcompaction is multiplied by this some fraction that will be inside this jitter value and then added/subtracted to determine when to run the next major compaction.

Following log snippet shows HBase regionserver logs during major compaction or largecompactions:

It provide following information:

[1] Displays the table and region undergoing major compaction.

[2] If the major compaction is triggered manually, then minor compaction is called internally.

[3] Intermittently the store file will be stored inside the .tmp folder.

[4]Provides information about the number of files under compaction, total size of new file generated, time taken for the compaction.

Major compaction logs: (Table name is 'hb' with column family 'c') 

2017-07-10 21:10:55,158 INFO  [PriorityRpcServer.handler=1,queue=1,port=16020] regionserver.RSRpcServices: Compacting hb,,1499720284228.0f0486e029334542705e66f401fa698b.

2017-07-10 21:10:55,159 DEBUG [PriorityRpcServer.handler=1,queue=1,port=16020] compactions.RatioBasedCompactionPolicy: Selecting compaction from 2 store files, 0 compacting, 2 eligible, 10 blocking

2017-07-10 21:10:55,159 DEBUG [PriorityRpcServer.handler=1,queue=1,port=16020] regionserver.HStore: 0f0486e029334542705e66f401fa698b - c: Initiating major compaction (all files)

2017-07-10 21:10:55,159 DEBUG [PriorityRpcServer.handler=1,queue=1,port=16020] regionserver.CompactSplitThread: Small Compaction requested: org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext@1ffa895a; Because: User-triggered major compaction; compaction_queue=(0:1), split_queue=0, merge_queue=        0

2017-07-10 21:10:55,159 INFO  [regionserver/vm52/10.10.XX.XX:16020-shortCompactions-1499725825449] regionserver.HRegion: Starting compaction on c in region hb,,1499720284228.0f0486e029334542705e66f401fa698b.

2017-07-10 21:10:55,160 INFO  [regionserver/vm52/10.10.XX.XX:16020-shortCompactions-1499725825449] regionserver.HStore: Starting compaction of 2 file(s) in c of hb,,1499720284228.0f0486e029334542705e66f401fa698b. into tmpdir=maprfs:/hbase/data/default/hb/0f0486e029334542705e66f401fa698b/.tmp, totalSize=10.9 K

2017-07-10 21:10:55,162 DEBUG [regionserver/vm52/10.10.XX.XX:16020-shortCompactions-1499725825449] compactions.Compactor: Compacting maprfs:/hbase/data/default/hb/0f0486e029334542705e66f401fa698b/c/0daeafd75a4c4ba4a53172a73b9ca4b0, keycount=41, bloomtype=ROW, size=6.0 K, encoding=NONE, seqNum=174, earliestPutTs=1499720300277

2017-07-10 21:10:55,164 DEBUG [regionserver/vm52/10.10.XX.XX:16020-shortCompactions-1499725825449] compactions.Compactor: Compacting maprfs:/hbase/data/default/hb/0f0486e029334542705e66f401fa698b/c/bfeea9e43bc347ec84863bdd4476e270, keycount=2, bloomtype=ROW, size=4.9 K, encoding=NONE, seqNum=182, earliestPutTs=1499735424849

2017-07-10 21:10:55,165 INFO  [regionserver/vm52/10.10.XX.XX:16020-shortCompactions-1499725825449] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=3, currentSize=1308744, freeSize=1249590072, maxSize=1250898816, heapSize=1308744, minSize=1188353920, minFactor=0.95, multiSize=594176960, multiFactor=0.5, singleSize=297088480, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false

2017-07-10 21:10:55,191 DEBUG [regionserver/vm52/10.10.XX.XX:16020-shortCompactions-1499725825449] regionserver.HRegionFileSystem: Committing store file maprfs:/hbase/data/default/hb/0f0486e029334542705e66f401fa698b/.tmp/da9d2b5454e640769a9b20c82124a010 as maprfs:/hbase/data/default/hb/0f0486e029334542705e66f401fa698b        /c/da9d2b5454e640769a9b20c82124a010

2017-07-10 21:10:55,208 DEBUG [regionserver/vm52/10.10.XX.XX:16020-shortCompactions-1499725825449] regionserver.HStore: Removing store files after compaction...

2017-07-10 21:10:55,219 DEBUG [regionserver/vm52/10.10.XX.XX:16020-shortCompactions-1499725825449] backup.HFileArchiver: Archiving compacted store files.

2017-07-10 21:10:55,231 DEBUG [regionserver/vm52/10.10.XX.XX:16020-shortCompactions-1499725825449] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:maprfs:/hbase/data/default/hb/0f0486e029334542705e66f401fa698b/c/0daeafd75a4c4ba4a53172a73b9ca4b0, to maprfs:/hbase/archive/data/default/hb/0f0486e029334542705e66f401fa698b/c/0daeafd75a4c4ba4a53172a73b9ca4b0

2017-07-10 21:10:55,243 DEBUG [regionserver/vm52/10.10.XX.XX:16020-shortCompactions-1499725825449] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:maprfs:/hbase/data/default/hb/0f0486e029334542705e66f401fa698b/c/bfeea9e43bc347ec84863bdd4476e270, to maprfs:/hbase/archive/data/default/hb/0f0486e029334542705e66f401fa698b/c/bfeea9e43bc347ec84863bdd4476e270

2017-07-10 21:10:55,244 INFO  [regionserver/vm52/10.10.XX.XX:16020-shortCompactions-1499725825449] regionserver.HStore: Completed major compaction of 2 (all) file(s) in c of hb,,1499720284228.0f0486e029334542705e66f401fa698b. into da9d2b5454e640769a9b20c82124a010(size=6.1 K), total size for store is 6.1 K. This selection was in queue for 0sec, and took 0sec to execute.

2017-07-10 21:10:55,246 INFO  [regionserver/vm52/10.10.XX.XX:16020-shortCompactions-1499725825449] regionserver.CompactSplitThread: Completed compaction: Request = regionName=hb,,1499720284228.0f0486e029334542705e66f401fa698b., storeName=c, fileCount=2, fileSize=10.9 K, priority=1, time=4237418695815609; duration=0sec

2017-07-10 21:10:55,246 DEBUG [regionserver/vm52/10.10.XX.XX:16020-shortCompactions-1499725825449] regionserver.CompactSplitThread: CompactSplitThread Status : compaction_queue=(0:0), split_queue=0, merge_queue=0

HBase Compaction - Part 1 - Why we need them?

HBase has two types of compaction. Minor compaction and Major compaction.

Why we need minor compaction?

When you insert some data to HBase, in normal write path (the other being bulk load, which does not go through normal write path), the data goes to WAL - Write Ahead Log. 

Regionserver reads the data and puts to its memstore. When data in memstore reaches a threshold, it is flushed to disk generating storefiles.

The threshold for memstore is normally in MBs. Hence, memstore flush will generate large number of small storefiles.

This will in turn cause performance issues while read operation. Hence we have minor compactions. 

Ideally minor compaction deals with small sized files. Minor compaction kicks in automatically when certain conditions are met (Discussed in HBase Compaction - Part 2).

It is not recommended to turn off minor compaction.

StoreFile contains many metadata information, some of the important ones are:

[1] MAX_SEQ_ID_KEY – maximum sequence ID in FileInfo

[2] MAJOR_COMPACTION_KEY – Major compaction flag info

[3] EXCLUDE_FROM_MINOR_COMPACTION_KEY  – Major compaction flag info

[4] HFILE_NAME_REGEX - HFiles are uuid ([0-9a-z]+). Bulk loaded hfiles has (_SeqId_[0-9]+_) has suffix.

Why we need major compaction?

Minor compaction deals with small files, while major compaction deals with larger files.

Major compaction combines multiple storefiles and generate one big HFile.

Also, when we update or delete a row in HBase table, HBase just write a deletion marker and masks the data from being accessed in further operations.

During major compaction, HBase reads each file and removes all the data marked for deletion.

This is impoertant, since if it is not performed it will eat up the disk space and will reduce the performance.

Hence major compactions. As of HBase 1.1.8, major compaction automatically happens once in 7 days unless turned off manually.

In busy clusters it is recommended to turn off automatic major compaction and trigger it manually at off peak hours.

Major compaction will read the complete table data and writes into a new file.
As a result it requires at least one full read and one full write of table data.
The two main benefits of major compaction are:
[1] Saving disk space by removing the expired and deleted data
[2] Improve the read performance by increasing disk seek performance

SQOOP Import failure to Hive parquet table

Sample query that fails:

sqoop import --connect <connection_string> --query "select * from alwin_test where \$CONDITIONS"  --create-hive-table --hive-import --hive-table <hive_table_name> --username <uname> -P --target-dir '/user/mapr/oracleimport' -m 1 --as-parquetfile

However the following will succeed:

sqoop import --connect <connection_string> --query "select * from alwin_test where \$CONDITIONS"  --create-hive-table --hive-import --hive-table <hive_table_name> --username <uname> -P --target-dir '/user/mapr/oracleimport' -m 1

Error thrown:

ERROR sqoop.Sqoop: Got exception running Sqoop: org.apache.avro.SchemaParseException: org.codehaus.jackson.JsonParseException: Unexpected end-out: was expecting closing quote for a string value 

at [Source: java.io.StringReader@13ae2297; line: 1, column: 6001] 

org.apache.avro.SchemaParseException: org.codehaus.jackson.JsonParseException: Unexpected end-of-input: was expecting closing quote for a string value 

at [Source: java.io.StringReader@13ae2297; line: 1, column: 6001] 

at org.apache.avro.Schema$Parser.parse(Schema.java:955) 

at org.apache.avro.Schema$Parser.parse(Schema.java:943) 

at org.kitesdk.data.DatasetDescriptor$Builder.schemaLiteral(DatasetDescriptor.java:463) 

at org.kitesdk.data.spi.hive.HiveUtils.descriptorForTable(HiveUtils.java:157) 

at org.kitesdk.data.spi.hive.HiveAbstractMetadataProvider.load(HiveAbstractMetadataProvider.java:104) 

at org.kitesdk.data.spi.filesystem.FileSystemDatasetRepository.load(FileSystemDatasetRepository.java:197) 

at org.kitesdk.data.spi.hive.HiveManagedDatasetRepository.create(HiveManagedDatasetRepository.java:82) 

at org.kitesdk.data.Datasets.create(Datasets.java:239) 

at org.kitesdk.data.Datasets.create(Datasets.java:307) 

at org.apache.sqoop.mapreduce.ParquetJob.createDataset(ParquetJob.java:107) 

at org.apache.sqoop.mapreduce.ParquetJob.configureImportJob(ParquetJob.java:89) 

at org.apache.sqoop.mapreduce.DataDrivenImportJob.configureMapper(DataDrivenImportJob.java:108) 

at org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:260) 

at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:673) 

at org.apache.sqoop.manager.OracleManager.importTable(OracleManager.java:444) 

at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:497) 

at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:606) 

at org.apache.sqoop.Sqoop.run(Sqoop.java:143) 

at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) 

at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179) 

at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218) 

at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227) 

at org.apache.sqoop.Sqoop.main(Sqoop.java:236) 

Caused by: org.codehaus.jackson.JsonParseException: Unexpected end-of-input: was expecting closing quote for a string value 

Root Cause:

The issue is because of limitation on the number of characters that can be stored in TBLPROPERTIES in hive metastore.

Inside hive metastore db:

mysql> describe TABLE_PARAMS;

+-------------+----------------+------+-----+---------+-------+

| Field       | Type           | Null | Key | Default | Extra |

+-------------+----------------+------+-----+---------+-------+

| TBL_ID      | bigint(20)     | NO   | PRI | NULL    |       |

| PARAM_KEY   | varchar(256)   | NO   | PRI | NULL    |       |

| PARAM_VALUE | varchar(4000)  | YES  |     | NULL    |       |

+-------------+----------------+------+-----+---------+-------+

3 rows in set (0.00 sec)

The default type for PARAM_VALUE is varchar(4000).
In this case, when hive parquet table is created, table property 'avro.schema.literal' is truncated to 4000 characters which results invalid JSON.
This will cause JSON parse exception resulting SQOOP import failure.

Resolution:

The solution is to either limit the character length to 4000 or increase character length for 'PARAM_VALUE'.

Following are the solutions:

[1] Increase character length for 'PARAM_VALUE'.

For example:

mysql> describe TABLE_PARAMS;

+-------------+----------------+------+-----+---------+-------+

| Field       | Type           | Null | Key | Default | Extra |

+-------------+----------------+------+-----+---------+-------+

| TBL_ID      | bigint(20)     | NO   | PRI | NULL    |       |

| PARAM_KEY   | varchar(256)   | NO   | PRI | NULL    |       |

| PARAM_VALUE | varchar(10000) | YES  |     | NULL    |       |

+-------------+----------------+------+-----+---------+-------+

3 rows in set (0.00 sec)

OR [2] Instead of using "select * from alwin_test where \$CONDITIONS" in SQOOP import, we can use alias name to reduce column name length and hence reduce character length for 'avro.schema.literal'

OR [3] Create a view of Oracle (source) tables with much smaller column name and then do SQOOP import on the view created.

Connect DRILL from JBOSS

AIM:  

Integrate DRILL and JBOSS.

STEPS:

1. Download JBOSS from http://jbossas.jboss.org/downloads (I used the following) 

http://download.jboss.org/jbossas/7.0/jboss-as-7.0.1.Final/jboss-as-web-7.0.1.Final.zip

2. Extract the zip file (Installed in my local windows machine)

Goto <JBOSS_HOME>\bin and run standalone.bat

3. Open the JBOSS administartor console in Web Browser

http://localhost:9990/console/App.html#server/datasources

4. Our DRILL is up and running in following node

http://<drillbit-node>:8047/

DRILL Server Version - 1.9r1 (MapR)

Make sure that we are using DRILL Server version which is newer or equal to following: (Earlier packages has a bug which will throw a 'NullPointerException' on DrillConnectionImpl.isReadOnly().

5. Add drill jdbc driver to this folder: $JBOSS_HOME/modules/org/apache/drill/main/

(Create folder 'org/apache/drill/main/' if it does not exist)

Added the following jar to specified location: drill-jdbc-all-1.9.0.jar

(The jar is available with DRILL installation under - '/opt/mapr/drill/drill-1.9.0/jars/jdbc-driver'  for MapR installtion for DRILL Server 1.9)

6. Add module.xml file to '$JBOSS_HOME/modules/org/apache/drill/main/' folder with the following content:

<module xmlns="urn:jboss:module:1.0" name="org.apache.drill">

    <resources>

         <resource-root path="drill-jdbc-all-1.9.0.jar"/> 

    </resources>

    <dependencies>

        <module name="javax.api"/>

        <module name="javax.transaction.api"/>

        <module name="sun.jdk"/>

    </dependencies>

</module>

7. Change datasources part in the $JBOSS_HOME/standalone/configuration/standalone.xml file:

                

<datasource jta="true" jndi-name="java:jboss/datasources/DRILL-DS1" pool-name="DRILL-DS1" enabled="true" use-java-context="true" use-ccm="true">

                    <connection-url>jdbc:drill:drillbit=<drillbit-node>:31010</connection-url>

                    <driver>drill</driver>

                    <security>

                        <user-name><username></user-name>

                        <password><Password></password>

                    </security>

                    <validation>

                        <check-valid-connection-sql>select * from sys.version</check-valid-connection-sql>

                        <validate-on-match>false</validate-on-match>

                        <background-validation>false</background-validation>

                        <use-fast-fail>false</use-fast-fail>

                    </validation>

                </datasource>

8. Add the following in 'drivers' section inside standalone.xml

        <drivers>

          <driver name="drill" module="org.apache.drill" />

        </drivers>

9. Restart the JBOSS server. Once you restart, you will see DRILL-DS1 under 'Registered Datasources'

10. If you are using RED HAT' JBOSS' ENTERPROSE APPLICATION PLATFORM, then you should see something like below:

11. You can test the connection by clicking on the particular datasource (DRILL-DS1) and then on 'Test Connection'. 

12. If the connection is successful, then we will see following screen:

HBase Archive Folder (/hbase/archive)

AIM: To understand the concept of HBase Archive folder.

Essentially, whenever an hfile would be deleted, it is moved to the archive directory. Removing hfiles occurs in - compaction, merge, catalog janitor and table deletes. In the below section, we see how this happens when compaction and table deletion takes place. 

Case 1 – Compaction:

Screenshot showing store files inside hbase region ‘fba74ad028a1532723da6e4cd1d9f4a6‘ for table ‘cfs’ with column family ‘c’. At this point, we have data inside ‘/hbase/data’ folder and ‘/hbase/archive’ folder is empty.

After minor compaction of table:

The original store files are moved to ‘/hbase/archive’ folder and ‘/hbase/data’ folder contains the new compacted hfile.

The files inside ‘/hbase/archive’ is cleaned up at specific intervals.

Case 2 - Deletion of a table

When a table is deleted, then the files are not instantly removed. It is moved to ‘/hbase/archive’ folder. One of the design consideration for this approach is that, there may be snapshots associated with the deleted table. This snapshots may in turn be used to create a new table. The new table uses references to old hfiles until compaction is completed for the table and it recreates the necessary files in the ‘/hbase/data’ folder.

HFileCleaner HBase chore

The cleaning of archive folder is taken care by HFileCleaner hbase chore (org.apache.hadoop.hbase.master.cleaner.HFileCleaner). This Chore, every time it runs, will clear the HFiles in the hfile archive folder that can be deleted. The hbase master will cleanup the FS for that table backup; first writes an 'invalid' marker for that table directory and then recursively deletes the regions and then the table directory in the archive.

(HBase CHORES: Chore is a task performed on a period in hbase. The chore is run in its own thread. This base abstract class provides while loop and sleeping facility.)

It is a single threaded activity, it collects the list of files in the archive directory, validates whether it can be deleted and then deletes it. Once one cycle is completed, it will wait for the specified time period and then kicks in again after that. One argument to this chore is ‘period’ - the period of time to sleep between each run.

If HBase snapshots are taken, then the snapshot have references to archive folder files. In that case, the HFileCleaner will not remove data from the ‘archive’ folder. (In this case manually removing data is snapshot data lose.)

Let’s assume the ‘period’ given for the activity is 10 minutes, then once it finishes one cycle, it will wait for another 10 minutes to kick in the next cycle. So, there can be a chance that the Hbase HFileCleaner cannot cope with the required deletion rate. Hence increasing the size of ‘/hbase/archive’ folder.

Related open bug for the same:

https://issues.apache.org/jira/browse/HBASE-12626 

https://issues.apache.org/jira/browse/HBASE-5547 à hbase/archive was introduced

A simple example explaining the issue caused due to above bug:

Let’s assume the following:

Deletion rate of files per minute in archive folder	50
Incoming rate of files per minute into archive folder	100

(Assumption: the cleaner kicks in for the next cycle as soon as it completes its one cycle and none of the files in archive folder is retained)

Minutes	Total number of files to be cleaned when the cleaner first kicks in - 100	Cleaning cycle
1		Cleaning cycle – 1 (Time taken – 2 minutes)
2
3	Total number of files to be cleaned at 3rd minute - 200	Cleaning cycle – 2 (Time taken – 4 minutes)
4
5
6
7	Total number of files to be cleaned at 7th minute - 400	Cleaning cycle – 3 (Time taken – 8 minutes)
8
9
10
11
12
13
14

Enable DRILL ODBC debug in Windows client

AIM: 

Steps to set log to DEBUG level for DRILL ODBC driver in Windows machine

STEPS:

If the client machine is Windows, then we can enable debug trace for Drill ODBC by following steps

- Go to ODBC Data Source Administrator

- Navigate to System DSN

- Select the specific driver, it will pop up the Driver DSN Setup window

- Click on 'Logging Options'

- Change the 'Log Level' to 'LOG_DEBUG'

- Specify the path