Hive log files not getting deleted even after retention number is reached from Hive 2.1 onward

Issue:

Hive log files not getting deleted even after the retention number is reached. Log rotation works fine.

Issue is observed from Hive 2.1 which uses log4j2 for logging.

Cause:

The issue is observed for 'TimeBasedTriggeringPolicy' in log4j2.

This is a know limitation in 'TimeBasedTriggeringPolicy' for log4j2 as mentioned in https://issues.apache.org/jira/browse/LOG4J2-435.

Workaround:

One workaround for the issue is to use SizeBasedTriggeringPolicy. 

To use 'SizeBasedTriggeringPolicy', make the following changes in 'hive-log4j2.properties' inside respective HIVE_CONF_DIR.

Comment out following:

appender.DRFA.filePattern = ${sys:hive.log.dir}/${sys:hive.log.file}.%d{yyyy-MM-dd}
appender.DRFA.policies.time.type = TimeBasedTriggeringPolicy
appender.DRFA.policies.time.interval = 1
appender.DRFA.policies.time.modulate = true

Add the following:

appender.DRFA.filePattern = ${sys:hive.log.dir}/${sys:hive.log.file}.%i
appender.DRFA.policies.size.type=SizeBasedTriggeringPolicy
appender.DRFA.policies.size.size=100MB    -----> Customize the size of each log file you need
appender.DRFA.strategy.max = 3   -----> Customize the number of log files to be retained

It will look similar to this after above activity:

It will look similar to this after above activity:
# daily rolling file appender
appender.DRFA.type = RollingRandomAccessFile
appender.DRFA.name = DRFA
appender.DRFA.fileName = ${sys:hive.log.dir}/${sys:hive.log.file}
# Use %pid in the filePattern to append <process-id>@<host-name> to the filename if you want separate log files for different CLI session
#appender.DRFA.filePattern = ${sys:hive.log.dir}/${sys:hive.log.file}.%d{yyyy-MM-dd}
appender.DRFA.filePattern = ${sys:hive.log.dir}/${sys:hive.log.file}.%i
appender.DRFA.layout.type = PatternLayout
appender.DRFA.layout.pattern = %d{ISO8601} %5p [%t] %c{2}: %m%n
appender.DRFA.policies.type = Policies
#appender.DRFA.policies.time.type = TimeBasedTriggeringPolicy
appender.DRFA.policies.size.type=SizeBasedTriggeringPolicy
appender.DRFA.policies.size.size=100MB
#appender.DRFA.policies.time.interval = 1
#appender.DRFA.policies.time.modulate = true
appender.DRFA.strategy.type = DefaultRolloverStrategy
appender.DRFA.strategy.max = 3