Hive - How to increase memory for Hive CLI, Hiveserver2, and Hivemetastore

You might encounter various cases where Hive CLI or Hiveserver2 or Hivemetastore is running out of memory. You will see following exception in metastore/hiveserver2 logs if they are running out of memory:

java.lang.OutOfMemoryError: Java heap space

As part of cluster planning, you might need to increase memory to one or more of the above services. If you want to increase memory for all the services, then specify the following property in 'hive-env.sh' file with required memory in all nodes where hiverserver2, metatstore is installed and from where Hive CLI will be called.

export HADOOP_CLIENT_OPTS="-Xmx<memory in MB>m"

This will require a restart of hiveserver2 and hivemetastore services.

However, if you want to increase memory for specific service, then you need to edit a specific part of 'hive-env.sh' file.

Case 1: Increase memory for Hive CLI

Add the following in 'hive-env.sh' file on nodes where Hive CLI will be used. When you reconnect to Hive CLI, it will take the new value.  

if [ "$SERVICE" = "cli" ]; then

  if [ -z "$DEBUG" ]; then

    export HADOOP_OPTS="$HADOOP_OPTS -XX:NewRatio=12 -Xmx<memory in MB>m -Xms10m -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:+UseParNewGC -XX:-UseGCOverheadLimit"

  else

    export HADOOP_OPTS="$HADOOP_OPTS -XX:NewRatio=12 -Xmx<memory in MB>m -Xms10m -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:-UseGCOverheadLimit"

  fi

fi

Case 2: Increase memory for HiveServer2

Add the following in 'hive-env.sh' file on all nodes where Hiveserver2 service is installed. This will require a restart of hiveserver2 service. 

if [ "$SERVICE" = "hiveserver2" ]; then

 if [ -z "$DEBUG" ]; then

    export HADOOP_OPTS="$HADOOP_OPTS -XX:NewRatio=12 -Xmx<memory in MB>m -Xms10m -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:+UseParNewGC -XX:-UseGCOverheadLimit"

  else

    export HADOOP_OPTS="$HADOOP_OPTS -XX:NewRatio=12 -Xmx<memory in MB>m -Xms10m -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:-UseGCOverheadLimit"

  fi

fi

Case 3: Increase memory for HiveMetastore

Add the following in 'hive-env.sh' file on all nodes where hivemetastore service is installed. This will require a restart of metastore service. 

if [ "$SERVICE" = "metastore" ]; then

 if [ -z "$DEBUG" ]; then

    export HADOOP_OPTS="$HADOOP_OPTS -XX:NewRatio=12 -Xmx<memory in MB>m -Xms10m -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:+UseParNewGC -XX:-UseGCOverheadLimit"

  else

    export HADOOP_OPTS="$HADOOP_OPTS -XX:NewRatio=12 -Xmx<memory in MB>m -Xms10m -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:-UseGCOverheadLimit"

  fi

fi