DRILL ON YARN - How to collect Drillbit log file

This blog discusses how to collect drillbit logs for troubleshooting.  We will focus on collecting logs specifically for DoY (Drill on Yarn).

There can be two scenarios where you end up collecting logs:

[1] After the application master is stopped.

[2] When the application master is still up and running. (Drillbit cluster is up and running.)

In scenario 1, collecting drillbit log is as simple as collecting the Yarn aggregated logs. Drillbit logs will be present in the application logs. For this to take effect, make sure that Yarn log aggregation is enabled in your cluster. You can execute the following command to collect the logs:

yarn logs -applicationId <application Id> [options] > <destination>

In scenario 2, since the application has not finished yet, the above command will not work. In this case, we need to collect 'Drillbit.log' file from each container launched for this application. This is not a straightforward process since it involves identifying the container IDs and then copying the file to the desired location from every node where drill bits are launched. 

Following script will help in such scenario. Copy the script and save as 'collect-doy-drillbitlog.sh'.  You can find the help by executing following command:

The script is also available in my Github repository - https://github.com/jamealwi2/doy-log-collector

collect-doy-drillbitlog.sh -h

#!/bin/bash

usage(){

echo "Please see usage below:"

echo "collect-doy-drillbitlog.sh -a <application id> -d <copy destination>"

echo "collect-doy-drillbitlog.sh -a <application id> (Default copy destination is '/tmp'.)"

}

while getopts 'ha:d:' option; do

        case "$option" in

                h) usage

                        exit

                        ;;

                a) application_id=$OPTARG

                        ;;

                d) copy_location=$OPTARG

                        ;;

        esac

done;

if [ -z $application_id ]; then

        echo "Please specify the application ID."

usage

        exit

fi

if [ -z $copy_location ]; then

        echo "Copy destination location not specified. The log will be copied to '/tmp'."

        echo -n "Please confirm: (y/n)? "

        read answer

        if echo "$answer" | grep -iq "^y" ;then

                copy_location="/tmp"

        else

                exit

        fi

fi

ls /opt/mapr/hadoop/hadoop-2.7.0/logs/userlogs/${application_id}/*/* |grep drillbit.log | while read line

do container_id=$(echo $line | awk -F"/" '{print $(NF-1)}')

cp $line /${copy_location}/`hostname`-${container_id}-drillbit.log

done;