Thursday, 17 August 2017

Can Classic Drill and Drill-On-Yarn co-exist? 


This combination is not completely tested but we can get it working. 

Following notes will help to configure Drill-Yarn and Drill-Classic to co-exist on the same cluster. 
The activity is performed in a MapR cluster.

[root@vm78-218 ~]# maprcli node list -columns svc 
service hostname ip 
historyserver,hbmaster,webserver,nodemanager,spark-historyserver,drill-bits,cldb,fileserver,hoststats vm78-217 IP1 
hbmaster,hbregionserver,webserver,cldb,fileserver,hoststats,hue vm78-210 IP2 
hivemeta,httpfs,webserver,drill-bits,fileserver,resourcemanager,hue vm78-218 IP3 

We have two classic drill-bits running on vm78-217 and vm78-218. 
We will install Drill On YARN (1.10) on vm78-210. (Note, this machine do not have any drill installed). 

yum install mapr-drill-yarn -y 

Now, configure Drill-YARN to co-exist with Classic Drill. 

[1] Change the drill log location in 'drill-env.sh'. Add the following: 
export DRILL_LOG_DIR="/opt/mapr/drill/drill-1.1.0-YARN/logs" 
Make sure that the directory is present in all the nodes running nodemanager with necessary permissions. 

[2] Make necessary changes in 'drill-on-yarn.conf' based on http://maprdocs.mapr.com/home/Drill/configure_drill_to_run_under_yarn.html 

[3] We need to change the following properties in 'drill-override.conf' 
• drill.exec.cluster-id 
• drill.exec.zk.root 
• drill.drill.exec.rpc.user.server.port 
• drill.exec.rpc.bit.server.port 
• drill.exec.http.port 
(The idea is to give different values to above properties other than the default properties. This is under the assumption that the default properties is used by classic drill.) 

Sample is given below: 
drill.exec: { 
cluster-id: "ajames-drillbitsonyarn", 
zk.root: "drillonyarn", 
rpc.user.server.port: 21010, 
rpc.bit.server.port: 21011, 
http.port: 7047, 
zk.connect: "<zk-qourum>" 

[4] Follow steps in http://maprdocs.mapr.com/home/Drill/step_4_launch_drill_under_yarn.html to launch the DRILL-ON-YARN. 

You can also refer https://community.mapr.com/docs/DOC-1188 

No comments:

Post a Comment