Installing Ganglia on MapR to monitor HBase

Node details:

Node1 – 10.10.YY.X1

Node2 - 10.10.YY.X2

Node3 - 10.10.YY.X3

 The basic packages required for ganglia to run are:

[1] ganglia-gmond – required on all nodes from where metrics needs to be collected

[2] ganglia-gmetad – required on node which will perform aggregation (typically on one node)

[3] ganglia-web – required on the node running the Ganglia web UI

Following is my setup: (Concentrating on Ganglia and HBase)

Node1 – 10.10.YY.X1 – Hbase regionserver, gmond

Node2 - 10.10.YY.X3 – Hbase regionserver, gmond

Node3 - 10.10.YY.X2 – Hbase regionserver, Hbase master, gmond, gemetad, ganglia-web

For installing the packages in CentOS, use the following command:

yum install ganglia-gmond -y

yum install ganglia-gmetad -y

yum install ganglia-web -y

Edit ‘/etc/ganglia/gmond.conf’ on all nodes running the gmond service (sample is shown below)

Part 1:

cluster {

  name = "ThreeNodeClusterAJames"

  owner = "unspecified"

  latlong = "unspecified"

  url = "unspecified"

}

Part 2:

udp_send_channel {

  bind_hostname = yes # Highly recommended, soon to be default.

                       # This option tells gmond to use a source address

                       # that resolves to the machine's hostname.  Without

                       # this, the metrics may appear to come from any

                       # interface and the DNS names associated with

                       # those IPs will be used to create the RRDs.

  #mcast_join = 239.2.11.71

  host = 10.10.YY.XX

  port = 8649

  ttl = 1

}

NOTE:

·        We are using unicast protocol instead of multicast protocol. Please comment ‘mcast_join = 239.2.11.71’ and add “host = <IP of localhost running the gmond service>”

·        If there is any change in the port used, please change the same in ‘port’ property. Here I am using the default port which is 8649.

Part 3:

udp_recv_channel {

  #mcast_join = 239.2.11.71

  port = 8649

  #bind = 239.2.11.71

  #bind = 10.10.72.154

  #retry_bind = true

  # Size of the UDP buffer. If you are handling lots of metrics you really

  # should bump it up to e.g. 10MB or even higher.

  # buffer = 10485760

}

Edit ‘/etc/ganglia/gmetad.conf’ in gmetad server and add the following:

data_source "ThreeNodeClusterAJames" 10.10.YY.X1:8649 10.10.YY.X2:8649 10.10.YY.X3:8649

The cluster name ‘ThreeNodeClusterAJames’ should be same in all the gmond servers and gmetad server.

Edit ‘/opt/mapr/conf/hadoop-metrics.properties’ file and add

Part 1:

# Configuration of the "cldb" context for ganglia

cldb.class=com.mapr.fs.cldb.counters.MapRGangliaContext31

cldb.period=10

cldb.servers=10.10.YY.X1:8649,10.10.YY.X2:8649,10.10.YY.X3:8649

cldb.spoof=1

Part 2:

# Configuration of the "fileserver" context for ganglia

fileserver.class=com.mapr.fs.cldb.counters.MapRGangliaContext31

fileserver.period=37

fileserver.servers=10.10.YY.X1:8649,10.10.YY.X2:8649,10.10.YY.X3:8649

fileserver.spoof=1

Execute the following in any of the node and restart cldb service on all cldb nodes:

maprcli config save -values {"cldb.ganglia.cldb.metrics":"1"}

maprcli config save -values {"cldb.ganglia.fileserver.metrics":"1"}

 Execute the following in all nodes

sudo setenforce 0

Execute the following gmetad server:

chown -R apache:apache /usr/share/ganglia

chown -R ganglia:ganglia /var/lib/ganglia/rrd*

chcon -R -t httpd_sys_content_t *

Edit ‘/etc/httpd/conf.d/ganglia.conf’ file:

<Location /ganglia>

  Order deny,allow

#  Deny from all

  Allow from 127.0.0.1

  Allow from ::1

  # Allow from .example.com

</Location>

Restart the httpd, gmond and gmetad:

/etc/init.d/httpd restart

/etc/init.d/gmond restart

/etc/init.d/httpd restart

Edit ‘/opt/mapr/hbase/hbase-<version>/conf/hadoop-metrics2-hbase.properties’ file

hbase.sink.ganglia.class=org.apache.hadoop.metrics2.sink.ganglia.GangliaSink31

hbase.sink.ganglia.servers=<ganglia-server>:8649

hbase.sink.ganglia.period=10

hbase.class=org.apache.hadoop.metrics.ganglia.GangliaContext31

hbase.period=10

hbase.servers=<server-running-hbase-regionserver-1>:8649, <server-running-hbase-regionserver-2>:8649

# Configuration of the "jvm" context for ganglia

# Pick one: Ganglia 3.0 (former) or Ganglia 3.1 (latter)

# jvm.class=org.apache.hadoop.metrics.ganglia.GangliaContext

jvm.class=org.apache.hadoop.metrics.ganglia.GangliaContext31

jvm.period=10

jvm.servers=<server-running-hbase-regionserver-1>:8649, <server-running-hbase-regionserver-2>:8649

...

# Configuration of the "rpc" context for ganglia

# Pick one: Ganglia 3.0 (former) or Ganglia 3.1 (latter)

# rpc.class=org.apache.hadoop.metrics.ganglia.GangliaContext

rpc.class=org.apache.hadoop.metrics.ganglia.GangliaContext31

rpc.period=10

rpc.servers=<server-running-hbase-regionserver-1>:8649, <server-running-hbase-regionserver-2>:8649

...

# Configuration of the "rest" context for ganglia

# Pick one: Ganglia 3.0 (former) or Ganglia 3.1 (latter)

# rest.class=org.apache.hadoop.metrics.ganglia.GangliaContext

rest.class=org.apache.hadoop.metrics.ganglia.GangliaContext31

rest.period=10

rest.servers=<server-running-hbase-regionserver-1>:8649, <server-running-hbase-regionserver-2>:8649

Restart Hbase master and regionserver in all nodes.

You should be able to see the hbase metrics in ganglia web UI.

Ganglia UI listing the metrics collected:

Ganglia UI showing HBase metrics: