Ganglia Yum 安装以及监控Hadoop 手记

Ganglia Yum 安装以及监控Hadoop 手记

Comments 0 Comment

最开始使用的源码安装，各种编译各种报错。然后发现epel中有yum源，于是就使用epel装了。下面是epel.repo的配置：

  1. [epel]
  2. name=Extra Packages for Enterprise Linux 6 - $basearch
  3. baseurl=http://download.fedoraproject.org/pub/epel/6/$basearch
  4. #mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-6&arch=$basearch
  5. failovermethod=priority
  6. enabled=1
  7. gpgcheck=1
  8. gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6
  10. [epel-debuginfo]
  11. name=Extra Packages for Enterprise Linux 6 - $basearch - Debug
  12. baseurl=http://download.fedoraproject.org/pub/epel/6/$basearch/debug
  13. #mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-debug-6&arch=$basearch
  14. failovermethod=priority
  15. enabled=0
  16. gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6
  17. gpgcheck=1
  19. [epel-source]
  20. name=Extra Packages for Enterprise Linux 6 - $basearch - Source
  21. baseurl=http://download.fedoraproject.org/pub/epel/6/SRPMS
  22. #mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-source-6&arch=$basearch
  23. failovermethod=priority
  24. enabled=0
  25. gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6
  26. gpgcheck=1
还用到了几个Centos6自带的rpm 包,下面这个是我本地的源，bill.repo配置如下，主要是Dvd1和Dvd2的包，其他的配置请忽略:
  1. [centos6.6-d1]
  2. name=centos6.6-dvd1
  3. enabled=1
  4. baseurl=http://yum-bill/centos6.6/Packages/
  5. gpgcheck=0
  6. #baseurl=file:///mnt/centos6.6
  7. #baseurl=http://192.168.24.49/centos6.6/Packages
  8. #gpgcheck=1
  9. #gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
  11. [centos6.6-d2]
  12. name=centos6.6-dvd2
  13. enabled=1
  14. gpgcheck=0
  15. baseurl=http://yum-bill/centos6.6/dvd2/Packages/
  17. [cloud]
  18. name=cloudstack4.5.1
  19. enabled=1
  20. gpgcheck=0
  21. baseurl=http://yum-bill/cloudstack4.5.1/
  23. [openvswitch]
  24. name=openvswitch
  25. enabled=1
  26. gpgcheck=0
  27. baseurl=http://yum-bill/openvswitch/
  29. [ceph6]
  30. name=ceph6
  31. enabled=1
  32. gpgcheck=0
  33. baseurl=http://yum-bill/ceph6/

安装

        服务端：

        下面的ganglia*包含了 ：ganglia ganglia-gmetad ganglia-gmond ganglia-web

  1. yum install rrdtool ganglia* pcre httpd php
        客户端：
  1. yum install ganglia-gmond
 配置
        服务端：
  1. #1、gmetad.conf的配置
  2. vi /etc/ganglia/gmetad.conf
  3. #去掉各种注释之后如下：
  4. data_source "hadoop-cluster" v1:8649 # 只修改了这个集群名称和服务器主机和端口，后面的都是默认值。集群名称和主机后面的gmond.conf需要用到
  5. setuid_username ganglia
  6. case_sensitive_hostnames 0
  8. #2、gmond.conf的配置，下面只列出修改过了配置。没列出来的即代表使用默认值
  9. vi /etc/gmond.conf
  10. cluster {
  11.   name = "hadoop-cluster" # 这里和上面gmetad.conf 主机一致
  12.   owner = "unspecified"
  13.   latlong = "unspecified"
  14.   url = "unspecified"
  15. }
  16. udp_send_channel {
  17. host = v1 // 使用host为单播，mcast_join为多播
  18. port = 8649
  19. ttl = 1
  20. }
  22. udp_recv_channel { // 如果使用单机广播，要删除“mcast_join”和“bind”
  23.   #mcast_join = 239.2.11.71
  24.   port = 8649
  25.   #bind = 239.2.11.71
  26.   retry_bind = true
  27.   # Size of the UDP buffer. If you are handling lots of metrics you really
  28.   # should bump it up to e.g. 10MB or even higher.
  29.   # buffer = 10485760
  30. }
  32. #3、开机启动相关
  33. # 开机运行采集进程
  34. chkconfig --levels 235 gmond on
  35. # 开机运行数据存储进程
  36. chkconfig --levels 235 gmetad on
  37. # 开机运行apache服务
  38. chkconfig --levels 235 httpd on
    客户端：
    
  1. # 在server端执行scp，将配置文件分发到client端。我这里发到了v2,v3,v4。也就是算上v1一共有4台机器
  2. scp /etc/ganglia/gmond.conf {ip}:/etc/ganglia/gmond.conf
  3. # 设置开机运行数据采集进程
  4. chkconfig --levels 235 gmond on
   

运行

    服务端：
         
  1. service gmond start
  2. service gmetad start
  3. service httpd start
    客户端：
    
  1. service gmond start
测试
    
  1. # 命令行打印当前活动client
  2. gstat -a
  4. # web显示当前client状态
  5. http://{your_ip}/ganglia

Apache密码验证

    通过web方式访问ganglia不需要密码，所以我们通过apache设置密码达到安全目的。
  1. ①
  2. htpasswd -c /etc/httpd/conf.d/passwords {your_name}
  4. ②
  5. cd /usr/share/ganglia
  7. vi .htaccess // 创建apache目录密码文件，并写入下面内容
  9. AuthType Basic
  10. AuthName "Restricted Files"
  11. AuthUserFile /etc/httpd/conf.d/passwords
  12. Require user {your_name} 
  14. ③
  15. vi /etc/httpd/conf/httpd.conf 
  16. <Directory />
  17.     Options FollowSymLinks
  18.     AllowOverride None
  19. </Directory>
  21. 修改为：
  23. <Directory />
  24.     Options FollowSymLinks
  25.     AllowOverride AuthConfig
  26. </Directory>
如果这个时候访问：http://v1/ganglia还是报错的话 （403 ERROR ），修改以下文件配置：
  1. vi /etc/httpd/conf.d/ganglia.con
  2. Alias /ganglia /usr/share/ganglia
  4. <Location /ganglia>
  5.   Order deny,allow
  6.   #Deny from all #将这行注释，写上下面那行
  7.   Allow from all
  8.   Allow from 127.0.0.1
  9.   Allow from ::1
  10.   # Allow from .example.com
  11. </Location
测试访问：
输入账号密码就可以了
进来之后是这样的：
可以看到4个节点都能监控到了
吸取个教训，以后但凡能通过yum装的，尽量用yum省时间。 安装篇差不多了，下面用它来监控Hadoop.
———————–—————––—————–—————–—————--Hadoop 监控分割线———————————–—————–—————–—————–
上面的监控是把4台机器都放到了一个组里面，和Hadoop并没有上面关系。下面要开始对Hadoop集群进行监控了，我目前4台机器搭建集群情况如下：
v1：Active Namenode/ResourceManager
v2：Standby Namenode/ResourceManager、DataNode
v3：DataNode
v4：DataNode

那么我这里会将原来的gmetad.conf和gmond.conf进行修改，同时还会修改Hadoop的hadoop-metrics2.properties的配置，这个文件修改后在Ganglia中可以看到很多Hadoop监控指标，超级爽！
修改如下配置：
    1.v1上的gmetad.conf
        原来的data_source只有一行，现在调整成两行,并且使用两个不同的端口,如下：
        
  1. data_source "hadoop-namenodes" v1:8649 v2:8649
  2. data_source "hadoop-datanodes" v3:8650 v4:8650 #注意这里是8650，后面datanode上的gmond.conf要用
  3. setuid_username ganglia
  4. case_sensitive_hostnames 0
    2.v1和v2的gmond.conf，这两个在这里我把它们当成hadoop-namenodes集群的配置
    原来的gmond.conf就改个cluser-name就好啦，别的例如端口不需要改，还是用8649
    
  1. /*
  2.  * The cluster attributes specified will be used as part of the <CLUSTER>
  3.  * tag that will wrap all hosts collected by this instance.
  4.  */
  5. cluster {
  6.   name = "hadoop-namenodes" #只需要修改这里
  7.   owner = "nobody"
  8.   latlong = "unspecified"
  9.   url = "unspecified"
  10. }
  12. /* The host section describes attributes of the host, like the location */
  13. host {
  14.   location = "unspecified"
  15. }
  17. /* Feel free to specify as many udp_send_channels as you like.  Gmond
  18.    used to only support having a single channel */
  19. udp_send_channel {
  20.   #bind_hostname = yes # Highly recommended, soon to be default.
  21.                        # This option tells gmond to use a source address
  22.                        # that resolves to the machine's hostname.  Without
  23.                        # this, the metrics may appear to come from any 
  24.                        # interface and the DNS names associated with
  25.                        # those IPs will be used to create the RRDs.
  26.   #mcast_join = 239.2.11.71
  27.   host = v1
  28.   port = 8649
  29.   ttl = 1 
  30. }
  32. /* You can specify as many udp_recv_channels as you like as well. */
  33. udp_recv_channel {
  34.   #mcast_join = 239.2.11.71
  35.   port = 8649
  36.   #bind = 239.2.11.71
  37. /*
  38.  * The cluster attributes specified will be used as part of the <CLUSTER>
  39.  * tag that will wrap all hosts collected by this instance.
  40.  */
  41. cluster {
  42.   name = "hadoop-namenodes"
  43.   owner = "nobody"
  44.   latlong = "unspecified"
  45.   url = "unspecified"
  46. }
  48. /* The host section describes attributes of the host, like the location */
  49. host {
  50.   location = "unspecified"
  51. }
  53. /* Feel free to specify as many udp_send_channels as you like.  Gmond
  54.    used to only support having a single channel */
  55. udp_send_channel {
  56.   #bind_hostname = yes # Highly recommended, soon to be default.
  57.                        # This option tells gmond to use a source address
  58.                        # that resolves to the machine's hostname.  Without
  59.                        # this, the metrics may appear to come from any 
  60.                        # interface and the DNS names associated with
  61.                        # those IPs will be used to create the RRDs.
  62.   #mcast_join = 239.2.11.71
  63.   host = v1
  64.   port = 8649
  65.   ttl = 1 
  66. }
  68. /* You can specify as many udp_recv_channels as you like as well. */
  69. udp_recv_channel {
  70.   #mcast_join = 239.2.11.71
  71.   port = 8649
  72.   #bind = 239.2.11.71
  73.   retry_bind = true
    3.修改v3和v4的gmond.conf,这里需要调整cluster-name和端口。将3个端口都改成8650同时把udp_send_channel-host修改成v3。当成hadoop-datanodes来配置
  1. /*
  2.  * The cluster attributes specified will be used as part of the <CLUSTER>
  3.  * tag that will wrap all hosts collected by this instance.
  4.  */
  5. cluster {
  6.   name = "hadoop-datanodes" #修改名称
  7.   owner = "nobody"
  8.   latlong = "unspecified"
  9.   url = "unspecified"
  10. }
  12. /* The host section describes attributes of the host, like the location */
  13. host {
  14.   location = "unspecified"
  15. }
  17. /* Feel free to specify as many udp_send_channels as you like.  Gmond
  18.    used to only support having a single channel */
  19. udp_send_channel {
  20.   #bind_hostname = yes # Highly recommended, soon to be default.
  21.                        # This option tells gmond to use a source address
  22.                        # that resolves to the machine's hostname.  Without
  23.                        # this, the metrics may appear to come from any 
  24.                        # interface and the DNS names associated with
  25.                        # those IPs will be used to create the RRDs.
  26.   #mcast_join = 239.2.11.71
  27.   host = v3  #修改
  28.   port = 8650 #修改
  29.   ttl = 1 
  30. }
  32. /* You can specify as many udp_recv_channels as you like as well. */
  33. udp_recv_channel {
  34.   #mcast_join = 239.2.11.71
  35.   port = 8650  #修改
  36.   #bind = 239.2.11.71
  37.   retry_bind = true
  38.   # Size of the UDP buffer. If you are handling lots of metrics you really
  39.   # should bump it up to e.g. 10MB or even higher.
  40.   # buffer = 10485760
  41. }
  43. /* You can specify as many tcp_accept_channels as you like to share
  44.    an xml description of the state of the cluster */
  45. tcp_accept_channel {
  46.   port = 8650 #修改
  47.   # If you want to gzip XML output
  48.   gzip_output = no
  49. }
    4.修改Hadoop的配置文件hadoop-metrics2.properties并且分发到另外3台机器，修改后的配置如下(这里只是把尾部的配置项目打开了，前面有些不相干的配置使用默认值)：
  1. #
  2. # Below are for sending metrics to Ganglia
  3. #
  4. # for Ganglia 3.0 support
  5. # *.sink.ganglia.class=org.apache.hadoop.metrics2.sink.ganglia.GangliaSink30
  6. #
  7. # for Ganglia 3.1 support
  8. *.sink.ganglia.class=org.apache.hadoop.metrics2.sink.ganglia.GangliaSink31
  10. *.sink.ganglia.period=10
  12. # default for supportsparse is false
  13. *.sink.ganglia.supportsparse=true
  15. *.sink.ganglia.slope=jvm.metrics.gcCount=zero,jvm.metrics.memHeapUsedM=both
  16. *.sink.ganglia.dmax=jvm.metrics.threadsBlocked=70,jvm.metrics.memHeapUsedM=40
  18. # Tag values to use for the ganglia prefix. If not defined no tags are used.
  19. # If '*' all tags are used. If specifiying multiple tags separate them with 
  20. # commas. Note that the last segment of the property name is the context name.
  21. #
  22. #*.sink.ganglia.tagsForPrefix.jvm=ProcesName
  23. #*.sink.ganglia.tagsForPrefix.dfs=
  24. #*.sink.ganglia.tagsForPrefix.rpc=
  25. #*.sink.ganglia.tagsForPrefix.mapred=
  27. namenode.sink.ganglia.servers=v1:8649
  29. datanode.sink.ganglia.servers=v3:8650
  31. resourcemanager.sink.ganglia.servers=v1:8649
  33. nodemanager.sink.ganglia.servers=v3:8650
  35. mrappmaster.sink.ganglia.servers=v1:8649
  37. jobhistoryserver.sink.ganglia.servers=v1:8649
    5.重启v1的gmetad和4台机器的gmond服务，然后重启整个Hadoop集群。
    
  1. #v1上执行
  2. service gmetad restart
  4. #4台机器上都执行
  5. service gmond restart
    6.再次访问v1上的ganglia，就可以看到两个cluster了。并且有很多Hadoop监控的指标，很方便！
    访问：192.168.30.31/ganglia，我这里用的v1的ip.
    结果：
    
选中hadoop-datanodes后：
选中hadoop-namenods后：
查看hadoop的相关指标：
好了，到这里就差不多了。 足够监控Hadoop集群使用了，下面研究下整合到Nagios实现报警。

发表回复

您的电子邮箱地址不会被公开。 必填项已用*标注