你的位置:首页 > 数据库

[数据库]朝花夕拾之


朝花夕拾之--大数据平台CDH集群离线搭建

Cloudera-Manager CDH Hadoop 部署 集群

摘要:管理、部署Hadoop集群需要工具,Cloudera Manager便是其一。本文先是简要对比了当前的类似工具,而后详细记录了以离线方式部署CDH集群的步骤。最后对“讲究”一词提出了自己的观点。


  • 朝花夕拾之--大数据平台CDH集群离线搭建
    • 前言
    • 部署步骤
      • 网络配置(所有节点)
      • 安装JDK(所有节点)
      • 安装MySQL(主节点)
      • 初始化MySQL(主节点)
      • 部署/启动CM Server(主节点)
      • 关闭防火墙(所有节点)
      • 部署/启动CM Agent(从节点)
      • 配置CDH
    • 其他
      • 停止集群步骤
      • 启动集群步骤
    • 题外话--论讲究

 


前言

以Apache Hadoop为主导的大数据技术的出现,使得中小型公司对于大数据的存储与处理也拥有了武器。目前Hadoop有不少发行版:华为发行版 收费、Intel发行版 收费、Cloudera发行版(Cloudera’s Distribution Including Apache Hadoop,简称 CDH免费、Hortonworks发行版(Hortonworks Data Platform,简称 HDP免费 等,所有这些发行版均是基于Apache Hadoop社区版衍生出来的。

部署、管理拥有数十数百甚至更多节点的Hadoop集群,也需要先进武器。Hortonworks公司的Apache Ambari项目的目的就是通过软件来配置、监控和管理Hadoop(HDP)集群,以使Hadoop的管理更加简单。Ambari提供了一个基于它自身RESTful的api实现的直观的、简单易用的web界面。Cloudera公司也提供了类似的工具:Cloudera Manager(简称 CM)来配置、监控和管理CDH集群。

本文主要内容即是本人早先搭建CDH集群之记录,故称作朝花夕拾。需特别注意的是Cloudera Manager与操作系统的版本关系 el7暂不支持,按照官方文档的要求来,否则安装会有问题。

注意用户。 
本文是基于操作系统CentOS 6.5, 64-bit;Cloudera Manager 5.3.6;JDK 1.7 版本进行部署的。



部署步骤


网络配置(所有节点)


  1. [root@cdh-server ~]# vi /etc/sysconfig/network #修改hostname:
  2. NETWORKING=yes
  3. HOSTNAME=cdh-server
  4. [root@cdh-server ~]# vi /etc/hosts #修改ip与主机名的对应关系:
  5. 192.168.180.173 cdh-server
  6. 192.168.180.175 node175
  7. [root@cdh-server ~]# service network restart #重启网络服务生效

安装JDK(所有节点)


  1. #卸载OpenJDK
  2. [root@cdh-server user1]# rpm -qa | grep java
  3. [root@cdh-server user1]# rpm -e --nodeps java-1.5.0-gcj-1.5.0.0-29.1.el6.x86_64
  4. [root@cdh-server user1]# rpm -e --nodeps java-1.6.0-openjdk-1.6.0.0-1.66.1.13.0.el6.x86_64
  5. [root@cdh-server user1]# rpm -e --nodeps java-1.7.0-openjdk-1.7.0.45-2.4.3.3.el6.x86_64
  6. #安装JDK
  7. [root@cdh-server user1]# chmod a+x jdk-7u79-linux-x64.rpm
  8. [root@cdh-server user1]# rpm -ivh jdk-7u79-linux-x64.rpm
  9. [root@cdh-server user1]# echo "JAVA_HOME=/usr/java/jdk1.7.0_79/" >>

安装MySQL(主节点)


  1. [user1@cdh-server]$ cd /home/user1
  2. [user1@cdh-server]$ tar -zxvf mysql-5.6.26-linux-glibc2.5-x86_64.tar.gz
  3. [user1@cdh-server]$ mv mysql-5.6.26-linux-glibc2.5-x86_64 mysql-5.6.26
  4. [user1@cdh-server]$ cd mysql-5.6.26/
  5. [user1@cdh-server]$ vi support-files/my.cnf #新建文件

  1. +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
  2. [mysqld]
  3. character-set-server=utf8
  4. default-storage-engine=INNODB
  5. # Uncomment the following if you are using InnoDB tables
  6. innodb_data_home_dir = /home/user1/mysql-5.6.26/data
  7. innodb_data_file_path = ibdata1:10M:autoextend
  8. innodb_log_group_home_dir = /home/user1/mysql-5.6.26/data
  9. # You can set .._buffer_pool_size up to 50 - 80 %
  10. # of RAM but beware of setting memory usage too high
  11. innodb_buffer_pool_size = 16M
  12. innodb_additional_mem_pool_size = 2M
  13. # Set .._log_file_size to 25 % of buffer pool size
  14. innodb_log_file_size = 5M
  15. innodb_log_buffer_size = 8M
  16. innodb_flush_log_at_trx_commit = 1
  17. innodb_lock_wait_timeout = 50
  18. +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

初始化MySQL(主节点)


  1. [user1@cdh-server]$ ./scripts/mysql_install_db --defaults-file=/home/user1/mysql-5.6.26/support-files/my.cnf --basedir=/home/user1/mysql-5.6.26 --datadir=/home/user1/mysql-5.6.26/data --user=user1
  2. [user1@cdh-server]$ ./bin/mysqld --defaults-file=/home/user1/mysql-5.6.26/support-files/my.cnf --basedir=/home/user1/mysql-5.6.26 --datadir=/home/user1/mysql-5.6.26/data > mysql.log 2>&1 &
  3. [user1@cdh-server]$ ./bin/mysqladmin -u root password '123456'

  1. [user1@cdh-server mysql-5.6.26]$ ./bin/mysql -uroot -p'123456'
  2. #hive
  3. mysql> create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
  4. Query OK, 1 row affected (0.00 sec)
  5. #Activity Monitor使用
  6. mysql> create database amon DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
  7. Query OK, 1 row affected (0.01 sec)
  8. #Navigator Audit Server使用
  9. mysql> create database audit DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
  10. Query OK, 1 row affected (0.01 sec)
  11. #Navigator Metadata Server
  12. mysql> create database metadata DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
  13. Query OK, 1 row affected (0.01 sec)
  14. mysql> grant all privileges on *.* to 'root'@'localhost' identified by '123456' with grant option;
  15. Query OK, 0 rows affected (0.00 sec)
  16. mysql> grant all privileges on *.* to 'root'@'cdh-server' identified by '123456' with grant option;
  17. Query OK, 0 rows affected (0.00 sec)
  18. #this user scm is for cloudera manager
  19. mysql> grant all privileges on *.* to 'scm'@'localhost' identified by 'scm' with grant option;
  20. Query OK, 0 rows affected (0.00 sec)
  21. mysql> grant all privileges on *.* to 'scm'@'cdh-server' identified by 'scm' with grant option;
  22. Query OK, 0 rows affected (0.00 sec)
  23. mysql> flush privileges;
  24. Query OK, 0 rows affected (0.00 sec)

部署/启动CM Server(主节点)


  1. [user1@cdh-server ~]$ tar -zxvf cloudera-manager-el6-cm5.3.6_x86_64.tar.gz
  2. [user1@cdh-server ~]$ cp mysql-connector-java-5.1.33-bin.jar ./cm-5.3.6/share/cmf/lib/
  3. [user1@cdh-server ~]$ su - root
  4. [root@cdh-server ~]# cd /home/user1/
  5. [root@cdh-server user1]# cp -rf cloudera /opt
  6. [root@cdh-server user1]# mv CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel /opt/cloudera/parcel-repo/CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel
  7. [root@cdh-server user1]# mv CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.sha /opt/cloudera/parcel-repo/CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.sha
  8. [root@cdh-server user1]# mv manifest.json /opt/cloudera/parcel-repo/manifest.json
  9. [root@cdh-server user1]# ./cm-5.3.6/share/cmf/schema/scm_prepare_database.sh mysql cm -hlocalhost:3306 -uroot -p123456 --scm-host localhost scm scm scm
  10. [root@cdh-server user1]# ./cm-5.3.6/etc/init.d/cloudera-scm-server start
  11. Starting cloudera-scm-server: [ OK ]
  12. [root@cdh-server user1]# tail -f ./cm-5.3.6/log/cloudera-scm-server/cloudera-scm-server.log

关闭防火墙(所有节点)


  1. #停止iptables
  2. [root@cdh-server user1]# service iptables stop
  3. #通过浏览器访问验证
  4. http://192.168.180.173:7180/

部署/启动CM Agent(从节点)


  1. [root@cdh-server user1]# tar -zxvf cloudera-manager-el6-cm5.3.6_x86_64.tar.gz
  2. [root@cdh-server user1]# vi cm-5.3.6/etc/cloudera-scm-agent/config.ini

  1. +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
  2. # Hostname of the CM server.
  3. #server_host=localhost
  4. server_host=cdh-server
  5. +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

  1. [root@cdh-server user1]# useradd -G sys --home=/home/user1/cm-5.3.6/run/cloudera-scm-server --no-create-home --comment "Cloudera SCM User" cloudera-scm
  2. [root@cdh-server user1]# useradd --comment "Cloudera SCM User" cloudera-scm #若上一步执行正确,则此步省略
  3. [root@cdh-server user1]# echo 0 > /proc/sys/vm/swappiness
  4. [root@cdh-server user1]# ./cm-5.3.6/etc/init.d/cloudera-scm-agent start
  5. Starting cloudera-scm-agent: [ OK ]
  6. [root@cdh-server user1]# tail -f ./cm-5.3.6/log/cloudera-scm-agent/cloudera-scm-agent.log

配置CDH

登陆Cloudera Manager http://192.168.180.173:7180/,并新建集群Cluster_user1,进行各服务的配置启动。


  1. #安装配置hive出错时,在hiveServer上:
  2. [root@hive-server user1]# cp mysql-connector-java-5.1.33-bin.jar /opt/cloudera/parcels/CDH-5.3.6-1.cdh5.3.6.p0.11/lib/hive/lib/
  3. #同理:use this jar for Navigator Audit Server and Navigator Metadata Server or Activity Server
  4. [root@cdh-server user1]# cp mysql-connector-java-5.1.33-bin.jar /usr/share/java/mysql-connector-java.jar

其他


停止集群步骤

  • 停止Cloudera Management Service和Cluster_user1
  • 从节点停止Agent

  1. [root@cdh-server user1]# ./cm-5.3.6/etc/init.d/cloudera-scm-agent stop
  • 主节点停止Server

  1. [root@cdh-server user1]# ./cm-5.3.6/etc/init.d/cloudera-scm-server stop

启动集群步骤

  • 主节点启动MySQL

  1. [user1@cdh-server]$ ./bin/mysqld --defaults-file=/home/user1/mysql-5.6.26/support-files/my.cnf --basedir=/home/user1/mysql-5.6.26 --datadir=/home/user1/mysql-5.6.26/data > mysql.log 2>&1 &
  2. [user1@cdh-server]$ ps -a | grep mysql
  • 从节点启动Agent

  1. [root@cdh-server user1]# ./cm-5.3.6/etc/init.d/cloudera-scm-agent start
  2. Starting cloudera-scm-agent: [ OK ]
  3. [root@cdh-server user1]# tail -f ./cm-5.3.6/log/cloudera-scm-agent/cloudera-scm-agent.log
  • 主节点启动Server

  1. [root@cdh-server user1]# ./cm-5.3.6/etc/init.d/cloudera-scm-server start
  2. Starting cloudera-scm-server: [ OK ]
  3. [root@cdh-server user1]# tail -f ./cm-5.3.6/log/cloudera-scm-server/cloudera-scm-server.log
  • 启动各服务 
    登陆Cloudera Manager http://192.168.180.173:7180/,进行各服务的检查启动。

题外话--论讲究

讲究 是一种情怀,也是一项技能。当处于卖方市场时,公司可能不需要太多的讲究就可以赚得盆满钵盈,然而如果它有更高的追求也就是情怀,可能会更讲究,比如更加注重细节、用户体验或者极力完美;而处于买方市场时,供需关系会使得公司不得不讲究起来,这对他们来说是一项必须技能。

讲究 既是推动人类社会发展的动力,也是人类社会发展的成果。当前很多互联网公司都处于买方市场,谁能赢得用户谁就拼得胜利,如履薄冰、小心翼翼、极致体验之讲究将他们推向了社会发展的浪尖,先进技术的发明和使用也是层出不穷。

讲究 不光体现于公司,也体现于个人、地域、国家等。愈是讲究的国家,愈是发达;讲究细节的公司,竞争力就越强。然而由于影响因素众多,讲究的个体有钱、赢钱与否则另当别论了。


作者 @王安琪 
我的头像 
 aitanjupt@hotmail.com 
2015 年 11月 24日



 此篇文章是本人另一英文文章的中文版本,英文版在此:A record--Offline deployment of Big Data Platform CDH Cluster,多谢指教!