How to install CDH6.0 Cluster on Centos7.5

2018-09-03 10:21:26
Cloudera 简介
安装Cloudera Manager和CDH
1
2
系统环境：CentOS7.5 
软件环境：Oracle JDK、Cloudera Manager Server 和 Agent 、数据库、CDH各组件

  • 系统初始化
    关闭防火墙 禁用selinux,服务器之间免密，时间保持同步

  • Cloudera安装,官方文档参考这里

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    #Configure a Repository
    wget https://archive.cloudera.com/cm6/6.0.0/redhat7/yum/cloudera-manager.repo -P /etc/yum.repos.d/
    sudo rpm --import https://archive.cloudera.com/cm6/6.0.0/redhat7/yum/RPM-GPG-KEY-cloudera

    #Installing the JDK
    sudo yum install oracle-j2sdk1.8

    #Install Cloudera Manager Packages
    sudo yum install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server

    #Install Databases
    wget http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm
    sudo rpm -ivh mysql-community-release-el7-5.noarch.rpm
    sudo yum update
    sudo yum install mysql-server
    sudo systemctl start mysqld

    #Set up the Cloudera Manager Database
    1. /opt/cloudera/cm/schema/scm_prepare_database.sh \
    [options] <databaseType> <databaseName> <databaseUser> <password>
    2. If it exists, remove the embedded PostgreSQL properties file:
    sudo rm /etc/cloudera-scm-server/db.mgmt.properties

    Example：
    sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm

    #Install CDH and Other Software
    sudo systemctl start cloudera-scm-server

    we go to browser http://<server_host>:7180,login admin/admin

  • 创建必需的数据库，可以参考这里

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    create database metastore DEFAULT CHARACTER SET utf8;
    grant all on metastore.* TO 'hive'@'%' IDENTIFIED BY 'hive';

    create database amon DEFAULT CHARACTER SET utf8;
    grant all on amon.* TO 'amon'@'%' IDENTIFIED BY 'amon';

    create database hue DEFAULT CHARACTER SET utf8;
    grant all on hue.* TO 'hue'@'%' IDENTIFIED BY 'hue';

    create database rman DEFAULT CHARACTER SET utf8;
    grant all on rman.* TO 'rman'@'%' IDENTIFIED BY 'rman';

    create database navms DEFAULT CHARACTER SET utf8;
    grant all on navms.* TO 'navms'@'%' IDENTIFIED BY 'navms';

    create database nas DEFAULT CHARACTER SET utf8;
    grant all on nas.* TO 'nas'@'%' IDENTIFIED BY 'nas';

    create database oos DEFAULT CHARACTER SET utf8;
    grant all on oos.* TO 'oos'@'%' IDENTIFIED BY 'oos';
卸载重装CM服务

如果，第一次没有安装成功，那这部分就对你有帮助了

1
2
3
4
5
6
7
8
9
# 安装CDH manager的服务器上面执行
yum remove cloudera-manager-server -y

# 在所有的服务器执行下面操作
systemctl stop  cloudera-scm-agent
yum remove cloudera-manager-agennt-y
ps -ef | grep cmf | grep -v grep | awk '{print $2}' | xargs kill -9
find / -name clouder* | xargs rm -rf 
find / -name cmf* | xargs rm -rf

Custom Installation Solutions

如果在线安装很慢，我们可以通过以下方式来加速安装

  • Creating a Permanent Internal Repository(Option)

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    #Setting Up a Web Server
    sudo yum install httpd
    sudo systemctl start httpd

    #Edit the Apache HTTP Server configuration file (/etc/httpd/conf/httpd.conf by default) to add or edit the following line in the <IfModule mime_module> section:
    AddType application/x-gzip .gz .tgz .parcel

    #Downloading and Publishing the Package Repository

    #Cloudera Manager 6
    sudo mkdir -p /var/www/html/cloudera-repos
    sudo wget --recursive --no-parent --no-host-directories https://archive.cloudera.com/cm6/6.0.0/redhat7/ -P /var/www/html/cloudera-repos
    sudo chmod -R ugo+rX /var/www/html/cloudera-repos/cm6

    #CDH 6
    sudo mkdir -p /var/www/html/cloudera-repos
    sudo wget --recursive --no-parent --no-host-directories https://archive.cloudera.com/cdh6/6.0.0/redhat7/ -P /var/www/html/cloudera-repos

    sudo wget --recursive --no-parent --no-host-directories https://archive.cloudera.com/gplextras6/6.0.0/redhat7/ -P /var/www/html/cloudera-repos

    sudo chmod -R ugo+rX /var/www/html/cloudera-repos/cdh6
    sudo chmod -R ugo+rX /var/www/html/cloudera-repos/gplextras6

  • Creating a Temporary Internal Repository(Option)

    1
    2
    3
    4
    5
    # Download the repository you need following the instructions in Downloading and Publishing the Package Repository.
    cd /var/www/html
    python -m SimpleHTTPServer 8900

    #Visit the Repository URL http://<web_server>:8900/cloudera-repos/

  • Configuring Hosts to Use the Internal Repository

    1
    2
    3
    4
    5
    6
    7
    8
    # Create /etc/yum.repos.d/cloudera-repo.repo files on cluster hosts with the following content, 
    where <web_server> is the hostname of the web server:

    [cloudera-repo]
    name=cloudera-repo
    baseurl=http://<web_server>/cm/5
    enabled=1
    gpgcheck=0
遇到的问题
1
2
3
4
5
6
7
8
# jdbc驱动未找到
mkdir -p /usr/share/java
rz mysql-connector-java-5.1.45-bin.jar
ln -s mysql-connector-java-5.1.45-bin.jar mysql-connector-java.jar
systemctl restart cloudera-scm-server.service

# host命令为找到
yum install bind-utils -y

ref
deploy-cm-cdh-on-centos7
Custom Installation Solutions
Using an Internally Hosted Remote Parcel Repository

您的鼓励是我写作最大的动力

俗话说，投资效率是最好的投资。 如果您感觉我的文章质量不错，读后收获很大，预计能为您提高 10% 的工作效率，不妨小额捐助我一下，让我有动力继续写出更多好文章。