# centos7離線安裝Ambari2.6+HDP2.6.4+ Hadoop集群 [TOC] # 第1章 概述 安裝總共分為四個部分 1.準備工作,服務器的基本配置 2.安裝Ambari 3.使用Ambari安裝HDP 4.排錯和調優 # 第2章 準備工作,服務器基本配置 ## 2.1 機器準備 hosts和主機名,這里選用三臺主機,1個控制節點,2個數據節點,控制節點的內存給大點 ``` 192.168.56.210 master.vim26.com 192.168.56.220 data1.vim26.com 192.168.56.230 data2.vim26.com ``` ## 2.2 軟件準備 因為默認的源都是在國外,而且很大,一共加起來差不多6個G左右,所以國內安裝的話很容易超時中斷導致安裝失敗,所以這里采用創建本地倉庫的方式來安裝Ambari以及HDP的CentOS7版本 ### 2.2.1 Ambari 2.6.1: ``` http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.6.1.0/ambari-2.6.1.0-centos7.tar.gz ``` ### 2.2.2 HDP-2.6.4.0: ``` http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.4.0/HDP-2.6.4.0-centos7-rpm.tar.gz ``` ### 2.2.3 HDP-UTILS-1.1.0.22: ``` http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.22/repos/centos7/HDP-UTILS-1.1.0.22-centos7.tar.gz ``` ### 2.2.4 HDP-GPL-2.6.4.0: ``` http://public-repo-1.hortonworks.com/HDP-GPL/centos7/2.x/updates/2.6.4.0/HDP-GPL-2.6.4.0-centos7-rpm.tar.gz ``` ### 2.2.5 其他操作系統版本的官方地址: ``` https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.1.0/bk_ambari-installation/content/ambari_repositories.html ``` ### 2.2.6 mysql的repo源,非必須 這里如果不添加的話當安裝HDP的時候會提示找不到mysql,當然你也可以手動安裝mysql,然后配置帳號密碼, ``` rpm -ivh http://repo.mysql.com/yum/mysql-5.7-community/el/7/x86_64/mysql-community-release-el7-7.noarch.rpm ``` ### 2.2.7 jdbc驅動,非必須 后面安裝服務的時候會有提示下載的地址: ``` https://cdn.mysql.com//Downloads/Connector-J/mysql-connector-java-5.1.45.tar.gz ``` ### 2.2.8 java環境 可以自己安裝,也可以使用默認安裝腳本自動安裝,不過腳本安裝的時候是從國外的源下載,可能會很慢 ### 2.2.9 apache httpd 提供離線倉庫的下載服務 yum install httpd -y ### 2.2.10 ntp時間服務 :所有節點都必須安裝ntpd服務,不然安裝的時候會提示警告 ``` yum install ntp -y ``` ## 2.3 環境設置 每個機器的hosts,hostname,關閉防火墻,selinux,ssh免密鑰登錄. ``` #hosts地址 192.168.56.210 master210.vim.com 192.168.56.220 data220.vim.com 192.168.56.230 data230.vim.com #關閉selinux sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config setenforce 0 #關閉防火墻 systemctl stop firewalld.service systemctl disable firewalld.service #生成密鑰并免密鑰認證 ssh-keygen ssh-copy-id -i .ssh/id_rsa.pub 192.168.56.240 ssh-copy-id -i .ssh/id_rsa.pub 192.168.56.230 ssh-copy-id -i .ssh/id_rsa.pub 192.168.56.220 ``` ## 2.4 所有節點安裝ntp ``` yum install ntp -y ``` ## 2.5 主節點安裝httpd ``` yum install httpd -y ``` ## 2.6 上傳倉庫軟件 ``` #把下載下來的文件都解壓到/var/www/html/ambari下,最終目錄結構如下: [root@master210 ~]#mkdir /var/www/html/ambari/ [root@master210 ~]# tree -Ld 3 /var/www/html/ambari/ /var/www/html/ambari/ ├── ambari │ └── centos7 │ └── 2.6.1.0-143 ├── HDP │ └── centos7 │ └── 2.6.4.0-91 ├── HDP-GPL │ └── centos7 │ └── 2.6.4.0-91 └── HDP-UTILS-1.1.0.22 ├── openblas ├── repodata ├── RPM-GPG-KEY └── snappy ``` ## 2.7 修改repo源文件 這里可以從每個目錄下的repo文件拷貝過來,或者直接新建文件然后復制以下內容: ambari.repo ``` [root@master210 ~]# cat /etc/yum.repos.d/ambari.repo #VERSION_NUMBER=2.6.1.0-143 [ambari-2.6.1.0] name=ambari Version - ambari-2.6.1.0 baseurl=http://192.168.56.210/ambari/ambari/centos7/2.6.1.0-143/ gpgcheck=1 gpgkey=http://192.168.56.210/ambari/ambari/centos7/2.6.1.0-143/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins enabled=1 priority=1 ``` hdp.repo ``` [root@master210 ~]# cat /etc/yum.repos.d/hdp.repo #VERSION_NUMBER=2.6.4.0-91 [HDP-2.6.4.0] name=HDP Version - HDP-2.6.4.0 baseurl=http://192.168.56.210/ambari/HDP/centos7/2.6.4.0-91/ gpgcheck=1 gpgkey=http://192.168.56.210/ambari/HDP/centos7/2.6.4.0-91/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins enabled=1 priority=1 [HDP-UTILS-1.1.0.22] name=HDP-UTILS Version - HDP-UTILS-1.1.0.22 baseurl=http://192.168.56.210/ambari/HDP-UTILS-1.1.0.22/ gpgcheck=1 gpgkey=http://192.168.56.210/ambari/HDP-UTILS-1.1.0.22/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins enabled=1 priority=1 ``` hdp.gpl.repo ``` [root@master210 ~]# cat /etc/yum.repos.d/hdp.gpl.repo #VERSION_NUMBER=2.6.4.0-91 [HDP-GPL-2.6.4.0] name=HDP-GPL Version - HDP-GPL-2.6.4.0 baseurl=http://192.168.56.210/ambari/HDP-GPL/centos7/2.6.4.0-91/ gpgcheck=1 gpgkey=http://192.168.56.210/ambari/HDP-GPL/centos7/2.6.4.0-91/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins enabled=1 priority=1 ``` 如果使用mysql的話也需要mysql的repo,不然安裝的時候會提示找不到,這里很奇怪,因為下載的離線源里并沒有mysql,并且安裝的時候會去mysql的官方源里去找,所以是個坑.下面有解決 ``` rpm -ivh http://repo.mysql.com/yum/mysql-5.7-community/el/7/x86_64/mysql-community-release-el7-7.noarch.rpm ``` 清理緩存并生成源文件 ``` yum clean all yum makecache ``` # 第3章 使用自定義mysql數據庫安裝Ambari 可以使用默認配置安裝的hadoop集群,也可以使用自己定義的環境,比如mysql和java ## 3.1 yum安裝mysql和java環境 ``` [root@hadoop-node1 html]# yum install -y java-1.8.0 mariadb mariadb-server [root@hadoop-node1 html]# java -version openjdk version "1.8.0_161" OpenJDK Runtime Environment (build 1.8.0_161-b14) OpenJDK 64-Bit Server VM (build 25.161-b14, mixed mode) [root@hadoop-node1 html]# systemctl start mariadb [root@hadoop-node1 html]# systemctl enable mariadb Created symlink from /etc/systemd/system/multi-user.target.wants/mariadb.service to /usr/lib/systemd/system/mariadb.service. [root@hadoop-node1 html]# systemctl status mariadb ● mariadb.service - MariaDB database server Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled) Active: active (running) since Mon 2018-01-29 17:44:54 CST; 23s ago Main PID: 1800 (mysqld_safe) CGroup: /system.slice/mariadb.service ├─1800 /bin/sh /usr/bin/mysqld_safe --basedir=/usr └─1962 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin- ................................. ``` ## 3.2 初始化mysql數據庫 ``` [root@hadoop-node1 html]# mysql_secure_installation NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB SERVERS IN PRODUCTION USE! PLEASE READ EACH STEP CAREFULLY! In order to log into MariaDB to secure it, we'll need the current password for the root user. If you've just installed MariaDB, and you haven't set the root password yet, the password will be blank, so you should just press enter here. Enter current password for root (enter for none): OK, successfully used password, moving on... Setting the root password ensures that nobody can log into the MariaDB root user without the proper authorisation. Set root password? [Y/n] New password: Re-enter new password: Password updated successfully! Reloading privilege tables.. ... Success! By default, a MariaDB installation has an anonymous user, allowing anyone to log into MariaDB without having to have a user account created for them. This is intended only for testing, and to make the installation go a bit smoother. You should remove them before moving into a production environment. Remove anonymous users? [Y/n] ... Success! Normally, root should only be allowed to connect from 'localhost'. This ensures that someone cannot guess at the root password from the network. Disallow root login remotely? [Y/n] ... Success! By default, MariaDB comes with a database named 'test' that anyone can access. This is also intended only for testing, and should be removed before moving into a production environment. Remove test database and access to it? [Y/n] - Dropping test database... ... Success! - Removing privileges on test database... ... Success! Reloading the privilege tables will ensure that all changes made so far will take effect immediately. Reload privilege tables now? [Y/n] ... Success! Cleaning up... All done! If you've completed all of the above steps, your MariaDB installation should now be secure. Thanks for using MariaDB! ``` ## 3.3 創建Ambari帳號和hive服務帳號以及數據庫 ``` [root@hadoop-node1 html]# mysql -u root -p123123 Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 16 Server version: 5.5.56-MariaDB MariaDB Server Copyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [(none)]> create database ambari default character set utf8; Query OK, 1 row affected (0.00 sec) MariaDB [(none)]> grant all on ambari.* to ambari@localhost identified by 'bigdata'; Query OK, 0 rows affected (0.00 sec) MariaDB [(none)]> grant all on ambari.* to ambari@'%' identified by 'bigdata'; Query OK, 0 rows affected (0.00 sec) MariaDB [(none)]> create database hive default character set utf8; Query OK, 1 row affected (0.00 sec) MariaDB [(none)]> grant all on hive.* to hive@localhost identified by 'hive'; Query OK, 0 rows affected (0.00 sec) MariaDB [(none)]> grant all on hive.* to hive@'%' identified by 'hive'; Query OK, 0 rows affected (0.00 sec) ``` ## 3.4 安裝Ambari服務 ``` [root@hadoop-node1 ~]# yum install ambari-server.x86_64 ``` ## 3.5 配置Ambari服務 這里配置的過程中需要修改java環境的變量以及導入mysql文件,先把完整過程放出,修改的部分在后面 ``` [root@hadoop-node1 ~]# ambari-server setup Using python /usr/bin/python Setup ambari-server Checking SELinux... SELinux status is 'disabled' Customize user account for ambari-server daemon [y/n] (n)? y Enter user account for ambari-server daemon (root):ambari Adjusting ambari-server permissions and ownership... Checking firewall status... Checking JDK... [1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8 [2] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7 [3] Custom JDK ============================================================================== Enter choice (1): 3 WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts. WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts. Path to JAVA_HOME: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/jre/ ###這里填寫本機已經安裝好的java軟件目錄 Validating JDK on Ambari Server...done. Checking GPL software agreement... GPL License for LZO: https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html Enable Ambari Server to download and install GPL Licensed LZO packages [y/n] (n)? Completing setup... Configuring database... Enter advanced database configuration [y/n] (n)? y Configuring database... ============================================================================== Choose one of the following options: [1] - PostgreSQL (Embedded) [2] - Oracle [3] - MySQL / MariaDB [4] - PostgreSQL [5] - Microsoft SQL Server (Tech Preview) [6] - SQL Anywhere [7] - BDB ============================================================================== Enter choice (1): 3 Hostname (localhost): Port (3306): Database name (ambari): Username (ambari): Enter Database Password (bigdata): Configuring ambari database... WARNING: Before starting Ambari Server, you must copy the MySQL JDBC driver JAR file to /usr/share/java and set property "server.jdbc.driver.path=[path/to/custom_jdbc_driver]" in ambari.properties. ###這里提示需要配置java的環境變量到ambari.properties這個文件里,參考下面java配置修改內容 Press <enter> to continue. Configuring remote database connection properties... WARNING: Before starting Ambari Server, you must run the following DDL against the database to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql Proceed with configuring remote database connection properties [y/n] (y)? ###這里提示需要導入SQL的DDL文件,操作參考下面mysql的部分 Extracting system views... ambari-admin-2.6.1.0.143.jar ........... Adjusting ambari-server permissions and ownership... Ambari Server 'setup' completed successfully. ``` ### 3.5.1 java路徑 默認yum安裝到/usr/lib/jvm ,后面的版本號可能不同,注意修改成自己本機的安裝位置 ``` /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/jre/ ``` ### 3.5.2 配置mysql的jdbc ``` [root@hadoop-node1 hadoop_soft]# ls ambari.repo hdp.repo mysql-connector-java-5.1.45.tar.gz [root@hadoop-node1 hadoop_soft]# tar zxf mysql-connector-java-5.1.45.tar.gz [root@hadoop-node1 hadoop_soft]# cd mysql-connector-java-5.1.45/ [root@hadoop-node1 mysql-connector-java-5.1.45]# ls build.xml CHANGES COPYING mysql-connector-java-5.1.45-bin.jar README README.txt src [root@hadoop-node1 mysql-connector-java-5.1.45]# mv mysql-connector-java-5.1.45-bin.jar /usr/share/java/ [root@hadoop-node1 mysql-connector-java-5.1.45]# vim /etc/ambari-server/conf/ambari.properties ###在jdbc區下面找個位置添加如下如下語句 server.jdbc.driver.path=/usr/share/java/mysql-connector-java-5.1.45-bin.jar ``` ### 3.5.3 導入mysql的DDL文件 ``` [root@hadoop-node1 ~]# mysql -uroot -p123123 ambari < /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql [root@hadoop-node1 ~]# ``` ## 3.6 啟動Ambari服務 ``` [root@hadoop-node1 ~]# ambari-server start Using python /usr/bin/python Starting ambari-server Ambari Server running with administrator privileges. Organizing resource files at /var/lib/ambari-server/resources... Ambari database consistency check started... Server PID at: /var/run/ambari-server/ambari-server.pid Server out at: /var/log/ambari-server/ambari-server.out Server log at: /var/log/ambari-server/ambari-server.log Waiting for server start................................... Server started listening on 8080 DB configs consistency check: no errors and warnings were found. Ambari Server 'start' completed successfully. ###查看端口 [root@hadoop-node1 ~]# lsof -i:8080 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME java 2932 ambari 1454u IPv6 31635 0t0 TCP *:webcache (LISTEN) ``` # 第4章 使用mysql數據庫配置Ambari Web安裝 下面的配置和前面的默認配置大部分一樣,但是選擇數據庫的時候有區別 ## 4.1 登錄 帳號密碼均為admin  ## 4.2 安裝向導創建集群  ## 4.3 集群命名  ## 4.4 選擇安裝版本并使用離線倉庫  ## 4.5 配置安裝節點信息  ## 4.6 安裝ambari客戶端 這里報了個錯,提示2個節點沒有安裝java,看來需要所有節點都安裝java環境  安裝完成后重新檢測一下警告消除了,顯示全部節點安裝成功  ## 4.7 選擇安裝服務 只選擇spark2之后點擊next會自動幫我們把需要依賴的服務都安裝上 ## 4.8 分配服務器角色 這里保持默認即可,或者將控制服務都安裝在一個節點上  ## 4.9 分配slaves服務 這里不太明白,不過可以都勾上  ? ## 4.10 配置服務 ### 4.10.1 hive修改 這里會有三處需要修改的地方: 1,填寫hive的密碼 2,選擇mysql 3,上傳mysql-jdbc包并運行設置命令  ``` [root@hadoop-node1 java]# cd /usr/share/java [root@hadoop-node1 java]# ambari-server setup --jdbc-db=mysql --jdbc-driver=mysql-connector-java-5.1.45-bin.jar Using python /usr/bin/python Setup ambari-server Copying mysql-connector-java-5.1.45-bin.jar to /var/lib/ambari-server/resources If you are updating existing jdbc driver jar for mysql with mysql-connector-java-5.1.45-bin.jar. Please remove the old driver jar, from all hosts. Restarting services that need the driver, will automatically copy the new jar to the hosts. JDBC driver was successfully initialized. Ambari Server 'setup' completed successfully. ``` ### 4.10.2 Ambari Metrics配置 密碼都設置為admin即可  ### 4.10.3 SmartSense修改 設置為admin即可  ## 4.11 查看和確認預配置  ## 4.12 安裝組件   點擊每個選項可以查看具體執行的過程內容輸出  ## 4.13 安裝完成  如果全部都是sucess那無疑是很開心的 ## 4.14 配置修改以及服務啟動 這里是展示頁面,還可以調整配置  配置服務參數的頁面  設置服務啟動頁面  如果用不到的服務也可以刪掉