**1. 首先檢查是否安裝了 jdk8。** **2. 下載安裝包并解壓** 直接訪問：https://archive.apache.org/dist/spark/spark-2.4.4/spark-2.4.4-bin-hadoop2.6.tgz a. 如需歷史版本下載請訪問：https://archive.apache.org/dist/spark/。 b. Spark 不是必須安裝 Hadoop，但如果已經有 Hadoop 集群或 HDFS，建議下載對應版本的 Spark。 下載完成后解壓到/opt/install 目錄。 ```shell tar -zvxf spark-2.4.4-bin-hadoop2.6.tgz -C /opt/install -- 創建軟連接 ln -s spark-2.4.4-bin-hadoop2.6 spark ``` **3. 配置環境變量** ```sql vim /etc/profile export SPARK_HOME=/opt/install/spark export PATH=$SPARK_HOME/bin:$SPARK_HOME/sbin:$PATH ``` 使環境變量生效 ``` source /etc/profile ``` **4. 配置文件** ```sql cd /opt/install/spark/conf cp spark-env.sh.template spark-env.sh vim spark-env.sh export JAVA_HOME=/opt/install/jdk export HADOOP_HOME=/opt/install/hadoop export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop #指定 master 的主機 export SPARK_MASTER_HOST=hadoop101 #指定 master 的端口 export SPARK_MASTER_PORT=7077 ``` 修改配置文件 {spark_home}/conf/slaves（單臺可以不配置，也可以按下面方式配置） ```sql -- 先把 slaves.template 重命名為 slaves mv slaves.template slaves vim slaves hadoop101 ``` **5. 啟動 Spark 集群，在 spark 目錄下，輸入：** ```sql sbin/start-all.sh ``` 查看是否有 Master 與 Worker 進程 ```sql [root@hadoop101 spark]# jps 3694 Worker 3599 Master ``` **6. 啟動 spark-shell 測試 scala 交互式環境** ```sql [root@hadoop101 spark]# spark-shell --master spark://hadoop101:7077 或者 [root@hadoop101 sbin]# spark-shell --master local[4] 21/01/04 10:05:37 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). Spark context Web UI available at http://hadoop101:4040 Spark context available as 'sc' (master = local[4], app id = local-1609725951951). Spark session available as 'spark'. Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 2.4.4 /_/ Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_171) Type in expressions to have them evaluated. Type :help for more information. scala> ``` 可以通過瀏覽器訪問：http://hadoop101:8080/  **7. 測試 Spark on YARN** ```sql -- 需提前啟動 yarn 進程。啟動yarn [root@hadoop101 sbin]# ./start-yarn.sh -- Spark on Yarn [root@hadoop101 sbin]# spark-shell --master yarn 21/01/04 14:23:46 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 21/01/04 14:23:58 WARN util.Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041. 21/01/04 14:23:59 WARN yarn.Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME. Spark context Web UI available at http://hadoop101:4041 Spark context available as 'sc' (master = yarn, app id = application_1609741310740_0001). Spark session available as 'spark'. Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 2.4.4 /_/ Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_171) Type in expressions to have them evaluated. Type :help for more information. scala> ```