[TOC] ## 一 安裝JDK8環境和reids ### A JDK安裝 jkd版本:`jdk-8u151-linux-x64.rpm`,大版本一致即可 jdk下載鏈接可能因為小版本更新導致不可用,可以進下面鏈接找到最新的jdk8的下載連接 系統版本選擇:https://pkgs.org/download/java 當前最小版本:https://forensics.cert.org/centos/cert/7/x86_64//jdk-8u221-linux-x64.rpm ```sh mkdir -p /server/tools/ cd /server/tools/ wget https://forensics.cert.org/centos/cert/7/x86_64//jdk-8u221-linux-x64.rpm rpm -ivh jdk-8u221-linux-x64.rpm ``` ### B redis快速安裝 redis不是這里要學習的重點,所以只需快速部署啟動即可 ```sh yum install -y redis sed -i 's#127.0.0.1#10.0.0.11#g' /etc/redis.conf sed -i '/# requirepass/i requirepass abcd1234e' /etc/redis.conf cp /etc/redis.conf{,.bak} egrep -v "^#|^$" /etc/redis.conf.bak >/etc/redis.conf systemctl start redis.service [root@file_redis ~]# ss -lntuo|grep 6379 tcp LISTEN 0 128 10.0.0.11:6379 *:* ``` ## 二 Filebeat快速安裝配置 **Filebeat的工作原理：** 啟動Filebeat時，它會啟動一個或多個inputs，這些inputs將查找指定的log的路徑。對于查找到的每個日志，Filebeat將啟動一個harvester。每個harvester讀取單個日志的新內容，并將新日志數據發送到libbeat，libbeat聚合事件并將聚合數據發送到配置的output。 ### A filebeat安裝 ```sh #ELK的安裝源在上一章已經部署好 yum install -y filebeat cp /etc/filebeat/filebeat.yml{,.bak} egrep -v "#|^$" /etc/filebeat/filebeat.yml.bak >/etc/filebeat/filebeat.yml ``` ### B 當前filebeat配置 ```yml [root@file_redis ~]# cat /etc/filebeat/filebeat.yml filebeat.inputs: - type: log enabled: false paths: - /var/log/*.log filebeat.config.modules: path: ${path.config}/modules.d/*.yml reload.enabled: false setup.template.settings: index.number_of_shards: 3 setup.kibana: output.elasticsearch: hosts: ["localhost:9200"] processors: - add_host_metadata: ~ - add_cloud_metadata: ~ ``` ### C 精簡并修改配置 收集ssh登錄日志,打上tag:`ssh`,存入redis第二個庫中,并制定key名為`filebeat-1101` ```sh [root@zhimai-test ~]# cat /etc/filebeat/filebeat.yml filebeat.inputs: - type: log enabled: true paths: - /var/log/secure tags: ["web","ssh"] output.redis: hosts: ["10.0.0.11"] port: 6379 password : 'abcd1234e' key: "filebeat-1011" db: 2 timeout: 5 ``` 清空安全ssh日志,啟動filebeat ```sh >/var/log/secure systemctl start filebeat.service redis-cli -h 10.0.0.11 -a abcd1234e # 查看redis是否有數據 10.0.0.11:6379> select 2 OK 10.0.0.11:6379[2]> keys * (empty list or set) #現在看到的redis庫中還沒有數據 ``` 新開xshell窗口登錄服務器后,再查看redis信息 ```sh 10.0.0.11:6379[2]> keys * 1) "filebeat-1101" 10.0.0.11:6379[2]> lrange filebeat-1011 0 -1 ...很多數據,省略顯示... ``` ## 三 安裝部署elasticserach ### A 安裝并配置es ```sh yum install -y elasticsearch sed -i '/^#network.host:/a network.host: 10.0.0.12' /etc/elasticsearch/elasticsearch.yml sed -i '/^#http.port:/a http.port: 9200' /etc/elasticsearch/elasticsearch.yml cp /etc/elasticsearch/elasticsearch.yml{,.bak} egrep -v "^#|^$" /etc/elasticsearch/elasticsearch.yml.bak >/etc/elasticsearch/elasticsearch.yml systemctl start elasticsearch.service ``` ### B 驗證es啟動結果 ```sh [root@log_es ~]# ss -lntup|grep 9200 tcp LISTEN 0 128 ::ffff:10.0.0.12:9200 :::* users:(("java",pid=31378,fd=197)) [root@log_es ~]# curl 10.0.0.12:9200 { "name" : "GCQ8mIk", "cluster_name" : "elasticsearch", "cluster_uuid" : "QIjZQfUrQwO0Keh-XPn8QQ", "version" : { "number" : "6.8.4", "build_flavor" : "default", "build_type" : "rpm", "build_hash" : "bca0c8d", "build_date" : "2019-10-16T06:19:49.319352Z", "build_snapshot" : false, "lucene_version" : "7.7.2", "minimum_wire_compatibility_version" : "5.6.0", "minimum_index_compatibility_version" : "5.0.0" }, "tagline" : "You Know, for Search" } ``` ## 四 安裝部署logstash ### A 安裝并驗證logstash **安裝logstash:** ```sh yum install -y logstash ``` **命令行啟動驗證** 通過命令行啟動,驗證屏幕輸入是否輸出是否正常,這是調試logstash的常用方式 ```sh [root@log_es ~]# /usr/share/logstash/bin/logstash -e "input { stdin { type => stdin } } output { stdout { codec => rubydebug } }" ....啟動過程....省略..... abcd test { "@timestamp" => 2019-11-15T03:43:55.953Z, "message" => "abcd test", "type" => "stdin", "host" => "log_es", "@version" => "1" } # "abcd test" 是鍵盤輸入的內容,后面的是輸出到屏幕的內容 ``` **配置文件啟動驗證** 將命令行內容寫入配置文件,然后啟動驗證 ``` #1. 配置文件 [root@log_es ~]# cat /etc/logstash/conf.d/test.conf input { stdin { type => "stdin" } } output { stdout { codec => rubydebug } } #2. 驗證配置文件 [root@log_es ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/test.conf -t ...... Configuration OK ...... 3. 用配置文件啟動 [root@log_es ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/test.conf ........ [INFO ] 2019-11-15 17:30:35.777 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600} conf test { "@version" => "1", "host" => "log_es", "@timestamp" => 2019-11-15T09:32:24.835Z, "message" => "conf test", "type" => "stdin" } ``` ### B 編寫配置文件 **配置文件** ```sh [root@log_es ~]# cat /etc/logstash/conf.d/test.conf input { redis { data_type => "list" key => "filebeat-1011" host => "10.0.0.11" port => 6379 password => 'abcd1234e' db => "2" threads => 5 codec => "json" } } output { elasticsearch { hosts => ["10.0.0.12:9200"] index => "1011-logs-%{+YYYY.MM}" } } ``` ### C 啟動logstash并在es中查看結果 ```sh [root@log_es ~]# systemctl restart logstash.service [root@log_es ~]# ss -lntup|grep 9600 tcp LISTEN 0 50 ::ffff:127.0.0.1:9600 :::* users:(("java",pid=33956,fd=104)) [root@log_es ~]# curl 10.0.0.12:9200/_cat/indices yellow open 1011-logs-2019.11 zJ-O37DWSzKfGWv6mXJnYw 5 1 5 0 45.3kb 45.3kb [root@log_es ~]# curl -XGET 'http://10.0.0.12:9200/1011-logs-2019.11' .....一大串json格式內容....... ``` ## 五 安裝部署kibana ### A **安裝啟動kibana:** ```sh yum install -y kibana cp /etc/kibana/kibana.yml{,.bak} # 寫入配置文件 cat >/etc/kibana/kibana.yml <<EOF server.port: 5601 server.host: "10.0.0.12" elasticsearch.url: "http://10.0.0.12:9200" EOF # 啟動kibana systemctl start kibana ``` ### B 瀏覽器訪問kibana 瀏覽器訪問`http://10.0.0.12:5601`,可以進入kibana的web界面即可 ## 結束語 至此,filebeat+redis+logstash+elasticsearch+kibana的ELK架構已經搭建完成,關于各個組件的詳細內容,在后續章節再討論