2. Logstash實戰 · 運維工作筆記

#### 請先看看結尾的坑 ### logstash注意事項 - ##### logstash可以啟動多個端口接收數據 - ##### 重啟logstash可能會卡住，一般由于輸出到其他服務導致（Reids服務沒啟動，驗證沒通過），一般會選擇kill - ##### 執行前先檢查語法，并前臺測試啟動 #### 檢查logstash語法 ```shell /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/system.conf -t # logstash前臺啟動 /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/system.conf ``` 測試階段可以將收集的日志輸出到本地文件中，調試后發往目標 ### 收集日志，存儲在本地文件 ```shell #cat /etc/logstash/conf.d/system.conf input{ file { type => "systemlog" path => "/var/log/messages" start_position => "beginning" stat_interval => "5" } } output { file { path => "/tmp/systemlog.log" } } ``` ### 使用logstash收集系統messages日志文件無權限報錯 ```shell [2017-08-29T16:50:00,834][INFO ][logstash.pipeline ] Pipeline main started [2017-08-29T16:50:00,852][WARN ][logstash.inputs.file ] failed to open /var/log/messages: Permission denied - /var/log/messages [2017-08-29T16:50:00,886][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600} ``` ```shell #系統日志，默認600,會報錯 chmod 644 /var/log/messages ``` #### logstash中增加配置（/etc/logstash/conf.d/systemlog.conf） ```shell #cat /etc/logstash/conf.d/system.conf input{ file { type => "systemlog" path => "/var/log/messages" start_position => "beginning" stat_interval => "5" } } output{ elasticsearch{ hosts => ["192.168.0.231:9200"] index => "logstash-systemlog-%{+YYYY.MM.dd}" } } ``` ### logstash收集Nginx日志 #### Nginx配置成Json格式日志配置摘自[jack.zhang的博客](http://www.cnblogs.com/zhang-shijie/p/5384624.html "jack.zhang的博客")，感謝分享 ```json log_format logstash_json '{"@timestamp":"$time_local",' '"remote_addr":"$remote_addr",' '"remote_user":"$remote_user",' '"body_bytes_sent":"$body_bytes_sent",' '"request_time":"$request_time",' '"status":"$status",' '"request":"$request",' '"request_method":"$request_method",' '"http_referrer":"$http_referer",' '"body_bytes_sent":"$body_bytes_sent",' '"http_x_forwarded_for":"$http_x_forwarded_for",' '"http_user_agent":"$http_user_agent"}'; access_log /var/log/nginx/access.log logstash_json; ``` json日志 ```json { "@timestamp": "30/Aug/2017:10:18:59 +0800", "remote_addr": "192.168.2.64", "remote_user": "-", "body_bytes_sent": "0", "request_time": "0.000", "status": "304", "request": "GET /nginxweb/ HTTP/1.1", "request_method": "GET", "http_referrer": "-", "http_x_forwarded_for": "-", "http_user_agent": "Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36" } ``` logstash中增加配置（/etc/logstash/conf.d/nginx.conf） ```json input { file { path => "/var/log/nginx/access.log" type => "nginx-accesslog" start_position => "beginning" } } output { if [type] == "nginx-accesslog" { elasticsearch { hosts => ["192.168.0.231:9200"] index => "nginx-accesslog-%{+YYYY.MM.dd}" } } } ``` ### logstash收集Tomcat訪問日志 #### 配置Tomcat日志為Json格式配置摘自[jack.zhang的博客](http://www.cnblogs.com/zhang-shijie/p/5384624.html "jack.zhang的博客")，感謝分享 #### 修改tomcat/conf/server.xml結尾部分的日志配置修改日志名稱，結尾格式，和pattern ```xml <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs" prefix="tomcat_access" suffix=".log" pattern="{"client":"%h","client user":"%l", "authenticated":"%u", "access time":"%t", "method":"%r", "status":"%s","send bytes":"%b","Query?string":"%q","partner":"%{Referer}i","Agent version":"%{User-Agent}i"}"/> ``` 重啟Tomcat，查看日志格式 ```json { "client": "192.168.2.64", "client user": "-", "authenticated": "-", "access time": "[31/Aug/2017:09:48:29 +0800]", "method": "GET /web/index.html HTTP/1.1", "status": "304", "send bytes": "-", "Query?string": "", "partner": "-", "Agent version": "Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36" } ``` logstash中增加配置（/etc/logstash/conf.d/tomcat.conf） ```json input{ file { path => "/app/tomcat1/logs/tomcat_access.*.log" type => "tomcatlog" start_position => "beginning" stat_interval => "5" } } output{ if[type] == "tomcatlog" { elasticsearch { hosts => ["192.168.0.231:9200"] index => "tomcatlog-%{+YYYY.MM.dd}" } } } ``` ### rsyslog收集HAproxy日志，發送到logstash #### haproxy日志配置設置HAproxy日志輸出到local6 ```shell global ...(略) log 127.0.0.1 local6 info ...(略) ``` #### rsyslog配置將local6的所有日志發送到logstash的5555端口 ```shell # Provides UDP syslog reception $ModLoad imudp $UDPServerRun 514 # Provides TCP syslog reception $ModLoad imtcp $InputTCPServerRun 514 local6.* @@192.168.0.230:5555 ``` #### logstash配置 ```shell input { syslog { type => "rsyslog-haproxy-8888" port => "5555" } } output{ if [type] == "rsyslog-haproxy-8888" { elasticsearch { hosts => ["192.168.0.232:9200"] index => "haproxy-%{+YYYY.MM.dd}" } } } ``` ### 一個logstash配置文件中，收集多個日志，建立不同索引 #### logstash中增加配置（/etc/logstash/conf.d/systemlog.conf） ```shell input{ file { #定義type類型，用于輸出判斷 type => "systemlog" path => "/var/log/messages" start_position => "beginning" stat_interval => "5" } file { path => "/var/log/lastlog" #定義type類型，用于輸出判斷 type => "system-last" start_position => "beginning" stat_interval => "5" } } output{ #判斷輸入類型，輸出不同索引名稱 if [type] == "systemlog" { elasticsearch{ hosts => ["192.168.0.231:9200"] index => "logstash-systemlog-%{+YYYY.MM.dd}" } } #判斷輸入類型，輸出不同索引名稱 if [type] == "system-last" { elasticsearch { hosts => ["192.168.0.231:9200"] index => "logstash-lastlog-%{+YYYY.MM.dd}" } } } ``` ### logstash收集beats組件信息，并轉發到不同的目標 ```shell input { beats { port => 5044 } } output { if [type] == "web02-tomcat-info" { redis { host => ["192.168.0.106"] data_type => "list" db => "3" key => "web02-tomcat-info" port => "6400" password => "123456" batch => "true" } } if [type] == "web02-tomcat-error" { elasticsearch { hosts => ["192.168.0.231:9200"] index => "web02-tomcat-error" } } } ``` ### logstash消費redis中的日志 ```shell input { redis { data_type => "list" db => "3" host => "192.168.0.106" port => "6400" key => "web02-tomcat-info" password => "123456" } } output { if [type] == "web02-tomcat-info" { elasticsearch { hosts => ["192.168.0.231:9200"] index => "tomcat-info-%{+YYYY.MM.dd}" } } } ``` ### 跳坑 #### 坑1. A數據，寫入到了B索引中，造成數據混亂 ##### ELK版本: 5.5.2 現狀描述：logstash的配置中，如果有A日志配置中使用了if [type]判斷，B日志收集配置未指定if[type]，那么，B的索引中，會寫入A中的數據（如果有C配置，設置了if[type]，C的數據也會寫入到B中）目前只判斷了type，其他判斷未測試結論：如果logstash中使用了if[type]判斷，所有配置中都要使用判斷 #### 坑2. 644也無法讀取日志也許有時候需要777，rpm裝的nginx，日志目錄和文件644也沒權限讀 nginx（rpm）日志權限 ```shell Nginx yum安裝，日志位置：/var/log/nginx/access.log /var/log/nginx/權限為700 /var/log/nginx/access.log權限為644 以上情況獲取不到日志把/var/log/nginx權限調整為644，依然獲取不到日志，直到調整為777，獲取日志正常。 ```