# 1.FileBeat ## 1. 1 讀取Java日志 **1. 配置input讀取Java日志** 日志格式不同，通過`multiline`分隔合并確定是一條日志，如下： ``` 2021-12-07 09:14:23.645 INFO 6 --- [io-10006-exec-4] com.yqcx.cp.core.filter.UserCPFilter : traceId=ef82e6f6b33a4e5e8d43ad77eecd91b7,URL:【http://localhost:8080/sysEquType/validateName】,耗時:【199】ms 2021-12-07 09:14:24.875 INFO 6 --- [io-10006-exec-5] com.yqcx.cp.core.filter.UserCPFilter : not allowedPath url [/sysEquType/validateName] 2021-12-07 09:14:24.875 INFO 6 --- [io-10006-exec-5] com.yqcx.cp.core.filter.UserCPFilter : UserCPFilter dealWithUser userId [1400279509741273090] Creating a new SqlSession SqlSession [org.apache.ibatis.session.defaults.DefaultSqlSession@15f227b] was not registered for synchronization because synchronization is not active JDBC Connection [com.alibaba.druid.proxy.jdbc.ConnectionProxyImpl@6f914542] will not be managed by Spring ==> Preparing: SELECT id,`name`,code,pid,company_id AS companyId,disable_flag AS disableFlag,disable_time AS disableTime,create_time AS createTime,create_user AS `createUser`,modified_time AS modifiedTime,modified_user AS modifiedUser FROM sys_equ_type WHERE (name = ? AND company_id = ?) ==> Parameters: eeeeeeeeeee(String), 1368801562103226369(Long) <== Total: 0 Closing non transactional SqlSession [org.apache.ibatis.session.defaults.DefaultSqlSession@15f227b] 2021-12-07 09:14:24.884 ERROR 6 --- [io-10006-exec-5] c.f.a.c.e.OpsGlobalExceptionHandler : 運行時異常: java.lang.NullPointerException: null at java.util.Objects.requireNonNull(Objects.java:203) at com.faw_qm.ad_ops.close_test.controller.SysEquTypeController.validateName(SysEquTypeController.java:109) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ``` 如下通過日期開始區分為一條日志，跨行合并，并簡單輸出到控制臺 ``` filebeat.inputs: - type: log enabled: true paths: - /root/*.log multiline.type: pattern multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}' multiline.negate: true multiline.match: after output: console: codec.json: pretty: true escape_html: true ``` 啟動，先不后臺啟動 ``` /usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml ``` 成功讀取到日志，并且按照日期分隔日志  ## 1.2 輸出到logstash **1.修改output配置** ``` vim /etc/filebeat/filebeat.yml filebeat.inputs: - type: log enabled: true paths: - /root/*.log multiline.type: pattern multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}' multiline.negate: true multiline.match: after output: #console: # codec.json: # pretty: true # escape_html: true logstash: enabled: true hosts: ["192.168.56.10:5044"] ``` **2. 后臺啟動** ``` systemctl start filebeat ``` # 2. logstash ## 2.1 接收filebeat傳入數據 logstash配置filebeat傳入，并控制臺打出測試 ``` input { beats { port => 5044 } } output { stdout { codec => rubydebug } } ``` 同樣不要后臺啟動，使得可以觀察到控制臺打印 ``` /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/filebeat.conf ``` 可以看到控制臺打印  ## 2.2 配置輸出到es ``` vim /etc/logstash/conf.d/filebeat.conf input { beats { port => 5044 } } output { # stdout { # codec => rubydebug # } elasticsearch { hosts => ["http://192.168.56.10:9200"] #user => "elastic" #password => "qEnNfKNujqNrOPD9q5kb" index => "sys-log-%{+YYYY.MM.dd}" } } ```  在es中存儲如下  異常日志（跨行也正常收集了） # 3. logstash過濾數據 ## 3.1 時間和無用字段