Beats 是一个免费且开放的平台，集合了多种单一用途数据采集器。它们从成百上千或成千上万台机器和系统向 Logstash 或 Elasticsearch 发送数据。
虽然利用 logstash 就可以收集日志，功能强大，但由于 Logtash 是基于Java实现，需要在采集日志的主机上安装JAVA环境，会消耗比较多的内存和磁盘空间，logstash运行时最少也会需要额外的500M的以上的内存，资源消耗很大,有些得不偿失。
可以采有基于Go开发的 Beat 工具代替 Logstash 收集日志，部署更为方便，而且只占用10M左右的内存空间及更小的磁盘空间。
官方链接
https://www.elastic.co/cn/beats/
Github 链接
https://github.com/elastic/beats
下载链接
https://www.elastic.co/cn/downloads/beats
Beats 是一些工具集,包括以下,其中 filebeat 应用最为广泛
| Beat | Description |
| Auditbeat | Collect your Linux audit framework data and monitor the integrity of yourfiles. |
| Filebeat | Tails and ships log files |
| Functionbeat | Read and ships events from serverless infrastructure. |
| Heartbeat | Ping remote services for availability |
| Metricbeat | Fetches sets of metrics from the operating system and services |
| Packetbeat | Monitors the network and applications by sniffing packets |
| Winlogbeat | Fetches and ships Windows Event logs |
| Osquerybeat | Runs Osquery and manages interraction with it. |
filebeat:收集日志文件数据。最常用的工具
packetbeat:用于收集网络数据。一般用zabbix实现此功能
metricbeat:从OS和服务收集指标数据，比如系统运行状态、CPU 内存利用率等。
winlogbeat: 从Windows平台日志收集工具。
heartbeat: 定时探测服务是否可用。支持ICMP、TCP 和 HTTP，也支持TLS、身份验证和代理
auditbeat:收集审计日志
Functionbeat:使用无服务器基础架构提供云数据。面向云端数据的无服务器采集器，处理云数据
注意: Beats 版本要和 Elasticsearch 相同的版本，否则可能会出错
利用 Metricbeat 监控性能相关指标
metricbeat 可以收集指标数据，比如系统运行状态、CPU、内存利用率等。
生产中一般用 zabbix等专门的监控系统实现此功能
官方配置说明
https://www.elastic.co/guide/en/beats/metricbeat/current/configuring-howto-metricbeat.html
下载 metricbeat 并安装
下载链接:
https://www.elastic.co/cn/downloads/beats/metricbeat
https://mirrors.tuna.tsinghua.edu.cn/elasticstack/7.x/apt/pool/main/m/metricbeat/
[root@ES-Node2 ~]#wget https://mirrors.tuna.tsinghua.edu.cn/elasticstack/7.x/apt/pool/main/m/metricbeat/metricbeat-7.15.0-amd64.deb
--2022-12-13 11:30:30-- https://mirrors.tuna.tsinghua.edu.cn/elasticstack/7.x/apt/pool/main/m/metricbeat/metricbeat-7.15.0-amd64.deb
正在解析主机 mirrors.tuna.tsinghua.edu.cn (mirrors.tuna.tsinghua.edu.cn)... 101.6.15.130, 2402:f000:1:400::2
正在连接 mirrors.tuna.tsinghua.edu.cn (mirrors.tuna.tsinghua.edu.cn)|101.6.15.130|:443... 已连接。
已发出 HTTP 请求，正在等待回应... 200 OK
长度： 43220906 (41M) [application/octet-stream]
正在保存至: “metricbeat-7.15.0-amd64.deb”
metricbeat-7.15.0-amd64.deb 100%[=========================================================================================================>] 41.22M 390KB/s 用时 64s
2022-12-13 11:31:35 (661 KB/s) - 已保存 “metricbeat-7.15.0-amd64.deb” [43220906/43220906])
[root@ES-Node2 ~]#dpkg -i metricbeat-7.15.0-amd64.deb
正在选中未选择的软件包 metricbeat。
(正在读取数据库 ... 系统当前共安装有 109695 个文件和目录。)
准备解压 metricbeat-7.15.0-amd64.deb ...
正在解压 metricbeat (7.15.0) ...
正在设置 metricbeat (7.15.0) ...
正在安装新版本配置文件 /etc/init.d/metricbeat ...
正在安装新版本配置文件 /etc/metricbeat/fields.yml ...
正在安装新版本配置文件 /etc/metricbeat/metricbeat.reference.yml ...
正在安装新版本配置文件 /etc/metricbeat/metricbeat.yml ...
正在安装新版本配置文件 /etc/metricbeat/modules.d/activemq.yml.disabled ...
正在安装新版本配置文件 /etc/metricbeat/modules.d/aerospike.yml.disabled ...
正在安装新版本配置文件 /etc/metricbeat/modules.d/apache.yml.disabled ...
正在安装新版本配置文件 /etc/metricbeat/modules.d/appsearch.yml.disabled ...
正在安装新版本配置文件 /etc/metricbeat/modules.d/aws.yml.disabled ...
正在安装新版本配置文件 /etc/metricbeat/modules.d/azure.yml.disabled ...
正在安装新版本配置文件 /etc/metricbeat/modules.d/beat-xpack.yml.disabled ...
正在安装新版本配置文件 /etc/metricbeat/modules.d/beat.yml.disabled ...
正在安装新版本配置文件 /etc/metricbeat/modules.d/ceph.yml.disabled ...
正在安装新版本配置文件 /etc/metricbeat/modules.d/cockroachdb.yml.disabled ...
正在安装新版本配置文件 /etc/metricbeat/modules.d/consul.yml.disabled ...
正在安装新版本配置文件 /etc/metricbeat/modules.d/coredns.yml.disabled ...
正在安装新版本配置文件 /etc/metricbeat/modules.d/couchbase.yml.disabled ...
正在安装新版本配置文件 /etc/metricbeat/modules.d/couchdb.yml.disabled ...
正在安装新版本配置文件 /etc/metricbeat/modules.d/docker.yml.disabled ...
正在安装新版本配置文件 /etc/metricbeat/modules.d/dropwizard.yml.disabled ...
正在安装新版本配置文件 /etc/metricbeat/modules.d/elasticsearch-xpack.yml.disabled ...
正在安装新版本配置文件 /etc/metricbeat/modules.d/elasticsearch.yml.disabled ...
正在安装新版本配置文件 /etc/metricbeat/modules.d/envoyproxy.yml.disabled ...
正在安装新版本配置文件 /etc/metricbeat/modules.d/etcd.yml.disabled ...
正在安装新版本配置文件 /etc/metricbeat/modules.d/golang.yml.disabled ...
正在安装新版本配置文件 /etc/metricbeat/modules.d/graphite.yml.disabled ...
正在安装新版本配置文件 /etc/metricbeat/modules.d/haproxy.yml.disabled ...
正在安装新版本配置文件 /etc/metricbeat/modules.d/http.yml.disabled ...
正在安装新版本配置文件 /etc/metricbeat/modules.d/jolokia.yml.disabled ...
正在安装新版本配置文件 /etc/metricbeat/modules.d/kafka.yml.disabled ...
正在安装新版本配置文件 /etc/metricbeat/modules.d/kibana-xpack.yml.disabled ...
正在安装新版本配置文件 /etc/metricbeat/modules.d/kibana.yml.disabled ...
正在安装新版本配置文件 /etc/metricbeat/modules.d/kubernetes.yml.disabled ...
正在安装新版本配置文件 /etc/metricbeat/modules.d/kvm.yml.disabled ...
正在安装新版本配置文件 /etc/metricbeat/modules.d/logstash-xpack.yml.disabled ...
正在安装新版本配置文件 /etc/metricbeat/modules.d/logstash.yml.disabled ...
正在安装新版本配置文件 /etc/metricbeat/modules.d/memcached.yml.disabled ...
正在安装新版本配置文件 /etc/metricbeat/modules.d/mongodb.yml.disabled ...
正在安装新版本配置文件 /etc/metricbeat/modules.d/mssql.yml.disabled ...
正在安装新版本配置文件 /etc/metricbeat/modules.d/munin.yml.disabled ...
正在安装新版本配置文件 /etc/metricbeat/modules.d/mysql.yml.disabled ...
正在安装新版本配置文件 /etc/metricbeat/modules.d/nats.yml.disabled ...
正在安装新版本配置文件 /etc/metricbeat/modules.d/nginx.yml.disabled ...
正在安装新版本配置文件 /etc/metricbeat/modules.d/oracle.yml.disabled ...
正在安装新版本配置文件 /etc/metricbeat/modules.d/php_fpm.yml.disabled ...
正在安装新版本配置文件 /etc/metricbeat/modules.d/postgresql.yml.disabled ...
正在安装新版本配置文件 /etc/metricbeat/modules.d/prometheus.yml.disabled ...
正在安装新版本配置文件 /etc/metricbeat/modules.d/rabbitmq.yml.disabled ...
正在安装新版本配置文件 /etc/metricbeat/modules.d/redis.yml.disabled ...
正在安装新版本配置文件 /etc/metricbeat/modules.d/sql.yml.disabled ...
正在安装新版本配置文件 /etc/metricbeat/modules.d/stan.yml.disabled ...
正在安装新版本配置文件 /etc/metricbeat/modules.d/statsd.yml.disabled ...
正在安装新版本配置文件 /etc/metricbeat/modules.d/system.yml ...
正在安装新版本配置文件 /etc/metricbeat/modules.d/tomcat.yml.disabled ...
正在安装新版本配置文件 /etc/metricbeat/modules.d/traefik.yml.disabled ...
正在安装新版本配置文件 /etc/metricbeat/modules.d/uwsgi.yml.disabled ...
正在安装新版本配置文件 /etc/metricbeat/modules.d/vsphere.yml.disabled ...
正在安装新版本配置文件 /etc/metricbeat/modules.d/windows.yml.disabled ...
正在安装新版本配置文件 /etc/metricbeat/modules.d/zookeeper.yml.disabled ...
正在处理用于 systemd (245.4-4ubuntu3.15) 的触发器 ...
[root@ES-Node2 ~]#systemctl start metricbeat.service
[root@ES-Node2 ~]#systemctl status metricbeat.service
● metricbeat.service - Metricbeat is a lightweight shipper for metrics.
Loaded: loaded (/lib/systemd/system/metricbeat.service; disabled; vendor preset: enabled)
Active: active (running) since Tue 2022-12-13 11:32:18 CST; 4s ago
Docs: https://www.elastic.co/beats/metricbeat
Main PID: 61618 (metricbeat)
Tasks: 8 (limit: 4575)
Memory: 71.2M
CGroup: /system.slice/metricbeat.service
└─61618 /usr/share/metricbeat/bin/metricbeat --environment systemd -c /etc/metricbeat/metricbeat.yml --path.home /usr/share/metricbeat --path.config /etc/metricbeat --path.data /var>
12月 13 11:32:19 ES-Node2.com metricbeat[61618]: 2022-12-13T11:32:19.642+0800 INFO [index-management.ilm] ilm/std.go:170 ILM policy metricbeat exists already.
12月 13 11:32:19 ES-Node2.com metricbeat[61618]: 2022-12-13T11:32:19.642+0800 INFO [index-management] idxmgmt/std.go:401 Set setup.template.name to '{metricbeat-7.15.>
12月 13 11:32:19 ES-Node2.com metricbeat[61618]: 2022-12-13T11:32:19.642+0800 INFO [index-management] idxmgmt/std.go:406 Set setup.template.pattern to 'metricbeat-7.1>
12月 13 11:32:19 ES-Node2.com metricbeat[61618]: 2022-12-13T11:32:19.642+0800 INFO [index-management] idxmgmt/std.go:440 Set settings.index.lifecycle.rollover_alias i>
12月 13 11:32:19 ES-Node2.com metricbeat[61618]: 2022-12-13T11:32:19.642+0800 INFO [index-management] idxmgmt/std.go:444 Set settings.index.lifecycle.name in template>
12月 13 11:32:20 ES-Node2.com metricbeat[61618]: 2022-12-13T11:32:20.183+0800 INFO template/load.go:132 Try loading template metricbeat-7.15.0 to Elasticsearch
12月 13 11:32:20 ES-Node2.com metricbeat[61618]: 2022-12-13T11:32:20.478+0800 INFO template/load.go:124 Template with name "metricbeat-7.15.0" loaded.
12月 13 11:32:20 ES-Node2.com metricbeat[61618]: 2022-12-13T11:32:20.478+0800 INFO [index-management] idxmgmt/std.go:297 Loaded index template.
12月 13 11:32:20 ES-Node2.com metricbeat[61618]: 2022-12-13T11:32:20.842+0800 INFO [index-management.ilm] ilm/std.go:140 Index Alias metricbeat-7.15.0 successfully cr>
12月 13 11:32:20 ES-Node2.com metricbeat[61618]: 2022-12-13T11:32:20.843+0800 INFO [publisher_pipeline_output] pipeline/output.go:151 Connection to backoff(elasticsea>
修改配置
vim /etc/metricbeat/metricbeat.yml
setup.kibana:
host: "10.0.0.208:5601" #指向kabana服务器地址和端口，非必须项，当使用dashboard时才需
要指定
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["10.0.0.209:9200"] #指向任意一个ELK集群节点即可
[root@elk-web2 ~]#grep -Ev "#|^$" /etc/metricbeat/metricbeat.yml
metricbeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
setup.template.settings:
index.number_of_shards: 1
index.codec: best_compression
setup.kibana:
host: "10.0.0.208:5601"
output.elasticsearch:
hosts: ["10.0.0.209:9200"]
processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~
通过 Kibana 查看收集的性能指标
利用 Heartbeat 监控
heartbeat 用来定时探测服务是否正常运行。支持ICMP、TCP 和 HTTP，也支持TLS、身份验证和代理
官方heartbeat配置文档:https://www.elastic.co/guide/en/beats/heartbeat/current/configuring-howto-heartbeat.html
下载并安装
下载链接
https://www.elastic.co/cn/downloads/beats/heartbeat
https://mirrors.tuna.tsinghua.edu.cn/elasticstack/7.x/apt/pool/main/f/filebeat/
[root@ES-Node2 ~]#wget https://mirrors.tuna.tsinghua.edu.cn/elasticstack/7.x/apt/pool/main/h/heartbeat-elastic/heartbeat-7.15.0-amd64.deb
--2022-12-13 11:50:44-- https://mirrors.tuna.tsinghua.edu.cn/elasticstack/7.x/apt/pool/main/h/heartbeat-elastic/heartbeat-7.15.0-amd64.deb
正在解析主机 mirrors.tuna.tsinghua.edu.cn (mirrors.tuna.tsinghua.edu.cn)... 101.6.15.130, 2402:f000:1:400::2
正在连接 mirrors.tuna.tsinghua.edu.cn (mirrors.tuna.tsinghua.edu.cn)|101.6.15.130|:443... 已连接。
已发出 HTTP 请求，正在等待回应... 200 OK
长度： 27757904 (26M) [application/octet-stream]
正在保存至: “heartbeat-7.15.0-amd64.deb”
heartbeat-7.15.0-amd64.deb 100%[=========================================================================================================>] 26.47M 3.64MB/s 用时 8.0s
2022-12-13 11:50:53 (3.31 MB/s) - 已保存 “heartbeat-7.15.0-amd64.deb” [27757904/27757904])
[root@ES-Node2 ~]#dpkg -i heartbeat-7.15.0-amd64.deb
正在选中未选择的软件包 heartbeat-elastic。
(正在读取数据库 ... 系统当前共安装有 110797 个文件和目录。)
准备解压 heartbeat-7.15.0-amd64.deb ...
正在解压 heartbeat-elastic (7.15.0) ...
正在设置 heartbeat-elastic (7.15.0) ...
正在处理用于 systemd (245.4-4ubuntu3.15) 的触发器 ...
#准备需要监控的服务httpd
[root@ES-Node2 ~]#apt -y install apache2
修改配置
官方参考:https://www.elastic.co/guide/en/beats/heartbeat/current/configuration-heartbeat-options.html
# heartbeat.yml
heartbeat.monitors:
- type: icmp
id: ping-myhost
name: My Host Ping
hosts: ["myhost"]
schedule: '*/5 * * * * * *'
- type: tcp
id: myhost-tcp-echo
name: My Host TCP Echo
hosts: ["myhost:777"] # default TCP Echo Protocol
check.send: "Check"
check.receive: "Check"
schedule: '@every 5s'
- type: http
id: service-status
name: Service Status
service.name: my-apm-service-name
hosts: ["http://localhost:80/service/status"]
check.response.status: [200]
schedule: '@every 5s'
heartbeat.scheduler:
limit: 10时间格式
https://github.com/gorhill/cronexpr#implementation
Field name Mandatory? Allowed values Allowed special characters
---------- ---------- -------------- --------------------------
Seconds No 0-59 * / , -
Minutes Yes 0-59 * / , -
Hours Yes 0-23 * / , -
Day of month Yes 1-31 * / , - L W
Month Yes 1-12 or JAN-DEC * / , -
Day of week Yes 0-6 or SUN-SAT * / , - L #
Year No 1970–2099 * / , -[root@ES-Node2 ~]#cat /etc/heartbeat/heartbeat.yml
# Configure monitors inline
heartbeat.monitors:
- type: http
# List or urls to query
urls: ["http://localhost:80"] #指向需要监控的服务器的地址和端口
# Configure task schedule
schedule: '@every 10s'
# Total test connection and data exchange timeout
timeout: 6s
- type: icmp #监控ICMP
id: ping-myhost
name: My Host Ping
hosts: ["10.0.0.209"]
schedule: '*/5 * * * * * *'
setup.kibana:
# Kibana Host
# Scheme and port can be left out and will be set to the default (http and5601)
# In case you specify and additional path, the scheme is required:http://localhost:5601/path
# IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
host: "10.0.0.208:5601" #指向kibana服务器地址和端口
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["10.0.0.208:9200"] #指向ELK集群服务器地址和端口
通过 Kibana 查看收集的性能指标
利用 Filebeat 收集日志
Filebeat 是用于转发和集中日志数据的轻量级传送程序。作为服务器上的代理安装，Filebeat监视您指定的日志文件或位置，收集日志事件，并将它们转发到Elasticsearch或Logstash进行索引。
Logstash 直接收集日志,需要安装JDK,并且会占用至少500M 以上的内存
生产一般使用filebeat代替logstash, 基于go开发,部署方便,重要的是只需要10M多内存,比较节约资源.
filebeat 支持从日志,Syslog,Redis,Docker,TCP,UDP,标准输入等读取数据,再输入至Elasticsearch,logstash,Redis,Kafka等
Filebeat的工作方式如下：
启动Filebeat时，它将启动一个或多个输入，这些输入将在为日志数据指定的位置中查找。对于Filebeat所找到的每个日志，Filebeat都会启动收集器。每个收集器都读取一个日志以获取新内容，并将新日志数据发送到libbeat，libbeat会汇总事件并将汇总的数据发送到为Filebeat配置的输出。
Filebeat 官方说明
https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-overview.html
https://www.elastic.co/guide/en/beats/filebeat/current/configuring-howto-filebeat.html
输入和输入官方说明:
https://www.elastic.co/guide/en/beats/filebeat/current/configuration-filebeat-options.html
https://www.elastic.co/guide/en/beats/filebeat/current/configuring-output.html
注意: Filebeat 支持多个输入,但不支持同时有多个输出
安装 Filebeat
[root@ES-Node2 ~]#wget https://mirrors.tuna.tsinghua.edu.cn/elasticstack/7.x/apt/pool/main/f/filebeat/filebeat-7.15.0-amd64.deb
--2022-12-13 18:26:11-- https://mirrors.tuna.tsinghua.edu.cn/elasticstack/7.x/apt/pool/main/f/filebeat/filebeat-7.15.0-amd64.deb
正在解析主机 mirrors.tuna.tsinghua.edu.cn (mirrors.tuna.tsinghua.edu.cn)... 101.6.15.130, 2402:f000:1:400::2
正在连接 mirrors.tuna.tsinghua.edu.cn (mirrors.tuna.tsinghua.edu.cn)|101.6.15.130|:443... 已连接。
已发出 HTTP 请求，正在等待回应... 200 OK
长度： 35823896 (34M) [application/octet-stream]
正在保存至: “filebeat-7.15.0-amd64.deb”
filebeat-7.15.0-amd64.deb 100%[=========================================================================================================>] 34.16M 2.55MB/s 用时 16s
2022-12-13 18:26:29 (2.18 MB/s) - 已保存 “filebeat-7.15.0-amd64.deb” [35823896/35823896])
[root@ES-Node2 ~]#dpkg -i filebeat-7.15.0-amd64.deb
正在选中未选择的软件包 filebeat。
(正在读取数据库 ... 系统当前共安装有 111544 个文件和目录。)
准备解压 filebeat-7.15.0-amd64.deb ...
正在解压 filebeat (7.15.0) ...
正在设置 filebeat (7.15.0) ...
正在处理用于 systemd (245.4-4ubuntu3.15) 的触发器 ...
官方说明:https://www.elastic.co/guide/en/beats/filebeat/8.3/configuration-general-options.html
添加新字段的配置说明
#添加新字段
vim /etc/filebeat/filebeat.yml
- type : log
enabled: true
paths:
- /var/log/syslog
fields:
project: test-syslog #添加fields.project和fields.env字段,可用于区分不同的日志
env: test
tags: ["syslog","test"] #添加标签,也可以用于区分不同的日志案例: 从标准输入读取再输出至标准输出
创建配置
[root@ES-Node2 ~]#cat /etc/filebeat/stdin.yml
filebeat.inputs:
- type: stdin
enabled: true
output.console:
pretty: true
enable: true执行读取
[root@ES-Node2 ~]#filebeat -e -c /etc/filebeat/stdin.yml
2022-12-13T18:35:36.508+0800 INFO instance/beat.go:665 Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2022-12-13T18:35:36.508+0800 INFO instance/beat.go:673 Beat ID: f87dddf0-89bf-46c9-be70-0f8e1784eb2a
2022-12-13T18:35:36.508+0800 INFO [seccomp] seccomp/seccomp.go:124 Syscall filter successfully installed
2022-12-13T18:35:36.508+0800 INFO [beat] instance/beat.go:1014 Beat info {"system_info": {"beat": {"path": {"config": "/etc/filebeat", "data": "/var/lib/filebeat", "home": "/usr/share/filebeat", "logs": "/var/log/filebeat"}, "type": "filebeat", "uuid": "f87dddf0-89bf-46c9-be70-0f8e1784eb2a"}}}
2022-12-13T18:35:36.508+0800 INFO [beat] instance/beat.go:1023 Build info {"system_info": {"build": {"commit": "9023152025ec6251bc6b6c38009b309157f10f17", "libbeat": "7.15.0", "time": "2021-09-16T03:16:09.000Z", "version": "7.15.0"}}}
2022-12-13T18:35:36.508+0800 INFO [beat] instance/beat.go:1026 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.16.6"}}}
2022-12-13T18:35:36.509+0800 INFO [beat] instance/beat.go:1030 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2022-12-13T06:27:05+08:00","containerized":false,"name":"ES-Node2.com","ip":["127.0.0.1/8","::1/128","10.0.0.209/24","fe80::20c:29ff:fefc:3b0c/64"],"kernel_version":"5.4.0-124-generic","mac":["00:0c:29:fc:3b:0c"],"os":{"type":"linux","family":"debian","platform":"ubuntu","name":"Ubuntu","version":"20.04.4 LTS (Focal Fossa)","major":20,"minor":4,"patch":4,"codename":"focal"},"timezone":"CST","timezone_offset_sec":28800,"id":"c0ae66620c874a90b249e470c0e9a204"}}}
2022-12-13T18:35:36.509+0800 INFO [beat] instance/beat.go:1059 Process info {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"ambient":null}, "cwd": "/root", "exe": "/usr/share/filebeat/bin/filebeat", "name": "filebeat", "pid": 75586, "ppid": 8270, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2022-12-13T18:35:35.540+0800"}}}
2022-12-13T18:35:36.509+0800 INFO instance/beat.go:309 Setup Beat: filebeat; Version: 7.15.0
2022-12-13T18:35:36.509+0800 INFO [publisher] pipeline/module.go:113 Beat name: ES-Node2.com
2022-12-13T18:35:36.510+0800 WARN beater/filebeat.go:178 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2022-12-13T18:35:36.510+0800 INFO [monitoring] log/log.go:142 Starting metrics logging every 30s
2022-12-13T18:35:36.510+0800 INFO instance/beat.go:473 filebeat start running.
2022-12-13T18:35:36.511+0800 INFO memlog/store.go:119 Loading data file of '/var/lib/filebeat/registry/filebeat' succeeded. Active transaction id=0
2022-12-13T18:35:36.512+0800 INFO memlog/store.go:124 Finished loading transaction log file for '/var/lib/filebeat/registry/filebeat'. Active transaction id=4
2022-12-13T18:35:36.512+0800 WARN beater/filebeat.go:381 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2022-12-13T18:35:36.513+0800 INFO [registrar] registrar/registrar.go:109 States Loaded from registrar: 1
2022-12-13T18:35:36.514+0800 INFO [crawler] beater/crawler.go:71 Loading Inputs: 1
2022-12-13T18:35:36.515+0800 INFO [crawler] beater/crawler.go:141 Starting input (ID: 11136643476161899408)
2022-12-13T18:35:36.515+0800 INFO [crawler] beater/crawler.go:108 Loading and starting Inputs completed. Enabled inputs: 1
2022-12-13T18:35:36.516+0800 INFO [stdin.harvester] log/harvester.go:309 Harvester started for file. {"harvester_id": "b2062d60-52a2-44a7-9ee7-457427cc67d1"}
hello,es #此处输入hello,es回车后输出下面信息
{
"@timestamp": "2022-12-13T10:35:41.440Z",
"@metadata": {
"beat": "filebeat",
"type": "_doc",
"version": "7.15.0"
},
"log": {
"file": {
"path": ""
},
"offset": 0
},
"message": "hello,es",
"input": {
"type": "stdin"
},
"agent": {
"name": "ES-Node2.com",
"type": "filebeat",
"version": "7.15.0",
"hostname": "ES-Node2.com",
"ephemeral_id": "19305c13-cd46-43b7-887c-262a9315a8b0",
"id": "f87dddf0-89bf-46c9-be70-0f8e1784eb2a"
},
"ecs": {
"version": "1.11.0"
},
"host": {
"name": "ES-Node2.com"
}
}
案例: 从标准输入读取再输出至 Json 格式的文件
创建配置
[root@ES-Node2 ~]#cat /etc/filebeat/stdout_file.yml
filebeat.inputs:
- type: stdin
enabled: true
json.keys_under_root: true #默认False会将json数据存储至message，改为true则会独立message外存储
output.file:
path: "/tmp"
filename: "filebeat.log"执行读取
[root@ES-Node2 ~]#filebeat -e -c /etc/filebeat/stdout_file.yml
...
#输入,回车后,输出如下
{"name" : "wangxiaochun", "age" : "18", "phone" : "0123456789" }
[root@ES-Node2 ~]#cat /tmp/filebeat.log
{"@timestamp":"2022-12-13T10:38:38.394Z","@metadata":{"beat":"filebeat","type":"_doc","version":"7.15.0"},"host":{"name":"ES-Node2.com"},"agent":{"hostname":"ES-Node2.com","ephemeral_id":"a2253f9f-fef4-468c-ba11-32f601695f45","id":"f87dddf0-89bf-46c9-be70-0f8e1784eb2a","name":"ES-Node2.com","type":"filebeat","version":"7.15.0"},"log":{"offset":0,"file":{"path":""}},"json":{},"message":"....","input":{"type":"stdin"},"ecs":{"version":"1.11.0"}}
{"@timestamp":"2022-12-13T10:38:43.544Z","@metadata":{"beat":"filebeat","type":"_doc","version":"7.15.0"},"message":"","input":{"type":"stdin"},"ecs":{"version":"1.11.0"},"host":{"name":"ES-Node2.com"},"agent":{"version":"7.15.0","hostname":"ES-Node2.com","ephemeral_id":"a2253f9f-fef4-468c-ba11-32f601695f45","id":"f87dddf0-89bf-46c9-be70-0f8e1784eb2a","name":"ES-Node2.com","type":"filebeat"},"log":{"offset":0,"file":{"path":""}},"json":{}}
{"@timestamp":"2022-12-13T10:38:44.333Z","@metadata":{"beat":"filebeat","type":"_doc","version":"7.15.0"},"log":{"offset":0,"file":{"path":""}},"json":{},"message":"","input":{"type":"stdin"},"ecs":{"version":"1.11.0"},"host":{"name":"ES-Node2.com"},"agent":{"ephemeral_id":"a2253f9f-fef4-468c-ba11-32f601695f45","id":"f87dddf0-89bf-46c9-be70-0f8e1784eb2a","name":"ES-Node2.com","type":"filebeat","version":"7.15.0","hostname":"ES-Node2.com"}}
{"@timestamp":"2022-12-13T10:38:53.029Z","@metadata":{"beat":"filebeat","type":"_doc","version":"7.15.0"},"input":{"type":"stdin"},"host":{"name":"ES-Node2.com"},"agent":{"version":"7.15.0","hostname":"ES-Node2.com","ephemeral_id":"a2253f9f-fef4-468c-ba11-32f601695f45","id":"f87dddf0-89bf-46c9-be70-0f8e1784eb2a","name":"ES-Node2.com","type":"filebeat"},"ecs":{"version":"1.11.0"},"age":"18","phone":"0123456789","log":{"offset":0,"file":{"path":""}},"name":"wangxiaochun"}
案例: 从文件读取再输出至标准输出
https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-log.html
filebeat 会将每个文件的读取数据的相关信息记录在/var/lib/filebeat/registry/filebeat/log.json文件中,可以实现日志采集的持续性,而不会重复采集
创建配置
[root@ES-Node2 ~]#cat /etc/filebeat/file.yml
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/syslog
output.console:
pretty: true
enable: true执行读取
[root@ES-Node2 ~]#filebeat -e -c /etc/filebeat/file.yml
.....案例: 利用 Filebeat 收集系统日志到 ELasticsearch
默认生成的索引名称为 filebeat-<版本>-<时间>*
[root@ES-Node2 ~]#vim /etc/filebeat/filebeat.yml
...
- type: filestream
# Change to true to enable this input configuration.
enabled: true
# Paths that should be crawled and fetched. Glob based paths.
paths:
- /var/log/filebeat.log
...
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["10.0.0.209:9200"]
...
[root@ES-Node2 ~]#systemctl restart filebeat.service 
案例: 自定义索引名称收集所有系统日志到 ELasticsearch
修改配置
[root@ES-Node2 ~]#vim /etc/filebeat/filebeat.yml
...
hosts: ["10.0.0.209:9200"]
index: "shu-%{[agent.version]}-%{+yyyy.MM.dd}" #自定义索引名称
setup.ilm.enabled: false #关闭索引生命周期ilm功能，默认开启时索引名称只能为filebeat-*
setup.template.name: "wang" #定义模板名称,要自定义索引名称,必须指定此项,否则无法启动
setup.template.pattern: "wang-*" #定义模板的匹配索引名称,要自定义索引名称,必须指定此项,否则无法启动
[root@ES-Node2 ~]#systemctl restart filebeat.service
[root@ES-Node2 ~]#systemctl status filebeat.service
案例: 利用 Filebeat 收集 Nginx Json 格式日志到Elasticsearch
官方文档
https://www.elastic.co/guide/en/beats/filebeat/7.6/filebeat-input-log.html
https://www.elastic.co/guide/en/beats/filebeat/7.6/redis-output.html
生产环境中我们经常需要获取Web访问用户的信息，比如:来源的IP是哪个地域，网站的PV、UV、状态码、访问时间等等;所以需要收集的Nginx访问日志
安装nginx以json格式输出日志
[root@ES-Node2 ~]#cat /etc/nginx/nginx.conf
....
log_format access_json '{"@timestamp":"$time_iso8601",'
'"host":"$server_addr",'
'"clientip":"$remote_addr",'
'"size":$body_bytes_sent,'
'"responsetime":$request_time,'
'"upstreamtime":"$upstream_response_time",'
'"upstreamhost":"$upstream_addr",'
'"http_host":"$host",'
'"uri":"$uri",'
'"domain":"$host",'
'"xff":"$http_x_forwarded_for",'
'"referer":"$http_referer",'
'"tcp_xff":"$proxy_protocol_addr",'
'"http_user_agent":"$http_user_agent",'
'"status":"$status"}';
access_log /var/log/nginx/access_json.log access_json;
....
#默认开启nginx的错误日志,但如果是ubuntu,还需要修改下面行才能记录错误日志
[root@ES-Node2 ~]#cat /etc/nginx/sites-enabled/default
....
#try_files $uri $uri/ =404;
....
#访问并生成日志
[root@ES-Node2 ~]#cat /var/log/nginx/access_json.log
{"@timestamp":"2022-12-15T16:19:21+08:00","host":"10.0.0.209","clientip":"10.0.0.210","size":10918,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"10.0.0.209","uri":"/index.html","domain":"10.0.0.209","xff":"-","referer":"-","tcp_xff":"-","http_user_agent":"curl/7.68.0","status":"200"}
{"@timestamp":"2022-12-15T16:19:23+08:00","host":"10.0.0.209","clientip":"10.0.0.210","size":10918,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"10.0.0.209","uri":"/index.html","domain":"10.0.0.209","xff":"-","referer":"-","tcp_xff":"-","http_user_agent":"curl/7.68.0","status":"200"}
{"@timestamp":"2022-12-15T16:19:24+08:00","host":"10.0.0.209","clientip":"10.0.0.210","size":10918,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"10.0.0.209","uri":"/index.html","domain":"10.0.0.209","xff":"-","referer":"-","tcp_xff":"-","http_user_agent":"curl/7.68.0","status":"200"}
{"@timestamp":"2022-12-15T16:19:24+08:00","host":"10.0.0.209","clientip":"10.0.0.210","size":10918,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"10.0.0.209","uri":"/index.html","domain":"10.0.0.209","xff":"-","referer":"-","tcp_xff":"-","http_user_agent":"curl/7.68.0","status":"200"}
{"@timestamp":"2022-12-15T16:19:25+08:00","host":"10.0.0.209","clientip":"10.0.0.210","size":10918,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"10.0.0.209","uri":"/index.html","domain":"10.0.0.209","xff":"-","referer":"-","tcp_xff":"-","http_user_agent":"curl/7.68.0","status":"200"}
{"@timestamp":"2022-12-15T16:19:35+08:00","host":"10.0.0.209","clientip":"10.0.0.210","size":162,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"10.0.0.209","uri":"/indexsss.html","domain":"10.0.0.209","xff":"-","referer":"-","tcp_xff":"-","http_user_agent":"curl/7.68.0","status":"404"}
{"@timestamp":"2022-12-15T16:19:37+08:00","host":"10.0.0.209","clientip":"10.0.0.210","size":162,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"10.0.0.209","uri":"/indexsss.html","domain":"10.0.0.209","xff":"-","referer":"-","tcp_xff":"-","http_user_agent":"curl/7.68.0","status":"404"}
{"@timestamp":"2022-12-15T16:19:37+08:00","host":"10.0.0.209","clientip":"10.0.0.210","size":162,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"10.0.0.209","uri":"/indexsss.html","domain":"10.0.0.209","xff":"-","referer":"-","tcp_xff":"-","http_user_agent":"curl/7.68.0","status":"404"}
{"@timestamp":"2022-12-15T16:19:37+08:00","host":"10.0.0.209","clientip":"10.0.0.210","size":162,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"10.0.0.209","uri":"/indexsss.html","domain":"10.0.0.209","xff":"-","referer":"-","tcp_xff":"-","http_user_agent":"curl/7.68.0","status":"404"}
[root@ES-Node2 ~]#tail -f /var/log/nginx/error.log
2022/12/15 16:24:27 [error] 5149#5149: *11 open() "/var/www/html/indexsss.html" failed (2: No such file or directory), client: 10.0.0.210, server: _, request: "GET /indexsss.html HTTP/1.1", host: "10.0.0.209"
2022/12/15 16:25:04 [error] 5149#5149: *12 open() "/var/www/html/indexsss.html" failed (2: No such file or directory), client: 10.0.0.210, server: _, request: "GET /indexsss.html HTTP/1.1", host: "10.0.0.209"
2022/12/15 16:25:48 [notice] 5337#5337: signal process started
2022/12/15 16:25:51 [error] 5339#5339: *13 open() "/var/www/html/indexsss.html" failed (2: No such file or directory), client: 10.0.0.210, server: _, request: "GET /indexsss.html HTTP/1.1", host: "10.0.0.209"
2022/12/15 16:25:56 [error] 5339#5339: *14 open() "/var/www/html/indexsss.html" failed (2: No such file or directory), client: 10.0.0.210, server: _, request: "GET /indexsss.html HTTP/1.1", host: "10.0.0.209"
2022/12/15 16:25:58 [error] 5339#5339: *15 open() "/var/www/html/indexsss.html" failed (2: No such file or directory), client: 10.0.0.210, server: _, request: "GET /indexsss.html HTTP/1.1", host: "10.0.0.209"
2022/12/15 16:25:59 [error] 5339#5339: *16 open() "/var/www/html/indexsss.html" failed (2: No such file or directory), client: 10.0.0.210, server: _, request: "GET /indexsss.html HTTP/1.1", host: "10.0.0.209"
2022/12/15 16:25:59 [error] 5339#5339: *17 open() "/var/www/html/indexsss.html" failed (2: No such file or directory), client: 10.0.0.210, server: _, request: "GET /indexsss.html HTTP/1.1", host: "10.0.0.209"
2022/12/15 16:26:00 [error] 5339#5339: *18 open() "/var/www/html/indexsss.html" failed (2: No such file or directory), client: 10.0.0.210, server: _, request: "GET /indexsss.html HTTP/1.1", host: "10.0.0.209"
2022/12/15 16:26:39 [notice] 5368#5368: signal process started
修改 Filebeat 配置文件
[root@ES-Node2 filebeat]#cat /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/nginx/access_json.log
json.keys_under_root: true #默认false会将全部数据存储至message字段，改为true则会以Json格式存储
json.overwrite_keys: true #设为true,覆盖默认的message字段，使用自定义json格式中的key
tags: ["nginx-access"] #指定tag,用于分类
- type: log
enabled: true
paths:
- /var/log/nginx/error.log
tags: ["nginx-error"]
output.elasticsearch:
hosts: ["10.0.0.208:9200","10.0.0.209:9200","10.0.0.210:9200"]
indices:
- index: "nginx-access-%{[agent.version]}-%{+yyy.MM.dd}"
when.contains:
tags: "nginx-access" #如果记志中有access的tag,就记录到nginx-access的索引中
- index: "nginx-error-%{[agent.version]}-%{+yyy.MM.dd}"
when.contains:
tags: "nginx-error" #如果记志中有error的tag,就记录到nginx-error的索引中
setup.ilm.enabled: false #关闭索引生命周期ilm功能，默认开启时索引名称只能为filebeat-*
setup.template.name: "nginx" #定义模板名称,要自定义索引名称,必须指定此项,否则无法启动
setup.template.pattern: "nginx-*" #定义模板的匹配索引名称,要自定义索引名称,必须指定此项,否则无法启动
[root@ES-Node2 filebeat]#systemctl restart filebeat.service 
案例: 利用 Filebeat 收集 Tomat 的 Json 格式的访问日志到Elasticsearch
安装tomcat
[root@ES-Node2 filebeat]#apt -y install tomcat9-admin tomcat9
[root@ES-Node2 ~]#vim /etc/tomcat9/server.xml
....
<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
prefix="localhost_access_log" suffix=".txt"
pattern="{"clientip":"%h","ClientUser":"%l","authenticated
":"%u","AccessTime":"%t","method":"%r","status":&
quot;%s","SendBytes":"%b","Query?string":"%q","partner":&qu
ot;%{Referer}i","AgentVersion":"%{User-Agent}i"}"/>
....
[root@ES-Node2 ~]#systemctl restart tomcat9.service
#访问后查看日志
[root@ES-Node2 ~]#tail -f /var/log/tomcat9/localhost_access_log.2022-12-15.txt
{"clientip":"10.0.0.210","ClientUser":"-","authenticated":"-","AccessTime":"[15/Dec/2022:16:55:31 +0800]","method":"GET / HTTP/1.1","status":"200","SendBytes":"1895","Query?string":"","partner":"-","AgentVersion":"curl/7.68.0"}
{"clientip":"10.0.0.210","ClientUser":"-","authenticated":"-","AccessTime":"[15/Dec/2022:16:57:27 +0800]","method":"GET / HTTP/1.1","status":"200","SendBytes":"1895","Query?string":"","partner":"-","AgentVersion":"curl/7.68.0"}
{"clientip":"10.0.0.210","ClientUser":"-","authenticated":"-","AccessTime":"[15/Dec/2022:16:57:27 +0800]","method":"GET / HTTP/1.1","status":"200","SendBytes":"1895","Query?string":"","partner":"-","AgentVersion":"curl/7.68.0"}
{"clientip":"10.0.0.210","ClientUser":"-","authenticated":"-","AccessTime":"[15/Dec/2022:16:57:28 +0800]","method":"GET / HTTP/1.1","status":"200","SendBytes":"1895","Query?string":"","partner":"-","AgentVersion":"curl/7.68.0"}
{"clientip":"10.0.0.210","ClientUser":"-","authenticated":"-","AccessTime":"[15/Dec/2022:16:57:38 +0800]","method":"GET /aaa HTTP/1.1","status":"404","SendBytes":"721","Query?string":"","partner":"-","AgentVersion":"curl/7.68.0"}
{"clientip":"10.0.0.210","ClientUser":"-","authenticated":"-","AccessTime":"[15/Dec/2022:16:57:40 +0800]","method":"GET /aaa HTTP/1.1","status":"404","SendBytes":"721","Query?string":"","partner":"-","AgentVersion":"curl/7.68.0"}
{"clientip":"10.0.0.210","ClientUser":"-","authenticated":"-","AccessTime":"[15/Dec/2022:16:57:42 +0800]","method":"GET /aaa HTTP/1.1","status":"404","SendBytes":"721","Query?string":"","partner":"-","AgentVersion":"curl/7.68.0"}
{"clientip":"10.0.0.210","ClientUser":"-","authenticated":"-","AccessTime":"[15/Dec/2022:16:57:46 +0800]","method":"GET / HTTP/1.1","status":"200","SendBytes":"1895","Query?string":"","partner":"-","AgentVersion":"curl/7.68.0"}修改 Filebeat 配置文件
[root@ES-Node2 ~]#vim /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/tomcat9/localhost_access_log.*
json.keys_under_root: true #默认False会将json数据存储至message，改为true则会独立message外存储
json.overwrite_keys: false #设为true,覆盖默认的message字段，使用自定义json格式中的key
tags: ["tomcat-access"]
- type: log
enabled: true
paths:
- /usr/local/tomcat/logs/catalina.*
tags: ["tomcat-error"]
output.elasticsearch:
hosts: ["10.0.0.208:9200"] #指定ELK集群服务器地址和端口
indices:
- index: "tomcat-access-%{[agent.version]}-%{+yyy.MM.dd}"
when.contains:
tags: "tomcat-access"
- index: "tomcat-error-%{[agent.version]}-%{+yyy.MM.dd}"
when.contains:
tags: "tomcat-error"
setup.ilm.enabled: false
setup.template.name: "tomcat"
setup.template.pattern: "tomcat-*"
[root@ES-Node2 ~]#systemctl restart filebeat.service 
案例: 利用 Filebeat 收集 Tomat 的多行错误日志到Elasticsearch
Tomcat 错误日志解析
Tomcat 是 Java 应用,当只出现一个错误时,会显示很多行的错误日志,如下所示
[root@ES-Node2 ~]#tail -25f /var/log/tomcat9/catalina.2022-12-15.log
15-Dec-2022 17:07:44.938 严重 [main] org.apache.tomcat.util.digester.Digester.fatalError Parse fatal error at line [171] column [7]
org.xml.sax.SAXParseException; systemId: file:/var/lib/tomcat9/conf/server.xml; lineNumber: 171; columnNumber: 7; 元素类型 "Host" 必须由匹配的结束标记 "</Host>" 终止。
at java.xml/com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:204)
at java.xml/com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:178)
at java.xml/com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:400)
at java.xml/com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:327)
at java.xml/com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1465)
at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1685)
at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2883)
at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:605)
at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:534)
at java.xml/com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:888)
at java.xml/com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:824)
at java.xml/com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
at java.xml/com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1216)
at java.xml/com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:635)
at org.apache.tomcat.util.digester.Digester.parse(Digester.java:1468)
at org.apache.catalina.startup.Catalina.load(Catalina.java:566)
at org.apache.catalina.startup.Catalina.load(Catalina.java:607)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:303)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:473)
15-Dec-2022 17:08:33.729 严重 [main] org.apache.tomcat.util.digester.Digester.fatalError Parse fatal error at line [171] column [7]
org.xml.sax.SAXParseException; systemId: file:/var/lib/tomcat9/conf/server.xml; lineNumber: 171; columnNumber: 7; 元素类型 "Host" 必须由匹配的结束标记 "</Host>" 终止。
at java.xml/com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:204)
at java.xml/com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:178)
at java.xml/com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:400)
at java.xml/com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:327)
at java.xml/com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1465)
at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1685)
at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2883)
at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:605)
at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:534)
at java.xml/com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:888)
at java.xml/com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:824)
at java.xml/com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
at java.xml/com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1216)
at java.xml/com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:635)
at org.apache.tomcat.util.digester.Digester.parse(Digester.java:1468)
at org.apache.catalina.startup.Catalina.load(Catalina.java:566)
at org.apache.catalina.startup.Catalina.load(Catalina.java:607)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:303)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:473)
多个行其实是同一个事件的日志的内容,而ES是根据每一行来区别不同的日志
可以将多个行合并成一个日志来解决此问题
官方文档:https://www.elastic.co/guide/en/beats/filebeat/7.0/multiline-examples.html
multiline.pattern: '^\['
multiline.negate: true
multiline.match: after修改 Filebeat 配置文件
[root@ES-Node2 ~]#vim /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/tomcat9/localhost_access_log.*
json.keys_under_root: true #默认False会将json数据存储至message，改为true则会独立message外存储
json.overwrite_keys: false #设为true,覆盖默认的message字段，使用自定义json格式中的key
tags: ["tomcat-access"]
- type: log
enabled: true
paths:
- /var/log/tomcat9/catalina.*.log
tags: ["tomcat-error"]
multiline.type: pattern #此为默认值,可省略
multiline.pattern: '^[0-3][0-9]-' #正则表达式匹配以两位,或者为'^\d{2}'
multiline.negate: true
multiline.match: after
multiline.maxlines: 10000 #默认只合并500行,指定最大合并1万行
output.elasticsearch:
hosts: ["10.0.0.208:9200"] #指定ELK集群服务器地址和端口
indices:
- index: "tomcat-access-%{[agent.version]}-%{+yyy.MM.dd}"
when.contains:
tags: "tomcat-access"
- index: "tomcat-error-%{[agent.version]}-%{+yyy.MM.dd}"
when.contains:
tags: "tomcat-error"
setup.ilm.enabled: false
setup.template.name: "tomcat"
setup.template.pattern: "tomcat-*"
[root@ES-Node2 ~]#systemctl restart filebeat.service
案例: 从标准输入读取再输出至 Logstash
vim /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: stdin
enabled: true
output.logstash:
hosts: ["10.0.0.104:5044","10.0.0.105:5044"]
index: filebeat
loadbalance: true #默认为false,只随机输出至一个可用的logstash,设为true,则输出至全部logstash
worker: 1 #线程数量
compression_level: 3 #压缩比案例: 利用 Filebeat 收集 Nginx 日志到 Redis
官方文档
https://www.elastic.co/guide/en/beats/filebeat/7.6/filebeat-input-log.html
https://www.elastic.co/guide/en/beats/filebeat/7.6/redis-output.html
生产环境中我们经常需要获取Web访问用户的信息，比如:来源的IP是哪个地域，网站的PV、UV、状态码、访问时间等等;所以需要收集的Nginx访问日志将 filebeat收集的日志,发送至Redis 格式如下
output.redis:
hosts: ["localhost:6379"]
password: "my_password"
key: "filebeat"
db: 0
timeout: 5
output.redis:
hosts: ["10.0.0.105:6379"]
password: "123456"
db: "0"
key: "filebeat" #所有日志都存放在key名称为filebeat的列表中，llen filebeat可查看长度，即日志记录数
#keys: #也可以用下面的不同日志存放在不同的key的形式
# - key: "nginx_access"
# when.contains:
# tags: "access"
# - key: "nginx_error"
# when.contains:
# tags: "error"安装 Nginx 配置访问日志使用 Json格式(参考前面nginx方法)
安装和配置 Redis
[root@ES-Node2 ~]#apt -y install redis
[root@ES-Node2 ~]#sed -i.bak '/^bind.*/c bind 0.0.0.0' /etc/redis/redis.conf
[root@ES-Node2 ~]#systemctl restart redis修改 Filebeat 配置文件
[root@ES-Node2 ~]#vim /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/nginx/access_json.log
json.keys_under_root: true #默认False会将json数据存储至message，改为true则会独立message外存储
json.overwrite_keys: true #设为true,覆盖默认的message字段，使用自定义json格式中的key
tags: ["nginx-access"]
- type: log
enabled: true
paths:
- /var/log/nginx/error.log
tags: ["nginx-error"]
output.redis:
hosts: ["10.0.0.209:6379"]
key: "filebeat"
#password: "123456"
#db: 0
[root@ES-Node2 ~]#systemctl restart filebeat.service
[root@ES-Node2 ~]#redis-cli
127.0.0.1:6379> llen filebeat
(integer) 0
127.0.0.1:6379> keys *
(empty list or set)
127.0.0.1:6379> keys *
1) "filebeat"
127.0.0.1:6379> llen filebeat
(integer) 3
127.0.0.1:6379> LINDEX filebeat 2
"{\"@timestamp\":\"2022-12-15T09:28:48.000Z\",\"@metadata\":{\"beat\":\"filebeat\",\"type\":\"_doc\",\"version\":\"7.15.0\"},\"upstreamhost\":\"-\",\"tcp_xff\":\"-\",\"log\":{\"offset\":8721,\"file\":{\"path\":\"/var/log/nginx/access_json.log\"}},\"host\":{\"name\":\"ES-Node2.com\"},\"responsetime\":0,\"http_user_agent\":\"curl/7.68.0\",\"ecs\":{\"version\":\"1.11.0\"},\"http_host\":\"10.0.0.209\",\"clientip\":\"10.0.0.210\",\"xff\":\"-\",\"uri\":\"/index.html\",\"upstreamtime\":\"-\",\"status\":\"200\",\"tags\":[\"nginx-access\"],\"agent\":{\"hostname\":\"ES-Node2.com\",\"ephemeral_id\":\"b31fd668-dfa4-422f-a810-45274d76e528\",\"id\":\"f87dddf0-89bf-46c9-be70-0f8e1784eb2a\",\"name\":\"ES-Node2.com\",\"type\":\"filebeat\",\"version\":\"7.15.0\"},\"domain\":\"10.0.0.209\",\"size\":10918,\"referer\":\"-\",\"input\":{\"type\":\"log\"}}"
127.0.0.1:6379> 案例: 从标准输入读取再输出至 Kafka
vim /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: stdin
enabled: true
output.kafka:
hosts: ["10.0.0.201:9092", "10.0.0.202:9092", "10.0.0.203:9092"]
topic: filebeat-log #指定kafka的topic
partition.round_robin:
reachable_only: true#true表示只发布到可用的分区，false时表示所有分区，如果一个节点down，会block
required_acks: 1 #如果为0，错误消息可能会丢失，1等待写入主分区（默认），-1等待写入副本分区
compression: gzip
max_message_bytes: 1000000 #每条消息最大长度，以字节为单位，如果超过将丢弃