Logstash 介绍
Logstash 是免费且开放的服务器端数据处理管道，能够从多个来源采集数据，转换数据，然后将数据发送到您最喜欢的“存储库”中
Logstash 可以水平伸缩，而且logstash是整个ELK中拥有最多插件的一个组件
Losgtash 基于 Ruby 语言开发
Logstash 主要功能：
输入 Input：用于日志收集,常见插件: Stdin、File、Kafka、Redis、Filebeat、Http
过滤 Filter：日志过滤和转换,常用插件: grok、date、geoip、mutate、useragent
输出 Output：将过滤转换过的日志输出, 常见插件: File,Stdout,Elasticsearch,MySQL,Redis,Kafka
Logstash 官网:https://www.elastic.co/cn/logstash/
Logstash 官方下载链接:
https://www.elastic.co/cn/downloads/logstash
https://www.elastic.co/cn/downloads/past-releases#logstash
Logstash 官方说明:
https://www.elastic.co/guide/en/logstash/7.6/index.html
Logstash 安装
Ubuntu 环境准备
apt -y install openjdk-11-jdk
apt -y install openjdk-8-jdkCentOS 环境准备
关闭防火墙和 SELinux
systemctl disable --now firewalld
sed -i '/SELINUX/s/enforcing/disabled/' /etc/selinux/config注意: Logstash 版本要和 Elasticsearch 相同的版本，否则可能会出错
镜像网站下载链接:https://mirrors.tuna.tsinghua.edu.cn/elasticstack/7.x
[root@ubuntu2004 ~]#apt -y install openjdk-11-jdk
[root@ubuntu2004 ~]#wget https://mirrors.tuna.tsinghua.edu.cn/elasticstack/7.x/apt/pool/main/l/logstash/logstash-7.15.0-amd64.deb
--2022-12-15 17:39:58-- https://mirrors.tuna.tsinghua.edu.cn/elasticstack/7.x/apt/pool/main/l/logstash/logstash-7.15.0-amd64.deb
正在解析主机 mirrors.tuna.tsinghua.edu.cn (mirrors.tuna.tsinghua.edu.cn)... 101.6.15.130, 2402:f000:1:400::2
正在连接 mirrors.tuna.tsinghua.edu.cn (mirrors.tuna.tsinghua.edu.cn)|101.6.15.130|:443... 已连接。
已发出 HTTP 请求，正在等待回应... 200 OK
长度： 370322504 (353M) [application/octet-stream]
正在保存至: “logstash-7.15.0-amd64.deb”
logstash-7.15.0-amd64.deb 100%[===================================================================>] 353.17M 800KB/s 用时 7m 7s
2022-12-15 17:47:06 (846 KB/s) - 已保存 “logstash-7.15.0-amd64.deb” [370322504/370322504])
[root@ubuntu2004 ~]#dpkg -i logstash-7.15.0-amd64.deb
正在选中未选择的软件包 logstash。
(正在读取数据库 ... 系统当前共安装有 161712 个文件和目录。)
准备解压 logstash-7.15.0-amd64.deb ...
正在解压 logstash (1:7.15.0-1) ...
正在设置 logstash (1:7.15.0-1) ...
Using bundled JDK: /usr/share/logstash/jdk
Using provided startup.options file: /etc/logstash/startup.options
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/pleaserun-0.0.32/lib/pleaserun/platform/base.rb:112: warning: constant ::Fixnum is deprecated
Successfully created system startup script for Logstash
修改 Logstash 配置
[root@ubuntu2004 ~]#egrep -v "^#|^$" /etc/logstash/logstash.yml
path.data: /var/lib/logstash
path.logs: /var/log/logstash
node.name: logstash-node01
pipeline.workers: 2
pipeline.batch.size: 1000 #批量写入的消息个数，可以根据ES的性能做性能优化
pipeline.batch.delay: 5 #批量写入的延时，可以根据ES的性能做性能优化
#内存优化
[root@ubuntu2004 ~]#egrep -v "^#|^$" /etc/logstash/jvm.options
-Xms1g
-Xmx1g
8-13:-XX:+UseConcMarkSweepGC
8-13:-XX:CMSInitiatingOccupancyFraction=75
8-13:-XX:+UseCMSInitiatingOccupancyOnly
-Djava.awt.headless=true
-Dfile.encoding=UTF-8
-Djruby.compile.invokedynamic=true
-Djruby.jit.threshold=0
-Djruby.regexp.interruptible=true
-XX:+HeapDumpOnOutOfMemoryError
-Djava.security.egd=file:/dev/urandom
-Dlog4j2.isThreadContextMapInheritable=true
#Logstash默认以logstash用户运行,如果logstash需要收集本机的日志,可能会有权限问题,可以修改为root
vim /etc/systemd/system/logstash.service
[Service]
User=root
Group=root
systemctl daemon-reload;systemctl restart logstashLogstash 使用
Logstash 命令
官方文档
https://www.elastic.co/guide/en/logstash/current/first-event.html
#各种插件
https://www.elastic.co/guide/en/logstash/7.6/input-plugins.html
https://www.elastic.co/guide/en/logstash/7.6/filter-plugins.html
https://www.elastic.co/guide/en/logstash/7.6/output-plugins.html
范例： 查看帮助
[root@ubuntu2004 ~]#/usr/share/logstash/bin/logstash --help
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Usage:
bin/logstash [OPTIONS]
Options:
-n, --node.name NAME Specify the name of this logstash instance, if no value is given
it will default to the current hostname.
(default: "ubuntu2004")
--enable-local-plugin-development Allow Gemfile to be manipulated directly
to facilitate simpler local plugin
development.
This is an advanced setting, intended
only for use by Logstash developers,
and should not be used in production.
(default: false)
-f, --path.config CONFIG_PATH Load the logstash config from a specific file
or directory. If a directory is given, all
files in that directory will be concatenated
in lexicographical order and then parsed as a
single config file. You can also specify
wildcards (globs) and any matched files will
be loaded in the order described above.
-e, --config.string CONFIG_STRING Use the given string as the configuration
data. Same syntax as the config file. If no
input is specified, then the following is
used as the default input:
"input { stdin { type => stdin } }"
and if no output is specified, then the
following is used as the default output:
"output { stdout { codec => rubydebug } }"
If you wish to use both defaults, please use
the empty string for the '-e' flag.
(default: nil)
--field-reference-parser MODE (DEPRECATED) This option is no longer
configurable.
Use the given MODE when parsing field
references.
The field reference parser is used to expand
field references in your pipeline configs,
and has become more strict to better handle
ambiguous- and illegal-syntax inputs.
The only available MODE is:
- STRICT: parse in a strict manner; when
given ambiguous- or illegal-syntax input,
raises a runtime exception that should
be handled by the calling plugin.
(default: "STRICT")
--modules MODULES Load Logstash modules.
Modules can be defined using multiple instances
'--modules module1 --modules module2',
or comma-separated syntax
'--modules=module1,module2'
Cannot be used in conjunction with '-e' or '-f'
Use of '--modules' will override modules declared
in the 'logstash.yml' file.
-M, --modules.variable MODULES_VARIABLE Load variables for module template.
Multiple instances of '-M' or
'--modules.variable' are supported.
Ignored if '--modules' flag is not used.
Should be in the format of
'-M "MODULE_NAME.var.PLUGIN_TYPE.PLUGIN_NAME.VARIABLE_NAME=VALUE"'
as in
'-M "example.var.filter.mutate.fieldname=fieldvalue"'
--setup Load index template into Elasticsearch, and saved searches,
index-pattern, visualizations, and dashboards into Kibana when
running modules.
(default: false)
--cloud.id CLOUD_ID Sets the elasticsearch and kibana host settings for
module connections in Elastic Cloud.
Your Elastic Cloud User interface or the Cloud support
team should provide this.
Add an optional label prefix '<label>:' to help you
identify multiple cloud.ids.
e.g. 'staging:dXMtZWFzdC0xLmF3cy5mb3VuZC5pbyRub3RhcmVhbCRpZGVudGlmaWVy'
--cloud.auth CLOUD_AUTH Sets the elasticsearch and kibana username and password
for module connections in Elastic Cloud
e.g. 'username:<password>'
--pipeline.id ID Sets the ID of the pipeline.
(default: "main")
-w, --pipeline.workers COUNT Sets the number of pipeline workers to run.
(default: 2)
--pipeline.ordered ORDERED Preserve events order. Possible values are auto (default), true and false.
This setting
will only work when also using a single worker for the pipeline.
Note that when enabled, it may impact the performance of the filters
and ouput processing.
The auto option will automatically enable ordering if the
pipeline.workers setting is set to 1.
Use true to enable ordering on the pipeline and prevent logstash
from starting if there are multiple workers.
Use false to disable any extra processing necessary for preserving
ordering.
(default: "auto")
--java-execution Use Java execution engine.
(default: true)
--plugin-classloaders (Beta) Load Java plugins in independent classloaders to isolate their dependencies.
(default: false)
-b, --pipeline.batch.size SIZE Size of batches the pipeline is to work in.
(default: 125)
-u, --pipeline.batch.delay DELAY_IN_MS When creating pipeline batches, how long to wait while polling
for the next event.
(default: 50)
--pipeline.unsafe_shutdown Force logstash to exit during shutdown even
if there are still inflight events in memory.
By default, logstash will refuse to quit until all
received events have been pushed to the outputs.
(default: false)
--pipeline.ecs_compatibility STRING Sets the pipeline's default value for ecs_compatibility,
a setting that is available to plugins that implement
an ECS Compatibility mode for use with the Elastic Common
Schema.
Possible values are:
- disabled (default)
- v1
- v2
This option allows the early opt-in (or preemptive opt-out)
of ECS Compatibility modes in plugins, which is scheduled to
be on-by-default in a future major release of Logstash.
Values other than disabled are currently considered BETA,
and may produce unintended consequences when upgrading Logstash.
(default: "disabled")
--path.data PATH This should point to a writable directory. Logstash
will use this directory whenever it needs to store
data. Plugins will also have access to this path.
(default: "/usr/share/logstash/data")
-p, --path.plugins PATH A path of where to find plugins. This flag
can be given multiple times to include
multiple paths. Plugins are expected to be
in a specific directory hierarchy:
'PATH/logstash/TYPE/NAME.rb' where TYPE is
'inputs' 'filters', 'outputs' or 'codecs'
and NAME is the name of the plugin.
(default: [])
-l, --path.logs PATH Write logstash internal logs to the given
file. Without this flag, logstash will emit
logs to standard output.
(default: "/usr/share/logstash/logs")
--log.level LEVEL Set the log level for logstash. Possible values are:
- fatal
- error
- warn
- info
- debug
- trace
(default: "info")
--config.debug Print the compiled config ruby code out as a debug log (you must also have --log.level=debug enabled).
WARNING: This will include any 'password' options passed to plugin configs as plaintext, and may result
in plaintext passwords appearing in your logs!
(default: false)
-i, --interactive SHELL Drop to shell instead of running as normal.
Valid shells are "irb" and "pry"
-V, --version Emit the version of logstash and its friends,
then exit.
-t, --config.test_and_exit Check configuration for valid syntax and then exit.
(default: false)
-r, --config.reload.automatic Monitor configuration changes and reload
whenever it is changed.
NOTE: use SIGHUP to manually reload the config
(default: false)
--config.reload.interval RELOAD_INTERVAL How frequently to poll the configuration location
for changes, in seconds.
(default: #<Java::OrgLogstashUtil::TimeValue:0x3382adc2>)
--http.enabled ENABLED Can be used to disable the Web API, which is
enabled by default.
(default: true)
--http.host HTTP_HOST Web API binding host (default: "127.0.0.1")
--http.port HTTP_PORT Web API http port (default: 9600..9700)
--log.format FORMAT Specify if Logstash should write its own logs in JSON form (one
event per line) or in plain text (using Ruby's Object#inspect)
(default: "plain")
--path.settings SETTINGS_DIR Directory containing logstash.yml file. This can also be
set through the LS_SETTINGS_DIR environment variable.
(default: "/usr/share/logstash/config")
--verbose Set the log level to info.
DEPRECATED: use --log.level=info instead.
--debug Set the log level to debug.
DEPRECATED: use --log.level=debug instead.
--quiet Set the log level to info.
DEPRECATED: use --log.level=info instead.
-h, --help print help
#常用选项
-e 指定配置内容
-f 指定配置文件
-t 语法检查
-r 修改配置文件,自动加载生效
#服务方式启动,由于默认没有配置文件,所以无法启动
[root@logstash ~]#systemctl start logstash各种插件帮助
https://www.elastic.co/guide/en/logstash/current/index.html
范例: 列出所有插件
[root@ubuntu2004 ~]#/usr/share/logstash/bin/logstash-plugin list
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
logstash-codec-avro
logstash-codec-cef
logstash-codec-collectd
logstash-codec-dots
logstash-codec-edn
logstash-codec-edn_lines
logstash-codec-es_bulk
logstash-codec-fluent
logstash-codec-graphite
logstash-codec-json
logstash-codec-json_lines
logstash-codec-line
logstash-codec-msgpack
logstash-codec-multiline
logstash-codec-netflow
logstash-codec-plain
logstash-codec-rubydebug
logstash-filter-aggregate
logstash-filter-anonymize
logstash-filter-cidr
logstash-filter-clone
logstash-filter-csv
logstash-filter-date
logstash-filter-de_dot
logstash-filter-dissect
logstash-filter-dns
logstash-filter-drop
logstash-filter-elasticsearch
logstash-filter-fingerprint
logstash-filter-geoip
logstash-filter-grok
logstash-filter-http
logstash-filter-json
logstash-filter-kv
logstash-filter-memcached
logstash-filter-metrics
logstash-filter-mutate
logstash-filter-prune
logstash-filter-ruby
logstash-filter-sleep
logstash-filter-split
logstash-filter-syslog_pri
logstash-filter-throttle
logstash-filter-translate
logstash-filter-truncate
logstash-filter-urldecode
logstash-filter-useragent
logstash-filter-uuid
logstash-filter-xml
logstash-input-azure_event_hubs
logstash-input-beats
└── logstash-input-elastic_agent (alias)
logstash-input-couchdb_changes
logstash-input-dead_letter_queue
logstash-input-elasticsearch
logstash-input-exec
logstash-input-file
logstash-input-ganglia
logstash-input-gelf
logstash-input-generator
logstash-input-graphite
logstash-input-heartbeat
logstash-input-http
logstash-input-http_poller
logstash-input-imap
logstash-input-jms
logstash-input-pipe
logstash-input-redis
logstash-input-s3
logstash-input-snmp
logstash-input-snmptrap
logstash-input-sqs
logstash-input-stdin
logstash-input-syslog
logstash-input-tcp
logstash-input-twitter
logstash-input-udp
logstash-input-unix
logstash-integration-elastic_enterprise_search
├── logstash-output-elastic_app_search
└── logstash-output-elastic_workplace_search
logstash-integration-jdbc
├── logstash-input-jdbc
├── logstash-filter-jdbc_streaming
└── logstash-filter-jdbc_static
logstash-integration-kafka
├── logstash-input-kafka
└── logstash-output-kafka
logstash-integration-rabbitmq
├── logstash-input-rabbitmq
└── logstash-output-rabbitmq
logstash-output-cloudwatch
logstash-output-csv
logstash-output-elasticsearch
logstash-output-email
logstash-output-file
logstash-output-graphite
logstash-output-http
logstash-output-lumberjack
logstash-output-nagios
logstash-output-null
logstash-output-pipe
logstash-output-redis
logstash-output-s3
logstash-output-sns
logstash-output-sqs
logstash-output-stdout
logstash-output-tcp
logstash-output-udp
logstash-output-webhdfs
logstash-patterns-coreGithub logstash插件链接
https://github.com/logstash-plugins
Logstash 输入 Input 插件
官方链接
https://www.elastic.co/guide/en/logstash/7.6/input-plugins.html
标准输入
范例: 交互式实现
#标准输入和输出,codec => rubydebug指输出格式，是默认值，可以省略
/usr/share/logstash/bin/logstash -e 'input { stdin{} } output { stdout{ codec => rubydebug }}'
[root@ubuntu2004 ~]#/usr/share/logstash/bin/logstash -e 'input { stdin{} } output { stdout{ codec => rubydebug }}'
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2022-12-15 18:09:07.112 [main] runner - Starting Logstash {"logstash.version"=>"7.15.0", "jruby.version"=>"jruby 9.2.19.0 (2.5.8) 2021-06-15 55810c552b OpenJDK 64-Bit Server VM 11.0.11+9 on 11.0.11+9 +indy +jit [linux-x86_64]"}
[INFO ] 2022-12-15 18:09:07.191 [main] writabledirectory - Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
[INFO ] 2022-12-15 18:09:07.224 [main] writabledirectory - Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
[WARN ] 2022-12-15 18:09:08.071 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2022-12-15 18:09:08.149 [LogStash::Runner] agent - No persistent UUID file found. Generating new UUID {:uuid=>"5cc3720e-856d-452a-aebe-2d10613cc42f", :path=>"/usr/share/logstash/data/uuid"}
[INFO ] 2022-12-15 18:09:11.181 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[INFO ] 2022-12-15 18:09:12.105 [Converge PipelineAction::Create<main>] Reflections - Reflections took 148 ms to scan 1 urls, producing 120 keys and 417 values
[WARN ] 2022-12-15 18:09:13.281 [Converge PipelineAction::Create<main>] line - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2022-12-15 18:09:13.364 [Converge PipelineAction::Create<main>] stdin - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[INFO ] 2022-12-15 18:09:14.088 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["config string"], :thread=>"#<Thread:0x58a64c22 run>"}
[INFO ] 2022-12-15 18:09:15.696 [[main]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>1.6}
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.jrubystdinchannel.StdinChannelLibrary$Reader (file:/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/jruby-stdin-channel-0.2.0-java/lib/jruby_stdin_channel/jruby_stdin_channel.jar) to field java.io.FilterInputStream.in
WARNING: Please consider reporting this to the maintainers of com.jrubystdinchannel.StdinChannelLibrary$Reader
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
[INFO ] 2022-12-15 18:09:15.851 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
The stdin plugin is now waiting for input:
[INFO ] 2022-12-15 18:09:15.951 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
hello world #标准输入此字符串
{
"message" => "hello world", #消息的具体内容
"@version" => "1", #事件版本号，一个事件就是一个ruby对象
"@timestamp" => 2022-12-15T10:09:34.667Z, #当前事件的发生时间
"host" => "ubuntu2004" #标记事件发生在哪里
}
#后续还可继续输入其它信息，按ctrl+c退出#指定输入信息为Json格式
logstash -e 'input { stdin{ codec => json } } output { stdout{ codec => rubydebug }}'
[root@ubuntu2004 ~]#/usr/share/logstash/bin/logstash -e 'input { stdin{ codec => json } } output { stdout{ codec => rubydebug }}'
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2022-12-15 18:12:30.672 [main] runner - Starting Logstash {"logstash.version"=>"7.15.0", "jruby.version"=>"jruby 9.2.19.0 (2.5.8) 2021-06-15 55810c552b OpenJDK 64-Bit Server VM 11.0.11+9 on 11.0.11+9 +indy +jit [linux-x86_64]"}
[WARN ] 2022-12-15 18:12:31.733 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2022-12-15 18:12:34.819 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[INFO ] 2022-12-15 18:12:35.627 [Converge PipelineAction::Create<main>] Reflections - Reflections took 174 ms to scan 1 urls, producing 120 keys and 417 values
[WARN ] 2022-12-15 18:12:37.031 [Converge PipelineAction::Create<main>] json - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2022-12-15 18:12:37.142 [Converge PipelineAction::Create<main>] stdin - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[INFO ] 2022-12-15 18:12:37.841 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["config string"], :thread=>"#<Thread:0x77ac69f4 run>"}
[INFO ] 2022-12-15 18:12:39.448 [[main]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>1.6}
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.jrubystdinchannel.StdinChannelLibrary$Reader (file:/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/jruby-stdin-channel-0.2.0-java/lib/jruby_stdin_channel/jruby_stdin_channel.jar) to field java.io.FilterInputStream.in
WARNING: Please consider reporting this to the maintainers of com.jrubystdinchannel.StdinChannelLibrary$Reader
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
[INFO ] 2022-12-15 18:12:39.573 [[main]-pipeline-manager] stdin - Automatically switching from json to json_lines codec {:plugin=>"stdin"}
[WARN ] 2022-12-15 18:12:39.628 [[main]-pipeline-manager] jsonlines - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[INFO ] 2022-12-15 18:12:39.639 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
The stdin plugin is now waiting for input:
[INFO ] 2022-12-15 18:12:39.769 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
{ "name":"shuhong","age": "18"}
{
"@timestamp" => 2022-12-15T10:57:02.692Z,
"age" => "18",
"@version" => "1",
"host" => "ubuntu2004",
"name" => "shuhong"
}范例: 以配置文件实现
[root@ubuntu2004 ~]#vim /etc/logstash/conf.d/stdin_to_stdout.conf
nput {
stdin {
type => "stdin_type" #自定义事件类型，可用于后续判断
tags => "stdin_tag" #自定义事件tag，可用于后续判断
codec => "json" #指定Json 格式
}
}
output {
stdout {
codec => "rubydebug" #输出格式,此为默认值,可省略
}
}
#语法检查
[root@ubuntu2004 ~]#/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/stdin_to_stdout.conf -t
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2022-12-15 19:02:01.095 [main] runner - Starting Logstash {"logstash.version"=>"7.15.0", "jruby.version"=>"jruby 9.2.19.0 (2.5.8) 2021-06-15 55810c552b OpenJDK 64-Bit Server VM 11.0.11+9 on 11.0.11+9 +indy +jit [linux-x86_64]"}
[WARN ] 2022-12-15 19:02:02.184 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2022-12-15 19:02:03.621 [LogStash::Runner] Reflections - Reflections took 238 ms to scan 1 urls, producing 120 keys and 417 values
[WARN ] 2022-12-15 19:02:04.838 [LogStash::Runner] json - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2022-12-15 19:02:04.945 [LogStash::Runner] stdin - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
Configuration OK
[INFO ] 2022-12-15 19:02:05.273 [LogStash::Runner] runner - Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
#执行[root@ubuntu2004 ~]#/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/stdin_to_stdout.conf
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2022-12-15 19:03:27.954 [main] runner - Starting Logstash {"logstash.version"=>"7.15.0", "jruby.version"=>"jruby 9.2.19.0 (2.5.8) 2021-06-15 55810c552b OpenJDK 64-Bit Server VM 11.0.11+9 on 11.0.11+9 +indy +jit [linux-x86_64]"}
[WARN ] 2022-12-15 19:03:28.985 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2022-12-15 19:03:32.162 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[INFO ] 2022-12-15 19:03:33.350 [Converge PipelineAction::Create<main>] Reflections - Reflections took 223 ms to scan 1 urls, producing 120 keys and 417 values
[WARN ] 2022-12-15 19:03:34.668 [Converge PipelineAction::Create<main>] json - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2022-12-15 19:03:34.764 [Converge PipelineAction::Create<main>] stdin - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[INFO ] 2022-12-15 19:03:35.821 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["/etc/logstash/conf.d/stdin_to_stdout.conf"], :thread=>"#<Thread:0x58d74f74 run>"}
[INFO ] 2022-12-15 19:03:37.386 [[main]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>1.55}
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.jrubystdinchannel.StdinChannelLibrary$Reader (file:/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/jruby-stdin-channel-0.2.0-java/lib/jruby_stdin_channel/jruby_stdin_channel.jar) to field java.io.FilterInputStream.in
WARNING: Please consider reporting this to the maintainers of com.jrubystdinchannel.StdinChannelLibrary$Reader
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
[INFO ] 2022-12-15 19:03:37.489 [[main]-pipeline-manager] stdin - Automatically switching from json to json_lines codec {:plugin=>"stdin"}
[WARN ] 2022-12-15 19:03:37.558 [[main]-pipeline-manager] jsonlines - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[INFO ] 2022-12-15 19:03:37.577 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
The stdin plugin is now waiting for input:
[INFO ] 2022-12-15 19:03:37.760 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
hello,shuhong
[WARN ] 2022-12-15 19:03:43.213 [[main]<stdin] jsonlines - JSON parse error, original data now in message field {:message=>"Unrecognized token 'hello': was expecting ('true', 'false' or 'null')\n at [Source: (String)\"hello,shuhong\"; line: 1, column: 6]", :exception=>LogStash::Json::ParserError, :data=>"hello,shuhong"}
{
"@timestamp" => 2022-12-15T11:03:43.248Z,
"message" => "hello,shuhong",
"type" => "stdin_type",
"tags" => [
[0] "_jsonparsefailure",
[1] "stdin_tag"
],
"@version" => "1",
"host" => "ubuntu2004"
}
{ "name":"shuhong","age": "18"}
{
"age" => "18",
"@timestamp" => 2022-12-15T11:04:07.878Z,
"type" => "stdin_type",
"@version" => "1",
"tags" => [
[0] "stdin_tag"
],
"host" => "ubuntu2004",
"name" => "shuhong"
}
从文件输入
Logstash 会记录每个文件的读取位置,下次自动从此位置继续向后读取
每个文件的读取位置记录在 /var/lib/logsstash/plugins/inputs/files/.sincedb_xxxx对应的文件中
此文件包括文件的 inode号, 大小等信息
范例:
[root@ubuntu2004 ~]#vim /etc/logstash/conf.d/file_to_stdout.conf
input {
file {
path => "/tmp/shu.*"
type => "shulog" #添加自定义的type字段,可以用于条件判断
exclude => "*.txt" #排除不想采集数据的文件，基于通配符glob匹配语法
start_position => "beginning" #第一次丛头开始读取文件,可以取值为:beginning和end
stat_interval => "3" #定时检查文件是否更新，默认1s
codec => json #如果文件是Json格式,需要指定此项才能解析,如果不是Json格式,却添加此行也不会影响结果
}
file {
path => "/var/log/syslog"
type => "syslog"
start_position => "beginning"
stat_interval => "3"
}
}
output {
stdout {
codec => rubydebug
}
}
[root@ubuntu2004 ~]#echo line1 >> /tmp/shu.log
[root@ubuntu2004 ~]#/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/file_to_stdout.conf -t
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2022-12-15 19:08:39.513 [main] runner - Starting Logstash {"logstash.version"=>"7.15.0", "jruby.version"=>"jruby 9.2.19.0 (2.5.8) 2021-06-15 55810c552b OpenJDK 64-Bit Server VM 11.0.11+9 on 11.0.11+9 +indy +jit [linux-x86_64]"}
[WARN ] 2022-12-15 19:08:40.369 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2022-12-15 19:08:41.805 [LogStash::Runner] Reflections - Reflections took 106 ms to scan 1 urls, producing 120 keys and 417 values
[WARN ] 2022-12-15 19:08:43.067 [LogStash::Runner] json - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2022-12-15 19:08:43.212 [LogStash::Runner] file - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2022-12-15 19:08:43.301 [LogStash::Runner] plain - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2022-12-15 19:08:43.338 [LogStash::Runner] file - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
Configuration OK
[INFO ] 2022-12-15 19:08:43.619 [LogStash::Runner] runner - Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
[root@ubuntu2004 ~]#/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/file_to_stdout.conf
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2022-12-15 19:11:09.459 [main] runner - Starting Logstash {"logstash.version"=>"7.15.0", "jruby.version"=>"jruby 9.2.19.0 (2.5.8) 2021-06-15 55810c552b OpenJDK 64-Bit Server VM 11.0.11+9 on 11.0.11+9 +indy +jit [linux-x86_64]"}
[WARN ] 2022-12-15 19:11:10.439 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2022-12-15 19:11:13.674 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[INFO ] 2022-12-15 19:11:14.990 [Converge PipelineAction::Create<main>] Reflections - Reflections took 179 ms to scan 1 urls, producing 120 keys and 417 values
[WARN ] 2022-12-15 19:11:16.458 [Converge PipelineAction::Create<main>] json - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2022-12-15 19:11:16.635 [Converge PipelineAction::Create<main>] file - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2022-12-15 19:11:16.837 [Converge PipelineAction::Create<main>] plain - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2022-12-15 19:11:16.879 [Converge PipelineAction::Create<main>] file - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[INFO ] 2022-12-15 19:11:17.440 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["/etc/logstash/conf.d/file_to_stdout.conf"], :thread=>"#<Thread:0x14e0381e run>"}
[INFO ] 2022-12-15 19:11:19.120 [[main]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>1.67}
[INFO ] 2022-12-15 19:11:19.251 [[main]-pipeline-manager] file - No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/usr/share/logstash/data/plugins/inputs/file/.sincedb_812e2de7445d78534a4b575c69114d72", :path=>["/tmp/shu.*"]}
[INFO ] 2022-12-15 19:11:19.324 [[main]-pipeline-manager] file - No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/usr/share/logstash/data/plugins/inputs/file/.sincedb_f5fdf6ea0ea92860c6a6b2b354bfcbbc", :path=>["/var/log/syslog"]}
[INFO ] 2022-12-15 19:11:19.349 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[INFO ] 2022-12-15 19:11:19.536 [[main]<file] observingtail - START, creating Discoverer, Watch with file and sincedb collections
[INFO ] 2022-12-15 19:11:19.547 [[main]<file] observingtail - START, creating Discoverer, Watch with file and sincedb collections
[INFO ] 2022-12-15 19:11:19.632 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[ERROR] 2022-12-15 19:11:20.409 [[main]<file] json - JSON parse error, original data now in message field {:message=>"Unrecognized token 'line1': was expecting ('true', 'false' or 'null')\n at [Source: (String)\"line1\"; line: 1, column: 11]", :exception=>LogStash::Json::ParserError, :data=>"line1"}
{
"tags" => [
[0] "_jsonparsefailure"
],
"path" => "/tmp/shu.log",
"message" => "line1",
"@version" => "1",
"host" => "ubuntu2004",
"@timestamp" => 2022-12-15T11:11:20.419Z,
"type" => "shulog"
}
{
"path" => "/var/log/syslog",
"message" => "Dec 15 15:45:47 ubuntu2004 rsyslogd: [origin software=\"rsyslogd\" swVersion=\"8.2001.0\" x-pid=\"806\" x-info=\"https://www.rsyslog.com\"] rsyslogd was HUPed",
"@version" => "1",
"host" => "ubuntu2004",
"@timestamp" => 2022-12-15T11:11:20.505Z,
"type" => "syslog"
}
.....
ogstash利用 sincedb 文件记录了logstash收集的记录文件的信息，比如位置，以方便下次接着从此位置继续收集日志
cat /var/lib/logstash/plugins/inputs/file/.sincedb_1ced15cfacdbb0380466be84d620085a
134219868 0 2064 29465 #记录了收集文件的inode和大小等信息
ll -li /elk/logs/ELK-Cluster.log
134219868 -rw-r--r-- 1 elasticsearch elasticsearch 29465 Apr 21 14:33 /elk/logs/ELK-Cluster.log从 Http 请求获取数
[root@ubuntu2004 ~]#vim /etc/logstash/conf.d/http_to_stdout.conf
input {
http {
port =>6666
codec => json
}
}
output {
stdout {
codec => rubydebug
}
}
[root@ES-Node2 ~]#curl http://10.0.0.207:6666
ok
[root@ES-Node2 ~]#curl -XPOST -d'test log message' http://10.0.0.207:6666
ok
[root@ubuntu2004 ~]#/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/http_to_stdout.conf -r
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2022-12-15 19:15:41.202 [main] runner - Starting Logstash {"logstash.version"=>"7.15.0", "jruby.version"=>"jruby 9.2.19.0 (2.5.8) 2021-06-15 55810c552b OpenJDK 64-Bit Server VM 11.0.11+9 on 11.0.11+9 +indy +jit [linux-x86_64]"}
[WARN ] 2022-12-15 19:15:42.226 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2022-12-15 19:15:45.286 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[INFO ] 2022-12-15 19:15:46.235 [Converge PipelineAction::Create<main>] Reflections - Reflections took 175 ms to scan 1 urls, producing 120 keys and 417 values
[WARN ] 2022-12-15 19:15:47.648 [Converge PipelineAction::Create<main>] json - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2022-12-15 19:15:47.802 [Converge PipelineAction::Create<main>] http - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[INFO ] 2022-12-15 19:15:48.709 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["/etc/logstash/conf.d/http_to_stdout.conf"], :thread=>"#<Thread:0x4fc6d834 run>"}
[INFO ] 2022-12-15 19:15:50.362 [[main]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>1.65}
[WARN ] 2022-12-15 19:15:50.446 [[main]-pipeline-manager] json - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[INFO ] 2022-12-15 19:15:50.698 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[INFO ] 2022-12-15 19:15:50.766 [[main]<http] http - Starting http input listener {:address=>"0.0.0.0:6666", :ssl=>"false"}
[INFO ] 2022-12-15 19:15:51.022 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[WARN ] 2022-12-15 19:16:10.932 [http-input-processor[T#2]] json - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2022-12-15 19:16:10.954 [http-input-processor[T#2]] json - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2022-12-15 19:16:46.718 [http-input-processor[T#1]] json - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2022-12-15 19:16:46.725 [http-input-processor[T#1]] json - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[ERROR] 2022-12-15 19:16:46.835 [http-input-handler-executor[T#2]] json - JSON parse error, original data now in message field {:message=>"Unrecognized token 'test': was expecting 'null', 'true', 'false' or NaN\n at [Source: (String)\"test log message\"; line: 1, column: 5]", :exception=>LogStash::Json::ParserError, :data=>"test log message"}
{
"host" => "10.0.0.209",
"tags" => [
[0] "_jsonparsefailure"
],
"message" => "test log message",
"headers" => {
"content_type" => "application/x-www-form-urlencoded",
"request_path" => "/",
"http_version" => "HTTP/1.1",
"request_method" => "POST",
"http_accept" => "*/*",
"http_host" => "10.0.0.207:6666",
"http_user_agent" => "curl/7.68.0",
"content_length" => "16"
},
"@version" => "1",
"@timestamp" => 2022-12-15T11:16:46.868Z
}从 Filebeat 读取数据
[root@ES-Node2 ~]#vim /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/nginx/access_json.log
json.keys_under_root: true #默认False会将json数据存储至message，改为true则会独立message外存储
json.overwrite_keys: true #设为true,覆盖默认的message字段，使用自定义json格式中的key
tags: ["nginx-access"]
- type: log
enabled: true
paths:
- /var/log/nginx/error.log
tags: ["nginx-error"]
output.logstash:
[root@ubuntu2004 ~]#vim /etc/logstash/conf.d/filebeat_to_stdout.conf
input {
beats {
port => 5044
}
}
output {
stdout {
codec => rubydebug
}
}
[root@ubuntu2004 ~]#/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/filebeat_to_stdout.conf
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2022-12-15 19:20:52.319 [main] runner - Starting Logstash {"logstash.version"=>"7.15.0", "jruby.version"=>"jruby 9.2.19.0 (2.5.8) 2021-06-15 55810c552b OpenJDK 64-Bit Server VM 11.0.11+9 on 11.0.11+9 +indy +jit [linux-x86_64]"}
[WARN ] 2022-12-15 19:20:53.337 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2022-12-15 19:20:56.482 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[INFO ] 2022-12-15 19:20:57.673 [Converge PipelineAction::Create<main>] Reflections - Reflections took 251 ms to scan 1 urls, producing 120 keys and 417 values
[WARN ] 2022-12-15 19:20:58.971 [Converge PipelineAction::Create<main>] plain - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2022-12-15 19:20:59.116 [Converge PipelineAction::Create<main>] beats - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[INFO ] 2022-12-15 19:20:59.973 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["/etc/logstash/conf.d/filebeat_to_stdout.conf"], :thread=>"#<Thread:0x6cf459b9 run>"}
[INFO ] 2022-12-15 19:21:01.624 [[main]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>1.64}
[INFO ] 2022-12-15 19:21:01.691 [[main]-pipeline-manager] beats - Starting input listener {:address=>"0.0.0.0:5044"}
[INFO ] 2022-12-15 19:21:01.744 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[INFO ] 2022-12-15 19:21:01.915 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[INFO ] 2022-12-15 19:21:02.165 [[main]<beats] Server - Starting server on port: 5044
[WARN ] 2022-12-15 19:21:45.107 [defaultEventExecutorGroup-4-1] plain - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
{
"http_user_agent" => "curl/7.68.0",
"domain" => "10.0.0.209",
"agent" => {
"name" => "ES-Node2.com",
"hostname" => "ES-Node2.com",
"version" => "7.15.0",
"id" => "f87dddf0-89bf-46c9-be70-0f8e1784eb2a",
"ephemeral_id" => "af027170-43be-4bb3-9c72-fff47cb94983",
"type" => "filebeat"
},
"host" => {
"name" => "ES-Node2.com"
},
"status" => "200",
"input" => {
"type" => "log"
},
"size" => 10918,
"clientip" => "10.0.0.209",
"tags" => [
[0] "nginx-access",
[1] "beats_input_raw_event"
],
"upstreamtime" => "-",
"uri" => "/index.html",
"log" => {
"file" => {
"path" => "/var/log/nginx/access_json.log"
},
"offset" => 9343
},
"http_host" => "10.0.0.209",
"referer" => "-",
"@timestamp" => 2022-12-15T11:21:41.000Z,
"responsetime" => 0,
"upstreamhost" => "-",
"xff" => "-",
"ecs" => {
"version" => "1.11.0"
},
"tcp_xff" => "-",
"@version" => "1"
}
{
"domain" => "10.0.0.209",
"agent" => {
"name" => "ES-Node2.com",
"hostname" => "ES-Node2.com",
"version" => "7.15.0",
"ephemeral_id" => "af027170-43be-4bb3-9c72-fff47cb94983",
"id" => "f87dddf0-89bf-46c9-be70-0f8e1784eb2a",
"type" => "filebeat"
},
"http_user_agent" => "curl/7.68.0",
"host" => {
"name" => "ES-Node2.com"
},
"status" => "200",
"input" => {
"type" => "log"
},
"size" => 10918,
"clientip" => "10.0.0.209",
"tags" => [
[0] "nginx-access",
[1] "beats_input_raw_event"
],
"upstreamtime" => "-",
"log" => {
"file" => {
"path" => "/var/log/nginx/access_json.log"
},
"offset" => 9032
},
"uri" => "/index.html",
"http_host" => "10.0.0.209",
"referer" => "-",
"@timestamp" => 2022-12-15T11:21:40.000Z,
"responsetime" => 0,
"upstreamhost" => "-",
"xff" => "-",
"ecs" => {
"version" => "1.11.0"
},
"tcp_xff" => "-",
"@version" => "1"
}
{
"http_user_agent" => "curl/7.68.0",
"domain" => "10.0.0.209",
"agent" => {
"name" => "ES-Node2.com",
"hostname" => "ES-Node2.com",
"version" => "7.15.0",
"id" => "f87dddf0-89bf-46c9-be70-0f8e1784eb2a",
"ephemeral_id" => "af027170-43be-4bb3-9c72-fff47cb94983",
"type" => "filebeat"
},
"status" => "200",
"host" => {
"name" => "ES-Node2.com"
},
"input" => {
"type" => "log"
},
"size" => 10918,
"clientip" => "10.0.0.209",
"tags" => [
[0] "nginx-access",
[1] "beats_input_raw_event"
],
"upstreamtime" => "-",
"log" => {
"file" => {
"path" => "/var/log/nginx/access_json.log"
},
"offset" => 9654
},
"uri" => "/index.html",
"http_host" => "10.0.0.209",
"referer" => "-",
"@timestamp" => 2022-12-15T11:21:41.000Z,
"responsetime" => 0,
"upstreamhost" => "-",
"xff" => "-",
"ecs" => {
"version" => "1.11.0"
},
"tcp_xff" => "-",
"@version" => "1"
}从 Redis 中读取数据
支持由多个 Logstash 从 Redis 读取日志,提高性能
Logstash 从 Redis 收集完数据后,将删除对应的列表Key
官方链接:https://www.elastic.co/guide/en/logstash/7.6/plugins-inputs-redis.html
范例:
[root@ES-Node2 ~]#vim /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/nginx/access_json.log
json.keys_under_root: true #默认False会将json数据存储至message，改为true则会独立message外存储
json.overwrite_keys: true #设为true,覆盖默认的message字段，使用自定义json格式中的key
tags: ["nginx-access"]
- type: log
enabled: true
paths:
- /var/log/nginx/error.log
tags: ["nginx-error"]
output.redis:
hosts: ["10.0.0.209:6379"]
key: "filebeat"
#password: "123456"
#db: 0
[root@ubuntu2004 ~]#vim /etc/logstash/conf.d/redis_to_stdout.conf
input {
redis {
host => '10.0.0.209'
port => "6379"
#password => "123456"
db => "0"
data_type => 'list'
key => "filebeat"
}
}
output {
stdout {
codec => rubydebug
}
}
[root@ubuntu2004 ~]#/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/redis_to_stdout.conf
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2022-12-15 19:26:30.762 [main] runner - Starting Logstash {"logstash.version"=>"7.15.0", "jruby.version"=>"jruby 9.2.19.0 (2.5.8) 2021-06-15 55810c552b OpenJDK 64-Bit Server VM 11.0.11+9 on 11.0.11+9 +indy +jit [linux-x86_64]"}
[WARN ] 2022-12-15 19:26:31.698 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2022-12-15 19:26:35.085 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[INFO ] 2022-12-15 19:26:36.367 [Converge PipelineAction::Create<main>] Reflections - Reflections took 181 ms to scan 1 urls, producing 120 keys and 417 values
[WARN ] 2022-12-15 19:26:37.752 [Converge PipelineAction::Create<main>] json - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[INFO ] 2022-12-15 19:26:38.491 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["/etc/logstash/conf.d/redis_to_stdout.conf"], :thread=>"#<Thread:0x6121e0d9 run>"}
[INFO ] 2022-12-15 19:26:40.049 [[main]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>1.55}
[INFO ] 2022-12-15 19:26:40.093 [[main]-pipeline-manager] redis - Registering Redis {:identity=>"redis://@10.0.0.209:6379/0 list:filebeat"}
[INFO ] 2022-12-15 19:26:40.108 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[INFO ] 2022-12-15 19:26:40.362 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
{
"@version" => "1",
"agent" => {
"hostname" => "ES-Node2.com",
"name" => "ES-Node2.com",
"version" => "7.15.0",
"ephemeral_id" => "b31fd668-dfa4-422f-a810-45274d76e528",
"id" => "f87dddf0-89bf-46c9-be70-0f8e1784eb2a",
"type" => "filebeat"
},
"tags" => [
[0] "nginx-access"
],
"responsetime" => 0,
"size" => 10918,
"http_user_agent" => "curl/7.68.0",
"upstreamhost" => "-",
"status" => "200",
"domain" => "10.0.0.209",
"host" => {
"name" => "ES-Node2.com"
},
"upstreamtime" => "-",
"ecs" => {
"version" => "1.11.0"
},
"input" => {
"type" => "log"
},
"@timestamp" => 2022-12-15T09:28:37.000Z,
"http_host" => "10.0.0.209",
"referer" => "-",
"uri" => "/index.html",
"clientip" => "10.0.0.209",
"log" => {
"offset" => 8410,
"file" => {
"path" => "/var/log/nginx/access_json.log"
}
},
"tcp_xff" => "-",
"xff" => "-"
}
{
"@version" => "1",
"tags" => [
[0] "nginx-access"
],
"agent" => {
"hostname" => "ES-Node2.com",
"name" => "ES-Node2.com",
"version" => "7.15.0",
"id" => "f87dddf0-89bf-46c9-be70-0f8e1784eb2a",
"ephemeral_id" => "b31fd668-dfa4-422f-a810-45274d76e528",
"type" => "filebeat"
},
"responsetime" => 0,
"size" => 10918,
"http_user_agent" => "curl/7.68.0",
"upstreamhost" => "-",
"status" => "200",
"domain" => "10.0.0.209",
"host" => {
"name" => "ES-Node2.com"
},
"upstreamtime" => "-",
"ecs" => {
"version" => "1.11.0"
},
"input" => {
"type" => "log"
},
"@timestamp" => 2022-12-15T09:28:35.000Z,
"http_host" => "10.0.0.209",
"referer" => "-",
"uri" => "/index.html",
"clientip" => "10.0.0.209",
"log" => {
"offset" => 8099,
"file" => {
"path" => "/var/log/nginx/access_json.log"
}
},
"tcp_xff" => "-",
"xff" => "-"
}
{
"@version" => "1",
"tags" => [
[0] "nginx-access"
],
"agent" => {
"hostname" => "ES-Node2.com",
"name" => "ES-Node2.com",
"version" => "7.15.0",
"ephemeral_id" => "b31fd668-dfa4-422f-a810-45274d76e528",
"id" => "f87dddf0-89bf-46c9-be70-0f8e1784eb2a",
"type" => "filebeat"
},
"responsetime" => 0,
"size" => 10918,
"http_user_agent" => "curl/7.68.0",
"upstreamhost" => "-",
"domain" => "10.0.0.209",
"status" => "200",
"host" => {
"name" => "ES-Node2.com"
},
"upstreamtime" => "-",
"ecs" => {
"version" => "1.11.0"
},
"input" => {
"type" => "log"
},
"@timestamp" => 2022-12-15T09:28:48.000Z,
"http_host" => "10.0.0.209",
"referer" => "-",
"uri" => "/index.html",
"clientip" => "10.0.0.210",
"log" => {
"offset" => 8721,
"file" => {
"path" => "/var/log/nginx/access_json.log"
}
},
"tcp_xff" => "-",
"xff" => "-"
}从 Kafka 中读取数据
官方链接:https://www.elastic.co/guide/en/logstash/7.6/plugins-inputs-kafka.html
cat /etc/logstash/conf.d/kakfa_to_stdout.conf
input {
kafka {
bootstrap_servers => "10.0.0.201:9092,10.0.0.202:9092,10.0.0.203:9092"
group_id => "logstash"
topics => ["nginx-accesslog","nginx-errorlog"]
codec => "json"
consumer_threads => 8
}
}
output {
stdout {
codec => rubydebug
}
}Logstash 过滤 Filter 插件
数据从源传输到存储库的过程中，Logstash 过滤器能够解析各个事件，识别已命名的字段以构建结构，并将它们转换成通用格式，以便进行更强大的分析和实现商业价值。
Logstash 能够动态地转换和解析数据，不受格式或复杂度的影响
常见的 Filter 插件:
利用 Grok 从非结构化数据中派生出结构
从 IP 地址破译出地理坐标
利用 useragent 从请求中分析操作系统、设备类型
简化整体处理，不受数据源、格式或架构的影响
官方链接:https://www.elastic.co/guide/en/logstash/7.6/filter-plugins.html
Grok 插件
Grok 是一个过滤器插件，可帮助您描述日志格式的结构。有超过200种 grok模式抽象概念，如IPv6地址，UNIX路径和月份名称。为了将行与格式匹配
生产环境常需要将如下非结构化的数据解析成 json 结构化数据格式
2016-09-19T18:19:00 [8.8.8.8:prd] DEBUG this is an example log message
使用 Grok 插件可以基于正则表达式技术用内置的正则表达式的别名来表示和匹配上面的日志
%{TIMESTAMP_ISO8601:timestamp} \[%{IPV4:ip};%{WORD:environment}\] % {LOGLEVEL:log_level} %{GREEDYDATA:message}
最终转换为以下格式
{
"timestamp": "2016-09-19T18:19:00",
"ip": "8.8.8.8",
"environment": "prd",
"log_level": "DEBUG",
"message": "this is an example log message"
}参考网站
https://www.elastic.co/cn/blog/do-you-grok-grok
http://grokdebug.herokuapp.com/
http://grokdebug.herokuapp.com/discover?#范例: 利用grokedebug 网站将nginx日志自动生成grok的内置格式代码
159.65.4.253 - - [28/Aug/2022:09:32:05 +0800] "POST /xmlrpc.php HTTP/1.1" 200 412 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36"%{COMBINEDAPACHELOG}或者使作Kibana 实现上面功能
范例：使用 grok pattern 将 Nginx 日志格式化为 json 格式
cat /etc/logstash/conf.d/http_grok_stdout.conf
input {
http {
port =>6666
}
}
filter {
#将nginx日志格式化为json格式
grok {
match => {
"message" => "%{COMBINEDAPACHELOG}" #将message字段转化为指定的Json格式
}
}
}
output {
stdout {
codec => rubydebug
}
}
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/http_grok_stdout.conf -rGeoip 插件
geoip 根据 ip 地址提供的对应地域信息，比如:经纬度,国家,城市名等,以方便进行地理数据分析
范例:
[root@ubuntu2004 ~]#vim /etc/logstash/conf.d/http_geoip_stdout.conf
input {
input {
http {
port =>6666
}
}
filter {
#将nginx日志格式化为json格式
grok {
match => {
"message" => "%{COMBINEDAPACHELOG}"
}
}
#以上面提取clientip字段为源,获取地域信息
geoip {
source => "clientip" #指定源IP的所在字段
}
}
output {
stdout {
codec => rubydebug
}
}
[root@ubuntu2004 ~]#/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/http_geoip_stdout.conf -r
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2022-12-16 10:14:36.537 [main] runner - Starting Logstash {"logstash.version"=>"7.15.0", "jruby.version"=>"jruby 9.2.19.0 (2.5.8) 2021-06-15 55810c552b OpenJDK 64-Bit Server VM 11.0.11+9 on 11.0.11+9 +indy +jit [linux-x86_64]"}
[WARN ] 2022-12-16 10:14:37.521 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2022-12-16 10:14:40.223 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[INFO ] 2022-12-16 10:14:41.840 [Converge PipelineAction::Create<main>] Reflections - Reflections took 237 ms to scan 1 urls, producing 120 keys and 417 values
[WARN ] 2022-12-16 10:14:43.558 [Converge PipelineAction::Create<main>] plain - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2022-12-16 10:14:43.752 [Converge PipelineAction::Create<main>] http - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2022-12-16 10:14:43.966 [Converge PipelineAction::Create<main>] geoip - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2022-12-16 10:14:44.410 [[main]-pipeline-manager] grok - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2022-12-16 10:14:47.438 [[main]-pipeline-manager] LazyDelegatingGauge - A gauge metric of an unknown type (org.jruby.RubySymbol) has been created for key: status. This may result in invalid serialization. It is recommended to log an issue to the responsible developer/development team.
[WARN ] 2022-12-16 10:14:47.442 [[main]-pipeline-manager] LazyDelegatingGauge - A gauge metric of an unknown type (org.jruby.RubySymbol) has been created for key: status. This may result in invalid serialization. It is recommended to log an issue to the responsible developer/development team.
[WARN ] 2022-12-16 10:14:47.495 [[main]-pipeline-manager] LazyDelegatingGauge - A gauge metric of an unknown type (org.jruby.RubySymbol) has been created for key: status. This may result in invalid serialization. It is recommended to log an issue to the responsible developer/development team.
[INFO ] 2022-12-16 10:14:51.285 [[main]-pipeline-manager] downloadmanager - new database version detected? true
[INFO ] 2022-12-16 10:15:44.532 [[main]-pipeline-manager] databasemanager - By not manually configuring a database path with database =>, you accepted and agreed MaxMind EULA. For more details please visit https://www.maxmind.com/en/geolite2/eula
[INFO ] 2022-12-16 10:15:44.539 [[main]-pipeline-manager] geoip - Using geoip database {:path=>"/usr/share/logstash/data/plugins/filters/geoip/1671156887/GeoLite2-City.mmdb"}
[INFO ] 2022-12-16 10:15:44.809 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["/etc/logstash/conf.d/http_geoip_stdout.conf"], :thread=>"#<Thread:0x72cea342 run>"}
[INFO ] 2022-12-16 10:15:46.560 [[main]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>1.74}
[WARN ] 2022-12-16 10:15:46.719 [[main]-pipeline-manager] json - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[INFO ] 2022-12-16 10:15:47.040 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[INFO ] 2022-12-16 10:15:47.134 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[INFO ] 2022-12-16 10:15:47.153 [[main]<http] http - Starting http input listener {:address=>"0.0.0.0:6666", :ssl=>"false"}
[WARN ] 2022-12-16 10:16:11.383 [http-input-processor[T#2]] plain - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2022-12-16 10:16:11.409 [http-input-processor[T#2]] json - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
{
"timestamp" => "27/Aug/2022:19:26:49 +0800",
"agent" => "\"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:104.0) Gecko/20100101 Firefox/104.0\"",
"geoip" => {
"longitude" => 116.3861,
"city_name" => "Beijing",
"ip" => "115.171.63.144",
"location" => {
"lat" => 39.9143,
"lon" => 116.3861
},
"country_code2" => "CN",
"region_code" => "BJ",
"region_name" => "Beijing",
"country_code3" => "CN",
"country_name" => "China",
"timezone" => "Asia/Shanghai",
"latitude" => 39.9143,
"continent_code" => "AS"
},
"host" => "10.0.0.210",
"ident" => "-",
"response" => "301",
"@version" => "1",
"clientip" => "115.171.63.144",
"@timestamp" => 2022-12-16T02:16:11.558Z,
"message" => "115.171.63.144 - - [27/Aug/2022:19:26:49 +0800] \"GET /favicon.ico HTTP/1.1\" 301 5 \"http://119.91.212.247/\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:104.0) Gecko/20100101 Firefox/104.0\"",
"headers" => {
"http_user_agent" => "curl/7.68.0",
"http_version" => "HTTP/1.1",
"request_path" => "/",
"request_method" => "POST",
"content_length" => "189",
"http_host" => "10.0.0.207:6666",
"content_type" => "application/x-www-form-urlencoded",
"http_accept" => "*/*"
},
"auth" => "-",
"bytes" => "5",
"httpversion" => "1.1",
"referrer" => "\"http://119.91.212.247/\"",
"verb" => "GET",
"request" => "/favicon.ico"
}
#使用curl命令通过http方式输入,输出上面信息
[root@ES-Node3 ~]#curl -XPOST -d '115.171.63.144 - - [27/Aug/2022:19:26:49 +0800] "GET /favicon.ico HTTP/1.1" 301 5 "http://119.91.212.247/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:104.0) Gecko/20100101 Firefox/104.0"' http://10.0.0.207:6666
ok
Date 插件
date插件可以将日志中的日期字符串解析为日志类型。然后替换@timestamp 字段(此字段默认为当前写放logstash的时间而非日志本身的时间)或指定的其他字段
match 类型为数组，用于指定日期匹配的格式，可以以此指定多种日期格式
target 类型为字符串，用于指定赋值的字段名，默认是 @timestamp
timezone 类型为字符串，用于指定时区域
官方说明
https://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html
时区格式参考
http://joda-time.sourceforge.net/timezones.html
[root@ubuntu2004 ~]#vim /etc/logstash/conf.d/http_grok_date_stdout.conf
input {
http {
port =>6666
}
}
filter {
#将nginx日志格式化为json格式
grok {
match => {
"message" => "%{COMBINEDAPACHELOG}"
}
}
#解析date日期格式为: 14/Jul/2020:15:07:27 +0800,
date {
match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
target => "access_time" #将时间写入新生成的access_time字段
#target => "@timestamp" #将时间覆盖原有的@timestamp字段
timezone => "Asia/Shanghai"
}
}
output {
stdout {
codec => rubydebug
}
}
[root@ubuntu2004 ~]#/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/http_grok_date_stdout.conf -r
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2022-12-16 10:23:17.776 [main] runner - Starting Logstash {"logstash.version"=>"7.15.0", "jruby.version"=>"jruby 9.2.19.0 (2.5.8) 2021-06-15 55810c552b OpenJDK 64-Bit Server VM 11.0.11+9 on 11.0.11+9 +indy +jit [linux-x86_64]"}
[WARN ] 2022-12-16 10:23:18.782 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2022-12-16 10:23:22.036 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[INFO ] 2022-12-16 10:23:23.666 [Converge PipelineAction::Create<main>] Reflections - Reflections took 178 ms to scan 1 urls, producing 120 keys and 417 values
[WARN ] 2022-12-16 10:23:25.192 [Converge PipelineAction::Create<main>] plain - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2022-12-16 10:23:25.286 [Converge PipelineAction::Create<main>] http - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2022-12-16 10:23:26.141 [[main]-pipeline-manager] grok - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[INFO ] 2022-12-16 10:23:26.857 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["/etc/logstash/conf.d/http_grok_date_stdout.conf"], :thread=>"#<Thread:0x5d15a961 run>"}
[INFO ] 2022-12-16 10:23:28.647 [[main]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>1.79}
[WARN ] 2022-12-16 10:23:28.830 [[main]-pipeline-manager] json - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[INFO ] 2022-12-16 10:23:29.127 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[INFO ] 2022-12-16 10:23:29.152 [[main]<http] http - Starting http input listener {:address=>"0.0.0.0:6666", :ssl=>"false"}
[INFO ] 2022-12-16 10:23:29.264 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[WARN ] 2022-12-16 10:23:36.853 [http-input-processor[T#2]] plain - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2022-12-16 10:23:36.872 [http-input-processor[T#2]] json - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
{
"auth" => "-",
"@version" => "1",
"host" => "10.0.0.210",
"response" => "200",
"agent" => "\"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:104.0) Gecko/20100101 Firefox/104.0\"",
"@timestamp" => 2022-12-16T02:23:37.061Z,
"verb" => "GET",
"bytes" => "1755",
"message" => "220.196.160.101 - - [27/Aug/2022:20:47:15 +0800] \"GET /wp-includes/js/dist/hooks.min.js?ver=c6d64f2cb8f5c6bb49caca37f8828ce3 HTTP/1.1\" 200 1755 \"https://www.shuzihan.com/wp-login.php?redirect_to=http%3A%2F%2F119.91.212.247%2Fwp-admin%2Foptions-permalink.php&reauth=1\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:104.0) Gecko/20100101 Firefox/104.0\"",
"request" => "/wp-includes/js/dist/hooks.min.js?ver=c6d64f2cb8f5c6bb49caca37f8828ce3",
"timestamp" => "27/Aug/2022:20:47:15 +0800",
"httpversion" => "1.1",
"headers" => {
"http_user_agent" => "curl/7.68.0",
"content_type" => "application/x-www-form-urlencoded",
"request_path" => "/",
"http_version" => "HTTP/1.1",
"request_method" => "POST",
"content_length" => "349",
"http_host" => "10.0.0.207:6666",
"http_accept" => "*/*"
},
"clientip" => "220.196.160.101",
"ident" => "-",
"referrer" => "\"https://www.shuzihan.com/wp-login.php?redirect_to=http%3A%2F%2F119.91.212.247%2Fwp-admin%2Foptions-permalink.php&reauth=1\"",
"access_time" => 2022-08-27T12:47:15.000Z
}
[root@ES-Node3 ~]#curl -XPOST -d '220.196.160.101 - - [27/Aug/2022:20:47:15 +0800] "GET /wp-includes/js/dist/hooks.min.js?ver=c6d64f2cb8f5c6bb49caca37f8828ce3 HTTP/1.1" 200 1755 "https://www.shuzihan.com/wp-login.php?redirect_to=http%3A%2F%2F119.91.212.247%2Fwp-admin%2Foptions-permalink.php&reauth=1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:104.0) Gecko/20100101 Firefox/104.0"' http://10.0.0.207:6666
okUseragent 插件
[root@ubuntu2004 ~]#vim /etc/logstash/conf.d/http_grok_useragent_stdout.conf
input {
http {
port =>6666
}
}
filter {
#将nginx日志格式化为json格式
grok {
match => {
"message" => "%{COMBINEDAPACHELOG}"
}
}
#解析date日期如: 10/Dec/2020:10:40:10 +0800
date {
match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
target => "@timestamp"
#target => "access_time"
timezone => "Asia/Shanghai"
}
#提取agent字段，进行解析
useragent {
source => "agent" #指定从哪个字段获取数据
target => "useragent" #转换后的新字段
}
}
output {
stdout {
codec => rubydebug
}
}
[root@ubuntu2004 ~]#/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/http_grok_useragent_stdout.conf -r
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2022-12-16 10:26:42.929 [main] runner - Starting Logstash {"logstash.version"=>"7.15.0", "jruby.version"=>"jruby 9.2.19.0 (2.5.8) 2021-06-15 55810c552b OpenJDK 64-Bit Server VM 11.0.11+9 on 11.0.11+9 +indy +jit [linux-x86_64]"}
[WARN ] 2022-12-16 10:26:43.982 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2022-12-16 10:26:47.256 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[INFO ] 2022-12-16 10:26:48.743 [Converge PipelineAction::Create<main>] Reflections - Reflections took 179 ms to scan 1 urls, producing 120 keys and 417 values
[WARN ] 2022-12-16 10:26:50.128 [Converge PipelineAction::Create<main>] plain - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2022-12-16 10:26:50.291 [Converge PipelineAction::Create<main>] http - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2022-12-16 10:26:50.696 [Converge PipelineAction::Create<main>] useragent - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2022-12-16 10:26:51.128 [[main]-pipeline-manager] grok - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[INFO ] 2022-12-16 10:26:52.090 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["/etc/logstash/conf.d/http_grok_useragent_stdout.conf"], :thread=>"#<Thread:0x6863bdd run>"}
[INFO ] 2022-12-16 10:26:54.002 [[main]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>1.9}
[WARN ] 2022-12-16 10:26:54.236 [[main]-pipeline-manager] json - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[INFO ] 2022-12-16 10:26:54.481 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[INFO ] 2022-12-16 10:26:54.549 [[main]<http] http - Starting http input listener {:address=>"0.0.0.0:6666", :ssl=>"false"}
[INFO ] 2022-12-16 10:26:54.730 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[WARN ] 2022-12-16 10:27:39.125 [http-input-processor[T#2]] plain - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2022-12-16 10:27:39.143 [http-input-processor[T#2]] json - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
{
"headers" => {
"http_host" => "10.0.0.207:6666",
"request_path" => "/",
"content_length" => "349",
"content_type" => "application/x-www-form-urlencoded",
"request_method" => "POST",
"http_accept" => "*/*",
"http_user_agent" => "curl/7.68.0",
"http_version" => "HTTP/1.1"
},
"host" => "10.0.0.210",
"verb" => "GET",
"bytes" => "1755",
"referrer" => "\"https://www.shuzihan.com/wp-login.php?redirect_to=http%3A%2F%2F119.91.212.247%2Fwp-admin%2Foptions-permalink.php&reauth=1\"",
"@version" => "1",
"timestamp" => "27/Aug/2022:20:47:15 +0800",
"request" => "/wp-includes/js/dist/hooks.min.js?ver=c6d64f2cb8f5c6bb49caca37f8828ce3",
"clientip" => "220.196.160.101",
"ident" => "-",
"@timestamp" => 2022-08-27T12:47:15.000Z,
"agent" => "\"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:104.0) Gecko/20100101 Firefox/104.0\"",
"auth" => "-",
"message" => "220.196.160.101 - - [27/Aug/2022:20:47:15 +0800] \"GET /wp-includes/js/dist/hooks.min.js?ver=c6d64f2cb8f5c6bb49caca37f8828ce3 HTTP/1.1\" 200 1755 \"https://www.shuzihan.com/wp-login.php?redirect_to=http%3A%2F%2F119.91.212.247%2Fwp-admin%2Foptions-permalink.php&reauth=1\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:104.0) Gecko/20100101 Firefox/104.0\"",
"response" => "200",
"httpversion" => "1.1",
"useragent" => {
"os_name" => "Windows",
"os_major" => "10",
"name" => "Firefox",
"device" => "Other",
"minor" => "0",
"os_version" => "10",
"version" => "104.0",
"major" => "104",
"os" => "Windows",
"os_full" => "Windows 10"
}
}
[WARN ] 2022-12-16 10:28:19.371 [http-input-processor[T#1]] plain - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2022-12-16 10:28:19.397 [http-input-processor[T#1]] json - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
{
"headers" => {
"http_host" => "10.0.0.207:6666",
"request_path" => "/",
"content_length" => "206",
"content_type" => "application/x-www-form-urlencoded",
"request_method" => "POST",
"http_accept" => "*/*",
"http_user_agent" => "curl/7.68.0",
"http_version" => "HTTP/1.1"
},
"host" => "10.0.0.210",
"verb" => "POST",
"bytes" => "412",
"referrer" => "\"-\"",
"@version" => "1",
"timestamp" => "07/Sep/2022:00:38:47 +0800",
"request" => "/xmlrpc.php",
"clientip" => "2220.204.174.94",
"ident" => "-",
"@timestamp" => 2022-09-06T16:38:47.000Z,
"agent" => "\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36\"",
"auth" => "-",
"message" => "2220.204.174.94 - - [07/Sep/2022:00:38:47 +0800] \"POST /xmlrpc.php HTTP/1.1\" 200 412 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36\"",
"response" => "200",
"httpversion" => "1.1",
"useragent" => {
"os_major" => "10",
"name" => "Chrome",
"minor" => "0",
"os_version" => "10",
"major" => "78",
"os_name" => "Windows",
"patch" => "3904",
"device" => "Other",
"os_full" => "Windows 10",
"os" => "Windows",
"version" => "78.0.3904.108"
}
}
[root@ES-Node3 ~]#curl -XPOST -d '220.196.160.101 - - [27/Aug/2022:20:47:15 +0800] "GET /wp-includes/js/dist/hooks.min.js?ver=c6d64f2cb8f5c6bb49caca37f8828ce3 HTTP/1.1" 200 1755 "https://www.shuzihan.com/wp-login.php?redirect_to=http%3A%2F%2F119.91.212.247%2Fwp-admin%2Foptions-permalink.php&reauth=1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:104.0) Gecko/20100101 Firefox/104.0"' http://10.0.0.207:6666
ok
[root@ES-Node3 ~]#]#curl -XPOST -d '20.204.174.94 - - [07/Sep/2022:00:38:47 +0800] "POST /xmlrpc.php HTTP/1.1" 200 412 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36"' http://10.0.0.207:6666
okMutate 插件
Mutate 插件主要是对字段进行、类型转换、删除、替换、更新等操作,可以使用以下函数
remove_field 删除字段
split 字符串切割,相当于awk取列
add_field 添加字段
convert 类型转换
gsub 字符串替换
官方链接:https://www.elastic.co/guide/en/logstash/7.6/plugins-filters-mutate.html
remove_field 删除字段
[root@ubuntu2004 ~]#vim /etc/logstash/conf.d/http_grok_mutate_remove_field_stdout.conf
input {
http {
port =>6666
}
}
filter {
#将nginx日志格式化为json格式
grok {
match => {
"message" => "%{COMBINEDAPACHELOG}"
}
}
#解析date日期如: 10/Dec/2020:10:40:10 +0800
date {
match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
target => "@timestamp"
#target => "access_time"
timezone => "Asia/Shanghai"
}
#mutate 删除操作
mutate {
remove_field => ["headers","message", "agent"]
}
}
output {
stdout {
codec => rubydebug
}
}
[root@ubuntu2004 ~]#/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/http_grok_mutate_remove_field_stdout.conf -r
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2022-12-16 10:33:06.528 [main] runner - Starting Logstash {"logstash.version"=>"7.15.0", "jruby.version"=>"jruby 9.2.19.0 (2.5.8) 2021-06-15 55810c552b OpenJDK 64-Bit Server VM 11.0.11+9 on 11.0.11+9 +indy +jit [linux-x86_64]"}
[WARN ] 2022-12-16 10:33:07.452 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2022-12-16 10:33:10.645 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[INFO ] 2022-12-16 10:33:12.278 [Converge PipelineAction::Create<main>] Reflections - Reflections took 208 ms to scan 1 urls, producing 120 keys and 417 values
[WARN ] 2022-12-16 10:33:13.758 [Converge PipelineAction::Create<main>] plain - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2022-12-16 10:33:13.943 [Converge PipelineAction::Create<main>] http - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2022-12-16 10:33:14.796 [[main]-pipeline-manager] grok - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[INFO ] 2022-12-16 10:33:15.416 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["/etc/logstash/conf.d/http_grok_mutate_remove_field_stdout.conf"], :thread=>"#<Thread:0x2d88844e run>"}
[INFO ] 2022-12-16 10:33:17.255 [[main]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>1.83}
[WARN ] 2022-12-16 10:33:17.490 [[main]-pipeline-manager] json - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[INFO ] 2022-12-16 10:33:17.774 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[INFO ] 2022-12-16 10:33:17.858 [[main]<http] http - Starting http input listener {:address=>"0.0.0.0:6666", :ssl=>"false"}
[INFO ] 2022-12-16 10:33:18.081 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[WARN ] 2022-12-16 10:33:18.990 [http-input-processor[T#2]] plain - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2022-12-16 10:33:19.012 [http-input-processor[T#2]] json - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
{
"auth" => "-",
"bytes" => "412",
"referrer" => "\"-\"",
"verb" => "POST",
"@timestamp" => 2022-09-06T16:38:47.000Z,
"request" => "/xmlrpc.php",
"httpversion" => "1.1",
"response" => "200",
"clientip" => "2220.204.174.94",
"timestamp" => "07/Sep/2022:00:38:47 +0800",
"ident" => "-",
"@version" => "1",
"host" => "10.0.0.210"
}
[root@ES-Node3 ~]#curl -XPOST -d '2220.204.174.94 - - [07/Sep/2022:00:38:47 +0800] "POST /xmlrpc.php HTTP/1.1" 200 412 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36"' http://10.0.0.207:6666
okSplit 切割
mutate 中的 split 字符切割， 指定字段做为分隔符,生成新的字段名
示例: 1000|提交订单|2020-01-08 09:10:21
范例: split 切割字符串取列
[root@ubuntu2004 ~]#vim /etc/logstash/conf.d/http_grok_mutate_split_stdout.conf
input {
http {
port =>6666
}
}
filter {
#mutate 切割操作
mutate {
#字段分隔符
split => { "message" => "|" } #将message字段按 | 分割成多个列表元素
}
}
output {
stdout {
codec => rubydebug
}
}
[root@ubuntu2004 ~]#/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/http_grok_mutate_split_stdout.conf -r
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2022-12-16 10:38:18.667 [main] runner - Starting Logstash {"logstash.version"=>"7.15.0", "jruby.version"=>"jruby 9.2.19.0 (2.5.8) 2021-06-15 55810c552b OpenJDK 64-Bit Server VM 11.0.11+9 on 11.0.11+9 +indy +jit [linux-x86_64]"}
[WARN ] 2022-12-16 10:38:19.724 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2022-12-16 10:38:22.728 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[INFO ] 2022-12-16 10:38:23.949 [Converge PipelineAction::Create<main>] Reflections - Reflections took 152 ms to scan 1 urls, producing 120 keys and 417 values
[WARN ] 2022-12-16 10:38:25.209 [Converge PipelineAction::Create<main>] plain - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2022-12-16 10:38:25.374 [Converge PipelineAction::Create<main>] http - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[INFO ] 2022-12-16 10:38:26.354 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["/etc/logstash/conf.d/http_grok_mutate_split_stdout.conf"], :thread=>"#<Thread:0x1dc494ad run>"}
[INFO ] 2022-12-16 10:38:27.844 [[main]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>1.48}
[WARN ] 2022-12-16 10:38:28.041 [[main]-pipeline-manager] json - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[INFO ] 2022-12-16 10:38:28.330 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[INFO ] 2022-12-16 10:38:28.407 [[main]<http] http - Starting http input listener {:address=>"0.0.0.0:6666", :ssl=>"false"}
[INFO ] 2022-12-16 10:38:28.644 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[WARN ] 2022-12-16 10:38:29.766 [http-input-processor[T#2]] plain - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2022-12-16 10:38:29.794 [http-input-processor[T#2]] json - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
{
"@version" => "1",
"headers" => {
"http_accept" => "*/*",
"request_path" => "/",
"request_method" => "POST",
"http_host" => "10.0.0.207:6666",
"http_user_agent" => "curl/7.68.0",
"http_version" => "HTTP/1.1",
"content_type" => "application/x-www-form-urlencoded",
"content_length" => "37"
},
"message" => [
[0] "1000",
[1] "提交订单",
[2] "2020-01-08 09:10:21"
],
"@timestamp" => 2022-12-16T02:38:29.876Z,
"host" => "10.0.0.210"
}
[root@ES-Node3 ~]#curl -XPOST -d '1000|提交订单|2020-01-08 09:10:21' http://10.0.0.207:6666
okadd_field 添加字段
[root@ubuntu2004 ~]#vim /etc/logstash/conf.d/http_grok_mutate_add_field_stdout.conf
input {
http {
port =>6666
}
}
filter {
#mutate 切割操作
mutate {
#字段分隔符
split => { "message" => "|" }
#添加字段,将message的列表的第0个元素添加字段名user_id
add_field => {
"user_id" => "%{[message][0]}"
"action" => "%{[message][1]}"
"time" => "%{[message][2]}"
}
#add_field => {"[@metadata][target_index]" => "app-%{+YYY.MM.dd}"}#添加字段做索引名
#删除无用字段
remove_field => ["headers","message"]
}
}
output {
stdout {
codec => rubydebug
}
}
[root@ubuntu2004 ~]#/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/http_grok_mutate_add_field_stdout.conf -r
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2022-12-16 10:41:12.821 [main] runner - Starting Logstash {"logstash.version"=>"7.15.0", "jruby.version"=>"jruby 9.2.19.0 (2.5.8) 2021-06-15 55810c552b OpenJDK 64-Bit Server VM 11.0.11+9 on 11.0.11+9 +indy +jit [linux-x86_64]"}
[WARN ] 2022-12-16 10:41:13.957 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2022-12-16 10:41:17.046 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[INFO ] 2022-12-16 10:41:18.648 [Converge PipelineAction::Create<main>] Reflections - Reflections took 165 ms to scan 1 urls, producing 120 keys and 417 values
[WARN ] 2022-12-16 10:41:20.086 [Converge PipelineAction::Create<main>] plain - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2022-12-16 10:41:20.298 [Converge PipelineAction::Create<main>] http - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[INFO ] 2022-12-16 10:41:21.002 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["/etc/logstash/conf.d/http_grok_mutate_add_field_stdout.conf"], :thread=>"#<Thread:0x6f4db68c run>"}
[INFO ] 2022-12-16 10:41:22.504 [[main]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>1.5}
[WARN ] 2022-12-16 10:41:22.682 [[main]-pipeline-manager] json - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[INFO ] 2022-12-16 10:41:22.943 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[INFO ] 2022-12-16 10:41:23.061 [[main]<http] http - Starting http input listener {:address=>"0.0.0.0:6666", :ssl=>"false"}
[INFO ] 2022-12-16 10:41:23.178 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[WARN ] 2022-12-16 10:41:23.971 [http-input-processor[T#2]] plain - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2022-12-16 10:41:23.989 [http-input-processor[T#2]] json - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
{
"action" => "提交订单",
"@timestamp" => 2022-12-16T02:41:24.184Z,
"@version" => "1",
"host" => "10.0.0.210",
"time" => "2020-01-08 09:10:21",
"user_id" => "1000"
}
[root@ES-Node3 ~]#curl -XPOST -d '1000|提交订单|2020-01-08 09:10:21' http://10.0.0.207:6666
okconvert 转换
mutate 中的 convert 可以实现数据类型的转换。 支持转换integer、float、string等类型
[root@ubuntu2004 ~]#vim /etc/logstash/conf.d/http_grok_mutate_convert_stdout.conf
input {
http {
port =>6666
}
}
filter {
#mutate 切割操作
mutate {
#字段分隔符
split => { "message" => "|" }
#添加字段
add_field => {
"user_id" => "%{[message][0]}"
"action" => "%{[message][1]}"
"time" => "%{[message][2]}"
}
#删除无用字段
remove_field => ["headers","message"]
#对新添加字段进行格式转换
convert => {
"user_id" => "integer"
"action" => "string"
"time" => "string"
}
#convert => ["excute_time","float] #此格式也可以支持
#convert => ["time","string" ]
}
}
output {
stdout {
codec => rubydebug
}
}
[root@ubuntu2004 ~]#/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/http_grok_mutate_convert_stdout.conf -r
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2022-12-16 10:45:24.708 [main] runner - Starting Logstash {"logstash.version"=>"7.15.0", "jruby.version"=>"jruby 9.2.19.0 (2.5.8) 2021-06-15 55810c552b OpenJDK 64-Bit Server VM 11.0.11+9 on 11.0.11+9 +indy +jit [linux-x86_64]"}
[WARN ] 2022-12-16 10:45:25.651 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2022-12-16 10:45:28.952 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[INFO ] 2022-12-16 10:45:30.548 [Converge PipelineAction::Create<main>] Reflections - Reflections took 222 ms to scan 1 urls, producing 120 keys and 417 values
[WARN ] 2022-12-16 10:45:32.136 [Converge PipelineAction::Create<main>] plain - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2022-12-16 10:45:32.270 [Converge PipelineAction::Create<main>] http - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[INFO ] 2022-12-16 10:45:33.093 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["/etc/logstash/conf.d/http_grok_mutate_convert_stdout.conf"], :thread=>"#<Thread:0x2f53298f run>"}
[INFO ] 2022-12-16 10:45:34.974 [[main]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>1.88}
[WARN ] 2022-12-16 10:45:35.133 [[main]-pipeline-manager] json - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[INFO ] 2022-12-16 10:45:35.322 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[INFO ] 2022-12-16 10:45:35.371 [[main]<http] http - Starting http input listener {:address=>"0.0.0.0:6666", :ssl=>"false"}
[INFO ] 2022-12-16 10:45:35.447 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[WARN ] 2022-12-16 10:45:41.468 [http-input-processor[T#2]] plain - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2022-12-16 10:45:41.499 [http-input-processor[T#2]] json - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
{
"@version" => "1",
"user_id" => "1000",
"action" => "提交订单",
"host" => "10.0.0.210",
"time" => "2020-01-08 09:10:21",
"@timestamp" => 2022-12-16T02:45:41.682Z
}
[root@ES-Node3 ~]#curl -XPOST -d '1000|提交订单|2020-01-08 09:10:21' http://10.0.0.207:6666
okgsub 替换
filter {
mutate {
gsub=>["message","\n", " "] #将message字段中的换行替换为空格
}
}条件判断
#vim /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/nginx/access.log
tags: ["access"]
- type: log
enabled: true
paths:
- /var/log/nginx/error.log
tags: ["error"]
output.logstash:
hosts: ["10.0.0.104:5044","10.0.0.105:5044",]
#loadbalance: true #负载均衡
#worker: 2 #number of hosts * workers #开启多进程
#vim /etc/logstash/conf.d/filebeat_logstash_es.conf
input {
beats {
port => 5044
}
}
filter {
if "access" in [tags][0] {
mutate {
add_field => { "target_index" => "access-%{+YYYY.MM.dd}"}
}
}
else if "error" in [tags][0] {
mutate {
add_field => { "target_index" => "error-%{+YYYY.MM.dd}"}
}
}
else if "system" in [tags][0] {
mutate {
add_field => { "target_index" => "system-%{+YYYY.MM.dd}"}
}
}
}
output {
elasticsearch {
hosts =>["10.0.0.101:9200","10.0.0.102:9200","10.0.0.103:9200"] #一般写data地址
index => "%{[target_index]}" #使用字段target_index值做为索引名
template_overwrite => true #覆盖索引模板
}
}Logstash 输出 Output 插件
官方链接:https://www.elastic.co/guide/en/logstash/7.6/output-plugins.html
Stdout 插件
stdout 插件将数据输出到屏幕终端，主要用于调试
output {
stdout {
codec => rubydebug
}
}File 插件
输出到文件，可以将将分散在多个文件的数据统一存放到一个文件
示例: 将所有 web 机器的日志收集到一个文件中，从而方便统一管理
output {
stdout {
codec => rubydebug
}
file {
path => "/var/log/test.log"
}
}Elasticsearch 插件
官方说明:https://www.elastic.co/guide/en/logstash/7.6/plugins-outputs-elasticsearch.html
索引的时间格式说明:https://www.joda.org/joda-time/apidocs/org/joda/time/format/DateTimeFormat.html
当日志量较小时，可以按月或周生成索引，当日志量比较大时，会按天生成索引，以方便后续按天删除
output {
elasticsearch {
hosts =>["10.0.0.101:9200","10.0.0.102:9200","10.0.0.103:9200"]#一般写ES中data节点地址
index => "app-%{+YYYY.MM.dd}" #指定索引名称,建议加时间,按天建立索引
#index => "%{[@metadata][target_index]}"#使用字段[@metadata][target_index]值做为索引名
template_overwrite => true #覆盖索引模板,此项可选,默认值为false
}
}注意: 索引名必须为小写
Redis 插件
Logstash 支持将日志转发至 Redis
官方链接:https://www.elastic.co/guide/en/logstash/7.6/plugins-outputs-redis.html
output {
if [type] == 'nginx-accesslog' {
redis {
host => 'Redis_IP'
port => "6379"
password => "123456"
db => "0"
data_type => 'list'
key => "nginx-accesslog"
}
}
}Kafka 插件
Logstash 支持将日志转发至 Kafka
官方链接:https://www.elastic.co/guide/en/logstash/7.6/plugins-outputs-kafka.html
output {
#stdout {}
if [type] == 'nginx-accesslog' {
kafka {
bootstrap_servers => '10.0.0.201:9092,10.0.0.202:9092,10.0.0.203:9092'
topic_id => 'nginx-accesslog'
codec => 'json' #如果是Json格式,需要标识的字段
}
}
if [type] == 'nginx-errorlog' {
kafka {
bootstrap_servers => 'Kafka_IP1:9092,Kafka_IP2:9092,Kafka_IP3:9092'
topic_id => 'nginx-errorlog'
codec => 'json' #为了保留logstash添加的字段,比如:type字段,也需要指定json格式,否则会丢失logstash添加的字段
}
}
}