Logstash Introduce ：

summary

Logstash yes Elastic Stack The central data flow engine , Used to collect 、 Enrich and unify all data , Regardless of format or mode . When and Elasticsearch,Kibana, And Beats When they are used together, they will have a very powerful real-time processing capability . In this video ,Elastic The technical evangelist Zeng Yong will know how to start Logstash Overview and demonstration .

Logstash It's a free and open server-side data processing pipeline , Ability to collect data from multiple sources , Conversion data , Then send the data to your favorite “ The repository ” in .

Logstash It can collect data dynamically 、 Converting and transferring data , Not affected by format or complexity . utilize Grok Derive structure from unstructured data , from IP The address decodes the geographic coordinates , Anonymous or exclude sensitive fields , And simplify the whole process .Logstash It mainly consists of three parts ：

• input： Get data from one or more data sources , Common plug-ins such as file、syslog、redis、beats etc. .
• filter： Used for data filtering 、 Format conversion, etc , Common plug-ins such as grok、mute、drop、geoip etc.
• output： Data output , Common plug-ins such as elastcisearch、file、statsd etc.

Logstash download ：

Tips ： Because it's using Ruby Written , So you need to install it first JAVA JDK, I won't elaborate on how to install it here JDK

Suggest kibana、es、logstash The versions of are consistent, otherwise compatibility problems will occur

• Download address of historical version
  

• Download address of the latest version of the official website
  

Logstash install ：

Decompressing the compressed package is very simple

$ tar -zxvf logstash-7.10.2-darwin-x86_64.tar.gz 

vim logstash-sample.conf 

# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.

input {
    
  beats {
    
    port => 5044
  }
}

output {
    
  elasticsearch {
    
    hosts => ["http://localhost:9200"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    #user => "elastic"
    #password => "changeme"
  }
}

Logstash start-up ：

Tips ： here -e It refers to starting and configuring through the command line

 $ logstash -e 'input { stdin {} } output { stdout {} }'
Using JAVA_HOME defined java: /Library/Java/JavaVirtualMachines/jdk1.8.0_291.jdk/Contents/Home
WARNING, using JAVA_HOME while Logstash distribution comes with a bundled JDK

Sending Logstash logs to /Users/xiaojialiang/module/logstash-7.10.2/logs which is now configured via log4j2.properties
[2022-07-18T19:24:33,639][INFO ][logstash.runner          ] Starting Logstash {
    "logstash.version"=>"7.10.2", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc Java HotSpot(TM) 64-Bit Server VM 25.291-b10 on 1.8.0_291-b10 +indy +jit [darwin-x86_64]"}
[2022-07-18T19:24:33,934][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2022-07-18T19:24:35,409][INFO ][org.reflections.Reflections] Reflections took 55 ms to scan 1 urls, producing 23 keys and 47 values 
[2022-07-18T19:24:36,585][INFO ][logstash.javapipeline    ][main] Starting pipeline {
    :pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, "pipeline.sources"=>["config string"], :thread=>"#<Thread:0x74a21a6f run>"}
[2022-07-18T19:24:37,477][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {
    "seconds"=>0.88}
[2022-07-18T19:24:37,520][INFO ][logstash.javapipeline    ][main] Pipeline started {
    "pipeline.id"=>"main"}
The stdin plugin is now waiting for input:
[2022-07-18T19:24:37,598][INFO ][logstash.agent           ] Pipelines running {
    :count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2022-07-18T19:24:37,970][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {
    :port=>9600}
{
    
       "message" => "",
      "@version" => "1",
    "@timestamp" => 2022-07-18T11:24:37.582Z,
          "host" => "xiaoDe-MacBook-Pro.local"
}