ELK Cluster architecture optimization

1. Cluster architecture extension

When the amount of data collected by the log reaches hundreds G, Then we need to introduce kafka There's a message queue ,kafka Processing message data ratio redis A lot of powerful , And support cluster mode .

When the equivalent level is very large , Query is slow , Then you can expand ES The number of nodes in the cluster ,ES It's a distributed cluster , It can easily complete the expansion and contraction of nodes .

If log data warehousing is slow , At this time, we need to add more logstash, At the same time, process the data , Speed up data warehousing .

2. About the optimization of log index library

Plan the index library according to the application , Try to avoid writing multiple applications into one index library , Affect query speed , It is better to have an application and a log index library , The advantage of this is that you can clean the log data of different applications anytime and anywhere , in addition , The name of the index library should preferably be named in days , You can clean up log data regularly through scripts , Avoid excessive disk usage .

Unused index libraries can be deleted or closed .

The command to close the index library is as follows , Both support wildcard forms .

# Close the index library 
curl -XPOST "http://127.0.0.1:9200/ The index name -*/_close?pretty"

# Open index library 
curl -XPOST "http://127.0.0.1:9200/ The index name -*/_open?pretty"

The command to delete the index library is as follows , Wildcard forms are supported .

curl -XDELETE "http://127.0.0.1:9200/ The index name -*"