Know and use log tool Loki

Loki It's a recipient Prometheus Inspired horizontal scalability 、 High availability 、 Multi tenant log aggregation system . Its design is very cost-effective and easy to operate . It doesn't index the contents of the log , Instead, set a set of labels for each log stream . The log data is compressed and stored in the object store ( for example S3 perhaps GCS) In the middle of the block .

For traditional log management tools ,Loki It has the following characteristics

• Do not full-text index logs , Storing compressed unstructured logs and index only metadata ,Loki Easier to operate , Lower operating costs .
• Use is already in Prometheus Index and group log streams with the same tags used in , Enable you to use already in Prometheus The same tags used in switch seamlessly between metrics and logs .
• Loki Is a particularly suitable storage Kubernetes Pod journal , Metadata ( Such as Pod label ) It will automatically crawl and index .
• Native support Grafana

Loki Through three modules to complete the log management , The three modules are Promtail,Loki,Grafana

Promtail
1. Send container logs to Loki perhaps Grafana Log collection tool on the service .
2. Discover the collection target and add data to the log stream Label, And send it to Loki.
3.Promtail Our service discovery is based on Prometheus The service discovery mechanism of , You can see configmap loki-promtail Understand the details

Loki
1.Loki Can expand horizontally 、 Highly available and multi tenant log aggregation system
2. Use and Prometheus The same service discovery mechanism , Add tags to the log stream instead of building a full-text index
3.Promtail Received logs and applied metrics Indicators have the same set of tags

Grafana
1.Grafana Is an open source platform for monitoring and visual observation , Support very rich data sources
2. stay Loki In the technology stack, it's dedicated to displaying from Prometheus and Loki Time series data from other data sources
3. Allow queries 、 visualization 、 Alarm and other operations , Can be used to create 、 Explore and share data Dashboard

Loki Itself can be divided into 3 A component , distributor 、 collector 、 Inquire device .

Distributor( distributor )
The allocator is responsible for processing the logs written by the client , Once the allocator receives the log data , It will be divided into several batches , And send them to multiple collectors in parallel .
The distributor passes through gRPC Communicate with the collector . The allocator is stateless , The distributor can be expanded according to actual needs .
Ingester( collector )
The collector is responsible for writing logs to the back end of long-term storage (S3,SynamoDB etc. ), The collector will check whether the collected logs are out of order . in addition , The collector verifies that the received log lines are received in the order of increasing timestamp , If the timestamp is incorrect , The log line is rejected with an error .
Querier( Inquire device )
The querier service handles LogQL Query statements to evaluate log data stored in long-term storage .

Previously, I introduced Loki Theoretical knowledge of , Next, let's take a look through practical examples Loki How to work . The first is k8s Install on cluster Loki

In the installation Loki When , At the same time installed grafana and prometheus( install prometheus when persistentVolume=false, Otherwise, you need to configure storage )

helm repo add grafana https://grafana.github.io/helm-charts
helm upgrade --install loki grafana/loki-stack --set grafana.enabled=true,prometheus.enabled=true,prometheus.alertmanager.persistentVolume.enabled=false,prometheus.server.persistentVolume.enabled=false

After installation , see pod, You can see loki Of pod Already in running state .

take Loki grafana service Change the type of to NodePort, Convenient for later access on the Internet Grafana

see secret, obtain Grafana Login password for , Note that you need to pass base64 -d Decrypt to login .

kubectl get secret loki-grafana -oyaml

After logging in , Data sources can be selected Loki perhaps Promethues, Choose here Loki, The following is a view of nginx pod Log information .

Select what you want to view through the tag pod Information is very important , As shown below ：Loki All installed on the cluster are automatically displayed pod, You can select whatever you want to view pod, Log view .

That's right Loki To understand and use .