System Overview

In modern software system operations, log collection and analysis are crucial. Our log collection system is built on Promtail, Loki, and Grafana, providing an efficient and powerful solution. It can collect logs from complex system environments, perform effective storage and management, and help operations and development personnel quickly obtain valuable information through an intuitive visualization interface for various scenarios such as system monitoring, troubleshooting, performance analysis, security auditing, ensuring stable system operation and continuous optimization.

Our log collection system, based on open-source components Promtail, Loki, and Grafana, has built an efficient log collection and analysis platform that can collect, store, and display application service log data in real-time.

System Architecture

As shown in the diagram below, the logging system contains three core components: Promtail, Loki, and Grafana. Promtail, as a log collection tool, is responsible for extracting logs from various services and sending them to Loki. Loki is responsible for storing log data. Grafana provides powerful log visualization and query capabilities, helping users quickly locate problems and monitor service health status.

Component Introduction

1. Promtail

• Function: Responsible for collecting logs from various sources (such as containers, application log files, etc.) and sending them to Loki. Supports log classification through log labels and parsing of multiple log formats.
• Advantages: Lightweight, able to efficiently collect logs from containers and file systems; supports integration of multiple log sources, flexible configuration options allow users to customize log collection strategies.

2. Loki

• Function: Loki is a horizontally scalable log aggregation system, similar to Prometheus but optimized for log data. It separates log storage from indexing, uses time and label-based indexing, and can efficiently store and query large amounts of logs.
• Advantages: Efficient storage and query capabilities, seamless integration with Promtail; low latency, distributed architecture, able to meet the needs of large-scale log data.

3. Grafana

• Function: Grafana is an open-source visualization tool used to display log data from Loki. Through Grafana, users can create powerful dashboards and charts, supporting real-time log querying, display, and alerting.
• Advantages: Supports multiple data sources, powerful visualization capabilities, flexible query functions, rich plugin ecosystem, convenient for users to create custom monitoring views.

Example Scenarios

Scenario 1: Log Anomaly Monitoring

Configure log collection rules so that when keywords (such as "ERROR" or "Exception") appear in application logs, Loki stores the log data and creates dashboards through Grafana to display the number and details of anomalous logs in real-time, helping operations personnel quickly locate problems.

Scenario 2: Slow Service Response Log Alert

Create a dashboard in Grafana to monitor response times for each service. When a service's response time exceeds 5 seconds, Loki records the event and displays it in Grafana. Operations personnel can view it in real-time on the dashboard, quickly discover slow-responding services, and notify relevant personnel by setting alert rules.

Scenario 3: Log Storage Optimization

As log volume increases, periodic cleaning of no longer needed log data is required. Log retention policies can be configured in Loki to automatically clean by time or log level, ensuring efficient operation of the logging system and saving storage space.