Host-Related Metrics
1. CPU
- CPU Utilization: Monitors the total CPU utilization and the utilization of each core. High utilization may indicate an overloaded system.
- System CPU Usage: The CPU time consumed by the operating system kernel. Persistently high values may require attention to system processes and configuration.
- User CPU Usage: The CPU time consumed by user processes, helping to assess the application's demand for CPU.
- IO Wait: The time the CPU spends waiting for disk or network I/O operations to complete. High I/O wait may indicate a disk or network bottleneck.
2. Memory
- Memory Usage: The current amount of memory used, helping to identify memory consumption. High usage may cause performance degradation.
- Available Memory: The amount of memory remaining in the system. Low available memory may indicate that the system is approaching its memory limit.
- Cached Memory: The amount of memory the operating system uses as cache, which helps speed up access to files and block devices.
3. Disk
- Disk Usage: The usage of disk partitions, helping to monitor whether disk space is sufficient.
- Disk Read/Write Rate: The number of bytes read from and written to disk per second, indicating I/O activity.
- Disk IO Wait Time: The time spent waiting for disk I/O operations to complete. High I/O wait time may indicate insufficient disk performance.
- Disk IO Operations: The number of read/write operations per second, helping to identify I/O bottlenecks.
4. Network
- Network Throughput: The amount of data transmitted per second, including inbound (received) and outbound (sent), helping to understand network bandwidth usage.
- Network Latency: The time data takes to travel across the network. Increased latency may indicate network congestion or insufficient bandwidth.
- Packet Loss: The proportion of packets lost during transmission. A high packet loss rate usually indicates a network problem.
- Network Connections: The number of active TCP/UDP connections, helping to identify network connection usage.
5. Processes
- CPU and Memory Usage of Key Processes: Monitors the CPU and memory usage of key processes to detect processes that consume resources abnormally in time.
- Process Uptime: Records how long a process has been running continuously, helping to detect process restarts or abnormal terminations.
K8S-Related Metrics
1. Pods and Containers
container_cpu_usage_seconds_total: Container CPU usage time (seconds).
container_memory_usage_bytes: Memory used by the container (bytes).
container_memory_working_set_bytes: Memory actively used by the container, excluding cache.
container_fs_usage_bytes: Disk space used by the container (bytes).
container_last_seen: The time the container was last checked.
container_start_time_seconds: The container's start timestamp (seconds).
kube_pod_status_phase: The current state of the Pod (Running, Pending, Succeeded, Failed, Unknown).
kube_pod_container_status_restarts_total: The number of times the Pod's containers have restarted.
kube_pod_container_status_running: Whether the Pod's container is running.
kube_pod_container_status_terminated: Whether the Pod's container has terminated.
kube_pod_container_status_waiting: Whether the Pod's container is in a waiting state.
2. Scheduling and Resource Allocation
- kube_pod_container_resource_requests_cpu_cores: CPU resources requested by the Pod's container (cores).
- kube_pod_container_resource_requests_memory_bytes: Memory resources requested by the Pod's container (bytes).
- kube_pod_container_resource_limits_cpu_cores: CPU limit of the Pod's container (cores).
- kube_pod_container_resource_limits_memory_bytes: Memory limit of the Pod's container (bytes).
- kube_node_status_capacity_cpu_cores: Total CPU capacity of the node (cores).
- kube_node_status_capacity_memory_bytes: Total memory capacity of the node (bytes).
- kube_node_status_allocatable_cpu_cores: Allocatable CPU capacity of the node (cores).
- kube_node_status_allocatable_memory_bytes: Allocatable memory capacity of the node (bytes).
3. Controllers and Replicas
- kube_deployment_status_replicas: The current number of replicas of the Deployment.
- kube_deployment_status_replicas_updated: The number of updated Deployment replicas.
- kube_replica_set_status_replicas: The number of replicas of the ReplicaSet.
- kube_replica_set_status_replicas_available: The number of available ReplicaSet replicas.
- kube_statefulset_status_replicas: The number of replicas of the StatefulSet.
- kube_statefulset_status_replicas_available: The number of available StatefulSet replicas.
4. API Server
- apiserver_request_duration_seconds: The time the API server takes to process requests (seconds).
- apiserver_request_count: The total number of requests processed by the API server.
- apiserver_request_latencies_seconds: The latency distribution of API server requests.
- controller_manager_reconcile_duration_seconds: The reconcile duration of the controller manager (seconds).
- controller_manager_reconcile_count: The number of reconciles performed by the controller manager.
5. Network and Load
- kube_service_info: Metadata of the Kubernetes service.
- kube_endpoint_address_count: The number of addresses of the service.
- kube_service_spec_type: The service type (ClusterIP, NodePort, LoadBalancer, etc.).
6. Storage
- kube_persistentvolumeclaim_resource_requests_storage_bytes: The storage space requested by the persistent volume claim (bytes).
- kube_persistentvolumeclaim_resource_requests_storage_bytes: The storage space actually used by the persistent volume claim (bytes).
- kube_persistentvolume_status_phase: The status of the persistent volume (Available, Bound, Released, Failed).
Redis-Related Metrics
1. Resource Usage
used_cpu_sys: CPU time used by the Redis process in system space (seconds).
used_cpu_user: CPU time used by the Redis process in user space (seconds).
used_cpu_sys_children: CPU time used by all child processes in system space (seconds).
used_cpu_user_children: CPU time used by all child processes in user space (seconds).
used_memory_rss: The amount of memory the operating system has allocated to the Redis process (RSS).
used_memory_peak: The peak memory usage of the Redis instance (bytes).
maxmemory: The maximum memory limit configured for Redis (bytes).
mem_fragmentation_ratio: The memory fragmentation ratio. Calculated as
used_memory_rss / used_memory, used to indicate memory waste.used_memory: The total amount of memory used by the Redis instance (bytes).
2. Key-Value Operations
- total_connections_received: The total number of connections received by Redis.
- total_commands_processed: The total number of commands processed by Redis.
- keyspace_hits: The number of cache hits for queries.
- keyspace_misses: The number of cache misses for queries.
- evicted_keys: The number of keys evicted (due to memory limits).
- expired_keys: The number of expired keys.
- persistent_connections: The number of persistent connections.
3. Persistence
- rdb_changes_since_last_save: The number of key changes since the last RDB persistence.
- rdb_last_bgsave_status: The status of the last RDB save operation (OK or ERROR).
- rdb_last_save_time: The timestamp of the last RDB persistence.
- aof_current_size: The current size of the AOF file (bytes).
- aof_last_bgrewrite_status: The status of the last AOF rewrite.
- aof_last_write_status: The status of the last AOF write operation.
4. Cluster and Replication
- role: The role of the Redis instance (master/slave).
- connected_slaves: The current number of connected replica nodes.
- master_last_io_seconds_ago: The time since the last interaction between the master and replica nodes (seconds).
- replication_backlog_active: Whether the replication backlog is enabled (1 means enabled, 0 means disabled).
5. Network
- total_net_input_bytes: The total number of bytes received by Redis.
- total_net_output_bytes: The total number of bytes sent by Redis.
- blocked_clients: The current number of blocked clients.
6. Slow Queries
- slowlog_len: The number of entries in the Redis slow query log.
- slowlog_get (N): Retrieve N entries from the slow query log.
MinIO-Related Metrics
1. Resource Usage
minio_memory_used_bytes: Current memory usage (bytes).minio_cpu_usage: CPU utilization.minio_cpu_load: CPU load (1-minute, 5-minute, and 15-minute average load).
2. Requests and Throughput
minio_api_requests_total: The total number of API requests.minio_api_request_duration_seconds: The response time of API requests (seconds).minio_api_requests_rate: The API request rate (requests per second).minio_api_errors_total: The total number of errors in API requests.minio_api_success_total: The total number of successfully processed API requests.minio_api_latency_seconds: The latency of API requests (seconds).
3. Network
minio_network_received_bytes_total: The total number of bytes received from clients.minio_network_sent_bytes_total: The total number of bytes sent to clients.minio_network_error_total: The total number of network errors.minio_network_requests_total: The total number of network requests.
4. Errors and Exceptions
minio_errors_total: The total number of errors.minio_error_rate: The error rate (errors per second).minio_object_errs: The number of errors in object storage.minio_request_errs: The number of errors in API requests.
5. Health and Status
minio_health_check: The health check status of the MinIO instance.minio_uptime_seconds: The uptime of the MinIO instance (seconds).minio_heartbeat: The status used to detect node heartbeats.minio_watchdog_timeouts: The number of watchdog timeouts.minio_cluster_health: The health of the MinIO cluster (for cluster deployments).minio_cluster_nodes: The number of available nodes in the cluster.
