Logging in containerized applications or Kubernetes involves running an agent (LogStash, FluentD, etc.) on the host (k8s nodes) to push the logs to a central database (ElasticSearch, Loki, etc.).

EFK Stack

• ‣ is used for log collection
• ElasticSearch is used as the DB to store the logs sent by FluentD
• Kibana is the web interface to view the logs stored in ElasticSearch

Grafana Loki

Must watch tutorial: Mastering Grafana Loki: Part 1

Grafana Loki is a log aggregation tool which uses Promtail as the log collecting agent by default (can be configured to use FluentBit instead). Promtail runs as a DaemonSet and pushes logs to Loki, which is the database that stores and indexes the logs. Once the logs are present in Loki, it can be queried by Grafana and displayed on the UI.

Unlike other logging systems, a Loki index is built from labels, leaving the original log message unindexed. This means, Loki is much more resource efficient compared to other logging tools.

Loki is built out of many component microservices, and is designed to run as a horizontally-scalable distributed system. It has three modes of operation:

• Monolithic: runs all of Loki’s microservice components inside a single process as a single binary or Docker image, can only use filesystem for storage
• Scalable: separates the reads and writes to the backend datastore to improve performance, requires a managed object store such as AWS S3 or a self-hosted store such as Minio
• Microservices: separates each component of Loki as a separate process for maximum scalability and efficiency (default in Helm chart installations)

Helm Installation of Loki (monolithic mode)

Refer Install the monolithic Helm chart | Grafana Loki documentation.

helm repo add grafana <https://grafana.github.io/helm-charts>
helm repo update

loki:
  commonConfig:
    replication_factor: 1
  storage:
    type: 'filesystem'
	auth_enabled: false

singleBinary:
  replicas: 1

helm install loki grafana/loki -n logging --values values.yaml 

Helm Installation of Promtail

If you added the Helm repo in the above section, you can just install Promtail as a chart. We don’t need to update the values.yaml file. Refer Promtail | Grafana Loki documentation.