Loki

First PublishedFeb 16, 2026ByAtif Alam

Grafana Loki is a log aggregation system designed to be cost-effective and easy to operate. It’s like Prometheus, but for logs — it indexes labels (metadata) instead of the full log content, making it much cheaper to run than Elasticsearch.

How Loki Works

┌──────────┐     push logs      ┌──────────┐     query      ┌──────────┐
│  Promtail │───────────────────►│   Loki   │◄───────────────│  Grafana │
│  (agent)  │                    │  (store) │                │  (UI)    │
└──────────┘                    └──────────┘                └──────────┘

1. Promtail (or another agent) reads log files, attaches labels, and pushes them to Loki.
2. Loki stores the logs, indexing only the labels (not the content). Log content is stored compressed in chunks.
3. Grafana queries Loki using LogQL and displays logs in the Explore panel or Log panels.

Why Not Elasticsearch?

	Loki	Elasticsearch
Indexing	Labels only	Full-text index
Storage cost	Low (compressed chunks)	High (full index)
Query speed	Fast for label-filtered queries; grep-like for content	Fast for any full-text search
Complexity	Simple (few components)	Complex (cluster, shards, mappings)
Best for	Kubernetes/cloud-native logs with known label structure	Full-text search across diverse log formats

Loki trades full-text search speed for much lower resource usage.

Labels

Labels are key-value pairs attached to log streams. They’re how you filter and query logs:

{app="my-app", namespace="production", pod="my-app-7b8f9c-x2k4d"}

Good Labels

Labels should have low cardinality (few distinct values):

{app="my-app"}                    # good — few apps
{namespace="production"}           # good — few namespaces
{env="staging", region="us-east"}  # good — few environments and regions

Bad Labels (Avoid)

High-cardinality labels explode the index:

{user_id="abc123"}        # bad — millions of users
{request_id="req-456"}    # bad — unique per request
{ip="192.168.1.50"}       # bad — thousands of IPs

Put high-cardinality data in the log line, not in labels. Query it with LogQL filters.

Promtail Configuration

Promtail is the default agent that ships logs to Loki:

promtail-config.yml

server:
  http_listen_port: 9080

positions:
  filename: /tmp/positions.yaml    # tracks read position in log files

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: system
    static_configs:
      - targets: [localhost]
        labels:
          job: syslog
          host: web1
          __path__: /var/log/syslog

  - job_name: app
    static_configs:
      - targets: [localhost]
        labels:
          job: myapp
          __path__: /var/log/myapp/*.log

Kubernetes (Auto-Discovery)

Promtail automatically discovers pods and attaches Kubernetes labels:

scrape_configs:
  - job_name: kubernetes-pods
    kubernetes_sd_configs:
      - role: pod
    pipeline_stages:
      - docker: {}
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        target_label: app
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace

Alternative Agents

• Grafana Alloy (successor to Promtail) — All-in-one collector for metrics, logs, traces.
• Fluentd / Fluent Bit — With the Loki output plugin.
• Vector — With the Loki sink.

LogQL Basics

LogQL is Loki’s query language, inspired by PromQL.

Stream Selectors

Filter by labels:

{app="my-app"}
{app="my-app", namespace="production"}
{app=~"my-app|api-gateway"}       # regex match
{namespace!="development"}         # not equal

Line Filters

Filter log content (like grep):

{app="my-app"} |= "error"                    # contains "error"
{app="my-app"} != "healthcheck"               # does not contain
{app="my-app"} |~ "timeout|connection refused" # regex match
{app="my-app"} !~ "DEBUG|TRACE"               # regex exclude

Chain filters:

{app="my-app"} |= "error" != "healthcheck" |~ "500|503"

Parser Stages

Extract structured fields from log lines:

# JSON logs
{app="my-app"} | json | status_code >= 500

# Logfmt logs (key=value)
{app="my-app"} | logfmt | level="error"

# Regex extraction
{app="my-app"} | regexp `(?P<method>\w+) (?P<path>/\S+) (?P<status>\d+)`
  | status >= 500

# Pattern (simpler than regex)
{app="my-app"} | pattern `<method> <path> <status>`
  | status >= 500

Label Filters (After Parsing)

{app="my-app"} | json | duration > 1s
{app="my-app"} | json | status_code = 500 | method = "POST"
{app="my-app"} | logfmt | level = "error" | line_format "{{.msg}}"

Metric Queries

Turn logs into metrics:

# Count of error logs per minute
count_over_time({app="my-app"} |= "error" [1m])

# Rate of logs per second
rate({app="my-app"}[5m])

# Bytes rate
bytes_rate({app="my-app"}[5m])

# Average extracted duration (from JSON field)
avg_over_time({app="my-app"} | json | unwrap duration [5m])

# Quantile of extracted values
quantile_over_time(0.95, {app="my-app"} | json | unwrap duration [5m])

Aggregations

# Error count by app
sum by (app) (count_over_time({namespace="production"} |= "error" [5m]))

# Top 5 apps by log volume
topk(5, sum by (app) (bytes_rate({namespace="production"}[5m])))

Viewing Logs in Grafana

1. Go to Explore (compass icon).
2. Select the Loki data source.
3. Write a LogQL query.
4. View log lines with timestamps, labels, and content.
5. Click a log line to expand and see all fields.

Log Panel on Dashboards

Add a Logs panel to any dashboard:

{app="my-app", namespace="production"} |= "error"

Combine with metric panels on the same dashboard — see error rate spikes alongside the actual error logs.

Structured Logging

Structured logging means writing logs as key-value pairs (usually JSON) instead of free-form text. This makes logs far easier to query and parse in Loki.

Unstructured vs Structured

Unstructured:

2026-02-16 10:23:45 ERROR Failed to process order 12345 for user abc - timeout after 5s

To extract the order ID, user, or duration, you need regex.

Structured (JSON):

{"ts":"2026-02-16T10:23:45Z","level":"error","msg":"Failed to process order","order_id":12345,"user":"abc","duration_s":5,"error":"timeout"}

LogQL can parse this instantly:

{app="orders"} | json | level="error" | duration_s > 3

Best Practices for Structured Logging

Practice	Why
Use JSON or logfmt format	Parseable by LogQL without regex
Include a level field	Filter by severity (error, warn, info, debug)
Include a msg field	Human-readable description of what happened
Add request/trace IDs	Correlate with distributed traces
Add domain fields	order_id, user_id, endpoint — searchable context
Don’t log sensitive data	No passwords, tokens, PII in plain text
Use consistent field names	duration_s everywhere, not dur / elapsed / time_ms in different services

Structured Logging in Code

Python (structlog):

import structlog

logger = structlog.get_logger()

logger.info("order_processed",
    order_id=12345,
    user="abc",
    duration_s=0.45,
    status="success"
)
# Output: {"event":"order_processed","order_id":12345,"user":"abc","duration_s":0.45,"status":"success","timestamp":"2026-02-16T10:23:45Z"}

Go (zerolog):

log.Info().
    Int("order_id", 12345).
    Str("user", "abc").
    Float64("duration_s", 0.45).
    Msg("order processed")

Node.js (pino):

const logger = require('pino')();

logger.info({ order_id: 12345, user: 'abc', duration_s: 0.45 }, 'order processed');

Log Pipelines

A log pipeline processes logs between the source (your app) and the destination (Loki). Pipelines parse, transform, filter, and enrich logs before storage.

Why Pipelines?

• Parse unstructured logs into structured fields.
• Enrich logs with metadata (Kubernetes labels, host info).
• Filter noisy logs (healthchecks, debug lines) to reduce storage costs.
• Redact sensitive data (emails, tokens) before storage.
• Route different logs to different destinations or tenants.

Promtail Pipeline Stages

Promtail has a built-in pipeline engine. Each scrape job can define pipeline_stages:

scrape_configs:
  - job_name: app
    static_configs:
      - targets: [localhost]
        labels:
          job: myapp
          __path__: /var/log/myapp/*.log
    pipeline_stages:
      # 1. Parse JSON logs
      - json:
          expressions:
            level: level
            msg: msg
            duration: duration_s

      # 2. Set 'level' as a label (low cardinality — only error/warn/info/debug)
      - labels:
          level:

      # 3. Filter out debug logs in production
      - match:
          selector: '{level="debug"}'
          action: drop

      # 4. Redact email addresses
      - replace:
          expression: '([a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,})'
          replace: '***REDACTED***'

      # 5. Add a timestamp from the log line
      - timestamp:
          source: ts
          format: RFC3339

Common Pipeline Stages

Stage	Purpose	Example
json	Parse JSON log lines	Extract level, msg, duration
logfmt	Parse logfmt (key=value) lines	Extract level, caller
regex	Parse with regex named groups	Extract fields from unstructured logs
labels	Promote extracted fields to Loki labels	Set level as a label for filtering
timestamp	Use a field as the log timestamp	Align Loki timestamp with app timestamp
output	Rewrite the log line	Change what gets stored
match	Conditionally apply stages	Drop debug logs, route by content
replace	Regex find-and-replace	Redact sensitive data
drop	Drop log lines entirely	Remove healthcheck noise
tenant	Set Loki tenant ID per line	Multi-tenant log routing
multiline	Merge multi-line logs (stack traces)	Combine Java exception lines into one entry

Multi-Line Log Handling

Stack traces and multi-line exceptions need special handling — otherwise each line becomes a separate log entry:

pipeline_stages:
  - multiline:
      firstline: '^\d{4}-\d{2}-\d{2}'  # new log starts with a date
      max_wait_time: 3s
      max_lines: 128

This merges lines until the next log entry starts, keeping stack traces together.

Pipeline in Grafana Alloy

Grafana Alloy (the successor to Promtail) uses a different config format but the same pipeline concepts:

loki.process "app_pipeline" {
  stage.json {
    expressions = { level = "level", msg = "msg" }
  }
  stage.labels {
    values = { level = "" }
  }
  stage.drop {
    expression = ".*healthcheck.*"
  }
  forward_to = [loki.write.default.receiver]
}

External Pipeline Tools

For complex pipelines or when you need to route logs to multiple destinations beyond Loki:

Tool	Strengths
Vector	High-performance Rust-based pipeline; transform, filter, route to Loki, Elasticsearch, S3, etc.
Fluent Bit	Lightweight C-based; popular in Kubernetes; many output plugins
Fluentd	Mature Ruby-based; huge plugin ecosystem; heavier than Fluent Bit
OTel Collector	Unified pipeline for metrics, logs, and traces via OpenTelemetry

Example: Use Fluent Bit to collect, Vector to transform and route, Loki to store, Grafana to visualize.

Key Takeaways

• Loki indexes labels only, not log content — cheap to run, but requires good label design.
• Keep labels low cardinality (app, namespace, environment). Put user IDs, request IDs, etc. in the log line.
• LogQL = stream selectors ({app="x"}) + line filters (|= "error") + parsers (| json) + metrics (count_over_time).
• Use metric queries to turn logs into charts (error counts, log volume, extracted latencies).
• Promtail (or Grafana Alloy) auto-discovers Kubernetes pods and attaches labels automatically.
• Structured logging (JSON/logfmt) makes logs instantly parseable — no regex needed.
• Log pipelines (Promtail stages, Alloy, Vector) parse, filter, redact, and route logs before they reach Loki.