Installing Prometheus and Grafana on k3s

First PublishedApr 24, 2026Last UpdatedMay 18, 2026ByAtif Alam

This page is a fast k3s-focused install using the kube-prometheus-stack Helm chart. For the same chart with more Kubernetes context (Loki, ServiceMonitor / PrometheusRule YAML, and patterns not repeated here), see Observability setup.

Quick glossary

• Prometheus — Scrapes HTTP metrics endpoints, stores time series, evaluates rules.
• Grafana — Dashboards and Explore; reads Prometheus as a data source (pre-wired by the chart).
• Alertmanager — Groups and routes alerts from Prometheus (installed with this stack).

The chart also installs the Prometheus Operator (CRDs + controllers), node-exporter, and kube-state-metrics. Conceptual background: Prometheus and Grafana.

Prerequisites

• kubectl pointed at your k3s cluster (see Kubeconfig Authentication if auth fails).
• Helm 3 installed:

helm version

If Helm is missing, see Helm.

• Storage: k3s ships local-path by default. Check classes before enabling persistence:

kubectl get storageclass

• Resources: first install can pull several images and schedule many pods. On a very small single-node lab, installs may go Pending from CPU/memory pressure; see Troubleshooting and consider lowering chart requests (Helm values or --set) after a failed attempt.

Before you install

Avoid a duplicate release or namespace clash:

helm list -n monitoring
kubectl get ns monitoring 2>/dev/null

If a release monitoring or namespace monitoring already exists, use helm upgrade or remove the old release before installing again.

Install

Add the chart repo and install the stack into namespace monitoring with release name monitoring (matches Observability setup so service names below stay predictable).

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

helm install monitoring prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace \
  --wait \
  --timeout 15m \
  --set grafana.adminPassword=admin

The --wait flag blocks until core resources become ready (can take several minutes).

Security note: grafana.adminPassword=admin is convenient for a homelab only. For anything shared, omit that --set so the chart generates credentials, then read the password from the Secret (see Grafana login).

Enable Durable Storage (Recommended)

The quick install above can run with ephemeral storage in some environments. If kubectl -n monitoring get pvc returns no PVCs and mounts show emptyDir, Grafana DB, Prometheus TSDB, and Alertmanager state are not durable.

Create kube-prometheus-stack-k3s-persistence-values.yaml:

kube-prometheus-stack-k3s-persistence-values.yaml

grafana:
  persistence:
    enabled: true
    storageClassName: local-path
    accessModes: ["ReadWriteOnce"]
    size: 5Gi

prometheus:
  prometheusSpec:
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: local-path
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 20Gi

alertmanager:
  alertmanagerSpec:
    storage:
      volumeClaimTemplate:
        spec:
          storageClassName: local-path
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 5Gi

Upgrade the existing release:

helm upgrade monitoring prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  -f kube-prometheus-stack-k3s-persistence-values.yaml \
  --wait --timeout 15m

Then verify durable storage:

kubectl -n monitoring get pvc
kubectl get pv
kubectl -n monitoring describe deploy monitoring-grafana
kubectl -n monitoring get sts | grep -E 'prometheus|alertmanager'

Expected:

• PVCs are present and Bound.
• PVs are provisioned (often by local-path in k3s).
• Data mounts are PVC-backed rather than emptyDir.

Note: local-path is a StorageClass, not a PV name. kubectl get pv local-path returning NotFound is expected.

Verify

kubectl get pods -n monitoring
kubectl get svc -n monitoring

Optional: confirm operator-managed CRs exist:

kubectl get prometheus,alertmanager -n monitoring

Discover exact Service names for port-forward (they always include the release name monitoring):

kubectl get svc -n monitoring | grep -E 'grafana|prometheus|alertmanager'

Typical names for release monitoring:

Role	Service (typical)	Ports
Grafana	monitoring-grafana	80
Prometheus	monitoring-kube-prometheus-prometheus	9090
Alertmanager	monitoring-kube-prometheus-alertmanager	9093

What you will see

After install, Grafana includes pre-built Kubernetes dashboards (folder names vary by chart version). In the UI: Menu → Dashboards (or use search).

You get strong cluster, node, and workload resource visibility (CPU, memory, pod state, kubelet/cAdvisor container metrics, kube-state object metrics). Full application-level or business KPI metrics (orders, revenue, funnels) are not included unless your app exposes Prometheus metrics and you add discovery (see Extending this setup).

Durability reminder: this visibility is durable only when storage is PVC-backed (see Enable Durable Storage (Recommended)).

Prometheus scrapes roughly: kube-state-metrics, node-exporter, kubelet/cAdvisor and other Kubernetes discovery jobs, Prometheus self-scrape, and Alertmanager. Exact jobs appear under Status → Targets in the Prometheus UI after you port-forward.

Access Grafana and Prometheus

Port-forward (recommended first)

Grafana (replace the Service name if kubectl get svc shows a different one):

kubectl port-forward -n monitoring svc/monitoring-grafana 3000:80

Open http://localhost:3000.

Prometheus:

kubectl port-forward -n monitoring svc/monitoring-kube-prometheus-prometheus 9090:9090

Open http://localhost:9090, then Status → Targets to confirm scrapes are UP.

Optional: Ingress or NodePort

Option A: NodePort (quick LAN access)

Patch Grafana and Prometheus Services to NodePort:

kubectl patch svc -n monitoring monitoring-grafana -p '{"spec":{"type":"NodePort"}}'
kubectl patch svc -n monitoring monitoring-kube-prometheus-prometheus -p '{"spec":{"type":"NodePort"}}'
kubectl get svc -n monitoring monitoring-grafana monitoring-kube-prometheus-prometheus -o wide

Use any reachable node IP from kubectl get nodes -o wide plus the shown node ports:

• Grafana: http://<node-ip>:<grafana-nodeport>
• Prometheus: http://<node-ip>:<prometheus-nodeport>

Option B: Traefik Ingress (hostname access)

1. Confirm Traefik exists and capture an entrypoint IP:

kubectl get svc -n kube-system traefik -o wide

Pick one Traefik EXTERNAL-IP (example: 10.0.0.60), then add host mappings in your workstation hosts file (/etc/hosts on macOS/Linux, C:\\Windows\\System32\\drivers\\etc\\hosts on Windows):

10.0.0.60 grafana.monitoring.local
10.0.0.60 prometheus.monitoring.local

2. Create Ingress resources:

monitoring-ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: grafana
  namespace: monitoring
spec:
  ingressClassName: traefik
  rules:
    - host: grafana.monitoring.local
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: monitoring-grafana
                port:
                  number: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: prometheus
  namespace: monitoring
spec:
  ingressClassName: traefik
  rules:
    - host: prometheus.monitoring.local
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: monitoring-kube-prometheus-prometheus
                port:
                  number: 9090

kubectl apply -f monitoring-ingress.yaml
kubectl get ingress -n monitoring

3. Test:

curl -I --resolve grafana.monitoring.local:80:10.0.0.60 http://grafana.monitoring.local/login
curl -I --resolve prometheus.monitoring.local:80:10.0.0.60 http://prometheus.monitoring.local/-/ready

Then open:

• http://grafana.monitoring.local
• http://prometheus.monitoring.local

If a browser cannot reach hosts that work with curl, check browser Secure DNS settings.

Grafana login

• Username: admin (chart default).
• Password: if you passed --set grafana.adminPassword=admin, use admin. Otherwise read the generated Secret:

kubectl get secret -n monitoring monitoring-grafana -o jsonpath='{.data.admin-password}' | base64 -d
echo

Extending this setup

1. Expose metrics from your app — Add a /metrics endpoint (Prometheus text format) or use a Prometheus client library; expose a named port on the Service (for example metrics).
2. Add a ServiceMonitor — So the Prometheus Operator scrapes your app; label it so the release selector matches (for example release: monitoring). Full YAML and label notes: Observability setup — Custom scrape targets.
3. Verify in Prometheus — Status → Targets: your job should be UP. If DOWN, check port names, NetworkPolicies, and RBAC.
4. Dashboards — Import a dashboard by ID or build panels in Explore using your metric names.
5. Alerts — Add PrometheusRule CRs with the same release label pattern; example: Observability setup — Custom alert rules. Alerting concepts: Alerting.
6. Optional stack growth — Logs: Loki and Observability setup — Add Loki. Traces: OpenTelemetry, Distributed tracing. Long-term metrics: Scaling Prometheus.
7. HPA on custom metrics — Prometheus Adapter HPA on k3s after this install; EKS: Prometheus Adapter for HPA on EKS.

Troubleshooting

Symptom	What to check
Pods Pending	kubectl describe pod -n monitoring <pod> — often PVC binding (kubectl get pvc -n monitoring) or insufficient CPU/memory.
CrashLoopBackOff	kubectl logs -n monitoring <pod> — often OOM; reduce chart resource requests or add node capacity.
Prometheus targets DOWN	Wrong TLS or scrape config for a component; compare with Status → Targets error text. k3s is usually straightforward; still verify kubelet reachability.
Data disappears after restart/reschedule	Check mounts for emptyDir and run the persistence upgrade in Enable Durable Storage (Recommended).
helm install fails “already exists”	helm list -n monitoring — upgrade or uninstall the old release.

Broader triage: Troubleshooting and Debugging.

Cleanup

helm uninstall monitoring -n monitoring

Optional: remove the namespace (deletes PVCs and data):

kubectl delete namespace monitoring

Related pages

• Observability setup
• Prometheus
• Grafana
• Helm