Prometheus Adapter HPA on k3s

First PublishedMay 18, 2026ByAtif Alam

This lab adds prometheus-adapter to an existing kube-prometheus-stack install on k3s, then scales a Deployment with HPA on a custom metric.

Prerequisites: Installing Prometheus and Grafana on k3s completed (monitoring namespace, release name monitoring). CPU-only HPA first: HPA on k3s.

EKS production path: Prometheus Adapter for HPA on EKS.

Step 1: Confirm Prometheus is reachable

1
kubectl get pods -n monitoring
2
kubectl port-forward -n monitoring svc/monitoring-kube-prometheus-prometheus 9090:9090

Open http://127.0.0.1:9090 and run Status → Targets — core targets should be UP.

Prometheus Service DNS (used by the adapter):

1
http://monitoring-kube-prometheus-prometheus.monitoring.svc:9090

If your release name differs, adjust: kubectl get svc -n monitoring.

Step 2: Adapter rules

Create adapter-rules.yaml:

1
rules:
2
  - seriesQuery: 'http_requests_total{namespace!="",pod!=""}'
3
    resources:
4
      overrides:
5
        namespace: { resource: "namespace" }
6
        pod: { resource: "pod" }
7
    name:
8
      matches: "^(.*)_total$"
9
      as: "${1}_per_second"
10
    metricsQuery: 'sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)'

Step 3: Install prometheus-adapter

Create prometheus-adapter-values.yaml:

1
prometheus:
2
  url: http://monitoring-kube-prometheus-prometheus.monitoring.svc
3
  port: 9090
4

5
rules:
6
  existing: adapter-rules
7

8
replicas: 1

1
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
2
helm repo update
3

4
helm upgrade --install prometheus-adapter prometheus-community/prometheus-adapter \
5
  --namespace monitoring \
6
  -f prometheus-adapter-values.yaml \
7
  --wait

Verify:

1
kubectl get apiservice v1beta1.custom.metrics.k8s.io
2
kubectl get pods -n monitoring -l app.kubernetes.io/name=prometheus-adapter
3
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | head -c 1500

Step 4: Sample app with metrics

Deploy httpbin (or any exporter of http_requests_total) in namespace hpa-promo:

1
kubectl create namespace hpa-promo

httpbin-promo.yaml:

1
apiVersion: apps/v1
2
kind: Deployment
3
metadata:
4
  name: httpbin
5
  namespace: hpa-promo
6
spec:
7
  replicas: 2
8
  selector:
9
    matchLabels:
10
      app: httpbin
11
  template:
12
    metadata:
13
      labels:
14
        app: httpbin
15
    spec:
16
      containers:
17
        - name: httpbin
18
          image: mccutchen/go-httpbin:v2.15.0
19
          ports:
20
            - containerPort: 8080
21
          resources:
22
            requests:
23
              cpu: 50m
24
              memory: 64Mi
25
---
26
apiVersion: v1
27
kind: Service
28
metadata:
29
  name: httpbin
30
  namespace: hpa-promo
31
  labels:
32
    app: httpbin
33
spec:
34
  selector:
35
    app: httpbin
36
  ports:
37
    - name: http
38
      port: 80
39
      targetPort: 8080

For kube-prometheus-stack to scrape the app, add a ServiceMonitor (labels must match your Prometheus serviceMonitorSelector — often release: monitoring):

1
apiVersion: monitoring.coreos.com/v1
2
kind: ServiceMonitor
3
metadata:
4
  name: httpbin
5
  namespace: hpa-promo
6
  labels:
7
    release: monitoring
8
spec:
9
  selector:
10
    matchLabels:
11
      app: httpbin
12
  endpoints:
13
    - port: http
14
      path: /metrics
15
      interval: 15s

go-httpbin may not expose Prometheus metrics by default — for a reliable lab, use an image with /metrics or add a small exporter sidecar. If no series appear, HPA will not scale; confirm a metric exists in Prometheus Explore before applying HPA.

For a known-good pattern, copy scrape config from Observability setup — ServiceMonitor for your app’s metric names.

1
kubectl apply -f httpbin-promo.yaml
2
# apply ServiceMonitor when your app exports /metrics

Step 5: HPA on custom metric

When Prometheus shows http_requests_per_second (or your rule’s output name):

1
apiVersion: autoscaling/v2
2
kind: HorizontalPodAutoscaler
3
metadata:
4
  name: httpbin-hpa
5
  namespace: hpa-promo
6
spec:
7
  scaleTargetRef:
8
    apiVersion: apps/v1
9
    kind: Deployment
10
    name: httpbin
11
  minReplicas: 2
12
  maxReplicas: 6
13
  metrics:
14
    - type: Pods
15
      pods:
16
        metric:
17
          name: http_requests_per_second
18
        target:
19
          type: AverageValue
20
          averageValue: "10"

1
kubectl apply -f httpbin-hpa.yaml
2
kubectl describe hpa httpbin-hpa -n hpa-promo

Generate load (port-forward + hey, or wget loop) as in HPA on k3s.

Troubleshooting

Symptom	Check
custom.metrics APIService missing	Adapter pod logs, Helm release
No metric in API	PromQL in Prometheus UI; adapter rules
HPA unknown	Metric name must match adapter `as:` rule
ServiceMonitor ignored	`release` label vs Prometheus selector

Cleanup

1
helm uninstall prometheus-adapter -n monitoring
2
kubectl delete namespace hpa-promo

To remove the full stack, see Prometheus + Grafana on k3s — Cleanup.