Autoscale with Edera zone metrics

6 min read · Intermediate

Edera exposes per-pod zone metrics at http://127.0.0.1:3035/metrics/kubernetes on each node by default. Because pods run inside zones, these metrics are not available through cAdvisor or the kubelet. Prometheus can be configured to scrape the Edera endpoint directly.

This guide recommends using a per-node Prometheus-compatible collector to scrape and forward metrics, as a secure and reasonable default. This is a worked example to demonstrate how Edera can work with a specific Prometheus-compatible stack, but the choice of Prometheus stack and topology is an operator and site decision.

[metrics]
metrics-endpoint-address = "0.0.0.0:3035"

Prerequisites

• A Prometheus-compatible store with remote_write enabled. For standard Prometheus that is --web.enable-remote-write-receiver, or spec.enableRemoteWriteReceiver: true on a Prometheus Operator custom resource. Mimir, Thanos Receive, and VictoriaMetrics accept remote write as well. The store does not have to run in the same cluster.
• Helm
• Edera running on each node and serving metrics on port 3035

Available metrics

Edera exposes per-pod zone metrics labeled with namespace, pod, and zone_id. For the full list, see the Edera metrics reference.

1. Scrape the Edera daemon

Scraping with a per-node collector (recommended)

This example uses Grafana Alloy, whose Helm chart deploys a DaemonSet by default. However, this is up to the operator - any Prometheus-compatible per-node collector works, including Prometheus in agent mode, vmagent, or the OpenTelemetry Collector, etc.

In this example, we run one collector per node as a DaemonSet. Each one shares its node’s network namespace, so it can reach 127.0.0.1:3035 locally, and forwards what it scrapes with remote_write to the Prometheus that Prometheus Adapter queries. Nothing on the node accepts unsecured inbound connections, and you can configure your Prometheus-compatible store to use any authentication or encryption mechanisms it supports for remote_write, in the normal fashion.

# alloy-values.yaml
controller:
  type: daemonset
  hostNetwork: true
  dnsPolicy: ClusterFirstWithHostNet

crds:
  create: false

alloy:
  listenAddr: 127.0.0.1
  enableHttpServerPort: false
  enableReporting: false

  extraEnv:
    - name: NODE_NAME
      valueFrom:
        fieldRef:
          fieldPath: spec.nodeName

  configMap:
    content: |-
      prometheus.scrape "edera" {
        targets      = [{__address__ = "127.0.0.1:3035"}]
        metrics_path = "/metrics/kubernetes"
        // Without this the job label becomes "prometheus.scrape.edera", the
        // component ID, rather than "edera".
        job_name     = "edera"
        forward_to   = [prometheus.remote_write.default.receiver]
      }

      prometheus.remote_write "default" {
        external_labels = { node = sys.env("NODE_NAME") }

        // Bearer tokens and mTLS are both supported and
        // strongly recommended, depending on what your implementation supports for `remote_write`. 
        // See https://grafana.com/docs/alloy/latest/reference/components/prometheus/prometheus.remote_write/
        // for examples.
        endpoint {
          url = "http://prometheus.monitoring.svc.cluster.local:9090/api/v1/write"
        }
      }      

Install it:

helm repo add grafana https://grafana.github.io/helm-charts
helm install edera-metrics-collector grafana/alloy \
  --namespace monitoring \
  -f alloy-values.yaml

Each node’s series arrive carrying a node label.

Verify scraping is working

Query your Prometheus-compatible store to validate metrics are being collected. In this example, the Prometheus-compatible store is assumed to reside in the same Kubernetes cluster and exposed via a K8S service named prometheus:9090:

kubectl port-forward -n monitoring svc/prometheus 9090:9090 &

# One result per node, each with value 1
curl -s 'http://localhost:9090/api/v1/query?query=up{job="edera"}' \
  | jq '.data.result[] | {node: .metric.node, up: .value[1]}'

# Confirm zone metrics arrived
curl -s 'http://localhost:9090/api/v1/query?query=zone_cpu_usage_percent' \
  | jq '.data.result | length'

Expected result: up is 1 for every node, and a non-zero series count once at least one zone-backed pod is running. up at 1 with a count of zero means no pods are running with runtimeClassName: edera yet, since the endpoint returns an empty body when the node has no zones. A missing up series for a node means that node’s collector is not reporting at all.

To inspect a single collector’s readiness from the Alloy example above, forward to the node-local collector and check its readiness state:

kubectl port-forward -n monitoring ds/edera-metrics-collector-alloy 12345:12345 &
curl -s http://localhost:12345/-/ready

Scraping across nodes with a centralized collector (not recommended)

A single Prometheus can scrape across every node directly without using per-node collectors or securable remote_write, but this requires exposing every node’s metrics endpoint insecurely as plaintext to anonymous consumers outside of the node.

The metrics endpoint has no authentication and no TLS, so exposing it to leaks exposes per-pod names, namespaces, and resource usage to anything that can reach port 3035. As such, this mode is supported, but is opt-in and not recommended.

To enable potentially insecure cross-node scraping, modify /var/lib/edera/protect/daemon.toml and restart the daemon, on each node, with the following config that explicitly binds the metrics endpoint to all public and private interfaces on the node:

[metrics]
metrics-endpoint-address = "0.0.0.0:3035"

sudo systemctl restart protect-daemon

Then configure your cross-node Prometheus-compatible collector (in this example, assumed to be deployed in a K8S cluster) to scrape across all nodes via node service discovery:

apiVersion: v1
kind: Secret
metadata:
  name: additional-scrape-configs
  namespace: monitoring
stringData:
  scrape-configs.yaml: |
    - job_name: edera
      metrics_path: /metrics/kubernetes
      kubernetes_sd_configs:
      - role: node
      relabel_configs:
      - source_labels: [__address__]
        regex: '(.+):\d+'
        target_label: __address__
        replacement: '${1}:3035'
      # Use the node name as the instance label
      - source_labels: [__meta_kubernetes_node_name]
        target_label: instance    

Reference the secret from your Prometheus Custom Resource:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: prometheus
  namespace: monitoring
spec:
  additionalScrapeConfigs:
    name: additional-scrape-configs
    key: scrape-configs.yaml
  # ... rest of your Prometheus spec

2. Configure the Prometheus Adapter

The adapter translates Prometheus metrics into the Kubernetes custom metrics API so HPAs can consume them. Because Edera zone metrics already carry namespace and pod labels, they map directly to Kubernetes pod resources.

# prometheus-adapter-values.yaml
prometheus:
  url: http://prometheus.monitoring.svc.cluster.local
  port: 9090

rules:
  default: false

  custom:
  - seriesQuery: 'zone_cpu_usage_percent{namespace!="",pod!=""}'
    resources:
      overrides:
        namespace:
          resource: namespace
        pod:
          resource: pod
    name:
      as: zone_cpu_usage_percent
    metricsQuery: 'avg by (namespace, pod) (zone_cpu_usage_percent{<<.LabelMatchers>>})'

Install or upgrade the adapter:

helm upgrade --install prometheus-adapter prometheus-community/prometheus-adapter \
  --namespace monitoring \
  -f prometheus-adapter-values.yaml

Verify the metric is available

# Should list zone_cpu_usage_percent
kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1 | jq '[.resources[].name]'

# Should return a value per pod
kubectl get --raw \
  "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/zone_cpu_usage_percent" \
  | jq '.items[] | {pod: .describedObject.name, value}'

3. Create a Horizontal Pod Autoscaler (HPA)

With the metric available in the custom metrics API, create an HPA that scales on average zone CPU usage across pods:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: my-edera-app
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-edera-app
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: Pods
    pods:
      metric:
        name: zone_cpu_usage_percent
      target:
        type: AverageValue
        averageValue: "70"   # scale up when average vCPU usage exceeds 70%

Verify the HPA is working

# Check HPA status and current metric value
kubectl get hpa my-edera-app

# Watch scaling events
kubectl describe hpa my-edera-app

The TARGETS column in kubectl get hpa shows the current metric value against the target. If it reads <unknown>, the adapter is not reaching the metric – recheck the adapter config and confirm the metric is present in the custom metrics API.

Next steps

• Edera metrics reference for the full list of available zone metrics
• Zone security model to understand what the zone boundary means for your workloads