Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How can I ungroup alerts in AlertManager #4246

Closed
tshuma1 opened this issue Feb 9, 2025 · 1 comment
Closed

How can I ungroup alerts in AlertManager #4246

tshuma1 opened this issue Feb 9, 2025 · 1 comment

Comments

@tshuma1
Copy link

tshuma1 commented Feb 9, 2025

Using the recommended notation for ungrouping (i.e. group_by: ['...']) , I have the following configuration to ungroup alerts but it is not working as expected:

      pagerduty_url: https://events.pagerduty.com/v2/enqueue
      resolve_timeout: 5m
      smtp_from: [email protected]
      smtp_require_tls: false
      smtp_smarthost: xxx.zw.corp.com:25
    inhibit_rules:
      - equal: ['alertname']
        source_matchers:
          - severity = critical
        target_matchers:
          - severity = warning
      - equal: ['namespace']
        source_matchers:
          - severity = warning
        target_matchers:
          - severity = info
      - source_matchers:
          - alertname = InfoInhibitor
        target_matchers:
          - severity = info
      - equal: ["host"]
        source_matchers:
          - alertname = HostDown
        target_matchers:
    receivers:
      - name: 'null'
      - name: pd_secops # https://mycorp.pagerduty.com/service-directory/PXXXX
        pagerduty_configs:
          - details:
            runbook_url: "{{ .CommonAnnotations.runbook_url }}"
            routing_key: "cd1xxxxx"
            severity: "{{ .CommonLabels.severity }}"
            send_resolved: false
      - name: 'email'
        email_configs:
          - to: '{{ .CommonLabels.email }}'
            from: '[email protected]'
            smarthost: 'xxx.zw.corp.com:25'
            require_tls: false
    route:
      group_by: ['alertname']
      group_interval: 2m
      group_wait: 1m
      receiver: 'null'
      repeat_interval: 8736h
      routes:
        - matchers:
            - alertname = Watchdog
          receiver: 'null'
        - matchers:
            - alertname = InfoInhibitor
          receiver: 'null'
        - matchers:
            - type = pagerduty
            - service = gitops
            - severity =~ warning|critical
          receiver: pd_gitops
          group_by: ['...']
          group_wait: 10s
          group_interval: 10s        

I can manually trigger several alerts on the associated Alert rule, within a 5m interval, but when I check the AlertManager UI I see only 1 alert instance.

A quick check of the Alert rule's PromQL returns the correct value as per the manual triggers I have done, but the number of alerts in AlertManager UI still does not match.

My expectation is that when I trigger, say 5 alerts within a 5m interval, the AlertManager UI should show , at most 5 separate alerts. For now I am only getting a single alert.

The Alert rule:

    alert = "repo.rename"
    expr  = "sum(count_over_time({stream=\"github_audit_log\"} | json | action = `repo.rename`[5m])) > 0"
    labels = {
      severity = "critical"
      source   = "github"
      type     = "pagerduty"
      service  = "gitops"
    }
    annotations = {
      managed_by  = "mycorp/iac"
      summary     = "A repository was renamed."
    }
  }```


What am I missing ?    Does the  `group_by: ['...']` notation actually work?
@grobinson-grafana
Copy link
Collaborator

Please use https://groups.google.com/g/prometheus-users for help. GitHub issues are for feature requests and bug reports.

@tshuma1 tshuma1 closed this as completed Feb 17, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants