Skip to content

Show webhook failure labels whenever last webhook failed

Summary

With #329849 (closed) merged, webhooks will be disabled if they fail to deliver the payload.

Currently, a Failed to connect or Fails to connect label is shown next to the failing webhooks based on the error codes returned. But, if the error type is error (response code), GitLab will show Fails to connect after the fourth failed attempt and then the label disappears after the webhook is disabled. This behaviour makes it difficult for users to identify which webhook has failed, and it requires them to check each webhook individually by clicking on the edit button and viewing the logs. So, either the Fails to connect label should persist or a new label must be added after a webhook is disabled automatically.

Also, two other feedbacks received on this matter:

  1. It's counterintuitive to click on edit button to view the logs.
  2. Errors related to timeout and network must be treated as 5xx type of errors so that they are always retried and re-enabled once the service comes back online.

ZD(internal link) where this issue was first reported.

Steps to reproduce

  1. Create a webhook with a non-existing IP address so that the request times out.
  2. Keep it running to let it fail 4 times so that it shows Fails to connect label.
  3. Check back after 10 minutes (probably after the update project statistics worker is run). You'd find the label disappeared.

Example Project

https://gitlab.com/psureshbabu/webhook-auto-disable

What is the current bug behavior?

Fails to connect label disappears after sometime.

What is the expected correct behavior?

Fails to connect label or a Disabled label should persist until the webhook is re-enabled.

Relevant logs and/or screenshots

Sample project: https://gitlab.com/psureshbabu/webhook-auto-disable

Webhook status after first four failures:

image

Webhook status after 10 minutes:

image

Output of checks

Results of GitLab environment info

Happens on GitLab.com and Self-Managed.

Results of GitLab application Check

Expand for output related to the GitLab application check 
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)


(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)


(we will only investigate if the tests are passing)

Possible fixes

Edited by Priyan Sureshbabu