Skip to content

feat: side panel data pipeline health - #44100

Merged
clr182 merged 8 commits into
masterfrom
HEALTH
Dec 30, 2025
Merged

clr182 merged 8 commits into
masterfrom
HEALTH

Conversation

@clr182

@clr182 clr182 commented Dec 30, 2025 •

Copy link
Copy Markdown
Member

Problem

Partially raised by this user: https://posthoghelp.zendesk.com/agent/tickets/46012 (moved to PostHog: https://us.posthog.com/project/2/support/tickets/48024)

Right now we don't have a great way of alerting customers of potential data pipeline issues. ideally this would be done by alerts in the future e.g. slack or email.

Changes

This returns failed/disabled data pipeline items and failed mat views and presents them in a new side panel. Side panel item is visible only when there is an issue. It lists all failed items with error message and a link to the item.
Screenshot 2025-12-30 at 12 00 36

How did you test this code?

Tested locally and added some dummy data, removed for PR.

Publish to changelog?

No

@pr-assigner-resolver-posthog
pr-assigner-resolver-posthog Bot requested a review from a team December 30, 2025 12:09
@clr182
clr182 requested review from Gilbert09 and andrewjmcgehee and removed request for a team December 30, 2025 12:09

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 issues found across 6 files

Prompt for AI agents (all issues)

Check if these issues are valid — if so, understand the root cause of each and fix them.


<file name="products/data_warehouse/backend/api/data_warehouse.py">

<violation number="1" location="products/data_warehouse/backend/api/data_warehouse.py:607">
P2: Silent exception handling with `except Exception: pass` hides errors and makes debugging difficult. Consider logging the exception at minimum.</violation>

<violation number="2" location="products/data_warehouse/backend/api/data_warehouse.py:653">
P2: Sorting logic puts `None` values first, not last as the comment states. With `reverse=True`, tuples where `failed_at is None` evaluate to `(True, &quot;&quot;)` which sorts before `(False, &quot;date&quot;)` in descending order.</violation>
</file>

Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.

Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>
}
)
except Exception:
pass

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question: why are we not logging this exception?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updating this to log a warning, same as transformations

)

# Get failed batch exports
failed_batch_exports = BatchExport.objects.filter(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we limit for the latest runs instead of loading all of them? (thinking about performance in the long term)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

^ that's a much better idea

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 1 file (changes from recent commits).

Prompt for AI agents (all issues)

Check if these issues are valid — if so, understand the root cause of each and fix them.


<file name="products/data_warehouse/backend/api/data_warehouse.py">

<violation number="1" location="products/data_warehouse/backend/api/data_warehouse.py:659">
P3: Consider using structured logging kwargs instead of f-string for better log searchability and aggregation. With f-strings, each unique `destination.id` creates a unique message, making it harder to query logs by destination ID.</violation>
</file>

Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.

Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>
@estefaniarabadan

Copy link
Copy Markdown
Contributor

overall looks good, the cubic-dev comment on the sorting situation looks legit, but that's it

@clr182
clr182 enabled auto-merge (squash) December 30, 2025 15:51
@clr182
clr182 merged commit 95eacfd into master Dec 30, 2025
179 checks passed
@clr182
clr182 deleted the HEALTH branch December 30, 2025 16:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants