-
Notifications
You must be signed in to change notification settings - Fork 4.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Analytics: measure and display throughput and integration metrics #392
Comments
Hello, I'm assuming this is not implemented yet right? Looking at the execution logs it seems that you already have this metrics beeing generated at least right?
Having some analytics on top of this, specially including some alerting features, would be very nice. Maybe expose as a prometheus exporter also. Is this on the roadmap? |
@vitoravancini - hiya! the metrics that you found above will be exposed in the UI in the next few days. please feel free to share any other metrics you think would be valuable.
definitely. this is something we know we want to do, but haven't quite figured out the priority yet. would not having this feature be a blocker for you to use the product? for your use case would basic email alerting be sufficient or would you need some more configurable setup (e.g. pushing to a SNS queue) to get off the ground?
I don't think we have thought through this one yet. Could you describe a little more what this would look like. Or if you could point to another project that does this, that would be helpful so that I can get a better sense of what this would look like. |
In my case simple email alerting would suffice, but without any kind of alerts or ways of fetching this metrics would be a show I'm not sure you aware of how prometheus works, but the basic idea is that you write a simple code for fetching metrics and expose them via http. Another prometheus service scrapes this exposed metrics and writes to prometheus timeseries database. Sorry if that part was obvious and I misunderstood your question. But as I wrote this I realize that it seems this way of collecting metrics wouldn't work too well for batch jobs. Prometheus team has some instructions for batch jobs(https://prometheus.io/docs/practices/instrumentation/#batch-jobs) Prometheus is not necessary, it just seems that would make easier for devops team to track success of Airbyte jobs and other systems in the same monitoring stack. |
I found this https://github.com/databand-ai/airflow-dashboards/ using Grafana and statsd. |
Do we have any update on this? |
Tell us about the problem you're trying to solve
I want to have visibility on how well and fast my integrations are performing.
Describe the solution you’d like
We instrument the scheduler and expose these metrics through an endpoint to make them consumable by the UI.
┆Issue is synchronized with this Asana task by Unito
The text was updated successfully, but these errors were encountered: