Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Analytics: measure and display throughput and integration metrics #392

Open
michel-tricot opened this issue Sep 22, 2020 · 5 comments
Open
Labels
frozen Not being actively worked on team/platform-move type/enhancement New feature or request

Comments

@michel-tricot
Copy link
Contributor

michel-tricot commented Sep 22, 2020

Tell us about the problem you're trying to solve

I want to have visibility on how well and fast my integrations are performing.

Describe the solution you’d like

We instrument the scheduler and expose these metrics through an endpoint to make them consumable by the UI.

┆Issue is synchronized with this Asana task by Unito

@michel-tricot michel-tricot added the type/enhancement New feature or request label Sep 22, 2020
@vitoravancini
Copy link

Hello, I'm assuming this is not implemented yet right?

Looking at the execution logs it seems that you already have this metrics beeing generated at least right?

DefaultSyncWorker(run):119 - sync summary: io.airbyte.config.StandardSyncSummary@6e082cf1[status=completed,recordsSynced=1847,bytesSynced=3377990,startTime=1607477357262,endTime=1607516339965]

Having some analytics on top of this, specially including some alerting features, would be very nice. Maybe expose as a prometheus exporter also.

Is this on the roadmap?

@cgardens
Copy link
Contributor

cgardens commented Dec 9, 2020

@vitoravancini - hiya! the metrics that you found above will be exposed in the UI in the next few days. please feel free to share any other metrics you think would be valuable.

Having some analytics on top of this, specially including some alerting features, would be very nice.

definitely. this is something we know we want to do, but haven't quite figured out the priority yet. would not having this feature be a blocker for you to use the product? for your use case would basic email alerting be sufficient or would you need some more configurable setup (e.g. pushing to a SNS queue) to get off the ground?

Maybe expose as a prometheus exporter also.

I don't think we have thought through this one yet. Could you describe a little more what this would look like. Or if you could point to another project that does this, that would be helpful so that I can get a better sense of what this would look like.

@vitoravancini
Copy link

In my case simple email alerting would suffice, but without any kind of alerts or ways of fetching this metrics would be a show
stopper.

I'm not sure you aware of how prometheus works, but the basic idea is that you write a simple code for fetching metrics and expose them via http. Another prometheus service scrapes this exposed metrics and writes to prometheus timeseries database. Sorry if that part was obvious and I misunderstood your question.

But as I wrote this I realize that it seems this way of collecting metrics wouldn't work too well for batch jobs. Prometheus team has some instructions for batch jobs(https://prometheus.io/docs/practices/instrumentation/#batch-jobs)

Prometheus is not necessary, it just seems that would make easier for devops team to track success of Airbyte jobs and other systems in the same monitoring stack.

@marcosmarxm
Copy link
Member

@vitoravancini 👋

I found this https://github.com/databand-ai/airflow-dashboards/ using Grafana and statsd.
Exporting metrics using statsd and consuming using grafana or prometheus could be developed as an external add-on of Airbyte? This will very convenient when working with k8s or deploying in production env.

@NipunaPrashan
Copy link
Contributor

Do we have any update on this?
developed grafana dashboard files will be handy as @marcosmarxm mentioned.

@bleonard bleonard added the frozen Not being actively worked on label Mar 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
frozen Not being actively worked on team/platform-move type/enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

7 participants