Use pointblank for Databricks pyspark. #351
Unanswered
dikeshmoody
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
Hi,
I have been a big fan of the Pointblank R package specially the ease of defining checks outside the code & the reporting capabilities.
So I am excited that the package has been ported to Python to support different execution engines.
I am exploring to check if it matured enough to be used for databricks pyspark sources.
I understand we use ibis framework in the backend.
Is the package matured enough to support spark.
Our use cases mainly involved defining dq checks in a config table & run them as part of our data pipeline & have reports generated & made available for the end users to review.
We have data volume ranging from 1 billion to 20 billion rows.
Beta Was this translation helpful? Give feedback.
All reactions