materialize-databricks: new connector

[mdibaiee]

Description:

• Databricks uses the Recovery Log is authoritative & Idempotent apply pattern of a materialization
• This has been tested using: integration tests, and manual testing on the 25m document collection (imported_25m)

Workflow steps:

(How does one use this feature, and how has it changed)

Documentation links affected:

(list any documentation links that you created, or existing ones that you've identified as needing updates, along with a brief description)

Notes for reviewers:

(anything that might help someone review this PR)

---

This change is 

[mdibaiee]

What remains to be tested: since I have changed the logic of the CounterWriter, we need to test whether compression is still working fine. The non-compression case is working as I have tested it with Databricks.

UPDATE: tested this by running bigquery's integration tests 👍🏽

[williamhbaker]

Some initial comments/considerations - looking good so far, this one is a bit of a beast.

[williamhbaker]

For consistency with other materializations and usability, I suggest:

• Not setting a default for schema in tableConfig
• Making schema optional in tableConfig
• In config, make schema_name a required field (note: I am assuming that you can't connect without some kind of schema specified), and maybe make the name of the equivalent fields in the tableConfig and config be the same
• In (*config).Validate, error if schema_name hasn't been provided
• In newTableConfig, by default initialize the schema to the schema from config, which will be over-written if there is a present schema property in the raw json of the resource config when it is unmarshalled elsewhere

This would support the case where most bindings use the same schema as set in the endpoint-level config, but just a few need to be edited to use a different schema, which can be done by just configuring it for those specific bindings. It would directly parallel the BigQuery and Snowflake materializations, which work in this way. As it is now, if you wanted to materialize all the bindings to a schema other than "default", you would need to set it for every single binding.

[williamhbaker]

LGTM % remaining unresolved comments

[mdibaiee]

Merging of this depends on gazette/core#352 so holding off until that PR is merged and landed