Pass custom metadata in sync

[justin-dropbase]

Tell us about the problem you're trying to solve

We'd like to be able to pass in custom metadata whenever we trigger a manual sync. We are using the S3 destination. This metadata would be used to link a file with which sync / connection it corresponds to, or to pass in some extra information that would be useful when processing the data further outside of Airbyte.

More details of the problem can be found in this Slack discussion.

Describe the alternative you’ve considered or used

Currently, we specify the prefix field when creating a connection to have some control over the name of the output file name. However, we'd like to be able to control the name whenever we sync too, not just when creating a connection.

Describe the solution you’d like

There are two possible ways to solve this - the first would be more useful for us, but the second may be more feasible to implement.

1. In the POST /v1/connections/sync endpoint, I'd like to be able to pass in an optional metadata field in the payload:

// POST /v1/connections/sync
{
connectionId*: uuid, 
syncCustomMetadata: {
  batch_id: "123e4567-e89b-12d3-a456-426614174000",
  another_field: 123,
  ...
  }
}

With the S3 destination, the fields in "syncCustomMetadata" can be assessed using S3's user-defined object metadata.

2. If the above is not feasible, alternatively, we would appreciate the ability to control the output name whenever we sync, perhaps by passing a prefix (like connections right now):

// POST /v1/connections/sync
{
connectionId*: uuid, 
prefix: "my_custom_prefix"
}

I know that we can achive this by simply updating the connection prefix before we sync every time, but we don't want to be doing that as it seems to be suboptimal for performance.