Implements AWS SigV4 for the HTTP output plugin.

[dlvenable]

Which issue(s) this PR fixes:
Fixes #4444

What this PR does / why we need it:

This PR adds a new authentication method to the HTTP output plugin for AWS sigV4. We need it so that customers can use Fluentd to send data to Amazon OpenSearch Ingestion which supports the Fluentd output plugin, but requires SigV4 authentication.

Docs Changes:

• aws_sigv4 - When this option is specified, Fluentd will sign requests using AWS Signature Version 4.

• aws_service - The AWS service to authenticate against

• aws_region - The AWS region to use when authenticating

• aws_role_arn - The AWS role ARN to assume when authenticating

You can optionally specify an aws_role_arn. If you provide it, Fluentd will assume that AWS role and send requests signing from that role. Otherwise, Fluentd will use the credentials found by the credential provider chain as defined in the AWS documentation.

Release Note:

The http output plugin supports AWS Signature Version 4 authentication.

[daipom]

@dlvenable Thanks so much for this enhancement!
I am very happy that adding AWS SigV4 authentication will allow Fluentd to work with AWS services such as Prepper!

[dlvenable]

Thank you all for the useful feedback. I've pushed changes to address these comments.

[daipom]

I commented on my concerns, but it looks basically good to me.
Thanks so much for this enhancement!

[cosmo0920]

Basically, it looks good to me. I added a comment about XML parsers.
aws-sdk-ruby needs to work with XML parsers to handle XML responses from AWS services.

[dlvenable]

@daipom , @cosmo0920 , I see that some tests are failing, but they don't appear to be related to http. Are these possibly flaky tests?

I'm very interested in getting this PR in to allow us to move forward. Thank you.

[dlvenable]

@ashie , It appears that the comment about gems still has this in changes requested.

[daipom]

I see that some tests are failing, but they don't appear to be related to http. Are these possibly flaky tests?

Yes. These tests are unstable. We can ignore it.

[cosmo0920]

@daipom , @cosmo0920 , I see that some tests are failing, but they don't appear to be related to http. Are these possibly flaky tests?

I'm very interested in getting this PR in to allow us to move forward. Thank you.

These failed tests are flaky tests on macOS and Windows. They should be able to be ignored for now.

@daipom @ashie Any ETA for getting merged this PR? In Fluent Bit side, SigV4 signature is set up in the almost HTTP related plugins. So, I'm also interested in to equip the similar functionality as a parity for Fluent Bit.
And this PR should be prioritized to process.

[ashie]

@daipom @ashie Any ETA for getting merged this PR?

It's almost ready for merge.
We'll merge soon when we get @dlvenable's response about #4459 (comment)

[daipom]

Yes. I think we can merge this soon, at least once the concerns pointed out by @ashie are resolved.

[dlvenable]

@daipom , @ashie , Thank you for the feedback. I have addressed the latest comments int the most recent commit. Please take another look.

[ashie]

Thanks for your contribution!

[daipom]

@dlvenable Thanks for your contribution!