Skip to content

fix: enforce user must specify access_key and secret_key using aws auth - #11120

Merged
tabVersion merged 6 commits into
mainfrom
tab/remove-aws-from-env
Jul 24, 2023
Merged

tabVersion merged 6 commits into
mainfrom
tab/remove-aws-from-env

Conversation

@tabVersion

@tabVersion tabVersion commented Jul 21, 2023 •

Copy link
Copy Markdown
Contributor

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

resolve #11086

Checklist

  • I have written necessary rustdoc comments
  • I have added necessary unit tests and integration tests
  • I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features Sqlsmith: Sql feature generation #7934).
  • My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
  • All checks passed in ./risedev check (or alias, ./risedev c)
  • My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)
  • My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

  • My PR contains user-facing changes.
Click here for Documentation

Types of user-facing changes

Please keep the types that apply to your changes, and remove the others.

  • Connector (sources & sinks)

Release note

access_key and corresponding secret_key become a must for all aws auth components.

@github-actions github-actions Bot added type/fix Type: Bug fix. Only for pull requests. user-facing-changes Contains changes that are visible to users labels Jul 21, 2023
@tabVersion
tabVersion requested review from arkbriar and fuyufjh July 21, 2023 07:36

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we reject it in fe?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But might be ok as a quick fix

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

still discussing with @arkbriar, whether we should continuously support public buckets.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

still discussing with @arkbriar, whether we should continuously support public buckets.

Prefer to keep it. It's up to user's choice, not our fault.

.with_role_provider(self.build_credential_provider().await?)
.with_role_provider(self.build_credential_provider()?)
.await?;
let config_loader = aws_config::from_env()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should change this?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the problem is about finding an alternative for from_env.

Important: Using the aws-config crate to configure the SDK is preferred to invoking this builder directly. Using this builder directly won’t pull in any AWS recommended default configuration values.

---- from doc

It is not recommended to build SdkConfig directly.

@huangjw806

Copy link
Copy Markdown
Contributor

If the bucket is Publicly accessible, can we don't provide AK/SK?

fuyufjh
fuyufjh previously approved these changes Jul 21, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

still discussing with @arkbriar, whether we should continuously support public buckets.

Prefer to keep it. It's up to user's choice, not our fault.

@fuyufjh
fuyufjh dismissed their stale review July 21, 2023 08:03

See @huangjw806 's comment

@tabVersion

Copy link
Copy Markdown
Contributor Author

If the bucket is Publicly accessible, can we don't provide AK/SK?

Just change to an empty string as default. We won't use our env var any more.

@tabVersion

tabVersion commented Jul 21, 2023 •

Copy link
Copy Markdown
Contributor Author

image

it looks like we cannot use a random or empty string as AK/SK, even for public access bucket.


conclusion: we provide the ability to access public buckets by using our own AK/SK. So we may not provide the func anymore. cc @fuyufjh @arkbriar

@fuyufjh

fuyufjh commented Jul 21, 2023

Copy link
Copy Markdown
Contributor

conclusion: we provide the ability to access public buckets by using our own AK/SK. So we may not provide the func anymore.

Well... Okay if that's too difficult, I can accept it

@codecov

codecov Bot commented Jul 21, 2023 •

Copy link
Copy Markdown

Codecov Report

Merging #11120 (0922e29) into main (7476772) will increase coverage by 0.00%.
The diff coverage is 0.00%.

@@           Coverage Diff           @@
##             main   #11120   +/-   ##
=======================================
  Coverage   69.92%   69.92%           
=======================================
  Files        1312     1312           
  Lines      223351   223347    -4     
=======================================
- Hits       156171   156170    -1     
+ Misses      67180    67177    -3     
Flag Coverage Δ
rust 69.92% <0.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
src/connector/src/aws_auth.rs 37.33% <0.00%> (+4.00%) ⬆️
src/connector/src/aws_utils.rs 0.00% <0.00%> (ø)
...c/connector/src/source/filesystem/s3/enumerator.rs 28.82% <0.00%> (-0.81%) ⬇️

... and 1 file with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@arkbriar arkbriar left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type/fix Type: Bug fix. Only for pull requests. user-facing-changes Contains changes that are visible to users 📖✓ Covered or will be covered in the user docs.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: credentials for external sources should be enforced

6 participants