feat: add allow_create_stream to avoid create nats stream by mistake - #22315
Conversation
…ector configuration
There was a problem hiding this comment.
Pull Request Overview
This PR adds an allow_create_stream flag to the NATS connector options to prevent accidental creation of JetStream streams by default.
- Introduces
allow_create_streamin source and sink YAML definitions with a default offalse. - Extends
NatsCommonto parse the new flag and defaults it tofalse. - Updates
build_or_get_streamto error when a stream is missing andallow_create_streamis not set, and refreshes integration tests and docs accordingly.
Reviewed Changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| src/connector/with_options_source.yaml | Add allow_create_stream property to source connector YAML |
| src/connector/with_options_sink.yaml | Add allow_create_stream property to sink connector YAML |
| src/connector/src/connector_common/common.rs | Add allow_create_stream field and enforce guard in stream builder |
| integration_tests/nats/* | Update SQL tests and README to include allow_create_stream |
Comments suppressed due to low confidence (5)
src/connector/src/connector_common/common.rs:956
- [nitpick] Consider renaming
stream_strtostream_namefor clarity and consistency with other parts of the codebase.
stream_str: String,
src/connector/src/connector_common/common.rs:838
- Add a rustdoc comment explaining the purpose of
allow_create_streamand its default behavior (false) so users understand its effect without reading the implementation.
pub allow_create_stream: bool,
src/connector/src/connector_common/common.rs:968
- Consider adding a unit test for
build_or_get_streamthat verifies it errors whenallow_create_streamisfalseand the stream does not exist.
if !self.allow_create_stream {
src/connector/with_options_source.yaml:695
- [nitpick] Using
Default::defaultin YAML may be ambiguous to users of generated docs; consider specifyingdefault: falseexplicitly for clarity.
default: Default::default
src/connector/src/connector_common/common.rs:969
- The error message includes the stream name and backticks around the config field; ensure this matches any consumers or tests that expect a specific substring.
return Err(anyhow!(
…ts on stream creation and existence requirements
| #[serde(rename = "allow_create_stream", default)] | ||
| #[serde_as(as = "DisplayFromStr")] | ||
| pub allow_create_stream: bool, |
There was a problem hiding this comment.
Will existing sources created before this PR also default to false, leading to them unable to create the stream automatically?
Similar issue: #22206
There was a problem hiding this comment.
The issue happens with little possibility.
For most existing streaming job, once they start the source, there must be a stream running. If the cluster experience a recovery, it will not hit the check here, it gets the stream directly without creating one.
One exception is, drop the stream and rely on RisingWave to recreate it. I think it is a wrong usage and we can ignore the case. Because creating a stream materializes some data on Nats server, this should be handled with caution.
#22315) Co-authored-by: tab <tabversion@bupt.icu>
|
✅ Cherry-pick PRs (or issues if encountered conflicts) have been created successfully to all target branches. |
I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.
resolve #21797
per request by poc user
What's changed and what's your intention?
Problem
The NATS connector currently creates streams automatically when they don't exist, which can lead to unintended stream creation in production environments. This poses security and operational risks:
Streams may be created with default configurations that don't match production requirements
Typos in stream names could result in unwanted streams being created
No explicit control over when RisingWave should have stream creation permissions
Solution
This PR introduces a new boolean configuration parameter allow_create_stream that provides explicit control over stream creation behavior:
Default: allow_create_stream = false - RisingWave will NOT create streams automatically
Explicit permission: Users must set allow_create_stream = true to enable stream creation
Clear error messaging: When a stream doesn't exist and creation is disabled, users get a helpful error message
Checklist
Documentation
Release note