Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fixed issue with leased connection leaks when threads executing HTTP … #4066

Merged
merged 6 commits into from
Jun 9, 2023
Merged

Fixed issue with leased connection leaks when threads executing HTTP … #4066

merged 6 commits into from
Jun 9, 2023

Conversation

joviegas
Copy link
Contributor

@joviegas joviegas commented Jun 6, 2023 •
edited
Loading

Fixed issue with leased connection leaks when threads executing HTTP connections with Apache HttpClient were interrupted while the connection was in progress.

Motivation and Context

  • The Apache connection pool leases were not released when Thread was interrupted while Connection was in progress.
  • The fix will release the resources in case if the thread was interrupted during the get.

Root cause

  • When a Thread is interrupted externally while httpClient.execute() is in progress , it is not necessary that the Httpclient interrupts the threads because of http.Client execute call completes successfully .
  • At a later stage in SDK Execute Pipeline the thread interrupt gets checked and throws execption whithout closing or aborting the stream , this causes the Resource leaks.

Modifications

  • Close the response output stream in AfterTransmissionExecutionInterceptorsStage by passing response in checkInterrupted API.

Testing

Screenshots (if appropriate)

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)

Checklist

  • I have read the CONTRIBUTING document
  • Local run of mvn install succeeds
  • My code follows the code style of this project
  • My change requires a change to the Javadoc documentation
  • I have updated the Javadoc documentation accordingly
  • I have added tests to cover my changes
  • All new and existing tests passed
  • I have added a changelog entry. Adding a new entry must be accomplished by running the scripts/new-change script and following the instructions. Commit the new file created by the script in .changes/next-release with your changes.
  • My change is to implement 1.11 parity feature and I have updated LaunchChangelog

License

  • I confirm that this pull request can be released under the Apache 2 license

@joviegas
Fixed issue with leased connection leaks when threads executing HTTP …
7b6c66e 
…connections with Apache HttpClient were interrupted while the connection was in progress.
@joviegas joviegas requested a review from a team as a code owner June 6, 2023 01:40
@joviegas
Copy link
Contributor Author

joviegas commented Jun 6, 2023 •
edited
Loading

After internal discussion, it has been decided to implement a change in the Client core. Moving forward, the responsibility of closing the stream will be shifted to the Client core instead of aborting when interrupted in the SdkApache layer. This adjustment is being made to ensure consistency in the contract of the Apache client, as we aim to avoid mere patching at the SdkApache layer. By making this change at the core level, we can maintain a unified approach across the entire SDK.

@joviegas joviegas force-pushed the joviegas/apache_lease_leak branch 3 times, most recently from 8e30f41 to d455b45 Compare June 7, 2023 20:16
@joviegas joviegas force-pushed the joviegas/apache_lease_leak branch from d455b45 to bf16fd5 Compare June 7, 2023 21:49
@joviegas
Moved the fix to AfterTransmissionExecutionInterceptorsStage to just …
31a12fc 
…close the stream instaed of aborting the reqyest in MakeHttpRequestStage
@joviegas
Removing test cases related to UrlConnectionHttp since adding depenen…
f0ce7cb 
…cy in protocol-test for urlConnectionClient cause failues since it uses default Client all the places
@joviegas joviegas force-pushed the joviegas/apache_lease_leak branch from 44131d5 to f0ce7cb Compare June 8, 2023 20:34
zoewangg
zoewangg reviewed Jun 8, 2023
View reviewed changes
...a/software/amazon/awssdk/protocol/tests/connection/SyncClientConnectionInterruptionTest.java Outdated
SdkHttpClient httpClient = ApacheHttpClient.builder().maxConnections(1).build();
ProtocolRestJsonClient client = getClient(httpClient, Duration.ofMillis(2L * LONG_DELAY)).build();

ExecutorService executorService = Executors.newFixedThreadPool(5);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we reuse the executor across all tests? This may speed up tests a bit

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

...a/software/amazon/awssdk/protocol/tests/connection/SyncClientConnectionInterruptionTest.java Outdated Show resolved Hide resolved
.changes/next-release/bugfix-ApacheHTTPClient-bf0095f.json Outdated
@@ -0,0 +1,6 @@
{
"type": "bugfix",
"category": "Apache HTTP Client",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: this is not just applicable to Apache HTTP client right? Should we use AWS SDK for Java v2 or whatever the right category for core?

...a/software/amazon/awssdk/protocol/tests/connection/SyncClientConnectionInterruptionTest.java Show resolved Hide resolved
...a/software/amazon/awssdk/protocol/tests/connection/SyncClientConnectionInterruptionTest.java Outdated


@Test
void sdkClientInterrupted_while_connectionIsInProgress() throws InterruptedException {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How is this different from connectionPoolsGetsReusedWhenInterruptedWith_1_MaxConnection?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was added to test default settings with UrlConnection , its no longer needed.

...a/software/amazon/awssdk/protocol/tests/connection/SyncClientConnectionInterruptionTest.java Outdated
}

@Test
void connectionPoolsGetsReusedWhenInterruptedWith_Multiple_MaxConnection() throws Exception {
Copy link
Contributor

@zoewangg zoewangg Jun 8, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we actually need this test? It seems we have verified the behavior (connection gets released back to the pool) in connectionPoolsGetsReusedWhenInterruptedWith_1_MaxConnection. My concern with tests covering the same area is that it could increase the build time and the likelihood of test flakiness

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removing this test case

@joviegas joviegas force-pushed the joviegas/apache_lease_leak branch from 131783a to 7494fb2 Compare June 8, 2023 23:21
zoewangg
zoewangg approved these changes Jun 8, 2023
View reviewed changes
...a/software/amazon/awssdk/protocol/tests/connection/SyncClientConnectionInterruptionTest.java Show resolved Hide resolved
...a/software/amazon/awssdk/protocol/tests/connection/SyncClientConnectionInterruptionTest.java Outdated
private final WireMockServer mockServer = new WireMockServer(new WireMockConfiguration()
.bindAddress("localhost").dynamicPort());

ExecutorService executorService = Executors.newCachedThreadPool();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit, private static

@joviegas joviegas force-pushed the joviegas/apache_lease_leak branch from 7494fb2 to 7d670cc Compare June 9, 2023 01:39
@sonarqubecloud
Copy link

sonarqubecloud bot commented Jun 9, 2023

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 3 Code Smells

95.8% 95.8% Coverage
0.7% 0.7% Duplication

@joviegas joviegas merged commit 15abae1 into master Jun 9, 2023
@klsince klsince mentioned this pull request Jun 12, 2023
Merged
@debora-ito debora-ito mentioned this pull request Jun 21, 2023
Closed
davidh44 added a commit that referenced this pull request Jun 21, 2023
@davidh44 @joviegas
@martinKindall @millems @aws-sdk-java-automation @StephenFlavin @zoewangg @allcontributors @debora-ito @breader124 @L-Applin @bmaizels @flittev
Merge from master (#4126)
95b5a0c 
* Fixed issue with leased connection leaks when threads executing HTTP … (#4066)

* Fixed issue with leased connection leaks when threads executing HTTP connections with Apache HttpClient were interrupted while the connection was in progress.

* Added logic in MakeHttpRequestStage to check and abort request if interrupted

* Add test cases for UrlConnectionHttpClient

* Moved the fix to AfterTransmissionExecutionInterceptorsStage to just close the stream instaed of aborting the reqyest in MakeHttpRequestStage

* Removing test cases related to UrlConnectionHttp since adding depenency in protocol-test for urlConnectionClient cause failues since it uses default Client all the places

* Updated after Zoe's comments

* Now it's possible to configure NettyNioAsyncHttpClient for non blocking DNS (#3990)

* Now it's possible to configure NettyNioAsyncHttpClient in order to use a
non blocking DNS resolver.

* Add package mapping for netty-resolver-dns.

---------

Co-authored-by: Matthew Miller <[email protected]>

* Amazon Connect Service Update: This release adds search APIs for Prompts, Quick Connects and Hours of Operations, which can be used to search for those resources within a Connect Instance.

* AWS Certificate Manager Private Certificate Authority Update: Document-only update to refresh CLI documentation for AWS Private CA. No change to the service.

* Release 2.20.83. Updated CHANGELOG.md, README.md and all pom.xml.

* Add "unsafe" AsyncRequestBody constructors for byte[] and ByteBuffers (#3925)

* Update to next snapshot version: 2.20.84-SNAPSHOT

* Use WeakHashMap in IdleConenctionReaper  (#4087)

* Use WeakHashMap in IdleConenctionReaper to not prevent connection manager from getting GC'd

* Checkstyle fix

* Update S3IntegrationTestBase.java (#4079)

* Amazon Rekognition Update: This release adds support for improved accuracy with user vector in Amazon Rekognition Face Search. Adds new APIs: AssociateFaces, CreateUser, DeleteUser, DisassociateFaces, ListUsers, SearchUsers, SearchUsersByImage. Also adds new face metadata that can be stored: user vector.

* Amazon DynamoDB Update: Documentation updates for DynamoDB

* Amazon FSx Update: Amazon FSx for NetApp ONTAP now supports joining a storage virtual machine (SVM) to Active Directory after the SVM has been created.

* Amazon SageMaker Service Update: Sagemaker Neo now supports compilation for inferentia2 (ML_INF2) and Trainium1 (ML_TRN1) as available targets. With these devices, you can run your workloads at highest performance with lowest cost. inferentia2 (ML_INF2) is available in CMH and Trainium1 (ML_TRN1) is available in IAD currently

* AWS Amplify UI Builder Update: AWS Amplify UIBuilder is launching Codegen UI, a new feature that enables you to generate your amplify uibuilder components and forms.

* Amazon OpenSearch Service Update: This release adds support for SkipUnavailable connection property for cross cluster search

* Amazon DynamoDB Streams Update: Documentation updates for DynamoDB Streams

* Updated endpoints.json and partitions.json.

* Release 2.20.84. Updated CHANGELOG.md, README.md and all pom.xml.

* Update to next snapshot version: 2.20.85-SNAPSHOT

* docs: add scrocquesel as a contributor for code (#4091)

* docs: update README.md [skip ci]

* docs: update .all-contributorsrc [skip ci]

---------

Co-authored-by: allcontributors[bot] <46447321+allcontributors[bot]@users.noreply.github.com>
Co-authored-by: Debora N. Ito <[email protected]>

* AWS CloudTrail Update: This feature allows users to view dashboards for CloudTrail Lake event data stores.

* AWS WAFV2 Update: You can now detect and block fraudulent account creation attempts with the new AWS WAF Fraud Control account creation fraud prevention (ACFP) managed rule group AWSManagedRulesACFPRuleSet.

* AWS Well-Architected Tool Update: AWS Well-Architected now supports Profiles that help customers prioritize which questions to focus on first by providing a list of prioritized questions that are better aligned with their business goals and outcomes.

* Amazon Lightsail Update: This release adds pagination for the Get Certificates API operation.

* Amazon Verified Permissions Update: GA release of Amazon Verified Permissions.

* EC2 Image Builder Update: Change the Image Builder ImagePipeline dateNextRun field to more accurately describe the data.

* Amazon CodeGuru Security Update: Initial release of Amazon CodeGuru Security APIs

* Amazon Simple Storage Service Update: Integrate double encryption feature to SDKs.

* Elastic Disaster Recovery Service Update: Added APIs to support network replication and recovery using AWS Elastic Disaster Recovery.

* AWS SimSpace Weaver Update: This release fixes using aws-us-gov ARNs in API calls and adds documentation for snapshot APIs.

* AWS SecurityHub Update: Add support for Security Hub Automation Rules

* Amazon Elastic Compute Cloud Update: This release introduces a new feature, EC2 Instance Connect Endpoint, that enables you to connect to a resource over TCP, without requiring the resource to have a public IPv4 address.

* Updated endpoints.json and partitions.json.

* Release 2.20.85. Updated CHANGELOG.md, README.md and all pom.xml.

* Update to next snapshot version: 2.20.86-SNAPSHOT

* Create secondary indices based on table bean annotations (#3923) (#4004)

* Create secondary indices based on table bean annotations (#3923)

* detect and group indices present in table schema into LSIs and GSIs
* pass request with indices information appended further

* Remove specifying provisioned throughput for GSIs (#3923)

* If there's no information about the billing mode of the new table,
  then it'll be using the PAY_PER_REQUEST one. It means that all
  GSIs related to this table will be doing the same and there's
  no need to hard code any provisioned throughput like it was done

* Allow passing empty indices list to CreateTableOperation (#3923)

* CreateTableRequest cannot handle empty list of indices of any type. It
  throws exception when given such a list. At the same time, it nicely
  handles the cases when indices lists are null. Make sure then that
  when empty indices list is passed CreateTableOperation, then in the
  CreateTableRequest it's just reflected as null.

---------

Co-authored-by: Adrian Chlebosz <[email protected]>
Co-authored-by: Olivier L Applin <[email protected]>

* Add EnhancedType parameters to static builder methods of StaticTableSchema and StaticImmitableTableSchema (#4077)

* Amazon Elastic File System Update: Documentation updates for EFS.

* Amazon GuardDuty Update: Updated descriptions for some APIs.

* Amazon Location Service Update: Amazon Location Service adds categories to places, including filtering on those categories in searches. Also, you can now add metadata properties to your geofences.

* AWS Audit Manager Update: This release introduces 2 Audit Manager features: CSV exports and new manual evidence options. You can now export your evidence finder results in CSV format. In addition, you can now add manual evidence to a control by entering free-form text or uploading a file from your browser.

* Updated endpoints.json and partitions.json.

* Release 2.20.86. Updated CHANGELOG.md, README.md and all pom.xml.

* Update to next snapshot version: 2.20.87-SNAPSHOT

* EnumAttributeConverter: enums can be identified by toString() or name(). toString() is the default for backward compatibility (#3971)

Co-authored-by: Zoe Wang <[email protected]>

* AWS Application Discovery Service Update: Add Amazon EC2 instance recommendations export

* AWS Account Update: Improve pagination support for ListRegions

* Amazon Simple Storage Service Update: This release adds SDK support for request-payer request header and request-charged response header in the "GetBucketAccelerateConfiguration", "ListMultipartUploads", "ListObjects", "ListObjectsV2" and "ListObjectVersions" S3 APIs.

* Amazon Connect Service Update: Updates the *InstanceStorageConfig APIs to support a new ResourceType: SCREEN_RECORDINGS to enable screen recording and specify the storage configurations for publishing the recordings. Also updates DescribeInstance and ListInstances APIs to include InstanceAccessUrl attribute in the API response.

* AWS Identity and Access Management Update: Documentation updates for AWS Identity and Access Management (IAM).

* Release 2.20.87. Updated CHANGELOG.md, README.md and all pom.xml.

* Update to next snapshot version: 2.20.88-SNAPSHOT

* Fix the StackOverflowException in WaiterExecutor in case of large retries count. (#3956)

* Move checksum calculation from afterMarshalling to modifyHttpRequest (#4108)

* Update HttpChecksumRequiredInterceptor

* Update HttpChecksumInHeaderInterceptor

* Update tests and remove constant

* Add back constant to resolve japicmp

* Add back javadocs

* docs: add dave-fn as a contributor for code (#4092)

* docs: update README.md [skip ci]

* docs: update .all-contributorsrc [skip ci]

* Removing unnecessary vscode file

---------

Co-authored-by: allcontributors[bot] <46447321+allcontributors[bot]@users.noreply.github.com>
Co-authored-by: Debora N. Ito <[email protected]>

* Amazon Route 53 Domains Update: Update MaxItems upper bound to 1000 for ListPricesRequest

* Amazon EC2 Container Service Update: Documentation only update to address various tickets.

* AWS CloudFormation Update: Specify desired CloudFormation behavior in the event of ChangeSet execution failure using the CreateChangeSet OnStackFailure parameter

* AWS Price List Service Update: This release updates the PriceListArn regex pattern.

* AWS Glue Update: This release adds support for creating cross region table/database resource links

* Amazon Elastic Compute Cloud Update: API changes to AWS Verified Access to include data from trust providers in logs

* Amazon SageMaker Service Update: Amazon Sagemaker Autopilot releases CreateAutoMLJobV2 and DescribeAutoMLJobV2 for Autopilot customers with ImageClassification, TextClassification and Tabular problem type config support.

* Release 2.20.88. Updated CHANGELOG.md, README.md and all pom.xml.

* Update to next snapshot version: 2.20.89-SNAPSHOT

* AWS Lambda Update: This release adds RecursiveInvocationException to the Invoke API and InvokeWithResponseStream API.

* AWS Config Update: Updated ResourceType enum with new resource types onboarded by AWS Config in May 2023.

* Amazon Appflow Update: This release adds new API to reset connector metadata cache

* Amazon Elastic Compute Cloud Update: Adds support for targeting Dedicated Host allocations by assetIds in AWS Outposts

* Amazon Redshift Update: Added support for custom domain names for Redshift Provisioned clusters. This feature enables customers to create a custom domain name and use ACM to generate fully secure connections to it.

* Updated endpoints.json and partitions.json.

* Release 2.20.89. Updated CHANGELOG.md, README.md and all pom.xml.

* Update to next snapshot version: 2.20.90-SNAPSHOT

* Move QueryParametersToBodyInterceptor to front of interceptor chain (#4109)

* Move QueryParametersToBodyInterceptor to front of interceptor chain

* Move customization.config interceptors to front of interceptor chain - for query protocols

* Refactoring

* Add codegen tests

* Refactoring

* Refactoring

---------

Co-authored-by: John Viegas <[email protected]>
Co-authored-by: Martin <[email protected]>
Co-authored-by: Matthew Miller <[email protected]>
Co-authored-by: AWS <>
Co-authored-by: aws-sdk-java-automation <[email protected]>
Co-authored-by: Stephen Flavin <[email protected]>
Co-authored-by: Zoe Wang <[email protected]>
Co-authored-by: allcontributors[bot] <46447321+allcontributors[bot]@users.noreply.github.com>
Co-authored-by: Debora N. Ito <[email protected]>
Co-authored-by: Adrian Chlebosz <[email protected]>
Co-authored-by: Adrian Chlebosz <[email protected]>
Co-authored-by: Olivier L Applin <[email protected]>
Co-authored-by: Benjamin Maizels <[email protected]>
Co-authored-by: flitt <[email protected]>
L-Applin pushed a commit that referenced this pull request Jul 24, 2023
@joviegas @L-Applin
Fixed issue with leased connection leaks when threads executing HTTP … (
3982769 
#4066)

* Fixed issue with leased connection leaks when threads executing HTTP connections with Apache HttpClient were interrupted while the connection was in progress.

* Added logic in MakeHttpRequestStage to check and abort request if interrupted

* Add test cases for UrlConnectionHttpClient

* Moved the fix to AfterTransmissionExecutionInterceptorsStage to just close the stream instaed of aborting the reqyest in MakeHttpRequestStage

* Removing test cases related to UrlConnectionHttp since adding depenency in protocol-test for urlConnectionClient cause failues since it uses default Client all the places

* Updated after Zoe's comments
@joviegas joviegas deleted the joviegas/apache_lease_leak branch August 30, 2023 22:58
@shourya035 shourya035 mentioned this pull request Jan 24, 2024
Closed
@sachinpkale sachinpkale mentioned this pull request Feb 8, 2024
Merged
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zoewangg zoewangg zoewangg approved these changes

@L-Applin L-Applin L-Applin approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@joviegas @L-Applin @zoewangg