Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: Add & improve concurrent tests #1820

Merged
merged 122 commits into from
Nov 14, 2024

Conversation

jbelkins
Copy link
Contributor

@jbelkins jbelkins commented Nov 13, 2024

Description of changes

S3Concurrency tests:

  • Add a new high concurrency (200x), small payload (256 byte) S3 concurrency test
  • All S3Concurrency tests are now enabled on Apple platforms & URLSessionHTTPClient as well as on Linux & the CRT client
  • Update the logic for dummy data to randomize data & create data of exact count
  • Added some clarifying comments

TranscribeStreaming tests:

  • Add a new concurrent (25x) TranscribeStreaming test (currently disabled on CRT HTTP client due to spurious connection-related failures)
  • Add custom retry logic to TranscribeStreaming tests to ensure throttling & service-imposed resource limits don't fail test.

note: A tracking ticket will be written for the CRT HTTP client not passing bidirectional streaming concurrency tests

New/existing dependencies impact assessment, if applicable

No new dependencies were added to this change.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

jbelkins and others added 30 commits August 30, 2024 22:30
* Add concurrent tests

* Add missing 100x to a test's name

* Remove 100MiB file and generate file instead.

* See if using UUID for filename fixes this possibly-concurrency-issue.

* Try temporaryDirectory property.

* Fix concurrency issue

* Comment fixes & remove STS getCallerIdentity concurrent calls.

* Remove unused import

* Remove concurrent test for event stream output

* Refactor tests a bit

* Use FileStream to write to file

* Swap out filemanager with writing emtpy data approach

* Tweak a bit

* Simplify; no need to safe generated dummy data to a file, just use 50MB memory.

* Change from 10x50MB to 100x5MB

* Reduce payload from 5MB to 3MB

* Reduce payload from 3MB to 1MB.

* Add test for 1.5 MB to test aws chunked & flexible checksum flow also.

---------

Co-authored-by: Sichan Yoo <chanyoo@amazon.com>
Co-authored-by: Josh Elkins <jbelkins@users.noreply.github.com>
@jbelkins jbelkins changed the title Jbe/20241113 reenable concurrent tests chore: Add & improve concurrent tests Nov 13, 2024
@jbelkins jbelkins marked this pull request as ready for review November 13, 2024 23:56
@jbelkins jbelkins merged commit faee5dd into main Nov 14, 2024
29 checks passed
@jbelkins jbelkins deleted the jbe/20241113_reenable_concurrent_tests branch November 14, 2024 17:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants