-
Notifications
You must be signed in to change notification settings - Fork 275
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
remove OpenSearch build.sh
: moved to OpenSearch repo
#2835
base: main
Are you sure you want to change the base?
remove OpenSearch build.sh
: moved to OpenSearch repo
#2835
Conversation
important: i'm not aware of a way to properly test this manually, thus this remains untested! |
6f52e99
to
e6904be
Compare
`build.sh` has been copied over to all active branches(*) of the OpenSearch repo and can now consequently be deleted here. this means, that from now on, the scripts from the respository will be picked up. for more information on the order of evaluation of the custom build scripts, see the [documentation][]. this gives the advantage to have branch-specific new logic in place there. this has already been used to add the no-jdk targets to the builds on main and 2.x. two tests were based on the assumption that there would be a `build.sh` for OpenSearch in this repository, however these are generic tests and just used OpenSearch as an example. they have been switched to be based on OpenSearch-Dashboards instead which - for the time being - still has its own `build.sh` here. (*): the following branches have been considered as active: - main - 2.x - 2.3 - 1.x - 1.3 this is part of opensearch-project#99 [documentation]: https://github.com/opensearch-project/opensearch-build/tree/main/src/build_workflow#custom-build-scripts Signed-off-by: Ralph Ursprung <Ralph.Ursprung@avaloq.com>
e6904be
to
e6288a3
Compare
Codecov Report
@@ Coverage Diff @@
## main #2835 +/- ##
=======================================
Coverage 94.15% 94.15%
=======================================
Files 157 157
Lines 4242 4242
=======================================
Hits 3994 3994
Misses 248 248 Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@peterzhuamazon any issues?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @rursprung
I wonder if we can block this until 2.4.0 is released to public.
We are at the edge of the release in about a week or so.
I want to make sure this has minimum impact.
Let me know what you think.
Thanks.
i was sort-of hoping to get this in for 2.4.0 😄. |
@peterzhuamazon it shouldn't break anything should it? we just moved a bash file |
We are building release candidate of 2.4.0 tomorrow. |
Hi @rursprung we have seen there is changes to add no-jdk build in build.sh.
We need some additional time to review and plan the no-jdk addition. Thanks again for the contribution and let us discuss more on this. Thanks. |
yes, that was added in opensearch-project/OpenSearch#4902 and ported to 2.x in opensearch-project/OpenSearch#4993 |
Based on the offline conversation I will test the build on local to ensure that no-jdk is not affecting the normal jdk build. Thanks. |
I have confirmed it will not impact the existing code base. But, instead of building both jdk and no-jdk in one take, we should add a --no-jdk param in build repo, so the target will be properly set as
instead of
This will also resolve the follow up assemble issue with no-jdk bundle. There are still a lot of steps to go, I would recommend hold this PR for a bit longer, and start changing the build.sh on OpenSearch Core repo. Ex:
Thanks. |
I also want to mention this seems significantly more complex than initially anticipated. Thanks. |
I am confused. We just need an old build.sh in those old opensearch branches. Why is this any more complicated? |
i guess the problem was that we (well, I) mixed two things:
it might make sense to just revert the latter (no-jdk) and get this one here merged to at least close the move job. and then we can look into the no-jdk discussion again? |
The issue is more that instead of adding a no-jdk, the original implementation is to run both jdk + no-jdk build in parallel, which is not supposed to be as the current code base does not support the actions after that. |
I am with you @rursprung. Let's focus on this PR! Can we merge just deleting the OpenSearch build.sh from this repo? |
…ch-project#4902)" This reverts commit 8c9ca4e. It seems that this wasn't entirely the correct way and is currently blocking us from removing the `build.sh` from the `opensearch-build` repository (i.e. this `build.sh` here is not yet being used). See the discussion in opensearch-project/opensearch-build#2835 for further details. Signed-off-by: Ralph Ursprung <Ralph.Ursprung@avaloq.com>
…ch-project#4902)" This reverts commit 8c9ca4e. It seems that this wasn't entirely the correct way and is currently blocking us from removing the `build.sh` from the `opensearch-build` repository (i.e. this `build.sh` here is not yet being used). See the discussion in opensearch-project/opensearch-build#2835 for further details. Signed-off-by: Ralph Ursprung <Ralph.Ursprung@avaloq.com>
…ch-project#4902)" This reverts commit 8c9ca4e. It seems that this wasn't entirely the correct way and is currently blocking us from removing the `build.sh` from the `opensearch-build` repository (i.e. this `build.sh` here is not yet being used). See the discussion in opensearch-project/opensearch-build#2835 for further details. Signed-off-by: Ralph Ursprung <Ralph.Ursprung@avaloq.com>
i've created the reversal PRs and they're ready for review:
but i just realised that one more step will be needed before this here can be done: |
…5467) This reverts commit 8c9ca4e. It seems that this wasn't entirely the correct way and is currently blocking us from removing the `build.sh` from the `opensearch-build` repository (i.e. this `build.sh` here is not yet being used). See the discussion in opensearch-project/opensearch-build#2835 for further details. Signed-off-by: Ralph Ursprung <Ralph.Ursprung@avaloq.com> Signed-off-by: Ralph Ursprung <Ralph.Ursprung@avaloq.com>
…5465) This reverts commit 8c9ca4e. It seems that this wasn't entirely the correct way and is currently blocking us from removing the `build.sh` from the `opensearch-build` repository (i.e. this `build.sh` here is not yet being used). See the discussion in opensearch-project/opensearch-build#2835 for further details. Signed-off-by: Ralph Ursprung <Ralph.Ursprung@avaloq.com> Signed-off-by: Ralph Ursprung <Ralph.Ursprung@avaloq.com>
…5466) This reverts commit 8c9ca4e. It seems that this wasn't entirely the correct way and is currently blocking us from removing the `build.sh` from the `opensearch-build` repository (i.e. this `build.sh` here is not yet being used). See the discussion in opensearch-project/opensearch-build#2835 for further details. Signed-off-by: Ralph Ursprung <Ralph.Ursprung@avaloq.com> Signed-off-by: Ralph Ursprung <Ralph.Ursprung@avaloq.com>
…5465) This reverts commit 8c9ca4e. It seems that this wasn't entirely the correct way and is currently blocking us from removing the `build.sh` from the `opensearch-build` repository (i.e. this `build.sh` here is not yet being used). See the discussion in opensearch-project/opensearch-build#2835 for further details. Signed-off-by: Ralph Ursprung <Ralph.Ursprung@avaloq.com> Signed-off-by: Ralph Ursprung <Ralph.Ursprung@avaloq.com>
…ature/identity (#5581) * Fix flaky ShardIndexingPressureConcurrentExecutionTests (#5439) Add conditional check on assertNull to fix flaky tests. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * Fix bwc for cluster manager throttling settings (#5305) Signed-off-by: Dhwanil Patel <dhwanip@amazon.com> * Update ingest-attachment plugin dependencies: Apache Tika 3.6.0, Apache Mime4j 0.8.8, Apache Poi 5.2.3, Apache PdfBox 2.0.27 (#5448) Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Enhance CheckpointState to support no-op replication (#5282) * CheckpointState enhanced to support no-op replication Signed-off-by: Ashish Singh <ssashish@amazon.com> Co-authored-by: Bukhtawar Khan<bukhtawa@amazon.com> * [BUG] org.opensearch.repositories.s3.RepositoryS3ClientYamlTestSuiteIT/test {yaml=repository_s3/20_repository_permanent_credentials/Snapshot and Restore with repository-s3 using permanent credentials} flaky: randomizing basePath (#5482) Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * [Bug] fix case sensitivity for wildcard queries (#5462) Fixes the wildcard query to not normalize the pattern when case_insensitive is set by the user. This is achieved by creating a new normalizedWildcardQuery method so that query_string queries (which do not support case sensitivity) can still normalize the pattern when the default analyzer is used; maintaining existing behavior. Signed-off-by: Nicholas Walter Knize <nknize@apache.org> * Support OpenSSL Provider with default Netty allocator (#5460) Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Revert "build no-jdk distributions as part of release build (#4902)" (#5465) This reverts commit 8c9ca4e. It seems that this wasn't entirely the correct way and is currently blocking us from removing the `build.sh` from the `opensearch-build` repository (i.e. this `build.sh` here is not yet being used). See the discussion in opensearch-project/opensearch-build#2835 for further details. Signed-off-by: Ralph Ursprung <Ralph.Ursprung@avaloq.com> Signed-off-by: Ralph Ursprung <Ralph.Ursprung@avaloq.com> * Add max_shard_size parameter for Shrink API (fix supported version after backport) (#5503) Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Sync CODEOWNERS with MAINTAINERS. (#5501) Signed-off-by: Daniel (dB.) Doubrovkine <dblock@amazon.com> Signed-off-by: Daniel (dB.) Doubrovkine <dblock@amazon.com> * Added jackson dependency to server (#5366) * Added jackson dependency to server Signed-off-by: Ryan Bogan <rbogan@amazon.com> * Updated CHANGELOG Signed-off-by: Ryan Bogan <rbogan@amazon.com> * Update build.gradle files Signed-off-by: Ryan Bogan <rbogan@amazon.com> * Add RuntimePermission to fix errors Signed-off-by: Ryan Bogan <rbogan@amazon.com> Signed-off-by: Ryan Bogan <rbogan@amazon.com> * Fix flaky test BulkIntegrationIT.testDeleteIndexWhileIndexing (#5491) Signed-off-by: Poojita Raj <poojiraj@amazon.com> Signed-off-by: Poojita Raj <poojiraj@amazon.com> * Add release notes for 2.4.1 (#5488) Signed-off-by: Xue Zhou <xuezhou@amazon.com> Signed-off-by: Xue Zhou <xuezhou@amazon.com> * Properly skip OnDemandBlockSnapshotIndexInputTests.testVariousBlockSize on Windows. (#5511) PR #5397 skipped this test in @before block but still frequently throws a TestCouldNotBeSkippedException. This is caused by the after block still executing and throwing an exception while cleaning the directory created at the path in @before. Moving the assumption to the individual test prevents this exception by ensuring the path exists. Signed-off-by: Marc Handalian <handalm@amazon.com> Signed-off-by: Marc Handalian <handalm@amazon.com> * Merge first batch of feature/extensions into main (#5347) * Merge first batch of feature/extensions into main Signed-off-by: Ryan Bogan <rbogan@amazon.com> * Fixed CHANGELOG Signed-off-by: Ryan Bogan <rbogan@amazon.com> * Fixed newline errors Signed-off-by: Ryan Bogan <rbogan@amazon.com> * Renaming and CHANGELOG fixes Signed-off-by: Ryan Bogan <rbogan@amazon.com> * Refactor extension loading into private method Signed-off-by: Ryan Bogan <rbogan@amazon.com> * Removed skipValidation and added connectToExtensionNode method Signed-off-by: Ryan Bogan <rbogan@amazon.com> * Remove unnecessary feature flag calls Signed-off-by: Ryan Bogan <rbogan@amazon.com> * Renaming and exception handling Signed-off-by: Ryan Bogan <rbogan@amazon.com> * Change latches to CompletableFuture Signed-off-by: Ryan Bogan <rbogan@amazon.com> * Removed unnecessary validateSettingKey call Signed-off-by: Ryan Bogan <rbogan@amazon.com> * Fix azure-core dependency Signed-off-by: Ryan Bogan <rbogan@amazon.com> * Update SHAs Signed-off-by: Ryan Bogan <rbogan@amazon.com> * Remove unintended dependency changes Signed-off-by: Ryan Bogan <rbogan@amazon.com> * Removed dynamic settings regitration, removed info() method, and added NoopExtensionsManager Signed-off-by: Ryan Bogan <rbogan@amazon.com> * Add javadoc Signed-off-by: Ryan Bogan <rbogan@amazon.com> * Fixed spotless failure Signed-off-by: Ryan Bogan <rbogan@amazon.com> * Removed NoopExtensionsManager Signed-off-by: Ryan Bogan <rbogan@amazon.com> * Added functioning NoopExtensionsManager Signed-off-by: Ryan Bogan <rbogan@amazon.com> * Added missing javadoc Signed-off-by: Ryan Bogan <rbogan@amazon.com> * Remove forbiddenAPI Signed-off-by: Ryan Bogan <rbogan@amazon.com> * Fix spotless Signed-off-by: Ryan Bogan <rbogan@amazon.com> * Change logger.info to logger.error in handleException Signed-off-by: Ryan Bogan <rbogan@amazon.com> * Fix ExtensionsManagerTests Signed-off-by: Ryan Bogan <rbogan@amazon.com> * Removing unrelated change Signed-off-by: Ryan Bogan <rbogan@amazon.com> * Update SHAs Signed-off-by: Ryan Bogan <rbogan@amazon.com> Signed-off-by: Ryan Bogan <rbogan@amazon.com> * Bump commons-compress from 1.21 to 1.22 (#5520) Bumps commons-compress from 1.21 to 1.22. --- updated-dependencies: - dependency-name: org.apache.commons:commons-compress dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * [Segment Replication] Trigger a round of replication for replica shards during peer recovery when segment replication is enabled (#5332) * Fix new added replica shards falling behind primary. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * Trigger a round of replication during peer recovery when segment replication is enabled. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * Remove unnecessary start replication overloaded method. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * Add test for failure case and refactor some code. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * Apply spotless check. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * Addressing comments on the PR. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * Remove unnecessary condition check. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * Apply spotless check. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * Add step listeners to resolve forcing round of segment replication. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * Adding support to register settings dynamically (#5495) * Adding support to register settings dynamically Signed-off-by: Ryan Bogan <rbogan@amazon.com> * Update CHANGELOG Signed-off-by: Ryan Bogan <rbogan@amazon.com> * Removed unnecessary registerSetting methods Signed-off-by: Ryan Bogan <rbogan@amazon.com> * Change setting registration order Signed-off-by: Ryan Bogan <rbogan@amazon.com> * Add unregisterSettings method Signed-off-by: Ryan Bogan <rbogan@amazon.com> * Remove unnecessary feature flag Signed-off-by: Ryan Bogan <rbogan@amazon.com> Signed-off-by: Ryan Bogan <rbogan@amazon.com> * Updated 1.3.7 release notes date (#5536) Signed-off-by: owaiskazi19 <owaiskazi19@gmail.com> Signed-off-by: owaiskazi19 <owaiskazi19@gmail.com> * Pre conditions check before updating weighted routing metadata (#4955) * Pre conditions check to allow weight updates for non decommissioned attribute Signed-off-by: Rishab Nahata <rnnahata@amazon.com> * Atomically update cluster state with decommission status and corresponding action (#5093) * Atomically update the cluster state with decommission status and its corresponding action in the same execute call Signed-off-by: Rishab Nahata <rnnahata@amazon.com> * Update Netty to 4.1.86.Final (#5529) Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Update release date in 2.4.1 release notes (#5549) Signed-off-by: Suraj Singh <surajrider@gmail.com> Signed-off-by: Suraj Singh <surajrider@gmail.com> * Update 2.4.1 release notes (#5552) Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Refactor fuzziness interface on query builders (#5433) * Refactor Object to Fuzziness type for all query builders Signed-off-by: noCharger <lingzhichu.clz@gmail.com> * Revise on bwc Signed-off-by: noCharger <lingzhichu.clz@gmail.com> * Update change log Signed-off-by: noCharger <lingzhichu.clz@gmail.com> Signed-off-by: noCharger <lingzhichu.clz@gmail.com> Co-authored-by: Daniel (dB.) Doubrovkine <dblock@amazon.com> * Upgrade lucene version (#5570) * Added bwc version 2.4.2 Signed-off-by: Daniel (dB.) Doubrovkine <dblock@amazon.com> * Added 2.4.2. Signed-off-by: Daniel (dB.) Doubrovkine <dblock@amazon.com> * Update Lucene snapshot to 9.5.0-snapshot-d5cef1c Signed-off-by: Suraj Singh <surajrider@gmail.com> * Update changelog entry Signed-off-by: Suraj Singh <surajrider@gmail.com> * Add 2.4.2 bwc version Signed-off-by: Suraj Singh <surajrider@gmail.com> * Internal changes post lucene upgrade Signed-off-by: Suraj Singh <surajrider@gmail.com> Signed-off-by: Daniel (dB.) Doubrovkine <dblock@amazon.com> Signed-off-by: Suraj Singh <surajrider@gmail.com> Co-authored-by: opensearch-ci-bot <opensearch-ci-bot@users.noreply.github.com> Co-authored-by: Daniel (dB.) Doubrovkine <dblock@amazon.com> * Add CI bundle pattern to distribution download (#5348) * Add CI bundle pattern for ivy repo Signed-off-by: Zelin Hao <zelinhao@amazon.com> * Gradle update Signed-off-by: Zelin Hao <zelinhao@amazon.com> * Extract path Signed-off-by: Zelin Hao <zelinhao@amazon.com> * Change with customDistributionDownloadType Signed-off-by: Zelin Hao <zelinhao@amazon.com> * Add default for exception handle Signed-off-by: Zelin Hao <zelinhao@amazon.com> * Add documentations Signed-off-by: Zelin Hao <zelinhao@amazon.com> Signed-off-by: Zelin Hao <zelinhao@amazon.com> * Bump protobuf-java from 3.21.9 to 3.21.11 in /plugins/repository-hdfs (#5519) * Bump protobuf-java from 3.21.9 to 3.21.11 in /plugins/repository-hdfs Bumps [protobuf-java](https://github.com/protocolbuffers/protobuf) from 3.21.9 to 3.21.11. - [Release notes](https://github.com/protocolbuffers/protobuf/releases) - [Changelog](https://github.com/protocolbuffers/protobuf/blob/main/generate_changelog.py) - [Commits](protocolbuffers/protobuf@v3.21.9...v3.21.11) --- updated-dependencies: - dependency-name: com.google.protobuf:protobuf-java dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * Updating SHAs Signed-off-by: dependabot[bot] <support@github.com> * Updated changelog Signed-off-by: Owais Kazi <owaiskazi19@gmail.com> Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Owais Kazi <owaiskazi19@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> Co-authored-by: Owais Kazi <owaiskazi19@gmail.com> Co-authored-by: Suraj Singh <surajrider@gmail.com> Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> Signed-off-by: Dhwanil Patel <dhwanip@amazon.com> Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Signed-off-by: Ashish Singh <ssashish@amazon.com> Signed-off-by: Nicholas Walter Knize <nknize@apache.org> Signed-off-by: Ralph Ursprung <Ralph.Ursprung@avaloq.com> Signed-off-by: Daniel (dB.) Doubrovkine <dblock@amazon.com> Signed-off-by: Ryan Bogan <rbogan@amazon.com> Signed-off-by: Poojita Raj <poojiraj@amazon.com> Signed-off-by: Xue Zhou <xuezhou@amazon.com> Signed-off-by: Marc Handalian <handalm@amazon.com> Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: owaiskazi19 <owaiskazi19@gmail.com> Signed-off-by: Rishab Nahata <rnnahata@amazon.com> Signed-off-by: Suraj Singh <surajrider@gmail.com> Signed-off-by: noCharger <lingzhichu.clz@gmail.com> Signed-off-by: Zelin Hao <zelinhao@amazon.com> Signed-off-by: Owais Kazi <owaiskazi19@gmail.com> Co-authored-by: Rishikesh Pasham <62345295+Rishikesh1159@users.noreply.github.com> Co-authored-by: Dhwanil Patel <dhwanip@amazon.com> Co-authored-by: Andriy Redko <andriy.redko@aiven.io> Co-authored-by: Ashish <ssashish@amazon.com> Co-authored-by: Nick Knize <nknize@apache.org> Co-authored-by: Ralph Ursprung <39383228+rursprung@users.noreply.github.com> Co-authored-by: Daniel (dB.) Doubrovkine <dblock@amazon.com> Co-authored-by: Ryan Bogan <10944539+ryanbogan@users.noreply.github.com> Co-authored-by: Poojita Raj <poojiraj@amazon.com> Co-authored-by: Xue Zhou <85715413+xuezhou25@users.noreply.github.com> Co-authored-by: Marc Handalian <handalm@amazon.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Owais Kazi <owaiskazi19@gmail.com> Co-authored-by: Rishab Nahata <rnnahata@amazon.com> Co-authored-by: Suraj Singh <surajrider@gmail.com> Co-authored-by: Louis Chu <lingzhichu.clz@gmail.com> Co-authored-by: opensearch-ci-bot <opensearch-ci-bot@users.noreply.github.com> Co-authored-by: Zelin Hao <87548827+zelinh@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
Hi @peterzhuamazon Should we move ahead with this PR? |
@peterzhuamazon @rursprung What's the next step on this PR? Please provide your comments. |
between the moment where i added the |
@peterzhuamazon What would you like to do with this (finish/close)? |
build.sh
has been copied over to all active branches(*) of the OpenSearch repo and can now consequently be deleted here. this means, that from now on, the scripts from the respository will be picked up.for more information on the order of evaluation of the custom build scripts, see the documentation.
this gives the advantage to have branch-specific new logic in place there. this has already been used to add the no-jdk targets to the builds on main and 2.x.
two tests were based on the assumption that there would be a
build.sh
for OpenSearch in this repository, however these are generic tests and just used OpenSearch as an example. they have been switched to be based on OpenSearch-Dashboards instead which - for the time being - still has its ownbuild.sh
here.(*): the following branches have been considered as active:
this is part of #99
Signed-off-by: Ralph Ursprung Ralph.Ursprung@avaloq.com
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.