Skip to content

Latest commit

 

History

History
81 lines (55 loc) · 9.06 KB

ONBOARDING.md

File metadata and controls

81 lines (55 loc) · 9.06 KB
Raw

Component Onboarding

This document describes steps to onboard a new plugin to release workflow for continuous integration and testing.

Onboard to OpenSearch Meta

opensearch-plugins

Add the new plugin to the opensearch-plugins meta, e.g. opensearch-plugins#97, which added cross-cluster-replication.

Onboard to Build Workflow

  1. Update a manifest for a particular release to include your plugin. For example to be included in the 1.1.0 release, you would update opensearch-1.1.0.yml. We require your plugin name, repository url, and git ref that should be used. For unreleased versions this should be a branch in your repository. Once a release is cut, these refs will be updated to build from a tag or specific commit hash.

  2. Create a scripts/build.sh if you have specific requirements that are not covered by the default build.sh script and commit it to your repository.

  3. Ensure your build.sh reads and passes along both -Dbuild.snapshot= and -Dopensearch.version= flags. Snapshot builds should produce a -SNAPSHOT tagged artifact for example opensearch-plugin-1.1.0.0-SNAPSHOT.zip where a release build of the same component would produce opensearch-plugin-1.1.0.0.zip.

  4. Execute ./build.sh to ensure your component builds and all artifacts are correctly placed into ./artifacts/ with correct output names.

  5. Execute ./assemble.sh to ensure the full bundle is assembled and placed in to /bundles/*.tar.gz. Unpack the tarball to ensure all your components are placed in their correct locations.

  6. Publish a PR to this repo including the updated manifest and the names of the artifacts being added.

Onboard to test-workflow

  1. Update the test configuration file (use 1.3.0 as an example), opensearch-1.3.0-test.yml, for a particular release, to include your plugin. This test configuration defines full suite of tests - integ, bwc, that can be run on the plugin.

  2. For integration testing, the test-workflow runs integration tests available in the plugin repository. You will need to add integ-test config for your plugin in opensearch-1.3.0-test.yml, example.

    1. It supports two test configs - with-security and without-security, which runs test with security plugin enabled and disabled respectively. Choose one or both depending on what your plugin integration tests support.

    2. If your plugin is dependent on job-scheduler zip, you can define that in build-dependencies in the config. Currently, the test workflow only supports job-scheduler as build dependency. Please create an issue if your plugin needs more support.

  3. For backward compatibility testing, the test-workflow runs backward compatibility tests available in the plugin repository, (see reference). Like integration test, it has a set of configurable options defined in opensearch-1.3.0-test.yml, example.

    1. It supports two test configs - with-security and without-security, which runs test with security plugin enabled and disabled respectively. Choose one or both depending on what your plugin integration tests support.

Onboarding to universal / 1-click release process:

This document describes steps to onboard a new component to universal or 1-click release process.

See opensearch-project#1234 for details about end to end workflow.

  1. Please ensure that opensearch-ci-bot has the write access to your repository. If not, request by creating an issue in this repository.
  2. Add a webhook token as credentials to CI system using configuration as code. To generate the token, use command head -30 /dev/urandom | sha512sum | awk '{print $1;}' that generates random alpha-numeric string.
  3. Create a Jenkins workflow that utilizes one of the build libraries to publish the artifacts to right platform. Please check the library requirements and retrieval methods before using it.
  4. For publishing to a new platform (other than ones specified above) a new library needs to be added. (ETA: 2 weeks)
  5. Release Drafter: Release drafter is a GitHub Action workflow that drafts a release that may or may not contain the release artifacts. The drafted release acts as a trigger to the Jenkins workflow. It also acts as a staging environment for release artifacts. This is to make sure the build environment remains the same even for release artifacts. Example
    • 2 Person Review It is highly recommended to add 2PR approval for any release workflow. In universal release process this can be added to release-drafter workflow as that is the starting point to trigger any release. See how to add the mechanism in the workflow. The mentioned solution creates an issues that notifies and assignes the reviewers. Example_
  6. Jenkins Workflow: Once the Jenkins workflow is added to the repository, onboard the workflow to publicly available CI system
    1. Create a New Item
    2. Name it <component-name>-release
    3. Select Pipeline as type of the project
    4. Hit Ok and scroll to the bottom of the page
    5. Select "Pipeline script from SCM" under Pipeline section
    6. Run the workflow once in order to update the configuration of the Jenkins Workflow. You can abort once the workflow starts pulling the docker image.
  7. GitHub Webhook: Add webhook to your GitHub repository by going to repository settings → Click Webhooks:
    Key Value
    Payload URL https://build.ci.opensearch.org/generic-webhook-trigger/invoke?token=tokenAddedInStep2
    Content type application/json
    SSL verification Enable
    Which events would you like to trigger this webhook Releases. Please ensure to deselect the default option "Pushes"
    Active Enable
  8. Once a webhook is added, it will send a ping to check the connectivity (✅). You can check the ping by going to repistory settings → Webhooks → Click on Recent Deliveries tabs
  9. Add RELEASING.md file to the repository documenting how to release the artifact. Example
  10. Adding tests: Each library has a respective library tester associated with it that can be used to test you jenkins workflow. This tests can be used to verify that the workflow is making the calls. The build system used is gradle. For example, this PublishToNpm test uses PublishToNpmLibTester with expected parameter that can be unique to your workflow. The assertions makes sure that calls to npm registry is made which is mandatory to release an artifact.