Allow setting order and hooks for test role #2374
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
Thanks for the PR! ❤️ |
eshulman2
force-pushed
the
test_operator_stages
branch
from
d749fd6
to
9e7ab2d
Compare
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/e1ceb9f2cf3d47658a3de25c1ed70de8 ✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 39m 16s |
eshulman2
force-pushed
the
test_operator_stages
branch
3 times, most recently
from
f24ddba
to
c4de17d
Compare
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/a25d6668f88443eeb5b2e354691eb8dc ✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 39m 51s |
eshulman2
force-pushed
the
test_operator_stages
branch
2 times, most recently
from
c175381
to
5efbd8f
Compare
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/ea84a8ceb689406f85a63f9e9f8d1718 ✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 39m 40s |
eshulman2
force-pushed
the
test_operator_stages
branch
3 times, most recently
from
1c61cd8
to
34ffac9
Compare
eshulman2
force-pushed
the
test_operator_stages
branch
2 times, most recently
from
c00804c
to
e6b67aa
Compare
eshulman2
requested review from
lpiwowar,
lewisdenny and
cjeanner
and removed request for
lpiwowar and
lewisdenny
eshulman2
force-pushed
the
test_operator_stages
branch
from
e6b67aa
to
d2b1fed
Compare
eshulman2
force-pushed
the
test_operator_stages
branch
from
d65bd64
to
03902d1
Compare
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/243d4e9a524745129e39edc840e9685c ✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 01m 35s |
|
recheck |
eshulman2
force-pushed
the
test_operator_stages
branch
from
03902d1
to
7ad379d
Compare
|
Hi 👋 , I think there need to be patches prepared for some other upstream repositories as well:
And it seems that there is missing an update for the |
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/585f0c06fb1b4f71b39c58dfbfa92426 ✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 51m 07s |
eshulman2
force-pushed
the
test_operator_stages
branch
from
7ad379d
to
79cef31
Compare
|
As of now the core of the PR looks good to me 👍 . The parameters should work as expected (experience from my manual testing session). I'm testing that the global vars are going to work with the test-operator job here (extra testing on top of what was agreed on). I'm not giving
|
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/88a7411e712e48ab877ce57526a53132 ❌ openstack-k8s-operators-content-provider FAILURE in 11m 35s |
|
recheck |
@lpiwowar created patches for all relevant projects updating the required var names: |
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/6a0917a358fb41229665d3f47f6e98ff ✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 23m 57s |
|
@lpiwowar @SeanMooney @vyzigold I can offer a compromise. I can probably do something like this in the CC: @pablintino any thoughts? |
i think that is fine. what i would propose is let have the fallback for now, "deprecate" the old value and remove it after FR2 is released. i.e support both for a short period of time to not break the existing ci jobs, update them to the new value and only delete the old one when that is complete. if the nova jobs (upstream and downstream) started running with concurrency 8 instead of the overridden values we have in those jobs they would start failing which is why we added the override in the first place. so as long as we do it in three stages (add new parameter, update old uages, remove old parameter) i have objection to the rename |
I think @SeanMooney and @vyzigold have a good point. The proposed solution by @eshulman2 should allow us to transition to the new parameters more smoothly 👍 . Just one small note: will not work IIUC as this change will lead to Also, |
eshulman2
force-pushed
the
test_operator_stages
branch
from
79cef31
to
d75e552
Compare
I'd be fine with this change as long as we can merge it without touching a single job. That trick in the defaults will do the trick and I'm all good with it. |
|
@lpiwowar @pablintino updated the patch and related patch with changes in the compromise |
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/5ab23c4b7d3f4804b46465355e232cf1 ✔️ openstack-k8s-operators-content-provider SUCCESS in 3h 07m 37s |
|
@eshulman2 👍 the changes you've made look reasonable. Now it is about the testing. |
This commit adds key improvements to the test_operator role: Support for defining test execution order. Ability to run the same testing controller multiple times. Pre-hooks and post-hooks for stage-specific customization. Per-stage variable overrides via files or test_vars. Refactored pre-set testing stages to enable more dynamic test management and extend hook behavior for complex scenarios. Jira: OSPRH-10106
eshulman2
force-pushed
the
test_operator_stages
branch
from
d75e552
to
c511e9c
Compare
This PR introduces improvements to the test_operator role, enabling ordering of testing stages and hooks for better control over test execution. Key updates include:
Customizable Testing Order:
Introduced the ability to define the testing order for the test_operator role, providing better control over test sequencing.
Repeatable Testing Controller:
Added support for running the same testing controller multiple times within a single test suite.
Stage-Specific Hooks:
Enabled configuration of pre-hooks and post-hooks for each testing stage, offering more granular customization.
Variable Overrides per Stage:
Allowed overriding variables for specific testing stages via either external files (
test_vars_file) or specific stage vars (test_vars) parameter.Refactoring for Enhanced Flexibility
The implementation refactors pre-set testing stage functionality, making test management more adaptable and significantly extending hook behavior when using the CI framework (cifmw). These changes enable the creation of more complex testing workflows, including scenarios requiring multiple and varied hooks or stage-specific configurations.
Impact and Use Cases
These updates allow users to:
Related Issue:
Jira: OSPRH-10106