Update AL2 ci images with unofficial nodejs build before EOL deprecation

[peterzhuamazon]

A lot of the github actions are failing now due to AL2 with older glibc does not support node 18 and 20 and above. (Ex: opensearch-project/job-scheduler#700) (Ex2: opensearch-project/neural-search#975)

Since AL2 will be deprecated on 2025/06 we opt to use the unofficial build to unblock us from this situation without doing too much changes to everyone's checks.

Thanks.

New Approach:

1. The image will have the unofficial build on disk, so it wont be blocked if nodejs mirror is down.
2. The workflow will do two volume mount, and force the hardcoded path of github runner binary pointing to the unofficial build on disk.

Therefore the actions of v4s will then use our binary to run, which support glibc2.17.

---

Plugin Owners, please follow this Sample PR to mitigate the issues:

• Enable custom start commands and options to resolve GHA issues job-scheduler#702

Thanks!

---

Fix Mechanism PRs:

• Support custom start-command/start-options on GHA container runner and openssl1.1.1 on AL2 docker images #5187

Infra/Plugin PRs:

• Enable custom start commands and options to resolve GHA issues job-scheduler#702
• Enable custom start commands and options to resolve GHA issues #5189
• Enable custom start commands and options to resolve GHA issues asynchronous-search#676
• Enable custom start commands and options to resolve GHA issues ml-commons#3223
• Updating setup-java, byte buddy, jacoco and mockito dependencies  anomaly-detection#1368
• Update codecov to check during PR neural-search#995
  • Switch codecov to v3 as it requires python3.11 instead of node20 neural-search#999
  • Update security runs to also support AL2 on node20 neural-search#1002
• Remove dco and update artifact action to v4 dashboards-observability#2265

[peterzhuamazon]

Apparently we can get over it with one more params, apart from ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION:

+        env:
+            ACTIONS_RUNNER_FORCE_ACTIONS_NODE_VERSION: node16
+            ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true

[peterzhuamazon]

Apparently checkout@v4 up to node20 but upload/download artifacts only up to 18/12 respectively.

[peterzhuamazon]

Seems like github prevent you from even using your own nodejs binaries to run actions:

mv: cannot move '/__e/node20/bin/node' to '/__e/node20/bin/node_bak': Read-only file system

[peterzhuamazon]

Even if you install nodejs 18 or 20 that is custom build and compatible with AL2 on docker images, github will enforce on using their host version of node binaries during action runs.

I am not seeing a good method to override that at the moment.

[peterzhuamazon]

Seems we can force with setup-node actions to some degree:
actions/setup-node#922

[peterzhuamazon]

More similar:

• supporting installing node into an alpine based container actions/setup-node#109
• Support specifying alternate node mirrors actions/setup-node#65
• Composite Actions using Node 20 don't work with CentOS 7 actions/runner#2906

[peterzhuamazon]

Some already proposed:

• use a glibc 2.17 compatible node20 build on x86_64 (better idea inside!) actions/runner#3128

[peterzhuamazon]

More deep dive and seems like happening since this where all node16 has been updated to node20.

• GHES 3.14 Upgrades runner to 2.317.0, this breaks ALL workflows on RHEL 7  actions/runner#3475

This pretty much means these params are not taking effect anymore:

+        env:
+            ACTIONS_RUNNER_FORCE_ACTIONS_NODE_VERSION: node16
+            ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true

[peterzhuamazon]

Try this tomorrow:

• Composite Actions using Node 20 don't work with CentOS 7 actions/runner#2906 (comment)

[peterzhuamazon]

And it runs:

• https://github.com/opensearch-project/job-scheduler/actions/runs/11843998084/job/33006210989?pr=702

[peterzhuamazon]

See descriptions for all the fix PRs.

[peterzhuamazon]

Close this as EE has completed research and initial fixes to the repos with the most issues.
The other repos can follow the PRs in description for fixes if needed.

Thanks.