Introduce ci_dcn_site role

The ci_dcn_site role may be used to deploy DCN sites for
testing. Each DCN site is a new EDPM nodeset with a
collocated Ceph cluster.

Co-authored-by: Sergey Bekkerman <sbekkerm@redhat.com>

docs/dictionary/en-custom.txt

@@ -21,6 +21,8 @@ authfile
 autoscale
 autostart
 awk
+az
+azs
 backend
 backends
 baremetal
@@ -139,6 +141,7 @@ dnsdata
 dnsmasq
 dockerfile
 dryrun
+dt
 dts
 ecdsa
 edecb
@@ -333,6 +336,7 @@ nodeexporter
 nodenetworkconfigurationpolicy
 nodeps
 nodeset
+nodesets
 nodetemplate
 noop
 nopasswd

playbooks/dcn.yml

@@ -0,0 +1,60 @@
+---
+# Copyright Red Hat, Inc.
+# All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License"); you may
+# not use this file except in compliance with the License. You may obtain
+# a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+# License for the specific language governing permissions and limitations
+# under the License.
+
+- name: Deploy DCN environment
+  hosts: localhost
+  vars:
+    _arch_repo_path: /home/zuul/src/github.com/openstack-k8s-operators/architecture
+    _cifmw_repo_path: /home/zuul/src/github.com/openstack-k8s-operators/ci-framework
+    _jinja_template_src: "{{ _cifmw_repo_path }}/roles/ci_dcn_site/templates"
+    _arch_deployment_path: "{{ _arch_repo_path }}/examples/dt/dcn"
+  tasks:
+    - name: Load reproducer-variables
+      ansible.builtin.include_vars:
+        file: "~/reproducer-variables.yml"
+
+    - name: Load networking-environment-definition
+      ansible.builtin.include_vars:
+        file: "/etc/ci/env/networking-environment-definition.yml"
+        name: cifmw_networking_env_definition
+
+    - name: Load openshift-login-params
+      ansible.builtin.include_vars:
+        file: "~/ci-framework-data/artifacts/parameters/openshift-login-params.yml"
+
+    - name: Create a network subnet list
+      ansible.builtin.set_fact:
+        _network_ranges: >-
+          {{
+            cifmw_networking_env_definition.networks
+            | dict2items
+            | selectattr('key', 'search', '^ctlplane')
+            | map(attribute='value.network_v4')
+            | list
+          }}
+
+    - name: Deploy EDPM
+      ansible.builtin.include_role:
+        name: ci_dcn_site
+      with_items: "{{ groups | dict2items | selectattr('key', 'search', 'compute') | list }}"
+      loop_control:
+        index_var: idx
+        loop_var: itm
+      vars:
+        _subnet: "subnet{{ idx + 1 }}"
+        _group_name: "{{ itm.key }}"
+        _az: "az{{ idx }}"
+        _subnet_network_range: "{{ _network_ranges[idx] }}"

roles/ci_dcn_site/README.md

@@ -0,0 +1,83 @@
+# ci_dcn_site
+
+Deploys DCN sites for testing. Each DCN site is a new EDPM nodeset
+with a collocated Ceph cluster.
+
+## Privilege escalation
+
+- Applies CRDs in openstack namespace
+- Runs openstack client commands to create aggregates and discover new
+  compute hosts
+
+## Parameters
+
+* `_az`: The name of the availability zone for the AZ, e.g. `az1`
+* `_group_name`: The name of the group of nodes to be deployed, e.g. `dcn1-computes`
+* `_subnet`: The name of the subnet the DCN site will use, e.g. `subnet2`
+* `_subnet_network_range`: The range of the subnet the DCN site will use, e.g. `192.168.133.0/24`
+
+## Examples
+
+To deploy two nodesets named dcn1-computes and dcn2-computes,
+the role may be called like this.
+```yaml
+- name: Deploy
+  include_role: ci_dcn_site
+  with_items: "{{ groups | dict2items | selectattr('key', 'search', 'compute') | list }}"
+  loop_control:
+    index_var: idx
+    loop_var: item
+  vars:
+    _subnet: "subnet{{ idx + 1 }}"
+    _group_name: "{{ item.key }}"
+    _az: "az{{ idx }}"
+    _subnet_network_range: "{{ _network_ranges[idx] }}"
+```
+The above assumes the following values for each iteration:
+```
+_subnet: subnet2 | _group_name: dcn1-computes | _az: az1 | _subnet_network_range: 192.168.133.0/24
+_subnet: subnet3 | _group_name: dcn2-computes | _az: az2 | _subnet_network_range: 192.168.144.0/24
+```
+It relies on the `ci-framework-data/artifacts/zuul_inventory.yml` which the
+ci-framework will populate correctly when the `dt-dcn.yml` scenario is used.
+The variables above can then be built with the following tasks before
+the above is run.
+```yaml
+  - name: Load reproducer-variables
+    ansible.builtin.include_vars:
+      file: "~/reproducer-variables.yml"
+
+  - name: Load networking-environment-definition
+    ansible.builtin.include_vars:
+      file: "/etc/ci/env/networking-environment-definition.yml"
+      name: cifmw_networking_env_definition
+
+  - name: Create a network subnet list
+    ansible.builtin.set_fact:
+      _network_ranges: >-
+        {{
+          cifmw_networking_env_definition.networks
+          | dict2items
+          | selectattr('key', 'search', '^ctlplane')
+          | map(attribute='value.network_v4')
+          | list
+        }}
+```
+
+## Integration with Architecture Repository
+
+The directions in the
+[DCN DT](https://github.com/openstack-k8s-operators/architecture/tree/main/examples/dt/dcn)
+end with deploying the first Availability Zone (AZ) called `az0`.
+Additional AZs may be deployed for testing by calling this role.
+
+The DCN DT contains values yaml files which may be passed to
+kustomize. This role generates additional instances of the same
+type of values files from jinja templates. The templates are populated
+with the values in the environment which are set when the `dt-dcn.yml`
+scenario is used. The role then calls kustomize to apply the CRDs.
+
+The role is executed by the dcn.yml playbook found in the playbooks
+directory. This same playbook is called by the automation structure
+in the DCN DT (`automation/vars/dcn.yaml`) by using a
+`post_stage_run`.

roles/ci_dcn_site/defaults/main.yml

@@ -0,0 +1,19 @@
+---
+# Copyright Red Hat, Inc.
+# All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License"); you may
+# not use this file except in compliance with the License. You may obtain
+# a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+# License for the specific language governing permissions and limitations
+# under the License.
+
+
+# All variables intended for modification should be placed in this file.
+# All variables within this role should have a prefix of "cifmw_ci_dcn_site"

roles/ci_dcn_site/meta/main.yml

@@ -0,0 +1,30 @@
+---
+# Copyright Red Hat, Inc.
+# All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License"); you may
+# not use this file except in compliance with the License. You may obtain
+# a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+# License for the specific language governing permissions and limitations
+# under the License.
+
+
+galaxy_info:
+  author: CI Framework
+  description: CI Framework Role -- ci_dcn_site
+  company: Red Hat
+  license: Apache-2.0
+  min_ansible_version: "2.14"
+  namespace: cifmw
+  galaxy_tags:
+    - cifmw
+
+# List your role dependencies here, one per line. Be sure to remove the '[]' above,
+# if you add dependencies to this list.
+dependencies: []

roles/ci_dcn_site/tasks/az.yml

@@ -0,0 +1,58 @@
+---
+# Copyright Red Hat, Inc.
+# All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License"); you may
+# not use this file except in compliance with the License. You may obtain
+# a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+# License for the specific language governing permissions and limitations
+# under the License.
+
+- name: Check if AZ exists has hosts
+  kubernetes.core.k8s_exec:
+    kubeconfig: "{{ cifmw_openshift_kubeconfig }}"
+    api_key: "{{ cifmw_openshift_token | default(omit) }}"
+    context: "{{ cifmw_openshift_context | default(omit) }}"
+    namespace: openstack
+    pod: openstackclient
+    command: >-
+      openstack aggregate show {{ _az }} -c hosts -f value
+  register: az_hosts
+  ignore_errors: true
+
+- name: Convert az_hosts.stdout string to list and remove extra text
+  ansible.builtin.set_fact:
+    az_hosts_list: "{{ az_hosts.stdout | default([]) | from_yaml
+      | map('regex_replace', 'edpm-compute-(.*)\\.ctlplane\\.example\\.com', 'compute-\\1')
+      | list }}"
+
+- name: Create AZ if it does not exist
+  when:
+    - az_hosts.rc == 1
+  kubernetes.core.k8s_exec:
+    kubeconfig: "{{ cifmw_openshift_kubeconfig }}"
+    api_key: "{{ cifmw_openshift_token | default(omit) }}"
+    context: "{{ cifmw_openshift_context | default(omit) }}"
+    namespace: openstack
+    pod: openstackclient
+    command: >-
+      openstack aggregate create {{ _az }} --zone {{ _az }}
+
+- name: Add only the missing edpm hosts to AZ
+  when:
+    - item.key not in az_hosts_list
+  kubernetes.core.k8s_exec:
+    kubeconfig: "{{ cifmw_openshift_kubeconfig }}"
+    api_key: "{{ cifmw_openshift_token | default(omit) }}"
+    context: "{{ cifmw_openshift_context | default(omit) }}"
+    namespace: openstack
+    pod: openstackclient
+    command: >-
+      openstack aggregate add host {{ _az }} edpm-{{ item.key }}.ctlplane.example.com
+  loop: "{{ _edpm_hosts | dict2items }}"