-
Notifications
You must be signed in to change notification settings - Fork 84
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #119 from Jaganathancse/ovs_dpdk_sriov
OVS DPDK & SRIOV VA
- Loading branch information
Showing
17 changed files
with
769 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| control-plane.yaml |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,31 @@ | ||
| # OpenStack OVS DPDK SRIOV | ||
|
|
||
| **Based on OpenStack K8S operators from the "main" branch of the [OpenStack Operator repo](https://github.com/openstack-k8s-operators/openstack-operator/tree/12c57baeca4ae33dd30a7707d330eb094309b4cd) on Jan 31st, 2024** | ||
|
|
||
| This is a collection of CR templates that represent a validated Red Hat OpenStack Services on OpenShift deployment that has the following characteristics: | ||
|
|
||
| - 3 master/worker combo-node OpenShift cluster | ||
| - 3-replica Galera database | ||
| - RabbitMQ | ||
| - OVN networking | ||
| - Network isolation over a single NIC | ||
| - 2 compute nodes | ||
| - OvS DPDK & SRIOV using ml2 OVN | ||
|
|
||
| ## Considerations | ||
|
|
||
| 1. These CRs are validated for the overall functionality of the OSP cloud deployed, but they nonetheless require customization for the particular environment in which they are utilized. In this sense they are _templates_ meant to be consumed and tweaked to fit the specific constraints of the hardware available. | ||
|
|
||
| 2. The CRs are applied against an OpenShift cluster in _stages_. That is, there is an ordering in which each grouping of CRs is fed to the cluster. It is _not_ a case of simply taking all CRs from all stages and applying them all at once. | ||
|
|
||
| 3. In stages 1 and 2 [kustomize](https://kustomize.io/) is used to generate the CRs dynamically. The `*-values.yaml` file(s) must be updated to fit your environment. kustomize version 5 or newer required. | ||
|
|
||
| 4.In stage 3 [kustomize](https://kustomize.io/) is used to generate the dataplane CRs dynamically. The edpm/values.yaml file must be updated to fit your environment. kustomize version 5 or newer required. | ||
|
|
||
| ## Stages | ||
|
|
||
| All stages must be executed in the order listed below. Everything is required unless otherwise indicated. | ||
|
|
||
| 1. [Install the OpenStack K8S operators and their dependencies](../../../common/) | ||
| 2. [Configuring networking and deploy the OpenStack control plane](control-plane.md) | ||
| 3. [Configure and deploy the data plane](dataplane.md) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,55 @@ | ||
| # Configuring networking and deploy the OpenStack control plane | ||
|
|
||
| ## Assumptions | ||
|
|
||
| - A storage class called `local-storage` should already exist. | ||
|
|
||
| ## Initialize | ||
|
|
||
| Switch to the "openstack" namespace | ||
| ``` | ||
| oc project openstack | ||
| ``` | ||
| Change to the nfv/ovs-dpdk-sriov directory | ||
| ``` | ||
| cd architecture/examples/va/nfv/ovs-dpdk-sriov | ||
| ``` | ||
| Edit the [nncp/values.yaml](nncp/values.yaml) and | ||
| [service-values.yaml](service-values.yaml) files to suit | ||
| your environment. | ||
| ``` | ||
| vi nncp/values.yaml | ||
| vi service-values.yaml | ||
| ``` | ||
|
|
||
| ## Apply node network configuration | ||
|
|
||
| Generate the node network configuration | ||
| ``` | ||
| kustomize build nncp > nncp.yaml | ||
| ``` | ||
|
|
||
| Apply the NNCP CRs | ||
| ``` | ||
| oc apply -f nncp.yaml | ||
| ``` | ||
| Wait for NNCPs to be available | ||
| ``` | ||
| oc wait nncp -l osp/nncm-config-type=standard --for jsonpath='{.status.conditions[0].reason}'=SuccessfullyConfigured --timeout=300s | ||
| ``` | ||
|
|
||
| ## Apply networking and control-plane configuration | ||
|
|
||
| Generate the control-plane and networking CRs. | ||
| ``` | ||
| kustomize build > control-plane.yaml | ||
| ``` | ||
| Apply the CRs | ||
| ``` | ||
| oc apply -f control-plane.yaml | ||
| ``` | ||
|
|
||
| Wait for control plane to be available | ||
| ``` | ||
| oc wait osctlplane controlplane --for condition=Ready --timeout=600s | ||
| ``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,35 @@ | ||
| # Configuring and deploying the dataplane | ||
|
|
||
| ## Assumptions | ||
|
|
||
| - The [control plane](control-plane.md) has been created and successfully deployed | ||
|
|
||
| ## Initialize | ||
|
|
||
| Switch to the "openstack" namespace | ||
| ``` | ||
| oc project openstack | ||
| ``` | ||
| Change to the nfv/ovs-dpdk-sriov/edpm directory | ||
| ``` | ||
| cd architecture/examples/va/nfv/ovs-dpdk-sriov/edpm | ||
| ``` | ||
| Edit the [values.yaml](edpm/values.yaml) file to suit | ||
| your environment. | ||
| ``` | ||
| vi values.yaml | ||
| ``` | ||
| Generate the dataplane CRs. | ||
| ``` | ||
| kustomize build > dataplane.yaml | ||
| ``` | ||
|
|
||
| ## Create CRs | ||
| ``` | ||
| oc apply -f dataplane.yaml | ||
| ``` | ||
|
|
||
| Wait for dataplane deployment to finish | ||
| ``` | ||
| oc wait osdpd edpm-deployment --for condition=Ready --timeout=1200s | ||
| ``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| dataplane.yaml |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| apiVersion: kustomize.config.k8s.io/v1beta1 | ||
| kind: Kustomization | ||
|
|
||
| components: | ||
| - ../../../../../va/nfv/ovs-dpdk-sriov/edpm/ | ||
| # - https://github.com/openstack-k8s-operators/architecture/va/nfv/ovs-dpdk-sriov/edpm/?ref=main | ||
| ## It's possible to replace ../../../../../va/nfv/ovs-dpdk-sriov/edpm/ with a git checkout URL as per: | ||
| ## https://github.com/kubernetes-sigs/kustomize/blob/master/examples/remoteBuild.md | ||
|
|
||
| resources: | ||
| - values.yaml |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,207 @@ | ||
| # local-config: referenced, but not emitted by kustomize | ||
| apiVersion: v1 | ||
| kind: ConfigMap | ||
| metadata: | ||
| name: edpm-values | ||
| annotations: | ||
| config.kubernetes.io/local-config: "true" | ||
| data: | ||
| ssh_keys: | ||
| # Authorized keys that will have access to the dataplane computes via SSH | ||
| authorized: CHANGEME | ||
| # The private key that will have access to the dataplane computes via SSH | ||
| private: CHANGEME2 | ||
| # The public key that will have access to the dataplane computes via SSH | ||
| public: CHANGEME3 | ||
| nodeset: | ||
| ansible: | ||
| ansibleUser: cloud-admin | ||
| ansiblePort: 22 | ||
| ansibleVars: | ||
| # CHANGEME -- see https://access.redhat.com/solutions/253273 | ||
| # edpm_bootstrap_command: | | ||
| # subscription-manager register --username <subscription_manager_username> --password <subscription_manager_password> | ||
| # podman login -u <registry_username> -p <registry_password> registry.redhat.io | ||
| service_net_map: | ||
| nova_api_network: internalapi | ||
| nova_libvirt_network: internalapi | ||
| timesync_ntp_servers: | ||
| - hostname: clock.redhat.com | ||
| # CPU pinning settings | ||
| # edpm nfv ovs dpdk config | ||
| # CHANGEME | ||
| edpm_kernel_args: "default_hugepagesz=1GB hugepagesz=1G hugepages=64 iommu=pt intel_iommu=on tsx=off isolcpus=2-11,14-23" | ||
| edpm_tuned_profile: "cpu-partitioning-powersave" | ||
| edpm_tuned_isolated_cores: "2-11,14-23" | ||
| edpm_nova_libvirt_qemu_group: "hugetlbfs" | ||
| edpm_ovs_dpdk_pmd_core_list: "1,13,2,14,3,15" | ||
| edpm_ovs_dpdk_socket_memory: "4096" | ||
| edpm_ovs_dpdk_memory_channels: "4" | ||
| edpm_ovs_dpdk_vhost_postcopy_support: "true" | ||
| edpm_ovn_bridge_mappings: ['dpdk2:br-link2','dpdk1:br-link1'] | ||
| # edpm nfv sriov config | ||
| edpm_neutron_sriov_agent_SRIOV_NIC_physical_device_mappings: 'sriov1:eno5,sriov2:eno6' | ||
| # edpm_network_config | ||
| # These vars are edpm_network_config role vars | ||
| edpm_network_config_hide_sensitive_logs: false | ||
| edpm_network_config_os_net_config_mappings: | ||
| # Need to provide nic mapping based on system uuid or system product name | ||
| # here used systetm uuid | ||
| edpm-compute-0: # CHANGEME | ||
| dmiString: "system-uuid" # CHANGEME | ||
| id: "4c4c4544-0034-3810-8050-cac04f325432" # CHANGEME | ||
| nic1: eno1 # CHANGEME | ||
| nic2: eno2 # CHANGEME | ||
| nic3: eno3 # CHANGEME | ||
| nic4: eno4 # CHANGEME | ||
| nic5: eno5 # CHANGEME | ||
| nic6: eno6 # CHANGEME | ||
| edpm_network_config_template: | | ||
| --- | ||
| {% set mtu_list = [ctlplane_mtu] %} | ||
| {% for network in role_networks %} | ||
| {{ mtu_list.append(lookup('vars', networks_lower[network] ~ '_mtu')) }} | ||
| {%- endfor %} | ||
| {% set min_viable_mtu = mtu_list | max %} | ||
| network_config: | ||
| - type: ovs_bridge | ||
| name: {{ neutron_physical_bridge_name }} | ||
| mtu: {{ min_viable_mtu }} | ||
| use_dhcp: false | ||
| dns_servers: {{ ctlplane_dns_nameservers }} | ||
| domain: {{ dns_search_domains }} | ||
| addresses: | ||
| - ip_netmask: {{ ctlplane_ip }}/{{ ctlplane_cidr }} | ||
| routes: {{ ctlplane_host_routes }} | ||
| members: | ||
| - type: interface | ||
| name: nic2 | ||
| mtu: {{ min_viable_mtu }} | ||
| # force the MAC address of the bridge to this interface | ||
| primary: true | ||
| {% for network in role_networks if network not in ['External', 'Tenant'] %} | ||
| - type: vlan | ||
| mtu: {{ lookup('vars', networks_lower[network] ~ '_mtu') }} | ||
| vlan_id: {{ lookup('vars', networks_lower[network] ~ '_vlan_id') }} | ||
| addresses: | ||
| - ip_netmask: | ||
| {{ lookup('vars', networks_lower[network] ~ '_ip') }}/{{ lookup('vars', networks_lower[network] ~ '_cidr') }} | ||
| routes: {{ lookup('vars', networks_lower[network] ~ '_host_routes') }} | ||
| {% endfor %} | ||
| - type: ovs_user_bridge | ||
| name: br-link1 | ||
| use_dhcp: false | ||
| ovs_extra: "set port br-link1 tag={{ lookup('vars', networks_lower['Tenant'] ~ '_vlan_id') }}" | ||
| addresses: | ||
| - ip_netmask: {{ lookup('vars', networks_lower['Tenant'] ~ '_ip') }}/{{ lookup('vars', networks_lower['Tenant'] ~ '_cidr') }} | ||
| mtu: {{ lookup('vars', networks_lower[network] ~ '_mtu') }} | ||
| rx_queue: 1 | ||
| members: | ||
| - type: ovs_dpdk_port | ||
| name: dpdk1 | ||
| members: | ||
| - type: interface | ||
| name: nic3 | ||
| - type: ovs_user_bridge | ||
| name: br-link2 | ||
| use_dhcp: false | ||
| mtu: 9000 | ||
| rx_queue: 1 | ||
| members: | ||
| - type: ovs_dpdk_port | ||
| name: dpdk2 | ||
| members: | ||
| - type: interface | ||
| name: nic4 | ||
| - type: sriov_pf | ||
| name: nic5 | ||
| numvfs: 10 | ||
| mtu: 9000 | ||
| use_dhcp: false | ||
| promisc: true | ||
| - type: sriov_pf | ||
| name: nic6 | ||
| numvfs: 10 | ||
| mtu: 9000 | ||
| use_dhcp: false | ||
| promisc: true | ||
| # These vars are for the network config templates themselves and are | ||
| # considered EDPM network defaults. | ||
| neutron_physical_bridge_name: br-ex | ||
| neutron_public_interface_name: nic1 | ||
| # edpm_nodes_validation | ||
| edpm_nodes_validation_validate_controllers_icmp: false | ||
| edpm_nodes_validation_validate_gateway_icmp: false | ||
| dns_search_domains: [] | ||
| gather_facts: false | ||
| enable_debug: false | ||
| # edpm firewall, change the allowed CIDR if needed | ||
| edpm_sshd_configure_firewall: true | ||
| edpm_sshd_allowed_ranges: | ||
| - 192.168.122.0/24 | ||
| # SELinux module | ||
| edpm_selinux_mode: enforcing | ||
| nodes: | ||
| edpm-compute-0: | ||
| ansible: | ||
| ansibleHost: 192.168.122.100 | ||
| hostName: edpm-compute-0 | ||
| networks: | ||
| - defaultRoute: true | ||
| fixedIP: 192.168.122.100 | ||
| name: CtlPlane | ||
| subnetName: subnet1 | ||
| - name: InternalApi | ||
| subnetName: subnet1 | ||
| - name: Storage | ||
| subnetName: subnet1 | ||
| - name: Tenant | ||
| subnetName: subnet1 | ||
| edpm-compute-1: | ||
| ansible: | ||
| ansibleHost: 192.168.122.101 | ||
| hostName: edpm-compute-1 | ||
| networks: | ||
| - defaultRoute: true | ||
| fixedIP: 192.168.122.101 | ||
| name: CtlPlane | ||
| subnetName: subnet1 | ||
| - name: InternalApi | ||
| subnetName: subnet1 | ||
| - name: Storage | ||
| subnetName: subnet1 | ||
| - name: Tenant | ||
| subnetName: subnet1 | ||
| services: | ||
| - bootstrap | ||
| - download-cache | ||
| - reboot-os | ||
| - configure-ovs-dpdk | ||
| - configure-network | ||
| - validate-network | ||
| - install-os | ||
| - configure-os | ||
| - run-os | ||
| - ovn | ||
| - neutron-metadata | ||
| - neutron-sriov | ||
| - libvirt | ||
| - nova-custom-ovsdpdksriov | ||
| - telemetry | ||
| nova: | ||
| compute: | ||
| conf: | | ||
| [DEFAULT] | ||
| reserved_host_memory_mb = 4096 | ||
| [compute] | ||
| cpu_shared_set = 0-3,24-27 | ||
| cpu_dedicated_set = 8-23,32-47 | ||
| migration: | ||
| ssh_keys: | ||
| private: CHANGEME4 | ||
| public: CHANGEME5 | ||
| pci: | ||
| conf: | | ||
| [pci] | ||
| device_spec = {"vendor_id":"8086", "product_id":"1572", "address": "0000:19:00.3", "physical_network":"sriov1", "trusted":"true"} #CHANGEME | ||
| device_spec = {"vendor_id":"8086", "product_id":"1572", "address": "0000:20:00.3", "physical_network":"sriov2", "trusted":"true"} #CHANGEME |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| apiVersion: kustomize.config.k8s.io/v1beta1 | ||
| kind: Kustomization | ||
|
|
||
| components: | ||
| - ../../../../va/nfv/ovs-dpdk-sriov/ | ||
| # - https://github.com/openstack-k8s-operators/architecture/va/nfv/ovs-dpdk-sriov?ref=main | ||
| ## It's possible to replace ../../../va/nfv/ovs-dpdk-sriov/ with a git checkout URL as per: | ||
| ## https://github.com/kubernetes-sigs/kustomize/blob/master/examples/remoteBuild.md | ||
|
|
||
| resources: | ||
| - nncp/values.yaml | ||
| - service-values.yaml |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| apiVersion: kustomize.config.k8s.io/v1beta1 | ||
| kind: Kustomization | ||
|
|
||
| transformers: | ||
| # Set namespace to OpenStack on all namespaced objects without a namespace | ||
| - |- | ||
| apiVersion: builtin | ||
| kind: NamespaceTransformer | ||
| metadata: | ||
| name: _ignored_ | ||
| namespace: openstack | ||
| setRoleBindingSubjects: none | ||
| unsetOnly: true | ||
| fieldSpecs: | ||
| - path: metadata/name | ||
| kind: Namespace | ||
| create: true | ||
| components: | ||
| - ../../../../../lib/nncp | ||
|
|
||
| resources: | ||
| - values.yaml |
Oops, something went wrong.