Final - Feedback Loop Implementation (#158)

* GH Actions branch for New PR and Close PR issues (#105)

* Sample Workload

* Sample Workload

* Sample Workload

* Sample Workload

* GH Actions

* GH Actions

* Github Workflow

* updated branch name

* added file removal on unsuccessful Close

* Close pull request remove configmap

* added file copying

* subdirectory check

* filecopy using branch checkout

* fix branch name

* Copy partner Directory

* Copy partner Directory update

* Moving sample workload

* remove unnecessary workflow

* workflow fix addition

* adding files to commit

* GH actions source check

* Update cross branch checkout path

* Check subdirectory

* Add files with ConfigMap

* close pull req updates to remove files

* remove commented out code

* update ghActions file

* remove workflow in branch

* add gh action workflow

* echo directory

* update directory reference

* update directory reference 2

* add files from Partner

* add files from Partner 2

* add files from Partner 3

* add files from Partner 4

* add files from Partner 5

* add files from Partner 6

* Reset new pull

* check env state

* list files check

* foreach fix

* update for each syntax

* update files variable

* subdirectory variable pull

* remove escape char

* remove brackets

* remove quotes and extra dollar sign

* add reference to files

* file checkout logic added

* Fix if else indent

* remove extra paren

* remove extra path

* change checkout to absolute path

* adding tracking comments and path edit

* recording Commit ID

* fix git ID variable

* fix git hash command

* return to pre-hash recording

* align commit messages

* align commit messages

* list automated Commits from this PR

* delete files test

* Git log addition

* fix namespace extraction

* update commits string ID and show hashes only

* commit tagging update

* git log --all addition

* update logging

* checkout dev branch

* checkout origin dev branch

* edit query string

* revert query string

* remove merge, set back to developer branch

* test git logging on PR open

* Update podinfo.yaml

* Revert "Feature/sample workload test 7.25"

* Push prev day changes

* Search for existing namespace file

* Reupload sample workload files

* Looking for existing namespace file

* Namespace file searchj

* Namespace file search

* Revert to working state

* Check Dir from updated files

* Check Dir from updated files 2

* Check Dir from updated files array

* Check Dir from updated files array 2

* test for existing namespace file

* test for existing namespace file, fix for statements

* test for existing namespace file, fix if statement

* test for existing namespace file update

* uncomment subdir assignment

* Update Logic to find namespace file

* Update configMap Message

* Update revert logic

* Update git log

* Update branch checkout

* Test Git log

* re-order commits

* revert instead of delete

* revert variable fix

* remove dir

* fix Commits variable

* push git revert

* more precise commit reverts

* more precise commit reverts, fix hash

* back to merge method

* reorder new PR file, add directory search

* Update time on pod def

* New file search logic

* lookup namespace file before searching committed files

* Add missing quote

* Alter directory check

* update file search logic

* variable test

* variable test 2

* step reduction

* fix close script

* add tag for conformitron bot

* fixing echo destination

* add rows on new pr

* Update Step language to be more descriptive of step being performed

* add Hash to ConfigMap

* update test

* update sha on configmap

* Update commit order

* Add config mapand updated files for podinfo_PR_16

* Update merge logic

* fix bad if statement

* Update merge logic

* Update merge logic 2

* Update merge logic with if statement

* Fix IF statement

* Fetch first

* infert IF test

* Comment out additional merge

* Add push

* Add config map for podinfo_PR_16

* Checkout dev branch at beginning

* Remove ConfirMap from Feature Branch

* Change File move commit process

* Close PR commit logging

* Update podinfo.yaml

* Update close workflow

* Delete sample workload

* Trying to fix Conflict

* Workflow Files

* Merge removal of Files

* add workflow files back in

* Updating workflow to work for Net new

* Spacing on new IF statement

* Spacing on new IF statement 2

* Create Subdirectory if not already in Dev branch

* Fix Filepath for ConfigMap file

* Fix Filepath for ConfigMap file

* Git branch and commit test

* Fetch commits not on the runner, list

* Massive Simplifying test

* add commit

* add author

* Fix Checkout to do deletions

* Spacing on IF statement

* Fixing files search

* Git sha testing

* Git sha testing quote pull

* Update git log

* Quote

* file extension

* file name

* Git log test

* Git log test 2

* Git log test 3

* Git log test 4

* Git log test 4

* Git log test 5

* Git log test 6

* Git log test 7

* Git log test 8

* Git log test 9

* Git log test 10

* Tags

* Tags 2

* Cleanup

* Tag Reverts

* Add Pull

* Add Pull to correct spot

* remove Long history

* Remove Tag on Close

* Edit revert command

* Use Commit Message Tag and rev-list grep to ID reverts

* Fix the rev-list command with HEAD variable

* Pull branch history

* Comment out tagging

* different pull strategy

* Check without tag filtering

* fetch depth

* Update Rev-List adding tag back in

* Clear out old commented code in Close PR

* Revert work

* Add Author before revert

* Remove other re-set method

* Add Error handling for Revert ops

* Fix error handling

* check delete

* Fixing weird git issue

* Delete eks-anywhere-common/Addons/Partner/workload_sample directory

* Space added

* Fix GH actions rollback

* Fix new namespace file search

* Fix new namespace file search commentary

* Adding then's

* adding fi

* Fixing branch checkouts

* Fixing branch fetch

* Move the ConfigMap over to Dev Branch

* Update the Addition of ConfigMap file to Dev Branch

* Commit new ConfigMap

* Commit new ConfigMap author addition

* Cleaning Up Merge

* Re-order branch checkout

* Update ConfigMapFile path

* Make Subdirectory

* Comment Out Extra Checkout

* adding -p option for mkdir

* E2E Testing Updates

* Cleanup commented Rows

* Set feature/e2e as target branch

---------

Co-authored-by: Elamaran Shanmugam <elamaran.shan@gmail.com>
Co-authored-by: Conformitron Bot <dev@null>

* E2E cross-fork testing branch (#130)

* Added Kube-Observer basics along with RBAC

* Created Observer Bot

* Added ObserverBot gitignore

* Added .idea into .gitignore

* Finalized the token, secrets, rbac, sa and observer properties

* Added high level skeleton, can find failing pods and return why they're failing

* Added .dev.env to gitignore

* Added information to make future work easier

* Added report generation into the system, still testing it

* Report generated, need to add fallback for a container/pod not returning logs

* Added logs for pods that doesn't produce any logs

* Reformatted the file

* Running into an issue where PATs can't create comments

* Comments are being sent to GitHub and Conformitron v1 should be good to go :)

* Added in the yamls used to test the feedback loop bot

* changed the pod_status.phase to check for failures

* Added in workloads

* Updated it so that the repo and repo owner are retrived from env vars

* Added configurability to what repo is used for observability

* Added Kustomization yamls for observer deployment

* Update new-pull-request.yaml

added namespace to workflow

* Update new-pull-request.yaml

* Added full yamls for observer bot

* Changes making observer-bot prod ready

* Delete close-pull-request.yaml

Removing GitHub Actions

* Removing Github Actions from PR

* Moved the bot.py file to src/ and created a separate config manager to operate on configmaps

* Removed redundant bot.py result of a botched recovery

* Reworked how the bot works with it's storage, confirmed that it actually prevents from sending multiple messages, will set it for an overnight trial

* Caught github failure

* loaded in-cluster config for config_data

* Gave * to observer-sa.yaml

* Least Privileged the SA for observer-bot

* Using least privilege for the things

* Fixed perms for accessing configmaps in the cluster

* Run haywire in the observer namespace

* Spelling mistake

* Modified documentation and added ADOT packages, Will instrument it as part of v1.5

* Changes to make conformitron bot prod ready

* Making changes for token-secret to work in the prod env

* Changed name

* fixed secret location

* Added CI Pipeline *ignores

* Added CI CDK Pipeline for Observer Bot

* Fixed the issue with raw pods throwing errors without reporting back to github

* Added the correct value for ECR_REPO_NAME

* Removed ObserverBot source code from the add-ons repo

* Transferring to main

* Trasnferring to main

* Better naming for Gh Actions

* Better names for Gh Actions

* better name v2 for gh actions

* testing cross-fork PRs

* testing cross-fork PRs

* viewing all the files in the repo at PR state

* getting to the bottom of why the incoming commit isn't available

* Maybe we fetch remote code

* Testing multi repo cloning

* testing with checking out two repos side-by-side

* looking for already downloaded namespace file

* Used the correct location of the configmap file

* #observability

* Copying files from side-by-side folder

* Fetch the PR branch

* Checked out PR for the namespace

* Removed one reminescent ./incoming

* reverted back to actions/checkout@v2

* Moved back to v4 and added more steps to ensure we can push to dev

* Fixed some errors

* Consolidated the steps because namespace error

* Testing with write-all permissions

* Added CI_TOKEN and moved pushing out to a different step

* Commented redundant code

* Added different method for CI_TOKEN

* Removed CI_TOKEN

* Using CI_TOKEN again

* Giving token to different step

* Removed uses

* Added permissions, contents: write to the workflow

* Testing pr_target in gh actions

* Validating prtarget stuff

* Fixed merge back to dev

* Testing with merge squash

* Removed for loop

* Overwriting files in the dev branch based on incoming files

* Force use 'ours' in merge squash

* I'm stupid and now using theirs for the squash strategy

* -X with theirs this time

* Reverting to just checking out files individually

* Fixed exponential revert problem

* Fixed the exponential revert errors

* Fixed some minor syntax

* Fixed indentation?

* Maybe fixed EoF error

* Silly me, forgor to done the loop

* I forgor to remove hardcoded tag

* I forgot to remove hardcoded tag

* Prepping for main merge

---------

Co-authored-by: Mike McDonald <61101829+mikemcd3912@users.noreply.github.com>
Co-authored-by: Elamaran Shanmugam <elamaran.shan@gmail.com>
Co-authored-by: Conformitron Bot <dev@null>

.github/workflows/close-pull-request.yaml

@@ -0,0 +1,30 @@
+name: PR Closed - purging developer_branch
+on: 
+  pull_request_target:
+    branches: [main]
+    types: [closed]
+jobs:
+  merge-master-back-to-dev:
+    if: github.event.pull_request.merged == false
+    timeout-minutes: 2
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v2
+      with:
+        ref: developer_branch
+        fetch-depth: 0
+    - name: Remove Updates from this PR
+      run: |
+        # Get Commits from this PR
+        TAG=PR_${{ github.event.pull_request.number }}
+        commits=$(git rev-list HEAD --grep=$TAG --max-count=1)     
+        echo "commits: $commits"
+    
+        # Revert Commits or Log that no change was made
+        git config --local user.email "dev@null"
+        git config --local user.name "Conformitron Bot"
+
+        git revert $commits --no-edit || echo "Commit $commits not reverted"
+
+        git push
+

.github/workflows/new-pull-request.yaml

@@ -0,0 +1,106 @@
+name: PR Opened - moving new ISV addon to developer_branch for E2E testing
+
+on:
+  pull_request_target:
+    branches: [main]
+    types: [opened, reopened, synchronize]
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+    - name: Checkout Base
+      uses: actions/checkout@v4
+
+    - name: Checkout PR Code
+      run:
+        git fetch origin pull/${{ github.event.pull_request.number }}/head:pr
+
+    - name: Parse Namespace data, Create ConfigMap
+      id: find-namespace-yaml
+      run: |
+        # Pull files down into a filename array
+        files=$(curl -s "https://api.github.com/repos/${{ github.repository }}/pulls/${{ github.event.pull_request.number }}/files" | jq -r '.[].filename')
+        echo $files
+        filearr=($files)
+        
+        # Find Namespace File in uploads list or in modified file subdirectory
+        namespace_file=$(echo "$files" | grep "namespace.yaml")
+        subdirectory=$(dirname ${filearr[0]})
+        if [ -z $namespace_file ]; then
+          git fetch --all
+          git config --local user.email "dev@null"
+          git config --local user.name "Conformitron Bot"
+          git checkout developer_branch
+          git pull
+          if [ -f "${subdirectory}/namespace.yaml" ]; then
+            namespace_file="${subdirectory}/namespace.yaml"
+            echo "namespace file = $namespace_file"
+          else
+            echo "No Namespace file found in existing subdirectory"
+            exit 200
+          fi
+        elif [[ -n $namespace_file ]]; then
+          echo "Namespace File: $namespace_file"
+        else
+          echo "No Namespace file found in commit or subdirectory"
+        fi
+
+        # Parse namespace data
+        namespace_file="${namespace_file}"
+        if [ -n "$namespace_file" ]; then
+          subdirectory=$(dirname $namespace_file)
+          echo "Sub Directory = $subdirectory"
+          
+          # get namespace from pr
+          git checkout pr
+          namespace_name=$(grep -E '^\s*metadata:\s*$|^\s*name:\s*' "$namespace_file" | awk -F':' '{gsub(/ /, "", $2); print $2}')
+          
+          git fetch --all
+          git config --local user.email "dev@null"
+          git config --local user.name "Conformitron Bot"
+          # fetch most recent update to dev
+          git checkout developer_branch
+          git pull
+        
+          if [[ ! -z $namespace_name ]]; then
+            echo "$namespace_name"
+            namespace=$(echo $namespace_name | xargs echo -n)
+            echo $namespace
+            config_map_file="config-map-${{ github.event.pull_request.number }}.yml"
+            echo $config_map_file
+            echo "apiVersion: v1" >> $config_map_file
+            echo "kind: ConfigMap" >> $config_map_file
+            echo "metadata:" >> $config_map_file
+            echo "  name: $namespace-configmap" >> $config_map_file
+            echo "  namespace: $namespace" >> $config_map_file
+            echo "  labels:" >> $config_map_file
+            echo "    bot: conformitron" >> $config_map_file
+            echo "data:" >> $config_map_file
+            echo "  Namespace: ${namespace}" >> $config_map_file
+            echo "  prNumber: \"${{ github.event.pull_request.number }}\"" >> $config_map_file
+            echo "  commitHash: ${{ github.event.pull_request.head.sha }}" >> $config_map_file
+            echo $subdirectory
+            echo $config_map_file
+            
+            mkdir -p ./$subdirectory/
+            mv "$config_map_file" ./$subdirectory/
+            git add .
+        
+          else
+            echo "No Namespace found"
+            exit 100
+          fi
+        else
+          echo "No namespace.yaml file found"
+          exit 200
+        fi
+        
+        # Move updated files over
+        for item in $files; do
+          git checkout pr -- $item
+          git add $item
+          echo $item
+        done
+        git commit -m "Adding new and changed files for ${namespace}_PR_${{ github.event.pull_request.number }}"
+        git push

eks-anywhere-common/Addons/Core/Kube-Observer/kustomization.yaml

@@ -0,0 +1,12 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+metadata:
+  name: observer-prod
+commonLabels:
+  app: observer-prod
+
+resources:
+  - namespace.yaml
+  - observer-sa.yaml
+  - observer-token-secret.yaml
+  - observer-cronjob.yaml

eks-anywhere-common/Addons/Core/Kube-Observer/namespace.yaml

@@ -0,0 +1,4 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: observer

eks-anywhere-common/Addons/Core/Kube-Observer/observer-cronjob.yaml

@@ -0,0 +1,28 @@
+apiVersion: batch/v1
+kind: CronJob
+metadata:
+  name: observer
+  namespace: observer
+spec:
+  # Run at the top of every hour
+  schedule: "0 * * * *"
+  jobTemplate:
+    spec:
+      template:
+        spec:
+          serviceAccountName: observer-sa
+          containers:
+          - name: observer-pod
+            image: public.ecr.aws/n5p5f4n3/conformitron-observer-bot:latest
+            imagePullPolicy: Always
+            env:
+              - name: GITHUB_TOKEN
+                valueFrom:
+                  secretKeyRef:
+                    name: observer-key
+                    key: observer-key
+              - name: REPO
+                value: eks-anywhere-addons
+              - name: OWNER
+                value: aws-samples
+          restartPolicy: Never

eks-anywhere-common/Addons/Core/Kube-Observer/observer-sa.yaml

@@ -0,0 +1,69 @@
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: observer-sa
+  namespace: observer
+
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: watch-pods
+rules:
+  - apiGroups: ["*"]
+    resources: 
+      - "pods"
+      - "namespaces"
+      - "replicasets"
+      - "deployments"
+      - "statefulsets"
+      - "daemonset"
+      - "jobs"
+      - "services"
+      - "configmaps"
+    verbs:
+      - "get"
+      - "list"
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: modified-configmap
+  namespace: observer
+rules:
+  - apiGroups:
+      - "*" # Represents core api group?
+    resources:
+      - "configmaps"
+    verbs:
+      - "*"
+
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: attach-observer-role-to-sa
+  namespace: observer
+subjects:
+  - kind: ServiceAccount
+    name: observer-sa
+    namespace: observer
+roleRef:
+  kind: ClusterRole
+  name: watch-pods
+  apiGroup: rbac.authorization.k8s.io
+
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: attach-configmap-role-to-sa
+  namespace: observer
+subjects:
+  - kind: ServiceAccount
+    name: observer-sa
+    namespace: observer
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: modified-configmap

eks-anywhere-common/Addons/Core/Kube-Observer/observer-token-secret.yaml

@@ -0,0 +1,17 @@
+apiVersion: external-secrets.io/v1beta1
+kind: ExternalSecret
+metadata:
+  name: observer-secret
+  namespace: observer
+spec:
+  refreshInterval: "1m"
+  secretStoreRef:
+    name: eksa-secret-store
+    kind: ClusterSecretStore
+  target:
+    name: observer-key
+    creationPolicy: Owner
+  data:
+  - secretKey: observer-key
+    remoteRef:
+      key: github-token