Skip to content

Commit

Permalink
[pre-commit.ci] auto fixes from pre-commit.com hooks
Browse files Browse the repository at this point in the history
for more information, see https://pre-commit.ci
  • Loading branch information
pre-commit-ci[bot] committed Oct 29, 2024
1 parent 3af22e5 commit 32ed46b
Show file tree
Hide file tree
Showing 62 changed files with 228 additions and 34 deletions.
3 changes: 3 additions & 0 deletions helm-charts/codegen-openshift-rhoai/Chart.yaml
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

apiVersion: v2
name: codegen
description: A Helm chart for deploying codegen on Red Hat OpenShift with Red Hat OpenShift AI
Expand Down
43 changes: 28 additions & 15 deletions helm-charts/codegen-openshift-rhoai/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,18 +2,19 @@

Helm chart for deploying CodeGen service on Red Hat OpenShift with Red Hat OpenShift AI.

Serving runtime template in this example uses model *ise-uiuc/Magicoder-S-DS-6.7B* for Xeon and *meta-llama/CodeLlama-7b-hf* for Gaudi.
Serving runtime template in this example uses model _ise-uiuc/Magicoder-S-DS-6.7B_ for Xeon and _meta-llama/CodeLlama-7b-hf_ for Gaudi.

## Prerequisites
## Prerequisites

1. **Red Hat OpenShift Cluster** with dynamic *StorageClass* to provision *PersistentVolumes* e.g. **OpenShift Data Foundation**) and installed Operators: **Red Hat - Authorino (Technical Preview)**, **Red Hat OpenShift Service Mesh**, **Red Hat OpenShift Serverless** and **Red Hat Openshift AI**.
1. **Red Hat OpenShift Cluster** with dynamic _StorageClass_ to provision _PersistentVolumes_ e.g. **OpenShift Data Foundation**) and installed Operators: **Red Hat - Authorino (Technical Preview)**, **Red Hat OpenShift Service Mesh**, **Red Hat OpenShift Serverless** and **Red Hat Openshift AI**.
2. Image registry to push there docker images (https://docs.openshift.com/container-platform/4.16/registry/securing-exposing-registry.html).
3. Access to S3-compatible object storage bucket (e.g. **OpenShift Data Foundation**, **AWS S3**) and values of access and secret access keys and S3 endpoint (https://docs.redhat.com/en/documentation/red_hat_openshift_data_foundation/4.16/html/managing_hybrid_and_multicloud_resources/accessing-the-multicloud-object-gateway-with-your-applications_rhodf#accessing-the-multicloud-object-gateway-with-your-applications_rhodf).
4. Account on https://huggingface.co/, access to model *ise-uiuc/Magicoder-S-DS-6.7B* (for Xeon) or *meta-llama/CodeLlama-7b-hf* (for Gaudi) and token with Read permissions.
4. Account on https://huggingface.co/, access to model _ise-uiuc/Magicoder-S-DS-6.7B_ (for Xeon) or _meta-llama/CodeLlama-7b-hf_ (for Gaudi) and token with Read permissions.

## Deploy model in Red Hat Openshift AI

1. Login to OpenShift CLI and run following commands to create new serving runtime and *hf-token* secret.
1. Login to OpenShift CLI and run following commands to create new serving runtime and _hf-token_ secret.

```
cd GenAIInfra/helm-charts/codegen-openshift-rhoai/
export HFTOKEN="insert-your-huggingface-token-here"
Expand All @@ -25,24 +26,29 @@ On Gaudi:
helm install servingruntime tgi --set global.huggingfacehubApiToken=${HFTOKEN} --values tgi/gaudi-values.yaml
```

Verify if template has been created with ```oc get template -n redhat-ods-applications``` command.
Verify if template has been created with `oc get template -n redhat-ods-applications` command.

2. Find the route for **Red Hat OpenShift AI** dashboard with below command and open it in the browser:

```
oc get routes -A | grep rhods-dashboard
```
3. Go to **Data Science Project** and clik **Create data science project**. Fill the **Name** and click **Create**.
4. Go to **Workbenches** tab and clik **Create workbench**. Fill the **Name**, under **Notebook image** choose *Standard Data Science*, under **Cluster storage** choose *Create new persistent storage* and change **Persistent storage size** to 40 GB. Click **Create workbench**.

3. Go to **Data Science Project** and click **Create data science project**. Fill the **Name** and click **Create**.
4. Go to **Workbenches** tab and click **Create workbench**. Fill the **Name**, under **Notebook image** choose _Standard Data Science_, under **Cluster storage** choose _Create new persistent storage_ and change **Persistent storage size** to 40 GB. Click **Create workbench**.
5. Open newly created Jupiter notebook and run following commands to download the model and upload it on s3:

```
%env S3_ENDPOINT=<S3_RGW_ROUTE>
%env S3_ACCESS_KEY=<AWS_ACCESS_KEY_ID>
%env S3_SECRET_KEY=<AWS_SECRET_ACCESS_KEY>
%env HF_TOKEN=<PASTE_HUGGINGFACE_TOKEN>
```

```
!pip install huggingface-hub
```

```
import os
import boto3
Expand All @@ -63,15 +69,21 @@ s3_resource = session.resource('s3',
aws_secret_access_key=s3_secretkey)
bucket = s3_resource.Bucket(bucket_name)
```
For Xeon download *ise-uiuc/Magicoder-S-DS-6.7B*:

For Xeon download _ise-uiuc/Magicoder-S-DS-6.7B_:

```
snapshot_download("ise-uiuc/Magicoder-S-DS-6.7B", cache_dir=f'./models', token=hf_token)
```
For Gaudi download *meta-llama/CodeLlama-7b-hf*:

For Gaudi download _meta-llama/CodeLlama-7b-hf_:

```
snapshot_download("meta-llama/CodeLlama-7b-hf", cache_dir=f'./models', token=hf_token)
```

Upload the downloaded model to S3:

```
files = (file for file in glob.glob(f'{path}/**/*', recursive=True) if os.path.isfile(file) and "snapshots" in file)
for filename in files:
Expand All @@ -80,8 +92,8 @@ for filename in files:
bucket.upload_file(filename, f'{path}{s3_name}')
```

6. Go to your project in **Red Hat OpenShift AI** dashboard, then "Models" tab and click **Deploy model** under *Single-model serving platform*. Fill the **Name**, choose newly created **Serving runtime**: *Text Generation Inference Magicoder-S-DS-6.7B on CPU* (for Xeon) or *Text Generation Inference CodeLlama-7b-hf on Gaudi* (for Gaudi), **Model framework**: *llm* and change **Model server size** to *Custom*: 16 CPUs and 64 Gi memory. For deployment with Gaudi select proper **Accelerator**. Click the checkbox to create external route in **Model route** section and uncheck the **Token authentication**. Under **Model location** choose *New data connection* and fill all required fields for s3 access, **Bucket** *first.bucket* and **Path**: *models*. Click **Deploy**. It takes about 10 minutes to get *Loaded* status.\
If it's not going to *Loaded* status and revision changed status to "ProgressDeadlineExceeded" (``oc get revision``), scale model deployment to 0 and than to 1 with command ``oc scale deployment.apps/<model_deployment_name> --replicas=1`` and wait about 10 minutes for deployment.
6. Go to your project in **Red Hat OpenShift AI** dashboard, then "Models" tab and click **Deploy model** under _Single-model serving platform_. Fill the **Name**, choose newly created **Serving runtime**: _Text Generation Inference Magicoder-S-DS-6.7B on CPU_ (for Xeon) or _Text Generation Inference CodeLlama-7b-hf on Gaudi_ (for Gaudi), **Model framework**: _llm_ and change **Model server size** to _Custom_: 16 CPUs and 64 Gi memory. For deployment with Gaudi select proper **Accelerator**. Click the checkbox to create external route in **Model route** section and uncheck the **Token authentication**. Under **Model location** choose _New data connection_ and fill all required fields for s3 access, **Bucket** _first.bucket_ and **Path**: _models_. Click **Deploy**. It takes about 10 minutes to get _Loaded_ status.\
If it's not going to _Loaded_ status and revision changed status to "ProgressDeadlineExceeded" (`oc get revision`), scale model deployment to 0 and than to 1 with command `oc scale deployment.apps/<model_deployment_name> --replicas=1` and wait about 10 minutes for deployment.

## Install the Chart

Expand All @@ -101,11 +113,12 @@ sed -i "s/insert-your-namespace-here/${NAMESPACE}/g" codegen-openshift-rhoai/llm
helm dependency update codegen-openshift-rhoai

helm install codegen codegen-openshift-rhoai --set image.repository=image-registry.openshift-image-registry.svc:5000/${NAMESPACE}/codegen --set llm-uservice.image.repository=image-registry.openshift-image-registry.svc:5000/${NAMESPACE}/llm-tgi --set react-ui.image.repository=image-registry.openshift-image-registry.svc:5000/${NAMESPACE}/react-ui --set global.clusterDomain=${CLUSTERDOMAIN} --set global.huggingfacehubApiToken=${HFTOKEN} --set llm-uservice.servingRuntime.name=${MODELNAME} --set llm-uservice.servingRuntime.namespace=${PROJECT}
```
```

## Verify

To verify the installation, run the command `oc get pods` to make sure all pods are running. Wait about 5 minutes for building images. When 4 pods achieve *Completed* status, the rest with services should go to *Running*.
To verify the installation, run the command `oc get pods` to make sure all pods are running. Wait about 5 minutes for building images. When 4 pods achieve _Completed_ status, the rest with services should go to _Running_.

## Launch the UI
To access the frontend, find the route for *react-ui* with command `oc get routes` and open it in the browser.

To access the frontend, find the route for _react-ui_ with command `oc get routes` and open it in the browser.
3 changes: 3 additions & 0 deletions helm-charts/codegen-openshift-rhoai/llm-uservice/Chart.yaml
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

apiVersion: v2
name: llm-uservice
description: A Helm chart for deploying llm-uservice on Red Hat OpenShift with Red Hat OpenShift AI
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

kind: BuildConfig
apiVersion: build.openshift.io/v1
metadata:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

apiVersion: v1
kind: ConfigMap
metadata:
Expand All @@ -7,7 +10,7 @@ data:
#!/bin/bash
EXISTS=$(oc get secret --ignore-not-found rhoai-ca-bundle)
if [[ -z "${EXISTS}" ]]; then
if [[ -z "${EXISTS}" ]]; then
oc create secret generic -n {{ .Release.Namespace }} rhoai-ca-bundle --from-literal=tls.crt="$(oc extract secret/knative-serving-cert -n istio-system --to=- --keys=tls.crt)"
else
echo "oc get secret --ignore-not-found rhoai-ca-bundle returned non-empty string, not creating a secret"
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

apiVersion: apps/v1
kind: Deployment
metadata:
Expand All @@ -14,7 +17,7 @@ spec:
template:
metadata:
labels:
{{- include "llm-uservice.selectorLabels" . | nindent 8 }}
{{- include "llm-uservice.selectorLabels" . | nindent 8 }}
spec:
securityContext: {}
containers:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

---
apiVersion: image.openshift.io/v1
kind: ImageStream
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

apiVersion: batch/v1
kind: Job
metadata:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

{{- range $key, $value := .Values.rbac.roles }}
{{- if $value.createRole }}
---
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

{{- range $key, $value := .Values.rbac.roleBindings }}
{{- if $value.createBinding }}
apiVersion: rbac.authorization.k8s.io/v1
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

apiVersion: v1
kind: ServiceAccount
metadata:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

apiVersion: v1
kind: Service
metadata:
Expand Down
5 changes: 4 additions & 1 deletion helm-charts/codegen-openshift-rhoai/llm-uservice/values.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,7 @@
image:
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

image:
repository: "insert-your-image-here" # image-registry.openshift-image-registry.svc:5000/<namespace>/llm-tgi
tag: "latest"
pullPolicy: IfNotPresent
Expand Down
3 changes: 3 additions & 0 deletions helm-charts/codegen-openshift-rhoai/react-ui/Chart.yaml
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

apiVersion: v2
name: react-ui
description: A Helm chart deploying OPEA CodeGen React UI
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

---
apiVersion: image.openshift.io/v1
kind: ImageStream
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

---
apiVersion: v1
data:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

apiVersion: console.openshift.io/v1
kind: ConsoleLink
metadata:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

---
apiVersion: apps/v1
kind: Deployment
Expand Down Expand Up @@ -46,9 +49,9 @@ spec:
httpGet:
path: /
port: http
resources:
resources:
{{- toYaml .Values.resources | nindent 10 }}
securityContext:
securityContext:
{{- toYaml .Values.securityContext | nindent 10 }}
volumeMounts:
- mountPath: /var/cache/nginx/
Expand All @@ -63,10 +66,10 @@ spec:
- name: nginx-cache-dir
emptyDir: {}
- name: default-conf
configMap:
configMap:
name: {{ include "codegen-react-ui.fullname" . }}-default-conf
- name: nginx-conf
configMap:
configMap:
name: {{ include "codegen-react-ui.fullname" . }}-nginx-conf
{{- with .Values.nodeSelector }}
nodeSelector:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

---
apiVersion: route.openshift.io/v1
kind: Route
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

---
apiVersion: v1
kind: Service
Expand Down
3 changes: 3 additions & 0 deletions helm-charts/codegen-openshift-rhoai/react-ui/values.yaml
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

# Default values for codegen-react-ui.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

kind: BuildConfig
apiVersion: build.openshift.io/v1
metadata:
Expand Down
3 changes: 3 additions & 0 deletions helm-charts/codegen-openshift-rhoai/templates/deployment.yaml
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

apiVersion: apps/v1
kind: Deployment
metadata:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

---
apiVersion: image.openshift.io/v1
kind: ImageStream
Expand Down
3 changes: 3 additions & 0 deletions helm-charts/codegen-openshift-rhoai/templates/route.yaml
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

apiVersion: route.openshift.io/v1
kind: Route
metadata:
Expand Down
3 changes: 3 additions & 0 deletions helm-charts/codegen-openshift-rhoai/templates/service.yaml
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

apiVersion: v1
kind: Service
metadata:
Expand Down
3 changes: 3 additions & 0 deletions helm-charts/codegen-openshift-rhoai/tgi/Chart.yaml
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

apiVersion: v2
name: tgi
description: A Helm chart for deploying servingruntime for ise-uiuc/Magicoder-S-DS-6.7B model on Red Hat OpenShift with Red Hat OpenShift AI
Expand Down
3 changes: 3 additions & 0 deletions helm-charts/codegen-openshift-rhoai/tgi/gaudi-values.yaml
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

global:
huggingfacehubApiToken: "insert-your-huggingface-token-here"

Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

apiVersion: v1
kind: Secret
metadata:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

---
apiVersion: template.openshift.io/v1
kind: Template
Expand Down
5 changes: 4 additions & 1 deletion helm-charts/codegen-openshift-rhoai/tgi/values.yaml
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

global:
huggingfacehubApiToken: "insert-your-huggingface-token-here"

Expand Down Expand Up @@ -25,7 +28,7 @@ resources: {}
volumes:
- emptyDir:
sizeLimit: 300Gi
name: model-volume
name: model-volume

volumeMounts:
- mountPath: /data
Expand Down
3 changes: 3 additions & 0 deletions helm-charts/codegen-openshift-rhoai/values.yaml
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

global:
clusterDomain: "insert-your-cluster-domain-here"

Expand Down
3 changes: 3 additions & 0 deletions helm-charts/codegen-openshift/Chart.yaml
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

apiVersion: v2
name: codegen
description: A Helm chart for deploying codegen on Red Hat OpenShift
Expand Down
Loading

0 comments on commit 32ed46b

Please sign in to comment.