Skip to content

Commit

Permalink
Clear up some older comments
Browse files Browse the repository at this point in the history
  • Loading branch information
Torec Luik committed Aug 29, 2023
1 parent d2bd51e commit 36ea451
Show file tree
Hide file tree
Showing 3 changed files with 37 additions and 6 deletions.
33 changes: 32 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,36 @@ The class provides methods for running commands on the remote Slurm host, submit
It also offers a `from_config` class method to create a `SlurmClient` object by reading configuration parameters from a file. Overall, the class provides a convenient way to work with Slurm clusters and manage job execution and monitoring.

# Prerequisites & Getting Started
Note: This library has only been tested on Slurm version 21.08.6 !

## Slurm Requirements
Note: This library has only been tested on Slurm version 21.08.6 and 22.05.09 !

Your Slurm cluster/login node needs to have:
1. SSH access w/ public key (headless)
2. SCP access (generally comes with SSH)
3. 7zip
4. Singularity/Apptainer
5. (Optional) Git

## Omero Requirements

Your Omero _processing_ node needs to have:
1. SSH client and access to the Slurm cluster (w/ private key / headless)
2. SCP access to the Slurm cluster
3. Python3.6+
4. This library installed (`python3 -m pip install 'git+https://github.com/NL-BioImaging/omero-slurm-client'`)
5. Configuration setup at `/etc/slurm-config.ini`
6. (Optional) requirements for some scripts: `python3 -m pip install ezomero==1.1.1 tifffile==2020.9.3`

Your Omero _server_ node needs to have:
1. Some Omero example scripts installed to interact with this library:
- My examples on github: `https://github.com/TorecLuik/docker-example-omero-grid-amc/tree/processors/scripts/slurm`
- Install those `scripts/slurm/*` at `/opt/omero/server/OMERO.server/lib/scripts/slurm/` (or make some more subfolders)




## Getting Started

To connect an Omero processor to a Slurm cluster using the `omero_slurm_client` library, users can follow these steps:

Expand Down Expand Up @@ -58,6 +87,8 @@ with SlurmClient.from_config(configfile=configfile,
```
7. With the configuration files in place, users can utilize the `SlurmClient` class from the Omero-Slurm library to connect to the Slurm cluster over SSH, enabling the submission and management of Slurm jobs from an Omero processor.

I have also provided a tutorial on connecting to a Local or Cloud Slurm. Those can give some more insights as well.


# slurm-config.ini
The `slurm-config.ini` file is a configuration file used by the `omero_slurm_client` Python package to specify various settings related to SSH and Slurm. Here is a brief description of its contents:
Expand Down
3 changes: 1 addition & 2 deletions resources/job_template.sh
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,7 @@
# force-stopped by the server. If you make the expected time too long, it will
# take longer for the job to start. Here, we say the job will take 15 minutes.
# d-hh:mm:ss
# Note that we use the timeout command in the actual script to requeue the job.
#SBATCH --time=00:15:00
#SBATCH --time=00:45:00

# How much memory you need.
# --mem will define memory per node
Expand Down
7 changes: 4 additions & 3 deletions resources/tutorials/tutorial_GoogleCloud_slurm.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ However, we are missing an ingredient: SSH access!
## 2. Add SSH access

### TL;DR:
1. Add your public SSH key (`~/.ssh/id_rsa.pub`) to the Google Cloud instance, like [here](https://cloud.google.com/compute/docs/connect/add-ssh-keys?cloudshell=true#gcloud).
1. Add your public SSH key (`~/.ssh/id_rsa.pub`) to the Google Cloud instance, like [here](https://cloud.google.com/compute/docs/connect/add-ssh-keys?cloudshell=true#gcloud). Easiest is with Cloud shell, upload your public key, and run `gcloud compute os-login ssh-keys add --key-file=id_rsa.pub`
2. Turn the [firewall](https://console.cloud.google.com/net-security/firewall-manager/firewall-policies/list) setting (e.g. `hpc-small-net-fw-allow-iap-ingress`) to allow `0.0.0.0/0` as IP ranges for `tcp:22`.
3. Promote the login node's IP address to a static one: [here](https://cloud.google.com/compute/docs/ip-addresses/reserve-static-external-ip-address#promote_ephemeral_ip)
4. Copy that IP and your username.
Expand All @@ -66,7 +66,7 @@ Follow the steps at [here](https://cloud.google.com/compute/docs/connect/add-ssh
0. Note that this tutorial by default seems to use the "OS Login" method, using the mail account you signed up with.
1. Open a Cloud Shell
2. Upload your public key to this Cloud Shell (with the `...` button).
3. Run the `gcloud compute ...` command they show, pointing at your newly uploaded public key. Leave out the optional `project` and `expire_time`.
3. Run the `gcloud compute os-login ssh-keys add --key-file=id_rsa.pub` command they show, pointing at your newly uploaded public key. Leave out the optional `project` and `expire_time`.

Then, we have to ensure that the firewall accepts requests from outside Google Cloud, if it doesn't already.

Expand Down Expand Up @@ -148,7 +148,7 @@ All on the same node that was spun up, on-demand, by Google Cloud. You should be

</details>

## 3b. Install Singularity / Apptainer
## 3b. Install requirements: Singularity / Apptainer and 7zip

### TL;DR:
1. Follow this [guide](https://cloud.google.com/architecture/deploying-containerized-workloads-slurm-cluster-compute-engine) to install Singularity in /app
Expand All @@ -157,6 +157,7 @@ All on the same node that was spun up, on-demand, by Google Cloud. You should be
```
echo 'export PATH=/apps/singularity/3.8.7/bin:/usr/sbin:${PATH}' >> ~/.bashrc && source ~/.bashrc
```
3. Install 7zip: `sudo yum install -y p7zip p7zip-plugins`

<details>

Expand Down

0 comments on commit 36ea451

Please sign in to comment.