Merge pull request #282 from ENCODE-DCC/dev

v2.2.0

README.md

@@ -3,36 +3,22 @@
 [![CircleCI](https://circleci.com/gh/ENCODE-DCC/chip-seq-pipeline2/tree/master.svg?style=svg)](https://circleci.com/gh/ENCODE-DCC/chip-seq-pipeline2/tree/master)
 
 
-## Download new Caper>=2.1
+## Conda environment name change (since v2.2.0 or 6/13/2022)
 
-New Caper is out. You need to update your Caper to work with the latest ENCODE ChIP-seq pipeline.
-```bash
-$ pip install caper --upgrade
+Pipeline's Conda environment's names have been shortened to work around the following error:
 ```
-
-## Local/HPC users and new Caper>=2.1
-
-There are tons of changes for local/HPC backends: `local`, `slurm`, `sge`, `pbs` and `lsf`(added). Make a backup of your current Caper configuration file `~/.caper/default.conf` and run `caper init`. Local/HPC users need to reset/initialize Caper's configuration file according to your chosen backend. Edit the configuration file and follow instructions in there.
-```bash
-$ cd ~/.caper
-$ cp default.conf default.conf.bak
-$ caper init [YOUR_BACKEND]
+PaddingError: Placeholder of length '80' too short in package /XXXXXXXXXXX/miniconda3/envs/
 ```
 
-In order to run a pipeline, you need to add one of the following flags to specify the environment to run each task within. i.e. `--conda`, `--singularity` and `--docker`. These flags are not required for cloud backend users (`aws` and `gcp`).
+You need to reinstall pipeline's Conda environment. It's recommended to do this for every version update.
 ```bash
-# for example
-$ caper run ... --singularity
+$ bash scripts/uninstall_conda_env.sh
+$ bash scripts/install_conda_env.sh
 ```
 
-For Conda users, **RE-INSTALL PIPELINE'S CONDA ENVIRONMENT AND DO NOT ACTIVATE CONDA ENVIRONMENT BEFORE RUNNING PIPELINES**. Caper will internally call `conda run -n ENV_NAME CROMWELL_JOB_SCRIPT`. Just make sure that pipeline's new Conda environments are correctly installed.
-```bash
-$ scripts/uninstall_conda_env.sh
-$ scripts/install_conda_env.sh
-```
 
+## Introduction
 
-## Introduction 
 This ChIP-Seq pipeline is based off the ENCODE (phase-3) transcription factor and histone ChIP-seq pipeline specifications (by Anshul Kundaje) in [this google doc](https://docs.google.com/document/d/1lG_Rd7fnYgRpSIqrIfuVlAz2dW1VaSQThzk836Db99c/edit#).
 
 ### Features
@@ -45,30 +31,44 @@ This ChIP-Seq pipeline is based off the ENCODE (phase-3) transcription factor an
 
 1) Make sure that you have Python>=3.6. Caper does not work with Python2. Install Caper and check its version >=2.0.
 	```bash
-	$ python --version
 	$ pip install caper
+
+	# use caper version >= 2.3.0 for a new HPC feature (caper hpc submit/list/abort).
+	$ caper -v
 	```
-2) Make a backup of your Caper configuration file `~/.caper/default.conf` if you are upgrading from old Caper(<2.0.0). Reset/initialize Caper's configuration file. Read Caper's [README](https://github.com/ENCODE-DCC/caper/blob/master/README.md) carefully to choose a backend for your system. Follow the instruction in the configuration file.
+2) Read Caper's [README](https://github.com/ENCODE-DCC/caper/blob/master/README.md) carefully to choose a backend for your system. Follow the instruction in the configuration file.
 	```bash
-	# make a backup of ~/.caper/default.conf if you already have it
+	# this will overwrite the existing conf file ~/.caper/default.conf
+	# make a backup of it first if needed
 	$ caper init [YOUR_BACKEND]
 
-	# then edit ~/.caper/default.conf
+	# edit the conf file
 	$ vi ~/.caper/default.conf
 	```
 
 3) Git clone this pipeline.
-	> **IMPORTANT**: use `~/chip-seq-pipeline2/chip.wdl` as `[WDL]` in Caper's documentation.
 	```bash
 	$ cd
 	$ git clone https://github.com/ENCODE-DCC/chip-seq-pipeline2
 	```
 
-4) (Optional for Conda) Install pipeline's Conda environments if you don't have Singularity or Docker installed on your system. We recommend to use Singularity instead of Conda. If you don't have Conda on your system, install [Miniconda3](https://docs.conda.io/en/latest/miniconda.html).
+4) (Optional for Conda) **DO NOT USE A SHARED CONDA. INSTALL YOUR OWN [MINICONDA3](https://docs.conda.io/en/latest/miniconda.html) AND USE IT.** Install pipeline's Conda environments if you don't have Singularity or Docker installed on your system. We recommend to use Singularity instead of Conda.
 	```bash
+	# check if you have Singularity on your system, if so then it's not recommended to use Conda
+	$ singularity --version
+
+	# check if you are not using a shared conda, if so then delete it or remove it from your PATH
+	$ which conda
+
+	# change directory to pipeline's git repo
 	$ cd chip-seq-pipeline2
-	# uninstall old environments (<2.0.0)
+
+	# uninstall old environments
 	$ bash scripts/uninstall_conda_env.sh
+
+	# install new envs, you need to run this for every pipeline version update.
+	# it may be killed if you run this command line on a login node.
+	# it's recommended to make an interactive node and run it there.
 	$ bash scripts/install_conda_env.sh
 	```
 
@@ -88,20 +88,22 @@ You can use URIs(`s3://`, `gs://` and `http(s)://`) in Caper's command lines and
 
 According to your chosen platform of Caper, run Caper or submit Caper command line to the cluster. You can choose other environments like `--singularity` or `--docker` instead of `--conda`. But you must define one of the environments.
 
-The followings are just examples. Please read [Caper's README](https://github.com/ENCODE-DCC/caper) very carefully to find an actual working command line for your chosen platform.
+PLEASE READ [CAPER'S README](https://github.com/ENCODE-DCC/caper) VERY CAREFULLY BEFORE RUNNING ANY PIPELINES. YOU WILL NEED TO CORRECTLY CONFIGURE CAPER FIRST. These are just example command lines.
+
     ```bash
-    # Run it locally with Conda (You don't need to activate it, make sure to install Conda envs first)
+    # Run it locally with Conda (DO NOT ACTIVATE PIPELINE'S CONDA ENVIRONEMT)
     $ caper run chip.wdl -i https://storage.googleapis.com/encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI_subsampled_chr19_only.json --conda
 
-    # Or submit it as a leader job (with long/enough resources) to SLURM (Stanford Sherlock) with Singularity
-    # It will fail if you directly run the leader job on login nodes
-    $ sbatch -p [SLURM_PARTITON] -J [WORKFLOW_NAME] --export=ALL --mem 4G -t 4-0 --wrap "caper chip chip.wdl -i https://storage.googleapis.com/encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI_subsampled_chr19_only.json --singularity"
+    # On HPC, submit it as a leader job to SLURM with Singularity
+    $ caper hpc submit chip.wdl -i https://storage.googleapis.com/encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI_subsampled_chr19_only.json --singularity --leader-job-name ANY_GOOD_LEADER_JOB_NAME
 
-    # Check status of your leader job
-    $ squeue -u $USER | grep [WORKFLOW_NAME]
+    # Check job ID and status of your leader jobs
+    $ caper hpc list
 
     # Cancel the leader node to close all of its children jobs
-    $ scancel -j [JOB_ID]    
+    # If you directly use cluster command like scancel or qdel then
+    # child jobs will not be terminated
+    $ caper hpc abort [JOB_ID]
 	```
 
 

chip.wdl

@@ -7,10 +7,10 @@ struct RuntimeEnvironment {
 }
 
 workflow chip {
-    String pipeline_ver = 'v2.1.6'
+    String pipeline_ver = 'v2.2.0'
 
     meta {
-        version: 'v2.1.6'
+        version: 'v2.2.0'
 
         author: 'Jin wook Lee'
         email: 'leepc12@gmail.com'
@@ -19,8 +19,8 @@ workflow chip {
 
         specification_document: 'https://docs.google.com/document/d/1lG_Rd7fnYgRpSIqrIfuVlAz2dW1VaSQThzk836Db99c/edit?usp=sharing'
 
-        default_docker: 'encodedcc/chip-seq-pipeline:v2.1.6'
-        default_singularity: 'https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.1.6.sif'
+        default_docker: 'encodedcc/chip-seq-pipeline:v2.2.0'
+        default_singularity: 'https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.0.sif'
         croo_out_def: 'https://storage.googleapis.com/encode-pipeline-output-definition/chip.croo.v5.json'
 
         parameter_group: {
@@ -71,11 +71,11 @@ workflow chip {
     }
     input {
         # group: runtime_environment
-        String docker = 'encodedcc/chip-seq-pipeline:v2.1.6'
-        String singularity = 'https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.1.6.sif'
-        String conda = 'encode-chip-seq-pipeline'
-        String conda_macs2 = 'encode-chip-seq-pipeline-macs2'
-        String conda_spp = 'encode-chip-seq-pipeline-spp'
+        String docker = 'encodedcc/chip-seq-pipeline:v2.2.0'
+        String singularity = 'https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.0.sif'
+        String conda = 'encd-chip'
+        String conda_macs2 = 'encd-chip-macs2'
+        String conda_spp = 'encd-chip-spp'
 
         # group: pipeline_metadata
         String title = 'Untitled'
@@ -228,7 +228,7 @@ workflow chip {
         Int xcor_time_hr = 24
         Float xcor_disk_factor = 4.5
 
-        Float subsample_ctl_mem_factor = 14.0
+        Float subsample_ctl_mem_factor = 22.0
         Float subsample_ctl_disk_factor = 15.0
 
         Float macs2_signal_track_mem_factor = 12.0
@@ -261,17 +261,17 @@ workflow chip {
         conda: {
             description: 'Default Conda environment name to run WDL tasks. For Conda users only.',
             group: 'runtime_environment',
-            example: 'encode-atac-seq-pipeline'
+            example: 'encd-chip'
         }
         conda_macs2: {
             description: 'Conda environment name for task macs2. For Conda users only.',
             group: 'runtime_environment',
-            example: 'encode-atac-seq-pipeline-macs2'
+            example: 'encd-chip-macs2'
         }
         conda_spp: {
             description: 'Conda environment name for tasks spp/xcor. For Conda users only.',
             group: 'runtime_environment',
-            example: 'encode-atac-seq-pipeline-spp'
+            example: 'encd-chip-spp'
         }
         title: {
             description: 'Experiment title.',

docs/build_genome_database.md

@@ -8,11 +8,7 @@
 
 # How to build genome database
 
-1. [Install Conda](https://conda.io/miniconda.html). Skip this if you already have equivalent Conda alternatives (Anaconda Python). Download and run the [installer](https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh). Agree to the license term by typing `yes`. It will ask you about the installation location. On Stanford clusters (Sherlock and SCG4), we recommend to install it outside of your `$HOME` directory since its filesystem is slow and has very limited space. At the end of the installation, choose `yes` to add Miniconda's binary to `$PATH` in your BASH startup script.
-    ```bash
-    $ wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
-    $ bash Miniconda3-latest-Linux-x86_64.sh
-    ```
+1. [Install Conda](https://conda.io/miniconda.html).
 
 2. Install pipeline's Conda environment.
     ```bash
@@ -22,7 +18,7 @@
 
 2. Choose `GENOME` from `hg19`, `hg38`, `mm9` and `mm10` and specify a destination directory. This will take several hours. We recommend not to run this installer on a login node of your cluster. It will take >8GB memory and >2h time.
     ```bash
-    $ conda activate encode-chip-seq-pipeline
+    $ conda activate encd-chip
     $ bash scripts/build_genome_data.sh [GENOME] [DESTINATION_DIR]
     ```
 

docs/input.md

@@ -302,7 +302,7 @@ Parameter|Default|Description
 
 Parameter|Default|Description
 ---------|-------|-----------
-`chip.subsample_ctl_mem_factor` | 14.0 | Multiplied to size of TAG-ALIGN BED to determine required memory
+`chip.subsample_ctl_mem_factor` | 22.0 | Multiplied to size of TAG-ALIGN BED to determine required memory
 `chip.macs2_signal_track_time_hr` | 24 | Walltime (HPCs only)
 `chip.subsample_ctl_disk_factor` | 15.0 | Multiplied to size of TAG-ALIGN BED to determine required disk
 

scripts/install_conda_env.sh

@@ -5,20 +5,20 @@ SH_SCRIPT_DIR=$(cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd)
 
 echo "$(date): Installing pipeline's Conda environments..."
 
-conda create -n encode-chip-seq-pipeline --file ${SH_SCRIPT_DIR}/requirements.txt \
+conda create -n encd-chip --file ${SH_SCRIPT_DIR}/requirements.txt \
   --override-channels -c bioconda -c defaults -y
 
-conda create -n encode-chip-seq-pipeline-macs2 --file ${SH_SCRIPT_DIR}/requirements.macs2.txt \
+conda create -n encd-chip-macs2 --file ${SH_SCRIPT_DIR}/requirements.macs2.txt \
   --override-channels -c bioconda -c defaults -y
 
-conda create -n encode-chip-seq-pipeline-spp --file ${SH_SCRIPT_DIR}/requirements.spp.txt \
+conda create -n encd-chip-spp --file ${SH_SCRIPT_DIR}/requirements.spp.txt \
   --override-channels -c r -c bioconda -c defaults -y
 
 # adhoc fix for the following issues:
 # - https://github.com/ENCODE-DCC/chip-seq-pipeline2/issues/259
 # - https://github.com/ENCODE-DCC/chip-seq-pipeline2/issues/265
 # force-install readline 6.2, ncurses 5.9 from conda-forge (ignoring dependencies)
-conda install -n encode-chip-seq-pipeline-spp --no-deps --no-update-deps -y \
+conda install -n encd-chip-spp --no-deps --no-update-deps -y \
   readline==6.2 ncurses==5.9 -c conda-forge
 
 echo "$(date): Done successfully."

scripts/uninstall_conda_env.sh

@@ -1,9 +1,9 @@
 #!/bin/bash
 
 PIPELINE_CONDA_ENVS=(
-  encode-chip-seq-pipeline
-  encode-chip-seq-pipeline-macs2
-  encode-chip-seq-pipeline-spp
+  encd-chip
+  encd-chip-macs2
+  encd-chip-spp
 )
 for PIPELINE_CONDA_ENV in "${PIPELINE_CONDA_ENVS[@]}"
 do

scripts/update_conda_env.sh

@@ -5,9 +5,9 @@ SH_SCRIPT_DIR=$(cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd)
 SRC_DIR=${SH_SCRIPT_DIR}/../src
 
 PIPELINE_CONDA_ENVS=(
-  encode-chip-seq-pipeline
-  encode-chip-seq-pipeline-macs2
-  encode-chip-seq-pipeline-spp
+  encd-chip
+  encd-chip-macs2
+  encd-chip-spp
 )
 chmod u+rx ${SRC_DIR}/*.py