Update interproscan.sh #111

mahesh-panchal · 2024-05-28T15:11:16Z

Update interproscan

Update Interproscan module to https://github.com/nf-core/modules/pull/5688/files locally
Automatically download DB if path not supplied.
Add cache dir so it doesn't need to be downloaded again.
Update README with instructions.

Misc

blast tag bug fix.

mahesh-panchal · 2024-06-17T07:52:45Z

Note to self:

Error in running [ps -o pid,vsize,rss,command -p 9897]

was in the interproscan stack trace. The biocontainers container doesn't support these flags.

$ ps -o pid,vsize,rss,command
ps: bad -o argument 'vsize', supported arguments: user,group,comm,args,pid,ppid,pgid,etime,nice,rgroup,ruser,time,tty,vsz,sid,stat,rss

However ps in the Seqera containers wave container does. Need to update the docker image too.

gbdias · 2024-06-20T07:48:31Z

I tried with the latest code and I'm getting an error that suggests the pipeline is looking for conda environments, although I'm not using the conda profile.

run command:

nextflow run /home/guibo205/git/pipelines-nextflow \
  -profile singularity,nbis \
  -params-file params.yml \
  -with-report report.html -with-trace \
  -resume

repo version

(base) guibo205@nac-login:~/git/pipelines-nextflow$ git log -1
commit ab2f579a929717924380755ae173f90ceafac0b9 (HEAD -> update_interproscan.sh, origin/update_interproscan.sh)
Author: Mahesh Binzer-Panchal <mahesh.binzer-panchal@nbis.se>
Date:   Mon Jun 17 07:49:10 2024 +0000

    Use seqera containers to fix ps error

Nextflow log

executor >  slurm (13)
[47/25b1d3] process > FUNCTIONAL_ANNOTATION:UNTAR... [100%] 1 of 1 ✔
[11/83012a] process > FUNCTIONAL_ANNOTATION:BLAST... [100%] 1 of 1 ✔
[69/045f65] process > FUNCTIONAL_ANNOTATION:GFF2P... [100%] 1 of 1 ✔
[84/297b5c] process > FUNCTIONAL_ANNOTATION:BLAST... [100%] 5 of 5 ✔
[eb/b77950] process > FUNCTIONAL_ANNOTATION:INTER... [100%] 1 of 1, failed: 1
[-        ] process > FUNCTIONAL_ANNOTATION:MERGE... -
Error executing process > 'FUNCTIONAL_ANNOTATION:INTERPROSCAN (braker)'

Caused by:
  Process `FUNCTIONAL_ANNOTATION:INTERPROSCAN (braker)` terminated with an error exit status (1)

Command executed:

  if [ -d 'data' ]; then
      # Find interproscan.properties to link data/ from work directory
      INTERPROSCAN_DIR="$( dirname "$( dirname "$( which interproscan.sh )" )" )"
      INTERPROSCAN_PROPERTIES="$( find "$INTERPROSCAN_DIR" -name "interproscan.properties" )"
      cp "$INTERPROSCAN_PROPERTIES" .
      sed -i "/^bin\.directory=/ s|.*|bin.directory=$INTERPROSCAN_DIR/bin|" interproscan.properties
      export INTERPROSCAN_CONF=interproscan.properties
  fi # else use sample DB included with conda ( testing only! )

  if false ; then
      gzip -c -d braker_proteins.2.fasta > braker_proteins.2.fasta
  fi

  interproscan.sh \
      --cpu 8 \
      --input braker_proteins.2.fasta \
      --goterms -pa -t p -dra \
      --output-file-base braker

  cat <<-END_VERSIONS > versions.yml
  "FUNCTIONAL_ANNOTATION:INTERPROSCAN":
      interproscan: $( interproscan.sh --version | sed '1!d; s/.*version //' )
  END_VERSIONS

Command exit status:
  1

Command output:
  (empty)

Command error:
  cp: cannot stat '/opt/conda/pkgs/interproscan-5.59_91.0-hec16e2b_1/info/recipe/interproscan.properties'$'\n''/opt/conda/pkgs/interproscan-5.59_91.0-hec16e2b_1/share/InterProScan/interproscan.properties'$'\n''/opt/conda/share/InterProScan/interproscan.properties': No such file or directory

Work dir:
  /projects/annotation/Babesia_ovis/annotation/func_test/work/87/5d10042f420270a13732d6a7dd4a93

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`

mahesh-panchal · 2024-07-01T12:42:10Z

Seqera containers uses micromamba internally to build images. It appears we're getting multiple hits that wasn't the case with biocontainers.

find "$INTERPROSCAN_DIR" -name "interproscan.properties"
/opt/conda/pkgs/interproscan-5.59_91.0-hec16e2b_1/info/recipe/interproscan.properties
/opt/conda/pkgs/interproscan-5.59_91.0-hec16e2b_1/share/InterProScan/interproscan.properties
/opt/conda/share/InterProScan/interproscan.properties

I'll see how best to solve this.

mahesh-panchal · 2024-07-01T13:23:59Z

See if this solves it.

gbdias · 2024-08-26T11:26:42Z

I tried to run the workflow with commit 1fa4a09 and am getting the following error:

WARN: Killing running tasks (4)

executor >  slurm (13)
[58/65f50a] process > FUNCTIONAL_ANNOTATION:UNTAR... [100%] 1 of 1 ✔
[58/be2693] process > FUNCTIONAL_ANNOTATION:BLAST... [100%] 1 of 1 ✔
[ab/b01d0b] process > FUNCTIONAL_ANNOTATION:GFF2P... [100%] 1 of 1 ✔
[4f/f90a15] process > FUNCTIONAL_ANNOTATION:BLAST... [100%] 5 of 5 ✔
[d9/d1e69f] process > FUNCTIONAL_ANNOTATION:INTER... [200%] 4 of 2, failed: 4
[-        ] process > FUNCTIONAL_ANNOTATION:MERGE... -
Error executing process > 'FUNCTIONAL_ANNOTATION:INTERPROSCAN (braker)'

Caused by:
  Process `FUNCTIONAL_ANNOTATION:INTERPROSCAN (braker)` terminated with an error exit status (231)

Command executed:

  if [ -d 'data' ]; then
      # Find interproscan.properties to link data/ from work directory
      INTERPROSCAN_DIR="$( dirname "$( dirname "$( which interproscan.sh )" )" )"
      INTERPROSCAN_PROPERTIES="$( find "$INTERPROSCAN_DIR/share" -name "interproscan.properties" )"
      cp "$INTERPROSCAN_PROPERTIES" .
      sed -i "/^bin\.directory=/ s|.*|bin.directory=$INTERPROSCAN_DIR/bin|" interproscan.properties
      export INTERPROSCAN_CONF=interproscan.properties
  fi # else use sample DB included with conda ( testing only! )

  if false ; then
      gzip -c -d braker_proteins.1.fasta > braker_proteins.1.fasta
  fi

  interproscan.sh \
      --cpu 8 \
      --input braker_proteins.1.fasta \
      --goterms -pa -t p -dra \
      --output-file-base braker

  cat <<-END_VERSIONS > versions.yml
  "FUNCTIONAL_ANNOTATION:INTERPROSCAN":
      interproscan: $( interproscan.sh --version | sed '1!d; s/.*version //' )
  END_VERSIONS

Command exit status:
  231

Command output:
  2024-08-26 08:46:56,777 [amqEmbeddedWorkerJmsContainer-7] [uk.ac.ebi.interpro.scan.jms.worker.LocalJobQueueListener:218] ERROR - The exception is :
  2024-08-26 08:46:56,778 [amqEmbeddedWorkerJmsContainer-7] [uk.ac.ebi.interpro.scan.jms.worker.LocalJobQueueListener:222] ERROR - StepExecution with errors - stepName: stepPIRSFRunBinary
  2024-08-26 08:46:56,800 [main] [uk.ac.ebi.interpro.scan.jms.master.StandaloneBlackBoxMaster:190] WARN - StepInstance 45 is being re-run following a failure.
  2024-08-26 08:47:00,569 [amqEmbeddedWorkerJmsContainer-6] [uk.ac.ebi.interpro.scan.management.model.implementations.RunBinaryStep:199] ERROR - Command line failed with exit code: 1
  Command: hmmscan -E 10 -Z 15438 --cpu 1 -o /scratch/nxf.etD05lvZne/temp/node-a02_20240826_084611226_hrhs//jobSuperFamily-1.75/000000000001_000000000878.raw.out data/superfamily/1.75/hmmlib_1.75 /scratch/nxf.etD05lvZne/temp/node-a02_20240826_084611226_hrhs//jobSuperFamily-1.75/000000000001_000000000878.fasta
  Error output from binary:

  Error: Failed to open binary auxfiles for data/superfamily/1.75/hmmlib_1.75: use hmmpress first


  2024-08-26 08:47:00,569 [amqEmbeddedWorkerJmsContainer-6] [uk.ac.ebi.interpro.scan.jms.worker.LocalJobQueueListener:216] ERROR - Execution thrown when attempting to executeInTransaction the StepExecution.  All database activity rolled back.
  java.lang.IllegalStateException: Command line failed with exit code: 1
  Command: hmmscan -E 10 -Z 15438 --cpu 1 -o /scratch/nxf.etD05lvZne/temp/node-a02_20240826_084611226_hrhs//jobSuperFamily-1.75/000000000001_000000000878.raw.out data/superfamily/1.75/hmmlib_1.75 /scratch/nxf.etD05lvZne/temp/node-a02_20240826_084611226_hrhs//jobSuperFamily-1.75/000000000001_000000000878.fasta
  Error output from binary:

  Error: Failed to open binary auxfiles for data/superfamily/1.75/hmmlib_1.75: use hmmpress first


  	at uk.ac.ebi.interpro.scan.management.model.implementations.RunBinaryStep.execute(RunBinaryStep.java:201) ~[interproscan-management-5.59-91.0.jar:?]
  	at uk.ac.ebi.interpro.scan.jms.activemq.StepExecutionTransactionImpl.executeInTransaction(StepExecutionTransactionImpl.java:87) ~[interproscan-5.jar:?]
  	at jdk.internal.reflect.GeneratedMethodAccessor103.invoke(Unknown Source) ~[?:?]
  	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
  	at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
  	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) ~[spring-aop-5.2.22.RELEASE.jar:5.2.22.RELEASE]
  	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) ~[spring-aop-5.2.22.RELEASE.jar:5.2.22.RELEASE]
  	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.2.22.RELEASE.jar:5.2.22.RELEASE]
  	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:367) ~[spring-tx-5.2.22.RELEASE.jar:5.2.22.RELEASE]
  	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:118) ~[spring-tx-5.2.22.RELEASE.jar:5.2.22.RELEASE]
  	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.2.22.RELEASE.jar:5.2.22.RELEASE]
  	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) ~[spring-aop-5.2.22.RELEASE.jar:5.2.22.RELEASE]
  	at com.sun.proxy.$Proxy141.executeInTransaction(Unknown Source) ~[?:?]
  	at uk.ac.ebi.interpro.scan.jms.worker.LocalJobQueueListener.onMessage(LocalJobQueueListener.java:200) [interproscan-5.jar:?]
  	at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:761) [spring-jms-5.2.22.RELEASE.jar:5.2.22.RELEASE]
  	at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:699) [spring-jms-5.2.22.RELEASE.jar:5.2.22.RELEASE]
  	at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:674) [spring-jms-5.2.22.RELEASE.jar:5.2.22.RELEASE]
  	at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:318) [spring-jms-5.2.22.RELEASE.jar:5.2.22.RELEASE]
  	at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:257) [spring-jms-5.2.22.RELEASE.jar:5.2.22.RELEASE]
  	at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1186) [spring-jms-5.2.22.RELEASE.jar:5.2.22.RELEASE]
  	at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1176) [spring-jms-5.2.22.RELEASE.jar:5.2.22.RELEASE]
  	at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1073) [spring-jms-5.2.22.RELEASE.jar:5.2.22.RELEASE]
  	at java.lang.Thread.run(Thread.java:829) [?:?]
  2024-08-26 08:47:00,570 [amqEmbeddedWorkerJmsContainer-6] [uk.ac.ebi.interpro.scan.jms.worker.LocalJobQueueListener:218] ERROR - The exception is :
  2024-08-26 08:47:00,571 [amqEmbeddedWorkerJmsContainer-6] [uk.ac.ebi.interpro.scan.jms.worker.LocalJobQueueListener:222] ERROR - StepExecution with errors - stepName: stepSuperFamilyRunHmmer3-1.75
  2024-08-26 08:47:00,589 [main] [uk.ac.ebi.interpro.scan.jms.activemq.NonZeroExitOnUnrecoverableError:25] FATAL - Analysis step 108 : Run HMMER3 for proteins 1 to 878 has failed irretrievably.  Available StackTraces follow.
  2024-08-26 08:47:00,589 [main] [uk.ac.ebi.interpro.scan.jms.activemq.NonZeroExitOnUnrecoverableError:42] FATAL - The JVM will now exit with a non-zero exit status.
  2024-08-26 08:47:00,589 [main] [uk.ac.ebi.interpro.scan.jms.master.StandaloneBlackBoxMaster:363] ERROR - Exception thrown by StandaloneBlackBoxMaster:
  java.lang.IllegalStateException: InterProScan exiting with non-zero status, see logs for further information.
  	at uk.ac.ebi.interpro.scan.jms.activemq.NonZeroExitOnUnrecoverableError.failed(NonZeroExitOnUnrecoverableError.java:43) ~[interproscan-5.jar:?]
  	at uk.ac.ebi.interpro.scan.jms.master.StandaloneBlackBoxMaster.run(StandaloneBlackBoxMaster.java:169) [interproscan-5.jar:?]
  	at uk.ac.ebi.interpro.scan.jms.main.Run.main(Run.java:413) [interproscan-5.jar:?]

Command error:
  	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:367)
  	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:118)
  	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
  	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
  	at com.sun.proxy.$Proxy141.executeInTransaction(Unknown Source)
  	at uk.ac.ebi.interpro.scan.jms.worker.LocalJobQueueListener.onMessage(LocalJobQueueListener.java:200)
  	at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:761)
  	at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:699)
  	at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:674)
  	at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:318)
  	at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:257)
  	at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1186)
  	at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1176)
  	at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1073)
  	at java.base/java.lang.Thread.run(Thread.java:829)
  java.lang.IllegalStateException: Command line failed with exit code: 1
  Command: hmmscan -E 10 -Z 15438 --cpu 1 -o /scratch/nxf.etD05lvZne/temp/node-a02_20240826_084611226_hrhs//jobSuperFamily-1.75/000000000001_000000000878.raw.out data/superfamily/1.75/hmmlib_1.75 /scratch/nxf.etD05lvZne/temp/node-a02_20240826_084611226_hrhs//jobSuperFamily-1.75/000000000001_000000000878.fasta
  Error output from binary:

  Error: Failed to open binary auxfiles for data/superfamily/1.75/hmmlib_1.75: use hmmpress first


  	at uk.ac.ebi.interpro.scan.management.model.implementations.RunBinaryStep.execute(RunBinaryStep.java:201)
  	at uk.ac.ebi.interpro.scan.jms.activemq.StepExecutionTransactionImpl.executeInTransaction(StepExecutionTransactionImpl.java:87)
  	at jdk.internal.reflect.GeneratedMethodAccessor103.invoke(Unknown Source)
  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
  	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
  	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
  	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
  	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:367)
  	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:118)
  	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
  	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
  	at com.sun.proxy.$Proxy141.executeInTransaction(Unknown Source)
  	at uk.ac.ebi.interpro.scan.jms.worker.LocalJobQueueListener.onMessage(LocalJobQueueListener.java:200)
  	at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:761)
  	at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:699)
  	at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:674)
  	at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:318)
  	at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:257)
  	at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1186)
  	at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1176)
  	at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1073)
  	at java.base/java.lang.Thread.run(Thread.java:829)
  java.lang.IllegalStateException: InterProScan exiting with non-zero status, see logs for further information.
  	at uk.ac.ebi.interpro.scan.jms.activemq.NonZeroExitOnUnrecoverableError.failed(NonZeroExitOnUnrecoverableError.java:43)
  	at uk.ac.ebi.interpro.scan.jms.master.StandaloneBlackBoxMaster.run(StandaloneBlackBoxMaster.java:169)
  	at uk.ac.ebi.interpro.scan.jms.main.Run.main(Run.java:413)
  InterProScan analysis failed. Exception thrown by StandaloneBlackBoxMaster. Check the log file for details

Work dir:
  /projects/annotation/Babesia_ovis/annotation/func_test/work/02/5fa3c8249e0c9ffef7189e4f2d3311

Tip: when you have fixed the problem you can continue the execution adding the option `-resume` to the run command line

This link mentions it might be an issue with incompatible HMMER version and specific HMM libraries from some databases
https://interproscan-docs.readthedocs.io/en/latest/UserDocs.html#hmmer-errors

mahesh-panchal added 7 commits May 28, 2024 14:24

Add local version of nf-core interproscan

dd23d58

Add nf-core untar module

4a75480

Add interproscan db to params and functional annotation pipeline

3702635

Add cache dir for interpro db

2e0ed32

Add NBIS cluster info

f45a595

Update usage instructions

ccbe7ea

Bug fix: blast tag

de5e6e3

mahesh-panchal requested a review from MartinPippel May 28, 2024 15:18

mahesh-panchal linked an issue May 28, 2024 that may be closed by this pull request

Inconsistent InterProScan results between GAAS and manual run #110

Open

mahesh-panchal removed a link to an issue May 28, 2024

Inconsistent InterProScan results between GAAS and manual run #110

Open

mahesh-panchal added 2 commits May 30, 2024 13:00

Fix function name

4afcf60

Remove meta from untar

2a755ec

mahesh-panchal requested a review from gbdias June 4, 2024 09:09

mahesh-panchal added 3 commits June 5, 2024 12:13

remove iprlookup as it's being skipped anyway

6af85a4

fix interproscan conda package

baf6656

Use seqera containers to fix ps error

ab2f579

Add share to interpro properties find path

1fa4a09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update interproscan.sh #111

Update interproscan.sh #111

mahesh-panchal commented May 28, 2024 •

edited

Loading

mahesh-panchal commented Jun 17, 2024 •

edited

Loading

gbdias commented Jun 20, 2024

mahesh-panchal commented Jul 1, 2024

mahesh-panchal commented Jul 1, 2024

gbdias commented Aug 26, 2024

Update interproscan.sh #111

Are you sure you want to change the base?

Update interproscan.sh #111

Conversation

mahesh-panchal commented May 28, 2024 • edited Loading