This file contains information concerning the use of the new job array 
features in TORQUE 2.5.

--- WARNING --- 
TORQUE 2.5 uses a new format for job arrays. It in not backwards
compatible with job arrays from version 2.3 or 2.4. Therefore, it is 
imperative that the system be drained of any job arrays BEFORE upgrading. 
Upgrading with job arrays queued or running may cause data loss, crashes, 
etc, and is not supported.


COMMAND UPDATES FOR ARRAYS
--------------------------

The commands qalter, qdel, qhold, and qrls now all support TORQUE 
arrays and will have to be updated. The general command syntax is: 

command <array_name> [-t array_range] [other command options]

The array ranges accepted by -t here are exactly the same as the array ranges
that can be specified in qsub.
(http://www.clusterresources.com/products/torque/docs/commands/qsub.shtml)


SLOT LIMITS
--------------------------

It is now possible to limit the number of jobs that can run concurrently in a
job array. This is called a slot limit, and the default is unlimited. The slot
limit can be set in two ways.

The first method can be done at job submission:

qsub script.sh -t 0-299%5

This sets the slot limit to 5, meaning only 5 jobs from this array can be
running at the same time. 

The second method can be done on a server wide basis using the server 
parameter max_slot_limit. Since administrators are more likely to
be concerned with limiting arrays than users in many cases the 
max_slot_limit parameter is a convenient way to set a global policy.
If max_slot_limit is not set then the default limit is unlimited. To set 
max_slot_limit you can use the following queue manager command.

qmgr -c 'set server max_slot_limit=10'

This means that no array can request a slot limit greater than 10, and 
any array not requesting a slot limit will receive a slot limit of 10.
If a user requests a slot limit greater than 10, the job will be
rejected with the message:

Requested slot limit is too large, limit is X. In this case, X would be 10.

It is recommended that if you are using torque with a scheduler like Moab or Maui
that you also set the server parameter moab_array_compatible=true. Setting 
moab_array_compatible will put all jobs over the slot limit on hold
so the scheduler will not try and schedule jobs above the slot limit.


JOB ARRAY DEPENDENCIES
--------------------------

The following dependencies can now be used for job arrays:

afterstartarray 
afterokarray 
afternotokarray 
afteranyarray 
beforestartarray 
beforeokarray 
beforenotokarray 
beforeanyarray

The general syntax is:

qsub script.sh -W depend=dependtype:array_name[num_jobs]

The suffix [num_jobs] should appear exactly as above, although the number of
jobs is optional. If it isn't specified, the dependency will assume that it is
the entire array, for example: 

qsub script.sh -W depend=afterokarray:427[] 

will assume every job in array 427[] has to finish successfully for the
dependency to be satisfied. The submission: 

qsub script.sh -W depend=afterokarray:427[][5] 

means that 5 of the jobs in array 427 have to successfully finish in order for
the dependency to be satisfied.

NOTE: It is important to remember that the "[]" is part of the array name.


QSTAT FOR JOB ARRAYS
--------------------------

Normal qstat output will display a summary of the array instead of displaying
the entire array, job for job. 

qstat -t will expand the output to display the entire array.


ARRAY NAMING CONVENTION
--------------------------

Arrays are now named with brackets following the array name, for example:

dbeer@napali:~/dev/torque/array_changes$ echo sleep 20 | qsub -t 0-299 
189[].napali 

Individual jobs in the array are now also noted using square brackets instead of
dashes, for example, here is part of the output of qstat -t for the above
array:

189[287].napali            STDIN[287]       dbeer                  0 Q batch          
189[288].napali            STDIN[288]       dbeer                  0 Q batch          
189[289].napali            STDIN[289]       dbeer                  0 Q batch          
189[290].napali            STDIN[290]       dbeer                  0 Q batch          
189[291].napali            STDIN[291]       dbeer                  0 Q batch          
189[292].napali            STDIN[292]       dbeer                  0 Q batch          
189[293].napali            STDIN[293]       dbeer                  0 Q batch          
189[294].napali            STDIN[294]       dbeer                  0 Q batch          
189[295].napali            STDIN[295]       dbeer                  0 Q batch          
189[296].napali            STDIN[296]       dbeer                  0 Q batch          
189[297].napali            STDIN[297]       dbeer                  0 Q batch          
189[298].napali            STDIN[298]       dbeer                  0 Q batch          
189[299].napali            STDIN[299]       dbeer                  0 Q batch