Center for Cognitive Neuroscience - User contributions [en]

MR

2018-05-11T19:01:50Z

Dmargolis: /* Transfer */

A collection of information about MR data from collection to processing/analysis.

==Collection==
How does one collect MR data?

===Staglin===
The [http://www.semel.ucla.edu/staglin Staglin IMHRO Center for Cognitive Neuroscience] is one of our main data collection facilities.
*[[StaglinCenter:MR]]

===EEG/FMRI===
Concurrent data collection of EEG and FMRI is possible at the Staglin Center.
*[[StaglinCenter:EEG-FMRI]]



==Directory Structure==
Standards are important. And we've got standards about how our MR data should be kept.
*[[MR:DirectoryStructure]]

==Processing==
How to process MR data.

===Tutorials===
Some helpful tutorials about how to process MR data.

*[http://fsl.fmrib.ox.ac.uk/fslcourse/ FSL and FreeSurfer Course]

===Preprocessing===
Certain preprocessing steps are done to all data. This is facilitated by an automated script.
*[[MR:Processing:Preprocessing]]

===DTI===
*[[MR:Processing:DTI]]

Hoffman2:Interactive Sessions

2018-04-18T15:29:52Z

Dmargolis: /* Tips */

[[Hoffman2|Back to all things Hoffman2]]

Interactive sessions on Hoffman2 let you have access to a computing node for up to 24 hours. This is ideal for:
* running a intensive program like MATLAB (in fact that's how it [[Hoffman2:MATLAB|works]]), [[Hoffman2:WEKA|WEKA]], [[Hoffman2:R|R]] or FSLView
* debugging a script you will be submitting to the queue later
* moving/tar'ing/untar'ing lots of files
* any other computing or graphics intensive operations
since you aren't supposed to use the login nodes for such heavy lifting.

==Basic Command==
To get one, you need to use the <code>qrsh</code> command.

For example
$ qrsh
will try to get you an interactive node.

Because you didn't specify a time limit this session will only last two hours after which you will be kicked off of the interactive node back to a login node.

And because you didn't specify a memory limit, the default is 1GB.

If you successfully get a node, your prompt will change from something like
[joebruin@login4 ~] $
to something like
[joebruin@n1234 ~] $
indicating you are on node 1234.

==Longer Time==
If you wanted to specify a a time limit for your interactive session (anything less than 24 hours), use the resource flag again and specify time in the HH:MM:SS format.

For example
$ qrsh -l h_rt=4:00:00
will try securing an interactive node for four hours with the default amount of RAM, but if they are all taken you will be kindly told you are out of luck.

==More Memory==
Doing something memory intensive? Like working with a lot of visualizations or multiple datasets? Use the resource flag again and specify a data request.

For example
$ qrsh -l h_rt=4:00:00,h_data=4G
will try securing an interactive node for four hours with four gigabytes of RAM, but if no such node is available the cluster will deny your request.


==Tips==
Sometimes inactivity on your computer will result in Hoffman2 connection break [ Broken Pipe ] (even while computing).

To prevent this from happening:
For Macs - in your /etc/ssh/ssh_config -add this line to the bottom
ServerAliveInterval 180

This will tell your ssh to ping the server every 180 seconds to prevent it from timing out.

==External Links==
*[http://www.ats.ucla.edu/clusters/hoffman2/computing/sge_qrsh.htm Getting an interactive node]

Hoffman2:Modules

2018-03-02T23:18:29Z

Dmargolis: /* Enabling the CCN Hoffman2 modules */

[[Hoffman2|Back to all things Hoffman2]]

[https://www.hoffman2.idre.ucla.edu/computing/modules/ Environmental Modules] allow users to dynamically modify their shell environment (e.g., $PATH, $LD_LIBRARY_PATH, etc.) in order to support a number of applications installed on the Hoffman2 Cluster. CCN supports several modules on Hoffman2 including: AFNI, ANTs, Caret,FSL, FreeSurfer, ITKGray, ITK-SNAP, and MRIcron.

==Enabling the CCN Hoffman2 modules==
# [[Hoffman2:Accessing the Cluster#SSH - Command Line|SSH into Hoffman2]]
# Use <code>gedit</code> or your favorite [[Text Editors|text editor]] (e.g. nano or vim) to edit the file <code>~/.bashrc</code>
#:* <pre>$ gedit ~/.bashrc</pre>
# Insert these lines at the '''bottom''' of the file
#:* <pre>module use /u/project/CCN/apps/modulefiles
umask 007</pre>
# '''Important:''' remove or comment out these lines (if present):
#:* <pre><nowiki>#</nowiki> Remove or comment out this line!
<nowiki>#</nowiki>source /u/project/FMRI/apps/etc/profile</pre>
#:* <pre><nowiki>#</nowiki> Remove or comment out this line!
<nowiki>#</nowiki>umask 0022</pre>
# Save the file
# Log out of Hoffman2 and the next time you log in, everything will be set for you to start working.

==CCN Hoffman2 modules==
===AFNI===
[https://afni.nimh.nih.gov/ AFNI] (Analysis of Functional NeuroImages) is a set of C programs for processing, analyzing, and displaying functional MRI (FMRI) data - a technique for mapping human brain activity.

# [[Hoffman2:Accessing the Cluster#SSH - Command Line|SSH into Hoffman2]]
# Start an [https://www.hoffman2.idre.ucla.edu/computing/interactive-session/ interactive session] on a compute node with qrsh. Then at the compute node shell prompt, enter:
#: <pre>module load afni</pre>
#To run AFNI, enter:
#: <pre>afni</pre>

===ANTs===

[http://stnava.github.io/ANTs/ ANTs] (Advanced Normalization Tools) computes high-dimensional mappings to capture the statistics of brain structure and function.

# [[Hoffman2:Accessing the Cluster#SSH - Command Line|SSH into Hoffman2]]
# Start an [https://www.hoffman2.idre.ucla.edu/computing/interactive-session/ interactive session] on a compute node with qrsh. Then at the compute node shell prompt, enter:
#: <pre>module load ants</pre>
# Please see the [http://stnava.github.io/ANTs/ ANTs website] for usage examples.

===Caret===

[http://brainvis.wustl.edu/wiki/index.php/Caret:About Caret] is a free, open-source, software package for structuraland functional analyses of the cerebral and cerebellar cortex.

# [[Hoffman2:Accessing the Cluster#SSH - Command Line|SSH into Hoffman2]]
# Start an [https://www.hoffman2.idre.ucla.edu/computing/interactive-session/ interactive session] on a compute node with qrsh. Then at the compute node shell prompt, enter:
#: <pre>module load caret</pre>
#To run Caret, enter:
#: <pre>caret5</pre>

===DCMTK===

[http://www.dcmtk.org/dcmtk.php.en DCMTK] is a collection of libraries and applications implementing large parts the DICOM standard. It includes software for examining, constructing and converting DICOM image files, handling offline media, sending and receiving images over a network connection, as well as demonstrative image storage and worklist servers.

# [[Hoffman2:Accessing the Cluster#SSH - Command Line|SSH into Hoffman2]]
# Start an [https://www.hoffman2.idre.ucla.edu/computing/interactive-session/ interactive session] on a compute node with qrsh. Then at the compute node shell prompt, enter:
#: <pre>module load dcmtk</pre>
#To run (for example) dcmdump, enter:
#: <pre>dcmdump [options] dcmfile(s)</pre>

===FSL===

[https://fsl.fmrib.ox.ac.uk/ FSL] is a comprehensive library of analysis tools for FMRI, MRI and DTI brain imaging data..

# [[Hoffman2:Accessing the Cluster#SSH - Command Line|SSH into Hoffman2]]
# Start an [https://www.hoffman2.idre.ucla.edu/computing/interactive-session/ interactive session] on a compute node with qrsh. Then at the compute node shell prompt, enter:
#: <pre>module load fsl</pre>
#To run FSL, enter:
#: <pre>fsl</pre>

CCN supports several versions of FSL on Hoffman2. To see the list of available FSL versions, enter:
module av fsl

To load a non-default version of FSL, enter (for example):
module load fsl/5.0.7

===FreeSurfer===

[https://surfer.nmr.mgh.harvard.edu/ FreeSurfer] is an open source software suite for processing and analyzing (human) brain MRI images.

# [[Hoffman2:Accessing the Cluster#SSH - Command Line|SSH into Hoffman2]]
# Start an [https://www.hoffman2.idre.ucla.edu/computing/interactive-session/ interactive session] on a compute node with qrsh. Then at the compute node shell prompt, enter:
#: <pre>module load freesurfer</pre>
#To run (for example) freeview, enter:
#: <pre>freeview</pre>

CCN supports several versions of FreeSurfer on Hoffman2. To see the list of available FreeSurfer versions, enter:
module av freesurfer

To load a non-default version of FreeSurfer, enter (for example):
module load freesurfer/5.1.0

===ITKGray===

[https://web.stanford.edu/group/vista/cgi-bin/wiki/index.php/ItkGray ITKGray] is a segmentation tool adapted and developed from ITKSnap by Bob Dougherty. ITKGray is a branch of the ITKSnap project. Bob added functionality from MrGray-Retired, including topology checking/fixing and a new flood-fill paintbrush that is similar to the MrGray-Retired auto-segmentation methods, but is run on small sections of an image rather than the entire image at once.

# [[Hoffman2:Accessing the Cluster#SSH - Command Line|SSH into Hoffman2]]
# Start an [https://www.hoffman2.idre.ucla.edu/computing/interactive-session/ interactive session] on a compute node with qrsh. Then at the compute node shell prompt, enter:
#: <pre>module load itkgray</pre>
#To run ITKGray, enter:
#: <pre>itkgray</pre>

CCN supports several versions of ITKGray on Hoffman2. To see the list of available ITKGray versions, enter:
module av itkgray

To load a non-default version of ITKGray, enter (for example):
module load itkgray/080128

===ITK-SNAP===

[http://www.itksnap.org/pmwiki/pmwiki.php ITK-SNAP] is a software application used to segment structures in 3D medical images.

# [[Hoffman2:Accessing the Cluster#SSH - Command Line|SSH into Hoffman2]]
# Start an [https://www.hoffman2.idre.ucla.edu/computing/interactive-session/ interactive session] on a compute node with qrsh. Then at the compute node shell prompt, enter:
#: <pre>module load itksnap</pre>
#To run ITK-SNAP, enter:
#: <pre>itksnap</pre>

CCN supports several versions of ITK-SNAP on Hoffman2. To see the list of available ITK-SNAP versions, enter:
module av itksnap

To load a non-default version of ITK-SNAP, enter (for example):
module load itksnap/3.4.0.QT4

===MRIcron===

[http://people.cas.sc.edu/rorden/mricron/MRIcron MRIcron] is a cross-platform NIfTI format image viewer. It can load multiple layers of images, generate volume renderings and draw volumes of interest. It also provides dcm2nii for converting DICOM images to NIfTI format and NPM for statistics.

# [[Hoffman2:Accessing the Cluster#SSH - Command Line|SSH into Hoffman2]]
# Start an [https://www.hoffman2.idre.ucla.edu/computing/interactive-session/ interactive session] on a compute node with qrsh. Then at the compute node shell prompt, enter:
#: <pre>module load mricron</pre>
#To run MRIcron, enter:
#: <pre>mricron</pre>

CCN supports several versions of MRIcron on Hoffman2. To see the list of available MRIcron versions, enter:
module av mricron

To load a non-default version of MRIcron, enter (for example):
module load mricron/20150622

===CCN Scripts===
Load the ''ccnscripts'' module to run CCN-supported scripts on Hoffman2: setup_subject, setup_subject_2, [[Hoffman2:Scripts:fix_perms.sh|fix_perms.sh]], etc.
# [[Hoffman2:Accessing the Cluster#SSH - Command Line|SSH into Hoffman2]]
# Start an [https://www.hoffman2.idre.ucla.edu/computing/interactive-session/ interactive session] on a compute node with qrsh. Then at the compute node shell prompt, enter:
#: <pre>module load ccnscripts</pre>

Hoffman2:Job Submission Templates

2017-12-04T23:28:40Z

Dmargolis: /* To work with these example scripts */

[[Hoffman2|Back to all things Hoffman2]]
===Job submission templates===
'''[[Hoffman2:Submit_job|Submit job]]'''
*example script for submitting a single job
'''[[Hoffman2:Submit_job_(tcsh)|Submit job (tcsh)]]'''
*same as above, written for tcsh
'''[[Hoffman2:Submit_jobarray|Submit jobarray]]'''
*example script for submitting a jobarray with hard-coded array values
'''[[Hoffman2:Submit_jobarray_(readarray)|Submit jobarray (readarray)]]'''
*example script for submitting a jobarray with an array read in from a file, e.g. 'subjects.txt'

===To work with these example scripts===
#Copy the contents of the script template into a new script, e.g. <code>myscript.sh</code>. ''Watch out for line ending errors caused by copying/pasting from a Mac or PC.'' Line ending issues can be fixed with <code>dos2unix myscript.sh</code>.
#Edit the "preamble" content at the top to adjust the memory (h_data) and run time (h_rt).
#*For jobs longer than 24 hours, you must specify the 'highp' option, e.g. <code>-l h_rt=36:00:00,h_data=4G,highp</code>
#*You can also adjust the number of cores: 2 cores is '-pe shared 2'. I recommend 2, 4, or 8 for this value. Be aware that the number of cores is a multiplier for the RAM. h_data=4G and 2 cores is 8G total.
#*Edit the mail notification options: '-m bea' means you want to receive a message when your job Begins, Ends, or Aborts (quits due to an error). You may use any combination of 'b', 'e', and 'a' for this setting.
#Put your script content at the bottom.
#Submit directly to the job scheduler like this: <code>qsub myscript.sh</code>

Hoffman2:Job Submission Templates

2017-12-04T23:27:40Z

Dmargolis: /* To work with these example scripts */

[[Hoffman2|Back to all things Hoffman2]]
===Job submission templates===
'''[[Hoffman2:Submit_job|Submit job]]'''
*example script for submitting a single job
'''[[Hoffman2:Submit_job_(tcsh)|Submit job (tcsh)]]'''
*same as above, written for tcsh
'''[[Hoffman2:Submit_jobarray|Submit jobarray]]'''
*example script for submitting a jobarray with hard-coded array values
'''[[Hoffman2:Submit_jobarray_(readarray)|Submit jobarray (readarray)]]'''
*example script for submitting a jobarray with an array read in from a file, e.g. 'subjects.txt'

===To work with these example scripts===
#Copy the contents of the script template into a new script, e.g. <code>myscript.sh</code>. ''Watch out for line ending errors caused by copying/pasting from a Mac or PC.'' Line ending issues can be fixed with <code>dos2unix myscript.sh</code>.
#Edit the "preamble" content at the top to adjust the memory (h_data) and run time (h_rt).
#*For jobs longer than 24 hours, you must specify the 'highp' option, e.g. <code>-l h_rt=8:00:00,h_data=4G,highp</code>
#*You can also adjust the number of cores: 2 cores is '-pe shared 2'. I recommend 2, 4, or 8 for this value. Be aware that the number of cores is a multiplier for the RAM. h_data=4G and 2 cores is 8G total.
#*Edit the mail notification options: '-m bea' means you want to receive a message when your job Begins, Ends, or Aborts (quits due to an error). You may use any combination of 'b', 'e', and 'a' for this setting.
#Put your script content at the bottom.
#Submit directly to the job scheduler like this: <code>qsub myscript.sh</code>

Hoffman2:Modules

2017-11-30T22:16:09Z

Dmargolis:

[[Hoffman2|Back to all things Hoffman2]]

[https://www.hoffman2.idre.ucla.edu/computing/modules/ Environmental Modules] allow users to dynamically modify their shell environment (e.g., $PATH, $LD_LIBRARY_PATH, etc.) in order to support a number of applications installed on the Hoffman2 Cluster. CCN supports several modules on Hoffman2 including: AFNI, ANTs, Caret,FSL, FreeSurfer, ITKGray, ITK-SNAP, and MRIcron.

==Enabling the CCN Hoffman2 modules==
# [[Hoffman2:Accessing the Cluster#SSH - Command Line|SSH into Hoffman2]]
# Use <code>gedit</code> or your favorite [[Text Editors|text editor]] (e.g. nano or vim) to edit the file <code>~/.bashrc</code>
#:* <pre>$ gedit ~/.bashrc</pre>
# Insert these lines at the '''bottom''' of the file
#:* <pre>module use /u/project/CCN/apps/modulefiles
umask 007</pre>
# '''Important:''' remove or comment out this line (if present):
#:* <pre><nowiki>#</nowiki> Remove or comment out this line!
<nowiki>#</nowiki>source /u/project/FMRI/apps/etc/profile</pre>
# Save the file
# Log out of Hoffman2 and the next time you log in, everything will be set for you to start working.

==CCN Hoffman2 modules==
===AFNI===
[https://afni.nimh.nih.gov/ AFNI] (Analysis of Functional NeuroImages) is a set of C programs for processing, analyzing, and displaying functional MRI (FMRI) data - a technique for mapping human brain activity.

# [[Hoffman2:Accessing the Cluster#SSH - Command Line|SSH into Hoffman2]]
# Start an [https://www.hoffman2.idre.ucla.edu/computing/interactive-session/ interactive session] on a compute node with qrsh. Then at the compute node shell prompt, enter:
#: <pre>module load afni</pre>
#To run AFNI, enter:
#: <pre>afni</pre>

===ANTs===

[http://stnava.github.io/ANTs/ ANTs] (Advanced Normalization Tools) computes high-dimensional mappings to capture the statistics of brain structure and function.

# [[Hoffman2:Accessing the Cluster#SSH - Command Line|SSH into Hoffman2]]
# Start an [https://www.hoffman2.idre.ucla.edu/computing/interactive-session/ interactive session] on a compute node with qrsh. Then at the compute node shell prompt, enter:
#: <pre>module load ants</pre>
# Please see the [http://stnava.github.io/ANTs/ ANTs website] for usage examples.

===Caret===

[http://brainvis.wustl.edu/wiki/index.php/Caret:About Caret] is a free, open-source, software package for structuraland functional analyses of the cerebral and cerebellar cortex.

# [[Hoffman2:Accessing the Cluster#SSH - Command Line|SSH into Hoffman2]]
# Start an [https://www.hoffman2.idre.ucla.edu/computing/interactive-session/ interactive session] on a compute node with qrsh. Then at the compute node shell prompt, enter:
#: <pre>module load caret</pre>
#To run Caret, enter:
#: <pre>caret5</pre>

===DCMTK===

[http://www.dcmtk.org/dcmtk.php.en DCMTK] is a collection of libraries and applications implementing large parts the DICOM standard. It includes software for examining, constructing and converting DICOM image files, handling offline media, sending and receiving images over a network connection, as well as demonstrative image storage and worklist servers.

# [[Hoffman2:Accessing the Cluster#SSH - Command Line|SSH into Hoffman2]]
# Start an [https://www.hoffman2.idre.ucla.edu/computing/interactive-session/ interactive session] on a compute node with qrsh. Then at the compute node shell prompt, enter:
#: <pre>module load dcmtk</pre>
#To run (for example) dcmdump, enter:
#: <pre>dcmdump [options] dcmfile(s)</pre>

===FSL===

[https://fsl.fmrib.ox.ac.uk/ FSL] is a comprehensive library of analysis tools for FMRI, MRI and DTI brain imaging data..

# [[Hoffman2:Accessing the Cluster#SSH - Command Line|SSH into Hoffman2]]
# Start an [https://www.hoffman2.idre.ucla.edu/computing/interactive-session/ interactive session] on a compute node with qrsh. Then at the compute node shell prompt, enter:
#: <pre>module load fsl</pre>
#To run FSL, enter:
#: <pre>fsl</pre>

CCN supports several versions of FSL on Hoffman2. To see the list of available FSL versions, enter:
module av fsl

To load a non-default version of FSL, enter (for example):
module load fsl/5.0.7

===FreeSurfer===

[https://surfer.nmr.mgh.harvard.edu/ FreeSurfer] is an open source software suite for processing and analyzing (human) brain MRI images.

# [[Hoffman2:Accessing the Cluster#SSH - Command Line|SSH into Hoffman2]]
# Start an [https://www.hoffman2.idre.ucla.edu/computing/interactive-session/ interactive session] on a compute node with qrsh. Then at the compute node shell prompt, enter:
#: <pre>module load freesurfer</pre>
#To run (for example) freeview, enter:
#: <pre>freeview</pre>

CCN supports several versions of FreeSurfer on Hoffman2. To see the list of available FreeSurfer versions, enter:
module av freesurfer

To load a non-default version of FreeSurfer, enter (for example):
module load freesurfer/5.1.0

===ITKGray===

[https://web.stanford.edu/group/vista/cgi-bin/wiki/index.php/ItkGray ITKGray] is a segmentation tool adapted and developed from ITKSnap by Bob Dougherty. ITKGray is a branch of the ITKSnap project. Bob added functionality from MrGray-Retired, including topology checking/fixing and a new flood-fill paintbrush that is similar to the MrGray-Retired auto-segmentation methods, but is run on small sections of an image rather than the entire image at once.

# [[Hoffman2:Accessing the Cluster#SSH - Command Line|SSH into Hoffman2]]
# Start an [https://www.hoffman2.idre.ucla.edu/computing/interactive-session/ interactive session] on a compute node with qrsh. Then at the compute node shell prompt, enter:
#: <pre>module load itkgray</pre>
#To run ITKGray, enter:
#: <pre>itkgray</pre>

CCN supports several versions of ITKGray on Hoffman2. To see the list of available ITKGray versions, enter:
module av itkgray

To load a non-default version of ITKGray, enter (for example):
module load itkgray/080128

===ITK-SNAP===

[http://www.itksnap.org/pmwiki/pmwiki.php ITK-SNAP] is a software application used to segment structures in 3D medical images.

# [[Hoffman2:Accessing the Cluster#SSH - Command Line|SSH into Hoffman2]]
# Start an [https://www.hoffman2.idre.ucla.edu/computing/interactive-session/ interactive session] on a compute node with qrsh. Then at the compute node shell prompt, enter:
#: <pre>module load itksnap</pre>
#To run ITK-SNAP, enter:
#: <pre>itksnap</pre>

CCN supports several versions of ITK-SNAP on Hoffman2. To see the list of available ITK-SNAP versions, enter:
module av itksnap

To load a non-default version of ITK-SNAP, enter (for example):
module load itksnap/3.4.0.QT4

===MRIcron===

[http://people.cas.sc.edu/rorden/mricron/MRIcron MRIcron] is a cross-platform NIfTI format image viewer. It can load multiple layers of images, generate volume renderings and draw volumes of interest. It also provides dcm2nii for converting DICOM images to NIfTI format and NPM for statistics.

# [[Hoffman2:Accessing the Cluster#SSH - Command Line|SSH into Hoffman2]]
# Start an [https://www.hoffman2.idre.ucla.edu/computing/interactive-session/ interactive session] on a compute node with qrsh. Then at the compute node shell prompt, enter:
#: <pre>module load mricron</pre>
#To run MRIcron, enter:
#: <pre>mricron</pre>

CCN supports several versions of MRIcron on Hoffman2. To see the list of available MRIcron versions, enter:
module av mricron

To load a non-default version of MRIcron, enter (for example):
module load mricron/20150622

===CCN Scripts===
Load the ''ccnscripts'' module to run CCN-supported scripts on Hoffman2: setup_subject, setup_subject_2, [[Hoffman2:Scripts:fix_perms.sh|fix_perms.sh]], etc.
# [[Hoffman2:Accessing the Cluster#SSH - Command Line|SSH into Hoffman2]]
# Start an [https://www.hoffman2.idre.ucla.edu/computing/interactive-session/ interactive session] on a compute node with qrsh. Then at the compute node shell prompt, enter:
#: <pre>module load ccnscripts</pre>

Hoffman2:Job Submission Templates

2017-11-29T19:36:42Z

Dmargolis:

[[Hoffman2|Back to all things Hoffman2]]
===Job submission templates===
'''[[Hoffman2:Submit_job|Submit job]]'''
*example script for submitting a single job
'''[[Hoffman2:Submit_job_(tcsh)|Submit job (tcsh)]]'''
*same as above, written for tcsh
'''[[Hoffman2:Submit_jobarray|Submit jobarray]]'''
*example script for submitting a jobarray with hard-coded array values
'''[[Hoffman2:Submit_jobarray_(readarray)|Submit jobarray (readarray)]]'''
*example script for submitting a jobarray with an array read in from a file, e.g. 'subjects.txt'

===To work with these example scripts===
#Copy the contents of the script template into a new script, e.g. <code>myscript.sh</code>. ''Watch out for line ending errors caused by copying/pasting from a Mac or PC.'' Line ending issues can be fixed with <code>dos2unix myscript.sh</code>.
#Edit the "preamble" content at the top to adjust the memory (h_data) and run time (h_rt).
#*You can also adjust the number of cores: 2 cores is '-pe shared 2'. I recommend 2, 4, or 8 for this value. Be aware that the number of cores is a multiplier for the RAM. h_data=4G and 2 cores is 8G total.
#*Edit the mail notification options: '-m bea' means you want to receive a message when your job Begins, Ends, or Aborts (quits due to an error). You may use any combination of 'b', 'e', and 'a' for this setting.
#Put your script content at the bottom.
#Submit directly to the job scheduler like this: <code>qsub myscript.sh</code>

Hoffman2:Software Tools

2017-11-29T19:36:25Z

Dmargolis:

[[Hoffman2|Back to all things Hoffman2]]

There is a CCN usergroup on Hoffman2 which is maintained for groups doing Neuroimaging work at UCLA. Tools like FSL, FreeSurfer, AFNI and Nibabel are maintained for this group separate from normal Hoffman2 programs. In order to take advantage of these tools, you need to setup your bash profile [[Hoffman2:Profile|properly]].

Below is a list of the available software tools. We will do our best to update it as changes are made.

==AFNI==
[http://afni.nimh.nih.gov/afni/ Official Webste]
{| class="wikitable"
! class="ccn-table-header" | Version
! class="ccn-table-header" | Install Date
! class="ccn-table-header" | Notes
|- class="ccn-table-odd"
| 2016.10.18-16.3.11 || 2016.11.20 || Current
|}

==BrainSuite==
[http://brainsuite.org/ Official Website]
{| class="wikitable"
! class="ccn-table-header"| Version Number
! class="ccn-table-header"| Install Date
! class="ccn-table-header"| Notes
|- class="ccn-table-odd"
| 15c || 2016.02.18 || Current
|}

==Caret==
[http://brainvis.wustl.edu/wiki/index.php/Caret:About Official Website]
{| class="wikitable"
! class="ccn-table-header" | Version
! class="ccn-table-header" | Install Date
! class="ccn-table-header" | Notes
|- class="ccn-table-odd"
| 5.65 (2012.01.27) || 2013.07.15 || Current, not folded into the main profile
|}

==Chronux==
[http://www.chronux.org Official Website]
{| class="wikitable"
! class="ccn-table-header" | Version
! class="ccn-table-header" | Install Date
! class="ccn-table-header" | Notes
|- class="ccn-table-odd"
| 2.10 || 2013.02.26 || Current
|}

==dcm2nii==
[http://www.mccauslandcenter.sc.edu/mricro/mricron/dcm2nii.html Official Website]
{| class="wikitable"
! class="ccn-table-header" | Version
! class="ccn-table-header" | Install Date
! class="ccn-table-header" | Notes
|- class="ccn-table-odd"
| 2013.06.06 || 2014.03.06 ||
|- class="ccn-table-even"
| 2011.11.11 || circa 2011 || Current
|}

==EEGLAB==
[http://sccn.ucsd.edu/eeglab/ Official Website]

[http://sccn.ucsd.edu/wiki/EEGLAB_revision_history Release Notes]
{| class="wikitable"
! class="ccn-table-header" | Version
! class="ccn-table-header" | Install Date
! class="ccn-table-header" | Notes
|- class="ccn-table-odd"
| 13.1.1b || 2014.01.29 || Current
|- class="ccn-table-even"
| 12.0.2.5b || 2013.11.14 ||
|- class="ccn-table-odd"
| 11.0.5.4b || 2013.11.14 ||
|- class="ccn-table-even"
| 12.0.0.0b || 2012.12.10 ||
|- class="ccn-table-odd"
| 11.0.0.0b || 2012.02.21 ||
|- class="ccn-table-even"
| 10.2.5.8b || 2012.02.21 ||
|}

==FreeSurfer==
[http://surfer.nmr.mgh.harvard.edu/ Official Website]

[http://freesurfer.net/fswiki/ReleaseNotes Release Notes]
{| class="wikitable"
! class="ccn-table-header" | Version
! class="ccn-table-header" | Install Date
! class="ccn-table-header" | Notes
|- class="ccn-table-odd"
| 5.3.0 || 2013.06.18 || Current
|- class="ccn-table-even"
| 5.2.0 || 2013.03.27 ||
|- class="ccn-table-odd"
| 5.1.0 || 2011.11.14 ||
|- class="ccn-table-even"
| 5.0.0 || circa 2010 ||
|- class="ccn-table-odd"
| 4.4.0 || circa 2009 ||
|- class="ccn-table-even"
| 4.0.5 || circa 2008 ||
|}

==FSL==
[http://fsl.fmrib.ox.ac.uk/fsl/fslwiki/ Official Website]

[http://fsl.fmrib.ox.ac.uk/fsl/fslwiki/WhatsNew Revision History]
{| class="wikitable"
! class="ccn-table-header"| Version Number
! class="ccn-table-header"| Install Date
! class="ccn-table-header"| Notes
|- class="ccn-table-even"
| 5.0.7 || 2013.10.17 || Current (2014.10.10-)
|- class="ccn-table-odd"
| 5.0.6 || 2013.12.18 || (2013.12.18-2014.10.10)
|- class="ccn-table-even"
| 5.0.5 || 2013.10.17 || (2013.10.17-2013.12.17)
|-class="ccn-table-odd"
| 5.0.4 || 2013.06.18 ||
|- class="ccn-table-even"
| 5.0.2 || 2013.02.19 ||
|- class="ccn-table-odd"
| 5.0.1 || 2012.10.01 ||
|- class="ccn-table-even"
| 5.0.0 || 2012.09.14 ||
|- class="ccn-table-odd"
| 4.1.9 || 2011.12.01 ||
|- class="ccn-table-even"
| 4.1.8 || circa 2011.06 ||
|- class="ccn-table-odd"
| 4.1.7 || circa 2011.11 ||
|- class="ccn-table-even"
| 4.1.4 || circa 2009 ||
|- class="ccn-table-odd"
| 4.1.3 || circa 2009 ||
|- class="ccn-table-even"
| 4.1.1 || circa 2008 ||
|- class="ccn-table-odd"
| 4.1.0 || circa 2008 ||
|- class="ccn-table-even"
| 4.0.4 || circa 2008 ||
|}

==ITKGray==
[http://vistalab.stanford.edu/newlm/index.php/ItkGray Official Website]
{| class="wikitable"
! class="ccn-table-header"| Version Number
! class="ccn-table-header"| Install Date
! class="ccn-table-header"| Notes
|- class="ccn-table-odd"
| 080803 || 2009.11.19 || Current
|- class="ccn-table-even"
| 080128 || 2009.11.13 ||
|}

==SPM==
[http://www.fil.ion.ucl.ac.uk/spm/ Official Website]
{| class="wikitable"
! class="ccn-table-header"| Version
! class="ccn-table-header"| Last Patch Applied
! class="ccn-table-header"| Last Checked Date
! class="ccn-table-header"| Notes
|- class="ccn-table-odd"
| SPM8 || 5236 || 2014.01 || Current
|- class="ccn-table-even"
| SPM5 || Unknown || N/A || No longer supported
|}

==TrackVis/Diffusion Toolkit==
[http://trackvis.org/ Official Website 1]

[http://trackvis.org/dtk/ Official Website 2]
{| class="wikitable"
! class="ccn-table-header"| Tool
! class="ccn-table-header"| Version Number
! class="ccn-table-header"| Last Checked Date
! class="ccn-table-header"| Notes
|- class="ccn-table-odd"
| TrackVis || 0.5.2.2 || 2014.03.06 ||
|- class="ccn-table-even"
| Diffusion Toolkit || 0.6.2.2 || 204.03.06 ||
|}

==WEKA==
[http://www.cs.waikato.ac.nz/ml/weka/ Official Website]
{| class="wikitable"
! class="ccn-table-header"| Version Number
! class="ccn-table-header"| Install Date
! class="ccn-table-header"| Notes
|- class="ccn-table-odd"
| 3.7.10 || 2014.03.03 || Current
|- class="ccn-table-even"
| 3.6.5 || circa 2011.08 ||
|}

Hoffman2:Software Tools

2017-11-29T19:36:13Z

Dmargolis:

[[Hoffman2|Back to all things Hoffman2]]

There is an CCN usergroup on Hoffman2 which is maintained for groups doing Neuroimaging work at UCLA. Tools like FSL, FreeSurfer, AFNI and Nibabel are maintained for this group separate from normal Hoffman2 programs. In order to take advantage of these tools, you need to setup your bash profile [[Hoffman2:Profile|properly]].

Below is a list of the available software tools. We will do our best to update it as changes are made.

==AFNI==
[http://afni.nimh.nih.gov/afni/ Official Webste]
{| class="wikitable"
! class="ccn-table-header" | Version
! class="ccn-table-header" | Install Date
! class="ccn-table-header" | Notes
|- class="ccn-table-odd"
| 2016.10.18-16.3.11 || 2016.11.20 || Current
|}

==BrainSuite==
[http://brainsuite.org/ Official Website]
{| class="wikitable"
! class="ccn-table-header"| Version Number
! class="ccn-table-header"| Install Date
! class="ccn-table-header"| Notes
|- class="ccn-table-odd"
| 15c || 2016.02.18 || Current
|}

==Caret==
[http://brainvis.wustl.edu/wiki/index.php/Caret:About Official Website]
{| class="wikitable"
! class="ccn-table-header" | Version
! class="ccn-table-header" | Install Date
! class="ccn-table-header" | Notes
|- class="ccn-table-odd"
| 5.65 (2012.01.27) || 2013.07.15 || Current, not folded into the main profile
|}

==Chronux==
[http://www.chronux.org Official Website]
{| class="wikitable"
! class="ccn-table-header" | Version
! class="ccn-table-header" | Install Date
! class="ccn-table-header" | Notes
|- class="ccn-table-odd"
| 2.10 || 2013.02.26 || Current
|}

==dcm2nii==
[http://www.mccauslandcenter.sc.edu/mricro/mricron/dcm2nii.html Official Website]
{| class="wikitable"
! class="ccn-table-header" | Version
! class="ccn-table-header" | Install Date
! class="ccn-table-header" | Notes
|- class="ccn-table-odd"
| 2013.06.06 || 2014.03.06 ||
|- class="ccn-table-even"
| 2011.11.11 || circa 2011 || Current
|}

==EEGLAB==
[http://sccn.ucsd.edu/eeglab/ Official Website]

[http://sccn.ucsd.edu/wiki/EEGLAB_revision_history Release Notes]
{| class="wikitable"
! class="ccn-table-header" | Version
! class="ccn-table-header" | Install Date
! class="ccn-table-header" | Notes
|- class="ccn-table-odd"
| 13.1.1b || 2014.01.29 || Current
|- class="ccn-table-even"
| 12.0.2.5b || 2013.11.14 ||
|- class="ccn-table-odd"
| 11.0.5.4b || 2013.11.14 ||
|- class="ccn-table-even"
| 12.0.0.0b || 2012.12.10 ||
|- class="ccn-table-odd"
| 11.0.0.0b || 2012.02.21 ||
|- class="ccn-table-even"
| 10.2.5.8b || 2012.02.21 ||
|}

==FreeSurfer==
[http://surfer.nmr.mgh.harvard.edu/ Official Website]

[http://freesurfer.net/fswiki/ReleaseNotes Release Notes]
{| class="wikitable"
! class="ccn-table-header" | Version
! class="ccn-table-header" | Install Date
! class="ccn-table-header" | Notes
|- class="ccn-table-odd"
| 5.3.0 || 2013.06.18 || Current
|- class="ccn-table-even"
| 5.2.0 || 2013.03.27 ||
|- class="ccn-table-odd"
| 5.1.0 || 2011.11.14 ||
|- class="ccn-table-even"
| 5.0.0 || circa 2010 ||
|- class="ccn-table-odd"
| 4.4.0 || circa 2009 ||
|- class="ccn-table-even"
| 4.0.5 || circa 2008 ||
|}

==FSL==
[http://fsl.fmrib.ox.ac.uk/fsl/fslwiki/ Official Website]

[http://fsl.fmrib.ox.ac.uk/fsl/fslwiki/WhatsNew Revision History]
{| class="wikitable"
! class="ccn-table-header"| Version Number
! class="ccn-table-header"| Install Date
! class="ccn-table-header"| Notes
|- class="ccn-table-even"
| 5.0.7 || 2013.10.17 || Current (2014.10.10-)
|- class="ccn-table-odd"
| 5.0.6 || 2013.12.18 || (2013.12.18-2014.10.10)
|- class="ccn-table-even"
| 5.0.5 || 2013.10.17 || (2013.10.17-2013.12.17)
|-class="ccn-table-odd"
| 5.0.4 || 2013.06.18 ||
|- class="ccn-table-even"
| 5.0.2 || 2013.02.19 ||
|- class="ccn-table-odd"
| 5.0.1 || 2012.10.01 ||
|- class="ccn-table-even"
| 5.0.0 || 2012.09.14 ||
|- class="ccn-table-odd"
| 4.1.9 || 2011.12.01 ||
|- class="ccn-table-even"
| 4.1.8 || circa 2011.06 ||
|- class="ccn-table-odd"
| 4.1.7 || circa 2011.11 ||
|- class="ccn-table-even"
| 4.1.4 || circa 2009 ||
|- class="ccn-table-odd"
| 4.1.3 || circa 2009 ||
|- class="ccn-table-even"
| 4.1.1 || circa 2008 ||
|- class="ccn-table-odd"
| 4.1.0 || circa 2008 ||
|- class="ccn-table-even"
| 4.0.4 || circa 2008 ||
|}

==ITKGray==
[http://vistalab.stanford.edu/newlm/index.php/ItkGray Official Website]
{| class="wikitable"
! class="ccn-table-header"| Version Number
! class="ccn-table-header"| Install Date
! class="ccn-table-header"| Notes
|- class="ccn-table-odd"
| 080803 || 2009.11.19 || Current
|- class="ccn-table-even"
| 080128 || 2009.11.13 ||
|}

==SPM==
[http://www.fil.ion.ucl.ac.uk/spm/ Official Website]
{| class="wikitable"
! class="ccn-table-header"| Version
! class="ccn-table-header"| Last Patch Applied
! class="ccn-table-header"| Last Checked Date
! class="ccn-table-header"| Notes
|- class="ccn-table-odd"
| SPM8 || 5236 || 2014.01 || Current
|- class="ccn-table-even"
| SPM5 || Unknown || N/A || No longer supported
|}

==TrackVis/Diffusion Toolkit==
[http://trackvis.org/ Official Website 1]

[http://trackvis.org/dtk/ Official Website 2]
{| class="wikitable"
! class="ccn-table-header"| Tool
! class="ccn-table-header"| Version Number
! class="ccn-table-header"| Last Checked Date
! class="ccn-table-header"| Notes
|- class="ccn-table-odd"
| TrackVis || 0.5.2.2 || 2014.03.06 ||
|- class="ccn-table-even"
| Diffusion Toolkit || 0.6.2.2 || 204.03.06 ||
|}

==WEKA==
[http://www.cs.waikato.ac.nz/ml/weka/ Official Website]
{| class="wikitable"
! class="ccn-table-header"| Version Number
! class="ccn-table-header"| Install Date
! class="ccn-table-header"| Notes
|- class="ccn-table-odd"
| 3.7.10 || 2014.03.03 || Current
|- class="ccn-table-even"
| 3.6.5 || circa 2011.08 ||
|}

Hoffman2:Submit jobarray (readarray)

2017-11-29T19:33:35Z

Dmargolis:

[[Hoffman2|Back to all things Hoffman2]]

In this example, subjects are read in from a subjects.txt file. subjects.txt is a text file with a single subject ID on each line. ''Watch out for line ending errors caused by copying/pasting from a Mac or PC.'' Line ending issues can be fixed with <code>dos2unix subjects.txt</code>.

<nowiki>#!/bin/bash
#$ -cwd
# error = Merged with joblog
#$ -o joblog.$JOB_ID.$TASK_ID
#$ -j y
#$ -pe shared 2
#$ -l h_rt=8:00:00,h_data=4G
# Email address to notify
#$ -M $USER@mail
# Notify when
#$ -m a
# Job array indexes
#$ -t 1-16:1

# Load the job environment:
. /u/local/Modules/default/init/modules.sh
module use /u/project/CCN/apps/modulefiles

# Load the FSL module
module load fsl

# This is optional
# More info here: https://www.ccn.ucla.edu/wiki/index.php/Hoffman2:FSL
export NO_FSL_JOBS=true

# Set up the subjects list
readarray -t subjects < subjects.txt
(( i=$SGE_TASK_ID - 1 ))

echo "This is sub-job $SGE_TASK_ID"
echo "This is subject ${subjects[$i]}"

# Your script content goes here...</nowiki>

Hoffman2:Submitting Jobs

2017-11-29T19:28:46Z

Dmargolis:

[[Hoffman2|Back to all things Hoffman2]]

If you remember from [[Hoffman2:Introduction#Sun Grid Engine|Anatomy of the Computing Cluster]], the Univa Grid Engine on Hoffman2 is the scheduler for all computing jobs. It takes your computing job request, considers what resources you are asking for and then puts your job in a line waiting for those resources to become available.

Ask for a simple 1GB of memory and a single computing core with a short time window, and your job will likely get placed at the front of the line and start running soon if not immediately.

Ask for a lot of memory or many computing cores, and your job will get put further back in the line because it will have to wait for more things to become available. Ask for too little RAM or too little time and your job will be killed or end prematurely leaving you with no results to examine.

= Job Submission Templates =
Please use these [[Hoffman2:Job_Submission_Templates|job submission template scripts]] to simplify job submission.

=Job Submission Types=

So how does one submit a computing job request? You've got some options:
# '''job.q'''
#: Use a simple tool that IDRE wrote. It has a menu and walks you through submitting things.
# '''qsub'''
#: Get under the hood and do it yourself. It can get messy but it can also be faster and you have more flexibility with options.
# '''command files'''
#: You've graduated to a higher level of operations, but we can help you get there with examples of our own command files.
# '''job arrays'''
#: You've got a lot of repetitive tasks to run, these will be your friend.



==job.q==
Once you've identified or written a script you'd like to run, [[Hoffman2:Accessing the Cluster#SSH - Command Line|SSH into Hoffman2]] and enter <code>job.q</code>. Then it is just a matter of following its step-by-step instructions.

From the tool's main menu, you can type ''Info'' to read up about how to use it and we highly encourage you to do so.

But we know patience is a virtue that most of us aren't blessed with. So we'll walk you through submitting a basic job so you can hit the ground running.

===Example===
# Once on Hoffman2, you'll need to edit one file so pull out your favorite [[Text Editors|text editor]] and edit the file
#: <pre>~/.queuerc</pre>
# Add the line
#: <pre>set qqodir = ~/job-output</pre>
# You've just set the default directory where your job command files will be created. Save the configuration file and close your text editor.
# Make that directory using the command
#: <pre>$ mkdir ~/job-output</pre>
# Now execute
#:<pre>$ job.q</pre>
# Press enter to acknowledge the message about some files that get created (READ IT FIRST THOUGH).
# Type ''Build <ENTER>'' to begin creating an SGE command file.
# The program now asks you which script you'd like to run, enter the following text to use our example script
#: <pre>/u/project/CCN/apps/examples/qsub/gather.sh</pre>
# The program now asks how much memory the job will need (in [http://en.wikipedia.org/wiki/Megabyte Megabytes]). This script is really simple, so let's go with the minimum and enter ''64''.
# The program now asks how long will the job take (in hours). Go with the minimum 1 hour; it will complete in much less than this.
# The program now asks if your job should be limited to only your resource group's cores. Answer ''n'' because you do not need to be limiting yourself here and the job is not going to be running for more than 24 hours.
# Soon, the program will tell you that ''gather.sh.cmd'' has been built and saved.
# When it asks you if you would like to submit your job, say no. Then type ''Quit <ENTER>'' to leave the program.
# Now you should be able to run
#: <pre>ls ~/job-output</pre>
#: and see ''gather.sh.cmd''. This file will stay there until you delete it and can be run over and over again. Making a command file like this is especially useful if there is a task you'll be running repeatedly on Hoffman2. But if this is something you only need to run once, you should delete the file so you don't needlessly approach your [[Hoffman2:Quotas|quota]].
# The time has come to actually run the program (thought we'd never get to that, didn't you?). Type
#: <pre>qsub job-output/gather.sh.cmd</pre>
#: and after hitting enter, a message similar to this will pop up:
#: <pre>Your job 1882940 ("gather.sh.cmd") has been submitted</pre>
#: where the number is your JobID, a unique numerical identifier for the computer job you have submitted to the queue.
# Now you can check if the job has finished running by doing
#: <pre>ls ~/job-output</pre>
# When two files named ''gather.sh.output.[JOBID]'' and ''gather.sh.joblog.[JOBID]'' (where JOBID is your job's unique identifier) appear, your job has run.
#: ''gather.sh.output.[JOBID]''
#:: This file has all the standard output generated by your script. In this case it will just have the line
#::: ''Standard output would appear here.''
#: ''gather.sh.joblog.[JOBID]''
#:: This file has all the details about when, where, and how your job was processed. Useful information if you are going to be running this job over and over and need to fine tune the resources it uses.
# Better ways of checking on your job can be found [[Hoffman2:Monitoring Jobs|here]].
# The script you ran is an aggregator. It looks in a list of directories, each assumed to contain a specifically named file, and gathers the contents of each of those files into one central file in your home directory. This file is named ''gather-[TIMESTAMP].txt'' where TIMESTAMP is when the script was run and follows [http://en.wikipedia.org/wiki/ISO_8601 ISO 8601] style encoding. You are encouraged to type
#: <pre>/u/project/CCN/apps/examples/qsub/gather.sh -h</pre>
#: or
#: <pre>/u/project/CCN/apps/examples/qsub/gather.sh --help</pre>
#: to see how this script works.
# Finally, go check the inbox of the email you used to sign up for your Hoffman2 account. There will be two emails from "root@mail.hoffman2.idre.ucla.edu" that indicate when the job was started and when the job was completed. This is one of the neat features of the queue so that you can be alerted about the progress of your job without having to stay logged into Hoffman2 and checking on it constantly.

==qsub==
Everything that job.q did can be done on the command line. And it can be done better.

===Example===
Run the command:
$ qsub -cwd -V -N J1 -l h_data=64M,h_rt=00:05:00 -M $USER -m bea /u/project/CCN/apps/examples/qsub/gather.sh

And something like the following will be printed out:
Your job 1875395 ("J1") has been submitted

Where the number is your JOBID, a unique numerical identifier for your job.

Let's break down the arguments in that command.

;<code>-cwd</code>
: Change working directory
: When your script runs, change the working directory to where you currently are in the filesystem.
:: e.g. If you were in the director /u/home/mscohen/data/ when you ran the command, the queue will change directories to that location and then execute the script you gave it. This means output and error directories will be placed here for that job.

;<code>-V</code>
: Export environment variables
: Exports all the environment variables to the context of the job. Useful if you have extra environment variables that are needed in your script.
:: e.g. If you had defined the variable SUBJECT_ID in your session on Hoffman2 (<code>export SUBJECT_ID=42</code>) before submitting a job and that variable was called on by your script, then you would need to use this flag. Tools like FreeSurfer look for certain environment variables to be set.

;<code>-N J1</code>
: Name my job
: Names your job "J1." When you [[Hoffman2:Monitoring Jobs#qstat|look at the queue]], this will be the text that shows up in the "name" column. This will also be the beginning of the output (<code>J1.o[JOBID]</code>) and error (<code>J1.e[JOBID]</code>) files for your job.

;<code>-l h_data=64M,time=00:05:00</code>
: Resource allocation (that's a lower case "elle")
: This is the resources flag meaning that the text immediately after it will ask for things like:
:* certain amount of memory, in [http://en.wikipedia.org/wiki/Megabyte Megabytes], or [http://en.wikipedia.org/wiki/Gigabyte Gigabytes]
:** h_data=64M (64 MB RAM) or h_data=1G (1 GB RAM)

: If your job uses more RAM than it requested, your job WILL be killed in order to avoid it hurting other jobs running on the same node. It is imperative that you set this RAM request properly.
:* certain length of computing time, in the form HH:MM:SS
:** h_rt=00:05:00
: In this case the script will complete its task rapidly, hence we are only asking for 5 minutes of computing time.

:* [http://www.ats.ucla.edu/clusters/hoffman2/computing/batch/policies.htm#highp highp]
:**: Job length maximum of 14 days but can only be run on nodes belonging to your resource group (type <code>mygroup</code> to see what type of resources you have available). If you are in the ccn group on Hoffman2, you have access to some of these highp nodes.



;<code>-M $USER</code>
: Define mailing list
: This defines the user that will be mailed if email updates are requested. The default address is that of the job-owner.
:: e.g. In this case, the email will be sent to the address on file for the user.

;<code>-m bea</code>
: Define mailing rules
: This defines when Hoffman2 should email you about your job. There are five options here
:* b - when the job begins
:* e - when the job ends
:* a - when the job is aborted (ends in an error state)


==Command Files==
Typing accurately can be difficult at times, so why put yourself through the trouble of having to retype the same arguments over and over if you will always be using about the same values? Enter command files.

You already have experience making a command file (~/job-output/gather.sh.cmd) from when you used the tool <code>job.q</code>. But did you know that you can edit that command file to make changes to how it runs, or write your own?

The command files generated by <code>job.q</code> are fairly well commented, so if you take a look at them with your favorite [Text Editors|text editor] you should be able to change their behavior. For instance, if you go into the command file from the job.q example, find the lines that say
# Notify at beginning and end of job
#$ -m bea
You recognize that this is the flag about when to send email messages. Go ahead and change this to
# Notify at the end and on abort
#$ -m ae
And you should only receive one email when your job finishes.


==Job Arrays==
There is an SGE qsub argument that allows you to submit multiple jobs in parallel that use the same script. It is
-t lower-upper:interval
where
;<code>lower</code>
: is replaced with the starting number
;<code>upper</code>
: is replaced with the ending number
;<code>interval</code>
: is replaced with the step interval
So adding the argument
-t 10-100:5
will step through the numbers 10, 15, 20, 25, ..., 100 submitting a job for each one.

In jobs that are called with this flag, there will be an [[Hoffman2:UNIX Tutorial#Environment Variables|environment variable]] called <code>SGE_TASK_ID</code> whose value will be incremented over the range you specified. Each possible value of SGE_TASK_ID will be submitted as its own job, so your work will be parallelized.

===Example===

Let's say you have a script, '''myFunc.sh''', that takes one numerical input and computes a bunch of values based on that input. But you need to run <code>myFunc.sh</code> for input values 1 to 100. One solution would be to write a wrapper script '''myFuncSlowWrapper.sh''' as
#!/bin/bash
# myFuncSlowWrapper.sh
for i in {1..100};
do
myFunc.sh $i;
done

The only drawback is that this will take quite a while since all 100 iterations will be done on a single processor. With job arrays, the computations will be split among many processors and can finish much more quickly. You would instead write a wrapper script called '''myFuncFastWrapper.sh''' as
#!/bin/bash
# myFuncFastWrapper.sh
echo $SGE_TASK_ID
myFunc.sh $SGE_TASK_ID

And submit it with
qsub -cwd -V -N PJ -l h_data=1024M,h_rt=01:00:00 -M $HOME -m bea -t 1-100:1 myFuncWrapper.sh

Hoffman2:Submit jobarray (readarray)

2017-11-29T19:27:38Z

Dmargolis:

[[Hoffman2|Back to all things Hoffman2]]

In this example, subjects are read in from a subjects.txt file. subjects.txt is a text file with a single subject ID on each line. ''Watch out for line ending errors caused by copying/pasting from a Mac or PC.'' Line ending issues can be fixed with <code>dos2unix myscript.sh</code>.

<nowiki>#!/bin/bash
#$ -cwd
# error = Merged with joblog
#$ -o joblog.$JOB_ID.$TASK_ID
#$ -j y
#$ -pe shared 2
#$ -l h_rt=8:00:00,h_data=4G
# Email address to notify
#$ -M $USER@mail
# Notify when
#$ -m a
# Job array indexes
#$ -t 1-16:1

# Load the job environment:
. /u/local/Modules/default/init/modules.sh
module use /u/project/CCN/apps/modulefiles

# Load the FSL module
module load fsl

# This is optional
# More info here: https://www.ccn.ucla.edu/wiki/index.php/Hoffman2:FSL
export NO_FSL_JOBS=true

# Set up the subjects list
readarray -t subjects < subjects.txt
(( i=$SGE_TASK_ID - 1 ))

echo "This is sub-job $SGE_TASK_ID"
echo "This is subject ${subjects[$i]}"

# Your script content goes here...</nowiki>

Hoffman2:Submit jobarray (readarray)

2017-11-29T19:27:26Z

Dmargolis:

[[Hoffman2|Back to all things Hoffman2]]

In this example, subjects are read in from a subjects.txt file. subjects.txt is a text file with a single subject ID on each line. '''Watch out for line ending errors caused by copying/pasting from a Mac or PC.'' Line ending issues can be fixed with <code>dos2unix myscript.sh</code>.

<nowiki>#!/bin/bash
#$ -cwd
# error = Merged with joblog
#$ -o joblog.$JOB_ID.$TASK_ID
#$ -j y
#$ -pe shared 2
#$ -l h_rt=8:00:00,h_data=4G
# Email address to notify
#$ -M $USER@mail
# Notify when
#$ -m a
# Job array indexes
#$ -t 1-16:1

# Load the job environment:
. /u/local/Modules/default/init/modules.sh
module use /u/project/CCN/apps/modulefiles

# Load the FSL module
module load fsl

# This is optional
# More info here: https://www.ccn.ucla.edu/wiki/index.php/Hoffman2:FSL
export NO_FSL_JOBS=true

# Set up the subjects list
readarray -t subjects < subjects.txt
(( i=$SGE_TASK_ID - 1 ))

echo "This is sub-job $SGE_TASK_ID"
echo "This is subject ${subjects[$i]}"

# Your script content goes here...</nowiki>

Hoffman2:Job Submission Templates

2017-11-29T19:26:49Z

Dmargolis:

===Job submission templates===
'''[[Hoffman2:Submit_job|Submit job]]'''
*example script for submitting a single job
'''[[Hoffman2:Submit_job_(tcsh)|Submit job (tcsh)]]'''
*same as above, written for tcsh
'''[[Hoffman2:Submit_jobarray|Submit jobarray]]'''
*example script for submitting a jobarray with hard-coded array values
'''[[Hoffman2:Submit_jobarray_(readarray)|Submit jobarray (readarray)]]'''
*example script for submitting a jobarray with an array read in from a file, e.g. 'subjects.txt'

===To work with these example scripts===
#Copy the contents of the script template into a new script, e.g. <code>myscript.sh</code>. ''Watch out for line ending errors caused by copying/pasting from a Mac or PC.'' Line ending issues can be fixed with <code>dos2unix myscript.sh</code>.
#Edit the "preamble" content at the top to adjust the memory (h_data) and run time (h_rt).
#*You can also adjust the number of cores: 2 cores is '-pe shared 2'. I recommend 2, 4, or 8 for this value. Be aware that the number of cores is a multiplier for the RAM. h_data=4G and 2 cores is 8G total.
#*Edit the mail notification options: '-m bea' means you want to receive a message when your job Begins, Ends, or Aborts (quits due to an error). You may use any combination of 'b', 'e', and 'a' for this setting.
#Put your script content at the bottom.
#Submit directly to the job scheduler like this: <code>qsub myscript.sh</code>

Hoffman2:Submit jobarray (readarray)

2017-11-29T19:25:32Z

Dmargolis:

[[Hoffman2|Back to all things Hoffman2]]

In this example, subjects are read in from a subjects.txt file. subjects.txt is a text file with a single subject ID on each line. '''Watch out for non-Linux (Mac or PC) line endings.''' Line ending issues can be fixed with <code>dos2unix myscript.sh</code>.

<nowiki>#!/bin/bash
#$ -cwd
# error = Merged with joblog
#$ -o joblog.$JOB_ID.$TASK_ID
#$ -j y
#$ -pe shared 2
#$ -l h_rt=8:00:00,h_data=4G
# Email address to notify
#$ -M $USER@mail
# Notify when
#$ -m a
# Job array indexes
#$ -t 1-16:1

# Load the job environment:
. /u/local/Modules/default/init/modules.sh
module use /u/project/CCN/apps/modulefiles

# Load the FSL module
module load fsl

# This is optional
# More info here: https://www.ccn.ucla.edu/wiki/index.php/Hoffman2:FSL
export NO_FSL_JOBS=true

# Set up the subjects list
readarray -t subjects < subjects.txt
(( i=$SGE_TASK_ID - 1 ))

echo "This is sub-job $SGE_TASK_ID"
echo "This is subject ${subjects[$i]}"

# Your script content goes here...</nowiki>

Hoffman2:Job Submission Templates

2017-11-29T19:25:23Z

Dmargolis: /* To work with these example scripts */

===Job submission templates===
'''[[Hoffman2:Submit_job|Submit job]]'''
*example script for submitting a single job
'''[[Hoffman2:Submit_job_(tcsh)|Submit job (tcsh)]]'''
*same as above, written for tcsh
'''[[Hoffman2:Submit_jobarray|Submit jobarray]]'''
*example script for submitting a jobarray with hard-coded array values
'''[[Hoffman2:Submit_jobarray_(readarray)|Submit jobarray (readarray)]]'''
*example script for submitting a jobarray with an array read in from a file, e.g. 'subjects.txt'

===To work with these example scripts===
#Copy the contents of the script template into a new script, e.g. <code>myscript.sh</code>. '''Watch out for non-Linux (Mac or PC) line endings.''' Line ending issues can be fixed with <code>dos2unix myscript.sh</code>.
#Edit the "preamble" content at the top to adjust the memory (h_data) and run time (h_rt).
#*You can also adjust the number of cores: 2 cores is '-pe shared 2'. I recommend 2, 4, or 8 for this value. Be aware that the number of cores is a multiplier for the RAM. h_data=4G and 2 cores is 8G total.
#*Edit the mail notification options: '-m bea' means you want to receive a message when your job Begins, Ends, or Aborts (quits due to an error). You may use any combination of 'b', 'e', and 'a' for this setting.
#Put your script content at the bottom.
#Submit directly to the job scheduler like this: <code>qsub myscript.sh</code>

Hoffman2:Job Submission Templates

2017-11-29T19:25:01Z

Dmargolis: /* To work with these example scripts */

===Job submission templates===
'''[[Hoffman2:Submit_job|Submit job]]'''
*example script for submitting a single job
'''[[Hoffman2:Submit_job_(tcsh)|Submit job (tcsh)]]'''
*same as above, written for tcsh
'''[[Hoffman2:Submit_jobarray|Submit jobarray]]'''
*example script for submitting a jobarray with hard-coded array values
'''[[Hoffman2:Submit_jobarray_(readarray)|Submit jobarray (readarray)]]'''
*example script for submitting a jobarray with an array read in from a file, e.g. 'subjects.txt'

===To work with these example scripts===
#Copy the contents of the script template into a new script, e.g. <code>myscript.sh</code>. '''Watch out for non-Linux (Mac or PC) line endings.''' Line ending issues can be fixed with <code>dos2unix</code>.
#Edit the "preamble" content at the top to adjust the memory (h_data) and run time (h_rt).
#*You can also adjust the number of cores: 2 cores is '-pe shared 2'. I recommend 2, 4, or 8 for this value. Be aware that the number of cores is a multiplier for the RAM. h_data=4G and 2 cores is 8G total.
#*Edit the mail notification options: '-m bea' means you want to receive a message when your job Begins, Ends, or Aborts (quits due to an error). You may use any combination of 'b', 'e', and 'a' for this setting.
#Put your script content at the bottom.
#Submit directly to the job scheduler like this: <code>qsub myscript.sh</code>

Hoffman2:Job Submission Templates

2017-11-29T19:23:34Z

Dmargolis: /* To work with these example scripts */

===Job submission templates===
'''[[Hoffman2:Submit_job|Submit job]]'''
*example script for submitting a single job
'''[[Hoffman2:Submit_job_(tcsh)|Submit job (tcsh)]]'''
*same as above, written for tcsh
'''[[Hoffman2:Submit_jobarray|Submit jobarray]]'''
*example script for submitting a jobarray with hard-coded array values
'''[[Hoffman2:Submit_jobarray_(readarray)|Submit jobarray (readarray)]]'''
*example script for submitting a jobarray with an array read in from a file, e.g. 'subjects.txt'

===To work with these example scripts===
#Edit the "preamble" content at the top to adjust the memory (h_data) and run time (h_rt).
#*You can also adjust the number of cores: 2 cores is '-pe shared 2'. I recommend 2, 4, or 8 for this value. Be aware that the number of cores is a multiplier for the RAM. h_data=4G and 2 cores is 8G total.
#*Edit the mail notification options: '-m bea' means you want to receive a message when your job Begins, Ends, or Aborts (quits due to an error). You may use any combination of 'b', 'e', and 'a' for this setting.
#Put your script content at the bottom.
#Submit directly to the job scheduler like this: <code>qsub myscript.sh</code>

Hoffman2:Submit job

2017-11-29T19:21:06Z

Dmargolis:

[[Hoffman2|Back to all things Hoffman2]]
<nowiki>#!/bin/bash
#$ -cwd
# error = Merged with joblog
#$ -o joblog.$JOB_ID
#$ -j y
#$ -pe shared 2
#$ -l h_rt=8:00:00,h_data=4G
# Email address to notify
#$ -M $USER@mail
# Notify when
#$ -m bea

# load the job environment:
. /u/local/Modules/default/init/modules.sh
module use /u/project/CCN/apps/modulefiles

# Load the FSL module
module load fsl

# This is optional
# More info here: https://www.ccn.ucla.edu/wiki/index.php/Hoffman2:FSL
export NO_FSL_JOBS=true

# Your script content goes here...</nowiki>

Hoffman2:Submit jobarray (readarray)

2017-11-29T19:20:49Z

Dmargolis:

[[Hoffman2|Back to all things Hoffman2]]

In this example, subjects are read in from a subjects.txt file. subjects.txt is a text file with a single subject ID on each line. '''Watch out for non-Linux (Mac or PC) line endings.''' Line ending issues can be fixed with <code>dos2unix</code>.

<nowiki>#!/bin/bash
#$ -cwd
# error = Merged with joblog
#$ -o joblog.$JOB_ID.$TASK_ID
#$ -j y
#$ -pe shared 2
#$ -l h_rt=8:00:00,h_data=4G
# Email address to notify
#$ -M $USER@mail
# Notify when
#$ -m a
# Job array indexes
#$ -t 1-16:1

# Load the job environment:
. /u/local/Modules/default/init/modules.sh
module use /u/project/CCN/apps/modulefiles

# Load the FSL module
module load fsl

# This is optional
# More info here: https://www.ccn.ucla.edu/wiki/index.php/Hoffman2:FSL
export NO_FSL_JOBS=true

# Set up the subjects list
readarray -t subjects < subjects.txt
(( i=$SGE_TASK_ID - 1 ))

echo "This is sub-job $SGE_TASK_ID"
echo "This is subject ${subjects[$i]}"

# Your script content goes here...</nowiki>

Hoffman2:Submit jobarray

2017-11-29T19:20:33Z

Dmargolis:

[[Hoffman2|Back to all things Hoffman2]]
<nowiki>#!/bin/bash
#$ -cwd
# error = Merged with joblog
#$ -o joblog.$JOB_ID.$TASK_ID
#$ -j y
#$ -pe shared 2
#$ -l h_rt=8:00:00,h_data=4G
# Email address to notify
#$ -M $USER@mail
# Notify when
#$ -m a
# Job array indexes
#$ -t 1-5:1

# Load the job environment:
. /u/local/Modules/default/init/modules.sh
module use /u/project/CCN/apps/modulefiles

# Load the FSL module
module load fsl

# This is optional
# More info here: https://www.ccn.ucla.edu/wiki/index.php/Hoffman2:FSL
export NO_FSL_JOBS=true

# Set up the subjects list
declare -a subjects

subjects[1]="su3v3hkaykw2"
subjects[2]="wxg5mk5u5xbz"
subjects[3]="6q2bgkqu5grp"
subjects[4]="whjue68jmwyh"
subjects[5]="pfx3ju9wz8rr"

echo "This is sub-job $SGE_TASK_ID"
echo "This is subject ${subjects[$SGE_TASK_ID]}"

# Your script content goes here...</nowiki>

Hoffman2:Submit job (tcsh)

2017-11-29T19:20:25Z

Dmargolis:

[[Hoffman2|Back to all things Hoffman2]]
<nowiki>#!/bin/tcsh
#$ -cwd
# error = Merged with joblog
#$ -o joblog.$JOB_ID
#$ -j y
#$ -pe shared 2
#$ -l h_rt=8:00:00,h_data=4G
# Email address to notify
#$ -M $USER@mail
# Notify when
#$ -m bea

# Load the job environment:
source /u/local/Modules/default/init/modules.csh
module use /u/project/CCN/apps/modulefiles

# Load the FSL module
module load fsl

# This is optional
# More info here: https://www.ccn.ucla.edu/wiki/index.php/Hoffman2:FSL
setenv NO_FSL_JOBS true

# Your script content goes here...</nowiki>

Hoffman2:Submit job (tcsh)

2017-11-29T19:20:09Z

Dmargolis:

[[Hoffman2|Back to all things Hoffman2]]
<nowiki>#!/bin/tcsh
#$ -cwd
# error = Merged with joblog
#$ -o joblog.$JOB_ID
#$ -j y
#$ -pe shared 2
#$ -l h_rt=8:00:00,h_data=4G
# Email address to notify
#$ -M $USER@mail
# Notify when
#$ -m bea

# Load the job environment:
source /u/local/Modules/default/init/modules.csh
module use /u/project/CCN/apps/modulefiles

# Load fsl module
module load fsl

# This is optional
# More info here: https://www.ccn.ucla.edu/wiki/index.php/Hoffman2:FSL
setenv NO_FSL_JOBS true

# Your script content goes here...</nowiki>

Hoffman2:Job Submission Templates

2017-11-29T19:16:42Z

Dmargolis: /* Job submission templates */

Hoffman2:Submit job example

2017-11-29T19:16:08Z

Dmargolis: Dmargolis moved page Hoffman2:Submit job example to Hoffman2:Submit job

#REDIRECT [[Hoffman2:Submit job]]

Hoffman2:Submit job

2017-11-29T19:16:08Z

Dmargolis: Dmargolis moved page Hoffman2:Submit job example to Hoffman2:Submit job

[[Hoffman2|Back to all things Hoffman2]]
<nowiki>#!/bin/bash
#$ -cwd
# error = Merged with joblog
#$ -o joblog.$JOB_ID
#$ -j y
#$ -pe shared 2
#$ -l h_rt=8:00:00,h_data=4G
# Email address to notify
#$ -M $USER@mail
# Notify when
#$ -m bea

# load the job environment:
. /u/local/Modules/default/init/modules.sh
module use /u/project/CCN/apps/modulefiles
module load fsl

# This is optional
# More info here: https://www.ccn.ucla.edu/wiki/index.php/Hoffman2:FSL
export NO_FSL_JOBS=true

# Your script content goes here...</nowiki>

Hoffman2:Submit jobarray example (readarray)

2017-11-29T19:15:51Z

Dmargolis: Dmargolis moved page Hoffman2:Submit jobarray example (readarray) to Hoffman2:Submit jobarray (readarray)

#REDIRECT [[Hoffman2:Submit jobarray (readarray)]]

Hoffman2:Submit jobarray (readarray)

2017-11-29T19:15:51Z

Dmargolis: Dmargolis moved page Hoffman2:Submit jobarray example (readarray) to Hoffman2:Submit jobarray (readarray)

[[Hoffman2|Back to all things Hoffman2]]

In this example, subjects are read in from a subjects.txt file. subjects.txt is a text file with a single subject ID on each line. '''Watch out for non-Linux (Mac or PC) line endings.''' Line ending issues can be fixed with <code>dos2unix</code>.

<nowiki>#!/bin/bash
#$ -cwd
# error = Merged with joblog
#$ -o joblog.$JOB_ID.$TASK_ID
#$ -j y
#$ -pe shared 2
#$ -l h_rt=8:00:00,h_data=4G
# Email address to notify
#$ -M $USER@mail
# Notify when
#$ -m a
# Job array indexes
#$ -t 1-16:1

# Load the job environment:
. /u/local/Modules/default/init/modules.sh
module use /u/project/CCN/apps/modulefiles
module load fsl

# This is optional
# More info here: https://www.ccn.ucla.edu/wiki/index.php/Hoffman2:FSL
export NO_FSL_JOBS=true

# Set up the subjects list
readarray -t subjects < subjects.txt
(( i=$SGE_TASK_ID - 1 ))

echo "This is sub-job $SGE_TASK_ID"
echo "This is subject ${subjects[$i]}"

# Your script content goes here...</nowiki>

Hoffman2:Submit jobarray example

2017-11-29T19:15:42Z

Dmargolis: Dmargolis moved page Hoffman2:Submit jobarray example to Hoffman2:Submit jobarray

#REDIRECT [[Hoffman2:Submit jobarray]]

Hoffman2:Submit jobarray

2017-11-29T19:15:42Z

Dmargolis: Dmargolis moved page Hoffman2:Submit jobarray example to Hoffman2:Submit jobarray

[[Hoffman2|Back to all things Hoffman2]]
<nowiki>#!/bin/bash
#$ -cwd
# error = Merged with joblog
#$ -o joblog.$JOB_ID.$TASK_ID
#$ -j y
#$ -pe shared 2
#$ -l h_rt=8:00:00,h_data=4G
# Email address to notify
#$ -M $USER@mail
# Notify when
#$ -m a
# Job array indexes
#$ -t 1-5:1

# Load the job environment:
. /u/local/Modules/default/init/modules.sh
module use /u/project/CCN/apps/modulefiles
module load fsl

# This is optional
# More info here: https://www.ccn.ucla.edu/wiki/index.php/Hoffman2:FSL
export NO_FSL_JOBS=true

# Set up the subjects list
declare -a subjects

subjects[1]="su3v3hkaykw2"
subjects[2]="wxg5mk5u5xbz"
subjects[3]="6q2bgkqu5grp"
subjects[4]="whjue68jmwyh"
subjects[5]="pfx3ju9wz8rr"

echo "This is sub-job $SGE_TASK_ID"
echo "This is subject ${subjects[$SGE_TASK_ID]}"

# Your script content goes here...</nowiki>

Hoffman2:Submit job example (tcsh)

2017-11-29T19:15:24Z

Dmargolis: Dmargolis moved page Hoffman2:Submit job example (tcsh) to Hoffman2:Submit job (tcsh)

#REDIRECT [[Hoffman2:Submit job (tcsh)]]

Hoffman2:Submit job (tcsh)

2017-11-29T19:15:24Z

Dmargolis: Dmargolis moved page Hoffman2:Submit job example (tcsh) to Hoffman2:Submit job (tcsh)

[[Hoffman2|Back to all things Hoffman2]]
<nowiki>#!/bin/tcsh
#$ -cwd
# error = Merged with joblog
#$ -o joblog.$JOB_ID
#$ -j y
#$ -pe shared 2
#$ -l h_rt=8:00:00,h_data=4G
# Email address to notify
#$ -M $USER@mail
# Notify when
#$ -m bea

# load the job environment:
source /u/local/Modules/default/init/modules.csh
module use /u/project/CCN/apps/modulefiles
module load fsl

# This is optional
# More info here: https://www.ccn.ucla.edu/wiki/index.php/Hoffman2:FSL
setenv NO_FSL_JOBS true

# Your script content goes here...</nowiki>

Hoffman2:Job Submission Templates

2017-11-29T19:14:45Z

Dmargolis: /* Job submission templates */

===Job submission templates===
'''[[Hoffman2:Submit_job_example|Submit job]]'''
*example script for submitting a single job
'''[[Hoffman2:Submit_job_example_(tcsh)|Submit job (tcsh)]]'''
*same as above, written for tcsh
'''[[Hoffman2:Submit_jobarray_example|Submit jobarray]]'''
*example script for submitting a jobarray with hard-coded array values
'''[[Hoffman2:Submit_jobarray_example_(readarray)|Submit jobarray (readarray)]]'''
*example script for submitting a jobarray with an array read in from a file, e.g. 'subjects.txt'

===To work with these example scripts===
#Edit the "preamble" content at the top to adjust the memory (h_data) and run time (h_rt).
#*You can also adjust the number of cores: 2 cores is '-pe shared 2'. I recommend 2, 4, or 8 for this value. Be aware that the number of cores is a multiplier for the RAM. h_data=4G and 2 cores is 8G total.
#Put your script content at the bottom.
#Submit directly to the job scheduler like this: <code>qsub myscript.sh</code>

Hoffman2:Submitting Jobs

2017-11-29T19:13:36Z

Dmargolis:

[[Hoffman2|Back to all things Hoffman2]]

If you remember from [[Hoffman2:Introduction#Sun Grid Engine|Anatomy of the Computing Cluster]], the Univa Grid Engine on Hoffman2 is the scheduler for all computing jobs. It takes your computing job request, considers what resources you are asking for and then puts your job in a line waiting for those resources to become available.

Ask for a simple 1GB of memory and a single computing core with a short time window, and your job will likely get placed at the front of the line and start running soon if not immediately.

Ask for a lot of memory or many computing cores, and your job will get put further back in the line because it will have to wait for more things to become available. Ask for too little RAM or too little time and your job will be killed or end prematurely leaving you with no results to examine.

=Job Submission Types=

So how does one submit a computing job request? You've got some options:
# '''job.q'''
#: Use a simple tool that IDRE wrote. It has a menu and walks you through submitting things.
# '''qsub'''
#: Get under the hood and do it yourself. It can get messy but it can also be faster and you have more flexibility with options.
# '''command files'''
#: You've graduated to a higher level of operations, but we can help you get there with examples of our own command files.
# '''job arrays'''
#: You've got a lot of repetitive tasks to run, these will be your friend.



==job.q==
Once you've identified or written a script you'd like to run, [[Hoffman2:Accessing the Cluster#SSH - Command Line|SSH into Hoffman2]] and enter <code>job.q</code>. Then it is just a matter of following its step-by-step instructions.

From the tool's main menu, you can type ''Info'' to read up about how to use it and we highly encourage you to do so.

But we know patience is a virtue that most of us aren't blessed with. So we'll walk you through submitting a basic job so you can hit the ground running.

===Example===
# Once on Hoffman2, you'll need to edit one file so pull out your favorite [[Text Editors|text editor]] and edit the file
#: <pre>~/.queuerc</pre>
# Add the line
#: <pre>set qqodir = ~/job-output</pre>
# You've just set the default directory where your job command files will be created. Save the configuration file and close your text editor.
# Make that directory using the command
#: <pre>$ mkdir ~/job-output</pre>
# Now execute
#:<pre>$ job.q</pre>
# Press enter to acknowledge the message about some files that get created (READ IT FIRST THOUGH).
# Type ''Build <ENTER>'' to begin creating an SGE command file.
# The program now asks you which script you'd like to run, enter the following text to use our example script
#: <pre>/u/project/CCN/apps/examples/qsub/gather.sh</pre>
# The program now asks how much memory the job will need (in [http://en.wikipedia.org/wiki/Megabyte Megabytes]). This script is really simple, so let's go with the minimum and enter ''64''.
# The program now asks how long will the job take (in hours). Go with the minimum 1 hour; it will complete in much less than this.
# The program now asks if your job should be limited to only your resource group's cores. Answer ''n'' because you do not need to be limiting yourself here and the job is not going to be running for more than 24 hours.
# Soon, the program will tell you that ''gather.sh.cmd'' has been built and saved.
# When it asks you if you would like to submit your job, say no. Then type ''Quit <ENTER>'' to leave the program.
# Now you should be able to run
#: <pre>ls ~/job-output</pre>
#: and see ''gather.sh.cmd''. This file will stay there until you delete it and can be run over and over again. Making a command file like this is especially useful if there is a task you'll be running repeatedly on Hoffman2. But if this is something you only need to run once, you should delete the file so you don't needlessly approach your [[Hoffman2:Quotas|quota]].
# The time has come to actually run the program (thought we'd never get to that, didn't you?). Type
#: <pre>qsub job-output/gather.sh.cmd</pre>
#: and after hitting enter, a message similar to this will pop up:
#: <pre>Your job 1882940 ("gather.sh.cmd") has been submitted</pre>
#: where the number is your JobID, a unique numerical identifier for the computer job you have submitted to the queue.
# Now you can check if the job has finished running by doing
#: <pre>ls ~/job-output</pre>
# When two files named ''gather.sh.output.[JOBID]'' and ''gather.sh.joblog.[JOBID]'' (where JOBID is your job's unique identifier) appear, your job has run.
#: ''gather.sh.output.[JOBID]''
#:: This file has all the standard output generated by your script. In this case it will just have the line
#::: ''Standard output would appear here.''
#: ''gather.sh.joblog.[JOBID]''
#:: This file has all the details about when, where, and how your job was processed. Useful information if you are going to be running this job over and over and need to fine tune the resources it uses.
# Better ways of checking on your job can be found [[Hoffman2:Monitoring Jobs|here]].
# The script you ran is an aggregator. It looks in a list of directories, each assumed to contain a specifically named file, and gathers the contents of each of those files into one central file in your home directory. This file is named ''gather-[TIMESTAMP].txt'' where TIMESTAMP is when the script was run and follows [http://en.wikipedia.org/wiki/ISO_8601 ISO 8601] style encoding. You are encouraged to type
#: <pre>/u/project/CCN/apps/examples/qsub/gather.sh -h</pre>
#: or
#: <pre>/u/project/CCN/apps/examples/qsub/gather.sh --help</pre>
#: to see how this script works.
# Finally, go check the inbox of the email you used to sign up for your Hoffman2 account. There will be two emails from "root@mail.hoffman2.idre.ucla.edu" that indicate when the job was started and when the job was completed. This is one of the neat features of the queue so that you can be alerted about the progress of your job without having to stay logged into Hoffman2 and checking on it constantly.

==qsub==
Everything that job.q did can be done on the command line. And it can be done better.

===Example===
Run the command:
$ qsub -cwd -V -N J1 -l h_data=64M,h_rt=00:05:00 -M $USER -m bea /u/project/CCN/apps/examples/qsub/gather.sh

And something like the following will be printed out:
Your job 1875395 ("J1") has been submitted

Where the number is your JOBID, a unique numerical identifier for your job.

Let's break down the arguments in that command.

;<code>-cwd</code>
: Change working directory
: When your script runs, change the working directory to where you currently are in the filesystem.
:: e.g. If you were in the director /u/home/mscohen/data/ when you ran the command, the queue will change directories to that location and then execute the script you gave it. This means output and error directories will be placed here for that job.

;<code>-V</code>
: Export environment variables
: Exports all the environment variables to the context of the job. Useful if you have extra environment variables that are needed in your script.
:: e.g. If you had defined the variable SUBJECT_ID in your session on Hoffman2 (<code>export SUBJECT_ID=42</code>) before submitting a job and that variable was called on by your script, then you would need to use this flag. Tools like FreeSurfer look for certain environment variables to be set.

;<code>-N J1</code>
: Name my job
: Names your job "J1." When you [[Hoffman2:Monitoring Jobs#qstat|look at the queue]], this will be the text that shows up in the "name" column. This will also be the beginning of the output (<code>J1.o[JOBID]</code>) and error (<code>J1.e[JOBID]</code>) files for your job.

;<code>-l h_data=64M,time=00:05:00</code>
: Resource allocation (that's a lower case "elle")
: This is the resources flag meaning that the text immediately after it will ask for things like:
:* certain amount of memory, in [http://en.wikipedia.org/wiki/Megabyte Megabytes], or [http://en.wikipedia.org/wiki/Gigabyte Gigabytes]
:** h_data=64M (64 MB RAM) or h_data=1G (1 GB RAM)

: If your job uses more RAM than it requested, your job WILL be killed in order to avoid it hurting other jobs running on the same node. It is imperative that you set this RAM request properly.
:* certain length of computing time, in the form HH:MM:SS
:** h_rt=00:05:00
: In this case the script will complete its task rapidly, hence we are only asking for 5 minutes of computing time.

:* [http://www.ats.ucla.edu/clusters/hoffman2/computing/batch/policies.htm#highp highp]
:**: Job length maximum of 14 days but can only be run on nodes belonging to your resource group (type <code>mygroup</code> to see what type of resources you have available). If you are in the ccn group on Hoffman2, you have access to some of these highp nodes.



;<code>-M $USER</code>
: Define mailing list
: This defines the user that will be mailed if email updates are requested. The default address is that of the job-owner.
:: e.g. In this case, the email will be sent to the address on file for the user.

;<code>-m bea</code>
: Define mailing rules
: This defines when Hoffman2 should email you about your job. There are five options here
:* b - when the job begins
:* e - when the job ends
:* a - when the job is aborted (ends in an error state)


==Command Files==
Typing accurately can be difficult at times, so why put yourself through the trouble of having to retype the same arguments over and over if you will always be using about the same values? Enter command files.

You already have experience making a command file (~/job-output/gather.sh.cmd) from when you used the tool <code>job.q</code>. But did you know that you can edit that command file to make changes to how it runs, or write your own?

The command files generated by <code>job.q</code> are fairly well commented, so if you take a look at them with your favorite [Text Editors|text editor] you should be able to change their behavior. For instance, if you go into the command file from the job.q example, find the lines that say
# Notify at beginning and end of job
#$ -m bea
You recognize that this is the flag about when to send email messages. Go ahead and change this to
# Notify at the end and on abort
#$ -m ae
And you should only receive one email when your job finishes.


==Job Arrays==
There is an SGE qsub argument that allows you to submit multiple jobs in parallel that use the same script. It is
-t lower-upper:interval
where
;<code>lower</code>
: is replaced with the starting number
;<code>upper</code>
: is replaced with the ending number
;<code>interval</code>
: is replaced with the step interval
So adding the argument
-t 10-100:5
will step through the numbers 10, 15, 20, 25, ..., 100 submitting a job for each one.

In jobs that are called with this flag, there will be an [[Hoffman2:UNIX Tutorial#Environment Variables|environment variable]] called <code>SGE_TASK_ID</code> whose value will be incremented over the range you specified. Each possible value of SGE_TASK_ID will be submitted as its own job, so your work will be parallelized.

===Example===

Let's say you have a script, '''myFunc.sh''', that takes one numerical input and computes a bunch of values based on that input. But you need to run <code>myFunc.sh</code> for input values 1 to 100. One solution would be to write a wrapper script '''myFuncSlowWrapper.sh''' as
#!/bin/bash
# myFuncSlowWrapper.sh
for i in {1..100};
do
myFunc.sh $i;
done

The only drawback is that this will take quite a while since all 100 iterations will be done on a single processor. With job arrays, the computations will be split among many processors and can finish much more quickly. You would instead write a wrapper script called '''myFuncFastWrapper.sh''' as
#!/bin/bash
# myFuncFastWrapper.sh
echo $SGE_TASK_ID
myFunc.sh $SGE_TASK_ID

And submit it with
qsub -cwd -V -N PJ -l h_data=1024M,h_rt=01:00:00 -M $HOME -m bea -t 1-100:1 myFuncWrapper.sh

Hoffman2:Submit jobarray (readarray)

2017-11-29T02:49:39Z

Dmargolis:

[[Hoffman2|Back to all things Hoffman2]]

In this example, subjects are read in from a subjects.txt file. subjects.txt is a text file with a single subject ID on each line. '''Watch out for non-Linux (Mac or PC) line endings.''' Line ending issues can be fixed with <code>dos2unix</code>.

<nowiki>#!/bin/bash
#$ -cwd
# error = Merged with joblog
#$ -o joblog.$JOB_ID.$TASK_ID
#$ -j y
#$ -pe shared 2
#$ -l h_rt=8:00:00,h_data=4G
# Email address to notify
#$ -M $USER@mail
# Notify when
#$ -m a
# Job array indexes
#$ -t 1-16:1

# Load the job environment:
. /u/local/Modules/default/init/modules.sh
module use /u/project/CCN/apps/modulefiles
module load fsl

# This is optional
# More info here: https://www.ccn.ucla.edu/wiki/index.php/Hoffman2:FSL
export NO_FSL_JOBS=true

# Set up the subjects list
readarray -t subjects < subjects.txt
(( i=$SGE_TASK_ID - 1 ))

echo "This is sub-job $SGE_TASK_ID"
echo "This is subject ${subjects[$i]}"

# Your script content goes here...</nowiki>

Hoffman2:Submit jobarray

2017-11-29T02:49:28Z

Dmargolis:

[[Hoffman2|Back to all things Hoffman2]]
<nowiki>#!/bin/bash
#$ -cwd
# error = Merged with joblog
#$ -o joblog.$JOB_ID.$TASK_ID
#$ -j y
#$ -pe shared 2
#$ -l h_rt=8:00:00,h_data=4G
# Email address to notify
#$ -M $USER@mail
# Notify when
#$ -m a
# Job array indexes
#$ -t 1-5:1

# Load the job environment:
. /u/local/Modules/default/init/modules.sh
module use /u/project/CCN/apps/modulefiles
module load fsl

# This is optional
# More info here: https://www.ccn.ucla.edu/wiki/index.php/Hoffman2:FSL
export NO_FSL_JOBS=true

# Set up the subjects list
declare -a subjects

subjects[1]="su3v3hkaykw2"
subjects[2]="wxg5mk5u5xbz"
subjects[3]="6q2bgkqu5grp"
subjects[4]="whjue68jmwyh"
subjects[5]="pfx3ju9wz8rr"

echo "This is sub-job $SGE_TASK_ID"
echo "This is subject ${subjects[$SGE_TASK_ID]}"

# Your script content goes here...</nowiki>

Hoffman2:Submit job (tcsh)

2017-11-29T02:49:15Z

Dmargolis:

[[Hoffman2|Back to all things Hoffman2]]
<nowiki>#!/bin/tcsh
#$ -cwd
# error = Merged with joblog
#$ -o joblog.$JOB_ID
#$ -j y
#$ -pe shared 2
#$ -l h_rt=8:00:00,h_data=4G
# Email address to notify
#$ -M $USER@mail
# Notify when
#$ -m bea

# load the job environment:
source /u/local/Modules/default/init/modules.csh
module use /u/project/CCN/apps/modulefiles
module load fsl

# This is optional
# More info here: https://www.ccn.ucla.edu/wiki/index.php/Hoffman2:FSL
setenv NO_FSL_JOBS true

# Your script content goes here...</nowiki>

Hoffman2:Submit job

2017-11-29T02:49:00Z

Dmargolis:

[[Hoffman2|Back to all things Hoffman2]]
<nowiki>#!/bin/bash
#$ -cwd
# error = Merged with joblog
#$ -o joblog.$JOB_ID
#$ -j y
#$ -pe shared 2
#$ -l h_rt=8:00:00,h_data=4G
# Email address to notify
#$ -M $USER@mail
# Notify when
#$ -m bea

# load the job environment:
. /u/local/Modules/default/init/modules.sh
module use /u/project/CCN/apps/modulefiles
module load fsl

# This is optional
# More info here: https://www.ccn.ucla.edu/wiki/index.php/Hoffman2:FSL
export NO_FSL_JOBS=true

# Your script content goes here...</nowiki>

Hoffman2:Submit job (tcsh)

2017-11-29T02:48:42Z

Dmargolis:

[[Hoffman2|Back to all things Hoffman2]]
<nowiki>#!/bin/tcsh
#$ -cwd
# error = Merged with joblog
#$ -o joblog.$JOB_ID
#$ -j y
#$ -pe shared 2
#$ -l h_rt=8:00:00,h_data=4G
# Email address to notify
#$ -M $USER@mail
# Notify when
#$ -m bea

# load the job environment:
source /u/local/Modules/default/init/modules.csh
module use /u/project/CCN/apps/modulefiles
module load fsl

# This is optional
# More info here: https://www.ccn.ucla.edu/wiki/index.php/Hoffman2:FSL
setenv NO_FSL_JOBS true

# Your script content goes here...
</nowiki>

Hoffman2:Submit job

2017-11-29T02:48:33Z

Dmargolis:

[[Hoffman2|Back to all things Hoffman2]]
<nowiki>#!/bin/bash
#$ -cwd
# error = Merged with joblog
#$ -o joblog.$JOB_ID
#$ -j y
#$ -pe shared 2
#$ -l h_rt=8:00:00,h_data=4G
# Email address to notify
#$ -M $USER@mail
# Notify when
#$ -m bea

# load the job environment:
. /u/local/Modules/default/init/modules.sh
module use /u/project/CCN/apps/modulefiles
module load fsl

# This is optional
# More info here: https://www.ccn.ucla.edu/wiki/index.php/Hoffman2:FSL
export NO_FSL_JOBS=true

# Your script content goes here...

Hoffman2:Submit jobarray (readarray)

2017-11-15T00:26:53Z

Dmargolis:

[[Hoffman2|Back to all things Hoffman2]]

In this example, subjects are read in from a subjects.txt file. subjects.txt is a text file with a single subject ID on each line. '''Watch out for non-Linux (Mac or PC) line endings.''' Line ending issues can be fixed with <code>dos2unix</code>.

<nowiki>#!/bin/bash
#$ -cwd
# error = Merged with joblog
#$ -o joblog.$JOB_ID.$TASK_ID
#$ -j y
#$ -pe shared 2
#$ -l h_rt=8:00:00,h_data=4G
# Email address to notify
#$ -M $USER@mail
# Notify when
#$ -m a
# Job array indexes
#$ -t 1-16:1

# Load the job environment:
. /u/local/Modules/default/init/modules.sh
module use /u/project/CCN/apps/modulefiles
module load fsl

# This is optional
# More info here: https://www.ccn.ucla.edu/wiki/index.php/Hoffman2:FSL
export NO_FSL_JOBS=true

# Set up the subjects list
readarray -t subjects < subjects.txt
(( i=$SGE_TASK_ID - 1 ))

echo "This is sub-job $SGE_TASK_ID"
echo "This is subject ${subjects[$i]}"

# Now do something with fsl, feat, etc.
sleep 600</nowiki>

Hoffman2:Submit jobarray (readarray)

2017-11-15T00:26:24Z

Dmargolis:

[[Hoffman2|Back to all things Hoffman2]]

In this example, subjects are read in from a subjects.txt file. subjects.txt is a text file with a single subject ID listed on each line. '''Watch out for non-Linux (Mac or PC) line endings.''' Line ending issues can be fixed with <code>dos2unix</code>.

<nowiki>#!/bin/bash
#$ -cwd
# error = Merged with joblog
#$ -o joblog.$JOB_ID.$TASK_ID
#$ -j y
#$ -pe shared 2
#$ -l h_rt=8:00:00,h_data=4G
# Email address to notify
#$ -M $USER@mail
# Notify when
#$ -m a
# Job array indexes
#$ -t 1-16:1

# Load the job environment:
. /u/local/Modules/default/init/modules.sh
module use /u/project/CCN/apps/modulefiles
module load fsl

# This is optional
# More info here: https://www.ccn.ucla.edu/wiki/index.php/Hoffman2:FSL
export NO_FSL_JOBS=true

# Set up the subjects list
readarray -t subjects < subjects.txt
(( i=$SGE_TASK_ID - 1 ))

echo "This is sub-job $SGE_TASK_ID"
echo "This is subject ${subjects[$i]}"

# Now do something with fsl, feat, etc.
sleep 600</nowiki>

Hoffman2:Submit job

2017-11-15T00:25:06Z

Dmargolis:

[[Hoffman2|Back to all things Hoffman2]]
<nowiki>#!/bin/bash
#$ -cwd
# error = Merged with joblog
#$ -o joblog.$JOB_ID
#$ -j y
#$ -pe shared 2
#$ -l h_rt=8:00:00,h_data=4G
# Email address to notify
#$ -M $USER@mail
# Notify when
#$ -m bea

# load the job environment:
. /u/local/Modules/default/init/modules.sh
module use /u/project/CCN/apps/modulefiles
module load fsl

# This is optional
# More info here: https://www.ccn.ucla.edu/wiki/index.php/Hoffman2:FSL
export NO_FSL_JOBS=true

# Now do something with fsl, feat, etc.
sleep 600</nowiki>

Hoffman2:Job Submission Templates

2017-11-14T22:54:06Z

Dmargolis:

===Job submission templates===
'''[[Hoffman2:Submit_job_example|Submit job example]]'''
*example script for submitting a single job
'''[[Hoffman2:Submit_job_example_(tcsh)|Submit job example (tcsh)]]'''
*same as above, written for tcsh
'''[[Hoffman2:Submit_jobarray_example|Submit jobarray example]]'''
*example script for submitting a jobarray with hard-coded array values
'''[[Hoffman2:Submit_jobarray_example_(readarray)|Submit jobarray example (readarray)]]'''
*example script for submitting a jobarray with an array read in from a file, e.g. 'subjects.txt'

===To work with these example scripts===
#Edit the "preamble" content at the top to adjust the memory (h_data) and run time (h_rt).
#*You can also adjust the number of cores: 2 cores is '-pe shared 2'. I recommend 2, 4, or 8 for this value. Be aware that the number of cores is a multiplier for the RAM. h_data=4G and 2 cores is 8G total.
#Put your script content at the bottom.
#Submit directly to the job scheduler like this: <code>qsub myscript.sh</code>

Hoffman2:Job Submission Templates

2017-11-14T22:53:12Z

Dmargolis:

'''[[Hoffman2:Submit_job_example|Submit job example]]'''
*example script for submitting a single job
'''[[Hoffman2:Submit_job_example_(tcsh)|Submit job example (tcsh)]]'''
*same as above, written for tcsh
'''[[Hoffman2:Submit_jobarray_example|Submit jobarray example]]'''
*example script for submitting a jobarray with hard-coded array values
'''[[Hoffman2:Submit_jobarray_example_(readarray)|Submit jobarray example (readarray)]]'''
*example script for submitting a jobarray with an array read in from a file, e.g. 'subjects.txt'

To work with these example scripts:
#Edit the "preamble" content at the top to adjust the memory (h_data) and run time (h_rt).
#*You can also adjust the number of cores: 2 cores is '-pe shared 2'. I recommend 2, 4, or 8 for this value. Be aware that the number of cores is a multiplier for the RAM. h_data=4G and 2 cores is 8G total.
#Put your script content at the bottom.
#Submit directly to the job scheduler like this: <code>qsub myscript.sh</code>

Hoffman2:Job Submission Templates

2017-11-14T22:53:03Z

Dmargolis:

'''[[Hoffman2:Submit_job_example|Submit job example]]'''
*example script for submitting a single job
'''[[Hoffman2:Submit_job_example_(tcsh)|Submit job example (tcsh)]]'''
*same as above, written for tcsh
'''[[Hoffman2:Submit_jobarray_example|Submit jobarray example]]'''
*example script for submitting a jobarray with hard-coded array values
'''[[Hoffman2:Submit_jobarray_example_(readarray)|Submit jobarray example (readarray)]]'''
*example script for submitting a jobarray with an array read in from a file, e.g. 'subjects.txt'

To work with these example scripts:
#Edit the "preamble" content at the top to adjust the memory (h_data) and run time (h_rt).
*You can also adjust the number of cores: 2 cores is '-pe shared 2'. I recommend 2, 4, or 8 for this value. Be aware that the number of cores is a multiplier for the RAM. h_data=4G and 2 cores is 8G total.
#Put your script content at the bottom.
#Submit directly to the job scheduler like this: <code>qsub myscript.sh</code>

Hoffman2:Job Submission Templates

2017-11-14T22:52:56Z

Dmargolis:

'''[[Hoffman2:Submit_job_example|Submit job example]]'''
*example script for submitting a single job
'''[[Hoffman2:Submit_job_example_(tcsh)|Submit job example (tcsh)]]'''
*same as above, written for tcsh
'''[[Hoffman2:Submit_jobarray_example|Submit jobarray example]]'''
*example script for submitting a jobarray with hard-coded array values
'''[[Hoffman2:Submit_jobarray_example_(readarray)|Submit jobarray example (readarray)]]'''
*example script for submitting a jobarray with an array read in from a file, e.g. 'subjects.txt'

To work with these example scripts:
#Edit the "preamble" content at the top to adjust the memory (h_data) and run time (h_rt).
**You can also adjust the number of cores: 2 cores is '-pe shared 2'. I recommend 2, 4, or 8 for this value. Be aware that the number of cores is a multiplier for the RAM. h_data=4G and 2 cores is 8G total.
#Put your script content at the bottom.
#Submit directly to the job scheduler like this: <code>qsub myscript.sh</code>

Hoffman2:Job Submission Templates

2017-11-14T22:52:06Z

Dmargolis:

'''[[Hoffman2:Submit_job_example|Submit job example]]'''
*example script for submitting a single job
'''[[Hoffman2:Submit_job_example_(tcsh)|Submit job example (tcsh)]]'''
*same as above, written for tcsh
'''[[Hoffman2:Submit_jobarray_example|Submit jobarray example]]'''
*example script for submitting a jobarray with hard-coded array values
'''[[Hoffman2:Submit_jobarray_example_(readarray)|Submit jobarray example (readarray)]]'''
*example script for submitting a jobarray with an array read in from a file, e.g. 'subjects.txt'

To work with these example scripts:
#Edit the "preamble" content at the top to adjust the memory (h_data) and run time (h_rt). You can also adjust the number of cores: 2 cores is '-pe shared 2'. I recommend 2, 4, or 8 for this value. Be aware that the number of cores is a multiplier for the RAM. h_data=4G and 2 cores is 8G total.
#Put your script content at the bottom.
#Submit directly to the job scheduler like this: <code>qsub myscript.sh</code>

Hoffman2:Job Submission Templates

2017-11-14T22:51:50Z

Dmargolis:

'''[[Hoffman2:Submit_job_example|Submit job example]]'''
*example script for submitting a single job
'''[[Hoffman2:Submit_job_example_(tcsh)|Submit job example (tcsh)]]'''
same as above, written for tcsh
'''[[Hoffman2:Submit_jobarray_example|Submit jobarray example]]'''
*example script for submitting a jobarray with hard-coded array values
'''[[Hoffman2:Submit_jobarray_example_(readarray)|Submit jobarray example (readarray)]]'''
*example script for submitting a jobarray with an array read in from a file, e.g. 'subjects.txt'

To work with these example scripts:
#Edit the "preamble" content at the top to adjust the memory (h_data) and run time (h_rt). You can also adjust the number of cores: 2 cores is '-pe shared 2'. I recommend 2, 4, or 8 for this value. Be aware that the number of cores is a multiplier for the RAM. h_data=4G and 2 cores is 8G total.
#Put your script content at the bottom.
#Submit directly to the job scheduler like this: <code>qsub myscript.sh</code>

Hoffman2:Job Submission Templates

2017-11-14T22:50:28Z

Dmargolis:

===[[Hoffman2:Submit_job_example|Submit job example]]===
*example script for submitting a single job
*[[Hoffman2:Submit_job_example_(tcsh)|Submit job example (tcsh)]]
**same as above, written for tcsh
*Submit jobarray example
**example script for submitting a jobarray with hard-coded array values
*Submit jobarray example (readarray)
**example script for submitting a jobarray with an array read in from a file, e.g. 'subjects.txt'

To work with these example scripts:
#Edit the "preamble" content at the top to adjust the memory (h_data) and run time (h_rt). You can also adjust the number of cores: 2 cores is '-pe shared 2'. I recommend 2, 4, or 8 for this value. Be aware that the number of cores is a multiplier for the RAM. h_data=4G and 2 cores is 8G total.
#Put your script content at the bottom.
#Submit directly to the job scheduler like this: <code>qsub myscript.sh</code>