Skip to Main Content

The University of Tennessee

Newton header banner

Frequently Used Tools:



Home » Documentation » Using the Grid Engine » Complex Resource Requests

Complex resource requests

The Grid Engine supports a powerful resource management concept that it calls resource "complexes". The idea is that there are parameter variables for each queue and host that define the resources available there. One example is the complex virtualfree_ that describes how much memory is free on a compute node. Most of these variables are Requestable, meaning that you can request them using the "-l" qsub option when submitting a job. Using resource complexes, you can request a minimum amount of RAM, a specific CPU architecture, and other resources for your job. The command qconf -sc will detail the complexes and their valid values. Consult the man page "complex" for more information.

Requesting RAM usage

The standard allocation of RAM is 2GB per requested processor core. If your job needs more than this, then you must request the specific upper limit of RAM usage when the job is submitted. For example, if you have a 128-processor MPI job that requires up to 3GB of RAM per MPI process, then you should use the following line in your batch submit script:

#$ -l mem=3G

You can also request that the Grid Engine enforce a memory usage limit on your job. This is useful if you are afraid that a job may use too much RAM and crash a compute node. To place the limit, you can request the "h_vmem" complex. You may also need to request a lower limit for the "h_stack" complex (some applications will not work correctly if the stack limit is set too high). This example also shows the recommended practice of also requesting the the "mem" complex in order to make sure that the RAM used is dedicated to my job (to avoid over-subscription of RAM and thus swapping):

#$ -l mem=48G,h_vmem=48G,h_stack=128M

Requesting a processor type

If your job is sensitive to the type of processor that it run on, you may wish to request the optimal CPU-type on which to execute. You may do this using the "prov_vendor" complex. For example, to request an Intel-based CPU, you could use the following line in your batch submit file:

#$ -l proc_vendor=Intel

For an AMD CPU:

#$ -l proc_vendor=AMD

List of default complexes

#name               shortcut   type        relop requestable consumable default  urgency 
#----------------------------------------------------------------------------------------
arch                a          RESTRING    ==    YES         NO         NONE     0
calendar            c          RESTRING    ==    YES         NO         NONE     0
cpu                 cpu        DOUBLE      >=    YES         NO         0        0
display_win_gui     dwg        BOOL        ==    YES         NO         0        0
h_core              h_core     MEMORY      <=    YES         NO         0        0
h_cpu               h_cpu      TIME        <=    YES         NO         0:0:0    0
h_data              h_data     MEMORY      <=    YES         NO         0        0
h_fsize             h_fsize    MEMORY      <=    YES         NO         0        0
h_rss               h_rss      MEMORY      <=    YES         NO         0        0
h_rt                h_rt       TIME        <=    YES         NO         0:0:0    0
h_stack             h_stack    MEMORY      <=    YES         NO         0        0
h_vmem              h_vmem     MEMORY      <=    YES         NO         0        0
hostname            h          HOST        ==    YES         NO         NONE     0
load_avg            la         DOUBLE      >=    NO          NO         0        0
load_long           ll         DOUBLE      >=    NO          NO         0        0
load_medium         lm         DOUBLE      >=    NO          NO         0        0
load_short          ls         DOUBLE      >=    NO          NO         0        0
mem_free            mf         MEMORY      <=    YES         NO         0        0
mem_total           mt         MEMORY      <=    YES         NO         0        0
mem_used            mu         MEMORY      >=    YES         NO         0        0
min_cpu_interval    mci        TIME        <=    NO          NO         0:0:0    0
np_load_avg         nla        DOUBLE      >=    NO          NO         0        0
np_load_long        nll        DOUBLE      >=    NO          NO         0        0
np_load_medium      nlm        DOUBLE      >=    NO          NO         0        0
np_load_short       nls        DOUBLE      >=    NO          NO         0        0
num_proc            p          INT         ==    YES         NO         0        0
qname               q          RESTRING    ==    YES         NO         NONE     0
rerun               re         BOOL        ==    NO          NO         0        0
s_core              s_core     MEMORY      <=    YES         NO         0        0
s_cpu               s_cpu      TIME        <=    YES         NO         0:0:0    0
s_data              s_data     MEMORY      <=    YES         NO         0        0
s_fsize             s_fsize    MEMORY      <=    YES         NO         0        0
s_rss               s_rss      MEMORY      <=    YES         NO         0        0
s_rt                s_rt       TIME        <=    YES         NO         0:0:0    0
s_stack             s_stack    MEMORY      <=    YES         NO         0        0
s_vmem              s_vmem     MEMORY      <=    YES         NO         0        0
seq_no              seq        INT         ==    NO          NO         0        0
slots               s          INT         <=    YES         YES        1        1000
swap_free           sf         MEMORY      <=    YES         NO         0        0
swap_rate           sr         MEMORY      >=    YES         NO         0        0
swap_rsvd           srsv       MEMORY      >=    YES         NO         0        0
swap_total          st         MEMORY      <=    YES         NO         0        0
swap_used           su         MEMORY      >=    YES         NO         0        0
tmpdir              tmp        RESTRING    ==    NO          NO         NONE     0
virtual_free        vf         MEMORY      <=    YES         NO         0        0
virtual_total       vt         MEMORY      <=    YES         NO         0        0
virtual_used        vu         MEMORY      >=    YES         NO         0        0

Back to Using The Grid Engine