SGE Guide

This page describes how hpc-runner maps job fields onto Sun Grid Engine (SGE) submission flags and directives.

Queues (default queue)

To set a default queue for all jobs:

[defaults]
queue = "batch.q"

To override per job:

hpc run --queue gpu.q "python train.py"

Or using passthrough flags directly:

hpc run -q gpu.q -- python train.py

CPU / slots (parallel environment)

cpu maps to SGE PE slots:

• config: [schedulers.sge].parallel_environment (default: smp)

• job field: cpu (e.g. 8)

• SGE: qsub -pe <parallel_environment> <cpu>

Example:

[schedulers.sge]
parallel_environment = "smp"

hpc run --cpu 8 "python train.py"

Memory and time resources

hpc-runner renders mem and time using SGE hard resources:

• config: [schedulers.sge].memory_resource (default: mem_free)

• config: [schedulers.sge].time_resource (default: h_rt)

• SGE: qsub -l <memory_resource>=<mem> -l <time_resource>=<time>

Example (site using h_vmem and h_rt):

[schedulers.sge]
memory_resource = "h_vmem"
time_resource = "h_rt"

Custom resources

Use resources to request arbitrary -l resources:

[types.gpu]
queue = "gpu.q"
resources = [{ name = "gpu", value = 1 }]

This renders as:

• #$ -l gpu=1 in the generated script

Native qsub passthrough

If you need site-specific flags not modeled as first-class fields, pass them through:

hpc run -P myproject -l exclusive=true -- python train.py