Jobs on c0-3 to c0-68 must be submitted through PBS on the frontend node. This page gives the relevant instructions. For more information on PBS (the Torque version that we have) see http://www.supercluster.org/projects/torque/ . For more information on Maui see http://www.supercluster.org/maui/ .
PBS is a queueing system similar to what is implemented at supercomputer centers etc. Jobs are submitted to the queue that reflects the resources needed, and the Maui scheduler decides which ones to run when nodes become available. These decisions are made on the basis of length of run, how long a job has been waiting, and (given the greatest weighting) fair sharing of resources among different users. Maui is more sophisticated than the FIFO scheduler used on komp, and includes the ability to backfill, i.e., fill 'holes' in the schedule with short jobs that are otherwise not the highest priority ones. This works best if people are accurate about how much time their jobs need.
Two X-windows based tools are available. xpbsmon shows the status of the various nodes (free, used, down etc.) and xpbs shows the various queues and jobs in them. By default xpbs shows only your jobs, but you can set it to show all users' jobs. Alternatively, the web browser mozilla can be used to look at the PBS queues. For non-graphical information use qstat (type 'man qstat' for more details). For detailed information on particular jobs or queues the Maui commands checkjob, showq, checknode, etc. are also useful.
Presently there are 8 queues. The details may be adjusted based on user experience.
Rather than having different queues for different duration runs, you can specify the time required when submitting a job (see below). If you don't specify a time then it defaults to 24 hours wall time. The maximum run duration is 1 or 2 weeks depending on the queue. After your time has elapsed PBS kills the job regardless of whether it has finished. Thus, it is best to specify a longer time than you need. However, shorter jobs are likely to start sooner so it is best not to overestimate the time needed. Time is measured as wall clock time, not CPU time.
e.g., qsub -q n2 myscript
This submits script file myscript to queue n2. When cpus get allocated, PBS will execute the script on the first of the allocated CPUs. Thus the script must contain the list of unix commands needed to run your job from a log in. For MPI jobs use mpiexec not mpirun - it is faster and you don't need to specify the number of nodes, it is the same as the number allocated. A minimalist script file for a parallel job might contain something like:
cd /home/pjt/+stag ; cp par.test par ; mpiexec stag3d
To specify run (wall/clock) time, use the -l option, e.g., qsub -q n2 -l walltime=48:00:00 myscript
To delete a job, use qdel, e.g., qdel 123.komp01.ess.ucla.edu
To monitor the job use qstat, xpbs, checkjob etc. as discussed earlier.
If your .login or .cshrc file sets terminal characteristics it may be necessary to bypass that for PBS login processes using an "if" block:
if ( ! $?PBS_ENVIRONMENT ) then
<terminal setting stuff>
end if