Scheduling and queuing

From IBERS Bioinformatics and HPC Wiki
Jump to: navigation, search

The first thing that is important to note when using HPC that what you're attempting to do is not possible on a desktop PC. If it is, then you don't really need to be using the HPC to run your job.

You have many more CPU cores, faster disk access speed and lots more memory than on your desktop computer. However, you don't have a graphical interface (i.e. you don't have a monitor attached to the HPC) and once you've submitted your job, it will wait until the resources you've requested are available and then run. At no point will you have control of your job whilst it is running. This is reminiscent of the days of punch-cards when you give your program to the person in charge and wait until it comes back, either complete or with an error.

It's unrealistic to expect that if you request a large percentage of the available resources from the scheduler, that it will instantly become available, it won't. The scheduler will attempt to run as many jobs as possible with the information it has been given.

There are several rules that the scheduler works by;

Fair Share

Assuming all things equal, i.e. if there are 100 slots in a queue, and two users submit 100 identical single jobs each into the queue, then each user will run 50 jobs at a time.

Resource Allocation

If you're accurate about what you are requesting, your jobs will be scheduled. If you omit information about your job, the scheduler will assume the worst. i.e. the defaults (see Complex submissions).

Backfilling

If there is space for smaller jobs to skip ahead, then the scheduler will allow this to keep the HPC usage at 100%.