LCRC has set up a new preemptable queue on the KNL partition (named as knl-preemptable). This covers all of the same nodes in the knlall partition. The preemptable queue is largely targeted towards users who need to run jobs but do not have sufficient time available in their projects or wish to stretch their allocations further. As the name implies, these jobs are preemptable immediately by other normal jobs if the partition (knlall) is full. The main advantage to users is that preemptable jobs are charged at 0.2 (20%) the normal core-hours.
The user has to have an active project with some remaining time to be able to submit jobs to the preemptable queue.
The rules for submitting jobs to the preemptable queue are as follows:
Maximum job size: 6 nodes
Maximum time: 24 hours
Maximum number of running jobs per project: 1
A single user can submit multiple jobs to the preemptable queue if they have more than one project. If a user submits two preemptable jobs in any given project, only one of the jobs will run while the other is queued.
As an example of the core-hour savings, a job run for 24 hours on 6 KNL nodes would be charged:
6x64x24x0.585 = 5391 core-hours (0.585 is the scaling factor for jobs run on the knl partition) whereas in the preemptable queue, the core-hours charged would be:
6x64x24x0.2 = 1843 core-hours (0.2 is the additional scaling factor for preemptable jobs).
It is important that the user have checkpointing available and enabled so intermediate solutions are being saved at regular intervals as the job is running. If a job is preempted, the job can be automatically requeued so it can start again as new resources become available if the requeue flag is included during job submission. It is also important that users make appropriate changes to their scripts and input files to have the requeued job start from the latest saved solution. It is recommended that users periodically clean up their project spaces to delete intermediate solution files they might not require after the preemptable job is completed.
Users wishing to submit jobs to the preemptable queue should include the line
in their batch script.
It is recommended that users include the following lines in their batch-scripts if they desire the job to be requeued and to be notified if their jobs have been preempted and requeued. This will enable them to make necessary changes to the input file to restart the jobs from the last saved solution (if need be).
#SBATCH –mail-user=<your email address>
If you have any questions, please let us know at support at lcrc.anl.gov
LCRC Support Team