All computing carried out on LCRC clusters is associated with a specific project (also known as an account in Slurm). Whenever a computing job runs on any computing node, the time the job uses will be counted and recorded as computing used by the associated project. A job must have a project in order to run on the computing nodes and will be assigned to your default project if none has been specified in your job script. A project must also have computing time available in order to run.
This page describes projects in detail, including policies, concepts, background, etc. Valuable links that will again be mentioned below include:
The Project Life Cycle
Projects normally go through the following life cycle. Detailed descriptions of the life cycle will be elaborated further down on this page.
- The PI requests a project using this web page: Requesting a New Project
- The project request is sent to the LCRC Allocations Administrator and to the LCRC Allocations Committee.
- The committee’s decision is reported via email to the project requester.
- If the project has been approved, the mail will include the number of hours allocated to the project. This number may be different from what was requested. This number may also be some initial allocation that will subsequently be augmented based on decisions by the LCRC Allocations Committee.
- As a part of creating the project, someone on the LCRC Allocations Committee will be affiliated with the project as a Point of Contact (PoC). That person will be the project’s contact on the committee. They are expected to become moderately familiar with the project, to act as the project’s advocate if necessary, and to be able to explain the project and the project’s status to the committee.
- Every quarter starting October 1, all projects will have their allocations zeroed out and be restarted with new allocations as requested and approved. This will be done in order to adjust and balance the use of the system. Unused time does not carry over to the next quarter.
- When making quarterly decisions, the Allocations Committee PoC will look at the usage for the previous quarter and may query any projects that have not been using most of their time to learn what their plan is for the next quarter. Sometimes they also query projects that have used a lot of time for their future plans.
- Once the project has been created, PIs can add other users to the project, appoint co-PIs, and administer the project’s allocation.
- In advance of the annual October (Argonne’s new fiscal year) re-allocation, PIs will be reminded to send in a request for the next year’s allocation for their project.
- Also in advance of the annual October re-allocation, PIs will also be required to provide a report on their project at that time. If reports had been sent in for previous quarters, those will be sufficient, if they are current. The basic goal here is to get at least one annual report for each project. This report will be used to help describe the overall use of the systems and will also be used when deciding on annual allocations for all projects. The format of the report will be roughly 2-3 pages long, with pictures highly encouraged. Projects that do not submit a report will not receive any new time for the new fiscal year until a report is received.
An Example Project
As an example for how projects work, consider a project named popcorn, which might have a project title of “Simulation of Popcorn Kernel Dynamics”.
Following the creation of the project during the Project Life Cycle, the project would get an allocation, perhaps of 20,000 initial hours for the year, and might have a PI and 3 other scientists collaborating on it.
If one of the scientists runs a job as part of the popcorn project for 10 hours across 100 8-core nodes, then that would use 8,000 hours of the total allocation, leaving 12,000 hours remaining.
Once those 12,000 hours are gone, the project will no longer be able to run jobs. At this point, the project might determine that it needs more time on the system. A designated PI of the project can request more time for the project at this point that will require approval from the LCRC Allocations Committee. The request approval and decision will be transmitted via email to the PI after completion of the web form.
Project PIs and User Accounts
Each project has at least one Primary Investigator, or PI. The PI is generally the person who requested that the project be created, and is the first point of contact for things like allocation decisions. The primary project PI must be a current Argonne employee. A project can have a co-PIs, who also has the ability to manage the project like the PI. Co-PIs do not need to be current Argonne employees.
A project can have one or more user accounts associated with it. Some projects may only have one user account (the PI), while others may have dozens. A user account can be associated with more than one project, and can also be the PI on more than one project.
The PI is responsible for adding and removing user accounts from projects.
The PI is responsible for administering the allocation associated with the project, i.e. describing who on the project can use how much of the allocation.
Startup Projects and Allocations
In order to help promote the use of the system and to foster new projects, all new user accounts belonging to active Argonne employees are given a “startup project” on Bebop only. The startup project can be used by someone getting familiar with the system, planning a new project, as “breeding ground” for new project ideas, or whatever the user wishes.
The startup project has an 20,000 hour allocation. The startup projects will not be allocated more time beyond the initial 20,000 hours. The 20,000 hours will never expire, i.e. it does not have to be used in the first year.
Startup projects should be used only by the account originally associated with that project.
It is expected that anyone wishing to use the system for more than 20,000 hours will either request their own project or join someone else’s project. Non-Argonne employees will always need to be a member of another active project belonging to an Argonne PI.
Joining an Existing Project
Whether you have a new or existing LCRC account, you may need to join an existing project. You may already have a project PI contact in LCRC with a project you are intending to join. If so, you can contact them to have them add you to their project if they have not already done so for you. If you would like to generate a request to the PI of a project or you know of a project you should have access to, you can request to join an existing project by visiting here: Requesting to Join a Project. Once the project PI approves your request to join the project, you should be able to use the project hours with your jobs. Please note that once your project access is approved, filesystem permissions make take up to an hour to grant access to the project folder on LCRC resources.
Allocation Usage and Tracking
General policies concerning how allocations are used and tracked:
- If a system error occurs that causes a program to crash while it is running, a project won’t be charged for that time. (This policy may be amended in the future in order to promote the use of user-based checkpointing.) The scheduler may or may not deduct the time used from the project’s allocation, depending on how the crash took place. If someone thinks their project should be credited time because of a system crash or other system problem, they should send email to email@example.com to get that time back into the project.
- If a project runs out of allocation time during a run, that run will be allowed to continue to completion.
- If a project runs out of allocation time, all users on the project will not be able to run any jobs until time is added again. At the present time, no steps will be taken to stop any jobs associated with that project from running and sitting idle in the job queue.
- PIs and Users can track their project’s usage by running the command mentioned in the next section of this page.
These policies are intended to keep things flexible in order to support the best overall scientific use of the system.
Project Allocation Queries and Management
The below commands will allow LCRC users to query their project balances, project transactions, default projects and allow you to change your default project. These commands are cluster specific and will only present the information for the current cluster you are on. You must also set your default project individually on each cluster.
|Command (Bebop & Blues)||Description|
|lcrc-sbank -q balance||Query all of your LCRC project balances.|
|lcrc-sbank -q balance <proj_name>||Query a specific LCRC project balance.|
|lcrc-sbank -q default||Query your default LCRC project.|
|lcrc-sbank -s default <proj_name>||Change your default LCRC project.|
lcrc-sbank -q trans <proj_name>
|Query all transactions on an LCRC project.|
|Print the help menu for the lcrc-sbank command.|
Requesting Additional Project Time
Occasionally, some LCRC projects might use up their quarterly time allocation well before the start of their next quarterly allocation. This can happen if the computations require more time than anticipated or if more case studies need to be completed. Under such circumstances, the project PIs can request additional time with proper justification via the web form here: Managing Existing Projects
If a PI needs to request time for a quarter other than the current quarter, please note this in your request’s justification. Secondly, if a PI needs to request time on Blues specifically, please also note this in the justification. By default, time is allocated to Bebop unless otherwise noted.
The LCRC core team meets every Wednesday and reviews all time requests for both new and existing projects. The LCRC core team can grant additional time up to 150K core-hours with requests above this needing approval from the LCRC allocations committee. PIs are reminded that any unused additional time will also expire at the end of the quarter. For instance, if a PI requested 100K core-hours for the first quarter and used only 40K core-hours, 60K core-hours would be lost at the end of the quarter.
The web form here is also used for requesting time for the entire next fiscal year after the first week of September. Quarterly allocations can still be made after the first quarter begins. PIs that require additional time after the first week of September for the current fiscal year up until the end of September, should send an email to firstname.lastname@example.org requesting additional time with proper justification.