All computing carried out on LCRC clusters is associated with a specific project (also known as an account in Slurm). Whenever a computing job runs on any computing node, the time the job uses will be counted and recorded as computing used by the associated project. A job must have a project in order to run on the computing nodes and will be assigned to your default project if none has been specified in your job script. A project must also have computing time available in order to run.
This page describes projects in detail, including policies, concepts, background, etc.
The Project Life Cycle
Projects normally go through the following life cycle. Detailed descriptions of the life cycle will be elaborated further down on this page.
- The Principal Investigator (PI) requests a project using at https://accounts.lcrc.anl.gov. The PI must be a current Argonne employee.
- The project request is sent to the LCRC Allocations Administrator and to the LCRC Allocations Committee.
- The committee’s decision is reported via email to the project requester.
- If the project has been approved, the mail will include the number of hours allocated to the project. This number may be different from what was requested. This number may also be some initial allocation that will subsequently be augmented based on decisions by the LCRC Allocations Committee.
- As a part of creating the project, someone on the LCRC Allocations Committee will be affiliated with the project as a Point of Contact (PoC). That person will be the project’s contact on the committee. They are expected to become moderately familiar with the project, to act as the project’s advocate if necessary, and to be able to explain the project and the project’s status to the committee.
- Every fiscal quarter starting October 1, all projects will have their allocations zeroed out and be restarted with new allocations as requested and approved. This will be done in order to adjust and balance the use of the system. Unused time does not carry over to the next quarter.
- When making quarterly decisions, the Allocations Committee PoC will look at the usage for the previous quarter and may query any projects that have not been using most of their time to learn what their plan is for the next quarter. Sometimes they also query projects that have used a lot of time for their future plans.
- Once the project has been created, PIs can add other users to the project, appoint Proxies (or Co-PIs), and administer the project’s allocation.
- In advance of the annual October (Argonne’s new fiscal year) re-allocation, PIs will be reminded to send in a request for the next year’s allocation for their project.
- Also in advance of the annual October re-allocation, PIs will also be required to provide a report on their project at that time. If reports had been sent in for previous quarters, those will be sufficient, if they are current. The basic goal here is to get at least one annual report for each project. This report will be used to help describe the overall use of the systems and will also be used when deciding on annual allocations for all projects. The format of the report will be roughly 2-3 pages long, with pictures highly encouraged. Projects that do not submit a report will not receive any new time for the new fiscal year until a report is received.
An Example Project
As an example for how projects work, consider a project named popcorn, which might have a project title of “Simulation of Popcorn Kernel Dynamics”.
Following the creation of the project during the Project Life Cycle, the project would get an allocation, perhaps of 20,000 initial hours for the year, and might have a PI and 3 other scientists collaborating on it.
If one of the scientists runs a job as part of the popcorn project for 10 hours across 100 8-core nodes, then that would use 8,000 hours of the total allocation, leaving 12,000 hours remaining.
Once those 12,000 hours are gone, the project will no longer be able to run jobs. At this point, the project might determine that it needs more time on the system. A designated PI of the project can request more time for the project at this point that will require approval from the LCRC Allocations Committee. The request approval and decision will be transmitted via email to the PI after completion of the web form.
Project PIs and User Accounts
Each project has at least one Primary Investigator, or PI. The PI is generally the person who requested that the project be created, and is the first point of contact for things like allocation decisions. The primary project PI must be a current Argonne employee. A project can have Proxies (or Co-PIs), who also has the ability to manage the project like the PI. Proxies also need to be current Argonne employees.
A project can have one or more user accounts associated with it. Some projects may only have one user account (the PI), while others may have dozens. A user account can be associated with more than one project, and can also be the PI on more than one project.
The PI is responsible for adding and removing user accounts from projects.
The PI is responsible for administering the allocation associated with the project, i.e. describing who on the project can use how much of the allocation.
Startup Projects and Allocations
In order to help promote the use of the system and to foster new projects, all new user accounts belonging to active Argonne employees are given a “startup project” on Bebop only. The startup project can be used by someone getting familiar with the system, planning a new project, as “breeding ground” for new project ideas, or whatever the user wishes.
The startup project has a 20,000 hour allocation. The startup projects will not be allocated more time beyond the initial 20,000 hours. The 20,000 hours will never expire, i.e. it does not have to be used in the first year.
Startup projects should be used only by the account originally associated with that project.
It is expected that anyone wishing to use the system for more than 20,000 hours will either request their own project or join someone else’s project. Non-Argonne employees will always need to be a member of another active project belonging to an Argonne PI. Your startup project will be set as your default account/project in Slurm. If you change your default project, you can also submit jobs to your startup allocations with the account name of: startup-<username>
Request a New LCRC Project
If you are an Argonne employee, you can request a new project in LCRC with an allocation as needed. Simply login to the LCRC Accounts page with your Argonne credentials and on the left hand sidebar, click on Request New LCRC Project link.
From here, fill out all of the required fields and hit the Request project button when finished. The project will go through an approval cycle and you will be emailed with a decision. Once the project is created, you will be able to add and remove users and Proxies (Co-PIs), request additional allocations and edit project data. A unix group of the same name will be applied to your project directory for membership access. Lastly, any request of over 1TB of storage will require a justification.
Join an Existing LCRC Project
Whether you have a new or existing LCRC account, you may need to join an existing project. You may already have a project PI contact in LCRC with a project you are intending to join. If so, you can contact them to have them add you to their project if they have not already done so for you. If you would like to generate a request to the PI of a project or you know of a project you should have access to, you can request to join an existing project by visiting the LCRC Accounts page. Login with your Argonne credentials and click the Join Project button the left hand side bar.
You should now search for the project name you intend to join.
Below the search box, the list of projects will update to show only the project named you’ve indicated. Click on the project name below followed by the Request Membership button.
Once the project owner approves your request to join the project, you should be able to use the project hours with your jobs.
Manage an LCRC Project
Project Owners (PIs) and Proxies (Co-PIs) can manage their project(s) at the LCRC Accounts page. To do so, login with your Argonne credentials. On the left hand side bar, you will see projects you can manage under the Projects > Owned menu item. Click on the name of the project you intend to edit. From here, you can easily:
- Add/Remove project members manually
- Approve/Reject any pending membership requests
- Add/Remove Proxies (Co-PIs)
- Request an additional project allocation
- Request additional project storage
- Edit project information
Modifying these fields should be straightforward and once you make any changes, simply press the Save Project info button to finish the request/apply any changes. Questions can be sent to firstname.lastname@example.org
Allocation Usage and Tracking
General policies concerning how allocations are used and tracked:
- If a system error occurs that causes a program to crash while it is running, a project won’t be charged for that time. (This policy may be amended in the future in order to promote the use of user-based checkpointing.) The scheduler may or may not deduct the time used from the project’s allocation, depending on how the crash took place. If someone thinks their project should be credited time because of a system crash or other system problem, they should send email to email@example.com to get that time back into the project.
- If a project runs out of allocation time during a run, that run will be allowed to continue to completion.
- If a project runs out of allocation time, all users on the project will not be able to run any jobs until time is added again. At the present time, no steps will be taken to stop any jobs associated with that project from running and sitting idle in the job queue.
- PIs and Users can track their project’s usage by running the command mentioned in the next section of this page.
These policies are intended to keep things flexible in order to support the best overall scientific use of the system.
Project Allocation Queries and Management
The below commands will allow LCRC users to query their project balances, project transactions, default projects and allow you to change your default project. These commands are cluster specific and will only present the information for the current cluster you are on. You must also set your default project individually on each cluster.
|Command (Bebop & Blues)||Description|
|lcrc-sbank -q balance||Query all of your LCRC project balances.|
|lcrc-sbank -q balance <proj_name>||Query a specific LCRC project balance.|
|lcrc-sbank -q default||Query your default LCRC project.|
|lcrc-sbank -s default <proj_name>||Change your default LCRC project.|
lcrc-sbank -q trans <proj_name>
|Query all transactions on an LCRC project.|
|Print the help menu for the lcrc-sbank command.|
Requesting Additional Project Time
Occasionally, some LCRC projects might use up their quarterly time allocation well before the start of their next quarterly allocation. This can happen if the computations require more time than anticipated or if more case studies need to be completed. Under such circumstances, the project PIs can request additional time with proper justification at the LCRC Accounts page via the project management screens.
The LCRC core team meets every Wednesday and reviews all time requests for both new and existing projects. The LCRC core team can grant additional time up to 150K core-hours with requests above this needing approval from the LCRC allocations committee. PIs are reminded that any unused additional time will also expire at the end of the quarter. For instance, if a PI requested 100K core-hours for the first quarter and used only 40K core-hours, 60K core-hours would be lost at the end of the quarter.
The LCRC Account page is also used for requesting time for the entire next fiscal year after the first week of September. Quarterly allocations can still be made after the first quarter begins.