All computing carried out on LCRC clusters is associated with a specific “project”. Whenever a computing job runs on any computing node, the time the job uses will be counted and recorded as computing use by the associated project. A job must have a project in order to run on the computing nodes and will be assigned to your default project if none has been specified in your job script.
This web page describes projects in detail, including policies, concepts, background, etc. Valuable links that will again be mentioned below include:
As an example for how projects work, consider a project named popcorn, which might have a project title of “Simulation of Popcorn Kernel Dynamics”.
The project would have an allocation, perhaps of 20,000 initial hours for the year, and might have a PI and 3 other scientists collaborating on it.
If one of the scientists runs a job as part of the popcorn project for 10 hours across 100 8-core nodes, then that would use 8,000 hours of the total allocation, leaving 12,000 hours remaining.
Once those 12,000 hours are gone, the project will have a negative allocation balance. You can still run jobs once your balance is negative, but they will run at the lowest priority. Essentially, this means that your jobs won’t run unless there is nothing else waiting in the queue. At this point, the project might determine that it needs more time on the system, which it could request from the LCRC Allocations Committee.
Project PIs and Accounts
Each project has exactly one Primary Investigator, or PI. The PI is generally the person who requested that the project be created, and is the first point of contact for things like allocation decisions.
PIs must be current Argonne employees.
A project can have one or more accounts associated with it. Some projects may only have one account (the PI), while others may have dozens. An account can be associated with more than one project, and can also be the PI on more than one project.
The PI is responsible for adding and removing accounts from projects.
The PI is responsible for administering the allocation associated with the project, i.e. describing who on the project can use how much of the allocation. There are a set of tools available for this type of management available on the Managing Existing Projects page.
A project can have a co-PI, who also has the ability to manage the project like the PI. Co-PIs do not need to be current Argonne employees.
Startup Projects and Allocations
In order to help promote the use of the system and to foster new projects, all new accounts belonging to active Argonne employees are given a “startup project” on Bebop only. The startup project can be used by someone getting familiar with the system, planning a new project, as “breeding ground” for new project ideas, or whatever the user wishes.
The startup project has an 20,000 hour allocation. The startup projects will not be allocated more time beyond the initial 20,000 hours. The 20,000 hours will never expire, i.e. it does not have to be used in the first year.
Startup projects should be used only by the account originally associated with that project.
It is expected that anyone wishing to use the system for more than 20,000 hours will either request their own project or join someone else’s project. Non-Argonne employees will always need to be a member of another active project belonging to an Argonne PI.
The Project Life Cycle
Projects normally go through the following life cycle:
- The PI requests a project using this web page: Requesting a New Project
- The project request is sent to the LCRC Allocations Administrator and to the LCRC Allocations Committee.
- The committee’s decision is reported via email to the project requester.
- If the project has been approved, the mail will include the number of hours allocated to the project. This number may be different from what was requested. This number may also be some initial allocation that will subsequently be augmented based on decisions by the LCRC Allocations Committee.
- As a part of creating the project, someone on the LCRC Allocations Committee will be affiliated with the project. That person will be the project’s contact on the committee. They are expected to become moderately familiar with the project, to act as the project’s advocate if necessary, and to be able to explain the project and the project’s status to the committee.
- Once the project has been created, PIs can add other users to the project, appoint co-PIs, and administer the project’s allocation.
- PIs will be asked approximately quarterly to report any results they might have from using the systems to the LCRC Allocations Committee. These quarterly reports are only requests. Information from these reports will be used to help guide the allocation of the system.
- Every October (in conjunction with the fiscal year here at Argonne), all projects will have their allocations zeroed out and be restarted with new allocations. This will be done in order to annually adjust and balance the use of the system.
- In advance of the annual reallocation, PIs will be reminded to send in a request for the next year’s allocation for their project. PIs will also be required to provide a report on their project at that time. If reports had been sent in for previous quarters, those will be sufficient, if they are current. The basic goal here is to get at least one annual report for each project. This report will be used to help describe the overall use of the systems and will also be used when deciding on annual allocations for all projects. The format of the report will be roughly 2-3 pages long, with pictures highly encouraged.
Joining an Existing Project
Whether you have a new or existing LCRC account, you may need to join an existing project. You may already have a project PI contact in LCRC with a project you are intending to join. If so, you can contact them to have them add you to their project if they have not already done so for you. If you would like to generate a request to the PI of a project or you know of a project you should have access to, you can request to join an existing project by visiting here: Requesting to Join a Project. Once the project PI approves your request to join the project, you should be able to use the project hours with your jobs.
Allocation Usage and Tracking
General policies concerning how allocations are used and tracked:
- If a system error occurs that causes a program to crash while it is running, a project won’t be charged for that time. (This policy may be amended in the future in order to promote the use of user-based checkpointing.) The scheduler may or may not deduct the time used from the project’s allocation, depending on how the crash took place. If someone thinks their project should be credited time because of a system crash or other system problem, they should send email to email@example.com to get that time back into the project.
- If a project runs out of allocation time during a run, that run will be allowed to continue to completion.
- If a project runs out of allocation time, all users on the project will be notified. At the present time, no steps will be taken to stop any jobs associated with that project from running. Usage of the system will be tracked, and the project’s allocation will have a negative balance. That negative balance will be taken into account by the LCRC Allocations Committee when considering overall allocations on the system.
These policies are intended to keep things flexible in order to support the best overall scientific use of the system. However, if they are exploited unfairly, then we will put in restrictions. For example, we may consider requiring a job to have a positive project allocation balance in order to be launched.
Requesting Additional Time
In the event your project runs out of time for the quarter/half-year, you can request additional time for your project. Please find out more information about this procedure here.