Laboratory Computing Resource Center

This page describes projects in detail, including policies, concepts, background, etc.

The Project Life Cycle

Projects normally go through the following life cycle. Detailed descriptions of the life cycle will be elaborated further down on this page.

• The Principal Investigator (PI) requests a project using at https://accounts.lcrc.anl.gov. The PI must be a current Argonne employee.
• The project request is sent to the LCRC Allocations Administrator and to the LCRC Allocations Committee.
• The committee’s decision is reported via email to the project requester.
• If the project has been approved, the mail will include the number of hours allocated to the project. This number may be different from what was requested. This number may also be some initial allocation that will subsequently be augmented based on decisions by the LCRC Allocations Committee.
• As a part of creating the project, someone on the LCRC Allocations Committee will be affiliated with the project as a Point of Contact (PoC). That person will be the project’s contact on the committee. They are expected to become moderately familiar with the project, to act as the project’s advocate if necessary, and to be able to explain the project and the project’s status to the committee.
• Every fiscal quarter starting October 1, all projects will have their allocations zeroed out and be restarted with new allocations as requested and approved. This will be done in order to adjust and balance the use of the system. Unused time does not carry over to the next quarter.
• When making quarterly decisions, the Allocations Committee PoC will look at the usage for the previous quarter and may query any projects that have not been using most of their time to learn what their plan is for the next quarter. Sometimes they also query projects that have used a lot of time for their future plans.
• Once the project has been created, PIs can add other users to the project, appoint Proxies (or Co-PIs), and administer the project’s allocation.
• In advance of the annual October (Argonne’s new fiscal year) re-allocation, PIs will be reminded to send in a request for the next year’s allocation for their project.
• Also in advance of the annual October re-allocation, PIs will also be required to provide a report on their project at that time. If reports had been sent in for previous quarters, those will be sufficient, if they are current. The basic goal here is to get at least one annual report for each project. This report will be used to help describe the overall use of the systems and will also be used when deciding on annual allocations for all projects. The format of the report will be roughly 2-3 pages long, with pictures highly encouraged. Projects that do not submit a report will not receive any new time for the new fiscal year until a report is received.
• LCRC allocations granted each quarter use Argonne’s fiscal calendar. Allocations are granted on the first business day of the new quarter around 10AM CST. Quarters are divided into the following time frames:
  • 1st Quarter (October 1 – December 31)
  • 2nd Quarter (January 1 – March 31)
  • 3rd Quarter (April 1 – June 30)
  • 4th Quarter (July 1 – September 30)

Mid-Quarter Allocations

PIs are expected to plan their computations so that the allotted time lasts through the quarter. Limited mid-quarter allocations are done by LCRC. Mid-quarter allocations are reviewed once a week. Please follow the example provided in https://accounts.lcrc.anl.gov/sample_project_request.pdf (also shown in the link below the Request Allocation box on the accounts page) to show your core-hour computations.

For requests more than 100K core-hours, strong scaling results should be provided as shown in the example to justify the choice of the nodes/cores used in the core-hour computations. Allocation requests with incomplete information can result in up to two weeks delay in the decision of the PI’s allocation request.

Allocation requests should be made only after PIs have exhausted all their allocated resources. Please check your current quarter usage on LCRC using the command: lcrc-usage <project_name>

A maximum of 250K or half of the initial request (whichever is less) is pro-rated with time remaining in the quarter can be made once during the quarter. The mid-quarter allocation decision is made on basis of the current load on the machine and the overall usage by the PI among other factors. If your initial request for the quarter was for 600K, and you request an additional 500K 6 weeks into the quarter, the maximum time you would get depending on the load on the machine would be 125K (250*6/12). If your initial request was for 200K and you request an additional 300K after 7 weeks, the maximum additional time you might be granted is (200*0.5*5/12) or approximately 58K.

Additional allocations are not made two weeks before the end of the quarter since the load on the machine is usually high during the end of the quarter.

PIs who need renewal of their projects should apply for time at the start of the fiscal year. PIs who fail to renew their projects before the allocation deadline will be granted allocation the following quarter (Q2 onwards).

PIs who have large core-hour needs or have a high core-hour burn rate should apply for additional time on ALCF via the Director’s Discretionary Allocation Program.

An Example Project

As an example for how projects work, consider a project named popcorn, which might have a project title of “Simulation of Popcorn Kernel Dynamics”.

Following the creation of the project during the Project Life Cycle, the project would get an allocation, perhaps of 20,000 initial hours for the year, and might have a PI and 3 other scientists collaborating on it.

If one of the scientists runs a job as part of the popcorn project for 10 hours across 100 8-core nodes, then that would use 8,000 hours of the total allocation, leaving 12,000 hours remaining.

Once those 12,000 hours are gone, the project will no longer be able to run jobs. At this point, the project might determine that it needs more time on the system. A designated PI of the project can request more time for the project at this point that will require approval from the LCRC Allocations Committee. The request approval and decision will be transmitted via email to the PI after completion of the web form.

For a more detailed, real-life project request example, please see:
Sample Project Request – https://accounts.lcrc.anl.gov/sample_project_request.pdf

Project PIs and User Accounts

Each project has at least one Primary Investigator, or PI. The PI is generally the person who requested that the project be created, and is the first point of contact for things like allocation decisions. The primary project PI must be a current Argonne employee. A project can have Proxies (or Co-PIs), who also has the ability to manage the project like the PI. Proxies also need to be current Argonne employees.

A project can have one or more user accounts associated with it. Some projects may only have one user account (the PI), while others may have dozens. A user account can be associated with more than one project, and can also be the PI on more than one project.

The PI is responsible for adding and removing user accounts from projects.

The PI is responsible for administering the allocation associated with the project, i.e. describing who on the project can use how much of the allocation.

Startup Projects and Allocations

In order to help promote the use of the system and to foster new projects, all new user accounts belonging to active Argonne employees are given a “startup project” on Bebop and Swing. The startup project can be used by someone getting familiar with the system, planning a new project, as “breeding ground” for new project ideas, or whatever the user wishes.

The startup project has a 20,000 hour allocation. The startup projects will not be allocated more time beyond the initial 20,000 hours. The 20,000 hours will never expire, i.e. it does not have to be used in the first year.

Startup projects should be used only by the account originally associated with that project.

It is expected that anyone wishing to use the system for more than 20,000 hours will either request their own project or join someone else’s project. Non-Argonne employees will always need to be a member of another active project belonging to an Argonne PI. Your startup project will be set as your default account/project in Slurm. If you change your default project, you can also submit jobs to your startup allocations with the account name of: startup-<username>

Request a New LCRC Project

If you are an Argonne employee, you can request a new project in LCRC with an allocation as needed. Simply login to the LCRC Accounts page with your Argonne credentials and on the left hand sidebar, click on Request New LCRC Project link. If you do not see this link, please make sure to join the lcrc project first by following this linked guide.

From here, fill out all of the required fields and hit the Request project button when finished. The project will go through an approval cycle and you will be emailed with a decision. Once the project is created, you will be able to add and remove users and Proxies (Co-PIs), request additional allocations and edit project data. A unix group of the same name will be applied to your project directory for membership access. Lastly, any request of over 1TB of storage will require a justification.

Join an Existing LCRC Project

Whether you have a new or existing LCRC account, you may need to join an existing project. You may already have a project PI contact in LCRC with a project you are intending to join. If so, you can contact them to have them add you to their project if they have not already done so for you. If you would like to generate a request to the PI of a project or you know of a project you should have access to, you can request to join an existing project by visiting the LCRC Accounts page. Login with your Argonne credentials and click the Join Project button the left hand side bar.

You should now search for the project name you intend to join.

Below the search box, the list of projects will update to show only the project named you’ve indicated. Click on the project name below followed by the Request Membership button.

Once the project owner approves your request to join the project, you should be able to use the project hours with your jobs.

Manage an LCRC Project

Project Owners (PIs) and Proxies (Co-PIs) can manage their project(s) at the LCRC Accounts page. To do so, login with your Argonne credentials. On the left hand side bar, you will see projects you can manage under the Projects > Owned menu item. Click on the name of the project you intend to edit. From here, you can easily:

• Add/Remove project members manually
• Approve/Reject any pending membership requests
• Add/Remove Proxies (Co-PIs)
• Request an additional project allocation
• Request additional project storage
• Edit project information

Modifying these fields should be straightforward and once you make any changes, simply press the Save Project info button to finish the request/apply any changes. Questions can be sent to [email protected]

Allocation Usage and Tracking

General policies concerning how allocations are used and tracked:

• If a system error occurs that causes a program to crash while it is running, a project won’t be charged for that time. (This policy may be amended in the future in order to promote the use of user-based checkpointing.) The scheduler may or may not deduct the time used from the project’s allocation, depending on how the crash took place. If someone thinks their project should be credited time because of a system crash or other system problem, they should send email to [email protected] to get that time back into the project.
• If a project runs out of allocation time during a run, that run will be allowed to continue to completion.
• If a project runs out of allocation time, all users on the project will not be able to run any jobs until time is added again. At the present time, no steps will be taken to stop any jobs associated with that project from running and sitting idle in the job queue.
• PIs and Users can track their project’s usage by running the command mentioned in the next section of this page.

These policies are intended to keep things flexible in order to support the best overall scientific use of the system.

Project Allocation Queries and Management

The below commands will allow LCRC users to query their project balances, project transactions, default projects and allow you to change your default project. These commands are cluster specific and will only present the information for the current cluster you are on. You must also set your default project individually on each cluster.

Requesting Additional Project Time

Occasionally, some LCRC projects might use up their quarterly time allocation well before the start of their next quarterly allocation. This can happen if the computations require more time than anticipated or if more case studies need to be completed. Under such circumstances, the project PIs can request additional time with proper justification at the LCRC Accounts page via the project management screens.

The LCRC core team meets every Tuesday and reviews all time requests for both new and existing projects. The LCRC core team can grant additional time up to 150K core-hours with requests above this needing approval from the LCRC allocations committee. PIs are reminded that any unused additional time will also expire at the end of the quarter. For instance, if a PI requested 100K core-hours for the first quarter and used only 40K core-hours, 60K core-hours would be lost at the end of the quarter.

The LCRC Account page is also used for requesting time for the entire next fiscal year after the first week of September until October 1st for existing projects. New projects will always have the full year to request time for the remaining quarters. After October 1st, existing projects can only request time for the current quarter.