Using Fusion
This area is meant to provide a basis and point of support for Fusion users to look at some commonly occurring questions. We hope that this may be a tool to enable you to get all the help you need in getting work done on Fusion without having to heavily rely on the support team for basic issues. This guide should enlighten users on the Fusion system as to some best practices that can be followed in order to have a pleasant experience with the system as well.
If you are looking for documentation on getting a specific program that has a softenv key working on Fusion please refer to the Installed software pages for more information on how to get the program working on Fusion.
Many issues that you may encounter are indeed not on this page as this is only a general overview and some very basic advice. A wide range of information is also available on the web. If for any reason you have an issue which you cannot resolve, you can always contact the LCRC Support Team via email at support@lcrc.anl.gov and we will be sure to help as quickly as possible. Your feedback is appreciated.
Accounts on Fusion
Any ANL employees may request an account on Fusion. To apply for an account, please fill out a Fusion Account Request. An initial starter allocation of computer hours will be provided. Information is available on the LCRC web pages for allocations, projects and related policies.
Non-ANL employees must be associated with an active Fusion project. More information on projects can be found on the Projects web page.
Login/SSH
SSH Client Software
SSH clients allow you to connect you computer to other computers or in this case with fusion our login servers. SSH is quite common today on many *nix like distributions and is considered a safe way to communicate and control a computer over a network. For some operating systems you may have to download a client in order to be able to use SSH.
- Linux:
Most Unix/Linux distributions come with ssh already installed so it is only necessary at this point to continue on a create your SSH Keys to be used for accessing Fusion.
If you want to double check that you have ssh working go into your terminal application and enter the following:
ssh -version
If it responds with nothing then you need to get OpenSSH. For most distributions this can be obtained from official repos with your package manager.
- Mac OS/X:
Openssh is installed with OS/X and can be invoked by opening up the terminal application and invoking the SSH commands as you would with any Unix/Linux distribution.
ssh -version
- Windows:
For accessing Fusion from a windows machine it is strongly suggested that you use PuTTY.
PuTTY
http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html
SSH Keys
To be able to login to Fusion you first need to create an ssh keypair to use for Fusion. Before continuing you need to decide what your passphrase will be. You passphrase should be a long memorable phrase that should be at least 12 characters or longer. Passphrases in this sense are not like passwords which are usually shorter in nature, this is to prevent brute force attempts on your keypair.
If you want to use another computer you will have to copy your private key to the new machine and setup ssh to use that key when connecting to Fusion. The key should be placed in a similar place as the one on your other machine unless you are on a windows machine. Please make sure that the file permissions are correct for the private key as well when you transfer it over.
The other safer and possibly easier option is to just create a new keypair for that computer to access Fusion. Once you create the keypair upload the public key to the LCRC Fusion accounts website and email LCRC Support to inform that that you would like to have that key put on Fusion as well.
Generating Keys
In order to login to the Fusion cluster you must setup a ssh keypair that will be used to authenticate you with the server. This is a much more secure method of access than a traditional password. If you really want to learn more about SSH Keys and how they work you can go here: http://www.ibm.com/developerworks/linux/library/l-keyc/index.html
- Generating SSH Keys on Linux
To generate a keypair on linux open up a terminal session and enter:
ssh-keygen -t rsa -C "fusion-username@fusion.lcrc.anl.gov"
After entering the previous command the terminal should then output the following:
Generating public/private rsa key pair.
Enter file in which to save the key (/Users/username/.ssh/fusionkeys):
At this time you can hit enter to let it put the keys in the default directory or a directory of your choosing.
NOTE: If you decide to give a different directory than the default directory it should be a fully qualified directory (e.g. /homes/username/directory and NOT ~/directory )
Hit enter after making a decision about where the keys will go at which point you will be prompted for a passphrase:
Enter passphrase (empty for no passphrase):
IMPORTANT: Enter a passphrase, otherwise we will not take your key.
After entering your passphrase it will prompt you again for that passphrase.
After entering in the passphrase for the second time you will get some output that looks something like this:
Your identification has been saved in id_rsa.
Your public key has been saved in id_rsa.pub.
The key fingerprint is:
3c:fb:bf:4b:71:13:dd:d5:36:0d:94:6a:c7:23:97:75 username@username.local
At this time you now can take the public key (in the examples case the contents of id_rsa.pub) and submit it on the MCS account page and then email LCRC support telling us to upload the recent public key to Fusion. After LCRC support installs the key you will be able to login to Fusion
Generating SSH Keys on Mac
To generate a keypair on mac open up the terminal app and enter:
ssh-keygen -t rsa -C "fusion-username@fusion.lcrc.anl.gov"
After entering the previous command the terminal should then output the following:
Generating public/private rsa key pair.
Enter file in which to save the key (/Users/username/.ssh/fusionkeys):
At this time you can hit enter to let it put the keys in the default directory or a directory of your choosing.
NOTE: If you decide to give a different directory than the default directory it should be a fully qualified directory (e.g. /homes/username/directory and NOT ~/directory )
Hit enter after making a decision about where the keys will go at which point you will be prompted for a passphrase:
Enter passphrase (empty for no passphrase):
IMPORTANT: Enter a passphrase, otherwise we will not take your key.
After entering your passphrase it will prompt you again for that passphrase.
After entering in the passphrase for the second time you will get some output that looks something like this:
Your identification has been saved in id_rsa.
Your public key has been saved in id_rsa.pub.
The key fingerprint is:
3c:fb:bf:4b:71:13:dd:d5:36:0d:94:6a:c7:23:97:75 username@username.local
At this time you now can take the public key (in the examples case the contents of id_rsa.pub) and submit it on the MCS account page and then email LCRC support telling us to upload the recent public key to Fusion. After LCRC support installs the key you will be able to login to Fusion
Generating SSH Keys on Windows
If you haven't already, first download a copy of PuTTY and PuTTYgen from PuTTY Official Download . PuTTY is the ssh client while PuTTYgen is a utility for generating SSH public key pairs. Next, you'll need to run PuTTYgen to create a new SSH Keypair.

Run PuTTYgen to generate a new SSH keypair. You should generate an "SSH2 RSA" key, and you should set the "number of bits" to 2048 or more. Make sure you enter a good passphrase when generating the key (something long with mixed case letters and numbers and/or some symbols but that you can remember and type easily).

Remember for this step to generate both the private and public keypair. For this next step only paste the public key. If you paste in your private key not only will logging in not work but you will have to regenerate a new keypair!
Copy and paste the new SSH public key into your LCRC Accounts webpage ; copy the public key from the top of the PuTTYgen window, open a web browser to https://accounts.lcrc.anl.gov/account.php , login using your LCRC username and password or your ANL username and password, scroll the web page down to the "SSH Public Key" section, paste the key you copied from PuTTYgen into the field, and finally click the "Update SSH Public Key" button on the web page. Again, see the instructions on the website mentioned above for more details, but note that you should go to the LCRC Accounts webpage.
After uploading your key, send us an email at support@lcrc.anl.gov letting us know that you've done so (you can reply to this email), and we'll install the key for you on fusion and reply back to you when you can try to log in.
Save your SSH key in PuTTYgen and configure PuTTY to use the new key. For this you simply need to select the profile you use to connect to Fusion, go to the left window and select Connection -> SSH - > Auth and then set the key that you want to use.

Once your key is saved and PuTTY is configured to use your key, and after we've confirmed with you that your key has been installed on fusion, then you can finally use PuTTY to log into fusion. Confirm that you are using the hostname "fusion.lcrc.anl.gov" and your LCRC username. If everything is configured correctly, PuTTY will prompt you for your key's passphrase (the one you entered when you generated the key with PuTTYgen) when you open a connection to fusion.
Logging in for the first time
If you followed all the steps preceding this and have SSH on your machine and a ssh keypair setup and confirmed to have the public key on the Fusion machine, then you are ready to finally login for the first time.
-
To login open your ssh client software of choice whatever it may be and do:
ssh fusion.lcrc.anl.gov
If your username on your local machine is different from your username on the Fusion cluster, you will need specify the Fusion username within the ssh command. This can be done two ways, by using the -l flag or by prepending your username to the hostname.
Using the -l flag:
ssh -l fusion_username fusion.lcrc.anl.gov
Prepending your username to the hostname is done with the @ symbol:
ssh fusion_username@fusion.lcrc.anl.gov
This format is particularly useful when copying files to and from Fusion.
Before being connected you might be prompted for your password if this is your first time logging into Fusion and or you don't have a keychain program setup.
The password that you would enter is the passphrase you used when first creating your ssh keypair. If you can't remember your passphrase then you will have to generate a new pair and upload
the new public key on the account page and email support to inform us to install your new key on Fusion.
Copying files to and from Fusion can be accomplished with the scp command. The format of the scp command is very similar to the format of using ssh:
scp localfile fusion.lcrc.anl.gov:path-to-remotefile
If you wish to copy an entire directory, use the -r option:
scp -r localdir fusion.lcrc.anl.gov:path-to-remotedir
Once again, if your local username differs from your Fusion username, you will need to specify the Fusion username within the scp command. The -l option will not work, so you must preprend the username:
scp localfile fusion_username@fusion.lcrc.anl.gov:path-to-remotefile
The method used to copy files to and from Fusion may need to take into account any firewalls between your local machine and the cluster. It is possible that there is a firewall that will prevent access to your local machine from the Fusion network. This can make copying files from Fusion to your local machine seem difficult. If you are logged onto one of the front-ends and you are unable to scp files to your local machine using the standard scp format, try pulling the Fusion files using scp on your local machine:
scp fusion.lcrc.anl.gov:path-to-remotefile localfile
Notice that you are simply swapping the local and remote filenames. The prepending of the username and use of the -r option will work with this format as well.
SSH X11 Forwarding
For some programs you must use X11 to most often view the results of the simulation you are trying to run on Fusion. Forwarding X11 sessions is fairly straightforward if you have X11 already installed on your machine.
-
In order to have X applications forwarded to your computer over SSH when you login you must specify the -X flag e.g.
ssh -X username@fusion.lcrc.anl.gov
Afterwards you should be able to display the X11 windows from fusion on your desktop given that you have X11 installed.
If for some reason it doesn't work enter the command:
echo $DISPLAY
If it is blank then try to reconnect with the -X flag. If it continues to return nothing then send the LCRC Support an email detailing the problem and any errors or details you can.
Login Shells
There are a few shells available on Fusion which you can pick from after logging in for the first time.
To view available shells enter in the command:
cat /etc/shells
Which will display a list like this:
/bin/sh
/bin/bash
/sbin/nologin
/bin/tcsh
/bin/csh
/bin/ksh
/bin/zsh
To view what your currently selected shell is enter the command:
echo $SHELL
To set a new shell enter the command:
chsh -s [path to shell from display command] [username]
Whatever you do, do not set your shell to /sbin/nologin unless you do not want to login to Fusion again.
From then on you should be using the shell of your choice and can go back and select another shell at anytime using the steps above. Do note that some environment variable conventions might be different from shell to shell so make sure to check your scripts and environment variables to account for any switch in your shell.
FIle Storage and Transfer Options
All fusion accounts get a home directory and access to the pvfs file system for use as scratch space on the compute nodes. While there is no quota on accounts, we appreciate it if you exercise diligence in keeping the amount of data stored in your home directory to a reasonable size given the work your doing on the system. Choosing the right location for your files, programs, etc. can have a serious impact on performance and the amount of difficulties you will experience.
The following table lists each of the three types, where you will find them mounted, the pros and cons of each and a discussion about how you might choose to use them.
- GPFS
- /home/username
- /fusion/gpfs/group/groupname
- Global Namespace
- Multi TB filesystem
- Large file support(>2GB)
- Backed up
- Raid protection
- Stable hardware
- Native Infiniband support
- Moderate performance
- /pvfs/username
- Parallel writes to same file
- Multi TB filesystem
- Large file support(>2GB)
- Performance
- Raid protection
- No cache issues
- Native Infiniband support
- Problems with many small files
- Not backed up
- May be cleaned at anytime
- Hardware is less stable
- /tmp
- /scratch
- Fast access
- Large file support(>2GB)
- Unique to each node; not shared between nodes
- GB filesystem
- Not backed up
- May be cleaned at anytime
- No raid protection
Your home directory is located on a GPFS filesystem that is shared by all nodes on the cluster. This filesystem is located on a raid array and is served by multiple fileservers. This provides both a performance increase and protection against the filesystem being inaccessible. If one server goes down, the other servers can continue to serve the filesystem.
We recommend that you keep all your programs and data in your home directory.
- Location
- PROS
- CONS
PVFS
If you will have multiple processes and/or multiple nodes writing to the same file(s), or need to write large scratch files during your job, then it is highly recommended to use PVFS for those file(s).
- Location
- PROS
- CONS
Local Disk
If you need a place to put temporary files that don't need to be accessed by other nodes, we recommend that you put them into one of these filesystems.
On the compute nodes, the $TMPDIR environment variable will be defined for your job script and will point to a directory created for you under /scratch on each node assigned to your job. $TMPDIR and its contents will be automatically removed at the end of the job.
- Location
- PROS
- CONS
File Transfer options
At some point you will probably want to transfer files off the fusion filesystems to somewhere else. For most small things, scp will do just fine. For larger transfers where data integrity and speed are important, for instance for large datasets and transfer of results, we would strongly advise you to take advantage of globus online.
Globus Online is a free online service that simplifies data movement -- with Globus Online, you hand-off data movement tasks to a hosted service that manages the entire operation: monitoring performance and errors, retrying failed transfers, correcting problems automatically whenever possible, and reporting status. Command line and web-based interfaces are available. The command line interface, which requires only ssh to be installed on the client, is the method of choice for grid-based workflows. Click here for more information or go here to sign up for an account.
Environment Variables and Management
How to use Softenv
Softenv is much like modules that some users maybe accustomed to from use of other clusters. In both cases these tools allow you to easily manage environmental variables enabling you to not have to worry about having to add things to your path, LDLIBRARY, or other necessary variables to use programs already installed on Fusion.
-
To see what is available through softenv simply invoke the command:
softenv
This will display all the different programs, compilers, and libraries available to be loaded with softenv.
To load a program, compiler, and or library with the command
soft add +namd-2.8
In this case it would load all necessary environmental variables related to NAMD version 2.8. At this point you should be able to invoke namd and start working with the software.
.soft environment file
In addition to being able to set softenv to load certain profiles on demand with the soft add command you can additionally setup a .soft file so that environment variables are always loaded when logging into Fusion. This can be extremely helpful for users that use mainly one program and stick to one set of compilers that are outside of the scope of those that are set as the default.
If you don't already have a .soft file in your home directory which you should you can create one and add @default to begin with. The @default allows you to access all basic system tools and provides use of our default compilers at the time for anyone logging into fusion. It is imperative that you do not comment out that line lest you want to have a hard time finding out where the basic system tools are. If you want to instead use compilers or programs outside of what is offered in the default key than you can add on to the file after the @default line. Subsequent lines will take precedence over older commands, so for instance as default MVAPICH2 is loaded for mpicc, but if you specify +openmpi1.4.3-intel11.1 on the line after then mpicc will be replaced with the path to OpenMPI and the CC, CXX, and F90 variables will be replaced with the path to the corresponding icc and intel fortran compiler as well.
Learning More
If you would like to learn more and take full advantage of all the benifits there are to managing your environmental variables effectively please review our extensive documentation on softenv.
Quotas
Looking at Project Quotas
Finding out how much time you have in your project quota is a relatively simple task and can found out with this command:
lcrc-qbank -q balance
Which should output something like below:
[10:20] user@flogin1~ lcrc-qbank -q balance
Project Balance (corehours)
------------------------- -------------------
*support 2923
startup-user 8000
For more information on using the lcrc-qbank utility to manage and set your default project on Fusion to deduct core hours from please see the Managing Allocations page.
Note on Reservations:
You can request to have a reservation of a certain amount of nodes for a time period, however you will be charged for the amount of core hours over the period of the reservation plus the hours while simulations on running. So if you ran a program on all the nodes for the entirety of the duration you would be charged at 2x the normal rate.
File Quotas
There are currently no file quotas on the cluster or the filesystems used across the cluster. We ask that you please be mindful and not write out to the cluster filesystems 90TB's of data without first contacting us so that we can make sure you can do it in a safe manner that doesn't adversely affect yourself or other users of Fusion.
Provided Applications
You can view the currently installed software here at this page. You can also get to this page by just clicking the Installed Software link in the left menu bar on every LCRC page. From there you can see which software is installed sorted by the types of program and can find out more about the software and how to use it on Fusion.
Compiling Programs
There are several options for what compilers, MPI compilers, and libraries to use when compiling your programs on the Fusion cluster. The most popular compilers can be found and set with softenv on the cluster with a set of default compilers which include intel C and Fortran Compilers and MVAPICH for MPI. For more information on how to take advantage of other compilers please see the section on softenv and management of environmental variables above or visit our extensive document on softenv.
Running Jobs on Fusion
Fusion utilizes Torque PBS manager and Maui PBS scheduler to allow users to submit jobs via PBS commands. The Portable Batch System (PBS) is a richly featured workload management system providing job scheduling and job management interface on computing resources, including Linux clusters. With PBS, a user requests resources and submits a job to a queue. The system will then take jobs from queues, allocate the necessary nodes, and execute them in as efficient a manner as it can.
Do NOT run large, long, multi-threaded, parallel, or CPU-intensive jobs on a front-end login host. All users share the front-end hosts, and running anything but the smallest test job will negatively impact everyone's ability to use Fusion. Always use PBS to submit your work as a job. You may even submit interactive sessions as jobs. This section of documentation will explain how to use PBS on Fusion.
Interactively
-
Interactive jobs can run on compute nodes. You can start interactive jobs either with specific time constraints (walltime=hh:mm:ss) or with the default time constraints of the queue to which you submit your job. PBS assigns to all jobs, even interactive jobs, the maximum wall time of their queue.
If you request an interactive job without a wall time option, PBS assigns to your job the default wall time limit for the queue to which you submit. If this is shorter than the time you actually need, your job will terminate before completion. If, on the other hand, this time is longer than what you actually need, you are effectively withholding computing resources from other users. For this reason, it is best to always pass a reasonable wall time value to PBS for interactive jobs.
Once your interactive job starts, you may use that connection as an interactive shell and invoke whatever other programs or other commands you wish. To submit an interactive job with one minute of wall time, use the -I option to qsub:
$ qsub -I -l walltime=00:01:00
waiting for job 100.fusion.lcrc.anl.gov to start
job 100.fusion.lcrc.anl.gov ready
If you need to use a remote X11 display from within your job , add the -v DISPLAY option to qsub as well:
$ qsub -I -l walltime=00:01:00 -v DISPLAY
waiting for job 101.fusion.lcrc.anl.gov to start
job 101.fusion.lcrc.anl.gov ready
To quit your interactive job:
logout
Scratch Files / Directories
For scratch directories you can either use the localized /tmp on each compute node or use the PVFS filesystem for cluster wide scratch space access. Location of your PVFS scratch space is at /pvfs/scratch/. Note that files left in either of these directories after completion of your job is subject to deletion and that all files you intend to keep and store should be placed in your home folder on the GPFS filesystem which is located at /home/ on all login and compute nodes of the cluster.
PBS scripts / submission files
Most users for their work will want to use pbs scripts and submission files to issue their jobs to the cluster to run. In order to submit a job in this method you will have to create a pbs script. A pbs script contains the parameters that you would usually issue with a qsub command.
#!/bin/sh
#PBS -N ufs
#PBS -l nodes=1:ppn=4
#PBS -l walltime=0:10:15
#PBS -j oe
export PATH=$PATH:/homes/wilshire/GFS/GSL/bin:/homes/wilshire/GFS/GTS/bin:/homes/wilshire/GFS/GERRIS/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/homes/wilshire/GFS/GSL/lib:/homes/wilshire/GFS/GTS/lib:/homes/wilshire/GFS/GERRIS/lib
export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/homes/wilshire/GFS/GSL/lib/pkgconfig:/homes/wilshire/GFS/GTS/lib/pkgconfig:/homes/wilshire/GFS/GERRIS/lib/pkgconfig
cd $PBS_O_WORKDIR
mpiexec /homes/wilshire/GFS/UFS/GFS-UFS3-snapshot/local/bin/gerris2D -m cyl_ufs-sfc.gfs
As you can see in the script, the first portion sets up variables needed to submit the job and have it distributed to machines assigned to you by the scheduler. In the next part you can see that you can also pass environment variables that you would like to be set for the job environment. Third portion of this script sets up immediate working directory and what program will be executing
Additionally while your pbsscript when it gets submitted will use what is defined in your .soft file that was discussed above in the Environments and Variables Management section, you can override this at job runtime. In order to do this you have to first identify what shell your using. So in the case of using a bash variant shell you can just add the following line to your job script:
source /etc/profile.d/z_softenv.sh
If you are using a C shell variant you would add the following line to your job script:
source /etc/profile.d/z_softenv.csh
After adding the correspondingly correct line, you can then invoke soft add commands to add whichever softenv key you need in order to override the defaults that are setup in your .soft file.
For more information regarding batch jobs please see visit the Submitting Batch Jobs page.
