Bebop

Quick Facts

  • 1024 public nodes
  • 128GB DDR4 (Intel Broadwell) / 96GB DDR4,16GB MCDRAM (Intel Knights Landing) of memory on each node
  • 36 cores (Intel Broadwell) / 64 cores (Intel Knights Landing) per compute node
  • Omni-Path Fabric Interconnect

Available Partitions

Bebop has several partitions defined, a partition is similar to a queue. Use the -p option with srun or sbatch to select a partition. The default partition is bdwall.


Bebop Partition Name Description Number of Nodes CPU Type Cores Per Node Memory Per Node Local Scratch Disk
bdwall All Broadwell Nodes 664 Intel Xeon E5-2695v4 36 128GB DDR4 15 GB or 4 TB
bdw Broadwell Nodes with 15 GB /scratch 600 Intel Xeon E5-2695v4 36 128GB DDR4 15 GB
bdwd Broadwell Nodes with 4 TB /scratch 64 Intel Xeon E5-2695v4 36 128GB DDR4 4 TB
bdws Broadwell Shared Nodes (Oversubscription / Non-Exclusive). 8 Intel Xeon E5-2695v4 36 128GB DDR4 15 GB
knlall All Knights Landing Nodes 352 Intel Xeon Phi 7230 64 96GB DDR4/16GB MCDRAM 15 GB or 4 TB
knl Knights Landing Nodes with 15GB /scratch 288 Intel Xeon Phi 7230 64 96GB DDR4/16GB MCDRAM 15 GB
knld Knights Landing Nodes with 4TB /scratch 64 Intel Xeon Phi 7230 64 96GB DDR4/16GB MCDRAM 4 TB

File Storage

There are no physical disks in most of the Bebop nodes themselves, and as such the OS running on every node runs in a diskless environment. Users that do take advantage of local scratch space currently on Blues will still have the option of using a scratch space on the node’s memory (15GB located at /scratch). A subset of our Broadwell and KNL nodes instead have a 4TB /scratch available. For details on which queues have which scratch space, please refer to the queue table above. The scratch space is essentially a RAM disk and consumes an amount of memory, so this should be taken into account if you are running a large job that requires a substantial amount of memory.

Please see our detailed description of the file storage used in LCRC here.

Architecture

Bebop runs mostly on Intel Broadwell and Knights Landing processors. Broadwell nodes can take advantage of using the AVX2 and AVX instruction sets while Knight Landing can use AVX-512, AVX2 and AVX.

Bebop is using an Intel Omni-Path interconnect for its network. This fact comes into play when considering MPI programs that would use Infiniband library as a means for communication. Omni-Path has its own communication method that only works with their gear (called PSM2), allowing for higher performance than you would see from using ibverbs or PSM. This means that you should recompile your code and use one of the MPI’s on Bebop which supports PSM2.

Running Jobs on Bebop

For detailed information on how to run jobs on Bebop, you can follow our documentation by clicking here: Running Jobs on Bebop.

Bebop utilizes the Slurm Workload Manager (formerly known as Simple Linux Utility for Resource Management or SLURM) for job management. Slurm is an open source, fault-tolerant, and highly scalable cluster management and job scheduling system for large and small Linux clusters.

As a cluster workload manager, Slurm has three key functions. First, it allocates exclusive and/or non-exclusive access to resources (compute nodes) to users for some duration of time so they can perform work. Second, it provides a framework for starting, executing, and monitoring work (normally a parallel job) on the set of allocated nodes. Finally, it arbitrates contention for resources by managing a queue of pending work.