Spring 2018 Newsletter

Lots of news this time. Here are the items that follow:

  • KNL Jobs Now “Cost” Less
  • Login Node Etiquette
  • Shared Queue on Bebop
  • Perfect Your Image in Print
  • Jenkins Continuous Integration
  • Requesting More Time
  • Upcoming Changes to Blues
  • Tons of New and Updated Software

KNL Jobs Now “Cost” Less

Since Bebop arrived last summer, we have been benchmarking the Broadwell and KNL nodes, and collecting user experiences with their performance. The consensus is that a KNL core is not as productive as a Broadwell core for most LCRC codes. To compensate, starting April 1, 2018, a KNL core hour will cost 0.585 bank core-hours, while a Broadwell core hour stays at 1.0 bank core-hours. It now becomes advantageous to preferentially run optimized community codes such as VASP, Quantum Espresso, NAMD and LAMMPS on the KNLs. Notes on how to use these codes are available on the LCRC web site. We may make further adjustments in the future, based on more experience with KNL.

Login Node Etiquette

Login nodes are only meant for operations such as editing, copying, compilation and submitting jobs to the compute nodes. The login nodes are used by our whole community. Please do not run long jobs (> 10 minutes) on the login nodes as they slow down the response time of the node to other users. You can submit short jobs (< 1 hr) for testing/debugging on the shared queue (see below). MPI has been disabled on login nodes (due to misuse), and long jobs running on a login node will be killed by the system administrators.

Shared Queue on Bebop

LCRC is now offering a shared queue on Bebop. The new job queue is named “bdws.”.
The main purpose of this queue is for users to have a place for short term testing of their workflow. This queue is comprised of 8 Broadwell nodes with a maximum walltime of 1 hour per job.

Each node in this queue allows up to 4 processes per CPU. This means you can have up to 4 full node jobs on one single node at a time, or any variation in between. There is no user limit, just this CPU limit, so nodes may have several users running at once until the CPU limit is reached. Because of this, you should not expect to see the performance you would expect on the normal queues. Jobs will be scheduled on the least used nodes first in order to help with performance.

LCRC users can submit to as many of these nodes as they require per job. Unlike the other Bebop queues, this queue only charges for the cores used, as opposed to a whole node per job. If you only use 18 cores for 1 hour, this is how much will be charged to your project.

Perfect Your Image in Print

Please remember to acknowledge LCRC in your publications, e.g., “We gratefully acknowledge use of the Bebop [or Blues] cluster in the Laboratory Computing Resource Center at Argonne National Laboratory.” Yes, we sometimes take a look at publications from list reported by the PI in their annual project report.

Jenkins Continuous Integration

LCRC is now running a Jenkins continuous integration server. LCRC users can login with their LCRC account to: https://jenkins-ci.lcrc.anl.gov
Builds that run on this service do not get charged any time to user projects.

What is Jenkins?
Jenkins is a self-contained, open source automation server which can be used to automate all sorts of tasks related to building, testing, and delivering or deploying software. We currently have 1 build slave where builds/tasks get scheduled. This is a Broadwell node with the same specifications as our other Broadwell nodes. If you wish to use the LCRC software stack (via modules) in your job, please add this as the first step to your build process: source /etc/profile

Requesting More Time

PIs can request more time for their project at any point during the year via the project management web pages: https://accounts.lcrc.anl.gov/

When you request more time,
please provide a detailed account of how you came up with the requested amount. This should include the typical problem size (nodes/cores), the wall-time for a run, and total number of runs planned. These numbers should be based on runs that have already been conducted or projections based on previous experience on Bebop. Additional time will be for the current quarter on Bebop, unless you say otherwise. These requests are reviewed by the Allocations Committee in the context of the overall usage of the cluster. Smaller requests are handled by the LCRC staff.

Upcoming Changes to Blues

This Summer, we plan to upgrade the software on Blues to match Bebop, which will streamline operations and enable some new capabilities planned for Blues. This will involve changing Blues to CentOS 7.4, updating libraries and tools, and installing Modules and Slurm. We will provide details and guidance at a later date.

Tons of New and Updated Software

Intel-17.0.4 Intel c/c++/Fortran programming languages
GCC-7.1.10 Gnu c/c++ programming languages
perl-5.24.1 Programming language
python-2.7.14 Programming language
python-3.6.2 Programming language
singularity-2.4.2 HPC container technology
arpack-ng-3.5.0 Collection of Fortran77 subroutines designed to solve large scale eigenvalue problems</td
boost-1.66.0 Boost provides free peer-reviewed portable C++ source libraries
chemzone-2.2 Simplified but extremely fast solver for analysis of plasma discharges and neutral reactive flows such as in combustion or chemical vapor deposition
cmake-3.10.1 Open-source, cross-platform family of tools designed to build, test and package software
darshan-3.1.5 Scalable HPC I/O characterization tool
eigen-3.3.3 High-level C++ library of template headers for linear algebra, matrix and vector operations, geometrical transformations, numerical solvers and related algorithms
fftw-3.3.7 Software library for computing discrete Fourier transforms developed by Matteo Frigo and Steven G. Johnson at the Massachusetts Institute of Technology
gdbm-1.13 GNU dbm (or GDBM, for short) is a library of database functions that use extensible hashing and work similar to the standard UNIX dbm
gettext- Internationalization and localization system commonly used for writing multilingual programs on Unix-like computer operating systems
hdf5-1.10.1 Data model, library, and file format for storing and managing data
intel-mkl-2018.1.163 Intel Math Kernel Library
libint-2.4.2 Software stack for computing integrals used in molecular quantum mechanics
libpsm2-10.3-17 Intel opa-psm2 library
llvm-5.0.1 Collection of modular and reusable compiler and toolchain technologies
lz4-1.7.5 LZ4 – Extremely fast compression
mathematica-10.2.0 High-powered computation with thousands of Wolfram Language functions, natural language input, real-world data, mobile support
metis-5.1.0 Set of serial programs for partitioning graphs, partitioning finite element meshes, and producing fill reducing orderings for sparse matrices
mumps-5.1.1 Multifrontal Massively Parallel sparse direct Solver
noaa/wct-4.0.6 NOAA’s Weather and Climate Toolkit (WCT)
OpenFOAM-5.4.0 Open source software for CFD from the OpenFOAM Foundation
openmpi-1.10.7 A High Performance Message Passing Library
parallel-20170322 GNU parallel is a shell tool for executing jobs in parallel using one or more computers
parmetis-4.0.3 ParMETIS is an MPI-based parallel library that implements a variety of algorithms for partitioning unstructured graphs, meshes, and for computing fill-reducing orderings of sparse matrices
petsc-3.8.1 Suite of data structures and routines for the scalable (parallel) solution of scientific applications modeled by partial differential equations
pkgconf-1.4.0 Helps configure compiler and linker flags for development frameworks
py-appdirs-1.4.3 Module for determining appropriate platform-specific dirs
py-cython-0.25.2 C-Extensions for Python
py-h5py-2.7.1 A Pythonic interface to the HDF5 binary data format
py-mpi4py-3.0.0 Python Message Passing Library
py-numpy-1.13.3 NumPy is a library for the Python programming language, adding support for large, multi-dimensional arrays and matrices
py-packaging-16.8 Core utilities for Python packages
py-pyparsing-2.2.0 Alternative approach to creating and executing simple grammars, vs. the traditional lex/yacc approach, or the use of regular expressions
py-scipy-0.19.1 Python-based ecosystem of open-source software for mathematics, science, and engineering
py-setuptools-35.0.2 Fully-featured, actively-maintained, and stable library designed to facilitate packaging Python projects
py-six-1.10.0 Six is a Python 2 and 3 compatibility library
readline-7.0 Provides a set of functions for use by applications that allow users to edit command lines as they are typed in
rosetta-2017.52.59948 Library based object-oriented software suite which provides a robust system for predicting and designing protein structures, protein folding mechanisms, and protein-protein interactions
scotch-6.0.4 Software package and libraries for sequential and parallel graph partitioning, static mapping and clustering, sequential mesh and hypergraph partitioning, and sequential and parallel sparse matrix block ordering
sqlite-3.20.0 SQLite is a relational database management system contained in a C programming library
suite-sparse-5.1.0 Suite of sparse matrix software
SuiteSparse-4.5.6 Suite of sparse matrix software
superlu-5.2.1 SuperLU is a general purpose library for the direct solution of large, sparse, nonsymmetric systems of linear equations
superlu-dist-5.2.2 Distributed memory, MPI based SuperLU
sxdefectalign-2.2 The charge model
taskfarmer-0.2-1 Utility to farm tasks onto a compute node
tcl-8.6.6 Tool Command Language
tcpdx-1.4 TCP proxy utility
texlive-live TeX Live is an easy way to get up and running with the TeX document production system
tkdiff-4.2 A graphical diff viewer based on the Tk framework
visit-2.13.0 Interactive, scalable, visualization, animation and analysis tool