The Farm
Versatile High-Performance Computing at the Sanger
The Farm hosts multiple HPC clusters at the Wellcome Sanger Institute, these are well-equipped and tailored to the science requirements of the research programmes at the institute and across the world.
Our system hosts over 20,000 CPU and 275 total GPU, connected with ultra-fast 400Gb/s networking, empowering a diverse range of workflows in science ranging from AI image processing to large de novo genome assemblies.
This page is your guide to the services and infrastructure we provide. The information displayed can be used for research grant applications.
Cluster Structure¶
All of the farms run on Ubuntu 22.04 Linux, and schedule jobs with IBM Spectrum LSF.
The main farm is farm22 with 165 total execution hosts including CPU and GPU
hosts. Some of these are reserved for specific purposes (e.g. the transfer machines
which are used for the transfer of large data between file systems and users).
Access to all compute farms (except a small test cluster for use in training - gen22)
is restricted to users who have attended a farm users' course,
delivered by our RSE and RTP team (see dedicated page on RSE and RTP).
We list the general resources available to all farm users here, emphasising that the list is not exhaustive since some resources are restricted to individual collaborations and research programs.
CPU Resources¶
There are 122 general purpose CPU hosts available on the main Sanger farm, these are listed below:
| Host name | Machines | Slots per machine | CPU type | Memory | Architecture | Operating System | LSF model type |
|---|---|---|---|---|---|---|---|
node-{5,10,11,12}-\* |
78 | 64 | 64x Intel Xeon Gold 6226R @ 2.90GHz | 750GB | X86-64 | Linux (64bit) | Intel_Platinum |
node-{13,14}-\* |
42 | 256 | 64x AMD EPYC 7713 64-Core | 1.9TB | X86-64 | Linux (64bit) | EPYC7713 |
b{11,12}-hm-{01,02} |
2 | 256 | 64x AMD EPYC 7713 64-Core | 3.9TB | X86-64 | Linux (64bit) | EPYC7713 |
The resources are arranged in queues to suit worklflows based on runtime and
resource requirements, as well as specific processes (as with the transfer queue
mentioned previously). The data in the table is not exhaustive, as there are more
CPU clusters at Sanger, including specific R&D systems.
GPU Resources¶
The GPU resources on the Sanger farms are similarly split by model. The common hosts accessible by all farm users are Nvidia-systems, and are comprised of the following GPU models:
| GPU Model | Total GPU across hosts |
|---|---|
| NVIDIA A100-SXM4-80GB | 56 |
| NVIDIA GH200 | 4 |
| NVIDIA H100 80GB HBM3 | 64 |
| NVIDIA H200 | 80 |
| NVIDIA L40S 48GB | 40 |
| Tesla T4 | 1 |
| Tesla V100-SXM2-16GB | 8 |
| Tesla V100-SXM2-32GB | 22 |
These are arranged in a variety of standard configurations in individual hosts, meaning the 275 GPU resources are installed in the cluster as part of multi-GPU hosts: - 2x Nvidia V100-32GB - 8x Nvidia A100-80GB - 4x Nvidia H100-80GB - 4x Nvidia A100-80GB - 8x Nvidia A100-80GB - 6x Nvidia V100-32GB - 8x Nvidia V100-16GB
The table and list are not exhaustive, and excludes resources for specific research programmes and collaborations, as well as R&D systems from other vendors.
Internal JupyterHub Service¶
We offer a JupyterHub interface to the HPC which enables scientists to hit the ground running and profit from the diversity of resource offered at the Sanger in a lightweight and accessible way.
The Jupyter server is hosted internally and supports bash, python, and R
sessions.
Documentation¶
The documentation for the farm, including useful how-to guides and tutorials, is hosted on the Sanger-internal confluence platform and maintained by the RSE and RTP team.