System Details

College of Science and Engineering

Compute Nodes

Compute (8x)

  • CPU: 2x Intel Xeon E5-2620

    • 2GHz, 2.5Ghz Max Turbo

    • 6-Core, HT

    • 15MB Cache

    • 32nm

    • 95W

  • RAM: 64GB

    • 8 x 8GB DDR3-1600 ECC Registered 2R DIMMs

    • Operating at 1600 MT/s Max

    • Maximum 256GB (8 x 32GB)

  • Networking

    • 2x Gigabit Ethernet

      • Intel Dual-Port I350 Gigabit Ethernet Controller - Integrated

    • 1x IPMI Management

      • 2.0 & KVM over with Dedicated LAN - Integrated

  • Storage

    • System (Hot Swap)

      • 1 x 1TB Seagate Constellation ES (6Gb/s, 7.2K RPM, 64MB Cache) 3.5” SATA

  • Priority

    • 4 are open / none

    • 4 are Kodner research group

  • HTCondor Provisioning

    • Partitionable (Dynamic)

    • Max: 24 cpus, ~64GB memory

    • ParallelSchedulingGroup: “no_ib”

Compute (8x)

  • CPU: 2x Intel Xeon E5-2620

    • 2GHz, 2.5Ghz Max Turbo

    • 6-Core, HT

    • 15MB Cache

    • 32nm

    • 95W

  • RAM: 64GB

    • 8 x 8GB DDR3-1600 ECC Registered 2R DIMMs

    • Operating at 1600 MT/s Max

    • Maximum 256GB (8 x 32GB)

  • Networking

    • 2x Gigabit Ethernet

      • Intel Dual-Port I350 Gigabit Ethernet Controller - Integrated

    • 1x IPMI Management

      • 2.0 & KVM over with Dedicated LAN - Integrated

    • 1x InfiniBand

      • ConnectX-3 QDR 40Gb/s Controller with QSFP Connector

  • Storage

    • System (Hot Swap)

      • 1 x 1TB Seagate Constellation ES (6Gb/s, 7.2K RPM, 64MB Cache) 3.5” SATA

  • Notes

    • Same as above, but with added 40Gb/s InfiniBand

  • Priority

    • Parallel jobs (but open to all groups)

  • HTCondor Provisioning

    • Partitionable (Dynamic)

    • Max: 24 cpus, ~64GB memory

    • ParallelSchedulingGroup: “ib40_1”

Compute (4x)

  • CPU: 2x Intel Xeon E5-2620 v4

    • 2.1GHz Base, 3.0Ghz Max Turbo

    • 8-Core, HT

    • 20MB Cache

    • 14nm

    • 85W

  • RAM: 128GB

    • 8 x 16GB DDR4-2400 ECC Registered 2R DIMMs

    • Operating at 2400 MT/s Max

    • Maximum 1TB (8 x 256GB)

  • Networking

    • 2x Gigabit Ethernet

      • Intel Dual-Port I350 Gigabit Ethernet Controller - Integrated

      • 1x IPMI Management

        • 2.0 & KVM over with Dedicated LAN - Integrated

      • 1x Infiniband

        • ConnectX-3 QDR 40Gb/s Controller with QSFP Connector

  • Storage

    • System (Hot Swap)

      • 1 x 400GB Intel SSD SATA 6Gb/s 2.5”

  • Priority

    • Berger research group

  • HTCondor Provisioning

    • Partitionable (Dynamic)

    • Max: 32 cpus, ~128GB memory

    • ParallelSchedulingGroup: “ib40_1”

Graphics Nodes

GPU (2x)

  • CPU: 2x Intel Xeon E5-2620

    • 2GHz, 2.5Ghz Max Turbo

    • 6-Core, HT

    • 15MB Cache

    • 32nm

    • 95W

  • RAM: 128GB

    • 8 x 16GB DDR3-1600 ECC Registered 2R DIMMs

    • Operating at 1600 MT/s Max

    • Maximum 128GB (8 x 16GB)

  • Networking

    • 2x Gigabit Ethernet

      • Intel Dual-Port I350 Gigabit Ethernet Controller - Integrated

    • 1x InfiniBand

      • Mellanox ConnectX-2 VPI Single-Port QDR 40Gb/s InfiniBand HCA, PCIe 2.0 x8

  • Storage

    • System (Hot Swap)

      • 1 x 1TB Seagate Constellation ES (6Gb/s, 7.2K RPM, 64MB Cache) 3.5” SATA

  • GPU: 3x Tesla M2075 GPUs

    • Architecture: Fermi

    • CUDA Cores: 448

    • Memory: 6 GB GDDR5

    • Clock Speed: 574 Mhz

  • Priority

    • Open / None

  • HTCondor Provisioning

    • Partitionable (Dynamic)

    • Max: 24 cpus, ~128GB memory

    • ParallelSchedulingGroup: “ib40_1”

GPU (1x)

  • CPU: 1x AMD EPYC™ 7742

    • 2.25GHz, 3.4GHz Max Boost Clock

    • 64-Core, 128 Threads

    • 256MB L3 Cache

    • 14nm

    • 225W

  • RAM: 512GB

    • 8 x 64GB SAMSUNG 3200MHZ ECC

    • Operating at 1600 MT/s Max

    • Maximum 1TB (8 x 128GB)

  • Networking

    • 2x 10GBase-T Ethernet

      • Dual 10GBase-T LAN with via Broadcom BCM57416

      • Operating in 1 Gigabit mode

    • 1x InfiniBand

      • Mellanox ConnectX-5 VPI Single-Port EDR 100Gb/s InfiniBand QSFP28, PCIe 3.0 x16

  • Storage

    • System (Hot Swap)

      • 1 x 960GB Intel S4610 (6Gb/s, 3 DWPD) 2.5” SATA

  • GPU: 2x NVIDIA Tesla T4 GPU

    • Architecture: Turing

    • CUDA Cores: 2,560

    • Tensor Cores: 320

    • Memory: 16 GB GDDR6

    • Clock Speed: 585 MHz Base, 1590 MHz Max Boost

    • Bus: PCIe 3.0 x16

    • Performance

      • Single Precision Performance (FP32)

        • 8.1 TFLOPS

      • Mixed Precision (FP16/FP32)

        • 65 FP16 TFLOPS

      • INT8 Precision

        • 130 INT8 TOPS

      • INT4 Precision

        • 260 INT4 TOPS

  • Priority

    • Berger research group

  • HTCondor Provisioning

    • Partitionable (Dynamic)

    • Max: 128 cpus, ~512GB memory

    • ParallelSchedulingGroup: “ib40_1”

    • Required extra settings:

GPU (1x)

  • CPU: 2x AMD EPYC™ 7543 32-Core 2.80GHz

    • 2.80GHz, 3.7GHz Max Boost Clock

    • 32-Core, 64 Threads

    • 256MB L3 Cache

    • 7nm+

    • 225W

  • RAM: 512GB

    • 16 x 32GB SAMSUNG 3200MHZ ECC

    • Maximum 2TB (16 x 128GB)

  • Networking

    • 2x 10GBase-T Ethernet

      • Dual 10GBase-T LAN with via Intel® X550-AT2 controller

      • Operating in 1 Gigabit mode

    • 1x InfiniBand

      • Mellanox ConnectX-5 VPI Single-Port EDR 100Gb/s InfiniBand QSFP28, PCIe 3.0 x16

  • Storage

    • System (Hot Swap)

      • 1 x 960GB Intel S4610 (6Gb/s, 3 DWPD) 2.5” SATA

    • Scratch space

      • 1 x 960GB Intel S4610 (6Gb/s, 3 DWPD) 2.5” SATA

  • GPU: 1x NVIDIA A100 GPU

    • Architecture: Ampere

    • CUDA Cores: 6,912

    • Tensor Cores: 432

    • Memory: 40 GB HBM2

    • Clock Speed: 765 MHz Base, 1410 MHz Max Boost

    • Bus: PCIe 4.0 x16

    • Multi-Instance GPU: Up to 7 MIGs @ 5 GB

    • Performance

    • Double Precision Performance (FP64)

      • 9.7 TFLOPS

    • Double Precision Tensor Core Performance (FP64)

      • 19.5 TFLOPS

    • Single Precision Performance (FP32)

      • 19.5 TFLOPS

    • Tensor Float 32 (TF32)

      • 156 TFLOPS

    • BFLOAT16 Tensor Core

      • 312 TFLOPS

    • FP16 Tensor Core

      • 312 TFLOPS

    • INT8 Tensor Core

      • 624 TOPS

  • Priority

    • Ramasubramanian research group

  • HTCondor Provisioning

    • Partitionable (Dynamic)

    • Max: 64 cpus, ~512GB memory

    • ParallelSchedulingGroup: “no_ib”

    • Required extra settings:

Support Systems & Miscellaneous

Head Node (x1)

  • CPU: 2 x Intel Xeon E5-2620

    • 2GHz

    • 6-Core, HT

    • 15MB Cache

    • 32nm

    • 95W

  • RAM: 128GB

    • 16 x 8GB DDR3-1600 ECC Registered 2R DIMMs

    • Operating at 1600 MT/s Max

    • Maximum 512GB (16 x 32GB)

  • Networking

    • 2x Gigabit Ethernet

      • Intel Dual-Port I350 Gigabit Ethernet Controller - Integrated

    • 1x IPMI Management

      • 2.0 & KVM with Dedicated LAN - Integrated

    • 1x Infiniband

      • Mellanox ConnectX-2 VPI Single-Port QDR 40Gb/s InfiniBand HCA, PCIe 2.0 x8

  • Storage

    • System - RAID 1

      • 2 x 1TB OS drives in fixed bays

  • Usage

    • This system is used to submit and monitor jobs. No jobs should be run directly on this system.

    • This is the only system that is reachable directly. You will connect to this system via Secure Shell (SSH) and leverage HTCondor to submit and monitor your jobs. Please see the Getting Started Guide for a quick introduction.

Network

InfiniBand

  • 18 port QDR 40Gb/s InfiniBand

    • Currently 14 nodes are connected

    • The connected nodes are part of the ParallelSchedulingGroup “ib40_1”

Ethernet

  • 1x Cisco C9200L-48T-4X

Computer Science

Compute Nodes

Compute (8x)

  • CPU: 2 x Intel Xeon Gold 6130

    • 2.1GHz Base, 3.7GHz Max Turbo

    • 16-Core, 32 Thread

    • 22MB L3 Cache

    • 14nm

    • 125W

  • RAM: 192GB

    • 12 x 16GB DDR4 2666MHz ECC Registered DIMMs

  • Networking

    • 2x Gigabit Ethernet

      • Intel Dual-Port I350 Gigabit Ethernet Controller - Integrated

      • 1x InfiniBand

        • ConnectX-5 VPI EDR 100Gb/s Controller with QSFP28 Connector

  • Storage

    • 1x 960GB Micron 5200 PRO Series (6Gb/s SATA, 540 MB/s Read, 520 MB/s Write) 2.5” SATA

  • Priority

    • Open / None

  • HTCondor Provisioning

    • Partitionable (Dynamic)

    • Max: 64 cpus, ~192GB memory

    • ParallelSchedulingGroup: N/A

Graphics Nodes

GPU (3x)

  • CPU: 2x Intel Xeon Gold 6130

    • 2.1GHz Base, 3.7GHz Max Turbo

    • 16-Core, 32 Thread

    • 22MB L3 Cache

    • 14nm

    • 125W

  • RAM: 192GB

    • 12x 16GB DDR4 2666MHz ECC Registered DIMMs

  • Networking

    • 2x Gigabit Ethernet

      • Intel Dual-Port I350 Gigabit Ethernet Controller - Integrated

    • 1x InfiniBand

      • ConnectX-5 VPI EDR 100Gb/s Controller with QSFP28 Connector

  • Storage

    • 1x 960GB Micron 5200 PRO Series (6Gb/s SATA, 540 MB/s Read, 520 MB/s Write) 2.5” SATA

  • GPU: 4x Nvidia GeForce RTX 2080 TI

    • Architecture: Turing

    • CUDA Cores: 4,352

    • Memory: 11 GB GDDR6

    • Clock Speed: 1350-1545 Mhz

  • Priority

    • Open / None

  • HTCondor Provisioning

    • Partitionable (Dynamic)

    • Max: 64 cpus, 4 gpus, ~192GB memory

    • Must request at least 1 GPU

    • ParallelSchedulingGroup: N/A

GPU (2x)

  • CPU: 2x Intel Xeon Gold 5218T

    • 2.1GHz Base, 3.8GHz Max Turbo

    • 16-Core, 32 Thread

    • 22MB L3 Cache

    • 14nm

    • 105W

  • RAM: 384GB

    • 12x 32GB DDR4 2666MHz ECC Registered DIMMs

  • Networking

    • 2x 10-Gigabit Ethernet

      • Intel Dual-Port X722 10-Gigabit Ethernet Controller - Integrated (10Gbase-T)

    • 2x 10-Gigabit Ethernet

      • Intel Dual-Port X710 10-Gigabit Ethernet Controller (SFP+)

    • 1x InfiniBand

      • ConnectX-5 VPI EDR 100Gb/s Controller with QSFP28 Connector

  • Storage

    • 1x 984GB Intel SSD SATA 6Gb/s 2.5”

  • GPU: 8x Nvidia GeForce RTX 2080 TI

    • Architecture: Turing

    • CUDA Cores: 4,352

    • Memory: 11 GB GDDR6

    • Clock Speed: 1350-1545 Mhz

  • Priority

    • Open / None

  • HTCondor Provisioning

    • Static

    • Max: 8 cpus, 1 gpu, ~46GB memory

    • Must request at least 1 GPU

    • ParallelSchedulingGroup: N/A

GPU (1x)

  • CPU: 2x Intel Xeon Gold 6434

    • 3.7GHz Base, 4.10GHz Max Turbo

    • 8-Core, 16 Thread

    • 22.5MB L3 Cache

    • 10nm

    • 195W

  • RAM: 1TB

    • 16x 64GB PC5-38400 4800MHz DDR5 ECC Registered DIMMs

  • Networking

    • 2x Gigabit Ethernet

      • Intel® X710-AT2 Gigabit Ethernet Controller - Integrated

  • Storage

    • 1x 960GB Micron 5400 PRO Series (6Gb/s SATA, 540 MB/s Read, 520 MB/s Write) 2.5” SATA

    • 1x 3.2TB Micron 7450 MAX Series (6800 MB/s Read, 5300 MB/s Write ) U.3 PCIe 4.0 x4 NVMe Solid State Drive

  • GPU: 3x Nvidia H100 GPU

    • Architecture: Hopper

    • CUDA Cores: 14,592

    • Tensor Cores: 456

    • Memory: 80GB HBM2e

    • Clock Speed: 1065 MHz Base, 1755 MHz Max Boost

    • Bus: PCIe 5.0 x16

    • Multi-Instance GPU: Up to 7 MIGs

    • Performance

    • Double Precision Performance (FP64)

      • 25.6 TFLOPS

    • Double Precision Tensor Core Performance (FP64)

      • 51 TFLOPS

    • Single Precision Performance (FP32)

      • 51 TFLOPS

    • Tensor Float 32 (TF32)

      • 756 TFLOPS

    • BFLOAT16 Tensor Core

      • 1513 TFLOPS

    • FP16 Tensor Core

      • 1513 TFLOPS

    • FP8 Tensor Core

      • 3026 TFLOPS

    • INT8 Tensor Core

      • 3026 TOPS

  • Priority

    • Slot 1

      • Hutchinson research group

      • Idriss research group

      • Wehrwein research group

    • Slot 2

      • Open / None

  • HTCondor Provisioning

    • Slot 1

      • Partitionable (Dynamic)

      • Max: 16 cpus, 1 gpus, ~502GB memory

      • Max run time before preemption: 3 days

      • Required extra settings:

      • ParallelSchedulingGroup: N/A

    • Slot 2

      • Partitionable (Dynamic)

      • Max: 2 gpus

      • Max per requested GPU: 8 cpus, 257423MB (~251GB) memory

      • Max run time before preemption: 3 days

      • Required extra settings:

      • ParallelSchedulingGroup: N/A

Support Systems & Miscellaneous

Head Node (1x)

  • CPU: 2 x Intel Xeon E5-2620

    • 2GHz

    • 6-Core, HT

    • 15MB Cache

    • 32nm

    • 95W

  • RAM: 128GB

    • 16 x 8GB DDR3-1600 ECC Registered 2R DIMMs

    • Operating at 1600 MT/s Max

    • Maximum 512GB (16 x 32GB)

  • Networking

    • 2x Gigabit Ethernet

      • Intel Dual-Port I350 Gigabit Ethernet Controller - Integrated

    • 1x InfiniBand

      • Mellanox ConnectX-2 VPI Single-Port QDR 40Gb/s InfiniBand HCA, PCIe 2.0 x8

  • Storage

    • System

      • RAID 1

        • 2 x 1TB OS drives in fixed bays

  • Usage

    • This system is used to submit and monitor jobs. No jobs should be run directly on this system.

    • This is the only system that is reachable directly. You will connect to this system via Secure Shell (SSH) and leverage HTCondor to submit and monitor your jobs. Please see the Getting Started Guide for a quick introduction.

Network

InfiniBand

  • 36 port EDR 100Gb/s InfiniBand

    • All CSCI nodes except g-3-0 are connected via InfiniBand

    • There is no ParallelSchedulingGroup because they are all connected to the same switch

Ethernet

  • 4x Cisco C9200L-48T-4X

Shared Storage

Main Storage (1x)

  • CPU: 2x Intel Xeon Gold 6226R

    • 2.9GHz Base, 3.9Ghz Max Turbo

    • 16-Core, HT

    • 22MB L3 Cache

    • 150W

    • 14nm

  • RAM: 384GB

    • 12 x 32GB DDR4-2933 ECC Registered DIMMs

  • Networking

    • 2x 10-Gigabit Ethernet (Bonded / 802.3ad)

      • Intel 82599ES Dual-Port 10 Gigabit Ethernet Controller

  • Storage

    • 2x Boot Mirror

      • 480GB INTEL SSD

    • 18x RAID Z2 Storage

      • 16TB Seagate Exos X16 Hard Disk Drive - 7200 RPM Class SAS 12Gb/s 256MB Cache 3.5”

        • 134 TB usable after formatting with 3x RAID Z2 vdevs

Backup Storage (1x)

  • CPU: 2x Intel Xeon E5-2609

    • 2.4GHz Base

    • 4-Core, HT

    • 10MB L3 Cache

    • 80W

    • 32nm

  • RAM: 32GB

    • 8 x 4GB DDR3-1600 ECC Registered DIMMs

  • Networking

    • 2x Gigabit Ethernet

      • Intel I350 Dual-Port Ethernet Gigabit Ethernet Controller - Integrated

    • 2x 10-Gigabit Ethernet

      • Intel 82599ES Dual-Port 10 Gigabit Ethernet Controller

  • Storage

    • 2x Boot Mirror

      • 1TB Western Digital Black 7200 RPM 32MB Cache SATA 3.0Gb/s 3.5”

    • 18x RAID Z2 Storage

      • 16TB Seagate Exos X16 Hard Disk Drive - 7200 RPM Class SAS 12Gb/s 256MB Cache 3.5”

        • 134 TB usable after formatting with 3x RAID Z2 vdevs

  • Notes

    • This system takes daily snapshots from the main storage.