System Details

College of Science and Engineering

Compute Nodes

Compute (8x)

  • CPU: 2x Intel Xeon E5-2620

    • 2GHz, 2.5Ghz Max Turbo

    • 6-Core, HT

    • 15 MB Cache

    • 32 nm

    • 95 W

  • RAM: 64 GB

    • 8 x 8 GB DDR3-1600 ECC Registered 2R DIMMs

    • Operating at 1600 MT/s Max

    • Maximum 256 GB (8 x 32 GB)

  • Networking

    • 2x Gigabit Ethernet

      • Intel Dual-Port I350 Gigabit Ethernet Controller - Integrated

    • 1x IPMI Management

      • 2.0 & KVM over with Dedicated LAN - Integrated

  • Storage

    • System (Hot Swap)

      • 1 x 1 TB Seagate Constellation ES (6Gb/s, 7.2K RPM, 64 MB Cache) 3.5” SATA

  • Priority

    • 4 are open / none

    • 4 are Kodner research group

  • HTCondor Provisioning

    • Partitionable (Dynamic)

    • Max: 24 cpus, ~64 GB memory

    • ParallelSchedulingGroup: “no_ib”

Compute (8x)

  • CPU: 2x Intel Xeon E5-2620

    • 2GHz, 2.5Ghz Max Turbo

    • 6-Core, HT

    • 15 MB Cache

    • 32 nm

    • 95 W

  • RAM: 64 GB

    • 8 x 8 GB DDR3-1600 ECC Registered 2R DIMMs

    • Operating at 1600 MT/s Max

    • Maximum 256 GB (8 x 32 GB)

  • Networking

    • 2x Gigabit Ethernet

      • Intel Dual-Port I350 Gigabit Ethernet Controller - Integrated

    • 1x IPMI Management

      • 2.0 & KVM over with Dedicated LAN - Integrated

    • 1x InfiniBand

      • ConnectX-3 QDR 40Gb/s Controller with QSFP Connector

  • Storage

    • System (Hot Swap)

      • 1 x 1 TB Seagate Constellation ES (6Gb/s, 7.2K RPM, 64 MB Cache) 3.5” SATA

  • Notes

    • Same as above, but with added 40Gb/s InfiniBand

  • Priority

    • Parallel jobs (but open to all groups)

  • HTCondor Provisioning

    • Partitionable (Dynamic)

    • Max: 24 cpus, ~64 GB memory

    • ParallelSchedulingGroup: “ib40_1”

Compute (4x)

  • CPU: 2x Intel Xeon E5-2620 v4

    • 2.1GHz Base, 3.0Ghz Max Turbo

    • 8-Core, HT

    • 20 MB Cache

    • 14 nm

    • 85 W

  • RAM: 128 GB

    • 8 x 16 GB DDR4-2400 ECC Registered 2R DIMMs

    • Operating at 2400 MT/s Max

    • Maximum 1 TB (8 x 256 GB)

  • Networking

    • 2x Gigabit Ethernet

      • Intel Dual-Port I350 Gigabit Ethernet Controller - Integrated

      • 1x IPMI Management

        • 2.0 & KVM over with Dedicated LAN - Integrated

      • 1x Infiniband

        • ConnectX-3 QDR 40Gb/s Controller with QSFP Connector

  • Storage

    • System (Hot Swap)

      • 1 x 400 GB Intel SSD SATA 6Gb/s 2.5”

  • Priority

    • Berger research group

  • HTCondor Provisioning

    • Partitionable (Dynamic)

    • Max: 32 cpus, ~128 GB memory

    • ParallelSchedulingGroup: “ib40_1”

Graphics Nodes

GPU (2x)

  • CPU: 2x Intel Xeon E5-2620

    • 2GHz, 2.5Ghz Max Turbo

    • 6-Core, HT

    • 15 MB Cache

    • 32 nm

    • 95 W

  • RAM: 128 GB

    • 8 x 16 GB DDR3-1600 ECC Registered 2R DIMMs

    • Operating at 1600 MT/s Max

    • Maximum 128 GB (8 x 16 GB)

  • Networking

    • 2x Gigabit Ethernet

      • Intel Dual-Port I350 Gigabit Ethernet Controller - Integrated

    • 1x InfiniBand

      • Mellanox ConnectX-2 VPI Single-Port QDR 40Gb/s InfiniBand HCA, PCIe 2.0 x8

  • Storage

    • System (Hot Swap)

      • 1 x 1 TB Seagate Constellation ES (6Gb/s, 7.2K RPM, 64 MB Cache) 3.5” SATA

  • GPU: 3x Tesla M2075 GPUs

    • Architecture: Fermi

    • CUDA Cores: 448

    • Memory: 6 GB GDDR5

    • Clock Speed: 574 Mhz

  • Priority

    • Open / None

  • HTCondor Provisioning

    • Partitionable (Dynamic)

    • Max: 24 cpus, ~128 GB memory

    • ParallelSchedulingGroup: “ib40_1”

GPU (1x)

  • CPU: 1x AMD EPYC™ 7742

    • 2.25GHz, 3.4GHz Max Boost Clock

    • 64-Core, 128 Threads

    • 256 MB L3 Cache

    • 14 nm

    • 225 W

  • RAM: 512 GB

    • 8 x 64 GB SAMSUNG 3200MHZ ECC

    • Operating at 1600 MT/s Max

    • Maximum 1 TB (8 x 128 GB)

  • Networking

    • 2x 10GBase-T Ethernet

      • Dual 10GBase-T LAN with via Broadcom BCM57416

      • Operating in 1 Gigabit mode

    • 1x InfiniBand

      • Mellanox ConnectX-5 VPI Single-Port EDR 100Gb/s InfiniBand QSFP28, PCIe 3.0 x16

  • Storage

    • System (Hot Swap)

      • 1 x 960 GB Intel S4610 (6Gb/s, 3 DWPD) 2.5” SATA

  • GPU: 2x NVIDIA Tesla T4 GPU

    • Architecture: Turing

    • CUDA Cores: 2,560

    • Tensor Cores: 320

    • Memory: 16 GB GDDR6

    • Clock Speed: 585 MHz Base, 1590 MHz Max Boost

    • Bus: PCIe 3.0 x16

    • Performance

      • Single Precision Performance (FP32)

        • 8.1 TFLOPS

      • Mixed Precision (FP16/FP32)

        • 65 FP16 TFLOPS

      • INT8 Precision

        • 130 INT8 TOPS

      • INT4 Precision

        • 260 INT4 TOPS

  • Priority

    • Berger research group

  • HTCondor Provisioning

    • Partitionable (Dynamic)

    • Max: 128 cpus, ~512 GB memory

    • ParallelSchedulingGroup: “ib40_1”

    • Required extra settings:

GPU (1x)

  • CPU: 2x AMD EPYC™ 7543 32-Core 2.80GHz

    • 2.80GHz, 3.7GHz Max Boost Clock

    • 32-Core, 64 Threads

    • 256 MB L3 Cache

    • 7 nm+

    • 225 W

  • RAM: 512 GB

    • 16 x 32 GB SAMSUNG 3200MHZ ECC

    • Maximum 2 TB (16 x 128 GB)

  • Networking

    • 2x 10GBase-T Ethernet

      • Dual 10GBase-T LAN with via Intel® X550-AT2 controller

      • Operating in 1 Gigabit mode

    • 1x InfiniBand

      • Mellanox ConnectX-5 VPI Single-Port EDR 100Gb/s InfiniBand QSFP28, PCIe 3.0 x16

  • Storage

    • System (Hot Swap)

      • 1 x 960 GB Intel S4610 (6Gb/s, 3 DWPD) 2.5” SATA

    • Scratch space

      • 1 x 960 GB Intel S4610 (6Gb/s, 3 DWPD) 2.5” SATA

  • GPU: 1x NVIDIA A100 GPU

    • Architecture: Ampere

    • CUDA Cores: 6,912

    • Tensor Cores: 432

    • Memory: 40 GB HBM2

    • Clock Speed: 765 MHz Base, 1410 MHz Max Boost

    • Bus: PCIe 4.0 x16

    • Multi-Instance GPU: Up to 7 MIGs @ 5 GB

    • Performance

    • Double Precision Performance (FP64)

      • 9.7 TFLOPS

    • Double Precision Tensor Core Performance (FP64)

      • 19.5 TFLOPS

    • Single Precision Performance (FP32)

      • 19.5 TFLOPS

    • Tensor Float 32 (TF32)

      • 156 TFLOPS

    • BFLOAT16 Tensor Core

      • 312 TFLOPS

    • FP16 Tensor Core

      • 312 TFLOPS

    • INT8 Tensor Core

      • 624 TOPS

  • Priority

    • Ramasubramanian research group

  • HTCondor Provisioning

    • Partitionable (Dynamic)

    • Max: 64 cpus, ~512 GB memory

    • ParallelSchedulingGroup: “no_ib”

    • Required extra settings:

Support Systems & Miscellaneous

Head Node (x1)

  • CPU: 2 x Intel Xeon E5-2620

    • 2GHz

    • 6-Core, HT

    • 15 MB Cache

    • 32 nm

    • 95 W

  • RAM: 128 GB

    • 16 x 8 GB DDR3-1600 ECC Registered 2R DIMMs

    • Operating at 1600 MT/s Max

    • Maximum 512 GB (16 x 32 GB)

  • Networking

    • 2x Gigabit Ethernet

      • Intel Dual-Port I350 Gigabit Ethernet Controller - Integrated

    • 1x IPMI Management

      • 2.0 & KVM with Dedicated LAN - Integrated

    • 1x Infiniband

      • Mellanox ConnectX-2 VPI Single-Port QDR 40Gb/s InfiniBand HCA, PCIe 2.0 x8

  • Storage

    • System - RAID 1

      • 2 x 1 TB OS drives in fixed bays

  • Usage

    • This system is used to submit and monitor jobs. No jobs should be run directly on this system.

    • This is the only system that is reachable directly. You will connect to this system via Secure Shell (SSH) and leverage HTCondor to submit and monitor your jobs. Please see the Getting Started Guide for a quick introduction.

Network

InfiniBand

  • 18 port QDR 40Gb/s InfiniBand

    • Currently 14 nodes are connected

    • The connected nodes are part of the ParallelSchedulingGroup “ib40_1”

Ethernet

  • 1x Cisco C9200L-48T-4X

Computer Science

Compute Nodes

Compute (8x)

  • CPU: 2 x Intel Xeon Gold 6130

    • 2.1GHz Base, 3.7GHz Max Turbo

    • 16-Core, 32 Thread

    • 22 MB L3 Cache

    • 14 nm

    • 125 W

  • RAM: 192 GB

    • 12 x 16 GB DDR4 2666MHz ECC Registered DIMMs

  • Networking

    • 2x Gigabit Ethernet

      • Intel Dual-Port I350 Gigabit Ethernet Controller - Integrated

      • 1x InfiniBand

        • ConnectX-5 VPI EDR 100Gb/s Controller with QSFP28 Connector

  • Storage

    • 1x 960 GB Micron 5200 PRO Series (6Gb/s SATA, 540 MB/s Read, 520 MB/s Write) 2.5” SATA

  • Priority

    • Open / None

  • HTCondor Provisioning

    • Partitionable (Dynamic)

    • Max: 64 cpus, ~192 GB memory

    • ParallelSchedulingGroup: N/A

Graphics Nodes

GPU (3x)

  • CPU: 2x Intel Xeon Gold 6130

    • 2.1GHz Base, 3.7GHz Max Turbo

    • 16-Core, 32 Thread

    • 22 MB L3 Cache

    • 14 nm

    • 125 W

  • RAM: 192 GB

    • 12x 16 GB DDR4 2666MHz ECC Registered DIMMs

  • Networking

    • 2x Gigabit Ethernet

      • Intel Dual-Port I350 Gigabit Ethernet Controller - Integrated

    • 1x InfiniBand

      • ConnectX-5 VPI EDR 100Gb/s Controller with QSFP28 Connector

  • Storage

    • 1x 960 GB Micron 5200 PRO Series (6Gb/s SATA, 540 MB/s Read, 520 MB/s Write) 2.5” SATA

  • GPU: 4x Nvidia GeForce RTX 2080 TI

    • Architecture: Turing

    • CUDA Cores: 4,352

    • Memory: 11 GB GDDR6

    • Clock Speed: 1350-1545 Mhz

  • Priority

    • Open / None

  • HTCondor Provisioning

    • Partitionable (Dynamic)

    • Max: 64 cpus, 4 gpus, ~192 GB memory

    • Must request at least 1 GPU

    • ParallelSchedulingGroup: N/A

GPU (2x)

  • CPU: 2x Intel Xeon Gold 5218T

    • 2.1GHz Base, 3.8GHz Max Turbo

    • 16-Core, 32 Thread

    • 22 MB L3 Cache

    • 14 nm

    • 105 W

  • RAM: 384 GB

    • 12x 32 GB DDR4 2666MHz ECC Registered DIMMs

  • Networking

    • 2x 10-Gigabit Ethernet

      • Intel Dual-Port X722 10-Gigabit Ethernet Controller - Integrated (10Gbase-T)

    • 2x 10-Gigabit Ethernet

      • Intel Dual-Port X710 10-Gigabit Ethernet Controller (SFP+)

    • 1x InfiniBand

      • ConnectX-5 VPI EDR 100Gb/s Controller with QSFP28 Connector

  • Storage

    • 1x 984 GB Intel SSD SATA 6Gb/s 2.5”

  • GPU: 8x Nvidia GeForce RTX 2080 TI

    • Architecture: Turing

    • CUDA Cores: 4,352

    • Memory: 11 GB GDDR6

    • Clock Speed: 1350-1545 Mhz

  • Priority

    • Open / None

  • HTCondor Provisioning

    • Static

    • Max: 8 cpus, 1 gpu, ~46 GB memory

    • Must request at least 1 GPU

    • ParallelSchedulingGroup: N/A

GPU (1x)

  • CPU: 2x Intel Xeon Gold 6434

    • 3.7GHz Base, 4.10GHz Max Turbo

    • 8-Core, 16 Thread

    • 22.5 MB L3 Cache

    • 10 nm

    • 195 W

  • RAM: 1 TB

    • 16x 64 GB PC5-38400 4800MHz DDR5 ECC Registered DIMMs

  • Networking

    • 2x Gigabit Ethernet

      • Intel® X710-AT2 Gigabit Ethernet Controller - Integrated

  • Storage

    • 1x 960 GB Micron 5400 PRO Series (6Gb/s SATA, 540 MB/s Read, 520 MB/s Write) 2.5” SATA

    • 1x 3.2 TB Micron 7450 MAX Series (6800 MB/s Read, 5300 MB/s Write ) U.3 PCIe 4.0 x4 NVMe Solid State Drive

  • GPU: 3x Nvidia H100 GPU

    • Architecture: Hopper

    • CUDA Cores: 14,592

    • Tensor Cores: 456

    • Memory: 80 GB HBM2e

    • Clock Speed: 1065 MHz Base, 1755 MHz Max Boost

    • Bus: PCIe 5.0 x16

    • Multi-Instance GPU: Up to 7 MIGs

    • Performance

    • Double Precision Performance (FP64)

      • 25.6 TFLOPS

    • Double Precision Tensor Core Performance (FP64)

      • 51 TFLOPS

    • Single Precision Performance (FP32)

      • 51 TFLOPS

    • Tensor Float 32 (TF32)

      • 756 TFLOPS

    • BFLOAT16 Tensor Core

      • 1513 TFLOPS

    • FP16 Tensor Core

      • 1513 TFLOPS

    • FP8 Tensor Core

      • 3026 TFLOPS

    • INT8 Tensor Core

      • 3026 TOPS

  • Priority

    • Slot 1

      • Hutchinson research group

      • Idriss research group

      • Wehrwein research group

    • Slot 2

      • Open / None

  • HTCondor Provisioning

    • Slot 1

      • Partitionable (Dynamic)

      • Max: 16 cpus, 1 gpus, ~502 GB memory

      • Max run time before preemption: 3 days

      • Required extra settings:

      • ParallelSchedulingGroup: N/A

    • Slot 2

      • Partitionable (Dynamic)

      • Max: 2 gpus

      • Max per requested GPU: 8 cpus, 257423 MB (~251 GB) memory

      • Max run time before preemption: 3 days

      • Required extra settings:

      • ParallelSchedulingGroup: N/A

Support Systems & Miscellaneous

Head Node (1x)

  • CPU: 2 x Intel Xeon E5-2620

    • 2GHz

    • 6-Core, HT

    • 15 MB Cache

    • 32 nm

    • 95 W

  • RAM: 128 GB

    • 16 x 8 GB DDR3-1600 ECC Registered 2R DIMMs

    • Operating at 1600 MT/s Max

    • Maximum 512 GB (16 x 32 GB)

  • Networking

    • 2x Gigabit Ethernet

      • Intel Dual-Port I350 Gigabit Ethernet Controller - Integrated

    • 1x InfiniBand

      • Mellanox ConnectX-2 VPI Single-Port QDR 40Gb/s InfiniBand HCA, PCIe 2.0 x8

  • Storage

    • System

      • RAID 1

        • 2 x 1 TB OS drives in fixed bays

  • Usage

    • This system is used to submit and monitor jobs. No jobs should be run directly on this system.

    • This is the only system that is reachable directly. You will connect to this system via Secure Shell (SSH) and leverage HTCondor to submit and monitor your jobs. Please see the Getting Started Guide for a quick introduction.

Network

InfiniBand

  • 36 port EDR 100Gb/s InfiniBand

    • All CSCI nodes except g-3-0 are connected via InfiniBand

    • There is no ParallelSchedulingGroup because they are all connected to the same switch

Ethernet

  • 4x Cisco C9200L-48T-4X

Shared Storage

Main Storage (1x)

  • CPU: 2x Intel Xeon Gold 6226R

    • 2.9GHz Base, 3.9Ghz Max Turbo

    • 16-Core, HT

    • 22 MB L3 Cache

    • 150 W

    • 14 nm

  • RAM: 384 GB

    • 12 x 32 GB DDR4-2933 ECC Registered DIMMs

  • Networking

    • 2x 10-Gigabit Ethernet (Bonded / 802.3ad)

      • Intel 82599ES Dual-Port 10 Gigabit Ethernet Controller

  • Storage

    • 2x Boot Mirror

      • 480 GB INTEL SSD

    • 18x RAID Z2 Storage

      • 16 TB Seagate Exos X16 Hard Disk Drive - 7200 RPM Class SAS 12Gb/s 256 MB Cache 3.5”

        • 134 TB usable after formatting with 3x RAID Z2 vdevs

Backup Storage (1x)

  • CPU: 2x Intel Xeon E5-2609

    • 2.4GHz Base

    • 4-Core, HT

    • 10 MB L3 Cache

    • 80 W

    • 32 nm

  • RAM: 32 GB

    • 8 x 4 GB DDR3-1600 ECC Registered DIMMs

  • Networking

    • 2x Gigabit Ethernet

      • Intel I350 Dual-Port Ethernet Gigabit Ethernet Controller - Integrated

    • 2x 10-Gigabit Ethernet

      • Intel 82599ES Dual-Port 10 Gigabit Ethernet Controller

  • Storage

    • 2x Boot Mirror

      • 1 TB Western Digital Black 7200 RPM 32MB Cache SATA 3.0Gb/s 3.5”

    • 18x RAID Z2 Storage

      • 16 TB Seagate Exos X16 Hard Disk Drive - 7200 RPM Class SAS 12Gb/s 256 MB Cache 3.5”

        • 134 TB usable after formatting with 3x RAID Z2 vdevs

  • Notes

    • This system takes daily snapshots from the main storage.