Infrastructure Overview

This page provides a high-level overview of ABI's computing and network infrastructure. For user-facing documentation, see Cluster Basics.

ABI's infrastructure runs on FreeBSD physical hosts with compute workloads running in bhyve VMs. Services are isolated using FreeBSD jails.

                            ┌──────────────┐
                            │   Internet   │
                            └──────┬───────┘
                                   │
                            ┌──────┴───────┐
                            │   Firewall   │
                            │   / Router   │
                            └──────┬───────┘
                                   │
       ┌───────────────────────────┼──────────────────────────┐
       │                           │                          │
 ┌─────┴──────────────┐   ┌────────┴─────────────┐   ┌────────┴─────────────┐
 │  GEONOSIS (nas0)   │   │  BANE                │   │  MUSTAFAR (nas1)     │
 │  FreeBSD           │   │  FreeBSD             │   │  FreeBSD             │
 │                    │   │                      │   │                      │
 │  ZFS: genomic data │   │  Jails:              │   │  ZFS: home, user,    │
 │  /mnt/nas0/proj    │   │   - nginx            │   │  proj, db            │
 │  /mnt/nas0/user    │   │   - LDAP             │   │  /mnt/home           │
 │                    │   │   - Forgejo (git)    │   │  /mnt/nas1/proj      │
 │  bhyve VMs:        │   │   - PostgreSQL       │   │  /mnt/nas1/db        │
 │   - ssh-01 (2C/4G) │   │   - MySQL            │   │                      │
 │   - ssh-02 (4C/8G) │   │   - DNS              │   └──────────────────────┘
 │   - thin-01(64C/384G)  │   - DHCP             │
 │   - thin-02(64C/384G)  │                      │   ┌───────────────────────┐
 │   - thick-01(64C/768G) │  Devuan VM on bhyve: │   │  HOTH (bak1)          │
 │   - dl-01  (2C/8G) │   │   - slurmctld        │   │  FreeBSD              │
 │   - dl-02  (2C/8G) │   │                      │   │                       │
 │   - rshiny0 (4C/16G)   └──────────────────────┘   │  ZFS backups via      │
 │   - rshiny1 (4C/16G)                              │  zelta (send/recv)    │
 │   - (+ legacy VMs) │                              │  homes + some projects│
 └────────────────────┘                              └───────────────────────┘

ABI runs on 4 physical FreeBSD servers:

All HPC VMs run on geonosis using bhyve. The Slurm controller runs in a Devuan VM on bane.

These VMs are for internal IT use. *TODO: Document when needed.*

All infrastructure services run in FreeBSD jails on bane (managed via jailer), providing isolation and easy management.

For the full jail list including stopped jails, see Server Inventory: bane.

See Network for detailed network documentation.

Summary:

• Public subnet: 37.26.174.176/28 – internet-facing services (www-01, git-01, mx-01, ssh.abi.am)
• Internal subnet 1: 172.20.42.0/24 – core infrastructure services (LDAP, DNS, DHCP)
• Internal subnet 2: 172.20.200.0/24 – internal services (databases, monitoring, wiki, UniFi, AdGuard)
• DNS: internal DNS served by dns-01 jail on bane
• DHCP: served by dhcp-01 jail on bane
• WiFi management: UniFi controller on unifi-01 jail
• *TODO: VLAN segmentation details, switch configuration, inter-subnet routing.*

ABI uses two ZFS-based NAS servers with storage served over NFS. ZFS provides transparent compression, snapshots, and data integrity.

• FreeBSD everywhere for physical hosts – provides ZFS, bhyve, and jails natively.
• bhyve for virtualization – all HPC workloads run in VMs on geonosis, providing isolation and resource control.
• FreeBSD jails for services on bane – lightweight isolation for LDAP, web, git, databases, DNS, DHCP.
• ZFS for all storage – transparent compression, snapshots, send/recv for backups.
• zelta for backups – ZFS replication from mustafar/geonosis to hoth.
• No environment modules – all bioinformatics software installed globally on compute VMs.

• Server Inventory – Detailed specs per machine
• Changelog – Infrastructure change log
• Network – Network topology and configuration
• LDAP Configuration – User directory setup
• Monitoring – System health monitoring
• Automation – Configuration management
• Admin Tips & Tricks – Quick admin commands