EXO: End-to-End Local AI Cluster Deployment Training Course

Introduction to EXO and Local AI Clustering

• Overview of the EXO framework and the exo-explore ecosystem
• Comparison between centralized cloud inference and distributed local inference
• Architecture overview: libp2p device discovery, MLX backend, dashboard, and API layers
• Hardware requirements: Apple Silicon (M3 Ultra, M4 Pro/Max), Thunderbolt 5, and shared storage solutions

Installing EXO on macOS

• Setting up Xcode, the Metal Toolchain, and other macOS prerequisites
• Installing uv, Node.js, and the Rust nightly toolchain
• Installing the specific macmon fork required for Apple Silicon monitoring
• Cloning the repository and building the dashboard using npm
• Running EXO from source code and verifying access via localhost:52415 dashboard

Installing EXO on Linux

• Installing necessary dependencies via apt or Homebrew on Linux
• Configuring uv, Node.js (version 18+), and the Rust nightly toolchain
• Building the dashboard and initiating EXO in CPU-only mode
• Understanding directory layout: utilizing XDG Base Directory paths for configuration, data, cache, and logs

Automatic Device Discovery and Cluster Formation

• Understanding libp2p-based auto-discovery mechanisms across local networks
• Configuring custom namespaces using EXO_LIBP2P_NAMESPACE to ensure cluster isolation
• Verifying node membership within the dashboard's cluster view
• Addressing discovery failures and network segmentation issues

Enabling RDMA over Thunderbolt 5

• Overview of RDMA architecture and its claim to reduce latency by 99 percent
• Activating RDMA in macOS Recovery mode via rdma_ctl
• Cable specifications and port topology constraints specific to Mac Studio devices
• The necessity of matching macOS versions across all cluster nodes
• Troubleshooting RDMA discovery processes and DHCP configuration

Deploying Frontier Models

• Using the dashboard to load and shard models such as DeepSeek v3.1, Qwen3-235B, and various Llama family models
• Previewing instance placements through the /instance/previews API endpoint
• Creating model instances using pipeline or tensor-parallel sharding strategies
• Configuring custom model cards sourced from the HuggingFace hub

Monitoring and Troubleshooting

• Reading EXO logs and comprehending distributed tracing mechanisms
• Interpreting cluster health indicators within the dashboard's cluster view
• Diagnosing worker node failures and observing reconnection behaviors
• Utilizing EXO_TRACING_ENABLED for in-depth performance bottleneck analysis

Cluster Maintenance and Updates

• Procedures for updating EXO binaries and rebuilding the dashboard
• Migrating model caches and managing pre-downloaded models over NFS
• Gracefully removing nodes and rebalancing workloads across the cluster