NixOS Intel Ultra 9 185H K3s with SR-IOV GPU Passthrough

Featuring the Following Projects

• Cert Manager
• Cluster Plex
• Dashboard
• Longhorn
• Flame
• Grafana
• Intel SR-IOV Plugins
• Kubevirt
• Kyverno
• MariaDB
• PHPMyAdmin
• Portainer
• Prometheus
• ReviewBoard
• Traefik
• Unifi Controller
• Wordpress

Prerequisites

• 3x Intel 13th Gen Processors with the 185H or similar. I use BeeLink.
  • Will also work on ARC dGPUs with a little fiddling
• Seperate VLAN for your Kube Cluster

Installation

• Contained in the repo are the main configurations required to build out each host.

1. Build a ThumbDrive or PXE Boot NixOS 24.05
   • Complete steps through SWAP setup
2. sudo nix-channel --update
3. git clone https://github.com/celesrenata/nixos-k3s-configs/
4. sudo nixos-generate-config --root /mnt
5. cp -r nixos-k3s-configs/nixos-kube-config/gremlin-1/* /mnt/etc/nixos/
6. sudo nixos-generate-config --root /mnt
   • Yes, again
7. You may now edit your hardware-configuration.nix file to your liking
8. nixos-install --root /mnt
9. sudo nixos-enter
10. passwd celes
11. exit
12. reboot

Repeat for Gremlins 2 and 3

Login to and add your own ssh configs to your account, root, and nixremote accounts:

• Add your own authroized keys, you will need these as the fleet does not work without passwordless SSH!

Networking

1. Set your network to expect 10.1.1.12, 10.1.1.13, 10.1.1.14 for your Cluster

Configuring NFS

1. These configs are setup for my NFS server, you will have to edit all your PVC files to meet your needs
2. Leaving these details in have been way more useful than not demonstrating how to create truely persistant volumes

Ensuring Cluster is Happy

1. I have included automation scripts for resetting the fleet to known good states as well as scripts to deploy all the services I have figured out!
2. ./resetfleet.sh.gremlin

Edit the Cluster Deployments

Each script is controlled by a runmefirst.sh file in the directory of the service, and is stood up by the following automation script:

• ./runmefirst.sh.gremlin
• Edit this file to turn off deployments you do not desire for your Cluster

Ollama via IPEX

Ollama is controlled via the IPEX fleet within the kubevirt directory If you have more than 32GB of ram per Node you can then use Ipex-LLM Ollama! kubevirt/ipex/runmefirst.sh

TODO

Verify with 96GB of RAM that OneAPI properly functions within Ubuntu VMs.