13 minute read

Here I will provide an overview of my homelab journey, including some analysis as to why I made some decisions and the hurdles I overcame while setting up this homelab.


Inspiration

I initially started with a decommissioned SuperMicro server from my university, RIT, which contained a TYAN S7002-LE motherboard. After lugging the server around for a few years, I decided to utilize the hardware in an effective manor along with several other pieces of older hardware I had around.

Homelab 1.0

Rack

My first task was to setup a central place for all my hardware. As I was moving around a decent amount, I wanted to purchase an enclosed server rack with wheels. I came across the 18U NavePoint Rack on Facebook Market. The listing also included a 4U server chassis which supports an EATX motherboard, fitting the TYAN S7002-LE motherboard.

Primary Server

The TYAN S7002-LE came with two Xeon 5645 processors that I opted to keep using. I originally wanted to use the power supply from the original server chassis but it did not have enough proper power connecters to power the board and an old Radeon 5770 I had. I ended up getting a Corsair RM750x to be the primary power supply.

For heatsinks, I got two be quiet! Shadow Rock LP BK002 as the heatsinks from the original chassis were passively cooled from the chassis’s (loud) fans.

SSDs used were two Kingston 256GB used in a software RAID 1.

I ordered two 8GB DDR3 DIMMS for 16GB of memory off Ebay.

For the underlying hypervisor, I choose ProxMox 7.4. I ran a few VMs to better understand using Proxmox but I did not launch anything for a specific purpose.

Network

I utilized my existing Ubiquiti setup which included:

In my apartment at the time, I had Comcast (ew) as an ISP where I used a Motorola MB8600 as it supports DOCSIS 3.1 and I did not want to manage/deal with the standard Comcast modem.

Power

APC 1500VA Back-UPS Pro

Other gear

I had an old Mac Mini (2011) that had some stability issues with the third party memory but I’ve lugged it around since college so I figured I including is as a mac centric node.

Laptop Setup

My primary Laptop is a 2013 15” MacBook Pro (retina) which I often use in ‘clamshell’ mode hooked up to a 32” LG display. I would run this through a Belkin Thunderbolt Dock which offers a wired network connection, an additional Thunderbolt out which I used for display out, and a USB hub. I had my laptop on a pull out rack shelf for convince and had a longer thunderbolt cable and power cable ziptied to the shelf. The video out went to an HDMI switch which fed the 32” LG display I used.

Homelab 1.1

I got tired of reaching into the rack to get to my networking gear, such as reseating connections (stupid Comcast), or reconnecting USB devices. I decided to add a customizable patch panel into the the rack.

Patch Panel

I ordered this Elite Core RP1U-16D with open punch outs so I can place different connection types.

The panel is as follows:

(blank) HDMI HDMI (blank) USB USB USB (blank) (blank) (blank) Ethernet Ethernet Ethernet Ethernet Ethernet (blank)

I used the first HDMI as a ‘spare’ in and followed by the main out for the LG 32” monitor and for easier ad-hoc usage into the display. Each USB panel had 2 USB 3.0 headers which were connected using various length of these UGreen USB cables to various devices. The ethernet punch panels I connected with custom cut cables (yes, I make my own Ethernet cables. T-568B).

Pi-Hole

I got a raspberry pi 4 B+ to use as a pi-hole. As I wanted to keep things rack mounted I got this mount as it also came with several micro-HDMI to HDMI breakout boards. Likewise, as I had a PoE switch, I got this PoE Pi HAT to remove the need for an external power supply. In the Ubiquiti Network console, I assigned a ‘Fixed IP’ of 10.0.0.53 (as DNS is port 53) and configured DHCP to hand out this IP for the DNS resolver.

Hue Bridge

I have a Philips Hue Bridge and which I have configured to passthrough to HomeKit to control several lights. As I was reworking the rack and setting up the pi-hole, I wanted to try and power the bridge via PoE. I used a TP-Link TL-PoE10R PoE splitter along with a 2.1mm to 2.5mm barrel plug adapter. This allowed for the Hue Bridge to be powered by PoE and reduce the need for another wall-wart power supply in the rack.

Homelab 1.2

NAS

I had several harddrives with my data in several places. I wanted to consolidate. Also, I ran out of space in my TimeMachine Drive and had not backed up my laptop in over a year, so it was needed. I opted for a Synology DS220+ and a single WD Red Pro WD181KFGX 18TB as there was a great from Newegg on these. I got a second WD Red Pro WD181KFGX 18TB on a black Friday deal to keep the drives in a RAID 1 for data resiliency.

HomeLab 2.0

I had moved apartments and learned from some shortfalls in my design and decided an overhaul of the rack was in order.

One major shortfall was that the handles on the Primary Server chassis prevented the rack from being closed. I normally had the front panel off so this was not an everyday issue. I had to remove the server from the rack in order for the rest of my equipment to be moved safely. During the moving process, I pulled the drives from the NAS and carried those with me in anti-static bags.

With the new apartment, my laptop being used in the rack no longer worked as my new apartment had an ‘IDF’ panel in a closet, where the rack now lives, rather then next to my desk. I also changed ISPs to a Fiber provider so the cable modem was no longer needed in the rack.

Network

I took this chance to upgrade my network core as the original gen 1 cloud key was starting to lack in feature capabilities and I never had consistent metric tracking. I ordered a Ubiquiti Dream Machine Special Edition (SE) along with another AP U6-Pro for additional WIFI coverage. I retired the USG and gen 1 cloud key and repurposed the Switch and AP into my office for work.

Primary server

The main changes here for the moment were to remove the handles on the chassis so I could close both the rack door and the closet door. In doing so, this removed the ‘normal’ mounting points for the chassis. I rectified this with some ‘shelf rails’ and some command strips to keep the faceplate on the server itself.

A few months after the move, I installed a Mellanox MCX311A-XCAT CX311A ConnectX-3 EN 10G Ethernet SFP+ card plus a ‘10Gtek#’ SFP+ cable to use with the Dream machine’s SFP+ port. (welcome to the world of 10G!!)

I also bought more RAM off Ebay bringing the total to 64GB for now.

Other Gear

I pulled the Mac Mini with the intention of working on the memory stability issue and reimage it with a flavor of linux on a new SSD.

The pull out rack shelf, thunderbolt dock, some older harddrives that had their data migrated to the NAS, and the old Cable modem were pulled from the rack. The remaining equipment was rearranged to accommodate the new Unifi switch and clean up the cable management.

Homelab 2.5

Mac Mini

I broke it trying to install the SSD. I stopped caring about it at this point as it is nearly a decade old. I will likely get some more Raspberry Pis or some other mini PC to use as external nodes with the primary server.

Primary Server

Name: Ark I name most of my equipment after Transformers and I decided on ‘Ark’ as it carried all the Autobots to Earth.

Not all the memory I installed was being recognized so I figured might as well check each DIMM stick and rearrange the memory to be recognized by the motherboard. In doing so, I was able to get 48GB recognized but while testing, the system would no longer POST and after some research I found the issue was with the BIOS’s Shadow RAM. At this point, I was getting annoyed with the TYAN board and its limitations, so I opted for some upgrades.

New Motherboard

I found a good deal for a ‘Supermicro X8DT3-LN4F’ on Ebay. Board arrived DoA

After getting a replacement from a different merchant, I went to work transplanting the CPUs and heatsinks to the new board. Due to the heatsinks form factor, I needed to remove the CPU’s retention bracket to install the heatsinks. Here lies the source of many future problems..

As I was still unsure if all the DIMMS I had were working or not, I went to testing each DIMM. In my testing, I had to remove the heatsink, which normally took the CPU with it. While re-installing the CPU during my tests, I bent some pins. After a few hours of attempting to realign the pings, I believe I ended up shorting my CPU as the motherboard would enter a power cycle loop when attempting to turn on.

As the CPUs were no longer viable, I ordered two Intel Xeon X5690.

After returning this board and getting a replacement, I learned my lesson and purchased different heatsinks, opting for the Noctua NH-U12DX i4. After CAREFUL testing I finally got all RAM (84GB total) recognized by the server and started a memtest86+ run to verify all was healthy. After about 2.5 hours, all tests passed and I installed the board in the chassis and prepared to mount the server. The heatsinks were a bit tall so the cover was not able to close all the way, but the system booted to the OS so I went ahead with the install. While trying to configure the IPMI, I was not seeing the interface in the Unifi dashboard so I went to manually turn the server on. There was an odd smell so I checked inside and saw an orange glow. The motherboard was on fire. I pulled power and investigated.

< add possible chip ID that caught fire>

Server got spicy🔥

New Motherboard 2 (Electric Boogaloo)

One more time, with feeling!!

I was able to return the motherboard that caught fire and I opted to return the pair of Noctua NH-U12DX i4. I was not able to find a similar deal to the X8DT3-LN4F but I found a Supermicro X8DTH-iF that would fit my needs and came with two Xeon E5645 and 4GB of RAM so I now had ‘spare parts’ to test other components if needed. For heatsinks, I ordered two NH-D9DX i4 3U as these will fit in the chassis and provide adequate cooling. During this time, I also upgraded the SSDs as one of the original SSD pairs were starting to have some hardware issues. I ordered 2 Sandisk Ultra 3D 1TB SSDs and reinstalled proxmox 8.1 onto the server. I also opted to remove the old Radeon 5770 as its power draw was overloading my apartments circuit breaker from time to time and was adding around 200W of idle power draw (per some quick readings from the APC’s load indicator).

During the installation process, I was having issues getting the X8DT3 to output to my monitor via VGA as it was outputting an unsupported resolution while trying to run the ProxMox 8.1 installer. After some forum searching, I was able to modify the boot config with the option nomodeset for the console instal method. This let me finally install proxmox 8.0 onto the new SSD. Proxmox does not like DHCP, but I did not initially set this up during install. After rebooting, I was not getting any networking from the node. Investigating led to determining that Proxmox does not do well with DHCP. I would prefer to set a ‘dynamically static’ IP via the Ubiquiti console (essentially a ‘fixed IP’), but Proxmox was not happy with this. I ended up modifying the /etc/network/interfaces file directly to configure the vmbr0 to have the static IP 10.0.0.168. running ifreload -a to apply the changes. The remaining interfaces, primarily the Mellanox 10G SFP+, I set the DHCP which were assigned a ‘Fixed IP’ in the Ubiquiti Networking console.

Now that I had networking, I upgraded Proxmox to 8.1 and installed a few other utilities, such as vim, and any other updates. Next, I needed to reset the X8DT3’s IPMI admin password. I installed ipmitool and followed this guide and confirmed by accessing the webUI and accessing via the IPMIView app.

IPMI update/notes

During the 2023 Holiday season I was traveling and while my girlfriend returned home for work, my UPS had tripped and I lost power. While she attempted to silence the alarm while I was on a plane back to the East Coast, she pulled the rack out which resulted in the primary server (Ark)’s main ethernet cable becoming lose and I was unable to reach the service during my time on the East Coast. Upon return, I first changed the server’s networking config to use ensp0f1 rather then ensp0f0 as the latching mechanism on the nic0 was faulty.

This also led me to investigate why the IPMI interface was not accessible when the OS nic was unplugged. This was due to the lan selection being set to Failover by default which from some forums has some issues:

Failover (factory default): On boot, detect if the dedicated IPMI interface is connected. If so, use the dedicated interface, otherwise fall back to the shared LAN1. I’ve never found a good use for this option. As best I can tell, this setup is fundamentally flawed - I haven’t tested it extensively, but I’ve heard reports it’ll fail to detect the dedicated interface in many circumstances because the upstream switch isn’t passing traffic - for example, after a power outage if the switch and system come up simultaneously, or if the switch is still blocking during the spanning tree detection. Combine this with the fact that the check only happens at boot, and it’s just generally hard to control what interface you end up using.

I used the ipmitool commands also mentioned in the same forums to change the lan selection to Dedicated: ipmitool raw 0x30 0x70 0x0c 1 0

This did not seem to persist, so I also changed the setting in the web interface which appeared to persist. The IPMI interface not properly appeared in the Unify Networking console as connecting from it’s dedicated connection to the switch.

Homelab 2.6

Moonbase 1 + 2

I purchased 2 HP Elitedesk G3 Mini PCs, each with an I5-7500T, 16GB of RAM, and I put in 2 working Kingston Drives I had.

I installed Proxmox onto each of these and added them to the existing Primary server (Ark).

I found while trying to add the nodes to the primary server, I first had to initiate a cluster which I did via the UI. Once a cluster (named ‘Cybertron’) was created, I tried to run pvecm add 10.0.0.168 but was running into authentication issues. From a forum I found that adding ark’s public ssh key to each node and adding all node’s IP addresses into each node’s /etc/hosts file resolved this.

/Current End. more TBD/

Updated: