Building a Data Lake with Spark and Iceberg at Home to over-complicate shopping for a House

Introduction I self host everything but email. I wrote about this here, here, or here. As a summary, at home, I run a 3 node Proxmox cluster with several services, powering a home network with Mikrotik router, Mikrotik switches, and UniFi WiFi, as well as an external VPS. This article is about two things: Why I still bother and what it has...

Introduction It’s been a while since I wrote about Bridge Four, my Scala 3 distributed data processing system from scratch. In this article, we’ll be discussing some major changes around Bridge Four’s state management, its new-and-improved consistency guarantees, and other features and improvements I’ve added since. In case you haven’t read the...

Introduction My home server is a Proxmox cluster. Recently, one of the host’s SSDs indicated it needed a replacement. I run TrueNas SCALE on it by passing through all my hard drives via LSI HBA so that zfs has access to the raw hardware, which makes the migration to a new SSD a bit tricker. For added difficulty, this process assumes the SSD is a...

Introduction Part 2 can be found here! Having found myself recently fun-employed in the great tech layoffs of 2023 (cue ominous thunder in the background) [1], I found myself in a bit of a conundrum: For landing a job, one needs to pass interviews. For passing interviews, one needs to jump through ridiculous hoops and, one of my favorite phrases,...

Introduction In 2019, I built a home server. It was pretty fun. Consumer hardware running Debian, all contained in a 19" Rack. It continued to hum along, basically unchanged (except for some RAM), serving a PiHole (DNS + DHCP), a UniFi controller for WiFi, smb shares for network drives (including TimeMachine), data backups, duplicati for system...