This commodity/consumer grade “server” has stood the test of time – Uptime of 1088 days with zero issues!

There is a unique server in my rack at the data center that performs a very specific task. Being strictly firewalled, both internally and externally, and having shown no signs of impending failure, I reluctantly rebooted it today to apply updates to the OS and various software packages. It had been humming along quietly for nearly three years. What’s under the hood of this nondescript 1U box may surprise you…

When my IT consulting business really got moving around 2004-2005, I was living in a small house in Myrtletown, CA, and ran the business from there. That house later became my company’s dedicated office (it just happened to be zoned for commercial use, with a lease and landlord combo that I wasn’t willing to let go). There was an odd closet in the center of the home where I built a makeshift server cabinet using basic lumber and fence brackets to create vertical-mount 19″ spaces to hold rack-mount gear, as well as some shelves for miscellaneous hardware. The house had a spacious attic and crawlspace, making it easy to run power and network cables wherever I needed. I wired up a couple of bathroom fans to move air from under the house, though the closet, and into the attic for rudimentary climate control. This worked well thanks to the cool coastal climate along the Humboldt Bay.

Sprux LLC’s server cabinet in Myrtletown, CA in the mid 2000s

Being on a limited budget, but still needing to run real services for my staff and general business infrastructure, I filled the closet with various machines pieced together with old parts or cheap used bits from eBay. I didn’t have enough depth in the closet to mount proper enterprise servers, like the Dell PowerEdge series I now run almost exclusively in our critical production environments, so I wound up with two or three short-depth 1U machines loaded with random components. At the height of my company’s tenure in Northern California that little cabinet was humming along and as busy as ever.

Since there were no colocation facilities in Humboldt County, CA, and I had yet to move my critical infrastructure to a proper data center in the SF Bay Area, we actually hosted production services from our small office. Though we did have both a DSL line and a cable uplink (using some clever firewalling and routing configs on a multi-homed machine appropriately named intersect), at the time there was only one fiber line for the entire county, which was buried along Highway 101 to an uplink provider north of San Francisco. This meant county-wide outages every time a contractor severed the fiber during roadwork along Highway 101. Nonetheless, we were primarily doing development work for local businesses, on-site tech support, hardware repair, and other jobs that didn’t involve a lot of internet service hosting. That said, I have always hosted my company’s email and web services on hardware I own and have physical control over. On top of all that, we deployed a high-power 802.11g public wifi hotspot to serve the Myrtletown area, which not only got us on the front page of the local paper but also provided dozens of local residents with free internet access. At least one individual in the hills north of Myrtletown pointed a directional antenna at our office and was able to get surprisingly good network performance over a distance of at least two miles!

Sprux software engineer and dear friend bangs out a website for a local organic farm in Humboldt County

The work we were doing at the time required typical on-premises services: networking gear (switches and routing/firewall machines); web, mail, and DNS; a file server hosting code repositories, staff home folders, backups, and other miscellaneous services; and of course UPSes to keep us up and running during power outages. While I didn’t skimp on the file server (proper RAID isn’t exactly cheap), those little cheapo half-depth boxes full of used consumer parts weren’t exactly expected to last long; we merely needed a stop-gap solution during the growth phase. The cases were perfect for my closet server rack, particularly because they were designed to expose hardware ports at the front rather than the back. Over the years, as the business moved to Austin and eventually Northern Nevada, these little 1U machines played countless roles in our infrastructure and were nearly bullet-proof. Perhaps a dead disk or a worn out fan here and there, but otherwise they ran in production environments for at least seven years, with two eventually being retired to free up valuable rack space.

Yet one machine remains in service. I have not upgraded or replaced any components since it was built 10 years ago. It currently lives in our rack at Roller Network‘s world-class colocation facility in Reno, NV. The machine runs only our monitoring systems: nagios and related trending/graphic tools. Being multi-homed, it is able to monitor the various dedicated VLANs we provide for our hosted clients while also handling remote monitoring services across the internet for customers with their own hosted environments. While I do plan to eventually virtualize these services to free up rack space, I am amazed at how well the piddly “server” has performed and how much value it has provided to my business. And the entire thing cost less than $200 a decade ago. When I rebooted it tonight (remotely, of course) I half expected it to never come back online. But it was up and running in under three minutes, humming away.

And so it is, sometimes you get lucky. Sure, I’ve seen enterprise Dell, HP, and Supermicro machines run for well over a decade without a single hardware failure. But in this case we’re talking about a box that was scraped together as inexpensively as possible with essentially no forethought. Here’s what’s under the hood:

• Desktop motherboard, model K7S41GX (American Megatrends…?)
• AMD Duron CPU, 1.2GHz, single core
• 1GB RAM
• 2 Seagate 40GB disks, model ST340014A, linux software RAID-1
• Intel 82546GB dual gigabit NIC

This hardware is so old, it actually has an ISA bus.

Anyway, that’s my linux nerd story for the day. I wrote this while waiting on some fairly massive database jobs to run in a development environment so I can bang out a bit of code to try to get ahead of my workload this week.

I would love to hear your stories of ancient hardware that keeps on going, whether it’s your old laptop, desktop, dev box, something you are still running in a production environment, or that mainframe you bought at an auction and has been sitting in your garage for longer than you want to admit. I suspect I’m not the only linux geek who’s seen 10+ year old machines run 24/7/365 without failure…