How a Redditor Ended Up With an Industrial-Grade Netflix Server (vice.com) 40
A Redditor says they've managed to get a hold of an old Netflix server for free, and has posted a detailed online look at the once mysterious hardware. The devices were part of Netflix's Open Connect Content Delivery Network (CDN), and can often be found embedded within major ISP networks to ensure your Netflix streams don't suck. From a report: Reddit user PoisonWaffle3 said the ISP he currently works for has been offloading old Netflix servers as they upgrade to more modern equipment. In a Reddit thread titled "So I got a Netflix cache server..." he posted a photo of the server, which is bright Netflix red, and explained how he was curious about what's inside the boxes given how little public information was available.
"All I could find online was overviews, installation/config guides for their proprietary software, etc.," he said. "No specs, no clue what was inside the red box." Dave Temkin, Netflix's former Vice President of Network Systems Infrastructure told Motherboard there's nothing too mysterious about what the servers can do, though they significantly help improve video streaming by shortening overall content transit time. "They're just an Intel FreeBSD box," he said. "We got Linux running on some of the generations of that box as well."
Netflix's Open Connect Content Delivery Network hardware caches popular Netflix content to reduce overall strain across broadband networks. Netflix lets major broadband ISPs embed a CDN server on the ISP network for free; the shorter transit time then helps improve video delivery, of benefit to broadband providers and Netflix alike. It took all of three screws for PoisonWaffle3 to get inside the mysterious red unit, at which point users discovered a "fairly standard" Supermicro board, a single Xeon E5 2650L v2 processor, 64GB of DDR3 memory, and a 10 gigabit ethernet card. They also found 36 7.2TB 7200RPM drives and six 500GB Micron solid state drives, for a grand total of 262 terabytes of storage.
"All I could find online was overviews, installation/config guides for their proprietary software, etc.," he said. "No specs, no clue what was inside the red box." Dave Temkin, Netflix's former Vice President of Network Systems Infrastructure told Motherboard there's nothing too mysterious about what the servers can do, though they significantly help improve video streaming by shortening overall content transit time. "They're just an Intel FreeBSD box," he said. "We got Linux running on some of the generations of that box as well."
Netflix's Open Connect Content Delivery Network hardware caches popular Netflix content to reduce overall strain across broadband networks. Netflix lets major broadband ISPs embed a CDN server on the ISP network for free; the shorter transit time then helps improve video delivery, of benefit to broadband providers and Netflix alike. It took all of three screws for PoisonWaffle3 to get inside the mysterious red unit, at which point users discovered a "fairly standard" Supermicro board, a single Xeon E5 2650L v2 processor, 64GB of DDR3 memory, and a 10 gigabit ethernet card. They also found 36 7.2TB 7200RPM drives and six 500GB Micron solid state drives, for a grand total of 262 terabytes of storage.
Nice Box (Score:2)
... users discovered a "fairly standard" Supermicro board, a single Xeon E5 2650L v2 processor, 64GB of DDR3 memory, and a 10 gigabit ethernet card. They also found 36 7.2TB 7200RPM drives and six 500GB Micron solid state drives, for a grand total of 262 terabytes of storage.
That's a decent little machine. I wouldn't mind one for free :)
what is the Wear Level on the ssd disks? (Score:2)
what is the Wear Level on the ssd disks?
Re:what is the Wear Level on the ssd disks? (Score:4, Funny)
what is the Wear Level on the ssd disks?
42, of course.
Re: (Score:2)
Re: (Score:2)
Your questions, answered (Score:4, Informative)
>How a Redditor Ended Up With an Industrial-Grade Netflix Server
He bought it
Re: (Score:2)
Yeah, it's not a big mystery. I worked for ISPs that had Akamai and Google CDN servers... at end-of-life, they'd remote wipe them. They'd ask if we wanted them to dispose of the servers (they'd send boxes and shipping labels if so), otherwise we were free to keep them or dispose of them locally. I think maybe one wanted the hard drives back but we could keep the server? Can't remember which did what now.
They aren't generally anything special. The first Akamai servers I received to install were just standard
Re: (Score:2)
Read the homelab thread. ISP was decommissioning the appliance and after things were wiped OP was allowed to keep it at no cost. Probably won't see these on eBay any time soon.
I am confused by the claim of no specs. Just open the chassis up and itemize the list of hardware.
Re: (Score:2)
Quite open (Score:5, Informative)
Netflix is actually quite open about their hardware configurations. Just search for "FreeBSD" instead of "Linux", because that's what their CDNs run on. In their latest offerings, they're now pushing 800gbps per node, which is just freaggin insane! https://freebsdfoundation.org/... [freebsdfoundation.org]
Re: (Score:2)
Netflix is actually quite open about their hardware configurations. Just search for "FreeBSD" instead of "Linux", because that's what their CDNs run on.
Netcraft confirms... ?
Re: (Score:2)
I heard Netflix is beleaguered, if that helps.
Re: (Score:3)
It isn't really FreeBSD, but ZFS which is doing all the heavy lifting. Something like an iXSystems TrueNAS machine can handle a surprising amount of IOPS very reliably because of ZFS behind the scenes.
Re: (Score:3)
Actually, quite the opposite. Netflix doesn't use ZFS / OpenZFS on their CDN boxes. But they DO run the head branch of FreeBSD, directly developing a lot of the bleeding edge kernel features in-house, such as in-kernel TLS for sending encrypted HTTPS video streams to customers by entirely bypassing userspace. Think what SENDFILE was in the earlier days of the web, but modernized for HTTPS encrypted traffic. https://papers.freebsd.org/201... [freebsd.org]
Re: (Score:3)
sendfile() is extremely useful these days, especially since the the two file descriptors can be anything - no longer did the destination need to be a socket.
If you ever need to make a copy of a file, or copy the contents of a file into another file, sendfile() makes it extremely easy and basically does all the error handling for you so you can check the final result
Don't knock it because you can achieve in a single line of code what would take a few doze
Re: (Score:2)
The world’s busiest ftp site ftp.cdrom.com famously ran FreeBSD, something like 5000 connections. It's had a long history of good performance.
Re: (Score:2)
It's amazing that still though FreeBSD persists and has use cases for high traffic web. I daresay it may h
Re: (Score:2)
Also parts lists here:
https://openconnect.netflix.co... [netflix.com]
I thought they had an open hardware design at one point but can't find it now.
Notably Tyan motherboards. It has my attention.
The ISP I worked for turned one down (Score:3)
Re:The ISP I worked for turned one down (Score:5, Interesting)
IIRC the Netflix servers don't really operate as a cache. They basically predict what will be popular the next day, and preload whatever will fit overnight.
Re:The ISP I worked for turned one down (Score:5, Informative)
Re: (Score:3)
In general a cache (especially in web terms) is fetching things it gets requests for and then also storing a copy. This is purely Netflix deciding what they think people will want and pushing it out in advance. If people requested something different, they got it from a different source; the near-by server didn't proxy it and pass it along.
Calling it a cache would be like calling a Linux/*BSD distribution's mirror server network a cache, rather than just a content distribution network.
Re: (Score:2)
These days I wonder if energy cost is an issue. These things have a lot of spinning rust, and the CPU will be working hard to handle all the parity checks.
Dumpster Diving (Score:2)
Hang out behind any datacenter and their dumpster will be periodically filled with old servers being tossed, *with data on drives that has not been wiped*.
find any weather channel hardware? cable ISP may (Score:2)
find any weather channel hardware? cable ISP may dumping the weather scan units soon and that software need to be saved.
I'm curious, (Score:3)
Re: (Score:2)
Re: (Score:2)
I don't know, but I wouldn't be surprised if it is something custom. It needs to store a small number of very large files. And nothing much else besides indexes into those files. It doesn't need to be general purpose, except the small root partition.
Re: (Score:3)
Definitely ZFS. This is probably why they are getting such good numbers, especially if the SSDs are being used as ZILs/SLOGs and L2ARC caching, as well as metadata caching for deduplication. I have obtained numbers similar to this with a SuperMicro (which was a destination server for backups) of a similar configuration and SAS cards used to the array, but was using Ubuntu as the OS that did the heavy lifting and ZFS on Linux.
For an array this large, there is no other way to go other than ZFS, and doing 8-
No RAID (Score:2)
Everything is cached from the upstream servers, so there's no reason to use RAID. If a drive fails, it would be faster to download the videos again instead of reconstructing them anyway. They would presumably want the configuration such that the loss of any one drive wouldn't impact the others, though it wouldn't be a big deal if the failure plan is for the system to mask of the failed drive and do a full reset, wiping all local data.
Re: (Score:2)
Checksumming means you're defeating zero-copy sendfile() by having the cpu read from ram. It will lower perf.
Re: (Score:2)
Think about what it's for. if there's any RAIDing, it's RAID0 and even that probably isn't necessary.
Re: (Score:2)
There are many other ways to go than ZFS. Further, Netflix frankly doesn't care about bit rot in cached files, so all that checksumming is just an unnecessary waste of performance they could achieve and electricity.
not a big machine ... (Score:2)
At work I use two wondows boxes, each 128 GB memory, 4 TB hard disk, 24 processors. And one linux box 1 TB memory, headless but with a superduper graphics card to run TurboVNC, 8 TB disk. Of course access to HPC network when I need even more computing power.
And as a lowly mesh maker I dont get machines the fem solver developers are eligible for. They easily get multiple 1TB machines.
HTF server hardware became illicit (Score:2)
Joining the ranks of contraband, now hardware is deemed non-consumer accessible though not unlawful. Side eye the poor chap whose fortune befalls the gift of salvage from generational obsolescence.
Gimme a break, servers flow out of corps and picking one up no different than playing with used cars and much safer than dynamite, which isn’t illegal either.