Stack Exchange Moves Everything to the Cloud, Destroys Servers in New Jersey (stackoverflow.blog) 64
Since 2010 Stack Exchange has run all its sites on physical hardware in New Jersey — about 50 different servers. (When Ryan Donovan joined in 2019, "I saw the original server mounted on a wall with a laudatory plaque like a beloved pet.")
But this month everything moved to the cloud, a new blog post explains. "Our servers are now cattle, not pets. Nobody is going to have to drive to our New Jersey data center and replace or reboot hardware..."
Over the years, we've shared glamor shots of our server racks and info about updating them. For almost our entire 16-year existence, the SRE team has managed all datacenter operations, including the physical servers, cabling, racking, replacing failed disks and everything else in between. This work required someone to physically show up at the datacenter and poke the machines... [O]n July 2nd, in anticipation of the datacenter's closure, we unracked all the servers, unplugged all the cables, and gave these once mighty machines their final curtain call...
We moved Stack Overflow for Teams to Azure in 2023 and proved we could do it. Now we just had to tackle the public sites (Stack Overflow and the Stack Exchange network), which is hosted on Google Cloud. Early last year, our datacenter vendor in New Jersey decided to shut down that location, and we needed to be out by July 2025. Our other datacenter — in Colorado — was decommissioned in June. It was primarily for disaster recovery, which we didn't need any more. Stack Overflow no longer has any physical datacenters or offices; we are fully in the cloud and remote...!
[O]ur Staff Site Reliability Engineer, got a little wistful. "I installed the new web tier servers a few years ago as part of planned upgrades," he said. "It's bittersweet that I'm the one deracking them also." It's the IT version of Old Yeller.
There's photos of the 50 servers, as well as the 400+ cables connecting them, all of which wound up in a junk pile. "For security reasons (and to protect the PII of all our users and customers), everything was being shredded and/or destroyed. Nothing was being kept... Ever have difficulty disconnecting an RJ45 cable? Well, here was our opportunity to just cut the damn things off instead of figuring out why the little tab wouldn't release the plug."
We moved Stack Overflow for Teams to Azure in 2023 and proved we could do it. Now we just had to tackle the public sites (Stack Overflow and the Stack Exchange network), which is hosted on Google Cloud. Early last year, our datacenter vendor in New Jersey decided to shut down that location, and we needed to be out by July 2025. Our other datacenter — in Colorado — was decommissioned in June. It was primarily for disaster recovery, which we didn't need any more. Stack Overflow no longer has any physical datacenters or offices; we are fully in the cloud and remote...!
[O]ur Staff Site Reliability Engineer, got a little wistful. "I installed the new web tier servers a few years ago as part of planned upgrades," he said. "It's bittersweet that I'm the one deracking them also." It's the IT version of Old Yeller.
There's photos of the 50 servers, as well as the 400+ cables connecting them, all of which wound up in a junk pile. "For security reasons (and to protect the PII of all our users and customers), everything was being shredded and/or destroyed. Nothing was being kept... Ever have difficulty disconnecting an RJ45 cable? Well, here was our opportunity to just cut the damn things off instead of figuring out why the little tab wouldn't release the plug."
Our servers are now cattle, not pets. (Score:2)
I hope they have a vegan option.
Re:Our servers are now cattle, not pets. (Score:5, Funny)
Re: (Score:2)
Not if they know what they are getting into. I give them 50/50 odds.
Re: Our servers are now cattle, not pets. (Score:2)
Re: (Score:3)
Re: (Score:2)
Stack Exchanges business model is dead after the advent of LLMs.
True.
The problem is that LLMs need to learn from somewhere, and Stack Exchange is a major source of knowledge. Once it's gone, LLMs may stagnate, or, even worse, suffer from model collapse [wikipedia.org], as they recursively rely on their own output.
New technologies (Score:2)
New technologies will have a lot less useful information for LLM training since the hordes of new user and slightly advanced user questions will be answered by AI preventing the foundation layers of AI training material upon which the expert level knowledge is laid.
Translation (Score:2)
Now we rent machines we will never own and pay for other people to poke the servers replacing disks and stuff...
I have to wonder, why would somebody have to drive to New Jersey to reset a server? Haven't they ever heard of remote management?
Re: (Score:2)
Very much this.
Re: (Score:2)
"The Cloud" is nothing more than "Someone else's computer". Do you keep your wallet at your neighbor's house?
That's a silly analogy.
I don't have a wallet (just a pouch on my cell phone).
I keep my money in my bank account in "someone else's building".
Re: Translation (Score:2)
Re: (Score:3)
I have to wonder, why would somebody have to drive to New Jersey to reset a server? Haven't they ever heard of remote management?
You've both asked and answered the question - no, it's likely they haven't.
Re: (Score:2)
Cause they're clearly incompetent.
I've run like 20 servers remotely for between 10 and 20 years, and have never even set foot in the state's they are in. The remote hands have never had to reboot them most of the time, and only after power issues at the data center (eg they literately went and powered off everyone's servers to do a maintenance)
They didn't need to do this, and they definitely aren't saving any money in doing so. If anything the site's probably one moronic AI away from being destroyed on acci
Sounds like they're going to sell and get gutted (Score:5, Insightful)
This is step 1 in trying to sell their flailing business. They obviously don't expect any more growth or they'd stick with running their own servers. Buyers want to know they can carve up the company easily, so migrating to a public cloud gives them some assurances this is possible. They're certainly hoping to get scooped up on their disintegrating brand awareness before there's no value left.
Dice will probably pick them up for pennies on the dollar to get ad spend as long as Google still sends users to the site.
Re:Sounds like they're going to sell and get gutte (Score:5, Insightful)
Re: (Score:3)
10s of thousands of businesses / sites have done this without ever intending to sell up. What makes you think Stack Exchange is doing it to sell? It also makes no sense, you don't shed your physical assets prior to a sale. Any potential buyer can migrate themselves.
Also when did we go from "the cloud scales" to "obviously you move to the cloud if you don't expect growth"? Like literally every point you made feels backwards.
Re: Sounds like they're going to sell and get gutt (Score:2)
The cloud - that's where you end up when you die.
everything shredded and/or destroyed (Score:4, Insightful)
For security reasons (and to protect the PII of all our users and customers), everything was being shredded and/or destroyed. Nothing was being kept...
Everything? Pretty sure just destroying the disks (SSDs/HDDs) would be sufficient. Destroying *everything* seems like wasteful overkill. I've worked places (DoD contractor) where drives being discarded had to be shredded and/or melted, but drives being returned to the Government were just securely wiped, several times. In practice, we securely wiped all drives, even ones destined to be physically destroyed.
I'm sure someone would by the servers...
Re: (Score:3)
Sure, but then you have to pay somebody to disassemble the servers and take inventory to make sure you got everything. If you just destroy the whole thing, it's much simpler and probably less expensive. It was probably all obsolete anyway.
Re: (Score:3)
If you just destroy the whole thing, it's much simpler and probably less expensive. It was probably all obsolete anyway.
I'm not sure that's true; if you destroy the entire computer, how do you verify that the important parts (e.g. the hard drives) were actually destroyed and not repurposed? Presumably they were inside the case, but if you don't open the case up and look, you can't prove that they weren't pilfered the night before and are in someone's bedroom now, waiting to be listed on eBay or somewhere worse.
If I was that paranoid, I'd want to manually inventory each hard drive and watch it being fed into the shredder wit
Re: (Score:2)
The plot thickens.
Re: everything shredded and/or destroyed (Score:3)
Re: (Score:2)
It was probably all obsolete anyway.
"Obsolete" hardware often works fine, and it's usually rather inexpensive.
For example, all my systems are old(er) and inherited from friends/family. My Windows 10 system is a Dell XPS 420 that a friend gave me in 2017 and it works great -- don't know when he got it, but the model was discontinued in 2009. Only thing I've replaced is the HDD, later replaced with a SSD. My Linux Mint system uses an ASRock Z77 Extreme3 motherboard with Intel Core i7-3770 CPU (32 GM RAM) -- old, but works great. My OPNs
Re: (Score:2)
All the systems you mentioned, are *personal* computers. I'm pretty sure Stack Exchange wasn't using home systems, but rack-mounted servers. Very different story.
Re: everything shredded and/or destroyed (Score:2)
Re: (Score:2)
Yeah, they're embellishing their story. Only the storage was shredded, nothing more.
Re: (Score:3)
Yep, local hospital does that too. The PII is on one device only, the HDD. One could remove the HDD, sell the rest for $50 and that would pay for extracting the HDD and putting a drill-bit through the HDD platter, thrice.
The hospital destroying the 'whole' computer, probably leaves the HDD platter intact. If someone cared, they could transplant a new circuit board and stepper-motor into the HDD. Then the PII can be copied.
Cloud = Servers (Score:3, Interesting)
Re: (Score:2)
Having equipment on-prem only provides the *illusion* of having full control. As soon as you hook up that on-prem equipment to the internet, you expose it to all kinds of potential intrusions, the same ones that they're exposed to in the cloud.
At the same time, your on-prem IT staff is probably stretched thin and is unlikely to do *everything* needed to keep the servers safe. You know, like failing to remove support for obsolete/insecure encryption protocols, or leaving back doors open for the convenience o
Re: (Score:2)
At the same time, your on-prem IT staff is probably stretched thin and is unlikely to do *everything* needed to keep the servers safe. You know, like failing to remove support for obsolete/insecure encryption protocols, or leaving back doors open for the convenience of IT staff. That small IT staff probably doesn't have the time or skillset to automate all the security deployments that cloud providers do.
Simply switching to someone else's servers accomplishes none of the above.
Re: (Score:2)
Nope, but switching to a major cloud provider's servers, does. Nobody understands server security better, or spends more money and manpower on security, than the big cloud providers. *Certainly* not your in-house 50-server farm with a couple of IT staff.
Re: Cloud = Servers (Score:3)
Re: (Score:2)
Nobody understands server security better, or spends more money and manpower on security, than the big cloud providers.
Who is responsible for configuring customers (i.e. _your_) leased space on a cloud provider? Who was responsible for misconfiguring the 'Tea' app [slashdot.org] bucket?
If you own your own servers, the answer is obvious. But if the cloud provider just sold your CIO on the cost savings, you'd better make sure they agreed to do so. And they don't come back and point the finger at your staff for checking the incorrect permissions box. Because your corporate cut your budget, figuring it would be the other guy's problem and yo
Re: (Score:2)
Sure, cloud hosting providers aren't going to prevent you from accidentally leaving your database open without credentials. That's not an advantage of either on-prem or cloud, they are both exactly the same in that regard. What cloud providers DO do, is make sure you don't run unpatched code with known vulnerabilities, and that you don't use obsolete vulnerable protocols. At least, not unless you're literally just renting hardware and managing it yourself. But then again, that's your own fault.
Your claim wa
if you have your VM at an cloud provider or even d (Score:2)
if you have your VM at an cloud provider or even docker images.
you still need to keep them up to date.
Re: (Score:2)
Yes indeed. But then, you can't blame the hosting provider for that.
If you use managed services, then yes, you do get the benefit of the cloud provider's security infrastructure and expertise.
Re: (Score:2)
How is this an improvement for anyone?
It moves liability away to somebody with deeper pockets
It obviates further need to drive to New Jersey.
Re: (Score:2)
Their operations are pretty small That's still a scale where cloud can be cheaper
They say 50 servers. Let's quote t4g.xlarge. they are 16 cores, 32GB nodes. Not the biggest, but not terrible either. That's about $1/h. That$435k per year
Storage say 1TB backups included. that seems reasonnable (probably over) since we are talking mostly text data (it is reported as 450GB uncompressed in database storage. but probably you can keep some compressed) it is about $.10/GB/h on SSDs and half that on spinning. So
Re: (Score:3)
(OK, that will teach me to post on my phone.)
OK, Storage is about $.10/GB/month not per hour. (That bill seems high.)
So even if you make storage 100TB. the storage bill is $120k/year.
So you are talking definitely under $1M a year for the whole operation.
Moving to the cloud probably makes financial sense at that scale and reduces the kind of expertise you need to hire.
Re: (Score:2)
and reduces the kind of expertise you need to hire.
What were these new hires going to do? Ask how to configure servers on Stack Overflow?
Re: (Score:2)
I imagine they had engineer double as server technicians
But you don't send untrained people to rewire a server or swap a hard drive. So I would imagine you would want to hire people who have physically worked in a data center before.
That's a smaller subset of the software engineer pool. That's what I meant.
Re: Cloud = Servers (Score:2)
No Servers, No Offices (Score:2)
First the complaints that AI stole their data.(AI stole everyone's contribution to Stack Overflow.)
Now they've eliminated servers and offices.
Sounds, to me, like Stack Overflow is dying.
Re: (Score:2)
Netcraft confirms Stack Overflow is dying.
Re: (Score:2)
Sounds, to me, like Stack Overflow is dying.
I'm rather shocked how small their operations are, for as big as they claim to be.
What a waste, and what an obvious lie (Score:5, Insightful)
The obvious lie: "For security reasons (and to protect the PII of all our users and customers), everything was being shredded and/or destroyed." If they were actually trying to protect PII, then they shouldn't have moved to the cloud. Not only have they thrown everything into a black box, they've thrown it into somebody else's black box that they have absolutely no control over -- and no visibility into. If some offshore contractor working at that cloud company decides to grab all the info and sell it as a side gig, (a) they can't stop it (b) they won't know it's happened until it's much too late and (c) they'll be unable to do anything meaningful about it.
Re: What a waste, and what an obvious lie (Score:2)
They took great care to prevent the sensitive data from falling into the wrong hands... *after having migrated it to freaking Azure*! The one public cloud where we can be most certain that it has already been entirely and lastingly compromised.
Re: What a waste, and what an obvious lie (Score:1)
Expectations (Score:2)
Expect the re-in-sourcing project to start next week when they figure out how much extra it will cost them, past the "projected" cost. Oh, maintenance, redundancy, replication, performance management, fat fat pipes, unmet SLAs.... It's the same fucking story over and over. So I hope the execs lined their pockets with those M$. Maybe they'll move to UAE, and we'll be rid of them. Then let the tens of thousands of unemployed tec workers help you put your business back together.
Old enough to remember (Score:1)
Ah well (Score:2)
Re: Ah well (Score:2)
Famous last words (Score:2)
"It was primarily for disaster recovery, which we didn't need any more. "
Talk about tempting fate. Wasn't there just an article about an AI deleting an important database?
I take they think they can sell the company before the disaster happens.
Re: Famous last words (Score:2)
And now it costs 3x as much (Score:2)
Sy Sims warehouse (Score:2)
I used to work on servers in the old, repurposed Sy Sims warehouse in New Jersey. There's tons of real estate just across the river from NYC.
Re: (Score:1)
Basecamp says hello (Score:2)
https://basecamp.com/cloud-exi... [basecamp.com]
Tabs (Score:1)
I always thought the tabs on those plugs was a stupid design. They either don't release or always get caught when trying to untangle them...eventually getting snapped off in the fight. Infuriating.
Whoever invented them should have had a good head slapping.