Stack Exchange Moves Everything to the Cloud, Destroys Servers in New Jersey (stackoverflow.blog) 87
Since 2010 Stack Exchange has run all its sites on physical hardware in New Jersey — about 50 different servers. (When Ryan Donovan joined in 2019, "I saw the original server mounted on a wall with a laudatory plaque like a beloved pet.")
But this month everything moved to the cloud, a new blog post explains. "Our servers are now cattle, not pets. Nobody is going to have to drive to our New Jersey data center and replace or reboot hardware..."
Over the years, we've shared glamor shots of our server racks and info about updating them. For almost our entire 16-year existence, the SRE team has managed all datacenter operations, including the physical servers, cabling, racking, replacing failed disks and everything else in between. This work required someone to physically show up at the datacenter and poke the machines... [O]n July 2nd, in anticipation of the datacenter's closure, we unracked all the servers, unplugged all the cables, and gave these once mighty machines their final curtain call...
We moved Stack Overflow for Teams to Azure in 2023 and proved we could do it. Now we just had to tackle the public sites (Stack Overflow and the Stack Exchange network), which is hosted on Google Cloud. Early last year, our datacenter vendor in New Jersey decided to shut down that location, and we needed to be out by July 2025. Our other datacenter — in Colorado — was decommissioned in June. It was primarily for disaster recovery, which we didn't need any more. Stack Overflow no longer has any physical datacenters or offices; we are fully in the cloud and remote...!
[O]ur Staff Site Reliability Engineer, got a little wistful. "I installed the new web tier servers a few years ago as part of planned upgrades," he said. "It's bittersweet that I'm the one deracking them also." It's the IT version of Old Yeller.
There's photos of the 50 servers, as well as the 400+ cables connecting them, all of which wound up in a junk pile. "For security reasons (and to protect the PII of all our users and customers), everything was being shredded and/or destroyed. Nothing was being kept... Ever have difficulty disconnecting an RJ45 cable? Well, here was our opportunity to just cut the damn things off instead of figuring out why the little tab wouldn't release the plug."
We moved Stack Overflow for Teams to Azure in 2023 and proved we could do it. Now we just had to tackle the public sites (Stack Overflow and the Stack Exchange network), which is hosted on Google Cloud. Early last year, our datacenter vendor in New Jersey decided to shut down that location, and we needed to be out by July 2025. Our other datacenter — in Colorado — was decommissioned in June. It was primarily for disaster recovery, which we didn't need any more. Stack Overflow no longer has any physical datacenters or offices; we are fully in the cloud and remote...!
[O]ur Staff Site Reliability Engineer, got a little wistful. "I installed the new web tier servers a few years ago as part of planned upgrades," he said. "It's bittersweet that I'm the one deracking them also." It's the IT version of Old Yeller.
There's photos of the 50 servers, as well as the 400+ cables connecting them, all of which wound up in a junk pile. "For security reasons (and to protect the PII of all our users and customers), everything was being shredded and/or destroyed. Nothing was being kept... Ever have difficulty disconnecting an RJ45 cable? Well, here was our opportunity to just cut the damn things off instead of figuring out why the little tab wouldn't release the plug."
Our servers are now cattle, not pets. (Score:4, Funny)
I hope they have a vegan option.
Re:Our servers are now cattle, not pets. (Score:5, Funny)
Re: (Score:2)
Not if they know what they are getting into. I give them 50/50 odds.
Re: Our servers are now cattle, not pets. (Score:3)
Re: (Score:3)
Re: (Score:3)
Stack Exchanges business model is dead after the advent of LLMs.
True.
The problem is that LLMs need to learn from somewhere, and Stack Exchange is a major source of knowledge. Once it's gone, LLMs may stagnate, or, even worse, suffer from model collapse [wikipedia.org], as they recursively rely on their own output.
New technologies (Score:3)
New technologies will have a lot less useful information for LLM training since the hordes of new user and slightly advanced user questions will be answered by AI preventing the foundation layers of AI training material upon which the expert level knowledge is laid.
Re: (Score:2)
There's already an LLM specific software documentation format. LLMs don't need to scrape stack exchange to find out how to implement a binary sort in powershell, they can consult the documentation directly. Model collapse as a worry is pretty outdated concept at this point.
Re: (Score:2)
Despite what some consulting firms think, Stack Exchange is where you go to find the actual correct answer, contrary to vendor expectation. It's exactly on stack exchange that you'll find the vendors answer, together with a comment about why it doesn't work, in what situation it explodes, how to work around it if that happens and what you can do which totally avoids the problem and exactly why the vendor is trying to avoid you finding that out.
Yes, 90% of answers are bad, but learning how to work out the di
Re: (Score:2)
Translation (Score:3)
Now we rent machines we will never own and pay for other people to poke the servers replacing disks and stuff...
I have to wonder, why would somebody have to drive to New Jersey to reset a server? Haven't they ever heard of remote management?
Re: (Score:1)
....why would somebody have to drive to New Jersey to reset a server? Haven't they ever heard of remote management?
Sometimes there are malfunctions that can't be fixed remotely. Shit happens.
Now we rent machines we will never own and pay for other people to poke the servers replacing disks and stuff...
"The Cloud" is nothing more than "Someone else's computer". Do you keep your wallet at your neighbor's house? Of course not. That would be stupid. Keeping your business on "someone else's computer" is just as stupid.
Re: (Score:2)
Very much this.
Re: (Score:3)
"The Cloud" is nothing more than "Someone else's computer". Do you keep your wallet at your neighbor's house?
That's a silly analogy.
I don't have a wallet (just a pouch on my cell phone).
I keep my money in my bank account in "someone else's building".
Re: (Score:2)
SO, on someone else's computer. You don't own your phone, Google/Apple and your cellphone providers own it. You're just paying the lease. Anytime one of those 2 want, they can bork it for you.
Re: Translation (Score:2)
Re: (Score:3)
I have to wonder, why would somebody have to drive to New Jersey to reset a server? Haven't they ever heard of remote management?
You've both asked and answered the question - no, it's likely they haven't.
Re: (Score:2)
Cause they're clearly incompetent.
I've run like 20 servers remotely for between 10 and 20 years, and have never even set foot in the state's they are in. The remote hands have never had to reboot them most of the time, and only after power issues at the data center (eg they literately went and powered off everyone's servers to do a maintenance)
They didn't need to do this, and they definitely aren't saving any money in doing so. If anything the site's probably one moronic AI away from being destroyed on acci
Re: (Score:2)
I'm currently in Santiago, Chile.
I have never ever had to step foot in the datacenter or even the continent where it resides.
Re: Translation (Score:2)
Upgrading spinning rust, PCIe, and RAM would require site visits.
The issue is rent for a place to put the rack when the business model is collapsing. The guy who upgrades the hw was probably let go.
Sounds like they're going to sell and get gutted (Score:5, Insightful)
This is step 1 in trying to sell their flailing business. They obviously don't expect any more growth or they'd stick with running their own servers. Buyers want to know they can carve up the company easily, so migrating to a public cloud gives them some assurances this is possible. They're certainly hoping to get scooped up on their disintegrating brand awareness before there's no value left.
Dice will probably pick them up for pennies on the dollar to get ad spend as long as Google still sends users to the site.
Re:Sounds like they're going to sell and get gutte (Score:5, Insightful)
Re: (Score:2)
SO was never the valuable FAQ. Look at how their search function sucks and how many duplicate questions are asked. They were the competent community. Then they didn't manage to moderate the elitist people, driving away new users keeping with the usual "The high karma accounts farm even more karma dilemma". LLM are only the final drop, the decay started much earlier.
Re:Sounds like they're going to sell and get gutte (Score:4, Insightful)
10s of thousands of businesses / sites have done this without ever intending to sell up. What makes you think Stack Exchange is doing it to sell? It also makes no sense, you don't shed your physical assets prior to a sale. Any potential buyer can migrate themselves.
Also when did we go from "the cloud scales" to "obviously you move to the cloud if you don't expect growth"? Like literally every point you made feels backwards.
Re: Sounds like they're going to sell and get gutt (Score:2)
Re: (Score:2)
If you need a lot of resources, the efficient way to host is to have your own servers. You go to the cloud if you need to be able to scale up and down. A healthy SO would only need to scale up, which means buying new servers from time to time. An unhealthy one needs to be able to scale down. And that is most easy if you're on the cloud and just remove a few instances you no longer need.
Re: Sounds like they're going to sell and get gutt (Score:2)
The cloud - that's where you end up when you die.
Re: (Score:1)
This is step 1 in trying to sell their flailing business. They obviously don't expect any more growth or they'd stick with running their own servers. Buyers want to know they can carve up the company easily, so migrating to a public cloud gives them some assurances this is possible. They're certainly hoping to get scooped up on their disintegrating brand awareness before there's no value left.
My company tried to buy them out. They responded:
\
Thank you for your interest in acquiring Stack Overflow. Unfortunately, your proposal has been closed for the following reasons:
everything shredded and/or destroyed (Score:5, Insightful)
For security reasons (and to protect the PII of all our users and customers), everything was being shredded and/or destroyed. Nothing was being kept...
Everything? Pretty sure just destroying the disks (SSDs/HDDs) would be sufficient. Destroying *everything* seems like wasteful overkill. I've worked places (DoD contractor) where drives being discarded had to be shredded and/or melted, but drives being returned to the Government were just securely wiped, several times. In practice, we securely wiped all drives, even ones destined to be physically destroyed.
I'm sure someone would by the servers...
Re: (Score:3)
Sure, but then you have to pay somebody to disassemble the servers and take inventory to make sure you got everything. If you just destroy the whole thing, it's much simpler and probably less expensive. It was probably all obsolete anyway.
Re:everything shredded and/or destroyed (Score:5, Interesting)
If you just destroy the whole thing, it's much simpler and probably less expensive. It was probably all obsolete anyway.
I'm not sure that's true; if you destroy the entire computer, how do you verify that the important parts (e.g. the hard drives) were actually destroyed and not repurposed? Presumably they were inside the case, but if you don't open the case up and look, you can't prove that they weren't pilfered the night before and are in someone's bedroom now, waiting to be listed on eBay or somewhere worse.
If I was that paranoid, I'd want to manually inventory each hard drive and watch it being fed into the shredder with my own eyeballs.
My suspicion is that most parts of the computers weren't destroyed, but rather they were sold off or given away to some third party that will figure out what to do with them. But it's easier and simpler to tell the public they were destroyed.
Re: (Score:2)
The plot thickens.
Re: everything shredded and/or destroyed (Score:3)
Re: (Score:2)
It was probably all obsolete anyway.
"Obsolete" hardware often works fine, and it's usually rather inexpensive.
For example, all my systems are old(er) and inherited from friends/family. My Windows 10 system is a Dell XPS 420 that a friend gave me in 2017 and it works great -- don't know when he got it, but the model was discontinued in 2009. Only thing I've replaced is the HDD, later replaced with a SSD. My Linux Mint system uses an ASRock Z77 Extreme3 motherboard with Intel Core i7-3770 CPU (32 GM RAM) -- old, but works great. My OPNs
Re: (Score:2)
All the systems you mentioned, are *personal* computers. I'm pretty sure Stack Exchange wasn't using home systems, but rack-mounted servers. Very different story.
Re: (Score:2)
All the systems you mentioned, are *personal* computers. I'm pretty sure Stack Exchange wasn't using home systems, but rack-mounted servers. Very different story.
Sure, but that doesn't change the fact that even "obsolete" hardware can be useful. As a sysadmin, I've used plenty of old(er) server class systems in production, as well as development and research, settings and they generally work fine; just be realistic about their performance and capability and plan accordingly. Not everything needs the latest and greatest.
Re: everything shredded and/or destroyed (Score:2)
Re: (Score:2)
Yeah, they're embellishing their story. Only the storage was shredded, nothing more.
Re: (Score:3)
Yep, local hospital does that too. The PII is on one device only, the HDD. One could remove the HDD, sell the rest for $50 and that would pay for extracting the HDD and putting a drill-bit through the HDD platter, thrice.
The hospital destroying the 'whole' computer, probably leaves the HDD platter intact. If someone cared, they could transplant a new circuit board and stepper-motor into the HDD. Then the PII can be copied.
Re: everything shredded and/or destroyed (Score:2)
Re: (Score:2)
> Destroy the hardware to protect PII
> Move the PII to the cloud
Are they serious?
Cloud = Servers (Score:3, Interesting)
Re: (Score:2)
Having equipment on-prem only provides the *illusion* of having full control. As soon as you hook up that on-prem equipment to the internet, you expose it to all kinds of potential intrusions, the same ones that they're exposed to in the cloud.
At the same time, your on-prem IT staff is probably stretched thin and is unlikely to do *everything* needed to keep the servers safe. You know, like failing to remove support for obsolete/insecure encryption protocols, or leaving back doors open for the convenience o
Re: (Score:2)
At the same time, your on-prem IT staff is probably stretched thin and is unlikely to do *everything* needed to keep the servers safe. You know, like failing to remove support for obsolete/insecure encryption protocols, or leaving back doors open for the convenience of IT staff. That small IT staff probably doesn't have the time or skillset to automate all the security deployments that cloud providers do.
Simply switching to someone else's servers accomplishes none of the above.
Re: (Score:2)
Nope, but switching to a major cloud provider's servers, does. Nobody understands server security better, or spends more money and manpower on security, than the big cloud providers. *Certainly* not your in-house 50-server farm with a couple of IT staff.
Re: Cloud = Servers (Score:3)
Re: (Score:2)
Nobody understands server security better, or spends more money and manpower on security, than the big cloud providers.
Who is responsible for configuring customers (i.e. _your_) leased space on a cloud provider? Who was responsible for misconfiguring the 'Tea' app [slashdot.org] bucket?
If you own your own servers, the answer is obvious. But if the cloud provider just sold your CIO on the cost savings, you'd better make sure they agreed to do so. And they don't come back and point the finger at your staff for checking the incorrect permissions box. Because your corporate cut your budget, figuring it would be the other guy's problem and yo
Re: (Score:2)
Sure, cloud hosting providers aren't going to prevent you from accidentally leaving your database open without credentials. That's not an advantage of either on-prem or cloud, they are both exactly the same in that regard. What cloud providers DO do, is make sure you don't run unpatched code with known vulnerabilities, and that you don't use obsolete vulnerable protocols. At least, not unless you're literally just renting hardware and managing it yourself. But then again, that's your own fault.
Your claim wa
if you have your VM at an cloud provider or even d (Score:2)
if you have your VM at an cloud provider or even docker images.
you still need to keep them up to date.
Re: (Score:2)
Yes indeed. But then, you can't blame the hosting provider for that.
If you use managed services, then yes, you do get the benefit of the cloud provider's security infrastructure and expertise.
Re: (Score:2)
Re: (Score:2)
What part of my statement are you responding to, that you feel is incorrect? What part of my statement do you disagree with? That cloud providers spend more money and effort on security than in-house server farms?
Re: (Score:2)
How is this an improvement for anyone?
It moves liability away to somebody with deeper pockets
It obviates further need to drive to New Jersey.
Re: (Score:2)
Their operations are pretty small That's still a scale where cloud can be cheaper
They say 50 servers. Let's quote t4g.xlarge. they are 16 cores, 32GB nodes. Not the biggest, but not terrible either. That's about $1/h. That$435k per year
Storage say 1TB backups included. that seems reasonnable (probably over) since we are talking mostly text data (it is reported as 450GB uncompressed in database storage. but probably you can keep some compressed) it is about $.10/GB/h on SSDs and half that on spinning. So
Re: (Score:3)
(OK, that will teach me to post on my phone.)
OK, Storage is about $.10/GB/month not per hour. (That bill seems high.)
So even if you make storage 100TB. the storage bill is $120k/year.
So you are talking definitely under $1M a year for the whole operation.
Moving to the cloud probably makes financial sense at that scale and reduces the kind of expertise you need to hire.
Re: (Score:2)
and reduces the kind of expertise you need to hire.
What were these new hires going to do? Ask how to configure servers on Stack Overflow?
Re: (Score:2)
I imagine they had engineer double as server technicians
But you don't send untrained people to rewire a server or swap a hard drive. So I would imagine you would want to hire people who have physically worked in a data center before.
That's a smaller subset of the software engineer pool. That's what I meant.
Re: (Score:2)
Nah, they'll hire "vibe" admins.
Re: Cloud = Servers (Score:2)
Re: (Score:2)
No Servers, No Offices (Score:2)
First the complaints that AI stole their data.(AI stole everyone's contribution to Stack Overflow.)
Now they've eliminated servers and offices.
Sounds, to me, like Stack Overflow is dying.
Re: (Score:2)
Netcraft confirms Stack Overflow is dying.
Re: (Score:2)
Sounds, to me, like Stack Overflow is dying.
I'm rather shocked how small their operations are, for as big as they claim to be.
Re: (Score:2)
SO questions and answers are CC-licensed. It's bad faith complaining, now they aren't the only ones to profit from providing the CC data anymore ... so what? The whole idea of having the site CC licensed is to have people doing useful stuff with the data, like training a code LLM that actually knows how to code. It's the same as other big publishers, they whine because they would like to be paid for data they provide for free. Only that in this case it isn't even their data.
What a waste, and what an obvious lie (Score:5, Insightful)
The obvious lie: "For security reasons (and to protect the PII of all our users and customers), everything was being shredded and/or destroyed." If they were actually trying to protect PII, then they shouldn't have moved to the cloud. Not only have they thrown everything into a black box, they've thrown it into somebody else's black box that they have absolutely no control over -- and no visibility into. If some offshore contractor working at that cloud company decides to grab all the info and sell it as a side gig, (a) they can't stop it (b) they won't know it's happened until it's much too late and (c) they'll be unable to do anything meaningful about it.
Re: What a waste, and what an obvious lie (Score:2)
They took great care to prevent the sensitive data from falling into the wrong hands... *after having migrated it to freaking Azure*! The one public cloud where we can be most certain that it has already been entirely and lastingly compromised.
Re: What a waste, and what an obvious lie (Score:1)
Expectations (Score:2)
Expect the re-in-sourcing project to start next week when they figure out how much extra it will cost them, past the "projected" cost. Oh, maintenance, redundancy, replication, performance management, fat fat pipes, unmet SLAs.... It's the same fucking story over and over. So I hope the execs lined their pockets with those M$. Maybe they'll move to UAE, and we'll be rid of them. Then let the tens of thousands of unemployed tec workers help you put your business back together.
Old enough to remember (Score:1)
Ah well (Score:2)
Re: Ah well (Score:2)
Famous last words (Score:2)
"It was primarily for disaster recovery, which we didn't need any more. "
Talk about tempting fate. Wasn't there just an article about an AI deleting an important database?
I take they think they can sell the company before the disaster happens.
Re: Famous last words (Score:2)
And now it costs 3x as much (Score:2)
Sy Sims warehouse (Score:2)
I used to work on servers in the old, repurposed Sy Sims warehouse in New Jersey. There's tons of real estate just across the river from NYC.
Re: (Score:1)
Basecamp says hello (Score:2)
https://basecamp.com/cloud-exi... [basecamp.com]
Tabs (Score:1)
I always thought the tabs on those plugs was a stupid design. They either don't release or always get caught when trying to untangle them...eventually getting snapped off in the fight. Infuriating.
Whoever invented them should have had a good head slapping.
Lemme predict the future... (Score:2)
Oh duh (Score:2)
So when the Cloud is down, everyone is F$%ked !!! (Score:2)
One should think they know better (Score:2)
Didn't they read their own site?