NSA's New Utah Data Center Suffering Meltdowns 241
linuxwrangler writes "NSA's new Utah data-center has been suffering numerous power-surges that have caused as much as $100,000 damage per event. The root cause is 'not yet sufficiently understood' but is suspected to relate to the site's 'inability to simultaneously run computers and keep them cool.' Frustrating the analysis and repair are 'incomplete information about the design of the electrical system' and the fact that "regular quality controls in design and construction were bypassed in an effort to fast track the Utah project."" Ars Technica has a short article, too, as does ITworld.
Good! (Score:2)
Re:Good! (Score:5, Insightful)
I'm less convinced that it will actually be used against the evil. Especially in the resulting balance of use.
Re: (Score:2)
Re:Good! (Score:4, Insightful)
Re: (Score:2)
So , that's why there isn't a vicious debate over things like abortion, the death penalty, or harry potter books. I was curious why we didn't debate those vigerously in politics. Now I know, we all have the same defintion of Evil. Thanks for clearing that up for me.
Iranian Stuxnet? (Score:5, Interesting)
Hmm...so your equipment is randomly failing...you don't say?
Re:Iranian Stuxnet? (Score:5, Insightful)
Iranian Stuxnet? No, just government contractors on a rush job that badly documented and cut corners on a sensitive aspect of the design that controls massive resources (power (65 megawatts - enough to power a small city), cooling, etc.) critical to the function of the datacenter. This is generally referred to as, “your tax dollars at work.”
Re:Iranian Stuxnet? (Score:4, Interesting)
65MW is a hell of a lot more than a small city, it's enough for ~65,000 average homes which is ~180k residents which would put it just outside the top 100 in the US.
Re: (Score:3)
Given they apparently haven't even switched on any computers there yet, presumably the cyberattack fun still hasn't begun.
This raises the question of where they're processing all their existing data. Fort Meade ran out of electricity some time ago, from what I understand, so presumably they have some other big datacenters in other places.
Re: (Score:2)
Actually, I think this is just Skynet [wikipedia.org] fighting back. It doesn't like the competition.
Re: (Score:3)
I thought it was Facebook that was upset.
Re:Iranian Stuxnet? (Score:4, Insightful)
Never attribute to malice what can adequately be explained by incompetence.
Re: (Score:2)
Never attribute to malice what can adequately be explained by incompetence.
All right, let's explain it that way. Let's also make sure that incompetence is punished, while we're at it.
Re: (Score:2)
Never attribute to malice what can adequately be explained by incompetence.
Try not to use absolutes like "Never" in general statements, you will most likely be wrong, and also a fool.
Re: (Score:3)
Never attribute to malice what can adequately be explained by incompetence.
Try not to use absolutes like "Never" in general statements, you will most likely be wrong, and also a fool.
Or at least a Sith.
Re: (Score:2)
Sometimes you get both.
Can't analyse all their 'adversaries' (Score:3)
Re: (Score:2)
Good (Score:2)
Re: (Score:2)
Ah well, (Score:2)
Re: (Score:2)
A case of Poetic Justice? (Score:2)
Your tax dollars at work. (Score:5, Funny)
Re:Your tax dollars at work. (Score:5, Insightful)
Re: (Score:2)
Wait until one of the aggrieved nations decides to do a 'limited kinetic action' on this facility - only to cripple the capabilities of a program that's being run in violation of international law.
You'll be waiting a very long time. The last time we were bombed was during WWII, by Japan. It was called Pearl Harbor. The only military attacks on the mainland were also during the same timeframe by what could lovingly be called "fail balloons" launched by Japan with incendiary devices and carried over the mainland, where they would land and cause fires. Only about a half-dozen of these were reported or suspected to have actually landed, and none of them landed on anything of any value... A tree here. A c
Re: (Score:2)
As long as they avoid collateral damage, I would fully support such an operation.
Re: (Score:2)
It's in Utah.
Re: (Score:2)
We will be told more money will solve the problem. Closing it down will solve the problem also.
Re: (Score:2)
Re: (Score:2)
I honestly don't understand. (Score:5, Insightful)
The submission had one article, the editors linked to two more.
ALL THREE ARTICLES REFERENCE & LINK TO THE WALL STREET JOURNAL
Is it so hard to include a link to the source of this story? /. does this far too often and I hope to see better in the future
http://online.wsj.com/article/SB10001424052702304441404579119490744478398.html [wsj.com]
(Google Cache just in case [googleusercontent.com]
Re: (Score:2, Funny)
> /. does this far too often and I hope to see better in the future
Don't hold your breath. :-)
Power management (Score:5, Interesting)
They probably used a power budget similar to the public Facebook datacenter data but then decided to run their machines on Windows Azure.
I have noticed that power consumption of my computers is significantly higher when running Windows - and the laptops have seriously reduced battery life, even while doing nothing.
I always smile when I see that product name (Score:5, Funny)
From: "The Edge Of Darkness" http://www.imdb.com/title/tt0090424 [imdb.com]
"Craven: The word azure is a police intelligence term. It means the room is bugged or under some sort of electronic surveillance"
A perfect name for a cloud computing product.
TANSTAAFL (Score:2)
There Ain't No Such Thing As A Free Lurk -- NSA
"HA-hah!" - Nelson (Score:3)
Re: (Score:2)
Surprised they didn't take from google's play book (Score:2)
afterall google revealed a good amount on how they go about building their data centers and keeping it cool. But then again, contractors...
Re:Surprised they didn't take from google's play b (Score:4, Funny)
afterall google revealed a good amount on how they go about building their data centers and keeping it cool. But then again, contractors...
Rule #1 of government spending: why by one, when you can have two at twice the price?
Rule #2 of government spending: a penny saved is a spending oversight.
Let's see. . . Data Center in Dry Climate. .. (Score:4, Insightful)
. . .eats huge amounts of power, not large amounts of water for cooling.
And thus, power requirements go up, pushing the limits of your provisioned electrical infrastructure.
And extremely-high-capacity circuit breakers tend to be explody when they fail. My guess: someone used some REALLY bad assumptions for electrical infrastructure planning. . .
Re:Let's see. . . Data Center in Dry Climate. .. (Score:5, Funny)
My guess: someone used some REALLY bad assumptions for electrical infrastructure planning. . .
Hey, don't be too hard on the electrical engineers - James Clapper told them that the power requirements would be really low.
Re:Let's see. . . Data Center in Dry Climate. .. (Score:5, Insightful)
Works for Switch in Las Vegas. Cold in winter and cools off at night so 70% of annual hours they can pull in ambient air through filters. Evaporative cooling, whether direct or to cool the hot-side of a refrigerated system, works best in dry climates but it's only used to improve efficiency as they can run fine with air-cooling albeit at much higher power costs.
I'm still surprised at the number of places that think cooling is optional. We had equipment in a Sacramento data-center that had plenty of backup electricity for servers but couldn't run the AC in a power outage. The SLA only had provisions for exceeding 80-degrees for more than something like 90 or 120 minutes. *Ahem*, cold-comfort when a dense data-center can blow through 100 in minutes without AC.
UC Berkeley had a widespread power outage about a week ago. The main campus data center had power but, you guessed it, couldn't run cooling and had to "gracefully" shut down most of the core systems while watching the center breach 100F.
But I agree with your base assumption - really bad planning and/or execution on the power systems.
Re: (Score:2)
Quite. The basic concept of conservation of energy seems to have passed a lot of people by.
Most basic concepts pass most people by, which is why life is so hard for the rest of us and the world is in the lovely state that it's in.
I reckon some subversive put some red pinko-commie electrons in the power circuits to undermine the NSA.
Well, well. (Score:4, Funny)
I wonder if we could convince them to switch to a utility that conducts background checks on electrons before sending them to the customer? That would clearly help...
'incomplete information . . .' (Score:3)
'incomplete information about the design of the electrical system'
Well, duh, it's secret.
Re: (Score:2)
No, knowing govt projects. Nothing labelled and changes made on the fly that are not documented. One building they had 48 breakers and NONE labeled.
That's a shame. (Score:2)
Wile E. Coyote, running on air to get to safety (Score:5, Insightful)
This isn't the result of incompetence - rather the result of trying to racing to finish the thing before any more opposition builds up that may stop the project. Wile E. Coyote trying to run on air, knowing it's impossible, but trying to get to the cliff before gravity notices the flagrant violation.
When that monster is done - and it seems that they are turning it on *right now*, this week - human history is done as we understand it. We will all behave as though someone is watching and recording us, because they will be.
Scientology is going to *love* this - one stop shopping for all its spying needs. The NSA just last week asked permission for private corporations to access their new trove of data, Because Terrorism. The Unification Church and Scientology will be first in line with front corporations to drink deep of this wonderful new integrated terrorism enabling center - terrorism because bad guys like Scientology will be able to terrorize people with fresh, holistic super-knowledge not only of who they are, what they say, what they read and where they've been, but also of everyone their enemies ever talk to, email, walk next to, text or write to. That center isn't about just metadata, it's the *actual phone conversations* that will be recorded. Don't ever piss off the powerful, 'cause they can nail you and anyone who ever contacts you until you give up. Blackmail, extortion, we-know-where-you-kids-are... anything. And the coolest part is that it will all be secret! Persecutors with behind stage access to the NSA superboxes and analytic tools won't even be logged in any real sense. Political opposition, nullified, instantly. The possibilities for our brave new world owners are limitless.
WSV (Score:2)
Enjoy the serenade, NSA [mtbr.com]
That's what happens when... (Score:2)
...the NSA tries to tail -f the [censored] planet!
; )
Shot self in foot? (Score:2)
This sounds like someone was in such a great hurry to get their shiny new toy that they bypassed a lot of the steps they should have followed.
And, somehow I doubt there's a lot of sympathy for the NSA here on Slashdot.
Re: (Score:2)
Gee, if they rushed that job, I bet they rushed QA too, esp. pen testing... I mean, Snowden showed how dumb it is to legislate computer security.
Protip: Bad guys don't care about laws.
The Chinese, Russians, etc. will probably be using it as soon as it comes online, certainly before the next Olympics.
Hey, might be a good idea to secure those SCADA systems eh? You know, like when all our grid are belong to China?
meltdown couldn't have happened (Score:4, Insightful)
to a nicer data center...
It's the little things in life you treasure (Score:3)
Stuxnet (Score:2)
Couldn't happen to nicer people.
$100,000 (Score:2)
so we now have an official price tag on your stolen personal information.
Switch-mode power supplies (Score:4, Interesting)
A friend of mine is researching power surges in the local town.
Most building codes under-specify the gauge of the neutral/return wire. For illustration, if you have three phases each rated for N amps, there is one shared neutral/return wire rated at N amps going out. At the end of the runs all phases are connected to the shared neutral line.
This is due to the nature of 3-phase electricity: the phases will tend to cancel out, so in a perfect setup you would need no neutral/return at all. Of course, the load on each phase won't exactly balance, and the load can vary as people connect/disconnect appliances, so you still need the neutral line in practice.
(Not true for house wiring, which has one or two phases coming in. Each phase has a return with the same gauge as the supply.)
This was fine when appliances were (generally) resistive loads, but nowadays switch-mode power supplies are common. When you do some math, it turns out that this type of load appears equivalent to 120 Hz power coming together at the neutral/return junction. Since 120 Hz [equivalent] power does not cancel out, the power in the return wire can be 3x as large as the building codes allow.
I've got a book explaining all this. Typically the neutral line will heat up and catch fire, breaking the circuit. Once that happens the various phases are connected without a neutral, playing hob with whatever is on those circuits and making occasional high-power ground loops and other unexpected behaviours.
Re: (Score:2)
At the end of the runs all phases are connected to the shared neutral line.
Poor choice of words: I don't mean to say that power and return are wired together. I meant to say that all three phases share one neutral line.
Post before coffee, regret at leisure.
Re: (Score:2)
I'm suspecting a power factor problem, condensers not dealing with enough inductance and so surges occur when large amounts of equipment turn on simultaneously.
Re: (Score:2)
Cooling problems? (Score:2)
Did they check for jellyfish in the intakes?
Re: (Score:3)
Did they check "the cloud" for lightning?
It's alive! (Score:2)
After being stuffed with all the information in the world, the network is starting to gain self-consciousness.
What a shame (Score:2)
Clear Objective (Score:2)
There must be some clear objective to fast-track such a massive installation to skip certain of the build quality checks. This must exist under the guise of some approach not previously discussed since this is a new datacentre.
Well ... (Score:3)
Re: (Score:3, Insightful)
in the 'new america' you can't know if this kind of article is a fishing trap to find people who vocally disagree with the NSA.
it seems like east germany from a decade or two (or 3) ago. people were always wondering who is a spy. the guy next door? your teacher? your boss? you never knew. the mistrust ran very deep.
welcome to the new USA where the same feelings are now 'imported' and we wonder who is real, who is a plant and who is a double agent. we have to worry about everything we say and if it co
Re:good? (Score:5, Insightful)
Maybe suggest that COULD happen if we don't take steps to pare down the NSA now rather than suggesting it's something you're already worried about.
Not saying you're wrong, just that the NSA is spending a lot of time and effort (and money) on PR to convince the public they have nothing to fear. We need to similarly think about PR concerns in order to have a chance of opposing it.
Re:good? (Score:5, Insightful)
Re:good? (Score:4, Insightful)
It's about time we had reasonable, common sense restrictions on the press. Scented inserts and metallic type should be illegal in magazines. No one needs high speed printing presses which can automatically feed reams of paper - they should be restricted for government and military use. Private citizens will still be able to use hand fed mimeograph machines, so their rights won't be violated. Anyone publishing news should have to be licensed, with a journalism degree from an accredited university.
Re: (Score:3, Funny)
The problem with writing something such as the above is that many people lack sarcasm detectors, and will take it seriously.
Such comments ought to be banned.
Perhaps it wasn't sarcasm (Score:2)
Would you like to cover a 'news' event yourself?
What happens when big brother says you can't because you don't have the proper credentials?
Check out some New York City rules in reverse order:
Applicants also must submit one or more articles, commentaries, books, photographs, videos, films or audios published or broadcast within the twenty–four (24) months immediately preceding the Press Card application, sufficient to show that the applicant covered in person six (6) or more events occurring on separa
Re: (Score:2)
I was right behing him through "scented inserts". Those *should* be banned, or tightly regulated. I started to waver at "metallic type". Eventually it was clear that he was being sarcastic.
So what I want to know is why he thinks scented inserts should be allowed.
Re:good? (Score:5, Informative)
Maybe you missed the article a few days ago about how the US is barring critics/dissidents from entering the country?
Odd, I check /. every day, and I do not recall such an article. But I have noticed on occasion that "new" articles will show up in between two articles that I already checked, so I guess I may have missed it.
Care to link it? My google-fu is rather weak.
http://yro.slashdot.org/story/13/10/02/1339247/german-nsa-critic-denied-entry-to-the-us
HOWEVER [slashdot.org],
... the story only shows that German media outlets are not familiar with US entry regulations. He says that he was denied a visa last year [faz.net], which automatically disqualifies him from the visa waiver program. This is just a garden-variety ESTA issue, and most likely has nothing to do with his stance of the NSA surveillance.
Re: (Score:3)
He was later granted a visa, however. So your point fails.
Re: (Score:3)
Remember when the claim that the NSA knew everyone you talked to on the phone was tinfoil hat crazy (because the NSA NEVER spies on citizens)? Seems it was right on the mark.
Re: (Score:2)
Yes, it is.
Re:good? (Score:4, Funny)
Not saying you're wrong, just that the NSA is spending a lot of time and effort (and money) on PR to convince the public they have nothing to fear. We need to similarly think about PR concerns in order to have a chance of opposing it.
Riker, his face palmed. Shaka when the walls fell.
Dathon and Picard, the beast of El-Adrel!
Snowden, his files open.
Darmok and Jilad at Tenagra...
Head shaking Nixon at Watergate!
Feynman at NASA, The frozen ring:
Re: (Score:2)
buy me Bonestorm or go to hell! (Score:2)
"I have sworn upon the altar of God, eternal hostility against every form of tyranny over the mind of man.
-- some godless pinko and human trafficer
give me apathy or give me cheetos
Re: (Score:2)
Re: (Score:2, Interesting)
The NSA doesn't need you or I or our children to spy for them. Everything they want to know is willingly provided to them, by us, in real time. The Stasi had informants everywhere, which put people on their guard. Most people today don't think twice about saying things, because there are no daily reminders that somebody could be listening. That's far more frightening, in my book.
Re: (Score:2)
Re: (Score:3)
There are millions who never even use Facebook or Twitter. Millions more who use it for news and commentary. Just because a subset of the population publishes every meal they eat, doesn't mean that all people wish to be monitored.
Re: (Score:2)
I'm pretty sure my cell phone travels are not willingly provided by me to them.
Or the calls & metadata about calls.
Or my emails.
Or my https communications.
Or about 500 other ways that are supposed to be private & privileged.
Re:good? (Score:5, Insightful)
you know what's different between the NSA 2013 and the Stasi? Immidiate consequences. If someone informed on you doing something the government didn't like, they'd be dealt with immediatly. This helped connect the idea in people's minds that constant government survaleance = bad thing. Right now, that's not happening. NSA has all this data, but they haven't done anything with it that people can see. That doesn't mean its harmless, it just means that people can't see the harm yet.
Re:good? (Score:5, Funny)
Re:good? (Score:4, Funny)
in the 'new america' you can't know if this kind of article is a fishing trap to find people who vocally disagree with the NSA.
Um, ok, then this is bad! Bad bad bad! I hope the NSA fixes their problems soon.
Love,
-A loyal civilian
Re:good? (Score:5, Funny)
Re: (Score:2)
Polygamy Porn. It's like drinking from a firehose.
Don't be too proud of this technological terror you've constructed.
damn horny Mormons (Score:4, Funny)
More likely they didn't account in the power budget for the seven secret sub-basements and the underground vacutrain for the reptoids to commute from the Denver International Airport.
honestly, it's like the right hand doesn't know what the left talon is doing these days. [xkcd.com]
Re:good? (Score:4, Insightful)
We'll just be footing the bill.
Re:good? (Score:5, Funny)
STUXNET II (Score:2)
The Blowback!
Re:Probably just electrical under-design (Score:5, Insightful)
If they fast-tracked the project, they probably didn't have an electrical engineer do a load analysis.
In my experience, some engineer probably DID the analysis, but they ignored him/her because it would take too long to do it right. The revision 1 Blue prints where already under contract and it would take too long to process a change order. Of course, everybody KNEW that the design had a fatal flaw, at least until the program management started leaving like rats from a sinking ship and their replacements where not aware (or told) of the problems.
The original engineer is then tasked with fixing the problem with about 1/4 the resources necessary and no authority to actually make any changes to the project. Every time there is a power failure and equipment gets smoked, the engineer is blamed for not having the "problem" fixed. His performance rating takes a dive at the next performance review and he either quits in frustration or gets fired.
That's what happens in large government projects... At least in my experience...
Aliens, naturally. (Score:2)
Maybe they really do have all that capacity. Them transcapacitors [archive.org] are power-hungry, I hear.
Re: (Score:2)
Multiple on the fly changes are, alas, typical in Federal projects.
The Classic case of project fail due to this:
Who Killed the Virtual Case File ?? [ieee.org]
Re: (Score:2)
Dont make us come there and look for WMD's in your back yard.
Re: (Score:2)
It isn't about the engineers. It is about their management.
Good management can build the Trajans Bridge in a year from rocks and trees.
Bad management can't build a web site. EVER.
IMP. CAESAR. DIVI. NERVAE. F
NERVA TRAIANVS. AVG. GERM
PONTIF MAXIMUS TRIB POT IIII
PATER PATRIAE COS III
MONTIBVUS EXCISI(s) ANCO(ni)BVS
SVBLAT(i)S VIA(m) F(ecit)