Can Commercial Storage Services Handle the NSA's Metadata? 67
itwbennett writes "In a review of NSA surveillance last month, President Obama called for a new approach on telephony metadata that will 'establish a mechanism that preserves the capabilities we need without the government holding this bulk metadata.' Obama said that a third party holding all the data in a single, consolidated database would be essentially doing what is a government function, and may not increase public confidence that its privacy is being protected. Now, an RFI (request for information) has been posted to get information on U.S. industry's commercially available capabilities, so that the government can investigate alternative approaches."
And? (Score:3)
And what if some commercial storage vendor can't or won't handle the NSA's metadata archiving requirements?
Re:And? (Score:1)
or keep it secure
http://science.slashdot.org/st... [slashdot.org]
Re:And? (Score:0)
What about Beta? What if they tried to make Beta successful and deployed it on something like this? What then?
metaBETA (Score:0)
Yeah, Mod parent up. Beta sucks!
Fire timothy "timmyboy" while you're at it.
Re:And? (Score:5, Funny)
Re:And? (Score:1)
We will just have to give the NSA even more money to develop the storage technology they need to spy on us.
Re:And? (Score:1)
and so-called "metadata" is only the NSA's selling point. they are also capturing boatloads of actual content under the legal principle "we can and nobody can stop us".
Re:And? (Score:0)
And what if some commercial storage vendor can't or won't handle the NSA's metadata archiving requirements?
Who cares? Obama has a pen and a phone and can just fix it with another executive order.
Give it to a private contractor. In Hawaii. (Score:5, Funny)
Re:Give it to a private contractor. In Hawaii. (Score:2)
It would be easier to just sub it out to China. It'll save them the bother of breaking into the servers.
Re:Give it to a private contractor. In Hawaii. (Score:1)
Not really a technology problem (Score:4, Interesting)
This is less of a technology problem than a policy question. The technology exists to build secure databases and make it accessible to only one remote client. The real controversy is over collecting the data, and who holds it. Private companies don't want to do it. Many are against the NSA, and by extension the Federal government doing it. If only there was somewhere in the middle, between the Federal government and private industry...
Re:Not really a technology problem (Score:3)
Well, one of the numerous problems with this whole situation is we can't rely on anything the govt, or the companies involved, have to say. Are these companies really against this, or do they just see the need to pretend to publicly? And even if they really are against it, would that change for sufficient compensation?
Either way, privatization is not going to make the underlying problems (such as much of the program being unconstitutional) go away.
Re:Not really a technology problem (Score:1)
This is less of a technology problem than a policy question. The technology exists to build secure databases and make it accessible to only one remote client. The real controversy is over collecting the data, and who holds it. Private companies don't want to do it. Many are against the NSA, and by extension the Federal government doing it. If only there was somewhere in the middle, between the Federal government and private industry...
How about nobody collect and store this so-called metadata? Too radical an idea for you and your government-centric, corporatist-centric worldview? Tough.
Re:Not really a technology problem (Score:2)
Re:Not really a technology problem (Score:0)
The ground between the federal government and private industry are...
The states.
Yes (Score:5, Insightful)
Given enough money.
Once the USA government asks for bids on this, you will get many companies wanting a share on this juicy contract. This is supposed to be with the intention of increasing security, but just wait a couple of years and stories will start to pop up as to how corners have been cut to turn a few extra dollars with the result that this data becomes available to all sorts.
Re:Yes (Score:5, Insightful)
Well, I'm sure one of the usual defense contractors built all the stuff the NSA is using in the first place, so having one build and run it someplace else doesn't seem like a problem. It just doesn't really seem like a solution either. How does moving around the lines on the org chart fix this issue?
Re:Yes (Score:2)
Re:Yes (Score:2)
Usually I would agree with this assessment but in this case not so much. The administrations responses to the public concern have been half measures at best.
I think the NSA does not really want to give up the data, and the Administration does not want to make but wants to be able to say they did something.
Clearly the plan here is for the NSA to tinker with the 'requirements' until nobody can meet them and use this as an excuse to delay any real changes indefinitely; meanwhile Obummer gets to sit back and say its being worked on.
Re:Yes (Score:2)
Re:Yes (Score:0)
It's Miss Manning you insensitive clod
Just what we needed! (Score:0)
Instead of a government theoretically beholden to the Constitution theoretically being held responsible for their actions, we get a corporation practically beholden to nobody but its shareholders selling the information to practically all comers (LexisNexis [krebsonsecurity.com], anyone [law360.com]?)
In theory, this is shit. In practice, it's worse.
Why even consider it? (Score:5, Insightful)
It's a bluff. A feint. A thinly veiled threat. It's not intended to actually come to pass. One of the things Obama proposed is to move the keys to the friggin kingdom from government controlled servers to nebulous "third parties". And in the very same damn speech he pointed out how this would be a ludicrously bad idea.
(Well, I mean, he also suggested that the telcom companies who move this data keep it until the NSA asks for it. That or third parties. I don't mean to harp on a stray comment or anything.)
But let me spell out the subtext here for anyone that can't read between the lines: If you try and keep the government from storing this data, we'll just go find someone else to hold it. And my, my, my, doesn't that sound just simply horrible? Be a REAL SHAME if someone were to try and enforce that 4th amendment 'round here.
Also, fuck beta. I have no way to tell if someone responded to me other than looking at that specific thread.
It's already being done. (Score:0)
If you try and keep the government from storing this data, we'll just go find someone else to hold it. And my, my, my, doesn't that sound just simply horrible?
Back in the 90s, I dated a lawyer. She said that with someone's SSN, she could find out everything about them.
Today in '14, not only do we have people voluntarily broadcasting their personal details, but the financial, medical, and retail industries has an obscene amount of data on all of us. And it's aggregated already by the credit bureaus and companies like ChoicePoint as well as search engines like Google who can do it on the fly. And the Medical Information Bureau has your health history. The phones companies have your calling history.
And let's not forget the intrusive information gathering by our governments.
Anyone who claims that your information is confidential is only speaking about some schmoe off the street. If you pay for it or send a real scary letter with law enforcement letterhead, all bets are off.
So, my point? The NSA's plan is redundant. It's be much cheaper just to force the above companies to do their bidding.
And if Google (Brin) or whoever don't like it - oops! Some N. Korean or Russian submarine mistook his yacht for a terrorist launching point. Our bad!
Re:Why even consider it? (Score:1)
But let me spell out the subtext here for anyone that can't read between the lines: If you try and keep the government from storing this data, we'll just go find someone else to hold it.
Nice attempt at misdirection, but the gov holding the data is only worse than the gov having access to said data to begin with, which is the real issue. 4th, 9th, and 10th Amendments and all, ya know
Re:Why even consider it? (Score:2)
Also, fuck beta. I have no way to tell if someone responded to me other than looking at that specific thread.
I know. That is the worst feature of the beta by far. Lacking direct navigation to comment threads from users' comments pages is a egregious omission.
USP (Score:0)
Ask a USENET service provider like giganews, they know the drill.
Here is the solution America... (Score:2)
This entire system is so f-ed up (Score:5, Insightful)
OK, so they want to store everything passing across the lines that they deem suspicious, promise us that no one will look at it with a warrant, then if you're ever suspected of something they can go back and find all your communications over the past X years. And, since the feds don't want the blame for holding onto this information (and looking as Big Brother-ish as they are), they want private industry to pony up the disk space? I'd almost trust the NSA more to house this info since they'll only snoop in on my conversations when I post/say a flagged word/phrase. Wheraeas I KNOW private companies will as soon as they figure out how they can commoditize it.
It's Orwellian enough seeing Google spam me with ads based on my email conversations.
Re:This entire system is so f-ed up (Score:4, Interesting)
OK, so they want to store everything passing across the lines that they deem suspicious,
No. Not really.
They really do want to store everything passing across the lines. Period. The "deeming suspicious" part only comes into play once they get a warrant to go look at the data they've already collected and stored.
The up-side to this idea is that the NSA isn't holding onto the data that they promise they're not looking at without a warrant. That's about it.
The down-side to this is that we SURE AS SHIT can't trust a third party to not look in the box. This third party is also implicitly alerted to who the NSA is investigating and when. That information alone is itself sensitive and not the sort of thing to be trusted to a third party.
Of course, you know, I guess I could extrapolate my answer and cut down your sentence even further:
OK, so they want to store everything
Re:This entire system is so f-ed up (Score:2)
aka PreCrime.
Re:This entire system is so f-ed up (Score:0)
It's Orwellian enough seeing Google spam me with ads based on my email conversations.
If you're using GMail, you're part of the problem.
It doesn't matter. (Score:3)
Re:It doesn't matter. (Score:0)
Be sure to use ROT13 encoding. I hear it's unbreakable...
Wrong question. (Score:5, Insightful)
Trouble is, that was never the fucking point. Do people want the NSA collecting a giant database about them? No. Does it make the slightest difference if the giant database is nominally Verizon's giant database, that just so happens to respond to all queries from the NSA? Aside from the greater likelihood that the database will be used for marketing and surveillance, not a bit. The ostensible '3rd party' won't remain at arm's length for long. Why would they? An entire organization with a single customer, dedicated to shovelling data toward them on command? Instant capture. The only time the 3rd party will be 'independent' is if somebody asks the NSA what that 3rd party is up to, in which case they'll oh-so-innocently-have-no-idea-what-that-independent-entity-does. For all other purposes, they'll be joined at the hip.
slashdot is trying to desensitize (Score:0)
and make the NSA's data collection program seem "cool", as if it's a cool technical consideration on how much storage it will take to store everyone's private conversation.
Beta is not the problem. While you turds complain about beta, these Psy-ops make their way to the front page of slashdot.
Re:Wrong question. (Score:2)
>Do people want the NSA collecting a giant database about them?
No.
> Does it make the slightest difference if the giant database is nominally Verizon's giant database, that just so happens to respond to all queries from the NSA?
Yes. Because this, if nothing else, creates a paper trail and at least a properly worded query to the database, whereas currently (as Snowden demonstrated) anyone with a modicum of coding experience can download the whole thing and make off with it and no one's the wiser.
>Aside from the greater likelihood that the database will be used for marketing and surveillance, not a bit
You realize there is nothing stopping companies from using this for marketing right now, anyway, right?
Beta first? (Score:-1)
Is the NSA going to beta their ressources first?
Great (Score:-1)
What do I read now that Beta sucks?
Interesting (Score:0)
Another way to illegally store people's information. I guess being the president he feels he doesn't have to follow the law or is subject to being held criminally responsible.
Dont keep it at all. (Score:1)
The problem isn't where the collected data is stored. The problem is that it is being collected. There is no reason that the bulk metadata of every phone call made in the US is stored for years or indefinitely. There is no need for this RAW data to be shared with other countries. So where it is kept makes no difference. This data shouldnt be kept at all, and from every independent analysis of the program it has had NO impact on fighting terrorism. So it is a colossal breach of the constitution and a massive waste of money and resources.
Well Certainly... (Score:2)
...well enough to be leaked.
Comment removed (Score:1)
Logic problem (Score:0)
The problem isn't who holds and maintains the data. The problem is that some entity is collecting in advance protected data.
The real method should be if a person has a signed court order granting an investigation against them then you can begin monitoring.
Innocent until proven guilty; Unmonitored until signed court order has been issued.
Please update laws accordingly thank you.
Doesn't solve any of the problems (Score:0)
Now you have to worry about the security of the third party. I give my credit card to a third party (say Target), Target follows the rules on what can be persisted, and yet you still have a data breach.
god, people are retards.. (Score:3, Funny)
The meta-data information provided by the President is a fucking cover story for hiding their spy games program. It's already been exposed that they are doing much more than saving meta-data; they're collecting word for word, every communication domestically and foreign, saving the content of our communications.
Lets focus on the meta-data for a minute thing: according to Bill Binney, previous NSA director on technology that helped design the system, anybody can store meta-data and equipment that fits inside a 20 by 12 foot room. FOR ALL COMMUNICATIONS, WORLD WIDE. So of course Verizon, AT&T, and these others douches can store this information. In a room probably the size of 5 by 5, because they'll be storing it themselves ; and providers are already storing this information anyway, which has been available for law enforcement use for some time. The Bluffdale data center in Utah is big enough to store 100 years of content data though, .. which means they're using it to store actual profiles and content of people, not just meta-data. Details @ http://www.pbs.org/newshour/bb... [pbs.org] "NSA Collects ‘Word for Word’ Every Domestic Communication, Says Former Analyst"
On top of that, they have a massive satellite and radar system with a variety of capabilities, which is being used to target Americans during continuous black operations. Mind reading capability, tracking from space, watching our movements wherever we are. look at the details @ http://www.oregonstatehospital... [oregonstatehospital.net]
Re:god, people are retards.. (Score:1)
Here's a few revelent articles: Phone companies already record and log all 'meta-data' and have for decades. Law enforcement have had full access to it through court-orders, warrants, etc. Generally, information is kept by phone companies for a period up to or a minimum of 3 years.
http://gizmodo.com/5795861/how... [gizmodo.com] ("How the police get your phone records" written, 2011)
https://www.aclu.org/blog/tech... [aclu.org] ("How Long Is Your Cell Phone Company Hanging On To Your Data?", 2011): this article covers cell phone only. Generally information is saved for 1 year minimum, but some carriers save it longer.
Re:god, people are retards.. (Score:1)
Number Calls = (330 X 10^6 People) X (3 Calls / Person / Day) = 1 X 10^9 Calls / Day
Assume each call lasts for 1 minute.
Seconds of Content = (1 X 10^9 Calls / Day) X (1 Minute / Call) X (60 Seconds / Minute) = 60 x 10^9 Seconds
Call audio data can be handled with a 4 KHz cutoff. It takes two samples per Hz to capture this data. Assume 2 Bytes per sample (actually too high).
Bytes per Day = (60 X 10^9 Seconds of Content) X (4 X 10^3 / Second) X (2 Samples) X (2 Bytes / Sample) = 9.6 X 10^14
Or, about 10^15 Bytes per Day to store raw content. One PetaByte. For perspective, this is just 1000 1 TB hard drives. The Utah facility has a capacity of about 30 ExaBytes, or 30 X 10^18 Bytes. This means that Utah could save about 30,000 days of U.S. content.
Lies for the sheeple- promoted by Slashdot (Score:-1)
The owners of Slashdot will accelerate their promotion of pro-NSA FUD in the foreseeable future.
The NSA doesn't store metadata- it stores ALL possible acquirable data on giant datacentres that use the EXACT same hardware and software designs that you will find in Google's own installations. The 'metadata 'meme' is the 'meme' mainstream media and George Soros controlled fake-indy media organs are requested to promote.
It is well understood that despite the clear and descriptive reports from Snowden, illustrating the fact that the intelligence agencies of the West actively seek to collect all possible information about every Human on the planet, and have unlimited technical budgets to achieve this end- the sheeple will forget they ever knew this fact within a year or so, if a sustained campaign of propaganda replaces the truth with something else.
Slashdot, and its army of vile shills, have told you here for years that the suggestion of excess by intelligence agencies was but the fever-dream of nut-case tinfoil hat wearing conspiracy nuts.
-Slashdot shills said "why would they be interesting in you?"
-Slashdot shills said "everyone knows the government is too incompetent to create such spy systems"
-Slashdot shills said "everyone knows they is no governments money for such extensive projects"
Slashdot shills said whatever they thought betas would believe.
And let me point out, before Snowden, not one of these programs was public knowledge. Strange eh, when the owners of Slashdot and their army of shills tell you that everything leaks from the government all the time. Snowden refers to NSA events where other intelligence agencies and private contractors were present as the programs were explicitly described in powerpoint presentations. And yet, pre-Snowden, not one program name leaked to the wider public.
And the same vile shills tell you that the false-flag of 9/11, on which EVERY act of abusive spying and aggressive warfare by the US government is based, couldn't have been a false-flag because the 'government' can't keep 'secrets'. And you, the dribbling betas, actually believe these endless liars.
And now you dribblers are dribbling about the laughable lie of 'external private data-storage contractors".
"We need to distract these morons"
"Simple- you take the laughable lie of 'we only collect and store metadata at worst' and give it a fake reality by creating a whole circus about who is paid to store this metadata"
"Are even American betas so thick that they'll fall for such a transparent ploy?"
"Look, they are not called 'chattering classes' for no reason. Give these dribblers something to chew on. Makes them feel 'powerful', 'significant'. And then, they are back to looking in every direction but the actual place we chose to operate."
"But what about the accurate descriptions of what we really do provided by Snowden?"
"These betas are PROUD to only get their knowledge from outlets we control. In a year, or two tops, OUR propaganda will have completely displaced the truth from Snowden. We'll paint ourselves 'fools' and 'incompetents' so the dribbling betas can laugh at us, and by doing so they'll take our lies as the real truth."
s/can/should (Score:2)
No problem (Score:2)
The world is globaliszed, don'tchaknow? I'll bet some Chinese firm would have *no* trouble offering to host the outsourcing of the data storage....
mark "on Chinese-made chips...."
What's The Worst That Can Happen? (Score:2)
Sure. Let's not shut down the horrible program that a ton of people oppose and instead hand the data over to a company to manage and keep secure. What's the worst that can happen?
Off the top of my head:
1 - Hackings. No database is secure. If anyone was to store the data securely (putting aside for the moment the question of whether they should have the data in the first place), I'd trust the NSA to do it over some random company. At the very least, this reduces the potential attack vectors.
2 - Profits. The company controls this data and realizes that they could make a ton of money off of it. Their federal contract might forbid it, but that's easily handled with a few lobbyists and sneaky riders on must-pass bills. Now, they can sell information to third parties legally. Maybe it's aggregate data/not personally identifiable (at least, at first to reduce any opposition) and maybe not. Either way, this information is now leaking out.
The answer to all of this, of course, is the answer to the question "Why does the NSA need to store metadata on EVERYONE?" They don't. However, they have fallen victim to a combination of lust for power and a "information gathering" fallacy. (Collecting some information proves useful against terrorists therefore collecting ALL THE DATA will prevent all the attacks. Except that they've just increased their signal to noise ratio to the point that they can't spot the tiny number of terrorist signals within all of the random noise.) If they scaled the program back to only collect metadata on a very limited number of individuals (proven to a judge enough to issue a warrant and with checks and balances to prevent abuse), they would have a higher signal to noise ratio and might actually catch more terrorists than from a random sweep.
Seriously? You people are NUTS! (Score:2)
I would assume that the methods used to collect this data are CLASSIFIED. Why else are they trying to get their hands on Snowden for leaking some of it?
IF you have classified information to store, you DON'T put it on third party systems unless they are under the necessary controls required to handle classified data. So, putting this data on contracted storage is NOT going to involve calling Amazon AWS for an account and just copy it up and pay the bill. So in reality you'd just be contracting somebody to build and run a storage solution for you.
Now *could* the government go out and *contract* with somebody to store their data someplace? Sure, it might even make sense to push it off to a number of contractors, but you NEVER, (and I mean NEVER) put classified data into public view (i.e. on systems you don't directly control), even encrypted, unless you have no choice. If you do, you are being STUPID. The more sensitive the information, the more this is true.
Assuming you don't use a one-time pad cypher, encryption doesn't mean that the adversary cannot read it only that they will have to break your encryption to see it. Brute forcing a key is *always* possible, the question is really "How Long" will it be, on average, before they will be able to view it, because they will eventually be able to.
As always, when the headline is a question mark... (Score:0)
The answer is "No."
Metadata (Score:3)
Seriously, all your data is perfectly safe. I have worked with GIS for 14 years. and I can tell your conclusively that absolutely no one reads metadata. :)
Just such a bad idea! (Score:0)
Let me list the ways. Oh, wait, there aren't enough bytes in the universe to expand this! 1. We only need to hack one site to get EVERYTHING! 2. See number 1.
Yes. (Score:0)
http://devnull-as-a-service.com/
I vote that these guys should get the contract.
Yes. (Score:0)
In the beginning was the Denelcor HEP. After Chapter 7, the chief scientist spent time in "Maryland" designing the successor. That became the Tera MTA. Tera became Cray, Inc, and the MTA system became the XMT. XMT was never a volume seller, but Cray would build one if you had the cash, and three-letter agencies did.
Now Yarcdata is a Cray subsidiary marketing it as a "graph appliance." See yarcdata.com.
Sub the job out to private industry to ensure ... (Score:1)
the entire Internet will have unfettered access to the data, without actually being able to access said data, thanks to the perpetual irreparable nature of the system's design. ... just visit http://404.nsa.gov
RFP (Score:1)
Yes. (Score:0)
That was easy. The commercial storage vendors do a very good job of pairing their spindles and flash with technologies from Oracle and various big-data vendors to make things work quite nicely. No issues at all. I've seen systems that can ingest many TB/hr. without problem.
A third party holding the data... (Score:2)
Wait... (Score:0)
Does anyone really think that this will happen?
Come on, they've (Big Govt) spent billions (or more) on data storage facilities and you really think they'll just close those down and let some 3rd party store data?
I have an escalator to the moon for sale..
What the CRAP is going on with these comments form (Score:0)
What the CRAP is going on with these comments formatting? Beta used to mean something was being improved but not ready for general use. Now it means crapping on users and destroying not only the super simple ease of use of a comments driven web site, but crapping on the users themselves in the process. Will someone show me how to change my homepage from slashdot to anything else!
Shouldn't have ... (Score:2)
Blowing smoke (Score:0)