Forgot your password?
typodupeerror
Government Data Storage Hardware

Obama Orders Federal Agencies To Digitize All Records 186

Posted by Soulskill
from the welcome-to-the-90s dept.
Lucas123 writes "President Obama this week issued a directive to all federal agencies to upgrade records management processes from paper-based systems that have been around since President Truman's administration to electronic records systems with Web 2.0 capabilities. Agencies have four months to come up with plans to improve their records keeping. Part of the directive is to have the National Archives and Records Administration store all long-term records and oversee electronic records management efforts in other agencies. Unfortunately, NARA doesn't have a stellar record itself (PDF) in rolling out electronic records projects. Earlier this year, due to cost overruns and project mismanagement, NARA announced it was ending a 10-year effort to create an electronic records archive."
This discussion has been archived. No new comments can be posted.

Obama Orders Federal Agencies To Digitize All Records

Comments Filter:
  • by wideBlueSkies (618979) * on Tuesday November 29, 2011 @11:21PM (#38210490) Journal

    So, how many Library of Congress equivalents worth of material are they intending to scan??

    • by Anonymous Coward on Tuesday November 29, 2011 @11:26PM (#38210526)

      At least 1.

    • by ron_ivi (607351)

      Brings up a great point -- why not have the Library of Congress manage it.

      The LoC is very good at managing digital content -- and making it searchable/available through partnerships with open source projects like the Univ of Michigan's Hathi Trust project:

      http://www.implu.com/federal_contracts/listing/LC-HathiTrust [implu.com]

      Could the National Archives and Records Administration outsource this project to the LoC?

      • by arkenian (1560563)
        So, NARAs problem is that there really are no standards, as of yet, for truly archival data storage mechanisms. For NARA to go purely digital, it has to be able to guarantee PERMANENT availability and accessibility of records. That's not, as has been discussed here before in other contexts, a trivial matter.
      • by sorak (246725)

        Does the LoC have adequate security? I may be wrong here, but I believe part of the NARA's job is to keep classified records (espionage and wikileaks fodder) so they can be released at a later date. I'm not sure if the LoC does anything with classified material.

  • by Jeremiah Cornelius (137) on Tuesday November 29, 2011 @11:21PM (#38210496) Homepage Journal

    When all the records are locked in 8x11 filing cabinets, sealed in Manila envelopes?

    And the FOIA headache!

    Destroying those records is hard, and some turn up - years after they were declared not to exist!

    • by rhizome (115711)

      They don't even want the garbagemen to know how much they're shredding.

      I'd say once Lockheed can actually implement this, you'll start seeing "now that we don't have to worry about paper records anymore" retention laws flowing through with YEAs.

    • by mirix (1649853)

      On one hand you've got images of the stasi shredding everything at headquarters as fast as they can... On the other, they never got hacked.

      I feel I should elaborate but I can't put it in words right now, so hopefully you get my drift.

  • by ChrisMaple (607946) on Tuesday November 29, 2011 @11:25PM (#38210520)
    I'd like to see 220 years of Congressional debates in digital form.
  • (the number of federal agencies)-odd number of completely incompatible digital records systems proposals.
  • by inKubus (199753) on Tuesday November 29, 2011 @11:29PM (#38210546) Homepage Journal

    This is actually the perfect place to incubate distributed object stores (e.g. Hadoop on one end, something like Zimbra on the other). One namespace .gov, with sub-namespaces. With a CMIS interface. Anyone see VMWare Project Octopus yet? Well, take that times 10,000 and you have a pretty nice records management system, platform independent. There's also Alfresco [alfresco.com] which is using the JCR spec which I believe can be moved to some type of distributed backend. But it implements CMIS, has a DoD spec records management system.. So the general spec would be a CMIS framework, each department/branch/whatever makes available a service for document retrieval, central .gov listing of the services, basically what Amazon does for literally everything it does. Do not compromise, executive order Jeff Bezos style, everything is a service with a public interface. I think it is possible, but it would take a lot of just plain buying in and our government (the bureaucratic, non-political side) has gotten really really good at dragging their feet and doing nothing. The cuts are coming though, and they will have to improve efficiency just like we all have in the private sector. Of course Defense is the worst, but education can use some work as well.

    • by oneiros27 (46144)

      You mean like OODT ( ) ? or something more like [apache.org] iRODS [irods.org] ? Both are used by various 'big data' groups (NASA, NIH, NOAA, NOAO, super computing centers) to share data across multiple sites.

      As for the indexes .... well, if science.gov [science.gov] and data.gov [data.gov] are any example, they could use some work. Although, hopefully in this case, you're describing bibliographic records, so the necessary metadata is a little more standardized.

      In some cases, I'd be better to just put the records out there under standardized open APIs,

  • Lockheed Martin (Score:5, Insightful)

    by dg41 (743918) on Tuesday November 29, 2011 @11:34PM (#38210592)
    Looking at the NARA article, as soon as I saw that some big IT contract was given to Lockheed Martin I saw all I needed to know about this initiative.
    • Looking at the NARA article, as soon as I saw that some big IT contract was given to Lockheed Martin I saw all I needed to know about this initiative.

      how much money did LM or someone closely associated with LM give to the present administration or someone closely associated with the present administration? Like "they" always say, follow the money.

  • by forkfail (228161) on Tuesday November 29, 2011 @11:42PM (#38210622)

    We must save our children's heritage. President Obama obviously hates America and it's legacy, otherwise, why would he be trying to destroy all the paper records? Undoubtedly, he'll claim that his long form birth certificate was destroyed during the digitization effort. It's obviously an Islamic socialist fascist communist ACORN black panther George Soros funded plot of some sort. Also.

    • by sorak (246725)

      Since this helps Lockheed Martin, I'm pretty sure the GOP will let it slide. They may make casual references to "Obama increasing scrutiny of US citizens", trying to portray it as an attempt to implement Orwellian Telescreens [wikipedia.org], but that will die out pretty quickly.

  • Archaeology (Score:4, Insightful)

    by GrahamCox (741991) on Tuesday November 29, 2011 @11:42PM (#38210628) Homepage
    In 1000 years or more, they'll have no idea what we were up to at all. At lease some paper records have a chance of surviving.
    • by Grave (8234)

      In a thousand years, if archaeologists cannot gather sufficient data from other observations besides paper records, then it really wasn't that important anyway.

      • by AgNO3 (878843)
        You are so right. Homer's works didn't help solve anything. Hieroglyphics didn't help use out at all. Cave paintings that where used to record historical events did nothing to help ups understand. Yup writing and graphical representations have done nothing to help the present understand the past what so ever.
    • by c0lo (1497653)

      In 1000 years or more, they'll have no idea what we were up to at all. At lease some paper records have a chance of surviving.

      I wouldn't worry that much... think of it... USofA has some pretty extraordinary archaeologists: Indiana Jones, Lara Croft, Rick O'Connell, Benjamin Gates [wikipedia.org]... should I continue?

    • by gl4ss (559668)

      oh really?

      what we are living is the _start_ of ridiculously accurate history, with video and images and schoolbooks all in neat packages for the people of the future to examine. 100 years ago contrast between year and year on that isn't going to be huge, but the contrast between now and just several decades ago is huge.

      never before has been so much information printed and recorded, never before have so many people lived who are doing their best to record information so that it's available later. never befo

    • by sorak (246725)

      It's just as easy to destroy a written document as an electronic one. The only way this information will be lost is if the powers-that-be intentionally destroy it, or if something so catastrophic occurs that the internet becomes a historic fad. In the internet age, information is a virus. The media may come and go, but the data will live on, so long as there is another remote system somewhere to copy it to.

  • While there is a certain amount of (justified) paranoia that the government would use digitizing records as an opportunity to engage in revisionist history, I have to say that despite a desire to do so, the odds are against the government being able to pull it off.

    In order for something like 1984's Ministry of Truth to function, the government would have to be far, far more competent and efficient than is ever to be likely.
    • So, we're relying on security through obscurity?
    • by AHuxley (892839)
      Thats why they can hand it over to the private sector. What the US gov cannot not collect, the private sector can share and build on.
      What the private sector cannot link over time, the US gov can do, medical, other govs.
      Any laws that stop the US gov, use private contractors or friendly govs outside the US e.g. Canada, UK.
      Databases are now very efficient, data entry is in place in most states in a shareable form.
      http://en.wikipedia.org/wiki/Information_Awareness_Office [wikipedia.org] showed the vision before it was los
  • by webdog314 (960286) on Tuesday November 29, 2011 @11:48PM (#38210676)

    Does that include the Declaration of Independence? I suppose it would be much easier to change in digital form...

    • Re: (Score:3, Insightful)

      by Anonymous Coward

      You mean the constitution?
      Or do you think they'll want to rejoin the British empire?

      • do you think they'll want to rejoin the British empire?

        That wouldn't be all bad. We'd at least be able to pawn off our debt on someone else.

    • Re: (Score:2, Insightful)

      The Declaration of Independence has no legal standing under the US Constitution (and hence US laws in general). For some reason conservatives cannot fit that idea into their heads.
      • by forkfail (228161)

        And yet that comment gets rec'ced up as being "insightful".

        Even more ironically, it's the same folks who love to talk about "life, liberty and the persuit of happiness" the most that seem to forget the bit about the next clause, "That to secure these rights, Governments are instituted among Men" more often than not.

        But then, and speaking of editing the Declaration of Independence, Texas did drop Jefferson from its textbooks:

        http://www.nytimes.com/2010/03/13/education/13texas.html [nytimes.com]

  • by actionbastard (1206160) on Tuesday November 29, 2011 @11:56PM (#38210718)
    So there isn't a repeat of this:
    http://www.archives.gov/st-louis/military-personnel/fire-1973.html [archives.gov]
  • by laing (303349) on Wednesday November 30, 2011 @12:18AM (#38210834)
    Why not outsource the whole task to somebody like Iron Mountain? They could get it done quickly and economically. It might even create a few jobs.
    • by SydShamino (547793) on Wednesday November 30, 2011 @12:37AM (#38210926)

      It was given out in a contract, so you are already getting your wish.

      Though I think we could save money by having the government do something itself instead of having to pay for Lockheed's profit and overhead.

      • by kiwimate (458274)

        Sorry, but I have to disagree. As I pointed out in another post, [slashdot.org] there are a lot of factors to think about when you do something like this, and if you don't have the experience you'll make mistakes.

        It's kind of like saying (my favorite distro) Linux/Windows 7 is so easy to set up these days that anyone can do it. If it's just a matter of clicking Next->Next->Next, then yes. And that might even be sufficient for a home computer (ignoring things like backups). But most people reading this will know ther

  • by MichaelCrawford (610140) on Wednesday November 30, 2011 @12:45AM (#38210958) Homepage Journal

    This would be a good time to write your congresscritter to point out the problems with undocumented file formats as well as Apis and network protocols.

    There are plenty of formats that could be used that are open and vendor neutral.

    If congress doesn't require that in it's funding authorization, many of our public records will be stored as word dos or in ms SQL databases.

  • by Rinisari (521266)

    IIRC, NARA didn't end the effort, it just stopped further development because it considered it complete.

  • by Temujin_12 (832986) on Wednesday November 30, 2011 @01:57AM (#38211234)

    Dunder Mifflin is gonna be pissed...

  • "And I want it to be implemented in less than four years. Then we can change all government records to show that presidents can have four terms in office. Then we'll change it to four decades. I call it my 4-4-4-4 plan. Get out your little red kindles, children. We're going to read about democracy."
  • by jlaprise1 (1042514) <j-laprise@northwestern.edu> on Wednesday November 30, 2011 @03:05AM (#38211488)

    As a professional historian who has worked in the National Archives in College Park, MD and at four different presidential libraries, which incidentally are also managed by NARA, I need to interject that this is an immense costly but valuable project.

    Remember "the warehouse" from the Indiana Jones movies? NARA is a little like that in terms of size but are better organized. Aisle upon aisle, shelf upon shelf, row upon row, room upon room, floor upon floor, building upon building of neatly indexed banker's boxes with labelled folders of documents. The labels may have been checked by the archivists at NARA, but they may also simply be the labels affixed to the records by the source federal agency. The individual documents in folders are almost never labelled. In the course of my work, I gathered 30k digital pictures of documents over the course of two months. The acquisition process sounds deceptively easy. Look in the index, find key words and request boxes from the archivist. Then you look through folders to locate individual documents. In point of fact, I probably visually scanned 3M pages to see if they were "interesting" and photo worthy for future research, usually taking only a few seconds per page to make a snap judgement. My decisions on which boxes of documents to request were far more time consuming. What is the right keyword for talking about computers in government in 1970? If you said "information automation" then you would be right. A few presidential (Ford especially) libraries have updated electronic files for indexing which is a huge advantage.

    On my trips to the archives, it was interesting to see both professionals and amateurs using a range of technologies. I saw really old school researchers using 3x5 note cards and taking notes on legal pads. They sometimes supplemented their work by photocopying really important documents at $.75/copy. Some researchers avoided this cost by using flat bed scanners which they carried in with them. Still other researchers brought in high end digital cameras and tripods. I used a digital camera freehanded. All of these people still need to find a way to actually get to physical proximity with the records. Digitalization would open up a new era in research.

    On the metadata issue, most of these records already have copious amounts of metadata recorded in well-established fields that are used by NARA.

    On the OCR issue, some documents have hand-written notes on them which would not be machine readable and sometimes are not human readable. It is likely that the documents will have to be digitally scanned and flagged if handwriting is detected.

    Making these records available to the general public would be a huge advantage to anyone interested in government and US history. Come to think of it, in terms of size and complexity, it would be a worthy challenge for Google. U.S. government documents run back to the founding of the country and the number of documents only increases over time.

    • Remember "the warehouse" from the Indiana Jones movies? NARA is a little like that in terms of size but are better organized.

      Does it play the music when you go in there? That's what really sets the mood, you know.

  • If a commercial vendor can manage to write the required code in the time given, the budget given, and meet all interface demands from the various perspective users. There is still no way that certain TLA (three letter agencies) will let all their documents be indexed. Thus, the project is DOA.
  • IMO
    Objective: Information Determines Social Change and Technology Application.
    Legacy: Technology Determines Social Change and Information Application.

    Yes, a paradigm change. Decision makers (.com/.gov/.mil...) are legacy mind-locked on technology always defining and providing the "Information Technology" (IT) solution.

    Yes, a paradigm change. Decision makers (.com/.gov/.mil...) must go to academia to help define the new "Information Management" (IM) market place. IM must determine the required IT ar

  • So I was at a data.gov meeting in the spring, and got to talking to someone from NARA ... he said their digital archive was um ... I can't remember the exact size, but I want to say it could all fit on a single disk, so given the time, 2TB or less.

    Some of the government agencies have PB of storage already ... we'd love to turn it over to NARA for long term archiving, but there's no procedures in place, and I don't think they currently have the infrastructure or personnel to deal with it.

    (note, I'm taking a

  • We did this at our office some time back. There's more to it than you might think, and I wish we'd done it sooner. First, the cost savings is pretty significant. You've no idea how much paper, files, file cabinets, and sheer storage space for all this paper that's involved until you don't have to use it anymore. Add to that the labor cost of constantly running somewhere to hunt down a paper file, or the labor cost of having someone file away a stack of papers into that paper file. It really is
  • I heard that Laserfiche is a great tool for document management. As it stands they are on the fore front of the anti-piracy movement, and seem to have a stable version to avoid security issues. Maybe this is what they need?

  • by ProppaT (557551) on Wednesday November 30, 2011 @10:39AM (#38213852) Homepage

    If so, I suggest creating your own business and get ready to bid on some work. No one is going to do this in house, they're going to take bids on conversions. I used to work at a company that made quite a bit of money off of paying people, per page, to OCR patents, correct OCR errors, and tag the document in XML. And I can assure you that, because of the way the government works, the majority of the work will go to minority owned small business. The work is easy and you can get college kids to do it for peanuts.

"In matters of principle, stand like a rock; in matters of taste, swim with the current." -- Thomas Jefferson

Working...