Follow Slashdot stories on Twitter


Forgot your password?
Privacy Security United States Hardware

How Cisco Is Trying To Prove It Can Keep NSA Spies Out of Its Gear ( 130

itwbennett writes: A now infamous photo [leaked by Edward Snowden] showed NSA employees around a box labeled Cisco during a so-called 'interdiction' operation, one of the spy agency's most productive programs,' writes Jeremy Kirk. 'Once that genie is out of the bottle, it's a hell of job to put it back in,' said Steve Durbin, managing director of the Information Security Forum in London. Yet that's just what Cisco is trying to do, and early next year, the company plans to open a facility in the Research Triangle Park in North Carolina where customers can test and inspect source code in a secure environment. But, considering that a Cisco router might have 30 million lines of code, proving a product hasn't been tampered with by spy agencies is like trying 'to prove the non-existence of god,' says Joe Skorupa, a networking and communications analyst with Gartner.
This discussion has been archived. No new comments can be posted.

How Cisco Is Trying To Prove It Can Keep NSA Spies Out of Its Gear

Comments Filter:
  • by kaka.mala.vachva ( 1164605 ) on Wednesday November 18, 2015 @02:49PM (#50957273)
    That is a lot of code, is that a realistic number for a router? I'm genuinely interested in knowing.
    • by Anonymous Coward

      It is a meaningless number without context (what language? did they count blank lines? etc)

    • by Anonymous Coward

      Considering that making routers is kinda their thing... What do you think? Best guess scenario is that it would be in the general ballpark no matter which way you go.

    • by Lennie ( 16154 ) on Wednesday November 18, 2015 @02:59PM (#50957357)

      If you add enough protocols you'll eventually get there ?

    • Re: (Score:3, Interesting)

      by Anonymous Coward

      BSD base.

      But a team of hundreds of highly talented people who are paid a full time wage to find vulnerabilities (you don't think the NSA has source too?) in everything from the application layer to the bare metal is going to do a better job of finding vulnerabilities than someone sent on a PR junket to "prove" that Cisco routers are secure.

      This is, alas, a technological solution to a social problem, and one with a very finite lifespan.

      In particular, observe that the first domino in the war against end-to-en

      • BSD? I thought that Cisco used Linux, whenever it didn't use QNX. Juniper are the guys who use BSD
        • by haruchai ( 17472 )

          I'm pretty sure the Nexus switches run Linux on the bare metal but the AsyncOS that powers the Ironport Web & Email appliances is supposedly running on top of FreeBSD.
          But in neither case does the customer have access to underlying OS - as far as I'm aware.

    • I'd say it's realistic. Depends on the router.

      A modern high-end router is really more of an IDS/IPS/firewall than just a router. There is a lot of stuff going on. And if you include all the code for the interface (both a console and a web-based interface), then it REALLY gets nutty.

      • by ArmoredDragon ( 3450605 ) on Wednesday November 18, 2015 @03:52PM (#50957763)

        Not only realistic, but I myself would be concerned with what is going on inside of the asic, and finding out would be very non trivial, even if they revealed the schematics.

        Also of concern is, how do we know they haven't received an NSL telling them to maintain two sets of code, with one of them being compromised and can't be shown to somebody without government clearance?

        • by AmiMoJo ( 196126 )

          To be fair, I think a backdoor in an ASIC is unlikely. It would be hard to hide from all the people working on the product, and would make it easier for other people to hack Cisco gear. The NSA doesn't want to open the door for everyone.

          That's why they were intercepting hardware being shipped to customers and planting bugs in it. Targeted, easy to update the bugs, easy to hide from Cisco engineers.

          • by Anonymous Coward

            To be fair, I think a backdoor in an ASIC is unlikely. It would be hard to hide from all the people working on the product, and would make it easier for other people to hack Cisco gear.

            I suspect such a backdoor would take to form of an extra processor that you can load code into if you know what you are doing. Sometimes there is a lightweight 8-bit controller that is only used to load the program into the main memory for the real processor to take over or a dedicated controller to deal with some I/O-stuff.
            It might be hard to realize that the code for a processor you didn't know existed is missing.

          • A backdoor might be hard to hide, but a backdoor enabling flaw might not be. Just as with any problem, you don't always have to solve it in one go, you take "bite sized" pieces and solve them.

            So you don't enable a backdoor, you just introduce a flaw which makes it easier to exploit another flaw downstream.

    • by Anonymous Coward

      It's actually around 185 million lines of code across 13 current release trains.

      So I hear.

    • by Tablizer ( 95088 )

      That is a lot of code, is that a realistic number for a router?

      It's divided into:

      A. 1 million lines that do real work.
      B. 10 million lines to verify nobody tampered with "A".
      C. 18 million lines to verify nobody tampered with "B".
      D. 1 million lines to display a disclaimer that says if somebody tampers with "C", you are S.O.L.

    • I don't know what those particular routers are running. Here is just me listing a few packages off the top of my head that could be in there:

      There are 12 million LOC in the kernel alone (linux?)
      Another million for libc
      2 millions for web server
      2 millions for php or whatever they use.
      6 million for java.

      I have not even included anything cisco might write themselves.
      As you can see, it would not be too hard to get to the 30 million LOC mark. The backdoors can be installed in any of these packages not only i

    • by shugah ( 881805 )
      It's certainly not the UI.
    • There are 15 million lines of code in the Linux kernel so this doesn't seem surprising at all. They probably have a smaller kernel and less userland but we're still within this order of magnitude.
    • I'm pretty sure it runs a version of the Linux kernel, so yes, that is realistic.
  • by ItsJustAPseudonym ( 1259172 ) on Wednesday November 18, 2015 @02:58PM (#50957349)
    More like "the devil", in this case.
  • It's the Law (Score:5, Interesting)

    by Anonymous Coward on Wednesday November 18, 2015 @02:59PM (#50957353)

    How can they convince anyone that they can keep the NSA out when the Law says they have to let the NSA in?

    • How can they convince anyone that they can keep the NSA out when the Law says they have to let the NSA in?

      Well, assuming your premise the only thing that can be done is to show everybody the code and let somebody not under NSL seal disclose it.

      Cisco's actions aren't inconsistent with that approach. The speculation is hardly proven, though.

    • How can they convince anyone that they can keep the NSA out when the Law says they have to let the NSA in?

      Which law is that, exactly?

      People on /. (and elsewhere) make a lot of invalid assumptions about what the law allows the government to do. National Security Letters, for example, are assumed to be able to compel anyone to do anything and keep their mouths shut about it. In fact, the law says that NSLs can only require the recipient to provide data already in possession (not set up long-lived back doors) and further can only demand metadata, not content. NSLs are only one legal vehicle for requests, but as

    • They ship signed binaries and post the hashes. That's not a perfect solution since somebody could load a different binary with a rootkit that makes it look like the real one. But if you download the firmware and verify the signature, that's a pretty good start. We all hated TPM when MSFT tried to introduce it to kill Linux but now it's starting to make sense.
  • by Lennie ( 16154 ) on Wednesday November 18, 2015 @03:01PM (#50957367)

    This might be useful only if I could bring my own compiler and could keep the resulting binary and I could install that myself on the hardware (never going to happen).

    Even than, the Cisco products includes hardware with sophisticated packet processing capabilities they could just built it into that.

    Maybe they should first find a way to ship the product in such a way that it can't be tampered with.

    • by cfalcon ( 779563 )

      > Maybe they should first find a way to ship the product in such a way that it can't be tampered with.

      I really and truly don't believe that is possible.

      In fact, the whole thing seems unlikely to be taken seriously.

      You need to be able to (at your site)- ensure the integrity of circuitry, ensure the integrity of code.

      I mean, holy crap.

      But when it comes to some random packing technique? No way.

    • This might be useful only if I could bring my own compiler

      You can (per the FAQ).

      and could keep the resulting binary and I could install that myself on the hardware (never going to happen).

      If Cisco defines the hash of the build binary as their IP, then the whole thing is doomed. If you can reproduce their build, a hash collision isn't going to be an actual risk.


      Q: What technologies or products can be reviewed?
      TVS includes all Cisco technologies, within the bounds of applicable Export Control Laws. W

  • by sasparillascott ( 1267058 ) on Wednesday November 18, 2015 @03:07PM (#50957407)
    Just like the documents showing Microsoft handing over their customers communication data to the NSA...once you've been fingered as a good "partner" with the U.S. intelligence apparatus your shelf life as a company has been time bombed...ignition is just waiting on an alternative supplier that can be reasonably trusted (IMHO this could take some years, but its coming...the market is too big and valuable...if given a true choice nobody wants to buy gear from companies that were shown to be stooges for government snooping).
  • by Anonymous Coward

    It's guaranteed that cisco is compromised by NSLs. Until this law is fixed, no big vendor can be trusted.

    • Is this less of a Cisco/Juniper problem and more of a FedEX/UPS/DHL problem?

      When I ship a package via FedEx et al, I don't expect it to be detoured thru the local NSA office to be 'enhanced'.

      I expect it to be delivered intact and not adulterated. Come on FedEx et al, do your job!

      • by swalve ( 1980968 )
        Exactly. If they have back doors, why would they bother with the mess of interdicting a shipment? Assuming the Snowden info is actually real, of course.
  • by silas_moeckel ( 234313 ) <silas.dsminc-corp@com> on Wednesday November 18, 2015 @03:22PM (#50957515) Homepage

    The NSA was supposedly loading code onto hardware. Cisco is a pretty closed environment if they pown the bootloader just exactly how are you going to detect this? You can review all the code you want if your can not trust the hardware it does you no good.

    • Answer is easy. Cisco routers ship naked and Cisco images each one on site personally for an extra fee

      • If you intercept it in shipping and replace hardware or install a rooted rommon you can install all the trusted images you like. Ultimately you have to trust rommon that it's updating itself or that IOS is actually writing out a new copy.

        • No the Cisco guy wipes out when he images. Unless you think NSA will put an ROM or eprom in it?

          • I rather doubt the Cisco guy is plugging in a jtag programmer (or similar), even if they did that's a fairly high level interface and could be hacked to deliver whatever responses are expected (probably requiring hardware to do so). It's not like any recent cisco box has a removable rom anymore (few PC's for that matter).

  • How about stop making and delivering interdicted custom gear for the NSA/CIA?

    I know, I have seen the equipment hooked into AT&Ts network.

    It isn't a joke what is happening. In the end we all know why this spying is happening and it is not to make you safer.

    It really is all about industrial espionage and taxes, all in pursuit of western bogeymen, they create.

    As long as they keep the bogeymen well funded expect more countries shredding freedom and liberty, and all of those that died before us to have give

    • by bluelip ( 123578 )

      You haven't seen anything.

      Read and _UNDERSTAND_ the article.

      • by hackus ( 159037 )

        I understand alright.

        Look at the original ARPA documents calling for a distributed non centralized architecture for the command and control of communications.

        I also understand that CISCOs products are perfectly designed to be easily compromised on a small scale with gigantic affects.

        The worse being the UCS manager. Who in there right mind would build a network where all you need is access to one box, and you can trash whole infrastructure.

        These products coming out of companies, not just CISCO, are the darli

        • by bluelip ( 123578 )

          You still don't understand anything. Good effort though.

          Cisco isn't "shipping and delivering interdicted" custom gear.

          Try again. Maybe you'll get a bronze star this time. For effort.

    • Re:CISCO (Score:5, Interesting)

      by AK Marc ( 707885 ) on Wednesday November 18, 2015 @03:35PM (#50957617)
      Use only Huawei in the core and Cisco on the edge, with a firewall rule to block traffic to/from China to block the Huawei back doors. Or vice versa. You can't trust either, but hopefully both aren't compromised by the same group.
      • by cfalcon ( 779563 )

        Mod parent up pls. This is a solid workaround based on realpolitik.

      • by wulfhere ( 94308 )

        You DO realize that 'China' could have servers sitting somewhere connected via 'non-China' IPs, right?

        • by AK Marc ( 707885 )
          You are thinking too literally. "block traffic to/from China" I don't care whether the server is in China, but whether the control of the networking gear is not from my management systems. Typical Slashdot style, someone implies that the correct solution is wrong because the correct solution wasn't specified to an irrelevant level of detail, and the idiot Slashdotter assumes incompetent implementation.

          Not everyone is as dumb as you. You don't blacklist your management system. You white list it. When y
      • by swalve ( 1980968 )
        This seems pretty simple. Can't you just make a box that looks at the data coming into a network and drops anything that is unexpected? Some kind of "firewall" between the trusted network and the untrusted?
        • by AK Marc ( 707885 )
          Yeah, but when the firewall is made by Cisco, how do you trust the firewall if you don't trust Cisco?
          • by vux984 ( 928602 )

            The solution was covered.

            2 firewalls in sequence.

            Cisco + Huewei

            Even if you trust neither to prevent the respective vendors government out, you can reasonably trust the cisco not to be in bed with the chinese, and the hauwei not to be in bed with the americans.

            So either state actor is blocked. If the chinese and americans are working *together* to break into your network... you've probably got a situation where your network shouldn't be connected to the internet period... transferring your data via u

  • by HighOrbit ( 631451 ) on Wednesday November 18, 2015 @03:26PM (#50957551)
    Back in March , in a related story, one of Cisco's VPs for security, John Stewart, was quoted in the press as saying that Cisco would ship to decoy addresses to circumvent interception by the Government. Supposedly, this was at a roundtable discussion during the Cisco-Live conference in Melbourne, but there is no video of the discussion on the Cisco-live website.

    I've heard he was misquoted and they don't actually do it. Does anybody have link to actual video of this discussion? Are they still doing this? Has anybody used that service?

    The original slashdot article is []
    • by swalve ( 1980968 )
      I mean, it only makes sense. You place an order and have it shipped to your buddy who works at KMart. Cisco doesn't even have to know!
  • by jtara ( 133429 ) on Wednesday November 18, 2015 @03:27PM (#50957555)

    Seen enough YouTube videos from cameras packed in shipments for the obvious answer...

    These boxes are costly enough to justify packaging it with some device that will record GPS, video, and sound. Make sure there is some good cryptographic signature on the device. Attach it to the router, and put a nasty anti-tamper dye spray to boot. (Although might have some regulatory issues with the explosive device for that, hmmm...).

    Give the customer a rebate for returning the tracking device. (After unlocking, of course.)

    Of course, the tracking device will need solid cryptographic signature/protection, but would have a lot fewer millions of lines of code than the router!

    Then the guy you see stumbling out of the FedEx office covered in dye... he's not with FedEx.

    The best the spys can do, then, is to "lose" the device in shipment, pay off the carrier's insurance company (otherwise, insurance rates will go sky-high), and then try to sell the router in the black market to spy on somebody other than the original target.

  • Step 1) Hacker (or rather "cracker") takes them on the offer to test their equipment in a secure environment.

    Step 2) Hacker understands how it works and notices a security issue, but does not reveal it.

    Step 3) Return to private home where they design an exploit of that issue.

    Frankly, their attempts to keep their security secret just make it harder for the white hats to detect the issues, without significantly affecting the black hats.

    • by AHuxley ( 892839 )
      Thats the problem with trap doors and back doors mandated by 5 eye governments. Sooner or later ex staff, former staff, smart people, private sector security experts find the extra code and let the world know.
      The UK has a plan to ban science that will try and extend the useful life of UK gov mandated trap doors and back doors.
      The US gov is trying to make people feel better about the US private sector again while private sector help for collect it all is the only tool the gov has.
      What can the public and
  • by CanadianMacFan ( 1900244 ) on Wednesday November 18, 2015 @03:56PM (#50957783)

    What good is checking the source code when the NSA is shown to be modifying the gear after it leaves Cisco? You're checking the code that ships from Cisco before the NSA gets it, not what you receive. And what if the NSA isn't touching something in the code but putting in a piece of their own hardware?

    • You're checking the code that ships from Cisco before the NSA gets it, not what you receive.

      Cisco could provide their customers with SHA-Checksums of the binaries, so they can be verified upon arrival.

  • Which means that they will be subject to all sorts of pressures to be 'helpful' about it. Let's be clear ladies and gentlemen, boys and girls, trusting any US produced hardware or software is a mistake if you want to be SECURE. That the tech firms haven't used this as excuse to move their domicile to somewhere with lower taxes as the real excuse for moving remains a surprise...
    • by cfalcon ( 779563 )

      You can trust free or open source software produced anywhere, because they give you the code.

      Proprietary code and almost any hardware... eh....

  • by EmperorOfCanada ( 1332175 ) on Wednesday November 18, 2015 @04:04PM (#50957831)
    Did they move their operations from the US and fire all their US developers and only hire ones from countries with the strongest data protection laws and the weakest spy agencies?

    No? Then they are NSA compromised. Here is a letter from the DOJ ordering you to cooperate with the NSA or go to jail. You can't show the letter to anyone or you go to jail. If you want to contest it you will first go to jail and then you will have to contest it in a special court where you can't get any evidence that is in your favour. So you stay in jail.

    If companies like Siemens are using Cisco equipment then they are fools.
  • by NotDrWho ( 3543773 ) on Wednesday November 18, 2015 @04:06PM (#50957839)

    Thank your government for the fact that no one in their right mind is ever going to trust any hardware coming out of the U.S. ever again. Ain't no putting that genie back in the bottle.

    • There are two versions of Cisco's 5500 LAN controller software: 1) the normal version 2) the special version which is only recommended for Russia where "Data DTLS Payload encryption" is regulated by the Government. Ironic that Snowden fled the US and went to Russia. Which version of software do you want on the wireless infrastructure you use?
  • ...Interdiction is where it's at: []

    Or maybe use IPSec / SSH with DH Group 19 - that's not looking too clever either: []

    All in all, if your threat model includes the NSA then reviewing 30m LOC may seem like a good place to start but in practice.....

  • by Billly Gates ( 198444 ) on Wednesday November 18, 2015 @05:34PM (#50958517) Journal

    And I wonder if the NSA root kit will wipe out the Chinese one?

  • No.

    Look, an out of control surveillance regime which can't even stop terrorists from getting 1000 weapons in the US will spy because they can, no matter what they say.

    There's your budget deficit.

  • The CIA and NSA specialize in intercepting items in transit, modifying them, carefully repacking them to hide any sign of tampering and sending them on to the end recipient.

    None of that is impacted in the slightest bit if customers are coming to a warehouse in NC to test it. So it tests clean and they sign off on it. And what happens next? It gets shipped. And if they want to intercept it, they will. And what has been accomplished? Nothing.

    And of course this is separate from the OTHER big Cisco issu

  • Better idea (Score:4, Interesting)

    by TheCarp ( 96830 ) <sjc&carpanet,net> on Wednesday November 18, 2015 @08:29PM (#50959673) Homepage

    We already have "did this package get dropped" sensors. So take that to the next level.

    Vacume seal an interior bag. Place a module inside the bag with:
    1. Internal Battery
    2. Sensor package including light and air pressure/composition sensors
    3. A small amount of memory
    4. A running program which will erase the memory if any of the sensors detect a change
    5. a small transmitter, capable of answering a challenge.

    Customer/Cisco generate a key using a key exchange protocol, key is loaded into box gaurdian module. Box is shipped. Customer uses an RF device to query the package to see if it has been tampered with, customer informs cisco for an immediate RMA, but accepts delivery, so as to be sure the box can be returned in tact for analysis.

Not only is UNIX dead, it's starting to smell really bad. -- Rob Pike