Catch up on stories from the past week (and beyond) at the Slashdot story archive

 



Forgot your password?
typodupeerror
Get HideMyAss! VPN, PC Mag's Top 10 VPNs of 2016 for 55% off for a Limited Time ×
Hardware Idle Technology

Plug In an Ethernet Cable, Take Your Datacenter Offline 150

New submitter jddj writes: The Next Web reports on a hilarious design failure built into Cisco's 3650 and 3850 Series switches, which TNW terms "A Network Engineer's Worst Nightmare". By plugging in a hooded Ethernet cable, you...well, you'll just have to see the picture and laugh. They write: "The cables, which are sometimes accidentally used in datacenters, feature a protective boot that sticks out over the top to ensure the release tab isn’t accidentally pressed or broken off, rendering the cable useless. That boot would hit the reset button which happened to be positioned directly above port one of the Cisco switch, which causes the device to quietly reset to factory settings."
This discussion has been archived. No new comments can be posted.

Plug In an Ethernet Cable, Take Your Datacenter Offline

Comments Filter:
  • Easy way.... (Score:5, Insightful)

    by blackfeltfedora ( 2855471 ) on Wednesday September 09, 2015 @10:12PM (#50491823)
    "There’s an easy way to prevent it happening at all, by disabling the button" Another easy way to prevent this from happening would be DON'T BUY THIS SWITCH
    • i can never find a single reason to go back to cisco (from juniper). it's just an expensive, awkward and scary experience.

    • by Anonymous Coward

      This flaw is over 2 years old and was fixed by shifting the button between ports 1 and 2. This news is old and the "Network Engineer's Worst Nightmare" comment is buzzfeed quality click bait rubbish.

      Slashdot needs to go stand in the corner for 10 minutes.

      • I could have fixed it for them without having to change the PCB and front panel: remove the panel button, and make engineers use a PEN to push the switch. You know, like how 99.9% of all other gear is reset to defaults.
  • Bad in any case (Score:5, Insightful)

    by Dan East ( 318230 ) on Wednesday September 09, 2015 @10:12PM (#50491827) Homepage Journal

    Regardless of the design of the connector, having the reset button directly above the port is a bad design. It's simply too easy to hit it with your thumb just plugging in or removing a cable. I suppose holding it down for several seconds resets to factory, which is what happens when using cables with the boot. Still, regardless of that more severe problem, it was a bad design in the first place.

    • Re:Bad in any case (Score:5, Insightful)

      by Drishmung ( 458368 ) on Wednesday September 09, 2015 @10:30PM (#50491897)
      Why didn't they at least recess the switch? You really don't want to accidentally press a reset switch. Poor design.

      Not that Cisco hasn't made faux pas before. The 25xx as I recall had socket for a PCMCIA card, but no slot in the front panel to access it! You had to take the case off to do that.

      • by Anonymous Coward

        Perhaps Cisco believes that the reset switch is used so frequently, they didn't want the network engineers to have to look around for a paperclip to push a recessed switch.

        The real WTF is that there is a "factory reset" button on the thing at all

        • Re: Bad in any case (Score:5, Interesting)

          by xous ( 1009057 ) on Wednesday September 09, 2015 @10:44PM (#50491955) Homepage

          The mode button triggers "express setup" which is basically a lazy way to configure the shit for retard small business/enterprise admins so they don't have to console the device via rs232 to configure it.

          I've had similar issues with older gear not racked properly. The mode button a 3750 (and other models) can still be accidentally depressed in a messy cabinet.

          • Re: Bad in any case (Score:5, Informative)

            by ArmoredDragon ( 3450605 ) on Wednesday September 09, 2015 @11:14PM (#50492053)

            The mode button triggers "express setup" which is basically a lazy way to configure the shit for retard small business/enterprise admins so they don't have to console the device via rs232 to configure it.

            For which model? In every Cisco device I've used (including the C3560 switches I own for CCIE training) the mode button only does anything at all if you have it held down while the switch is powering on. Doing so goes into ROMMON, which allows you to change the configuration register to ignore the startup-config.text file on the flash (the startup-config.text file is what contains all of the password information, so if it doesn't execute, then you effectively have a factory configuration switch, although your configuration files are still present if you need to use them.)

            By the way, you can also modify the configuration register so that if the mode button is held at bootup, then it simply wipes the configuration files entirely, that way you don't have to worry about somebody stealing your configuration data if you have a switch that's in a geographic location that you can't reasonably have physically secured.

            • Re: Bad in any case (Score:5, Informative)

              by TWX ( 665546 ) on Thursday September 10, 2015 @01:54AM (#50492597)
              If I'm remembering correctly...

              If there's a TFTP server properly configured... If there's bootp on the LAN properly configured... If there's a switch configuration saved to that TFTP server and If it's named correctly such that there's a mechanism for associating it with a given request, some Cisco equipment can autoconfigure by pulling the config down off of TFTP without administrator intervention. I've seen some C2960S and C3560G do this; had to clear-out, IOS update, and put config templates on about 160 switches over a few days, watching it complain about not being able to find a TFTP server is just a little burned into my brain.

              No one that I've spoken with has ever used this feature in production, and honestly it would take so much advance-setup to make it work that no boss would choose that path out of laziness instead of getting out a console cable, but technically if the switch were reset with the mode button it might make the attempt.

              Again, if I'm remembering correctly.

              I wish that Cisco would make it harder to press that button. Some older switches were REALLY bad, the button was the whole left end of the panel. If the closet is racked incorrectly the component above or below the switch could press the button and hold it down. I've seen it happen a few times.
              • by Anonymous Coward

                You are right, Cisco has a zero-touch provisioning feature called Smart Install. It is a pain in the ass to configure and it has some limitations like ths switches don't have an RSA key generated by default so you only have telnet access on your first boot so you need to generate byt hand or run a script with a list of hosts using expect afterwards.

                There's a post install script option but it's only available if your director (Switch that manages the init config for the client switches) supports 15.2.(2)E.

                It

            • by xous ( 1009057 )

              Express setup is confined to newer models such as the 3650 and 3750s.

              You are correct that on the 3750 and 3560 that depressing the mode button will only break off and arrive at a rommon> prompt if the mode button is depressed while it's powering it on. I've had a power outage happen when a device was in a messy cabinet and not racked properly... it pressed up against the door and the mode button was held.

              Made quite a bit cleaning up the cabinet and mounting things properly over the weekend.

        • "Factory Reset" has 2 purposes.

          1. When some genius has mis-configured the hell out of it, you use this button to clear everything back to known values.

          2. When the unit is retired from service and placed into service somewhere else, this is a quick way to clear settings that don't apply to the new environment instead of having to walk the entire option list.

          In neither case should it be a simple tap of a button.

    • Well, kind of, but take into account that a switchs front side is almost completly network sockets, so anywhere on the front, it would have been directly above or below a network socket. (You don't want it to be on any other side than the front side for a racked device)

      • What's wrong with the rear, where I presume the DC terminals are located? You should have little access holes in your rack so you can feed cables between racks if necessary. Doesn't take much, an inch of clearance is way more than you need - but it's enough to shove a console cable through if you need to access it from the wrong side,

  • by __roo ( 86767 ) on Wednesday September 09, 2015 @10:15PM (#50491843) Homepage

    Are 'config t' and 'write erase' too difficult to remember? Bothered by all those inconvenient keystrokes? Try the new EasyBoot(TM) from Cisco, the most convenient way to reset your router!

    • by CrankyFool ( 680025 ) on Wednesday September 09, 2015 @10:31PM (#50491903)

      You've got to log in as enabled in order to be able to use 'config' or 'write', which of course means you can't use either to recover from a lost enable password (of course, that's what starting up and interrupting the boot sequence and 0x2102 (which, BTW, I last used about 18 years ago and could still remember -- scary) are for.

  • Wait, what? (Score:5, Funny)

    by Anonymous Coward on Wednesday September 09, 2015 @10:26PM (#50491885)

    From the article:

    The cables, which are sometimes accidentally used in datacenters, feature a protective boot that sticks out over the top to ensure the release

    and then

    Such a situation could cause a problem in any size datacenter, where these switches and cables are commonly used

    So are they commonly used on accident? Accidentally used commonly? I was reading the article to figure out what type of cable was often used, but apparently it's these cables but only by accident all the time.

    • "Sometimes," "commonly" and "accidentally" are not exclusive conditions. They can all be true at once.
      • "Sometimes," "commonly" and "accidentally" are not exclusive conditions. They can all be true at once.

        That's what the AC was asking about, yes. We still haven't heard a clarification.

      • "Sometimes," "commonly" and "accidentally" are not exclusive conditions.

        One of these things is not like the other
        One of these things just doesn't belong!
        Can you tell me which thing is not like the other
        Before I finish this song?

    • by TWX ( 665546 )
      If the alternative is the company buying cables without tab protectors on them, these are actually quite nice cables. We have a ton of 'em (probably quite literally) and so long as hardware designers aren't idiots they work fine. This is an idiot trying to blame the cable when the button is in the wrong spot.
  • by cosm ( 1072588 ) <thecosm3@nOSpAm.gmail.com> on Wednesday September 09, 2015 @10:26PM (#50491889)
    If a single device brings down your entire data center, you've got design problems and your architect should be fired or retrained. These days everything is redundant in triplicate at minimum and new devices spin up automatically based on automatic provisioning and chef/puppet type setups. Even if your core router (why would you have just one!?!?!?!) shits the bed and resets to factory defaults with VLAN 1 and basic STP with no routing interfaces configured, if your NOC folks did a good job, a proper MSTP / VRF / TRILL / SDN ( OpenFlow, etc) / etc like setup should route around that shit and QA will have already tested the "core clos spine device reboots to factory defaults" test case at which point you have just another device for a low paid lackey to swap out based on your network monitor going yellow.

    If you work in a Fortune 500 datacenter and you can't handle this sort of outage, get the fuck out. You're the reason shit's going downhill. Also if a Cisco 3650 or 3850 bring down your datacenter, see previous negative asshole sentiment or get a new job if your manager is responsible for the confines of such a clusterfuck. No participation trophy for such asshattery.
    • by iggymanz ( 596061 ) on Wednesday September 09, 2015 @10:33PM (#50491931)

      blah blah blah

      Reality is single device failures bring down large chunks of the net including valuable peers of your "enterprise datacenter"

      Of course, sometimes identical cisco models used in redundant tuples also cause outages together after upgrade by common bug that didn't show up in test

      so pontificate all you want, you're vulnerable to a lot of bad things

      • Re: (Score:2, Flamebait)

        by cosm ( 1072588 )

        blah blah blah

        Reality is single device failures bring down large chunks of the net including valuable peers of your "enterprise datacenter"

        Of course, sometimes identical cisco models used in redundant tuples also cause outages together after upgrade by common bug that didn't show up in test

        so pontificate all you want, you're vulnerable to a lot of bad things

        (1) I guarantee if you emailed that explanation to a DC manager you'd be shitcanned. I agree that we are all vulnerable to bad things, but avoidance of a single point of failure device in the DC like op highlights is network ops 101 stuff.

        (2) Show me a datacenter that's an all cisco shop. Most are whitebox/greybox now. Welcome to the 21st century. Most "big-data" shops have firmware experts who know their hardware down to the MMU register level and order stuff directly from places like Taiwan with nary a

        • > (2) Show me a datacenter that's an all cisco shop

          I saw two small business datacenters, basically single company server rooms, that had critical core Cisco switches with no redundancy.

          • To be fair, a small company's server room, isn't really a datacenter by most people's use of the word, but it is true that in the vast majority of businesses there's no money for redundant everything. Most small company's are lucky if they can afford two of anything.
        • Your experience must be very limited, the biggest data centers use Cisco.

          Funny thing about blow hards like your hypothetical D.C. manager, they make redundant everything and shoot off their mouths about failure being impossible, and then something like a misconfigured router elsewhere advertising blocks it doesn't own suck up traffic and bring important service access down even if a person has multiple providers. Single point of failure, hundreds of miles away outside their control. Ha! Happened quite a

      • by xous ( 1009057 ) on Wednesday September 09, 2015 @10:48PM (#50491967) Homepage

        The problem is that 3650 and 3850 are not designed for a "Datacenter" deployment.

        They aren't even designed as top of rack switches.Their use case is access or distribution for end-users. They belong in a wiring closet.

        That, of course, doesn't stop morons or small companies deploying them as "Core" routers or switches in their datacenters....

    • by Antique Geekmeister ( 740220 ) on Wednesday September 09, 2015 @11:04PM (#50492019)

      > If a single device brings down your entire data center, you've got design problems and your architect should be fired or retrained.

      Please: if your data center has the time, and skill, and is willing to take the service interruptions to make the whole setup properly immune to single points of failure, that's great. But very, very few live business environments have that kind of resource, time, and willingness to enable critical switches with robust failover.

      • by cosm ( 1072588 )

        > If a single device brings down your entire data center, you've got design problems and your architect should be fired or retrained.

        Please: if your data center has the time, and skill, and is willing to take the service interruptions to make the whole setup properly immune to single points of failure, that's great. But very, very few live business environments have that kind of resource, time, and willingness to enable critical switches with robust failover.

        As other posters have mentioned the level of switches discussed by op are not DC switches. SMB switches, sure, but enterprise datacenter, no.

        • > As other posters have mentioned the level of switches discussed by op are not DC switches. SMB switches, sure, but enterprise datacenter, no.

          I acknowledge you rpoint. From Cisco's specs, they're not aimed at the "enterprise datacenter". With the integrated wireless support, they seem aimed at the corporate datacenter. Frankly, I see a lot more of those these days than of core ISP data centers and switch configurations. But even in an enterprise datacenter, with businesses or individual departments in i

    • by gl4ss ( 559668 )

      yeah I'm sure everyone runs 3x the servers and 3x the switches they need.

      yeah. sure.

      it's not just for fortune 500 datacenters. and plenty of fortune 500 companies have office or whatever serving centralized servers that don't have triple redundancy because it's not really practical.

      plenty of places where that switch could have 10-20 devices behind it that weren't redundant on another switch. in fact if you just stopped to think of how practical world works, it's more than likely.

      (furthermore, "the low paid

      • by cosm ( 1072588 )
        ...small backhaul offices with a couple hundred servers do not a datacenter make. we're arguing semantics at this point but my point still stands for anybody switching petabytes by the hour, i.e. 1k switches at 24-48 10G fiber links switching at 5-10% loading every second.
    • If you work in a Fortune 500 datacenter and you can't handle this sort of outage, get the fuck out. You're the reason shit's going downhill. Also if a Cisco 3650 or 3850 bring down your datacenter, see previous negative asshole sentiment or get a new job if your manager is responsible for the confines of such a clusterfuck. No participation trophy for such asshattery.

      In your Fortune 500 datacenter what happens when a high density edge switch with lots of ports fries? Are all of those systems dead until a monkey pulls all the cables out and replaces the hardware? Do you have redundant connections to every system just to guard against this?

    • If a single device brings down your entire data center, you've got design problems and your architect should be fired or retrained.

      Go fire the architect. http://www.tech-faq.com/how-do... [tech-faq.com]

  • by Anonymous Coward
    Little slow to blog about this then huh ? Bad enough this is considered news, worse still, as usual slashdot bought your cluck-bait.
    • I voted this article down for the use of the term "Hilarious" but it got in anyways.
    • Yep very old news. I laughed when I heard about it nearly 2 years ago.

      Relevant Field Notice from October 2013. http://www.cisco.com/c/en/us/support/docs/field-notices/636/fn63697.html
  • by hilather ( 1079603 ) on Wednesday September 09, 2015 @10:50PM (#50491977)
    You're plugging it in wrong.
  • On our 3850's the button is placed above and in between the Ethernet ports 1 & 3, not directly above Ethernet port 1 as shown in the article.
  • Novel! (Score:5, Interesting)

    by adolf ( 21054 ) <flodadolf@gmail.com> on Wednesday September 09, 2015 @11:22PM (#50492075) Journal

    While I like the auto-LART feature, I wonder what the switch is doing there at all: If the switch is working properly, it doesn't need a reset button.

    If the switch is not working properly, it needs to be burdensome to power-cycle it, to encourage people to complain loudly to the responsible vendor(s) until the product actually works.

    In these modern times, I think an accessible reset switch is like: "Yo dawg, I heard you like to 'fix' things by pushing buttons, so we put buttons on your Enterprise switches so you can reset one-handed while you [...]"

    ObTopic: I once helped take down an enterprise LAN with an Ethernet cable. It was 10-ish years ago, and we just installed a new-fangled VoIP phone system. Each VoIP deskset had a built-in unmanaged 10/100 switch. This was a very handy thing before our modern enlightened structured cabling roll-outs, because it could be trivially daisy-chained with a desktop computer and standardized PoE was not yet a thing.

    Anyhow, we started late on a Wednesday, and finished just before start of business Thursday: Record time for replacing an old Nortel with a few hundred extensions, I tell you. And I went home and died on my couch, having been awake and actually working (prep, etc) for about 40 hours.

    At 7:23AM, my phone rang. It was my manager. Their entire network had crashed, hard. They blamed us. They were livid. I read my manager the NSFW riot act, hung up, and went back to sleep.

    Turns out that after we left, some unknown person had plugged both external switched ports of a deskset into both ports on a wallplate connected to a then high-end HP Procurve switch, which itself connected to a factory and office tower full of other HP Procurve switches carefully set up in a redundant "mesh fabric" mode. This carefully-constructed, redundant network then died in a broadcast packet storm.

    Once they found the error and unplugged that one extraneous heads-will-roll wayward wire, things more-or-less instantly returned to normal.

    (STP would've instantly made this a complete non-issue, but at that time STP and HP's mesh conflicted with eachother and could not cohabitate. I understand that this was subsequently resolved, though I don't deal with HP switches often enough to verify.)

  • On Catalyst 3850s this has been fixed since the release of 3.3.5SE code (release November 2014), so this is old news. Even on older code, the problem can be fixed by using the command "no setup express". I have to say running into this the first time and trying to figure out why the switch had a blank config was a head scratcher...
  • For starters, assuming you fall prey to this, all you lose is the configuration of a single switch. If losing a single fixed configuration 1U switch causes your entire datacenter to go down, your datacenter is badly designed.

    Second, this requires a particular style of booted cable, not just any booted cable. Most datacenters I've worked in don't use booted cables in their switch ports. Their cables are cut to length and crimped by hand. Booted cables can be a bitch to get out of the port, especially on 1U 4

  • by HockeyPuck ( 141947 ) on Thursday September 10, 2015 @01:43AM (#50492565)

    Sure this is funny, but the workaround in TFA is pretty straightforward.

    Disable Express Setup with this command while in config mode:

    3850(config)# no setup express

    Someone explain to me why you'd run Express Setup after deploying this switch?

  • The cables, which are sometimes accidentally used in datacenters

    In my opinion there's not any specific definition on that they shouldn't be used in datacenters - they do have the advantage of protecting the tab on the RJ-45 connector pretty good and would actually be preferred over unprotected connectors.

    Overall the button placement is pretty stupid, and is probably the result of optimizing the size of the unit. So if you run a data center, then you will learn to deal with the button location.

    Realize that this problem is just annoying, there are bigger design flaws in t

  • Ahh, is that the switch and cable combo Ubisoft is using for Uplay? So it's all really Cisco's fault then!

  • There is a reason that vital reset commands need more than one action to complete. It would be a minor inconvenience if the router were to reboot when you press this button (by accident), but to have the complete configuration be wiped by this, and have it situated so that an involuntary application of said button is easy, is just epically stupid.
    But ok. It's Cisco. You'd expect that from them.
  • If this is a problem, you have more serious issues to worry about, such as looking for a mental institution to house you.
  • I once did something similar. I had a screen on a web app which had a form. On the next screen the Delete button was at the same place the submit button on the form.

    The nice lady user had a habit of DOUBLE clicking for some reason. Which means she submitted the form and then deleted the record directly in the next step because the second click went to the delete button.

    Took us a bit to figure out why the docs were deleted.

  • I agree that this is a a crappy design. I was never a fan of the way Cisco designed equipment anyway, but back in the day I cut off boots on any Ethernet cable I used in either the data center or wiring closet simply because SOME equipment had ports slightly recessed and the boots would prevent the cable from locking in reliably. Caused a number of hard to find intermittent problems before we figured out what was going on.
  • The 3650 and 3850 are access layer switches. These are used in closets to connect client devices (desktops, phones, wireless AP's, etc). These are not top-of-rack server switches or core switches for datacenter usage.

  • The cables, which are sometimes accidentally used in datacenters, feature a protective boot that sticks out over the top to ensure the release tab isn’t accidentally pressed or broken off, rendering the cable useless.

    I'm not a network engineer but why are those types of cables not supposed to be used? The article seems to imply that using these hooded cables is wrong. I can see why they wouldn't be cost effective or not necessary but why wrong?

  • I think the first thing we all need to understand is that the button mentioned is NOT a reset button. It's the display button for the lights and is clearly labeled "mode". It cycles between the different information modes such as speed, duplex, stack ID, POE usage, etc. See this article from the Cisco Support forums detailing how to determine which stack ID the different switches are as one example: https://supportforums.cisco.co... [cisco.com]

  • Okay, you gotta admit- that's some funny shit. Poor design allows you to bork your entire network by plugging in a cable. Hilarity ensues.

    And what's this crap: "The cables, which are sometimes accidentally used in datacenters..."

    Cables are "accidentally" used? WTF?

  • ... and this is 'current news' because?

  • just saw of the reset button - leave a ditch. For resetting you can always prick with a pin on that ditch :).

Kiss your keyboard goodbye!

Working...