Plug In an Ethernet Cable, Take Your Datacenter Offline 150
New submitter jddj writes: The Next Web reports on a hilarious design failure built into Cisco's 3650 and 3850 Series switches, which TNW terms "A Network Engineer's Worst Nightmare". By plugging in a hooded Ethernet cable, you...well, you'll just have to see the picture and laugh. They write: "The cables, which are sometimes accidentally used in datacenters, feature a protective boot that sticks out over the top to ensure the release tab isn’t accidentally pressed or broken off, rendering the cable useless. That boot would hit the reset button which happened to be positioned directly above port one of the Cisco switch, which causes the device to quietly reset to factory settings."
Easy way.... (Score:5, Insightful)
Re: (Score:2)
i can never find a single reason to go back to cisco (from juniper). it's just an expensive, awkward and scary experience.
Re: (Score:1)
This flaw is over 2 years old and was fixed by shifting the button between ports 1 and 2. This news is old and the "Network Engineer's Worst Nightmare" comment is buzzfeed quality click bait rubbish.
Slashdot needs to go stand in the corner for 10 minutes.
Re: (Score:1)
Comment removed (Score:5, Insightful)
Re: (Score:2)
I largely agree however why add *another* source of failure?
Re: (Score:1)
Yeah but that would require 1) planning ahead, and 2) knowing what you're buying (see 1). This is a non-starter among average Americans.
Because average citizens of any nation are commonly found purchasing 60-port ethernet switches? You accuse the "average American" of not thinking about their purchase, yet you have failed to think about how your statement doesn't really apply to the subject at hand.
Re:Easy way.... (Score:5, Insightful)
That's even worse, only qualified IT departments would be buying these switches so you have every reason to expect that they *should* research their purchase before buying.
Normally a reset button needs to be pressed with a pin to prevent accidental pressing...
Probably designed by a millenial (Score:3)
This. I've never seen anything where it wasn't recessed like that.
Re: (Score:2)
That. So there.
I have an old DFI server. Reset button sticks right out there. I bumped it more than once, sometimes just because my knee hit it and it was right next to my chair.
Re: (Score:2)
Now you have and Cisco invented it first! I look forward to the patent so Cisco can prevent other network equipment manufacturers from implementing this vital feature. It needs to be exclusive to Cisco.
Re: (Score:2)
Pin? For real?
You wouldn't be bringing small slivers of conducting metal loose into any data centre I ran, I wouldn't be buying something requiring a pin either.
Re: (Score:3)
So you've never bought a current generation phone, tablet, PC, or a linksys router? Each of those require a pin for different things. Phones and tablets to remove their SIM chips, PCs and linksys routers to reset their CMOS settings or any device with a CD/DVD/Bluray drive to open it in case it gets "stuck".
Re: (Score:2)
Sometimes those pins come with non conducting handles, these pins are usually called push pins. (thumb tacks)
Re: (Score:3)
"Average" anyones don't buy datacenter equipment.
Re: (Score:1)
Average datacenter equipment buyers do?
Re: (Score:3)
The average American has 0.98 Testicles.
Re: (Score:2)
The average American has 0.98 Testicles.
I surely hope the average American has at least 1.0195 testicle more than you say. 1.9995 testicle seems a good number even though a lady friend of mine says the real value is 1.982142 based upon a spot check.
Re: (Score:2)
Did you remember to factor in all the ones and twos, plastic balls don't count.
Re: (Score:2)
Then who does? Datacenter jobs are just jobs. Average people work them, just like they do every job. And they do so under a corporate structure which excels at encouraging indifference and ass-covering at best and actively sabotaging each other at worst, just like any other dictatorship.
Re: (Score:2)
You know, I was driving in traffic up to a big clover leaf with a short merge area the other day and thinking, you know....theres probably enough people out there that there is almost always someone trying to do this for the first time right here.
Everyone who does something does it while inexperienced first. Every day old people die and new are born, and new people are being inexperienced and making errors....and..... even experienced people miss things.
Presumably "experienced people miss things" is how thi
Re: (Score:2)
Well, I'd hope that the Cisco lab people tested this thing before they released it to mass-production. You'd think that someone would have noticed.
Re: (Score:2)
Yes but testing can only catch so much. Do you really think anyone ever considered "Hey we need to stock the lab with booted cables too, because it might make a difference someday"? Maybe now that will be standard procedure but....I wouldn't have expected anyone to have expected such a test to matter, or even think to try it.
Re: (Score:2)
I use booted cables extensively, since plugging and unplugging frequently is a good way to snap the retainer clip. Not that people would be doing that in a test lab...
There's also the suggestion that if you have fat enough fingers even plugging in naked connectors would be enough to trip the button.
Re: (Score:3)
Also, it would require that the person most qualified to make such decissions is also the person actually making those decissions.
How many of us had to suffer the fate of a "golfclub"; where the boss decides to force a certain product upon his employees because his buddy from the golfclub sells it.
Re: Easy way.... (Score:1)
Wow, America- AND fat-shaming in one go. Surely we should all follow your enlightened example.
Fucktard.
Re: (Score:2, Funny)
joke is on you because WE WILL NOT BE SHAMED! #americanpride #fatpride
Re: Easy way.... (Score:5, Funny)
Re: (Score:2, Funny)
Bad in any case (Score:5, Insightful)
Regardless of the design of the connector, having the reset button directly above the port is a bad design. It's simply too easy to hit it with your thumb just plugging in or removing a cable. I suppose holding it down for several seconds resets to factory, which is what happens when using cables with the boot. Still, regardless of that more severe problem, it was a bad design in the first place.
Re:Bad in any case (Score:5, Insightful)
Not that Cisco hasn't made faux pas before. The 25xx as I recall had socket for a PCMCIA card, but no slot in the front panel to access it! You had to take the case off to do that.
Re: Bad in any case (Score:2)
Perhaps Cisco believes that the reset switch is used so frequently, they didn't want the network engineers to have to look around for a paperclip to push a recessed switch.
The real WTF is that there is a "factory reset" button on the thing at all
Re: Bad in any case (Score:5, Interesting)
The mode button triggers "express setup" which is basically a lazy way to configure the shit for retard small business/enterprise admins so they don't have to console the device via rs232 to configure it.
I've had similar issues with older gear not racked properly. The mode button a 3750 (and other models) can still be accidentally depressed in a messy cabinet.
Re: Bad in any case (Score:5, Informative)
The mode button triggers "express setup" which is basically a lazy way to configure the shit for retard small business/enterprise admins so they don't have to console the device via rs232 to configure it.
For which model? In every Cisco device I've used (including the C3560 switches I own for CCIE training) the mode button only does anything at all if you have it held down while the switch is powering on. Doing so goes into ROMMON, which allows you to change the configuration register to ignore the startup-config.text file on the flash (the startup-config.text file is what contains all of the password information, so if it doesn't execute, then you effectively have a factory configuration switch, although your configuration files are still present if you need to use them.)
By the way, you can also modify the configuration register so that if the mode button is held at bootup, then it simply wipes the configuration files entirely, that way you don't have to worry about somebody stealing your configuration data if you have a switch that's in a geographic location that you can't reasonably have physically secured.
Re: Bad in any case (Score:5, Informative)
If there's a TFTP server properly configured... If there's bootp on the LAN properly configured... If there's a switch configuration saved to that TFTP server and If it's named correctly such that there's a mechanism for associating it with a given request, some Cisco equipment can autoconfigure by pulling the config down off of TFTP without administrator intervention. I've seen some C2960S and C3560G do this; had to clear-out, IOS update, and put config templates on about 160 switches over a few days, watching it complain about not being able to find a TFTP server is just a little burned into my brain.
No one that I've spoken with has ever used this feature in production, and honestly it would take so much advance-setup to make it work that no boss would choose that path out of laziness instead of getting out a console cable, but technically if the switch were reset with the mode button it might make the attempt.
Again, if I'm remembering correctly.
I wish that Cisco would make it harder to press that button. Some older switches were REALLY bad, the button was the whole left end of the panel. If the closet is racked incorrectly the component above or below the switch could press the button and hold it down. I've seen it happen a few times.
Re: (Score:1)
You are right, Cisco has a zero-touch provisioning feature called Smart Install. It is a pain in the ass to configure and it has some limitations like ths switches don't have an RSA key generated by default so you only have telnet access on your first boot so you need to generate byt hand or run a script with a list of hosts using expect afterwards.
There's a post install script option but it's only available if your director (Switch that manages the init config for the client switches) supports 15.2.(2)E.
It
Re: (Score:2)
Express setup is confined to newer models such as the 3650 and 3750s.
You are correct that on the 3750 and 3560 that depressing the mode button will only break off and arrive at a rommon> prompt if the mode button is depressed while it's powering it on. I've had a power outage happen when a device was in a messy cabinet and not racked properly... it pressed up against the door and the mode button was held.
Made quite a bit cleaning up the cabinet and mounting things properly over the weekend.
Re: (Score:2)
"Factory Reset" has 2 purposes.
1. When some genius has mis-configured the hell out of it, you use this button to clear everything back to known values.
2. When the unit is retired from service and placed into service somewhere else, this is a quick way to clear settings that don't apply to the new environment instead of having to walk the entire option list.
In neither case should it be a simple tap of a button.
Re: (Score:2)
Well, kind of, but take into account that a switchs front side is almost completly network sockets, so anywhere on the front, it would have been directly above or below a network socket. (You don't want it to be on any other side than the front side for a racked device)
Re: (Score:2)
What's wrong with the rear, where I presume the DC terminals are located? You should have little access holes in your rack so you can feed cables between racks if necessary. Doesn't take much, an inch of clearance is way more than you need - but it's enough to shove a console cable through if you need to access it from the wrong side,
It's not a bug, it's a feature (Score:5, Funny)
Are 'config t' and 'write erase' too difficult to remember? Bothered by all those inconvenient keystrokes? Try the new EasyBoot(TM) from Cisco, the most convenient way to reset your router!
Re:It's not a bug, it's a feature (Score:4, Interesting)
You've got to log in as enabled in order to be able to use 'config' or 'write', which of course means you can't use either to recover from a lost enable password (of course, that's what starting up and interrupting the boot sequence and 0x2102 (which, BTW, I last used about 18 years ago and could still remember -- scary) are for.
Wait, what? (Score:5, Funny)
From the article:
The cables, which are sometimes accidentally used in datacenters, feature a protective boot that sticks out over the top to ensure the release
and then
Such a situation could cause a problem in any size datacenter, where these switches and cables are commonly used
So are they commonly used on accident? Accidentally used commonly? I was reading the article to figure out what type of cable was often used, but apparently it's these cables but only by accident all the time.
Re: (Score:1)
Re: (Score:2)
"Sometimes," "commonly" and "accidentally" are not exclusive conditions. They can all be true at once.
That's what the AC was asking about, yes. We still haven't heard a clarification.
Re: (Score:3)
"Sometimes," "commonly" and "accidentally" are not exclusive conditions.
One of these things is not like the other
One of these things just doesn't belong!
Can you tell me which thing is not like the other
Before I finish this song?
Re: (Score:2)
i work in enterprise datacenter (Score:4, Interesting)
If you work in a Fortune 500 datacenter and you can't handle this sort of outage, get the fuck out. You're the reason shit's going downhill. Also if a Cisco 3650 or 3850 bring down your datacenter, see previous negative asshole sentiment or get a new job if your manager is responsible for the confines of such a clusterfuck. No participation trophy for such asshattery.
Re:i work in enterprise datacenter (Score:5, Insightful)
blah blah blah
Reality is single device failures bring down large chunks of the net including valuable peers of your "enterprise datacenter"
Of course, sometimes identical cisco models used in redundant tuples also cause outages together after upgrade by common bug that didn't show up in test
so pontificate all you want, you're vulnerable to a lot of bad things
Re: (Score:2, Flamebait)
blah blah blah
Reality is single device failures bring down large chunks of the net including valuable peers of your "enterprise datacenter"
Of course, sometimes identical cisco models used in redundant tuples also cause outages together after upgrade by common bug that didn't show up in test
so pontificate all you want, you're vulnerable to a lot of bad things
(1) I guarantee if you emailed that explanation to a DC manager you'd be shitcanned. I agree that we are all vulnerable to bad things, but avoidance of a single point of failure device in the DC like op highlights is network ops 101 stuff.
(2) Show me a datacenter that's an all cisco shop. Most are whitebox/greybox now. Welcome to the 21st century. Most "big-data" shops have firmware experts who know their hardware down to the MMU register level and order stuff directly from places like Taiwan with nary a
Re: (Score:2)
> (2) Show me a datacenter that's an all cisco shop
I saw two small business datacenters, basically single company server rooms, that had critical core Cisco switches with no redundancy.
Re: (Score:2)
Re: (Score:2)
Your experience must be very limited, the biggest data centers use Cisco.
Funny thing about blow hards like your hypothetical D.C. manager, they make redundant everything and shoot off their mouths about failure being impossible, and then something like a misconfigured router elsewhere advertising blocks it doesn't own suck up traffic and bring important service access down even if a person has multiple providers. Single point of failure, hundreds of miles away outside their control. Ha! Happened quite a
Re:i work in enterprise datacenter (Score:5, Interesting)
The problem is that 3650 and 3850 are not designed for a "Datacenter" deployment.
They aren't even designed as top of rack switches.Their use case is access or distribution for end-users. They belong in a wiring closet.
That, of course, doesn't stop morons or small companies deploying them as "Core" routers or switches in their datacenters....
Re: i work in enterprise datacenter (Score:2)
If used in such a case it would not result in such a sensational "complete data center outage" unless the network was designed by a retard.
Yes, it is a terrible design problem but anyone using the hardware in such a way that it takes out an entire data center deserves what they get.
Re:i work in enterprise datacenter (Score:5, Insightful)
> If a single device brings down your entire data center, you've got design problems and your architect should be fired or retrained.
Please: if your data center has the time, and skill, and is willing to take the service interruptions to make the whole setup properly immune to single points of failure, that's great. But very, very few live business environments have that kind of resource, time, and willingness to enable critical switches with robust failover.
Re: (Score:2)
> If a single device brings down your entire data center, you've got design problems and your architect should be fired or retrained.
Please: if your data center has the time, and skill, and is willing to take the service interruptions to make the whole setup properly immune to single points of failure, that's great. But very, very few live business environments have that kind of resource, time, and willingness to enable critical switches with robust failover.
As other posters have mentioned the level of switches discussed by op are not DC switches. SMB switches, sure, but enterprise datacenter, no.
Re: (Score:2)
> As other posters have mentioned the level of switches discussed by op are not DC switches. SMB switches, sure, but enterprise datacenter, no.
I acknowledge you rpoint. From Cisco's specs, they're not aimed at the "enterprise datacenter". With the integrated wireless support, they seem aimed at the corporate datacenter. Frankly, I see a lot more of those these days than of core ISP data centers and switch configurations. But even in an enterprise datacenter, with businesses or individual departments in i
Re: (Score:3)
yeah I'm sure everyone runs 3x the servers and 3x the switches they need.
yeah. sure.
it's not just for fortune 500 datacenters. and plenty of fortune 500 companies have office or whatever serving centralized servers that don't have triple redundancy because it's not really practical.
plenty of places where that switch could have 10-20 devices behind it that weren't redundant on another switch. in fact if you just stopped to think of how practical world works, it's more than likely.
(furthermore, "the low paid
Re: (Score:2)
Re: (Score:2)
If you work in a Fortune 500 datacenter and you can't handle this sort of outage, get the fuck out. You're the reason shit's going downhill. Also if a Cisco 3650 or 3850 bring down your datacenter, see previous negative asshole sentiment or get a new job if your manager is responsible for the confines of such a clusterfuck. No participation trophy for such asshattery.
In your Fortune 500 datacenter what happens when a high density edge switch with lots of ports fries? Are all of those systems dead until a monkey pulls all the cables out and replaces the hardware? Do you have redundant connections to every system just to guard against this?
Re: (Score:2)
Go fire the architect. http://www.tech-faq.com/how-do... [tech-faq.com]
Re: (Score:2)
No, the Fortune 500 company replacing 3 competent network engineers with an H-1B who works 12x6 for $65K and can't handle the outage, is the reason shit's going downhill. Don't hate the player, hate the game.
How many games would exist or continue with no players?
Re: (Score:3)
Re: (Score:2)
What crimper? Just use a connector with no cable connected to it.
Re: (Score:2)
The transparent plastic type I buy won't fit - the contacts stand up until you crimp them. It's the pressing down that pushes the spikes into the wires.
Re: (Score:2)
Ahh. That is when you snip the ends off of old and questionable cables, perhaps with enough of a tail to attach a label that says "dead port" or "beware reset switch".
Re: (Score:2)
For $9,000 I'll sell you 40 bottle caps and two - not just one - rolls of duct tape.
In 2013, Cisco issued a ‘field notice’ (Score:1)
Re: (Score:1)
Re: (Score:1)
Relevant Field Notice from October 2013. http://www.cisco.com/c/en/us/support/docs/field-notices/636/fn63697.html
Cisco's official response.. (Score:5, Funny)
Re:Cisco's official response.. (Score:5, Funny)
You're plugging it in wrong.
To be fair, it is running IOS.
Not on ours... (Score:1)
Novel! (Score:5, Interesting)
While I like the auto-LART feature, I wonder what the switch is doing there at all: If the switch is working properly, it doesn't need a reset button.
If the switch is not working properly, it needs to be burdensome to power-cycle it, to encourage people to complain loudly to the responsible vendor(s) until the product actually works.
In these modern times, I think an accessible reset switch is like: "Yo dawg, I heard you like to 'fix' things by pushing buttons, so we put buttons on your Enterprise switches so you can reset one-handed while you [...]"
ObTopic: I once helped take down an enterprise LAN with an Ethernet cable. It was 10-ish years ago, and we just installed a new-fangled VoIP phone system. Each VoIP deskset had a built-in unmanaged 10/100 switch. This was a very handy thing before our modern enlightened structured cabling roll-outs, because it could be trivially daisy-chained with a desktop computer and standardized PoE was not yet a thing.
Anyhow, we started late on a Wednesday, and finished just before start of business Thursday: Record time for replacing an old Nortel with a few hundred extensions, I tell you. And I went home and died on my couch, having been awake and actually working (prep, etc) for about 40 hours.
At 7:23AM, my phone rang. It was my manager. Their entire network had crashed, hard. They blamed us. They were livid. I read my manager the NSFW riot act, hung up, and went back to sleep.
Turns out that after we left, some unknown person had plugged both external switched ports of a deskset into both ports on a wallplate connected to a then high-end HP Procurve switch, which itself connected to a factory and office tower full of other HP Procurve switches carefully set up in a redundant "mesh fabric" mode. This carefully-constructed, redundant network then died in a broadcast packet storm.
Once they found the error and unplugged that one extraneous heads-will-roll wayward wire, things more-or-less instantly returned to normal.
(STP would've instantly made this a complete non-issue, but at that time STP and HP's mesh conflicted with eachother and could not cohabitate. I understand that this was subsequently resolved, though I don't deal with HP switches often enough to verify.)
Only happens in old code (Score:2)
What a shitty headline (Score:2)
For starters, assuming you fall prey to this, all you lose is the configuration of a single switch. If losing a single fixed configuration 1U switch causes your entire datacenter to go down, your datacenter is badly designed.
Second, this requires a particular style of booted cable, not just any booted cable. Most datacenters I've worked in don't use booted cables in their switch ports. Their cables are cut to length and crimped by hand. Booted cables can be a bitch to get out of the port, especially on 1U 4
Slow news day? (Score:3)
Sure this is funny, but the workaround in TFA is pretty straightforward.
Disable Express Setup with this command while in config mode:
Someone explain to me why you'd run Express Setup after deploying this switch?
More interesting wording (Score:2)
The cables, which are sometimes accidentally used in datacenters
In my opinion there's not any specific definition on that they shouldn't be used in datacenters - they do have the advantage of protecting the tab on the RJ-45 connector pretty good and would actually be preferred over unprotected connectors.
Overall the button placement is pretty stupid, and is probably the result of optimizing the size of the unit. So if you run a data center, then you will learn to deal with the button location.
Realize that this problem is just annoying, there are bigger design flaws in t
It all makes sense now (Score:1)
Ahh, is that the switch and cable combo Ubisoft is using for Uplay? So it's all really Cisco's fault then!
Single point of failure (Score:2)
But ok. It's Cisco. You'd expect that from them.
If this is a problem... (Score:1)
Easy mistake (Score:2)
I once did something similar. I had a screen on a web app which had a form. On the next screen the Delete button was at the same place the submit button on the form.
The nice lady user had a habit of DOUBLE clicking for some reason. Which means she submitted the form and then deleted the record directly in the next step because the second click went to the delete button.
Took us a bit to figure out why the docs were deleted.
Cut the damn boot off (Score:2)
These are not datacenter switches (Score:2)
The 3650 and 3850 are access layer switches. These are used in closets to connect client devices (desktops, phones, wireless AP's, etc). These are not top-of-rack server switches or core switches for datacenter usage.
"Accidentally" why not intentionally (Score:2)
I'm not a network engineer but why are those types of cables not supposed to be used? The article seems to imply that using these hooded cables is wrong. I can see why they wouldn't be cost effective or not necessary but why wrong?
NOT A RESET BUTTON (Score:2)
I think the first thing we all need to understand is that the button mentioned is NOT a reset button. It's the display button for the lights and is clearly labeled "mode". It cycles between the different information modes such as speed, duplex, stack ID, POE usage, etc. See this article from the Cisco Support forums detailing how to determine which stack ID the different switches are as one example: https://supportforums.cisco.co... [cisco.com]
I laughed (Score:2)
Okay, you gotta admit- that's some funny shit. Poor design allows you to bork your entire network by plugging in a cable. Hilarity ensues.
And what's this crap: "The cables, which are sometimes accidentally used in datacenters..."
Cables are "accidentally" used? WTF?
while funny, how is something from 2013 "news"? (Score:1)
... and this is 'current news' because?
Simple solution (Score:2)
Re: (Score:3)
I've seen a few of them, but they're pretty rare. I avoid them because usually the boot does more harm than good - getting stuck under the tab, sliding to the side and making it hard to push the tab, getting stuck next to the jack/port, especially if it's slightly recessed like you might find in an IP phone. And, apparently, breaking Cisco switches. Something like This [showmecables.com] would probably do it.
Incidentally, I'm not really a Cisco guy, but I have helped recover a couple secondhand switches for friends and I'm pr
Re: (Score:2)
The good "tab protector" cables actually use a hood,, not just a fragile tab, second reversed tab above the connector tab. I've had some problems with even those where the recess for the connector was too deep and too tightly encased, making it impossible to get a hooded cable in place. Those are especially handy because they cost considerably more, and can require a small screwdriver to lever under the hoold and release the connector tab.
Re: (Score:2)
Those are especially handy because they cost considerably more
Does not compute.
and can require a small screwdriver to lever under the hoold and release the connector tab.
Still not seeing what is "especially handy" about that.
Re: (Score:2)
The phrase "especially handy" was meant to be ironic.
Re: (Score:2)
I usually take a knife or scissors to that hood in those cases, and give it a circumcision.
Re: (Score:3)
Re: (Score:3)
Article only show drawings/illustrations - where's an actual picture
That's exactly what I said in Sex Ed!
Re: (Score:3)
You home-schooled kids are so funny.