Duke Wireless Problem Caused by Cisco, not iPhone

jpallas writes "Following up to a previous Slashdot story, it now turns out that the widely reported problems with Duke University's wireless network were not caused by Apple's iPhone. The problem was actually with their Cisco network. Duke's Chief Information Officer praises the work of their technical staff. Does that include the assistant director for communications infrastructure who was quoted as saying, "I don't believe it's a Cisco problem in any way, shape, or form?""

I'll feel bad...

[CCFreak2K (930973)]

...for the poor guy who said it wasn't a Cisco problem when he starts getting those Apple fanboy death threats.

More information?

[physicsnick (1031656)]

I'm curious to find more information on this. TFA just says "Cisco has provided a fix". What nature of fix was this? Was it actually a flaw in the routers, or did someone just configure them wrong?

Given the widespread use of Cisco routers compared to the isolated nature of the problem, it sounds a bit like Duke is just trying to save face.

Re:More information?

[ZWithaPGGB (608529)]

"Given the widespread use of Cisco". So Windows must be pretty good too, right?

Cisco is the Microsoft of networking gear. Their stuff is complete crap compared to the alternatives in every category. It's also overpriced.

People buy Cisco for the same reason Chambers used to be able to get them to buy IBM Front End Processors (where he cut his teeth as an exec), because No-one gets fired for buying what everyone else buys. They SHOULD be, because they are just buying on inertia, but they don't.

Re:More information?

[physicsnick (1031656)]

"Given the widespread use of Cisco". So Windows must be pretty good too, right?

You misunderstood. I wasn't implying anything about the quality of Cisco routers.

Suppose Duke University (and only Duke university) suddenly has problems with all of their Windows boxes. Do you think it's a Windows problem? Given the widespread use of Windows compared to the isolated nature of the problem, it's far more likely that they themselves configured something incorrectly, otherwise all universities should be encountering similar problems.

This isn't to say that there aren't such problems; just as you said, both Cisco and Windows have widespread flaws that affect all universities. But for THIS particular problem, it's more likely to be just a misconfiguration, simply because of the fact that it's localized to Duke.

Re:More information?

[HockeyPuck (141947)]

Cisco is the Microsoft of networking gear. Their stuff is complete crap compared to the alternatives in every category. It's also overpriced.

I think you hit the nail on the head. Alternatives in every catagory. Which means you have 500 different vendors. From core routers, to access switches to firewall appliances, to Content/Caching engines to telephony to wireless, heck Cisco even makes storage switches. If there's a nework problem, you call up ONE company. You sign one large support contract, makes it very easy to have 'one neck to choke' when there's an issue.

When you build a server (not a hobbiest linux box at home) would you rather buy all the parts (cpu, ram, disk, etc..) from ONE vendor, or would you rather buy each component from someone else? You'd call up IBM/hp/dell/sun and order a server, so when the ram breaks you call the same vendor as when the CPU breaks.

While cisco gear may not be the best in every catagory, the solution as a whole is pretty good and there's not a networking vendor that can provide an 'end to end' solution. Plus there's something to be said for being able to put firewall/content/PoE/WAN modules in a single chassis.

Integration and consolidation does save power.

Re:More information?

[ZWithaPGGB (608529)]

Juniper for routers. Extreme for Network Switches. Juniper/Netscreen, Fortinet, or even Checkpoint for firewalls. Intruvert for IDP. Aventail for VPN. Aruba for Wireless.

Even a Vyatta or other OSS router is as good as or better than all but the biggest, and most horribly expensive, Ciscos.

But you knew that, because you couldn't point to any evidence that refuted my opinion that Cisco has more than just market share in common with MS.

Re:More information?

[HockeyPuck (141947)]

I'm not saying that there aren't vendors with single produts that are better, but NOT all companies/customers are looking for 500 different vendors. You wouldn't build a server farm from 50 different linux servers b/c "IBM w/redhat is better at dns, HP w/Ubuntu is better at samba, Dell/slackware is better at sendmail..." you'd go outta your mind supporting such a hetergenous infrastructure.

Cisco/MSFT have plenty in common. All religions aside, when you hire someone it's much easier to find someone that is familiar (CCIE) with a broad range of cisco products than to find one that has (as you put it), "Juniper for routers. Extreme for Network Switches. Juniper/Netscreen, Fortinet, or even Checkpoint for firewalls." The same holds true if you were hiring someone with office skills. It's much easier to find someone that is well versed in MS-Office than it is to find someone that has the same skillset in lotusnotes, wordperfect, etc...

Building an IT infrastructure is more than just having the 'fastest, best out there'. It's building the best solution for YOUR environment. I work with plenty of clients that have Juniper in the core and cisco at the access/distribution layer.

Correlation is not cause and effect

[PIPBoy3000 (619296)]

This is unfortunately a common issue with people. When two events happen at about the same time, people assume they're somehow connected. The autism and vaccine link, for example, is one of those things where they get their shots and soon afterwards, they notice their child is acting strangely. Then there's the old "this coincidence must be a sign of the divine" theory.

We run into this all the time when doing server administration. For example, one of our developers found that web pages were slower on our new virtual servers. The obvious thought is that virtualization=slow. It turns out that compression hadn't been turned on for those servers. Since he was going over a slow VPN connection, it made a fairly significant difference. Once switched on, they worked about the same as real servers.

Most Don't understanding networking

[henryhbk (645948)]

Many network IT folks just understand how to change settings on routers (what you learn to do in a "certification" course on a router) and understanding networking. Networking is more than just some router settings, and understanding the organic interdependent flowing nature of a network is critical to debugging problems. Just knowing something is causing a problem, and blaming the most recent change as the cause (as opposed to some underlying problem that this change simply brings to light). A senior IT official should, even if he doesn't know the exact problem, know that weird entworking problems are often way more complex than they seem, and should not jump to knee-jerk conclusions (especially based on some 1994 anti-mac bias about networking)

Everyone is a winner

[BillGatesLoveChild (1046184)]

Cool. Cisco screws up, iPhone gets blamed, but nobody minds, because iPhones are so cool.

Boss: "Did you get those reports done?"
Underling: "Sorry Boss, I Couldn't. iPhone Congestion."
Boss: "iPhone? ... (smiles) iPhones are cool aren't they!"
Underling: "They sure are boss!"

Boss wanders off feeling good.
Underling returns to screwing around with his iPhone.

Cisco gear just isn't that good.

[CRC'99 (96526)]

I think that after spending a number of years working in Cisco only networks, I'm constantly amazed at the generally poor compatibility and functionality of Cisco equipment.

This ranges from critical recovery steps being removed from the 7200 series G2 NPE (NEVER make one of these crash to ROMMON on boot. The fix is to RMA the NPE) for Xmodem recovery of bootloaders - something a basic 827 router has to their latest 7961 VoIP SIP phones that are apparently RFC compliant for SIP communications - but aren't.

There are MANY things that make Cisco equipment worse and worse as the years go by. Part of it I believe is the outsourcing of the people who write the software for these things now. Chances are that they weren't even around with Xmodem was in use - and I bet a lot of the coders have NEVER admin'ed a network of Cisco gear. This is the only thing I can think behind removing essential recovery procedures for $35,000AU routers.

There's a whole new direction that Cisco is heading, and with the stupid things missing from their new gear, I'm starting to wonder if it's a direction that will have huge impacts for the worse in the network admin side of life.

Re:Cisco gear just isn't that good.

[jkbull (453632)]

...(NEVER make one of these crash to ROMMON on boot. The fix is to RMA the NPE)...

I understand the ROMMON, RMA, and NPE acronyms, but what's NEVER stand for?

To be fair....

[Vacuous (652107)]

To be fair, who hasn't had an issue where you were SURE it wasn't one thing, when it actually was. I would imagine most of you, like me, have seen issues where you still can't explain how you fixed it.

Obviously a Cisco Problem All Along

[smack.addict (116174)]

The sick thing is that it was OBVIOUS it was a Cisco problem from the start. If you make the assumption that the iPhones are somehow defective, it's still a Cisco problem because any defective behavior from an iPhone would be indistinguishable from malicious behavior from a student. The fact that the iPhone was involved really was a non-issue all along.

It was terribly irresponsible of them to go off blaming Apple and, worse, absolving Cisco of responsibility.

Jumping to conclusions

[faloi (738831)]

Seems to be all the rage at Duke. One would think they'd learn from their past mistakes.

Re:idiots

[nosilA (8112)]

I used to work with the "hair trigger IT moron." He has a CS degree from one of the best CS schools in the country, he has been running college networks since 2000, and he does, in fact, know what he's doing.

I will admit though, that he has been known to get ahead of himself. When he looked at the logs and saw a bunch of iPhone MAC addresses spewing garbage, but no other devices are, he assumed that it's an iPhone problem. The quote in Network World is unfortunate, but he is no "hair trigger IT moron." He continued working on getting to the root of the problem and solved it yesterday.

Re:idiots

[peragrin (659227)]

go back and read the slashdot article on the subject when this first came out. Dozen of slashdot guys were reporting that cisco routers and WAP's have a flaw that would enable just such a solution and that you had to patch the routers with a patch that Cisco already had made.

Cisco makes some solid equipment, but when they let flaky stuff loose it's really flaky. It is also not something you announce to the world first, without throughly checking out your own equipment first, especially when the iPhone was working perfectly fine with tens of thousands of other access points around the country.

Re:idiots

[eli pabst (948845)]

He continued working on getting to the root of the problem and solved it yesterday.

See, that's what he gets for not reading Slashdot. If he would've just sat there eagerly refreshing his browser, he would've seen several people post the solution to their problem last week and could've taken the weekend off. Hope this is a lesson.

Re:idiots

[eipo (1131163)]

Your assumption at the skill of the network folks at Duke is sadly mistaken. The people at Duke are very intelligent and experienced group of folks that detected a problem on the network that seemed to be related to the iPhone. In turn they contacted BOTH Apple and Cisco and began running dumps to try to figure out what was going on. In the beginning it did appear to be caused by the iPhone and only after a lot of testing and help from Cisco was the true problem discovered. They had Cisco network that functioned perfectly until iPhones started popping up, it wasn't a far stretch to suspect the new device introduced into a working system.

The only thing they did poorly was fail to realize how much the techie world is hot and bothered over ANY news about the iPhone. Had the cause seemed to have been the the latest Crackberry this would have never sweep through the iPhone loving media/techie-verse this quickly.

So come off your superiority complex a bit and cut them some slack. They managed to detect and solve this issue within a week on a massive University network with half the tech world breathing down their collective necks. It wasn't the work of inexperienced MIS folks but group of talented network professionals that had the misfortune of publicly grappling with the iPhone juggernaut and half million know-it-alls on forums like this.

Re:idiots

[EGSonikku (519478)]

Perhaps they should have waited a week then and not announced prior to discovering the true issue that the iPhone was at fault.

Re:idiots

[jayhawk88 (160512)]

Uh huh. Because you've never made a mistake or misdiagnosed a problem when something is broken and your entire customer base is screaming at you to fix it.

Jesus, I love how you all are posting here like you single handly created the first router and invented TCP/IP. Let's try and look at this from the Duke IT perspective: 1. Wireless network is (presumably) working great. 2. iPhone is released, students start showing up with it. 3. Wireless starts getting slammed. Yes it was a wrong conclusion and faulty logic but come on, was it really that horrible? When something breaks the first thing you ask is "What has changed", in this case iPhones were introduced to the network. I guarentee that would have been the first thing I would have looked at.

Re:idiots.. But it is true...

[Fallen Kell (165468)]

I and my group have experienced this at work all the time almost whenever a new person is hired into the network team. Cisco gear do NOT play nice with Sun Microsystems, be it their desktop workstations or their servers. The Cisco gear refuses to properly auto-negotiate with the equipemnt causing issues such as duplex/simplex mis-matches (i.e. the workstation thinks it is connected at 100 Full duplex, while the switch thinks it is connected at 10 Half duplex). Needless to say this causes all kinds of collisions, IErrors, OErrors, etc., on the system and the network. All the Sun gear must have their associate network partner's port forced to 100 Full, and we do the same for the system as well. How do I know the problem is with the Cisco gear? Because the workstation/server works fine if you use a HP, Xylan, Baynetworks, or other switch. The net network engineers immediately believe it is the Sun equipment because they have been brainwashed into believing that Cisco can't make a mistake or a poor product. It usually takes us to demonstrate using 2 or more other switches that the problem only happens on the Cisco. Cisco still denies that there is a problem as well.

Oh and if you don't believe me, do a google "Cisco problems with Sun"...

Re:you're

[uglyduckling (103926)]

Shouldn't that be "fuck's sake" or "fucks' sake" - certainly not "fucks sake"..?

Poor Sake

[Oyume (464420)]

Why is everyone picking on Sake? I mean, sure it's served slightly warm and in tiny cups, but it's not all that bad to drink, is it?

Re:So what was it

[sammy baby (14909)]

I prefer Charles Wang's assessment of the folks at Gartner.

"I want to choose my words carefully here, so I'm not misunderstood," he said. "They're a bunch of fucking idiots."

Sorry I can't provide an authoritative cite... but even if it's apocryphal, it's so perfect that I can't care.