Duke Wireless Problem Caused by Cisco, not iPhone 195
jpallas writes "Following up to a previous Slashdot story, it now turns out that the widely reported problems with Duke University's wireless network were not caused by Apple's iPhone. The problem was actually with their Cisco network. Duke's Chief Information Officer praises the work of their technical staff. Does that include the assistant director for communications infrastructure who was quoted as saying, "I don't believe it's a Cisco problem in any way, shape, or form?""
deficient (Score:2, Insightful)
I would say that the network was deficient until the patch was applied. For him to say otherwise implies that there was no problem to begin with.
I'll feel bad... (Score:5, Funny)
Re: (Score:3, Insightful)
You mean when hack journalists start reporting unsubstantiated rumors of death threats.
More information? (Score:5, Interesting)
Given the widespread use of Cisco routers compared to the isolated nature of the problem, it sounds a bit like Duke is just trying to save face.
Re: (Score:2, Interesting)
http://hardware.slashdot.org/comments.pl?sid=25112 9&cid=19886053 [slashdot.org]
Our other routers and access points work perfectly. For instance, I have a dozen PCs with Intel network cards that when set to autonegotiate, they get pretty much crippled speeds (feels like dialup)...I hav
Re: (Score:2)
I don't know about anyone else, but I'm
Re:More information? (Score:4, Informative)
Cisco is the Microsoft of networking gear. Their stuff is complete crap compared to the alternatives in every category. It's also overpriced.
People buy Cisco for the same reason Chambers used to be able to get them to buy IBM Front End Processors (where he cut his teeth as an exec), because No-one gets fired for buying what everyone else buys. They SHOULD be, because they are just buying on inertia, but they don't.
Re:More information? (Score:5, Insightful)
Suppose Duke University (and only Duke university) suddenly has problems with all of their Windows boxes. Do you think it's a Windows problem? Given the widespread use of Windows compared to the isolated nature of the problem, it's far more likely that they themselves configured something incorrectly, otherwise all universities should be encountering similar problems.
This isn't to say that there aren't such problems; just as you said, both Cisco and Windows have widespread flaws that affect all universities. But for THIS particular problem, it's more likely to be just a misconfiguration, simply because of the fact that it's localized to Duke.
Re: (Score:3, Interesting)
Re: (Score:2)
Tell me, do you work for Oracle's support dept? Seriously, I think we've spoken...
Re: (Score:2)
Or even some sites that do these kind of comparisons?
Thanks in advance.
Re:More information? (Score:5, Insightful)
When you build a server (not a hobbiest linux box at home) would you rather buy all the parts (cpu, ram, disk, etc..) from ONE vendor, or would you rather buy each component from someone else? You'd call up IBM/hp/dell/sun and order a server, so when the ram breaks you call the same vendor as when the CPU breaks.
While cisco gear may not be the best in every catagory, the solution as a whole is pretty good and there's not a networking vendor that can provide an 'end to end' solution. Plus there's something to be said for being able to put firewall/content/PoE/WAN modules in a single chassis.
Integration and consolidation does save power.
Re: (Score:3, Insightful)
There are also costs, like lock-in -- not only are in a position to be taken advantage of by your single provider in terms of price, but you're actually likely to dimiss technically superior solutions if they don't come from your provider, and your solutions will be inflexible outside the bounds set by your provider.
Take Exchange email as an example. It's not a terrible way to do mail folders, and the i
Re: (Score:3, Informative)
Unfortunately, the "one neck" often turns out to be yours, rather than the vendor's...
The reason Cisco's gear is dominant in the networking marketplace has nothing to do with superior hardware, software, or service. It does, however, have quite a lot to do with the fact that Cisco was one of the first players in the IP router market, with products that frequently failed to interoperate with other brands
Re:More information? (Score:5, Informative)
Even a Vyatta or other OSS router is as good as or better than all but the biggest, and most horribly expensive, Ciscos.
But you knew that, because you couldn't point to any evidence that refuted my opinion that Cisco has more than just market share in common with MS.
Re:More information? (Score:4, Insightful)
Cisco/MSFT have plenty in common. All religions aside, when you hire someone it's much easier to find someone that is familiar (CCIE) with a broad range of cisco products than to find one that has (as you put it), "Juniper for routers. Extreme for Network Switches. Juniper/Netscreen, Fortinet, or even Checkpoint for firewalls." The same holds true if you were hiring someone with office skills. It's much easier to find someone that is well versed in MS-Office than it is to find someone that has the same skillset in lotusnotes, wordperfect, etc...
Building an IT infrastructure is more than just having the 'fastest, best out there'. It's building the best solution for YOUR environment. I work with plenty of clients that have Juniper in the core and cisco at the access/distribution layer.
Re: (Score:2)
Where is the benefit to everyone when the fix was subject to NDA?
In my experience, Cisco almost always pushes back with the same thing as Microsoft any time there is an interoperability problem: "We ARE the standard, so it must be them."
Old joke:
Q: How many big_dominant_company engineers does it take to change a Light Bulb?
A:
Correlation is not cause and effect (Score:5, Interesting)
We run into this all the time when doing server administration. For example, one of our developers found that web pages were slower on our new virtual servers. The obvious thought is that virtualization=slow. It turns out that compression hadn't been turned on for those servers. Since he was going over a slow VPN connection, it made a fairly significant difference. Once switched on, they worked about the same as real servers.
Re: (Score:3, Interesting)
Yea , but it was still 'something' related to the change that was made.
The dev may
Re: (Score:2)
If there is an exisiting network that "works" and then a new device is put into use on the network - and then the network breaks... it is reasonable to conclude either the device is a potential source for the issue.
While details are sparse I suspect the Appl
Re: (Score:2)
Re: (Score:2, Insightful)
So how was he wrong? The virtual servers were slower.
Re: (Score:2)
Re: (Score:2)
But, what we know now is that there is a definite correlation between pointing out someone's mistake, and them flaming you on slashdot. Oh, and modding you troll. That's also much more likely to happen apparently.
Re: (Score:2)
Re: (Score:2, Informative)
Not the iPhone but the iPhone (Score:2, Funny)
Most Don't understanding networking (Score:5, Insightful)
Re: (Score:2, Funny)
Damn those Ents and their slow decision making. First they nearly refused to act to stop the downfall of middle Earth and now, even worse, they are causing problems with Steve's divine creation. Personally I think we should ban them from having I phones if they are going to do this.
Re: (Score:2)
Everyone is a winner (Score:5, Funny)
Boss: "Did you get those reports done?"
Underling: "Sorry Boss, I Couldn't. iPhone Congestion."
Boss: "iPhone?
Underling: "They sure are boss!"
Boss wanders off feeling good.
Underling returns to screwing around with his iPhone.
Cisco gear just isn't that good. (Score:5, Interesting)
This ranges from critical recovery steps being removed from the 7200 series G2 NPE (NEVER make one of these crash to ROMMON on boot. The fix is to RMA the NPE) for Xmodem recovery of bootloaders - something a basic 827 router has to their latest 7961 VoIP SIP phones that are apparently RFC compliant for SIP communications - but aren't.
There are MANY things that make Cisco equipment worse and worse as the years go by. Part of it I believe is the outsourcing of the people who write the software for these things now. Chances are that they weren't even around with Xmodem was in use - and I bet a lot of the coders have NEVER admin'ed a network of Cisco gear. This is the only thing I can think behind removing essential recovery procedures for $35,000AU routers.
There's a whole new direction that Cisco is heading, and with the stupid things missing from their new gear, I'm starting to wonder if it's a direction that will have huge impacts for the worse in the network admin side of life.
Re:Cisco gear just isn't that good. (Score:5, Funny)
Re: (Score:3, Interesting)
I understand the ROMMON, RMA, and NPE acronyms, but what's NEVER stand for?
The NEVER stands for what I mean when I don't want to sit through 8+ weeks of rubbish from Cisco to get the thing RMA'ed (lucky it was in our testing phase and not live equipment). The TAC closed the case off and refused the warranty and it's been put on the account managers plate to fix. You can think of it as _never_ or never - which ever you like. I still refuse to use the flash tag though ;)
Re: (Score:2)
I think that after spending a number of years working in Cisco only networks, I'm constantly amazed at the generally poor compatibility and functionality of Cisco equipment.
There's a very simple explanation. Cisco, Oracle and Microsoft are 3 big gorillas who make a business of selling closed-source complicated implementations that are supposed-to-be standards-compliant. Unfortunately they are not truly standards-compliant, besides being overly complicated and obfuscated. Hence such compatibility issues.
Notice we have no clue in this particular incident, of what exactly triggered this response from Cisco equipment. It is like a Service Pack from Microsoft - we only get vague d
Re: (Score:2)
1) ROMMON - this is responsible for initialising the device and loading the bootloader from the bootflash: device. The bootflash is soldered onto the NPE.
2) Bootloader - This starts up and initialises the network. It parses the startup-config and then launches the IOS image.
3) IOS - This again parses the config and loads the actual unit.
The bootloader does have networking
To be fair.... (Score:5, Insightful)
Obviously a Cisco Problem All Along (Score:5, Insightful)
It was terribly irresponsible of them to go off blaming Apple and, worse, absolving Cisco of responsibility.
Re: (Score:2)
Re: (Score:2)
It Was the Stripper's Fault (Score:2)
Though the expert officials blaming the wrong party should find a new line of work [cnn.com]. I suggest politics.
Re: (Score:2)
Or the part where I linked to how the prosecutor was fired and disbarred (though it looks like the CNN article to which I linked is now broken).
Anonymous illiterate Coward can't even think straight when they agree with me. Maybe something's jamming the plate in their head, like fevered lust to join the Duke lacrosse team, though they're wedged between a gamer's chair and their keyboard, tied by a gamer's catheter to their mom's basement.
First blame the user (Score:2)
After all you and your staff would never mess up, your high status, reputation and salaries are at stake. The fact that you have outstanding trouble tickets, perform patches and upgrades without testing are coincidental.
So something goes wrong, you blame the user, remove them and claim problem solved. In the background you quietly find the problem and fix it (as part of routine maintenance). Your reputation is intact and all is good with the world.
Jumping to conclusions (Score:5, Insightful)
Re: (Score:3, Funny)
*ducks*
Haha Called It (Score:2)
rm999, blindbat, Doctor Memory, and Funk_dat69, you all owe me a beer. Go drink it for me and think about what you've done.
Re: (Score:2)
Something is still unexplained though (Score:3, Interesting)
first think, then post (Score:2)
It's kind of funny: most posts here complain about the Duke IT staff, either about their lack of competence or that they didn't wait until they had all the facts before claiming that this was an iPhone related problem.
Some people here who know the IT staff at Duke defended them and objected the claim of lacking competence, and there is no reason not to believe them, since everybody else is just guessing.
So most posters rushed to explain what happened without having seen the whole picture, didn't look in
Those responsible for the earlier comments... (Score:2)
As someone in another forum pointed out, and it's a good point...
Cisco provided a NEW patch, or just finally got Duke's IT staff off their ass and over to a patch set that's been readily available for some time?
Re: (Score:3, Funny)
Isn't that the entire basis for wikipedia?
Re: (Score:2)
Re: (Score:2, Informative)
Why do developer types always have be hating on your friendly IT folk? Developers are some of my best friends, and we ask each-other for advice all the time. I don't meet one stupid developer who I have to explain to what NAT, proxies or TCP options are and say 'ugh, those damn CS geeks!'
Re: (Score:2)
I know maybe a half dozen IT people that are worth their weight in gold pressed latinum, but I know many times that number more that are useless, and cannot figure out how to do simple things without having to have their hand held through the entire process, for something they do on a weekly basis. But they have certificates out the wazoo, so they look good on
Re: (Score:3, Insightful)
It's actually rare to find an industry where almost everybody is top-notch, simply because most companies don't want to pay the premiums for these folks. I can only think of a few off the top of my head: NASA, Google, most engineering firms...
Re: (Score:2, Insightful)
I don't meet one stupid developer who I have to explain to what NAT, proxies or TCP options are
That's because we already studied them in our Networking class and wrote our own implementations from scratch for homework assignments while we were getting four-year degrees... ;-P
;) I have a feeling you know a lot more by now than anyone finishing a one semester course... and a lot of CS grads get through without taking a networking course in the first place... but the trick is, we both know a lot more than the new recruits wearing their certification diapers and intonin
Just pulling your leg (mostly)
Re: (Score:2, Insightful)
Re: (Score:2, Insightful)
Of course there are arguments to work in a homogenous environment, I encountered the same arguments in my first year IS classes. My teacher insisted it was much more efficient to buy mass amounts of generic-branded PCs because the "support was better" in case of hardware failure. Of course I argue that if I build them myself, I already know b
Re:idiots (Score:4, Informative)
You've never worked in a large-scale IT environment. At my company, we deploy over 7000 machines per year (1/3 of the entire infastructure) in hundreds of sites around the world.
Are you going to build and "burn in" 20 machines per day? How many people are you going to hire (probably at least two dedicated employees, which is at least $300k/year in expenses)?
Who's going to handle packaging and shipping the machines (HINT: Dell/HP/Lenovo spend a LOT of time testing to make sure the PCs arrive intact)?
When there's a problem, are you going to be able to repair them locally, or will you have to ship them back to headquarters? You can't have a dedicated tech for a 10-man site, but major manufacturers can offer support pretty much anywhere in the world.
How do you know that your images are going to work? You don't want to find out that some chipset mismatch on 2% of your PCs is causing kernel panics.
When you have a problem, who's going to fix it? HP/Dell release BIOS updates for years to fix bugs. Good luck getting ANY support out of AsusTek/ECS/Tyan/Biostar/MSI/Gigabyte/Whoever after even 1 year.
Where do you dispose of your PCs? HP/Dell have extensive recycling programs in place.
How do you handle your purchase orders? HP/Dell are very good at working with your accounting department. It's not as simple as "put it on the Visa".
Of the 7396 PCs (desktop and notebook) we deployed in 2005, 143 have failed (1.9%). Generally, we find that the lifetime failure rate is below 3%. You're not even going to get close to that by building them in-house. One of my friends runs a custom-built PC business, and he sees a failure rate closer to 5%, with a large percentage being damaged during shipping.
As for "lasting quite some time", this indicates that you've never worked in a large IT environment at all. All major IT environments have some sort of lifecycle in place, typically 3 years but sometimes 4 or 5. A typical employee costs the company $150,000 per year (salary + benefits + taxes + etc) - if you replace a $1500 PC every three years, you're only spending $500 per year on the PC. It makes precisely zero sense to stick your $150,000 employee with old technology - if the new PC makes them even 0.5% more productive, you are saving $750 per year.
You may think that the big manufacturers just throw together parts, but nothing could be further from the truth.
Re: (Score:2)
Granted Dell gets huge discounts due to their size but you can build systems in house for around the same cost as a rebuilt Dell if you build a lot of systems. (AKA Assuming you don't pay full retail cost for windows.)
Don't forget Support Contracts make Dell a lot of cash and they outsource a lot of thi
Re:idiots (Score:5, Insightful)
I will admit though, that he has been known to get ahead of himself. When he looked at the logs and saw a bunch of iPhone MAC addresses spewing garbage, but no other devices are, he assumed that it's an iPhone problem. The quote in Network World is unfortunate, but he is no "hair trigger IT moron." He continued working on getting to the root of the problem and solved it yesterday.
Re: (Score:3, Insightful)
Well, he HAD to continue to work the problem, that's his job, he didn't really have the option of simply ignoring the situation, did he? Not sure that constitutes a pat on the back.
The quote in Network World is unfortunate, but he is no "hair trigger IT moron."
It does, in fact, show that he's willing to drop statements to the press such as "I don't believe it's a Cisco problem in any way, shape, or form," quite pre
Re: (Score:2)
Although I suppose I should say that they elucidate a plethora of needless added extra redundancy to give more force, pizazz and oomph to their otherwise mundane, banal and uncertain statements to elicit illicit confidence.
Re: (Score:2, Insightful)
Three things:
1.) if you think that Duke students will put up with an IT staff too closed minded to resolve a problem with their own network, which would result in making their $500 iPhones useless on wifi, you're crazy. The kids pay the big $$$ to go there, IT toads can be easily replaced.
2.) Duke U. has be
Re:idiots (Score:5, Informative)
Cisco makes some solid equipment, but when they let flaky stuff loose it's really flaky. It is also not something you announce to the world first, without throughly checking out your own equipment first, especially when the iPhone was working perfectly fine with tens of thousands of other access points around the country.
Re:idiots (Score:5, Funny)
See, that's what he gets for not reading Slashdot. If he would've just sat there eagerly refreshing his browser, he would've seen several people post the solution to their problem last week and could've taken the weekend off. Hope this is a lesson.
Re: (Score:2)
hound (Score:2)
Re:idiots (Score:4, Insightful)
The only thing they did poorly was fail to realize how much the techie world is hot and bothered over ANY news about the iPhone. Had the cause seemed to have been the the latest Crackberry this would have never sweep through the iPhone loving media/techie-verse this quickly.
So come off your superiority complex a bit and cut them some slack. They managed to detect and solve this issue within a week on a massive University network with half the tech world breathing down their collective necks. It wasn't the work of inexperienced MIS folks but group of talented network professionals that had the misfortune of publicly grappling with the iPhone juggernaut and half million know-it-alls on forums like this.
Re:idiots (Score:5, Insightful)
Re: (Score:2)
Re: (Score:2)
It's like say
Re:idiots (Score:4, Insightful)
No, what they did poorly was their job as problem solvers. They made the classic mistake of trying to solve a problem in reverse -- they started out with an assumption and then looked for evidence to support that assumption. For whatever reason, they wanted to rule out Cisco as the cause, so they did. This prevented them from finding the real cause as quickly as they might have.
Had they kept an open mind, they would have looked for more evidence before making a determination. For example, they could have asked some other universities (who undoubtedly now have iPhones on their wireless networks) whether or not the same type of problem was occurring there. With the answer being "no", they would have learned that the problem must have something to do with the combination of the iPhone and their specific network. That would have opened the doors to start looking at network configuration and/or faulty networking equipment. Obviously, that's exactly what happened in the end, but my point is that they erected a barrier in the problem solving process by "trusting" their Cisco equipment rather than suspecting it along with everything else.
I don't think the Duke IT people are incompetent, unintelligent, lacking in education, etc. Rather, I think this is an interesting little case study that illustrates how even highly competent people can allow their preconceptions to undermine their problem solving efforts. After all, our instincts, gut reactions, feelings, etc. are extremely useful when diagnosing a problem. They are often correct or at least highly informed on a level that is difficult to quantify. So, it's not easy to consider that your instincts may be completely wrong -- that you may be looking at an entirely new and surprising situation in which your instincts only serve to mislead you. Effective problem solving requires creativity, deliberate role-playing (e.g. "playing devil's advocate") and a certain amount of (forced) objectivity. Unfortunately, too few technical professionals display these traits when attempting to diagnose a problem and fail to understand that problem solving, in a general sense, is a discipline unto itself.
Re: (Score:3, Insightful)
This hardly seems like a minor error. They apparently went public with a premature conclusion about a new product before they actually knew what was going on, thereby holding themselves and Duke University up to ridicule.
And it is an unfortunately typical knee-jerk reaction reflects the arrogance typical of many IT departments: "The problem isn't with our network; it must be your c
Re:idiots (Score:4, Insightful)
Jesus, I love how you all are posting here like you single handly created the first router and invented TCP/IP. Let's try and look at this from the Duke IT perspective: 1. Wireless network is (presumably) working great. 2. iPhone is released, students start showing up with it. 3. Wireless starts getting slammed. Yes it was a wrong conclusion and faulty logic but come on, was it really that horrible? When something breaks the first thing you ask is "What has changed", in this case iPhones were introduced to the network. I guarentee that would have been the first thing I would have looked at.
Re: (Score:3, Insightful)
That word "presumably" was pretty close to the heart of this entire debacle. It's an assumption and those are the first things you should throw out when performing logical troubleshooting.
Come on what? Logically fixing this problem is their prim
Re: (Score:3, Insightful)
One does not need a technical background to know that if the iPhone caused problems for these kind of networks, we should be seeing them all over the place. A simple, logical process of elimination would soon cast strong doubt on the iPhone as the cause. It had to be the way that particular network interacted with the iPhone. Hence, it was the network and not the phone.
Re: (Score:2)
Re: (Score:2)
Re:idiots.. But it is true... (Score:4, Interesting)
Oh and if you don't believe me, do a google "Cisco problems with Sun"...
Re: (Score:2)
Re: (Score:2)
I just don't get this. Auto-detect has never failed me, except when I messed up myself. When one end is hard-coded to full duplex and the other end is autonegotiating, the other end picks half duplex. That's what the standard says must happen,
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Notorious? You sound like you've done your homework on this one. What are some examples that make these problems "notorious"?
I haven't seen the problems in a while, because I've been on the applications side for some time, but once upon a time you could see this problem just by connecting autodetecting, 100Mbps+FDX ports on two catalyst 5000 switches together. Sometimes they would figure things out and play nice. Sometimes they wouldn't, and bad things happened.
Beyond that concrete example (cat5ks aren't generally even in service any more because the backplane was somehow non-y2k-compliant) I've simply lost track of the number
Re: (Score:2)
Who puts together the best router solution?
Who's switches would run your ideal network?
Who kills at the SANS device space?
Who's VoIP gateways are seen as the best?
Who's Security devices rule?
What manufacturer offers all these type devices?
Who supports their products better?
Re: (Score:2, Informative)
Re:you're (Score:5, Funny)
Poor Sake (Score:4, Funny)
Re: (Score:2)
Sake is only "bad to drink" until you figure out how wildly dangerous hot liquor can be. Once you get used to the difference, it's great. Just don't forget to take the cap off the bottle before you heat it up.
</pedantic>
Re: (Score:2)
The tradition of heating sake started because it makes shitty sake more drinkable. Those who care still drink it cold.
And those who don't care, but prefer unfiltered sake, also drink it cold.
Re: (Score:2)
Re: (Score:3, Interesting)
Re:So what was it (Score:5, Funny)
Sorry I can't provide an authoritative cite... but even if it's apocryphal, it's so perfect that I can't care.
Re: (Score:2)
Are you sure you don't work for Gartner?
Re: (Score:2)
If you're turning off autonegotiate... (Score:3, Informative)
If you aren't, then the devices will come up as half duplex (assuming they properly implement the standards), you have a duplex mismatch, and you _will_ have network problems. 802.3u requires an end which is set to autonegotiate to assume half duplex if the other end will not autonegotiate.
Except, some Suns can not be forced and will only autonegotiate, in which case you MUST set the switch port to half duplex if you're forcing.
Re: (Score:2)
Re: (Score:3, Funny)
Re: (Score:2)