Build Your Own NOC

Geminus writes "Ever wanted to build a cheap NOC but had difficulty explaining tech stuff to bean counting managers? Here's the basics on building one for under two grand. Makes for a pretty good dog-n-pony show, and proves useful too! Damn, I want to be an Armchair Network Operations Center General."

NOC????

[CyberBill]

What in gods name is NOC?
Nerds on Crack...
Nice/Naughty old Chicks...

-Bill

The scary thing is....

[beeudoublez]

what if your boss/manager saw this and decided this is all you needed for your budget?
Hard to justify higher costs when your proof of concept is some webpage discovered by your boss, we've all been there.

For a real opensource NOC

[losttoy]

You need:
1. A good network management system (Open-NMS)
2. A good systems monitoring system (MRTG+RRD Tool)
3. A good helpdesk software to follow trouble tickets.

My NOC is 66 square feet,3TB of traffic

[Anonymous Coward]

Bashed out a window so a fan can circulate air, installed 4 of the cheap open frame racks, use a OpenBSD firewall and all of our servers run FreeBSD. It costs next to nothing to set up. Idiots down the hall from us spend $1.5 million on their room, $100K just for the air conditioner. The funny thing is they do 1/100th of the traffic we do. Believe me, the "IT" industry is set up to rip you off if you don't know what you're doing. This stuff can be done a lot cheaper than the suits lead you to believe. This is how we survived the bubble while the floor outside our door got marked up from other occupants expensive equipment getting moved in, and then out!

Please hook me up with your vendor!

[Zero__Kelvin]

The article calls for:

1) At least three big monitors (the bigger the better), two smaller ones (17"), a KVM switch, and OOB dialup.
2) A 333Mhz system with 64MB of RAM and a 2.1GB Hard-Drive.
3) A barebones 600 Mhz system
4) A 333Mhz Windows based system.
5) A 1.2Ghz, 512MB, 20GB computer, with dual head Matrox card, with dual booting OS (Linux & Windows), Preferably Linux with a Windows VMWARE guest OS

All the above for under $2000.00? Can we also assume that the author works for free, so that setup cost is $0.00? I haven't priced VMWARE in a long time, but if memory serves, that should be near or over the 2K mark by itself. Perhaps the author meant under $20,000.00? What am I missing here folks?

WTF?

[bazik]

The best Linux Dual-Head OS is SuSE 8.3.

WTF has Dual-Head support to do with the distribution?

The Christmas tree

[BrookHarty]

How many other people out there, went over the correct shade of yellow for the alarm lights with a vendor? Funny stories about NOC design. This thread could have some very interesting stuff, if people would let some company secrets slip. ;)

But onto my point.

Biggest thing about a noc, is you need to see the alarm, other than taking action, missing an alarm is the worst design flaw. Filter, Page, auto-ticket, there are many things a professional NOC can lend some experience on design. Not everything has to cost, in fact many opensource software works great. (Big Brother anyone?)

BTW, windows and vmware? Pfft.. Worst thing you want is a crash in the middle of working, Solaris and xterms. Eye-candy is the worst thing to get in the way of working outages.

Humm, also a good ticketing system is important, if you want to page out someone, you need to have enough detail for the person to do their job.

Oh yea, give me an Aeron [google.com] Chair also. I know, its .com ish, but they do feel great.

Akamai NOC Tour

[mcbridematt]

You might want to have a look at Akamai's NOC at http://www.akamai.com//en/html/about/nocc_tour.htm l

Pictures of Akamai's NOC also were in the Wired article about the Slammer Virus a few months ago.

You really don't want to work there...

[yalla]

I used to work in a NOC of a major cellphone carrier. Working in shifts, staring at your HP Openview, no coffee/food at your desk, boring calls from the staff "Oh, the connection to server ABC isn't working. Do something!" - and when really something goes wrong you feel you want to be an octopus - you need 8 arms for 8 phones.

Essentially the job is: Stare at network map, wait for thingys to blink, make calls.

Yalla.

Dual-headed video

[John Courtland]

...is indeed the greatest thing since sliced bread. I've had it for about 2.5 years now, and one day when my primary monitor went out, I almost couldn't function. Being able to have Visual studio open in one screen and All sorts of Docs and a web browser in the other, I don't know how I did it before...

In the same vein, nVidia included a really nice feature in their latest drivers (I think it's been around since the 4x.xx series, but it wasn't as refined) that lets you "throw" a window. Pure genius, whoever invented that. With 2048 pixels of desktop space, it actually takes over an entire mousepad to move a window across the desktop. With throwing, I just flick my mouse. If I have a few IM windows open, a few Putty terminals, etc etc, it's great to just get stuff out of the way real fast and put it all into a known area.

Re:Please hook me up with your vendor!

[Soko]

Perhaps he meant "$2000 Capital Investment"?

Most of what he calls for can usually be gleaned from the office "PC Bone Yard". The most expensive item is the big dual head computer with associated software. Getting it all for under $2K would be a challenge, but not impossible. As for working for free - he set this up for his employer (An assumption - I'll RTFA when it's not /.ed), so they'd be paying him anyway. Since he's trying to make himself more productive, they'll get more for less in the end. I can't see a problem with that, as long as his other duties are kept up as well.

Sliping stuff you need in under the coprorate radar is easily done with FOSS. When setting up a NOC, if you spread any purchases you need out a bit most of them will be cheap enough that they can be bought on an expense account or with petty cash - you avoid Budget Comittees and/or the Accounting Dept. Call it a "Test Case", and use it to prove that a NOC is a good investment, not just some toy or geeky buzzword. Being able to have concrete numbers that say "See? My NOC isn't really expensive, but it adds a ton of value." will keep the bean counters happy. Once the NOC is in place and you show it has value, you will get to keep it - and sometimes expand it.

This is one of the ways that FOSS shines - you can (most times) just get the job done without getting caught up in coprorate red tape, since the inital capital outlay is usually minimal.

Soko

Worthless article..

[Thomas Charron]

This article was a complete waste of time..

I could just as easily post an article saying 'Get *4* Tires, *2* axells, and engine, and a few other things. Toss them all together, and you just made your own CAR!!'

I mean cripes. It's not talking about ANYTHING besides 'buy cheap puters and put neat graphics up'.

I've had bosses that could have written this article.. Heck, I bet they did. 'Whatcha wantt a fluke for? I mean, we BUILT you a NOC for a grand!!' Bear in mind, the 'NOC' was a closet with two monitors I salvaged..

I dunno, perhaps I'm just getting old but..

I fee like I just wastes a good minute of my life reading that..

Re:The article.

[SkewlD00d]

NOCs... oh, like the one Enron had for petrochem market trading? HAHA. All u need is nmap, snort, ethereal, neotrace pro (runs on wine i think), dshield's log generator, etherape, and nagios (netsaint). Nagios is fucking l337. But a whole solution that integrates CRM (ticket manager) and monitor/response would be nifty w/ a slick interface. Something like neotrace + etherape + DIDS monitoring + nagios would be awesome.

Lol, u can't find wardrivers if they have their transmitters turned off. ;)

lmao... red phone... a simple circuit can be used to direct dial a hard line to the boss's office or something. Hell, a VoIP setup should be ez (assuming u have real encryption goin).

BTW, I dont see anywhere to download source for Coyote (www.coyotelinux.com) (Vortech Consulting, www.vortech.net). Isnt that a GPL violation? *Sigh* Yet Another closed-source whoring of modified GPL projects for monetary gain. (YACSWOMGPFMG).

Re:The article.

[boaworm]

Another way of doing that is to connect the machines with a Hub instead of a Switch, and have one machine configured without an IP, only raw logging of network traffic.

The idea is that whatever goes on out there will be logged/dumped, but never executed/analyzed, on this machine. And since it has no IP, it does not show and cannot be addressed. So if you have an intrusion, this machine is uncontactable, but still will hold all network traffic for you to analyze later.

Kind of like making
bash# ln -s /dev/lp /var/log/messages

Pretty hard to clear up the trace now, huh ? :)

Re:My NOC is 66 square feet,3TB of traffic

[Anonymous Coward]

We wanted better air....

so I installed a window air conditioner through the wall behind every 2 racks, and then we walled off the front of the racks so you had 36 inches in front of them and 24 behind them. plenty of room to work, and swap out heavy equipment and servers. and my airconditioning costs $250.00 per air conditioner, and cince each unit only draws 7 amps, no wiring needed.

We have a "rogue" NOC here cine corperate is filled with prima-donnas that want it their way and not right... so we firewall off from corperate and run our own NOC.

we boast 99% uptime.. they can't. we havent had a virus infection for over 2 years in our WAN (16 offices covering 3 states) they cant keep them out for a month, we have fended off 3 break in attempts (ALL FROM THE CORPERATE FEED BTW!) they cant.

CNN

[pyite]

Can't underestimate the importance of some news channel on at all times. During August of this year, we were in our NOC and we saw our power blip for a second and heard the UPS alarms from the adjacent machine room. Shortly thereafter, we found out we were on diesel power. Our monitoring tools began to show remote devices going down, some coming back, some not. I noticed my SSH session to home died around the same time. I began to worry. I called my house to see if my answering machine would pick up. No dice. It was at this point we realized a big power failure had hit us. A few minutes later, the reports started coming in on CNN that all of New York had gone down, etc. Eventually it all made sense, but it was definitely important to have CNN... even if we knew about the power failure before they did.

Re:For a real opensource NOC

[Cramer]

• (MRTG/RRD, OpenNMS) are mediocre to the point of unusability

I cannot speak to OpenNMS, although I am aware of it. However, MRTG is quite usable and valuable. No, it's not the best, most optimal traffic collection system out there, but it is simple, fast, and gets the jobs done when used appropriately. I've used it for many, many years. It does suck if you try to have one instance monitor thousands of interfaces on hundreds of devices, but more than one instance is perfectly functional. I suspect what you want is far beyond what MRTG was designed to provide. I'll admit, I'd like to have a database filled with millions of data points, but MRTG isn't designed to do that. (And at my previous job, monitoring all the ports I'd've liked to would've consumed a few hundred meg per year without archiving or consolidation.)

(FWIW, I know of one place that uses MRTG instead of HP OpenView, for which they paid $$$$$$, because MRTG is faster and simpler and runs on a 200$ PC.)

Homegrown applications are great... when the company will allow it. Many places simply do not want the responsibility or liability of creating and maintaining their own software. If it doesn't work correctly or fails, who do they have to blame but themselves? Plus, the people who wrote the app may not be there in a year thus creating a support issue. I've created a number of homegrown apps to deal with my job, but I'm the only one who completely understands them; when I'm no longer there, that's a problem. Additionally, let's be real here. Given the quality of commercial software, just how good do you expect internally developed software from one or two programmers (who may not understand the problem they are fixing) will be? The best stuff will be coming from the grunts who have to work with and fix stuff everyday -- shell, perl, tcl, etc. scripts born out of necessity. That stuff will not be "quality" nor will it make much sense to anyone other than the author. (I've been here way too many times.)