SANs and Excessive Disk Utilization?

pnutjam asks: "I work for a small to medium mental health company as the Network Administrator. While I think a SAN is a bit of overkill for our dozen servers, it was here when I got here. We currently boot 7 servers from our SAN, which houses all of their disks. Several of them have started to show excessive disk load, notably our SQL server, and our old domain controller (which is also the file/print server). I am in the process of separating our file/print server from our domain controller, but right now I get excessive disk load during the morning when people log on (we use roaming profiles). I think the disks need to be defragged, but should this be done on the servers, or on the SAN itself? When it comes to improving performance, I get conflicting answers when I inquire whether I would get better throughput from newer fibre-channel cards (ours are PCI-x, PCI-e is significantly faster), or mixing in some local disks, or using multiple fibre channel cards. Has anyone dealt with a similar situation or has some expertise in this area?"

That seems like an odd configuration

[hattmoward]

I've always kept the system disks local so the server isn't dependent on the SAN connection to boot. That said, do you have this SAN configured as a single shared filesystem or as a group of raid containers that are isolated from one another and provisioned to a single server? If it's shared, I'd say you need to take all but one server down and defragment from that. If it's not shared, they can all defragment their private filesystems at once (though I'd only do one or two at a time anyway).

Not enough information

[PapaZit]

First, what do you mean by excessive disk load? I'm not being facetious here. Do you mean that the SAN unit is pegged. How do you know that? Are the servers spending a lot of time waiting for I/O? Is the unit making loud noises? Or are the machines that are connected to the server just slow without the processor being pegged?

Also, while "have you tried defragging?" is a common home troubleshooting tip, it's not clear how you came up with the idea that the SAN has to be defragged. If you have reasons and you're just simplifying to keep the post short, great. Defrag away according to the SAN manufacturer's recommendations. However, don't become obsessed with it unless you know that fragmentation's an issue.

You need to spend some time benchmarking the whole system. Figure out how much disk, processor, network IO, and SAN IO are being used. Know what percentage of the total that is. Figure out exactly which servers are causing performance problems at which times.

"Find the problem" is always the first step in "fix the problem."
Once you know what's going on, you can deal with the problem intelligently. Are all the servers booting at the same time? Give them different spindles to work from or stagger the boot times. Are all of the users logging in at once? Figure out why that's slow (network speed, SAN, data size, etc.) and split the data across multiple servers and SANS or improve the hardware.

If you can make the case with hard data that the SAN is swamped, you can probably pry money from management to fix the problem. However, guessing that it -might- be something won't get you very far. They don't want to spend $20k on a fix to be told, "Nope. It was something else."