Ask Slashdot: Best Kit For a Home Media Server? 355
First time accepted submitter parkejr writes "I started off building a media library a few years ago with an old PC running Ubuntu. Folders for photos, ogg vorbis music from my CD collection, and x264 encoded mkv movies. I have a high spec machine for encoding, but over the years I've moved the server to a bigger case, with 8 TB of disk capacity, and reverted back to Debian, but still running with the same AMD Sempron processor and 2GB RAM. It's working well, it's also the family mail server, and the kids are starting to use it for network storage, and it runs both link and twonkyserver, but my disks are almost full, and there are no more internal slots. The obvious option to me is to add in a couple of SATA PCI cards, to give me 4 more drives, and buy an externally powered enclosure, but that doesn't feel very elegant. I'm a bit of an amateur, so I'd like some advice. Should I start looking at a rack system? Something that can accommodate, say, 10 3.5" drives (I'm thinking long term, and some redundancy)? Also, what about location — I could run some cat6 to the garage and move it out of the house, in case noise is an issue. Finally, what about file format, file system, and OS/software? I'm currently running with ext3 and Debian Squeeze. Happy with my audio encoding choice, but not sure about x264 and mkv. I'd also consider different media server software, too. Any comments appreciated."
raid (Score:2, Informative)
start playing linux mdadm and run raid 5 array with 1 or 2 spare drives. I built my first 1TB system about 9 years ago with 14 120G drives. ran without major failure for 6 years. Play with the array (remove drives, add new drives, etc) BEFORE you have a drive failure so you understand how to repair the failure. Let me say again, learn BEFORE you have failure. Otherwise, you'll freak out at losing all your data because you screwed something up during repair.
Larger disks (Score:5, Informative)
Larger disks. 4 TB should be available very soon (maybe now?).
http://www.extremetech.com/computing/108665-hitachi-ships-worlds-first-4tb-hard-drive-sticks-it-to-thor
Re:Why? (Score:2, Informative)
@Bonch: Congratulations, you are officially the Grinchy douche on Christmas.
OP: Great questions, I can not wait to see answers. Would love to hear more about what your software solution is for encoding, I've got a bunch of DVDs & Bluerays I would really like to get on the network, but a streamlined rip+encode+publish I have yet to achieve :-/ What are you using on the frontend? I've got various iDevices roaming about, and a Roku that does 720p for the projector, but haven't had much luck with mt-daapd so far.
I do not trust wireless for my projection room, I ran cat 6 everywhere when I moved in as nothing beats the reliability of copper. I've got a basement, so I put the server is right by the patch panel in the furnace room. Noise is definitely an issue. The only thing that scares me about the garage is bugs and what not gumming up the cooling enough to cause catastrophic failure. With a bunch of drives you need a bunch of airflow to keep everything cool, I can just imagine the intake fan sucking up mosquitoes, and coating the CPU cooling block with my dog's blood.
As far as rack vs tower, I started with a rack and ended up with towers. The rack systems just took up way too much floor space, and that space was poorly utilized to boot. It made a very large utility room feel much smaller.
HP Microserver (Score:5, Informative)
For a slightly more sane solution than rackmounting at home, consider the HP microserver.
Very low power (12W CPU), small, quiet, cheap, server grade, no Windows tax, holds four pluggable 3.5" drives plus optical (which some people swap for a 5th HDD for RAID5.)
http://blog.thestateofme.com/2011/05/14/review-hp-microserver/ [thestateofme.com]
http://www.silentpcreview.com/HP_Proliant_MicroServer [silentpcreview.com]
http://h10010.www1.hp.com/wwpc/us/en/sm/WF05a/15351-15351-4237916-4237918-4237917-4248009.html [hp.com]
http://forums.overclockers.com.au/showthread.php?t=905262 [overclockers.com.au]
If 8TB is full, you need to stop the obsessive collection of warez/pr0n/torrentz you are never likely to watch again.
Re:Why? (Score:2, Informative)
The submitter has very specific tastes and would rather keep conversion to a minimum because of the overhead and liabilities involved.
You see, the submitter is storing guro, bestiality, snuff, squirt/scat, and crush videos; all featuring underage animals, so the overhead must be kept to a minimum. He must make do with wired networking only, adding parts to his existing configuration as needed so his horrific pornography may be available to him immediately, whenever he wants to jack off.
I've seen his workstation. He lays a towel over his cheap office chair so he can plant his bare ass on it in comfort. He once remarked that he likes to keep his pants around his ankles so he can pull them up in a hurry when his wife or kids knock on the usually-locked door. The interior of all windows in that room are coated with aluminum foil. On the floor next to his chair lies a long athletic sock crusted stiff by what appears to be dry super glue. A marijuana pipe sits on his desk, next to the razor scratches, with loose nuggets and ampule shells of crystal meth and amyl nitrate respectively. He also keeps a noose handy so his free hand can choke his neck while his other hand chokes his chicken.
Yep, hard drives.
Re:Mac mini or apple Tv (Score:5, Informative)
A mac mini is a perfectly valid (if expensive) media solution. I personally use 2 synology NASs feeding a Mac Mini which is a dedicated iTunes server feeding 4 apple TVs (When I win the lotto I might try out a promise pegasus but until then the synology NASs are growing the library nicely). It's a bit of a pain to remux mkvs to mp4s but it works and it's a really nice solution once you've gone through the headaches of setting it up (yes better than a boxee box -- tried it, now gathering dust, better than every DLNA solution I've tried). The Apple TVs are just really, really nice media end-points and iTunes is a perfectly good management system.
But yea iCloud has no place in any video solution (your purchased shows can be streamed -- movies can not, and this solution will introduce you to your ISP's bandwidth caps very quickly).
Hard drive math and hobbyist heuristics (Score:5, Informative)
Ballpark figures, this isn't exact, redo it with your preferred constants, I'm just trying to explain my reasoning against huge enclosures with > 10 drives,
Standard drive idle usage (W) ~ 10W [1]
Low-power (green) drive idle usage (W) ~ 5W [1]
Cost of power ~0.20 $/KWH
Cost of an older drive per year = $17
Cost of a green drive per year = $8.50
Replacing 6x500GB older drives with one 3TB green power savings = $95/yr
So think about that for a sec. At $150[2] for a 3TB drive, you cover the price in power savings in 18 months. That's assuming that there is zero fixed-cost per drive. At the point where you are talking about adding SATA controllers or fancy multi-bay enclosures or, worse, external enclosures with their own PSUs (and fans!), the turnaround-point for older drives is far sooner.
I'm a hobbyist, I understand that it's really cool to make do with older hardware and feel like you aren't letting anything to go waste but sometimes using old hardware instead of buying new is penny-wise and pound-foolish. Spending money on increasing how many hard drives you can accommodate instead of just buying newer high-capacity lower-wattage drives is absolutely batty; especially when you get into the price for anything remotely good in the RAID dept.
My advice, move everything to the largest capacity drives that are reasonably priced (after the flood damage is sorted). Replace the drives when you can do between 4:1 and 6:1 replacement -- should be every 3-4 years. Live happily, quietly and simpler. Save money on power transparently.
[1] http://hothardware.com/Reviews/Western-Digital-2TB-Caviar-Green-Power-Hard-Drive/ [hothardware.com]
[2] I bought some Hitachi 3TBs before the Thailand floods at $130 on Newegg. Of course you would be silly as heck to buy hard drives now for your hobby storage project before they at least fall back to pre-flood level.
[3] http://www.newegg.com/Product/Product.aspx?Item=N82E16817182221 [newegg.com]
[4] Older drives need not go to waste, they can become offline storage with a simple USB dock[3] -- make a backup, throw it in an anti-static bag, leave it at your relative's house when you visit!
[5] http://en.wikipedia.org/wiki/File:Hard_drive_capacity_over_time.svg [wikipedia.org]
Thoughts from my home storage server experience. (Score:5, Informative)
I wrote about the latest storage server I built back in 2008, and a lot of my thoughts at the time are written up in http://www.tummy.com/Community/Articles/ultimatestorage2008/ [tummy.com]
However, to answer a few of your questions...
External disc enclosures? Avoid them like the plague. My initial experience with the 5 bay eSATA enclosures was pretty good -- sometimes it wouldn't pick up the external drives, but usually I could get it to find them after some tweaking, rebooting, etc... I ended up getting 3 of them, the AMS DS-2350S, which at the time were well reviewed, etc... I have since pulled all 3 of them out of active use and have them just sitting around. I don't know exactly the mode of the failures, but eventually after replacing some with others, I finally put them in internal SATA enclosures, which have been very reliable (I used these Supermicro CSE-M35T-1.
Also note that eSATA connectors don't really hold on that well. If anything, they're not as robust as internal SATA connectors, despite being outside the case where they can get banged around.
If I were to do it over again, I'd probably stick with the case I started with, with 5 internal 3.5" bays, and 3 front 5.25" bays, and put the Supermicro in there. I'd also probably go with fewer big drives rather than more smaller drives like I did previously (even though at the time the drives were free, I had them from another project).
As far as running it in the garage, don't even think it, unless your garage is not where you store your cars. I have some computers that I've run in the garage for the last 9 months, and they are filthy, I've had a lot of fan failures, lots of dust, insects, and random other crap. I put mine in our furnace room, which has enough extra space.
As far as using a server case? Hard to see the payback there unless you have a cabinet. Most server cases are HUGE, heavy, and expensive. A 3U case with 12 drive bays likely costs $500, plus you usually have to deal with special form-factor power supplies, expect to spend another $200 on one of those. I wouldn't do it, and I have a 3U 12-bay Chenbro case just sitting at my office that I could re-purpose.
As far as the file-system, I selected ZFS (via zfs-fuse under Linux) and I've been VERY happy with it. The primary benefit is that it checksums *ALL* data and can recover from some types of corruption or at least alert about corruption if it can't correct it. So, if you are storing photos or home videos that you may not be accessing very often, that's good peace of mind to have, I know in 10 years I won't go to look at some photographs I've taken and find they were silently corrupted. Of course, you could get similar benefits by saving off a database of file checksums and checking and alerting if they are bad. Really the only downside of ZFS that I've seen is that if you need to do a RAID rebuild it is a seek-heavy task rather than just streaming. I have a 8x2TB drive array that I'm currently rebuilding (drive failure, at work), and it's 33% done after 31 hours. A normal RAID-5 array would have rebuilt that in what, 10? The system is idle except for the rebuild.
If you care about the data going into it, make sure you checksum and verify the files regularly.
The 8 port PCI SATA card I got is fantastic, it's a Supermicro with the Marvel chipset and is very well supported (even supported by Nexenta).
Finally, all this data is encrypted, so if someone were to burgle us I only have to worry about them getting the hardware, I don't have to worry about them now having scanned bills and other documents and other personal and private data, etc... This is why I'm running ZFS in Linux, it gave me encryption plus ZFS (not available otherwise in 2008), as well as being an OS I'm very familiar with.
As far as OS, I am personally running CentOS on my system because that means I can install and set it up and then forget about it for quite a few years, except for regularly running "yum update". Debian should be fine, but you will get/have to track upstream changes more frequently.
Re:No reason to change from H.264 (Score:5, Informative)
1: More open than MP4 -- it has none of the ugly MPEG-LA overtones.
2: More codec support than MP4.
3: More consistency -- there aren't PS3-oriented versions or AppleTV versions of Matroska. MP4's device support might be wide, but when every player and device seems to have its own version with its own ridiculous, restrictive standards, it doesn't really qualify as supported at all.
4: Content management: Matroska is the easiest to mux and makes it great to work with alternate audio and video streams, subtitles, etc. You get the widest range of options of any container out there.
While Matroska might be used by some because it is perceived as more "elite", there are many very valid reasons for using it, mainly revolving around flexibility and openness. As someone who has worked with both containers, I can say that Matroska far more easily delivers what I want and the tools for working with it are generally free (in both senses), more usable, and more powerful.
Re:No reason to change from H.264 (Score:5, Informative)
Ditch .mkv as soon as possible. It's an almost completely unsupported container. Even among software that supposedly supports it there can be compatibility issues. It's popular in the ripping/pirate communities precisely because it's a pain to use. Just getting your videos to work on a regular basis is a mark of distinction.
I disagree. Having just converted my entire collection over to mkv, I'm never looking back. There are some great reasons to use it:
1. I've found MKV to have better support for chapters
2. MKV has heaps better support for subtitles (I could never manage to get subtitles to properly work across players using MP4)
3. MKV can hold just about any video and audio encoding, not just H.263 and H.264
4. (If you care) MP4 has patent issues, whereas MKV does not
Re:No reason to change from H.264 (Score:2, Informative)
Re:Why? (Score:4, Informative)
Agreed. In addition, rack is generally designed with the premise that space is at a premium and that it's well worth cramming as much as possible into the smallest space possible and then compensating for the poor natural airflow with high speed fans everywhere. That's fine if space is that big a premium, but ion a home environment, it's rarely THAT tight. A big roomy tower will run quieter, have less problem with failing fans, and probably will run a bit cooler. It'll also be less of a pain to work on.
Re:No reason to change from H.264 (Score:2, Informative)
1) This is debatable, but not really relevant for the discussion. Depends on how you judge the difference between known licenses and god-knows-what patent issues.
2) Not sure if this counts as a positive feature. AVIs are famous for the ten-million random codecs they might contain. Just because you can get the container to wrap around a particular stream doesn't mean you will ever be able to play it back again.
3) Those 'versions' are actually a feature of the h264 codec, not the container. You get the same thing on mp4 or mkv. You may not have realized it because it's just assumed that mkv is incompatible ;)
It's a combination of the supported profiles/levels. Basically rising levels of extra quality in exchange for higher hardware requirements. They are fully forward-compatible, so anything compressed with the standard profiles will play on any device that supports higher profiles, but something that has been recorded specifically for a high-end system will not play on a low-end one. Even phones are now starting to support level 5.1 high profile, which is the highest, so in another year or so the entire issue will disappear.
4) Again, I'm not sure this is a feature. Being able to jam anything you can dream of into a container is a poor choice if the goal is to play it back on any system but the one that was used to make the file.
Re:Mac mini or apple Tv (Score:5, Informative)
Ummmm, why two nic? The HDD and it's bus are the bottleneck - I cannot max out a gig-E nic now on my server but can play 1080P via x.264 with surround sound DTS encoded on a 100meg connection. No stuttering, no issues, no muss no fuss. It's wired for gig-E but sadly this nic refuses to synch at it.
I WOULD run some sort of redundant storage. However I wouldn't go traditional RAID. Striping data across disks along with parity makes for lots of speed, also means the disks never shut down. No thanks! I happen to use unRAID from Lime Technology. Parity isn't striped, it's held on a single disk. Each disk uses a standard format - ResierFS (ick). But's journaled and standard enough that recovery is easier. On top of that I can pull a disk and get data from it on another machine - one disk removed I still see all my data. Last but not least - if I lose multiple disks at once I only lose the data on THOSE disks and not the entire thing. Losing one disk I lose nothing. In 6+ years or so I've never lost more than one disk. Since parity isn't striped and neither is data one of my servers has ALL drives spun down and quiet, the other has just 2. Between them I have a bit over 22TB worth of disk BTW and not all are 2TB disks but as I fill smaller ones I swap in 2TB disks, the system migrates the data fine. Other programs can be run on this system, it's Linux based, but it's trickier than a full OS install - which has been done by some too.
Honestly to me it sounds like his current setup is working save for disk space. Why not just upgrade to 2TB disks? Surely he has more than 4 ports? If add-on cards are needed they are plentiful. I would also suggest using 4n1 cages for easy swapping. A low speed CPU is fine, underclock it maybe too - HDD bus is the bottleneck!
Re:Mac mini or apple Tv (Score:5, Informative)
He wants a media server not a HTPC.
then pretty much any old PC with a bunch of 3.5" drives would work.
I googled "build a media server" and found several guides. Here's one for $300 with six 3.5" bays [hothardware.com] and here's a 4U rackmount server case with twenty 3.5" drive bays for roughly $1,000 [hometheatershack.com]
Re:No reason to change from H.264 (Score:5, Informative)
Others have touched on your other points, so I wanted to address this:
Plenty of thin set-top box clients play mkvs already. Devices from Western Digital (WD TV Live), Netgear, Seagate, Roku, Popcorn Hour, Boxee, and many others all support mkv out of the box, with header compression support, subtitles, chapters, multiple audio and video streams, and some even support 3D (not the new mk3d format yet, but SBS works) and will play subtitles correctly. Most mkv files contain MPEG2, h264, or VC-1 video and AC3 or DTS (or the newer Dolby TrueHD, DTS-HD, etc) and all of these players handle those just fine. Don't blame the container for being flexible enough to allow any random codec. Blame whatever crap source you stole your videos from for using a random codec. Those of us archiving our DVDs and Blu-Rays will either encode in H264 or remux the original MPEG2/H264/VC-1 streams untouched and have no problems.
If you want to move videos to a phone, that's easy enough to do. The beauty of being a completely open container format means that it's trivial to demux Matroska containers into their component streams, which than then be remuxed into mp4 for devices that suck. Since you'll probably want to down-res the videos anyway for handheld formats (even on tablets you won't want higher than 720p), there's no reason to keep the originals in mp4. Keep your original, untouched videos in mkv and re-encode at lower resolution and bitrates into mp4 using Handrake for mobile devices.
Divx (yeah, whatever, they're still relevant) has adopted MKV as their HD container format, and the proliferation of "networked media tank" devices plus Matroska's openness makes it not only relevant but desirable for long-term video storage. Using 5+ year old devices like Xbox 360 and PS3 as your benchmark for what containers to use would be a bad idea (Xbox still doesn't even support 6-channel AAC in mp4, never mind supporting AC3).
Define the problem a little bit better (Score:4, Informative)
a) i am running out of hd space
b) i feel (=movies dont play without interrupting) that the processor is a little slow
c) i am bored over the christmas holidays
d) i am worried the thing explodes or falls apart
if only a): attach a NAS and make an archiving system
if a) and b) or d): there are enough shopping guides for off-the shelf pc's out there. You priority should be energy consumption, reliablility and space for more hds. Condiser external sata boxes
c) play around with something differnt.
Re:Why? (Score:4, Informative)
If you're putting a system into a dirty environment such as a basement then either buy or build an environmental enclosure to put the server in. It's basically a sealed box with large filters for cleaning the airflow through the hardware inside it. Enclosure fans are optional, an overtemp alarm/shutdown system isn't. Replace the filters every six months or so and it should be good.
Re:Mac mini or apple Tv (Score:4, Informative)
iTunes for Windows? Because iTunes for Mac is quite a bit different. i.e. it works much better. The Windows port is a bit of a hack in my opinion and doesn't work very well.