Ask Slashdot: Best Kit For a Home Media Server? 355
First time accepted submitter parkejr writes "I started off building a media library a few years ago with an old PC running Ubuntu. Folders for photos, ogg vorbis music from my CD collection, and x264 encoded mkv movies. I have a high spec machine for encoding, but over the years I've moved the server to a bigger case, with 8 TB of disk capacity, and reverted back to Debian, but still running with the same AMD Sempron processor and 2GB RAM. It's working well, it's also the family mail server, and the kids are starting to use it for network storage, and it runs both link and twonkyserver, but my disks are almost full, and there are no more internal slots. The obvious option to me is to add in a couple of SATA PCI cards, to give me 4 more drives, and buy an externally powered enclosure, but that doesn't feel very elegant. I'm a bit of an amateur, so I'd like some advice. Should I start looking at a rack system? Something that can accommodate, say, 10 3.5" drives (I'm thinking long term, and some redundancy)? Also, what about location — I could run some cat6 to the garage and move it out of the house, in case noise is an issue. Finally, what about file format, file system, and OS/software? I'm currently running with ext3 and Debian Squeeze. Happy with my audio encoding choice, but not sure about x264 and mkv. I'd also consider different media server software, too. Any comments appreciated."
No reason to change from H.264 (Score:5, Insightful)
Re:Mac mini or apple Tv (Score:5, Insightful)
I'm sorry, did you read the summary? Do you honestly think a Mac mini is a step up from that or solves the problems presented?
Re: (Score:2)
I was going to suggest Apple as a joke, but rereading it his main issue is storage and he does ask about OS as part of his issue. I don't consider storage and front end coupled so the Mac mini and iTunes is a legitimate interface suggestion as long as he understands the implications to the content.
Re:Mac mini or apple Tv (Score:5, Informative)
A mac mini is a perfectly valid (if expensive) media solution. I personally use 2 synology NASs feeding a Mac Mini which is a dedicated iTunes server feeding 4 apple TVs (When I win the lotto I might try out a promise pegasus but until then the synology NASs are growing the library nicely). It's a bit of a pain to remux mkvs to mp4s but it works and it's a really nice solution once you've gone through the headaches of setting it up (yes better than a boxee box -- tried it, now gathering dust, better than every DLNA solution I've tried). The Apple TVs are just really, really nice media end-points and iTunes is a perfectly good management system.
But yea iCloud has no place in any video solution (your purchased shows can be streamed -- movies can not, and this solution will introduce you to your ISP's bandwidth caps very quickly).
Re: (Score:2)
Atom based boards are pretty good. Make sure you get one of the 5xx series, the dual core hyperthreaded ones. Earlier versions have terrible performance but the 5xx cores are actually pretty good. Won't win any benchmarks for computationally intensive tasks but for responsive file serving and some background downloading they are ideal. I have an Intel mobo with the CPU soldered one and it draws about 11W including the HDD and 2GB of RAM most of the time, just be sure to get a low power DC PSU rather than th
Re: (Score:3)
Re:Mac mini or apple Tv (Score:4, Informative)
iTunes for Windows? Because iTunes for Mac is quite a bit different. i.e. it works much better. The Windows port is a bit of a hack in my opinion and doesn't work very well.
Comment removed (Score:5, Interesting)
Re: (Score:3)
Re: (Score:2)
Re:Mac mini or apple Tv (Score:5, Informative)
He wants a media server not a HTPC.
then pretty much any old PC with a bunch of 3.5" drives would work.
I googled "build a media server" and found several guides. Here's one for $300 with six 3.5" bays [hothardware.com] and here's a 4U rackmount server case with twenty 3.5" drive bays for roughly $1,000 [hometheatershack.com]
Re:Mac mini or apple Tv (Score:5, Interesting)
Just because you don't understand the request doesn't mean it's gobbledegooke, it means your knowledge is limited.
Personally, I don't trust any auto-encoding solutions as they easily go haywire. I'd suggest doing that all by hand.
Ext3 is fine, & a rack mount is a necessity. If you want smooth operation of the system, at least 2 network cards are a must (I run 3, 2 bonded for media/SMB, 1 for management & VPN). I'd suggest having a decent 16 port switch in the house & running the 2 (or more) cables to the box.
For DLNA I just run miniDLNA & for torrents I've just set up uTorrent with a web interface. There's very little my desktop actually does.
Re:Mac mini or apple Tv (Score:5, Informative)
Ummmm, why two nic? The HDD and it's bus are the bottleneck - I cannot max out a gig-E nic now on my server but can play 1080P via x.264 with surround sound DTS encoded on a 100meg connection. No stuttering, no issues, no muss no fuss. It's wired for gig-E but sadly this nic refuses to synch at it.
I WOULD run some sort of redundant storage. However I wouldn't go traditional RAID. Striping data across disks along with parity makes for lots of speed, also means the disks never shut down. No thanks! I happen to use unRAID from Lime Technology. Parity isn't striped, it's held on a single disk. Each disk uses a standard format - ResierFS (ick). But's journaled and standard enough that recovery is easier. On top of that I can pull a disk and get data from it on another machine - one disk removed I still see all my data. Last but not least - if I lose multiple disks at once I only lose the data on THOSE disks and not the entire thing. Losing one disk I lose nothing. In 6+ years or so I've never lost more than one disk. Since parity isn't striped and neither is data one of my servers has ALL drives spun down and quiet, the other has just 2. Between them I have a bit over 22TB worth of disk BTW and not all are 2TB disks but as I fill smaller ones I swap in 2TB disks, the system migrates the data fine. Other programs can be run on this system, it's Linux based, but it's trickier than a full OS install - which has been done by some too.
Honestly to me it sounds like his current setup is working save for disk space. Why not just upgrade to 2TB disks? Surely he has more than 4 ports? If add-on cards are needed they are plentiful. I would also suggest using 4n1 cages for easy swapping. A low speed CPU is fine, underclock it maybe too - HDD bus is the bottleneck!
Re:Mac mini or apple Tv (Score:5, Interesting)
unraid sounds just like what I was *designing* for myself (doh!).
the idea of lots of disks always spinning is just moronic for home-users! we are NOT a data center! the game is different, here; and it has taken the industry a while to learn this.
I had a gazillion disks spinning in big-time home raid. for years. blech! lots of noise, heat, failure modes and like you said, you lose too much, you lose it all! the idea of being able to take a single spinner out, mount it on a 'dumb' system and read it, that's super powerful! don't discount that, folks! its worth its weight in gold.
my approach is to just be an idiot human (I do that part well) and save lots of copies of the files to lots of places. I do that anyway. then run a database job that will traverse your filesystem, get every file's size, date, md5 hash and any other tag info you want to peek inside, for. and run a smart differ on it. keep 'n' copies; report partial copies; trim the ones that are beyond what you need (if you want 3 copies, kill any 4,5,etc version you see). use ANY format you want for the disks, ntfs, ext3, jfs, whatever. each disk has a disk-id ('blkid' on linux shows this easily) and so each file has a disk-id to show where its located.
the final part of this is to keep all disks spun down and have the database know which disk the file is on (the one you want to watch/play) and have it do power-mgmt (as I call it) and spin the drive UP. either have it auto-spin down or spin down via mgmt when your 'session' is over.
spinning the disks is easy if they are all external esata/usb/fw. even simple x10 powerline relays can do this (my first proto will use this style of distrib tech). for internal drives, you power their molex's on/off via relays, via software control.
noise is stupid! power/heat wastage is stupid! raid is stupid for most of us. lets get beyond the gazillion spinners and be smarter about our large disk collections.
(oblig: I'm using arduinos and linux as the controller on all this)
Re: (Score:3)
the idea of lots of disks always spinning is just moronic for home-users! we are NOT a data center!
I agree, and I used to have my media server under XP and the disks would sleep unless accessed. But for the last few years I'm Linux all the way (work/home/laptop/media server). And no matter what I try, the disks won't ever spin down. If I send a low-level SLEEP command with smartctl, it spins down and then right back up within 10 seconds. I've researched this problem without solution. Maybe someone here is brighter than me.
Re:No reason to change from H.264 (Score:5, Informative)
1: More open than MP4 -- it has none of the ugly MPEG-LA overtones.
2: More codec support than MP4.
3: More consistency -- there aren't PS3-oriented versions or AppleTV versions of Matroska. MP4's device support might be wide, but when every player and device seems to have its own version with its own ridiculous, restrictive standards, it doesn't really qualify as supported at all.
4: Content management: Matroska is the easiest to mux and makes it great to work with alternate audio and video streams, subtitles, etc. You get the widest range of options of any container out there.
While Matroska might be used by some because it is perceived as more "elite", there are many very valid reasons for using it, mainly revolving around flexibility and openness. As someone who has worked with both containers, I can say that Matroska far more easily delivers what I want and the tools for working with it are generally free (in both senses), more usable, and more powerful.
Re: (Score:2, Informative)
1) This is debatable, but not really relevant for the discussion. Depends on how you judge the difference between known licenses and god-knows-what patent issues.
2) Not sure if this counts as a positive feature. AVIs are famous for the ten-million random codecs they might contain. Just because you can get the container to wrap around a particular stream doesn't mean you will ever be able to play it back again.
3) Those 'versions' are actually a feature of the h264 codec, not the container. You get the same t
Re:No reason to change from H.264 (Score:5, Insightful)
Device support might be important to you, but I've just never understood why people contort themselves into making decision which are, frankly, idiotic simply because they have a crappy DVD player or something which only supports a few things in a handful of configurations. The most sensible and flexible solution is
What I care about most is media support. What can you fit in? MKV's subtitle handling is simply the best around bar none. You get complete flexibility to store hardcoded or softcoded subtitles in any way you want and you can easily do the same with audio, something which cannot be said for most containers. Unless you are trying to square the circle by wrestling with uncooperative devices you are unwilling to give up, there's really no reason to use anything else, at least in the present.
Re:No reason to change from H.264 (Score:5, Informative)
Ditch .mkv as soon as possible. It's an almost completely unsupported container. Even among software that supposedly supports it there can be compatibility issues. It's popular in the ripping/pirate communities precisely because it's a pain to use. Just getting your videos to work on a regular basis is a mark of distinction.
I disagree. Having just converted my entire collection over to mkv, I'm never looking back. There are some great reasons to use it:
1. I've found MKV to have better support for chapters
2. MKV has heaps better support for subtitles (I could never manage to get subtitles to properly work across players using MP4)
3. MKV can hold just about any video and audio encoding, not just H.263 and H.264
4. (If you care) MP4 has patent issues, whereas MKV does not
Re:No reason to change from H.264 (Score:5, Interesting)
1) I've never had problems with using Handbreak for chapters. But one anecdote is about as useless as another for this kind of thing. .mp4 but not the subtitles is the PS3, and all indications are that it was deliberate. So fuck Sony and the horse they rode in on (slung underneath). On the other hand, PS3 doesn't support mkv at all
2) Only system I've ever found that supports
3) This is not a good thing if your goal is to play back the content on any system but the one it was made on. Ever gone internet hunting for that one weird codec that you used for a few months a couple years ago? No? Me neither because I'm not dumb enough to think that 'can jam anything into it' is a good thing in a media format.
4) This is... debatable. Both formats are open standards and open source. You can look at the specs and the code for either. The patents for .mp4 are known and need to be licensed if you are a large commercial operation. The patents for mkv are god-knows-what and may or may not get eaten alive the first time that the patents ever become important. No one knows. Pick your poison.
Re: (Score:2)
The original claim wasn't that it's better, it was that it's an industry standard.
I own more devices that will play avi but not mkv than the other way round. Maybe it's just because it's newer, but it doesn't seem very standard to me.
Re: (Score:2, Informative)
Re: (Score:2)
Knowing a thing or to about genetics not much short of string theory makes Dna look simple
Re: (Score:2, Interesting)
Although I do agree that using x264 in mp4 is a good idea, the fact is that MKV (Matroska) is a very well accepted/supported container. You must feel like a complete idiot for claiming that is not.
The issue with some MKV files is that they include UNSUPPORTED video / sound formats. (in case poeple didn't know, MKV is not a video/audio format ... it is a container ... a kind of zip for media files). For example a lot of completely delusional people love to use Ogg Vorbis as sound, which is barely suppo
Re:No reason to change from H.264 (Score:5, Interesting)
While Handbrake is an awesome piece of software, it's not a remuxer. It doesn't support passthrough for the video streams. They will always be re-compressed. Often with little or no loss of visible quality, but some loss will occur.
Re:No reason to change from H.264 (Score:5, Informative)
Others have touched on your other points, so I wanted to address this:
Plenty of thin set-top box clients play mkvs already. Devices from Western Digital (WD TV Live), Netgear, Seagate, Roku, Popcorn Hour, Boxee, and many others all support mkv out of the box, with header compression support, subtitles, chapters, multiple audio and video streams, and some even support 3D (not the new mk3d format yet, but SBS works) and will play subtitles correctly. Most mkv files contain MPEG2, h264, or VC-1 video and AC3 or DTS (or the newer Dolby TrueHD, DTS-HD, etc) and all of these players handle those just fine. Don't blame the container for being flexible enough to allow any random codec. Blame whatever crap source you stole your videos from for using a random codec. Those of us archiving our DVDs and Blu-Rays will either encode in H264 or remux the original MPEG2/H264/VC-1 streams untouched and have no problems.
If you want to move videos to a phone, that's easy enough to do. The beauty of being a completely open container format means that it's trivial to demux Matroska containers into their component streams, which than then be remuxed into mp4 for devices that suck. Since you'll probably want to down-res the videos anyway for handheld formats (even on tablets you won't want higher than 720p), there's no reason to keep the originals in mp4. Keep your original, untouched videos in mkv and re-encode at lower resolution and bitrates into mp4 using Handrake for mobile devices.
Divx (yeah, whatever, they're still relevant) has adopted MKV as their HD container format, and the proliferation of "networked media tank" devices plus Matroska's openness makes it not only relevant but desirable for long-term video storage. Using 5+ year old devices like Xbox 360 and PS3 as your benchmark for what containers to use would be a bad idea (Xbox still doesn't even support 6-channel AAC in mp4, never mind supporting AC3).
Arduino. (Score:5, Funny)
Just kidding.
Re: (Score:3, Funny)
I think you would need to imagine a beowulf of arduino...
sysadmin'ed by natalie portman with grits in her pants who gets hugged by cowboyneal
Re: (Score:3)
and for a serious take on it, net.search on 'spinmaster DIY'.
I designed and built that to solve some staggered spin-up power supply issues, but it does have a back-end CLI and if you connect your usb-TTLserial cable and run a term program, you can spin up/down any disk you want at the 5/12v molex power point.
add in a database that knows which disk file X is on and you have a nice power-aware system that keeps noise down and only spins up drives (media drives) that are needed for a 'session'.
(the arduino par
A little under capacity, but perfect otherwise! (Score:2)
With drive capacities soaring, I wonder if you'll really need 10 drives.
You might want to try something like a fractal design array. It's a small htpc case for a microatx board. It has mounts for six 3.5" drives, and these days a microatx board will have everything you need, including integrated video for all your playback needs.
raid (Score:2, Informative)
start playing linux mdadm and run raid 5 array with 1 or 2 spare drives. I built my first 1TB system about 9 years ago with 14 120G drives. ran without major failure for 6 years. Play with the array (remove drives, add new drives, etc) BEFORE you have a drive failure so you understand how to repair the failure. Let me say again, learn BEFORE you have failure. Otherwise, you'll freak out at losing all your data because you screwed something up during repair.
Re: (Score:2)
Re: (Score:3)
You should not be dependent on a single array.
You can't just depend on RAID to save your butt. You need to have a backup. Given the size of these arrays, it has to be another array. There's really no other good option. It's kind of painful but unavoidable.
You should never be in the position to panic about your data. Your stuff should be safe even if you need to rebuild an array from scratch.
Larger disks (Score:5, Informative)
Larger disks. 4 TB should be available very soon (maybe now?).
http://www.extremetech.com/computing/108665-hitachi-ships-worlds-first-4tb-hard-drive-sticks-it-to-thor
I use mythtv (Score:3)
I use mythtv. It does pretty much everything. I love it.
Re: (Score:2)
I use mythtv. It does pretty much everything. I love it.
Out of space? Just add another box. The idea that your entire media store is attached to a single server seems a bit old fashioned.
Re: (Score:3)
Putting everything in a single server has its advantages - lower power consumption for one (because you do not need to heat additional CPUs, RAM etc) and less physical space used.
I used the "just add another box" method some years ago, so now I have ~3TB (in >10 hard drives) and 6 computers (including the main one), total power consumption ~1.2kW. I would love to replace 5 of them with a single server, but servers are really expensive, especially the ones that have more than 4 HDD slots and a rackmount (
Re: (Score:3)
Putting everything in a single server has its advantages - lower power consumption for one (because you do not need to heat additional CPUs, RAM etc) and less physical space used.
Do you think the OP really needs to keep his/her 8TB of disk powered up and online _all_ the time, just on the off chance they might want to watch something right now that can't wait 30 seconds for a second box to boot up?
We're still using DVD's at our house and the kids seem to watch a new movie a few times (or a few hundred times... :) in the first month or so then, with a few exceptions, not really watch it again for ages. All those older movies could go on a box that only boots up when required. In fact
Re: (Score:3)
I use LTO-2 tapes to archive the stuff that I already watched (as they take up less space than DVDs).
How many movies have you ever seen than you'd want to watch again?
Movies? Probably not that many, I do not watch that many movies anyway, mainly TV shows. TV shows? Sure, I watched B5 and DS9 like 6 times each.
As for the archiving, I archive because I do not know if I am able to download the same movie in the future. Maybe everybody stops seeding (the downside of BitTorrent compared to, say, Gnutella or ed2k, is that you have to specifically seed the file, instead of point
Re: (Score:3)
As for the archiving, I archive because I do not know if I am able to download the same movie in the future.
Ah. I made the flawed assumption that people would actually own the shows/movies they were watching and could just re-load from the original DVD.
Re: (Score:2)
Myth is not suitable for a modern HTPC, because it doesn't support streaming from netflix. That's a fatal flaw for a huge number of people these days.
what's a netflix?
Re:I use mythtv (Score:4, Insightful)
If you really want Netflix cheap you can buy a Western Digital Live TV Plus. They are under $100 and do extremely well with Netflix and a bunch of other apps. I have gone through several firmware updates and they keep adding new apps. In addition to the apps you can do streaming from just about any kind of server and USB connected flash drives.
HDMI and even optical audio out if you needed it. Netflix has a lot of HD titles, plus 5.1 audio on quite a bit too. I believe the Ethernet is gigabit, but I don't know off the top of my head.
Only thing it does not do well at all is DVD ISO files.
For under $100 though with a nifty little remote you will find yourself using it for more than just Netflix on a regular basis.
Consider it a nice little addition to a HTPC.
Re: (Score:2)
Re: (Score:2)
I am curious about this. I've been a MythTV user for many years now. I don't *need* Netflix, but it certainly would be nice to have. Part of the draw of a well built HTPC is having everything in a single place controlled by a single remote.
But, you're right that MythTV does not natively support Netflix. Maybe it could be accessed via MythBrowser? Or is a real Win/Mac (on a PC) client necessary?
Is there any other Linux alternative to getting Netflix running? Or, for that matter, any other movie streami
Re: (Score:2)
But, you're right that MythTV does not natively support Netflix. Maybe it could be accessed via MythBrowser? Or is a real Win/Mac (on a PC) client necessary?
Netflix depends on Silverlight. Silverlight is not supported (or usable) on Linux. Doesn't even work under Wine [winehq.org]. A good alternative to Netflix is Hulu [hulu.com]. It uses Flash for video.
Re: (Score:2)
Hmmm... that is unfortunate.
The suggestion to use Hulu is a good one, but I don't think it will particularly fit my needs. Hulu is focused much more on television shows rather than movies. According to their list, Hulu Plus has only 1800 movies available, of which only ~700 are in HD. That's not enough to justify the monthly cost since there is only a small subset of those movies that I will want to watch.
Re: (Score:3)
Well... you do know MythTV can run in Windows now right? I'm betting with a little work NetFlix could indeed be up and running....
http://www.mythtv.org/wiki/MythTV_on_Windows [mythtv.org]
Separate servers (Score:2)
Instead of tacking function after function onto the same server, I'd encourage you to use several small ones, including the $35 Raspberry Pi. That way one one piece of software starts to go haywire, it doesn't bring the whole shebang down with it.
Also, I'd go for fewer, larger disks. As long as you do backups it's not more risky, and it's a lot more practical. HDs reach 4TB these days, so you're talking 2 HD for your existing data, a 3rd one more more capacity, and double that for backups.
Re: (Score:2)
Sure! You just have to get the right hardware and after tinkering with ESX for awhile I gave up on it. None of my onboard NICs were supported! No thanks, wake me when it's running on broader hardware support.
you may want to have raid 5 or 6 (Score:3)
10 disks in a raid 0 type setup is a big risk. Also lot's of pci cards eat up PCI bus io and maybe even the same io used for the network also your board like only has 100M e-net. Now if your system has a pci-e slot then a 10 port non raid card + software raid may work and is cheap then a raid card. But you may also want to get a newer MB + cpu most new amd and intel boards max out at 8 sata ports. 8 ports may work out ok or you can get a new board with a dual core cpu + a raid or non raid card also a new MB will get you GIG-E + pci-e IO.
Re:you may want to have raid 5 or 6 (Score:5, Insightful)
Yup, start again.
Pick a board with plenty of SATA ports, put a modest amount of RAM and CPU in it. Make sure it's got PCI-E slots (what hasn't these days) and go from there.
Use bigger drives than you are currently, it's a bad time to buy drives so wait if you can but just build a new box from scratch and save yourself the headache of trying to migrate drives or retain data while upgrading drives one at a time in an existing array.
New machine, 3TB drives x as many as you want (6 would about double your capacity), add a 4 port PCI-E SATA card if you need it and rsync all the data across, job done
I was saying maybe a raid card but non raid card (Score:2)
useing software raid is ok most boards have about 6 ports so if you want like 10 then maybe a x4 or better pci-e card may be needed.
Re:I was saying maybe a raid card but non raid car (Score:5, Interesting)
useing software raid is ok most boards have about 6 ports so if you want like 10 then maybe a x4 or better pci-e card may be needed.
Or, get an actual server board (this is gonna be a server, right?), like this one [supermicro.com]. That's six SATA ports and 8 SAS ports. If you flash the SAS ROM to the "no-RAID" version, the controller is recognized natively by Linux. In addition, you get lots of PCIe connectivity, a pair of Gigabit Ethernet ports, and IPMI (allowing remote power cycle).
Then, find a full-tower case with lots of 5-1/4" drive bays, and add hot swap bays [newegg.com]. There are smaller versions, as well...just budget what you need for drives.
I use the motherboard I referenced along with an add-on 8-port SATA card (anything supported by Linux would be fine) and two of the drive bays for ten 2TB drives in RAID-10. I boot Fedora off a pair of SSDs in RAID-1 and also have four 2-1/2" 750GB drives in RAID-10. The 10TB array serves iSCSI over 10Gbit Ethernet to ESX systems that hold all my VMs, with the 1.5TB array as local and NFS storage. There's still PCIe slots available if you need more controller cards.
With this setup, the VMs are how everything is accessed, so you can pick whatever OS you want to face client machines.
Re: (Score:2)
You do know, any ROM flash is just software raid right? Even SAS controllers don't have hardware RAID unless you buy a real raid card for $$$. Real raid cards have write back memory and a BBU.
That being the case you're better off using something like linux software RAID so if your SAS controller happens to die, you can still recover your array by simply plugging it into another machine
Re: (Score:2, Insightful)
Raid seem silly in a home setup. People see it as a backup solution when it's not, and I doubt a few extra tenths of a percent of uptime is really significant at home. Just have one or two big disks for all your media and stuff, then buy the equivalent to use for regular rsync backups. You can get a third to take "off site" occasionally (e.g. your sister's house, a friends house, a drawer at work) if you want to be really careful - just rotate the backup disks every so often.
with 8TB data you are looking at 3 3 TB HDD's (Score:3)
now just to fit the data on one set is like 3 HDD's and then raid 5 or 6 starts to look like a good idea.
Re: (Score:3)
Re: (Score:2)
Raid seem silly in a home setup.
Once you start to use more than one hard drive worth of space, RAID is pretty much required unless you really like to spend days restoring data when a hard drive fails.
Re: (Score:2)
Failure rate of disk drives in consumer environments is about 3%/year. With N drives you'll see N*3% failures per year. You do the math.
HP Microserver (Score:5, Informative)
For a slightly more sane solution than rackmounting at home, consider the HP microserver.
Very low power (12W CPU), small, quiet, cheap, server grade, no Windows tax, holds four pluggable 3.5" drives plus optical (which some people swap for a 5th HDD for RAID5.)
http://blog.thestateofme.com/2011/05/14/review-hp-microserver/ [thestateofme.com]
http://www.silentpcreview.com/HP_Proliant_MicroServer [silentpcreview.com]
http://h10010.www1.hp.com/wwpc/us/en/sm/WF05a/15351-15351-4237916-4237918-4237917-4248009.html [hp.com]
http://forums.overclockers.com.au/showthread.php?t=905262 [overclockers.com.au]
If 8TB is full, you need to stop the obsessive collection of warez/pr0n/torrentz you are never likely to watch again.
Re:HP Microserver (Score:5, Interesting)
If 8TB is full, you need to stop the obsessive collection of warez/pr0n/torrentz you are never likely to watch again.
As opposed to the obsessive collecting of physical media that can be scratched and takes up 10x+ the space?
My family since the 80s has amassed literally 10k CD/DVDs as well as almost 100 laserdisc titles. Not to mention a buttload of VHS tapes that we offloaded years ago.
It has all been converted to digital storage. Since it is on multiple RAID 5 devices and I run a cron job that checks the MD5 sigs against a database I know that it is in good condition.
Of course this requires constant rollover of the data from hard drive to hard drive. Half the drives have failed over the years and it has moved between multiple NAS systems. We still have all the data.
In addition to that, we have over 100k family photos collected from all of our relatives scanned and tagged as well.
Our collection is nearing 20 TB. With the low cost of drives we have backups in lead lined containers in safety deposit boxes at two banks. We swap them out every year or so adding to it. I am really looking forward to long term archival storage that is write once and designed to last 100 years plus. I'll pay for that.
Now I know you may be thinking obsession, but we *paid* for it. Paying twice for music or movies is just plain insane and we never fell for the HD/Bluesuck shit they were shoving down our throats. Well my parents did, but Spiderman solved that problem the first time it could not be played because the encryption changed. Since then they are back on DVD only and we are waiting for a HD storage method that does not involve constant Big Brother monitoring and DRM in our houses.
Then there is the most obvious benefit of all. You only have to rip the music or movie one time. Been years since we bought an actual CD, but you get my point.
The convenience of having all of your media at your fingertips without touching physical media is pretty damn nice.
Guess how much storage space you need for thousands of DVD/CDs when they are packed into spindles and put into storage? A heck of lot less than you would expect. Fits in a closet.
Re: (Score:2)
Umm, why not strip the DRM from the BD and store those too? the picture really is much better even if you re-encode them for smaller storage. I have a post above about this but yeah it's doable. I also have about the same amount of storgae as you but it's not backed up other than the unRAID software's protections. It's simply too expensive to have a full dupe for me :(
Re: (Score:3)
BD is never an option for multiple reasons.
1) Even if doable, which keeps changing, a true BD backup takes up a *lot* more storage. I have seen torrents that are 25-30GB for a BD rip. Why even download it or rip it?
2) To rip it, I would have to purchase it. I would never do so because that would be voting with my wallet in the wrong direction.
3) Re-encoding has variable quality depending on the algorithm and the parameters. Piracy groups do it better, so I would never do it myself. In the end I would
Re: (Score:2)
Perhaps so. There really is no choice unless you want to eventually become a victim of Big Content. The GP strongly implied piracy as the only cause of such massive storage requirements. It's not at all.
It has to be stored *somewhere* and since managing physical media is a pain in the ass and *every* single use causes damage, another solution must be found. Be it ever so slight, there is damage with physical media. It makes no sense if we own the legal entitlements to a copyrighted work to keep it stor
Re: (Score:3)
You DO realize that normal SD DVD had DRM too right? It had the SAME sorts of restrictions that BD has now and so did HD-DVD! The only difference was that the DRM was easier to defeat because they couldn't swap keys as they were compromised. You're not doing anything different now with DVD than you would be with BD. Strip the DRM from BD, re-encode if you want it smaller, and then watch that much nicer picture. It's harder, it's more time consuming, and you don't end up with menus and advertisements. You ca
Re: (Score:2)
You DO realize that normal SD DVD had DRM too right? It had the SAME sorts of restrictions that BD has now and so did HD-DVD! The only difference was that the DRM was easier to defeat because they couldn't swap keys as they were compromised
I covered that in my other response to you.
I'd also argue that a laser beam does no damage to a DVD or BD. The damage these gets is from oxygen leaking into the media and oxidizing the metal. This is real, it's happened to some of my CD. Scratches happen but so long as they aren't on the label side they can be recovered with a buffer - a car wax like Nu-Finish with distillates works wonders too.
I never said laser beams did damage. The act of putting the media in the player does damage, even if ever so slightly. Your point about oxidation is also understood. Which is why even the best storage methods, short of a vacuum, will only marginally prolong storage life. Ultimately, shifting and validating the data across different mediums is the only long term storage solution.
BTW - where did the GP imply piracy at ALL in his posting?
I thought the reference to warez and torrents was more than enough. Torrents can be
No Garage (Score:3, Insightful)
You need a NAS. (Score:2)
Since you're not complaining about processing power or ram, you're in the market for a NAS. There are several good brands. I personally use Synology. It's a bit pricy but you get what you pay for. Personally I'd just add a few external USB drives until the prices fall (they're pretty outrageous now). When prices fall, get a nice 5 bay and stock it with 3TB drives that will give you ~12 TB in raid 5 and ~9 TB in raid 6 (recommended unless you like living on the edge).
You'll probably find the Synology c
Hard drive math and hobbyist heuristics (Score:5, Informative)
Ballpark figures, this isn't exact, redo it with your preferred constants, I'm just trying to explain my reasoning against huge enclosures with > 10 drives,
Standard drive idle usage (W) ~ 10W [1]
Low-power (green) drive idle usage (W) ~ 5W [1]
Cost of power ~0.20 $/KWH
Cost of an older drive per year = $17
Cost of a green drive per year = $8.50
Replacing 6x500GB older drives with one 3TB green power savings = $95/yr
So think about that for a sec. At $150[2] for a 3TB drive, you cover the price in power savings in 18 months. That's assuming that there is zero fixed-cost per drive. At the point where you are talking about adding SATA controllers or fancy multi-bay enclosures or, worse, external enclosures with their own PSUs (and fans!), the turnaround-point for older drives is far sooner.
I'm a hobbyist, I understand that it's really cool to make do with older hardware and feel like you aren't letting anything to go waste but sometimes using old hardware instead of buying new is penny-wise and pound-foolish. Spending money on increasing how many hard drives you can accommodate instead of just buying newer high-capacity lower-wattage drives is absolutely batty; especially when you get into the price for anything remotely good in the RAID dept.
My advice, move everything to the largest capacity drives that are reasonably priced (after the flood damage is sorted). Replace the drives when you can do between 4:1 and 6:1 replacement -- should be every 3-4 years. Live happily, quietly and simpler. Save money on power transparently.
[1] http://hothardware.com/Reviews/Western-Digital-2TB-Caviar-Green-Power-Hard-Drive/ [hothardware.com]
[2] I bought some Hitachi 3TBs before the Thailand floods at $130 on Newegg. Of course you would be silly as heck to buy hard drives now for your hobby storage project before they at least fall back to pre-flood level.
[3] http://www.newegg.com/Product/Product.aspx?Item=N82E16817182221 [newegg.com]
[4] Older drives need not go to waste, they can become offline storage with a simple USB dock[3] -- make a backup, throw it in an anti-static bag, leave it at your relative's house when you visit!
[5] http://en.wikipedia.org/wiki/File:Hard_drive_capacity_over_time.svg [wikipedia.org]
Was going to suggest... (Score:2)
Re: (Score:2)
I swear I've changed that setting before...
Dammit.
A better chassis (Score:2)
Dedicated unRAID server box (Score:2)
Thoughts from my home storage server experience. (Score:5, Informative)
I wrote about the latest storage server I built back in 2008, and a lot of my thoughts at the time are written up in http://www.tummy.com/Community/Articles/ultimatestorage2008/ [tummy.com]
However, to answer a few of your questions...
External disc enclosures? Avoid them like the plague. My initial experience with the 5 bay eSATA enclosures was pretty good -- sometimes it wouldn't pick up the external drives, but usually I could get it to find them after some tweaking, rebooting, etc... I ended up getting 3 of them, the AMS DS-2350S, which at the time were well reviewed, etc... I have since pulled all 3 of them out of active use and have them just sitting around. I don't know exactly the mode of the failures, but eventually after replacing some with others, I finally put them in internal SATA enclosures, which have been very reliable (I used these Supermicro CSE-M35T-1.
Also note that eSATA connectors don't really hold on that well. If anything, they're not as robust as internal SATA connectors, despite being outside the case where they can get banged around.
If I were to do it over again, I'd probably stick with the case I started with, with 5 internal 3.5" bays, and 3 front 5.25" bays, and put the Supermicro in there. I'd also probably go with fewer big drives rather than more smaller drives like I did previously (even though at the time the drives were free, I had them from another project).
As far as running it in the garage, don't even think it, unless your garage is not where you store your cars. I have some computers that I've run in the garage for the last 9 months, and they are filthy, I've had a lot of fan failures, lots of dust, insects, and random other crap. I put mine in our furnace room, which has enough extra space.
As far as using a server case? Hard to see the payback there unless you have a cabinet. Most server cases are HUGE, heavy, and expensive. A 3U case with 12 drive bays likely costs $500, plus you usually have to deal with special form-factor power supplies, expect to spend another $200 on one of those. I wouldn't do it, and I have a 3U 12-bay Chenbro case just sitting at my office that I could re-purpose.
As far as the file-system, I selected ZFS (via zfs-fuse under Linux) and I've been VERY happy with it. The primary benefit is that it checksums *ALL* data and can recover from some types of corruption or at least alert about corruption if it can't correct it. So, if you are storing photos or home videos that you may not be accessing very often, that's good peace of mind to have, I know in 10 years I won't go to look at some photographs I've taken and find they were silently corrupted. Of course, you could get similar benefits by saving off a database of file checksums and checking and alerting if they are bad. Really the only downside of ZFS that I've seen is that if you need to do a RAID rebuild it is a seek-heavy task rather than just streaming. I have a 8x2TB drive array that I'm currently rebuilding (drive failure, at work), and it's 33% done after 31 hours. A normal RAID-5 array would have rebuilt that in what, 10? The system is idle except for the rebuild.
If you care about the data going into it, make sure you checksum and verify the files regularly.
The 8 port PCI SATA card I got is fantastic, it's a Supermicro with the Marvel chipset and is very well supported (even supported by Nexenta).
Finally, all this data is encrypted, so if someone were to burgle us I only have to worry about them getting the hardware, I don't have to worry about them now having scanned bills and other documents and other personal and private data, etc... This is why I'm running ZFS in Linux, it gave me encryption plus ZFS (not available otherwise in 2008), as well as being an OS I'm very familiar with.
As far as OS, I am personally running CentOS on my system because that means I can install and set it up and then forget about it for quite a few years, except for regularly running "yum update". Debian should be fine, but you will get/have to track upstream changes more frequently.
Comment removed (Score:3)
Re: (Score:2)
Very true indeed. On top of the great set of features already included, it's easy to enhance those things with extra packages. It's just so much less hassle to maintain these things if your main job isn't being a sysadmin. Anybody can basically do it and in a fraction of the time needed for regular server management, too.
Some might argue that these NAS things are too expensive compared to self-built systems. If you consider the time spent on them however, NAS systems beat DIY-systems hands down.
Get a NAS (Score:2)
As many others have already stated, a NAS definitely is the way to go here. There are 2 good manufacturers that accomodate any need and have vibrant communities providing excellent support on top of what the manufacturers themselves offer: QNAP and Synology.
Both of them basically use custom Linux builds on their otherwise very PC-like hardware that is open to all sorts of tweaking and readily allows for adding all sorts of extra software.
encoding (Score:2)
Define the problem a little bit better (Score:4, Informative)
a) i am running out of hd space
b) i feel (=movies dont play without interrupting) that the processor is a little slow
c) i am bored over the christmas holidays
d) i am worried the thing explodes or falls apart
if only a): attach a NAS and make an archiving system
if a) and b) or d): there are enough shopping guides for off-the shelf pc's out there. You priority should be energy consumption, reliablility and space for more hds. Condiser external sata boxes
c) play around with something differnt.
Re: (Score:2, Informative)
@Bonch: Congratulations, you are officially the Grinchy douche on Christmas.
OP: Great questions, I can not wait to see answers. Would love to hear more about what your software solution is for encoding, I've got a bunch of DVDs & Bluerays I would really like to get on the network, but a streamlined rip+encode+publish I have yet to achieve :-/ What are you using on the frontend? I've got various iDevices roaming about, and a Roku that does 720p for the projector, but haven't had much luck with mt-daap
Re: (Score:3)
Not the submitter but.... i do lots of this :)
Atom machines running Ubuntu and XBMC are my front-ends, I use MCE remotes and HDMI for sound and video. 1080P with an ION chipset accelerated using VDPAU is wonderful. Sound can be a hassle to get working with Alsa but it can be done. If this sounds too expensive or complex or whatever - look at the WDLive that was just revamped. $99, supports local storage, wireless, and network shares. XBMC is way nicer though! ;-)
Backend I mentioned this above - unRAID with
Re:Why? (Score:4, Informative)
If you're putting a system into a dirty environment such as a basement then either buy or build an environmental enclosure to put the server in. It's basically a sealed box with large filters for cleaning the airflow through the hardware inside it. Enclosure fans are optional, an overtemp alarm/shutdown system isn't. Replace the filters every six months or so and it should be good.
Re: (Score:3)
Yes... "the age of Facebook".
Every one I know that still had a Yahoo mail account got their accounts hacked and then used as a carrier for virus infested spam.
Yes. "the era of Facebook".
Perhaps I don't want Facebook screwing it up.
People put a lot of faith in "the cloud" but the fact is that it really is not really warranted. Big prominent services get hacked. They sell your information. They do stupid annoying things. Then there's the problem that the network itself still sucks. We simply don't have the in
Re: (Score:3, Interesting)
Re: (Score:2)
Re:Why? (Score:4, Informative)
Agreed. In addition, rack is generally designed with the premise that space is at a premium and that it's well worth cramming as much as possible into the smallest space possible and then compensating for the poor natural airflow with high speed fans everywhere. That's fine if space is that big a premium, but ion a home environment, it's rarely THAT tight. A big roomy tower will run quieter, have less problem with failing fans, and probably will run a bit cooler. It'll also be less of a pain to work on.
Re: (Score:3)
because it saves space
highly standardized
The guy isn't going to be hosting so many servers
Those are the guys three problems.
1) I have the entire "under the basement stairs" space plus two partial wiring closets in my house. Other than hoarding for the sake of hoarding, I can not fill them up. Do not spend money and severely constrain yourself to save space, that you do not need to save.
2) Do not highly standardize. My best purchases have been weird special one off this weekend only deals. Surplus sales, etc. In a corporate environment you Really need to standardize on one specific model of
Re: (Score:2)
Of course, he's retarded for even considering buying a hard drive at this time. He could wait until the price fixing is over in a few months and get 3TB HDDs for $150, instead of
Re: (Score:3, Insightful)
I'd say it would be better for him to get a separate system for the kids. Some time they are going to want to move away from home and move to college/into their own flat.
At that point, they are going to have to buy their own data storage then and transfer everything across.
Re:Why? (Score:4, Insightful)
Power costs. Having a bunch of always on machines starts to add up. That, plus the progress of technology (by the time the kids move away we might be on to using SSDs for everything) means that it's usually better to try and aggregate costs. Plus it can be an interesting hobby!
Re: (Score:2, Informative)
The submitter has very specific tastes and would rather keep conversion to a minimum because of the overhead and liabilities involved.
You see, the submitter is storing guro, bestiality, snuff, squirt/scat, and crush videos; all featuring underage animals, so the overhead must be kept to a minimum. He must make do with wired networking only, adding parts to his existing configuration as needed so his horrific pornography may be available to him immediately, whenever he wants to jack off.
I've seen his works
Re: (Score:2)
Re: (Score:2)
Re: (Score:3, Interesting)
Another option with FreeBSD is ZFS, which is pretty sexy.
Re: (Score:2)
Given that OpenSolaris is already deployed all over the place, it's the least adventurous of the options.
I'm using snv_130 for my personal media server and it works fine for me.
6 drives: 2 on ICH, 4 on sil3124. Pentium Dual-Core Mobile 1.73 on a Mini-ITX board with dual-gigE.
If people are unhappy about the lack of a support community, there'a Illumos/OpenIndiana.
Or just download the free Solaris 11 Express install DVD from Oracle's site for the latest and greatest.
I'm almost tempted to make a 2nd server ent
Re: (Score:3)
btrfs has been very stable for me since 3.1 - with RAID10 at least. Sometimes it feels a little slower than ext4 but overall it works well (4x 3TB in RAID10, 3.1TB in use, 1.3M files right now.)
Re: (Score:2)
XBMC however on the old hardware it's not going to do high bitrate HD. Better to go ION hardware or an aTV with XBMC, it's been ported to Linux, Windows, and OSX - Plex is based on it.
This!! ^^^^ (Score:2)
Except with unRAID, sorry I don't need enterprise RAID at home but the hardware you chose is solid and the 5n1 adapters rock! I can stream 1080P video with DTS on a 100meg NIC, I sure don't need RAID 5 or 6 spinning my drives all day, let them spin down :)
Re: (Score:2)
unRAID is NOT OpenSourced. It does run on Linux but it has closed source portions and it is locked to a USB key that you register. If the key dies Tom is damned good about replacing the license for you - been there done that. Tom does comply with the GPL, any code he modifies that's GPL is included with unRAID. His closed source stuff he keeps to himself and for the service he provides I'm okay with that. I run two of his servers and he's been very helpful over the many years. Slow to update software, bug f
Re: (Score:2)
Each disk node should have dedicated 1Gbit connection without switches between the master server and the disk node.
If direct links are not possible and switches are needed, those should be manageable and configured with iSCSI in its own vlan.
Also important, unbind all other services from the NIC dedicated to iSCSI.
The disk node can be connected to the local network using