I need to a an external file storage unit primarily to store video footage. I’ve used external USB HDDs for years but I’m considering a unit that could give some redundancy and expansion. I want a pretty good value for the money and I’m thinking something like 4TB to begin with. There are a ton of devices out there. Micro center has quite a few.
Need:
Start with empty device plus one drive (4TB? ) I realize there is no redundancy there.
Want to be able to add additional drives with little fuss as far as configuration goes, so a companion app that goes with the box would be great - to take care of configuration and such.
Would be nice if it could automatically configure the files transparently when a new drive is added for more storage.
Does not have to be hot swappable or do the reconfiguration in real time / transparent to the OS. This is just for backup.
Am I right in the assumptions below? I’m guessing that if one drive is a parity drive then . . .
1 drive - Capacity = 1, no redundancy
2 drives - Capacity = 1, mirrored, single drive failure tolerant
N drives - Capacity = (N-1), single drive failure tolerant
Since this is purely for backup storage, this makes options a lot more flexible. Please do remember, RAID is not a backup; don’t rely entirely on this unit.
If you’re technically inclined I like to recommend FreeNAS. The ZFS system allows you to simply yank drives one by one, replacing them with larger drives and rebuilding their share; once you’ve done this for each spot expand for the entire volume to become available.
Otherwise, something like a Drobo or QNAP in terms of performance, though if it will be exposed outside the local network QNAP has had some issues (in reality most consumer and pro-sumer lever NAS do).
With traditional RAID setups you want to try and get the whole array first, rather than add drives later, as it requires rebuilding the entire array. If you don’t mine the effort, and have enough swap storage not on the device handy, this works fine anyway.
Right now 6TB is probably best bang for the buck (at least last I checked).
As for RAID levels, that’s going to be base a bit off of what you find in terms of budget. Since you don’t have a workflow requiring editing off the drive, then anything that provides fault tolerance works but here are the caveats for reference:
in any raid involving more than 2 drives, it’s not simply N-1 capacity (though it works out like that for some groupings). RAID 5 has a space efficiency of 1-(1/n) and RAID 6 has an efficiency of (1-2/n)
RAID 5 can still only support a single drive failure, like RAID 1, but if you can budget 3 slightly smaller drives instead of 2 larger ones this may work better. However, it has what’s called a “write hole” in that if it’s being written to and the controller suddenly loses power, there will be corruption since the Parity is not an atomic transaction. For purely backup systems this may or may not be an issue depending how often the drive is written to, and whether or not you can check the last written files after a backup operation.
RAID 6 is like RAID 5, but can tolerate 2 drive failures. It requires at least 4 drives
RAID 1+0 (also known as RAID 10) is a combination of RAID 1 and RAID 0. It will have a 50% space efficiency, but usually gives the best performance whilst having redundancy. It can support at least one drive failure. Depending on which drive fails next, it may or may not survive a second drive failure.
For a purely backup system, I like the KISS method; I’d stick with RAID 1 unless an economical thing opens up for a higher drive count, or you choose to go the ZFS route. There are a lot of fun things to do with FreeNAS, such as running Emby/Plex media servers natively in jails (which would allow streaming of the videos etc with nice interfaces. Emby is my go-to), support for various document sync systems, etc on top of being an overall well rounded F/OSS project.
Thanks so much! That’s some seriously good info. I’ll research FreeNAS.
I took a quick look and will delve into when I have a moment. To set up your own system with FreeNAS / ZFS you would need what hardware? It says it’s an OS to run on any hardware. So you would either buy a box from them / Amazon or you would put a bunch of drives in an old PC and that would be your file server?
Yes, FreeNAS runs on almost anything. The unit you linked to is made by the developers of FreeNAS, who also run commercial hardware / versions. I haven’t used their hardware personally, but I’ve heard good things.
When I need a box, it’s usually going in a rackmount so I build it from a local supplier (The Server Store). If you have a home rack it’d get something from there, either a Dell r7xx series or a SuperMicro with the SuperQuiet power supplies so you have a manageable system.
If not in a rack, it’s really a matter of preference. If you build your own here are a few pieces of advice:
try to avoid mixing SATA controllers if you can. Not all motherboards use the same controller for every SATA port. Some may be the AMD/Intel native, others may be a Marvel expansion controller.
if you use something like an LSI card, make sure it is set to IT mode since FreeNAS handles all the RAID info and not the local raid card
rule of thumb for ZFS & RAM is to have enough RAM to cache at least 8 seconds of continuous writes (the network would be your bottleneck, so don’t reference the drive speed for that generally unless you’re bonding multiple 10G ports). This RAM is so when ZFS does some fine, fancy parity or other under the hood data validation you can hold enough for it to sort and organize things
you don’t need a Threadripper, but ZFS does have to calculate parity and the like. A mid-range CPU should do the trick, though you can get away with lower. These units you linked to use an 8 core and likely were chosen because they have optimizations available, though I don’t know enough about that processor to say exactly what.
get a couple of small SSDs in traditional RAID 1 to make your boot drive for FreeNAS; backup the configuration every so often, especially if you choose to use a single boot drive instead.
if using an old PC, check what NIC it has. Some network interfaces hate being used for network storage for one reason or another. Realtek seems to be the worst with this. I like to get a decent Intel NIC; you can save the original NIC for management interfaces and items like Jails if you use them.
What make / model of 6TB do you favor. I did a little reading. Thought WD looked good but that’s apparently only if you never try to invoke the warranty. Nothing but terrible reviews about how they will insist you go back to the original vendor or if they do send you another drive it’s a refurb that will fail.
Generally, I try to stick with HGST. However, if you built a unit and got a SAS controller then the Server Store (in Carrolton) has a trove of zero hour Seagate enterprise 6TB drives for a good price. I note the zero hour here because it’s not retail, but the drives are unused. Those drives are very reliable and give great performance. In the past few years Seagate’s really come up in their high end offerings. However, for anything but those I’d go with HGST personally. I think the seagate enterprise ones I got cost about $50 more each than the HGST consumer ones, but their SAS rather than SATA.
Note, I’ve never once tried to claim a drive warranty; but I’ve had HGST spinning in my datacenters and home round clock in RAID for everything from media editors to hypervisors.