BUILD Advice and comment welcome on my proposed FreeNAS build

[cyberjock]

No, but if it fits it would be a close fit.

 

[trionic]

By the end of this week I should have all the hardware for this build. In the UK, the next weekend is a long public holiday and so I should have time to build both the ZFS box and a general-purpose server (both 4U rack mounted) plus finish that kitchen refit project that's been dragging on for a while.

I have also spent whole days messing around with FreeNAS running in VirtualBox, testing configurations, resilvering, damaging disks, yanking out the (virtual) plug and anything else I can think of to trip up ZFS. That process has given me more confidence in using FreeNAS and ZFS. My advice to anyone building a ZFS server is to do the same.

FWIW the Supermicro motherboard's on-board SAS controller has a total capacity of 8 devices; 1 device per port. I have ordered a single M1015 for connection to the RM424 Pro's SAS backplane.

I have spent many hours reading forum posts and articles on ZFS, in an attempt to reduce the chances of pool loss. To that end, two Seasonic SS-600H2U PSUs in redundant configuration will join the APC Smart-UPS SC UPS. The Seasonics will replace the single Corsair 1200W ATX PSU.

With time I may build another 4U enclosure to handle pool replication. However, that'll be after my bank balance has recovered and I have had time to properly research the subject.

 

[trionic]

Now that the hardware's been ordered I am thinking more about which RAIDZ configuration to use. I had previously settled on RAIDZ2 with initially two 6 x 3TB VDEVs in this 24-bay enclosure. I already have fifteen or so 3TB drives but since the per TB cost of 4TB drives is now nearly the same as 3TB drives, the last one or two 6-disk VDEVs may be populated with 4TB drives instead of 3TB drives. So the configuration will end up being either three 6x3TB VDEVs plus one 6x4TB VDEV or two 6x3TB VDEVs plus two 6x4TB VDEVs.

I will keep tested cold spare drives on the shelf.

I *think* that Z2 will be sufficient for use with 3TB drives and that scheme seems to be what most folk here use. What about 4TB VDEVs? The probability is low of a further two drives failing during a resilver but given that VDEV loss causes pool loss I am now wondering whether Z2 is enough.

I'd rather fill the enclosure with working drives and so a 7-disk Z3 VDEV will occupy only 21 of the 24 drive bays. An 11-disk Z3 VDEV would seem to increase risk of further drive failures during a resilver.

Anyone using Z3? Am I now being too paranoid? I am aware that increasing paranoia would be better served with mirroring/replication and that is something I am planning for the future.

 

[cyberjock]

I'd do RAIDZ2 with 6 disks. RAIDZ3 is a bit paranoid, especially if you do a proper setup of FreeNAS. ;)

 

[pbucher]

FWIW the Supermicro motherboard's on-board SAS controller has a total capacity of 8 devices; 1 device per port. I have ordered a single M1015 for connection to the RM424 Pro's SAS backplane.

Hmm I think you read something wrong, the on-board SAS controller can control 256 devices via it's 8 ports. You just need the usual SAS expander to have more then 8 devices. The M1015 is the same thing as the one on the motherboard, just an older version. As someone mentioned previously you just need a reverse SAS cable to bring the individual ports back on the motherboard back to a 8087 multi-lane connector. I can't tell from the X-case website how many SAS ports it requires(talks about 6 and then having 2 sets of 6 each), the standard LSI cards supply 8 ports(some only 4, but the ones your talking about here both are 8 port versions).

 

[trionic]

That's really interesting.

I struggled to find reliable information about the X9SRH-7F's on-board SAS controller. The manual doesn't help, couldn't find anything on forums and Scan (Supermicro's UK reseller) didn't know! In the end I called Supermicro UK and they told me that each invididual mobo port supports just one device. The ports are just SATA-form ports.

I remained less than 100% convinced but without concrete evidence to the contrary I had to accept the answer that I was given. If only eight devices are supported then I could not see why the expense of an on-board SAS controller was justified instead of simply adding more SATA ports (but the mobo has about eight or so already). I kinda got fed up with thinking about it and chasing an answer, gave up and went the well trodden route with an M1015.

On the XCase website, the photos of the RM424 Pro showing six SAS ports are incorrect. Upon delivery I opened the case and found that the SAS backplane has two outputs and one input. I'll take some photos tomorrow if I get a chance.

I still have time to send back the M1015 if I know for certain that (a) the X9SRH-7F can do what you suggest (and you seem very clear on that) and (b) that the on-board SAS controller can be flashed to IT mode.

 

[pbucher]

I still have time to send back the M1015 if I know for certain that (a) the X9SRH-7F can do what you suggest (and you seem very clear on that) and (b) that the on-board SAS controller can be flashed to IT mode.

Here how to do (b): link

 

[panz]

RM 424 comes in 2 versions: the n. 6 SFF-8087 backplane model (the one I have) and the "Expander backplane" model. I purchased the first version because I didn't want to be tied to a backplane with a built-in expander and it's complexity. I have 1 M1015 and another for redundancy.

 

[trionic]

I have the SAS expander backplane model.

I received the rest of the parts yesterday: UPS, motherboard, memory, redundant PSUs and some other bits. I didn't order any SAS reverse breakout cables and so I will not be able to play around with SAS configurations. No matter though because for a couple of weeks at least the server will be undergoing smoke tests, giving me time to get the necessary cables. I will flash the on-board SAS controller to IT mode though and get that job out of the way.

I also forgot to order more 3TB WD Red hard disk drives. However, I have just written out a plan for VDEV creation, migration of data and reuse of existing disks. Using 3TB hard disks this server will have a RAW capacity of 72TB, available capacity of 48TB available but an 80% capacity of 38.4TB. Subtracting from 38.4TB the amount of data that I already have to store, I am left with 9.72TB free space. Take away from that regular laptop/phone/tablet backups and I'll quickly find that I have run out of space (mental). So, 3TB disks may prove to be a false economy, especially given that the 3TB/4TB £/TB is roughly the same. I may have to bite the bullet and buy 4TB drives instead to have at least two VDEVs with an 80% available capacity of 12.8TB.

The alternative would be finding myself having to build a JBOD enclosure at great expense, much greater in fact than equipping the current enclosure with 4TB disks.

 

[trionic]

I think I may have screwed up with the power supplies. I bought two Seasonic SS-600H2U PSUs without giving any thought as to how configure them for redundancy and infact without understanding enough about the subject. I just have two individual PSUs; essentially one would be a cold spare which is no use at all. Urgh.

 

[pbucher]

Using 3TB hard disks this server will have a RAW capacity of 72TB, available capacity of 48TB available but an 80% capacity of 38.4TB. Subtracting from 38.4TB the amount of data that I already have to store, I am left with 9.72TB free space. Take away from that regular laptop/phone/tablet backups and I'll quickly find that I have run out of space (mental). So, 3TB disks may prove to be a false economy, especially given that the 3TB/4TB £/TB is roughly the same. I may have to bite the bullet and buy 4TB drives instead to have at least two VDEVs with an 80% available capacity of 12.8TB.

Just a heads up ZFS performance starts to go down when your vdevs get full. It's a CopyOnWrite file system so all writes go to your empty space and when empty space gets to be fragmented all over the vdev and you write a big chuck of data it's going to be any ugly write. That said it depends upon your individual environment to say at what point your write performance will tank. At some point your free space will become fragmented regardless, but the degree and size of the fragments will depend upon the history of size of what's getting written and rewritten so there is no magic # of % of free space you should keep.

Anyways I'd highly suggest since this is a modern setup go with the lz4 compression on your pool and hope it can save you some space. I believe it's the default now for new pool anyways, I've seen some really nice results with it so far.

 

[trionic]

I recall from messing around with FreeNAS in a VM that lz4 is indeed the default compression. This server will be used for digital media that is already compressed in some form (generally H.264 or FLAC). How much additonal compression can I expect with lz4?

The majority of files stored will be multi-gigabyte sized. Is there any way of monitoring ZFS pool fragmentation? I assume that there aren't any ZFS defrag tools out there

I have just worked out another migration plan, this time based on 4TB disks throughout. It's a no brainer - 4TB it must be the way to go. However, I absolutely cannot right now take the hit of 24 4TB hard disks (£3120 inc tax!) and so some half-way point is needed. I may be able to afford to buy 12 4TB drives now.

One pool with two 6x4TB VDEVs will be *almost* enough space for migrating existing data. The plan below is intended to resilvering 12 hard disks and the (perceived) risk from doing so.

So, pool A contains two 6x4TB VDEVs. Pool B and pool C are each made from 6x3TB VDEVs. When I can afford to buy a further six 4TB disks, I remove pool C from the enclosure and create a third 6x4TB VDEV and add that to pool A. I copy all data from pool B to the new expanded pool A. I remove the 6x3TB disks that composed pool C and put back the 6x3TB disks that composed pool B (we now have three 6x4TB VDEVs in one pool and one 6x3TB VDEV). When I can afford another six 4TB disks I do the same, but this time I'll have to copy pool C to a different machine (the other server I am building which has 10x 3.5" hotswap bays) before replacing pool C with the fourth and final 6x4TB VDEV to be added to pool A. Phew!

Does that sound crazy/stoopid/sensible/<insert-adjective-here>?

 

[Ericloewe]

I recall from messing around with FreeNAS in a VM that lz4 is indeed the default compression. This server will be used for digital media that is already compressed in some form (generally H.264 or FLAC). How much additonal compression can I expect with lz4?

While I haven't tried it in FreeNAS, don't count on much compression for media.

 

[trionic]

Quick update - other stuff has meant I've not had the time yet to build this thing but hopefully this weekend I will get stuck in. What I have done is to build another server, using existing parts and to begin finding out more about the burn-in tests that are required.

In the past I have used the SystemRescue CD distro to recover data from broken hard disks. Usefully, SystemRescue includes iozone, used for stress testing hard disks.

I have just ordered a Zippy Emacs 1+1 620W C2W-5620V redundant PSU plus two StarTech 50cm SFF-8087 to 4x SATA Internal Mini SAS to SATA reverse cables.

When I get building I'll update this thread with photos.

If the enclosure has expanders, you just connect it "normally" to the motherboard (to four connectors instead of a single big one, with a big one on the enclosure end). If not, you will need additional controllers or provide your own expander. Since yours provides an expander, here's what your setup would look like:

This is a formal warning that bad ASCII Art lies ahead.
...

Eric, I forgot to say thank you for the (actually awesome) ASCII art :) and the explanation for the components that I need.

I confess that I didn't really get the explanation when I first read it but now I know a little more about SAS I better understand what I need to do.

http://lime-technology.com/forum/index.php?topic=7003.0
While the 4 ports on the board are part of a SAS controller, they are SATA connectors, i.e. they require a crossover (reverse) cable to go to 8087.

That helped a lot too.

 

[trionic]

Today I started building this ZFS server. Lots of information and photos to come.

Question though: from the Zippy EMACs PSU I have three unidentified cables:

• Red and black cables to a 2-pin connector (like a small fan header)
• Yellow and black cable also to a 2-pin connector (like a small fan header)
• Purple, black and white to a black 5-pin header

Only the 5-pin purple/black/white cable has a matching socket on the motherboard labelled JP12C1 "power supply SMBus 12C header".

I cannot see any 2-pin sockets on the motherboard for the yellow/black or red/black cables so presumably they're not used.

SM motherboard manual not helpful. PSU manual absent.

Any ideas?

 

[cyberjock]

No clue myself.. never heard of the brand(run away!?)

 

[pbucher]

You could put your 6x3TB vdevs into

I cannot see any 2-pin sockets on the motherboard for the yellow/black or red/black cables so presumably they're not used.

That would be my guess. I'd guess they are for hardwired fans, being that they have only 2 wires.

 

[c32767a]

No clue myself.. never heard of the brand(run away!?)

Zippy is not uncommon in OEM cases. They don't really have a huge presence in the direct to end-user market though.
Good hardware, just not well documented.

 

[trionic]

And a bit noisy too. Those four small (20mm?) fans have an irritating high-pitched component. I wonder if an FSP supply would be quieter.

Been figuring out the SAS backplane/on-board LSI setup. There are actually two SAS backplanes in the enclosure. I bought two reverse breakout cables; one mini-SAS plug on one end and four SATA tails on the other.

For a breakout cable attached to one backplane, any of those tails connected to any on-board LSI SATA socket connects all drives on that backplane. The conclusion is that two 1x mini-SAS to 1x SATA tail would be enough to connect all 24 drives.

Equally if I connect all eight SATA tails to all eight on-board LSI SATA sockets then all drives are also visible.

Can anyone unravel that for me?

 

[Ericloewe]

And a bit noisy too. Those four small (20mm?) fans have an irritating high-pitched component. I wonder if an FSP supply would be quieter.

Been figuring out the SAS backplane/on-board LSI setup. There are actually two SAS backplanes in the enclosure. I bought two reverse breakout cables; one mini-SAS plug on one end and four SATA tails on the other.

For a breakout cable attached to one backplane, any of those tails connected to any on-board LSI SATA socket connects all drives on that backplane. The conclusion is that two 1x mini-SAS to 1x SATA tail would be enough to connect all 24 drives.

Equally if I connect all eight SATA tails to all eight on-board LSI SATA sockets then all drives are also visible.

Can anyone unravel that for me?

Easy, the expander(s) will use whatever bandwidth it has available. Yes, the stuff that is non-trivial in ethernet is automagic in SAS.

There are plenty of complicated setups that are possible, especially with multiple expanders and SAS controllers, be sure to read the manual.