ZFS Pool Recommendations?

[Kris Jacobs]

We just built a new FreeNAS v9.2 server to be used as a backup-to-disk target for various production servers. We like to store 6 full backups and five daily incrementals for each server.

It's a SuperMicro chassis with single Xeon CPU, 32GB of ECC RAM, an LSI HBA, and 12 2.0TB SAS drives. The chassis can support a maximum of 16 drives. We will not use ZFS dedupe, we will use the recommended lz4 compression.

I think RAIDZ2 would be fine for redundancy's sake, hot spares are not necessary.

Tentative plan:

Two RAIDZ2 pools, 6 drives each.

Does this make sense?

Thanks!

Edit:
We want to maximize write performance while maintaining redundancy.

 

[Henry Miller]

There is one potential downside: you don't have room to expand if you find out in the future that you don't have enough storage. Is it too late to change your drives? 6 4tb drives is probably cheaper than 12 2tb drives, or at least not significantly more money. Either way you get the same amount of storage now, but you have those 6 slots free.

Of course either way you still have 4 drive bays free. You can add a mirror in the future if you are low on space, but if you are already using 16tb of space adding 1 more drive probably isn't going to debt your out of space problem.

 

[Kris Jacobs]

Thanks Henry.

Space isn't a concern really - we plan to build at least one more of these.

 

[jgreco]

The SAS is basically a waste, unless you happened to have the drive set laying around. If you build another, Henry's suggestion is good. The WD RED drives appear to be one of the best currently available options for this sort of project.

Do not make the mistake of saying "oh we need 15TB, let's see, two vdevs of 6 * 2TB drives in RAIDZ2 is 8TB each or 16TB total usable. Ah! I need 2TB drives."

ZFS can write faster when it has large amounts of free space available, so one of the most-often-missed ways to make sure your ZFS pool stays fast is to make sure there's lots of free space. A ZFS write transaction group can be very large, like gigabytes large, and the less seeking around that the drives have to do in order to blat that out to disk, the better it is for write performance.

So in one of those ironic sorts of things, a 5400RPM 4TB WD RED SATA drive that's only 45% full is likely to be able to write faster than a 15K RPM 2TB SAS drive that's 90% full. ZFS can be counterintuitive in that way.

 

[Kris Jacobs]

The SAS is basically a waste, unless you happened to have the drive set laying around. If you build another, Henry's suggestion is good. The WD RED drives appear to be one of the best currently available options for this sort of project.

Do not make the mistake of saying "oh we need 15TB, let's see, two vdevs of 6 * 2TB drives in RAIDZ2 is 8TB each or 16TB total usable. Ah! I need 2TB drives."

ZFS can write faster when it has large amounts of free space available, so one of the most-often-missed ways to make sure your ZFS pool stays fast is to make sure there's lots of free space. A ZFS write transaction group can be very large, like gigabytes large, and the less seeking around that the drives have to do in order to blat that out to disk, the better it is for write performance.

So in one of those ironic sorts of things, a 5400RPM 4TB WD RED SATA drive that's only 45% full is likely to be able to write faster than a 15K RPM 2TB SAS drive that's 90% full. ZFS can be counterintuitive in that way.

Thanks for the advice J! I appreciate the comments about free space.

We went with SAS for backplanes, expanders, etc. and ease of cabling that comes with SAS instead of SATA.
With an LSI HBA that has 4 sockets, we can connect a total of 16 SATA drives. That single HBA can control far more than 16 drives if you use SAS & expanders properly.
Because we are using SAS, we can control 16 drives on a single HBA socket - a single cable from the HBA to the SAS backplane.
If we want to add another shelf full of disk, we simply mount a bracket in the back of the server, plug it into the second socket on the HBA, then bingo - a second shelf full of 16 more disks.

 

[jgreco]

I have some bad news for you, I'm sorry to say ... what you say, it works with SATA too.

Don't worry, don't feel bad, this stuff is kind of esoteric and nobody really wants to advertise that you can do the cheaper thing. Because they want to sell you stuff. Me, I'm a natural cheapskate, so within a range of options I will evaluate and typically pick the thing that gets me what I want (in the case of a hard drive, usually space) at the best price point.

So let's walk through a system and consider stuff. Let's take a chassis, like Supermicro's SC846-BE26-R920B, which has support for 24 SAS drives, and includes two SAS expanders. Connection to the host system is accomplished via SFF8087, of which there are two for upstream and four for downstream connections. Obviously the secondary expander is only usable/useful if you happen to have dual-ported SAS drives. We will ignore it and treat the unit as though it were the -BE16, the model without the secondary expander.

Now we stick into it an IBM M1015, which is basically just a favored LSI HBA around here that works well with FreeNAS. The M1015, as with most of the 6Gbps LSI HBA's, has two SFF8087's on it. A single SFF8087 cable to connect the controller to the SAS backplane allows for a 24Gbps link to the primary SAS expander chip on the backplane.

Then comes the gut-puncher. You can plug SATA drives into the SAS expander and it is perfectly happy with that. BUT! The bus bandwidth is shared; you have to make sure you don't get too close to the capacity of the 24Gbps SAS link or you start seeing performance problems.

I find that current generation SATA drives are capable of about 150MBytes/sec or 1.2Gbps, and multiplied by 24 that's almost 30Gbps. That would be uncomfortably full, I think... but on the other hand I've never actually seen ZFS totally saturate a large number of drives doing a scrub. I have some ambivalence about how that will hold out into the future but for today it seems like a 24Gbps SAS link for 24 drives is not a total train wreck.

So basically it ends up looking like this:

Code:

mps0: <LSI SAS2008> port 0x4000-0x40ff mem 0xfd3fc000-0xfd3fffff,0xfd380000-0xfd3bffff irq 18 at device 0.0 on pci3
mps0: Firmware: 15.00.00.00, Driver: 14.00.00.01-fbsd
mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
ses0 at mps0 bus 0 scbus3 target 24 lun 0
ses0: <LSI SAS2X36 0e0b> Fixed Enclosure Services SCSI-5 device
ses0: 600.000MB/s transfers
ses0: Command Queueing enabled
ses0: SCSI-3 ENC Device
da6 at mps0 bus 0 scbus3 target 47 lun 0
da6: <ATA ST4000DM000-1F21 CC51> Fixed Direct Access SCSI-6 device
da6: 600.000MB/s transfers
da6: Command Queueing enabled
da6: 3815447MB (7814037168 512 byte sectors: 255H 63S/T 486401C)
da6: quirks=0x8<4K>
da8 at mps0 bus 0 scbus3 target 49 lun 0
da8: <ATA ST4000DM000-1F21 CC51> Fixed Direct Access SCSI-6 device
da8: 600.000MB/s transfers
da8: Command Queueing enabled
da8: 3815447MB (7814037168 512 byte sectors: 255H 63S/T 486401C)
da8: quirks=0x8<4K>
da7 at mps0 bus 0 scbus3 target 48 lun 0
da7: <ATA ST4000DM000-1F21 CC51> Fixed Direct Access SCSI-6 device
da7: 600.000MB/s transfers
da7: Command Queueing enabled
da7: 3815447MB (7814037168 512 byte sectors: 255H 63S/T 486401C)
da7: quirks=0x8<4K>
da10 at mps0 bus 0 scbus3 target 51 lun 0
da10: <ATA ST4000DM000-1F21 CC51> Fixed Direct Access SCSI-6 device
da10: 600.000MB/s transfers
da10: Command Queueing enabled
da10: 3815447MB (7814037168 512 byte sectors: 255H 63S/T 486401C)
da10: quirks=0x8<4K>
da12 at mps0 bus 0 scbus3 target 53 lun 0
da12: <ATA ST4000DM000-1F21 CC52> Fixed Direct Access SCSI-6 device
da12: 600.000MB/s transfers
da12: Command Queueing enabled
da12: 3815447MB (7814037168 512 byte sectors: 255H 63S/T 486401C)
da12: quirks=0x8<4K>
da13 at mps0 bus 0 scbus3 target 54 lun 0
da13: <ATA ST4000DM000-1F21 CC52> Fixed Direct Access SCSI-6 device
da13: 600.000MB/s transfers
da13: Command Queueing enabled
da13: 3815447MB (7814037168 512 byte sectors: 255H 63S/T 486401C)
da13: quirks=0x8<4K>
da4 at mps0 bus 0 scbus3 target 45 lun 0
da4: <ATA ST4000DM000-1F21 CC52> Fixed Direct Access SCSI-6 device
da4: 600.000MB/s transfers
da4: Command Queueing enabled
da4: 3815447MB (7814037168 512 byte sectors: 255H 63S/T 486401C)
da4: quirks=0x8<4K>
da11 at mps0 bus 0 scbus3 target 52 lun 0
da11: <ATA ST4000DM000-1F21 CC52> Fixed Direct Access SCSI-6 device
da11: 600.000MB/s transfers
da11: Command Queueing enabled
da11: 3815447MB (7814037168 512 byte sectors: 255H 63S/T 486401C)
da11: quirks=0x8<4K>
da14 at mps0 bus 0 scbus3 target 55 lun 0
da14: <ATA ST4000DM000-1F21 CC52> Fixed Direct Access SCSI-6 device
da14: 600.000MB/s transfers
da14: Command Queueing enabled
da14: 3815447MB (7814037168 512 byte sectors: 255H 63S/T 486401C)
da14: quirks=0x8<4K>
da3 at mps0 bus 0 scbus3 target 44 lun 0
da3: <ATA ST4000DM000-1F21 CC52> Fixed Direct Access SCSI-6 device
da3: 600.000MB/s transfers
da3: Command Queueing enabled
da3: 3815447MB (7814037168 512 byte sectors: 255H 63S/T 486401C)
da3: quirks=0x8<4K>
da5 at mps0 bus 0 scbus3 target 46 lun 0
da5: <ATA ST4000DM000-1F21 CC52> Fixed Direct Access SCSI-6 device
da5: 600.000MB/s transfers
da5: Command Queueing enabled
da5: 3815447MB (7814037168 512 byte sectors: 255H 63S/T 486401C)
da5: quirks=0x8<4K>
da9 at mps0 bus 0 scbus3 target 50 lun 0
da9: <ATA ST4000DM000-1F21 CC52> Fixed Direct Access SCSI-6 device
da9: 600.000MB/s transfers
da9: Command Queueing enabled
da9: 3815447MB (7814037168 512 byte sectors: 255H 63S/T 486401C)
da9: quirks=0x8<4K>

which you can see is a dozen SATA hard drives all attached to an LSI 2008. One SFF8087 port is free. Twelve more drive slots are also free in the chassis.

So please understand that I have a bit of an idea what I'm talking about when I say that the SAS drives are a waste in that application. If they were fast 15K drives in an application where you needed lots of pool IOPS, perhaps that'd be worthwhile. But for ZFS in a write-mostly scenario, lower speed drives with substantially higher capacity will get you further.

 

[Kris Jacobs]

I'm not worried, and I don't feel bad - thanks for the information, I appreciate your input!