Confused about that LSI card? Join the crowd ...

[pbucher]

care to explain why would loosing 4 drives will be "the best setup"

Depends if you care about performance or just available storage and how much redundancy you want. I'd recommend reading up on optimal # of disks for the various volume types.

 

[Fermin Rodriguez]

I dont feel very motivated about loosing 4 drives for parity.

 

[cyberjock]

I dont feel very motivated about loosing 4 drives for parity.

You'll consider a change in that motivation when you lose your pool. ;)

 

[Fermin Rodriguez]

Why would i lose my pool if i use say z2 or 3 on all 12 drives. That setup should protect me agaisnt 2 or 3 drives dying at the same tim

 

[cyberjock]

With more disks in a vdev you have a higher chance of having additional disk failures. This is because a larger vdev will take longer to rebuild, and more disks means larger chance for disk failures. Remember, rebuilding a vdev is very disk intensive and any disk that may not be 100% may fail during the rebuild.

Also, wider vdevs have their own performance problems that are related to the extreme wideness.

 

[jgreco]

You have a backplane that uses SFF8087's for host attachment and probably a mainboard with some empty SATA ports, right?

You can probably get away with a single M1015 and then a reverse SATA-to-SFF8087 cable as well. This lets you reduce your power consumption by about 10 watts.

 

[Fermin Rodriguez]

I see. I tho 12 disks were "standard" as almost every storage brand (powervault, emc, equalogic) have at least 12 drives.
I might do 11z3 + one hot spare then

 

[jgreco]

I see. I tho 12 disks were "standard" as almost every storage brand (powervault, emc, equalogic) have at least 12 drives.
I might do 11z3 + one hot spare then

None of those brands are using ZFS. ZFS has a rather generalized design that doesn't forcibly marry it to some of the preconceptions in conventional technologies, which sometimes means it works differently. Conventional RAID technology has some problems "going wide" too. ZFS does offer RAIDZ3, though, which is pretty cool.

The 11 in Z3 plus one spare is a good idea, except be aware that FreeBSD doesn't actually have a way to do "hot" spares. It is more of a warm, ready-to-go spare. It gives you the optimal 8 data drive vdev width. From a failure point of view, it gives you the ability to remotely command the replacement of a failed drive while ALSO maintaining the ability to withstand up to two other failures without data loss, which is totally where I wanted to be, so I actually have a nice array in the 11-Z3-plus-spare configuration. When it can take days to move data around, there's a convenience factor to just having that redundancy.

However, that being said, RAIDZ3 is slow. Read speeds on the 11 drive array vary from 500-900MBytes/sec. Write speeds vary from 100-200. For our uses here, where this is mainly archival "nearline" style use, the speed loss is fine.

I believe someone else suggested a 2 vdev in 6+6 RAIDZ2 configuration. This would be a higher-performance solution, and doesn't include a spare drive. It would recover faster in the event you had a failed drive that needed to be rebuilt, because only the single affected vdev would need to be taking the rebuild hit, and there are fewer devices involved (and less data to analyze and fewer bits that can go wrong). On the other hand, if you do manage to lose three disks in a single vdev, your whole pool's toast.

Tradeoffs, tradeoffs.

 

[pbucher]

Here's a snippet from a ZFS best practices document that sums it up well:

For the number of disks in the storage pool, use the "power of two plus parity" recommendation. This is for storage space efficiency and hitting the "sweet spot" in performance. So, for a RAIDZ-1 VDEV, use three (2+1), five (4+1), or nine (8+1) disks. For a RAIDZ-2 VDEV, use four (2+2), six (4+2), ten (8+2), or eighteen (16+2) disks. For a RAIDZ-3 VDEV, use five (2+3), seven (4+3), eleven (8+3), or nineteen (16+3) disks. For pools larger than this, consider striping across mirrored VDEVs.

So based on that the 11 in a raidz3 + spare is probably the best for a 12 disk setup, though I'd benchmark it against a pool with 2 6 disk raidz2's in it. Also if you have a failure the pair of raidz2s will rebuild much quicker since it has less disks to pull data from for the rebuild and performance wise it might win out also. ZFS works very well with multiple zdev's in a pool, it's best not to add a zdev to a pool with data in it already because it won't strip the existing data across to the new zdev.

 

[jgreco]

it's best not to add a zdev to a pool with data in it already because it won't strip the existing data across to the new zdev.

Strictly speaking that's true, but the old data continues to read at the same speed it used to. For new data, ZFS will heavily prefer the new vdev (because it's empty) which will also mean that you won't see magic new write speeds right away, but as the pool vdev utilization levels out, it'll all work out and balance the writes, and then it is faster. And it does all of that without administrator intervention. Since there isn't really a case where performance gets WORSE, I'm not sure I'd worry about adding new vdevs. Just be aware of how it all works.

 

[pbucher]

Strictly speaking that's true, but the old data continues to read at the same speed it used to.

Agreed.

Off Topic: Play with ESXi 5.5 yet? I've got my develop/test/mirror server running 5.5 with my virtual SAN setup for the last week without problem. Upgrading my vCenter server though was slightly less then smooth(pretty much just do the what it says here kb.vmware.com/kb/2060511).

 

[jgreco]

Yeah, whattaloadacrap.

http://forums.freenas.org/index.php...ce-Comparison-with-VMXNET-and-Intel-EM.15320/

Basically management has been made a disaster. Also managed to kill an iSCSI SAN that had 4.1 traffic on it. More paranoid now (I wasn't before?)

ugh something broke the link, lets try this

 

[titan_rw]

However, that being said, RAIDZ3 is slow. Read speeds on the 11 drive array vary from 500-900MBytes/sec. Write speeds vary from 100-200. For our uses here, where this is mainly archival "nearline" style use, the speed loss is fine.

Really?

I have an 11 disk z3 pool, and can write to it at just over 700 MB/sec.

What kind of CPU is only giving you 100-200?

 

[jgreco]

E5-2697v2. Only four cores, and only 32GB. It is quite possible some of the tuning to increase responsiveness has impacted performance (virtually guaranteed in fact). Still, Z3 is noticeably slower than Z1.

 

[pbucher]

I have an 11 disk z3 pool, and can write to it at just over 700 MB/sec.

What kind of CPU is only giving you 100-200?

Well like everything else ZFS performance is going to be impacted by the design of the entire system, such has CPU, hard disk type & speed, disk controller, etc. It takes a lot more then a fast CPU to push 700 MB/sec write to a 11 disk raidz3 pool.

 

[jgreco]

It takes a lot more then a fast CPU to push 700 MB/sec write to a 11 disk raidz3 pool.

Like someone actually wanting or needing that. Continuing the discussion of bug 1530, my use cases for a filer that goes catatonic for many seconds at a time are much fewer than a filer that remains responsive but has lower performance. I can deal with the lower performance.

 

[pbucher]

Like someone actually wanting or needing that. Continuing the discussion of bug 1530, my use cases for a filer that goes catatonic for many seconds at a time are much fewer than a filer that remains responsive but has lower performance. I can deal with the lower performance.

I'm being slow, but I'm not seeing a 1530 that makes sense for this thread. I read your bug 1530 about 3ware formatted disks causing the filer to panic but that doesn't make sense for this thread.....

 

[jgreco]

Insufficient coffee. 1531

 

[jyavenard]

In regards to using LSI based controllers. My experience is that if you set the LSI controller in the BIOS to jbod and format the disk there; then yes. The disk's data becomes unusable without a LSI adapter: you can't take out the disk and use it in another machine.

However, if you format the disk elsewhere (or the same machine, but using the intel sata controller) and then plug it to the LSI ports: then it works as one would expect. And then you can still use it on another machine without an LSI adapter later.

smart should work (certainly did with my LSI megaraid card)

 

[jgreco]

I think you're right but I also seem to recall there are some significant caveats to that. I just don't remember what they all are. HBA's in IR mode definitely work that way, but we generally encourage users to use IT mode, where it isn't an option or an issue. I think it's worse under MFI (mpt driver). Um, but I should probably not try to fuzzy recall stuff, and we should keep this verified-fact based.

Does anyone have an MFI (SAS2 RAID) card around to play with right now?