I know there are others trying to figure out how to build a san on the cheap (using that term loosely) to host virtual machines. I thought i'd share my current setup as it has been hugely successful for us, even with room to improve.
We started with a Super Micro 24 Bay Chassis, 64GB Ram, E52620 CPU, LSI 9207 (Geniune, not a reflashed ibm) HBA Chelsio S320E-LP-CR, 24 2TB WD Black, 2 Intel 400GB PCIe SSD.
We are running 11 Mirrored Vdevs, keeping two drives as spares.
We use one ssd for Zil (Log) and the other for L2Arc (Cache)
We use nfs and currently do no use lagg.
NFS is set sync=always
We have one link dedicated for nfs and the other for ZFS Replication.
We have two nfs stores, one that replcated to the other server and one that doesn't replicate.
We replicate ever yhour and keep 4 hours.
We use nfs over iscsi because of how easy it is to access the replicated data in the event of a major failure.
Our system could use more memory and we're looking at adding another 96gb of ram, but so far performance is really great.
We are currently using the default auto tune settings, with the exception of:
vfs.zfs.vdev.trim_on_init = 0
You need this or the server will take forever to boot.
We tried intel 520 10gb nics but ultimately chose the chelsio nics as they are recommended.
Don't worry about 12gb controllers etc right now, stick to the LSI 9207 controller. It's easily fast enough.
12gb is still "beta" at the point of writing this.
I hope this helps someone else out.
We started with a Super Micro 24 Bay Chassis, 64GB Ram, E52620 CPU, LSI 9207 (Geniune, not a reflashed ibm) HBA Chelsio S320E-LP-CR, 24 2TB WD Black, 2 Intel 400GB PCIe SSD.
We are running 11 Mirrored Vdevs, keeping two drives as spares.
We use one ssd for Zil (Log) and the other for L2Arc (Cache)
We use nfs and currently do no use lagg.
NFS is set sync=always
We have one link dedicated for nfs and the other for ZFS Replication.
We have two nfs stores, one that replcated to the other server and one that doesn't replicate.
We replicate ever yhour and keep 4 hours.
We use nfs over iscsi because of how easy it is to access the replicated data in the event of a major failure.
Our system could use more memory and we're looking at adding another 96gb of ram, but so far performance is really great.
We are currently using the default auto tune settings, with the exception of:
vfs.zfs.vdev.trim_on_init = 0
You need this or the server will take forever to boot.
We tried intel 520 10gb nics but ultimately chose the chelsio nics as they are recommended.
Don't worry about 12gb controllers etc right now, stick to the LSI 9207 controller. It's easily fast enough.
12gb is still "beta" at the point of writing this.
I hope this helps someone else out.
