Good morning all ... I've been using FreeNAS now in my ESXi environment for about 7 months, and have been very pleased. @jgreco will likely remember talking to me when I stood up my last system. There was a lot of great info you all provided back then to get me started. While my last build was kind of a "hackers hobby", now I'm trying to standardize on a solid build to migrate over to, and provide increased speed along with path for future growth.
Just a refresher. Right now, Ive got a storage environment of a FreeNAS box with about 15TB of usable storage, running RAIDZ2. (7 3TB disks, with a hot spare). Yes, I know that isn't idea for ESXi. I was dealing with what I had at the time, with the space I needed. The network backend on this old box is two 1GB ethernet connections. I have both interfaces IPd up, and I use MPIO from ESXi to talk to the storage backend, not LACP, since that doesn't seem to work with a darn. System is dual XEON hex cores, with 96GB RAM. I have 20GB of Intel S3700 space (from 2 x 100GB drives) set for SLOG. I have a 20GB part on each, mirrors. I then have 160GB of JBOD space with the 2 remaining 80GB parts for L2ARC.
All in all, for my workloads, performance is acceptable. There are times the system comes under load, like with an Microsoft Exchange server does maintenance on a Exchange DB. But again, it's be tolerable.
I have a 2nd NAS that is running Windows Storage Server 2012 R2. I won't bother with the specs, because this box is going away.
Right now, the 2 NAS boxes are used solely for iSCSI storage for ESXi.
The goal now is to build two new boxes, with "best practices" and migrate everything over to the new FreeNAS boxes, and retire these old hodge podge pieces of crap.
I'm going to start out an say right now, the plan is to build with 1GB interfaces for network to start with. We are looking to upgrade network to 10GB next year when funds are available, but for now, I'm going to start on 1GB. I will be dropping 10GB dual port NICS in to these new units, so the HBAs are there, when the time comes.
Having said all that, here is what I was looking to do ...
I have already purchased 2 Supermicro 36 Bay Servers. Specs are below:
Supermciro 847E16-R1400LPB SuperChassis
Come with 2 SAS2 backplane:
Front: SAS2-846EL1 (Backplane support up to 24 3.5" Ports SAS/SATA Expander)
Rear: SAS2-826EL1 (12 Ports 3.5 "SATA SAS 826 backplane with single LSI SAS2x28 expander Chip)
Supermicro X8DTN+
Dual Intel Xeon E5645 2.26Ghz Hex Core CPU 144GB (18x 8GB DDR3 ECC REG Memory)
LSI 9211-8i HBA JBOD card On board Intel® 82576 Dual-Port Gigabit Ethernet Controller Dual 1400Watt power supply
My intent is to drop in a 2nd Intel 82576 dual port nic, giving me 4 1GB ports, and also drop in a Chelsio T520 for the future 10GB expansion.
With all of that said, the next thing is to ensure that we purchase drives, SLOG, L2ARC appropriately so that it works now, works when we move to 10GB, and provides a path for expansion. With that said, here is my intended layout on each machine:
2 x Verbatim 64GB USB Flash Drives for OS
2 x Intel DC S3700 200GB Drives mirrored for SLOG
2 x Samsung 850 Pro 256GB Drives striped for 512GB L2ARC
14 x 3TB Seagate Enterprise ES.3 7200RPM Drives for storage.
The layout of the storage would be to use 12 of the drives, in 2 drive mirrors, stripped together to give me 18TB of usable storage with 2 hot spare drives available in the chassis. (and I know about not "filling" it all the way for performance reasons. I say 18TB usable, because with compression, which I have seen give me about 1.5 to 1, i'll come in at a comfortable number under the 18TB of RAW space)
Thoughts on this layout? Clearly, with the handful of 1GB network ports, throughput will be limited to network. And a 200GB SLOG is overkill. I wanted to make sure it is sized right though for the future when we go live with 10GB.
And a small question about this layout. This layout leaves me with 16 available bays. From what I read, it "sounds" like I can add additional 3TB of space, by dropping in 2 additional 3TB drives, mirroring them, and adding them to the volume. If I'm wrong with that, lemme know.
Any thoughts are appreciated. The only thing I'm currently locked in with is the chassis, as I have already purchased them. All of the drive specs for volumes, SLOG, L2ARC are all open to be discussed, and honestly I want to make sure I get it right.
@jgreco Your wonderful insight here would be greatly appreciated!
Just a refresher. Right now, Ive got a storage environment of a FreeNAS box with about 15TB of usable storage, running RAIDZ2. (7 3TB disks, with a hot spare). Yes, I know that isn't idea for ESXi. I was dealing with what I had at the time, with the space I needed. The network backend on this old box is two 1GB ethernet connections. I have both interfaces IPd up, and I use MPIO from ESXi to talk to the storage backend, not LACP, since that doesn't seem to work with a darn. System is dual XEON hex cores, with 96GB RAM. I have 20GB of Intel S3700 space (from 2 x 100GB drives) set for SLOG. I have a 20GB part on each, mirrors. I then have 160GB of JBOD space with the 2 remaining 80GB parts for L2ARC.
All in all, for my workloads, performance is acceptable. There are times the system comes under load, like with an Microsoft Exchange server does maintenance on a Exchange DB. But again, it's be tolerable.
I have a 2nd NAS that is running Windows Storage Server 2012 R2. I won't bother with the specs, because this box is going away.
Right now, the 2 NAS boxes are used solely for iSCSI storage for ESXi.
The goal now is to build two new boxes, with "best practices" and migrate everything over to the new FreeNAS boxes, and retire these old hodge podge pieces of crap.
I'm going to start out an say right now, the plan is to build with 1GB interfaces for network to start with. We are looking to upgrade network to 10GB next year when funds are available, but for now, I'm going to start on 1GB. I will be dropping 10GB dual port NICS in to these new units, so the HBAs are there, when the time comes.
Having said all that, here is what I was looking to do ...
I have already purchased 2 Supermicro 36 Bay Servers. Specs are below:
Supermciro 847E16-R1400LPB SuperChassis
Come with 2 SAS2 backplane:
Front: SAS2-846EL1 (Backplane support up to 24 3.5" Ports SAS/SATA Expander)
Rear: SAS2-826EL1 (12 Ports 3.5 "SATA SAS 826 backplane with single LSI SAS2x28 expander Chip)
Supermicro X8DTN+
Dual Intel Xeon E5645 2.26Ghz Hex Core CPU 144GB (18x 8GB DDR3 ECC REG Memory)
LSI 9211-8i HBA JBOD card On board Intel® 82576 Dual-Port Gigabit Ethernet Controller Dual 1400Watt power supply
My intent is to drop in a 2nd Intel 82576 dual port nic, giving me 4 1GB ports, and also drop in a Chelsio T520 for the future 10GB expansion.
With all of that said, the next thing is to ensure that we purchase drives, SLOG, L2ARC appropriately so that it works now, works when we move to 10GB, and provides a path for expansion. With that said, here is my intended layout on each machine:
2 x Verbatim 64GB USB Flash Drives for OS
2 x Intel DC S3700 200GB Drives mirrored for SLOG
2 x Samsung 850 Pro 256GB Drives striped for 512GB L2ARC
14 x 3TB Seagate Enterprise ES.3 7200RPM Drives for storage.
The layout of the storage would be to use 12 of the drives, in 2 drive mirrors, stripped together to give me 18TB of usable storage with 2 hot spare drives available in the chassis. (and I know about not "filling" it all the way for performance reasons. I say 18TB usable, because with compression, which I have seen give me about 1.5 to 1, i'll come in at a comfortable number under the 18TB of RAW space)
Thoughts on this layout? Clearly, with the handful of 1GB network ports, throughput will be limited to network. And a 200GB SLOG is overkill. I wanted to make sure it is sized right though for the future when we go live with 10GB.
And a small question about this layout. This layout leaves me with 16 available bays. From what I read, it "sounds" like I can add additional 3TB of space, by dropping in 2 additional 3TB drives, mirroring them, and adding them to the volume. If I'm wrong with that, lemme know.
Any thoughts are appreciated. The only thing I'm currently locked in with is the chassis, as I have already purchased them. All of the drive specs for volumes, SLOG, L2ARC are all open to be discussed, and honestly I want to make sure I get it right.
@jgreco Your wonderful insight here would be greatly appreciated!
