I'm building a high performance ZFS SAN for education

Status
Not open for further replies.

mikesm

Dabbler
Joined
Mar 20, 2013
Messages
36
Agree that the non-HT CPU's seem to be kind of lethargic, but I have an exceedingly small sample size I'm working from in making that statement. But since you were suggesting it, ours is an an E5-2609 too, which seems to compare poorly against our common E3-1230's.

....

3) Do not connect any ZFS pool drives to the LSI 2208 on the system mainboard. You already have a 9207-4i4e for that. I expect that'd be a reasonable choice but haven't actually used one with ZFS, but it's a 2308 based controller and I believe it can be configured to look like an HBA.

4) And now, for something clever. Omit the ZeusRAM you wanted to buy. Instead, buy the BBU for the onboard LSI 2208. Then attach the 2208 to a conventional hard drive sufficiently fast to handle your sync write load. The BBU plus cabling should be about $300. Now the thing is, by doing this, you gain a SLOG device that has virtually unlimited write endurance, and the latency of having the mpt driver shove data out to the RAID controller write cache is very low compared to having a write command issued and completion waited-for over the SAS channel. Bonus: the cache is MUCH faster than 6Gbps, so your SLOG device can accept the first NNN megabytes of sync writes at incredible speed.

I'm doing something entirely different than you, so I'll explain a bit first: we're using the 2208 with a pair of Seagate Momentus XT's in RAID1 for ESXi local storage and boot. Since most of our ESXi storage is SAN based, the 2208 is only lightly used. The system is slated to handle a bunch of VM's and then also a FreeNAS instance for general fileservice. There is normally no need for a SLOG device, but by just adding a virtual disk on the local ESXi server, I magically get this SLOG device with awesome characteristics... and since the 2208 is usually not at all busy, I get a SLOG device that can sustain almost 50MB/sec (craptacular throughput on the XT's) but bursts the first few hundred MB in a second.

Sorry to refresh an old thread, but I am about to upgrade my existing ZFS server to a new one based on this same case and Supermicro X9DR7-TF+. I have M1015's from the older server to connect 20 2 TB and 4 TB drives, and I'd like to try doing your trick for use of the onboard 2208 and cache (+BBU). I was planning to run freenas native on the hardware, but I do run ESX hypervisors on two other home servers. Would you recommend I run ESXi on this server and keep the system disks on the 2208 with cache? I am not sure how that creates a SLOG device for FreeNAS.

If I was doing this today, it would seem SSD's would be more appropriate for the disks instead of the Momentus XT's, correct?

thanks!
mike

mod note: separated from ancient thread, edited for link
 
Last edited by a moderator:
Joined
Oct 2, 2014
Messages
925
Sorry to refresh an old thread, but I am about to upgrade my existing ZFS server to a new one based on this same case and Supermicro X9DR7-TF+. I have M1015's from the older server to connect 20 2 TB and 4 TB drives, and I'd like to try doing your trick for use of the onboard 2208 and cache (+BBU). I was planning to run freenas native on the hardware, but I do run ESX hypervisors on two other home servers. Would you recommend I run ESXi on this server and keep the system disks on the 2208 with cache? I am not sure how that creates a SLOG device for FreeNAS.

If I was doing this today, it would seem SSD's would be more appropriate for the disks instead of the Momentus XT's, correct?

thanks!
mike
i would suggest starting your own thread, so you can add all the HW spec's/details and all without digging up old posts.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
You can do whatever makes sense. In the case I described, I didn't really need a SLOG device - I just noticed I could have one for "free".

The problem with using SSD for SLOG is that you have a limited amount of endurance. After X amount of data is written, it will die.

The thing I noticed was that the write cache on the RAID controller is extremely fast (not quite NVMe speed, but still...) and the hard disks making that datastore have a very high level of endurance because they're HDD.

The downside was that the 2208 had poor support in FreeNAS directly, but that wasn't really a problem since I intended to use it as a hypervisor anyways. So FreeNAS happily runs under ESXi, the 2208 and two HDD's form the boot datastore, and the datastore's pretty fast so it can be abused as a SLOG device.
 
Status
Not open for further replies.
Top