Questions for best performance

Status
Not open for further replies.

Jimmy Tran

Dabbler
Joined
Dec 27, 2015
Messages
33
Hi All, I'm a noob to FreeNAS but have done quite a bit of research already. I just need some final recommendations on which way to go. I look forward to everyone's responses.

Setup:
SSG-6028R-E1CR12L Chassis
E5-2630V3 2.4 GHz
4 x 16GB DDR4 ECC REG DIMM
8 x WD 4TB RE Drives: Pool currently has 2 striped vdevs NO Compression
3 x Dell PE R610 X5550 8 x 2.659GHz
10Gbe Storage Network

1. I like VAAI and it looks like iSCSI is the only way to get it. Is there really any other benefit going with iSCSI as I don't have any apps that require block level access? I just would like the cloning and migrations to be quick but maybe that isn't necessary with a 10Gb network?

2. When testing with NFS, I was getting terrible speeds. After much reading (wish I did it first before testing) it looks like I need a SLOG due to the write sync. Is there a better option than the Intel SSD s3700 in a RAID 1?

3. Write Cache: My write cache would allow up to 8GB per transaction group. (64GB x 1/8). The pool has 8 disks capable of writing at 150MB/s. Would be Transaction Group be 6GB based on the info below?

150MB/s x 8 = 1200MB/s
6GB transaction group size (5 seconds)

If my calculations are correct, my write cache in RAM is more than my disks can handle. Does that mean ZFS will stop and wait for my flush to disk to finish? I'm a little confused about the IO stopping and waiting...

What if I have two separate vdevs with 4 disks each. Would the calculations above still be the same?

4. SLOG Size: Should the SLOG partition be 32GB since that is half of my available RAM or should it be equivalent to two times my Transaction Group size? I'm getting some mixed info...

5. iSCSI zvol: I was reading it should be more than 1TB and less than 50% full for optimum performance. Is there such a requirement for NFS?

I think those are all of my immediate concerns to get things going. Appreciate the responses and guidance!
 

DrKK

FreeNAS Generalissimo
Joined
Oct 15, 2013
Messages
3,630
It sounds like you did your homework. I feel like coherent commentary though requires maybe @cyberjock or one of the other guys more familiar with larger systems and/or iSCSI to weigh in. My first impression is that your selection of components is very good, at least.
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
4. SLOG Size: Should the SLOG partition be 32GB since that is half of my available RAM or should it be equivalent to two times my Transaction Group size? I'm getting some mixed info...
I wouldn't worry too much. Pick the larger of the two, more won't hurt. The drive might possibly perhaps maybe not have as easy a time doing garbage collection, but the effect is small enough that nobody has complained about using the full drive being slower.

less than 50% full for optimum performance. Is there such a requirement for NFS?
Well, if it's handling, in essence, a few big-ass files, it'll have the same issues as iSCSI.

2. When testing with NFS, I was getting terrible speeds. After much reading (wish I did it first before testing) it looks like I need a SLOG due to the write sync. Is there a better option than the Intel SSD s3700 in a RAID 1?
Yeah, the Intel P3700. It's 4x PCI-e 3.0 NVMe (PCI card or 2.5" with U.2 form factors available). Best SSD you can get these days for SLOG.

("RAID1" considerations apply to it, too)
 

Jimmy Tran

Dabbler
Joined
Dec 27, 2015
Messages
33
Yeah, the Intel P3700. It's 4x PCI-e 3.0 NVMe (PCI card or 2.5" with U.2 form factors available). Best SSD you can get these days for SLOG.

("RAID1" considerations apply to it, too)

I just looked up that SSD and didn't realize it costs so much! I might just have to bite the bullet since I've spend so much money on this server already...
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Few things:

Your transaction size is not based on the 150MB/sec. It's based on a really crude benchmark test that your system uses to "guess" the disks performance. It'll never, ever be 150MB/sec, even if your disks can do that at the beginning of the disk (where throughput is the fastest). The benchmark just sucks like that. In short, it's very convoluted and whatnot, and you shouldn't worry about transaction size at this point in time. ;)

Let the slog use the full disk. There's no gain and nothing lost by letting the full disk be used.

The 50% disk space is to allow the VMs to write data to the zpool, and for ZFS to be able to choose free space easily. Regardless of if you use iSCSi, NFS, or SMB (if it were possible) the 50% "requirement" would apply because it doesn't matter what protocol you are trying to use.

You can "simulate" having the slog by setting sync to disabled for the zvol or dataset in question.
 

Jimmy Tran

Dabbler
Joined
Dec 27, 2015
Messages
33
Thanks for the info @cyberjock. Do you have any suggestions regarding VAAI and if it is work sticking with iSCSI even if I have a 10Gb network?
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
4. SLOG Size: Should the SLOG partition be 32GB since that is half of my available RAM or should it be equivalent to two times my Transaction Group size? I'm getting some mixed info...

It needs to be a MINIMUM of two times the transaction group size. Period. It can be larger. But the useful data stored on your SLOG will NEVER be more than two times the transaction group size. So the real question is whether or not it is worth the trouble to calculate the maximum amount of data that could be, or if an intelligent guess is sufficient. I say an intelligent guess is fine.

So the thing is, if you've got a system with 64GB of RAM, the transaction group might possibly get as large as 8GB, so there might be 16-17GB of active data on your SLOG SSD. But they don't make SSD's that small anymore. So if you have a 400GB SSD, you maybe only really need ~18GB of it. In that case, I say make a 20GB SLOG partition.

iXsystems feels that it is fine for the SLOG to occupy the entire SSD. This could possibly be true-ish with modern SSD's with TRIM. The issue is that SSD takes some time to erase pages and return them to the pool of free pages. Under a situation where you have heavy write pressure, that is, writes streaming in as fast as the SSD can go, if the free page pool is small, you can blow through it very quickly, at which point the SSD starts writing more slowly.

So a smart admin will increase the size of the pool of free pages. This means that the SLOG device will be able to write more transaction groups before it hits that speed bump.

My argument is that if you reduce the SLOG size to ~20GB, then that 400GB SSD we were talking about retains an extra 380GB of zeroed pages in the free page pool, which implies that if the SSD is not capable of keeping up with the data that is coming in at, then you have a 380GB ADDITIONAL "buffer" that can be burned through at high speed before you hit that speed bump of "out of pages, have to erase-on-demand." The iXsystems guys are betting that this never happens. That used to be false, but now with TRIM, this comes down to a question of the speed with which the SSD is able to do page zeroing in the background.

But the flip side to this is that even if they're right, there's no HARM done in having the big bucket of free pages available, so the clever admin will go and underprovision the SLOG anyways. If there's a way to force something to behave the way you want it, without negative side effects, my vote is to just do it.

By the way, while the P3700's an awesome device, the Intel 750 400GB is a LOT cheaper. If you do not have a very heavy write load (please do the math!) then you may wish to consider this option.
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
By the way, while the P3700's an awesome device, the Intel 750 400GB is a LOT cheaper. If you do not have a very heavy write load (please do the math!) then you may wish to consider this option.
Right, good point: The Intel 750 is a P3700 tuned for client workloads.
 

Jimmy Tran

Dabbler
Joined
Dec 27, 2015
Messages
33
How do I properly determine what my write loads are? I'll do some searching tonight but if someone has a quick answer, by all means let me know!
 

Fuganater

Patron
Joined
Sep 28, 2015
Messages
477
How do you plan on getting around the Dell PERC card?
 

Jimmy Tran

Dabbler
Joined
Dec 27, 2015
Messages
33
How do you plan on getting around the Dell PERC card?

I'm not using it for storage. The PERC only holds the OS for ESXi. All VM's like on shared storage on FreeNAS.
 

Fuganater

Patron
Joined
Sep 28, 2015
Messages
477
Ah ok I missed the very first line which was your chassis. Now that I look at it it is WAY overkill if you are using 3x ESXi hosts. Dual 2011 chips? You can get a 12 bay SAS2 chassis for less than $400 or a 24 bay for around $430-450. Would save you like $800 for some SSDs.

Question (Pure curiosity), how are you managing all 3 hosts? Do you have a vCenter license? I currently have 2x Dell R710 and I am using a trial of vCenter to manage both of them, and for HA, but will need something else once the trial is over.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
How do I properly determine what my write loads are? I'll do some searching tonight but if someone has a quick answer, by all means let me know!

In general, you don't, it's largely handwavey B/S and knowledge of what you're doing. If you REALLY have no clue at all, you can look at the stats in FreeNAS under Reporting->Disk and in ESXi, but that seems kinda like judging your vehicle driving habits by checking the odometer rather than actually being aware of what sort of driving you actually do.

In a VM environment, the thing you really want to try to do is to minimize trite writes. Every time you write something to the datastore, you may be increasing fragmentation. So if you make an important change to an important system file, okay, fine, but if you're merely updating the last access timestamp on your /etc/motd file as you log in to your FreeBSD VM, that's about as worthless as it gets.

So anyways, there's simple stuff you can to to reduce your write load, including small things like setting "noatime" on your VM's fstab entries where appropriate, or bigger stuff like not trying to run portupgrade out of cron every night. This all goes to help answer the question of "what sort of write environment am I promoting here." But there'll also be useful writes your system does as part of whatever task it has been designed to do. You just kinda have to think about that.

Does it seem reasonable that you might be writing lots and lots of data out to disk from your VM's every day? If so - do your VM's store lots of data locally (which means on what appears to be a local disk, i.e. iSCSI datastore), or do you take advantage of the FreeNAS box as a fileserver and store files on the network? The latter is much more likely to be friendly to SLOG devices.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
how are you managing all 3 hosts? Do you have a vCenter license? I currently have 2x Dell R710 and I am using a trial of vCenter to manage both of them, and for HA, but will need something else once the trial is over.

vSphere Essentials 6 doesn't have all the bells and whistles (no vmotion etc) but is only $670 for three years. The Essentials Plus with most of the basic bells and whistles (vmotion, HA, etc) is like 10x that.
 

Jimmy Tran

Dabbler
Joined
Dec 27, 2015
Messages
33
Does it seem reasonable that you might be writing lots and lots of data out to disk from your VM's every day? If so - do your VM's store lots of data locally (which means on what appears to be a local disk, i.e. iSCSI datastore), or do you take advantage of the FreeNAS box as a fileserver and store files on the network? The latter is much more likely to be friendly to SLOG devices.

All my VMDKs are stored on the FreeNAS. Only the OS is local on my ESXi host. The only exception are my Veeam backups which do copy to local storage. I am a MSP so I have a few VM's that are pretty write intensive, Exchange, N-Central (MSP Software), SQL. I have a file server, DC and a few other servers that aren't so dependent on writes. I definitely will be getting a SLOG once I decide which one.

Ah ok I missed the very first line which was your chassis. Now that I look at it it is WAY overkill if you are using 3x ESXi hosts. Dual 2011 chips? You can get a 12 bay SAS2 chassis for less than $400 or a 24 bay for around $430-450. Would save you like $800 for some SSDs.

Question (Pure curiosity), how are you managing all 3 hosts? Do you have a vCenter license? I currently have 2x Dell R710 and I am using a trial of vCenter to manage both of them, and for HA, but will need something else once the trial is over.

I am only using a single socket and I know it is overkill. I don't want to have to purchase another server. I have an Enterprise license for vCenter as I am a VMware partner and have NFR licenses.
 
Status
Not open for further replies.
Top