Performance Questions

[HANDLEric]

Hello,

i'm doing some testing with FreeNAS and iSCSI as a POC. I stood up FreeNAS on an old R710 we had around with these specs:

(2) Intel Xeon CPU E5620 @ 2.40GHz - 8 total cores/16 htreads
48GB RAM
(2) WD RE4 300GB in RAID 1 for the FreeNAS OS
(6) 300GB 10K SAS drives for a Testing Pool
(1) 1GB dedicated management connection
(1) 10GB iSCSI Network Connection

I was able to configure the storage pool as a stripe which gave me ~1,500GB of usable storage and then provision a LUN and present it to some ESXi hosts. My question is, after I created a datastore on the storage I started copying over a 60GB VM and the rate seems to be far slower than I expected and I also notice the FreeNAS web GUI is very sluggish during this time- I'm also seeing gaps in the performance graphs, but what data I do have does not indicate (at least to me) a performance bottleneck. Would adding cache help with this?

 

[HoneyBadger]

What is the storage controller in this machine?

 

[HANDLEric]

It's currently running a PERC H710- the OS drives are set to RAID 1, but the data drives are set to single drive raid 0 volumes and the stripe is managed by the OS

 

[HoneyBadger]

It's currently running a PERC H710- the OS drives are set to RAID 1, but the data drives are set to single drive raid 0 volumes and the stripe is managed by the OS

That's likely a major contributor to your issues. Replace this with a PERC H200 or similar HBA (although if you flash the official LSI firmware, you may need to use a dedicated PCIe slot rather than the "storage slot") and I wager your performance will improve.

The other thing to note is that iSCSI by default is not using "safe" sync writes, so you'll need to consider the use of an SSD as a separate log (SLOG) device.

 

[HANDLEric]

That's likely a major contributor to your issues. Replace this with a PERC H200 or similar HBA (although if you flash the official LSI firmware, you may need to use a dedicated PCIe slot rather than the "storage slot") and I wager your performance will improve.

I can make this happen, but how would I handle the RAID for the OS drives?

The other thing to note is that iSCSI by default is not using "safe" sync writes, so you'll need to consider the use of an SSD as a separate log (SLOG) device.

Okay, so ill add a cache device like a PCIe drive, then I just add it to the pool as cache vir the GUI right?

 

[HoneyBadger]

I can make this happen, but how would I handle the RAID for the OS drives?

Just select both drives when installing FreeNAS, and it will create a ZFS mirror of them.

Okay, so ill add a cache device like a PCIe drive, then I just add it to the pool as cache vir the GUI right?

You'll want a certain type of PCIe drive - see the SLOG thread in my signature, but the short answer for best performance is "Intel DC P3700 or Intel Optane" and then you need to add it as a "log" device ("cache" is for reads) then set the option sync=always for the ZVOL you want to protect.

 

[HANDLEric]

Okay, so I will create a new Pool containing the SLOG drive and then whatever volume regardless of pool will automatically know to use it once I set the sync-always option?

 

[HoneyBadger]

Okay, so I will create a new Pool containing the SLOG drive and then whatever volume regardless of pool will automatically know to use it once I set the sync-always option?

If you set the sync=always at the top level of the pool itself, yes, all of the child datasets or ZVOLs from it will inherit this setting.

 

[HANDLEric]

If you set the sync=always at the top level of the pool itself, yes, all of the child datasets or ZVOLs from it will inherit this setting.

But the log volume doesn't have to be inside the same pool as the volumes I want to use it for right?

I only ask because I will have different pools that represent different storage tiers, it would be nice to share the log resource among all of them.

 

[HoneyBadger]

But the log volume doesn't have to be inside the same pool as the volumes I want to use it for right?

I only ask because I will have different pools that represent different storage tiers, it would be nice to share the log resource among all of them.

Log devices can only be attached to a single pool, unfortunately.

There have been a couple users who have partitioned up a high-performance log device, such as an Optane 900p or P4800X, and used that to service multiple pools at once; but you still have to share the total performance of the device across all pools.

 

[HANDLEric]

Log devices can only be attached to a single pool, unfortunately.

There have been a couple users who have partitioned up a high-performance log device, such as an Optane 900p or P4800X, and used that to service multiple pools at once; but you still have to share the total performance of the device across all pools.

Well I would rather be on the safe side, and I don't have enough PCI slots to accommodate log drives for 3 or more pools. How difficult is it to split a drive like that, I assume this has to be done using f disk or similar methods?

 

[HoneyBadger]

Well I would rather be on the safe side, and I don't have enough PCI slots to accommodate log drives for 3 or more pools. How difficult is it to split a drive like that, I assume this has to be done using f disk or similar methods?

Not difficult at all, but does have to be done from the command line because the GUI will just assign the entire drive.

See command list below; the -a 1m in the first gpart add is there to try to start your partitions on a 1M boundary which should align things nicely with NAND program/erase pages.

Code:

gpart create -s gpt nvd0
gpart add -i 1 -a 1m -t freebsd-zfs -s 16g nvd0
gpart add -i 2 -t freebsd-zfs -s 16g nvd0
gpart add -i 3 -t freebsd-zfs -s 16g nvd0

glabel status | grep nvd0 #note the GUIDs for each partition

zpool add POOL1 log gptid/first-nvd-guid-goes-here
zpool add POOL2 log gptid/second-nvd-guid-goes-here
zpool add POOL3 log gptid/third-nvd-guid-goes-here

 

[HANDLEric]

@HoneyBadger is 16g substantial enough size wise? I mean it's an 800GB drive in total, so it would be easy to allocate more space if it would be in any way beneficial.

 

[HoneyBadger]

16G is more than sufficient; the default amount of outstanding data (per pool) is 4GB. This could be adjusted upwards but it eats into your main memory as well, and with 48GB this could mean up to 12GB is being held for pending writes.

Which device are you considering for the SLOG?

 

[HANDLEric]

Okay,

so I ordered an Intel DC P3700 and a PERC H310, is there anything else you would recommend that I change here? I am running encryption and would like to run de-duplication and compression if possible

 

[HoneyBadger]

So I ordered an Intel DC P3700 and a PERC H310, is there anything else you would recommend that I change here?

Is the H310 a regular PCIe version? The "embedded" or "mini/mono" one cannot be flashed with the official LSI firmware, and the stock firmware has a very poor I/O queue depth that makes it unsuitable for high performance solutions.

I am running encryption and would like to run de-duplication and compression if possible

Encryption is generally recommended only if there is a legal or regulatory framework that requires it; there are a number of ways to irreparably lose data if you make a mistake with encryption. Be very careful if you do this and make sure you fully understand the solution. (Experiment first!)

Compression is fine and is basically "free" on any modern processor; LZ4 is extremely efficient.

Deduplication on the other hand is a massive, massive performance and resource hog (think "dozens of GB of RAM per TB of data stored, with the potential to cripple your pool if you undersize) and no easy way to disable it after the fact. Strong recommendation against it.