Slow SAS drives Volume, SSD Volume fast

[Ulrich Jorgensen]

Hi,

I have a FreeNAS-11.1-U4 server, based on a SuperMicro 3U server, 8-core CPU, 128GB RAM, 2x SAS12 controller connected to a SAS12 backplane with 2 SAS-expanders.
I have 6x Kingston DC400 800GB SATA SSD's and 6x Seagate 2.4TB SAS 10K HDD's, both configured as striped mirrors.

Kingston SSD Volume performs 1-1.5 GB/s, which is fine, but Seagate SAS volume performs at 25MB/s, and that is simply too slow.

1 of the Seagate SAS drive can perform up to 270 MB/s alone, so I cannot understand why 3 striped mirrors should perform only 25MB/s...and shouldn't my ARC of approx 120GB kick in and give me very high results when I test with working sets of only 4-16GB?

Any assistance would be greatly appreciated.

 

[Stux]

How are you benchmarking?

 

[Ulrich Jorgensen]

FreeNAS is connected to VMware cluster over 10Gbit/s network, and we are using NFS as protocol.
I have been using a couple of test vm’s that I have been live storage migrating both vm’s files and vmdk files between the SAS HDD’s and the SSD’s.
On the windows vmwe have tested with AJA System test, and on the linux vm with dd.

Both windows and linux vm have been tested on both storage several times, and shows similar results.

We can also feel that the vm’s are slow on the SAS HDD’s which again, they should not be with 3 striped mirrors of the fastest SAS HDD’s on the market

 

[Stux]

What you are benchmarking is the sync write speed via nfs of your array.

In order to see sync write speeds better than say 5MB/s per vdev, you need a SLOG drive.

Preferably an PCIe NVME with PLP. The best is the Intel p4800x. It’s also the most expensive.

You could perform some testing like this:
https://forums.freenas.org/index.php?threads/testing-the-benefits-of-slog-using-a-ram-disk.56561/

 

[Ulrich Jorgensen]

Interesting thread, thank you for the link.

Can you tell me what makes spinning disks so slow compared to SSD’s in FreeNAS?

I have 6 seagate 2.4 TB SAS 10K drives in striped mirrors, and I only get about 25-30MB/s from the volume over NFS.

Each drive in it self can perform 250+ MB/s, so what makes them so slow in FreeNAS.

If I would put them on a HW raid controller in RAID 10, I am sure the speeds would be much better.

 

[Stux]

ESXi forces all NFS writes to be synchronous writes.

Ie, uncached.

HDs are slow when writing synchronously. SSDs not so much. It’s because the HD has to physically rotate and position the head, write the data, then do that again to update the metadata before returning control back to ESXi.

With async writes, the writes are buffered and executed when the hd can.

If sync writes are slow. Use a SLOG, and then the sync writes will be written to the SLOG synchronously and asynchronously to the pool.

HW raid controllers need a BBU in order to safely cache sync writes. The SLOG serves the same purpose to FreeNAS

 

[Ulrich Jorgensen]

Thank you for clarifying that.

So when I see 1-1.5GB/s from the SSD’s, that id actually sync write speeds, that could even have been faster without sync. I actually though that was the ARC helping that result.

Would iSCSI actually give us caching on the FreeNAS then, so the 128GB RAM is not pure waste?

So a SLOG should help this, is a SLOG a better option than going iSCSI?
Do we need a SLOG for each volume, that is one for the SSD’s and one for the HDD’s or can they share a SLOG?

SLOG should have PLP, but should there be other security on a production system, like ie. mirror?

Thank you again.