Expected I/O Performance Question

[websmith]

Hi People,
I am fairly new to ZFS and FreeNAS in particular.

I have been using ZFS on Linux for about a year in VM with a LSI SAS card in passthrough mode, but decided to set up a dedicated box for my NAS.

My build is:

SilverStone CS380 (8x3.5" hot plug)
Cremax ICY Dock MB455SPF (8x2.5" hot plug)
SuperMicro X10SRi-F
1xXeon E5-1620v3 (4c, 8t)
64GB ECC REG, 4 modules so I can upgrade to 128GB if needed

1xIBM 46M0912 (M1015 i think) SAS2, 8i, flashed to IT mode, P20
1xDell H200 8i, Flashed to IT mode, P20 (Planning to use for storage pool)
1xLSI SAS 921116e flashed to IT mode, P20 (Planning to add external storage)
1xOptane 900P, PCI-e, 280GB

1xIntel X540 T2, where one port is being used for management/my use and the other port is being used as a dedicated point to point connection to my esxi server with NFS.

3xWD RED, 2TB
3x Seagate NAS 2TB
4x STEA4000400, taken out of case, not in use yet, but plan is to use them as a backup pool

My main pool:

NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
gptid/d0195541-a267-11e8-a5cc-000c293f95ce ONLINE 0 0 0
gptid/d101bb1c-a267-11e8-a5cc-000c293f95ce ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
gptid/d1da3874-a267-11e8-a5cc-000c293f95ce ONLINE 0 0 0
gptid/d29d9899-a267-11e8-a5cc-000c293f95ce ONLINE 0 0 0
mirror-2 ONLINE 0 0 0
gptid/efed68cb-aba1-11e8-87b7-002590a3474e ONLINE 0 0 0
gptid/f0b4111a-aba1-11e8-87b7-002590a3474e ONLINE 0 0 0
logs
gpt/optane-log0 ONLINE 0 0 0
cache
gpt/optane-cache0 ONLINE 0 0 0


The L2ARC is 64GB
the ZIL is 16GB

Output from top:

last pid: 93494; load averages: 0.10, 0.24, 0.23 up 2+13:32:24 11:51:35
90 processes: 1 running, 89 sleeping
CPU: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 99.9% idle
Mem: 156M Active, 1254M Inact, 44G Wired, 16G Free
ARC: 29G Total, 9042M MFU, 20G MRU, 20M Anon, 62M Header, 197M Other
27G Compressed, 46G Uncompressed, 1.70:1 Ratio
Swap: 6144M Total, 6144M Free



Output from dmesg|grep mps

mps0: <Avago Technologies (LSI) SAS2008> port 0xe000-0xe0ff mem 0xfb040000-0xfb04ffff,0xfb000000-0xfb03ffff irq 26 at device 0.0 on pci2
mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd
mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
mps1: <Avago Technologies (LSI) SAS2008> port 0xd000-0xd0ff mem 0xfa600000-0xfa603fff,0xfa580000-0xfa5bffff irq 40 at device 0.0 on pci4
mps1: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd
mps1: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
mps2: <Avago Technologies (LSI) SAS2116> port 0xb000-0xb0ff mem 0xfae9c000-0xfae9ffff,0xfae40000-0xfae7ffff irq 16 at device 0.0 on pci7
mps2: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd
mps2: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
mps1: SAS Address for SATA device = 3c2b3d30817c3f40
mps1: SAS Address from SATA device = 3c2b3d30817c3f40
mps1: SAS Address for SATA device = 382a5233918a373f
mps1: SAS Address from SATA device = 382a5233918a373f
mps1: SAS Address for SATA device = 34634434dda2be6e
mps1: SAS Address from SATA device = 34634434dda2be6e
mps1: SAS Address for SATA device = 383053306b7b5633
mps1: SAS Address from SATA device = 383053306b7b5633
mps1: SAS Address for SATA device = 34634230e4a7c06c
mps1: SAS Address from SATA device = 34634230e4a7c06c
mps1: SAS Address for SATA device = d1766451f8b9bd87
mps1: SAS Address from SATA device = d1766451f8b9bd87
da1 at mps1 bus 0 scbus1 target 1 lun 0
da0 at mps1 bus 0 scbus1 target 0 lun 0
da2 at mps1 bus 0 scbus1 target 2 lun 0
da4 at mps1 bus 0 scbus1 target 4 lun 0
da3 at mps1 bus 0 scbus1 target 3 lun 0
da5 at mps1 bus 0 scbus1 target 5 lun 0



So wall of text completed.

My question is what kind of write speed should I be able to expect with the above pool? I know that in raw terms, I should not expect more than 3xsingle drive write speed, since i'm running with a striped mirror zvol.

Initially I had a Intel DC S3520 SSD for ZIL/L2ARC, but exchanged it for the Optane since I hoped that the better speed would give me better write speed, but it does not seems like it.

using dd from my esxi with sync=forced:
time dd if=/dev/zero of=./9G.bin bs=9000 count=1000000

1000000+0 records in
1000000+0 records out
real 0m 37.83s
user 0m 28.52s
sys 0m 0.00s

time dd if=/dev/zero of=./9G_2.bin bs=1M count=9000
9000+0 records in
9000+0 records out
real 0m 36.00s
user 0m 26.89s
sys 0m 0.00s

time dd if=/dev/zero of=./9G_3.bin bs=4k count=2200000
2200000+0 records in
2200000+0 records out
real 0m 39.51s
user 0m 28.41s
sys 0m 0.00s


which translates to roughly 230MB'ish write speed no matter the block size

Mounting an iscsi zvol, also with sync=forced from my windows and testing with CrystalDiskMark I get similar'ish results.

One of the reasons for going with a dedicated NAS was that I could install a 10GB netcard in my main rig and get "local like" speeds to my nas, and also for durability reasons, so my esxi server(s) could share fast storage.

I was hoping that with the above setup I could actually use the speed of the ZIL to boost my 3xmirror write speed so I could get around 600MB/s.

Is that impossible?, would I need to add three extra mirrors to my pool to get that kind of performance?

I am happy with my 900P, since the latency is extremely low on it and when the L2ARC is filled as it should be by my esxi servers, at least the esxi server will have low latency access to the nfs share, just not crazy read/write speeds.

Anything I should do differently, I have spent lots of money on this setup, which is fine, since computers is my hobby - I just need advice on how to best utilize the current gear and advice to make it better.

I have read a lot of the previous posts about performance issues and my own conclusion is that my performance will probably not be much better with the amount of hardrives i have now, but I would rather ask the experts.

Thanks in advance for any replies.

Bjorn

P.S. Yes I know I should have asked for advice before spending stacks of cash, but I wanted to learn by doing as well and spending money is no big issue (within limits)
P.P.S I have a Intel DC S4600 960GB lying on my desk that I consider buying a twin for and use for a mirrored pool for my esxi, not sure if thats worth it.

 

[kdragon75]

I should not expect more than 3xsingle drive write speed, since i'm running with a striped mirror zvol.

This is correct no matter what SLOG you use for the ZIL. Optane is a great choice for the SLOG as its extremely low latency.

which translates to roughly 230MB'ish write speed no matter the block size

This looks close to right to me.

I was hoping that with the above setup I could actually use the speed of the ZIL to boost my 3xmirror write speed so I could get around 600MB/s.

Again, the SLOG will reduce the negative performance impact of sync writes but it will never be faster than setting sync=disabled on the dataset/zvol. This setting can be changed on the fly safely. Try turning it off and test again. It WILL be faster (and a bit unsafe in the event of power loss) but will tell you the upper limit of the pool. To be honest, a peak of 435MB/s sync writes on a 3 vdev system is not bad.

1xIntel X540 T2, where one port is being used for management/my use and the other port is being used as a dedicated point to point connection to my esxi server with NFS.

That's odd... What not use the 1g for management and multipath for the NFS? I guess it depends on the host setup...

P.P.S I have a Intel DC S4600 960GB lying on my desk that I consider buying a twin for and use for a mirrored pool for my esxi, not sure if thats worth it.

I hope you didn't pay full price for that. You could get a dual PCIe NVMe adapter and two large drives for that price while tripling your performance. Assuming your board supports PCIe bifurcation. I did see an NVME adapter that has a built in PCIe switch chip but I can't seem to find it now.

 

[websmith]

This looks close to right to me.

Well, that just sucks, but that was my suspision - so I guess I would need to double the number of VDEV's at least to be able to saturate my 10G connection?

That's odd... What not use the 1g for management and multipath for the NFS? I guess it depends on the host setup...

Well I wanted to have one fast connection to my own computer, but I guess I could just pop in another 10G card that I have at hand and do multipath - but from my understanding its only for failover right?

I hope you didn't pay full price for that. You could get a dual PCIe NVMe adapter and two large drives for that price while tripling your performance. Assuming your board supports PCIe bifurcation. I did see an NVME adapter that has a built in PCIe switch chip but I can't seem to find it now.

Nope, got a sweet deal on it, paid around 200 USD for it, with 127 hours usage on it and 90 GB data written, so I could not say no to that :)