Max write speed on nosync dataset

[daves_garage]

Greetings! I have a Storinator Q30 with thirty 14TB. I have an Optane stick for cache and logs. The system has 128GB of RAM. TrueNAS Scale. Xeon Silver 4216.

My issue is that read speeds are in excess of 1GB/sec (on a 10gbit NIC), but write speeds max out at 700MB/sec. This is true on any dataset, including a test I created with sync disabled.

In my mind, it should be able to take data as fast as you can send it if it's just going to the TrueNAS cache. Yet it seems to be constrained by the physical write speed. Since the 30 drives are broken up into 3 vdevs of 9 drives plus a spare each, 700MB/sec might be the limit for writing to 9 drives. But again, with sync off I thought it should just be writing to RAM cache until the cache gets full, which it's nowhere near.

What am I missing that I can try in order to get write speeds equal to the cache write speed?

Thanks!

 

[HoneyBadger]

Morning Dave,

ZFS doesn't have a traditional "tiered" style of caching as some other filesystems do; so you won't be writing to your Optane device and then later "de-staging" it to the pool. Writes are collected into RAM, with any sync writes also going to your log device for safety.

Can you describe the pool configuration (I see 3x 9-wide VDEVs - I assume a RAIDZ type?) as well as what type of benchmark you used to get the 1GB/s read and 700MB/s write numbers?

Setting up a test dataset using a larger-than-default (eg: 1M) recordsize may also help, because a larger chunk of data to be spread across the disks can allow more disks at a time to contribute to the I/O.

 

[daves_garage]

It turns out to have been Mac-limited. When I tested from a PC with a 25Gbe card, I was writing at 1750MB/sec, so it's performing fine. Not sure why the Mac (Blackmagic Disk Speed) can't write at full speed, but it's not the NAS's fault!

 

[HoneyBadger]

It turns out to have been Mac-limited. When I tested from a PC with a 25Gbe card, I was writing at 1750MB/sec, so it's performing fine. Not sure why the Mac (Blackmagic Disk Speed) can't write at full speed, but it's not the NAS's fault!

Assuming you're connecting the Mac over SMB, the default behavior for MacOS is to leverage synchronous writes to some degree - but the fact that you still saw this limit with a sync=disabled dataset means that might not be the case. Could be the chipset in the Mac machine or the way the adapter is connected if it's an external/TB based one.

Glad to hear it soars along at over twice the speed from a PC on 25Gbps!