Expectable raidz1 SMB write performance with 5x HDD at 10 Gbit ethernet?

[flashdrive]

Hello,

what kind of write speed can I expect from this setup when transfering large sized files from Win 10 to TrueNAS?

In the beginning of the file transfer the 10 GBit ethernet link is saturated at about 1 GByte / s.

However after a couple of seconds it will go down as low as constant 230 MByte / s.

I did not tune the system at all, it is stock. Only Jumbo Frames at 9014 at Windows 10 and TrueNAS. No switch, it is a direct LAN connection.

Disabling file encryption doesn't seem to make a difference.

5x 2 TByte HDD (very old, I know)

 

[flashdrive]

compare https://www.truenas.com/community/t...ware-still-very-slow-smb-file-transfer.92788/

 

[jgreco]

Yes, you fill the write cache (that's in your system memory) for the first moments, and then it degrades to whatever the pool can sustain.

Jumbo frames is generally a bad idea. It causes unusual code paths to be used, and you may run into problems that other people don't hit.

"5x 2 TByte HDD" .. "very old"? They still make those, you know.

 

[flashdrive]

reading material

ZFS Raidz Performance, Capacity and Integrity Comparison @ Calomel.org

LSI MegaRAID 9265-8i

FreeBSD Network Performance Tuning @ Calomel.org

 

[flashdrive]

Hello @jgreco

do you have a recommendation / howto were I can have a look into the topic?

I know my system is DIY - however I am learning step by step this way.

My goal is to get as near as possible to a sustained 10 GBit ethernet write saturation to TrueNAS.

 

[jgreco]

Well, the math for that is pretty simple. IF your drives are mostly empty, meaning that writes will happen to contiguous free space, AND your drives can sustain ~1Mbit/s (~100MByte/sec) write speeds average, then you need 20 drives in mirror pairs, because you need 10 vdevs each capable of 1Mbit/s.

The caveats are that drives stop giving you fast write speeds once there is fragmentation, and that while it might be tempting to use a faster number for what drives can write at, what you really need is the pessimistic inner track numbers, not the optimistic benchmark-favorite outer track numbers.

The 9265-8i is not suitable for use with FreeNAS.

https://www.truenas.com/community/t...s-and-why-cant-i-use-a-raid-controller.81931/

 

[flashdrive]

Thank you for the link and explanation.

~1Mbit/s (~100MByte/sec) should be
~1Gbit/s imho

So now I also do need to read into all that ZIL topic as well - having an SSD as write cache? I do not want to go all out onto the Optane highway - there was a great post into that topic

A bit about SSD perfomance and Optane SSDs, when you're planning your next SSD....

NOTE: I'll be referring in this page, to a type of SSD developed by Intel and Micron, called 3D X-Point (pronounced "crosspoint"). It's most widely sold as Intel's Optane. The Optane devices I mean are things like the 900p, 905p, and P48xx...

www.truenas.com

 

[jgreco]

So now I also do need to read into all that ZIL topic as well - having an SSD as write cache? I do not want to go all out onto the Optane highway - there was a great post into that topic

A bit about SSD perfomance and Optane SSDs, when you're planning your next SSD....

NOTE: I'll be referring in this page, to a type of SSD developed by Intel and Micron, called 3D X-Point (pronounced "crosspoint"). It's most widely sold as Intel's Optane. The Optane devices I mean are things like the 900p, 905p, and P48xx...

www.truenas.com

ZFS does not support SSD or Optane as a write cache. It only supports using main memory for write cache.

If you have made the usual error of thinking that "ZIL" or "SLOG" is some sort of write cache, I guarantee it isn't, and encourage you to go read

https://www.truenas.com/community/threads/some-insights-into-slog-zil-with-zfs-on-freenas.13633/