Transfer Speed Brakes Down

[heldchen]

Hi,
i have a "problem" with truenas 13-u1. testet on 12.x before and same happened.
My Hardware: Supermicro Storage Server with Xeon 6 Cores and 128gb of ram.
10x16TB HDD(data), 2xSSD(system)

Tested with Raid z1 or z2 with all of 10 drives and tested with 4 drives z2 and 3 drives z1, and with 3 ssd z1 via onboard controller to make sure that the HDD controller isnt the bottleneck.

All of this configuration tested against a server and a test workstation via 10gb ethernet, with iscsi share and smb.

Every time it starts very speeded up with 700mb/s to 900mb/s then after 5 to 10 minutes it brakes down, sometimes to 0kb, sometimes to 70-90mb/s and hold this speed for a while. after some minutes it speeds up most time to the first rates. And this is repeating evere 5-10 minutes.

when i make a stripe of 3 of these disks, without raid-z mode, nothing brakes down, only with raid-z level. what to do?

 

[heldchen]

My Test Object at this time is one big 2TB file and speed is going up and down as described. But now a saw something.

 

[heldchen]

No Tips or hints for me?

 

[Samuel Tai]

With any RAIDZ*, you only get the write performance of at most a single disk, because each write needs to write a data block and some number of parity blocks on different drives. So you're seeing the raw write performance of your slowest drive.

For your screen shot, that indicates you may have a bad controller, which is disconnecting from the PCI bus under load.

 

[heldchen]

Thanks for reply. So what will be a better Setting? This storage will only be a Target for Backups via Veeam filled by a single Server. My Controller is a S3008L-L8e AOC-S3008L-L8E bought with the supermicro Board.

 

[indy]

With any RAIDZ*, you only get the write performance of at most a single disk, because each write needs to write a data block and some number of parity blocks on different drives. So you're seeing the raw write performance of your slowest drive.

Why am I able to write incompressible data to a raidz2 pool, way past the dirty data limit, at greater than single hard-disk speed then?

Anyways, I reported a maybe similar problem transfering from Truenas here:

TrueNAS 13.0-RELEASE has been released

Is this going to be rebased to Freebsd 13.1 since it was released on May 16th? Or is this already based on it? The reason I ask is that once I upgrade to it on the U1 release, does that mean our jails will only be valid for a few months after upgrading them and then the updates will no...

www.truenas.com

No feedback though and I did not not diagnose any further.

 

[mav@]

Why am I able to write incompressible data to a raidz2 pool, way past the dirty data limit, at greater than single hard-disk speed then?

RAIDZ vdev has IOPS of single disk, but throughput (especially on large recordsize write and if pool is not fragmented) should be close to all disks minus used parity.

What's about pauses in writes, I'd try to look on CPU (top -SHIz) and disk (gstat -I 1s -pa) activity during those pauses to try and see what is going on. There were some discussions in upstream OpenZFS recently (https://github.com/openzfs/zfs/pull/12868) that some write patterns may cause spikes in existing write throttling algorithm, and there were some improvements, but nothing is perfect.