Transfers perform much better on second attempt to copy.

[RP@45]

A strange issue I came across that I'm scratching my head at.

Attempting to transfer 160 GB file over 10GB network, it was only transferring average 250MBps. After that was complete, I did deleted from the folder it was copied to, transferred again and it performed better (450MBps - 500MBps) on average.

Crystal Disk Mark has both R & W over 1GB and we can see the full pipe being used in the Netdata GUI of freeNAS during the benchmark. During the transfers, the pipe is only being used half to a quarter of it's ability according to Netdata GUI.

It's a large .mov file. Our pool was 4 x 10 Raidz2 10 TB WD Gold.

SMB share, windows client, active directory enabled.

During testing, we created a RAM disk that was benchmarking over 3GBs both R & W but when transferring to that drive off the share, same transfer speeds (250MBps)

From the FreeNAS reporting, during the transfers it looked like everything was being pulled from the Arc

Any thoughts or input would be appreciated!

 

[kdragon75]

What are you testing parameters in Crystal disk mark? By face value I would suspect the file in question is getting cached in the ARC but then I would expect 40 drives (even in z2) to do better than 250MB/s

Per forum rules please provide full system and network specs.

How full is your pool?

 

[Chris Moore]

During testing, we created a RAM disk that was benchmarking over 3GBs both R & W

This is giving you the results of the network adapter and various software controls.

but when transferring to that drive off the share, same transfer speeds (250MBps)

This is giving you the results of how fast your hard drives are.
You have a problem when trying to get 10Gb network speed out of only four spinning disk drives. The problem is that the drives are not that fast. I have a iSCSI pool in my server with 16 drives arranged in 8 mirror vdevs and I get almost 800MB/s and that isn't too bad but it would be faster with better drives.
You simply can't get much more speed than that out of the drives you have.

 

[Chris Moore]

Our pool was 4 x 10 Raidz2 10 TB WD Gold.

Is that 4 drives at 10TB each arranged in RAIDz2? This could be more clear.

 

[kdragon75]

Is that 4 drives at 10TB each arranged in RAIDz2? This could be more clear.

Thats a good point I just read it as having 4 x (10 drive raidz2) vdevs striped together for a total of 40 drives. Now that I write it out, a pool of 400TB is kind of excessive for 99% of use cases.

 

[RP@45]

Hi guys,

It's 40 drives in the unit, each are 10 TB WD Golds.

Pool is below 3% full.

 

[Ericloewe]

That seems rather slow. While transferring stuff, have a look (or several) at zpool iostat and show us the output.

 

[Chris Moore]

Hi guys,

It's 40 drives in the unit, each are 10 TB WD Golds.

Pool is below 3% full.

We still need all the detail you can give us on the hardware you are using and how it is configured. There may be some basic hardware incompatibility that is at the root of this.

It is in the rules for a reason:
https://forums.freenas.org/index.php?threads/updated-forum-rules-4-11-17.45124/