Write speeds: Why is one pool 5 times slower than the other?

[gobygoby]

I am confused by the results I am seeing and if anyone can help me, I would really appreciate it.

In this scenario I am using two computers:
1) Win10-Pro
2) TrueNAS Scale

The TrueNAS server has 2 pools in it:
-"fishpond"
-"fishpool"

I start by curating a media folder on the Win10 computer. I add files, rename them, move then around, etc.

Then I use FreeFileSync to update the files onto a dataset in "fishpond" using an NFS share over ethernet. The write speeds are ~97MB/sec which I am very pleased with.

Lastly, I try to use the same program to update the same files onto "fishpool" (also from the Win10 computer) but the speeds drop to ~16MB/sec

"fishpool" is a single HDD while "fishpond" is a collection of drives, so that may be part of it, but I suspect that something else is going on I don't understand yet.

Also, some of my drives are connected to an HBA (flashed to IT mode) and others are connected to the motherboard SATA port, but I don't know if this could explain the fivefold difference in write speeds.

Does anyone have any suggestions on how I can figure out why the writes to one pool are so much slower than the other pool?

 

[gobygoby]

 

[Etorix]

So the pool with three times more data vdevs and a (NVMe?) special vdev performs much better than a single drive. And I suspect that the file sync app makes heavy use of the metadata in the special vdev.

Where's the surprise?

 

[gobygoby]

"fishpond" does have (2) 200GB SSDs for metadata, but I thought those fusion pool drives were mainly used for small files.

My most recent sync operations is going to take 14 hours on "fishpool" and contains 723 GB, and very few of them are small files.

So I don't think the SSDs can explain the difference in write speeds.

 

[gobygoby]

Is there a way to measure the write speeds to the two pools from within Truenas? (to remove potential network or software bottlenecks)

I've seen "zpool iostat" on the forums, but I'm not sure if that applies here or how to interpret the results.

 

[AlexGG]

My most recent sync operations is going to take 14 hours on "fishpool" and contains 723 GB, and very few of them are small files.

So I don't think the SSDs can explain the difference in write speeds.

The SSDs are used not only for small files, but also for the metadata, of which ZFS has quite an amount, and which is also heavily used by the sync.

 

[Patrick M. Hausen]

Is there a way to measure the write speeds to the two pools from within Truenas? (to remove potential network or software bottlenecks)

There's iozone available on TrueNAS. Don't have much experience but would suggest something like

Code:

# prepare scratch dataset
zfs create <pool>/scratch
zfs set compression=off <pool>/scratch

# run benchmark with async writes
cd /mnt/<pool>/scratch
iozone -a -b results.xls

# run benchmark with ssync writes
cd /mnt/<pool>/scratch
iozone -a -o -b results.xls

# remove scratch dataset
cd
zfs destroy <pool>/scratch