TrueNAS CORE 13 slower then 12 perhaps, raw disk copy seems slow, general performance seems OK

[damjank]

Hello,

I have TrueNAS Core version 13, no beta, went from 12 to 13 directly.
Anyway, I have pool-a and pool-b, pool/b is composed from 4 x 6TB HGST Ultrastar 7K6000 disks (HUS726060ALE610) for data and 1 x Huawei ES3600S V5 800GB SAS for cache. Now, I have 2 zvols shared over 10GB iSCSI working fine, no issues sofar. The VMs are responsive and thats it.

I recently purchased 4 new 14TB Toshibas and I am transferring some files from older pool-a that is made up from 3 x 4TB HGST Deskstar HDN726040ALE614. No biggie, just some old documents, nothing directly running, archive only.

I logged on to SSH session, added screen, and inside I run cp -a /mnt/pool-a/docu/* .
This is now copying but as far as I look, this is doing at about 35-50MB/s, in 40min it coped about 134GB. There is a long way to go towards about 3TB.

Now to what should I account this?are the drives pwned (no SMART errors) or I made mistake with drives themselves (no drive has ever been offline, marked as bad or whatever in the last 14 months) or there is speed difference from TNS12, initially I coped to this rig with about 135MB/s.

Any pointer or hints would be greatly appreciated. I know, that eventually it will copy but - it would be nice to know if I made complete crap from it and I can fix it or there is something else going on...

PS: the rig is a Supermicro board having Intel(R) Xeon(R) CPU E3-1220 v6 @ 3.00GHz and 48GB of ECC RAM.

Thx in advance!
rgD

 

[sretalla]

You're measuring in throughput, but you're potentially copying a large number of smaller files, so you should consider file count as a better measure of this job (and thinking about IOPS).

Also with an IOPS-heavy job like that, your pool is likely suffering from a lack of IOPS to handle it any faster.

Matadata-only L2ARC or a metadata special VDEV may be worth consideration of it's an objective to be able to copy large numbers of small files quickly.

 

[damjank]

Hm, did not think of that. Seems that ent ssd is ont even making that much of a big deal. Perhaps to invest into another to add meta vded for cache. All i know is that curently it is at stage that there are much 3-4gb files…
What even more disturbing, vms making many small writes are gling to be battling for io and/or writes….

 

[sretalla]

vms making many small writes are gling to be battling for io and/or writes

Exactly, and those will also likely be sync writes, so maybe there's some value to evaluating SLOG (if you also add some RAM... 64GB is considered a good starting point for block storage)... https://www.truenas.com/community/threads/the-path-to-success-for-block-storage.81165/