Single virtual disk in Truenas, good read very poor write performance

[eRJe]

Hi there,

I have a virtualised TrueNAS for fairly simple file hosting in my local network. The RAIDz1 composed out of 4x12TB HDD's have been in there from the beginning and performs good. I get approx 200 MB/s read and 125 MB/s write when I transfer a 32GB file between VM's. I did dd tests but I guess something is in between pushing the numbers unrealisticly up (compression is off)

root@truenas[/mnt/RAIDz-4HDD]# dd if=/dev/zero of=testfile1 bs=512k count=204800 oflag=direct
204800+0 records in
204800+0 records out
107374182400 bytes transferred in 318.558280 secs (337062915 bytes/sec) [321 MB/s]

Anyway, I'm happy with the RAIDz1, my issue is with a disk that I recently added for backups.

I created a new pool with a single-disk vdev. The disk is a new WD 4TB HDD. I want to use this disk for backing up VM's locally in the HPE, taking advantage of the fast VMX network. (I have a second system that also makes backups for redundancy).

Because I did not have a free SATA slot on the passthrough controller, I decided to add the disk as datastore to the ESXi host and add an extra hdisk to TrueNAS. Thus the disk in the vdev is a virtual disk sitting on vmfs.

When I copy a file from the RAIDz to the backup disk in the TrueNAS shell, I get only 12 MB/s write. A copy from the backup to RAIDz gives me 130 MB/s read.

Also a dd on the backup disk gives me the same results (no compression).

root@truenas[/mnt/Backup]# dd if=/dev/zero of=testfile5 bs=512k count=204800
204800+0 records in
204800+0 records out
107374182400 bytes transferred in 8355.877308 secs (12850139 bytes/sec) [12,25 MB/s]

Is this because I didn't provide this disk directly to TrueNAS but as virtual disk instead?

# zpool status
pool: Backup
state: ONLINE
config:

NAME STATE READ WRITE CKSUM
Backup ONLINE 0 0 0
gptid/df8786bb-6e6a-11ec-8e29-00505684703c ONLINE 0 0 0

errors: No known data errors

pool: RAIDz-4HDD
state: ONLINE
scan: scrub repaired 0B in 11:02:37 with 0 errors on Sun Dec 12 11:02:38 2021
config:

NAME STATE READ WRITE CKSUM
RAIDz-4HDD ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
gptid/e74c9924-df18-11e9-9fef-00505684703c ONLINE 0 0 0
gptid/e8114859-df18-11e9-9fef-00505684703c ONLINE 0 0 0
gptid/e8d4c816-df18-11e9-9fef-00505684703c ONLINE 0 0 0
gptid/e999a3d1-df18-11e9-9fef-00505684703c ONLINE 0 0 0

errors: No known data errors

pool: freenas-boot
state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(5) for details.
scan: scrub repaired 0B in 00:00:44 with 0 errors on Thu Jan 6 03:45:44 2022
config:

NAME STATE READ WRITE CKSUM
freenas-boot ONLINE 0 0 0
da0p2 ONLINE 0 0 0

errors: No known data errors

 

[Redcoat]

The disk is a new WD 4TB HDD.

Could this be an SMR drive? What is its model number?

See this from the Resource section: https://www.truenas.com/community/resources/list-of-known-smr-drives.141/

 

[eRJe]

Thanks for the hint. I wasn't aware of that at all. 'Unfortunately' after checking I can confirm its a CMR disk.

Recording Technology
CMR: WD5000AZLX, WD5000AZRZ, WD10EZEX, WD10EZRZ, WD20EZRZ, WD30EZRZ, WD40EZRZ, WD80EAZZ
SMR: WD20EZAZ, WD20EZBX, WD30EZAZ, WD40EZAZ, WD60EZAZ

I'm reconsidering if this is all worth the effort and perhaps just delete this backup store and use the main one on the 2nd server. Afterall, with 130 mb/s reading max between the VM's over a VMX network, I don't gain so much compared to the 1GB network to the other server...

 

[Redcoat]

'Unfortunately' after checking I can confirm its a CMR disk.

Thanks for the feedback. Time for a pragmatic decision - sounds like you are there!

 

[eRJe]

Yup, decided it's not worth the effort. Specially considering it's somewhat a cheesy setup anyway.

Thanks again. I now know at least which drives to avoid in the future.