TrueNas Core v12-U2 Slow Performance via iSCSI and SMB

[NightGhost123]

Hello Guys ,

It took quite a long time before i decided to post a thread about this topic that seems to elude me in every way possible ... So long story short i was using FreeNas 11 last year and was getting quite decent speeds of read and write via iSCSI and SMB , but then TrueNas Core v12 was out and i upgraded in order to see the full feature list and wanting a better system but due to my quickness i also upgraded the zpool that eventually locked me to v12 for good and from there the problems started to show .

As some background into the system i am using and the specs :

Server : HPE ProLiant DL380e Gen8

CPU : 2x Intel(R) Xeon(R) CPU E5-2450L 0 @ 1.80GHz
RAM : 192 GB RDIMM , 48 GB Online Spare usable 144 GB due to Ram Hot Swap
HDD : 14x 7200 RPM Seagate Model => ST2000DM008-2FR1
OS Disk : SanDisk Ultra Fit 30 GB Model => 4C531001520223
HBA/Controller : Smart Array P420 Controller in HBA Mode
NIC 01 : Integrated 4 Port Gbit Nic 4x 1 Gbe => Used for esxi iSCSI connection
NIC 02 : Qlogic 8250 2x SFP 10Gbe Ports => Used for iSCSI and SMB Traffic

ql0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1514
description: ql0
options=8013b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,TSO4,LINKSTATE>
ether 28:80:23:91:92:78
inet 172.250.1.101 netmask 0xffffff00 broadcast 172.250.1.255
media: Ethernet autoselect (10Gbase-SR <full-duplex>)
status: active
nd6 options=9<PERFORMNUD,IFDISABLED>
ql1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1514
description: ql1
options=8013b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,TSO4,LINKSTATE>
ether 28:80:23:91:92:7c
inet 172.250.1.102 netmask 0xffffff00 broadcast 172.250.1.255
media: Ethernet autoselect (10Gbase-SR <full-duplex>)
status: active
nd6 options=9<PERFORMNUD,IFDISABLED>
igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9014
options=8120b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER>
ether 2c:59:e5:4a:e8:dc
inet 10.250.1.101 netmask 0xffffff00 broadcast 10.250.1.255
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=9<PERFORMNUD,IFDISABLED>
igb1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9014
options=e527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether 2c:59:e5:4a:e8:dd
inet 10.250.2.102 netmask 0xffffff00 broadcast 10.250.2.255
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=9<PERFORMNUD,IFDISABLED>
igb2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9014
options=e527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether 2c:59:e5:4a:e8:de
inet 10.250.3.103 netmask 0xffffff00 broadcast 10.250.3.255
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=9<PERFORMNUD,IFDISABLED>
igb3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9014
options=e527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether 2c:59:e5:4a:e8:df
inet 10.250.4.104 netmask 0xffffff00 broadcast 10.250.4.255
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=9<PERFORMNUD,IFDISABLED>

I use vlan for iSCSI trafic on the Integrated Nics for esxi iSCSI traffic and no vlan for iSCSI and SMB Traffic for the qlogic card

Both adaptors were set to an MTU of 9014 connected to HPE Procurve with Jumbo Mode Enabled on vlan and global config , also the 10gbe is connected to a UNIFI switch with 9216 MTU on Global Config (Now the qlogic is set to 1514 for some tests)

Using Latest Firmware for all the components , HPE SPP Gen8 2019 version

The problems with performance started to show just after the upgrade to TrueNas Core v12 and zPool update ... usualy on the v11 the speeds i got on random reads writes were above 250 - 300 mb/s while using iSCSI on the esxi with Round Rubin and 1 IOPS path Switch now i get ... almost 0 ..

Example if i try to un suspend a vm let's say with 16 gb of ram it takes about 40 mins to 1 hour ... but if i try to suspend it works as intended all of the 4 gigabit interfaces jump at 110 mb/s so 440 mb/s ...

The SMB is the same , now i get about 11 - 20 mb/s ... it used to go to about 900 - 950 mb/s write or read ...

I hoped that the update to U2 will fix the issue but seems that the same issue is present as per this bugs https://jira.ixsystems.com/browse/NAS-107593 , https://jira.ixsystems.com/browse/NAS-108081 , https://jira.ixsystems.com/browse/N...ssuetabpanels:comment-tabpanel#comment-123102 .

The thing is quite weird if i run Disk Mark it gets almost perfect scores as if nothing is wrong ...

In IDLE mode or just hitting a refresh on desktop within an esxi 6.7 U3 vm shows like this ...

Also i have to mention that i reinstalled TrueNas Core v12 from new iso so that there were no issues with the upgrade and same issue ..

Below is the zPool Setup :

root@freenas[~]# zpool status -v
pool: Storage-Pool
state: ONLINE
scan: scrub in progress since Fri Feb 12 16:16:40 2021
10.5T scanned at 2.82G/s, 3.65T issued at 1002M/s, 11.2T total
0B repaired, 32.55% done, 02:11:55 to go
config:

NAME STATE READ WRITE CKSUM
Storage-Pool ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/74d63f90-dd7c-11ea-86d8-2c59e54ae8dc ONLINE 0 0 0
gptid/76cce573-dd7c-11ea-86d8-2c59e54ae8dc ONLINE 0 0 0
gptid/76dc2a3f-dd7c-11ea-86d8-2c59e54ae8dc ONLINE 0 0 0
gptid/79b323dd-dd7c-11ea-86d8-2c59e54ae8dc ONLINE 0 0 0
gptid/796f6cc6-dd7c-11ea-86d8-2c59e54ae8dc ONLINE 0 0 0
gptid/79a824cf-dd7c-11ea-86d8-2c59e54ae8dc ONLINE 0 0 0
gptid/7a52c699-dd7c-11ea-86d8-2c59e54ae8dc ONLINE 0 0 0
gptid/78d255b6-dd7c-11ea-86d8-2c59e54ae8dc ONLINE 0 0 0
gptid/7b46b25a-dd7c-11ea-86d8-2c59e54ae8dc ONLINE 0 0 0
gptid/7b825580-dd7c-11ea-86d8-2c59e54ae8dc ONLINE 0 0 0
gptid/495f71f5-30b4-11eb-bdf7-288023919278 ONLINE 0 0 0
gptid/7d7b974b-dd7c-11ea-86d8-2c59e54ae8dc ONLINE 0 0 0
gptid/7d8645f0-dd7c-11ea-86d8-2c59e54ae8dc ONLINE 0 0 0
gptid/7eaf1d84-dd7c-11ea-86d8-2c59e54ae8dc ONLINE 0 0 0

errors: No known data errors

pool: boot-pool
state: ONLINE
config:

NAME STATE READ WRITE CKSUM
boot-pool ONLINE 0 0 0
da14p2 ONLINE 0 0 0

errors: No known data errors

Most of the other settings on the NAS are default by Auto Tune (also tried without them same issues)

There does not seem to be a network issue since it hits full speed while reading and writing vm's from the ARC Cache ...

Do you guys have any ideea ?

This is a production env and it's quite unacceptable this kind of performance issue ...

Thank you in advance !

 

[HoneyBadger]

Firstly, welcome to the forums.

Unfortunately I'm about to lay a lot of bad news on you. Please don't shoot the messenger, but you've got a combination of hardware/configuration issues.

Bear with me here, this is going to be a long post.

HDD : 14x 7200 RPM Seagate Model => ST2000DM008-2FR1
Those are SMR (shingled) drives. I'm not sure if you're familiar with them, but they offer extremely bad performance in RAID configurations. There is no real "band-aid" for this, and your ultimate, very ugly fix is replacing the drives with CMR drives. 2TB SAS drives are cheap though, and you could probably pull this off for a few hundred dollars if you're willing to use refurbished/pulled drives.

Single wide raidz2 vdev for block data/VMFS
This is hurting your performance and likely your space-efficiency quite a bit. ZFS performance is tied much more closely to "number of vdevs" rather than "width of vdev" - you have 14 drives in a single RAIDZ2. Seven drives in two-way mirrors will deliver roughly 7x the random I/O numbers.
Check here for some more details: https://www.truenas.com/community/r...and-why-we-use-mirrors-for-block-storage.112/

ESXi/VM workload on async/no SLOG (danger - your data is at risk)
There's no way that sync writes on a 14-wide SMR Z2 would have been fast enough even on fresh disks to have your iSCSI ZVOLs be running with sync=always - which means that the last few seconds/MB's of data being written to your VMFS datastores are only being written to the RAM on the FreeNAS machine. If you suffer a power failure or system crash, that data is gone. For many SMB workloads this isn't a problem (just copy the file again) but for VMFS, losing important metadata can mean corrupt VMs or completely unmountable datastores. You need to get an SLOG device in there (Optane DC P-series or a Radian RMS-200 are my suggestions for NVMe, WD DC SS530 for SAS)

Those are the big three things that need to be addressed. Some minor hardware issues are below.
HBA/Controller : Smart Array P420 Controller in HBA Mode
While it "works" in HBA mode, the FreeBSD driver for the HP SmartArray cards (ciss) is significantly less mature than the one for the LSI SAS family (mps/mpr) so you may experience performance issues. Swapping to an LSI card might be a worthwhile exercise.

RAM : 192 GB RDIMM , 48 GB Online Spare usable 144 GB due to Ram Hot Swap
RAM sparing is neat from a functional perspective, but have you actually tried this to ensure that it's stable on your board under FreeNAS (without running live data, of course?) You're also leaving 48GB of read-cache on the table from a performance perspective.

CPU : 2x Intel(R) Xeon(R) CPU E5-2450L 0 @ 1.80GHz
Per-core speed is rather low here. Swapping these for faster units might improve things, but you've got much bigger fish to fry here.

This is a production env and it's quite unacceptable this kind of performance issue

Right now I'm genuinely concerned for the safety and stability of your environment, but happy that as it's "production" you will hopefully be able to pull some dollars behind this to fix it. Fixing the safety/stability will do a lot to improve performance; there's fine-tuning that can be done afterwards of course, but safety first, speed second.

Here's the ugly part though. You can add an SLOG with a brief bit of downtime to slot in an NVMe-based SSD (Optane is your friend, or a used Radian RMS-200) - you can hot-swap your SMR Barracudas with Barracuda Pro, WD Red Plus, or even some Constellation ES SAS drives on the cheap, and you can even use SLOG-add downtime to change CPUs and RAM config.

But there is no way to change your RAIDZ2 into mirrors or a faster pool config without migrating all the data off, destroying the pool, and building it again. Do you have sufficient secondary/backup storage to do this, or would it require a long cycle of "back up everything, take a business/production downtime, rebuild, and restore"?

Again; I apologize if this seems like a "harsh welcome" - but the reality is that your system is not in a good state, and I just want to see it get better.

 

[NightGhost123]

Hello and Thank you ,

I was Kind of expecting this response :)) so no warries , as for your answers please see below :

HDD : 14x 7200 RPM Seagate Model => ST2000DM008-2FR1
Those are SMR (shingled) drives. I'm not sure if you're familiar with them, but they offer extremely bad performance in RAID configurations. There is no real "band-aid" for this, and your ultimate, very ugly fix is replacing the drives with CMR drives. 2TB SAS drives are cheap though, and you could probably pull this off for a few hundred dollars if you're willing to use refurbished/pulled drives.

Yes i am aware , but this still does not explain what it worked perfectly in the past with version 11.x and after the upgrade the performance was no ware to be found . Also i have to add that there are no more the 30 vm's running and just 2 of them have quite big workloads like a visual svn on a windows server and of course the vCenter Server . There is also a Veeam server that in v 11.xx worked at full speed now it can't even get to 11 - 12 mb throughput . That being said yes better drives would yield better performance and i agree , but the question remains why did it work ok and now it's not ...

Single wide raidz2 vdev for block data/VMFS
This is hurting your performance and likely your space-efficiency quite a bit. ZFS performance is tied much more closely to "number of vdevs" rather than "width of vdev" - you have 14 drives in a single RAIDZ2. Seven drives in two-way mirrors will deliver roughly 7x the random I/O numbers.

Actually though there is a raidz2 it's compliant in the sense that it has 4 parity drives and 10 data drives and was automatically done by freenas in this config . Please forgive me if I misunderstood this ...

ESXi/VM workload on async/no SLOG (danger - your data is at risk)
There's no way that sync writes on a 14-wide SMR Z2 would have been fast enough even on fresh disks to have your iSCSI ZVOLs be running with sync=always - which means that the last few seconds/MB's of data being written to your VMFS datastores are only being written to the RAM on the FreeNAS machine. If you suffer a power failure or system crash, that data is gone. For many SMB workloads this isn't a problem (just copy the file again) but for VMFS, losing important metadata can mean corrupt VMs or completely unmountable datastores. You need to get an SLOG device in there (Optane DC P-series or a Radian RMS-200 are my suggestions for NVMe, WD DC SS530 for SAS)

Yes true , but we use default , and the drives don't seem to take a punishment from that .... As shown Below :

dT: 1.016s w: 1.000s
L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name
1 33 0 0 0.0 33 287 2.9 9.6| da0
1 45 0 0 0.0 45 323 0.4 1.8| da1
1 37 0 0 0.0 37 299 2.3 8.6| da2
1 33 0 0 0.0 33 283 0.6 1.9| da3
1 43 0 0 0.0 43 303 0.3 1.5| da4
1 43 0 0 0.0 43 307 0.4 1.7| da5
1 45 0 0 0.0 45 323 0.3 1.6| da6
1 36 0 0 0.0 36 295 0.5 1.6| da7
1 39 0 0 0.0 39 268 0.2 0.9| da8
0 44 0 0 0.0 43 283 0.4 27.2| da9
1 36 0 0 0.0 36 307 2.4 8.6| da10
1 46 0 0 0.0 46 327 0.4 1.8| da11
1 36 0 0 0.0 36 268 0.2 0.9| da12
0 36 0 0 0.0 35 295 0.4 25.0| da13
0 0 0 0 0.0 0 0 0.0 0.0| da14

This is how the drive are staying at the moment with 24 vm,s running .

Minor hardware issues:
HBA/Controller : Smart Array P420 Controller in HBA Mode

While it "works" in HBA mode, the FreeBSD driver for the HP SmartArray cards (ciss) is significantly less mature than the one for the LSI SAS family (mps/mpr) so you may experience performance issues. Swapping to an LSI card might be a worthwhile exercise.

Yes , but this is a problem since it's an HPE built system we can not change the card without voiding the warranty and support ... Also as a mention tried with a dell card same issue and also with an adaptec one ...

RAM : 192 GB RDIMM , 48 GB Online Spare usable 144 GB due to Ram Hot Swap
RAM sparing is neat from a functional perspective, but have you actually tried this to ensure that it's stable on your board under FreeNAS (without running live data, of course?) You're also leaving 48GB of read-cache on the table from a performance perspective.

Tested it and it works as a charm in prod and test env , used it since v9 of FreeNas and never had an issue .

CPU : 2x Intel(R) Xeon(R) CPU E5-2450L 0 @ 1.80GHz
Per-core speed is rather low here. Swapping these for faster units might improve things, but you've got much bigger fish to fry here.

CPU is at most at 30% in full load ... so yeah maybe better cpu's do a better job but they don't even hit a bottle neck ..

As for doing a full rebuild sadly it would take the array offline for more the 3-4 days ... and as of now we can not do that .

SLOG i don't think it will help since ... the problem seems to be at HDD=>Network pass ... since it worked fine in the v11 version ... and now it topples down ...

Thank you in advance :)

 

[Samuel Tai]

12 was a full replacement of ZFS from the FreeBSD in-kernel module to unification with OpenZFS used in ZFS on Linux. It would stand to reason this new module will behave differently from that in 11. If you haven’t upgraded your pool, your best option is to boot back into 11, and address the disk and pool issues @HoneyBadger identified.

If you already upgraded your pool, then you’re living on borrowed time, and it’s only a matter of when, not if, you’ll be facing a complete rebuild under duress.

 

[NightGhost123]

12 was a full replacement of ZFS from the FreeBSD in-kernel module to unification with OpenZFS used in ZFS on Linux. It would stand to reason this new module will behave differently from that in 11. If you haven’t upgraded your pool, your best option is to boot back into 11, and address the disk and pool issues @HoneyBadger identified.

If you already upgraded your pool, then you’re living on borrowed time, and it’s only a matter of when, not if, you’ll be facing a complete rebuild under duress.

Sadly as i mention i already upgraded the pool .. so as it stands , the only thing i could think of was of the bugs i mentioned , and they seem to also happen for me on v12 U2 build ...

Thank you in advance

 

[HoneyBadger]

Yes i am aware , but this still does not explain what it worked perfectly in the past with version 11.x and after the upgrade the performance was no ware to be found . Also i have to add that there are no more the 30 vm's running and just 2 of them have quite big workloads like a visual svn on a windows server and of course the vCenter Server . There is also a Veeam server that in v 11.xx worked at full speed now it can't even get to 11 - 12 mb throughput . That being said yes better drives would yield better performance and i agree , but the question remains why did it work ok and now it's not ...

If the performance issue happened immediately after the TN12 upgrade, it might be related to drivers. 12.0-U2 does make reference to a fix for slow Chelsio and Intel network performance, but that's with the iflib driver, not igb, and you've mentioned in your original post that your speeds are fine when reading from ARC and I assume iperf testing would show the same: https://jira.ixsystems.com/browse/NAS-107593

There have been some reports of poor SMB performance in general on 12.0 - but right now I'm far more concerned about the impact of SMR/Z2/SLOG on your VMware environment.

If the performance was fine for a bit and then tanked, you're likely seeing the SMR re-shingling action coming into play now. This basically means that you'll be suffering from a "read-modify-write" cycle on the drives instead of just being able to write to free space.

What does a few screens of gstat -dp look like under a heavy workload? I can't recall if Seagate SMR exposes their reshingling operations via the delete queue.

Actually though there is a raidz2 it's compliant in the sense that it has 4 parity drives and 10 data drives and was automatically done by freenas in this config . Please forgive me if I misunderstood this ...

Your zpool status shows that Storage-Pool is backed by a single RAIDZ2 vdev with 14 drives. This is fine for large sequential data, but poor for random I/O.

Yes true , but we use default , and the drives don't seem to take a punishment from that .... As shown Below :

The drives aren't taking punishment because the I/O is being collected in RAM, and RAM only. Your ESXi hosts believe that data is safe on disk, but it's only in the RAM on your FreeNAS machine - if it crashes or loses power and hasn't flushed that data to disk, it's gone.

Yes , but this is a problem since it's an HPE built system we can not change the card without voiding the warranty and support ... Also as a mention tried with a dell card same issue and also with an adaptec one ...

The P420 in HBA mode will work; but I'm fairly confident you'd be better off with an LSI, even an older SAS2008.

Tested (hot swap RAM) and it works as a charm in prod and test env , used it since v9 of FreeNas and never had an issue .

Glad to hear! Make sure your iLO/IPMI is set up to alert you via email/SNMP on failing modules since it may mask them from the OS.

CPU is at most at 30% in full load ... so yeah maybe better cpu's do a better job but they don't even hit a bottle neck ..

The overall CPU percentage is an aggregate - it's simply averaging load across all cores, and your CPU setup has 16c/32t assuming you haven't disabled hyperthreading. A CPU with fewer but faster cores might offer a performance boost in single-threaded or lightly-threaded workloads, and unless things have changed significantly both iSCSI and SMB fit that bill.

As for doing a full rebuild sadly it would take the array offline for more the 3-4 days ... and as of now we can not do that .

Please don't take it lightly when I suggest that you need to start making plans for this. It's better to do it in a controlled environment than under duress or in response to a failure.

SLOG i don't think it will help since ... the problem seems to be at HDD=>Network pass ... since it worked fine in the v11 version ... and now it topples down ...

SLOG is tied in with data safety as well as performance here; the idea is that async writes are unsafe. Sync writes are safe, but often end up reducing performance. Adding an SLOG is a way to regain some or most of that performance back.

 

[anodos]

If the performance issue happened immediately after the TN12 upgrade, it might be related to drivers. 12.0-U2 does make reference to a fix for slow Chelsio and Intel network performance, but that's with the iflib driver, not igb, and you've mentioned in your original post that your speeds are fine when reading from ARC and I assume iperf testing would show the same: https://jira.ixsystems.com/browse/NAS-107593

There have been some reports of poor SMB performance in general on 12.0 - but right now I'm far more concerned about the impact of SMR/Z2/SLOG on your VMware environment..

The slow SMB performance issues that users were seeing were specifically related to the aforementioned driver problems. We have a lot of SMB users in the community and so it's easy to ID an OS/driver issue as an application issue.

 

[NightGhost123]

The slow SMB performance issues that users were seeing were specifically related to the aforementioned driver problems. We have a lot of SMB users in the community and so it's easy to ID an OS/driver issue as an application issue.

Yes that is true but i have already updated to v12-U2 and it didn't fix the problem ... same as before .

Thank you in advance .

 

[NightGhost123]

Hello ,

And thank you for your reply , please see below my answers :

he P420 in HBA mode will work; but I'm fairly confident you'd be better off with an LSI, even an older SAS2008.

Agreed , but for the moment that is not possible in changing the device , also it worked before with no issues on older versions ...

Your zpool status shows that Storage-Pool is backed by a single RAIDZ2 vdev with 14 drives. This is fine for large sequential data, but poor for random I/O.

Yes True , but also keep in mind that we do no have that much iops coming in , also it would be more the ok to deliver at leat 2k iops ... and since this is not the case ... and it was on older versions ...

What does a few screens of gstat -dp look like under a heavy workload? I can't recall if Seagate SMR exposes their reshingling operations via the delete queue.

Under load it looks likte this :

dT: 1.064s w: 1.000s
L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name
1 3 0 0 0.0 2 15 429.5 0 0 0.0 84.2| da0
1 3 0 0 0.0 2 11 187.5 0 0 0.0 40.0| da1
0 4 0 0 0.0 2 15 169.4 0 0 0.0 37.8| da2
7 3 0 0 0.0 0 0 0.0 0 0 0.0 99.8| da3
0 3 0 0 0.0 1 11 793.4 0 0 0.0 81.5| da4
1 3 0 0 0.0 2 11 293.2 0 0 0.0 59.9| da5
1 2 0 0 0.0 1 11 231.1 0 0 0.0 26.5| da6
1 3 0 0 0.0 2 11 268.4 0 0 0.0 55.2| da7
0 4 0 0 0.0 2 15 335.2 0 0 0.0 68.8| da8
1 2 0 0 0.0 1 11 807.2 0 0 0.0 80.6| da9
1 3 0 0 0.0 2 15 397.2 0 0 0.0 79.4| da10
0 3 0 0 0.0 1 11 629.6 0 0 0.0 65.2| da11
0 2 0 0 0.0 1 4 0.2 0 0 0.0 0.8| da12
0 3 0 0 0.0 1 11 884.2 0 0 0.0 88.7| da13
0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da14

Glad to hear! Make sure your iLO/IPMI is set up to alert you via email/SNMP on failing modules since it may mask them from the OS.

Done :)

The overall CPU percentage is an aggregate - it's simply averaging load across all cores, and your CPU setup has 16c/32t assuming you haven't disabled hyperthreading. A CPU with fewer but faster cores might offer a performance boost in single-threaded or lightly-threaded workloads, and unless things have changed significantly both iSCSI and SMB fit that bill.

Yes true , but average core rate is 20-30% under full load , and no core has actually hit more then 58% in full load .

Please don't take it lightly when I suggest that you need to start making plans for this. It's better to do it in a controlled environment than under duress or in response to a failure.

Yes i agree , but for the moment it would be quite hard to do so , we are in the process of adding a new TrueNas Core but it's far from finished as of now .

SLOG is tied in with data safety as well as performance here; the idea is that async writes are unsafe. Sync writes are safe, but often end up reducing performance. Adding an SLOG is a way to regain some or most of that performance back.

Agree , but as i specified it worked in the past (I don't want to be weird user that says that everything worked before he deleted a file :)) , but in this case older versions of FreeNas worked fine without SLOG) we also have 4 UPS on 18.4 kwh so ... the Nas can stay online for more the 4 days without any power input from network , also 2 redundant supplies so for the moment i assume the data is ok'ish to say the least .

Thank you in advance

 

[NightGhost123]

Hello ,

If the performance issue happened immediately after the TN12 upgrade, it might be related to drivers. 12.0-U2 does make reference to a fix for slow Chelsio and Intel network performance, but that's with the iflib driver, not igb, and you've mentioned in your original post that your speeds are fine when reading from ARC and I assume iperf testing would show the same: https://jira.ixsystems.com/browse/NAS-107593

There have been some reports of poor SMB performance in general on 12.0 - but right now I'm far more concerned about the impact of SMR/Z2/SLOG on your VMware environment.

If the performance was fine for a bit and then tanked, you're likely seeing the SMR re-shingling action coming into play now. This basically means that you'll be suffering from a "read-modify-write" cycle on the drives instead of just being able to write to free space.

What does a few screens of gstat -dp look like under a heavy workload? I can't recall if Seagate SMR exposes their reshingling operations via the delete queue.

After a few more tests seems that the read performance is almost unusable as seen below ...

This is while trying to un suspend a few vm's ... While i have done some more research it seems that SMR drives should have no problem at read speeds but only have issues with writes , and in terms of that i do not see any issues ... but with reads it seems is horribly slow

Also the performance on the SMB share starts at 200 mb and drops to 1-2 mb/s ...

Here are some iperf stats for the interfaces running SMB and iSCSI over 10gb :

Here are some top info :

system is allmost idle from CPU stand Point :

while the HDD's are dead ...

This is quite weird ... i intend on assuming that there may be an issue upgrading the pool from v11 to v12 ... But i am not sure ... also if i do remember , it did seem that performance died just after i upgraded to v12 and made the zPool update as well , and since i couldn't find anything i assumed it would have been due to those bugs with drivers ...

Below is also a zilstat :

Also please let me know if i need to share anything else .

Thank you in advance

 

[HoneyBadger]

Your zilstat output indicates that you aren't doing any sync writes, which again means that your data-in-flight is at risk. You have UPSes and redundant power, but that won't help you if your HBA decides to let its magic blue smoke out. If this is an acceptable risk then that's fine, just as long as you're taking it on with full awareness.

The symptoms here are lining up with SMR reshingling - this is done entirely "on-drive" which doesn't present any workload to your CPU, but instead just shows up as the drives being incredibly slow to respond.

When an SMR drive is "clean" it can easily de-stage its CMR "cache zone" to these clean SMR zones with a minimum of delay; but once it's time to overwrite an SMR zone, you're now performing a doing a "read modify write" operation. SMR zones are 256MB - when you're a consumer writing multi-gigabyte backups or media files, it's a tolerable delay. But when you have to do a "read modify write" to change a few kilobytes inside a VMDK, now it's a problem. Even more so if the changed data in that VMDK is spread across multiple SMR zones.

 

[sretalla]

raidz2 it's compliant in the sense that it has 4 parity drives and 10 data drives

I don't think you're counting correctly... RAIDZ2 has 2 parity drives, leaving 12 Data drives.

The best practice would have RAIDZ2 at a maximum of 12 wide, so you're already over that recommendation (so in my opinion, not compliant).