Hello,
Let me start by saying I am still new to True/FreeNAS. This environment was setup by a former employee and I am merely living in their world.
Anyways onto the issue. Everything has been running smooth for years until last Friday. A coworker noticed that the HDD Pool reached 89% full when performance took a big hit. We found several .zstream files that were used to move snapshots around from environment to environment, after removing that and some VMware snapshots the disk came down to 67% free however performance has not increased. We have checked so many different things but nothing seems to help.
The systems specs are as follows;
1 single vdev for one zpool containing 11x8TB 7200RPM disks. RAIDZ2
The second pool is all the SSDs (900gbx12) split into two RAIDZ2 groups of 6 disks each
The connect between the TrueNAS and ESXi is iSCSI on a 10Gbe link.
Using iPERF3 I can get:
On the ESXi/Vcenter (6.7 latest) side I am seeing the dqlen get stuck at 32 and build a queue as well as the datastore almost constantly dropping.
I have done my best to provide as much info as possible but I am sure I am missing a huge amount that you need in order to help me figure out whats going on I will do my best to provide additional information as quickly and accurately as possible. Again I am still new to this.
Let me start by saying I am still new to True/FreeNAS. This environment was setup by a former employee and I am merely living in their world.
Anyways onto the issue. Everything has been running smooth for years until last Friday. A coworker noticed that the HDD Pool reached 89% full when performance took a big hit. We found several .zstream files that were used to move snapshots around from environment to environment, after removing that and some VMware snapshots the disk came down to 67% free however performance has not increased. We have checked so many different things but nothing seems to help.
The systems specs are as follows;
1 single vdev for one zpool containing 11x8TB 7200RPM disks. RAIDZ2
Code:
zpool list HDD NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT HDD 80T 50.3T 29.7T - - 10% 62% 1.00x ONLINE /mnt
The second pool is all the SSDs (900gbx12) split into two RAIDZ2 groups of 6 disks each
Code:
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT ssd 10.4T 2.07T 8.37T - - 27% 19% 1.00x ONLINE /mnt
The connect between the TrueNAS and ESXi is iSCSI on a 10Gbe link.
Using iPERF3 I can get:
- Host1 <> Host2 = 10gbe
- Host1 -> NAS = sub 1gbe
- Host2 -> NAS = sub 1gbe
- NAS -> Host1/2 = 10gbe
- Intel(R) Xeon(R) CPU E3-1220 v5 @ 3.00GHz
- 64GB RAM
- TrueNAS-12.0-U4
On the ESXi/Vcenter (6.7 latest) side I am seeing the dqlen get stuck at 32 and build a queue as well as the datastore almost constantly dropping.
I have done my best to provide as much info as possible but I am sure I am missing a huge amount that you need in order to help me figure out whats going on I will do my best to provide additional information as quickly and accurately as possible. Again I am still new to this.
