Hey everybody! So ive been having some problems with read speeds when running a verification job (100% read) from a Commvault backup system. Read speeds on the job will be an acceptable ~170MB/sec for about 30-45 minutes and then something i can't figure out happens.. speeds drop to around ~20MB/sec.
Checking out gstat, drive usage is optimal and then when the speeds tank i get drives that have a very high MS/R, killing performance. Its completely random which drives they are, and how many it consists of.. sometimes its 4.. sometimes 6, the only thing i have noticed is they tend to be in the same vdev. Stopping all services that cause disk activity returns the read speeds to normal.. again for about 30-45 minutes.
Pool Specs:
3x RAIDZ2 vdevs striped , all vdevs consist of 11 3TB WD RED drives.
41 TiB zvol iSCSI extent, about 60% of pool size.. zpool get fragmentation reports 13%.
Data on the vdevs are balanced, and iostat shows reads are even across vdevs.
SLOG is Intel P3700 200GB.
System Specs:
SUPERMICRO X9DRH-iTF , 2X Intel Xeon E5-2609V2
128GB ECC memory
2x LSI 9207 running firmware/driver P16
Intel X540 10GbE NIC
Supermicro SC847 E26
running FreeNAS-9.3-STABLE-201506042008
Autotune disabled, no fancy tweaking or sysctl's.
The Backup server has the same X540, running Server 2008 R2. iSCSi is connecting with MS iSCSI initiator. using MPIO with seperate subnets and VLAN's. Drive is NTFS and 64K sectors, as the backup files tend to be large. I can give more info about this but i really think the problem is on the storage side.
Has anyone else had problems with drives "choking" after prolonged reads? I've attached screenshots of gstat running 10 minutes in, and what happens after 30-45 minutes.
Checking out gstat, drive usage is optimal and then when the speeds tank i get drives that have a very high MS/R, killing performance. Its completely random which drives they are, and how many it consists of.. sometimes its 4.. sometimes 6, the only thing i have noticed is they tend to be in the same vdev. Stopping all services that cause disk activity returns the read speeds to normal.. again for about 30-45 minutes.
Pool Specs:
3x RAIDZ2 vdevs striped , all vdevs consist of 11 3TB WD RED drives.
41 TiB zvol iSCSI extent, about 60% of pool size.. zpool get fragmentation reports 13%.
Data on the vdevs are balanced, and iostat shows reads are even across vdevs.
SLOG is Intel P3700 200GB.
System Specs:
SUPERMICRO X9DRH-iTF , 2X Intel Xeon E5-2609V2
128GB ECC memory
2x LSI 9207 running firmware/driver P16
Intel X540 10GbE NIC
Supermicro SC847 E26
running FreeNAS-9.3-STABLE-201506042008
Autotune disabled, no fancy tweaking or sysctl's.
The Backup server has the same X540, running Server 2008 R2. iSCSi is connecting with MS iSCSI initiator. using MPIO with seperate subnets and VLAN's. Drive is NTFS and 64K sectors, as the backup files tend to be large. I can give more info about this but i really think the problem is on the storage side.
Has anyone else had problems with drives "choking" after prolonged reads? I've attached screenshots of gstat running 10 minutes in, and what happens after 30-45 minutes.
Attachments
Last edited: