Specs TrueNas Z-35
CPU 2x Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60
RAM 262073MB
Network
GUI/WEB/SSH 1Gbit
ISCSI 10GBit -> Cisco 4500 Fiber Switch -> Cisco UCS
Pool
Encrypted
Default LZ4 compression
de-duplication disabled
42 x 2 Drive Vdevs 4TB 7200 rpm drives
5 Hot Spares
3 x 8GB SSD's for ZIL in stripe
3 x 400GB SSD's for L2ARC
Environment
10 vmware hosts and 85 windows/linux VMs attached via ISCSI datastores on the vmware hosts.
15.5TB used of 151TB
We are receiving a large amount of data move aborts when doing a storage vmotion between datastores. As we are trying to split our servers from 1 lun each host to multiple luns to provide more room queue depth wise.
I performed a ctladm dumpooa when this was happening and was greeted with quite a few Blocked: read(16) entries with high latency's coming from the lun we are migrating off of.
Disk usage has not gone over 40% and CPU does not exceed 50% during this time.
LUN 0 tag 0x135e87 BLOCKED: READ(16). CDB: (410 ms)
LUN 0 tag 0x135e88 BLOCKED: READ(16). CDB: (410 ms)
LUN 0 tag 0x135e89 BLOCKED: READ(16). CDB: (410 ms)
LUN 0 tag 0x135e8a BLOCKED: READ(16). CDB: (390 ms)
LUN 0 tag 0x135e8b BLOCKED: READ(16). CDB: (390 ms)
LUN 0 tag 0x135e8c BLOCKED: READ(16). CDB: (390 ms)
LUN 0 tag 0x135e8d BLOCKED: READ(16). CDB: (390 ms)
LUN 0 tag 0x135e8e BLOCKED: READ(16). CDB: (390 ms)
LUN 0 tag 0x135e8f BLOCKED: READ(16). CDB: (390 ms)
LUN 0 tag 0x135e90 BLOCKED: READ(16). CDB: (390 ms)
LUN 0 tag 0x135e91 BLOCKED: READ(16). CDB: (390 ms)
LUN 0 tag 0x135e93 BLOCKED: READ(16). CDB: (390 ms)
LUN 0 tag 0x135e94 BLOCKED: READ(16). CDB: (390 ms)
LUN 0 tag 0x135e95 BLOCKED: READ(16). CDB: (390 ms)
LUN 0 tag 0x135e96 BLOCKED: READ(16). CDB: (390 ms)
LUN 0 tag 0x135e97 BLOCKED: READ(16). CDB: (390 ms)
LUN 0 tag 0x135e98 BLOCKED: READ(16). CDB: (390 ms)
LUN 0 tag 0x135e99 BLOCKED: READ(16). CDB: (390 ms)
LUN 0 tag 0x135e9a BLOCKED: READ(16). CDB: (390 ms)
LUN 0 tag 0x135e9c BLOCKED: READ(16). CDB: (370 ms)
LUN 0 tag 0x135e9d BLOCKED: READ(16). CDB: (370 ms)
LUN 0 tag 0x135e9e BLOCKED: READ(16). CDB: (370 ms)
LUN 0 tag 0x135e9f BLOCKED: READ(16). CDB: (370ms)
LUN 0 tag 0x135ea0 BLOCKED: READ(16). CDB: (370 ms)
LUN 0 tag 0x135ea1 BLOCKED: READ(16). CDB: (370 ms)
LUN 0 tag 0x135ea2 BLOCKED: READ(16). CDB: (370 ms)
LUN 0 tag 0x135ea3 BLOCKED: READ(16). CDB: (370 ms)
LUN 0 tag 0x135ea4 BLOCKED: READ(16). CDB: (360 ms)
LUN 0 tag 0x135ea5 BLOCKED: READ(16). CDB: (360 ms)
LUN 0 tag 0x135ea6 BLOCKED: READ(16). CDB: (360 ms)
LUN 0 tag 0x135ea7 BLOCKED: READ(16). CDB: (360 ms)
LUN 0 tag 0x135ea9 BLOCKED: READ(16). CDB: (360 ms)
LUN 0 tag 0x135eaa BLOCKED: READ(16). CDB: (360 ms)
LUN 0 tag 0x135eab BLOCKED: READ(16). CDB: (360 ms)
LUN 0 tag 0x135eac BLOCKED: READ(16). CDB: (360 ms)
LUN 0 tag 0x135ead BLOCKED: READ(16). CDB: (350 ms)
LUN 0 tag 0x135eae BLOCKED: READ(16). CDB: (350 ms)
LUN 0 tag 0x135eaf BLOCKED: READ(16). CDB: (350 ms)
LUN 0 tag 0x135eb0 BLOCKED: READ(16). CDB: (350 ms)
LUN 0 tag 0x135eb2 BLOCKED: READ(16). CDB: (350 ms)
LUN 0 tag 0x135eb3 BLOCKED: READ(16). CDB: (350 ms)
LUN 0 tag 0x135eb4 BLOCKED: READ(16). CDB: (350 ms)
LUN 0 tag 0x135eb5 BLOCKED: READ(16). CDB: (350 ms)
LUN 0 tag 0x135eb6 BLOCKED: READ(16). CDB: (340 ms)
LUN 0 tag 0x135eb7 BLOCKED: READ(16). CDB: (340 ms)
LUN 0 tag 0x135eb8 BLOCKED: READ(16). CDB: (340 ms)
LUN 0 tag 0x135eb9 BLOCKED: READ(16). CDB: (340 ms)
Any ideas would be greatly appreciated.
CPU 2x Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60
RAM 262073MB
Network
GUI/WEB/SSH 1Gbit
ISCSI 10GBit -> Cisco 4500 Fiber Switch -> Cisco UCS
Pool
Encrypted
Default LZ4 compression
de-duplication disabled
42 x 2 Drive Vdevs 4TB 7200 rpm drives
5 Hot Spares
3 x 8GB SSD's for ZIL in stripe
3 x 400GB SSD's for L2ARC
Environment
10 vmware hosts and 85 windows/linux VMs attached via ISCSI datastores on the vmware hosts.
15.5TB used of 151TB
We are receiving a large amount of data move aborts when doing a storage vmotion between datastores. As we are trying to split our servers from 1 lun each host to multiple luns to provide more room queue depth wise.
I performed a ctladm dumpooa when this was happening and was greeted with quite a few Blocked: read(16) entries with high latency's coming from the lun we are migrating off of.
Disk usage has not gone over 40% and CPU does not exceed 50% during this time.
LUN 0 tag 0x135e87 BLOCKED: READ(16). CDB: (410 ms)
LUN 0 tag 0x135e88 BLOCKED: READ(16). CDB: (410 ms)
LUN 0 tag 0x135e89 BLOCKED: READ(16). CDB: (410 ms)
LUN 0 tag 0x135e8a BLOCKED: READ(16). CDB: (390 ms)
LUN 0 tag 0x135e8b BLOCKED: READ(16). CDB: (390 ms)
LUN 0 tag 0x135e8c BLOCKED: READ(16). CDB: (390 ms)
LUN 0 tag 0x135e8d BLOCKED: READ(16). CDB: (390 ms)
LUN 0 tag 0x135e8e BLOCKED: READ(16). CDB: (390 ms)
LUN 0 tag 0x135e8f BLOCKED: READ(16). CDB: (390 ms)
LUN 0 tag 0x135e90 BLOCKED: READ(16). CDB: (390 ms)
LUN 0 tag 0x135e91 BLOCKED: READ(16). CDB: (390 ms)
LUN 0 tag 0x135e93 BLOCKED: READ(16). CDB: (390 ms)
LUN 0 tag 0x135e94 BLOCKED: READ(16). CDB: (390 ms)
LUN 0 tag 0x135e95 BLOCKED: READ(16). CDB: (390 ms)
LUN 0 tag 0x135e96 BLOCKED: READ(16). CDB: (390 ms)
LUN 0 tag 0x135e97 BLOCKED: READ(16). CDB: (390 ms)
LUN 0 tag 0x135e98 BLOCKED: READ(16). CDB: (390 ms)
LUN 0 tag 0x135e99 BLOCKED: READ(16). CDB: (390 ms)
LUN 0 tag 0x135e9a BLOCKED: READ(16). CDB: (390 ms)
LUN 0 tag 0x135e9c BLOCKED: READ(16). CDB: (370 ms)
LUN 0 tag 0x135e9d BLOCKED: READ(16). CDB: (370 ms)
LUN 0 tag 0x135e9e BLOCKED: READ(16). CDB: (370 ms)
LUN 0 tag 0x135e9f BLOCKED: READ(16). CDB: (370ms)
LUN 0 tag 0x135ea0 BLOCKED: READ(16). CDB: (370 ms)
LUN 0 tag 0x135ea1 BLOCKED: READ(16). CDB: (370 ms)
LUN 0 tag 0x135ea2 BLOCKED: READ(16). CDB: (370 ms)
LUN 0 tag 0x135ea3 BLOCKED: READ(16). CDB: (370 ms)
LUN 0 tag 0x135ea4 BLOCKED: READ(16). CDB: (360 ms)
LUN 0 tag 0x135ea5 BLOCKED: READ(16). CDB: (360 ms)
LUN 0 tag 0x135ea6 BLOCKED: READ(16). CDB: (360 ms)
LUN 0 tag 0x135ea7 BLOCKED: READ(16). CDB: (360 ms)
LUN 0 tag 0x135ea9 BLOCKED: READ(16). CDB: (360 ms)
LUN 0 tag 0x135eaa BLOCKED: READ(16). CDB: (360 ms)
LUN 0 tag 0x135eab BLOCKED: READ(16). CDB: (360 ms)
LUN 0 tag 0x135eac BLOCKED: READ(16). CDB: (360 ms)
LUN 0 tag 0x135ead BLOCKED: READ(16). CDB: (350 ms)
LUN 0 tag 0x135eae BLOCKED: READ(16). CDB: (350 ms)
LUN 0 tag 0x135eaf BLOCKED: READ(16). CDB: (350 ms)
LUN 0 tag 0x135eb0 BLOCKED: READ(16). CDB: (350 ms)
LUN 0 tag 0x135eb2 BLOCKED: READ(16). CDB: (350 ms)
LUN 0 tag 0x135eb3 BLOCKED: READ(16). CDB: (350 ms)
LUN 0 tag 0x135eb4 BLOCKED: READ(16). CDB: (350 ms)
LUN 0 tag 0x135eb5 BLOCKED: READ(16). CDB: (350 ms)
LUN 0 tag 0x135eb6 BLOCKED: READ(16). CDB: (340 ms)
LUN 0 tag 0x135eb7 BLOCKED: READ(16). CDB: (340 ms)
LUN 0 tag 0x135eb8 BLOCKED: READ(16). CDB: (340 ms)
LUN 0 tag 0x135eb9 BLOCKED: READ(16). CDB: (340 ms)
Any ideas would be greatly appreciated.