SOLVED ESXi host hangs when cloning a VM on iSCSI Datastore, SCSI Sense Codes in console. Output attached.

Status
Not open for further replies.
Joined
Apr 11, 2013
Messages
7
Hi,

I have faced a strange issue as of the past few days where any time I perform a clone activity on my ESXi host (Dell r710) it will essentially stun the entire host for 700-900 seconds.

Here is a bit more background on my environment:

Server: Dell r710
NAS: Infortrend EonNAS PRO 850-2 running FreeNAS-9.10.2-U3 (e1497f269)

Both systems are directly connected (no switch) using DAC cables between an x520-da2 on each system.

The 2 x Crucial MX300 525GB SSDs (shown below) are together to make a 1050GB iSCSI Datastore, this is the Datastore that is causing issues. I am cloning to/from the same datastore.

I have never had any issues with this previously, and only recently started having issues now. I can power on VMs and do regular tasks with no problems, it's only when I am cloning a VM that it stuns the host and eventually fails the clone operation.

I have noticed that there is a lot of SCSI Sense Code spam on the physical console monitor connected to my NAS, which is something I've not seen before; these will only present themselves when I attempt a clone operation.

I have also noticed that the 10GbE interfaces (ix0, ix1) are flipping a bit.

Attached are some relevant logs.

Here is some extra output that may be useful:

Code:
[root@freenas] /var/log# zpool status
  pool: freenas-boot
state: ONLINE
  scan: scrub repaired 0 in 0h5m with 0 errors on Wed Sep 13 03:19:12 2017
config:

	NAME										  STATE	 READ WRITE CKSUM
	freenas-boot								  ONLINE	   0	 0	 0
	  gptid/a648eab9-bac9-11e6-b688-888888888888  ONLINE	   0	 0	 0

errors: No known data errors

  pool: ssd1050
state: ONLINE
  scan: scrub repaired 0 in 0h28m with 0 errors on Sun Aug 20 00:28:50 2017
config:

	NAME											STATE	 READ WRITE CKSUM
	ssd1050										 ONLINE	   0	 0	 0
	  gptid/78f597c9-c4ef-11e6-9802-888888888888	ONLINE	   0	 0	 0
	  gptid/7947b50b-c4ef-11e6-9802-888888888888	ONLINE	   0	 0	 0
	logs
	  mirror-2									  ONLINE	   0	 0	 0
		gptid/79973781-c4ef-11e6-9802-888888888888  ONLINE	   0	 0	 0
		gptid/79e4dbde-c4ef-11e6-9802-888888888888  ONLINE	   0	 0	 0

errors: No known data errors

  pool: ssd120
state: ONLINE
  scan: scrub repaired 0 in 0h0m with 0 errors on Sun Aug 13 00:00:05 2017
config:

	NAME										  STATE	 READ WRITE CKSUM
	ssd120										ONLINE	   0	 0	 0
	  gptid/1b1bc380-badb-11e6-9ebc-888888888888  ONLINE	   0	 0	 0

errors: No known data errors


Code:
[root@freenas] /var/log# smartctl -a -q noserial /dev/ada0
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:	 Crucial/Micron MX1/2/300, M5/600, 1100 Client SSDs
Device Model:	 Crucial_CT525MX300SSD1
Firmware Version: M0CR021
User Capacity:	525,112,713,216 bytes [525 GB]
Sector Size:	  512 bytes logical/physical
Rotation Rate:	Solid State Device
Form Factor:	  2.5 inches
Device is:		In smartctl database [for details use: -P show]
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.2, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:	Thu Sep 14 04:59:46 2017 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x03)	Offline data collection activity
					is in progress.
					Auto Offline Data Collection: Disabled.
Self-test execution status:	  (   0)	The previous self-test routine completed
					without error or no self-test has ever
					been run.
Total time to complete Offline
data collection:		 (  705) seconds.
Offline data collection
capabilities:			 (0x7b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:			(0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:		(0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine
recommended polling time:	 (   2) minutes.
Extended self-test routine
recommended polling time:	 (   6) minutes.
Conveyance self-test routine
recommended polling time:	 (   3) minutes.
SCT capabilities:			(0x0035)	SCT Status supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME		  FLAG	 VALUE WORST THRESH TYPE	  UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate	 0x002f   100   100   000	Pre-fail  Always	   -	   0
  5 Reallocate_NAND_Blk_Cnt 0x0032   100   100   010	Old_age   Always	   -	   0
  9 Power_On_Hours		  0x0032   100   100   000	Old_age   Always	   -	   6164
12 Power_Cycle_Count	   0x0032   100   100   000	Old_age   Always	   -	   18
171 Program_Fail_Count	  0x0032   100   100   000	Old_age   Always	   -	   0
172 Erase_Fail_Count		0x0032   100   100   000	Old_age   Always	   -	   0
173 Ave_Block-Erase_Count   0x0032   099   099   000	Old_age   Always	   -	   26
174 Unexpect_Power_Loss_Ct  0x0032   100   100   000	Old_age   Always	   -	   17
183 SATA_Interfac_Downshift 0x0032   100   100   000	Old_age   Always	   -	   0
184 Error_Correction_Count  0x0032   100   100   000	Old_age   Always	   -	   0
187 Reported_Uncorrect	  0x0032   100   100   000	Old_age   Always	   -	   0
194 Temperature_Celsius	 0x0022   074   059   000	Old_age   Always	   -	   26 (Min/Max 19/41)
196 Reallocated_Event_Count 0x0032   100   100   000	Old_age   Always	   -	   0
197 Current_Pending_Sector  0x0032   100   100   000	Old_age   Always	   -	   0
198 Offline_Uncorrectable   0x0030   100   100   000	Old_age   Offline	  -	   0
199 UDMA_CRC_Error_Count	0x0032   100   100   000	Old_age   Always	   -	   0
202 Percent_Lifetime_Used   0x0030   099   099   001	Old_age   Offline	  -	   1
206 Write_Error_Rate		0x000e   100   100   000	Old_age   Always	   -	   0
246 Total_Host_Sector_Write 0x0032   100   100   000	Old_age   Always	   -	   9224971207
247 Host_Program_Page_Count 0x0032   100   100   000	Old_age   Always	   -	   318853612
248 Bckgnd_Program_Page_Cnt 0x0032   100   100   000	Old_age   Always	   -	   173829129
180 Unused_Reserve_NAND_Blk 0x0033   000   000   000	Pre-fail  Always	   -	   1925
210 Success_RAIN_Recov_Cnt  0x0032   100   100   000	Old_age   Always	   -	   0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description	Status				  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Vendor (0xff)	   Completed without error	   00%	  6164		 -
# 2  Vendor (0xff)	   Completed without error	   00%	  6044		 -
# 3  Reserved (0x19)	 Aborted by host			   90%	  6425		 -
# 4  Reserved (0x18)	 Aborted by host			   80%	  6168		 -
# 5  Reserved (0x18)	 Aborted by host			   80%	  6168		 -
# 6  Reserved (0x18)	 Aborted by host			   80%	  6168		 -
# 7  Reserved (0x18)	 Aborted by host			   80%	  6168		 -
# 8  Reserved (0x19)	 Aborted by host			   90%	  6425		 -

SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
	1		0		0  Not_testing
	2		0		0  Not_testing
	3		0		0  Not_testing
	4		0		0  Not_testing
	5		0		0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.


Code:
[root@freenas] /var/log# smartctl -a -q noserial /dev/ada1
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:	 Crucial/Micron MX1/2/300, M5/600, 1100 Client SSDs
Device Model:	 Crucial_CT525MX300SSD1
Firmware Version: M0CR021
User Capacity:	525,112,713,216 bytes [525 GB]
Sector Size:	  512 bytes logical/physical
Rotation Rate:	Solid State Device
Form Factor:	  2.5 inches
Device is:		In smartctl database [for details use: -P show]
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.2, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:	Thu Sep 14 05:00:16 2017 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x03)	Offline data collection activity
					is in progress.
					Auto Offline Data Collection: Disabled.
Self-test execution status:	  (   0)	The previous self-test routine completed
					without error or no self-test has ever
					been run.
Total time to complete Offline
data collection:		 (  727) seconds.
Offline data collection
capabilities:			 (0x7b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:			(0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:		(0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine
recommended polling time:	 (   2) minutes.
Extended self-test routine
recommended polling time:	 (   7) minutes.
Conveyance self-test routine
recommended polling time:	 (   3) minutes.
SCT capabilities:			(0x0035)	SCT Status supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME		  FLAG	 VALUE WORST THRESH TYPE	  UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate	 0x002f   100   100   000	Pre-fail  Always	   -	   0
  5 Reallocate_NAND_Blk_Cnt 0x0032   100   100   010	Old_age   Always	   -	   0
  9 Power_On_Hours		  0x0032   100   100   000	Old_age   Always	   -	   6164
12 Power_Cycle_Count	   0x0032   100   100   000	Old_age   Always	   -	   18
171 Program_Fail_Count	  0x0032   100   100   000	Old_age   Always	   -	   0
172 Erase_Fail_Count		0x0032   100   100   000	Old_age   Always	   -	   0
173 Ave_Block-Erase_Count   0x0032   099   099   000	Old_age   Always	   -	   26
174 Unexpect_Power_Loss_Ct  0x0032   100   100   000	Old_age   Always	   -	   18
183 SATA_Interfac_Downshift 0x0032   100   100   000	Old_age   Always	   -	   0
184 Error_Correction_Count  0x0032   100   100   000	Old_age   Always	   -	   0
187 Reported_Uncorrect	  0x0032   100   100   000	Old_age   Always	   -	   0
194 Temperature_Celsius	 0x0022   074   059   000	Old_age   Always	   -	   26 (Min/Max 19/41)
196 Reallocated_Event_Count 0x0032   100   100   000	Old_age   Always	   -	   0
197 Current_Pending_Sector  0x0032   100   100   000	Old_age   Always	   -	   0
198 Offline_Uncorrectable   0x0030   100   100   000	Old_age   Offline	  -	   0
199 UDMA_CRC_Error_Count	0x0032   100   100   000	Old_age   Always	   -	   0
202 Percent_Lifetime_Used   0x0030   099   099   001	Old_age   Offline	  -	   1
206 Write_Error_Rate		0x000e   100   100   000	Old_age   Always	   -	   0
246 Total_Host_Sector_Write 0x0032   100   100   000	Old_age   Always	   -	   9202926991
247 Host_Program_Page_Count 0x0032   100   100   000	Old_age   Always	   -	   317888359
248 Bckgnd_Program_Page_Cnt 0x0032   100   100   000	Old_age   Always	   -	   172505953
180 Unused_Reserve_NAND_Blk 0x0033   000   000   000	Pre-fail  Always	   -	   1949
210 Success_RAIN_Recov_Cnt  0x0032   100   100   000	Old_age   Always	   -	   0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description	Status				  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Vendor (0xff)	   Completed without error	   00%	  6164		 -
# 2  Vendor (0xff)	   Completed without error	   00%	  6045		 -
# 3  Reserved (0x17)	 Aborted by host			   70%	  5911		 -
# 4  Reserved (0x17)	 Aborted by host			   70%	  5911		 -
# 5  Reserved (0x16)	 Aborted by host			   60%	  5654		 -
# 6  Reserved (0x16)	 Aborted by host			   60%	  5654		 -
# 7  Reserved (0x16)	 Aborted by host			   60%	  5654		 -
# 8  Reserved (0x16)	 Aborted by host			   60%	  5654		 -

SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
	1		0		0  Not_testing
	2		0		0  Not_testing
	3		0		0  Not_testing
	4		0		0  Not_testing
	5		0		0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.


Code:
[root@freenas] /var/log# smartctl -a -q noserial /dev/ada2
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:	 Samsung based SSDs
Device Model:	 Samsung SSD 840 Series
Firmware Version: DXT08B0Q
User Capacity:	120,034,123,776 bytes [120 GB]
Sector Size:	  512 bytes logical/physical
Rotation Rate:	Solid State Device
Device is:		In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ATA8-ACS T13/1699-D revision 4c
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:	Thu Sep 14 05:00:42 2017 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)	Offline data collection activity
					was never started.
					Auto Offline Data Collection: Disabled.
Self-test execution status:	  (   0)	The previous self-test routine completed
					without error or no self-test has ever
					been run.
Total time to complete Offline
data collection:		 (	0) seconds.
Offline data collection
capabilities:			 (0x53) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					No Offline surface scan supported.
					Self-test supported.
					No Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:			(0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:		(0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine
recommended polling time:	 (   2) minutes.
Extended self-test routine
recommended polling time:	 (  30) minutes.
SCT capabilities:			(0x003d)	SCT Status supported.
					SCT Error Recovery Control supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME		  FLAG	 VALUE WORST THRESH TYPE	  UPDATED  WHEN_FAILED RAW_VALUE
  5 Reallocated_Sector_Ct   0x0033   100   100   010	Pre-fail  Always	   -	   0
  9 Power_On_Hours		  0x0032   098   098   000	Old_age   Always	   -	   8075
12 Power_Cycle_Count	   0x0032   099   099   000	Old_age   Always	   -	   444
177 Wear_Leveling_Count	 0x0013   090   090   000	Pre-fail  Always	   -	   111
179 Used_Rsvd_Blk_Cnt_Tot   0x0013   100   100   010	Pre-fail  Always	   -	   0
181 Program_Fail_Cnt_Total  0x0032   100   100   010	Old_age   Always	   -	   0
182 Erase_Fail_Count_Total  0x0032   100   100   010	Old_age   Always	   -	   0
183 Runtime_Bad_Block	   0x0013   100   100   010	Pre-fail  Always	   -	   0
187 Uncorrectable_Error_Cnt 0x0032   100   100   000	Old_age   Always	   -	   0
190 Airflow_Temperature_Cel 0x0032   079   058   000	Old_age   Always	   -	   21
195 ECC_Error_Rate		  0x001a   200   200   000	Old_age   Always	   -	   0
199 CRC_Error_Count		 0x003e   100   100   000	Old_age   Always	   -	   0
235 POR_Recovery_Count	  0x0012   099   099   000	Old_age   Always	   -	   243
241 Total_LBAs_Written	  0x0032   099   099   000	Old_age   Always	   -	   16932685867

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
	1		0		0  Not_testing
	2		0		0  Not_testing
	3		0		0  Not_testing
	4		0		0  Not_testing
	5		0		0  Not_testing
  255		0	65535  Read_scanning was never started
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.


Code:
[root@freenas] /var/log# smartctl -a -q noserial /dev/ada3
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:	 Intel 730 and DC S35x0/3610/3700 Series SSDs
Device Model:	 INTEL SSDSC2BA100G3T
Add. Product Id:  DELL(tm)
Firmware Version: 5DV1DL06
User Capacity:	100,030,242,816 bytes [100 GB]
Sector Sizes:	 512 bytes logical, 4096 bytes physical
Rotation Rate:	Solid State Device
Form Factor:	  2.5 inches
Device is:		In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 T13/2015-D revision 3
SATA Version is:  SATA 2.6, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:	Thu Sep 14 05:01:17 2017 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x02)	Offline data collection activity
					was completed without error.
					Auto Offline Data Collection: Disabled.
Self-test execution status:	  (   0)	The previous self-test routine completed
					without error or no self-test has ever
					been run.
Total time to complete Offline
data collection:		 (	2) seconds.
Offline data collection
capabilities:			 (0x79) SMART execute Offline immediate.
					No Auto Offline data collection support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:			(0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:		(0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine
recommended polling time:	 (   1) minutes.
Extended self-test routine
recommended polling time:	 (  60) minutes.
Conveyance self-test routine
recommended polling time:	 (  60) minutes.
SCT capabilities:			(0x003d)	SCT Status supported.
					SCT Error Recovery Control supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME		  FLAG	 VALUE WORST THRESH TYPE	  UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate	 0x000f   130   130   039	Pre-fail  Always	   -	   694771
  5 Reallocated_Sector_Ct   0x0033   100   100   001	Pre-fail  Always	   -	   0
  9 Power_On_Hours		  0x0032   100   100   000	Old_age   Always	   -	   20655
12 Power_Cycle_Count	   0x0032   100   100   000	Old_age   Always	   -	   54
13 Read_Soft_Error_Rate	0x0032   130   100   000	Old_age   Always	   -	   694771
179 Used_Rsvd_Blk_Cnt_Tot   0x0033   100   100   010	Pre-fail  Always	   -	   0
180 Unused_Rsvd_Blk_Cnt_Tot 0x0032   100   100   000	Old_age   Always	   -	   19112
181 Program_Fail_Cnt_Total  0x003a   100   100   000	Old_age   Always	   -	   0
182 Erase_Fail_Count_Total  0x003a   100   100   000	Old_age   Always	   -	   0
184 End-to-End_Error		0x0032   100   100   000	Old_age   Always	   -	   0
194 Temperature_Internal	0x0022   100   100   000	Old_age   Always	   -	   27
195 Hardware_ECC_Recovered  0x0032   100   100   000	Old_age   Always	   -	   0
198 Offline_Uncorrectable   0x0010   100   100   000	Old_age   Offline	  -	   0
199 CRC_Error_Count		 0x003e   100   100   000	Old_age   Always	   -	   0
201 Unknown_SSD_Attribute   0x0033   100   100   010	Pre-fail  Always	   -	   532982268554
226 Workld_Media_Wear_Indic 0x0032   100   100   000	Old_age   Always	   -	   40
227 Workld_Host_Reads_Perc  0x0032   100   100   000	Old_age   Always	   -	   3
228 Workload_Minutes		0x0032   100   100   000	Old_age   Always	   -	   1238987
233 Media_Wearout_Indicator 0x0032   100   100   000	Old_age   Always	   -	   44925
245 Unknown_Attribute	   0x0032   100   100   000	Old_age   Always	   -	   0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description	Status				  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline	   Completed without error	   00%		 2		 -
# 2  Extended offline	Completed without error	   00%		 1		 -
# 3  Short offline	   Completed without error	   00%		 1		 -

SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
	1		0		0  Not_testing
	2		0		0  Not_testing
	3		0		0  Not_testing
	4		0		0  Not_testing
	5		0		0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.


Code:
[root@freenas] /var/log# smartctl -a -q noserial /dev/ada4
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:	 Intel 730 and DC S35x0/3610/3700 Series SSDs
Device Model:	 INTEL SSDSC2BA100G3T
Add. Product Id:  DELL(tm)
Firmware Version: 5DV1DL06
User Capacity:	100,030,242,816 bytes [100 GB]
Sector Sizes:	 512 bytes logical, 4096 bytes physical
Rotation Rate:	Solid State Device
Form Factor:	  2.5 inches
Device is:		In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 T13/2015-D revision 3
SATA Version is:  SATA 2.6, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:	Thu Sep 14 05:01:45 2017 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x02)	Offline data collection activity
					was completed without error.
					Auto Offline Data Collection: Disabled.
Self-test execution status:	  (   0)	The previous self-test routine completed
					without error or no self-test has ever
					been run.
Total time to complete Offline
data collection:		 (	2) seconds.
Offline data collection
capabilities:			 (0x79) SMART execute Offline immediate.
					No Auto Offline data collection support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:			(0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:		(0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine
recommended polling time:	 (   1) minutes.
Extended self-test routine
recommended polling time:	 (  60) minutes.
Conveyance self-test routine
recommended polling time:	 (  60) minutes.
SCT capabilities:			(0x003d)	SCT Status supported.
					SCT Error Recovery Control supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME		  FLAG	 VALUE WORST THRESH TYPE	  UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate	 0x000f   130   130   039	Pre-fail  Always	   -	   689341
  5 Reallocated_Sector_Ct   0x0033   100   100   001	Pre-fail  Always	   -	   0
  9 Power_On_Hours		  0x0032   100   100   000	Old_age   Always	   -	   20653
12 Power_Cycle_Count	   0x0032   100   100   000	Old_age   Always	   -	   54
13 Read_Soft_Error_Rate	0x0032   130   100   000	Old_age   Always	   -	   689341
179 Used_Rsvd_Blk_Cnt_Tot   0x0033   100   100   010	Pre-fail  Always	   -	   0
180 Unused_Rsvd_Blk_Cnt_Tot 0x0032   100   100   000	Old_age   Always	   -	   19111
181 Program_Fail_Cnt_Total  0x003a   100   100   000	Old_age   Always	   -	   0
182 Erase_Fail_Count_Total  0x003a   100   100   000	Old_age   Always	   -	   0
184 End-to-End_Error		0x0032   100   100   000	Old_age   Always	   -	   0
194 Temperature_Internal	0x0022   100   100   000	Old_age   Always	   -	   26
195 Hardware_ECC_Recovered  0x0032   100   100   000	Old_age   Always	   -	   0
198 Offline_Uncorrectable   0x0010   100   100   000	Old_age   Offline	  -	   0
199 CRC_Error_Count		 0x003e   100   100   000	Old_age   Always	   -	   7
201 Unknown_SSD_Attribute   0x0033   100   100   010	Pre-fail  Always	   -	   532982268565
226 Workld_Media_Wear_Indic 0x0032   100   100   000	Old_age   Always	   -	   40
227 Workld_Host_Reads_Perc  0x0032   100   100   000	Old_age   Always	   -	   7
228 Workload_Minutes		0x0032   100   100   000	Old_age   Always	   -	   1238893
233 Media_Wearout_Indicator 0x0032   100   100   000	Old_age   Always	   -	   43461
245 Unknown_Attribute	   0x0032   100   100   000	Old_age   Always	   -	   0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description	Status				  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline	   Completed without error	   00%		 2		 -
# 2  Extended offline	Completed without error	   00%		 1		 -
# 3  Short offline	   Completed without error	   00%		 1		 -

SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
	1		0		0  Not_testing
	2		0		0  Not_testing
	3		0		0  Not_testing
	4		0		0  Not_testing
	5		0		0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
 

Attachments

  • dmesg.today.txt
    38.2 KB · Views: 497
  • dmesg.yesterday.txt
    255 KB · Views: 472
  • messages.txt
    41.9 KB · Views: 473
Joined
Apr 11, 2013
Messages
7
Di0CSpw.jpg


Here is some further output during a clone operation, ada0 and ada1 are the 2 SSDs that make up the datastore. The wait time on the left pane is ever-increasing.

miJhs4k.png


This is the output of esxtop for the datastore (3rd row).

Any ideas?

Cheers,
Kane.
 
D

dlavigne

Guest
Your logs are full of listen queue overflows (sonewconn). Are you able to update this system to 11.0-U3 to see if that resolves it?
 
Joined
Apr 11, 2013
Messages
7
Your logs are full of listen queue overflows (sonewconn). Are you able to update this system to 11.0-U3 to see if that resolves it?

Hi, thanks for your response.

I have upgraded to FreeNAS-9.10.2-U6 (561f0d7a1) and the issue is still present.

I adjusted the queue depth on my esxi host to 128 for the iscsi lun but it appears to be capping at 100, which I'd say is being enforced at the freenas box.

Any further help I would be incredibly grateful for .

Kane.
 
D

dlavigne

Guest
The new kernel and drivers in 11 might resolve this. If they do not, it would be worth making a ticket at bugs.freenas.org to see if something additional could be done for your configuration.
 

JoshDW19

Community Hall of Fame
Joined
May 16, 2016
Messages
1,077
The new kernel and drivers in 11 might resolve this. If they do not, it would be worth making a ticket at bugs.freenas.org to see if something additional could be done for your configuration.

I agree with Dru. I'd upgrade and see if the issue persists. Many issues were fixed between these versions.
 
Joined
Apr 11, 2013
Messages
7
Hi Guys,

I'd like to confirm that upgrading to 11 u3 seems to have fixed the issue. To anyone else experiencing similar behaviour, I imagine the upgrade would be your first point of troubleshooting too.

Cheers,
Kane.
 
Status
Not open for further replies.
Top