Disk Fail and unexpected reboot

[Archer27]

Hello,
hope posting in the right section.

I build an FreeNAS on a Fujitsu Server

The problems starts after some Tb R/W on 2 iSCSI Targets (2 zvol) on one Pool.
The Pool ist from 8x1Tb SSD RAIDZ2
I use two seperate servers as initator. Each with 1 gbit direct connected to the FreeNAS Server.

New alerts:
* Pool Pool1 state is DEGRADED: One or more devices are faulted in response to persistent errors. Sufficient replicas exist for the pool to continue functioning in a degraded state..

OK right, this is shown
Name: da4
GUID: 15200565966193669901
Read Errors: 117
Write Errors: 3165
Self Healed: 0

The stange things:
After Reboot it is not longer degraded. It shows "online".

* Pool Pool1 state is ONLINE: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected..
The following alert has been cleared:
* Pool Pool1 state is ONLINE: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state..

Thie SSD is fully work after reboot and not loger shown af faulty

Some Tb R/W later ...

had an unscheduled system reboot.
The operating system successfully came back online at Wed Feb 26 16:24:25 2020.

If i use the onboard SATA-Connectors this not happen.
Only with the 9305-16i HBA.

Any hints for me to solve this problem?

Thank you

Chris

 

[Archer27]

Push

 

[sretalla]

Any hints for me to solve this problem?

So what is smartctl -a /dev/da4 telling you?

 

[Archer27]

Hello,

the Answer from smartctl -a /dev/da4 is (in short)

=== START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED

All devices that "fail" have no SMART problems.
At this time i think it is a controller problem. I changed the controller and have the same behavior.
Now i installed the latest freeNAS Version for a new test.
Maybe the problem is gone with the bradcom 9x00 fixes.

Chris

 

[sretalla]

All devices that "fail" have no SMART problems.

I'm not sure I follow you (failing SMART test can point to SMART tests not running)... the lines you quoted do not indicate the status of the disk, just the status of the test... share (or at least look) at the full test results for that drive.

At this time i think it is a controller problem. I changed the controller and have the same behavior.

Seems you confirmed it's not a controller problem (unless you have 2 bad controllers).

 

[Archer27]

Hello,

i am sorry for the incomplete output.
smartctl -a /dev/da4 smartctl 7.0 2018-12-30 r4883 [FreeBSD 11.3-RELEASE-p7 amd64] (local build) Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Samsung based SSDs Device Model: Samsung SSD 860 EVO 1TB Serial Number: S3Z9NB0KC57078J LU WWN Device Id: 5 002538 e40b30c6b Firmware Version: RVT02B6Q User Capacity: 1,000,204,886,016 bytes [1.00 TB] Sector Size: 512 bytes logical/physical Rotation Rate: Solid State Device Form Factor: 2.5 inches Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-4 T13/BSR INCITS 529 revision 5 SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Tue Apr 14 15:49:20 2020 CEST SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x00) Offline data collection activity was never started. Auto Offline Data Collection: Disabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: ( 0) seconds. Offline data collection capabilities: (0x53) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. No Offline surface scan supported. Self-test supported. No Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 85) minutes. SCT capabilities: (0x003d) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 1 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0 9 Power_On_Hours 0x0032 099 099 000 Old_age Always - 1664 12 Power_Cycle_Count 0x0032 099 099 000 Old_age Always - 57 177 Wear_Leveling_Count 0x0013 099 099 000 Pre-fail Always - 10 179 Used_Rsvd_Blk_Cnt_Tot 0x0013 100 100 010 Pre-fail Always - 0 181 Program_Fail_Cnt_Total 0x0032 100 100 010 Old_age Always - 0 182 Erase_Fail_Count_Total 0x0032 100 100 010 Old_age Always - 0 183 Runtime_Bad_Block 0x0013 100 100 010 Pre-fail Always - 0 187 Uncorrectable_Error_Cnt 0x0032 100 100 000 Old_age Always - 0 190 Airflow_Temperature_Cel 0x0032 080 064 000 Old_age Always - 20 195 ECC_Error_Rate 0x001a 200 200 000 Old_age Always - 0 199 CRC_Error_Count 0x003e 100 100 000 Old_age Always - 0 235 POR_Recovery_Count 0x0012 099 099 000 Old_age Always - 34 241 Total_LBAs_Written 0x0032 099 099 000 Old_age Always - 11068089204 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Short offline Completed without error 00% 1010 - # 2 Short offline Completed without error 00% 1009 - # 3 Short offline Completed without error 00% 1008 - # 4 Short offline Completed without error 00% 1007 - # 5 Short offline Completed without error 00% 1006 - # 6 Short offline Completed without error 00% 1005 - # 7 Short offline Completed without error 00% 1004 - # 8 Short offline Completed without error 00% 1003 - # 9 Short offline Completed without error 00% 1002 - #10 Short offline Completed without error 00% 1001 - #11 Short offline Completed without error 00% 1000 - #12 Short offline Completed without error 00% 999 - #13 Short offline Completed without error 00% 998 - #14 Short offline Completed without error 00% 997 - #15 Short offline Completed without error 00% 996 - #16 Short offline Completed without error 00% 995 - #17 Short offline Completed without error 00% 994 - #18 Short offline Completed without error 00% 993 - #19 Short offline Completed without error 00% 992 - #20 Short offline Completed without error 00% 991 - #21 Short offline Completed without error 00% 990 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay.


And he did it again,
while a heavy load test the system reboot.

WARNING
had an unscheduled system reboot. The operating system successfully came back online at Tue Apr 14 15:43:42 2020.
Tue, 14 Apr 2020 03:43:42 PM (Europe/Berlin)

 

[sretalla]

So that confirms the disk itself is OK. You already eliminated the controller (as far as I can see).

How about cabling or your backplane/expander? (try the disk in a different bay/slot/port/cable)

 

[sretalla]

235 POR_Recovery_Count 0x0012 099 099 000 Old_age Always - 34

OK, so there is an issue with the disk... it is being powered off without graceful shutdown... could it be an issue with the power supply in the system?

 

[Archer27]

Hello,
cabeling and backplane are original Fujitsu.
I have controlled and unplug/plug all Cables.
Seems be OK
Changing the disk is possible (but make no sense if you see the syslog. Its every time another disk)
Power supply looks good and the server don´t tell me that something is wrong.
The syslog is attatched.
Maybe you can see what happen at the time the system reboots.
I think this is a problem with the HBA and the firmware because with the onboard SATA Ports this do not happen, but i loose 25% performance.
If you need a longer version from the syslog please tell me.

 

[Archer27]

Next system Fail.
Here the last two minutes.
The Raid-Z is online without errors afer the unexpected reboot.
No errors shown on any disk.

Code:

Thu Apr 16 09:45:00 2020;130.10.100.70; <78>Apr 16 09:45:00 STORAGE /usr/sbin/cron[18150]: (root) CMD (/usr/libexec/atrun > /dev/null 2>&1)
Thu Apr 16 09:45:56 2020;130.10.100.70; <30>Apr 16 09:45:55 STORAGE smartd[1427]: Device: /dev/da7 [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 72 to 71
Thu Apr 16 09:45:56 2020;130.10.100.70; <30>Apr 16 09:45:55 STORAGE smartd[1427]: Device: /dev/da6 [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 72 to 71
Thu Apr 16 09:45:56 2020;130.10.100.70; <30>Apr 16 09:45:55 STORAGE smartd[1427]: Device: /dev/da4 [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 72 to 71
Thu Apr 16 09:45:56 2020;130.10.100.70; <30>Apr 16 09:45:55 STORAGE smartd[1427]: Device: /dev/da2 [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 72 to 71
Thu Apr 16 09:45:56 2020;130.10.100.70; <30>Apr 16 09:45:55 STORAGE smartd[1427]: Device: /dev/da1 [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 72 to 71
Thu Apr 16 09:45:56 2020;130.10.100.70; <30>Apr 16 09:45:55 STORAGE smartd[1427]: Device: /dev/da0 [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 72 to 71
Thu Apr 16 09:45:56 2020;130.10.100.70; <30>Apr 16 09:45:55 STORAGE smartd[1427]: Device: /dev/ada1, SMART Usage Attribute: 194 Temperature_Celsius changed from 81 to 80
Thu Apr 16 09:47:40 2020;130.10.100.70; <13>Apr 16 09:47:39 STORAGE     (da4:mpr0:0:4:0): WRITE(10). CDB: 2a 00 13 89 3a 50 00 00 48 00 length 36864 SMID 709 Aborting command 0xfffffe0002140b30
Thu Apr 16 09:47:40 2020;130.10.100.70; <13>Apr 16 09:47:39 STORAGE mpr0: Sending reset from mprsas_send_abort for target ID 4
Thu Apr 16 09:47:40 2020;130.10.100.70; <13>Apr 16 09:47:39 STORAGE     (da2:mpr0:0:2:0): WRITE(10). CDB: 2a 00 16 fe f1 58 00 00 60 00 length 49152 SMID 131 Aborting command 0xfffffe000210cc50
Thu Apr 16 09:47:40 2020;130.10.100.70; <13>Apr 16 09:47:39 STORAGE mpr0: Sending reset from mprsas_send_abort for target ID 2
Thu Apr 16 09:47:40 2020;130.10.100.70; <13>Apr 16 09:47:39 STORAGE     (da3:mpr0:0:3:0): WRITE(10). CDB: 2a 00 13 89 36 b8 00 00 08 00 length 4096 SMID 820 Aborting command 0xfffffe000214aac0
Thu Apr 16 09:47:40 2020;130.10.100.70; <13>Apr 16 09:47:39 STORAGE mpr0: Sending reset from mprsas_send_abort for target ID 3
Thu Apr 16 09:47:40 2020;130.10.100.70; <13>Apr 16 09:47:39 STORAGE     (da7:mpr0:0:7:0): WRITE(10). CDB: 2a 00 16 fe f1 28 00 00 28 00 length 20480 SMID 372 Aborting command 0xfffffe00021226c0
Thu Apr 16 09:47:40 2020;130.10.100.70; <13>Apr 16 09:47:39 STORAGE mpr0: Sending reset from mprsas_send_abort for target ID 7
Thu Apr 16 09:47:40 2020;130.10.100.70; <13>Apr 16 09:47:39 STORAGE     (pass10:mpr0:0:5:0): LOG SENSE. CDB: 4d 00 4d 00 00 00 00 00 40 00 length 64 SMID 291 Aborting command 0xfffffe000211b250
Thu Apr 16 09:47:40 2020;130.10.100.70; <13>Apr 16 09:47:39 STORAGE mpr0: Sending reset from mprsas_send_abort for target ID 5
Thu Apr 16 09:47:40 2020;130.10.100.70; <13>Apr 16 09:47:39 STORAGE     (pass5:mpr0:0:0:0): LOG SENSE. CDB: 4d 00 4d 00 00 00 00 00 40 00 length 64 SMID 653 Aborting command 0xfffffe000213bab0
Thu Apr 16 09:47:40 2020;130.10.100.70; <13>Apr 16 09:47:39 STORAGE mpr0: Sending reset from mprsas_send_abort for target ID 0
Thu Apr 16 09:47:40 2020;130.10.100.70; <13>Apr 16 09:47:39 STORAGE     (pass6:mpr0:0:1:0): LOG SENSE. CDB: 4d 00 4d 00 00 00 00 00 40 00 length 64 SMID 847 Aborting command 0xfffffe000214d190
Thu Apr 16 09:47:40 2020;130.10.100.70; <13>Apr 16 09:47:39 STORAGE mpr0: Sending reset from mprsas_send_abort for target ID 1
Thu Apr 16 09:47:40 2020;130.10.100.70; <13>Apr 16 09:47:39 STORAGE     (da6:mpr0:0:6:0): WRITE(10). CDB: 2a 00 16 fe f3 30 00 00 e8 00 length 118784 SMID 336 Aborting command 0xfffffe000211f300
Thu Apr 16 09:47:40 2020;130.10.100.70; <13>Apr 16 09:47:39 STORAGE mpr0: Sending reset from mprsas_send_abort for target ID 6
Thu Apr 16 09:47:40 2020;130.10.100.70; <13>Apr 16 09:47:39 STORAGE mpr0: mprsas_action_scsiio: Freezing devq for target ID 5
Thu Apr 16 09:47:40 2020;130.10.100.70; <13>Apr 16 09:47:39 STORAGE (da5:mpr0:0:5:0): READ(10). CDB: 28 00 68 6a 58 08 00 00 08 00
Thu Apr 16 09:47:40 2020;130.10.100.70; <13>Apr 16 09:47:39 STORAGE (da5:mpr0:0:5:0): CAM status: CAM subsystem is busy
Thu Apr 16 09:47:40 2020;130.10.100.70; <13>Apr 16 09:47:39 STORAGE (da5:mpr0:0:5:0): Retrying command
Thu Apr 16 09:47:40 2020;130.10.100.70; <27>Apr 16 09:47:39 STORAGE collectd[1797]: Traceback (most recent call last):
  

 

[sretalla]

No errors shown on any disk.

/dev/ada1, SMART Usage Attribute: 194 Temperature_Celsius changed from 81 to 80

That's a hot disk... technically not an error, but 80 degrees celsius is going to be a problem.

You are also queueing for da5.

I think you should consider looking at the heat in your system and getting that below 50 degrees. It is possible your HBA is going to fry itself.

 

[Archer27]

Hello,

first time i see this i was shocked.
Running to the Serverroom. 20 °C ;-)
The Server and disks are cold like a Fish.
I think this is an Celsius - Farenheit Error.
This will explain the high Temp .
70°F ~ 21 °C

 

[sretalla]

Except the counter is specifically called Temperature_Celsius... seems unlikely the temperature would be shown in fahrenheit.

Is there a chance the disks are hot inside the server (just not at the front where you can touch)?

 

[Archer27]

I pulled out two random SSD.
Both are cold.
After the last crash i find a thread what maybe point to the error.
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=224496

 

[Archer27]

Happy,

final Solution.

The thread https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=224496 push me in the right direction.
It was an firmware issue from the Broadcom SAS 9305-16i Controller.
After using sas3flash and bringing the controller to the latest firmware the problems are gone.
It runns without reboots.
Nevermind that Broadcom sell new controllers with such an old firmware.