GUI access lost and rando errors happenning.

[TheUsD]

Freshly installed TrueNAS 13.0u3
HW:
Make/Model: Oracle FS1-2 Flash
CPUs: Intel E5-2620
RAM 208GB ecc DDR3
PCI1: Sun Storage 16 Gb Fibre Channel PCIe Universal FC HBA, Qlogic
PCI2: Sun Storage 16 Gb Fibre Channel PCIe Universal FC HBA, Qlogic
PCI3: Adaptec 70165H Host Bus Adapter - Connected to DE2-24p w/ 24 Sun 7044376 Oracle 900GB 10K RPM 2.5in SAS Hard Drive Disk
PCI4: Adaptec 70165H Host Bus Adapter - Connected to DE2-24p w/ 12 Sun 7044376 Oracle 900GB 10K RPM 2.5in SAS Hard Drive Disk and 12 SUN O7093044 1.2TB 10K RPM
PCI6: Adaptec 70165H Host Bus Adapter

TrueNAS is advertising iSCSi zvols to two esxi hosts.

When idle, SMART short/long tests, or SCRUBS the system works fine. Once I start moving storage to the drives, all sorts of weird errors, such as checksum errors and eventually the system reboots. No alerts, not even the common "your system rebooted unexpectedly" alert is reported. If I attempt to do a scrub after a system reboot, the scrub freezes at random stops.

Today, with only a single VM on datastore that, again, is attached to the TrueNAS via iscsi and these errors came up. I lost access to the GUI but the VM is still running.
Here are the errors being reported as of now:

The only configuration changes made are:

1. Installing a cert for the GUI
2. Enabling iSCSi
3. Enabling SSH
4. Enabling LLDP
5. Changing the NTP
6. Created Pool-01 with the 24 900GB SAS drives - 4 vdevs of 6 raidz3
7. Created Pool-02 with the 12 900GB and 12 1.2TB SAS drives 2 vdevs raidz2

I've read this has been a SNMP issue on other forums but that was on 12.x. Lastly, temp intake is 29c and output is 37c

*Update* I know I am stacking issues on issues, but I also wanted to mention that sometimes the zvol on Pool-02, which is the combo 12 900GB and 12 1.2TB does not always stay connected to the esxi hosts. Both SAN3-Boot-01 and SAN3-Storage-01 are being advertised off the same initiator and target group from TrueNAS. ESXi can see the device but not the datastore.
vCenter

ESXi host devices. The 5.5TB and the 13.8TB are advertised from the host having the issue. The other two devices, 3.58TB and 18.76 are on two separate Dell R720's. No issues with either and they are running the same version and build of TrueNAS.

esxi datastores

 

[TheUsD]

Is there any more information I can provide?

 

[HoneyBadger]

The Adaptec cards are supported by a PMC/Microsemi driver which appears to have had official vendor support stopped as of FreeBSD 10 according to their site. Being as it's Oracle original hardware they might have just developed their own Solaris driver.

Can you find the dmesg line that describes which driver is being used? dmesg | grep -i 'sas\|sata' should pull it out.

 

[TheUsD]

The Adaptec cards are supported by a PMC/Microsemi driver which appears to have had official vendor support stopped as of FreeBSD 10 according to their site. Being as it's Oracle original hardware they might have just developed their own Solaris driver.

Can you find the dmesg line that describes which driver is being used? dmesg | grep -i 'sas\|sata' should pull it out.

My apologies if I gave too much fluff

Code:

pmspcv0: <PMC Sierra SPC-V 16 Port SAS-SATA Card> mem 0xc7a10000-0xc7a1ffff,0xc7a00000-0xc7a0ffff irq 16 at device 0.0 numa-domain 0 on pci3
ahci0: <Intel Patsburg AHCI SATA controller> port 0x9070-0x9077,0x9060-0x9063,0x9050-0x9057,0x9040-0x9043,0x9020-0x903f mem 0xc7f22000-0xc7f227ff irq 19 at device 31.2 numa-domain 0 on pci0
pmspcv1: <PMC Sierra SPC-V 16 Port SAS-SATA Card> mem 0xfbe10000-0xfbe1ffff,0xfbe00000-0xfbe0ffff irq 16 at device 0.0 numa-domain 1 on pci18
pmspcv2: <PMC Sierra SPC-V 16 Port SAS-SATA Card> mem 0xfbc10000-0xfbc1ffff,0xfbc00000-0xfbc0ffff irq 16 at device 0.0 numa-domain 1 on pci19
ses1: pass25 in 'Slot 04', SATA Slot: scbus5 target 0
ses0: da0,pass0 in 'DISK  0', SAS Slot: 1+ phys at slot 0
ada0: <Samsung SSD 860 EVO 250GB RVT02B6Q> ACS-4 ATA SATA 3.x device
ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 512bytes)
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0: da1,pass1 in 'DISK  1', SAS Slot: 1+ phys at slot 1
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0: da2,pass2 in 'DISK  2', SAS Slot: 1+ phys at slot 2
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses2: da24,pass27 in 'DISK  0', SAS Slot: 1+ phys at slot 0
ses0: da3,pass3 in 'DISK  3', SAS Slot: 1+ phys at slot 3
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses2:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0: da4,pass4 in 'DISK  4', SAS Slot: 1+ phys at slot 4
ses2: da25,pass28 in 'DISK  1', SAS Slot: 1+ phys at slot 1
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses2:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses2: da26,pass29 in 'DISK  2', SAS Slot: 1+ phys at slot 2
ses0: da5,pass5 in 'DISK  5', SAS Slot: 1+ phys at slot 5
ses2:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses2: da27,pass30 in 'DISK  3', SAS Slot: 1+ phys at slot 3
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0: da6,pass6 in 'DISK  6', SAS Slot: 1+ phys at slot 6
ses2:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses2: da28,pass31 in 'DISK  4', SAS Slot: 1+ phys at slot 4
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0: da7,pass7 in 'DISK  7', SAS Slot: 1+ phys at slot 7
ses2:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses2: da29,pass32 in 'DISK  5', SAS Slot: 1+ phys at slot 5
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses2:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses2: da30,pass33 in 'DISK  6', SAS Slot: 1+ phys at slot 6
ses0: da8,pass8 in 'DISK  8', SAS Slot: 1+ phys at slot 8
ses2:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses2: da31,pass34 in 'DISK  7', SAS Slot: 1+ phys at slot 7
ses2:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses2: da32,pass35 in 'DISK  8', SAS Slot: 1+ phys at slot 8
ses2:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses2: da33,pass36 in 'DISK  9', SAS Slot: 1+ phys at slot 9
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses2:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses2: da34,pass37 in 'DISK 10', SAS Slot: 1+ phys at slot 10
ses0: da9,pass9 in 'DISK  9', SAS Slot: 1+ phys at slot 9
ses2:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0: da10,pass10 in 'DISK 10', SAS Slot: 1+ phys at slot 10
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0: da11,pass11 in 'DISK 11', SAS Slot: 1+ phys at slot 11
ses2: da35,pass38 in 'DISK 11', SAS Slot: 1+ phys at slot 11
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0: da12,pass12 in 'DISK 12', SAS Slot: 1+ phys at slot 12
ses2:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses2: da36,pass39 in 'DISK 12', SAS Slot: 1+ phys at slot 12
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0: da13,pass13 in 'DISK 13', SAS Slot: 1+ phys at slot 13
ses2:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses2: da37,pass40 in 'DISK 13', SAS Slot: 1+ phys at slot 13
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0: da14,pass14 in 'DISK 14', SAS Slot: 1+ phys at slot 14
ses2:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses2: da38,pass41 in 'DISK 14', SAS Slot: 1+ phys at slot 14
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses2:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses2: da39,pass42 in 'DISK 15', SAS Slot: 1+ phys at slot 15
ses0: da15,pass15 in 'DISK 15', SAS Slot: 1+ phys at slot 15
ses2:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses2: da40,pass43 in 'DISK 16', SAS Slot: 1+ phys at slot 16
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0: da16,pass16 in 'DISK 16', SAS Slot: 1+ phys at slot 16
ses2:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses2: da41,pass44 in 'DISK 17', SAS Slot: 1+ phys at slot 17
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0: da17,pass17 in 'DISK 17', SAS Slot: 1+ phys at slot 17
ses2:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses2: da42,pass45 in 'DISK 18', SAS Slot: 1+ phys at slot 18
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0: da18,pass18 in 'DISK 18', SAS Slot: 1+ phys at slot 18
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses2:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses2: da43,pass46 in 'DISK 19', SAS Slot: 1+ phys at slot 19
ses2:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0: da19,pass19 in 'DISK 19', SAS Slot: 1+ phys at slot 19
ses2: da44,pass47 in 'DISK 20', SAS Slot: 1+ phys at slot 20
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses0: da20,pass20 in 'DISK 20', SAS Slot: 1+ phys at slot 20
ses2:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses2: da45,pass48 in 'DISK 21', SAS Slot: 1+ phys at slot 21
ses0:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses2:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses2: da46,pass49 in 'DISK 22', SAS Slot: 1+ phys at slot 22
ses2:  phy 0: SAS device type 1 phy 1 Target ( SSP )
ses2: da47,pass50 in 'DISK 23', SAS Slot: 1+ phys at slot 23
ses0: da21,pass21 in 'DISK 21', SAS Slot: 1+ phys at

I ended up just rebooting the device via the ILOM. Upon reboot, I received these errors:

Ran a new SMART test on drives 24-47 and recieved:

Code:

Errors:
da38
smartctl failed for disk da38:
smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p2 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

Standard Inquiry (36 bytes) failed [Input/output error]
Retrying with a 64 byte Standard Inquiry
Standard Inquiry (64 bytes) failed [Input/output error]
A mandatory SMART command failed: exiting. To continue, add one or more '-T permissive' options.

da36
smartctl failed for disk da36:
smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p2 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

Smartctl open device: /dev/da36 failed: INQUIRY failed

da34
smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p2 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

Long (extended) offline self test failed [device not ready]

da33
smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p2 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

Long (extended) offline self test failed [device not ready]

da29
smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p2 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

Long (extended) offline self test failed [device not ready]

da27
smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p2 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

Long (extended) offline self test failed [device not ready]

da26
smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p2 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

Long (extended) offline self test failed [device not ready]

da25
smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p2 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

Long (extended) offline self test failed [device not ready]

 

[TheUsD]

I forgot to mention @HoneyBadger I had to reboot via ILOM in order to run your command. I could have SSH'ed in, but I didn't think about that at the time, so I did a reboot instead. When I ran that SMART test, those errors were Immedient. Like .5ms fast, haha. First time I received errors.

 

[HoneyBadger]

Those SMART errors are popping up fast because it's not able to actually issue any commands against the disks. Try running against the passX devices.

But it looks like you're using the pms driver for those cards. I don't have a lot of experience with them but they might not have the same maturity under FreeBSD vs the standard LSI family of cards.

Is grabbing a cheap LSI external HBA an option?

 

[TheUsD]

Those SMART errors are popping up fast because it's not able to actually issue any commands against the disks. Try running against the passX devices.

But it looks like you're using the pms driver for those cards. I don't have a lot of experience with them but they might not have the same maturity under FreeBSD vs the standard LSI family of cards.

Is grabbing a cheap LSI external HBA an option?

I will look into how to run passx. Thats something new for me.

I did not choose any specific driver, just allowed TrueNAS to load whatever drivers it deemed necessary when it did its install. Getting LSI Cards is not out of the question, was just hoping to use what was free and given to me. Do you have any recommendations on specific LSI eHBA cards I use obtain?

 

[TheUsD]

Try running against the passX devices

Understanding what you mean now. I ran said and there did not seem to be any current issues with da36 or da44 which are the two disks that showed alerts on post 4, pic 1

I am running Long tests on them both. They've been running for roughly 122mins

Code:

root@SAN-03[~]# smartctl -a /dev/da44
smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p2 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:               SEAGATE
Product:              ST1200IN9SUN1.2T
Revision:             ORA9
Compliance:           SPC-4
User Capacity:        1,200,243,695,616 bytes [1.20 TB]
Logical block size:   512 bytes
Formatted with type 1 protection
8 bytes of protection information per logical block
LU is fully provisioned
Rotation Rate:        10020 rpm
Form Factor:          2.5 inches
Logical Unit id:      0x5000c500ca677057
Serial number:        002022L8XXXX        WFK8XXXX
Device type:          disk
Transport protocol:   SAS (SPL-3)
Local Time is:        Sat Nov 19 11:37:20 2022 CST
SMART support is:     Available - device has SMART capability.
SMART support is:     Enabled
Temperature Warning:  Enabled

=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK

Grown defects during certification <not available>
Total blocks reassigned during format <not available>
Total new blocks reassigned = 0
Power on minutes since format <not available>
Current Drive Temperature:     34 C
Drive Trip Temperature:        68 C

Accumulated power on time, hours:minutes 9482:43
Manufactured in week 22 of year 2020
Specified cycle count over device lifetime:  10000
Accumulated start-stop cycles:  22
Specified load-unload count over device lifetime:  300000
Accumulated load-unload cycles:  551
Elements in grown defect list: 0

Vendor (Seagate Cache) information
  Blocks sent to initiator = 1582166843
  Blocks received from initiator = 3039535313
  Blocks read from cache and sent to initiator = 1778246658
  Number of read and write commands whose size <= segment size = 435529175
  Number of read and write commands whose size > segment size = 10770

Vendor (Seagate/Hitachi) factory information
  number of hours powered up = 9482.72
  number of minutes until next internal SMART test = 15

Error counter log:
           Errors Corrected by           Total   Correction     Gigabytes    Total
               ECC          rereads/    errors   algorithm      processed    uncorrected
           fast | delayed   rewrites  corrected  invocations   [10^9 bytes]  errors
read:   529314550        0         0  529314550          0     136994.128    0
write:         0        0         0         0          0      19532.797  0
verify: 124912797        0         0  124912797          0         64.961    0

Non-medium error count:        0

SMART Self-test log
Num  Test              Status                 segment  LifeTime  LBA_first_err [SK ASC ASQ]
     Description                              number   (hours)
# 1  Background long   Self test in progress ...   -     NOW                 - [-   -    -]
# 2  Background short  Completed                   -    9479                 - [-   -    -]

Code:

root@SAN-03[~]# smartctl -a /dev/da36
smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p2 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:               SEAGATE
Product:              ST1200IN9SUN1.2T
Revision:             ORA9
Compliance:           SPC-4
User Capacity:        1,200,243,695,616 bytes [1.20 TB]
Logical block size:   512 bytes
Formatted with type 1 protection
8 bytes of protection information per logical block
LU is fully provisioned
Rotation Rate:        10020 rpm
Form Factor:          2.5 inches
Logical Unit id:      0x5000c500a6db304b
Serial number:        001834L1XXXX        WFK1XXXX
Device type:          disk
Transport protocol:   SAS (SPL-3)
Local Time is:        Sat Nov 19 11:39:17 2022 CST
SMART support is:     Available - device has SMART capability.
SMART support is:     Enabled
Temperature Warning:  Enabled

=== START OF READ SMART DATA SECTION ===
SMART Health Status: FAILURE PREDICTION THRESHOLD EXCEEDED: ascq=0x5 [asc=5d, ascq=5]

Grown defects during certification <not available>
Total blocks reassigned during format <not available>
Total new blocks reassigned = 7879
Power on minutes since format <not available>
Current Drive Temperature:     33 C
Drive Trip Temperature:        68 C

Accumulated power on time, hours:minutes 17053:28
Manufactured in week 34 of year 2018
Specified cycle count over device lifetime:  10000
Accumulated start-stop cycles:  37
Specified load-unload count over device lifetime:  300000
Accumulated load-unload cycles:  999
Elements in grown defect list: 8190

Vendor (Seagate Cache) information
  Blocks sent to initiator = 3077942699
  Blocks received from initiator = 2781769790
  Blocks read from cache and sent to initiator = 2459120950
  Number of read and write commands whose size <= segment size = 1471166783
  Number of read and write commands whose size > segment size = 132656

Vendor (Seagate/Hitachi) factory information
  number of hours powered up = 17053.47
  number of minutes until next internal SMART test = 3

Error counter log:
           Errors Corrected by           Total   Correction     Gigabytes    Total
               ECC          rereads/    errors   algorithm      processed    uncorrected
           fast | delayed   rewrites  corrected  invocations   [10^9 bytes]  errors
read:   3091598738     1950         0  3091600688       1952     528471.777      2
write:         0        0         0         0          0      53926.247  0
verify: 397293343      128         0  397293471        128        207.140    0

Non-medium error count:        0

SMART Self-test log
Num  Test              Status                 segment  LifeTime 

 

[HoneyBadger]

I did not choose any specific driver, just allowed TrueNAS to load whatever drivers it deemed necessary when it did its install. Getting LSI Cards is not out of the question, was just hoping to use what was free and given to me. Do you have any recommendations on specific LSI eHBA cards I use obtain?

Going off the photos in your other thread, your current cards use the higher-density 12Gbps SAS plug - but the DE2 shelves use the older, flatter SAS 6Gbps generation.

In your shoes, I'd buy a nice inexpensive 6Gbps SAS HBA like the Dell PERC H200e for around USD$30, and a set of SFF-8088 cables - two perhaps, as 48 10K SAS drives is likely enough to feel a choke from a 4x6Gbps line on things like scrubs or heavy sequential workloads. Look for Dell part numbers 12DNW/D687J - the LSI 9200-8e IT firmware should be a match if you need to flash. (I think I might have one or two of these kicking around, I'll try checking them out to see if the flash process is the same.)

The more expensive option would be to buy a 12Gbps SAS HBA like the Dell "12Gbps HBA" - part numbers 2PHG9/T93GD - that's two 12Gbps ports, and would let you use your existing cables, but would run closer to the USD$115-120 mark. Those will be SAS3008 based, but stock firmware should be fine.

 

[TheUsD]

Going off the photos in your other thread, your current cards use the higher-density 12Gbps SAS plug - but the DE2 shelves use the older, flatter SAS 6Gbps generation.

You are correct. The cards are 12Gbps with the square, physical inserts with the flat 6Gbps inserts at the other end. These are the default cards and cables that came with the FS1-2 and the shelves from Oracle. I thought that was weird.

In your shoes, I'd buy a nice inexpensive 6Gbps SAS HBA like the Dell PERC H200e for around USD$30, and a set of SFF-8088 cables - two perhaps

I actually already own 4 of these because I purchased thinking TrueNAS could support HBAe connection setup such as ESXi to TrueNAS. Now I might have a purpose for these cards, haha!

I'll try checking them out to see if the flash process is the same.

If you have steps that you think could work, feel free to shoot them over and I'll do the flash attempts myself and possibly save you some time / effort. You've already been generous enough to always come to my aid when I get on these forums, flaying around like a fish out of water.

 

[TheUsD]

I thought that was weird.

I take that back, I have a good suspicion as to why. There were two FS1-2 controllers hooked up to a total of 3 DE2-24P. Meaning two Oracle controllers of all three storage "containers". Those Controllers were also connected to each other via the HBAe'.

Does TrueNAS support two controllers to manage multiple "containers" or in this case DE2-24Ps? I ask because I have the other FS1-2 and a 3rd DE2-24P. The 3rd DE2-24P is going to be all SSD. 12, 1TB and 12, 2TB. zRAID2