FreeNAS updates and errors/controller firmware

[Rob G]

Good morning. I posted the below info/questions on Reddit regarding one of our FreeNAS servers (SuperMicro hardware). I am not certain what direction to take with the updates and firmware errors at this point. Any help with narrowing things down is greatly appreciated.

I attempted to update Server 2 in our pair of FreeNAS servers yesterday and received the error "cannot remove file /tmp". After doing some digging on Google, I found that the files in /var/db/system/update can be deleted allowing for the update to proceed. Has anyone verified that this works?

Code:

Contests of the directory:
-rw-r--r-- 1 root wheel	38K Nov 10 02:58 ChangeLog.txt
-rw-r--r-- 1 root wheel 5.6M Nov 10 02:58 FreeNASUI-9.10.1-U4-d61691852044f7425790380209f30ef9.tgz
-rw-r--r-- 1 root wheel 3.7K Nov 10 02:56 MANIFEST
-rw-r--r-- 1 root wheel	32B Nov 10 02:58 SEQUENCE
-rw-r--r-- 1 root wheel	 7B Nov 10 02:58 SERVER
-rw-r--r-- 1 root wheel 380M Nov 10 02:58 base-os-9.10.1-U4-d61691852044f7425790380209f30ef9.tgz
-rw-r--r-- 1 root wheel 7.7M Nov 10 02:58 docs-9.10.1-U3-8b3f08f46416c0be9315277f6fb8f075.tgz
-rw-r--r-- 1 root wheel 4.2M Nov 10 02:58 freebsd-pkgdb-9.10.1-U4-d61691852044f7425790380209f30ef9.tgz
-rw-r--r-- 1 root wheel	62K Nov 10 02:58 freenas-pkg-tools-9.10.1-U3-8b3f08f46416c0be9315277f6fb8f075.tgz

We are currently using 9.10: Current Train: FreeNAS-9.10-STABLE (Installed OS)


The other issue we are seeing provides the following alerts:

FreeNAS Server1

WARNING: Oct. 10, 2016, 5:55 p.m. - Firmware version 3 does not match driver version 12 for /dev/mpr0. Please flash controller to P12 IT firmware. WARNING: Oct. 10, 2016, 5:55 p.m. - Firmware version 2 does not match driver version 12 for /dev/mpr1. Please flash controller to P12 IT firmware.

FreeNAS Server2

WARNING: Firmware version 6 does not match driver version 9 for /dev/mpr1. Please flash controller to P9 IT firmware. WARNING: Firmware version 3 does not match driver version 9 for /dev/mpr0. Please flash controller to P9 IT firmware.

Is this a FreeNAS 9.10 issue with SuperMicro servers, or should I ignore it?

 

[joeschmuck]

Rob,
Might I ask who setup all these FreeNAS servers and why they are not maintaining them? Sorry, I'm just assuming it wasn't you because you are asking a few basic questions that should be understood before building a FreeNAS server.

Also please list all the hardware for each server, ensure you include the internal components as well.

While I could just tell you that you need to flash the current IT firmware, I think it important to know the hardware before making that kind of move.

Cheers!

 

[BigDave]

The other issue we are seeing provides the following alerts:

Freenas Server1

WARNING: Oct. 10, 2016, 5:55 p.m. - Firmware version 3 does not match driver version 12 for /dev/mpr0. Please flash controller to P12 IT firmware. WARNING: Oct. 10, 2016, 5:55 p.m. - Firmware version 2 does not match driver version 12 for /dev/mpr1. Please flash controller to P12 IT firmware.

Freenas Server2

WARNING: Firmware version 6 does not match driver version 9 for /dev/mpr1. Please flash controller to P9 IT firmware. WARNING: Firmware version 3 does not match driver version 9 for /dev/mpr0. Please flash controller to P9 IT firmware.

Is this a Freenas 9.10 issue with SuperMicro servers, or should I ignore it?

FreeNAS requires matching firmware/drivers for HBA cards. Ignoring these warnings can lead to serious problems.
Update your firmware ASAP.

 

[Rob G]

BigDave & joeschmuck thanks for the response. I inherited the Freenas environment for our company from our wmWare guru. I mainly focus on Networking and Windows servers, but have other duties assigned, with Freenas being one of them. I have 2 Freenas systems, one at our office, and one in our main data center offsite. They are SuperMicro servers with the below listed hardware. Anyway, when I updated (a couple months ago) to 9.10 stable, from 9.3 stable I started seeing the firmware/driver errors.

If more information is needed please let me know. I ran dmidecode, but maybe there is a better command to gather a hardware summary?

Offsite DC Freenas Server 1:

• Computer
  • System
    • Manufacturer: Supermicro
    • Product Name: SSG-5028R-E1CR12L
    • Serial No.: xxxxxx
  • BIOS
    • Vendor: American Megatrends Inc.
    • Version: 1.0a
    • Release: 09/19/2014
  • CPU
    • CPU1
      • Processor: Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz
      • Speed: 2400 MHz
      • Core: 6
      • Core Active: 6
      • Manufacturer: Intel
  • DIMM
    • DIMMC1
    • DIMMC2
    • DIMMD1
    • DIMMD2

Code:

BIOS Information
   Vendor: American Megatrends Inc.
   Version: 1.0a
   Release Date: 09/19/2014
   Address: 0xF0000
   Runtime Size: 64 kB
   ROM Size: 16384 kB
   Characteristics:
	 PCI is supported
	 BIOS is upgradeable
	 BIOS shadowing is allowed
	 Boot from CD is supported
	 Selectable boot is supported
	 BIOS ROM is socketed
	 EDD is supported
	 5.25"/1.2 MB floppy services are supported (int 13h)
	 3.5"/720 kB floppy services are supported (int 13h)
	 3.5"/2.88 MB floppy services are supported (int 13h)
	 Print screen service is supported (int 5h)
	 8042 keyboard services are supported (int 9h)
	 Serial services are supported (int 14h)
	 Printer services are supported (int 17h)
	 ACPI is supported
	 USB legacy is supported
	 BIOS boot specification is supported
	 Targeted content distribution is supported
	 UEFI is supported
   BIOS Revision: 5.6

Processor Information
   Socket Designation: CPU1
   Type: Central Processor
   Family: Xeon
   Manufacturer: Intel
   ID: F2 06 03 00 FF FB EB BF
   Signature: Type 0, Family 6, Model 63, Stepping 2
   Flags:
	 FPU (Floating-point unit on-chip)
	 VME (Virtual mode extension)
	 DE (Debugging extension)
	 PSE (Page size extension)
	 TSC (Time stamp counter)
	 MSR (Model specific registers)
	 PAE (Physical address extension)
	 MCE (Machine check exception)
	 CX8 (CMPXCHG8 instruction supported)
	 APIC (On-chip APIC hardware supported)
	 SEP (Fast system call)
	 MTRR (Memory type range registers)
	 PGE (Page global enable)
	 MCA (Machine check architecture)
	 CMOV (Conditional move instruction supported)
	 PAT (Page attribute table)
	 PSE-36 (36-bit page size extension)
	 CLFSH (CLFLUSH instruction supported)
	 DS (Debug store)
	 ACPI (ACPI supported)
	 MMX (MMX technology supported)
	 FXSR (FXSAVE and FXSTOR instructions supported)
	 SSE (Streaming SIMD extensions)
	 SSE2 (Streaming SIMD extensions 2)
	 SS (Self-snoop)
	 HTT (Multi-threading)
	 TM (Thermal monitor supported)
	 PBE (Pending break enabled)
   Version: Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz
   Voltage: 1.8 V
   External Clock: 100 MHz
   Max Speed: 4000 MHz
   Current Speed: 2400 MHz
   Status: Populated, Enabled
   Upgrade: Socket LGA2011-3
   L1 Cache Handle: 0x006A
   L2 Cache Handle: 0x006B
   L3 Cache Handle: 0x006C
   Serial Number: Not Specified
   Asset Tag: Not Specified
   Part Number: Not Specified
   Core Count: 6
   Core Enabled: 6
   Thread Count: 12
   Characteristics:
	 64-bit capable
	 Multi-Core
	 Hardware Thread
	 Execute Protection
	 Enhanced Virtualization
	 Power/Performance Control

Handle 0x006E, DMI type 13, 22 bytes
BIOS Language Information
   Language Description Format: Long
   Installable Languages: 1
	 en|US|iso8859-1
   Currently Installed Language: en|US|iso8859-1

Handle 0x006F, DMI type 127, 4 bytes
End Of Table

Onsite Freenas Server 2:

Firmware Revision : 01.51

Firmware Build Time : 06/28/2014 BMC MAC address :
BIOS Version : 1.0 System LAN1 MAC address :
BIOS Build Time : 07/02/2014 System LAN2 MAC address
System LAN3 MAC address :
System LAN4 MAC address :

Code:

# dmidecode 3.0
Scanning /dev/mem for entry point.
SMBIOS 2.8 present.
111 structures occupying 4444 bytes.
Table at 0x000EB000.

Handle 0x0000, DMI type 0, 24 bytes
BIOS Information
   Vendor: American Megatrends Inc.
   Version: 1.0
   Release Date: 07/02/2014
   Address: 0xF0000
   Runtime Size: 64 kB
   ROM Size: 16384 kB
   Characteristics:
	 PCI is supported
	 BIOS is upgradeable
	 BIOS shadowing is allowed
	 Boot from CD is supported
	 Selectable boot is supported
	 BIOS ROM is socketed
	 EDD is supported
	 5.25"/1.2 MB floppy services are supported (int 13h)
	 3.5"/720 kB floppy services are supported (int 13h)
	 3.5"/2.88 MB floppy services are supported (int 13h)
	 Print screen service is supported (int 5h)
	 8042 keyboard services are supported (int 9h)
	 Serial services are supported (int 14h)
	 Printer services are supported (int 17h)
	 ACPI is supported
	 USB legacy is supported
	 BIOS boot specification is supported
	 Targeted content distribution is supported
	 UEFI is supported
   BIOS Revision: 5.6

Processor Information
   Socket Designation: CPU1
   Type: Central Processor
   Family: Xeon
   Manufacturer: Intel
   ID: F2 06 03 00 FF FB EB BF
   Signature: Type 0, Family 6, Model 63, Stepping 2
   Flags:
	 FPU (Floating-point unit on-chip)
	 VME (Virtual mode extension)
	 DE (Debugging extension)
	 PSE (Page size extension)
	 TSC (Time stamp counter)
	 MSR (Model specific registers)
	 PAE (Physical address extension)
	 MCE (Machine check exception)
	 CX8 (CMPXCHG8 instruction supported)
	 APIC (On-chip APIC hardware supported)
	 SEP (Fast system call)
	 MTRR (Memory type range registers)
	 PGE (Page global enable)
	 MCA (Machine check architecture)
	 CMOV (Conditional move instruction supported)
	 PAT (Page attribute table)
	 PSE-36 (36-bit page size extension)
	 CLFSH (CLFLUSH instruction supported)
	 DS (Debug store)
	 ACPI (ACPI supported)
	 MMX (MMX technology supported)
	 FXSR (FXSAVE and FXSTOR instructions supported)
	 SSE (Streaming SIMD extensions)
	 SSE2 (Streaming SIMD extensions 2)
	 SS (Self-snoop)
	 HTT (Multi-threading)
	 TM (Thermal monitor supported)
	 PBE (Pending break enabled)
   Version: Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz
   Voltage: 1.8 V
   External Clock: 100 MHz
   Max Speed: 4000 MHz
   Current Speed: 2400 MHz
   Status: Populated, Enabled
   Upgrade: Socket LGA2011-3
   L1 Cache Handle: 0x0069
   L2 Cache Handle: 0x006A
   L3 Cache Handle: 0x006B
   Serial Number: Not Specified
   Asset Tag: Not Specified
   Part Number: Not Specified
   Core Count: 6
   Core Enabled: 6
   Thread Count: 12
   Characteristics:
	 64-bit capable
	 Multi-Core
	 Hardware Thread
	 Execute Protection
	 Enhanced Virtualization
	 Power/Performance Control

Handle 0x006D, DMI type 13, 22 bytes
BIOS Language Information
   Language Description Format: Long
   Installable Languages: 1
	 en|US|iso8859-1
   Currently Installed Language: en|US|iso8859-1

Handle 0x006E, DMI type 127, 4 bytes
End Of Table

 

[SweetAndLow]

What type of disk controlled do these systems have?

Sent from my Nexus 5X using Tapatalk

 

[Rob G]

Hi @SweetAndLow Looks like Avago Tech LSI SAS3008 for both servers. They do have different driver levels though, but the firmware looks to be the same.


Offsite Freenas Server 1:

Code:

# dmesg | grep mpr
mpr0: <Avago Technologies (LSI) SAS3008> port 0xe000-0xe0ff mem 0xfb400000-0xfb40ffff irq 26 at device 0.0 on pci1
mpr0: IOCFacts   :
mpr0: Firmware: 03.00.06.136, Driver: 09.255.01.00-fbsd
mpr0: IOCCapabilities: 6985c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,IR,MSIXIndex>
mpr1: <Avago Technologies (LSI) SAS3008> port 0xc000-0xc0ff mem 0xfb200000-0xfb20ffff irq 16 at device 0.0 on pci7
mpr1: IOCFacts   :
mpr1: Firmware: 06.00.00.00, Driver: 09.255.01.00-fbsd
mpr1: IOCCapabilities: 6985c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,IR,MSIXIndex>
mpr1: SAS Address for SATA device = d2634236d9c3bd6f
mpr1: SAS Address for SATA device = 9c944925a6a1cf83
mpr1: SAS Address for SATA device = 9c944925a7c0ce9e
mpr1: SAS Address for SATA device = 8b96584dc293ab7f
mpr1: SAS Address for SATA device = 8b944a25caa6c6a1
mpr1: SAS Address for SATA device = 8b94553fa7a9d3a7
mpr1: SAS Address for SATA device = 8b944a26afbabfa1
mpr1: SAS Address for SATA device = 8b995b49ccb8c97f
mpr1: SAS Address for SATA device = 8b94492cccb8b0a1
mpr1: SAS Address for SATA device = d262583401d9da8d
mpr1: SAS Address for SATA device = 9c944929c6bbb297
mpr1: SAS Address for SATA device = 8b94553ea9b6d19d
mpr1: SAS Address from SAS device page0 = 5003048001a19500
mpr1: SAS Address from SATA device = d2634236d9c3bd6f
mpr1: Found device <81<SataDev>,End Device> <12.0Gbps> handle<0x000a> enclosureHandle<0x0002> slot 0
mpr1: At enclosure level 0 and connector name (  )
mpr1: SAS Address from SAS device page0 = 5003048001a19501
mpr1: SAS Address from SATA device = 9c944925a6a1cf83
mpr1: Found device <81<SataDev>,End Device> <12.0Gbps> handle<0x000b> enclosureHandle<0x0002> slot 1
mpr1: At enclosure level 0 and connector name (  )
mpr1: SAS Address from SAS device page0 = 5003048001a19502
mpr1: SAS Address from SATA device = 9c944925a7c0ce9e
mpr1: Found device <81<SataDev>,End Device> <12.0Gbps> handle<0x000c> enclosureHandle<0x0002> slot 2
mpr1: At enclosure level 0 and connector name (  )
mpr1: SAS Address from SAS device page0 = 5003048001a19503
mpr1: SAS Address from SATA device = 8b96584dc293ab7f
mpr1: Found device <81<SataDev>,End Device> <12.0Gbps> handle<0x000d> enclosureHandle<0x0002> slot 3
mpr1: At enclosure level 0 and connector name (  )
mpr1: SAS Address from SAS device page0 = 5003048001a19504
mpr1: SAS Address from SATA device = 8b944a25caa6c6a1
mpr1: Found device <81<SataDev>,End Device> <12.0Gbps> handle<0x000e> enclosureHandle<0x0002> slot 4
mpr1: At enclosure level 0 and connector name (  )
mpr1: SAS Address from SAS device page0 = 5003048001a19505
mpr1: SAS Address from SATA device = 8b94553fa7a9d3a7
mpr1: Found device <81<SataDev>,End Device> <12.0Gbps> handle<0x000f> enclosureHandle<0x0002> slot 5
mpr1: At enclosure level 0 and connector name (  )
mpr1: SAS Address from SAS device page0 = 5003048001a19506
mpr1: SAS Address from SATA device = 8b944a26afbabfa1
mpr1: Found device <81<SataDev>,End Device> <12.0Gbps> handle<0x0010> enclosureHandle<0x0002> slot 6
mpr1: At enclosure level 0 and connector name (  )
mpr1: SAS Address from SAS device page0 = 5003048001a19507
mpr1: SAS Address from SATA device = 8b995b49ccb8c97f
mpr1: Found device <81<SataDev>,End Device> <12.0Gbps> handle<0x0011> enclosureHandle<0x0002> slot 7
mpr1: At enclosure level 0 and connector name (  )
mpr1: SAS Address from SAS device page0 = 5003048001a19508
mpr1: SAS Address from SATA device = 8b94492cccb8b0a1
mpr1: Found device <81<SataDev>,End Device> <12.0Gbps> handle<0x0012> enclosureHandle<0x0002> slot 8
mpr1: At enclosure level 0 and connector name (  )
mpr1: SAS Address from SAS device page0 = 5003048001a19509
mpr1: SAS Address from SATA device = d262583401d9da8d
mpr1: Found device <81<SataDev>,End Device> <12.0Gbps> handle<0x0013> enclosureHandle<0x0002> slot 9
mpr1: At enclosure level 0 and connector name (  )
mpr1: SAS Address from SAS device page0 = 5003048001a1950a
mpr1: SAS Address from SATA device = 9c944929c6bbb297
mpr1: Found device <81<SataDev>,End Device> <12.0Gbps> handle<0x0014> enclosureHandle<0x0002> slot 10
mpr1: At enclosure level 0 and connector name (  )
mpr1: SAS Address from SAS device page0 = 5003048001a1950b
mpr1: SAS Address from SATA device = 8b94553ea9b6d19d
mpr1: Found device <81<SataDev>,End Device> <12.0Gbps> handle<0x0015> enclosureHandle<0x0002> slot 11
mpr1: At enclosure level 0 and connector name (  )
mpr1: SAS Address from SAS device page0 = 5003048001a1953d
mpr1: Found device <4411<SmpInit,SspTarg,SepDev>,End Device> <12.0Gbps> handle<0x0016> enclosureHandle<0x0002> slot 12
mpr1: At enclosure level 0 and connector name (  )
ses0 at mpr1 bus 0 scbus5 target 22 lun 0
da1 at mpr1 bus 0 scbus5 target 11 lun 0
da0 at mpr1 bus 0 scbus5 target 10 lun 0
da3 at mpr1 bus 0 scbus5 target 13 lun 0
da2 at mpr1 bus 0 scbus5 target 12 lun 0
da5 at mpr1 bus 0 scbus5 target 15 lun 0
da8 at mpr1 bus 0 scbus5 target 18 lun 0
da4 at mpr1 bus 0 scbus5 target 14 lun 0
da7 at mpr1 bus 0 scbus5 target 17 lun 0
da6 at mpr1 bus 0 scbus5 target 16 lun 0
da9 at mpr1 bus 0 scbus5 target 19 lun 0
da10 at mpr1 bus 0 scbus5 target 20 lun 0
da11 at mpr1 bus 0 scbus5 target 21 lun 0

Onsite Freenas Server 2:

Code:

# dmesg | grep mpr
mpr0: <Avago Technologies (LSI) SAS3008> port 0xe000-0xe0ff mem 0xfb400000-0xfb40ffff irq 26 at device 0.0 on pci1
mpr0: IOCFacts   :
mpr0: Firmware: 03.00.06.136, Driver: 13.00.00.00-fbsd
mpr0: IOCCapabilities: 6985c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,IR,MSIXIndex>
mpr1: <Avago Technologies (LSI) SAS3008> port 0xc000-0xc0ff mem 0xfb200000-0xfb20ffff irq 16 at device 0.0 on pci7
mpr1: IOCFacts   :
mpr1: Firmware: 02.00.02.00, Driver: 13.00.00.00-fbsd
mpr1: IOCCapabilities: 2985c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,IR,MSIXIndex>
mpr1: SAS Address for SATA device = d27e6453ddd4db8b
mpr1: SAS Address for SATA device = d27a5b49d9dbba89
mpr1: SAS Address for SATA device = d2755344fdc0cc75
mpr1: SAS Address for SATA device = d2765556eee2cf75
mpr1: SAS Address for SATA device = d277425bfed2ca96
mpr1: SAS Address for SATA device = d2765556eec9e06e
mpr1: SAS Address for SATA device = d280535decd0be6f
mpr1: SAS Address for SATA device = 9e944b2dc6a2d1a1
mpr1: SAS Address for SATA device = d280535decbfcb80
mpr1: SAS Address for SATA device = d2765556eeccbb80
mpr1: SAS Address for SATA device = d280535deccccb87
mpr1: SAS Address for SATA device = d2765556eedbc881
mpr1: SAS Address from SAS device page0 = 50030480015cb600
mpr1: SAS Address from SATA device = d27e6453ddd4db8b
mpr1: Found device <81<SataDev>,End Device> <12.0Gbps> handle<0x000a> enclosureHandle<0x0002> slot 0
mpr1: SAS Address from SAS device page0 = 50030480015cb601
mpr1: SAS Address from SATA device = d27a5b49d9dbba89
mpr1: Found device <81<SataDev>,End Device> <12.0Gbps> handle<0x000b> enclosureHandle<0x0002> slot 1
mpr1: SAS Address from SAS device page0 = 50030480015cb602
mpr1: SAS Address from SATA device = d2755344fdc0cc75
mpr1: Found device <81<SataDev>,End Device> <12.0Gbps> handle<0x000c> enclosureHandle<0x0002> slot 2
mpr1: SAS Address from SAS device page0 = 50030480015cb603
mpr1: SAS Address from SATA device = d2765556eee2cf75
mpr1: Found device <81<SataDev>,End Device> <12.0Gbps> handle<0x000d> enclosureHandle<0x0002> slot 3
mpr1: SAS Address from SAS device page0 = 50030480015cb604
mpr1: SAS Address from SATA device = d277425bfed2ca96
mpr1: Found device <81<SataDev>,End Device> <12.0Gbps> handle<0x000e> enclosureHandle<0x0002> slot 4
mpr1: SAS Address from SAS device page0 = 50030480015cb605
mpr1: SAS Address from SATA device = d2765556eec9e06e
mpr1: Found device <81<SataDev>,End Device> <12.0Gbps> handle<0x000f> enclosureHandle<0x0002> slot 5
mpr1: SAS Address from SAS device page0 = 50030480015cb606
mpr1: SAS Address from SATA device = d280535decd0be6f
mpr1: Found device <81<SataDev>,End Device> <12.0Gbps> handle<0x0010> enclosureHandle<0x0002> slot 6
mpr1: SAS Address from SAS device page0 = 50030480015cb607
mpr1: SAS Address from SATA device = 9e944b2dc6a2d1a1
mpr1: Found device <81<SataDev>,End Device> <12.0Gbps> handle<0x0011> enclosureHandle<0x0002> slot 7
mpr1: SAS Address from SAS device page0 = 50030480015cb608
mpr1: SAS Address from SATA device = d280535decbfcb80
mpr1: Found device <81<SataDev>,End Device> <12.0Gbps> handle<0x0012> enclosureHandle<0x0002> slot 8
mpr1: SAS Address from SAS device page0 = 50030480015cb609
mpr1: SAS Address from SATA device = d2765556eeccbb80
mpr1: Found device <81<SataDev>,End Device> <12.0Gbps> handle<0x0013> enclosureHandle<0x0002> slot 9
mpr1: SAS Address from SAS device page0 = 50030480015cb60a
mpr1: SAS Address from SATA device = d280535deccccb87
mpr1: Found device <81<SataDev>,End Device> <12.0Gbps> handle<0x0014> enclosureHandle<0x0002> slot 10
mpr1: SAS Address from SAS device page0 = 50030480015cb60b
mpr1: SAS Address from SATA device = d2765556eedbc881
mpr1: Found device <81<SataDev>,End Device> <12.0Gbps> handle<0x0015> enclosureHandle<0x0002> slot 11
mpr1: SAS Address from SAS device page0 = 50030480015cb63d
mpr1: Found device <4411<SmpInit,SspTarg,SepDev>,End Device> <12.0Gbps> handle<0x0016> enclosureHandle<0x0002> slot 12
ses0 at mpr1 bus 0 scbus5 target 22 lun 0
da2 at mpr1 bus 0 scbus5 target 12 lun 0
da0 at mpr1 bus 0 scbus5 target 10 lun 0
da1 at mpr1 bus 0 scbus5 target 11 lun 0
da5 at mpr1 bus 0 scbus5 target 15 lun 0
da8 at mpr1 bus 0 scbus5 target 18 lun 0
da11 at mpr1 bus 0 scbus5 target 21 lun 0
da3 at mpr1 bus 0 scbus5 target 13 lun 0
da6 at mpr1 bus 0 scbus5 target 16 lun 0
da4 at mpr1 bus 0 scbus5 target 14 lun 0
da7 at mpr1 bus 0 scbus5 target 17 lun 0
da10 at mpr1 bus 0 scbus5 target 20 lun 0
da9 at mpr1 bus 0 scbus5 target 19 lun 0
   (noperiph:mpr1:0:4294967295:0): SMID 1 Aborting command 0xfffffe00012801f0
mpr1: Sending reset from mprsas_send_abort for target ID 19
   (da9:mpr1:0:19:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 length 0 SMID 347 terminated ioc 804b scsi 0 sta(da9:mpr1:0:19:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00
(da9:mpr1:0:19:0): CAM status: Command timeout
mpr1: (da9:Unfreezing devq for target ID 19
mpr1:0:19:0): Retrying command
(da9:mpr1:0:19:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00
(da9:mpr1:0:19:0): CAM status: CCB request completed with an error
(da9:mpr1:0:19:0): Retrying command
(da9:mpr1:0:19:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00
(da9:mpr1:0:19:0): CAM status: SCSI Status Error
(da9:mpr1:0:19:0): SCSI status: Check Condition
(da9:mpr1:0:19:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)
(da9:mpr1:0:19:0): Error 6, Retries exhausted
(da9:mpr1:0:19:0): Invalidating pack
   (noperiph:mpr1:0:4294967295:0): SMID 2 Aborting command 0xfffffe0001284780
mpr1: Sending reset from mprsas_send_abort for target ID 14
   (da4:mpr1:0:14:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 length 0 SMID 465 terminated ioc 804b scsi 0 state c xfer 0
   (da4:mpr1:0:14:0): READ(10). CDB: 28 00 81 4f 59 38 00 00 28 00 length 20480 SMID 599 terminated ioc 804b scsi 0 state c xfer(da4:mpr1:0:14:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00
(da4:mpr1:0:14:0): CAM status: CCB request completed with an error
   (da4:mpr1:0:14:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 length 0 SMID 1014 terminated ioc 804b scsi 0 st(da4:ate c xfer 0
mpr1:0:mpr1: 14:Unfreezing devq for target ID 14
(da4:mpr1:0:14:0): READ(10). CDB: 28 00 81 4f 59 38 00 00 28 00
(da4:mpr1:0:14:0): CAM status: CCB request completed with an error
(da4:mpr1:0:14:0): Retrying command
(da4:mpr1:0:14:0): WRITE(10). CDB: 2a 00 7a 1e a7 70 00 00 50 00
(da4:mpr1:0:14:0): CAM status: Command timeout
(da4:mpr1:0:14:0): Retrying command
(da4:mpr1:0:14:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00
(da4:mpr1:0:14:0): CAM status: CCB request completed with an error
(da4:mpr1:0:14:0): Retrying command
(da4:mpr1:0:14:0): WRITE(10). CDB: 2a 00 7a 1e a7 70 00 00 50 00
(da4:mpr1:0:14:0): CAM status: SCSI Status Error
(da4:mpr1:0:14:0): SCSI status: Check Condition
(da4:mpr1:0:14:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)
(da4:mpr1:0:14:0): Retrying command (per sense data)
mpr1: mprsas_prepare_remove: Sending reset for target ID 19
da9 at mpr1 bus 0 scbus5 target 19 lun 0
mpr1: clearing target 19 handle 0x0013
mpr1: Unfreezing devq for target ID 19
(da9:mpr1:0:19:0): Periph destroyed
mpr1: SAS Address for SATA device = d2655a58fbbeba92
mpr1: SAS Address from SAS device page0 = 50030480015cb609
mpr1: SAS Address from SATA device = d2655a58fbbeba92
mpr1: Found device <81<SataDev>,End Device> <12.0Gbps> handle<0x0013> enclosureHandle<0x0002> slot 9
da9 at mpr1 bus 0 scbus5 target 19 lun 0

 

[Rob G]

@SweetAndLow Correction, mpr0 firmware for both servers is the same, but different Driver level, and mpr1 is different firmware and Driver level.

Offsite Freenas Server 1:

Code:

# dmesg | grep mpr
mpr0: <Avago Technologies (LSI) SAS3008> port 0xe000-0xe0ff mem 0xfb400000-0xfb40ffff irq 26 at device 0.0 on pci1
mpr0: IOCFacts   :
mpr0: Firmware: 03.00.06.136, Driver: 09.255.01.00-fbsd
mpr0: IOCCapabilities: 6985c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,IR,MSIXIndex>
mpr1: <Avago Technologies (LSI) SAS3008> port 0xc000-0xc0ff mem 0xfb200000-0xfb20ffff irq 16 at device 0.0 on pci7
mpr1: IOCFacts   :
mpr1: Firmware: 06.00.00.00, Driver: 09.255.01.00-fbsd

Onsite Freenas Server 2:

Code:

# dmesg | grep mpr
mpr0: <Avago Technologies (LSI) SAS3008> port 0xe000-0xe0ff mem 0xfb400000-0xfb40ffff irq 26 at device 0.0 on pci1
mpr0: IOCFacts   :
mpr0: Firmware: 03.00.06.136, Driver: 13.00.00.00-fbsd
mpr0: IOCCapabilities: 6985c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,IR,MSIXIndex>
mpr1: <Avago Technologies (LSI) SAS3008> port 0xc000-0xc0ff mem 0xfb200000-0xfb20ffff irq 16 at device 0.0 on pci7
mpr1: IOCFacts   :
mpr1: Firmware: 02.00.02.00, Driver: 13.00.00.00-fbsd

 

[MatthewSteinhoff]

when I updated (a couple months ago) to 9.10 stable, from 9.3 stable I started seeing the firmware/driver errors.

Doh! Staggered upgrades and making sure the first system is fully-functional without errors before updating the second system should be considered a best practice. Especially when you're taking over an unfamiliar system outside your core expertise.

Flogging aside, I'm not terrified by mismatched firmware. Concerned, yes. Terrified, no. You're going to need to grab the recommended firmware from either Supermicro (first choice) or Avagotech (second choice) and perform the upgrade.

The firmware upgrade procedure can be time consuming and non-trivial but basically comes down to create bootable USB device, put firmware and tools on the USB device, boot to USB device, upgrade firmware, reboot, cross fingers.

Given that you're already playing with fire, do the least important server first, make sure you have the procedure perfected, wait a week or so then do the other server.

Cheers,
Matt

 

[SweetAndLow]

Driver is different because you are probably running 2 different versions of FreeNAS. You need your firmware to match your driver. Since you can't change the FreeNAS driver that means you need to change the firmware.

Sent from my Nexus 5X using Tapatalk

 

[joeschmuck]

You know, it's really too bad that the firmware must be changed on these cards all the time. It would be nice if FreeNAS was able to run with a few different versions of the drivers from say the last 5 years just to ensure someone doesn't need to be reflashing all the time. But, it is what it is.

 

[SweetAndLow]

You know, it's really too bad that the firmware must be changed on these cards all the time. It would be nice if FreeNAS was able to run with a few different versions of the drivers from say the last 5 years just to ensure someone doesn't need to be reflashing all the time. But, it is what it is.

Just don't update all the time.

Sent from my Nexus 5X using Tapatalk

 

[Rob G]

We really don't upgrade all the time. We thought we were safe going to 9.10 stable, and it turned out differently, so in hindsight, I would have stayed on 9.3 Stable and avoided this mess. With both servers having very important data on them, we can't risk losing either one. We should probably move to another support storage system, but the requires space and $$.

 

[MatthewSteinhoff]

We should probably move to another support storage system, but the requires space and $$.

Take a deep breath, @Rob G . FreeNAS is a robust, reliable storage system and your problem stems entirely from human error. No storage system can prevent system administrators from destabilizing the platform.

Upgrade the HBA firmware as above and you'll be fine.

Cheers,
Matt

 

[SweetAndLow]

We really don't upgrade all the time. We thought we were safe going to 9.10 stable, and it turned out differently, so in hindsight, I would have stayed on 9.3 Stable and avoided this mess. With both servers having very important data on them, we can't risk losing either one. We should probably move to another support storage system, but the requires space and $$.

Should read the change logs for every update and test it before using it in production. Updating the firmware is no big deal, probably take your a day to do all the server's once you know how.

Sent from my Nexus 5X using Tapatalk

 

[Rob G]

@SweetAndLow That sounds encouraging. I will study up on the firmware updates and then proceed one server at a time. For future reference, you can bet I will double check the release notes for firmware/driver compatibility. Thanks for all the advice.

On a side note, I have a FreeNAS system at home running on an older desktop with AMD proc, 8 GB RAM, and a couple drives. So far it has been very stable and has taken every update. I was thinking of doing something a bit better with an Intel proc, and maybe 4 redundant swap-able drives, but I need to find time to invest in the project. I am going to start a backup to Crashplan, so I don't lose data as well.

 

[SweetAndLow]

@SweetAndLow That sounds encouraging. I will study up on the firmware updates and then proceed one server at a time. For future reference, you can bet I will double check the release notes for firmware/driver compatibility. Thanks for all the advice.

On a side note, I have a Freenas system at home running on an older desktop with AMD proc, 8 GB RAM, and a couple drives. So far it has been very stable and has taken every update. I was thinking of doing something a bit better with an Intel proc, and maybe 4 redundant swap-able drives, but I need to find time to invest in the project. I am going to start a backup to Crashplan, so I don't lose data as well.

You can also just rollback to the old boot environment. Use the GUI to pick the one you want and reboot. This way you can prepare for the update on your own time and not be rushed.

Sent from my Nexus 5X using Tapatalk

 

[danb35]

You can also just rollback to the old boot environment.

But that won't really solve the problem. Sure, the alert will go away, but the situation that caused it (mismatch firmware and driver) won't. Both of those controllers are on ancient firmware and need to be updated in any case.

And there's no reason to burn a bootable USB stick to flash the firmware; the sas3flash utility is part of FreeNAS.

 

[MatthewSteinhoff]

the sas3flash utility is part of FreeNAS.

I'm one for five when using sas2flash to update LSI cards. Maybe sas3flash is better?

You're right though. It wouldn't hurt to try it from the server first.

Cheers,
Matt

 

[SweetAndLow]

I'm one for five when using sas2flash to update LSI cards. Maybe sas3flash is better?

You're right though. It wouldn't hurt to try it from the server first.

Cheers,
Matt

Depends on if you have a sas2 or sas3 card.

Sent from my Nexus 5X using Tapatalk

 

[danb35]

I'm one for five when using sas2flash to update LSI cards. Maybe sas3flash is better?

You'd use sas2flash for a SAS2 card, and sas3flash for a SAS3 card. But yes, I've seen others have trouble with the process as well. No idea what makes it more difficult for others.