kernel: mfi0: IOCTL 0x40704d07 not handled

[ahgu]

I am getting this error :
kernel: mfi0: IOCTL 0x40704d07 not handled
several times a day.
Anyone know the cause of it? I don't seem to have a bad disk. Wondering what is wrong?

Here is the storcli dump:




Generating detailed summary of the adapter, it may take a while to complete.

CLI Version = 007.1316.0000.0000 Mar 12, 2020
Operating system = FreeBSD 12.3-RELEASE-p5
Controller = 0
Status = Failure
Description = None

Detailed Status :
===============

------------------------------------------------
Ctrl Status ErrCd ErrMsg
------------------------------------------------
0 Failed 255 Failed to get Bus Information
------------------------------------------------



Basics :
======
Controller = 0
Model = ServeRAID M5014 SAS/SATA Controller
Serial Number = SV12302325
Current Controller Date/Time = 07/25/2022, 02:17:21
Current System Date/time = 07/25/2022, 02:17:23
SAS Address = 500605b0038012b0
PCI Address = NA
Mfg Date = 05/29/12
Rework Date = 00/00/00
Revision No = 71B


Version :
=======
Firmware Package Build = 12.13.0-0179
Firmware Version = 2.130.383-2854
Bios Version = 3.29.00_4.14.05.00_0x05290000
Preboot CLI Version = 04.04-020:#%00009
WebBIOS Version = 6.0-52-e_48-Rel
NVDATA Version = 2.09.03-0046
Boot Block Version = 2.02.00.00-0000
Bootloader Version = 09.250.01.219
Driver Name = N/A
Driver Version = N/A



Bus :
===
Vendor Id = 0x1000
Device Id = 0x79
SubVendor Id = 0x1014
SubDevice Id = 0x3C7
Host Interface = PCI-E
Device Interface = SAS-6G


Pending Images in Flash :
=======================
Image name = No pending images


Status :
======
Controller Status = Optimal
Memory Correctable Errors = 0
Memory Uncorrectable Errors = 0
ECC Bucket Count = 0
Any Offline VD Cache Preserved = No
BBU Status = 0
PD Firmware Download in progress = No
Support PD Firmware Download = No
Lock Key Assigned = No
Failed to get lock key on bootup = No
Lock key has not been backed up = No
Bios was not detected during boot = No
Controller must be rebooted to complete security operation = No
A rollback operation is in progress = No
At least one PFK exists in NVRAM = No
SSC Policy is WB = No
Controller has booted into safe mode = No
Controller shutdown required = No


Supported Adapter Operations :
============================
Rebuild Rate = Yes
CC Rate = Yes
BGI Rate = Yes
Reconstruct Rate = Yes
Patrol Read Rate = Yes
Alarm Control = Yes
Cluster Support = No
BBU = Yes
Spanning = Yes
Dedicated Hot Spare = Yes
Revertible Hot Spares = Yes
Foreign Config Import = Yes
Self Diagnostic = Yes
Allow Mixed Redundancy on Array = No
Global Hot Spares = Yes
Deny SCSI Passthrough = No
Deny SMP Passthrough = No
Deny STP Passthrough = No
Support more than 8 Phys = Yes
FW and Event Time in GMT = No
Support Enhanced Foreign Import = Yes
Support Enclosure Enumeration = Yes
Support Allowed Operations = Yes
Abort CC on Error = Yes
Support Multipath = Yes
Support Odd & Even Drive count in RAID1E = No
Support Security = Yes
Support Config Page Model = Yes
Support the OCE without adding drives = Yes
Support EKM = Yes
Snapshot Enabled = No
Support PFK = No
Support PI = No
Support Ld BBM Info = No
Support Shield State = No
Block SSD Write Disk Cache Change = No
Support Suspend Resume BG ops = No
Support Emergency Spares = Yes
Support Set Link Speed = No
Support Boot Time PFK Change = No
Support JBOD = No
Disable Online PFK Change = No
Support Perf Tuning = No
Support SSD PatrolRead = Yes
Real Time Scheduler = Yes
Support Reset Now = Yes
Support Emulated Drives = No
Headless Mode = Yes
Dedicated HotSpares Limited = No
Point In Time Progress = No
Extended LD = No
Support Uneven span = No
Support Config Auto Balance = No
Support Maintenance Mode = No
Support Diagnostic results = No
Support Ext Enclosure = No
Support Sesmonitoring = No
Support SecurityonJBOD = No
Support ForceFlash = No
Support DisableImmediateIO = No
Support LargeIOSupport = No
Support DrvActivityLEDSetting = No
Support FlushWriteVerify = No
Support CPLDUpdate = No
Support ForceTo512e = No
Support discardCacheDuringLDDelete = No
Support JBOD Write cache = No
Support Large QD Support = No
Support Ctrl Info Extended = No
Support IButton less = No
Support AES Encryption Algorithm = No
Support Encrypted MFC = No
Support Snapdump = No
Support Force Personality Change = No
Support Dual Fw Image = No
Support PSOC Update = No
Support Secure Boot = No
Support Debug Queue = No
Support Least Latency Mode = Yes
Support OnDemand Snapdump = No
Support Clear Snapdump = No
Support PHY current speed = No
Support Lane current speed = No
Support NVMe Width = No
Support Lane DeviceType = No
Support Extended Drive performance Monitoring = No


Enterprise Key management :
=========================
Capability = Supported
Boot Agent = Not Available
Configured = No


Supported PD Operations :
=======================
Force Online = Yes
Force Offline = Yes
Force Rebuild = Yes
Deny Force Failed = No
Deny Force Good/Bad = No
Deny Missing Replace = No
Deny Clear = No
Deny Locate = No
Support Power State = No
Set Power State For Cfg = No
Support T10 Power State = No
Support Temperature = Yes
NCQ = No
Support Max Rate SATA = No
Support Degraded Media = No
Support Parallel FW Update = No
Support Drive Crypto Erase = No
Support SSD Wear Gauge = No


Supported VD Operations :
=======================
Read Policy = Yes
Write Policy = Yes
IO Policy = Yes
Access Policy = Yes
Disk Cache Policy = Yes
Reconstruction = Yes
Deny Locate = No
Deny CC = No
Allow Ctrl Encryption = No
Enable LDBBM = No
Support FastPath = No
Performance Metrics = Yes
Power Savings = No
Support Powersave Max With Cache = No
Support Breakmirror = No
Support SSC WriteBack = No
Support SSC Association = Yes
Support VD Hide = No
Support VD Cachebypass = No
Support VD discardCacheDuringLDDelete = No
Support VD Scsi Unmap = No


HwCfg :
=====
ChipRevision = B4
BatteryFRU = N/A
Front End Port Count = 0
Backend Port Count = 8
BBU = Present
Alarm = Disable
Serial Debugger = Present
NVRAM Size = 32KB
Flash Size = 8MB
On Board Memory Size = 256MB
CacheVault Flash Size = NA
TPM = Absent
Upgrade Key = Present
On Board Expander = Absent
Temperature Sensor for ROC = Absent
Temperature Sensor for Controller = Absent
Upgradable CPLD = Absent
Upgradable PSOC = Absent
Current Size of CacheCade (GB) = 0
Current Size of FW Cache (MB) = 96


Policies :
========

Policies Table :
==============

------------------------------------------------
Policy Current Default
------------------------------------------------
Predictive Fail Poll Interval 300 sec
Interrupt Throttle Active Count 16
Interrupt Throttle Completion 50 us
Rebuild Rate 30 % 30%
PR Rate 30 % 30%
BGI Rate 30 % 30%
Check Consistency Rate 30 % 30%
Reconstruction Rate 30 % 30%
Cache Flush Interval 4s
------------------------------------------------

Flush Time(Default) = 4s
Drive Coercion Mode = 1GB
Auto Rebuild = On
Battery Warning = On
ECC Bucket Size = 15
ECC Bucket Leak Rate (hrs) = 24
Restore Hot Spare on Insertion = Off
Expose Enclosure Devices = On
Maintain PD Fail History = On
Reorder Host Requests = On
Auto detect BackPlane = SGPIO/i2c SEP
Load Balance Mode = Auto
Security Key Assigned = Off
Disable Online Controller Reset = Off
Use drive activity for locate = Off


Boot :
====
BIOS Enumerate VDs = 1
Stop BIOS on Error = On
Delay during POST = 4
Spin Down Mode = None
Enable Ctrl-R = No
Enable Web BIOS = Yes
Enable PreBoot CLI = Yes
Enable BIOS = Yes
Max Drives to Spinup at One Time = 4
Maximum number of direct attached drives to spin up in 1 min = 120
Delay Among Spinup Groups (sec) = 2
Allow Boot with Preserved Cache = Off


High Availability :
=================
Topology Type = None
Cluster Permitted = No
Cluster Active = No


Defaults :
========
Phy Polarity = 0
Phy PolaritySplit = 0
Strip Size = 128 KB
Write Policy = WB
Read Policy = No Read Ahead
Cache When BBU Bad = Off
Cached IO = Off
VD PowerSave Policy = Controller Defined
Default spin down time (mins) = 30
Coercion Mode = 1 GB
ZCR Config = Unknown
Max Chained Enclosures = 16
Direct PD Mapping = No
Restore Hot Spare on Insertion = No
Expose Enclosure Devices = Yes
Maintain PD Fail History = Yes
Zero Based Enclosure Enumeration = No
Disable Puncturing = Yes
EnableLDBBM = No
DisableHII = No
Un-Certified Hard Disk Drives = Allow
SMART Mode = Mode 6
Enable LED Header = No
LED Show Drive Activity = No
Dirty LED Shows Drive Activity = No
EnableCrashDump = No
Disable Online Controller Reset = No
Treat Single span R1E as R10 = No
Power Saving option = Enabled
TTY Log In Flash = No
Auto Enhanced Import = No
BreakMirror RAID Support = No
Disable Join Mirror = No
Enable Shield State = No
Time taken to detect CME = 60 sec


Capabilities :
============
Supported Drives = SAS, SATA
RAID Level Supported = RAID0, RAID1(2 or more drives), RAID5, RAID6, RAID00, RAID10(2 or more drives per span), RAID50, RAID60
Enable JBOD = No
Mix in Enclosure = Allowed
Mix of SAS/SATA of HDD type in VD = Not Allowed
Mix of SAS/SATA of SSD type in VD = Not Allowed
Mix of SSD/HDD in VD = Not Allowed
SAS Disable = No
Max Arms Per VD = 32
Max Spans Per VD = 8
Max Arrays = 128
Max VD per array = 16
Max Number of VDs = 64
Max Parallel Commands = 1008
Max SGE Count = 80
Max Data Transfer Size = 8192 sectors
Max Strips PerIO = 42
Max Configurable CacheCade Size(GB) = 0
Max Transportable DGs = 0
Enable Snapdump = No
Enable SCSI Unmap = Yes
FDE Drive Mix Support = No
Min Strip Size = 8 KB
Max Strip Size = 1.000 MB


Scheduled Tasks :
===============
Consistency Check Reoccurrence = 168 hrs
Next Consistency check launch = 07/30/2022, 03:00:00
Patrol Read Reoccurrence = 168 hrs
Next Patrol Read launch = 07/30/2022, 03:00:00
Battery learn Reoccurrence = 672 hrs
Next Battery Learn = 08/20/2022, 16:00:00
OEMID = Lenovo

Drive Groups = 2

TOPOLOGY :
========

-----------------------------------------------------------------------------
DG Arr Row EID:Slot DID Type State BT Size PDC PI SED DS3 FSpace TR
-----------------------------------------------------------------------------
0 - - - - RAID1 Optl N 464.729 GB dsbl N N none N N
0 0 - - - RAID1 Optl N 464.729 GB dsbl N N none N N
0 0 0 :255 9 DRIVE Onln N 464.729 GB dsbl N N none - N
0 0 1 :255 11 DRIVE Onln N 464.729 GB dsbl N N none - N
1 - - - - RAID6 Optl N 10.913 TB dsbl N N none N N
1 0 - - - RAID6 Optl N 10.913 TB dsbl N N none N N
1 0 0 252:0 8 DRIVE Onln N 2.727 TB dsbl N N none - N
1 0 1 :255 10 DRIVE Onln N 2.727 TB dsbl N N none - N
1 0 2 252:1 12 DRIVE Onln N 2.727 TB dsbl N N none - N
1 0 3 252:3 13 DRIVE Onln N 2.727 TB dsbl N N none - N
1 0 4 252:2 14 DRIVE Onln N 2.727 TB dsbl N N none - N
1 0 5 :255 15 DRIVE Onln N 2.727 TB dsbl N N none - N
-----------------------------------------------------------------------------

DG=Disk Group Index|Arr=Array Index|Row=Row Index|EID=Enclosure Device ID
DID=Device ID|Type=Drive Type|Onln=Online|Rbld=Rebuild|Optl=Optimal|Dgrd=Degraded
Pdgd=Partially degraded|Offln=Offline|BT=Background Task Active
PDC=PD Cache|PI=Protection Info|SED=Self Encrypting Drive|Frgn=Foreign
DS3=Dimmer Switch 3|dflt=Default|Msng=Missing|FSpace=Free Space Present
TR=Transport Ready

Virtual Drives = 2

VD LIST :
=======

---------------------------------------------------------------
DG/VD TYPE State Access Consist Cache Cac sCC Size Name
---------------------------------------------------------------
0/0 RAID1 Optl RW Yes NRWBD - ON 464.729 GB
1/1 RAID6 Optl RW Yes NRWBD - ON 10.913 TB
---------------------------------------------------------------

VD=Virtual Drive| DG=Drive Group|Rec=Recovery
Cac=CacheCade|OfLn=OffLine|Pdgd=Partially Degraded|Dgrd=Degraded
Optl=Optimal||dflt=Default|RO=Read Only|RW=Read Write|HD=Hidden|TRANS=TransportReady|B=Blocked|
Consist=Consistent|R=Read Ahead Always|NR=No Read Ahead|WB=WriteBack|
AWB=Always WriteBack|WT=WriteThrough|C=Cached IO|D=Direct IO|sCC=Scheduled
Check Consistency

Physical Drives = 8

PD LIST :
=======

-----------------------------------------------------------------------------------
EID:Slt DID State DG Size Intf Med SED PI SeSz Model Sp Type
-----------------------------------------------------------------------------------
252:0 8 Onln 1 2.727 TB SATA HDD N N 512B ST3000DM001-1CH166 U -
252:1 12 Onln 1 2.727 TB SATA HDD N N 512B HGST HUS724030ALA640 U -
252:2 14 Onln 1 2.727 TB SATA HDD N N 512B ST3000DM001-1ER166 U -
252:3 13 Onln 1 2.727 TB SATA HDD N N 512B HGST HUS724030ALA640 U -
:255 10 Onln 1 2.727 TB SATA HDD N N 512B ST3000DM001-1ER166 U -
:255 9 Onln 0 464.729 GB SATA HDD N N 512B WDC WD5000AVDS-61U7B1 U -
:255 15 Onln 1 2.727 TB SATA HDD N N 512B ST3000DM001-1ER166 U -
:255 11 Onln 0 464.729 GB SATA HDD N N 512B WDC WD5000AVDS-61U7B1 U -
-----------------------------------------------------------------------------------

EID=Enclosure Device ID|Slt=Slot No.|DID=Device ID|DG=DriveGroup
DHS=Dedicated Hot Spare|UGood=Unconfigured Good|GHS=Global Hotspare
UBad=Unconfigured Bad|Sntze=Sanitize|Onln=Online|Offln=Offline|Intf=Interface
Med=Media Type|SED=Self Encryptive Drive|PI=Protection Info
SeSz=Sector Size|Sp=Spun|U=Up|D=Down|T=Transition|F=Foreign
UGUnsp=UGood Unsupported|UGShld=UGood shielded|HSPShld=Hotspare shielded
CFShld=Configured shielded|Cpybck=CopyBack|CBShld=Copyback Shielded
UBUnsp=UBad Unsupported|Rbld=Rebuild

Enclosures = 1

Enclosure LIST :
==============

--------------------------------------------------------------------
EID State Slots PD PS Fans TSs Alms SIM Port# ProdID VendorSpecific
--------------------------------------------------------------------
252 OK 4 4 0 0 0 0 1 - SGPIO
--------------------------------------------------------------------

EID=Enclosure Device ID |PD=Physical drive count |PS=Power Supply count|
TSs=Temperature sensor count |Alms=Alarm count |SIM=SIM Count ||ProdID=Product ID


BBU_Info :
========

-----------------------------------------------------------------------
Model State RetentionTime Temp Mode MfgDate Next Learn
-----------------------------------------------------------------------
iBBU08 Optimal 48 hours + 32C 4 2018/04/16 2022/08/20 16:48:27
-----------------------------------------------------------------------

Mode 4: 48+ Hrs retention with a non-transparent learn cycle
and balanced service life.

 

[Ericloewe]

Anyone know the cause of it?

The cause of it is that you're using a RAID controller (and the mfi driver). There are many reasons why this is considered an inappropriate setup and "mystery errors" is high on that list.

 

[sretalla]

What's all the noise about HBA's, and why can't I use a RAID controller?

This is relevant to FreeNAS and TrueNAS CORE. Some parts of it might also be relevant to Scale, but I don't really know how reliable the Linux drivers are. 1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with...

www.truenas.com

 

[ahgu]

The cause of it is that you're using a RAID controller (and the mfi driver). There are many reasons why this is considered an inappropriate setup and "mystery errors" is high on that list.

Would you please elaborate ? This is a RAID controller with a freeBSD driver, why inappropriate setup? What is considered as good setup?

 

[Ericloewe]

The document linked in Post #3 is a good start. The Introduction to ZFS also touches on the subject.

 

[ahgu]

The document linked in Post #3 is a good start. The Introduction to ZFS also touches on the subject.

I don't see anywhere talk about the IOCTL not handled error. I don't know what is the cause of it. I suspect it might be a failing disk. I noticed there is one disk that has seek error count.

I cannot change the setup now. If I get a new raid card, not sure it will recognize the old RAID 6 format. Best thing is to fix the error on the existing setup.

 

[jgreco]

Best thing is to fix the error on the existing setup.

No, that's not the best thing. The MFI stuff is known to have shred pools in the past, so the best thing is to evacuate your pool, yank the RAID card, install an HBA, create a proper pool, and get on with life. This is the "best thing" to do, even if it's inconvenient.

get a new raid card

In no case do you want a "new raid card". Please refer to my linked post above that discusses why you do not want to be using a RAID card. Using a RAID card is basically just asking for problems down the road.

 

[ahgu]

No, that's not the best thing. The MFI stuff is known to have shred pools in the past, so the best thing is to evacuate your pool, yank the RAID card, install an HBA, create a proper pool, and get on with life. This is the "best thing" to do, even if it's inconvenient.



In no case do you want a "new raid card". Please refer to my linked post above that discusses why you do not want to be using a RAID card. Using a RAID card is basically just asking for problems down the road.

What HBA card do you recommend for RAID 6? LSI 9240-8i good? Please recommend a good reliable one.

evaucate your pool: does that mean make a backup? will the new HBA card recognize the old RAID6 configuration?

 

[ahgu]

serveraid m5014 not HBA?


Controller = 0
Model = ServeRAID M5014 SAS/SATA Controller
Serial Number = SV12302325
Current Controller Date/Time = 07/25/2022, 02:17:21
Current System Date/time = 07/25/2022, 02:17:23
SAS Address = 500605b0038012b0
PCI Address = NA
Mfg Date = 05/29/12
Rework Date = 00/00/00
Revision No = 71B

 

[Ericloewe]

What HBA card do you recommend for RAID 6?

With ZFS, you would never be running hardware RAID (6, 1, whatever), so the first step is to strip down the HBA to the bare minimum: A device that speaks PCIe on one end and SATA/SAS on the other. ZFS will handle all the volume management, parity, etc.

LSI 9240-8i good

Kinda. You would need to crossflash it or get one already crossflashed, since the 9240 is a low-end RAID controller - although the hardware is also available under the 9211 model designation, which is a plain HBA. You could also buy a 9211 that was manufactured as such. Note that this is older SAS2 hardware - it's still fine, particularly if you're not running expanders or SSDs.

SAS3 stuff is what most people would aim for these days, since it's still current. The keyword there is LSI SAS 9300, although there are newer models (with expensive features of limited usefulness such as NVMe support, PCIe 4.0 support and high-density connectors).

 

[ahgu]

With ZFS, you would never be running hardware RAID (6, 1, whatever), so the first step is to strip down the HBA to the bare minimum: A device that speaks PCIe on one end and SATA/SAS on the other. ZFS will handle all the volume management, parity, etc.

Kinda. You would need to crossflash it or get one already crossflashed, since the 9240 is a low-end RAID controller - although the hardware is also available under the 9211 model designation, which is a plain HBA. You could also buy a 9211 that was manufactured as such. Note that this is older SAS2 hardware - it's still fine, particularly if you're not running expanders or SSDs.

SAS3 stuff is what most people would aim for these days, since it's still current. The keyword there is LSI SAS 9300, although there are newer models (with expensive features of limited usefulness such as NVMe support, PCIe 4.0 support and high-density connectors).

WHY is software raid in ZFS a good thing? That will take a lot of resource from the CPU. For Freebsd, I was always using UFS.
I always thought hardware raid is desired.

 

[Ericloewe]

The Introduction to ZFS is probably where you should start, since it directly addresses the core advantages of ZFS, though it is not written as a "ZFS vs. Hardware RAID" analysis.

 

[ahgu]

The Introduction to ZFS is probably where you should start, since it directly addresses the core advantages of ZFS, though it is not written as a "ZFS vs. Hardware RAID" analysis.

Well, I agree software RAID in ZFS have more control and easier to debug, but Hardware RAID will always be better in performance.
When I configure ZFS, should I choose software raid or use the RAID disk already created by the hardware?

 

[Samuel Tai]

Please do NOT use HW RAID with ZFS. You WILL lose data eventually, because ZFS and the RAID controller will be working at cross-purposes.

What's all the noise about HBA's, and why can't I use a RAID controller?

This is relevant to FreeNAS and TrueNAS CORE. Some parts of it might also be relevant to Scale, but I don't really know how reliable the Linux drivers are. 1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with...

www.truenas.com

 

[ahgu]

If I am only doing two disk raid-1, I don't need any LSI HBA or raid controller as long as I see the two hard drives on the BIOS.
Is that correct? Or Do I need get LSI HBA for some reason unknown to me?

 

[Samuel Tai]

If you can connect your disks directly to the motherboard SATA ports, you don't need an HBA, or your RAID controller.

 

[ahgu]

If you can connect your disks directly to the motherboard SATA ports, you don't need an HBA, or your RAID controller.

thank you, Mr. Tai.