LSI 3108 IT Mode

[CookiesLikeWhoa]

Hello all!

Searching around on the forum and on Google hasn't given the results I'm looking for so here we are! Does any one know if the on board LSI 3108 can be flashed to IT mode?

Thank you in advance!

 

[Ericloewe]

I'm fairly certain the answer is no.

However, I think that the LSI SAS3 stack allows for HBA operation even with megaraid firmware.

 

[CookiesLikeWhoa]

The few things I have managed to read say that. But there's nothing concrete about it :/


Sent from my iPhone using Tapatalk

 

[Ericloewe]

Honestly, if you get SMART data and nothing looks out of place during burn-in, I'd give it a try.

My realistic worst-case scenario is that it's slower than an SAS 3008, but safe.

 

[CookiesLikeWhoa]

Well shot an email to Supermicro Support. We shall what they have to say on the subject.

 

[neuro]

Did you get any answer from the Supermicro Support?

I did some tests with a X10DRC-LN4+ and FreeNAS 9.10.1. It's possible to configure the onboard LSI 3108 in JBOD mode and if you force FreeNAS to load the mrsas driver instead of mfi (see "mrsas driver instead of mfi driver") everything looks ok.

I did some basic testing on a Volume (2 x 6 disk raidz2), short/long smart tests, write a few TB data on it, took a lot of snapshots and executed some scrubs. No problems so far.

 

[CookiesLikeWhoa]

Yeah the answer I got from them was that it can be configured into a JBOD mode. But after probing some more they confirmed that it won't pass the smart data on.

 

[neuro]

I have just checked, it does only pass some SMART information, for example "Error counter log", "SMART Healt Status" but it does not pass the SMART attributes along with their raw values.

 

[CookiesLikeWhoa]

Yeah that's my fear is that it won't pass everything a long.

 

[Ericloewe]

I have just checked, it does only pass some SMART information, for example "Error counter log", "SMART Healt Status" but it does not pass the SMART attributes along with their raw values.

With what drives? Can you show us the output?

 

[neuro]

With what drives? Can you show us the output?

Drives are "HGST Ultrastar He8" SAS Version. I guess as long as the "SMART Self-test log" data is accessible it should be enough for the smart monitoring.

Code:

~# smartctl -a /dev/da0
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:  HGST
Product:  HUH728080AL5200
Revision:  A7J0
Compliance:  SPC-4
User Capacity:  8,001,563,222,016 bytes [8.00 TB]
Logical block size:  512 bytes
Physical block size:  4096 bytes
LU is fully provisioned
Rotation Rate:  7200 rpm
Form Factor:  3.5 inches
Logical Unit id:  0x5000cca23bd81b3c
Serial number:  2EKUUBRZ
Device type:  disk
Transport protocol:  SAS (SPL-3)
Local Time is:  Fri Oct 14 18:57:02 2016 CEST
SMART support is:  Available - device has SMART capability.
SMART support is:  Enabled
Temperature Warning:  Enabled

=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK

Current Drive Temperature:  30 C
Drive Trip Temperature:  85 C

Manufactured in week 05 of year 2016
Specified cycle count over device lifetime:  50000
Accumulated start-stop cycles:  25
Specified load-unload count over device lifetime:  600000
Accumulated load-unload cycles:  37
Elements in grown defect list: 0

Vendor (Seagate) cache information
  Blocks sent to initiator = 85624736448512

Error counter log:
  Errors Corrected by  Total  Correction  Gigabytes  Total
  ECC  rereads/  errors  algorithm  processed  uncorrected
  fast | delayed  rewrites  corrected  invocations  [10^9 bytes]  errors
read:  0  1  0  1  1771  514.891  0
write:  0  3  0  3  7199  170.982  0
verify:  0  0  0  0  1005  0.008  0

Non-medium error count:  0

SMART Self-test log
Num  Test  Status  segment  LifeTime  LBA_first_err [SK ASC ASQ]
  Description  number  (hours)
# 1  Background short  Completed  -  281  - [-  -  -]
# 2  Background short  Completed  -  257  - [-  -  -]
# 3  Background long  Completed  -  229  - [-  -  -]
# 4  Background short  Completed  -  185  - [-  -  -]
# 5  Background short  Completed  -  161  - [-  -  -]
# 6  Background short  Completed  -  137  - [-  -  -]
# 7  Background short  Completed  -  112  - [-  -  -]

Long (extended) Self Test duration: 65535 seconds [1092.2 minutes]

 

[CookiesLikeWhoa]

Hmm that is an interesting read out for the data. Can we be sure it's not just that type of HDD's output?

 

[Ericloewe]

Yeah, SAS drives vary a lot more in terms of their SMART output. The most important information is there: Elements in grown defect list, SMART test results, etc.

 

[neuro]

Hmm that is an interesting read out for the data. Can we be sure it's not just that type of HDD's output?

I guess it's not the HD, Backblaze has written about reading the Helium level status of this drive through a unknown smart attribute. But I didn't test had the chance to check it myself on a system with only a HBA.

 

[Ericloewe]

Maybe there's an option missing for smartctl.

 

[Antok]

Hey, we're in the same boat you are; How did you get the drives visible to smart? We're running into the same issues as the OP from this thread a few years ago. We've put most of the drives into JBOD mode, but some other threads have implied this might not be a true passthrough/IT mode.

We have also read this might be an mfi vs mpr driver issue.

We're running FreeNAS v9.10.1 in a SSG-6028R-E1CR12H with an AOC-S3108L-H8iR HBA/Raid controller, with a raid volume for the ZIL, L2Arc, and the rest of the disks in the external JBOD chassis running in JBOD mode, with HGST HUH728080AL5200 drives.

If all else fails, we intend to switch to an AOC-S3008L-L8e; Are we SoL?

Finally, if we create a pool using mfisyspd, would this create complications if we switched swapped Raid/HBA cards (moved to a 3008 IT mode card?)

All drives save for the USB boot stick are presented as both /dev/pass* and /dev/mfisyspd* simultaneously; smart queries to pass* succeeds, but mfisyspd fails; Only mfisyspd is visible to the GUI, and only mfisyspd can be used to create pools (via GUI or CLI).

Details:

Code:

[root@server] ~# MegaCli -AdpAllinfo -aALL
 
Adapter #0

==============================================================================
  Versions
  ================
Product Name  : AVAGO 3108 MegaRAID
Serial No  :
FW Package Build: 24.15.0-0018

  Mfg. Data
  ================
Mfg. Date  : 00/00/00
Rework Date  : 00/00/00
Revision No  :
Battery FRU  : N/A

  Image Versions in Flash:
  ================
BIOS Version  : 6.31.03.0_4.17.08.00_0x06140200
Ctrl-R Version  : 5.16-0300
Preboot CLI Version: 01.07-05:#%0000
FW Version  : 4.650.00-6223
NVDATA Version  : 3.1602.00-0003
Boot Block Version : 3.07.00.00-0003

  Pending Images in Flash
  ================
None

  PCI Info
  ================
Controller Id   : 0000
Vendor Id  : 1000
Device Id  : 005d
SubVendorId  : 15d9
SubDeviceId  : 0809

Host Interface  : PCIE

ChipRevision  : C0

Link Speed	 : 0
Number of Frontend Port: 0
Device Interface  : PCIE

Number of Backend Port: 8
Port  :  Address
0  500304801f14fe3f
1  0000000000000000
2  0000000000000000
3  0000000000000000
4  0000000000000000
5  0000000000000000
6  0000000000000000
7  0000000000000000

  HW Configuration
  ================
SAS Address  : 500304801b9b7501
BBU  : Absent
Alarm  : Present
NVRAM  : Present
Serial Debugger  : Present
Memory  : Present
Flash  : Present
Memory Size  : 2048MB
TPM  : Absent
On board Expander: Absent
Upgrade Key  : Absent
Temperature sensor for ROC  : Present
Temperature sensor for controller  : Absent

ROC temperature : 64  degree Celsius

  Settings
  ================
Current Time  : 18:20:4 10/19, 2016
Predictive Fail Poll Interval  : 300sec
Interrupt Throttle Active Count  : 16
Interrupt Throttle Completion  : 50us
Rebuild Rate  : 30%
PR Rate  : 30%
BGI Rate  : 30%
Check Consistency Rate  : 30%
Reconstruction Rate  : 30%
Cache Flush Interval  : 4s
Max Drives to Spinup at One Time : 2
Delay Among Spinup Groups  : 12s
Physical Drive Coercion Mode  : Disabled
Cluster Mode  : Disabled
Alarm  : Enabled
Auto Rebuild  : Enabled
Battery Warning  : Disabled
Ecc Bucket Size  : 15
Ecc Bucket Leak Rate  : 1440 Minutes
Restore HotSpare on Insertion  : Disabled
Expose Enclosure Devices  : Enabled
Maintain PD Fail History  : Enabled
Host Request Reordering  : Enabled
Auto Detect BackPlane Enabled  : SGPIO/i2c SEP
Load Balance Mode  : Auto
Use FDE Only  : No
Security Key Assigned  : No
Security Key Failed  : No
Security Key Not Backedup  : No
Default LD PowerSave Policy  : Controller Defined
Maximum number of direct attached drives to spin up in 1 min : 10
Auto Enhanced Import  : No
Any Offline VD Cache Preserved  : No
Allow Boot with Preserved Cache  : No
Disable Online Controller Reset  : No
PFK in NVRAM  : Yes
Use disk activity for locate  : No
POST delay		: 90 seconds
BIOS Error Handling	  : Stop On Errors
Current Boot Mode	   :Normal
  Capabilities
  ================
RAID Level Supported  : RAID0, RAID1, RAID5, RAID6, RAID00, RAID10, RAID50, RAID60, PRL 11, PRL 11 with spanning, SRL 3 supported, PRL11-RLQ0 DDF layout with no span, PRL11-RLQ0 DDF layout with span
Supported Drives  : SAS, SATA

Allowed Mixing:

Mix in Enclosure Allowed
Mix of SAS/SATA of HDD type in VD Allowed

  Status
  ================
ECC Bucket Count  : 0

  Limitations
  ================
Max Arms Per VD  : 32
Max Spans Per VD  : 8
Max Arrays  : 128
Max Number of VDs  : 64
Max Parallel Commands  : 928
Max SGE Count  : 60
Max Data Transfer Size  : 8192 sectors
Max Strips PerIO  : 128
Max LD per array  : 16
Min Strip Size  : 64 KB
Max Strip Size  : 1.0 MB
Max Configurable CacheCade Size: 0 GB
Current Size of CacheCade  : 0 GB
Current Size of FW Cache  : 1718 MB

  Device Present
  ================
Virtual Drives  : 2
  Degraded  : 0
  Offline  : 0
Physical Devices  : 51
  Disks  : 47
  Critical Disks  : 0
  Failed Disks  : 0

  Supported Adapter Operations
  ================
Rebuild Rate  : Yes
CC Rate  : Yes
BGI Rate  : Yes
Reconstruct Rate  : Yes
Patrol Read Rate  : Yes
Alarm Control  : Yes
Cluster Support  : No
BBU  : Yes
Spanning  : Yes
Dedicated Hot Spare  : Yes
Revertible Hot Spares  : Yes
Foreign Config Import  : Yes
Self Diagnostic  : Yes
Allow Mixed Redundancy on Array : No
Global Hot Spares  : Yes
Deny SCSI Passthrough  : No
Deny SMP Passthrough  : No
Deny STP Passthrough  : No
Support Security  : No
Snapshot Enabled  : No
Support the OCE without adding drives : Yes
Support PFK  : Yes
Support PI  : Yes
Support Boot Time PFK Change  : No
Disable Online PFK Change  : No
Support LDPI Type1  : No
Support LDPI Type2  : No
Support LDPI Type3  : No
PFK TrailTime Remaining  : 0 days 0 hours
Support Shield State  : Yes
Block SSD Write Disk Cache Change: Yes
Support Online FW Update   : Yes

  Supported VD Operations
  ================
Read Policy  : Yes
Write Policy  : Yes
IO Policy  : Yes
Access Policy  : Yes
Disk Cache Policy  : Yes
Reconstruction  : Yes
Deny Locate  : No
Deny CC  : No
Allow Ctrl Encryption: No
Enable LDBBM  : Yes
Support Breakmirror  : Yes
Power Savings  : No

  Supported PD Operations
  ================
Force Online  : Yes
Force Offline  : Yes
Force Rebuild  : Yes
Deny Force Failed  : No
Deny Force Good/Bad  : No
Deny Missing Replace  : No
Deny Clear  : No
Deny Locate  : No
Support Temperature  : Yes
NCQ  : Yes
Disable Copyback  : No
Enable JBOD  : Yes
Enable Copyback on SMART  : No
Enable Copyback to SSD on SMART Error  : Yes
Enable SSD Patrol Read  : No
PR Correct Unconfigured Areas  : Yes
Enable Spin Down of UnConfigured Drives : Yes
Disable Spin Down of hot spares  : No
Spin Down time  : 30
T10 Power State  : No
  Error Counters
  ================
Memory Correctable Errors  : 0
Memory Uncorrectable Errors : 0

  Cluster Information
  ================
Cluster Permitted  : No
Cluster Active  : No

  Default Settings
  ================
Phy Polarity  : 0
Phy PolaritySplit  : 0
Background Rate  : 30
Strip Size  : 256kB
Flush Time  : 4 seconds
Write Policy  : WB
Read Policy  : Adaptive
Cache When BBU Bad  : Disabled
Cached IO  : No
SMART Mode  : Mode 6
Alarm Disable  : Yes
Coercion Mode  : None
ZCR Config  : Unknown
Dirty LED Shows Drive Activity  : No
BIOS Continue on Error  : 0
Spin Down Mode  : None
Allowed Device Type  : SAS/SATA Mix
Allow Mix in Enclosure  : Yes
Allow HDD SAS/SATA Mix in VD  : Yes
Allow SSD SAS/SATA Mix in VD  : No
Allow HDD/SSD Mix in VD  : No
Allow SATA in Cluster  : No
Max Chained Enclosures  : 16
Disable Ctrl-R  : No
Enable Web BIOS  : No
Direct PD Mapping  : No
BIOS Enumerate VDs  : Yes
Restore Hot Spare on Insertion  : No
Expose Enclosure Devices  : Yes
Maintain PD Fail History  : Yes
Disable Puncturing  : No
Zero Based Enclosure Enumeration : No
PreBoot CLI Enabled  : No
LED Show Drive Activity  : Yes
Cluster Disable  : Yes
SAS Disable  : No
Auto Detect BackPlane Enable  : SGPIO/i2c SEP
Use FDE Only  : No
Enable Led Header  : Yes
Delay during POST  : 0
EnableCrashDump  : Yes
Disable Online Controller Reset  : No
EnableLDBBM  : Yes
Un-Certified Hard Disk Drives  : Allow
Treat Single span R1E as R10  : No
Max LD per array  : 16
Power Saving option  : Don't Auto spin down Configured Drives
Max power savings option is  not allowed for LDs. Only T10 power conditions are to be used.
Default spin down time in minutes: 30
Enable JBOD  : No
TTY Log In Flash  : No
Auto Enhanced Import  : No
BreakMirror RAID Support  : Yes
Disable Join Mirror  : Yes
Enable Shield State  : Yes
Time taken to detect CME  : 60s

Exit Code: 0x00

Code:

[root@server] ~# MegaCli -PDList -aALL
  
Adapter #0

[....first JBOD drive...]

Enclosure Device ID: 31
Slot Number: 0
Enclosure position: 2
Device Id: 58
WWN: 5000CCA25405C023
Sequence Number: 2
Media Error Count: 0
Other Error Count: 0
Predictive Failure Count: 0
Last Predictive Failure Event Seq Number: 0
PD Type: SAS

Raw Size: 7.277 TB [0x3a3812ab0 Sectors]
Non Coerced Size: 7.276 TB [0x3a3712ab0 Sectors]
Coerced Size: 7.276 TB [0x3a3710000 Sectors]
Sector Size:  512
Logical Sector Size:  512
Physical Sector Size:  4096
Firmware state: JBOD
Device Firmware Level: A7J0
Shield Counter: 0
Successful diagnostics completion on :  N/A
SAS Address(0): 0x5000cca25405c021
SAS Address(1): 0x0
Connected Port Number: 0(path0)
Inquiry Data: HGST  HUH728080AL5200 A7J0VKG3517X  
FDE Capable: Not Capable
FDE Enable: Disable
Secured: Unsecured
Locked: Unlocked
Needs EKM Attention: No
Foreign State: None
Device Speed: 12.0Gb/s
Link Speed: 12.0Gb/s
Media Type: Hard Disk Device
Drive:  Not Certified
Drive Temperature :34C (93.20 F)
PI Eligibility:  No
Drive is formatted for PI information:  No
PI: No PI
Port-0 :
Port status: Active
Port's Linkspeed: 12.0Gb/s
Port-1 :
Port status: Active
Port's Linkspeed: 12.0Gb/s
Drive has flagged a S.M.A.R.T alert : No

Code:

[root@server] ~# smartctl -a /dev/pass24
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:  HGST
Product:  HUH728080AL5200
Revision:  A7J0
Compliance:  SPC-4
User Capacity:  8,001,563,222,016 bytes [8.00 TB]
Logical block size:  512 bytes
Physical block size:  4096 bytes
LU is fully provisioned
Rotation Rate:  7200 rpm
Form Factor:  3.5 inches
Logical Unit id:  0x5000cca25405631c
Serial number:  VKG2YV6X
Device type:  disk
Transport protocol:  SAS (SPL-3)
Local Time is:  Wed Oct 19 11:33:18 2016 PDT
SMART support is:  Available - device has SMART capability.
SMART support is:  Enabled
Temperature Warning:  Enabled

=== START OF READ SMART DATA SECTION ===
....snip....

[root@server] ~# smartctl -a /dev/mfisyspd25
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
/dev/mfisyspd25: Unable to detect device type
Please specify device type with the -d option.

[root@server] ~# smartctl -a /dev/mfisyspd41 -d
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=======> ARGUMENT REQUIRED FOR OPTION: d
=======> VALID ARGUMENTS ARE: ata, scsi, nvme[,NSID], sat[,auto][,N][+TYPE], usbcypress[,X], usbjmicron[,p][,x][,N], usbprolific, usbsunplus, 3ware,N, hpt,L/M/N, cciss,N, areca,N/E, atacam, auto, test <=======

root@server] ~# mfiutil show drives
mfi0 Physical Drives:
27 (  224G) ONLINE <INTEL SSDSC2BB24 0150 serial=PHWA632002DS240AGN> SATA E1:S0
28 (  224G) ONLINE <INTEL SSDSC2BB24 0150 serial=PHWA632001QH240AGN> SATA E1:S2
29 (  224G) ONLINE <INTEL SSDSC2BB24 0150 serial=PHWA63060589240AGN> SATA E1:S3
30 (  224G) ONLINE <INTEL SSDSC2BB24 0150 serial=PHWA6316023W240AGN> SATA E1:S1
32 ( 7452G) JBOD  <HGST HUH728080AL5200 A7J0 serial=VKG357AX> SCSI-6 E2:S5
33 ( 7452G) JBOD  <HGST HUH728080AL5200 A7J0 serial=VKG3VXNX> SCSI-6 E2:S1
34 ( 7452G) JBOD  <HGST HUH728080AL5200 A7J0 serial=VKG3TPYX> SCSI-6 E2:S4
35 ( 7452G) JBOD  <HGST HUH728080AL5200 A7J0 serial=VKG2YK8X> SCSI-6 E2:S10
[.... etc ... ]

​

Code:

[root@server] ~# zpool create test3 pass40 pass41
cannot create 'test3': no such pool or dataset
[root@server] ~# zpool create test3 mfisyspd40 mfisyspd41
[root@server] ~# zpool list
NAME  SIZE  ALLOC  FREE  EXPANDSZ  FRAG  CAP  DEDUP  HEALTH  ALTROOT
freenas-boot  7.19G  1.24G  5.95G  -  -  17%  1.00x  ONLINE  -
test3  14.5T  79K  14.5T  -  0%  0%  1.00x  ONLINE  -

mfisyspd succeeds where pass fails.

 

[happydude57]

Hello All,

Newb with the willingness to dig before asking. I've been digging, but haven't found anything real recent or convincing.

In thread [hd-controller-recommendation.28699] jgreco replied with, "Supermicro AOC-S3108L-H8iR right here which is a similar card. Great for ESXi, not good (at all) for FreeNAS."

It feels like this current thread [lsi-3108-it-mode.46407] was asking if the SAS3108 provides pure enough passthrough of JBOD disks to satisfy the implicit requirements of ZFS to be considered safe for production?

We just received a Supermicro 2028U-E1CNR4T+ with the AOC-S3108L-H8iR controller and 24-2TB SSDs in it. During POST, we configured half the drives RAID6 (not sure why) and the other half as JBOD. Latest stable FreeNAS install sees the JBOD drives like one would expect.

After reading forums and docs all over the place for the past couple weeks, I'm a little leery about using this card for production. I'm not finding anything that says, "Yes, absolutely, it will work."

This current thread is about as recent a thread as I can find, and it's 4 months old since the last reply. I can't tell from reading it that SAS3108 (or anything anywhere that suggests in particular the AOC-S3108L-H8iR) can do the job. Like the previous post by Antok, we're wondering if we'd be better off acquiring and running with the AOC-S3008L-L8e.

Anxious to get feedback on this very long-running topic about the SAS3108 ... it's been around for several years as far as I can tell.

Thanks a million!

 

[Ericloewe]

It feels like this current thread [lsi-3108-it-mode.46407] was asking if the SAS3108 provides pure enough passthrough of JBOD disks to satisfy the implicit requirements of ZFS to be considered safe for production?

It has a lot less testing than IT mode, but it's supposed to work with the mrsas driver.

At some point, someone is going to have to be the Guinea pig for mrsas and everyone needs to ask themselves if they want to be it.

An SAS3008 in IT mode (or SAS2008/2308) is the safer bet, but it's quite likely that mrsas will work just as well.

By the way, did you have to manually load mrsas instead of mfi? FreeNAS used to default to mfi, even for controllers that supported mrsas, but that was supposed to have been patched out.

 

[happydude57]

FreeNAS install was essentially default and OOTB. We did nothing special to load mrsas drivers, JBOD drives appeared right out of the gate.

This box is going to be used as a PoC for us, trying to decide if we want to run FreeNAS or CentOS and build a file server with it. We configured 12 of the drives JBOD and the other 12 as a single RAID6 pool. It's a curiosity to me, I'll have to take a look. I'm wondering if we can turn cache on for the RAID6 pool, and off for all of the JBOD drives (per best practices). I'm going to guess it's all or nothing ... so we may go back to the drawing board and try one of a couple other things.

We might continue testing the RAID card, go JBOD across the board and disable cache, or we may add a SAS3008 card (probably Supermicro AOC-S3008L-L8e), swap cables to the JBOD drives to it, keep the RAID6 drives on the SAS3108 card and use the features it has to offer.

Not sure, but we will start out testing SAS3108 with JBOD and cache disabled. Ours is a very recent Supermicro AOC-S3108L-H8iR so we'll let you know what we see.

 

[Ericloewe]

If you run into any oddities with the mrsas driver, please file a bug report. It'd be nice to have more options in the HBA market that work with FreeNAS, even if they're also by LSI.