FreeNAS on Sunfire Thor system

[Doug McIntyre]

I'm trying out FreeNAS 9.3 on a Sunfire Thor system. So far, seems better than most linux distributions at actually detecting and using the hardware in this system.

Main problem is that I can see only 40 of the 48 disks in the system. I could see this old of thing having some bad disks, but 40 seems like such a nice round human number, that it sounds like some built in. Is there anyway to increase the # of disks the system can detect?

Otherwise things seem to work well. I'd have liked an option to create the pool with spares, but I suppose RAIDZ3 will be enough parity disks as it is.

 

[9C1 Newbee]

I am unfamiliar with that hardware. But right off I would be thinking some sort of hardware raid is mucking things up. What kind of controller does it use?

 

[9C1 Newbee]

"A significant feature of both systems is that the I/O framework was designed to handle high throughput on all disks simultaneously. These were the first systems designed specifically with ZFS in mind, so no hardware RAID is included."

Sounds like it should work.

 

[Doug McIntyre]

Yes, indeed, the 2nd gen of the first system designed for ZFS.

In looking deeper at the controller type, I think the correct question to ask is if there is a limit on the # of lsi mpt controllers set to 5?

And is there a way to override that limit to go to the 6 controllers that are built into this system?
Otherwise, they are LSI SAS1068E PCI-Express Fusion-MPT SAS, and in reviewing dmesg/lspci output, only the first 5 controllers are detected, and the 6th is MIA, which has another 8 disks on it.

 

[9C1 Newbee]

That thing sounds like a monster even if it is 8 years old. Have you played around in the bios? Maybe the controller is disabled for whatever reason.

 

[HoneyBadger]

Thor is an absolute beast, it was the replacement to Thumper.

I don't think there's a limit on the number of controllers. Quick way to test though would be to boot something like an OpenIndiana liveCD and see if it finds all of the controllers properly.

 

[Doug McIntyre]

Solaris and CentOS 6.5 do find the correct number of disks/controllers in the system. It is just FreeNAS that does not.

 

[HoneyBadger]

File a bug report, I'd say. Maybe attach the contents of lspci/dmesg as a text file here?

X4540 service manual here:
https://docs.oracle.com/cd/E19469-01/819-4359-19/819-4359-19.pdf

Section 2.15 gives the bus addresses of the LSI controllers, you should be able to see which one is "missing" as well.

 

[cyberjock]

I'd attach a debug file too.

I'm gonna guess that the firmware on that 6th controller is an odd-man-out and is somehow responsible for the behavior. But that's just a gut instinct with no information to back up the claim. ;)

 

[Doug McIntyre]

FWIW: This seems to be some weird hardware mode that FreeNAS set the Thor system into. I suspect that the way the graphics video card was initialized kicked it out of control of the onboard IPMI system, so I can no longer do remote console. And I think that graphic card issue has blocked the last SAS controller from initializing due to out of Option ROM space.

I've had some searches for simular issues on loaded up SuperMicro boards, but nothing on this particular combination.

SO, I don't think FreeNAS is doing anything specificly wrong, but just some weird combination with BIOS negotiation and PCI bus/card initiation has resulted in the last controller not coming up fully and in a usable state by anything, including the LSI bios util.

So, off to plumbing the deeps of BIOS and PCI, and not looking for anything within FreeNAS at all about this. I know I have at least 2 bad disks, perhaps I'll just use the last 8 as cold spares anyway.

 

[david kennedy]

I'm trying out FreeNAS 9.3 on a Sunfire Thor system. So far, seems better than most linux distributions at actually detecting and using the hardware in this system.
is.

interesting to see someone else with some old sun gear on their hands.

I ran FREENAS on my thumper for a while before reverting back to Solaris.

My only issue was getting it to boot off USB. If memory serves, your thor has a cf flash slot you can use instead?

 

[Doug McIntyre]

That is correct, there is a CF slot I can use. I suppose I should sacrafice some of my cameras CF cards to see how that works out rather than running off a pair of boot drives.

 

[jblakeney]

I have recently setup a FreeNAS system using this server. I only have 24 drives but all is well. I used the 2 USB ports on the motherboard of the system controller for the boot drives. On the back are two retaining screws below the power supplies. You can remove the motherboard chassis and take the protective cover off the top. There are two USB ports near the processors that can be used. I installed 2 USB drives and mirrored the install.

Also, Are you sure you only see 5 controllers. The boot process shows all 6 controllers on mine but they are numbered 0-5 not 1-6. Currently, I have 4 drives on each controller and all show properly.

 

[Ben Bliss]

I am running a Thor system on 9.3 stable. I see all 48 disks and am booting off of mirrored usb stick on the back. The BMC still works as well. The system has been stable and runs like a champ. Currently this system is a DR storage array for vCenter Replication. If needed I can help troubleshoot as this system can go offline if needed for short periods.

 

[Doug McIntyre]

I did find out the hardware was bad for sure, and a whole HBA Controller is offline and not detectable, nor any of the 8 disks on that one controller. Its something like #5 out of 6 HBA controllers on this system. I wasn't expecting any bad hardware on this due to providence of the hardware, expected them to have fixed on contract if anything had fried.

Which did work out okay in the end, as I had 4 bad disks in the working disk set, so the 8 disks offline became cold spares to replace the bad ones.

The KVM system is a bit wonky too, when using it, generates a bit of excess oddness that doesn't show up when just using local console.

OOTH, the CF card slot works fine to boot the system, and that is where FreeNAS lives now. The 4 ethers bound nicely into two LAG groups for public/private side functions.

Otherwise, been doing mostly iSCSI shares off here to mostly backup & archival systems, and working like a champ for all other FreeNAS serving needs. I guess my initial reaction is from all my ancient past dealing with systems that did have hard limits after a reasonable numbers.

I've seen the motherboard module come up on eBay from time to time, but at too high a cost. Maybe some day I'll see one at a reasonable cost and have all 48 disks be able to go online.

 

[9C1 Newbee]

Just curious what the power consumption numbers are.

 

[Doug McIntyre]

Right now with 44 disks spinning (4 bad ones removed, 4 left spinning on the bad controller), so 40 disks working, it is consuming 817W of power.

 

[Ben Bliss]

Mine is running around 920w depending on load. Right now it gets a lot of usage.