Supermicro Backplane Woes

[pollytheparrot]

So I have an SC846 box running Freenas. I have the SAS2 backplane. On it is a RAIDZ2+0 pool configuration of two 10 drive RAIDZ2 vdevs that are 4TB Reds.

I'm looking to replace one of the vdevs with 8TB reds, and rather than run in a degraded state (pulling the drive to replace), I wish to follow the steps outlined in Freenas 8.1.11. Basically adding drives into a spare slot and then replacing it.

Problem is, I believe one of the sections of the backplane is not working right. Here is a picture of how it looks:

Green is the first vdev, blue is the second vdev, and red is a sector that appears to not display drives. The blue LED status indicator on the front of the hotswap bays comes on when drives are plugged in, but drives plugged in do not show up in the "View Disks" functionality of Freenas, nor do they show up as an available disk to extend a volume with. Obviously this presents a complication to my plan to grow the vdev. On the inside I have several M1015s in IT mode (P15), these are connected to the backplane across the SFF-8087 cables.

I'm not really sure how to possibly test this or fix this and was looking for advice before I go screwing things up.

 

[DrKK]

I vote for swapping around your 8087's, and see if the problem follows the cables, or not.

 

[Ericloewe]

Also, P15 is a very weird version to use. You need P20.00.0.7.

 

[danb35]

I have the SAS2 backplane

It would be helpful if you clarified exactly which backplane you have, but from the fact that you say you have several HBAs, it sounds like a -A model. In that case, I concur with @DrKK--try moving the cables around to see if the problem follows them.

 

[Spearfoot]

So I have an SC846 box running Freenas. I have the SAS2 backplane. On it is a RAIDZ2+0 pool configuration of two 10 drive RAIDZ2 vdevs that are 4TB Reds.

I'm looking to replace one of the vdevs with 8TB reds, and rather than run in a degraded state (pulling the drive to replace), I wish to follow the steps outlined in Freenas 8.1.11. Basically adding drives into a spare slot and then replacing it.

Problem is, I believe one of the sections of the backplane is not working right. Here is a picture of how it looks:

Green is the first vdev, blue is the second vdev, and red is a sector that appears to not display drives. The blue LED status indicator on the front of the hotswap bays comes on when drives are plugged in, but drives plugged in do not show up in the "View Disks" functionality of Freenas, nor do they show up as an available disk to extend a volume with. Obviously this presents a complication to my plan to grow the vdev. On the inside I have several M1015s in IT mode (P15), these are connected to the backplane across the SFF-8087 cables.

I'm not really sure how to possibly test this or fix this and was looking for advice before I go screwing things up.

Welcome to the forums!

Please post your full system specs (per the Forum Rules) - this really helps the folks here to better help you.

You didn't say which backplane you have... but you mentioned having "several M1015s in IT mode", so I'm guessing you have the direct-attached BPN-SAS-846A model (user manual available here) and not one of the units with an expander.

The 4 red bays you show here correspond to the IPASS3 connection on the BPN-SAS-846A: so the first thing I would try is to re-seat (at both ends!) the cable connecting IPASS3 to whichever HBA card you've got it connected to. Hopefully re-seating the cable will restore full functionality of the 4 'red' bays.

But if re-seating the cable doesn't work, then either the cable is bad, the corresponding port on the HBA card driving IPASS3 is bad, or (shudder!) those 4 ports on the backplane are bad.

How to tell which? Easy: but you'll need to detach all of your pools before start, and for good measure, remove all of your disks: you don't want to put your data at risk and the next steps involve performing a little major surgery on the server, moving cards and cables around and what-not. Make good notes and a diagram of your current HBA connections. After you've finished these preliminaries...

First replace the cable with one of your 'known-good' cables (viz., any of the other cables in your system, which all seem to work). If the 'red' bays start working then you know the cable is bad. Replace the cable.

If they still don't work, then either the HBA or the backplane itself is bad. So the next step is then to connect IPASS3 to any other 'known good' port on one of the other HBAs. If the 'red' ports start working, then you know the HBA is bad (really it's 'half-bad' because one of the two ports seems to be working). Replace the HBA.

However, if they still don't work this means your backplane is at fault. Hopefully your problem is a bad cable or HBA and not the backplane, as it's much more expensive and a real bear to change out!

When you've finished and replaced the bad part(s), reassemble the system as it was configured when you started, then boot FreeNAS and re-import all of your volumes. You'll be back where you started only (hopefully!) with the 4 'red' bays now functioning.

Oh! If you're running FreeNAS 9.x or later, you really ought to install the P20.00.07.00 IT firmware on your M1015s. The perfect time to do this would be after you've finished troubleshooting and repairing the system and before you install the hard drives.

Hopefully I haven't made a bad guess about your setup...

Good luck!

 

[pollytheparrot]

Welcome to the forums!

Please post your full system specs (per the Forum Rules) - this really helps the folks here to better help you.

You didn't say which backplane you have... but you mentioned having "several M1015s in IT mode", so I'm guessing you have the direct-attached BPN-SAS-846A model (user manual available here) and not one of the units with an expander.

The 4 red bays you show here correspond to the IPASS3 connection on the BPN-SAS-846A: so the first thing I would try is to re-seat (at both ends!) the cable connecting IPASS3 to whichever HBA card you've got it connected to. Hopefully re-seating the cable will restore full functionality of the 4 'red' bays.

But if re-seating the cable doesn't work, then either the cable is bad, the corresponding port on the HBA card driving IPASS3 is bad, or (shudder!) those 4 ports on the backplane are bad.

How to tell which? Easy: but you'll need to detach all of your pools before start, and for good measure, remove all of your disks: you don't want to put your data at risk and the next steps involve performing a little major surgery on the server, moving cards and cables around and what-not. Make good notes and a diagram of your current HBA connections. After you've finished these preliminaries...

First replace the cable with one of your 'known-good' cables (viz., any of the other cables in your system, which all seem to work). If the 'red' bays start working then you know the cable is bad. Replace the cable.

If they still don't work, then either the HBA or the backplane itself is bad. So the next step is then to connect IPASS3 to any other 'known good' port on one of the other HBAs. If the 'red' ports start working, then you know the HBA is bad (really it's 'half-bad' because one of the two ports seems to be working). Replace the HBA.

However, if they still don't work this means your backplane is at fault. Hopefully your problem is a bad cable or HBA and not the backplane, as it's much more expensive and a real bear to change out!

When you've finished and replaced the bad part(s), reassemble the system as it was configured when you started, then boot FreeNAS and re-import all of your volumes. You'll be back where you started only (hopefully!) with the 4 'red' bays now functioning.

Oh! If you're running FreeNAS 9.x or later, you really ought to install the P20.00.07.00 IT firmware on your M1015s. The perfect time to do this would be after you've finished troubleshooting and repairing the system and before you install the hard drives.

Hopefully I haven't made a bad guess about your setup...

Good luck!

I greatly appreciate the detailed response! My apologies for not including the full system specifications. The M1015's are the SAS9220-8i model and I'm running FreeNAS 9.10.2-U3. The system itself is two X5660s running on top of a Supermicro X8DTN+ with 72GB of ECC RAM.

As far as the manual goes, I want to say that JSM4 (#11 on the key map) is the problematic port here.

You have outlined a great strategy for looking at this problem. I had a few fragments of it in my head but this really works to put it all together.

Regarding the "Detach Volume", I should leave the two boxes unchecked on the box that comes up, correct?

This should allow me to shut it down, remove all the disks, and then boot it back up for my testing purposes correct? When I'm done, I should put all of the disks back in and then start it up and use the import volume functionality and that should display my pool as a volume that can be imported and everything should be back to normal?

Also, P15 is a very weird version to use. You need P20.00.0.7.

It was the current version when I flashed them. I want to say it was about two years ago. I fall pretty far into the "if it ain't broke don't fix it" camp regarding firmware. Have there been significant improvements in functionality between the two versions?

 

[Ericloewe]

The M1015's are the SAS9220-8i

Then they're not M1015s, are they? They're LSI SAS 9220s. No matter...

Regarding the "Detach Volume", I should leave the two boxes unchecked on the box that comes up, correct?

Definitely don't check the "delete everything" box.
In fact, it might be simpler to just use a fresh install of FreeNAS, to minimize configuration woes.

Have there been significant improvements in functionality between the two versions?

No, but the software stack is known to blow up if there's a mismatch.

 

[Spearfoot]

I greatly appreciate the detailed response!

You're very welcome!

As far as the manual goes, I want to say that JSM4 (#11 on the key map) is the problematic port here.

My bad! I referred to a setup spreadsheet I made for my main system (see 'my systems' below), where I labeled the port blocks as IPASS0=JSM1, IPASS1=JSM2, IPASS2=JSM3, IPASS3=JSM4, etc. JSM4 is indeed the port we're interested in.

Regarding the "Detach Volume", I should leave the two boxes unchecked on the box that comes up, correct?

Correct! You most definitely do not want to erase the disks! That would destroy your data! I can't see the image you posted, but note that I only see a single checkmark when I detach a volume, named 'Mark the disks as new (destroy data)'; leave this unchecked! See below:

This should allow me to shut it down, remove all the disks, and then boot it back up for my testing purposes correct? When I'm done, I should put all of the disks back in and then start it up and use the import volume functionality and that should display my pool as a volume that can be imported and everything should be back to normal?

Yes, sir. Detaching all of your volumes and removing the disks means you can't destroy the data on them. Be sure you detach both volumes/pools on your system.

You'll find it handy for testing purposes to have one or more 'old' disks (or new ones that don't contain any data) mounted in trays and ready to pop in to the 'red' bays. You can run smartctl --scan to see if the drives load up, create 'scratch' pools to exercise the HBA/backplane subsystem, etc. (Be sure to destroy any temporary pools you create when you're through testing with them.)

When you've finished working on the system, just boot FreeNAS and import both volumes. Once the volumes are imported, your system should be fully functional again. You'll see these two forms during the import process:

It was the current version when I flashed them. I want to say it was about two years ago. I fall pretty far into the "if it ain't broke don't fix it" camp regarding firmware. Have there been significant improvements in functionality between the two versions?

I understand and generally adhere to your philosophy, but in the case of these LSI cards I strongly urge you to flash the newer firmware that matches what FreeNAS expects to find.