Migrated drives from 730xd to 730xd. OS booted fine but pools disappeared with no way to import

[EnragedZeus05]

as the title states I moved by 2 boot drives and 16 HDDs from one 730 to another. I moved the HBA as well. Booted into the OS but I only see 14 new disks with no way to import a pool. I put everything back in the old chassis and im in the same boat. looking for a way to move forward before I send all my disks into a data recovery place...

 

[morganL]

as the title states I moved by 2 boot drives and 16 HDDs from one 730 to another. I moved the HBA as well. Booted into the OS but I only see 14 new disks with no way to import a pool. I put everything back in the old chassis and im in the same boat. looking for a way to move forward before I send all my disks into a data recovery place...

please follow Forum Rules. More information gets you better answers.

 

[EnragedZeus05]

please follow Forum Rules. More information gets you better answers.

sorry, Had a freak out moment.

• Motherboard make and model: Dell 730XD
• CPU make and model: Intel Xeon E5-2643 V3
• RAM quantity: 128gb
• Hard drives, quantity, model numbers, and RAID configuration, including boot drives:
  • Boot drives: Intel SSD DC S3500 120gb raid z1
  • Storage pool: Seagate Exos X18 18tb - 16 total for two raid z2
• Hard disk controllers: dell Perc HBA 330
• Network cards: Dell Intell X540 Base-T2 Quad port

Moved drives from one chassis to another and moved the HBA as well. Drives are only showing up as "new" and not associated with the pool. Moved back to old chassis and moved the HBA as well. Same result.

 

[morganL]

You should provide some screenshots and be very accurate with terminology.

Pool import is usually the 1st step:

/scale/scaletutorials/storage/importpoolscale/

 

[EnragedZeus05]

You should provide some screenshots and be very accurate with terminology.

Pool import is usually the 1st step:

/scale/scaletutorials/storage/importpoolscale/

Yea, i get it. it shouldnt be a pool import at all since i moved drives/etc to the same chassis w/ the same firmware on the board/server.

But, i swapped things over back and forth a few more times today and on the 100th swap of troubleshooting all the pools spun up. It is complete black magic but as of now everything is online.

I have a very basic home setup. single cpu, not many pools, etc. But as for now im never going to reboot and just lay claim that as of today bluefin is RIP for me :/

 

[morganL]

Yea, i get it. it shouldnt be a pool import at all since i moved drives/etc to the same chassis w/ the same firmware on the board/server.

But, i swapped things over back and forth a few more times today and on the 100th swap of troubleshooting all the pools spun up. It is complete black magic but as of now everything is online.

I have a very basic home setup. single cpu, not many pools, etc. But as for now im never going to reboot and just lay claim that as of today bluefin is RIP for me :/

This is the way OpenZFS works.... I doubt its specifically related to SCALE.

When you moved drives they probably appeared as different linux devices on the new system On the 100th attempt they might have reappeared the same way by luck.

Did pool import not work? That would be an issue worth reporting.

"bluefin" was not declared in your problem statement. It's possible there is a bug... its RC1 code, not yet "released" and widely tested.

 

[EnragedZeus05]

This is the way OpenZFS works.... I doubt its specifically related to SCALE.

When you moved drives they probably appeared as different linux devices on the new system On the 100th attempt they might have reappeared the same way by luck.

Did pool import not work? That would be an issue worth reporting.

"bluefin" was not declared in your problem statement. It's possible there is a bug... its RC1 code, not yet "released" and widely tested.

I am still using stable. I just meant rip bluefin because i dont want to reboot.

i cant say pool import didnt work as there was no pool to import in both gui and cli. The disks were showing as part of no pool and gave me the option to create a new pool. this afternoon i moved the disks back to the new chassis since it has a enterprise idrac and gave me more troubleshooting options remotely. I always kept the disks in the same slots but for whatever reason when it booted this time the pools showed as available. No import needed.

 

[morganL]

I've looked through our bug database and can't find any "pool import" issues with 22.02.4. From your hardware description, I can't see what would be unusual. There have been issues with missing devices (e.g 1 drive not attached), but nothing else I can see. Link here is an example: https://www.truenas.com/community/t...ation-failed-to-bring-the-pool-online.101908/

If you have the time, it would be worthwhile reporting the issue with your hardware and the specific webUi and CLI steps you took.

I doubt you'll see any similar issues rebooting.. regardless of whether you update to Bluefin, you will need that.

If anyone finds a similar issue, please comment.

 

[HoneyBadger]

Hard disk controllers: dell Perc HBA 330

Are you very certain that these controllers are HBA330s? This kind of behavior is what I would expect to see when moving drives between units with an H330 that is importing a foreign RAID configuration. Order of drives shouldn't matter at all for pool import as long as the internal labels are present.

The other thing I would look at is the cabling to your drive backplane/midplane as you've got what looks like an R730XD with the four-bay midplane that swivels up, as well as the rear 2.5" flexbays. There's lots of opportunities there for cables to be jostled loose or pinched even in routine operations.

 

[EnragedZeus05]

Are you very certain that these controllers are HBA330s? This kind of behavior is what I would expect to see when moving drives between units with an H330 that is importing a foreign RAID configuration. Order of drives shouldn't matter at all for pool import as long as the internal labels are present.

The other thing I would look at is the cabling to your drive backplane/midplane as you've got what looks like an R730XD with the four-bay midplane that swivels up, as well as the rear 2.5" flexbays. There's lots of opportunities there for cables to be jostled loose or pinched even in routine operations.

it is 100% an HBA330. My friend who uses truenas rocks n H330 and had a ton of issues w/ it and I didnt want to deal with them so I forked it out for an HBA 330.

The cables were fine. I reseated them and compared them between both chassis.

I've looked through our bug database and can't find any "pool import" issues with 22.02.4. From your hardware description, I can't see what would be unusual. There have been issues with missing devices (e.g 1 drive not attached), but nothing else I can see. Link here is an example: https://www.truenas.com/community/t...ation-failed-to-bring-the-pool-online.101908/

If you have the time, it would be worthwhile reporting the issue with your hardware and the specific webUi and CLI steps you took.

I doubt you'll see any similar issues rebooting.. regardless of whether you update to Bluefin, you will need that.

If anyone finds a similar issue, please comment.

yea, I know I said pool import but I shouldn't have even have to look into that since I swapped the OS drives into an identical chassis. Its not like I loaded truenas onto new drives and uploaded my config.

it was a straight swap.
Power down 730XD server A.
Moves OS and Data drives and HBA330 to 730XD server B
Power on server B.

the boot pool showed up fine and booted into truenas but all the data drives were only showing up to create a new pool.

 

[morganL]

it is 100% an HBA330. My friend who uses truenas rocks n H330 and had a ton of issues w/ it and I didnt want to deal with them so I forked it out for an HBA 330.

The cables were fine. I reseated them and compared them between both chassis.

yea, I know I said pool import but I shouldn't have even have to look into that since I swapped the OS drives into an identical chassis. Its not like I loaded truenas onto new drives and uploaded my config.

it was a straight swap.
Power down 730XD server A.
Moves OS and Data drives and HBA330 to 730XD server B
Power on server B.

the boot pool showed up fine and booted into truenas but all the data drives were only showing up to create a new pool.

The chassis are always similar, but not identical. Things like MAC addresses change. If pool import processes were not available that would be a problem, but there is no guarantee that pool import is not required when changing systems.

 

[depasseg]

Hmm, something strange might be going on in your setup. I happened to just do this same exact thing (except I didn't move the HBA) and everything just booted up on the new system like it did on the old one. I didn't need to import anything. Maybe something in the BIOS is presenting the disks differently?

 

[HoneyBadger]

it is 100% an HBA330. My friend who uses truenas rocks n H330 and had a ton of issues w/ it and I didnt want to deal with them so I forked it out for an HBA 330.

The cables were fine. I reseated them and compared them between both chassis.

Suggest to your friend that they should crossflash their H330 to an HBA330 - it can be done, and that will get them off the mrsas driver and onto mpr which is better tested.

But in your situation, an HBA330 should have absolutely no considerations for physical disk slot location. I don't think it's possible for them to inherit an Hx30 RAID configuration either (through retained iDRAC info from the host board) - assuming you have a BIOS/EFI bootrom on your card still, are you able to see all of the devices through that if you interrupt the boot via Ctrl+C or F2 and open it via Device Options?

 

[EnragedZeus05]

Hmm, something strange might be going on in your setup. I happened to just do this same exact thing (except I didn't move the HBA) and everything just booted up on the new system like it did on the old one. I didn't need to import anything. Maybe something in the BIOS is presenting the disks differently?

Yea, I went through screen for screen before swapping drives to make sure everything was the same. Took some time but I didn’t want any “gotchas” with 90tb of data.

Suggest to your friend that they should crossflash their H330 to an HBA330 - it can be done, and that will get them off the mrsas driver and onto mpr which is better tested.

But in your situation, an HBA330 should have absolutely no considerations for physical disk slot location. I don't think it's possible for them to inherit an Hx30 RAID configuration either (through retained iDRAC info from the host board) - assuming you have a BIOS/EFI bootrom on your card still, are you able to see all of the devices through that if you interrupt the boot via Ctrl+C or F2 and open it via Device Options?

He did cross flash his. I didn’t want to do that and I went the HBA route.

Yea, the devices/hdds were showing up. My boot pool survived because it booted into truenas but all my pools were gone. Tried to import…nothing to import. For shits and giggles went through creating a new pool and there’s all my drives. Swapped back to old hardware and same issue. Freaked out for a few hours swapping hardware, trying anything I could find on forums, gave up and opened a case with a data recovery company.

The next morning I woke up and swapped the drives into the new chassis. I had calmed down since I accepted everything was gone and figured I would put some time into troubleshooting via cli.

I boot the server up and my pools showed online.

 

[Daisuke]

Glad you had that resolved. So technically you added the drives into original new server and when you booted, the pools showed?

 

[EnragedZeus05]

Glad you had that resolved. So technically you added the drives into original new server and when you booted, the pools showed?

no, when I swapped them into the new chassis

 

[Daisuke]

Was this a different chassis, from the new R730xd you mentioned into OP?

 

[EnragedZeus05]

Was this a different chassis, from the new R730xd you mentioned into OP?

No,

only been operating out of chassis A (original) and chassis B (new).

 

[morganL]

So, its resolved, but something was not as reliable as expected.
Anyone seen it before or have a theory?