Truenas Forces Reboot When I import My Raid Z1 Pool

[Danthewhale]

I was doing a file transfer and the server suddenly shutdown and was in a boot loop, i plugged in a hdmi and checked the console and it came up with this error, cpu_reset_proxy: Stopped CPU 1 and this number would change from 1 to 0 to 3 etc so it wasn't always the same number, also sometimes it would say this error too, 0 :kdb.enter.default> textdump dump then would reset. I installed a fresh install of Truenas several times, Unplugged the cpu, ram and drives then put them back in and the problem persisted, I've narrowed it down to my Raid Z1 pool because when I try to import the pool it shows that error and forces the machine into a constant reboot. The other pools i have managed to import fine with no problems. I need some advice on what this could be because I'm just a noob with this stuff and I'm at a loss here on if I have made a mistake somewhere or worst case scenario I've lost really important data. The Raid Z1 has 4x4TB IronWolf Pro Drives and they are still in their warranty, I have only had them for 1 year and 6 months so far. I'll write the specs and put pictures below, thankyou if you can help me in anyway.

The Version of Truenas it's running is 13.0 U4 and I did try to go back to 12.0 and still the same problem, but I don't know how to download/find older versions of Truenas.

The Specs:
Processor: Ryzen 3 1200
Ram: I have all Ram sticks clocked at 2133mhz for stability
- 2x8gb 2400mhz Corsair Vengence LPX DDR4
- 2x8gb 3000mhz Corsair Vengence LPX DDR4
Motherboard: Gigabyte b450m ds3h, Bios version 63f
Graphics Card: Nvidia Geforce GT 710
Power Supply: Corsair VS350W
Also I have a 8 port PCIE to SATA board for the drives to connect to since I don't have enough SATA ports on my motherboard.

 

[NugentS]

Well the PCIe to SATA board is almost certainly not a good idea, depending on exactly what it is.
TrueNAS is designed for enterprise hardware - and using cheap PCIe X1 Sata expanders is NOT a good idea.

If you remove all the disks (and the PCIe SATA card) - it boots correctly?

Can you please be very specific about what SATA expansion card you are using?

 

[Danthewhale]

Yes it boots fine without all of the drives connected and only gives that error when the 10TB Raid Z1 is connected, it goes through the process of importing it then it just crashes and goes into a boot loop and gives me one of them two errors.

The Sata Card I have just posted pictures of it here.
Before I bought it i did some research on the best sata card to get and they said to get one with a marvell chip on it, they were the most reliable which yes i have seen that they can cause problems and so far until now it has been fine but a problem that i have always had with this server is that if i am transferring files constantly for more than 20 minutes one after the other it would often crash for some reason and im not sure why.

I did try and plug the 10TB Raid Z1 straight into the 4 ports of my motherboard and the same problems happened, if I took it out and had the other 3 drives in there it would be fine. I was doing a file transfer and was moving large TV Shows over to the drives from my pc and suddenly it crashed and it went into a boot loop and this is how it all started. The crashing under heavy transfers has always been a problem with this server ever since I built it 2 years ago and idk why.

 

[NugentS]

Well the best "SATA" card was not what you have there. There are very few SATA addin cards that work with ZFS
Something like ebay.co.uk Link would have been a much better idea - but do you have the slots for one? I don't think you do as you have an X16, an X4 and a largely useless X1
You could put the GPU into the X4 slot, depends on what you use the GPU for and put a proper HBA into the X16 slot. Use the HBA for HDD's and the ports on the motherboard for any SSD's you may have. What do you use the GPU for?

What version of freenas / truenas are you using?

Nevertheless you ought to be able to import the pool - such SATA cards "mostly" cause problems under load. What happens if you plug the Z1 into the motherboard and leave other HDD's disconnected. Also can you set the motherboard SATA ports to hotswap. If so then try booting the device without HDD's and then add the array to the motherboard ports and import - maybe

1. 1 x PCI Express x16 slot, running at x16 (PCIEX16)(Note)
   * For optimum performance, if only one PCI Express graphics card is to be installed, be sure to install it in the PCIEX16 slot.
   (The PCIEX16 slot conforms to PCI Express 3.0 standard.)
2. 1 x PCI Express x16 slot, running at x4 (PCIEX4)
3. 1 x PCI Express x1 slot
   (The PCIEX4 and PCI Express x1 slots conform to PCI Express 2.0 standard.)

 

[blanchet]

Marvel SATA chips are not well supported by TrueNAS Core (and also TrueNAS Scale). Only Intel embedded SATA controller work well. If you need more SATA/SAS ports then you should pick a supported HBA.

The best HBA for TrueNAS are
- LSI/Broadcom 9300 (with the SAS3008 chip that supports SAS3 12 Gbps) flashed in IT-mode with the 16.00.12.00 firmware https://www.truenas.com/community/resources/lsi-9300-xx-firmware-update.145/
- LSI SAS 2308 chip is older (6 Gbps) but it works also well if you connect only spinning disks. Typically IBM M1015

More recent Broadcom HBA like 9400 and 9500 series should work. But I have never tried them.

 

[Danthewhale]

Well the best "SATA" card was not what you have there. There are very few SATA addin cards that work with ZFS
Something like ebay.co.uk Link would have been a much better idea - but do you have the slots for one? I don't think you do as you have an X16, an X4 and a largely useless X1
You could put the GPU into the X4 slot, depends on what you use the GPU for and put a proper HBA into the X16 slot. Use the HBA for HDD's and the ports on the motherboard for any SSD's you may have. What do you use the GPU for?

What version of freenas / truenas are you using?

Nevertheless you ought to be able to import the pool - such SATA cards "mostly" cause problems under load. What happens if you plug the Z1 into the motherboard and leave other HDD's disconnected. Also can you set the motherboard SATA ports to hotswap. If so then try booting the device without HDD's and then add the array to the motherboard ports and import - maybe

Yes, what i did was i took the SATA add in card out and moved the raid 10TB Z1 drives into the only 4 SATA ports on my motherboard and the same problem happened, yeah with the SATA card i did know about the SAS cards and stuff but it wasnt in my budget and i didn't know enough at the time. In the original post i said that the truenas version is 13.0 U4 and i tried an older 12.0 version. I have been using the gpu just for video since my cpu doesnt have graphics.

As far as i know i have the SATA card on the top PCIE port and the gpu at the bottom which i presume the top is x16 and the bottom is x4?

This is why i am at a loss because the problems all seem to point towards the drives unless i'm missing something, like i've always had stability problems where it would just reboot/crash under heavy load but it hasn't done this before. Sorry guys for not explaining this stuff properly i'm just sad that i hope i haven't lost data and i'm too dumb to understand :(

 

[NugentS]

Crashing under load would be a symptom of a crappy PCIe SATA port card. It could be other things as well (such as a weak PSU) - but a crappy SATA adapter would be a prime suspect.
The top port is likley the X16 but putting an X1 card into it won't help. Putting the GPU into the X4 slot is a good idea given the circumstances

My opinion is that you have likley lost the pool - do you have any backups (I am guessing not)?

There are commands you can use to force a pool import that I am not terribly familiar with - unfortunately the NAS needs to boot first so they won't help.

"Also can you set the motherboard SATA ports to hotswap. If so then try booting the device without HDD's and then add the array to the motherboard ports and import - maybe" - Did you try this - its kinda a desperation tactic though

 

[Danthewhale]

Crashing under load would be a symptom of a crappy PCIe SATA port card. It could be other things as well (such as a weak PSU) - but a crappy SATA adapter would be a prime suspect.
The top port is likley the X16 but putting an X1 card into it won't help. Putting the GPU into the X4 slot is a good idea given the circumstances

My opinion is that you have likley lost the pool - do you have any backups (I am guessing not)?

There are commands you can use to force a pool import that I am not terribly familiar with - unfortunately the NAS needs to boot first so they won't help.

"Also can you set the motherboard SATA ports to hotswap. If so then try booting the device without HDD's and then add the array to the motherboard ports and import - maybe" - Did you try this - its kinda a desperation tactic though

Yeah ill try that next, if i remove the pool from truenas and boot with them still connected it will work but it crashes as soon as i go through the import, i will see if i can find that command to force import the pool and see how it goes because not all of the data is important but some of it is very important. Also with these Seagate IronWolf Pro Drives they are still in their data rescue warranty, would they be able to recover the data if all else fails?

 

[NugentS]

I imagine that there is nothing wrong with the HDD's which given my cynical view towards warranties in general I suspect will be limited to "if there is a catastropic hardware failure", and only on a single drive - not across an array. The devil will be in the details of the warranty. Note that I haven't read anything about the warranty - so I could be talking out of my ass.

 

[Danthewhale]

I imagine that there is nothing wrong with the HDD's which given my cynical view towards warranties in general I suspect will be limited to "if there is a catastropic hardware failure", and only on a single drive - not across an array. The devil will be in the details of the warranty. Note that I haven't read anything about the warranty - so I could be talking out of my ass.

Yeah i will have a look at the warranty thanks for the help so far man, im at least happy i know what i need to know and its my fault in the end of the day for trusting the IronWolf Pro drives...

I will reply back with anything i find with the force importing or the warranties.

And with me reading your last reply, are you saying that the SATA card caused this problem?

 

[NugentS]

I doubt its the ironwolf drives - its probably (in order of likleyhood)

1. SATA Card
2. PSU being weak - but you haven't told us how many HDD's (I think 7) you have - but a theoretical 350W isn't a powerful PSU. Thats a "budget" PSU for what is probably the most important device in the entire system. IMHO its underpowered
3. Other hardware issue - not related to HDD's or PSU
4. One or more of the HDD's has gone kaput - but I think this is a low order probability

You could, before trying to import the pool, try running a smart test (smartctl -t short /dev/...) on each drive. Make sure each drive responds correctly.

Sorry to be blunt - but this is your fault. You ignored a serious rebooting issue and are using ill advised hardware to run enterprise grade software on, and you have no backups (assumed).

I suppose I should add the obligatory "RAID is not a backup" here.