Pool visible via CLI but not in the GUI. GUI import says an error occurred.

[Shoop83]

Do you have any backup of the data on your NAS?

I have a reasonable backup though it's not up to date. Rebuilding back to where I was will be time consuming but not impossible. I was horribly naive and thought the zraid array in my FreeNAS would be robust enough to not fail.

 

[Chris Moore]

Running memory test. 35 minutes in it hit a bad patch and found 8 errors. I'll let it keep running.

This might be a reasonable explanation for the problem. Bad memory will cause all sorts of problems with any operating system.


Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[Shoop83]

Assuming that's the issue, I should replace it with something like this? https://www.newegg.com/Product/Product.aspx?Item=9SIAD8U7EK4399&cm_re=ECC_RAM-_-9SIAD8U7EK4399-_-Product

 

[P-Chan]

Assuming that's the issue, I should replace it with something like this? https://www.newegg.com/Product/Product.aspx?Item=9SIAD8U7EK4399&cm_re=ECC_RAM-_-9SIAD8U7EK4399-_-Product

First and foremost, if you are using GELI, make sure you get a good copy of your key from /data/geli/*.key

Your motherboard won't support ECC memory. Here's my suggestion - to avoid any further damage, first fix the hardware issues, the memory issues could be the memory, or the memory controller, or even dust in a socket. Maybe try removing all the memory physically, dusting really well with compressed air, try reseating and retesting memory. If you can afford to, move to a board that supports ECC and get ECC memory (also, make sure you're using a CPU that supports ECC). ZFS reliability is highly dependent on having ECC memory.

If you are not using GELI to encrypt the devices, you can try booting up with System Rescue CD (http://list.zfsonlinux.org/zfs-iso/) and see if the pool is ok (just run zpool import, but don't actually import the pool, just to check it's status).

Once you know that the pool is ok and your hardware is ok, do a fresh install of FreeNAS and re-import your pool.

 

[Chris Moore]

Assuming that's the issue, I should replace it with something like this? https://www.newegg.com/Product/Product.aspx?Item=9SIAD8U7EK4399&cm_re=ECC_RAM-_-9SIAD8U7EK4399-_-Product

The problem with that memory is not just that it is ECC, it is Registered ECC. Registered ECC (RDIMM) or LRDIMM (which is Load Reduced, Registered) memory will only work with Xeon E5 (and better) processors regardless of any other factors. In addition to the fact that your Ggigabyte system board would not work at all with Registered memory, it does not support ECC memory at all. This is not to say that it would not work with Unbuffered ECC memory, but it would not allow you to take advantage of the ECC capabilities. If you have ECC memory, and a system board that supports ECC in firmware and a processor that supports ECC, when there is a memory error, the system will notify you. Without ECC memory support, you will have no notice of a memory error, the system will just crash.
If you only want to get the defective memory replaced to get the system back in operation, you can use regular Unbuffered ECC memory with the system board you have, but it will not protect you from memory errors without the firmware support in the system board.

The If you wanted to go to a system board with ECC support, this one would be compatible with your existing CPU, which does support ECC function:
Supermicro X10SLH-F Socket H3 (LGA1150) motherboard with I/O Shield
https://www.ebay.com/itm/Supermicro...1150-motherboard-with-I-O-Shield/113016159243
Price: US $124.00

If you want to get ECC memory to go with that new system board, this would be compatible with the new system board I just pointed to:
16GB 2X8GB MEMORY RAM
https://www.ebay.com/itm/16GB-2X8GB...10SLL-F-X10SLH-F-Motherboard-B89/272412461489
Price: US $160.00

I would suggest going to 16GB of memory because of the amount of storage you have, but you cold stay at 8GB if you really don't want to upgrade.

 

[Shoop83]

Chris, that's a whole lot to chew on, but that might be in the cards. Thank you so much for putting that together. First I'll try what P-chan (thanks for chiming in!) suggested and give it a good cleaning then run memtest again and report back.

It's possible I can obtain a new stick of RAM to try if this memtest finds faults again. But ultimately it looks like I really should try to replace some hardware for peace of mind if nothing else.

 

[Chris Moore]

If cleaning it doesn't change anything, the next step would be to buy a piece of memory, or if you can borrow some, just to make sure the system board isn't the source of the problem.
If your system board isn't defective then you can always just go back with a replacement memory module without having to upgrade to ECC.
In the long run, for reliability sake, ECC is the recommended type of memory.
Let me know what you want to do and I'll find some good parts for you.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[Johnny Fartpants]

Can you import the pool via the UI when booting into a fresh install of FreeNAS?

Might be worth looking into clearing out your zpool cache files.

 

[Johnny Fartpants]

Also just checking that after you imported the zpool via the command line you did export the pool before trying to import it again via the UI?

 

[Shoop83]

Can you import the pool via the UI when booting into a fresh install of FreeNAS?
Might be worth looking into clearing out your zpool cache files.

Also just checking that after you imported the zpool via the command line you did export the pool before trying to import it again via the UI?

I'm unable to import the pool from the GUI into a fresh install of FreeNAS. It panics.

Yep, I did export the pool after importing it in the command line.

 

[hescominsoon]

Shoop...if you have ram causing bits to be written in scribbles onto your pool ZFS(nor any other filesystem) can stop it past a certain point. ZFS, despite all of it's precautions, is not a suitable substitute for good backups...which would have given you a reasonable assurance you could get your data back in the case of a hardware failure...like you experienced. I ahve to second p-chan's advice. You need to get some real hardware into that box to reduce the chances of further scribbles.

 

[Chris Moore]

I'll try what P-chan (thanks for chiming in!) suggested and give it a good cleaning then run memtest again and report back.

Did you find anything further on this?

 

[Shoop83]

Did you find anything further on this?

Haven't had time to try it yet. Going to get canned air after work and should have time to try tonight.

 

[Chris Moore]

Haven't had time to try it yet. Going to get canned air after work and should have time to try tonight.

Good luck.

 

[Shoop83]

Good luck.

After cleaning things, ran MemTest all night and found no errors this morning. I'm letting it run all day while I'm at work. This seems promising.

 

[Chris Moore]

After cleaning things, ran MemTest all night and found no errors this morning. I'm letting it run all day while I'm at work. This seems promising.

It wouldn't be the first time I saw an error clear up by cleaning and re-seating parts. This could save you from needing to buy any new parts, other than what you already bought. Sorry about that.

 

[Shoop83]

It wouldn't be the first time I saw an error clear up by cleaning and re-seating parts. This could save you from needing to buy any new parts, other than what you already bought. Sorry about that.

No worries. Having an SSD as a boot device should be more reliable than a USB stick.

What's the next step assuming I get home and find no memory errors? Do a fresh install of FreeNAS and try to import? Or would it be wise to run the solnet array test you mentioned earlier to test the disks? Something else?

 

[Chris Moore]

What's the next step assuming I get home and find no memory errors? Do a fresh install of FreeNAS and try to import?

I would say yes, with the memory working, a fresh install and importing the config file should bring you back to where you were before this problem started.
When you import the config file, the system should reboot, that is normal, but after the reboot, everything should be the way it was when the config file was saved including the pool should be imported already and shared to the network.
Later, you can consider purchasing other hardware, but you have time to plan for it instead of needing to run out and make a rush job of it.
I very much recommend the Supermicro server boards because of the IPMI remote management. It allows you to remotely connect to the system with a virtual keyboard, mouse, monitor and CD drive that are run from a window on your main PC instead of needing to connect physical devices to the server. The only wires going to my server are the power and network. I do all the management remotely from another room, unless I need to touch it for some reason.

 

[Chris Moore]

No worries. Having an SSD as a boot device should be more reliable than a USB stick.

Also, you can figure on it lasting for a very long time. Under normal use, it should last 10 years, but it might last even longer with the use in a FreeNAS boot pool.

 

[Shoop83]

I would say yes, with the memory working, a fresh install and importing the config file should bring you back to where you were before this problem started.

Famous last words! Ah hahahahaaaaaa. <-- defeated laughter of the disheartened. Shucks. It hit me with this after importing the configuration while it was coming back from rebooting. Any thoughts on this one?

Edit:
Well, after the above error I shut it down, re-cleaned the RAM and started another MemTest to run over night. Found more errors this morning. Drat.

Still looks like hardware failure, doesn't it? That fatal trap doesn't look very good.

Seriously, thanks for all your help so far. I really appreciate it.