FreeNAS deadlocked

[Wisdom]

So here's a problem I haven't been able to find anyone else having: FreeNAS works (mostly), but I can't change anything about it. Even directly through the console.

The story: I moved recently, so I turned off all my services that I wouldn't want cropping up all of a sudden on a new internet connection. I shut FreeNAS down, packed the case back up in its box, and got set up again at the new place. While getting my internet together on this side of the move, I had to move my entire subnet due to some awful TP-Link hardware (who knew 192.168.100.0 was so popular?), which included FreeNAS. I hopped into the webGUI, changed the interface information, and thought it would be fine.

Switching back to the Networking tab, I wanted to update the domain, default gateway, etc, but every time I tried to save, I immediately got a little "unexpected error!" prompt at the top of the page. Well, okay, annoying, but maybe I can restart my way around it. Nope.

I get this same error whenever I try to do just about anything through the webGUI. From restarting and shutting down to editing network information and jails. Even the plugins I already have installed I can't turn on and off! Furthermore, I haven't been able to make any real progress with the console - I'm hesitant to start hammering lines in just because I'm getting bitter about everything going to shit in my network today.

For what it's worth, I also have issues pinging out from the console. I can hit 1.1.1.1 (my primary DNS server) just fine, but anything else, if it connects, is in the ~350ms+ range. That said, LAN times are normal (<4ms).

So, thoughts?

Specs:
G4400
Asrock C236 WSI
8GB Kingston ECC RAM
6x3TB WD Reds
Corsair RMi550
Node 304

These are from memory, you'll have to forgive me if something doesn't look quite right. PCPP is also giving me errors on all my parts lists!

 

[millst]

Console setup menu maybe?
https://doc.freenas.org/11/quick.html#

-tm

 

[Wisdom]

Console setup menu maybe?
https://doc.freenas.org/11/quick.html#

-tm

Yeah, like I said, I've had a bit of a mixed bag with the console. I got all my services up and running (their jails were still autostarting), but I don't know how to change the IP of the jail through the CLI. I'll poke around and see what I can find, but am still getting those "an error occurred!" messages constantly through the webGUI.

That's my biggest concern. Working around something like the jails and services is annoying, but manageable. What I really don't like is that something is wrong with FreeNAS, and I can't do jack inside the webGUI. That screams long term red flag to me - if this isn't working now, what issues will I be facing down the line? The lack of common complaints about this is also a little frustrating, since I don't imagine I'm the first person to ever encounter something like this.

 

[Wisdom]

Okay, now I'm really getting salty.

I've massaged the network side of things to get FreeNAS a clear view of the internet. I can ping across the world, access is great.

This matters because now I can hit the update server again. However, FreeNAS will not let me update!!!

I'm totally stuck. Can't do jack through the webGUI, and am uncomfortable pushing to do more through the command line without some more hand-holding/clear go-aheads, since I've read a few horror stories about how doing too much not through the webGUI can compromise my shares.

I've been periodically, randomly, getting django crashes as well. I have to constantly be restarting the service through the CLI to get the webGUI to be responsive. Anything from trying to see my jails to the network information will cause the service to stop and start hemorraging error messages (10+/second) over the console.

I'm really tapping out for ideas here. Anybody got something that will point me forwards?

 

[Wisdom]

Continuing my one man parade, I came at this from a different angle. I put together a whole new alternative LAN in pfsense, back on the old subnet. Reconnected to the webGUI, managed to get the half of things that a never got moved to the new subnet up and working... but am still unable to interact with anything through the webGUI. As a bonus, all the stuff on the new subnet are just as broken as the things on the old subnet were before. It's a damned-if-I-do, damned-if-I-don't kind of situation. I can pick and choose which half of my plugins and services I want to use, but I can't have both. We're pretty deep into "unacceptable" territory here.

This endless stream of "an error occurred!" messages is really making me see red. Not sure what I'm supposed to do when this seem this fucked up.

I've continued my endless googling and forum-trawling for something that relates back to this, but there isn't much. By and large, there are posts on redmine about tangentially related stuff that was fixed two years and two versions ago (we're talking about 9.3 issues that I'm having on 11.1).

E: got on 11.1-U5, through the beta UI. However, it still spits out loads of errors whenever I try to navigate around (likely a more detailed version of "an error occurred"? They seem to be related to drawing the pages though mostly, so far as I can tell).

However, in the beta-UI, the jails section won't populate at all. This is the last section I really need to have some semblance of normal - I can't find any way to change jail IPs through the CLI, nor can I use the webGUI. So SOL here. It still doesn't address the dramatically broken webGUI experience I'm having, which is still concerning, but I'd like some semblance of functionality across my plugins along the way.

Community, feel free to chip in at any time here. I'm going to keep plugging away at this, but really have little sense of direction.

 

[danb35]

However, in the beta-UI, the jails section won't populate at all.

The beta UI shows iocage jails, while the old UI shows warden jails. So that isn't going to work.

 

[Jailer]

Sounds like your install is corrupt, USB boot devices are notoriously unreliable. I'd save your config and try re installing FreeNAS and restore your config.

Why would you have to reconfigure your subnet? What does popularity have to do with it? If it's in your LAN it can be any private address space you like and shouldn't cause any issues. At any rate you'll have to update /etc/resolv.conf in each jail/plugin that you created before your subnet change and change the nameserver to reflect your new default gateway.

Also you haven't listed just how many jails, plugins or services you are running but it sounds like you are trying to do an awful lot on just 8GB of RAM.

 

[Stux]

The correct way to fix networking issues which are breaking the GUI is via the console.

It can be confusing, but I think you can fix most networking issues with the console.

 

[danb35]

...except the console doesn't do anything for the jails, which seem to be the primary current concern.

 

[Wisdom]

Holy community heavyweights, Batman!

Thanks for the suggestions everybody! I made a crosspost over onto the freenas subreddit and got some similar advice, it looks like my next step is going to be trying to reinstall from a backup of the config.

I'm glad to see that I was headed in the right direction though. I've been editing the jails one at a time (those that won't connect), and have been slowly seeing progress. However, the greater issue of the broken webGUI is something that takes priority for me, so I'll see about getting that solved first, now that I have a direction to work in.

As for jails: I've been running Plex, Transmission, CouchPotato, Sickrage, Headphones, and Syncthing (with only a single client) for a little over a year now. I don't have a lot of simultaneous heavy transcodes that hits the CPU, but most of the content I watch has subtitles that need to be burned in on the fly. I used to fuss with Maraschino, but it seems to have been abandoned for a while (and I can't be asked to care enough about external access when I don't really want that kind of opening back to my NAS anyway). I also played with an OpenVPN server, which was really great to have up and running, but have sinced axed it in favor of pushing the work off to pfsense.

I don't feel like that's a super heavy load (generally these plugins are the only thing the NAS is working on), but I do want to expand my RAM. My original build was always marked for having 16GB, but I never got around to picking up another stick, and have since been waiting for prices to fall, especially considering how pleased I've been with the performance of my build. Fitting all this, 8TB of usable, redundant storage, into the footprint of the Node 304 and stuffing it onto a shelf under my desk, has been exactly what I was looking for in a NAS. FreeNAS has been a great fit.

As for why I was fussing with the subnets: I just moved, and my new internet service required getting a box to fit between my router (pfsense standalone machine) and the coax cable coming out of the wall. The documentation for said box indicated that in order to connect to it's webGUI, I needed to connect to 192.168.100.1, the IP of pfsense on my LAN. Furthermore, the documentation seemed to indicate that anything connected to it (notably the WAN-facing side of pfsense) would need to be configured for DHCP, getting a lease on the 192.168.100.0 subnet. Since pfsense won't let the LAN and WAN sit on the same subnet, the LAN needed to move. The kicker is that the webGUI IP can't be changed, and the webGUI itself is so stupidly barebones it may as not even be a feature. TL;DR: Avoid the TC-7610.

Turns out that how it actually works is that while the webGUI is bound to 192.168.100.1, the device is otherwise configured to be permanently in bridge mode. The WAN facing side of pfsense did need to be set to DHCP, but it was to get a lease from my ISP, not the ethernet-to-coax box. Hope you enjoyed the prologue to this mess!

Back on topic: to try to avoid this issue in the future, I do have a spare 240GB SSD just sitting around. While I lack free SATA ports on my motherboard, I do have an open PCIe slot. I understand it's possible to get SATA expansion cards to allow me more drives, anyone have suggestions for something to let me move my install off of thumb drives?

Thanks to ya'll for offering some support. Sorry my last couple posts came across as kind of passive aggressive, I should have kept it together better. I'll post back when I have some fresh thumb drives and am working to copy my config over.

 

[Wisdom]

@danb35 @Jailer

I'm back for round two, with fresh boot drives.

After getting everything installed, copied over, and my config restored, it seems like things are still having trouble.

I'm routinely getting django crashes (quick fixed by

Code:

service django onerestart

through the console), which is frustrating.

However, I can write changes through the webGUI, when it's up. Progress! I've been able to get some of my services up and running (manually, through the shell in the webGUI), but others are still giving me trouble.

PLEX is working, now that I've set it to DHCP and it's getting a lease off the static mapping in pfSense. Evidently this is the best practice? However, I've tried updating it, but the webGUI is having none of it (however, it's producing real errors this time, which hopefully are being automatically submitted? That's how it reads to me).

Any more advice, or is there any more information I can fill in for you?

 

[jde]

There have been some recent posts on this forum about how a change to syncthing was causing machines to crash or other unexpected events like ssh locking up. I think it had something to do with the way syncthing was implementing folder monitoring. You can search the forum for those posts. IIRC, the problems others were having are quite similar to those you are encountering. I bet if you disable syncthing, your problems will disappear.

 

[Wisdom]

I bet if you disable syncthing, your problems will disappear.

Worth a shot, when I can get that far.

Right now I'm having issues with booting FreeNAS. I'm really not sure what the issue is here: my board can see my boot drive (a single USB, to narrow things down), the installer can see the boot drive, it can install to the boot drive, but I can't get the boot drive to actually boot. It's #1 in the boot order, with all the other possible options coming between the USB and attempting to boot off the zfs pool, but without fail I end up dumped at a system halt message, because the zfs pool has no bootable file.

The kicker is that even selecting the USB specifically as the target boot device immediately yields a failure.

I've moved ports, switched drives, reinstalled FreeNAS, fussed with the specific boot order, but nothing is letting me back in. I would say I feel like I'm backsliding, if it wasn't for the ~2 hours I was able to successfully make progress on my previous issues.

 

[jde]

Here is the link to the thread I referenced

https://forums.freenas.org/index.ph...nal-when-running-syncthing.64044/#post-458463

 

[jde]

I'm back for round two, with fresh boot drives.

The kicker is that even selecting the USB specifically as the target boot device immediately yields a failure.

Even though you got a new USB drive, I'm betting that it is still causing the boot problem. Forum posts are replete with posts about USB drives flaking out. Do you have a spare SSD or HDD that you can try installing to and booting from? I think a USB issue is much more likely than your MB or BIOS failing out of the blue.

 

[Wisdom]

Do I have a spare drive? You bet. I've got a 250GB Kingston SSD just sitting in its box.

Do I have a spare SATA port on my motherboard? No. Like my signature says, I run the ASRock E3C236D2I. It supports 6 SATA drives, all of which are in my pool (6x3TB WD Reds).

I had built my NAS out with the intention of using USB boot drives since they were considered to be acceptable forms of storage for the OS. It's disappointing that it's coming back to bite me, I had felt that the recommendations from the community were strong enough to warrant planning on its success. What's particularly surprising is that I've got through 4 different USBs now (albeit all the same brand/model) and none of them are playing nice. I'm using these super low profile Sandisk drives that I think came recommended off of Fester's FreeNAS guide (my bible for setting this whole thing up, way back when).

I asked earlier about PCI-to-SATA adapters (I think they're called HBA cards or something like that? It's been a while since I read about them), but didn't hear anything back. They're a piece of hardware that I'm wholly unfamiliar with, so I didn't want to just start trawling the forums for old recommendations in case the advice had changed, for the people who are more up to speed on this kind of stuff.

 

[Wisdom]

Here is the link to the thread I referenced

https://forums.freenas.org/index.ph...nal-when-running-syncthing.64044/#post-458463

I took a look and checked the syncthing clients I have still running, and it is not enabled. As I just received the prompt to activate it the first time I checked (again, just now) I suspect that it isn't enabled on FreeNAS either - though it's a little difficult to check for sure right now! Great information though, thanks for pointing it out for me.

 

[jde]

Do I have a spare SATA port on my motherboard? No. Like my signature says, I run the ASRock E3C236D2I. It supports 6 SATA drives, all of which are in my pool

For testing purposes you could disconnect one (or all) of the drives in your pool, hook up the SSD, and just try to get your machine booting. At least then you would know if it's a USB issue or not.

I've got through 4 different USBs now (albeit all the same brand/model) and none of them are playing nice. I'm using these super low profile Sandisk drives

What are the specs on the USB drives. It seems that there used to be an issue about trying to boot off of USB2 drives vs. USB3 drives. Perhaps someone else can chime in on that. If not search the forums. There are definitely old posts about it.

If you need to add more SATA devices, you could destroy your 6 disk pool and create a 5 disk pool. You would of course have to have a suitable backup for your data during that process. It may not be an option for you if you need all of the storage space in the 6 disk pool. How is your pool currently configured?

I asked earlier about PCI-to-SATA adapters (I think they're called HBA cards or something like that? It's been a while since I read about them), but didn't hear anything back. They're a piece of hardware that I'm wholly unfamiliar with, so I didn't want to just start trawling the forums for old recommendations in case the advice had changed, for the people who are more up to speed on this kind of stuff.

I believe the IBM 1015 host bus adapter (HBA) flashed to IT mode is still regularly recommended in this forum. They are readily available second hand on eBay for a reasonable price. Be a little careful to avoid counterfeits though. You will also need a SAS to SATA breakout cable to use with the card.

 

[Wisdom]

For testing purposes you could disconnect one (or all) of the drives in your pool, hook up the SSD, and just try to get your machine booting. At least then you would know if it's a USB issue or not.

Fair. I may do this, but on another day. It's a bit of a process to actually get the case out from inside my network stack, but if the problem persists I'll give this a shot.

What are the specs on the USB drives. It seems that there used to be an issue about trying to boot off of USB2 drives vs. USB3 drives. Perhaps someone else can chime in on that. If not search the forums. There are definitely old posts about it.

I use this drive: https://www.amazon.com/gp/product/B005FYNSZA/ref=oh_aui_detailpage_o00_s00?ie=UTF8&psc=1

It's quite nice that it's so low profile, stashing a couple of them on the back of the box was super easy.

If you need to add more SATA devices, you could destroy your 6 disk pool and create a 5 disk pool. You would of course have to have a suitable backup for your data during that process. It may not be an option for you if you need all of the storage space in the 6 disk pool. How is your pool currently configured?

This really isn't an option for me. The upload speed on my current connection has made it next to impossible to lean on my previous offsite backup, so I'm really breaking the 3-2-1 rules here.

My pool is RAID-Z2, so I have ~12TB usable in the pool.

I believe the IBM 1015 host bus adapter (HBA) flashed to IT mode is still regularly recommended in this forum. They are readily available second hand on eBay for a reasonable price. Be a little careful to avoid counterfeits though. You will also need a SAS to SATA breakout cable to use with the card.

That's the one. I'll take a look and see about picking one up, along with the breakout cables. Space is tight in the 304, but I think I can swing it.

In any case, is the need to mirror the boot drive as important when using a real SSD as opposed to something like a USB? I only have the single spare drive on hand right now.

E:

Looking at something that doesn't lean on the SuperMicro UEFI to flash the card, would something simpler like this still play nice with FreeNAS? To me, it seems like it shouldn't be a big deal to drop that in. Given the space of the enclosure, I don't need 8 more drive options, just a couple.

https://www.amazon.com/QNINE-Contro...words=pci+to+sata&refinements=p_85:2470955011

 

[jde]

I use this drive: https://www.amazon.com/gp/product/B005FYNSZA/ref=oh_aui_detailpage_o00_s00?ie=UTF8&psc=1

These are specced as USB 2, so I don't think that is the problem.

stashing a couple of them on the back of the box was super easy.

Are you installing to mirrored USB's? If so, try installing to a single USB and remove the rest. I have a FreeNAS backup target (not in my sig) that also has all six SATA connections used on the pool with mirrored USB's as boot. Occasionally the mirror goes sideways and I have to pull one or the other USB to get it to boot. You can always come in later and add a mirror to the single USB.

A bit of advice for mirrored USB's. Mark the backs of the USB's with something like different color paint pens. Add them to the system one at a time, and run

Code:

camcontrol inquiry

to get the serial number. Then note which color USB corresponds to which serial number. That way if one flakes out later you can know which one to pull.

is the need to mirror the boot drive as important when using a real SSD as opposed to something like a USB?

No

would something simpler like this still play nice with FreeNAS? To me, it seems like it shouldn't be a big deal to drop that in. Given the space of the enclosure, I don't need 8 more drive options, just a couple.

https://www.amazon.com/QNINE-Contro...words=pci+to+sata&refinements=p_85:2470955011

I don't know. I'd say it's a crapshoot at best. The IBM 1015 has been the go-to workhorse HBA in these forums for years. It is known to work. Personally, I would not put the SATA controller you listed in my system. I saw a reference in another thread to a SYBA controller that someone got to work. (EDIT https://forums.freenas.org/index.php?threads/cheap-pci-sata-controller.42537/#post-276363) IIRC the price of that model of SYBA controller was $30-40. personally I would rather pay an additional $30 to get the 1015 if it were my system.