Boot stall or fail.

[kennis942]

Recently as my pool has been filling up, ive reached 89% capacity. (above recomended)
and so full it can no longer write logs or tell me what is wrong.
after v11.1+ and i have never accepted these updates (original 9.11) my system and its functions have been contenually randomly been stopped (Plex, smb, transmission etc) I dont know what to look for but boot looks normal untuil it comes to booting root on freenas [version]:blah blah blah ... and takes unusually long to boot compared to a few weeks ago or ever before. there are no errors as i can see, and its so bad now i dont dare to reboot it becasue it takes about 15 reboots to succeed. and when it succeds, my services (jails) are started but only two system prosesses are visable via top. (so jail is not starting propperly)
I have now deleted about 1 TB and the system still have this problem. i have not changed any settings recently all i have done is loaded it with maybe too much data.
Sometimes in the past it would stop smb due to probably network overflow, and ive had to hard reset as i cannot putty or use the GUI after that, this migth be whats casuing my problem, but i highly doubt it as it is extremly rare.

Setup :
Bootdrive: 126 Gb usb drive.
Pool : 4x 6 TB WD red
Build FreeNAS-11.2-RELEASE-U1
Platform Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz
Memory 16235MB

My question is: how do i get my system in working order agian and or troubleshoot this when i dont even have an error mesage to go by ? it has been running for 2 years 24/7 with no issues other than what i stated above.
I do not have the option to add more drives or do anything that costs money. (that was the plan)

 

[Chris Moore]

it has been running for 2 years 24/7 with no issues other than what i stated above.

That is long enough to wear out a USB memory stick.

Bootdrive: 126 Gb usb drive.

Bigger doesn't make it better. It is probably going bad and needs to be replaced. You only need 32GB.

Recently as my pool has been filling up, I've reached 89% capacity.

Probably not related to the issue you are having with the system not booting, but you should either delete something or expand capacity.

I do not have the option to add more drives or do anything that costs money. (that was the plan)

What was the plan, that this system would last forever and never need to be repaired or upgraded?

 

[kennis942]

TTo upgrade the system was the plan but i no longer have the finacial capability of doing so in the next 3 years. ( and i lost that ability about 6 months ago)
It was about 14 Tibi , i have deleted more than 50% of the content through SFTP/WinScp, but the capazity does not reflect that. ( it still shows way more than what i know i deleted. (in one session about 700 GB)
I also have issues acessing the system at all so im really out of options, it is a miracle if i get ssh connection, and takes several reboots.
granted all of this data is not critical, but 1.3 TB of it is invaluable and can never be recovered or gained from anywhere else than these drives.
As you suggested, i deleted files, it took me 3 days non stop to do that but the problem remains.

Bigger doesn't make it better. It is probably going bad and needs to be replaced. You only need 32GB.

Replace the usb then ?
How would i go about that ? I know there are guides on installing / upgrading but i have to be 100% sure i am doing the rigth thing and looking at the rigth place be fore i even touch the machine, in any way, at this point as it is barely alive. and that is why i descided to post this thread in spite of how incredably stupid i portray my problem.

 

[Apollo]

You can run at 89% capacity without issues, it is not going to give you trouble with read capability if you have enough RAM. Writing shouldn't really be too bad either. You are just going to be on the slowest side of the HDD but not an issue to make your system unusable. It will just not be that snappy as on its prime days.
I think the issue you are experiencing is directly related to 11.1 which has been terrible with iocage jails support, simply jails and netwokr wouldn't work nicely together.,
When you reboot your system, and let's assume you don't enable the iocage jails, there will be some time when Freenas (ZFS) will be loading the content of the pool and it can take a bit of time. I think it is part of the cache being restored in order to improve performance.

If you have backups, and it seems you don't, I would have suggested you upgrade to 11.2, but in your case it is not without risk.
When you delete files through SMB or any other means, you will not free the space if and when you have snapshots pointing to that data.
To clear up and make room, you will need to get rid of some of the snapshots.

 

[SweetAndLow]

Let's just do a fresh install on a new usb stick to test things out. You don't even need your backup database if you don't want to but I would grab it anyways.

If the fresh install doesn't change anything then you can boot up the old USB with no harm done.

Seeing the output of zpool status, zpool list and your full hardware specs would also be helpful. Motherboard specifically needs to be included.

 

[kennis942]

I dont know where to find the logs of the boot readout.
zfs pool of the 4xTB WD RED has no errors.
MB: MSI H110I Pro, Socket-1151
http://prntscr.com/niq2rm

One of my jails is failing to boot those times it does not stall, i am amble to aceess files via smb, and transmission(torrentclient) service which is on that jail says the disk is full, but it is definitely not. Marked out in red are deactivated by pourpuse jail containers.

 

[kennis942]

I think the issue you are experiencing is directly related to 11.1 which has been terrible with iocage jails support, simply jails and netwokr wouldn't work nicely together.,
When you reboot your system, and let's assume you don't enable the iocage jails, there will be some time when Freenas (ZFS) will be loading the content of the pool and it can take a bit of time. I think it is part of the cache being restored in order to improve performance.

If you have backups, and it seems you don't, I would have suggested you upgrade to 11.2, but in your case it is not without risk.
When you delete files through SMB or any other means, you will not free the space if and when you have snapshots pointing to that data.
To clear up and make room, you will need to get rid of some of the snapshots.

Deleting snapshots seems to temporarily have fixed my problem everything seems to be okay now.
...for the future should i just wait for a freenas release or application to migrate jails and icage crap to newer os versions ?
Also how do i go about copying exactly the boot drive to a newer smaller flash drive (32GB) since i do not trust the one i am currently using ?

 

[Apollo]

Deleting snapshots seems to temporarily have fixed my problem everything seems to be okay now.
...for the future should i just wait for a freenas release or application to migrate jails and icage crap to newer os versions ?
Also how do i go about copying exactly the boot drive to a newer smaller flash drive (32GB) since i do not trust the one i am currently using ?

Unless Freenas is screaming for free space, deleting snapshots shouldn't have a direct effect.
For the boot disk, I would have recommended mirroring it, but going with a smaller size drive would be impossible I think.
There are ways you can go about.
You can perform a fresh install to a new disk and you need to copy the main config file your system is on and also the seeds and .ssh files. Can't remember where but I believe I posted something a year or so ago.
Theoritically, replication should be doable but with Grub or the boot partition I don't know if it is possible.
It maybe possible to make an image of the disk with Rufus, Filezilla or the like and restore its content while resizing to the new partitions. Maybe "dd" could be used, but I haven't really experimented with this approach myself, so I can't confirm on the process.

The config file and the .ssh and the like are closer to the easiest thing to do.