Updates fail

[billsey]

The only errors I see are:

Code:

Jan  5 13:38:28 nas01 syslog-ng[5607]: I/O error occurred while writing; fd='28', error='Unknown error: 122 (122)'
Jan  5 13:38:28 nas01 syslog-ng[5607]: Suspending write operation because of an I/O error; fd='28', time_reopen='60

Over and over and over... when I SSH in and do a 'df' I don't see anything that's full and I can't seem to find out what the fd='28' refers to. Nothing else seems to be failing, so I'm hoping if someone can let me in on how to tell what's really causing the error I can fix it and the updates will finally happen. I'm running FreeNAS-9.10.1-U4 (ec9a7d3) on a Dell server with 20GB of ram and a single 8TB partition (PERC controller sets the partition up and provides it as a single drive).

 

[dlavigne]

It might be useful to save your config, burn the latest version to a different stick, restore your config and see how that works.

 

[m0nkey_]

Two things:
1. Replace the boot drive as suggested by @dlavigne.
2. Using a RAID controller on FreeNAS is not recommended. If you have a OEM branded LSI card, there's a chance you can cross flash it to IT mode. However, doing that with your current set-up will cause you to lose you pool, so you will need to backup your data first.

 

[billsey]

The boot drive is a small partition on the main drive. Not really feasible to replace. No one knows what's wrong with the update process that causes these errors?

Code:

[root@nas01] /# df
Filesystem  1K-blocks  Used  Avail Capacity  Mounted on
freenas-boot/ROOT/9.10.1-U4  1316114  641894  674219  49%  /
devfs  1  1  0  100%  /dev
tmpfs  32768  8440  24328  26%  /etc
tmpfs  4096  8  4088  0%  /mnt
tmpfs  6979012  96884  6882128  1%  /var
freenas-boot/grub  680704  6485  674219  1%  /boot/grub
fdescfs  1  1  0  100%  /dev/fd
User_Data  7538711976 2264611988 5274099988  30%  /mnt/User_Data
User_Data/.system  5274509428  409440 5274099988  0%  /var/db/system
User_Data/.system/cores  5274153420  53432 5274099988  0%  /var/db/system/cores
User_Data/.system/samba4  5274122760  22772 5274099988  0%  /var/db/system/samba4
User_Data/.system/syslog-86d99cfd26b4471f8f9359adef356993  5276017864  1917876 5274099988  0%  /var/db/system/syslog-86d99cfd26b4471f8f9359adef356993
User_Data/.system/rrd-86d99cfd26b4471f8f9359adef356993  5274100132  144 5274099988  0%  /var/db/system/rrd-86d99cfd26b4471f8f9359adef356993
User_Data/.system/configs-86d99cfd26b4471f8f9359adef356993 5274291988  192000 5274099988  0%  /var/db/system/configs-86d99cfd26b4471f8f9359adef356993
linprocfs  4  4  0  100%  /compat/linux/proc

As you can see, there doesn't seem to be any obvious issues with the free space available.

 

[sef]

Given that it said I/O error, why do you think it's a space issue?

 

[billsey]

In addition to the many, many errors flowing past that match the previous, each attempt at an update generates:

Code:

Jan 18 16:08:45 nas01 manage.py: [middleware.exceptions:37] [MiddlewareError: 'Insufficent space to install update']

 

[sef]

Run "freenas-update -v check" and "freenas-update -v update" from the shell.

I suspect that an exception is being caught and reported as out of space.

 

[SweetAndLow]

The boot drive is a small partition on the main drive. Not really feasible to replace. No one knows what's wrong with the update process that causes these errors?

Code:

[root@nas01] /# df
Filesystem  1K-blocks  Used  Avail Capacity  Mounted on
freenas-boot/ROOT/9.10.1-U4  1316114  641894  674219  49%  /
devfs  1  1  0  100%  /dev
tmpfs  32768  8440  24328  26%  /etc
tmpfs  4096  8  4088  0%  /mnt
tmpfs  6979012  96884  6882128  1%  /var
freenas-boot/grub  680704  6485  674219  1%  /boot/grub
fdescfs  1  1  0  100%  /dev/fd
User_Data  7538711976 2264611988 5274099988  30%  /mnt/User_Data
User_Data/.system  5274509428  409440 5274099988  0%  /var/db/system
User_Data/.system/cores  5274153420  53432 5274099988  0%  /var/db/system/cores
User_Data/.system/samba4  5274122760  22772 5274099988  0%  /var/db/system/samba4
User_Data/.system/syslog-86d99cfd26b4471f8f9359adef356993  5276017864  1917876 5274099988  0%  /var/db/system/syslog-86d99cfd26b4471f8f9359adef356993
User_Data/.system/rrd-86d99cfd26b4471f8f9359adef356993  5274100132  144 5274099988  0%  /var/db/system/rrd-86d99cfd26b4471f8f9359adef356993
User_Data/.system/configs-86d99cfd26b4471f8f9359adef356993 5274291988  192000 5274099988  0%  /var/db/system/configs-86d99cfd26b4471f8f9359adef356993
linprocfs  4  4  0  100%  /compat/linux/proc

As you can see, there doesn't seem to be any obvious issues with the free space available.

Boot environment on the same raid as your pool. Where is the popcorn this is going to be fun.

Sent from my Nexus 5X using Tapatalk

 

[billsey]

[freenasOS.Configuration:83] CheckFreeSpace(path=None, pool=freenas-boot, required=1354721280)
[freenasOS.Configuration:109] pool_used (3170806272) + required (1354721280) > pool_max (3593679667)
[freenasOS.Update:371] Cannot prune clone default since keep is None
[freenasOS.Update:403] Clone default not eligible for pruning
[freenasOS.Update:371] Cannot prune clone FreeNAS-9.3-STABLE-201605170422 since keep is None
[freenasOS.Update:403] Clone FreeNAS-9.3-STABLE-201605170422 not eligible for pruning
[freenasOS.Update:371] Cannot prune clone FreeNAS-ca82ba222c0be179a6983636c50732c3 since keep is None
[freenasOS.Update:403] Clone FreeNAS-ca82ba222c0be179a6983636c50732c3 not eligible for pruning
[freenasOS.Update:371] Cannot prune clone 9.10.1-U2 since keep is None
[freenasOS.Update:403] Clone 9.10.1-U2 not eligible for pruning
[freenasOS.Update:371] Cannot prune clone 9.10.1-U4 since keep is None
[freenasOS.Update:403] Clone 9.10.1-U4 not eligible for pruning
[freenasOS.Update:406] Done with prune loop. Must have failed.
[freenas-update:270] 'Insufficent space to install update'
Insufficent space to install update

 

[billsey]

So, I set the latest two boot images to Keep, then deleted each of the older images. That freed up enough room to do the update. I'll keep the boot images trimmed to the latest two from now on.

It turns out there is a 4GB thumb drive internal to the machine that it's booting from. I hadn't pulled the machine to open it before...

 

[billsey]

Still getting the syslog errors though. Where are syslog messages saved?

 

[SweetAndLow]

Still getting the syslog errors though. Where are syslog messages saved?

You do realize that your hardware setup is insane and causing all your problems? You can't fix them unless you remove your raid configuration. Your performance will be terrible(proof because of your write error), your data will end up being corrupt, you won't be able to rebuild your pool when a disk fails, you can't expand your pool and you won't be able to move it to different hardware. Try reading the noob guide and you might start to understand.

I'll reinterate to fix your

I/O error occurred while writing; fd='28', error='Unknown error: 122 (122)'

error you need to replace your raid card with a hba card.

 

[billsey]

I fail to see how a system with a hardware raid controller is different than a system with one disk drive. I have already rebuilt the raid once when a disk failed, so that point isn't valid. I do realize that I can't expand my pool, but all drive slots are currently filled so expanding would require new hardware anyway. Why should my data end up being corrupt?

 

[SweetAndLow]

I fail to see how a system with a hardware raid controller is different than a system with one disk drive. I have already rebuilt the raid once when a disk failed, so that point isn't valid. I do realize that I can't expand my pool, but all drive slots are currently filled so expanding would require new hardware anyway. Why should my data end up being corrupt?

Have you even tried to read the guides or best practices? It's all over the place, zfs expect raw access to the disk and when you use a raid card you introduce an extra layer that does stuff behind zfs back. Not to mention the write cache that tells zfs the data is written to disk but it actually isn't. This is super common knowledge.

Sent from my Nexus 5X using Tapatalk

 

[billsey]

I'm sorry, I'm not trying to be confrontational, but do you understand what a hardware raid controller is? It looks to the file system just as if it were a disk and in the case of the PERC 6/i handles caching with a battery backup, so cached data isn't lost even when the power is cycled. I could see your point if it were running one of the software raid controllers, but it just doesn't make sense when talking about a hardware raid.

 

[billsey]

Anyway, that's pretty much off topic. If no one knows where syslog is trying to write files I'll just ignore the messages...

 

[SweetAndLow]

I'm sorry, I'm not trying to be confrontational, but do you understand what a hardware raid controller is? It looks to the file system just as if it were a disk and in the case of the PERC 6/i handles caching with a battery backup, so cached data isn't lost even when the power is cycled. I could see your point if it were running one of the software raid controllers, but it just doesn't make sense when talking about a hardware raid.

Ok this conversation should be over because you are not going to change. But anyone else reading this just know that you should not use a raid controller with zfs in most cases because it will result in data corruption and loss of your pool.

 

[sef]

That's not true. You shouldn't use a raid controller because it offers less flexibility, compatibility, and power than zfs does, but it's no more likely to have data corruption with zfs than without.