Why do my boot drives keep failing?

[Fuganater]

I had a boot drive "fail" so I replace it via the GUI. Now it has happened again. I tried 3 flash drives and I can't believe that all 3 are bad. 16GB Sandisk Cruiser Fit. Server has been running for almost 2 years with no issues. I need some serious help.

 

[William Grzybowski]

Do you have replication tasks set up in that server?

We have just discovered an issue which could cause excessive write to the boot pool.

 

[Fuganater]

Yes. It replicates every night to my backup FreeNAS server.

 

[William Grzybowski]

So it only replicates once per day? How often are the snapshots taken?

That seems unlikely to be the cause.
Can you post a screenshot of your boot disk from Reporting -> Disk? A time-frame of a day and a week would be OK.

 

[Fuganater]

So it only replicates once per day? How often are the snapshots taken?

That seems unlikely to be the cause.

Snapshots happen once every 12 hours. Replication occurs between 00:00 and 07:00 daily. ( As far as I can remember)

Can you post a screenshot of your boot disk from Reporting -> Disk? A time-frame of a day and a week would be OK.

How do I do that when I can't even boot the system?

 

[Vito Reiter]

There are several reasons that this happens, Flash Drives don't really handle tons and tons of reads and writes well. So look into these few things:

• Verbose Logging
• System Datasets
• Syslog and Reporting Database

These can definitely do a number on your boot device especially if it's a lower-end flash drive. Hope this helps a bit and just remember to keep I/O's on the boot device as low as possible.

Regards,
Vito Reiter

 

[William Grzybowski]

Snapshots happen once every 12 hours. Replication occurs between 00:00 and 07:00 daily. ( As far as I can remember)



How do I do that when I can't even boot the system?

The device of boot pool can be found in System -> Boot -> Status.

Where is your system dataset (system -> system dataset) located?

 

[Fuganater]

The device of boot pool can be found in System -> Boot -> Status.

Where is your system dataset (system -> system dataset) located?

Ok I got in booting in "Verbose Mode". "Normal Mode" produces the OP errors.

The System Dataset is located in Vol1 which is my main volume.

 

[William Grzybowski]

Ok I got in booting in "Verbose Mode". "Normal Mode" produces the OP errors.

The System Dataset is located in Vol1 which is my main volume.

I am not sure what you mean by Verbose/Normal nor what difference it makes.

 

[Fuganater]

I am not sure what you mean by Verbose/Normal nor what difference it makes.

When you boot FreeNAS, you get an option of which "boot" you want to boot to. IDK the actual name but you get a list,
Initial-Install
default
FreeNAS-9.3-STABLE-NUMBERS
FreeNAS-9.3-STABLE-NUMBERS
FreeNAS-9.3-STABLE-NUMBERS
FreeNAS-9.3-STABLE-NUMBERS

Then once you pick that, you get 3 mode options:
Normal Mode
Single-User Mode
Verbose Mode


Normal Mode produces all those errors, Verbose let me boot all the way up.

 

[Vito Reiter]

I didn't realize you couldn't boot in normal mode, verbose mode, as far as I know, just tells you what's going on. It wouldn't be catastrophic to backup your config and reinstall (Although inconvenient). Perhaps look into some log settings while in verbose mode and see if there's anything that makes sense.

 

[PhilipS]

As far as boot volume flash wear, if you are using kerberos: I have discovered that everytime an SMB client connects, samba/krb5 will rewrite a file called /.rnd - several times. This is stored on your boot volume and I was getting almost 1GB of writes to my flash each day due to this - as well as those writes caused by replication that William mentioned - but I replicate every 15 minutes.

I found that I can remove the kerberos realm and restart directory services and stop this behavior - as long as your installation doesn't depend on kerberos (like impersonation).

Edit: It appears FreeNAS is smart enough to setup the kerberos realm for you so removing the realm did not work. Also, the developers have not been able to reproduce this yet so it could be something unique to my systems.