TrueNAS SCALE /etc/hosts file not persistent across reboots

[Chin-Fah HEOH]

Hi

Anyone encountered this issue?

I have just installed TrueNAS SCALE 22.02.0.1 (latest one as of last week) for the first time, and configurations went well. I set up 6 nodes of TrueNAS SCALE with a full /etc/hosts file which I edited with the vi editor. But the /etc/hosts file resets to default (empties all my entries) after every reboot. No real harm except being a hassle.

Planning a 2x3 Distributed Replicated volume and a 4+2 Dispersed volume for Gluster testing. The 6 node Gluster cluster is working fine so far, but haven't tested the volumes integration with the Gluster client on another laptop (probably testing with Fedora or Ubuntu).

Note: All these 6 nodes are running in Virtualbox VMs with 4GB of RAM each. My server has Intel i7 Gen 2, and limited RAM and that's all I can afford. Vbox version is 6.1.32 r149290.

Looking for answers and advice. Thanks

 

[Samuel Tai]

You're not supposed to edit /etc/hosts directly, but via the middleware, using Network->Global Configuration->Other Settings->Host Name Database.

Host Name Database

Additional hosts to append to /etc/hosts. Separate entries by pressing Enter. Use the format IP_address space hostname where multiple hostnames can be used if separated by a space. Hosts defined here are still accessible by name even when DNS is not available. See hosts for additional information.

 

[sretalla]

All these 6 nodes are running in Virtualbox VMs with 4GB of RAM each

Plan to have many ZFS and general stability issues. That's not enough to run ZFS alone, but with Gluster and anything else on top, forget it.

All that your testing will prove is that you get instability and bad results with that little RAM. Stop wasting your time on an invalid test, you won't learn anything useful.

 

[Kris Moore]

Plan to have many ZFS and general stability issues. That's not enough to run ZFS alone, but with Gluster and anything else on top, forget it.

All that your testing will prove is that you get instability and bad results with that little RAM. Stop wasting your time on an invalid test, you won't learn anything useful.

Have to 2nd this. 16G would be the bare minimum you'd need to get anything approaching a valid test with Gluster, anything less and we'd expect there to be issues.

 

[Chin-Fah HEOH]

You're not supposed to edit /etc/hosts directly, but via the middleware, using Network->Global Configuration->Other Settings->Host Name Database.

Host Name Database

Additional hosts to append to /etc/hosts. Separate entries by pressing Enter. Use the format IP_address space hostname where multiple hostnames can be used if separated by a space. Hosts defined here are still accessible by name even when DNS is not available. See hosts for additional information.

Thanks. I was following the Linux way.

 

[Chin-Fah HEOH]

Thanks. Don't plan to build anything super powerful, and just for knowledge sake.

Plan to have many ZFS and general stability issues. That's not enough to run ZFS alone, but with Gluster and anything else on top, forget it.

All that your testing will prove is that you get instability and bad results with that little RAM. Stop wasting your time on an invalid test, you won't learn anything useful.

 

[Chin-Fah HEOH]

Have to 2nd this. 16G would be the bare minimum you'd need to get anything approaching a valid test with Gluster, anything less and we'd expect there to be issues.

Thanks Kris. Really doing it for knowledge sake to advise a couple of resellers here about using Gluster. No plans to go beyond other than testing the work. Unless someone wants to sponsor some high spec server for me. ...

 

[Arwen]

Thanks Kris. Really doing it for knowledge sake to advise a couple of resellers here about using Gluster. No plans to go beyond other than testing the work. Unless someone wants to sponsor some high spec server for me. ...

Well, the problem of testing with that low of memory, is that if you get crashes or strange behavior, you may never know if it is the low memory or something else.

 

[Chin-Fah HEOH]

Well, the problem of testing with that low of memory, is that if you get crashes or strange behavior, you may never know if it is the low memory or something else.

Thanks. It is just a functionality test.

 

[Arwen]

Thanks. It is just a functionality test.

Then that makes more sense.

Good luck.

Oh, and one last comment. Both TrueNAS Core & SCALE are designed as appliances. While they may use standard kernels, (FreeBSD & Linux), as well as many stock / standard Unix tools, they are not designed for under the hood changes. Like using VI to edit "/etc/hosts", (as you found out).

So, if you need to add some custom code, implement it in a container of some sort, (docker or VM). Otherwise it may be lost during an update, (though rebooting to the prior boot environment may bring it back).

Plus, special boot time options need to be done in such a way that the middleware knows about it. (Either through the GUI or the middleware's command line, which is NOT a Unix shell.)

 

[John Rushford]

I just did a fresh install of TrueNAS-SCALE-22.12.2. I was trying to update the hosts file on the machine for my needs. I went through the UI and added my entries to the hostname database through the Network global configurations. After saving my entries, I noticed the /etc/hosts did not reflect my additions unil after a reboot. You shouldn't have to reboot the machine to persist changes to the /etc/hosts. I'd call this a bug.

One of the reasons I added to the hosts database was to add my stratum one NTP server so that I could add it in to my TrueNAS. I was able to update the NTP servers list through the UI and after saving the addition, I saw that /etc/ntp.conf was updated and ntp was restarted after saving. No reboot was required, that's how it should be through the UI. Unfortunately, changes to the hostname database require a reboot.

 

[jgreco]

You shouldn't have to reboot the machine to persist changes to the /etc/hosts. I'd call this a bug.

It could be seen either way. Since editing /etc/hosts has impacts all over the system, the only way to make the change globally is to reboot.

To use your case as an example, if you are adding or overriding a name that would normally be answered by DNS, how would ntpd know about the change in the hosts file? It would need to be restarted in many cases. This applies to many other cases where local names are used in exports or other ACL uses as well. While there are cases where a reboot isn't needed, the middleware cannot be aware of all of them, and its intent is to reliably achieve a particular given system state. Ambiguity is bad. I would say that the correct fix would be to force a reboot after hosts file changes.

 

[John Rushford]

It could be seen either way. Since editing /etc/hosts has impacts all over the system, the only way to make the change globally is to reboot.

To use your case as an example, if you are adding or overriding a name that would normally be answered by DNS, how would ntpd know about the change in the hosts file? It would need to be restarted in many cases. This applies to many other cases where local names are used in exports or other ACL uses as well. While there are cases where a reboot isn't needed, the middleware cannot be aware of all of them, and its intent is to reliably achieve a particular given system state. Ambiguity is bad. I would say that the correct fix would be to force a reboot after hosts file changes.

I don't buy that, rebooting to update the hosts file impacts the service it provides pretty dramatically. In my particular case, I have a small private network and if the NTP server were in DNS I wouldn't have needed to add it to /etc/hosts, I added to hosts so that I could add it by name to ntp.conf. That aside, I disagree with you. I think a reboot to update /etc/hosts is ridiculous, can you imagine as a systems administrator having to reboot each time you updated the hosts file on any machine? I could see it now, writing up tickets and scheduling downtime just to make a minor change to an entry in the hosts file, that's crazy.

 

[sretalla]

As it stands, if you want an immediate change without reboot, you can edit the file directly and then also make the changes in Network Global Configuration to ensure it persists after that.

If you see that as a bug, file one in Jira.

 

[jgreco]

Can you imagine as a systems administrator having to reboot each time you updated the hosts file on any machine?

You can disagree all you like, but without any sort of mechanism to be able to identify what subsystems are affected by the changes you make, the ability of the system to properly reinstantiate services is a real problem.

I guess this is probably why we invented DNS back in the 1980's. Some of us still remember the days of actually using hosts files (anyone remember $env['HOSTALIASES']?)

Anyways, you're being a little ridiculous because a sysadmin is intended to understand the context of the changes he/she is making to a hosts file, and then manually restart those services. In the appliance model, it is the middleware's job to be able to understand that, making it unnecessary for there to be a sysadmin who understands all the dependencies. Sometimes this is done by brute force, such as when you make a netwoek interface change and the system essentially deprovisions all the IP networking before reprovisioning it. We have not actually reached a point where a machine can reliably be taught how to do this, so sometimes the brute force solution or just ignoring the problem entirely are the practical options.

 

[Kris Moore]

I think there is a fundamental misunderstanding of editing files like /etc/hosts by hand. TrueNAS isn't some generic *Nix server OS that you administrate by SSH'ing in and doing things directly at the OS level. The middleware is our state engine that makes it run like an appliance. If there is some need to control things in /etc/hosts, then the proper request would a new feature to the MW / API which allows CRUD of hosts entries. This is so that they are persistent across upgrades, and can be backed up / restored in the configuration DB file.

 

[John Rushford]

I think there is a fundamental misunderstanding of editing files like /etc/hosts by hand. TrueNAS isn't some generic *Nix server OS that you administrate by SSH'ing in and doing things directly at the OS level. The middleware is our state engine that makes it run like an appliance. If there is some need to control things in /etc/hosts, then the proper request would a new feature to the MW / API which allows CRUD of hosts entries. This is so that they are persistent across upgrades, and can be backed up / restored in the configuration DB file.

No, I understand that and I use the UI to update through the middleware. I just believe that once you hit save it should also update the hosts file. Having to reboot to get the change in /etc/hosts is nuts in my opinion.

 

[John Rushford]

You can disagree all you like, but without any sort of mechanism to be able to identify what subsystems are affected by the changes you make, the ability of the system to properly reinstantiate services is a real problem.

I guess this is probably why we invented DNS back in the 1980's. Some of us still remember the days of actually using hosts files (anyone remember $env['HOSTALIASES']?)

Anyways, you're being a little ridiculous because a sysadmin is intended to understand the context of the changes he/she is making to a hosts file, and then manually restart those services. In the appliance model, it is the middleware's job to be able to understand that, making it unnecessary for there to be a sysadmin who understands all the dependencies. Sometimes this is done by brute force, such as when you make a netwoek interface change and the system essentially deprovisions all the IP networking before reprovisioning it. We have not actually reached a point where a machine can reliably be taught how to do this, so sometimes the brute force solution or just ignoring the problem entirely are the practical options.

I fully understand the ramifications of updating /etc/hosts so, no need to patronize. I've been working with unix systems since 1980 as a sysadmin and developer. I disagree that the truenas needs to behave this way when updating that file. I added a hosts entry for an NTP server, I know what uses it on Truenas and I took the steps to update the ntp.config to use the new entry. I found that Truenas restarted ntp for me when i hit save and I thought that was pretty awesome actually but, having to restart my entire NAS to get host change seems ridiculous to me. Let me ask you, do you reboot your DNS servers when you make updates there? Of course you don't. You'd be better insuring software that may query /etc/hosts is able to tolerate changes when they happen.

I'll write up a bug report and see where that goes. While waiting for it, I'll update the host database and manually edit /etc/hosts to avoid the boot.

 

[John Rushford]

As it stands, if you want an immediate change without reboot, you can edit the file directly and then also make the changes in Network Global Configuration to ensure it persists after that.

If you see that as a bug, file one in Jira.

based upon the replies I see here, my bug report would probably end up in /dev/null

 

[Kris Moore]

based upon the replies I see here, my bug report would probably end up in /dev/null

Nah, please do share the bug report ID with me, I'll take a look and make sure we give it a proper review :)