TrueNAS 12.0-U1.1 - nginx stopping after restart, webUI not accessible

Oriann

Explorer
Joined
Dec 15, 2018
Messages
84
Hello folks

I have came back to TrueNAS after some time and I am having problem accessing web UI after restart.
Everytime I need to manually start nginx from cli with service nginx start

Does somebody have clue whats going on ?

I have tried to make my own certificate cus log say it has problem with that, but without success... or should I reinstall whole TrueNAS from scratch ? (for third time today)

2021/02/04 17:04:44 [emerg] 1968#100918: cannot load certificate "/etc/certificates/freenas_default.crt": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/certificates/freenas_default.crt','r') error:2006D080:BIO routines:BIO_new_file:no such file)
2021/02/04 17:04:44 [emerg] 1969#100918: cannot load certificate "/etc/certificates/freenas_default.crt": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/certificates/freenas_default.crt','r') error:2006D080:BIO routines:BIO_new_file:no such file)
2021/02/04 18:04:21 [emerg] 1876#100747: cannot load certificate "/etc/certificates/freenas_default.crt": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/certificates/freenas_default.crt','r') error:2006D080:BIO routines:BIO_new_file:no such file)
2021/02/04 18:04:21 [emerg] 1877#100747: cannot load certificate "/etc/certificates/freenas_default.crt": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/certificates/freenas_default.crt','r') error:2006D080:BIO routines:BIO_new_file:no such file)
2021/02/04 19:07:24 [error] 2119#100860: send() failed (54: Connection reset by peer)
2021/02/05 02:16:21 [emerg] 1986#101053: cannot load certificate "/etc/certificates/freenas_default.crt": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/certificates/freenas_default.crt','r') error:2006D080:BIO routines:BIO_new_file:no such file)
2021/02/05 02:16:21 [emerg] 1987#101053: cannot load certificate "/etc/certificates/freenas_default.crt": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/certificates/freenas_default.crt','r') error:2006D080:BIO routines:BIO_new_file:no such file)
2021/02/05 03:24:18 [error] 2220#100975: send() failed (54: Connection reset by peer)
2021/02/05 02:30:26 [emerg] 1986#100665: cannot load certificate "/etc/certificates/freenas_default.crt": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/certificates/freenas_default.crt','r') error:2006D080:BIO routines:BIO_new_file:no such file)
2021/02/05 02:30:26 [emerg] 1987#100665: cannot load certificate "/etc/certificates/freenas_default.crt": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/certificates/freenas_default.crt','r') error:2006D080:BIO routines:BIO_new_file:no such file)
2021/02/05 03:34:17 [error] 2220#100695: send() failed (54: Connection reset by peer)
2021/02/05 02:38:06 [emerg] 1987#100884: cannot load certificate "/etc/certificates/hello2.crt": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/certificates/hello2.crt','r') error:2006D080:BIO routines:BIO_new_file:no such file)
2021/02/05 02:38:06 [emerg] 1988#100884: cannot load certificate "/etc/certificates/hello2.crt": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/certificates/hello2.crt','r') error:2006D080:BIO routines:BIO_new_file:no such file)
 
Last edited:

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,740
There should be some hints on what is going wrong in /var/log/nginx/error.log.
 

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,399
You'll have to create a valid cert. Currently, your config is pointing to certs that don't exist.

In System->Certificates, you'll have to paste in the ASCII text of the certificate and the private key, along with the CN identifier.

1612535665801.png


After clicking Save, this will result in the cert files showing up in /etc/certificates.
1612535720711.png


Then set this certificate active in System->General

1612535792931.png
 

Oriann

Explorer
Joined
Dec 15, 2018
Messages
84
You'll have to create a valid cert. Currently, your config is pointing to certs that don't exist.

I have already done that but without success.
After restart the scenario is like this
1612542993861.png

Then I want connect through SSH to the server via Mobaxterm, SIDENOTE - everytime server restarts it changes server key fingerprint, this is not normal behavior

1612543066122.png


After login this is the problem
1612543099623.png


Error still the same cert does not exist even I have it in /etc/certificates and keys are valid
 

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,399
There's a fairly good chance either your machine's been hacked, or your boot pool is going bad.
 

Oriann

Explorer
Joined
Dec 15, 2018
Messages
84
I will try to reinstall once more because I am out of ideas.

None of the options are possible, SSD drive boot pool is good via SMART and I dont suppose hacked device because it started right upon first install
 

Oriann

Explorer
Joined
Dec 15, 2018
Messages
84
So from my investigation there is something bad with latest TrueNAS, after clean install I set up static ipaddress then rebooted and GUI was not accesible. I have still same error with nginx not starting at boot.

2021/02/05 19:45:45 [emerg] 1775#100778: cannot load certificate "/etc/certificates/freenas_default.crt": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/certificates/freenas_default.crt','r') error:2006D080:BIO routines:BIO_new_file:no such file)
2021/02/05 19:45:45 [emerg] 1776#100778: cannot load certificate "/etc/certificates/freenas_default.crt": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/certificates/freenas_default.crt','r') error:2006D080:BIO routines:BIO_new_file:no such file)
2021/02/05 19:52:58 [emerg] 1777#100679: cannot load certificate "/etc/certificates/freenas_default.crt": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/certificates/freenas_default.crt','r') error:2006D080:BIO routines:BIO_new_file:no such file)
2021/02/05 19:52:58 [emerg] 1778#100679: cannot load certificate "/etc/certificates/freenas_default.crt": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/certificates/freenas_default.crt','r') error:2006D080:BIO routines:BIO_new_file:no such file)
2021/02/05 20:07:33 [emerg] 1786#100808: cannot load certificate "/etc/certificates/freenas_default.crt": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/certificates/freenas_default.crt','r') error:2006D080:BIO routines:BIO_new_file:no such file)
2021/02/05 20:07:33 [emerg] 1787#100808: cannot load certificate "/etc/certificates/freenas_default.crt": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/certificates/freenas_default.crt','r') error:2006D080:BIO routines:BIO_new_file:no such file)
 

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,399
Did you verify the checksum of the ISO after download? How are you creating the install media?
 

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,399
In DD mode?
 

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,399
Also, please describe your hardware.
 

Oriann

Explorer
Joined
Dec 15, 2018
Messages
84
In DD mode?
Yes its default for TrueNAS iso

EDIT:

HW:
CPU- AMD Ryzen 5 3600 6-Core Processor
MB- Asrock fatal1ty b450 gaming k4
RAM 32GB, HyperX (2x16GB)
1xSSD NVMe Samsung boot drive
2x1TB Seagate Ironwolf / RAID1
2x4TB WD Purple / RAID1
2x6TB Seagate Skyhawk / RAID 1
 

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,399
Ah, Ryzen and a gaming motherboard. This is becoming clearer.

Are you overclocking the motherboard? Please disable all overclocks.
For Ryzen, also disable Cool'n Quiet and C6 states in the BIOS.
The motherboard also hosts a Realtek Gigabit NIC; these are known to be problematic in TrueNAS, and should be replaced with an Intel Gigabit NIC.
Also disable hardware that won't be used, like the sound card, and any RAID functionality. All SATA ports should be in just plain AHCI mode.
 
Last edited:

Oriann

Explorer
Joined
Dec 15, 2018
Messages
84
Ah, Ryzen and a gaming motherboard. This is becoming clearer.

Are you overclocking the motherboard? Please disable all overclocks.
For Ryzen, also disable Cool'n Quiet and C6 states in the BIOS.
The motherboard also hosts a Realtek Gigabit NIC; these are known to be problematic in TrueNAS, and should be replaced with an Intel Gigabit NIC.
Also disable hardware that won't be used, like the sound card, and any RAID functionality. All SATA ports should be in just plain AHCI mode.
How is my HW related to nginx that is not starting at boot ? Everything runs smoothly but something is bad compiled in TrueNAS I suppose.
Also I am not overclocking anything I run my home server as stock and everything is disabled in BIOS as you wrote.
I will now try install older TrueNAS release...see if I can dodge this bug and I will report back
 

Oriann

Explorer
Joined
Dec 15, 2018
Messages
84
So as I was expecting...older release FreeNAS 11.3-U5 booted up and restarted without a question with certificate built into system, I have done nothing but installed and run.

TrueNAS is failing in this and after first restart, nginx will crash because of some cert, there is something bad compiled in new release, can devs look into this ?

For now I have set up post-init command from gui to start nginx after boot is completed
 
Last edited:

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,740
There are literally thousands of TrueNAS 12.0-U1.1 systems world wide that don't show this problem. It must be something particular to your individual installation. I am not blaming you, your hardware, or anyone. But to get this problem debugged and solved it won't help to state "there must be a bug" on the public community forum.

Please open an issue in iXsystems' JIRA and attach a debug file of your system. That would help you and everyone who should ever encounter the same problem on their system.


Thank you for helping.
 

Oriann

Explorer
Joined
Dec 15, 2018
Messages
84
There are literally thousands of TrueNAS 12.0-U1.1 systems world wide that don't show this problem. It must be something particular to your individual installation. I am not blaming you, your hardware, or anyone. But to get this problem debugged and solved it won't help to state "there must be a bug" on the public community forum.

Please open an issue in iXsystems' JIRA and attach a debug file of your system. That would help you and everyone who should ever encounter the same problem on their system.


Thank you for helping.
Thanks I would really like to report this bug to jira, but your bug tracker is now loading for 20 minutes and still I dont see even icon of ixsystems, spamming F5 and switching between browsers is a lost duel. On cellular its almost same. ( and I dont live in jungle)
I dont blame you or dev team but from start of TrueNAS something is not right. I remember gold times with FreeNAS
 
Top