SOLVED HELP! I killed my NAS by upgrading to version 12.0-U8!!!

NumberSix

Contributor
Joined
Apr 9, 2021
Messages
188
Help!
Please help somebody! I have my entire digital life and history on my NAS, and I did a supposedly routine upgrade to version 12.0-U8 this evening and that's the last of the NAS. It hung at the time - no more communication. Powering on and off changes nothing. The web gui cannot see anything to try to connect to.

Just to add to the crisis, I no longer have the ancient cable that connected to it's old VGA output, so it's been running fine headless for many months, and many previous upgrades, but this one has killed all contact with it.

What can I do?!!!!!!!!!!

You can only imagine how desperate I am. Please help if you're able. Thank you so very much.

Frank.
 

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,464
First, calm down. In all likelihood, your data is perfectly fine. But for some reason, the system is having trouble booting. I'd speculate that your boot USB has died, but to really figure it out, you'll need to get a monitor hooked up to it.

Or just do a clean installation of 12.0-U8 to a new USB stick (or better yet, to a SSD), boot from that, then upload a copy of your config file that I'm sure you have saved--your system will be up and running.
 

NumberSix

Contributor
Joined
Apr 9, 2021
Messages
188
First, calm down. In all likelihood, your data is perfectly fine. But for some reason, the system is having trouble booting. I'd speculate that your boot USB has died, but to really figure it out, you'll need to get a monitor hooked up to it.

Or just do a clean installation of 12.0-U8 to a new USB stick (or better yet, to a SSD), boot from that, then upload a copy of your config file that I'm sure you have saved--your system will be up and running.
Hi Dan
Thank you for your calming words. However, I made (and previously tested) a clone of my Boot USB, and I just substituted it now. It's not the boot USB that's failed. Where does that leave me?

Thank you.
 

joeschmuck

Old Man
Moderator
Joined
May 28, 2011
Messages
10,972
Where does that leave me?
I recommend that you find a video cable to connect up a monitor and see what is going on, or you are just flying blind and guessing. This is the only real path forward. If you cannot figure it out once you have the video up and running, you need to provide us what you are seeing on the monitor. My guess is the BIOS needs to be configured to boot from the USB stick again, short of any other failures.
 

NumberSix

Contributor
Joined
Apr 9, 2021
Messages
188
I recommend that you find a video cable to connect up a monitor and see what is going on, or you are just flying blind and guessing. This is the only real path forward. If you cannot figure it out once you have the video up and running, you need to provide us what you are seeing on the monitor. My guess is the BIOS needs to be configured to boot from the USB stick again, short of any other failures.
Thanks Joe. You (and Dan) are quite right of course. I don't handle this kind of stress well so it's easy to panic & loose sight of the critical path. I'll buy a replacement VGA cable & get back to you when I have more details.
Thank you guys!
 

NumberSix

Contributor
Joined
Apr 9, 2021
Messages
188
Epilogue:
It's fixed!!!!
Thank you guys, both! I found the VGA cable (more on that in a moment) and booted up with a connected monitor. There was a hitherto unseen system message onscreen saying there was a 'Hard Disk' error 'on disk 2' - whatever that means. The boot drive is a USB. The system drive is a SSD. Only the data drives are hard disks, so, deciding it must be refering to the boot or system drives, I went ahead and hit F2 (check hard drive) and then selected the 'quick check' option. Sixty seconds later it 'passed' whatever it was checking, and... booted correctly! A heart stopping 18 hours for me there. Thanks again you two for 'talking me down' from elevated panic mode!

Going off topic but bizarre tale of the critical VGA cable I'll shamelessly share with you. I have a drawer in my desk where cables, current and antique, live. It was getting so full it wouldn't close properly, so a few days ago I decided to cull the oldest. Out went RF connectors (from my 8-bit BBC 'B'), out went SCART connectors from Lord knows what, and so on. Fast forward to yesterday and I can't find the VGA cable when I need it. You're probably ahead of me at this point - but 'realising' I wouldn't have thrown out a useful cable like the VGA, I searched the house for it top to bottom, three times. Nothing. So this lunchtime, finding it nowhere, I had to entertain the 'impossible', and I went bin diving through my garbage outside. We're talking old food cans, rotting vegetable matter and the detritus of a hundered torn teabags, all lightly dusted in fifty or so tiny flies. Yum it was not. And there, in the heart of this miasma - the absent-mindly discarded vital VGA cable! I never imagined poking around in a pile of trash could make me so happy. So, all's well that ends well.

Thanks again. You guys are the best!
 

NumberSix

Contributor
Joined
Apr 9, 2021
Messages
188
Hi once more.
I just noticed, following getting the NAS back, a 'Critical Error' message, saying:

CRITICAL
Update failed. See /data/update.failed for details.​

Rather than work out how to visit a folder there's no GUI link to, I'll mention the corresponding email it sent me which has an impossible amount of detail, as follows: my question is, can I ignore this?
====================================================================

New alerts:
* Update failed. See /data/update.failed for details.
* Failed to check for alert CertificateRevoked:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1247, in _execute_context
self.dialect.do_execute(
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute
cursor.execute(statement, parameters)
sqlite3.OperationalError: no such column: services_s3.s3_console_bindport

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/alert.py", line 740, in __run_source
alerts = (await alert_source.check()) or []
File "/usr/local/lib/python3.9/site-packages/middlewared/alert/source/certificates.py", line 75, in check
((await self.middleware.call('s3.config'))['certificate'], 'S3', 'certificate', 'certificate'),
File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1256, in call
return await self._call(
File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1213, in _call
return await methodobj(*prepared_call.args)
File "/usr/local/lib/python3.9/site-packages/middlewared/schema.py", line 975, in nf
return await f(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/middlewared/service.py", line 404, in config
return await self._get_or_insert(
File "/usr/local/lib/python3.9/site-packages/middlewared/service.py", line 385, in _get_or_insert
return await self.middleware.call('datastore.config', datastore, options)
File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1256, in call
return await self._call(
File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1213, in _call
return await methodobj(*prepared_call.args)
File "/usr/local/lib/python3.9/site-packages/middlewared/schema.py", line 975, in nf
return await f(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/datastore/read.py", line 186, in config
return await self.query(name, [], options)
File "/usr/local/lib/python3.9/site-packages/middlewared/schema.py", line 975, in nf
return await f(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/datastore/read.py", line 157, in query
result = await self.middleware.call("datastore.fetchall", qs)
File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1256, in call
return await self._call(
File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1213, in _call
return await methodobj(*prepared_call.args)
File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/datastore/connection.py", line 81, in fetchall
return await self.middleware.run_in_executor(self.thread_pool, self._fetchall, *args)
File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1128, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 52, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/datastore/connection.py", line 84, in _fetchall
cursor = self.connection.execute(query, params or [])
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 984, in execute
return meth(self, multiparams, params)
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/elements.py", line 293, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1097, in _execute_clauseelement
ret = self._execute_context(
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1287, in _execute_context
self._handle_dbapi_exception(
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1481, in _handle_dbapi_exception
util.raise_(
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
raise exception
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1247, in _execute_context
self.dialect.do_execute(
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such column: services_s3.s3_console_bindport
[SQL: SELECT services_s3.id, services_s3.s3_bindip, services_s3.s3_bindport, services_s3.s3_console_bindport, services_s3.s3_access_key, services_s3.s3_secret_key, services_s3.s3_mode, services_s3.s3_disks, services_s3.s3_certificate_id, services_s3.s3_browser, services_s3.s3_tls_server_uri, system_certificate_1.id, system_certificate_1.cert_type, system_certificate_1.cert_name, system_certificate_1.cert_certificate, system_certificate_1.cert_privatekey, system_certificate_1."cert_CSR", system_certificate_1.cert_signedby_id, system_certificate_1.cert_acme_uri, system_certificate_1.cert_domains_authenticators, system_certificate_1.cert_renew_days, system_certificate_1.cert_acme_id, system_certificate_1.cert_revoked_date, system_certificateauthority_1.id, system_certificateauthority_1.cert_type, system_certificateauthority_1.cert_name, system_certificateauthority_1.cert_certificate, system_certificateauthority_1.cert_privatekey, system_certificateauthority_1."cert_CSR", system_certificateauthority_1.cert_revoked_date, system_certificateauthority_1.cert_signedby_id, system_certificateauthority_2.id, system_certificateauthority_2.cert_type, system_certificateauthority_2.cert_name, system_certificateauthority_2.cert_certificate, system_certificateauthority_2.cert_privatekey, system_certificateauthority_2."cert_CSR", system_certificateauthority_2.cert_revoked_date, system_certificateauthority_2.cert_signedby_id, system_acmeregistration_1.id, system_acmeregistration_1.uri, system_acmeregistration_1.directory, system_acmeregistration_1.tos, system_acmeregistration_1.new_account_uri, system_acmeregistration_1.new_nonce_uri, system_acmeregistration_1.new_order_uri, system_acmeregistration_1.revoke_cert_uri
FROM services_s3 LEFT OUTER JOIN system_certificate AS system_certificate_1 ON system_certificate_1.id = services_s3.s3_certificate_id LEFT OUTER JOIN system_certificateauthority AS system_certificateauthority_1 ON system_certificateauthority_1.id = system_certificate_1.cert_signedby_id LEFT OUTER JOIN system_certificateauthority AS system_certificateauthority_2 ON system_certificateauthority_2.id = system_certificateauthority_1.cert_signedby_id LEFT OUTER JOIN system_acmeregistration AS system_acmeregistration_1 ON system_acmeregistration_1.id = system_certificate_1.cert_acme_id]
(Background on this error at: http://sqlalche.me/e/e3q8)

====================================================================

Can I ignore this? I note that the Dashboard of my NAS now says it has
'Version: TrueNAS-12.0-U8'

Thanks gang!
 

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,464
The boot drive is a USB. The system drive is a SSD.
Can you explain this a bit? Because it seems like a pretty strange arrangement.
 

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,399
Can I ignore this? I note that the Dashboard of my NAS now says it has 'Version: TrueNAS-12.0-U8'

Thanks gang!

The error is complaining about a certificate issue with the S3 service. If you don't use this service, as I suspect, try turning it on, then off, to populate the required tables in the configuration database.
 

NumberSix

Contributor
Joined
Apr 9, 2021
Messages
188
The error is complaining about a certificate issue with the S3 service. If you don't use this service, as I suspect, try turning it on, then off, to populate the required tables in the configuration database.
Hi
Thank you for that advice! How do I do it though? Somewhere in the GUI? I'm not terribly comfortable with the CLI so really I'd appreciate a step-by-step description of how to do it. Thank you!
 

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,399
In the GUI, navigate to Services. Click the pencil to edit the S3 service, and then just OK the default settings. Once back at the Services screen, blip the toggle to turn the service on, then off.
 

NumberSix

Contributor
Joined
Apr 9, 2021
Messages
188
Can you explain this a bit? Because it seems like a pretty strange arrangement.
Hi Dan
I'm not sure I can explain it well. Bear in mind please I'm always at the edge of my understanding with TRUENAS.

On initial setup, I had 2 HDDs to hold everything, and boot via USB. Over time, I read/believed that it would be an improvement to have the system on a different disk (minimally, it let's me back the system up to the 2 Data HDDs). When it came to finding a system disk, I chose to buy (my first ever) SSD, mainly because there's no more free drive bays in my small case, but also for added speed. In short it's evolved this way.

I know many people advocate putting the boot onto the SSD. I haven't done this, for three reasons. The first, and killer reason is that I simply don't know how to. The second reason is that data from Grafana tells me that the boot disk is constantly accessed, even when idle. If a drive is going to fail, it seems to me that it's the over-accessed boot that will fail first, and it's so easily replaced (I keep a clone on standby!), whereas replacing the state would be much more complex for me, not just to fit, but to reconfigure, I suspect. And the third reason is that old adage - if it's not broke, don't fix it.

And that's how my setup came to be.
 

joeschmuck

Old Man
Moderator
Joined
May 28, 2011
Messages
10,972
As for the boot disk, maybe the output of zpool status and zfs list would help. You do this from the Shell screen or a Putty SSH window (my preference of course) and you may need to scroll your window up to see all the data generated.

I'm not sure how you have the SSD configured in your system, for all we know you added it as a data drive, a cache drive, or nothing at all.

I want to tell you how to create a boot device from the SSD but until we know what the SSD is doing, it would be reckless of me.

And one good piece of advice, always backup your data before attempting any significant change if you are unsure what is going to happen. It's the smart thing to do.
 
Joined
Oct 22, 2019
Messages
3,589
I think the words "boot", "system", and "OS" can overlap if used casually.

When you refer to "the system" (or "the system drive"), do you mean the System Dataset? Is it that you want to spare the constant I/O operations on your USB stick (an effect of the System Dataset and logs), so you bought an SSD to specifically house the System Dataset in its own dedicated pool, separate from the boot or data pools?

If that is the case, you would be better off to install TrueNAS to the SSD, select "yes" during the installation to create a 16GB swap on the boot drive, and then once you're up and running, make sure to set the System Dataset to live on the boot-pool, as well as the system logs to also reside in the System Dataset.

This will yield a setup like so:
  • data-pool
    • mirror vdev
      • HDD
      • HDD
    • contains data
  • boot-pool
    • striped vdev (single drive)
      • SSD
    • contains 16GB swap
    • contains TrueNAS OS
    • contains System Dataset
    • contains log files
 

NumberSix

Contributor
Joined
Apr 9, 2021
Messages
188
In the GUI, navigate to Services. Click the pencil to edit the S3 service, and then just OK the default settings. Once back at the Services screen, blip the toggle to turn the service on, then off.
Hi!
It looks like I don't have a S3 service?
 

Attachments

  • screenshot.1.jpg
    screenshot.1.jpg
    76.5 KB · Views: 98

NumberSix

Contributor
Joined
Apr 9, 2021
Messages
188
When you refer to "the system" (or "the system drive"), do you mean the System Dataset? Is it that you want to spare the constant I/O operations on your USB stick (an effect of the System Dataset and logs), so you bought an SSD to specifically house the System Dataset in its own dedicated pool, separate from the boot or data pools?
Yes! That's exactly right. Thank you for stating it less ambiguously than I might have done.

Unfortunately your next paragraph fills me with more questions than is perhaps appropriate for this thread, so once I've resolved the current question (the crash, and the associated certificate service question), I'll probably get to inviting some clarification on that much postponed question, but thank you for trying!
 

NumberSix

Contributor
Joined
Apr 9, 2021
Messages
188

NumberSix

Contributor
Joined
Apr 9, 2021
Messages
188
In the GUI, navigate to Services. Click the pencil to edit the S3 service, and then just OK the default settings. Once back at the Services screen, blip the toggle to turn the service on, then off.
It says "Error starting service S3" when I try to do as you explained.

I should add that when I click the pencil to edit the default settings, the 'Save' is greyed out. IF I try to edit any field, just to see if 'Save' wakes up, nothing has any effect on it. So I come back out of configure and try to turn the service on and I get the above error message.
 

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,399
OK, it looks like a corrupted install. Try going to System->Boot, and activating your previous 12.0-U7 boot environment. Reboot back into U7, and try upgrading again.
 
Top