nextcloud 13 on FreeNAS 11 just stopped working one day

[cunningorb]

A couple of weeks ago I noticed when I visited my nextcloud from work that the ssl cert had lost renewal. I couldn't immediately troubleshoot and finally have a modicum of time but my snapshots don't go back that far so I need to know what's up with my nextcloud in iocage jail.

My nextcloud was setup using the script from @danb35 (thanks!) and it was working great for several months. Now I can't get the certbot renew command to work and I think it has something to do with the jail. The jail is not able to update packages at all. It has internet, I pinged google.com which was successful and ran the command host pkg.freebsd.org which returned:

Code:

pkg.freebsd.org is an alias for pkgmir.geo.freebsd.org.
pkgmir.geo.freebsd.org has address 96.47.72.71
pkgmir.geo.freebsd.org has IPv6 address 2610:1c1:1:606c::50:1

Here's my attempt to update packages in the jail:

Code:

pkg update
Updating local repository catalogue...
pkg: Repository local load error: access repo file(/var/db/pkg/repo-local.sqlite) failed: No such file or directory
pkg: file:///usr/ports/packages/meta.txz: No such file or directory
repository local has no meta file, using default settings
pkg: file:///usr/ports/packages/packagesite.txz: No such file or directory
Unable to update repository local
Error updating repositories!

Where do I even start here. One day it worked, the next my nextcloud was brought to its knees. I know my files are fine but I have other people using this and I can't just wipe it out and start over.

 

[danb35]

Now I can't get the certbot renew command to work and I think it has something to do with the jail.

If you used my script, it doesn't use certbot but acme.sh. Log in to the jail ( iocage exec nextcloud) and try acme.sh --cron.

The issue with pkg upgrade looks like you've lost your mountpoint for the portsnap dataset. What's the output of iocage fstab -l nextcloud?

 

[cunningorb]

If you used my script, it doesn't use certbot but acme.sh. Log in to the jail ( iocage exec nextcloud) and try acme.sh --cron.

The issue with pkg upgrade looks like you've lost your mountpoint for the portsnap dataset. What's the output of iocage fstab -l nextcloud?

root@freenas:~ # acme.sh --cron
acme.sh: Command not found.

and

Code:

root@freenas:~ # iocage fstab -l nextcloud
+-------+--------------------------------------------------------------------------------------+
| INDEX |									 FSTAB ENTRY									  |
+=======+======================================================================================+
| 0	 | /mnt/orb1/portsnap/ports /mnt/iocage/jails/nextcloud/root/usr/ports nullfs rw 0 0	|
+-------+--------------------------------------------------------------------------------------+
| 1	 | /mnt/orb1/portsnap/db /mnt/iocage/jails/nextcloud/root/var/db/portsnap nullfs rw 0 0 |
+-------+--------------------------------------------------------------------------------------+
| 2	 | /mnt/orb1/files /mnt/iocage/jails/nextcloud/root/mnt/files nullfs rw 0 0			 |
+-------+--------------------------------------------------------------------------------------+
| 3	 | /mnt/orb1/db /mnt/iocage/jails/nextcloud/root/var/db/mysql nullfs rw 0 0			 |
+-------+--------------------------------------------------------------------------------------+

Just to be safe rather than ssh'ing right into the jail I did the iocage exec nextcloud from shell. Here's the output:

Code:

[Wed Aug  1 22:20:59 EDT 2018] Renew: 'myawesomedomain.com'						
[Wed Aug  1 22:21:00 EDT 2018] Standalone mode.								
netstat: kvm not available: /dev/mem: No such file or directory				
[Wed Aug  1 22:21:00 EDT 2018] Single domain='myawesomedomain.com'				  
[Wed Aug  1 22:21:00 EDT 2018] Getting domain auth token for each domain		
[Wed Aug  1 22:21:00 EDT 2018] Getting webroot for domain='myawesomedomain.com'	
[Wed Aug  1 22:21:00 EDT 2018] Getting new-authz for domain='myawesomedomain.com'	
[Wed Aug  1 22:21:02 EDT 2018] The new-authz request is ok.					
[Wed Aug  1 22:21:02 EDT 2018] Verifying:myawesomedomain.com						
[Wed Aug  1 22:21:02 EDT 2018] Standalone mode server						  
2018/08/01 22:21:02 socat[51837] E bind(5, {LEN=0 AF=2 0.0.0.0:80}, 16): Address
 already in use																
[Wed Aug  1 22:21:06 EDT 2018] myawesomedomain.com:Verify error:Invalid response from
 http://myawesomedomain.com/.well-known/acme-challenge/upuUYy5IpI4DlglS8YmAb8ed2X7Aid
6b7RDReNohqbE:																
/root/.acme.sh/acme.sh: line 2087: kill: (51837) - No such process			
[Wed Aug  1 22:21:06 EDT 2018] Please add '--debug' or '--log' to check more det
ails.																		  
[Wed Aug  1 22:21:06 EDT 2018] See: https://github.com/Neilpang/acme.sh/wiki/How
-to-debug-acme.sh															  
[Wed Aug  1 22:21:07 EDT 2018] Error renew myawesomedomain.com.					
[Wed Aug  1 22:21:07 EDT 2018] ===End cron===			

Thanks for helping with this!

 

[danb35]

root@freenas:~ # acme.sh --cron

No, in the jail, not in FreeNAS itself.

[Wed Aug 1 22:21:00 EDT 2018] Standalone mode.

OK, there's your problem--for some reason it didn't update to use webroot instead of standalone. Try this, inside the jail:

Code:

acme.sh --issue -d ${HOST_NAME} -w /usr/local/www/apache24/data -k 4096 --fullchain-file /usr/local/etc/pki/tls/certs/fullchain.pem --key-file /usr/local/etc/pki/tls/private/privkey.pem --reloadcmd "service apache24 reload"

 

[cunningorb]

No, in the jail, not in FreeNAS itself.

OK, there's your problem--for some reason it didn't update to use webroot instead of standalone. Try this, inside the jail:

Code:

acme.sh --issue -d ${HOST_NAME} -w /usr/local/www/apache24/data -k 4096 --fullchain-file /usr/local/etc/pki/tls/certs/fullchain.pem --key-file /usr/local/etc/pki/tls/private/privkey.pem --reloadcmd "service apache24 reload"

As soon as I get back from work I am trying this for sure! Someone needs to get danb35 a medal, everyone I've seen who asks a question gets a quick answer. This is a community but it is in many ways better and faster than support you would pay for elsewhere. Amazing!

 

[cunningorb]

No, in the jail, not in FreeNAS itself.

OK, there's your problem--for some reason it didn't update to use webroot instead of standalone. Try this, inside the jail:

Code:

acme.sh --issue -d ${HOST_NAME} -w /usr/local/www/apache24/data -k 4096 --fullchain-file /usr/local/etc/pki/tls/certs/fullchain.pem --key-file /usr/local/etc/pki/tls/private/privkey.pem --reloadcmd "service apache24 reload"

I'm actually getting this:
acme.sh: Command not found.

 

[danb35]

acme.sh: Command not found.

And you ran that inside the jail? Not at the FreeNAS prompt, but at the jail's prompt?

 

[cunningorb]

And you ran that inside the jail? Not at the FreeNAS prompt, but at the jail's prompt?

I've been using putty to SSH directly into the IP of the jail, hopefully that's an acceptable way to run these commands. I prefer it over using freenas gui shell or even using SSH to remote into the freenas IP, then running command iocage exec nextcloud
I can always switch and use a different method though!
Also, I assume {HOST_NAME} gets replaced with myawesomedomain.com or www.myawesomedomain.com unless that was meant to be the host name of the machine. I ran the command first exactly as you typed it, and got an error something about host name not being defined. So then I ran it using no curly brackets and myawesomedomain.com and that's when I got the command not found error.

 

[danb35]

Also, I assume {HOST_NAME} gets replaced with myawesomedomain.com or www.myawesomedomain.com

${HOST_NAME} (including the dollar sign) gets replaced with whatever domain name you're giving the jail, so it would look like -d www.myawesomedomain.com.

Connecting directly to the jail via SSH is fine, but it would mean you've done some additional configuration beyond what's in my script. But just for the sake of eliminating variables, can you try connecting to FreeNAS via SSH, then iocage console nextcloud, then try the command again?

 

[cunningorb]

${HOST_NAME} (including the dollar sign) gets replaced with whatever domain name you're giving the jail, so it would look like -d www.myawesomedomain.com.

Connecting directly to the jail via SSH is fine, but it would mean you've done some additional configuration beyond what's in my script. But just for the sake of eliminating variables, can you try connecting to FreeNAS via SSH, then iocage console nextcloud, then try the command again?

Sure thing, here's the output from that:

Code:

root@nextcloud:/ # acme.sh --issue -d myawesomedomain.com -w /usr/local/www/apache24/data -k 4096 --fullchain-file /usr/local/etc/pki/tls/certs/fullchain.pem --key-file /usr/local/etc/pki/tls/private/privkey.pem --reloadcmd "service apache24 reload"
[Fri Aug  3 22:11:49 EDT 2018] Single domain='myawesomedomain.com'
[Fri Aug  3 22:11:49 EDT 2018] Getting domain auth token for each domain
[Fri Aug  3 22:11:49 EDT 2018] Getting webroot for domain='myawesomedomain.com'
[Fri Aug  3 22:11:49 EDT 2018] Getting new-authz for domain='myawesomedomain.com'
[Fri Aug  3 22:11:50 EDT 2018] The new-authz request is ok.
[Fri Aug  3 22:11:50 EDT 2018] Verifying:myawesomedomain.com
[Fri Aug  3 22:11:53 EDT 2018] myawesomedomain.com:Verify error:Invalid response from http://myawesomedomain.com/.well-known/acme-challenge/D2qI2kWQdyrCUu567n-HCtt3QmEHha6gm5mi2iCa2EU:
[Fri Aug  3 22:11:53 EDT 2018] Please add '--debug' or '--log' to check more details.
[Fri Aug  3 22:11:53 EDT 2018] See: https://github.com/Neilpang/acme.sh/wiki/How-to-debug-acme.sh

 

[danb35]

Can you post the contents of /usr/local/etc/apache24/Includes/${HOST_NAME}.conf? From inside the jail, that is.

 

[cunningorb]

Can you post the contents of /usr/local/etc/apache24/Includes/${HOST_NAME}.conf? From inside the jail, that is.

Here you go

Code:

<VirtualHost 192.168.1.2:80>
  DocumentRoot "/usr/local/www/apache24/data/nextcloud"
  ServerName myawesomedomain.com
  Redirect / https://myawesomedomain.com/
</VirtualHost>

<VirtualHost 192.168.1.2:443>
  ServerAdmin admin@example.com
  ServerName myawesomedomain.com
  DocumentRoot "/usr/local/www/apache24/data/nextcloud"
  RewriteEngine On
  RewriteCond %{HTTP:Authorization} ^(.*)
  RewriteRule .* - [e=HTTP_AUTHORIZATION:%1]
  #ProxyPassMatch ^/(.*\.php(/.*)?)$ fcgi://127.0.0.1:9000/usr/local/www/apache24/data/nextcloud/$1
  <FilesMatch \.php$>
		 SetHandler "proxy:fcgi://localhost:9000"
  </FilesMatch>
  DirectoryIndex /index.php index.php
  SSLCertificateFile /usr/local/etc/pki/tls/certs/fullchain.pem
  SSLCertificateKeyFile /usr/local/etc/pki/tls/private/privkey.pem
  SSLEngine on
  # Modern configuration, tweak to your needs
  SSLProtocol			 all -SSLv3 -TLSv1 -TLSv1.1
  SSLCipherSuite		  ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
  SSLHonorCipherOrder  on
  SSLCompression  off
#  SSLUseStapling on
  SSLSessionTickets	   off
#  SSLOpenSSLConfCmd DHParameters "/usr/local/etc/pki/tls/private/dhparams_4096.pem"
  SSLOptions +StrictRequire
  <Directory /usr/local/www/apache24/data/nextcloud>
  AllowOverride all
  </Directory>
  <IfModule mod_headers.c>
#  Header always set Strict-Transport-Security "max-age=15552000; includeSubDomains"
  </IfModule>
</VirtualHost>

 

[danb35]

Here you go

OK, I've updated that config file with the script since your installation. It'd probably be best if you updated it to look like the one here, specifically by adding these lines:

Code:

  Alias /.well-known/acme-challenge/ "/usr/local/www/apache24/data/.well-known/acme-challenge/"
  <Directory "/usr/local/www/apache24/data/.well-known">			 
	AllowOverride None
	Options MultiViews Indexes SymLinksIfOwnerMatch IncludesNoExec
	Require method GET POST OPTIONS
  </Directory>	

...right before the "DirectoryIndex /index.php index.php" line. Alternatively, you could change the acme.sh command to use -w /usr/local/www/apache24/data/nextcloud.

 

[cunningorb]

OK, I've updated that config file with the script since your installation. It'd probably be best if you updated it to look like the one here, specifically by adding these lines:

Code:

  Alias /.well-known/acme-challenge/ "/usr/local/www/apache24/data/.well-known/acme-challenge/"
  <Directory "/usr/local/www/apache24/data/.well-known">			 
		AllowOverride None	
		
		Options MultiViews Indexes SymLinksIfOwnerMatch IncludesNoExec	
		
		Require method GET POST OPTIONS	
		
  </Directory>	
		

...right before the "DirectoryIndex /index.php index.php" line. Alternatively, you could change the acme.sh command to use -w /usr/local/www/apache24/data/nextcloud.

I changed the myawesomedomain.com.conf file to this:

Code:

<VirtualHost 192.168.1.2:80>
  DocumentRoot "/usr/local/www/apache24/data/nextcloud"
  ServerName myawesomedomain.com
  Redirect / https://myawesomedomain.com/
</VirtualHost>

<VirtualHost 192.168.1.2:443>
  ServerAdmin admin@example.com
  ServerName myawesomedomain.com
  DocumentRoot "/usr/local/www/apache24/data/nextcloud"
  RewriteEngine On
  RewriteCond %{HTTP:Authorization} ^(.*)
  RewriteRule .* - [e=HTTP_AUTHORIZATION:%1]
  #ProxyPassMatch ^/(.*\.php(/.*)?)$ fcgi://127.0.0.1:9000/usr/local/www/apache24/data/nextcloud/$1
  <FilesMatch \.php$>
		 SetHandler "proxy:fcgi://localhost:9000"
  </FilesMatch>
   Alias /.well-known/acme-challenge/ "/usr/local/www/apache24/data/.well-known/acme-challenge/"
  <Directory "/usr/local/www/apache24/data/.well-known">			 
		AllowOverride None	 
		
		Options MultiViews Indexes SymLinksIfOwnerMatch IncludesNoExec	 
		
		Require method GET POST OPTIONS	 
		
  </Directory>	 
  DirectoryIndex /index.php index.php
  SSLCertificateFile /usr/local/etc/pki/tls/certs/fullchain.pem
  SSLCertificateKeyFile /usr/local/etc/pki/tls/private/privkey.pem
  SSLEngine on
  # Modern configuration, tweak to your needs
  SSLProtocol			 all -SSLv3 -TLSv1 -TLSv1.1
  SSLCipherSuite		  ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
  SSLHonorCipherOrder  on
  SSLCompression  off
#  SSLUseStapling on
  SSLSessionTickets	   off
#  SSLOpenSSLConfCmd DHParameters "/usr/local/etc/pki/tls/private/dhparams_4096.pem"
  SSLOptions +StrictRequire
  <Directory /usr/local/www/apache24/data/nextcloud>
  AllowOverride all
  </Directory>
  <IfModule mod_headers.c>
#  Header always set Strict-Transport-Security "max-age=15552000; includeSubDomains"
  </IfModule>
</VirtualHost>

And re-running in the jail the acme.sh command from above resulted in this:

Code:

root@nextcloud:/ # acme.sh --issue -d myawesomedomain.com -w /usr/local/www/apache24/data -k 4096 --fullchain-file /usr/local/etc/pki/tls/certs/fullchain.pem --key-file /usr/local/etc/pki/tls/private/privkey.pem --reloadcmd "service apache24 reload"
[Sat Aug  4 08:37:55 EDT 2018] Single domain='myawesomedomain.com'
[Sat Aug  4 08:37:56 EDT 2018] Getting domain auth token for each domain
[Sat Aug  4 08:37:56 EDT 2018] Getting webroot for domain='myawesomedomain.com'
[Sat Aug  4 08:37:56 EDT 2018] Getting new-authz for domain='myawesomedomain.com'
[Sat Aug  4 08:37:57 EDT 2018] The new-authz request is ok.
[Sat Aug  4 08:37:57 EDT 2018] Verifying:myawesomedomain.com
[Sat Aug  4 08:38:00 EDT 2018] myawesomedomain.com:Verify error:Invalid response from http://myawesomedomain.com/.well-known/acme-challenge/dtLK740zU_H9PWv7bk7nnqI734g7zu8uYlgMTULLaHw:
[Sat Aug  4 08:38:00 EDT 2018] Please add '--debug' or '--log' to check more details.
[Sat Aug  4 08:38:00 EDT 2018] See: https://github.com/Neilpang/acme.sh/wiki/How-to-debug-acme.sh

 

[danb35]

I changed the myawesomedomain.com.conf file to this:

Did you reload the apache configuration afterward (which I should have mentioned)? service apache24 reload.

Then, before you burn through a rate limit for failed authorizations, try a test.
nano
/usr/local/www/apache24/data/.well-known/acme-challenge/test, and put some text in there (doesn't really matter what). Then, browse to http://myawesomedomain.com/.well-known/acme-challenge/test. Do you see that text file?

 

[cunningorb]

Did you reload the apache configuration afterward (which I should have mentioned)? service apache24 reload.

Then, before you burn through a rate limit for failed authorizations, try a test.
nano
/usr/local/www/apache24/data/.well-known/acme-challenge/test, and put some text in there (doesn't really matter what). Then, browse to http://myawesomedomain.com/.well-known/acme-challenge/test. Do you see that text file?

Shoot, I had thought about reloading apache and just didn't do it. Did that now and when I go to the hyperlink you noted I am getting redirected to https and getting the invalid cert page in chromium. So the page itself is there but I can't view it on port 80.

Code:

Not Found
The requested URL /.well-known/acme-challenge/test was not found on this server.

About to head out for a few hours but thanks for keeping in touch with me on this!

 

[danb35]

I am getting redirected to https and getting the invalid cert page in chromium.

Both of those are expected, and wouldn't be a problem. But after accepting the invalid cert you're getting a 404, and that isn't expected. This may take some digging in your logs to track down--anything of note in (in the jail) /var/log/httpd-{access|error}.log?

 

[cunningorb]

Both of those are expected, and wouldn't be a problem. But after accepting the invalid cert you're getting a 404, and that isn't expected. This may take some digging in your logs to track down--anything of note in (in the jail) /var/log/httpd-{access|error}.log?

This is all I have of note from the error log:

Code:

[Sat Aug 04 13:17:00.128078 2018] [mpm_event:notice] [pid 3111:tid 34397577216] AH00493: SIGUSR1 received.  Doing graceful restart
[Sat Aug 04 13:17:00.154542 2018] [ssl:warn] [pid 3111:tid 34397577216] AH01873: Init: Session Cache is not configured [hint: SSLSessionCache]
[Sat Aug 04 13:17:00.156053 2018] [mpm_event:notice] [pid 3111:tid 34397577216] AH00489: Apache/2.4.33 (FreeBSD) OpenSSL/1.0.2k-freebsd configured -- resuming normal operations
[Sat Aug 04 13:17:00.156077 2018] [core:notice] [pid 3111:tid 34397577216] AH00094: Command line: '/usr/local/sbin/httpd -D NOHTTPACCEPT'
[Sat Aug 04 15:17:12.005132 2018] [proxy_fcgi:error] [pid 73409:tid 34469060096] (70007)The timeout specified has expired: [client 23.112.128.141:61118] AH01075: Error dispatching request to : (reading input brigade)

fcgi error?

 

[danb35]

This is what happens when I assume things rather than testing them... I need to work some more on the virtual host configuration, stand by...

 

[cunningorb]

This is what happens when I assume things rather than testing them... I need to work some more on the virtual host configuration, stand by...

Hey @danb35 after a full reboot of freenas the ssl connection is now working again. I didn't check ssl labs but it's good enough for the moment. I did not re-run acme.sh so it must have just need a recycle after one of the times it was run.