Is the built-in S3 (MinIO) service currently broken on RC.2 ?

[fsociety3765]

Hi all,

I have been using the built-in S3 service in TrueNAS for a while. Previously on Core and also since migrating to SCALE. It was definitely working before. I haven't used it in a little while though. I needed to use it today but there seem to have been some changes. The port is set to 9000 as it always has been. Yet when I browse to it, it redirects to some random port number that seems to also change each time you access it. The UI is completely different too. It comes up as MinIO Console. The login doesn't work with the self-signed "freenas_default" cert anymore. The only way I can actually log in is to disable SSL and use HTTP.

I usually access the MinIO web interface via HAProxy on pfSense which does the SSL termination with Let's Encrypt. But this is no longer working by pointing it at port 9000 / unencrypted. I'm guessing it has something to do with the random port redirect that happening.

Anyone else seen this and know what's going on?

Thanks,

FS

 

[morganL]

Was it working with RC1.x.... before you updated?
Can you revert to check out that its a software issue and not a configuration change?

 

[fsociety3765]

Hi,

I can't be sure in all honestly. I know it has been working on SCALE since I migrated as I was using it previously.

Happy to give reverting a shot to see if anything changes. I've not had to revert before. I assume it can be done from the "Boot" menu? Looks like there are versions there going back to BETA.1. Are there any major risks in reverting that I should be aware of?

 

[morganL]

I haven't seen anyone report a similar issue. If we hear, then its likely to be software.
If not, something in the configuration may have been changed (or been broken).

If you have the time and willingness to test.... then the rollback is described here: https://www.truenas.com/docs/scale/systemsettings/bootenvironment/#changing-boot-environments

 

[fsociety3765]

OK. I have just reverted back to RC.1-2 and it now works again as I expect it to.

It seems like in RC.2 the version of MinIO is much newer. The UI is completely different.

RC.1-2:

Following login, the about page shows:

RC.2:

Take note of the random port number (33671). I did not enter that. I entered 9000 and it automatically changed to the above port.

Using the "freenas_default" SSL cert, I cannot log in and get this error:

After unselecting the "freenas_default" SSL cert in the S3 service settings I get the same page as above using HTTP, but notice the different port this time:

Again, I entered port 9000 and it automatically changed o that random port (36709).

Using HTTP, I can however log in now:

As you can see the version of MinIO between RC.1-2 and RC.2 has changed significantly.

Hope this helps. Surprised no one else has come up against this yet.

FS

 

[c77dk]

I actually experienced the same on CORE today (-U7) - never used minio before so can't tell if it has changed in CORE.
I am however able to use port 9000 as endpoint for Velero k8s backups without TLS - the TLS part is next on my todo.
Could this be something specific to the minio webinterface, and not affecting the service itself?

 

[fsociety3765]

I actually experienced the same on CORE today (-U7) - never used minio before so can't tell if it has changed in CORE.
I am however able to use port 9000 as endpoint for Velero k8s backups without TLS - the TLS part is next on my todo.
Could this be something specific to the minio webinterface, and not affecting the service itself?

Confirmed. The service itself is still usable on port 9000 using the MinIO CLI tool (MC).

 

[fdodd]

The error on web access requires CA certificate being created on DNS host name for your truenas, then you can assign a self signed certificate associated to the CA - the issue with mount data store out side the jail is located in this image but still can not get it to work of the bat, it did map the mount point but could not access. If any one knows the correct flags for jail_zfs - jail_zfs_mountpoint done another post over this but this tides in with MinIO. Need help on this mount point within the jail of minIO to external mount point, if you allow data to map to the jail\minIO its gone if you re-install MinIO

 

[fdodd]

It looks good but its not mapping to my data point the fix is in advance properties but can not get it to resolve, tried about 20 plus times driving me mad, help ;-)

 

[Axemann]

Hi All! I was just running into this same issue and did some poking around in the filesystem. While looking through the MinIO startup script at /conf/base/etc/local/rc.d/minio, there is a variable named 'minio_console_address' that is unset by default in the TrueNAS config. I added an rc.conf tunable of 'minio_console_address'=':9001' (see below), then restarted the S3 service and was able to access it via HTTP at truenas_ip:9001. I also did not select a certificate in the service config page, as I am reverse-proxying it and the S3 service port (9000), but either way it should work with the scheme you select.

Hopefully this helps you guys/gals/both/neither with this, as it was driving me nuts, lol.

Cheers!

 

[fdodd]

Fixed my datastore mapping to my truenas jail

 

[fsociety3765]

Does anyone know where the MinIO config is located on a SCALE system? Searching around but can't find anything.

 

[zuntaruk]

Does anyone know where the MinIO config is located on a SCALE system? Searching around but can't find anything.

There isn't a config file from the looks of it. Poked around a bit, found that they have a custom startup script for minio that starts the process based on the data you entered in the Services form for S3.

File that I found: `/usr/bin/minio-truenas`

How I found it:

Code:

systemctl status minio

# Sample output
minio.service - High Performance Object Storage
     Loaded: loaded (/lib/systemd/system/minio.service; enabled; vendor preset: disabled)
     Active: active (running) since Sun 2022-01-23 19:25:44 CST; 16min ago

# Find the loaded service file from the `Loaded` line
cat /lib/systemd/system/minio.service

# Here you'll see the ExecStart point to a file
cat /usr/bin/minio-truenas

With all the above, it doesn't look like there is currently a way to (from the UI) modify and add the parameter to run the Admin interface on a different port (9001 was used above, as an example).

 

[zuntaruk]

Looks like they've got a fix coming in an upcoming release:

https://jira.ixsystems.com/browse/NAS-114137

Here's the MR for the middleware:

NAS-114116 / 13.0 / Make minio console port configurable by Qubad786 · Pull Request #8082 · truenas/middleware

MinIO by default selects a random port for the MinIO Console on each server startup. Browser clients accessing the MinIO Server are automatically redirected to the MinIO Console on its dynamically ...

github.com