Tried Minio directly, but since they don't maintain the BSD package, I'm trying here. I'm got a number of TrueNAS Minio installs running for me on various systems without problems except for this one. Any insight would be appreciated.
<!--- Provide a general summary of the issue in the Title above -->
Using Minio as an S3 offload for Veeam 10. The setup was running happily for about 36 hours into the initial offload when it started crashing.
## Expected Behavior
Not crashing?
## Current Behavior
Runs for anywhere between 1 and 5 minutes before crashing. Restarting manually does bring it back online for a little while before crashing again. One oddity that I noted is that is has created an additional directory in the same place as the others with a space at the end (I have data2 data2 data3 and data4) and on reboot, something creates "/mnt/data1 ".
During the start process I'm getting a number of TLS errors
I'm a little unsure of which evens follow each other but I think this is after one of the crashes: (text OCR from screen shot so there are some errors in the text)
```
Jan 8 09:14:21 Veeam-53 minio[6117]: github.com/minio/minio/cmd.(*xlStorageDiskIDCheck).CreateFile(0xc0014530e0, Ox2230, Oc00042d,x, Oxe, Oxc0c668e640, 0x50, 0x1916e, 0x2203be0, 0xc036fc5950, ...1
Jan 8 09:14:21 Veeam-$3 minio[6117]: /wrkdirs/usr/ports/www/minio/work/minio-RELEASE.2020-12-29T23-29-292/cmd/x1-storage-disk-id-check.go:181 +0xdb
Jan 8 09:14:21 Veeam-S3 minio[6117]: github.com/minio/minio/cmd.newStreamingBitrotWriter.func1(0x1914e, 0x500000, Ox2240780, Oxc0cb60340, Ox226460, 0xc001453 Oe0, Oxlef6e34, Oxe, Oxc0cb68e640, 0x50, ...)
Jan 8 09:14:21 Veeam-S3 minio[6117]: /wrkdirs/usr/ports/www/minio/work/minio-RELEASE.2020-12-29T23-29-29z/cmd/bitrot-streaming.go:84 +0x104
Jan 8 09:14:21 Veeam-S3 minio[6117]: created by github.com/minio/minio/cmd.newstreamingBitrotWriter
Jan 8 09:14:21 Veeam-$3 minio[6117]: /wrkdirs/usr/ports/www/minio/work/minio-RELEASE.2020-12-29T23-29-29z/cmd/bitrot-streaming.go:78 +0xZb3
Jan 8 09:14:21 Veeam-$3 minio[6117]:
Jan 8 09:14:21 Veeam-$3 minio[6117]: rax 0x0
Jan 8 09:14:21 Veeam-53 minio[6117]: rbx 0x 6
Jan 8 09:14:21 Veeam-S3 minio[6117]: rex 0x80306bc4a
Jan 8 09:14:21 Veeam-S3 minio[6117]: rdx 0x0
Jan 8 09:14:21 Veeam-S3 minio[6117]: rdi 0x18889
Jan 8 09:14:21 Veeam-53 minio[6117]: rsi Ox6
Jan 8 09:14:21 Veeam-$3 minio[6117]: rbp 0x7fffdfffda90
Jan 8 09:14:21 Veeam-S3 minio[6117]: rsp 0x7fffdfffda68
Jan 8 09:14:21 Veeam-53 minio[6117]: r8 0x0
Jan 8 09:14:21 Veeam-S3 minio[6117]: r9 0x0
Jan 8 09:14:21 Veeam-S3 minio[6117]: r10 0x802ed5680
Jan 8 09:14:21 Veeam-S3 minio[6117]: r11 0x202
Jan 8 09:14:21 Veeam-53 minio[6117]: r12 0xf1
Jan 8 09:14:21 Veeam-53 minio[6117]: r13 0x0
Jan 8 09:14:21 Veeam-S3 minio[6117]: r14 0x8032980a0
Jan 8 09:14:21 Veeam-$3 minio[6117]: r15 0x0
Jan 8 09:14:21 Veeam-$3 minio[6117]: rip 0x80306bc2a
Jan 8 09:14:21 Veeam-53 minio[6117]: rflags 0x206
Jan 8 09:14:21 Veeam-S3 minio[6117]: cs 0x43
Jan 8 09:14:21 Veeam-S3 minio[6117]: fs 0x13
Jan 8 09:14:21 Veear-$3 minio[6117]: gs 0xlb
Jan 8 09:16:09 Veeam-53 minio[6159]: 2021-01-08 09:16:09.394824 I http: TLS handshake error from 10.10.255.5:63651: EOF
Jan 8 09:16:10 Veeam-S3 minio[6159]: Status:
4 Online, 0 Offline.
Jan 8 09:16:11 Veeam-53 minio[6159]: 2021-01-08 09:16:11.946341 I
http: TLS handshake error from 10.10.255.5:63717: EOF
```
The other log entries before a crash are:
```
Jan 8 09:17:59 Veeam-S3 minio[6159]: /wrkdirs/usr/ports/Www/minio/work/minio-RELEASE.2020-12-29T23-29-29z/cmd/x1-storage.go:1805 +0x378
Jan 8 09:17:59 Veeam-S3 minio[6159]: github.com/minio/minio/cmd.(*xlStorageDiskIDCheck) .CheckFile(0xc0010fa980, Ox223b040, Oxc0640bd0, Ox05514 0xOxc 05cc51a4b, 0x63, 0x442076, 0x1f865001
Jan 8 09:17:59 Veeam-$3 minio[6159]: /wrkdirs/usr/ports/www/minio/work/minio-RELEASE.2020-12-29T23-29-292/cmd/x1-storage-disk-id-check.go:221 +0xab
Jan 8 09:17:59 Veeam-53 minio[6159]: github.com/minio/minio/cmd.erasureObjects.isObject.func1(0xc0501525b0, Ox1f86818)
Jan 8 09:17:59 Teeam-53 minio[6159]: /wrkdirs/usr/ports/www/minio/work/minio-RELEASE.2020-12-29T23-29-292/cmd/erasure-common.go:239 +0x8d
Jan 8 09:17:59 Teeam-S3 minio[6159]: github.com/minio/minio/pkg/sync/errgroup.(*Groupl.Go.func1(0xc064691530, Oxc0547b5e00, 0x21
Jan 8 09:17:59 Veeam-$3 minio[6159]: /wrkdirs/usr/ports/www/minio/work/minio-RELEASE.2020-12-29T23-29-29z/pkg/sync/errgroup/errgroup.go:55 +0x55
Jan 8 09:17:59 Veeam-$3 minio[6159]: created by github.com/minio/minio/pkg/sync/errgroup. (*Group). Go
Jan 8 09:17:59 Veeam-S3 minio[6159]: /wrkdirs/usr/ports/Www/minio/work/minio-RELEASE.2020-12-29T23-29-29z/pkg/sync/errgroup/errgroup.go:52 +0x6c
```
Then some transactions will go through
## Possible Solution
No idea
## Steps to Reproduce (for bugs)
<!--- Provide a link to a live example, or an unambiguous set of steps to -->
<!--- reproduce this bug. Include code to reproduce, if relevant -->
<!--- and make sure you have followed https://github.com/minio/minio/tree/release/docs/debugging to capture relevant logs -->
1.
2.
3.
4.
## Context
This is a setup that is trying to run in production as means to a have secured copy of onsite backups combining Minio immutability with TrueNAS 2FA. Similar configurations have been installed without issues.
## Regression
<!-- Is this issue a regression? (Yes / No) --> No
## Your Environment
* Version used (`minio --version`): minio version RELEASE.2020-12-29T23-29-29Z
* Server setup and configuration:
I have a vSphere (6.5) hosted VM running TrueNAS 12.0 with 4 vCPU and 20Gb of RAM in which I have installed the Minio plugin (not the built-in service). Storage is an Raw Device Mapping to a NAS volume mounted over iSCSI.
The basic setup has 4 directories created on the mounted ZFS dataset in order to use the immutable option with these directories noted in the jail's /etc/rc.conf file
* Operating System and version (`uname -a`): FreeBSD 12.2§RELEASE-p2 (TrueNAS)
<!--- Provide a general summary of the issue in the Title above -->
Using Minio as an S3 offload for Veeam 10. The setup was running happily for about 36 hours into the initial offload when it started crashing.
## Expected Behavior
Not crashing?
## Current Behavior
Runs for anywhere between 1 and 5 minutes before crashing. Restarting manually does bring it back online for a little while before crashing again. One oddity that I noted is that is has created an additional directory in the same place as the others with a space at the end (I have data2 data2 data3 and data4) and on reboot, something creates "/mnt/data1 ".
During the start process I'm getting a number of TLS errors
I'm a little unsure of which evens follow each other but I think this is after one of the crashes: (text OCR from screen shot so there are some errors in the text)
```
Jan 8 09:14:21 Veeam-53 minio[6117]: github.com/minio/minio/cmd.(*xlStorageDiskIDCheck).CreateFile(0xc0014530e0, Ox2230, Oc00042d,x, Oxe, Oxc0c668e640, 0x50, 0x1916e, 0x2203be0, 0xc036fc5950, ...1
Jan 8 09:14:21 Veeam-$3 minio[6117]: /wrkdirs/usr/ports/www/minio/work/minio-RELEASE.2020-12-29T23-29-292/cmd/x1-storage-disk-id-check.go:181 +0xdb
Jan 8 09:14:21 Veeam-S3 minio[6117]: github.com/minio/minio/cmd.newStreamingBitrotWriter.func1(0x1914e, 0x500000, Ox2240780, Oxc0cb60340, Ox226460, 0xc001453 Oe0, Oxlef6e34, Oxe, Oxc0cb68e640, 0x50, ...)
Jan 8 09:14:21 Veeam-S3 minio[6117]: /wrkdirs/usr/ports/www/minio/work/minio-RELEASE.2020-12-29T23-29-29z/cmd/bitrot-streaming.go:84 +0x104
Jan 8 09:14:21 Veeam-S3 minio[6117]: created by github.com/minio/minio/cmd.newstreamingBitrotWriter
Jan 8 09:14:21 Veeam-$3 minio[6117]: /wrkdirs/usr/ports/www/minio/work/minio-RELEASE.2020-12-29T23-29-29z/cmd/bitrot-streaming.go:78 +0xZb3
Jan 8 09:14:21 Veeam-$3 minio[6117]:
Jan 8 09:14:21 Veeam-$3 minio[6117]: rax 0x0
Jan 8 09:14:21 Veeam-53 minio[6117]: rbx 0x 6
Jan 8 09:14:21 Veeam-S3 minio[6117]: rex 0x80306bc4a
Jan 8 09:14:21 Veeam-S3 minio[6117]: rdx 0x0
Jan 8 09:14:21 Veeam-S3 minio[6117]: rdi 0x18889
Jan 8 09:14:21 Veeam-53 minio[6117]: rsi Ox6
Jan 8 09:14:21 Veeam-$3 minio[6117]: rbp 0x7fffdfffda90
Jan 8 09:14:21 Veeam-S3 minio[6117]: rsp 0x7fffdfffda68
Jan 8 09:14:21 Veeam-53 minio[6117]: r8 0x0
Jan 8 09:14:21 Veeam-S3 minio[6117]: r9 0x0
Jan 8 09:14:21 Veeam-S3 minio[6117]: r10 0x802ed5680
Jan 8 09:14:21 Veeam-S3 minio[6117]: r11 0x202
Jan 8 09:14:21 Veeam-53 minio[6117]: r12 0xf1
Jan 8 09:14:21 Veeam-53 minio[6117]: r13 0x0
Jan 8 09:14:21 Veeam-S3 minio[6117]: r14 0x8032980a0
Jan 8 09:14:21 Veeam-$3 minio[6117]: r15 0x0
Jan 8 09:14:21 Veeam-$3 minio[6117]: rip 0x80306bc2a
Jan 8 09:14:21 Veeam-53 minio[6117]: rflags 0x206
Jan 8 09:14:21 Veeam-S3 minio[6117]: cs 0x43
Jan 8 09:14:21 Veeam-S3 minio[6117]: fs 0x13
Jan 8 09:14:21 Veear-$3 minio[6117]: gs 0xlb
Jan 8 09:16:09 Veeam-53 minio[6159]: 2021-01-08 09:16:09.394824 I http: TLS handshake error from 10.10.255.5:63651: EOF
Jan 8 09:16:10 Veeam-S3 minio[6159]: Status:
4 Online, 0 Offline.
Jan 8 09:16:11 Veeam-53 minio[6159]: 2021-01-08 09:16:11.946341 I
http: TLS handshake error from 10.10.255.5:63717: EOF
```
The other log entries before a crash are:
```
Jan 8 09:17:59 Veeam-S3 minio[6159]: /wrkdirs/usr/ports/Www/minio/work/minio-RELEASE.2020-12-29T23-29-29z/cmd/x1-storage.go:1805 +0x378
Jan 8 09:17:59 Veeam-S3 minio[6159]: github.com/minio/minio/cmd.(*xlStorageDiskIDCheck) .CheckFile(0xc0010fa980, Ox223b040, Oxc0640bd0, Ox05514 0xOxc 05cc51a4b, 0x63, 0x442076, 0x1f865001
Jan 8 09:17:59 Veeam-$3 minio[6159]: /wrkdirs/usr/ports/www/minio/work/minio-RELEASE.2020-12-29T23-29-292/cmd/x1-storage-disk-id-check.go:221 +0xab
Jan 8 09:17:59 Veeam-53 minio[6159]: github.com/minio/minio/cmd.erasureObjects.isObject.func1(0xc0501525b0, Ox1f86818)
Jan 8 09:17:59 Teeam-53 minio[6159]: /wrkdirs/usr/ports/www/minio/work/minio-RELEASE.2020-12-29T23-29-292/cmd/erasure-common.go:239 +0x8d
Jan 8 09:17:59 Teeam-S3 minio[6159]: github.com/minio/minio/pkg/sync/errgroup.(*Groupl.Go.func1(0xc064691530, Oxc0547b5e00, 0x21
Jan 8 09:17:59 Veeam-$3 minio[6159]: /wrkdirs/usr/ports/www/minio/work/minio-RELEASE.2020-12-29T23-29-29z/pkg/sync/errgroup/errgroup.go:55 +0x55
Jan 8 09:17:59 Veeam-$3 minio[6159]: created by github.com/minio/minio/pkg/sync/errgroup. (*Group). Go
Jan 8 09:17:59 Veeam-S3 minio[6159]: /wrkdirs/usr/ports/Www/minio/work/minio-RELEASE.2020-12-29T23-29-29z/pkg/sync/errgroup/errgroup.go:52 +0x6c
```
Then some transactions will go through
## Possible Solution
No idea
## Steps to Reproduce (for bugs)
<!--- Provide a link to a live example, or an unambiguous set of steps to -->
<!--- reproduce this bug. Include code to reproduce, if relevant -->
<!--- and make sure you have followed https://github.com/minio/minio/tree/release/docs/debugging to capture relevant logs -->
1.
2.
3.
4.
## Context
This is a setup that is trying to run in production as means to a have secured copy of onsite backups combining Minio immutability with TrueNAS 2FA. Similar configurations have been installed without issues.
## Regression
<!-- Is this issue a regression? (Yes / No) --> No
## Your Environment
* Version used (`minio --version`): minio version RELEASE.2020-12-29T23-29-29Z
* Server setup and configuration:
I have a vSphere (6.5) hosted VM running TrueNAS 12.0 with 4 vCPU and 20Gb of RAM in which I have installed the Minio plugin (not the built-in service). Storage is an Raw Device Mapping to a NAS volume mounted over iSCSI.
The basic setup has 4 directories created on the mounted ZFS dataset in order to use the immutable option with these directories noted in the jail's /etc/rc.conf file
* Operating System and version (`uname -a`): FreeBSD 12.2§RELEASE-p2 (TrueNAS)
