Hello, I have been using TrueNAS core for a while as a networked storage system. Primary use cases are to provide persistent volumes to my kubernetes cluster(Using democratic-csi), providing network shares to my home devices and using s3 object storage using minio.
S3 and NFS shares work without a problem and perform exactly as expected. For iscsi I created a share using the default procedure (pretty much all options were kept as default). Then I connected to the target, mounted the volume and formatted it to ext4 which again worked fine (https://manjaro.site/how-to-connect-to-iscsi-volume-from-ubuntu-20-04/). Then when I log out of the target and log back in I get immidiate filesystem errors. These are so bad that I cant actually fix them sometimes, when I run fsck it detects and fixes so many problems that it gets oom killed.
This doesnt happen every time this way. For example just now I tried to mount an iscsi share on my computer and it worked. However the kernel has already detected some problems:
I also tried to use iscsi with democratic-csi in kubernetes, which automatically sets up and provisions iscsi shares. Any pod that tried to use it as soon as it restarted would crash complaining of excessive filesystem errors. Again using NFS here works without a problem.
Am I missing something regarding the setup of these shares? When I check the zvol status there are no errors or issues as far as I can tell and scrubbing doesnt return any errors. Since NFS works without issues I would think that the problem isnt with the hardware and there is something going on with the setup of iscsi but I cant tell for the life of me what it is.
Any help/advice would be greatly appreciated, thank you.
S3 and NFS shares work without a problem and perform exactly as expected. For iscsi I created a share using the default procedure (pretty much all options were kept as default). Then I connected to the target, mounted the volume and formatted it to ext4 which again worked fine (https://manjaro.site/how-to-connect-to-iscsi-volume-from-ubuntu-20-04/). Then when I log out of the target and log back in I get immidiate filesystem errors. These are so bad that I cant actually fix them sometimes, when I run fsck it detects and fixes so many problems that it gets oom killed.
This doesnt happen every time this way. For example just now I tried to mount an iscsi share on my computer and it worked. However the kernel has already detected some problems:
Code:
[Sun Mar 3 12:22:03 2024] scsi host12: iSCSI Initiator over TCP/IP [Sun Mar 3 12:22:03 2024] scsi 12:0:0:0: Direct-Access TrueNAS iSCSI Disk 0123 PQ: 0 ANSI: 7 [Sun Mar 3 12:22:03 2024] scsi 12:0:0:0: Power-on or device reset occurred [Sun Mar 3 12:22:03 2024] sd 12:0:0:0: Attached scsi generic sg3 type 0 [Sun Mar 3 12:22:03 2024] sd 12:0:0:0: [sde] 13107204 4096-byte logical blocks: (53.7 GB/50.0 GiB) [Sun Mar 3 12:22:03 2024] sd 12:0:0:0: [sde] 16384-byte physical blocks [Sun Mar 3 12:22:03 2024] sd 12:0:0:0: [sde] Write Protect is off [Sun Mar 3 12:22:03 2024] sd 12:0:0:0: [sde] Mode Sense: 7f 00 10 08 [Sun Mar 3 12:22:04 2024] sd 12:0:0:0: [sde] Write cache: enabled, read cache: enabled, supports DPO and FUA [Sun Mar 3 12:22:04 2024] sd 12:0:0:0: [sde] Optimal transfer size 8388608 bytes [Sun Mar 3 12:22:04 2024] sde: sde1 [Sun Mar 3 12:22:04 2024] sd 12:0:0:0: [sde] Attached SCSI disk [Sun Mar 3 12:22:05 2024] /dev/sdd1: Can't lookup blockdev [Sun Mar 3 12:22:17 2024] EXT4-fs (sde1): warning: mounting fs with errors, running e2fsck is recommended [Sun Mar 3 12:22:17 2024] EXT4-fs (sde1): recovery complete [Sun Mar 3 12:22:17 2024] EXT4-fs (sde1): mounted filesystem f1295757-6adb-47f2-b38c-b6565cfc375e r/w with ordered data mode. Quota mode: none.
I also tried to use iscsi with democratic-csi in kubernetes, which automatically sets up and provisions iscsi shares. Any pod that tried to use it as soon as it restarted would crash complaining of excessive filesystem errors. Again using NFS here works without a problem.
Am I missing something regarding the setup of these shares? When I check the zvol status there are no errors or issues as far as I can tell and scrubbing doesnt return any errors. Since NFS works without issues I would think that the problem isnt with the hardware and there is something going on with the setup of iscsi but I cant tell for the life of me what it is.
Any help/advice would be greatly appreciated, thank you.
