Parallel iSCSI access possible?

JoHapp

Dabbler
Joined
Jul 29, 2021
Messages
44
I am checking options for our network, and next to using SMB or NFS, iSCSI would be an option. As we have a mixed environment, i would purchase the ATTO XtendSan clients for the macs. They are compatible with latest Mac OS and M1.

I have learned from a technician, that it depends on the storage, if a parallel access on iSCSI is supported.
How is it with Truenas (Scale)?
a) Is it capable of such a service, that 2 or more iSCSI connections are accessing the same volume at the same time?
b) is it possible, that those connections read the same source at the same time?

Thanks in advance! :smile:
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
a technician, that it depends on the storage, if a parallel access on iSCSI is supported.

It doesn't depend on the storage. That's the dumbest thing I've heard today, though the day is young. Some iSCSI targets offer an option to prohibit concurrent access, but that's only as a safety belt for the issue we're about to discuss.

Is it capable of such a service, that 2 or more iSCSI connections are accessing the same volume at the same time?

Of course it is.

is it possible, that those connections read the same source at the same time?

Of course it is. People use SAN storage for hypervisors and other stuff like that ALL THE TIME.

However, there's a problem. Concurrent connections require a cluster-aware filesystem, such as VMFS. iSCSI is a block level protocol.

Good search term: "cluster aware filesystem"


So when an operating system mounts a hard disk, it generally caches useful things in memory and assumes that no one else is writing to the hard disk. You can substitute "SATA disk" or "USB disk" or "USB flash" in there if those feel more comfortable to the discussion. A typical filesystem that is NOT cluster-aware makes this assumption.

Imagine simultaneously plugging a single SATA disk or USB flash device into two PC's. This is what iSCSI effectively makes possible. Now you have a situation where two PC's may be trying to access the same data. Device one reads a file named "abc" and happens to read in a directory block and caches it. Device two wants to write "def". Reads the same directory block. Finds the slot right after "abc" is open, so writes a directory entry there, now the physical disk block contains directory entries for "abc", "def". Device one now decides to write "ghi". It has the cached block, sees "abc", sees the empty entry right after it, so writes "ghi" there. Now the physical disk block contains "abc", "ghi". And we've lost information. What happened to "def"? And the data that the second PC is writing to that file?

Cluster-aware filesystems allow multiple hosts to simultaneously mount a single disk. This is possible because the cluster-aware filesystems take extra steps to signal between nodes the writes that are being staged out, and also they tend to do much less read caching. VMFS is one of the best examples of this.

Your typical consumer grade filesystems, NTFS, Apple, etc., are optimized for speed, and try to maximize caching, and are NOT cluster-aware. They will happily trample your disk and can corrupt everything if you get multiple mounts going.

So you absolutely need to use AFP, CIFS, NFS, or other file-level sharing protocols for multi-host access in most situations. It does not matter if the backing disk is connected via SATA, USB, SAS, iSCSI, Fiber Channel, etc. -- as long as a single host has the filesystem's disk mounted, and is responsible for coordinating the integrity of the filesystem, you'll be fine. This is why most network storage works with these protocols. You only have one kernel accessing and one filesystem mount active, which means that there's no chance of inadvertent trampling.
 

JoHapp

Dabbler
Joined
Jul 29, 2021
Messages
44
Wow, thanks for that detailed information!
That explains me the need of the filesystem above the blocklevel.
 

JoHapp

Dabbler
Joined
Jul 29, 2021
Messages
44
Concurrent connections require a cluster-aware filesystem, such as VMFS.
Am I right, that it would mean purchasing VMware vSphere Standard or using the free Hypervisor?
Also this ressource is very helpful, after understanding, that a filesystem is needed:
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
Am I right, that it would mean purchasing VMware vSphere Standard or using the free Hypervisor?
Also this ressource is very helpful, after understanding, that a filesystem is needed:

Just so we're clear:

1) Yes, all versions of ESXi support VMFS.

2) However, this ONLY applies to the hypervisor. The guest VM's do not magically inherit some new ability to utilize a shared filesystem. It is the VM virtual disks that make up the VM's that are on the VMFS.

This is why NFS or SMB are normally still a big part of virtual machine networks.
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,776
You need a Windows server for that according to their spec sheet. And it's completely proprietary vendor lock-in if you go that route. What happens to your data if that company goes bankrupt and you need support?

For a mixed Windows and Mac environment SMB is your best bet, IMHO.
 

simonj

Dabbler
Joined
Feb 28, 2022
Messages
32
Yeah I know. I was just testing (trial license) it with the metadata controller running in a VM on our Scale server and serving the ISCSI shares from the same server. It’s a bit cumbersome to setup but it works.
I’m also a bit worried as it seems like outdated tech. But the underlying data is NTFS so you could access it even if they go out of business.

Background is that SMB is totally broken in TrueNAS for Mac clients. We have been looking for a workaround for some time and might go the Tiger Store Route. Performance is similar for single files and much better for file sequences, which we use a lot in video post-production. Also the annoying lockups when accessing folders with many files or deleting many files don’t happen with that setup.
 
Top