Clustered iSCSI Performance

[NickF]

Hi all,

This is probably untreaded water, but I am just wondering if any other folks out there have been playing with this. I have 3 mini computers (6c/12t, 16GB RAM, 2x512GB ssd mirrored) and I am playing around with iSCSI inside of TrueCommand.

Performance consistency is odd.
This is setup with 3 nodes in TC:

This is a single node setup in TN SCALE (one of same computers from above), notice the difference in read IOPS and latency, as well as the higher write and mixed performance in general.

To be clear, I am not expecting a high level of performance with my systems here. I am just trying to figure out what relative performance should look like vs a single node. I am aware that 16GiB of RAM is not recommended for iSCSI, and I am not trying to build a production system here. Rather, I am trying to figure out how this feature works...

I also only see one node at a time actually seeing IO...which is why I am confused. If theres a performance penalty for syncing the data between 3 nodes, I get that. BUt I am only seeing IO on a single node at a time.....

Am I misunderstanding this feature?

iSCSI Volume Management

Provide information on managing iSCSI block shares and iSCSI volumes in TrueCommand.

www.truenas.com

This is supposed to sync the data between all the nodes right??

 

[Kris Moore]

No. Clustering only works for SMB shares that you create through the TrueCommand UI. iSCSI is not part of a cluster, and usually is a poor fit for clustering due to the fact that most block workloads are highly latency sensitive. Clustering adds additional latency to the mix (thanks physics!) which usually makes it less desirable for block workloads.

 

[NickF]

No. Clustering only works for SMB shares that you create through the TrueCommand UI. iSCSI is not part of a cluster, and usually is a poor fit for clustering due to the fact that most block workloads are highly latency sensitive. Clustering adds additional latency to the mix (thanks physics!) which usually makes it less desirable for block workloads.

I hear you. But I’m confused.
When I went through the wizard in TrueCommand and then subsequently went to to Windows ISCSI imitator, after putting only one nodes IP in it auto discovered the other nodes

And only one device shows up when I auto configured on the next tab. From a user perspective it seems like it’s building a cluster…See my other thread. If it’s not then the wizard in TC is fundamentally confusing by it's design. Why would it present it to me like this?

It even calls it a Block Device Group

And then when i look at the nodes themselves, they have these weird notes

Like theres some sort of GUID that TC is tracking and its the same on each node.

Thread 'Missing Features for ISCSi, Plus Data Loss Possible...'
https://www.truenas.com/community/threads/missing-features-for-iscsi-plus-data-loss-possible.107842/

 

[Kris Moore]

Ok, I think I see why that is confusing. The TC iSCSI wizard is primarily aimed at helping to make a bunch of devices at once. But that is != to clustering. I'll point some of the devs at this thread to review and see if we can help make this clearer.

 

[NickF]

Ok, I think I see why that is confusing. The TC iSCSI wizard is primarily aimed at helping to make a bunch of devices at once. But that is != to clustering. I'll point some of the devs at this thread to review and see if we can help make this clearer.

To your original point that’s why I was playing around with this, I was trying to figure out what the expected performance penalty was supposed to be for a feature that doesn’t exist how I had understood it. Lol

What’s stranger is that I actually did see a difference in performance that now I cannot explain why. More tinkering to do

 

[HoneyBadger]

Interesting that it picked up portals on the other nodes through dynamic discovery. Do you get different behavior if you connect/discover targets through a VMware initiator vs. a Windows one?

Re: performance; do the systems all have similar BIOS power-saving configurations? RND4K I've found to be especially sensitive to that.

 

[NickF]

Well I can

Interesting that it picked up portals on the other nodes through dynamic discovery. Do you get different behavior if you connect/discover targets through a VMware initiator vs. a Windows one?

Re: performance; do the systems all have similar BIOS power-saving configurations? RND4K I've found to be especially sensitive to that.

My VMWare host has been decomed, I can unbury it and try when I have some time.

In any case, the WWNs of the LUNs are different (they are supposed to be if its not a cluster) I also have COVID with a high fever so I could be losing my mind.

 

[HoneyBadger]

FWIW, I didn't see the same auto-discovery behavior on VMware, and I have different serial/NAA IDs for the LUNs as well.