SOLVED iSCSI udev recommended values

[nicolasb827]

Hello,

I have searched for udev settings but I found nothing.
I connected a zvol over iSCSI from my backup server to my FreeNAS 11.1-U7.
It is a 4 TB volume.
I encounter many iscsi disconnect due to high I/O when backup occurs.
can you help me (and others) about the udev settings ?

I have that for the moment:

Code:

ATTR{device/timeout}="120"
ATTR{queue/scheduler}="deadline"
ATTR{queue/nr_requests}="512"
ATTR{queue/read_ahead_kb}="1024"
ATTR{queue/max_sectors_kb}="32767"

Volume is presented through multipathd (2 connections) to my backup server.

Thank you for your help

Nicolas.

 

[jgreco]

What are "udev settings"?

Also, please follow the forum rules and provide a detailed accounting of your hardware and configuration.

 

[nicolasb827]

Hi,
sorry.
The volume where errors are set is the 2x8TB (mirror) + SSD

I am searching for udev rules.d file for the iscsi initiator (linux centos 7).

Storage is connected using multipathd on linux, using 2 differents nic:

 

[nicolasb827]

To complete, i see holes in graphs during backup. It show that no more data are received by freenas no ?

 

[jgreco]

Hi,
sorry.
The volume where errors are set is the 2x8TB (mirror) + SSD
View attachment 29763
I am searching for udev rules.d file for the iscsi initiator (linux centos 7).

You'll need to contact your OS vendor for that, then. FreeBSD is completely different than Linux.

Storage is connected using multipathd on linux, using 2 differents nic:

View attachment 29764

You probably need to get rid of the jumbo frames. Incorrect configuration is known to cause stalling and all sorts of other crazy problems. You should probably also get rid of multipath. Get a working, stable, basic configuration going FIRST. Walk before you try to run. Once you have a stable basic configuration, introduce one new thing at a time until you identify your problem.

 

[HoneyBadger]

I see MFI in your device name so I smell hardware RAID, and whenever see 8TB drives I get worried about the possibility of SMR.

Please post full hardware specs as requested, including your HBA model and disk connection method, and the full model number of those 8T disks.

 

[jgreco]

I see MFI in your device name so I smell hardware RAID

Oh, good catch! That is ALSO known to be very problematic. You need an HBA.

and whenever see 8TB drives I get worried about the possibility of SMR.

And SMR drives on the back end of MFI would be... a catastro-fsck. LSI RAID controllers are notoriously fickle about poorly performing drives, and yes, that'd be a double whammy of non-awesome-sauce.

 

[nicolasb827]

Thank you.

You'll need to contact your OS vendor for that, then. FreeBSD is completely different than Linux.

Sure, I was speaking about 'client' part of iscsi connection :)
I am working for more than 20 years with BSD, but adjusting /proc/sys is a difficult task on linux

Get a working, stable, basic configuration going FIRST.

I already have a good stable connection with 2 differents SANs constructor on my client server (Equallogic and Compellent).
I cannot disable jumbo frames. I checked from bge* to eth* and there is no errors on my client side.
I re-checked the FreeNAS side, and configuration is good (mtu 9216 on juniper SW, on both interfaces).

The 8TB disks are:

As you can see, from hardware point of view, they are not confgured as raid device. Mirror mode is configured into freenas GUI.

 

[jgreco]

I cannot disable jumbo frames. I checked from bge* to eth* and there is no errors on my client side.

Of course you can disable jumbo frames. "ifconfig <int> mtu 1500".

As you can see, from hardware point of view, they are not confgured as raid device. Mirror mode is configured into freenas GUI.

"I am only hauling groceries with my truck, therefore I call it a car".

If you are connecting your drives through a hardware RAID controller, they are RAID devices. The drive is being presented to the host as a virtual disk abstraction. The fact that it happens to be a JBOD abstraction is irrelevant. The problem is that you're attaching them through a driver that's known to be a bit flaky and problematic. FreeNAS is known to flood out hardware RAID controllers. The MFI driver is known to stall. The exact interaction there isn't clear, but it hasn't been worth it to anyone to figure it out, because it's the wrong way to attach drives.

If you're not willing to listen to the suggestions I have, that's fine, of course, but then I can't help you.

 

[nicolasb827]

Of course you can disable jumbo frames. "ifconfig <int> mtu 1500".

Yes, but on the other side, i will have a problem !

If you're not willing to listen to the suggestions I have, that's fine, of course, but then I can't help you.

Sorry to ask dumb question ...
so, it seems that FreeNAS is not compatible to HDD through PERC Controller. ok, I read that.

 

[HoneyBadger]

As you can see, from hardware point of view, they are not confgured as raid device. Mirror mode is configured into freenas GUI.

As @jgreco mentioned, they are still connected through a hardware controller. The H330 is not the HBA330 - the former is a cut-down MegaRAID with poor queue depths, a RAID firmware, and a lousy FreeBSD driver. The latter - well, I'm not sure they make it in a version that's compatible with your R720xd. You might need to sacrifice a PCIe slot to fit a proper LSI HBA like the 9207-8i or similar.

 

[jgreco]

Yes, but on the other side, i will have a problem !

Set both sides.

so, it seems that FreeNAS is not compatible to HDD through PERC Controller. ok, I read that.

This is PC hardware. In the old days, when you bought a Sun server, and you bought that network or storage controller, the manufacturer had spent significant time and effort in making a driver that performed flawlessly with Solaris or SunOS, because that was literally its only purpose in life.

With the PC architecture, and FreeBSD, you sometimes get a driver from a vendor, but more often no driver, and someone has to write one, and there's perilously little documentation, and even for the vendor drivers, FreeBSD and Linux are niche markets because they aren't Windows. Now this might kind of make sense to you at some underlying level, but it's worth making the point very explicitly -

Rather than hoping for the best with some random bit of hardware, which probably works "okay" with FreeBSD on a lightly loaded platform, it is important to remember that ZFS is going to push your hardware hard. In order to have pleasant operation of your NAS, your storage and networking layers both need to be impeccable. If they are not, this shows up as end user unhappiness.

We have a number of recommended bits of hardware that are recommended precisely because they are expected to be trouble-free. For example, LSI 2008-based HBA's that are crossflashed to 20.00.07.00 in IT mode are known to be trouble-free, having probably a billion aggregate hours of problem-free runtime across all FreeNAS installs.

Likewise, multipathing and jumbo frames are known to suffer from a variety of quirks and defects.

So the first thing we do when people show up with these things is we try to establish a stable system for them. Once stable, you can move forward from there.