Looking for dedicated, slightly masochistic BETA testers!

[reqlez]

I would expect that each connection would be a thread or something like that...

Well... you made your point, sir ! I was expecting the requests from a single connection to be split between 2 threads ... maybe i'm asking for too much :P

 

[jgreco]

No idea how to make that work well given the paradigm ... probably possible but most people will be using multiple initiators.

 

[reqlez]

So the question is ... each LUN has its own connection, right ? ( since you can specify different chap passwords, etc ) or you would HAVE to use several ESXi hosts ?

 

[jgreco]

"its own connection"? No, it would have "its own connections" - at least one to each initiator, possibly more if multipathed.

 

[reqlez]

"its own connection"? No, it would have "its own connections" - at least one to each initiator, possibly more if multipathed.

Okay i guess I'll make another target for a windows workstation and run IOMETER on both the VM and the windows initiator at the same time and see.

 

[reqlez]

Okay ... i can't get the WCPU on iscsi service to go over 120% ... multithread or not ... i'm starting to think that the option is not even working... but i did realize something today, the windows iscsi initiator sucks ! very high latency, and take forever to reconnect when the connection drops ...

 

[pbucher]

I think a lot of the problem stems from the amount of poorly informed or misinformed bullshit too easily found on the Internet on the topic;

Quite frankly if you want to make a 4GB non-ECC FreeNAS with a pile of USB2 drives all attached to a single powered hub with no redundancyand then bludgeon sync=disabled, that's fine by me as long as you understand the heaps of badness.

There is an unending parade of such choices that flow in and want help. There are also just as many bad choices on the ESXi side of things. Since many of us are here for the benefits of ZFS, that implies certain things about the expectations and values. We tend to err on the "correctness" side of most issues

I understand all too well...but really after all those bad choices does it really matter what the sync setting is? I also whole heartily agree that their is way too much bad info on the internet on these topics. But let's also remember that some folks have to make due with less then mirrored SLOGs much less things like ECC ram.

Finally regardless of how perfect of a setup you have, you are a plain fool if you don't have a backup(things like the toddler knocks the server over and all the drives crash happen). So for folks who loose the family photo album because they used a bunch of USB drives there is no need to educate them about their string of stupid choices just teach to make a backup(it's far easier). Nothing wrong with some education but it's been waring on everyone trying to straighten out every noob who shows up on the forums(it seems that FreeNAS has become a major magnet for noobs too).


Now back to debating multi-threaded iSCSI.

 

[cyberjock]

Sorry, I was conflating the options in my own posting. Yes, sync=always is the zfs dataset option. "Option WriteCache Disable" (which can be set on a per-LUN basis) is the iSCSI (istgt.conf) configuration parameter for this.

So how do you set that value in istgt.conf in FreeNAS? I'm not seeing anywhere you can actually set that....

 

[Weeblie]

So this is where things get *really* messy. After tons of research I'm convinced that doing NFS with sync=disabled is about the same danger level as using iSCSI with sync=standard or sync=disabled. Also, if you want to use iSCSI with approximately(keyword: approximately) the same level of protection as NFS with its sync writes, then you must set sync=enabled when using iSCSI. The reason for the "approximately" is that iSCSI may internally return the write as complete to the iSCSI initiator *before* issuing the write to ZFS. So there may be a chance that you could lose data even with sync=enabled. Keep in mind that in a properly operating system we are talking fractions of a millisecond, but that is the "write hole" that ZFS is supposed to prevent. But its still not quite the same as using NFS with sync writes. The safest is to use NFS with its sync writes.

There's 2 things to realize about the sync=disabled situation:

1. This will tell the system to never honor sync writes(this is obviously not recommended, but you may want to take that risk). If you have other services that are dependent on that sync write being honored and performed, you may be in for serious trouble.
2. There is no significant possibility of harm to the pool if you recognize that every write to the file system is an atomic write. That is to say that any single write to the file system either hasn't been performed or has been performed and completed. There is no such thing as a "partial" transaction. ZFS will automatically discard any incomplete transaction when the pool is remounted after a "partial" transaction. Obviously if ZFS discards a partial transaction then that data is lost.

So, if so many people are using sync=standard with iSCSI and not having problems, why am I hesitant to turn around and start recommending setting sync=disabled? I just said they are the same thing, right?

Excellent post, although I have a few extra remarks of my own:

Assuming that ESXi's implementation is sane; using "sync=standard" with iSCSI should be a bit safer than using "sync=disabled" with either iSCSI or NFS.

Let me try to explain my reasoning!

SCSI/SATA (and iSCSI) actually does provide the ability to perform non-cached writes through a Force Unit Access flag, but what usually ends up happening on consumer grade hardware is that they cheat and ignores it for the sake of performance. I had a brief look at the istgt sources at Google Code (hopefully not too different from the one used by FreeNAS) and it seems to me like istgt ignores it too.

But fortunately for us, it's a situation that modern OS makers are taking into account and compensating for. Newer versions of Windows will for example issue a full blown Sync/Flush after these writes and which are honored. The most logical way for ESXi to handle iSCSI is simply to forward all commands to the target, and istgt seems to respond to that with a fsync.

So iSCSI with "sync=standard" should therefore from a "successful writes are lost after a power outage" point-of-view be no more dangerous than using regular consumer grade HDDs in your ZFS pool.

What about "sync=disabled"?

Well, that's telling ZFS to also ignore the follow-up flush from the VM so all bets are really off... :)

TL;DR: iSCSI with "sync=standard" is the way to go (unless you are super paranoid or super lucky).

 

[jgreco]

Thst's mostly nonsense. Both sync=disabled and sync=standard are risky. "A little less risky" is a hell of a strange thing to argue in a storage array...

The ZFS transsction group write model blats out massive txg's that are going to tend to force data out to the disk, and in practice you need to take how that all works into account.

As noted above the txg commit is more or less an all or nothing deal. The best idea is to have a SLOG device to guarantee that the data is committed; in that one specific case it is actually interesting to debate the finer points of disk caching IMO.

 

[jkh]

So how do you set that value in istgt.conf in FreeNAS? I'm not seeing anywhere you can actually set that....

In the web GUI, nowhere - you'd have to sneak behind its back and modify the istgt configuration from the CLI. I know, hardly optimal. If it weren't for the fact that kernel iSCSI is about to kick istgt to the curb, I'd suggest adding some additional configuration options to the GUI. That's all about to be substantially roto-tilled, however, so better just to see how the configuration morphs overall and then DTRT with the new code.

Ahem. Can we get back to discussing BETA testing in this thread now? I haven't seen one mention of people talking about their experiences with 9.2.1.6-BETA recently in this thread and am almost tempted to just rename this thread to "iSCSI discussion" and start a completely new one. :)

 

[cyberjock]

I've had some problems with the 9.2.1.6-beta. Tickets are in for both of them and both are slated to 9.2.1.6-RELEASE so I'm just waiting for those to be handled.

 

[Weeblie]

Thst's mostly nonsense. Both sync=disabled and sync=standard are risky. "A little less risky" is a hell of a strange thing to argue in a storage array...

The ZFS transsction group write model blats out massive txg's that are going to tend to force data out to the disk, and in practice you need to take how that all works into account.

As noted above the txg commit is more or less an all or nothing deal. The best idea is to have a SLOG device to guarantee that the data is committed; in that one specific case it is actually interesting to debate the finer points of disk caching IMO.

What "a little less risky" actually means is "what developers would expect from any other filesystem". Any "important data storage engine" such as databases won't assume that all writes will hit the disk immediately - you have to do a flush first. Using "sync=standard" is perfectly safe for these programs since that's the sort of behavior they are designed for.

Using "sync=always" would not solve the problem for other software, especially if they are in VMs. Their "successful" writes could still be stuck in the write cache of the VM's OS when the power outage hits (i.e. lost before even making it to the iSCSI target).

It also doesn't really matter that ZFS batches writes into TXGs since the user program in the VM has no control over it. TXGs are neither guaranteed to prevent partial writes nor out-of-order writes.

An extreme example of the former is if you issue a single 1 GB write operation from your user program in the VM. It will be chopped up by the guest OS and not guaranteed to be squeezed into a single TXG.

An extreme example of the latter is if you do writes that are moving backwards. The VM's OS will most likely reorder them.

In either case; it's ultimately up to each person to decide what settings they want to use.

Ahem. Can we get back to discussing BETA testing in this thread now? I haven't seen one mention of people talking about their experiences with 9.2.1.6-BETA recently in this thread and am almost tempted to just rename this thread to "iSCSI discussion" and start a completely new one.

Alright, back on track! Only done some minor testing but the 20140516 build is working just fine for me (it's mainly a CIFS server). :)

 

[survive]

Hi Jordan,

I've been running the 9.2.1.6 betas since you first posted about it & the only problem I've had involves the new feature that moves the rrd data to the system dataset.

What i can tell you is ticking the box in the new "Settings" -> "System Dataset" tab appeared to try & move the rrd data, but it didn't work & it appears to have lost several months of reporting data as a result. Nothing to terrible, but the "volatility" of the reporting data has long been a pet peeve of mine about FreeNAS.

-Will

 

[jkh]

the only problem I've had involves the new feature that moves the rrd data to the system dataset.

What i can tell you is ticking the box in the new "Settings" -> "System Dataset" tab appeared to try & move the rrd data, but it didn't work & it appears to have lost several months of reporting data as a result. Nothing to terrible, but the "volatility" of the reporting data has long been a pet peeve of mine about FreeNAS.

Can you file a bug report for that? That's *exactly* the kind of thing we want to find out about (and fix) between 9.2.1.6-BETA and -RELEASE! It should "just work" and be seamless. Please also indicate which BETA (date) you were running when you saw this. Thanks!

 

[survive]

Hi Jordan,

Of course I filed a bug report!

https://bugs.freenas.org/issues/4976

-Will

 

[reqlez]

LoL at the iSCSI thread comment. Looks like I started something bad ... But at least people know where freenas is headed with iSCSI now ! So I'm not sure if this is a big or what, but before I was able to enable SSL on the interface, now it always says configuration file is bad or something and it reverts back to using http for the management interface ... What's up with that ?

 

[zambanini]

upgrade from 9.2.1.3:
first reboot (something with system dataset .bak) was ok, on the second reboot i saw a message like touch /boot.config, read only filesystem - system was stuck for 15minutes, had to reboot it - iscsi, cifs und nfs work as expected (AD domain joined, iscsi export - one lun only)

things that should be fixed (in my opinion):
* changing tunables still freeze the system für 3-7 minutes (system pool is on the disks, ssh and web is unresponsive in this time) - at least this is annyoing on a system with 96gb ram and 16 sas disks
* "view disks" tab does only query one sas path on multipath systems - so you just miss most of the drives in the disk overview

 

[jkh]

Hi folks!
Well, it’s been awhile since a good candidate for 9.2.1.6-BETA testing was announced, and we are happy to say that http://download.freenas.org/nightlies/9.2.1.6/BETA/20140529/ is that build!
Lots of good stuff in this one. We’ve fixed 41 tickets so far in 9.2.1-BRANCH since 9.2.1.5 was released, some of the most notable bugs/features being:

• Added extra safety belts and sane ACL behavior for CIFS sharing.

Now when you create a ZFS dataset with the type set to Windows, or you create a Windows share on an existing dataset with “Default Permissions” checked (that’s a new GUI option in the CIFS sharing dialog), FreeNAS will make sure that the ACLs / ownerships are set properly for Windows and it will also make sure that the behavior of chmod(2) is set in ZFS such that it does not also destroy the Inherit, Delete or Delete Child ACLs whenever it’s used. That was the old behavior, making chmod essentially a destructive operation as far as ACLs were concerned, and we’re pretty sure this explains a good number of the “I upgraded from 9.2.0->9.2.1.x and now my permissions are all broken in Windows!” reports, because one of the first thing folks try to do when they have a permission problem is reach for chown/chgrp/chmod, and that made Windows very unhappy up until now because Unix permissions and ACLs fought.

• The System Dataset now has its own Settings UI that allows you to control the placement of various sub-datasets under it. RRD (collectd) data, for example, can now live on it’s own sub-dataset of the .system dataset
• The System Dataset will now migrate if you change the pool assignment, preserving your data and also moving it off the previous pool.
• A new Virtualbox jail template has been added. Supporting Virtualbox in a jail is as easy as creating an instance of this template.

Please test this build and let us know what your experience is like. We’re down to just 11 tickets for 9.2.1.6-RELEASE, and some of them are pretty obscure, so we’re hoping that this build is actually pretty close to release quality for 99% of the FreeNAS user community!

Thanks,​

- The FreeNAS Development Team

 

[Yatti420]

Great to see progress being made.. Glad ACLs / CIFS (hopefully) wont be an issue anymore..