10Gb tunables on 9.10

[jgreco]

jgreco and the need for speed

Ironically I don't tune for speed that much, I tune for massive parallelism. There's not much point to optimizing for single transfer speed. I just need it to be reasonable in the face of many other things also going on with the pool.

 

[cyberjock]

I'm not sure whether to apologize for that, or to point out that I simply took what you said at face value.

Either way, I've been unhappy with those defaults for some time now. It'd be really nice if autotune was updated to be a little smarter at the same time, but I know you're not a code guy.

Few things:

1. iXsystems is not going to change the defaults just because I said so. I'm gonna need serious evidence that shows this isn't detrimental for some scenario that FreeNAS is likely to be used in.
2. I may still not get the defaults changed even if I have an encyclopedia of evidence supporting some changes. (why fix it if it ain't truly broken?)
3. Even if I get the defaults changed, there's probably going to be some scenario that it's not "the best" in. Them's the breaks though. ;)

As for the autotune being changed significantly, I agree (and all of the devs agree). It's just not a high priority at this point (9.x world). The plan, last I heard, was that in 10.x we'd have the autotuner be a bit smarter, and you'd choose the workload (file sharing, VMs, etc.) and the autotune would change things that go beyond some silly dumb code like what autotune uses right now. It might do things like adjust ZFS tunables, disable prefetch if your workload chosen is VMs, adjust network parameters based on the NICs you have installed, etc. It's just not a priority and since the original intent of the autotuner (prevent ZFS from being overzealous with its ARC and causing kernel panics) is still being met, getting resources to write the new tuner is just too low on the priority. Everyone wants to see FreeNAS 10 go out the door. ;)

 

[jgreco]

Well, it *is* autotune, and we know it breaks some things right now. There's certainly a chance I could release a patched version, or we could just maintain a database of known tweakables and maybe some better documentation on the how-and-why (I get the sinking feeling that I'm the obvious party for that unenviable task).

 

[cyberjock]

Well, it *is* autotune, and we know it breaks some things right now. There's certainly a chance I could release a patched version, or we could just maintain a database of known tweakables and maybe some better documentation on the how-and-why (I get the sinking feeling that I'm the obvious party for that unenviable task).

Can you cite an example? I'm not aware of anything that "break" with autotune.

 

[jgreco]

Can you cite an example? I'm not aware of anything that "break" with autotune.

Naw. I've never really kept a list of issues by-issue. I just kinda know that things are not sized well for large-and-up boxes (64GB++) and can actually break small boxes sometimes (8GB).

 

[cyberjock]

I could see autotune not working right for systems with 8GB of RAM. But that's not the intent of autotune. It's to limit excessively large systems (which a system with 8GB of RAM is not). So I don't consider it a bug, more of a "don't use the wrong tool for the job".

 

[jgreco]

I could see autotune not working right for systems with 8GB of RAM. But that's not the intent of autotune. It's to limit excessively large systems (which a system with 8GB of RAM is not). So I don't consider it a bug, more of a "don't use the wrong tool for the job".

Not that I disagree with you, but I've spent ~5 years on this forum, watching people come in doing things that you and I know are dumb. But if you go to the GUI and actually look at what it says next to "Enable autotune", it makes no mention of such a limit, and even kinda implies otherwise... "Attempt to automatically tune ... based on memory available"

This isn't a PEBCAK class error in my opinion. I've had to explain it too many times.

 

[cyberjock]

Not that I disagree with you, but I've spent ~5 years on this forum, watching people come in doing things that you and I know are dumb. But if you go to the GUI and actually look at what it says next to "Enable autotune", it makes no mention of such a limit, and even kinda implies otherwise... "Attempt to automatically tune ... based on memory available"

This isn't a PEBCAK class error in my opinion. I've had to explain it too many times.

Oh, I totally agree with you. Unfortunately, as I said before, getting resources to fix this (even just changing the autotune note to be a bit more accurate) is difficult. :/

The autotune thing is one of my sore spots because it doesn't clearly describe "when" to use it. It really is one of those magical things that I know because I work with the devs so I know what the intent is. I really wish that the autotuner would be fixed (even if the note just makes it more obvious as to when to use it and when you shouldn't bother). If I knew my way around python and such, I'd try to fix it myself. I'm sure its dead simple to fix... if I could figure out where it is and such.

 

[xhoy]

The autotune thing is one of my sore spots because it doesn't clearly describe "when" to use it. It really is one of those magical things that I know because I work with the devs so I know what the intent is. I really wish that the autotuner would be fixed (even if the note just makes it more obvious as to when to use it and when you shouldn't bother). If I knew my way around python and such, I'd try to fix it myself. I'm sure its dead simple to fix... if I could figure out where it is and such.

Well devs can only improve it when they have a list of things that can be fixed/improved. So i would suggest starting with a list of stuff that COULD change and conditions that COULD be improved.

If there is a list *somebody* could improve autotune, as long as there isn't a list its never going to beimproved :)

I would suggest starting a list of possible(usefull) use cases

Use cases:

1. Network file storage (mostly cif)
2. Backup file storing (mostly cif)?? (write oriented?)
3. VM storage (NFS)
4. VM storage (iscsi??? is that even recommended atm?)
   
5. Freenas backup machine (zfs sync optimised)

Then there would be several system scenarios:
In all scenario's reliability is most important then performance :)

1. Small home use
   1. <16gig, slowish cpu, 4/6 disks raidz2?
      
   2. mostly for use case 1 and 2?)
   3. 1Gb
2. large home use
   1. >16GB, medium cpu (i3+), 8+ disks raidz3?
   2. mostly use case 1 and 2
      
   3. 1Gb
3. medium business
   1. 16GB+, medium CPU (i3+?) 4 disks?
   2. mostly use case 1/2
   3. 1Gb
4. large business
   1. 20+ users (light users) or 5+ heavy users
      
   2. 32GB+, CPU (e3+?) 8+ disks?
   3. mostly use case 1/2
   4. 1gb (maybe dual nic/teaming?)
5. small Virtualization infrastructure (NFS)
   1. < 10vm's (max 2 vsphere servers via NFS?)
      
   2. 32GB+, CPU ?, disks? slog?
   3. use case 3?
   4. 1Gb nic
6. medium Virtualization infrastructure (NFS)
   1. < 20vm's (max 2 vsphere servers via NFS?)
   2. 32GB+, CPU ?, disks? slog?
   3. use case 3?
   4. 10Gb nic?
7. big Virtualization infrastructure
   1. < 25vm's (max 2 vsphere servers via NFS?)
   2. 64GB+, CPU E3+, disks? slog?
   3. use case 3?
   4. 10Gb nic's
8. Same as scenarios 5/6/7 but then with ISCSI? (if recommended?)
   
9. Freenas as a backup for another freenas machine
   1. 16gb, CPU ?, disks? slog?
   2. use case 5
   3. 1GB nic (possible WAN speed?)

As I am writing this down is notice that the scenario's and the hardware recommendations kinda of align, so maybe the hardware guide needs an update with some scenario's and some hardware recommendations?

PS: I am just guess what this could be but imho "changing" something is easier the working from scratch so feel free to enhance :)

 

[cyberjock]

Well devs can only improve it when they have a list of things that can be fixed/improved. So i would suggest starting with a list of stuff that COULD change and conditions that COULD be improved.

If there is a list *somebody* could improve autotune, as long as there isn't a list its never going to beimproved :)

I disagree. The devs will actually be doing real-world tests with various hardware configurations and such to come up with numbers that have been shown via empirical data to provide benefits with little (or no) downsides (and especially no kernel panics, etc.).

Us (the community) providing a list is going to mean nothing to them without lots, and lots of data, lots and lots of explaining of how these values came into play, how we covered "all the bases" and such. I doubt anyone in the community has the required hardware, time, and money to really take on this task easily. To boot, they aren't like to be particular interested in data that isn't from TrueNAS or FreeNAS certified because there's no way to know what kinds of tweaks or other things you may have done to the system that may have invalidated the data.

Believe it or not, quite often people with problems on FreeNAS are because of poor hardware choices, bad hardware, etc.

The iXsystems wormhole (catches the data you allow to be submitted when you allow telemetry data from the WebGUI option) provides the cause for kernel panics to iXsystems for lots of reasons (don't worry, it's not particularly useful for trying to find "you" out of a bunch unless you call and give us your host's GUID number. Something like 95% of them are due to RAM errors/failures. Not. Even. Joking.

With that kind of info, the community has lost a bit of credibility with regards to "building good reliable systems". So providing info on something as sensitive, as debatable, and as potentially catastrophic as tuning values is not going to get far in iXsystems. Even my system really doesn't carry much weight with iXsystems IMO. Sure, I did virtually "all the right things" but the reality is that my system RAM is still smaller than anything iXsystems sells. The smallest you can buy is 64GB of RAM and 2x4c/4t CPUs. I've got 1/2 the RAM and a relatively outdated CPU (E3-1230v2).

I won't lie. I used to give the community a lot of credit for the things we did. The reality is that the install-base has a good portion of the people that *are* doing things right, but there's a whole lot doing lots of things wrong. The data is pretty clear and really paints a picture that isn't all rosy with regards to "doing things right". This issue with trying to separate out hardware problems from software problems is one of the many "road hazards" that the developers have to work through with bug tickets.

If I were the deciding factor on what we put in and don't, I'd give much less value to data provided by the community than data provided by iXsystems own internal research and testing. Not that you (or I) couldn't do a good honest effort, but its just not the same as someone who has touched the code and can consider nuances you and I might not be able to.

 

[bartnl]

but then again if many people test the FreeNAS software this by pure numbers statistically would be meaningful. I suspect that's the whole reason of releasing daily alpha and beta releases to the public.
If iXsystems would just be selling iXsystems there would be no need for these. Data by iXsystems will be limted to the hardware they sell which is meaningless to 99% of the user as they are using non-identical hardware. I am dissapointrd that iXsystems / FreeNAS is not advertising proven hardware setups based on their experience. Whithout this info any autotune generated parameters provided by iXsystems / FreeNAS are less meaningful as I as user have no clue if these are optimised for anything like my FreeNAS setup.

 

[jgreco]

With that kind of info, the community has lost a bit of credibility with regards to "building good reliable systems".

You know, I've seen the hardware iX sells, and overall, ... not that impressed. Your message is overall rather insulting, especially seeing as how much effort some of us have put into helping others build good, reliable systems.

Some of us have been tuning FreeBSD since FRICKIN BEFORE IT WAS CALLED FREEBSD, and have spent time doing network tuning at levels no TrueNAS or FreeNAS box will ever see.

 

[xhoy]

Lets not start a flamwar here!

I think everybody has a point. BUT there are different interests for different ppl/organisations. iXsystems Needs to make money on something so i dont think we can give them a hard time for releasing there software/updates to the community.

Then back to @cyberjock's point.

Us (the community) providing a list is going to mean nothing to them without lots, and lots of data, lots and lots of explaining of how these values came into play, how we covered "all the bases" and such.

I dont agree with this. There are so many options that i can image a dev doesn't have the time to "look" or "test" all of them. If we could compile a "short" list of options we "consider" (since I agree with you they should NOT just dump that in a new tune script!) relevant in some situations. This would give the devs a shortlist of options and could focus there time.

I see it this way. If the community proposes some changes of enhanchment a dev only needs to validate them. If we do nothing a dev needs to work from crash. I would think that a shortlist could potentially shorten the dev cycle. Ofcz only if the community can supply some validation of the quality of the changes and a basic test.

 

[cyberjock]

You know, I've seen the hardware iX sells, and overall, ... not that impressed. Your message is overall rather insulting, especially seeing as how much effort some of us have put into helping others build good, reliable systems.

Some of us have been tuning FreeBSD since FRICKIN BEFORE IT WAS CALLED FREEBSD, and have spent time doing network tuning at levels no TrueNAS or FreeNAS box will ever see.

So you are hurt because you cannot differentiate between what "should" be done, and what people "actually" do?

Fantastic! Thanks for letting me know.

@xhoy,

You need to understand something. We don't do support contracts for FreeNAS. So at the end of the day, FreeNAS has no bearing on how we do tunables, etc. Yes, they are similar, but they are not identical. If we push an update for TrueNAS that hurts performance, causes kernel panics, etc then iXsystems would be on the hook to deal with the problem, and quickly. Tunables require lots of time and testing. iXsystems just couldn't deal with the onslaught of calls that could come in if tunables were borked in some serious way. So iXsystems naturally takes changes to things like tunables very, ver seriously. They are NOT going to just listen to a bunch of people in the forums. They are going to do their own testing, etc. Too much is at risk to trust what some people say they did without doing the tests themselves. That's why the Autotuner hasn't been fixed. There is serious risks involved with changing the defaults. The risk versus benefits of trusting a guy(s) that did some testing of their own on FreeNAS systems leans so heavily towards the risk side of the scale that there's really no discussion at all.

 

[jgreco]

So you are hurt because you cannot differentiate between what "should" be done, and what people "actually" do?

Fantastic! Thanks for letting me know.

No, I'm offended that you paint a wide swath with that condescending paintbrush of yours. I say that as someone who is largely responsible for having pushed this userbase from a collection of hobbyist users trying to run FreeNAS on 2GB AMD E350 systems into a community where we all kinda snicker when n00bs are using inappropriate hardware, because the benefits of using server-oriented hardware have been well documented. There is nothing all that magic about iXsystems or tuning FreeBSD systems. A FreeNAS system isn't particularly stressy on the TCP side of things; sometime try optimizing a system to support 10K simultaneous data transfer connections of varying bandwidth and latency over the public Internet on an 8GB RAM system and you'll learn more about mbuf/mbufcluster/etc sizing and kernel panicks than you ever wanted to know. By comparison, FreeNAS is a gorgeous platform because it tends towards small numbers of high bandwidth connections on a platform where there are tremendous resources available.

I know the devs don't think much of the forum users, but really, that's their problem.

 

[cyberjock]

Well, I guess you'll have to be upset. I'm really not seeing how my "condescening paintbrush" ever painted over you or anything you've done in the forums.

What you recommend, and what people actually do are not the same thing. That's the problem. Plain and simple.

 

[BigDave]

The FreeNAS version of Godzilla vs. King Kong...
popcorn anyone?

 

[jgreco]

The FreeNAS version of Godzilla vs. King Kong...
popcorn anyone?

I musta missed it, when did Godzilla arrive?

 

[xhoy]

We could still make a Community recommended settings thing. I know its a longs shot to get it in the releases but we COULD just create a list for our selfs and try to improve freenas :)
Users who read the guides will learn and follow them, ppl who run crappy hardware and lose there data will do that anyway, regardless of the recommended stuff :)

 

[jgreco]

We could still make a Community recommended settings thing. I know its a longs shot to get it in the releases but we COULD just create a list for our selfs and try to improve freenas :)
Users who read the guides will learn and follow them, ppl who run crappy hardware and lose there data will do that anyway, regardless of the recommended stuff :)

Yeah, it's on my list of things I'm working on.