64 bit only FreeNAS distributions?

[joeschmuck]

Well thought I'd chime in as well...

I vote to remove support for any further 32 bit support like the others here and it really is a valid move. You have seen arguments on why this should occur but the simple truth is it's going to be more difficult to support it from the developers point of view. I think this needs to be communicated to the community clearly when it is done. For anyone really needing a 32 bit version then the older versions of FreeNAS should be able to support them or they could migrate over to NAS4Free which is a similar product. If security is an issue then they should figure out a firewall or upgrade the hardware to support 64 bit.

The other question in this thread is the RAM system requirements. Since UFS is a valid and supported file system there should be two limits listed. An example might be:

For UFS only support: 2GB RAM Minimum
For ZFS support: 4GB RAM Minimum, 8GB RAM Recommended (see RAM Performance)

RAM Performance: 8GB RAM is recommended for total drive pool sizes of 8TB or less. The rule of thumb is 1GB of RAM for each TB of Pool space.

Of course you would have to get into some lengthy discussion about pool calculations and how more RAM benefits the NAS. This isn't something we all haven't previously discussed before and should be easily conveyed.

 

[jkh]

I think what we're going to do is this:

Release FreeNAS 9.1.2 as x86_64 / i386 (no firm ETA, but around 90 days from now)
Switch to x86_64 only for FreeNAS 9.2 (about 12 months from now).

I think this should give everyone the benefit of enough warning while still allowing the FreeNAS debs to see a small circle of light at the end of the tunnel on supporting those damned i386 builds / plugins / jails / etc. :)

Thanks for all the feedback!

 

[jgreco]

I think what we're going to do is this:

Release FreeNAS 9.1.2 as x86_64 / i386 (no firm ETA, but around 90 days from now)
Switch to x86_64 only for FreeNAS 9.2 (about 12 months from now).

What exactly is the version numbering strategy, anyways? With the release of FreeBSD 9.2R, is the NanoBSD framework being updated? What relationship, if any, is there intended to be between FreeBSD and FreeNAS release numbering?

 

[cheezehead]

All for dropping 32-bit support. The ram requirement is a bit different, how about a slight twist on it with ZFS creation disabled if the system has less than 8GB...ie UFS only for <8GB? ...ZFS if mounted with 4GB would work...ie your primary box tanked and the box in the closet has 4GB and you only really need to get a couple of files off it scenario.

 

[cyberjock]

ZFS if mounted with 4GB would work...ie your primary box tanked and the box in the closet has 4GB and you only really need to get a couple of files off it scenario.

Do you know how many users have lost their entire pool because they used 4GB of RAM? I wouldn't dare mount one of my pools with that little RAM.

 

[cheezehead]

Do you know how many users have lost their entire pool because they used 4GB of RAM? I wouldn't dare mount one of my pools with that little RAM.

Depending on the size of the pool 8GB wouldn't be enough either.

 

[cyberjock]

Depending on the size of the pool 8GB wouldn't be enough either.

Yes, but nobody has actually killed their pool despite its size with 8GB of RAM. Sure, it pay perform poorly, but at least it didn't kill the pool. We've had people that crashed the system and on reboot never could get the pool to mount again.

There's a big difference between "performs poorly" and "the pool is unmountable". And let me point out that your standard hard drive recovery tools that look for files without a file system doesn't work with ZFS, so if its unmountable your data is gone unless you have a backup.

 

[joeschmuck]

I don't believe that 4GB RAM would cause any pool loss if someone was running 4 to 6 TB of pool space. I can believe that there have been losses that people have theorized were attributed to low RAM amounts but that wasn't very scientific and who knows what those folks really did when trying to fix the problem, or how flaky their system was. I'm speaking from experience that 4GB was fine (FreeNAS 8.x), never had an issue. Once I was able to, I did bump my RAM to 8GB and now to 16GB but I was planning to create a larger pool as well and gain a little better performance. I'm certain there are folks still running 4GB RAM (non-ECC) without any issue as well. You may not hear that often but that's because they set up the NAS and left it alone and it runs. I like those types of systems myself.

So I think there should be a very objectionable view taken when determining minimum RAM requirements. And let's not forget to toss in the recommendation of using ECC RAM which I personally believe it should be a requirement if using ZFS to prevent data corruption. I myself wouldn't recommend less than 8GB ECC RAM to anyone building a system and would try to talk them into 16GB if possible due to performance issues but not out of stability concerns.

 

[jkh]

What exactly is the version numbering strategy, anyways? With the release of FreeBSD 9.2R, is the NanoBSD framework being updated? What relationship, if any, is there intended to be between FreeBSD and FreeNAS release numbering?

I don't know the disposition of the NanoBSD framework in FreeBSD 9.2R - you'd have to ask the FreeBSD folks (I'm not even a committer these days). There is no relationship intended between FreeBSD and FreeNAS release numbering. Up until now, it's been sort of "loosely tracking" but I don't think we'll continue that since it sets false expectations.

 

[jgreco]

I meant the NanoBSD framework in FreeNAS. How closely does the project try to track the upstream FreeBSD source?

 

[cyberjock]

I don't believe that 4GB RAM would cause any pool loss if someone was running 4 to 6 TB of pool space. I can believe that there have been losses that people have theorized were attributed to low RAM amounts but that wasn't very scientific and who knows what those folks really did when trying to fix the problem, or how flaky their system was. I'm speaking from experience that 4GB was fine (FreeNAS 8.x), never had an issue.

I agree, but also disagree. My disagree is only based on observation though.

ZFS shouldn't be having problems with insufficient RAM. It's designed to rollback incomplete transactions and whatnot. So it shouldn't be causing problems.

What does peak my interest though and this is only from observation is that we've had plenty of people that had kernel panics from insufficient RAM in the past. We've also had many of those users lose their pool on reboot. No matter what we did the darn things wouldn't mount. And generally when we can't put the finger on exactly what is wrong and just say "your pool must be corrupt" despite zfs not identifying it as corrupt they have always had less than or equal to 6GB of RAM. I've never seen a user with more than 6GB of RAM have the same kind of failure that is unexplained. Once you get to 8GB of RAM, we've never had an unexplained failure regardless of the size of their pool. At one point when I had RAM on order I had 12GB of RAM with a 36TB zpool. You and I talked about it in PM back in January too. I had a crash or two because of improper commands on the CLI. Pools came right back up without issue on reboot.

So, while I'm skeptical about the RAM limit being the cause too, it seems to be a pretty sure bet based on users in the forum over my tenure here. I'd really like to understand the cause for the problems more because ZFS is supposed to be able to rollback potential incomplete transactions when the system crashes during a transaction write. But, I've got no data to go on. Just observation. If it were a ZFS issue where transaction rollback was somehow broken, then I'd also expect that everyone and not just low RAM users would be susceptible. After all, even big servers have crashes from time to time. And surely some of them would have the same issue as the low RAM users, right?

So while I agree that RAM shouldn't matter, based on observation I just have to push the "I believe" button and recognize that this is a problem for many users and is currently unexplained. Of course, if someone figured out the exact cause and it just happens to be unrelated to insufficient RAM, I'm all ears. I'd love to see a techinical reason for the failures for those that have shorted their system RAM.

 

[jgreco]

I'll add, informationally, that some of the kernel parameters are what I'd consider "horrifyingly high" in FreeNAS, but this is largely a side effect of my being a product of the days when 32MB was a maxed out FreeBSD server (yay 486!) and even today many of our general service machines operate on 256MB. I feel reasonably certain that some of the problems experienced with panics are directly the result of tuning choices that are aimed at (and pretty good at) making large FreeNAS systems just magically work. The downside seems to be that once you get below 8GB, you're dangerously tight on memory.

The takeaway from that is that it's probably quite possible to have a FreeNAS system that's totally fine with ZFS at 4GB or maybe even 2GB, but it would require at a minimum some tuning, and quite possibly a different kernel image, depending on whether or not there are wired-in tunables that cannot be batted around in loader.conf.

 

[joeschmuck]

@Cyberjock,
I knew I would finally get a response out of you ;) and I'm not saying you are not correct either. Your observations are crucial to the enhancement of this product and to helping so many people in trouble, even though you were a fast attack sailor.

I hope the low RAM thing isn't due to a memory leak or RAM not being released properly because that could take a long time to locate and isolate.

 

[paleoN]

I meant the NanoBSD framework in FreeNAS. How closely does the project try to track the upstream FreeBSD source?

I thought they switched to trueos for the source, but it's been a while since I built FreeNAS.

 

[cyberjock]

I'll add, informationally, that some of the kernel parameters are what I'd consider "horrifyingly high" in FreeNAS, but this is largely a side effect of my being a product of the days when 32MB was a maxed out FreeBSD server (yay 486!) and even today many of our general service machines operate on 256MB. I feel reasonably certain that some of the problems experienced with panics are directly the result of tuning choices that are aimed at (and pretty good at) making large FreeNAS systems just magically work. The downside seems to be that once you get below 8GB, you're dangerously tight on memory.

But, that shouldn't affect ZFS' stability. At least, even if you tune it horribly, it shouldn't end up unmountable. Clearly, at some very deep level, something sinister is happening. The only way that I can deal with it is to put in 8GB of RAM minimum. For whatever reason once you get to 8GB of RAM, no matter how big your pool is nobody has had their pool become unmountable.

The takeaway from that is that it's probably quite possible to have a FreeNAS system that's totally fine with ZFS at 4GB or maybe even 2GB, but it would require at a minimum some tuning, and quite possibly a different kernel image, depending on whether or not there are wired-in tunables that cannot be batted around in loader.conf.

I totally agree. But the question to ask is how likely that is for the average user. Even I've goofed around with ZFS tuning and it doesn't come easy or fast. And users want an OS they install and begin using immediately. For 99.999% of them, tuning before using ZFS is out of the question.

 

[Alvin]

Please consider the people who use FreeNAS without ZFS. (I've used ZFS for years on FreeBSD atom servers with a 2GB ram limit, without any problems. It's low, but stable and fine if your expectations aren't too high.)

I use FreeNAS with UFS, and virtualized. Why? For having a gui on small file servers that can failover fast in case a motherboard dies. Since FreeBSD 9.2-RELEASE, we have virtio in the kernel, and UFS requirements are way lower than the proposed minimum of 2GB.

 

[jgreco]

I thought they switched to trueos for the source, but it's been a while since I built FreeNAS.

That could be, but there are still NanoBSD bits floating around, such as the scripts in /root or /etc/nanobsd.conf. I don't know enough about TrueOS to know what the history there is supposed to be.

 

[jgreco]

But, that shouldn't affect ZFS' stability. At least, even if you tune it horribly, it shouldn't end up unmountable. Clearly, at some very deep level, something sinister is happening. The only way that I can deal with it is to put in 8GB of RAM minimum. For whatever reason once you get to 8GB of RAM, no matter how big your pool is nobody has had their pool become unmountable.

Yes, in theory I agree, but see, the code was designed that way on Solaris. And I'm guessing it was tested extensively. But with FreeBSD, the code was imported, and then modified to fit within the FreeBSD framework, and there are ways in which it is substantially different because of it. So what really needs to happen is for someone to spend some time rendering test pools unmountable and then doing postmortem analysis to determine how they wound up that way. Unfortunately this is kind of difficult because unlike Solaris, there's no commercial relationship between affected customers and the operating system vendor, and the crew doing the ZFS is, I think, sizeof(1) ... just Pawel Jakub Dawidek last I recall. And it could be complicated further because kernel panics are interesting things. They typically stop most system activity dead cold, so if, for example, there was something in the kernel that was causing blocks to be written out in an optimized, reordered fashion that ZFS wasn't expecting, it is something that might only come into the light through forced panics, and then only if you were lucky...

I totally agree. But the question to ask is how likely that is for the average user. Even I've goofed around with ZFS tuning and it doesn't come easy or fast. And users want an OS they install and begin using immediately. For 99.999% of them, tuning before using ZFS is out of the question.

Well, yes, I agree. That's why my first post in this thread suggested something along those lines. But ... I think it needs to be done by developers who actually understand the ins and outs. It doesn't actually seem to be all that difficult to just bludgeon down the defaults, but knowing where to do it most effectively is best done by someone with intimate familiarity with the issues.

 

[Dusan]

That could be, but there are still NanoBSD bits floating around, such as the scripts in /root or /etc/nanobsd.conf. I don't know enough about TrueOS to know what the history there is supposed to be.

This is how it works: NanoBSD does not "contain" any FreeBSD source code. NanoBSD is a tool (it's basically just few shell scripts) that will help you create a pruned down FreeBSD system image, but you need to supply your own FreeBSD source tree. The FreeBSD source code used by the FreeNAS build process comes from the TrueOS git repo. I.e., FreeNAS is TrueOS trimmed down by the NanoBSD tool + all the FreeNAS specific bits.

 

[jgreco]

This is how it works: NanoBSD does not "contain" any FreeBSD source code. NanoBSD is a tool (it's basically just few shell scripts) that will help you create a pruned down FreeBSD system image, but you need to supply your own FreeBSD source tree.

I am actually aware of what NanoBSD is. I was the original author of XKERNEL, the first framework to build a minimalist FreeBSD-on-a-floppy, which Andrzej took, generalized, and made into PicoBSD. Someone else started from scratch for NanoBSD, but I think I can reasonably say it was inspired by the examples of XKERNEL and PicoBSD. So I understand how that sort of thing works.

You've now just indicated that the sources are derived from TrueOS. Fair enough. Then the question becomes, what's the delta between TrueOS and FreeBSD?

Basically I don't understand exactly what to expect in the way of differences between what's in the FreeBSD tree (both in terms of features and code) and what winds up in FreeNAS. Right now, for example, newnfs is not enabled in FreeNAS. Or the various ZFS differences. It seemed for a while that they were trying to map FreeNAS release numbers to more or less match FreeBSD. But overall it is kind of opaque, and rather than just diff'ing code trees and trying to read tea leaves, it seems to make sense to just ask the decisionmakers.