64 bit only FreeNAS distributions?

[jkh]

Hi folks,

So, since the dawn of time FreeNAS has been released for both i386 and x86_64 architectures, and as much as this may have seemed "painless", it really hasn't been.

For each distribution we also require 64/32 bit PBIs, you see, and in FreeNAS trunk you'll notice that support for jail templates has also been added (so that custom jails can be supported, including linux jails). Each FreeBSD jail template must also be, of course, either 32 or 64 bit since you can't run a 64 bit jail on a 32 bit host.

It's becoming somewhat unsustainable from a release engineering perspective, and some plugins (like Plex) don't even support 32 bit environments because the upstream binary providers (in this case, Plex) are only providing 64 bit versions of their stuff.

ZFS is also memory hungry, as many folks in the forums have noted, and a 32 bit address limit really penalizes it here. So, to cut to the point: I would like to see what people think of the idea of going 64 bit only for future FreeNAS distributions. Folks with older hardware can always stick with FreeNAS 9.1.1, of course, as we're not going to pull the older distribution media or anything (and we can even commit to keeping the 32 bit plugins we built for 9.1.1 around for awhile), but it sure would make our lives easier to be able to just generate x86_64 binaries going forward!

Comments? Thanks!

 

[cyberjock]

I think that is a great idea. Deprecate the old 32-bit stuff, but leave it around for those that want to continue using it. Offer the 64-bit for those that have modern hardware.

While we're on the topic, maybe consider setting a RAM limit to boot to stop people from using FreeNAS with significantly less RAM than is generally deemed safe. Note that I don't think it should prevent installation of FreeNAS onto a USB stick, but the USB stick would boot up and stop with a warning if you have less than the required RAM.

I'd recommend 8GB, but would be willing to go as low as 6GB of RAM. Strictly based on forum users though, 8GB of RAM seems to be the point at which there isn't significant reliability issues for FreeNAS(although people with large systems will still have performance issues due to short RAM).

Both setting the RAM limit on bootup as well as forcing 64bit would (in my opinion) go a long way to getting people to not do either of those very bad decisions. Of course, the forum will still instead fill up with:

1. People that want to use 9.1.1 forever and complain about their hardware not being supported.
2. People that refuse to go to 8GB of RAM and will happily complain about the "overly high requirements".
3. People that refuse to go to 64 bit CPUs and will happily complain about the "overly high requirements".

Another option is to simply remove the ZFS/jails/plugins options from the 32 bit version so they can have updates and still use UFS. Seems a small subset of people still use UFS.

Just a thought, and I know several people are going to disagree with this idea. I welcome both comments and criticism on the idea.

My vote is set a 6GB(or 8GB of RAM) boot up limit and release only a 64 bit version.

PS. Could have used a poll for this thread. :)

 

[jkh]

While we're on the topic, maybe consider setting a RAM limit to boot to stop people from using FreeNAS with significantly less RAM than is generally deemed safe.

Yep, there's a bug tracking this but I can't find it at the moment. There are basically two "minimum RAM" requirements. The first is, I believe, somewhere around 1GB to interact with every single option in the GUI before any swap is configured or the kernel will shoot django in the head. I've heard apocryphal stories of folks getting the swapless GUI to work in 512MB, but I've never managed it. The second is, as you point out, ZFS itself. We've been going around and around on the right value for this since some BIOSes steal memory for nefarious purposes like video memory and some don't, and it's hard to account for "RAM stealing" to the point where you know that a 4GB system *actually* has 4GB available or just reports that. Nonetheless, we agree, simply attempting to use ZFS on a low-memory configuration system should pop up a warning of some sort.

PS. Could have used a poll for this thread. :)

What's a poll? Seriously, I didn't even know you could do that. :) Is that something you'd be willing to run, perhaps?

Thanks!

 

[cyberjock]

Yep, there's a bug tracking this but I can't find it at the moment. There are basically two "minimum RAM" requirements. The first is, I believe, somewhere around 1GB to interact with every single option in the GUI before any swap is configured or the kernel will shoot django in the head. I've heard apocryphal stories of folks getting the swapless GUI to work in 512MB, but I've never managed it. The second is, as you point out, ZFS itself. We've been going around and around on the right value for this since some BIOSes steal memory for nefarious purposes like video memory and some don't, and it's hard to account for "RAM stealing" to the point where you know that a 4GB system *actually* has 4GB available or just reports that. Nonetheless, we agree, simply attempting to use ZFS on a low-memory configuration system should pop up a warning of some sort.

I'd think that RAM stealing is a non-issue. If I have a video card that is hogging 4GB of RAM and I have 8GB installed, I really only have 4GB of RAM to play with. Obviously, that's not enough for FreeNAS. So a limit of 7.5GB would cover most people that have onboard video and other small things.

Or is it a matter of detecting how much is actually available to the OS?

What's a poll? Seriously, I didn't even know you could do that. :) Is that something you'd be willing to run, perhaps?

Thanks!

When you create the thread you have the option of creating a poll. Now that the thread is created I don't think you can add a poll. I know I can't with my permissions...

 

[diedrichg]

64bit should truly be the only option. And why would 32bit be offered anyways since 64bit is backward compatible anyways?

 

[jgreco]

It has really only been in this year we've finally seen a larger number of users being willing to commit more significant resources to their NAS.

An i386 system might be able to use more than 4GB (thinking: PAE) but the real problem is ZFS pigginess. Since the kernel address space is still limited, PAE's not as helpful as one would like it to be, and of course PAE is kind of dodgy for compatibility with random end-user systems.

I don't think I'm a fan of dropping i386 entirely. Generally speaking, it is extremely useful in various contexts, ranging from being able to repurpose an older machine, to being more practical as a VM. The resource penalty for amd64 is substantial especially in a virtualization environment. I've had FreeNAS i386 with ZFS running on a 1GB (no not swapless and also not exercising the GUI) VM providing iSCSI services, took a little bit of convincing and I don't think I'd do it with precious data. But the E3 platform still has a 32GB limit, and someone who wants to experiment with a VM that demands 8GB - fully a quarter of that - just to run, that strikes me as bad.

I keep hearing iX people saying "yay ZFS on i386" but I think that the consensus of those of us working the forums over the past two-plus years are that this is pretty much crap. It doesn't work, at least not reliably and not without some tuning to tame the pigginess. It'd be great to see some tuning done to make ZFS on i386 an actual practical reality, but I expect the compromise would be to simply disable ZFS and go UFS only. But of course, problems there. What do you do for the people who upgrade their box. Leave ZFS code in the kernel but strip it out of FreeNAS? Just disable the creation of new ZFS pools in the GUI? I don't see any happy solutions, except perhaps to dump more time into i386 to find reasonable ZFS defaults for it. I can understand the reluctance to do this though.

Looking at another iX product, PC-BSD, dropping i386 has been pretty much a train wreck. Those of us who had been repurposing older PC's for PC-BSD essentially just found machines suddenly hit EOL. Very unpleasant and ugly. Not a good feeling.

Seriously, if you plan to go that route, it might make more sense to make the transition with a major version bump, like FreeNAS 10. That gives people some warning, allows a few more months of time for old 32-bit hardware to commit seppuku on its own, etc.

 

[cyberjock]

64bit should truly be the only option. And why would 32bit be offered anyways since 64bit is backward compatible anyways?

You got that backwards. 32bit OSes will run on 64bit hardware. The whole reason we're discussing this is that if we get rid of the 32-bit OS then all of the 32 bit only hardware people have will not be usable with FreeNAS.

 

[Daisuke]

Ditch the 32bits packages and free yourself from the developing burden OR keep it to allow probably 2% of global users running FreeNAS on an old laptop? I don't know any admin running a server on 32bits OS. Yet, every distro offers it...

 

[jgreco]

I don't know any admin running a server on 32bits OS. Yet, every distro offers it...

You say that generally enough that I'm going to take it as "distros" other than FreeNAS.

Lots of admins run 32 bit servers. Walk into an environment with virtualization, and any experienced admin is likely to be doing 32 anywhere possible. It reduces footprint. Why run an image that eats 20% more disk and needs more RAM?

I'm reasonably sure that MAYBE only 10% of our VM's are 64.

It's kind of funny. We spent years complaining about the hard limits of 32 bit systems. Virtualization makes it easy to break up what used to be one bigger server into several smaller ones.

 

[Fran Aquino]

I find it reasonable to go 64 bit only.
That boot time memory limit thing looks like an awful idea, at least for me. I'm used to install first inside a VM, to test the UI etc. and I'm sure a lot of people does the same ¿6GB? Overkill. Just put alerts in the UI stating you shouldn't be using it for production if less than X Gigs.

 

[jkh]

Lots of good comments in this thread - thanks (seriously). That was kind of the point of raising it here.

Just some general observations, however:

1. I think the NAS appliance market and the desktop market are rather different. I can totally see pulling an old PC out of a closet to repurpose as a desktop because all I want to do is run Firefox and Thunderbird on it, or something suitably limited in scope (OK, so I am obviously a Mac user, but I can see this happening "in the abstract" :) ) but for a NAS, that's really not the best idea in the world and we have the crashdumps / UI tracebacks to prove it (and Cyberjock will readily agree that your hardware should be at least >this< tall to ride the NAS rollercoaster). A NAS is a fileserver. Its purpose in life is to get beaten on, and if you don't need that level of service then why not simply keep your data on the client and eliminate the extra moving parts and power cost of a server box at all?

2. The point above also speaks fairly strongly against running a NAS in a VM for any length of time. Sure, for testing purposes it's fine, and I run a lot of FreeNAS VMs for the purpose of hacking UI features or testing new stuff, but I'd never do that in production because VMWare is essentially abstracting away all of FreeNAS's (and specifically ZFS's) ability to provide fault tolerance and performance. It would be a bit like converting a top fuel dragster to run on Ethanol for the fuel savings. Kind of missing the point. :)

So, in short, while there may be other reasons to keep the i386 builds around, I have a hard time seeing those two justifications as being particularly strong ones...

 

[jgreco]

There's actually a nice guide around here somewhere that someone wrote on how to successfully virtualize FreeNAS.

Quite frankly there's a lot of attractiveness in an E3-1230/32GB box that runs FreeNAS on 4 drives AND simultaneously runs other VM's. At less than 100 watts.

Or this nifty 12-core beast with 128GB, 12 drives, and 200-350 watts, load-dependent.

But to turn this around and stab your former employer a bit... not all NAS users beat on their servers, but when Apple no longer supports ZFS, what's a poor Mac user to do? ;-) Point is, there are lots of use cases that maybe don't involve doing buildworld over NFS on your NAS hourly but where a NAS is still very convenient. I think the elimination of i386 would wipe out access to FreeNAS for a certain base of users who are currently choosing to run it rather than buying the small NAS appliances that have been popping up based on SoC and Atom solutions.

 

[cyberjock]

Why not just offer a stripped down version of FreeNAS that's 32-bit? No jails function and only UFS for the i386 build?

 

[paleoN]

Why not just offer a stripped down version of FreeNAS that's 32-bit?

Actually, this is what I thinking. No plugins, linux jails, etc, but ideally keep the basic FreeBSD jail available.

It'd be great to see some tuning done to make ZFS on i386 an actual practical reality, but I expect the compromise would be to simply disable ZFS and go UFS only. But of course, problems there. What do you do for the people who upgrade their box. Leave ZFS code in the kernel but strip it out of FreeNAS? Just disable the creation of new ZFS pools in the GUI? I don't see any happy solutions, except perhaps to dump more time into i386 to find reasonable ZFS defaults for it.

Couldn't have said it better myself.

 

[cyberjock]

I was thinking of yanking out the jail with the 32-bit version. As it is some of the jail functions already call out for ZFS only. And if the jail exists newbies won't recognize the difference between the jail and the plugins, so they'll be posting how to make it work in the 32 bit world when it was deliberately removed. I had to look it up to understand the difference between the jail and the plugins. :P

 

[HolyK]

I am for both x64 and memory limits. I agree with Cyberjock.

When you create the thread you have the option of creating a poll. Now that the thread is created I don't think you can add a poll. I know I can't with my permissions...

No, you can't do that. Another "amazing XenForo feature" .. hell with that, there is a workaround so why would someone bother to implement this feature?

 

[Savell Martin]

64bit only...

Get rid of 32bit.
Even older hardware can run 64bit for the last few years.

 

[jgreco]

64bit only...

Get rid of 32bit.
Even older hardware can run 64bit for the last few years.

That's great, but doesn't add much to the discussion. With the Opteron having burst onto the scene in 2003, it is more than just "the last few years." However, the 64-bit bandwagon was a bit laggy, with Intel's Core offerings in 2007 (Yonah) still being only 32-bit. AMD was a little quicker to supporting 64-bit, of course. In some cases there are even platforms that went both ways, such as LGA775, which started with 32-bit Intel Pentium 4's, and moved on upwards from there.

Problem is, a lot of people seem to get into FreeNAS through repurposing, which is hardly ideal, but still the way it goes. Requiring people to use a 64-bit platform is a significant barrier to getting the foot in the door, and they'll end up with OpenFiler or something like that instead.

 

[pirateghost]

That's great, but doesn't add much to the discussion. With the Opteron having burst onto the scene in 2003, it is more than just "the last few years." However, the 64-bit bandwagon was a bit laggy, with Intel's Core offerings in 2007 (Yonah) still being only 32-bit. AMD was a little quicker to supporting 64-bit, of course. In some cases there are even platforms that went both ways, such as LGA775, which started with 32-bit Intel Pentium 4's, and moved on upwards from there.

Problem is, a lot of people seem to get into FreeNAS through repurposing, which is hardly ideal, but still the way it goes. Requiring people to use a 64-bit platform is a significant barrier to getting the foot in the door, and they'll end up with OpenFiler or something like that instead.

I wouldn't wish open filer upon anyone.

 

[cyberjock]

That's great, but doesn't add much to the discussion. With the Opteron having burst onto the scene in 2003, it is more than just "the last few years." However, the 64-bit bandwagon was a bit laggy, with Intel's Core offerings in 2007 (Yonah) still being only 32-bit. AMD was a little quicker to supporting 64-bit, of course. In some cases there are even platforms that went both ways, such as LGA775, which started with 32-bit Intel Pentium 4's, and moved on upwards from there.

Yonah was a laptop chip, and it was released in January 2006. I had one of those laptops in February 2006 and I loved it! I don't think that's a fair comparison though.

Some of the Prescott Pentium 4s had 64 bit support, but that goes back to 2004. Not sure exactly when the first P4 with EMT64 hit the marke t though

But, we'd all agree that a even a 2007 desktop machine is not capable as function as a FreeNAS ZFS server(mostly because of the limited RAM size). Maybe UFS though.