Greater than 4GB on 32-Bit

[gzartman]

I have a RAID box that has dual 2.67ghz xeons, 12GB of RAM, and 12x2GB SATA drives hooked to an Areca RAID card. I was running a flavor of linux on it and using the built in raid function on the card, but I wanted to switch to freenas for the zfs. However, the 32-bit version of the distro doesn't see beyond 4GB. With linux, you can use the PAE kernel and see all the memory on a 32-bit install. Is this at all possible on FreeNAS?

Thanks.

 

[cyberjock]

Nope. You can't do PAE with FreeNAS in that way. Go 64bit for >4GB of RAM as that's your only option. :(

 

[gzartman]

That's my point. It doest seem the 64bit version will install on my box, correct?

I couldn't get it to work.

Seems I might be SOL for freenas on this box

 

[cyberjock]

That's my point. It doest seem the 64bit version will install on my box, correct?

I couldn't get it to work.

Seems I might be SOL for freenas on this box

I don't know. Your answer of "dual 276ghz xeons" is way too vague. You'd need to look up your motherboard model and CPU model to figure out if it supports 64-bit.

 

[jgreco]

FreeBSD also supports PAE, but I do not believe that it is compiled into the FreeNAS kernel. In any case, PAE has limited benefits - the kernel memory map is still a fixed, sub-4GB size, and that's where you really need the memory to be with ZFS. For running a large web server with lots of large independent processes, PAE is a win. For FreeNAS and ZFS, not so much.

I would think that most Xeons capable of supporting 12GB are also likely to have 64-bit support, unless the box is at least 10 years old.

 

[gzartman]

I think the age might be the case here. This is the MB: http://www.supermicro.com/products/motherboard/Xeon/E7501/X5DMS-8GM.cfm

I'm pretty sure now that it has the 32 bit Xeons in it. If I recall correctly, Intel lagged quite a bit in moving to a 64-bit server processor. Th AMD had the Opterons well before Intel created the 64-bit xeon.

Looks like I'm headed back to linux.

 

[jgreco]

E7501? Out of luck I think...

 

[gzartman]

Yep, that's the chipset.

I don't understand the PAE limitation of the freenas kernel though. 32-bit kernel on linux works fine up to 32GB. I'm running the ZFS for linux modules on a Centos 5 32-bit install and it works great.

 

[jgreco]

It works fine on FreeBSD too, and you can use 32GB. But PAE never lets you have an address space larger than 4GB per process. Your Linux box's ZFS is probably just living within that 4GB.

What do your ARC stats look like on the Linux box?

 

[gzartman]

I don't have the arc_summary app on my box, just arcstat.

Here is what I see with a 1 second refresh interval:

[root@bacula arcstat]# python arcstat.py 1
time read miss miss% dmis dm% pmis pm% mmis mm% arcsz c
11:38:59 0 0 0 0 0 0 0 0 0 405M 421M
11:39:00 0 0 0 0 0 0 0 0 0 405M 421M

 

[jgreco]

Ok, so, actually it is using a HELL of a lot less than 4GB. I'm guessing the kernel/user split is about 2G/2G, but less than 500MB is still tiny.

By comparison a typical 32GB FreeNAS system:

~ % arcstat.py 1
time read miss miss% dmis dm% pmis pm% mmis mm% arcsz c
12:54:19 0 0 0 0 0 0 0 0 0 17G 17G
12:54:20 0 0 0 0 0 0 0 0 0 17G 17G
12:54:21 0 0 0 0 0 0 0 0 0 17G 17G

This system is doing NOTHING because it's long-term archival yet it still manages to be using 17G.

So your ZFS on Linux isn't using ... well, basically, any of it (405M being substantially closer to 0M than 12288M). PAE may be helping your userland, but basically with PAE you could probably have 3GB KVA and be using half it for ARC. So I tend to think whatever is going on there is crappy at best.

 

[gzartman]

The zfs pool is basically a backup of a backup. I'm backing up my various vms to a bacula virtual tape library located on hardware raid. The VTL is then mirrored over to the zfs pool. More or less this is just a non-critical test deployment in a production envir. However, read/write performance is just fine in this setup. This would likley change significantly if I were trying to use zfs in place of my hardware raid device.

 

[jgreco]

Your question was implying that your Linux ZFS on a PAE Linux system was able to make use of your 12GB. It doesn't. ZFS on FreeBSD works just like this too. I was mainly interested in the validity of what you were saying. Theoretically, one could create a system on i386 with PAE that allowed multiple address spaces (multiple processes) to access a larger amount of space. I didn't think anyone had bothered to implement such a thing for Linux ZFS though, because basically it's cheaper and easier just to replace an old server board that is i386-only than it is to invest the time writing such complicated code.

But I also wanted clarification so that I could offer you some guidance: even if you had a FreeNAS i386 kernel with PAE, it would require some tuning to make it work correctly. The problem we've noted with FreeNAS is that it comes with a kernel that appears to be tuned for correct operation on larger systems (32+GB) but this appears to stress out small-memory systems. Here in the forums, we simply advise people to go with 8GB and amd64 and problems with random panics go away. It is possible to run ZFS on i386 with 4GB, but it takes work. Further, the dev team is looking to kill off the i386 version in the next year. So it is possible that FreeNAS is "not for you" if you must continue to use the i386 platform.

On the flip side, please do consider that ZFS makes totally awesome use of large amounts of memory... as long as it can address it (heh). A modern Xeon E3-1230 system with 32GB would provide a huge performance jump over that poor little PAE system.

 

[gzartman]

Thanks Jgreco for the detailed explanation. Clarifies several things for me. The box in question has a pretty decent Areca RAID card in it, so perhaps it's better to stick with the hardware RAID until I decide to chuck the box. It is just my home machine at present and stores my avi library, personal files, and an offsite copy of my office virtual tape library.

I've just had my eye on FreeNAS for awhile now and have been waiting for the opportunity to deploy it in some fashion. Perhaps on another box.

Greg

 

[jgreco]

We're here to help.