What's the real MINIMUM for running ZFS?

[freshfeesh]

I'm mainly interested in the minimum RAM. I did read the user guide. It says "typically requires a minimum of 6GB". If 6GB was the true min, I'd be done, but it says "typically". Then it goes on to say that with less than 4GB, prefetching is disabled. It doesn't say that with less than 6GB, ZFS doesn't work. Prefetching would be great, but it's not a necessity. I don't need to saturate two aggregated gigabit pipes all day long, I want to reuse an old AMD DDR setup with some old and a couple new drives to backup my family's precious music and photos in the most reliable way possible (using this nas box and a couple other modes of backup). I love the inherent file integrity that ZFS brings to the picture, and I'm willing to let performance suffer a bit compared to a UFS system to get that. Secondarily, I'd like to stream music an movies when that becomes available with Freenas v8.

I think the processor is an Athlon 64 X2 3500+ dual core, and there's 1GB of DDR ram. Getting 4GB of ram in there would basically cost $100, at which point I may as well spend $150 and get a new proc, mobo, and 8GB of DDR3, at which point I'd really be better off spending $250 to upgrade my oldest PC, which is unfortunately also a DDR machine. This all just defeats the purpose of reusing my old stuff for my nas box. Buying 2GB of DDR I may be willing to do if it makes a big difference for ZFS. If the new ram is stable with the old ram, I'd leave those old sticks in and have 3GB. I don't expect much from the performance, but I think I'd be disappointed with less than 35GB/s.

I looked around for this already being discussed and didn't see it. I'd happily read some other threads if someone points me in the right direction.

Edit: I posted here based on the subheading of the forum, but I think this really belongs in "Hardware". I'll repost there.

 

[cubix]

I'm not able to give you the answer you are looking for, however, the larger the storage, the more ram your system will want. Rule of thumb is one GB of ram for every TB of space for ZFS.

A few people have run into issues running 4gb or less.

In an effort to utilise spare hardware, don't write off UFS too quickly.

 

[Durkatlon]

On i386 builds, ZFS gets no more than about 512 MB. You need to tweak the kernel vars a bit but this can certainly be tuned to run quite stable.

 

[freshfeesh]

Thanks. I reposted in the Hardware forum, but wanted to give a quick reply. You say "issues", can you give specifics? Lowish performance I can live with. Instability and snails pace transfers I wouldn't want to deal with. UFS wouldn't be a bad option for me, it just sounds second best.

 

[cubix]

Errors and timeouts - especially during large operations. Here's some light reading http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide#System.2FMemory.2FSwap_Space

 

[freshfeesh]

I've seen a few threads where people ask if their old systems like mine will work with ZFS. The standard response (and I understand why) is to either use UFS or upgrade the hardware. If anyone went ahead and ran ZFS with 1GB of ram anyway, I'd love to hear your tips and experiences, especially kernal var settings like Durkathlon mentions. The link that Cubix provided says "One GB or more of memory is recommended", so I feel like I'm in OK shape to jump in and try it (although I didn't make it much further down the page before coming back to post). Can anyone recommend tools/procedures to benchmark performance and stress test it for stability? Just throw a huge transfer at it from one of my Windows machines? Extra karma points if I don't have to use SSH or the CLI.

 

[Durkatlon]

Are you planning to run amd (i.e. 64-bit) or i386 (i.e. 32-bit) ? For 64-bit I don't think you have to optimize much, since the kernel can use a lot more of the RAM than it can out of the box on 32-bit. I would recommend setting vfs.zfs.arc_max to a very small value (like 30M or even less). Other than that you'll probably be OK. Then to test, just do a lot of sustained reading and writing.

If you need to run i386, you'll run into the unfortunate situation right now on FreeNAS8 that KVA_PAGES is set to 256 which means that the maximum kernel RAM is 1GB, no matter how much RAM you put into the system. Unless you want to recompile and repackage the entire system with a modified kernel configuration, I would recommend going to FreeNAS7 where KVA_PAGES is set to 512 by default, meaning the kernel can take 2GB of RAM if you supply it with sufficient RAM (more than 2GB of course since userspace wants RAM too).

 

[freshfeesh]

I'm running AMD64, although with only 1GB RAM physically installed anyway. I'll make the tweak you suggested and start hammering away! I'll report back on how it goes. Thanks.

 

[Digidoc]

Prior to me upgrading my FreeNAS box yesterday, I was running the following with very good results:

-Asus A8N32-SLI Deluxe motherboard
-4GB DDR RAM (maxing out the board)
-Opteron 185 (basically the server version of the X2 4800+ s939 processor)
-Seven 2TB HDD's in RAIDZ2.
-SilImage 3112 based two-port SATA card (I was using the four nforce SATA ports, the internal Marvell SATA port, and the two ports on the 3112 based card)
-Intel EXPI9400PT Server NIC card
-580W PSU
(I also had an old nvidia 7800 based video card in it for setup/diagnostic purposes too).

I am also using FreeNAS 8.0.2 AMD64.

It ran remarkably well, but one thing that was really annoying was that it would sometimes "stutter" when opening directories. After they opened though it ran fine. Keep in mind that I am using this server to store almost two terabytes of photos in Camera RAW format, plus the edits I've done in Photoshop .PSD format, with a few TIFF and JPG files thrown in for good measure (and not including acting as a storage repository for my software installation files too). It's also a "low volume" environment; meaning that it's only me accessing the server.

In short, you should be able to run what you're looking to do with the hardware you have. It doesn't sound like you're trying to put this in a high volume environment so you should be able to get away with what you already have on-hand.

One thing I do suggest though is that you do the following in your system BIOS to maximize performance:

- Turn OFF Cool n' Quiet
- Turn OFF Spread Spectrum
- DO NOT overclock the CPU or the RAM.
- (on newer AMD systems, turn OFF turbo-core)

When I upgraded the system over the weekend, I went to a MSI NF980 board w/a Phenom-II X6 1090T CPU, and 16GB of RAM (I migrated the HDD's, NIC, and SilImage controller over from the previous FreeNAS8 build). At first everything was working fine but I noticed that I still had the same "hiccuping" going on when accessing folders. I ended up having to reset my BIOS (this board used to be in my main PC) to resolve a BIOS setting mistake I made. After that the system was incredibly unstable; it'd lock up for no apparent reason. In my own case I swapped out the RAM (I had DDR3-1600 RAM in it initially so I swapped that with the 16GB of DDR3-1333 RAM I had in my new desktop system, so now the desktop's got the faster RAM which works out better really). I also discovered that those settings had reverted back to defaults when I had reset the BIOS earlier. I reset everything, let the system pick default timings for the RAM, and it ran all night without hanging. It also now doesn't hiccup when accessing network shares! Go me! :D

[edit]I just discovered about an hour ago that it may have just been the RAM causing the problems I was experiencing. Even though the RAM ran flawlessly in my system for over a year, apparently resetting the memory timings to defaults caused it to go haywire. When i put the RAM into my other desktop initially it worked fine, but it started to blue-screen after a while. I ended up having to decrease the speed on the RAM to DDR3-1333 speeds to get it stable. Oh well... :/ ).[/edit]

To sum it up, yes prefetching was disabled in my old build with only 4GB of RAM, but I was able to use ZFS/RAIDZ2 without any issues really. It ran reasonably well (typical write speeds were a little more than 50MB/sec) over my gigabit network too. The new system w/the six core processor and 16GB of RAM completely annihilates it though (I was seeing 117MB/sec transfer rates in some instances).

I'm mainly interested in the minimum RAM. I did read the user guide. It says "typically requires a minimum of 6GB". If 6GB was the true min, I'd be done, but it says "typically". Then it goes on to say that with less than 4GB, prefetching is disabled. It doesn't say that with less than 6GB, ZFS doesn't work. Prefetching would be great, but it's not a necessity. I don't need to saturate two aggregated gigabit pipes all day long, I want to reuse an old AMD DDR setup with some old and a couple new drives to backup my family's precious music and photos in the most reliable way possible (using this nas box and a couple other modes of backup). I love the inherent file integrity that ZFS brings to the picture, and I'm willing to let performance suffer a bit compared to a UFS system to get that. Secondarily, I'd like to stream music an movies when that becomes available with Freenas v8.

I think the processor is an Athlon 64 X2 3500+ dual core, and there's 1GB of DDR ram. Getting 4GB of ram in there would basically cost $100, at which point I may as well spend $150 and get a new proc, mobo, and 8GB of DDR3, at which point I'd really be better off spending $250 to upgrade my oldest PC, which is unfortunately also a DDR machine. This all just defeats the purpose of reusing my old stuff for my nas box. Buying 2GB of DDR I may be willing to do if it makes a big difference for ZFS. If the new ram is stable with the old ram, I'd leave those old sticks in and have 3GB. I don't expect much from the performance, but I think I'd be disappointed with less than 35GB/s.

I looked around for this already being discussed and didn't see it. I'd happily read some other threads if someone points me in the right direction.

Edit: I posted here based on the subheading of the forum, but I think this really belongs in "Hardware". I'll repost there.