How much RAM is needed for 4x2TB RAID-Z 5?

[HolyK]

How much RAM is needed for 4x2TB RAID-Z ?

Hello,

i would like to ask more experienced people, how many RAM is required for ZFS RAIDZ created from 4x2TB WD Green.

I'm thinking about following HW, but i'm not sure, if the 8GB is enough (MB limit).

MB/CPU/GPU: ASUS E35M1-M PRO - AMD Hudson M1, AMD Zacate E-350, VGA AMD Radeon HD6310
RAM: Zeppelin Gold - Evolve 8GB (kit 2x 4GB) 1333MHz
PSU: Seasonic S12II-330 330W Bronze - i think this is enough for four discs and Zecate MB.
Case: Eurocase MC 41

Any ideas?

 

[Brand]

I read on one of the Sun, now Oracle, Solaris pages that ZFS works best with 1GB of RAM for every 1TB of storage. That was included using some of the advanced ZFS features including deduplication. Right now I have 10 x 2TB with 4GB of ECC RAM but plan to at leat double it as soon as possible. Also the ZFS file system is not capable of RAID5 array, you probably mean RAIDZ which is similar.

 

[HolyK]

Yep, i mean RAIDZ, anyway thanks for the info, so 8GB should do the work properly :)

BTW: What version of WD Green is "better" to buy ? Classic 512B/sector or new "Advanced format"

WD AV-GP WD20EURS 3.5" 2TB - 512B/sector
WD Caviar Green WD20EARS 3.5" 2TB - 4kB/sector aka "Advanced format" (and a bit cheaper)

Any performance boost / issues?

Thanks

 

[esamett]

2gb RAN OK WITH 3 DRIVES, BOGGED DOWN WITH 8 DRIVES. 4GB IMPROVED PERFORMANCE. OTHERS HAVE POSTED THAT 8GB HAS A GOOD RESULT FROM 4GB.

 

[HolyK]

Thanks ;)

Now i need to choose right disk :p

 

[esamett]

I am using the Hitachi slow 2tb drives. they use native 512b sector size rather than 4k sector size with emulation. while future builds will support large sector size there is an issue about the ideal number of drives in a zfs array that I think the 512b disks avoid. price points on newegg range from $80-60 depending on sales, coupons and rebates.

 

[jfr2006]

Hi:

I'm building my freenas around the same platform, kust some minor differences (not the asus pro version motherboard, an extra SATA controller, 4GB RAM).

I will be using some drives i have and buying some more new ones. All will be WD20EACS and Samsung HD204UI. I will be creating 2 volumes: one with 5 or 4 discs and a second one with 4 discs.
My power supply will be a Be Quiet! Pure Power 300W.

If you build your first (i'm still waiting for some components), let me know how it behaves.

Regards.

 

[HolyK]

Online shop where i want to buy all components don't have memory modules on stock, so i need to wait too ^^

 

[HolyK]

Hmmm, i googled a bit and i foud information about 4 discs arent recomanded for RAIDZ ... only 3 or 5 (...), but i can't find "WHY" ? Is there a performance issue so i should get 5 discs? Or?

Thanks

 

[torrin]

Hmmm, i googled a bit and i foud information about 4 discs arent recomanded for RAIDZ ... only 3 or 5 (...), but i can't find "WHY" ? Is there a performance issue so i should get 5 discs? Or?

Thanks

It should not matter that much, but 5 drives is more efficient than 4 in RAIDZ. Read up on how raid5 works, and it will help you understand why. 3 drives doing RAIDz is sort of a waste IMHO, please you have less drive heads to do the work. I am running 6 drives in a RAIDZ2 and it performs well, though not as well as a 5 drive RAIDz.

 

[HolyK]

I know how RAID5 works, but obviously the bigger difference is in 4k/sector discs. Anyway ...

128 divided by (total_nr_disks - parity_disks); so do not count the parity disks.

That means that for 4K disks the following combinations are optimal:
RAID-Z: 2, 3, 5, 9 or 17 disks
RAID-Z2: 3, 4, 6, 10 or 18 disks (anything under 6 disks produces low results for another reason, though)

If you want a final list of optimal configurations, use this one:
RAID-Z: 5 disks
RAID-Z2: 6 disks or 10 disks

SOURCE

So ok, ill get 5x WD Caviar Green WD20EARS 3.5" 2TB :D

 

[torrin]

I know how RAID5 works, but obviously the bigger difference is in 4k/sector discs. Anyway ...





SOURCE

So ok, ill get 5x WD Caviar Green WD20EARS 3.5" 2TB :D

If it is a RaidZ, then yup, that would be optimal, if you want RaidZ2, 6 drives would be optimal. Remember, if you are running RaidZ and you lose on drive, you will be running degraded, and another crash while trying to rebuild the first failed drive kills all your data.

 

[jfr2006]

Can someone explain me why 4 drives is not recommended ?

 

[esamett]

reporting the work of others, the number of disks matters if they use the "advanced format" 4k sector size. "old style" 512b sector size disks are said to be relatively immune to the problem. Currently Hitachi 2TB drives are 512b sector size so you should be able to use any number of drives you want. The theoretical issue is if you want to "future proof" your array.

 

[Artur Meinild]

I have the same setup: 4 x 2TB WD green in Raid-Z. I use 4 GB ram, and it is satisfactory. So 8 GB should definitely be enough.

 

[jfr2006]

What do you consider satisfactory? What are the transfer rates you have and the cpu usage?

I still did not get wtaht is the problem with 4 disks that have 4KB sector..Is it a performance problem? If so, why? Is it a safety problem?

 

[HolyK]

... and another crash while trying to rebuild the first failed drive kills all your data.

That's why i want to buy each disc from different series, so the "chance" that two or more disc fail at "same" time is minimized (i think :D)

 

[esamett]

i have read that the number of drives is only really relevant if you use "advanced format" 4ksector size drives. true 512b sector size disks such as the hitachi essentially avoid this problem. there is an issue about divisibility by 128 involved. hope this helps.

 

[jwhitt]

Where were you seeing this information. And is there any limitation on a raidz with 512b Drives, ideal numbers? Im currently running with 8 1TB in raidz? the bottleneck that im currently having is that Im using 6 on board sata 2's, a pci 2 port sata 2, a pci nic for iscsi traffic, and the on board nic for LAN/SMB/NFS traffic.... that will b changing soon, however that requires a overhaul of the mobo proc, a pci-express sata 3.0 card and pci express nics - IE expensive.

 

[ixdwhite]

The RAIDZ sizing data comes from the ZFS Best Practices page from the Solaris Internals wiki (a page you should keep bookmarked AND under your pillow):

http://www.solarisinternals.com/wik...onfiguration_Requirements_and_Recommendations

The recommendation there is copied here:

A RAIDZ configuration with N disks of size X with P parity disks can hold approximately (N-P)*X bytes and can withstand P device(s) failing before data integrity is compromised.

• Start a single-parity RAIDZ (raidz) configuration at 3 disks (2+1)
• Start a double-parity RAIDZ (raidz2) configuration at 5 disks (3+2)
• Start a triple-parity RAIDZ (raidz3) configuration at 8 disks (5+3)
• (N+P) with P = 1 (raidz), 2 (raidz2), or 3 (raidz3) and N equals 2, 4, or 8
• The recommended number of disks per group is between 3 and 9. If you have more disks, use multiple groups.

The odd sizing has to do with how RAIDZ distributes blocks across disks ("columns" in RAIDZ parlance). If the number of columns is not a power of 2 then ZFS generates "fake" I/Os to fill the stripe so there is I/O going to every column. RAIDZ is like RAID3 in that it uses sector-sized stripes (512 bytes for normal disks) so reading and writing less than full stripes causes contention and thus performance problems. This characteristic is why ZFS wants to have lots of RAM to use for RAIDZ readahead; otherwise, RAIDZ performance is utterly abysmal.

As far as I know there aren't requirements for RAM vs array size though there are guidelines based on some rough assumptions about working set size vs. array size. If you don't tend to access the same data then any amount of cache will do since it will never cache anything of value (except maybe a few top level directories).

(There are requirements for total RAM due to ARC being a hungry pig, though that is a different problem.)