New build questions

[ixidor]

I have a server that "fell off of the back of the truck", as in was free to me, was basically brand new. It has a
SuperMicro mobo, X8DTH-iF.cfm, and a 12 bay chassis, and a M105 flashed to IT mode.

1. is there something inherently wrong with doing a 6 drive raidz2? as opposed to say something 2^n like 4 or 8 drives?

2. if i have 16TB space16gb ram move up to 32tb space 32gb ram, is there a reason i would want/need more? do not plan to use dedupe.

3. anyone have any idea how much more power just the cpu (s) + mobo would draw over say a smaller mobo + i3 over course of a year?

4. the WD red 4tb drives look tasty. however seems both newegg and amazon are not great places to get them right now. Newegg says they have firmware v2, but may get version3. amazon has this note " While this item is available from other marketplace sellers on this page, it is not currently offered by Amazon.com because customers have told us there may be something wrong with our inventory of the item, the way we are shipping it, or the way it's described here. (Thanks for the tip!)" is it worth waiting until can get a reliable source of version 3? or is it worth it to pay ~$85 more per drive go go for the enterprise drive ( and 2 more years of warranty, and no fuss over # drives in chassis) ?

5. suggestions for use of a ssd drive for slog, l2arc so on.

 

[BigDave]

1. No, a six drive RAIDz2 pool is what I have and afaik there's nothing wrong with that configuration
2.No one can tell you how much space you want or need
3. Older hardware is not as efficent at saving power when compared to the newer hardware
4. I do not buy hard disk drives that have been shipped in a box, I purchase local retail only
5. cyberjocks guide is a good source for learning about SLOG and L2arc with ZFS, you been around
here long enough (3-1/2 years), you have not read any of those posts?

 

[marbus90]

1. nope, it's imho the best practice for raidz2 since you'll be having 2^n data drives plus 2 redundancy drives.
2. only if you run it as an enterprise fileserver or are running many plex-sessions etc on the system.
3. hard to tell without knowing the exact CPU specs. If it comes with 2 CPUs, remove the CPU2. You may need to relocate the M1015 to plug into a slot which is run via the CPU1.
4. A good alternative are the HGST NAS 4TB drives, they're going for the same price as the WD Reds on newegg.
5. You wouldn't need them, but here are the recommendations: two S3700 or S3710 for mirrored SLOG, the L2ARC SSD isn't nearly as critical. Can be anything like the Crucial MX100.

 

[danb35]

Some of your questions make me suspect that you've gathered some bad info about ZFS and FreeNAS--specifically your questions about drive count and SSDs. To the extent that drive count matters at all with ZFS since the advent of compression (which is minimally, at most), the guideline has always been 2^n + p -- some power of two, plus the number of parity/redundancy disks. Six disks fits this guideline perfectly: four data disks plus two parity. There has never been a guideline suggesting simply 2^n disks for any flavor of RAIDZ vdev. And again, since compression became the default, this guideline has pretty much gone away. I'm curious where you got the idea that 2^n was recommended.

Similarly, it's pretty unlikely that you'll have much, if any, use for SLOG or L2ARC devices. What's your use case? Why do you think you'd want or need either of these?

As to the drives, as far as I can tell, WD's "up to n disks" is pure marketing. I haven't seen any documented difference between NASWare 2.0 and 3.0, nor any reason that the former should be limited to 5 disks while the latter is good for up to 8 (nor any reason why they can't both work with 12 disks or more). FWIW, nine of my 12 disks are WD reds, but I have no idea which "NASWare" version they're running. Seagate and HGST have "NAS" drives as well for about the same price--pick whatever flavor your prefer. The higher-speed WD Red Pro will likely be wasted in this application.

As to RAM, FreeNAS will use all you give it, but whether you'd need more than 16-32 TB depends a lot on what you're planning to do with the box.

 

[marbus90]

As to the drives, as far as I can tell, WD's "up to n disks" is pure marketing. I haven't seen any documented difference between NASWare 2.0 and 3.0, nor any reason that the former should be limited to 5 disks while the latter is good for up to 8 (nor any reason why they can't both work with 12 disks or more). FWIW, nine of my 12 disks are WD reds, but I have no idea which "NASWare" version they're running. Seagate and HGST have "NAS" drives as well for about the same price--pick whatever flavor your prefer. The higher-speed WD Red Pro will likely be wasted in this application.

Now, that isn't entirely true. With many disks in a single chassis without vibration damping the HDDs can disturb each other, leading to read errors, lots of head repositioning, shorter lifespan and the lot. Therefore: If you go big, don't go Red, but SAS drives. Avoids issues with the SAS Expanders too.

 

[Ericloewe]

Now, that isn't entirely true. With many disks in a single chassis without vibration damping the HDDs can disturb each other, leading to read errors, lots of head repositioning, shorter lifespan and the lot. Therefore: If you go big, don't go Red, but SAS drives. Avoids issues with the SAS Expanders too.

Is there really any empirical evidence of this? Plenty of people have run Greens and Reds with no issues on large systems.

 

[marbus90]

I haven't bothered about searching about the issue, but here's some youtube vid showing what voice can do:
Also, if it breaks in 10-15% of all cases, it's still considered the worst stuff ever happened in history. I'd rather design systems which are a bit more on the safe side.

 

[cyberjock]

I personally ran 24xWD Greens for more than 3 years with no problems. In fact, I had only 3 drives fail over the entire 3 year lifespan. 6 of the drives are still in the chassis, but there's only 16 drives now (10 are WD Red 6TB).

The voice thing is pretty old, and the voice frequencies are different than vibration that emanates from typical disks. So the voice thing proves that voices can affect disks (and pretty significantly too) but that doesn't really prove that lots of disks in a chassis require SAS.

There's lots of evidence around here that:

1. SAS are more reliable than SATA.
2. SAS are the same reliability than SATA.
3. SAS are less reliable than SATA.
4. Lots of disks means you should use SAS.
5. Lots of disks with SATA has no downsides.
6. SAS expanders + SATA drives = problems.
7. SAS expanders + SATA drives = no problems.

Notice lots of conflicts? Yeah. That's the problem. Some configurations are well documented to be problems. Others haven't been shown to cause problems, but the manufacturers provide recommendations that have been ignored by plenty. It's a very mixed bag, so YMMV.

 

[ixidor]

alright lemme clear up a few things. question 2 was about, would i need more ram. i can get it to 64gb, is there any benefit with 16tb of space of stuffing all the ram slots?
i know i have read, in more than a few places 2^n being optimal. here is one example http://pivotallabs.com/high-performing-mid-range-nas-server/#non_optimal we have reservations about the statement, “the number of disks should be a power of 2 for best performance”.

would having the 2nd cpu sped up/slow down anything, or taking it out speed up/slow down anything?

i cant find it right now, but was reading on the improvements from nasware 3 over 2. it is physical changes to the drives to make them more vibration tolerant. my current file server, which has over 4 years powered on on the drives smart, just used regular desktop drives, so hopefully any of the nas drives, hgst, wd, seagate etc should be plenty.

regarding #5, after reading here, http://pivotallabs.com/high-performing-mid-range-nas-server-part-2-performance-tuning-iscsi/ scroll down to section 1.0
seems determining the sized needed for zil, slog, l2arc takes a bit of finesse.

in the end, this will be going to be data store for only a few concurrent devices. i have a 2 port lacp bond for current file server, and for desktop. i would like to consistently saturate the 2g link, see reads maxxed out at about 200mb/s. i will eventually end up with a 4 port bond on the new server.
i plan to use 8 ports on the sas card, and 4 or 5 on the motherboard, no need in 12 bay for the sas expander.

also on the drives, im not so much concerned on the up to x drives. just how much hassle would WD give me, if they somehow determine that 12 drives in the chassis is what killed it, then maybe deny a warranty claim. plus paying a bit more now to get the RE4 drives means a longer warranty, and less hassle.

 

[danb35]

i know i have read, in more than a few places 2^n being optimal. here is one example http://pivotallabs.com/high-performing-mid-range-nas-server/#non_optimal we have reservations about the statement, “the number of disks should be a power of 2 for best performance”.

You're misreading that page--immediately before the part you quote is "We aren’t concerned about a non-optimal configuration (i.e. the number of disks (less parity) should optimally be a power of 2)". That is, the recommendation for "optimal" had been 2^n+p, as I noted in my post--which your proposed 6-disk RAIDZ2 array meets perfectly. However, as they went on to discuss, the concept is pretty questionable with the advent of default compression, and this post by one of the core ZFS developers further discusses why the 2^n+p rule just doesn't apply any more (to the extent it ever did).