Need a Plan Of Attack

[Binary Buddha]

So I need some help on planning a transition and setup for my NAS. My usage is primarily Plex related; with personal backups for home computers and "everything drawer" scenarios. My RAIDZ pool is my primary pool; Jails is a 1TB for jails (obviously). All the drives in RAIDZ are 2TB; with a spare SSD I had laying around for L2ARC. My eventual plan is to add another SATA controller to replace and max out the case space by having 15 8TB drives and max out the RAM to 24GB. Possibly add a PCI-e SSD card for the L2ARC and ZIL. The OCZ SSDs with RAID are recognized as having two drives that I can use for the L2ARC and ZIL. With the two Controllers I should have a few extra SATA ports to dangle a few drives for the JAILS zpool.

Current Hardware
MSI X58M
IO Crest SATA III 8 Port Controller
Marvell 88SE9705 chipset
RAM 16349MB
Intel(R) Core(TM) i7 CPU 960 @ 3.20GHz

Rosewill Server Chassis/Server Case/Rackmount Case, 4U Metal Rack Mount Server Chassis with 15 bays


[root@freenas /mnt/RAIDZ/HOME]# zpool status
pool: Jails
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
Jails ONLINE 0 0 0
gptid/cdf94e24-eaa1-11e5-8757-4061868cf681 ONLINE 0 0 0

errors: No known data errors

pool: RAIDZ
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
RAIDZ ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
gptid/731b8173-e82e-11e5-bb01-4061868cf681 ONLINE 0 0 0
gptid/74405aa9-e82e-11e5-bb01-4061868cf681 ONLINE 0 0 0
gptid/75151ffc-e82e-11e5-bb01-4061868cf681 ONLINE 0 0 0
gptid/75df5d80-e82e-11e5-bb01-4061868cf681 ONLINE 0 0 0
gptid/76da9dc9-e82e-11e5-bb01-4061868cf681 ONLINE 0 0 0
gptid/7785a4ef-e82e-11e5-bb01-4061868cf681 ONLINE 0 0 0
cache
gptid/77e28a10-e82e-11e5-bb01-4061868cf681 ONLINE 0 0 0

errors: No known data errors

pool: freenas-boot
state: ONLINE
scan: scrub repaired 0 in 0h4m with 0 errors on Tue Apr 12 03:49:34 2016
config:

NAME STATE READ WRITE CKSUM
freenas-boot ONLINE 0 0 0
gptid/ef59c7ed-e5de-11e4-80e6-4061868cf681 ONLINE 0 0 0

errors: No known data errors

 

[depasseg]

I'm not really sure what you are looking for, but I will caution you that you don't have enough RAM for a large L2ARC. And 24 GB won't be nearly enough for the ~120TB RAW, unless this is just a replication target = (no other shares). And I think there are other SSD's recommended for a SLOG (SLOG SSD's need internal capacitors for power protection), I haven't heard of OCZ being used for this.

 

[SweetAndLow]

I agree that I don't know what you want but I'll tell you your setup is not very good and you will run into problems with this hardware.

No need for the cache devices. Poor motherboard choice, poor CPU choice and your port controlled seems sketchy.

 

[nightshade00013]

What they said. Really.

Read a bit and you will understand why. https://forums.freenas.org/index.php?threads/hardware-recommendations-read-this-first.23069/

It's a deal where it may be fine for a week, a month, a year or six years but you have a much higher risk of data corruption among other problems. Most of us would suggest turning the SSD, MB, CPU and RAM into a gaming computer and buy recommended parts. A Xeon E5-2670 can be had for cheap the ECC RAM isn't super expensive if you find a board that has quad channel DDR3 to put the CPU in.

 

[Binary Buddha]

Well... what would be optimal? The MB and most of it was free. I don't need it to be the best... Just the best that it could be. I just don't like seeing perfectly fine hardware go to waste. As for building it into something else... I have no need for another workstation at the moment. I do however have a need to store things.

 

[Binary Buddha]

so I just read it's about 1GB RAM per TB. So the largest I "should" make the NAS is about 24TB?

 

[SweetAndLow]

That number is a little rough. Once you get above 16GB of memory you can get away with a bigger pool without being in danger of losing data. It might just be a little slow depending on your workflow.

 

[Binary Buddha]

As long as Plex doesn't lag out... I'm happy.

 

[danb35]

what would be optimal?

We do have a thread here with hardware recommendations, you know. It'd be a good idea to read it. But with that said, a little perspective. For 15 x 8 TB drives like you're talking about, you'll be spending $5k in drives alone. The RAM rule of thumb of 1 GB RAM / 1 TB storage is a rule of thumb, and a rough one, that gets more flexible as you get past 16 GB, but you're still going to want a fair bit more than 24 GB to support the 120 TB of storage space you're talking about. You don't have, and your motherboard can't support, enough RAM to make an L2ARC sensible--it would likely actually reduce performance.

Consider these components instead:

• Xeon E5-2670: $60.90
• SuperMicro X9SRL-F-B: $252.99
• 2 x 16 GB DIMMs: $101.99 ea.

Using 16 GB DIMMs, you can expand the system up to 128 GB, which should be plenty, but the board will support up to 512 GB of RAM. You'll have ECC RAM and a CPU/chipset that support it, IPMI (which can be very handy when running a server headless), and an 8-core, 2.6 GHz CPU.

 

[Binary Buddha]

Sorry, I'm probably not conveying information correctly... Or there's such a thing as selective reading...

With my MB at 24GB.... What size drives should I get to fill the 15 slots and in what sort of vdev configuration? Why use all the slots? Because depending on the size of the drives, I'm honestly just going to pick up a bunch of crap drives at Best Buy or something. So, I'm trying to increase the number of allowable drive failures whilst maintaining the maximum amount of storage as the top priority.

Yes, I know my MB is a pos... It's free hardware. I'M NOT BUYING A NEW MOTHERBOARD. The only new hardware I'm picking up is drives, SATA controllers, and RAM.

You'll be wasting both our time if you tell me to get a new MB. If I was going to "plan" to build a proper NAS I would have gone with @cyberjock 's list of recommended hardware. Shall we move on?

 

[mattbbpl]

" I don't need it to be the best... Just the best that it could be."
Unlike a gaming PC/workstation, the concern here is not speed - it's integrity. With this hardware your data is at risk, and you could lose everything on it. That being said, if you don't care, we don't care Although people in the community will be less inclined to offer support while running on inadequate hardware, that is also your choice. Just know the tradeoffs and risks. (Although is seems silly to me to purchase somewhere between $1500 to $4500 in drives, and then skimp out on the extra few hundred dollars to make those drives stable, but that's just me)

"15 8TB drives and max out the RAM to 24GB"
That's not going to fly. Not on FreeNAS. You might be able to get away with RAW storage of something like 40TB on 24GB of RAM (that would mean something like 2TB or 3TB drives if you wanted to get gutsy). I'm pushing it with 32GB on 60TB of RAW storage., and sometimes that causes performance issues - but it has been stable so far, so there's that.

 

[depasseg]

I believe you came here looking for feedback. Feedback has been given. You can chose to follow it or not. It's your choice to proceed with a plan of attaching an expensive JATO rocket onto a free go-cart without seatbelts.

In summary -
If you care about your data, use ECC RAM and corresponding MB.
If you plan to use an L2ARC, you need to closely follow the RAM sizing requirements or your experience will be slower than without the L2ARC. I don't think you need, nor do you have enough RAM to support an L2ARC.
24GB RAM might be enough, or might not, for your needs.

 

[danb35]

You probably need to give some thought to your priorities. In the same sentence you say you're trying to increase your redundancy and (as "top priority") maximize storage capacity; as you know, those two goals are directly opposed. If your top priority is to maximize storage capacity, just stripe all 15 drives together. You'll get full use of all of them, but when one of them dies your data will go away. If max possible redundancy is your goal, you could have three, 5-disk RAIDZ3 vdevs. Plenty of redundancy--you could lose as many as 9 drives--but you give up a lot of space to get it.

With the max of 24 GB of RAM, I wouldn't suggest more than 30-40 TB of storage total. If you absolutely have to fill all 15 bays, maybe 3 TB drives are the way to go. A better bet would probably be 10 x 4 TB drives in a single RAIDZ2 vdev. That would give you 32 TB (~29 TiB) of capacity, of which 23 TiB would be usable to comply with the 80% rule. But with that said, none of what you've said about what you're trying to do sounds like a good idea.

 

[SweetAndLow]

Your motherboard also has realtek NICs which are terrible so you should get a Intel nic for $30. The onboard audio might also be a problem but you never know until you try.

 

[mattbbpl]

24GB RAM might be enough, or might not, for your needs.

I'm curious on the rationale for this point. Under what use cases would 24GB of RAM be sufficient for a 120TB pool and a Plex jail?

 

[pirateghost]

I'm curious on the rationale for this point. Under what use cases would 24GB of RAM be sufficient for a 120TB pool and a Plex jail?

In case you wanted to replicate using a file server across a 56k modem?

 

[mattbbpl]

In case you wanted to replicate using a file server across a 56k modem?

Touche!

 

[AVB]

I did run 10 3TB drives on 16GB of memory and it was OK, not great but OK. Moving up to 24GB didn't seem like much of an improvement but when I got to 32GB the system seemed happier. No measurements to prove it but just from using it. I've since gone to 16 3TB drives in 2 raidz2 vdevs and it has slowed. New MB, processor and another 32GB of ram should fix that and I'll know in about a week if it does.

 

[depasseg]

I'm curious on the rationale for this point. Under what use cases would 24GB of RAM be sufficient for a 120TB pool and a Plex jail?

A combination of me using 8GB RAM for 32 TB of storage (replication and light use) and this: https://www.reddit.com/r/DataHoarder/comments/47qedb/how_much_ram_for_100tb_zfs_pool/

If the OP is just using this as a gigantic datastore for Plex supporting 1-2 users, RAM usage doesn't need to be that high (in my opinion).

 

[mattbbpl]

https://www.reddit.com/r/DataHoarder/comments/47qedb/how_much_ram_for_100tb_zfs_pool/

Holy cow, interesting data points in that thread.

I will say that my system (32GB/60TB) experiences some significant slowdowns during large/long write operations - sometimes drastically so. Given my usage patterns it's something I can tolerate, but still. I would feel nauseous attempting to run 100 TB on 16 GB.