Memory usage when expanding Striped Mirrors

[jutny]

Full disclosure: I'm a freeNAS noob. That being said I managed to get my box up and running without any issues. I'm even getting what I think is decent performance out of it, ~80-100MB/s over the network with a shitty switch.

My question has to do with memory usage and striped mirrors. I have read everywhere that ZFS loves memory, but mostly that seems to have to do with RAIDz1 or RAIDz2 where there is parity calculation.

My rig is as follows (I know, I went non-ecc... mini-ITX board choice was a bit limiting for socket 1150):

• Asus H87I-Plus
• Core i3 4130T 35w Haswell
• 16gb crucial 1333/1600
• Seasonic 400w 80+platinum
• 4x4tb WD Red - striped mirror (2 en-route literally now)
• Lian-Li Q08b w/ Noctua Fans

1. Will my memory usage requirements increase when I expand my storage with another two 4tb drives? Then i'll be 3x2 striped mirrors. Will my 16GB be enough? It's the max the board can take (same problem i ran into with a lot of the miniITX server boards.)
2. Performance increase or decrease? I can rationalize a small performance increase as workload is split across three drives.
3. AND finally, can I add the two new, empty drives into the existing vdev? (I think that's the proper terminology) Will freeNAS be able to reconfigure the array on its own?

Thanks in advance, and please be gentle with me. I tried searching a much as possible, but it was hard to find anything that was about striped mirrors specifically, everything else seems to be about RAIDz1 and z2.

 

[joelmusicman]

1. Will my memory usage requirements increase when I expand my storage with another two 4tb drives? Then i'll be 3x2 striped mirrors. Will my 16GB be enough? It's the max the board can take (same problem i ran into with a lot of the miniITX server boards.)
2. Performance increase or decrease? I can rationalize a small performance increase as workload is split across three drives.
3. AND finally, can I add the two new, empty drives into the existing vdev? (I think that's the proper terminology) Will freeNAS be able to reconfigure the array on its own?

80-100MB/s is pretty much saturating gigabit Ethernet. It would be hard to get better than that without upgrading your entire network.
mITX boards with ECC: Asrock E3C224D2I, or E3C226D2I. You can use your existing processor when you upgrade. :)

1. For home use 16gb should be fine for 12tb storage.
2. Might have a slight increase on IO performance, peak transfer rates will still be bottlenecked by GbE.
3. The two new drives will be a NEW vdev, which you can add to your existing POOL (you were close!). You'll need to tell it what to do in the Storage tab.

 

[jutny]

Awesome! Thank you so much for the speedy and polite reply (some forums... not so much)

I figured I was basically maxing out the gigabit, but was interested in the theoretical IO performance as you mentioned. I'm glad to hear that 16GB should be good, because my board maxes out at 16GB, so anything from here on out is going to have to be a motherboard upgrade (is it bad I'm already planning v2?) I can roll over this board and RAM to a HTPC build, and then maybe not constrain myself to mini-ITX for NASv2 when I make the jump to ECC. I purposely went with a 4th gen CPU so I could use it either way, plus... 35w TDP :)

So just so I'm crystal here: My current setup is 2 drives mirrored (1 vdev) and another 2 drives mirrored (1 vdev), striped together to create a pool? A pool to which I will add another vdev to when i get the 2nd two drives? Then I will have the ability to have it(freeNAS/ZFS) figure out which data from the existing striped mirrors will be moved over to the new mirror to create a 3-way stripe? If I'm way off here, it's because I'm thinking I currently have 2 vdev's but I can think it through either way. I thought the inital setup of the 4 drives was ultra easy, though it never said anything about stripe/mirror, just mirror (stripe option offered 16TB capacity) but I assume that's what I wanted as I have the 8TB capacity from 4 drives, so it must be striped mirrors.

I realize this may sound like a stupid question, but it's hard to wrap my head around never having any real experience with this sort of thing. Thanks!

 

[joelmusicman]

I figured I was basically maxing out the gigabit, but was interested in the theoretical IO performance as you mentioned. I'm glad to hear that 16GB should be good, because my board maxes out at 16GB, so anything from here on out is going to have to be a motherboard upgrade (is it bad I'm already planning v2?) I can roll over this board and RAM to a HTPC build, and then maybe not constrain myself to mini-ITX for NASv2 when I make the jump to ECC. I purposely went with a 4th gen CPU so I could use it either way, plus... 35w TDP :)

Those are pretty good thoughts. Of course an HTPC would probably require a better video card, but the $50ish ones will do the trick.

So just so I'm crystal here: My current setup is 2 drives mirrored (1 vdev) and another 2 drives mirrored (1 vdev), striped together to create a pool? A pool to which I will add another vdev to when i get the 2nd two drives? Then I will have the ability to have it(freeNAS/ZFS) figure out which data from the existing striped mirrors will be moved over to the new mirror to create a 3-way stripe?

Yes, yes, and kinda. AFAIK it doesn't really move data that's already in place, just provisions the storage and new data writes will be spread evenly among the six.

If I'm way off here, it's because I'm thinking I currently have 2 vdev's but I can think it through either way. I thought the inital setup of the 4 drives was ultra easy, though it never said anything about stripe/mirror, just mirror (stripe option offered 16TB capacity) but I assume that's what I wanted as I have the 8TB capacity from 4 drives, so it must be striped mirrors.

Striped & mirrored setup is pretty good, but less optimal from a data protection standpoint than a RAIDZ2 for a new pool of six. If you have solid backups in place (TEST them first!), you might want to consider destroying the pool and create a new one of RAIDZ2 when the new drives arrive.

RAIDZ2: If you lose one drive you won't be in a zero-parity scenario. You'll also gain an extra drive's worth of storage (two total parity drives vs three). You also won't have a TOTAL failure if you lose the "wrong" two drives.

 

[cyberjock]

Awesome! Thank you so much for the speedy and polite reply (some forums... not so much)

You ever, EVER call us polite again and I will shank you! I'm so offended I'm gonna have to go call my mom and cry!

As for mirrors versus RAIDZ2, unless you plan to run VMs on your pool, RAIDZ2 is better for home users as it provides more disk space and better redundancy for certain situations. ;)

 

[jutny]

Those are pretty good thoughts. Of course an HTPC would probably require a better video card, but the $50ish ones will do the trick.

I think that's the route I'll go in the future. I have a couple of decent 3d cards that aren't in use, albeit power hungry.

Yes, yes, and kinda. AFAIK it doesn't really move data that's already in place, just provisions the storage and new data writes will be spread evenly among the six.

Striped & mirrored setup is pretty good, but less optimal from a data protection standpoint than a RAIDZ2 for a new pool of six. If you have solid backups in place (TEST them first!), you might want to consider destroying the pool and create a new one of RAIDZ2 when the new drives arrive.

Considering I have only transferred data onto the NAS in order to test it thus far (my uptime is about 3 days) that won't be a problem. I would have started from scratch with the three drives anyway, based on what you said about how it would manage the expansion. Just seems a bit "dirty" to me. My real question comes next.

RAIDZ2: If you lose one drive you won't be in a zero-parity scenario. You'll also gain an extra drive's worth of storage (two total parity drives vs three). You also won't have a TOTAL failure if you lose the "wrong" two drives.

You ever, EVER call us polite again and I will shank you! I'm so offended I'm gonna have to go call my mom and cry!

As for mirrors versus RAIDZ2, unless you plan to run VMs on your pool, RAIDZ2 is better for home users as it provides more disk space and better redundancy for certain situations. ;)

Hah! Thanks for the welcome :)

I am a home user. I will now have 6 identical 4tb drives. I don't really need 16TB of space but why not. My thoughts so far are that I didn't go the ECC route, and in that way, going with the striped mirror setup would be not only lower overhead, but a little faster as well. The lower overhead bit being where the risk was minimized due to non-ECC. I don't know if this was sound logic at all. Considering I am maxing out my gigabit connection with my striped mirrors right now, do you think I would see a performance hit with my hardware if I went to RAIDZ2?

Seriously, if it's really the better route, I'll do it. I think I'm at the point where I don't know enough about this stuff, to have really been able to tell what the best option for me, even though I've read quite a lot about it. I really like the idea of RAIDz2, but I'm not sure I have the right build for it?

 

[joelmusicman]

I am a home user. I will now have 6 identical 4tb drives. I don't really need 16TB of space but why not. My thoughts so far are that I didn't go the ECC route, and in that way, going with the striped mirror setup would be not only lower overhead, but a little faster as well. The lower overhead bit being where the risk was minimized due to non-ECC. I don't know if this was sound logic at all. Considering I am maxing out my gigabit connection with my striped mirrors right now, do you think I would see a performance hit with my hardware if I went to RAIDZ2?

No worries about "overprovisioning." It's more cost effective to buy it now than to buy cheaper stuff and upgrade later. It's also much more difficult to add extra drives later on than to start with the final number.

As far as performance goes, I have no trouble maxing out GbE on a Pentium G3220 with RAIDZ2, and in fact DD benchmarks (local filesystem) show 400MB/s writes and 600MB/s reads. I long for 10G but that stuff is pretty spendy. It struggles at Plex transcoding, but that's a different issue.

 

[jutny]

That was going to be my final line of questioning. What sort of ways can I benchmark this beast? My IT buddy is curious to compare it to certain desktop RAID setups he has. Can I use IOmeter from a network machine? For that purpose, local filesystem benchmarks aren't exactly relevant for comparison. That being said, I'd like to know how to do the local benchmarks so I can see the true performance (non-network limited) and how things change as I upgrade the drives and switch drive setups around before settling on one setup.

My drives showed up today so I intend to pop them in when I get home after work today. I also got my proper gigabit switch (D-Link DGS 1100-16) so I should see a bit of improvement in my network as a whole. Yay new toys!

I probably don't want to go down this hole, but what's Plex transcoding?

And now to start my research on how to get my foscam IP cam to record to the NAS.

 

[joelmusicman]

The "Performance" forum has the info you seek.

Plex is a media player that can serve content across the internet. It converts videos on the fly, and therefore requires a decent amount of CPU horsepower.

 

[jutny]

Great, thanks for the direction. I haven't really had the time to explore the forums fully yet.

Thanks for all your help, I really appreciate you taking the time to answer my silly questions. I feel a lot more confident in the setup, and I'm excited to try out RAIDz2 and effectively double my capacity with these two new drives. :)

I have a ton of things to look into now... including how I might be able to utilize it with my smartTV. Hopefully anyone else with my inital questions will now be able to search the web and find this thread.

Cheers! :D