VDEV setup

Status
Not open for further replies.

LIGISTX

Guru
Joined
Apr 12, 2015
Messages
525
I am about to build out a new server, just ordered a X8 based server, to be more accurate: SUPERMICRO 4U 846E16-R1200B X8DTE-F

I currently have 10 3 TB drives that I am using in a old NAS box, and want to later utilize them in my new build, but first want to start fresh with 4 TB drives, migrate the data over, and then pop the old drives in as a new VDEV.

I am just trying to make sure my redundancy assumptions and ideas are correct, don't want to do all this work, overlook some silly statistical issue, and have a situation where I don't have enough redundancy.

My "new" 4TB vdev is probably going to be 8 4 TB drives, with this I plan on using RAIDZ2, that should be enough with this many drives/volume size, correct?

Then I will plan on throwing my old 3 TB's in, and have all 10 of those as a RAIDZ3 vdev. Sound good?

And now, one more question, because you can't write a forum post without thinking of something else midway through.

Are there any issues I may have running these two vdevs in the same pool? Would it be a "better" idea to run them as different independent pools? I am new to freenas and ZFS, and I know just enough to be dangerous as some would say. Just trying to make sure my assumptions are all correct.

For reference, I will have a HBA with LSI 9211-8I firmware on it and a 24 port backplane. As far as I have been told thus far, I will only need to run 1 SAS cable to the backplane, will that have any issue with my proposed setup?

Thanks guys!
 
Last edited:

DrKK

FreeNAS Generalissimo
Joined
Oct 15, 2013
Messages
3,630
My "new" 4TB vdev is probably going to be 8 4 TB drives, with this I plan on using RAIDZ2, that should be enough with this many drives/volume size, correct?
8x4TB, brand new drives in a RAID-Z2 is somewhere near the edge of the green zone. If these are brand new NAS-oriented drives (e.g., WD reds), and you know exactly what you are doing and are going to aggressive maintain the vdev, then you should be OK. But, generally speaking, most of us would consider 8 larger (so, 4TB or more) in a RAID-Z2 to be pushing your luck. Effectively, that's 25% redundancy. "Stupid" starts at 20%. "Conservative" starts at 30%. So you're in the yellow zone, I think, if you were to poll the top 20 posters on this forum. I'd personally sign off on it, if the user had a good grip on ZFS and FreeNAS. Cyberjock, who is much more conservative, would [expel] blood. So it depends who you ask.

Then I will plan on throwing my old 3 TB's in, and have all 10 of those as a RAIDZ3 vdev. Sound good?
That sounds OK, if those drives are still in their youth or early middle age (i.e., under 25000 hours).

Are there any issues I may have running these two vdevs in the same pool? Would it be a "better" idea to run them as different independent pools? I am new to freenas and ZFS, and I know just enough to be dangerous as some would say. Just trying to make sure my assumptions are all correct.
Nah. If you'd rather have the storage capitated in one pool, that will be fine. Doesn't really matter here if you have one or two pools. It's whatever works for you. Neither of these vdev configurations are particularly high performance, so putting them together doesn't have any adverse impact.

For reference, I will have a HBA with LSI 9211-8I firmware on it and a 24 port backplane. As far as I have been told thus far, I will only need to run 1 SAS cable to the backplane, will that have any issue with my proposed setup?
Someone else will have to answer that. I don't know [feces] about HBAs.

Hope that was helpful sir.
 

Stux

MVP
Joined
Jun 2, 2016
Messages
4,358
FWIW,

That's pretty much what I started with, Ie 8x4TB in RAIDZ2 and a bunch of older 3TB drives.

Two vdevs will stripe across each other, which will improve iops and total sequential performance.

FWIW2,

I believe your understanding re HBA/SAS/Backplane is correct. Basically, you'll have 24gbps of total bandwidth to the backplane with a single mini-sas connector (6gbps x 4). And I believe the card has a total of 32gbps of bandwidth to the CPU (PCIe2 x 8).

8i means two 4i internal mini sas ports on the card. You can later route one of the ports to the outside of the chassis and connect another 24 bay expander chassis if you want ;).

If the backplane has two ports, then you could actually connect both sas ports to it to double the bandwidth to the backplane... I think... which may or may not work, but this is where my sas knowledge ends ;)
 
Last edited:

LIGISTX

Guru
Joined
Apr 12, 2015
Messages
525
8x4TB, brand new drives in a RAID-Z2 is somewhere near the edge of the green zone. If these are brand new NAS-oriented drives (e.g., WD reds), and you know exactly what you are doing and are going to aggressive maintain the vdev, then you should be OK. But, generally speaking, most of us would consider 8 larger (so, 4TB or more) in a RAID-Z2 to be pushing your luck. Effectively, that's 25% redundancy. "Stupid" starts at 20%. "Conservative" starts at 30%. So you're in the yellow zone, I think, if you were to poll the top 20 posters on this forum. I'd personally sign off on it, if the user had a good grip on ZFS and FreeNAS. Cyberjock, who is much more conservative, would [expel] blood. So it depends who you ask.


That sounds OK, if those drives are still in their youth or early middle age (i.e., under 25000 hours).


Nah. If you'd rather have the storage capitated in one pool, that will be fine. Doesn't really matter here if you have one or two pools. It's whatever works for you. Neither of these vdev configurations are particularly high performance, so putting them together doesn't have any adverse impact.


Someone else will have to answer that. I don't know [feces] about HBAs.

Hope that was helpful sir.

Ok, good info. I guess I may have to play with multiple Z3 vdevs... But, actually, I am not sure on the age of the 3TB drives. They have been running some of them for a few years now..... Some are newer, but maybe half are 2-3 years old. They don't get hit hard, and have been in a swell ventilated setup, but still, don't want to be dumb here. That being said..... There isn't a RaidZ4. I do plan to follow the guides of the more knowledgable people to the letter, make sure SMART reports are all in order, scrub the drives etc. I have a LOT of reading and learning to do before I actually deploy the setup.


Sent from my jailbroke iPhone using Tapatalk
 

CraigD

Patron
Joined
Mar 8, 2016
Messages
343
Loved looking at your home built case!

If you have a backup, thrash the 3TB drives until they die, you have nothing to lose

HDDs are durable if they are looked after last a long time, and you wooden case looks to have had great air flow

Have Fun
PS In the last month I have had 2 HDDs die with 40000+ hours on them, Strangely another drive has 50000+ hours and it is running great
 

nojohnny101

Wizard
Joined
Dec 3, 2015
Messages
1,478
build looks good. i'm envious of all the drives you have to play with :)

what @DrKK said, FreeNAS documentation recommends no larger than 8 drives for a raidz2 so that is the upper limit. you should be fine and as long as you have a backup. the upper limit for raidz3 is 12 drives I believe so you're also good there.

keeping them together in the same pool will be no problem as others have said. make management of data easier if you ask me!

best of luck!
 

LIGISTX

Guru
Joined
Apr 12, 2015
Messages
525
Loved looking at your home built case!

If you have a backup, thrash the 3TB drives until they die, you have nothing to lose

HDDs are durable if they are looked after last a long time, and you wooden case looks to have had great air flow

Have Fun
PS In the last month I have had 2 HDDs die with 40000+ hours on them, Strangely another drive has 50000+ hours and it is running great
build looks good. i'm envious of all the drives you have to play with :)

what @DrKK said, FreeNAS documentation recommends no larger than 8 drives for a raidz2 so that is the upper limit. you should be fine and as long as you have a backup. the upper limit for raidz3 is 12 drives I believe so you're also good there.

keeping them together in the same pool will be no problem as others have said. make management of data easier if you ask me!

best of luck!

Thanks guys, yea the wood case was a fun setup, but it has outlived its useful lifespan. I'll prob do something with it as it can hold 16 drives.... But we are clearly moving onto bigger and better things going into the future.

That being said, I don't have an actual backup, and I know I know, RAID isn't a backup. But, that being said, none of the data is mission critical, if it was all to be lost I would be sad, but nothing more than some low level emotional pain. Everything stored on it that IS critical data is stored in other locations, but that is maybe 2-3 TB of the total pool. Obviously, I don't want to play the statistics game like a moron, so I think I may throw another 4TB into the budget and run a 9x3TB Z3 and a 9x4TB Z3.

On that note, I have had a drive fail in my current 8x3 Z2 setup, thankfully it didn't take too long to recover and rebuild, but, I understand that was a sketchy situation... Having that extra parity drive would be appreciated.

My only concern at this point is I didn't plan on having two Z3's in here, I hope the Dual socket L5520 Xeon server I got will have enough headroom to do scrubs, run plex and some other jails. The scrubs I will schedule when there is least use, so plex shouldn't interfere with scrubs, but I was planning to use the extra CPU power for folding at home since, being a millennial, I fully support the idea of turning watts into health benefits for the future ;). Hopefully this server will have the guts to do all of this, guess we will find out!

Also, I was planning on running encryption on these drives. Is that going to be possible from a CPU standpoint....?


Sent from my jailbroke iPhone using Tapatalk
 

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,456
Also, I was planning on running encryption on these drives. Is that going to be possible from a CPU standpoint....?
There are lots of caveats regarding FreeNAS's encryption, such that the general recommendation is not to use it unless you have a legal or regulatory requirement to do so. With that said, the L5520 doesn't appear to support AES-NI, so the encryption would slow you down a bit. I'm sure you'd still have plenty of horsepower, but it wouldn't be hardware-accelerated as it would be with a more modern CPU.
 
Status
Not open for further replies.
Top