Benefit/issues with 10 disk vdev

[rwslippey]

The title might be a little unclear but it was that or a really long title.

I'm getting ready to do an upgrade on my current Freenas system. I'm planning on adding memory and a few hard drives. I'm planning on offloading all of my data and moving it to a temporary location until I get the upgrade complete. Basically, I'm going to end up with 10 disks. For now, 5 will be 4TB the other 5 will be 2TB and increased in size and resilvered over the next few months (budgets, got to love them)

My question is: What would be the benefit of splitting my vdevs up. So instead of having a 10 drive vdev would having 2 5 disk vdevs make any sense or be at all useful. I only ask because I saw the 10 disk recommended limit all over the place but I couldn't figure out why one would want to split them up beyond that?

My thought is that it might improve performance but that's a complete guess. I'd like to get 10 GB ethernet going at some point in the future but it's a little ways off but if I can plan for it now and I'm indeed correct on this being a performance thing, it might be worth it.

Would there be any other arguments to my 10 disk configuration? I'm planning on going RAIDz3 for the added protection if it helps to have that information.

 

[Chris Moore]

What would be the benefit of splitting my vdevs up. So instead of having a 10 drive vdev would having 2 5 disk vdevs make any sense or be at all useful.

Two vdevs would give you faster access to the data vs one large vdev. I switched from on 12 drive vdev to two 6 drive vdevs several years ago because of this. Because I eliminated deduplication at the same time, it more than doubled my speed on the same hardware. You should run RAIDz2 and 6 drive vdevs are pretty much ideal because of the way the padding lays out on the drives. Less wasted space.

I only ask because I saw the 10 disk recommended limit all over the place but I couldn't figure out why one would want to split them up beyond that?

The wider the vdev, the longer it takes to scrub or resilver. It is best to keep vdevs narrow for performance and space allocation. I have a server at work that was bought before I started that has vdevs with 15 drives and it takes about five days to do a scrub or to resilver a replacement disk.

My thought is that it might improve performance but that's a complete guess. I'd like to get 10 GB ethernet going at some point in the future but it's a little ways off but if I can plan for it now and I'm indeed correct on this being a performance thing, it might be worth it.

Yes, you are correct. When I had one vdev with 12 drives, I couldn't even fill the 1Gbit pipe, but with two vdevs, I get much better speed and I upgraded to 10Gbit also. It is nice.

Would there be any other arguments to my 10 disk configuration? I'm planning on going RAIDz3 for the added protection if it helps to have that information.

I when I went to 2 RAIDz2 vdevs, I was moving away from a 12 drive RAIDz3 pool. I have 4 dives of parity now instead of 3, and I have more speed. I wouldn't do RAIDz3, I tried it and this is better.

 

[Chris Moore]

If you want to do 10Gig networking, you should watch this series of YouTube videos:
https://www.youtube.com/watch?v=MgNpI6VAAhI
This is the first one and I followed these directions. It works.

 

[rwslippey]

Two vdevs would give you faster access to the data vs one large vdev. I switched from on 12 drive vdev to two 6 drive vdevs several years ago because of this. Because I eliminated deduplication at the same time, it more than doubled my speed on the same hardware. You should run RAIDz2 and 6 drive vdevs are pretty much ideal because of the way the padding lays out on the drives. Less wasted space.

The wider the vdev, the longer it takes to scrub or resilver. It is best to keep vdevs narrow for performance and space allocation. I have a server at work that was bought before I started that has vdevs with 15 drives and it takes about five days to do a scrub or to resilver a replacement disk.

Yes, you are correct. When I had one vdev with 12 drives, I couldn't even fill the 1Gbit pipe, but with two vdevs, I get much better speed and I upgraded to 10Gbit also. It is nice.

I when I went to 2 RAIDz2 vdevs, I was moving away from a 12 drive RAIDz3 pool. I have 4 dives of parity now instead of 3, and I have more speed. I wouldn't do RAIDz3, I tried it and this is better.

Wow, I was right then... I think I might go this route. Is two 5 disk vdevs still a good route to go or should I really push for 6 each. I’ve been working with freenas for a few years but still can’t understand the nuances of the numbers of disks and why they were selected


Sent from my iPhone using Tapatalk

 

[rwslippey]

If you want to do 10Gig networking, you should watch this series of YouTube videos:
https://www.youtube.com/watch?v=MgNpI6VAAhI
This is the first one and I followed these directions. It works.

This was actually the video that got me wanting 10 Gb Ethernet. I think it’d be awesome to be able to keep files on the NAS and work on them. Iv tried before but it just didn’t like the slow connection. On the other hand even if 10 isn’t enough I can still enjoy much faster copies


Sent from my iPhone using Tapatalk

 

[Chris Moore]

Is two 5 disk vdevs still a good route to go or should I really push for 6 each.

If you can fit them in your case and afford them, I would suggest 6.

I’ve been working with freenas for a few years but still can’t understand the nuances of the numbers of disks and why they were selected

It is about how the blocks of data lay out on disk and the amount of padding needed.

 

[rwslippey]

If you can fit them in your case and afford them, I would suggest 6.

It is about how the blocks of data lay out on disk and the amount of padding needed.

I can definitely fit them... can afford them too... but then there is the other half...

Just spent 750 on new drives and a power supply n case. But I’ll see how I can work in the 2 extra drives


Another quick question. Would it matter if 1 vdev was full of 2Tb drives and the 4Tb I’ll eventually up them all to 4 just not at once. I know I won’t get all the space until they all get replaced though





Sent from my iPhone using Tapatalk

 

[Stux]

Wow, I was right then... I think I might go this route. Is two 5 disk vdevs still a good route to go or should I really push for 6 each. I’ve been working with freenas for a few years but still can’t understand the nuances of the numbers of disks and why they were selected


Sent from my iPhone using Tapatalk

The reason to go with 6 drives instead of 5 is you should be using raidz2.

The extra drive is good value as it increases your available storage by 33%

 

[Chris Moore]

Another quick question. Would it matter if 1 vdev was full of 2Tb drives and the 4Tb I’ll eventually up them all to 4 just not at once. I know I won’t get all the space until they all get replaced though.

I am working on a similar change from 2TB to 4TB drives and right now I have one vdev of 4TB drives and the other is 2TB drives. It works out fine and you get the space when all the drives of a vdev have the same size.



Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[rwslippey]

I am working on a similar change from 2TB to 4TB drives and right now I have one vdev of 4TB drives and the other is 2TB drives. It works out fine and you get the space when all the drives of a vdev have the same size.



Sent from my SAMSUNG-SGH-I537 using Tapatalk

Sweet,

as it stands now I'll gain a bit of space anyway and I don't really think I need a quick jump to the 20 some TB that'll I'd get with the 4TB drives but atleast its an option down the road.... Who knows I might start in 4k video someday .... that'd be painful

 

[Chris Moore]

This was actually the video that got me wanting 10 Gb Ethernet.

I built the switch he talks about in video 3 using an old socket 1366 board from SuperMicro that I had laying around doing nothiing. It had two integrated 1Gig network interfaces and I added two dual port 10Gig cards and two quad port 1Gig cards. So I have 10 ports at 1Gb and 4 ports at 10Gb and all it cost me was the purchase price of the network cards, which was less than $100 if I recall correctly. I did that last year and it has been really nice. I have two desktops and two servers connected on the 10Gig ports and all the other things in the house take up most of the 1Gig ports, but it works like a champ.

 

[rwslippey]

I built the switch he talks about in video 3 using an old socket 1366 board from SuperMicro that I had laying around doing nothing. It had two integrated 1Gig network interfaces and I added two dual port 10Gig cards and two quad port 1Gig cards. So I have 10 ports at 1Gb and 4 ports at 10Gb and all it cost me was the purchase price of the network cards, which was less than $100 if I recall correctly. I did that last year and it has been really nice. I have two desktops and two servers connected on the 10Gig ports and all the other things in the house take up most of the 1Gig ports, but it works like a champ.

Nice, I haven't gotten past video 1 yet. Still in the need to get the storage right phase. I probably watched it some time ago but don't remember all of the details. I'll check it out when I get home. I can't image how it'll be to be able to move files from the workstation without waiting forever... Only wish I had was that dell had some sort of way to integrate 10GBe into a laptop.... I've got an XPS and have been thinking about their thunderbolt dock and making the laptop my only workstation but if I'm going for 10GB it's not worth it, I'll keep the desktop up..... I had some other plans for the desktop hardware but that's a totally different story

 

[diedrichg]

If you are going to have the money for the larger drives in a few months then just wait. Save the money - I know, it's hard to let it sit in the bank and not use it. Then when you have the money to buy the proper sized drives you'll be all set for creating the exact sized pool you require.

 

[Stux]

Of course, you could start with a single vdev and add another later.

 

[Chris Moore]

Only wish I had was that dell had some sort of way to integrate 10GBe into a laptop....

Same here. I use a laptop most of the time and that is almost making me build a new desktop for myself. My wife is the one that uses the desktop we have on the 10Gig and it is nice for her. I get to use it sometimes. When I first did the 10Gig switch, I would copy files just to see how fast it was.