How should I configure 14 3tb drives?

[Joe Goldthwaite]

I kind of quit working on my new FreeNAS machine because I got discouraged. I had built it with 12 2tb drives in a big RaidZ2 array. I found out that that specific number of drives makes for a very inefficient array.

I added a 13th drive and it looked like I got most of the missing space back but when I copied my files from the old NAS to the new one, they took up a lot more space. My old NAS had 9 3tb drives. The new one had 13. After copying the data over the new NAS has less available space than the old one. It's like I lost more than 4 drives of space. That's when I got discouraged and took some time off.

I'm ready to try again. This time I'm thinking of creating the array with 7 vdevs made up of two mirrored 3tb arrays each. No RaidZ just 7 mirrored sets. Any advice or thoughts?

 

[SweetAndLow]

Hardware specs? Read the rules. Possible good options are 2vdevs with 7disks each in raidz3 or 2vdevs of 6drives in raidz2.

Sent from my Nexus 5X using Tapatalk

 

[Joe Goldthwaite]

Asrock C2750D4i motherboard with 16gb ECC ram. It's got 12 SATA ports and I've added a four port SATA adapter to the only PCI slot so I could hook up an additional two drives.

Edit: Freenas 9.10

 

[SweetAndLow]

Asrock C2750D4i motherboard with 16gb ECC ram. It's got 12 SATA ports and I've added a four port SATA adapter to the only PCI slot so I could hook up an additional two drives.

Edit: Freenas 9.10

Actually that board only has 6 maybe 8 usable sata ports. The 4 Marvell SE9230 ports do not work under freebsd. Also sata cards are usually a bad idea. You should use a hba card and forwards break out cable to add sata ports.

Also tell us what HDD you have, what your storage needs are and what workflows you use.

Sent from my Nexus 5X using Tapatalk

 

[CraigD]

I am confused

You complain about storage capacity of RAIDz2 losing less than 20% of your space

Your looking into faster, less redundant, storage solution that you can use less than 50% of space?

Depending on your use case, Mirrors can be beneficial. If you need the speed great choice, if you desire capacity not so much

SATA port availability is not really an issue, if you can add an HBA

Have Fun

 

[gpsguy]

How did you add the 13th drive? Hopefully you didn't add it as a single drive into your existing volume. If so, should that disk fail, you will lose your entire volume.


Sent from my iPhone using Tapatalk

 

[Joe Goldthwaite]

I am confused
You complain about storage capacity of RAIDz2 losing less than 20% of your space

I'm not really complaining. It's just that I had an old FreeNAS box with 9 3tb drives in Raid2Z. When I built the new one with 12 3tb drives in Raid3Z I expected to go from 14tb of usable space to 20tb. I didn't. I ended up with 16tb. SweetAndLow posted this link to explain it. That's why I added an extra drive to bump it up to 13. According to that article, 13 drives would be more efficient. I ended up with a little more that 20tb of space.

When I started copying the data over though, it looks like each file is taking up more space than it does on the 9 drive NAS. Once all the files copied over, I had less free space on the 20tb box than I did on the 14tb box. I don't know why but my current configuration of 13 drives in a large Raid3z array just isn't working.

I've since read that mirroring the drives is better than Raidz. If you lose a drive it's much quicker to rebuild the mirror. If you want to add space it's easier. Upgrading the array is easier. This spells it out.

I'm not worried about performance. I'd like as much space as possible *but* creating a big RaidZ pool doesn't seem to be giving me that. Since I was only one drive away from having 7 mirrored vdevs and that would give me my 20tb, I thought I'd try that. I thought it would be a good idea to ask the experts before I actually went ahead and did it.

How did you add the 13th drive? Hopefully you didn't add it as a single drive into your existing volume. If so, should that disk fail, you will lose your entire volume.

Don't worry. All my data is on my old NAS so I can wipe the new one and re-copy the data. That's what I've done with all my changes.

 

[depasseg]

I'm confused. You seem to need more usable space than your current array configuration can give you, but then you are considering mirrors which gives you the least amount of usable space compared to raw.
Have you seen @Bidule0hm 's ZFS sizing calculator? https://forums.freenas.org/index.php?threads/zfs-raid-size-and-reliability-calculator.28191/
And don't forget that the target usable size shouldn't exceed 80% without expecting possibly serious performance degradation.

 

[Joe Goldthwaite]

Hi depasseg. That calculator doesn't take into account the efficiency problem mentioned above. It thinks my 12 drive array would give me 19.33TiB of usable space. I actually got about 3TiB less than that.

 

[depasseg]

Ahh, interesting. And mirrors don't suffer from that same efficiency problem?

 

[Joe Goldthwaite]

Ahh, interesting. And mirrors don't suffer from that same efficiency problem?

I honestly don't know. A 13 drive array had a lot more space than a 12 drive array. The space disappeared when I coped my files over to it though. I'm hoping that a pool made up of 7 mirrored vdevs will work more like my old drive made up of 9 disks in RaidZ2. I guess I'll have to try it to find out.

P.S Sorry, I've been describing my old NAS wrong. It's made up of 9 2tb drives in a RaidZ2. That's where my 14TiB is coming from.

 

[VladTepes]

This time I'm thinking of creating the array with 7 vdevs made up of two mirrored 3tb arrays each. No RaidZ just 7 mirrored sets. Any advice or thoughts?

Leaving speed aside for a moment....
With your example (if I understand it correctly) there is only a single drive redundancy in any given vDev. Failure of any VDev will kill your pool. Risky.

 

[Mirfster]

Leaving speed aside for a moment....
With your example (if I understand it correctly) there is only a single drive redundancy in any given vDev. Failure of any VDev will kill your pool. Risky.

Correct. I am running some Mirror vDevs for VM Storage, however I try to maintain 1 Hot Spare for every 2 Mirrors I use.

Kinda janky but does give a little more security. Think if it as a pseudo 3 Way Mirror in a relaxed sense. Also, I have Cold Spares that are readily available.

Others run 3 Way Mirrors which is safer but at a higher cost in disks.

Lastly, keep in mind that Mirrors Re-Silver a heck of a lot faster.

 

[VladTepes]

" Mirrors Re-Silver a heck of a lot faster".- Yes, there is that.

 

[Murac]

Actually that board only has 6 maybe 8 usable sata ports. The 4 Marvell SE9230 ports do not work under freebsd. Also sata cards are usually a bad idea. You should use a hba card and forwards break out cable to add sata ports.

I've got the 2550 version of that board and I use some of the Marvel ports. They also worked but gave errors. I updated the marvell firmware and the errors went away. granted i never had 12 drives plugged in but yeah, worked for me.

Read SMART Error Log Failed after every test [SOLVED]