*

[TinTIn]

So my kit arrives in a week or two consisting of one Supermicro server node with 256GB RAM and 3 x 45 bay Supermicro JBODs and 90 x 4TB SAS drives.

It has been agreed that a 10 disk per vdev layout with RAIDZ2 with 9 vdevs is the most appropriate.

Any thoughts on the disk layout within the vdevs? Should I try and scatter the disks across the JBODs as evenly as possible per vdev or would this make no difference?

Any other thoughts or suggestions on the setup would be most welcome.

Thanks


Sent from my iPhone using Tapatalk

 

[jgreco]

3 JBOD chassis? Or two (2 x 45 = 90)?

If you're actually doing three, then, yes, ideally spread the drives out in order to help manage heat dissipation and hotspots. Those 45 drive chassis have the downside that you're exhausting (slightly?) warm air past the back drives. For a 24-drive chassis, something like leaving the second and fifth rows empty (just drive fillers in the trays). You should probably also have several warm spare disks ready to go. With 90 disks, you will be frequently replacing disks, and with only Z2 protection, you can't really afford to mess around too much.

As for the vdev-to-disk layout, this is more a function of how you're attaching the disks. The two backplanes in the 45's may be wired independently (to the four external SAS connectors) but you get some complexity depending on whether you're going for the single (E16) or dual (E26) models. The dual can do multipath SAS. Your situation is simpler because you're not doing dual heads, which FreeNAS doesn't really support. What is your attachment plan? As long as you're not daisy-chaining (either chassis-to-chassis or internally within a chassis, backplane-to-backplane) you're likely to be fine.

 

[TinTIn]

Thanks for your thoughts jgreco.

3 JBODs are being used with 30 disks per JBOD.

I'm going to have 2 warm spares to start with so 92 disks in total.

The JBODs will be cabled twice as the server has three HBAs with two channels on each. I will cable them twice to different HBAs to create redundancy in the event of a HBA or cable failure and perhaps help throughput.


Sent from my iPhone using Tapatalk

 

[jgreco]

Three HBA's with two channels each is six channels.

Each of your JBOD devices has two separate backplanes in it. Three chassis times two backplanes means you have six backplanes.

Six backplanes to six HBA channels is single link. No redundancy.

You can internally connect the two backplanes within a JBOD together, which might make redundancy, but you'll have to figure out all the multipath stuff.

 

[TinTIn]

I don't understand jgreco.

The JBODs only require one connection to the HBA so I figured cable them twice to different HBAs and that would provide multi path (perhaps) and failover. Is that wrong?


Sent from my iPhone using Tapatalk

 

[jgreco]

Not exactly. Re-read my immediately preceding response. Depending on their internal configuration, the JBOD's might require one or two HBA connections.

 

[TinTIn]

Ok ta. I'm sure these only require one. Many thanks for your advice.


Sent from my iPhone using Tapatalk

 

[jgreco]

If they only require one, that is because the expanders inside have been connected together. This is not necessarily an optimal configuration because there can be contention on the SAS multilane. It's just something to be aware of. These things can be arranged in a number of ways.

 

[TinTIn]

I really appreciate the feedback. Many thanks.


Sent from my iPhone using Tapatalk

 

[Darren Myers]

I want to see pictures when this is done, it sounds awesome

 

[TinTIn]

Will do. Probably about two weeks from now.


Sent from my iPhone using Tapatalk