SOLVED DELL R720 16xHDD RAID/HBA-configuration question

[atomique90]

Hello FreeNAS-Community,

I plan to build up a new NAS Server to shutdown my old Iomega-NAS. My plan is, to buy a Dell R720 with 16 Slots (I want it 19" for my rack) and build it up for ESXi and FreeNAS.

Plan:

- Use 6 HDDs for ESXi-Datastore (RAID10)
- Use 10 HDDs for FreeNAS (I want to passthrough them to the machine, running FreeNAS)

Questions:

- Is this possible? I heard I cant configure some disks to a RAID and the other ones not
- Which RAID-Controller should be the best for this scenario? Should I split it into 2 RAID-COntrollers / HBA to get it running?

I know that I can upgrade my Dell Server with H310 and H720p Controllers, but they told me on reddit, that I cant use passthrough on the 720p and the H310 is not so good. So I thought, okay.. lets make this possible with external cards!

Available Ports in that server should be 1x PCIe x16 Full Profile - 3x PCIe x8 Full Profile - 3x PCIe x8 Low.

Idea:

- How about two Broadcom 9211-8i + 1 SAS Extender? I could do the following:

- Card0: Connect 6 Drives to create a Datastore for ESXi
- Card1: Connect SAS Extender and 10 Drives to this connector to passthrough them to FreeNAS

I know.. you may ask, why dont you use an dedicated server for that FreeNAS-Stuff? I already got an Dell R420 ESXi and I want to merge this machine to one "big" machine. Its a homelab and I dont want to run a bunch of machines - just to try to keep the costs as "cheap" as possible.

Maybe you guys can help me out with this.

Thank you!

Atomique90

PS.: Let me know if you need further information!

 

[sretalla]

The way I understand it is you would need to have 2 distinct HBAs (not an HBA + an expander) in order for ESXi to be able to allocate the chip that's doing the HBA job into passthrough mode, so for one chip to be in passthrough and another not, you need 2 of them.

You're free to experiment with the expander scenario you mention (maybe ESXi will see the expander as a separate card that's available to be enabled for passthrough... I am under the impression that it's not actually a controller though), but my guess is that you'll be returning the expander.

 

[atomique90]

Hi sretalla,

thanks for your reply :) appreciate it - I just read some threads and googled around and "found out" that the "IBM ServeRaid M1015 SATA / SAS HBA Controller" can handle up to 16 disks. What about buying two of them and connect 10 disks to raid-controller 1 (with that IT-Mode/HBA mentioned everywhere) and 6 disks in RAID 10 (normal firmware)?

But my big problem now is the following: how do I connect 16 disks to that raid controller? It has two mini-SAS connectors and I thought I only can connect 4 disks with one mini-SAS?

Thank you!

Atomique

 

[sretalla]

I think your proposal to use an expander is good assuming you don't expect to present disks attached to the same controller (through the expander or directly) to both ESXi AND FreeNAS.

If you do as you mention and have one controller to pass through to FreeNAS (and add the expander to it), then have another for direct access by ESXi, I think all will be good.

You could also simplify your life and have the second controller you buy be able to handle 16 disks on its own (4 mini-SAS ports onboard)... I use one with 6, allowing me to populate my 24-bay chassis on 1 HBA alone.

Also worth consideration... does your MoBo have SATA ports? (I would recommend using this only for ESXi, not FreeNAS in this case as passthrough for the onboard controller will usually rob you of your boot option). There's nothing really "broken" about onboard SATA controllers in terms of performance or reliability if you have enough ports available to run the disks you need.

 

[neb50]

See this thread on a discussion on how to do this on a r720xd
https://forums.freenas.org/index.ph...720xd-h310-mini-or-esxi-vm.64239/#post-460175

That thread has the information on how a couple people setup an ESXI host with FreeNAS as a VM with other VM's hosted from the FreeNAS based storage.

On a r720 non-xd model, I would suggest using the sata ports with a small ssd as a datastore for FreeNAS (if you have the SATA power connector available for the Tape backup) and then use FreeNAS to provide the ESXI datastore back to ESXI for the rest of your VM's. ESXI can be installed to a USB drive, the dual SD card's (if installed) or on the SSD. Or you can use a m.2 NVME adaptor card and drive in one of the card slots to provide your initial datastore like we did on the r720xd.

The 16 SFF bays would be connected to an HBA like this thread discussion.
https://forums.freenas.org/index.php?threads/storage-controller-suggestions.68451/#post-469066

There are also 2 versions of the 16 bay SFF r720 with one having a backplane with expander for all 16 drives using one HBA and another one that splits it into 2 8 bay backplanes with 2 HBA's.

 

[Elliot Dierksen]

There are also 2 versions of the 16 bay SFF r720 with one having a backplane with expander for all 16 drives using one HBA and another one that splits it into 2 8 bay backplanes with 2 HBA's.

If you don't mind an external device for drive bays, The HP D2700 (25 x 2.5" SAS/SATA) can be had pretty inexpensively on eBay. They support 6G SAS and SATA drives, and it works great for me with an LSI 9207-8E drive controller.

 

[atomique90]

Hey guys,

thanks again for your help :)

If you do as you mention and have one controller to pass through to FreeNAS (and add the expander to it), then have another for direct access by ESXi, I think all will be good.

Okay, but I "found" a new problem. Just was interested in the way the HDDs are connected and found this Backplane that is used inside the Dell 720 I think. If I assume that SAS A (Disk 0-7) is going to RAID Controller A I cant add the two more disks to extend my "NAS" to 10 disks because I should connect SAS B to connect it to the other RAID Controller B. It seems the best idea is to buy one Broadcom 9211-8i, connect it with two SFF-8087-Cables (right?) to SAS A and SAS B and get FreeNAS running on an extra SSD-Datastore on a SSD-NVMe-Storage? Are there any tipps hardware that are compatible with ESXi 6.7? (Just need a direction) :)

Also worth consideration... does your MoBo have SATA ports? (I would recommend using this only for ESXi, not FreeNAS in this case as passthrough for the onboard controller will usually rob you of your boot option). There's nothing really "broken" about onboard SATA controllers in terms of performance or reliability if you have enough ports available to run the disks you need.

On a r720 non-xd model, I would suggest using the sata ports with a small ssd as a datastore for FreeNAS (if you have the SATA power connector available for the Tape backup) and then use FreeNAS to prov connected to an HBA like this thread discussion.
https://forums.freenas.org/index.php?threads/storage-controller-suggestions.68451/#post-469066
ide the ESXI datastore back to ESXI for the rest of your VM's. ESXI can be installed to a USB drive, the dual SD card's (if installed) or on the SSD. Or you can use a m.2 NVME adaptor card and drive in one of the card slots to provide your initial datastore like we did on the r720xd.

Sounds like a plan. I usually install ESXi on an USB-Stick. I could install FreeNAS on a normal SATA-SSD, but I think the better option is the NVMe-SSD (because I cant install that 2,5inch SSD inside the case I think (assuming that there is no place under the DVD-Drive, where I could double-side-Tape it).

If you don't mind an external device for drive bays, The HP D2700 (25 x 2.5" SAS/SATA) can be had pretty inexpensively on eBay. They support 6G SAS and SATA drives, and it works great for me with an LSI 9207-8E drive controller.

I want to prevent using an external device for my drives. Usually I would do this, but I want it a bit compact for my homelab, but also thank you for your suggestion!

Conclusion:

- I need one RAID Controller (Broadcom 9211-8i)
- I need to SFF-8087-Cables
- I need a SSD PCI-Card (any suggestions maybe? :) )

Greetings from Germany - Atomique90

EDIT:

Additional Question: If I would buy two Broadcom-Cards (they are really cheap on ebay oO), would there be more speed gain?

 

[Elliot Dierksen]

Okay, but I "found" a new problem. Just was interested in the way the HDDs are connected and found this Backplane that is used inside the Dell 720 I think.

I am not familiar with the Dell hardware specifically (More HP and Cisco myself), but I will make an semi-educated guess. My Cisco C240 M3S servers have 24 internal 2.5" SAS/SATA bays. The 2 cables from the drive controller go to two connections on the backplane for those drives. I expect that the backplane is also a SAS expander. That means you just need the two cables from whatever "disk controller" you decide to use.

- I need one RAID Controller (Broadcom 9211-8i)

I am being a bit of the semantics police here, but you mean "drive controller" instead of "RAID controller". IMHO, "RAID controller" implies that the controller firmware is doing the RAID which is not what you want with FreeNAS/ZFS. You want a drive controller (which is what the 9211 is) and ZFS handles the RAID/redundancy part of it. I know that probably seems overly fussy, but sometimes I can't help myself...

 

[Elliot Dierksen]

Additional Question: If I would buy two Broadcom-Cards (they are really cheap on ebay oO), would there be more speed gain?

I assume you mean NIC cards, right? It depends. What kind of switch are you using? If you are using something that support ether-channel (ideally with LACP), then yes. If the switch is un-managed or not capable of some kind of link aggregation, no.

 

[atomique90]

I know that probably seems overly fussy, but sometimes I can't help myself...

Haha no, thats okay, just need to learn this. Are HBAs and "disk-controllers" the same? Because a RAID-Controller is no HBA, right?

I assume you mean NIC cards, right? It depends. What kind of switch are you using? If you are using something that support ether-channel (ideally with LACP), then yes. If the switch is un-managed or not capable of some kind of link aggregation, no.

No no, I meant that Broadcom "drive-controller". Is it faster to use two cards? SAS A -> CARD1 and SAS B to CARD2 (means 8 disks per drive-controller-card, right?)

 

[Elliot Dierksen]

Haha no, thats okay, just need to learn this. Are HBAs and "disk-controllers" the same? Because a RAID-Controller is no HBA, right?

I would say that disk controller and HBA are reasonably interchangeable terms. https://searchstorage.techtarget.com/definition/host-bus-adapter

I think you can also say that a RAID controller has to be able to do HBA functions, and then it adds RAID features on top of that.

No no, I meant that Broadcom "drive-controller". Is it faster to use two cards? SAS A -> CARD1 and SAS B to CARD2 (means 8 disks per drive-controller-card, right?)

Back to my original statement about the backplane, I don't think you would want to do that in a chassis that has (I am almost positive) a built in SAS expander. In strictly theoretical terms, IF (really big if here) your drives could oversubscribe either bandwidth of the physical connection or the controller itself (or the PCIe) slot, then yes. In the situation you are describing, I don't think that is possible or would help. Look at the FreeNAS hardware guides to see what ones are well supported and recommended. That is what lead me to the selection of a 9207-8E for my external chassis.

 

[neb50]

Hey guys,

thanks again for your help :)



Okay, but I "found" a new problem. Just was interested in the way the HDDs are connected and found this Backplane that is used inside the Dell 720 I think. If I assume that SAS A (Disk 0-7) is going to RAID Controller A I can't add the two more disks to extend my "NAS" to 10 disks because I should connect SAS B to connect it to the other RAID Controller B. It seems the best idea is to buy one Broadcom 9211-8i, connect it with two SFF-8087-Cables (right?) to SAS A and SAS B and get FreeNAS running on an extra SSD-Datastore on a SSD-NVMe-Storage? Are there any tipps hardware that are compatible with ESXi 6.7? (Just need a direction) :)





Sounds like a plan. I usually install ESXi on an USB-Stick. I could install FreeNAS on a normal SATA-SSD, but I think the better option is the NVMe-SSD (because I can't install that 2,5inch SSD inside the case I think (assuming that there is no place under the DVD-Drive, where I could double-side-Tape it).



I want to prevent using an external device for my drives. Usually I would do this, but I want it a bit compact for my homelab, but also thank you for your suggestion!

Conclusion:

- I need one RAID Controller (Broadcom 9211-8i)
- I need to SFF-8087-Cables
- I need a SSD PCI-Card (any suggestions maybe? :) )

Greetings from Germany - Atomique90

EDIT:

Additional Question: If I would buy two Broadcom-Cards (they are really cheap on ebay oO), would there be more speed gain?

I would go with one HBA for the SAS A and B connections.

If there is a DVD drive installed, there may be a standard SATA power plug available there for the optional tape drive and space to put a SSD drive under the DVD drive. If the r720xd chassis/backplane/??? didn't disable the SATA ports on the motherboard, this is what I would have done except the ssd would have been in one of the rear 2.5" bays without the rear backplane installed

If you go with an NVMe drive, you can either go with a larger drive like gallyjh did and store multiple VM's on it or a 16Gb Optane drive like I did for just the FreeNAS VM ($25 at Microcenter) and a ~$15 cheap adaptor card. The rest of my VM storage is from a pair of Intel S3500 SSD's in a mirror configuration hosted as an NFS share.

 

[atomique90]

Okay nice, now I have something I can work with - really nice. Thank you guys for your ideas and clarification! Its really hard to get into the topic.. seems like everyone only wants a 4 bay NAS from synology/qnap :D

Have a nice weekend! :)