Bad JBOD Controller Experience

[Adam Tyler]

Both support more than 8 drives but only when using a SAS expander.

"SAS" expander is new speak for me.. Is this just a different kind of cable or....? Can you use a SAS expander with SATA drives?

I ended up purchasing these cables originally: https://www.amazon.com/gp/product/B012BPLYJC/ref=oh_aui_detailpage_o02_s00?ie=UTF8&psc=1

So I have them on hand. Bad move? Could I use a different kind of cable and potentially connect a few SSDs and be better prepared for expansion?

I just purchased this controller: https://www.ebay.com/itm/New-in-Sea...e=STRK:MEBIDX:IT&_trksid=p2057872.m2749.l2649

Thanks for your help guys!

 

[Chris Moore]

Does the 9207-8i actually support more drives than that?

See this server chassis:

It has two SAS expander backplanes that are each able to accept 24 drives. Each backplane has an "out" port that can be connected to another SAS expander in another chassis. All the drives on all the SAS expanders can 'daisy chain" back to a single SAS controller, up to the number of drives the controller is able to handle. I can't say from memory what those limits are, but I know the 9207 chip supports over 200 drives.

 

[Chris Moore]

"SAS" expander is new speak for me.. Is this just a different kind of cable or....? Can you use a SAS expander with SATA drives?

Yes. My server (pictured above) now has 32 SATA drives installed on the SAS backplane and room for 16 more.

So I have them on hand. Bad move? Could I use a different kind of cable and potentially connect a few SSDs and be better prepared for expansion?

You would need a SAS expander to connect more than 8 drives to the SAS controller.
Do you need SSDs? What would you use them for?

I just purchased this controller

It is a fine one. I have two like that.

 

[Johnnie Black]

"SAS" expander is new speak for me.. Is this just a different kind of cable or....? Can you use a SAS expander with SATA drives?

An expander can be included in a backplane like Chris posted, or used separately, and both support SATA devices, e.g.:

 

[Adam Tyler]

Money! We are on the same page now. Thank you for the photo.. I get it! So the expander isn't an option for me at the moment. I've already built my NAS/SAN PC into this chassis:
https://www.amazon.com/gp/product/B01MA6NJE8/ref=oh_aui_detailpage_o03_s00?ie=UTF8&psc=1

I am going to use this for a 2 node VMware ESXi cluster @ home connected via iSCSI and multipath IO. VMware has this great feature called flash cache where you can actually place the SSDs directly into the host. It is a "write through", but "read cache" you can enable on a per VM basis. Because I only have 2, 1 Gb NICs connected to the array, I think I will probably go that route before allocating SSDs directly into the SAN.

 

[Chris Moore]

Money! We are on the same page now. Thank you for the photo.. I get it! So the expander isn't an option for me at the moment. I've already built my NAS/SAN PC into this chassis:

It depends entirely on what you want to do with it. If you want to add some more drives, there are the 5.25" bays at the top. They make an adapter like this:
https://www.newegg.com/Product/Product.aspx?Item=N82E16817994095

Put some SSDs or laptop size drives in there...
What system board did you use?

I am going to use this for a 2 node VMware ESXi cluster @ home connected via iSCSI and multipath IO.

I think we should have discussed this earlier. You definitly want to put some SSDs in there, but not for ESXi. You will need a SLOG device and possibly an L2ARC and those would both be part of you ZFS pool.
We need to revisit some design concepts here.

 

[Adam Tyler]

An expander can be included in a backplane like Chris posted, or used separately, and both support SATA devices, e.g.:

This photo is showing a card that actually connects to the motherboard... Hmm.. So this has an "in" from the controller and allows you to connect multiple break out data cables from there?

Do the expanders built into the case actually connect to the motherboard in some way or only to the controller...?

 

[Chris Moore]

Please give a full rundown on your hardware.

 

[Chris Moore]

This photo is showing a card that actually connects to the motherboard... Hmm.. So this has an "in" from the controller and allows you to connect multiple break out data cables from there?

Do the expanders built into the case actually connect to the motherboard in some way or only to the controller...?

That card can either connect to the system board for power OR (not both) be connected to power by the Molex connector at the top of the card. The data all travels through SAS cables. It doesn't need to connect to the system board at all.

 

[Chris Moore]

This is the best photo I could find quickly.

 

[Adam Tyler]

Full rundown of hardware.. Well it has been a long emotional road. Lol.. Trying to stick to a budget and I had no idea how many "ifs", "ands", "buts", I would run into. I began using ZFS with NAS4Free originally and it had just been rock solid. Used 6 drives on the motherboard and never really had to deal with much to get all working. Some CLI, but meh... I decided to go with FreeNAS on this new build because it seems to have more development support and use the same underlying technology.

So, you have the case:
https://www.amazon.com/gp/product/B01MA6NJE8/ref=oh_aui_detailpage_o03_s00?ie=UTF8&psc=1

Bought a motherboard from Supermicro originally that required a XEON and ECC RAM.. After figuring out that ECC RAM was going to cost an arm+leg, I returned the board and bought one that was more geared for a desktop. Ended up with this mobo and a compatible i5 processor.
https://www.ebay.com/itm/SuperMicro...e=STRK:MEBIDX:IT&_trksid=p2057872.m2749.l2649

At least it still has dual NICs on the board. I was worried that the x16 slot wouldn't support anything other than a video card, but Supermicro actually tested this for me. It will work with a controller. PCIe 2 only I think.

I had 16 Gb RAM laying around in two 8 Gb modules. I know ZFS likes RAM, I may upgrade that to 32 down the road. Not ECC, I realize that isn't ideal for the NAS, but should work.

We've talked the controller to death, will end up with 9207-8i..

I got the SATA break out cables:
https://www.amazon.com/gp/product/B012BPLYJC/ref=oh_aui_detailpage_o02_s00?ie=UTF8&psc=1

An 800w Corsair PSU..


And a partridge in a pear tree..! :)

 

[Adam Tyler]

This is the best photo I could find quickly.

Wild. Thank you! This and the previous photos make it very clear. Wonder why that expander has PCIe connections if just a pass-through type device....? Maybe it CAN function as an independent controller?

 

[Adam Tyler]

That card can either connect to the system board for power OR (not both) be connected to power by the Molex connector at the top of the card. The data all travels through SAS cables. It doesn't need to connect to the system board at all.

Disregard my last post. I just saw this. Thank you. Very clear now.

 

[Chris Moore]

Bought a motherboard from Supermicro originally that required a XEON and ECC RAM.. After figuring out that ECC RAM was going to cost an arm+leg, I returned the board and bought one that was more geared for a desktop. Ended up with this mobo and a compatible i5 processor.
https://www.ebay.com/itm/SuperMicro...e=STRK:MEBIDX:IT&_trksid=p2057872.m2749.l2649

I hope you will consider returning that or selling it on eBay and getting a server board like this instead:
https://www.ebay.com/itm/Super-Micr...ATX-server-motherboard-with-i-o-/272782545683
It will work with the same i5 and RAM you already have, but it gives you more PCIe slots to put expansion cards in and has IPMI for remote management. It like these boards so well that I have three of them myself. They are a little older, but they get the job done.

On to the "new" issue. With iSCSI, the thing that 16 of the 32 drives in my server are dedicated to, you can't fill it past 50% without the performance tanking. Also, if you want latency to be reasonable, you will want to have two things at the minimum. First, a SLOG (Separate Log Device) and you will need a fast connection (10GB) to the ESXi systems.

Rather than tell the same story that has already been told, I would ask you to do some reading and post back here with your questions.

Build Report: Node 304 + X10SDV-TLN4F [ESXi/FreeNAS AIO]
https://forums.freenas.org/index.ph...node-304-x10sdv-tln4f-esxi-freenas-aio.57116/

Testing the benefits of SLOG
https://forums.freenas.org/index.php?threads/testing-the-benefits-of-slog-using-a-ram-disk.56561

 

[Chris Moore]

PS. I don't want you to think you must go with the absolute fastest SLOG device out there. So you might want to take a look at this also:

https://forums.freenas.org/index.php?threads/multiple-volumes-or-not.45545/#post-308867

https://forums.freenas.org/index.php?threads/slow-writes-on-ixsystems-hardware.46032/page-3

https://forums.freenas.org/index.php?threads/looking-for-your-input-microsoft-lab-hyper-v.62365

There are many others if you want to do a little searching.

 

[Adam Tyler]

I hope you will consider returning that or selling it on eBay and getting a server board like this instead:
https://www.ebay.com/itm/Super-Micr...ATX-server-motherboard-with-i-o-/272782545683
It will work with the same i5 and RAM you already have, but it gives you more PCIe slots to put expansion cards in and has IPMI for remote management. It like these boards so well that I have three of them myself. They are a little older, but they get the job done.

On to the "new" issue. With iSCSI, the thing that 16 of the 32 drives in my server are dedicated to, you can't fill it past 50% without the performance tanking. Also, if you want latency to be reasonable, you will want to have two things at the minimum. First, a SLOG (Separate Log Device) and you will need a fast connection (10GB) to the ESXi systems.

Rather than tell the same story that has already been told, I would ask you to do some reading and post back here with your questions.

Build Report: Node 304 + X10SDV-TLN4F [ESXi/FreeNAS AIO]
https://forums.freenas.org/index.ph...node-304-x10sdv-tln4f-esxi-freenas-aio.57116/

Testing the benefits of SLOG
https://forums.freenas.org/index.php?threads/testing-the-benefits-of-slog-using-a-ram-disk.56561

I think I am done spending money on this rig.. Seems like I could drop tons more.. lol... Good suggestions for future upgrades... I wish I could do 10Gb connectivity, but I am stuck at 1 Gb unless you know where I can get some cheapo gear? I would need 10 Gb NICs in both my ESXi hosts and some kind of switch to connect them. I have tons of 1 Gb ports, maybe I could do more 1 Gb and load balance iSCSI across more links. If I did my math correctly, my 8 WD RED drives mirrored and stripped with ZFS should just about soak a 2 Gb link if each mirror is doing 60 MB/s..

I do want IPMI, I will check that board out.

As for the SLOG.. I am using a 128Gb SSD drive for the FreeNAS boot volume. Will that not work?

Regards,
Adam Tyler

 

[danb35]

As for the SLOG.. I am using a 128Gb SSD drive for the FreeNAS boot volume. Will that not work?

No, the SLOG needs to be its own device.

 

[Adam Tyler]

No, the SLOG needs to be its own device.

Crap.. Well. Thanks for letting me know.. Should I install the FreeNAS OS on its own older spinny disk and dedicate the 128 Gb SSD to SLOG? Would that work or do I need more SLOG space?

Regards,

 

[Chris Moore]

I am stuck at 1 Gb unless you know where I can get some cheapo gear?

I will share what I have done, but it will still cost some money. If you enjoy tinkering it is a fun thing to work on.
Watch these videos on the YouTube:

https://www.youtube.com/watch?v=MgNpI6VAAhI&t

https://www.youtube.com/watch?v=KHiucyRYX_w&t

https://www.youtube.com/watch?v=p39mFz7ORco&t

Crap.. Well. Thanks for letting me know.. Should I install the FreeNAS OS on its own older spinny disk and dedicate the 128 Gb SSD to SLOG? Would that work or do I need more SLOG space?

If you want, you can do what I did and use a pair of laptop drives in a mirror to install FreeNAS on. The FreeNAS installer handles setting up the mirror and I picked up a pair of "New, old stock" 40GB laptop drives for about $8 each on eBay.
The SSD you have may not be ideal, but you can use it. I would suggest getting better hardware as quickly as you can afford it.

Any time you have a question, just ask and someone here will try to help.

 

[Chris Moore]

or do I need more SLOG space?

PS. No, the amount of space that will actually be used for SLOG is limited by the network speed and even with 10GB, you will probably never use more than about 32GB and that is an extreme outside maximum. The point of SLOG is to mask latency by making the initial write to fast SSD instead of slow HDD and then the system can move the data over to the HDD in a normal transaction group, 'behind the scenes' instead of making the VM wait.