SOLVED Best HDD config for general purpose FreeNAS build (I'm desperate)

[steveroch-rs]

Not really a lottery.
- Shuck only 8TB and above, as WD has no SMR drives there. 3.5" only, as those are the ones with an actual SATA port.
- Before you shuck, run CrystalDiskInfo on the drive, make sure what's in there is something you want.
- Shuck exceedingly carefully, using a credit card or similar flexible prying tool. See YouTube videos.
- Burn that drive in well. https://www.ixsystems.com/community...for-freenas-scripts-including-disk-burnin.28/
- If there are any errors at all, return the drive to its enclosure and RMA it
- Yes, no warranty. WD Elements is only 1 year to begin with, and drive failure rate goes up after year 3 -> anything but a Pro drive is likely out of warranty by the time it fails, anyway. Exceptions are drives that fail early in their life, and that's what burn-in is there for.

Thank you for the information!
I will consider once I upgrade. For now I will populate the 12 bays with 4TB WD40EFRX to get maximum performance. Had the 6TB version for 4years now and it works great. 18TB total storage is more than enough right now for me.

 

[steveroch-rs]

I got HGST Ultrastar HE10 out of 8TB Elements, with firmware that limits them to 8TB and 5400 rpm. Helium drives. No complaints. That was late 2018, not sure what's in those now. Hence CrystalDiskInfo first.

Those are great drives I figured.
Once I require more storage I will start shucking as well.

 

[steveroch-rs]

I think it is time to draw my conclusion and bring this to a close.
Again thank you everybody for taking your time and sharing your expertise. I have never been to a forum this helpful before!

My FreeNAS box will consist of the following:

• 2x Xeon E5-2450L 1.8GHz 8C/16T CPUs
• 128GB DDR3 regECC RAM
• 12 WD 4TB drives (WD40EFRX) in 2-way mirror (+2 as hot spares)
• a mirror of 120GB SSDs as boot drives
• redundant 750W PSUs

The only questions still remaining are which HBA for the disk-array and whether I can still use the on-board SATA3 connectors for my boot SSDs?

 

[mouseskowitz]

The only questions still remaining are which HBA for the disk-array and whether I can still use the on-board SATA3 connectors for my boot SSDs?

Server the Home is my go to for questions like that. Here's their HBA article. You should be able to use the HBA and on-board connectors without any issues.

 

[Yorick]

The only questions still remaining are which HBA for the disk-array and whether I can still use the on-board SATA3 connectors for my boot SSDs?

You can use the onboard SATA, no worries there. One thing about mirrored boot drives, and that is how the BIOS chooses which one to boot from. If you have a failed drive but it’s physically still visible, BIOS will attempt to boot from it. I don’t know that there’s a good solution other than manual intervention. That said, boot SSDs are expected to last.

That case has a SAS backplane, right? Then I’d expect an IBM M1015 in IT mode (off eBay) would do well. That’s an LSI 2008 under the hood.
That serve the home list with more modern controllers is also great.
Edit: The older controller, suitable for HDD but not SSD, is around 50; a newer 9300-8i is around 300. I’d cross the SSD bridge if you ever come to it. Replacing the controller is easy, and by the time you move to SSD, prices might have come down.

 

[steveroch-rs]

Server the Home is my go to for questions like that. Here's their HBA article. You should be able to use the HBA and on-board connectors without any issues.

Thank you :) I have the complete "Top Picks" guide from STH saved in my bookmarks for future reference regarding ZIL and L2ARC etc.

 

[steveroch-rs]

You can use the onboard SATA, no worries there. One thing about mirrored boot drives, and that is how the BIOS chooses which one to boot from. If you have a failed drive but it’s physically still visible, BIOS will attempt to boot from it. I don’t know that there’s a good solution other than manual intervention. That said, boot SSDs are expected to last.

Okay then I'll use onboard for internal drives such as boot SSD. Is a SATADOM still the preferable choice over a regular cheap SSD?

That case has a SAS backplane, right? Then I’d expect an IBM M1015 in IT mode (off eBay) would do well. That’s an LSI 2008 under the hood.
That serve the home list with more modern controllers is also great.

The server comes with an HP SmartArray P420 Controller which is SAS 6G/ SATA 6G so I have to assume the backplane in that thing is SASII as well. The connector on the RAID-Controller seems to be SFF-8087.

I am not that familiar with SAS that's why I have another question: Assuming I buy an HBA with 2 SFF-8087 ports on them like an LSI-9211-8i I could run 8 SATA drives directly off them via those SFF->4x SATA splitter cables OR I run 2 SAS cables directly to my backplane which acts as a SAS expander for those 12 drives am I right?

The Proliant case has 2 drive bays in the back which I wanted to use to hold 2 hot spares an connect them to on-board SATA ports. Will there be any problem when e.g. 1 drive in my front bays running off the HBA fails and I use one of the hot spares in the rear to resilver until I can replaced the faulty drive in the front?

Using this calculator with my drives' performance data I get 1,7GB/s as throughput for 100% read. Is there going to be a bottleneck with using a 6G SATA/SAS controller?

 

[winnielinnie]

ADDING: What about configuring a separate SSD-only pool for the databases?

This is pretty much what I did, except I used two cheap 2.5" (laptop) HDDs in a mirrored vdev. My jails, plugins, user homes, logs, and system-dataset all reside on here. I named it "auxillary-pool" to make its purpose clear.

The way I see it, only my "data" matters: the stuff that is irreplaceble. I don't want all the auxillary stuff (which can easily be recreated / redone) to be intertwined with my data disks.

EDIT: I misread your post. I thought you wrote system dataset, not databses. Unless you meant the same thing?

 

[Yorick]

Assuming I buy an HBA with 2 SFF-8087 ports on them like an LSI-9211-8i I could run 8 SATA drives directly off them via those SFF->4x SATA splitter cables OR I run 2 SAS cables directly to my backplane which acts as a SAS expander for those 12 drives am I right?

Yup.

Is there going to be a bottleneck with using a 6G SATA/SAS controller?

Well first and foremost your bottleneck is going to be that 1GBit/s Ethernet link LONG before you tax either the drives or the controller.
Secondly, that controller gives you eight (8) lanes of 6Gb/s. Nope, you're not maxing that out, not with HDDs, not even with more HDDs than you have there. See https://docs.broadcom.com/docs-and-downloads/host-bus-adapters/LSISAS9211-8i_UG_v1-1.pdf for that 8 lane reference.

Edit: Found a great offer on the 9211-8i with 2 SATA breakout cables, just may need a firmware flash. https://www.ebay.com/itm/142898356107
Cross-flashing guide: https://blog.michael.kuron-germany....dell-perc-h200-to-lsi-9211-8i/comment-page-1/

 

[steveroch-rs]

This is pretty much what I did, except I used two cheap 2.5" (laptop) HDDs in a mirrored vdev. My jails, plugins, user homes, logs, and system-dataset all reside on here. I named it "auxillary-pool" to make its purpose clear.

The way I see it, only my "data" matters: the stuff that is irreplaceble. I don't want all the auxillary stuff (which can easily be recreated / redone) to be intertwined with my data disks.

EDIT: I misread your post. I thought you wrote system dataset, not databses. Unless you meant the same thing?

Thanks for replying.
No I really meant databases as in MySQL, MariaDB etc.
I have various pieces of software that require different types of database engines which I'm planning to run on a Docker host and I want to use my FreeNAS as a central storage for all of them.

 

[steveroch-rs]

Well first and foremost your bottleneck is going to be that 1GBit/s Ethernet link LONG before you tax either the drives or the controller.

I thought just in case I'll upgrade to 10GbE at some point.

Secondly, that controller gives you eight (8) lanes of 6Gb/s. Nope, you're not maxing that out, not with HDDs, not even with more HDDs than you have there.

Ah ok I didn't know that. That should be more than sufficient :D

See https://docs.broadcom.com/docs-and-downloads/host-bus-adapters/LSISAS9211-8i_UG_v1-1.pdf for that 8 lane reference.

Have read it. Thank you for helping me so much and providing information!

Edit: Found a great offer on the 9211-8i with 2 SATA breakout cables, just may need a firmware flash. https://www.ebay.com/itm/142898356107

I guess this LSI 3008 is a little overkill right? At ServeTheHome I read that the order of desirability is LSI 3200, 3008, 2308, 2008 so I thought why not get the 3008? Just in case I buy the 3008 with SFF-8643 connectors I could use SFF-8643->SFF-8087 adapter cables and the entire apparatus will drop down to the SASII speeds of the backplane without any problems right?

 

[Yorick]

I guess this LSI 3008 is a little overkill right?

Little bit, but a 100 bucks ain't bad. I didn't know they'd come down that far already. As for compatibility with the backplane, I don't have a clue. I'd expect it'd work just fine. Maybe @jgreco has something smart to say.

 

[steveroch-rs]

Little bit, but a 100 bucks ain't bad.

I thought that as well :) Can't hurt having one for the future...

As for compatibility with the backplane, I don't have a clue. I'd expect it'd work just fine.

Then I'll do some further research.

 

[Ludensen]

Regarding the 2 controller/HBA-offerings from ebay;
Am I totally out on a limb - seeing ghosts and misfortune everywhere, when I notice the "seller doesn't accept returns" and think about all the stories about counterfeit controllers from China..?
Of course you can't eliminate all risk and there's often a relationship between price and risk ;-)

Yes, I'm also tempted by the LSI 3008, but my need isn't imminent and I have learned the hard (read expensive) way;
Don't buy any HW before the need is imminent! or else you will pay too much or miss new features/technology.

steveroch-rs: But a nice server-plan! Glückwunch

 

[Yorick]

Am I totally out on a limb - seeing ghosts and misfortune everywhere, when I notice the "seller doesn't accept returns"

A little bit. eBay/Paypal protects you from "item not as advertised". A seller can say "no returns", but that only means "we don't want this back if it's what we said it was and it works". You can always get your money back if it's not the right item or it's defective.

Yes, I'm also tempted by the LSI 3008, but my need isn't imminent and I have learned the hard (read expensive) way;
Don't buy any HW before the need is imminent! or else you will pay too much or miss new features/technology.

This, a thousand times. By the time it's 2 to 3 years later and I want the functionality, something new has come along that's better than what I bought 2-3 years ago. "Future-proofing" only gets you so far in tech. It's one thing to say "these switches need to last me 7 years and I have a WiFi 6 refresh coming up in 1 year, I'll buy mGig now", and another to say "I might want to do WiFi 6 eventually, maybe in 2-3 years? Should I buy mGig now?". In the latter case, unless there's some serious cash flow / procurement issue, hold off until the project is real.

As a non-storage related example :)

 

[steveroch-rs]

Of course you can't eliminate all risk and there's often a relationship between price and risk ;-)

I will BE careful. I Just don't know what to expect price-wise for these controllers.

Yes, I'm also tempted by the LSI 3008, but my need isn't imminent and I have learned the hard (read expensive) way;
Don't buy any HW before the need is imminent! or else you will pay too much or miss new features/technology.

Yeah I just followed STH's list of desirability that's why I picked this 3008 because I will have to spend $50 in an HBA anyway.

But I think the 2008 and 2308 especially are plenty for my use-case and are cheap to get even as non-counterfeit parts.

@steveroch-rs: But a nice server-plan! Glückwunch :)

Danke! Been planning this one since November now... Learning ZFS was the hardest part.

 

[steveroch-rs]

This, a thousand times. By the time it's 2 to 3 years later and I want the functionality, something new has come along that's better than what I bought 2-3 years ago. "Future-proofing" only gets you so far in tech. It's one thing to say "these switches need to last me 7 years and I have a WiFi 6 refresh coming up in 1 year, I'll buy mGig now", and another to say "I might want to do WiFi 6 eventually, maybe in 2-3 years? Should I buy mGig now?". In the latter case, unless there's some serious cash flow / procurement issue, hold off until the project is real.

Yes I agree with that!
But I just went and said "Hey for STH this ist the second most desirable HBA so I'll Take that one!"
But I think the 2308 is still a great choice isn't it? Even the 2008.

 

[MRBIQ]

To much comments
use 7200 only if wanted make backup or normal data of PC if you have another thing best to be far from all HDD, its cheap ,but iops not very high
If you want my advice and keep your mind from all these questions if iops enough or no , if this drive give me performance or no if if if and many if best to go to SSD from Samsung maybe the price expensive right but will not think about all this also if you want use sas 15k or 10k very close from HDD iops but ssd is x50 or more this best from HDD and sas if you said I can't or dont have enough money so try to use sas 10k or 15k with raid10 always to get speed , redundancy , fast in building . If think in sas or sata always make them raid10 will lose half capacity but will get something good and be in safety side