My TrueNAS Build - HDD configuration and setup advice :)

[SmItLeR]

Hello hello :)

New user here, albeit I've spent the better part of 22 years killing/rebuilding and messing around with IT gear but for the better half of 10 years I've had very little to do with builds (have grown fond of buy, set and forget).....until now :)

So I've landed myself the following piece of kit that I'm looking at building into my home TrueNAS network/storage:

1) Cisco WS-C3750X-24P Switch
2) Cisco UCS C240 M3 server with the following specs:
- 2 x Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz
- 64gb RAM @ 1600mhz
- 4 x 1gb NIC
- 2 x PSU
- RAID LSI 9271-8i
- 10 x 900gb SAS 10k 2.5"
- 12 x 1.2tb SAS 10k 2.5"

My basic use for my server will be:
1) Home Assistant - Home automation
2) Internal Plex server
3) Data storage. I also do a bit of photo editing using Lightroom/Photoshop so I'd like to try and migrate my catalogue across to the NAS, but apart from that I won't be need huge write speeds to the server
4) NVR Home Security using 6 x POE Cameras (I've yet to settle on which cameras I'm going with, still doing my research on that). With the camera setup I'm going to be running BlueIris and programming in detection zones for recording, but I'd like to try and store a weeks worth of recordings on the server but the most important thing with the NVR setup is I'll be creating a separate VLAN later for the system to operate on. I'm thinking of dedicating 2 or 3tb worth of storage for this system (open to recommendations for more if need).

So my question is that I have no idea what I want to configure in terms of HDDs. I basically came from a 5 bay Synology system (which I'll be maintaining on the current network as just mass storage) and I have a spare 1.2tb and 900gb drives sitting in my draw as replacements, so redundancy wise I'm happy with a RAID5 style config (single HDD failure) but again, open to suggestions.

I've been reading quite a bit into the whole hardware RAID vs ZFS config and I cannot come to a conclusion as to what to run. I'm not sure if my server has any SATA slots where I could possibly slot in a small SSD I have laying around which I could use as a boot drive?

Would love some technical advice on how to proceed.

Thanks again :)

Brad

 

[SmItLeR]

Ooo forgot to ask if my system is able to pass through SMART data to TrueNAS ?

 

[jgreco]

I've been reading quite a bit into the whole hardware RAID vs ZFS config and I cannot come to a conclusion as to what to run.

Really? Because it doesn't sound like it. Everyone says not to run hardware RAID with ZFS. That's not just a TrueNAS thing, Proxmox, Nexenta, Delphix, Joyent, etc. have all warned against it.

Feel free to visit this forum's summary of the issue here:

What's all the noise about HBAs, and why can't I use a RAID controller?

1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with, a server. RAID controllers typically aggregate several disks into a Virtual Disk abstraction of some sort...

www.truenas.com

which is the most comprehensive discussion of the multiple facets of this issue that I'm aware of. Not all of the facets are "fatal" issues, but taken as a whole, you're risking your data, and that kinda goes against the point of ZFS. On TrueNAS, a properly crossflashed LSI HBA is the winning bet.

- RAID LSI 9271-8i

So this is useless. Nice controller ... for ESXi 6 or some non-ZFS use. I have a bunch of them here that are being retired because they're no longer supported in modern ESXi.

 

[sretalla]

Everyone says not to run hardware RAID with ZFS

Well, you made me go looking for an article that I read a few years back from some complete e-diot, which I couldn't even find by trying now (proving your point a bit more strongly), but if I were to summarize it in a sentence:

You ZFS guys are all dumb... my excellent hardware RAID system works just fine if I install ZFS on it and look how great it works... I get the best of both worlds.

I never saw the follow-up article which I presume was titled "How I lost all my data by not listening to people who know what they are talking about".

He was so confident he was right, he was even writing at the end that he expected "us" to try to challenge or debunk his article, but he was "ready for it".

 

[jgreco]

I stand corrected. Everyone competent to render an opinion on the topic says not to run hardware RAID with ZFS.

The fact that a thing does not instantaneously blow up in most circumstances is that most awful of fallacious arguments, because it's like riding around in your car without a seatbelt, cocksure that you're just fine. Which you are. Until you get into a head-on accident, in which case the being jettisoned out the windshield and into the crash-in-progress is not too healthy and detrimental to your wellbeing. Which admittedly doesn't happen too often. Hopefully.

The reasons not to do certain things can be subtle, such as the people who come in with 2Gbps FC disk shelves and an otherwise competent and proven robust HBA to attach it, for which the primary issue becomes the 2Gbps link for 16 drives, which is murder on scrubs/resilvers.

Hardware RAID is one of those things where yes it really ought to work, and with enough cooperation with vendors could probably be made to work, but vendors are oriented towards making profit off Windows servers and not worrying about the finer points of ZFS I/O contention, ordering, flooding, etc., issues.

 

[Arwen]

...
I've been reading quite a bit into the whole hardware RAID vs ZFS config and I cannot come to a conclusion as to what to run. I'm not sure if my server has any SATA slots where I could possibly slot in a small SSD I have laying around which I could use as a boot drive?
...

Second not using hardware RAID with ZFS.

On the subject of SATA slots, Serial Attached SCSI, (aka SAS), is 100% backward compatible with SATA II, (aka 3Gbps), and if it is SAS 6Gbps, then it's also compatible with SATA III, (aka 6Gbps). If you use a backplane slot which seems to have an expander, it more than likely does SATA tunneling over SAS protocol.

I have some of those Cisco M240 servers in production, (running RHEL though). Nice servers.

Sorry, no opinion on pool layout.


PS: The "big" boys use ZFS over hardware RAID all the time, and reliably. But, that's with EMC, Hitachi, and other serious SAN storage. And with multiple 8 or 16Gbps FC links, (as in 4 or more links for visualizing Solaris 11 using Logical Domains). (With the caveat the any ZFS detected corruption will require restores from backup...)

 

[SmItLeR]

I have been delving into all the threads here about Hardware RAID vs ZFS and I can see the consensus, but since I have barely any experience with RAID controllers or enterprise grade servers, what should I do in my case?

I'm going to look into what Arwen mentioned, but I've yet to take apart my server to find out how else to run these 22 x HDDs that I have sitting in the server.

Can anyone point me in the direction to read up more on this or figure out what specific hardware I need in order to bypass the LSI RAID controller?

Cheers all, appreciate the feedback!

 

[SmItLeR]

I did chuckle over this video re: using ZFS with Hardware RAID - https://www.youtube.com/watch?v=S_YN1vluLws&t=17s

His sarcasm is priceless !

 

[jgreco]

But, that's with EMC, Hitachi, and other serious SAN storage.

True SAN storage is typically a large (six, seven, etc) figure investment managed by a dedicated professional on hardware that is incredibly robust and competent not to be losing random blocks.

With the caveat the any ZFS detected corruption will require restores from backup..

Or you run a ZFS mirror of two SAN partitions.

This basically works out to the exact same situation I suggest in the Virtually FreeNAS post.

Virtually FreeNAS ... an alternative for those seeking virtualization

[---- 2014/12/24: Note, there is another post describing how to create a full-sized FreeNAS VM directly managing a bunch of disks. THIS post is aimed at people wanting to use FreeNAS to provide a little basic file sharing off of virtual disks on shared VM storage. ----] There's been a lot of...

www.truenas.com

You can offload the storage responsibilities but then FreeNAS loses visibility into that aspect of it. This is okay in a virtualization environment because theoretically the hypervisor (or SAN system in @Arwen 's big SAN example) is raising alarms a different way. It is not okay on a bare metal FreeNAS host with an LSI RAID controller, because nothing's monitoring the RAID controller for developing hardware problems. Plus all the related driver, etc., issues already outlined in the HBA post.

 

[Arwen]

...

I'm going to look into what Arwen mentioned, but I've yet to take apart my server to find out how else to run these 22 x HDDs that I have sitting in the server.

Can anyone point me in the direction to read up more on this or figure out what specific hardware I need in order to bypass the LSI RAID controller?

Cheers all, appreciate the feedback!

I don't have my Cisco server access handy, but if the backplane has a SAS expander, then any LSI type HBA / IT Mode controller will work.

Edit: I checked one of my Cisco servers, and it has a LSI SAS3508 Fusion-MPT Tri-Mode RAID on a chip. This seems to work okay in pass through mode, because I can get full SMART data out of the disks. We only use Linux software RAID, (and application based redundancy), so I am not sure what this chip set is capable of.

 

[ChrisRJ]

I have been delving into all the threads here about Hardware RAID vs ZFS and I can see the consensus, but since I have barely any experience with RAID controllers or enterprise grade servers, what should I do in my case?

What do you mean? Is this to be read as "should I really not use a RAID controller", as "what HBA should I get", or as something completely different?

 

[SmItLeR]

So basically what I've done so far is configure the server's LSi RAID controller to create drive groups and Virtual drives for each of the HDDs installed, then pass this through to TrueNAS where I'm assigning my pools and data sets based off what I need.

Everything seems to be working, I'm also able to access S.M.A.R.T. data from the raid controller and I haven't yet run in to any issues.

Chris: Basically I was trying to figure out how the more experienced users would assign their HDDs given my specs above. So I just went ahead with the above config for now and will do some tests and see how everything works together. I actually got a lot of info from the video I posted above, someone who's way more tech savvy than I am and has also done the Hardware RAID controller and ZFS config option like I just described.

 

[ChrisRJ]

@SmItLeR , I wish you luck with this configuration.

 

[SmItLeR]

It's a bit hard to figure out which direction to go down when I'm getting mixed reports from both sides of the fence So I can only try once direction and see for myself.

Seems to be a lot of attitude here and less "education" so to speak. Not everyone is a guru, hence the reason why I reached out for assistance, so it seems to be that it'll be trial and error in my case.

 

[jgreco]

It's a bit hard to figure out which direction to go down when I'm getting mixed reports from both sides of the fence So I can only try once direction and see for myself.

Seems to be a lot of attitude here and less "education" so to speak. Not everyone is a guru, hence the reason why I reached out for assistance, so it seems to be that it'll be trial and error in my case.

Where are you getting "mixed reports"?

You were already referred to the correct answer in #3, which includes an article that explains it in some detail.