Making a TrueNAS SCALE server as a first IT project

[TheGeekn°72]

Hi there fellas,

I've just started on TrueNAS and since I'm a beginner in this general area of IT, I wanted to inquire People Who Know on my project and tell me if they have any advices, comments, if there's anything I should look out for while handling this and what not

I put everything in a thread on the LTT server so I'll just link it here to avoid just making doubloons across the internet : https://linustechtips.com/topic/1496088-my-first-personal-tinker-project-as-an-it-student/

I got almost all the hardware, will probably get the last of it next week, planning to RAID-Z1 3x4TB Ironwolf drives to it, fire comments away !

Geek, over and out

 

[TheGeekn°72]

Hi there fellas,

I'm getting my last orders delivered tomorrow and so inching ever closer to actual setup time, issue is, as I'm looking up info regarding RAIDing drives, I'm not quite sure if I can be confident about the content I'm reading and as such I wanted to ask to people familiar with RAID-Z and if RZ1 is actually the way to go for my planned 3x4TB config or if there was other options I missed/didn't consider
At first I was planning on using RAID 5 then realised that due to the file system used, RAID-Z1 would be better then there went the rabbit hole of googling info and having 3445864 results to read opinions from lol

Fire away, I'm all eyes !

Geek, over and out

 

[Ericloewe]

What's the actual goal here? It's hard to provide any meaningful info for an abstract tinkering thing, much like working on a car - it's not a goal, it's a means to an end.

 

[TheGeekn°72]

What's the actual goal here? It's hard to provide any meaningful info for an abstract tinkering thing, much like working on a car - it's not a goal, it's a means to an end.

I put literally everything there is to know in the post just above mate :D

 

[sretalla]

The only feedback I can see that would be relevant is that RAIDZ1 isn't ideal for disks larger than 2TB, but is preferable to RAID 5 via hardware as ZFS will not play well with that and you will eventually lose data.

You may find somewhere down the road that when one disk fails, you're unprotected against data corruption and also against a further disk failure while resilvering (which is hard on the remaining disks). If the pool were to be full-ish, that could take at least 24 hours with big disks...

 

[TheGeekn°72]

RAIDZ1 isn't ideal for disks larger than 2TB

damn, so no bueno for my 4TB HDDs or I can still go ahead with relative safety ?

but is preferable to RAID 5 via hardware as ZFS will not play well with that and you will eventually lose data.

I... lost you there. You mean hardware RAID-Z1 instead of any software RAID ? (because I wasn't planning on buying a RAID PCIe card, that stuff is expensive and I've already put quite a lot of budget on this already)

You may find somewhere down the road that when one disk fails, you're unprotected against data corruption

ok, I figured that one part, looking that up was pretty straight forward

and also against a further disk failure while resilvering (which is hard on the remaining disks). If the pool were to be full-ish, that could take at least 24 hours with big disks...

I got Ironwolf HDDs which are *supposed* to be quality NAS designed drives, did I got played like a pigeon or was it the right thing to do ?
is having a second drive failure a common thing when resilvering ? would it be better to copy all data from the "OK" drives on new ones and resilver from them ? is it even possible ?

 

[Heracles]

because I wasn't planning on buying a RAID PCIe card

No-Go here. No hardware raid. TrueNAS must see and manage physical drives itself.

I got Ironwolf HDDs which are *supposed* to be quality NAS designed drives

Indeed, a good choice and I use them myself.

is having a second drive failure a common thing when resilvering ?

More often than you think but must important is that when it happen, you can loose up to your entire pool. So even if probability is low, impact is extreme. As such, it is a pretty high risk.

Do not forget about backups. No single server, TrueNAS or other, can be more than a single point of failure. See my signature about that...

 

[TheGeekn°72]

More often than you think but must important is that when it happen, you can loose up to your entire pool. So even if probability is low, impact is extreme. As such, it is a pretty high risk.

gonna reiterate my question just in case you might have missed it : would it be better to copy all data from the "OK" drives on new ones and resilver from them ? is it even possible ?

Do not forget about backups. No single server, TrueNAS or other, can be more than a single point of failure. See my signature about that...

this is exactly why I'm making this NAS, not only to learn stuff about server management but also to make a back up of all the files I've accumulated so far, I'm not cramming any of my current drives by any means but I also wanted a headstart regarding my data storage capability anyway
and for what I've planned, I'm not even nearing 40% of my max capacity post dump and it's pretty low priority content too so eh
eventually in a few years, I'll get me enough hardware to spawn a couple high capacity NAS servers and have them in mirror together in two locations
I've always wanted to make myself a large, heavily geared server room with lots of cool tech in it lol, a pipe dream at this time but one day eh ?

 

[Heracles]

would it be better to copy all data from the "OK" drives on new ones and resilver from them ? is it even possible ?

The only proper solution is not using Raid-Z1. No matter is it resilvering a new disk from the other members in the Raid-Z1 or emptying the remaining drives in the Raid-Z1, you face the same risks :
--No protection until the task is completed
--The task itself is heavy load and may very well push the other drives to a fault
--All of these data must be retrieved without a single error because there will be no more redundancy to detect and fix it
--All drives, buses and others have a small normal error rate. To read that much data, even with such a low error rate, will statistically leaves you with a few errors. As such, despite being technically possible, there is no way to make it a proper solution.

Mirrors are way better because they penalize only a single drive when it is time to rebuild something. For critical systems, it is easy to use 3-way mirrors to increase the protection.

Raid-Z2 will avoid the heavy load of re-reading everything because the extra drive is "pre-" silvered for the failure of any other drive. Also, it will keep you safe with some redundancy left once a first drive died. Raid-Z3 will increase that protection one step further.

So there are no proper way of securing Raid-Z1 with large drives, other than destroying that pool and re-creating it as Raid-Z2.

 

[TheGeekn°72]

The only proper solution is not using Raid-Z1. No matter is it resilvering a new disk from the other members in the Raid-Z1 or emptying the remaining drives in the Raid-Z1, you face the same risks :
--No protection until the task is completed
--The task itself is heavy load and may very well push the other drives to a fault
--All of these data must be retrieved without a single error because there will be no more redundancy to detect and fix it
--All drives, buses and others have a small normal error rate. To read that much data, even with such a low error rate, will statistically leaves you with a few errors. As such, despite being technically possible, there is no way to make it a proper solution.

Mirrors are way better because they penalize only a single drive when it is time to rebuild something. For critical systems, it is easy to use 3-way mirrors to increase the protection.

Raid-Z2 will avoid the heavy load of re-reading everything because the extra drive is "pre-" silvered for the failure of any other drive. Also, it will keep you safe with some redundancy left once a first drive died. Raid-Z3 will increase that protection one step further.

So there are no proper way of securing Raid-Z1 with large drives, other than destroying that pool and re-creating it as Raid-Z2.

That makes a great deal of sense... But being an IT student, I don't have the budget to get that many drives, even the minimum amount lol
3 is already pushing it for me (at 100€ a pop, 3x4TB is the best deal I could get for my budget) add to that an SSD (for boot drive of course, didn't have one at hand) and a few spare parts here and there, I'm hoping that this low usage NAS will hold for a while til I actually get a job that allows me to dump cash into loads of gear and allow me to fund bigger projects like a proper R-Z3 pool with a dozen hi-cap drives
In the mean time, that project of mine should be way more than enough to learn how actually properly use, manage, upgrade, tweak and maintain such a system (hopefully nothing goes drastically wrong)

 

[Heracles]

Well, maybe you are shooting too high for a first project then ? How about putting in place a secured DNS infrastructure ?

Step 1
A local DNS server hosting your own local domain (like home.lab) (ex: using isc bind)
Step 2
Make it a recursive server to resolve Internet names, including access control to answer your local network only
Step 3
Implement DNS Sec validation
Step 4
Query remote DNS servers over TLS (853)
Step 5
Increase security by adding DNS sink hole to block adwares, trackers and more (ex: pi-hole)
Step 6
Increase feature by registering extra stuff in your DNS like SSH keys
Step 7
Start doing network monitoring by collecting and reviewing your DNS logs

Just with that, you can have fun for months. Also, you will not need anything more than a Raspberry PI. Of course, whatever hardware you have will be able to run something capable of that.

DNS is a fundamental in IT, network and security. To master it would be a great point to begin with.

 

[TheGeekn°72]

Well, maybe you are shooting too high for a first project then ? How about putting in place a secured DNS infrastructure ?

Step 1
A local DNS server hosting your own local domain (like home.lab) (ex: using isc bind)
Step 2
Make it a recursive server to resolve Internet names, including access control to answer your local network only
Step 3
Implement DNS Sec validation
Step 4
Query remote DNS servers over TLS (853)
Step 5
Increase security by adding DNS sink hole to block adwares, trackers and more (ex: pi-hole)
Step 6
Increase feature by registering extra stuff in your DNS like SSH keys
Step 7
Start doing network monitoring by collecting and reviewing your DNS logs

Just with that, you can have fun for months. Also, you will not need anything more than a Raspberry PI. Of course, whatever hardware you have will be able to run something capable of that.

DNS is a fundamental in IT, network and security. To master it would be a great point to begin with.

We've actually started touching that and so far we've mostly been doing an overhaul of our hardware/software knowledge and a bunch of stuff regarding tools and tricks good to master and we aren't done just yet with network planning, setup and configuration
And I already have all the equipment, might as well make something of it ! And I could do that either on my RasPi (that I'll have to unearth from somewhere in my storage) or this server since it'll have way more than enough processing power to handle that and I planned to use it as such a platform too anyway, adding services and containers and stuff like that
Also, I genuinely don't think I've aimed that high to be honest, it's probably gonna be rough but not all that hard, I'm not that helpless either when it comes to IT :p

 

[Heracles]

And I already have all the equipment

For a TrueNAS project, not really.

TrueNAS is an enterprise-grade solution that relies on enterprise-grade resources. To deploy it without these requirements is not recommended. Because you wish to learn, I think it would be important to learn how to do it properly. To have things that looks like functioning at first is not good enough in enterprise. You need to be able to guarantee that the solution will keep working under all conditions, will survive problems, will safeguard the intellectual property and more.

So if you still wish to go about storage, here is another idea :

Step 1
Deploy a Linux server using only 1 hard drive at first
Step 2
Add drives in that server and use them to create a logical raid using mdadm
Step 3
Install MinIO on that server (MinIO is an S3-compatible storage solution)
Step 4
Deploy a database in that server
Step 5
Deploy a Nextcloud service that will connect to that database and use the S3 storage for its main storage
Step 6
Deploy a reverse proxy with ACME certificate to access your private cloud remotely

Again, you have a ton of fun waiting for you on that path and again, you can build this following best practices.

 

[WI_Hedgehog]

First Lesson: Ask questions BEFORE buying hardware.

Second Lesson: Thorough planning avoids unnecessary expense. (Keep in mind "unknowns" exist, so don't over-plan either; experience will dictate what's prudent.)

Third Lesson: Buy used. Rolling a car off the lot incurs an immediate 20% depreciation, computer equipment is 30%. Buying used can get you to the same endpoint for 70% less initial investment, though your electricity and somewhat more frequent replacement costs will drive that up 20% over time (potentially a 50% over-all savings). Since you're not paying interest on debt not incurred (as you're a college student, not a business with working capital), your over-all savings is around 70%, in rough numbers.

Fourth Lesson: These are your working years. Forget about women, gaming, hobbies, and anything that doesn't solidly invest in your future. Work smart now and retire early with a young, beautiful trophy wife. You won't understand how this all works until you're 40, so take this advice now and you'll be really happy with yourself later.

Fifth Lesson: Don't get married. It's a legally-binding contract that doesn't follow the rules of law and has no up-side for men; it's a horrible decision from a business standpoint. You won't understand how this all works until 15 years after your divorce, so take this advice now and you'll be really happy with yourself later. When you're 40 with a home, vacation property, home-theater, four luxury vehicles, nobody on your @$$ about how much anything cost but rather admiring you for growing your wealth, and your pick of incredibly gorgeous women who haven't let themselves go because they're married and have no reason to do anything they don't want to, you'll realize how important this advice was. Conversely, if you're 40 with three kids, loads of debt, and a fat, nagging spouse, you'll realize how important this advice was.

Sixth Lesson: Learning how to cook is easy, meals are inexpensive, taste great, and your health will be far easier to maintain. Invest in a few basic cooking items and real ingredients after browsing a few free online recipe sites. Your friends will admire you and later on when you're 40 you'll find that cooking a meal at your place will pretty much land any woman you want. That huge fireplace you retire in front of after the meal will land you whatever else you want. (pack her a lunch in the morning and send her on her way, you've got a life to lead, when you're 40)

 

[NugentS]

WI_Hedgehog seems to be a bit cynical about life!!

Not that he's 100% wrong though (not even close)

 

[LarsR]

Damn, came to this forum for tech support, instead we get life support...

 

[NugentS]

TrueNAS is for Life (just like a puppy)

 

[TheGeekn°72]

For a TrueNAS project, not really.

TrueNAS is an enterprise-grade solution that relies on enterprise-grade resources. To deploy it without these requirements is not recommended. Because you wish to learn, I think it would be important to learn how to do it properly. To have things that looks like functioning at first is not good enough in enterprise. You need to be able to guarantee that the solution will keep working under all conditions, will survive problems, will safeguard the intellectual property and more.

So if you still wish to go about storage, here is another idea :

Step 1
Deploy a Linux server using only 1 hard drive at first
Step 2
Add drives in that server and use them to create a logical raid using mdadm
Step 3
Install MinIO on that server (MinIO is an S3-compatible storage solution)
Step 4
Deploy a database in that server
Step 5
Deploy a Nextcloud service that will connect to that database and use the S3 storage for its main storage
Step 6
Deploy a reverse proxy with ACME certificate to access your private cloud remotely

Again, you have a ton of fun waiting for you on that path and again, you can build this following best practices.

With absolutely no ill intent to ignore your advice, I respectfully disagree on whether or not I should use TrueNAS for my project.
And since I'm using SCALE which is Linux based, I can probably install, setup and use it along the steps you have listed as a learning curve.
I have a couple relations that work professionally in cybersec and storage solutions that I can rely on to help me out learn about it and a friend of mine showed me a heavily "modded" (?) TrueNAS he built for himself with services that do sounds like some of the steps you listed (that was a few months ago and I wasn't knowledgeable enough then to fully understand what I was shown but looking back there are similarities, that is, as long as I recall properly).
Again, it is not my intent to dismiss what looks to me like valuable advice, I just feel quite bitter about scrapping my carefully researched plan that took me close to two weeks of educated research and scavenging for info here and there to make something that looks like it holds its ground and validated by those same relations I mentioned earlier.
However, I thank you for your time and your replies, until next time ? I plan on updating my LTT forum thread (link in the first post) if you are interested in eyeing my progress with this project over time

 

[TheGeekn°72]

First Lesson: Ask questions BEFORE buying hardware.

Second Lesson: Thorough planning avoids unnecessary expense. (Keep in mind "unknowns" exist, so don't over-plan either; experience will dictate what's prudent.)

Third Lesson: Buy used.

First : that I did ! I looked up info about what I was planning to do for at least a solid week and pushed so far into it, I was even reading about HDD technologies and the difference between SMR, CMR and other types

Second : it may have looked like I was complaining about the cash I've put into this but so far, every single part I've acquired has been on the planned purchases list since day one, I just had other unrelated expenses ("unknowns" shall we say) that made me grind my teeth but I fixed that budget hole thankfully so I don't have any issues regarding that anymore

Third : that I did too ! 80% of the cash spent went into brand new drives (because I didn't have any and I wasn't about to play Russian roulette on ebay with NAS type HDDs that can't be certified regarding their usage duration until after delivery) the last 20% went into a refurbished system from a reliable company here that I mostly got for "spare parts" : had a full system that was perfectly workable but had the space for a singular HDD and nothing else and half the SATA ports I needed, RAM and CPU are otherwise far more than enough, so I was planning on gutting it from the good parts and put them into that low value system I got just for the bigger case and larger motherboard :p (honestly a great deal when compared to similar listings for either a full machine or just the spare parts I needed)

 

[WI_Hedgehog]

@TheGeekn°72 : Good to hear! Your system should work as intended.

As a note, you bought the data drives at the same time, they'll see the same workload, and they'll die at the same time (unless one kicks off early). If one starts dying of old age, statistically there's a 10% chance you'll lose your data before the Z1 array is rebuilt. If you had a Z2 array it's 10% of 10%, or a 1% chance. When you consider there's a statistical probability your (untested?) backup is corrupt (the % depending on many variables), you may lose some data. I've lost data intermittently over the years despite reasonable efforts to the contrary, so run Z3. (Most members here run Z2.)

The other note is ZFS heavily uses RAM for caching, RAM has a lot of errors (Zebras All the Way Down), so ECC RAM is highly suggested. Note I'm not telling you to use it, though I use it and Xenon processors for 4-bit ECC.

One final note, if you lose your OS drive your data array *might* get trashed--though I don't know how accurate that assessment is. efs3 and efs4 have tools to recover a wrecked array under LVM, I'm not so sure how ZFS will handle it. It's something to look into.