Excited about my first build. Would appreciate your guidance.

[Haig]

Hello Everyone,

Good to be here – finally. I have been reading the forums and guidance docs for a good while now. I think I know where I am and where I would like to be with my build of my zfs Truenas machine, and am open to the collective experience and insight. It would be my first time building a nas – up until this point I have been using Qnap and Seagate nas devices as I have never had the capacity to build one.

I am setting up a new machine – it is to replace the separate nas machines I currently have in use. It is for the purpose of my own data, and valuable data, and as a result I want to be able to rely on the setup and machine fully.

My primary, and virtually only, purpose with the machine is static storage, and backups of my other machines. The most critical of data on my Truenas pools will be backed up (via rsync I believe would be the appropriate mechanism?) to another machine (I may set up a mirrored qnap offsite for such backups). I incidentally would stream content from the Truenas server, but this is so infrequent and insignificant that I do not even calculate that into any facets here.

I will be using Truenas Core, I don’t believe that I would require more than its functionality. I will be using the latest build unless there is a reason for me not to.

I have the following hardware:

Dell T340 (new machine)
Xeon E-2224 3.4GHz, 8M c ache, 4C/4T
Other than the Dell code for the motherboard I cannot find any details	DK9CR or R171RR1 or 17086-1	one of these should be the code
8gb Udimm ECC memory (this post to address this in tldr)
PERC H330 (backplane on the machine chassis for 8 hotswap hdd) – flashed to a HBA
PERC H200 card to connect the cache drive (will crossflash to HBA)
3 x 8tb SAS drives (Mirrored 3 way)	HGST -- HUH728080AL5204	8tb total in storage (24tb raw)
4 x 6tb SAS drives (Mirrored 2 way each set)	Seagate -- ST6000NM0034	12tb storage (24tb raw)
1 x 2tb SAS drive	Seagate – ST2000NM013A	2tb storage (2tb raw)
2 x 8tb drives as cold spares	HGST -- HUH728080AL5204
1 x 120gb enterprise sata ssd (boot drive)	WD 120gb Ultrastar DC SA210
1 x 200gb enterprise sas2 (6gb/s) ssd (cache)	Sandisk LB206M

All the unused (5-year-old) SAS drives that I obtained (8tb and 6tb drives) came with 520-byte logical sector emulation, but they are 4224-byte physical sector drives. The 8tb drives I have reformatted to 4096. The 6tb drives I am struggling with, but they are already convertible to 512 bytes logical. And I am speaking to the OEM in terms of conversion to 4096, so hopefully I can get them converted too. Once done I can do a burn-in on all the drives, which I will do, including the cold spares.

The Dell Poweredge T340 motherboard cannot deal with rdimms. Thus, it is udimm memory in use (but it is ecc). My understanding is that this is reliable and safe, and that most here use udimms (but ecc). I am now able to obtain memory and am mulling the difference between adding 32 or 16 gb (for a total of respectively 40gb or 24gb memory). It is 16gb modules (the motherboard has 4 memory slots). I don’t see a reason to not use the 8gb module currently in the machine, although I prefer multiples of doubled memory. But ecc udimms are pricey, so I can get over that!

Also, the T340 has only 2 onboard (gb) network ports at present, so there is a quad port Intel network card on the way too (the exact chip I do not know yet). But I can swap from another machine too if push comes to shove. I know I will not saturate this many gbe ports, but for the sake of overnight backups it won’t hurt.

Thermals should be good seeing as it is a Dell designed server (albeit a simplistic entry level server). I don’t seem to pick up any excessive heat on either the HBA or the drives up to this point, and I will monitor before I put the machine in use.

I can add in a redundant PSU, but I believe I will do so later. The machine is brand new and is being put through its paces as to burn-in already. Further, there is double power backup availability (I run an enterprise UPS and the space that we are in run a battery backup system, which backs up my UPS – seeing as we have what is called here “loadshedding”!). I love South Africa (and no, I am not being facetious).

TLDR, please give me guidance on the hardware that I am setting up.
I would appreciate your guidance on the following, please:
a) Generally the hardware compatibility with Truenas.
b) Would the PERC H330 (now a HBA330) work appropriately here? Is there consideration to be had for the version of firmware for the HBA330 (SAS3008)? If so, which version should I flash it with? I believe I read that there were concerns with a specific version on the SAS2008 hba’s but I have not been able to pick anything up on the SAS3008s?
c) I considered mirroring the boot drive, but a short downtime would not bother me sufficiently to warrant that. Instead, I have a ssd in wait (and will automate configuration backups if possible).
e) Should I stay with 24gb of ram? Or just set it up with 40gb from the get-go? I understand that there are no absolute answers here, and that it is dictated fundamentally by my use of the server. Your experienced thoughts appreciated.
f) I understand that using 4096-byte drives and 512-byte drives in the same machine would not be a concern, so long as they are not in the same pool. Is my interpretation correct? It seems logical to me too, but what do I know.
g) I want to use the storage the way I described above so that I have more redundant and less redundant storage for different purposes – 3 drives mirrored for the most valuable data, 2 mirrored for somewhat less valuable data, etc. Does setting it up like this above make sense, or are there other suggestions for such a setup? Or am I going to lose way too much storage space versus the benefit of mirroring vs striped storage?
h) From my reading the Xeon E-2224 seem up to the job. Any thoughts appreciated.
j) Any other thoughts that may add to this build? Missed anything?

Thank you in advance, sincerely appreciate all the guidance provided on this platform.

 

[artlessknave]

Other than the Dell code for the motherboard I cannot find any details

dells are usually proprietary, so only dell part numbers can replace them. these forums are usually proponents of more open hardware, like supermicro, but ultimately its server grade and if thats what you want to use, or already have, *shrugs*.

8gb Udimm ECC memory (this post to address this in tldr)

16GB is the minimum. ECC ram is ECC ram is ECC RAM. UDIMM's vs RDIMMs mostly only matter for the amount of ram they allow (RDIMM's are slightly slower but you wont notice for a homeserver - think a few nanoseconds).
you can absolutely start with the minimum and expand later if needed. if you can afford it, though, zfs will use any and all RAM available, and the more you feed it, the better it will NAS.

I understand that using 4096-byte drives and 512-byte drives in the same machine would not be a concern, so long as they are not in the same pool

no, they can be mixed, as long as you have ashift12, which has been the default on pool creations for a few years now. ashift9 makes a pool 512 only. the pool will always perform at the speed of the slowest vdev. hacking these drives I am not sure about, if they perform poorly, you might need to replace them, but if you do some burnin and testing first, you should be able to tell and decide then.

3 drives mirrored for the most valuable data, 2 mirrored for somewhat less valuable data, etc.

ultimately, up to you, but adds complexity. i would recommend one pool with 2-way mirrors, and 1 spare 8TB, OR a raidz2 of 8 drives +8TB spare,
and backups. which you already mentioned so thats good.

via rsync I believe would be the appropriate mechanism?

ZFS replication is ideal, requiring another zfs system, usually freenas, but anything with openzfs will work fine. rsync is about the best you will get otherwise, but, once you use replication, you will see how rsync pales in comparison. (rsync has to read the entire filesystem, every time; zfs snapshots already know which files changed and only sends block level time-based differentials)

1 x 200gb enterprise sas2 (6gb/s) ssd (cache)

generally, only do cache/SLOG/metadata/etc drives AFTER maxing the RAM, and only if your zfs stats show that you would benefit from them, such as if ARC (RAM cache) misses the data often. you probably wont have this unless you are really pushing the server (think databases or VMs storage under constant use).
RAM (ARC) cache is best cache. if you add L2ARC (cache disk) without enough RAM, you will probably reduce your performance. l2ARC is read ONLY, it does nothing for writes, and writes are slowest part.
there is no "write cache" disks in the zfs world, RAM is the write cache. SLOG is the closest thing but ONLY applies to sync writes, and is more for heavy use scenarios where lost writes corrupt data (think databases or VMs). zfs is Copy-on-write, data at rest is never overwritten. smb and nfs are async by default.
most people just need more RAM, and so, I would just do 2 boot drives here.

2 onboard (gb) network ports at present

how many users? for most people 2x1GB is plenty fine. 4x1GB isnt going to do much, each data stream is still limited to 1GB anyway (even in LACP, you dont get 4GB, you still get 4x 1GB).
a single 10gb would be more useful, but requires the infrastructure (10GB switch, 10GB nics in the clients, etc)
your hodgepodge spindle disks will likely max at 150MB/s anyway, plus overhead. you will probably be limited more by the pool itself than the nics.
obviously, since you say the 4port NIC is already on the way, there is no reason not to use it, but it probably won't do as much as you thought.

Would the PERC H330 (now a HBA330)

if its a true HBA, yes. anything that lets ZFS access the disks directly is fine (and isnt like cheap amazon SATA pcie cards - though, to be fair, these are what i started out with myself....)

you should get about the best performance you are going to from this, so I don't see any huge problems, mostly just tweaks and deciding how you want to config things.

 

[Haig]

Apologies for the tardy response. I designate time I can spend on each of my projects :)
Thank you for all of your insight.

dells are usually proprietary, so only dell part numbers can replace them. these forums are usually proponents of more open hardware, like supermicro, but ultimately its server grade and if thats what you want to use, or already have, *shrugs*.

Yes. I thought as much. And as to the open platforms, I agree. I was able to pick this machine up 18 or so months ago, for a song and a dance, and like Dell (albeit for its quality of equipment in most instances) and decided to run with it.

16GB is the minimum. ECC ram is ECC ram is ECC RAM. UDIMM's vs RDIMMs mostly only matter for the amount of ram they allow (RDIMM's are slightly slower but you wont notice for a homeserver - think a few nanoseconds).
you can absolutely start with the minimum and expand later if needed. if you can afford it, though, zfs will use any and all RAM available, and the more you feed it, the better it will NAS.

I think I am going to go for more rather than less. Since I can get the ram for a reasonable price now it makes sense to put the 40gb in.

no, they can be mixed, as long as you have ashift12, which has been the default on pool creations for a few years now. ashift9 makes a pool 512 only. the pool will always perform at the speed of the slowest vdev. hacking these drives I am not sure about, if they perform poorly, you might need to replace them, but if you do some burnin and testing first, you should be able to tell and decide then.

Thanks. I am definitely trying to get them all to run with logical 4096 blocks. And I will definitely also run burn-in testing on all the hardware. Just makes sense - gives that much more peace of mind.

ultimately, up to you, but adds complexity. i would recommend one pool with 2-way mirrors, and 1 spare 8TB, OR a raidz2 of 8 drives +8TB spare,
and backups. which you already mentioned so thats good.

Backups, let me get that out of the way. The data that I cannot lose will definitely be backed up, daily at that, to another mirror.
The structure of the pools I must say I am not yet finalised on in my mind. I will go read more on raidz2 and mirroring. Is there a difference in terms of loss of space on the drives in using either raidz2 vs mirroring? I understand that raidz2 you lose about 25% of the drive capacity, or am I mistaken?

ZFS replication is ideal, requiring another zfs system, usually freenas, but anything with openzfs will work fine. rsync is about the best you will get otherwise, but, once you use replication, you will see how rsync pales in comparison. (rsync has to read the entire filesystem, every time; zfs snapshots already know which files changed and only sends block level time-based differentials)

Can Truenas run as a vm with drive space provided to it? If so, I could setup a backup server for this purpose, and can then use zfs replication. Speed, I don't think, would be significant here seeing as it is onsite (at present) albeit over wireless bridge. I didn't realise zfs replication does block level backups, thanks for that snippet!

generally, only do cache/SLOG/metadata/etc drives AFTER maxing the RAM, and only if your zfs stats show that you would benefit from them, such as if ARC (RAM cache) misses the data often. you probably wont have this unless you are really pushing the server (think databases or VMs storage under constant use).
RAM (ARC) cache is best cache. if you add L2ARC (cache disk) without enough RAM, you will probably reduce your performance. l2ARC is read ONLY, it does nothing for writes, and writes are slowest part.
there is no "write cache" disks in the zfs world, RAM is the write cache. SLOG is the closest thing but ONLY applies to sync writes, and is more for heavy use scenarios where lost writes corrupt data (think databases or VMs). zfs is Copy-on-write, data at rest is never overwritten. smb and nfs are async by default.
most people just need more RAM, and so, I would just do 2 boot drives here.

Yes, this cements my thinking as to the ram. And I don't think I am then going to need the cache drive. So I can start without it. I don't read the data perpetually.
You say you would do 2 boot drives. I pretty much concluded that I don't need that, seeing as I am okay with no access to the data for a while, as long as I do not need to rebuild, but just recover (reload) the configuration onto a fresh install. The data and its integrity and safety is more important to me.
Would you suggest a dual drive would make sense here based on what you know? Also, I understand it is not a 'hot' spare boot drive, it is a cold. Thus, Truenas would not fail-over to the mirrored boot drive by itself in any case? Am I missing something?

how many users? for most people 2x1GB is plenty fine. 4x1GB isnt going to do much, each data stream is still limited to 1GB anyway (even in LACP, you dont get 4GB, you still get 4x 1GB).
a single 10gb would be more useful, but requires the infrastructure (10GB switch, 10GB nics in the clients, etc)
your hodgepodge spindle disks will likely max at 150MB/s anyway, plus overhead. you will probably be limited more by the pool itself than the nics.
obviously, since you say the 4port NIC is already on the way, there is no reason not to use it, but it probably won't do as much as you thought.

At present, me, but in future, two users. I could do fibre in future, but that would only be between compute machine and Truenas, basically.
So there is no form of teaming in Truenas (FreeBSD)? I have not looked at that component as yet - appreciate your insight.
Do SAS3 discs run out of steam at 150MB/s? The HBA I am running is 12GB/s. I didn't realise it is that slow. But anyhow, I don't really have a bigger need than that. I got the SAS discs for the reliability of the drives and not the speed.

if its a true HBA, yes. anything that lets ZFS access the disks directly is fine (and isnt like cheap amazon SATA pcie cards - though, to be fair, these are what i started out with myself....)

According to what I can see it is passing the information through to the OS - I ran tests in both Centos and in MS server to confirm just that. The card was a part of the Dell T340 purchase, so it made sense to flash it and use it. I really bought the H200 (still have to flash that one) as a backup if I do not use the ssds for cache.

you should get about the best performance you are going to from this, so I don't see any huge problems, mostly just tweaks and deciding how you want to config things.

Fantastic, that sounds promising. I will be starting the burn-in testing in the next few weeks and can then start looking at the configuration in more detail.

Thank you again for your insight. Much appreciated

 

[artlessknave]

Thanks. I am definitely trying to get them all to run with logical 4096 blocks. And I will definitely also run burn-in testing on all the hardware. Just makes sense - gives that much more peace of mind.

ZFS is 4096/512 aware, and will attempt to coalesce reads and writes into chunks that will perform the best on the drives. probably try and put the same type in each vdev if possible, but ultimately, zfs should handle this part on its own pretty well.

understand that raidz2 you lose about 25% of the drive capacity, or am I mistaken?

raidz2 loses 2 drives. that is not always 25%. in a 6 drive raidz2, you loose like 33%, but in a 10 disk raidz2, you loose, umm, 10%? math not my strong area.
2 way mirrors is always half, 3-way mirrors is always 66%
mirrors can give you more space overall if you have dramatically different sizes of drives. also note that disks AND vdevs can be added/removed with mirrors, but not with raidz2 (this might not apply to your mixed block disks though)

Can Truenas run as a vm with drive space provided to it?

technically, yes, but no, do not do this. zfs needs direct access to the hardware, putting it on top of abstractions like VM disks and RAIDs is a bad idea. there is a right way way to virtualize it, I believe its in my sig. somewhere. its generally not recommended. i ran my backup server that way for awhile, but I loathed it, though that was partly cuz I was always trying things on the host.

Would you suggest a dual drive would make sense here based on what you know? Also, I understand it is not a 'hot' spare boot drive, it is a cold. Thus, Truenas would not fail-over to the mirrored boot drive by itself in any case? Am I missing something?

it's up to you, that's why I said what *I* would do. that drive will be doing nothing really useful.
I do not understand what you mean by "not a 'hot' spare boot drive, it is a cold". mirrored drives are not spares, they are mirrors, they cannot be either hot or cold spares.
the boot drives are mirrored. either one will boot, as long as your bios is set to boot both of them. that is kind of the whole point.
that misunderstanding somewhat explains why you don't think a 2nd boot drive is useful. a mirrored boot drives means the system stays online till you get a spare resilvered (and if you have external hotswap bays, you dont even need to shut it off to replace the failed drive)
since you have no other use for the drive anyway, why not make it a boot mirror? you can detach it later if you find a use for it (I cant think of any though)

to another mirror

another mirror in the same system? that only protects against the first pool having issues, not the whole system killing your drives, being run over, or something.
if it's not huge in size, a cloud copy might be worth considering, but a backup ideally is on another system.

So there is no form of teaming in Truenas (FreeBSD)? I have not looked at that component as yet - appreciate your insight.

LAGG is the applicable term, but pretty much everything but failover LAGG requires a compatible *managed* switch. failover just sets every included interface a priority and uses whichever the first in line is that works, failing over if any die. I have mine set with failover, with 10g the primary and 1g the secondary (across 2 switches that technically have link aggregation but they are decomed amazon or something Quantas and the software is a complete undocumented labyrith). nothing else matters.
and even if you use LACP, you dont get 1 link at 4GB, you get 4 links at 4GB, because each stream across the network has to use the same interface or it becomes an out-of-order mess (windows enables this but there are whole threads about why it shouldnt and why freeBSD doesnt)