Optimal configuration for SCALE

[ItsMeWolfE]

I'm about to build my first NAS system. I have the following hardware:
Altermiter X99-H v1.41 Motherboard with Xeon E5-2680V4
32GB (2x16) ECC REG DDR4 RAM 2133mhz
1TB Samsung 980 PRO NVME SSD
2x6TB Toshiba Enterprise 7200RPM 512MB Cache drives (Model MG08ADA600E)

I have these spare drives lying around, if there is potentially any use for them, I'd like to know;
256 GB Sandisk SATA SSD PLUS
128 GB Samsung 850 EVO SSD

What's the best I can do with this hardware using TrueNAS SCALE? I have researched a bit and currently thinking about RAIDZ1, but I was thinking I could employ the 980 pro for cache while using one of the spare SSDs for a boot drive - or perhaps using both of them for a mirrored boot?

 

[Arwen]

RAID-Z1 requires a minimum of 3 disks, and is not recommended for disks over 2TB due to the long rebuild times.

The 128GB Samsung 850 EVO SSD would make a decent boot device. And yes, you can Mirror the boot pool, though it is far more important to backup your configuration.

L2ARC Cache devices generally should not be used until you have maxed out your physical RAM. And one rule of thumb for size, is only 5 to 10 times the size of memory. Like 32GB of RAM would suggest a L2ARC Cache size of 160GB to 320GB. Their are exceptions, for example using one as a Metadata only.

Here is a suggested reading;

Welcome to TrueNAS SCALE - Beginners Intro

Since mid 2021 we have had many new users of TrueNAS wanting to try SCALE. Here are some things that are helpful for new TrueNAS SCALE users to know; SCALE is a specialized & targeted OS based on a Linux distro. However, SCALE is NOT a Linux...

www.truenas.com

What are your uses for a NAS?

Their are differences in how you would layout the disks in a ZFS pool depending on use case.

 

[ItsMeWolfE]

RAID-Z1 requires a minimum of 3 disks, and is not recommended for disks over 2TB due to the long rebuild times.

The 128GB Samsung 850 EVO SSD would make a decent boot device. And yes, you can Mirror the boot pool, though it is far more important to backup your configuration.

L2ARC Cache devices generally should not be used until you have maxed out your physical RAM. And one rule of thumb for size, is only 5 to 10 times the size of memory. Like 32GB of RAM would suggest a L2ARC Cache size of 160GB to 320GB. Their are exceptions, for example using one as a Metadata only.

Here is a suggested reading;

Welcome to TrueNAS SCALE - Beginners Intro

Since mid 2021 we have had many new users of TrueNAS wanting to try SCALE. Here are some things that are helpful for new TrueNAS SCALE users to know; SCALE is a specialized & targeted OS based on a Linux distro. However, SCALE is NOT a Linux...

www.truenas.com

What are your uses for a NAS?

Their are differences in how you would layout the disks in a ZFS pool depending on use case.

Hi, thanks a lot for the information. My uses for the NAS are just home backup and/or sync - my PC in its entirety, along with three phones of family members in the house including mine (Parents and I) using syncthing (I've heard it's the best offering, correct me if i'm wrong).

So you're basically saying I didn't really need to buy the 980 pro for this build? It was on sale and I've read a bit about SSD cache.

And at last, about the boot disk mirror, it doesn't matter that these are two different SSD models with different capacities, yeah? Because it's just a boot drive. And if it matters, both drives have been sitting unused for a couple of years, but they are supposed to be okay.

 

[ItsMeWolfE]

Hi, thanks a lot for the information. My uses for the NAS are just home backup and/or sync - my PC in its entirety, along with three phones of family members in the house including mine (Parents and I) using syncthing (I've heard it's the best offering, correct me if i'm wrong).

So you're basically saying I didn't really need to buy the 980 pro for this build? It was on sale and I've read a bit about SSD cache.

And at last, about the boot disk mirror, it doesn't matter that these are two different SSD models with different capacities, yeah? Because it's just a boot drive. And if it matters, both drives have been sitting unused for a couple of years, but they are supposed to be okay.

Can't seem to find an edit button, new to the forum, so I guess i'll just reply to this lol
My reasoning for a cache in the beginning is to make moving large files take less time- if ZFS takes care of that for me thanks to caching on the RAM (I specifically bought ECC ram for this in case someone told me an SSD cache is irrelevant), then I guess it takes care of that.

 

[Arwen]

L2ARC Cache devices can always be added after, (this is not the case for other devices, like special vDevs).

Yes, you can use 2 different sized devices for the boot pool. Just select the smaller one first.

I have no knowledge of Syncthing.

An L2ARC device is a read cache, and won't affect any writing. Except in the unusual case if L2ARC is used for Metadata Caching AND you need to access a lot of different Metadata, (for example directory entries), for writing.

You will be able to edit posts after a certain amount of post, (I don't know the number).


With your use case, you don't need a Mirrored pool. But, with only 2 x 6TB disks, that is all you can create.


Not that you can not change your pool layout easily. Or at all in most cases. If it is needed after creation, it requires a full backup and restore to fix some pool layout issues.

 

[ItsMeWolfE]

L2ARC Cache devices can always be added after, (this is not the case for other devices, like special vDevs).

Yes, you can use 2 different sized devices for the boot pool. Just select the smaller one first.

I have no knowledge of Syncthing.

An L2ARC device is a read cache, and won't affect any writing. Except in the unusual case if L2ARC is used for Metadata Caching AND you need to access a lot of different Metadata, (for example directory entries), for writing.

You will be able to edit posts after a certain amount of post, (I don't know the number).


With your use case, you don't need a Mirrored pool. But, with only 2 x 6TB disks, that is all you can create.


Not that you can not change your pool layout easily. Or at all in most cases. If it is needed after creation, it requires a full backup and restore to fix some pool layout issues.

So in short, the 980 pro is basically unnecessary for my use case? Again, I specifically bought it for the NAS because it was on sale. Not that big of a deal, I just want my NAS to be fast and responsive when it comes to accessing and transferring lots of small files.
And for my 6tb drives it's just called a mirror pool, like a raid1 but with ZFS? I've read the document you sent but I'm still not 100% sure.

 

[indivision]

Altermiter X99-H v1.41 Motherboard with Xeon E5-2680V4
32GB (2x16) ECC REG DDR4 RAM 2133mhz
1TB Samsung 980 PRO NVME SSD
2x6TB Toshiba Enterprise 7200RPM 512MB Cache drives (Model MG08ADA600E)

I have these spare drives lying around, if there is potentially any use for them, I'd like to know;
256 GB Sandisk SATA SSD PLUS
128 GB Samsung 850 EVO SSD

Much good advice and information given already.

If I had only this hardware, I would configure like this:

850 EVO SSD = Boot Drive
Sandisk SSD = Applications Pool (Where your installed server applications get installed. SSD can make a big performance difference because they do a lot of internal processing.)
2x6TB Drives = 1 Mirrored Pool (for data that need a bit more safety/redundancy)
1TB 980 = 1 Additional Pool (a bit riskier due to lack of redundancy)

TBH, the 1TB 980 probably is a bit pointless in a home NAS. Unless you get 10 Gigabit ethernet set up, this drive will not be noticeably faster than a crusty old metal drive over the network. It's speed can be helpful for running apps on it. But, it's way more size than you likely need as the apps don't really take up much space. It might be helpful to use for storing media for Plex. Then the drive speed may help transcoding speed.

Read or write cache become more relevant in cases outside of home use (many users accessing the same files, etc).

 

[indivision]

I just want my NAS to be fast and responsive when it comes to accessing and transferring lots of small files.

That SSD will not help in this regard. The bottleneck is your network speed. It will only help for processes that aren't leaving the server.

What are the specifics about the small file case you are thinking of? Small files? Or, small blocks written randomly? There are some cases I've seen where a write cache can help. But, even then, that SSD is not only way larger than necessary, it's not really the right kind of SSD. There are special SSD made for cache (more durable for repeat writing and have different features to prevent dataloss).

And for my 6tb drives it's just called a mirror pool, like a raid1 but with ZFS?

Yes.

 

[ItsMeWolfE]

Much good advice and information given already.

If I had only this hardware, I would configure like this:

850 EVO SSD = Boot Drive
Sandisk SSD = Applications Pool (Where your installed server applications get installed. SSD can make a big performance difference because they do a lot of internal processing.)
2x6TB Drives = 1 Mirrored Pool (for data that need a bit more safety/redundancy)
1TB 980 = 1 Additional Pool (a bit riskier due to lack of redundancy)

TBH, the 1TB 980 probably is a bit pointless in a home NAS. Unless you get 10 Gigabit ethernet set up, this drive will not be noticeably faster than a crusty old metal drive over the network. It's speed can be helpful for running apps on it. But, it's way more size than you likely need as the apps don't really take up much space. It might be helpful to use for storing media for Plex. Then the drive speed may help transcoding speed.

Read or write cache become more relevant in cases outside of home use (many users accessing the same files, etc).

Alright, so I completely disregarded the fact that this motherboard has only three SATA ports. It's absurd but that's what i'll have to work with. I'll probably use the 980 pro as the applications pool instead. But, there is another problem. I just assembled the system today and I can't seem to install the OS (TrueNAS SCALE). On boot, I see FIPS self test fails, along with something about the ECC ram (will try to get pics of it all), and then it shows me unable to load firmware rtl_nic, and when I try to select a disk for installation (any disk) it shows me another error spam of something along the lines of 128 bit and 0 bit - i'm just not next to the system right now, i've read something about how the setup doesn't like initialized disks and I think the 850 evo has something on it since i can boot from it, so i'm in the process of making a bootable windows USB to completely erase the partition on it. Will update if I succeeded in installation

 

[indivision]

Alright, so I completely disregarded the fact that this motherboard has only three SATA ports. It's absurd but that's what i'll have to work with. I'll probably use the 980 pro as the applications pool instead. But, there is another problem. I just assembled the system today and I can't seem to install the OS (TrueNAS SCALE). On boot, I see FIPS self test fails, along with something about the ECC ram (will try to get pics of it all), and then it shows me unable to load firmware rtl_nic, and when I try to select a disk for installation (any disk) it shows me another error spam of something along the lines of 128 bit and 0 bit - i'm just not next to the system right now, i've read something about how the setup doesn't like initialized disks and I think the 850 evo has something on it since i can boot from it, so i'm in the process of making a bootable windows USB to completely erase the partition on it. Will update if I succeeded in installation

To address limited SATA ports you may want to get an HBA like this: https://www.amazon.com/gp/product/B002RL8I7M

I've never even heard of Atermiter. Hopefully, it's some configuration issue. But, it may have a compatibility issue with TrueNAS. Is it new? If so, you may want to return this stuff and take a look at the hardware guides available here.

 

[ItsMeWolfE]

To address limited SATA ports you may want to get an HBA like this: https://www.amazon.com/gp/product/B002RL8I7M

I've never even heard of Atermiter. Hopefully, it's some configuration issue. But, it may have a compatibility issue with TrueNAS. Is it new? If so, you may want to return this stuff and take a look at the hardware guides available here.

I forgot to update but I got it all working. I just... Pressed enter instead of space to select a disk. And had to create the USB using balena etcher instead of Rufus. Anyways, the server is currently working perfectly fine, just wondering how to sync my PC and phones to it. Veeam doesn't recognize my NAS and it's also backup only, not sync, nextcloud won't let me create a user and I'm also not sure it's for my use case, and I can't seem to find how to install syncthing. What do you recommend?

 

[indivision]

My preference there is to use direct connections instead of syncing apps.

I've only tried Nextcloud from those options. When it's working, it's very similar to something commercial like DropBox. But, I've found that it's constantly updated and prone to issues between updates. So, not the most stable solution.