New member, little knowledge, 1st time build NAS, compatibility, need help/advise

[mariosk9gr]

Hello everyone,
So glad that I pull the plug to build my 1st NAS system and I would like your help to guide me regarding hardware.
So, first of all I have 2x hackintoshes which I use for my work as a video editor and colorist. I work with huge files most of the times like braw, canon CRL, r3d raw etc.
My main issue the last couple of years is that I'm out of space very often and mainly my HDDs are not fast enough to handle the work I'm doing. So workflow goes from doing my editing/coloring stuff in NVMEs and then offload the projects to HDDs. It's a pain process and many times I need to copy again a project from my backup HDDs to NVME to make changes/corrections. So, finally I took the decision that I need a NAS system which can connect through 10Gbit on both of my systems so they can work simultaneously (for example 5Gbit per system). Im searching for the last couple of weeks for a ready to go NAS like from Qnap or Synology but I haven't found anything that can provide true 10Gbit speeds sub 1000€. For example Qnap TVS-672xt is plus 2000€ and this price w/o disks ofc. I have a small company and I cannot pay that amount of money so I decided to build it myself. I have experience building systems for many years but not very good experience with networks. 1st step was to visit truenas and check hardware requirements. After that the hardware parts I chose are these:

Intel i3-10320
Asus Prime B560K-M
Corsair 16Gb DDR4 2400 C14 (1 module for start and if needed I can upgrade to 32Gb)
Be Quiet! Pure Power 11 400w Gold+ Certified
Fractal Design Node 804
Chelsio T520 CR T520-CR 10GbE 2-Port PCIe Unified Wire Adapter Card 110-1160-50 (which I found on eBay for 105€)

On my hacks I will use 2x Gigabyte GC-AQC113C or 2x Asus XG-C100C which are 10Gbit ofc and are compatible under macOS.

So, the critical question for start is If I'm going to have any hardware incompatibility issues? Mainly I'm asking about Chelsio T520 CR and if it's compatible with latest truenas OS.
Second, what do you think about the parts I chose.. are them enough to give straight raw power of 10Gbit on both of my systems. Is i3-10320 strong enough to handle it?

I forgot to mention that I will use 5x Ironwolf 8Tb/12Tb (7200rpm/256mb cache) HDDs in RAID5 mode (motherboard has 6x sata ports which I can use them all in the future). Also I won't use a switch for start and my goal is to connect ny NAS directly to my hacks and work it as offline file server for start... maybe later I will experiment with apps and virtualisation but for now all I need is a fast file server with 10Gbit raw performance to handle my files for my editing work.

P.S. I hope I didn't make you bored with all of my noob questions! Please do not hesitate for any suggestions except that I don't have the luxury to go far away from my budget!

Best Regards,
Marios

 

[NugentS]

1. Gamer Gear - this is a general comment
2. 16GB is sorta close to minimum memory for TN. Not ECC (this will not stop it working)
3. NIC is a good one. I am assuming its in your sole PCIe 3.0 x16 slot
4. An HDD is capable of 150MB/s approximately. This is about 1.2Gb/second. In RAIDZ1 (no such thing as RAID5 in ZFS) you get the performance of a single disk although transfer rate of large sequential files will go up "a bit". But this will not in any way flood a 10Gb NIC. This is your bottleneck.
5. How are you booting this NAS as you don't specifiy a boot device. You do have a single M.2 PCIe3.0 x4 Slot the PCIe 4.0 x4 slot being disabled due to the CPU

Given 4. above - your only hope is to hold your working set in memory (ARC) so you can read at 10Gb (still can't write) - but you only have 16GB and have stated "huge files". Your NAS design is severly bottlenecked for your use case.

 

[mariosk9gr]

1. Gamer Gear - this is a general comment
2. 16GB is sorta close to minimum memory for TN. Not ECC (this will not stop it working)
3. NIC is a good one. I am assuming its in your sole PCIe 3.0 x16 slot
4. An HDD is capable of 150MB/s approximately. This is about 1.2Gb/second. In RAIDZ1 (no such thing as RAID5 in ZFS) you get the performance of a single disk although transfer rate of large sequential files will go up "a bit". But this will not in any way flood a 10Gb NIC. This is your bottleneck.
5. How are you booting this NAS as you don't specifiy a boot device. You do have a single PCIe3.0 x4 Slot the PCIe 4.0 x4 slot being disabled due to the CPU

Given 4. above - your only hope is to hold your working set in memory (ARC) so you can read at 10Gb (still can't write) - but you only have 16GB and have stated "huge files". Your NAS design is severly bottlenecked for your use case.

1.Yes I understand that the gear is not what it should be in first place but I don't want to spend so much money for a supermicro mobo with support of ECC, including cpu and ram.
2.What is TN? I didn't expect 16Gb of Ram to be not enough. I can buy another stick (motherboard supports 2) for 32Gb total. Is it going to be enough and of not why when this NAS will work only as a file server to store my projects and do editing/coloring from there?
3. I haven't build the NAS yet but I'm ready to buy the parts in the next couple of days. Motherboard has 1 pci-ex and 2 pci slots. I won't use gpu so yes I can put Chelsio T520-CR into pci-ex slot.
4/5. So, with 5 disks in RAIDZ1 what is the total maximum speed I can get if you can speculate? I have seen ready servers from Qnap that can catch 900mb/s with 4 disks and they are also underwhelmed compared to my configuration. For example Qnap TVS-472xt can get these speeds. Maybe it's the ssd caching that provides these speeds? If yes, the motherboard I chose has also 2 m.2 slots where in the 1st one I will use the wn770 250gb for OS and the other I can use for SSD caching. What speeds can I get with the addition of NVME for caching?

Can you please provide/suggest without exceeding the budget a lot what would you recommend for a budget dual 10Gbit NAS?
Thank you for your help!!

 

[jgreco]

I didn't expect 16Gb of Ram to be not enough.

Sorry to hear it. TrueNAS is based on ZFS, and ZFS is resource-hungry. Where your little QNAP or Synology unit might think of 16GB as "lots" of RAM, for TrueNAS, it's really the minimum. 128GB is a reasonable amount of RAM for many high performance uses, might be a bit tight for video editing, hard to say.

4/5. So, with 5 disks in RAIDZ1 what is the total maximum speed I can get if you can speculate? I have seen ready servers from Qnap that can catch 900mb/s with 4 disks and they are also underwhelmed compared to my configuration. For example Qnap TVS-472xt can get these speeds.

Can it? In optimal (possibly even staged) conditions, possibly. Hard drives are inherently a performance issue, and you're not actually going to get 150MBytes/sec sustained out of most hard drives. That works out to 750MBytes/sec which is well less than 10Gbps, much less dual 10Gbps.

If yes, the motherboard I chose has also 2 m.2 slots where in the 1st one I will use the wn770 250gb for OS and the other I can use for SSD caching. What speeds can I get with the addition of NVME for caching?

If you have sufficient ARC and L2ARC, you can cache virtually all of your working set. But you need a SUBSTANTIAL amount of ARC to support enough L2ARC to manage this little trick. People who do this find ZFS read speeds to blow everything else out there out of the water though. But we might be talking numbers like 256GB ARC and 1TB L2ARC, and even then, you have to remember that for a cache to work, you have to read the data out of the pool and write it into the cache first.

 

[mariosk9gr]

Sorry to hear it. TrueNAS is based on ZFS, and ZFS is resource-hungry. Where your little QNAP or Synology unit might think of 16GB as "lots" of RAM, for TrueNAS, it's really the minimum. 128GB is a reasonable amount of RAM for many high performance uses, might be a bit tight for video editing, hard to say.



Can it? In optimal (possibly even staged) conditions, possibly. Hard drives are inherently a performance issue, and you're not actually going to get 150MBytes/sec sustained out of most hard drives. That works out to 750MBytes/sec which is well less than 10Gbps, much less dual 10Gbps.



If you have sufficient ARC and L2ARC, you can cache virtually all of your working set. But you need a SUBSTANTIAL amount of ARC to support enough L2ARC to manage this little trick. People who do this find ZFS read speeds to blow everything else out there out of the water though. But we might be talking numbers like 256GB ARC and 1TB L2ARC, and even then, you have to remember that for a cache to work, you have to read the data out of the pool and write it into the cache first.

As I understand I have to read a lot before I buy the parts and then regret it!
My thought was to build a NAS so 2x systems can do simultaneously video editing work. So backup with efficient RAIDZ1 and speed with 10Gbit network cards. That was my main goal. I would be using it as an offline file server directly attached to my systems. No switches, routers, special cables, etc. In reality I need a DAS but DAS cannot support 2 systems! That's why I concluded to build a NAS. So I thought it would work just like when you setup a hardware RAID configuration from your system or as a big fast external disk which connects to 2 systems and provides reliability and speed. Now I understand it's much more complicated...

 

[NugentS]

Now I understand it's much more complicated...

It is and it isn't. TrueNAS is designed for business grade hardware. We are just lucky that it runs on consumer grade hardware reasonably well with some caveats.

HDD's are slow. RAIDZ isn't geared to performance, its geared to bulk. Mirrors are geared to performance measured in IOPS. Sometimes a large RAIDZ array can beat mirrors, but only if the data is held sequentially on the array which isn't something you can rely on. Also 150MB/s is a good day for an HDD. They get a lot slower once they start to fill up.

What you need is either:
1. Lots of additional vdevs, but that will tend to mean lots of additional disks if you want bulk space as well
2. An Array of SSD's. SSD's are rated at say 600MB/s = 4.8Gb/s and much lower latency OR
3. An array of NVMe drives - but we are into bifurcation or switched cards here - which is a whole different subject (and likley expensive). You will also need another PCIe x16 slot and associated lanes to utilise the NVMe drives properly.

Given you want to edit from the NAS then I suggest the following as a possible solution. Its not ideal as you will see:

Have two arrays, 1 Bulk HDD for long term storage, 1 of SSD's or NVMe for working from. Move data between the two as required. Edit from the SSD's and store on the HDD's

The problem you have here is that you don't have enough SATA slots OR anyway of adding more as you have used your only useful PCIe Slot. To add additional SATA you need a PCIe x8 slot - which you have already used for the NIC. Solution better motherboard.

A motherboard with 2*16 slots should run both electrically as x8 (make sure) which is all you need. One slot for the NIC and one slot for an LSI HBA 920x-8i flashed to IT Mode. Do not be tempted to use one of the shitty PCIe x1 cards that supplies 'n' SATA ports - thats a good way to lose data. Buy the LSI card second hand from a dismantler. Be very careful of anything from China - it may be a fake

Both cards will require significant cooling. If they cook they will corrupt data (and break). This isn't quite so critical on the NIC but is absolutely vital on the HBA to avoid corrupted data.

Lastly can you fit enough disks (and keep everything cool) in the Node 804? I don't know

ECC isn't compulsory, but it is considered important. Your chosen CPU can support 128GB of memory - but that will be expensive as the modules get bigger. Consider second hand memory but (as with new or second hand) test it thoroughly using mentest for a minimum couple of days before using in anger.

Lastly if you use SSD's then check carefully what their long term performance is. For example the Samsung QVO is utter utter shit at doing anything other than reading from (at which it is OK) but writing to it, performance swiftly drops off a cliff - not what you need. They are cheap for a reason
------------
Oh the reason you get such good numbers from QNAP / Synology is that they:
1. Use SSD's
2. Use stripes / RAID 0

 

[AOK]

And what is TN?

Thanks in advance for the answer.

 

[WN1X]

TN is TrueNAS

 

[NugentS]

BTW - if you use a multi vdev array of SSD's to get your 10Gb then a 9200 probably won't cut it - you will need to move to the 9300 HBA range.
9200 is great for HDD's but not so hot for a bunch of SSD's

 

[mariosk9gr]

I still can't understand why cpu,ram are so important when in my case I wont use a plex media share but I will use it only as an offline file server. All the heavy lifting goes to my system (gpu, ram, cpu, etc) and NAS is only to provide fast access to my data. What am I missing here? I'm reading about transcoding, that some heavy video files won't play because of the lack of cpu power and I'm still wondering why I can't use it just as file server that the only thing it has to do is to backup and work with video files. Just like an internal array of HDDs into a system!

 

[jgreco]

I still can't understand why cpu,ram are so important when in my case I wont use a plex media share but I will use it only as an offline file server. All the heavy lifting goes to my system (gpu, ram, cpu, etc) and NAS is only to provide fast access to my data. What am I missing here? I'm reading about transcoding, that some heavy video files won't play because of the lack of cpu power and I'm still wondering why I can't use it just as file server that the only thing it has to do is to backup and work with video files. Just like an internal array of HDDs into a system!

You've already been given the answer; ZFS is heavy on resource consumption. If you want fast access to your data, you need it to have lots of cache (ARC), lots of CPU (to handle the RAIDZ and checksums), and lots of I/O capacity (to keep everything flowing smoothly).

Lots of people come to TrueNAS having never owned a filesystem larger than maybe a few terabytes. Just like any other filesystem, TrueNAS requires stuff like metadata to keep track of your files and where the free space. Unlike most other filesystems, ZFS may be trying to keep track of that for a filesystem that could be a petabyte(!) or larger, and part of the design tradeoffs Sun made was to make it super-scalable by not making it dependent on things like hardware RAID. It's all done in software. All the things that be done in hardware assist on a conventional Windows server are instead done in software in ZFS. That's why a ZFS system can be scaled to thousands(!) of hard drives. You are trading off the compute and memory resources (relatively inexpensive) along with inexpensive I386 I/O resources like ethernet and HBA, and you're not paying six figures for the custom silicon solutions of the big SAN vendors.

Put another way, go try it. It'll probably work, as long as you're generally following the rules. I have a 140TB pool with only 32GB of RAM on one system here, which is way below recommended limits. I do it because I understand the tradeoffs I'm making and why it works. It isn't fast, but for my uses I don't need it to be. It will never transfer data at 10Gbps. But it is just fine doing a few Gbps.

 

[Davvo]

I still can't understand why cpu,ram are so important when in my case I wont use a plex media share but I will use it only as an offline file server. All the heavy lifting goes to my system (gpu, ram, cpu, etc) and NAS is only to provide fast access to my data. What am I missing here? I'm reading about transcoding, that some heavy video files won't play because of the lack of cpu power and I'm still wondering why I can't use it just as file server that the only thing it has to do is to backup and work with video files. Just like an internal array of HDDs into a system!

Quoting the Introduction to ZFS:

To this end, ZFS is completely Copy-on-Write (CoW) and checksums all data and metadata.
Checksums are kept separate from their blocks, so ZFS can verify that data is valid and that is it the correct data and not something else that your evil disks or storage controller sent your way.
When ZFS detects an error, it immediately corrects it if possible.

This is an enormous work. If you want to understand more, you have the entirity of the internet (including our lovely jgreco) to help you do so.

 

[Etorix]

Hello everyone,
So glad that I pull the plug to build my 1st NAS system and I would like your help to guide me regarding hardware.
So, first of all I have 2x hackintoshes which I use for my work as a video editor and colorist. I work with huge files most of the times like braw, canon CRL, r3d raw etc.

Welcome here!

Without repeating in full what I wrote on the Hackintosh side, and because you already got lots of good advices above:

Intel i3-10320
Asus Prime B560K-M

It will work, but if you're buying parts to build, going for actual server-grade hardware is better.
(From earlier posts by Greek users, I understand that the local market is limited and there are few opportunities for second hand tough.)

Be Quiet! Pure Power 11 400w Gold+ Certified
Fractal Design Node 804

PSU is adequate for just 5 drives, but not for a full Node 804 case. Adding a second vdev would double performance, so this is an option to keep in mind.

Proper Power Supply Sizing Guidance

I've seen about 1,000 threads like this one where people decide that they can power a dozen hard drives off a 360 watt supply. DO NOT DO THIS. I've seen another 1,000 threads where people decide to buy the cheapest power supply that they can...

www.truenas.com

Mainly I'm asking about Chelsio T520 CR and if it's compatible with latest truenas OS.

Yes.

Second, what do you think about the parts I chose.. are them enough to give straight raw power of 10Gbit on both of my systems. Is i3-10320 strong enough to handle it?

CPU is fine. 16 GB RAM is fine for 1 GbE link, but is likely to fall short for a 10 GbE link.
The HDD array will not saturate a 10 GbE link. Can you put hard numbers on the red/write throughputs you need?

How large is a typical working set of RAW files? What total capacity do you need?

Also, is the NAS intended just to serve the rushes you currently work on, or is it intended for long term archival of the finished work?
Raidz1 (ZFS equivalent to RAID5) is only moderately secure against hardware failures; it is fine for short-term work on RAW files which can be loaded anew from the original cards or from a backup. For long-term archival, raidz2 (or raidz3) would be recommended (plus a regular external backup of the NAS…).

I still can't understand why cpu,ram are so important when in my case I wont use a plex media share but I will use it only as an offline file server. All the heavy lifting goes to my system (gpu, ram, cpu, etc) and NAS is only to provide fast access to my data. What am I missing here? I'm reading about transcoding, that some heavy video files won't play because of the lack of cpu power and I'm still wondering why I can't use it just as file server that the only thing it has to do is to backup and work with video files. Just like an internal array of HDDs into a system!

Ninja'ed by our ever-helpful Resident Grinch…
CPU is fine. SMB (if that's how you'd share to Mac clients) is a single-threaded affair; high clocks with few cores are enough. But ZFS loves RAM and makes all it can of it to cache data and metadata. There's a rule of thumb of 1 GB RAM by TB of data—very roughly counted, and relaxing for large arrays. Note that (my emphasis):

I have a 140TB pool with only 32GB of RAM on one system here, which is way below recommended limits. I do it because I understand the tradeoffs I'm making and why it works. It isn't fast, but for my uses I don't need it to be. It will never transfer data at 10Gbps. But it is just fine doing a few Gbps.

A system with 50-60 TB on 16 GB RAM will likely be acceptable on 1 GbE; it won't make full use of 10 GbE.
Since your NAS is expected to do 10 GbE, you have a toe on the "performance" track, which begins with "increase RAM above the minimum (and some more if you can)".

 

[mariosk9gr]

First of all, thank you all so much for your help! I really appreciate it!!

It will work, but if you're buying parts to build, going for actual server-grade hardware is better.
(From earlier posts by Greek users, I understand that the local market is limited and there are few opportunities for second hand tough.)

Yes that's true. I'm thinking about Asus WS C246 Pro which has 8 sata ports, 2 nvme slots and 4 DDR slots also. I can find with ease a 9th gen cpu also with ecc support (I have seen which Coffelake cpus from intel have ecc support). The problem is that ecc Ram is too expensive and I'm afraid to buy 2nd hand ram, especially ram!

PSU is adequate for just 5 drives, but not for a full Node 804 case. Adding a second vdev would double performance, so this is an option to keep in mind.

Proper Power Supply Sizing Guidance

I've seen about 1,000 threads like this one where people decide that they can power a dozen hard drives off a 360 watt supply. DO NOT DO THIS. I've seen another 1,000 threads where people decide to buy the cheapest power supply that they can...

www.truenas.com

So a Be Quiet! 700-800w Gold shall be enough for 6 mechanical drives, 2 Nvmes and a dual 10Gbe network card?

CPU is fine. 16 GB RAM is fine for 1 GbE link, but is likely to fall short for a 10 GbE link.
The HDD array will not saturate a 10 GbE link. Can you put hard numbers on the red/write throughputs you need?

How large is a typical working set of RAW files? What total capacity do you need?

I was thinking with 5x HDDs but now with the new information, I'm going for 6x Ironwolf 8tb each.
For example in 10 days I will shoot a music video clip and I will finish with 1-1.5tb of 4k/6k raw video files (Braw and Canon CRL). Except from that, I will put into my timeline 5-9 streams for multi cam editing.I don't expect to work w/o making proxies! That's a lof of data even for Nvme!

Also, is the NAS intended just to serve the rushes you currently work on, or is it intended for long term archival of the finished work?
Raidz1 (ZFS equivalent to RAID5) is only moderately secure against hardware failures; it is fine for short-term work on RAW files which can be loaded anew from the original cards or from a backup. For long-term archival, raidz2 (or raidz3) would be recommended (plus a regular external backup of the NAS…).

The NAS has a dual purpose. To work from there and as backup also which means every project I'm working on is going to stay there also with reliability in mind.

Ninja'ed by our ever-helpful Resident Grinch…
CPU is fine. SMB (if that's how you'd share to Mac clients) is a single-threaded affair; high clocks with few cores are enough. But ZFS loves RAM and makes all it can of it to cache data and metadata. There's a rule of thumb of 1 GB RAM by TB of data—very roughly counted, and relaxing for large arrays. Note that (my emphasis):

A system with 50-60 TB on 16 GB RAM will likely be acceptable on 1 GbE; it won't make full use of 10 GbE.
Since your NAS is expected to do 10 GbE, you have a toe on the "performance" track, which begins with "increase RAM above the minimum (and some more if you can)".

So if I need 1Gb per 1Tb then that calculates that with 64Gb of Ram (6x8=48) I will be fine, right?

P.S. I forgot to mention that I will use RaidZ1. If I use Raidz2 I find it difficult to climb on those high reads/writes with only 6 drives...

 

[Etorix]

Yes that's true. I'm thinking about Asus WS C246 Pro which has 8 sata ports, 2 nvme slots and 4 DDR slots also. I can find with ease a 9th gen cpu also with ecc support (I have seen which Coffelake cpus from intel have ecc support).

WS C246 is ATX sized; perhaps you meant WS C246M? Anyway if you can find a C246 motherboard and a matching Coffee Lake Core i3 (or Xeon E-2100/2200) at acceptable price, go for it.

The problem is that ecc Ram is too expensive and I'm afraid to buy 2nd hand ram, especially ram!

RAM hardly wears out, and second-hand/refurbished server parts is a good way to save money on a build.
The only parts I would avoid buying second-hand are the data drives and PSU.

So a Be Quiet! 700-800w Gold shall be enough for 6 mechanical drives, 2 Nvmes and a dual 10Gbe network card?

Should be good for a full Node 804 (10*3.5" HDD) (see "Xeon E3"-class system).

I was thinking with 5x HDDs but now with the new information, I'm going for 6x Ironwolf 8tb each.

Side note: Ironwolf are fine drives, but shop around and also consider enterprise-class HDDs (Seagate Exos, Toshiba MG, WD Gold) in addition to NAS drives (Ironwolf, N300, Red Plus/Pro) and pick the best price.

The NAS has a dual purpose. To work from there and as backup also which means every project I'm working on is going to stay there also with reliability in mind.

Here comes a design issue: Raidz1 with large HDDs is not-so-safe; for resilience and reliability, raidz2 would be better (plus backup…).
Long term storage would do fine with a 6-10 wide raidz2 vdev.
For live work, many striped vdevs ("RAID 0") provides maximal performance. Possibly even a stripe of single drive vdev; no redundancy in case of failure, so it would be a pure scratch volume but it may be acceptable with a backup to the raidz2 archival pool. But two different, dedicated, pools would mean a lot of drives.

So if I need 1Gb per 1Tb then that calculates that with 64Gb of Ram (6x8=48) I will be fine, right?

Absolutely. It's a very loose guidance anyway, but the more RAM the better (until you go so high that the whole "working set" fits in ARC, but that's not going to happen).
64 GB and above opens the possibility to add a NVMe drive as L2ARC. Managing a L2ARC requires RAM, so it's not advised with less than 64 GB and there's another rule of thumb of having a 5:1 ratio of L2ARC:RAM (possibly up to but no more than 10:1, and checking the consequences on ARC). So with 64 GB, you may consider a 256-512 GB NVMe drive as L2ARC (may be added later to spread costs); with maxxed-up 128 GB RAM, that could be 512 GB-1 TB L2ARC, fitting a good part of your 1-1.5 TB working set and that should be really fast.

 

[Davvo]

The problem is that ecc Ram is too expensive and I'm afraid to buy 2nd hand ram, especially ram!

Adding at what Etorix already wrote, generally good RAM manifacturers give you lifetime warranties.

 

[MrGuvernment]

Adding at what Etorix already wrote, generally good RAM manifacturers give you lifetime warranties.

And you can also run memtest on it for a day or 2 as well to try and test it through for any issues (which everyone should be doing from the recommended reading here at TrueNAS)

 

[mariosk9gr]

WS C246 is ATX sized; perhaps you meant WS C246M? Anyway if you can find a C246 motherboard and a matching Coffee Lake Core i3 (or Xeon E-2100/2200) at acceptable price, go for it.

No I meant WS C246 with 8x sata ports and another case, I'm thinking about Fractal Design Define 7 which I think is good for nas build

Here comes a design issue: Raidz1 with large HDDs is not-so-safe; for resilience and reliability, raidz2 would be better (plus backup…).
Long term storage would do fine with a 6-10 wide raidz2 vdev.
For live work, many striped vdevs ("RAID 0") provides maximal performance. Possibly even a stripe of single drive vdev; no redundancy in case of failure, so it would be a pure scratch volume but it may be acceptable with a backup to the raidz2 archival pool. But two different, dedicated, pools would mean a lot of drives.

With zraid2 that would mean that performance goes down by 1 disk and reliability goes up by 1 disk also. I don't know how much faster can Ram and Nvme can provide while caching? What should I expect with zraid2, 6 mechanical drives and 1 Nvme drive, for example a wd black 1tb sn850x?

Absolutely. It's a very loose guidance anyway, but the more RAM the better (until you go so high that the whole "working set" fits in ARC, but that's not going to happen).
64 GB and above opens the possibility to add a NVMe drive as L2ARC. Managing a L2ARC requires RAM, so it's not advised with less than 64 GB and there's another rule of thumb of having a 5:1 ratio of L2ARC:RAM (possibly up to but no more than 10:1, and checking the consequences on ARC). So with 64 GB, you may consider a 256-512 GB NVMe drive as L2ARC (may be added later to spread costs); with maxxed-up 128 GB RAM, that could be 512 GB-1 TB L2ARC, fitting a good part of your 1-1.5 TB working set and that should be really fast.

Yes 2 sticks of 32Gb of Ram at the beginning with the potential to add another 2 in the future for 128gb total.

 

[mariosk9gr]

Can I ask also if a 11th gen motherboard P12R-E-10G-2T will work with zraid? It has also a dual 10Gbe on board but I don't think it will be supported under truenas...

​

 

[mariosk9gr]

And you can also run memtest on it for a day or 2 as well to try and test it through for any issues (which everyone should be doing from the recommended reading here at TrueNAS)

I was working back in 2008-2013 in a big computer sales company which I was on IT department and also sales. The most hardware part with issues were HDDs and Ram modules and the latter because even if you touch with your hands the pins you may provide bigger volt than 1.2-1.35 and destroy the modules. Also Memtest sometimes needed 2-3 days or more to just show 1-2 errors! I'm scared of when thinking about used Rams, especially where in my case I will buy the modules from EU and if something happens then the procedure will take time...