Lots of noob’s questions inside

[starche.old]

Before I’ll start with the questions, let me tell you what I have, what I want, what I already understood reading the internet for a few days now and finally – the questions I need your answers on.



I have:

• Hardware: Pretty old, but solid LGA1366 mobo – P6T6 WS Revolution (https://www.asus.com/Commercial-Servers-Workstations/P6T6_WS_Revolution) on Intel® X58 / ICH10R paired with i7-920 and 6GB (3*2GB) DDR3 RAM.
• Drives: SSD – Samsung 850 PRO (128GB) for system plus a single 8TB IronWolf and a bunch of 2TB and 3TB drives as storage.
• All that runs on zfs (RAID-Z) on Debian 10.x

What I want:

• I want to get a usable and reliable storage solution based on TrueNAS 12 (a long-time dream to play with FreeBSD after so many years on various Linux distros).
• Get rid of 2TB and 3TB drives and replace them with bigger, but fewer drives. Simple setup – 4 * 8TB drives in RAID-Z2.
• The storage will be filled (and filled now) with tons of photos, videos and such, so I do not want to lose all that.

After investing the 3 days already (and keep counting) I think I’m getting a better idea of what I need to do. I’ll put this short list – just to ensure I’ve got everything right and the community members are in agreement with that.

• I was thinking to buy two 12-16TB drives to complement my single 8TB IronWolf. But after going through how zfs is working as well as the zfs storage calculator (for example – https://wintelguy.com/zfs-calc.pl) I decided to buy three 8TB drives. That will give RAID-Z2 and almost 16TB of storage (which should be enough for me – at least for now).

I’ve got quite a list of the questions:

• Does my mobo and i7-920 CPU are good enough for the purpose?
• How critical is the RAM requirement? Yes, I’ve read that 8GB of RAM is absolutely a minimum. But will my 6GB do the trick for 4 drives only? If no, then the chances to get another matching set of three DDR-3 sticks are close to zero, thus I will need to build a new rig.
• I’m thinking to move the data from my computer and try to run TrueNAS on 6GB anyway – to see by myself how bad it might be. But maybe someone can tell me right off the bat – “it’s OK” or “don’t even waste your time on that”.
• What vendor is preferable these days? I’m a long-term user of Seagate drives, but TrueNAS is talking about WD. Does WD is any better than Seagate, or both vendors are on par?
• How much of a difference between NAS drives (WD Red/Red pro), enterprise (gold) and data center (UltraStar) drives in terms of reliability, speed, etc.? Does it make much sense to get Exos or UltraStar instead of IronWolf or WD Red?
• How bad is an idea to buy cheap Barracuda drives instead of IronWolf or WD Red if I'm using RAID-Z2?
• I've read quite a few articles about SATA vs SAS as an interface - https://store.hp.com/us/en/tech-takes/sas-vs-sata or https://www.diffen.com/difference/SATA_vs_Serial_Attached_SCSI, but I’m still confused.
• I do realize that the theoretical speed of SAS is double against SATA (12 vs 6Gb/s). But with sequential read from modern HDDs around 250-270MB/s (2.0-2.5 Gb/s), even SATA’s 6Gb/s won’t be a bottleneck, thus it makes no sense in SAS for at least 7200rpm HDDs (15k rpm HDDs in those days were the beasts). Am I missing something?
• Do I need a controller (like 9211-8i, 9300-8i and similar)?. What’s the benefit of using them? Can built-in southbridge chip handle 4 drives or not?
• If I need a controller, which one do I need? I’ve read here the topic (https://www.truenas.com/community/threads/sas-controller.89195/) that “Any LSI HBA, notably not a "RAID card" should be fine”. Is that correct?
• Why are a lot of advice to buy a Dell controller and reflash it with LSI firmware (like this one – https://fohdeesha.com/docs/perc/). Is it just cheaper this way or something else?

Sorry for so many questions and thanks in advance for the answers.

 

[mjt5282]

. check your calcs, the wintelguy website says 3 wide 8TB raidz2 is 7TB not 16. I would strongly recommend 6 wide minimum or better, 8 wide raidz2.
. minimum RAM for truenas is I believe 8 GB. For a larger ARC (in memory cache), 16 GB is recommended by me.
. Both Seagate and WD make excellent NAS drives, just don't buy SMR drives. WD calls CMR drives "WD Red Plus" now, to differentiate.
. A SAS HBA card made by Broadcom (nee LSI) will work with SAS and/or SATA drives. A SAS2 HBA card is probably fast enough for spinning rust.
I have around 65 TB 8x8TBx2 vdev pool , and when I upgraded from SAS2 to SAS3 HBA, speed increased at most 10% for a total pool scrub.
. I'd strongly recommend buying a SAS2 Broadcom/LSI card and flashing it to IT mode. This is what ZFS expects/deserves.
. flashing a Dell HBA to IT mode is a good way to start with TrueNAS, there are plenty on Ebay. ArtofServer guy on ebay sells pre-flashed HBA cards, too.

 

[ThreeDee]

I want to get a usable and reliable storage solution based on TrueNAS 12 (a long-time dream to play with FreeBSD after so many years on various Linux distros).

Then you are going to need 16GB at least (in my opinion) of ECC RAM .. but your i7-920 does not support ECC RAM .. so a conundrum right off the bat it would seem.

 

[fidgety]

I'd suggest you check the hardware compatibility list for that MB - to be sure about chipset and CPU support in FreeBSD. Eventually, I think you will want to upgrade, it looks like that MB was reviewed in 2009.... capacitors don't last forever. I try to replace my hardware every ten years or so, on general principles, as I don't want something to fail.

Your Asus web link says that 24GB is supported. I've not tried TrueNAS with 6GB but upgrading the memory would be the safe option. I've just checked the Crucial website for a quick view on this - it looks like it's still possible to buy DDR3-1600 DIMMs - £100 for 16GB.That would be money well spent if you keep your existing MB and you want to run a stable NAS. 16GB is probably not an ideal amount of memory in the long run, or if your disk space keeps expanding (it will ). You could try TrueNAS on this basis, I think, but it would be dead spend if you decide to replace the MB anyway.

Re: "which vendor" - it's worth reviewing the Backblaze hard disk reports. They buy drives in bulk and keep stats on the failure rates. My experience has been that the helium-filled drives run a lot cooler, and therefore the server is a lot quieter, if that makes a difference to you.

Re: SATA vs SAS - I think you are right, you probably wouldn't see much benefit. If you had an external enclosure with a SAS expander and 20 or 30 spinners in it then SAS makes more sense at that point.

The southbridge chip may be supported by TrueNAS - if so then you could probably get by without buying a disk controller in the short term. I'm never sure whether the MB manufacturers build these for performance, or only to the "it works now" standard. I haven't tested the performance of the MB ports as I always end up needing more ports and the controllers are cheap (see next point).

"If I need a controller, which one do I need?" - looks like your MB only supports PCIe v2.0. So a v3 controller is a waste of money (for now) as it will only run in v2 mode anyway. So that Dell H310 at $11.99 on ebay looks like a very economical way forward. You can buy disk controllers new - my usual supplier wants £200 or so for a new one but unless you have to have new hardware with a warranty (for operational support reasons) the used cards typically work just fine and the price is right.

Hope this helps,

 

[sretalla]

Does my mobo and i7-920 CPU are good enough for the purpose?

Maybe. Probably not a complete showstopper.

How critical is the RAM requirement? Yes, I’ve read that 8GB of RAM is absolutely a minimum. But will my 6GB do the trick for 4 drives only?

Very critical. 8GB is the minimum recommended due to the experience of the forum (and iX) that ZFS is not able to work reliably with less than that.

don’t even waste your time on that

Does WD is any better than Seagate, or both vendors are on par?

If you get the right version (WD RED PLUS) they spin slower and consume less power per disk than the 7200RPM options from seagate. I have found WD to be reliable.

How bad is an idea to buy cheap Barracuda drives instead of IronWolf or WD Red if I'm using RAID-Z2?

WD RED PLUS are good.

• I've read quite a few articles about SATA vs SAS as an interface - https://store.hp.com/us/en/tech-takes/sas-vs-sata or https://www.diffen.com/difference/SATA_vs_Serial_Attached_SCSI, but I’m still confused.
• I do realize that the theoretical speed of SAS is double against SATA (12 vs 6Gb/s). But with sequential read from modern HDDs around 250-270MB/s (2.0-2.5 Gb/s), even SATA’s 6Gb/s won’t be a bottleneck, thus it makes no sense in SAS for at least 7200rpm HDDs (15k rpm HDDs in those days were the beasts). Am I missing something?

In a 4-wide RAIDZ2, you're not going to blind anyone with speed, no matter how fast the disks. SATA is fine (connected via a SAS HBA also fine if you want to)

Do I need a controller (like 9211-8i, 9300-8i and similar)?. What’s the benefit of using them? Can built-in southbridge chip handle 4 drives or not?

If you need more SATA ports or if your onboard SATA is a Marvell controller, it can be a good option to get more (or better behaved ones)

If I need a controller, which one do I need? I’ve read here the topic (https://www.truenas.com/community/threads/sas-controller.89195/) that “Any LSI HBA, notably not a "RAID card" should be fine”. Is that correct?

You've got it right.

Why are a lot of advice to buy a Dell controller and reflash it with LSI firmware (like this one – https://fohdeesha.com/docs/perc/). Is it just cheaper this way or something else?

There are more available on the second-hand market and the LSI chip is on there, so no loss compared to a "genuine LSI card". You may find them cheaper... depends on the day and the seller.

 

[mjt5282]

if your storage requirements are not large, maybe raidz2 is not the right choice for you. Perhaps buy a 4th 8TiB drive, create a mirrored pool from the first two disks, then add a second vdev of the other two drives. 50% storage penalty but quick resilvers and easy to add more storage later. YMMV.

 

[starche.old]

. check your calcs, the wintelguy website says 3 wide 8TB raidz2 is 7TB not 16. I would strongly recommend 6 wide minimum or better, 8 wide raidz2.
. minimum RAM for truenas is I believe 8 GB. For a larger ARC (in memory cache), 16 GB is recommended by me.
. Both Seagate and WD make excellent NAS drives, just don't buy SMR drives. WD calls CMR drives "WD Red Plus" now, to differentiate.
. A SAS HBA card made by Broadcom (nee LSI) will work with SAS and/or SATA drives. A SAS2 HBA card is probably fast enough for spinning rust.
I have around 65 TB 8x8TBx2 vdev pool , and when I upgraded from SAS2 to SAS3 HBA, speed increased at most 10% for a total pool scrub.
. I'd strongly recommend buying a SAS2 Broadcom/LSI card and flashing it to IT mode. This is what ZFS expects/deserves.
. flashing a Dell HBA to IT mode is a good way to start with TrueNAS, there are plenty on Ebay. ArtofServer guy on ebay sells pre-flashed HBA cards, too.

My calcs are correct. I want to buy 3 HDDs, but I already have one 8TB Irowolf, so in total, I have 4 HDDs so around 15TB of available space.

Got QVL for my mobo from the manual. Checked on Amazon - nothing, but there are few sets on eBay. Now, the question is will they work for me or not (not a 100% match in part#). I need to try them hoping one of them will work.

SAS2 HBA (9211-8i for example) is good enough and there's no practical need to go to 93xx ones? Great news. The question - why I need the card? Can mobo handle zpool? Another question - can I create zpool now and add SAS2 HBA card? Will that break the zpool?

 

[starche.old]

Then you are going to need 16GB at least (in my opinion) of ECC RAM .. but your i7-920 does not support ECC RAM .. so a conundrum right off the bat it would seem.

The more I'm thinking and researching, the more I'm tending to think about the new build. That will eliminate RAM search issues and in general feels like a better idea in a long run.
I have 3 options if my current build won't be sufficient enough:
1. Asus Pro WS X570-ACE (workstation mobo, supports ECC RAM - https://www.asus.com/Commercial-Servers-Workstations/Pro-WS-X570-ACE/overview/) plus AMD Ryzen 5 3600X 6-Core, 12-Thread
2. Non-WS, just gaming Asus mobo (for example -TUF GAMING X570-PRO (WI-FI)) plus the same CPU
3. SuperMicro mobo and Xeon. But that would be crazy expensive (I guess?) and a bit of overkill for a home server. Or I'm wrong and that's the way to go?

for RAM - if I need ECC RAM, than I'm looking at (2 sticks of each):
1. 16GB Micron DDR4-2666 ECC UDIMM Memory - MTA18ASF2G72AZ-2G6E2
2. Kingston 16GB 2400MHz DDR4 ECC CL17 DIMM 2Rx8 Micron E
3. KINGSTON 16GB 2666MHz DDR4 ECC CL19 SODIMM 2Rx8 Hynix D (KSM26SED8/16HD)


What do you think about all that if I'll have to go with a new build?

 

[starche.old]

I'd suggest you check the hardware compatibility list for that MB - to be sure about chipset and CPU support in FreeBSD. Eventually, I think you will want to upgrade, it looks like that MB was reviewed in 2009.... capacitors don't last forever. I try to replace my hardware every ten years or so, on general principles, as I don't want something to fail.

Your Asus web link says that 24GB is supported. I've not tried TrueNAS with 6GB but upgrading the memory would be the safe option. I've just checked the Crucial website for a quick view on this - it looks like it's still possible to buy DDR3-1600 DIMMs - £100 for 16GB.That would be money well spent if you keep your existing MB and you want to run a stable NAS. 16GB is probably not an ideal amount of memory in the long run, or if your disk space keeps expanding (it will ). You could try TrueNAS on this basis, I think, but it would be dead spend if you decide to replace the MB anyway.

Re: "which vendor" - it's worth reviewing the Backblaze hard disk reports. They buy drives in bulk and keep stats on the failure rates. My experience has been that the helium-filled drives run a lot cooler, and therefore the server is a lot quieter, if that makes a difference to you.

Re: SATA vs SAS - I think you are right, you probably wouldn't see much benefit. If you had an external enclosure with a SAS expander and 20 or 30 spinners in it then SAS makes more sense at that point.

The southbridge chip may be supported by TrueNAS - if so then you could probably get by without buying a disk controller in the short term. I'm never sure whether the MB manufacturers build these for performance, or only to the "it works now" standard. I haven't tested the performance of the MB ports as I always end up needing more ports and the controllers are cheap (see next point).

"If I need a controller, which one do I need?" - looks like your MB only supports PCIe v2.0. So a v3 controller is a waste of money (for now) as it will only run in v2 mode anyway. So that Dell H310 at $11.99 on ebay looks like a very economical way forward. You can buy disk controllers new - my usual supplier wants £200 or so for a new one but unless you have to have new hardware with a warranty (for operational support reasons) the used cards typically work just fine and the price is right.

Hope this helps,

I installed FreeBSD (I think that was the 11th or even 10th version of FreeBSD) on that mobo, had DE and such. It worked no problem. The only thing which was not supported at that time was a sound card (Asus Xonar Essense STX), which I was not ready to give up.

I'm thinking to upgrade. I built that rig in 2008 or early 2009 and it's time to get something faster than 12 years old desktop. I put my thoughts in one post above regarding the hardware. What do you think about that? The only thing here - I'd like to spread the costs ($2k plus - as it sits right now) in time. In other words, it would be great if I'll buy HDDs right now (like a grand), create zpool and spend another grand in 2-3 months later - if my current rig can handle that.

Backblaze is a great source of the information regarding reliability. The only thing - they are carrying data center HHDs, not NAS ones. My hope is RAIDZ2 is reliable enough to handle occasional HDD failures.

Thank you for clarifying the story with SAS. No way I'll need to run 20-30 drives. If that will be the case then it would be a completely different discussion . So, I guess, I'm good to go with SATA?

Oh, so all the discussion about the controller is the compatibility with TrueNAS? Now I've got it. But if my mobo can work in TrueNAS (it works I FreeBSD for sure), then temporarily I can run without it? If I'll keep my current rig, I would definitely need a card coz SATA ports on mobo are one 3GB, not 6GB ones. I can easily afford $12-$50 and flash it for IT mode or something like that (I saw a manual on how to do so).

But I'll build a new rig with PCIe 4.0, will that make sense to get something better than a Dell card? Or let's put it this way: with a new PCIe 4.0 rig, do I need a card? And which one you'll suggest?

Your answers helped a lot. Thank you.

 

[starche.old]

Maybe. Probably not a complete showstopper.


Very critical. 8GB is the minimum recommended due to the experience of the forum (and iX) that ZFS is not able to work reliably with less than that.




If you get the right version (WD RED PLUS) they spin slower and consume less power per disk than the 7200RPM options from seagate. I have found WD to be reliable.


WD RED PLUS are good.


In a 4-wide RAIDZ2, you're not going to blind anyone with speed, no matter how fast the disks. SATA is fine (connected via a SAS HBA also fine if you want to)


If you need more SATA ports or if your onboard SATA is a Marvell controller, it can be a good option to get more (or better behaved ones)


You've got it right.


There are more available on the second-hand market and the LSI chip is on there, so no loss compared to a "genuine LSI card". You may find them cheaper... depends on the day and the seller.

Got it. There are 2 options on the table right now: to add RAM to the current rig or to build a new one. I'm leaning towards a new build but after some time.

I would like to get 7200 rpm drives. I've got a good case with good soundproofing and quite a few fans. Noise and hot air (I hope) should not be a problem, but let's see how that would be in reality.

4 drives in RAIDZ2 won't be fast, I agree. I'm planning to add 1 or 2 drives to the mix a bit later. Will that help with the speed and overall performance?

I'm definitely looking into that. $50 won't make me to mortgage my house .

BTW, what do you think about the DELL PERC H710 card? Should I get that one or H310 is good enough for 4-6 drives?

 

[starche.old]

After some time spending reading and such, I'm thinking to get IronWolf NAS 8TB drives (ST8000VN004 to be specific). There is another model of the same drive - ST8000VN0022. From what I understood, 0022 has 6 disks while the 004 has 7 disks. That (in my mind) mean s that 004 will be noisier and hotter, but less data density (better for reliability). Also, ST8000VN0022 is older (Jan 2018 against Jul 2019).

Should I get 004 or 0022 ones?

Thank you everyone for the very helpful discussion. I continue to bug you with the questions if you don't mind.

 

[starche.old]

if your storage requirements are not large, maybe raidz2 is not the right choice for you. Perhaps buy a 4th 8TiB drive, create a mirrored pool from the first two disks, then add a second vdev of the other two drives. 50% storage penalty but quick resilvers and easy to add more storage later. YMMV.

Do you mean like RAID10 setup? From what I understand, RAID10 and RAIDZ2 have the same 50% penalty on 4 HDDs, but RAIDZ2 gets ahead with adding the 5th drive. Am I wrong?

Or you mean to have 2 separate mirrored pools? How that will work with adding the 5th and the 6th drive?

 

[ThreeDee]

If you are looking to build an AMD based server .. you might want to look into ASRock Rack X570D4U or the X470D4U variants.

I recently acquired the X470D4U to unnecessarily replace the setup in my sig (waiting on 2 x 16GB ECC DDR4 UDIMM's) I'm just wanting a setup that is a bit more "green" to save a couple bucks on my power bill over time seeing how my TrueNAS box is on 24/7.

 

[ThreeDee]

AMD Ryzen 5 3600X 6-Core, 12-Thread

What do you think about all that if I'll have to go with a new build?

I'd recommend the Ryzen 5 3600 over the 3600x (65wtt tdp vs 105wtt with negligible performance gains) ..or go with the Ryzen 7 3700x (65wtt 8core/16thread)

.. or go with a 5600x.

The ASRock Rack server motherboards I mentioned both support Ryzen 5000 series. But .. most here would recommend Intel setups for a TrueNAS setup... with a server motherboard and not a desktop gaming/workstation motherboard. If this is going to only be a server and not repurposed into a gaming box down the road ... then I would recommend you look at server motherboards/setups only.

 

[starche.old]

If you are looking to build an AMD based server .. you might want to look into ASRock Rack X570D4U or the X470D4U variants.

I recently acquired the X470D4U to unnecessarily replace the setup in my sig (waiting on 2 x 16GB ECC DDR4 UDIMM's) I'm just wanting a setup that is a bit more "green" to save a couple bucks on my power bill over time seeing how my TrueNAS box is on 24/7.

I heard that X470 does not support 5000 series CPUs. If that's the fact, then X570 looks a better option in the long run.

I agree, lower TDP is better. And you know what? 3700X looks like a great choice for me - 8C/16T and 65W TDP.

For mobo - I was always thinking that Asrock is a cheaper version of Asus. I guess, that changed by now? The only objection from my side will be micro-ATX form factor. I would prefer ATX or even eATX, my case easily allows that.

That what I was always thinking, that Intel (Xeon especially) is better for server applications> But with current development with AMD (I never bought anything on AMD until recently a Lenovo T14 Gen1 AMD laptop with 4750PRO - I was really impressed, especially in comparison with twice as expensive Lenovo X1 Extreme Gen3 with i7-10850H). Plus the price tag for Intel setup will be higher.

P.S. It will be a server only - I'm done with gaming for quite a few years now. Too old for that

What about the choice of the drives - are they good enough for "casual" (not heavy usage) home server in RAIDZ2?

 

[starche.old]

What are the advantages of Intel setup? I know in those days (10-15 years ago, when I was involved into that) that was out of the discussion, but what about nowadays?

 

[ThreeDee]

B350/X370 doesn't support 5000 series Ryzen.

size of motherboard shouldn't matter as long as it has what you need/want on it.

ASRock was originally spun off from Asus in 2002 in order to compete with companies like Foxconn for the commodity OEM market. ... ASRock currently produces consumer, server, workstation and HTPC motherboards. ASRock has distributors in over 90 countries around the world and branches in Europe, US and China.

 

[mjt5282]

...
SAS2 HBA (9211-8i for example) is good enough and there's no practical need to go to 93xx ones? Great news. The question - why I need the card? Can mobo handle zpool? Another question - can I create zpool now and add SAS2 HBA card? Will that break the zpool?

if you are running SATA hard drives, the speed improvement of SAS3 is incrementally very small. Also, for home users, SAS3 HBA controllers are still rather unaffordable. SAS2 is a good budget choice, and worked well for me for many years. You absolutely can use your motherboard SATA connections and create the pool, and migrate to a HBA card later, if you want/need to.
good luck with the learning curve and remember to have fun while learning!

 

[starche.old]

B350/X370 doesn't support 5000 series Ryzen.

size of motherboard shouldn't matter as long as it has what you need/want on it.

I wanna get X570 mobo only. It's more future proof than anything else at this time.

 

[starche.old]

if you are running SATA hard drives, the speed improvement of SAS3 is incrementally very small. Also, for home users, SAS3 HBA controllers are still rather unaffordable. SAS2 is a good budget choice, and worked well for me for many years. You absolutely can use your motherboard SATA connections and create the pool, and migrate to a HBA card later, if you want/need to.
good luck with the learning curve and remember to have fun while learning!

Thank you.

The Dell card looks great value for the $$$ spent. For around $30-$50, I can get an improvement (speed + reliability) over the mobo-connected SATA drives. Am I correct?

4 questions here:
1. Does IronWolf NAS 8TB drive (ST8000VN004 one) is a good choice for NAS? Can't wait to order them.

2. AMD vs Intel in terms of NAS. I do realize that this is a big question and potentially it can start a "holy war" (that's NOT my intention). I went through SuperMicro mobos and Xeon CPUs. In general, that's an easy $2k price tag while AMD combo is well under $1k. I guess, my question is - do I really need to pay that extra? In other words, does AMD server more than twice worse (less reliable or smth else?) than its Intel counterpart?

3. For an AMD-based server - does X570 is a good choice? Is there anything better than that?

4. From what I already saw (keep looking tho), I have 2 mobos which are appealing to me:
- ASRock X570D4U-2L2T (https://www.asrockrack.com/general/productdetail.asp?Model=X570D4U-2L2T#Specifications)
- Asus Pro WS X570-ACE (https://www.asus.com/Commercial-Servers-Workstations/Pro-WS-X570-ACE/overview/)
Any other mobos which would be good for me?
I guess gaming/desktop mobos are not appropriate for my scenario.