Newbie Build

[FlyingHacker]

I am scoping out hardware for a TrueNAS Core build designed to support heavy video file use. (Projects on the order of a few hundred GB)

I was looking for opinions on a setup like such. I am not married to anything here, and am opening to suggestions.



AMD Ryzen 5 5600 Vermeer 3.5GHz 6-Core AM4 Boxed Processor - Wraith Stealth Cooler Included


Asrock x570 Phantom Gaming 4

https://www.asrock.com/MB/AMD/X570 Phantom Gaming 4/index.asp#Specification

This supports ECC RAM, which I would have 64 GB from the QVL


A pair of Samsung EVO M.2 drives - probably 256-512GB each


8 Toshiba N300 8TB NAS drives using the 6Gb SATA ports on the mobo. Would want RAID 5 or 6.

~800W power supply (80plus gold or better)

Case will be some 4U rackmount.

Will add a 10GigE ethernet card (Chelsio)

Should I be looking at something else for a ZIL?

Is this type of consumer grade mobo/cpu a terrible choice? Any other suggestions?

Thanks in advance.

 

[Jessep]

Do you have the parts already or building a shopping list?

• If not, look at used enterprise gear.
• Example rack mount pretty much good to go (if US based)
  • https://www.ebay.com/itm/334431851829?hash=item4dddb11535:g:hyQAAOSweqVh1gZ9

Are you planning/wanting to work on the files directly on the NAS?

• If so plan for a working pool SSD/NVME drive set for working files and a separate pool for bulk storage with HDD.
• Suggest using iSCSI for access.

What is the end game for drives? 8? 16? 24? etc.

Please read the terminology primer, as you will draw ire for incorrect terms, i.e. its RaidZ1 not Raid5.

 

[FlyingHacker]

Thanks for the reply.

Building a shopping list, with immediate need.

The eBay link looks decent. I wonder if there is a way to get a 10GigE card in there? I definitely need that. I assume you would configure the HBA as JBOD and use TrueNAS for the actual RAID.

Files remain on the NAS, some files get automagically cached to local M.2, but they should be assumed to be on the server.

Interesting thought to keep those two separate in terms of working and bulk storage. So there is no magical caching on the NAS to handle this? It is best left as an exercise for the user?

Is iSCSI faster than mounting SMB volumes?

OK. RaidZ1. RAID capable of loosing at least one unit (possibly 2).

Thanks!

 

[FlyingHacker]

Also, how about something like this for 12 bays? At used prices perhaps I go with more bays and put SSDs in some. There is no M.2 in these, though.

https://www.ebay.com/itm/325192888705?hash=item4bb701b181:g:RyUAAOSwaBxii8BF

Is the H310 that much worse than the HBA330?

 

[FlyingHacker]

OK. I think I will go all the way for one of these and add 12 drives to it and a boot drive:

https://www.ebay.com/itm/3741409722...Auy56ObAHzrxJ4Ar0Rt8ZVHSmg==|tkp:BFBM8oOek7Vg

It's also got 256GB of memory already. So that should be good for the cache.

 

[Ericloewe]

Is the H310 that much worse than the HBA330?

Yeah, but it can be lobotomized to something that would work: https://fohdeesha.com/docs/H310.html

OK. I think I will go all the way for one of these and add 12 drives to it and a boot drive:

https://www.ebay.com/itm/374140972205?epid=1266085304&hash=item571c8a2cad:g:wJIAAOSwEnthRkkM&amdata=enc:AQAHAAAA8CBoPUut1y+/QDBB5BK15a2GQyjj5s7Bt2fndb5zopCcbjo0ZL3eCqjQ0Y/4g8AtRLuFd6r3X7UhcehXcFHILIhXsbIq4JBrUjFfJ4cyCuPIjNvCqd4EabIA5GNnKamUKNyMJ7BuoNJLOdNCbl5YDokmWfQY2ku1b7Cc8WQsA1yuCNNFt4rs7G8ltPzwnv9+WwTVk8kDDiPV5tjVRgoSYnD645L+tBYkBX1rDoIlvPabTi78hm8cW3RFS0X7W7yIvK9PdERWGUJrLCuVq8XfXyk/LkyxzfDuRAkqINe2Auy56ObAHzrxJ4Ar0Rt8ZVHSmg==|tkp:BFBM8oOek7Vg

It's also got 256GB of memory already. So that should be good for the cache.

Gen 13 is a decent upgrade over Gen 12, so if it works for you price-wise go for it. Up to four bays should support U.2 NVMe disks, if you wire them up to a PCIe x16 slot.

 

[Jessep]

Agree the 730xd is a better choice, will need a new HBA (Dell HBA330, mini micro).

This would the the correct choice

Dell HBA330 mini monolithic (=9300-8i) w/ P16 IT mode H330 ZFS FreeNAS unRAID | eBay

I have fully tested them using all 12 SAS HDD slots. Listed for sale are Dell PERC HBA330 mini monolithic cards (Dell P/N: P2R3R) for 13th generation Dell PowerEdge servers. They have been flashed with Dell IT mode firmware equivalent to Avago / LSI 9300-8i IT (Initiator Target) firmware version...

www.ebay.com

 

[FlyingHacker]

Thanks for the feedback, guys. Most of this data center HW is foreign to me. So I appreciate all the input I can get.

That 730xd comes with a "Perc H730 1gb Controller with Battery." Would I would still want the HBA 330 mini micro instead? Again, I assuming the preferred method is to disable the hardware RAID and use TrueNAS to handle that.

Thanks again.

 

[FlyingHacker]

Up to four bays should support U.2 NVMe disks, if you wire them up to a PCIe x16 slot.

Would this require additional hardware, or is this built in? Sorry, I have never seen inside one of these, and there is a lot to digest.

 

[Ericloewe]

Would I would still want the HBA 330 mini micro instead?

100% yes.

Would this require additional hardware, or is this built in? Sorry, I have never seen inside one of these, and there is a lot to digest.

The backplane supports it. Some servers may ship with a suitable adapter and cables, others might not. The official Dell kit is a bit expensive, but generic equivalents should work.

 

[Etorix]

Should I be looking at something else for a ZIL?

The proper terminology here is "SLOG", and this device is useless if there are no sync writes.
But with a SLOG and sync writes enabled, writes are still an order of magnitude slower than with async writes. So set sync=never and don't bother with a SLOG.

Is this type of consumer grade mobo/cpu a terrible choice? Any other suggestions?

It is a poorer choice compared with actual server motherboards: No BMC, not necessarily engineered for 24/7 stability.
If you stick with Ryzen, look at X470DU4, X570D4 and B550DU4 boards from AsRockRack; some variants even come with 10 GbE on board (but Base-T, not SFP+).

 

[FlyingHacker]

OK. Thank you all for your input. I will update when I get everything going, but ordered that Dell Poweredge R730xd 2x E5-2666 v3 2.9ghz 256gb and the Art of Server's H330 (linked above). I also ordered 10 SAS HGST drives (used, but will run full test on them, and I only plan on using likely 10 of them.)

We will then see if we need any of the NVMe drives or not. With 256GB of RAM that will likely provide for plenty of read cache. And based on some of the Lawrence Systems videos on Youtube it would appear with our load of primarily larger files from few users that a ZIL cache might not help much anyway.

Now I have to figure out how many drive to use in my pool(s). Do I do 10 in one pool, or two 5 drive pools? I am trying for RaidZ2. Some of the benchmarks I have seen seem to indicate the 5 drive pool may actually be faster. But first I have to get everything, test the drive for like 6 days, and get TrueNAS setup. So this is a ways off!

 

[FlyingHacker]

Oops. I meant I ordered 12 of those drives, but plan on using 10, with a couple spares.

 

[Jessep]

Not much point in "hot" spares for RaidZ2, would make more sense for RaidZ3 in that case. However if you are doing 2x RaidZ3 of 6 drives each, you may as well do 5x Mirror pairs with 2 hot spares for significantly more performance.

Your system also has 2x 2.5" drives in the back for OS/Boot.
2x of these would work well:

Intel S3500 120GB 2.5" SATA Solid State Drive for sale online | eBay

Find many great new & used options and get the best deals for Intel S3500 120GB 2.5" SATA Solid State Drive at the best online prices at eBay! Free shipping for many products!

www.ebay.com

There will be a large learning curve on Enterprise gear, plenty of folks here can help you out with that. You will want to do a few things first before even thinking of working with TN, this is a short list (not complete).

• Get access to iDRAC (remote access management) it comes with enterprise license so you will be able to remote KVM and virtual media.
• Update ALL firmware using SUU (system update utility) or the like (lifecycle online etc.).
• Run built in Dell diagnostics, both quick and through
• Plan your backups and redundancy
  • If your business is using this as primary storage, what happens if it goes down?
  • Think about extending warranty if possible, check here with service tag
    • https://www.dell.com/support/incidents-online/en-us/careforms/extendwarranty
  • If nothing else once you have PoC (proof of concept) put the budget in place for a second unit as backup ASAP
• At least skim through the service manual to familiarise yourself with the basic hardware
  • https://www.dell.com/support/home/en-us/product-support/product/poweredge-r730xd/drivers
  • https://www.dell.com/support/home/en-us/product-support/product/poweredge-r730xd/docs
    • https://dl.dell.com/topicspdf/poweredge-r730xd_owners-manual_en-us.pdf
• Test all drives thoroughly (may as well use windows if that is what you are familiar with, download trial server 2019 or 2022)
  • Badblocks
    • https://www.truenas.com/community/threads/unclefesters-freenas-beginners-guide.43889/
    • https://www.familybrown.org/dokuwiki/doku.php?id=fester:hvalid_hdd
  • HD Tune, this goofy guy has some decent starter videos, this one shows good vs. bad drives that is good to check on used hdd
    • https://youtu.be/JQdf6WF0kMo?t=811
• ASK QUESTIONS, assumptions will break things or destroy data.

Welcome to a fun adventure.

 

[FlyingHacker]

Not much point in "hot" spares for RaidZ2, would make more sense for RaidZ3 in that case. However if you are doing 2x RaidZ3 of 6 drives each, you may as well do 5x Mirror pairs with 2 hot spares for significantly more performance.

Ah, good point. This is likely the ideal setup.

Your system also has 2x 2.5" drives in the back for OS/Boot.
2x of these would work well:

Intel S3500 120GB 2.5" SATA Solid State Drive for sale online | eBay

Find many great new & used options and get the best deals for Intel S3500 120GB 2.5" SATA Solid State Drive at the best online prices at eBay! Free shipping for many products!

www.ebay.com

Yep. That was the plan. Those would be for boot. A ZFS mirrored pair would be ideal, assuming TrueNAS lets you do that on install, like the normal FreeBSD's installer does. ??

There will be a large learning curve on Enterprise gear, plenty of folks here can help you out with that. You will want to do a few things first before even thinking of working with TN, this is a short list (not complete).

• Get access to iDRAC (remote access management) it comes with enterprise license so you will be able to remote KVM and virtual media.
• Update ALL firmware using SUU (system update utility) or the like (lifecycle online etc.).
• Run built in Dell diagnostics, both quick and through

All on the list, and generally what I do with any hardware, new or used. Get all those updates out of the way before the machine becomes important. I have experience with some enterprise gear in the past, but everything is different.

• Plan your backups and redundancy
  • If your business is using this as primary storage, what happens if it goes down?
  • Think about extending warranty if possible, check here with service tag
    • https://www.dell.com/support/incidents-online/en-us/careforms/extendwarranty
  • If nothing else once you have PoC (proof of concept) put the budget in place for a second unit as backup ASAP

Likely the old NAS will server as backup with some sort of rsync or similar. Due to the nature of the work this type of system has served us well for backups for a long time ( it had provisions for ransomware encryption as well ). But this needs further evaluation. ZFS Snapshots would be ideal in the future.

• At least skim through the service manual to familiarise yourself with the basic hardware
  • https://www.dell.com/support/home/en-us/product-support/product/poweredge-r730xd/drivers
  • https://www.dell.com/support/home/en-us/product-support/product/poweredge-r730xd/docs
    • https://dl.dell.com/topicspdf/poweredge-r730xd_owners-manual_en-us.pdf
• Test all drives thoroughly (may as well use windows if that is what you are familiar with, download trial server 2019 or 2022)
  • Badblocks
    • https://www.truenas.com/community/threads/unclefesters-freenas-beginners-guide.43889/
    • https://www.familybrown.org/dokuwiki/doku.php?id=fester:hvalid_hdd
  • HD Tune, this goofy guy has some decent starter videos, this one shows good vs. bad drives that is good to check on used hdd
    • https://youtu.be/JQdf6WF0kMo?t=811
• ASK QUESTIONS, assumptions will break things or destroy data.

Welcome to a fun adventure.

I'll likely use the badblocks concurrent script shown in this video to test all 12 drives at once.
https://www.youtube.com/watch?v=9bh5ZK8z4ZA

I've been using BSD since about 1995, and linux almost as long. So none of that is an issue, and is generally preferred (by me) over Windows for anything server related.

Thanks for all the info. Always more to learn.

 

[FlyingHacker]

You said keep asking questions.. What is the best way to test used SSDs (for the boot drive)? I assume a multiple full write and read session (like badblocks is bad due to adding additional wear... Or is that still the way to go?

Thanks.

 

[Whattteva]

Is this type of consumer grade mobo/cpu a terrible choice? Any other suggestions?

My first build was consumer and, honestly, it ran fine for a couple years (I even had a year uptime once), but frustrated me everytime something happens (usually a bad USB stick boot drive back in the days when FreeNAS still uses those) and I have to scavenge a kb/monitor combo from another machine. The best part of going server-grade, IMO, is IPMI. The ability to run headless and use virtually any remote machine connected to the network to be a thin client terminal. The ability to shutdown, power cycle, and watch your machine boot from even hundreds of miles away from home through VPN is priceless in my humble personal opinion.

 

[Ericloewe]

You said keep asking questions.. What is the best way to test used SSDs (for the boot drive)? I assume a multiple full write and read session (like badblocks is bad due to adding additional wear... Or is that still the way to go?

Thanks.

Badblocks-esque testing is significantly less useful for SSDs because of the additional layers of abstraction that mean you can't effectively test every bit. What is the best option for testing an SSD? I'm not sure I have a good answer for you. Spares on hand is what I'd do.

 

[FlyingHacker]

So an extended SMART test and a mirror ought to be adequate?

 

[Ericloewe]

Yeah. A dry run period before the system becomes critical would also be advisable.