PCI HBA SAS vs Sata chipset and general use case questions

[XOC_JOE]

Hi Group!
Thanks for having me in the group and talking the time to talk to me. A little about me quick.
I started with FreeNas back when it was 8.00 something I think. I put that system together back in 08-09ish used it as a gaming computer for a short time and then switched it over to a Freenas box. It was running a AMD Phenom II X4 940, 16gigs of ram and 6 drives. This system worked great as a File storage system and a Plex box. The only issue I ever had was drives dying every couple of years and I would just wipe it and start over with new drives. It used a flash drive as a boot device that I loved because it opened up one more Sata port for a drive. I shut down this system a little over a year ago and its time to rebuild from the ground up. Its been so long since I put this system together I have to start over myself. I did very little for management with the Freenas box due to it just working so I know little about Truenas itself. I do a lot within the PC world of hardware mainly AMD stuff. I build a lot of gaming computers and for fun I XOC systems with LN2. I always have extra parts laying around because of this. I have spent a good amount of time rereading all the guilds and searching the forums trying to build the best system I can with some of the parts I already had laying around here and purchasing the best possible parts for my use case. I am hoping to get 10 plus years again out of this build if possible. My use case has grown a little for this build since the last build. I want to be able to store files again pictures and raw video plus run Plex again. I want to be able to transcode two streams of Plex from 4K to what ever the device needs with subs outside my local network at the same time. Thats why I used this CPU instead of a smaller one. I have a 10GBe network now so I can move large files to the Nas a little better then before with just a 1GBe network. My goal is to max out the 10GBe network from my shop PC to the Nas box. I have Cat 8 cabling and 10GBe switches with Intel X540-T2s for both ends.
My build so far is.
AMD 5950X
Asrack X570D4U mother board PCI 4.0
128 gigs 3200 ECC ram from Nemix (never heard of them any input would be great)
Intel X540-T2 10GBe Nic card
Samsung 850 EVO 120gig SSD sata Boot drive
Qty 5 Samsung EVO 2TB Sata SSDs (more on this later)
PCIe HBA SAS 3008 9003-8I???? (more on this later)

Would my use case benefit from adding a NVME drive for a SLOG or any other type of Cache? I know some benefits can be had from this when running a huge Plex library. This Plex will Be servicing about 6 connections of different types. I also want to send large files to the Nas as fast as possible I think the benefits come from use of spinner drives whereas I am using all SSDs and a larger ram pool do I need this? I have a bunch of 1 and 2 TB NVME drives sitting here doing nothing and a unused NVME 4.0 port open that wont affect any of my other lanes. Do I need/Use this?

I am considering a HBA SAS card but do I need this? I have a total of 5 drives and can add more if I order the HBA I would just order a couple more drives and fully populate the card. What do I gain from adding this to my system over the X570 chipset? Both have PCIe 4.0 So bandwidth is not a issue that I know of. The boot drive and Pool would be on the same chipset which could be a down side maybe? Do I gain speed, reliability, ease of use, better power consumption? I could run a total of 7 drives with the chipset and 8 drives with the HBA. These drives have gotten so cheap its hard not to go with more of them.

Any info anyone has will be greatly appreciated. If anyone has input on my server build to this point I would like to hear what you have to say. It will save me heart ache in the future. Thanks so much. I cant wait to get this running again and the TrueNas looks great from what I have seen so far. I also want to build a great Nas box so if you see anything wrong with what I am doing please speak up.

 

[jgreco]

I started with FreeNas back when it was 8.00 something I think.

Welcome back.

I have a 10GBe network now so I can move large files to the Nas a little better then before with just a 1GBe network. My goal is to max out the 10GBe network from my shop PC to the Nas box. I have Cat 8 cabling and 10GBe switches with Intel X540-T2s for both ends.
My build so far is.
AMD 5950X
Asrack X570D4U mother board PCI 4.0
128 gigs 3200 ECC ram from Nemix (never heard of them any input would be great)
Intel X540-T2 10GBe Nic card

Well I personally think 10GBase-T copper cabling's a hot crappy mess. See the 10 Gig Networking Primer for how to do it right. As for "Nemix", there are a lot of companies that take the rejects from the major manufacturers and make 'em into DIMM's anyways. Especially with ECC, this isn't particularly harmful. Run memtest for a month and you can be pretty sure it'll be good.

Would my use case benefit from adding a NVME drive for a SLOG or any other type of Cache?

SLOG is not any sort of cache. See

Some insights into SLOG/ZIL with ZFS on FreeNAS

What is the ZIL? POSIX provides a facility for the system or an application to make sure that data requested to be written is actually committed to stable storage: a synchronous write request. Upon completion of a sync write request, the underlying filesystem is supposed to guarantee that a...

www.truenas.com

You can turn off sync writes in many cases. A SLOG can only be slower than a pool with sync writes disabled.

I am considering a HBA SAS card but do I need this? I have a total of 5 drives and can add more if I order the HBA I would just order a couple more drives and fully populate the card. What do I gain from adding this to my system over the X570 chipset?

Probably none. An LSI HBA is basically a little MIPS CPU that handles SAS communications, but it represents another thing handling your data. This is fine for HDD's but if you put SATA SSD's on, you can actually see it is usually a bit slower than PCH SATA ports on your chipset. On the other hand, you cannot hook 100 HDD's up to your X570 chipset. You can to an LSI HBA. The HBA will typically consume about 12 watts.

 

[XOC_JOE]

Welcome back.

Welcome back.

Thanks!

Run memtest for a month and you can be pretty sure it'll be good.

I planned on heavily tuning my ram. The great thing with doing XOC all the time. I can tune ram to run like crazy if needed.

SLOG is not any sort of cache. See

I reread this again. To a beginner some of this seems to contradict itself. I know its not but thats how I am interpreting it for what ever reason. My biggest take away was that for a given amount of ram like mine 128 gigs I have to make sure I can write at least 16gigs or 1/8 my system memory every 5 seconds to the drive pool. If I cant then I am creating a bottle neck. (This would be a worse case) My network wont be able to move 16gigs in 5 seconds so the transaction group will start to clear from the ram and be written to the pool by the 5 second timer. My network should only be able to move about 7ish gigs of data to the transaction group before the timer limit hits again. My 5 current drives running through the chipset can handle that but to be safe I think I will add two more drives and fill all my ports just for some extra head room in the pool. I know there is more to consider here but this one thing tells me to increase the number of drives in the pool so I can write fast enough. If I read this all right then I shouldnt need to add the NVME. Does this sound right? Am I in the right ball part or am I still lost? LN2arc, SLOG or any reason I should be adding the NVME? I would like to just rule it out as wasted hardware if I am reading everything right? I would like to get the best performance possible from this and this is all backed up before it goes to the Nas so if it dies I am not out anything.

You can turn off sync writes

I thought I read this was a huge no no? I thought it is disabling all the data write protections I will call it. Kind of like using ECC vs none ECC? Does sync write have a big performance hit?

Thanks for responding to me.

 

[jgreco]

I thought I read this was a huge no no? I thought it is disabling all the data write protections I will call it. Kind of like using ECC vs none ECC? Does sync write have a big performance hit?

Thanks for responding to me.

It depends on what you're writing. If you work for a bank and are using it to maintain the ledger of customer accounts, you want sync. If you are using your filer to store iSCSI vmdk images for hypervisors, you want sync. If you are storing database files, you want sync. In other cases, you might not want or need to.

 

[XOC_JOE]

It depends on what you're writing. If you work for a bank and are using it to maintain the ledger of customer accounts, you want sync. If you are using your filer to store iSCSI vmdk images for hypervisors, you want sync. If you are storing database files, you want sync. In other cases, you might not want or need to.

Ok... That makes sense being this is not mission crucial info we are dealing with.. Did you see my other response to your comment about the SLOG? The forum hides it on the screen for some reason. I am not sure how I did that. I am still wondering if I have that info correct.

 

[HoneyBadger]

I planned on heavily tuning my ram. The great thing with doing XOC all the time. I can tune ram to run like crazy if needed.

The "tuning" here should be optimized towards hardware stability first. Performance can come second, and is usually best obtained from your RAM by "adding more."

I reread this again. To a beginner some of this seems to contradict itself. I know its not but thats how I am interpreting it for what ever reason. My biggest take away was that for a given amount of ram like mine 128 gigs I have to make sure I can write at least 16gigs or 1/8 my system memory every 5 seconds to the drive pool. If I cant then I am creating a bottle neck. (This would be a worse case) My network wont be able to move 16gigs in 5 seconds so the transaction group will start to clear from the ram and be written to the pool by the 5 second timer. My network should only be able to move about 7ish gigs of data to the transaction group before the timer limit hits again. My 5 current drives running through the chipset can handle that but to be safe I think I will add two more drives and fill all my ports just for some extra head room in the pool. I know there is more to consider here but this one thing tells me to increase the number of drives in the pool so I can write fast enough. If I read this all right then I shouldnt need to add the NVME. Does this sound right? Am I in the right ball part or am I still lost? LN2arc, SLOG or any reason I should be adding the NVME? I would like to just rule it out as wasted hardware if I am reading everything right? I would like to get the best performance possible from this and this is all backed up before it goes to the Nas so if it dies I am not out anything.

The original post is a little bit "vintage" at this point, heralding from 2013. I posted an update in-thread that's also a bit aged (2019) but is a bit more up-to-date:

Some insights into SLOG/ZIL with ZFS on FreeNAS

they (all of a sudden) care more about the "speed" (exactly latency) than about the (small) possible data loss You can work with this. The lack of power loss prevention (PLP) on the drive is (or rather will be) directly related to the poor SLOG performance; most drives are very slow if you...

www.truenas.com

The old "1/8th every 5 seconds" doesn't apply any longer - I did my best to simplify the explanation there, but it is a bit generalized.

Your described use case of "file storage and Plex media" doesn't appear to demand an SLOG or L2ARC.

 

[XOC_JOE]

I did my best to simplify the explanation there

That made a lot of sense. Thanks. Sounds like adding a NVME would just be a waste of hardware at this point.