Hey! I'm relatively new to TrueNAS. I've asked a few questions around the forums during my first test build. For the most part everything went smooth and I've managed to get everything I wanted up and running.
This post may be a bit loaded.
What I want to ask about was general hardware specifications for a specific build/setup that I have in mind, but due to lack of experience with ZFS, I'm curious where hardware starts to become a waste.
Ultimately, I'd like to build a 12 or 24 drive bay chassis for high read storage for video editing projects. We often have multiple editors (up to 8-10 at times, but usually 4-5). I just can't afford to purchase a turnkey solution for some projects, it's easy to bleed into the $50k+ range on those.
Currently I have a 2U Chassis with Dual Xeon 2600 v3, 64gb RAM, 2x M2 PCIE, 2x 10gbe w/ 12 drive bays and 2x SSD's mirroring the boot drive. I'm leaning towards 14TB WD DC drives for the 12 bays.
How important is CPU?
Costs drive up exponentially as you move to dual CPU's (especially Intel), and the faster you get, it's really easy to spec out a $15k+ server before you even get to storage. So where do you draw that line? What are you looking for in a CPU for a ZFS system? I just don't want to skimp.
How much RAM?
I'll be going with ECC. But after 64-128gb, what are the gains? How do you calculate how much RAM you will actually need for a large storage pool? Obviously ZFS can use it for caching...but to what extent? I haven't seen a system absolutely hammered in a real world situations.
L2ARC
On paper, this seems logical. But in practice, the community seems very mixed on the benefits of L2ARC. I'm aiming for focus on higher maintained read speeds but will be dealing with large amounts of data being pulled from multiple clients. Write speeds are nice, but less critical. L2ARC isnt' a large cost edition on a small scale. Is there a risk or a point where it degrades performance?
SLOG
I'm still trying to wrap my head around this one...benifits? Not needed?
OS SSD / SWAP
When I install TrueNAS, it asks me if I want to enable SWAP on the boot SSD's. Suggestions on that? I read if you're going to be running SSD/L2ARC then the answer is no, looking for suggestions based on experience. Seems mixed in the forums. I couldn't find small SSD's in stock when I was building out my test system, so they are overkill on Size. 500gb SanDisk 3D Plus drives mirrored.
Storage
Advantage over a pure spinning disk configuration? Eliminates the need for L2ARC?
Mirrored vDevs or RaidZ? All research points to mirrored vDevs, but most of that is based on words written 3-8 years ago. I don't see RAID 10 setups that often through work anymore.
Current Build
I am just experimenting with the hardware and learning TrueNAS at the moment. This is NOT a production unit in usage. I don't have 12x platter drives to even install as of yet. So I pulled a few spares to get mirrored vdevs up and running and see how to generate L2ARC, etc and get a feel for the configuration and setup of TrueNAS so I can do some testing.
This post may be a bit loaded.
What I want to ask about was general hardware specifications for a specific build/setup that I have in mind, but due to lack of experience with ZFS, I'm curious where hardware starts to become a waste.
Ultimately, I'd like to build a 12 or 24 drive bay chassis for high read storage for video editing projects. We often have multiple editors (up to 8-10 at times, but usually 4-5). I just can't afford to purchase a turnkey solution for some projects, it's easy to bleed into the $50k+ range on those.
Currently I have a 2U Chassis with Dual Xeon 2600 v3, 64gb RAM, 2x M2 PCIE, 2x 10gbe w/ 12 drive bays and 2x SSD's mirroring the boot drive. I'm leaning towards 14TB WD DC drives for the 12 bays.
How important is CPU?
Costs drive up exponentially as you move to dual CPU's (especially Intel), and the faster you get, it's really easy to spec out a $15k+ server before you even get to storage. So where do you draw that line? What are you looking for in a CPU for a ZFS system? I just don't want to skimp.
How much RAM?
I'll be going with ECC. But after 64-128gb, what are the gains? How do you calculate how much RAM you will actually need for a large storage pool? Obviously ZFS can use it for caching...but to what extent? I haven't seen a system absolutely hammered in a real world situations.
L2ARC
On paper, this seems logical. But in practice, the community seems very mixed on the benefits of L2ARC. I'm aiming for focus on higher maintained read speeds but will be dealing with large amounts of data being pulled from multiple clients. Write speeds are nice, but less critical. L2ARC isnt' a large cost edition on a small scale. Is there a risk or a point where it degrades performance?
SLOG
I'm still trying to wrap my head around this one...benifits? Not needed?
OS SSD / SWAP
When I install TrueNAS, it asks me if I want to enable SWAP on the boot SSD's. Suggestions on that? I read if you're going to be running SSD/L2ARC then the answer is no, looking for suggestions based on experience. Seems mixed in the forums. I couldn't find small SSD's in stock when I was building out my test system, so they are overkill on Size. 500gb SanDisk 3D Plus drives mirrored.
Storage
Advantage over a pure spinning disk configuration? Eliminates the need for L2ARC?
Mirrored vDevs or RaidZ? All research points to mirrored vDevs, but most of that is based on words written 3-8 years ago. I don't see RAID 10 setups that often through work anymore.
Current Build
I am just experimenting with the hardware and learning TrueNAS at the moment. This is NOT a production unit in usage. I don't have 12x platter drives to even install as of yet. So I pulled a few spares to get mirrored vdevs up and running and see how to generate L2ARC, etc and get a feel for the configuration and setup of TrueNAS so I can do some testing.
Code:
capacity operations bandwidth
pool alloc free read write read write
-------------------------------------- ----- ----- ----- ----- ----- -----
boot-pool 1.16G 475G 15 3 201K 14.4K
mirror 1.16G 475G 15 3 201K 14.4K
ada0p2 - - 9 1 107K 7.18K
ada1p2 - - 6 1 94.1K 7.18K
-------------------------------------- ----- ----- ----- ----- ----- -----
pool1 65.4M 9.08T 5 19 62.5K 162K
mirror 21.9M 5.45T 2 10 26.5K 87.4K
gptid/57b0789d-a453-11eb-93de-0cc47a6ea7ec - - 1 5 14.3K 43.7K
gptid/5bf40307-a453-11eb-93de-0cc47a6ea7ec - - 1 5 12.2K 43.7K
mirror 43.5M 3.62T 2 8 36.0K 74.3K
gptid/57e45194-a453-11eb-93de-0cc47a6ea7ec - - 1 4 19.4K 37.1K
gptid/5c0e82ab-a453-11eb-93de-0cc47a6ea7ec - - 1 4 16.5K 37.1K
cache - - - - - -
gptid/2127ca05-a587-11eb-aa0f-0cc47a6ea7ec 1.82M 447G 0 0 317 5.75K
gptid/213c7d4a-a587-11eb-aa0f-0cc47a6ea7ec 1.41M 447G 0 0 317 4.55K
-------------------------------------- ----- ----- ----- ----- ----- -----
