New Build - Looking for advice to see if there is any holes

dome

Cadet
Joined
Apr 21, 2021
Messages
3
Currently, the home setup is as follows

2 x DL380 G9, 2697 V3, 512GB Ram
2 x Intel Nuc ( always on hosts )

The function of the TrueNAS box will be to move some/all of the local storage from the DL380 G9 hosts to Shared Storage and to move my Network Attached Storage away from my QNAP. My idea was to build a box that would fulfill both of these.

The internal Network is 10GB via SFP+ and some TGBe. DAC is what's used apart from a few fiber runs to the office.

Regardless All of the hosts above run inside a Netshelter CX Rack where the new Supermicro chassis will be going. I'm really just looking for a sanity check to make sure I'm not making any huge mistakes on the below config.

What I've purchased so far

Supermciro CSE-829U with the X10DRU-i+ Mobo
CPU X 2: 2630 V4
RAM: 128GB DDR4 2133 ECC ( 4 x 32GB )
HBA: LSI 9400-16i SAS/SATA/NVME U.2 ( Already in IT Mode )
Boot: Intell SSD's connected to the onboard SATA connectors in a mirror

What I have still to Purchase

Disks:
I don't really have a preference for disks, However, around 40TB of total storage would be the goal. Whether that is 6x10TB in Z2 etc. room for expansion would be an advantage too. oh and cost, so I don't mind getting refurbished drives and just grabbing a few extra to sit of the shelf

read write Cache:
2 X Optane 280GB 900p u.2. These can be sourced on eBay for around the £150 mark for the u.2 SAS versions so a pair of these would utilize the 2 spare mini SAS connectors from the HBA


Thanks in advance for checking the post out.
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
Thoughts:
  • The tri-mode cards are iffy. The drivers are less than stable and much of the value proposition hinges on massively expensive tri-mode expander backplanes. A plain SAS3008 HBA plus separate PCIe connections are likely to cause fewer headaches.
  • For 40 TB of available storage, aim for 6x12 TB instead of 10 TB. Of course, YMMV depending on your data.
  • I have doubts that L2ARC or an SLOG would be beneficial. You may be optimizing something that doesn't need it. Again, depends on your workload.
  • Nitpick: two DRAM channels @ 2133 MHz per CPU is a bit on the "starved" side of memory bandwidth. It might not matter at all, but it's something to keep in mind.
 

dome

Cadet
Joined
Apr 21, 2021
Messages
3
Thoughts:
  • The tri-mode cards are iffy. The drivers are less than stable and much of the value proposition hinges on massively expensive tri-mode expander backplanes. A plain SAS3008 HBA plus separate PCIe connections are likely to cause fewer headaches.
  • For 40 TB of available storage, aim for 6x12 TB instead of 10 TB. Of course, YMMV depending on your data.
  • I have doubts that L2ARC or an SLOG would be beneficial. You may be optimizing something that doesn't need it. Again, depends on your workload.
  • Nitpick: two DRAM channels @ 2133 MHz per CPU is a bit on the "starved" side of memory bandwidth. It might not matter at all, but it's something to keep in mind.
Thanks for the response,

  • Tri-mode. I plan on tinkering around before I put it into so-called production. I'm lucky that the backplane on my chassis did come with the backplane in which it supports NVME as well as SAS. so if I feel it's not stable enough I can relist this on eBay and source something else. and no doubt save money in the process
  • Storage. This comes down to how much I'm willing to spend really. but that 40TB is the goal.
  • L2ARC. the workload can be from 10 vms running off the nucs to 90+ when the main hosts are up and I'm testing a new release of various products that are used at work. I was more hoping it would improve io performance.
  • Nitpick, Yes this was a deal I couldn't refuse last week. it will work for testing purposes but I do plan on getting a matching set and doubling that to 256. It should then improve the memory bandwidth as you mentioned.
thanks again for taking the time to check over the
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
the backplane on my chassis did come with the backplane in which it supports NVME as well as SAS
But what kind of backplane is it? If it has separate connectors for SATA/SAS and PCIe, it's not going to take advantage of the tri-mode stuff (other than the HBA being a glorified PCIe switch) and it should be easy to wire up with separate SAS and PCIe. This is the most likely scenario for a Supermicro X10 server.
If it's passive and has a single connector (i.e. it's a U.3 backplane), it'll need a tri-mode controller most likely. The DIY ecosystem for this sort of thing still feels very experimental to me and I'm not sure what details may end up being problematic. One thing I do know is that you'd need U.3 disks, specifically.
If it's an expander backplane, then it'll definitely need a compatible HBA.

L2ARC. the workload can be from 10 vms running off the nucs to 90+ when the main hosts are up and I'm testing a new release of various products that are used at work. I was more hoping it would improve io performance.
Alright, so running block storage... If you can live without sync writes and the added protection that comes with them, just disable sync writes and skip the SLOG. For L2ARC, it's going to depend on your ARC hit rate and deadlist hit rates. If the deadlists have a bunch of hits, you'd benefit from more ARC or L2ARC.
 

dome

Cadet
Joined
Apr 21, 2021
Messages
3
But what kind of backplane is it? If it has separate connectors for SATA/SAS and PCIe, it's not going to take advantage of the tri-mode stuff (other than the HBA being a glorified PCIe switch) and it should be easy to wire up with separate SAS and PCIe. This is the most likely scenario for a Supermicro X10 server.
If it's passive and has a single connector (i.e. it's a U.3 backplane), it'll need a tri-mode controller most likely. The DIY ecosystem for this sort of thing still feels very experimental to me and I'm not sure what details may end up being problematic. One thing I do know is that you'd need U.3 disks, specifically.
If it's an expander backplane, then it'll definitely need a compatible HBA.


Alright, so running block storage... If you can live without sync writes and the added protection that comes with them, just disable sync writes and skip the SLOG. For L2ARC, it's going to depend on your ARC hit rate and deadlist hit rates. If the deadlists have a bunch of hits, you'd benefit from more ARC or L2ARC.


Apologies for the delay in reply, The backplane is the Supermicro BPN-SAS3-826EL1-N4. I believe this comes with the option of using up to 4 u.2 discs. I realize that the bottleneck will be the actual PCIe bus, but I suppose it's to futureproof myself.

On the SLOG, L2ARC, ill try without and see what it's like. I can always throw in some test discs to see if there is a noticeable performance gain afterwords.

Thanks again for your comments.
 
Top