Building a 'Little' storage box for first time

[BulgarianHealer]

Hello,

I have been using my personal PC for storage, but with the allocation of the HDD, I had to build something more serious. After looking for a user-friendly OS, I stopped on Truenas. So I started searching for MB, CPU, etc., after buying the components. I had to build a custom server case because buying a second-hand or new one for 24bays HDD is a F* nightmare because of the price.


What I bought for the system is:

• MB: Huananzhi x99 f8
• CPU: Intel Xeon E5-2699v3
• RAM: 8x8GB DDR4 2666MHz ECC
• PSU: Seasonic FOCUS GX-850
• HBA: LSI 9217-8i
• BACKPLANE: 2xSuperMicro bpn-sas826a
• EXPANDER: INTEL RES2SV240
• COOLER: NH-U12S redux
• FANS: 3x NF-A14 iPPC-3000 PWM

as for the HDDs I have 12x4TB ST4000NM0023, 12x6TB ST6000NM0014 and some nvmes.

So the HDD are a mix of 6/12Gb/s and i don't think that's gonna be a problem.

For boot, I'm using 2x PC401 NVMe SK hynix 256GB, I have created two pools:

Pool 1: 12x4TB Seagate ST4000NM0023 and 2x PC401 NVMe SK hynix 512GB for LOG in mirror
Pool 2: 12x6TB Seagate ST6000NM0014 and 2x PC401 NVMe SK hynix 512GB for LOG in mirror

Both pools are in raidz3 in order to have time to fix possible single/multiple HDD failures in a short/long period of time.

Also in the feature, i think to buy one more HBA (with 8e/16e) and EXPANDER(maybe same intel) to create a new server case with custom HDD cages with SuperMicro bpn-sas826a backplane and another PSU.

My main question is the configuration good enough for video storage(+PLEX(transcoding with the cpu)) and VM's(small web projects) and do you think that there will be any bottleneck aside from the network card(i have plans to get 3x10gbps)?

Thanks.

 

[NugentS]

Why do you have a log (SLOG) vdev?
What do you think it will achieve?

 

[BulgarianHealer]

Why do you have a log (SLOG) vdev?
What do you think it will achieve?

1. The idea was to use SLOG was improve the writing performance(like a cache drive, maybe I'm mistaking it) over the SMB
2. Better and stable writing speed on sync.

Seems I have mistaken the whole way of what is SLOG and how it works, but ready to learn from others. :)

 

[NugentS]

A Slog only effects sync writes, essentially NFS and iSCSI. Does squat all to SMB
A Slog also slows things down, rather than speed things up. Its a data safety thing. It can speed up sync writes, but not as much as setting sync=disabled (although that isn't data safe in the event of an unexpected shutdown.) Note that a SLOG is not, never has been and never will be a write or read cache of any kind

Lastly and perhaps most importantly a SLOG has very specific and quite tough hardware requirements - which you are very unlikley to get from a consumer drive (you need enterprise hardware)

Very High Speed
Very high endurance - its always written to, and never read from under a steady state, and then only with sync writes
Power Loss Protection - this drives the enterprise grade hardware requirement. Or Optane 900p - which is close

I notice you mention VM's - are these external VM's or VM's on Scale. Either way VM's need IOPS, and you are running RAIDZn which is low on IOPS. VM's need mirrors. I would suggest taking those 4 proposed SLOG drives and setting them up as a 2*2 Striped Mirror and using them for VM's and then consider a SLOG, using appropriate enterprise hardware, for that pool. Also a SLOG is not pool critical so does not have to be resilient (although if a single drive fails at the same time as an unscheduled reboot that can be a problem). If a SLOG fails during normal use then apart from a slowdown it doesn't really matter.

 

[elvisimprsntr]

Welcome!

HBA: LSI 9217-8i
EXPANDER: INTEL RES2SV240

RAID Controllers? Have you flashed both to IT mode?

 

[BulgarianHealer]

Welcome!


RAID Controllers? Have you flashed both to IT mode?

The LSI 9217-8i is with IR mode, I will flash it to IT in the upcoming weekend.

Firmware Product ID : 0x2714 (IR)
Firmware Version : 20.00.07.00

The expander not sure
Device is a Enclosure services device
Enclosure # : 2
Slot # : 24
SAS Address : 5001e67-7-be42-affd
State : Standby (SBY)
Manufacturer : Intel
Model Number : RES2SV240
Firmware Revision : 0d00
Serial No : x362541300
GUID : N/A
Protocol : SAS
Device Type : Enclosure services device

One backplane is fully connected to the expander, and 2 slots are for the 2-nd expander, and 1 slot for connection to the HBA, and from the HBA one cable is going to the 3-rd port on the 2-nd backplane.

 

[BulgarianHealer]

A Slog only effects sync writes, essentially NFS and iSCSI. Does squat all to SMB
A Slog also slows things down, rather than speed things up. Its a data safety thing. It can speed up sync writes, but not as much as setting sync=disabled (although that isn't data safe in the event of an unexpected shutdown.) Note that a SLOG is not, never has been and never will be a write or read cache of any kind

Lastly and perhaps most importantly a SLOG has very specific and quite tough hardware requirements - which you are very unlikley to get from a consumer drive (you need enterprise hardware)

Very High Speed
Very high endurance - its always written to, and never read from under a steady state, and then only with sync writes
Power Loss Protection - this drives the enterprise grade hardware requirement. Or Optane 900p - which is close

I notice you mention VM's - are these external VM's or VM's on Scale. Either way VM's need IOPS, and you are running RAIDZn which is low on IOPS. VM's need mirrors. I would suggest taking those 4 proposed SLOG drives and setting them up as a 2*2 Striped Mirror and using them for VM's and then consider a SLOG, using appropriate enterprise hardware, for that pool. Also a SLOG is not pool critical so does not have to be resilient (although if a single drive fails at the same time as an unscheduled reboot that can be a problem). If a SLOG fails during normal use then apart from a slowdown it doesn't really matter.

Okay, for the VM's will stash those 4 nvmes.

 

[oordonezm]

What I bought for the system is:

• MB: Huananzhi x99 f8
• CPU: Intel Xeon E5-2699v3
• RAM: 8x8GB DDR4 2666MHz ECC
• PSU: Seasonic FOCUS GX-850
• HBA: LSI 9217-8i
• BACKPLANE: 2xSuperMicro bpn-sas826a
• EXPANDER: INTEL RES2SV240
• COOLER: NH-U12S redux
• FANS: 3x NF-A14 iPPC-3000 PWM

Please, can you share the purchase links for your components? I plan to build the same thing, and how is it working with the Huanazhi x99 f8 motherboard?