Hi all,
First things first, I'm in Australia, so keep that in mind when making part suggestions (although I do buy a lot of refurbished parts off eBay)
I've made quite a few upgrades to my current TrueNAS build but I'm thinking of splitting off my iSCSI pool into a separate system.
The reasoning is simple, I want maximum performance and lowest latency out of my iSCSI pool and I feel the best way to do this is to isolate it from my main TrueNAS box which mostly manages SMB/CIFS traffic. I don't run any VMs or Jails on my TrueNAS systems as I have an ESXi box which runs VMs for that sort of stuff.
I plan to build the system in two steps, the first a prototype of sorts to confirm I can get the performance I want with hardware I own, and then the final system with some extra special sauce like 100gbE Chelsio NICs that can do iSCSI offload and/or iSER.
Now... my system does some things that may break some TrueNAS rules, but please keep in mind the data on this iSCSI ZVOL is NOT critical - it can easily be redownloaded or reobtained so if I get corruption or break things, that's totally OK. I am experienced with redlining systems though so I think I can get it stable.
The prototype will consist of the following hardware:
Case: Corsair 900D or Xigmatek Elysium
CPU: 1660v2, locked to 4ghz (or above) on all cores
RAM: 64GB 1866mhz DDR3 ECC
Mobo: X79 Extreme11 - plenty of PCIe due to PLX chips, no need for IPMI as I'll only turn it on/off when I'm physically near it
HBA: mobo's onboard LSI SAS 2308
HDDs: 8 x Toshiba 7200rpm HDDs in striped mirrors
SLOG: 2 x Optane 900p 280GB that i'll "borrow" from my main TrueNAS system
NIC: Mellanox ConnectX-3 dual port with 56gbE enabled and directly connected via DAC cable to the client
Boot: One of my spare cheap SSDs
The client system also has a ConnectX-3 56gbE card.
The proposed final system will consist of the following:
Case: TGC-316 - 16 bay, 3U
CPU: same as above
RAM: 128GB 1866mhz DDR3 ECC (maximum supported by CPU)
Mobo: same as above
HBA: onboard LSI SAS 2308 + PCIe LSI SAS 2308, eg. LSI 9217-8i
HDDs: 14 x Toshiba 7200rpm HDDs in striped mirrors, 2 slots left open for potential hot spares or easy expansion in the future
SLOG: 2 x Optane 905p M.2 or 2 x Optane P4800X or 2 x Radian RMS-200 8GB
L2ARC: 4 (or more?) x Optane 900p 280GB via 8 port Intel OCuLink card
NIC: Chelsio T62100-CR dual port for client and TrueNAS
Boot: 2 x 80gb Intel 320 SSDs
I can get a reasonable deal on a Xeon W-2123 and a Supermicro X11SRL-F, but it has less PCIe lanes and I'm not sure the IPC of the Xeon W will outscale an overclocked E5-1660v2. It'd also require me to buy DDR4 ECC memory, which isn't as cheap as I'd like. Plus again, that'd be new parts I'd have to buy which would take away from the budget for say, the SLOGs, NICs and L2ARC.
I'm sure one question may be "why not SSDs?" and the answer is simple, they're too expensive currently to justify, and I feel having HDDs with SSD caches will provide much greater capacity without requiring me to buy ~$500 AUD 4TB SSDs. I'm also hoping that by going to 14 spindles in the final system that'll give plenty of IOPS.
The client system will be using the following:
CPU: Ryzen 7 5800X (watercooled)
Mobo: ASUS X570 Pro WS (only ryzen board with 3 x x8 4.0 slots)
RAM: 32gb 3600mhz CL14 DDR4
GPU: EVGA RTX 3090 FTW3 24GB (watercooled)
OS: Windows 10 Pro for Workstations
NIC: Mellanox ConnectX-3 dual port
Now, I know I am limited by PCH bandwidth from the CPU to the NIC - it's an x4 4.0 link to the x8 4.0 slot that my current NIC is in. I'm OK with that trade off, it still means I get around 7880MB/sec theoretical bandwidth which is still pretty good. Plus... I may upgrade to a Threadripper system in the future when the 5000 series Threadrippers come out (assuming there's a 16 core variant - my workloads don't need 24 or 32 cores).
So, are there any major issues/flaws with the proposals I've made here? I know the first thing people will point out is the overclocked CPU. I realise it's totally non-standard and not the "done thing" but like I said, the data isn't essential and I'm after pure performance. I won't RAID-0/stripe the disks as I feel RAID-10/stripe mirrors still provide plenty of IOPs while also making it less likely stuff will get corrupted.
Thank you so much for any advice that can be provided :D
I should mention, I'm also open to running pure FreeBSD if it means I can take advantage of iSER. I'd try the iSCSI offload on the Chelsio cards first, and then based off performance I'd then try iSER if plain iSCSI wasn't enough.
First things first, I'm in Australia, so keep that in mind when making part suggestions (although I do buy a lot of refurbished parts off eBay)
I've made quite a few upgrades to my current TrueNAS build but I'm thinking of splitting off my iSCSI pool into a separate system.
The reasoning is simple, I want maximum performance and lowest latency out of my iSCSI pool and I feel the best way to do this is to isolate it from my main TrueNAS box which mostly manages SMB/CIFS traffic. I don't run any VMs or Jails on my TrueNAS systems as I have an ESXi box which runs VMs for that sort of stuff.
I plan to build the system in two steps, the first a prototype of sorts to confirm I can get the performance I want with hardware I own, and then the final system with some extra special sauce like 100gbE Chelsio NICs that can do iSCSI offload and/or iSER.
Now... my system does some things that may break some TrueNAS rules, but please keep in mind the data on this iSCSI ZVOL is NOT critical - it can easily be redownloaded or reobtained so if I get corruption or break things, that's totally OK. I am experienced with redlining systems though so I think I can get it stable.
The prototype will consist of the following hardware:
Case: Corsair 900D or Xigmatek Elysium
CPU: 1660v2, locked to 4ghz (or above) on all cores
RAM: 64GB 1866mhz DDR3 ECC
Mobo: X79 Extreme11 - plenty of PCIe due to PLX chips, no need for IPMI as I'll only turn it on/off when I'm physically near it
HBA: mobo's onboard LSI SAS 2308
HDDs: 8 x Toshiba 7200rpm HDDs in striped mirrors
SLOG: 2 x Optane 900p 280GB that i'll "borrow" from my main TrueNAS system
NIC: Mellanox ConnectX-3 dual port with 56gbE enabled and directly connected via DAC cable to the client
Boot: One of my spare cheap SSDs
The client system also has a ConnectX-3 56gbE card.
The proposed final system will consist of the following:
Case: TGC-316 - 16 bay, 3U
CPU: same as above
RAM: 128GB 1866mhz DDR3 ECC (maximum supported by CPU)
Mobo: same as above
HBA: onboard LSI SAS 2308 + PCIe LSI SAS 2308, eg. LSI 9217-8i
HDDs: 14 x Toshiba 7200rpm HDDs in striped mirrors, 2 slots left open for potential hot spares or easy expansion in the future
SLOG: 2 x Optane 905p M.2 or 2 x Optane P4800X or 2 x Radian RMS-200 8GB
L2ARC: 4 (or more?) x Optane 900p 280GB via 8 port Intel OCuLink card
NIC: Chelsio T62100-CR dual port for client and TrueNAS
Boot: 2 x 80gb Intel 320 SSDs
I can get a reasonable deal on a Xeon W-2123 and a Supermicro X11SRL-F, but it has less PCIe lanes and I'm not sure the IPC of the Xeon W will outscale an overclocked E5-1660v2. It'd also require me to buy DDR4 ECC memory, which isn't as cheap as I'd like. Plus again, that'd be new parts I'd have to buy which would take away from the budget for say, the SLOGs, NICs and L2ARC.
I'm sure one question may be "why not SSDs?" and the answer is simple, they're too expensive currently to justify, and I feel having HDDs with SSD caches will provide much greater capacity without requiring me to buy ~$500 AUD 4TB SSDs. I'm also hoping that by going to 14 spindles in the final system that'll give plenty of IOPS.
The client system will be using the following:
CPU: Ryzen 7 5800X (watercooled)
Mobo: ASUS X570 Pro WS (only ryzen board with 3 x x8 4.0 slots)
RAM: 32gb 3600mhz CL14 DDR4
GPU: EVGA RTX 3090 FTW3 24GB (watercooled)
OS: Windows 10 Pro for Workstations
NIC: Mellanox ConnectX-3 dual port
Now, I know I am limited by PCH bandwidth from the CPU to the NIC - it's an x4 4.0 link to the x8 4.0 slot that my current NIC is in. I'm OK with that trade off, it still means I get around 7880MB/sec theoretical bandwidth which is still pretty good. Plus... I may upgrade to a Threadripper system in the future when the 5000 series Threadrippers come out (assuming there's a 16 core variant - my workloads don't need 24 or 32 cores).
So, are there any major issues/flaws with the proposals I've made here? I know the first thing people will point out is the overclocked CPU. I realise it's totally non-standard and not the "done thing" but like I said, the data isn't essential and I'm after pure performance. I won't RAID-0/stripe the disks as I feel RAID-10/stripe mirrors still provide plenty of IOPs while also making it less likely stuff will get corrupted.
Thank you so much for any advice that can be provided :D
I should mention, I'm also open to running pure FreeBSD if it means I can take advantage of iSER. I'd try the iSCSI offload on the Chelsio cards first, and then based off performance I'd then try iSER if plain iSCSI wasn't enough.
Last edited: