Wanting to build high performance iSCSI server for single Win10 client

Sirius

Dabbler
Joined
Mar 1, 2018
Messages
41
Hi all,

First things first, I'm in Australia, so keep that in mind when making part suggestions (although I do buy a lot of refurbished parts off eBay)

I've made quite a few upgrades to my current TrueNAS build but I'm thinking of splitting off my iSCSI pool into a separate system.
The reasoning is simple, I want maximum performance and lowest latency out of my iSCSI pool and I feel the best way to do this is to isolate it from my main TrueNAS box which mostly manages SMB/CIFS traffic. I don't run any VMs or Jails on my TrueNAS systems as I have an ESXi box which runs VMs for that sort of stuff.

I plan to build the system in two steps, the first a prototype of sorts to confirm I can get the performance I want with hardware I own, and then the final system with some extra special sauce like 100gbE Chelsio NICs that can do iSCSI offload and/or iSER.

Now... my system does some things that may break some TrueNAS rules, but please keep in mind the data on this iSCSI ZVOL is NOT critical - it can easily be redownloaded or reobtained so if I get corruption or break things, that's totally OK. I am experienced with redlining systems though so I think I can get it stable.

The prototype will consist of the following hardware:

Case: Corsair 900D or Xigmatek Elysium
CPU: 1660v2, locked to 4ghz (or above) on all cores
RAM: 64GB 1866mhz DDR3 ECC
Mobo: X79 Extreme11 - plenty of PCIe due to PLX chips, no need for IPMI as I'll only turn it on/off when I'm physically near it
HBA: mobo's onboard LSI SAS 2308
HDDs: 8 x Toshiba 7200rpm HDDs in striped mirrors
SLOG: 2 x Optane 900p 280GB that i'll "borrow" from my main TrueNAS system
NIC: Mellanox ConnectX-3 dual port with 56gbE enabled and directly connected via DAC cable to the client
Boot: One of my spare cheap SSDs

The client system also has a ConnectX-3 56gbE card.

The proposed final system will consist of the following:

Case: TGC-316 - 16 bay, 3U
CPU: same as above
RAM: 128GB 1866mhz DDR3 ECC (maximum supported by CPU)
Mobo: same as above
HBA: onboard LSI SAS 2308 + PCIe LSI SAS 2308, eg. LSI 9217-8i
HDDs: 14 x Toshiba 7200rpm HDDs in striped mirrors, 2 slots left open for potential hot spares or easy expansion in the future
SLOG: 2 x Optane 905p M.2 or 2 x Optane P4800X or 2 x Radian RMS-200 8GB
L2ARC: 4 (or more?) x Optane 900p 280GB via 8 port Intel OCuLink card
NIC: Chelsio T62100-CR dual port for client and TrueNAS
Boot: 2 x 80gb Intel 320 SSDs

I can get a reasonable deal on a Xeon W-2123 and a Supermicro X11SRL-F, but it has less PCIe lanes and I'm not sure the IPC of the Xeon W will outscale an overclocked E5-1660v2. It'd also require me to buy DDR4 ECC memory, which isn't as cheap as I'd like. Plus again, that'd be new parts I'd have to buy which would take away from the budget for say, the SLOGs, NICs and L2ARC.

I'm sure one question may be "why not SSDs?" and the answer is simple, they're too expensive currently to justify, and I feel having HDDs with SSD caches will provide much greater capacity without requiring me to buy ~$500 AUD 4TB SSDs. I'm also hoping that by going to 14 spindles in the final system that'll give plenty of IOPS.

The client system will be using the following:
CPU: Ryzen 7 5800X (watercooled)
Mobo: ASUS X570 Pro WS (only ryzen board with 3 x x8 4.0 slots)
RAM: 32gb 3600mhz CL14 DDR4
GPU: EVGA RTX 3090 FTW3 24GB (watercooled)
OS: Windows 10 Pro for Workstations
NIC: Mellanox ConnectX-3 dual port

Now, I know I am limited by PCH bandwidth from the CPU to the NIC - it's an x4 4.0 link to the x8 4.0 slot that my current NIC is in. I'm OK with that trade off, it still means I get around 7880MB/sec theoretical bandwidth which is still pretty good. Plus... I may upgrade to a Threadripper system in the future when the 5000 series Threadrippers come out (assuming there's a 16 core variant - my workloads don't need 24 or 32 cores).

So, are there any major issues/flaws with the proposals I've made here? I know the first thing people will point out is the overclocked CPU. I realise it's totally non-standard and not the "done thing" but like I said, the data isn't essential and I'm after pure performance. I won't RAID-0/stripe the disks as I feel RAID-10/stripe mirrors still provide plenty of IOPs while also making it less likely stuff will get corrupted.

Thank you so much for any advice that can be provided :D
I should mention, I'm also open to running pure FreeBSD if it means I can take advantage of iSER. I'd try the iSCSI offload on the Chelsio cards first, and then based off performance I'd then try iSER if plain iSCSI wasn't enough.
 
Last edited:

Sirius

Dabbler
Joined
Mar 1, 2018
Messages
41
So I built the prototype to do some testing.

I was able to fit in 128GB ECC 1866mhz memory however, so that's good. I also ended up partitioning up my 2 x Optane 900p into both SLOG and L2ARC (32gb each for SLOG, 128gb each for L2ARC)

I've attached my CrystalDiskMark results for the setup - all 6 cores/12 threads are locked to 4ghz. I'm going to keep bumping the multiplier and see if the performance scales.

I also wonder, if due to ZFS's ARC, would faster memory help? Say, 2133mhz or above?
 

Attachments

  • iscsi performance - test 2 - 4ghz 1660v2.png
    iscsi performance - test 2 - 4ghz 1660v2.png
    127.4 KB · Views: 214

Sirius

Dabbler
Joined
Mar 1, 2018
Messages
41
I ran the Anvil SSD benchmark and I was surprised at how close iSCSI is getting to a direct attached PCIe Gen 3 SSD.

I'm still after advice... Would it be worth upgrading to the Chelsio 100gbE NICs? I'm thinking that even if I can't max out the bandwidth the latency should be lower and using iSCSI offload on both the initiator and target should free up CPU on both ends.

I assume then I'd be better off with 4 x 900p in RAID 10 for the SLOG in order to keep up with the 100gbE or faster individual device like the 905p or DC P4800X

I assume the Radian RMS-200 8GB wouldn't work due to it being so small? Maybe I could do something crazy like stripe 2 of them?
 

Attachments

  • anvil - 970 pro 1TB score.png
    anvil - 970 pro 1TB score.png
    377 KB · Views: 169
  • anvil score - 1660v2 at 4ghz.png
    anvil score - 1660v2 at 4ghz.png
    361.5 KB · Views: 154
Joined
Jul 18, 2022
Messages
4
I ran the Anvil SSD benchmark and I was surprised at how close iSCSI is getting to a direct attached PCIe Gen 3 SSD.

I'm still after advice... Would it be worth upgrading to the Chelsio 100gbE NICs? I'm thinking that even if I can't max out the bandwidth the latency should be lower and using iSCSI offload on both the initiator and target should free up CPU on both ends.

I assume then I'd be better off with 4 x 900p in RAID 10 for the SLOG in order to keep up with the 100gbE or faster individual device like the 905p or DC P4800X

I assume the Radian RMS-200 8GB wouldn't work due to it being so small? Maybe I could do something crazy like stripe 2 of them?
Did you end up going with the RAID of 900p for the SLOG here if you ended up going to 100Gb? Any takeaways?
 
Top