Suggestions for replacing TrueNAS iSCSI-server

[Thomas_VDB]

Hi,

I am replacing a iXsystems TrueNAS core server (6 years old) that we are using as iSCSI storage for 2 ESXi-hosts.

The new quotation from iXsystems is pretty expensive. That's why we 're thinking of building it ourself.
We need similar or better specs. Requirements haven't changed. The old server is still performing well.

Old server :
- Neptune iX2112 : 2U 12 bay LFF - 1200W redundant (1+1) PSU
- Intel Xeon E5-1620 v3@ 3.5GHz
- 32GB DDR4 2133Mhz 17000 ECC/REG
- 10x Hitachi 3TB Ultrastar Enterprise SAS 6Gb/s 7200RPM 64MB Cache (1 big tank in mirroring)
- ZIL : Samsung 128GB 850 Pro Series Solid State Drive (MLC) MZ-7KE128BW
- L2ARC : Samsung 256GB 850 Pro Series Solid State Drive (MLC)
- OS : 32GB SATADOM MLC
- 1x PCIe : 2 port 10GbE SFP+

My budget is around 5k€.

First we were thinking going all-flash, but with enterprise NVME our budget is too tight.
Then I thought of building it with a HP Proliant DL380 Gen10 12x LFF bay.
However the built-in drive/raid controller (P408i) isn't the best option for TrueNAS.

Edit : I think the Supermicro SSG-620P-ACR12L seems a better option over the HP. It has an IT-mode card.

Can you guys give me some suggestions? HP proliant or SuperMicro?
Would WD Gold HD's be good?
Which SSD's for ZIL/ARC? And better up the RAM?

Thx.

 

[HoneyBadger]

Hey Thomas!

Sorry to hear that the quotation from iX didn't come in at the budget you were targeting. Feel free to DM me if you'd like to talk specifics on that side of things.

Regarding your two systems, I would definitely prefer the Supermicro over the HP, as it's already sporting a proper LSI HBA (a tri-mode one, to boot) and Intel networking (although it's 10Gbase-T and your existing environment is using SFP+)

WD Gold drives are certainly fine, WD Red Pro is also an option as well. What are your capacity requirements? This will help drive the question of which size of drives to aim for, as well as general L2ARC guidance.

Your specific choice of Supermicro system gives you a few unique options thanks to the included motherboard, the X12DPi-NT6:

X12DPi-NT6 | Motherboards | Products | Supermicro

Ideally, I'd configure using the internal M.2 slot as a boot device (a simple WD Blue or similar 250G will be fine), assign the two rear-mounted 2.5" bays for L2ARC, and leverage the Optane DC Persistent Memory slots for mirrored SLOG devices.

Note that you can only have one Optane DIMM per CPU socket, so enabling mirrored SLOG this way would require two CPUs as well. You'll also have to pick from a smaller selection of CPUs with Optane PMEM support; the Gold/Platinum line support it, but notably so does the Xeon Silver 4314 for an entry-level option.

RAM should be "as much as your budget will allow" as it's one of the best ways to improve performance on a random-read heavy workload such as virtualization. Try to obtain the largest individual DIMMs that your board supports in order to provide an easy path for future upgrades if desired.

I'm not too familiar with the EU market for costs, but hopefully these suggestions help.

Cheers!

 

[Thomas_VDB]

Auwch this a bit uncomfortable . But seriously: thanks for your advice much appreciated.
We have not yet decided on the ixsystems proposal or going self build.

About the Supermicro capacity requirements :
I was thinking about 10x 4TB in mirroring, and use the 2 remaining bays for hot spares.
Could you suggest some specific devices for the L2ARC and SLOG? I will probably go with only 1 processor because of budget.
Also I read somewhere that for iSCSI, L2ARC and SLOG doesn't really help. Would that be true?
If I were to go for 128GB RAM. Would it be better to have 4x 32 or would 2x 64 be okay.
I know I need ECC, but there are so many types, (2Rx4,...)
About the boot device in the M2 socket : wouldn't it be better to use the 2 SuperDOM sockets to make a boot volume in mirror?
And about 4 of the 12 drives being NVMe hybrid, is that something I could use?

Thanks!

 

[Chris Moore]

Requirements haven't changed. The old server is still performing well.

If your requirements have not changed and the old server is still performing, why are you looking at new hardware?
Where I in your position, I think I would simply replace the old drives with new ones. You can do that one drive at a time, with a rebuild in between, and upgrade from the 3 TB drives you have to new 4 TB drives. Once all drives in a vdev are replaces, you see the capacity automatically expand.
I did that a couple of other times with my system over the years. The electronic components don't fail nearly as often as disks.

 

[HoneyBadger]

Auwch this a bit uncomfortable . But seriously: thanks for your advice much appreciated.
We have not yet decided on the ixsystems proposal or going self build.

About the Supermicro capacity requirements :
I was thinking about 10x 4TB in mirroring, and use the 2 remaining bays for hot spares.
Could you suggest some specific devices for the L2ARC and SLOG? I will probably go with only 1 processor because of budget.
Also I read somewhere that for iSCSI, L2ARC and SLOG doesn't really help. Would that be true?
If I were to go for 128GB RAM. Would it be better to have 4x 32 or would 2x 64 be okay.
I know I need ECC, but there are so many types, (2Rx4,...)
About the boot device in the M2 socket : wouldn't it be better to use the 2 SuperDOM sockets to make a boot volume in mirror?
And about 4 of the 12 drives being NVMe hybrid, is that something I could use?

Thanks!

Just looking to get you the right solution regardless of the way it arrives.

10x4TB in mirrors with two hotspares should give you a very good degree of redundancy. You'll have 20TB of space "after mirroring" but note that performance will be impacted by fragmentation, and the fuller the drives get the more this could be an issue. If you are using much more than 10TB on your current system, I'd recommend using larger drives and building in some extra overhead.

For L2ARC, any "enterprise" targeted SATA or SAS should be fine - just as an example, the Micron 5400 PRO might be a good choice - but indexing the contents of a L2ARC uses a small amount of system memory. A 480GB SSD could use approximately 8GB of RAM for block workloads such as virtual machines (assuming a mix of 4K/8K records [1] ) but the up to 480GB indexed would be read at SSD-speed instead of HDD-speed.

For SLOG, unfortunately the pricing on Optane PMEM Series 200 devices makes them a bit of a no-go for your target budget. Other Optane devices like the DC P4800X (U.2) and DC P4801X (M.2 sized) could be an option for you, especially since you do have the capability to use your front drive bays with NVMe devices. Hot-plugging PCIe is a different challenge than SATA/SAS though, so I wouldn't count on being able to do that quite seamlessly yet. Check out the SLOG thread linked in my signature - maybe even start from the newest posts and skip backwards a few pages at a time looking through results. Optane makes a frequent and powerful showing, although it's officially "end of life" from Intel now.

SLOG benchmarking and finding the best SLOG

@NickF hit some odd snags with regards to the visibility of devices through the UI, likely due to the identifier fields from the device being the same, so it would be a CLI-configured option (and thus technically "unsupported"). You'd also have to be cautious about ensuring that your mirrors are...

www.truenas.com

iSCSI doesn't benefit from SLOG by default, but in a scenario where you're using it to store virtual machines, you absolutely want to set sync=always on every zvol with a live VM - which will mean that every incoming write will hit the SLOG as well. So it needs to be a high-performance, high-endurance device. The random-read workload of VMs is also a prime candidate for L2ARC, so I'm hoping that it was a bit of an incorrect generalization made on the part of whoever told you that.

Considering a single-processor build, probably using something like the aforementioned Xeon Silver, you would probably want to opt for 2x64G, as that lets you expand all the way up to 512G (8x64G) with a single socket. Note that you'll only be able to use three out of six PCIe slots (the furthest three from the socket, I believe) if you only have one processor; if one's in use by a 10Gbps SFP+ NIC that only leaves two more.

Boot devices - you could mirror two SuperDOMs, but in my opinion you'll likely have better overall reliability (and much better performance) with a single NVMe SSD. Another option could be to mount a pair of something like the Silverstone SST-ECM22 cards into the remaining two PCIe slots (https://www.amazon.com/SilverStone-Port-Converter-Adapter-SST-ECM22/dp/B0771HW52P) - use the NVMe M.2 to hold an Optane P4801X, mirror across cards for SLOG, and the SATA M.2 to hold whatever inexpensive SATA M.2 SSD you can source as a mirrored boot device. Best of both worlds, perhaps.

[1] We can check your existing pool for details if you give me the output of zdb -Lbb -U /data/zfs/zpool.cache YourPoolNameHere)

 

[Thomas_VDB]

If your requirements have not changed and the old server is still performing, why are you looking at new hardware?
Where I in your position, I think I would simply replace the old drives with new ones. You can do that one drive at a time, with a rebuild in between, and upgrade from the 3 TB drives you have to new 4 TB drives. Once all drives in a vdev are replaces, you see the capacity automatically expand.
I did that a couple of other times with my system over the years. The electronic components don't fail nearly as often as disks.

Thanks for your reply. I understand your suggestion about keeping the old server going. However I also needto replace a very old 12-bay NAS, and I was thinking about using the current TrueNAS server as a the new NAS, and then replacing the TrueNAS with a new TrueNAS.
So I need to replace it anyhow.

 

[Thomas_VDB]

Just looking to get you the right solution regardless of the way it arrives.

10x4TB in mirrors with two hotspares should give you a very good degree of redundancy. You'll have 20TB of space "after mirroring" but note that performance will be impacted by fragmentation, and the fuller the drives get the more this could be an issue. If you are using much more than 10TB on your current system, I'd recommend using larger drives and building in some extra overhead.

For L2ARC, any "enterprise" targeted SATA or SAS should be fine - just as an example, the Micron 5400 PRO might be a good choice - but indexing the contents of a L2ARC uses a small amount of system memory. A 480GB SSD could use approximately 8GB of RAM for block workloads such as virtual machines (assuming a mix of 4K/8K records [1] ) but the up to 480GB indexed would be read at SSD-speed instead of HDD-speed.

For SLOG, unfortunately the pricing on Optane PMEM Series 200 devices makes them a bit of a no-go for your target budget. Other Optane devices like the DC P4800X (U.2) and DC P4801X (M.2 sized) could be an option for you, especially since you do have the capability to use your front drive bays with NVMe devices. Hot-plugging PCIe is a different challenge than SATA/SAS though, so I wouldn't count on being able to do that quite seamlessly yet. Check out the SLOG thread linked in my signature - maybe even start from the newest posts and skip backwards a few pages at a time looking through results. Optane makes a frequent and powerful showing, although it's officially "end of life" from Intel now.

SLOG benchmarking and finding the best SLOG

@NickF hit some odd snags with regards to the visibility of devices through the UI, likely due to the identifier fields from the device being the same, so it would be a CLI-configured option (and thus technically "unsupported"). You'd also have to be cautious about ensuring that your mirrors are...

www.truenas.com

iSCSI doesn't benefit from SLOG by default, but in a scenario where you're using it to store virtual machines, you absolutely want to set sync=always on every zvol with a live VM - which will mean that every incoming write will hit the SLOG as well. So it needs to be a high-performance, high-endurance device. The random-read workload of VMs is also a prime candidate for L2ARC, so I'm hoping that it was a bit of an incorrect generalization made on the part of whoever told you that.

Considering a single-processor build, probably using something like the aforementioned Xeon Silver, you would probably want to opt for 2x64G, as that lets you expand all the way up to 512G (8x64G) with a single socket. Note that you'll only be able to use three out of six PCIe slots (the furthest three from the socket, I believe) if you only have one processor; if one's in use by a 10Gbps SFP+ NIC that only leaves two more.

Boot devices - you could mirror two SuperDOMs, but in my opinion you'll likely have better overall reliability (and much better performance) with a single NVMe SSD. Another option could be to mount a pair of something like the Silverstone SST-ECM22 cards into the remaining two PCIe slots (https://www.amazon.com/SilverStone-Port-Converter-Adapter-SST-ECM22/dp/B0771HW52P) - use the NVMe M.2 to hold an Optane P4801X, mirror across cards for SLOG, and the SATA M.2 to hold whatever inexpensive SATA M.2 SSD you can source as a mirrored boot device. Best of both worlds, perhaps.

[1] We can check your existing pool for details if you give me the output of zdb -Lbb -U /data/zfs/zpool.cache YourPoolNameHere)

Hi, thanks for the great advice! The last part is a bit confusing to me, and also I don't think I can use P4800X or P4801X devices (+1k€) because of budget.
So far I have :
- Server : SuperMicro SSG-620P-ACR12L
- RAM : 2x 64 GB ECC
- Processor : Intel xeon Silver 4310
- Storage : 12x 4TB WD Red
(With these 4 items I am already over 5k€ budget )
- L2ARC : Micron 5400 Pro 2,5" 480 GB
- SLOG : ?
- CPU cooler ?
- Boot : Silverstone SST-ECM22 + 2 M.2 NVME Drives (Would 2x WD Green SN350 240GB be okay?)

So the L2Arc probably goes in on of the 2 rear mounted 2.5" bays. It's not very clear for me what you want to put in the onboard M.2 bay, and what goes in the Optane DC Persistent Memory slots? The xeon 4310 takes a big bite out of the budget. Is it necessary?

 

[Thomas_VDB]

About our current setup :
- the "zdb -lbb ..."-command : it would take 12 hours to complete. That's okay?
- current layout :

- tank details : (sync = standard)

Could it be taht because sync = standard, that the log (=slog?) device da0 is not being accessed?

I remember that setting sync= always resulted in a huge performance hit.

Thomas.

 

[HoneyBadger]

Hi, thanks for the great advice! The last part is a bit confusing to me, and also I don't think I can use P4800X or P4801X devices (+1k€) because of budget.
So far I have :
- Server : SuperMicro SSG-620P-ACR12L
- RAM : 2x 64 GB ECC
- Processor : Intel xeon Silver 4310
- Storage : 12x 4TB WD Red
(With these 4 items I am already over 5k€ budget )
- L2ARC : Micron 5400 Pro 2,5" 480 GB
- SLOG : ?
- CPU cooler ?
- Boot : Silverstone SST-ECM22 + 2 M.2 NVME Drives (Would 2x WD Green SN350 240GB be okay?)

So the L2Arc probably goes in on of the 2 rear mounted 2.5" bays. It's not very clear for me what you want to put in the onboard M.2 bay, and what goes in the Optane DC Persistent Memory slots? The xeon 4310 takes a big bite out of the budget. Is it necessary?

Hi Thomas,

I'm not too familiar with the EU market - could you suggest a site or two where you're obtaining prices and comparing them? This will help out in further suggestions.

In general, the Xeon Silver 4310 should be the least expensive of the 3rd-generation Intel Scalable processors, but they do seem to have taken a price bump over the 2nd-generation, even within the family, and the entry-level models have been thinned out. There's no equivalent to the Silver 4208, for example.

Is obtaining an older 2nd-generation Scalable system like this model an option?

SSG-5029P-E1CTR12L | SuperStorage | Products | Super Micro Computer, Inc.

With the budget constraints we might need to make some different choices regarding the SLOG and boot device configuration as well. The Silverstone SST-ECM22 cards are capable of holding both an NVMe and a SATA M.2 on the same card - with two of them, that allows for a mirrored NVMe slog (using the Optane P4801X, or perhaps the P1600X if budget is a concern) and two SATA M.2 SSDs for the boot device. This leaves the rear bays open for two SSDs for L2ARC.

 

[HoneyBadger]

About our current setup :
- the "zdb -lbb ..."-command : it would take 12 hours to complete. That's okay?

Yes, it's expected to take a fair bit of time to check all the metadata. The flags are also case-sensitive - did you use -L or -l?

Could it be taht because sync = standard, that the log (=slog?) device da0 is not being accessed?

I remember that setting sync= always resulted in a huge performance hit.

sync=always does result in reduced performance, but this is required for the safety of remote writers that assume a non-volatile write cache (like VMware over iSCSI) - VMware's NFS client demands sync writes by default, but iSCSI needs to be forced. Have a look at the resource below for details:

Resource - Sync writes, or: Why is my ESXi NFS so slow, and why is iSCSI faster?

This post is not specific to ESXi, however, ESXi users typically experience more trouble with this topic due to the way VMware works. When an application on a client machine wishes to write something to storage, it issues a write request. On a...

www.truenas.com

sync=standard will effectively bypass the log device, speeding things up, but also losing the write safety that it provides.

 

[Thomas_VDB]

Hi, again thx for the tips.

Regarding the prcies: I am looking at US stores/prices, and assuming that I will get similar pricing.
However most components (except for the Supermicro-stuff), I can buy via a big retailer (Ingram) as we have a reseller account.
So I can't give you a link to that as you haven't got an account.
Thx for the tip about the 2nd generation Xeon. I will decide when I have a decision about the other components (SLog).
It is increasingly difficult to get hands on optane devices here. I am considering not using them because they are end-of-life and it will also be very hard to replace them in case of failure.
What do you recommend as an (affordable) alternative? My first choice would be 'not' using an slog and let the sync=standard, so that there is no performance issue. What can I do to reduce the dataloss risk. The server is already hooked up to a UPS. Is there anything else I can consider (to reduce data loss, or as an alternative to optane)?
Or would the P1600x - 120GB (that I can only find at Amazon), be the wisest choice?

What would you recommend as a CPU cooler?

 

[HoneyBadger]

What do you recommend as an (affordable) alternative? My first choice would be 'not' using an slog and let the sync=standard, so that there is no performance issue. What can I do to reduce the dataloss risk. The server is already hooked up to a UPS. Is there anything else I can consider (to reduce data loss, or as an alternative to optane)?

Unfortunately sync=standard inherently introduces that risk of data loss, as the writes from ESXi aren't guaranteed to be on stable storage. A UPS mitigates the failure condition of external power loss, but it doesn't prevent internal failures of components like the PSU, HBA, backplane, motherboard, etc - using an SLOG and sync writes add multiple-9s of safety.

Devices that perform well as SLOGs tend to be expensive on a per-GB basis because of the requirements for the fast sync writes and power-loss-prevention. The Optane P1600X 118GB is rated at 6 DWPD which is quite robust for a light-usage server and it's relatively low-cost when considering performance-per-dollar metrics. There are certainly competing devices, on SAS/SATA or NVMe, that can be had for less cost outlay - but they may not do as well on the performance-per-dollar metric. That's something I should consider compiling from the listing of benchmarks in the SLOG thread.

What would you recommend as a CPU cooler?

Because of potential socket clearance and mounting compatibility issues, it's likely best to look at the system/board in question and pick something from the compatibility list (on the Supermicro page under Resources/Chassis) or that's been verified by another user to work well. For the single-socket 2nd-gen 2U system, Supermicro suggests this active cooler.

Supermicro 2U Active CPU Heat Sink Socket LGA3647-0 (SNK-P0068APS4)

2U Active CPU Heat Sink Intel® Xeon® Scalable Processors Socket LGA3647-0 Narrow Mounting Mechanism Supermicro Certified

store.supermicro.com

 

[Thomas_VDB]

Hi,

I think I am finalizing the Supermicro build.
- Server : SuperMicro SSG-520P-ACR12L (slightly different model as I am using only 1 cpu socket and no front NVME slots)
- 128GB RAM
- CPU Xeon Silver 4310
- Boot : 2x 256GB in rear mounted Sata slots. This is because I am afraid that it won't boot of the PCIe to m.2 adapter.
- L2Arc : 1x WD Blue M.2 NVME SSD 512 GB, in the onboard M.2 slot. I don't see an advantage of 2 disks in mirror. Correct?
- Storage : 7x WD Gold 6TB (6 drives in mirror + hot spare)
- ZIL : 2x Optane 1600x 120GB, via 2x Startech PEX4M2E1 PCIe to M.2 adapter
- PCIe SFP+ adapter

This build costs just over 5k€. Could you comment on this build?

Do you have any advice regarding the sector size that I must choose?

thx!

 

[HoneyBadger]

I'm not finding a result on that exact server - I do see the SSG-520P-ACTR12L though which I believe only differs by having onboard 10GbE through the Intel X550.

For your PCIe SFP+ adapter, selecting one listed in the 10Gig Networking Primer resource is probably ideal:

10 Gig Networking Primer

This is a discussion of high speed networking for newcomers, with specific emphasis on practical small- to medium-scale deployments for home labs or small office users. It originated with a forum thread located here that has many pages of...

www.truenas.com

Chelsio is used by iX internally, but other vendors such as Intel are an option as well - ensure that you check the authenticity of any Intel cards using their Yottamark/date codes, as counterfeit hardware unfortunately has appeared in the supply chain from time to time.

The rest of the component selection looks good. Ensure that you pick RAM from the motherboard QVL (qualified vendor list) to guarantee compatibility, and note that the rear 2x2.5" kit on that server is an additional option and not included by default.

For sector size, TrueNAS supports both 512e and 4Kn sectors, and defaults to treating all devices as having a 4K minimum sector size, so there shouldn't be any functional differences in that choice.

 

[Thomas_VDB]

Hi, it is indeed the SSG-520P-ACTR12L, which also has only 1 CPU socket and less memory banks I believe.

Regarding the CPU, RAM, SFP+ adapter : I was planning to just use the ones that I can add in the SuperMicro estore configurator. It also automatically adds the rear kit if you add rear drives.
The SFP+ adapter is probably this one : Intel X710-BM2 2-port SFP+. I guess that's okay?

 

[HoneyBadger]

Hi, it is indeed the SSG-520P-ACTR12L, which also has only 1 CPU socket and less memory banks I believe.

Regarding the CPU, RAM, SFP+ adapter : I was planning to just use the ones that I can add in the SuperMicro estore configurator. It also automatically adds the rear kit if you add rear drives.
The SFP+ adapter is probably this one : Intel X710-BM2 2-port SFP+. I guess that's okay?

Using the Supermicro configurator should definitely give you compatible hardware, and the X710 is a fine card with an Intel-supplied driver, but the SFP module lock is in the firmware - so you'll have to ensure that you buy Intel-compatible optical modules as well.