ZFS Pool creation advice

Fred974

Contributor
Joined
Jul 2, 2016
Messages
190
I am building a TrueNAS system with 1x Dell PowerEdge R610 as host. Attached to it are
1x Dell PowerVault MD1200 (12x 3TB LFF HDD)
1x Dell PowerVault MD1220 (24x 1TB SFF HDD)

The storage are connected via daisy chain and the connected to TrueNAS host using a Dell PERC H200 IT mode HBA card.

I will be be creating:
1x pool with the 12 HDD with 4x SSD for metadata for my backup storage
1x pool with the 24 SSD for XCP-NG VM where high IOP is preferred

I have been reading a fair bit on ZFS and I come to the conclusion that for my xcp-NG use case I would need to go for either
8x VDEV with 3x SSD in mirror
or
4x VDEV with 4x SSD in raidz2

Could anyone please tell me if the sacrifice in storage using the mirrored solution is woth it over the radz2 solution? Will the performence be significentl higher or just a little bit?

As for as the backup pool is concern, would you recommend that I use:
2x VDEV with 6x HHD
or
3x VDEV with 4x HDD

Thank you all in advance
 

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,504
 

Fred974

Contributor
Joined
Jul 2, 2016
Messages
190
@danb35 thank you for the link you posted but I have already read that page. I guess my question is more about in real life experience, how much of a difference this is configuration are
8x VDEV with 3x SSD in mirror
or
4x VDEV with 4x SSD in raidz2
 

ChrisRJ

Wizard
Joined
Oct 23, 2020
Messages
1,919
I have been reading a fair bit on ZFS and I come to the conclusion that for my xcp-NG use case I would need to go for either
8x VDEV with 3x SSD in mirror
or
4x VDEV with 4x SSD in raidz2

Could anyone please tell me if the sacrifice in storage using the mirrored solution is woth it over the radz2 solution? Will the performence be significentl higher or just a little bit?
Two things:
  1. The two SSD pool alternatives do not match in terms of the number of drives used. Option 1 uses 24 drives while option 2 uses only 16. Is that intentional? If yes, why?
  2. The question about which configuration is faster, albeit a very common approach to performance, is wrong. The question should be "what do I need to fulfill my performance requirements?"
I realize that many people don't exactly know their current or future performance requirements. So in a way my second argument is sort-of academic. Still, I would advise to try approaching things from the "what do I need" perspective. Can you elaborate on 1) your hardware in quite a bit more detail, and 2) explain what VMs you are planning to run?
 

Fred974

Contributor
Joined
Jul 2, 2016
Messages
190
@ChrisRJ the harsward is as follow:

TrueNAS computing server:
Dell PowerEdge R610 with 2x Intel(R) Xeon(R) CPU X5650 @ 2.67GHz and 72GB DDR3 RAM PERC H200 SAS IT mode 2x 10Gbase-Twinax NIC
2x 146GB SAS 15k HDD for boot-pool
4x 350GB SSD for metatdata

JBOB Storage:
1x Dell PowerVault MD1200 (12x 3TB LFF HDD)
1x Dell PowerVault MD1220 (24x 1TB SFF HDD)

The 2x JBOD are connected via daisy chain and the connected to TrueNAS host via the SAS card.

The requirement is to have storage to host xcp-NG VMs and have storage for backup. The VMs will be over iSCSI and NFS connection, the backup will be over NFS. We will not be using the TrueNAS server for any jails or plugin.

We want the maximum performance for the VMs and good ratio storage/resilance for the backup.
The VMs will host anything from websser LAMP stack to Collabora server, Nextcloud, Email server, Unifi controller etc.
The backup storgae will be the storage for our Asigra dedicated server as well as the storage for our MinIO server

Going forward we are also looking at reselling VPS but that will using or Dell EqualLogic storage as we are aware hat our gear is a few years old.

We are planning to buy 256GB or RAM to put in the R610 or the ARC cache.
The backup pool will use the 4x SSD for metadata.

I hope this is enough information.

Please ask if you need more details
 

ChrisRJ

Wizard
Joined
Oct 23, 2020
Messages
1,919
Well, the DELL R610 is a really old system and pretty bad in terms of power efficiency. For a commercial offering I am not sure it is something I would choose. But that depends on the details.

I am not sure why you want to use SSDs for metadata on the backup pool.
 

Fred974

Contributor
Joined
Jul 2, 2016
Messages
190
I am not sure why you want to use SSDs for metadata on the backup pool.
Because the backup pool are spinning disks. My understanding is that adding a metadata to a pool of spinning disk provide better performence. Am I wrong?
 

ChrisRJ

Wizard
Joined
Oct 23, 2020
Messages
1,919
You are not wrong. My line of thinking was, however, that for VM backup this may not be needed. So that the SSDs could be used somewhere else with bigger value.
 

Fred974

Contributor
Joined
Jul 2, 2016
Messages
190
@ChrisRJ the 4x SSD are 400GB Pliant LB400M from my previous Dell EqualLogic. What would you use them for? I am not sure if they have power lost technology, so not sure if they can be used for LARC or SLOG. I know the Dell R610 is old but we don't have huge budget so we will be replacing it at some point. Unless you can advice on a budget friendly 2nd hand server I could get of the the famous auction site?

Thank you
 
Last edited:

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,112
It's moreso that adding SSDs may not benefit your workflow enough to be worth the additional risk of having your metadata rest on them. Typically a meta SSD is useful for when you're handling a large number of files that need to be listed or investigated frequently, in more of a "random pattern" - whereas a backup job is mostly a single stream of writes (or reads, if a recovery is in progress)

The Pliant LB400M (aka Sandisk Lightning) disks don't use a volatile write cache either (they're a direct-to-NAND write technology) which means they "have PLP" in the sense of an SLOG - this isn't required for an L2ARC though. They likely will not be fast enough for SLOG duties against an all-SSD pool though.

TrueNAS computing server:
Dell PowerEdge R610 with 2x Intel(R) Xeon(R) CPU X5650 @ 2.67GHz and 72GB DDR3 RAM PERC H200 SAS IT mode 2x 10Gbase-Twinax NIC
2x 146GB SAS 15k HDD for boot-pool
4x 350GB SSD for metadata

JBOD Storage:
1x Dell PowerVault MD1200 (12x 3TB LFF HDD)
1x Dell PowerVault MD1220 (24x 1TB SFF HDD)

The structure of "head unit with expansion shelves" works well, but my concern is that a 1U system like the R610 may not have enough physical room for PCIe cards to handle things correctly. The R610 only has two PCIe x8 slots and a "storage slot" that is PCIe x4 and has a device whitelist to only accept "integrated" Dell storage cards.

You'll need to use one of those PCIe x8 slots for your external HBA to connect to the MD12xx shelves - the other, I presume, has the 10Gbps NIC in it, as I don't think the R610 has any embedded 10GbE options. The "storage slot" in the Dell 11G servers can be fooled with an SBR hack (see the video from ArtOfServer tagged as footnote [1] or the Reddit post as footnote [2] for details) but that still leaves you without another slot to use for an NVMe SLOG device - so you'd be limited to SAS/SATA devices in either the "head unit" or the MD1220 shelf. See footnote [3] or my signature for the SLOG benchmark thread - the most popular SLOG devices are almost exclusively NVMe (Optane DC P-series, Intel DC P-series) which would consume a PCIe slot on their own. The other option is to use a "Write-Intensive" SAS drive like the WD DC SS530 (WUSTM*) or Toshiba PX04SHB*/PX05SM* - you'll likely be spending a fair amount, but you would get things like easy hot-swap capability.

We are planning to buy 256GB or RAM to put in the R610 or the ARC cache.

The R610 only officially supports up to 192GB.

Pool vdev layout:

I'd do 2x 6-wide Z2 for the backup pool. No L2ARC or SLOG here.
I'd do 12x (or fewer, if you need to fit SLOG) 2-wide mirrors for the main pool, with regular backups to the main pool being the reason I'd be comfortable with this level of redundancy.

Potential alternate hardware:

Moving up even one generation to the R620 will give you access to integrated 10GbE options on the motherboard, better power efficiency in the processors, more RAM (up to 768GB!), more front bay slots (10x 2.5" vs 6x 2.5") and options for three PCIe slots (2 x16, 1 x8) if desired. Going to a 2U solution like the R720 gives you even more slots and bays if you find it necessary, but the 12G 3-slot with integrated (Intel) 10GbE will likely do the trick.

Questions:

1. What model of SSDs are you planning to use for the all-SSD pool?
2. Can you detail the layout of your PCIe cards?
3. Can you afford a downtime if you need to swap a PCIe SLOG device?

Footnotes:
[1] https://www.youtube.com/watch?v=v0AEHVdc_go
[2] https://www.reddit.com/r/homelab/comments/cyre3o/how_to_move_an_h200_into_the_dedicated_slot_h200a/
[3] https://www.truenas.com/community/threads/slog-benchmarking-and-finding-the-best-slog.63521/
 

Fred974

Contributor
Joined
Jul 2, 2016
Messages
190
@HoneyBadger thank you very much for taking the time to write such a details post. It is very appreciated :)

1. What model of SSDs are you planning to use for the all-SSD pool?

Dell R610 (6x disks)
2x Dell 146GB SAS 15k HDDl
4x Pliant LB400M 400GB SSD

Dell PowerVault MD1200 (12x disks)
12x Dell 3TB SAS 7.2k 3.5" 6G Hard Drive 91K8T

1x Dell PowerVault MD1220 (24x disks)
24x HPE Intel DC S3520 Series 1.6TB SSD 2.5"

2. Can you detail the layout of your PCIe cards?
I have 1x Chelsio 10G NIC card connected to 1 PCI Slot and the second PCI slot has the Dell PERC H200 SAS card in IT mode (screenshot bellow)
1666606648611.png


3. Can you afford a downtime if you need to swap a PCIe SLOG device?

Not really keen on having the server down for swapping SLOG

If the 4x Pliant LB400M are not good as SLOG due to the speed, will I not be better using then as mirror SLOG on the Spinning disk or metadata? Our main storage is the Dell EqualLogic at this moment so we will be replacing the r610 as sugested but not before Q1 of next year.

Thank you
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,112
Dell R610 (6x disks)
2x Dell 146GB SAS 15k HDDl
4x Pliant LB400M 400GB SSD

You certainly don't need 15K HDDs for a boot pool, but it won't hurt anything other than power consumption.

The Pliant/Sandisk SSDs may end up getting the boot for faster SLOG devices; I'll touch on that later.

Dell PowerVault MD1200 (12x disks)
12x Dell 3TB SAS 7.2k 3.5" 6G Hard Drive 91K8T

Magic decoder ring says "Seagate Constellation ES.2" - the 3TB model was known to be less-than-reliable historically, I'm afraid. Not quite as horrific as the consumer ST3000DM001 (to my knowledge, there was a class-action launched over that drive) but it still wouldn't be something I'd build a backup pool out of. If you don't have any alternative models you can use, make sure to keep several offline spares, set up alert notifications over email, and respond to those alerts promptly. Scheduled scrubs and RAIDZ2 will help mitigate this, but still - tread lightly here.

1x Dell PowerVault MD1220 (24x disks)
24x HPE Intel DC S3520 Series 1.6TB SSD 2.5"

Good choice - the S3520 while still being a "read focused" drive picked up a lot of endurance over the earlier S3500, your drives being rated for just under 3PBW lifetime vs. the 880TBW lifetime of their earlier cousins. The HPE models might hide their total writes in SMART data though - check to see if you can monitor this value.

I have 1x Chelsio 10G NIC card connected to 1 PCI Slot and the second PCI slot has the Dell PERC H200 SAS card in IT mode (screenshot bellow)

How are the internal bays connected? I assume you've still got a card in the "Storage" slot (where the PERC 6i is in the picture) - while a hardware RAID card will be acceptable for a boot device (and possibly even L2ARC) I wouldn't use it for anything like SLOG/meta where data integrity is paramount. You might need to look into the "Moving an H200 into the internal storage slot" process I described in my last post.

Not really keen on having the server down for swapping SLOG

Unfortunately a moot point here as you don't have the PCIe slots for an NVMe SLOG; but is worth noting for the potential R620 upgrade in the future.

If the 4x Pliant LB400M are not good as SLOG due to the speed, will I not be better using then as mirror SLOG on the Spinning disk or metadata? Our main storage is the Dell EqualLogic at this moment so we will be replacing the r610 as sugested but not before Q1 of next year.

If your spinning disks are only handling an asynchronous sequential write workload (such as "store these backups, and hopefully we don't ever need them") then they won't make use of an SLOG at all. Rapid metadata writes are similarly not as crucial in a backup workload - and if you really needed rapid metadata reads, those can be more safely handled by a metadata-only L2ARC device.

With your pool devices having power-loss-protection and fast-ish sync writes, it's theoretically possible although not necessarily recommended to run sync writes without an SLOG here. I would still recommend that you use a pair of write-intensive SAS SSDs like the WD DC SS530 "WUSTM" model I mentioned above, as the goal is to push write latency as low as possible.

The VMs will be over iSCSI and NFS connection

Will you be choosing "one or the other" or do you have to run both? One of these is a NAS protocol, one is a SAN protocol, and they may have different methods of bundling connections for redundancy and throughput (NFS is okay with LACP - iSCSI isn't)

They also might not take too kindly to sharing the same physical interface in terms of congestion avoidance. If you have to run both at once, I'd suggest tuning the hosts to behave in an active/passive manner, where all of the NFS traffic goes on interface cxgbe0 by default, reverting to cxgbe1 on failover - and iSCSI having the inverse behaviour.
 

Fred974

Contributor
Joined
Jul 2, 2016
Messages
190
@HoneyBadger thank you again for the details post. What would you replace the PERC 6i storage card with? I don't think i can connect the H200 to the "storage slot".

For the protocol I am thinking of only using NFS at this point as it allow for thin provisioning.
The R610 only officially supports up to 192GB.
is that not 192GB per CPU? If that's the case do you thing 16x 16GB RAM will be ok?
 

Fred974

Contributor
Joined
Jul 2, 2016
Messages
190
In regard to the Pliant LB400M, would you just use then as boot devise or not use them at all?
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,112
@HoneyBadger thank you again for the details post. What would you replace the PERC 6i storage card with? I don't think i can connect the H200 to the "storage slot".

I would use a second H200 and convert it to an H200i using the SBR modifications. If you have a PERC 6/i though, you will need to get replacement cables as they're using the older SFF-8484 types on the "card side" and possibly "backplane side" as well.

For the protocol I am thinking of only using NFS at this point as it allow for thin provisioning.

iSCSI ZVOLs can be thin ("sparse") provisioned, but NFS is fine of course if it is what you are familiar with and designing for - you'll also be using it for the backup traffic, so only having to coordinate one protocol is probably ideal.

is that not 192GB per CPU? If that's the case do you thing 16x 16GB RAM will be ok?

No, it's 192GB for the entire system board. The R610 also only has 12 physical DIMM slots, so 16x-anything definitely won't fit - it will give you some spares though in case of problems or failed DIMMs though.

The only results indicating that these systems support 32GB DIMMs are all from websites offering RAM for sale, so I'm disinclined to trust them in favour of the official spec sheet [1] that says 12x16GB is the maximum - and supporting 16GB DIMMs requires an updated BIOS, so make sure you do that part first.

In regard to the Pliant LB400M, would you just use then as boot devise or not use them at all?

They'll certainly be an excellent (if overly large) boot device - you could use two for boot, two more for L2ARC duties against the spinning-disk pool, and have two slots available for faster SAS SLOG devices.

[1] https://i.dell.com/sites/csdocuments/Shared-Content_data-Sheets_Documents/en/R610-SpecSheet.pdf
 
Top