Help with setting up a TrustNAS pool with SLOG, L2ARC and Metadata

[hysel]

Hi All

I am running the following setup:

• AMD 5900X CPU
• x570 Motherboard
• 128GB RAM
• SAS 9300-8i 8-port
• ASUS Hyper M.2 X16 PCIe 3.0 X4 Expansion Card V2
• Storage:
  • 6 x Seagate IronWolf 10TB NAS Drives
  • 2 x Optane Memory M10 64Gb M.2 80 installed on the Asus Card
  • 2 x SSDSC2BB120G401 Solid-State Drive DC S3500 Series 120GB.
  • 2 x SABRENT 256GB Rocket NVMe PCIe M.2
  • The Seagate and Intel SSD drives are connected to the SAS Card using SFF-8643 Internal Mini SAS HD to (4) 29pin SFF-8482 connectors with SAS 15pin Power Port 12GB/S Cables.
  • The Optane drives are connected to the Asus PCIe card (PCIe x3 speed)
  • The NVMe drives are connected to the Motherboard (PCIe x4 speed)

I am running ESXi 8 with TrueNAS running as VM with the hardware being passthrough.

Everything is working fine, but I recently purchased the Optane and Intel SSD, and I am looking for a recommendation on how to utilize my setup better.

My usages are mainly around streaming (Plex) and VMware (not heavy use).

Here are my thoughts about how to set this up. Please let me know if I am on the right path:

• Set the 6 Seagate in a RaidZ-1 setup
• Use the Optane drives as Metadata special drives (Mirrored)
• Use the Intel SSD as SLOG (Mirrored)
• Use the NVMe as L2ARC (Stripped)

Is this sound ok for my requirements?

Also, what else should I look for (block size), etc.?

I appreciate any help you can provide.

Itamar

 

[jgreco]

Use the Intel SSD for Caching (Mirrored)

What is "caching"? If I play process of elimination with the possibilities, I suspect you mean SLOG, but SLOG is in no way, shape, or form any kind of cache.

Some insights into SLOG/ZIL with ZFS on FreeNAS

What is the ZIL? POSIX provides a facility for the system or an application to make sure that data requested to be written is actually committed to stable storage: a synchronous write request. Upon completion of a sync write request, the underlying filesystem is supposed to guarantee that a...

www.truenas.com

 

[hysel]

What is "caching"? If I play process of elimination with the possibilities, I suspect you mean SLOG, but SLOG is in no way, shape, or form any kind of cache.

Some insights into SLOG/ZIL with ZFS on FreeNAS

What is the ZIL? POSIX provides a facility for the system or an application to make sure that data requested to be written is actually committed to stable storage: a synchronous write request. Upon completion of a sync write request, the underlying filesystem is supposed to guarantee that a...

www.truenas.com

Thank you. I totally messed that up. I updated my post.

 

[jgreco]

Thank you. I totally messed that up. I updated my post.

It brings up the next question, which is why do you think you need SLOG? I mean, it certainly won't kill you, but the S3500 isn't particularly fast, and mirroring it isn't something that's required these days. Compared to just using async writes, SLOG always slows you down. If you have something where you want or need sync writes, then that's fine, of course.

 

[Arwen]

• Use the NVMe as L2ARC (Mirrored)

You can't mirror L2ARC. Stripe yes, (if you have enough RAM for additional L2ARC Cache devices to make sense).

Any failure of an L2ARC device, simply causes ZFS to read from the source pool. Slower yes, but seemless access.

From the zpoolconcepts manual page:

Cache devices cannot be mirrored or part of a raidz configuration. If a read error is encountered on a
cache device, that read I/O is reissued to the original storage pool device, which might be part of a mir-
rored or raidz configuration.

You should probably read up on both ZFS terminology & concepts.

Terminology and Abbreviations Primer

We realize that new users have a lot to learn when they come to FreeNAS. There's a certain amount of confusion added to discussions when users pick random/approximate terms to describe things. I've spent a lot of time quietly trying to...

www.truenas.com

Introduction to ZFS

This is a short introduction to ZFS. It is really only intended to convey the bare minimum knowledge needed to start diving into ZFS and is in no way meant to cut Michael W. Lucas' and Allan Jude's book income. It is a bit of a spiritual...

www.truenas.com

I don't have a link to the ZFS concepts, can't find it easily. Will edit post if I find it.

 

[hysel]

It brings up the next question, which is why do you think you need SLOG? I mean, it certainly won't kill you, but the S3500 isn't particularly fast, and mirroring it isn't something that's required these days. Compared to just using async writes, SLOG always slows you down. If you have something where you want or need sync writes, then that's fine, of course.

Interesting point.

What I want to do is to maximize both read and write performance, especially for those times I do run some heavy duty VMs or streaming an high resolution movie.

When using async writes, is large memory allocation to the server plays much higher factor or can I use something else?

Itamar

 

[hysel]

You can't mirror L2ARC. Stripe yes, (if you have enough RAM for additional L2ARC Cache devices to make sense).

Any failure of an L2ARC device, simply causes ZFS to read from the source pool. Slower yes, but seemless access.

From the zpoolconcepts manual page:


You should probably read up on both ZFS terminology & concepts.

Terminology and Abbreviations Primer

We realize that new users have a lot to learn when they come to FreeNAS. There's a certain amount of confusion added to discussions when users pick random/approximate terms to describe things. I've spent a lot of time quietly trying to...

www.truenas.com

Introduction to ZFS

This is a short introduction to ZFS. It is really only intended to convey the bare minimum knowledge needed to start diving into ZFS and is in no way meant to cut Michael W. Lucas' and Allan Jude's book income. It is a bit of a spiritual...

www.truenas.com

I don't have a link to the ZFS concepts, can't find it easily. Will edit post if I find it.

Thank you, I updated my post

 

[jgreco]

When using async writes, is large memory allocation to the server plays much higher factor or can I use something else?

Not clear what you're asking. The ZFS write cache is always in main memory. Sync writes act as a speed brake, slowing down the system (possibly dramatically), which is a performance issue a SLOG device is meant to address. The transaction group process (the write cache) uses as much memory as it needs to, though there are some practical limits. This is typically enough memory to cache two transaction groups in main memory.

 

[hysel]

Not clear what you're asking. The ZFS write cache is always in main memory. Sync writes act as a speed brake, slowing down the system (possibly dramatically), which is a performance issue a SLOG device is meant to address. The transaction group process (the write cache) uses as much memory as it needs to, though there are some practical limits. This is typically enough memory to cache two transaction groups in main memory.

Thank you. I think I have a better understanding of what I need to do.

Appreciate the help

Itamar

 

[NugentS]

Just to add the S3500 is a read-intensive device designed for static data that is often read, but rarely written to. A SLOG is (in a steady state) always written to and never read. The S3500 is also (for an enterprise SSD) 1 DWPD which is low endurance in comparison to the S3600 (5 DWPD) and S3700 (10 DWPD). Thus the S3500 is not a good SLOG from that PoV.

• Set the 6 Seagate in a RaidZ-1 setup
  • - Low IOPS - High Capacity - not good for VM's, but good for bulk storage. IOPS of a single drive
• Use the Optane drives as Metadata special drives (Mirrored)
  • - These are M.10 and not terribly fast. Yes they are Optane and have good endurance but .......
• Use the Intel SSD as SLOG (Mirrored)
  • - Nope. Its does have PLP, which is the only plus point in this scenario
• Use the NVMe as L2ARC (Stripped)
  • - Probably unnessesary as you have 128GB of RAM anyway.

In my opinion the drive (beyond the data vdev) you have got right(ish) is the NVMe as L2ARC, and its almost certainly not required. It WILL make a difference in storing metadata and thus making browsing etc feel snappier but I suspect thats about it. I would be tempted to use the Optanes, striped (or one on its own) as L2ARC (metadata only). Ditch the special and the SLOG.

You have Special (performing metadata storage) and L2ARC (performing metadata storage). Pick one. Not both. The Optanes aren't big enough to do much in the way of small files and are pool critical where L2ARC isn't

 

[hysel]

Awesome, that is exactly what I was looking for

Thank you very much