TrueNAS and fusion pool

Fred974

Contributor
Joined
Jul 2, 2016
Messages
190
Hi,

I have building and TrueNAS server with a Super Micro 1U CSE-510T-200B Chassis for my Homelab setup. The server will have 2 ssd for the TrueNAS OS and I will connect a Dell MD1200 JBOD via SAS H200E.

The MD1200 has a 12 bays support. I will have 6x 3TB spinning HDD and 6x SAS SSD. Could anyone please advise on how the fusion pool would be setup in my case?

The plan is to have 2x pools in raidz2 1x ssd pool for fast storage like XCP-NG etc and 1x spinning for slower stuff like files.

The case can host 4x more ssd if needed.

Could anyone please help me understand how to create the pool in ‘hybrid’ mode? Do I create 2 pool with 2 metadata vdev made of 2 ssd each?
Using a Mirror layout is possible, but it is strongly recommended to keep the layout identical to the other vdevs.

Does this mean I need another 12 HDD for the metadata vdev?

Thank you
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
Could anyone please help me understand how to create the pool in ‘hybrid’ mode? Do I create 2 pool with 2 metadata vdev made of 2 ssd each?
Does this mean I need another 12 HDD for the metadata vdev?
What's important is to consider the fault tolerance level you have selected for the pool...

If your data VDEV(s) are structured to tolerate loss of 2 disks, then the metadata VDEV should probably allow the same (in this case, that means a 3-way mirror is the best option).

If your metadata VDEV is lost, your pool goes with it, so I hope that makes sense to match the VDEVs in terms of redundancy level.

Your pool only needs one metadata VDEV (but you need one per pool if you want the pool to benefit from fast metadata). There was some talk originally about the desire to have a small number of SSDs divided into partitions in order to be used to cover more than one pool, but that's another level of (unsupported) messing around at the CLI, so decide if you want to go down that road.
 

Fred974

Contributor
Joined
Jul 2, 2016
Messages
190
Hi @sretalla
If I create 2 pool with 6 HDD in Raidz2, am I correct to say that the tolerate loss of 2 disks ? So if I follow your logic, I will need 3 drive for the metadata. Is my understanding correct?

Also Is fusion pool worth it? Will I be better creating 2x classic raidz2 pool and call it a day?
 

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,828
There is a big benefit to having a sVDEV if you have to deal with lots of small files and/or want the metadata to be always super quick (think browsing, rsync, and like tasks).

That said, I got good rsync performance out of a metadata-only L2ARC and as of 12.x, said L2ARC can be made persistent between reboots. The beauty of L2ARC is that it is not critical to system performance and hence can be detached, fail, etc. with nary a consequence (L2ARC stores a copy of the underlying data, not the data itself). That stands in stark contrast to the sVDEV which needs n-way redundancy, quality SSDs that can handle a lot of writes, etc. since the pool will go kaput if the sVDEV fails. Hence, my Z3 pool features a 3-way SSD sVDEV mirror, with a cold but qualified spare SSD nearby.

However, for your application, two pools might make more sense since you have a complete set of SSDs and so on. I'd consider a persistent, metadata-only L2ARC for the HDD pool to make browsing faster. My metadata-only L2ARC sizing was 1TB for a 40-TB pool. Any SSD will do for L2ARC duty, though faster ones will help somewhat. L2ARC consumes some RAM, the recommendation generally is to have at least 32GB of RAM, with 64+GB being preferable.
 
Last edited:

Fred974

Contributor
Joined
Jul 2, 2016
Messages
190
@Constantin Thank you for your advise, I thinl L2ARC is a better option for as you said, I don't have to worry loosing disk then.
Will L2ARC cache both pool?
 

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,828
L2ARc is applied on a per pool basis, IIRC. I’d only use it for the HDD-based pool. No benefit for the SSD pool.
 

Fred974

Contributor
Joined
Jul 2, 2016
Messages
190
L2ARc is applied on a per pool basis, IIRC. I’d only use it for the HDD-based pool. No benefit for the SSD pool.
Thank you very much for all your advice. I'm learning quite a bit here... I will find an old ssd to use on the HDD pool. However the motherboard supports max of 32gb and has currently got 12gb installed. Can I get away with it? Do I have to get the max 32gb ram?

Thank you
 

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,828
Hmmm. The 3-way mirror of SSDs in a SVDEV will only show up as a single drive. So you will lose capacity vs. having two z2 pools consisting of 6 drives each.

If you want a fast scratch drive and a slower, higher capacity drive then a two pool solution makes more sense to me. Especially with just 12GB of RAM.
 

Fred974

Contributor
Joined
Jul 2, 2016
Messages
190
I was meaning adding 3 ssd in the server case. I will still keep the 6x ssd for the pool.
Does the size of the drive matter for the sVDEV SSD drive?
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
If I create 2 pool with 6 HDD in Raidz2, am I correct to say that the tolerate loss of 2 disks ? So if I follow your logic, I will need 3 drive for the metadata. Is my understanding correct?
Yes

Does the size of the drive matter for the sVDEV SSD drive?
You probably don't need a lot... think something like 10-20% of your total pool capacity as a maximum.
 

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,828
I’m running something even smaller - 1.6TB for the sVDEV in a 50TB pool, but I limited small files to less than 32kb and pretty consistently eliminated all small files by putting my old Mac stuff into disk images which present themselves as large “band” files to ZFS.

in my installation, the max. disk capacity needed for metadata projects to be about 500GB, assuming the pool were filled to its max. suggested capacity of 40TB, the added data has the same format / size as the extant data, etc. You can set the small file limits inside the GUI.
 

PackElend

Explorer
Joined
Sep 23, 2020
Messages
60
Hi there,
my thoughts go in the same direction, so I may add my question here.

I'm building my very first TN (scale) system using Overview - UDOO BOLT GEAR Docs, which is built around the AMD Ryzen™ Embedded V1605B.
It will be replaced by a more adequate hardware platform when I will have made my experience with TN.
Currently, I have an
  1. 1 TB 970 EVO Plus NVMe M.2 SSD | MZ-V7S1T0BW | Samsung SG
  2. 32 GB M.2 SSD 400S | SATA III M.2 SSDs - Transcend Information, Inc. (transcend-info.com)
  3. 32 GB DDR4 2400
  4. 3x 4 TB NAS HDDs
I addition, I do have some USB 3.1 ports spare, which could be used to create a Fusion Pool and/or add L2ARC/ZIL SLOG.

There will be two
  1. vdev, RAIDZ: The HDD will hold photos, music, data etc. connected via USB 3.1 (what generation exactly is currently inquired)
  2. vdev, Stripe: The NVMe is used to run VMs, Containers (Kubernetes, Docker), DBs and so on.
  3. boot: SATA III M2.SSD
I would say, there is not that much to be improved at the NVMe, if it fails, I can restore it using backup configurations

Now I'm wondering if I can improve HDD pool performance by adding a USB 3.1 Gen2 SSD creating a Fusion Pool that would store the most used data (not sure if TN does it automatically or I can only control the data stored on the SSD by the file size filter).

I don't know if I could improve the pool performance by adding L2ARC/SLOG considering 32 GB RAM (accepting data losses in case of power loss)
 

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,828
this config is somewhat confusingly described (at least to me). I would not recommend attaching anything other than the boot pool via USB.

Is this a 2 pool layout with one pool in a raidz, the other a SSD stripe for fast I/o?
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
I don't know if I could improve the pool performance by adding L2ARC/SLOG considering 32 GB RAM (accepting data losses in case of power loss)
No.

If you are accepting data loss with power loss SLOG is totally the opposite of what you want, just set sync=disabled and you'll get the maximum speed possible without needing SLOG (which is only about data protection during power loss).

I don't anything else left besides a single sata port (details will come up with the update tomorrow)
If you use USB for a permanent data pool member or L2ARC, expect carnage at some point. USB controllers don't handle ZFS transaction groups well and I have seen many fail when I have seen the combination of USB and ZFS in heavily used pool disks. (remember resilvers and scrubs are extreme load for disks and the controllers attaching them).

You would absolutely never do that for SLOG.
 

PackElend

Explorer
Joined
Sep 23, 2020
Messages
60
@sretalla thx for the feedback
expect carnage at some point.
it is only for familiarization testing.
My current data is back up on an external hard disk and only a copy will be created on the ZFS pool. Data that could be used daily (bills, administration etc.) is backed up to cloud storage live.
Containers & VMs will run on NVMe.

Is this a 2 pool layout with one pool in a raidz, the other an SSD stripe for fast I/o?
yes, in nutshell :)


I have just got preliminary feedback in regard to USB speed, so the Udoo Bolt has these interfaces:
  1. Mass Storage:
    1. 1x SATA III 6Gbit/s standard connector
      MAYBE AS A BACKUP DRIVE
    2. 1x SSD SATA module's Slot M.2 Socket 2 Key B 2260 (featured also PCI-e x2)
      BOOT
    3. 1x NVMe modules Slot M.2 Socket 3 Key M 2280 (PCI-e x4 Gen 3 interface)
      SINGEL DRIVE STRIPE: CONTAINER & VMs
  2. Networking:
    1. Gigabit Ethernet Realtek RTL8111G
      USED
    2. Slot M.2 Socket 1 Key E 2230 for optional WiFi/BT combo
      NOT USED
  3. USB
    1. 4x USB 3.2 Gen 2×1 (Superspeed+ 10 Gbit/s) directly connected to the AMD Ryzen™ Embedded V1605B.
      1. 1x RAIDZ HDD POOL CONNECTED VIA USB-C HUB | UDOO Shop USING 3x HDDs
      2. 1x ...?
      3. 1x ...?
      4. 1x ...?
(better?)
 
Top