Make Pool Status screen show labels/partitions instead of disk geom

[InQuize]

Is there a way to make GUI screen 'Pool Status' to show meaningful info about pool members and not just a disk that it resides on?
As it is now, this screen does not necessarily correspond with 'config' part of `zpool status` output.

Running TrueNAS-13.0-U3.1
Yesterday I was experimenting with GPT labels and in the end had this pool:

Code:

root@nas:~ # zpool status poly
  pool: poly
 state: ONLINE
  scan: scrub repaired 0B in 00:00:01 with 0 errors on Thu Feb  9 04:09:48 2023
config:

        NAME                   STATE     READ WRITE CKSUM
        poly                   ONLINE       0     0     0
          mirror-0             ONLINE       0     0     0
            gpt/poly-3XG29K2T  ONLINE       0     0     0
            gpt/poly-2PGX5G8T  ONLINE       0     0     0

errors: No known data errors
root@nas:~ #

And webUI was showing same info.
The following screenshot is only an example, because I had no reason to save it then, so I edited page code to match what I was seeing, the only difference might be prepending '/dev/gpt/' part of the path, not sure:

But today after a reboot, it I for some reason reverted to useless usual:

Also noticed that partitions are displaying in some cases as well, it is demonstrated on current Docs page:

Show

which is a lot more informative that what I get on my instance.

Last days were spent in preparations to import multiple pools spanning same disks with custom partitioning scheme.
My goal was to have GPT labels with `poolname-Serial` naming schema on each partition to make CLI maintenance easier since webUI Disk screens are for some reason not partition aware despite the fact that middleware itself uses only partitions and not whole disks. Moreover, I can't point pool device replace dialog to partition instead of empty disk..

I get the appliance part, but the whole disk management section is contradicting itself in logic and is so limiting that it is not even funny. I mean, it is the core part of storage management solution and should be comprehensive and not beating users into submission for a single theoretical "best practice" storage architecture of the multitude possible ways to utilize ZFS. The amount of effort I am making at this point to not sound negative about my battles with the whole restrictive part of TrueNAS ideology is ridiculous.

 

[Samuel Tai]

I think you would have a smoother experience if you didn't fight the default partitioning scheme. It's a default for a reason, and trying to shoehorn your preferred partitioning into the middleware is an exercise in futility, as you've experienced.

You can always create a handy table of GPTID to device partition by running glabel status. To get an idea of what TrueNAS is doing under the covers, you can run the API call midclt call disk.query | jq. Each disk uses a combination of its serial number and world-wide name (LUN ID) as its unique identifier in the TrueNAS configuration database. Each entry per disk ID also lists the current device name and ZFS GUID. You can match these to pool VDEV members via zpool status -g.

 

[InQuize]

I think you would have a smoother experience if you didn't fight the default partitioning scheme.

Actually, I did not. I took webUI created partitioning and implemented it precisely for my use case to fit additional zfs partition for additional pool.

These two are disks for tests:

Code:

root@nas:~ # gpart show -l da3 da4
=>         40  31251759024  da3  GPT  (15T)
           40           88       - free -  (44K)
          128      4194304    1  swap_3XG29K2T  (2.0G)
      4194432   7809842696    2  poly-3XG29K2T  (3.6T)
   7814037128   7809842696    3  test-enc-3XG29K2T  (3.6T)
  15623879824   7809842696    4  node0-3XG29K2T  (3.6T)
  23433722520   7809842696    5  node1-3XG29K2T  (3.6T)
  31243565216      8193848       - free -  (3.9G)

=>         40  31251759024  da4  GPT  (15T)
           40           88       - free -  (44K)
          128      4194304    1  swap_2PGX5G8T  (2.0G)
      4194432   7809842696    2  poly-2PGX5G8T  (3.6T)
   7814037128   7809842696    3  node4-2PGX5G8T  (3.6T)
  15623879824   7809842696    4  node2-2PGX5G8T  (3.6T)
  23433722520   7809842696    5  node3-2PGX5G8T  (3.6T)
  31243565216      8193848       - free -  (3.9G)

As you may see, the rules derived of having 2GB swap starting at sector 128 (at least in my case with 512b disks) and 4k aligned ZFS partitions (that are even matched to real disk sizes, 4TB disks in this case) are met.

In the end, planned there is only one zpool across multiple drives and single-partition pools on some of them larger than base ones in size. So not much to conflict. Data is primarily large file archival tiered in redundancy, nothing that would fight for IO. Makes perfect sense for me and gives flexibility without violating actual best practices for ZFS.

trying to shoehorn your preferred partitioning into the middleware is an exercise in futility, as you've experienced.

My testing shows quite the contrary. It all functions very well, haven't found anything not working.
And I am comfortable with shell, aware and ready to manage drive replacements in CLI.

Even though the only thing that needs tweeking for it to work in GUI is to modify drive replacement dialog to show partitions on the list for the same `zpool replace` command that is being used for gptid equivalent of `da#p2` in case of every disk now. Yes, sure there are couple more tests&glue under the hood, but replacement is really the only missing piece here.

While this is not happening, I would love the thing to at least assist me with info that is "already there" by design in webUI and not resist by masking real devices with whole disk geom in this case.
Sure, I can get the same info in CLI, but this does not change the fact that UI screen is still there, and useless, showing same pool devices for different pools due to some assumption that user is better off with over-simplified abstractions:

UPD:

To get an idea of what TrueNAS is doing under the covers, you can run the API call midclt call disk.query | jq. Each disk uses a combination of its serial number and world-wide name (LUN ID) as its unique identifier in the TrueNAS configuration database. Each entry per disk ID also lists the current device name and ZFS GUID

Don't think info in midclt call disk.query | jq output is related here, because before reboot Storage - Disks screen was showing correct info, but Pool Status screen was showing partition labels instead of disks.
Maybe has to do with pool.import_find (API) process or something similar that could have different methods for boot time and for manual import UI button that was cached somewhere until reboot. Just guessing, not that familiar with code.

 

[Patrick M. Hausen]

1. Partitioning disks for multiple pools/vdevs is not supported. Everything in TrueNAS assumes a disk belongs to one vdev only.
2. That being said, if you insist, always refer to partitions via GPTID when creating pools. That will at least keep things consistent.

HTH,
Patrick

 

[InQuize]

Partitioning disks for multiple pools/vdevs is not supported.

Yeah, on Core nothing is supported beyond group help from amazing community on this forum and bugfixes from dev team.
Virtualizing was never supported either, and yet it is essential for many folks nowadays.

Partitioning as a feature is not being suggested or advocated here. This is a process that should belong to CLI in my opinion.
But do you really think that TrueNAS should only be able to import and use ZFS-valid pools created inside the eco-system and nothing more?

Everything in TrueNAS assumes a disk belongs to one vdev only.

It looks like it in UI, but the team behind middleware is smarter that this, since everything works as intended even during my exercises.
I tested: importing/exporting, native encryption, SMB/NFS sharing, snapshoting, replication, jail mounts..

always refer to partitions via GPTID when creating pools.

This command is what I came up with during my homework:

Code:

zpool create -o altroot=/mnt \
-o feature@encryption=enabled \
-O encryption=aes-256-gcm \
-O keylocation=prompt \
-O keyformat=hex \
poly mirror gpt/poly-3XG29K2T gpt/poly-2PGX5G8T

That will at least keep things consistent.

From what I understand, GPT labels should be a functional substitute to GPT IDs here.

 

[Patrick M. Hausen]

No, they are not - because the middleware and the UI rely on GPT IDs only. Of course you won't break the pool by using GPT labels instead, just as you can use raw device names on the command line, too. I'm only arguing that you should expect inconsistent results if you are not sticking to GPT IDs. I wrote the first guide on partitioning your boot media for an additional data pool. So I am not advocating you should never ever do what you are doing. But you will make your life easier in the UI using IDs instead of labels.

 

[InQuize]

I'm only arguing that you should expect inconsistent results if you are not sticking to GPT IDs.

Ok, noted. Thank you.

My assumption is based on previous experiences documented in this thread, and info from freebsd man pages, but I could very well be missing middleware code quirks.

Any examples of such inconsistences you may recall now?

 

[Patrick M. Hausen]

Any examples of such inconsistences you may recall now?

Partly partition names, partly disk devices shown in the UI?

OK, with GPT IDs you get only the disk devices, but at least consistently so

 

[InQuize]

OK, with GPT IDs you get only the disk devices, but at least consistently so

I am getting disk names no matter what is used during pool creation.
Inconsistency described in the initial post is probably only temporary and exists after you imported freshly created pool for the first time and before reboot that erases some volatile part of appliance filesystem.
I plan to recreate this pool and try to reproduce it.

UPD: and this particular inconsistency is so minor, that I wouldn't consider not having convenience of labels because of it.

 

[Johnny Fartpants]

My assumption is based on previous experiences documented in this thread,

Wow, I wrote that a long time ago. Everything I said in that thread is still true however I do tend to agree with @Patrick M. Hausen in that trying your best to stick with the UI and only deviate when absolutely necessary. Sometimes at a glance all appears fine until you uncover something and then think ah shucks.

I've learned a lot over the years and a lot of that is down to this forum so thanks everyone for contributing. After running FreeNAS/TrueNAS for the last 10ish years and currently at 30+ PBs I can confirm that sticking with the UI is best in general and will give you better and more consistent outcomes.

 

[InQuize]

currently at 30+ PBs I can confirm that sticking with the UI is best in general and will give you better and more consistent outcomes.

Yeah, as TrueNAS is oriented exactly at this type of setups, and I have a mere homelab with completely different budgets for both the system and its yearly maintenance.
I was balancing ZFS storage architecture for my setup ever since FN 9.3 and I have not come up with anything even close to the same amounts of usable space at this kind of budget. So it is necessary for me and not only a homelab exercise. Practicing such things in enterprise production is probably not the best idea, and is the case where you really consider that it is "not supported".

Sometimes at a glance all appears fine until you uncover something and then think ah shucks.

That's why I am not jumping at the idea and doing ton of testing first.

Wow, I wrote that a long time ago.

Did not expect to actually hear from the author (:
How was your experience with labels in real use?

I created mine like so:

Code:

gpart modify -i 2 -l poly-3XG29K2T da3

and not with glabel, still have to make sure that there is no critical difference between the two.

 

[Patrick M. Hausen]

GPT label and GEOM label are two different things.

 

[Johnny Fartpants]

Sure, and you are doing the right thing by playing and learning outside of production as that's how we all learn.

I love this https://twitter.com/rogerskaer/status/1576025818182332416?lang=en

I used glabel simply because I was very familiar with it as I used gmultipath to label my multipath systems so it seemed the right thing to do to keep things consistent.

Im not trying to brag by the above statement but just to highlight that I've been around the block a bit and hope to share some of the things I've learned with others as they have with me over the years for which Im very grateful.

I primarily started labelling my drives because I was so scared about disk identification in the early days. Some systems had over 100 drives across three JBODs and I needed to be sure which was which when it came to disk replacement. Over the years I have become more confident and dare I say skilled at identifying drives without the need of my makeshift labels so have slowly done away with them on newer systems (still have some in production). I never had an issue personally with labels but it does add another layer of complexity when building and managing pools.

 

[InQuize]

GPT label and GEOM label are two different things.

Yes, and I chose GPT label exactly because it seems more universal and less invasive.

From what I gathered, geom automatic label is written to the last block of partition. Which sounds bad considering ZFS fragility around metadata.
It should not be a problem when zpool is created after labeling as file system simply sees 1 block less space at this time.
But should not allow to add label to existing unlabeled pool device since god knows what will be overwritten with that one last block.
Might be ok to relabel existing pool device that already has label.

On the other hand GPT label should be part of GPT metadata that is not part of filesystem blocks, which makes it a lot less harmful.
Moreover, I tested adding GPT label to 4TB clones of my existing 96% allocated pools and it continues to function without any noticeable consequence. Not ready to bet that there are none at this point, though.
Another great part is that unlike geom, GPT labels are not BSD specific, and supported in Linux as well, which is a huge bonus to universality. Windows uses GPT labels as well for its default partitioning during installation.

During my testing, I was fatigued after dealing with GPT IDs so quick, that my gut predicts mistakes outside of scripting being only a matter of time, so the goal with labels was to mimic /dev/disk/by-id/ tree of Linux sysfs, adapting for the context of zpools, making it human readable, unique, portable and definitive.

I love this https://twitter.com/rogerskaer/status/1576025818182332416?lang=en

So true

 

[Patrick M. Hausen]

You are perfectly correct. GEOM label stems from a time when GPT did not yet exist or was at least not yet in wide adoption.

 

[InQuize]

My initial attempts to reproduce UI inconsistency were not successful, but I took a little dive in code.

There is a helpful comment pointing out that I was presented with path of device:
/src/app/pages/storage/volumes/volume-status/volume-status.component.ts#L564
because device name was absent in datastore for some reason.
I did not test bulk actions with pools mixed on same disks, maybe it has smth to do.
Still haven't found point at which data gets parsed into datastore in the first place.

But as for UI, call pool.query is used get values for Pool Status screen, and all possible required info is already there
under topology - data - children for each pool device, including path, partition is called device and disk:

Code:

root@nas:~ # midclt call pool.query | jq
{
    "id": 49,
    "name": "poly",
    "guid": "3833979279566409292",
    "encrypt": 0,
    "encryptkey": "",
    "path": "/mnt/poly",
    "status": "ONLINE",
    "scan": {
      "function": null,
      "state": null,
      "start_time": null,
      "end_time": null,
      "percentage": null,
      "bytes_to_process": null,
      "bytes_processed": null,
      "bytes_issued": null,
      "pause": null,
      "errors": null,
      "total_secs_left": null
    },
    "topology": {
      "data": [
        {
          "type": "MIRROR",
          "path": null,
          "guid": "7253049314549764203",
          "status": "ONLINE",
          "stats": {
            "timestamp": 10033019491180,
            "read_errors": 0,
            "write_errors": 0,
            "checksum_errors": 0,
            "ops": [
              0,
              61,
              513,
              0,
              0,
              0,
              0
            ],
            "bytes": [
              0,
              1568768,
              5898240,
              0,
              0,
              0,
              0
            ],
            "size": 3985729650688,
            "allocated": 602112,
            "fragmentation": 0,
            "self_healed": 0,
            "configured_ashift": 12,
            "logical_ashift": 9,
            "physical_ashift": 12
          },
          "children": [
            {
              "type": "DISK",
              "path": "/dev/gpt/poly-3XG29K2T",
              "guid": "12705696833310567128",
              "status": "ONLINE",
              "stats": {
                "timestamp": 10033019533581,
                "read_errors": 0,
                "write_errors": 0,
                "checksum_errors": 0,
                "ops": [
                  0,
                  44,
                  255,
                  0,
                  0,
                  0,
                  0
                ],
                "bytes": [
                  0,
                  839680,
                  2949120,
                  0,
                  0,
                  0,
                  0
                ],
                "size": 0,
                "allocated": 0,
                "fragmentation": 0,
                "self_healed": 0,
                "configured_ashift": 12,
                "logical_ashift": 9,
                "physical_ashift": 12
              },
              "children": [],
              "device": "da3p2",
              "disk": "da3",
              "unavail_disk": null
            },
            {
              "type": "DISK",
              "path": "/dev/gpt/poly-2PGX5G8T",
              "guid": "769435533208944199",
              "status": "ONLINE",
              "stats": {
                "timestamp": 10033019556067,
                "read_errors": 0,
                "write_errors": 0,
                "checksum_errors": 0,
                "ops": [
                  0,
                  17,
                  258,
                  0,
                  0,
                  0,
                  0
                ],
                "bytes": [
                  0,
                  729088,
                  2949120,
                  0,
                  0,
                  0,
                  0
                ],
                "size": 0,
                "allocated": 0,
                "fragmentation": 0,
                "self_healed": 0,
                "configured_ashift": 12,
                "logical_ashift": 9,
                "physical_ashift": 12
              },
              "children": [],
              "device": "da4p2",
              "disk": "da4",
              "unavail_disk": null
            }
          ],
          "unavail_disk": null
        }
      ],
      "log": [],
      "cache": [],
      "spare": [],
      "special": [],
      "dedup": []
    },
    "healthy": true,
    "status_detail": null,
    "autotrim": {
      "value": "off",
      "rawvalue": "off",
      "parsed": "off",
      "source": "DEFAULT"
    },
    "encryptkey_path": null,
    "is_decrypted": true
  }

So we could use same trick (placed around here after mentioned block) in webUI code to display GPT label if it is present in "path":

Code:

// if path points to GPT label use it instead for device name
if ( (data as VDev).path.startsWith("/dev/gpt/") ) {
      (data as any).disk = data.path.substring(5);
    }

Not sure if my use of type assertion is correct, but general idea is there.
It won't change default behavior, but at the same time should give desired info for devices imported using GPT label.

I am not that handy with dev environments to test it immediately.
Setting it up will take me time, so anyone could probably beat me to it.

---

But I did spend time reading related middleware (pool_, disk_, zfs_ plugins) code, and I got to say it is a lot more intelligent than one would expect seeing it only operate with disks in UI. It gathers everything about disks there is to know. For example, lists all partitions, compares their UUID to freebsd_zfs (and Linux one as well) and freebsd_swap ones as part of checks before doing anything to disk. Also, looks for both types of disk (only for boot devices actually) and partition for the rest in /dev/ tree when interrogating OS.

I haven't found much that could hurt my use case if one would not perform disk operations via webUI to logical disks and pools that are sharing physical disk. Which does not mean that my review was comprehensive and sufficient.
The only silently dangerous part I encountered is pool device Detach process is working ahead of things, trying to wipe the disk clean and not only performing zpool detach itself. This makes sense knowing that other parts of middleware check if disk is clean to be considered available for use. Still, IMO there should always be a warning and a dialog to enter text as a confirmation for such action and it is not. The box assumes that disk belongs to it and not user, which is quite rude.
Wipe is failing if neighboring partition zpool is currently imported, thus disk is in a busy state, but I don't see why it won't otherwise.

---

Couple useful midclt calls to correlate current UI representation to libzfs info as seen by middleware:

Code:

root@nas:~ # midclt call zfs.pool.get_disks poly | jq
[
  "da3",
  "da4"
]

root@nas:~ # midclt call zfs.pool.get_devices poly | jq
[
  "gpt/poly-3XG29K2T",
  "gpt/poly-2PGX5G8T"
]

And couple that might help around encrypted swap mirrors, when a moment to manually replace disk comes:

Code:

# midclt call disk.swaps_remove_disks da3 da4
midclt call disk.swaps_configure

Not sure how to escape list syntax for midclt, though, not tested much yet.