Custom widgets ?

[gaston.lu]

I hope this is the right forum for my question. Am I right, that there is no official way of creating custom widgets for TrueNAS (SCALE in my case) ?

Background:

After I have experienced poor support of my previous NAS vendor on several occasions, I decided to build my own custom NAS. I have currently two software options, one already running, the other being TrueNAS, mainly because of its better ZFS integration. Not being able to have custom widgets would however be a drawback.

While custom widgets just generally make a solution more future-proof, one important one for me would be the possibility to display the physical bay layout with disk status. The main question is not if such custom widget already exists, just if there is a documented way to develop such a widget.

I read somewhere that such functionality (not sure if widget or separate page) exists only for iX hardware.

If custom widgets are at all possible I would consider developing one myself. If not, then the decision on moving to TrueNAS would be much harder for me.

 

[danb35]

I read somewhere that such functionality (not sure if widget or separate page) exists only for iX hardware.

A display of drives-to-bays exists only for certain iX hardware. I'm not aware of a way to build a custom widget/page into the UI.

 

[gaston.lu]

A display of drives-to-bays exists only for certain iX hardware. I'm not aware of a way to build a custom widget/page into the UI.

Ok, thanks for confirmation. It's a pity, but I think I will then stay with my current solution. Thanks!

 

[ragametal]

@gaston.lu i can confirm that some IX Hardware have this functionality. I have a Truenas Mini X+ in my office that has it

However, i have a second system at home that is a DIY and doesn't have this.
It would be awesome to have this same functionality on DIY systems but as far as I know it is not possible.

Out of curiosity, which other solution are you trying that has this?

 

[Arwen]

Ok, thanks for confirmation. It's a pity, but I think I will then stay with my current solution. Thanks!

Yes, it is a missed feature.

Part of the issue is that the Enterprise customers pay for the software development. Adding a custom method to display any storage enclosure layout is not needed by the Enterprise customers.

That said, iX has been kind and has helped with other less complex issues that affect mostly SOHO users who use the NAS software for free. Though I doubt this issue will get any attention any time soon... (SCALE is still under development, some bugs to squash and features to add.)

 

[gaston.lu]

@gaston.lu i can confirm that some IX Hardware have this functionality. I have a Truenas Mini X+ in my office that has it
View attachment 72863

However, i have a second system at home that is a DIY and doesn't have this.
It would be awesome to have this same functionality on DIY systems but as far as I know it is not possible.

Out of curiosity, which other solution are you trying that has this?

Well claiming that things are not possible is often just business protective. I don't see why it would not be possible while I have to admit that there is a tiny chance of mixing up drives. I have an algorithm in head with which I think I could exclude most mixup cases. Only if more than one drive is replaced at the same time than a software update occured, or drives added and swapped at the same time with shutdown system it could be possible that the algorithm mixes up drives. But it could detect such instances and notify for manual checking

This should at least work with SAS backplanes where one could control the locator LED. FOr othe configurations it might be more tricky to identify the drives for the setup.

The idea is to map every bay to the drive path interface/controller/channel/drive, its WWN and serial number. With these information most situations could be resolved, other need to be manually resolved or confirmed. For example if the path changes for a given drive WWN it could be due to a reboot and changes in driver loading or it could be that the drives have been moved around. Many of these cases could be correctly identified and resolved, while others not.

I am currently using UnRaid 6.12 which introduces native ZFS support. On this platform there is a plugin with a widget which allows to define and map drives to bays. But I don't know how robust it is. Once I am finally decided I will have a closer look.

 

[gaston.lu]

Yes, it is a missed feature.

Part of the issue is that the Enterprise customers pay for the software development. Adding a custom method to display any storage enclosure layout is not needed by the Enterprise customers.

I do not expect such thing, I do only expect a possibility to develop custom widgets. There should be such API for the developers official widget not to reinvent the wheel for each widget.

A very easy way to allow custom widgets (especially in the context of Apps being the future of plugins) would be a standard widget which can be configured to retrieve its content via HTTP(S) with an interval. Optionally it could be possible to configure an open TCP channel allowing to push updates.

That said, iX has been kind and has helped with other less complex issues that affect mostly SOHO users who use the NAS software for free. Though I doubt this issue will get any attention any time soon... (SCALE is still under development, some bugs to squash and features to add.)

Maybe iX could consider a paid Pro version involving a small fee (like Unraid does) for SOHO version allowing more customization.

After some more thinking an specially considering the current shortcomings of Unraid (especially currently limited ZFS support, and no user groups) I decided to give TrueNAS a try. I do just hope that importing the ZFS from Unraid and sharing the existing datasets will work fine. I have the data still on the old NAS but the transfer takes several days.

 

[Arwen]

I do not expect such thing, I do only expect a possibility to develop custom widgets. There should be such API for the developers official widget not to reinvent the wheel for each widget.

A very easy way to allow custom widgets (especially in the context of Apps being the future of plugins) would be a standard widget which can be configured to retrieve its content via HTTP(S) with an interval. Optionally it could be possible to configure an open TCP channel allowing to push updates.

Perhaps. iX may very well have a straight forward procedure to implement a new chassis into the GUI. You can certainly download the source code and see. And potentially make it work for you.

But, I think what you want is a user tool. I brought this up years ago as well. I have no answers.

Maybe iX could consider a paid Pro version involving a small fee (like Unraid does) for SOHO version allowing more customization.
...

This has been discussed here in the forums before. Don't know what iX is thinking. I can however see lots of potential problems.

 

[gaston.lu]

This has been discussed here in the forums before. Don't know what iX is thinking. I can however see lots of potential problems.

I agree with you there, but at the end virtually every plugin (on any platform) has the potential of problems in some way. In the case of the suggested plugin it could for instance not be developed by the community as strict naming conventions (id, classes, javascript entities, available libraries...) must be established to avoid clashes with the native UI and other widgets. Also default UI theme classes must be documentded for use by the widgets.

I was working as freelance product consultant for many years and the potential of problems was always a heavy argument of my customers to avoid plugins in their products.

Off topic:

Well, I bought 2 SSDs today and installed my first proxmox and TrueNAS Scale. Truenas is now running like a charm with pathrough SAS card. The zpool was also imported just had a small hiccup in mounted it on /mnt/mnt/data but this was quickly fixed by "zfs set mountpoint"

I have to admit that I like the TrueNAS interface much more than Unraid. It all comes a bit more professional and the flexibility of configuration is a huge change, giving access to virtually every configuration option (and this not only for zfs) that exists.

The only drawback for me is the lack of support for installing additional linux packages (I need ImageMagick). Unraid boots a ramdisk and installs packages at boot time. If one causes problems, simply delete the package and it vanishes from the installation after boot. But I think I can adjust my package build script and make a "sidecar" installation. As my script is also checking availability of shared library dependencies I could quickly evaluate the effort it would take to make it work.

One thing that initially pushed me away from TrueNAS was a comment in a youtube video comparing both, stating that the TrueNAS community was stiff-necked. Well I cannot confirm this and by that occasion want to thank everybody in this thread for the help provided. It might be that this community is more professionally populated which may clash with some pure amateur approach (like "why should I do it correctly, if it works like that", etc...). Me having worked in the Enterprise storage business (the real big SAN boys with hundreds of disks, multipath, multisite,...) in 2nd level support and some time with R&D, I do not fear but appreciate such community.

This said, I am quite confident that proxmox/TrueNAS will most probably be my final setup. Except if my performance testing the next days would disappoint, but I don't expect that.

Many thanks again to everbody!

 

[Arwen]

Several comments:

Proxmox is not considered well tested for running TrueNAS SCALE as a VM. No one is trying to stop you, that is just a statement of fact. Most here running TrueNAS as a VM are using VMWare.

Installing additional packages is not something that is supported. TrueNAS is more like "firmware", than a real OS. That said, at some point in Cobia, iX seemed to have a developer mode. You run a script and it allows non-supported, non-standard things like additional packages. Any bugs found need to be re-created with a vanilla TrueNAS before being accepted. Of course running an App could suffix for your use case.

I wrote a Resource about TrueNAS SCALE for very new users. Mostly giving them an idea of what SCALE is and is not, to allow them to make better decisions:

Welcome to TrueNAS SCALE - Beginners Intro

Since mid 2021 we have had many new users of TrueNAS wanting to try SCALE. Here are some things that are helpful for new TrueNAS SCALE users to know; SCALE is a specialized & targeted OS based on a Linux distro. However, SCALE is NOT a Linux...

www.truenas.com

Your comment about "stiff-necked" may be somewhat true. But, I would use other wording that long time forum members may also use, "conservative when it comes to data safety". Meaning we get people doing some outrageous things as a NAS, and the way we find out is that that are having trouble, then want help. Even data loss trouble. Even loosing their entire pool trouble. Thus, "conservative".

 

[gaston.lu]

Several comments:

Proxmox is not considered well tested for running TrueNAS SCALE as a VM. No one is trying to stop you, that is just a statement of fact. Most here running TrueNAS as a VM are using VMWare.

Understood, but I use this for home and cannot afford ESXi which is as far as I know (I used to be VMware certified, but that's many years back) the only VMware option to pass through hardware to the clients. Additionally, as I am passing through the complete controller directly to Truenas and my platform IOMMU supporting ACS, I value the risk being fairly low.

Installing additional packages is not something that is supported. TrueNAS is more like "firmware", than a real OS. That said, at some point in Cobia, iX seemed to have a developer mode. You run a script and it allows non-supported, non-standard things like additional packages. Any bugs found need to be re-created with a vanilla TrueNAS before being accepted. Of course running an App could suffix for your use case.

Yes, read this before and the disabled apt witnesses it as well. I will have a look into the App path, but my approach of "sidecar" install would be not to use any debian package but build my own (I have a complete script that I developped and for UnRaid which only needs minor changes for TrueNAS, which downloads and updates sources, build them, installs them to a virtual empty path (prefix) and then builds a package from there. This package would then be "installed" into a virtual root (ZFS dataset) with linux standard tree and the processes requiring it would just add this to their path. So TrueNas "firmware" is not changed at any point.

Just for the record, my script also checks all shared libraries required by the package against the host (TrueNAS in this case) so that I can also inject missing libraries into the "sidecar".

I have 2 scripts, the one described above and one for migrating shares which I may publish in the future. The one for migrating shares uses tar for performance on empty destination directories, and rsync if not empty. I also allows to check source and destination to be equal in file count and size, optionally listing the differences. This is pretty useful to make sure all has been copied correctly. I now have plan to add automated share creation on truenas as well, but are struggeling with TrueNAS scale CLI (will open other forum thread for this).

I wrote a Resource about TrueNAS SCALE for very new users. Mostly giving them an idea of what SCALE is and is not, to allow them to make better decisions:

Welcome to TrueNAS SCALE - Beginners Intro

Since mid 2021 we have had many new users of TrueNAS wanting to try SCALE. Here are some things that are helpful for new TrueNAS SCALE users to know; SCALE is a specialized & targeted OS based on a Linux distro. However, SCALE is NOT a Linux...

www.truenas.com

Thanks, read this now. Very useful stuff, but found nothing real new to me, which does not mean it is not useful to me as it is kind of verification.

Your comment about "stiff-necked" may be somewhat true. But, I would use other wording that long time forum members may also use, "conservative when it comes to data safety". Meaning we get people doing some outrageous things as a NAS, and the way we find out is that that are having trouble, then want help. Even data loss trouble. Even loosing their entire pool trouble. Thus, "conservative".

Just to make sure: "stiffneck" was not my own wording, but one I "heard" when searching for my best solution. I only later realized that this might just be due to the different type of community. I completely agree that "conservative" is the right word from the right perspective.

 

[Patrick M. Hausen]

Understood, but I use this for home and cannot afford ESXi

ESXi is free for home lab use.

 

[gaston.lu]

ESXi is free for home lab use.

Thanks for that information. I admit I did not know. While I have implemented ESXi at customer sites (years ago), this is my first installation at home where a type 1 hypervisor came to my mind. I use Rasberry Pi's and Nipogi servers for 24/7 services (DHCP, DNS,...) as they use far less power than a NAS. And so far I used pre-build NAS from Synology and QNap and VMs for "on-demand", like a local wiki, etc.. services on top of those.

However with ESXi alone it is not done as it has very tight hardware support matrix. My ASRock E3C256D4U-2L2T/BCM server mainboard with Xeon E-2378G CPU is not supported.

 

[Davvo]

Just a heads up since you want virtualize TN:

Resource - "Absolutely must virtualize TrueNAS!" ... a guide to not completely losing your data.

[---- 2024/01/16: Still relevant. Virtualization really doesn't change much. Updates made as appropriate. ----] [---- 2018/02/27: This is still as relevant as ever. As PCIe-Passthru has matured, fewer problems are reported. I've updated some...

www.truenas.com

 

[gaston.lu]

Just a heads up since you want virtualize TN:

Thanks for that information. I believe this is exactly how I implemented my solution with the exception of using proxmox instead of ESXi.

Just to be sure, did you make any particular problematic discovery in my previous writing, for you to give me that link, or just for general information ? Don't get this question wrong, I just want to make sure I am not missing anything here.

I use proxmox raw device pass through which gives the Broadcom/LSI 9500 8i controller as-is to TrueNAS. I added PCI vendor and device ID to increase security while not really necessary. There will be no other SAS controller ever in this server. Also device passthrough will be limited to the TrueNAS VM. I did also verify that proxmox does not "see" the drives attached to the SAS controller.

I was considering adding a script that keeps proxmox in single user mode if ever it detects a change in disk configuration, i.e. failure of mapping the controller and possibly rearranging disk device mapping. But given the exclusive use of zfs I rate this as not necessary.

 

[Davvo]

Thanks for that information. I believe this is exactly how I implemented my solution with the exception of using proxmox instead of ESXi.

Just to be sure, did you make any particular problematic discovery in my previous writing, for you to give me that link, or just for general information ? Don't get this question wrong, I just want to make sure I am not missing anything here.

From what I read, not really: just wanted to make sure you checked all the boxes before going production.
I could not find your intended use of Proxmox though, maybe I missed it.

 

[gaston.lu]

From what I read, not really: just wanted to make sure you checked all the boxes before going production.

Ok, great, thanks for that.

I could not find your intended use of Proxmox though, maybe I missed it.

Until now I had a QNap NAS which did run LXD containers and VMs for services which are not required to run all the time. With proxmox I can move these out to a platform specialized in virtualizations instead of running it in the NAS. It also allows for better memory management for TrueNAS.