bunch of noob questions

[jamsven]

Hello everyone,

I recently acquired a Supermicro server with four JBODs, offering space for approximately 150 HDDs, as I was in need of significant storage capacity. My previous experience was primarily with QNAP and Synology systems, which I appreciated for their user-friendly GUIs. However, I encountered several bugs and reliability issues with QNAP.

To address my storage needs, I set up a TrueNAS system and initiated a simple array consisting of 60 x 20 TB drives, configured as 6 groups of 10 drives each with RAID-Z2. This array is now 85% full, prompting me to seek additional drives to fill the remaining slots.

Fortunately, I discovered about 90 x 18 TB drives in our storage room. Unsure of their condition, I created a single large vdev with all these drives to test for any issues. This process revealed 14 faulty or degraded drives, and some did not appear at all, leaving me with 69 functional HDDs.

A crucial reason for my preference for GUIs in NAS systems is their ability to simplify storage management with minimal command-line interaction. Typically, these systems provide a clear overview of all slots, indicating any problematic HDDs and their exact locations, which greatly facilitates troubleshooting and saves time. After encountering many faulty drives and the subsequent challenges in destroying the pool, I am keen to find a similar feature in TrueNAS.

However, despite my search, I couldn't locate such a drive/slot overview in TrueNAS, a functionality I had expected to be standard.

Questions and Issues:

1. Drive/Slot Overview: From a practical standpoint, how can I obtain a detailed overview of the drives and slots in TrueNAS? Is there a way to manually map the drives once and then view them through the GUI?
Similar to

2. SMB Shares and Permissions: When creating SMB shares, I had to log in as the root user, as the admin user's permissions were inadequate. This issue seems to be common, as noted by other users on YouTube.

3. Reporting Page Errors: The reporting page consistently shows 'NULL' for CPU, HDD, etc. I'm still investigating this - I've set the timezone to UTC-0 and rebooted several times without success.

4. GRUB Boot Loader: The lack of an auto-timer in GRUB is inconvenient. Each time I boot up, I have to log in via IPMI to manually select TrueNAS for booting, which is quite bothersome. I know it should have an autotimer and I never touched any grub settings.

Overall, my experience with TrueNAS feels more glitchy than with QNAP, although I'm hopeful this might be due to my limited familiarity with the system.

Any advice or insights would be greatly appreciated. Thank you in advance for your help.


Software:
- Linux Kernel 5.15.131+TrueNAS
- TrueNAS-SCALE-22.12.4

Hardware:
- Server: Supermicro CSE-218U X11QPH+ 19" 2U
- 24x 2.5" SAS SFF, 4x NVMe, 4x Intel XEON LGA3647, up to 12TB DDR4, 1.5 TB RAM
- LSI SAS9300-8i 9300-8i 9311-8i PCIe x8, 12x SFF-8643

- JBODs: Supermicro JBOD Storage 19" 4U 45x SAS S-ATA 847E16-RJBOD1

 

[Ericloewe]

Please note that external image hosts are not allowed. Please upload your images directly to the forum.

Drive/Slot Overview: From a practical standpoint, how can I obtain a detailed overview of the drives and slots in TrueNAS

You don't, for a variety of reasons, that is a feature exclusive to iXSystems hardware.

SMB Shares and Permissions: When creating SMB shares, I had to log in as the root user, as the admin user's permissions were inadequate. This issue seems to be common, as noted by other users on YouTube

Most Youtube tutorials range from "outright dangerous" to "byzantine and pointless". The manual contains instructions on how to correctly setup SMB shares and user permissions.

Reporting Page Errors: The reporting page consistently shows 'NULL' for CPU, HDD, etc.

That's a weird one, maybe it's because the interfaces are different for the quad-socket platforms? It's niche hardware, though not that niche...

GRUB Boot Loader: The lack of an auto-timer in GRUB is inconvenient. Each time I boot up, I have to log in via IPMI to manually select TrueNAS for booting, which is quite bothersome. I know it should have an autotimer

It does, something is probably sending spurious keypresses. Try resetting the iKVM in the BMC WebGUI.

 

[jamsven]

Please note that external image hosts are not allowed. Please upload your images directly to the forum.


You don't, for a variety of reasons, that is a feature exclusive to iXSystems hardware.

Most Youtube tutorials range from "outright dangerous" to "byzantine and pointless". The manual contains instructions on how to correctly setup SMB shares and user permissions.

That's a weird one, maybe it's because the interfaces are different for the quad-socket platforms? It's niche hardware, though not that niche...


It does, something is probably sending spurious keypresses. Try resetting the iKVM in the BMC WebGUI.

Thank you for your quick response. I understand that the Drive/Slot Overview is exclusive to iXSystems hardware, which is a bit disheartening.

I've recently acquired used hardware that has the iXSystems logo on it, so it seems to be official hardware from iXSystems. In light of this, I'm wondering how I can access the non-limited version of TrueNAS that includes this feature. Could anyone advise on whom I should contact to discuss this?

I recognize the necessity for iXSystems to monetize their software, but a comprehensive GUI for NAS setup is one of the main appeals of TrueNAS, especially when compared to other commercial solutions. Relying more on CLI and custom scripts seems counterintuitive when the GUI lacks basic functions. Any guidance or suggestions from the community would be greatly appreciated.

I red that some writing down all Serial numbers to match them in Excel sheets with their coresponding slots... This sounds like a workflow from 20 years ago. Are there more effient ways to solving this issue ?

 

[Ericloewe]

Any vaguely-decent hardware, yours included, should support SES, so as to allow a disk to be identified (from the lights on the front panel) using sesutil:
sesutil locate daX enable/disable

 

[jamsven]

Thank you for your suggestion regarding the use of sesutil for drive identification. While this tool is useful for pinpointing specific drives via their front panel lights, it doesn't address my core requirement: an intuitive and comprehensive overview of drive health and status.

In NAS systems like QNAP or Synology, such a feature is standard and incredibly beneficial. These systems offer an easy-to-navigate GUI that quickly highlights problematic drives and provides detailed information about their status, which is a critical feature missing in TrueNAS.

For instance, I recently created a test pool with a drive known to have defects. The shell indicated "/dev/sdp [SAT], 71 Offline uncorrectable sectors". Despite this, both the topology and ZFS health checks showed the drive as healthy, with no issues reported. It was only after conducting long SMART tests that a warning symbol appeared under disk health. However, I still had to manually search through all the disks to locate this warning. This process is neither efficient nor user-friendly, especially when managing a large number of drives.

Moreover, I've encountered a peculiar issue regarding user permissions in TrueNAS. When logged in as an admin, I am unable to execute 'zpool' or 'zfs' commands, which are essential for managing storage pools and file systems. Strangely, these commands are accessible when logged in as the root user. This restriction is inconvenient and seems counterintuitive, considering the admin user should have sufficient permissions for these fundamental tasks.

In summary, while TrueNAS offers robust storage solutions, it lacks in critical areas of user interface and ease of management, particularly when compared to other NAS systems. The absence of a straightforward, GUI-based overview for drive health and status significantly hampers the efficiency of managing large storage arrays. Additionally, the inconsistent user permission settings add unnecessary complexity to system administration.

I'm keen to understand why such essential features, standard in other NAS systems, are not present in TrueNAS. Any insights or solutions to these challenges would be greatly appreciated.

Thank you for your assistance.

 

[Ericloewe]

Moreover, I've encountered a peculiar issue regarding user permissions in TrueNAS. When logged in as an admin, I am unable to execute 'zpool' or 'zfs' commands, which are essential for managing storage pools and file systems. Strangely, these commands are accessible when logged in as the root user. This restriction is inconvenient and seems counterintuitive, considering the admin user should have sufficient permissions for these fundamental tasks.

That's what sudo is for. You can delegate certain permissions, but Linux imposes some limitations (e.g. only root can mount filesystems).

The absence of a straightforward, GUI-based overview for drive health and status significantly hampers the efficiency of managing large storage arrays.

Do you have a proposal for such a GUI that would work on somewhat-arbitrary hardware? Because that's step zero and I haven't seen anyone propose something concrete. Then someone has to implement it, and that's another can of worms. How do you deal with different chassis? Different backplanes (e.g. Supermicro's old SAS/SATA-only backplanes vs. the newer -N4 backplanes that also have four U.2 slots)? Different cabling to different HBAs or directly to PCIe?
This problem gets way out of hand very quickly.

In NAS systems like QNAP or Synology

...which are limited to their hardware, with known, limited configurations.

I've recently acquired used hardware that has the iXSystems logo on it, so it seems to be official hardware from iXSystems. In light of this, I'm wondering how I can access the non-limited version of TrueNAS that includes this feature. Could anyone advise on whom I should contact to discuss this?

As I understand it, it should just work with modern TrueNAS. It may be an older system from before this feature was thing, or possibly hacked-together and no longer recognized as iX hardware.

 

[jamsven]

The HDD mapping could take inspiration from other systems. Unraid also has the ability for manual mapping, which looks a bit oldschool...

I envision a user interface where all disks are listed, allowing users to click on each to activate their status LED. Further, the ability to drag and drop these disks into a grid layout ans save those as the physical layout. This approach seems feasible and would significantly improve the system's usability, especially for large-scale storage environments.



Addressing the issue of disk health reporting, it's perplexing that drives with multiple issues and failed SMART tests still appear as error-free in the devices overview. This inconsistency is evident in the attached screenshot and raises concerns about the reliability of the system's health monitoring capabilities. A more intuitive and accurate reporting mechanism is crucial for effective storage management.

Are red leds from the hardware supported? Usually with other software, the status LED of the drive should be red or is there an additional setup involved?


Also I don´t get, how to interpret my temperatures correctly....

Automatic updates gives me: Cannot connect to host update.ixsystems.com:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate is not yet valid (_ssl.c:1123)')]: Automatic update check failed. Please check system network settings.
which seems to be bios timezone related (i will further investigate tomorrow)



Another point of contention is the difference in permissions between the root and admin accounts. For instance, I was unable to create Samba shares using the admin account through the web GUI; however, it was possible when logged in as root. This inconsistency isn't just inconvenient but also counterintuitive. It took a considerable amount of time and research, including watching YouTube tutorials, to find the same issue for others and that they used root login as a workaround for this issue.

More missing features between payed and free version?
I really wanna love Truenas and still trying finding good arguments. If I can´t run the pro/enterprise version, what are the other deficits of it? What else did they not implemented? I couldn´t find a comparison, between free with all the stripped down features and the pro version and how reliable would it be in the future, that they don´t strip out more features to force people using only their hardware?

 

[danb35]

[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate is not yet valid

Your system date is set incorrectly. Fix it. It's not likely just the timezone--that would affect this only during a window of a few hours. Your system clock needs to be set to the current date/time in UTC.

Another point of contention is the difference in permissions between the root and admin accounts.

There should not be any difference in the web UI--if you're seeing one with the current release, that would be a bug, and I'd recommend you report it using the Report a Bug link at the top of the page.

If I can´t run the pro/enterprise version, what are the other deficits of it? What else did they not implemented?

Looks like this page sets out the differences:

Compare TrueNAS Editions - Powerful Storage Platform

TrueNAS is the ultimate Open Source storage platform. Choose from CORE, Enterprise, or SCALE to get the best performance and reliability.

www.truenas.com

If you need FIPS security, you need Enterprise. If you need (supported) Fibre Channel, you need Enterprise. Other than enclosure management in the GUI (because Enterprise runs only on iX hardware), those look like the only significant differences.

 

[Ericloewe]

Are red leds from the hardware supported?

Yes, via SCSI Enclosure Services. Linux seems to have the sg_ses utility. I have no experience with it, but FreeBSD's sesutil generally seems to work well with Supermicro hardware.

 

[Tony-1971]

Hello,
In my Scale I can use:

Code:

# cat /sys/class/enclosure/6:0:0:0/Slot\ 00/fault
0

# echo 1 > /sys/class/enclosure/6:0:0:0/Slot\ 00/fault
# echo 0 > /sys/class/enclosure/6:0:0:0/Slot\ 00/fault

to turn on and off red leds.
Best Regards,
Antonio

 

[jamsven]

Another weird thing for me are the stats.
For a name like TrueNas, nearly all my stats are false or missing.

On GUI it shows 9% CPU usage

and with htop it shows 75%

running copy over samba with approx 1,4GB/s so it seems to max out the CPU.

At least HDD usage seems to be correct, but I don´t know which numbers I can trust in the GUI.

 

[Patrick M. Hausen]

I don't know how htop calculates the "CPU%", but the "stats per thread" in the UI seem to agree with the top three rows of your screenshot. The displayed average is just that: the average across all threads.

 

[danb35]

and with htop it shows 75%

...because you don't understand what htop is telling you--the 75% is the percentage of a single core/thread that's being used by that process. No conflict there.

 

[Ericloewe]

I think that's configurable now. In the past two-three years, htop has seen a bunch of improvements.

 

[NugentS]

Yo get stats on HDD's you could try running the script @joeschmuck created called multi_report which you should be able to find in the resources section.

multi_report.sh version for Core and Scale

TooMuchData submitted a new resource: multi_report.sh versions for Core and Scale - ZPool & SMART status report ZPool & SMART status report. Original script by joeschmuck, modified by Bidelu0hm, then by melp. Minor corrections by TooMuchData. Read more about this resource...

www.truenas.com

 

[jamsven]

I don't know how htop calculates the "CPU%", but the "stats per thread" in the UI seem to agree with the top three rows of your screenshot. The displayed average is just that: the average across all threads.

Thanks for clarification.
So to understand correctly my stats in htop. The first smb copy process utilize across multiple threads with a few percent, but utilizing only one core with 75% ?

So in total 2 copy processes only utilize 2 cores and the rest of the cores are idle?
So for me its good news, because I thought my CPU is already the bottleneck.

Could you suggest a direction or setting, how I can utilize all cores with only few smb clients?
On google I found mainly settings for windows server in combination with SMB Direct to archive more thoughput.
Locally I can write with about 3 GB/s on the ZFS while on SMB I´m limited to 1.3GB/s-1.6GB/s

As a next project I would love to setup a flash based NAS (I still have 36x 3,6TB Intel drives), but the smb limitations of Linux seem to be a bootleneck if im right? At least on youtube I only found videos where people tried it with linux and than had to switch to windows for full SMB performance. I would love to saturate my 200GB ethernet.

 

[jamsven]

Another weird issue, NAS was working for several days, I had to move it a few racks over and, recabled everything as before and now truenas doesent boot up anymore.
I´m now getting

Out of Memory and You Need to Load the Kernel First​

Message and booting after grub takes like 7mins untill I see the truenas boot option...
Is it possible to just reinstall truenas and import the existing ZFS back into the system?

 

[NugentS]

Yes