Supermicro Drive Indicator LEDs?

[isopropyl]

I have seen a few scripts for drive status leds for various platforms including supermicro. Some are years old.
I was curious, what the latest is, or is it supported natively?

All my drives have a blue led showing on the ones that are in the system. And they seem to do a short flicker every few seconds and I hear the arms moving on the drives. But when a drive was giving errors last time in TrueNAS, none of them changed colors. So I just wasn't sure.
I am not sure how they are supposed to work either in all honesty.

 

[jyavenard]

The H11SSL-i only does SATA.

Looking at your backplane manual, you need to connect the SGPIO header of the backplane to the motherboard's SGPIO, which the H11SSL-i doesn't have (I recently go the H11SS-NC myself and tried the same thing)

In any case, I don't think TrueNAS supports this. Making the LED blinks a proper colour requires some scripts last I looked into it.

 

[isopropyl]

The H11SSL-i only does SATA.

All my drives are SAS except my boot SSD is SATA. I'm wondering if I got the wrong backplane model listed..
Lemme figure that out and get back to you.

 

[danb35]

The H11SSL-i only does SATA.

...but the listed HBA does SAS. And the SAS cable to the backplane should take care of any other enclosure management signaling that's required.

@isopropyl, I cobbled together a script to light the alert LEDs on disk failure. I assume it still works, but as I don't use CORE any more I can't really keep it up. But it's here if you want to check it out:

Disk failure LEDs for Supermicro SAS backplanes

Many hot-swap backplanes have disk-failure LEDs that can be turned on when a disk fails. Unfortunately, there doesn't seem to be a standard mechanism to do this, and FreeNAS doesn't have any built-in support for this feature. Using inspiration...

www.truenas.com

TrueNAS won't do this by itself except on iX hardware.

 

[isopropyl]

@isopropyl, I cobbled together a script to light the alert LEDs on disk failure. I assume it still works, but as I don't use CORE any more I can't really keep it up. But it's here if you want to check it out:

Disk failure LEDs for Supermicro SAS backplanes

Many hot-swap backplanes have disk-failure LEDs that can be turned on when a disk fails. Unfortunately, there doesn't seem to be a standard mechanism to do this, and FreeNAS doesn't have any built-in support for this feature. Using inspiration...

www.truenas.com

Thanks, I will give this a try.
Does it show anything for spare drives, or drives that are replaced?

And by disk fail, is it only if it flat out dies, or if it is "failing" too?

 

[danb35]

IIRC, the script is based on the ZFS status--if ZFS reports a disk as anything other than ONLINE, it lights the LED.

 

[isopropyl]

I assume it still works, but as I don't use CORE any more I can't really keep it up. But it's here if you want to check it out:

zsh: ./zpscan.sh: bad interpreter: /usr/local/bin/bash^M: no such file or directory

The Syntax:

Code:

#! /usr/local/bin/bash
if [ ! "$1" ]; then
  echo "Usage: zpscan.sh pool "
  echo "Scan a pool, send email notification and activate leds of failed drives"
  exit
fi
if [ ! -d /root/.sas2ircu ]; then
  mkdir /root/.sas2ircu
  touch /root/.sas2ircu/drives
  touch /root/.sas2ircu/locs
fi
if [ "$2" ]; then
  email="$2"
else
  email="root"
fi
condition=$(/sbin/zpool status $1 | egrep -i '(DEGRADED|FAULTED|OFFLINE|UNAVAIL|REMOVED|FAIL|DESTROYED|corrupt|cannot|unrecover)')
if [ "${condition}" ]; then
  emailSubject="`hostname` - ZFS pool - HEALTH fault"
  mailbody=$(zpool status $1)
  echo "Sending email notification of degraded zpool $1"
  echo "$mailbody" | mail -s "Degraded Zpool $1 on hostname" $email
  drivelist=$(zpool status $1 | grep -E "(DEGRADED|FAULTED|OFFLINE|UNAVAIL|REMOVED|FAIL|DESTROYED)" | grep -vE "^\W+($1|NAME|mirror|logs|spares)" | sed -E $'s/.*was \/dev\/([0-9a-z]+)/\\1/;s/^[\t  ]+([0-9a-z]+)[\t ]+.*$/\\1/')
  echo "Locating failed drives."
  for drive in $drivelist;
  do
  record=$(grep -E "^$drive" /root/.sas2ircu/drives)
  location=$(echo $record | cut -f 3 -d " ")
  echo Locating: $record
  sas2ircu 0 locate $location ON
  if [ ! "$(egrep $location /root/.sas2ircu/locs)" ]; then
  echo $location >> /root/.sas2ircu/locs
  fi
  done
else
  echo "Saving drive list."
  drivelist=$(zpool status $1 | grep -E $'^\t  ' | grep -vE "^\W+($1|NAME|mirror|logs|spares)" | sed -E $'s/^[\t ]+//;s/([a-z0-9]+).*/\\1/')
  saslist=$(sas2ircu 0 display)
  printf "" > /root/.sas2ircu/drives
  for drive in $drivelist;
  do
  sasaddr=$(sg_vpd -i -q $drive 2>/dev/null | sed -E '2!d;s/,.*//;s/  0x//;s/([0-9a-f]{7})([0-9a-f])([0-9a-f]{4})([0-9a-f]{4})/\1-\2-\3-\4/')
  encaddr=$(echo "$saslist" | grep $sasaddr -B 2 | sed -E 'N;s/^.*: ([0-9]+)\n.*: ([0-9]+)/\1:\2/')
  echo $drive $sasaddr $encaddr >> /root/.sas2ircu/drives
  done

  for loc in $(cat /root/.sas2ircu/locs);
  do
  sas2ircu 0 locate $loc OFF
  done
  printf "" > /root/.sas2ircu/locs
fi

  for loc in $(cat /root/.sas2ircu/locs);
  do
  sas2ircu 0 locate $loc OFF
  done
  printf "" > /root/.sas2ircu/locs
fi

#! /usr/local/bin/bash
Does bash just need to be changed to zsh?

 

[jyavenard]

All my drives are SAS except my boot SSD is SATA. I'm wondering if I got the wrong backplane model listed..
Lemme figure that out and get back to you.

then you don't have a H11SSL-i
It doesn't have a HBA (only the H11SSL-N and H11SSL-NC do). It does have 16 SATA ports

Edit: sorry, didn't realise you had an HBA card. my bad

 

[danb35]

then you don't have a H11SSL-i

Did you not read that he has a SAS HBA? Which I've now told you twice, and is mentioned in the signature of every one of his posts?

 

[danb35]

zsh: ./zpscan.sh: bad interpreter: /usr/local/bin/bash^M: no such file or directory

It looks like you've edited this file under Windows, which has messed with the line endings. Either edit it directly on your NAS (using, e.g., nano), or use a Windows editor that's aware of Unix line endings, like Notepad++ or VSCode.

 

[isopropyl]

It looks like you've edited this file under Windows, which has messed with the line endings. Either edit it directly on your NAS (using, e.g., nano), or use a Windows editor that's aware of Unix line endings, like Notepad++ or VSCode.

Ah, I see what happened. I missed the Github link in the updates, so I grabbed it off serverfault and threw it in pastebin and fetched it that way haha.
So I ran it and it said it ran no issues. Just trying to get it setup on the cron.
To confirm the syntax, is it like this? brackets and spaces and everything?

/path/to/zpscan.sh poolname [ user@domain.com ]

and as far as the Cron timing, in the web gui it doesn't look like it's possible to let it run every 15 minutes?
The closest I can get it is every hour?

Also any safe way to test the drive led functionality?

 

[danb35]

it doesn't look like it's possible to let it run every 15 minutes?

You should be able to enter */15 for minutes.

Also any safe way to test the drive led functionality?

You should be able to offline a disk through the GUI and wait. Once it picks up on the error, online the disk again.

 

[isopropyl]

You should be able to enter */15 for minutes.

Ah didn't realize the fields accepted the syntax like that. Awesome.

/root/bin/failled-supermicro/zpscan.sh PrimaryPool [ user@domain.com ]
I'm running into a error when running. /root/bin/failled-supermicro/zpscan.sh: /sbin/zpool: No such file or directory

 

[Redcoat]

but as I don't use CORE any more

I guess I've been asleep at the wheel here - how long have you been transitioned to Scale?

 

[danb35]

I guess I've been asleep at the wheel here - how long have you been transitioned to Scale?

I'd say about six months or so. Not entirely smooth sailing.

 

[Redcoat]

Thanks. Are you tempted to summarize the experience at some point for the benefit of us luddites?

 

[isopropyl]

Thanks. Are you tempted to summarize the experience at some point for the benefit of us luddites?

Even though it is off topic, I would be interested in this as well. It seems like it is inevitable I will end up migrating in the semi-near future.
Also did you make a script for the leds to work on SCALE?

 

[Redcoat]

Yes - off topic question - my apologies.

 

[danb35]

It's hard to give a good summary, but here are some thoughts that come to mind. Keep in mind, I'm a home user, local users only, no AD/LDAP directory service.

• The basic file-sharing services are grossly comparable to CORE. They're the same services, they work in the same way, and I don't notice a particular performance difference, though it's not like I've run any benchmarks. I use SMB and NFS, and was using iSCSI for a while.
• SCALE doesn't use all RAM the way CORE does; in my system that means about 30 GB is wasted at any given time. My ARC hit ratio is still pretty good (it's averaging over 99% over the past week), so I guess this isn't a critical thing, but it does seem wasteful.
• I'm not seeing the 10-15% CPU use at idle from k3s that others have reported, but my server has dual Xeon E5s, so...
• The "killer app" that got me to switch to SCALE was the apps, particularly TrueCharts' implementation of them with Traefik/Ingress (AFAIK, there's nothing comparable with iX' official or community apps). That covers a feature I asked for in FreeNAS almost 8 years ago, and does it better than I'd even thought to ask for at that time--reverse proxies are really easy to implement with Caddy, but this way they're automatic (although I haven't yet figured out whether it's possible to have it get certs from my local CA rather than from Let's Encrypt). But it hasn't been entirely smooth sailing:
  • The upgrade from Angelfish to Bluefin completely lost all my apps, and they couldn't be recovered. After working with both the iX folks on a bug ticket, and the TrueCharts folks on their Discord, I ultimately had to destroy the ix-applications dataset and reinstall the apps from scratch. iX' release engineering hasn't improved in the last few years. Though to be fair, I seem to be the only one who encountered this bug, at least to such an intractable extent.
  • TrueCharts is a rapidly-moving target, and there's been at least one breaking change from them that has required reinstalling all the apps from scratch too. But at least with this one, the old apps continued to run so I could back up their data before destroy/reinstall.
  • TrueCharts' support is via their Discord, which as a platform I'm not a fan of--but I've gotten prompt, helpful responses there. Others have reported some, um, abrasive personalities there though.
  • The Apps platform doesn't give anything quite like a standard jail under CORE--there's no supported way to set up something like the LXC containers I use under Proxmox (though there is a thread, and I think a resource, about a "Linux jail" that's said to work on SCALE and survive updates, and there's been some discussion with iX about making this an official feature). As a result, I've moved youtube-dl (or now yt-dlp) into a VM on my Proxmox cluster (I was having trouble making it work in a LXC), mounting storage from the NAS via NFS.
• Although iX deny this, it really does look like they've abandoned CORE (and no, the release of CORE 13 does nothing to change this impression--dropping the same middleware/GUI over a new version of the base OS, which is what I see as a user, doesn't give the appearance that they're putting active development resources into the product), putting all the shiny stuff into SCALE and not even porting simple UI fixes to CORE. You still can't get a cert using Cloudflare's DNS in CORE, but you can in SCALE. CORE still has the same old horribly broken web shell that it's had since, well, before it was TrueNAS; SCALE apparently has a working one. And bug/feature tickets against CORE are being addressed in SCALE. I'd like to see CORE continue to be an actively-maintained product, but I believe its days are numbered.
• It's probably relevant to note that I don't have ideological or technical precommitments for or against either Linux or FreeBSD. Those who do will have to factor those in accordingly.

So what would I recommend?

• If you're just looking for the core file-sharing services, CORE all the way. SCALE is catching up, but CORE is by far the more mature solution here. Migration to SCALE, if/when it becomes necessary or desirable, is a simple thing as long as you aren't using jails.
• If you want to be able to install arbitrary software into a lightweight container, CORE's jails fit the bill, and SCALE doesn't currently have a comparable feature. But the jails will not migrate to SCALE unless something significant changes.
• If you want to use your NAS as a hypervisor, SCALE is probably a better choice--KVM seems to have much broader guest OS compatibility than bhyve. Though I only run one VM under TrueNAS, and it's a Linux VM for Pi-Hole on my parents' CORE system; any other virtualization is on my Proxmox cluster (which also uses kvm).
• If you want a plugin-like experience, where you can point-and-click to install additional software, SCALE is the only option--plugins in CORE have been a dead end for a long time, and iX have finally admitted it.

Edit: I think it's probably at least generally accurate to say that SCALE is what The Release That Must Not Be Named wanted to be. I've seen Jordan quoted as saying that SCALE demonstrates that he should have pushed harder to migrate then-FreeNAS to Linux that the time of that misbegotten abomination.

 

[isopropyl]

Ah didn't realize the fields accepted the syntax like that. Awesome.

/root/bin/failled-supermicro/zpscan.sh PrimaryPool [ user@domain.com ]
I'm running into a error when running. /root/bin/failled-supermicro/zpscan.sh: /sbin/zpool: No such file or directory

Still running into this issue here, haven't been able to figure out. Any ideas?

It's hard to give a good summary, but here are some thoughts that come to mind.

Good rundown, lots of knowledge appreciate it.

sn't use all RAM the way CORE does; in my system that means about 30 GB is wasted at any given time.

Isn't the high RAM usage on Core intentional though? I thought it did not affect things.

The "killer app" that got me to switch to SCALE was the apps

Yes, "plugins" on Core are absolutely broken. Even iX addressed this and their "solution" was pretty much migrate to Scale lol.
Which is kinda absurd, seeing they advertise "plugins" so heavily on Core.. yet they are unusable.

The Future of TrueNAS Plugins is Apps

TrueNAS Plugins makes it easy to customize your NAS storage. Create a secure and scalable data storage with TrueNAS.

www.truenas.com

I had one person I know, switch to Scale, and he said while he was on Scale, there was an update that was pushed to update the containers. But they were not simply "update-able", but needed to be destroyed. Basically a "we don't support apps created before x version anymore, destroy them all and re-make".. which is kinda absurd.
Maybe that was during some early release though, I'm not sure.

Although iX deny this, it really does look like they've abandoned CORE

This is the big thing which is making me consider switching. The GUI on Scale looks better, and the Core GUI has lots of broken stuff.


I know Jails can't migrate. But beyond that, it was a pretty simple migration correct? I assume it's probably safest to take your pool offline and remount it when migrating?
It's just an in place update, correct? Just go to GUI, System > Update, and select the Scale release train. It updates and reboots, and bam?
And you can't roll back to Core correct? It does nothing to the pool though correct? So you can always pop a new SSD in with TrueNAS core running and just mount your pool on it if you dislike/have issues with Scale?