SOLVED System and daily emails work correctly, email alert service fails to send

[prairiemoose]

Long-timer lurker but first-time poster here. I have run up against a wall in troubleshooting a FreeNAS email issue and am hoping for some pointers for next steps!


Overview:
- New FreeNAS install (FreeNAS-11.2-U4, Build Date: May 7, 2019 0:46)
- Email was successfully configured in System > Email using a generic SMTP service with TLS authentication
- The "Send Mail" button in System > Email successfully sends to the root user's configured email address
- The daily mailouts generated by 'periodic daily' send successfully
HOWEVER:
- The email alert service in "System > Alert Services" shows a popup with the text "Failed sending test alert!" when I use the "SEND TEST ALERT" button, and alerts are not being sent.


Hardware:
- SuperMicro 5029A-2TN4
- CPU: Intel Atom C3338
- RAM: 8 GB ECC UDIMM
- Storage: 1 x SuperMicro 16GB SATADOM, 2 x Seagate 6TB ST6000NM0115 Enterprise SATA disks


Troubleshooting steps I have attempted so far:
- Tested the Email Alert Service by triggering a real alert (pulling one of the disk drives while the system is running). A system alert appeared in the WebUI, but no alert email was sent, and no messages appeared in /var/log/messages or /var/log/maillog
- Changed the email account to Office365 using an app password - test emails send successfully, alerts do not send successfully
- Changed the root user's email to a different Office365 account - no change
- Changed the "Email Address" field of the Email Alert Service in "System > Alert Services > Edit Alert Service" - no change


Further info:
- No errors appear in /var/log/messages or /var/log/daemon.log or /var/log/maillog when attempting to send the alert service test email.
- I tested setting up a custom alert service through Slack using only a Slack webhook URL in the Slack Alert Service "Webhook URL" field - this also fails to send the alert with the same "Failed sending test alert!" message. (I have validated that a cURL call to the Slack webhook URL produces a Slack message successfully)


Screenshots:
(You will need to make the first screenshot fullscreen to see the error popup)



Based on these tests, it seems like I might be running into a deeper error or bug in the FreeNAS Alert Service subsystem that is not directly related to my email configuration.

Any pointers on troubleshooting options, logs I could check, or tests I could run would be greatly appreciated before I resort to a reinstall or bug report.

Thank you for your time!

 

[prairiemoose]

Some additional info: I located some debug output that is produced each time I use the "SEND TEST ALERT" button in the Email Alert Service config field:

From /var/log/middlewared.log

Code:

[2019/05/10 17:29:00] (ERROR) AlertServiceService.test():633 - Error in alert service 'Mail'
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/alert.py", line 631, in test
    await alert_service.send([test_alert], [], [test_alert])
  File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/../alert/service/mail.py", line 23, in send
    "text": await self._format_alerts(alerts, gone_alerts, new_alerts),
  File "/usr/local/lib/python3.6/site-packages/middlewared/alert/base.py", line 135, in _format_alerts
    return format_alerts(product_name, hostname, node_map, alerts, gone_alerts, new_alerts)
  File "/usr/local/lib/python3.6/site-packages/middlewared/alert/base.py", line 183, in format_alerts
    text += "New alerts:\n" + "".join(["* %s\n" % format_alert(alert, node_map) for alert in new_alerts]) + "\n"
  File "/usr/local/lib/python3.6/site-packages/middlewared/alert/base.py", line 183, in <listcomp>
    text += "New alerts:\n" + "".join(["* %s\n" % format_alert(alert, node_map) for alert in new_alerts]) + "\n"
  File "/usr/local/lib/python3.6/site-packages/middlewared/alert/base.py", line 195, in format_alert
    return (f"{node_map[alert.node]} - " if node_map else None) + alert.formatted
TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'

 

[dlavigne]

Thanks for posting the traceback. We've identified an issue in the alerts system and will be releasing U4.1 to address it (most likely later today if QE can finish testing the hot fix today).

 

[prairiemoose]

@dlavigne With the new update FreeNAS-11.2-U4.1 applied to my spare NAS, notifications are working as expected to both E-Mail and Slack alert services.

Thank you again!