Adventures in TrueNAS SCALE on QNAP Hardware

[elvisimprsntr]

BACKGROUND

4+ months ago, I got fed up with the QNAP QTS hardcoded credentials and vulnerabilities, and successfully installed TrueNAS CORE on all three of my QNAP systems. I was able to get all the services I need (APACHE, LACP, RSYNC, SMB, TFTP, TM) and even a Windows 10 VM running.

I ultimately ended up replacing the OEM 512MB eUSB DOMs with 16GB SLC eUSB DOMs for the boot-pool.

SCALE
I decided to take TrueNAS-SCALE-21.08-BETA.1 for a spin on a TS-453A NAS

WHAT WORKED SO FAR
1. Installed on SanDisk NVMe 1TB Extreme v2 Portable SSD
2. Importing existing CORE data pool (ZFS pool upgraded)
3. UPS slave monitoring
4. SMB/TM shares
5. RSYNC tasks
6. Control of 2x16 LCD display
7. Installed on 16GB SLC eUSB DOM
8. External USB SSD data pool (ZFS pool upgraded)
9. Email alerts

WHAT HAS NOT WORKED
1. UPS reporting missing (Known issue: https://jira.ixsystems.com/browse/NAS-110579)
2. LACP 802.3ad layer2+3 (Not working: https://jira.ixsystems.com/browse/NAS-112401)

TO BE TESTED
1. Windows 10 VM

 

[morganL]

Impressive work to transform a QNAP box ...

 

[elvisimprsntr]

Well I think I spoke too soon on one area. It appears LACP link aggregation is not working on SCALE.

Re: NAS-112401

Results from iperf3 testing show only ~940Mbps total. I tried switching to layer3+4, but the changes will not stick.

I was able to achieve close to ~1.8Gbps with CORE performing the same test.

Switch confirmed to be in LACP

Looks like SCALE is using the same virtual MAC address for both members of the LAGG interface and is not using the physical HW MAC address.

On CORE, the LAGG interface uses the physical HW MAC address.

 

[Kris Moore]

Please file a ticket on this LAGG issue at https://jira.ixsystems.com so we can investigate it for 21.10.

Welcome to the SCALE family BTW :)

 

[NugentS]

@elvisimprsntr
I have a TS-853A and have been running TrueNAS Core on it for a month or so. Works extremely well for a somewhat hardware limited platform
I boot from USB's stuck in the back of the box, and this works well.

What does not well is booting from eUSB from the header on the motherboard where I removed the QNAP widgit.
I bought an Innodisk module (16GB) eUSB and installed TrueNAS on it. It was so slow it took overnight. But did install. Booting seems to work, but I will be old and gray before the boot process finishes - I gave it 4+ hours before giving up. It was working its way through but was very very veeeerrrryyyyy slow.

So I tried a USB to NVMe drive off a USB Header cable attached to the same onboard USB header. Same issues.

Would be interested to hear about your attempts to use the USB header and a eUSB module

 

[elvisimprsntr]

Please file a ticket on this LAGG issue at https://jira.ixsystems.com so we can investigate it for 21.10.

Welcome to the SCALE family BTW :)

One step ahead of you. Filed this morning.

https://jira.ixsystems.com/browse/NAS-112401

 

[elvisimprsntr]

@elvisimprsntr
I have a TS-853A and have been running TrueNAS Core on it for a month or so. Works extremely well for a somewhat hardware limited platform
I boot from USB's stuck in the back of the box, and this works well.

What does not well is booting from eUSB from the header on the motherboard where I removed the QNAP widgit.
I bought an Innodisk module (16GB) eUSB and installed TrueNAS on it. It was so slow it took overnight. But did install. Booting seems to work, but I will be old and gray before the boot process finishes - I gave it 4+ hours before giving up. It was working its way through but was very very veeeerrrryyyyy slow.

So I tried a USB to NVMe drive off a USB Header cable attached to the same onboard USB header. Same issues.

Would be interested to hear about your attempts to use the USB header and a eUSB module

I've been running CORE on my three QNAP TS-x53A systems for 4+ months. 3.5 months off eUSB DOMs without any issues. UEFI install took a little longer (~15 min vs ~30 min) than on a USB SSD (Crucial X8), but no issues booting off eUSB DOM (UEFI)

What type of flash does your eUSB have? (SLC, MLC, TLC, QLC, etc.)? SLC is the fastest and the best write life cycle, but cost a bit more assuming you can find them. https://www.hardwaretimes.com/difference-between-slc-mlc-tlc-qlc-nand-ssds-which-one-is-better/

 

[NugentS]

I think MLC - I only bought it as a test to see if I could get something to work on the USB Header which hasn't turned out that well.

 

[elvisimprsntr]

I think MLC - I only bought it as a test to see if I could get something to work on the USB Header which hasn't turned out that well.

I haven't had any problems like you seem to have experienced. [knock on wood] I paid the premium price for SLC to avoid potential durability problems and the need to boot off a SSD in a USB enclosure.

 

[elvisimprsntr]

Installed SCALE daily with the hope link aggregation was working.

TrueNAS-SCALE-21.10-MASTER-20211007-033612

Although one can now change BOND interface settings (rate, hash policy, etc.), unfortunately still getting only ~940 Mbps combined bandwidth.

https://jira.ixsystems.com/browse/NAS-112401

 

[Trexx]

Good to see you over here @elvisimprsntr ! I haven't taken the plunge yet on my flipping my Ryzen Qnap to Scale yet as I need to find a cheap Nvidia card for it (main one was moved to SuperMicro chassis). Overall I am really liking Truenas Scale(ZFS) although their VM capabilities are still very 1.x compared to Virt Station.

 

[elvisimprsntr]

@Trexx Thanks. I deactivated my QNAP forum account. I didn't see any point keeping it, since migrating all my QNAP HW to TrueNAS CORE or SCALE.

Yeah, VMs are still a bit finicky, although I did get a Window VM working on CORE.

 

[filipya]

Any idea what it would take to get Core going on a TVS-872XT? Does it require a DOM swapout? I haven't been able to find anyone on any forums that has done this with the 872XT.

And is it possible to even play around with Core on the 872 without losing my current install? Something like remove the HDDs & NVMe, then boot of USB just to play for a while? Or must Truenas be written to the DOM? Or perhaps just disable the DOM in BIOS and boot off a USB stick?

Thanks in advance.

 

[MikeLowry]

@elvisimprsntr - Interested in installing SCALE on my QNAP 470 Pro. I have 16GB RAM with 4 x 4TB WD RED Plus. The noise issue and worsening QTS firmware would be reasons to move. SCALE would be the choice due to Debian OS.

Are you able to provide the process to replace the DOM (is there only one inside) and what model DOM needs to be purchased?

 

[elvisimprsntr]

@elvisimprsntr - Interested in installing SCALE on my QNAP 470 Pro. I have 16GB RAM with 4 x 4TB WD RED Plus. The noise issue and worsening QTS firmware would be reasons to move. SCALE would be the choice due to Debian OS.

Are you able to provide the process to replace the DOM (is there only one inside) and what model DOM needs to be purchased?

The QNAP 470 Pro has an Apacer 512 MB SLC eUSB DOM. Specs are typically 5v, 2x5 pin header, 2.54 mm pitch, horizontal (90D STD)
Apacer eUSB DOM data sheet and ICP Electronics LCD Manual are here: https://github.com/elvisimprsntr/QnapFreeLCD

To remove the Apacer eUSB DOM, you will likely need to partially disassemble the NAS, apply a heat gun to the non-conductive white staking compound and pinch the plastic stand off tabs. A hair dryer might suffice, if you don't have a heat gun.

Might be able to remove the DOM without to much disassembly.

SLC eUSB DOMs are hard to come by and can be expensive. You will need a minimum of 16GB capacity.

Industrial Embedded USB / USB EDC | Flash Storage | Innodisk

Embedded USB (eUSB) and USB EDC (USB Disk On Module) comply with high-speed USB 2.0 and 3.0 interfaces, featuring in fast data transmission and high reliability.

www.innodisk.com

Another less expensive alternative is to connect an external USB3 SSD enclosure. I ran CORE and SCALE on USB3 Crucial X8's until I was able to source 16GB SLC eUSB DOMs here in the US. Not sure about availability in AUS

If you want to kick the tires on SCALE, you can temporality install SCALE on a USB flash drive. Not recommend for a permanent installation.

 

[NugentS]

Any idea what it would take to get Core going on a TVS-872XT? Does it require a DOM swapout? I haven't been able to find anyone on any forums that has done this with the 872XT.

And is it possible to even play around with Core on the 872 without losing my current install? Something like remove the HDDs & NVMe, then boot of USB just to play for a while? Or must Truenas be written to the DOM? Or perhaps just disable the DOM in BIOS and boot off a USB stick?

Thanks in advance.

Looks to me that you have everything you need. HDMI, USB & NIC all on the motherboard.
As @elvisimprsntr says you may need to remove the DOM (I would as that way its not at risk and you can always put it back later)
Attach monitor (at front) and keyboard (don't bother with a mouse)

Attach 2 USB flash drives, one with a TN installer, the other 16GB+ Blank
Boot and enter BIOS, change the BOOT order and install TN to the blank USB flash drive

See if it works. If it does (after some playing around) then replace the "Blank" drive with a USB to M.2 (SSD or NVME) adapter and repeat as a more permanent option.

I tried a eUSB module but didn't for some reason have a lot of luck with it - it worked but was silly slow so I have stuck with 2 USB to M.2 adapters with a couple of cheap M10 optanes as a mirrored boot pool.

 

[elvisimprsntr]

Another option is to install a Mini-PCIe SSD or use the two eSATA ports on the back of the unit, if you are not using it for something else.

 

[NugentS]

but test with the USB Flash drives first - to make sure TN will run on the box

 

[Astraea]

I just converted 3 QNAP boxes to TrueNAS though I am running CORE not SCALE. I did have to set the smart mode for the fans to be disabled so they run at full power all the time otherwise they just barely spun. I also used an LCD script on one of them to allow the front LCD to work otherwise it would just say system booting forever. The models I converted were two TS-451 (with 16GB ram) and a TS-853A (with 16 GB) ram. I have installed TrueNAS on mirrored USB thumb drives (I did move the system dataset to the storage pool to minimize writes to the flash drives) and while not advised by TrueNAS works for me atm until I find a way to mount/secure some SSDs to the boxes and get some USB to SATA adapters.

 

[crkinard]

Well I think I spoke too soon on one area. It appears LACP link aggregation is not working on SCALE.

Re: NAS-112401

Results from iperf3 testing show only ~940Mbps total. I tried switching to layer3+4, but the changes will not stick.


View attachment 49446

I was able to achieve close to ~1.8Gbps with CORE performing the same test.

View attachment 49449

Switch confirmed to be in LACP

View attachment 49447


Looks like SCALE is using the same virtual MAC address for both members of the LAGG interface and is not using the physical HW MAC address.

View attachment 49450

On CORE, the LAGG interface uses the physical HW MAC address.
View attachment 49451

I noticed the issue of Scale using a virtual MAC for LACP as well but using Supermicro hardware. When I used Core the bond0 interface used the HW MAC of one of the two interfaces.