Adventures in TrueNAS SCALE on QNAP Hardware

[trionic]

Thank you for the replies and the very helpful suggestions.

When I posted, I did not own a QNAP HX1688 but decides to take a risk and buy one just to try TrueNAS. So it arrived today. I prepared a TrueNAS SCALE boot USB and a (temporary for testing purposes only) installation USB, installed TNS, booted the QNAP and it works! Can't believe it. I'll need to have to a proper play and add some disks but it's a good start.

The QNAP is fairly well built but the hotswap disk caddies are all plastic and not even good plastic (ABS) so they're a real weak point. It does feel a big step down from my server-grade 4U chassis with Supermicro motherboard etc but this QNAP represents a different set of compromises to the big server (actually it's a 4U server with another 4U JBOD expansion chassis, plus a 4U fileserver - all to be replaced by QNAP + TNS).

The thing is quiet! There's an optional 10Gb Ethernet card installed with this really small LOUD fan so I disconnected that! Then the unit is quiet. We'll see how it is when it's full of disks and working hard.

I have two Samsung SSD 980 M.2 NVMe cards to put on the QNAP's motherboard for the permanent install of TNS.

I chose TrueNAS SCALE because I thought that there was a better chance of support for the QNAP hardware. My FreeNAS server of course is FreeBSD based and I will miss BSD. Maybe I'll give TrueNAS CORE try to see if that'll work.

I'll play around and see what happens!

 

[Zero95]

Thank you for the replies and the very helpful suggestions.

When I posted, I did not own a QNAP HX1688 but decides to take a risk and buy one just to try TrueNAS. So it arrived today. I prepared a TrueNAS SCALE boot USB and a (temporary for testing purposes only) installation USB, installed TNS, booted the QNAP and it works! Can't believe it. I'll need to have to a proper play and add some disks but it's a good start.

The QNAP is fairly well built but the hotswap disk caddies are all plastic and not even good plastic (ABS) so they're a real weak point. It does feel a big step down from my server-grade 4U chassis with Supermicro motherboard etc but this QNAP represents a different set of compromises to the big server (actually it's a 4U server with another 4U JBOD expansion chassis, plus a 4U fileserver - all to be replaced by QNAP + TNS).

The thing is quiet! There's an optional 10Gb Ethernet card installed with this really small LOUD fan so I disconnected that! Then the unit is quiet. We'll see how it is when it's full of disks and working hard.

I have two Samsung SSD 980 M.2 NVMe cards to put on the QNAP's motherboard for the permanent install of TNS.

I chose TrueNAS SCALE because I thought that there was a better chance of support for the QNAP hardware. My FreeNAS server of course is FreeBSD based and I will miss BSD. Maybe I'll give TrueNAS CORE try to see if that'll work.

I'll play around and see what happens!

This is amazing news! especially buying a ots unit and just going for it - I’m another h1288x looking to be free of QuTS after all the shenanigans. After doing a clean backup I’ll have to give this a try as well! thanks so much for sharing your insight!

 

[trionic]

Thanks! It was as easy as falling off a log. The only thing that doesn't work is the LCD display but IIRC there's a fix for that. Weirdly, the drive activity lights did not work but after about the third reboot they did.

Just follow the TrueNAS SCALE instructions and it should work. I did not disconnect the DOM on the QNAP's motherboard but (for testing purposes) just installed a USB drive into one of the rear USB ports to contain the installation. I have two Samsung 980 M.2 NVMe cards here which will be going onto the QNAP's motherboard for permanent installation. I'll also upgrade the memory to at least 64Gb (I have already seen TrueNAS eat all the unit's 32Gb memory once data begins to add up and the drives get busy).

I have been playing around with this now for several days and I have tried all sorts of things: removing disks upto and beyond the redundancy limit, thrashing the drives and hardware as much as I can, installing VMs and the thing's solid.

Yes, the HX1688X is proprietary hardware and it's unlikely that I'll be able to upgrade it and it's expensive for what it is. The drive trays are cheap and the case is a bit "tinny" (but that can be fixed with some sound deadening) but it represents a set of compromises which are acceptable to me right now. The thing is so quiet and the drives run coolish (30 to 42C). I have eight spare 4TB Western Digital Reds in there at the mo while I play around with it but it'll be fully populated with 12 Western Digital UltraStar HC550 18TB drives in (probably) two VDEVs for a usable capacity of 111TB. I am not such a data hoarder these days.

The rig that the HX1688X is replacing is two 4U 24-bay rackmount chassis that I built back in 2014, one with 24 Western Digital Red 4TB drives and the other is a JBOD expansion chassis with 12 Western Digital 3TB Greens and Reds, running FreeNAS 9.2.1.7. The thing is LOUD and HOT (drives always >40C) and I am fed up with it. The HX1688X will have similar storage capacity, similarly robust NAS software. Hopefully the HX1688X will manage to keep all 12 drives cool enough so that I can run it 24/7.

I also run another 4U rackmount chassis with Windows 7 on it, with about 8 drives of varying capacities and three optical drives. I need to get a storage tower to connect to the HX1688X and an additional USB controller so that I can pass the lot through to a Windows 7 (or maybe 10) VM on TrueNAS and then I can get rid of that 4U chassis too.

All this lot is in my home office and it'll be a relief to simplify down to just the single QNAP box. If I had a basement or small spare room then all this stuff would have been in there and out of earshot and I'd have kept it, but it's just not suitable for a small home office.

Let me know how you get on with the install :)

 

[NugentS]

I just put 16GB into my TS853A (QNAS) - which is only meant to support 8GB. But 16GB seems to work.

 

[emsicz]

Found this thread through search -- the ticket https://ixsystems.atlassian.net/browse/NAS-112401 is marked duplicate and closed. I'm on TrueNAS-SCALE-22.02.2.1 and the problem is still present. Bonded NICs only work as fast as one of them.

 

[elvisimprsntr]

Found this thread through search -- the ticket https://ixsystems.atlassian.net/browse/NAS-112401 is marked duplicate and closed. I'm on TrueNAS-SCALE-22.02.2.1 and the problem is still present. Bonded NICs only work as fast as one of them.

It was never fixed, and I gave up all hope of it ever working.

 

[Patrick M. Hausen]

I don't 100% understand the problem even when reading the JIRA issues. A single flow will always max out just one port. You are aware of that? Are you using iperf with multiple threads?

 

[emsicz]

I don't 100% understand the problem even when reading the JIRA issues. A single flow will always max out just one port. You are aware of that? Are you using iperf with multiple threads?

I have 3 boxes on the desk. 1 is running Scale with bonded NICs and the other two are trying to push data into the first one. 1 client is TrueNAS Core trying to run replication task on it's datasets and the other is a Windows 11 PC pushing large files to Scale through SMB share on the Scale box. Would that count as multiple threads?

 

[Patrick M. Hausen]

Yes, but there is no guarantee that they will be balanced across both links. It might be the case that the hash computation gives the same result (odd/even) for both so they use just one link.

You can try to configure a strictly layer 3 hash/balance algorithm and use one system with an odd and one with an even IP address in the last octet. Then use tcpdump on TrueNAS to observe what's happening on the individual links.

 

[emsicz]

So I have invested some time into reading about how the networking abstraction layer works in Linux, also this was quite helpful. It is currently my understanding that the link aggregation or bonding is not supposed to guarantee to make the thing work twice as fast during file transfer, or during multiple file transfers even. There is no guarantee about how the NICs are actually utilized during receiving or sending out data, methinks. I had two other boxes (OLD1, OLD2) that my new NAS (NAS1) needs to ingest all of their data from. Only the new NAS1 has 2 NICs. This is a one-time migration operation. Here is what I did:

1. All boxes are on the same subnet, and yes, all their NICs are on the same subnet and they all have DHCP assigned IP addresses (I hope @jgreco doesn't get a stroke reading this).
   1. A personal note, TrueNAS SCALE will not permit you to have more than 1 DHCP-enabled NIC, but yet it is it's own default setting - both NICs have DHCP checkbox checked. I know it doesn't have a choice out of the box, but me being a software developer, it just burns my insides to have it implemented this way. It creates confused users as to why was it permitted by default and later on they can't revert to this state by clicking the DHCP option, because once they change this setting, the UI won't let them go back unless they clear their entire network config. Nothing in the UI tells them this. UX-wise this is an antipattern.
2. BOX1 uploading to IP1 on NAS1/NIC1.
3. BOX2 uploading to IP2 on NAS1/NIC2.

I did get 2 times the upload speed in total, reliably, throughout the whole migration. But when that migration was done, I threw the two NICs into LACP and then I put that into a Bridge so VMs can see the host also.

The implementation is quite buggy on this and I don't mean my Ubiquiti router taking too long to distribute assigned addresses, ARP tables or anything like that. The sure-fire way of doing this is to first bond the NICs together, restart the whole TrueNAS, create empty bridge, restart TrueNAS, assign the LACP to the bridge, restart TrueNAS. I could never get it to work with just setting up the networking config the way I wanted, pressing Apply, then testing the changes after applying that entire transaction at once. It would never work, the box would be inaccessible on the network, it would not even lease an IP, I would not even see it in the network overview in the router UI, nada. I wasn't just waiting the 60 seconds, too - the thing has IPMI, so I committed the changes always, even if it meant the connectivity will break and observed how the network behaves, so it wasn't a question of not having enough time to re-lease addresses or restart the networking stack. But splitting the same task into smaller ones and doing restarts along the way, I ended up with the same thing, but working. Maybe this just needs some more testing thrown at it.

Also, it would be super helpful if the UI or the console-wizard gave me the MAC address of the interfaces it's creating for LACP and Bridge. It would make it easier for me to set up the DHCP reservation lists before these interfaces are created and enabled, instead of waiting for the communication to stop working, refreshing router client table, and copypasting the observed MAC address into DHCP reservation list, creating the record and then restarting TrueNAS or router or both to force it to release the IP address.

 

[naderelos]

Hi all, seeking advise and knowledge,

Has anyone been able to get the Disk LEDs to work?

I am testing the waters of truenas scale on a QNAP TS-253D . Although not a deal breaker, it’s usefull seeing disk activity.

 

[naderelos]

I had to grab QNAP-EC and also the .deb package for Fancontrol to get the fans working on my TS-653D, with the Bios/UEFI on smart the thing was hitting 103c cpu core temp under load (Apparently critical is 105)

Will need to re-apply after an update.

Code:

sensors
qnap_ec-isa-0000
Adapter: ISA adapter
fan1:        2311 RPM
fan2:        2243 RPM
fan3:        2257 RPM
fan33:       1161 RPM
fan34:       5108 RPM
fan35:       12140 RPM
fan36:       1222 RPM
temp1:        +25.0°C
temp6:        +33.0°C

acpitz-acpi-0
Adapter: ACPI interface
temp1:        +82.0°C  (crit = +95.0°C)

coretemp-isa-0000
Adapter: ISA adapter
Package id 0:  +83.0°C  (high = +105.0°C, crit = +105.0°C)
Core 0:        +83.0°C  (high = +105.0°C, crit = +105.0°C)
Core 1:        +83.0°C  (high = +105.0°C, crit = +105.0°C)
Core 2:        +83.0°C  (high = +105.0°C, crit = +105.0°C)
Core 3:        +83.0°C  (high = +105.0°C, crit = +105.0°C)

Code:

# Configuration file generated by pwmconfig, changes will be lost
INTERVAL=10
DEVPATH=hwmon0=devices/virtual/thermal/thermal_zone0 hwmon2=devices/platform/qnap-ec
DEVNAME=hwmon0=acpitz hwmon2=qnap_ec
FCTEMPS= hwmon2/pwm1=hwmon0/temp1_input
FCFANS= hwmon2/pwm1=hwmon2/fan3_input+hwmon2/fan2_input
MINTEMP= hwmon2/pwm1=45
MAXTEMP= hwmon2/pwm1=75
MINSTART= hwmon2/pwm1=150
MINSTOP= hwmon2/pwm1=0

@dragonn any chance you can detail the steps you used to for this?

 

[naderelos]

@dragonn any chance you can detail the steps you used to for this?

Just to be more clear, I can install “QNAP-EC” and “lm-sensors”, I have even configured the fan with “pwmconfig”. I’m more interested in cleanly persisting all of this on reboot.

 

[jgreco]

A personal note, TrueNAS SCALE will not permit you to have more than 1 DHCP-enabled NIC, but yet it is it's own default setting - both NICs have DHCP checkbox checked. I know it doesn't have a choice out of the box, but me being a software developer, it just burns my insides to have it implemented this way. It creates confused users as to why was it permitted by default and later on they can't revert to this state by clicking the DHCP option, because once they change this setting, the UI won't let them go back unless they clear their entire network config. Nothing in the UI tells them this. UX-wise this is an antipattern.

So the fundamental problem here is that it is unsupported to have multiple ethernet interfaces on the TrueNAS host attached to a single broadcast domain. For the purposes of setup, however, if you have a host with (for example) two or four ethernet interfaces, you want the appliance to pick up SOME usable address for management, so if you happen to attach ethernet #3 to a switch instead of ethernet #1, the appliance will pick up DHCP and provide you with a management address.

Now, the problem is, if you didn't have all four interfaces configured for DHCP, that wouldn't be able to happen. And of course you can't go back to this state, it's a bootstrap state. Once the administrator can reach the appliance, the administrator is expected to provide a correct networking configuration, and -- now don't have a stroke -- multiple DHCP is not a correct networking configuration, regardless of whether it is on a single network or even on multiple networks. So there would be no reason to allow someone to specify such a configuration. It's inherently broken, and the appliance is designed to guide you towards workable configurations.

UX-wise, the design is correct. It could be more correct if a little robot arm could be made to shoot out of the NAS's butt to unplug the superfluous and IP-violating additional ethernet interfaces, but so far we haven't gotten there, and the software prohibition against misconfiguration seems sufficient, except where we run into someone who is really Absolutely Cocksure That They Are Right And The Standards And RFC's Are Wrong.

If you have a more correct suggestion about how to implement this, feel free to make a suggestion. The two obvious candidates, first is only running DHCP on the first ethernet in the system, and the second is allowing an incorrect provisioning of multiple DHCP via the GUI, have both been discussed to death and are both wrong for reasons previously explained.

 

[naderelos]

Posting a QNAP-EC package to install (rather than having to clone the source code and then building it). I went through the forum rules and didn't see anything that says I shouldn't do it, so here it is:

1. Firstly run this command: `sensors`
2. You should see temperatures but no fan speeds
3. Download the attached `qnap-ec_1.1.2-1_amd64.deb` package into your NAS
4. Run the command `sudo apt install ./qnap-ec_1.1.2-1_amd64.deb`
5. I you get an error to do with `apt`, then run this command to enable it `chmod +x /usr/bin/apt*` and then repeat step 4.
6. Run this command `sudo modprobe qnap-ec check-for-chip=no`
7. Run `sensors` again and with any luck, you should see fan temps.

NOTE: if you restart, then you will need to run `sudo modprobe qnap-ec check-for-chip=no` again. I am note sure if you will need to reinstall this package on firmware updates (anyone with some knowledge is welcomed to give feedback).

I built this package myself and so let me know if this failed for you as I have only tested this on my QNAP TS-253D. (post the error if there was one). There is a package post install script to run the modprobe command but it isn't executing after install and haven't debugged it yet, hence why you have to run it manually.

I am still playing around with a truenas script for this and fancontrol at the moment but my unix script foo is non existent at the moment.

 

[ChrisD.]

Has anyone recently done this? Will a 5.0.1 ZFS pool import into TrueNAS?

 

[trionic]

For anyone considering buying a QNAP HX1688X, be aware that the chassis is unbelievably noisy due to cabinet resonance from drive and fan vibration. As I type this, the QNAP in a room on the opposite corner of this 4-bed house can be heard rattling away. This is with 12x Western Digital UltraStar HC550 18TB drives. Some improvement could be had by installing sound-deadening on the panels.

This is a combination of the lightweight chassis and harshly vibrating drives. Quite different from the old 4TB WD Reds I used to run, which were silent in comparison.

Very noisy IMO for what is intended as a domestic/small office NAS chassis.

I still have the old 4U server and may go back to that, as now I am in a new home and could put that into a cabinet in the detached garage. The thicker steel and generally heavier-weight chassis would absorb a lot of he drive vibration. Plus, I'd re-gain IPMI on the old Supermicro mobo, which I miss. I am glad I never did sell it.

 

[Zoltar]

Good afternoon, I would also like to kick qts out of my house. Aside from the vulnerabilities and hardcoded credentials, I've been putting up with their bad software since the early builds of version 4 when there was a problem with the video codecs I think it was and they removed codexpack. I don't know how it behaves on other devices but mine has become garbage at all levels. The fact is that I have been thinking about changing it for TrueNAS for a long time, but I have not dared due to my limited technical knowledge, but the last straw is that they have disabled an old TwonkyServer license that I bought as a result of the problems that I have commented, they have left it useless. I have read all the threads but I have doubts about which SATADOM I should buy to change it, use Core or Scale and which version. My model is a bit old, the TS-453PRO with a TR-004 expansion box, and I can't find which eUSB it takes exactly, although I've already read that I need a 16Gb one and better if it's SLC instead of MLC. I've found these: http://www.altec-cs.com/Distribution/IndustrialStorage/USB/DoM/SLC and

Could someone recommend which one to look for and where I can buy it (EU)?, I can't find it in any online store. And which is better to install, Core or Scale, and which version. The most current or an older one better? Thank you so much.

 

[elvisimprsntr]

@Zoltar

Welcome!

Most QNAP units use a 512MB Apacer eUSB DOM, which is not large enough for TrueNAS (16GB minimum). The OEM eUSB DOM is 512MB, SLC, +5V, 10-pin (1 key), 2.54 mm pitch, 90 degree, with mounting hole. (data sheet attached)

The data sheet you included in your post is for SATA DOMs, which is a completely different form factor and is not compatible with QNAP internal eUSB DOM. The link you provided in your post includes the Low Profile (LP) version, which is a 2.0 mm pitch header.

Assuming you can even source a compatible eUSB DOM, you will have to take apart the entire chassis, use a heat gun (hair dryer might work) to soften the white staking compound, use a tool to remove the staking compound, pinch the plastic riser, then remove the OEM eUSB DOM.


OPTIONS

Assuming you don't have an internal M.2 or SATA port, your options are as follows:

1. Source a compatible eUSB DOM and replace the smaller OEM eUSB DOM (difficulty high)
2. Purchase an external USB SSD enclosure and SSD. (difficulty easy)


OTHER

1. CORE vs SCALE depends on what you plan to use it for, but don't expect a full replacement for QTS features (transcoding, etc.)
2. TrueNAS does not support expansion chassis like the TR-004

 

[Zoltar]

Good morning, Thank you very much for her answer. I am very aware of the compound that holds the plate, I have colleagues who have broken it when trying to replace it. Thank you very much also for the data sheet, now I understand the specifications type A, B, C, etc... I will review it in detail to choose the correct one. Then we'll see if I can find it.
My model does not have PCI for expansion, and I only have two free bays in the TR-004 (I have already read that it does not accept it, but I only have it with a couple of disks for backup that I also extract to an external usb disk) . The option of installing it on external SSDs doesn't really satisfy me.
I have already lost the QTS features, apart from the fact that the transcoding worked very badly for a long time, and I had disabled the video and photo station, when I got angry because they did not want to accept the license that Twonky paid for, I have deleted my account and the free Cayin license has also been permanently disabled. When I bought it, it used all the functions, File Server, Qsync, Transcoding, Video and Photo Station, Virtualization, Containers, iSCSI Storage, Backup of mobile devices, etc... the whole family used it, but over time I've gone disabling options due to poor performance, and I've had it for years without internet access, which makes it work even worse, QNAP has excelled.
Given the great experience that you seem to have, I would like to ask you a question. I had planned to replace the NAS with a more modern model, or mount a new chassis with SSD or NVMe, but due to economic problems in the end two WD failed and I changed the four disks. Currently I use it as Storage videos and photo family, FileServer, MediaServer, QSync for the PCs and mobile devices, and iSCSI Storage. If I manage to find the eUSB, and even if it's installing an older version of TrueNAS, will I notice the performance? I understand that apart from losing the transcoding I also lose the hardware acceleration of the raid, right? Thank you very much for your time, and for all the information in the forum, it is very helpful.