-
Important Announcement for the TrueNAS Community.
The TrueNAS Community has now been moved. This forum will now become READ-ONLY for historical purposes. Please feel free to join us on the new TrueNAS Community Forums
Google Coral TPU support
- Thread starter StanAccy
- Start date
This is/will be a pain. Is there nothing we can configure that says "after you update, please install the following packages" ? A config file that gets added on automatically after updates?
If you can script it, there is Init/Shutdown Scripts section in System Settings > Advanced
But it's best to make a jira suggestion for official support.
jgreco
Resident Grinch
- Joined
- May 29, 2011
- Messages
- 18,680
FreeNAS/TrueNAS is an appliance, and you're not really supposed to be installing packages in the host system. The idea is for you to use jails or virtual machines for additional processing tasks.
overeat
Dabbler
- Joined
- Aug 31, 2021
- Messages
- 20
I have the same problem. I bought a Coral TPU for Frigate NVR system, which I use via multiple docker images to do AI computer vision.
I recently updated to BETA2 and this uninstalled some of my changes for Google Coral Drivers and such.
While TrueNAS Scale is an appliance, I personally want some tweaks on the base system to persist. I like having screen & vim on all machines I SSH into, including my TrueNAS Scale box. Also it appears I need to install Google Coral TPU drivers on the base system, since it's a PITA to modify docker/helm deploys to integrate this.
With that said, I've recently wrote this script which installs Google Coral TPU (mines m.2 card in PCIe adaptor) drivers which then can be used in Frigate. This doesn't require a reboot, as mentioned in the Google Coral install documents, as I load the driver via modprobe and it seems to work for me.
Code:
#!/bin/sh echo "deb https://packages.cloud.google.com/apt coral-edgetpu-stable main" | tee /etc/apt/sources.list.d/coral-edgetpu.list curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add - apt-get update apt-get install gasket-dkms libedgetpu1-std sh -c "echo 'SUBSYSTEM==\"apex\", MODE=\"0660\", GROUP=\"apex\"' >> /etc/udev/rules.d/65-apex.rules" groupadd apex modprobe apex udevadm control --reload-rules
I can't seem to get my coral to show up correctly in Truenas Scale no matter what I do. I can get it configured on a pi and a windows machine no problem, but Truenas is not cooperating. It just shows up as "1a6e:089a Global Unichip Corp." and not the correct "18d1:9302 Google Inc." device, even after installing drivers on the base system. Any advice?
As an update: Ive got Frigate running in a docker container inside an Ubuntu 20.04 VM running on Truenas Scale 22.02. You have to add the Coral as a PCIe passthrough in the VM config, then it becomes available to the VM, then the docker container can pick it up. Add the Coral drivers (see Coral docs on how to install those and load the modules). Inference speed is reported around 8.2 - so I think that's pretty good given the additional VM and docker. I noticed a Frigate Application in the incubator channel of TrueCharts, but I couldnt get that to work - it bombed out on initialization/config. Ive currently got 6 cameras running through the Coral, and Im blown away by the object detection ability of this tiny PCIe board. Now I just need to figure out how to leverage the ZFS aspect of TrueNAS without going through the ext4 of the VM.
good stuff. I might just continue using my current solution until the frigate truecharts app starts working. I did fork the incubator/frigate app and removed the coral (edge-tpu) requirement in the config and got it to install but after install it gets stuck on deploying, haven't had time to dig into that yet.
I've also fixed this issue in the meantime, if I run a "demo" example from the docs after installing the pycoral API on the Truenas base system it then shows up in lsusb as the correct "18d1:9302 Google Inc." device
How do you passthrough the TPU to Frigate?
I managed to install drivers for the TPU on my TrueNAS Box:
Code:
root@truenas[~]# lsmod | grep apex apex 24576 0 gasket 114688 1 apex root@truenas[~]# lspci |grep -i tpu 05:00.0 System peripheral: Global Unichip Corp. Coral Edge TPU root@truenas[~]# ll /dev/apex* crw-rw---- 1 root 120, 0 Mar 7 17:22 /dev/apex_0
I shared this device with my Frigate Docker-Container and configured Frigate to use the TPU:
Code:
detectors:
coral0:
type: edgetpu
device: pci:0
coral1:
type: edgetpu
device: pci:1
But Frigate complains that no TPU was found:
Code:
[2022-03-07 19:29:55] frigate.edgetpu ERROR : No EdgeTPU was detected. If you do not have a Coral device yet, you must configure CPU detectors.
Yes, indeed (and I'm sorry for not sharing this information).
When you run a container in "Privileged Mode", ALL devices will be passed to the container. Frigate is then able to access the TPU.
But I don't know if this is the best solution. I would feel more secure if I could only pass this specific Device to Frigate.
Hi,
I have the USB version. Thanks for the script but is it also ok for usb version?
because when i try to check with
It's blank
I have the USB version. Thanks for the script but is it also ok for usb version?
because when i try to check with
It's blank
I have a similar setup with a dual TPU card; my backordered TPUs just arrived. I've installed it and run the steps from @j0rd's script, after chmod on apt. I see what I assume to be two TPUs in the output of `lspci |grep -i tpu` and `ll /dev/apex*`.
Can I use the Incubator Truechart for Frigate with this? This is the first time I've ever run TrueNAS SCALE and the second time I've ever tried to install Frigate. (Never finished the first one.)
Can I use the Incubator Truechart for Frigate with this? This is the first time I've ever run TrueNAS SCALE and the second time I've ever tried to install Frigate. (Never finished the first one.)
This seems to work if you first:
Code:
chmod +x /usr/bin/apt*
and
Code:
chmod +x /usr/bin/dpkg*
However, it complains that apt-key is deprecated:
Code:
# "Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8))."
Note: the following is based on some googling and my own experimentation - if someone spots an error in it, please correct me!
Note 2: All the commands need to be run as root
Note 3: As mentioned earlier in this thread, all of this is unsupported, and will not persist Truenas upgrades (which is why it is scripted ;) )
Note 4: After enabling apt-get, do NOT install any updates that are suggested! This will likely break your NAS. It is an appliance, and all these workarounds to install drivers onto Truenas are hacks!
Anyways, to fix this and get rid of the apt-key warning - which I suppose is valid since apt-key is deprecated for a good reason:
Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead
I was installing elasticsearch following this guide, but elasticsearch is not really the part of this question. In the first step, I need to add the key: wget -qO - https://artifacts.elastic.co/GPG...
stackoverflow.com
What commands (exactly) should replace the deprecated apt-key?
Setting up keys for a local repository on a new ubuntu 20.10 virtual machine, I got a message that apt-key add was deprecated and I should read the apt-key(8) man page. The apt-key(8) man page is a
askubuntu.com
If you have already tried @j0rd 's script, first list your existing apt keys:
Code:
apt-key list
Find the key that looks like " [ unknown] Rapture Automatic Signing Key (cloud-rapture-signing-key-2022-03-07-08_01_01.pub)"
look at the 40 character key id, e.g.
"A362 B822 F6DE DC65 2817 EA46 B53D C80D 13ED EF05"
take the last 8 characters removing the space between:
e.g.
13EDEF05
and run:
Code:
apt-key del 13EDEF05
Thereafter, you can run a modified version of the script (just start here if this is the first time you are installing the coral driver):
Code:
#!/bin/sh mkdir /etc/apt/keyrings echo "deb [signed-by=/etc/apt/keyrings/coral.gpg] https://packages.cloud.google.com/apt coral-edgetpu-stable main" | tee /etc/apt/sources.list.d/coral-edgetpu.list wget -O- https://packages.cloud.google.com/apt/doc/apt-key.gpg | gpg --dearmor | sudo tee /etc/apt/keyrings/coral.gpg > /dev/null apt-get update apt-get install gasket-dkms libedgetpu1-std sh -c "echo 'SUBSYSTEM==\"apex\", MODE=\"0660\", GROUP=\"apex\"' >> /etc/udev/rules.d/65-apex.rules" groupadd apex modprobe apex udevadm control --reload-rules
This places Google's gpg key into a file in /etc/apt/keyrings folder called coral.gpg, and references it only from the corresponding source.list
/etc/apt/sources.list.d/coral-edgetpu.list
thus avoiding cross-signing with other gpg keys.
The script might need yet another modification when Debian moves to Deb822 format using individual .sources files instead of .list files.
The result should look like:
Code:
# modinfo apex
filename: /lib/modules/5.15.79+truenas/updates/dkms/apex.ko
author: John Joseph <jnjoseph@google.com>
license: GPL v2
version: 1.2
description: Google Apex driver
srcversion: 700E8BBBE9CC23C6EC17712
alias: pci:v00001AC1d0000089Asv*sd*bc*sc*i*
depends: gasket
retpoline: Y
name: apex
vermagic: 5.15.79+truenas SMP mod_unload modversions
parm: allow_power_save:int
parm: allow_sw_clock_gating:int
parm: allow_hw_clock_gating:int
parm: bypass_top_level:int
parm: trip_point0_temp:int
parm: trip_point1_temp:int
parm: trip_point2_temp:int
parm: hw_temp_warn1:int
parm: hw_temp_warn2:int
parm: hw_temp_warn1_en:bool
parm: hw_temp_warn2_en:bool
parm: temp_poll_interval:int
# modinfo gasket
filename: /lib/modules/5.15.79+truenas/updates/dkms/gasket.ko
author: Rob Springer <rspringer@google.com>
license: GPL v2
version: 1.1.4
description: Google Gasket driver framework
srcversion: 2CA68DA0268ABC8C7117109
depends:
retpoline: Y
name: gasket
vermagic: 5.15.79+truenas SMP mod_unload modversions
parm: dma_bit_mask:int
# lspci -vvv | grep -e Unichip
04:00.0 System peripheral: Global Unichip Corp. Coral Edge TPU (prog-if ff)
Subsystem: Global Unichip Corp. Coral Edge TPU
The 04:00:0 part gives you the PCI slot number that you need to expose to a VM - in a sort of reverse notation. In the UI, 04:00:0 should show up as pci_0000_00_04_0 .
At least for me, Scale's UI only showed a few of all the PCI devices once it populated the list (note that you need to give it a minute when opening the "PCI Passthrough Device*" dropdown), and it was missing the 04:
To get around this, instead of choosing from the populated list, just start typing 04 (or what ever your PCI number happens to be) in the Search box:
Choose the one that most resembles in notation the ones that are populated automatically if nothing is typed.
This seems to pass the device correctly into a VM.
If you want to go via the Truechart route, just select the "Render Configuration" checkbox right after the MQTT options in the chart configuration, click on "Add", and fill a name (can be anything I suppose), choose "Edge TPU" as the Type, and type "pci" in the Device.
If you have several coral boards, choose one for each detector:
detectors:
coral1:
type: edgetpu
device: pci:0
coral2:
type: edgetpu
device: pci:1
Or mix usb and pci:
detectors:
coral_usb:
type: edgetpu
device: usb
coral_pci:
type: edgetpu
device: pci
Last edited:
I just setup the incubator chart - see my post above. Works fine, inference dropped to 1/10th of when using CPU.
PS: If someone has figured out a way to map a hostpath to /config inside the app, please share.
Tried this in the chart config, but it did not work (/config still comes from .../ix-applications/k3s/kubelet):
I guess I have to try again starting from scratch, since I did not add the additional storage during initial install.
That seems to be the case.
Is there any way to be able to modify the /config/config.yml directly with an editor, outside or inside the container?
Last edited:
