FreeNAS with Raspberry Pi. Is It Possible?

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Well, speaking as someone who builds and deploys OS images for both i386 and amd64, I can say that the differences in supporting those two platforms aren't huge, and if someone came to me with ARM hardware that was reasonably supported by FreeBSD, I suspect I would quite likely be able to support it without a lot of effort.

The problem with the low end Raspberry Pi grade hardware is that it's made to be cheap, which means you get a single crappy ethernet controller. If I could get a hundred ARM cores with a respectable amount of memory and decent networking controllers, that could be a great jail server, without a lot of the x86 hardware drama. If they can keep the cost down, there's a lot of stuff where a larger number of lower performance cores could be a winner. And if we can find the equivalent of LSI HBA for ARM, then a FreeNAS solution starts to look attractive... even if it isn't ideal for every use case.

So. The thing is, it's coming. Within ten years. Back when ZFS started about 15 years ago, the resources required, CPU and memory, were onerous because the typical large x86 server had maybe two core/two socket and 4-8GB of RAM, and boy was that EXPENSIVE. The advent of cell phones and the decline of Windows in the data center has created a ripe environment for the explosion of non-x86 architectures, and as things keep moving towards being able to define infrastructure as code, the actual platform doesn't matter as much, as long as it runs Go or Python or PHP or Java. The higher level languages hide a lot of platform quirks, so it doesn't matter as much if you're running ${particular-flavor-of} Linux or FreeBSD or whatever. That just acts as an accelerant for alternative cheaper CPU's to eat into x86.
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
And if we can find the equivalent of LSI HBA for ARM
LSI HBAs, most likely. At this point, I can't imagine a new player seriously entering the SATA/SAS market. Microchip is trying, but I worry that the SATA/SAS part is an afterthought.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
LSI HBAs, most likely. At this point, I can't imagine a new player seriously entering the SATA/SAS market. Microchip is trying, but I worry that the SATA/SAS part is an afterthought.

Possibly, but I suspect this will only work if the FreeBSD LSI driver compiles cleanly and runs correctly.

One of the problems that we are going to be running into is that the LSI 92xx stuff is at end of lifecycle. I haven't seen them available in the channel for some time, and you haven't been able to buy them from OEM's like Dell or IBM in years, at least not as far as I've seen. At some point, LSI/Avagotech/Broadcom/${nextbuyer} will notice that they are supporting drivers for a chipset they haven't sold in some time. So yes this could just move to LSI 93xx stuff for awhile, but the support lifespan here doesn't give me super-good feelings. I don't expect major efforts by LSI to port the driver to ARM, but we could luck out and maybe there aren't any major blockers.

But the other side to this is that a big part of ARM is being aimed at lower power consumption and application-specific use. We've already seen boards like the ASRock C2750D4I with 12x SATA and a focus on storage, so I wouldn't be shocked to see similar specialization in the ARM world.
 
Joined
Jul 2, 2019
Messages
648
One of the drivers to more ARM support from hardware (and software, for that matter) vendors is Apple's apparent pending announcement of moving to ARM - Apple will announce move to ARM-based Macs later this month, says report

Not that Apple has anything in the server space anymore (and back in the day there wasn't much, anyway) this could also boost options.

Of course, the move to ARM in the supercomputing area would be the biggest driver - Cray to license Fujitsu Arm processor for supercomputers.

Maybe that really aligns with @jgreco's statement "The thing is, it's coming. Within ten years." ;)
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Apple has clearly been on a path to deprecating HDD storage for years, and I do not expect new devices are likely to have HDD's, so the appearance of new high quality SATA chipsets to drive Apple's ARM-based server needs is ... unlikely.

ARM in the supercomputing arena is also probably not that helpful. The HPC guys are usually all network, and if they had to store stuff locally and wait more than 0.00000000000001 seconds for it, they're hating it, so they're probably all like "NVMe on PCIe 4+ is so damn slow". Wouldn't expect to see SATA from them either. :-/

There are already some Marvell SATA controllers that are okay out there. I don't remember the specifics, but one of the ones on the C2750D4I was stable under FreeBSD, while the other one sucked. It's just a matter of time before some manufacturer decides that they want to supply a whitebox part that becomes a QNAP/Synology-killer, and coughs out an ARM board with 2x10GbE, 12 or 24 ports of SATA, and some NVMe capability. That will be in demand if it's not too expensive.
 

colinng

Cadet
Joined
Oct 1, 2012
Messages
6
FreeNAS isn't available for non-x86 or non-x64 architectures. So I tried installing ZFS on Linux on a Pi 4, as per https://www.oshelp.co.uk/?p=204

Here are my findings:
- USB for storage is extremely unreliable. Frequent checksum errors. Frequent crashes. Frequent disks going offline after crash.
- I am speculating but maybe that is why FreeNAS Web UI does not permit creation of pools over USB drives. Even pools created by CLI are hidden from the Web UI. I support that! It prevents people from shooting themselves in the foot!
- Maybe my problem is the disks (someone more knowledgeable please confirm/deny), but I suspect that it is USB that is causing the Pi (Pi 4 4GB) to crash so badly that I cannot SSH in.
- I plug the same disks (Seagate External 5TB Backup Plus) into FreeNAS atop VirtualBox atop macOS, and do a snapshot clone (read from ZoL-created mirror, write to a single O3 (OpenZFS on OS X) disk, and I get many such messages in `dmesg`:

```
(da0:umass-sim0:0:0:0): WRITE(10). CDB: 2a 00 a4 d4 e6 30 00 00 80 00
(da0:umass-sim0:0:0:0): CAM status: CCB request completed with an error
(da0:umass-sim0:0:0:0): Retrying command
(da0:umass-sim0:0:0:0): WRITE(10). CDB: 2a 00 a4 de 9b d8 00 00 80 00
(da0:umass-sim0:0:0:0): CAM status: CCB request completed with an error
(da0:umass-sim0:0:0:0): Retrying command
(da0:umass-sim0:0:0:0): WRITE(10). CDB: 2a 00 a1 a4 b7 d8 00 00 80 00
(da0:umass-sim0:0:0:0): CAM status: CCB request completed with an error
(da0:umass-sim0:0:0:0): Retrying command
(da0:umass-sim0:0:0:0): WRITE(10). CDB: 2a 00 a5 14 88 10 00 00 80 00
(da0:umass-sim0:0:0:0): CAM status: CCB request completed with an error
(da0:umass-sim0:0:0:0): Retrying command
```

My Pi NAS (haha) was so unreliable, I christened it FlakyNAS.

I eventually built a 6 x 5TB 2.5" drive NAS using an ODROID-H2 and modified their Type 1 Case so it would hold everything neatly. After I discovered that the RealTek network driver must be upgraded to the one in FreeBSD (or you will crash under any moderate network activity), the system is now rock-solid. **Very** happy with the reliability of the ODROID-H2, and the size and price didn't hurt either!
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
- USB for storage is extremely unreliable. Frequent checksum errors. Frequent crashes. Frequent disks going offline after crash.
- I am speculating but maybe that is why FreeNAS Web UI does not permit creation of pools over USB drives. Even pools created by CLI are hidden from the Web UI. I support that! It prevents people from shooting themselves in the foot!
- Maybe my problem is the disks (someone more knowledgeable please confirm/deny), but I suspect that it is USB that is causing the Pi (Pi 4 4GB) to crash so badly that I cannot SSH in.
The findings behind the minimum spec for FreeNAS (=8GB RAM) were that system stability suffers under 8GB when using ZFS (if I am remembering and paraphrasing correctly). ... at least for FreeBSD. Possibly applies to Linux too.

I doubt that it's specifically USB that's the problem here, but we also know that most USB controllers and Marvel SATA controllers don't play well with the way that ZFS does IO.

For both of those reasons, I suspect that the Pi hardware is a poor choice (caveat: I notice there is now an 8GB Pi, so maybe a different story for that one?) for ZFS.
 

kiriak

Contributor
Joined
Mar 2, 2020
Messages
122
You can create and use pools of USB drives with the FeeNAS web GUI.
This is what I do with an old NUC, just for learning about FreeNAS and until I feel comfortable to move from my Synology to FreeNAS using proper hardware.
But it can be done and worked flawlessly until now.
Of course it is not recommended for the main pool(s) of your NAS.
I'm trying to find if it could be a valid solution for keeping backups in the form of replicated snapshots. I'm playing with this and try to find any caveats.
 

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,828
FWIW, I am about to deploy two sets of ubuntu server on RPi to serve as a
  • Sonos server (iTunes is replicated to this server and Sonos reads from it) - SMB v1 (yay) and
  • TimeMachine target for the three kids - AFP - I manually transfer the data periodically to FreeNAS
The Sonos server is a throwaway. It limits Sonos' ability to wreak havoc with the iTunes archive, which took a long time to rip. I have little to no faith re: Sonos and security plus my local FW is locked at 8.4 to keep the CR100 around. While my sonos gear cannot reach much of the sonos mothership (everything but sslvalidator.sonos.com is blocked to allow radio use on TuneIn), I cannot discount that there is a backdoor somewhere. After all, my Pi-holes are reporting hundreds of attempts to reach the mothership each day.

The time machine server is another risk-mitigation strategy - allow the kids to back up their CPUs but on a separate server from which the data is periodically synchronized. That way, their machines do not have a key to the FreeNAS, cannot even reach it (different VLAN).

Some leftover mSATA and NVME modules plus bcache may even make these units quite responsive. For the time being, the Sonos is being served just fine from a 2.5" spinner hanging from an Airport extreme. Speaking of which, that Airport is a RISC processor whose USB port can deal with printers and SSDs/HDDs.

I have no doubt that ARM will enter into the server industry eventually. Especially for storage-oriented solutions, the current Intel architecture is frequently overkill, yet runs hot, uses a lot of power, and costs a lot to boot (unless you buy used). Major data center operators like MS, Apple, LinkedIn, Google, etc. are already specifying custom boards to minimize the cost of ownership (for example: heat-tolerant stuff to allow the data center to run warmer).

It is conceivable that Apple may even be one of the "pushers" for ARM use in their iCloud data centers as a means of lowering acquisition and operational cost, especially after they have already made the lift of adding PCIe ports to their "Pro" machines in a few years' time. From there, it's only a hop, skip, and a jump to adding a HBA chip or two and you have a storinator / drive45 motherboard. Unfortunately, it's doubtful that Apple will give back to the open source community in the form of drivers, etc. For all we know, even the HBA chips will be Apple-only proprietary stuff that no one else can source, just like some of the Intersil power chips on their MacBook Pro motherboards.

I do think that Apple will continue to support SATA for the first or second generation of ARM motherboards - either to allow add-on drives inside their Pro machines, or externally to allow bulk storage.
 

Pen1

Cadet
Joined
Oct 18, 2020
Messages
1
Maybe there is a way to use Freenas on Rpi 4... Xen Project has now officially ports hypervisor to Raspberry Pi 4. FreeNAS support Virtualization so it is
possible to install Freenas on hypervisor and run it.
Ok raspberrys has only 8Gb ram, but i have successfully use my Virtualizated FreeNAS on only 3Gb ram.
 

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,828
For me it would come down to answering why this is a good use of your time. If this is a hobby, a cool quest (like running Doom on a modified pregnancy test kit), etc then enjoy the journey! :smile: To me, the complexity of FreeNAS/TrueNAS is about bitrot-Proof long-term storage. In that context, to me, it makes sense to stick to platforms that sport ECC memory and other hardware ZFS mainstays.

FWIW, the RPi storage servers for Time Machine and Sonos are working great on the Pi’s. They can handle 50MB/s writes, which is 2x or more what a time machine backup usually sends, and the Sonos players only run on fast Ethernet (ie cannot possibly read more than ~10MB/s ea). So for what they are (inexpensive, potentially disposable storage platforms) Raspberry Pis are great.

But I wouldn’t use them as a FreeNAS server due to the lack of built in SATA ports. Yes, you can work around the issue by adding USB-SATA port multiplier, etc but as @jgreco has noted, such chips have a sordid history in FreeNAS / FreeBSD land due to driver issues. To me, the reward is not worth the risk.

Instead, I’d try to find a mini ITX ARM board that has a storage focus with solid server hardware (ie not a Realtek Ethernet Chip, for example). That could be a real game changer re: bringing a performant hardware platform to the FreeNAS universe that is low in power consumption, heat, and potentially cost. Same challenge (Ie making FreeNAS run on ARM) but a much bigger potential payoff to the user base and iXSystems.
 
Last edited:
Joined
Jul 2, 2019
Messages
648

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,828
Last edited:

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
instead, I’d try to find a mini ITX ARM board that has a storage focus with solid server hardware (ie not a Realtek Ethernet Chip, for example). That could be a real game changer re: bringing a performant hardware platform to the FreeNAS universe that is low in power consumption, heat, and potentially cost. Same challenge (Ie making FreeNAS run on ARM) but a much bigger potential payoff to the user base and iXSystems.

I've been hunting for a Gigabyte MP30-AR1 for a while (UEFI-based X-Gene board) but I can't seem to find them, and I wasn't about to stump up the $550 or so when they released, because at those prices I could buy a whole X9/X10 Xeon setup, let alone just a board/chip.

If I can find a used one though I'm going for it, it's got potential as an ARM-based solution (8 core, 45W TDP, 8 RDIMM slots, a pair of PCIe 3.0 x8, and onboard 10GbE) but I imagine I'd be spending a lot of time building kernel/mods from source tarballs.
 

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,828
That thing looks like a unicorn! Gigabyte doesn't even advertise it on its server main page anymore - just AMD and Intel stuff shown there now. Curious they'd include a SD card slot but not more SATA ports for something allegedly "server". I wonder what its original intended market was - billboard display driver?
 

ilmarmors

Dabbler
Joined
Dec 27, 2014
Messages
25
Latest Rasberry Pi Compute Module 4 IO Board has PCIe Gen 2 x1 socket. Compute Module 4 you plug into IO board maxes out with 8GB RAM. OS could be installed in 8GB, 16GB or 32GB eMMC Flash memory (depending on variant)

Jeff Geerling tried to plug in video card into that PCIe socket. Socket is meant more for things like NVMe drives and SATA adapters. Jeff is compiling testing results of PCIe cards that could be pluged into RP4 IO board. See Raspberry Pi PCI Express device compatibility database. Most relevant to this thread is IO Crest 4 Port SATA III PCIe x1 with Marvell 9215 that fully works with driver and currently tested IBM ServeRAID BR10i / LSI SAS3082E-R SAS RAID controller.
 

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,828
I still don't quite understand the appeal of a $150 motherboard that lacks many of the basics needed to allow ZFS to perform its intended function.
  1. RAM is lacking, it's not ECC either
  2. Form Factor appears non-standard (good luck finding a case?)
  3. A SATA HBA with a Marvell Chip. What possibly could go wrong? (@jgreco may have an opinion on that one)
Look, I get the "because it's there!" appeal to making TrueNAS work on officially-unsupported platforms but just because you can shoehorn TrueNAS into a given platform does not make it suitable for that application.

For $150 you can get a proper older motherboard ($30) with a Intel NIC, decent CPU ($32), a heat sink for same, and another $60 for 32GB of ECC RDIMM memory. All that fits into a standard PC case, accepts a standard ATX power supply, and has oodles of standard PCIe slots to plug standard LSI (Not Marvell) HBA cards into, perhaps a 10GbE NIC, etc. Overall purchase cost, likely lower than the RPi solution, far greater expansion capacity, likely much better performance, better hardware, albeit at a higher operational (power) cost.

But if the journey is more important than the end-result, I'll be the first one to cheer you on. Not because this version of the RPi is a suitable TrueNAS platform (it doesn't meet critical performance criteria in my book) but because you made it work. Calling for someone else to do the work is easy, getting it to work is usually a lot harder. (Or in my case, likely impossible). Good luck!
 
Last edited:

Dan Tudora

Patron
Joined
Jul 6, 2017
Messages
276
But if the journey is more important than the end-result, I'll be the first one to cheer you on.
hello
I am second !! :wink:
I wait for some "clusterware/TrueNAS SCALE/Beowolf" Pi. My son have a new tablet with 16 Gb RAM and 32 GB storage and want to join in that cluster
 
Top