Re-using old workstation for FreeNAS, but not sure I'm clever enough

[Bikerchris]

Hello all,

Hope you're safe and well, wherever you are in the world.

Please don't feel obliged to read this, it's just me thinking aloud in case anyone that has knowledge can contribute thoughts!

I've been meaning to transfer over to a more reliable OS for storage for some time, but work/family has always taken priority. I've never had data loss as I've had online backup as well as duplicate files on multiple hard drives, but the day will come if I keep up my current ways. Using plain old windows as a sort of file server, I have had minor data corruption, such as photos etc., but fortunately so far nothing critical.

I would like to give FreeNAS a go and intend to buy a few used hard drives to have a play with it, but if it goes well, below is the spec of my spare workstation that I intend to use. I have to admit, I'm not too academic and want to dive into FreeNAS to make sure I have the mental ability to do it properly, I'm hoping that I don't need to become a FreeNAS pro' to get a stable and reliable system.

Motherboard make and model: Asus X99-E WS
CPU make and model: Xeon E5-1650 v4
RAM quantity: 2 x 8GB ECC 2133MHz Registered
Hard drives, quantity, model numbers, and RAID configuration, including boot drives Hard disk controllers: USB Drive for FreeNAS, 1 x 250GB NVMe SSD, 4 x 3TB WD, 1 old 2TB.
Network cards: Onboard 2 x 1GB
Case: A very old Thermaltake Armor (circa. 2008).

I have around 8TB of stuff, this includes disk images (no larger than DVD capacity), work and personal images, programmes, films, etc. Around 2-5GB is added per month, I use Dropbox/G Drive for regular things and work files.

Something I am wondering is how I get my data from the old, to the new, without having to buy too many more drives, but this is something I'll look into more as I get closer to doing it.

Final use of this machine will be work file server, Plex (not used before) and family machines backup.

Thanks for reading if you got this far, compliments to Level1Tech Youtube channel for promoting FreeNAS.

All the best,

Chris

 

[Inxsible]

Your board, CPU and RAM supports ECC, and it also has dual Intel NICs. So that's a good start.

I usually don't like having fluff in my server boards -- like sound cards, overclocking etc. but if you already have the hardware lying around then I wouldn't go out and buy new. Your cpu and RAM should more than suffice your stated goal of using it as a file server, Plex and backup.

You might want to get an understanding of transcoding in Plex especially if you intend to run multiple streams simultaneously.

 

[Bikerchris]

Your board, CPU and RAM supports ECC, and it also has dual Intel NICs. So that's a good start....

Hi Inxsible,

Thank you for taking the time to respond, very much appreciated.

Glad I'm starting with a good foundation of hardware.

Oh, please be assured it's been running in a clean environment for many years, so it's fluff free and an un-overclockable CPU. No sound cards and although it has a GPU in there, I'd probably get a cheap GPU that I can use just for initial config and then pull it and remote in once setup. I could always leave it in if that doesn't cause a problem? It has a severely under used GTX 1070 in there at the moment, but I'd probably sell it to pay for extra HDD's.

Thanks for the headsup with plex, I think there would only be 2-3 users at most. May be the GTX 1070 could be used for transcoding, otherwise my other main machine has a Quadro P2000 that I could re-purpose, as I've been wanting to upgrade that for some time.

Thank you!

 

[sretalla]

Motherboard make and model: Asus X99-E WS

I run that board in two of my systems and it has served me well as both FreeNAS bare-metal and virtualized FreeNAS on ESXi.

 

[jgreco]

but not sure I'm clever enough

You asked the right question. This shows you're clever enough. You can work your way through this. We worry more about the people with random desktops, or worse, laptops.

 

[Yorick]

Couple comments.

- Dont boot from USB, you’re asking for pain, as in, failed boot drive. That can be recovered from quickly with a reinstall and restore of config backup (take one daily, we have scripts), and, it’s still a pain. Get a cheap ssd - one of those 20 bucks jobs will do - and boot from that.
- what’s the intended use of the NVMe ssd? You won’t need L2ARC for Plex and backup (large files), though it won’t hurt you to have it. You certainly won’t need slog, you’re not sync writing.
- you can’t hardware transcode with an NVidia card in freebsd. Not supported by Plex. Plex exclusively supports Intel’s iGPU on FreeBSD, and that comes with some manual setup work. That said, unless you intend to transcode 4k (don’t, just keep a 1080p copy alongside), you’ll be fine with software transcode.
- Most importantly: You can’t change the layout of your vdev after the fact. You will have 4x3TB in either raidz2 (6TB usable) or raidz1 (9TB usable, real risk of data loss of another drive fails during resilver). You may consider getting another two drives and going for a 6-drive raidz2, with 12TB usable. You don’t want to go above roughly 80% use, ZFS starts to slow down when there’s not enough space. And you can’t expand the raidz vdev (*). You can add another vdev to the pool later of the same error recovery type - so you could add another raidz2 with six or more drives, of the same or higher capacity. You can also replace all drives in your raidz2 one by one, and once the last drive is replaced, you’ll have the higher capacity available for your vdev. This behavior of ZFS is good to understand ahead of time. It’s not RAID5 or RAID6, and you can’t just “add a drive” after the fact - you’d need to add an entire vdev or replace all drives in the vdev with higher-capacity ones. For reference, my smallish Plex library uses 9TiB or thereabouts, and PC backups use 6 TiB.

(*) raidz expansion exists in a very early pre-alpha build. Whether that ever becomes real and when - 2021, or a few years after that - is anyone’s guess. If this comes to pass, you will be able to “just add a drive” to raidz, but you can’t plan with it in mind.

 

[Bikerchris]

You asked the right question. This shows you're clever enough. You can work your way through this. We worry more about the people with random desktops, or worse, laptops.

That's very kind jgreco, that gives me some confidence. People don't really install it on Laptops, good grief!

 

[Bikerchris]

Hi Yorick,

Thank you very much for taking the time to reply and provide your comments and suggestions. Invaluable stuff!

Couple comments.

- don't boot from USB, you’re asking for pain, as in, failed boot drive. That can be recovered from quickly with a reinstall and restore of config backup (take one daily, we have scripts), and, it’s still a pain. Get a cheap ssd - one of those 20 bucks jobs will do - and boot from that.

Thank you, I will definitely do that - I did watch the video and noted that an SSD would be more reliable, happy to buy a small SSD, 32GB was recommended. I'm using 2 memory sticks to start with, only for 'is this beyond me or not' purposes.

- what’s the intended use of the NVMe ssd? You won’t need L2ARC for Plex and backup (large files), though it won’t hurt you to have it. You certainly won’t need slog, you’re not sync writing.

I thought it could act as a cache drive, plus I have it in the machine already, so thought I could make use of it. I do have another somewhere (lost in house move), but if I were to find it, would be be good to use 2 x NVMe's and have the FreeNAS OS on both of them, as a fall back?

- you can’t hardware transcode with an NVidia card in freebsd. Not supported by Plex. Plex exclusively supports Intel’s iGPU on FreeBSD, and that comes with some manual setup work. That said, unless you intend to transcode 4k (don’t, just keep a 1080p copy alongside), you’ll be fine with software transcode.

Thank you for letting me know that, very useful. The CPU I'm using doesn't have an iGPU, so I guess that means I can't use Plex? So sorry for what I'm sure is a naive question.

- Most importantly: You can’t change the layout of your vdev after the fact. You will have 4x3TB in either raidz2 (6TB usable) or raidz1 (9TB usable, real risk of data loss of another drive fails during resilver). You may consider getting another two drives and going for a 6-drive raidz2, with 12TB usable. You don’t want to go above roughly 80% use, ZFS starts to slow down when there’s not enough space. And you can’t expand the raidz vdev (*). You can add another vdev to the pool later of the same error recovery type - so you could add another raidz2 with six or more drives, of the same or higher capacity. You can also replace all drives in your raidz2 one by one, and once the last drive is replaced, you’ll have the higher capacity available for your vdev. This behavior of ZFS is good to understand ahead of time. It’s not RAID5 or RAID6, and you can’t just “add a drive” after the fact - you’d need to add an entire vdev or replace all drives in the vdev with higher-capacity ones. For reference, my smallish Plex library uses 9TiB or thereabouts, and PC backups use 6 TiB.

(*) raidz expansion exists in a very early pre-alpha build. Whether that ever becomes real and when - 2021, or a few years after that - is anyone’s guess. If this comes to pass, you will be able to “just add a drive” to raidz, but you can’t plan with it in mind.

Thank you for that last paragraph, very kind - with the greatest respect do leave it with me as I'll have to read it several times over before responding. I will only raise more questions once I've searched for the answer on this forum. Thank you very much!

 

[Bikerchris]

@Yorick @jgreco @sretalla @Inxsible

The test install went well at least, very impressed as it only took 15-20mins or so!

I must say something, watching the code going down the screen humbles me immensely, to think many people spent many hours creating the code that tells things what to do. If the species is capable of this, there's hope.

And I do appreciate the lack of RAM, final system will have between 16-32GB.

 

[sretalla]

Looking good. RAM is important as you note.

 

[Bikerchris]

Looking good. RAM is important as you note.

Thank you for confirming that @sretalla , much appreciated.

 

[Yorick]

I thought it could act as a cache drive, plus I have it in the machine already, so thought I could make use of it. I do have another somewhere (lost in house move), but if I were to find it, would be be good to use 2 x NVMe's and have the FreeNAS OS on both of them, as a fall back?

You can, that's called L2ARC. Only: You likely won't need it. Your drives can supply roughly 100MB/sec, which saturates a Gbit link. Since you're dealing with a few large files - Plex and Backup - additional read cache isn't going to do much, if anything, for you. Once you're up and running, you can see what your throughput is, and from there decide whether more RAM or an L2ARC is needed.

I'm running with 32GiB of RAM and no L2ARC. The 32GiB is likely overkill, and I get GBit speeds for Plex and backup without issue from the drives.

Thank you for letting me know that, very useful. The CPU I'm using doesn't have an iGPU, so I guess that means I can't use Plex? So sorry for what I'm sure is a naive question.

Oh, you can use plex, it will use the CPU to transcode. You have a powerful CPU, so I'm not worried there. You just can't hardware transcode, which means that 4k transcodes aren't the best idea. But those aren't the best idea to begin with, as HDR->SDR conversion in Plex is atrocious, so you're better off keeping a 1080p file alongside - Plex can handle that - and use software transcode for 1080p down to "smartphone" resolution for those times you're streaming to mobile devices.

 

[Inxsible]

don't boot from USB, you’re asking for pain, as in, failed boot drive.

I'd disagree -- having the same USB stick running since almost 5 years. call me lucky !!

@Bikerchris -- you would be right in assuming that you'd need more RAM. I run with 16GB and only use Emby (Plex alternative) on the FreeNAS box. The rest of my VMs and containers are located in a separate VM server. My CPU on the FreeNAS box is also just a measly Pentium G3240. So your configuration should be more than adequate to run Plex or Emby and also do backups.

 

[Bikerchris]

You can, that's called L2ARC. Only: You likely won't need it. Your drives can supply roughly 100MB/sec, which saturates a Gbit link. Since you're dealing with a few large files - Plex and Backup - additional read cache isn't going to do much, if anything, for you. Once you're up and running, you can see what your throughput is, and from there decide whether more RAM or an L2ARC is needed.

I'm running with 32GiB of RAM and no L2ARC. The 32GiB is likely overkill, and I get GBit speeds for Plex and backup without issue from the drives.

Ah, thank you for the translation. Perhaps it's best not to have it - if I fancied tinkering with a VM OS, is it still not worth having? I suppose the fewer drives, the fewer failure points perhaps?

Thank you for the reassurance with your own setup as an example.

Oh, you can use plex, it will use the CPU to transcode. You have a powerful CPU, so I'm not worried there. You just can't hardware transcode, which means that 4k transcodes aren't the best idea. But those aren't the best idea to begin with, as HDR->SDR conversion in Plex is atrocious, so you're better off keeping a 1080p file alongside - Plex can handle that - and use software transcode for 1080p down to "smartphone" resolution for those times you're streaming to mobile devices.

Thank you again Yorick, I can understand that which is a relief! I'm fairly sure I have no 4k TV's in the home, so I guess it's no problem to me.

 

[Bikerchris]

I'd disagree -- having the same USB stick running since almost 5 years. call me lucky !!

I'm calling you lucky! :) I guess this is a judgement call on personal experience, and as I've had more USB sticks fail than SSD's, I'll go with SSD for this. No offence intended at all to your config, may you have another 5 years of reliable service!

@Bikerchris -- you would be right in assuming that you'd need more RAM. I run with 16GB and only use Emby (Plex alternative) on the FreeNAS box. The rest of my VMs and containers are located in a separate VM server. My CPU on the FreeNAS box is also just a measly Pentium G3240. So your configuration should be more than adequate to run Plex or Emby and also do backups.
[/QUOTE]
That is really interesting, thank you for sharing. I'll get more RAM if funds permit or the system demands it....may not be able to add more drives, but more RAM seems FreeNAS friendly :) You certainly get a lot of use out of that Pentium, may be I'll turn a few cores off, if there's a worthwhile electric saving or reliability increase (probably not!).

 

[Bikerchris]

Important to note, so far I'm only able to install FreeNAS, I know it could be a long road to understanding. Bit like owning a book, doesn't mean you can read :)

 

[Inxsible]

I'm calling you lucky! :) I guess this is a judgement call on personal experience, and as I've had more USB sticks fail than SSD's, I'll go with SSD for this. No offence intended at all to your config, may you have another 5 years of reliable service!

No offence taken. I would have gone with an SSD as well if I had enough SATA ports available. My case supported 6 drives and so did my board. And I wanted to have 6 drives. So no port left for the boot drive. That's the only reason I chose to install on a USB. Thankfully, for me at least, it was a good decision.

As for CPU, i believe in the theory of using components to the limit -- which is why I chose a Pentium G3240 because for me it was primarily a file server & I knew I wasn't going to transcode more than 1 stream via Plex/Emby. In 5 years, I still don't have the need to transcode more than 1, so I guess in that terms my choice was sound because you should always plan your setup in such a way that it would support any and all upgrades for at least 5 years.

If you do plan to transcode more than 1 stream, you might need a beefier CPU. Check the passmark to make sure you are in the ballpark.

 

[Yorick]

Ah, thank you for the translation. Perhaps it's best not to have it - if I fancied tinkering with a VM OS, is it still not worth having? I suppose the fewer drives, the fewer failure points perhaps?

L2ARC is used for reads exclusively. It can fail and the ZFS checksumming will catch that and stop using it. So, it's entirely safe to add one.

One of the nice things about ZFS is that you don't have to guess whether L2ARC would be beneficial. For starters, you'd see whether you're getting close to Gig speeds on read when using your actual data in the way you typically use it. If so, you're golden, nothing else needs to be done.

If speeds are not optimal, then you can see what ARC is doing with "zfs-stats -a | less". In my case, you can see I have over 98% ARC hit rate. I could add an L2ARC, but it wouldn't help me performance-wise. I am also getting "close to Gig", subject to what Windows and its network drivers are doing, so I'm good.

As for VMs, there's an important point to be made about FreeNAS. FreeNAS 11.3 is based on FreeBSD 11.3, which uses bhyve as its hypervisor. It works well enough for something like a text-only (headless) Ubuntu Linux service VM - I have one running to run Foundry VTT for D&D games, for example. Smooth as butter. But if you're trying graphics-heavy VMs - Windows 10 is a prime example - you are going to have a very bad time. This might improve with FreeBSD 12.1, and it might improve if the PR for PCIe passthrough ever makes it into FreeNAS: And, don't hold your breath on that.

Rule of thumb: FreeNAS as a file storage with some light services, such as a jails and Linux VMs, perfectly fine. FreeNAS as a full-on virtualization platform with demanding VMs, particularly ones that do graphics output - not a great idea. In no way shape or form will you have a good time with resource-hungry VMs. I'd look at Proxmox, which also supports ZFS, for that "lots of heavy VMs" use case.

From what you said, your use case lends itself well to FreeNAS.

Here's the output of my ARC statistics. This is with a very simple use case. SMB file sharing to Windows; Windows PCs back up using Veeam Free Agent; Plex jail with 4k HDR and 1080p files for media streaming (hardware transcode works on my rig); LMS jail for whole-house audio with Raspi 4 and Max2Play and HiFiBerry audio HATs; Ubuntu VM to run Foundry VTT. I could run Foundry in a jail, but it's designed for and tested on Linux, and I don't want to be the odd person out trying something the dev doesn't run in their CI/CD environment.

Code:

ARC Eviction Statistics:
        Evicts Total:                           462848
        Evicts Eligible for L2:         96.46%  446464
        Evicts Ineligible for L2:       3.53%   16384
        Evicts Cached to L2:                    0

ARC Efficiency
        Cache Access Total:                     3143497
        Cache Hit Ratio:                98.32%  3090801
        Cache Miss Ratio:               1.67%   52696
        Actual Hit Ratio:               98.03%  3081670

        Data Demand Efficiency:         84.74%
        Data Prefetch Efficiency:       20.62%

        CACHE HITS BY CACHE LIST:
          Anonymously Used:             0.29%   9131
          Most Recently Used (mru):     5.18%   160145
          Most Frequently Used (mfu):   94.52%  2921525
          MRU Ghost (mru_ghost):        0.00%   0

 

[Yorick]

If you do plan to transcode more than 1 stream, you might need a beefier CPU. Check the passmark to make sure you are in the ballpark.

He's got a Xeon E5-1650 v4, with a passmark a little north of 14k. Conservatively, that's seven (7) transcoded 1080p streams. Streaming to a TV doesn't transcode, as it's native. So unless more than 7 people will watch stuff from that Plex while on the road, I'd say it's perfectly fine - and if more than 7 people do, the upstream bandwidth of the ISP link is likely to be the bottleneck much sooner than the CPU will be.

 

[Inxsible]

He's got a Xeon E5-1650 v4, with a passmark a little north of 14k. Conservatively, that's seven (7) transcoded 1080p streams. Streaming to a TV doesn't transcode, as it's native. So unless more than 7 people will watch stuff from that Plex while on the road, I'd say it's perfectly fine - and if more than 7 people do, the upstream bandwidth of the ISP link is likely to be the bottleneck much sooner than the CPU will be.

when I said "you might need a beefier CPU" I was actually comparing it to my Pentium G3240. But I can see how my statement could be misinterpreted.