Advice on a TrueNAS device

[robbrown99]

Hello, I'm not a TrueNAS/FreeNAS user yet, but I having been running an OpenZFS zpool on my Mac for the last 6 years. I have been mostly successful, but have suffered a couple of hiccups, which is making me think I should be considering a self contained NAS instead to avoid such problems again. Hence the look at TrueNAS.

Issues I've had so far with OpenZFS on OSX:

1. When I upgraded from Yosemite to El Capitan in 2017, I had some permissions issues
2. Last week one of my drives (two 1.5TB disks, mirrored in a single zpool) threw a bunch of errors. During the process of resilvering, the process stalled and both drives ended up erroring and now won’t mount. I had to backup from an HFS+ drive.

With all of the above, I am getting fed up worrying about support between OpenZFS and Apple OSX. I’d like to stop worrying, and get something that will just manage my data. I'd also like a flexible solution that doesn't care which operating system I run. So if I want to run Windows, or Linux instead, I can just do that.

My use cases are:

1. For storing my large photo (and a very small number of videos) collection as the primary disk storage, and having fault tolerance / bit rot protection for that. Ideally here I need decent speed, and for the next 5 years I’m figuring I need approx 6TB storage to cover that use case. I'm currently just under 1.5TB. My computer is an old 2009 Mac Pro. At some point I want to ‘upgrade’ to a new Atom based Mac Mini.
2. I'd like a really easy way of sucking the images from my camera off my SD card, onto the NAS, rename them and copy up to flickr. Not sure if there are plugins that allow this?
3. There’s a growing collection of computers in my house, so I’d like to have an easy method of backing those up by giving access to a NAS over wireless. I would need maybe 4TB storage to cover that.
4. I don’t yet do plex or anything, but the ability to do such things is intriguing to me. However, I get most of my content from online services, and don’t have a movie collection, so it would be just for the odd TV show recording here and there. Very light usage! No idea how much I’d need for that. 2TB?
5. I’m toying with the idea of having a simple surveillance camera setup to record a couple of weeks of video at a time from 2-3 cameras. No idea how much for that. 1TB?
6. I'm toying with running my website and own email off the NAS also, because why not? It costs me $90 a year to host elsewhere (but maybe my electricity bill would go up)?
7. My son is getting into gaming, would be fun for him to set up a server on the NAS if that's possible (e.g. minecraft, steam)

Question 1 - With regards to networking setup, I really want to have good performance between my Mac Pro and the NAS. So ideally I can connect directly to my mac from the NAS via 1G ethernet, AND separately also connect to the NAS to the router. However:

• my 2600AC Synology wireless router is upstairs (unfortunately) in the middle of the house for good coverage. My photo editing computer is downstairs in my office. I’m ok with just normal wireless for the other computers in the house to backup, but that photo editing computer needs a direct connect I think. My current Mac only has n capability, not ac.
  I guess therefore my options for connecting my NAS to the network are:
  i) put NAS in back of router upstairs. Ethernet over power line from office to router. Wireless to all rest of devices.
  ii) put NAS in office, plug into my computer via RJ45. Get a wireless mesh expander and plug that into the NAS. Synology sell a mesh hub that connects with my main wireless router.

Question 2
I’m confused by all of the RAID options though. RAIDZ2 looks interesting for performance and some protection. I also assume I need 5 drives for that. I would probably be going with Seagate Ironwolf red 6TB disks. I bought 2x of them, so now I’m committed I think, so guess I would just buy 3x more.

Question 3: For the above use cases, I'm figuring a TrueNAS MiniX may be a good fit? It has >1 1GB ethernet (shame it doesn't have 10G), has 5 bays.

Question 4: compared with my direct attached drives today (2x mirrored disk directly into motherboard), I assume I will be seeing a performance hit over 1GB Ethernet, to a NAS... but how bad? 2x slower? 4x slower? Should I instead be considering going with a simple large single disk in my computer for main storage, then a NAS for backup? Put it this way - does any use their TrueNAS directly attached to their computer over 1GB and actually enjoy editing photos from it?

Thanks for any tips

 

[Nick2253]

Based on your use case and concerns, I think a dedicated NAS makes a lot of sense.

Q1:

Networking everything as you want might be tricky. Given the limitations of the location of your router and your desktop, I'm not sure how to easily fix that. Powerline ethernet is terrible in my opinion. If you can do MoCA (ethernet over coax), that is significantly better. In theory, you can definitely direct connect your Mac Pro to your NAS at the same time as having both connect to the main network, though it sometimes can cause oddities, especially if your Mac really wants to connect via the main network instead of the direct connection. In my opinion, it's always better to keep the connection to one network if possible.

As such, I'd recommend the following: get a small 4 port gigabit switch for your office, and directly connect your Mac Pro and your NAS to that. If you can make MoCA work, use MoCA to bridge the connection from your office to the upstairs router. If MoCA won't work, then get a mesh hub for your office, and connect that into the switch. Now, your Mac Pro and NAS can communicate directly through the switch, and you'll all be on one network. If you go the mesh route, you might put your router in the office and the mesh node upstairs o that you don't have to get a separate switch for the office.

Q2:

RAIDZ isn't super performant. For a photo library, it's probably plenty, but stripped mirrors are the best performance option. RAID-Zx gives you "x" drives of redundancy, or perhaps from a different perspective, against "x" number of simultaneous failures/losses. Therefore, RAID-Z2 gives you protection against two failures, which might take the form of lost drives or unreadable sectors, etc. Technically, you only need 1+x drives, so for RAID-Z2, you could have just 3 drives (but if you only had three drives and wanted x=2, you'd be better off with a 3-way mirror). Realistically, most people don't consider RAID-Z2 until they get to four drives.

In your case, 5 drives in RAID-Z2 would be fine. Depending on your backup strategies and appetite for failure, RAID-Z1 might be fine for you as well. If you go with 5x 6TB in RAID-Z2, you'd be looking at ~15TiB of available space. With ZFS, you generally want to keep at least 20% free (because ZFS is a copy-on-write file system), which gives you a total of ~12TiB usable.

Q3:

The MiniX is a great system. However, you may be able to get more bang-for-your-buck with a DIY system or a repurposed used system. For example, you can buy an older Dell PowerEdge T320 server with 8 hot-swappable bays for around $300-$400 shipped on eBay, or even cheaper if you're patient.

At least older Mini systems used to have an expansion PCIe slot, but according to the specs on the website, only the X+ and XL systems offer an expansion option. If you really want 10G, I would probably recommend a DIY or repurposed system to give you that expansion capability.

If you're serious about adding more VMs and services to your system, I would no go with the Mini. Its relatively weak Atom processor is great for a NAS, but not well suited to much more. In this case, I would definitely recommend a repurposed used system.

Q4:

Photos, even gi-normous RAW photos, aren't all that big. Depending on exactly what you're doing with them, I would guess that your photo editing software will load the entire photo to memory, so any performance issues would be limited to opening or saving a photo.

1 Gbps (little b "bits") is equivalent to about 100 MB/s (big B "bytes") transfer speed. Even for some of the largest RAW photos around 100MB, you're talking 1-ish seconds to transfer that. Even if that's 10x slower, it's probably not all that noticeable, because, again, it's only happening during opening or saving.

If you go into more detail about your photo workflow, I might be able to give you a better answer. But I wouldn't think even a "slow" RAID-Z2 array is going to be very noticeable.

 

[robbrown99]

Hi Nick2253, thanks so much for the thoughtful resonse.

1. MoCA - I had not heard of that, but makes a lot of sense. I do have cable connnections in my office and of cours where my wireless router is. I will look into that.

2. RAIDZ2, yes I think this path makes sense. I really like the idea of fault tolerance across more than one drive given my last issue.

3. Thanks for the tip on that Dell T320. I guess that would be a good cheaper option, and more performant too. Only two downsides to that in particular:
- size (I'd really like something more compact, but I guess if I'm getting rid of the Mac Pro some day, something similarly sized is fine
- power consumption - I like the idea of the low stand-by power consumption of the 26W of the mini X, but to be fair I should really just look at that Dell. I assume the Xeon processor is quite power hungry when running, or on standby.
- noise, would like something quiet.

I will look at similar devices for ideas of off the shelf boxes that are pre-built. You have got my gears turning on this one. I guess I also have the option of Unraid too, but I'm still assessing the diffs of that vs TrueNAS SW (it appears I can run Unraid with ZFS via a plugin)

4. My usage for photos is:
- generally to download approx 20-500 photos in one go from an SD card. So write performance needs to be fast enough that this just gets done quick so I don' get frustrated
- on my mac I have a script that runs, renames files, edits some metadata and then a flickr plugin automatically uploads. I'm fine doing this from the Mac still, it works.
- editing photos - I tend to use a RAW editing program like Lightroom, or Luminar. That library is stored on my SSD for speed. The photos are on my ZFS pool. In the program, they get read each time I click on one if there's not a preview already. When i do load them, performance tends to be in the order of a couple of secs for RAW decode and display, or sub 1 sec for JPEG. They are approx 20 megapixel files.
- Occasionally I do other editing on files but rarely so.

All this is done via a 2 disk zpool mirror, directly inserted into the motherboard. So it is fairly snappy.

I also do a very tiny fraction of viewing videos, but really not interested in it. I don't have any movies on my computer or anything.

Now, that's ME. My son is becoming a computer nerd, and may be interested in more heavy lift stuff like recording TV shows etc. I haven't crossed that bridge yet. He may also want to set up gaming servers etc, so that's coming some day soon I think.

 

[danb35]

Technically, you only need 1+x drives, so for RAID-Z2, you could have just 3 drives

I'm pretty sure you need 2+x drives at a minimum with parity RAID--otherwise you just have a mirror. This would mean that four disks would be the smallest possible vdev size for RAIDZ2. In any event, five disks is fine.

 

[Nick2253]

Yeah, if you go with a repurposed system, there's really no way around the size. If you go DIY, you could get down into the Mini ITX or Micro ATX form factors which allow for extremely compact systems, but that's just basically what the Mini family is.

As far as power use goes, once you get to Sandy Bridge or newer, your idle power consumption is pretty low. I have a couple dual E5 systems at work as VM hypervisors, and they idle at under 100W (though with all the VMs, the system is rarely truly idle in the way a NAS can idle). According to this review of the T320, power usage at idle is around 68W. Definitely more than the mini, but not a huge amount more. Based on a power cost of around $0.12/kWh, that's about $3.50 extra per month.

When it comes to noise, it usually goes that the larger the footprint, the quieter the system. This is because you can use larger fans, have more interior space for ducting and passive cooling (which means you can run fans slower), and add more noise-dampening materials to a system. I can't directly comment on the noise of either system since I have no personal experience with them, but I would guess that the T320 will be quieter.

Regarding Unraid vs TrueNAS, as you might guess, I'm biased towards TrueNAS. The biggest difference, in my opinion, is that TrueNAS is open source, and Unraid is not. This means a couple things: (1) you have to pay for Unraid, which significantly decreases the community size, which means fewer users providing help, providing guides, and providing bug reports; and (2) you have no way to audit what Unraid is doing with your data, since you can't dig into the source yourself (or perhaps, more importantly, security auditors can't dig into the source). TrueNAS is also playing in the enterprise space, which speaks to its robustness and reliability: both key features for me when I'm looking at data storage. Unraid is more focused on the prosumer category, and directly competes with the likes of QNAP and Synology units.

Based on your photo workflow, I would think a RAID-Z2 NAS would be plenty fast enough for you. You'll probably see slight increases in latency when browsing through photos, and running your SD card dumps might take a bit longer, but I wouldn't imagine that it would be that meaningfully slower. The biggest slowdown is likely when you first connect to the NAS after the connection has gone idle, as your Mac negotiates the new network connection to your NAS. I definitely "feel" that delay, but once I'm connected, everything goes very smoothly.

 

[robbrown99]

Great, thanks Nick!

The detail here is very useful, thanks again. The extra power concern may not be to worry about then. I guess unless I start adding graphics cards etc. I really like the idea of starting small, then expanding. I'm interested in lots of things, including maybe running VMs for OSs on that eventually, but I'm conscious I dont want to get ahead of myself and over-complicate things.

I think the way to go for me is going for a mini-ITX case with a bunch of bays, a reliable motherboard that has expansion options for graphics cards, or PCI based SSDs etc, and a CPU that has some oomph for some headroom (e.g. 8 cores), and some built in graphics on-board (e.g. AMD have some chips I think).

I have been looking at builds on youtube and looks like there are some nice cases like this Silverstone https://www.amazon.com/dp/B07PCH47Z2/?ref=exp_bytemybits_dp_vv_d with easy access drive bays.

Ultimate p2000/Ryzen Powered Micro-Unraid server build guide!

This is a step-by-step guide on how to build a small, but super powerful Unraid/Plex Media Server! Powered by an NVIDIA Quadro P200 and a Ryzen 3700x.Find mo...

www.youtube.com

The comment about TrueNAS being open source (vs Unraid) is interesting, and a stand-out for me. I need to explore the TrueNAS ecosystem a little more, and maybe download TrueNAS core and get it running on a drive in my Mac Pro is a good place to start there.

Thanks for the comments, appreciated.

 

[Nick2253]

I have that Silverstone case, and I wouldn't necessarily recommend it. My drives regularly overheat, and I've had to extensively modify the case in order to improve airflow.

Easy-access bays are really nice in some cases, but for a typical home use, they are overkill. If you have to shut down your home system for 10 mins to remove and replace your drive, that's not costly downtime in the way it is for an enterprise use-case. If I were to go back and do it again, I would get one of the Fractal Design cases.

 

[sremick]

Easy-access bays are really nice in some cases, but for a typical home use, they are overkill. If you have to shut down your home system for 10 mins to remove and replace your drive, that's not costly downtime in the way it is for an enterprise use-case. If I were to go back and do it again, I would get one of the Fractal Design cases.

That was my logic too. 2 failed drives replaced and 6 drives upgraded later, and I still don't regret my choice. I got a Fractal Design case.

Back when I built this system, it had to live in the living room so size and noise were big concerns and design constraints. Now I have wired network into the basement, so if I rebuild it then I'll go big to open up my options. I might go with external drive cage access then, but it's not a priority.