FreeNAS Hardware Guide (Up-To-Date)

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
I got semi-advance notice, but I haven't read the whole thing yet. What I have read is positive, though.
In related news, I'm working on an update to mine, with the help of a few people at iX. I probably won't have the time to sit down and finish it today, but I'd like to get it done tomorrow.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681

If you have any suggestions for this guide, then please comment here.

s/right of passage/rite of passage/

It feels a little devoid of information. For example, there's a well-reasoned guide to proper power supply sizing here on the forums, because so many people were coming in and saying "Oh but this random Internet site said I only needed a 300W PSU for my 12 drive NAS." There are cases especially where people want to be able to spin down drives, where ZFS can try to spin up a whole bunch of drives simultaneously, and the idle current calcs that most people seem to think are sufficient for sizing a PSU turn into a brownout.
 
Joined
Jul 2, 2019
Messages
648
Nowhere near the experts here, but here are some thoughts for consideration:
  • Section "Storage Device Considerations" - Maybe add a link/reference to the BackBlaze hard disk failure reports? I'm not endorsing BackBlaze but the reports are a good reference
  • Under "Central Processing Unit (CPU) Selection" - It is not clear, to me at least, what is meant by "Watch for VT-d/AMD-Vi device virtualization support on the CPU and motherboard to pass PCIe devices to virtual machines." This could be taken as do not have the capability or you need to have the capability.
  • Under "Remote Management: IPMI" - May want to add a caution on the older IPMI boards that use Java. It can be a royal pain if you are not using Oracle Java, using Chrome, etc.
Noting that I have already said I am nowhere near any type of FreeNAS expert - it might be useful to note that a good level of technical competence is needed with FreeNAS. FreeNAS is not a simple application (that is why there is QNAP, Synology, even OMV :p). If you have spent a significant amount of time running FreeNAS it is easy to forget your learning curve (with almost 30 years in all aspects of IT, I have to remind myself of this often when talking to others..) - it is often not obvious to those asking the questions. Maybe this is something for the forum sticky post: Here is what you should not do! ;)
 
Joined
Jul 2, 2019
Messages
648
I just realised that there is nothing here on backup. While not technically part of the a "Hardware Guide" there needs to be some type of reference around the need to back up outside of FreeNAS. "RAID isn't backup!" :) Maybe a guide on backup (I feel "dirty" making this suggestion as I am not the person to write it... :rolleyes:)
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
I feel "dirty" making this suggestion as I am not the person to write it... :rolleyes:)

... Why?

Not being the person to write it is meaningless when someone is literally soliciting feedback and comments on the things they have written and where it has come up short. I have wrestled with this frequently over the years, trying to balance a tendency to be overly thorough (which might put off beginners) with trying to write in a manner that is accessible. Your feedback is probably THE MOST IMPORTANT.
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
I might also need to grab 11.3-RC1 and deploy it somewhere to see if behavior has changed or not, but I'd like to delve into the mess of SLOG and sync write behavior.

Here's the section in question:

A SLOG device need not be large as it only needs to service five seconds of data writes delivered by the network or a local application. A high-endurance, low-latency device between 8 GB and 32 GB in size is adequate for most modern networks, and multiple devices can be striped or mirrored for either performance or redundancy. Paying attention to the published endurance claims of the device is imperative, since a SLOG will be the funnel point for a majority of the writes made to the system.

It is also vital that a SLOG device has power protection. The purpose of the ZFS intent log (ZIL), and thus the SLOG, is to keep sync writes safe in the event of a crash or power failure. If the SLOG isn’t power protected and its data is lost after a power failure, it defeats the purpose of using a SLOG in the first place! Check the manufacturer’s specifications to ensure the SLOG device is power safe or has power loss/failure protection.

I think that there needs to be a section here on highlighting what exactly the difference between "sync" and "async" writes are, as they're kind of just dropped into the conversation without any real introduction - yes, it's brought up in terms of "NFS and databases" but there's nothing referencing another major use of them which is when backing virtualization infrastructure, and nothing that really denotes that async writes aren't guaranteed to survive a sudden outage. I recently wrote a forum post explaining this to another user here:


And there's also the SLOG benchmarking/information thread from js_level2 here:


Which I've been meaning to go through, pull data, and rewrite to a v2.0 of the resource.

Regarding the sizing of the SLOG device, unless something has changed very recently in OpenZFS, the rule is no longer "five seconds of data writes" but has to do with the zfs_dirty_data_max tunable, exposed in FreeBSD as sysctl vfs.zfs.dirty_data_max and defaulting to either 1/10th of your physical RAM or 4GB, whichever is smaller. So without adjustment, no one will use more than 4GB of their SLOG device - although the GUI might still be creating a large swap partition on log vdevs, which is a separate issue (that I think has a bug open in Jira.) Adjustments do exist, but are likely outside the scope of a general "hardware guide." A wording about the general relationship between SSD sizes and performance would be beneficial as well - as size goes up, speed tends to as well, but finding a device that's purpose-built for a write-intensive environment is best. A small 100GB "write intensive" SSD will likely crush a 512GB "general purpose" or "consumer" one.

Expanding on the "low-latency" part of the device is also important, as most users are surprised when their "Super L33T Gamer Edition" SSD rated to max out the SATA bus turns in a poor showing as an SLOG. A word about hunting for the performance specs at single or very low queue depths would be advised, since those numbers tend to reflect the diskinfo -wS "benchmark" results as well as real-world SLOG speeds.

With regards to the power protection piece, while an SLOG technically only needs to provide power-loss-protection for data at rest, those without power-loss-protection for data in flight (aka a "non-volatile write buffer" or other marketing terms) will obviously perform much better as SLOG devices, since they'll be able to effectively "ignore" the request to flush bits to stable storage - the controller knows it can consider the write buffer as stable, so it doesn't need to. But it's vital that if a device doesn't have PLP for in-flight data that it doesn't lie about it.

Integrating the diskinfo -wS benchmark into the FreeNAS webUI might be beneficial; call it a "simple SLOG benchmark," give the red-screen warning that it will destroy the data on the drive, and provide the results from 4K-128K in a bar or line chart.

Afterthought:

Keep in mind that for every data block in the L2ARC, the primary ARC needs an 88 byte entry; this can cause the ARC to fill up unexpectedly and actually reduce performance in a poorly-designed system. For example, a 480GB L2ARC filled with 4KiB blocks will need more than 10GiB of metadata storage in the primary ARC!

Don't the L2ARC headers also benefit from compressed ARC? I swear I've seen lower usage than the quoted 88 bytes/record.
 

joeinaz

Contributor
Joined
Mar 17, 2016
Messages
188
I just realised that there is nothing here on backup. While not technically part of the a "Hardware Guide" there needs to be some type of reference around the need to back up outside of FreeNAS. "RAID isn't backup!" :) Maybe a guide on backup (I feel "dirty" making this suggestion as I am not the person to write it... :rolleyes:)

Backup of the system could be in one of several formats. Ideally, the backed up storage is or can be stored offsite. Some options include:

1. Backup to an attached USB device.
The benefit is it's the easiest option to deploy, cheaper for smaller FreeNAS installations and easily removable.
The challenge here is with large FreeNAS arrays getting a disk big enough in a USB enclosure.

2. Backup to another NAS (could be FreeNAS) device.
The benefit is the ease of deployment and a single NAS can backup multiple network storage devices.
The challenges here are the cost, and the fact that a NAS device is usually at the same location and not easily removable.

3. Backup to tape.
The benefit of tape is tape still the cheapest storage medium and is easily removable offsite.
The challenges are the cost effectiveness of deploying tape for a small NAS and restoration of data in terms of time and complexity. (unless LTFS is deployed)

4. Cloud based backup
The benefit of a cloud based solution is it's inherently offsite and it can be easy to deploy and manage for a casual user.
The challenges are the cost, and the performance of both backing up and restoring.

5. A combination of any of the aforementioned methods.

Having a backup strategy is key and need is directly proportional to the value of your data. If necessary, we can get into the specifics of each solution.
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
I don't want to discount the need for a backup, it is one of the first things on my list, but is that part of a "Hardware Guide" document?

A "User Guide" or "Administrator Guide" might be a better place for it, although some discussion of the hardware involved would belong in the hardware guide.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
I don't want to discount the need for a backup, it is one of the first things on my list, but is that part of a "Hardware Guide" document?

A "User Guide" or "Administrator Guide" might be a better place for it, although some discussion of the hardware involved would belong in the hardware guide.

"You should also make sure you acquire all the hardware for your second FreeNAS system, or better yet, the FreeNAS Mini is an awesome solution for a ready made backup target!"

Seems hardware guide-y enough to me.
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
Don't the L2ARC headers also benefit from compressed ARC? I swear I've seen lower usage than the quoted 88 bytes/record.
That doesn't seem right. The blocks exist in ARC as they do on disk, which is what makes the whole thing viable in the first place, with no additional compression. Might you be looking at memory compression by the OS (which is fairly popular these days)?
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
That doesn't seem right. The blocks exist in ARC as they do on disk, which is what makes the whole thing viable in the first place, with no additional compression. Might you be looking at memory compression by the OS (which is fairly popular these days)?
No, I mean the behavior of ZFS to compress its own metadata (including the L2ARC headers) with LZ4, which would further reduce the memory overhead.

But for large L2ARC sizes you should consider the option of just building an all-flash pool.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
But for large L2ARC sizes you should consider the option of just building an all-flash pool.

Pricing on that is still way off. Let's build two basic pools:

Pool 1) -- Black Friday pricing of shucked WD 12TB drives is $180. Current pricing of 1TB SN750 SSD is $150. Total cost for 6TB of 50%-rule usable datastore space is $510, with up to 1/6th of it readable at NVMe SSD speeds.

Pool 2) -- Cheap 1TB SATA is $100/TB. If you do RAID1 and follow 80%-rule, you actually need 16TB raw space, or 16 * $100 = $1600. This is more than 3x the cost of the HDD option, but you do actually get full SSD speeds.

As a note 6 x 1TB cheap SATA is $600, no data protection, still more expensive than pool 1!

Depending on your workload, lots of people will have working sets smaller than 1/6th of the pool size, so HDD is still incredibly attractive there. A full install of FreeBSD 12.1R (defined as including sufficient basic ports, source, and sufficient space to build world) is 30GB, but the active image there is only about 10GB, and the working set is probably less than 1GB.

Now it's easy to forget the other half of this, which is that Pool 1, when filled to 6TB, has 6TB *more* space available. The 50% rule says that you will start hurting yourself if you actually use it on an ongoing basis, but that can still be handy in a crunch. By comparison, Pool 2 only has 7.5TB total, so when filled to 6TB, it really has very little extra space.

I mean, yes, I totally see that it makes sense to ask the question, but I think for many the numbers still won't make sense. We're definitely getting closer though!
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
Right-sizing a pool is getting a little off-base from "review the contents of the hardware guide" though, let's diverge into another thread if we want to continue. I think we can agree though that each problem needs a tailored solution that considers both the desired performance and available budget.

I do appreciate that the impact of L2ARC on primary RAM has been identified and quantified; even if the impact is being overstated slightly it helps end-users realize that they can't simply strap a 1TB cache drive onto a pool with 8GB of RAM and expect everything to turn up roses.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Right-sizing a pool could arguably be within the scope of the hardware guide, because if nothing else there are a large number of people who are perplexed by the 80% limit, and a HUGE number of people who have no clue about fragmentation and only filling a database/VM/etc pool to 25-50% and needing mirrors to do so.
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
Since this came up as a likely root cause to a user's issue today - how about a bullet-point (or a 72pt flashing neon bold italic underlined header?) stating that the use of SMR drives should be discouraged?
 
Joined
Dec 16, 2018
Messages
2
I apologize if this has been answered elsewhere (I have searched and didn't find anything) - but once I have built my FreeNAS machine (it is working well), can I remove the dedicated gpu (the CPU does NOT have built in graphics), or does FreeNAS require a GPU at all times? Thanks!
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
FreeNAS doesn't, but your system might not be happy booting without a GPU.
 
Top