First FreeNAS build

[donk]

Hello
After a lot of reading, I thought I'd post my plan for my freenas box before buying anything.

Motherboard: Asus P10S-I Mini Itx
CPU: Intel Core i3 7300
2x16 GB Samsung 2133 MHz ECC DDR4 (From manufacturer QVL)
HBA: IBM/Lenovo M1215, to be flashed to IT mode
PSU: Seasonic Prime 850 Watt 80+ Gold
System drive: Intel 540S SSD
HDD's: 10x Seagate Ironwolf 4 TB
Case: Lian Li PC-Q26 (bought a year ago)
CPU cooler: Noctua NH-D9L (If the stock cooler is too noisy)
Fans: Noctua Static pressure optimized. (If the stock fans isn't up to the task)

I also need an UPS, any recommendations here?

The HDD's will be in a 10 disk raid-z2 vdev
The nas will only have one client (me). And it won't be running any jails, except some r-sync in the future.
It will be running encryption. The I3 7300 supports AES-NI, but will the I3 be enough? Or should i consider a Xeon?
The HBA might be overkill, but I can't find any M1015 locally (Denmark). And I don't trust the cheap ebay ones from China.
The 32 GB ram is low balling it with 40 TB, according to the official hardware guide.
But since it will only serve me, and only do storage, it should be fine?
The content it will be storing is: 70% blu ray mkv rips, 20% smaller video files and 10% pictures, documents & music.

I know encryption isn't recommended, mostly due to people being bad at managing keys, passwords etc?
But I have been running everything I own encrypted for +8 years, and have never lost data, since i backup my keys to multiple locations, like a box in my bank.

I have been going back and fourth between the Seagate's Ironwolfs, WD reds, HGST's and enterprise grade drives. But I have decided on the Ironwolfs.
I run both WD Blues and Seagate Desktops in my desktop pc. Neither WD or Seagate have failed me, but my Seagates run 5°C cooler. Which will benefit me in the noise department, since it will be placed in my room. (I live in an apartment with others)

The Freenas will be replacing my current setup which is: 2x4 tb backed up to an 8 tb external, 1x2 tb backed up to my laptop with rslsync, and a 1x2 tb black game drive. No drives is in raid, jbod. Just a lot of different drives in windows. It's a pretty annoying to manage..

Have I missed something? Or is there something I could improve on? Pleas let me know, and thanks in advance :)

 

[Chris Moore]

I also need an UPS, any recommendations here?

I guess that you need a 240 volt unit, so you will want to be sure to get the right model, but I use the APC Back-UPS Pro 1500 and have been very pleased with it. It provides plenty of run time to allow the server to shutdown gracefully if the power is out long enough for the battery to get to the 'low' threshold. The UPS service that is built into FreeNAS works great to monitor the status of the UPS and initiate shutdown when needed. You might be able to get away with a smaller model, but I went with this one to ensure there was enough run time and I have it protecting my router, network switch and NAS so that the network remains operational during any momentary outage.

The HDD's will be in a 10 disk raid-z2 vdev
The nas will only have one client (me). And it won't be running any jails, except some r-sync in the future.
It will be running encryption. The I3 7300 supports AES-NI, but will the I3 be enough? Or should i consider a Xeon?

Will you be using 1GB networking? If you are and if you are only using it for storage, that should be plenty of CPU power. My quad core Xeon idles with less than 10% utilization most of the time only spiking up to 80% when I backup my main storage pool to the external enclosure where the backup storage pool is located. Even transcoding video for Plex, it usually never goes over 50%.

The 32 GB ram is low balling it with 40 TB, according to the official hardware guide.
But since it will only serve me, and only do storage, it should be fine?
The content it will be storing is: 70% blu ray mkv rips, 20% smaller video files and 10% pictures, documents & music.

The memory should be fine for storage only. With all that video inventory, I am a bit surprised that you are not planning to run Plex in a jail.

I know encryption isn't recommended, mostly due to people being bad at managing keys, passwords etc?
But I have been running everything I own encrypted for +8 years, and have never lost data, since i backup my keys to multiple locations, like a box in my bank.

That is the biggest thing that kills a storage pool with encryption, not being able to unlock it after a reboot, and it is almost always down to key management. If you replace a drive in the pool, for example, you need to generate a new key and recovery key and keep those handy because you will either need the key or the recovery key (depending on the situation) along with the pass phrase, to gain access to the pool after every reboot. I suggest putting a little test data on the system and go through a number of scenarios to familiarize yourself with the intricacies of how it works, including reboots and simulated drive failure / replacements, before you put any important data on the system. If you prepare in advance and you are careful, you should be fine, but there is always a risk and you should keep a backup of anything that is not replaceable.

I run both WD Blues and Seagate Desktops in my desktop pc. Neither WD or Seagate have failed me, but my Seagates run 5°C cooler. Which will benefit me in the noise department, since it will be placed in my room.

I use Seagate drives in my NAS also and for the same reason, they run cooler. I have 16 drives in my primary NAS and all that heat adds up. I had all WD drives in one NAS and all HGST in the other several years ago and replaced all of those drives with the Seagate drives after doing some testing and I found that the Seagates (for me) ran as much as 10c cooler in the same chassis. I had to make the change because of that.

I don't really like the super small cases because they are so small, it makes them hard to cool. Bigger case, more airflow, cooler drives. Just keep an eye on the temperatures of the drives and be prepared to make adjustments to the fan situation.

 

[donk]

I guess that you need a 240 volt unit, so you will want to be sure to get the right model, but I use the APC Back-UPS Pro 1500 and have been very pleased with it. It provides plenty of run time to allow the server to shutdown gracefully if the power is out long enough for the battery to get to the 'low' threshold. The UPS service that is built into FreeNAS works great to monitor the status of the UPS and initiate shutdown when needed. You might be able to get away with a smaller model, but I went with this one to ensure there was enough run time and I have it protecting my router, network switch and NAS so that the network remains operational during any momentary outage.

The same model is available here in DK, so I'll pick up one of those. I'm also a fan of a larger battery, so it won't shut down immediately.
Power outages are very rare here, but I would hate losing the pool to a power outage.

Will you be using 1GB networking? If you are and if you are only using it for storage, that should be plenty of CPU power. My quad core Xeon idles with less than 10% utilization most of the time only spiking up to 80% when I backup my main storage pool to the external enclosure where the backup storage pool is located. Even transcoding video for Plex, it usually never goes over 50%.

The motherboard support nic teaming, so I might look into that, to get up to 2gbit speeds. But no more than that.

The memory should be fine for storage only. With all that video inventory, I am a bit surprised that you are not planning to run Plex in a jail.

I watch all my movies/shows on my tv through my htpc with Kodi. So it will just get a share, so no need for plex :)

That is the biggest thing that kills a storage pool with encryption, not being able to unlock it after a reboot, and it is almost always down to key management. If you replace a drive in the pool, for example, you need to generate a new key and recovery key and keep those handy because you will either need the key or the recovery key (depending on the situation) along with the pass phrase, to gain access to the pool after every reboot. I suggest putting a little test data on the system and go through a number of scenarios to familiarize yourself with the intricacies of how it works, including reboots and simulated drive failure / replacements, before you put any important data on the system. If you prepare in advance and you are careful, you should be fine, but there is always a risk and you should keep a backup of anything that is not replaceable.

That's good advice :) I will be sure to familiarize myself with the procedures for failures, reboots etc before I deploy the nas.
All of my data which can't be replaced will be backed up to SOS Online Backup, which I have been using for a long time. So everything except the blu ray/dvd rips. I have the discs in the closet.
And the hard to replace data can also be backed up to my 8 tb external hard drive. On that note, I assume freenas can handle that, if i plug the 8 tb external directly into the nas?

I use Seagate drives in my NAS also and for the same reason, they run cooler. I have 16 drives in my primary NAS and all that heat adds up. I had all WD drives in one NAS and all HGST in the other several years ago and replaced all of those drives with the Seagate drives after doing some testing and I found that the Seagates (for me) ran as much as 10c cooler in the same chassis. I had to make the change because of that.

That's very nice to hear. I haven't been able to find much if any feedback on the Seagate drives, so it's reassuring to hear they do well:)

I don't really like the super small cases because they are so small, it makes them hard to cool. Bigger case, more airflow, cooler drives. Just keep an eye on the temperatures of the drives and be prepared to make adjustments to the fan situation.

From the reviews of my chosen case, it should do fairly well. But I will be keeping an eye on temps during disk burn in, and if the temps is too high, I will start with fans. And should that not be enough, I will look into other cases.

 

[Ericloewe]

I know encryption isn't recommended, mostly due to people being bad at managing keys, passwords etc?
But I have been running everything I own encrypted for +8 years, and have never lost data, since i backup my keys to multiple locations, like a box in my bank.

You'll have to backup the GELI metadata for the drives, too.

 

[donk]

You'll have to backup the GELI metadata for the drives, too.

Is that something that can be automated? Or is just upon creation, that the metadata have to be backed up?

 

[Ericloewe]

The metadata won't change, so you only need to back it up once. Plus all new drives. You'll have to do it yourself from the CLI, because the operation isn't supported in the GUI yet (long story...).

 

[Chris Moore]

Is that something that can be automated? Or is just upon creation, that the metadata have to be backed up?

For the testing I did, I did not do a drive replacement, but I did export and import the pool a couple times and reboot a few times and the only thing I needed was the pass phrase, and encryption key or the recovery key. The encryption key and recovery key can both be created and saved from the GUI. After you encrypt the pool, it does not prompt you to save the key or recovery key, but without them you can't access the pool after a reboot because the first thing you need to do is "decrypt" the pool and that wants the encryption key. If you export the pool and then import it, you need the recovery key. It is best to have them both, but I don't claim all understanding of this as I only did it for a couple days of testing. Everything I did was able to be done through the GUI and the documentation was a little cryptic. That is why I said to become familiar with it before you put important data on the system. Just during testing I managed to lock myself out of the pool twice and each time I had to wipe it out and start over.

On that note, I assume freenas can handle that, if i plug the 8 tb external directly into the nas?

People have tried and it usually causes the NAS to crash at some point. The most reliable option is to transfer data in and out of the NAS through the network.

The motherboard support nic teaming, so I might look into that, to get up to 2gbit speeds.

This is generally only useful if you have multiple people accessing the NAS simultaneously.

 

[danb35]

You'll have to do it yourself from the CLI, because the operation isn't supported in the GUI yet (long story...).

Is it just me, or is the encryption support in FreeNAS seriously half-assed?

 

[Ericloewe]

Is it just me, or is the encryption support in FreeNAS seriously half-assed?

It's good enough for the big IT folks who keep meticulous backups (in theory) and don't mind erring on the side of "oops, your data is gone", because that is better than "oops, your cleaning crew made it out of here with your database".

For FreeBSD/GELI noobs, it is seriously half-assed. That's what Corral was supposed to fix (and it did, to its credit), but we know how that story went in the end. That's two years of these small, but necessary improvements that went down the drain with the generic and bland honeycomb logo (sorry, winner of the logo design contest, I never really liked it). Others include mirror device manipulation, improvements to replication and snapshot management and configurable alerts. They're all in the works, but it's going to take a while.

 

[donk]

The metadata won't change, so you only need to back it up once. Plus all new drives. You'll have to do it yourself from the CLI, because the operation isn't supported in the GUI yet (long story...).

Alright, I'll backup the metadata, the encryption key & recovery key to multiple locations.
In case of a drive replacement, will I be able to point out the new drives meta data, or would it be better practice just to backup all drives meta data once again including the new one of course?
But I will be sure to practice first :)

For the testing I did, I did not do a drive replacement, but I did export and import the pool a couple times and reboot a few times and the only thing I needed was the pass phrase, and encryption key or the recovery key. The encryption key and recovery key can both be created and saved from the GUI. After you encrypt the pool, it does not prompt you to save the key or recovery key, but without them you can't access the pool after a reboot because the first thing you need to do is "decrypt" the pool and that wants the encryption key. If you export the pool and then import it, you need the recovery key. It is best to have them both, but I don't claim all understanding of this as I only did it for a couple days of testing. Everything I did was able to be done through the GUI and the documentation was a little cryptic. That is why I said to become familiar with it before you put important data on the system. Just during testing I managed to lock myself out of the pool twice and each time I had to wipe it out and start over.

Is it just one encryption key/recovery key for the entire pool? Or pr. drive/vdev?
So far I have managed not to lock myself out of anything through backup of keys, and use of a password manager (with a strong master password of course :)), but my friends do think I'm ever so slightly paranoid. I just call them careless..

People have tried and it usually causes the NAS to crash at some point. The most reliable option is to transfer data in and out of the NAS through the network.

I'll set up rsync to my pc instead :)

This is generally only useful if you have multiple people accessing the NAS simultaneously.

I thought as much. The only time I could benefit from a higher bandwidth connection to the nas, is when I'm transferring new uncompressed blu ray rips. If it proves to much of a hassle I'll abandon it. Since 40 GB is done in a few minutes anyway :)

 

[Ericloewe]

In case of a drive replacement, will I be able to point out the new drives meta data, or would it be better practice just to backup all drives meta data once again including the new one of course?
But I will be sure to practice first :)

The old drives' metadata won't change. The important part is the salt that is used for the encryption - since the disks aren't reencrypted when you swap one out (the keys just get changed), the salt needs to stay the same.

I thought as much. The only time I could benefit from a higher bandwidth connection to the nas, is when I'm transferring new uncompressed blu ray rips. If it proves to much of a hassle I'll abandon it. Since 40 GB is done in a few minutes anyway :)

You could be a Guinea Pig for SMB multichannel: https://forums.freenas.org/index.php?resources/setting-up-smb-3-multichannel-on-freenas.76/

 

[donk]

The old drives' metadata won't change. The important part is the salt that is used for the encryption - since the disks aren't reencrypted when you swap one out (the keys just get changed), the salt needs to stay the same.

So only the new disk get encrypted, which changes the salt for the pool, resulting in a new key to unlock the pool?

You could be a Guinea Pig for SMB multichannel: https://forums.freenas.org/index.php?resources/setting-up-smb-3-multichannel-on-freenas.76/

I could very well be that before I deploy the nas :)

 

[Ericloewe]

Actually, let me fix what I said:

The GELI metadata stores the disk's Master Key, in encrypted form. It is decrypted using the user keys. Each disk's Master Key is never changed, since it would imply re-encrypting everything.

When adding/swapping disks, FreeNAS generates new user keys, which are used to re-encrypt the Master Keys. So you will have to make a new backup (well, the old backup + the old keys would work, but that seems like more trouble than it's worth).

 

[donk]

Actually, let me fix what I said:

The GELI metadata stores the disk's Master Key, in encrypted form. It is decrypted using the user keys. Each disk's Master Key is never changed, since it would imply re-encrypting everything.

When adding/swapping disks, FreeNAS generates new user keys, which are used to re-encrypt the Master Keys. So you will have to make a new backup (well, the old backup + the old keys would work, but that seems like more trouble than it's worth).

So to make sure I understand this correctly:
-Only one key is required to unlock the pool - the user key (or recovery key).
-The GELI Meta data MUST be backed up, since it contains each drives master key.
-When replacing a drive, it's best practice to do a new backup of all drives meta data, as to not mix old and new.
-For extra good practice, it would be to take the old geli metadata backups and user key, put them in a folder, name that folder old geli backup+date. Backup the metadata again, with the new drives metadata + user key, put it in a folder named current. And of course store this in multiple locations.

Will FreeNAS encrypt the new drive automatically, as part of the re-silver?

 

[Ericloewe]

-Only one key is required to unlock the pool - the user key (or recovery key).

Yes, GELI has two sets: In FreeNAS, the first has a key plus optional passphrase, the second only has a key. You need one of these sets.

-The GELI Meta data MUST be backed up, since it contains each drives master key.

It should be. It's rather easy to lose it and, unlike the partition setup, impossible to recreate without a backup.

When replacing a drive, it's best practice to do a new backup of all drives meta data, as to not mix old and new.

Not just best practice, required with FreeNAS.

-For extra good practice, it would be to take the old geli metadata backups and user key, put them in a folder, name that folder old geli backup+date. Backup the metadata again, with the new drives metadata + user key, put it in a folder named current. And of course store this in multiple locations.

Sure, just don't let yourself get confused.

Will freenas encrypt the new drive automatically, as part of the re-silver?

If you follow the manual, yes.