Is this sane planning? (Larger home use FreeNAS, 40TB)

[Bidule0hm]

You are neither.

But Backblaze ist Backblaze and their drive stats have an evident message when it comes to drive selection.



May backup plan is switching from status quo with no backups at all to RAIDZ3.

Well, given how they use their drives their stats are biased, at best.

RAID isn't a backup.

 

[Yoshimo]

You can cable all of your drives to the HBA via a SAS expander. If you want to direct cable, you could go to an 9211-8i, giving you an extra 4 channels.

After reading up on the subject, this stuff with the SAS Expander sounds more and more interesting. And these SAS Expander Backplanes are even better!

Does someone know where [in the EU] to buy a [used] bare storage server case with 12x or 24x 3.5" HDD bays plus caddys, preinstalled SAS Expander Backplane, preinstalled fans and redundant PSU? This would give me the basics to install my own hardware. (24 bays would be great in order to add a second 12x4TB RAIDZ3 vdev later.)

 

[Yoshimo]

Well, given how they use their drives their stats are biased, at best.

RAID isn't a backup.

I just ordered the 12x Seagate ST4000DM000 drives in order to put an end to that discussion. Don't get me wrong: All your input is wonderful. And I followed the vast majority of the community's advises. But when it comes to drive selection, I just decided to follow my gut feeling.

And about the backups: I won't burn hundreds of optical discs or mirror my data into the cloud. The 3-drives-failover will do the trick for me. And really important data is backed up on some VeraCrypt-encrypted M-Discs now and then.

 

[Arwen]

...
One thing I forgot to mention in my initial post was that encryption will be turned on. But thanks to AES-NI this won't be a problem. So the Intel Pentium G4520 it is.
...

Several things about FreeNAS disk encryption. You may know this already, but others reading may benefit from this information.

1. The encryption is on the disk level, meaning you build your pool on top of FreeBSD Geli encrypted devices.
2. Disk replacement can be a pain, and people HAVE lost their pools due to not understanding the process. My suggestion is to practice before you have real, un-backed up data on the pool. Then document the process as your hardware needs it.
   
3. Every reboot will require entering the encryption keys and or passphrase, before the pool will be available
4. Using encrypted drives REALLY benefits from having complete, (and multiple), backups in case of lost keys or passphrase.
   
5. This type of encryption is mostly useful for theft of the disks and or server. Or returning disk(s) for replacement where you can't wipe them first.
   
6. This type of encryption will NOT protect the data during normal operation. Meaning if a cracker breaks into your server, the encryption is meaningless, (unless they reboot).

It's the last 2 many people mis-understand. While I like the concept of data encryption, it's sometimes better to use it at the file level. For example, my personal password list is GPG encrypted at the file level. Unless I leave a copy un-encrypted, (not likely since it's mostly used R/O), it's pretty safe.

Last, OpenZFS is working on dataset encryption. This would allow some datasets in a pool to be plain, un-encrypted and others to have encrytion of varying strength. However, this will likely not be available for at least a year.

Edit: Minor syntax and phrasing improvements.

 

[Amu W Ramappa]

As @Stux and @Arween mentioned RAIDZ + encryption without full backup is a disaster waiting to happen.

Please consider a backup solution at least for your critical datasets.

Sent from my Nexus 6P using Tapatalk

 

[tvsjr]

At $REALJOB, we consider encryption at rest *of servers* to provide the following safeguards:
Protection from physical theft of servers/drives
Protection from improper data sanitization/device destruction prior to asset disposal

Unless there's a mandate that says, for X type of data, thou shalt encrypt at rest... we don't, assuming we can ensure the system is physically secured (typically in a room with badge access, then inside a locked cabinet with either badge or controlled key access, with the cabinet securely bolted to the floor/wall/etc. in a manner that would require significant time, heavy tools, or explosives to remove) and we ensure proper asset disposal practices are occurring (codified in policy and frequently audited).

So, unless you're worried about someone stealing your FreeNAS box, or you're worried you won't properly wipe the drives before you get rid of them, encryption buys you nothing beyond headaches. From a legal standpoint, here in the US, you can be compelled to turn over encryption keys... I would assume things are similar (or worse) in Germany? If that's the case, encryption isn't going to protect you from the BND or their ilk.

As Her Majesty the Princess above has pointed out, encryption buys you precisely nothing against network-based attacks from a system that's on the network and running.

Long story short, make sure you understand exactly what encryption will protect you from, and ensure that the risks are worth it.

 

[Stux]

ilover will do the trick for me. And really important data is backed up on some VeraCrypt-encrypted M-Discs now and then.

Well, that's a backup plan of sorts ;)

Make sure your desktop drives are set to not spin down and also try to reduce the error recovery timeout.

Remember to burn them in

 

[Bidule0hm]

I just ordered the 12x Seagate ST4000DM000 drives in order to put an end to that discussion. Don't get me wrong: All your input is wonderful. And I followed the vast majority of the community's advises. But when it comes to drive selection, I just decided to follow my gut feeling.

And about the backups: I won't burn hundreds of optical discs or mirror my data into the cloud. The 3-drives-failover will do the trick for me. And really important data is backed up on some VeraCrypt-encrypted M-Discs now and then.

No problem, I just want to inform you, then you can make the decision you want ;)

 

[Arwen]

One note about both ZFS and returning disks for replacement. OpenZFS is so radically different from all file systems before it, that unless the receiver understands a bit about ZFS, it's un-likely they will be able to get much off a single returned disk.

Especially if using compression on all datasets, along with RAID-Zx, so a single file is really stripped across multiple disks. A pool made up with a single Mirrored vDev that does not use any data compression, would be much easier to extract text files using a single disk from the pool. And potentially even binary data files.

 

[Simon Sparks]

I would swap out your LSI SAS 9211-4i for a pair of LSI SAS 9211-8i or LSI SAS 9210-8i and avoid using the ports on the motherboard this will reduce the jitter as both paths will have different latencies.

This would give you 16 ports off the SAS card assuming you install 12 HDDs and you place 6 drives on each SAS card that leaves 2 ports per SAS card free for a potential SSD read and write cache in the future should you want blazing fast IO.

I would maybe suggest a pair of SMALL mirrored SSDs for the ZFS Intent Log ( ZIL )

4 x 1 Gigabit Ethernet Connections = 4000 Megabits per second maximum transfer speed divided by 8 bits = 500 Megabytes per second maximum transfer speed.

As a rule of thumb your ZFS Intent Log ( ZIL ) would be roughly 15 seconds multiplied by the maximum transfer speed per second of 500 Megabytes.

This totals 7,500 Megabytes or roughly 8 Gigabytes so when I say a pair of SMALL mirrored SSDs I really mean SMALL, you can always buy say a 120GB SSD and over provision it so that the 8 Gigabyte partition is moved around on the SSD and it lasts approximately 15 times longer.

 

[diedrichg]

I'm concerned about the CPU choices. You started with a Xeon and decided to go with the Pentium based on suggestions. I recall you saying that this box would be for file storage but you didn't specifically mention that you would use it to serve TV and movies - although you did mention storing Blu-ray movies.

Your storage space is huge so I'm assuming those huge movies will be living on the NAS. I therefore assume you intend to serve them to clients. If this is the case, then you will definitely want to go with at least the Xeon E3-1240 v5.

 

[Yoshimo]

Several things about FreeNAS disk encryption. You may know this already, but others reading may benefit from this information.
The encryption is on the disk level, meaning you build your pool on top of FreeBSD Geli encrypted devices.

You mean it's like VeraCrypt when using neither containers nor the encryption of partitions, but encrypting the whole drive instead? You than get a disk that appears like having been filled with random data, just like after wiping it with DBAN using the PRNG stream. This would be great, as it provides plausible deniability on top of the encryption. [Of course you must not initialize such drives, even when prompted to do so by an OS.]

Disk replacement can be a pain, and people HAVE lost their pools due to not understanding the process. My suggestion is to practice before you have real, un-backed up data on the pool. Then document the process as your hardware needs it.

Okay, I will migrate my data to the NAS without erasing the source drives. Sounds like a good idea, so I can test drive replacement, rebuilt time and general performance as well as making sure I'm able to master the encryption.

Every reboot will require entering the encryption keys and or passphrase, before the pool will be available

Of course. Else it would be quite useless, right? ;)

Using encrypted drives REALLY benefits from having complete, (and multiple), backups in case of lost keys or passphrase.

Objection! Needless to say that I will create backups of my key(s), just as VeraCrypt header backups are mandatory. But backing up the data itself in unencrypted state is self-defeating the purpuose of encryption. It's like writing down your passphrase on a piece of paper and pinning it to your monitor, hidding it in your bedroom closet or putting it in a safe deposit locker at you bank. In both cases an adversary can obtain the data without much effort.

Encryption is encryption, meaning a forgotten passphrase or a lost key makes you data go into oblivion.

However, creating file-level encrypted backups of the most important data on optical media (e.g. M-Discs) is a good idea. I'm already doing this for years. However, as I said, this is only for important data (images, text files, eMail archive, chat logs, password database, work-related stuff, etc). Backing up the media archive isn't worth the trouble and cost.

This type of encryption is mostly useful for theft of the disks and or server. Or returning disk(s) for replacement where you can't wipe them first.

It can also be useful in other cases...

This type of encryption will NOT protect the data during normal operation. Meaning if a cracker breaks into your server, the encryption is meaningless, (unless they reboot).

The passphrase decrypts the key, and the key decrypts the data. So the key needs to be held in RAM during operation. There is no way around this. You can purge an unlocked PGP private key from RAM after decryption of a message, but you can't purge a permanently needed key. But I consider an attack from the outside unlikely, as the NAS won't be reachable from the internet.

Last, OpenZFS is working on dataset encryption. This would allow some datasets in a pool to be plain, un-encrypted and others to have encrytion of varying strength. However, this will likely not be available for at least a year.

Not needed. No unencrypted data in my house. Although I could use AES for the media archive and cascaded encryption on important data.

I'm concerned about the CPU choices. You started with a Xeon and decided to go with the Pentium based on suggestions. I recall you saying that this box would be for file storage but you didn't specifically mention that you would use it to serve TV and movies - although you did mention storing Blu-ray movies.

Your storage space is huge so I'm assuming those huge movies will be living on the NAS. I therefore assume you intend to serve them to clients. If this is the case, then you will definitely want to go with at least the Xeon E3-1240 v5.

Initially I thought the same way as you do, but was convinced to switch from the Xeon E3-1240v5 to a much cheaper Pentium G4520. But the Xeon E3-1240v5 would feel better. Maybe I will buy a Xeon E3-1240v5 or even a Xeon E3-1245v5, as the latter has an integrated GPU so attaching a monitor would be possible.

 

[diedrichg]

Initially I thought the same way as you do, but was convinced to switch from the Xeon E3-1240v5 to a much cheaper Pentium G4520. But the Xeon E3-1240v5 would feel better. Maybe I will buy a Xeon E3-1240v5 or even a Xeon E3-1245v5, as the latter has an integrated GPU so attaching a monitor would be possible.

If you ONLY run a file server then the Pentium is plenty. If you plan to serve the videos through Plex or Emby or you plan to run any VMs then the Xeon is a must.

As mentioned before, we can't stress enough how useful the IPMI is, spend the money on the board with the IPMI and save the money on the processor without graphics.

 

[Yoshimo]

I would swap out your LSI SAS 9211-4i for a pair of LSI SAS 9211-8i or LSI SAS 9210-8i and avoid using the ports on the motherboard this will reduce the jitter as both paths will have different latencies.

Hey, thanks for the tip! Didn't know there are different latencies - but I should have suspected it. It makes total sense.

This would give you 16 ports off the SAS card assuming you install 12 HDDs and you place 6 drives on each SAS card that leaves 2 ports per SAS card free for a potential SSD read and write cache in the future should you want blazing fast IO.

What about a single LSI SAS 9211-4i plus an SAS Expander? An SAS Expander Backplane with 12 ports would be nice as well. Of course I would need a 12 bay storage server case for this, but it should work, correct?

But I admit it: That topic with SAS Controllers, SAS Expanders and SAS Expander Backplanes is new to me. So far I got it this way:

(1) SAS cables can be plugged into SATA drives, too.
(2) Things like the LSI SAS 9211-4i are called Host Bus Adapters.
(3) A Host Bus Adapter is not a Controller!
(4) Host Bus Adapter is abbreviated as HBA.
(5) The LSI SAS 9211-4i is a SAS HBA.
(6) On a SAS HBA you find one or more Mini-SAS ports.
(7) One Mini-SAS port can be connected to up to 4 SAS or SATA drives, (There are adapter cables available for this.)
(8) With the help of a SAS Expander, a single Mini-SAS port can make use of up to 256 SAS or SATA drives.
(9) There are SAS Expander Backplanes where SAS or SATA HDDs can easily be plugged into.

But: Do SAS Expander Backplanes have integrated SAS Expanders? After seeing some images of SAS Expanders and SAS Expander Backplanes, it seems as the SAS Expander Backplanes are simply components for signal distribution and power supply. So you need to plug the SAS Expander Backplane via Mini-SAS into the SAS Expander sitting on a PCIe slot and the SAS Expander via Mini-SAS into the SAS HBA sitting on another PCIe slot, right?

I would maybe suggest a pair of SMALL mirrored SSDs for the ZFS Intent Log ( ZIL )

4 x 1 Gigabit Ethernet Connections = 4000 Megabits per second maximum transfer speed divided by 8 bits = 500 Megabytes per second maximum transfer speed.

As a rule of thumb your ZFS Intent Log ( ZIL ) would be roughly 15 seconds multiplied by the maximum transfer speed per second of 500 Megabytes.

This totals 7,500 Megabytes or roughly 8 Gigabytes so when I say a pair of SMALL mirrored SSDs I really mean SMALL, you can always buy say a 120GB SSD and over provision it so that the 8 Gigabyte partition is moved around on the SSD and it lasts approximately 15 times longer.

Please remember: The NAS will be plugged into a single gigabit port on a consumer-grade router. So 125 MB/s is the highest speed possible.

And I have plenty of smaller SSDs left, so setting up this ZIL log (never heard of it before) will be no problem.

 

[Yoshimo]

If you ONLY run a file server then the Pentium is plenty. If you plan to serve the videos through Plex or Emby or you plan to run any VMs then the Xeon is a must.

To be frank: I just want a big, fat network drive showing up in my Windows Explorer. That's it. I want to play the media files stored on it with the VLC media player.

In case an encode of audio or video files is wanted, the Intel Xeon E5-1680v4 on my workstation will do it before the file is transferred onto the NAS.

As mentioned before, we can't stress enough how useful the IPMI is, spend the money on the board with the IPMI and save the money on the processor without graphics.

Okay, I will get the board with IPMI. But I'll likely never use it, as I don't know how.

 

[Kevin Horton]

If you ONLY run a file server then the Pentium is plenty. If you plan to serve the videos through Plex or Emby or you plan to run any VMs then the Xeon is a must.

My first FreeNAS server had a G3258 CPU, and served videos through Plex without breaking a sweat (less than 20% CPU usage per client). Mind you, I had set up the system so it never needed to transcode - everything was served straight from the video files on the NAS.

 

[Stux]

then you will definitely want to go with at least the Xeon E3-1240 v5.

I recommend the E3-1230v5 over the 1240. 1240 is 4% faster and 10% more expensive.

1230 is significantly faster than the 1220.

1245 is a waste of money with the SSM-F board. Does it even have the hardware to use the iGPU?

 

[diedrichg]

I recommend the E3-1230v5 over the 1240. 1240 is 4% faster and 10% more expensive.

1230 is significantly faster than the 1220.

1245 is a waste of money with the SSM-F board. Does it even have the hardware to use the iGPU?

PassMark has the 1240 performing 7% better and I see U.S. pricing at 10% higher than the 1230. That 7% performance increase roughly translates to an additional 720p transcode. Is it worth the extra $28? Probably not. But who doesn't want moar pawer!

 

[Arwen]

@Yoshimo
Sounds like you now know what you are getting into with encryption on FreeNAS.
Good luck.

 

[Yoshimo]

@Yoshimo
Sounds like you now know what you are getting into with encryption on FreeNAS.
Good luck.

I hope so. Encryption always fascinated me. Sometimes it makes you mad, and sometimes it makes you think: "Hey; math is so cool, crazy and beautiful ... it's like magic!"